You’re right, this is exactly what happens currently. I added a comment in the documentation a year ago about this.
Note : If you want to connect to IPv4 and IPv6 relays, you must either not specify host addresses when starting cardano-node or make sure to specify both an IPv4 and IPv6 host address.
I once made a PR to change this behavior so it would allow outbound IPv6 without specifying a host address, but it was not accepted:
So currently unfortunately this is expected behavior.