The issue you describe could be a firewalling problem. But, without knowing how your network is set up it is hard to say.
Do some testing using cardano-cli ping from each node to the other.
I would recommend that you put both your relay and your block producer in P2P mode. For some example mainnet-topology.json files for each check out this post.