That seems odd to me. If your relays were remaining synced then this means they were pulling blocks from other relays and therefore should have been able to pull blocks from your block producer equally well. Maybe there is something in the P2P protocol which caused your relay to label your block producer as “tainted” or “untrusted” because it wasn’t properly synced and thereby refused to re-establish a connection with it, even after it became synced???
It is frustrating to get a problem like this which you never properly get to the bottom of, because restarting the service just made things work. I feel your frustration and I would like to know the explanation too.