Sudden stake pool topology/sync issue

Hey, I’m a new SPO who just set up a pool. I’m an experienced systems admin but just today after the pool running smoothly for a couple of weeks, I encountered the following issue:

Suddenly I noticed my block producer node in gLiveView is saying ‘Syncing 99.9%’ with no incoming or outgoing connections. I check my relay, and topology logs are saying this (since the last reboot of the node, which happens every 24h to run new topology)

{ "resultcode": "502", "datetime":"2022-02-17 23:04:45", "clientIp": "<SNIP>", "msg": "invalid blockNo []" }

I restarted both my block producer and my relay and now both are just stuck on “Syncing 99.9%”, with 1 incoming 1 outgoing connection on the BP, and 1 incoming 23 outgoing on the relay

Anyone know why this might have happened and how I can prevent it from occurring in future? There doesn’t seem to be any general connectivity issue on the servers

your BP not starting because has only one connection with your relay… you can add inside the topology file the IOHK nodes (will be used only for OUT/syncing connections) and restart the node… or forget the BP till you will fix the relay issue

now for relay… try to restart the node

sudo systemctl restart cardano-node 
journalctl -e -f -u cardano-node

and check the logs since the restart… check if there are any errors