Sudden stake pool topology/sync issue

Staking & Delegation Operate a Stake Pool
1

Hey, I’m a new SPO who just set up a pool. I’m an experienced systems admin but just today after the pool running smoothly for a couple of weeks, I encountered the following issue:

Suddenly I noticed my block producer node in gLiveView is saying ‘Syncing 99.9%’ with no incoming or outgoing connections. I check my relay, and topology logs are saying this (since the last reboot of the node, which happens every 24h to run new topology)

{ "resultcode": "502", "datetime":"2022-02-17 23:04:45", "clientIp": "<SNIP>", "msg": "invalid blockNo []" }

I restarted both my block producer and my relay and now both are just stuck on “Syncing 99.9%”, with 1 incoming 1 outgoing connection on the BP, and 1 incoming 23 outgoing on the relay

Anyone know why this might have happened and how I can prevent it from occurring in future? There doesn’t seem to be any general connectivity issue on the servers

2

your BP not starting because has only one connection with your relay… you can add inside the topology file the IOHK nodes (will be used only for OUT/syncing connections) and restart the node… or forget the BP till you will fix the relay issue

now for relay… try to restart the node

sudo systemctl restart cardano-node 
journalctl -e -f -u cardano-node

and check the logs since the restart… check if there are any errors