Block producer freezing at the ti

Hi Folks,
When my block producer starts, the tip node reaches the top of the blockchain, then stops in gLiveView…, The tip chain keeps counting up, but the tip node never increases. It polls every two seconds and the tip node starts as a happy face, then a line for a mouth, Then sad face, then 100% synching, then it sticks at 99.9% synching. Any suggestions are appreciated.

Thank you

when this issue started? after the alonzo HF ?

After Alonso ….

In prior messages, I previously used AWS with1.27 and never went live due to security issues that I could not solve due to inexperience with Network security.

Now I have my own hardware with three separate PCs running Ubuntu, no VMs, network+ training, Gig business fiber, current model firewall properly configured with port forwarding and private subnets. I installed a Coin cashew guided fresh install of 1.29.0 just before Alonso. I watched it tick over …. Relays have 15 nodes, crontab working fine …

Thanks

journalctl -e -f -u cardano-node

I’m not at my system right now, but data was scrolling when I executed that exact command line this morning.

Hi, I just watched it count up on the BP and then suddenly journalctl returns:
Trying to connect to <RELAY 1 IP:6000>
Skipping Peer <Relay 2 IP:6000>
Failed to Start All required subscriptions
In gLiveView, pressing ‘p’ return relay1 as in and relay2 as out. Will post images soon.


So, the producer is up and running but how about the relay?

I have two relays. I am trying to determine if Relay1 has a hardware issue - both are i5 with 8GB RAM, but Relay 1 is clearly much slower than Relay 2 for some reason with the same SSD. Relay 2 seems OK.

I’ve just configured the BP to using only Relay2. i removed Relay1 from mainnet-topology.json and executing ‘sudo systemctl restart cardano-node’. With Relay2 only, I watched the BP get to 13 :), then the Tip (node) just stopped at 40414310 while Tip (ref) continues to increment.

gLiveView displays 1 peer in and 1 peer out - both are the private IP of Relay2.

Using a single relay displaying status of 2 :smile: with peers 16 out and 3 in, on the same subnet as the BP, the BP displays synching (100.00%) and is about 4500 difference Tip (ref) to Tip (node).

All traffic is internal on new, brand name, hardware firewall/switch rated beyond gigabit speeds on gigabit fiber. I can ping on the subnet in both directions. BP to Relay and Relay to BP.

Add to your producer, inside the topology file the IOHK nodes, there is OUT connection so you will not need to modify any firewall rule

Check if now it’s staying sync

It did not work … I added relays-new.cardano-mainnet.iohk.io port 3001 to my BP. The BP cannot talk outside of the subnet. Then restarted the service. It did count up again and is now stuck at Tip (node) 40420873 while to tip (ref) is counting up at 40421232 at this moment …

gLiveView reports 1 In and 1 Out. Both are my Relay 2. Out Port 6000 (private IP) and in port 45nnn (private IP).

Jouranlctl reported Domain: relays-new.cardano-mainnet.iohk.io the message is Failed to start all required subscriptions

Ok, do u have OUT firewall rules implemented? I think can be related with the internet traffic
The private relay has the same issue?

Configuration is:
Relay 1 and Relay 2 on 10.45.10.nnn for example
BP on 10.45.10.nnn also
Port forward from 6000 public IP addresses to Relay 1 and Relay 2
Relay 1 and Relay 2 can talk to Internet on DMZ config
BP can only talk on subnet - no other out or in traffic for BP. Not even SSH, tunnel or other …

The private relay has the same issue?

I’m using a single Relay now that works 100%. It stays pretty close to always having a happy face …

Try to aet the TraceMempool to false and start the BP … same issue?

Or monitor both servers with htop

I’m checking firewall policies one by one. Will try tracemempool soon.

Your help is much appreciated.

Thank you

There is another thread(s) on this. My guess is that the database structure changed along with the genesis files. Sooo … At this time - because of my other shenanigans - I’m rebuilding all Cardano databases and will regen my KES keys as soon as I get the block chain synched. I hope to be able to report back tonight …

1 Like