# Block producer freezing at the ti **URL:** \ **Category:** Setup a Stake Pool\ **Created:** [17 September 2021 16:17 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839 "2021-09-17T16:17:01Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![timeforprunejuice](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/timeforprunejuice/32/37147_2.png) [@timeforprunejuice](https://forum.cardano.org/u/timeforprunejuice)\ **Post date:** [17 September 2021 16:17 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/1 "2021-09-17T16:17:01Z")
Hi Folks, When my block producer starts, the tip node reaches the top of the blockchain, then stops in gLiveView…, The tip chain keeps counting up, but the tip node never increases. It polls every two seconds and the tip node starts as a happy face, then a line for a mouth, Then sad face, then 100% synching, then it sticks at 99.9% synching. Any suggestions are appreciated. Thank you ---
**Author:** ![Alexd1985](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/alexd1985/32/23759_2.png) [@Alexd1985](https://forum.cardano.org/u/Alexd1985)\ **Post date:** [17 September 2021 16:18 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/2 "2021-09-17T16:18:39Z")
when this issue started? after the alonzo HF ? ---
**Author:** ![timeforprunejuice](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/timeforprunejuice/32/37147_2.png) [@timeforprunejuice](https://forum.cardano.org/u/timeforprunejuice)\ **Post date:** [17 September 2021 18:18 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/3 "2021-09-17T18:18:54Z")
After Alonso …. In prior messages, I previously used AWS with1.27 and never went live due to security issues that I could not solve due to inexperience with Network security. Now I have my own hardware with three separate PCs running Ubuntu, no VMs, network+ training, Gig business fiber, current model firewall properly configured with port forwarding and private subnets. I installed a Coin cashew guided fresh install of 1.29.0 just before Alonso. I watched it tick over …. Relays have 15 nodes, crontab working fine … Thanks ---
**Author:** ![Alexd1985](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/alexd1985/32/23759_2.png) [@Alexd1985](https://forum.cardano.org/u/Alexd1985)\ **Post date:** [17 September 2021 18:54 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/4 "2021-09-17T18:54:37Z")
journalctl -e -f -u cardano-node ---
**Author:** ![timeforprunejuice](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/timeforprunejuice/32/37147_2.png) [@timeforprunejuice](https://forum.cardano.org/u/timeforprunejuice)\ **Post date:** [17 September 2021 20:34 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/5 "2021-09-17T20:34:03Z")
I’m not at my system right now, but data was scrolling when I executed that exact command line this morning. ---
**Author:** ![timeforprunejuice](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/timeforprunejuice/32/37147_2.png) [@timeforprunejuice](https://forum.cardano.org/u/timeforprunejuice)\ **Post date:** [18 September 2021 01:11 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/6 "2021-09-18T01:11:08Z")
Hi, I just watched it count up on the BP and then suddenly journalctl returns: Trying to connect to \ Skipping Peer \ Failed to Start All required subscriptions In gLiveView, pressing ‘p’ return relay1 as in and relay2 as out. Will post images soon. ![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/f/2/f233f94b3721759065fdd11b615da8defe84b02b.jpeg) ![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/5/b/5b210bc152a5f3d46c25f9be905036b5b3651217.jpeg) ---
**Author:** ![Alexd1985](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/alexd1985/32/23759_2.png) [@Alexd1985](https://forum.cardano.org/u/Alexd1985)\ **Post date:** [18 September 2021 05:13 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/7 "2021-09-18T05:13:52Z")
So, the producer is up and running but how about the relay? ---
**Author:** ![timeforprunejuice](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/timeforprunejuice/32/37147_2.png) [@timeforprunejuice](https://forum.cardano.org/u/timeforprunejuice)\ **Post date:** [18 September 2021 16:05 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/8 "2021-09-18T16:05:28Z")
I have two relays. I am trying to determine if Relay1 has a hardware issue - both are i5 with 8GB RAM, but Relay 1 is clearly much slower than Relay 2 for some reason with the same SSD. Relay 2 seems OK. I’ve just configured the BP to using only Relay2. i removed Relay1 from mainnet-topology.json and executing ‘sudo systemctl restart cardano-node’. With Relay2 only, I watched the BP get to 13 :), then the Tip (node) just stopped at 40414310 while Tip (ref) continues to increment. gLiveView displays 1 peer in and 1 peer out - both are the private IP of Relay2. ---
**Author:** ![timeforprunejuice](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/timeforprunejuice/32/37147_2.png) [@timeforprunejuice](https://forum.cardano.org/u/timeforprunejuice)\ **Post date:** [18 September 2021 17:16 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/9 "2021-09-18T17:16:14Z")
Using a single relay displaying status of 2 😄 with peers 16 out and 3 in, on the same subnet as the BP, the BP displays synching (100.00%) and is about 4500 difference Tip (ref) to Tip (node). All traffic is internal on new, brand name, hardware firewall/switch rated beyond gigabit speeds on gigabit fiber. I can ping on the subnet in both directions. BP to Relay and Relay to BP. ---
**Author:** ![Alexd1985](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/alexd1985/32/23759_2.png) [@Alexd1985](https://forum.cardano.org/u/Alexd1985)\ **Post date:** [18 September 2021 17:31 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/10 "2021-09-18T17:31:29Z")
Add to your producer, inside the topology file the IOHK nodes, there is OUT connection so you will not need to modify any firewall rule Check if now it’s staying sync ---
**Author:** ![timeforprunejuice](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/timeforprunejuice/32/37147_2.png) [@timeforprunejuice](https://forum.cardano.org/u/timeforprunejuice)\ **Post date:** [18 September 2021 17:50 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/11 "2021-09-18T17:50:52Z")
It did not work … I added [relays-new.cardano-mainnet.iohk.io](http://relays-new.cardano-mainnet.iohk.io) port 3001 to my BP. The BP cannot talk outside of the subnet. Then restarted the service. It did count up again and is now stuck at Tip (node) 40420873 while to tip (ref) is counting up at 40421232 at this moment … gLiveView reports 1 In and 1 Out. Both are my Relay 2. Out Port 6000 (private IP) and in port 45nnn (private IP). Jouranlctl reported Domain: [relays-new.cardano-mainnet.iohk.io](http://relays-new.cardano-mainnet.iohk.io) the message is Failed to start all required subscriptions ---
**Author:** ![Alexd1985](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/alexd1985/32/23759_2.png) [@Alexd1985](https://forum.cardano.org/u/Alexd1985)\ **Post date:** [18 September 2021 18:00 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/12 "2021-09-18T18:00:21Z")
Ok, do u have OUT firewall rules implemented? I think can be related with the internet traffic The private relay has the same issue? ---
**Author:** ![timeforprunejuice](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/timeforprunejuice/32/37147_2.png) [@timeforprunejuice](https://forum.cardano.org/u/timeforprunejuice)\ **Post date:** [18 September 2021 18:04 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/13 "2021-09-18T18:04:18Z")
Configuration is: Relay 1 and Relay 2 on 10.45.10.nnn for example BP on 10.45.10.nnn also Port forward from 6000 public IP addresses to Relay 1 and Relay 2 Relay 1 and Relay 2 can talk to Internet on DMZ config BP can only talk on subnet - no other out or in traffic for BP. Not even SSH, tunnel or other … ---
**Author:** ![timeforprunejuice](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/timeforprunejuice/32/37147_2.png) [@timeforprunejuice](https://forum.cardano.org/u/timeforprunejuice)\ **Post date:** [18 September 2021 18:07 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/14 "2021-09-18T18:07:41Z")
> The private relay has the same issue? I’m using a single Relay now that works 100%. It stays pretty close to always having a happy face … ---
**Author:** ![Alexd1985](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/alexd1985/32/23759_2.png) [@Alexd1985](https://forum.cardano.org/u/Alexd1985)\ **Post date:** [18 September 2021 18:21 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/15 "2021-09-18T18:21:54Z")
Try to aet the TraceMempool to false and start the BP … same issue? Or monitor both servers with htop ---
**Author:** ![timeforprunejuice](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/timeforprunejuice/32/37147_2.png) [@timeforprunejuice](https://forum.cardano.org/u/timeforprunejuice)\ **Post date:** [18 September 2021 18:33 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/16 "2021-09-18T18:33:36Z")
I’m checking firewall policies one by one. Will try tracemempool soon. Your help is much appreciated. Thank you ---
**Author:** ![timeforprunejuice](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/timeforprunejuice/32/37147_2.png) [@timeforprunejuice](https://forum.cardano.org/u/timeforprunejuice)\ **Post date:** [19 September 2021 21:32 UTC](https://forum.cardano.org/t/block-producer-freezing-at-the-ti/74839/17 "2021-09-19T21:32:19Z")
There is another thread(s) on this. My guess is that the database structure changed along with the genesis files. Sooo … At this time - because of my other shenanigans - I’m rebuilding all Cardano databases and will regen my KES keys as soon as I get the block chain synched. I hope to be able to report back tonight …