Nodes slower to restart following 1.25.1 update, anyone?

Staking & Delegation
#1

Hi all,

I just wanted to gauge the opinions on the topic as per title. I run the nodes as a systemd service and am used to them responding within roughly 15 seconds to a systemctl restart command. Since the update the time it takes is a minute or more, especially one node, which takes 5+ minutes.

I have explored the logs and found nothing sinister. One message is unusual though, and seems to point at the overall number of connected peers on one of the relays (15 in this case). Another observation is that the ā€œslowerā€ node is the one responsible for running sendmytip. Lastly, I have also found that topologyUpdater is no longer updating the topology file it points to.

Has anyone else experienced similar issues?

Cheers,

A

#2

Hello,

I didnā€™t observe that; tbh I didnā€™t cared how much it takes because I have 3 relays and are seted to restart at 4 hours differenceā€¦ for me 1 minute will not be a real problemā€¦

I looked in grafana and seems to have the same behavior with one of my node

image

regarding topology updater issue, check if the service is still active, if it pushed the config to topology files, etc

you can also check the logs with journalctl -e -f -u cnode-tu-fetch.service ( * cnode-tu-fetch.service : fetches a fresh topology file before cnode.service file is started/restarted)

cheers,

2 Likes
#3

Probably needless to say, but does the service use SIGINT kill signal?

1 Like
#4

Yes, it uses SIGINT as it always has. I have changed nothing except for the node software update.

Seeing as topologyUpdater also stopped working, i imagine it has to do with some of the metrics changes in the config. I have not had a chance to test this, but as soon as i do Iā€™ll post my findings here.

Thank you all for your contribution so far, keep it coming! The more the merrier!

Cheers A

1 Like
#5

thank you @Alexd1985 and @bjorn,

I have checked and topologyUpdater is still running and sending back a happy message in the logs. Iā€™ll have to do some further digging to find out why itā€™s output is not updating.

Cheers,

A

PS on a side note @Alexd1985, I appreciate what you say about the relaysā€™ delay (pun intended), I have three also.

But, letā€™s give the hypothetical were your BP goes down for whatever reason (power surge, kernel reboot, space-critters invasion, etc), I like to be reasonably sure that it will bounce back like a basketball, not after 5 minutes.

#6

u are right, maybe they are aware about this behavior and it will be fixed on next updateā€¦

I remember I have this behavior also on older version, but they fixed in 1.24.2 or 1.23.x (donā€™t remember well)

you said you have a node which takes 5+ minutesā€¦ has a different hw configuration?

1 Like
#7

Mmm, ok, thatā€™s interesting. This is my first experience of it. I remember there were some issues with kill signals but sice SIGINT was accepted as standard I have never had nodes take longer than 45 seconds to come back after a full reboot.

Anyway, your input is, as usual, greatly appreciated Alex :pray:. Iā€™m sure weā€™ll get to the bottom of it.

Cheers, A

1 Like
#8

Just going to jump in here to say a big thank you to all commenting here and on other threads :clap:

To see fellow pool operators and the Cardano community as a whole, helping and supporting is such a beautiful thing.

There are a number of things that seem a ā€˜bit oddā€™ at the moment with the world, so to have a space to come to and witness collaboration, support and goodwill to others is truly heart warming :purple_heart:

3 Likes
#9

Hi all,

thank you for your contributions :point_up_2:

As discussed, I finally got around to check what was going on, and it was just a -config file issue. All sorted. It also fixed the topologyUpdater thing.

All the best,

A

#10

hey @Alexd1985, nah, same HW everywhere. It seems like I had left some unwanted metrics in the new config file. This was causing topologyUpdater to have a hissy fit, and some confusion in the peer handling.

Itā€™s all good now, back to 15-second bounces as usual :smile:

Thank you for your time,

A

1 Like
#11

Hi, do i need to update each time, when there is new version? can it be done automatically? I have 1.24.2 version. I am stake pool operator.

#12

Hello,

Nope, as a pool operator you need to prepare/do these updates manually each time an update is required.

Cheers,

1 Like
#13

+1 to what @Alexd1985 said. Check out the ā€œbuilding your nodeā€ section of the stake pool school. It is now up to date (albeit not all of it) and gives instructions how to update your software. Itā€™s a rather quick process, particularly if you parallelize it over all your infrastructure.

Iā€™m on mobile but will post links later.

Cheers A

PS Installing Cardano-node - Stake pool course

The section youā€™re interested in is down the bottom. Checkout with version 1.25.1

1 Like