Node not starting

Hi there,
I’ve updated my gLiveView.sh and cardano-node on my both Relay and BP nodes.

Relay node (with port 6000) is up and running fine:
Screenshot 2023-02-27 at 15.23.58

But my BP Node (with port 6001) hangs at starting stage forever?

Screenshot 2023-02-27 at 15.21.25

Can you please tell me what am I missing?

sometimes, when you restart the node often you will hit this issue and you will need to perform the steps

  • stop the node
sudo systemctl stop cnode
  • rename ledger, immutable and volatile folders
cd $CNODE_HOME/db
ls -l
mv immutable imm
mv ledger led
mv volatile vol
ls -l
  • start/stop the node
sudo systemctl start cnode
stop the node after 10 sec
sudo systemctl stop cnode
  • you should also see now the new folders ledger, immutable and volatile
    delete the new folders created (not the old one renamed)
ls -l
rm -R ledger
rm -R immutable
rm -R volatile
ls -l
  • rename back the original folders
mv imm immutable
mv led ledger
mv vol volatile
ls -l
  • start the node and check glive (you should see now Mem RSS slowly increasing)
2 Likes

@Alexd1985 What is actually happening there? What do the logs show when this “issue” happens? I don’t think I have seen that happen???

1 Like

@Alexd1985 Thank you, it helped. Now it’s up and running.
Screenshot 2023-02-27 at 15.40.29

2 Likes

Nothing happens I guess… but from time to time when I use sudo systemctl restart cnode instead stop/start its staying in starting for hours.
I believe because it was not stoped correctly…

That is interesting. I have never seen it happen and I restart my block producer and 1 or 2 relays at least every day for 1.5 years.

One thing to check, maybe, is that your systemctl service file has the following:

Restart=always
RestartSec=5
TimeoutStopSec=300
KillSignal=SIGINT

In particular, the RestartSec setting. The manpage says it is only 100ms by default:

RestartSec=

Configures the time to sleep before restarting a service (as
configured with Restart=). Takes a unit-less value in
seconds, or a time span value such as “5min 20s”. Defaults to
100ms.

Unrelated but important is the TimeoutStopSec=300 because without it the default is 90 seconds and sometimes cardano-node takes a bit longer to cleanly stop. If graceful shutdown doesn’t end within that time the systemd sends the process a SIGKILL and that will cause cardano-node to recheck its entire blockchain database on restart which could take a few hours. I have been bitten by that one a few times.
Note also that cardano-node expects SIGINT, not SIGTERM for graceful shutdown.

here is my config

Description=Cardano Node
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=600
StartLimitBurst=5

[Service]
Type=simple
Restart=on-failure
RestartSec=60
User=user
LimitNOFILE=1048576
WorkingDirectory=/opt/cardano/cnode/scripts
ExecStart=/bin/bash -l -c "exec /opt/cardano/cnode/scripts/cnode.sh"
ExecStop=/bin/bash -l -c "exec /opt/cardano/cnode/scripts/cnode.sh -s"
KillSignal=SIGINT
SuccessExitStatus=143
SyslogIdentifier=cnode
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target

As I said its not happening each time (my relays auto restart once/24 hours)

1 Like

As @Terminada said: That might be a bit short.

CoinCashew also has this at 300: https://www.coincashew.com/coins/overview-ada/guide-how-to-build-a-haskell-stakepool-node/part-ii-configuration/creating-startup-scripts

Your 60 probably come from cnode.sh: https://github.com/cardano-community/guild-operators/blob/alpha/scripts/cnode-helper-scripts/cnode.sh#L76-L98

3 Likes

Thanks, I will set it to 300

Another follow-up question:
My supposedly BP node is showing “Relay - Mainnet” instead of “Core - Mainnet”. I thought it’ll automatically show as Core once syncing is complete, but it is not. What I might have done wrong?
Screenshot 2023-02-28 at 10.04.22

Have you entered the pool ticker in the env file on the BP node?

No, not previously. Now that you told me, I’ve added it and restarted it. But still, it is showing ‘Relay’

it will take few minutes to resync again - after it is completed should change to Core

The sync is complete.

Screenshot 2023-02-28 at 11.27.54

and you are certain that you have added you ticker and uncommented the line in env file on the BP node:
POOL_NAME=“xxx”

Yes
Screenshot 2023-02-28 at 11.31.19

Is there any reason the line POOL_FOLDER to be uncommented?

@SPICE Thank you. I’ve commented the line again and restarted the node. Now it says ‘Core’.
Screenshot 2023-02-28 at 11.45.11

1 Like