NODE stuck in start

Developers Cardano Projects
1

Hello,

Need some help, my NODE was working fine until about 2hrs ago.
It is now stuck in starting, see below logs.
I do use this NODE as my monitoring server as well and had tmux going for Prometheus and node exporter (shut those down before troubleshooting). See Sending SIGINT to cardano-node process, not a clean shutdown?

I rebooted server, still stuck in starting.
env settings, topology file correct (checked them but no reason they wouldn’t be).
node0.socket missing but quessing it is because NODE isn’t fully starting.
I uncommented the line: SOCKET="${CNODE_HOME}/sockets/node0.socket, no change.

Suggestions?

Last lines of journalctl -e -f -u cnode

Jan 27 04:07:27 xxx cnode[11184]: Sending SIGINT to cardano-node process…
– Reboot –
Jan 27 04:08:07 xxx cnode[1943]: Listening on http://127.0.0.1:12798

CPU @ 100%
2129 node1 20 0 1.001t 121336 57796 R 103.7 0.7 40:25.82 cardano-node

ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2023-01-27 04:07:58 CET; 40min ago
Process: 1179 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 1230 (sshd)
Tasks: 7 (limit: 4915)
CGroup: /system.slice/ssh.service
├─1230 /usr/sbin/sshd -D
├─2158 sshd: node1 [priv]
├─2335 sshd: node1@pts/0
├─2336 -bash
├─7369 sudo systemctl status ssh
├─7699 systemctl status ssh
└─7700 pager

From env, running montoring server from it.

“hasEKG”: 12788,
“hasPrometheus”: [
“127.0.0.1”,
12798

/opt/cardano/cnode/files$ ls -l
total 1060
-rw-r–r-- 1 node1 node1 9459 Jan 20 02:50 alonzo-genesis.json
-rw-r–r-- 1 node1 node1 1056360 Jan 20 02:50 byron-genesis.json
-rw-r–r-- 1 node1 node1 2626 Jan 20 05:34 config.json
-rw-r–r-- 1 node1 node1 2632 Jan 20 02:50 dbsync.json
-rw-r–r-- 1 node1 node1 2486 Jan 20 02:50 shelley-genesis.json
-rw-r–r-- 1 node1 node1 2889 Jan 27 05:20 topology.json
node1@vmi1135243:/opt/cardano/cnode/files$

nano ${HOME}/.bashrc

if ! shopt -oq posix; then
if [ -f /usr/share/bash-completion/bash_completion ]; then
. /usr/share/bash-completion/bash_completion
elif [ -f /etc/bash_completion ]; then
. /etc/bash_completion
fi
fi
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

[ -f “/home/node1/.ghcup/env” ] && source “/home/node1/.ghcup/env” # ghcup-env
export CNODE_HOME=/opt/cardano/cnode

2

type
sudo systemctl restart cnode
sudo journalctl -e -f -u cnode
and check for errors. If everything is fine (no errors)
apply

sometimes, when you stop/start/restart the node often you will hit this issue and you will need to perform the steps

  • stop the node
sudo systemctl stop cnode
  • rename ledger, immutable and volatile folders
cd $CNODE_HOME/db
ls -l
mv immutable imm
mv ledger led
mv volatile vol
ls -l
  • start/stop the node
sudo systemctl start cnode
stop the node after 10 sec
sudo systemctl stop cnode
  • you should also see now the new folders ledger, immutable and volatile
    delete the new folders created (not the old one renamed)
ls -l
rm -R ledger
rm -R immutable
rm -R volatile
ls -l
  • rename back the original folders
mv imm immutable
mv led ledger
mv vol volatile
ls -l
  • start the node and check glive (you should see now Mem RSS slowly increasing)
3

That did the trick, thank you sir.

1 Like