Odd errors in code log, missed a block last night

Try to rebuild again the node

1 Like

ok, will report back

Oct 24 07:38:10 staking01 systemd[1]: Stopped Cardano Node.
Oct 24 08:06:00 staking01 systemd[1]: Started Cardano Node.
Oct 24 08:06:01 staking01 cnode[253069]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Oct 24 08:06:02 staking01 cnode[253535]: Listening on http://127.0.0.1:12798
Oct 24 08:09:42 staking01 systemd[1]: Stopping Cardano Node…
Oct 24 08:09:42 staking01 systemd[1]: cnode.service: Control process exited, code=killed, status=2/INT
Oct 24 08:09:42 staking01 cnode[253535]: Shutting down…
Oct 24 08:09:42 staking01 systemd[1]: cnode.service: Failed with result ‘signal’.
Oct 24 08:09:42 staking01 systemd[1]: Stopped Cardano Node.
Oct 24 08:09:42 staking01 systemd[1]: Started Cardano Node.
Oct 24 08:09:43 staking01 cnode[266418]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Oct 24 08:09:45 staking01 cnode[266711]: Listening on http://127.0.0.1:12798
Oct 24 08:11:32 staking01 systemd[1]: Stopping Cardano Node…
Oct 24 08:11:32 staking01 systemd[1]: cnode.service: Control process exited, code=killed, status=2/INT
Oct 24 08:11:32 staking01 cnode[266711]: Shutting down…
Oct 24 08:11:32 staking01 systemd[1]: cnode.service: Failed with result ‘signal’.
Oct 24 08:11:32 staking01 systemd[1]: Stopped Cardano Node.
Oct 24 08:11:32 staking01 systemd[1]: Started Cardano Node.
Oct 24 08:11:33 staking01 cnode[272947]: ERROR: You specified 12788 as your EKG port, but it looks like the cardano-node (PID: 273060 ) is not listening on this port. Please update the config or kill the conflicting process first.
Oct 24 08:11:33 staking01 cnode[272947]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Oct 24 08:11:33 staking01 cnode[273082]: mkdir: cannot create directory ‘/archive’: Permission denied
Oct 24 08:11:34 staking01 cnode[273086]: Listening on http://127.0.0.1:12798
Oct 24 08:11:34 staking01 cnode[273086]: Shutting down…
Oct 24 08:11:34 staking01 cnode[273086]: cardano-node: FatalError {fatalErrorMessage = “Cardano.Node.Run.handleSimpleNode.readTopologyFile: Cardano.Node.Configuration.Topology.readTopologyFile: : openBinaryFile: does not exist (No such file or directory)”}
Oct 24 08:11:34 staking01 systemd[1]: cnode.service: Main process exited, code=exited, status=1/FAILURE
Oct 24 08:11:34 staking01 systemd[1]: cnode.service: Failed with result ‘exit-code’.
Oct 24 08:11:39 staking01 systemd[1]: cnode.service: Scheduled restart job, restart counter is at 1.
Oct 24 08:11:39 staking01 systemd[1]: Stopped Cardano Node.
Oct 24 08:11:39 staking01 systemd[1]: Started Cardano Node.
Oct 24 08:11:40 staking01 cnode[273285]: ERROR: You specified 12788 as your EKG port, but it looks like the cardano-node (PID: 273501 ) is not listening on this port. Please update the config or kill the conflicting process first.
Oct 24 08:11:40 staking01 cnode[273540]: mkdir: cannot create directory ‘/archive’: Permission denied
Oct 24 08:11:41 staking01 cnode[273552]: Listening on http://127.0.0.1:12798

started it after the build. 1st time was ok, second time had one error, third caused a shitshow. I waited until gLiveView said that the node was finished starting and then waited a minute or so before restarting the node.

It looks like more instances are running on the server

how should I be restarting the node?
I’ve been using

sudo systemctl restart cnode
is it possible doing it that way is the issue?

Nope, try to start it manually so

sudo systemctl stop cnode
sudo systemctl disable cnode

Now go to scripts folder and type ./cnode.sh

what is the output?

/opt/cardano/cnode/scripts$ ./cnode.sh
WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Listening on http://127.0.0.1:12798

killed that and ran it again and it even got rid of the unclean shutdown warning!

$ ./cnode.sh
Listening on http://127.0.0.1:12798

edit: did it again and the warning is back… but JUST that warning:

$ ./cnode.sh
WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Listening on http://127.0.0.1:12798

Ok, press Ctrl+C

now run again ./deploy-as-systemd.sh and press N for topology updater and also N for pooltool processes in case u are not sending informations to pooltool (sendmytip and slots)

then sudo systemctl enable cnode
sudo systemctl restart cnode

journalctl -e -f -u cnode

Oct 24 08:52:54 staking01 systemd[1]: Started Cardano Node.
Oct 24 08:52:55 staking01 cnode[26511]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Oct 24 08:52:56 staking01 cnode[26938]: Listening on http://127.0.0.1:12798

first startup seems ok

edit: except gLiveView says it’s a relay…

Ok, this is the only output since the restart?
Can u type now top? Do u see the CPU 100%? How about glive?

that’s all the output since the service restart
glive says it’s a relay. --do I need to edit my env to fix that?
htop shows less than 5% CPU usage for the entire system.

Perfect!
Now edit again the topology file (replace the IOHK nodes with your relay)
also edit the env file (pool name line and uncomment it)
Then restart the node

sudo systemctl restart cnode

Wait for the node to start and check again on glive
at the end you can delete all files from scripts/files folder which have _bkp_xxxx in the name

use rm filename_bkp_xxxx

Oct 24 08:52:56 staking01 cnode[26938]: Listening on http://127.0.0.1:12798
Oct 24 09:10:38 staking01 systemd[1]: Stopping Cardano Node…
Oct 24 09:10:38 staking01 systemd[1]: cnode.service: Control process exited, code=killed, status=2/INT
Oct 24 09:10:38 staking01 cnode[26938]: Shutting down…
Oct 24 09:10:38 staking01 systemd[1]: cnode.service: Failed with result ‘signal’.
Oct 24 09:10:38 staking01 systemd[1]: Stopped Cardano Node.
Oct 24 09:10:38 staking01 systemd[1]: Started Cardano Node.
Oct 24 09:10:38 staking01 cnode[86061]: ERROR: You specified 12788 as your EKG port, but it looks like the cardano-node (PID: 86284 ) is not listening on this port. Please update the config or kill the conflicting process first.
Oct 24 09:10:38 staking01 cnode[86061]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Oct 24 09:10:38 staking01 cnode[86351]: mkdir: cannot create directory ‘/archive’: Permission denied
Oct 24 09:10:39 staking01 cnode[86355]: Listening on http://127.0.0.1:12798
Oct 24 09:10:40 staking01 cnode[86355]: Shutting down…
Oct 24 09:10:40 staking01 cnode[86355]: cardano-node: FatalError {fatalErrorMessage = “Cardano.Node.Run.handleSimpleNode.readTopologyFile: Cardano.Node.Configuration.Topology.readTopologyFile: : openBinaryFile: does not exist (No such file or directory)”}
Oct 24 09:10:40 staking01 systemd[1]: cnode.service: Main process exited, code=exited, status=1/FAILURE
Oct 24 09:10:40 staking01 systemd[1]: cnode.service: Failed with result ‘exit-code’.
Oct 24 09:10:45 staking01 systemd[1]: cnode.service: Scheduled restart job, restart counter is at 1.
Oct 24 09:10:45 staking01 systemd[1]: Stopped Cardano Node.
Oct 24 09:10:45 staking01 systemd[1]: Started Cardano Node.
Oct 24 09:10:47 staking01 cnode[87126]: Listening on http://127.0.0.1:12798

I don’t know what I’m doing wrong!
I only updated my pool’s name and port in env,
ticker, PT api key in cncli.sh,
and the number of retries in gLiveView.sh

I haven’t used cntools so not sure what might be the problem but as I see from the logs it’s something like this:

  • User running the cardano node service does not have permission to create /archive folder at some location (I assume that is not the absolute path shown in the logs)
  • Process initiates a “shut down”
  • During the shut-down phase, it’s unable to read the topology file (maybe it keeps that “temporary” file in the /archive folder it is unable to create)

Maybe do some investigation around the /archive directory, what that might be and where it’s located (ideally on a working node). Then see if the process has permissions in the parent folder to create it.

Ok, so if u comment the pool name from env the node is running in glive (as a relay)?

Can u test again?

I think /archive is in the /opt/cardano/cnode/logs folder. in fact, it’s there now. so the user must have permissions, right?

well it’s been stuck at “starting” for a while now… and one of my cores is ~100%

Oct 24 09:44:37 staking01 systemd[1]: Started Cardano Node.
Oct 24 09:44:39 staking01 cnode[154652]: Listening on http://127.0.0.1:12798

Wait more time, it looks now is trying to start the node

it’s started.

but not showing as a relay?