Block producer crashes every ~24hours status=137

Staking & Delegation Operate a Stake Pool
1

My block producer crashes every 24 hours since I run a stake pool (since about 6 months). Both relay nodes don’t show this problem. Any idea if this is a known issue or how to fix it? Would be greatly appreciated.

Dec 30 15:16:02 server2 systemd[1]: cardano-node.service: Main process exited, code=exited, status=137/n/a
Dec 30 15:16:02 server2 systemd[1]: cardano-node.service: Failed with result ‘exit-code’.
Dec 30 15:16:07 server2 systemd[1]: cardano-node.service: Scheduled restart job, restart counter is at 1.
Dec 30 15:16:07 server2 systemd[1]: Stopped Cardano node service.
Dec 30 15:16:07 server2 systemd[1]: Started Cardano node service.

2

Hi,

journalctl -e -f -u cardano-node and check if u see the killing message or any other errors

Cheers,

3

This is the killing message:
Dec 30 15:16:02 server2 cardano-node[2750627]: /home/xxx/cardano-my-node/startBlockProducingNode.sh: line 11: 2750636 Killed

4

ok, now type to check how much RAM has the server

free -m

5 
          total        used        free      shared  buff/cache   available

Mem: 15998 8953 148 0 6896 6742
Swap: 4095 48 4047

6

You suggested to another user having status=137 a while ago to set TraceMempool=false. I just changed this and will upgrade to 32GB tomorrow to see if the problem persists. Or do you have any other ideas?

7

The RAM should be enough… can u check if u have something added in crontab which is restarting the node each 24 hours?

8

calculate slots assignment for the next epoch

15 21 * * * /home/xxx/cardano-my-node/scripts/cncli-fivedays.sh && /home/xxx/cardano-my-node/scripts/cncli-leaderlog.sh

send previous and current epochs slots to pooltool

15 22 * * * /home/xxx/cardano-my-node/scripts/cncli-fivedays.sh && /home/xxx/cardano-my-node/scripts/cncli-sendslots.sh

query ledger-state and dump to /home/xxx/cardano-my-node/scripts/ledger-state.json

15 15 * * * /home/xxx/cardano-my-node/scripts/ledger-dump.sh

9

I could remove those and see if it helps?

10

Yeah, those scripts which are running daily are crashing the node (due to insufficient RAM)… try to increase the SWAP FILE to 8G and keep it under monitoring

PS: the scripts should run only once with 1,5 days before the next epoch start

11

Ok thanks a lot!!

1 Like
12

To increase the SWAP file

sudo swapoff /swapfile
sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

free -m and now u should see 8G

13

You should stop doing a ledger dump and use cncli “active-stake” method. It uses a lot less resources and is faster.