Cardano-node slow to start

Staking & Delegation Operate a Stake Pool
1

I am doing some maintenance on my relay node that required me to stop/restart cnode (using cntools). I’m seeing the service restart times take longer and longer… I am approaching 1 hour for the node to be restarted. I am seeing these types of logs messages in $CNODE_HOME/logs/node0.json:

{"host":"cardano-","pid":"43058","loc":null,"at":"2022-08-12T18:20:24.49Z","ns":["cardano.node.ChainDB"],"sev":"Info","env":"1.34.1:73f9a","data":{"finalChunk":"3181","initialChunk":"2840","kind":"TraceImmutableDBEvent.StartedValidatingChunk"},"msg":"","thread":"7","app":[]}
{"host":"cardano-","pid":"43058","loc":null,"at":"2022-08-12T18:20:27.13Z","ns":["cardano.node.ChainDB"],"sev":"Info","env":"1.34.1:73f9a","data":{"kind":"TraceImmutableDBEvent.ValidatedChunk","chunkNo":"2840"},"msg":"","thread":"7","app":[]}
{"host":"cardano-","pid":"43058","loc":null,"at":"2022-08-12T18:20:27.13Z","ns":["cardano.node.ChainDB"],"sev":"Info","env":"1.34.1:73f9a","data":{"finalChunk":"3181","initialChunk":"2841","kind":"TraceImmutableDBEvent.StartedValidatingChunk"},"msg":"","thread":"7","app":[]}
{"host":"cardano-","pid":"43058","loc":null,"at":"2022-08-12T18:20:28.78Z","ns":["cardano.node.ChainDB"],"sev":"Info","env":"1.34.1:73f9a","data":{"kind":"TraceImmutableDBEvent.ValidatedChunk","chunkNo":"2841"},"msg":"","thread":"7","app":[]}
{"host":"cardano-","pid":"43058","loc":null,"at":"2022-08-12T18:20:28.78Z","ns":["cardano.node.ChainDB"],"sev":"Info","env":"1.34.1:73f9a","data":{"finalChunk":"3181","initialChunk":"2842","kind":"TraceImmutableDBEvent.StartedValidatingChunk"},"msg":"","thread":"7","app":[]}

How can I decrease service restart times? My node specs:
16GB RAM
8 core CPU
SSD storage

2

How much storage?

3

SSD storage is 160GB

4

It might be a side issue, but 160gb might be getting a bit tight now. Just check you have enough spare storage.

5

I have 51GB free. Is that not enough free space?

6

can u check glive again? If Mem RSS not increasing then apply the steps:

  1. cd db folder
  2. ls -l (here u should see more files and 3 folders)
  3. sudo systemctl stop cardano-node
  4. mv ledger ledger_bkp
    mv immutable immutable_bkp
    mv volatile volatile_bkp
  5. ls - l
  6. sudo systemctl start cardano-node
  7. after 10 -15 seconds type sudo systemctl stop cardano-node
  8. ls -l (now u should have the bkp files and also the new files)
  9. delete the new files (not the bkp-ed ones)
    rm -R ledger
    rm -R immutable
    rm -R volatile
  10. ls -l (now u should have the old files only)
  11. rename back the old files
    mv ledger_bkp ledger
    mv immutable_bkp immutable
    mv volatile_bkp volatile
  12. ls -l
  13. start the node and check glive
    sudo systemctl start cardano-node
7

image
I’ve followed your steps and it looks like it’s syncing the whole chain again…? Does this mean that my relay is now down for a whole day? I’m looking at the estimated time remaining and it says 1d 00:18:39…

8

are u sure u followed the right steps?
go to db folder type ls -l and share the output

9 
drwxr-xr-x 3 cardano users 4096 Aug 12 21:34 immutable
drwxr-xr-x 3 cardano users 4096 Aug 12 21:28 ledger
-rw-r--r-- 1 cardano users    0 Apr 26  2021 lock
-rw-r--r-- 1 cardano users    9 Apr 26  2021 protocolMagicId
drwxr-xr-x 3 cardano users 4096 Aug 12 21:36 volatile
10

Let me try going through the steps again

11

It’s still showing 1 day for time remaining.

13

try cd ledger and ls -l
I want to check if u moved the folders inside folderS

14

image

15

exactly… u copied the folder inside folder

now type

cd ledger 
mv ledger_bkp /opt/cardano/cnode/db
cd immutable 
mv immutable_bkp /opt/cardano/cnode/db
cd volatile
mv volatile_bkp /opt/cardano/cnode/db

stop the node
now inside db folder u should have the new files and the bkp ones… delete the new ones

sudo rm -R ledger
sudo rm -R immutable
sudo rm -R volatile
ls -l

now u should see the bkped files only … if not try the above commands without sudo

then rename the files

1 Like
16

Oh jeez… I goofed that up! Thanks for catching that! So, I’ve fixed that and restarted the node. I still see that “initialChunk” output that I referenced in the beginning

{"host":"cardano-","pid":"693738","loc":null,"at":"2022-08-13T01:56:44.20Z","ns":["cardano.node.ChainDB"],"sev":"Info","env":"1.34.1:73f9a","data":{"finalChunk":"3181","initialChunk":"104","kind":"TraceImmutableDBEvent.StartedValidatingChunk"},"msg":"","thread":"5","app":[]}
{"host":"cardano-","pid":"693738","loc":null,"at":"2022-08-13T01:56:45.25Z","ns":["cardano.node.ChainDB"],"sev":"Info","env":"1.34.1:73f9a","data":{"kind":"TraceImmutableDBEvent.ValidatedChunk","chunkNo":"104"},"msg":"","thread":"5","app":[]}
{"host":"cardano-","pid":"693738","loc":null,"at":"2022-08-13T01:56:45.25Z","ns":["cardano.node.ChainDB"],"sev":"Info","env":"1.34.1:73f9a","data":{"finalChunk":"3181","initialChunk":"105","kind":"TraceImmutableDBEvent.StartedValidatingChunk"},"msg":"","thread":"5","app":[]}
{"host":"cardano-","pid":"693738","loc":null,"at":"2022-08-13T01:56:46.43Z","ns":["cardano.node.ChainDB"],"sev":"Info","env":"1.34.1:73f9a","data":{"kind":"TraceImmutableDBEvent.ValidatedChunk","chunkNo":"105"},"msg":"","thread":"5","app":[]}
{"host":"cardano-","pid":"693738","loc":null,"at":"2022-08-13T01:56:46.43Z","ns":["cardano.node.ChainDB"],"sev":"Info","env":"1.34.1:73f9a","data":{"finalChunk":"3181","initialChunk":"106","kind":"TraceImmutableDBEvent.StartedValidatingChunk"},"msg":"","thread":"5","app":[]}

I am back to where I started it seems.