This observed cycle is from the ledger snapshot alone, currently taken by default at a 72 minute interval, set in the code by DefaultSnapshotInterval
. The interval is adjustable in the config.json
file with SnapshotInterval
, measured in seconds. (One of the more cooperative IOG devs stepped in just now to the thread above to tell us this.)
Under some conditions even copying files, let alone writing the ledger snapshot, will use a lot of swap… I believe this is because file pages are first put in the buffer cache before writing them out to disk. On thinly provisioned servers, this pushes colder cardano-node
memory pages out to swap even if you have vm.swappiness
set to 0
like we have on our servers.
Then as @Vadimus said the kernel needs to come back and herd the swapped out pages back into memory. This is how the ledger snapshot can cause a cascade of CPU use. It’s not hard to break it down once you know that the cycle of activity begins with the CPU+disk activity of generating the ledger snapshot.