I am experiencing sudden high memory usage on my core/block node running cardano-node 1.34:
This screenshot is for over the past 6 hours. The only things running on these boxes are cardano. 2vCPU 16GB of RAM. I’m not sure where to look for the errors because the OS kills the cnode.service (using CNTools).
Any advice on where I can begin to look to troubleshoot here? I’ve increases the swap file to 32GB from 6GB.
72 minutes is the interval between nodes taking ledger snapshots… but that normally causes the load & CPU usage to spike, not the memory usage.
I don’t know what “Memory Utilization” is in this graph, but it’s certainly not your percentage of 16GB RAM being used, since the Cardano node needs at least 6.5GB of memory space to perform normally at all. But it does sound like your system is running out of memory if the node process is getting killed. You could try:
1 - upgrading from cardano-node 1.34 to 1.34.1 since as I remember 1.34 was a tag used internally by IOG & not an official release… maybe there are some memory bugs that come out when doing the ledger snapshot?
2 - you can change the variables for the ledger snapshot interval if you need to do any more testing:
your node keeps restarting. view the journal to find out whats happening
journalctl -fu cnode
see whats going on. act accordingly.
