Background: I have both my cardano-relay and cardano-block nodes running in seperate docker container.
Before I touched anything, the disk usage looked like this:
Looks pretty normal.
Then I decided to add
flock to my cronjobs on both my relay node and block nodes:
Crontab for relay-node:
* * * * * flock -n /root/grafana.lock -c 'cd /usr/share/grafana && /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini' * * * * * flock -n /root/prometheus.lock -c 'prometheus' * * * * * flock -n /root/prometheus-node-exporter -c 'prometheus-node-exporter' * * * * * flock -n /root/start-relay-node.lock -c '/root/cardano-my-node/startRelayNode1.sh' * * * * * echo "Hello world" >> /var/log/cron.log 2>&1 0 * * * * /root/cardano-my-node/topologyUpdater.sh && echo "DONE" >> /var/log/cron.log 2>&1
Crontab for block-node:
* * * * * flock -n /root/pne.lock -c 'prometheus-node-exporter' * * * * * flock -n /root/start-bp-node.lock -c '/root/cardano-my-node/startBlockProducingNode.sh'
Now all of a sudden, disk usage became 100 MB/s!!! Why? Is this flock related? Why is the nodes constantly writing so much data?
(The gaps in between are me stopping and starting the nodes)
Edit 1: using
iotop I see that it’s the
prometheus that keeps writing the ~90-100 MB/s to disk. Not sure why though…
Edit 2: running `watch -n 2 “df -h”’ to constantly watch my overall storage, I see that the storage decreases by 2 GB then increases by 2 GB constantly, so I’m guessing that data isn’t actually just being written but also being deleted which is a big relief.