Running out of disk space on relay

Hello guys, can some one share best practice regarding disk space maintenance.
I have two relays and both are slowly running out of space please see attached. Each drive 30 GB in size.
Relay 1 - just a relay nothing else installed
Relay 2 - has Prometheus and Grafana installed on top
My question: is there is anything that could be deleted to free up some space or I just add more and keep going.
2021-03-23 01_39_54-Kairos1 Pool - Grafana - Brave

what you can remove is the logs of the node - can you check what is the size of the log folder ( du -h)

Hi @laplasz it is 8.4 GB
Please let me know what is safe to to delete.

yes - list the folder content of the logs folder (ls -l logs)


wow - you have an archive folder - it seems every logs except the current one stored under the archive and that size is 8.4 G - so it is save to delete all the files under archive folder… what is the content of that folder (ls logs/archive)

Here you go

burn it… can you please share what is the log settings in your config.json, like this:

"rotation": {
    "rpKeepFilesNum": 10,
    "rpLogLimitBytes": 100000000,
    "rpMaxAgeHours": 24
  "setupScribes": [
      "scFormat": "ScText",
      "scKind": "FileSK",
      "scName": "/logs/node.log",
      "scRotation": null

rotation actually removes old log files automatically

“rotation”: {
“rpKeepFilesNum”: 10,
“rpLogLimitBytes”: 50000000,
“rpMaxAgeHours”: 24
“setupBackends”: [
“setupScribes”: [
“scKind”: “FileSK”,
“scName”: “/opt/cardano/cnode/logs/node0.json”,
“scFormat”: “ScJson”,
“scRotation”: null

interesting - then might be some external scripts archives your logs… do you have a cron job, or something which moving logs to archive?

I have not setup anything deliberately. I used prerequisites from cntools.
Where else can I check?

Ok will go through what cntools actually do with logs… probably there is something what I am not aware right now

Thanks man

there is an archiveLog function in
archiveLog # archive current log and cleanup log archive folder

I think it is called everytime the script is called. so how often you call the

I do not call cntools automaticaly, being called every 3 hours

yes - actually archiveLog would delete the content of the archive folder… something else is controlling the content of the logs/

just found it… the making this kind of behavior - which I think it is not so good - will contact them to ask about this behavior.
so since you start the relay every 3 hours probably with systemctl - it calls which archive the logs.

Yes, you are right Looking at it right now, could you please ask them how to clean or rotate logs. Thanks in advance

as a workaround just remove the related lines from

[[ ! -d "${LOG_DIR}/archive" ]] && mkdir -p "${LOG_DIR}/archive"

[[ $(find "${LOG_DIR}"/*.json 2>/dev/null | wc -l) -gt 0 ]] && mv "${LOG_DIR}"/*.json "${LOG_DIR}"/archive/

then the rotation will start to happen because cardano-node now can remove older files

Do I really need to keep those files? can just delete older files from Archive (node0-XXXXXXXXXXX.json) ?