Running out of disk space on relay

Staking & Delegation Operate a Stake Pool
1

Hello guys, can some one share best practice regarding disk space maintenance.
I have two relays and both are slowly running out of space please see attached. Each drive 30 GB in size.
Relay 1 - just a relay nothing else installed
Relay 2 - has Prometheus and Grafana installed on top
My question: is there is anything that could be deleted to free up some space or I just add more and keep going.
2021-03-23 01_39_54-Kairos1 Pool - Grafana - Brave

2

what you can remove is the logs of the node - can you check what is the size of the log folder ( du -h)

3

Hi @laplasz it is 8.4 GB
cardano_cnode
Please let me know what is safe to to delete.

4

yes - list the folder content of the logs folder (ls -l logs)

5

cardano_cnode1

6

wow - you have an archive folder - it seems every logs except the current one stored under the archive and that size is 8.4 G - so it is save to delete all the files under archive folder… what is the content of that folder (ls logs/archive)

7

Here you go

cardano_cnode2
cardano_cnode21131Ă—754 28.7 KB

8

burn it… can you please share what is the log settings in your config.json, like this:

"rotation": {
    "rpKeepFilesNum": 10,
    "rpLogLimitBytes": 100000000,
    "rpMaxAgeHours": 24
  },
  "setupScribes": [
    {
      "scFormat": "ScText",
      "scKind": "FileSK",
      "scName": "/logs/node.log",
      "scRotation": null
    }
  ],

rotation actually removes old log files automatically

9

“rotation”: {
“rpKeepFilesNum”: 10,
“rpLogLimitBytes”: 50000000,
“rpMaxAgeHours”: 24
},
“setupBackends”: [
“KatipBK”,
“EKGViewBK”
],
“setupScribes”: [
{
“scKind”: “FileSK”,
“scName”: “/opt/cardano/cnode/logs/node0.json”,
“scFormat”: “ScJson”,
“scRotation”: null
}
]
}

10

interesting - then might be some external scripts archives your logs… do you have a cron job, or something which moving logs to archive?

11

I have not setup anything deliberately. I used prerequisites from cntools.
Where else can I check?

12

Ok will go through what cntools actually do with logs… probably there is something what I am not aware right now

13

Thanks man

14

there is an archiveLog function in cnTools.sh
archiveLog # archive current log and cleanup log archive folder

I think it is called everytime the cntools.sh script is called. so how often you call the cnTools.sh?

15

I do not call cntools automaticaly, topologyUpdater.sh being called every 3 hours

16

yes - actually archiveLog would delete the content of the archive folder… something else is controlling the content of the logs/

17

just found it… the cnode.sh making this kind of behavior - which I think it is not so good - will contact them to ask about this behavior.
so since you start the relay every 3 hours probably with systemctl - it calls cnode.sh which archive the logs.

18

Yes, you are right Looking at it right now, could you please ask them how to clean or rotate logs. Thanks in advance

19

as a workaround just remove the related lines from cnode.sh

[[ ! -d "${LOG_DIR}/archive" ]] && mkdir -p "${LOG_DIR}/archive"

[[ $(find "${LOG_DIR}"/*.json 2>/dev/null | wc -l) -gt 0 ]] && mv "${LOG_DIR}"/*.json "${LOG_DIR}"/archive/

then the rotation will start to happen because cardano-node now can remove older files

21

Do I really need to keep those files? can just delete older files from Archive (node0-XXXXXXXXXXX.json) ?