Relay using a lot of CPU, a lot

[CheffWallets]

I have two relay, both are same configuration, 6 core, 16gb ram, so far so good. one of them, which that runs grafana by the way, started to look very bad today.
It was doing OK but all of sudden, it eats more than 100% cpu now. I don’t have any idea whats going on, I already filtered all ICMP packages, no avail and I wanted to know what can I do about it…
this just very bad because BP will lose the connection and will just lose its peer…

I do feel myself very bad, and I do apologize to any delegator looking at this post and asking “hell this guy doesnt know what he’s doing” I’m really sorry about that.

[jeremyisme]

Sorry for a dumb question, but have you restarted the node lately? It can be a magic solution sometimes…

[CheffWallets]

hi I just ran again the prereq.sh script, at least glview go back to work it was dead since a while.
let’s try to see if this miserable issue was fixed!
thank you

[CheffWallets]

hi I have no idea whats going on with this particular relay. it just cannot start!

[Alexd1985]

journalctl -e -f -u cardano-node for coincashew guide

journalctl -e -f -u cnode for cntools guide

[CheffWallets]

I found some strange logs today

“thread”:“167”,“loc”:null,“data”:{“address”:“18.188.75.201:3001”,“kind”:“ErrorPolicyTrace”,“event”:“ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (HeaderError (At (Block {blockPointSlot = SlotNo 43372972, blockPointHash = 0b5b67d45414810c66248c9992748fccdc5388866c3946570f7aafd6ff093902})) (HeaderEnvelopeError (OtherHeaderEnvelopeError (HardForkEnvelopeErrFromEra S (S (S (S (Z (WrapEnvelopeErr {unwrapEnvelopeErr = ObsoleteNodeCHAIN 6 5})))))))) (Tip (SlotNo 43372792) 97a5392f156a97b2a66ec3568b1ae49433c52dde3aaabddb9ffee4db2a174814 (BlockNo 6405205)) (Tip (SlotNo 43516901) c7c02c22a38326762a2eb75d9a392bfaab84ef8e3fa9f915e47808cbaee190f5 (BlockNo 6412017))))) 200s 200s”},“sev”:“Warning”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“hostname”,“pid”:“1022”,“ns”:[“cardano.node.ErrorPolicy”],“at”:“2021-10-24T13:46:46.91Z”}
{“thread”:“2149”,“loc”:null,“data”:{“kind”:“SubscriptionTrace”,“domain”:"“relays-new.cardano-mainnet.iohk.io”",“event”:“Connection Attempt Start, destination 18.132.127.241:3001”},“sev”:“Notice”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“hostname”,“pid”:“1022”,“ns”:[“cardano.node.DnsSubscription”],“at”:“2021-10-24T13:46:47.71Z”}
{“thread”:“2149”,“loc”:null,“data”:{“kind”:“SubscriptionTrace”,“domain”:"“relays-new.cardano-mainnet.iohk.io”",“event”:“Connection Attempt End, destination 18.132.127.241:3001 outcome: ConnectSuccess”},“sev”:“Notice”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“hostname”,“pid”:“1022”,“ns”:[“cardano.node.DnsSubscription”],“at”:“2021-10-24T13:46:47.73Z”}
{“thread”:“2150”,“loc”:null,“data”:{“kind”:“SubscriptionTrace”,“domain”:"“relays-new.cardano-mainnet.iohk.io”",“event”:“Connection Attempt Start, destination 18.188.75.201:3001”},“sev”:“Notice”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“hostname”,“pid”:“1022”,“ns”:[“cardano.node.DnsSubscription”],“at”:“2021-10-24T13:46:47.74Z”}
{“thread”:“2164”,“loc”:null,“data”:{“kind”:“SubscriptionTrace”,“domain”:"“relays-new.cardano-mainnet.iohk.io”",“event”:“Connection Attempt Start, destination 3.128.217.217:3001”},“sev”:“Notice”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“hostname”,“pid”:“1022”,“ns”:[“cardano.node.DnsSubscription”],“at”:“2021-10-24T13:46:47.76Z”}
{“thread”:“2165”,“loc”:null,“data”:{“kind”:“SubscriptionTrace”,“domain”:"“relays-new.cardano-mainnet.iohk.io”",“event”:“Connection Attempt Start, destination 18.133.133.57:3001”},“sev”:“Notice”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“hostname”,“pid”:“1022”,“ns”:[“cardano.node.DnsSubscription”],“at”:“2021-10-24T13:46:47.79Z”}
{“thread”:“2165”,“loc”:null,“data”:{“kind”:“SubscriptionTrace”,“domain”:"“relays-new.cardano-mainnet.iohk.io”",“event”:“Connection Attempt End, destination 18.133.133.57:3001 outcome: ConnectSuccessLast”},“sev”:“Notice”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“hostname”,“pid”:“1022”,“ns”:[“cardano.node.DnsSubscription”],“at”:“2021-10-24T13:46:47.81Z”}
{“thread”:“167”,“loc”:null,“data”:{“address”:“18.188.75.201:3001”,“kind”:“ErrorPolicyTrace”,“event”:“ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = “Parallel connection cancelled”, seStack = }))) 1s”},“sev”:“Notice”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“hostname”,“pid”:“1022”,“ns”:[“cardano.node.ErrorPolicy”],“at”:“2021-10-24T13:46:47.81Z”}
{“thread”:“167”,“loc”:null,“data”:{“address”:“3.128.217.217:3001”,“kind”:“ErrorPolicyTrace”,“event”:“ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = “Parallel connection cancelled”, seStack = }))) 1s”},“sev”:“Notice”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“hostname”,“pid”:“1022”,“ns”:[“cardano.node.ErrorPolicy”],“at”:“2021-10-24T13:46:47.81Z”}

[CheffWallets]

Oct 24 15:41:36 myhost cnode[952]: Guild scripts has now been upgraded to support cardano-node 1.30.1 or higher (1.29.0 found).
Oct 24 15:41:36 myhost cnode[952]: Please update cardano-node (note that you should ideally update your config too) or use tagged branches for older node version.
Oct 24 15:41:36 myhost cnode[952]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Oct 24 15:41:39 myhost cnode[1022]: Listening on http://127.0.0.1:12798

I mean, it always says I didnt updated but thats a lie, I’m running 1.30.1 100% sure about that!

[Alexd1985]

cardano-node - -version
which cardano-node

It looks like u still have the old binari files (1.29.0); this is happening because u used another guide before cntools guide
cd /usr/local/bin
ls -l

[CheffWallets]

@CheffWallets

Guild scripts has now been upgraded to support cardano-node

cardano-node 1.30.1 - linux-x86_64 - ghc-8.10
git rev 0fb43f4e3da8b225f4f86557aed90a183981a64f

/home/cardano/.cabal/bin/cardano-node

drwxr-xr-x 5 cardano cardano 4096 Jun 11 15:27 auto-completion

-rwxr-xr-x 1 cardano cardano 5121272 Jun 11 15:27 bech32

-rwxr-xr-x 1 cardano cardano 8973768 Jun 11 15:27 cardano-address

-rwxrwxr-x 1 cardano cardano 116460856 Sep 21 14:21 cardano-cli

-rwxrwxr-x 1 cardano cardano 127623584 Sep 21 14:21 cardano-node

-rwxrwxr-x 1 cardano cardano 105378504 Sep 21 14:21 cardano-node-chairman

-rwxrwxr-x 1 cardano cardano 125601040 Sep 21 14:21 cardano-submit-api

-rwxrwxr-x 1 cardano cardano 28314968 Sep 21 14:21 cardano-topology

-rwxr-xr-x 1 cardano cardano 33086248 Jun 11 15:27 cardano-wallet

-rwxrwxr-x 1 cardano cardano 29680832 Sep 21 14:21 locli

-rwxrwxr-x 1 cardano cardano 79189032 Sep 21 14:21 plutus-example

-rwxrwxr-x 1 cardano cardano 129827808 Sep 21 14:21 tx-generator

-rwxrwxr-x 1 cardano cardano 3188488 Sep 21 14:21 unit

[Alexd1985]

@Alexd1985

cd /usr/local/bin
ls -l

show me this output

[CheffWallets]

-rwxr-xr-x 1 root root 7540728 Sep 21 16:08 bech32

-rwxr-xr-x 1 root root 30816992 Sep 21 16:08 cardano-address

-rwxr-xr-x 1 root root 117337808 Sep 21 16:08 cardano-cli

-rwxr-xr-x 1 root root 128344016 Sep 21 16:08 cardano-node

-rwxr-xr-x 1 root root 105841152 Sep 21 16:08 cardano-node-chairman

-rwxr-xr-x 1 root root 126413008 Sep 21 16:08 cardano-submit-api

-rwxr-xr-x 1 root root 28316968 Sep 21 16:08 cardano-topology

-rwxr-xr-x 1 root root 113670616 Sep 21 16:08 create-script-context

-rwxr-xr-x 1 root root 29718944 Sep 21 16:08 locli

-rwxr-xr-x 1 root root 112942328 Sep 21 16:08 plutus-example

-rwxr-xr-x 1 root root 130795936 Sep 21 16:08 tx-generator

-rwxr-xr-x 1 root root 3195096 Sep 21 16:08 unit

[Alexd1985]

@CheffWallets

/home/cardano/.cabal/bin/

and for the cd /home/cardano/.cabal/bin/
ls -l

[CheffWallets]

@Alexd1985

@CheffWallets

/home/cardano/.cabal/bin/

lrwxrwxrwx 1 cardano cardano 117 Sep 29 16:51 bech32 → …/store/ghc-8.10.4/bech32-1.1.0-e-bech32-e124a333ef663f7a6cfdad83014e06b58d0ac64d7eb42ddeab35b09d20ac39a3/bin/bech32

lrwxrwxrwx 1 cardano cardano 150 Sep 29 16:51 cardano-address → …/store/ghc-8.10.4/cardano-addresses-cli-3.5.0-e-cardano-address-bf39b20d433ebca4b3ecac23b63305b93f254ad40b598fe4fa818880959675f1/bin/cardano-address

-rwxrwxr-x 1 cardano cardano 116365544 Sep 29 17:28 cardano-cli

-rwxrwxr-x 1 cardano cardano 127526736 Sep 29 17:28 cardano-node

-rwxrwxr-x 1 cardano cardano 105283056 Sep 29 17:28 cardano-node-chairman

-rwxrwxr-x 1 cardano cardano 125504736 Sep 29 17:28 cardano-submit-api

-rwxrwxr-x 1 cardano cardano 28314968 Sep 21 00:11 cardano-topology

-rwxrwxr-x 1 cardano cardano 29680832 Sep 21 00:11 locli

-rwxrwxr-x 1 cardano cardano 79086584 Sep 29 17:28 plutus-example

-rwxrwxr-x 1 cardano cardano 129730648 Sep 29 17:28 tx-generator

-rwxrwxr-x 1 cardano cardano 3188488 Sep 21 00:11 unit

[Alexd1985]

cd /usr/local
rm -R bin

restart the node and check again with jorunalctl…

[CheffWallets]

ok. I’ll try that.

check this CPU usage, it can’t be normal.

[Alexd1985]

It is while the node will start, after that you should see ~3-4%

[CheffWallets]

at least now glview shows syncing message i+ll let you know

thank you

[Alexd1985]

It will start

[CheffWallets]

hi ALex, looks its back in the business now! thank you very much for that!

[map84]

Hi there! I have a similar problem but the issue seems to have a different cause (the relay and block producer have been syncing for around 24hours) but the CPU consumption may be why i struggle to get a SSH connection every now and then.

:~/cardano-my-node$ which cardano-node
/usr/local/bin/cardano-node

~/cardano-my-node$ cd /home/cardano/.cabal/bin/
-bash: cd: /home/cardano/.cabal/bin/: No such file or directory

/usr/local/bin$ ls
cardano-cli cardano-node

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
715203 ubuntu 20 0 257.1g 7.5g 57052 S 172.1 48.3 44:17.99 cardano-node
197 root 19 -1 361868 219084 217780 S 0.3 1.4 17:31.39 systemd-journal
1 root 20 0 104764 10528 7000 S 0.0 0.1 0:06.61 systemd

Any help would be greatly appreciated:) Im just starting out in the world of stake pool operation and am impressed and slightly overwhelmed by the amount of resources