Excessive incoming traffic on node causes high CPU usage

Hi All,
This is my first message in this forum. Thank you a lot to everyone for your good advice on forum with guides etc in setting up pool relays and block producers, and all the trouble shooting.

I’ve noticed one of my Relays is experiencing excessive incoming traffic (on the same port as the Cardano node), and this is visible from the Cardano node text graph interface as well. It’s about 30-120 mbytes/sec, but it results in Cardano node requiring periodic excessive cpu usage, despite me adding more resources to the machine.

I am concerned some party is trying to break their way into the relay…for whatever reason, maybe to find out where the block producer is, etc…

At the moment this is looking more like a denial of service/resources on one of my relays. All other relays are behaving normally, without any significant incoming traffic (after having all synchronised etc). Transactions are showing up all well on the relays.

Question: has anyone noticed any similar patterns of strange Incoming traffic towards your relays?

Of course as this traffic is on the same port as the one Cardano node is listening on, I can’t do much about it with the firewall…yet.

Regards,
lenesul

1 Like

Sorry, meant this strange traffic is 30-100 Kbytes/sec

Hello lenesul,

I would recommend installing fail2ban, but I don’t think that would address this specific issue.

Make sure your relay topology peer count has less than 20 total sum of valencies. I recommend a total valency count between 15-18. Anything higher than 20 and you will have issues as outbound connections require far more resources than inbound.

And just to make sure, Is your relay syncing, or already in sync?

I would recommend checking topology and restarting to see if the issue perists.

Your friend, FROG

1 Like

Thank you for your reply. The relay is fully synced, and has 18 peers. Have restarted it a few times, but the issue didn’t go away. Have even restarted the Linux machine. The other identical relays (Apart from topology) on other machines don’t have the issue. In fact this machine has double the resources of the other relays I run.

Most interesting is that the incoming Strange high traffic (For the last 9 hours) has just stopped a short time after my post in the forum, and the cpu usage is back to normal. :slight_smile:

Thank you for your help!

Will keep monitoring it.

Awesome, Lenesul. Yes that does sound suspicious.

Definitely let us know if the problem persists and take care.

Hi All,

I am also experiencing a similar issue. For the past 24 hours, one of my relays is bombarded with an unusual amount of traffic up to 500kb/s, which causes the CPU to work excessively.

Can anybody help me shed some light on this, please?

The relay has been working fine for 2 weeks in a row and there is nothing in the logs that shows a possible malfunction.

Many thanks in advance,
Gilberto

Tagging @ADAfrog since he had commented last August and asked about whether the node is syncing.

Is it typical for a node to chew up a lot of cpu and memory when it has just started syncing? cardano-node has been running at about 31% memory (out of 4GB) and around 190% cpu (two dedicated cores). This is against mainnet It’s been running for the last few hours and initially the tip query showed it in the Byron era. It’s now in Shelley and I’m assuming still syncing. Will this usage decrease once it’s synced?

I

{
“epoch”: 213,
“hash”: “5cd24cb34b1ae3fd4db0aff613ec3a0ec7a4029dc52accfd6fb91f48a6e84350”,
“slot”: 6806466,
“block”: 4605403,
“era”: “Shelley”
}

How about changing the port. I know it requires a new certificate, but it might help. Especially, because your other relay is not affected.

You know - hackers are everywhere!