High Transaction Event on 2021-04-04 2200 UTC. SPO Thoughts?

Hi Everyone,
I received a bunch of Grafana alerts on my phone due to a surge in transactions where my pool performance degraded noticeably. I wanted to query the community to see what other SPOs experienced during this event.

A CardanoBits auction event started around 22:00 UTC. I saw a noticeable degradation in my pool performance during this time and wanted to make some changes to prevent future issues. Here is what I saw happen to [NEON] during this time:

CPU was pegged at 95% for minutes at a time
image

The block producer missed 132 Slots in a 1 hour time period
image

Transactions Per Second briefly reached 80 Tx/Second!
image

Transactions in Mempool remained above 350 for a period of time
image

First off, it’s awesome to see this type of usage on the blockchain but an aws T3.Large instance did not have the ability to handle it. I moved the my block producer from T3.large to C5.xlarge to try and mitigate this as a future risk. For those of you unfamiliar with AWS, T3 is general purpose and C5 is compute optimized but noticeably more expensive.

So… What did everyone else see during this event? I don’t think I’ve seen a transaction surge like this before. I feel this is a good growing pain but something that must be discussed.

2 Likes

I had a high transaction peak at the same time (my dashboard shows time in cet so its 2 hours later)

However my system load did not have an issue at all (as you can see)

I use 4 core 8gb server

Can you share the metrics you use to show missed nodes please. I don’t have that in my dashboard yet. Thanks.

1 Like

cardano_node_metrics_slotsMissedNum_int

2 Likes

Thanks alex :+1:

2 Likes

@Alexd1985 I don’t “no data” when i add this metric. Is the info not being exported or maybe i never missed a slot? If the first is the case where can i find instructions to export this metric? Thanks.

1 Like

I tested and it will appear data when you will miss slots. till then it will show you no data:)

image

you can test it too if u want :slight_smile:

also u can interogate the metric:
curl localhost:12798/metrics | grep slotsMissed

Cheers,

2 Likes


all is about cardanobits sells.

2 Likes

BP - 6X2.8GHz + 4GB ram + 4GB swap.
i lost 3 slot it’s because i did turn off TraceMempool. also other 3 relay run in 1.26.1 -

1 Like

1.26.1 not officially released… however, can u see a performance increased for 1.26.1?

Cheers,

yes. i did test them . it’s work much better . less CPU/RAM uses, less slot lost but get crash.

1 Like