Mempool issue on 2020-01-17 21:45-21:50UTC

Staking & Delegation Stake Pool Security
1

Nodes are monitored using Grafana. Between 2020-01-17 21:45-21:50UTC there was a significant increase in CPU utilisation across all relays and BPNs. There was a drop in chain synchronisation speed and a number of peers dropped. The nodes then recovered. Mempool spiked to about 60 txns suggested that the nodes could not process incoming txns fast enough.

Checking logs shows events like:

{“at”:“2021-01-17T21:46:28.20Z”,“env”:“1.24.2:400d1”,“ns”:[“cardano.node.Mempool”],“data”:{“tx”:{“txid”:“txid: TxId {_unTxId = “96bc76c023114b06cd1520d07ad219ea786743dd538b4245c797ef233349e838”}”},“kind”:“TraceMempoolRejectedTx”,“mempoolSize”:{“numTxs”:28,“bytes”:15565},“err”:{“kind”:“BadInputsUTxO”,“error”:“The transaction contains inputs that do not exist in the UTxO set.”,“consumed”:0,“badInputs”:[[“064add209fa393f8ab32d38a95226a397144ec759882d87af4dd7c0e549626d9”,25],[“4cf4c49f96cbbd1a0f846740814096faa6a3a679393a7c4b60c3ce42e0d6648d”,14],[“4cf4c49f96cbbd1a0f846740814096faa6a3a679393a7c4b60c3ce42e0d6648d”,22],[“60659adeb66e7e9f38918a41dd4a44b3a084f6a0c5ca935dca500443c11d95fa”,21],[“6a129268c2429ca32998bc639402a8f3da69791383c3fdabfda14d66dc83719c”,23],[“74b631133463c8f1fe2afaf1e908287e47a2a82ff31f4d538913db85844c4b99”,26],[“a7ca89d2f026e10065097c09a4d18d7bb7bc599fe6833b49923d295c18827453”,4],[“aebb3725e1c7e5a6c88584f8282b7e6f44325372fd66dca53a6ad91a8e469797”,0],[“c15161c55bf77d705e5f223526f522c4913b5de64b871396223288ac8435058b”,8],[“c15161c55bf77d705e5f223526f522c4913b5de64b871396223288ac8435058b”,14],[“c15161c55bf77d705e5f223526f522c4913b5de64b871396223288ac8435058b”,16],[“e3a47af3d7e6666455616ad2f7044165622b3f86a79e3b44d0bd34b8941a7cdc”,21],[“f029b6feef9098ec35241d996f0d0c476412d2cd1ebea8d0a523d2664e608272”,13],[“f4d2dfeb353172406860a32eaaa32a8be7f07ab31faf37e9d38353d9c37a8c07”,5]],“produced”:17427345978019}},“app”:,“msg”:"",“pid”:“18414”,“loc”:null,“host”:“vducdn46”,“sev”:“Info”,“thread”:“616”}

and:

{“at”:“2021-01-17T21:46:36.32Z”,“env”:“1.24.2:400d1”,“ns”:[“cardano.node.Mempool”],“data”:{“tx”:{“txid”:“txid: TxId {_unTxId = “94b9131bc937ffcd3b90f6a4c0715b47f370cfdd0d3801eb7bc0f4d42427046d”}”},“kind”:“TraceMempoolRejectedTx”,“mempoolSize”:{“numTxs”:31,“bytes”:17946},“err”:{“kind”:“StakeKeyAlreadyRegisteredDELEG”,“error”:“Staking credential already registered”,“consumed”:0,“credential”:“KeyHashObj (KeyHash “6b2b992a14a836b0d31fd4d072172bf389fd5500e8a4e18cdffe458f”)”,“badInputs”:[[“e3a47af3d7e6666455616ad2f7044165622b3f86a79e3b44d0bd34b8941a7cdc”,19]],“produced”:117980562}},“app”:,“msg”:"",“pid”:“18414”,“loc”:null,“host”:“vducdn46”,“sev”:“Info”,“thread”:“577”}

Anyone seen similar? Was this an attack? It essentially DoS’d our pool for the duration.

Let me know if you want logs from the time period concerned.

2

I hope this helps answer your concern: I’ve checked & we saw no such anomalies during that period. We have two relays in NA & Asia and they each have a very different character of incoming peers. Neither of them had anything like this in their log files & had no such interruptions in service. Thanks for the report in any case :sunglasses:

3

Thanks for the feedback.

Just checked the transactions in the explorer and they are good (even though the node thinks they are bad):
https://explorer.cardano.org/en/transaction?id=96bc76c023114b06cd1520d07ad219ea786743dd538b4245c797ef233349e838

So I guess that means the nodes locally got out of sync somehow? Strange, second time we have seen this since start of shelley so not a common occurrence by any means…

4

Have you checked to see if your symptoms resemble those mentioned in this issue? (Note there can be a cascade effect from other peers so it’s not only a problem when one of the connected nodes has pre-Allegra software.)

5

Cheers for feedback. RAM use is consistent with that issue (and presumably is a background issue given we are running 1.24.2), but we allow 8GB for nodes so unlikely to be limiting them in any way.
The other characteristics are not consistent so I suspect this has a different cause.
Will scan through the rest of the GitHub issues, nothing jumps out so far…

I’ll open an issue there if needed…