Error: Switchboard's queue full, dropping log items!

Gilberto · 4 March 2021 21:18

I found another post that explains very well what might be the cause of the spikes

It mentioned also an open issue that focuses on fixing the bug

github.com/input-output-hk/ouroboros-network

Emit a unique Mempool tracer event for duplicate txs

opened 10:23PM - 10 Feb 21 UTC

closed 12:46AM - 07 Apr 21 UTC

nfrisby

good first issue

consensus

Node and Consensus chatted about input-output-hk/cardano-node#2350 on Slack. Thi…s Issue captures our first thought about how to assist. The SPOs are seeing high CPU usage until they disable mempool tracing. The leading suspect is that we're seeing duplicate rejection events when we fetch the same tx from multiple peers. See that `cardano-node` ticket for more info. The node's TxSub logic -- in particular contrast to BlockFetch -- makes no attempt to fetch a tx from only one peer. In other words, the various TxSub server threads running on a node only first synchronize by adding their fetched txs to the node's single mempool. In general, only the first of the per-peer TxSub server thread to do so will be able to add each tx. So we expect these duplicate rejections given our design (favoring the simplicity of not prematurely synchronizing the TxSub server threads), and have not seen them causing negligible (edit: I meant "non-negligible") CPU/network overhead eg. This is supported by the SPOs seeing less utilization by merely disabling the mempool tracers. This Issue is therefore to emit a distinct event for duplicate transactions, separate from txs that are rejected for more interesting reasons. This will allow the node team to easily avoid the apparent expense of these duplicate tx rejection ids without losing all the mempool tracing logs.

In the meantime, I had to disable TraceMempool in config.json and lose the ability to monitor transactions.

Topic		Replies	Views
Relay node - Info log with error Operate a Stake Pool	6	916	4 January 2022
Node remains on status syncing 100% Operate a Stake Pool	16	1225	9 March 2021
[IN PROGRESS] CPU load at maximum since 1.25.1 Operate a Stake Pool	21	2558	10 April 2021
SlotsMissedNum_int Operate a Stake Pool	4	706	30 December 2021
System time sync issue, causing relay wreck Operate a Stake Pool stake-pools , stakepool , pool , pool-operator	5	1387	30 November 2020

Error: Switchboard's queue full, dropping log items!

Related topics