Pool not processing transaction anymore

Staking & Delegation Operate a Stake Pool
1

Hi,

I’ve been operating a pool since the beginning of Shelley and haven’t got a problem until few days ago that’s I’ve been trying to resolve.

All 2 relays and the bp seem to have stopped producing transactions (cardano_node_metrics_txsProcessedNum_int and cardano_node_metrics_txsInMempool_int don’t report anything anymore), and I see those messages in the logs:

Apr 21 18:44:29 ns3030548 cardano-node-relay[2588]: [ns303054:cardano.node.ChainDB:Notice:158] [2023-04-21 18:44:29.02 UTC] Chain extended, new tip: 40511e17f6f0525007f1ed9830aa927539ab301aa29e7a63ff1d279da202b59b at slot 90536375

Apr 21 18:44:44 ns3030548 cardano-node-relay[2588]: [ns303054:cardano.node.BlockFetchDecision:Info:168] [2023-04-21 18:44:44.14 UTC] [TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:40725, remoteAddress = 18.133.40.48:3001}) (Right [At (Block {blockPointSlot = SlotNo 90536393, blockPointHash = 00c857bf1e707093f8d74bf6b7d91b80c27ca2cf56669d492dfcf7fbbdf2f8a8})]),TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:43361, remoteAddress = 54.36.178.85:6000}) (Left FetchDeclineChainNotPlausible),TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:41463, remoteAddress = 65.21.58.20:3001}) (Left FetchDeclineChainNotPlausible),TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:39881, remoteAddress = 149.28.44.125:3001}) (Left FetchDeclineChainNotPlausible),TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:41625, remoteAddress = 173.50.95.212:3001}) (Left FetchDeclineChainNotPlausible),TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:33129, remoteAddress = 3.128.250.3:3001}) (Left FetchDeclineChainNotPlausible),TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:43081, remoteAddress = 173.50.95.210:3001}) (Left FetchDeclineChainNotPlausible)]

Apr 21 18:44:44 ns3030548 cardano-node-relay[2588]: [ns303054:cardano.node.BlockFetchDecision:Info:168] [2023-04-21 18:44:44.15 UTC] [TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:43361, remoteAddress = 54.36.178.85:6000}) (Left (FetchDeclineConcurrencyLimit FetchModeDeadline 1)),TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:41463, remoteAddress = 65.21.58.20:3001}) (Left FetchDeclineChainNotPlausible),TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:40725, remoteAddress = 18.133.40.48:3001}) (Left FetchDeclineInFlightThisPeer),TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:39881, remoteAddress = 149.28.44.125:3001}) (Left FetchDeclineChainNotPlausible),TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:41625, remoteAddress = 173.50.95.212:3001}) (Left FetchDeclineChainNotPlausible),TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:33129, remoteAddress = 3.128.250.3:3001}) (Left FetchDeclineChainNotPlausible),TraceLabelPeer (ConnectionId {localAddress = 149.202.88.213:43081, remoteAddress = 173.50.95.210:3001}) (Left FetchDeclineChainNotPlausible)]
Apr 21 18:44:44 ns3030548 cardano-node-relay[2588]: [ns303054:cardano.node.ChainDB:Notice:158]

[2023-04-21 18:44:44.18 UTC] Chain extended, new tip: 00c857bf1e707093f8d74bf6b7d91b80c27ca2cf56669d492dfcf7fbbdf2f8a8 at slot 90536393

I tried looking up for someone with similar issue but I can’t see anything.

The version of the node running on all 3 is:

cardano-node 1.35.7 - linux-x86_64 - ghc-8.10
git rev f0b4ac897dcbefba9fa0d247b204a24543cf55f6

It just suddenly stopped processing transaction, and I did the upgrades a while back before that.

On top of that, I deleted all 3 dbs and resynchronised from genesis, but nothing changed

Would really appreciate if someone had an idea of what I can try to solve this

Thanks,

Matt

2

hi,

check:

  • if inside configuration file do you have TraceMempool set to true (in case is set to false you will need to set to true)
  • if you have more IN peers, other than your BP

Cheers,

1 Like
3
  • the TradeMempool didn’t change for the last year and is set to true
  • I have exactly the same peers I used for the last year as well (and that includes IOHK relays)

I tried rollbacking to 1.35.5, but no luck there either

4

can u share glive output from one relays? Also can u share the pool ticker?