Failed Adoption (Over incremented during re certification (I think))

Hi Everyone,

My pool “DOJO” was selected to mint today. As far as I can tell the block was not adopted. I’m trying to troubleshoot the event to ensure I have as much a chance to mint and successfully have a block adopted next time. Currently I have 2 questions.

  1. What are the optimal logs to verify transpired events that contain the most important information to what transpired during the minting? I have currently pulled the logs using journalctl for the node application on the producer and have included the logs during assigned slot times.

  2. Assuming the above mentioned logs that have been pulled are the correct logs, my understanding to the error is that during the issuing of a new cert, an error was made during the incrementation of the counter. How do I go about correcting the number (retract by 1)?

Any help is always appreciated. Thanks a bunch!

Logs:

[TareqSer:cardano.node.LeadershipCheck:Info:187] [2023-12-09 01:18:24.00 UTC] {"chainDensity":4.9608234e-2,"credentials":"Cardano","delegMapSize":1322405,"kind":"TraceStartLeadershipCheck","slot":110518413,"utxoSize":10495685}

[TareqSer:cardano.node.Forge:Info:187] [2023-12-09 01:18:24.04 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 1.10518413e8)]))]

[TareqSer:cardano.node.LeadershipCheck:Info:187] [2023-12-09 01:18:25.01 UTC] {"chainDensity":4.9608234e-2,"credentials":"Cardano","delegMapSize":1322405,"kind":"TraceStartLeadershipCheck","slot":110518414,"utxoSize":10495685}

[TareqSer:cardano.node.Forge:Info:187] [2023-12-09 01:18:25.06 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeIsLeader"),("slot",Number 1.10518414e8)]))]

[TareqSer:cardano.node.Forge:Info:187] [2023-12-09 01:18:25.06 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("block",String "aaa68aca6621977e270eba85ef063f3ff031e096193d2e0bd016f24fec4fc453"),("blockNo",Number 9649376.0),("blockPrev",String "e36409d66f412661537c8b9ca22c550a1cbe8b26f4132538b20052dce283e79d"),("kind",String "TraceForgedBlock"),("slot",Number 1.10518414e8)]))]

[TareqSer:cardano.node.ChainDB:Error:177] [2023-12-09 01:18:25.06 UTC] Invalid block aaa68aca6621977e270eba85ef063f3ff031e096193d2e0bd016f24fec4fc453 at slot 110518414: ExtValidationErrorHeader (HeaderProtocolError (HardForkValidationErrFromEra S (S (S (S (S (Z (WrapValidationErr {unwrapValidationErr = CounterOverIncrementedOCERT 2 5}))))))))

[TareqSer:cardano.node.ErrorPolicy:Warning:207] [2023-12-09 01:18:25.06 UTC] IP 192.168.1.4:33147 ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError (MuxIOException Network.Socket.recvBuf: resource vanished (Connection reset by peer)) "(recv errored)"))) 1s 20s

[TareqSer:cardano.node.ChainDB:Info:177] [2023-12-09 01:18:25.06 UTC] Valid candidate e36409d66f412661537c8b9ca22c550a1cbe8b26f4132538b20052dce283e79d at slot 110518409

[TareqSer:cardano.node.Forge:Error:187] [2023-12-09 01:18:25.06 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceForgedInvalidBlock"),("reason",Object (fromList [("error",Object (fromList [("error",Object (fromList [("currentCounter",Number 5.0),("kind",String "CounterOverIncrementedOCERT"),("lastCounter",Number 2.0)])),("kind",String "HeaderProtocolError")])),("kind",String "ValidationError")])),("slot",Number 1.10518414e8)]))]

[TareqSer:cardano.node.LeadershipCheck:Info:187] [2023-12-09 01:18:26.01 UTC] {"chainDensity":4.9608234e-2,"credentials":"Cardano","delegMapSize":1322405,"kind":"TraceStartLeadershipCheck","slot":110518415,"utxoSize":10495685}

[TareqSer:cardano.node.Forge:Info:187] [2023-12-09 01:18:26.06 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 1.10518415e8)]))]

run this command (adapt the path/files name) to check if the certificate is valid or not

cardano-cli query kes-period-info --mainnet --socket-path /home/cardano/cardano-my-node/db/socket --op-cert-file $NODE_HOME/node.cert
1 Like

Thank you @Alexd1985, it returned:
"Operational certificate’s KES period is within the correct KES period interval.

The operational certificate counter too far ahead of the node protocol state counter in the operational certificate at: /home/tareq/cardano-my-node/mainnet/node.cert
On disk operational certificate counter: 5
Protocol state counter: 2"

So how do I correct the over incrementation?

Renew the certificate but use the right incremental number… check on adapools.org for the last number used when u created the last block… and use next one

Cheers,

2 Likes

Thank you @Alexd1985, I have found the information as directed. Most recent block creation indicates i was using my 2nd certificate. I will use 3 for this th
image
e new certificate

1 Like