Missed a block: Validation error on Operational Certicate

Staking & Delegation Operate a Stake Pool
1

Hi Guys, we updated our relays and BP to 1.29.0 a few days ago ánd rotated KES keys as well. This morning we shoulded mint a block but it was not adopted. :sob: :sob:
This is what the log says:

{“thread”:“567”,“loc”:null,“data”:{“kind”:“TraceStartLeadershipCheck”,“chainDensity”:4.845044e-2,“slot”:38997933,“delegMapSize”:798734,“utxoSize”:3020093,“credentials”:“Cardano”},“sev”:“Info”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“bnode1”,“pid”:“520687”,“ns”:[“cardano.node.LeadershipCheck”],“at”:“2021-09-02T06:30:24.00Z”}

{“thread”:“567”,“loc”:null,“data”:{“val”:{“kind”:“TraceNodeIsLeader”,“slot”:38997933},“credentials”:“Cardano”},“sev”:“Info”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“bnode1”,“pid”:“520687”,“ns”:[“cardano.node.Forge”],“at”:“2021-09-02T06:30:24.00Z”}

{“thread”:“567”,“loc”:null,“data”:{“val”:{“kind”:“TraceForgedBlock”,“blockPrev”:“dd45fdcedd60d40fa50fc33e3a3dfacf82dc9f383798769dfd7fcc4ab6d1d9b8”,“slot”:38997933,“block”:“dc437ec04142ad568bb4ab090bba699d1145a8232242433623e53c400126f3d4”,“blockNo”:6191124},“credentials”:“Cardano”},“sev”:“Info”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“bnode1”,“pid”:“520687”,“ns”:[“cardano.node.Forge”],“at”:“2021-09-02T06:30:24.01Z”}

{“thread”:“557”,“loc”:null,“data”:{“kind”:“TraceAddBlockEvent.AddBlockValidation.InvalidBlock”,“block”:{“kind”:“Point”,“hash”:“dc437ec04142ad568bb4ab090bba699d1145a8232242433623e53c400126f3d4”,“slot”:38997933},“error”:“ExtValidationErrorHeader (HeaderProtocolError (HardForkValidationErrFromEra S (S (S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (OcertFailure (CounterTooSmallOCERT 3 1))]}))))))”},“sev”:“Error”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“bnode1”,“pid”:“520687”,“ns”:[“cardano.node.ChainDB”],“at”:“2021-09-02T06:30:24.02Z”}

{“thread”:“557”,“loc”:null,“data”:{“kind”:“TraceAddBlockEvent.AddBlockValidation.ValidCandidate”,“block”:“dd45fdcedd60d40fa50fc33e3a3dfacf82dc9f383798769dfd7fcc4ab6d1d9b8@38997884”},“sev”:“Info”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“bnode1”,“pid”:“520687”,“ns”:[“cardano.node.ChainDB”],“at”:“2021-09-02T06:30:24.02Z”}

{“thread”:“567”,“loc”:null,“data”:{“val”:{“kind”:“TraceForgedInvalidBlock”,“slot”:38997933,“reason”:{“kind”:“ValidationError”,“error”:{“kind”:“HeaderProtocolError”,“error”:{“failures”:[{“lastKESCounter”:“3”,“kind”:“CounterTooSmallOCert”,“currentKESCounter”:“1”,“error”:“The operational certificate’s last KES counter is greater than the current KES counter.”}],“kind”:“ChainTransitionError”}}}},“credentials”:“Cardano”},“sev”:“Error”,“env”:“1.29.0:4c594”,“msg”:"",“app”:,“host”:“bnode1”,“pid”:“520687”,“ns”:[“cardano.node.Forge”],“at”:“2021-09-02T06:30:24.02Z”}

We just generated the operational certificate as usual.

cardano-cli node issue-op-cert --kes-verification-key-file kes.vkey --cold-signing-key-file cold.skey --operational-certificate-issue-counter cold.counter --kes-period 299 --out-file node.cert

In gLiveView I can see expiration date of the operational Certificate:
KES current/remaining : 301 / 60 │
KES expiration date : 2021-12-01 09:44:51 UTC

Any clou? Thanks in avance!

2

Looks like the counter number is smaller than the previously used one

Maybe you have deleted the cold.counter file and therfore when you created a new set of keys/cert it again started to count from 0

You will need to rotate few times the keys and grow the counter value (in cold.counter file)

cardano-cli node issue-op-cert --kes-verification-key-file kes.vkey --cold-signing-key-file cold.skey --operational-certificate-issue-counter cold.counter --kes-period 299 --out-file node.cert

This is a security mechanism is there to protect you when someone steals a still valid keys. You just generate new set - and as the couner for new set will have a higher value, then the blockchain will ignore the keys/cert with lower counter value.

Hope this helps

Lauris

3

Yes it make sense the way you explain it, but how to know this kind of things? Running a stakepool for about a year now and never suspected such an issue. Sadly, because we are a small pool and every block is more than welcome, also to keep delegators attracted. :sweat: Anyway, I will generate the keys a couple of times. Many thanks for your help!

4

Maybe this script can help you. https://github.com/ada-piggy-bank/pool-utils
I haven’t used it as is, so look through the code, check all lines and see if it’ll do the job for future rotations. (I recommend not running anything you are not familiar with unless it comes from a trusted source)

Edit: you can use some tricks from the script to get your current rotation and check it against the counter file (i guess the counter file is not encoded, don’t have it with me now to check)

5

Indeed it is a useful script and one use case covers exactly my case: * Node certificate was generated with a stale node.counter file (retrieved from backup) Will run the test, make sure my current node.counter is higher than the current one.

Love this community with you nice fellow SPO’s always willing to help. Thanks!

6

:+1: let us know if that was indeed the problem