Missed Block(s)


Recently saw my BP node show Leader: 1 without any blocks being Adopted or Invalid. Ran the below journalctl and received the below.

sudo journalctl --unit=cardano-node | grep -A 2 “NodeIsLeader”

Dec 05 02:28:19 BP-node cardano-node[1606]: [BP-node:cardano.node.Forge:Info:154] [2021-12-05 08:28:19.00 UTC] fromList [(“val”,Object (fromList [(“kind”,String “TraceNodeIsLeader”),(“slot”,Number 4.7126608e7)])),(“credentials”,String “Cardano”)]

Dec 05 02:28:19 BP-node cardano-node[1606]: [BP-node:cardano.node.Forge:Info:154] [2021-12-05 08:28:19.01 UTC] fromList [(“val”,Object (fromList [(“block”,String “c7032412d17a81a153233f544fa9b0c158092f6655f102a84d93f50a759bc47f”),(“blockNo”,Number 6587861.0),(“blockPrev”,String “989516b7ba57fda51edf63d8fb6210dd81c248e9864b9bb5b7a1452fdb1f9744”),(“kind”,String “TraceForgedBlock”),(“slot”,Number 4.7126608e7)])),(“credentials”,String “Cardano”)]

Dec 05 02:28:19 BP-node cardano-node[1606]: [BP-node:cardano.node.ChainDB:Error:144] [2021-12-05 08:28:19.02 UTC] Invalid block c7032412d17a81a153233f544fa9b0c158092f6655f102a84d93f50a759bc47f at slot 47126608: ExtValidationErrorHeader (HeaderProtocolError (HardForkValidationErrFromEra S (S (S (S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (OcertFailure (InvalidKesSignatureOCERT 363 328 35 “Reject”))]})))))))

Do this only indicate that the KES is wrong? I previously looked into this and ran

ls -l kes.skey node.cert

This came back with the KES being a minute or two older with the old KES, which in the SPOT KES checker says it may have not be an issue.

Well since the missed block, I rotated my KES and now the KES is younger than the node.cert which would make sense in order or creation. So hopefully this will solve the KES issue, but just wanted to get some insight if there is anything else I should look for or do before the next block.

This was fixed by rotating the KES and making sure that the code pointed to the correct KES.

I built from cashew guide so KES was in $NODE_HOME and not cold-keys. I never copied it to the $NODE_HOME location but had copied to the cold-keys backup location. So when I ran the code it used the old KES making it invalid but showing on Guild Live View it being correct because the counter value was correct.

So always make sure your PATHS are correct!

1 Like