Invalid KES signature OCERT but gLiveView shows KES is valid

Have recently rotated KES following steps.
I cannot get any block to get adopted despite being selected as a slot leader number of times already. As you can see Guild LiveView tool shows that KES kes are valid. But the logs from cardano-node show otherwise.

enter image description here

Jun 17 16:39:29 wladyslawa bash[1021152]:
[wladysla:cardano.node.ChainDB:Error:82] [2021-06-17 15:39:29.01 UTC]
Invalid block
95def156e23112ebe3f9c6832ac96059d10ff5c446bdbbbc275b2698dfc336d0 at
slot 32378078: ExtValidationErrorHeader (HeaderProtocolError
(HardForkValidationErrFromEra S (S (S (Z (WrapValidationErr
{unwrapValidationErr = ChainTransitionError [OverlayFailure
(OcertFailure (InvalidKesSignatureOCERT 249 243 6 “Reject”))]}))))))

More errors:

maciej@ola:~/Downloads$ grep -ir ":error" logs.txt  -C 1
Jun 17 14:00:38 wladyslawa bash[1021152]: [wladysla:cardano.node.Forge:Info:92] [2021-06-17 13:00:38.00 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 3.2368547e7)])),("credentials",String "Cardano")]
Jun 17 14:00:38 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Error:73120] [2021-06-17 13:00:38.48 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Application Exception: 192.168.0.12:6000 ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (ServerAgency TokNext TokMustReply)
Jun 17 14:00:38 wladyslawa bash[1021152]: [wladysla:cardano.node.ErrorPolicy:Notice:100] [2021-06-17 13:00:38.48 UTC] IP 192.168.0.12:6000 ErrorPolicySuspendConsumer (Just (ApplicationExceptionTrace ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (ServerAgency TokNext TokMustReply))) 20s
--
Jun 17 14:04:25 wladyslawa bash[1021152]: [wladysla:cardano.node.Forge:Info:92] [2021-06-17 13:04:25.00 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 3.2368774e7)])),("credentials",String "Cardano")]
Jun 17 14:04:25 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Error:99511] [2021-06-17 13:04:25.20 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Application Exception: 192.168.0.12:6000 ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (ServerAgency TokNext TokMustReply)
Jun 17 14:04:25 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Info:99511] [2021-06-17 13:04:25.20 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Closed socket to 192.168.0.12:6000
--
Jun 17 14:05:56 wladyslawa bash[1021152]: [wladysla:cardano.node.Forge:Info:92] [2021-06-17 13:05:56.00 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 3.2368865e7)])),("credentials",String "Cardano")]
Jun 17 14:05:56 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Error:99547] [2021-06-17 13:05:56.21 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Application Exception: 192.168.0.12:6000 ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (ServerAgency TokNext TokMustReply)
Jun 17 14:05:56 wladyslawa bash[1021152]: [wladysla:cardano.node.ErrorPolicy:Notice:100] [2021-06-17 13:05:56.21 UTC] IP 192.168.0.12:6000 ErrorPolicySuspendConsumer (Just (ApplicationExceptionTrace ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) ('[] *))))))) (ServerAgency TokNext TokMustReply))) 20s
--
Jun 17 14:05:57 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Info:102] [2021-06-17 13:05:57.21 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Skipping peer 192.168.0.12:6000
Jun 17 14:05:57 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Error:102] [2021-06-17 13:05:57.21 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Failed to start all required subscriptions
Jun 17 14:05:57 wladyslawa bash[1021152]: [wladysla:cardano.node.Mempool:Info:99577] [2021-06-17 13:05:57.23 UTC] fromList [("tx",Object (fromList [("txid",String "txid: TxId {_unTxId = SafeHash \"fb5f593592735aa5507b2881705e902611729620968b4c890718cb775ce9d4fc\"}")])),("mempoolSize",Object (fromList [("bytes",Number 87490.0),("numTxs",Number 152.0)])),("kind",String "TraceMempoolRejectedTx"),("err",Object (fromList [("badInputs",Array [String "d22db824b458406b322b8e2620768bd3c5ff1120208de2fbce57d5b553a69ff5#1"]),("consumed",Object (fromList [("lovelace",Number 0.0),("policies",Object (fromList []))])),("error",String "This transaction consumed Value 0 (fromList []) but produced Value 284069674 (fromList [(PolicyID {policyID = ScriptHash \"defca4b95934aec82b818cbafd4bd0ab45511c1d690b58759f07d9b1\"},fromList [(\"VyFi\",24900)])])"),("kind",String "ValueNotConservedUTxO"),("produced",Object (fromList [("lovelace",Number 2.84069674e8),("policies",Object (fromList [("defca4b95934aec82b818cbafd4bd0ab45511c1d690b58759f07d9b1",Object (fromList [("56794669",Number 24900.0)]))]))]))]))]
--
Jun 17 14:06:07 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Info:102] [2021-06-17 13:06:07.21 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Skipping peer 192.168.0.12:6000
Jun 17 14:06:07 wladyslawa bash[1021152]: [wladysla:cardano.node.IpSubscription:Error:102] [2021-06-17 13:06:07.21 UTC] IPs: 0.0.0.0:0 [192.168.0.12:6000] Failed to start all required subscriptions
Jun 17 14:06:08 wladyslawa bash[1021152]: [wladysla:cardano.node.LeadershipCheck:Info:92] [2021-06-17 13:06:08.00 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":5.040153e-2,"slot":32368877,"delegMapSize":644495,"utxoSize":2660038,"credentials":"Cardano"}
--
Jun 17 16:39:29 wladyslawa bash[1021152]: [wladysla:cardano.node.Forge:Info:92] [2021-06-17 15:39:29.00 UTC] fromList [("val",Object (fromList [("kind",String "TraceForgedBlock"),("slot",Number 3.2378078e7)])),("credentials",String "Cardano")]
Jun 17 16:39:29 wladyslawa bash[1021152]: [wladysla:cardano.node.ChainDB:Error:82] [2021-06-17 15:39:29.01 UTC] Invalid block 95def156e23112ebe3f9c6832ac96059d10ff5c446bdbbbc275b2698dfc336d0 at slot 32378078: ExtValidationErrorHeader (HeaderProtocolError (HardForkValidationErrFromEra S (S (S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (OcertFailure (InvalidKesSignatureOCERT 249 243 6 "Reject"))]}))))))
Jun 17 16:39:29 wladyslawa bash[1021152]: [wladysla:cardano.node.ChainDB:Info:82] [2021-06-17 15:39:29.01 UTC] Valid candidate 82548006ab042d802e1f885e028ea93038282d028b9530999e9dfc63473ff247 at slot 32378076
Jun 17 16:39:29 wladyslawa bash[1021152]: [wladysla:cardano.node.Forge:Error:92] [2021-06-17 15:39:29.01 UTC] fromList [("val",Object (fromList [("kind",String "TraceForgedInvalidBlock"),("reason",Object (fromList [("error",Object (fromList [("error",Object (fromList [("failures",Array [Object (fromList [("error",String "Reject"),("kind",String "InvalidKesSignatureOCERT"),("opCertExpectedKESEvolutions",String "6"),("opCertKESCurrentPeriod",String "249"),("opCertKESStartPeriod",String "243")])]),("kind",String "ChainTransitionError")])),("kind",String "HeaderProtocolError")])),("kind",String "ValidationError")])),("slot",Number 3.2378078e7)])),("credentials",String "Cardano")]
Jun 17 16:39:30 wladyslawa bash[1021152]: [wladysla:cardano.node.LeadershipCheck:Info:92] [2021-06-17 15:39:30.00 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":5.0332207e-2,"slot":32378079,"delegMapSize":644766,"utxoSize":2661388,"credentials":"Cardano"}

All logs at Dropbox.

1 Like

There was such case earlier for @LuckySlam

but not sure what was the solution

1 Like

Related on cardano.stackexchange.com
stake pool - Invalid KES signature OCERT but gLiveView shows KES is valid - Cardano Stack Exchange

1 Like

I know this is an old post, but I had this same problem after moving my node to a different machine. Even though my KES keys were up to date and I had copied the right files over, my journal was throwing errors when trying to mint blocks as if the KES period was off.
I found the errors by searching like this:

sudo journalctl --unit=cardano-node -g InvalidKesSignature

Just wanted to leave this post for people with similar problems. Rotating your KES keys and node.cert files was the quickest way to solve my the problem.

-Sully at RADAR