Selected for slot leader but failed to mint new block

Staking & Delegation Operate a Stake Pool
1

Hi, we have missed a block and I can’t figure out why.
Node is running well no issues and /.gLive is counting up the leader slots

Screenshot 2021-07-01 at 13.57.41
Screenshot 2021-07-01 at 13.57.411168×860 163 KB 

> {"thread":"868","loc":null,"data":{"kind":"TraceAddBlockEvent.AddBlockValidation.InvalidBlock","block":{"kind":"Point","hash":"bea1facbb2499fc1187c925b403f4e691865b346b38dd45be64adb94892bf822","slot":33566834},"error":"ExtValidationErrorHeader (HeaderProtocolError (HardForkValidationErrFromEra S (S (S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (OcertFailure (InvalidKesSignatureOCERT 259 253 6 \"Reject\"))]}))))))"},"sev":"Error","env":"1.27.0:8fe46","msg":"","app":[],"host":"chris-Po","pid":"50521","ns":["cardano.node.ChainDB"],"at":"2021-07-01T09:52:05.04Z"}
> {"thread":"868","loc":null,"data":{"kind":"TraceAddBlockEvent.AddBlockValidation.ValidCandidate","block":"9c1bbea16714f89bd644ef014cc24d06d68ec94f968c953222310977dcaf71e4@33566779"},"sev":"Info","env":"1.27.0:8fe46","msg":"","app":[],"host":"chris-Po","pid":"50521","ns":["cardano.node.ChainDB"],"at":"2021-07-01T09:52:05.04Z"}
> {"thread":"878","loc":null,"data":{"val":{"kind":"TraceForgedInvalidBlock","slot":33566834,"reason":{"kind":"ValidationError","error":{"kind":"HeaderProtocolError","error":{"failures":[{"opCertExpectedKESEvolutions":"6","kind":"InvalidKesSignatureOCERT","opCertKESStartPeriod":"253","opCertKESCurrentPeriod":"259","error":"Reject"}],"kind":"ChainTransitionError"}}}},"credentials":"Cardano"},"sev":"Error","env":"1.27.0:8fe46","msg":"","app":[],"host":"chris-Po","pid":"50521","ns":["cardano.node.Forge"],"at":"2021-07-01T09:52:05.05Z"}
2

The error has looks to have something to do with the KesSignature

I followed CoinCashew to renew my KES period a few days ago

3

Full slot log output:

{"thread":"878","loc":null,"data":{"kind":"TraceStartLeadershipCheck","chainDensity":5.079044e-2,"slot":33566834,"delegMapSize":670001,"utxoSize":2752458,"credentials":"Cardano"},"sev":"Info","env":"1.27.0:8fe46","msg":"","app":[],"host":"chris-Po","pid":"50521","ns":["cardano.node.LeadershipCheck"],"at":"2021-07-01T09:52:05.00Z"}
> {"thread":"878","loc":null,"data":{"val":{"kind":"TraceNodeIsLeader","slot":33566834},"credentials":"Cardano"},"sev":"Info","env":"1.27.0:8fe46","msg":"","app":[],"host":"chris-Po","pid":"50521","ns":["cardano.node.Forge"],"at":"2021-07-01T09:52:05.00Z"}
> {"thread":"878","loc":null,"data":{"val":{"kind":"TraceForgedBlock","slot":33566834},"credentials":"Cardano"},"sev":"Info","env":"1.27.0:8fe46","msg":"","app":[],"host":"chris-Po","pid":"50521","ns":["cardano.node.Forge"],"at":"2021-07-01T09:52:05.01Z"}
> {"thread":"868","loc":null,"data":{"kind":"TraceAddBlockEvent.AddBlockValidation.InvalidBlock","block":{"kind":"Point","hash":"bea1facbb2499fc1187c925b403f4e691865b346b38dd45be64adb94892bf822","slot":33566834},"error":"ExtValidationErrorHeader (HeaderProtocolError (HardForkValidationErrFromEra S (S (S (Z (WrapValidationErr {unwrapValidationErr = ChainTransitionError [OverlayFailure (OcertFailure (InvalidKesSignatureOCERT 259 253 6 \"Reject\"))]}))))))"},"sev":"Error","env":"1.27.0:8fe46","msg":"","app":[],"host":"chris-Po","pid":"50521","ns":["cardano.node.ChainDB"],"at":"2021-07-01T09:52:05.04Z"}
> {"thread":"868","loc":null,"data":{"kind":"TraceAddBlockEvent.AddBlockValidation.ValidCandidate","block":"9c1bbea16714f89bd644ef014cc24d06d68ec94f968c953222310977dcaf71e4@33566779"},"sev":"Info","env":"1.27.0:8fe46","msg":"","app":[],"host":"chris-Po","pid":"50521","ns":["cardano.node.ChainDB"],"at":"2021-07-01T09:52:05.04Z"}
> {"thread":"878","loc":null,"data":{"val":{"kind":"TraceForgedInvalidBlock","slot":33566834,"reason":{"kind":"ValidationError","error":{"kind":"HeaderProtocolError","error":{"failures":[{"opCertExpectedKESEvolutions":"6","kind":"InvalidKesSignatureOCERT","opCertKESStartPeriod":"253","opCertKESCurrentPeriod":"259","error":"Reject"}],"kind":"ChainTransitionError"}}}},"credentials":"Cardano"},"sev":"Error","env":"1.27.0:8fe46","msg":"","app":[],"host":"chris-Po","pid":"50521","ns":["cardano.node.Forge"],"at":"2021-07-01T09:52:05.05Z"}
4

after u refreshed the KES, you restarted the node right?

5

Ive restarted the node several times yes. Will restart again just in case

Grafana dashboard recognised that the key rotation has reset if thats any indicator that it worked…

6

search inside the logs if u still have this error

7

When the node attempts to mint a new block? Or is there a command I can run to simulate the process?

8

try to search about this error | grep InvalidKesSignatureOCERT

9

hmm, after u rotated the kes, which files u copied to the producer? also a new pool.cert was generated; did u copied to the producer?

Check this topic

10

Node.cert
The new pool.cert was to include dns names for both relays
Would the bp run if the registered pool.cert was different from the pool.cert on my bp?

11

I think yes, it will work with the old one, didn’t u copied the new pool.cert on the Producer?

but pool.cert and op.cert are 2 different files
you error tells that there is an issue with the op.cert so, the new op.cert was copied to the BP after u refreshed the KES?

12

No Im pretty sure I did, both the files were updated on the same date. Node.cert matches my node.cert on the cold environment same with the pool.cert

13

aaa, ok then

14

try to copy the files again and restart the node… and wait for the 2nd block

15

So copy updated node.cert and pool.cert from cold env to bp. No better way to check if it works other than waiting for a block?

16

ok, I wanted to ask you, what files did u copied from cold server to the BP after u refreshed the KES?

17

just node.cert as stated in the guide:

Copy kes.vkey to your cold environment.

Create the new node.cert file with the following command. Update <startKesPeriod> with the value from above.

air-gapped offline machine:


cd $NODE_HOME

chmod u+rwx $HOME/cold-keys

cardano-cli node issue-op-cert \

--kes-verification-key-file kes.vkey \

--cold-signing-key-file $HOME/cold-keys/node.skey \

--operational-certificate-issue-counter $HOME/cold-keys/node.counter \

--kes-period <startKesPeriod> \

--out-file node.cert

chmod a-rwx $HOME/cold-keys

Copy node.cert back to your block producer node.

Stop and restart your block producer node to complete this procedure.

18

ok, I refreshed the KES on 28 of June and I checked now and I have 4 files modified since then:

  • hot.skey
  • hot.vkey
  • kes.start
  • op.cert

you are using these 3 files when u start the producer:

  • KES=${DIRECTORY}/kes.skey

  • VRF=${DIRECTORY}/vrf.skey

  • CERT=${DIRECTORY}/node.cert

so check if kes.skey and node.cert were updated same time/date

19

they were updated yes but at different times… so Im guessing they were modified when copied over to my cold env. I did not copy them back. I will copy the kes.v/skey to the hot env and hope it solves the issue. Thanks for your help Alex

1 Like
20

let us know if solved or not the issue

meantime u can check also other topics, same issue

https://forum.cardano.org/search?q=InvalidKesSignatureOCERT