KES Certificate Question when Swapping between Block Producer and Backup Block Producer

I experienced an “invalid block” because of this issue:
{"host":"someserver","pid":"65351","loc":null,"at":"2022-03-31T23:11:06.10Z","ns":["cardano.node.Forge"],"sev":"Error","env":"1.34.1:73f9a","data":{"val":{"kind":"TraceForgedInvalidBlock","reason":{"kind":"ValidationError","error":{"kind":"HeaderProtocolError","error":{"kind":"ChainTransitionError","failures":[{"lastKESCounter":"10","currentKESCounter":"9","kind":"CounterTooSmallOCert","error":"The operational certificate's last KES counter is greater than the current KES counter."}]}}},"slot":57201975},"credentials":"Cardano"},"msg":"","thread":"653","app":[]}

Now, I swap between my block producer and my backup block producer now and then. The issue is that I was running my backup block producer for epoch 329. At the time I rotated the KES certificate on the backup producer. Then I returned to my main production node for the current epoch (330) but didn’t update the KES certificate as it still had some time on it before it expires.

So did I lose that block because my KES certificate on the main block producer was older than the one from off my backup blockproducer. So each time we swap block producing nodes it is a good idea to rotate the KES certificate?



Once u rotated the KES (create a new set of KES) on a BP and created blocks it’s mandatory to upload the files also on bkp BP because there will be a mismatch between operational certificate

Fo example if on a BP u created blocks with op certificate counter 6 u can’t use on another BP(bkp) a certificate with a smaller counter… the counter used should be equal or greater than the latest used

Best practice: when u rotate the KES on a BP also upload the files regenerated to bkp BP