Hello,
It was not noticed until afterwards, but the kes key was rotated when the block producer came back online from the 1.31.0 upgrade while the node was still catching up. Following this guide for initial setup and kes rotation - 18. Operational and Maintenance Tips - CoinCashew
Additional details if applicable:
-Kes was expired for many days prior to rotation.
-gLiveView shows expired as well as running curl localhost:12798/metrics
cardano_node_metrics_remainingKESPeriods_int 0
tailing the logs I am seeing the following:
Dec 11 04:17:26 infra-cardano-blockproducer-node-01 cardano-node[776447]: [infra-ca:cardano.node.Forge:Error:365] [2021-12-11 04:17:26.00 UTC] fromList [(“val”,Object (fromList [(“kind”,String “TraceForgeStateUpdateError”),(“reason”,Object (fromList [(“forgeStateUpdateError”,Object (fromList [(“kesInfo”,Object (fromList [(“endPeriod”,Number 72.0),(“evolution”,Number 0.0),(“kind”,String “KESInfo”),(“startPeriod”,Number 10.0)])),(“kind”,String “KESKeyAlreadyPoisoned”),(“targetPeriod”,Number 367.0)])),(“kind”,String “HardForkForgeStateUpdateError”)])),(“slot”,Number 4.7629955e7)])),(“credentials”,String “Cardano”)]
Screen grab from glive:
