I missed a block today. please help me understand the logs

Sorry but I restarted it before this post

This is probably the last option.
The biggest problem is not that I lost the block, but that I do not know why :cry:

Do you have grafana? If you have, you can check those metrics there.

Thank you for the quick replies. :slight_smile:

if the producer was not aware about that block can be an error generated by cncli (this is happen when you are using wrong vrf files or perhaps cncli db corrupted) can u try to search again on cncli and check the status for that slot?

I made a VM clone to continue the research. I will post the findings (if any)

Also Kes rotating keys should not be expired, not sure this was covered in the replies.

I don’t think the problem has anything to do with cncli. The problem is most likely solely related to cardano-node and something to do with the block producer keys.

The important part in the logs is:

If you are the slot leader, the next log after should be something like:

Jan 05 04:04:00 node1 cardano-node[151911]: [node1:cardano.node.Forge:Info:174
] [2022-01-05 04:04:00.00 UTC] fromList [("val",Object (fromList [("kind",String
 "TraceNodeIsLeader"),("slot",Number 4.9778704e7)])),("credentials",String "Card
ano")]

The important part is: “TraceNodeIsLeader”

If your logs don’t say that then the node didn’t assess that it was the leader. This indicates some problem related to the leadership calculation. As I understand the leadership calculation depends on the keys, stake and nonce values. The stake and nonce values are on-chain whereas the keys are only on your system. Therefore, most likely the problem relates to your keys and whether your cardano-node loaded them.

Do the following:

  1. Determine the start time of your cardano-node with:
startsec=$(curl -s -H 'Accept: application/json' http:/localhost:12788 | jq '.cardano.node.metrics.nodeStartTime.int.val'); startdate="$(date -d @${startsec})"; echo $startdate
  1. Subtract 1-2 seconds from the printed startdate value and use this value to check your logs to see exactly what the cardano-node process logged as its configuration used. Eg:
journalctl --since "2022-01-07 08:48:24" | head -n 10
  1. Review the details of the configuration log which will start with “Node configuration:” and look something like:
Jan 07 08:48:24 node1 cardano-node[166454]: Node configuration: NodeConfiguration {ncNodeIPv4Addr = Just 0.0.0.0, ncNodeIPv6Addr = Nothing, ncNodePortNumber = Just 3000, ncConfigFile = "/opt/cardano/etc/mainnet-config.json", ncTopologyFile = "/opt/cardano/etc/mainnet-topology.json", ncDatabaseFile = "/opt/cardano/db", ncProtocolFiles = ProtocolFilepaths {byronCertFile = Nothing, byronKeyFile = Nothing, shelleyKESFile = Just "/opt/cardano/priv/kes.skey", shelleyVRFFile = Just "/opt/cardano/priv/vrf.skey", shelleyCertFile = Just "/opt/cardano/priv/node.cert", shelleyBulkCredsFile = Nothing}, ncValidateDB = False, ncShutdownIPC = Nothing, ncShutdownOnSlotSynced = NoMaxSlotNo, ... snip ...

The important bit related to your keys is:

shelleyKESFile = Just "/opt/cardano/priv/kes.skey", shelleyVRFFile = Just "/opt/cardano/priv/vrf.skey", shelleyCertFile = Just "/opt/cardano/priv/node.cert"

Note: that these details are for my block producer’s keys which I have stored in “/opt/cardano/priv/”

  1. Check that your keys are stored where your node is looking for them and that they are the correct keys.

Good luck.

1 Like