Missing blocks on 1.35.3 with fresh operational certificate

Staking & Delegation Operate a Stake Pool
, ,
1

Been trying to figure this one out for a few days but no joy.
Steps to repro:

  1. Upgrade to cardano-node v. 1.35.3

  2. Issue new op. certificate by following the official guidelines found here and echoed by this article.

Before issuing new certificate cardano-cli query kes-period-info outputs this:
Screenshot 2022-08-24 at 21.42.08

After issuing new certificate cardano-cli query kes-period-info outputs this:
Screenshot 2022-08-24 at 21.43.12

Then, when we eventually reach the slot assigned to the node for block production the logs on the BP output this:

Aug 24 16:30:45 cardano-node[558984]: [:cardano.node.LeadershipCheck:Info:253] [2022-08-24 16:30:45.00 UTC] {"chainDensity":4.8052527e-2,"credentials":"Cardano","delegMapSize":1213997,"kind":"TraceStartLeadershipCheck","slot":69792354,"utxoSize":8428808}
Aug 24 16:30:45 cardano-node[558984]: [cardano.node.Forge:Info:253] [2022-08-24 16:30:45.02 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("kind",String "TraceNodeIsLeader"),("slot",Number 6.9792354e7)]))]
Aug 24 16:30:45 cardano-node[558984]: [cardano.node.Forge:Info:253] [2022-08-24 16:30:45.04 UTC] fromList [("credentials",String "Cardano"),("val",Object (fromList [("block",String "350ec4c630a803bc9b6df2e2c9b0ab22f5f34ba5d294ede7a8fbe223c4c6f766"),("blockNo",Number 7667871.0),("blockPrev",String "82f0d662df2182fa2b817ab1efe457f5f3c86406363933013b46a1ccc4ec39fe"),("kind",String "TraceForgedBlock"),("slot",Number 6.9792354e7)]))]
Aug 24 16:30:45 cardano-node[558984]: [cardano.node.ChainDB:Notice:243] [2022-08-24 16:30:45.07 UTC] Chain extended, new tip: 350ec4c630a803bc9b6df2e2c9b0ab22f5f34ba5d294ede7a8fbe223c4c6f766 at slot 69792354

While the relay logs output this:
Aug 24 16:30:51 cardano-node[3620399]: [:cardano.node.ChainDB:Info:12737] [2022-08-24 16:30:51.14 UTC] Block fits onto some fork: 350ec4c630a803bc9b6df2e2c9b0ab22f5f34ba5d294ede7a8fbe223c4c6f766 at slot 69792354

I initially thought that I lost the block to a slot battle however pooltool does not show any slot battles for the lost block (at height 7667871):
Screenshot 2022-08-24 at 21.58.10

After missing a couple of blocks because of this issue, I reverted back to my old node.cert and kes.skey files, restarted the BP without changing any other setting and lo and behold, a block was minted!

Anyone else bump into this issue? Any help would be appreciated!

2 Likes
2

Here is an explanation… and according to it ur new certificate was fine…

3

Not sure what the explanation is, there’s a lot of info there. Have you encountered this on your own node?

4

Nope. No issues… all blocks were minted with success

5

With newly rotated KES keys + new op.cert as well?

6

Why you need rotate cert? It is not necessary for 1.35.3

7

i rotated the keys before to upgrade the node to 1.35.3

8

Because as coincidence has it, they are about to expire in about 2 days from now.

9

Lucky you!:laughing:

10

But it is wired… try to open a ticket to IOHK… something must happened… if it was an issue with certificate u should see it inside the logs

11

Will do. BTW, if you run the query kes-period-info command on your current build, do you get "qKesKesKeyExpiry": null or an actual date instead?

12

This shouldn’t be an issue until the Vasil hardfork (I thought). Or has 1.35.3 required it before the hardfork?

13

No official information regarding that, nor was rotating KES keys required before or right after upgrading to 1.35.3 - i just rotated the keys because mine are about to expire.

14

You need to collect all logs from that time, I think your block wasn’t fetched.

Logs from relay are at 16:30:51, 6 seconds after block was forged.

You should see around 16:30:45 on both relays and producer, that your block was fetched.

15

not sure what that has to do with anything - as per the pasted snippets you can see that the very same slot and block hash are identical

16

Yes, thank you for pointing that out, I didn’t notice. I’m still learning, like anyone else here(I assume). In any case good luck figuring out what went wrong for you. Cheers!

17

Thank you for trying to be of help! Posted this on IOHK’s github backlog as well, hope there’s some resolution.

18

I was just checking everything on my mainnet upgrade and i get the same! was trying to find out if thats normal before my blocks are due

19

Are your new KES keys + op cert generated using cardano-cli version 1.35.3?

20

This is not important, I am minting blocks on preprod and preview with kes keys and operational certificates generated with 1.34.1 on the cold machine.

1 Like