Third block minted? But rewards not showing? Did I lose slot battle?

[Gentleman_Goat]

So something weird happened with the staking pool I am running.

I updated my KES keys 4 days before it was about to expire and in that same epoch on the 20th was supposed to be minted a block.

 SlotNo                          UTC Time              

---

 114192698                   2024-01-20 13:56:29 UTC

This was when it was supposed to have minted a block according to the leaderboard and my node even says it adopted a block interestingly also.

When you look at the previous epoch though and even during it the minted block did not seem to register?

TTSP1 Trading Tools Software Pool - Cardanoscan

It minted 2 blocks the previous epoch so I’m pretty sure I have this setup correctly. Did changing/updating my KES keys in the same epoch (I did it the 17th of January), did that mess it up? Did I mess up the KES keys somehow? It looks like I did it right in my screenshot with it showing a 4-16-2024 as the next expire date

Did I miss something stupid to cause this to happen? Did I lose a slot battle? If so, how do I check this?

[Alexd1985]

The value of the qKesNodeStateOperationalCertificateNumber key indicates the current counter value for your stake pool registered by the blockchain protocol. The value of the qKesOnDiskOperationalCertificateNumber key indicates the counter value of the current operational certificate that your stake pool uses.

For a new operational certificate, the counter value must be exactly one (1) greater than the current value of the qKesNodeStateOperationalCertificateNumber key.

This is what u need to understand… in ur case qKesNodeStateOperationalCertificateNumber is 0 acording to adapools.org and now qKesOnDiskOperationalCertificateNumber this is also 0 but now u must to rotate the KES and the next number is 0 +1 = 1

[Gentleman_Goat]

✓ Operational certificate’s KES period is within the correct KES period interval
✓ The operational certificate counter ahead of the node protocol state counter by 1

{
    "qKesCurrentKesPeriod": 882,
    "qKesEndKesInterval": 939,
    "qKesKesKeyExpiry": "2024-04-16T09:44:51Z",
    "qKesMaxKESEvolutions": 62,
    "qKesNodeStateOperationalCertificateNumber": 0,
    "qKesOnDiskOperationalCertificateNumber": 1,
    "qKesRemainingSlotsInKesPeriod": 7381875,
    "qKesSlotsPerKesPeriod": 129600,
    "qKesStartKesInterval": 877
}

[Alexd1985]

@Gentleman_Goat

qKesNodeStateOperationalCertificateNumber": 0,

For the last 2 blocks minted used op 0, so Inthink now needs to be 1 (+1)

Cheers,

[Gentleman_Goat]

How can I verify that? Also the message states that the KES is okay? Is it just lying then? I’m just super confused. I believe I did it correctly? Thanks for your help in advance by the way, I’m just super worried I’ll miss another block and have no idea why and every tool I have appears to be saying ’ good job you are doing it correctly >.< ’

My main question is why the third block it tried to mint on the 20th failed to mint. Is there a site I can use to see if I lost a slot battle maybe? (or some console command to test this?).

Are these two values supposed to be the same?

"qKesNodeStateOperationalCertificateNumber": ?,
"qKesOnDiskOperationalCertificateNumber": ? ,

In my notes I noticed they were, and now there is a divergence,

If they do need to be the same how do I fix this? Doesn’t the .counter file I have now corrupted? I wonder why it gave me a green checkmark when it said

✓ The operational certificate counter ahead of the node protocol state counter by 1

haha, that is just very misleading when I created it. Shouldn’t it show some kind of red error?

In anycase I’m not sure how to get them in sync at this point now, so any help is appreciated!

According to my notes when I updated my KES kes I ran these commands

cardano-cli-mainnet node key-gen-KES \
  --verification-key-file mainnet_kes.vkey \
  --signing-key-file mainnet_kes.skey

This created the new kes verify and signing keys, then the certificate command below

  cardano-cli-mainnet node issue-op-cert --kes-verification-key-file mainnet_kes.vkey \
  --cold-signing-key-file mainnet_cold.skey \
  --operational-certificate-issue-counter-file mainnet_opcert.counter \
  --kes-period <current kes period> \
  --out-file mainnet_opcert.cert

I put the value of qKesCurrentKesPeriod at the time in there, which I believe is still 882.

Was my counter file somehow changed/corrupted when it should not have been? If so I’m not sure how to get that in sync? (assuming that is even the issue? on this post on the forum it says +1 is okay Correct value for node.counter, but maybe that was a special situation for a hardfork, in anycase just very confused )

When is it okay to be +1 ahead and when it should it be the same?

[Gentleman_Goat]

Okay, so after further research and reading here (Adjust Node.Counter for KES - How to Guides for Coincashew Method Cardano SPOs), it turns out they do have to be the same. When using the command to issue a new node certificate it will automatically increment the value which may or may not be what is needed based on the current. After running the correct command to change the opcert.counter it now shows as follows

✓ Operational certificate's KES period is within the correct KES period interval
✓ The operational certificate counter agrees with the node protocol state counter
{
    "qKesCurrentKesPeriod": 882,
    "qKesEndKesInterval": 944,
    "qKesKesKeyExpiry": "2024-04-23T21:44:51Z",
    "qKesMaxKESEvolutions": 62,
    "qKesNodeStateOperationalCertificateNumber": 0,
    "qKesOnDiskOperationalCertificateNumber": 0,
    "qKesRemainingSlotsInKesPeriod": 7990750,
    "qKesSlotsPerKesPeriod": 129600,
    "qKesStartKesInterval": 882
}

So I’m hoping now it will work?

This might help someone else

cardano-cli node new-counter \
--cold-verification-key-file mainnet_cold.vkey \
--counter-value 0 \
--operational-certificate-issue-counter-file mainnet_opcert.counter

When you cat the .counter file it should be whatever the value of qKesNodeStateOperationalCertificateNumber +1 is after you have issued the node certificate. The first time you run new-counter it will be blank when you cat the file and after running the node certification creation command it will fill it in. You just have to make sure it says that the next certificate number will be 1

[Alexd1985]

hmm, u used 0 last blocks… (see adapools.org) next one should be 1

[Gentleman_Goat]

When I open the counter file it says 1 inside the file, I think it just displays as 0 for some reason, plus now it says it agrees with it from that node. Try using cat on your counter file and let me know it will be +1 what your [quote=“Gentleman_Goat, post:7, topic:126775”]
qKesNodeStateOperationalCertificateNumber
[/quote]

is

[Alexd1985]

if now the counter shows 1 use it again to renew the certs… if u used 0 for the last blocks now it should be 1
after u should see 2 inside the file (next time when u will rotate the kes it will use 2 but this is valid only if u will create a block with 1, if not next time use 1 again)

Cheers

[Gentleman_Goat]

@Alexd1985 I guess I’m just super confused with the whole KES counter thing. Let me know if I understand the rules correctly.

1.) If you have never minted a block the counter should always be 0 when resetting your keys
2.) If you have minted a block, whatever the counter value was used with that last block, the counter for your new KES cert should be +1 that.
3.) qKesNodeStateOperationalCertificateNumber and qKesOnDiskOperationalCertificateNumber should always be the same number?

For example

cardano-cli query kes-period-info --mainnet --op-cert-file ./mainnet_opcert.cert yields the result of…

✓ Operational certificate’s KES period is within the correct KES period interval
✓ The operational certificate counter agrees with the node protocol state counter

{
    "qKesCurrentKesPeriod": 882,
    "qKesEndKesInterval": 944,
    "qKesKesKeyExpiry": "2024-04-23T21:44:51Z",
    "qKesMaxKESEvolutions": 62,
    "qKesNodeStateOperationalCertificateNumber": 0,
    "qKesOnDiskOperationalCertificateNumber": 0,
    "qKesRemainingSlotsInKesPeriod": 7942719,
    "qKesSlotsPerKesPeriod": 129600,
    "qKesStartKesInterval": 882
}

Note that the values are the same for qKesNodeStateOperationalCertificateNumber and qKesOnDiskOperationalCertificateNumber AND it says The operational certificate counter agrees with the node protocol state counter

What confuses me is that according to this article (Renew KES Keys - How to Guides for Coincashew Method Cardano SPOs), near the bottom it seems to contradict what it said at the top about them being the same and suddenly it’s different!

Refer to screenshots below

and the contradiction here?

Suddenely the qKesNodeStateOperationalCertificateNumber and qKesOnDiskOperationalCertificateNumber in their example are different?

Why is this?

I hope this explains why I am so confused.

The only explanation I can possibly have is maybe it changes after you mint a block and the ondisk will show +1 once the first minted block after you get your keys comes in? This was not made clear in the article or any tutorials I read (including from carlos from the cardano foundation unless I missed it?)

I appreciate any confirmation or explanation on this, and I hope I don’t seem like I’m dumb or crazy for asking these questions. Thank you!

I wanted to be even more extra clear and share the article itself (Renew KES Keys - How to Guides for Coincashew Method Cardano SPOs) AND I wanted to show the literal contents of my current .counter file when you ‘cat’ it.

On my air gap machine the counter files, it looks like this, I just … the hex characters for security reasons

{
    "type": "NodeOperationalCertificate",
    "description": "Next certificate issue number: 1",
    "cborHex": "..."
}

Not sure why they are different, maybe it deleted the description on the server after the node used it? The cborHex is identical though.

[Alexd1985]

@Gentleman_Goat

    "qKesNodeStateOperationalCertificateNumber": 0,
    "qKesOnDiskOperationalCertificateNumber": 0,

This is what u need to understand… in ur case qKesNodeStateOperationalCertificateNumber is 0 acording to adapools.org and now qKesOnDiskOperationalCertificateNumber is also 0 but now u must to rotate the KES and the next number is 0 +1 = 1

[Gentleman_Goat]

Which number exactly needs to have +1? which variable? qKesOnDiskOperationalCertificateNumber? Is what I have correct from my counter file correct since it’s says next value is ‘1’?

Just so you know @Alexd1985 I have rotated the keys already, so what you see is what it is AFTER I have rotated, you can tell I rotated because of the expire date 2024-04-23 21:44 UTC, I am just trying to verify I did it correctly this time. Maybe I need to make another thread for this.

[Alexd1985]

U used number 0 for the latest blocks, now set incremental number to 1 inside counter file and rotate again… then check again

[Gentleman_Goat]

I was on the schedule to mint a block, but it failed again :(, I don’t need the newest binary to mint the block do I? I made sure it followed the rules we discussed, so frustrated as to what I’m doing wrong. Not angry at you or anything @Alexd1985 , if anything I appreciate your help I’m just trying to figure out what is going wrong especially since I had minted blocks before okay, and I’m pretty sure I set up the KES keys correctly this time with the numbers matching.

Was supposed to win this slot 115986490 2024-02-10 08:13:01 UTC

but this failed to mint

Is there anyway to find out why I lost that slot?

I am upgrading to 8.7.3 from 8.1.2 now in the meantime to see if that helps and will report back on KES statistics

UDPATE: Successfully got 8.7.3 installed on the block producer and two relays, just waiting for them to catch up on validating the chunks and will give KES data shortly. Luckily they had binaries available for download now on them.

[Alexd1985]

check inside the logs, you didn’t find anything?

also, can u share again the output?

cardano-cli query kes-period-info --mainnet --op-cert-file ./mainnet_opcert.cert

[Gentleman_Goat]

@Alexd1985

cardano-cli query kes-period-info --mainnet --op-cert-file ./mainnet_opcert.cert

✓ Operational certificate’s KES period is within the correct KES period interval
✓ The operational certificate counter agrees with the node protocol state counter
{
“qKesCurrentKesPeriod”: 895,
“qKesEndKesInterval”: 944,
“qKesKesKeyExpiry”: “2024-04-23T21:44:51Z”,
“qKesMaxKESEvolutions”: 62,
“qKesNodeStateOperationalCertificateNumber”: 0,
“qKesOnDiskOperationalCertificateNumber”: 0,
“qKesRemainingSlotsInKesPeriod”: 6283432,
“qKesSlotsPerKesPeriod”: 129600,
“qKesStartKesInterval”: 882
}
Let me know if you need anything else? @Alexd1985

How do I search for specific things in the logs or at all?

Attached is permissions of he keys also, do they look correct? Note date is same for opcert and kes

When running the command showing all matches with that slot number, I didn’t see any errors? Unless clocks fits onto some fork is an issue

journalctl --unit=cardano-node -g "115986490"

I got

Feb 10 08:13:01 vmi555.contaboserver.net cardano-node[3629849]: [b8fa3eb4:cardano.node.LeadershipCheck:Info:4702] [2024-02-10 08:13:01.00 UTC] {"chainDensity":4.739014e-2,"credentials":"Cardano","delegMapSize":1335342,"kind":"TraceStartLeadershipCheck","slot>

Feb 10 08:13:01 vmi555.contaboserver.net cardano-node[3629849]: [b8fa3eb4:cardano.node.ChainDB:Notice:4692] [2024-02-10 08:13:01.15 UTC] Chain extended, new tip: 420e60c347882fb6191f64861321a7268b719b1361778f031c6e5cc89593f622 at slot 115986490

Feb 10 08:13:02 vmi555.contaboserver.net cardano-node[3629849]: [b8fa3eb4:cardano.node.ChainDB:Info:4692] [2024-02-10 08:13:02.62 UTC] Block fits onto some fork: d5054885eb2d5931ec9ea42a4433ca8fcfffcfba2032dcd68bea72f103ddf782 at slot 115986490

Feb 10 08:13:02 vmi555.contaboserver.net cardano-node[3629849]: [b8fa3eb4:cardano.node.ChainDB:Notice:4692] [2024-02-10 08:13:02.69 UTC] Switched to a fork, new tip: d5054885eb2d5931ec9ea42a4433ca8fcfffcfba2032dcd68bea72f103ddf782 at slot 115986490

Let me know if you want to run any other log command to see anything

[Alexd1985]

The certificate is wrong

cat cold.counter

Whats the incremental number now?

Try

journalctl --unit=cardano-node -g "InvalidKesSignatureOCERT"

[Gentleman_Goat]

@Alexd1985

journalctl --unit=cardano-node -g “InvalidKesSignatureOCERT”

@Alexd1985 the journal commands gives

– No entries –

So it seems that is not the error maybe?

When I cat the .counter file from my air gap machine it says
“Next certificate issue number: 1”

also note that the kes command said this

✓ Operational certificate’s KES period is within the correct KES period interval
✓ The operational certificate counter agrees with the node protocol state counter

Does what this say above not matter? Can it be incorrect?

[Alexd1985]

I want to see the number for:

“qKesNodeStateOperationalCertificateNumber”: ,
“qKesOnDiskOperationalCertificateNumber”: ,

[Gentleman_Goat]

@Gentleman_Goat

“qKesNodeStateOperationalCertificateNumber”: 0,
“qKesOnDiskOperationalCertificateNumber”: 0,

“qKesNodeStateOperationalCertificateNumber”: 0,
“qKesOnDiskOperationalCertificateNumber”: 0,