Whats Happening with KES Keys Rotation

Hey Guys,

Can you confirm for me:

  1. When rotating KES keys, the op.cert and cold.counter are updated, any other files changed?

  2. Is there a penalty for rotating pool keys to early? or too many times? should we wait until expiration before rotating?

  3. I think I read that rotating KES Keys has an impact on your pool being chosen for minting blocks…ie. while not expired, you can delay when your pool is chosen for slots and/or minting blocks if done ‘incorrectly’ or not according to best practices.

I think for many of us, rotating KES keys sounds like a way for the network to make sure all of the nodes are not being abandoned however, it’s more complex then that, right? It will affect the slots that you’re picked for(?) Any input is appreciated…it is definitely better to understand the ‘behind the scenes’ and how NOT to rotate keys when still being within expiration.

  1. a. On a side note: can you confirm that I am able to delete all wallet files from producer node? (obviously keeping these files on offline machine)
    b. and just for additional confirmation…With CNTools, the only files I NEED on producer to mint blocks are the vrf.skey, op.cert, kes.start, right? (I don’t need the hot.skey, hot.vkey to mint blocks?)

ugh, I’m a new SPO with 500k stake (3rd epoch) and still waiting for first block…I’m getting nervous that something is wrong but all signs point to ‘GOOD’ so, Im TRYING to be patient. lol

Hello RRusso84

I rotated the KEYs 2 days ago; look what can I confirm to you:

  1. These are the files which will be updated:
    Jan 8 21:10 hot.vkey
    Jan 8 21:10 kes.start
    Jan 8 21:10 op.cert

  2. You can rotate the KES Keys earlier, I rotated with 8 days before to expire; you should not wait till last day because who knows what can happens and you will not have enough time to fix it

  3. I prefer if I have a slot assigned to not performed any actions; After that I can do what I have to do (upgrade, etc); So yes, if your KEYS are valid better to not change them if you have a slot assigned

  4. You can delelte all files you have also it is recomended to do that but you need to keep on BP :
    hot.skey
    op.cert
    vrf.skey

You can go in cnode.sh script and you will see at the end what files it use when it’s start the node

cardano-node run
–topology “{TOPOLOGY}" \ --config "{CONFIG}”
–database-path “{DB_DIR}" \ --socket-path "{SOCKET}”
–host-addr 0.0.0.0
–shelley-kes-key “{POOL_DIR}/{POOL_HOTKEY_SK_FILENAME}”
–shelley-vrf-key “{POOL_DIR}/{POOL_VRF_SK_FILENAME}”
–shelley-operational-certificate “{POOL_DIR}/{POOL_OPCERT_FILENAME}”
–port ${CNODE_PORT}

Be patient, sooner you will make blocks, d parameter is changing in SPO’s favor each epoch.

Cheers

Ps: the BP/node must be restarted in order to start with new keys

2: Not that I know of, but you only need to rotate them before they expire or if you think they have been compromised.
3: Not that I know of
4a: Wallet files are not need once you have signed the transactions. However if you are using cntools (a good choice IMO) you may prefer to simply encrypt them with a very strong random password (32 chars recommended so you have similar entropy in password to encryption strength)
4b: I would suggest to keep them there, the pool is designed to run with hot keys online, and the cold keys can be offline (hence the names)

On the last comment I would suggest setting up cncli (given you are using cntools) according to the GuildOps docs and that way you will actually know when your node is going to produce blocks down to the specific slot. It takes all the guesswork out and is highly recommended as it also allows you to plan maintenance windows.

HTH.

I see I was beaten to the response, so simply read what’s above :slight_smile:

1 Like

Alex,

As always, thank you!

How do you know if you have a slot assigned? I’m not seeing it in gLiveView.sh

I was able to see it in rtView but have since switched to Grafana (I using the standard IOHK-Cardano dashboard right now and don’t see slot assignment).

Thanks!

You are able to see via cncli script
If u run ./cncli leaderlog or something like that… I don’t remeber the name well but if u type ./cncli.sh u will see the options

Thanks! Your input was great.

I had cncli set up but ran into some issues then I disabled everything except pooltool sendtip. I figured right now, I’d take a deep breath and stop messing around with things…

correction: I’ll stop messing around after I do alex’s recommended idea for cncli-leaderlog…looks like I’m diving back in. lol

Hei, if u are running only the cncli leaderlog I am sure u will be ok, u need to run once at the beggining of the epoch to see if u have slots assigned… after that u can use only pooltool sendtip

And heads up… I’m sure the first slot is arround the corner

CNCLI just crashed my node and now it won’t start-up up.

I’ve disabled cncli from running at systemd restart but the node still wont start…any ideas?

Its working now…

your node was crashed, as you tried to create the ledger.json file.
you can create this file from another system which is not your BP or relays.
prepare a node with cnode at home and prepare the file,
cardano-cli query ledger-state --mainnet --allegra-era --out-file ledger.json

then you can move file to your BP,
rsync -avz -e “ssh -p 222” /home/user/file/ledger.json BP_user@192.168.1.1:/opt/cardano/cnode/scripts

then connect on BP, and run the scripts
python3 getSigma.py --pool-id 437240728470741THISisYOURpoolID3489759279279 --ledger /opt/cardano/cnode/scripts/ledger.json

you receive a Sigma
building active stake
Sigma: 2.338473482938294074e-06

and then you check the leaderlog
python3 leaderLogs.py --vrf-skey /opt/cardano/cnode/priv/pool/tts17/vrf.skey --sigma 2.338473482938294074e-06

please adapt all the necessary paremeters based on your outputs

Hmmm… why crashed?
Have you ran cncli migrate path…?
But did u tried to run in tmux session?

Awesome, Thanks!!

For anyone else having this issue in the future, you can find additional info under “18.12 Slot Leader Schedule” here:

looks like a super easy way to get slot leader schedule.

1 Like

No, that’s on me. I need to start using tmux more often.

Hi RRusso84: For the actual view of being a slotleader: In Grafana you can add a ‘stat’ with the following variabels: (cardano_node_metrics_Forge_forge_about_to_lead_int - cardano_node_metrics_Forge_node_not_leader_int)
The subtraction gives you the result if you are a slot leader or not.

I run this leaderlogs as well, by starting a local node.
You don’t have to move files to your BP node, you can run all the commands locally. I put it all together in a script.

hi,
do you mean from grafana, without any script or ledger file, can we have this info with this metric?

if yes,
means… 36 hours before the new EPOCH, we can see this info here,. right?

I think it is not possible, otherwise why they built the script?

1 Like

Ok, Haha! No, sorry: For Grafana it is only (nearly) realtime info. Just to check on the specific time if you become the slotleader. For prediction you need to run indeed the leaderlogs.

2 Likes

exactly, that was my question / point… i have realized it is realtime…

1 Like

i use also the metric below

cardano_node_metrics_Forge_forged_int / cardano_node_metrics_Forge_node_is_leader_int * 100

1 Like