KES keys expired - trouble rotating

For the producer, nothing since 22:02 last night (GMT): Listening on http://127.0.0.1

Perfect, and how is the relay?

Relay
Screenshot 2021-09-15 at 09.30.05

And if u type top on producer… do u see the CPU ~100%

I think it’s re-syncing the DB and could take more time

no, we’re idle at 0-3%
glive still “starting”

Interestingly, if I load CNTOOLS back up it says the KES keys have expired.

Yes, because the producer didn’t started

Do u have teams? Would u like to take a look?

That would be great.
I’m working the day job now till 6. I’ll try to find some time to look at this.

Send me a DM with your email and what time might suit for you and I’'ll book something in. I appreciate the evening after 6pm (GMT) may not be any good for you.

just give me a sign when you will be available and I will send the meeting link

I’m having a very similar issue.

Transfer the BP hdd to a another system with 16gb ram due to the 1.29 update, at first started up fine and was resyncing then it reset stayed on “starting…” ever since. I’ve tried what has been suggested above without any success.

When I run cntools says the KES has expired (from memory still had a couple of months to go) and when trying to rotate them using cntools the new date is for yesterday…

Any help would be greatly appreciated

Hi Alex, I am having also similar issue, 1 node looks ok (but not connected to others), another relay and BP stayed starting (they are connected to each other). I have also all nodes 1.29. (first 10 days after update was everything ok) 16 GB ram.

When I run CNTOOLS says also the KES has expired (I rotate them on 10th or 11th) and when trying to rotate them again using cntools the new date is also for yesterday…

journalctl -e -f -u cnode says
– Reboot –
Sep 15 07:22:44 Guild scripts has now been upgraded to support cardano-node 1.29.0 or higher (1.25.1 found).
Sep 15 07:22:44 Please update cardano-node (note that you should ideally update your config too) or use tagged branches for older node version.
Sep 15 07:22:44 WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Sep 15 07:22:45 Listening on http://127.0.0.1:12798
Sep 15 14:07:26 Shutting down…
Sep 15 14:07:28 Guild scripts has now been upgraded to support cardano-node 1.29.0 or higher (1.25.1 found).

Sep 15 14:07:28 Please update cardano-node (note that you should ideally update your config too) or use tagged branches for older node version.

Sep 15 14:07:28 WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.

Sep 15 14:07:30 Listening on http://127.0.0.1:12798

When I try to send transaction from BP node it says:
Command failed: transaction submit Error: Error while submitting tx: ShelleyTxValidationError ShelleyBasedEraMary (ApplyTxError [DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash “8ca543af31722dd951b20af603ae601608cf719e6b793d75d98bd701”)}

I will appreciate also any help, tried restarted, updated, change the CN port, nothing helps.

did you solve the issue?

Hi, it simple to fix it, understand that u are using cntools but before u used/tried another guide

so go inside the env file and uncomment the lines

  • CCLI
  • CNODE_HOME

save the file and restart the node, check again with journalctl -e -f -u cnode

It should start now

Not yet, Just trying what Alex has suggested now

Hi Alex, thank you for replying. Tried what you have suggested, giving the node some time to start.

Unfortunately through when a run cntools the “expired KES key” warning still pops up and when I do a key rotate the “new date” is still for the 2021-09-15 which is now two days ago

Man, type journalctl -e -f -u cnode

I have, here’s the output…

Sep 16 00:07:09 Stopping Cardano Node…
Sep 16 00:07:09 Shutting down…
Sep 16 00:07:09 cnode.service: Succeeded.
Sep 16 00:07:09 Stopped Cardano Node.
Sep 16 00:07:09 Started Cardano Node.
Sep 16 00:07:09 Guild scripts has now been upgraded to support cardano-node 1.29.0 or higher (1.27.0 found).
Sep 16 00:07:09 Please update cardano-node (note that you should ideally update your config too) or use tagged branches for older node version.
Sep 16 00:07:09 WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Sep 16 00:07:10 Listening on http://0.0.0.0:12798
Sep 16 00:40:30 Stopping Cardano Node…
Sep 16 00:40:30 Shutting down…
Sep 16 00:40:30 cnode.service: Succeeded.
Sep 16 00:40:30 Stopped Cardano Node.
Sep 16 00:40:30 Started Cardano Node.
Sep 16 00:40:30 Guild scripts has now been upgraded to support cardano-node 1.29.0 or higher (1.27.0 found).
Sep 16 00:40:30 Please update cardano-node (note that you should ideally update your config too) or use tagged branches for older node version.
Sep 16 00:40:30 WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Sep 16 00:40:31 : Listening on http://0.0.0.0:12798
Sep 16 22:37:21 Stopping Cardano Node…
Sep 16 22:37:21 Shutting down…
Sep 16 22:37:21 cnode.service: Succeeded.
Sep 16 22:37:21 Stopped Cardano Node.
Sep 16 22:37:21 Started Cardano Node.
Sep 16 22:37:22 WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Sep 16 22:37:23 Listening on http://0.0.0.0:12798

Says the node is running 1.27.0 which is strange as I i’ve updated to 1.29 (following your guide) twice, and it was running fine on the previous computer (apart from running out of memory, hence the new computer).

gLiveView_Screenshot

You are not following the logs, that message dissapeared

started the node again and this was the the output with journalctl -e -f -u cnode running…

Sep 17 04:51:33 Stopping Cardano Node…
Sep 17 04:51:33 Shutting down…
Sep 17 04:51:33 cnode.service: Succeeded.
Sep 17 04:51:33 Stopped Cardano Node.
Sep 17 04:51:33 Started Cardano Node.
Sep 17 04:51:34 WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Sep 17 04:51:35 Listening on http://0.0.0.0:12798

is this the message you are referring to that disappeared?
could you please explain what you are referring to?

This is fine, it means the node started properly… what is the issue?

type top
Do u see the cpu ~100%?
How the glive looks?