Cannot Upgrade to v10.1.4 Block Producer

hello, I need some help updating my stake pool. I tried upgrading my core node from version 9.2.0 to the latest version 10.1.4. The relay upgraded fine but when i restarted the core machine it would not start with a error:

“The KES key provided does not match the KES key specified”

when I try and run cardano-cli commands to verify the KES, i am getting errors:

/ Network.Socket.connect: <socket: 11>: does not exist (Connection refused).

I figured that this was happening because the nodes could not start so I reverted the virtual machine to the snapshot I took before upgrading. The cardano node is back on but I am worried about upgrading and I still cant run cardano-cli command, I get the same error. I notice there are two sockets in the /opt/cardano/cnode/sockets/ folder:

node0.socket node.socket.

I suspect that this might be causing the problem. I tried changing the config export: export

CARDANO_NODE_SOCKET_PATH=/opt/cardano/cnode/sockets/node.socket

I can now run cardano-node commands:

cardano-cli query tip --mainnet
{
“block”: 11345784,
“epoch”: 533,
“era”: “Conway”,
“hash”: “3bd7be71cd141a66d444616a02ab5ec37d142215d5d07bae4e9afd1fdfd5212d”,
“slot”: 145192688,
“slotInEpoch”: 299888,
“slotsToEpochEnd”: 132112,
“syncProgress”: “100.00”
}

Here is the output of the KES Query:

cardano-cli query kes-period-info \
–op-cert-file /opt/cardano/cnode/priv/pool/stake_pool/op.cert
–mainnet
✓ Operational certificate’s KES period is within the correct KES period interval
✓ The operational certificate counter ahead of the node protocol state counter by 1
{
“qKesCurrentKesPeriod”: 1120,
“qKesEndKesInterval”: 1181,
“qKesKesKeyExpiry”: “2025-04-14T09:44:51Z”,
“qKesMaxKESEvolutions”: 62,
“qKesNodeStateOperationalCertificateNumber”: 14,
“qKesOnDiskOperationalCertificateNumber”: 15,
“qKesRemainingSlotsInKesPeriod”: 7864775,
“qKesSlotsPerKesPeriod”: 129600,
“qKesStartKesInterval”: 1119
}

I then tried to restart the node and now it won’t start with this error:

CardanoProtocolInstantiationError (CardanoProtocolInstantiationPraosLeaderCredentialsError (MismatchedKesKey “/opt/cardano/cnode/priv/pool/stake_pool/hot.skey” “/opt/cardano/cnode/priv/pool/stake_pool/op.cert”))
Jan 13 09:08:13 cardano-core cnode[4029990]: cardano-node: The KES key provided at: “/opt/cardano/cnode/priv/pool/stake_pool/hot.skey” does not match the KES key specified in the operational certificate at: “/opt/cardano/cnode/priv/pool/stake_pool/op.cert”
Jan 13 09:08:13 cardano-core systemd[1]: cnode.service: Main process exited, code=exited, status=1/FAILURE
Jan 13 09:08:13 cardano-core systemd[1]: cnode.service: Failed with result ‘exit-code’.

This seems to be the result of: CARDANO_NODE_SOCKET_PATH=/opt/cardano/cnode/sockets/node.socket

1 Like

Do you have two processes running? There shouldn’t be two sockets.

Hi Jeremy,

No, I don’t have two processes running:

core:~$ ps aux | grep cardano-node
core 536936 15.7 70.7 1073813004 14171928 ? Ssl Jan12 232:18 /home/core/.local/bin/cardano-node +RTS -N4 -RTS run --topology /opt/cardano/cnode/files/topology.json --config /opt/cardano/cnode/files/config.json --database-path /opt/cardano/cnode/db --socket-path /opt/cardano/cnode/sockets/node.socket --shelley-kes-key /opt/cardano/cnode/priv/pool/stake_pool/hot.skey --shelley-vrf-key /opt/cardano/cnode/priv/pool/stake_pool/vrf.skey --shelley-operational-certificate /opt/cardano/cnode/priv/pool/stake_pool/op.cert --port 3001 --host-addr 0.0.0.0
core 4000183 0.0 0.0 6612 2224 pts/0 S+ 21:51 0:00 grep --color=auto cardano-node

To get this issue fixed had to revert my virtual machine to the snapshot, remove the node0.socket file, then rotate my KES again. I think part of the problem was that in my .bashrc file I had the path pointing to the wrong socket:

export CARDANO_NODE_SOCKET_PATH=/opt/cardano/cnode/sockets/node0.socket

instead of:

export CARDANO_NODE_SOCKET_PATH=/opt/cardano/cnode/sockets/node.socket

I think at one stage it was node0.socket but maybe after a restart it switched to node.socket. What is the best practice here? As, I imagine, that the socket may change after restarts, is it best not to put this variable in the bashrc file?

When you get to do the ugprade again, you will need to let it replay the whole ledger. It can take many many hours. So make sure you wait for that to be completed before you try do any cli actions, otherwise the socket file won’t be there, as the node won’t be synced.

Yep, doing that right now. It’s about 40% done after a couple of hours

2 Likes

Do you mind taking a quick look at my gLiveView and seeing if everything looks ok to you?

Looks synced which is good, but if this is your BP, you shouldn’t have that many outgoing. Should have only connections to your relays. So check you p2p topology and make sure it is right. P2P Topology File Explained | How to Guides for Coincashew Method Cardano SPOs

Thank Jeremy,

I have update the topology file as per the example with two relays. Here are the gViewLIve for BP and Relay:

1 Like

Double check the connected connections on the BP, it is showing 2 out and 1 in. It may be 2 outgoing to your relay (assuming you have 1). Or if you have two, then you’re only getting 1 incoming.

1 Like

A long time ago another kind person from the forum helped me with a few problems on the stake pool and he gave me his relay ip address to connect to as well. Perhaps it was for testing purposes but I kept it ever since. Not sure if there’s any benefit in keeping it?

I would only connect relay to relay. Your BP should only ever connect to your own relays. Nothing else.

1 Like

Got it, I changed the topology to only my relay. One in and One out now. Cheers.

1 Like