1.35.3 stuck in epoch 364

[manonthemat]

I moved our stake pool LOXE to 1.35.3 (git rev 80b5637a5520648d50b0763d7677ddbe374cd598) on August 20th and it was producing blocks regularly. I noticed yesterday that our block producer was up as a slot leader, but the block was not adopted by the chain. Only later did I realize that our pool was stuck in Alonzo at the last block of epoch 364 (7791698).

I stopped the nodes and deleted some recent files from the db, fo force replaying of blocks on restart with no difference.

I complied cardano-node and cardano-cli - this time from commit ea6d78c7 - transferred the binaries and configs to my nodes and compiled the libsodium on the nodes as well. I changed the topology on the relays to just connect to the IOHK servers for the moment to try and just sync up, but I’m still getting nowhere with it.

Attached is a view from my journalctl from the relay in question.

[Alexd1985]

Can u share glive output?

[Alexd1985]

Also can u check which version u installed?
I think u installed a wrong version (1.35.3 config)

You’ll need to install

Release Cardano Node 1.35.3 · input-output-hk/cardano-node

[manonthemat]

It’d show “starting” as the tip is on block 7791698 and no connections stay open to other nodes.

As per the wrong version, I installed the release version(s). Cardano PoolTool - The most comprehensive staking statistics for Cardano on the web. shows also that 9% of nodes are running with the same commit I had running since August 20th. 8% are running on the git commit I checked out recently and have run. You can check this release branch for the commit history: Commits · input-output-hk/cardano-node · GitHub

All of those commits should work though.

[Alexd1985]

Understand but on adapools.org → blocks it’s saying that ur last block was made with version6

try again to install 1.35.3 … first time it will take more time to start… check Mem RSS on glive, it should slowly increase

[manonthemat]

I don’t know what version 6 refers to. My last accepted block was in epoch 364 with version 1.35.3 (git commit 80b5637a5520648d50b0763d7677ddbe374cd598).

As I mentioned, I have installed 1.35.3 again - this time from source.

I am taking the remove (part of) the freaking blockchain approach right now and hope for ChainDB to be done before the epoch ends and catch up with missing blocks.
Not sure if that’ll help though.

[Alexd1985]

do not remove the db
Can u share glive?

If u deleted u can download the db from here

Download the latest Cardano blockchain snapshot

[Alexd1985]

@manonthemat

I don’t know what version 6 refers to

should be 7

[manonthemat]

The recent snapshot from cnapshots.io results in:

cardano-node: decodeTelescope: invalid telescope length
CallStack (from HasCallStack):
error, called at src/Ouroboros/Consensus/HardFork/Combinator/Serialisation/Common.hs:443:27 in ouroboros-consensus-0.1.0.0-6Kpqbym2b1s1arV4kigRIW:Ouroboros.Consensus.HardFork.Combinator.Serialisation.Common[

[Alexd1985]

this should not happen…

[HeptaSean]

@manonthemat

As I mentioned, I have installed 1.35.3 again - this time from source.

Are you sure that the 1.35.3 you install is also the one run, e.g., by your systemd units?

This all looks like the version running is still 1.34.1 or something.

[lauris]

All the indications tell me, that you are running old veraion of cardano node

How are you starting the cardano node service? Are you running uaing systemd?

You can try to type commmand to check the path to cardano node bin:

which cardano-node

I have seen that people update the cardano-node bin in ~/.local/bin but scripts/systemd uses ~/.cargo/bin/

Also, restart the node and check the ligfile

[lauris]

Looka like gou have older Carano node version which ia not compatible with the latest (1.35.1) snapshot

[HeptaSean]

@lauris

the latest (1.35.1) snapshot

1.35.1 is not the latest. All nodes should be 1.35.3 now. (… except for the legacy testnet, which should only be used by enthusiasts with 1.35.2 now.)

[manonthemat]

Yes, cardano-node and cardano-cli are both version 1.35.3 as also verified with the --version argument. My systemd script use the same (and only) cardano-node binary that’s installed on the machine.

[Alexd1985]

Do u have this revision number?
950c4e222086fed5ca53564e642434ce9307b0b9

[manonthemat]

@manonthemat

ea6d78c7

No, as mentioned above, I’m on ea6d78c7.

This shouldn’t matter as evident in the output for git diff 950c4e222086fed5ca53564e642434ce9307b0b9 ea6d78c7

diff --git a/docker-compose.yml b/docker-compose.yml
index c85bc17c8..f6d6f9c70 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -2,7 +2,7 @@ version: "3.5"
 
 services:
   cardano-node:
-    image: inputoutput/cardano-node:${CARDANO_NODE_VERSION:-1.35.3}
+    image: inputoutput/cardano-node:${CARDANO_NODE_VERSION:-1.35.3-configs}
     environment:
       - NETWORK=${NETWORK:-mainnet}
     volumes:
@@ -15,7 +15,7 @@ services:
         max-file: "10"
 
   cardano-submit-api:
-    image: inputoutput/cardano-submit-api:${CARDANO_SUBMIT_API_VERSION:-1.35.3}
+    image: inputoutput/cardano-submit-api:${CARDANO_SUBMIT_API_VERSION:-1.35.3-configs}
     environment:
       - NETWORK=${NETWORK:-mainnet}
     depends_on:
diff --git a/flake.lock b/flake.lock
index 4855ece8f..751268fcc 100644
--- a/flake.lock
+++ b/flake.lock
@@ -4521,11 +4521,11 @@
         ]
       },
       "locked": {
-        "lastModified": 1653579289,
-        "narHash": "sha256-wveDdPsgB/3nAGAdFaxrcgLEpdi0aJ5kEVNtI+YqVfo=",
+        "lastModified": 1661431587,
+        "narHash": "sha256-KsWH17SMi+In5J9xkGfwxa8971GAsNi9Cp7zNpEtZdM=",
         "owner": "input-output-hk",
         "repo": "iohk-nix",
-        "rev": "edb2d2df2ebe42bbdf03a0711115cf6213c9d366",
+        "rev": "b8376719bf495694f538acf3e7eed5086cde303c",
         "type": "github"
       },
       "original": {

[manonthemat]

Since IO’s shipped binaries (80b5637a5520648d50b0763d7677ddbe374cd598) ceased to work, I’m compiling the binaries on my kvm host machine and pull it via scp into my kvm clients, so there’s no rust build machinery there, thus a .cargo directory would be absent. Of course, I had to install some build tools on the kvm clients after all for the shared libraries (libsodium namely).

I downloaded the snapshot, but after running into the telescope length error, I’ve removed files that fall within the last few days (epoch 365). My nodes are still replaying the blocks.

[manonthemat]

actually, I think you’re right and I messed up… will stop my relay, pull down another snapshot and test my hypothesis… will report back shortly.

[Alexd1985]

Ok, I thought 1.35.3 config is not compatible with mainnet

Release 1.35.3-configs

Latest

This release provides configuration file updates for the new Cardano test environements (Preview, Pre-Production)