Invalid snapshot DiskSnapshot on 1.35.0 mainnet

Staking & Delegation Operate a Stake Pool
1

Hi all,
giving 1.35.0 a try on mainnet on a development machine. getting “Invalid snapshot DiskSnapshot” then cardano-node hangs. ideas? was syncing and running fine for about 12hours, then this

Jun 28 09:09:26 dev1 cardano-node[776112]: [dev1:cardano.node.ChainDB:Info:15] [2022-06-27 21:09:26.86 UTC] Started opening Chain DB
Jun 28 09:09:26 dev1 cardano-node[776112]: [dev1:cardano.node.ChainDB:Info:15] [2022-06-27 21:09:26.86 UTC] Started opening Immutable DB
Jun 28 09:09:27 dev1 cardano-node[776112]: [dev1:cardano.node.ChainDB:Info:15] [2022-06-27 21:09:27.00 UTC] Validating chunk no. 2997 out of 2997. Progress: 99.97%
Jun 28 09:09:28 dev1 cardano-node[776112]: [dev1:cardano.node.ChainDB:Info:15] [2022-06-27 21:09:28.10 UTC] Validated chunk no. 2997 out of 2997. Progress: 100.00%
Jun 28 09:09:28 dev1 cardano-node[776112]: [dev1:cardano.node.ChainDB:Info:15] [2022-06-27 21:09:28.11 UTC] Found a valid last location at chunk 2997 with tip 6dde18909f3fa2b66b3ffb74e64ff587a2e6b651be807edeb5835e4891331950@64748421.
Jun 28 09:09:28 dev1 cardano-node[776112]: [dev1:cardano.node.ChainDB:Info:15] [2022-06-27 21:09:28.11 UTC] Opened imm db with immutable tip at 6dde18909f3fa2b66b3ffb74e64ff587a2e6b651be807edeb5835e4891331950 at slot 64748421 and chunk 2997
Jun 28 09:09:28 dev1 cardano-node[776112]: [dev1:cardano.node.ChainDB:Info:15] [2022-06-27 21:09:28.11 UTC] Started opening Volatile DB
Jun 28 09:09:35 dev1 cardano-node[776112]: [dev1:cardano.node.ChainDB:Info:15] [2022-06-27 21:09:35.17 UTC] Opened vol db
Jun 28 09:09:35 dev1 cardano-node[776112]: [dev1:cardano.node.ChainDB:Info:15] [2022-06-27 21:09:35.17 UTC] Started opening Ledger DB
Jun 28 09:10:23 dev1 cardano-node[776112]: [dev1:cardano.node.ChainDB:Error:15] [2022-06-27 21:10:23.37 UTC] Invalid snapshot DiskSnapshot {dsNumber = 64748421, dsSuffix = Nothing}InitFailureRead (ReadFailed (DeserialiseFailure 397070315 "end of input"))
2

Have you got enough disk space?

If all else fails, delete the db folder and resync. But that will take some time…

3

Yeah heaps of space.
was trying to avoid a re-sync… but looks like this might have to be the way

4

This seems to have a similar error, and time solved it UPDATE CARDANO NODE to 1.29.0 - FOR CNTOOLS users - #86 by Juan_Echeverry

Maybe just give your node some more time. A watched pot never boils!

5

Seems like the node just hangs though, as in it doesnt look like its talking to anything and just stops.

ive started a re-sync and will report back. Im wondering if its an issue with the replay of the ledger after 1.35.0 was run for the first time on an existing db

6

The ledger spec changed, making the old snapshots incompatible with 1.35.0 - it will need a resync. This is expected behavior.

1 Like
7

So re-sync’d the entire db, still getting this error upon node startup:

Jul 01 11:55:56 dev1 cardano-node[2267462]: [dev1:cardano.node.ChainDB:Error:15] [2022-06-30 23:55:56.94 UTC] Invalid snapshot DiskSnapshot {dsNumber = 64620415, dsSuffix = Nothing}InitFailureRead (ReadFailed (DeserialiseFailure 182145009 "end of input"))
8

After that, does it show any info regarding it rebuilding the ledger snapshot?

9

no, it just stops at “end of input” and does not do anything else. I cant see it attempting to restart, rebuild, or re open the ledger.


Jul 02 10:10:58 dev1 cardano-node[2299704]: node-to-node versions:
Jul 02 10:10:58 dev1 cardano-node[2299704]: NodeToNodeV_7        HardForkNodeToNodeEnabled HardForkSpecificNodeToNodeVersion1 (EraNodeToNodeEnabled ByronNodeToNodeVersion2 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* EraNodeToNodeDisabled :* Nil)
Jul 02 10:10:58 dev1 cardano-node[2299704]: NodeToNodeV_8        HardForkNodeToNodeEnabled HardForkSpecificNodeToNodeVersion1 (EraNodeToNodeEnabled ByronNodeToNodeVersion2 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* EraNodeToNodeDisabled :* Nil)
Jul 02 10:10:58 dev1 cardano-node[2299704]: NodeToNodeV_9        HardForkNodeToNodeEnabled HardForkSpecificNodeToNodeVersion1 (EraNodeToNodeEnabled ByronNodeToNodeVersion2 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* EraNodeToNodeEnabled ShelleyNodeToNodeVersion1 :* Nil)
Jul 02 10:10:58 dev1 cardano-node[2299704]: node-to-client versions:
Jul 02 10:10:58 dev1 cardano-node[2299704]: NodeToClientV_9        HardForkNodeToClientEnabled HardForkSpecificNodeToClientVersion2 (EraNodeToClientEnabled ByronNodeToClientVersion1 :* EraNodeToClientEnabled ShelleyNodeToClientVersion4 :* EraNodeToClientEnabled ShelleyNodeToClientVersion4 :* EraNodeToClientEnabled ShelleyNodeToClientVersion4 :* EraNodeToClientEnabled ShelleyNodeToClientVersion4 :* EraNodeToClientDisabled :* Nil)
Jul 02 10:10:58 dev1 cardano-node[2299704]: NodeToClientV_10        HardForkNodeToClientEnabled HardForkSpecificNodeToClientVersion2 (EraNodeToClientEnabled ByronNodeToClientVersion1 :* EraNodeToClientEnabled ShelleyNodeToClientVersion4 :* EraNodeToClientEnabled ShelleyNodeToClientVersion4 :* EraNodeToClientEnabled ShelleyNodeToClientVersion4 :* EraNodeToClientEnabled ShelleyNodeToClientVersion4 :* EraNodeToClientDisabled :* Nil)
Jul 02 10:10:58 dev1 cardano-node[2299704]: NodeToClientV_11        HardForkNodeToClientEnabled HardForkSpecificNodeToClientVersion2 (EraNodeToClientEnabled ByronNodeToClientVersion1 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* EraNodeToClientDisabled :* Nil)
Jul 02 10:10:58 dev1 cardano-node[2299704]: NodeToClientV_12        HardForkNodeToClientEnabled HardForkSpecificNodeToClientVersion2 (EraNodeToClientEnabled ByronNodeToClientVersion1 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* EraNodeToClientDisabled :* Nil)
Jul 02 10:10:58 dev1 cardano-node[2299704]: NodeToClientV_13        HardForkNodeToClientEnabled HardForkSpecificNodeToClientVersion2 (EraNodeToClientEnabled ByronNodeToClientVersion1 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* EraNodeToClientEnabled ShelleyNodeToClientVersion5 :* Nil)
Jul 02 10:10:58 dev1 cardano-node[2299704]: [dev1:cardano.node.ChainDB:Info:15] [2022-07-01 22:10:58.22 UTC] Started opening Chain DB
Jul 02 10:10:58 dev1 cardano-node[2299704]: [dev1:cardano.node.ChainDB:Info:15] [2022-07-01 22:10:58.22 UTC] Started opening Immutable DB
Jul 02 10:10:58 dev1 cardano-node[2299704]: [dev1:cardano.node.ChainDB:Info:15] [2022-07-01 22:10:58.51 UTC] Validating chunk no. 2967 out of 2967. Progress: 99.97%
Jul 02 10:11:00 dev1 cardano-node[2299704]: [dev1:cardano.node.ChainDB:Info:15] [2022-07-01 22:11:00.90 UTC] Validated chunk no. 2967 out of 2967. Progress: 100.00%
Jul 02 10:11:00 dev1 cardano-node[2299704]: [dev1:cardano.node.ChainDB:Info:15] [2022-07-01 22:11:00.91 UTC] Found a valid last location at chunk 2967 with tip ba452331eca19a73982ffb38e008fd4d1a80121a1ea9ce35e4387f61d6a1912b@64106396.
Jul 02 10:11:00 dev1 cardano-node[2299704]: [dev1:cardano.node.ChainDB:Info:15] [2022-07-01 22:11:00.91 UTC] Opened imm db with immutable tip at ba452331eca19a73982ffb38e008fd4d1a80121a1ea9ce35e4387f61d6a1912b at slot 64106396 and chunk 2967
Jul 02 10:11:00 dev1 cardano-node[2299704]: [dev1:cardano.node.ChainDB:Info:15] [2022-07-01 22:11:00.91 UTC] Started opening Volatile DB
Jul 02 10:12:02 dev1 cardano-node[2299704]: [dev1:cardano.node.ChainDB:Info:15] [2022-07-01 22:12:02.13 UTC] Opened vol db
Jul 02 10:12:02 dev1 cardano-node[2299704]: [dev1:cardano.node.ChainDB:Info:15] [2022-07-01 22:12:02.13 UTC] Started opening Ledger DB
Jul 02 10:12:02 dev1 cardano-node[2299704]: [dev1:cardano.node.ChainDB:Error:15] [2022-07-01 22:12:02.13 UTC] Invalid snapshot DiskSnapshot {dsNumber = 64651740, dsSuffix = Nothing}InitFailureRead (ReadFailed (DeserialiseFailure 0 "end of input"))
10

I can share a snapshot of mine on 1.35.0 and see if it starts with that if you want.

11

That would be great!

12

Here you go:

https://x230.weebl.me/public/65070743

Make sure you copy the latest immutable and volatile databases from an up-to-date running 1.34.x node, otherwise the snapshot will probably be too new!

13

Had the same issue. The solution is wait. It will begin finishing the synch just like @jeremyisme says. I’m running an EVO 970, decent cpu with a pretty slow internet speed (50mbps on a good day) (I’m putting specs here to see if you’re getting some hardware effect increasing wait time
and it took maybe 3 minutes to get passed the error and continue synching.

1 Like
14

Yes, got the same issue upon restarting node using version 1.35.7. After about 5 minute wait, the startup process continued. So, just waiting did the trick for me.

Here is the snippet from the logs after getting past the error.

Apr 05 09:51:14 xxx cardano-node[865]: [vmi73080:cardano.node.ChainDB:Error:11] [2023-04-05 14:51:14.08 UTC] Invalid snapshot DiskSnapshot {dsNumber = 89094018, dsSuffix = Nothing}InitFailureRead (ReadFailed (DeserialiseFailure 1192968317 "end of input"))
Apr 05 09:56:02 xxx cardano-node[865]: [vmi73080:cardano.node.ChainDB:Info:11] [2023-04-05 14:56:02.10 UTC] Replaying ledger from snapshot at 9ddec40b8cc0997bc42b15599ad2fe70cabafa26f624a8c82c0c75723381fe1a at slot 89089455