SPO Restart from Snapshot Takes a Long Time to Process Ledger

Staking & Delegation Setup a Stake Pool
1

:wave: everyone,

I’m running an SPO node bootstrapped from a fresh snapshot on cnapshot. I have noticed that when I want to restart my block producer node or one of my relayers, it takes a long time (~40 minutes) to process the ledger state.

Here you can find the logs when I restart my node.

Started opening Volatile DB
Opened vol db
Started opening Ledger DB
Replaying ledger from snapshot at 5ddaaa4229d94f293a0035236af0ad20ffa1d6b0ae89ee16a37e74ef33475aa0 at slot
 127696529
Replayed block: slot 127696544 out of 127712203. Progress: 0.10%
Replayed block: slot 127699197 out of 127712203. Progress: 17.02%
Opened lgr db
Started initial chain selection
Pushing ledger state for block 2ecf057e416552be051e148e9e7464a5a9816612ba83a6e16e7784bd53f98f5a at slot 12
7712209. Progress: 0.00%
before next, messages elided = 127712244
Pushing ledger state for block 55da99e2b0435e6bc62116f71ab19259956775e3e2df83a2e5dc78afab092795 at slot 12
7722229. Progress: 20.64%
Pushing ledger state for block 605de6fc8c2edab21281fccfad595b5c39e203cb5005614a303a3867b642ca95 at slot 12
7722263. Progress: 20.71%

[...]
Pushing ledger state for block da498bdf58d64ef70357a36811f4912806d9eba3288eff514b6841888df37d14 at slot 127760751. Progress: 100.00%

When I’m checking the content of the ledger folder I can find a snapshot at the slot 127696529 but its a very old slot (~19h)

I would like to know if anyone else is experiencing the same behavior ?

I have an instance with 4CPU and 32GB of RAM. I’m using the default configuration and I leave the default value of 72 minutes for the Snapshot Interval and I’ve an NVME disk and a size of 200 GB allocated to my instance

Thanks.

2

There has been a big load of transactions recently (over the last 12 hours). So the replay will be working through all of those, which can take a very long time.

3

Thanks for your reply @jeremyisme

Can we do something on our side ? I’ve also seen that an issue has been opened on Github.

Also do you know why at each restart the node start from a old ledger state ? should we not have a new ledger snapshot every 72 minutes by default ?