Still having Syncing issues post hardfork?

This post will walk you through the issue that I had on the hardfork transition. It might not solve your exact problem but the steps that I will go through might help pinpoint your issue as well. Shout out to xSPO alliance. The community there is helpful and many members helped me and others to resolve issues post hardfork. If your pool has less than 1m ADA staked you really should think about joining xSPO, come by and join our discord channel: discord.gg/VGMnRjeH9d

So the TLDR: all my nodes would sync up to tip 39916796 and then stop both at the hard fork and when I resynced the database. The root cause was that I was running an older build (6510764) of alonzoGenesis and while it will show running 1.29.0 it will stop syncing at the end of Epoch 289. Updating to the correct build (7578887) of the alonzoGenesis and config file (or adding in the correct hash) and letting the system run for 15-20 minutes got my node back up and running and fully synced

pic01

If you want to read more on troubleshooting steps here continue reading. First off my root issue was not fully understanding the code I was using with which I was pulling down the files. Guides are great and I was using the coincashew guide, but it’s always important to understand what it is doing. My mistake was not updating the node build in my bashrc file. This caused me to download older versions that causes all my mess.

So let’s first start with the issue. From the screenshot above my node was syncing and running but couldn’t get past 39916796.

Here is the log entry on where it stops
Sep 13 18:04:58 cardano-node[942]: #033[31m[cardano.node.ChainDB:Error:49]#033[0m [2021-09-13 18:04:58.56 UTC] Invalid block 8959c0323b94cc670afe44222ab8b4e72cfcad3b5ab665f334bbe642dc6e9ef4 at slot 39916975: ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [ShelleyInAlonzoPredFail (LedgersFailure (LedgerFailure (UtxowFailure (WrappedShelleyEraFailure (UtxoFailure (OutputTooBigUTxO [(1870,1000,(Addr Mainnet (KeyHashObj (KeyHash “409153eaf3a2f4266f64e74030ae6cac8b687612c4ffbe3ab4b8be7f”)) (StakeRefBase (KeyHashObj (KeyHash “1c42b5e9ac1d0515ccba0c526091abea80c7f731a2d64ad831c2b5f3”))),Value 101169294…
From there it tries to go back to the previous block
Sep 13 18:04:58 cardano-node[942]: #033[34m[cardano.node.ChainDB:Info:49]#033[0m [2021-09-13 18:04:58.56 UTC] Valid candidate e72579ff89dc9ed325b723a33624b596c08141c7bd573ecfff56a1f7229e4d09 at slot 39916796

Two things that I like to use when troubleshooting Cardano Node: The first is htop monitoring the CPU and RAM
Let’s start off with a server not currently running cardano node. As you can see on this build my Memory is low at 344MB

When you start running the node it will start using more RAM.

This is a great place to start looking when your node is having problems. If it stays around 600MB or 1GB of RAM something is not running correctly. During the syncing it will run up to around 8GB of ram, and when your node is fully synced,your startup RAM usage might show 4-6GB, but that will grow over time.

The second tool is syslog on Ubuntu it is as /var/log/syslog
I used this to really get a good understanding of what was going on with the node. You can view the entire log or use journalctl and filter it down. I like viewing the entire log, but that is just me

So my Node is sitting there stuck on 39916796. My first thing that I tried is changing the topology file to only sync to a node relay that was fully synced. Then I removed the DB folder and fully synced the node. Again it stopped at the same place with the same error in syslog.

Then I was looking at my config file and it hit me: my GenesisHash was off

The correct Hash is: 7e94a15f55d1e82d10f09203fa1d40f8eede58fd8066542cf6566008068ed874

Finally I was pointed in the right direction.
Going to the latest build: http://hydra.iohk.io/job/Cardano/iohk-nix/cardano-deployment/latest-finished/download/1/index.html I downloaded the correct alonzoGensis and updated the config file to the correct hash.

From there you need to stop the cardano node, start it back up. Again using htop I saw the node get back up to the correct memory usage, but still stuck on starting. Looking at the syslog I could tell that the node is resyncing something in the database. It took 20-30 minutes for the database to resync (you could follow the progress in the syslog). Once that was done, the node started syncing past tip 39916796 and has been running healthily since

pic05

2 Likes

Great post! Thanks to the community, i had very similar issue, i’ve notice i get msg in logs and it show the env version of 1.26.2
My node wss at 1.29 at this point, but something is messed up with the config. After i made sure i have the right one everything was solved in seconds
I was hard hard fork for us. Couple hours down. But now we are live again !
Thanks xSPO!

I have been running a geth testnet instance for the last 2 months and learning this. But after the hard-fork i have seen the syncing to be extremely slow.syncing issue appears to have snarled a number of companies and … by client sync issue following Ethereum’s Berlin hard fork.

targetpayandbenefits