Updating Cardano with Minimal Down Time

Hello community,

I am upgrading to 1.29 and have built and synced the update on by build machine which took many hours. Instead of going through the same process on the core and relay machine would it be possible to just copy relevant files from the build machine to the other machines? I would assume I could simply copy all of the updated executables (cardano-node, cardano-cli, etc) which would save build time but in order to save down time which other files would I need to copy? I am using CNTOOTLS and would perhaps assume the db folder there? Any other files?

Binari files should be located to
cd ${HOME}/.cabal/bin/

Hi Alex,

Yes, I have those but wondering whether I could bypass the whole sync process as it too overnight for gLiveView to move from starting to started.

Why so long? It should start normally

I waited for about 3 hours and then went to bed, it was started when I woke up.

Strange… and they didn’t mentioned something about this…

I can’t remember where but I read somewhere that you can just copy the synced db over to shorted the process. I haven’t tried it.

Yeah, u can do this also… by copying the DB

as in /opt/cardano/cnode/db ?

1 Like

It should only take a few minutes of real downtime. I usually do this:

  • compile on one of the relays
  • update binary and restart the first relay (zero downtime for the node, since the other relay is still running)
  • compile and update the second relay (zero downtime for the node)
  • compile (or copy from relay) on the core
  • update the node and restart (a few minutes of downtime unless a ledger rebuild is required)

The last step is executed when I’m sure no blocks will be arriving (query leaderlog for this), so overall the effective downtime is zero.
If you are on cloud services that support snapshots, remember to make them on relays and core before doing the updates, so that you can do a fast rollback if needed.

1 Like

Seems like you didn’t reuse the existing configs and database somehow?

When running ps aux | grep cardano-node does it display --database-path /opt/cardano/cnode/db ?

I did the whole CoinCashew installation process…EXCEPT the actual compile of the cardano binarys. I skipped that. I did the libsodium install, and all the ā€˜apt install’ packages then:

  1. Download the pre-built binaries from the IOHK repository.
  2. Create a directory ~/bin/1.29.0 and untar the pre-builts into this directory.
  3. Create a link ā€œln -s ~/bin/1.29.0 ~/.bin/cardanoā€
  4. Make sure all your scripts are looking for ā€œ~/bin/cardano/cardano-nodeā€
  5. run your node.

When 1.30.0 comes out:

  1. make a directory ~/bin/1.30.0 and untar the latest binaries into that directory.
  2. Stop the node.
  3. Re-create the link ā€œln -sf ~/bin/1.30.0 ~/bin/cardanoā€
  4. Restart the node.

My downtime from start to finish was about 1.3 seconds.

If the pre-built binaries work, as they do for me, there isn’t a need that I’ve seen yet to compile them from source. You just need to make sure all of the dependencies a ā€œbuild-from-sourceā€ install has you install.

@mcrio Runing the command does indeed shows the output:

--topology /opt/cardano/cnode/files/topology.json --config /opt/cardano/cnode/files/config.json --database-path /opt/cardano/cnode/db --socket-path /opt/cardano/cnode/sockets/node0.socket --port 3001 --host-addr 0.0.0.0

So looks good to me.

@scinexus this is a nice way to do it because it easily allows you to rollback in case of failure

I eventually managed to update pretty well to 1.29.0 by building the binaries on my build machine, syncing and running cardano on the build machine and then copying the binaries and db files to the core and relay server. Downtime was minimal and the startup time for the relay was about 5 min (until fully synced). There was about a day gap between uploading the files and running the update. It was my first update so I took it slowly.

It’s strange that the update required you to download the whole ledger again. Usually that is not the case.

I assume you downloaded the additional alonzo genesis file and added the alonzo hashes to the config.

The first time I ran the update I missed the Alonzo hash on the config file. That may have been the reason it triggered a full update, not sure. Other’s on the other thread by @Alexd1985 also seem to have a major rebuild. I don’t mind having to practice an update of the whole ledger as it will probably be the case when the smart contracts update comes along. @mcrio do you normally upgrade whenever you see a new version on git? I notice that v1.29.1 is already out. Or do you wait for some official announcement to update?

1.29.0 is te latest release

@demetrio Watch out for what is the latest stable release on Github:
https://github.com/input-output-hk/cardano-node
You can subscribe to Github notifications for releases.

github-cardano-latest

Additionally, follow the official SPO announcements and upgrade on mainnet when they say it’s safe to do so:

Also subscribe to the SPO newsletter if you haven’t already done so:
https://mailchi.mp/iohk/spo

This is the way…