Updating Offline Nodes

Staking & Delegation Operate a Stake Pool
,
1

Sharing this as it may be useful for all SPO’s out there.

Administrators of Linux machines may sometimes be required to update the Operating system of servers that are not connected to the internet. These servers may not have any internet access for a variety of reasons including:

  • They are at locatiions that have extremely slow and unusable or even no internet connections.

  • They are were purposefully disconnected from the internet for security reasons

These servers are called offline or air gapped machines.

In Cardano, stake pool operators are required to maintain an offline machine. For security reasons, this machine should never be connected to the internet.

On a regular basis (sometimes even daily), Operating System updates are released. These may include bug fixes, security updates and new features or enhancements.

  • Online machines that are Internet connected can benefit from these updates whenever the operator does regular maintenance or sometimes automatically for some updates that can be auto installed.

  • Offline machines that are not internet connected receive none of these updates. Nevertheless, they can be used for months or even years without having any updates as long as there is no need to update them.

For Cardano Stake Pool Operators, there was no real requirement to update the Operating System (OS) of their offline machine until version 1.35 required the installation of a new package (libsecp256k1). While only one new package was required, installing this package required other OS packages to be installed beforehand. These other packages were dependent on other packages to also be installed first and the cycle repeats.

Some operators are able to quickly resolve the issue by connecting their offline machine to the internet. This is not a valid solution in my opinion as it can compromise security— defeating the purpose of having an offline machine in the first place.

For this reason, I documented the steps to assist anyone needing to update an offline machine.

Options Available

As I was researching this article, I came across various other articles— many of which were written in a way that makes the reader think that their solution is the only way to update an offline machine.

Unfortunately, I soon faced issues when I tried them out. Further research and troubleshooting led me to four of the better ways to update an offline machine. They are:

  1. Manual Installation

  2. Apt-Offline

  3. Package Managers (e.g. Synaptic)

  4. Rebuild

Of course there are other ways including creating your own package repository. However, these are the methods that I recommend as they are easier and more practical to use. Each method has their advantages and disadvantages. You can do your own research on how to use the 4 options or you can check out this article that we wrote for Cardano SPOT Check which has more technical details:

Hope this helps!

3 Likes
2

That URL is private. You maybe want to replace it by the public URL https://blockchainlens.gitbook.io/cardano-spot-check/tips/updating-offline-nodes to save people one login plus one click.

1 Like
3

Thanks! Have updated the link.

1 Like
4

Good reminder.
It is always a bit tempting to just connect your cold machine quickly. Just for a moment. It won’t hurt… (don’t do it)

6

Thanks for this!

7

Knowing how to upgrade and install packages on an air-gapped computer using a package manager and offline repository is very helpful in general. Thank you.

Copying the compiled libsecp256k1 shared library files to the air-gapped computer also seems possible:

  1. On the air-gapped computer, add the following lines at the end of the ~/.bashrc file:
export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"
export PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH"
  1. To reload the ~/.bashrc file, type:
source ~/.bashrc
  1. Using removable media, copy the following files from the /usr/local/lib folder on a computer having libsecp256k1 installed to the /usr/local/lib folder on the air-gapped computer, ensuring that file ownership and permissions match:
-rw-r--r-- root root libsecp256k1.a
-rwxr-xr-x root root libsecp256k1.la
-rwxr-xr-x root root libsecp256k1.so.0.0.0
  1. To create symbolic links, type the following commands in a terminal window on the air-gapped computer:
cd /usr/local/lib
sudo ln -s libsecp256k1.so.0.0.0 libsecp256k1.so
sudo ln -s libsecp256k1.so.0.0.0 libsecp256k1.so.0
  1. Confirm that in step 4 you created the following symbolic links in the /usr/local/lib folder:
lrwxrwxrwx root root libsecp256k1.so -> libsecp256k1.so.0.0.0
lrwxrwxrwx root root libsecp256k1.so.0 -> libsecp256k1.so.0.0.0
  1. On the air-gapped, offline computer, type:
sudo ldconfig

The procedure above is written using Ubuntu 20.04 LTS

If any issues are found with copying the libsecp256k1 shared libraries to the offline computer, please let me know.

CHG

8

If you are using apt as the package manager (Debian, Ubuntu, etc.) and you want to compile libsecp256k1 as a deb package, I have provided a Github repository for doing this. You can build your own deb and then just copy that to your offline machine. This way you let the package manager take care of everything. Build once and install on as many machines as you like with no changes to LD_LIBRARY_PATH, PKG_CONFIG_PATH and no manual copying of files.

9

The next pull request for the Coin Cashew guide includes a link to your GitHub repository (line 68):

https://github.com/coincashew/coincashew/pull/141/files#diff-4c28786d149ae07343199efa599c24884f6747f536ecc3432129ab9912628955

To value your time and effort, I also delegated some ADA to your pool.

CHG

1 Like
10

That is very nice of you to delegate some Ada to my pool. I don’t expect that. I actually advise the following on my very crappy website:

I do not recommend you stake with Terminada pool yet because the protocol does not allow setting the fixed fee any lower than 340 Ada.

With a fixed fee of 340 Ada you will lose too much of your rewards to fees until the pool size is over 10 million Ada.

When I started up my stake pool, I went through every guide I could find including Coin Cashew. Being a long term Debian user, I hate installing software manually on my servers. I always look for deb packages but I couldn’t find any so I built my own. They work well, but I am embarrassed to note that if a proper Debian maintainer looked at them they would complain because they don’t include all the copyright docs and other things they should. They only contain Cardano necessary things.

I see so many people getting upgrade problems with library permissions, locations, paths, and just removing old versions of the binaries. I keep notes of all my installation + configuration steps and I still often stuff up when repeating them.

I do believe pool operators should compile their own cardano-node software and shouldn’t blindly trust others. However, once built, you can simply stick your debs in a local repository. Then updating all your individual nodes becomes a simple ‘apt dist-upgrade’. That is how running Debian and Ubuntu is supposed to be.

If you see any problems with where the debs install files or how they set things up please let me know because I could certainly benefit from hints on how to be a better Debian/Ubuntu maintainer.

11

This is a very good, straightforward and quick solution to installing libsecp256k1 without having to compile. Thanks!

Just want to add that you should also copy the file /usr/local/lib/pkgconfig/libsecp256k1.pc onto your airgap machine.

If you don’t mind, I will update SPOT Check with this solution with of course credit to yourself.

12

Thanks @bclens