Some thoughts concerning the node build process

Recently I did some analyzing of my node build process, and here are a couple of conclusions I drew from it. I thought to share them here, so you can make your own.

Libsodium fork

First of all, I see some guides (e.g. Coincashew) saying to add the following lines to the cabal.project.local file:

package cardano-crypto-praos
  flags: -external-libsodium-vrf

Cardano uses a custom fork of libsodium which exposes some internal functions and adds some other new functions, that’s the library we all have to install. If I understood correctly, adding the lines above will disable the need for that custom install (the minus disables the flag) and will use a standard libsodium installation. There is C-code internal to the cardano-crypto-praos package which also contains the custom code (apparently GHC can also compile an link C-code…) and this will be used where it’s needed. The documentation on cardano-node-wiki/docs/getting-started/install.md at main · input-output-hk/cardano-node-wiki · GitHub clearly states that this internal C-code should ONLY be used for development purposes, so the developers don’t have to deal with custom libsodium installations. I guess that internal C-code contains a lot of copy/paste of those internal functions (that needs to be exposed) spoken off.

So for compiling a node, you SHOULD NOT add the lines above (so the custom code from the libsodium library will be used and NOT the internal C-code)!

Optimization

Some guides (e.g. Coincashew again) say to execute the following:

cabal configure -O0 -w ghc-8.10.7

This will add two lines to the cabal.project.local file:

with-compiler: ghc-8.10.7
optimization: 0

The first is ok, the second one disables optimization however… But why would you want to disable this? It will compile faster and probably give smaller binaries without optimization, but I rather spend a little more time on compiling and have a little bigger binary if that means if my node can run more optimized, read faster… So better change the -O0 in -O2 (which will give the most optimization).

Then after doing some digging, it seems that this flag only applies to the code in the current package. But most of the code (that matters) is located in other packages! So in order to truly enable optimization, you’ll have to add the following to enable optimization for all packages:

package *
  optimization: 2

LLVM

I also compile the node with the llvm backend, it’ll take longer to compile, will give bigger binaries (about twice as big I thought), but it’ll also contain more optimized code! The same applies as with the optimization from above: you’ll need to enable it for the package itself and for all other packages if you want all packages compiled with llvm:

My complete cabal.project.local file looks as follows:

ignore-project: False
with-compiler: ghc-8.10.7
optimization: 2
program-options
  ghc-options: -fllvm  
package *
  optimization: 2
  ghc-options: -fllvm

When compiling with GHC 8.10.7, you’ll need to install an llvm version between 9 and 12 an the machine where you compile your nodes. I’ve seen somewhere (not sure where anymore) that 13 would also work. I use llvm version 12.0.0, so I can’t confirm the former.

Be sure to add the folder that contains the opt and llc binaries to your path. Check with opt --version and llc --version if the binary can be located and if the right versions are used!

Remarks

  • You don’t need to clear your cabal package store when changing the optimization level or when switching between the native and llvm backend. It won’t use the already compiled ones if one of those is different than the already compiled one, it will compile the packages again and add it to the store with a different hash. You can however to save space.
  • I’m certainly no expert on this. I drew these conclusions from what I’ve read in the docs and from some experimenting of my own. Feel free to correct me if I drew the wrong conclusions.
  • If (some of) my conclusions seems to be correct, the guides should be updated (especially concerning the first topic). Feel free to initiate this process yourself… :slight_smile:
2 Likes

Thanks for this. I did notice that the latest install.md instruction don’t include the external-libsodium-vrf flag. So I suspect you are right in saying we shouldn’t include that in the build as flag. I had a feeling it was there earlier though, but I can’t be certain.

I’m keen to see what others think too.

2 Likes

Maybe it was there before without the minus which would be the same as omitting it and hence will use the right code for production?

1 Like

What do you think about also adding the following to cabal.project.local?

split-sections: True
executable-stripping: True
library-stripping: True

This is what cabal reference says:

  • split-sections: “This reduces the final size of the executables that use the library by allowing them to link with only the bits that they use rather than the entire library. The downside is that building the library takes longer and uses a bit more memory.”
  • executable-stripping: “When installing binary executable programs, run the strip program on the binary. This can considerably reduce the size of the executable binary file. It does this by removing debugging information and symbols.”
  • library-stripping: “When installing binary libraries, run the strip program on the binary, saving space on the file system.”

I have been adding those but I don’t know if they make any difference to performance?

I have also been compiling using ghc-9.8.2 and cabal 3.10.2.1 because I want to use the Haskell nonmoving garbage collector on my ARM machines which performs better in later versions.

1 Like

Don’t know about these options. My guess is they won’t have an effect on performance.

These only save space in terms of the executable size, which probably isn’t that important given that there is the entire block-chain and ledger states laying around somewhere in the filesystem as well.

I wish cardano-node had a benchmark suit so we could compare which options matter and which don’t.

I think if you’ve installed the specific libsodium version noted in the earlier section of cardano-node-wiki/docs/getting-started/install.md at main · input-output-hk/cardano-node-wiki · GitHub

i.e.

: ${SODIUM_VERSION:='dbb48cc'}
git clone https://github.com/intersectmbo/libsodium
cd libsodium
git checkout $SODIUM_VERSION
./autogen.sh
./configure
make
make check
sudo make install

Then I think the flag below is redundant, as mentioned by the install.md notes:

In order to avoid having to install the custom version of libsodium for development purposes, cardano-crypto-praos defines a cabal flag that makes use of C code located here.

The C code is merely a port of the bits missing in a normal libsodium installation. To enable this code, one has to add the following code in the cabal.project.local file:

package cardano-crypto-praos flags: -external-libsodium-vrf

What do you think?

I’ve always build with the flag as noted by coincashew, and I’ve had no problems minting blocks. But keen to make sure I’m following the right practice.

1 Like

Be aware, that line disables the flag (because of the minus sign)! The default is enabling the flag, wich you should do; so not including it is ok. The CoinCashew guide let you disable it, the guide even says it:

  1. Using a text editor, open the file named cabal.project.local located in the current folder, and then add the following lines at the end of the file to avoid installing a custom libsodium library:

But the documentation on cardano-node-wiki/docs/getting-started/install.md at main · input-output-hk/cardano-node-wiki · GitHub says (even in a warning):

The ported c code should not be used to run the node, and should only be used for development purposes.

Even if you’ve installed the modified libsodium, disabling the flag (so adding those lines) will still have the consequence of using the ported C code. So the CoinCashew guide is WRONG!
It’s a bit funny that in an earlier step, it gives instructions how to install the modified libsodium and then it says do this to avoid installing the modified libsodium (which you already did).

If no one beats me to it, I’m going to make a PR for the CoinCashew guide to remove that part somewhere in the near future.

The CoinCashew guide is following the IOHK documentation, i.e. using the ‘-’. I understand by using the ‘-’ flag, it tells cabal to not use the external libsdoium-vrf but use the internal one in the praos package.
I see now there is a bit of chat on the discord server from late last year that discusses this and with the new libsodium build dbb48cce then this flag is redundant. So while CoinCashew has used the flag as per the IOHK instructions, it shouldn’t be used now, as you point out.

I’ll race you to make the PR request on CoinCashew!

1 Like

Where were those instructions? The IOHK instructions say to use it only for development purposes. Or was this different in the past (didn’t check it)?

I meant the "-’ flag is as per the IOHK instructions. I agree that the instructions says for development, but they may have been different a while back (referring to some old discussion in the discord server), as it didn’t build unless you used the -eternal-libosdium-vrf flag.

I was including the extra IOHK ported C code to compile cardano-node with the following lines in cabal.project.local:

package cardano-crypto-praos
  flags: -external-libsodium-vrf

Those lines in cabal.project.local were allowing me to compile cardano-node against the Debian stable version of libsodium.

I think this comment in the build instructions is relatively new:

The ported c code should not be used to run the node, and should only be used for development purposes.

I don’t recall that instruction being so strongly worded in the past.

I also notice that IOHK have gotten more sophisticated with their instructions about the versions they require for libsodium, libsecp256k1 and libblst libraries. They now have a web link providing the specific version for each library depending upon the cardano-node version:

CARDANO_NODE_VERSION='8.7.3'
IOHKNIX_VERSION=$(curl https://raw.githubusercontent.com/IntersectMBO/cardano-node/$CARDANO_NODE_VERSION/flake.lock | jq -r '.nodes.iohkNix.locked.rev')
echo "iohk-nix version: $IOHKNIX_VERSION"
SODIUM_VERSION=$(curl https://raw.githubusercontent.com/input-output-hk/iohk-nix/$IOHKNIX_VERSION/flake.lock | jq -r '.nodes.sodium.original.rev')
echo "Using sodium version: $SODIUM_VERSION"
SECP256K1_VERSION=$(curl https://raw.githubusercontent.com/input-output-hk/iohk-nix/$IOHKNIX_VERSION/flake.lock | jq -r '.nodes.secp256k1.original.ref')
echo "Using secp256k1 version: ${SECP256K1_VERSION}"
2 Likes

@brouwerQ PR done for Coincashew.

1 Like