[Need Support] Relay node keeps dying after restart (testnet) undefined crypto_vrf_publickeybytes?

Staking & Delegation Setup a Stake Pool
1

Hey guys, stumped on this and no answers online seem to get it working for me.

I have a basic, fresh relay node (1.25.1) sitting on a digitalocean droplet, and setting up stuff went well – got it restarting with systemd and what not. Problem is now every time I start the node it dies and gives me this about 15 seconds after starting up and opening the db on the immutable tip:

/home/node/.local/bin/cardano-node: symbol lookup error: /home/node/.local/bin/cardano-node: undefined symbol: crypto_vrf_publickeybytes

Here is the script I am using to run the node:

#!/bin/bash
/home/node/.local/bin/cardano-node run --topology /home/node/src/testnet-topology.json --database-path /home/node/src/db --socket-path /home/node/src/db/node.socket --host-addr 0.0.0.0 --port 3001 --config /home/node/src/testnet-config.json

Initial googling tells me that “I didn’t install libsodium” but I did and followed every step very closely.

Other answers will say that I don’t have the following set:

export PATH=“~/.local/bin:$PATH”
export CARDANO_NODE_SOCKET_PATH=~/src/db/node.socket
export LD_LIBRARY_PATH=“/usr/local/lib:$LD_LIBRARY_PATH”
export PKG_CONFIG_PATH=“/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH”

But indeed I do… and its in my .bashrc file and being loaded every time into the environment. I have confirmed the results with env and everything seems to be good there as far as I can tell.

PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:
LD_LIBRARY_PATH=/usr/local/lib:
CARDANO_NODE_SOCKET_PATH=/home/node/src/db/node.socket
PATH=/home/node/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

Both my non-root user and root user have these paths confirmed in their env

I’ve tried multiple rebuilds, re-installs, new droplets, more memory, and have not had any different results. If I try to run the script as a systemd service then the service will fail with code 127.

systemd[1]: cnode-restart.service: Main process exited, code=exited, status=127/n/a

Node is running a DigitalOcean droplet with 8GB RAM, 4CPUs, 80GB – Libsodium, ghc 8.10.2, and I am using 1.25.1 for the node software.

I am aware there are other topics on this posted a few months ago, but I have followed their instructions and it has not changed my output (as shown above).

Any experienced node operators that have seen this issue before and know where I might still be messing up? Again all of this is running on the testnet and seems to happen once my node has synced for a bit and ran for a few hours.

Thanks for any help ~

2

why do you run the 1.25.2?

3

Sorry I meant to post 1.25.1

I had problems running another version and saw some people had success with that version.

When you take a look at the tags list you get

1.0.0 1.1.0 1.10.0 1.10.1 1.11.0 1.12.0 1.13.0 1.13.0-rewards 1.14.0 1.14.1 1.14.2 1.15.0 1.15.1 1.16.0 1.17.0 1.18.0 1.18.1 1.19.0 1.19.1 1.2.0 1.20.0 1.21.0 1.21.1 1.21.2 1.22.0 1.22.1 1.23.0 1.24.0 1.24.1 1.24.2 1.25.0 1.25.1 1.3.0 1.4.0 1.5.0 1.6.0 1.7.0 1.8.0 1.9.0 1.9.1 1.9.2 1.9.3 pioneer pioneer-1 pioneer-2 pioneer-3 pioneer-4 pioneer-5 pioneer-new-address pioneer-wave2

Am I supposed to be using pioneer-wave2?

4

on this script, do you have host IP or you use 0.0.0.0 ?
please use, 0.0.0.0

Screenshot 2021-03-03 at 21.17.32
Screenshot 2021-03-03 at 21.17.321476×338 34.6 KB

5

I was originally using the host IP.

I changed it to 0.0.0.0 and I am still getting the same error message.

6

This would typically occur if you did not follow pre-requisites to ensure your machine has libsodium installed and available in your LD_LIBRARY_PATH.

7

Correct. Which I discovered upon initial googling of my problem.

But as my post says, I can assure you I did follow all pre-requisites verbatim, and I do believe I have libsodium installed and available in my LD path.

This is what leads me to believe that:

node@relay-alpha:~$ echo $LD_LIBRARY_PATH
/usr/local/lib:/usr/local/lib:

and

node@relay-alpha:~$ ls -la /usr/local/lib
total 9984
drwxr-xr-x 5 root root 4096 Mar 3 04:43 .
drwxr-xr-x 10 root root 4096 Oct 16 21:39 …
drwxr-xr-x 41 root root 4096 Mar 3 04:40 ghc-8.10.2
-rw-r–r-- 1 root root 6474828 Mar 3 04:43 libsodium.a
-rwxr-xr-x 1 root root 959 Mar 3 04:43 libsodium.la
lrwxrwxrwx 1 root root 19 Mar 3 04:43 libsodium.so → libsodium.so.23.3.0
lrwxrwxrwx 1 root root 19 Mar 3 04:43 libsodium.so.23 → libsodium.so.23.3.0
-rwxr-xr-x 1 root root 3723104 Mar 3 04:43 libsodium.so.23.3.0
drwxr-xr-x 2 root root 4096 Mar 3 04:43 pkgconfig
drwxrwsr-x 3 root staff 4096 Oct 16 21:39 python3.8

Unless there was some unstated configuration step I did not follow, or my node software version of 1.25.1 is outdated/incorrect.

8

Checking against env command may not always be useful especially if you’re starting something via systemd (as systemd invocation will not necessarily spawn bash session or invoke bashrc file unless specifically being sourced).

9

While that makes sense, disregarding systemd, I get the same result if I simply run the node from a non-root user.

If checking against env and using export LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH" are not the correct actions to take to make libsodium available to my environment, then I am a little confused as to what are. How can I check if am I configuring correctly?

10

In the same shell session where you get the error for starting node, execute the below:
id && ldd /home/node/.local/bin/cardano-node

11

Output from your command was

node@relay-alpha:~/src$ id && ldd /home/node/.local/bin/cardano-node
uid=1000(node) gid=1000(node) groups=1000(node),27(sudo)
linux-vdso.so.1 (0x00007ffe9e0e6000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f6ec511e000)
libsodium.so.23 => /usr/local/lib/libsodium.so.23 (0x00007f6ec50bf000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f6ec509c000)
libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x00007f6ec4fed000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f6ec4fe2000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f6ec4fdd000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6ec4fd5000)
libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f6ec4f51000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6ec4d5f000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f6ec4c10000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6ec5143000)
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f6ec4be7000)
liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f6ec4bc6000)
libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f6ec4aa6000)
libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f6ec4a83000)

Unfortunately no change in output for node start

[relay-al:cardano.node.ChainDB:Info:5] [2021-03-04 05:38:08.70 UTC] Opened imm db with immutable tip at 42f4273b834d1da550a287b5cbe6fb3bdff5da8cbaa96932941c918c5e3e3207 at slot 1596122 and chunk 73
[relay-al:cardano.node.ChainDB:Info:5] [2021-03-04 05:38:13.23 UTC] Opened vol db
[relay-al:cardano.node.ChainDB:Info:5] [2021-03-04 05:38:13.28 UTC] Replaying ledger from snapshot DiskSnapshot {dsNumber = 1594071, dsSuffix = Nothing} at cf0bc2d0ace060e7cc074814973b51060203ef7657c23b6c19dd12d2a89d8178 at slot 1594071
[relay-al:cardano.node.ChainDB:Info:5] [2021-03-04 05:38:13.28 UTC] Replayed block: slot SlotNo 1594072 of At (SlotNo 1596122)
[relay-al:cardano.node.ChainDB:Info:5] [2021-03-04 05:38:13.43 UTC] block replay progress (%) = 99.9
[relay-al:cardano.node.ChainDB:Info:5] [2021-03-04 05:38:13.57 UTC] block replay progress (%) = 100.0
[relay-al:cardano.node.ChainDB:Info:5] [2021-03-04 05:38:13.58 UTC] before next, messages elided = 1594072
[relay-al:cardano.node.ChainDB:Info:5] [2021-03-04 05:38:13.58 UTC] Replayed block: slot SlotNo 1596122 of At (SlotNo 1596122)
[relay-al:cardano.node.ChainDB:Info:5] [2021-03-04 05:38:13.58 UTC] Opened lgr db
[relay-al:cardano.node.ChainDB:Info:5] [2021-03-04 05:38:14.24 UTC] Opened db with immutable tip at 42f4273b834d1da550a287b5cbe6fb3bdff5da8cbaa96932941c918c5e3e3207 at slot 1596122 and tip 9e3aafec3a36e6b8a571b358cb83811b29af5480b4095d601a5d6c7e3f27e055 at slot 1598282
[relay-al:cardano.node.DnsSubscription:Notice:109] [2021-03-04 05:38:14.24 UTC] Domain: “relays-new.cardano-testnet.iohkdev.io” Connection Attempt Start, destination 54.250.11.33:3001
[relay-al:cardano.node.DnsSubscription:Notice:110] [2021-03-04 05:38:14.27 UTC] Domain: “relays-new.cardano-testnet.iohkdev.io” Connection Attempt Start, destination 18.177.214.255:3001
[relay-al:cardano.node.DnsSubscription:Notice:111] [2021-03-04 05:38:14.29 UTC] Domain: “relays-new.cardano-testnet.iohkdev.io” Connection Attempt Start, destination 54.177.23.120:3001
[relay-al:cardano.node.DnsSubscription:Notice:112] [2021-03-04 05:38:14.32 UTC] Domain: “relays-new.cardano-testnet.iohkdev.io” Connection Attempt Start, destination 54.151.49.138:3001
[relay-al:cardano.node.DnsSubscription:Notice:113] [2021-03-04 05:38:14.35 UTC] Domain: “relays-new.cardano-testnet.iohkdev.io” Connection Attempt Start, destination 52.74.88.13:3001
[relay-al:cardano.node.DnsSubscription:Notice:111] [2021-03-04 05:38:14.36 UTC] Domain: “relays-new.cardano-testnet.iohkdev.io” Connection Attempt End, destination 54.177.23.120:3001 outcome: ConnectSuccess
[relay-al:cardano.node.DnsSubscription:Notice:114] [2021-03-04 05:38:14.37 UTC] Domain: “relays-new.cardano-testnet.iohkdev.io” Connection Attempt Start, destination 54.177.182.253:3001
[relay-al:cardano.node.DnsSubscription:Notice:109] [2021-03-04 05:38:14.39 UTC] Domain: “relays-new.cardano-testnet.iohkdev.io” Connection Attempt End, destination 54.250.11.33:3001 outcome: ConnectSuccessLast
[relay-al:cardano.node.ErrorPolicy:Notice:99] [2021-03-04 05:38:14.39 UTC] IP 18.177.214.255:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = “Parallel connection cancelled”, seStack = }))) 1s
[relay-al:cardano.node.DnsSubscription:Notice:112] [2021-03-04 05:38:14.39 UTC] Domain: “relays-new.cardano-testnet.iohkdev.io” Connection Attempt End, destination 54.151.49.138:3001 outcome: ConnectValencyExceeded
[relay-al:cardano.node.ErrorPolicy:Notice:99] [2021-03-04 05:38:14.39 UTC] IP 52.74.88.13:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = “Parallel connection cancelled”, seStack = }))) 1s
[relay-al:cardano.node.ErrorPolicy:Notice:99] [2021-03-04 05:38:14.39 UTC] IP 54.177.182.253:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = “Parallel connection cancelled”, seStack = }))) 1s
/home/node/.local/bin/cardano-node: symbol lookup error: /home/node/.local/bin/cardano-node: undefined symbol: crypto_vrf_publickeybytes

12

Given that the ldd shows the correct path for libsodium, the two things to verify would be whether your node was built using statically linked libsodium dependency or a system dependency one.

To start with, execute the commands below:

echo -e "#include <sodium.h>\nint main(){if (sodium_init() < 0) {return 1;}\nreturn 0;}" | gcc -x c++ -E -W - 2>&1 | grep -e crypto_vrf_publickeybytes -e crypto_vrf_ietfdraft03_proofbytes

If it shows a hit for both the symbols, then you should strictly concentrate on the instructions used for building cardano-node. Did you copy the cardano-node build binaries from another machine, or was it built locally?

13

Specify the env variables in a EnvironmentFile that your systemd unit loads.

14

So it seems that the symbols are hitting.

node@relay-alpha:~# echo -e “#include <sodium.h>\nint main(){if (sodium_init() < 0) {return 1;}\nreturn 0;}” | gcc -x c++ -E -W - 2>&1 | grep -e crypto_vrf_publickeybytes -e crypto_vrf_ietfdraft03_proofbytes
size_t crypto_vrf_ietfdraft03_proofbytes(void);
size_t crypto_vrf_publickeybytes(void);

I followed, line for line, very closely, this tutorial to build my node: https://docs.cardano.org/projects/cardano-node/en/latest/getting-started/install.html

The cardano binaries were built locally after cloning the source code from their git repo (1.25.1), as per their instructions.

Thank you for your help so far and forgive me but I’m not understanding what exactly you mean by “strictly concentrate on the instructions used to build the node” – I have not used any other instructions besides those.

15

While that is likely a solution when I get back to running it on systemd, my current problem is that even running a simple command as a non-root user to run the node leaves me with this same error message.

16

Try installing the libsodium-dev system package and rebuild 1.25.1.

17

Unfortunately that did not fix the error. Still failing on undefined symbol: crypto_vrf_publickeybytes

I ran

sudo apt-get update -y
sudo apt-get install -y libsodium-dev
cabal build all
cabal install --installdir ~/.local/bin cardano-cli cardano-node

Not sure if you meant for me to do a full wipe and spin up another droplet to rebuild, this time with libsodium-dev installed before initial cabal build – but I might try that and see if works.

18

I rebuilt completely with a DigitalOcean 8GB droplet and Ubuntu 18.04, following https://cardano-foundation.gitbook.io/stake-pool-course/stake-pool-guide/getting-started/install-node very carefully, and also installed libsodium-dev before building my node – these two things seemed to solve the issue.

My old droplet was Ubuntu 20.04 so not sure if that had something to do with it, but most likely it was a copy-paste error when following build steps.

For anyone reading in the future, be sure you’re verifying you ran each build step correctly.