Cardano-node: internal error: evacuate: strange closure type 0

Staking & Delegation Setup a Stake Pool
1

I am having weird errors with the cardano-node. It stops woking with the next:

[papal01:cardano.node.ErrorPolicy:Notice:70] [2022-01-09 13:58:43.63 UTC] IP 35.158.3.162:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s
[papal01:cardano.node.ChainDB:Notice:51] [2022-01-09 13:58:44.77 UTC] Chain extended, new tip: 5c55abe80728f78c7055b12740aabb8aa367d4086481c445774997a7f06414e8 at slot 418779
[papal01:cardano.node.ChainDB:Notice:51] [2022-01-09 13:58:46.02 UTC] Chain extended, new tip: f3ede6fb926970fef8f646b26c676ffc7c5cf14ffa6a62541f957b706c822b48 at slot 419765
cardano-node: internal error: evacuate: strange closure type 0
    (GHC version 8.10.7 for x86_64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
cardano-node/startRelayNode1.sh: line 9: 588098 Aborted                 (core dumped) /usr/local/bin/cardano-node run +RTS -N -A16m -qg -qb -RTS --topology ${TOPOLOGY} --database-path ${DB_PATH} --socket-path ${SOCKET_PATH} --host-addr ${HOSTADDR} --port ${PORT} --config ${CONFIG}

The coredump does not show anything intersting. I am having this problem on 1 relay. The other relay is working as expected even that both are now running on version 1.33.0

Additionaly:

Jan 09 07:07:20 papal01 kernel: traps: cardano-node:w[559803] general protection fault ip:3fb7c1c sp:7fa6dfffa9b8 error:0 in cardano-node[400000+4158000]

From abrt:

[root@papal01 ccpp-2022-01-09-08:22:12-588709]# abrt-cli list
id bd71c25cb38740881c36141855ae632e213bfae1
reason:         ouroboroszmconsensuszmbyronzm0zi1zi0zi0zm4d134515e213dcc3681c0703cc35ae242e6ca92e0294fc42c6216482d532beef_OuroborosziConsensusziByronziLedgerziBlock_zdfGetPrevHashByronBlockzuzdcheaderPrevHash_info(): cardano-node killed by SIGSEGV
time:           Sun 09 Jan 2022 08:22:12 AM CST
cmdline:        /usr/local/bin/cardano-node run +RTS -N -A16m -qg -qb -RTS --topology /home/adamin/cardano-node/mainnet-topology.json --database-path /home/adamin/cardano-node/db --socket-path /home/adamin/cardano-node/db/socket --host-addr 0.0.0.0 --port 6000 --config /home/adamin/cardano-node/mainnet-config.json
uid:            1000 (adamin)
count:          1
Directory:      /var/spool/abrt/ccpp-2022-01-09-08:22:12-588709

It is looping because everytime crashes. Frecuently showing:

Jan 09 08:32:08 papal01 cardano-node[589014]: [papal01:cardano.node.ChainDB:Error:59] [2022-01-09 14:32:08.95 UTC] Invalid block c4227b378a644f8cebf57a21fbde8442d18703340d78f18652641d6db4ffc489 at slot 530988: ExtValidationErrorLedger (HardForkLedgerErrorFromEra Z (WrapLedgerErr {unwrapLedgerErr = ChainValidationUTxOValidationError (UTxOValidationUTxOError (UTxOMissingInput (TxInUtxo 43faba30c9752a166efabde4934038dff08cd30c09ebff23e83db4057f608762 0)))}))
Jan 09 08:32:08 papal01 cardano-node[589014]: [papal01:cardano.node.BlockFetchDecision:Info:69] [2022-01-09 14:32:08.96 UTC] before next, messages elided = 4290
Jan 09 08:32:08 papal01 cardano-node[589014]: [papal01:cardano.node.BlockFetchDecision:Info:69] [2022-01-09 14:32:08.96 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.100.201:44161, remoteAddress = 18.158.118.230:3001}) (Left (FetchDeclineConcurrencyLimit FetchModeBulkSync 1)),TraceLabelPeer (ConnectionId {localAddress = 192.168.100.201:45147, remoteAddress = 192.168.100.211:6000}) (Left (FetchDeclinePeerBusy 234993 196608 393216)),TraceLabelPeer (ConnectionId {localAddress = 192.168.100.201:36803, remoteAddress = 18.224.177.133:3001}) (Left (FetchDeclineConcurrencyLimit FetchModeBulkSync 1)),TraceLabelPeer (ConnectionId {localAddress = 192.168.100.201:43507, remoteAddress = 54.151.113.34:3001}) (Left (FetchDeclineConcurrencyLimit FetchModeBulkSync 1)),TraceLabelPeer (ConnectionId {localAddress = 192.168.100.201:38751, remoteAddress = 52.15.49.197:3001}) (Left (FetchDeclineConcurrencyLimit FetchModeBulkSync 1))]
Jan 09 08:32:17 papal01 cardano-node[589014]: [papal01:cardano.node.ErrorPolicy:Warning:88] [2022-01-09 14:32:17.50 UTC] IP 192.168.100.211:43815 ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError (MuxIOException Network.Socket.recvBuf: resource vanished (Connection reset by peer)) "(recv errored)"))) 20s 20s
2

Somebody help?

3

Shot in the dark, but do you have enough resources on that node?

4

Yes. I have 2 relay nodes and this one gives such problems. Both relays have the same specs over CPU, RAM and disks.

5

Try to clean install?

6

I would not like to try it because, otherwise, it is not a workaround. But at this moment without any other clues… First a bit wait, maybe more ideas appear.

7

Did you resolve this?
I’ve bumped from cardano-node 1.31 to 1.33.
Running a node repeated hit a seg fault before syncing.
I deleted the db and tried again but to sync from scratch.
I then got a “strange closure” error similar to the above.
I’ve restarted the node and it seems to be continuing. For now.
Any tips?

8

Hey are you still having this issue? Trying using pool.vet and see if that gives you any insight into your problems. Let me know your stake pool ticker and I can look as well.

If all else fails I would probably just rebuild the node from scratch personally. Especially because you have another relay to support your pool in the meantime.

9

I was only trying to run a relay node from the binary from iohks repo.
Subsequent attempts all ended in seg faults and eventually i gave up and used another machine where it just worked.
:person_shrugging: It’ll do for now but I might circle back to this

1 Like
10

I continue to see this error, just going through the basic getting started, installing, running cardano-node (v 1.34.1).

> [dev:cardano.node.ChainDB:Notice:72] [2022-04-17 05:06:05.07 UTC] Chain extended, new tip: 2b4c3a1c20f0bbc9296cc0bcefe3e5abe5dd83e744a255f20b42791c124501ce at slot 17792675
> [dev:cardano.node.ChainDB:Notice:72] [2022-04-17 05:06:06.33 UTC] Chain extended, new tip: 9bf9dc14762b6da038f6555d47cfa9ea368e177ba7087fb7d88b14c9821c8b20 at slot 17793307
> cardano-node: internal error: evacuate: strange closure type 0
>     (GHC version 8.10.7 for x86_64_unknown_linux)
>     Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug

I don’t see a core file on my system however, my ulimit is set to unlimited.

11

I have faced the same problem when I updated from 1.35.3 to 1.35.4
I have cncli sync working as well and thought it was the reason. I went and deleted both the DB for cnode and for cncli and still have the same recurring issue.

12

I have the same issue on my machine. It is an intel NUC with 16GB of RAM running 1.36 as a relay. I have consistently had this issue on this system across multiple OS installs.

[intelNUC:cardano.node.ChainDB:Notice:332] [2022-12-14 06:32:20.88 UTC] Chain extended, new tip: a6a1eca035bd545a56c2ca18e1846f895a2c6c8b0a5826ecdaf9c6cdbe46d612 at slot 79433249

cardano-node: internal error: evacuate: strange closure type -998029057

(GHC version 8.10.7 for x86_64_unknown_linux) Please report this as a GHC bug: https://www.haskell.org/ghc/reportabug /home/bismuth/.nix-profile/bin/node_launch: line 2: 1039463 Aborted (core dumped)

13

Cardano-node version 1.36???

14

Yes. Since I am not running more than a relay node, I tend to compile and run the latest.

15

Then it might be an answer why it crashed, 1.36 is not official released yet, so I would expect that it might not work stable as you wish.

16

I disagree.

It crashed with the exact same error code when I used version 1.35.4. I suspect that the real answer is that the machine I am running isn’t powerful enough to run a node since 1.36 runs without issue for weeks on my other machine running a near identical NixOS derivation.