Cardano-node: internal error: evacuate: strange closure type 0

I am having weird errors with the cardano-node. It stops woking with the next:

[papal01:cardano.node.ErrorPolicy:Notice:70] [2022-01-09 13:58:43.63 UTC] IP 35.158.3.162:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s
[papal01:cardano.node.ChainDB:Notice:51] [2022-01-09 13:58:44.77 UTC] Chain extended, new tip: 5c55abe80728f78c7055b12740aabb8aa367d4086481c445774997a7f06414e8 at slot 418779
[papal01:cardano.node.ChainDB:Notice:51] [2022-01-09 13:58:46.02 UTC] Chain extended, new tip: f3ede6fb926970fef8f646b26c676ffc7c5cf14ffa6a62541f957b706c822b48 at slot 419765
cardano-node: internal error: evacuate: strange closure type 0
    (GHC version 8.10.7 for x86_64_unknown_linux)
    Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug
cardano-node/startRelayNode1.sh: line 9: 588098 Aborted                 (core dumped) /usr/local/bin/cardano-node run +RTS -N -A16m -qg -qb -RTS --topology ${TOPOLOGY} --database-path ${DB_PATH} --socket-path ${SOCKET_PATH} --host-addr ${HOSTADDR} --port ${PORT} --config ${CONFIG}

The coredump does not show anything intersting. I am having this problem on 1 relay. The other relay is working as expected even that both are now running on version 1.33.0

Additionaly:

Jan 09 07:07:20 papal01 kernel: traps: cardano-node:w[559803] general protection fault ip:3fb7c1c sp:7fa6dfffa9b8 error:0 in cardano-node[400000+4158000]

From abrt:

[root@papal01 ccpp-2022-01-09-08:22:12-588709]# abrt-cli list
id bd71c25cb38740881c36141855ae632e213bfae1
reason:         ouroboroszmconsensuszmbyronzm0zi1zi0zi0zm4d134515e213dcc3681c0703cc35ae242e6ca92e0294fc42c6216482d532beef_OuroborosziConsensusziByronziLedgerziBlock_zdfGetPrevHashByronBlockzuzdcheaderPrevHash_info(): cardano-node killed by SIGSEGV
time:           Sun 09 Jan 2022 08:22:12 AM CST
cmdline:        /usr/local/bin/cardano-node run +RTS -N -A16m -qg -qb -RTS --topology /home/adamin/cardano-node/mainnet-topology.json --database-path /home/adamin/cardano-node/db --socket-path /home/adamin/cardano-node/db/socket --host-addr 0.0.0.0 --port 6000 --config /home/adamin/cardano-node/mainnet-config.json
uid:            1000 (adamin)
count:          1
Directory:      /var/spool/abrt/ccpp-2022-01-09-08:22:12-588709

It is looping because everytime crashes. Frecuently showing:

Jan 09 08:32:08 papal01 cardano-node[589014]: [papal01:cardano.node.ChainDB:Error:59] [2022-01-09 14:32:08.95 UTC] Invalid block c4227b378a644f8cebf57a21fbde8442d18703340d78f18652641d6db4ffc489 at slot 530988: ExtValidationErrorLedger (HardForkLedgerErrorFromEra Z (WrapLedgerErr {unwrapLedgerErr = ChainValidationUTxOValidationError (UTxOValidationUTxOError (UTxOMissingInput (TxInUtxo 43faba30c9752a166efabde4934038dff08cd30c09ebff23e83db4057f608762 0)))}))
Jan 09 08:32:08 papal01 cardano-node[589014]: [papal01:cardano.node.BlockFetchDecision:Info:69] [2022-01-09 14:32:08.96 UTC] before next, messages elided = 4290
Jan 09 08:32:08 papal01 cardano-node[589014]: [papal01:cardano.node.BlockFetchDecision:Info:69] [2022-01-09 14:32:08.96 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.100.201:44161, remoteAddress = 18.158.118.230:3001}) (Left (FetchDeclineConcurrencyLimit FetchModeBulkSync 1)),TraceLabelPeer (ConnectionId {localAddress = 192.168.100.201:45147, remoteAddress = 192.168.100.211:6000}) (Left (FetchDeclinePeerBusy 234993 196608 393216)),TraceLabelPeer (ConnectionId {localAddress = 192.168.100.201:36803, remoteAddress = 18.224.177.133:3001}) (Left (FetchDeclineConcurrencyLimit FetchModeBulkSync 1)),TraceLabelPeer (ConnectionId {localAddress = 192.168.100.201:43507, remoteAddress = 54.151.113.34:3001}) (Left (FetchDeclineConcurrencyLimit FetchModeBulkSync 1)),TraceLabelPeer (ConnectionId {localAddress = 192.168.100.201:38751, remoteAddress = 52.15.49.197:3001}) (Left (FetchDeclineConcurrencyLimit FetchModeBulkSync 1))]
Jan 09 08:32:17 papal01 cardano-node[589014]: [papal01:cardano.node.ErrorPolicy:Warning:88] [2022-01-09 14:32:17.50 UTC] IP 192.168.100.211:43815 ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError (MuxIOException Network.Socket.recvBuf: resource vanished (Connection reset by peer)) "(recv errored)"))) 20s 20s

Somebody help?

Shot in the dark, but do you have enough resources on that node?

Yes. I have 2 relay nodes and this one gives such problems. Both relays have the same specs over CPU, RAM and disks.

Try to clean install?

I would not like to try it because, otherwise, it is not a workaround. But at this moment without any other clues… First a bit wait, maybe more ideas appear.

Did you resolve this?
I’ve bumped from cardano-node 1.31 to 1.33.
Running a node repeated hit a seg fault before syncing.
I deleted the db and tried again but to sync from scratch.
I then got a “strange closure” error similar to the above.
I’ve restarted the node and it seems to be continuing. For now.
Any tips?

Hey are you still having this issue? Trying using pool.vet and see if that gives you any insight into your problems. Let me know your stake pool ticker and I can look as well.

If all else fails I would probably just rebuild the node from scratch personally. Especially because you have another relay to support your pool in the meantime.

I was only trying to run a relay node from the binary from iohks repo.
Subsequent attempts all ended in seg faults and eventually i gave up and used another machine where it just worked.
:person_shrugging: It’ll do for now but I might circle back to this

1 Like

I continue to see this error, just going through the basic getting started, installing, running cardano-node (v 1.34.1).

> [dev:cardano.node.ChainDB:Notice:72] [2022-04-17 05:06:05.07 UTC] Chain extended, new tip: 2b4c3a1c20f0bbc9296cc0bcefe3e5abe5dd83e744a255f20b42791c124501ce at slot 17792675
> [dev:cardano.node.ChainDB:Notice:72] [2022-04-17 05:06:06.33 UTC] Chain extended, new tip: 9bf9dc14762b6da038f6555d47cfa9ea368e177ba7087fb7d88b14c9821c8b20 at slot 17793307
> cardano-node: internal error: evacuate: strange closure type 0
>     (GHC version 8.10.7 for x86_64_unknown_linux)
>     Please report this as a GHC bug:  https://www.haskell.org/ghc/reportabug

I don’t see a core file on my system however, my ulimit is set to unlimited.

I have faced the same problem when I updated from 1.35.3 to 1.35.4
I have cncli sync working as well and thought it was the reason. I went and deleted both the DB for cnode and for cncli and still have the same recurring issue.