Need help with cardano-node operation

English Community Technical Support
1

Hi everyone. I’m finding the build of a Cardano node highly unstable. in Test Mode, the Node will run and this start you can see the tip and the blocks increment, but let it run a few hours and its is throwing errors all the time and does not recover. Restart the node, continues to fail. Reboot AWS, no success. Start & Stop AWS, no success. This can’t be the environment that works on the mainet as this is not enterprise ready so I’m sure I’m doing something wrong. I cant see a balance with utxo even though the Faucet confirmed it. I cant move forward to doing a simple transaction as I dont have a Node that runs or a balance.

Has anyone else had the same issues?

Please let me know.

Richard

2

Uh this sounds very strange and definitely not like literally any operator experience so far, even back in pre-launch testnets.

I run block producers and many relays in main and testnets, on different cloud and baremetal infrastructures. Even on small arm boards.

All nodes are permanently monitored at log, api and port levels and show continuous uptime.

Can you tell us more about your server setup?
Resources, operating system, logged errors, memory usage, …?

3

thanks for the response. Excuse my ignorance but where are my logs? I feel a bit stupid with this. Config is AWS EC2 cloud, 24GB 8G ram. Seems to unpack stuff very quickly. i did not install a Ubuntu server as it islinux/unix and I thought it wouldn’t need it and the instructions eluded to the fact that it wasn’t needed for AWS. My experience and skill it of a unix admin and dev in my early 20’s; im now in my 50’s (CEO) so i might have to kill -9 myself. I’m looking to see if Cardano (awesome) can be part of our business operations moving forward. I could do a teams call to show this but im aware we need to add to the knowledge base here.

I get this when i run it:
[ip-172-3:cardano.node.DnsSubscription:Error:587] [2021-01-06 09:59:33.76 UTC] [String “Application Exception: 18.197.197.49:3001 MuxError MuxBearerClosed “<socket: 23> closed when reading data, waiting on next header True””,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[ip-172-3:cardano.node.ErrorPolicy:Warning:53] [2021-01-06 09:59:33.76 UTC] [String “ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 23> closed when reading data, waiting on next header True”))) 20s 20s”,String “ErrorPolicyTrace”,String “18.197.197.49:3001”]
[ip-172-3:cardano.node.DnsSubscription:Error:603] [2021-01-06 09:59:33.78 UTC] [String “Application Exception: 3.9.80.183:3001 MuxError MuxBearerClosed “<socket: 27> closed when reading data, waiting on next header True””,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[ip-172-3:cardano.node.ErrorPolicy:Warning:53] [2021-01-06 09:59:33.78 UTC] [String “ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 27> closed when reading data, waiting on next header True”))) 20s 20s”,String “ErrorPolicyTrace”,String “3.9.80.183:3001”]
[ip-172-3:cardano.node.DnsSubscription:Notice:623] [2021-01-06 09:59:42.70 UTC] [String “Connection Attempt Start, destination 18.177.214.255:3001”,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[ip-172-3:cardano.node.DnsSubscription:Notice:624] [2021-01-06 09:59:42.72 UTC] [String “Connection Attempt Start, destination 18.197.197.49:3001”,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[ip-172-3:cardano.node.DnsSubscription:Notice:624] [2021-01-06 09:59:42.74 UTC] [String “Connection Attempt End, destination 18.197.197.49:3001 outcome: ConnectSuccess”,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[ip-172-3:cardano.node.DnsSubscription:Notice:625] [2021-01-06 09:59:42.75 UTC] [String “Connection Attempt Start, destination 54.177.182.253:3001”,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[ip-172-3:cardano.node.DnsSubscription:Notice:640] [2021-01-06 09:59:42.78 UTC] [String “Connection Attempt Start, destination 54.248.196.117:3001”,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[ip-172-3:cardano.node.DnsSubscription:Notice:641] [2021-01-06 09:59:42.80 UTC] [String “Connection Attempt Start, destination 3.9.80.183:3001”,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[ip-172-3:cardano.node.DnsSubscription:Notice:641] [2021-01-06 09:59:42.80 UTC] [String “Connection Attempt End, destination 3.9.80.183:3001 outcome: ConnectSuccessLast”,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[ip-172-3:cardano.node.ErrorPolicy:Notice:53] [2021-01-06 09:59:42.81 UTC] [String “ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = “Parallel connection cancelled”, seStack = }))) 1s”,String “ErrorPolicyTrace”,String “18.177.214.255:3001”]
[ip-172-3:cardano.node.ErrorPolicy:Notice:53] [2021-01-06 09:59:42.81 UTC] [String “ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = “Parallel connection cancelled”, seStack = }))) 1s”,String “ErrorPolicyTrace”,String “54.177.182.253:3001”]
[ip-172-3:cardano.node.ErrorPolicy:Notice:53] [2021-01-06 09:59:42.81 UTC] [String “ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = “Parallel connection cancelled”, seStack = }))) 1s”,String “ErrorPolicyTrace”,String “54.248.196.117:3001”]
[ip-172-3:cardano.node.DnsSubscription:Error:624] [2021-01-06 09:59:43.79 UTC] [String “Application Exception: 18.197.197.49:3001 MuxError MuxBearerClosed “<socket: 26> closed when reading data, waiting on next header True””,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[ip-172-3:cardano.node.ErrorPolicy:Warning:53] [2021-01-06 09:59:43.79 UTC] [String “ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 26> closed when reading data, waiting on next header True”))) 20s 20s”,String “ErrorPolicyTrace”,String “18.197.197.49:3001”]
[ip-172-3:cardano.node.DnsSubscription:Error:641] [2021-01-06 09:59:43.81 UTC] [String “Application Exception: 3.9.80.183:3001 MuxError MuxBearerClosed “<socket: 27> closed when reading data, waiting on next header True””,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[ip-172-3:cardano.node.ErrorPolicy:Warning:53] [2021-01-06 09:59:43.81 UTC] [String “ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 27> closed when reading data, waiting on next header True”))) 20s 20s”,String “ErrorPolicyTrace”,String “3.9.80.183:3001”]

In another terminal i dont see new blocks:
[ec2-user@ip-xxx-xx-xx-36 ~] cardano-cli shelley query tip --testnet-magic 1097911063 { "blockNo": 2143526, "headerHash": "50e64f0a5133007b5a85a355081589e94146308185f860b253cf775af2ce62dc", "slotNo": 13694320 } [ec2-user@ip-xxx-xx-xx-36 ~] cardano-cli shelley query tip --testnet-magic 1097911063
{
“blockNo”: 2143526,
“headerHash”: “50e64f0a5133007b5a85a355081589e94146308185f860b253cf775af2ce62dc”,
“slotNo”: 13694320
}

Why?

4

The default config (from hydra or from IOG github) will configure your logging to be at stdout (which is the extract you have pasted). The log entries shared so far do not necessarily indicate any errors that you need to worry about. That does not indicate a failure of node.

Perhaps taking a step back, what document are you following? and what exactly are you trying to achieve as part of running the node (eg: interact with API, start a stake pool, etc)? From simplistic usage perspective, you can also simply download and run Daedalus Testnet which will run the node in background and show you the status of sync/way to check balance in GUI/etc

There are quite a few versions of documentations available, and depending on your usage you could be pointed to an appropriate one.

5

You should be able to start a (relay) node like this …

docker run --detach \
    --name=relay \
    -p 3001:3001 \
    nessusio/cardano run

docker logs -f relay

There would be no build step (i.e. ghc+cabal) involved.

This also comes with a built-in gLiveView

relay-glview

and a topology updater, so you don’t have to worry about setting up a cron job for that.

The image supports x86_64 and arm64. Instructions that help you to setup Docker are here

Details about this image are here and here.

6

I am sorry but you really need to add a lot of disclaimers when suggesting to use docker images with precompiled binaries. Docker architecture in itself can have a lot of security loopholes depending on the familiarity and awareness of the end user using it (and examining base image used)

1 Like
7

With this approach I follow the general trend of what we do at IBM/RedHat. How the Docker container is built should of course be open source, like the chain of base images, like the container runtime itself.

RedHat is among others one of the major drivers in the Open Container Initiative (OCI) with all of these fine folks sharing a common understanding that the IT infrastructure (of tomorrow) will be containerized. Not all folks may share this understanding, I can tell you however that over the last years or so we’ve seen a dramatic increase in containerized payloads.

Anyhow, the build process of a Cardano node is intriguing and complicated and when you succeed, you have still not done this for various hardware architectures. This effort aims to provide the best-possible container image for Cardano. Perhaps over time, there will be folks joining this effort and of course, we will be doing stuff according to the highest quality and open source standards that are being used by Cardano itself.

Perhaps your comment aims at a specific container runtime environment (Docker) instead of using an OCI runtime. In this case, you’d be asking for disclaimers associated with the use of Docker as a runtime, which would be independent of the (docker) image that is mentioned above.

Major VPS providers like GCE, AWS have server images that use OCI runtimes. You have a good point if this is about OCI vs. Docker (and not so much about running a containerized node). Let me follow up with this and provide instructions to setup OCI on for various VPS providers.