Mainnet 1.34.1 first sync - cardano-cli query tip --mainnet

[mattmco]

I left the server syncing over the weekend. Today the the hard drive is over 90GB, am hoping that’s close to finishing ?

It was unresponsive to SSH so I had to do a restart.

Now ./gLiveview is still starting. So I wanted to check the progress manually.

cardano-cli query tip --mainnet

gives the error about socket path ?

Am see this in systemctl status is this anything to be concerned about and or requires any further action ?

WARN: A prior running Cardano node was not cleanly shutdown,
socket file still exists. Cleaning up.

Two questions :

What do I need to specify to check the tip using cardano-cli ?

Will gLiveview eventually start working or does that need an update or something ?

[mattmco]

ps -ax produces this

1021 ? Sl 0:00 /home/matt/.cabal/bin/cardano-submit-api --config /opt/cardano/cnode/files/config.json --testnet-magic 764824073 --socket-path /opt/cardano/cnode/sockets/node0.socket --listen-address 127.0.0.1 --port 8090
1035 ? Rl 38:39 /home/matt/.cabal/bin/cardano-node run --topology /opt/cardano/cnode/files/topology.json --config /opt/cardano/cnode/files/config.json --database-path /opt/cardano/cnode/db --socket-path /opt/cardano/cnode/sockets/node0.s

[mattmco]

Does the above look right ?

Do I need to run this kind of command ?

export CARDANO_NODE_SOCKET_PATH=/opt/cardano/cnode/sockets/node0.socket

before

cardano-cli query tip --mainnet

[HeptaSean]

@mattmco

I left the server syncing over the weekend. Today the the hard drive is over 90GB, am hoping that’s close to finishing ?

Should be, at the moment. But it will continue to grow several GiB per month.

@mattmco

Do I need to run this kind of command ?

export CARDANO_NODE_SOCKET_PATH=/opt/cardano/cnode/sockets/node0.socket

Yes, exactly that. cardano-cli needs to know, where to find the socket to communicate with cardano-node.

Usually, all guides tell you to put that in .bashrc, so that it gets executed automatically for your shell (if you are using bash and not some other shell) and you don’t have to do it again each time.

[mattmco]

@mattmco

cardano-cli query tip --mainnet

Hi Sean

Still not working for me I have added this line

export CNODE_HOME=/opt/cardano/cnode
export CARDANO_NODE_SOCKET_PATH=/opt/cardano/cnode/sockets/node0.socket

$CARDANO_NODE_SOCKET_PATH

-bash: /opt/cardano/cnode/sockets/node0.socket: No such file or directory

The file does not exist how is the current process running then ?

035 ? Rl 38:39 /home/matt/.cabal/bin/cardano-node run --topology /opt/cardano/cnode/files/topology.json --config /opt/cardano/cnode/files/config.json --database-path /opt/cardano/cnode/db --socket-path /opt/cardano/cnode/sockets/node0.socket

[HeptaSean]

@mattmco

–socket-path /opt/cardano/cnode/sockets/node0.s

Is that just cut off or does have you told your cardano-node to use /opt/cardano/cnode/sockets/node0.s instead of /opt/cardano/cnode/sockets/node0.socket?
You can just look in that directory, what the filename of the created socket is.
Of course, you have to use the same.

@mattmco

export CARDANO_NODE_SOCKET_PATH=/opt/cardano/cnode/sockets/node0.socket

$CARDANO_NODE_SOCKET_PATH

-bash: /opt/cardano/cnode/sockets/node0.socket: No such file or directory

Why do you put $CARDANO_NODE_SOCKET_PATH on a line of its own? Then, it tries to execute /opt/cardano/cnode/sockets/node0.socket (since that’s the content you have just put into that variable). And it’s a socket. It is not supposed to be executable.

[mattmco]

@HeptaSean

/opt/cardano/cnode/sockets/node0.s instead of /opt/cardano/cnode/sockets/node0.socket?

Yes it was shortened by the screen see I amended as you typed this response

[mattmco]

@HeptaSean

Why do you put $CARDANO_NODE_SOCKET_PAT

To test if it existed

Apologies it should be

echo $CARDANO_NODE_SOCKET_PATH

[mattmco]

cardano-cli query tip --mainnet

cardano-cli: Network.Socket.connect: <socket: 11>: does not exist (No such file or directory)

Second Question to help resolve this.

Is there a way to get gLiveview working again its forever in starting mode ?

Any updates ?

[mattmco]

@mattmco

cardano-cli: Network.Socket.connect: <socket: 11>: does not exist

Also could it be related to earlier nodes being killed off looking at this post

Socket goes missing

Thanks for the help guys! It was very strange, it was fixed after i rebooted my server to encourage it to create socket in the correct directory. Might be related to this, because something happened earlier where I might have prematurely killed my node off:

[mattmco]

Is there Anyone out there that can help ?

[HeptaSean]

@mattmco

cardano-cli query tip --mainnet

cardano-cli: Network.Socket.connect: <socket: 11>: does not exist (No such file or directory)

Second Question to help resolve this.

Is there a way to get gLiveview working again its forever in starting mode ?

During the first phase of start-up, both – cardano-cli and gLiveView – will not work. Might be that you just have to wait some minutes.

Does journalctl tell you errors the node runs into?

[mattmco]

@HeptaSean

During the first phase of start-up, both – cardano-cli and gLiveView – will not work. Might be that you just have to wait some minutes.

Could be its just I have waited days before and it didn’t change on other nodes.

So have a look at this can you see anything ?

Aug 08 08:51:10 node1vlv systemd[1]: Started Cardano Node.

Aug 08 08:51:14 node1vlv cnode[505]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.

Aug 08 08:51:16 node1vlv cnode[1035]: Listening on http://127.0.0.1:12798

Aug 08 10:02:36 node1vlv systemd[1]: Stopping Cardano Node…

Aug 08 10:02:36 node1vlv systemd[1]: cnode.service: Control process exited, code=exited, status=1/FAILURE

Aug 08 10:02:36 node1vlv cnode[1035]: Shutting down…

Aug 08 10:02:36 node1vlv cnode[1035]: Node configuration: NodeConfiguration {ncSocketConfig = SocketConfig {ncNodeIPv4Addr = Last {getLast = Just 0.0.0.0}, ncNodeIPv6Addr = Last {getLast = Nothing}, ncNodePortNumber = Last {getLast = Just 6707}, ncSocketPath = Last {getLast = Just “/opt/cardano/cnode/sockets/node0.socket”}}, ncConfigFile = “/opt/cardano/cnode/files/config.json”, ncTopologyFile = “/opt/cardano/cnode/files/topology.json”, ncDatabaseFile = “/opt/cardano/cnode/db”, ncProtocolFiles = ProtocolFilepaths {byronCertFile = Nothing, byronKeyFile = Nothing, shelleyKESFile = Nothing, shelleyVRFFile = Nothing, shelleyCertFile = Nothing, shelleyBulkCredsFile = Nothing}, ncValidateDB = False, ncShutdownConfig = ShutdownConfig {scIPC = Nothing, scOnSlotSynced = Just NoMaxSlotNo}, ncProtocolConfig = NodeProtocolConfigurationCardano (NodeByronProtocolConfiguration {npcByronGenesisFile = “/opt/cardano/cnode/files/byron-genesis.json”, npcByronGenesisFileHash = Nothing, npcByronReqNetworkMagic = RequiresNoMagic, npcByronPbftSignatureThresh = Nothing, npcByronApplicationName = ApplicationName {unApplicationName = “cardano-sl”}, npcByronApplicationVersion = 1, npcByronSupportedProtocolVersionMajor = 3, npcByronSupportedProtocolVersionMinor = 0, npcByronSupportedProtocolVersionAlt = 0}) (NodeShelleyProtocolConfiguration {npcShelleyGenesisFile = “/opt/cardano/cnode/files/shelley-genesis.json”, npcShelleyGenesisFileHash = Nothing}) (NodeAlonzoProtocolConfiguration {npcAlonzoGenesisFile = “/opt/cardano/cnode/files/alonzo-genesis.json”, npcAlonzoGenesisFileHash = Nothing}) (NodeHardForkProtocolConfiguration {npcTestEnableDevelopmentHardForkEras = False, npcTestShelleyHardForkAtEpoch = Nothing, npcTestShelleyHardForkAtVersion = Nothing, npcTestAllegraHardForkAtEpoch = Nothing, npcTestAllegraHardForkAtVersion = Nothing, npcTestMaryHardForkAtEpoch = Nothing, npcTestMaryHardForkAtVersion = Nothing, npcTestAlonzoHardForkAtEpoch = Nothing, npcTestAlonzoHardForkAtVersion = Nothing}), ncDiffusionMode = InitiatorAndResponderDiffusionMode, ncSnapshotInterval = DefaultSnapshotInterval, ncTestEnableDevelopmentNetworkProtocols = False, ncMaxConcurrencyBulkSync = Nothing, ncMaxConcurrencyDeadline = Just 2, ncLoggingSwitch = True, ncLogMetrics = True, ncTraceConfig = TracingOnLegacy (TraceSelection {traceVerbosity = NormalVerbosity, traceAcceptPolicy = OnOff {isOn = False}, traceBlockFetchClient = OnOff {isOn = True}, traceBlockFetchDecisions = OnOff {isOn = True}, traceBlockFetchProtocol = OnOff {isOn = False}, traceBlockFetchProtocolSerialised = OnOff {isOn = True}, traceBlockFetchServer = OnOff {isOn = False}, traceBlockchainTime = OnOff {isOn = False}, traceChainDB = OnOff {isOn = True}, traceChainSyncBlockServer = OnOff {isOn = True}, traceChainSyncClient = OnOff {isOn = False}, traceChainSyncHeaderServer = OnOff {isOn = False}, traceChainSyncProtocol = OnOff {isOn = False}, traceConnectionManager = OnOff {isOn = True}, traceConnectionManagerCounters = OnOff {isOn = True}, traceConnectionManagerTransitions = OnOff {isOn = False}, traceDebugPeerSelectionInitiatorTracer = OnOff {isOn = False}, traceDebugPeerSelectionInitiatorResponderTracer = OnOff {isOn = False}, traceDiffusionInitialization = OnOff {isOn = False}, traceDnsResolver = OnOff {isOn = False}, traceDnsSubscription = OnOff {isOn = True}, traceErrorPolicy = OnOff {isOn = True}, traceForge = OnOff {isOn = True}, traceForgeStateInfo = OnOff {isOn = True}, traceHandshake = OnOff {isOn = False}, traceInboundGovernor = OnOff {isOn = False}, traceInboundGovernorCounters = OnOff {isOn = True}, traceInboundGovernorTransitions = OnOff {isOn = True}, traceIpSubscription = OnOff {isOn = False}, traceKeepAliveClient = OnOff {isOn = False}, traceLedgerPeers = OnOff {isOn = False}, traceLocalChainSyncProtocol = OnOff {isOn = False}, traceLocalConnectionManager = OnOff {isOn = False}, traceLocalErrorPolicy = OnOff {isOn = True}, traceLocalHandshake = OnOff {isOn = False}, traceLocalInboundGovernor = OnOff {isOn = False}, traceLocalMux = OnOff {isOn = False}, traceLocalRootPeers = OnOff {isOn = False}, traceLocalServer = OnOff {isOn = False}, traceLocalStateQueryProtocol = OnOff {isOn = False}, traceLocalTxMonitorProtocol = OnOff {isOn = False}, traceLocalTxSubmissionProtocol = OnOff {isOn = False}, traceLocalTxSubmissionServer = OnOff {isOn = False}, traceMempool = OnOff {isOn = False}, traceMux = OnOff {isOn = False}, tracePeerSelection = OnOff {isOn = False}, tracePeerSelectionCounters = OnOff {isOn = True}, tracePeerSelectionActions = OnOff {isOn = False}, tracePublicRootPeers = OnOff {isOn = False}, traceServer = OnOff {isOn = False}, traceTxInbound = OnOff {isOn = False}, traceTxOutbound = OnOff {isOn = False}, traceTxSubmissionProtocol = OnOff {isOn = False}, traceTxSubmission2Protocol = OnOff {isOn = False}}), ncMaybeMempoolCapacityOverride = Nothing, ncProtocolIdleTimeout = 5s, ncTimeWaitTimeout = 60s, ncAcceptedConnectionsLimit = AcceptedConnectionsLimit {acceptedConnectionsHardLimit = 512, acceptedConnectionsSoftLimit = 384, acceptedConnectionsDelay = 5s}, ncTargetNumberOfRootPeers = 100, ncTargetNumberOfKnownPeers = 100, ncTargetNumberOfEstablishedPeers = 50, ncTargetNumberOfActivePeers = 20, ncEnableP2P = DisabledP2PMode}

Aug 08 10:02:37 node1vlv systemd[1]: cnode.service: Failed with result ‘exit-code’.

Aug 08 10:02:37 node1vlv systemd[1]: Stopped Cardano Node.

– Reboot –

Aug 08 10:04:08 node1vlv systemd[1]: Started Cardano Node.

Aug 08 10:04:14 node1vlv cnode[924]: Listening on http://127.0.0.1:12798

[mattmco]

So basically I have no way of checking the sync is that right ?

cardano-cli query tip --mainnet ==> errors on socket

gLiveview ===> stuck on starting …

Any other ideas / ways to check the sync / or any imaginative suggestions ???

[HeptaSean]

@mattmco

Aug 08 10:04:08 node1vlv systemd[1]: Started Cardano Node.

Aug 08 10:04:14 node1vlv cnode[924]: Listening on http://127.0.0.1:12798

That looks like it started. It could take ten minutes or so, but not half an hour until the socket is created.

Also: There should be more messages than those. Have you tried journalctl -bu cnode?

[mattmco]

@HeptaSean

but not half an hour until the socket is created.

Exactly right somethings up !

/opt/cardano/cnode/scripts$ journalctl -b -u cnode

– Logs begin at Thu 2022-07-14 14:14:41 UTC, end at Mon 2022-08-08 10:46:57 UTC. –

Aug 08 10:04:08 node1vlv systemd[1]: Started Cardano Node.

Aug 08 10:04:14 node1vlv cnode[924]: Listening on http://127.0.0.1:12798

[HeptaSean]

This looks like it helped in a quite similar situation:

Block Producer does not start after 1.34.1 Update

ok, you will need to perform a WA: coincashew or cntools? cntools… wait 1 min sometimes, when you stop/start/restart the node often you will hit this issue and you will need to perform the steps stop the node sudo systemctl stop cnode rename ledger, immutable and volatile folders cd $CNODE_HOME/db ls -l mv immutable imm mv ledger led mv volatile vol ls -l start/stop the node sudo systemctl start cnode stop the node after 10 sec sudo systemctl stop cnode you should also see now the …

[mattmco]

Hi thanks not sure which bit helped the renaming of the DB files or the memory swap I tried the db files first no difference.

Stuck on starting worth doing the memory swap ?

[mattmco]

As there is no clear answer or any way I can see here to troubleshoot.

I will shut this node down as its costs money to run and our project is already behind.

If you do have any different to add I’ll check back in this thread.

Currently trying to run 1.34.1 in Docker instead on another VM.

[rdlrt]

It can take hour or so if a node crashed (eg: due to OOM) as it would go through verifying ledger-db), the evidence would be in node logs in that case.