Thrown Everything but the kitchen sink at my BP and she won't start

Ok, its me again. Ive been trying to deal with my BP for the last 3 days without much success. I have recently updates from ubuntu 20 to 22 and upgraded the nodes to 1.35.5 to 1.35.7. The relay is working fine but my BP has been stuck in “starting” status for over 20 hours. mem RSS seems to increase in the first 10 mins but stops at 6.8 gigs.

I have rebuilt the node twice

I have tried alex’s method of renaming the contents of the DB folder and restarting the node.

I copied the DB immutable folder contents from my relay to my BP but i can’t even get past starting status to sync.

I dont see any errors. journalctl -e -f -u cnode


May 03 11:44:43 $$$$$$ cnode[1796612]: Node configuration: NodeConfiguration {ncSocketConfig = SocketConfig {ncNodeIPv4Addr = Last {getLast = Just 0.0.0.0}, ncNodeIPv6Addr = Last {getLast = Nothing}, ncNodePortNumber = Last {getLast = Just 6001}, ncSocketPath = Last {getLast = Just (SocketPath {unSocketPath = "/opt/cardano/cnode/sockets/node0.socket"})}}, ncConfigFile = "/opt/cardano/cnode/files/config.json", ncTopologyFile = "/opt/cardano/cnode/files/topology.json", ncDatabaseFile = "/opt/cardano/cnode/db", ncProtocolFiles = ProtocolFilepaths {byronCertFile = Nothing, byronKeyFile = Nothing, shelleyKESFile = Just "/opt/cardano/cnode/priv/pool/Fresh/hot.skey", shelleyVRFFile = Just "/opt/cardano/cnode/priv/pool/Fresh/vrf.skey", shelleyCertFile = Just "/opt/cardano/cnode/priv/pool/Fresh/op.cert", shelleyBulkCredsFile = Nothing}, ncValidateDB = False, ncShutdownConfig = ShutdownConfig {scIPC = Nothing, scOnSyncLimit = Just NoShutdown}, ncProtocolConfig = NodeProtocolConfigurationCardano (NodeByronProtocolConfiguration {npcByronGenesisFile = "/opt/cardano/cnode/files/byron-genesis.json", npcByronGenesisFileHash = Nothing, npcByronReqNetworkMagic = RequiresNoMagic, npcByronPbftSignatureThresh = Nothing, npcByronApplicationName = ApplicationName {unApplicationName = "cardano-sl"}, npcByronApplicationVersion = 1, npcByronSupportedProtocolVersionMajor = 3, npcByronSupportedProtocolVersionMinor = 0, npcByronSupportedProtocolVersionAlt = 0}) (NodeShelleyProtocolConfiguration {npcShelleyGenesisFile = "/opt/cardano/cnode/files/shelley-genesis.json", npcShelleyGenesisFileHash = Nothing}) (NodeAlonzoProtocolConfiguration {npcAlonzoGenesisFile = "/opt/cardano/cnode/files/alonzo-genesis.json", npcAlonzoGenesisFileHash = Nothing}) (NodeHardForkProtocolConfiguration {npcTestEnableDevelopmentHardForkEras = False, npcTestShelleyHardForkAtEpoch = Nothing, npcTestShelleyHardForkAtVersion = Nothing, npcTestAllegraHardForkAtEpoch = Nothing, npcTestAllegraHardForkAtVersion = Nothing, npcTestMaryHardForkAtEpoch = Nothing, npcTestMaryHardForkAtVersion = Nothing, npcTestAlonzoHardForkAtEpoch = Nothing, npcTestAlonzoHardForkAtVersion = Nothing, npcTestBabbageHardForkAtEpoch = Nothing, npcTestBabbageHardForkAtVersion = Nothing}), ncDiffusionMode = InitiatorAndResponderDiffusionMode, ncSnapshotInterval = DefaultSnapshotInterval, ncTestEnableDevelopmentNetworkProtocols = False, ncMaxConcurrencyBulkSync = Nothing, ncMaxConcurrencyDeadline = Just 2, ncLoggingSwitch = True, ncLogMetrics = True, ncTraceConfig = TracingOnLegacy (TraceSelection {traceVerbosity = NormalVerbosity, traceAcceptPolicy = OnOff {isOn = False}, traceBlockFetchClient = OnOff {isOn = True}, traceBlockFetchDecisions = OnOff {isOn = True}, traceBlockFetchProtocol = OnOff {isOn = False}, traceBlockFetchProtocolSerialised = OnOff {isOn = True}, traceBlockFetchServer = OnOff {isOn = False}, traceBlockchainTime = OnOff {isOn = False}, traceChainDB = OnOff {isOn = True}, traceChainSyncBlockServer = OnOff {isOn = True}, traceChainSyncClient = OnOff {isOn = True}, traceChainSyncHeaderServer = OnOff {isOn = False}, traceChainSyncProtocol = OnOff {isOn = False}, traceConnectionManager = OnOff {isOn = True}, traceConnectionManagerCounters = OnOff {isOn = True}, traceConnectionManagerTransitions = OnOff {isOn = False}, traceDebugPeerSelectionInitiatorTracer = OnOff {isOn = False}, traceDebugPeerSelectionInitiatorResponderTracer = OnOff {isOn = False}, traceDiffusionInitialization = OnOff {isOn = False}, traceDnsResolver = OnOff {isOn = False}, traceDnsSubscription = OnOff {isOn = True}, traceErrorPolicy = OnOff {isOn = True}, traceForge = OnOff {isOn = True}, traceForgeStateInfo = OnOff {isOn = True}, traceHandshake = OnOff {isOn = False}, traceInboundGovernor = OnOff {isOn = False}, traceInboundGovernorCounters = OnOff {isOn = True}, traceInboundGovernorTransitions = OnOff {isOn = True}, traceIpSubscription = OnOff {isOn = False}, traceKeepAliveClient = OnOff {isOn = False}, traceLedgerPeers = OnOff {isOn = False}, traceLocalChainSyncProtocol = OnOff {isOn = False}, traceLocalConnectionManager = OnOff {isOn = False}, traceLocalErrorPolicy = OnOff {isOn = True}, traceLocalHandshake = OnOff {isOn = False}, traceLocalInboundGovernor = OnOff {isOn = False}, traceLocalMux = OnOff {isOn = False}, traceLocalRootPeers = OnOff {isOn = False}, traceLocalServer = OnOff {isOn = False}, traceLocalStateQueryProtocol = OnOff {isOn = False}, traceLocalTxMonitorProtocol = OnOff {isOn = False}, traceLocalTxSubmissionProtocol = OnOff {isOn = False}, traceLocalTxSubmissionServer = OnOff {isOn = False}, traceMempool = OnOff {isOn = True}, traceMux = OnOff {isOn = False}, tracePeerSelection = OnOff {isOn = False}, tracePeerSelectionCounters = OnOff {isOn = True}, tracePeerSelectionActions = OnOff {isOn = False}, tracePublicRootPeers = OnOff {isOn = False}, traceServer = OnOff {isOn = False}, traceTxInbound = OnOff {isOn = False}, traceTxOutbound = OnOff {isOn = False}, traceTxSubmissionProtocol = OnOff {isOn = False}, traceTxSubmission2Protocol = OnOff {isOn = False}}), ncTraceForwardSocket = Nothing, ncMaybeMempoolCapacityOverride = Nothing, ncProtocolIdleTimeout = 5s, ncTimeWaitTimeout = 60s, ncAcceptedConnectionsLimit = AcceptedConnectionsLimit {acceptedConnectionsHardLimit = 512, acceptedConnectionsSoftLimit = 384, acceptedConnectionsDelay = 5s}, ncTargetNumberOfRootPeers = 100, ncTargetNumberOfKnownPeers = 100, ncTargetNumberOfEstablishedPeers = 50, ncTargetNumberOfActivePeers = 20, ncEnableP2P = DisabledP2PMode}
May 03 11:44:48 $$$$$$ cnode[1806354]:   Sending SIGINT to cardano-node process..
May 03 11:47:36 $$$$$$ cnode[1806562]: INFO: Cleaned-up stale socket file
May 03 11:47:36 $$$$$$ cnode[1806678]: Listening on http://127.0.0.1:12798

topology.json 
{
  "Producers": [
    {
      "addr": "192.168.0.145",
      "port": 6000,
      "valency": 1
    }
  ]
}




bashrc 
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
  if [ -f /usr/share/bash-completion/bash_completion ]; then
    . /usr/share/bash-completion/bash_completion
  elif [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
  fi
fi
[ -f "/home/user/.ghcup/env" ] && source "/home/user/.ghcup/env" # ghcup-env
export CNODE_HOME=/opt/cardano/cnode
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
export PATH="${HOME}/.local/bin:${PATH}"
export CARDANO_NODE_SOCKET_PATH=“/opt/cardano/cnode/sockets/node0.socket”

BP has 512 Gb ssd and 32 gigs of ram with a 8gb swap 
![freem|761x71](upload://h1SCuVbVQjUlDcDKrZdQC6hJQbI.png)
![image|918x632](upload://kZI2c1syaslFcidJq4RXyxznqL7.png)
![image|663x522](upload://w9KxnnkSWrV4ofc2AcDc43Hzz07.png)

anyone have any ideas? should i try to downgrade? or just nuke this thing from space and start from a fresh install of ubuntu?

Have you worked through any of this? Upgrade Ubuntu OS 20.0.4.5 TLS -> 22.04 Jammy / with running Cardano Node 1.35.5

2 Likes

ill have a look. thanks

Try to add inside topology file also the IOHK relays, see if its starting… perhaps there is an issue connecting to ur relay

Thanks Alex, i actually found a solution of yours where i just delete the volatile and ledger folders. and it started syncing after a restart of the node. Althought… syncing all the way from 0%. even tho i had copied the immutable folder from my fully synced relay. should i try copying the folder again?

Yes, or u can download the latest db snapshot

I tried moving my immutable folder from relay to BP. its sooo close but seem it hasn’t moved at all in the last few hours. Should i just try to restart it or add those IOHK address into topology? or should I just leave it be?

BP
image
Relay
image

Thanks Alex, after coping my DB folder over and still not being able to sync. I tried the snapshot and it stuck! up and running now!

1 Like