After updating to 1.26.1 / 1.26.2 my BP won't start anymore

niki_niki_airmax · 26 April 2021 10:04

Hello,

so after updating to 1.26.1 and later to 1.26.2 I am not able to start my block producer anymore. Temporary fix was to reset the database, but after a couple of hours (about 10) it crashed and did not start back up again. The error I am getting is:

Apr 26 09:52:36 aa bash[5419]: /home/cardano/cardano-my-node/startBlockProducingNode.sh: line 11: 5420 Killed cardano-node run --topology {TOPOLOGY} --database-path {DB_PATH} --socket-path {SOCKET_PATH} --host-addr {HOSTADDR} --port {PORT} --config {CONFIG} --shelley-kes-key {KES} --shelley-vrf-key {VRF} --shelley-operational-certificate ${CERT}

Anybody has the same issue/fix?

Thanks,

Matic

Alexd1985 · 26 April 2021 10:37

I can tell u that once u updated the nodes to a higher version than 1.25.1 the nodes need to re-sync the db (first time when it start). This process can take a while (from 40 minutes to few hours)

What is the status on glive?

niki_niki_airmax · 26 April 2021 10:43

Yep, I know they have to resync, but that is not the problem. As you can see from my post, as soon as the block producer replays the block progress, it fires up the start script for the block producer which then effectively crashes the process.

Alexd1985 · 26 April 2021 10:59

journalctl -e -f -u cardano-node.service

niki_niki_airmax · 26 April 2021 11:14

Apr 26 10:51:29 maticbp bash[5897]: /home/cardano/cardano-my-node/startBlockProducingNode.sh: line 11: 5898 Killed cardano-node run --topology {TOPOLOGY} --database-path {DB_PATH} --socket-path {SOCKET_PATH} --host-addr {HOSTADDR} --port {PORT} --config {CONFIG} --shelley-kes-key {KES} --shelley-vrf-key {VRF} --shelley-operational-certificate {CERT} Apr 26 10:51:35 maticbp bash[5936]: Listening on http://127.0.0.1:12798 Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.networkMagic:Notice:5] [2021-04-26 10:51:35.78 UTC] NetworkMagic 764824073 Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.protocol:Notice:5] [2021-04-26 10:51:35.78 UTC] Byron; Shelley Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.version:Notice:5] [2021-04-26 10:51:35.78 UTC] 1.26.2 Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.commit:Notice:5] [2021-04-26 10:51:35.78 UTC] 3531289c9f79eab7ac5d3272ce6e6821504fec4c Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.nodeStartTime:Notice:5] [2021-04-26 10:51:35.78 UTC] 2021-04-26 10:51:35.788640424 UTC Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.systemStartTime:Notice:5] [2021-04-26 10:51:35.78 UTC] 2017-09-23 21:44:51 UTC Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.slotLengthByron:Notice:5] [2021-04-26 10:51:35.78 UTC] 20s Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.epochLengthByron:Notice:5] [2021-04-26 10:51:35.78 UTC] 21600 Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.slotLengthShelley:Notice:5] [2021-04-26 10:51:35.78 UTC] 1s Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.epochLengthShelley:Notice:5] [2021-04-26 10:51:35.78 UTC] 432000 Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.slotsPerKESPeriodShelley:Notice:5] [2021-04-26 10:51:35.78 UTC] 129600 Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.slotLengthAllegra:Notice:5] [2021-04-26 10:51:35.78 UTC] 1s Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.epochLengthAllegra:Notice:5] [2021-04-26 10:51:35.78 UTC] 432000 Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.slotsPerKESPeriodAllegra:Notice:5] [2021-04-26 10:51:35.78 UTC] 129600 Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.slotLengthMary:Notice:5] [2021-04-26 10:51:35.78 UTC] 1s Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.epochLengthMary:Notice:5] [2021-04-26 10:51:35.78 UTC] 432000 Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.basicInfo.slotsPerKESPeriodMary:Notice:5] [2021-04-26 10:51:35.78 UTC] 129600 Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.addresses:Notice:5] [2021-04-26 10:51:35.78 UTC] [SocketInfo 0.0.0.0:6000] Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.diffusion-mode:Notice:5] [2021-04-26 10:51:35.78 UTC] InitiatorAndResponderDiffusionMode Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.dns-producers:Notice:5] [2021-04-26 10:51:35.78 UTC] [DnsSubscriptionTarget {dstDomain = "relay", dstPort = 1111111, dstValency = 1}] Apr 26 10:51:36 maticbp bash[5936]: [maticbp:cardano.node.ip-producers:Notice:5] [2021-04-26 10:51:35.78 UTC] IPSubscriptionTarget {ispIps = [], ispValency = 0} Apr 26 10:58:58 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:58:58.09 UTC] Opened imm db with immutable tip at fdece46415f52bc132db524027cd8213706adbb7e3a2d5d7f91e6cba2a1bdf8d at slot 27777881 and chunk 1286 Apr 26 10:59:00 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:00.57 UTC] Opened vol db Apr 26 10:59:07 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Error:5] [2021-04-26 10:59:07.61 UTC] Invalid snapshot DiskSnapshot {dsNumber = 27823598, dsSuffix = Nothing}InitFailureRead (ReadFailed (DeserialiseFailure 164577270 "end of input")) Apr 26 10:59:28 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:28.78 UTC] Replaying ledger from snapshot DiskSnapshot {dsNumber = 27733554, dsSuffix = Nothing} at 4bcb2cbcbeab69615e4c0a8b3633be03c8c34e5ba270c6025e262b0f37d4fcd6 at slot 27733554 Apr 26 10:59:29 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:29.06 UTC] Replayed block: slot SlotNo 27733614 of At (SlotNo 27777881) Apr 26 10:59:29 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:29.11 UTC] before next, messages elided = 27733625 Apr 26 10:59:29 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:29.11 UTC] Replayed block: slot SlotNo 27734342 of At (SlotNo 27777881) Apr 26 10:59:29 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:29.11 UTC] Replayed block: slot SlotNo 27734352 of At (SlotNo 27777881) Apr 26 10:59:29 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:29.46 UTC] block replay progress (%) = 99.9 Apr 26 10:59:30 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:30.21 UTC] block replay progress (%) = 99.9 Apr 26 10:59:30 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:30.29 UTC] before next, messages elided = 27734432 Apr 26 10:59:30 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:30.29 UTC] Replayed block: slot SlotNo 27755968 of At (SlotNo 27777881) Apr 26 10:59:30 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:30.29 UTC] Replayed block: slot SlotNo 27755980 of At (SlotNo 27777881) Apr 26 10:59:30 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:30.92 UTC] block replay progress (%) = 100.0 Apr 26 10:59:31 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:31.25 UTC] before next, messages elided = 27755999 Apr 26 10:59:31 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:31.25 UTC] Replayed block: slot SlotNo 27777562 of At (SlotNo 27777881) Apr 26 10:59:31 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:31.27 UTC] Replayed block: slot SlotNo 27777588 of At (SlotNo 27777881) Apr 26 10:59:31 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:31.28 UTC] before next, messages elided = 27777600 Apr 26 10:59:31 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:31.28 UTC] Replayed block: slot SlotNo 27777881 of At (SlotNo 27777881) Apr 26 10:59:31 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 10:59:31.28 UTC] Opened lgr db Apr 26 11:00:15 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Info:5] [2021-04-26 11:00:15.52 UTC] Opened db with immutable tip at fdece46415f52bc132db524027cd8213706adbb7e3a2d5d7f91e6cba2a1bdf8d at slot 27777881 and tip d7695ab6d0716e437e83419d304b6c05e740538fdb2d669cdb37e965df51f516 at slot 27867982 Apr 26 11:00:15 maticbp bash[5936]: [maticbp:cardano.node.LeadershipCheck:Info:558] [2021-04-26 11:00:15.62 UTC] {"kind":"TraceStartLeadershipCheck","slot":27868524,"credentials":"Cardano"} Apr 26 11:00:15 maticbp bash[5936]: [maticbp:cardano.node.Forge:Info:558] [2021-04-26 11:00:15.74 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 2.7868524e7)])),("credentials",String "Cardano")] Apr 26 11:00:15 maticbp bash[5936]: [maticbp:cardano.node.DnsSubscription:Notice:582] [2021-04-26 11:00:15.80 UTC] Domain: "relay" Connection Attempt Start, destination 91173:110 Apr 26 11:00:15 maticbp bash[5936]: [maticbp:cardano.node.DnsSubscription:Notice:582] [2021-04-26 11:00:15.86 UTC] Domain: "relay" Connection Attempt End, destination 11:1110 outcome: ConnectSuccessLast Apr 26 11:00:16 maticbp bash[5936]: [maticbp:cardano.node.LeadershipCheck:Info:558] [2021-04-26 11:00:16.01 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":4.92091e-2,"slot":27868525,"delegMapSize":465154,"utxoSize":2014752,"credentials":"Cardano"} Apr 26 11:00:16 maticbp bash[5936]: [maticbp:cardano.node.Forge:Info:558] [2021-04-26 11:00:16.09 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 2.7868525e7)])),("credentials",String "Cardano")] Apr 26 11:00:16 maticbp bash[5936]: [maticbp:cardano.node.BlockFetchDecision:Info:560] [2021-04-26 11:00:16.26 UTC] [TraceLabelPeer (ConnectionId {localAddress = bp, remoteAddress = relay}) (Right [At (Block {blockPointSlot = SlotNo 27868004, blockPointHash = 08f8e646a7000d0750b7cf11d4e38c32aa0d0eb1adefde0fce79f61b607d6a87})])] Apr 26 11:00:16 maticbp bash[5936]: [maticbp:cardano.node.BlockFetchDecision:Info:560] [2021-04-26 11:00:16.34 UTC] [TraceLabelPeer (ConnectionId {localAddress = bp, remoteAddress = relay}) (Right [At (Block {blockPointSlot = SlotNo 27868015, blockPointHash = 1e7f8ecbeea979ba962bd2a286f4f1ac34f192d101d1c1a2c00dc17b30b8da73})])] Apr 26 11:00:16 maticbp bash[5936]: [maticbp:cardano.node.BlockFetchDecision:Info:560] [2021-04-26 11:00:16.40 UTC] [TraceLabelPeer (ConnectionId {localAddress = bp, remoteAddress = relay}) (Right [At (Block {blockPointSlot = SlotNo 27868020, blockPointHash = 60bf42304253318e2bae6e2e89a0ee5c95197791510538164fe0b95a3d72e34b})])] Apr 26 11:00:16 maticbp bash[5936]: [maticbp:cardano.node.BlockFetchDecision:Info:560] [2021-04-26 11:00:16.54 UTC] [TraceLabelPeer (ConnectionId {localAddress = bp, remoteAddress = relay}) (Right [At (Block {blockPointSlot = SlotNo 27868025, blockPointHash = 54e9c3c5ec6166f63901e59a9e04acb686577ed3ae02752536efac6ff94b1b15}),At (Block {blockPointSlot = SlotNo 27868072, blockPointHash = f07fa360a234446882f0045cb50488995466e4cf30169db657a8f80632265ea1}),At (Block {blockPointSlot = SlotNo 27868073, blockPointHash = c380f59272276cd7e3e5d58a0aae60ec21c8f0f6f7eff1c766d1e7b759f7b8fb}),At (Block {blockPointSlot = SlotNo 27868077, blockPointHash = cb2a1565cab4617350ed144802cffe6620371522332ab347a5e5ef90a3f5acc2}),At (Block {blockPointSlot = SlotNo 27868079, blockPointHash = 3940d9b4b41d5e2e135ea1a91eca92977e44e01cdc3945c9e645ca0c77674b86}),At (Block {blockPointSlot = SlotNo 27868129, blockPointHash = eb4ad2ea24e8a392ef2377cd3693f6957a1d0520c65718f518d13d3ca42772df}),At (Block {blockPointSlot = SlotNo 27868146, blockPointHash = 5d944b9c58d83eef0f07b65bb84b000310d47927a715f78a1c6c42328eda3d0d}),At (Block {blockPointSlot = SlotNo 27868147, blockPointHash = 415beee6af487c3571990f225be681fb79b650045be032b752f772c501318d6f}),At (Block {blockPointSlot = SlotNo 27868172, blockPointHash = 289c89d5ebb7c483cf077e0ec39c0fc1e80e6b6f1ea32d40d8ff801a0419b542}),At (Block {blockPointSlot = SlotNo 27868178, blockPointHash = af56097f88bfc62d195facbdbaaace5caeefc915113e0868ded473679c7af9a4}),At (Block {blockPointSlot = SlotNo 27868188, blockPointHash = c31d6534ca950e3e2d317b9c66cb697f91ecbb0a19211807f8465b8fcc9f1a46}),At (Block {blockPointSlot = SlotNo 27868189, blockPointHash = ed5f0f8adaf039a3aa8c96fe3831b0cc0f0555de243fbf9712be6150a985ee65})])] Apr 26 11:00:16 maticbp bash[5936]: [maticbp:cardano.node.BlockFetchDecision:Info:560] [2021-04-26 11:00:16.54 UTC] [TraceLabelPeer (ConnectionId {localAddress = bp, remoteAddress = relay}) (Right [At (Block {blockPointSlot = SlotNo 27868194, blockPointHash = 80a2675b15c0c6a26a33f1a949f8ed289dee02f072619c3a7ed761e7b9d5a2d1})])] Apr 26 11:00:16 maticbp bash[5936]: [maticbp:cardano.node.BlockFetchDecision:Info:560] [2021-04-26 11:00:16.63 UTC] [TraceLabelPeer (ConnectionId {localAddress = bp, remoteAddress = relay}) (Right [At (Block {blockPointSlot = SlotNo 27868202, blockPointHash = a9d93f697e12fbc55d19f49eed789a92e91f889d41fee7742a56e77ae4aa6056}),At (Block {blockPointSlot = SlotNo 27868274, blockPointHash = 0a4fb72c8f7ed8bc191530a373472486918d14c216dcf3d4e3a2a439eb20dab3}),At (Block {blockPointSlot = SlotNo 27868290, blockPointHash = b839f9efd1e48361740ae5e1e6c08f9340fc688a152bb5fa9ada61f3a2f8b5b3}),At (Block {blockPointSlot = SlotNo 27868314, blockPointHash = bfb49a60464d0d32cd10ca30c2198f54263dfe6d5b63dca30542e136023c5199}),At (Block {blockPointSlot = SlotNo 27868326, blockPointHash = 3d217419ce44e6b5dc27e93ac609e26ade9a81bd2d917ba745700fdc79037249}),At (Block {blockPointSlot = SlotNo 27868344, blockPointHash = 56f4d4b8408be4c16bf8a670cf4bc948c7a06a178e8854feda2e831f3a4dea39})])] Apr 26 11:00:16 maticbp bash[5936]: [maticbp:cardano.node.BlockFetchDecision:Info:560] [2021-04-26 11:00:16.64 UTC] [TraceLabelPeer (ConnectionId {localAddress = bp, remoteAddress = relay}) (Right [At (Block {blockPointSlot = SlotNo 27868349, blockPointHash = eef1c9c9438a36049f50d6d85c158e7855e1ebcf0f4389f9a9da8ba0a7974e38}),At (Block {blockPointSlot = SlotNo 27868352, blockPointHash = 99e9785f97dd60d8e86aa7e5f96f3b664d499a2f52451d1f46d17423490e91f4}),At (Block {blockPointSlot = SlotNo 27868372, blockPointHash = 37511e42152450c9eebc7b3b2cc84e2a2e3d30257b72e86c047afdc616799fb0}),At (Block {blockPointSlot = SlotNo 27868374, blockPointHash = d8d25d57e0b210cb45f4ec6f068da47f22a8c7efcfd545791c67909237f0354a})])] Apr 26 11:00:17 maticbp bash[5936]: [maticbp:cardano.node.BlockFetchDecision:Info:560] [2021-04-26 11:00:16.66 UTC] [TraceLabelPeer (ConnectionId {localAddress = bp, remoteAddress = relay}) (Right [At (Block {blockPointSlot = SlotNo 27868393, blockPointHash = 3cd9ad71b310e1090611e9d20555ba29d7cc4a80e515e22d1496862e63f29142}),At (Block {blockPointSlot = SlotNo 27868417, blockPointHash = f08cdbd2944ab0a068b94be3cf13621008ac5681a8b31b53baf24e6874477763}),At (Block {blockPointSlot = SlotNo 27868424, blockPointHash = 1631478ad98fedc321e7fb5b2815358d19c2a90804af51f310d2753dd7cf40aa})])] Apr 26 11:00:17 maticbp bash[5936]: [maticbp:cardano.node.BlockFetchDecision:Info:560] [2021-04-26 11:00:16.67 UTC] [TraceLabelPeer (ConnectionId {localAddress = bp, remoteAddress = relay}) (Right [At (Block {blockPointSlot = SlotNo 27868443, blockPointHash = beea65375de5eda6eb9a7552974e43cbab097d025609488461ce9e4b8156c727}),At (Block {blockPointSlot = SlotNo 27868467, blockPointHash = 104e6699d451600e087fe8292ed377509917212f516d48ec5318679bd6316e97})])] Apr 26 11:00:17 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Notice:548] [2021-04-26 11:00:16.72 UTC] Chain extended, new tip: 08f8e646a7000d0750b7cf11d4e38c32aa0d0eb1adefde0fce79f61b607d6a87 at slot 27868004 Apr 26 11:00:17 maticbp bash[5936]: [maticbp:cardano.node.LeadershipCheck:Info:558] [2021-04-26 11:00:17.01 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":4.9187798e-2,"slot":27868526,"delegMapSize":465154,"utxoSize":2014755,"credentials":"Cardano"} Apr 26 11:00:17 maticbp bash[5936]: [maticbp:cardano.node.Forge:Info:558] [2021-04-26 11:00:17.33 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 2.7868526e7)])),("credentials",String "Cardano")] Apr 26 11:00:18 maticbp bash[5936]: [maticbp:cardano.node.LeadershipCheck:Info:558] [2021-04-26 11:00:18.00 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":4.921583e-2,"slot":27868527,"delegMapSize":465155,"utxoSize":2014760,"credentials":"Cardano"} Apr 26 11:00:18 maticbp bash[5936]: [maticbp:cardano.node.Forge:Info:558] [2021-04-26 11:00:18.12 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 2.7868527e7)])),("credentials",String "Cardano")] Apr 26 11:00:18 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Notice:548] [2021-04-26 11:00:18.15 UTC] Chain extended, new tip: 54e9c3c5ec6166f63901e59a9e04acb686577ed3ae02752536efac6ff94b1b15 at slot 27868025 Apr 26 11:00:19 maticbp bash[5936]: [maticbp:cardano.node.LeadershipCheck:Info:558] [2021-04-26 11:00:19.01 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":4.9224805e-2,"slot":27868528,"delegMapSize":465155,"utxoSize":2014760,"credentials":"Cardano"} Apr 26 11:00:19 maticbp bash[5936]: [maticbp:cardano.node.Forge:Info:558] [2021-04-26 11:00:19.10 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 2.7868528e7)])),("credentials",String "Cardano")] Apr 26 11:00:19 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Notice:548] [2021-04-26 11:00:19.43 UTC] Chain extended, new tip: 289c89d5ebb7c483cf077e0ec39c0fc1e80e6b6f1ea32d40d8ff801a0419b542 at slot 27868172 Apr 26 11:00:20 maticbp bash[5936]: [maticbp:cardano.node.LeadershipCheck:Info:558] [2021-04-26 11:00:20.00 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":4.9219195e-2,"slot":27868529,"delegMapSize":465160,"utxoSize":2014869,"credentials":"Cardano"} Apr 26 11:00:20 maticbp bash[5936]: [maticbp:cardano.node.Forge:Info:558] [2021-04-26 11:00:20.04 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 2.7868529e7)])),("credentials",String "Cardano")] Apr 26 11:00:21 maticbp bash[5936]: [maticbp:cardano.node.LeadershipCheck:Info:558] [2021-04-26 11:00:21.01 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":4.9219195e-2,"slot":27868530,"delegMapSize":465160,"utxoSize":2014869,"credentials":"Cardano"} Apr 26 11:00:21 maticbp bash[5936]: [maticbp:cardano.node.Forge:Info:558] [2021-04-26 11:00:21.11 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 2.786853e7)])),("credentials",String "Cardano")] Apr 26 11:00:22 maticbp bash[5936]: [maticbp:cardano.node.LeadershipCheck:Info:558] [2021-04-26 11:00:22.05 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":4.9219195e-2,"slot":27868531,"delegMapSize":465160,"utxoSize":2014869,"credentials":"Cardano"} Apr 26 11:00:22 maticbp bash[5936]: [maticbp:cardano.node.Forge:Info:558] [2021-04-26 11:00:22.34 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 2.7868531e7)])),("credentials",String "Cardano")] Apr 26 11:00:22 maticbp bash[5936]: [maticbp:cardano.node.BlockFetchDecision:Info:560] [2021-04-26 11:00:22.66 UTC] [TraceLabelPeer (ConnectionId {localAddress = bp, remoteAddress = relay}) (Right [At (Block {blockPointSlot = SlotNo 27868531, blockPointHash = 1a1d68f097b5fa9e22e8c23e1fed033dcc2add55c4a13f2994b4aca3dddc9a0b})])] Apr 26 11:00:25 maticbp bash[5936]: [maticbp:cardano.node.LeadershipCheck:Info:558] [2021-04-26 11:00:24.36 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":4.9151964e-2,"slot":27868532,"delegMapSize":465161,"utxoSize":2014877,"credentials":"Cardano"} Apr 26 11:00:25 maticbp bash[5936]: [maticbp:cardano.node.ChainDB:Notice:548] [2021-04-26 11:00:24.52 UTC] Chain extended, new tip: 1a1d68f097b5fa9e22e8c23e1fed033dcc2add55c4a13f2994b4aca3dddc9a0b at slot 27868531 Apr 26 11:00:25 maticbp bash[5936]: [maticbp:cardano.node.Forge:Info:558] [2021-04-26 11:00:25.13 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 2.7868532e7)])),("credentials",String "Cardano")] Apr 26 11:00:25 maticbp bash[5936]: [maticbp:cardano.node.LeadershipCheck:Info:558] [2021-04-26 11:00:25.14 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":4.9151964e-2,"slot":27868534,"delegMapSize":465161,"utxoSize":2014877,"credentials":"Cardano"} Apr 26 11:00:25 maticbp bash[5936]: [maticbp:cardano.node.Forge:Info:558] [2021-04-26 11:00:25.34 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 2.7868534e7)])),("credentials",String "Cardano")] Apr 26 11:00:26 maticbp bash[5936]: [maticbp:cardano.node.LeadershipCheck:Info:558] [2021-04-26 11:00:26.01 UTC] {"kind":"TraceStartLeadershipCheck","chainDensity":4.9151964e-2,"slot":27868535,"delegMapSize":465161,"utxoSize":2014877,"credentials":"Cardano"} Apr 26 11:00:26 maticbp bash[5936]: [maticbp:cardano.node.Forge:Info:558] [2021-04-26 11:00:26.08 UTC] fromList [("val",Object (fromList [("kind",String "TraceNodeNotLeader"),("slot",Number 2.7868535e7)])),("credentials",String "Cardano")] Apr 26 11:00:27 maticbp bash[5935]: /home/cardano/cardano-my-node/startBlockProducingNode.sh: line 11: 5936 Killed cardano-node run --topology {TOPOLOGY} --database-path {DB_PATH} --socket-path {SOCKET_PATH} --host-addr {HOSTADDR} --port {PORT} --config {CONFIG} --shelley-kes-key {KES} --shelley-vrf-key {VRF} --shelley-operational-certificate {CERT}

niki_niki_airmax · 26 April 2021 11:15

I hope you can read through this, otherwise tell me and I’ll figure something out.

Alexd1985 · 26 April 2021 11:21

nano or cat /home/cardano/cardano-my-node/startBlockProducingNode.sh and check the line 11

niki_niki_airmax · 26 April 2021 11:48

Line 11 is this:

cardano-node run --topology {TOPOLOGY} --database-path {DB_PATH} --socket-path {SOCKET_PATH} --host-addr {HOSTADDR} --port {PORT} --config {CONFIG} --shelley-kes-key {KES} --shelley-vrf-key {VRF} --shelley-operational-certificate ${CERT}

Alexd1985 · 26 April 2021 11:51

ok, try to start the node manually

disable and stop the systemd node

sudo systemctl stop cardano-node
sudo systemctl disable cardano-node

and try to start manually

./startBlockProducingNode.sh

if u have the same issue, then try to recreate the script from the guide, also try to check the paths for files, etc

should looks like:

cat > $NODE_HOME/startBlockProducingNode.sh << EOF 
DIRECTORY=$NODE_HOME
PORT=6000
HOSTADDR=0.0.0.0
TOPOLOGY=\${DIRECTORY}/${NODE_CONFIG}-topology.json
DB_PATH=\${DIRECTORY}/db
SOCKET_PATH=\${DIRECTORY}/db/socket
CONFIG=\${DIRECTORY}/${NODE_CONFIG}-config.json
KES=\${DIRECTORY}/kes.skey
VRF=\${DIRECTORY}/vrf.skey
CERT=\${DIRECTORY}/node.cert
cardano-node run --topology \${TOPOLOGY} --database-path \${DB_PATH} --socket-path \${SOCKET_PATH} --host-addr \${HOSTADDR} --port \${PORT} --config \${CONFIG} --shelley-kes-key \${KES} --shelley-vrf-key \${VRF} --shelley-operational-certificate \${CERT}
EOF

also for line 11 error u can take a look here

https://forum.cardano.org/search?q=startBlockProducingNode.sh:%20line%2011

niki_niki_airmax · 26 April 2021 12:14

So, it still crashed. I even tried using the relay node start script to minimise files, but it still crashed. I’m thinking it’s something about the database as it spat out this error:

[maticbp:cardano.node.ChainDB:Error:5] [2021-04-26 12:09:37.88 UTC] Invalid snapshot DiskSnapshot {dsNumber = 27827376, dsSuffix = Nothing}InitFailureRead (ReadFailed (DeserialiseFailure 176891292 “end of input”))

Now, as I mentioned in my post it DID work after I reset the database but 10 hours later it crashed and maybe (corrupted) the database somehow?

Alexd1985 · 26 April 2021 12:22

Hmm, can be; perhaps better to delete the db and resync again the node? Or try to copy the db from one of ur relay

niki_niki_airmax · 26 April 2021 12:24

Yeah, I will copy the database from the relay, but I don’t wanna do this every 10 hours if it crashes.

laplasz · 28 April 2021 08:31

Any update on this issue - do you have a synced db?

tpokorra · 29 April 2021 04:23

I had the same problem.
The reason: I was running out of memory, and that made my node crash after upgrading to 1.26
I am testing on a virtual machine that has less than recommended memory and was running other applications as well.

niki_niki_airmax · 29 April 2021 11:37

Yep, will be switching VPS plan to a better one as it seems like RAM is a bit of a problem after the latest release. It is running stable for 24 hours now, but better safe than sorry.

niki_niki_airmax · 29 April 2021 11:39

Yep, figured so. Will be switching to a better VPS plan (in this case another provider), do I only have to copy the operational keys and leave it be on another machine or do i have to re-register the pool if you know? @Alexd1985

Alexd1985 · 29 April 2021 11:43

Don’t need to register again the pool;
Move what u have now in priv folder to the new server

niki_niki_airmax · 29 April 2021 12:19

alright, thanks