check this topic as well - detailed info about what @Alexd1985 is suggesting:
Thank you Gents.
This helped a bit but cnode does nothing now. When I run LiveView I see only “starting”. I deleted all DB and DB folder is empty with correct permissions. So I should be seeing “syncing” and some progress.
I modified topology to is looks now like:
px2@sm3:/opt/cardano/cnode/scripts$ cat ../files/topology.json
{
"Producers":
[
{
"addr": "110.174.184.26",
"port": 3001,
"continent": "Oceania",
"state": "AU",
"valency": 1
},
{
"addr": "61.69.254.98",
"port": 3001,
"continent": "Oceania",
"state": "AU",
"valency": 1
},
{
"addr": "14.201.73.146",
"port": 3001,
"continent": "Oceania",
"state": "AU",
"valency": 1
},
{
"addr": "relay02.stakepoolcentral.com",
"port": 17654,
"continent": "Europe",
"state": "NL",
"valency": 1
}
]
}
cnode is running, what is shown by systemctl status and I see process running by ps.
● cnode.service - Cardano Node
Loaded: loaded (/etc/systemd/system/cnode.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-06-02 19:34:37 NZST; 22min ago
Main PID: 3227352 (bash)
Tasks: 10 (limit: 14295)
Memory: 91.3M
CGroup: /system.slice/cnode.service
├─3227352 bash /opt/cardano/cnode/scripts/cnode.sh
└─3227431 cardano-node run --topology /opt/cardano/cnode/files/topology.json --config /opt/cardano/cnode/files/config.json --database-path /mnt/data/cardano>
cze 02 19:34:37 sm3 systemd[1]: Started Cardano Node.
cze 02 19:34:39 sm3 cnode[3227431]: Listening on http://127.0.0.1:12798
Logs doesn’t move any further than:
…
{“thread”:“5”,“loc”:null,“data”:{“message”:“129600”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotsPerKESPeriodAllegra”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“1s”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotLengthMary”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“432000”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.epochLengthMary”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“129600”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotsPerKESPeriodMary”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“1s”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotLengthAlonzo”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“432000”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.epochLengthAlonzo”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“129600”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotsPerKESPeriodAlonzo”],“at”:“2021-06-02T07:34:39.22Z”}
It sits there, doing nothing, logging nothing.
I think because topology file is contains invalid fields… like state, and continent… go through on the linked topic again.
@laplasz , I checked it out and I seen such fields in example there. I haven’t added those fields by myself, just copied “preferred peers” sections to my topology.
Anyway… odd but just restarting it again, without changing anything, made it syncing.
Literally, from my last reply, only what I did was “systemctl restart cnode” .
It goes now. I will leave it overnight (Yes, night, I’m in NZ) and check status tomorrow. Hopefully it will pass 55%.
Thanks to both of you so far and I’ll update this thread tomorrow with progress.
so here is the relevant part
Indeed…
Note that “continent” and “state” keys should be replaced with “valency”:
Apologies for my reading inability…
Anyway, it went ok after restart with those fields… I know, I know… just saying.
hmm, ok that is good to know! thanks!
Alright, so we’re back in square 1.
It was syncing overnight but again, stuck on 54.7%.
Log show:
{“thread”:“64”,“loc”:null,“data”:{“address”:“203.159.80.226:45726”,“kind”:“ErrorPolicyTrace”,“event”:“ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 34> closed when reading data, waiting on next header True”))) 20s 20s”},“sev”:“Warning”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ErrorPolicy”],“at”:“2021-06-02T20:03:50.60Z”}
{“thread”:“64”,“loc”:null,“data”:{“address”:“172.16.0.151:43865”,“kind”:“ErrorPolicyTrace”,“event”:“ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 40> closed when reading data, waiting on next header True”))) 20s 20s”},“sev”:“Warning”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ErrorPolicy”],“at”:“2021-06-02T20:05:54.73Z”}
{“thread”:“152274”,“loc”:null,“data”:{“kind”:“ChainSyncServerEvent.TraceChainSyncServerRead.RollBack”,“slot”:17020929,“block”:“c8e0010c753b5e6ce284926def428498ded9064cd7f085f190494f8d2208f794”,“blockNo”:5107443},“sev”:“Info”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ChainSyncHeaderServer”],“at”:“2021-06-02T20:05:54.73Z”}
{“thread”:“64”,“loc”:null,“data”:{“address”:“172.16.0.151:41881”,“kind”:“ErrorPolicyTrace”,“event”:“ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 94> closed when reading data, waiting on next header True”))) 20s 20s”},“sev”:“Warning”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ErrorPolicy”],“at”:“2021-06-02T20:07:25.74Z”}
{“thread”:“152329”,“loc”:null,“data”:{“kind”:“ChainSyncServerEvent.TraceChainSyncServerRead.RollBack”,“slot”:17020929,“block”:“c8e0010c753b5e6ce284926def428498ded9064cd7f085f190494f8d2208f794”,“blockNo”:5107443},“sev”:“Info”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ChainSyncHeaderServer”],“at”:“2021-06-02T20:07:45.75Z”}
Still has connection to all peers and shows no other issues (as far as I can asses…)
172.16.0.151 is a producer connected only to this relay and currently this producer is syncing as well to whatever relay can sync to.
what is the version of cardano-node? all config files up-to-date?
1.27.0 built from source.
This is not version used in tutorial but latest. Because of that I faced issue with missing alonzo config entry which I added (entry to config and config file downloaded from era config page). Otherwise node didn’t want to start at all. So above was the modification on general config I made.
alonzo is from that site but shelley and byron are default which came with the source of 1.27.0. I assumed those should be fine, was it wrong assumption? I will compare what I have with what’s on the site. Thanks!
As per below output, no difference between mine and site available genesis configs:
px2@sm3:~/cardano$ wget https://hydra.iohk.io/build/6498473/download/1/mainnet-byron-genesis.json
--2021-06-03 11:39:59-- https://hydra.iohk.io/build/6498473/download/1/mainnet-byron-genesis.json
Translacja hydra.iohk.io (hydra.iohk.io)... 147.75.83.98
Łączenie się z hydra.iohk.io (hydra.iohk.io)|147.75.83.98|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 200 OK
Długość: 1056360 (1.0M) [application/json]
Zapis do: `mainnet-byron-genesis.json'
mainnet-byron-genesis.json 100%[======================================================================================>] 1.01M 402KB/s w 2.6s
2021-06-03 11:40:04 (402 KB/s) - zapisano `mainnet-byron-genesis.json' [1056360/1056360]
px2@sm3:~/cardano$ diff mainnet-byron-genesis.json /opt/cardano/cnode/files/
alonzo-genesis.json byron-genesis.json config.json dbsync.json genesis.json topology.json
px2@sm3:~/cardano$ diff mainnet-byron-genesis.json /opt/cardano/cnode/files/byron-genesis.json
px2@sm3:~/cardano$ wget https://hydra.iohk.io/build/6498473/download/1/mainnet-shelley-genesis.json
--2021-06-03 11:40:43-- https://hydra.iohk.io/build/6498473/download/1/mainnet-shelley-genesis.json
Translacja hydra.iohk.io (hydra.iohk.io)... 147.75.83.98
Łączenie się z hydra.iohk.io (hydra.iohk.io)|147.75.83.98|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 200 OK
Długość: 2486 (2.4K) [application/json]
Zapis do: `mainnet-shelley-genesis.json'
mainnet-shelley-genesis.json 100%[======================================================================================>] 2.43K --.-KB/s w 0s
2021-06-03 11:40:45 (432 MB/s) - zapisano `mainnet-shelley-genesis.json' [2486/2486]
px2@sm3:~/cardano$ diff mainnet-shelley-genesis.json /opt/cardano/cnode/files/
alonzo-genesis.json byron-genesis.json config.json dbsync.json genesis.json topology.json
px2@sm3:~/cardano$ diff mainnet-shelley-genesis.json /opt/cardano/cnode/files/genesis.json
px2@sm3:~/cardano$ wget https://hydra.iohk.io/build/6498473/download/1/mainnet-alonzo-genesis.json
--2021-06-03 11:41:27-- https://hydra.iohk.io/build/6498473/download/1/mainnet-alonzo-genesis.json
Translacja hydra.iohk.io (hydra.iohk.io)... 147.75.83.98
Łączenie się z hydra.iohk.io (hydra.iohk.io)|147.75.83.98|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 200 OK
Długość: 309 [application/json]
Zapis do: `mainnet-alonzo-genesis.json'
mainnet-alonzo-genesis.json 100%[======================================================================================>] 309 --.-KB/s w 0s
2021-06-03 11:41:29 (51.0 MB/s) - zapisano `mainnet-alonzo-genesis.json' [309/309]
px2@sm3:~/cardano$ diff mainnet-alonzo-genesis.json /opt/cardano/cnode/files/alonzo-genesis.json
px2@sm3:~/cardano$
Grepping log for errors (and ignoring ones between my relay and my producer), this is what I get:
{"thread":"38","loc":null,"data":{"kind":"TraceAddBlockEvent.AddBlockValidation.InvalidBlock","block":{"kind":"Point","hash":"448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d","slot":17020941},"error":"ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))"},"sev":"Error","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainDB"],"at":"2021-06-02T17:31:33.87Z"}
{"thread":"128365","loc":null,"data":{"kind":"ChainSyncClientEvent.TraceException","peer":{"remote":{"addr":"51.145.240.140","port":"17654"},"local":{"addr":"172.16.0.153","port":"42047"}},"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainSyncClient"],"at":"2021-06-02T17:31:33.88Z"}
{"thread":"128316","loc":null,"data":{"kind":"ChainSyncClientEvent.TraceException","peer":{"remote":{"addr":"110.174.184.26","port":"3001"},"local":{"addr":"172.16.0.153","port":"41377"}},"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainSyncClient"],"at":"2021-06-02T17:31:33.88Z"}
{"thread":"128330","loc":null,"data":{"kind":"ChainSyncClientEvent.TraceException","peer":{"remote":{"addr":"14.201.73.146","port":"3001"},"local":{"addr":"172.16.0.153","port":"39649"}},"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainSyncClient"],"at":"2021-06-02T17:31:33.88Z"}
{"thread":"139602","loc":null,"data":{"kind":"ChainSyncClientEvent.TraceException","peer":{"remote":{"addr":"61.69.254.98","port":"3001"},"local":{"addr":"172.16.0.153","port":"36303"}},"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainSyncClient"],"at":"2021-06-02T17:31:33.89Z"}
{"thread":"64","loc":null,"data":{"address":"203.159.80.226:45726","kind":"ErrorPolicyTrace","event":"ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed \"<socket: 34> closed when reading data, waiting on next header True\"))) 20s 20s"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ErrorPolicy"],"at":"2021-06-02T20:03:50.60Z"}
this looks for me like protocol or blockchain error, not really specific to my deployment. Can anyone confirm or provide some guidance how to read that?
What command did u used to compile the node… did u also used git pull origin master ?
mkdir “$HOME/tmp”
cd “$HOME/tmp”
curl -sS -o prereqs.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/prereqs.sh
chmod 755 prereqs.sh
./prereqs.sh
. “${HOME}/.bashrc”
cd ~/git
git clone GitHub - input-output-hk/cardano-node: The core component that is used to participate in a Cardano decentralised blockchain.
cd cardano-nodegit fetch --tags --all
git checkout 1.27.0
git pull origin masterecho -e “package cardano-crypto-praos\n flags: -external-libsodium-vrf” > cabal.project.local
$CNODE_HOME/scripts/cabal-build-all.sh -o
Yeah, u should not use git pull origin master
go to cd ~/.cabal/bin folder delete the cardano-node and rebuild the node
If u will want also to use import wallet option use the commands :
ghcup install cabal 3.2.0.0
ghcup set cabal 3.2.0.0
Build the node
cd ~/git
cd cardano-node
git fetch --tags --all
git checkout 1.27.0
echo -e "package cardano-crypto-praos\n flags: -external-libsodium-vrf" > cabal.project.local
$CNODE_HOME/scripts/cabal-build-all.sh -o
might be related:
so lets see what happens in that topic… could you try the config files from release page @erikd suggesting?
Recompiling (without pull origin master) did the trick. So far it’s syncing and on 77%. I think we’re good so far. Once that accomplished I will (recompile &) re-sync producer.
Laplasz, thanks for digging this one out anyway.
I know, because compiling with master will run all features that are preparing for alonzo hardfork
For producer u can download the db from relay and upload it on Producer node… it will reduce the time for syncing