Public peer IPs & sync issues

check this topic as well - detailed info about what @Alexd1985 is suggesting:

1 Like

Thank you Gents.

This helped a bit but cnode does nothing now. When I run LiveView I see only “starting”. I deleted all DB and DB folder is empty with correct permissions. So I should be seeing “syncing” and some progress.

I modified topology to is looks now like:

px2@sm3:/opt/cardano/cnode/scripts$ cat ../files/topology.json 
{
  "Producers": 
  [
    {
      "addr": "110.174.184.26",
      "port": 3001,
      "continent": "Oceania",
      "state": "AU",
      "valency": 1
    },
    {
      "addr": "61.69.254.98",
      "port": 3001,
      "continent": "Oceania",
      "state": "AU",
      "valency": 1
    },
    {
      "addr": "14.201.73.146",
      "port": 3001,
      "continent": "Oceania",
      "state": "AU",
      "valency": 1
    },
    {
      "addr": "relay02.stakepoolcentral.com",
      "port": 17654,
      "continent": "Europe",
      "state": "NL",
      "valency": 1
    }
  ]
}

cnode is running, what is shown by systemctl status and I see process running by ps.
● cnode.service - Cardano Node
Loaded: loaded (/etc/systemd/system/cnode.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-06-02 19:34:37 NZST; 22min ago
Main PID: 3227352 (bash)
Tasks: 10 (limit: 14295)
Memory: 91.3M
CGroup: /system.slice/cnode.service
├─3227352 bash /opt/cardano/cnode/scripts/cnode.sh
└─3227431 cardano-node run --topology /opt/cardano/cnode/files/topology.json --config /opt/cardano/cnode/files/config.json --database-path /mnt/data/cardano>

cze 02 19:34:37 sm3 systemd[1]: Started Cardano Node.
cze 02 19:34:39 sm3 cnode[3227431]: Listening on http://127.0.0.1:12798

Logs doesn’t move any further than:

{“thread”:“5”,“loc”:null,“data”:{“message”:“129600”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotsPerKESPeriodAllegra”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“1s”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotLengthMary”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“432000”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.epochLengthMary”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“129600”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotsPerKESPeriodMary”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“1s”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotLengthAlonzo”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“432000”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.epochLengthAlonzo”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“129600”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotsPerKESPeriodAlonzo”],“at”:“2021-06-02T07:34:39.22Z”}

It sits there, doing nothing, logging nothing.

I think because topology file is contains invalid fields… like state, and continent… go through on the linked topic again.

@laplasz , I checked it out and I seen such fields in example there. I haven’t added those fields by myself, just copied “preferred peers” sections to my topology.
Anyway… odd but just restarting it again, without changing anything, made it syncing.
Literally, from my last reply, only what I did was “systemctl restart cnode” .
It goes now. I will leave it overnight (Yes, night, I’m in NZ) and check status tomorrow. Hopefully it will pass 55%.
Thanks to both of you so far and I’ll update this thread tomorrow with progress.

so here is the relevant part

Indeed…

Note that “continent” and “state” keys should be replaced with “valency”:

Apologies for my reading inability… :wink:
Anyway, it went ok after restart with those fields… I know, I know… just saying.

hmm, ok that is good to know! thanks!

Alright, so we’re back in square 1.
It was syncing overnight but again, stuck on 54.7%.
Log show:
{“thread”:“64”,“loc”:null,“data”:{“address”:“203.159.80.226:45726”,“kind”:“ErrorPolicyTrace”,“event”:“ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 34> closed when reading data, waiting on next header True”))) 20s 20s”},“sev”:“Warning”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ErrorPolicy”],“at”:“2021-06-02T20:03:50.60Z”}
{“thread”:“64”,“loc”:null,“data”:{“address”:“172.16.0.151:43865”,“kind”:“ErrorPolicyTrace”,“event”:“ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 40> closed when reading data, waiting on next header True”))) 20s 20s”},“sev”:“Warning”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ErrorPolicy”],“at”:“2021-06-02T20:05:54.73Z”}
{“thread”:“152274”,“loc”:null,“data”:{“kind”:“ChainSyncServerEvent.TraceChainSyncServerRead.RollBack”,“slot”:17020929,“block”:“c8e0010c753b5e6ce284926def428498ded9064cd7f085f190494f8d2208f794”,“blockNo”:5107443},“sev”:“Info”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ChainSyncHeaderServer”],“at”:“2021-06-02T20:05:54.73Z”}
{“thread”:“64”,“loc”:null,“data”:{“address”:“172.16.0.151:41881”,“kind”:“ErrorPolicyTrace”,“event”:“ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 94> closed when reading data, waiting on next header True”))) 20s 20s”},“sev”:“Warning”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ErrorPolicy”],“at”:“2021-06-02T20:07:25.74Z”}
{“thread”:“152329”,“loc”:null,“data”:{“kind”:“ChainSyncServerEvent.TraceChainSyncServerRead.RollBack”,“slot”:17020929,“block”:“c8e0010c753b5e6ce284926def428498ded9064cd7f085f190494f8d2208f794”,“blockNo”:5107443},“sev”:“Info”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ChainSyncHeaderServer”],“at”:“2021-06-02T20:07:45.75Z”}

Still has connection to all peers and shows no other issues (as far as I can asses…)
172.16.0.151 is a producer connected only to this relay and currently this producer is syncing as well to whatever relay can sync to.

what is the version of cardano-node? all config files up-to-date?

1.27.0 built from source.
This is not version used in tutorial but latest. Because of that I faced issue with missing alonzo config entry which I added (entry to config and config file downloaded from era config page). Otherwise node didn’t want to start at all. So above was the modification on general config I made.

ok, so genesis files are also up-to-date - from this site - Cardano Configurations

alonzo is from that site but shelley and byron are default which came with the source of 1.27.0. I assumed those should be fine, was it wrong assumption? :slight_smile: I will compare what I have with what’s on the site. Thanks!

As per below output, no difference between mine and site available genesis configs:

px2@sm3:~/cardano$ wget https://hydra.iohk.io/build/6498473/download/1/mainnet-byron-genesis.json
--2021-06-03 11:39:59--  https://hydra.iohk.io/build/6498473/download/1/mainnet-byron-genesis.json
Translacja hydra.iohk.io (hydra.iohk.io)... 147.75.83.98
Łączenie się z hydra.iohk.io (hydra.iohk.io)|147.75.83.98|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 200 OK
Długość: 1056360 (1.0M) [application/json]
Zapis do: `mainnet-byron-genesis.json'

mainnet-byron-genesis.json                 100%[======================================================================================>]   1.01M   402KB/s    w 2.6s     

2021-06-03 11:40:04 (402 KB/s) - zapisano `mainnet-byron-genesis.json' [1056360/1056360]

px2@sm3:~/cardano$ diff mainnet-byron-genesis.json /opt/cardano/cnode/files/
alonzo-genesis.json  byron-genesis.json   config.json          dbsync.json          genesis.json         topology.json        
px2@sm3:~/cardano$ diff mainnet-byron-genesis.json /opt/cardano/cnode/files/byron-genesis.json 
px2@sm3:~/cardano$ wget https://hydra.iohk.io/build/6498473/download/1/mainnet-shelley-genesis.json
--2021-06-03 11:40:43--  https://hydra.iohk.io/build/6498473/download/1/mainnet-shelley-genesis.json
Translacja hydra.iohk.io (hydra.iohk.io)... 147.75.83.98
Łączenie się z hydra.iohk.io (hydra.iohk.io)|147.75.83.98|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 200 OK
Długość: 2486 (2.4K) [application/json]
Zapis do: `mainnet-shelley-genesis.json'

mainnet-shelley-genesis.json               100%[======================================================================================>]   2.43K  --.-KB/s    w 0s       

2021-06-03 11:40:45 (432 MB/s) - zapisano `mainnet-shelley-genesis.json' [2486/2486]

px2@sm3:~/cardano$ diff mainnet-shelley-genesis.json /opt/cardano/cnode/files/
alonzo-genesis.json  byron-genesis.json   config.json          dbsync.json          genesis.json         topology.json        
px2@sm3:~/cardano$ diff mainnet-shelley-genesis.json /opt/cardano/cnode/files/genesis.json 
px2@sm3:~/cardano$ wget https://hydra.iohk.io/build/6498473/download/1/mainnet-alonzo-genesis.json
--2021-06-03 11:41:27--  https://hydra.iohk.io/build/6498473/download/1/mainnet-alonzo-genesis.json
Translacja hydra.iohk.io (hydra.iohk.io)... 147.75.83.98
Łączenie się z hydra.iohk.io (hydra.iohk.io)|147.75.83.98|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 200 OK
Długość: 309 [application/json]
Zapis do: `mainnet-alonzo-genesis.json'

mainnet-alonzo-genesis.json                100%[======================================================================================>]     309  --.-KB/s    w 0s       

2021-06-03 11:41:29 (51.0 MB/s) - zapisano `mainnet-alonzo-genesis.json' [309/309]

px2@sm3:~/cardano$ diff mainnet-alonzo-genesis.json /opt/cardano/cnode/files/alonzo-genesis.json 
px2@sm3:~/cardano$

Grepping log for errors (and ignoring ones between my relay and my producer), this is what I get:

{"thread":"38","loc":null,"data":{"kind":"TraceAddBlockEvent.AddBlockValidation.InvalidBlock","block":{"kind":"Point","hash":"448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d","slot":17020941},"error":"ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))"},"sev":"Error","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainDB"],"at":"2021-06-02T17:31:33.87Z"}
{"thread":"128365","loc":null,"data":{"kind":"ChainSyncClientEvent.TraceException","peer":{"remote":{"addr":"51.145.240.140","port":"17654"},"local":{"addr":"172.16.0.153","port":"42047"}},"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainSyncClient"],"at":"2021-06-02T17:31:33.88Z"}
{"thread":"128316","loc":null,"data":{"kind":"ChainSyncClientEvent.TraceException","peer":{"remote":{"addr":"110.174.184.26","port":"3001"},"local":{"addr":"172.16.0.153","port":"41377"}},"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainSyncClient"],"at":"2021-06-02T17:31:33.88Z"}
{"thread":"128330","loc":null,"data":{"kind":"ChainSyncClientEvent.TraceException","peer":{"remote":{"addr":"14.201.73.146","port":"3001"},"local":{"addr":"172.16.0.153","port":"39649"}},"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainSyncClient"],"at":"2021-06-02T17:31:33.88Z"}
{"thread":"139602","loc":null,"data":{"kind":"ChainSyncClientEvent.TraceException","peer":{"remote":{"addr":"61.69.254.98","port":"3001"},"local":{"addr":"172.16.0.153","port":"36303"}},"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainSyncClient"],"at":"2021-06-02T17:31:33.89Z"}
{"thread":"64","loc":null,"data":{"address":"203.159.80.226:45726","kind":"ErrorPolicyTrace","event":"ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed \"<socket: 34> closed when reading data, waiting on next header True\"))) 20s 20s"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ErrorPolicy"],"at":"2021-06-02T20:03:50.60Z"}

this looks for me like protocol or blockchain error, not really specific to my deployment. Can anyone confirm or provide some guidance how to read that?

What command did u used to compile the node… did u also used git pull origin master ?

1 Like

mkdir “$HOME/tmp”
cd “$HOME/tmp”
curl -sS -o prereqs.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/prereqs.sh
chmod 755 prereqs.sh
./prereqs.sh
. “${HOME}/.bashrc”
cd ~/git
git clone GitHub - input-output-hk/cardano-node: The core component that is used to participate in a Cardano decentralised blockchain.
cd cardano-node

git fetch --tags --all
git checkout 1.27.0
git pull origin master

echo -e “package cardano-crypto-praos\n flags: -external-libsodium-vrf” > cabal.project.local
$CNODE_HOME/scripts/cabal-build-all.sh -o

Yeah, u should not use git pull origin master

go to cd ~/.cabal/bin folder delete the cardano-node and rebuild the node

If u will want also to use import wallet option use the commands :


ghcup install cabal 3.2.0.0
ghcup set cabal 3.2.0.0

Build the node
cd ~/git
cd cardano-node

git fetch --tags --all
git checkout 1.27.0


echo -e "package cardano-crypto-praos\n  flags: -external-libsodium-vrf" > cabal.project.local
$CNODE_HOME/scripts/cabal-build-all.sh -o

might be related:

so lets see what happens in that topic… could you try the config files from release page @erikd suggesting?

Recompiling (without pull origin master) did the trick. So far it’s syncing and on 77%. I think we’re good so far. Once that accomplished I will (recompile &) re-sync producer.
Laplasz, thanks for digging this one out anyway.

1 Like

I know, because compiling with master will run all features that are preparing for alonzo hardfork

For producer u can download the db from relay and upload it on Producer node… it will reduce the time for syncing