Public peer IPs & sync issues

Hi,
How I can find public peers IPs?

I’m trying to set up pool based on relay+producer. (based on How to set up a POOL in a few minutes - and register using CNTOOLS) Got them already connected and relay sync DB from network peers and producer syncs from relay but…
It goes really slow and I blame for that fact that I’m connected to peers which are really distant from me.
Therefore I was thinking about defining a closer peer to connect to.

However now I see that my relay stopped syncing at about 54.8% and in its log I see:
{"thread":"56","loc":null,"data":{"address":"50.18.12.114:3001","kind":"ErrorPolicyTrace","event":"ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))))) 200s 200s"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"1670668","ns":["cardano.node.ErrorPolicy"],"at":"2021-06-02T06:28:48.85Z"}

I won’t be lying that I understand above entry but it repeats in similar shape. And LiveView shows (on relay) 0 peers out.
I’d think that I’ve done something wrong but that’s not the case if till this moment it was syncing correctly for about 8 hours.

So, who can “decipher” meaning of that log entry and advise how I should progress?
Thanks in advance!
Px2

Hi!
How did you find other peers addresses?

Default topology file have generic dns name which resolves to few. I was wondering about changing that to static IP(s) closest peers.
Anyway, error in sync seems to be more important to solve now. :frowning:

Check this topology, here u can find all registered relays

https://explorer.mainnet.cardano.org/relays/topology.json

check this topic as well - detailed info about what @Alexd1985 is suggesting:

1 Like

Thank you Gents.

This helped a bit but cnode does nothing now. When I run LiveView I see only “starting”. I deleted all DB and DB folder is empty with correct permissions. So I should be seeing “syncing” and some progress.

I modified topology to is looks now like:

px2@sm3:/opt/cardano/cnode/scripts$ cat ../files/topology.json 
{
  "Producers": 
  [
    {
      "addr": "110.174.184.26",
      "port": 3001,
      "continent": "Oceania",
      "state": "AU",
      "valency": 1
    },
    {
      "addr": "61.69.254.98",
      "port": 3001,
      "continent": "Oceania",
      "state": "AU",
      "valency": 1
    },
    {
      "addr": "14.201.73.146",
      "port": 3001,
      "continent": "Oceania",
      "state": "AU",
      "valency": 1
    },
    {
      "addr": "relay02.stakepoolcentral.com",
      "port": 17654,
      "continent": "Europe",
      "state": "NL",
      "valency": 1
    }
  ]
}

cnode is running, what is shown by systemctl status and I see process running by ps.
● cnode.service - Cardano Node
Loaded: loaded (/etc/systemd/system/cnode.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2021-06-02 19:34:37 NZST; 22min ago
Main PID: 3227352 (bash)
Tasks: 10 (limit: 14295)
Memory: 91.3M
CGroup: /system.slice/cnode.service
├─3227352 bash /opt/cardano/cnode/scripts/cnode.sh
└─3227431 cardano-node run --topology /opt/cardano/cnode/files/topology.json --config /opt/cardano/cnode/files/config.json --database-path /mnt/data/cardano>

cze 02 19:34:37 sm3 systemd[1]: Started Cardano Node.
cze 02 19:34:39 sm3 cnode[3227431]: Listening on http://127.0.0.1:12798

Logs doesn’t move any further than:

{“thread”:“5”,“loc”:null,“data”:{“message”:“129600”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotsPerKESPeriodAllegra”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“1s”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotLengthMary”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“432000”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.epochLengthMary”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“129600”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotsPerKESPeriodMary”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“1s”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotLengthAlonzo”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“432000”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.epochLengthAlonzo”],“at”:“2021-06-02T07:34:39.22Z”}
{“thread”:“5”,“loc”:null,“data”:{“message”:“129600”,“kind”:“LogMessage”},“sev”:“Notice”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3227431”,“ns”:[“cardano.node.basicInfo.slotsPerKESPeriodAlonzo”],“at”:“2021-06-02T07:34:39.22Z”}

It sits there, doing nothing, logging nothing.

I think because topology file is contains invalid fields… like state, and continent… go through on the linked topic again.

@laplasz , I checked it out and I seen such fields in example there. I haven’t added those fields by myself, just copied “preferred peers” sections to my topology.
Anyway… odd but just restarting it again, without changing anything, made it syncing.
Literally, from my last reply, only what I did was “systemctl restart cnode” .
It goes now. I will leave it overnight (Yes, night, I’m in NZ) and check status tomorrow. Hopefully it will pass 55%.
Thanks to both of you so far and I’ll update this thread tomorrow with progress.

so here is the relevant part

Indeed…

Note that “continent” and “state” keys should be replaced with “valency”:

Apologies for my reading inability… :wink:
Anyway, it went ok after restart with those fields… I know, I know… just saying.

hmm, ok that is good to know! thanks!

Alright, so we’re back in square 1.
It was syncing overnight but again, stuck on 54.7%.
Log show:
{“thread”:“64”,“loc”:null,“data”:{“address”:“203.159.80.226:45726”,“kind”:“ErrorPolicyTrace”,“event”:“ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 34> closed when reading data, waiting on next header True”))) 20s 20s”},“sev”:“Warning”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ErrorPolicy”],“at”:“2021-06-02T20:03:50.60Z”}
{“thread”:“64”,“loc”:null,“data”:{“address”:“172.16.0.151:43865”,“kind”:“ErrorPolicyTrace”,“event”:“ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 40> closed when reading data, waiting on next header True”))) 20s 20s”},“sev”:“Warning”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ErrorPolicy”],“at”:“2021-06-02T20:05:54.73Z”}
{“thread”:“152274”,“loc”:null,“data”:{“kind”:“ChainSyncServerEvent.TraceChainSyncServerRead.RollBack”,“slot”:17020929,“block”:“c8e0010c753b5e6ce284926def428498ded9064cd7f085f190494f8d2208f794”,“blockNo”:5107443},“sev”:“Info”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ChainSyncHeaderServer”],“at”:“2021-06-02T20:05:54.73Z”}
{“thread”:“64”,“loc”:null,“data”:{“address”:“172.16.0.151:41881”,“kind”:“ErrorPolicyTrace”,“event”:“ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 94> closed when reading data, waiting on next header True”))) 20s 20s”},“sev”:“Warning”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ErrorPolicy”],“at”:“2021-06-02T20:07:25.74Z”}
{“thread”:“152329”,“loc”:null,“data”:{“kind”:“ChainSyncServerEvent.TraceChainSyncServerRead.RollBack”,“slot”:17020929,“block”:“c8e0010c753b5e6ce284926def428498ded9064cd7f085f190494f8d2208f794”,“blockNo”:5107443},“sev”:“Info”,“env”:“1.27.0:98a06”,“msg”:"",“app”:,“host”:“sm3”,“pid”:“3267799”,“ns”:[“cardano.node.ChainSyncHeaderServer”],“at”:“2021-06-02T20:07:45.75Z”}

Still has connection to all peers and shows no other issues (as far as I can asses…)
172.16.0.151 is a producer connected only to this relay and currently this producer is syncing as well to whatever relay can sync to.

what is the version of cardano-node? all config files up-to-date?

1.27.0 built from source.
This is not version used in tutorial but latest. Because of that I faced issue with missing alonzo config entry which I added (entry to config and config file downloaded from era config page). Otherwise node didn’t want to start at all. So above was the modification on general config I made.

ok, so genesis files are also up-to-date - from this site - Cardano Configurations

alonzo is from that site but shelley and byron are default which came with the source of 1.27.0. I assumed those should be fine, was it wrong assumption? :slight_smile: I will compare what I have with what’s on the site. Thanks!

As per below output, no difference between mine and site available genesis configs:

px2@sm3:~/cardano$ wget https://hydra.iohk.io/build/6498473/download/1/mainnet-byron-genesis.json
--2021-06-03 11:39:59--  https://hydra.iohk.io/build/6498473/download/1/mainnet-byron-genesis.json
Translacja hydra.iohk.io (hydra.iohk.io)... 147.75.83.98
Łączenie się z hydra.iohk.io (hydra.iohk.io)|147.75.83.98|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 200 OK
Długość: 1056360 (1.0M) [application/json]
Zapis do: `mainnet-byron-genesis.json'

mainnet-byron-genesis.json                 100%[======================================================================================>]   1.01M   402KB/s    w 2.6s     

2021-06-03 11:40:04 (402 KB/s) - zapisano `mainnet-byron-genesis.json' [1056360/1056360]

px2@sm3:~/cardano$ diff mainnet-byron-genesis.json /opt/cardano/cnode/files/
alonzo-genesis.json  byron-genesis.json   config.json          dbsync.json          genesis.json         topology.json        
px2@sm3:~/cardano$ diff mainnet-byron-genesis.json /opt/cardano/cnode/files/byron-genesis.json 
px2@sm3:~/cardano$ wget https://hydra.iohk.io/build/6498473/download/1/mainnet-shelley-genesis.json
--2021-06-03 11:40:43--  https://hydra.iohk.io/build/6498473/download/1/mainnet-shelley-genesis.json
Translacja hydra.iohk.io (hydra.iohk.io)... 147.75.83.98
Łączenie się z hydra.iohk.io (hydra.iohk.io)|147.75.83.98|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 200 OK
Długość: 2486 (2.4K) [application/json]
Zapis do: `mainnet-shelley-genesis.json'

mainnet-shelley-genesis.json               100%[======================================================================================>]   2.43K  --.-KB/s    w 0s       

2021-06-03 11:40:45 (432 MB/s) - zapisano `mainnet-shelley-genesis.json' [2486/2486]

px2@sm3:~/cardano$ diff mainnet-shelley-genesis.json /opt/cardano/cnode/files/
alonzo-genesis.json  byron-genesis.json   config.json          dbsync.json          genesis.json         topology.json        
px2@sm3:~/cardano$ diff mainnet-shelley-genesis.json /opt/cardano/cnode/files/genesis.json 
px2@sm3:~/cardano$ wget https://hydra.iohk.io/build/6498473/download/1/mainnet-alonzo-genesis.json
--2021-06-03 11:41:27--  https://hydra.iohk.io/build/6498473/download/1/mainnet-alonzo-genesis.json
Translacja hydra.iohk.io (hydra.iohk.io)... 147.75.83.98
Łączenie się z hydra.iohk.io (hydra.iohk.io)|147.75.83.98|:443... połączono.
Żądanie HTTP wysłano, oczekiwanie na odpowiedź... 200 OK
Długość: 309 [application/json]
Zapis do: `mainnet-alonzo-genesis.json'

mainnet-alonzo-genesis.json                100%[======================================================================================>]     309  --.-KB/s    w 0s       

2021-06-03 11:41:29 (51.0 MB/s) - zapisano `mainnet-alonzo-genesis.json' [309/309]

px2@sm3:~/cardano$ diff mainnet-alonzo-genesis.json /opt/cardano/cnode/files/alonzo-genesis.json 
px2@sm3:~/cardano$

Grepping log for errors (and ignoring ones between my relay and my producer), this is what I get:

{"thread":"38","loc":null,"data":{"kind":"TraceAddBlockEvent.AddBlockValidation.InvalidBlock","block":{"kind":"Point","hash":"448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d","slot":17020941},"error":"ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))"},"sev":"Error","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainDB"],"at":"2021-06-02T17:31:33.87Z"}
{"thread":"128365","loc":null,"data":{"kind":"ChainSyncClientEvent.TraceException","peer":{"remote":{"addr":"51.145.240.140","port":"17654"},"local":{"addr":"172.16.0.153","port":"42047"}},"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainSyncClient"],"at":"2021-06-02T17:31:33.88Z"}
{"thread":"128316","loc":null,"data":{"kind":"ChainSyncClientEvent.TraceException","peer":{"remote":{"addr":"110.174.184.26","port":"3001"},"local":{"addr":"172.16.0.153","port":"41377"}},"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainSyncClient"],"at":"2021-06-02T17:31:33.88Z"}
{"thread":"128330","loc":null,"data":{"kind":"ChainSyncClientEvent.TraceException","peer":{"remote":{"addr":"14.201.73.146","port":"3001"},"local":{"addr":"172.16.0.153","port":"39649"}},"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainSyncClient"],"at":"2021-06-02T17:31:33.88Z"}
{"thread":"139602","loc":null,"data":{"kind":"ChainSyncClientEvent.TraceException","peer":{"remote":{"addr":"61.69.254.98","port":"3001"},"local":{"addr":"172.16.0.153","port":"36303"}},"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 17020941, blockPointHash = 448a53bfbb7aa2671d426e01c6a5e7ade16ab14af8d18417aa8b73c3a000397d})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"e1f6cb9f65d11425be3aefc5ff5ac2f2d33f885ddfde3c258c4b23f5\")},Coin 351635274)])))),LedgersFailure (LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \"a16b3e08f2ee91cf9b62075e45e3908f523014f9eba3ca188dd3c1b7\")},Coin 1018419043)]))))])}))))))"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ChainSyncClient"],"at":"2021-06-02T17:31:33.89Z"}
{"thread":"64","loc":null,"data":{"address":"203.159.80.226:45726","kind":"ErrorPolicyTrace","event":"ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed \"<socket: 34> closed when reading data, waiting on next header True\"))) 20s 20s"},"sev":"Warning","env":"1.27.0:98a06","msg":"","app":[],"host":"sm3","pid":"3267799","ns":["cardano.node.ErrorPolicy"],"at":"2021-06-02T20:03:50.60Z"}

this looks for me like protocol or blockchain error, not really specific to my deployment. Can anyone confirm or provide some guidance how to read that?

What command did u used to compile the node… did u also used git pull origin master ?

1 Like

mkdir “$HOME/tmp”
cd “$HOME/tmp”
curl -sS -o prereqs.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/prereqs.sh
chmod 755 prereqs.sh
./prereqs.sh
. “${HOME}/.bashrc”
cd ~/git
git clone GitHub - input-output-hk/cardano-node: The core component that is used to participate in a Cardano decentralised blockchain.
cd cardano-node

git fetch --tags --all
git checkout 1.27.0
git pull origin master

echo -e “package cardano-crypto-praos\n flags: -external-libsodium-vrf” > cabal.project.local
$CNODE_HOME/scripts/cabal-build-all.sh -o