Block Producing sync not showing up in GView after 10 hours

Staking & Delegation Setup a Stake Pool
#1

My block producer node has been syncing for over 10 hours but GView is still not showing any progress. This is what GView looks like on my block producer:

Screenshot from 2021-10-29 09-58-51

My relay is fully synced and shows 0 in 2 out. The nodes are on two separate machines in same lan. Port 6000 is showing as open. Topology for both is my public IP. Both BP and relay journals appear to show syncing occurring, but also sometimes show the famous “Failed to start all required subscriptions” message.

My relay is using a Cat6 Ethernet cable while this machine is on Cat5E, don’t know if that makes a difference really. Below is some of the recent BP journal logs with my IP address removed:

Oct 29 10:04:03 jack-MS-7A74 cardano-node[2613]: [jack-MS-:cardano.node.IpSubscription:Error:64815] [2021-10-29 14:04:03.11 UTC] IPs: 0.0.0.0:0 [MY PUBLIC IP:6000] Application Exception: MY PUBLIC IP:6000 ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) (': * (ShelleyBlock (AlonzoEra StandardCrypto)) ('[] *)))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) (': * (ShelleyBlock (AlonzoEra StandardCrypto)) ('[] *)))))))) (ServerAgency TokNext TokMustReply)
Oct 29 10:04:03 jack-MS-7A74 cardano-node[2613]: [jack-MS-:cardano.node.IpSubscription:Info:64815] [2021-10-29 14:04:03.11 UTC] IPs: 0.0.0.0:0 [MY PUBLIC IP:6000] Closed socket to MY PUBLIC IP:6000
Oct 29 10:04:03 jack-MS-7A74 cardano-node[2613]: [jack-MS-:cardano.node.ErrorPolicy:Notice:53] [2021-10-29 14:04:03.11 UTC] IP MY PUBLIC IP:6000 ErrorPolicySuspendConsumer (Just (ApplicationExceptionTrace ExceededTimeLimit (ChainSync (Header (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) (': * (ShelleyBlock (AlonzoEra StandardCrypto)) ('[] *)))))))) (Tip HardForkBlock (': * ByronBlock (': * (ShelleyBlock (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Allegra StandardCrypto)) (': * (ShelleyBlock (ShelleyMAEra 'Mary StandardCrypto)) (': * (ShelleyBlock (AlonzoEra StandardCrypto)) ('[] *)))))))) (ServerAgency TokNext TokMustReply))) 20s
Oct 29 10:04:04 jack-MS-7A74 cardano-node[2613]: [jack-MS-:cardano.node.IpSubscription:Info:55] [2021-10-29 14:04:04.11 UTC] IPs: 0.0.0.0:0 [MY PUBLIC IP:6000] Restarting Subscription after 181.008426782s desired valency 1 current valency 0
Oct 29 10:04:04 jack-MS-7A74 cardano-node[2613]: [jack-MS-:cardano.node.IpSubscription:Info:55] [2021-10-29 14:04:04.11 UTC] IPs: 0.0.0.0:0 [MY PUBLIC IP:6000] Trying to connect to MY PUBLIC IP:6000
Oct 29 10:04:04 jack-MS-7A74 cardano-node[2613]: [jack-MS-:cardano.node.IpSubscription:Notice:55] [2021-10-29 14:04:04.11 UTC] IPs: 0.0.0.0:0 [MY PUBLIC IP:6000] Waiting 0.025s before attempting a new connection
Oct 29 10:04:04 jack-MS-7A74 cardano-node[2613]: [jack-MS-:cardano.node.IpSubscription:Info:65128] [2021-10-29 14:04:04.11 UTC] IPs: 0.0.0.0:0 [MY PUBLIC IP:6000] Connection Attempt Start, destination MY PUBLIC IP:6000
Oct 29 10:04:04 jack-MS-7A74 cardano-node[2613]: [jack-MS-:cardano.node.ErrorPolicy:Warning:63] [2021-10-29 14:04:04.11 UTC] IP 192.168.10.1:33801 ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed "<socket: 108> closed when reading data, waiting on next header True"))) 20s 20s
Oct 29 10:04:04 jack-MS-7A74 cardano-node[2613]: [jack-MS-:cardano.node.IpSubscription:Notice:65128] [2021-10-29 14:04:04.11 UTC] IPs: 0.0.0.0:0 [MY PUBLIC IP:6000] Connection Attempt End, destination MY PUBLIC IP:6000 outcome: ConnectSuccessLast
Oct 29 10:04:04 jack-MS-7A74 cardano-node[2613]: [jack-MS-:cardano.node.ChainSyncHeaderServer:Info:65134] [2021-10-29 14:04:04.11 UTC] fromList [("kind",String "ChainSyncServerEvent.TraceChainSyncServerRead.RollBack"),("slot",Number 0.0),("block",String "d2d90d10993b3eb309f5485a5413cb9c9fe4f088e79b7b4e815dbbe8fcb8e4b7"),("blockNo",Number 0.0)]

Any ideas why this is taking so long? Should I just wait it out?

#2

You should use the private/LAN IP to connect them…

now, inside the Producer topology files, which node do you have ? Only the Relay? If yes add also the IOHK nodes and restart the node

try
sudo systemctl restart cardano-node
journalctl -e -f -u cardano-node

share the output

1 Like
#3

Thanks for your reply Alex. You’ve been very helpful throughout this process. My producer topology currently looks like this:

 {
    "Producers": [
      {
        "addr": "MY PUBLIC IP",
        "port": 6000,
        "valency": 1
      }
    ]
  }

Could you clarify what you mean by add the IOHKs nodes? The output I received using
sudo systemctl restart cardano-node
journalctl -e -f -u cardano-node
was the same output as I shared above.

#4

Ok, delete all from topology file and add


{
  "Producers": [
    {
      "addr": "relays-new.cardano-mainnet.iohk.io",
      "port": 3001,
      "valency": 2
    }
  ]
}

save the file and restart the node
then check with top you should see the CPU ~100%
then check glive

Now check if the relay port is open

https://www.yougetsignal.com/tools/open-ports/

You will need a port forward on your local router for the relay port 6000

All incoming traffic with destination port 6000 should be forwarded to Relay_private_IP port 6000

The relay and producer should be connected via private IP

  • on Producer allow ONLY the relay to connect on port 6000 (the rule should be added on FW)
  • on Relay allow any to connect on port 6000 (FW rule)

You can test with

  • from Producer : telnet Relay_private_IP Relay_port (u should see connected)
  • from Relay : telnet Producer_private_IP Producer_port (u should see connected)
#5

Alex you’re the man! My BP is syncing now. The only thing is it only shows 2 peers out now, no peers in. GView also reads as “Relay” at the top which is strange, but I definitely configured this node as a producer…weird. The yougetsignal site is reading port 6000 as open still which is great.

Are you saying I should change the topology of my Relay to private IP instead of public?

#6

Yeah, if are under the same LAN u should use private IPs

PS: the Producer will start as a CORE node when it will be 100% synced

1 Like
#7

Awesome. Can’t thank you enough man. :slightly_smiling_face:

#8

no problem, anytime

1 Like
#9

Hey Alex, both relay and block producer are fully synced to the blockchain, the only issue I’m still having is that my BP is still showing up as a relay, despite my startup scripts appearing to be correct. Here’s what is inside the “startBlockProducingNode.sh” file:

#!/bin/bash
DIRECTORY=/home/jack/cardano-my-node
PORT=6000
HOSTADDR=0.0.0.0
TOPOLOGY=${DIRECTORY}/mainnet-topology.json
DB_PATH=${DIRECTORY}/db
SOCKET_PATH=${DIRECTORY}/db/socket
CONFIG=${DIRECTORY}/mainnet-config.json
/usr/local/bin/cardano-node run +RTS -N -A16m -qg -qb -RTS --topology ${TOPOLOGY} --database-path ${DB_PATH} --socket-path ${SOCKET_PATH} --host-addr ${HOSTADDR} --port ${PORT} --config ${CONFIG}
#10

It’s not ok, u didn’t updated with the pool files (check the step on coicashew guide)

1 Like
#11

Ahh I think I understand. I haven’t moved on from step 8 yet, wanted to make sure both nodes were good to go before generating keys. But it looks like I need to do step 9 prior to the node showing up as Core.

#12

Just follow the steps, if u will have issues let me know

1 Like
#13

Alex I’m having some trouble setting up the air-gap offline machine. I managed to move the cardano-cli file from my block producer via USB to my offline machine, then used the mv command to move that file into the /usr/local/bin folder. However, upon inputting the following into the offline machine’s terminal:

cardano-cli node key-gen \
    --cold-verification-key-file node.vkey \
    --cold-signing-key-file $HOME/cold-keys/node.skey \
    --operational-certificate-issue-counter node.counter

…I get this message:

-bash /usr/local/bin/cardano-cli: Permission denied

I have tried logging into the terminal as the root user, but this has the same result. Any ideas?

#14

yes, because… you need to change the permision of the files

cd /usr/local
ls -l

cand u check the permision? are the same with the live server?

1 Like
#15

Hey Alex, both my block producer and offline machine appear to show the same result with that input:

BP

jack@jack-MS-7A74:~$ cd /usr/local
ls -l
total 32
drwxr-xr-x 2 root root 4096 Oct 25 12:36 bin
drwxr-xr-x 2 root root 4096 Aug 19 06:29 etc
drwxr-xr-x 2 root root 4096 Aug 19 06:29 games
drwxr-xr-x 3 root root 4096 Oct 27 20:19 include
drwxr-xr-x 4 root root 4096 Oct 27 20:19 lib
lrwxrwxrwx 1 root root    9 Oct 25 11:01 man -> share/man
drwxr-xr-x 2 root root 4096 Aug 19 06:29 sbin
drwxr-xr-x 7 root root 4096 Aug 19 06:40 share
drwxr-xr-x 2 root root 4096 Aug 19 06:29 src

Offline machine:

root@jack-HP-EliteBook-8440p:~# cd /usr/local
ls -l
total 32
drwxr-xr-x 2 root root 4096 Oct 30 16:06 bin
drwxr-xr-x 2 root root 4096 Aug 19 06:29 etc
drwxr-xr-x 2 root root 4096 Aug 19 06:29 games
drwxr-xr-x 3 root root 4096 Oct 25 18:20 include
drwxr-xr-x 4 root root 4096 Oct 25 18:20 lib
lrwxrwxrwx 1 root root    9 Oct 25 17:48 man -> share/man
drwxr-xr-x 2 root root 4096 Aug 19 06:29 sbin
drwxr-xr-x 7 root root 4096 Aug 19 06:40 share
drwxr-xr-x 2 root root 4096 Aug 19 06:29 src
#16

Try this on offline machine

cd /usr/local/bin
ls -l

chmod +x cardano-cli

1 Like