Failed to start all required subscriptions with new relay

I am getting crazy with my setup. I have now a working BP and Relay node. The BP is already working with his KES keys.
I added another relay to be connected to that BP. I made a full rsync of the “cardano-node” directory from the exsting relay to the new one. Then all the necesary “make install” were made to have the same binaries and DB.
After that the new ralay does not sync and it does not make any connection with the public relays or the BP

The BP shows the next errors:
Dec 20 07:27:10 n01 cardano-node[1425463]: [n01:cardano.node.Forge:Info:114] [2021-12-20 13:27:10.00 UTC] fromList [(“val”,Object (fromList [(“kind”,String “TraceNodeNotLeader”),(“slot”,Number 4.8440539e7)])),(“credentials”,String “Cardano”)]
Dec 20 07:27:10 n01 cardano-node[1425463]: [n01:cardano.node.IpSubscription:Info:124] [2021-12-20 13:27:10.22 UTC] IPs: 0.0.0.0:0 [192.168.100.201:6000,192.168.100.202:6000] Skipping peer 192.168.100.201:6000
Dec 20 07:27:10 n01 cardano-node[1425463]: [n01:cardano.node.IpSubscription:Info:124] [2021-12-20 13:27:10.22 UTC] IPs: 0.0.0.0:0 [192.168.100.201:6000,192.168.100.202:6000] Trying to connect to 192.168.100.202:6000
Dec 20 07:27:10 n01 cardano-node[1425463]: [n01:cardano.node.IpSubscription:Error:124] [2021-12-20 13:27:10.22 UTC] IPs: 0.0.0.0:0 [192.168.100.201:6000,192.168.100.202:6000] Failed to start all required subscriptions
Dec 20 07:27:11 n01 cardano-node[1425463]: [n01:cardano.node.LeadershipCheck:Info:114] [2021-12-20 13:27:11.00 UTC] {“kind”:“TraceStartLeadershipCheck”,“chainDensity”:4.888597e-2,“slot”:48440540,“delegMapSize”:1006025,“utxoSize”:4665795,“credentials”:“Cardano”}

And the new relay:
Dec 20 07:27:06 p01cardano-node[10082]: [p01:cardano.node.ErrorPolicy:Warning:74] [2021-12-20 13:27:06.90 UTC] IP 52.58.171.193:3001 ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (InvalidBlock (At (Block {blockPointSlot = SlotNo 39196055, blockPointHash = ba5318c5af530fd3d7b1b42e0219d5a92b1a32ac300c75a9261336a1a0ce703e})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (UtxowFailure (InvalidWitnessesUTXOW [VKey (VerKeyEd25519DSIGN “393b3d3cab6be0897cbc871434c3a24825dc1d8249df4d37969502931cf856ff”)]))),LedgersFailure (LedgerFailure (UtxowFailure (MissingVKeyWitnessesUTXOW (WitHashes (fromList [KeyHash “6904e8b2c26f3dda6c4a5db4b3ec9e31d581c9960977cfe9c6917a43”]))))),InvalidBodyHashBBODY “0b7f3003e19d94ccc9fba4b5aaf30db493480dedf2021d1da9d6eeb5db6a2e7b” “46cf12e2f414252f0eb893472b2a581610d55fb48e7c9cfc56bfcea10ec100b3”])})))))))))) 200s 200s
Dec 20 07:27:07 p01cardano-node[10082]: [papalote:cardano.node.IpSubscription:Info:73] [2021-12-20 13:27:07.17 UTC] IPs: 0.0.0.0:0 [192.168.100.211:6000] Restarting Subscription after 1.001047234s desired valency 1 current valency 0
Dec 20 07:27:16 p01cardano-node[10082]: [p01:cardano.node.IpSubscription:Info:73] [2021-12-20 13:27:16.17 UTC] IPs: 0.0.0.0:0 [192.168.100.211:6000] Skipping peer 192.168.100.211:6000
Dec 20 07:27:16 p01cardano-node[10082]: [p01:cardano.node.IpSubscription:Error:73] [2021-12-20 13:27:16.17 UTC] IPs: 0.0.0.0:0 [192.168.100.211:6000] Failed to start all required subscriptions
Dec 20 07:27:16 p01cardano-node[10082]: [p01:cardano.node.DnsSubscription:Notice:1000] [2021-12-20 13:27:16.18 UTC] Domain: “relays-new.cardano-mainnet.iohk.io” Connection Attempt Start, destination 3.9.116.228:3001

I do not understand why.

Here the topology file of the BP:

 {
    "Producers": [
      {
        "addr": "192.168.100.201",
        "port": 6000,
        "valency": 1
      },
      {
        "addr": "192.168.100.202",
        "port": 6000,
        "valency": 1
      }
    ]
  }

And the topolgy of the new relay (to be noted that both relays have extact the same topology because they point to the same BP):

 {
    "Producers": [
      {
        "addr": "192.168.100.211",
        "port": 6000,
        "valency": 1
      },
      {
       "addr": "relays-new.cardano-mainnet.iohk.io",
       "port": 3001,
       "valency": 2
      }
    ]
  }

What version did u installed?

Hi,
It is
$ cardano-cli --version
cardano-cli 1.31.0 - linux-x86_64 - ghc-8.10

download again the files from here and restart the node

https://hydra.iohk.io/build/7654130/download/1/index.html

Hi,
I do not know why I need to download them again if they were a rsync copy from a working relay. After all I download it and started. No changes in the topology file, so my BP not there:
``

[adamin@papalote01 ~]$ cat cardano-node/mainnet-topology.json
{
  "Producers": [
    {
      "addr": "relays-new.cardano-mainnet.iohk.io",
      "port": 3001,
      "valency": 2
    }
  ]
}

And again:

Dec 20 14:18:26 p01 cardano-node[2215]: [p01:cardano.node.DnsSubscription:Warning:76] [2021-12-20 20:18:26.43 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Failed to start all required subscriptions

Now, I added my BP into. Checking for another logs, I see the next too:

Dec 20 14:31:12 p01 cardano-node[6448]: [p01:cardano.node.ErrorPolicy:Warning:73] [2021-12-20 20:31:12.24 UTC] IP 3.21.187.168:3001 ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (InvalidBlock (At (Block {blockPointSlot = SlotNo 39196055, blockPointHash = ba5318c5af530fd3d7b1b42e0219d5a92b1a32ac300c75a9261336a1a0ce703e})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (UtxowFailure (InvalidWitnessesUTXOW [VKey (VerKeyEd25519DSIGN "393b3d3cab6be0897cbc871434c3a24825dc1d8249df4d37969502931cf856ff")]))),LedgersFailure (LedgerFailure (UtxowFailure (MissingVKeyWitnessesUTXOW (WitHashes (fromList [KeyHash "6904e8b2c26f3dda6c4a5db4b3ec9e31d581c9960977cfe9c6917a43"]))))),InvalidBodyHashBBODY "0b7f3003e19d94ccc9fba4b5aaf30db493480dedf2021d1da9d6eeb5db6a2e7b" "46cf12e2f414252f0eb893472b2a581610d55fb48e7c9cfc56bfcea10ec100b3"])})))))))))) 200s 200s
Dec 20 14:31:12 p01 cardano-node[6448]: [p01:cardano.node.IpSubscription:Info:75] [2021-12-20 20:31:12.90 UTC] IPs: 0.0.0.0:0 [192.168.100.211:6000] Restarting Subscription after 1.026103991s desired valency 1 current valency 0
Dec 20 14:32:51 p01 cardano-node[6448]: [p01:cardano.node.IpSubscription:Info:75] [2021-12-20 20:32:51.88 UTC] IPs: 0.0.0.0:0 [192.168.100.211:6000] Skipping peer 192.168.100.211:6000

How did u started the node? Did u used the IP address or 0.0.0.0?

Yes, 0.0.0.0

I was wondering: Did it have a limitation with the hostname? I mean, longer than 8 chars?

Now I added my BP. Grabbing for more logs:

Dec 20 14:37:01 p01cardano-node[6448]: [p01:cardano.node.IpSubscription:Info:75] [2021-12-20 20:37:0
1.92 UTC] IPs: 0.0.0.0:0 [192.168.100.211:6000] Skipping peer 192.168.100.211:6000
Dec 20 14:37:02 p01cardano-node[6448]: [p01:cardano.node.DnsSubscription:Notice:1789] [2021-12-20 20:37:02.13 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt End, destination 54.250.21.52:3001 outcome: ConnectSuccessLast
Dec 20 14:37:02 p01cardano-node[6448]: [p01:cardano.node.ErrorPolicy:Notice:73] [2021-12-20 20:37:02.13 UTC] IP 18.136.216.144:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s
Dec 20 14:32:51 p01cardano-node[6448]: [p01:cardano.node.IpSubscription:Info:75] 12-20 20:32:51.88 UTC] IPs: 0.0.0.0:0 [192.168.100.211:6000] Skipping peer 192.168.100.211:600

What do u mean… u must use the username only when u have a DNS set for ur IP, try to set it to default and restart the node
and use 0.0.0.0
also did u used port 6000?

I am talking about this script

cat > $NODE_HOME/startRelayNode1.sh << EOF 
#!/bin/bash
DIRECTORY=$NODE_HOME
PORT=6000
HOSTADDR=0.0.0.0
TOPOLOGY=\${DIRECTORY}/${NODE_CONFIG}-topology.json
DB_PATH=\${DIRECTORY}/db
SOCKET_PATH=\${DIRECTORY}/db/socket
CONFIG=\${DIRECTORY}/${NODE_CONFIG}-config.json
/usr/local/bin/cardano-node run +RTS -N -A16m -qg -qb -RTS --topology \${TOPOLOGY} --database-path \${DB_PATH} --socket-path \${SOCKET_PATH} --host-addr \${HOSTADDR} --port \${PORT} --config \${CONFIG}
EOF

On those messages
“p01:cardano.node.ErrorPolicy:Notice”
I am replacing here the hostname (security reasons) for the “p01” string in the forum/support. Originally I have something as next:
“pxxxxxxx:cardano.node.ErrorPolicy:Notice”
But the hostname is longer than 8 chars, like: “pxxxxxxx01”. In all messages from the working relay I see the hostname without the “01”. After the rsync to the new relay, I started the service. My surprise is to see the same hostname “pxxxxxxx:cardano.node.ErrorPolicy:Notice” in place of “pxxxxxxx02”

Yes, there I use 0.0.0.0. It is a rsync from the a working relay.

and where did u set the hostnane? Ahh u are refering to logs…

if u type

telnet 192.168.100.211 6000 do u see the port opened?

Yes, I mean in the logs. because it let me think that the relays are “registered” on some way using his hostname to get such “Ipsubscriptions” and, because the working relay has the name pxxxxxxx01 but is reguistered as pxxxxxxx, then, later on, comes the new relay with te name pxxxxxxx02 and is “registered” without the 02.
The port is open. The working relay reaches it.

My assumption comes from the next:

] [2021-12-20 20:46:36.81 UTC] IP 18.182.219.85:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s
 [2021-12-20 20:46:36.81 UTC] IP 54.250.21.52:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s

After 16hrs we can “chat” further (limitation of the forum)

can u share the glive output?

Hi,
Since yesterday is syncing! As I noted, the hostnames (at OS level) should no be longer than 8 chars. Deleted the DB on the new one and re-sync it from the working node. Now it is working.

I would suggest to put it on the documentation.

1 Like

let’s see if will cross epoch 297

Thx for the feedbach

What do you mean? It is already in sync on epoch 309
01

perfect, then … the hostname was the issue :thinking: