Stuck with topologyUpdater.. Network help

Staking & Delegation Operate a Stake Pool
1

I moved my nodes to new hardware and just about everything transferred nicely except my topologyUpdater.sh doesnt want to work properly… Getting the following error:

{ “resultcode”: “403”, “datetime”:“2022-02-28 15:33:20”, “clientIp”: “107.190.67.120”, “iptype”: 4, “│
msg”: “glad you want to stay with us, but please check and enable your IP:port reachability” }

However. My ports are configured properly. https://pool.vet/#yyc shows my nodes are connectable. and my SLiveView shows incoming peers. What else can I look at?

image
image1637×1266 322 KB

2

check if u opened the port on fw…

3

After looking at your information at pool.vet, here are my comments:

  • Both of your node1.cardanoyyc.ca and node2.cardanoyyc.ca has the IP of 107.190.67.120.There is something wrong here, because the two must not have the same IP.
  • You are using port 3000 for relay1 and 3006 for relay2. My recommendation is to leave things as they are, both nodes can use the default port 6000 with no problem. You don’t need to make them different.
  • I have checked your IP 107.190.67.120, and port 3000 and 3006 are open.

Therefore, I assume your problem is in the topo files.

4

Or, if the nodes ore behind same public IP (private network) then check the PF configuration… u will need 2 rules

relay 1:
All traffic from outside with destination port relay1 → forward the traffic to relay1 port relay1

relay 2:
all traffic from outside with destination port relay2 → forward the traffic to relay2 port relay2

5

Firewall is inactive for diagnosing this issue.

@DevJohn I fail to see how the topo files are interfering with the topologyUpdater.sh script? Ports appear to be open as you stated.

@Alexd1985 Yes I already have my LAN configured from the previous setup. nothing changed there except some internal IP. See image:

This just seems super weird. the ports are working. traffic is flowing 2 ways. but topologyUpdater.sh complains?

Most appreciated your help.

image

6

Then is something related with the DNS, please check perhaps something changed…

7

We could use more information.

Are you using topologyUpdater.sh? If so, please share its contents for both of relay1 and relay2.

Also share the topology.json for the node, and the two relays.

8

Internally 192.168.42.9 is my BP.

Here is my topologyUpdater.sh for Node 1

#!/usr/bin/env bash
# shellcheck disable=SC2086,SC2034
# shellcheck source=/dev/null

PARENT="$(dirname $0)"
[[ -f "${PARENT}"/env ]] && . "${PARENT}"/env offline

######################################
# User Variables - Change as desired #
######################################

CNODE_HOSTNAME="node1.cardanoyyc.ca"  # (Optional) Must resolve to the IP you are requesting from
CNODE_PORT="3000"
CNODE_VALENCY=1             # (Optional) for multi-IP hostnames
MAX_PEERS=10                 # Maximum number of peers to return on successful fetch (note that a single peer may include valency of up to 3)
CUSTOM_PEERS="192.168.42.8:3000|192.168.42.9:3000"        # *Additional* custom peers to (IP,port[,valency]) to add to your target topology.json
BATCH_AUTO_UPDATE=Y        # Set to Y to automatically update the script if a new version is available without user interaction

######################################
# Do NOT modify code below           #
######################################

PARENT="$(dirname $0)"
[[ -f "${PARENT}"/.env_branch ]] && BRANCH="$(cat ${PARENT}/.env_branch)" || BRANCH="master"

usage() {
  cat <<-EOF
		Usage: $(basename "$0") [-b <branch name>] [-f] [-p]
		Topology Updater - Build topology with community pools

		-f    Disable fetch of a fresh topology file
		-p    Disable node alive push to Topology Updater API
		-b    Use alternate branch to check for updates - only for testing/development (Default: master)
		
		EOF
  exit 1
}

TU_FETCH='Y'
TU_PUSH='Y'

while getopts :fpb: opt; do
  case ${opt} in
    f ) TU_FETCH='N' ;;
    p ) TU_PUSH='N' ;;
    b ) echo "${OPTARG}" > "${PARENT}"/.env_branch ;;
    \? ) usage ;;
  esac
done
shift $((OPTIND -1))

[[ -z "${BATCH_AUTO_UPDATE}" ]] && BATCH_AUTO_UPDATE=N

#######################################################
# Version Check                                       #
#######################################################
clear

if [[ ! -f "${PARENT}"/env ]]; then
  echo -e "\nCommon env file missing: ${PARENT}/env"
  echo -e "This is a mandatory prerequisite, please install with prereqs.sh or manually download from GitHub\n"
  exit 1
fi

. "${PARENT}"/env offline &>/dev/null # ignore any errors, re-sourced later

if [[ "${UPDATE_CHECK}" == "Y" ]] && [[ "${BATCH_AUTO_UPDATE}" == "N" ]]; then
  echo "Checking for script updates..."
  # Check availability of checkUpdate function
  if [[ ! $(command -v checkUpdate) ]]; then
    echo -e "\nCould not find checkUpdate function in env, make sure you're using official guild docos for installation!"
    exit 1
  fi
  # check for env update
  ! checkUpdate env ${BATCH_AUTO_UPDATE} && exit 1
  ! checkUpdate topologyUpdater.sh ${BATCH_AUTO_UPDATE} && exit 1
  # source common env variables in case it was updated
  . "${PARENT}"/env offline &>/dev/null
  case $? in
    0) : ;; # ok
    2) echo "continuing with topology update..." ;;
    *) exit 1 ;;
  esac
fi

# Check if old style CUSTOM_PEERS with colon separator is used, if so convert to use commas
if [[ -n ${CUSTOM_PEERS} && ${CUSTOM_PEERS} != *","* ]]; then
  CUSTOM_PEERS=${CUSTOM_PEERS//[:]/,}
fi

if [[ ${TU_PUSH} = "Y" ]]; then
  fail_cnt=0
  while ! blockNo=$(curl -s -f -m ${EKG_TIMEOUT} -H 'Accept: application/json' "http://${EKG_HOST}:${EKG_PORT}/" 2>/dev/null | jq -er '.cardano.node.metrics.blockNum.int.val //0' ); do
    ((fail_cnt++))
    [[ ${fail_cnt} -eq 5 ]] && echo "5 consecutive EKG queries failed, aborting!"
    echo "(${fail_cnt}/5) Failed to grab blockNum from node EKG metrics, sleeping for 30s before retrying... (ctrl-c to exit)"
    sleep 30
  done
fi

if [[ -n ${CNODE_HOSTNAME} && "${CNODE_HOSTNAME}" != "CHANGE ME" ]]; then
  T_HOSTNAME="&hostname=${CNODE_HOSTNAME}"
else
  T_HOSTNAME=''
fi

if [[ ${TU_PUSH} = "Y" ]]; then
  if [[ ${IP_VERSION} = "4" || ${IP_VERSION} = "mix" ]]; then
    curl -s -f -4 "https://api.clio.one/htopology/v1/?port=${CNODE_PORT}&blockNo=${blockNo}&valency=${CNODE_VALENCY}&magic=${NWMAGIC}${T_HOSTNAME}" | tee -a "${LOG_DIR}"/topologyUpdater_lastresult.json
  fi
  if [[ ${IP_VERSION} = "6" || ${IP_VERSION} = "mix" ]]; then
    curl -s -f -6 "https://api.clio.one/htopology/v1/?port=${CNODE_PORT}&blockNo=${blockNo}&valency=${CNODE_VALENCY}&magic=${NWMAGIC}${T_HOSTNAME}" | tee -a "${LOG_DIR}"/topologyUpdater_lastresult.json
  fi
fi
if [[ ${TU_FETCH} = "Y" ]]; then
  if [[ ${IP_VERSION} = "4" || ${IP_VERSION} = "mix" ]]; then
    curl -s -f -4 -o "${TOPOLOGY}".tmp "https://api.clio.one/htopology/v1/fetch/?max=${MAX_PEERS}&magic=${NWMAGIC}&ipv=${IP_VERSION}"
  else
    curl -s -f -6 -o "${TOPOLOGY}".tmp "https://api.clio.one/htopology/v1/fetch/?max=${MAX_PEERS}&magic=${NWMAGIC}&ipv=${IP_VERSION}"
  fi
  if [[ -n "${CUSTOM_PEERS}" ]]; then
    topo="$(cat "${TOPOLOGY}".tmp)"
    IFS='|' read -ra cpeers <<< "${CUSTOM_PEERS}"
    for cpeer in "${cpeers[@]}"; do
      IFS=',' read -ra cpeer_attr <<< "${cpeer}"
      case ${#cpeer_attr[@]} in
        2) addr="${cpeer_attr[0]}"
           port=${cpeer_attr[1]}
           valency=1 ;;
        3) addr="${cpeer_attr[0]}"
           port=${cpeer_attr[1]}
           valency=${cpeer_attr[2]} ;;
        *) echo "ERROR: Invalid Custom Peer definition '${cpeer}'. Please double check CUSTOM_PEERS definition"
           exit 1 ;;
      esac
      if [[ ${addr} = *.* ]]; then
        ! isValidIPv4 "${addr}" && echo "ERROR: Invalid IPv4 address or hostname '${addr}'. Please check CUSTOM_PEERS definition" && continue
      elif [[ ${addr} = *:* ]]; then
        ! isValidIPv6 "${addr}" && echo "ERROR: Invalid IPv6 address '${addr}'. Please check CUSTOM_PEERS definition" && continue
      fi
      ! isNumber ${port} && echo "ERROR: Invalid port number '${port}'. Please check CUSTOM_PEERS definition" && continue
      ! isNumber ${valency} && echo "ERROR: Invalid valency number '${valency}'. Please check CUSTOM_PEERS definition" && continue
      topo=$(jq '.Producers += [{"addr": $addr, "port": $port|tonumber, "valency": $valency|tonumber}]' --arg addr "${addr}" --arg port ${port} --arg valency ${valency} <<< "${topo}")
    done
    echo "${topo}" | jq -r . >/dev/null 2>&1 && echo "${topo}" > "${TOPOLOGY}".tmp
  fi
  mv "${TOPOLOGY}".tmp "${TOPOLOGY}"
fi
exit 0

And the resulting mainnet-topology.json for Node 1

{
  "resultcode": "201",
  "networkMagic": "764824073",
  "ipType": 4,
  "requestedIpVersion": "4",
  "max": "10",
  "Producers": [
    {
      "addr": "adarelay02.psilobyte.io",
      "port": 3002,
      "valency": 1,
      "distance": 282,
      "continent": "NA",
      "country": "CA",
      "region": "AB"
    },
    {
      "addr": "relay0.8sqrdpools.com",
      "port": 6000,
      "valency": 1,
      "distance": 1921,
      "continent": "NA",
      "country": "US",
      "region": "CA"
    },
    {
      "addr": "3.18.168.22",
      "port": 6000,
      "valency": 1,
      "distance": 2671,
      "continent": "NA",
      "country": "US",
      "region": "OH"
    },
    {
      "addr": "54.159.255.195",
      "port": 6000,
      "valency": 1,
      "distance": 3110,
      "continent": "NA",
      "country": "US",
      "region": "VA"
    },
    {
      "addr": "75.119.133.13",
      "port": 3000,
      "valency": 1,
      "distance": 3947,
      "continent": "NA",
      "country": "US",
      "region": "FL"
    },
    {
      "addr": "antrel.dyn.ydns.io",
      "port": 54833,
      "valency": 1,
      "distance": 7158,
      "continent": "EU",
      "country": "NL",
      "region": "NH"
    },
    {
      "addr": "94.237.99.68",
      "port": 8001,
      "valency": 1,
      "distance": 7512,
      "continent": "EU",
      "country": "FI",
      "region": "HE"
    },
    {
      "addr": "157.90.119.14",
      "port": 6000,
      "valency": 1,
      "distance": 7687,
      "continent": "EU",
      "country": "DE",
      "region": "BY"
    },
    {
      "addr": "207.180.243.108",
      "port": 6000,
      "valency": 1,
      "distance": 7815,
      "continent": "EU",
      "country": "DE",
      "region": "BY"
    },
    {
      "addr": "34.80.179.174",
      "port": 6000,
      "valency": 1,
      "distance": 9952,
      "continent": "AS",
      "country": "TW",
      "region": "CYI"
    },
    {
      "addr": "192.168.42.8",
      "port": 3000,
      "valency": 1
    },
    {
      "addr": "192.168.42.9",
      "port": 3000,
      "valency": 1
    }
  ]
}

Here is the topologyUpdater.sh for node 2

#!/usr/bin/env bash
# shellcheck disable=SC2086,SC2034
# shellcheck source=/dev/null

PARENT="$(dirname $0)"
[[ -f "${PARENT}"/env ]] && . "${PARENT}"/env offline

######################################
# User Variables - Change as desired #
######################################

CNODE_HOSTNAME="node2.cardanoyyc.ca"  # (Optional) Must resolve to the IP you are requesting from
CNODE_PORT="3006"
CNODE_VALENCY=1             # (Optional) for multi-IP hostnames
MAX_PEERS=8                 # Maximum number of peers to return on successful fetch (note that a single peer may include valency of up to 3)
CUSTOM_PEERS="192.168.42.6:3000|192.168.42.9:3000"        # *Additional* custom peers to (IP,port[,valency]) to add to your target topology.json
BATCH_AUTO_UPDATE=Y        # Set to Y to automatically update the script if a new version is available without user interaction

######################################
# Do NOT modify code below           #
######################################

PARENT="$(dirname $0)"
[[ -f "${PARENT}"/.env_branch ]] && BRANCH="$(cat ${PARENT}/.env_branch)" || BRANCH="master"

usage() {
  cat <<-EOF
		Usage: $(basename "$0") [-b <branch name>] [-f] [-p]
		Topology Updater - Build topology with community pools

		-f    Disable fetch of a fresh topology file
		-p    Disable node alive push to Topology Updater API
		-b    Use alternate branch to check for updates - only for testing/development (Default: master)
		
		EOF
  exit 1
}

TU_FETCH='Y'
TU_PUSH='Y'

while getopts :fpb: opt; do
  case ${opt} in
    f ) TU_FETCH='N' ;;
    p ) TU_PUSH='N' ;;
    b ) echo "${OPTARG}" > "${PARENT}"/.env_branch ;;
    \? ) usage ;;
  esac
done
shift $((OPTIND -1))

[[ -z "${BATCH_AUTO_UPDATE}" ]] && BATCH_AUTO_UPDATE=N

#######################################################
# Version Check                                       #
#######################################################
clear

if [[ ! -f "${PARENT}"/env ]]; then
  echo -e "\nCommon env file missing: ${PARENT}/env"
  echo -e "This is a mandatory prerequisite, please install with prereqs.sh or manually download from GitHub\n"
  exit 1
fi

. "${PARENT}"/env offline &>/dev/null # ignore any errors, re-sourced later

if [[ "${UPDATE_CHECK}" == "Y" ]] && [[ "${BATCH_AUTO_UPDATE}" == "N" ]]; then
  echo "Checking for script updates..."
  # Check availability of checkUpdate function
  if [[ ! $(command -v checkUpdate) ]]; then
    echo -e "\nCould not find checkUpdate function in env, make sure you're using official guild docos for installation!"
    exit 1
  fi
  # check for env update
  ! checkUpdate env ${BATCH_AUTO_UPDATE} && exit 1
  ! checkUpdate topologyUpdater.sh ${BATCH_AUTO_UPDATE} && exit 1
  # source common env variables in case it was updated
  . "${PARENT}"/env offline &>/dev/null
  case $? in
    0) : ;; # ok
    2) echo "continuing with topology update..." ;;
    *) exit 1 ;;
  esac
fi

# Check if old style CUSTOM_PEERS with colon separator is used, if so convert to use commas
if [[ -n ${CUSTOM_PEERS} && ${CUSTOM_PEERS} != *","* ]]; then
  CUSTOM_PEERS=${CUSTOM_PEERS//[:]/,}
fi

if [[ ${TU_PUSH} = "Y" ]]; then
  fail_cnt=0
  while ! blockNo=$(curl -s -f -m ${EKG_TIMEOUT} -H 'Accept: application/json' "http://${EKG_HOST}:${EKG_PORT}/" 2>/dev/null | jq -er '.cardano.node.metrics.blockNum.int.val //0' ); do
    ((fail_cnt++))
    [[ ${fail_cnt} -eq 5 ]] && echo "5 consecutive EKG queries failed, aborting!"
    echo "(${fail_cnt}/5) Failed to grab blockNum from node EKG metrics, sleeping for 30s before retrying... (ctrl-c to exit)"
    sleep 30
  done
fi

if [[ -n ${CNODE_HOSTNAME} && "${CNODE_HOSTNAME}" != "CHANGE ME" ]]; then
  T_HOSTNAME="&hostname=${CNODE_HOSTNAME}"
else
  T_HOSTNAME=''
fi

if [[ ${TU_PUSH} = "Y" ]]; then
  if [[ ${IP_VERSION} = "4" || ${IP_VERSION} = "mix" ]]; then
    curl -s -f -4 "https://api.clio.one/htopology/v1/?port=${CNODE_PORT}&blockNo=${blockNo}&valency=${CNODE_VALENCY}&magic=${NWMAGIC}${T_HOSTNAME}" | tee -a "${LOG_DIR}"/topologyUpdater_lastresult.json
  fi
  if [[ ${IP_VERSION} = "6" || ${IP_VERSION} = "mix" ]]; then
    curl -s -f -6 "https://api.clio.one/htopology/v1/?port=${CNODE_PORT}&blockNo=${blockNo}&valency=${CNODE_VALENCY}&magic=${NWMAGIC}${T_HOSTNAME}" | tee -a "${LOG_DIR}"/topologyUpdater_lastresult.json
  fi
fi
if [[ ${TU_FETCH} = "Y" ]]; then
  if [[ ${IP_VERSION} = "4" || ${IP_VERSION} = "mix" ]]; then
    curl -s -f -4 -o "${TOPOLOGY}".tmp "https://api.clio.one/htopology/v1/fetch/?max=${MAX_PEERS}&magic=${NWMAGIC}&ipv=${IP_VERSION}"
  else
    curl -s -f -6 -o "${TOPOLOGY}".tmp "https://api.clio.one/htopology/v1/fetch/?max=${MAX_PEERS}&magic=${NWMAGIC}&ipv=${IP_VERSION}"
  fi
  if [[ -n "${CUSTOM_PEERS}" ]]; then
    topo="$(cat "${TOPOLOGY}".tmp)"
    IFS='|' read -ra cpeers <<< "${CUSTOM_PEERS}"
    for cpeer in "${cpeers[@]}"; do
      IFS=',' read -ra cpeer_attr <<< "${cpeer}"
      case ${#cpeer_attr[@]} in
        2) addr="${cpeer_attr[0]}"
           port=${cpeer_attr[1]}
           valency=1 ;;
        3) addr="${cpeer_attr[0]}"
           port=${cpeer_attr[1]}
           valency=${cpeer_attr[2]} ;;
        *) echo "ERROR: Invalid Custom Peer definition '${cpeer}'. Please double check CUSTOM_PEERS definition"
           exit 1 ;;
      esac
      if [[ ${addr} = *.* ]]; then
        ! isValidIPv4 "${addr}" && echo "ERROR: Invalid IPv4 address or hostname '${addr}'. Please check CUSTOM_PEERS definition" && continue
      elif [[ ${addr} = *:* ]]; then
        ! isValidIPv6 "${addr}" && echo "ERROR: Invalid IPv6 address '${addr}'. Please check CUSTOM_PEERS definition" && continue
      fi
      ! isNumber ${port} && echo "ERROR: Invalid port number '${port}'. Please check CUSTOM_PEERS definition" && continue
      ! isNumber ${valency} && echo "ERROR: Invalid valency number '${valency}'. Please check CUSTOM_PEERS definition" && continue
      topo=$(jq '.Producers += [{"addr": $addr, "port": $port|tonumber, "valency": $valency|tonumber}]' --arg addr "${addr}" --arg port ${port} --arg valency ${valency} <<< "${topo}")
    done
    echo "${topo}" | jq -r . >/dev/null 2>&1 && echo "${topo}" > "${TOPOLOGY}".tmp
  fi
  mv "${TOPOLOGY}".tmp "${TOPOLOGY}"
fi
exit 0

And the mainnet-topology.json for Node 2

{
  "resultcode": "201",
  "networkMagic": "764824073",
  "ipType": 4,
  "requestedIpVersion": "4",
  "max": "8",
  "Producers": [
    {
      "addr": "adarelay02.psilobyte.io",
      "port": 3002,
      "valency": 1,
      "distance": 282,
      "continent": "NA",
      "country": "CA",
      "region": "AB"
    },
    {
      "addr": "relay0.8sqrdpools.com",
      "port": 6000,
      "valency": 1,
      "distance": 1921,
      "continent": "NA",
      "country": "US",
      "region": "CA"
    },
    {
      "addr": "3.18.168.22",
      "port": 6000,
      "valency": 1,
      "distance": 2671,
      "continent": "NA",
      "country": "US",
      "region": "OH"
    },
    {
      "addr": "54.159.255.195",
      "port": 6000,
      "valency": 1,
      "distance": 3110,
      "continent": "NA",
      "country": "US",
      "region": "VA"
    },
    {
      "addr": "75.119.133.13",
      "port": 3000,
      "valency": 1,
      "distance": 3947,
      "continent": "NA",
      "country": "US",
      "region": "FL"
    },
    {
      "addr": "antrel.dyn.ydns.io",
      "port": 54833,
      "valency": 1,
      "distance": 7158,
      "continent": "EU",
      "country": "NL",
      "region": "NH"
    },
    {
      "addr": "94.237.99.68",
      "port": 8001,
      "valency": 1,
      "distance": 7512,
      "continent": "EU",
      "country": "FI",
      "region": "HE"
    },
    {
      "addr": "157.90.119.14",
      "port": 6000,
      "valency": 1,
      "distance": 7687,
      "continent": "EU",
      "country": "DE",
      "region": "BY"
    },
    {
      "addr": "192.168.42.6",
      "port": 3000,
      "valency": 1
    },
    {
      "addr": "192.168.42.9",
      "port": 3000,
      "valency": 1
    }
  ]
}
9

A good portion of the above is complicated, incorrect, and unnecessary.

My recommendation is to visit this: Guide: How to build a Cardano Stake Pool - CoinCashew

And follow the below steps:

    1. Configure the block-producer node
    1. Configure the relay node(s)
    1. Configure your topology files

If you apply it as it’s written everything should work fine. Just pay attention to your cardano version, and this guide’s version.

10

Hi, I am not able to resolve DNS for api.clio.one which is the one used on topology. what has been changed?

11

You had me worried I was hacked… Where did you get this? Clio looks like a service for checking wallets.

YYC Pool Node 1 is node1.cardanoyyc.ca:3000 and should resolve to 107.190.67.120.

12

this is in topologyUpdater.sh
https://github.com/cardano-community/guild-operators/blob/89c1016d415c5344fca7a9c02815b364db3cb477/scripts/cnode-helper-scripts/topologyUpdater.sh

13

Anyway now it’s working I didn’t do anything.

14

There was a problem with DNS resolution of api.clio.one for a couple of hours. It works again now. The monitoring part of topologyUpdater.py managed to send me email accordingly.

1 Like
15

Update. I was eventually able to solve my issue by removing the /opt/cardano/cnode directory. removing all my “env” and “gLiveview.sh” files. Running the prereqs.sh and using the gLiveview.sh provided there. Only needed to change the port number.

PHEW.

Aside, the error messaging could be really improved in gLiveview. Maybe ill learn bash and put some hours in there.