Topology Updater Service

[werkof]

Since the test network has to get along without P2P network module for the time being, some static topology files are our way to handle this meanwhile.

This TopologyUpdater service is intended to be a temporary (yes centralistic) solution to allow everyone to activate their relay nodes without having to postpone and wait for manual topology extension requests (message with other pool operators or do pull requests on the IOHK repository).

• The TopologyUpdater shell script must be executed on the relay node as a cronjob exactly every 60 minutes.
• After 4 consecutive requests (3 hours) the node is considered a new relay node, and so listed in the topology file.
• If the node is turned off, it’s automatically delisted after 3 hours.

Read more about at https://cardano-community.github.io/guild-operators/Scripts/topologyupdater.html

Based on first participation experiences the topology-Files can be fetched soon from the same relay nodes.

[werkof]

Here’s a basic concept of how topologies without P2P peer discovery will/can look like

First, let’s imagine how a everyone “I wanna peer with IOHK” will look like.

there might be some manual connections between pool operator who know each other.

Now let’s start with an unconnected set of relays and how they will connect if each only connects to a subset of other nodes.

Your node connects to just a couple of other nodes

and there is another node using the same subset (you both act as backup of each other to run this sub-set)

Now if most involved nodes run such a subset of just some few connections it will look like this

This is just to give an idea what a good p2p protocol has to achieve in a decentralised way.

The proposed topology service can generate such sub-sets in a centralized way, by also ensuring removal of orphaned nodes and automated addition of new nodes.

by participating and announcing you node you help to achieve a good interim solution until P2P is in place.

[igghibu]

hi @werkof,

thanks for this. I have only one issue: my relays have a load balancer in front and topolgy update script complaints about the domain not matching the IP when I run it. Is there a solution for this?

cheers

[werkof]

That’s interesting, because the loadbalancer shouldn’t cause any issue.

Whatever router/firewall/balancer/proxy you have in front: The hostname you announce will DNS-resolve to one (or multiple) IPs. And at each of that IPs one of your relays should answer to requests.

Feel free to DM me the hostname so I can look into.

[werkof]

We have reached a 150 peers who continuously update their tip state, guaranteeing they are actually good and usable peers for your node

.
.
Here’s where they are located around the globe (open link to see the map)

https://gist.github.com/gufmar/2fdc4f4582015404d19038331c3d5091

[werkof]

The topologyUpdater Service also forked into Shelley mainnet.

Here’s how this looked like, while Pool-Operators switched existing MC4 Testnet nodes over to Shelley Mainnet

And this is where the relay nodes are located all over the world

[adatainment]

That is really nice @werkof

[werkof]

Only two months ago, in the middle of the testnet preparations we decided to provide our CLIO1 server infrastructure and develop this topologyUpdater service.

Now, two months later, and less than two weeks after Shelley mainnet launch we are a 450 and still growing subscribed peer nodes.

We still have the same peer selection logic in place, looking at the individual requesting node’s geographic distance to all other subscribers. The service always returns a list of very close to as far as possible peers, around the whole globe. This should ensure a global connected network without isolated areas or preferred cluster zones.

If interested you can read more about at
https://cardano-community.github.io/guild-operators/#/Scripts/topologyupdater

[jdloft]

@werkof, @igghibu brings up something we have been facing as well. We run in a cluster environment with a load balancer to handle incoming requests. It is able to handle public connections on port 3001, while our requests/all egress come from each cluster node IP address. Even though those are public-facing IPs, they do not listen on any port.

Is there a way that we could set the client IP in the headers with X-Forwarded-For or something similar?

[werkof]

Technically it wouldn’t be a problem to accept X-Forwarded-For or even simply a &LbRealIp
But both modes debunk the attempt to design and use this service without user-accounts and authentication, just by the real connecting TCP-IP.

I expected the P2P to be ready (enabled) around one month after mainnet-launch (now) but that probably will take some more weeks. One thing I’m working on, is fetching the on-chain registered relays, and “import” it to the topologyUpdater list. Then do only the public reachability test, without knowing if the node is on tip or not.

For me it’s just very important to not introduce something that can easily be used to infiltrate the peer list.

[jdloft]

Right, I think at this point it wouldn’t be worth it to try and support every use case out there. We’ll just wait for P2P to be implemented.

[werkof]

TopologyUpdater subscriptions first reached 1000 nodes. +100 in one month.

On average one update every 3.6 seconds, from Cardano relay nodes on all 5 continents.

https://gist.github.com/gufmar/2fdc4f4582015404d19038331c3d5091

[regel]

hi @werkof

I have the same problem as @igghibu

My relay(s) are running inside Kubernetes and the service is exposed behind a LoadBalancer. This is a very common setup nowadays with Cloud managed Kubernetes. See the External IP and port below, this is how the relay can be reached externally.

                                             TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                        AGE
5473/TCP                       13d
testnet         foo-cardano-headless                                 ClusterIP      None           <none>          6000/TCP                       12d
testnet         foo-cardano-producer                                 ClusterIP      10.0.181.141   <none>          6000/TCP                       12d
testnet         foo-cardano-relay                                    LoadBalancer   10.0.83.217    51.105.220.72   6000:31972/TCP                 12d
testnet         foo-redis-headless                                   ClusterIP      None           <none>          6379/TCP                       12d

However, this public IP (51.105.220.72) is only assigned to the relay service.

Other pods, such as topology updater bash script, or the simple curl commands have to go through standard NAT and will have a different clientIP address when they reach api.clio.one.

As you can see from the following curl command, it complains that the hostname and clientIP do not match. They cannot match. Routing is asymmetric due to how Kubernetes and Cloud providers handle the IP traffic.

$ k run -ti --restart=Never --rm -n testnet --image=curlimages/curl foo -- sh
If you don't see a command prompt, try pressing enter.
/ $ curl -s -f -4 'https://api.clio.one/htopology/v1/?port=3001&blockNo=2890699&magic=1097911063&hostname=regel-testnet.westeurope.cloudapp.azure.com'
{ "resultcode": "501", "datetime":"2021-09-06 20:27:30", "clientIp": "20.93.174.215", "msg": "invalid hostname not pointing to client IP" }
/ $ curl -s -f -4 'https://api.clio.one/htopology/v1/?port=6000&blockNo=2890699&magic=1097911063&hostname=regel-testnet.westeurope.cloudapp.azure.com'
{ "resultcode": "501", "datetime":"2021-09-06 20:27:47", "clientIp": "20.93.174.215", "msg": "invalid hostname not pointing to client IP" }

To try to resolve this issue, if I remove the hostname, then I get a “welcome to the topology” output but the clientIP registered by api.clio.one will be wrong because it does not match the relay DNS address.

curl -s -f -4 'https://api.clio.one/htopology/v1/?port=3001&blockNo=2890699&magic=1097911063'
{ "resultcode": "203", "datetime":"2021-09-06 20:00:21", "clientIp": "20.93.174.215", "iptype": 4, "msg": "welcome to the topology" }

20.93.174.215 is not the relay IP address.

I hope this clarifies the issue.

We’ve all been waiting for P2P in cardano-node but it’s not clear when it will be out, or even if it will be able to resolve the above issue.

Let me know your thoughts and please share P2P information if available.

If anyone reads this thread and is using Kubernetes to run cardano-node, please let me know how you managed the above issue.

[werkof]

Hi Regel

I know the issue in such a setup. The whole and fundamental idea in how I designed TU was to make it permissionless, and possible for everyone to join without special setup or support.

However, due to some previous demand, I also implemented a solution to allow such setups too. please DM me and confirm the public IP your subscribe & fetch requests will come from.

[a13519]

Hi, How to determine the value of CNODE_HOSTNAME in TopologyUpdater.sh?

Thanks,
Song

[werkof]

It’s an optional value, only if you have and want to use a domain hostname.

[eladws]

Hi @werkof ,
Given the fact that P2P is still not operational, did you manage to implement this (“fetching the on-chain registered relays, and “import” it to the topologyUpdater list”)?

Running nodes behind NATs or Kubernetes Load balancers (as described below in this thread) is still a pain without it…

Thank you so much for your efforts

[werkof]

Hi,
We have a solution in place for such NAT’ed/proxied setups. We haven’t opened the service for all on-chain registered relays because we wanted and needed to keep an incentive for subscribers to actually subscribe and show their correct tip height.

I understand the special situation for certain nodes and just didn’t expect it to take this long for p2p.

Feel free to DM me for a dedicated access key

[nemo83]

Hey,

would you know how P2P is going to discover node behing NAT/Kube?

I spun a P2P relay and I’m not sure it knows on what ip/port it should receive connections.

Thanks

[werkof]

Around one year ago TopologyUpdater crossed the 1000 participating relay nodes.
since then many more joined the cooperation. now we are at around 2700 participants on mainnet and another 160 on testnet

(February 2022: mainnet relays subscribed to TopologyUpdater)

The plan is to continue this service until P2P becomes ready, even if this now took way longer than expected.

I also have new additional ideas to add more functionality for Stakepool Operators and other applications on the Cardano Blockchain. Again permissionless and free of charge for every participant.

The idea had been floating around for a while, and became topical again with the congestion phase of the Mainnet in early 2022. The goal is to give pool operators the possibility to monitor their nodes performance and compare it with others in the network.

• Where are the opportunities for optimisation?
• Why does something unexpected happen?
• What do we think about possible parameter changes?
• What is the scope for scaling?

The main task of the SPOs is to ensure that the network infrastructure works as well as possible and that the blocks with the transactions and scripts are distributed ideally, quickly and reliably.

In order to implement this project and to continue to operate the TopologyUpdater, I have submitted a Catalyst proposal:

https://cardano.ideascale.com/c/idea/404897

I hereby ask for your support for this proposal.
The vote will take place between 14 and 28 April. Until then, it would be helpful if you support the proposal as follows:

• Log in to Catalyst and give the proposal (above link) a Like (clap hands) You can also repeat this on several days in a row.
• Like this post here
• or even better: copy the link and share it with a few SPO friends or in the Cardano community, with a short reason why you find the TopologyUpdater and the block performance project useful.