topologyUpdater reliance and weakness

Staking & Delegation Setup a Stake Pool
1

Hi Community,

Over the last few weeks I have been working to set up a stake pool. I’m the kind of person that likes to do everything manually, so I’ve spent quite a bit of time investigating how things work.

I’m at the stage where I feel ready to deploy my stake pool, but I’m a bit concerned with what I perceive as an over-reliance of topologyUpdater in the network of stake pools - at least that’s how it appears from what I’ve read in this forum and others.

OK, Cardano doesn’t currently have a P2P solution for automatically taking care of this, so we need to somehow negotiate with other pools to be included in their topology files and accepted into the network - which is obviously not ideal.

topologyUpdater is the most commonly recommended solution to this problem, but who controls it and what are its dependencies?

topologyUpdater relies on a web service provided by clio.one, which in my mind is a single point of failure and rather opaque.

My concern further deepened when I read the code for topologyUpdater and saw that every time it runs, by default, it automatically reaches out to the web and, if it so desires, downloads a new version of topologyUpdater itself and overwrites the original script. IMO this is a huge vulnerability waiting to happen, any attacker or malicious actors would only need access to 1 account that has publishing rights of that script to potentially attack a significant portion of the network, not to mention steal funds from the less savvy, or lazy pool operators who keep keys on their server.

Anyone else share this worry?

How are others navigating this issue rather than resorting to topologyUpdater?

I feel I should state, this post is not intended as an attack on clio.one or anyone else involved with these tools, just voicing my concerns as a potential stake pool operator.

Also, not a hater, have been a keen follower of the Cardano project since 2018, and looking forward to the impending smart contract era.

Thank you for reading.

2 Likes
2

Hello,

Did u checked the topologyUpdater script?
U have an option to stop automatically updates

#BATCH_AUTO_UPDATE=N # Set to Y to automatically update the script if a new version is available without user interaction

uncomment this line if u wish to not be updated auto

Cheers,

2 Likes
3

Hi @phera !
I had similar doubts than you when I started.
Compare: Stakepool Operation Tools as a potential risk?

Being careful with updates is a good practice.
This is what i learned so far about the topic:

  • The scripts which are part of the Guild Operator tools (includes topologyUpdater) are asking for updates before just running them
  • As mentioned above it is a good idea to validate an update before just installing it.
  • To avoid manipulation of your local config it is an option to use Intrustion detection. Some use OSSEC. Here is a guide which was shared on the forum: Cardano Node Security Monitoring - ADAvault
  • Specificaly for TopologyUpdater some are not using it, instead looking for peers which they trust and build a custom topology from there.
  • This also allows to put more concrete firewall rules on relay to avoid traffic from “unknown” sources which also eliminates DDOS attack vectors. Anyways I’m not sure if this is a good practice since it also does not allow all the other relays to connect to your server. A downside might be that some of the Pool Comparison platforms (like Pooltool) would also not be able to ping your relay, so some specific handling would be required.

If someone with custom topo based on the known peers approach and IP limitations only for those peers reads this. Would be interesting to hear your experience with that.

3 Likes
4

Yes, this threat is real. It is actually even worse. The default branch for guild-operators is alpha.

image
image1692×304 19.9 KB

Every time someone installs the topology updater, he/she gets an arbitrary commit from some alpha branch. There are no releases to the project and the few (outdated) tags are of no much use either.

Eventually, I resorted to a hack like this to pin to a specific git commit for gLiveView. For topology updates, I borrowed from the original code (and many thanks for that) and removed the update logic and a few other kinks.

A docker image needs to be immutable and cannot be built from unknown sources let alone depend on scripts that may potentially update themselves behind the scenes. The result of which is here.

3 Likes
5

Hi @Alexd1985,

Thank you for your response.

Yes, I read the code for topologyUpdater script, and was aware of the flag you mentioned. My main issue was that the default policy is auto update. I now see that the script asks before updating, so that reduces my concerns a little, but I still feel that this update behaviour is non-standard - e.g. Python, NGINX, Apache, GCC, GHC… none of these offer or attempt to update themselves, and it doesn’t seem to hamper people’s ability to use them.

Anyhow, I’ll not be using these tools for my pool and will roll my own solution, while taking inspiration from community contributions and discussions.

Regards

6

Hi @Markus-VITAL

Thank you for your considered response. Good to see I’m not the only one who has these concerns. I did try searching prior to posting, but failed to find anything related.

Thank you for pointing out that the script asks before updating, I must’ve skipped that line!

I’m still not comfortable using the guild tools, so I’ll just opt for my own scripts until there’s an actual P2P discovery solution provided by IOG.

Thanks for the OSSEC guide link, hadn’t saw that before and quite interesting.

I did consider strict firewall rules, as you suggest, but also thought that might be a bit too much and may have unforeseen downsides.

Thanks again for your response, it provided context and gave reassurance.

All the best.

7

Thank you for pointing this out @tomdx, I was reading the master branch while completely unaware it wasn’t the default!

I see you’ve embraced nix for your solution, I like this approach! I’ll probably continue rolling my own, especially as it’s nearly ready to use, but there’s definitely points of inspiration to take from yours. Thank you for sharing.

Further on nix, it seems to be massively underrated, probably due to the steep learning curve, but I can see why IOG are using it.

Hopefully IOG will provide a P2P discovery feature in the near future and we can further streamline our dependencies.

Thanks again for your response, it has been reassuring and helpful.

All the best.