Guild Operator scripts are security holes

Before explaining my concern, I will say that I would love to read a response from the guild operator scripts maintainers proving me wrong.

Now I am extremely concerned that many pool operators are basing and standardising their work based on these tools, I see security issues in the sense that:

  • You are executing scripts directly downloaded from a master branch of a github repo which most contributions come from 2 person, that have no clear/transparent governance, nor checks and balances set.

  • These scripts are having access to your servers AND keys, including the keys for managing the pledges.

  • The scripts and tools are not easy to verify/analyse, the cost of verifying ALL scripts before executing each time and after accepting automatic updates is too high and makes for introduction of malicious code easy.

It would be relatively easy for a single or coordinated group of contributors to introduce a stealthy commit that steals private keys or does transactions. This would put the whole Cardano network and community at risk, the network depends on the pools, and if most pools depend on the same shared script library, that library becomes a HUGE single point of failure and of incredible risk.

I hope my concerns are completely incorrect, but it is important to consider that for TRUE decentralisation and resilience of the network, the Pools MUST be independent of each other, both technologically and politically, if they get too involved with each other the amalgamation becomes a source of centralisation and single points of failure.

4 Likes

Hi!

you can reach them on github:

Ok I’ve posted to them, lets see what they say :slight_smile: Security concern · Issue #897 · cardano-community/guild-operators (github.com)

those guys mostly pool operators so I guess their goals is the same as others - having a healthy, decentralized network.

I bet they do, still we should not trust the good will of important actors, anyone can get corrupted at any moment and elaborated security attacks do exist, even if these guys are all well intentioned if their git accounts get hacked because of minor mistakes then everyone is at risk. Big systems depend on the well functionality of SPOs and on controlling security holes, complete nations eventually will depend on what we as a community achieve. This should NOT be taken lightly

2 Likes

Check this similar topic: Stakepool Operation Tools as a potential risk?

2 Likes

I do trust the guys behind CNTools enough to run most of my relays on their scripts. They’ve done an awesome job, made all simple and user friendly.
My BP, however, is and will stay hand installed to minimize potential attack surface.
It is up to each SPO to take their security decisions.

To those following this topic, the conversation expanded further here: Security concern · Issue #897 · cardano-community/guild-operators (github.com)

After talking with the contributors I can only say the obvious, these tools are very helpful but should be used VERY consciously, USE cold environments for ALL your keys, if you don’t know what is that ASK. Don’t be lazy! The whole network is counting on us being diligent!

4 Likes

Even better than cold, if you are using cloud based deployments, use temporary, home made instances/containers.
Cattle not pets is the rule. It reduce the surface of attack by a great deal.
Thanks for this post @FrancoAra I think this is a very serious matter and should be well/better understood.

I have respect for the guys behind the guild scripts and am friends with some of them. However, I have never used their script mainly because of the same concerns as you. Thank you for explaining it very clearly.

While I do trust the Guild Operator scripts - and they’ve done a really great job providing all that many scripts - I started to re-implement topologyUpdater in python instead of bash. The result can be found here: https://github.com/Josef3110/stakepool_python_tools

As an addition it provides monitoring by sending emails if a problem occurs. Because it mimics the original shell script it can be used in with the systemd services scripts from Guild Operators.

It uses a JSON file for configuration, and therefore - no editing of the script itself is necessary. It does not make lookups for updates and has no additional env file with lots of functions not needed. IMO it is easier to maintain and simple to audit because of it’s straight forward implementation.

The current source of centralisation is IOG who solely manage the cardano codebase, not guild tools, Martin’s scripts, Andrew’s JorManager or any other avenue. It is very naive and silly to try to throw mud at wrapper scripts and then on contrary trust the codebase and even pre-created docker images/binaries from single organisation. Also, not a single statement above has actually shown to be a security hole - unless you assume 20 guild members (multiple current/ex-ambassadors) suddenly go rogue giving away 4 years of value they’ve built, and no one catches them (incl. entities who have their identity documents too).

There seems a lot of intentional misinformation/FUD being spread around about what scripts are and are not, trying to achieve a higher ground based on actions / lack of reading of documentations that comes from an end user. Also, it is a lie that there isnt a reviewal as well as test cycle involved - the tests are performed against alpha branch before merging to master, and users are invited to check them via announcement groups. You try to question ‘verifying scripts’ as hard - which in the end are simple bash scripts, yet - you feel free to update the node without any queries and understanding of haskell codebase. To paste the final reply from the issue:

You are confusing between what SPOs should do versus what the wrapper scripts to the binaries do. It is like saying “cardano-node should not run in online mode”. They will all recommend it, but they cannot enforce something like that in code, being restrictive and hindering knowledgable use cases too.

It is all about finding a good balance between multiple opinions, options and practices - and being flexible between those. The responsibility of how node is run, how script is run, whether recommendations are followed lie on the SPOs, which is captured below:

Do not download automatic updates

Not sure what your understanding is, the scripts dont auto update without user prompts. Besides, gLiveView has NO_INTERNET_MODE mode to avoid update checks as well

Downloading the tools should come from immutable stable releases (that maybe one day can be peer reviewd/audited)

That’s a user action, on our ends we do have compulsory reviewal process, and a test process before it’s merged to master for users. The guild itself comprises of over 20 active members, who have been around (some from day 1 of project) for years, with multiple ambassadors and with different roles.

Commands that access keys should check first that the script is being run under a cold environment (for example by trying to fail a ping to the internet) and only doing the operation in a “risky environment” after confirmation of a statement

That’s again a user preference, not everyone runs key operations in cold mode. We cant enforce our ideology to consumers of those scripts beyond a point. :slight_smile:

In spite of explaining on individual points and sharing screenshots, this comes as bit of a disgusting/demoralising approach to conclude under false assumptions about security holes (calling it on github instead of SPOs usage) - they’re not on the guild-operators, but on SPOs skipping to read the home page - I kindly request to adjust your title accordingly, and none of the concerns raise actually differ from IOG codebase OR any other open source methodology incl. docker - you cannot conclude a codebase to have a security hole, if an end user does not follow recommendation or skips steps.

One thing that we have ofcourse onboarded is disabling option to show update prompt as default , consolidating all checks to be from common NO_INTERNET variable, which is set to Y by default (to avoid checking) - as not many understand or review the code or history of members before coming to conclusions like above (and this is not about a single post, but many - including accounts formed with less than 2 hours of life time prior to their post - that we have not responded to).

4 Likes