Skills for Operators in Cardano

English General Discussions
1

A little rant:

I am seeing much too much copy and paste solutionism in this and other Cardano forums!

In this great community, you will often, nearly always find someone to hold your hand through any and all problems you encounter. But, please, if you want to operate financial infrastructure – a stake pool, an exchange, a marketplace – you should have a quite good knowledge of what runs on your system and how it works.

In my opinion, it would be good if you had seen a Linux system, a command line, a .bashrc, and an editor, before you even start, but if you want to learn all that also on the way, in parallel to all the Cardano-specific stuff, that’s also okay with me.

All these copy and paste guides are nice to have a quick reference, to not forget a step in a process, to remember something you seldomly do. They are even a necessity, since the more or less official documentation is scattered among all those systems IOG and other players experimented with in Cardano’s almost five-year journey. But, please, don’t follow them blindly! Try to understand what is in there!

After some time you should really know, what most of cardano-cli’s options mean and why they have to be there and have exactly that value or why not. You should have at least a basic understanding of what the non-Cardano technologies you maybe use – from systemd to Docker – do and how. You should know, which parts of the “create a transaction” walkthrough are the necessary steps and which are just some shell shenanigans to keep the guide at a reasonable size. (Looking at you, awfully complicated while read loop aggregating the UTxOs!)

It really is a lot more fun to help people out, where they learn something on the way (and often me also), have a better understanding, have a better mental model of their system and its place in the Cardano network in the end, instead of just: “Ah, yeah, now GLiveView says it’s running. We left a lot of unused files in all corners of the filesystem in the process and have no idea, what exactly was the problem, but it’s running. Kthxbye!”

Even seasoned developers or operators have some places in their projects and systems with a label: “I don’t know why it works this way, but I’ve tried everything else. Just don’t touch!” But they should be rare!

Please, try to learn, not just get something to work somehow! Thank you!

10 Likes
2

Thank you. That did so need to be said.

Can I further encourage people to make use of their linux packaging tools to automate the software installation and upgrades. This allows installing the executables in standard locations so that custom modifications to PATH, LD_LIBRARY_PATH, and other variables is not required. It also reduces errors from forgetting steps when upgrading software.

I have provided a Github repository explaining how to build cardano-node as a deb package. Feel free to read through it and use it, or recommend improvements: GitHub - TerminadaPool/cardano-node-debian

I don’t think it is a good idea to blindly trust a script to download, compile and install software on your secure cardano node running machines. When I often need to run the same series of commands, I find it better to create a note file with the series of commands recorded in a way that I can simply cut and paste them. This way I clearly know what the commands are doing.

I think setting up a separate build machine which can be just a virtual machine is a good idea. Build your deb packages on that and then only install the built debs on each of your relays and block producer.

I don’t think it is a good idea to have all the compiler, build chain, and development libraries sitting on every one of your relays and your block producer. I think it is much safer, security wise, to have only the minimum software required to run cardano-node on each relay / block producer.

Furthermore, complex scripts like GLiveView are unnecessary and their complexity increases the risk. I think it is better to learn how to run a couple of different commands for checking things. You can experiment with looking at the data already provided at your node’s EKG port 12788. Try:

curl -s -H 'Accept: application/json' http:/localhost:12788 | jq '.'

And, adjust the jq command to extract the data you want.

Or take a look at some scripts written by others and rewrite them. Eg: this script for monitoring block delay times: cardano-node-debian/cn-monitor-block-delay at master · TerminadaPool/cardano-node-debian · GitHub.

1 Like
3

Sounds like a great idea!

Personally, I have no problem installing a select few pieces of software outside of distribution package management if I interact more intensively with them, compile myself, … Cardano’s tools are a candidate for that. But you have good arguments for your way, also.

Coming back to the more across-the-board scope of my post: People, use @Terminada’s packaging tools, use the Coincashew guide, or use CNTOOLS, but only use any of them, after you understand what it does and why you think you prefer it to the other options!

2 Likes
4

The complexity of the cntools suite of scripts is overwhelming. They attempt to automate everything to the point that stake pool operators blindly trust without understanding them.

I just found out that the cntools suite encourages pool operators to keep their cold keys on their block producer machines. It provides a feature to encrypt and decrypt the cold key files.

See my thread comment: Encrypting pool files using cntools question - #8 by Terminada

I don’t think this “feature” is a good idea at all. But maybe I am wrong because maybe some pool operators will still have their cold keys on their block producer but leave them unencrypted instead. Not that encryption would make much difference since an attacker with access will easily keylog the password.

I wonder if people using hosted servers realise that the hosting provider has complete access to their machines and can copy any file they want?

Cold keys should never be on the block producer. Not even once. Providing a false sense of security encourages people to do this.