Back to Basics — Security measures every Cardano Stake Pool Operators should know and implement

Staking & Delegation Stake Pool Security
,
1

Recently we saw some unsettling news about a stake pool operator (HAPPY) that was hacked and had their 1,000,000 ADA Pledge stolen. Following this, a fellow SPO and friend did a quick analysis and found this startling revelation:

This is what prompted the article, a “back to basics” security measures every Cardano Stake Pool Operators should know and implement.

  1. DO NOT run the block producer and relay on the same host — as the relay is public facing and if it is hacked or under attack (DDOS), your block producer will also be offline, which means you will miss the chance to produce a block and it can affect your ranking & rewards later on.
  2. Your block producer should be behind a firewall and should only connect to your relay nodes. And your relay nodes may connect to the block producer and other peer relays.
  3. Only keep the bare minimum ( kes.skey , vrf.skey , node.cert equivalents) to keep your cardano-node up on the server. DO NOT keep your wallet/pool files (other than required files) on the server.
    Credits to https://twitter.com/SafeStak for recommended file list.
  4. Never run anything (cardano-node, web server, docker, etc…) with root privileges. Setup an unprivileged account(s) and use it.
  5. Remember, docker runs with root privilege by default — unless you know what you are doing, don’t use docker. If you are going to use it, invest some time into learning about how to properly setup docker namespaces.
  6. Use a hardware/software firewall on your server and lock down the ports.
  7. DO NOT leave any unused ports open on your server.
  8. Change the default SSH port 22 to something random that only you know.
  9. Disable access to root login on the SSH config. Only use unprivileged account(s) with SUDO access. Also disable password login in SSH.
  10. DO NOT use username/password authentication for SSH.
  11. Use public key/private key ssh authentication method for SSH. Don’t forget to password protect your private key when you generate them.
  12. Use SFTP to transfer files to/from your server.
  13. Use a secure operating system with long term support for your server. For example Ubuntu 20.04 LTS release.
  14. Don’t forget to keep your server’s operating system and installed software up to date with latest updates.
  15. Disable IPv6 on your server if you are not using it. Here’s why.
  16. Ensure your day-to-day machine user account and cloud provider accounts have secure auth — strong passwords, 2FA authentication, etc. Keep your set up simple and reduce the surface area.
    Credits to https://twitter.com/SafeStak for this recommendation.
  17. Implement Fail2Ban to block unauthorised access attempts.

Security is complex and has many layers to it. It starts with the person. I highly recommend this video released by Charles Hoskinson:

I am aware quite a few non-techy aspiring stake pool operators are joining the ecosystem; to them — please educate yourself. We need more people to participate — so let’s do it in a secure way that is mutually beneficial (you the SPO as well as the delegator and the Cardano eco system).

Useful Links

This is not the final list of everything a stake pool operator must do — it’s simply a start. In addition to the above, please do your own research.

If the stake pool operators don’t follow these basic security measures, their pools will likely get compromised and worst case scenario — pledge get’s stolen. As for the delegators — they won’t see any rewards for staking at the compromised pool (as it won’t be making any blocks).

Thank You.

25 Likes
Air Gap required?
Postponing our stake pool until hacker issue is solved
SPOs, Do not repeat my mistakes, Keep your Core Node Safe
Port 3001 (relays-new.cardano-mainnet.iohk.io) mainnet-topology
2020 Cardano Forum Stats - The Year in Review
2

Super valuable @ADABros! And called for.
I hope this will help many SPOs to steer clear of danger.

@Donnybaseball @necropuddi @Trigger can we help spread this post through the SPO community somehow?

@papa_carp perhaps we can link/share this on pooltool.io?

3 Likes
3

Gave it some link juice my friend :vulcan_salute:

2 Likes
4

I love this post. Thanks a lot!

To all readers, also please always assume that you will be hacked. Any mistakes you make security wise will very likely be found. For example, after being live only for a short period of time, our security scanner already notices known attackers knocking on our door. Be prepared for this:

"service":{
   "serviceName":"guardduty",
   "action":{
      "actionType":"PORT_PROBE",
      "portProbeAction":{
         "portProbeDetails":[
            {
               "localPortDetails":{
                  "port":3000,
                  "portName":"Unknown"
               },
               "remoteIpDetails":{
                  "ipAddressV4":"195.54.160.21",
                  "organization":{
                     "asn":"49505",
                     "asnOrg":"OOO Network of data-centers Selectel",
                     "isp":"OOO Network of data-centers Selectel",
                     "org":"OOO Network of data-centers Selectel"
                  },
                  "country":{
                     "countryName":"Russia"
                  },
                  "city":{
                     "cityName":""
                  },
                  "geoLocation":{
                     "lat":55.7386,
                     "lon":37.6068
                  }
               }
            }
         ],
         "blocked":false
      }
   },
   "additionalInfo":{
      "threatName":"Brute Forcer",
      "threatListName":"ProofPoint"
   },
},
"description":"EC2 instance has an unprotected port which is being probed by a known malicious host."
4 Likes
5

When an SPO is compromised, can one still undelegate stakes to another SPO ?

6

I believe these are unrelated @pcsasd, the delegation certificate exits on the blockchain and isn’t located on the system of any any specific SPO (or more correctly, it should exist on all of them). Happy Shelly!:beers:

1 Like
7

oh my god, this is crazy. but then again, it is to be expected when real world funds are involved. I like your report on hack attempt - is that an AWS product?

1 Like
8

Thanks. Yes this is coming from AWS GuardDuty

1 Like
9

Awesome post. Sometimes when inexperience is at play one can make some unintentional blunders. I have a couple times. It cost me dearly. My inexperience and questions for some reasons I’m still trying to understand, were not received well by a number of people. The technical answers that were disclosed in this case were well thought out and described. I was unable to be as efficient.

Impressed to such see a very well thought out set of solutions and recommendations. This situation should also help those of us who stake understand just how important our pool managers are to the process. Their technical awareness and attention to detail will be vital to the success of the entire Cardano ecosystem.

I’m personally grateful for their expertise.

1 Like
10

Great list. I would add some types of deception to your infrastructure. Canary tokens and other items. Make sure you are able to be alerted of the threat and have a plan to take action. Check out this deception video by a leader in the information security industry. “Because attacker tears make the best wine” haha! https://www.blackhillsinfosec.com/webcast-getting-started-in-cyber-deception/

11

One question about security:

If the block-producer gets hacked, or even the physical machine is stolen, what will I have lost exactly?

Assume that I only had kes.skey, vrf.skey & node.cert on the server, and that my “air gapped / offline signing machine” was never compromised.

Basically having the offline machine safe is easy. But I want to know if I pledge $100k to my stake pool that I wont lose it if my block producer gets hacked.

2 Likes
12

You won’t lose it if your block producer gets hacked. Your funds are always kept securely offline.
I’m pretty sure they can steal your credit for producing blocks. So your node would begin working for somebody else.

13

I don’t understand point 4, whats the difference between running your node from root user compared to running it without root user but with sudo privs. its the same thing no?

14

Point 4 says do not run anything as root, use a regular user account, no sudo.

15

yea but you cant do much without sudo