Securing a node on Linux?

Staking & Delegation Stake Pool Security
#1

Reading about pool intrusions just few days after their launch demonstrate that people out there, are ready to exploit any security breach.

For people like me (not security expert) I think it is honouring @junada ADA sacrifice’s than to do one’s best to take proper counter measures, but personally I would need help. (defence is more collaborative than attack I believe)

Typically my nodes are running on fresh install of Debian 10 on separate VPS in different countries. I use a 1GB RAM servers with 25GB SSD Drive. Binaries are built and transferred to those servers with ssh configured to connect via key. I build libsodium one each node and use CNTools to run the node (after changing default path).

I want to close the maximum ports and services. But can you tell me those that are mandatory ?

  • ntp: for time synchronisation?
  • DNS: to update packages a little bit more easily?
  • ssh: with password authentication disabled and default port changed?
  • https: for communication with a white list of IP addresses ?

I’m starting the iptable configuration with something from serverfault but it probably needs extensions.

#Flush existing rules
iptables -F

# Allow existing connections to continue
iptables -A INPUT -i eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT

# Accept I/O from the 1.2.3.4 IP
iptables -A INPUT -i ens3 -s  1.2.3.4 -j ACCEPT
iptables -A OUTPUT -o ens3 -d 1.2.3.4 -j ACCEPT

# default DROP rule for all protocol to ens3 but ping (icmp) 
iptables -P INPUT -i ens3 ! -p icmp DROP

Could a security expert share a template of his iptables ?

#2

Majority of operators are running on Ubuntu, using ufw firewall

The ufw firewall rules I have offhand are as follows, and can be translated into iptables by referencing the port activity

on relay nodes:

ufw default deny incoming
ufw default allow outgoing
ufw limit proto tcp from any to any port [custom ssh port]
ufw allow [relay port]/tcp

on block producer (bp) node:

ufw default deny incoming
ufw default allow outgoing
ufw limit proto tcp from any to any port [custom ssh port]
sudo ufw allow from [relay 1 ip] to any port [bp node port] proto tcp
sudo ufw allow from [relay 2 ip] to any port [bp node port] proto tcp
sudo ufw allow from [relay 3 ip] to any port [bp node port] proto tcp

You will have a difficult time establishing a stable node with 1GiB ram - I recommend 4GiB, but have relays in test doing alright on 2GiB for the time being (change of epoch is currently a bit rough on the nodes)

I strongly recommend learning to build transactions manually (not sure if you’re using CNTools for tx building too), as doing so will provide you the framework for building and signing all transactions offline on a cold machine (and then you can move the signed transactions to your hot nodes for submission to the chain). This practice ensures your keys always remain cold and would mitigate theft of funds or pool keys regardless of unauthorized system breach.

6 Likes
Relay node syncing to core node but when I add other Pool operators nodes to topology it syncs
Relay ports and ip
Opening ports for relays
No TXs processed
Opening ports for relays
#3

Yes, I just found that when the node reach block 208…
I thought 4GiB was the minimum for building the node, not running it.
I guess I’ll just have to upgrade…then.

As for CNTools, you are right it’s important to learn the building and signing of transactions, and I surely need more practice but I think I got the basics.

Thank your for the rules !

1 Like
#4

Regarding ssh many have now moved on to more than a single factor authentication. (2FA/3FA) if you need to have SSH.

I am still learning many of this myself but I found some valuable audit software (https://cisofy.com/lynis/) that gave me many pointers on what I could improve on my servers both in network and otherwise. An overview can be found on https://linuxsecurity.expert/tools/lynis/

I am sure many in the community can contribute on this topic and I think it is very important we learn security well as a community.

4 Likes
#5

If you have a hardware wallet, it can provide SSH authentication, which is a bit inconvenient, but pretty cool and as secure as pubkey + 2FA:

3 Likes
#6

Do think the error I report here is related to the Gio requirements ? Transfert node binaries to production server

#7

My dream is to catchup with memotechnics and learn a riddle that enable me to memorize some SSH key :o)

#8

No, the node will crash if too little RAM is present and no swap area is defined. I believe the issue you see on that error seems to be compiling without libsodium - I’ve had it before and will comment over there

#9

Yes indeed, that was the problem. I forgot to rebuild libsodium before running the node on my-other-server.

As a summary of this thread and Transfert node binaries to production server.

A secured relay-node can be a Virtual Private Server (avoid docker images) running with 2Gio where cardano’s binaries are transferred (securely eg. rsync + SSH ) after a proper local off internet compilation with libsodium.
On that VPS, one should also :

  • build and install libsodium
  • close all unnecessary ports
  • change default port for an eventual SSH authentication by key.

Ideally 2 or more node-relays should be distributed in different datacenters (maybe even on continent ?)
Obviously all servers should have exactly the same OS version.

Do you see something else ?

#10

Your summary looks good - thank you for that.

The only recommendation I would have is to use (at least) 4GiB RAM for production nodes

1 Like
#11

Hi all,

Reading @junada made me shiver, thanks for being so strong and sharing your experience, I wish you all the best.

Quick question, I hope it’s on topic, if not please help me better direct it. I would like to have the relay node(s) as AWS instances. So far it seems OK as long as system requirements are met. However:

-AWS security groups only seem to allow SSH on port 22 (and I want to change that);
-people have reported major issues when trying to install ufw to AWS instances (eg get locked out)

Am I missing something? Actually, I know I am, as I am really new to this :slight_smile: But how can I get the core to talk to AWS relay(s) securely if it must use port 22 to do so?

Thanks in advance for any advice,

A

1 Like
#12

AWS security groups don’t care which port you have for SSH - just open access to tcp/2222 or whatever. A better security measure would be to only allow SSH from your Home IP.
Also, Personally, I wouldn’t use UFW if I was using AWS - just use security groups, and properly segment your network. A simple VPC with public and private subnets should suffice. Place your relay nodes in the public and Block producing node in private. You might need a NAT for your block producing node to talk to the outside world if you place it in a private subnet.

4 Likes
#13

hi Greg,

thank you for your input. The first part of your answer put my mind at ease a little (without being complacent). I only allow SSH and other TCP from my personal IPs. I will not even try UFW, even if I love it for local machines.

In the second part you loose me a little (I am very new to this). Why would I need a virtual cloud and a network address translator (eg router)? I guess you are thinking of a setup where the core AND relays are all on VMs (AWS, Google VM, etc)? But, if I have the relay(s) as VMs, connecting to the blockchain, and the core as a physical machine (eg home or office) only talking to them, I wouldn’t need the VPC and NAT, right?

While I have your ear, seeing as you clearly know more than I: let’s say I have 2 relays (VMs) and 1 core (local). I want to sign all transactions offline and then launch them back signed. Can the core take downtime for that to happen securely? Can I double the set up and run 2 cores, connected to 2 relays each (essentially one as backup during downtime)? And if so, can the 2 cores be an identical copy of each other?

Hope the last bit made sense :slight_smile:

Thank you in advance for your time,

A

#14

I’ll answer the first question as that’s where my expertise lies, and not on cardano specifically.

Yes, I was under the impression you would be hosting the whole thing inside AWS so I suggested the typical way you would structure a VPC. The NAT wouldn’t be necessary if you don’t have any nodes in a private subnet. You can still use a simple VPC with public subnets in that case. I believe you can no longer create ec2 instances without VPC’s.

2 Likes
#15

I’m sorta new to firewall rules and IP addressing conventions. I’ve come across them in the past, but haven’t delved into them much. AWS security groups looks like a firewall, filtering incoming and outbound traffic by ports and services/protocols, etc.
The only rule I have so far is the default rule AWS provided, allowing incoming ssh traffic from my home public IP subnet to my AWS relay…I did try to select “my IP” from the drop-down menu as a new rule, but it said “no changes made” after I saved the rule, so I assume the best you can do in my case is to allows traffic from a particular subnet instead of a particular IP address my BP uses, which I’m hoping it means that the public IP addresses on the subnet my ISP (my BP is at home, not on AWS) has provided to me (with network address x.x.x.0/32 in this instance) has all the host IP addresses on that subnet reserved only for my use; otherwise, different hosts on the same subnet as mine would have an SSH port available them to try to connect to my AWS relay I suppose, or is my reasoning wrong?
I get the error: ‘CIDR block, a security group ID or a prefix list has to be specified’ when I try to choose the IP specific to my BP.
Maybe a firewall like ufw would allow for more specificity (like just one IP address) than AWS security groups - I don’t know, and I don’t know if it would make any difference in terms of having more security.

Oops! Just found out /32 means there is only 1 network IP address and no host IP address, so I believe my default gateway is the only IP address allowed to ssh to my AWS relay.
Should have looked into subnet masks…didn’t know there was such a big difference between /24 and /32.

Too many edits on this one, but I’m just starting to understand IP addressing. :confused:

#16

Hardware wallets scare me…lol. I’m afraid I’ll screw up the process to securing the device and lock myself out permanently. I feel safer using LUKS-encrypted micro sd cards and USBs, although they are very likely not as safe as a hardware wallet.

#17

Hey guys I have these as security groups for my relay, I’m still getting used to this… would recommend more or less?

Default sg- open to everything (should I delete?)
1st - open inbound 22 from my IP address, open outbound.
2nd - Limit inbound and outbound through (custom port number).

Also for my BP
allow default outgoing
deny default incoming
limit proto tcp from any to any port through (same custom port as ‘2nd’)
allow from relay IP to any port (same custom port as '2nd) proto tcp

#18

Only allow outgoing you actually need (updates and other relays in your topology);
Incoming from your BP and other relays only on the port you are running the node on;
Change the ssh port to something other than 22 and only allow incoming to it from your local ip.

For your BP you should do much the same, except only to and from your own relays.

Cheers,

A

#20

So if I’m understanding you correctly;
delete the default SG and 2nd SG
for 1st- open inbound (custom port #) from my BP IP address, allow outgoing.

BP node should stay the same

Sorry if this seems redundant I just want to make sure i get the security parameters correct and I am not at all an expert. Hope to continue to learn over time.

1 Like
#21

Regarding how to secure your server

  • change ssh port
  • deactivate ssh for root user
  • install google_auth
  • install fail2ban
  • etc,

And more here…

4 Likes