SPOs, Do not repeat my mistakes, Keep your Core Node Safe

Staking & Delegation Stake Pool Security
1

Dear Awesome Cardano Staking Pool Operators,

My name is Jun, I am a Korean pool operator of HAPPY.

The reason I am posting this to here is to share my mistakes and prevent similar incident happening to you guys.

Just consider that I am just a graduate student in Korea, not a security expert, and I think most of you are not expert neither.

I made a huge mistake, and make a security breach to my Core node.

First, Let me share my security settings.

  1. My SSH config: Password disabled, only accessible with my private ssh key.

  2. My Port setting: Originally Open to 2222(ssh port), 3001(Core node), 3002~3005 (my ITN node)

  3. My private key setting: Originally stored in cold usb storage as a zipped format with password (using zip -er)

  4. My Core and Relay Setup: Core is in my home, bare metal, 5 Relays are in AWS with protected by each ssh keys and UFW.

I think the above settings were okay, but I made mistake from here.

I personally experimenting the docker, to dockerize my node for easy diploy and setup.

During the experimentation, I opened port 2375. and link that to my docker socket. (MISTAKE)

The docker user group have sudo previllege.

As I didn’t imagine I can be hacked with docker (I am just learning docker) I just liked docker socket to external port and opened it, So I can access my docker engine from outside easily.

I know this is very stupid mistake now, But I didn’t even imagine about this possiblility. (Yes you can blame me).

Yesterday, My pool pledge is suddenly drop to 740k, the original amount I set before I add my ITN rewards.

I thought that might be caused by a very long lived tx, because I usually set TTL to 10000.

So I uploaded my cold key without doubt. and unlock the zipfile (SECOND MISTAKE, never upload key file to machine when your pool is changed not by youself.)

And I increased pledge my 1M again.

1 hour later, The pledge dropped to 740k again. Then I notice that something is seriously wrong.

Then right after that, My all pledge transferred to the hacker’s wallet. (TX: https://explorer.cardano.org/en/transaction?id=ef8ac1c667084018315cd080001a3d62d513afa51f1bcf1847684760afac2747)

During the investigation, I found out what my mistakes, and I found an alien docker image “zbrtgwlxz:latest” is in my docker image list.

(!!! MAKE SURE that you don’t have this alien image if you are using docker !!!)

And that image was a hacking tool. (Threat Alert: Attacker Building Malicious Images Directly on Your Host)

I am so regretful for what I missed, and for my mistake, But I need to share exactly what happened to me and prevent similar incident.

Thank you for reading this, and Be safe.

Jun.

76 Likes
2

Hopefully everyone else that has this open can learn from your experience. Thanks for sharing!

4 Likes
3

@junada, thank you for the courage to publish this and possibly protect others. I can’t imagine how you must be feeling right now and wish you all the best for the future.

15 Likes
4

Thank you for sharing your story! Even at the cost of possibly being regarded “insecure” - for me personally your openness is the opposite. :slight_smile:

As a former colleague from work repeated endlessly: Always keep complexity as low as possible! And docker is not really part of this mentality…

4 Likes
6

@junada as said before I am very sorry for your loss.

Take all the time that is needed to feel better again. I hope you will get stronger out of this and continue to run a pool after what has happened.

If there is anything I can do feel free to send me a DM here or on Slack.

8 Likes
7

Thank you for your story mate. I think this is really important sharing such stories to prevent further attacks on other pool operators. This will be really challenging to stay strong so wish you patience and strength to forget it and go further!

3 Likes
8

I am very sorry to hear about your loss. Thank you for the courage to tell your story and also let community learn from it. This witness of strong character and I wish you will rise from the ashes of this.

4 Likes
9

@junada very impressed that you have the guts to share this helpful & detailed incident report under these circumstances… I am sorry for your experience :sweat: but our community as a whole is so much better for your coming forward about it :heart_eyes:

6 Likes
10

Thank you for having the courage and discipline to describe what happened so transparently (a characteristic of the Cardano community) and a sobering reminder to us all not to be complacent. The security reviews everyone will be doing today can be more efficient and you’ve possibly saved people hundreds of hrs.

3 Likes
11

Sorry sorry to hear about the funds you have lost. Major kudos for sharing your story to try and ensure this does not happen to anyone else. Very impressed and major respect for doing that

5 Likes
12

I feel sorry for you man. It’s sad that people trying to run a legit business are being targeted by hackers.
Are you still going to run your stakepool?

1 Like
13

Thank you Jun for sharing your story with us, it breaks my heart. :disappointed_relieved:
Please take all the time you need to recover yourself, you will get out of this being stronger than before.

I belive in you and the operator community will support you! :heart:

Thanks again for sharing to protect others out there. :pray:

4 Likes
14

Thank you very much.

You experience is more than useful for all of us who are not security experts.

We all know that the best way to scale is diagonaly : more operators (horizontal scaling) with better security knowledge (vertical scaling).
This is exactly what you are helping us to do.

Thank you.

2 Likes
15

It really shows your strong personality if you still care and want to protect others from the same mistake after you just lost so much money…

I really hope you come back stronger after this event. We are a community and help each other if someone made a mistake.

So thanks again for sharing your story and stay strong!

6 Likes
16

We appreciate you sharing, Jun. Thank you for your courage.

2 Likes
17

Sorry to hear that. Maybe we, the community, can help you somehow to get back on track. Stay strong!

2 Likes
18

I’m really sorry for what happened. Guess we all pay what we learn… Hope this will make you stronger, keep up the good work. And thank you for honest info

2 Likes
19

As others have written before, this is an awesome demonstration of generosity and humility, exemplary.

And there are things we can do as a community to soften the blow this valuable member has suffered @Gilianscheatday, we don’t even have to wait for a formal proposal.

@junada if you would be so kind as to share a wallet address with me in DM, I would be honored to send you some ADA, once Yoroi is back up. A gift, and a token of my appreciation for the wonderful contributions you have made in our community :pray::dove:.

It won’t be much, but if others feel as I do, perhaps it will make a difference in your life :rainbow:.
Even the best experts make mistakes, not everybody can use those to grow. Good luck @junada!

10 Likes
20

Hi Jun, I can imagine how do you feel now, but you will recover from this. I’m sure the community will do everything to help. Thanks for sharing your experience with us. Stay strong! :muscle:

2 Likes