No Space Left on Device shortly after Running Relay

Staking & Delegation Setup a Stake Pool
1

Problem

The cardano-node relay stops running after about 15min, and can no longer SSH into the VM (the serial port log on the VM in GCP says ‘not enough disk space’ or no space left)

My machine is a e2-medium (2 vCPU, 4GB memory) w/ 10GB SSD VM hosted as a compute engine in GCP. I’m following instructions from the Stake Pool Course.

Here’s the log right when it happened:

no-space-left-on-device
no-space-left-on-device1797×692 210 KB

Debugging & Questions

command used:

cardano-node run \
 --topology testnet-topology.json \
 --database-path db \
 --socket-path db/node.socket \
 --host-addr 0.0.0.0 \
 --port 3001 \
 --config testnet-config.json

  1. When I do various ls and du commands to check the size of my ~/cardano-node/db directory, its contents don’t look that large (the .dat files), does this imply that maybe it’s a file that’s not in ~/cardano-node/db that is consuming too much disk space? Log file hanging around somewhere?

  2. Here is the output from running htop, why is cardano-node listed so many times, that normal? 1 instance of it running but using or sharing more more CPU’s?

    htop
    htop1899×871 357 KB

  3. I run the commands tree and df to observe memory and disk consumption, any key metrics to look for when running those? When I ran ‘tree’ I noticed the “available memory” was slowly dropping over time.

I captured this log right after a fresh cardano-node install but before running a node.
before-running-cardano-node

  1. Any params I can tweak? Flags passed to cardano-node run. I’m checking them now.

  2. Need to update the testnet-config.json at all?

I have fresh install backup up via machine image and snapshot so can iterate on any suggestions! Thanks

2

Hello,

10Gb ssd it’s not enough, you will need more, at least 40gb to be safe.

You should check the size of ledger folder, logs folder, etc

Minimum requirements for running a stake pool

In terms of hardware, you should have the following available:

  • 4 GB of RAM
  • 24 GB of hard disk space
  • a good network connection and about 1 GB of bandwidth per hour
  • a public IP4 address

Note that processor speed is not a significant factor for running a stake pool.

Cheers,

2 Likes
3

Thanks Alexd1985 I will try that.

Does a relay node download the entire blockchain? Is it validating transactions?

I think since I’m running cardano-node with --database-path db the answer is Yes.

I believe it does, but it’s hard to tell from these docs:
https://docs.cardano.org/en/latest/getting-started/stake-pool-operators/index.html.

https://docs.cardano.org/projects/cardano-db-sync/en/latest/getting-started/validation.html?highlight=blockchain%20history#validation

https://docs.cardano.org/en/latest/explore-cardano/cardano-architecture-overview/index.html

most docs seem to not distinguish if it’s a Relay or Core/BP node. I understand it’s still the same 'cardano-node` program.

I will start at +24GB.
https://cardano-foundation.gitbook.io/stake-pool-course/stake-pool-guide/system-setup/aws recommends 24GB

https://cardano-foundation.gitbook.io/stake-pool-course/stake-pool-guide/getting-started/install-node recommends 10GB

https://docs.cardano.org/projects/cardano-node/en/latest/getting-started/install.html recommends 10GB

Will

4

Relay or BP are the same, downloading all blockchain, validating transaction, etc

The only difference is that only the BP can create blocks…

1 Like
5

First and foremost, thanks @Alexd1985 for your answer.
This is at least the third time I’m seeing something like this, and it is really frustrating.
I get that minimum requirements are just that – minimum, but you need “at least 40gb [of storage] to be safe”. You also need 16GB of memory for the node to actually try to function.
I think “Realistic Hardware Requirements” would really help noobs like me, rather than having to fail at setting up a node several times then search the internet for error messages and cobble together useful bits of information like this from disparate sites.
Again, I really appreciate this answer because I was using 30GB of storage, thinking that was above the requirements and might be sufficient.
I understand that things are relatively new and under intense development, but as someone trying to set up and run a node, I am having serious trouble finding the information I need.
I worry that other people trying to do the same thing are not finding what they need, getting frustrated, and giving up without speaking up. So, we end up with people turned off by Cardano, rather than more stake pool operators. Maybe that’s by design–so only people who are truly dedicated get to run a node?

6

40 G should be at limit… try 100G minim for disk space

7

Wow. OK. Thanks for the heads up.
BTW, I just found your How to set up a pool in a few minutes. Thank you for that!
What are the real world requirements? Is an updated list maintained somewhere?
So far, here is what I’ve found:

  • CPU – compute power is not that important and the minimums do seem sufficient.
  • MEMORY – 16 GB seems to be what you actually need
  • STORAGE – 100 GB is a realistic amount
  • NETWORK – needs to be able to handle ~1GB / hour
  • STATIC IP ADDRESS – kind of obvious, but necessary

Is this correct? What am I missing?

Thank you so much for your help!

8
  • CPU – compute power is not that important and the minimums do seem sufficient.

min 4vCPU

  • MEMORY – 16 GB seems to be what you actually need

16G will be enough

  • STORAGE – 100 GB is a realistic amount

min 100G

  • NETWORK – needs to be able to handle ~1GB / hour

Gigabit connection will be enough

  • STATIC IP ADDRESS – kind of obvious, but necessary

U should use static IP otherwise u must use dns (if u will use IPs and they will change u will need again to register the pool with the new IPs)

1 Like