Hey,
Recently I started my own Cardano stakepool with two relay nodes.
I made a flow overview. Any feedback on this concerning RTT / …?
All good,
Don’t forget to install chrony on all nodes
Cheers,
1 Like
Just implemented,
Thanks Alex
1 Like
One more thing… I see your BP is currently running as a Relay (you should see CORE) , did u set the start script for BP?
Indeed,
A bit further in the guide I applied this to the block producer
https://www.coincashew.com/coins/overview-ada/guide-how-to-build-a-haskell-stakepool-node/part-iii-operation/generating-keys-for-the-block-producing-node#:~:text=Update%20your%20startup%20script%20with%20the%20new%20KES%2C%20VRF%20and%20Operation%20Certificate.
However, on my block producer I can see two start files:
startBlockProducingNode.sh
startCardanoNode.sh
Should I swap the content of startBlockProducingNode.sh into startCardanoNode.sh and remove startBlockProducingNode.sh afterwards to make it work?
Hello,
Yes you should only have one startup script. I think your systemctl service runs “startCardanoNode.sh”
So copy the content of your startBlockProducingNode.sh to the startCardanoNode.sh and delete the first one. Then restart your cardano service.
In find this part of the coincashew guide confusing. I will propose a change soon.
1 Like
Perfect, it say’s Core - Mainnet now at the top 
Thanks Kirael and Alex
2 Likes
Hi,
My stake pool has been running for almost 4 days and I’m thrilled to have seen 1 block adopted. However, the number of missed slot leader checks seems high. I’ve read that this shouldn’t be too high, but I don’t have a reference point.
Is this normal?
Hi,
Should be fine, the missed blocks/slots is because the HW setup and can be seen more often for VPS setup… but should be fine
Cheers,
1 Like
I would say that’s quite a high number of missed slots. How much ram do you have? I’m running all of my nodes on 24GB VPS servers with ZRAM Swap added and I don’t have any missed slots. I’ve also added the following RTS settings to the startup script “–nonmoving-gc --disable-delayed-os-memory-return”.
1 Like
Hi,
I’m using 16GB RAM in my Block producer and the 2 relay nodes each.
I’ll have a look at the ZRAM Linux module.
1 Like
If you want your haskell nonmoving garbage collector to work better, ensure you compile your node software with ghc version 9.2.7. The cardano-node does compile with ghc 9.2.7 now but I am not sure what the coincashew scripts do.
2 Likes
Thanks for the tip. I’ll give that a go!
1 Like
I’ve added the command at the ~/.bashrc file and will monitor it for a few days.
I’ll keep you posted.
I’ll test the Garbage Collector script first.
If the missed slot leader keeps around 3% I’ll test the ZRAM implementation.
I’m planning to follow this guide:
https://fosspost.org/enable-zram-on-linux-better-system-performance/#:~:text=zRAM%20is%20a%20Linux%20kernel,on%20SSD%20or%20HDD%20devices.
Should I do this on all my nodes (Block and relay’s) and will this be fine with16GB of RAM?
1 Like
I would say you definitely need zram swap as the min required RAM for cardano node 8.1.1 is 24GB. I wouldn’t risk running it without it. My servers are all 24GB they use the swap I’ve set up. I have it set up on my core and all three relays. Cardano node is regularly using over 20GB.
I haven’t yet tried @Terminada’s suggestion of compiling with ghc 9.2.7.
I’ve found that the only way to ensure no missed slots is by using --nonmoving-gc. I’ve inserted it into my startup scripts as follows. N is the number of CPUs.
Core: cardano-node +RTS -N8 --nonmoving-gc --disable-delayed-os-memory-return -RTS run
Relays: cardano-node +RTS -N8 --disable-delayed-os-memory-return -RTS run
2 Likes
The problem with cardano-node and the haskell garbage collector (gc) is that it needs to copy significant amounts of memory and it needs to ensure that this memory is not changed whilst it is doing the garbage collection. The normal garbage collector “stops the world” in order to copy the memory holus bolus, whereas the nonmoving-gc tries to keep parts and copy others (but the implementation details are over my head). When the gc “stops the world”, then if your block producer needs to do a slot leadership check during this time, it won’t do it because the thread won’t run in time. On a relay this won’t matter as much because it isn’t doing anything so time critical like a slot leadership check.
Consequently, I run my BP with the nonmoving-gc but my relays with the normal gc. Using the nonmoving-gc was the only way I could be sure to not miss any slot leader checks on my BP, and only if I restarted the cardano-node process within a day of needing to make a block.
I have a couple of relays on x86 machines that I have left with only 16G ram (and 16G swap) and these have kept running without any problems in relay mode with the default haskell gc. I haven’t seen any OOM (out of memory) killing events. But, the cardano-node software uses more computational resources when running in BP mode.
The previous versions of cardano-node were not able to be compiled with ghc 9.2.x and running my bp with the nonmoving-gc resulted in a gradual increase in RAM usage. Eventually the cardano-node process would get OOM killed if I didn’t restart it every day or two. I used to run my bp on an x86 machine that had 24G ram and 24G swap and it wasn’t using zram.
I also have a low powered ARM machine which is the equivalent of 4 Raspberry Pis (16 arm Cortex A72 cores clocked at 2GHz with 64G ram). It is an awesome networking board but with 8yr old processor technology. I experimented with running cardano-node in a virtual machine with 32G ram and 32G swap on this machine. The node software was compiled with ghc 9.2.7 and I set it to run in BP mode (on mainnet) between the times I had blocks to produce. I left it to run for up to 1 week at a time and it seemed to go pretty well. It only missed around 8 slot leader checks at epoch transitions but otherwise it ran mostly with only a few missed slot leader checks every now and then (presumably at gc events). I could make it miss a few more slot leader checks if I made it do an epoch leader log calculation. When I left it running without a restart for a week, RAM usage seemed to plateau at 25G and swap around 5G.
This was a much better result than I was seeing with previous versions of cardano-node. I tried a similar experiment with cardano-node version 1.35.7 compiled with ghc 8.10.7 and this process got OOM killed after exhausting all RAM and swap a couple of days into continuous running. Plus, it missed hundreds of slot leader checks.
The nonmoving-gc with ghc 8.10.7 doesn’t properly release RAM back to the OS. ghc 9.2.7 has a much better performing nonmoving-gc implementation, at least in terms of releasing memory back to the OS. The haskell garbage collector is under active development and apparently will get better in later versions of haskell. So this is why I would recommend compiling cardano-node with the latest version of ghc that IOG has made compatible, especially if you want to use the nonmoving-gc.
I am currently running my mainnet bp on my arm machine and this is what it’s RAM and swap usage looks like after 34hrs of continuous running using the nonmoving-gc:
free
total used free shared buff/cache available
Mem: 32812936 24539144 2606584 696 6005188 8273792
Swap: 32260088 342736 31917352
It has missed 9 slot leader checks during this 34hrs.
To give you an idea of how under powered this ARM machine is: When I restart cardano-node, it takes around 10-11 minutes to complete all the usual blockchain checks before it is fully running, establishing incoming / outgoing connections, and doing slot checks. Unfortunately this start up sequence seems to use only 1 core, with the other 7 cores the vm has access to, remaining unused.
I don’t want to chance failing to make a block, so I still restart my BP several hours before I know a block will be due. This is pretty easy for me to monitor since my pool is small and I only get from zero to a few blocks to produce each epoch.
3 Likes
Interestingly my bp, running on my low power ARM machine, just went through the epoch transition with 0 extra missed slot leader checks. Here is how things currently look for my bp:
Node Started: Tue 18 Jul 2023 08:17:01 AEST (Running: 48 hrs 7 mins)
RTS settings: cardano-node +RTS -N --nonmoving-gc -RTS
Missed slots: 9
Memory use:
total used free shared buff/cache available
Mem: 32812936 24543024 2677644 696 5930192 8269912
Swap: 32260088 331984 31928104
The memory usage has plateaued using the nonmoving-gc if you compare with my previous post.
Most of the 9 missed slot leader checks occurred a day ago when I made it do a leadership log for the next epoch which obviously puts more load on the processor at the same time as it is running in block producer mode.
1 Like
I tested my pool for 2 days with the GC settings.
It’s a little bit better. Around (3500) 1.8% missed slot leader checks.
I’ll implemented the ZRAM on my 16GB RAM BP and monitor it for a few days as well.
Otherwise I will upgrade to 30GB of RAM.
I’ll keep you posted.
Thanks for the help folks
1 Like
Nice detailed description.
How did you get that nice overview containing the Node Started, RTS settings, …
1 Like