Missed slot leader checks

I have a 6vCPU with 32G RAM and I also have missed slots… but this will not lead to lose blocks … as far as I know everyone has these missed slots… ignore them

Thanks @Alexd1985

Probably need more data but following [SHARD] Pools great article Solved: How to minimise Missed Slot Leader checks / missed blocks with stake pool ADA Cardano node - How to get stuff done. to “change the Garbage Collection to use Nonmoving-GC which uses the newer concurrent mark-and-sweep garbage collector, and it doesn’t block other processes.” Seems to have reduced the missed slots I have been seeing. I also tried changing from timedatectl NTP to chrony NTP which did not seem to have much impact - but using chronyc tracking is a great tool to see how the timesync is performing.

As per the recommendation -

add the following line to your .bashrc file (credit orpheus-ant)
export GHCRTS=’-N -T -I0 -A16m --disable-delayed-os-memory-return --nonmoving-gc’

Its too early to tell but initial stats look promising for my case as the missed slots have roughly halved

Hi, thanks for your feedback. Glad to hear that this seems to be working for you. I have already tried this, and unfortunately for me, it didn’t change anything.

I am using this for the last two and a half epochs and i am at 0 missed slots except at epoch change +RTS -N -H3G -qg -qb --nonmoving-gc -RTS
I tried a lot of parameters but this is the only one that is so stable and with 0 missed slots.
A really important note, parameters tested vary from a machine to another, HW plays an important role so don’t expect the same parameters to work on everyone.
For the one i am currently using 16GB are not enough, i got 32 so i am good.
Don’t look to just copy someone else parameters, play around depending on your system,
to understand more about the options read 5.7. Running a compiled program — Glasgow Haskell Compiler 9.2.1 User's Guide

Hi guys, just a point. I was just updating to cardano node to 1.31.0 by following this awesome post → Update Cardano Node to 1.31.0 - for Coincashew users

But when running ghcup upgrade command I was getting this message → ghcup: Most RTS options are disabled. Link with -rtsopts to enable them.

To resolve this, I had to clear the environment variable GHCRTS which was set as -N -T -I0 -A16m --disable-delayed-os-memory-return --nonmoving-gc from the above on trying to reduce the missing slot leader checks. This was cleared by running unset GHCRTS, and I now can subsequently upgrade :+1:

Thanks for the advice. I will look into this :+1:

1 Like

@Xpriens
Do you have the +RTS -N -H3G -qg -qb --nonmoving-gc -RTS in the startBlockProducingNode.sh file on the line cardano-node run --topology..... or as the GHCRTS environment variable? Thanks

only on the node’s start up script, no reason to use it anywhere else

:+1: Thanks

cardano-node +RTS -N -H3G -qg -qb --nonmoving-gc -RTS run .....
to be more precise

Hi, I’ve tried it both in the startBlockProducingNode.sh file and as a GHCRTS environment variable. Neither seemed to make much of a difference

What exactly do you mean? You still have missed slot checks or no difference on system load?

Oh yeah - if people are still strugging with this, try the following:

+RTS -N(num of PHYSICAL CPUs) -xn -G2 -T -S -A128m -n1m -C0 -I0.3 -Iw3600 -O4000m -RTS

I think the magic is in -C0 - this tells GHC to do as MANY context switches as it can and not wait the default 20ms before trying to context switch.

-O4000m means don’t do old GC before it reaches 4000 MB. -n1m means split up chunks in heap to 1 MB

3 Likes

but on version 8.10.7 right?

If possible, yes. 8.10.7 has improvements for non copying GC

1 Like

When I tried before, I had tried both variations but neither made any difference, I still had missed slot checks. But today, I updated the startBlockProducingNode.sh file with what you suggested cardano-node +RTS -N -H3G -qg -qb --nonmoving-gc -RTS run ..... (I had variations of this before) and fingers crossed, but it seems to be working - after 3 hours 45 minutes I have no missed slots

2 Likes

great keep us informed and if it won’t work take a look at @weebl2000 post

1 Like

23 hours and zero missed slot leader checks :pray: :pray: :bowing_man:

Thanks so much

nice but keep checking, 23h is not that much. goal is the 0 missed slot checks but also check system health over time to make sure everything is fine.

1 Like