U see the missed slots also during the epoch or only when the epoch change?
Unfortunately, I get them throughout the epoch which makes me nervous because I have a small pool (Just got our first Block this Epoch). I don’t have any slots assigned Epoch 320 so I will be able to play with the settings a bit.
For example I have been up for 4 hours after I rebooted for an update and there is 1:21:08 left in the Epoch and I have 59 missed slots. .43% currently.
I wanted to tell u that will not affect ur blocks… u can ignore the missed slots
or, u can try something which I did and now I don’t see missed blocks anymore
edit the cnode.sh to use 4vCPUs
nano cnode.sh
CPU_CORES=4
save the file and restart the node, keep it under monitoring and check if u still see missed slots during the epoch
If yes then replace e erything inside cnode.sh with this one
and modify the lines
CPU_CORES=4 # Number of CPU cores cardano-node process has access to (please don't set higher than physical core count, 2-4 recommended)
MEMPOOL_BYTES=8388608 # Override mempool in bytes (Default: Do not override)
Awesome, I will try that after the new Epoch starts. To clarify, when you say the missed slots don’t impact making the block I know the chance is really low but are you saying that if my Missed Slot Leader check happened on the actual second I was supposed to make the block it would still go through?
Yeah, I think so…
@Alexd1985 I don’t think that is correct.
If you miss a slot leader check right at the slot your node is supposed to be the leader then you will miss producing the block. After all, this is what your node does to check if it should proceed with making a block.
I tried a few combinations with the processor settings but this one mentioned by Xpriens in the thread works really well for my setup.
+RTS -N -H3G -qg -qb --nonmoving-gc -RTS run
Thanks for all the help and advice!
Do you think it is any better than just?:
+RTS -N --nonmoving-gc -RTS
I think it is the nonmoving-gc that produces the minimum missed slot leader checks and I am not sure if any other additional options make any significant difference.
I know if I try to change the -N option to -N4 or -N6 it actually gets worse. I can try some other combinations and see what the total memory used is as another factor compared to missed slots. That might be a good indicator. I guess one advantage of being a smaller pool and knowing you are not getting a block for a full Epoch is you get more chances to test. : )
The nonmoving-gc isn’t working fully in the version of Haskell we need to compile the cardano-node: ghc 8.10.7. I believe it works better in version 9.2.x. Apparently with version 8.10.7 the memory doesn’t get released back to the OS properly from Haskell after garbage collections.
What I see with monmoving-gc is that cardano-node runs very well and I get zero missed slot leader checks but the memory usage, according to the OS, increases until it uses all the RAM (16G) and then starts using more swap. Gradually swap increases too and eventually the node gets slower. If I let it run for days like this then I do start seeing missed slot checks eventually.
Consequently, I restart cardano-node every day or so.
Memory not being returned to the OS by GHC 8.x is by design. To combat the “dead” memory usage you can use zramswap, it will compact empty regions into memory.
I usually end up at 10-16gb of memory used after a few days of running:
I do run with very exotic RTS params though and I run with eatmydata 
eatmydata cardano-node run
+RTS -N2 -w -xn -qb1 -G2 -T -S -A50m -AL2048m -n1m -C0 -I0.3 -Iw4500 -O2500m -F1.15 -RTS
1 Like
I have some 400+ missed checks for 15 hours so I will be trying this tweak too. Do I also need to do it with my relay node or just on the block producing node?
@stakeside you only need to do the RTS “tweaks” on the block producer because this is the one doing the slot leadership checks. You could try what @weebl2000 recommended above, except for the “eatmydata” wrapper part. Or you could try nonmoving-gc.
I am not sure whether eatmydata could result in a corrupted blockchain db if your cardano-node executable was to crash. Maybe @weebl2000 could clarify re: eatmydata?
1 Like
It could, but it’s unlikely. The largest write is the ledger anyway and if that gets corrupted it will just use the one older ledger.
The cardano-node behiavor with regards to determining when the database is “clean” is not very smart imho. Even if the cardano-node isn’t writing any files and the process gets killed forcibly (kill -9) it will not mark the database as clean and will verify the entire immutable chain. Like what?
Immutable will hardly ever get corrupted. I would opt for developing something like only marking immutable possibly not clean when writing to immutable, then removing that flag right after.
I think a lot of Daedalus users would be very happy if the “clean database logic” becomes smarter.
tl;dr: Yes eatmydata increases the possibility of corruption, but I think it’s highly unlikely and the cardano-node is currently very aggressive in checking everything when it’s not shut down cleanly so meh.
2 Likes
Thanks @weebl2000 what you said makes a lot of sense. In my systemd service file for cardano-node I had set: TimeoutStopSec=300 to make systemd wait longer before sending a kill signal. Watching it re-verify the entire immutable chain is annoying.
Do you find that wrapping the cardano-node executable with eatmydata results in less missed slot leader checks?
Also, I saw a note on github about getting cardano-node to compile with ghc 9.2.x. Do you know anything about that? If it could be compiled with 9.2.x I understand the nonmoving-gc will work better in terms of releasing the memory back to the OS.
I haven’t done an extensive deep dive into the difference between running with eatmydata and without. I think on most fast disks, SSD/NVMe it shouldn’t make too big of an impact. I do know cardano-node waits for fsyncs usually, but GHC is smart enough to schedule other threads while it waits for I/O. I think impact might be larger for slower disks, but again: nothing to substantiate this claim.
So it might not do anything, or help a little. The reason I did not investigate further is because I still get on average 0 slots missed per epoch running on 10 year old hardware with my GHC RTS params. 
I’m running my BP on a Intel(R) Core™ i5-3320M CPU @ 2.60GHz (dual-core with HT) and 16 GB of ram. It’s an old Lenovo ThinkPad X230 laptop, lol!
I also give the cardano-node process more more scheduling priority and higher IO scheduling.
In [Service] section:
Nice=-15
IOSchedulingClass=realtime
IOSchedulingPriority=1
I tried building cardano-node a few months ago with 9.2.x GHC but a lot of dependencies weren’t compatible with the newest GHC. Not sure if it can build on GHC 9 already now.
With reagrds to GHC 9, yes it should release memory back to the OS. Currently let’s say a chunk of 1 MB only has one 1 KB object residing in it, GHC will keep the entire 1 MB block claimed (but the OS will know only 1 KB is occupying it, so it can swap out the other 999KB). In the newer GHC, it should be smarter and not keep the 1 MB block claimed forever.
In the end though, cardano-node seems to equalize around a certain memory usage after running long enough to have hit epoch changeover and ledger snapshots writting. At this point memory claimed might be let’s say 14 GB when running with -xn. Only 3-4 GB will be used, the other 10~ ish GB will be mostly empty regions.
I’m not sure how GHC will behave in GHC 9. If peak memory usage still requires 14GB heap at one point, it will still use up those 14 GB at some points during running, just release more back to the OS after using it. But again: the OS knows these mostly empty blocks are usable by swapping them out. So if you are currently running with zramswap which can swap these regions out quite fast I doubt GHC 9 will reduce memory usage as opposed to that. It will probably be more efficient than zramswapping though.
tl;dr: I’m not 100% read up on the changes they made to non-copying collector but it should definitely reduce memory fragmentation. Probably yielding 2-3 GB of less wasted memory. That’s the number I’m usually seeing.
14718 MiB total memory in use (2883 MB lost due to fragmentation)
I’m currently running:
+RTS -N2 -w -xn -qb1 -G2 -T -S -A200m -AL2048m -n1m -C0 -I0.3 -Iw4500 -O2500m -F1.15 -RTS
Hopefully this is helps explaining it.
On a side note: I think P2P code might be a little more efficient cpu/memory wise than the current connectivity too.
1 Like
So far so good. I’ve been using Xpriens’ tweak for more than 5 hours now without any missed checks.
The first ledger snapshot will take place anywhere between 0 seconds and 12 hours. You need to wait 12 hours to see how it behaves to make sure it has taken a ledger snapshot. (To be very precise: the first snapshot is taken after it sees 2160 (k parameter) blocks after your last snapshot. So the time it will take is about: (2160 - lastLedgerSnapshotBlockHeight) * 20 seconds, not counting for missed slots by pools)
Only after your first snapshot after node start will it start taking snapshot at configured snapshot interval. (with some side notes)
1 Like
Thanks for mentioning those scheduling settings. I was setting Nice=-7 as I figured going much lower wouldn’t make much difference. I hadn’t set the IOScheduling Class and Priority but those make sense too. I can’t see any disadvantage in using all of those settings including the Nice=-15.
I haven’t been using zRam because I have 16G of RAM and my pool is small, so it is easy to restart the block producer 1hr before producing a block. This way the swap is never used around the time of block production so I assumed that zram could actually slow things.
Thanks for the great performance tips. I need to test them on my ARM machine which has pretty low performance.