Have read/tried pretty much all I could find on the boards in regards to this.
I’ve got 3 servers on Contabo with below stats.
Setting clientloglimit 100000000 has been latest entry, it appears to have lessened the misses (they where typicall as high as 2%) but still incrementing. I’ve been researching timing because one NODE shows more time sync errors than other 2 (setup same on all 3).
Any other suggestions or these are unavoidable to a degree?
Can be related (missed slots) with servers which are running on VPS services… I had them before to migrate my BP to dedicated/bare metal server but never lost a block.
Time synchronization looks ok (my BP has 15 ms)…
You can try to edit cnode.sh script on all nodes (+ restart) and check again; uncomment the lines
Forgot to mention I did set CPU_CORES to 8 (8 core on Contabo), then tried 6.
Now set to 4 (is that the max it should be set to?).
Uncommented MEMPOOL, we’ll see.
Thanks Alex.
If you are using a VPS then you are sharing CPU resources with other users. This means that if your node does not get scheduled a CPU right at the start of each new slot then it will skip doing that slot leadership check.
If you use dedicated hardware then you can ensure that some CPUs are dedicated to your block producer so that they are always available. Nevertheless, you can still get missed leadership checks when Haskell does its garbage collection memory copying because this also can tie up the processor at the wrong time.
One way around the garbage collection cause is to then run your node with RTS settings that make the Haskell executable use the “nonmoving” garbage collector: “+RTS -N --nonmoving-gc -RTS”. But this has other problems: Using the nonmoving-gc results in freed memory not being returned to the OS properly so then you need to restart your node every day or so. Arrrgh!
I wish IOG would upgrade the code base to compile under ghc version 9.2.x instead of the older 8.10.7 because apparently the Hasekll nonmoving-gc works properly in 9.2.x or later.
I run my block producer on a low power machine and I get no missed leadership checks at all when running with the nonmoving-gc. I just restart often.
Others on this forum have different opinions on how to minimise missed leadership checks. If you do some searches on this forum you will find numerous posts about the problem.
Do NOT uncomment the MEMPOOL, and when it comes to Contabo you could have 1000 vCPUs,
IF the host-node is overshared you get nothing.
Contabo uses Desktop-PC (so no real server hardware) and that hardware is overshared on top of that.
So, basically, you can be lucky and end up on a new host-server and get some resources, but this being lucky ends the moment the host-server gets filled up with customers. And when all other customers have high loads, while you run your nodes, then your VPS just gets like nothing at all.
Imagine you took a VPS from them with 8 vCPUs, which are basically CPU-Threads, and on an overshared host-server you maybe get 5% out of 1 CPU-Thread, that means with all your 8 vCPUs you get maybe around ~40% of 1 CPU thread,…
So long story short, if you want to run your Pool seriously, do NOT use Contabo.
It’s like crossing the ocean in a plane fully made out of paper on a rainy day - good luck!
come on! Seriouslly? My relay is on contabo (VPS) and I don’t have any issues… maybe its not the best provider but for a small pool the price is more than ok
I can see the differences (CPU) only when I’m building/compiling the node… but to mint blocks there is no need for powerful CPU
I am also using Contabo and never had a problem. For the block producer I am using VDS (S). Having dedicated CPUs for that particular time constrained job is good enough. Together with some tweeking Haskell runtime, missed slot leader is no problem.
This is not true. And I’ve never had the experience you call “overshared”. Whatever your problems you have with Contabo - spreading FUD here is - IMO not ok.
This is the same for all VPS - as it follows the definition of VPS. If you want bare metal you can order that from Contabo as well. If you order VPS from any of the other hosting service provider you don’t get bare metal either.
I, for myself, did go for VDS for the block producer and had never a missed scheduled block. Loosing a height battle does not count here.
A cardano node does not need a lot of bandwidth, my nodes are using less than 1 Mbps in average. Latency is very important, of course, and you need more bandwidth to transfer the blocks quickly, but having “good network performance” is easy. But good CPU and disk performance are more difficult to provide.
Just take a look at the “stolen” CPU here in this vmstat output, which is from a Contabo VPS with 32 GB RAM struggling to run a cardano-node:
You are probably lucky, at least for now. Anyway, I tested many hosting providers, and Contabo was the only one where I saw stolen CPU. And most of the people complaining about many missed slot leader checks are using Contabo. For other applications, Contabo might be ok, but I would prefer any other hosting for a cardano node, after what I’ve heard and seen about them.
There’s a simple solution: go ahead for a bare metal host. That’s also available from Contabo and you can be sure it’s your machine only. I, for myself went for VDS which does not share CPU’s, thus being able to produce a block in time. When I started operations I did spend quite some time (actually weeks) to optimize (reduce) missed slot leader checks. After going VDS and optimizing Haskell runtime parameters, the bp was fine and I did not miss any scheduled block since then.
I’m going to challenge that; actually it does happen to all VPS, because that is the nature of VPS. You don’t get dedicated CPU’s - hence the name virtual in VPS.
Over and over I’ve seen people using AWS, Azure, and other services having their nodes crashing because of limited resources which includes RAM as well. IMO Contabo sells what they promise - not more not less. So far, I haven’t found any VPS hosting services with guaranteed real time performance.
Don’t get mad at me, but IMO it’s not the fault of Contabo, that you did not do your homework to set up your system properly. Others did, and by the number of Contabo hosted Cardano pools they seem to be successful.
And, btw., I’m not in any way affiliated with Contabo, and if you don’t like their services then use some other one. But I don’t like bad-mouthing some company because of your own mistakes.
No, it does not happen to any VPS. I tested many providers. Try Hetzner and see the quality. There are also other provider a lot more expensive which also have excellent quality. I also have a few VPSes on rented hardware in a DC and there are no missed slot leader checks.
An I have also bare metal servers with SSD disks where I see missed slot leader checks.
