Too many Missed Slots happens, help me

Hello again,
I run my pool since 6 months ! Now I see why I never ever have written any blocks. My pool missed more than 500 slot leader checks in 9 hours.
image

1 Like

All POs had missed slots last night during the epoch transition (~400): I hope it will be fixed in the next software release

2 Likes

I had none missed other than 2 on startup since the changes I detailed above. However; I had ~200 missed right on the Epoch transition and the node seemed to go crazy during the transition (so did many others because I was also monitoring the connections to the other peers and 9 of them fell over at the transition). After about 15 minutes it was all fine again and it seemed to be the same with the peers as they all came back online too - I noticed lots of memory spikes during the transition - as others have pointed out hopefully it’s fixed in the next patch. Since then it’s been at 0 missed.

I am having the same issues, too many missed slots

It is frustrating :frowning: I dont know how to fix it :frowning:

hey, my specs:

  • running on AWS w 4vCPU and 16GB Ram
  • have 8GB swap
  • I was still getting significant missed slot leader checks (~0.25% - still more than I was comfortable with)

I followed suggestion from @shard and it looks to have drastically improved my missed leadership slots (been running for 9 hours - 4 missed slots leadership checks, will continue to monitor)

  1. Added to ~/.bashrc (still had a couple of missed leadership slots)
sudo nano ~/.bashrc
export GHCRTS='-N -T -I0 -A16m --disable-delayed-os-memory-return --nonmoving-gc'
source ~/.bashrc
  1. Edit the cardano-node run
cardano-node +RTS -N -RTS run

EDIT: I’m not sure how this affect CNCLI setup users as there is a
cardano-node ${CPU_RUNTIME[@]} run
- if you’ve successfully edit config, can you please share to help others? thanks

Depending if you are using a pi or not, I saw one config for arm64 cardano-node

3 Likes

hello:
do l lscpu (if you distro does not have that command there are other, just look it up) to find out how many cores you have, then go into cnode.sh and the variable CPU_CORES=xx where x is not higher that your cores, if you don’t have that variable add it here:

#!/usr/bin/env bash
# shellcheck disable=SC2086
#shellcheck source=/dev/null

. "$(dirname $0)"/env offline

######################################
# User Variables - Change as desired #
# Common variables set in env file   #
######################################

CPU_CORES=4            # Number of CPU cores cardano-node process has access to (please dont set higher than physical core count, 2-4 recommended)

######################################
# Do NOT modify code below           #
######################################

I was having a ton of misses, after that in 16 hours I only have 4 (0.0280%)
Let us know the results.
regards

2 Likes

UPDATE, after doing the CPU_CORES i am getting let than 1 missed per hour, after 37 hours i got:
Missed slot leader checks : 26 (0.0288%)

Dear Alex
I’ve tried to find when the missed slots occurred, but didn’t catch exact time…
Anyway I did not do anything till now because the blocks have been well produced.
Thank you so much for your help and concern…

this is a common issue so, ignore it for the moment.

And increasing… I upgraded 1.30.1 yesterday It appears to be missing every check since then.
image

sudo journalctl -e -f -u cnode
– Logs begin at Mon 2021-10-04 20:36:52 CEST. –
Oct 04 20:36:53 vmi660486.contaboserver.net systemd[1]: Started Cardano Node.
Oct 04 20:36:53 vmi660486.contaboserver.net cnode[560]: Guild scripts has now been upgraded to support cardano-node 1.30.1 or higher (1.29.0 found).
Oct 04 20:36:53 vmi660486.contaboserver.net cnode[560]: Please update cardano-node (note that you should ideally update your config too) or use tagged branches for older node version.
Oct 04 20:36:53 vmi660486.contaboserver.net cnode[560]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Oct 04 20:36:56 vmi660486.contaboserver.net cnode[560]: Listening on http://127.0.0.1:12798
Oct 04 20:38:14 vmi660486.contaboserver.net systemd[1]: Stopping Cardano Node…
Oct 04 20:38:19 vmi660486.contaboserver.net systemd[1]: cnode.service: State ā€˜stop-sigterm’ timed out. Killing.
Oct 04 20:38:19 vmi660486.contaboserver.net systemd[1]: cnode.service: Killing process 560 (bash) with signal SIGKILL.
Oct 04 20:38:19 vmi660486.contaboserver.net systemd[1]: cnode.service: Killing process 623 (cardano-node) with signal SIGKILL.
Oct 04 20:38:19 vmi660486.contaboserver.net systemd[1]: cnode.service: Main process exited, code=killed, status=9/KILL
Oct 04 20:38:19 vmi660486.contaboserver.net systemd[1]: cnode.service: Killing process 623 (cardano-node) with signal SIGKILL.
Oct 04 20:38:19 vmi660486.contaboserver.net systemd[1]: cnode.service: Failed with result ā€˜timeout’.
Oct 04 20:38:19 vmi660486.contaboserver.net systemd[1]: Stopped Cardano Node.
Oct 04 21:21:52 vmi660486.contaboserver.net systemd[1]: Started Cardano Node.
Oct 04 21:21:53 vmi660486.contaboserver.net cnode[18809]: Listening on http://127.0.0.1:12798

6 core, 16g of ram.

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 40 bits physical, 48 bits virtual
CPU(s): 6
On-line CPU(s) list: 0-5
Thread(s) per core: 1
Core(s) per socket: 6
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 49
Model name: AMD EPYC 7282 16-Core Processor
Stepping: 0
CPU MHz: 2794.748
BogoMIPS: 5589.49
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 64K
L1i cache: 64K
L2 cache: 512K
L3 cache: 16384K
NUMA node0 CPU(s): 0-5

224 now in less than a hour

1 Like

I did this and still missing what appears to be every check

I am not 100% sure but I dont think the issue was there before I updated. Would that still be wallet related?
I also dont know how to contact their support.

1 Like

The instructions tell me to login to a website thats asking for my seed phrase. No thanks

2 Likes

it’s a scammer, just reported

the missed slots can be related with Hardware configuration of the servers…

  • what is the Hardware configuration?
  • how many relays do you have?
  • can u add also the IOHK relays and check again? any improvements?
1 Like

Cantonbo VPS, no hardware.
2 relays. 1 relay is in the USA, the producer is in Germany along with the other relay. Would that cause the possible missed?

1 Like

I’m now using one of their VDS servers instead of VPS for the bp node. No serious issues with missed slots since migration from VPS to VDS.

Btw. it’s also important to use chrony as time sync service. Otherwise the clock skew of your server is too high.

2 Likes

I have a very small pool. The VDS servers would almost not pay for themselves

ADAAT is also quite small. Still - to me it’s more important to have a solid setup instead of making a quick buck.

It pays off since we have now 3 blocks validated. This attracts delegations which is the best way to grow. IMO.

Hello, are you using the RTS option when running the cardano-node? Like /usr/local/bin/cardano-node run +RTS … -RTS … ?