20 Epochs without a block averaging around 300,000 in stake

Hi everybody - I have just gone 20 Epochs now without being allocated slot leader. My stake has been between approx 270,000 and 350,000 ADA with a Block Production ideal of between 0.23 and 0.3. I know this can be down to luck but this appears to be a bit extreme given that I was getting runs of blocks quite well for the first 18 months. At the end of the day it is extremely difficult to keep delegators when going this long without a block and I have to admit there appears to be little point in continuing at the moment. I have checked and double checked everything and see no reason or issues as to why my Stakepool is not getting allocated as slot leader anymore. Maybe it is just a poor run of luck by I am not sure how small pools are supposed to ever get of the ground when the system is designed this way.

2 Likes

If cncli shows no blocks, it means you will not get blocks. If in doubt with newest cardano-node version you can check leadership using built in tools cardano-cli query leadership-schedule ....

I had 8 epochs with stake between 100k and 200k+ without blocks and then I got 2 blocks with 20k. I didnā€™t believed cncli output, so I verified that with cardano-cli query leadership-schedule ....

I think you just need to stay patient.

2 Likes

The dead spots in that Poisson distribution are bloody awful but they can always be counted on to appear, as blocks can also always be counted to lump together in streaks. To me the frequency spectrum of that distribution does indeed seem to favour the large pools, because small pools canā€™t survive the thin parts of the block allocation schedule. It matters little if the sum of all outcomes will still be ā€œfairā€ if all our delegators have left in the meantime.

Roughly the chances of this happening to you, assuming a geometric average over those 20 epochs of 25% chance of block production, are 0.75^20 = 0.3% ā€¦ Iā€™ve not had a run literally like this but have been hit with similar anomalies in the block distribution that were equally unlikely.

I suppose the system architects would make the argument that if blocks were given more regularly (i.e. a ā€œlower frequencyā€ probability distribution) it would be more possible to anticipate when individual pools would produce blocks & therefore more possible to disrupt and/or game the system. Thatā€™s also why I believe, if you were ever able to attract the attention of one of the IOG scientists about this often-reported issue, that the Ouroboros function would be as unlikely as ever to change. :face_with_monocle:

2 Likes

Thanks - yes over time things should average out but as you say it doesnā€™t help much if you are left with no delegators by the time that happens :grinning:

With regards to the block allocation are we saying then that it is not just based on chance? Are unlikely streaks of bad luck just like getting 10 reds in a row in Roulette in a Casino when betting on black or does the protocol work differently and somehow the chances of winning are determined by other factors as well. I am still unclear on this?

1 Like

I built out this rough calculation for my pool to calculate my expected blocks per epoch for a pool.


Based on this I would have expected your pool to make around 5 blocks in this period. Iā€™m also calculating the probability of this happening around 0.57% of the time, which isnā€™t that impossible, but is really really bad luck.

I would say just keep the nodes synced, keep morale up and hope for a block soon. Iā€™d give it another few epochs before closing up shop.

P.S. You havenā€™t forgotten to rotate KES keys right?

2 Likes

Thanks - keys are fine and still have 40 days left - 0.57% luck seems really unlikely which worries me a bit. I had minted 26 blocks up until this period and while erratic this period appears way too long - just trying to think what else might be wrong :thinking:

1 Like

To confirm (and repeat) what was said here & in the two threads that were linked, yes it works differently than roulette spins, dice rolls, etc. which are said to be independent events.

The times at which blocks are chosen for a pool ā€” according to Ouroboros which is locally a pseudorandom number generator ā€” appear random within any given time interval, but these chosen times (and therefore the block counts) are not independent. The longer the interval is, the more likely the total number of blocks will be the expected value.

If you want to find out why this is true, you could follow the links above into the mathematical background or just adopt some of the analogies along the way that explain why Ouroboros and similar random number sequences are not really randomā€¦ i.e., they are mathematically assured to cover all possible outcomes eventually, which is the opposite of ā€œrandomā€. :face_with_monocle:

2 Likes

Thanks for the explanation - that is good to know

Appreciate everybodyā€™s help here. Does anyone have a simple check list that they could provide just to be sure my Node is operating OK. I did move it to the cloud some time ago and have checked with a couple of people but if anyone can add anything to the following if they think of anything that I could have missed.

Following checked

  • Node is showing as Core in gLiveView
  • Tip is increasing as it should
  • KES keys are current with 41 days remaining
  • Both relays can be seen from Block Producing Node
  • Both relays can see the BP Node
  • Node is 100% synced
  • Chrony is active and running
  • vrf key hash matches what is showing on Pooltool
  • Pool is showing as active on Pooltool and all other websites
  • All relevant keys and certificates are in the correct location

Anything else I can check to be sure that the Pool is registered correctly on the network to be eligible for slot leadership?

Hi

Having a similar issue, 6 epochs with no blocks with 364k staked. A few things? Does it matter that no new coins are being staked? I.e does the algo use this as a variable in the block distribution calculation? Does versioning of the bp and relay nodes matter. We have 1 relay that cant get to the latest version no matter what we do to upgrade. So we have bp and 1 relay on latest versiob and 1 relay on a previous version.

Thanks

1 Like

Is this solved?, we are having the same problem with 1.5M staked - No blocks for 4 epochs - Also some one else is having the same problem. check the forum topicsā€¦

1 Like

I do not believe this has been solved. I am now at 22 Epochs without being assigned as slot leader. I have checked every possible thing I can think of and followed all advice. The problem is that having to just wait and see if a block ever comes I think is unfair on my delegators and I wish there was just a simple check list that I could go to to make sure everything is set up right. My block allocations were reasonably regular then bang they just stopped altogether

You could downgrade to 1.33, Not sure how possible, it seems like ever since I upgraded itā€™s been zero slots. I have a feeling they are trying to bring the pool numbers down as they have yet to change the ā€˜Kā€™ parameter which would help small pools.

1 Like

Thanks - it would be pretty easy I think to downgrade to an earlier version but donā€™t really want to do that given all the changes that will have taken place on the network. It would be useful if someone working on the development of Cardano could provide some comment as right now I donā€™t know where else to look for information on this.

From my limited knowledge it should not matter that no new coins are being staked. I also do not think the version would matter as from my understanding the allocation of blocks is based purely on the registration status of the pool and the current live stake. All the other settings and configuration parameters of the pool should only come into effect once you go to mint the allocated slot.

You are right but I stopped following this thread when it appeared you had moved on to operational issuesā€¦ and as you said, everything about your pool seems to check out.

But since you never responded in this thread to the earlier reference to cncli and since youā€™re right about the now 22 dry epochs in a row being hugely unlikely, I have to make sure you are indeed running this command every epoch to determine if and when you are supposed to be the slot leaderā€¦ rather than either checking the log file or some UI based on the logs:

https://github.com/AndrewWestberg/cncli/blob/develop/USAGE.md#leaderlog-command

My apologies if you had already been checking this all along, since itā€™s what people generally assume by ā€œslot leaderā€ and Iā€™m just double-checking in your case. Everything I said about maths and statistical inevitabilities would only apply to Ouroboros and the leadership schedule it generates (i.e. the output of this cncli leaderlog command), while many things about your operational environment could affect whether your BP recognises itself as slot leader and then produces a block.

If you are indeed so uncommonly unlucky then cncli leaderlog would have been the only way to be sure, and if anything is amiss then running this command (or the cardano-cli equivalent) every epoch will be the only way to be sure going forward. :face_with_monocle:

1 Like

Thank you for responding to this and taking the time to help. I have been running the cardano-cli query for the past few Epochs and have seen no leaderslots - below is the query I run and the result it produces which is just an empty result

cardano-cli query leadership-schedule
ā€“mainnet
ā€“genesis /opt/cardano/cnode/files/shelley-genesis.json
ā€“stake-pool-id pool1fv9f8phzn7hp623ypw6ctf73a98hd7nrh8wm7glpcuhf64856g2
ā€“vrf-signing-key-file /opt/cardano/cnode/priv/pool/GNP1/vrf.skey
ā€“current

The output is below

SlotNo UTC Time


I have also run ./cncli.sh init and shows that no blocks have been minted. stolen missed etc for every Epoch since Epoch 320

I have also run the command ./cncli.sh leaderlog and I just get the result

~ CNCLI Leaderlog started ~
Node in sync, sleeping for 60s before running leaderlogs for current epoch
Leaderlogs already calculated for epoch 343, skipping!

In addition to this queries from within cnTools just shows no Blocks for each Epoch

1 Like

23 Epochs now without a slot leader assigned - there has to be something wrong - anyone have any ideas on what I can do? Is there a checklist of everything I could check with my BP node to see if something is wrong - this does not make sense anymore.

According to adapools.org your luck is still 92%ā€¦ Itā€™s not extremely low

But yeah if your leaderlogs donā€™t show anything, itā€™s just bad luck. :slight_smile:

Thanks for looking at this but it has been 23 Epochs in a row and this has been since I migrated my node to the cloud. I was regularly minting blocks with 26 minted over 18 months then after migrating the node there was complete silence - yes it is possible to be just poor luck but this is getting less likely with each passing Epoch and I need to be sure it is not an issue with the node migration.