The problem I see with the cardano-cli query leadership-schedule command is that you can’t see the internal workings unless you are a Haskell developer (and a brilliant one at that). It is a black box to you because you supply your key data and it tells you that you got zero leadership slots. Hence your questions about how you can be sure that there isn’t some bug causing unfair bias against your pool.
I think it would be useful if you could actually see the output of the “leadership Verifiable Random Function” and then you could compare this to your relative stake value. It would give you a much better feel for how it all works. It would be good if IOG provided a tool, or a simple tutorial on how to write some code, that can get the actual output from the “verifiable random function” used in the leadership calculation.
I have been thinking about this verifiable random function myself recently. I am currently writing a CIP seeking to address the problem that geographically decentralised pools are unfairly treated because they suffer from greater network delays. As part of this process, I needed to model the effect of various changes and so I have started writing a small program (in Haskell). This process is giving me further insight. However, my program does not use the proper “verifiable random function” but rather simulates it’s output through using pseudorandom data. And, the other BIG problem is that I am only a beginner programmer.
Hello, I am experiencing the same issue having “incredibly bad luck”.
dtDn had 130-219k ADA staked from epoch 364 until 381 then 1.68M ADA staked this current epoch 382.
The last validated block occurred on epoch 365, at the time it had a lot of luck because it validated a block in two consecutive epoch (though the one on epoch 365 was not in the leadership schedule and therefore I suspect that it took a block from someone who didn’t update to 1.35, so maybe not so much down to luck)
Anyway, since then the pool hadn’t one single slot assigned, that means that with a 10-15% chance of validating a block per epoch between epoch 367 and 381 included, the probability of having no block assigned is about, considering 11% chance to get a slot per epoch on average which is conservative : 0.89 ^ 15 ~= 17.4% …
BUT I issued a 1.5M ADA bond through Optim Finance and the staking rights are active since the current epoch (382) and dtDn has still no slot assigned with a 75% chance of having at least one block assigned for this epoch that means the probability of having no block assigned since epoch 367 is down to 4.4%. Tomorrow dtDn will generate the leadership schedule for epoch 383 and if there are still no slot assigned the probability for having no slot assigned between epoch 367 and 383 included will be down to 1.1%.
I checked the verification secret key file so many times, the file didn’t change since before epoch 364 when the leadership schedule correctly returned the assigned slot. I don’t see where I could be making a mistake really the leadership schedule command is kind of straightforward (not running out of memory btw).
The leadership schedule calculations are deterministic. If you are doing it correctly there is no way that you can mint a block for a slot which wasn’t predicted by your leadership schedule.
Are you sure you don’t have multiple different copies of executables or keys lying around where you could be running the command with erroneous data?
I am pretty sure yes (I’ll check again but I did check so many times already)
Isn’t it possible to validate a block that is not in your leadership schedule if another pool failed to validate the block?
No, that’s not possible. Only pools/block producers assigned a block can mint a block. If you minted a block that was not predicted by your leadership schedule then your calculation of leadership schedule is wrong.
Here comes a rant not directed at anyone in particular:
Most cardano-node operator problems seem to be caused by the following:
No standard filesystem location used for installation of executables.
No standard filesystem location used for installation of libraries.
Use of modified $PATH environment which is different for different users on the system.
Use of, and dependence on, custom environment variables.
Use of automated scripts for updating the cardano software which are massive and only understood by the original author of the script. Most people seem to just blindly trust the script.
Mix all of the above together and combine with several software update cycles +/- some changes to the massive automated script which often automates its own upgrade changes.
I really don’t understand why most Cardano people don’t like to use their linux package manager.
I just got the result of the leadership schedule for next epoch (384) and dtDn has 5 slots assigned (probability for this to happen with current stake is 1%)
So I won’t make any conclusion but for information the numbers are
Between epoch 367 to 382 included, dtDn had a stake around 130-219k ADA and didn’t validate any block, the probability for this event to happen is (made the calculation with more precision) : 10%
Then no block in epoch 383 (1.68M ADA staked) with a 25% probability for this to happen.
And now for 384, 5 slots assigned (same stake as for 383) with a probability for this to happen of 1%.
I have no idea why the leadership schedule didn’t return the slot dtDn validated on epoch 365 but really since I updated the node on version 1.35.4, nothing changed much.
For info my system is as close as it can be to the standard installation described on developers.cardano.org and except for monitoring I do basically everything by hand to be 100% sure to master every aspect of the configuration.
If you are then there is a lot of environment variables and functions defined in those scripts and many seem to inherit stuff from each other. I don’t know how others can get their head around those mammoth scripts. I can’t.
Does the user that is running the cardano-node process have different environment variables or a different $PATH setting compared to the user you are using to output the leadership logs? You need to figure out how the different processes could be somehow using different data?
If you are minting blocks that are not shown in your leaderlog output then you really need to figure out why. One of the scripts you are using is accessing different information compared to what your actual cardano-node service is using.
By “standard” I meant the compilation of sources, key generation and run using a daemon as described on developers.cardano.org
I am not using any script to manage the node and relays, only some scripts I wrote for monitoring (that should not impact the node).
I do have hard coded paths in the script that calculates the leadership schedule so it might come from there, I’ll investigate. Thanks for the tip. (and also thanks for letting me know that it is abnormal to have blocks produced that aren’t shown in the leadership schedule)
About the main subject of this thread I don’t have much more to say so I’ll…
Happy holiday season everyone - just for the record I am now at 62 Epochs and only 3 slots over this time - really don’t feel that I am ever going to be allocated slots anymore
65 now and only 3 slots - ok I am going to change my vrf keys as there is definitely a bias here - we need to get more interest in this for the sake of small pools
They did but answered the wrong support ticket still waiting for mine but they have assured me that they will attend to it. By the way I do have new vrf keys so if anyone can do analysis on the old ones they are welcome to as I would really like to know why I am getting no slots.
Ok this was the response which is ridiculous. It is now close to 70 Epochs with only 3 slots allocated and now 22 in a row again with no slots allocated. My expected number per Epoch has been around 0.25 and without going into binomial theory etc you would expect a block every 4 to 5 Epochs. Yes of course this can vary widely but over time should approach this. I ask myself why am I doing this when things don’t even work as they should?
Dear Russell,
Thank you for contacting the IOHK Technical Service Desk. We look forward to helping you with your request.
Please let me tell you from the conclusion first.
We could not find any operational issues on your pool.
Yes, I do understand that you’ve been working as a Stake pool operator for 2.5 years.
But the Cardano network and general environment have changed significantly over the past year.
The number of Stake pools is increased so much since you register your stake pool.
At the moment, the stake pool should have more than 2M ADA to be constantly chosen as slot leaders.
Currently, K-parameter is set as 500 so most of the delegators go to go the large stake pool but in the future, there is a chance that we change this k-parameter to 1,000.
If that happens your pool increases the chance to be chosen as a slot leader.
So there are 2 ways for your pool to be chosen as slot leader.
Increase your delegator.
If there is difficulty to increase the delegator, you will need to wait for the new K-parameter. (But we don’t know when this happens at the moment)
I have changed my vrf keys and if anyone with better technical knowledge than I have would like to test my old keys to see if there is any kind of bias I am happy to send them to you as I doubt that my small pool size will prompt any action from IOG.
I finally got a slot allocated and minted a block today so I will mark this thread as solved. I still think there is some bias towards small pools based on the frequency of slot selection but in the absence of any evidence there is not much more I can say.
The good thing however is that I know the pool is configured right and there was no issue with my pool, the infrastructure, or the network as the block minted flawlessly like the 29 before it. Thanks everyone for your assistance with this.
still waiting for mine but they have assured me that they will attend to it. By the way I do have new vrf keys so if anyone can do analysis on the old ones they are welcome to as I would really like to know why I am getting no slots.