Daedalus rankings are broken - let's figure out why

Staking & Delegation Stake Delegation
, ,
1

I want to start another discussion on pool rankings.

Following the delegation (https://hydra.iohk.io/build/3744897/download/1/delegation_design_spec.pdf) and rankings (https://hydra.iohk.io/build/4213276/download/1/pool-ranking.pdf) designs, there is no way to re-create the pool ranking we currently have in Daedalus.

The argument for wrongly implemented pool ranking system (i.e. not following the published design specifications) is empowered by the fact that multiple members have come to the same conclusions with independent calculations - including u/iiLap, u/Homersapiens and myself that I know of.

I believe I have pinpointed the issue with the implementation and it lies in the calculation of the non-myopic member reward (function getNonMyopicMemberRewards :: in the code). In the end, according to the specifications, the pool ranking uses exactly this value (non-myopic member reward) to rank the pools in the wallet, with top pools having the highest non-myopic member reward for a given amount of delegation, which is what is causing significant distortion to what we are expecting to see as rankings.

According to the specification, if a pool’s non-myopic member reward is less than the pool cost, it should be set to 0. However, this is not the case as I will exemplify, and I encourage everyone in here to confirm my (and u/iiLap’s and possibly u/HomerJ’s) calculations.

This issue is well illustrated in u/iiLap’s gist (https://gist.github.com/ilap/ad088d31e542f73685a3a245b3ad6c50#cardano-pool-ranking-in-daedalus), but I will just set one example here as well:

Instead of showing all calculations for a 1PCT pool, I will jump straight to its non-myopic member reward as that is what is finally used to rank the pools. The important thing to understand is that any 1PCT pool, based on it’s desirability (f(pledge, cost, margin)) is not in the top 150 pools by desirability, therefore, for its non-myopic member reward calculation, the system should only use its pledge (111k ADA).

When queried, the non-myopic member reward for a 1PCT pool should be 0.00 according to the formula posted at the end which reads “if a pool’s non-myopic member reward is less than pool cost, it is set to 0” [1].

This means that there should be absolutely no way for that pool to be near the top, because even assuming 300% apparent performance would only get it close to having the non-myopic member reward > cost, and that is by no means enough to get near the top rankings.

However, when the non-myopic member reward for 1PCT#3 is queried from daedalus-wallet, it comes back as 73.504539 (in ADA, by default it is returned in lovelaces), placing it at rank 11 in the current system state for an intended stake of 100k ADA.

I see two possible reasons for this:

  1. The non-myopic member reward formula is wrongly implemented, where the calculation for 1PCT pool (and all pools) is using it’s live stake rather than pledge for the calculation (i.e. the first order selection by desirability is not working).
  1. The non-myopic member reward formula is wrongly implemented, where it does not set the non-myopic member reward to 0 if non-myopic member reward < cost.

To confirm which of these is the case, I have calculated the non-myopic member reward myself in both the case of using just the pledge, and the case of using live stake for the calculation, then compared it with the actual non-myopic member reward value we get when quering the cardano-wallet for it. Pool in question is 1PCT#3, and the intended delegation is 100k ADA.

Firstly, considering the case of using only the pledge - 111k ADA (the way it should work), the non-myopic member reward is 82.92054846 according to the general rewards formula posted below [2]. This means it is less than the pool cost (340) and the non-myopic member reward should be at that point set to 0.

Secondly, considering the case of using the live stake - 50.61M ADA (the way it should not work), the non-myopic member reward is 37813.184, which is higher than the pool cost, hence the calculation continues according to the formula:

(37813.184 - cost) * (1 - 0.01) * 100000 / 50.61 * 1000000 -> 73.30261244

Now, to make the final conclusion on which path is being taken, we must consider what the system assigned for the non-myopic member reward to 1PCT#3 - which is 73.504539. This is a remarkably close number to the one calculated using its live stake, hence my conclusion is that option 1) is what is wrong with the system.

Daedalus rankings play if not the most important role in delegators decisions, yet this issue has been addressed in a nonchalant way.

I would like some questions answered:

  1. Why is there such a nonchalant approach to this matter? Is it not agreed that Daedalus rankings play a crucial role?
  2. Given what is presented here, can you explain the current pool ranking?
  3. The protocol is code. Why are there all these speculations on what apparent performance is causing - i.e. where are the tests that confirm correct behaviour?
  4. Why are there “opinions” on this matter? Some people seem to think that the only issue with the rankings is the saturation warnings being absent, while the specifications are correctly implemented. I believe most people think it is not correctly implemented, but there is only 1 right answer and it should not be difficult to prove which one it is.

Update:

my calculations for the case of 1PCT#3 being in the top150 were miscalculated. That number indeed matches that what’s in the wallet, hence 1PCT#3 is in the top150 according to desirability, and that can only be explained by hit rate estimates.

I think the way the protocol assigns hit rates is “punishing” small pools that don’t have a lot of blocks produced - so to compensate for the lack of data, it assigns the lower end of the estimates to those pools. That means the top pools are ranked the way they are because they produced a lot of blocks and small pools haven’t. This should then improve over time as slowly block production increases the total number of blocks produced for even the small pools. This is my understanding now. Or there is an issue with the estimations.

Please raise awareness of this issue so we can get to the bottom of it. Discuss here or at:

The formulas:

non-myopic member rewards formula [1]

NMformual

general rewards formula [2]

rewards

5 Likes
2

Excellent!

As the leader goes, so does the Cardano Nation. Forgive me Ezekiel, but so apropos.

Obviously, the problem isn’t rank and file maths comprehension. The problem is ill leadership.

You’re pulling on one of many exposed threads.

Stay with it👍

3

I think we agreed on how to confirm whether or not there is a bug, and that is:

  1. Calculate the non-myopic member rewards for 1PCT#3 assuming it is not in the top150 pools by desirability.

  2. Calculate the non-myopic member rewards for 1PCT#3 assuming it is in the top150 pools by desirability.

These are the only 2 cases possible for the non-myopic member rewards, so we can then:

  1. Compare both values to the value that cardano-wallet calculates for non-myopic member rewards (we get that value by directly querying the wallet)

From there, either one of the values we calculated will match or neither will match.

If neither value matches, there is a bug as the code is calculating the reward using some formula/values that are not in the specifications. In that case, we can:

  1. Calculate the non-myopic member rewards for 1PCT#3 using its live stake for the non-myopic member rewards calculation.

If that value matches the one from the wallet, there is strong indication that live stake is being used for the calculation, which is not intended behaviour. My calculations show this is the case, but now it’s time for others to do them to prove me right or wrong.

On the other side, if one of the values from 1) and 2) matches the one from the wallet, then:

a) the non-myopic member rewards formula is either correctly implemented, but hit rate is causing the disparity with my calculations and 1PCT#3 is actually ending up in the top150 pools by desirability (unlikely in my opinion as somehow the hit rate among all 1PCT pools is very high relative to other pools in that case)

b) the non-myopic member rewards formula is either correctly implemented, but something else is wrong and messing with the rankings

Both a) and b) seem unlikely, but let’s see. I invite everyone to run the calculations and report.

Parameters:
pledge = 111000 ADA
cost = 340 ADA
margin = 0.01
hit rate is not important, as you should calculate for both the case of 1PCT#3 in top150 and not in top150.

4

Update:

my calculations for the case of 1PCT#3 being in the top150 were miscalculated. That number indeed matches that what’s in the wallet, hence 1PCT#3 is in the top150 according to desirability, and that can only be explained by hit rate estimates.

I think the way the protocol assigns hit rates is “punishing” small pools that don’t have a lot of blocks produced - so to compensate for the lack of data, it assigns the lower end of the estimates to those pools. That means the top pools are ranked the way they are because they produced a lot of blocks and small pools haven’t. This should then improve over time as slowly block production increases the total number of blocks produced for even the small pools. This is my understanding now. Or there is an issue with the estimations.

5

Thanks for your efforts. But I have a simple question, how does a pool get into group 150, what are the optimal parameters and how many blocks need to be made to earn my desirability rating?

6

Section 5.6.1 describes the calculation for the top k desirable pools. It’s based on the the pool rewards formula for a fully saturated pool (taking cost, pledge and margin into account) and multiplying the result with the assumed pool performance which is basically number of produced blocks divided by number of expected blocks.

7

I addition to the (potential) shortcoming of incorrect rankings in Daedalus there are other vital features missing that were defined in the specs.

grafik

Daedalus should present the user the average apparent performance of each pool, as well as other figures like the non-myopic member rewards.
The wallet should also warn the user in case the current delegation becomes less favorable:
Section 4.3
grafik

8

Reading https://hydra.iohk.io/build/4213276/download/1/pool-ranking.pdf I’m not really sure why they introduced this new mechanism. I don’t know which approach is more fair, but the orignial one is definietly more simple and elegant.

In the design spec pool performance is defined like this:

Blockquote
In order to decrease the influence of these fluctuations, and to get a long-term picture, whenevaluating the desirability and non-myopic rewards, one should insert the apparent performanceaveraged over several epochs asp, not the value ofpfrom just the last epoch.Averaging can be done efficiently via an exponential moving average, or by using a movingwindow and keeping track of the apparent performance during the last couple of epochs.

and

grafik

with
n = amount of blocks the pool added successfully to the chain this epoch,
N_ = Total blocks added to the chain this epoch
sigma_a = pool’s relative active stake (which is basically the pool’s chance to win a slot)

9

The apparent performance is now substituted with hit rate estimate:

10

I had the same thought, they say hit rate deals better with variance than just exponential moving average or simple moving average of apparent performance.

11

I found this document only by chance and as far as I can tell it’s not linked on the docs.cardano page. Is there another source to get the latest papers and specs that IOG releases?

13

Here’s an observation of my ranking within Daedalus. Within the past 7 days I updated my margin and pledge amounts on my new pool that has been running for over a month. Prior to updating my margin and pledge my rank was low 200s . Within a few hours after I updated my margin and pledge my rank dropped to mid 900s. I believe the Daedalus rank may be basing their rank on some historical/longevity basis that is incorrectly associating an update to pool details as a newly created pool (thus lowering the rank). Are others pools that update their margins or pledges also seeing a drop in rank? Just my observations.

As always mates,
Cheers and beers!

14

Hi,
the thing with ranks past ~226 is that all those pools have non-myopic member reward = 0. Daedalus ranks them randomly on each machine since they get the same “score”.

15

Wow, great work!

16

Take a look at this pull request:

Seems like Jared fixed the issue, at least as far as the implementations of the formulas go. The rest of the updates I cannot judge due to my lacking Haskell knowledge. But please take a look as well @Dostrelith @_ilap

Edit: They also changed the definition of sigma_nm in the design specs based on our suggestions. It is now defined as max(Sigma + t , z0). Feeling a little proud that we were able to contribute a tiny bit to this amazing project =)

4 Likes
17

Yep, it’s great to see how discussions can lead to improvements - even though I was wrong in my calculations, we managed to get a better understanding of the hit rate estimates and ranking in general, as well as point out an inconsistency between the code and design specs!

Great work

5 Likes