Does cncli database store correct leader_vrf values?

I noticed a couple of slot battles shown on pooltool recently and was trying to see the leader vrf values to confirm how they were resolved. I pulled the data from my cncli database and found that the leader_vrf_0 values were not always less for the winner. But, I thought that the winner should have the lower leader vrf value according to this reference:

Here is a query that lists the 5 most recent slot battles where the winner had the higher leader_vrf_0 result stored in the cncli database (when the winner should have the lower value):

sqlite3 /var/lib/cardano/cncli.db 'select winner.block_number, winner.slot_number, winner.pool_id, winner.hash, winner.leader_vrf_0, loser.leader_vrf_0 from chain winner inner join chain loser on winner.slot_number = loser.slot_number and winner.block_number = loser.block_number and winner.orphaned=0 and loser.orphaned=1 and winner.leader_vrf_0 > loser.leader_vrf_0 order by winner.block_number desc limit 5;'

BlockNo | SlotNo | Winner PoolID | Winner block hash | Winner Leader VRF | Loser Leader VRF

7814905|72793921|ac5ba151fae6889e25021acc723b0e34655828bf3328b6f1180e1d65|fd15a900f46e8bf9c03ad987da1d99e9102dea99ae1e0423113ba5c7ad1f903b|0003b0ad7390b0da547e3e43b1adae072c709d5bd06d4962ef992ac42221deb3|00016073b18d7c4f9ae4174085401da8426eae37646a29c0027b0ba57ee8f456
7814595|72787725|ee98a72dfb3ba43e9de2e39ef825a69d5d45627d2e918ea3e260f2dd|7b02eb92d0f5ac4e820d7535f18825d242086336413a6c51e30a1c9d581594d2|0004106510affa94afddb43845a59dc6041b3887ce563e7f1114018f75329072|0000d4662f2c7d626857c552b6063b2c70f891b2d602412a6a15baa2fd200a43
7814482|72785510|b619d366a0acc2e5a8e0f8e373d6eb926a01811f05a381aaa7a8564c|99656c88d38916a8ce932d53a5a06db8ebbafb0ff1a37112ea86c248a03539c1|000703cbe5d02357b7dcd72b6319679ac58089eb91a8ff9a961e3050d41ec0fb|00024b9090d39771158f8426e876f687053c0efe2fe4f8335b34f656006183b5
7814446|72784819|971212db48d9390662faa0fa725950f3b9ac1a898227f5fcf1096399|c97725be2f3854730037af89ace0f6a4e1dabedf70040b56842b87795c26f2e3|00039b61db164c09c820bf86194d5b8e58f2c5400f6f6ec8f017ec397389a343|0003307de189e839e1a5ee43dd37553f505383f3b1407071b316a8ac63426c90
7814351|72782882|ed40b0a319f639a70b1e2a4de00f112c4f7b7d4849f0abd25c4336a4|dc67f5747cfbe163ef6efd7a29e5e49296504757372bdb8fa7575e65eb79ae26|0008cdbb377d1a589908b8a763f95a3878ccc916110dd5575fcdb89094d13698|00003e02943eba2de8de080a022878350b26d1a0bff472c6b370496519b9d722

If you take the top example. The leader_vrf_0 value for the winner was 0003b…, but the loser had a lower value of 00016…

Pooltool shows this battle and confirms the winner block has hash fd15… which is the same as my query confirms was the winner : Cardano PoolTool - The most comprehensive staking statistics for Cardano on the web.

According to the reference, the lower leader vrf value should win the slot battle.

The format of the chain table in the cncli database is:

sqlite3 /var/lib/cardano/cncli.db '.schema chain'

CREATE TABLE chain (
id INTEGER PRIMARY KEY AUTOINCREMENT,
block_number INTEGER NOT NULL,
slot_number INTEGER NOT NULL,
hash TEXT NOT NULL,
prev_hash TEXT NOT NULL,
eta_v TEXT NOT NULL,
node_vkey TEXT NOT NULL,
node_vrf_vkey TEXT NOT NULL,
eta_vrf_0 TEXT NOT NULL,
eta_vrf_1 TEXT NOT NULL,
leader_vrf_0 TEXT NOT NULL,
leader_vrf_1 TEXT NOT NULL,
block_size INTEGER NOT NULL,
block_body_hash TEXT NOT NULL,
pool_opcert TEXT NOT NULL,
unknown_0 INTEGER NOT NULL,
unknown_1 INTEGER NOT NULL,
unknown_2 TEXT NOT NULL,
protocol_major_version INTEGER NOT NULL,
protocol_minor_version INTEGER NOT NULL,
orphaned INTEGER NOT NULL DEFAULT 0 ,
pool_id TEXT NOT NULL DEFAULT ‘’,
block_vrf_0 TEXT NOT NULL DEFAULT ‘’,
block_vrf_1 TEXT NOT NULL DEFAULT ‘’
);
CREATE INDEX idx_chain_slot_number ON chain(slot_number);
CREATE INDEX idx_chain_orphaned ON chain(orphaned);
CREATE INDEX idx_chain_hash ON chain(hash);
CREATE INDEX idx_chain_block_number ON chain(block_number);
CREATE INDEX idx_chain_node_vkey ON chain(node_vkey);
CREATE INDEX idx_chain_pool_id ON chain(pool_id);

Are the leader_vrf_0 values that cncli database stores not the proper leader vrf values?

The leader_vrf_1 values are empty and I don’t know what eta_vrf_0 or eta_vrf_1 values represent.

Can anyone explain this?

Is there some other way to look up what the leader vrf values were for each block in a slot battle? The system logs don’t show these values and nor does pooltool when it shows the slot battle winner.

2 Likes

Interesting question!

Tried to understand it from https://github.com/cardano-community/cncli/blob/develop/src/nodeclient/sync.rs, but couldn’t.

Do block_vrf_0 or block_vrf_1 make more sense?

1 Like

Indeed it is interesting!

I just did some more investigating and this is starting to look like a change (or bug) introduced more recently.

Here is what I have found so far:

I did a query to retrieve all “slot battles” where the winner did not have the lower leader vrf value (cncli database leader_vrf_0). I noticed that these were very infrequent before the last few epochs (when everyone started running node version 1.35.3).

So I started at the earliest examples and looked through my node logs. The earliest such example was 2022, July 09. As I worked forwards in time, I found that in each case the “slot battle” turned out to be a “height battle”. What happened in each case that I checked so far is that my node did initially prefer the block that was ultimately orphaned switching tip to it (because it had a lower leader vrf). But, it subsequently received another block that forced it to switch tip again shortly after.

So it appears that what actually happened in these early cases (before node version 1.35.3) is the following:

  • Node A and Node B both produce blocks for same slot
  • Node A initially wins the slot battle (because it has a lower leader vrf) and nodes that received both blocks switch tip to node A’s block
  • Node C produces a block on top of node B’s losing block (presumably because it didn’t see node A’s block yet)
  • Now the combination of blocks created by node B and node C are the longest chain
  • All other nodes upon seeing node C’s block then switch their tips to this longer fork

So the above is exactly how I thought things should work. Slot battles are resolved by lowest leader vrf and height battles are resolved by longest chain. Sometimes a slot battle can later become a height battle if the next producer didn’t see the other block prior to making its block. This is exactly what Duncan Coutts outlined as the current state at: Consensus should favor expected slot height to ward off delay attack · Issue #2913 · input-output-hk/ouroboros-network · GitHub

Here is an example of one such slot battle that subsequently became a height battle.

Jul 11 18:35:04 relay1 cardano-node[962057]: [relay1:cardano.node.ChainDB:Info:143] [2022-07-11 08:35:04.87 UTC] Valid candidate ea82dc57bda01a664b6aa8591d788458b885baef2c685b9ca9fb5d9495e0acdd at slot 65962213
Jul 11 18:35:04 relay1 cardano-node[962057]: [relay1:cardano.node.ChainDB:Notice:143] [2022-07-11 08:35:04.87 UTC] Chain extended, new tip: ea82dc57bda01a664b6aa8591d788458b885baef2c685b9ca9fb5d9495e0acdd at slot 65962213
Jul 11 18:35:05 relay1 cardano-node[962057]: [relay1:cardano.node.ChainDB:Info:143] [2022-07-11 08:35:05.26 UTC] Block fits onto some fork: 90a921ca2c93cb69cf52fe5d2cc532df7990d38a344e3db3daf22457dba6a370 at slot 65962213
Jul 11 18:35:05 relay1 cardano-node[962057]: [relay1:cardano.node.BlockFetchDecision:Info:153] [2022-07-11 08:35:05.27 UTC] before next, messages elided = 38
Jul 11 18:35:05 relay1 cardano-node[962057]: [relay1:cardano.node.BlockFetchDecision:Info:153] [2022-07-11 08:35:05.27 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 192.168.27.9:2700}) (Right [At (Block {blockPointSlot = SlotNo 65962213, blockPointHash = 90a921ca2c93cb69cf52fe5d2cc532df7990d38a344e3db>
Jul 11 18:35:05 relay1 cardano-node[962057]: [relay1:cardano.node.ChainDB:Notice:143] [2022-07-11 08:35:05.29 UTC] Switched to a fork, new tip: 90a921ca2c93cb69cf52fe5d2cc532df7990d38a344e3db3daf22457dba6a370 at slot 65962213
Jul 11 18:35:05 relay1 cardano-node[962057]: [relay1:cardano.node.BlockFetchDecision:Info:153] [2022-07-11 08:35:05.81 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 88.99.143.230:3001}) (Right [At (Block {blockPointSlot = SlotNo 65962214, blockPointHash = 36fc6ae41f912253084168b1582fb6beb97a8e8ce2fa05>
Jul 11 18:35:05 relay1 cardano-node[962057]: [relay1:cardano.node.BlockFetchDecision:Info:153] [2022-07-11 08:35:05.86 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 52.59.248.215:4321}) (Right [At (Block {blockPointSlot = SlotNo 65962214, blockPointHash = 36fc6ae41f912253084168b1582fb6beb97a8e8ce2fa05>
Jul 11 18:35:05 relay1 cardano-node[962057]: [relay1:cardano.node.BlockFetchDecision:Info:153] [2022-07-11 08:35:05.87 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 94.136.30.220:3001}) (Left (FetchDeclineConcurrencyLimit FetchModeDeadline 2)),TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:>
Jul 11 18:35:06 relay1 cardano-node[962057]: [relay1:cardano.node.ChainDB:Info:143] [2022-07-11 08:35:06.08 UTC] Block fits onto some fork: 36fc6ae41f912253084168b1582fb6beb97a8e8ce2fa05834159c9f865dfe30b at slot 65962214
Jul 11 18:35:06 relay1 cardano-node[962057]: [relay1:cardano.node.BlockFetchDecision:Info:153] [2022-07-11 08:35:06.08 UTC] before next, messages elided = 11
Jul 11 18:35:06 relay1 cardano-node[962057]: [relay1:cardano.node.BlockFetchDecision:Info:153] [2022-07-11 08:35:06.08 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 54.148.179.11:3001}) (Left (FetchDeclineConcurrencyLimit FetchModeDeadline 2)),TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:>
Jul 11 18:35:06 relay1 cardano-node[962057]: [relay1:cardano.node.ChainDB:Notice:143] [2022-07-11 08:35:06.09 UTC] Switched to a fork, new tip: 36fc6ae41f912253084168b1582fb6beb97a8e8ce2fa05834159c9f865dfe30b at slot 65962214

Notice how my node did prefer the ultimately orphaned block initially and switched its tip to it, but then received a subsequent block which made it switch again.

I have reviewed several of the earliest examples retrieved by my query and they all fit this pattern. According to the Duncan Coutts reference above, this is the correct action.

But, this is not how things appear to be working now.

Starting from the most recent blocks retrieved by my query and reviewing logs at these times shows that the node seems to prefer the later received block. Here is an example:

Sep 29 04:50:57 relay1 cardano-node[972972]: [relay1:cardano.node.ChainDB:Notice:178] [2022-09-28 18:50:57.48 UTC] Chain extended, new tip: 552a6898460ccaa343437c8aad7ba9648c2f57bee06444dd8dbc6a1abe71cb01 at slot 72824766
Sep 29 04:50:57 relay1 cardano-node[972972]: [relay1:cardano.node.ChainDB:Info:178] [2022-09-28 18:50:57.87 UTC] Block fits onto some fork: f1bbac626bc5d6ad88fdf86a84d25d299cdb337a2bb9c8a3c2277bad244fc4d9 at slot 72824766
Sep 29 04:50:57 relay1 cardano-node[972972]: [relay1:cardano.node.BlockFetchDecision:Info:188] [2022-09-28 18:50:57.88 UTC] before next, messages elided = 16
Sep 29 04:50:57 relay1 cardano-node[972972]: [relay1:cardano.node.BlockFetchDecision:Info:188] [2022-09-28 18:50:57.88 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 35.247.85.163:3001}) (Left (FetchDeclineConcurrencyLimit FetchModeDeadline 2)),TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:>
Sep 29 04:50:57 relay1 cardano-node[972972]: [relay1:cardano.node.ChainDB:Notice:178] [2022-09-28 18:50:57.91 UTC] Switched to a fork, new tip: f1bbac626bc5d6ad88fdf86a84d25d299cdb337a2bb9c8a3c2277bad244fc4d9 at slot 72824766

See how my node switched tip to the later received block. But this later received block did not have the lower leader_vrf_0 in the cncli database. I don’t know what made the node switch its tip to this later block. Maybe just because it was received later??? The logs following this just continue on with blocks built on top as usual.

So I continued reviewing the more recent slot battles and in every case I noticed that the later received block was preferred irrespective of the leader_vrf_0 value in cncli database. For example:

Sep 29 06:57:19 relay1 cardano-node[972972]: [relay1:cardano.node.ChainDB:Notice:178] [2022-09-28 20:57:19.54 UTC] Chain extended, new tip: 5396e4543b3ee688bcaced6f515a4e54f1b22a5fd1f612be88625ab7b491040d at slot 72832348
Sep 29 06:57:19 relay1 cardano-node[972972]: [relay1:cardano.node.ChainDB:Info:178] [2022-09-28 20:57:19.55 UTC] Block fits onto some fork: c6651e4413b482d82dcc50f24eaeec8fdd1792938d6fe2bb2c43e8ec970f7cfc at slot 72832348
Sep 29 06:57:19 relay1 cardano-node[972972]: [relay1:cardano.node.ChainDB:Notice:178] [2022-09-28 20:57:19.55 UTC] Switched to a fork, new tip: c6651e4413b482d82dcc50f24eaeec8fdd1792938d6fe2bb2c43e8ec970f7cfc at slot 72832348

In that case the later block did have a lower leader_vrf_0 score.

But another example:

Sep 29 08:58:08 relay1 cardano-node[972972]: [relay1:cardano.node.ChainDB:Info:178] [2022-09-28 22:58:08.54 UTC] Valid candidate 4089eac285b5a0170974f1d2aced7f1a4035e6a329d7569f1ef1c36068d08b5c at slot 72839597
Sep 29 08:58:08 relay1 cardano-node[972972]: [relay1:cardano.node.ChainDB:Notice:178] [2022-09-28 22:58:08.54 UTC] Chain extended, new tip: 4089eac285b5a0170974f1d2aced7f1a4035e6a329d7569f1ef1c36068d08b5c at slot 72839597
Sep 29 08:58:08 relay1 cardano-node[972972]: [relay1:cardano.node.ChainDB:Info:178] [2022-09-28 22:58:08.60 UTC] Block fits onto some fork: 1643b6b22508648d8170af937548d1edf645f450dda88ad32e8472119cf1d306 at slot 72839597
Sep 29 08:58:08 relay1 cardano-node[972972]: [relay1:cardano.node.ChainDB:Notice:178] [2022-09-28 22:58:08.65 UTC] Switched to a fork, new tip: 1643b6b22508648d8170af937548d1edf645f450dda88ad32e8472119cf1d306 at slot 72839597

In that case the later block had a higher leader_vrf_0 score in cncli database.

Why did my node immediately prefer this later received block for the same slot when its leader_vrf_0 score was higher?

One thing that might be going on here is that the cncli database value stored as leader_vrf_0 might be incorrect. I notice that the length of this value has changed to be half the size more recently.

If you do this query to list all slot battles where the block with the higher leader_vrf_0 value won:

sqlite3 /var/lib/cardano/cncli.db 'select winner.block_number, winner.slot_number, winner.pool_id, winner.hash, winner.leader_vrf_0, loser.leader_vrf_0 from chain winner inner join chain loser on winner.slot_number = loser.slot_number and winner.block_number = loser.block_number and winner.orphaned=0 and loser.orphaned=1 and winner.leader_vrf_0 > loser.leader_vrf_0 order by winner.block_number desc;'

You will see that the top (latest) results look like:

7817452|72845687|e645c034522f32522bf1c64a7f46128c562880abc26abe5f65f25a84|25f0f2262cd6d4fc6afdcbcb20643028745bc5035ab3897dc75f5d19e40039e6|0005e6d2b1c42a5bcc81758673ea3c6837a9369d1fc4af8191064c3685accd55|0000d7fc65939da69a675516637da68eca480f4d8df8128dc97282364ea4b059

But the bottom (earliest) results look like:

7473625|65745540|43bf2e2c7601bd6dfb38564fc2c48f5b1a773fdb4095dee412f52e50|1c42cd7e8d7bb007828f7480844185b0c20998f9c5eb8ba9b87b2a8d20a5c7c9|0002efabf7f6a5f5fb85cf410817e33a6ee09dcf6fc0d7b806dea6f9b07505cabb0d42462c2bff7bfa76831dfd9872f606e3534fc8fd1736c07466be7b469fb3|00014cc9956e4511fc7aba887c157c617e74b36e69a237507bf768d9d5c66c58cae30ca9d9b009978a714b7a830038f980016829eb78a916bc9818885a8d17ff

Note how the leader_vrf_0 values are much larger for the bottom (earliest) results.

Maybe cncli is now storing incorrect leader_vrf_0 values??? Or maybe the cardano-node is no longer considering the leader vrf value when deciding on slot battles???

I am not sure how to investigate this any further.

Can someone check my logic and do a bit of investigating on their machines because I don’t want to start making claims that there is a bug if I am just hopelessly confused.

I do recall IOHK saying something about being able to remove an extra vrf calculation in each block for an efficiency gain. Could it be something related to this?

Is there some way to pull the leader vrf values for blocks directly from the chain???

1 Like

I think it is a bug with cncli since the length of the hex string leader_vrf_0 values in the chain table has changed recently.

I submitted a bug with cncli: Incorrect leader_vrf_0 values in chain table? · Issue #19 · cardano-community/cncli · GitHub

1 Like

The lowest VRF value rule only applies on chains of the same length, if another pool already built on top of the block with the higher VRF value (because he didn’t receive the one with the lower value in time), that chain is already longer and will be used by the next slot leader (if he received all the blocks in time).

Normally this situation will only happen if the next block slot is very close to the one(s) with the slot or height battle and the propagation of the one with the lowest VRF value took longer to reach that BP.

I don’t know if this is the reason for what you saw, because that situation won’t happen that often with current propagation times I think, so I might be something else.

1 Like

It is looking like the cncli database leader_vrf_0 values are not the correct leader vrf values used by nodes to decide slot battles.

I looked at logs in two different relays on opposite sides of the world for the last couple of slot battles. The blocks were received only a few milliseconds apart by both relays but in different orders. In both cases the same block was preferred immediately. Presumably the nodes compared leader vrf scores to determine the winner. This is expected behaviour. However the cncli database leader_vrf_0 values were actually lower for the losing blocks.

I have reviewed logs of several similar cases too. Note that the slot battles I am considering are simple slot battles, not ones where a subsequent block arrived quickly which could have been built on one fork before seeing the other, which would cause the node to decide based on longest chain rule.

So what we can say is that the leader_vrf_0 values in cncli database do not correspond with the “leader vrf” values the nodes are using to determine slot battles.

Therefore either cncli is wrong or cardano-node is wrong. Querying the cncli database shows that the correlation between slot winner and lower leader_vrf_0 values is 50%. In other words, the cncli leader_vrf_0 values are not correlated with the slot battle winner.

Note that the cncli leader_vrf_0 values were the correct values prior to the Vasil upgrade.

@AndrewWestberg said he was going to look into it next week: Incorrect leader_vrf_0 values in chain table? · Issue #19 · cardano-community/cncli · GitHub

For those interested he also recorded this excellent video in June: Cardano: NerdOut - Babbage LeaderLogs - YouTube

1 Like

It turns out that the answer to my original question: “Does cncli database store correct leader_vrf values?” is YES. @AndrewWestberg 's cncli tool is storing the correct values.

Instead, there is a bug in cardano-node which has now been confirmed.

Slot battles are currently being determined based on the “block vrf” value rather than the “leader vrf” value. In essence this means that there is a 50% chance to win a slot battle now, whereas before smaller pools were much more likely to win.

Small pool operators should campaign for this bug to be fixed soon as they have now lost their slot battle advantage.

2 Likes