Orphaned block, three way Height Battle

natefilm · 29 October 2022 19:55

Hi, I’m the operator of BIKES and was wondering if there was anything I could’ve done to prevent this or was this the luck of the draw?

Could it be the performance of my nodes that could’ve made this happen? Seems likely if my pool had the lower Block and Leader VRF value?

When digging into BP logs I get (Block fits onto some fork: “” at slot 75460095), which was the slot North minted the block at.

Ran:

curl -s localhost:12798/metrics | grep cardano_node_metrics_Forge

Received:

cardano_node_metrics_Forge_adopted_int 1
cardano_node_metrics_Forge_forged_int 1
cardano_node_metrics_Forge_forge_about_to_lead_int 513083
cardano_node_metrics_Forge_node_is_leader_int 1
cardano_node_metrics_Forge_node_not_leader_int 513082

One idea was time sync, all nodes are running chrony.

Or is it the new parameter changes with VRF values?

github.com/input-output-hk/ouroboros-network

Leader VRF value no longer settling ties

opened 10:56PM - 30 Sep 22 UTC

JaredCorduan

I do not know whether or not we've found a bug or just discovered the consequenc…es of an intentional decision. --- The final tie breaker for "slot battles" is the leader VRF value: https://github.com/input-output-hk/ouroboros-network/blob/9249a70ed9e2365f3963e47cb31b4b1589bca8f6/ouroboros-consensus-protocol/src/Ouroboros/Consensus/Protocol/Praos/Common.hs#L62-L68 In the `TPraos` protocol (used prior to the Vasil HF), `csvLeaderVRF` was the leader VRF value. In the `Praos` protocol, however, `csvLeaderVRF` is being set to the single VRF value in the block header (prior to the range extension). **This removes a small advantage that small pools previously enjoyed.** Small pools are more likely to win this tie breaker, since by being a small pool they need a smaller leader VRF value in order to win the leader check. Using the the VRF value before the range extension is applied removes this small advantage. --- The Evidence: The view, `PraosChainSelectView`, is populated by the `BlockSupportsProtocol` class method `selectView`, which uses the `ProtocolHeaderSupportsProtocol` class method `pHeaderVRFValue` to set `csvLeaderVRF` in the view. * In `TPraos`, `pHeaderVRFValue` uses the leader VRF value: https://github.com/input-output-hk/ouroboros-network/blob/9249a70ed9e2365f3963e47cb31b4b1589bca8f6/ouroboros-consensus-shelley/src/Ouroboros/Consensus/Shelley/Protocol/TPraos.hs#L111 * In `Praos`, `pHeaderVRFValue` uses the raw header VRF value: https://github.com/input-output-hk/ouroboros-network/blob/9249a70ed9e2365f3963e47cb31b4b1589bca8f6/ouroboros-consensus-shelley/src/Ouroboros/Consensus/Shelley/Protocol/Praos.hs#L141 --- This was discover here: https://github.com/cardano-community/cncli/issues/19

Stumped on this if anyone has any insight would be greatly appreciated, thanks.

Terminada · 29 October 2022 22:09

Your clock is out of sync.

Here is what I have in my logs:

Oct 29 16:52:56 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Info:207] [2022-10-29 06:52:56.87 UTC] Valid candidate eebf634dca68962f78bc71b9978a6a868365097916d89fc155244ae7c05c41d1 at slot 75460085
Oct 29 16:52:56 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Notice:207] [2022-10-29 06:52:56.87 UTC] Chain extended, new tip: eebf634dca68962f78bc71b9978a6a868365097916d89fc155244ae7c05c41d1 at slot 75460085
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.BlockFetchDecision:Info:217] [2022-10-29 06:53:03.34 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 13.59.88.248:3001}) (Right [At>
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.BlockFetchDecision:Info:217] [2022-10-29 06:53:03.35 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 52.14.17.13:6000}) (Right [At >
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.BlockFetchDecision:Info:217] [2022-10-29 06:53:03.58 UTC] before next, messages elided = 12
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.BlockFetchDecision:Info:217] [2022-10-29 06:53:03.58 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 180.150.96.245:2700}) (Right [>
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Info:207] [2022-10-29 06:53:03.59 UTC] before next, messages elided = 2493813056099743
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Info:207] [2022-10-29 06:53:03.59 UTC] Valid candidate 31082ac6c8beac8490ffa70a83b8e5693786dc9da2e8316212a1c3f528dd6492 at slot 75460099
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Error:207] [2022-10-29 06:53:03.59 UTC] Candidate contains blocks from future exceeding clock skew limit: 31082ac6c8beac8490ffa70a83b8e5693786dc9da2e8316212a1c3f>
Oct 29 16:53:06 relay1 cardano-node[617434]: [relay1:cardano.node.BlockFetchDecision:Info:217] [2022-10-29 06:53:06.26 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 13.59.88.248:3001}) (Right [At>
Oct 29 16:53:06 relay1 cardano-node[617434]: [relay1:cardano.node.BlockFetchDecision:Info:217] [2022-10-29 06:53:06.27 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 45.32.153.230:3000}) (Right [A>
Oct 29 16:53:06 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Notice:207] [2022-10-29 06:53:06.50 UTC] Chain extended, new tip: c1fa0c0ea3fb46010cf5b26c11532e8c060c996bc63c352182240dcc60a73f5c at slot 75460095
Oct 29 16:53:06 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Info:207] [2022-10-29 06:53:06.56 UTC] Block fits onto some fork: 90c80844fc579a10e8bd4a7f4fa769dda1506f03f1876e7a955265b409de9bd2 at slot 75460095
Oct 29 16:53:06 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Notice:207] [2022-10-29 06:53:06.56 UTC] Switched to a fork, new tip: 90c80844fc579a10e8bd4a7f4fa769dda1506f03f1876e7a955265b409de9bd2 at slot 75460095

Logs start after my node received block for slot 75460085. The next block received is yours for slot 75460099. Note the next log message “Candidate contains blocks from future exceeding clock skew limit”. Then my node receives the two blocks for slot 75460095 and switches to the slot battle winner (90c80844fc579a10e8bd4a7f4fa769dda1506f03f1876e7a955265b409de9bd2).

By the way, if you look at the logs on one of your relays, you would at least see the ordering of how the blocks arrived. In other words, your own relay’s logs would show the problem too.

natefilm · 29 October 2022 22:34

Thanks for digging this up. Found on my relay as well.

Maybe the newly added relay was the issue? We’ll see, thanks again.

Terminada · 30 October 2022 00:16

Your block producer produced its block too early. Therefore at least you block producer’s clock is wrong.

It is strange that the rest of the network nodes forwarded your early (thus invalid) block though. I didn’t think that was supposed to happen. I know we have pipelining now but I thought the headers were still verified before propagating the block.

After thinking that through a bit more: Maybe the policy about this is that different nodes on the network can have different times and so you don’t want one particular node refusing to propagate a block based on its concept of correct time???

natefilm · 30 October 2022 19:59

Yeah it was my block producer, chrony was actually not installed and gives me an error when trying to start the service. Service can’t be found after making and moving the .conf file, assume it’s a path issue. I did install and setup NTP for the time being. All 3 relays have chrony installed and is running correctly.

I understand the value in having a hardware server clock now.

It’s a strange issue but I think you’re right on the concept of correct time. It’s been awhile since I made a bock, BP was ready to go!

EDIT: Instead of trying to start chronyd, I started just chrony and that ended up fixing my issue on the install. Strange.

Topic		Replies	Views
Does cncli database store correct leader_vrf values? Operate a Stake Pool	7	550	1 October 2022
Understanding slot battles Education	1	594	5 August 2023
Slot battles: what do they look like & what can be done about them? Operate a Stake Pool ouroboros , stake-pools	7	3423	10 January 2021
Increased number of Orphaned blocks Operate a Stake Pool	21	1437	11 March 2022
I just received a a ghosted block, what can I do to prevent this? Community Technical Support	1	294	19 May 2023

Orphaned block, three way Height Battle

Related Topics