Orphaned block, three way Height Battle

Hi, I’m the operator of BIKES and was wondering if there was anything I could’ve done to prevent this or was this the luck of the draw?

Could it be the performance of my nodes that could’ve made this happen? Seems likely if my pool had the lower Block and Leader VRF value?

When digging into BP logs I get (Block fits onto some fork: “” at slot 75460095), which was the slot North minted the block at.

Ran:

curl -s localhost:12798/metrics | grep cardano_node_metrics_Forge

Received:

cardano_node_metrics_Forge_adopted_int 1
cardano_node_metrics_Forge_forged_int 1
cardano_node_metrics_Forge_forge_about_to_lead_int 513083
cardano_node_metrics_Forge_node_is_leader_int 1
cardano_node_metrics_Forge_node_not_leader_int 513082

One idea was time sync, all nodes are running chrony.

Or is it the new parameter changes with VRF values?

Stumped on this if anyone has any insight would be greatly appreciated, thanks.

Your clock is out of sync.

Here is what I have in my logs:

Oct 29 16:52:56 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Info:207] [2022-10-29 06:52:56.87 UTC] Valid candidate eebf634dca68962f78bc71b9978a6a868365097916d89fc155244ae7c05c41d1 at slot 75460085
Oct 29 16:52:56 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Notice:207] [2022-10-29 06:52:56.87 UTC] Chain extended, new tip: eebf634dca68962f78bc71b9978a6a868365097916d89fc155244ae7c05c41d1 at slot 75460085
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.BlockFetchDecision:Info:217] [2022-10-29 06:53:03.34 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 13.59.88.248:3001}) (Right [At>
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.BlockFetchDecision:Info:217] [2022-10-29 06:53:03.35 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 52.14.17.13:6000}) (Right [At >
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.BlockFetchDecision:Info:217] [2022-10-29 06:53:03.58 UTC] before next, messages elided = 12
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.BlockFetchDecision:Info:217] [2022-10-29 06:53:03.58 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 180.150.96.245:2700}) (Right [>
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Info:207] [2022-10-29 06:53:03.59 UTC] before next, messages elided = 2493813056099743
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Info:207] [2022-10-29 06:53:03.59 UTC] Valid candidate 31082ac6c8beac8490ffa70a83b8e5693786dc9da2e8316212a1c3f528dd6492 at slot 75460099
Oct 29 16:53:03 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Error:207] [2022-10-29 06:53:03.59 UTC] Candidate contains blocks from future exceeding clock skew limit: 31082ac6c8beac8490ffa70a83b8e5693786dc9da2e8316212a1c3f>
Oct 29 16:53:06 relay1 cardano-node[617434]: [relay1:cardano.node.BlockFetchDecision:Info:217] [2022-10-29 06:53:06.26 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 13.59.88.248:3001}) (Right [At>
Oct 29 16:53:06 relay1 cardano-node[617434]: [relay1:cardano.node.BlockFetchDecision:Info:217] [2022-10-29 06:53:06.27 UTC] [TraceLabelPeer (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 45.32.153.230:3000}) (Right [A>
Oct 29 16:53:06 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Notice:207] [2022-10-29 06:53:06.50 UTC] Chain extended, new tip: c1fa0c0ea3fb46010cf5b26c11532e8c060c996bc63c352182240dcc60a73f5c at slot 75460095
Oct 29 16:53:06 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Info:207] [2022-10-29 06:53:06.56 UTC] Block fits onto some fork: 90c80844fc579a10e8bd4a7f4fa769dda1506f03f1876e7a955265b409de9bd2 at slot 75460095
Oct 29 16:53:06 relay1 cardano-node[617434]: [relay1:cardano.node.ChainDB:Notice:207] [2022-10-29 06:53:06.56 UTC] Switched to a fork, new tip: 90c80844fc579a10e8bd4a7f4fa769dda1506f03f1876e7a955265b409de9bd2 at slot 75460095

Logs start after my node received block for slot 75460085. The next block received is yours for slot 75460099. Note the next log message “Candidate contains blocks from future exceeding clock skew limit”. Then my node receives the two blocks for slot 75460095 and switches to the slot battle winner (90c80844fc579a10e8bd4a7f4fa769dda1506f03f1876e7a955265b409de9bd2).

By the way, if you look at the logs on one of your relays, you would at least see the ordering of how the blocks arrived. In other words, your own relay’s logs would show the problem too.

1 Like

Thanks for digging this up. Found on my relay as well.

Maybe the newly added relay was the issue? We’ll see, thanks again.

Your block producer produced its block too early. Therefore at least you block producer’s clock is wrong.

It is strange that the rest of the network nodes forwarded your early (thus invalid) block though. I didn’t think that was supposed to happen. I know we have pipelining now but I thought the headers were still verified before propagating the block.

After thinking that through a bit more: Maybe the policy about this is that different nodes on the network can have different times and so you don’t want one particular node refusing to propagate a block based on its concept of correct time???

1 Like

Yeah it was my block producer, chrony was actually not installed and gives me an error when trying to start the service. :sweat: Service can’t be found after making and moving the .conf file, assume it’s a path issue. I did install and setup NTP for the time being. All 3 relays have chrony installed and is running correctly.

I understand the value in having a hardware server clock now.

It’s a strange issue but I think you’re right on the concept of correct time. It’s been awhile since I made a bock, BP was ready to go!

EDIT: Instead of trying to start chronyd, I started just chrony and that ended up fixing my issue on the install. Strange.