Anyone else missing blocks since Vasil?

Just checked my BP and so far I’ve been ‘missed’ 7 blocks since the epoch change… Never had a miss before today, so something seems strange. Anyone else having issues?!

1 Like

What’s the message? Did u check inside the logs?

I checked mine after ur post and I had 3 missed, I upgraded the cncli, deleted the blocks for epoch 365 then run again leaderlog force… and now all good

If u are using cncli to calculate the slots… check this out

:exclamation:CNCLI leaderlog Vasil HFE 1-epoch edge case

The reason:

Cardano mainnet merge into Babbage era 22 Sep @ 21:44 UTC. One of the changes is that we go from TPraos(Transitional Praos) to Praos ouroboros consensus. Due to this change, cncli has a new --consensus parameter that sets the correct type.

The problem:

The cncli.sh wrapper script in Guild script suite has this parameter implemented, but there is an edge case for the transition. As leaderlog is calculated before the HFE occur, the wrong consensus for upcoming epoch is identified.

The solution:

Make sure that you have the latest version of cncli.sh wrapper script deployed. Also make sure to run the latest version of cncli binary, v5.1.2 as of this message. Both can be deployed/updated using prereqs.sh script.

After HFE event, ie after crossing into epoch 365, run the below command from the scripts folder to re-calculate leaderlog for current epoch.

./cncli.sh leaderlog force

Optional:

If pooltool sendslots is active, you might also want to run the below command to force submission of slots to pooltool after calculating correct leaderlog.

./cncli.sh ptsendslots force

5 Likes

I’ve missed blocks since Vasil too.
result1
This is what I get from ./cncli.sh leaderlog force. Does this look OK?


and this from ./cncli.sh ptsendslots force

and can you please explain “deleted the blocks for epoch 365”. How to do that? Thank you

Assuming you have the standard CNTools directory structure, navigate to

/opt/cardano/cnode/guild-db/blocklog

(which has in it the blocklog.db file)

then run the script and manually fill in the ${BLOCKLOG_DB} location

$ sqlite3 “./blocklog.db” “DELETE FROM blocklog WHERE epoch=365;”

once epoch 365 is deleted from blocklog.db, then you can navigate back to $CNODE_HOME/scripts and the

./cncli.sh leaderlog force

./cncli.sh ptsendslots force

commands work as expected, and cntools/gLiveView should now match the cncli raw output

1 Like

This worked for me - no missed blocks! Appreciate it.

1 Like

I’m still losing block. Looks like I’m not doing it right.

I have standard CNTools directory structure as you mentioned /opt/cardano/cnode/guild-db/blocklog

Could you please clarify exactly what do I have to type.
$ sqlite3 …

Appreciate your help.

$ sqlite3 “./blocklog.db” “DELETE FROM blocklog WHERE epoch=365;”

once epoch 365 is deleted from blocklog.db, then you can navigate back to $CNODE_HOME/scripts and the

./cncli.sh leaderlog force

.

I don’t have /opt/cardano/cnode/guild-db/blocklog in my directory or sqlite3 installed (Never needed it before) I seems to be miss calculation leaderlogs.
Cnc tools running 5.1.2 so unsure whats the problem.

Try to run ./cncli.sh leaderlog force
if still see the missed slot go to pooltool.io → orphans and check if it was a battle for that block

Used Cardano CLI query which shows
72357501 2022-09-23 09:XXXXX
Missed block i think, Not shown as orphan on Pooltool :frowning:

You can use cardano-cli to check your leaderlog as follows:

CARDANO_NODE_SOCKET_PATH='/run/cardano/mainnet-node.socket' cardano-cli query leadership-schedule --mainnet --genesis /etc/cardano/mainnet-shelley-genesis.json --stake-pool-id "blahblahblah" --vrf-signing-key-file /etc/cardano/private/vrf.skey --current
1 Like

Yes that’s working - Missed slot but can’t find any battle not sure what happed -Possibly i was running a leader log script at the time of my scheduled block - Typical.

7793648 Block

72357501 Schedule 09/23 09:xxxxx

1 Like

You have your leaderlog, so you know the timing of when your block should have been produced. You also have cncli storing all block data seen by your node.

  1. Check your systemd journal logs right at the time of the block production. Eg:
journalctl --since "2022-09-23 20:50:15" --no-pager | head -n 10

Sep 23 20:50:15 bprod1 cardano-node[1176602]: [bprod1:cardano.node.LeadershipCheck:Info:199] [2022-09-23 10:50:15.00 UTC] {“chainDensity”:4.8593294e-2,“credentials”:“Cardano”,“delegMapSize”:1219549,“kind”:“TraceStartLeadershipCheck”,“slot”:
72363924,“utxoSize”:8583870}
Sep 23 20:50:15 bprod1 cardano-node[1176602]: [bprod1:cardano.node.Forge:Info:199] [2022-09-23 10:50:15.02 UTC] fromList [(“credentials”,String “Cardano”),(“val”,Object (fromList [(“kind”,String “TraceNodeIsLeader”),(“slot”,Number 7.2363924e7)]))]
Sep 23 20:50:15 bprod1 cardano-node[1176602]: [bprod1:cardano.node.Forge:Info:199] [2022-09-23 10:50:15.04 UTC] fromList [(“credentials”,String “Cardano”),(“val”,Object (fromList [(“block”,String “d622cfe93efcbd0735c6f2b71926e1e8a5eb6438adde61479fe410147167fb0c”),(“blockNo”,Number 7793988.0),(“blockPrev”,String “3500a6a24587fc13f02dac6a8b44c6f87eadd181f0437e1036992564d1485466”),(“kind”,String “TraceForgedBlock”),(“slot”,Number 7.2363924e7)]))]
Sep 23 20:50:15 bprod1 cardano-node[1176602]: [bprod1:cardano.node.ChainDB:Notice:189] [2022-09-23 10:50:15.04 UTC] Chain extended, new tip: d622cfe93efcbd0735c6f2b71926e1e8a5eb6438adde61479fe410147167fb0c at slot 72363924

The logs should confirm that your block was created as shown by the key aspects I highlighted in bold.

  1. Confirm your block was adopted on chain

One way to quickly check is to go to pooltool.io and check your own pool blocks.

Another way is to look at the block data stored by your cncli database. You can view your last 10 blocks with this query:

sqlite3 /var/lib/cardano/cncli.db 'select block_number, slot_number, hash, orphaned from chain where pool_id="08f05bcfaada3bb5c038b8c88c6b502ceabfd9978973159458c6535b" order by slot_number desc limit 10;'

7797962|72445831|88e58e6795b1fa753a5da6548f2dd2e58429149f2d351fcd6afdf92297b5f18d|0
7793988|72363924|d622cfe93efcbd0735c6f2b71926e1e8a5eb6438adde61479fe410147167fb0c|0
… snip …

The last column in the output indicates if the block was orphaned or not. In my case the last 2 blocks were not orphaned (‘0’). A ‘1’ indicates orphaned.

You won’t necessarily see your orphaned block data on pooltool.io since it will only show there if your block got propagated to pooltool’s nodes. For example, it won’t show at pooltool if other relays chose not to propagate your block because it lost a slot battle or height battle and therefore didn’t make it to pooltool. However, if you are running cncli and have your database sync’ed then your orphaned block will be in that database if your own node made the block.

Doing the above investigation work should identify any problems.

1 Like

It’s solved. Everything looks normal. No missed blocks. I tried running
./cncli.sh leaderlog force (which took quite a while)

Then check in cntools and it will show new calculation.

Thank you Alex

1 Like