Hi I am running a pool that mints ~10 blocks an epoch ~500ms propagation and am discovering that I will lose at least 1-2 blocks every epoch and is causing quite a toll on RoS. These blocks rarely show as orphaned. I have tried playing around with topologies and system resources with no luck. The pool runs on 16gb of ram and 2 Vcores. I am wondering if its just because the pool is located in Australia and the propagation times aren’t snappy enough?
This epoch a block was expected and not adopted.
Link to logs at expected slot - Block Mint - Pastebin.com
To add, these were the system graphs at the time
Hello SeeEm!
I dont think your nodes’s region is the cause of the issue. Because there are 152 relays located in Australia.
You should check your relays’s connection. They need to be connected to more than 15 relays.
Hey there, I have 3 Relay Nodes with ~20 connections in 3 distinct regions.
Hi @SeeEm1
Do you have chrony installed and running on ALL nodes?
If not this could be the cause of your problem.
Also in grafana add the cardano_node_metrics_slotsMissedNum_int metric. How many missed slots you got on average each hour?
Hi,
Is your uptime good?
If the blocks are not showing up as orphans means you are not losing the slot or height battles which in turn means propagation is not the culprit.
My guess, you are simply missing the blocks
Hey there, I do have chrony installed on all of my nodes.
as for missed slots, I only lose a few on epoch transition. I am a bit confused as the logs look like the block was minted.
OK, so that is good.
Did you look up the slotnumber in the explorer to see if you minted it?
I Have just had a look and another pool minted a block in that slot. I don’t see anything about an orphan in the logs or on pool tool etc. Are there cases where a block is orphaned but not logged? Is there an easier way to know if a block is missed or orphaned?
You know which slot you were leader for. If you then lookup that slot in the explorer and find that it was minted by another pool then you are 100% sure you lost a slot battle. Except of course if you weren’t online.
The log should show you minted a block and then say it was adopted and a few lines down tell you it was rolled back (because the other leader won the battle).
The easiest way, imo, to get a good overview about all blocks adopted, confirmed, missed or stolen is to use cncli. It has a validate function that shows you that. The best setup, again imo, is to go for the guild-operators setup including cntools plus cncli. Then glliveview will show you all the info:
