About missed slots

[Jose_Carlos]

I have this metric in prometheus and really don’t understand what it means or if its a problem or not.
Can somebody clarify what’s the missed slots and how can be fixed?

[laplasz]

Hi!

Have you checked these topics?

Investigating missed slots

Hello everyone, Last week I started seeing my missed slots count increase in Grafana and I’m trying to figure out why. Right now, my core is missing a slot every 3-4 minutes. This seems to be a constant rate. The core has 4 relays connected to/from in, 2 that are within the same LAN and 2 that are outside. Each of the 4 relays have ± 20 peers (in and out) All machine (relays & cores) are at ~5-6% CPU and 40% RAM Nothing changed in the setup/configuration for weeks The core logs: Mar 03 02:1…

Slots missed when logging to file

I’m getting missed slots, so I through I would play around with my block node and see if I could do something about it. I through maybe the logging was a bit excessive, so I changed the log to goto /dev/null. Making the above change seem to have fixed it, at least I went more then 12 hours without missing a slot. Previously, I would go about 8 hours and then I would start missing slots. I’m using Grafana to view the missed slot metric, and wondering if my change actually fixed it, or just stopp…

Missed slots

Hi, I’m a SPO and I just wondered, how many missed slots by BP is normal, if any. My pool gets about 20-30 missed slots in 24 hours.

[Jose_Carlos]

sure looks like we can’t lose blocks because of this, I’ll try to ignore by now

[laplasz]

Could give it a try to calculate your network delay, I would be curious what is the value for your BP node…

Network delay calculation based on logs

Hi There! Recently my pool missed to create a valid block. and the reason was that the block was produced not on the highest block but on the previous one. So I tried to calculate what was the delay time between the highest block production time and the actual time when the pool was received it. so here is the command - right now it only works on text format, not json format: grep -h "new tip" node.log | sed -rn "s/\[(.*)\].\[(.*)\].* ([0-9]+)/\2,\3/p" | awk -F, -v threshold=2 '{"date +%s.%3N…

[ToTheMoonADA]

I’ve been experiencing some missed slots too but i’ve notice it only happens after the end of ever block and the beginning of a new one.

[laplasz]

@ToTheMoonADA

block

you mean epoch? are you running 1.26.2 version of cardano-node?

[ToTheMoonADA]

sorry yes epoch. And yes I’m running 1.26.2

[laplasz]

can you give it a try calculate what is the delay of your network?

[Jose_Carlos]

how can I do that? glview won’t display RTT more than a few ms

[laplasz]

ok - I post it again:

Network delay calculation based on logs

Hi There! Recently my pool missed to create a valid block. and the reason was that the block was produced not on the highest block but on the previous one. So I tried to calculate what was the delay time between the highest block production time and the actual time when the pool was received it. so here is the command - right now it only works on text format, not json format: grep -h "new tip" node.log | sed -rn "s/\[(.*)\].\[(.*)\].* ([0-9]+)/\2,\3/p" | awk -F, -v threshold=2 '{"date +%s.%3N…

[ToTheMoonADA]

Here is a sample of the delay I got using your post.

2021-05-05 12:44:12.53 UTC’
2021-05-05 12:44:18.23 UTC’
2021-05-05 12:44:48.29 UTC’
2021-05-05 12:44:53.27 UTC’
2021-05-05 12:45:08.34 UTC’
2021-05-05 12:45:11.30 UTC’
2021-05-05 12:45:13.08 UTC’
2021-05-05 12:45:25.27 UTC’
2021-05-05 12:45:35.14 UTC’

[laplasz]

Will dm you, since this is not the desired output of the command…

[Jose_Carlos]

sorry I cannot execute any of these commands are they broken?

[laplasz]

will dm you - should work…

[Jose_Carlos]

{“thread”:“837”,“loc”:null,“data”:{“kind”:“TraceTxSubmissionOutboundSendMsgReplyTxs”,“peer”:“ConnectionId {localAddress = 62.171.147.209:45967, remoteAddress = 52.76.9.195:6000}”},“sev”:“Info”,“env”:“1.26.2:35312”,“msg”:"",“app”:,“host”:“vmi58081”,“pid”:“376087”,“ns”:[“cardano.node.TxOutbound”],“at”:“2021-05-07T15:28:20.81Z”}

{“thread”:“9973”,“loc”:null,“data”:{“kind”:“TraceTxInboundCannotRequestMoreTxs”,“peer”:“ConnectionId {localAddress = 0.0.0.0:6000, remoteAddress = 68.183.194.220:34789}”,“count”:0},“sev”:“Info”,“env”:“1.26.2:35312”,“msg”:"",“app”:,“host”:“vmi58081”,“pid”:“376087”,“ns”:[“cardano.node.TxInbound”],“at”:“2021-05-07T15:28:20.83Z”}

{“thread”:“22915”,“loc”:null,“data”:{“kind”:“TraceTxInboundCannotRequestMoreTxs”,“peer”:“ConnectionId {localAddress = 0.0.0.0:6000, remoteAddress = 54.151.49.15:35095}”,“count”:0},“sev”:“Info”,“env”:“1.26.2:35312”,“msg”:"",“app”:,“host”:“vmi58081”,“pid”:“376087”,“ns”:[“cardano.node.TxInbound”],“at”:“2021-05-07T15:28:20.89Z”}

{“thread”:“11082”,“loc”:null,“data”:{“kind”:“TraceTxSubmissionOutboundRecvMsgRequestTxs”,“peer”:“ConnectionId {localAddress = 62.171.147.209:43851, remoteAddress = 106.72.39.130:5513}”},“sev”:“Info”,“env”:“1.26.2:35312”,“msg”:"",“app”:,“host”:“vmi58081”,“pid”:“376087”,“ns”:[“cardano.node.TxOutbound”],“at”:“2021-05-07T15:28:20.89Z”}

{“thread”:“11082”,“loc”:null,“data”:{“kind”:“TraceTxSubmissionOutboundSendMsgReplyTxs”,“peer”:“ConnectionId {localAddress = 62.171.147.209:43851, remoteAddress = 106.72.39.130:5513}”},“sev”:“Info”,“env”:“1.26.2:35312”,“msg”:"",“app”:,“host”:“vmi58081”,“pid”:“376087”,“ns”:[“cardano.node.TxOutbound”],“at”:“2021-05-07T15:28:20.89Z”}

{“thread”:“9023”,“loc”:null,“data”:{“kind”:“TraceTxInboundCannotRequestMoreTxs”,“peer”:“ConnectionId {localAddress = 0.0.0.0:6000, remoteAddress = 172.104.144.137:46207}”,“count”:0},“sev”:“Info”,“env”:“1.26.2:35312”,“msg”:"",“app”:,“host”:“vmi58081”,“pid”:“376087”,“ns”:[“cardano.node.TxInbound”],“at”:“2021-05-07T15:28:21.02Z”}

[laplasz]

yes, that is json format… I will update the script to handle that format, and get back to you