Missed a block but not an orphan. Can you figure out what happened?

[mdsullivan]

So just missed a block for no apparent reason.
Both relays are running fine, and BP has been up for 2 days, processing transactions and we minted a block just a few hours prior.
{
“no”: 3,
“slot”: 39210028,
“slotInEpoch”: 157228,
“at”: “2021-09-04T13:25:19-04:00”
},

A search on Cardano Explorer doesn’t tell me who minted this block over us.
This is our first missed block that wasn’t an ‘orphan’ and I’m trying to make sure I know what’s happened to prevent any possible future problems.
-Sully at RADAR

[Anti.biz]

I missed a block also

Could we be missing blocks because not everyone is on 1.29?

[mdsullivan]

I don’t think so, Anti.biz. My cardano-node log file from the missed block is below.
Looks like there was a fork and someone else’s tip was accepted over mine.
Could someone mind taking a look at this log and telling me that’s all this was?
Usually when this happens, Pooltool.io will log an “orphaned” block, but for some reason it’s not showing up. Just making sure my this is just that random bad luck that will cause you to lose a slot battle about 1 in 20 blocks…
Thanks!
-Sully at RADAR

Sep 04 13:25:19 sully-OptiPlex-7050 bash[2067]: [sully-Op:cardano.node.LeadershipCheck:Info:80] [2021-09-04 17:25:19.00 UTC] {“kind”:“TraceStartLeadershipCheck”,“chainDensity”:4.9415212e-2,“slot”:39210028,“delegMapSize”:806007,“utxoSize”:3071773,“credentials”:“Cardano”}
Sep 04 13:25:19 sully-OptiPlex-7050 bash[2067]: [sully-Op:cardano.node.Forge:Info:80] [2021-09-04 17:25:19.00 UTC] fromList [(“val”,Object (fromList [(“kind”,String “TraceNodeIsLeader”),(“slot”,Number 3.9210028e7)])),(“credentials”,String “Cardano”)]
Sep 04 13:25:19 sully-OptiPlex-7050 bash[2067]: [sully-Op:cardano.node.Forge:Info:80] [2021-09-04 17:25:19.03 UTC] fromList [(“val”,Object (fromList [(“kind”,String “TraceForgedBlock”),(“slot”,Number 3.9210028e7)])),(“credentials”,String “Cardano”)]
Sep 04 13:25:19 sully-OptiPlex-7050 bash[2067]: [sully-Op:cardano.node.ChainDB:Notice:70] [2021-09-04 17:25:19.04 UTC] Chain extended, new tip: f320e7479cc53890c1abfc9c382d8ba62388ef75b0c0b319afb86a3bfb11b633 at slot 39210028
Sep 04 13:25:19 sully-OptiPlex-7050 bash[2067]: [sully-Op:cardano.node.Forge:Info:80] [2021-09-04 17:25:19.05 UTC] fromList [(“val”,Object (fromList [(“blockHash”,String “f320e7479cc53890c1abfc9c382d8ba62388ef75b0c0b319afb86a3bfb11b633”),(“blockSize”,Number 25538.0),(“kind”,String “TraceAdoptedBlock”),(“slot”,Number 3.9210028e7)])),(“credentials”,String “Cardano”)]
Sep 04 13:25:19 sully-OptiPlex-7050 bash[2067]: [sully-Op:cardano.node.Mempool:Info:78] [2021-09-04 17:25:19.05 UTC] fromList [(“txs”,Array [Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “9b799eb296b2d1ce51d5e096c9e2aa015436a149bb2ac966a9f5fbaad765de40”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “e5f89b76d30175ae4ad72ed49f882429d82ebbe0a3183b8c21aa7058a98cac6f”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “ca956786edc94151c1459a168440b55beb63b1cce06d6b72a06904f8ca1c8229”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “a6234de47b94382aaa7b6eace156a8ec384add62d0eb6ed3601e3d34c5eced0e”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “73cbebc53c760ca2b444b2a1c390557f92973ae3682698af4d00a6575d326330”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “3ab02cc3fe30ca7ac18fbc9cf0813e256763f8f1316dec4e302f8679937a7a8a”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “c02c9ea5985f2e9cdadccb99dee47f40b8f4df29364380384659a0126c3e3eb1”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “0c1b9ae841cf178a195a4e01ddbd8d5b44a39c39da9f1af8ee56bc563b4834c9”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “714ebcd6d1cdb4f4f42ff7c6554e093787b10e08ec7098beaa62b87c687b2190”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “c8ad7a648cb1c647f04e51886cd331873bdd6534bd98060529a29e5ef73da524”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “12ca19cd6948f71173a66f82f63c8a6441672ea99d17f8e64ba44a512571a5a6”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “80a281fded9c170d0e25672d668de5bb077cf2d4cf8e4748c6ce7e562382b3cc”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “8e49dbee3e72a5831b42bce6b62b126c7a5196b1b6158308ee08633ea77d0eb9”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “fcf8d77b851b173cf184a2f98f650ebfb96a21b091a88e769d402c42ac90ad77”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “533c518a7fc089c4b0308dc65996ba934c01279e8b83a97ee5196fc7e554f4b9”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “b704206176982f48568623e2450a2c562ff8a96cf571f7195fc5f275f86dcbef”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “6d2b999adc724268eea5b04748325387213f48bd67d6cdabed9a74ed348de1e7”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “cc6cca93445bb5896a74a6d1e4b008c0dfc82a875f8c28866706c74782928b98”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “3788fc7f1b2c66a6aafd8fbacdb3808adf578b0b69d3148f200c30f23e2b9254”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “7599db4965efa86b3342d97a4eb2e5da9f6d0e5c088cc409dd20dbeaab759c43”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “4041963f0b33b57bfb53d64f421ec71d60ce2352e86954486cb23013aee136fa”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “20d3825890b7caa74e47521f8faec95407f3cdd81663454e9d58ed8c5d7e07e6”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “d8e705cde68493684559e0251311fa70b6327f3c06ae41d66ba35abd0f3d84f8”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “0ff288d4a1b95cf91810936ed1d1306fd623049fc785c03bbc1f4bccbfd60530”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “26fec5ad036b50dda565dce313fcfe1ba8cb7716eb634f89e9540057bf957613”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “eff5718cdb43bf4123d944e8af559e57ad4c4f9dbd0838a4a6604f1cf5539dd1”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “d200ae8554105f09da684af10682df08054b42fb728f65587f09ee41e6850456”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “6fdcb4925344e9dc63b881daa2d81f7042a70d1bcceb2946976e5430f62510c4”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “4380e6789e66cceb159b545b20c93c1bfa41bebeccd129a80af3df12a8ee7ef9”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “4dbbbb3127e6c1fe8a7ecd37a4270ce2bd4d13ecc4f4af95b0aa0a574c38ff11”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “452279c1bcb3ea8030d09fa5e5221ac537adf612e8f7913bb8828e82d8cb9157”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “088f6c6c301ce71ce552ea2a556d71ac9673f8716d2f5cc2d561b3e4c6be5646”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “614805143889024d99156965b1519fb9180fe5d3bbb4d01c62a4af63881d4f6a”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “59d90bd9e5a2cc857921c0ab315ded694798ed05f3731eed31f36a376a54c98b”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “72e9e4700405b0a6c44036606f05eb8ebf973edd3d03ff230ad6674fe694776a”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “3662e59d1159e4d6859a7caf19646a206d2b480ee2872d595ffe5d8e63586cf8”}”)]),Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “c45916a1795b4918b8fa1108df8748f20a70983a7072a1ccfd6d54308cbb39c4”}”)])]),(“mempoolSize”,Object (fromList [(“bytes”,Number 0.0),(“numTxs”,Number 0.0)])),(“kind”,String “TraceMempoolRemoveTxs”)]
Sep 04 13:25:19 sully-OptiPlex-7050 bash[2067]: [sully-Op:cardano.node.ChainDB:Info:70] [2021-09-04 17:25:19.43 UTC] Block fits onto some fork: dc9e22bf3d65f39442abcb120f7aba5eff44948e64999e214cd1f1bbe20ba9ef at slot 39210023
Sep 04 13:25:19 sully-OptiPlex-7050 bash[2067]: [sully-Op:cardano.node.ChainDB:Info:70] [2021-09-04 17:25:19.46 UTC] Valid candidate dc9e22bf3d65f39442abcb120f7aba5eff44948e64999e214cd1f1bbe20ba9ef at slot 39210023
Sep 04 13:25:19 sully-OptiPlex-7050 bash[2067]: [sully-Op:cardano.node.ChainDB:Notice:70] [2021-09-04 17:25:19.46 UTC] Switched to a fork, new tip: dc9e22bf3d65f39442abcb120f7aba5eff44948e64999e214cd1f1bbe20ba9ef at slot 39210023
Sep 04 13:25:19 sully-OptiPlex-7050 bash[2067]: [sully-Op:cardano.node.Mempool:Info:17018271] [2021-09-04 17:25:19.49 UTC] fromList [(“tx”,Object (fromList [(“txid”,String “txid: TxId {_unTxId = SafeHash “781863cf75becbaa671dbc8153d8424db397df8d718da52e433d02bb4021ade3”}”)])),(“mempoolSize”,Object (fromList [(“bytes”,Number 2607.0),(“numTxs”,Number 1.0)])),(“kind”,String “TraceMempoolAddedTx”)]

[jeremyisme]

I’ve had the same problem just yesterday. The first time it has happened. I have a similar log as you. Has anyone got any thoughts? It doesn’t show up as an orphaned block on pooltool, it’s as though the block doesn’t exist…

[Chris.AXN]

Hi guys, I just missed a block a few hours ago as well. I tried using Cardano explorer to see who minted the block but I can’t even find the slot or any info regarding the slot. My logs also appear to stop a few thousand slots before the missed slot so I cant find any info regarding why I missed the slot.

[mdsullivan]

In my situation, I believe it was just a slot battle I lost. There is a chance it didn’t appear in pooltool because I wasn’t sending my tip at that moment.
But I have had the same experience since then one other time when I know I was sending, yet pooltool didn’t show it. Still not clear on what would cause that.

[extramileit]

I also got bit by this error just now (see log output below). It shows up as Invalid in gLiveView. It appears it wasn’t a slot battle, since I can’t find any Block Details for epoch 323, slot 428489, i.e. search term “323.428489” in https://cardanoscan.io.

Does this mean we still have pools with invalid configuration for their multiple BP nodes as discussed here? If so, this is really lame!

Invalid block - no known cause

late to this party but this seems to me like a classic issue with multiple valid forks on the network, you appended block to a fork that was later discarded. This is being caused by pools that run multiple BPs in parallel, where each BP creates a valid, but different block. Then these 2 different chains are propagated through network, and shorter chain is later discarded. You were a victim of their negligence. This was mentioned recently on many places, including reddit and twitter. Here are so…

[2022-03-01 20:46:20.37 UTC] Block fits onto some fork: 21c0bee54e48a59af12fdb9ae4498