Invalid block - no known cause

Staking & Delegation Operate a Stake Pool
1

It appears we had an “invalid block”. Our server was not offline at all (we monitor it constantly), the KES is valid (until 2022), it was minting blocks this epoch without problems. So this is unexpected.

The next block was fine. So it’s definitely not a KES issue. We have 6 relays, and they’re all well-connected. We did everything perfectly, so WTF?

Has this happened to anyone else? We’re using node version 1.31.0.

Does anyone know what might have caused it?

2

Block status

  • Leader : Scheduled to make block at this slot
  • Ideal : Expected/Ideal number of blocks assigned based on active stake (sigma)
  • Luck : Leader slots assigned vs ideal slots for this epoch
  • Adopted : Block created successfully
  • Confirmed : Block created validated to be on-chain with the certainty set in cncli.sh for CONFIRM_BLOCK_CNT
  • Missed : Scheduled at slot but no record of it in CNCLI DB and no other pool has made a block for this slot
  • Ghosted : Block created but marked as orphaned and no other pool has made a valid block for this slot → height battle or block propagation issue
  • Stolen : Another pool has a valid block registered on-chain for the same slot
  • Invalid : Pool failed to create block, base64 encoded error message can be decoded with echo <base64 hash> | base64 -d | jq -r
3

Thanks but what the hell does this mean, and how can we investigate and remedy the situation to prevent problems? Please explain it like I’m Biden without a teleprompter.

4

yeah but are u sure the block was Invalid and not ghosted or missed?
check inside the logs more infos about the slot

5

Where are the logs exactly?

6

I can see logs with “journalctl --unit=cardano-node --since=today”. Is this what you mean? If so, how can I download the logs so I’m not scrolling through countless lines via ssh?

7

you can check the log files inside the log folder (u can download via sftp or scp or ftp )

but also on server you can filter the output… check for the proper log file (by time/date) then try something like that:

cat file_name | grep slot_number

8

Ok thanks. I think I have the right one. It says “Block fits onto some fork”.

So what the fork happened?

9

I don’t know exactly, can be related with

10

That’s the message when you lose a slot battle or when the block was ghosted because of propagation issues, the previous block was usually not received before minting the block.
If the other block had the same slot as yours, then it is a lost slot battle, you could have done nothing to prevent it, it’s just bad luck.

11

I have one ghosted but it’s saying ghosted not Invalid

12

This is an example of a slot battle that happened a few minutes ago:

[2021-11-26 11:17:52.52 UTC] Chain extended, new tip: 177a1a333604fe937515c4e939768d0b251a5ef13229fc11804fc57cb26e4e9b at slot 46359181
[2021-11-26 11:17:52.77 UTC] Block fits onto some fork: 8b88349ff96239ace02dd9b23098cb6d2d205c69d1193eac59512f5445dfc036 at slot 46359181
[2021-11-26 11:17:52.83 UTC] Valid candidate 8b88349ff96239ace02dd9b23098cb6d2d205c69d1193eac59512f5445dfc036 at slot 46359181
[2021-11-26 11:17:52.83 UTC] Switched to a fork, new tip: 8b88349ff96239ace02dd9b23098cb6d2d205c69d1193eac59512f5445dfc036 at slot 46359181
13

late to this party but this seems to me like a classic issue with multiple valid forks on the network, you appended block to a fork that was later discarded.

This is being caused by pools that run multiple BPs in parallel, where each BP creates a valid, but different block. Then these 2 different chains are propagated through network, and shorter chain is later discarded. You were a victim of their negligence. This was mentioned recently on many places, including reddit and twitter. Here are some posts where pools that are malicious in this way were detected: https://twitter.com/SmaugPool/status/1469602098090323977/photo/1

14

there is open issue regarding this where it’s proposed that misbehaving pools get punished for this behaviour - [FR] - need to implement detection/punishing of multiple blocks produced for a given slot by the same pool · Issue #1132 · input-output-hk/cardano-node · GitHub it seems not much can be done about it right now