Looking for help on forged_block_from_future

Hello folks, I am running a pool, and sometimes see forged_block_from_future int becomes positive. I have chrony installed and have a relay.

Also, i have been searching and reading and changed the poll interval of the chrony from [2 seconds to 4 seconds] into [0.5 seconds to 1 second], After around 1 week or so recently, I saw the forged block from future becomes [2], and my produced blocks since last restart were [2] as well. Seems like there is a significant space for me to improve

My question is could this be improved by some configuration or setup?
If this is indeed the “orphaned block”, is it indeed something unavoidable?

Any idea is greatly appreciated :pray:

just wanted to follow up on this thread, I checked twice after I posted the thread, and I noticed the forged block from future raised a little bit. No orphaned block seen on the pooltool

Sorry, I hate to see questions left unanswered. Have you tried posting on Cardano StackExchange as well? Personally, I don’t know the answer to this, but it drives me crazy that here we are 1/4 of the way through the 21st century, and it is still so difficult to research issues like this.

That is why we started a project last year that I think you may find interesting in the long term. We are going to submit a Proposal for Catalyst F10. The idea we are calling Integrated Centralized Error Management, or ICEM. With ICEM, we centralize the reporting, lookup, research, and solutioning of errors like the one you got in Cardano-CLI. The software, instead of presenting you with some esoteric error code, will also present you with the ICEM-provided input from the community on what it means, and how to solve it, as well as the error code. The solution is called CodaEA, which AlexD is also piloting on scanning the Cardano logs and identifying/reporting issues.

Our request to the community would be for IOHK to incorporate CodaEA as an ICEM system into the Cardano Node code, so that when it reports an error, it also references community input and links (e.g. to this forum) along with the code.

If you like this idea, please look for us when Fund10 gets underway and vote your support!

1 Like

Where exactly you see forged_block_from_future is it Prometheus metric or what?

1 Like

I just grep’ed through my logs on my block producer for the word “future” (case insensitive) over the past 15 days and got no hits at all.

If you are seeing these messages in your logs (journal) then I suspect that either you have a clock problem or one of the block producers you got a block from had a clock problem. Perhaps your node is receiving blocks that it perceives are from the future because your node’s clock is set in the past???

I do recall there was a poster on this forum that had a problem with his block producer clock that resulted in his block being rejected by the rest of the network. In that case, his block producer’s clock was set in the future and consequently he made his block early and it was rejected by other relays. In that case, I believe other relays produced logs that said something like “forged block from future”.

thanks! this is prometheus, came from curl http://127.0.0.1:12798/metrics 2>/dev/null | grep -e IsLeader -e orge | grep -v KES

sounds great! will do when this happens

thanks! I just grep’ed the log on the keyword “future” and I only saw around 7 messages saying Candidate contains blocks from future exceeding clock skew limit
happened in October and November. But the forged_block_from_future has been 20+, with some of them happened in December and January as well. I also grep’ed on “clock”, got the exactly the same results

Thus it seems the log doesn’t reflect anything about the forged_block_from_future

The concern that I have is that if your relay / bp is logging messages like that and other relays receiving the same blocks are not logging such messages then this would seem to indicate that the problem is with your clock.

Doe the computer get a lot of clock drift causing chrony to make lots of changes??? I don’t know how to look into that properly but it is likely that chrony logs the changes it makes.

I see, the log was off by default, I have just turned it on, will see the log in the future

I don’t have metric forged_block_from_future at all and I have metrics history for 6+ months.

I looked at the log so far, seems the IP ends with 240 isn’t doing well because:
-some discussion mentioned the numbers should always be 11 111 1111, which I think refers to the columns 123 567 ABDC, and the IP ends with 240 constantly displays 0 at one of the digits

a typical part of the log is:

I am going to remove this IP and see if this makes a difference