Why all my blocks are ghosted?

Hello!

I’m relative new as pool operator. Thanks to @Benagain I could take 5 blocks this epoch. The two first blocks are ghosted and the problem is that I don’t know how to debug it. I read the producer node’s journalctl but I didn’t see any error or notice. I suppose it’s the machine maybe too slow, 2 vCpu and 4 GB ram. Although the node’s ram is alway below 2GB and I don’t have many missed slot leader.

Can someone help me to understand why I’m having so many ghosted blocks and how I can debug it?

Thanks

EDIT:
I forgot to mention that I’m working on testnet :slight_smile:

Sounds like you are making good progress @Juantxuthree. I believe ghosted blocks are typically related to propagation times being too long. Here is a great post on the topic.

Could be time synchronization, are you running NTP?

I would also make sure that your topology is updating periodically.

1 Like

Hi Ben, I’ll install the ntp service in the producer and relay nodes. The topology is running in the relays but not in the producer. Is it right?

I read this article and I was watching the Block Propagation and it’s normally below 500 ms.

correct, topology is running only on your relays.

Ok, I installed the ntp service in the producer and relay nodes. I observed that the tip (diff) is frequently above 40 with a :expressionless: face. Which value is normal for the tip (diff).

Your Tip Diff sounds normal. Are you using coincashew or cntools? Also where are you seeing the ghosted block indication?

I used CNTOOLS. I’ve just installed Grafana for monitoring. I hope to have more data when a block is ghosted. I’ve also realized that my producer node has only 4 GB. I raise the memory to 8 GB. Maybe, if there wasn’t enough ram, the node started swapping. Therefore I had propagation delays. We will see the next block.

ok, those are all good things to try, but for reference, my Testnet BP is running fine with 4gb ram (not hosting Grafana). There is a useful tool called glances that you can install/run and it will give you info as far as swap usage.

Hi Juan,
I personally tested a lot off different VPS’s from different providers and it looks like your setup is not performant enough, maybe you get away with it on the testnet, but certainly not in real production on the mainnet. Your setup is below the requirements according to IOHK for node 1.32.1.

Minimum System Requirements:

  • An Intel or AMD x86 processor with two or more cores, at 1.6GHz or faster (2GHz or faster for a stake pool or relay)
  • 12GB of RAM
  • 20GB of free storage

Here is my suggestion:

  • Your 2vCPU’s (2GHz or faster) should be ok, as long as they are dedicated
    *16GB of RAM increasable to 32GB
  • 100GB free SSD storage (keep at least 20% of your entire diskspace free)
  • 200 Gb/s network connection

One of my relays is configured like so and it works without issues:

  • Digital Ocean droplet - Basic (shared cpu) / 16 GB RAM / 8 vCPUs / 320 GB SSD

Good luck :+1: :metal:

Thank you all for your response!! I found the error and I’m producing blocks now. The problem was that I’didnt include correctly in the relay’s topologyUpdater.sh the producer node, what a mistake!. :man_facepalming: That’s why I have a problem with the propagation, in fact any propagation :rofl:

3 Likes