Stolen Block? Pooltool updates?

[Alexd1985]

hi,

Can be more options:

- Adopted   : Block created successfully
- Confirmed : Block created validated to be on-chain with the certainty set in `cncli.sh` for `CONFIRM_BLOCK_CNT`
- Missed    : Scheduled at slot but no record of it in CNCLI DB and no other pool has made a block for this slot
- Ghosted   : Block created but marked as orphaned and no other pool has made a valid block for this slot -> height battle or block propagation issue
- Stolen    : Another pool has a valid block registered on-chain for the same slot
- Invalid   : Pool failed to create block, base64 encoded error message can be decoded with `echo <base64 hash> | base64 -d | jq -r`

You can check with CNCLI what was the issue (if was an issue) or u can check inside the logs by slot/block number

Also u can check if the BP is working well :

• check if u have IN/OUT peers (your relay)
• check if u see TX processed, if not check inside config file if TraceMempool is set to true (but setting to true it will use more RAM, ~11G); if is set to true and u have IN/OUT peers and still no TX processed then check on relay if u have IN/OUT peers (IN peers other then ur BP)
• check if the certificate is valid
  U can use the command:

cardano-cli query kes-period-info --op-cert-file node.cert --mainnet

! Adapt the path to node.cert

If everything is fine probably was a fake block/information provided by CNCLI (if u use CNCLI)

[T_rrxx]

i think everything is up and running correctly. i have noticed that the pool tools are generally indicating my relay is down (0/1). here is my bp nodes data:
note: says 0 leader/ 0 adopted because i restarted the node after the supposed minting.

the cli command returned a bunch of info including the following:
“✗ No blocks minted so far with the operational certificate at: node.cert
On disk operational certificate counter: 0”
“qKesNodeStateOperationalCertificateNumber”: null,
dos that second line above indicate it cant find the cert number?

also how would is use the cncli to search the block/slot?

[os11k]

@T_rrxx

i have noticed that the pool tools are generally indicating my relay is down (0/1)

What is your ticker? Are you sure your relay is accessible from outside world?

Is cexplorer showing that your relay is down too?

Your relay must be accessible from everywhere on port what you specified, otherwise other pools will not be able to connect to it and fetch a block.

[T_rrxx]

Ticker: DOJO
Im not very familiar with cexplorer and could not find a relay metric however my relay seems to be connected to several other relays as well as the bp:

[T_rrxx]

and wierdly enough after about 5 days of displaying that its down (it was not down) it magically popped back online just now. thanks for the relay help!

[os11k]

@T_rrxx

DOJO

Now pooltool show that it is up, so I would guess something was with your relay, it should not be shown as down, I think that the reason why you lost block. 7 incoming connections is definitely not much and it would imply that relay just started to receive incoming connections.

I would recommend you to setup monitoring of your relay, you can use cexplorer bot for it, it will send you telegram message when your relay is down and it has some other neat functionality too. You can build your own monitoring system using AWS for example, it is not very expensive. You should start using Grafana too and monitor several metrics including incoming connections and maybe setup alerts too based on them…

Monitoring is a must!

Additionally start to think about 2nd relay, you should have HA setup, running one relay will be a big problem in future, especially for small pool like yours or mine where each block is super valuable. Imagine next time getting a slot and loose internet.

[T_rrxx]

interesting, the relay has been online for almost 20h, and before that about 10 days, within that time i have had incoming & outgoing connections every time i checked. Additionally my BP was receiving 1 in and 1 out and it was always the relay. how can the relay be connected to the network but be seen as down on pooltool? What exactly determines the number of incoming connections? Additionally how do i increase the number of connections if it is low?

im already working on the funds for relay #2

[os11k]

You should use either topology updater or p2p to increase connection to your relay. I personally recommend p2p

[T_rrxx]

ok, ive been using topology updater, how often should the script be used?
ill look into p2p

[os11k]

Coincashew has good guide about topology updater:

Configuring Network Topology

Additionally you need to re-fetch peers once a week or so and then restart relay, so it start using new peers…

[T_rrxx]

ok, thanks for the help!

[natefilm]

Do you have a slot number that you can reference from when you were suppose to mint the block? You can or others can check the logs of the relay to see if there’s anything there.

How did you run leaderlogs to see that you were scheduled?

CNCLI:

GitHub - cardano-community/cncli

[T_rrxx]

above image is the slot number for epoch 371 where supposed minted took place

thanks for the cncli how to!

[natefilm]

Have you restarted your block producer since this happened? You could also try running this on your BP:

curl -s localhost:12798/metrics | grep cardano_node_metrics_Forge

It’ll show if you were actually a scheduled leader.

[T_rrxx]

here is what it returned:

[T_rrxx]

oh and i checked leader logs using the cardano node cli on bp:
#check leader slotes for current epoch (mainnet):
cardano-cli query leadership-schedule
–mainnet
–genesis
–stake-pool-id
–vrf-signing-key-file
–current

[Terminada]

I just created another topic with a troubleshooting example: Troubleshooting why my block got orphaned

Maybe reading through this example will help you troubleshoot what went wrong.

[T_rrxx]

Perfect! Ill be sure to read thru it. thank you!

[kirael]

Hello

I wrote a little guide on how to monitor Peers In connections using Grafana/Prometheus. It’s a simple script, but it works fine ! And you can even set some alarms if Peers number goes down on your Relay

I hope it’ll help !

Guide on how to monitor Peers In connections on Grafana

Hello there o/ While setting up my Grafana Dahsboards a little while ago, i couldn’t find any “Connected peers IN” metrics, out of the box. So i wrote a little and simple script that can do that, and export it to Prometheus every minute. I though someone else might be interested, so i wrote a guide on my pool Blog there : It’s certainly not perfect, but well, it works

[T_rrxx]

thank you so much, ill read thru it!