Please Share Your Grafana Dashboards For Cardano-Node

I would be wary of using docker for highly private operations. We test these images routinely for work on HPCs and they have multiple security flaws. But that’s just my opinion. In reality you are experiencing first hand why an incomplete container has limitations.

I wouldn’t call CNCLI mandatory. I installed all software manually and run most things as they are intended standalone. For monitoring your performance you can rely on Prometheus. Having said this, knowing your slot allocation in advance is handy to time your maintenance. An alternative to CNCLI for this purpose is papacarp’s leaderlogs.sh, but last i checked the api was not working properly.

If you’re handy with coding you might consider re scripting either of those tools to suit your needs: they are both open sources on GitHub. Also, it may be worth asking this question as an issue on CNCLI’s repo, the devs may have better answers than mine.

Let me know if you need anything else to progress this.

Regards,

A

1 Like

Hi! Thank you for explaining!

I have a bit different view on docker. :slight_smile:
So there are no way to get slot allocation in advance/leadership chances from grafana, right?

I will try to run cncli from host, maybe I will be able to run it.

No problem,

No, the Prometheus-grafana duo are only giving you metrics exposed by your node software. To find your leaderlog you need to have a working, synced copy of the db as a single file and query the ledger with your pool id, its total stake etc.

You should be able to run the necessary commands by installing CNCLI. I understand the installation itself is problematic from within the image, but you should be able to run this on the host. The sync command may be throwing an error if you sync to the wrong server, but that should be an easy fix. From there you should simply end up with a local .db file and query it for leadership.

Let me know if you get it to work.

Cheers,

A

Hi @Adrem ! Thank you for detail explanation, in this case, yes CNCLI will be possible to run from host, I will try it a bit later.

Can you clarify one more thing, currently pool performance in my dashboard grafana are calculated like this:

(cardano_node_metrics_Forge_forged_int{alias=“producer”} / cardano_node_metrics_Forge_node_is_leader_int{alias=“producer”} * 100)

Is it correct way? I have run my pool in testnet for 2 days already and it is still shows “No Data”, what is a bit alarming. Thank you!

Hi,

No problem at all. I am currently on mobile and cannot verify your code 100%, but it looks correct. You are dividing the number of blocks minted by the assigned slots, times 100. This is a decent way of measuring performance.

The fact that is still on 0 should indicate that either you have had no slots assigned (and therefore no blocks) or that for some reason you didn’t mint a block in the assigned slot.

In Cardano’s PoS the number of assigned slots is proportional to your pool’s stake. Can i ask you how much that is on testnet?

Also, please note that once your nodes are up and running you should connect them to 10-15 peers each (friendly pools or using topology updater) and see a steady flow of txns through your nodes (typically 1 per second or so).

Regards,

A

Hi @Adrem

I have 1498 Active Stake, from where 1000 is pledge and rest is just on same wallet. I can’t even find my pool in Daedalus Testnet. I see transactions in grafana:

Seems peers are ok too, I ran “relay-topology_pull.sh” script and here I see my peers in Grafana:

image

test-producer-backup, my standby producer shows 2 peers. As far as I understood relay must have correct config and currently it points to test-producer only + other peers, what I got running “relay-topology_pull.sh”, so all should be good here. Or not?

Seem test-producer & test-producer-backup are both connected to relays, but only test-producer has transactions going. So nothing to worry about that part? Or I should not allow my test-producer-backup to connect towards relays in case if I don’t want him to produce blocks

1 Like

@Adrem
I hope you forgive me for spamming your inbox.

I got cncli working and actually it works quite well.

so this is result of running ./cncli leaderlog xxxxxxxxxxxxxx --ledger-set current


{
  "status": "ok",
  "epoch": 160,
  "epochNonce": "695cfb14d621541b2b2319d839659041f6bd57044762a3dbf564d404ad0f604c",
  "epochSlots": 0,
  "epochSlotsIdeal": 0.0,
  "maxPerformance": null,
  "poolId": "e82e9f7930bcf2811111111111111111111112222222222",
  "sigma": 0.0,
  "activeStake": 0,
  "totalActiveStake": 14315783994496876,
  "d": 0.0,
  "f": 0.05,
  "assignedSlots": []
}

What it shows? Seems I didn’t get any slots assigned to me yet. Right? This means in nearest future I don’t have any chance to Forge a block right?

Sorry for such an embarrassing question, but could you please explain, how this assigned slots and forged blocks are related? As far as I understand, first I got a slot and then in that time I can forge a block and then I will get reward for this. Something like this?

I’m bit overwhelmed with all new information what I’m getting for couple of the last days.

1 Like

This is correct, your BP should be only connected to your relays, which in turn connect to the wider network;

Your backup BP should be running “passive” eg with no keys as having two BPs registered and minting simultaneously is a big no-no. I would have the backup configured and ready to go in case of need, maybe even with an automation to spin it up when necessary, but not running in parallel. this will save you ongoing server costs as well (provided your backup is in a discrete VM, which it should be or it might as well not exist);

No problem at all @os11k , happy to help;

Great;

The output from leaderlog shows you have no active stake "activeStake": 0, no assigned slots "assignedSlots": [] and an expected null performance "maxPerformance": null,

Just to clarify on the reasons why this may be and answer your question above:

My pool on mainnet [RABIT], currently has 400K $ADA staked to it. It’s expected performance is 0.4 blocks per epoch (5 days). This means essentially 4 blocks every 10 epochs. To consistently be assigned at least one slot to mint into every epoch, the stake would have to be 3-4 times higher (eg 1.2-1.6M $ADA). This is not to say that the pool is then GUARANTEED to mint one block per epoch, but the expectation is there and over time it will average that figure (eg some epochs it will be assigned 2 slots, some none, etc). You can see the performance of my pool here (to get a feel for what I mean): [RABIT] Rabbitholepools | Cardano Staking

So essentially, with 1.5K tADA staked, your sigma is so low that being assigned a slot may take months. I and other SPOs had the same issue on Testnet about a year ago when I started: the faucet just didn’t provide enough funds to properly evaluate performance. You may be able to apply for more tADA or ask fellow Testnet operators to stake to your pool just for testing. I and others skipped this step once we were satisfied that the pools were technically sound and moved to mainnet. Naturally back then $ADA was at 0.1 AUD, so I managed to get enough delegation to have a small pool. These days it may be a tad harder given that $ADA is now at 3 AUD (more capital required).

One thing worth noting is that both IOG and CF have a rotation program to delegate 3M and 15M $ADA every three months to pools that meet certain criteria. You can find more info about these programs here:

IOG: Cardano stake pool operator delegation – a new round - IOHK Blog
CF: Cardano Foundation announces its delegation methodology - CLOSED - #210 by Alexd1985

Let me know if you need more help to get your operation going.

Best regards,
A

PS: the reason why your active stake still shows 0, instead of 1498 is possibly because delegation preferences take 2 epochs to come into play.

PPS: if you don’t see your pool in Daedalus testnet, you may need to wait (the metadata server is notoriously slow) or you may have botched the registration (in this case try to re-register). You may also be able to check your pool here: https://pooltool.io/ (down the bottom left is the button to switch to Testnet view).

Hi @Adrem ! Thank you a lot for detailed explanation!

I see my pool in testnet.adatools.io:

https://testnet.adatools.io/pools/e82e9f7930bcf28a15369de34d4f5c32adad5cbd0ab050f88b52ce8b

And all info seems to be correct, like website and ticker.

But if I look here:

All data is missing, like ticker, description and etc. Did I messed up my pool set up?

AFAICT your pool’s data are nearly the same on adapools and pooltool. The ticker may take a while to sync with the smash server.

I would run the nodes as they are for a couple of epochs to wait for the stake to appear as active. See what the data look like then. Alternatively you can try to re register, just without paying the fee. This is something you will do anyway in the future when you change things like your pledge etc.

It seems to me you’re doing everything the right way, slow and steady. This is a plan for the long run, keep up the good work👍

Ping me here or on my pool’s contacts if you want to chat more.

Best regards,

A

Hi @Adrem ! Thank you a lot for your help!

Probably I will leave it as it is for now.

P.S. If I will not succeeded to build my own pool, I know a pool where I will stake my ada and it is RABIT.

1 Like

Hi @os11k,

no worries at all. Let me know if you require further assistance during your testing phase or when you are ready to go mainnet!

Best regards,

A

2 Likes

Hello Rodrigo,

How did you get your Core node system health and Relay 1 system health to works?

Would you mind sharing your prometheus.yml file?

Thanks!

Probably everybody is aware of this dashboard, but I personally find it one of the best to check my server resources, it just work out of the box and personally used by me in several projects, this is developed by unknown person to me so big kudos to that person and maybe somebody in cardano community besides me will find it useful too

3 Likes

I just created simple dashboard to show prices from Kraken in grafana, so maybe somebody will find it useful.

I would assume you already did this:

or this, what is same but form me it was much more easier to understand:

So requirements is that we have node exporter reading metrics from some directory in our case lets it be /adapools/stats and our script will be located in /adapool directory, so node exporter should be configured with something like this: --collector.textfile.directory=/adapools/stats

Here is a code to script:

It is quite easy to update with your favorite crypto prices if needed

And we need to add this crontab entry, so our price is updating every minute:

* * * * * /adapools/price.sh

After this, you should be able to get prices in grafana:

image

In my examples following metrics will be available: adaeur, adausd, btcusd & ethusd

And then you can create a dashboard with all of this data. if somebody needs, I can share my dashboard with prices, but this is super easy to create and I doubt anybody needs this. :slight_smile:

4 Likes

Sharing some dashboards and accompanying guides I’ve made up, any feedback is appreciated.

v1: PANL - Dashboards
v1 dashboard: https://gist.githubusercontent.com/rp5210/22223fd4dd2d7b9d452a81d737e6fcda/raw/e2ae0933664acef707f776b704f6cfc475930e9f/dashboard.json

v2: Welcome / Introductions - Grafana monitoring setup for Cardano stake pool
v2 dashboard: https://gist.githubusercontent.com/rp5210/90810787ba696494803f2c0ecb59fac8/raw/be86a6658f428c7b46ce9f8a98aae4637babce82/dashboard-v2.json

2 Likes

The PANL one is pretty cool! Thanks for sharing! Unfortunately it’s forcing me to install Prometheus in each node which is a bad practice. Prometheus is already a pretty powerful time series DB and a dashboard should access the necessary information from a single data source. I’ll see if I can rework it a bit to access a single data source with multiple targets (bp, relayN, …)

2 Likes

Easily possible, I separated them out in the guide in an attempt to make it easier to follow for those who may be new. You should be able to adjust the scraper to pull data from each node with different aliases, something like the following:

  # The job name is added as a label job=<job_name> to any timeseries scraped from this config.
  - job_name: 'prometheus'

    static_configs:
      - targets: ['localhost:9100']

        labels:
          alias: 'relaynode1'
          type:  'cardano-node'

      - targets: ['<relay node 2 public ip address>:9100']

        labels:
          alias: 'relaynode2'
          type:  'cardano-node'
      - targets: ['<block producer public ip address>:9100']

        labels:
          alias: 'block-producer-node'
          type:  'cardano-node'
     - targets: ['localhost:12798']
        labels:
          alias: 'relaynode1'
          type:  'cardano-node'

     - targets: ['<relay node 2 public ip address>:12798']

        labels:
          alias: 'relaynode2'
          type:  'cardano-node'

     - targets: ['<block producer public ip address>:12798']
        labels:
          alias: 'block-producer-node'
          type:  'cardano-node'

EDIT: @slux83 I’ve updated both sets of documentation to use a single data source and uploaded a new dashboard JSON based on your feedback.

Thanks! I’ll give it a try for sure :slight_smile:

Sharing my Grafana dashboards / monitoring setup with guides that I made. Hopefully it will help somebody with their initial setup. If you have any questions or run into trouble, let me know, I’m happy to help you out.