After the upgrade (using the Guild scripts), a number of node-based metrics are no longer being reported.
As a primary example the cardano_node_BlockFetchDecision_peers_connectedPeers_int is not returning data.
I’m using Prometheus and Grafana with the standard SKY Core Node dashboard installed. The data sources are confirmed as working (many other stats showing).
Further, the gLiveView shows connected peers.
So I’m convinced everything with the relay node is fine, however am concerned that these metrics are not being shown in the monitoring tools.
I’m tipping the upgrade has changed something. I’m just not sure what, or if there is a setting / param that I am missing somewhere.
Would greatly appreciate any assistance.
from the release notes:
Stake Pool Operators (SPOs) and Exchanges should take note that the metric namespace has undergone consolidation, so all metrics now reside in cardano.node.metrics :
- cardano.node.Forge.metrics.* → cardano.node.metrics.Forge.*
- cardano.node.ChainDB.metrics.* → cardano.node.metrics.ChainDB.*
- cardano.node.BlockFetchDecision.connectedPeers → cardano.node.metrics.connectedPeers
The node configs require no changes, but allow dropping entries that became redundant: wherever cardano.node.Forge.metrics.* , cardano.node.ChainDB.metrics.* or cardano.node.BlockFetchDecision.connectedPeers were mentioned, those entries can be removed. Only cardano.node.metric needs to remain. Please see #2281 for further details.
Modify ur metrics in grafana pannel like this
Old
cardano_node_BlockFetchDecision_peers_connectedPeers_int
New
cardano_node_meterics_connectedPeers_int
PS: new format: cardano_node_metrics_xxxx
Cheers,
2 Likes
Amazing and thankyou! Yes I was so close, and hadn’t seen these metrics available in the options!
Appreciate the very quick reply guys!
New
cardano_node_meterics_connectedPeers_int
Shouldn’t this be:
cardano_node_metrics_connectedPeers_int
1 Like
Hey everyone, I had the same problem and was able to get the fields fixed except 1: uptime. The key in it seems to follow the newer convention: cardano_node_metrics_upTime_ns I’ve tried a lot of different values but nothing has worked. Does anyone know what the correct key is? Also, what would I reference to get this information for the future?
I use this one, but is the uptime of vps: (time()-node_boot_time_seconds)
Hi there, I just recently updated the dashboard and for the uptime metric I use this:
Add the following metric:
- time() - cardano_node_metrics_nodeStartTime_int
Add the following override:
- Fields returned by query = A
- Add override property = Unit
- Option = duration (d hh:mm:ss)
You should see something like the following:
core-node-0
8 d 05:05:40
relay-node-0
2 d 07:26:58
Both solutions work great!!! Thank you both for your solutions. Love this dashboard
Am i the only one which is missing some metrics?
For example i cant find “cardano_node_meterics_connectedPeers_int”
All i get when calling http://localhost:12798/metrics is
rts_gc_par_max_bytes_copied 17933718304
cardano_node_metrics_blockNum_int 5465808
rts_gc_num_gcs 15654
cardano_node_metrics_operationalCertificateExpiryKESPeriod_int 248
rts_gc_mutator_wall_ms 19120737
rts_gc_gc_cpu_ms 44935
rts_gc_par_avg_bytes_copied 22327027600
cardano_node_metrics_RTS_gcLiveBytes_int 1916809768
rts_gc_init_cpu_ms 1
cardano_node_metrics_RTS_gcMinorNum_int 15112
cardano_node_metrics_delegMapSize_int 351069
rts_gc_bytes_allocated 465588156856
cardano_node_metrics_epoch_int 253
rts_gc_num_bytes_usage_samples 542
cardano_node_metrics_currentKESPeriod_int 187
rts_gc_current_bytes_slop 93162968
cardano_node_metrics_RTS_mutticks_int 22990
cardano_node_metrics_Forge_node_not_leader_int 18954
cardano_node_metrics_Stat_cputicks_int 27486
rts_gc_max_bytes_slop 6970552
rts_gc_par_tot_bytes_copied 22327027600
cardano_node_metrics_remainingKESPeriods_int 61
cardano_node_metrics_RTS_gcMajorNum_int 542
cardano_node_metrics_slotNum_int 24277969
rts_gc_cumulative_bytes_used 9140410056
cardano_node_metrics_slotInEpoch_int 345169
ekg_server_timestamp_ms 1615844261952
rts_gc_gc_wall_ms 22856
cardano_node_metrics_operationalCertificateStartKESPeriod_int 186
cardano_node_metrics_Mem_resident_int 3192479744
rts_gc_mutator_cpu_ms 229908
cardano_node_metrics_nodeStartTime_int 1615825120
rts_gc_peak_megabytes_allocated 3020
rts_gc_init_wall_ms 1
cardano_node_metrics_utxoSize_int 1546378
rts_gc_max_bytes_used 1034263648
rts_gc_bytes_copied 22327027600
rts_gc_wall_ms 19143593
rts_gc_current_bytes_used 1916809768
rts_gc_cpu_ms 274843
cardano_node_metrics_Stat_threads_int 13
cardano_node_metrics_RTS_gcticks_int 4493
cardano_node_metrics_density_real 4.77516912057302e-2
cardano_node_metrics_Forge_forge_about_to_lead_int 18954
do i have to change mainnet-config.json to get more metrics?
Btw i’m running the docker image inputoutput/cardano-node which is on version 1.25.1
How is ur metric defined in grafana for
“cardano_node_meterics_connectedPeers_int”
Cheers,
i tried just
cardano_node_metrics_connectedPeers_int{alias="core"}
other metrics like
cardano_node_metrics_epoch_int{alias="core"}
do work as they are contained in the metrics endpoint
but i think it doesn’t matter since the metrics endpoint does not output it, and even prometheus can’t find it:
I updated the metrics in configuration file and I kept there
"cardano.node-metrics": [
"EKGViewBK"
],
"cardano.node.metrics.connectedPeers": [
"EKGViewBK"
],
.....
Do u have it?
mh the config from
https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/
looks like that
should i change “cardano.node.metrics” to “cardano.node-metrics” ?
Nope, my config is old one adapted… try to add only the connected peer
and restart the node, try first on relay
ok i’ve added the part with cardano.node.metrics.connectedPeers but that didn’t change anything
i also tried adding EKGViewBK to defaultBackends, but that didn’t help either.
i also tried to screw up my config, to ensure that the node is reading the file, which lead as expected to the output: InvalidYaml
Then, I don’t know why ur nodes not exporting these metrics
1 Like
Anyway thanks four your help, are you running a node in version 1.25.1 which is outputting these metrics?
Would you mind showing me your whole config.json?