Please Share Your Grafana Dashboards For Cardano-Node

Staking & Delegation Operate a Stake Pool
,
61

Does anyone know which config tracing flags need to be enabled to see the cardano_node_metrics_txsProcessedNum_int and cardano_node_metrics_txsInMempool_int reported in Prometheus? I’m not seeing either of these values.

62

From config file

TraceMempool=true

But I understood that if u don’t have enough resources it’s better to keep them off (at least on relays)

63

Thanks for the quick reply! i have that option enable, but i’m not finding either reported?

  "TraceBlockFetchClient": true,
  "TraceBlockFetchDecisions": true,
  "TraceBlockFetchProtocol": false,
  "TraceBlockFetchProtocolSerialised": false,
  "TraceBlockFetchServer": false,
  "TraceChainDb": true,
  "TraceChainSyncBlockServer": false,
  "TraceChainSyncClient": false,
  "TraceChainSyncHeaderServer": false,
  "TraceChainSyncProtocol": false,
  "TraceDNSResolver": true,
  "TraceDNSSubscription": true,
  "TraceErrorPolicy": true,
  "TraceForge": true,
  "TraceHandshake": false,
  "TraceIpSubscription": true,
  "TraceLocalChainSyncProtocol": false,
  "TraceLocalErrorPolicy": true,
  "TraceLocalHandshake": false,
  "TraceLocalTxSubmissionProtocol": false,
  "TraceLocalTxSubmissionServer": false,
  "TraceMempool": true,
  "TraceMux": false,
  "TraceTxInbound": false,
  "TraceTxOutbound": false,
  "TraceTxSubmissionProtocol": false,
  "TracingVerbosity": "NormalVerbosity",
  "TurnOnLogMetrics": true,
  "TurnOnLogging": true,

$ curl -s 127.0.0.1:12798/metrics | sort
cardano_node_metrics_blockNum_int 5633086
cardano_node_metrics_connectedPeers_int 5
cardano_node_metrics_density_real 4.837300965220702e-2
cardano_node_metrics_epoch_int 261
cardano_node_metrics_Mem_resident_int 2155229184
cardano_node_metrics_nodeStartTime_int 1619288600
cardano_node_metrics_RTS_gcLiveBytes_int 1949680200
cardano_node_metrics_RTS_gcMajorNum_int 18
cardano_node_metrics_RTS_gcMinorNum_int 938
cardano_node_metrics_RTS_gcticks_int 2277
cardano_node_metrics_RTS_mutticks_int 2169
cardano_node_metrics_served_header_counter_int 67
cardano_node_metrics_slotInEpoch_int 334738
cardano_node_metrics_slotNum_int 27723538
cardano_node_metrics_Stat_cputicks_int 4447
cardano_node_metrics_Stat_threads_int 13
64

These are mines, if can help you

  "TraceBlockFetchClient": true,
  "TraceBlockFetchDecisions": true,
  "TraceBlockFetchProtocol": true,
  "TraceBlockFetchProtocolSerialised": true,
  "TraceBlockFetchServer": true,
  "TraceChainDb": true,
  "TraceChainSyncBlockServer": true,
  "TraceChainSyncClient": true,
  "TraceChainSyncHeaderServer": true,
  "TraceChainSyncProtocol": true,
  "TraceDNSResolver": true,
  "TraceDNSSubscription": true,
  "TraceErrorPolicy": true,
  "TraceForge": true,
  "TraceHandshake": true,
  "TraceIpSubscription": true,
  "TraceLocalChainSyncProtocol": true,
  "TraceLocalErrorPolicy": true,
  "TraceLocalHandshake": false,
  "TraceLocalTxSubmissionProtocol": true,
  "TraceLocalTxSubmissionServer": true,
  "TraceMempool": true,
  "TraceMux": false,
  "TraceTxInbound": true,
  "TraceTxOutbound": true,
  "TraceTxSubmissionProtocol": true,
  "TracingVerbosity": "NormalVerbosity",
  "TurnOnLogMetrics": true,
  "TurnOnLogging": true,
65

well, must be something else entirely… i matched my settings exactly and still not seeing those values being exported. very strange. is it possible that some of these other configuration options impact these values?

  "defaultBackends": [
    "KatipBK"
  ],
  "defaultScribes": [
    [
      "StdoutSK",
      "stdout"
    ]
  ],
  "hasEKG": 12788,
  "hasPrometheus": [
    "0.0.0.0",
    12798
  ],
  "minSeverity": "Info",
  "options": {
    "mapBackends": {
      "cardano.node.metrics": [
        "EKGViewBK"
      ],
      "cardano.node.resources": [
        "EKGViewBK"
      ]
    },
    "mapSubtrace": {
      "cardano.node.metrics": {
        "subtrace": "Neutral"
      }
    }
  },
  "rotation": {
    "rpKeepFilesNum": 10,
    "rpLogLimitBytes": 5000000,
    "rpMaxAgeHours": 24
  },
  "setupBackends": [
    "KatipBK"
  ],
  "setupScribes": [
    {
      "scFormat": "ScText",
      "scKind": "StdoutSK",
      "scName": "stdout",
      "scRotation": null
    }
  ]
66

But u restarted the node right?
How is the env file ? Did u modified something inside?

67

Question: ur nodes are synced and also have IN PEERS?

68

i have 5 peers out only one in which i guess is my core node? I did restart the relay (testing these configuration changes on my relay first). but i see the trans count is not shown using the liveview.sh script either?

image

69

Ok, u have just 1 IN peer which probably is ur BP; that’s way u don’t see tx… did u ran the topology updater script once/hour (on Relay only)?

Cheers

70

I’m actually not running the topology updater. I did not install using CNTools. So I manually update the topology and keep the list down to trusted endpoints.

71

but shouldn’t my endpoint be discoverable automatically?

72

Nope, for the moment u need to run the script.
U can find it in coincashew guide

73

Hoping to use your Grafana dash (USONE). Got it imported but it seems like a few of the metrics arent populating like: Core + Relay node system health (not showing CPU, RAM, SWAP, or DISK usage), system load, time synchronization, and memory utilization.

Has anyone else had this issue / figured it out?

Thanks in advance!

image

74

switch from alias to use instance. ie 'alias=“block-producer-node” to instance=“xx.xx.xx.xx” and you should start getting metrics.
Congrats on the blocks assigned btw!

75

Thanks! Will try it later today. Btw, did not get assigned any blocks. For some reason it shows 4 on the dash always but 0 on leaderlog

76

tried switching to instance but still not working. This is what my CPU metric looks like:

100 - (avg by(alias) (irate(node_cpu_seconds_total{mode=“idle”,instance=“xx.xx.xx.xx”}[1m])) * 100)

where xx.xx.xx.xx is my Block prod IP

Still nothing though

77

Hi,
Try this:
100 - (avg(irate(node_cpu_seconds_total{mode=“idle”, instance=“yourip:yourport”}[30m])) * 100)

and ensure you have node explorer installed and the ports opened.

78

Does any one have in their dashboard a way to monitor tip difference?

79

sorry, was away for a bit. Tried the instance=“yourIP:port” but no luck. Node exporter is installed + working and the ports are open. Wondering if it might have something to do with my mainnet-config? I’m getting some of the dash being populated just not all of it

80

Yes, you have to enable many of the fields. Example below. But it will become very resource intensive by doing so. Glad you were able to get some to of the dashboards to work =)

“TraceBlockFetchClient”: true,
“TraceBlockFetchDecisions”: true,
“TraceBlockFetchProtocol”: true,
“TraceBlockFetchProtocolSerialised”: true,
“TraceBlockFetchServer”: true,
“TraceChainDb”: true,
“TraceChainSyncBlockServer”: true,
“TraceChainSyncClient”: true,
“TraceChainSyncHeaderServer”: true,
“TraceChainSyncProtocol”: true,
“TraceDNSResolver”: true,
“TraceDNSSubscription”: true,
“TraceErrorPolicy”: true,
“TraceForge”: true,
“TraceHandshake”: true,
“TraceIpSubscription”: true,
“TraceLocalChainSyncProtocol”: true,
“TraceLocalErrorPolicy”: true,
“TraceLocalHandshake”: false,
“TraceLocalTxSubmissionProtocol”: true,
“TraceLocalTxSubmissionServer”: true,
“TraceMempool”: true,
“TraceMux”: false,
“TraceTxInbound”: true,
“TraceTxOutbound”: true,
“TraceTxSubmissionProtocol”: true,
“TracingVerbosity”: “NormalVerbosity”,
“TurnOnLogMetrics”: true,
“TurnOnLogging”: true,