Issue with CoinCashew Guide on Grafana/Prometheus setup

Staking & Delegation Operate a Stake Pool
1

I was following this guide to get Grafana up and running. For some reason, Processor Usage, Disk Usage, Network Usage and Peers are all displaying as “No Data”. I’ve been searching around for a few hours but can’t seem to isolate the issue. Any one knows how to resolve this one?

https://www.coincashew.com/coins/overview-ada/guide-how-to-build-a-haskell-stakepool-node#16-setup-prometheus-and-grafana-dashboard

2

Did u checked the ports… I think 12798 and 9091 or 9090? Did u opened? Also inside the configuration file of each server u will need to replace 127.0.0.1 with 0.0.0.0 and restart the nodes

3

Hi! Yeah, I already did that.
12798 and 9100 opened on both node_exporters.
mainnet-config also updated to 0.0.0.0 and rebooted all nodes.
Still having the same issue. funny thing is that the Memory related graphs are all working, so it’s not like it totally does not work. Just very bizarre that only the Processor Usage, Disk Usage, Network Usage and Peers graphs/fields are not working. Even the local machine does not have that data.

4

Did u checked if the nodes are exporting the metrics?

curl localhost:12798/metrics

5

Here’s the outputs from that:
curl localhost:12798/metrics
rts_gc_par_tot_bytes_copied 967433733008
rts_gc_num_gcs 36203
rts_gc_max_bytes_slop 28545784
cardano_node_metrics_Stat_threads_int 16
cardano_node_metrics_density_real 4.982239239747197e-2
cardano_node_metrics_epoch_int 276
rts_gc_num_bytes_usage_samples 134
cardano_node_metrics_txsInMempool_int 8
rts_gc_wall_ms 29769217
cardano_node_metrics_RTS_gcMinorNum_int 36041
rts_gc_max_bytes_used 3096107920
rts_gc_init_cpu_ms 7
rts_gc_bytes_copied 967433733008
cardano_node_metrics_txsProcessedNum_int 7602
rts_gc_init_wall_ms 27
cardano_node_metrics_RTS_gcLiveBytes_int 5583059464
cardano_node_metrics_mempoolBytes_int 3731
rts_gc_peak_megabytes_allocated 8991
cardano_node_metrics_nodeStartTime_int 1625675612
ekg_server_timestamp_ms 1625705377081
cardano_node_metrics_RTS_gcMajorNum_int 133
cardano_node_metrics_blockNum_int 5950962
rts_gc_par_max_bytes_copied 720495867400
cardano_node_metrics_Stat_cputicks_int 1144311
rts_gc_current_bytes_slop 18424304
cardano_node_metrics_RTS_gcticks_int 937457
cardano_node_metrics_RTS_mutticks_int 206852
rts_gc_cpu_ms 11471348
rts_gc_bytes_allocated 1070419727848
rts_gc_current_bytes_used 2958093840
rts_gc_par_avg_bytes_copied 967433733008
cardano_node_metrics_served_header_counter_int 26891
rts_gc_mutator_cpu_ms 2070384
cardano_node_metrics_Mem_resident_int 7790915584
rts_gc_gc_wall_ms 12679095
rts_gc_mutator_wall_ms 17090122
rts_gc_gc_cpu_ms 9400965
rts_gc_cumulative_bytes_used 351711795448
cardano_node_metrics_slotInEpoch_int 270196
cardano_node_metrics_slotNum_int 34138996
cardano_node_metrics_RTS_gcHeapBytes_int 9427746816

Looks like it’s working. I also tried it on port 9100, and also did curl requests on the other nodes from the grafana node to see if the ports are open on 12798 and 9100. All looks good from that front.

Do you think the issue is with the json file?
https://raw.githubusercontent.com/coincashew/coincashew/master/.gitbook/assets/grafana-monitor-cardano-nodes-by-kaze.json

6

U can try mine for test

https://charity-pool.ro/files/CHRTY_dashboard.txt

7

image
image1822×785 39.8 KB

Doesn’t look like it’s working either.

8

Then… please check the prometheus.yml file… did u updated with the right IP address?

9

IP addresses are all correct. But maybe the config file is crafted wrongly?

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label job=<job_name> to any timeseries scraped from this config.
  - job_name: 'prometheus'

    static_configs:
      - targets: ['localhost:9100']
      - targets: ['<BP_IP>:9100']
      - targets: ['<BP_IP>:12798']
        labels:
          alias: 'SGCO BP'
          type:  'cardano-node'
      - targets: ['localhost:9100']
      - targets: ['<Relay2_IP>:12798']
      - targets: ['<Relay2_IP>:9100']
        labels:
          alias: 'Relay2'
          type:  'cardano-node'
      - targets: ['localhost:12798']
        labels:
          alias: 'Relay1'
          type:  'cardano-node'
10

Asummimg the relay is the grafana server this is my prometheus configuration… the forum changed the format of few characthers so please correct first then restart the prometheus
Also replace 9091 with ur port 9100

  • job_name: ‘BP_cnode’
    static_configs:
    • targets: [‘PRODUCER_IP:12798’]
      labels:
      instance: “Producer”
  • job_name: ‘BP_node_exporter’
    static_configs:
    • targets: [‘PRODUCER_IP:9091’]
      labels:
      instance: “Producer”
  • job_name: ‘Relay_cnode’
    static_configs:
    • targets: [‘127.0.0.1:12798’]
      labels:
      instance: “Relay”
  • job_name: ‘Relay_nexporter’
    static_configs:
    • targets: [‘127.0.0.1:9091’]
      labels:
      instance: “Relay”

Should looks like

image

12

Replacing the earlier post. The solution is here: