Prometheus not fetching any metrics from core to Grafana

Hey there,

i finally find some time again to setup Prometheus and Grafana. I tried it back in december but i failed at the same point today like the last time.

In Grafana i can fetch all metrics from my relays but i dont see any from my core node.

I searched today almost 4 hours for a solution but i dont finde one…

What ive done so far:

  • node exporter running without any failure
  • changed the config.json to 0.0.0.0
  • curl the metrics on the core node - all where showing up
  • changed the prometheus.yml on the “prometheus server”
  • set up ufw rules

I think the problem is that my BP running on a ipv6 and prometheus or grafana cant fetch the metrics from the core node

Greetings
Ben

Hmm, if u type sudo systemctl status prometheu-node-exporter what is the output? and I believe inside configuration file u should not use 0.0.0.0 (IPv4) try with :: (IPv6… didn’t tested)

Good morning,

after restarting the core and the grafana server it now fetching the node metrics from the core. But still not a single cardano related metric.

I tried it with the :: (IPv6) in the mainnet config but after this the cnode dont start .

prometheus-node-exporter.service - Prometheus exporter for machine metrics
     Loaded: loaded (/lib/systemd/system/prometheus-node-exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2022-01-30 16:51:53 UTC; 17h ago
       Docs: https://github.com/prometheus/node_exporter
   Main PID: 849 (prometheus-node)
      Tasks: 27 (limit: 18685)
     Memory: 26.0M
     CGroup: /system.slice/prometheus-node-exporter.service
             └─849 /usr/bin/prometheus-node-exporter --web.listen-address=:12666

so from this port i get the systemhealth related metrics and 12798 should give out the cardano related metrics but i only get this

scrape_duration_secondsscrape_samples_post_metric_relabelingscrape_samples_scrapedscrape_series_addedup

What happens when you try to curl from grafana server? Something like:

curl -s  core-node-ipv4:12798/metrics

Can you try same on your core node?

Hello there,

it happen nothing, no output. Can´t curl with the ipv6. Tried the Cardano Port and the “System health” Port.
i tried it with

curl -S -6 core-node-IPv6:12798/metrics

The strange thing is, that i get the system health over the port 12666 from my core node. not via curl on the terminal but over the Grafana Dashboard.

This is only for listening for external connections right?

  "hasEKG": 12788,
  "hasPrometheus": [
    "0.0.0.0",
    12798

By the way im using the CNCLI if it is important

So this worked from BP and didn’t work from grafana server right?

Could you please show output what you see when you try this from Grafana server.

After some research i used the cURL wrong. need it to wrap it to work.

curl -S -6 'http://[IPv6]:12798/metrics'
curl: (7) Failed to connect to IPv6 port 12798: Connection refused

I checked the ufw status and the port is open for the ip of the relay with the grafana server.

curl -S -6 'http://[IPv6]:12666/metrics'
# HELP apt_upgrades_pending Apt package pending updates by origin.
# TYPE apt_upgrades_pending gauge
apt_upgrades_pending{arch="all",origin="Ubuntu:20.04/focal-updates"} 1
apt_upgrades_pending{arch="amd64",origin="Ubuntu:20.04/focal-updates"} 8
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.024e-06
go_gc_duration_seconds{quantile="0.25"} 7.911e-06
go_gc_duration_seconds{quantile="0.5"} 2.7033e-05
go_gc_duration_seconds{quantile="0.75"} 9.5014e-05
go_gc_duration_seconds{quantile="1"} 0.000366446
go_gc_duration_seconds_sum 3.564344939
go_gc_duration_seconds_count 60292
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 9
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.13.8"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.245576e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.10373235064e+11

...and many more

so port 12666 is reachable but 12798 not.

if u check from relay if it has connectivity on IPv6 and port 12798… it has or not?
on BP you should open 2 ports on FW 12798 and 9100

Unable to connect to remote host: Connection refused

hm… im going to try to delete and renew the ufw ports

yes 12798 and 9100 is just open to the grafana server

nope did not help to renew the ports in ufw, still get the “connection refused” on port 12798

do you have another FW? the BP is a local server?

Yes it´s a local server. Having a FW on my router aswell but ofcourse i opend the port there too.

what confuses me is that the port 12666 is reached but 12798 is not. they are the same settings

hmm did u configured the PF (port forward) on your local router?

yes i did. just checked it again.

check the ports from extern tool, still closed? if yes, then it must be an issue with the FW (blocking)
you can try to disable it just for tests

can’t use this on the server as i only use the terminal version. tried telnet: connection refused.
even if i disable the FW on the router completely for the device.

so permission is not denied so the port should be open but i get refused