About my setup:
1 relay node and 1 block producing node.
Node exporter is installed on both the nodes mentioned above.
Grafana is installed on relay node as mentioned here
Following is the output of the ports relay node is listening on :
tcp 0 0 127.0.0.1:12788 0.0.0.0:* LISTEN 76978/cardano-node
Following is the output of the ports core node is listening on :
tcp 0 0 0.0.0.0:12083 0.0.0.0:* LISTEN 602/sshd: /usr/sbin
Grafana status on relay node :
grafana-server.service - Grafana instancehttp://docs.grafana.org
Running prometheus on local machine with following config
scrape_configs:
job_name: “Relay Node”
targets: [“x.x.x.x:20800”]
job_name: “Relay Node Node Exporter”
targets: [“x.x.x.x:9100”]
job_name: “Core Node”
targets: [“x.x.x.x:20800”]
job_name: “Core Node Node Exporter”
targets: [“x.x.x.x:9100”]
job_name: “Relay Node EKG”
targets: [“x.x.x.x:12788”]
job_name: “Block Producing Node Node Exporter EKG”
targets: [“x.x.x.x:12788”]
The job name is added as a label job=<job_name> to any timeseries scraped from this config.
job_name: “prometheus”
metrics_path defaults to ‘/metrics’
scheme defaults to ‘http’.
static_configs:
targets: [“localhost:9090”]
Problems I am facing :
Following command when executed on relay node doesn’t work as mentioned in the document linked above.
curl -s 127.0.0.1:12788/metrics
<!DOCTYPE html>
<html>
<head>
<title>Not found</title>
</head>
<body>
<code>No handler accepted "/metrics"</code>
After I run grafana on my local machine after running prometheus and try to add new datasource I get following error :
Error reading Prometheus: Get “http://localhost:9090/api/v1/query?query=1%2B1&time=1662965276.513 ”: dial tcp [::1]:9090: connect: connection refused
But when I paste the above URL in the browser it works fine with following output :
{“status”:“success”,“data”:{“resultType”:“scalar”,“result”:[1662965233.712,“2”]}}
My local machine a mac book running Big Sur. Firewall settings seem fine as well.
Can you guys please help me understand what am I missing?
os11k
12 September 2022 13:33
2
Hi! @Prateek_Sachdeva !
You are looking on wrong port, for grafana it should not be EKG, but rather prometheus, like here:
"hasPrometheus": [
"127.0.0.1",
12798
]
So if you are using default port you should curl -s 127.0.0.1:12798/metrics
Where did you install Prometheus? On separate machine(not BP and relay)?
In default configuration prometheus metrics are exposed on loopback ip(127.0.0.1), so cardano node metrics will be accessible only locally from those machines. If you want to get those metrics from other servers, then you need to have something like this:
"hasPrometheus": [
"0.0.0.0",
12798
]
But use it with caution, that will expose your metrics to outside world, so please make sure you have proper FW rules.
I’m not sure, but seems you are running Prometheus on separate machine from Grafana, not sure what is idea behind, but for most setups, I personally would recommend to run it on same box.
Hi @os11k
prateek@relay:~$ curl -s x.x.x.x:20800/metrics
Following is the snippet of the testnet-config.json on relay node.
“hasEKG”: 12788,
I have installed node_exporter on my relay & core node. I have installed prometheus on my local machine and I use the following command to access it :
cd /Users/xyz/cardano_testnet/prometheus/prometheus-2.35.0-rc1.darwin-amd64
./prometheus --config.file=prometheus.yml
I am able to run prometheus successfully on my local macbook and I am able to connect to the node exporters. I have attached the screenshot
I am running grafana on my macbook as well and I access using this urlhttp://x.x.x.x:3000/?orgId=1
So to answer your last question - prometheus and grafana are running on same machine. I am really confused what I am missing.
I am not still not able to setup the data source on grafana when I run it on my mac book. Attached screenshots of failure to add datasource to grafana and output of the link which grafana complains about after I try it in the same browser.
os11k
17 September 2022 21:02
4
Hi @Prateek_Sachdeva
In your print screen you have typo:
it is: http://localhost:9090 not locahost !
You missed l between a & h.
