Hi - got a question about Grafana: I installed my node and Grafana according to Coincashews guide and it works quite well. The only thing not working is that the KES periods and left days dont show up in grafana. It just states “no data” - how can i troubleshoot this? Has anyone had the same issue? Help would be appriciated!
BRGDS
1 Like
The only time I see no data is when my nodes are down. I’m also using their dashboard.
It’s expecting it’s datasource to be named prometheus all lower case.
Then you could check on the left menu there’s an explore section that allows you to test queries. If you only have prometheus as a datasource it should already be selected, if not there’s a dropdown on top.
On the query section there’s a metric dropdown that allows you to see all the metrics being scraped. It’s organized by job. So I would then check if you see the node metrics, specifically the cardano_node_Forge_metrics.
I checked the metrics and there are quite a lot jobs in it but the Forge jobs are missing… prometheus is also all in small letters so no problem there…
KES periods are available ONLY from BP nodes which have successfully loaded certificates. If you don’t see KEs statiatics on your BP node, then you are in trouble
Check your BP start script for common mistakes:
- check if paths to certificates/key are correct
- check if you don’t miss the line break \ at the end of each line (except the last one)
2 Likes
I think my paths and everything are okay because if I type into my terminal curl localhost:12798/metrics I can see all KES data and it also states 57 periods left. But I cant see it in Grafana. the cardano_metrics_Forge_remainingKES…etc is not showing up in the “edit” page. You know wjat i mean? Im currently at work maybe I can post some pics later
So the node puts out the metric that’s good. But it seems like something’s happening at the prometheus level. I don’t know if you’re doing anything “fancy” with your config, like re-labelling.
Maybe you can check the prometheus web ui directly?
1 Like
Okay here we go, you can see on the pics that there is no KES metric on grafana. I installed prometheus and grafana exactly like stated in coincashews guide:
And I can grep the KES date when I’m in the terminal, thats no problem.
Can you double check that in your Grafana settings you have pointed the data source to your BP Node and not to relay.
If you have more than 1 node running, check your node config files if each node has a different port for EKG and Prometheus data, check also Prometheus config file if you are collecting the data from all the nodes you are running
cat /etc/prometheus/prometheus.yml
check if your cardano-node config file ports are correct in prometheus config file.
I checked it all - it’s all correct. Which ports do I have to open in the firewall? Only 3000?
I guess it all depends what your setup is like. Is everything on 1 server?
do you run several nodes on the server?
i checked the guide you were referring to, and I see that they are using not the default Prometheus ports but 12700 and 12701, are you sure you have in your mainnet-config file the same ports set for prometheus?
here is what you have to search in your mainnet-config.json file
“hasEKG”: 12788,
“hasPrometheus”: [
“0.0.0.0”,
12798
],
in prometheus ( /etc/prometheus/prometheus.yml )
static_configs:
- targets: [‘127.0.0.1:12798’]
labels:
especially check the BP nodes ports, from what i see in the guide and your issue - it looks to me that prometheys does’t have access to BP nodes EKG data - that’ s why you don’t see KES metrics on Grafana.
as you reported, you see the KES metrics when you do:
curl localhost:12798/metrics | grep KES
then check that port 12798 is written in the targets in your prometheus config file.
let me know if this was helpful.
Lauris
1 Like
I had the same issue, while using the “official” grafana dashboard (the one linked as a json file in the pool school gitbook).
I found the problem was in the query in the panel for KES periods. I edited the panel’s query to scrape the correct metric (you’ll be able to find this in prometheus, I am on mobile and cannot remember it by heart). And it all worked a treat.
Let me know if this helps, if I get to a machine before you get this working I’ll post the correct metric for KES periods.
Cheers,
A
Grafana:
cardano_node_Forge_metrics_currentKESPeriod_int
Yep, that’s the one.
I’ve also scraped remaining KES periods in a separate panel, and given it thresholds for when it gets close to renewal. I find that helpful for visual reminding.
Thanks @needshelps