Node Relays Swapperooo having trouble with Graphana and Prometheus

Staking & Delegation Operate a Stake Pool
1

I had one of those pre best practice setups where the relays were on the same server as the block producing node. I recently split everything up so now the block producing node is on it’s own server, and both relays are on their own independent servers. I have everything all linked up and working fine, no issues there that I am able to see* so as of yet.

I updated the yml with the appropriate targets. Made sure everything is matching in the config.json’s. I am trying to run prometheus from the 2nd relay node. The block producing node and relay node 1 both have node-exporter running.

Expected result: See the node and both relays the same as I did pre server swapperooo.

Actual result: It shows the old setup plus the relay 2 three times. (I did try changing the alias of the relay, not sure if that has something to do with it showing multiple times if this is a caching issue.) Could this be a caching issue? Is there a clear cache button hidden somewhere I am not seeing?

Anyone ran into this or similar issue before? Any thoughts or assistance would be greatly appreciated.

I know I can get these all running on their own independent graphana / prometheus setups but I feel like I should be able to tie these all together to see them in one chart like I did before.

Thanks!

2

UPDATE: I did a complete uninstall and purge of prometheus and graphana from relay 2, now all the old data and duplicates are gone. Still not pulling in data from the other relay and block producing node.

3

UPDATE 2:

This is what I have in my prometheus.yml. I notice that if I remove one of the targets I can get two nodes to display in prometheus. It doesn’t seem to matter which one I remove so long as localhost is the one running prometheus. I feel like it’s staring me in the face but for what ever reason I can’t figure this out.

Any thoughts or suggestions based off this?

global:
scrape_interval: 15s

external_labels:
monitor: ‘codelab-monitor’

scrape_configs:
– job_name: ‘prometheus’

static_configs:
  - targets: ['localhost:9100']
  - targets: ['< block producers ip >:12700']
    labels:
      alias: 'node_core'
      type:  'cardano-node'
  - targets: ['< relay 1s ip >:12701']
    labels:
      alias: 'node_r1'
      type:  'cardano-node'
  - targets: ['localhost:12702']
    labels:
      alias: 'node_r2'
      type:  'cardano-node'