Any SENSIBLE way to get peer connection metrics directly from a node? (Incoming AND Outgoing)

I see lots of scripts (liveview - im looking at you) screwing around with the output of netstat/ss and established connections and pretending its real monitoring, but thats lame, since TCP established does NOT mean a real actual connection with real actual Cardano information flowing.

What I want is to be able to query a node for

number of incoming connections (actually talking cardano protocol)
number of outgoing connections (actually talking cardano protocol)
list of incoming IP addresses (actually talking cardano protocol)
list of outgoing IP addresses (actually talking cardano protocol)

I cant see anything even close in my prometheus metrics, is there something I can turn on in a config file?

maybe more too, but lets start with those.

thanks for any suggestions.

edit - to discourage more gliveview suggestions. gliveview is fine for someone who wants to sit and watch the screen all the time. i do not wish to do that. i need to script my checks. end of story.

2 Likes

use gLiveView and u can see the IN/OUT number connections + the IPs of the peers

screwing around with the output of netstat/ss and established connections and pretending its real monitoring, but thats lame, since TCP established does NOT mean a real actual connection with real actual Cardano information flowing.

What do u mean here? Can u elaborate?

Cheers,

2 Likes

established TCP connection does not mean that its a genuine cardano peer.

gLiveview works in 2 ways, both seem to start by guessing the number of peers using the output of the OS utility ‘ss’ (similar to netstat) which is messy at best, and inaccurate at worst.

…then, (in some cases), handing the output of the aforementioned ss off to cncli to get more info. which means that for any of those guessed peers which isnt really a peer, cncli is called and tries to start gibbering at it for a ping.

Its a mess for something that should be grabbable directly out of the cardano-node metrics.

I want to get the number of REAL peers. Not the number of connections to the port in question.

I’ve read that it should be possible to get the built-in metrics to deliver cardano_node_metrics_connectedPeers_int - which should show at least 1 of the number I want, but I cant get it to appear in my list of metrics no matter which options i turn on.

please dont start defending gLiveview. If it works for you and you’re happy with it - carry on.
** I’m not here to change your mind. I’m just looking for ways to make my own monitoring better.**

2 Likes

I am not defending… just that gLive shows me the correct numbers… if I go on BP (where the topology is manually configured) on gLive I see the correct number of peers and IPs configured by me … :thinking:

Forgive me… perhaps I don’t understand it… I’m not an english man … I’ll let other people to comment

Cheers,

1 Like

I suspect if you telnet to your own node you might see the number of peers rise.

…and yet an empty telnet connection is not a peer.

I dont care about how many TCP connections I have. I want to know how many real active peers I have, and I dont want to have to do it by iterating all my established connections with an OS tool and hand them all off to a 3rd party app to check if there is a node at the other end or not.

1 Like

I don’t think… but I will check right now… you made me curious

Well it’s not true… glive it’s showing only connections with other cardano node peers

You can press P (on gLive) for IPs + port infos for each peer

2 Likes

kudos to you for checking, were you telnetting from localhost? it looked like it might filter those out. i got bored of reading through it.

…but it still iterates the output of ‘ss’ (or lsof) and (eventually) passes it to ‘cncli’ which is messy.

1 Like

From home to my relay from cloud

perhaps its only if you have a port listening on the port your connection is sourced from.

anyway. all off topic.

i need my metrics directly from my node. and i think ive just worked out 1 of the numbers i need.

1 done. 3 to go.

Yes. I feel your pain.

I wrote my own script to look at this but it also uses ‘ss’ and is susceptible to the problem you identify using telnet. I don’t use any of the gLiveview, cntools scripts because they are overly complex and I need to understand what the scripts are doing.

Some good news:

When you switch your node to P2P mode, it provides much more statistics on connected peers. It logs changes with messages like this:

[relay1:cardano.node.ConnectionManager:Info:332] [2022-02-20 00:33:19.92 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 2, unidirectionalConns = 66, inboundConns = 19, outboundConns = 50})                                     
[relay1:cardano.node.InboundGovernor:Info:332] [2022-02-20 00:33:19.92 UTC] TrPromotedToWarmRemote (ConnectionId {localAddress = 192.168.27.7:2700, remoteAddress = 52.58.89.148:8381}) (OperationSuccess (InboundIdleSt Unidirectional))                                         
[relay1:cardano.node.InboundGovernor:Info:332] [2022-02-20 00:33:19.92 UTC] TrInboundGovernorCounters (InboundGovernorCounters {coldPeersRemote = 0, idlePeersRemote = 0, warmPeersRemote = 14, hotPeersRemote = 5})

You also get cardano_node_metrics like:

cardano_node_metrics_connectionManager_incomingConns{}

I can’t wait until P2P is fully rolled out.

1 Like

is that only available in p2p?

thanks

Only in P2P mode unfortunately it seems.