P2P - why producer has many peers?

Jun_Sun · 16 July 2023 17:12

I have set up a preprod testing pool (NSPPP) with p2p network. It seems running fine. To my surprise, though, I found the producer node has many peers. ( See screenshot attached.) Specifically it has 20 hot peers, 50 outgoing and 1 incoming!

Shouldn’t it only have 1 connection to the relay node? Did I configure it wrong?

Below is my topology for producer and relay. Thanks!

producer:
{
  "localRoots": [
    {
      "accessPoints": [ { "address": "10.197.172.61", "port": 3001 } ],
      "advertise": false,
      "valency": 1
    }
  ],
  "publicRoots": [],
  "useLedgerAfterSlot": 4642000
}

relay:
{
  "localRoots": [
    {
      "accessPoints": [{"address": "10.197.172.165", "port": 3001 }],
      "advertise": false,
      "valency": 1
    }
  ],
  "publicRoots": [
    {
      "accessPoints": [
        {
          "address": "preprod-node.world.dev.cardano.org",
          "port": 30000
        }
      ],
      "advertise": false
    }
  ],
  "useLedgerAfterSlot": 4642000
}

Zyroxa · 16 July 2023 18:11

Just disable p2p on your BP in the config file.

Terminada · 17 July 2023 22:44

It is no problem to leave your block producer in P2P mode and there may be some advantages in doing so. For example, the duplex connections that P2P mode enables may be preferable???

You simply need to ensure your block producer doesn’t seek other relays to connect with other than your local peers (your relays). Just tell it not to use the ledger to look for other peers.

Delete the line: “useLedgerAfterSlot”.

Then when you start your node ensure you see a log like this:

Jul 18 08:17:01 cardano-node[85210]: Don't use ledger to get root peers.

You can (maybe should) also set how many peers your node will attempt to maintain in your config file (mainnet-config.json):

  "TargetNumberOfRootPeers": 5,
  "TargetNumberOfKnownPeers": 5,
  "TargetNumberOfEstablishedPeers": 5,
  "TargetNumberOfActivePeers": 5,

I have my bp config set to seek 5 peers because I have up to 5 relays. If you only want to rely on 1 relay then you can set these values to 1. Though, you might want to setup another relay, even if it is behind a firewall, only known to you, and not registered on-chain.

Note: You can have a relay running behind a firewall (which blocks all incoming ports) and this relay can still enable blocks to be pulled from it by external relays. This is because this relay can initially set up half duplex connections outwards to external relays and then these can be later upgraded to full duplex if the other external relays are also running in P2P mode. You effectively had your block producer running this way due to the “useLedgerAfterSlot” line you had in your topology file.

Peter_Vouk · 21 July 2023 15:36

I had a similar issue so I kept my BP in Non-p2p mode. I have followed your directions and I am in p2p on relays and BP now. Thanks for the detailed response.

Terminada · 23 July 2023 00:35

Here is an example. This relay is running on my internal network and the firewall prevents any incoming connections to it from outside the network.

curl -s -H 'Accept: application/json' http://localhost:12788 | jq '.cardano.node.metrics.connectionManager'
{
  "duplexConns": {
    "type": "g",
    "val": 9
  },
  "incomingConns": {
    "type": "g",
    "val": 0
  },
  "outgoingConns": {
    "type": "g",
    "val": 50
  },
  "prunableConns": {
    "type": "g",
    "val": 0
  },
  "unidirectionalConns": {
    "type": "g",
    "val": 41
  }
}

After 30hrs of running, it has set up 50 outgoing connections and 9 of these have been upgraded to duplex. These duplex connections will allow the other end to pull blocks from this hidden relay.

token · 5 August 2023 02:09

Here is a guide for p2p, topology template for relay and bp

Jun_Sun · 10 August 2023 18:14

I have set up the relay node in p2p mode while leaving producer node with static topology. I have set them to run in mainnet for 3 days now. Everything seems fine except that there is still no TX recorded. Is this normal?

See gLiveview pictures below for both nodes.

token · 10 August 2023 20:12

Hi @Jun_Sun

You also need to enable p2p in core.

Here is a guide on p2p networking:

Zyroxa · 11 August 2023 05:53

Hey @Jun_Sun

Check if

“TraceMempool”: true

in your config

Jun_Sun · 11 August 2023 06:09

Thanks for reminding. Yes, it is true.

Right now, I have followed @token and @Terminada advice and enabled P2P for block producer node as well. I also added ““useLedgerAfterSlot”: 79387772” to relay’s topology json file. It has been up for about 10 hours now. Still no TX.

token · 11 August 2023 06:26

Are the relay and bp on the same network?
Does your firewall (UFW) allow node ports?

I have a relay and bp on the same network. For that I use Tailscale:

It can be easily installed on both relay and bp. It does not need further firewall settings. Then, you
check their new Tailscale IPs and modify both topology files. Finally, restart the nodes

sudo systemctl restart cardano-node

Terminada · 11 August 2023 06:29

You might need to enable Tx monitoring in your mainnet-config.json file. I have set my BP and relays to not monitor Txs so I have the following:

  "TraceTxInbound": false,
  "TraceTxOutbound": false,
  "TraceTxSubmissionProtocol": false,

Check your values are true for these.

You might also need these to be turned on:

  "TraceLocalTxSubmissionProtocol": false,
  "TraceLocalTxSubmissionServer": false,

Jun_Sun · 11 August 2023 16:55

Yes. Both producer and relay are in the same network. I double-checked and firewall is not the issue.

In addition, I don’t see TX in relay node anyway.

Jun_Sun · 11 August 2023 16:57

I have turned them to true and will see if they change anything.

I kind of doubt it since I could see TXs with the same config options before. It is either related to P2P or newer version of cardano node software.

Jun_Sun · 11 August 2023 17:04

BTW, I should mention that my relay node and producer node are behind NAT. I have not registered the pool or set up port forwarding yet. As such, I only have 1 incoming connection to the relay node (from my own producer node).

Could this be the reason for relay node not seeing any TX? It shouldn’t though, at least with earlier versions of Cardano and without P2P.

token · 11 August 2023 18:34

Tailscale solves port forwarding and goes straight through the NAT.

Wireguard was the usual approach, but Tailscale has proved to have an easier setup.

Terminada · 12 August 2023 01:30

@token I don’t know why you keep recommending tailscale. Are you affiliated with the project in some way?

On their website they make lots of comments about how they love open source, but then they say this:

A closed source coordination server.

I don’t know what this coordination server is used for but it sounds like, well, “coordination”. So to me this means you are sending your meta data about which end points you are communicating with to their “coordination server”.

Consequently I thought it necessary to add this post so that anyone reading might think twice.

You don’t need to do that with wireguard which is what Tailscale is using under the hood. It might be better to invest the time to get wireguard working properly because it is a fantastic tool that works flawlessly and then you don’t need to send your private metadata to someone you don’t know.

Terminada · 12 August 2023 01:38

It might be true that tailscale is easier. It is also easier to leave your money in a TradFi bank and outsource the control over your money.

I don’t think we want to encourage SPOs to outsource any control over their servers. These guys are running the Cardano network.

If you want wireguard VPN features then just use wireguard. It is fully open source, easy to use, and doesn’t require any “closed source coordination server”.

token · 12 August 2023 02:10

For those who prefer wireguard, this is how you can set it up:

@Jun_Sun If you need any help with wg0.conf, port forwarding and firewall, let me know.

Jun_Sun · 13 August 2023 05:07

OK. To prove my sanity, I have disabled P2P and reverted back to the clio.one ping/pull style. Sure enough, only after a few hours my nodes start to see incoming TXs.

After looking at the whole issue again, here is my hypothesis. Please confirm or rebut.

My pool is not registered yet and relay node is behind NAT. As a result its public IP/port are not published in ledger. Nor can they be discovered automatically. Consequently when P2P is enabled, I think that is the reason why there is only 1 incoming connection as shown by gLiveview. (my own producer node)
With P2P disabled, my relay node IP/ports are registered with clio.one server. Pretty soon clio.one server will tell other relay node and I start to see incoming connections reach 2.
It seems only when my relay node has external incoming connections (other than my own producer node obviously), relay node starts to see TXs.

Can anyone confirm this? This might be a bug somewhere…

BTW It is confusing to see all the different p2p listings in gLiveview, including incoming, outgoing, uni-dir, bi-dir, duplex. Is there an explanation somewhere?

Another BTW, I will switch back to P2P right now for more testing. My predication is that, since I already registered with clio.one server, this time I should see more than 1 incoming connections even with P2P enabled and I should see TX.

Topic		Replies	Views
How do I connect to more than 2 hot peers on pre-prod? Operate a Stake Pool	3	328	25 August 2023
Producer Server says its a relay? Setup a Stake Pool	15	595	28 March 2021
P2P on block producer and warm/hot peer connections with relays Operate a Stake Pool	2	290	1 February 2024
Low custom peers question (P2P) Operate a Stake Pool	4	318	4 April 2023
P2P - Yes or No? Operate a Stake Pool	4	412	18 August 2023

P2P - why producer has many peers?

Related Topics