Docs and procedures for p2p (how to run p2p?)

Hello,

Looking to run p2p on testnet.

Searching the forums I’ve noticed
"EnableP2P": true

Searching I found IOG have a pre-production p2p config
The only other documentation I found was this IOG blog post

I would like to know if other operators are running the ‘official’ p2p and how they set it up. I’m assuming that from the lack of good documentation that it’s not currently a priority or fully supported yet (unless I missed the docs)?

Any input from fellow SPO’s would be great!

There’s been a few topics here about p2p, the docs aren’t clear yet from IOG, but there’s been a few that have it going.

I’ve got my pre-prod machine running p2p. I can post the topology if that helps, but there is a topology on the docs.cardano website too https://docs.cardano.org/cardano-testnet/getting-started

I’m running p2p on preprod. Its pretty much plug and play using CNTOOLS. You can follow the guide I wrote here on setting up on pre-production or preview testnets. You can adjust and add peers to your p2p topology fairly easily. Once your node is discoverable, it may take some time to start connecting to peers. I think it took mine around a day to start connecting to random folks.

There’s some great info on the IOHK Technical discord that is worth searching for if you are seeking deeper understanding.

1 Like

ive been trying p2p on mainnet - because someone has to

I generally get between 0 and 1 connections. The most I ever got was 2.

tldr; not worth it on mainnet at the moment.

Here are some info:

Here is my topology file what I use in prod and testnet networks with some details:

For me p2p works out of the box in most cases(I do run more then half year in mainnet and forged almost all my blocks using p2p), just don’t forget to monitor incoming connections to your relay and maybe enable TraceMempool on your bp and/or relays.

1 Like

I have had most of my relays and my only block producer running in P2P mode on mainnet for several months now and have produced blocks without any problems. Here is how I set my configs:

Relay config:

/etc/cardano/mainnet-config.json changes:

  "TestEnableDevelopmentNetworkProtocols": true,
  "EnableP2P": true,
  "MaxConcurrencyBulkSync": 2,
  "MaxConcurrencyDeadline": 2,
  "TargetNumberOfRootPeers": 100,
  "TargetNumberOfKnownPeers": 100,
  "TargetNumberOfEstablishedPeers": 50,
  "TargetNumberOfActivePeers": 20,

/etc/cardano/mainnet-topology.json complete file:

{
  "LocalRoots": {
    "groups": [
      {
        "localRoots": {
          "accessPoints": [
            {
              "address": "bp1.terminada.io",
              "port": 2700
            }
          ],
          "advertise": false
        },
        "valency": 1
      }
    ]
  },
  "PublicRoots": [
    {
      "publicRoots" : {
        "accessPoints": [
          {
            "address": "relays-new.cardano-mainnet.iohk.io",
            "port": 3001
          }
        ],
        "advertise": true
      },
      "valency": 1
    }
  ],
  "useLedgerAfterSlot": 0
}

Note: LocalRoots is only the block producer. You need your relay to connect to your block producer in order to propagate the blocks produced.
Note: “useLedgerAfterSlot”: 0 tells the node to gather data about registered relays from the blockchain after slot 0 (ie. from the start of the blockchain).
Note: My block producer and relays use port 2700. Most people have theirs configured to use 3001 like the IOHK relays are.

Block producer config

/etc/cardano/mainnet-config.json changes:

  "TestEnableDevelopmentNetworkProtocols": true,
  "EnableP2P": true,
  "MaxConcurrencyBulkSync": 2,
  "MaxConcurrencyDeadline": 2,
  "TargetNumberOfRootPeers": 5,
  "TargetNumberOfKnownPeers": 5,
  "TargetNumberOfEstablishedPeers": 5,
  "TargetNumberOfActivePeers": 5,

/etc/cardano/mainnet-topology.json complete file:

{
  "LocalRoots": {
    "groups": [
      {
        "localRoots": {
          "accessPoints": [
            {
              "address": "relays.terminada.io",
              "port": 2700
            }
          ],
          "advertise": false
        },
        "valency": 4
      }
    ]
  },
  "PublicRoots": []
}

Note: PublicRoots is empty
Note: Absence of useLedgerAfterSlot.

Check your logs at node start. You should see:

cardano-node[232505]: Public Roots:
cardano-node[232505]: Don’t use ledger to get root peers.

This is very important for your block producer because you only want it connecting with your own relays (LocalRoots) and not any external relays. In other words, you do not want your block producer to gather registered relays (root peers) information from the blockchain.

I have an internal DNS server that returns multiple IP addresses for “relays.terminada.io” - in fact it currently returns 4. Hence “valency”: 4 under “LocalRoots”.

Note the different TargetNumberOf[Root|Known|Established|Active]Peers for the block producer which I set to 5 when I currently only have 4 relays. I set it one higher than the valency of the LocalRoots but it will be limited to 4 anyway because there is no potential to connect any via PublicRoots. I just set this at 5 to allow me to add another relay easily and only update the topology file.

I set “MaxConcurrencyBulkSync”: 2 and “MaxConcurrencyDeadline”: 2. I set these to 2 because I didn’t want my block producer attempting to sync a block from more than 2 relays at once in case this would just waste processing time and potentially lead to increased risk of missed slot leader checks. Maybe it would be better at only 1? I don’t know if it would be beneficial to set these higher on relays but I set these the same on my P2P relays.

Under LocalRoots I set “advertise”: false. I also set “advertise”: false under LocalRoots for the relay. I am not sure about this but I believe that if set to true this would allow the block producer and relay to tell incoming connections about known local access points. I wondered if this could in some way cause leakage of information about how to connect to my block producer via my P2P relays gossiping with external P2P relays. In my case, it may not matter if set to true, because my firewall would prevent any external connection to my block producer.

Results:

I monitor block delay times on all my nodes and found that my P2P relays had less delay compared with my relays running in normal mode. I do still run one relay in normal mode. I have produced several blocks over the past few months using these settings.

I think that P2P works really well. However, obviously IOHK is not 100% confident yet under all conditions. I believe this is why it is hard to find proper documentation about what all the settings mean. For example, it is possible that it works very well if only a few nodes use it but may cause some sort of network partition if all nodes use it. This is why it is still being tested I guess. So be aware of the risks if you choose to run in P2P mode since it has not been officially recommended on mainnet yet.

One final point: You still need to ping api.clio.one with your topology data every hour (port, hostname, and latest block). If you don’t continue to ping this data to api.clio.one then you may find that your relays don’t get added to topology files used by other pool operators still running in normal mode. This could result in less incoming connections to your relays which could result in poor propagation of your produced blocks.

3 Likes

Thanks for the responses :slight_smile:

I also had to take into consideration the latest testnet environments. I won’t be using p2p in mainnet until I’ve done some extensive playing around in testnet.

Hi!

Great summary.

I would like to add 2 points:

  1. Local roots in relay topology file should contain not only your BP, but other your relays too. So if you have 1BP and 1 relay, then 2 records should be there.
  2. Regarding api.clio.one I don’t bother to use it. I do not run topology updater anymore for more then half year and 200+ blocks were produced just fine without it, so I would say it is overkill.

Advertise is for future gossip feature, which seems doesn’t yet work and you should not enable it for your BP.

2 Likes

I did think about that but chose not to configure things that way. My reasoning was as follows:

I don’t see any point in including all your own relays in each relay’s P2P topology file. What you care about is fastest transmission of new blocks to your block producer. You don’t care which relay delivers it to your block producer only that it gets there fast. Having your relays connect with each other won’t speed up transmission to your block producer. Furthermore, I don’t think you care if your other relays get the latest block via your block producer. I think that having a star shaped topology with your block producer at the centre is likely to be faster for the block producer.

If you consider the situation where you have a mesh topology then you could have one of your relays request block data from the first receiving relay at the same time as your block producer is requesting it. This could slow the transmission of data to your block producer should the first relay choose to transmit to the second relay before the block producer.

1 Like

Hi @Terminada !

Thank you for explonation! That is an interesting approach, I never thought about that.

I personally would stick with topology file containing not only BPs, but your own relays too, I think that is most common and recommended approach.

1 Like

Thanks so much for sharing your experience and recommendations. I have upgraded to 1.35.4 with one relay and my BP on p2p. I have left one relay on normal mode as well for now. Your guide has been a great help!

1 Like