Hello,
Looking to run p2p on testnet.
Searching the forums I’ve noticed
"EnableP2P": true
Searching I found IOG have a pre-production p2p config
The only other documentation I found was this IOG blog post
I would like to know if other operators are running the ‘official’ p2p and how they set it up. I’m assuming that from the lack of good documentation that it’s not currently a priority or fully supported yet (unless I missed the docs)?
Any input from fellow SPO’s would be great!
There’s been a few topics here about p2p, the docs aren’t clear yet from IOG, but there’s been a few that have it going.
I’ve got my pre-prod machine running p2p. I can post the topology if that helps, but there is a topology on the docs.cardano website too https://docs.cardano.org/cardano-testnet/getting-started
I’m running p2p on preprod. Its pretty much plug and play using CNTOOLS. You can follow the guide I wrote here on setting up on pre-production or preview testnets. You can adjust and add peers to your p2p topology fairly easily. Once your node is discoverable, it may take some time to start connecting to peers. I think it took mine around a day to start connecting to random folks.
There’s some great info on the IOHK Technical discord that is worth searching for if you are seeking deeper understanding.
ive been trying p2p on mainnet - because someone has to
I generally get between 0 and 1 connections. The most I ever got was 2.
tldr; not worth it on mainnet at the moment.
Here are some info:
Here is my topology file what I use in prod and testnet networks with some details:
For me p2p works out of the box in most cases(I do run more then half year in mainnet and forged almost all my blocks using p2p), just don’t forget to monitor incoming connections to your relay and maybe enable TraceMempool on your bp and/or relays.
I have had most of my relays and my only block producer running in P2P mode on mainnet for several months now and have produced blocks without any problems. Here is how I set my configs:
/etc/cardano/mainnet-config.json changes:
"TestEnableDevelopmentNetworkProtocols": true,
"EnableP2P": true,
"MaxConcurrencyBulkSync": 2,
"MaxConcurrencyDeadline": 2,
"TargetNumberOfRootPeers": 100,
"TargetNumberOfKnownPeers": 100,
"TargetNumberOfEstablishedPeers": 50,
"TargetNumberOfActivePeers": 20,
/etc/cardano/mainnet-topology.json complete file:
{
"LocalRoots": {
"groups": [
{
"localRoots": {
"accessPoints": [
{
"address": "bp1.terminada.io",
"port": 2700
}
],
"advertise": false
},
"valency": 1
}
]
},
"PublicRoots": [
{
"publicRoots" : {
"accessPoints": [
{
"address": "relays-new.cardano-mainnet.iohk.io",
"port": 3001
}
],
"advertise": true
},
"valency": 1
}
],
"useLedgerAfterSlot": 0
}
Note: LocalRoots is only the block producer. You need your relay to connect to your block producer in order to propagate the blocks produced.
Note: “useLedgerAfterSlot”: 0 tells the node to gather data about registered relays from the blockchain after slot 0 (ie. from the start of the blockchain).
Note: My block producer and relays use port 2700. Most people have theirs configured to use 3001 like the IOHK relays are.
/etc/cardano/mainnet-config.json changes:
"TestEnableDevelopmentNetworkProtocols": true,
"EnableP2P": true,
"MaxConcurrencyBulkSync": 2,
"MaxConcurrencyDeadline": 2,
"TargetNumberOfRootPeers": 5,
"TargetNumberOfKnownPeers": 5,
"TargetNumberOfEstablishedPeers": 5,
"TargetNumberOfActivePeers": 5,
/etc/cardano/mainnet-topology.json complete file:
{
"LocalRoots": {
"groups": [
{
"localRoots": {
"accessPoints": [
{
"address": "relays.terminada.io",
"port": 2700
}
],
"advertise": false
},
"valency": 4
}
]
},
"PublicRoots": []
}
Note: PublicRoots is empty
Note: Absence of useLedgerAfterSlot.
Check your logs at node start. You should see:
cardano-node[232505]: Public Roots:
cardano-node[232505]: Don’t use ledger to get root peers.
This is very important for your block producer because you only want it connecting with your own relays (LocalRoots) and not any external relays. In other words, you do not want your block producer to gather registered relays (root peers) information from the blockchain.
I have an internal DNS server that returns multiple IP addresses for “relays.terminada.io” - in fact it currently returns 4. Hence “valency”: 4 under “LocalRoots”.
Note the different TargetNumberOf[Root|Known|Established|Active]Peers for the block producer which I set to 5 when I currently only have 4 relays. I set it one higher than the valency of the LocalRoots but it will be limited to 4 anyway because there is no potential to connect any via PublicRoots. I just set this at 5 to allow me to add another relay easily and only update the topology file.
I set “MaxConcurrencyBulkSync”: 2 and “MaxConcurrencyDeadline”: 2. I set these to 2 because I didn’t want my block producer attempting to sync a block from more than 2 relays at once in case this would just waste processing time and potentially lead to increased risk of missed slot leader checks. Maybe it would be better at only 1? I don’t know if it would be beneficial to set these higher on relays but I set these the same on my P2P relays.
Under LocalRoots I set “advertise”: false. I also set “advertise”: false under LocalRoots for the relay. I am not sure about this but I believe that if set to true this would allow the block producer and relay to tell incoming connections about known local access points. I wondered if this could in some way cause leakage of information about how to connect to my block producer via my P2P relays gossiping with external P2P relays. In my case, it may not matter if set to true, because my firewall would prevent any external connection to my block producer.
I monitor block delay times on all my nodes and found that my P2P relays had less delay compared with my relays running in normal mode. I do still run one relay in normal mode. I have produced several blocks over the past few months using these settings.
I think that P2P works really well. However, obviously IOHK is not 100% confident yet under all conditions. I believe this is why it is hard to find proper documentation about what all the settings mean. For example, it is possible that it works very well if only a few nodes use it but may cause some sort of network partition if all nodes use it. This is why it is still being tested I guess. So be aware of the risks if you choose to run in P2P mode since it has not been officially recommended on mainnet yet.
One final point: You still need to ping api.clio.one with your topology data every hour (port, hostname, and latest block). If you don’t continue to ping this data to api.clio.one then you may find that your relays don’t get added to topology files used by other pool operators still running in normal mode. This could result in less incoming connections to your relays which could result in poor propagation of your produced blocks.
Thanks for the responses 
I also had to take into consideration the latest testnet environments. I won’t be using p2p in mainnet until I’ve done some extensive playing around in testnet.
Hi!
Great summary.
I would like to add 2 points:
Advertise is for future gossip feature, which seems doesn’t yet work and you should not enable it for your BP.
I did think about that but chose not to configure things that way. My reasoning was as follows:
I don’t see any point in including all your own relays in each relay’s P2P topology file. What you care about is fastest transmission of new blocks to your block producer. You don’t care which relay delivers it to your block producer only that it gets there fast. Having your relays connect with each other won’t speed up transmission to your block producer. Furthermore, I don’t think you care if your other relays get the latest block via your block producer. I think that having a star shaped topology with your block producer at the centre is likely to be faster for the block producer.
If you consider the situation where you have a mesh topology then you could have one of your relays request block data from the first receiving relay at the same time as your block producer is requesting it. This could slow the transmission of data to your block producer should the first relay choose to transmit to the second relay before the block producer.
Hi @Terminada !
Thank you for explonation! That is an interesting approach, I never thought about that.
I personally would stick with topology file containing not only BPs, but your own relays too, I think that is most common and recommended approach.
Thanks so much for sharing your experience and recommendations. I have upgraded to 1.35.4 with one relay and my BP on p2p. I have left one relay on normal mode as well for now. Your guide has been a great help!
Just bumping this thread as P2P will hopefully be the norm with node release 1.35.6
@Terminada summarised how to do this.
For myself I only needed
topology.json
{
"LocalRoots": {
"groups": [
{
"localRoots": {
"accessPoints": [
{
"address": "<BP_IP_ADDR>",
"port": 6000
}
],
"advertise": false
},
"valency": 1
}
]
},
"PublicRoots": []
}
config.json just copy and paste this at the top of the file. In 1.35.6 I believe the TestEnable will be dropped, but having it in the config won’t break anything.
"TestEnableDevelopmentNetworkProtocols": true,
"MaxConcurrencyBulkSync": 2,
"MaxConcurrencyDeadline": 2,
"TargetNumberOfRootPeers": 100,
"TargetNumberOfKnownPeers": 100,
"TargetNumberOfEstablishedPeers": 50,
"TargetNumberOfActivePeers": 20,
"EnableP2P":true,
Official resources:
1.35.6 + What is required
Understanding new topology changes
Update:
Have a read of the release notes for 1.35.6. After reading the 1.35.6 release nodes, here is what I have been using for my P2P setup:
mainnet-config.json:
"EnableP2P":true,
"MaxConcurrencyBulkSync": 2,
"MaxConcurrencyDeadline": 2,
"TargetNumberOfRootPeers": 100,
"TargetNumberOfKnownPeers": 100,
"TargetNumberOfEstablishedPeers": 50,
"TargetNumberOfActivePeers": 20,
"TraceInboundGovernorCounters": true,
"TraceConnectionManagerTransitions": true,
"TraceLedgerPeers": false,
"TracePeerSelectionCounters": true,
"TracePeerStateActions": true,
"TracePublicRootPeers": false,
Those were just the changes I made from the default file. Many of the defaults listed in the release notes document were already set. You don’t need to explicitly set many of the above if they are the defaults, I just set them to be clear.
There is no longer a need for the setting: “TestEnableDevelopmentNetworkProtocols”: true.
relays mainnet-topology.json:
{
"localRoots": [
{ "accessPoints": [
{ "address": "bp.terminada.io", "port": 3001 }
]
, "advertise": false
, "valency": 1
}
]
, "publicRoots" : [
{ "accessPoints": [
{ "address": "relays-new.cardano-mainnet.iohk.io", "port": 3001 }
]
, "advertise": true
, "valency": 1
}
]
, "useLedgerAfterSlot": 0
}
block producer mainnet-topology.json:
{
"localRoots": [
{ "accessPoints": [
{ "address": "relays.terminada.io", "port": 2700 }
]
, "advertise": false
, "valency": 4
}
]
, "publicRoots" : []
}
I recall reading a discussion about simplifying the topology file. There is no longer a need to have “localRoots” within “LocalRoots” or publicRoots within “PublicRoots”. You only require “localRoots” and “publicRoots” now. Sorry that I can’t find the link to this discussion easily. The old style topology still works though.
The official resource explaining the topology file uses the new style.
I have been running my block producer and most relays in P2P mode since Jan 2022. Initially I saw some problems with my block producer (running in normal mode) dropping connections to my P2P relays and then not re-establishing them. Once I converted my block producer to P2P as well this problem ceased.
For over a year now all my relays and my block producer have been running P2P except for just 1 relay which has been left running in normal mode.
By the way, the “useLedgerAfterSlot” configuration in the topology file is confusing to me. I have interpreted this as meaning: Look in the ledger for stake pool certificates to get data about registered relays, from this slot onwards. IE: My stake pool was registered over 2yrs ago so if you look in the ledger from a more recent slot then you wouldn’t see it’s registration certificate.
Other people have interpreted the meaning of this configuration differently and are using more recent slot numbers (as opposed to the beginning of the ledger which is slot 0). Though, once your relay is fully synchronised it will find peers via the P2P chat protocol, so this setting may not matter then.
Thanks. That might explain why I currently have such a low peer count of 3 (including one of my own relays). Cheers for the update appreciate it.
The optimal configuration for mainnet is (IMO)
"useLedgerAfterSlot": 0
It looks like 1.35.7 apparently fixes this issue. The release notes say:
“Node 1.35.7 is a bug fix release that addresses an issue with the legacy network stack connecting to a P2P relay.”
Here is the diff file for my block producer mainnet-config.json showing all the changes from the default config downloaded from https://book.world.dev.cardano.org/environments/mainnet/config.json
diff mainnet-config.json mainnet-config.json.orig
2c2
< "AlonzoGenesisFile": "mainnet-alonzo-genesis.json",
---
> "AlonzoGenesisFile": "alonzo-genesis.json",
6c6
< "ByronGenesisFile": "mainnet-byron-genesis.json",
---
> "ByronGenesisFile": "byron-genesis.json",
14c14
< "ShelleyGenesisFile": "mainnet-shelley-genesis.json",
---
> "ShelleyGenesisFile": "shelley-genesis.json",
16,22d15
< "EnableP2P": true,
< "MaxConcurrencyBulkSync": 2,
< "MaxConcurrencyDeadline": 2,
< "TargetNumberOfRootPeers": 5,
< "TargetNumberOfKnownPeers": 5,
< "TargetNumberOfEstablishedPeers": 5,
< "TargetNumberOfActivePeers": 5,
25c18
< "TraceBlockFetchDecisions": true,
---
> "TraceBlockFetchDecisions": false,
42,43d34
< "TraceInboundGovernorCounters": true,
< "TraceConnectionManagerTransitions": true,
45c36
< "TraceLedgerPeers": false,
---
> "TraceLedgerPeers": true,
52c43
< "TraceMempool": false,
---
> "TraceMempool": true,
56,58c47
< "TracePeerSelectionCounters": true,
< "TracePeerStateActions": true,
< "TracePublicRootPeers": false,
---
> "TracePublicRootPeers": true,
77c66
< "0.0.0.0",
---
> "127.0.0.1",
Notes:
We @MonedaCloud [MOC] just finished testing and deploying one of our relays using P2P.
This is a simple technical guide to follow for those that have not enabled P2P yet. This configuration works only on versions 1.35.6 and 1.35.7.