Docs and procedures for p2p (how to run p2p?)

Staking & Delegation Operate a Stake Pool
21

My approach of connecting my own relays to each other is mainly for redundancy between them, given they are at different data center and the fact that I trust my own relays more than others.

Your reasoning related to block propagation is interesting and something to think about, but block producers chose one relay at a time to propagate the blocks, so I don’t see any race condition that can happen between relays. Keeping in mind that relays are connected to many other relays, even if your first relay propagate the block to your second relay, then both will distribute it to their own connected list.

22

I guess the point I am making is that you don’t really care how quickly each relay gets the latest block. Instead you care about how quickly your block producer gets each block. And, you care about this only when you are about to make a block yourself. This is because you want to make your block on top of the latest block in order to minimise the chance of creating a fork which could result in your block getting orphaned. Passing the latest block between relays doesn’t matter. In fact, I would argue, that it doesn’t even matter if each other relay gets the block indirectly via your block producer.

However, if I argue for the other side: Depending on how the P2P mechanism works, it might be advantageous for each relay to get the latest block as quickly as possible because then they might get a better ranking in P2P metrics with other external relays. This could lead to these relays being preferred by others and thus better connected within the Cardano network. Something to consider???

Another consideration is that each relay node can only have a limited number of connections to other nodes before its communication will slow down. If this wasn’t true then it would be best for each of your own relays to be connected to every relay of the 3000 odd currently active stake pools. The current recommendation is to limit the number of hot connections to around 20 in order to minimise the performance impact.

So, if you are taking up 1,2, or 3 of these limited hot positions with your own relays, and this choice is not speeding up the transmission to your block producer… Well…???

23

Thank you for the configuration example, that really helps.

I have a question regarding the relay not running in P2P. As far as i understand, i do have to include it in my BP P2P syntax config file even though it’s not running P2P :

{
  "localRoots": [
    { "accessPoints": [
        { "address": "A.A.A.A", "port": 6000 },
        { "address": "B.B.B.B", "port": 6000 }
      ]
      , "advertise": false
      , "valency": 2
    }
  ]
, "publicRoots" : []
}

With Relay A in P2P mode and relay B in standard topology mode. Is that correct ?

24

Yes, that looks good to me for the block producer topology.

The legacy mode relay (B) should include the block producer in it’s topology:

{
  "Producers": [
    { "addr": "blockproducerIP", "port": 6000, "valency": 1 }
  , {... etc.}
  ]
}

Where {… etc.} is around 10-19 other Cardano relays that you obtain from a service like api.clio.one. If you don’t have a script for pulling a list of relays from api.clio.one then take a look at my script here and steal from it.

And the P2P mode relay (A) should include the block producer in it’s topology:

{
  "localRoots": [
    { "accessPoints": [
        { "address": "blockproducerIP", "port": 6000 }
      ]
      , "advertise": false
      , "valency": 1
    }
  ]
, "publicRoots" : [
    { "accessPoints": [
        { "address": "relays-new.cardano-mainnet.iohk.io", "port": 3001 }
      ]
      , "advertise": true
      , "valency": 1
    }
  ]
, "useLedgerAfterSlot": 0
}
3 Likes
25

Yes i’ve been running my pool for 7 month with the standard topology, and i’m switching to P2P so i just wanted to be sure that i understood correctly :slight_smile: Thank you very much !

1 Like
26

Anyone running this on mainnet?

1 Like
27

I have been running my relays and BP in P2P mode on mainnet for over a year. I think since Jan 2022 actually. I did have one relay in legacy mode until a few months ago but I even converted it to P2P. The P2P nodes just end up with more reliable peers and quicker block propagation times. I am desperately trying to keep my block delays to less than 1 second which is sometimes difficult from Australia.

If you don’t keep your block delays under 1 second then you will end up with 3 times the number of “fork battles”. If your block delays are greater than 2 seconds then you will get 5 times the number of “fork battles”. Every fork battle comes with a 50% chance of having your block orphaned.

28

Thanks Terminada, are you able to share your config for this.

1 Like
29

Relay mainnet-topology.json:

{
  "localRoots": [
    { "accessPoints": [
        { "address": "bp1.terminada.io", "port": 2700 }
      ]
      , "advertise": false
      , "valency": 1
    }
  ]
, "publicRoots" : [
    { "accessPoints": [
        { "address": "relays-new.cardano-mainnet.iohk.io", "port": 3001 }
      ]
      , "advertise": true
      , "valency": 1
    }
  ]
, "useLedgerAfterSlot": 84916732
}

Each of my relays has the exact same topology configuration. Note that I don’t list my other relays as localRoots accessPoints in each relay topology. I don’t see any point in doing that because I don’t care if my relay that first gets a block provides it quickly to my other relays. I only care how quickly this block gets delivered to my block producer (“bp1”). My other relays can then get that block from my bp1 if they don’t receive it from another external relay first. The goal is to get each new block to your BP as quickly as possible from whichever relay receives it first.

In other words, I believe a “fan out” topology is best, with your BP at the centre, and no deliberate interconnections between your own relays. You don’t want them wasting time delivering blocks between themselves, just get on with delivering each block to your BP.

BP mainnet-topology.json:

{
  "localRoots": [
    { "accessPoints": [
        { "address": "relays.terminada.io", "port": 2700 }
      ]
      , "advertise": false
      , "valency": 6
    }
  ]
, "publicRoots" : []
}

Note: the name “relays.terminada.io” resolves to many IP addresses on my bp1 machine. That is why I have valency 6, because there are up to 6 at any time. You can configure a name to resolve to multiple IP addresses like I have or simply list the IP address of each of your relays in the localRoots accessPoints array.

30

Hi @Terminada , I thought I should link you to another post I made in which I was using your setup here as reference (Advantage of relays? - #2 by HeptaSean), I’m not sure if you are still using the subdomain bp1.terminada.io, but thought you should know that is a possible security risk since it exposes the block producing node ip address which you want to keep secret as possible. If I’m wrong about this and what was talked about in my post let me know. I greatly appreciate you sharing in the first place and it has been very helpful!

1 Like
31

Thanks for your concern and bringing it to my attention. However it is not a problem. I run a separate nameserver on my internal network and “bp1.terminada.io” resolves to a private IP which is not possible to directly connect to from outside. Actually, bp1.terminada.io doesn’t resolve to anything from outside my internal network (not that it would matter if it did because it is still not possible to connect with it from outside my private network).

I only use the domain names for convenience. In particular it is nice to use the “relays.terminada.io” name which I have configured to resolve to many IPs on my internal network. This allows me to shutdown and spin-up several different relays on different machines (ARM / AMD) more easily without having to change the various topology files.

The only nodes that can directly connect with my block producer are my own relays.

I do appreciate your concern though. We do want to look out for each other and try to keep Cardano as the most secure, most reliable, blockchain.

2 Likes
32

Might be a matter of taste, but I would use a separate internal domain – e.g., bp1.terminada and relays.terminada – for things that are not supposed to be publicly resolvable (which would also make more clear to people reading your guidance that making bp1.terminada.io publicly resolvable would be a very bad idea).

2 Likes
33

Thanks @HeptaSean I do agree with your general sentiment. Even though I do know what I am doing with name resolutions on my internal network and I run my own caching nameserver internally which is well firewalled from outside. I probably should have modified those example configs to remove the dns names because I am now seeing that this might confuse some users possibly resulting in security issues.

It is probably best to provide example configs using only IPs in the reserved private network range.

Hmmm. I am starting to regret recommending a caching nameserver as a good idea in this other thread as this might require a bit more security and firewalling understanding.

1 Like
34

Just out of curiosity: Do you set up a VPN to be able to reach BP and relays in the private network? Or are they really physically at one location? Because the idea is to have relays spread world-wide.

1 Like
35

@HeptaSean thank you for asking.

I have 3 publicly registered relays. 2 in Australia and one in the USA. I also have several other unregistered relays at different locations.

I have fibre connections at different locations for relays and also have at least 1 backup mobile network associated with my BP. I leverage using friends and family fibre home internet connections for most of this.

One problem with any simple setup is that there is usually only 1 internet connection wherever the BP is located and so this becomes critical. Even though my fibre connection where the BP is housed is around 99.9+% reliable (I estimate), I also wanted an alternative connection that didn’t involve some sort of fail-over strategy taking time to switch. For this I just use a relatively cheap mobile network using a different provider. I use wireguard and network namespaces with a different routing table to ensure the BP is connected to other relays simultaneously via the different network links. Thus if the fibre network does go down, hopefully the mobile network remains.

However, I have formed the opinion that this is probably significant overkill for a small pool like mine (and even a large pool). As I said, I believe my fibre connection is around 99.9+% reliable so this would mean that even without any backup link this would result in less than 0.1% of lost blocks. I actually think that if the fibre link goes down for any significant amount of time it is quite likely that someone broke some major infrastructure point in Australia or the undersea cable, and this will mean that my BP will be disconnected from the rest of the world until fixed. IE. it won’t matter if I have a redundant link and it won’t matter if I have 10 other relays in 10 different countries because none of them will be able to connect with the BP in Australia.

I think we overstate the reliability needs and this is costing SPOs and their delegators unnecessarily. Ouroboros is designed so that pools can disconnect and reconnect. If a block is missed here and there the protocol keeps ticking along.

99% reliability is good enough and trying to get much more reliable than this will cost SPOs more, which means they need to charge more in fees to their delegators. It is totally crazy for a SPO to increase his fees from 1% to say 5% to pay for increasing his reliability from 99% to 99.95% because his delegators will actually lose more rewards.

From a protocol point of view, we are more resilient, and more decentralised, by having lots of small pool operators that are around 99% reliable. I am not suggesting you have put any pressure on anyone, but I think we need to be careful about pressuring small pool operators to achieve 99.95% reliability as this will simply result in more centralisation and multi-pool operators.

By the way, I know the cause of every block I have missed or had orphaned and none have been caused by a network disconnection over nearly 2 years and 400 on-chain blocks.

2 Likes
36

Yes, that pretty much sounds like overkill. :crazy_face: But like overkill that sparks fun for you. So that’s okay. :grinning:

So, I think my question is answered by: Yes, the network where the relays reach the BP by a private IP address, is a VPN across all these different relays and their sites?

1 Like
37

I use a wireguard VPN as a separate network between various nodes but I also route to my BP via my main link using DNAT (nftables). Eg:
Relay4 IP → IP of firewall machine : Port 3030 → DNAT to 192.168.0.10:2700.

So I use both techniques to allow the relays to initiate connections to the BP. The BP is behind the firewall obviously and it can initiate connections out to everywhere.

I also ensure there is nothing else running on all relays and BP. No other services apart from ssh. And ssh has only key based authentication with the key on a hardware smart card.

Yes and it is tax deductible because it is an operating expense. If I added up what it all really costs me, it would be a bit scary. However, I don’t see it this way. I would do all this even if not running a stake pool. Running the stake pool just commits me to staying more focused. I see it as investing in learning which hopefully will create opportunities in the future.

38

Speaking of hidden relays. I have one running on my Wife’s computer at home and she doesn’t even notice it, except that the computer stays on 24/7 which she finds convenient anyway.

1 Like
39

That is funny haha, I was thinking of making my personal computer ‘one of’ the relays also, even though it would be unreliable since it would be shutdown once in a while and fail when the internet goes out from time to time on wifi haha, would that be a problem you think? I have two other paid relays I’m running using contabo

1 Like
40

Why not do it? The benefit of the extra relay is around the time you make a block because this is the time you want fastest possible block receipts and propagation. Depending on how many blocks you get, you can at least try to ensure your personal computer, with extra relay, is working well at these times.

1 Like