No Txs / Blocks missed

hi,
I set up a new stakepool on the mainnet (ticker AMP). I went through the all process on testnet with no problem.
My two first block were forged/adopted but not minted on the blockchain. No error, no battle.
Attached are my gliveview pics. P2P is enable on my relays and disabled on my BP (I did it one hour ago). I still have no Txs on BP and it could be the problem.
Any chance someone look and the pics and tell me if I missed something?
Cheers, Laurent
Core
Relay1
Relay2

1 Like

with 0 Incoming PEERS your blocks will not be distributed to the blockhain
you need to fix this issue (firewall rules on BP or check if you added BP IP+port on Relays topology update script)

Cheers,

Thanks for your quick answer Alex.
I cannot see anything wrong in my files.
Topology_BP
Topology_Relay1
Topology_Relay2
UFW_BP

Ok Alex I replace MyDNS with the local IP. Got incoming peers now.
So from now some TXS should be done on the BP, right?
Laurent

Yes, shoukd be fine now, did u set tracemempool to true inside configuration file?

Yes I have done that. I have some blocks served on BP but still no transaction. And previously My BP was set to enableP2P=true but it did not make any Txs even with incoming peers.
When I created the BP keys (node.cert…) the BP IP in the relays topology.json was wrong. I am wondering if I need to recreate the keys. Do u think it could matter?

Nope, should be fine
Do u run topology updater on relays? Can u check the message from logs?

Your relays also seem to have only the incoming connections from the BP and the other relay. Might that be the issue, why no transactions are coming in?

Does your firewall let through connections to the relays? (By the way: Are both your relays in your local network? The idea is actually to distribute them to different datacentres in different locations.)

Are your relays registered correctly with your pool certificate?

Also: As far as I understood it, connections in P2P mode are scaled by stake. If you don’t have much, you maybe won’t get incoming connections.

No I am not. Bit of journalctl attached. According to Gliveview I have many peers on relays.
Journalctl from relay

u have 2 relays right? Keep one relay on P2P mode and the other set it static + topology updater script
This way u can test more… without your blocks being affected

Ok thanks alex. I’ll do that.
Laurent

1 Like

Are you sure about that? I don’t think that is the case. I think the nodes are basing their decisions about which nodes to prefer simply on which ones reliably provide them new blocks the quickest. In other words, your relay could end up preferring a node that is being run for a Daedalus wallet and is not even associated with a pool, if it is well connected, and fast. But, I can’t read the Haskell code well enough to do a deep dive into checking such things.

If you are right about P2P nodes making selection decisions based on stake distribution, then I will need to think about that more in terms of whether that opens some way to manipulate things by the bigger pools.

Hi @adaspear

Your block producer is running topology updater, while your relays are on p2p.

Have you tried enabling p2p on bp?

Not totally. I was just inferring from the usual questions if an SPO has no incoming connections on a P2P relay being “Is it registered on-chain?” and “How much stake?” in the #p2p-mainnet channel in the IOG Discord.

According to https://iohk.io/en/blog/posts/2023/03/16/dynamic-p2p-is-coming-to-cardano/ you are right.

But, on the other hand, according to https://github.com/input-output-hk/cardano-node/releases/tag/1.35.6:

Ledger peers are the relays registered on the chain. Currently we use square of the stake distribution to randomly pick new ledger peers.

And the only(?) way to initially get a relay to be known by other nodes is by being a “ledger peer”, by registering it on-chain.

1 Like

Yep. I had the same problem when BP on P2P => No Txs.
I will look through my setup today. Maybe I will spot something…
thanks, Laurent

That is really interesting. I hadn’t noticed that. Thanks for pointing it out.

I guess that probably makes sense during initial node start-up since that would be a good method of picking the most active relays.

Once the node has been running for a while this probably matters less because I believe each node then finds out about other available nodes via the “gossip” protocol. I think the “advertise: true” setting under publicRoots enables your node to advertise which peers it thinks are good to others using this gossip mechanism and this is why I have this set to false for my localRoots. (Because you don’t want your block producer being advertised.)

The topology of your BP lists ports 62020 and 63230, while your firewall (UFW) only allows 60110/tcp.

You can set your firewall for each node and check if you can ping between them.

Well, if your relay never gets picked up by any other nodes, because you have close to zero stake, there will also be no nodes to gossip it further.

Moreover, I think gossiping is not active up to now as per this conversation with @ATADA:


https://discord.com/channels/826816523368005654/1086025344311627826/1134904223935516762

1 Like

Thanks again. Presumably IOG are being super cautious about a possible network partitioning if P2P with gossiping is fully enabled?

I have noticed a significant increase in the number of incoming connections to my relays over the past few weeks. I presume this is because more people are switching on P2P mode. Interestingly my relays have a lot more incoming connections now compared to under the old system where operators were downloading topology files from api.clio.one. This P2P mechanism may be benefiting my more physically decentralised pool over here on the other side of the world, through quicker block propagation mitigating my increased risk of 1 second delay fork battles.

All 3 of my chain registered relays are quite different at present. One is on an AMD machine, one on a low powered ARM machine, and one is a Contabo vps. I need to watch more over time, but I think other relays tend to prefer my relays in the following order:

  1. AMD relay
  2. ARM relay
  3. Contabo relay

This is purely based on the number of incoming and duplex connections each has. If this is the case, then I think it is because this is in the order of processor speed. When each relay has 20+ incoming/duplex connections it might be a tiny bit slower for my ARM machine to check the block headers and forward them and even more with the block bodies. The Contabo vps is a cheap service where they are over-allocating cpu so that is also likely to be a bit slower.

I probably need to look at the processor utilisation more and experiment with increasing these settings:

"TargetNumberOfEstablishedPeers": 50,
"TargetNumberOfActivePeers": 20,

to see what happens if I tell my ARM relay to maintain more active peers. Presumably there will be a sweet spot where block receipt delay is minimised relative to processor and network bandwidth limits.

Hi Alex.
I decided to renew my node.cert this morning.
As you can see I forgot in the first place to change the “null” value to 0!
Can I change it manually to one (1) or do I need to do it another way?
thanks
Laurent