Syncing Blocks - curious about the bottle neck

daedalus

#1

I understand that the wallet is basically a frontend on an SL node.

It takes time to “sync blocks” - the node must be working through all the blocks that were added since the node was last running finding UTXOs owned by the wallet. Why does this take as long as it does?

It could be internet connection speed, although with the amount of data being downloaded this seems unlikely. That leaves CPU or memory. What is the bottleneck?


#2
  1. Downloading the chain, and processing transactions - are two different operations. First is done when you open the Daedalus for the first time (and “tail” is downloaded on each opening). And the second is done when you’re restoring a wallet, so already downloaded local chain is analysed.

  2. DB-1.0 folder with the chain currently takes 7.3 GB on my MacOS. Downloading 7.3 GB of data even with 1 Mb\s would take about ~7500 seconds or > 2 hours. And that’s not a torrent client (yet, afaik), so the speed is probably far less in practice (remember that it depends not only with what speed you can take the data, but with what speed other node can give it to you). Once it’s switched to torrents (possibly) - full download should be better.

  3. When a wallet is restored - Daedalus has to go thru all 7.3 GBs of chain and to scan every input and output in every transaction, so it’s just a lot of IO-load on the disk and a lot of CPU processing (matching hashes). Nothing strange here, imo. Some indexing could be implemented in the feature. Also - lite-clients, epoch-snapshots, etc.


#3

I researched the Bittorrent model, after I watched a presentation from CH on this. Intuitively if it works as outlined then the larger the network - numbers and geographic - the faster is will perform. I didn’t realize the first iteration of the network would not be using the Torrent model. I’d assumed that once the network was decentralized that it would be using that. Do you have any more color on this?


#4

А list of HTTP mirrors ran by IOHK shall be sufficient for a start.

In process of time, we plan to maintain a Bittorrent-based or Bittorrent-like solution to distribute updates. In general, P2P update distribution is a crucial business requirement due to legal concerns. It is to be decided which Bittorrent-like solution particular we will use.

Basically, for now, we’re all downloading the chain from the same set of nodes that runs the network (IOHK/Emurgo/Foundation federation). Torrents are planned, but not yet implemented.

Imo, would be kinda weird, to not let random nodes to run the protocol, but let them distribute the chain thru torrents. When SL moves to full decentralisation - then torrents will make more sense.


#5

They use Kademlia DHT and only for node discovering and not for all type of nodes. Also, they do not use the KV lookup feature at the moment.

For example, wallets (Edge nodes) usually behind some NAT/FW (ISP and router), which usually does not enabled for torrent protocol and therefore they do not allow incoming connections.

Kademlia DHT is affected by Spartacus attack, where any adversory can join to the network and claim that it has the same done ID as another nodes. So, I think that’s why we have some semi-centralised nodes in the beginnig to evenly distribute the node IDs to prevent this kind of attack.

That’s why the network topology is a kind of mixed protocols, but the bitcoin one is similar and it’s constatly (24/7) attacked for more than 9 yrs.