I’m back using cncli instead of my own script. It is also much faster computing the leaderlog compared to cardano-cli.
This is one matter where I have to agree 100%. Due to a cncli bug which manifested before the advent of node v8, we were using cardano-cli query leadership-schedule for several epochs, then got cncli working again when changed our hosting platform… now it runs in several seconds vs. several minutes, without pushing the node’s memory pages out into swap space. 
1 Like
If I understand correctly, CNCLI stores some data in its own DB that allows it to predict the allocated slots for the next epoch just like the leadership-schedule query but faster and by consuming less resources?
1 Like
cncli calculates the epoch nonce using it’s sqlite database of blocks for the past epoch. This epoch nonce calculation is based on all blocks for the past epoch up to “stability window” blocks from the end of the epoch. (Or something close to that?) Presumably the cardano-cli calculates this by directly using the blockchain data on your running cardano-node. Whereas cncli grabs this by querying it’s sqlite database and I believe it essentially builds this epoch nonce value over time as the epoch progresses.
So, in other words, the reason why cncli might be quicker at calculating the epoch nonce is because it has partly amortised the cost of the calculation over the entire epoch. Then it can more quickly grab the data from sqlite and crunch it to produce the epoch nonce seed for the vrf function. IE: It is faster because it is not a fair comparison. If you add in the cost of running cncli for the entire epoch then maybe cardano-cli is actually less computationally expensive?
Now to calculate the vrf value for the block you produce you only need the following:
- epoch nonce
- slot number
- pool vrf.skey
Then the leader vrf value seems to be produced by hashing this vrf value with the string “L” (for leader). See: this code here.
1 Like
Likewise we’ve recently found it works optimally when the network load is similarly amortised. As I mentioned in the bug report above, we’ve had to arrange cncli sync to run as a system service rather than just syncing once per epoch as we formerly did: only when needing to produce the leadership schedule.
So now we begin the process with a fully synced cncli database whose network load has been finely broken down over the previous 5 days. I’ve noticed cncli will even sync from the public load-balanced nodes at relays-new.cardano-mainnet.iohk.io at about 4x the speed at which the chain itself is expanding: so with the network load distributed by continuous syncing (and getting a head start with a recently synced cncli DB from elsewhere) you can use cncli to generate the leadership schedule outside your own stake pool nodes. 
1 Like
Thank you for your answers 
Thank you for this script, very nice
I am not familiar with reading Python though, could you please tell me where you perform the encryption of the slot list?
So you are saying that cncli seeks data from external relays. I hadn’t realised that. I am not sure how I feel about that. On the one hand it is public data, but on the other hand this is something that I didn’t know cncli was doing. The configuration involves telling it only about your own node’s host / port information so the assumption is that it is only communicating with it.
Regarding the speed of cncli sync: Obviously cncli doesn’t need to do all the checking that your cardano-node process does while it syncs. cardano-node is verifying every block, all the transactions, vrf checks, pool signatures, etc.
An interesting comparison would be to use ogmios to grab the header data of every block directly from your running node and dump the bits you want to a sqlite database. That should be very fast. I am going to try to build something to do that, but I would be glad if someone else did. I plan to try using haskell, but I only know enough to be dangerous.
I think you have already known it, but I’m just expressing a different (kind of antisocial) use case. cncli doesn’t do this by default (or choose any host/port by default) but users and operators can seek data from external relays, and they don’t have to be yours nor do you need the relay’s permission: though I guess they can ban you if they realise you’re not a relay.
I used relays-new.cardano-mainnet.iohk.io when testing this because it would be a drop in the bucket compared to Daedalus load, but you could use any relay in theory & in fact this also works (if you don’t mind waiting a couple months for it to build the DB from scratch):
cncli sync --no-service --host relays.digitalfortress.online --port 3001
For me this runs at exactly the same speed as syncing from the IOHK servers… an average of 1 block per 5 seconds (4x the chain growth rate)… which is slower than what you’d expect for fast processors and bandwidth. I actually don’t know what’s bandlimiting it in this case… it might in fact have to re-verify all the cryptographic structure, but far less efficiently than cardano-node does.
1 Like
There’s a simple reason for cncli having an advantage: it only stores block headers and not block content in its database.
To be more exact: cncli stores all block headers from the beginning of the shelley era.
1 Like
Well, the basic idea of using cncli is to send tip and slot information to pooltool.io. Especially for sending tip it makes sense to use your own node (bp that is) to provide the proper information for the metric data shown in pooltool.io.
I’m using leaderlog from cncli, because I have it’s synced db anyways and because it is more efficient compared to cardano-cli (for the discussed reasons).
1 Like
Ahhh. Of course. I now understand what you were saying before. I had thought you were indicating that cncli was communicating with external relays even without you configuring it to (which was alarming me). 
1 Like
OK then, maybe you have some insight about why cncli takes 5 seconds to sync each block from an external relay, while a properly hosted cardano-node can sync a couple hundred blocks in the same amount of time? And how that is defined as an “advantage” for cncli in this case?
1 Like
If you sync a cncli db from your own node you’ll see how much faster cncli can sync compared to cardano-node. There are some more reasons: cncli is written in Rust not in Haskell. I.e., it is fully compiled into native binary code and does not need any interpreter or garbage collector. And it uses sqllite as database. Which makes it easier to query data from outside.
yes, we’ve all done that, but your last reply still doesn’t answer the question about why the cncli sync to an external relay goes at 1/100th the speed of a node-to-node sync. I thought you were saying you had an answer for that.
1 Like
I don’t get your question - obviously. A block is appended to the chain every 5 seconds; i.e. cncli stores the new block every 5 seconds. Forgive me, if this was not your question.
cncli does not make any difference between relays or bp nodes. It just uses the Cardano network protocol to aquire blocks from the configured source. The speed depends on the performance of the source.
It is only a starting point. Parsing of cardano-cli result and hashing is not yet implemented.
Yes I saw your comment on github. It would be great if someone could provide an example of the data to be sent before encrypting, maybe from cncli?
1 Like
I have added the missing parts according to the comments from papacarp to the script. Now I need someone to test it. Because my pool is to small to produce a block every epoch.
There’s also an example config file sendslots.json in the config directory to provide the data for calling cardano-cli with the proper files.