A part of our infra is to keep processing new blocks to check if any currency(here ADA) is sent to address from a list.
To do this in case of other blockchains we listen to events of new blocks and process them, we also store the blockheight so in case of a reboot the databases can be synchronized.
But in case of ADA I guess there aren’t any events, and also there isn’t an api to fetch the data based on block-height.
Right now my option is to query the node with /api/blocks/pages/total then if the pageno has increased then subsequently call /api/blocks/pages, scan for the slots/blocks and for each blocks call /api/blocks/txs/{blockhash}, then scan the outputs for the matching address.
This looks like a long process. If we can have events of new blocks a lot of the above steps can be reduced. Or as an alternative if there is an api that can directly query on the blockheight.
Hi! From what I understood - you don’t really need to process the whole history of the blockchain, sp there’s no need to process every block and every transaction, because if you have, for example, a list of 10 addresses - then you would process 21K blocks and 15K transactions every epoch just to find maybe 1 or 2, or even 0 transfers to your addresses of interest.
So you can configure the poll-time as you wish (for example once a minute) and then just iterate thru all addresses and query a summary for each one, and see if there’re any new transactions. This way you will only be processing the information of direct interest to you, and you can store in the database the last known number of transactions for each address, or an ID of the last known transaction.
But if at some point you will need to have a history of the blockchain processed, or some blocks-metadata collected - then the way you have described it is the only one, afaik, for now.
My bad when I said list I should have mentioned the volume.
We have around 3mil addresses to begin with, so the method you mentioned though would work for a smaller list but in our case it is not possible.
The the Goal here is to continuously listen to new confirmed transactions and check if there is any ADA incoming to one of our addresses and subsequently update our database with the amount.
Well, it seems like it should support the “Subscribe SubTx” event and similar, but I haven’t actually tried those out yet, so cannot provide any more reliable info about it (And also couldn’t find any other documentation on it).
It requires a running SL node on the same host, so you could connect to localhost:8110. If you manage to get it working - would be really cool and might be really useful. If you ready to wait a bit - I could try it out tonight (couldn’t find a reason to do it for long time ) and some manuals if I manage to get it working.
Yeah, I had the same feeling about this thing. I’ve literally seen zero discussion about socket-API before your post Let’s hope they answer something, and maybe I’ll take a look into code to see if it’s still running according to the docs.
I will change the topic-title a bit, maybe someone with some experience on the issue will help us out.
The first few bytes are 00 09 07 ef bf bd instead of “97:”, the EFBFBD is the UTF8 for the Unicode’s invalid character FFFD , means that the explorer somewhy could not build the proper string. The 00 09 07 is CBOR representation of 0 9 7, the length of the payload, the string w/ the curly brackets. The 0 before the “{” means type is connet.
I was wrong, the ebbfbd, comes from node.js Socket.io, what really comes from Explorer is
000907FF, and socket.io’s UTF converted the 0xFF to the Unicode’s invalid char.
Anyway, that result directly from explorer does not make sense as it’s not a valid CBOR representation, something missing. FF is the break/stop code of the indefinite length of some major types (map, array, bytestring etc.).
UPDATE: Nothing wrong w/ 000907FF, Socket.io should handle it, but the underlying XHR convert it to UTF8 (reponseText instead of response), despite the “Content-Type” is “application/octet-stream”. Probably, it got confused w/ the “Transfer-Encoding: chunked” header that comes from Cloudflare.
So, the socket.io-client or engine.io-client should be forced to use binaryType, somehow. I will check it when I am back from work.
UPDATE2: It cannot be forced to binaryType, as socket.io-client and engine.io-client use xmlhttprequest-ssl which is not XHR2 compliant, means the socket.io-client implementation does not support binary payload encode/decode in node.js, that’s sad.
What can be requested from IOHK, is to use string representation of the payload in their code instead.
I can use /api/epochs/{epoch}/{slot} api to get new blocks.
So I can have any block as my genesis block(starting point). Then my process would check the latest epoch/slot and then increment the slot by 1 to query the next block. (if the block is not yet formed, the slot is returned as empty array). Now once the slot no reaches the max_slot_per_epoch I could increment the epoch by one and start with 1st slot.
Yeah, I think you may use it, if your program really requires having a block without any delay, so you gonna make network calls every 20 seconds anyway.
On the other hand - if your program does not require such immediate requirement - I would make rarer network calls and receive more data at once (block pages).
But I would not probably recommend this approach, since it might cause some complications:
Currently an epoch is 21600 slots (with 20 seconds per slot). Unfortunately I couldn’t find any automatic way to get the epoch length, except calling /api/epochs/{epoch} and checking if you have processed the same amount of blocks as total amount of pages =\
Be aware that slot-length might change when network transitions to Ouroboros Praos (no definite date yet), and idk about an epoch length.
Also be aware:
Slots may be empty. The protocol does not guarantee that every slot will have a block in it, and Ouroboros Praos - in opposite - guarantees that some slots will remain empty. So your program must be ready for that.
That’s why I would consider processing blocks by height to be the recommended way of dealing with it.
That’s why I would consider processing blocks by height to be the recommended way of dealing with it.
Yup, that is the best way according me as-well, but the problem remains there is no api for getting blocks by height.
You only get height in tx summary api call.
I think you should also keep a close eye on the community, imo, it should definitely cause some noise if/when epoch/slot length would change And keep an eye on technical reports.
(And also couldn’t find any other documentation on it).
I meant just