Cardan-db-sync not syncing - behind by 5 days

English Community Technical Support
1

Hello everyone,

I’ve been working on upgrading our entire infrastructure. We recently upgraded all of our nodes to version 8.1.2. We also run a Cardano DB Sync instance with one of our nodes with its latest version

cardano-node 8.1.2 - linux-x86_64 - ghc-8.10
git rev d2d90b48c5577b4412d5c9c9968b55f8ab4b9767

Cardano-db-sync 13.1.1.3 - linux-x86_64 - ghc-8.10
git revision 6e69a80797f2d68423b25ca7787e81533b367e42

Now, for the past five days, the DBSync cannot stay up to date. I restored the whole thing from a snapshot.

https://update-cardano-mainnet.iohk.io/cardano-db-sync/13.1/db-sync-snapshot-schema-13.1-block-9492777-x86_64.tgz

I upgraded my PostgreSQL instance to version 15; in addition, I optimized the instance following the Guild Operators instructions

https://cardano-community.github.io/guild-operators/Appendix/postgres/#tuning-your-instance

And for some reason, the system is not capable of staying up to date.

cexplorer=# select now () - max (time) as behind_by from block ;
       behind_by
------------------------
 5 days 15:27:47.289634
(1 row)

cexplorer=# select now () - max (time) as behind_by from block ;
       behind_by
------------------------
 5 days 15:27:51.986572
(1 row)

We have plenty of memory available, as you can see.

image
image1183×167 6.36 KB

Although the DBSync seems like it is not consuming as much memory as it should.

image

Looking at the logs for dbsync and cnode, I don’t see any error. All I see is “Handling input offset …”

dbsync logs

{"app":[],"at":"2023-11-02T13:37:19.29Z","data":{"kind":"LogMessage","message":"Handling input offset 4900000"},"env":"<unknown>:67e42","host":"","loc":null,"msg":"","ns":["db-sync-node"],"pid":"1962038","sev":"Info","thread":"6"}

Node is 100% Synced and fully working. I did notice many incoming connections (47 at this time)

gLiveView
image

We would really appreciate if anyone can shine some light on this problem. It seems many of us are going through the same issue.

Thank you,

2

I also tried cardano-db-sync before and my experience was also not very good. Very expensive to have a full Cardano node up to sync with the whole Cardano network, then get cardano-db-sync to pick up from a snapshot, then hopefully it can catch up with the latest block. Then you need to monitor all these services in case they got disconnected or stop…

While I was searching for a postgresql service which already sync up-to-day (not successful of course), I came across below projec which was funded from Project Catalyst. If cardano-db-sync is a deadend, maybe give it a try?

Gideon

1 Like
3

Thank you @gideont . Yes I saw this project a while ago, In fact, I was looking at this project to make the jump eventually, but I didn’t want another feature creep to come along while I was working on other projects. I was wondering if anybody knew how to solve this issue, otherwise, I’m going to have to make a refactorization to use carp instead of the db sync for our API, which sucks really.

Another thing that I would like to point out is that It isn’t as mature as the dbsync with the snapshots and recovery. It takes up to 5 days to be up to date. I already have a fallback system with a third party, although I don’t like to add these types of dependencies to our system. This is only to keep the service up and running for our users.

Having said that, I met Sebastian a while ago in Austin, and the guy is great besides all of the things he already made (Flint anybody?). I know Carp will eventually surpass dbsync but it just needs time.

4

Curious on what kind of data are you looking to access? I’m developing a customizable indexer called Mafoc where you can configure the data-to-be-indexed, which might greatly help with sync speed and memory use if you only need a subset of what DB Sync indexes.

5

Oh cool! I’ll keep an eye on this @Markus_Lall . Essentially, we are looking to get basic asset information like fingerprints easily. I understand that having an entire indexer just for this is a complete overkill (600 GB of disk storage), although it removes dependencies from 3 parties, which is my concern. It’s interesting to see how many Dapps depend on these services that can potentially fail, which is very often, and therefore creating a single point of failure.

  # Python ORM

  policy_id = "<policyId>"
  hex_name = bytes(str("RandomTokenName").encode()).hex()

  fingerprintQry = MultiAsset.objects.raw(
      f"SELECT * FROM multi_asset WHERE policy = decode('{policy_id}', 'hex') \
          AND name = decode('{hex_name}', 'hex')"
  )

  if len(list(fingerprintQry)):
      for i in fingerprintQry:
          print(i.fingerprint) # Prints: asset1auvcmvw4j8txr8xf4vam9e0atykedk05fkgfjn
6

i Wolfy, have you had any luck with this so far? Your htop and ps screenshots look familiar to my preferences, and I would like to see if there’s anything I can help with.

Markus’s Mafoc project seems interesting too.

I’m a data architect and am interested in drawing insights from a fully synchronized database. However, it seems such a waste to spin up all these resources for occasional instances when I need some answers. Is there any way we can team up to solve these problems?

7

Hi @Wolfy I’ve added an indexer to emit fingerprints, if you have docker you can run it as follows:

docker run --rm -it -v /path/to/node.socket:/socket markus77/mafoc \
   fingerprint  -s /socket --mainnet \
   --fingerprint asset1auvcmvw4j8txr8xf4vam9e0atykedk05fkgfjn \
   --interval alonzo-

Notice that you’ll need to replace /path/to/node.socket with real path to cardano-node’s socket.

The above command starts indexing from the Alonzo era and prints only custom assets with the specified fingerprint. Leaving --fingerprint option off would print every custom asset (this would be quite verbose). You could also specify --policy-id and --asset-name to filter fingerprints.

The output is a line of json for every custom asset, for fingerprint asset1auvcmvw4j8txr8xf4vam9e0atykedk05fkgfjn it would be:

{
  "assetName": "436f6d70657465547261636b546f7465",
  "blockHeaderHash": "9ec2396dcbf90fe3cc4247275e063050ad93207ec9fe48fcb5f70258dfc8cc47",
  "fingerprint": "asset1auvcmvw4j8txr8xf4vam9e0atykedk05fkgfjn",
  "policyId": "737cf2b20bdcc703791e83491493fe87f32ebe0b0b96137baadd913b",
  "slotNo": 105924239
}

EDIT: To get to the asset1auvcmvw4j8txr8xf4vam9e0atykedk05fkgfjn fingerprint really quickly then using --interval 105924203:57a1fa24f5deddb2a67ed0c2af5461333473188ac85130872506d1d2e0b61880+40 will do. It starts from slot 105924203 and indexes 40 slots after that, which effectively means only the next block at slot 105924239. (We do this due to how chainsync protocol works: whatever you specify as the starting point, it starts getting blocks after that point, so we must start from a previous block.)