Cardano db-sync stuck at 95% progress

i’ve been migrating data into the postgres database but have been stuck at 95% for over a day now, here is the error being shown:

[db-sync-node:Error:65518] [2022-03-17 15:13:15.54 UTC] runDBThread: DbInsertException "Many TxIn" (SqlError {sqlState = "40P01", sqlExecStatus = FatalError, sqlErrorMsg = "deadlock detected", sqlErrorDetail = "Process 918356 waits for ShareLock on transaction 12424; blocked by process 1322037.\nProcess 1322037 waits for ShareLock on transaction 12422; blocked by process 918356.", sqlErrorHint = "See server log for query details."})
[db-sync-node:Error:65517] [2022-03-17 15:13:15.54 UTC] recvMsgRollForward: AsyncCancelled
[db-sync-node:Error:65510] [2022-03-17 15:13:15.54 UTC] ChainSyncWithBlocksPtcl: DbInsertException "Many TxIn" (SqlError {sqlState = "40P01", sqlExecStatus = FatalError, sqlErrorMsg = "deadlock detected", sqlErrorDetail = "Process 918356 waits for ShareLock on transaction 12424; blocked by process 1322037.\nProcess 1322037 waits for ShareLock on transaction 12422; blocked by process 918356.", sqlErrorHint = "See server log for query details."})

anyone know how to fix this ShareLock issue? Thanks!

Hi,

perhaps no space left on hdd? Can u check?

1 Like

hey Alex, thanks for the quick response :slight_smile:

disk space seems to be okay

Filesystem      Size  Used Avail Use% Mounted on
udev             34G     0   34G   0% /dev
tmpfs           6.8G  1.4M  6.8G   1% /run
/dev/vda2       405G  186G  199G  49% /
tmpfs            34G  848k   34G   1% /dev/shm
tmpfs           5.3M     0  5.3M   0% /run/lock
tmpfs            34G     0   34G   0% /sys/fs/cgroup
tmpfs           6.8G     0  6.8G   0% /run/user/1000

what seems concerning is this:

"Process 232989 waits for ShareLock on transaction 12548; blocked by process 420119.\nProcess 420119 waits for ShareLock on transaction 12547; blocked by process 232989."

ultimately it keeps rolling back to epoch 310:

[db-sync-node:Info:322] [2022-03-17 16:06:17.27 UTC] Rolling back to slot 48556794, hash b0336f5eee8c433b899661d9c6563d6b86bc1aa8fe29635c25a49976aff3842d
[db-sync-node:Info:322] [2022-03-17 16:06:17.27 UTC] No blocks need to be deleted
[db-sync-node.Subscription:Notice:55] [2022-03-17 16:06:17.94 UTC] Identity Required subscriptions started
[db-sync-node:Info:322] [2022-03-17 16:08:08.05 UTC] Found snapshot file for slot 48556794, hash b0336f5eee8c433b899661d9c6563d6b86bc1aa8fe29635c25a49976aff3842d
[db-sync-node:Info:322] [2022-03-17 16:08:16.39 UTC] Flushing remaining 1002 BulkOperations
[db-sync-node:Info:322] [2022-03-17 16:08:59.56 UTC] insertEpochInterleaved: Epoch 310, 1017022 stake addresses
[db-sync-node:Info:322] [2022-03-17 16:08:59.56 UTC] File ledger-state/mainnet/48556794-b0336f5eee-309.lstate exists
[db-sync-node:Info:322] [2022-03-17 16:09:04.69 UTC] validateEpochRewards: total rewards that become spendable in epoch 310 is 14350297.414804 ADA

memory and CPU seem to be okay as well:

They block each other. Is it possible that you have started cardano-db-sync twice? Can you just kill one of the two?

1 Like

oh wow, i think you’re right
image
what’s the best way to kill these (i’m a linux newb)? just restart? or kill them through htop?

U can testart the server or use

kill -9 process_ID

1 Like

cool, seems to be back to normal, thanks everyone :slight_smile:

1 Like

Since your newish to linux note that -9 here is the SIGKILL switch so be careful with that one :wink:

kill pid is usually enough as the default signal will be SIGTERM or -15 switch. Check out the man page for more details.