Parsing ledger-state stucks with cntools

Hi all,
trying to parsing ledger-state with cntools (pool > show) but it stucks. I got a couple of timeout errors, so I’ve doubled timeout time, but still no luck.
Node 1.25.1
Any ideas? How long usually it takes?

Your node is synced? What is the output?

Cheers

Schermata da 2021-02-11 18-25-18
This is what is going on on my pool.
Do you think all is fine or I miss somthing?

You don’t have tx processing… do u run topology updater on relay?

What is ur ticker?

Yes I do
Schermata da 2021-02-11 18-33-45 Schermata da 2021-02-11 18-34-22

My ticker is ITEU1
Do you think something is wrong? Firewall?

What is the last message from topology updater log?

{ “resultcode”: “502”, “datetime”:“2021-02-11 17:04:02”, “clientIp”: “2a02:7b40:5928:998::1”, “msg”: “invalid blockNo ” }

Ok, check ur config file

Can u paste it here?

As you mention here the configs must be the same on the pool and relay?

I hope you mean the relay one… Here it is. Anyway can you explain?

{
“ApplicationName”: “cardano-sl”,
“ApplicationVersion”: 1,
“ByronGenesisFile”: “mainnet-byron-genesis.json”,
“ByronGenesisHash”: “5f20df933584822601f9e3f8c024eb5eb252fe8cefb24d1317dc3d432e940ebb”,
“LastKnownBlockVersion-Alt”: 0,
“LastKnownBlockVersion-Major”: 3,
“LastKnownBlockVersion-Minor”: 0,
“MaxKnownMajorProtocolVersion”: 2,
“Protocol”: “Cardano”,
“RequiresNetworkMagic”: “RequiresNoMagic”,
“ShelleyGenesisFile”: “mainnet-shelley-genesis.json”,
“ShelleyGenesisHash”: “1a3be38bcbb7911969283716ad7aa550250226b76a61fc51cc9a9a35d9276d81”,
“TraceBlockFetchClient”: false,
“TraceBlockFetchDecisions”: false,
“TraceBlockFetchProtocol”: false,
“TraceBlockFetchProtocolSerialised”: false,
“TraceBlockFetchServer”: false,
“TraceChainDb”: true,
“TraceChainSyncBlockServer”: false,
“TraceChainSyncClient”: false,
“TraceChainSyncHeaderServer”: false,
“TraceChainSyncProtocol”: false,
“TraceDNSResolver”: true,
“TraceDNSSubscription”: true,
“TraceErrorPolicy”: true,
“TraceForge”: true,
“TraceHandshake”: false,
“TraceIpSubscription”: true,
“TraceLocalChainSyncProtocol”: false,
“TraceLocalErrorPolicy”: true,
“TraceLocalHandshake”: false,
“TraceLocalTxSubmissionProtocol”: false,
“TraceLocalTxSubmissionServer”: false,
“TraceMempool”: true,
“TraceMux”: false,
“TraceTxInbound”: false,
“TraceTxOutbound”: false,
“TraceTxSubmissionProtocol”: false,
“TracingVerbosity”: “NormalVerbosity”,
“TurnOnLogMetrics”: true,
“TurnOnLogging”: true,
“defaultBackends”: [
“KatipBK”
],
“defaultScribes”: [
[
“StdoutSK”,
“stdout”
]
],
“hasEKG”: 12788,
“hasPrometheus”: [
“127.0.0.1”,
12798
],
“minSeverity”: “Info”,
“options”: {
“mapBackends”: {
“cardano.node.metrics”: [
“EKGViewBK”
],
“cardano.node.resources”: [
“EKGViewBK”
]
},
“mapSubtrace”: {
“cardano.node.metrics”: {
“subtrace”: “Neutral”
}
}
},
“rotation”: {
“rpKeepFilesNum”: 10,
“rpLogLimitBytes”: 5000000,
“rpMaxAgeHours”: 24
},
“setupBackends”: [
“KatipBK”
],
“setupScribes”: [
{
“scFormat”: “ScText”,
“scKind”: “StdoutSK”,
“scName”: “stdout”,
“scRotation”: null
}
]
}

The config looks fine but about err 502… I don’t know the reason… can u trry to run the script manually?

After, check the message from topology log file

topologyUpdater.sh needed an update, then I’ve change curl -4 for dual network: now I got a good response

{ “resultcode”: “201”, “datetime”:“2021-02-11 21:47:36”, “clientIp”: “89.40.9.152”, “iptype”: 4, “msg”: “nice to meet you” }

Still I can’t show my pool with cntools. You know what I noticed: when I run pool → show gLiveView disconnected I guess my pool crash or restart. That’s weird. Any idea? Low RAM?

How long show pool takes usually?

keep the topology updater running only on relays… after 4 runs (3hours) u should see glade for staying with us!

Ur nodes are fully synced?

It runs on relays only and yes fully sync.
I got this from the topologyupdater log

{ “resultcode”: “201”, “datetime”:“2021-02-11 21:47:36”, “clientIp”: “89.40.9.152”, “iptype”: 4, “msg”: “nice to meet you” }
{ “resultcode”: “504”, “datetime”:“2021-02-11 22:04:19”, “clientIp”: “89.40.9.152”, “iptype”: 4, “msg”: “one request per hour please” }
{ “resultcode”: “502”, “datetime”:“2021-02-11 22:04:33”, “clientIp”: “89.40.9.152”, “msg”: “invalid blockNo ” }

Yes, 1 per hour… are u run it mannualy or in crontab?

It runs with systemctl. I’ve checked this morning (more then 4 hours waiting) and the log/archive/ says

{ “resultcode”: “502”, “datetime”:“2021-02-12 07:04:52”, “clientIp”: “89.40.9.152”, “msg”: “invalid blockNo ” }

so I run topologyUpdater.sh manually (by the way it always asks me to update it because I’ve added -4 to curls) and I got

{ “resultcode”: “201”, “datetime”:“2021-02-12 07:15:25”, “clientIp”: “89.40.9.152”, “iptype”: 4, “msg”: “nice to meet you” }

in /log/ outside archive folder.
Something is no working properly I guess… Do you think is better cronjob it? And see what happens?

Yes, something wrong with crontab… try to edit all full path in the crontab for topology updater script

I saw now, or the issue is that the topology updater always ask you to updated…

For me topologyupdater works fine, is this

“invalid blockNo

that looks like what I get from

curl -s -H ‘Accept: application/json’ “http://127.0.0.1:12788/

which is a curl in the topologyupdater.sh, is a wrong object or its data are not correct for updater API.
Do you think I must redownload configs files? I mean here Cardano Configurations

Anyway the object is this

{“cardano”:{“node”:{“metrics”:{“blockNum”:{“int”:{“type”:“g”,“val”:5332038}},“slotInEpoch”:{“int”:{“type”:“g”,“val”:211019}},“slotNum”:{“int”:{“type”:“g”,“val”:21551819}},“Mem”:{“resident”:{“int”:{“type”:“g”,“val”:1463762944}}},“nodeStartTime”:{“int”:{“type”:“g”,“val”:1613117104}},“Stat”:{“cputicks”:{“int”:{“type”:“g”,“val”:12442}},“threads”:{“int”:{“type”:“g”,“val”:15}}},“RTS”:{“gcticks”:{“int”:{“type”:“g”,“val”:1187}},“gcMinorNum”:{“int”:{“type”:“g”,“val”:887}},“gcMajorNum”:{“int”:{“type”:“g”,“val”:17}},“mutticks”:{“int”:{“type”:“g”,“val”:11253}},“gcLiveBytes”:{“int”:{“type”:“g”,“val”:875926280}}},“density”:{“real”:{“type”:“l”,“val”:“4.97581202487906e-2”}},“connectedPeers”:{“int”:{“type”:“g”,“val”:22}},“epoch”:{“int”:{“type”:“g”,“val”:247}}}}},“iohk-monitoring version”:{“type”:“l”,“val”:“0.1.10.1”},“ekg”:{“server_timestamp_ms”:{“type”:“c”,“val”:1613118118557}},“rts”:{“gc”:{“gc_cpu_ms”:{“type”:“c”,“val”:11877},“mutator_wall_ms”:{“type”:“c”,“val”:1007794},“mutator_cpu_ms”:{“type”:“c”,“val”:112543},“gc_wall_ms”:{“type”:“c”,“val”:9457},“wall_ms”:{“type”:“c”,“val”:1017252},“bytes_copied”:{“type”:“c”,“val”:3672296272},“init_wall_ms”:{“type”:“c”,“val”:9},“init_cpu_ms”:{“type”:“c”,“val”:8},“max_bytes_used”:{“type”:“g”,“val”:572770520},“max_bytes_slop”:{“type”:“g”,“val”:3419848},“num_bytes_usage_samples”:{“type”:“c”,“val”:17},“peak_megabytes_allocated”:{“type”:“g”,“val”:1353},“cpu_ms”:{“type”:“c”,“val”:124420},“current_bytes_used”:{“type”:“g”,“val”:875926280},“bytes_allocated”:{“type”:“c”,“val”:26174335608},“par_max_bytes_copied”:{“type”:“g”,“val”:3360845448},“current_bytes_slop”:{“type”:“g”,“val”:7474424},“cumulative_bytes_used”:{“type”:“c”,“val”:1420869128},“num_gcs”:{“type”:“c”,“val”:904},“par_tot_bytes_copied”:{“type”:“g”,“val”:3672296272},“par_avg_bytes_copied”:{“type”:“g”,“val”:3672296272}}}}

what IP do you have set in config file is the same? 127.0.0.0 ? how about the port? the same 12788?

Lines are commented in env and in config.json

“hasEKG”: 12788,
“hasPrometheus”: [
“127.0.0.1”,
12798
],

only port is set.

should be fine… possible to be something related with the script
can you download again?