Parsing ledger-state stucks with cntools

You don’t have tx processing… do u run topology updater on relay?

What is ur ticker?

Yes I do
Schermata da 2021-02-11 18-33-45 Schermata da 2021-02-11 18-34-22

My ticker is ITEU1
Do you think something is wrong? Firewall?

What is the last message from topology updater log?

{ “resultcode”: “502”, “datetime”:“2021-02-11 17:04:02”, “clientIp”: “2a02:7b40:5928:998::1”, “msg”: “invalid blockNo ” }

Ok, check ur config file

Can u paste it here?

As you mention here the configs must be the same on the pool and relay?

I hope you mean the relay one… Here it is. Anyway can you explain?

{
“ApplicationName”: “cardano-sl”,
“ApplicationVersion”: 1,
“ByronGenesisFile”: “mainnet-byron-genesis.json”,
“ByronGenesisHash”: “5f20df933584822601f9e3f8c024eb5eb252fe8cefb24d1317dc3d432e940ebb”,
“LastKnownBlockVersion-Alt”: 0,
“LastKnownBlockVersion-Major”: 3,
“LastKnownBlockVersion-Minor”: 0,
“MaxKnownMajorProtocolVersion”: 2,
“Protocol”: “Cardano”,
“RequiresNetworkMagic”: “RequiresNoMagic”,
“ShelleyGenesisFile”: “mainnet-shelley-genesis.json”,
“ShelleyGenesisHash”: “1a3be38bcbb7911969283716ad7aa550250226b76a61fc51cc9a9a35d9276d81”,
“TraceBlockFetchClient”: false,
“TraceBlockFetchDecisions”: false,
“TraceBlockFetchProtocol”: false,
“TraceBlockFetchProtocolSerialised”: false,
“TraceBlockFetchServer”: false,
“TraceChainDb”: true,
“TraceChainSyncBlockServer”: false,
“TraceChainSyncClient”: false,
“TraceChainSyncHeaderServer”: false,
“TraceChainSyncProtocol”: false,
“TraceDNSResolver”: true,
“TraceDNSSubscription”: true,
“TraceErrorPolicy”: true,
“TraceForge”: true,
“TraceHandshake”: false,
“TraceIpSubscription”: true,
“TraceLocalChainSyncProtocol”: false,
“TraceLocalErrorPolicy”: true,
“TraceLocalHandshake”: false,
“TraceLocalTxSubmissionProtocol”: false,
“TraceLocalTxSubmissionServer”: false,
“TraceMempool”: true,
“TraceMux”: false,
“TraceTxInbound”: false,
“TraceTxOutbound”: false,
“TraceTxSubmissionProtocol”: false,
“TracingVerbosity”: “NormalVerbosity”,
“TurnOnLogMetrics”: true,
“TurnOnLogging”: true,
“defaultBackends”: [
“KatipBK”
],
“defaultScribes”: [
[
“StdoutSK”,
“stdout”
]
],
“hasEKG”: 12788,
“hasPrometheus”: [
“127.0.0.1”,
12798
],
“minSeverity”: “Info”,
“options”: {
“mapBackends”: {
“cardano.node.metrics”: [
“EKGViewBK”
],
“cardano.node.resources”: [
“EKGViewBK”
]
},
“mapSubtrace”: {
“cardano.node.metrics”: {
“subtrace”: “Neutral”
}
}
},
“rotation”: {
“rpKeepFilesNum”: 10,
“rpLogLimitBytes”: 5000000,
“rpMaxAgeHours”: 24
},
“setupBackends”: [
“KatipBK”
],
“setupScribes”: [
{
“scFormat”: “ScText”,
“scKind”: “StdoutSK”,
“scName”: “stdout”,
“scRotation”: null
}
]
}

The config looks fine but about err 502… I don’t know the reason… can u trry to run the script manually?

After, check the message from topology log file

topologyUpdater.sh needed an update, then I’ve change curl -4 for dual network: now I got a good response

{ “resultcode”: “201”, “datetime”:“2021-02-11 21:47:36”, “clientIp”: “89.40.9.152”, “iptype”: 4, “msg”: “nice to meet you” }

Still I can’t show my pool with cntools. You know what I noticed: when I run pool → show gLiveView disconnected I guess my pool crash or restart. That’s weird. Any idea? Low RAM?

How long show pool takes usually?

keep the topology updater running only on relays… after 4 runs (3hours) u should see glade for staying with us!

Ur nodes are fully synced?

It runs on relays only and yes fully sync.
I got this from the topologyupdater log

{ “resultcode”: “201”, “datetime”:“2021-02-11 21:47:36”, “clientIp”: “89.40.9.152”, “iptype”: 4, “msg”: “nice to meet you” }
{ “resultcode”: “504”, “datetime”:“2021-02-11 22:04:19”, “clientIp”: “89.40.9.152”, “iptype”: 4, “msg”: “one request per hour please” }
{ “resultcode”: “502”, “datetime”:“2021-02-11 22:04:33”, “clientIp”: “89.40.9.152”, “msg”: “invalid blockNo ” }

Yes, 1 per hour… are u run it mannualy or in crontab?

It runs with systemctl. I’ve checked this morning (more then 4 hours waiting) and the log/archive/ says

{ “resultcode”: “502”, “datetime”:“2021-02-12 07:04:52”, “clientIp”: “89.40.9.152”, “msg”: “invalid blockNo ” }

so I run topologyUpdater.sh manually (by the way it always asks me to update it because I’ve added -4 to curls) and I got

{ “resultcode”: “201”, “datetime”:“2021-02-12 07:15:25”, “clientIp”: “89.40.9.152”, “iptype”: 4, “msg”: “nice to meet you” }

in /log/ outside archive folder.
Something is no working properly I guess… Do you think is better cronjob it? And see what happens?

Yes, something wrong with crontab… try to edit all full path in the crontab for topology updater script

I saw now, or the issue is that the topology updater always ask you to updated…

For me topologyupdater works fine, is this

“invalid blockNo

that looks like what I get from

curl -s -H ‘Accept: application/json’ “http://127.0.0.1:12788/

which is a curl in the topologyupdater.sh, is a wrong object or its data are not correct for updater API.
Do you think I must redownload configs files? I mean here https://hydra.iohk.io/build/5367762/download/1/index.html

Anyway the object is this

{“cardano”:{“node”:{“metrics”:{“blockNum”:{“int”:{“type”:“g”,“val”:5332038}},“slotInEpoch”:{“int”:{“type”:“g”,“val”:211019}},“slotNum”:{“int”:{“type”:“g”,“val”:21551819}},“Mem”:{“resident”:{“int”:{“type”:“g”,“val”:1463762944}}},“nodeStartTime”:{“int”:{“type”:“g”,“val”:1613117104}},“Stat”:{“cputicks”:{“int”:{“type”:“g”,“val”:12442}},“threads”:{“int”:{“type”:“g”,“val”:15}}},“RTS”:{“gcticks”:{“int”:{“type”:“g”,“val”:1187}},“gcMinorNum”:{“int”:{“type”:“g”,“val”:887}},“gcMajorNum”:{“int”:{“type”:“g”,“val”:17}},“mutticks”:{“int”:{“type”:“g”,“val”:11253}},“gcLiveBytes”:{“int”:{“type”:“g”,“val”:875926280}}},“density”:{“real”:{“type”:“l”,“val”:“4.97581202487906e-2”}},“connectedPeers”:{“int”:{“type”:“g”,“val”:22}},“epoch”:{“int”:{“type”:“g”,“val”:247}}}}},“iohk-monitoring version”:{“type”:“l”,“val”:“0.1.10.1”},“ekg”:{“server_timestamp_ms”:{“type”:“c”,“val”:1613118118557}},“rts”:{“gc”:{“gc_cpu_ms”:{“type”:“c”,“val”:11877},“mutator_wall_ms”:{“type”:“c”,“val”:1007794},“mutator_cpu_ms”:{“type”:“c”,“val”:112543},“gc_wall_ms”:{“type”:“c”,“val”:9457},“wall_ms”:{“type”:“c”,“val”:1017252},“bytes_copied”:{“type”:“c”,“val”:3672296272},“init_wall_ms”:{“type”:“c”,“val”:9},“init_cpu_ms”:{“type”:“c”,“val”:8},“max_bytes_used”:{“type”:“g”,“val”:572770520},“max_bytes_slop”:{“type”:“g”,“val”:3419848},“num_bytes_usage_samples”:{“type”:“c”,“val”:17},“peak_megabytes_allocated”:{“type”:“g”,“val”:1353},“cpu_ms”:{“type”:“c”,“val”:124420},“current_bytes_used”:{“type”:“g”,“val”:875926280},“bytes_allocated”:{“type”:“c”,“val”:26174335608},“par_max_bytes_copied”:{“type”:“g”,“val”:3360845448},“current_bytes_slop”:{“type”:“g”,“val”:7474424},“cumulative_bytes_used”:{“type”:“c”,“val”:1420869128},“num_gcs”:{“type”:“c”,“val”:904},“par_tot_bytes_copied”:{“type”:“g”,“val”:3672296272},“par_avg_bytes_copied”:{“type”:“g”,“val”:3672296272}}}}

what IP do you have set in config file is the same? 127.0.0.0 ? how about the port? the same 12788?

Lines are commented in env and in config.json

“hasEKG”: 12788,
“hasPrometheus”: [
“127.0.0.1”,
12798
],

only port is set.

should be fine… possible to be something related with the script
can you download again?

Which script?

topology updater? I don’t think it’s necessary do download again the configuration file (but u can try) because when u ran the script manually it worked… but when u ran with crontab it doesn’t

Updater works. What is wrong is the blockNo that

https://api.clio.one/htopology/v1/

get from topologyupdater. What this curl request pass is these variables

?port=6000&blockNo=5334296&valency=1&magic=764824073

Now I’m pretty sure about port and valency, magic comes from this

jq -r .networkMagic < mainnet-shelley-genesis.json

and blockNo from

curl -s -m ${EKG_TIMEOUT} -H ‘Accept: application/json’ “http://${EKG_HOST}:${EKG_PORT}/” 2>/dev/null | jq -er ‘.cardano.node.metrics.blockNum.int.val //0’

I guess something wrong from cardano.node.metrics.blockNum.int.val. Any idea from where this comes from?