Try to search for this one 49778704
Can you check the hash of your VRF verification key:
cardano-cli node key-hash-VRF --verification-key-file vrf.vkey
and compare it to the one displayed on cardanoscan.io for your stake pool (“Vrf Hash” field)? Are they the same?
Yes. It is the same
$ cardano-cli node key-hash-VRF --verification-key-file vrf.vkey
e52cd41c94e36f6e4bfce6cdff08dd83cacfd123c44dc323d97a67ba9829fd89
When I missed block in testnet, for me was very usefull to search for that time in logs on both relays and producer.
Something like this:
more ./cardano.log | grep "2021-12-31 08:32"
In my case I had a problem with connectivity from relay to BP, so BP minted block correctly, but it wasn’t pulled.
I would recommend to search for time frame when block is minted vs time frame when it doesn’t and then compare those logs, should be quite straightforward.
What grafana shows, by the way? Do you still have 100% as pool performance?
Are you running chrony on all the nodes (block producer and relay(s))?
I get this results but I can’t find the problem 
cat node0-20220104* | grep "2022-01-05T01:09:55"
{"thread":"208","sev":"Info","data":{"kind":"TraceStartLeadershipCheck","chainDensity":4.8247967e-2,"slot":49778704,"delegMapSize":1029624,"utxoSize":4731961,"credentials":"Cardano"},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"producer","at":"2022-01-05T01:09:55.00Z","ns":["cardano.node.LeadershipCheck"],"pid":"12314"}
{"thread":"208","sev":"Info","data":{"val":{"kind":"TraceNodeNotLeader","slot":49778704},"credentials":"Cardano"},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"producer","at":"2022-01-05T01:09:55.00Z","ns":["cardano.node.Forge"],"pid":"12314"}
{"thread":"344919","sev":"Info","data":{"mempoolSize":{"bytes":65441,"numTxs":48},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"ce8e61bba55381f5b132c025657f965367f6bf02a1c058aaf5d10849b7534919\"}"},"kind":"TraceMempoolAddedTx"},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"producer","at":"2022-01-05T01:09:55.07Z","ns":["cardano.node.Mempool"],"pid":"12314"}
{"thread":"344919","sev":"Info","data":{"mempoolSize":{"bytes":66145,"numTxs":49},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"77afb15e02568da5c401477a87f1c321402905116469109eceaaf3fa21a3a253\"}"},"kind":"TraceMempoolAddedTx"},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"producer","at":"2022-01-05T01:09:55.11Z","ns":["cardano.node.Mempool"],"pid":"12314"}
{"thread":"339274","sev":"Info","data":{"mempoolSize":{"bytes":66145,"numTxs":49},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"ce8e61bba55381f5b132c025657f965367f6bf02a1c058aaf5d10849b7534919\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":2049478087},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["01123543180f03b20df190a7e4b8110ce898cbaf415f9280176005f6563bad9a#1"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 2049478087 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"producer","at":"2022-01-05T01:09:55.13Z","ns":["cardano.node.Mempool"],"pid":"12314"}
{"thread":"344919","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"179ae7305cf8d4a4973a09b8fe87eb0da69c12ec9c2a31bd4b1341a5f9dcc979\"}"},"kind":"TraceMempoolAddedTx"},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"producer","at":"2022-01-05T01:09:55.23Z","ns":["cardano.node.Mempool"],"pid":"12314"}
{"thread":"339274","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"179ae7305cf8d4a4973a09b8fe87eb0da69c12ec9c2a31bd4b1341a5f9dcc979\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26":{"586d617332303231443031333131":1},"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3":{"4c4f4253544552":1000000}},"lovelace":160795573},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["1d4cd2b3fb0b65fe1a01495278aa982eb048812f1409a2ddd1335e657cb97df8#1","55da61d71f7731763c5b7691c7f65e08689abd06b247e04ccded90522f40ebb1#0","bc3c24fe3fc2a67491fd561c67f2a442a22d4271b4b15fa592b272595e291723#3"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 160795573 (fromList [(PolicyID {policyID = ScriptHash \"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26\"},fromList [(\"Xmas2021D01311\",1)]),(PolicyID {policyID = ScriptHash \"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3\"},fromList [(\"LOBSTER\",1000000)])])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"producer","at":"2022-01-05T01:09:55.28Z","ns":["cardano.node.Mempool"],"pid":"12314"}
{"thread":"344919","sev":"Info","data":{"mempoolSize":{"bytes":67243,"numTxs":51},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"e65cf24e751f35ad0b8325c8d22ced7cd83ea115d167a821dca987b43f6326d3\"}"},"kind":"TraceMempoolAddedTx"},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"producer","at":"2022-01-05T01:09:55.98Z","ns":["cardano.node.Mempool"],"pid":"12314"}
I checked the connectivity and it is fine in both sides (relays to bp, bp to relays)
I don’t have performance metric in grafana. What dashboard do you use?
This is the relevant part from the logs:
{“thread”:“208”,“sev”:“Info”,“data”:{“val”:{“kind”:“TraceNodeNotLeader”,“slot”:49778704},“credentials”:“Cardano”},“loc”:null,“env”:“1.32.1:4f65f”,“msg”:"",“app”:,“host”:“producer”,“at”:“2022-01-05T01:09:55.00Z”,“ns”:[“cardano.node.Forge”],“pid”:“12314”}
Your Block Produced did not know he is the slot leader. This probably neans that the VRF signature key is not correct. You already confirmed it is correct, but I guess you took the VRF verification key from somewhere else, not from the block producer.
Please do the following:
- take a look at the startup script (for the cardano-node service) and see exactly which VRF sign key it is specified as parameter.
- run the following commands with that vrf.skey from the startup script:
cardano-cli key verification-key --signing-key-file vrf.skey --verification-key-file /tmp/vrf.vkey
cardano-cli node key-hash-VRF --verification-key-file /tmp/vrf.vkey
and check if the output is tha same one as before (e52cd41c94e36f6e4bfce6cdff08dd83cacfd123c44dc323d97a67ba9829fd89).
I don’t see any messages that block was minted, so this is something not ok, I would assume. Are you sure that times are correct?
What you have in logs when you do mint log successfully?
Can you post your logs from relay at that time?
Can you run this on your BP?
curl -s localhost:12798/metrics | grep cardano_node_metrics_Forge
I don’t think the key is the problem
$ cardano-cli key verification-key --signing-key-file vrf.skey --verification-key-file /tmp/vrf.vkey
$ cardano-cli node key-hash-VRF --verification-key-file /tmp/vrf.vkey
e52cd41c94e36f6e4bfce6cdff08dd83cacfd123c44dc323d97a67ba9829fd89
Absolutely. I use chrony but also I checked it manually today
I found many of this only in one relay (relay2). Total 358471 lines!
{"thread":"32456","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"77afb15e02568da5c401477a87f1c321402905116469109eceaaf3fa21a3a253\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":593960842},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["29608e905240c781827e10adafbd89d087afc2bbdccfeeec1ba10100c36188d9#0","687dc5113e0afeba26241f224148d7c7910a2faf802ef2f655ae01f83eb28ca3#1","8c2e816adf8a81b2790b6bdc3616aba8f25c65c373472a1f7112015e2addf17a#1","e24c8fec017a324787e24c0f16cd1699d8418cb0d2b7f4545fd08352aeb1287c#0"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 593960842 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay2","at":"2022-01-05T01:09:55.48Z","ns":["cardano.node.Mempool"],"pid":"228770"}
{"thread":"32456","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"179ae7305cf8d4a4973a09b8fe87eb0da69c12ec9c2a31bd4b1341a5f9dcc979\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26":{"586d617332303231443031333131":1},"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3":{"4c4f4253544552":1000000}},"lovelace":160795573},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["1d4cd2b3fb0b65fe1a01495278aa982eb048812f1409a2ddd1335e657cb97df8#1","55da61d71f7731763c5b7691c7f65e08689abd06b247e04ccded90522f40ebb1#0","bc3c24fe3fc2a67491fd561c67f2a442a22d4271b4b15fa592b272595e291723#3"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 160795573 (fromList [(PolicyID {policyID = ScriptHash \"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26\"},fromList [(\"Xmas2021D01311\",1)]),(PolicyID {policyID = ScriptHash \"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3\"},fromList [(\"LOBSTER\",1000000)])])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay2","at":"2022-01-05T01:09:55.48Z","ns":["cardano.node.Mempool"],"pid":"228770"}
{"thread":"28799","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"ce8e61bba55381f5b132c025657f965367f6bf02a1c058aaf5d10849b7534919\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":2049478087},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["01123543180f03b20df190a7e4b8110ce898cbaf415f9280176005f6563bad9a#1"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 2049478087 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay2","at":"2022-01-05T01:09:55.48Z","ns":["cardano.node.Mempool"],"pid":"228770"}
{"thread":"28799","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"77afb15e02568da5c401477a87f1c321402905116469109eceaaf3fa21a3a253\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":593960842},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["29608e905240c781827e10adafbd89d087afc2bbdccfeeec1ba10100c36188d9#0","687dc5113e0afeba26241f224148d7c7910a2faf802ef2f655ae01f83eb28ca3#1","8c2e816adf8a81b2790b6bdc3616aba8f25c65c373472a1f7112015e2addf17a#1","e24c8fec017a324787e24c0f16cd1699d8418cb0d2b7f4545fd08352aeb1287c#0"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 593960842 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay2","at":"2022-01-05T01:09:55.48Z","ns":["cardano.node.Mempool"],"pid":"228770"}
{"thread":"28799","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"179ae7305cf8d4a4973a09b8fe87eb0da69c12ec9c2a31bd4b1341a5f9dcc979\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26":{"586d617332303231443031333131":1},"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3":{"4c4f4253544552":1000000}},"lovelace":160795573},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["1d4cd2b3fb0b65fe1a01495278aa982eb048812f1409a2ddd1335e657cb97df8#1","55da61d71f7731763c5b7691c7f65e08689abd06b247e04ccded90522f40ebb1#0","bc3c24fe3fc2a67491fd561c67f2a442a22d4271b4b15fa592b272595e291723#3"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 160795573 (fromList [(PolicyID {policyID = ScriptHash \"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26\"},fromList [(\"Xmas2021D01311\",1)]),(PolicyID {policyID = ScriptHash \"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3\"},fromList [(\"LOBSTER\",1000000)])])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay2","at":"2022-01-05T01:09:55.48Z","ns":["cardano.node.Mempool"],"pid":"228770"}
{"thread":"41538","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"ce8e61bba55381f5b132c025657f965367f6bf02a1c058aaf5d10849b7534919\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":2049478087},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["01123543180f03b20df190a7e4b8110ce898cbaf415f9280176005f6563bad9a#1"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 2049478087 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay2","at":"2022-01-05T01:09:55.48Z","ns":["cardano.node.Mempool"],"pid":"228770"}
{"thread":"41538","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"77afb15e02568da5c401477a87f1c321402905116469109eceaaf3fa21a3a253\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":593960842},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["29608e905240c781827e10adafbd89d087afc2bbdccfeeec1ba10100c36188d9#0","687dc5113e0afeba26241f224148d7c7910a2faf802ef2f655ae01f83eb28ca3#1","8c2e816adf8a81b2790b6bdc3616aba8f25c65c373472a1f7112015e2addf17a#1","e24c8fec017a324787e24c0f16cd1699d8418cb0d2b7f4545fd08352aeb1287c#0"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 593960842 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay2","at":"2022-01-05T01:09:55.48Z","ns":["cardano.node.Mempool"],"pid":"228770"}
{"thread":"41538","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"179ae7305cf8d4a4973a09b8fe87eb0da69c12ec9c2a31bd4b1341a5f9dcc979\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26":{"586d617332303231443031333131":1},"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3":{"4c4f4253544552":1000000}},"lovelace":160795573},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["1d4cd2b3fb0b65fe1a01495278aa982eb048812f1409a2ddd1335e657cb97df8#1","55da61d71f7731763c5b7691c7f65e08689abd06b247e04ccded90522f40ebb1#0","bc3c24fe3fc2a67491fd561c67f2a442a22d4271b4b15fa592b272595e291723#3"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 160795573 (fromList [(PolicyID {policyID = ScriptHash \"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26\"},fromList [(\"Xmas2021D01311\",1)]),(PolicyID {policyID = ScriptHash \"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3\"},fromList [(\"LOBSTER\",1000000)])])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay2","at":"2022-01-05T01:09:55.48Z","ns":["cardano.node.Mempool"],"pid":"228770"}
{"thread":"33763","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"ce8e61bba55381f5b132c025657f965367f6bf02a1c058aaf5d10849b7534919\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":2049478087},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["01123543180f03b20df190a7e4b8110ce898cbaf415f9280176005f6563bad9a#1"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 2049478087 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay2","at":"2022-01-05T01:09:55.48Z","ns":["cardano.node.Mempool"],"pid":"228770"}
{"thread":"33763","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"77afb15e02568da5c401477a87f1c321402905116469109eceaaf3fa21a3a253\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":593960842},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["29608e905240c781827e10adafbd89d087afc2bbdccfeeec1ba10100c36188d9#0","687dc5113e0afeba26241f224148d7c7910a2faf802ef2f655ae01f83eb28ca3#1","8c2e816adf8a81b2790b6bdc3616aba8f25c65c373472a1f7112015e2addf17a#1","e24c8fec017a324787e24c0f16cd1699d8418cb0d2b7f4545fd08352aeb1287c#0"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 593960842 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay2","at":"2022-01-05T01:09:55.48Z","ns":["cardano.node.Mempool"],"pid":"228770"}
{"thread":"33763","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"179ae7305cf8d4a4973a09b8fe87eb0da69c12ec9c2a31bd4b1341a5f9dcc979\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26":{"586d617332303231443031333131":1},"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3":{"4c4f4253544552":1000000}},"lovelace":160795573},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["1d4cd2b3fb0b65fe1a01495278aa982eb048812f1409a2ddd1335e657cb97df8#1","55da61d71f7731763c5b7691c7f65e08689abd06b247e04ccded90522f40ebb1#0","bc3c24fe3fc2a67491fd561c67f2a442a22d4271b4b15fa592b272595e291723#3"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 160795573 (fromList [(PolicyID {policyID = ScriptHash \"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26\"},fromList [(\"Xmas2021D01311\",1)]),(PolicyID {policyID = ScriptHash \"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3\"},fromList [(\"LOBSTER\",1000000)])])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay2","at":"2022-01-05T01:09:55.48Z","ns":["cardano.node.Mempool"],"pid":"228770"}
{"thread":"33155","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"ce8e61bba55381f5b132c025657f965367f6bf02a1c058aaf5d10849b7534919\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":2049478087},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["01123543180f03b20df190a7e4b8110ce898cbaf415f9280176005f6563bad9a#1"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 2049478087 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay2","at":"2022-01-05T01:09:55.49Z","ns":["cardano.node.Mempool"],"pid":"228770"}
The other relay (relay1) has only this at the same time (13 lines)
{"thread":"5095","sev":"Info","data":{"mempoolSize":{"bytes":65441,"numTxs":48},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"ce8e61bba55381f5b132c025657f965367f6bf02a1c058aaf5d10849b7534919\"}"},"kind":"TraceMempoolAddedTx"},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.13Z","ns":["cardano.node.Mempool"],"pid":"39759"}
{"thread":"546","sev":"Info","data":{"mempoolSize":{"bytes":65441,"numTxs":48},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"ce8e61bba55381f5b132c025657f965367f6bf02a1c058aaf5d10849b7534919\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":2049478087},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["01123543180f03b20df190a7e4b8110ce898cbaf415f9280176005f6563bad9a#1"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 2049478087 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.14Z","ns":["cardano.node.Mempool"],"pid":"39759"}
{"thread":"330","sev":"Info","data":{"mempoolSize":{"bytes":65441,"numTxs":48},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"ce8e61bba55381f5b132c025657f965367f6bf02a1c058aaf5d10849b7534919\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":2049478087},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["01123543180f03b20df190a7e4b8110ce898cbaf415f9280176005f6563bad9a#1"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 2049478087 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.15Z","ns":["cardano.node.Mempool"],"pid":"39759"}
{"thread":"6174","sev":"Info","data":{"mempoolSize":{"bytes":66145,"numTxs":49},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"77afb15e02568da5c401477a87f1c321402905116469109eceaaf3fa21a3a253\"}"},"kind":"TraceMempoolAddedTx"},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.17Z","ns":["cardano.node.Mempool"],"pid":"39759"}
{"thread":"5095","sev":"Info","data":{"mempoolSize":{"bytes":66145,"numTxs":49},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"77afb15e02568da5c401477a87f1c321402905116469109eceaaf3fa21a3a253\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":593960842},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["29608e905240c781827e10adafbd89d087afc2bbdccfeeec1ba10100c36188d9#0","687dc5113e0afeba26241f224148d7c7910a2faf802ef2f655ae01f83eb28ca3#1","8c2e816adf8a81b2790b6bdc3616aba8f25c65c373472a1f7112015e2addf17a#1","e24c8fec017a324787e24c0f16cd1699d8418cb0d2b7f4545fd08352aeb1287c#0"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 593960842 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.17Z","ns":["cardano.node.Mempool"],"pid":"39759"}
{"thread":"716","sev":"Info","data":{"mempoolSize":{"bytes":66145,"numTxs":49},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"ce8e61bba55381f5b132c025657f965367f6bf02a1c058aaf5d10849b7534919\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":2049478087},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["01123543180f03b20df190a7e4b8110ce898cbaf415f9280176005f6563bad9a#1"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 2049478087 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.19Z","ns":["cardano.node.Mempool"],"pid":"39759"}
{"thread":"546","sev":"Info","data":{"mempoolSize":{"bytes":66145,"numTxs":49},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"77afb15e02568da5c401477a87f1c321402905116469109eceaaf3fa21a3a253\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":593960842},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["29608e905240c781827e10adafbd89d087afc2bbdccfeeec1ba10100c36188d9#0","687dc5113e0afeba26241f224148d7c7910a2faf802ef2f655ae01f83eb28ca3#1","8c2e816adf8a81b2790b6bdc3616aba8f25c65c373472a1f7112015e2addf17a#1","e24c8fec017a324787e24c0f16cd1699d8418cb0d2b7f4545fd08352aeb1287c#0"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 593960842 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.20Z","ns":["cardano.node.Mempool"],"pid":"39759"}
{"thread":"330","sev":"Info","data":{"mempoolSize":{"bytes":66145,"numTxs":49},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"77afb15e02568da5c401477a87f1c321402905116469109eceaaf3fa21a3a253\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{},"lovelace":593960842},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["29608e905240c781827e10adafbd89d087afc2bbdccfeeec1ba10100c36188d9#0","687dc5113e0afeba26241f224148d7c7910a2faf802ef2f655ae01f83eb28ca3#1","8c2e816adf8a81b2790b6bdc3616aba8f25c65c373472a1f7112015e2addf17a#1","e24c8fec017a324787e24c0f16cd1699d8418cb0d2b7f4545fd08352aeb1287c#0"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 593960842 (fromList [])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.20Z","ns":["cardano.node.Mempool"],"pid":"39759"}
{"thread":"5095","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"179ae7305cf8d4a4973a09b8fe87eb0da69c12ec9c2a31bd4b1341a5f9dcc979\"}"},"kind":"TraceMempoolAddedTx"},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.29Z","ns":["cardano.node.Mempool"],"pid":"39759"}
{"thread":"330","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"179ae7305cf8d4a4973a09b8fe87eb0da69c12ec9c2a31bd4b1341a5f9dcc979\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26":{"586d617332303231443031333131":1},"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3":{"4c4f4253544552":1000000}},"lovelace":160795573},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["1d4cd2b3fb0b65fe1a01495278aa982eb048812f1409a2ddd1335e657cb97df8#1","55da61d71f7731763c5b7691c7f65e08689abd06b247e04ccded90522f40ebb1#0","bc3c24fe3fc2a67491fd561c67f2a442a22d4271b4b15fa592b272595e291723#3"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 160795573 (fromList [(PolicyID {policyID = ScriptHash \"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26\"},fromList [(\"Xmas2021D01311\",1)]),(PolicyID {policyID = ScriptHash \"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3\"},fromList [(\"LOBSTER\",1000000)])])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.31Z","ns":["cardano.node.Mempool"],"pid":"39759"}
{"thread":"6174","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"179ae7305cf8d4a4973a09b8fe87eb0da69c12ec9c2a31bd4b1341a5f9dcc979\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26":{"586d617332303231443031333131":1},"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3":{"4c4f4253544552":1000000}},"lovelace":160795573},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["1d4cd2b3fb0b65fe1a01495278aa982eb048812f1409a2ddd1335e657cb97df8#1","55da61d71f7731763c5b7691c7f65e08689abd06b247e04ccded90522f40ebb1#0","bc3c24fe3fc2a67491fd561c67f2a442a22d4271b4b15fa592b272595e291723#3"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 160795573 (fromList [(PolicyID {policyID = ScriptHash \"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26\"},fromList [(\"Xmas2021D01311\",1)]),(PolicyID {policyID = ScriptHash \"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3\"},fromList [(\"LOBSTER\",1000000)])])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.32Z","ns":["cardano.node.Mempool"],"pid":"39759"}
{"thread":"546","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"179ae7305cf8d4a4973a09b8fe87eb0da69c12ec9c2a31bd4b1341a5f9dcc979\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26":{"586d617332303231443031333131":1},"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3":{"4c4f4253544552":1000000}},"lovelace":160795573},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["1d4cd2b3fb0b65fe1a01495278aa982eb048812f1409a2ddd1335e657cb97df8#1","55da61d71f7731763c5b7691c7f65e08689abd06b247e04ccded90522f40ebb1#0","bc3c24fe3fc2a67491fd561c67f2a442a22d4271b4b15fa592b272595e291723#3"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 160795573 (fromList [(PolicyID {policyID = ScriptHash \"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26\"},fromList [(\"Xmas2021D01311\",1)]),(PolicyID {policyID = ScriptHash \"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3\"},fromList [(\"LOBSTER\",1000000)])])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.32Z","ns":["cardano.node.Mempool"],"pid":"39759"}
{"thread":"716","sev":"Info","data":{"mempoolSize":{"bytes":66805,"numTxs":50},"tx":{"txid":"txid: TxId {_unTxId = SafeHash \"179ae7305cf8d4a4973a09b8fe87eb0da69c12ec9c2a31bd4b1341a5f9dcc979\"}"},"kind":"TraceMempoolRejectedTx","err":{"produced":{"policies":{"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26":{"586d617332303231443031333131":1},"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3":{"4c4f4253544552":1000000}},"lovelace":160795573},"consumed":{"policies":{},"lovelace":0},"kind":"ValueNotConservedUTxO","badInputs":["1d4cd2b3fb0b65fe1a01495278aa982eb048812f1409a2ddd1335e657cb97df8#1","55da61d71f7731763c5b7691c7f65e08689abd06b247e04ccded90522f40ebb1#0","bc3c24fe3fc2a67491fd561c67f2a442a22d4271b4b15fa592b272595e291723#3"],"error":"This transaction consumed Value 0 (fromList []) but produced Value 160795573 (fromList [(PolicyID {policyID = ScriptHash \"18f59066d205b9ffc4208d9b6cfad2dbdee84dc4a0097e0b9de7db26\"},fromList [(\"Xmas2021D01311\",1)]),(PolicyID {policyID = ScriptHash \"8654e8b350e298c80d2451beb5ed80fc9eee9f38ce6b039fb8706bc3\"},fromList [(\"LOBSTER\",1000000)])])"}},"loc":null,"env":"1.32.1:4f65f","msg":"","app":[],"host":"relay1","at":"2022-01-05T01:09:55.34Z","ns":["cardano.node.Mempool"],"pid":"39759"}
$ curl -s localhost:12798/metrics | grep cardano_node_metrics_Forge
cardano_node_metrics_Forge_node_not_leader_int 18321
cardano_node_metrics_Forge_forge_about_to_lead_int 18321
check your logfile - look what happened before your slot: Slot: 49778704, SlotInEpoch: 357904 as well as after…
if you are using journal:
journalctl -uq cardano-node.service -o cat | grep 49778704 -B5 -A10
check if your BP is performing leadership checks
if you can’t find anything in logs with slot nr 49778704, then the BP node was either down or overutilized
just open the log file and check what is happening before the 49778704 slot…
Hi @thomaspb !
Based on metrics from your BP, seems BP didn’t know that it was a SlotLeader. (You didn’t restarted your BP after missed slot, right?)
You should see something like this:
curl -s localhost:12798/metrics | grep cardano_node_metrics_Forge
cardano_node_metrics_Forge_adopted_int 19
cardano_node_metrics_Forge_forged_int 19
cardano_node_metrics_Forge_forge_about_to_lead_int 491329
cardano_node_metrics_Forge_node_is_leader_int 19
cardano_node_metrics_Forge_node_not_leader_int 491310
Can you remove CNCLI DB and recalculate everything from scratch?
That’s what I would also suggest as next step.
oh, somehow missed the TraceNodeNotLeader post, yeah, @georgem1976 is right… looks like you have an issue with cncli
Sorry but I restarted it before this post
This is probably the last option.
The biggest problem is not that I lost the block, but that I do not know why
Do you have grafana? If you have, you can check those metrics there.
Thank you for the quick replies. 
if the producer was not aware about that block can be an error generated by cncli (this is happen when you are using wrong vrf files or perhaps cncli db corrupted) can u try to search again on cncli and check the status for that slot?
I made a VM clone to continue the research. I will post the findings (if any)