**External**
**Area**
*Other*
Forging and Traces impact
**Summary**
**B…lock producer** nodes with `TraceMemPool` set to true miss more slots than ones without despite very few traces (compared to a relay with the same setting) and a low CPU consumption.
Default config should set it to false or the trace implementation should be improved not to impact slots performance.
**Steps to reproduce**
Steps to reproduce the behavior:
1. Run a BP with [current official default config](https://hydra.iohk.io/build/5822084/download/1/mainnet-config.json) that has `TraceMemPool` set to true
2. Monitor `cardano_node_metrics_slotsMissedNum_int` during several days
3. Run a BP with same config except `TraceMemPool` set to false
4. Monitor `cardano_node_metrics_slotsMissedNum_int` during several days
**Expected behavior**
With a system powerful enough, both should have almost the same number of missed slots or not at all.
In practice, the node with `TraceMemPool` set to true has much more missed slots.
**System info (please complete the following information):**
- OS: GNU/Linux Debian
- Version 10 Buster
- 1.25.1
**Screenshots and attachments**
This graph compare 2 BP nodes with the same setup running during the same period. The steps at epochs transitions are due to another cardano-node issue, but the missed slots increase inside epochs with `TraceMemPool` set to true is unexpected.
CPU consumption during the same range is really reasonable.
Here are the logs of a missed leader slot 25309533 (slots 25309532 and 25309534 are checked) likely because of the `TraceMemPool` `StakeKeyAlreadyRegisteredDELEG` trace just before the slot:
```
mars 27 21:10:23 bp1.smaug.pool.pm cardano-node[738]: {"at":"2021-03-27T20:10:23.00Z","env":"1.25.1:9a733","ns":["cardano.node.LeadershipCheck"],"data":{"credentials":"Cardano","kind":"TraceStartLeadershipCheck","delegMapSize":389166,"slot":25309532,"chainDensity":4.946956e-2,"utxoSize":1693420},"app":[],"msg":"","pid":"738","loc":null,"host":"bp1","sev":"Info","thread":"79"}
mars 27 21:10:23 bp1.smaug.pool.pm cardano-node[738]: {"at":"2021-03-27T20:10:23.00Z","env":"1.25.1:9a733","ns":["cardano.node.Forge"],"data":{"credentials":"Cardano","val":{"kind":"TraceNodeNotLeader","slot":25309532}},"app":[],"msg":"","pid":"738","loc":null,"host":"bp1","sev":"Info","thread":"79"}
mars 27 21:10:23 bp1.smaug.pool.pm cardano-node[738]: {"at":"2021-03-27T20:10:23.86Z","env":"1.25.1:9a733","ns":["cardano.node"],"data":{"time(ps)":3082099509739250000,"kind":"MeasureTxsTimeStart","mempoolNumBytes":10504,"mempoolNumTxs":19},"app":[],"msg":"","pid":"738","loc":null,"host":"bp1","sev":"Info","thread":"615015"}
mars 27 21:10:23 bp1.smaug.pool.pm cardano-node[738]: {"at":"2021-03-27T20:10:23.86Z","env":"1.25.1:9a733","ns":["cardano.node.Mempool"],"data":{"tx":{"txid":"txid: TxId {_unTxId = \"d43d0be62b323e5f88975114295a03a5971b8427be0011acec8695876c2c728c\"}"},"kind":"TraceMempoolAddedTx","mempoolSize":{"numTxs":19,"bytes":10504}},"app":[],"msg":"","pid":"738","loc":null,"host":"bp1","sev":"Info","thread":"615015"}
mars 27 21:10:25 bp1.smaug.pool.pm cardano-node[738]: {"at":"2021-03-27T20:10:23.87Z","env":"1.25.1:9a733","ns":["cardano.node.Mempool"],"data":{"tx":{"txid":"txid: TxId {_unTxId = \"d43d0be62b323e5f88975114295a03a5971b8427be0011acec8695876c2c728c\"}"},"kind":"TraceMempoolRejectedTx","mempoolSize":{"numTxs":19,"bytes":10504},"err":{"kind":"StakeKeyAlreadyRegisteredDELEG","error":"Staking credential already registered","consumed":{"lovelace":0,"policies":{}},"credential":"KeyHashObj (KeyHash \"448ebf9d10c636d30c7309f9876f0c6f32e57d5d19b6b725217c7177\")","badInputs":[["b4129da1ebc9469113628712551188a315fd415ee7181e359ae036ce277127e4",0]],"produced":{"lovelace":962573798,"policies":{}}}},"app":[],"msg":"","pid":"738","loc":null,"host":"bp1","sev":"Info","thread":"687010"}
mars 27 21:10:25 bp1.smaug.pool.pm cardano-node[738]: {"at":"2021-03-27T20:10:25.01Z","env":"1.25.1:9a733","ns":["cardano.node.LeadershipCheck"],"data":{"credentials":"Cardano","kind":"TraceStartLeadershipCheck","delegMapSize":389166,"slot":25309534,"chainDensity":4.946956e-2,"utxoSize":1693420},"app":[],"msg":"","pid":"738","loc":null,"host":"bp1","sev":"Info","thread":"79"}
```
The other BP without the `StakeKeyAlreadyRegisteredDELEG` trace checked it as expected:
```
mars 27 20:10:23 bp2 cardano-node[28595]: {"at":"2021-03-27T20:10:23.00Z","env":"1.25.1:9a733","ns":["cardano.node.LeadershipCheck"],"data":{"credentials":"Cardano","kind":"TraceStartLeadershipCheck","delegMapSize":389166,"slot":25309532,"chainDensity":4.946956e-2,"utxoSize":1693420},"app":[],"msg":"","pid":"28595","loc":null,"host":"bp2","sev":"Info","thread":"67"}
mars 27 20:10:23 bp2 cardano-node[28595]: {"at":"2021-03-27T20:10:23.00Z","env":"1.25.1:9a733","ns":["cardano.node.ForgeTime"],"data":{},"app":[],"msg":"","pid":"28595","loc":null,"host":"bp2","sev":"Info","thread":"67"}
mars 27 20:10:23 bp2 cardano-node[28595]: {"at":"2021-03-27T20:10:23.00Z","env":"1.25.1:9a733","ns":["cardano.node.Forge"],"data":{"credentials":"Cardano","val":{"kind":"TraceNodeNotLeader","slot":25309532}},"app":[],"msg":"","pid":"28595","loc":null,"host":"bp2","sev":"Info","thread":"67"}
mars 27 20:10:24 bp2 cardano-node[28595]: {"at":"2021-03-27T20:10:24.00Z","env":"1.25.1:9a733","ns":["cardano.node.LeadershipCheck"],"data":{"credentials":"Cardano","kind":"TraceStartLeadershipCheck","delegMapSize":389166,"slot":25309533,"chainDensity":4.946956e-2,"utxoSize":1693420},"app":[],"msg":"","pid":"28595","loc":null,"host":"bp2","sev":"Info","thread":"67"}
mars 27 20:10:24 bp2 cardano-node[28595]: {"at":"2021-03-27T20:10:24.00Z","env":"1.25.1:9a733","ns":["cardano.node.Forge"],"data":{"credentials":"Cardano","val":{"kind":"TraceNodeIsLeader","slot":25309533}},"app":[],"msg":"","pid":"28595","loc":null,"host":"bp2","sev":"Info","thread":"67"}
```
**Additional context**
Note that I have observed since that some other traces also have impact on missed slots, but to a lesser degree.
I think we need to be prepare for 1.27.1 soon 
