I have the same problem of one of my relay, this allowed me to implement leak detection and automatic restart of this node when the memory used reaches a certain threshold. You should never wait for swapping to begin before restarting you node, swapping may cause I/O to stale and your node may timeout on queries and appear dead to other peers.
A note about the “bad tx”, from what I’ve noticed these TX are not bad, they are just received several times and of course, only the first one is valid. This might be a logging concurrency problem, I think I saw a GitHub issue about this some time ago. So the more peers a node is connected to, the more “bad TX” it will receive, it might even be seen as a sign of good peer connectivity…