BP restarting every few mins

Suddenly core node starts to crash and restarting every few minutes,
server have 16GB RAM, 160GB SSD and 8 vCPUs.
Ubuntu apt is updated and latest node ver. and latest Cntools.

Check with journalctl -e -f -u cnode

Any errors?

Nov 03 10:44:59 SADA-P cnode[805]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Nov 03 10:45:01 SADA-P cnode[1405]: Listening on http://127.0.0.1:12798
Nov 05 09:46:36 SADA-P cnode[805]: /opt/cardano/cnode/scripts/cnode.sh: line 60: 1405 Killed cardano-node “${CPU_RUNTIME[@]}” run --topology “${TOPOLOGY}” --config “${CONFIG}” --database-path “${DB_DIR}” --socket-path “${CARDANO_NODE_SOCKET_PATH}” --shelley-kes-key “${POOL_DIR}/${POOL_HOTKEY_SK_FILENAME}” --shelley-vrf-key “${POOL_DIR}/${POOL_VRF_SK_FILENAME}” --shelley-operational-certificate “${POOL_DIR}/${POOL_OPCERT_FILENAME}” --port ${CNODE_PORT} “${host_addr[@]}”
Nov 05 09:46:42 SADA-P cnode[1889786]: ERROR: You specified 12788 as your EKG port, but it looks like the cardano-node (PID: 1890042 ) is not listening on this port. Please update the config or kill the conflicting process first.
Nov 05 09:46:42 SADA-P cnode[1889786]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Nov 05 09:46:42 SADA-P cnode[1890086]: mkdir: cannot create directory ‘/archive’: Permission denied
Nov 05 09:46:44 SADA-P cnode[1890097]: Listening on http://127.0.0.1:12798
Nov 05 09:46:44 SADA-P cnode[1890097]: Shutting down…
Nov 05 09:46:44 SADA-P cnode[1890097]: cardano-node: FatalError {fatalErrorMessage = “Cardano.Node.Run.handleSimpleNode.readTopologyFile: Cardano.Node.Configuration.Topology.readTopologyFile: : openBinaryFile: does not exist (No such file or directory)”}
Nov 05 09:46:55 SADA-P cnode[1890367]: ERROR: You specified 12788 as your EKG port, but it looks like the cardano-node (PID: 1890642 ) is not listening on this port. Please update the config or kill the conflicting process first.
Nov 05 09:46:55 SADA-P cnode[1890671]: mkdir: cannot create directory ‘/archive’: Permission denied
Nov 05 09:46:56 SADA-P cnode[1890677]: Listening on http://127.0.0.1:12798
– Reboot –
Nov 06 08:15:40 SADA-P cnode[1440]: Listening on http://127.0.0.1:12798
Nov 06 08:35:19 SADA-P cnode[801]: /opt/cardano/cnode/scripts/cnode.sh: line 60: 1440 Killed cardano-node “${CPU_RUNTIME[@]}” run --topology “${TOPOLOGY}” --config “${CONFIG}” --database-path “${DB_DIR}” --socket-path “${CARDANO_NODE_SOCKET_PATH}” --shelley-kes-key “${POOL_DIR}/${POOL_HOTKEY_SK_FILENAME}” --shelley-vrf-key “${POOL_DIR}/${POOL_VRF_SK_FILENAME}” --shelley-operational-certificate “${POOL_DIR}/${POOL_OPCERT_FILENAME}” --port ${CNODE_PORT} “${host_addr[@]}”
Nov 06 08:35:25 SADA-P cnode[52682]: ERROR: You specified 12788 as your EKG port, but it looks like the cardano-node (PID: 52950 ) is not listening on this port. Please update the config or kill the conflicting process first.
Nov 06 08:35:25 SADA-P cnode[52682]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Nov 06 08:35:25 SADA-P cnode[52988]: mkdir: cannot create directory ‘/archive’: Permission denied
Nov 06 08:35:26 SADA-P cnode[52997]: Listening on http://127.0.0.1:12798
Nov 06 08:35:27 SADA-P cnode[52997]: Shutting down…
Nov 06 08:35:27 SADA-P cnode[52997]: cardano-node: FatalError {fatalErrorMessage = “Cardano.Node.Run.handleSimpleNode.readTopologyFile: Cardano.Node.Configuration.Topology.readTopologyFile: : openBinaryFile: does not exist (No such file or directory)”}
Nov 06 08:35:39 SADA-P cnode[53677]: Listening on http://127.0.0.1:12798
Nov 06 08:55:09 SADA-P cnode[53219]: /opt/cardano/cnode/scripts/cnode.sh: line 60: 53677 Killed cardano-node “${CPU_RUNTIME[@]}” run --topology “${TOPOLOGY}” --config “${CONFIG}” --database-path “${DB_DIR}” --socket-path “${CARDANO_NODE_SOCKET_PATH}” --shelley-kes-key “${POOL_DIR}/${POOL_HOTKEY_SK_FILENAME}” --shelley-vrf-key “${POOL_DIR}/${POOL_VRF_SK_FILENAME}” --shelley-operational-certificate “${POOL_DIR}/${POOL_OPCERT_FILENAME}” --port ${CNODE_PORT} “${host_addr[@]}”
Nov 06 08:55:15 SADA-P cnode[117287]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Nov 06 08:55:16 SADA-P cnode[117757]: Listening on http://127.0.0.1:12798
Nov 06 09:15:00 SADA-P cnode[117287]: /opt/cardano/cnode/scripts/cnode.sh: line 60: 117757 Killed cardano-node “${CPU_RUNTIME[@]}” run --topology “${TOPOLOGY}” --config “${CONFIG}” --database-path “${DB_DIR}” --socket-path “${CARDANO_NODE_SOCKET_PATH}” --shelley-kes-key “${POOL_DIR}/${POOL_HOTKEY_SK_FILENAME}” --shelley-vrf-key “${POOL_DIR}/${POOL_VRF_SK_FILENAME}” --shelley-operational-certificate “${POOL_DIR}/${POOL_OPCERT_FILENAME}” --port ${CNODE_PORT} “${host_addr[@]}”
Nov 06 09:15:06 SADA-P cnode[192416]: WARN: A prior running Cardano node was not cleanly shutdown, socket file still exists. Cleaning up.
Nov 06 09:15:07 SADA-P cnode[192854]: Listening on http://127.0.0.1:12798

Did u tried to restart the machine?
Can u monitor with top or htop how is the memory? I saw the killing message on logs

cnode is using 62.4% memory
but i see cncli spike CPU up to 100%

Try to stop the cncli and keep it under monitoring

sudo systemctl | grep running
init.scope loaded active running System and Service Manager
session-13.scope loaded active running Session 13 of user user
session-24.scope loaded active running Session 24 of user user
accounts-daemon.service loaded active running Accounts Service
atd.service loaded active running Deferred execution scheduler
chrony.service loaded active running chrony, an NTP client/server
cnode-cncli-leaderlog.service loaded active running Cardano Node - CNCLI Leaderlog
cnode-cncli-sync.service loaded active running Cardano Node - CNCLI Sync
cnode-cncli-validate.service loaded active running Cardano Node - CNCLI Validate
cnode-logmonitor.service loaded active running Cardano Node - Log Monitor
cnode.service loaded active running Cardano Node
cron.service loaded active running Regular background program processing daemon
dbus.service loaded active running D-Bus System Message Bus
do-agent.service loaded active running The DigitalOcean Monitoring Agent
fail2ban.service loaded active running Fail2Ban Service
getty@tty1.service loaded active running Getty on tty1
irqbalance.service loaded active running irqbalance daemon
multipathd.service loaded active running Device-Mapper Multipath Device Controller
networkd-dispatcher.service loaded active running Dispatcher daemon for systemd-networkd
polkit.service loaded active running Authorization Manager
rsyslog.service loaded active running System Logging Service
serial-getty@ttyS0.service loaded active running Serial Getty on ttyS0
snapd.service loaded active running Snap Daemon
ssh.service loaded active running OpenBSD Secure Shell server
systemd-journald.service loaded active running Journal Service
systemd-logind.service loaded active running Login Service
systemd-networkd.service loaded active running Network Service
systemd-resolved.service loaded active running Network Name Resolution
systemd-udevd.service loaded active running udev Kernel Device Manager
udisks2.service loaded active running Disk Manager
unattended-upgrades.service loaded active running Unattended Upgrades Shutdown
user@1000.service loaded active running User Manager for UID 1000
dbus.socket loaded active running D-Bus System Message Bus Socket
multipathd.socket loaded active running multipathd control socket
snapd.socket loaded active running Socket activation for snappy daemon
syslog.socket loaded active running Syslog Socket
systemd-journald-audit.socket loaded active running Journal Audit Socket
systemd-journald-dev-log.socket loaded active running Journal Socket (/dev/log)
systemd-journald.socket loaded active running Journal Socket
systemd-networkd.socket loaded active running Network Service Netlink Socket
systemd-udevd-control.socket loaded active running udev Control Socket
systemd-udevd-kernel.socket loaded active running udev Kernel Socket
ua-timer.timer loaded active waiting Ubuntu Advantage Timer for running repeated jobs

which cncli service to stop?

All, for the moment, then start one by one while u are monitoring

No clue, restarted server several times, relays also.
it works at the moment, will monitor it.
I hope for more stable node with the new version soon.

You check the health of your SSD? May be IO failures.