No more peers suddenly

So I was in the process of syncing my node and all of a sudden at 95% I can see now that no more OUT peers are there to sync with. I would also like to mention my node has been syncing for a VERY long time. its been over 24 hours since I started the sync process. Im running my node on Ubuntu 20.04 t2.medium instance on AWS. On my other node (relay node) a different issue occurred where during the sync the socket file seemed to disappear and now when i try to query the tip or run the monitor it returns an error saying the socket file was not found. I don’t know if these things are related but any advice on where to begin the triage here would be super helpful.

I’d also like to mention that I had recently setup 2FA on my nodes although it didn’t seem t affect the node sync when I checked after setting it up initially. I came back 2 hours later and these things had occurred. Not sure if they are related but i wouldn’t think so.

First of all keep the iohk nodes in topology file till the nodes will be 100% synced and 2nd the 2FA installation should not affect the OUT connection

Got it. that explains why my BPN was not receiving any more peers because my relay node failed and that was the only node it was configured to talk with. Now to my second issue, The relay node, which is configured to talk to both my BPN and the IOHK nodes, will not start back up for some reason. It seems to just hang on this point during startup.

May 06 13:12:00 ip-172-31-77-188 cardano-node[525048]: [ip-172-3:cardano.node.diffusion-mode:Notice:5] [2021-05-06 13:12:00.58 UTC] InitiatorAndResponderDiffusionMode
May 06 13:12:00 ip-172-31-77-188 cardano-node[525048]: [ip-172-3:cardano.node.dns-producers:Notice:5] [2021-05-06 13:12:00.58 UTC] [DnsSubscriptionTarget {dstDomain = "relays-new.cardano-m>
May 06 13:12:00 ip-172-31-77-188 cardano-node[525048]: [ip-172-3:cardano.node.ip-producers:Notice:5] [2021-05-06 13:12:00.58 UTC] IPSubscriptionTarget {ispIps = [3.239.0.203:6000], ispVale>
lines 1-20/20 (END)

I also can see that there is no socket file in the db location. Any idea on what this might be?

Type

sudo systemctl status cardano-node

journalctl -e -f -u cardano-node

This is the entire log output. Its been this way for the entire time.

May 06 14:45:09 ip-172-31-77-188 systemd[1]: Started Cardano node service.
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: Listening on http://127.0.0.1:12798
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.networkMagic:Notice:5] [2021-05-06 14:45:11.23 UTC] NetworkMagic 764824073
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.protocol:Notice:5] [2021-05-06 14:45:11.23 UTC] Byron; Shelley
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.version:Notice:5] [2021-05-06 14:45:11.23 UTC] 1.26.2
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.commit:Notice:5] [2021-05-06 14:45:11.23 UTC] 3531289c9f79eab7ac5d3272ce6e6821504fec4c
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.nodeStartTime:Notice:5] [2021-05-06 14:45:11.23 UTC] 2021-05-06 14:45:11.237578746 UTC
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.systemStartTime:Notice:5] [2021-05-06 14:45:11.23 UTC] 2017-09-23 21:44:51 UTC
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.slotLengthByron:Notice:5] [2021-05-06 14:45:11.23 UTC] 20s
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.epochLengthByron:Notice:5] [2021-05-06 14:45:11.23 UTC] 21600
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.slotLengthShelley:Notice:5] [2021-05-06 14:45:11.23 UTC] 1s
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.epochLengthShelley:Notice:5] [2021-05-06 14:45:11.23 UTC] 432000
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.slotsPerKESPeriodShelley:Notice:5] [2021-05-06 14:45:11.23 UTC] 129600
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.slotLengthAllegra:Notice:5] [2021-05-06 14:45:11.23 UTC] 1s
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.epochLengthAllegra:Notice:5] [2021-05-06 14:45:11.23 UTC] 432000
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.slotsPerKESPeriodAllegra:Notice:5] [2021-05-06 14:45:11.23 UTC] 129600
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.slotLengthMary:Notice:5] [2021-05-06 14:45:11.23 UTC] 1s
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.epochLengthMary:Notice:5] [2021-05-06 14:45:11.23 UTC] 432000
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.basicInfo.slotsPerKESPeriodMary:Notice:5] [2021-05-06 14:45:11.23 UTC] 129600
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.addresses:Notice:5] [2021-05-06 14:45:11.23 UTC] [SocketInfo 0.0.0.0:6000]
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.diffusion-mode:Notice:5] [2021-05-06 14:45:11.23 UTC] InitiatorAndResponderDiffusionMode
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.dns-producers:Notice:5] [2021-05-06 14:45:11.23 UTC] [DnsSubscriptionTarget {dstDomain = “relays-new.cardano-mainnet.iohk.io”, dstPort = 3001, dstValency = 2}]
May 06 14:45:11 ip-172-31-77-188 cardano-node[939]: [ip-172-3:cardano.node.ip-producers:Notice:5] [2021-05-06 14:45:11.23 UTC] IPSubscriptionTarget {ispIps = [3.239.0.203:6000], ispValency = 1}

And how it’s looking the glive output?

Guild LiveView version check…
Looks like cardano-node is running with socket-path as /home/ubuntu/cardano-relay-node/db/socket, but the actual socket file does not exist.
This could occur if the node hasnt completed startup or if a second instance of node startup was attempted!
If this does not resolve automatically in a few minutes, you might want to restart your node and try again.

ERROR: CNTools failed to load common env file
Please verify set values in ‘User Variables’ section in env file or log an issue on GitHub

Ok, show me the topology file

Try sudo systemctl status cardano-node

{
“Producers”: [
{
“addr”: “My BPN Nodes Ip”,
“port”: 6000,
“valency”: 1
},
{
“addr”: “relays-new.cardano-mainnet.iohk.io”,
“port”: 3001,
“valency”: 2
}
]
}

I left out my BPN IP address on purpose, theres an IP there in the actual file

sudo systemctl status cardano-node

● cardano-node.service - Cardano node service
Loaded: loaded (/etc/systemd/system/cardano-node.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-05-06 14:59:17 UTC; 1min 30s ago
Main PID: 1319 (startRelayNode.)
Tasks: 10 (limit: 4706)
Memory: 2.1G
CGroup: /system.slice/cardano-node.service
├─1319 /bin/bash /home/ubuntu/cardano-relay-node/startRelayNode.sh
└─1333 /usr/local/bin/cardano-node run --topology /home/ubuntu/cardano-relay-node/mainnet-topology.json --database-path /home/ubuntu/cardano-relay-node/db --socket-path /home/>

May 06 14:59:19 ip-172-31-77-188 cardano-node[1333]: [ip-172-3:cardano.node.basicInfo.slotLengthAllegra:Notice:5] [2021-05-06 14:59:18.81 UTC] 1s
May 06 14:59:19 ip-172-31-77-188 cardano-node[1333]: [ip-172-3:cardano.node.basicInfo.epochLengthAllegra:Notice:5] [2021-05-06 14:59:18.81 UTC] 432000
May 06 14:59:19 ip-172-31-77-188 cardano-node[1333]: [ip-172-3:cardano.node.basicInfo.slotsPerKESPeriodAllegra:Notice:5] [2021-05-06 14:59:18.81 UTC] 129600
May 06 14:59:19 ip-172-31-77-188 cardano-node[1333]: [ip-172-3:cardano.node.basicInfo.slotLengthMary:Notice:5] [2021-05-06 14:59:18.81 UTC] 1s
May 06 14:59:19 ip-172-31-77-188 cardano-node[1333]: [ip-172-3:cardano.node.basicInfo.epochLengthMary:Notice:5] [2021-05-06 14:59:18.81 UTC] 432000
May 06 14:59:19 ip-172-31-77-188 cardano-node[1333]: [ip-172-3:cardano.node.basicInfo.slotsPerKESPeriodMary:Notice:5] [2021-05-06 14:59:18.81 UTC] 129600
May 06 14:59:19 ip-172-31-77-188 cardano-node[1333]: [ip-172-3:cardano.node.addresses:Notice:5] [2021-05-06 14:59:18.81 UTC] [SocketInfo 0.0.0.0:6000]
May 06 14:59:19 ip-172-31-77-188 cardano-node[1333]: [ip-172-3:cardano.node.diffusion-mode:Notice:5] [2021-05-06 14:59:18.81 UTC] InitiatorAndResponderDiffusionMode
May 06 14:59:19 ip-172-31-77-188 cardano-node[1333]: [ip-172-3:cardano.node.dns-producers:Notice:5] [2021-05-06 14:59:18.81 UTC] [DnsSubscriptionTarget {dstDomain = "relays-new.cardano-mai>
May 06 14:59:19 ip-172-31-77-188 cardano-node[1333]: [ip-172-3:cardano.node.ip-producers:Notice:5] [2021-05-06 14:59:18.81 UTC] IPSubscriptionTarget {ispIps = [3.239.0.203:6000], ispValenc>
lines 1-20/20 (END)

I see the relay is restarting

1min 30s ago

try to monitor the node

type more times (3-5 min interval)
sudo systemctl status cardano-node

What is the hw configuration of the node?

1 Like

Having the same exact issue.

Relay completely syncs then pauses and restarts by itself every five minutes (estimated).

The status says almost exactly what the above says and gets stuck not syncing anymore.

I deleted the DB numerous of times.

I’m running v1.27.0.

Script host --host-addr 0.0.0.0

Config.json set from 127.0.0.1 to 0.0.0.0

Stuck…

Hey Active, the solution for me was to increase the memory size of my instance. I was running a t2.medium on AWS and I upped it to a t3.large. I was running 1.26.2 though so it could be different. Let me know if that works!

I have two R610s with 48gb RAM and two Xeon six core processors each. I only have two 146GB enterprise SAS drives on both. I reverted the config to 127.0.0.1.

The ram sticks are 4gb each… would that be an issue? I’ll look into the BIOS settings to see if I can enable optimization - I still have no luck.

The 4gb ram sticks might be the issue. You have plenty of processing power otherwise. What are the ram sticks for?

They’re servers with 48gb ram.

Basically it’s 4gb ram × 12 so wouldn’t it make it 48gb ram? All same brand and frequency.

But why are u running 1.27.0 version? It’s not official released. Try 1.26.2 and check if u still hVe this issue

Yes still having the same issue. Crashing after syncing or close to it. Ram reaches 5.3gb and crashes. Processor up to 230%…

I switched version same issue… Seems like Cardano likes to use a single card and not take advantage of the others. Unless I’m wrong. It reached a peak of 5.6gb and crashed.