Daedalus: Relay Node Configuration Help?

Actually the memory usage grows over time as the node runs in the background until it runs out (thus using swap space / virtual memory on disk). It doesn’t matter it you have 8, 16, or 32 GB of ram as it will burn through it all very quickly … this is indicative of a rampant memory leak somewhere. Restarting the machine or node periodically is the only work around I have found thus far.

Looks like Raph made a similar python script based on timing the response by actually connecting a socket to each node rather than just using a port scan to see if they are available. He put a 3 second timeout on it then sorts the results after by the diffs, if successful. That works too although a bit slower to run. I like his inclusion of max_num_rows as well since I found the more valency and connections the faster the memory gets eaten up!

I actually modified the previous workaround script a tad this weekend to include state filtering and a shorter 250ms timeout. As Raph mentioned ICMP is almost always blocked by firewall so you got to do something a tad more creative than a ping to see if there node is good. I think I will add a max connection ratio based on system memory for the final touch.

It will be much better when they fix connectivity and discovery. Regular users are not going to write code, dig through log files, and conduct their own workaround research. They are going to say “this is broken” and go somewhere else that doesn’t have fundamental usability issues.

#!/bin/python3
import json
import requests


# TODO: SET YOUR REGION HERE!
# e.g. 'North America', 'Europe', 'Asia', 'Africa', 'South America', 'Oceania'
region = 'North America'
# TODO: SET YOUR STATES HERE! (optional)
# e.g. ['Arizona', 'California', 'Nevada', 'Utah', 'Colorado', 'New Mexico']
# e.g. ['FR', 'GB', 'BE', 'DE', 'CH', 'IT', 'ES']
states = ['Arizona']
# TODO: SET MAXIMUM NODES TO ADD HERE!
max_nodes = 4
# collect relay nodes
cardano_nodes = []
# port check courtesy of https://www.yougetsignal.com/tools/open-ports/
uri = 'https://ports.yougetsignal.com/check-port.php'


# helper to add responsive nodes
def transform_node(node):
  keys = ['addr', 'port']
  res = {x:node[x] for x in keys}
  res['valency'] = 1
  cardano_nodes.append(res)


def main():
  # fetch latest relay nodes from Cardano explorer
  response = requests.get('https://explorer.cardano-mainnet.iohk.io/relays/topology.json')
  if response.ok:
    body = json.loads(response.text)
    # filter local region nodes
    nodes = (n for n in body['Producers'] if n['continent'] == region)
    # optional filter by states as well
    if states:
      nodes = (n for n in nodes if n['state'] in states)
    for node in nodes:
      port = node['port']
      addr = node['addr']
      success = 'is open on ' + addr
      raw = {'remoteAddress': addr, 'portNumber': port}
      try:
        # check if node is available
        yougetsignal = requests.post(uri, data = raw, timeout=0.25)
        if yougetsignal.ok:
          print(yougetsignal.text)
          if success in yougetsignal.text:
            print('Adding ', node)
            transform_node(node)
            if len(cardano_nodes) == max_nodes:
              break;
          else:
            print('Skipping ', node)
        else:
          print('Request failed for YouGetSignal!')
      except Exception as ex:
        print('Request timeout for ', node)
    # always add IOHK default relays
    cardano_nodes.append({'addr':'relays-new.cardano-mainnet.iohk.io','port':3001,'valency':1})
    topo = {'Producers': cardano_nodes}
    with open('topology.yaml', 'w') as out:
      out.write(json.dumps(topo))
  else:
    print('Request failed for Cardano explorer!')


if __name__ == '__main__':
  main() 

Edit 2: For giggles I ran the new script to make sure I didn’t have any typos before posting and then dumped the topo file into Daedalus. Just had a record breaking startup and sync in under two minutes and that has never happened before! Completely unfair test as I had previously messed around earlier today but it’s the little things …

2 Likes

could you visualize the memory usage over time?

Yes, there are many tools for graphing profiling data. Unfortunately all the ones I know are for C, C++, C#, Java, etc. I have been using top to monitor. I am just beginning to learn Haskell so I am not very familiar with tools for it yet. I think I can google around and maybe find something more generalized at the process level that might do the trick though.

Edit: GHC and GCC seems to be close friends such that some of the flags I know may work on Cardano node as well. I will try building one from source with extra debug options and see if that yields more information.

I mean can you monitor your system resources? prometheus+grafana

I found a 3rd party library with Haskell for prometheus (GitHub - fimad/prometheus-haskell: Haskell client library for exposing prometheus.io metrics.) It appears to be hosted and available on hackage … I have never used prometheus+grafana outside of Java-land so can’t vouch for how well this would work.

GHC seems to come with some decent profiling capability out of the box though. I was going to follow the instructions here (8. Profiling — Glasgow Haskell Compiler 8.3.20171025 User's Guide) and build a cardano node locally to test and profile in more detail.

1 Like

Oddly enough, it happens that a node, yet declared outside your continent, is faster to connect to than others in your own country :thinking: In this particular case the node with IP address 13.248.171.210 is using AWS Global Accelerator technology.

Sample of results:
                  addr  port      continent         state         RTT
1768     95.179.214.87  6001         Europe            FR    0.006507
2466    13.248.171.210  3001  North America            CA    0.007263
1082     51.15.191.124  3001         Europe            FR    0.007273
2019      51.159.2.140  5002         Europe            FR    0.008190
572       51.159.3.130  3001         Europe            FR    0.008332
...                ...   ...            ...           ...         ...
1302     34.209.220.93  6000  North America        Oregon  255.000000
2040      207.148.1.91  6001  North America         Texas  255.000000
2594    131.153.78.114  6002           Asia            SG  255.000000
741   one.goodpanj.org  6001  North America  Pennsylvania  255.000000
924      54.84.119.195  3001  North America      Virginia  255.000000
1 Like

Interesting, it’s true with edge deployed cloud networks the physical location of node matters significantly less.

This could actually be a great case study to increase Cardano network reliability and speed in general if thousands or millions of relay nodes were deployed for passive participation in the network. It wouldn’t take any of the onus or need away from dedicated servers and block producing nodes but it would greatly benefit users as far as overall performance. As it stands there isn’t really an incentive to leave Daedalus running or deploying passive nodes in cloud hosting other than for giggles.

Other than becoming a SPO, the problem of creating incentive for network participation may solve itself once there are enough popular DApps in the future …