Daedalus: Relay Node Configuration Help?

@Alexd1985 You seem extremely active on Cardano forums. Any chance you are also a python aficionado? I rarely use this language so it is not my strongest coding tool it just seemed the best for this particular task of automating the workaround. While I know how to use nmap on the terminal it seems this script is getting KeyError where the dictionary value for host name is not in the scan result as expected. I could switch it up to a more generalized ping check but I suspect most nodes block ICMP as a general firewall rule, at least I would if I was a stake pool operator setting up firewalls etc.

Also kudos! I am also a nerd from 1985 :wink:

Nope, sorry :frowning:

I think you can call system commands from python…

import os
cmd = 'nmap'
os.system(cmd)

The scan method here invokes the command like nmap -p PORT -sV ADDR or equivalent. That part is working fine although a tad slow. It’s checking the result on the subsequent line to see if the host relay node port has open and available state for TCP protocol that gets a dictionary key error. Basically it can’t find the ADDR used in the scan from the result info …

I think I will trace what requests yougetsignal sends and adjust it to be similar. I might be able to just piggy back off their web server with a simple POST request instead of using nmap as a port scan is overkill for this use case.

It’s 90% done but that last 10% is usually 90% of the work :wink:
I will come back to debug and simplify this later tonight!

if I will have time I will do a shell script or a batch file for windows…

Request payload for you-get-signal

remoteAddress=relays-new.cardano-mainnet.iohk.io&portNumber=3001

Response Body for you-get-signal OK

<p><img src="/img/flag_green.gif" alt="Open" style="height: 1em; width: 1em;" /> Port <a href="https://en.wikipedia.org/wiki/Port_3001" target="_blank" />3001</a> is open on relays-new.cardano-mainnet.iohk.io.</p>

Response Body for you-get-signal !OK

<p><img src="/img/flag_red.gif" alt="Closed" style="height: 1em; width: 1em;" /> Port <a href="https://en.wikipedia.org/wiki/Port_3001" target="_blank" />3001</a> is closed on relays.cardano-mainnet.iohk.io.</p>

Complete curl-style request for you-get-signal

curl -X POST 'https://ports.yougetsignal.com/check-port.php' --compressed --data-raw 'remoteAddress=relays-new.cardano-mainnet.iohk.io&portNumber=3001'

And the question is, have the wallet started to sync?

Yup, it was much better. Roughly 100x faster than using IOHK alone in my very unscientific test.

However during testing I exceeded the daily limit of requests on yougetsignal. Of course proxies, VPN, and other methods can be used to circumvent this but I am not interested in doing anything to bypass the reasonable expectation of usage for educational purposes.

I am emailing the owner of the site to strike up a dialog as I can understand why the limit exists to deter people from using the tools for nefarious purposes rather than development and education. He might be interested in officially hosting something for the interim that is limited to known Cardano relay nodes perhaps?

Of course if they don’t like this activity, as harmless as the intent may be, then there will need to be another workaround.

1 Like

FYI: No reply from Kurt but he has probably moved on to other more interesting projects since college.

Also note 100x faster than original is still pathetically slow bordering on unusable in the 21st century. I will start tracing memory and CPU activity as there is no legitimate reason for it to be so high given the network usage is basically less than 5% of available bandwidth.

Hi!

in ideal conditions is till need ~6 hours to get synced… I dont know the reason why can not use more bandwidth…

Me either,

I am actually becoming more concerned with the other resource usage. Only two types of applications peg out all CPUs and hit swap and virtual memory within a few minutes of launching. The question is becoming is the Cardano implementation malicious or atrocious?

The latter means the project is full of bugs and not ready for use yet. I don’t want to talk about what the former means :exploding_head:

high CPU and memory usage happens only at startup - to open up the databases…

related:

Actually the memory usage grows over time as the node runs in the background until it runs out (thus using swap space / virtual memory on disk). It doesn’t matter it you have 8, 16, or 32 GB of ram as it will burn through it all very quickly … this is indicative of a rampant memory leak somewhere. Restarting the machine or node periodically is the only work around I have found thus far.

Looks like Raph made a similar python script based on timing the response by actually connecting a socket to each node rather than just using a port scan to see if they are available. He put a 3 second timeout on it then sorts the results after by the diffs, if successful. That works too although a bit slower to run. I like his inclusion of max_num_rows as well since I found the more valency and connections the faster the memory gets eaten up!

I actually modified the previous workaround script a tad this weekend to include state filtering and a shorter 250ms timeout. As Raph mentioned ICMP is almost always blocked by firewall so you got to do something a tad more creative than a ping to see if there node is good. I think I will add a max connection ratio based on system memory for the final touch.

It will be much better when they fix connectivity and discovery. Regular users are not going to write code, dig through log files, and conduct their own workaround research. They are going to say “this is broken” and go somewhere else that doesn’t have fundamental usability issues.

#!/bin/python3
import json
import requests


# TODO: SET YOUR REGION HERE!
# e.g. 'North America', 'Europe', 'Asia', 'Africa', 'South America', 'Oceania'
region = 'North America'
# TODO: SET YOUR STATES HERE! (optional)
# e.g. ['Arizona', 'California', 'Nevada', 'Utah', 'Colorado', 'New Mexico']
# e.g. ['FR', 'GB', 'BE', 'DE', 'CH', 'IT', 'ES']
states = ['Arizona']
# TODO: SET MAXIMUM NODES TO ADD HERE!
max_nodes = 4
# collect relay nodes
cardano_nodes = []
# port check courtesy of https://www.yougetsignal.com/tools/open-ports/
uri = 'https://ports.yougetsignal.com/check-port.php'


# helper to add responsive nodes
def transform_node(node):
  keys = ['addr', 'port']
  res = {x:node[x] for x in keys}
  res['valency'] = 1
  cardano_nodes.append(res)


def main():
  # fetch latest relay nodes from Cardano explorer
  response = requests.get('https://explorer.cardano-mainnet.iohk.io/relays/topology.json')
  if response.ok:
    body = json.loads(response.text)
    # filter local region nodes
    nodes = (n for n in body['Producers'] if n['continent'] == region)
    # optional filter by states as well
    if states:
      nodes = (n for n in nodes if n['state'] in states)
    for node in nodes:
      port = node['port']
      addr = node['addr']
      success = 'is open on ' + addr
      raw = {'remoteAddress': addr, 'portNumber': port}
      try:
        # check if node is available
        yougetsignal = requests.post(uri, data = raw, timeout=0.25)
        if yougetsignal.ok:
          print(yougetsignal.text)
          if success in yougetsignal.text:
            print('Adding ', node)
            transform_node(node)
            if len(cardano_nodes) == max_nodes:
              break;
          else:
            print('Skipping ', node)
        else:
          print('Request failed for YouGetSignal!')
      except Exception as ex:
        print('Request timeout for ', node)
    # always add IOHK default relays
    cardano_nodes.append({'addr':'relays-new.cardano-mainnet.iohk.io','port':3001,'valency':1})
    topo = {'Producers': cardano_nodes}
    with open('topology.yaml', 'w') as out:
      out.write(json.dumps(topo))
  else:
    print('Request failed for Cardano explorer!')


if __name__ == '__main__':
  main() 

Edit 2: For giggles I ran the new script to make sure I didn’t have any typos before posting and then dumped the topo file into Daedalus. Just had a record breaking startup and sync in under two minutes and that has never happened before! Completely unfair test as I had previously messed around earlier today but it’s the little things …

1 Like

could you visualize the memory usage over time?

Yes, there are many tools for graphing profiling data. Unfortunately all the ones I know are for C, C++, C#, Java, etc. I have been using top to monitor. I am just beginning to learn Haskell so I am not very familiar with tools for it yet. I think I can google around and maybe find something more generalized at the process level that might do the trick though.

Edit: GHC and GCC seems to be close friends such that some of the flags I know may work on Cardano node as well. I will try building one from source with extra debug options and see if that yields more information.

I mean can you monitor your system resources? prometheus+grafana

I found a 3rd party library with Haskell for prometheus (GitHub - fimad/prometheus-haskell: Haskell client library for exposing prometheus.io metrics.) It appears to be hosted and available on hackage … I have never used prometheus+grafana outside of Java-land so can’t vouch for how well this would work.

GHC seems to come with some decent profiling capability out of the box though. I was going to follow the instructions here (8. Profiling — Glasgow Haskell Compiler 8.3.20171025 User's Guide) and build a cardano node locally to test and profile in more detail.

1 Like

Oddly enough, it happens that a node, yet declared outside your continent, is faster to connect to than others in your own country :thinking: In this particular case the node with IP address 13.248.171.210 is using AWS Global Accelerator technology.

Sample of results:
                  addr  port      continent         state         RTT
1768     95.179.214.87  6001         Europe            FR    0.006507
2466    13.248.171.210  3001  North America            CA    0.007263
1082     51.15.191.124  3001         Europe            FR    0.007273
2019      51.159.2.140  5002         Europe            FR    0.008190
572       51.159.3.130  3001         Europe            FR    0.008332
...                ...   ...            ...           ...         ...
1302     34.209.220.93  6000  North America        Oregon  255.000000
2040      207.148.1.91  6001  North America         Texas  255.000000
2594    131.153.78.114  6002           Asia            SG  255.000000
741   one.goodpanj.org  6001  North America  Pennsylvania  255.000000
924      54.84.119.195  3001  North America      Virginia  255.000000
1 Like

Interesting, it’s true with edge deployed cloud networks the physical location of node matters significantly less.

This could actually be a great case study to increase Cardano network reliability and speed in general if thousands or millions of relay nodes were deployed for passive participation in the network. It wouldn’t take any of the onus or need away from dedicated servers and block producing nodes but it would greatly benefit users as far as overall performance. As it stands there isn’t really an incentive to leave Daedalus running or deploying passive nodes in cloud hosting other than for giggles.

Other than becoming a SPO, the problem of creating incentive for network participation may solve itself once there are enough popular DApps in the future …