DNS Address fails while IP address is working

I am trying to establish a Stakepool with Kubernetes.
Now i faced an issue with the peer nodes.
If i have configure “addr” attribute in the topology.json file with a dns Address i get a warning like this:

[testnet-:cardano.node.DnsSubscription:Warning:54] [2021-05-26 15:22:42.12 UTC] Domain: "relay02.orkastaker.com" Failed to start all required subscriptions
[testnet-:cardano.node.DnsSubscription:Warning:60] [2021-05-26 15:22:42.15 UTC] Domain: "tn.adastack.net" Failed to start all required subscriptions
[testnet-:cardano.node.DnsSubscription:Warning:57] [2021-05-26 15:22:42.15 UTC] Domain: "topo-test-bp.topopool.com" Failed to start all required subscriptions
[testnet-:cardano.node.DnsSubscription:Warning:63] [2021-05-26 15:22:42.15 UTC] Domain: "testnet-seeland-stakepool-producing" Failed to start all required subscriptions

I also tried it with the official iohk node relays-new.cardano-testnet.iohkdev.io with the same result.
Strangely it works if i give the ip address in the topology.json file.
First i was thinking about an issue with Kubernetes but if i use telnet from within an ubuntu pod it is able to connect to the address sucessfully:

root@ubuntu:/# telnet relays-new.cardano-testnet.iohkdev.io 3001
Trying 18.132.238.21...
Connected to relays-new.cardano-testnet.iohkdev.io.

I build the Cardano Node inside a Docker Container with Version 1.27.0. is there something special i need to consider when using a dns address?

The topology.json file for the relay node looks like this:

{
  "Producers": [
    {
      "addr": "relay02.orkastaker.com",
      "port": 6000,
      "valency": 1
    },
    {
      "addr": "topo-test-bp.topopool.com",
      "port": 3001,
      "valency": 1
    },
    {
      "addr": "tn.adastack.net",
      "port": 3001,
      "valency": 1
    },
    {
    "addr": "18.132.238.21",
      "port": 3001,
      "valency": 1
    },
    {
    "addr": "testnet-seeland-stakepool-producing",
    "port": 3000,
    "valency": 1
    }
  ]
}

While testnet-seeland-stakepool-producing is the kubernetes service name for the block producing node.

Your node/public IP should have a dns entry in order to use dns instead of IP

Do u have a dns registration/record for your public IP?

What i understood from here, i need to connect my relay noded to other peer nodes. The nodes which are listed in my topology.json are nodes which i found in this list (exept of course the block producing node inside of my kubernetes Cluster testnet-seeland-stakepool-producing).
My node has a dns entry but only for my domain seelandstakepool.ch which points to the IP address of my VM.

Sorry, I misunderstood :slight_smile:

Did u started the node with host-address 0.0.0.0 ?

No issues :slight_smile: yes i did start my node with --host-addr 0.0.0.0

Ok, try to keep only the iohk nodes in topology files and restart the node

https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/testnet-topology.json

Same error?

Yes unfortunately the same error:

[seeland-:cardano.node.dns-producers:Notice:5] [2021-05-26 19:08:49.68 UTC] [DnsSubscriptionTarget {dstDomain = "relays-new.cardano-testnet.iohkdev.io", dstPort = 3001, dstValency = 2}]
[seeland-:cardano.node.ip-producers:Notice:5] [2021-05-26 19:08:49.68 UTC] IPSubscriptionTarget {ispIps = [], ispValency = 0}
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Opened imm db with immutable tip at genesis (origin) and chunk 0
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Opened vol db
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Replaying ledger from genesis
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Opened lgr db
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Opened db with immutable tip at genesis (origin) and tip genesis (origin)
[seeland-:cardano.node.ChainDB:Notice:34] [2021-05-26 19:08:49.73 UTC] Chain extended, new tip: 1e64e74bd7ac76d6806480a28017deb0aedd356fb61844ec95c429ae2f30c7c3 at slot 0
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:08:49.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:08:59.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:09:09.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[seeland-:cardano.node.ChainDB:Notice:76] [2021-05-26 19:09:18.86 UTC] before next, messages elided = 2259510500755060
[seeland-:cardano.node.ChainDB:Notice:76] [2021-05-26 19:09:18.86 UTC] Chain extended, new tip: 1e64e74bd7ac76d6806480a28017deb0aedd356fb61844ec95c429ae2f30c7c3 at slot 0
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:09:19.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:09:29.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions

Are you able to do a manual DNS query ? Using dig or nslookup ?

U said that if u are replace dns names with ip is working right?
Then I’m suspecting a FW issue

Yes if i use an ubuntu pod withing the same namespace i am able to query the dns:

root@ubuntu:/# nslookup relays-new.cardano-testnet.iohkdev.io
Server:         10.96.0.10
Address:        10.96.0.10#53

Non-authoritative answer:
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 54.151.49.138
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 3.128.220.55
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 18.132.238.21
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 3.9.80.183
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 54.250.11.33
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 3.124.178.139
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 175.41.160.108
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 52.58.117.45

Sorry, I’m not familiar with pods configuration. From Kubernetes doc it appears there may be a few known issues related to the way systemd handles DNS resolution, see Debugging DNS Resolution | Kubernetes

Hi!

That is interesting…
Have you tired to start the pod on host network?

No but i already executed dnslookup directly from the releay pod which was successful. Therefore i do not expect an issue with the firewall or kubernetes:

root@seeland-stakepool-relay1-77587998b4-zj6k5:/config# nslookup relays-new.cardano-testnet.iohkdev.io
Server:		10.96.0.10
Address:	10.96.0.10#53

Non-authoritative answer:
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 54.250.11.33
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 3.129.133.68
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 3.9.80.183
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 3.124.178.139
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 175.41.160.108
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 18.159.64.253
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 54.241.77.32
Name:	relays-new.cardano-testnet.iohkdev.io
Address: 3.128.220.55

yes, I think the cardano-node has issues as well… but as a workaround just to see what happens if host network is used…

The same if i run the Pod with hostNetwork: true.
I can use the IP address as a workaround but do i need to report this as an issue in the cardano-node Github Repository?

Yes, I recommend to submit an issue…
And please share the ticket once you are done with it.

I just created the this Bug in the Github Repository.

1 Like