Cardano-node in docker container exception when Shelley period starts

Developers Misc Dev Talk
1

Hello, when I run docker container for cardano-node from IOHK docker hub, everything is running smoothly, but after the node reaches the Shelley era (last blocks synced in postgres DB are from 29.7. 2020) it stops on the exceptions outlined below. I tried it on two different cardano-node versions (1.19.1 and 1.20.0) completely from scratch, both produced the same result.

Does anyone experienced this? Any idea where could be the problem? Thank you.

[String "Application Exception: 54.150.68.245:3001 HeaderError 
(At 
    (Block 
      {blockPointSlot = SlotNo 4492800, blockPointHash = aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de}
    )
  ) 
  (HeaderProtocolError 
    (HardForkValidationErrFromEra S 
      (Z 
        (WrapValidationErr 
          {unwrapValidationErr = ChainTransitionError 
            [OverlayFailure 
              (VRFKeyBadNonce 
                (Nonce \"81e47a19e6b29b0a65b9591762ce5143ed30d0261e5d24a3201752506b20f15c\") 
                (SlotNo 4492800) 
                (Nonce \"c75a45fd987dd42259583064ed847c43d18317509f3e72c97cd53dd9243c2af3\") 
                (CertifiedVRF 
                  {certifiedOutput = OutputVRF 
                    {getOutputVRFBytes = \"6\\236Sx\\209\\245\\EOT\\SUBY\\235\\141\\150\\230\\GS\\233o\\tP\\251A\\180\\159\\245\\DC1\\247\\188\\DEL\\209\\t\\212\\&8>\\GS$\\190p4\\230t\\156f\\DC2p\\r\\213\\206\\176\\198ew\\184\\138\\EM\\174(k\\DC3!\\209[\\206\\SUB\\183\\&6\"}, 
                    certifiedProof = CertPraosVRF \"@Z\\163p\\255\\NUL\\149D\\162\\190J\\165\\188R\\196Ec3\\244\\249\\182W\\GSf\\197Y\\r.V)\\208\\139:6\\t\\244\\189\\ENQ\\STX\\237]\\v\\225\\171\\219\\DEL*\\183j\\174\\174G\\254\\DC1\\ESC\\ETX5\\164\\228\\222\\246F\\147\\SYN'\\148\\184\\211\\193\\202qP\\SI\\SYN\\177\\226DrL\\ETX\"
                  }
                )
              )
            ]
          }
        )
      )
    )
  ) 
  (Tip 
    (SlotNo 4492799) f8084c61b6a238acec985b59310b6ecec49c0ab8352249afd7268da5cff2a457 
    (BlockNo 4490510)
  ) 
  (Tip 
    (SlotNo 9549648) 5c51d3544abba7f631987c8246f479302e67718c96f6756aaa0aae245bc96493 
    (BlockNo 4739742)
  )",
  String "SubscriptionTrace",
  String "\"relays-new.cardano-mainnet.iohk.io\""
] 
[8ae4fa72:cardano.node.ErrorPolicy:Warning:96] 


[String "ErrorPolicySuspendPeer 
  (Just 
    (ApplicationExceptionTrace 
      (HeaderError 
        (At 
          (Block 
            {blockPointSlot = SlotNo 4492800, blockPointHash = aa83acbf5904c0edfe4d79b3689d3d00fcfc553cf360fd2229b98d464c28e9de}
          )
        ) 
        (HeaderProtocolError 
          (HardForkValidationErrFromEra S 
            (Z 
              (WrapValidationErr 
                {unwrapValidationErr = ChainTransitionError 
                  [OverlayFailure 
                    (VRFKeyBadNonce 
                      (Nonce \"81e47a19e6b29b0a65b9591762ce5143ed30d0261e5d24a3201752506b20f15c\") 
                      (SlotNo 4492800) 
                      (Nonce \"c75a45fd987dd42259583064ed847c43d18317509f3e72c97cd53dd9243c2af3\") 
                      (CertifiedVRF 
                        {certifiedOutput = OutputVRF 
                          {getOutputVRFBytes = \"6\\236Sx\\209\\245\\EOT\\SUBY\\235\\141\\150\\230\\GS\\233o\\tP\\251A\\180\\159\\245\\DC1\\247\\188\\DEL\\209\\t\\212\\&8>\\GS$\\190p4\\230t\\156f\\DC2p\\r\\213\\206\\176\\198ew\\184\\138\\EM\\174(k\\DC3!\\209[\\206\\SUB\\183\\&6\"}, 
                          certifiedProof = CertPraosVRF \"@Z\\163p\\255\\NUL\\149D\\162\\190J\\165\\188R\\196Ec3\\244\\249\\182W\\GSf\\197Y\\r.V)\\208\\139:6\\t\\244\\189\\ENQ\\STX\\237]\\v\\225\\171\\219\\DEL*\\183j\\174\\174G\\254\\DC1\\ESC\\ETX5\\164\\228\\222\\246F\\147\\SYN'\\148\\184\\211\\193\\202qP\\SI\\SYN\\177\\226DrL\\ETX\"
                        }
                      )
                    )
                  ]
                }
              )
            )
          )
        ) 
        (Tip 
          (SlotNo 4492799) f8084c61b6a238acec985b59310b6ecec49c0ab8352249afd7268da5cff2a457 
          (BlockNo 4490510)
        ) 
        (Tip 
          (SlotNo 9549648) 5c51d3544abba7f631987c8246f479302e67718c96f6756aaa0aae245bc96493 
          (BlockNo 4739742)
        )
      )
    )
  ) 200s 200s",
  String "ErrorPolicyTrace",
  String "54.150.68.245:3001"
]
2

Has anyone else experience this? I’m having the same error with my testnet node at block 1598400.

Mainnet deployment is working fine but testnet is not having any of it. I’ve delete both the db and the ledger state folder. Did a new build with a new machine still failed with the same error at block 1598400.

3

Yes, I’ve setup a new node yesterday and I get the same errors at exactly the same block. Anyone got this fixed yet?

4

ps, I’m still having this issue but only with my test nodes. My mainnet nodes are working just find.
I just updated both nodes to 1.27.0 yesterday and deleted my testnest and rsync and had the exact same error at the exact same epoch. I notice the latest version of the config file (and my mainnet) has “ApplicationVersion”: 0; my test config previously had 1.

I just redownloaded all the config files and deleted my db files again with a new fresh sync and node version 1.27.0. We’ll see where we’re at tomorrow!

If you get it to work, please update this thread. I’ve posted the thread to many of the discord channel with no luck.

5

The issue starts to get discussed here

1 Like
6

If anyone else is having this issue with the testnet, here is a github ticket tracking, for the the testnet only: [BUG] - Sync error when runnning on testnet · Issue #2734 · input-output-hk/cardano-node · GitHub