Help with Testnet relay error

bclens · 14 February 2021 23:39

I am new to creating stake pools and am a bit lost. Hope someone can assist me. I started building a test relay node on an Ubuntu Virtualbox yesterday and left it running overnight to build the db. This morning I discovered that my host OS (Windows) rebooted because of Microsft updates

I started running the node again and I got all sort of errors (see snippet below):

[relay1:cardano.node.DnsSubscription:Error:6743] [2021-02-14 23:26:19.16 UTC] [String “Application Exception: 3.14.157.200:3001 InvalidBlock (At (Block {blockPointSlot = SlotNo 18023840, blockPointHash = c9881ce23d270f6b00369017015621e5735f029c8006e2077ab37c54dc7ef4d2})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (UtxowFailure (UtxoFailure (OutputTooSmallUTxO [(Addr Testnet (KeyHashObj (KeyHash “53c493ab318b98d4bd5bfebee157f33a2c2dcf2752e818d558f8867d”)) (StakeRefBase (KeyHashObj (KeyHash “54251fc51d128c9808585c95f66b0c5146dddf8fafe8dbf13de65774”))),Value 1500000 (fromList [(PolicyID {policyID = ScriptHash “9cebd568ac4ad908cdd2d45a52327fc4711911054c9951b90a603303”},fromList [(“BerryIndigo”,1)])]))]))))])})))))))”,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[relay1:cardano.node.ErrorPolicy:Warning:56] [2021-02-14 23:26:19.16 UTC] [String “ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (InvalidBlock (At (Block {blockPointSlot = SlotNo 18023840, blockPointHash = c9881ce23d270f6b00369017015621e5735f029c8006e2077ab37c54dc7ef4d2})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (UtxowFailure (UtxoFailure (OutputTooSmallUTxO [(Addr Testnet (KeyHashObj (KeyHash “53c493ab318b98d4bd5bfebee157f33a2c2dcf2752e818d558f8867d”)) (StakeRefBase (KeyHashObj (KeyHash “54251fc51d128c9808585c95f66b0c5146dddf8fafe8dbf13de65774”))),Value 1500000 (fromList [(PolicyID {policyID = ScriptHash “9cebd568ac4ad908cdd2d45a52327fc4711911054c9951b90a603303”},fromList [(“BerryIndigo”,1)])]))]))))])})))))))))) 200s 200s”,String “ErrorPolicyTrace”,String “3.14.157.200:3001”]

It does not seem to be running properly as the slot number does not update when I run this command:
$ cardano-cli query tip --testnet-magic 1097911063
{
“blockNo”: 2287371,
“headerHash”: “aef965e0a82c1103e78454cfc6153996410126aa826176a3a16bb38d729d9f27”,
“slotNo”: 18023833
}

Tried reading through the forum and there was a suggestion to make sure Protocol is set to Cardano in the testnet-config.json. Mine is already set to Cardano so the problem may be something else.

Can someone be kind enough to give any suggestions please?

bclens · 15 February 2021 11:01

Update. I decided to try again so I deleted the db directory and ran the node again. It started again from the first epoch… but then a few long hours later, it again shows similar errors.

Can current Stake Pool Operators please share your experience if it really is this difficult to setup a node? Have you encountered similar errors? Excerpt of the part where it fails is below. Hope someone can help…

[relay1:cardano.node.ChainDB.metrics:Critical:41] [2021-02-15 08:41:35.77 UTC] slotInEpoch = 9433
[relay1:cardano.node.ChainDB.metrics:Critical:41] [2021-02-15 08:41:35.77 UTC] epoch = 112
[relay1:cardano.node.ChainDB:Error:41] [2021-02-15 08:41:35.77 UTC] Invalid block c9881ce23d270f6b00369017015621e5735f029c8006e2077ab37c54dc7ef4d2 at slot 18023840: ExtValidationErrorLedger (HardForkLedgerErrorFromEra S (S (S (Z (WrapLedgerErr {unwrapLedgerErr = BBodyError (BlockTransitionError [LedgersFailure (LedgerFailure (UtxowFailure (UtxoFailure (OutputTooSmallUTxO [(Addr Testnet (KeyHashObj (KeyHash “53c493ab318b98d4bd5bfebee157f33a2c2dcf2752e818d558f8867d”)) (StakeRefBase (KeyHashObj (KeyHash “54251fc51d128c9808585c95f66b0c5146dddf8fafe8dbf13de65774”))),Value 1500000 (fromList [(PolicyID {policyID = ScriptHash “9cebd568ac4ad908cdd2d45a52327fc4711911054c9951b90a603303”},fromList [(“BerryIndigo”,1)])]))]))))])})))))
[relay1:cardano.node.ChainDB:Info:41] [2021-02-15 08:41:35.77 UTC] Valid candidate aef965e0a82c1103e78454cfc6153996410126aa826176a3a16bb38d729d9f27 at slot 18023833
[relay1:cardano.node.DnsSubscription:Error:24798] [2021-02-15 10:38:07.77 UTC] [String “Application Exception: 54.151.49.138:3001 MuxError MuxBearerClosed “<socket: 26> closed when reading data, waiting on next header True””,String “SubscriptionTrace”,String ““relays-new.cardano-testnet.iohkdev.io””]
[relay1:cardano.node.ErrorPolicy:Warning:57] [2021-02-15 10:38:07.77 UTC] [String “ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed “<socket: 26> closed when reading data, waiting on next header True”))) 20s 20s”,String “ErrorPolicyTrace”,String “54.151.49.138:3001”]

bclens · 15 February 2021 13:33

Realized I posted this in the Operate A Stake Pool section. Probably who no one is replying. Reposting it here in the Setup A Stake Pool Section.

Additional info. I did a check on the tip and I got the same slotNo when it hanged the first time.

$ cardano-cli query tip --testnet-magic 1097911063
{
“blockNo”: 2287371,
“headerHash”: “aef965e0a82c1103e78454cfc6153996410126aa826176a3a16bb38d729d9f27”,
“slotNo”: 18023833
}

Seems it is getting stuck at 18023833. Any way for me to clean up / delete the last transaction or epoch so I can proceed?

bclens · 17 February 2021 09:23

Hello. Any kind soul want to point me in the right direction please?

oneforall · 17 February 2021 23:51

I ran into the same problem while following the stake pool course. Public testnet now requires Cardano Node 1.25.1. After updating I was able to continue the sync.

Hoang · 18 February 2021 17:54

Hi I got the same error. So I will try update cardano. Do I need to set up everything or simply cardano-node ?

oneforall · 19 February 2021 03:18

Just Cardano Node.

bclens · 19 February 2021 22:17

Thanks oneforall. I redid all the steps but used 1.25.1. Unfortunately, it stopped at the same slotno again (18023833). Do you have any other tip on what I may be doing wrong?

oneforall · 21 February 2021 04:01

Hmm. What does “cardano-node --version” show?

allfs · 21 February 2021 16:14

I found the solution. Just start with a fresh cardano-node folder, checkout to 1.25.1 and build it. When you are building, don’t do anything else if you dont have enough ram memory could throw some errors and dont finish properly.

cd
mkdir -p ~/src
cd ~/src
git clone https://github.com/input-output-hk/cardano-node.git
cd cardano-node
git fetch --all --tags
git checkout tags/1.25.1
cabal configure --with-compiler=ghc-8.10.2
echo "package cardano-crypto-praos" >>  cabal.project.local
echo "  flags: -external-libsodium-vrf" >>  cabal.project.local
cabal update
cabal build cardano-node cardano-cli
cp -p dist-newstyle/build/x86_64-linux/ghc-8.10.2/cardano-node-1.25.1/x/cardano-node/build/cardano-node/cardano-node ~/.local/bin/
cp -p dist-newstyle/build/x86_64-linux/ghc-8.10.2/cardano-cli-1.25.1/x/cardano-cli/build/cardano-cli/cardano-cli ~/.local/bin/
cardano-cli --version

You can reuse your last blockchain. Now it should continue extending new blocks.

bclens · 25 February 2021 04:29

Thanks. It didn’t work for me. Had reused my db. Will retry later maybe with a fresh install and new db

ryan.baron · 13 March 2021 12:35

This was the fix for me, thank you @allfs. After updating the cardano-node from 1.24.2 to 1.25.1 the chain continued to be extended past 18023833.