AMD Ryzen firmware - report on recent patch

You may have seen some recent commentary on social media about an issue a stake pool operator encountered related to an older firmware variant on AMD Ryzen processors. In line with common industry practice, we will very rarely comment on or publicize such discoveries until fully fixed and patched. Due to the slightly unusual nature of this issue, and some social media coverage, we wanted to share further detail this time in order to allay any community concerns.

We’d first like to start by thanking ADAPH stake pool for their support and professionalism in this; first, by reporting this issue to IOHK’s infosec team and for their diligent and committed support to the community in identifying the root cause, and confirming the fix, which was successfully applied as part of the recent 1.19.1 node update.

WHAT HAPPENED

The issue was discovered as ADAPH set up its pool for the first time. After getting everything compiled per the documentation, the next step took them through generating the required keys and addresses. Upon generating a new wallet address, ADAPH discovered that the ‘new’ wallet already contained some ADA.

Believing this to be an error, they tried testing a new address several times and each time, the same address was generated, containing the same amount of ADA. At this point, they reached out to our DevOps team, who in turn brought in me as Director of Cybersecurity.

Our immediate priority was confirming this was an edge case and not an issue with wider implications. We first checked with our TSD team that no similar issues had been reported. Confirming this to be the case, we immediately conducted extensive testing across a wide number of platforms and configurations to try and recreate the issue. We were unable to do so.

After testing extensively, we discovered that the issue was recreatable ONLY on a specific firmware variant of AMD Ryzen-based machine. This issue was caused by a known hardware issue with random number generation on AMD Ryzen CPUs. The firmware fix for that issue was released more than 1-year ago, so in fact, this was caused by older, unpatched hardware. We were unable to recreate the issue on other platforms or on Ryzen machines running updated firmware. However, to be completely sure, we wrote a small piece of code that would address the issue and prepared it for deployment.

We were unable to identify the original creator of the wallet - and thus, the owner of that ADA. So we transferred the ADA to a new wallet address, to safeguard it, and expecting that the owner would likely step forward. Indeed, this happened (CHARM pool) who initially thought their funds had been stolen. Upon identifying themselves publicly in social media, we immediately reached out and returned the funds to them. CHARM subsequently confirmed the issue was resolved, again publicly: in an abundance of caution, we did ask them to remove one sensitive tweet

RESOLUTION

The issue was patched as part of release 1.19.1. To be clear, this was an edge-case issue caused by an old firmware, on a CPU with a known vulnerability. This was an extremely unlikely occurrence - and we reached out to our independent auditors who confirmed this. After continued testing, we are clear that this already edge-case issue will not be repeatable.

At no time was any ADA holder’s stake at risk from within Daedalus. And the chances of another SPO encountering the same issue was extremely low. We’d again like to thank ADAPH pool for their responsible reporting of this issue. It is important that SPOs ensure their systems are running all the latest software and firmware, to minimize the possibility of such security issues. We shall be stepping up our education activities in this area to ensure we are giving the community our full support in keeping their operations secure and safe.

15 Likes

Very nice explanation Charles, thank you! I find it amazing that a random number generating error would link to a wallet address with preexisting funds in it, what are the chances!

#ADAPH main operator here, all I can say is that the IOHK and Cardano team handled the issue very well. I couldn’t ask for a better team to handle this issue.

We will have a post about our perspective on this issue soon.

Thanks,
Clark

3 Likes

Welcome back @Mercurial and thank you much. I do not know if others would have acted the same way. :raised_hands:t2:

when two of a thousand faced these issue i wouldnt call it an edge-case, frankly speaking

1 Like

The core problem was, that the random number generator (under above stated circumstances) always spit out the same number, so it always generated exactly this one key.

2 Likes

LOL, as if we didn’t have enough challenges already. Thanks for clarification. I’m happy your team and IOHK and the SPO were able to resolve this with minimal losses.

1 Like

When I created the pool I spent a lot of time thinking about how much pledge to put. So for start I just put ADA earned by pool on ITN. If I had put all the ADA gathered in the last 3 years and seen them missing, I think I would have gone crazy.

Many thanks to #ADAPH SPO and to Charles Morgan for the professional way in which they treated the problem.

2 Likes