You may have seen some recent commentary on social media about an issue a stake pool operator encountered related to an older firmware variant on AMD Ryzen processors. In line with common industry practice, we will very rarely comment on or publicize such discoveries until fully fixed and patched. Due to the slightly unusual nature of this issue, and some social media coverage, we wanted to share further detail this time in order to allay any community concerns.
We’d first like to start by thanking ADAPH stake pool for their support and professionalism in this; first, by reporting this issue to IOHK’s infosec team and for their diligent and committed support to the community in identifying the root cause, and confirming the fix, which was successfully applied as part of the recent 1.19.1 node update.
WHAT HAPPENED
The issue was discovered as ADAPH set up its pool for the first time. After getting everything compiled per the documentation, the next step took them through generating the required keys and addresses. Upon generating a new wallet address, ADAPH discovered that the ‘new’ wallet already contained some ADA.
Believing this to be an error, they tried testing a new address several times and each time, the same address was generated, containing the same amount of ADA. At this point, they reached out to our DevOps team, who in turn brought in me as Director of Cybersecurity.
Our immediate priority was confirming this was an edge case and not an issue with wider implications. We first checked with our TSD team that no similar issues had been reported. Confirming this to be the case, we immediately conducted extensive testing across a wide number of platforms and configurations to try and recreate the issue. We were unable to do so.
After testing extensively, we discovered that the issue was recreatable ONLY on a specific firmware variant of AMD Ryzen-based machine. This issue was caused by a known hardware issue with random number generation on AMD Ryzen CPUs. The firmware fix for that issue was released more than 1-year ago, so in fact, this was caused by older, unpatched hardware. We were unable to recreate the issue on other platforms or on Ryzen machines running updated firmware. However, to be completely sure, we wrote a small piece of code that would address the issue and prepared it for deployment.
We were unable to identify the original creator of the wallet - and thus, the owner of that ADA. So we transferred the ADA to a new wallet address, to safeguard it, and expecting that the owner would likely step forward. Indeed, this happened (CHARM pool) who initially thought their funds had been stolen. Upon identifying themselves publicly in social media, we immediately reached out and returned the funds to them. CHARM subsequently confirmed the issue was resolved, again publicly: in an abundance of caution, we did ask them to remove one sensitive tweet
RESOLUTION
The issue was patched as part of release 1.19.1. To be clear, this was an edge-case issue caused by an old firmware, on a CPU with a known vulnerability. This was an extremely unlikely occurrence - and we reached out to our independent auditors who confirmed this. After continued testing, we are clear that this already edge-case issue will not be repeatable.
At no time was any ADA holder’s stake at risk from within Daedalus. And the chances of another SPO encountering the same issue was extremely low. We’d again like to thank ADAPH pool for their responsible reporting of this issue. It is important that SPOs ensure their systems are running all the latest software and firmware, to minimize the possibility of such security issues. We shall be stepping up our education activities in this area to ensure we are giving the community our full support in keeping their operations secure and safe.
