Sundae Swap Launch Forensics: "Queue Jumping"

Sundae Swap Launch Forensics: “Queue Jumping”

There has been a lot of talk about the Sundae Swap release on other forums, the queue jumping and who did such a thing. Much of the work published on other forums has been very poorly done, representing a widespread misunderstanding about the way the UTXO model works and leading to confusion and invective rather than clarity. I hope this helps steer the community discussion in more productive directions.

Overview

Looking at dApp releases, there is often a strong financial incentive to being first in line. In the recent Sundae release, we had the equivalent of a night club having a large queue out front waiting to get in, while someone slipped in through the unlocked back door and was able to buy drinks at the bar before anyone else got in. Looking at the history of extreme gas fees that users on platforms like Ethereum were willing to pay to get to the front of the line, this is neither a new problem, nor should it have been an unexpected one.

We do not approve of this type of behavior, it goes against our deeply held convictions of fair play and is not representative of the type of community we want to be a part of. While code is law on the blockchain - we hope to see better code coming out of this, building on the lessons learned.

Background:

Sundae Swap is an automated market maker that allows for trading of different pairs of digital assets. It uses a constant product market maker model (similar to Uniswap or SushiSwap), such that when you buy a token, the price goes up according to a set formula. When SundaeSwap went live, there was a limited supply of their governance token in circulation such that buying it on the DEX was the only way for most buyers to acquire the coin. This one sided market created the expectation that the price of SUNDAE would rise.

Sundae Swap is an open protocol, such that any properly formatted and funded transaction will be accepted. When the protocol was enabled in advance of the web page, this created a window of opportunity for people to submit transactions prior to the users of the webpage.

Prior to the Sundae Swap launch, two transactions were constructed that allowed their owners to jump the queue and get their transactions executed first. When the protocol went live, this allowed their owners to transact first and achieve the most favorable price, given the expectation of a rapid rise in price.

The two transactions that “jumped the queue” and were executed before anyone else:

700K Ada Transaction #1
txn_01 = 2d6a2469cc7d6901c8540860dd7754eab09c03862c0a18c6f5df0ffa1d7e53aa

400K Ada Transaction #2
txn_02 = 46c9ff237001f0d30882fab86a739da6c5e94e8857da4f487d212e20b2b4fad5

The UTXO Model

This is a slight digression on how the Cardano transaction model works; which may already be familiar with many experienced users of the platform, so feel free to skip this section. If you are not very familiar with the model, this may help you understand how to track transactions on-chain and what the explorer is showing you.

The heart of the idea is that transactions have outputs. Each output must be fully spent; any left over amount is returned to you as a change transaction.

Once the transactions are used once, they can never be used again. The platform tracks all the outputs that have not yet been spent, which are called “unspent transaction outputs” or UTXOs.

When a transaction has only one output, it is often a user transferring a full balance to somewhere else that they control. When a transaction has multiple outputs, one of them is probably going back to the original user and often represents a change in control of the funds.

While the Cardano platform is pseudonymous, you don’t need to directly identify yourself, all records are permanent and immutable, allowing you to track funds all the way back to when they originally were introduced into the system. By analyzing when and how a user uses funds, you can learn a significant amount about who is using the system, what times of day, week or month they tend to work, and even whether they are holding Ada as an investment, participating in the ecosystem or actively trading.

As funds move around, such as moving onto or off of an exchange (or as we will later see) a custody provider, funds from many different sources can become intermingled. One the blockchain, “7 degrees” of separation is more like “3 degrees”.

Tools:

I will be using links to two public tools to help visualize what is happening.

https://explorer.cardano.org/en

In practice, these tools are not very useful for actually doing the analysis though - automated tools are far more effective at analyzing the data, classifying common patterns. If you are interested in digging deeper, you will want to use a programmatic interface such:

A full node Cardano wallet will also allow you to customize your queries via the CLI or the associated postgres database (db-sync).

Forensics

Following from the two transactions used to trade on the DEX, they can be traced to both originating from the same wallet.

Wallet #1: The “user” wallet

addr1qxw876l6pafu63z25muwvzt7fgjf9l7qtfmh4j5dq0v44xhtj5fxf9czefjucs9nkmlyrkuguj0rdll0knagj9wull7qh79w83

stakeKey = stake1u84e2ynyjupv5ewvgzemdljpmwywf83kllhmf75fzhw0llqgpvlgt

Wallet #1 on Cardano Explorer

Wallet #1 at Pool.pm

This address was created on 20th Jan 2022 and was last used on 13 Feb 2022. By looking at the timing and value of the transactions, we might suspect that since the bulk of the transactions happen during the American day that the user in question lives anywhere from Brazil to the Pacific Coast - but probably not Asia or Europe.

Wallet #2: The “hot” wallet

The funds for the user wallet came from a second, larger wallet; one with only a small number of transactions associated with it. However, this address had an associated staking key. Normally wallets belonging to the same user share a stake key, so this indicates that this might be a separate entity than wallet #1. However, we can use that staking key to identify other addresses associated with this one. We see funds flowing from a single large wallet, through this address on the way to the user wallet.

addr1q86z744rp9q3jv7jvz8e0fhfzg46jp95zm52w56mv5qk53swq3wh8ycd3n4ulgmjsd7yg99dexmrjn5jvq02zndyl53spnpsuw

stakeKey = stake1uxf53lmw7479rehspdstly4vqmgrtsuzev5nghplh748wrsz8scjy

Wallet #2 on Cardano Explorer

Looking at the timing of the transactions, 03:00 UTC is 19:00 PST, at the end of the US business day, so we can infer that this transaction was initiated by a user during the day and was paid out after the close of business for a requestor based in the Americas.

Wallet #3: The “Cold” wallet

Of the associated addresses linked by that staking ID, there is one of particular interest. This wallet holds close to a billion Ada; several percent of the total supply. It is one of the biggest of the big whales.

addr1q8g77agasft90nnrln487665tf4nz9tt0z9df0l5z637yavnfrlkaatu28n0qzmqh7f2cpksxhpc9jefx3wrl0a2wu8qu5g9nq

stakeKey = stake1uxf53lmw7479rehspdstly4vqmgrtsuzev5nghplh748wrsz8scjy

Wallet #3 on Cardano Explorer

Wallet #3 on Pool.pm

This is a rather more interesting wallet, we can see that it is used around the clock, and while activity does spike in the middle of the European business day, we see heavy use around the clock.

The inputs are highly structured, implying an automated process that triggers slightly after midnight UTC to move client funds from their incoming accounts to what appears to be a cold wallet.

The timing of the payments by day is also interesting, we see large initial inflows on the 29th of October, followed by much lower daily volumes of around 50 transactions a day involving one or two million Ada.

If you dig into the addresses feeding this account, there was a large initial transfer from a single wallet, and then subsequently around 3500 active accounts each that tended to deposit and withdraw funds with half a million Ada on quarterly frequency.

Drawing a picture will help visualize what is happening here.

Based on what we know about the flows in (they are working in different time zones, there are several thousand of them, they each own more than $500,000 worth of Ada), this looks like a custody solution for high net worth individuals. Exchanges by contrast would show more frequent, lower value transactions. Individual firms would tend to show more specific timings when they moved funds around (large flows of funds typically need to be approved by a senior executive and tend to happen towards the end of a business day of the region where that officer resides.)

If you are not familiar with a custody business, one well known example is Coinbase Custody. For a fee, they provide value added services like securing digital assets, insuring them against loss and various types of financial and regulatory reporting services.

Timing, types of activity, and a basket of native assets or the presence of NFTs are all techniques that could potentially be used here.

Wallet #4: The Previous “Cold” wallet

Following the big inflow of funds in Wallet #3, we can track back the initial transfer as coming from another account. This account uses an older style address (a “Byron” style address) where we can see a more organic growth of funds. This particular custody business likely started supporting Ada on the 3rd of July, 2020.

What this all means

The user who jumped the queue for Sundae Swap withdrew the funds from a custody solution. He or she is likely located geographically somewhere from Brazil to the Pacific coast, they are relatively wealthy, owning more than a million Ada, but probably less than 3 million (which can be determined by looking at the flows into the custody platform - there are a couple very large holders but the vast majority are in the one to three Ada range.) If legal proceedings were to be taken against this individual, then a subpoena to the custody provider would be able to positively identify them.

Coinbase Custody is a large, well-regarded and well-regulated custody business. It holds funds from several thousand parties, ranging from companies with significant crypto holdings (such as IOHK) to individuals (which will include both current and former members of staff.) Around 10% of all Ada is directly connected to IOHK funds, and practically all Ada is within 3 transactions of being on an exchange where it would have mixed with funds originating from IOHK. Having funds originating from IOHK deposited at a large custody company, along with funds from thousands of others, is a statistical certainty - not a smoking gun.

Lessons Learned

One lesson coming out of the SundaeSwap launch was that auditing needs to go beyond just checking the security and correctness of the code, but also expanded to include the launch strategy. We have moved beyond a point where you can just “do your own research”; all too often that means just recreating the mistakes of the past. Instead, there is a huge need for better standards and a review of best practices for launching dApps. These are issues where we need broad engagement from the wider community.

28 Likes

Thanks alot for this well done research @Colin_Edwards :slight_smile:

8 Likes

Great breakdown colin
This has been rattling around the internut for ages
Really glad you put the smoking gun in the statistical certainty bin …

Here’s looking forward to the next level of auditing as per your suggestion

2 Likes

Thank you for taking the time to provide clarity.

2 Likes

Because of this thread

I have incidentally looked at wallets #2 and #3 in some detail just today.

I don’t know if your conclusion of a custody service is correct.

During the last few hours, addr1…a2wu8qu5g9nq has moved an impressive amount of ADA to addr1…swdfpqxu74xt and addr1…dyl53sgu4pen in very regular chunks of 800k ADA.

In the past days before that, it had incoming transactions like 207ace0a100bc64c8910a64bc3a920d49e1ecd9cc09a3032d6a5af4e07451f3d, where a lot of small amounts of ADA were collected from lots and lots of Byron addresses.

The other two wallets regularly have outgoing transactions like 58b0601cea08faf3243d95fb08234ad324d2f1d869b626d17ba64c930027eb78 or 06fb39945277d0c25a79ea5db427c3b4339eecd90d8616fc3e892ced24233ee1, where widely varying including very small amounts go to a wide variety of addresses.

I would interpret this altogether as three wallets belonging to the same exchange and they internally distribute their assets in 800k chunks between these three wallets, where two are used for payouts to customers and one was used to get funds from old Byron addresses, but is just in this moment in the process of being retired or something like that.

EDIT: When you look at the two transactions funding wallet #1b7418577cb7265c7a986595d57b9ae9b355b4e51dfda400e1bb0732ae6cca48f and e6225c16c04e22c7286de58ed0cb2bcd57b02ff85829e0757875e7c7f1fb3a72 – they come exactly from these two wallets and they both collect a huge amount of UTxOs to get the sum of two times almost a million ADA together. These UTxOs look rather typical for a regular exchange, not necessarily a custody for high amounts.

Why might this be important? In the other thread, someone identified this as the end location of ADA scammed from them. If it was a custody solution that might mean that the scammer could be identified as one of these few high-wealth individual customers. If it is just a regular exchange, the chances are much slimmer.

3 Likes

A couple things would indicate otherwise - the average account coming in was worth 500K+, there were only several thousand accounts and maybe 50 transactions a day. An exchange would see many more accounts and the transactions would be in much smaller amounts and higher frequency.

The activity tended to be much heavier during European and American hours, rather than Asia and besides that, there aren’t that many billion dollar Ada exchanges once you exclude the Asian exchanges.

If it was a custody solution that might mean that the scammer could be identified as one of these few high-wealth individual customers.

Yes, any product whose customers start at half a million USD and go up from there is going to have not just KYC but KYC++.

3 Likes

Thank you for the great information… definitely an eye opener for me!

2 Likes

Interesting, thanks a lot.

2 Likes

To the Reddit accusers, if anyone honestly thought Charles would jeopardize all his work by pulling a stunt like front running Sundae Swap, they need their head examined.

2 Likes

@almahmoud84
Cardano is a proof-of-stake blockchain so it doesn’t do mining in the Bitcoin sense. You can stake your coins to a stake pool and this contributes to the security of the network. If you stake your coins then you get a share of the staking rewards generated by the protocol.

Have a read about staking and pool delegation here:
https://cardano.org/stake-pool-delegation/

2 Likes

Most of the economics are very similar for pools with more than about 5 million in stake; selecting a pool based on its mission or community is a perfectly reasonable thing to do.

A couple useful resources:

2 Likes

Is there ever going to be a follow up here on these queue skippers?

Or is the point of the exercise just to see how we might avoid same/similar in the future?

Professionally, I thought it was an interesting problem. I mostly just wanted to see if I could figure out what happened.

1 - That’s a question for the SundaeSwap team, but my feeling is there is not much that can be done. (My understanding is that a properly formatted hand-crafted transaction will still work - you don’t need to use their webpage.)

2 - Other product launches in the future should consider possibilities like this; audits should be more holistic than just looking at code correctness and cover things like release timing.