USDC v2: Upgrading a multi-billion dollar ERC-20 token
By Pete Kim
USD Coin (USDC) is a stablecoin brought to you by Centre, a consortium of which Coinbase and Circle are the founding members. Each USDC token is backed by one US dollar held in a bank, enabling the stablecoin to maintain a 1:1 exchange rate with the US Dollar.
USDC has grown significantly since its inception. It reached $500 million in market capitalization for the first time in December 2019, $1 billion in July 2020, and $3 billion in November 2020. The growth of USDC in 2020 was in large part fueled by the growth of Decentralized Finance (DeFi), where USDC remains the number one fiat-backed stablecoin of choice by both users and developers. DeFi’s core innovation is that it enables a wide array of applications such as lending, borrowing, and trading, in a global and permissionless manner. Various DeFi protocols can also be combined thanks to its programmable nature, and USDC acts as a medium of exchange between different protocols.
With cryptocurrencies at the center, DeFi presents huge opportunities for financial innovation. However, the volatile prices of many cryptocurrencies have been a barrier to mainstream adoption. For this reason, building a strong stablecoin infrastructure has been a critical part of Coinbase’s mission to build an open financial system for the world. The USDC stablecoin inherits many of the core innovations of cryptocurrency while maintaining the price stability of the US Dollar, making it ideal for use in DeFi applications.
The USDC Smart Contract
USDC is now available on multiple blockchain platforms, but it was originally launched on Ethereum. The original USDC smart contract, a fairly standard ERC20 token deployed in 2018, operated efficiently for two years. However, it has been long overdue for an upgrade.
Over the years, we’ve received lots of feedback from both users and developers. One aspect of the original ERC-20 smart contract that confuses many users is that in order to spend the tokens, you also need ETH to pay for the transaction fees. For example, if you bought USDC on Coinbase and transferred it to a user-controlled wallet such as Coinbase Wallet or MetaMask that contained no ETH, you could not spend USDC unless you also bought and transferred some ETH to that wallet. For developers, this limitation complicated onboarding as they had to ensure their users had both USDC and ETH, and this also made Venmo-type of use-cases difficult to build. In addition to addressing that issue, we wanted to bring many other general improvements to make USDC more secure for our users and developers.
Upgradeable Smart Contracts
Technically, a smart contract deployed on Ethereum is immutable. While this property is necessary for fully trustless applications, the caveat is that bugs or security flaws in the code cannot be corrected afterwards once the code is committed on the blockchain. On Ethereum, the proxy pattern can be used as a workaround for this limitation.
The general idea of the proxy contract pattern is to have users interact with a proxy contract, which forwards all function calls to the implementation contract that houses the actual logic. The implementation contract can be replaced, which makes the contract “upgradable”. The proxy contract can be constructed through the use of a special Ethereum opcode called DELEGATECALL. This opcode lets a contract borrow and execute code from another contract while preserving the calling contract’s context, such as the storage and the caller (msg.sender). With this opcode and a fallback function to catch any arbitrary function call, the proxy contract can keep the contract state in its storage and a separate implementation contract can contain the logic.
The proxy contract contains a variable that stores the address of the implementation contract, to which the fallback function relays the function calls. To upgrade the contract, the operator of the smart contract can simply deploy a new implementation contract and update the implementation contract address in the proxy contract so that it points to the newly deployed contract.
At first glance, the upgrade process described above may appear trivial. However, severe data loss and unexpected behaviors can occur if special care is not taken in designing the replacement implementation contract. There are two important factors to consider: 1. the way in which the state variables are laid out in the contract storage 2. that the contract state resides in the proxy contract rather than the implementation contract itself.
Storage Slots
In Ethereum, the state variables for a smart contract are laid out sequentially in what’s called storage slots, starting from position zero. There are complex rules that determine the storage slot positions for state values of different types and sizes, but in general, the slots are assigned in the order the variables are declared in the code.
Let’s consider the following example: contract Foo has two state variables called alpha and bravo, contract Bar has one state variable charlie, and contract Baz has two state variables delta and echo. Baz is the contract to be deployed on the blockchain.
Since Baz inherits from Foo and Bar in that order, Baz ends up having 5 state variables, declared in the following sequence: alpha, bravo, charlie, delta, and echo. As a result, the five variables are assigned storage slots at positions 0 through 4.
Now, if I were to update the code and add a new state variable called foxtrot to contract Bar, the order of the variables and the corresponding storage slot positions would change.
As illustrated above, this change causes a misalignment in the storage slot positions. If I replace the implementation contract with this new contract, the state variable foxtrot will then be located at position 3, delta at position 4, and echo at position 5. This causes the new variable foxtrot to erroneously have the value of delta prior to the upgrade, delta to have the value of echo, which isn’t even of the same data type, and echo to lose its value.
In the example shown above, instead of modifying the existing contract Bar, the new variable is introduced as a part of a new parent called Qux contract from which Baz inherits. Unfortunately, this results in the same misalignment in storage slot positions. The correct approach to avoid storage slot misalignment here would be to either introduce the new state variable in Baz after echo, or in a new contract that inherits from Baz.
There are other ways to avoid this problem besides carefully enumerating all of the state variables declared and ensuring that the order does not change. One simple way is to dedicate one contract to hold all of the state variables and have all other contracts inherit from it. Another way is to use a mapping to wrap the state variable so that the name of the field plays a role in the derivation of the storage slot. Finally, it’s also possible to specify the storage slots directly by using the EVM opcodes SLOAD and SSTORE.
There is no single best solution since each approach has drawbacks such as increased contract size, higher gas cost, or more complex code. If you are starting a new project, I recommend checking out newer design patterns such as the EIP-2535 Diamond Standard that are created with the upgradeability and composability in mind.
Testing Storage Slots
In the case of USDC v2, an accidental change in the storage slots could result in a loss of funds of more than a billion dollars. This would undoubtedly cause irreparable damage to the trust our users have placed in the protocol. Thus, the first task in developing the v2 upgrade was to create a unit test that verifies that the original storage slots are retained.
The table above describes how the various state variables are laid out in the storage for USDC v1. To read the storage at a specific slot position, you can use web3.eth.getStorageAt (web3.js) or provider.getStorageAt (ethers), which returns the content of the a storage slot in hexadecimal format, with the preceding zeros stripped.
Multiple adjacent state variables that are smaller than 32 bytes can share a single storage slot, starting from the lower-order bytes (right-aligned). For example, the storage slots 1 and 8 in USDC contain both an address and a boolean value.
Strings that are at most 31 bytes long are encoded in the storage slot with the text stored in the higher-order bytes (left-aligned) and its length × 2 stored in the lower-order byte.
Reading a mapping from the contract storage is a little tricky. Since a mapping does not have a predefined size, the slot position for each value in the mapping is calculated by performing a Keccak-256 hash of the key (k) concatenated with the main storage slot position (p) of the mapping (keccak256(k . p)). The main storage slot is left blank and does not hold any data.
I highly recommend all upgradeable smart contract projects to include a storage slot test, as it offers the developer the confidence to make large changes to the codebase without the risk of causing accidental data loss. The full source code for the USDC’s storage slot test can be found here.
Testing in “prod”
Unit tests are helpful in catching potential errors in the code and USDC v2 boasts a 100% test coverage. However, unit tests do not fully replicate the production environment, and manual testing is still valuable. Fortunately, it is very easy to spin up a local fork of the Ethereum Mainnet using Ganache. By specifying one or more --unlock arguments, you can also make transactions from accounts for which you don’t possess the private keys. In other words, I am able to perform the USDC upgrade in this simulated mainnet without actually having access to the administrator key for the USDC proxy contract, which is kept in cold storage.
Another major benefit is that this allows you to test the interoperability and compatibility of your contract with other applications that are deployed on the mainnet. By configuring MetaMask to talk to your local fork, you can even test using the frontends of applications like Uniswap.
Upgrader Contract
In spite of all the testing that had been done, our confidence level about the upgrade was still not at 100%. USDC market capitalization had grown to $1.4 billion at the time of the upgrade and our careers were on the line — no one wanted to go down in history as the developer that set a billion dollars on fire.
If issues are found after an upgrade, it is technically possible to roll back simply by setting the implementation contract back to the original address. However, any downtime incurred by a botched upgrade can potentially cause serious financial damage to the users, and being able to recover is also predicated on the assumption that the failed upgrade did not mangle the contract state.
The solution to our worries was, of course, more code: an upgrader contract. The upgrader contract upgrades the USDC contract, initializes it, runs various tests to ensure everything is working as expected, and self-destructs itself when everything is OK. This is all done in a single atomic transaction, and if issues are detected, it rolls back the entire upgrade process as if nothing had happened. In other words, there is zero down time regardless of the outcome of the upgrade.
Transaction Confirmed
At 8:30 in the morning on August 27th, it was finally time for the engineers from Coinbase and Circle gathered in a virtual war room to face the moment of truth. The necessary contracts were deployed the day before, and all that was remaining was to flip the switch by calling the upgrade() function. A transaction was created and verified once and once more by everyone in the war room before it was sent into the Ether.
The end of the saga was somewhat anticlimactic: a green checkmark in Etherscan appeared in just seconds after the transaction was submitted and the USDC smart contract carried on steadily and faithfully. The upgrade was complete and funds were safe.
The main takeaway from this is that a small group of engineers could upgrade a billion dollar global financial service securely with zero downtime. This was never before possible with the legacy financial system, and it is a perfect example of how powerful this new technology really is.
Acknowledgements: The author would like to thank the following people for their feedback: Olivia Thet, Mike Cohen, and Dan Bravender.
If building the financial system of the future using this new technology sounds exciting to you, Coinbase is hiring.
This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.
Unless otherwise noted, all images provided herein are by Coinbase.
USDC v2: Upgrading a multi-billion dollar ERC-20 token was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
OhNoCoinbase via https://www.ohnocrypto.com/ @Coinbase, @Khareem Sudlow