On Managing Secure Upgradability
How our security systems are prepared to protect in a constantly evolving protocol
Upgradability, the Conduit for Innovation
At this stage of early growth, zkSync is a rapidly evolving protocol, which requires the zkSync protocol to remain upgradable. If we were to give up upgradability, it would require all users and developers to withdraw and deposit to migrate into the new version every six months. Mastering the powers of upgradability is an ongoing focus for us (we’ve already written two articles on it), but we are happy to announce a few more changes in how we are managing upgradability at Matter Labs.
While our previous articles on upgradability focus on finding the balance between trust-minimization and reactivity to bugs with timelocks and an external check and balance by a security council, this article will focus on an upgraded security approach in dealing with the subsequent bugs upgrades may introduce.
The Illusion of Security
System security is a function of time. But upgrades distort that perception of time. A project that has been battle-tested on mainnet for a long time is widely considered to be secure, but every upgrade introduces additional surface area for an attack that can put the entire system at risk. And while the first version had time to slowly accrue in total value locked as users of decreasing security tolerances joined overtime, an upgrade jumps straight into the deep end, at a high TVL with a high expectation of security by its users.
On July 26, 2021, a vulnerability in our smart contracts that could have resulted in frozen funds was reported to the Matter Labs team. The fix was deployed on July 27, and all funds are safe. At the time of the report, zkSync had $8.5m USD in total value locked, and was the only severe vulnerability in the zkSync protocol since genesis to date. Before publishing any details, we conducted a thorough internal review of our security approach and processes, introducing a number of systematic improvements not only to fix the root cause, but also to make our entire system more robust in bug prevention and reaction, which is explored in depth in the next sections.
The vulnerability was introduced in zkSync’s Version 5.1 upgrade on July 1, and at the time, although the fix was finished and deployed the next day, the timelock could only be shortened to the 3-day minimum after receiving 12/15 signatures by members of our security council. Similarly, Compound, who launched v2 in May 2019, had a $80m bug introduced in a governance proposal accepted two years later, on September 29, 2021. They too, had a 2-day minimum timelock for governance upgrades, and because the fix was not effective immediately, they incurred an additional $67m in damage for a total of a $147m bug.
The resemblance of the two is uncanny, and not a coincidence. Both should serve as a cautionary tale for the large majority of protocols on Ethereum that remain upgradable. With each upgrade, we need to reset our internal security clocks and treat them as a completely new deployment of the protocol.
Here are a few changes we’ve introduced to improve our system security against upgradability:
Removing the Timelock in Security Council 2.0
Upgrades to zkSync’s smart contracts can only be initiated by Matter Labs and are subject to a 1-month timelock before it is live in production. And while such timelocks protect users from the Matter Labs team upgrading zkSync and conducting a rug-pull, it also restricts our ability to fix bugs quickly. After an internal reassessment, we decided that while zkSync is still rapidly upgrading, the probability of bugs is significantly higher than malicious collusion between both the Matter Labs team and 9/15 members of the security council, who are all well-known members of the Ethereum community.
On November 12, we announced Security Council 2.0, which switched from the 3-day minimum timelock to instant upgradability with 9/15 signatures from the security council. It was activated on December 17, 2021.
Bug Bounty Incentive Rework
Next, we are happy to announce that we are partnering with ImmuneFi to improve our bug bounty system. The bug bounty will go live with our incoming upgrade!
Conclusion
With another upgrade to zkSync 1.x coming soon, and zkSync 2.0 on the horizon, we are constantly updating our internal processes for mitigating the risk that comes with upgrades. As upgradability is baked into most protocols, including many Layer 2 solutions, let’s work together in moving forward more securely as an ecosystem!
