Arbitrum 2023 Downtime. Can It Happen Again? What Else Can Go Wrong?

Do repost and rate:

It’s important to note that as promising as rollup technology is, it’s still a new technology not without risk. Arbitrum One experienced downtime for ~45 minutes in September 2021 when a bug caused a large burst of transactions to overload the system. Optimism (O?) also experienced a temporary outage (~one hour) in November 2021 in which its L2 transactions were halted. 

Beyond downtime, Arbitrum (and most rollups) remained very centralized around the core team/company. Control over the core code (and thus, users’ funds) in Arbitrum is maintained with a 4/6 multisig design. This means only four people are needed to collude/be compromised in order to have complete control over the rollup. The multi-sig address is seen 

“The admin of all contracts in the system, capable of issuing upgrades without notice and delay. This allows it to censor transactions, upgrade bridge implementation, potentially gain access to all funds stored in a bridge, and change the sequencer or any other system component (unlimited upgrade power). It is also the admin of the special purpose smart contracts used by validators.” -

Additionally, rollups currently use sequencers to collect and order users' transactions in a mempool before they get executed and posted to the DA layer. This poses a potential issue regarding the Maximum Extractable Value (MEV) as the sequencer is generally centralized (one company) in current rollup implementations.  is built by Offchain Labs, which currently operates Arbitrum's sequencer and illustrates the long road ahead many rollups have towards achieving true decentralization. The current solution would be to decentralize the sequencers, which many rollups plan to do, although this presents its own challenges. 

In the event the L2 node is offline, users must be able to withdraw funds from the bridge. Up until recently, becoming a validator for Arbitrum has not been completely permissionless. Arbitrum implemented a whitelist. As previously discussed, validators send new L2 state roots to Ethereum mainnet.  Due to the existence of a whitelist, if all validators were to fail, nobody could publish L2 state roots, and users' funds would subsequently be frozen. But as of Q3 2022, the whitelist has been effectively removed with a new process in place should a validator go offline. 

December 2023 Downtime

On December 15th, the Arbitrum One network experienced a significant disruption due to an outage in the Arbitrum One Sequencer. This incident resulted in irregularities in network traffic, causing downstream issues, particularly in gas pricing. The Offchain Labs team, acting on behalf of the Arbitrum Foundation, spearheaded the investigation and resolution of the issue. What follows is a comprehensive account of the event, inclusive of a root cause analysis, the learnings acquired, and the preventive measures enacted to mitigate future recurrences.

In the early hours of the specified date, the batch poster component of the Arbitrum One network, responsible for posting transaction data to Ethereum, began to accumulate a backlog. This was primarily due to complications with an Ethereum consensus client that the sequencer relied upon, coupled with the manner in which the backlog was being managed. The situation was further compounded by a high volume of inscriptions being minted at the time. As the backlog burgeoned, the feed component of the sequencer malfunctioned, resulting in a severed connection for both 3rd party node providers and the public RPC fleet. This led to users encountering out-of-sync chain data and subsequent transaction failures.

In response, the team expedited the deployment of a developmental version of the node software, which was already under testing, onto Arbitrum Sepolia. Following successful internal trials, this updated build was deployed to the Arbitrum One sequencer, promptly restoring its functionality.

Concurrently, the incident prompted an inadvertent issue with the onchain pricing system. Typically, this system adjusts the L1 data price automatically and trustlessly. However, due to the backlog's impact on timely data availability, the system was inadvertently undercharging users at a high volume. Once the sequencer was restarted and the backlog cleared, it became evident that there was a significant deficit in fees collected versus the amount spent posting data to L1. To rectify this, the pricing mechanism increased the gas fees, inadvertently rendering transactions prohibitively expensive for most users.

Acknowledging the impact on users and the network, the Arbitrum Foundation took decisive actions to correct the pricing deficit. This involved allocating funds and employing zero-value transactions directed to a burn address to neutralize the deficit. Subsequently, the pricing mechanism was recalibrated, stabilizing transaction costs and resuming regular network activity.

This incident underscores the complexities inherent in maintaining robust blockchain networks and the importance of agile and effective incident management. The Arbitrum Foundation and Offchain Labs are committed to continuous improvement and have implemented measures to fortify against similar incidents, ensuring the reliability and efficiency of the Arbitrum network for all users.

In the new system, if validators do not submit new state roots within a week,  can become a validator and submit the transactions. This is a big step in the right direction for decentralizing the rollup chain.

However, other attack vectors still exist. ORs and their challenge period are susceptible to 51% attacks. In this scenario, the attacker would try to introduce “bad” transaction data into the rollup and attempt to censor any attempts to challenge it during the challenge period. The attacker is ultimately trying to corrupt the state of the rollup (with fraudulent data for their own self-interest) and stop anyone from challenging the submission.

This is why an adequately lengthy withdrawal/challenge period (one to two weeks) is needed. An attacker may be able to censor or sneak a transaction through if the window was short enough, but the longer the window, the harder it is to fool the rest of the chain. 

To withdraw from Arbitrum, you first submit the withdrawal transaction on the rollup. Once the transaction is finalized on L1 (up to ~seven days), your funds are free to claim with another L1 transaction (requiring a Merkle proof).

Arbitrum also uses fraud proofs, which, as discussed before, refer to the concept of anyone (with a vested interest) being able to track, backtest, and dispute a transaction’s claim within a challenge window. Arbitrum uses interactive (multi-round) fraud proofs.

Below, you’ll read about different types of fraud proofs and the advantages of interactive (multi-round) fraud proofs, based on which type of fraud proof Offchain Labs (builders of Arbitrum) decided to use.

Two Types of Fraud proofs:

1. Non-interactive (single round) - Optimism

Whenever someone submits a challenge regarding the state of the rollup. The dispute can be solved without requiring other parties' involvement. Re-execution of disputed transactions takes place on layer 1 (Ethereum) using a verifier contract on layer 1. After the re-execution of transactions, if it was found that the challenger’s claim about the correct rollup state was right, the operator who sent the invalid rollup block, its stake, will be slashed. In contrast, the challenger will be rewarded with tokens.

The biggest disadvantage of using non-interactive fraud proofs is that the rollup protocol can’t execute transactions on their rollup, which exceed the Ethereum gas limit, and large transactions can’t be verified in case of dispute on layer 1. Rollups are also required to publish state commitments for individual transactions, increasing the data posted on-chain. It’s not efficient for rollups, and layer 2 users need to pay more fees due to this method being less cost-efficient. 

2. Interactive (Multi round) - Arbitrum

Whenever someone submits a challenge over the state of rollup. The dispute requires two or more parties to resolve. The asserter (the party making the assertion) and the challenger participate in the dispute resolution process, overseen by a layer 1 verifier contract, which ultimately identifies which party is lying.

To resolve the dispute, the challenger requests the asserter to divide the assertion into two assertions. This dividing process continues until both parties reach an assertion representing a small enough operation that can be executed on Ethereum. Now, the layer 1 verifier contract will resolve the dispute by executing the one instruction on layer 1. The result will then determine which party was lying. 

If the challenger's claim is right, the assertor stake will be slashed and part of the slashed stake will be awarded to the challenger while the remaining part is burned. This is done to prevent bogus challenges from happening, which will delay withdrawals of assets from layer 2 to layer 1.

There are various advantages of Interactive fraud proofs, which are absent in non-interactive fraud proofs. Interactive fraud proofs minimize the work layer 1 needs to do to resolve the dispute. As most of the work is done by both parties off-chain, layer 1 needs to execute one small instruction.

A rollup using interactive fraud proofs can also execute rollup transactions that exceed the Ethereum gas limits. There’s also no limit on contract size as it doesn’t require contracts to fit within Ethereum’s contract size limit. It also allows more flexibility in implementation. For example, a rollup can add instructions that don’t exist in the EVM.  

Arbitrum, however, does multiple rounds, only stopping for a disagreement disputed on the L1. The disagreement is examined and deconstructed until the precise cause of the conflict is identified. The sequencer and verifier perform the dissection and only the stage at which the dispute is narrowed down will be computed on the L1 to establish the proper end state.

Regulation and Society adoption

Ждем новостей

Нет новых страниц

Следующая новость