Fallback oracle vulnerability fixed

On 17/09/2022 we received a bug report from @paco0x (https://twitter.com/paco0x), detailing a vulnerability in Liquity’s TellorCaller.sol contract - specifically, that Liquity uses its fallback oracle Tellor in an unsafe way, allowing an attacker to make Liquity see a fake ETH-USD price from Tellor at relatively low attack cost.

The Tellor team has now implemented a custom fix which makes Liquity’s usage of Tellor safe again. The bug was due to Liquity’s incorrect usage of Tellor, and as far as we know Tellor remained secure since its launch. No user funds were at risk prior to the fix since Liquity was purely using price data from its primary oracle, Chainlink.

‍Background and details of the vulnerability

‍Liquity needs a current and accurate ETH-USD price to determine the value of ETH in user operations. The system pulls the ETH-USD prices from two oracles - a primary (Chainlink) and a fallback (Tellor). As long as the primary oracle is functioning correctly, Liquity only uses the price from the primary oracle for all operations.

However, if the primary oracle ever fails or freezes, Liquity promptly switches to using price data from the fallback oracle, Tellor.

How Tellor Works

‍Tellor operates via proof of stake: price reporters stake some of Tellor’s native TRB token on their price reports for a given query (e.g. ETH-USD). Tellor records all reported prices, and they can be fetched by external systems such as Liquity.

Anyone can also dispute a price report by paying a small fee in TRB. When an ETH-USD price report is disputed, it is removed from the list of prices that Liquity can see. TRB holders then vote on the validity of a price, and determine whether or not the staker should be slashed.

Tellor is designed such that dApps which consume their price data can select how far back in time they want to look for price updates. This allows a dApp team to effectively choose the length of the “dispute period” for their price data. As soon as a price is disputed (regardless of the outcome of that dispute), it is removed from the list of prices the dApp can see.

Since disputing fake price data is mostly profitable, then as long as fake prices get disputed within the dispute period, then only correct ETH-USD prices will be seen by Liquity.

‍Liquity’s original Tellor usage

‍Liquity was deployed with no dispute period - it used Tellor’s most current ETH-USD price. This was an implementation error. If Liquity had ever switched to using Tellor for price data, then with no dispute period, an attacker could submit a fake ETH-USD price and have Liquity instantly consume it. This would obviously be disastrous for the system.

Was this ever exploited?

‍No, not on mainnet. However, it appears to have been exploited on the ETHW fork shortly after the merge. The ETHW fork at the time of the merge contained a clone of all state on Ethereum with instances of all dApps, Liquity included.

Since Chainlink do not push price updates on this chain, The Liquity instance on ETHW asserted that Chainlink was frozen, the fallback logic kicked in, and the Liquity instance switched to using Tellor for the ETHW-USD price data.

Someone then managed to stake TRB and report a fake ETHW price, and minted trillions of LUSD.

As a team, we made it clear pre-merge that the only “real” instance of Liquity is the mainnet version, since there was broad expectation of chaos/dysfunction/collapse on the ETHW fork - much of which came to pass. Still, witnessing Liquity’s incorrect usage of Tellor be exploited on the ETHW chain as well was jarring to say the least.

Were funds at risk on Mainnet?

‍No. Since Liquity only switches to using the Tellor price if Chainlink’s ETH-USD price feed fails or freezes, funds were not at risk. Chainlink’s ETH-USD price feed has functioned flawlessly since Liquity’s launch with accurate and frequent price updates, and Liquity on mainnet has never switched to using Tellor for price data.

As mentioned in our community announcement Liquity was essentially relying solely on Chainlink until the Tellor fix was implemented. Now that the fix is in place, Liquity has a secure fallback oracle contingency as originally intended.

How did this happen?

‍It boils down to an implementation error on Liquity’s part. Liquity’s fallback oracle logic is complex, and despite the strong focus on testing, security and ensuring all branches of the oracle logic were correct, this error slipped past Liquity’s own engineering team and external security auditors.

Nature of the fix

‍Since Liquity’s code is completely immutable, it is not possible to alter how the system interacts with Tellor.

Fortunately, the bug was discovered before Tellor made their final upgrade to “Tellor360”, their finalized and immutable system on mainnet. When the bug was found, Tellor still had upgrade capabilities. The Tellor team were extremely responsive and helpful. They were able to incorporate a custom fix for this issue in their final upgrade.

The Tellor team suggested the following fix: when the Liquity system requests the ETH price from Tellor, Tellor will return the latest price that was submitted at least 15 minutes ago. Since disputed prices are removed from the price list, this gives a period of at least 15 minutes after a price is submitted for someone to dispute it.

Disputing is permissionless and low-cost, and price monitoring and disputing can be automated. This fix makes it extremely hard for an attacker to make Liquity consume a fake price from Tellor: any fake price will be disputed and removed, and Liquity will only see undisputed prices.

The fix has been audited by Coinspect, and no major issues were found.

With this fix in place, Liquity now uses Tellor in a safe way as originally intended.

How was the delay period of 15 minutes chosen?

In terms of disputing fake prices on Tellor, any delay over a few minutes is sufficient - a responsive bot can dispute a fake price in its subsequent block, thus removing it from the list of Tellor price updates that Liquity sees.

A longer delay gives disputers more of a time buffer to dispute a fake price report.

However, for a Liquity system that has fallen back to using the Tellor ETH-USD price, a shorter delay is better: a shorter delay leads to a smaller differential between the price the system sees and the real market price. Keeping this differential small is important for ensuring profitability of liquidations and minimizing excessive LUSD redemptions.

The delay period of 15 minutes was based on careful analysis of the impact of a delayed price on a Liquity system. We used historical ETH price data and looked at the impact of different delay lengths. 15 minutes was chosen as a sweet spot that gives plenty of time for disputers to respond to fake prices, while keeping any adverse impacts on Liquity to a minimum.

‍How will disputing happen?

‍Disputing fake prices is permissionless and mostly profitable, so there is an economic incentive to dispute a fake price. Anyone with an interest in the correct functioning of Liquity can also monitor price reports and submit disputes. The process is easy to automate and it only takes one dispute per fake price to remove it.

Technical details of the fix

‍Prior to the fix, Liquity’s TellorCaller.sol contract (deployed here: https://etherscan.io/address/0xad430500ecda11e38c9bcb08a702274b94641112) gets the latest Tellor price from Tellor with the following code:

Liquity calls Tellor twice. The _requestId identifies the ETH-USD price query that Liquity needs. The returned _count gives the index of the latest reported price, and the returned _time is the timestamp of the latest reported price. The code then retrieves _value, the actual latest price value.

This code is part of the core Liquity system and is immutable.

As part of Tellor’s 360 update, they deployed new contracts which incorporate a fix for Liquity. The relevant code is here in Tellor’s Tellor360.sol contract (deployed here: https://etherscan.io/address/0xd3b9a1dcabd16c482785fd4265cb4580b84cded7)

The function fetches the timestamp of the price at the requested position inside the try{...} block.

Here’s the custom fix: when the _requestId is 1 (i.e. when it is the ID of the ETH-USD price) and the value is less than 15 minutes old, it actually searches again for the most current value that is at least 15 minutes old with getDataBefore.

Since Liquity also calls Tellor to retrieve the price value, Tellor also put a custom fix in the retrieveData function:

After Tellor’s upgrade, the legacy ID of 1 now triggers the catch{...} block. Within it we see that when the _requestId is 1, Tellor returns the latest price that was at least 15 minutes old with getDataBefore.

‍Conclusion

‍We’re extremely grateful to the Tellor team for incorporating a custom fix into their final system upgrade at short notice, and thus making Liquity’s usage of Tellor safe as originally intended. We’re also grateful to @paco0x for his clear bug report, and we’ve awarded him a bug bounty in line with the importance of the issue.