DSS

DSS 2026 / Day 2 / Pearl Hall

RPC Trust Failures: A Checklist for Builders and Monitoring Tools

Jork Leonhardt

13:59 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:01So hi, good morning.

0:04I hope you can all hear me.

0:12Okay, so yeah,

0:14as I've been introduced, my name is Jörg

0:16Lenhardt. I'm from Corpus Core.

0:19You might also know us from the Colibri

0:21client.

0:23And in the next 15 minutes, I want to

0:25give you two checklists

0:28that you can take back to your team and

0:30hopefully discuss it with them.

0:34So you might remember

0:36just this year in April,

0:39um an attacker took

0:41116,500

0:44RS ETH to so

0:46that's roughly $292 million

0:50from Kelp DAO's Layer Zero bridge.

0:54So there's some nodding. You remember

0:55that.

0:57I know it's not as much money as uh

1:00all of the hacks from last week, but

1:02still it's an important one

1:04because the contracts were audited. So

1:07Henri just said they're also going

1:09through an audit and this is something

1:10that we all do.

1:12But their contracts were audited and uh

1:15every postmortem since then agrees that

1:17the contracts did exactly what they were

1:19supposed to do.

1:22The bridge released the funds because

1:23the verifier signed a message

1:25saying the tokens had been burned on

1:27Unichain.

1:29However, they had not really been

1:31burned.

1:33The verifier believed it because

1:36the RPC nodes said

1:38that

1:39this was the case.

1:42One fact of fairness, um the bridge

1:44executive is a single verifier and Kelp

1:46and Layer Zero are currently in court

1:49uh discussing who's responsible for that

1:50setup. So I'm not going to take any

1:53sides here. This talk is just about the

1:56facts and where the verifier got his got

1:58his data from.

2:00So quick show of hands, who of you

2:02verifies what their RPC returns?

2:06I expected no hands.

2:09Let me ask a different question. Who's

2:10actually listening?

2:13Okay, there are some hands now. That's

2:14good. That's good.

2:16So all of you who are listening

2:18does not verify. You just trust your

2:20RPC.

2:22And that is usually the case. That's

2:23what we do. But from the KELP DAO um

2:27hack, we saw that

2:29your RPC can lie to you.

2:31So almost every component in DeFi

2:34read chain change state through JSON RPC

2:37endpoints. So that's the front ends that

2:40we use, wallets, keepers, liquidation

2:42bots,

2:43oracles, bridge verifiers, and so on.

2:46Most of the times that endpoint is run

2:48by a third party.

2:50So we audit our contracts line by line

2:52and then we let an unverified data pipe

2:54decide what those contracts get to see.

2:58KELP DAO is just a loud example, but

3:00it's one of four different failure

3:02failure modes that we uh see keep coming

3:05back again and again and again.

3:08So here's the four failure modes that I

3:11want to talk about today.

3:13So availability first.

3:15On November 11th, 2020,

3:18parts of Infura

3:20were still running older Geth versions

3:22with a consensus bug and they ended up

3:24as a minority chain.

3:26And Infura's mainnet API was down for

3:29about 6 and 1/2 hours.

3:31MetaMask broke for many users and

3:33Binance, Bitfinex, and other exchanges

3:35paused ETH

3:37and ERC 20 withdrawals.

3:39If we look closely, this was also a

3:41trust problem.

3:43The provider served a different version

3:45of the chain and downstream systems

3:47could not could not tell the difference.

3:50Then the next is censorship.

3:53Uh in March 22, Infura changed its

3:55configuration to follow new sanction

3:58directives and set the blocks more

3:59broadly

4:01than they intended to do.

4:03So users in Venezuela lost access to

4:05metamask for several hours.

4:08Users in sanctioned sanctioned regions

4:10like Iran

4:12state blocked by design. That was the

4:14original intent.

4:16So whatever you think of those policies,

4:18your users access depends on one

4:21provider's compliance settings.

4:23That's something we always need to keep

4:24in mind.

4:26Now the next one's privacy.

4:28In November 22

4:29a consensus privacy policy update

4:32disclosed that Infura

4:34and that's the default RPC from metamask

4:36collects users IP addresses and wallet

4:39addresses when they send transactions.

4:42After a lot of backlash, Consensys

4:44clarified that it stores neither read

4:46requests and keeps those two those those

4:49two data points separately

4:51and would narrow retention for seven

4:53days.

4:54The structural point, however, stays.

4:56Whoever broadcasts your users

4:58transaction sees where they come from.

5:01And then the last one, Keltou

5:04um I'm going to talk more in detail

5:06about that one.

5:08So from

5:10the Keltou incident report

5:13um

5:15here's something that matters for

5:16everyone in this room. First

5:18the malicious code software returned

5:20forged data only to the verifier and

5:23accurate data to everyone else in the

5:24system.

5:26From the outside

5:27the nodes looked healthy. Everything was

5:29fine.

5:31And second, the attacker knocked out the

5:33uncompromised external RPC providers

5:36with a DDoS.

5:37So the failover logic itself sent every

5:39request to the poisoned nodes.

5:42The redundancy was there originally and

5:44then it was used against Keltou.

5:52This is the point I'm coming to.

5:58So here's a checklist.

6:02I just let you read it.

6:04I don't need to read for you. I expect

6:06you can do it yourself.

6:08But these are the four questions that

6:11you should answer

6:12with your own stack. Probably not this

6:15week since you're all going to be busy

6:16here, but whenever you return home and

6:19discuss with your team what you've

6:20learned this week.

6:25Now let's go through these questions one

6:26by one.

6:28So the first question,

6:30for each read

6:32that can move value,

6:34how many independent sources stay behind

6:37it?

6:38If it's three endpoints with the same

6:40provider or on the same cloud,

6:43they just count as one. So in Keldau's

6:46instance, the two poisoned nodes ran on

6:48separate clusters,

6:50but both set in the same company's cloud

6:52environment and one set of stolen

6:54developer session keys reached both.

6:57So that's only

6:59one source.

7:03You can take pictures if you like or in

7:05the end either QR code but you can just

7:07download the whole list. So you better

7:09listen carefully and then you get it.

7:12Um

7:14now the second question.

7:16That's an uncomfortable one.

7:19So is any response actually verified or

7:22is it just trusted?

7:24If your keeper or your verifies verifier

7:28takes the result of an ETH call at face

7:30value,

7:31then whoever runs that node controls

7:33your system.

7:35Ethereum gives us the tools to check

7:37state proof via ETH get proof, receipts

7:40and headers that can be verified against

7:42the sync committee,

7:43but most stacks

7:45do not use them.

7:51Third question. What does the system do

7:54when two sources disagree?

7:57In 2020, Binance took two block block

8:00explorers storing two different chains

8:02and passed withdrawals. That was a smart

8:04move

8:05and resumed once the picture was clear.

8:09So they already had some security in

8:10case, but yeah, we know that we've seen

8:13last week that that's not all.

8:15So in KelpDAO, the verifier never saw

8:17disagreement because the attacker took

8:19every source that could have disagreed

8:21offline.

8:22So the question has a second half. What

8:24happens

8:26if sources disappear?

8:28If losing three endpoints means

8:30you quite don't trust the fourth,

8:32an attacker only needs to DDoS the other

8:34ones.

8:36So you should define a quorum and treat

8:39losing it as an incident.

8:44Fourth question,

8:46concentration.

8:49Write down who serves the reads,

8:52where those nodes hosted,

8:54and which legal regime they answer to.

8:57Include the defaults in the wallets your

8:59users bring. The Venezuela case shows

9:01that one compliance setting at one

9:03provider can switch off a whole country.

9:06The privacy case shows that the provider

9:08that broadcasts the user's transaction

9:10also sees where they come from.

9:13Neither is a bug in the code

9:15and both are a user's problem.

9:17If changing providers needs a redeploy,

9:21the fallback that you intend exists only

9:24on paper.

9:28Now it's not just those four questions.

9:30There's also

9:32something for a different part of

9:34audience. There might also be some

9:35monitoring and tooling providers in

9:37here. So there's two gaps

9:39and they also both show up in the Kelp

9:42timeline.

9:44So the first gap,

9:46Chain analysis described the KelpDAO

9:48exploit as indistinguishable from normal

9:51bridge activity at the transaction

9:52level. There was a valid signature,

9:55valid message format, the correct

9:58contract behavior.

9:59So at the transaction level, the anomaly

10:01detection had nothing to catch.

10:04And remember the timeline, the poisoned

10:06nodes gave correct answers to everyone

10:09except the verifier.

10:10So the gap in response integrity is a

10:13useful tool that would record a data

10:15component, acted on, and compared it

10:18with state verified independently from a

10:20vantage point the attacker cannot reach.

10:22That's the fallback we should use. So,

10:24cross-chain invariant checks,

10:27burns against releases, that's a good

10:28start.

10:29And they still fire after the um after

10:32the fact.

10:34And the second gap

10:36is visibility.

10:39There we go.

10:41So, you should ask your protocol team

10:43what share of a reads goes through a

10:44single provider.

10:46And most can probably not answer this.

10:49No dashboard shows it.

10:51A monitoring product

10:53actually could.

10:54So, an an inventory of your RPC

10:56dependencies per component

10:59that should alert you when a failover

11:00happens or a quorum is lost.

11:02And you should have a link between the

11:04DDoS on the provider and every protocol

11:06that depends on that provider.

11:08So, in the KELP tower instant, the DDoS

11:10of the external RPC providers began

11:13um on that day that was April 18th at

11:165:20 p.m.

11:18And according to Misari's timeline,

11:20roughly 50 minutes

11:22um after

11:23the forged message. So, there was a time

11:26window of 15 minutes.

11:28And if anyone had been been watching the

11:30quorum or what the other providers would

11:33have done, they could have stopped

11:35everything.

11:36And then the KELP tower attack would

11:37have failed.

11:42So, what we do what do we do to route

11:43it?

11:45There's two directions and they both

11:47work best together.

11:49Redundancy done properly with

11:50independent operators and a quorum

11:53covers availability and most censorship

11:55cases.

11:57It does not solve trust though.

11:59If every source can be poisoned and

12:01knocked out, more sources only

12:03only raise the cost of an attack.

12:06The second direction is verification.

12:08A light client checks each response

12:10against Merkel proof and against block

12:12headers signed by Ethereum Sync

12:14Committee.

12:15Then a note that is lying can only

12:17produce invalid proofs and your system

12:20rejects it.

12:21So our team built Kolibri as I said.

12:24That's one implementation of this

12:26approach

12:27and it's small enough for wallets,

12:29backends

12:30and embedded devices.

12:32So if you have like a very small chip,

12:34it probably runs on that as well.

12:37So you can use it as a reference. You

12:38could also build your own.

12:40What matters is that verification

12:42happens where the data is consumed, not

12:44anywhere in the cloud, but locally on

12:46your device or within your DApp.

12:51So three things that you should take

12:52home.

12:54And please go ahead and scan that QR

12:55code then it you can also download it as

12:57as a PDF.

13:00So first audited code running on

13:02unverified data

13:04in the end is an un audited system.

13:09Second, take the four questions home to

13:11your team this week or maybe next week

13:13when you are back home then and

13:15do a full checklist.

13:17Um

13:18yeah, scan the QR code then you have

13:20those four questions.

13:21And last if you build monitoring, the

13:24next blind

13:25blind spot is between the chain and the

13:27protocol in the data layer.

13:30So please scan that and if you like

13:32there's a little blog post that we wrote

13:35attached also as the checklist of the

13:37questions that you should ask yourself.

13:40And uh on our site you then also um

13:43find our GitHub link and there you can

13:45download Kolibri and also check that out

13:47yourself. It's open source of course.

13:52Okay, thank you for your time.

13:55And now I'm heading over to Ofer.

Source and accuracy details

Source export SHA256: 17c70e5eb562f66420c1d691ba8415e59fa4b6e0c2b2b2d2c945db3065be4f54