DSS

DSS 2026 / Day 1 / Amber Hall

AI-FV: Slop or Prop? Explorations in Agentic Formal Verification

Jonathan Mevs

25:57 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:00Um, well, hello everybody. Uh, thanks

0:03DeFi Security Summit for having me here.

0:05I'm John.

0:06I'm a senior engineer with Quantstamp,

0:08and the topic I'll be talking about

0:09today is like formal verification and

0:12how far can AI support such uh and some

0:16experiments we did uh surrounding a

0:17couple different tools.

0:21Click.

0:24Okay, great. So these are the guiding

0:26questions that inspired the talk and

0:27like what we'll be focusing on today.

0:30And we wanted to see like to what extent

0:32does AI make existing formal

0:34verification tools accessible to

0:36experienced smart contract auditors?

0:39And to do such, we like ran some

0:40experiments on three prominent tools,

0:43like AutoProver by Certora, Control by

0:46Runtime Verification, and the SMT

0:48Checker by the Solidity team.

0:50And then ultimately we wanted to

0:52conclude where does formal verification

0:54fit in the audit cycle and like some

0:57takeaways that we can have as smart

0:58contract auditors.

1:01Next please.

1:04So this was sort of like the starting

1:06idea, right? Like over the past year um

1:09with like LLMs and like agents getting

1:12really really good,

1:13we've seen like non-technical people be

1:15able to produce like some legitimate

1:17technical output. So this is like your

1:20typical business guy like sitting down

1:22with Claude and being like, you know,

1:24build me 100 million ARR SaaS, make no

1:27mistakes, and actually getting a website

1:30output. So there's like um

1:33this non-technical person doesn't really

1:35need to understand anything about

1:38websites in order to like actually get a

1:40website

1:41um out there and sort of running with

1:44it.

1:45So next slide please.

1:47So then uh we wanted to take that step

1:49further and see like with us as like

1:52pretty technical people, bug hunters,

1:54reading a bunch of Solidity code, um you

1:56know, breaking apart systems, stuff like

1:58that, could we make a similar leap to

2:01the extremely technical realm of formal

2:03verification? So oftentimes people that

2:06are like experts in formal verification

2:07may be studying mathematics at like the

2:10PhD level or

2:13having like many years of this like

2:15targeted study.

2:17So if

2:19we spent like years doing smart

2:22contracts, can AI help us get that leap

2:26similar to like the BD guy to the

2:27website, the smart contract auditor to

2:29the formally verify expert?

2:33So if Thread Guy can do it, maybe I

2:35could do it, right?

2:37Now that's just a meme checkpoint.

2:39Okay, so a slight primer on formal

2:42verification. So let's say we have a

2:45spec like a deposit never decreases

2:47total shares. So if you were to write a

2:49unit test for such, you would just like

2:51do something like deposit 100 and make

2:54sure that the assets you got back

2:57was greater than or equal to 100.

2:59If you were to fuzz that spec, you would

3:01probably

3:02you would run it through a fuzzer and

3:04then it would test a deposit of 100,

3:06test a deposit of 1,000, test a deposit

3:08of

3:091,001, so on and so forth, thousands of

3:11random deposits. But with formal

3:13verification, you want to mathematically

3:17prove that for every deposit from any

3:20state

3:21the shares will never decrease.

3:25So now let's take a look at to what

3:27extent did these tools actually make

3:31formal verification accessible to

3:33experienced smart contract auditors.

3:37So this sort of sets the stage for how

3:39we went about this experiment, like just

3:41sitting down at the computer, hey

3:43Claude, formally verify this

3:45believe in yourself, make no mistakes,

3:47right?

3:49So in order to do so, we needed an ideal

3:51codebase. So this was one that was like

3:54simple for the point of like

3:56experimentation, so if and when we

3:58needed to look under the hood, you could

4:00go read the code. It was like uh had

4:03manageable line of code so that the

4:04prover didn't run forever, which was

4:06actually something that I underestimated

4:08how long those things take. Um and then

4:11we This was really important here is we

4:12needed like a public human formal

4:14verification report that actually had

4:17some real bugs in order for us to catch.

4:20So what we landed on was this Aave vault

4:23that Certora audited both with formal

4:25verification and a manual code review in

4:272023.

4:29It's a simple ERC 4626 vault where

4:32deposits are in yield. The owner can

4:34take a fee and it's just shy of 700

4:36lines of code.

4:38And there were 11 known bugs uh to see

4:40if the formal verification tools

4:42actually caught it.

4:45So now how do we pick the tools? Well

4:47the ideal tool um was tailored

4:49specifically to Solidity smart

4:51contracts. It was open source so we can

4:53use it and download it or um tinker with

4:56it.

4:58It was already like adopted by the

4:59community in some extent um and had

5:01already undergone several client

5:03engagements um and it was thoroughly

5:06documented so that we could basically

5:08build agents uh to use the tools without

5:12uh having the agent to figure too much

5:14out.

5:15So we ran each of these tools on their

5:17own EC2 instances so they had tons of

5:19resources and didn't need to share

5:21anything um and they could sort of just

5:24run the provers on and on and on.

5:27So the first tool that we uh took a look

5:30at was Certora's auto prover which came

5:32out this past summer in July.

5:34Uh this is the only tool that we looked

5:36at that was agentic out of the box. So

5:39Certora has had their prover out for

5:41some time but in July they actually

5:44released this agentic solution.

5:46Um what's unique about this tool is that

5:49it's written in the Certora verification

5:50language uh so that's their own language

5:53where you write the specs. They're

5:55verified at the bytecode level and they

5:56support

5:58uh several different solvers.

6:00And the technique that they approached

6:02to formal verification is verification

6:04condition generation where per rule you

6:07build one formula and ask the solver,

6:09oh, is any input going to break this

6:11rule?

6:14So this is an example of like what the

6:16agent would actually create. Um, this is

6:18the tool or a spec written in uh Certora

6:22verification language,

6:23never decreasing shares.

6:25Um, so this is what would actually get

6:27fed to the solver.

6:29And humans don't usually read this part,

6:31but it's just good to sort of like

6:33get a look at what's actually being fed

6:35into the solver, um which which is this

6:39assert statement

6:40is trying to be proven holding true for

6:42all inputs.

6:52Could you go next?

6:53Thank you. Uh,

6:55the next tool is really interesting.

6:58It's controlled by runtime verification.

7:00Uh, what's cool about this is it runs on

7:02the bytecode in the KVM, which is a

7:05mathematical definition of the Solidity

7:08EVM according to the yellow paper.

7:11So it verifies bytecode.

7:14Uh, it uses the Z3 solver and what this

7:17tool focuses on is symbolic execution,

7:20which is running each test with like

7:23these symbolic inputs at each branch in

7:26the test. You traverse each path and

7:29you're either going to get to an error

7:30that's going to show that the spec

7:33doesn't hold or you're going to get to a

7:35terminating condition and see that the

7:36spec does indeed hold.

7:39Uh, I think this might be out again.

7:42If you could go next. Thank you.

7:44So this is what um you would get from

7:48or what the agent created for running

7:50the control test.

7:51We see that it's extending this control

7:55test library and actually running

7:58the test within the KVM directly.

8:03Next, please.

8:08And

8:09yeah, this an example of what was being

8:11uh fed to the solver.

8:13Um and it's not too important, but you

8:16could see that like there's actually a

8:18different question being asked to the

8:20solver versus what uh Certora asked.

8:24Next, please.

8:27So this one uh SMT Checker is actually

8:29just built directly into the Solidity

8:31compiler. So this tool is actually like

8:33most native to Solidity and probably

8:36like easiest for like a Solidity smart

8:38contract auditor to jump straight into.

8:41Um so this is provided by the Solidity

8:43team. It uses uh constrained Horn

8:45clauses model checking, which is just

8:48asking um a solver for

8:51an invariant that holds for like all

8:53deployments for every call. And then if

8:56that holds, it rules out the bug.

8:58Please go next.

9:02And like I mentioned, this is like most

9:04native to Solidity because it's in the

9:06compiler itself. So there's no extra

9:08libraries. There's no um other languages

9:11that are used to write the spec.

9:14You can just extend a vault,

9:17uh make a call to the contract or the

9:19function that you want to test and add

9:21an assert afterward.

9:23So uh we know that like usually as

9:26Solidity devs, you aren't too keen to

9:28having asserts like within uh the

9:31contracts or the functions themselves.

9:33You usually tend towards require

9:34statements, so you would usually just

9:36extend and put in the assert for the

9:38condition that you're looking for. Um

9:41and then run it through the compiler.

9:43And then yes, it will create uh an

9:46output that you can ask the solver.

9:49Next, please.

9:51So this is just kind of wrapping that

9:53all up. I'll invite you to take a look

9:56um

9:57a diagram like comparing all three of

10:00the tools that we looked at. So all of

10:02them are open source, all of them

10:04generate counterexamples, and all of

10:06them support the Z3 solver, which is

10:08like the main solver being used here.

10:11Um, as I mentioned, Certora's is the

10:13only one that's agentic out of the box

10:15and the only one that has a um

10:18separate language used, the CVL.

10:21Both Control and SMT Checker are written

10:24in Solidity,

10:26and they each have their own uh separate

10:29technique for verification.

10:31Next, please.

10:33Okay, so now for the exciting part, uh

10:36harness development.

10:37Could you go next?

10:39Thank you.

10:40So the idea when we started to write the

10:43harnesses to test the formal

10:44verification tools

10:46was we assumed like, okay, Certora's

10:48AutoProver, that's like sort of the

10:50template, right? Uh that's what an

10:52agentic solution to formal verification

10:54would look like right now.

10:56So we extracted the properties and the

10:59architecture from Certora AutoProver,

11:01and then basically in conjunction with

11:03the documentation on a specific tool,

11:05either Control or SMT Checker,

11:08we fed them um into Claude to actually

11:11write the harness.

11:12And then

11:15Hold on, let me not get uh let me not

11:17lose my train of thought here. So

11:20like with AutoProver, we wanted to

11:22basically like take what they were doing

11:24and see if it could also work with

11:26Control and SMT Checker, um and then

11:28like compare the results of such.

11:31So that wasn't just enough to write the

11:33harness. Each Control and SMT Checker

11:36came with their own like example code

11:38and test cases to work with.

11:40So Control, I think they had uh like a

11:44simple counter contract and a WETH9

11:47contract uh that we could fine-tune the

11:49harness on.

11:51And SMT Checker, I think it was like a

11:53mutex, like a lock

11:55uh contract

11:57and ERC777 that we got to fine-tune on.

12:01So SMT Checker was actually fairly

12:03straightforward with fine-tuning, but

12:06what we found with working with Control,

12:08if you read the Control documentation

12:10for like that formal verification tool

12:12specifically, A lot of it is focused on

12:15um

12:17it's written so that there's a formal

12:19verification human expert like

12:21frequently in the loop. So a lot of the

12:23control documentation talks about

12:25writing a test, okay, it fails, go back

12:28and like tweak it a little bit, add a

12:30lemma, so on and so forth. Um so that

12:33actually didn't do too well with the

12:35agent, which we'll see in a little bit.

12:37But after we basically fine tuned it to

12:40actually solve the example contracts or

12:43example tests that were provided within

12:45the documentation for the tool itself,

12:47it sort of graduated to move on to the

12:49data set.

12:51Next please.

12:52Okay, now the exciting part. So how did

12:54each of these tools actually perform

12:56when driven by state of the art models?

12:59Next please.

13:02Uh so actually one thing on the models

13:04before I talk about the metrics, um for

13:07Sertora's AutoProver,

13:10there's the ability to uh change what

13:13models you were running it on,

13:16but

13:17we so like out of the box it's running

13:19on Opus 5 and Sonnet 5.

13:22And then once I wanted to go tweak the

13:25models uh a little bit,

13:27they have like a high like a high power

13:29model and a low power model. So their

13:31high power was Opus 5 out of the box and

13:33their low power was like Sonnet 5.

13:36When I bumped them up to the newer

13:37generation like 5.5 and Sonnet 5.5, uh

13:40for some reason Sonnet 5.5 like didn't

13:43extract the properties the right way

13:45that Sonnet 5 did. So then I was like,

13:47all right, let's not mess with this,

13:49just test AutoProver out of the out of

13:51the box. So Control and SMT Checker are

13:54checked on Opus 5.5, but AutoProver is

13:57just coming right off the shelf. So

13:59perhaps if I had more time I could tweak

14:01and try to figure out what's going on

14:02with Sonnet 5.5, but for the sake of

14:05experimentation we just let it run.

14:07Uh just a small note before we look at

14:09some of these metrics. So we wanted to

14:11test like recall. So of those 11 bugs

14:14that were in the original report, like

14:16how many did the tool actually catch?

14:18Precision. So like we know with like AI

14:21scanning tools, one of the worst things

14:22you could have as an auditor is like

14:24noisy results that like end up wasting

14:26more of your time. So we wanted to see

14:28like out of everything that the tool

14:30ended on, how many of them actually hit

14:32a known uh hit a known bug.

14:36Soundness is a really interesting one.

14:37Like can the finding actually happen? So

14:40we extended all the provers to actually

14:41ask an agent to write POCs in Foundry

14:44saying, oh, based on like this Aave

14:46Vault deployment, is this bug actually

14:49feasible within like uh the execution of

14:51the contracts?

14:53And rule recall was like, okay, out of

14:55all the rules that the provers

14:57themselves checked, how many were

14:58actually also written by a human? And

15:01then we have some cost and runtime

15:02metrics we'll look at.

15:04So yeah, I I talked about this a little

15:07bit, but it's just basically like uh

15:10recall

15:11of the

15:12bugs caught by the tool, how many were

15:14actually in the report.

15:15Next, please.

15:18So this is a really interesting slide

15:19here. It sort of highlights like uh the

15:22medium, low, and informational bugs that

15:24actually made it to the report and how

15:26each tool did. So Auto Provers tool off

15:29the shelf actually did pretty well as

15:31compared to their human teams that

15:32reviewed the Aave Vault.

15:34It caught two of the four um medium bugs

15:37and um one of the informational bugs and

15:39scored the highest among the other two

15:41um

15:43tools.

15:44Uh

15:45SMT Checker got one of the mediums,

15:47actually one of them that Auto Prover

15:49didn't.

15:50Uh and Control

15:52scored a couple of the info findings,

15:54which was like good.

15:56Uh next, please.

15:58So uh this one was like

16:02trying to see how noisy things were,

16:04right? Uh could you go next?

16:09So this shows like all of the findings

16:11like after the tools completely ran

16:13through, this was like Auto Prover gave

16:16us nine findings, Control gave us five

16:18findings, and SMT Checker gave us five

16:21findings.

16:22Two out of the nine of Autoprover

16:24actually made it to the final report.

16:26Two out of the five for Control made it

16:28to the final report, and only one

16:30uh of from SMT Checker made it to the

16:32final report.

16:33What's really interesting here is

16:35actually like you could see in

16:36Autoprover and an SMT Checker like one

16:40formal verification break or spec break

16:42actually led to two separate bugs in the

16:44finding or in the report, which I

16:46thought was pretty interesting.

16:48Um and also speaks to the fact that like

16:50the human is still very involved in this

16:52because you as the auditor would have to

16:54see the formal verification expert and

16:56that uh

16:57formal verification um output and sort

17:00of determine okay like what more can I

17:02break with this? If this spec doesn't

17:04hold, what contracts are really going to

17:06go wrong or what problems are really

17:08going to go wrong in the contracts uh

17:10once they hit production?

17:12Um and then this like

17:15the Autoprover doesn't have this out of

17:17the box and um

17:20like it just sort of came to my uh mind

17:22when I was uh like developing these

17:25tools kind of like

17:27just to like Lazy Max, right? Like just

17:30to be as lazy as possible and see like

17:33give as much to the uh

17:34AI as possible.

17:36So after like the formal verification

17:39actually found some breaks in the spec,

17:42it got fed to another agent and like I'd

17:44mentioned earlier, I had the um agent

17:46write a POC

17:48um in Foundry to see like oh can this

17:50actually happen

17:52from the forked block like from the

17:54Aave's test suite?

17:57Um so surprisingly all the tools were

18:00actually producing sound bugs. Um so all

18:03of the things well like they have high

18:06percentages for like what could actually

18:08happen in the code. So

18:10these things weren't like very noisy. It

18:12was actually very valuable input to like

18:15guide the uh smart contract audit

18:17itself. Um we know sometimes that if

18:20you're like a smart contract and you're

18:23going back and forth with Claude,

18:25you can waste hours being convinced that

18:27there's a bug somewhere where it's not.

18:29So this was actually pretty cool to see.

18:32Um, you could go to the next slide.

18:35And

18:37the caveat here is that like

18:40some bugs Well, many bugs are sound in

18:43the report, but it was actually like you

18:45wouldn't want to bring it to the client

18:47because it actually like doesn't mean

18:48much to the client.

18:50One very simple example here is like

18:52abuse of owner privilege. Like when

18:54you're working with a client, you're

18:55probably going to assume that they want

18:56their protocol to do well and um that

18:59you know they're going to be treating

19:01owner functions uh with like at least

19:04some responsibility. I'll let the OpSec

19:05guys talk more about how you can make

19:08sure those owner functions are working

19:10the way they should, but there wouldn't

19:11be any blatant abuse of owner privilege.

19:13So that's why some of the bugs were

19:15sound, but didn't actually count to

19:17being in the report itself.

19:19Another thing is like unrealistic

19:21deployment conditions. This one vault

19:23that was um

19:25being reviewed by these tools was an

19:28upgrade. Um, so you would assume like

19:30the proxies in place and some of these

19:31things are already in place. Um, and

19:34some things were sound

19:36in the context of what the agent saw,

19:38but they were actually just handled by

19:39the dependencies or elsewhere.

19:41Um, so

19:46if you could go to the previous slide

19:47actually.

19:48Thank you. So like yes, many were sound,

19:51but not all of them made it to the

19:53actual report because of some of the

19:55things we talk about. All right, two

19:56slides forward. Thank you, sir.

19:59Okay, so now we wanted to see like out

20:01of all the rules that the AI produced,

20:04how many of those

20:06rules

20:09actually matched what the human did

20:10versus like all of the rules that were

20:12in the human report? Okay, hold on. Let

20:14me say that one more time.

20:16So like the AI produced a bunch of

20:18rules, like way more rules than the

20:19human did. So of those rules that

20:22actually matched the human. So of like

20:25the formal verification tool produced

20:27100 rules, 10 of those were also

20:30produced by the human.

20:32That divided by all the human rules.

20:35Okay, I hope that makes sense. Next

20:36slide.

20:38So they actually performed fairly fairly

20:40low here. Um

20:43I think I have some ideas why. We'll see

20:45like once we actually check um some more

20:47results later, but on the left side you

20:50could see like all of the rules that the

20:52human wrote or at least groups of rules

20:55that the human wrote

20:56um and the rules that the um agents were

21:00actually able to come up with.

21:02And for time sake I'm going to go to the

21:05next slide.

21:07Um yeah, so this is just sort of you can

21:10go to the next one.

21:12Yeah, uh

21:13this is just a breakdown of runtime,

21:16token spent, and cost. And I think that

21:19this uh slide is a big takeaway.

21:23And it sort of shows why Auto Prover

21:26performed well in uh several different

21:29of the tests. Um probably because it's

21:32spending like almost a magnitude of 10

21:35more tokens than the harnesses we were

21:38able to develop. So that's probably a

21:40testament to Auto Prover being a more

21:43mature piece of software. Um and there's

21:45still being work to be done on the

21:47harnesses uh themselves.

21:50So what was unexpected to me as an

21:52auditor was just how long these provers

21:54take to run, like

21:5758 hours um is a long time.

22:01And yeah, you've got to

22:05hope your CEO is funding your tokens.

22:09Next slide.

22:10Okay, so is the slop or prop? Is like

22:15can we actually have AI power and

22:18experienced smart contract auditors

22:19access to formal verification tools?

22:22So if we look at the numbers, it's

22:24tending towards slop.

22:26But honestly,

22:28as an auditor that worked on it, I am

22:30tending towards definitely a value prop

22:32here.

22:33Um, at the very least,

22:36as an auditor, you have like access to

22:40sort of start to learn this new

22:42specialized skill, like at the very

22:44least. Um, on the next slide.

22:47Uh, actually, could you go to the slide

22:49after this?

22:52Okay, I'll go here and then I'll go

22:54back. Um,

22:56I think the biggest thing why I think

22:58it's prop is that you get valuable

23:00results for edge cases that can just

23:02sort of further guide your manual

23:04review. Um, so

23:07we saw that like the soundness scored

23:09really high. Um, so you could look at

23:11some of those bugs and then like see, oh

23:13okay, like how bad is this really?

23:16Um,

23:17another really cool thing you could do

23:19with this is if you have like the formal

23:22verification running sort of maybe

23:24alongside like your manual review,

23:26um,

23:28maybe you give the report to the client

23:30and they come back uh and they fix a

23:32small part, you could like

23:34run your specs again and make sure that

23:36nothing else broke.

23:38Um,

23:40I do think that like the results alone

23:42do not suggest that like LLMs could

23:44drive a proper standalone formal

23:46verification engagement right now or at

23:48least that that could be driven by a

23:50sole smart contract auditor. I think

23:52it's still a very, very specialized

23:55thing that's going to need like practice

23:57um and expertise. Could you go to the

23:59previous slide actually?

24:01Thank you. Uh, so this sort of goes to

24:03the shortcomings. It's it's hard to

24:04learn. Um, the results that we got are

24:07actually dependent on the harness, so

24:09you could tell me to just prompt better.

24:11Okay, that's sort of like the next step.

24:13We'll work on it together. Um, results

24:16are on the whim of the model itself. So

24:18you can see a real screenshot that

24:21Claude gave me. Like this is probably

24:23the worst thing you could see after like

24:2559 hours of proving and hundreds of

24:28dollars spent on tokens. Um, luckily,

24:32like

24:33it was in the conversation history, so

24:35the fallout wasn't too bad, but like

24:38there's always this risk of like the

24:40models going rogue and just sort of

24:42like, you know, how much can you really

24:44trust them? Uh, we'll go forward to uh

24:48two slides or

24:50Yes. Okay. So like the next step

24:53uh would be like open sourcing some of

24:55these harnesses we developed. So once

24:58that's like out and shared with the

24:59community, like we can all sort of like

25:02take a crack at it and like tweaking the

25:03harnesses.

25:05Also the workflows um that we did were

25:09ultimately like prompt output prompt um

25:13in like a workflow instead of like a

25:15truly agentic solution. So eventually I

25:17would want to like develop towards that

25:19where you sort of had um either like one

25:22agent or a swarm of agents sort of

25:26like really really looping through all

25:29of these different stages instead of

25:30just like a prompting workflow.

25:33And then of course, there's a lot more

25:35work that should be done with like

25:36testing on multiple models and multiple

25:38code bases.

25:41Next slide.

25:42Okay.

25:43Well, thank you all for the time. Um,

25:46and I hope you learned something. I'll

25:47be around like today and tomorrow uh to

25:50talk more and I'm interested to hear

25:52your thoughts.

25:53Thank you.

Source and accuracy details

Source export SHA256: 6145dc50240426d431cfd4126b2d2574aca2809814c95bf00db010656e35c3e7