DSS 2026 / Day 1 / Amber Hall
AI-FV: Slop or Prop? Explorations in Agentic Formal Verification
Jonathan Mevs
Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.
Transcript
0:00Um, well, hello everybody. Uh, thanks
0:03DeFi Security Summit for having me here.
0:05I'm John.
0:06I'm a senior engineer with Quantstamp,
0:08and the topic I'll be talking about
0:09today is like formal verification and
0:12how far can AI support such uh and some
0:16experiments we did uh surrounding a
0:17couple different tools.
0:21Click.
0:24Okay, great. So these are the guiding
0:26questions that inspired the talk and
0:27like what we'll be focusing on today.
0:30And we wanted to see like to what extent
0:32does AI make existing formal
0:34verification tools accessible to
0:36experienced smart contract auditors?
0:39And to do such, we like ran some
0:40experiments on three prominent tools,
0:43like AutoProver by Certora, Control by
0:46Runtime Verification, and the SMT
0:48Checker by the Solidity team.
0:50And then ultimately we wanted to
0:52conclude where does formal verification
0:54fit in the audit cycle and like some
0:57takeaways that we can have as smart
0:58contract auditors.
1:01Next please.
1:04So this was sort of like the starting
1:06idea, right? Like over the past year um
1:09with like LLMs and like agents getting
1:12really really good,
1:13we've seen like non-technical people be
1:15able to produce like some legitimate
1:17technical output. So this is like your
1:20typical business guy like sitting down
1:22with Claude and being like, you know,
1:24build me 100 million ARR SaaS, make no
1:27mistakes, and actually getting a website
1:30output. So there's like um
1:33this non-technical person doesn't really
1:35need to understand anything about
1:38websites in order to like actually get a
1:40website
1:41um out there and sort of running with
1:44it.
1:45So next slide please.
1:47So then uh we wanted to take that step
1:49further and see like with us as like
1:52pretty technical people, bug hunters,
1:54reading a bunch of Solidity code, um you
1:56know, breaking apart systems, stuff like
1:58that, could we make a similar leap to
2:01the extremely technical realm of formal
2:03verification? So oftentimes people that
2:06are like experts in formal verification
2:07may be studying mathematics at like the
2:10PhD level or
2:13having like many years of this like
2:15targeted study.
2:17So if
2:19we spent like years doing smart
2:22contracts, can AI help us get that leap
2:26similar to like the BD guy to the
2:27website, the smart contract auditor to
2:29the formally verify expert?
2:33So if Thread Guy can do it, maybe I
2:35could do it, right?
2:37Now that's just a meme checkpoint.
2:39Okay, so a slight primer on formal
2:42verification. So let's say we have a
2:45spec like a deposit never decreases
2:47total shares. So if you were to write a
2:49unit test for such, you would just like
2:51do something like deposit 100 and make
2:54sure that the assets you got back
2:57was greater than or equal to 100.
2:59If you were to fuzz that spec, you would
3:01probably
3:02you would run it through a fuzzer and
3:04then it would test a deposit of 100,
3:06test a deposit of 1,000, test a deposit
3:08of
3:091,001, so on and so forth, thousands of
3:11random deposits. But with formal
3:13verification, you want to mathematically
3:17prove that for every deposit from any
3:20state
3:21the shares will never decrease.
3:25So now let's take a look at to what
3:27extent did these tools actually make
3:31formal verification accessible to
3:33experienced smart contract auditors.
3:37So this sort of sets the stage for how
3:39we went about this experiment, like just
3:41sitting down at the computer, hey
3:43Claude, formally verify this
3:45believe in yourself, make no mistakes,
3:47right?
3:49So in order to do so, we needed an ideal
3:51codebase. So this was one that was like
3:54simple for the point of like
3:56experimentation, so if and when we
3:58needed to look under the hood, you could
4:00go read the code. It was like uh had
4:03manageable line of code so that the
4:04prover didn't run forever, which was
4:06actually something that I underestimated
4:08how long those things take. Um and then
4:11we This was really important here is we
4:12needed like a public human formal
4:14verification report that actually had
4:17some real bugs in order for us to catch.
4:20So what we landed on was this Aave vault
4:23that Certora audited both with formal
4:25verification and a manual code review in
4:272023.
4:29It's a simple ERC 4626 vault where
4:32deposits are in yield. The owner can
4:34take a fee and it's just shy of 700
4:36lines of code.
4:38And there were 11 known bugs uh to see
4:40if the formal verification tools
4:42actually caught it.
4:45So now how do we pick the tools? Well
4:47the ideal tool um was tailored
4:49specifically to Solidity smart
4:51contracts. It was open source so we can
4:53use it and download it or um tinker with
4:56it.
4:58It was already like adopted by the
4:59community in some extent um and had
5:01already undergone several client
5:03engagements um and it was thoroughly
5:06documented so that we could basically
5:08build agents uh to use the tools without
5:12uh having the agent to figure too much
5:14out.
5:15So we ran each of these tools on their
5:17own EC2 instances so they had tons of
5:19resources and didn't need to share
5:21anything um and they could sort of just
5:24run the provers on and on and on.
5:27So the first tool that we uh took a look
5:30at was Certora's auto prover which came
5:32out this past summer in July.
5:34Uh this is the only tool that we looked
5:36at that was agentic out of the box. So
5:39Certora has had their prover out for
5:41some time but in July they actually
5:44released this agentic solution.
5:46Um what's unique about this tool is that
5:49it's written in the Certora verification
5:50language uh so that's their own language
5:53where you write the specs. They're
5:55verified at the bytecode level and they
5:56support
5:58uh several different solvers.
6:00And the technique that they approached
6:02to formal verification is verification
6:04condition generation where per rule you
6:07build one formula and ask the solver,
6:09oh, is any input going to break this
6:11rule?
6:14So this is an example of like what the
6:16agent would actually create. Um, this is
6:18the tool or a spec written in uh Certora
6:22verification language,
6:23never decreasing shares.
6:25Um, so this is what would actually get
6:27fed to the solver.
6:29And humans don't usually read this part,
6:31but it's just good to sort of like
6:33get a look at what's actually being fed
6:35into the solver, um which which is this
6:39assert statement
6:40is trying to be proven holding true for
6:42all inputs.
6:52Could you go next?
6:53Thank you. Uh,
6:55the next tool is really interesting.
6:58It's controlled by runtime verification.
7:00Uh, what's cool about this is it runs on
7:02the bytecode in the KVM, which is a
7:05mathematical definition of the Solidity
7:08EVM according to the yellow paper.
7:11So it verifies bytecode.
7:14Uh, it uses the Z3 solver and what this
7:17tool focuses on is symbolic execution,
7:20which is running each test with like
7:23these symbolic inputs at each branch in
7:26the test. You traverse each path and
7:29you're either going to get to an error
7:30that's going to show that the spec
7:33doesn't hold or you're going to get to a
7:35terminating condition and see that the
7:36spec does indeed hold.
7:39Uh, I think this might be out again.
7:42If you could go next. Thank you.
7:44So this is what um you would get from
7:48or what the agent created for running
7:50the control test.
7:51We see that it's extending this control
7:55test library and actually running
7:58the test within the KVM directly.
8:03Next, please.
8:08And
8:09yeah, this an example of what was being
8:11uh fed to the solver.
8:13Um and it's not too important, but you
8:16could see that like there's actually a
8:18different question being asked to the
8:20solver versus what uh Certora asked.
8:24Next, please.
8:27So this one uh SMT Checker is actually
8:29just built directly into the Solidity
8:31compiler. So this tool is actually like
8:33most native to Solidity and probably
8:36like easiest for like a Solidity smart
8:38contract auditor to jump straight into.
8:41Um so this is provided by the Solidity
8:43team. It uses uh constrained Horn
8:45clauses model checking, which is just
8:48asking um a solver for
8:51an invariant that holds for like all
8:53deployments for every call. And then if
8:56that holds, it rules out the bug.
8:58Please go next.
9:02And like I mentioned, this is like most
9:04native to Solidity because it's in the
9:06compiler itself. So there's no extra
9:08libraries. There's no um other languages
9:11that are used to write the spec.
9:14You can just extend a vault,
9:17uh make a call to the contract or the
9:19function that you want to test and add
9:21an assert afterward.
9:23So uh we know that like usually as
9:26Solidity devs, you aren't too keen to
9:28having asserts like within uh the
9:31contracts or the functions themselves.
9:33You usually tend towards require
9:34statements, so you would usually just
9:36extend and put in the assert for the
9:38condition that you're looking for. Um
9:41and then run it through the compiler.
9:43And then yes, it will create uh an
9:46output that you can ask the solver.
9:49Next, please.
9:51So this is just kind of wrapping that
9:53all up. I'll invite you to take a look
9:56um
9:57a diagram like comparing all three of
10:00the tools that we looked at. So all of
10:02them are open source, all of them
10:04generate counterexamples, and all of
10:06them support the Z3 solver, which is
10:08like the main solver being used here.
10:11Um, as I mentioned, Certora's is the
10:13only one that's agentic out of the box
10:15and the only one that has a um
10:18separate language used, the CVL.
10:21Both Control and SMT Checker are written
10:24in Solidity,
10:26and they each have their own uh separate
10:29technique for verification.
10:31Next, please.
10:33Okay, so now for the exciting part, uh
10:36harness development.
10:37Could you go next?
10:39Thank you.
10:40So the idea when we started to write the
10:43harnesses to test the formal
10:44verification tools
10:46was we assumed like, okay, Certora's
10:48AutoProver, that's like sort of the
10:50template, right? Uh that's what an
10:52agentic solution to formal verification
10:54would look like right now.
10:56So we extracted the properties and the
10:59architecture from Certora AutoProver,
11:01and then basically in conjunction with
11:03the documentation on a specific tool,
11:05either Control or SMT Checker,
11:08we fed them um into Claude to actually
11:11write the harness.
11:12And then
11:15Hold on, let me not get uh let me not
11:17lose my train of thought here. So
11:20like with AutoProver, we wanted to
11:22basically like take what they were doing
11:24and see if it could also work with
11:26Control and SMT Checker, um and then
11:28like compare the results of such.
11:31So that wasn't just enough to write the
11:33harness. Each Control and SMT Checker
11:36came with their own like example code
11:38and test cases to work with.
11:40So Control, I think they had uh like a
11:44simple counter contract and a WETH9
11:47contract uh that we could fine-tune the
11:49harness on.
11:51And SMT Checker, I think it was like a
11:53mutex, like a lock
11:55uh contract
11:57and ERC777 that we got to fine-tune on.
12:01So SMT Checker was actually fairly
12:03straightforward with fine-tuning, but
12:06what we found with working with Control,
12:08if you read the Control documentation
12:10for like that formal verification tool
12:12specifically, A lot of it is focused on
12:15um
12:17it's written so that there's a formal
12:19verification human expert like
12:21frequently in the loop. So a lot of the
12:23control documentation talks about
12:25writing a test, okay, it fails, go back
12:28and like tweak it a little bit, add a
12:30lemma, so on and so forth. Um so that
12:33actually didn't do too well with the
12:35agent, which we'll see in a little bit.
12:37But after we basically fine tuned it to
12:40actually solve the example contracts or
12:43example tests that were provided within
12:45the documentation for the tool itself,
12:47it sort of graduated to move on to the
12:49data set.
12:51Next please.
12:52Okay, now the exciting part. So how did
12:54each of these tools actually perform
12:56when driven by state of the art models?
12:59Next please.
13:02Uh so actually one thing on the models
13:04before I talk about the metrics, um for
13:07Sertora's AutoProver,
13:10there's the ability to uh change what
13:13models you were running it on,
13:16but
13:17we so like out of the box it's running
13:19on Opus 5 and Sonnet 5.
13:22And then once I wanted to go tweak the
13:25models uh a little bit,
13:27they have like a high like a high power
13:29model and a low power model. So their
13:31high power was Opus 5 out of the box and
13:33their low power was like Sonnet 5.
13:36When I bumped them up to the newer
13:37generation like 5.5 and Sonnet 5.5, uh
13:40for some reason Sonnet 5.5 like didn't
13:43extract the properties the right way
13:45that Sonnet 5 did. So then I was like,
13:47all right, let's not mess with this,
13:49just test AutoProver out of the out of
13:51the box. So Control and SMT Checker are
13:54checked on Opus 5.5, but AutoProver is
13:57just coming right off the shelf. So
13:59perhaps if I had more time I could tweak
14:01and try to figure out what's going on
14:02with Sonnet 5.5, but for the sake of
14:05experimentation we just let it run.
14:07Uh just a small note before we look at
14:09some of these metrics. So we wanted to
14:11test like recall. So of those 11 bugs
14:14that were in the original report, like
14:16how many did the tool actually catch?
14:18Precision. So like we know with like AI
14:21scanning tools, one of the worst things
14:22you could have as an auditor is like
14:24noisy results that like end up wasting
14:26more of your time. So we wanted to see
14:28like out of everything that the tool
14:30ended on, how many of them actually hit
14:32a known uh hit a known bug.
14:36Soundness is a really interesting one.
14:37Like can the finding actually happen? So
14:40we extended all the provers to actually
14:41ask an agent to write POCs in Foundry
14:44saying, oh, based on like this Aave
14:46Vault deployment, is this bug actually
14:49feasible within like uh the execution of
14:51the contracts?
14:53And rule recall was like, okay, out of
14:55all the rules that the provers
14:57themselves checked, how many were
14:58actually also written by a human? And
15:01then we have some cost and runtime
15:02metrics we'll look at.
15:04So yeah, I I talked about this a little
15:07bit, but it's just basically like uh
15:10recall
15:11of the
15:12bugs caught by the tool, how many were
15:14actually in the report.
15:15Next, please.
15:18So this is a really interesting slide
15:19here. It sort of highlights like uh the
15:22medium, low, and informational bugs that
15:24actually made it to the report and how
15:26each tool did. So Auto Provers tool off
15:29the shelf actually did pretty well as
15:31compared to their human teams that
15:32reviewed the Aave Vault.
15:34It caught two of the four um medium bugs
15:37and um one of the informational bugs and
15:39scored the highest among the other two
15:41um
15:43tools.
15:44Uh
15:45SMT Checker got one of the mediums,
15:47actually one of them that Auto Prover
15:49didn't.
15:50Uh and Control
15:52scored a couple of the info findings,
15:54which was like good.
15:56Uh next, please.
15:58So uh this one was like
16:02trying to see how noisy things were,
16:04right? Uh could you go next?
16:09So this shows like all of the findings
16:11like after the tools completely ran
16:13through, this was like Auto Prover gave
16:16us nine findings, Control gave us five
16:18findings, and SMT Checker gave us five
16:21findings.
16:22Two out of the nine of Autoprover
16:24actually made it to the final report.
16:26Two out of the five for Control made it
16:28to the final report, and only one
16:30uh of from SMT Checker made it to the
16:32final report.
16:33What's really interesting here is
16:35actually like you could see in
16:36Autoprover and an SMT Checker like one
16:40formal verification break or spec break
16:42actually led to two separate bugs in the
16:44finding or in the report, which I
16:46thought was pretty interesting.
16:48Um and also speaks to the fact that like
16:50the human is still very involved in this
16:52because you as the auditor would have to
16:54see the formal verification expert and
16:56that uh
16:57formal verification um output and sort
17:00of determine okay like what more can I
17:02break with this? If this spec doesn't
17:04hold, what contracts are really going to
17:06go wrong or what problems are really
17:08going to go wrong in the contracts uh
17:10once they hit production?
17:12Um and then this like
17:15the Autoprover doesn't have this out of
17:17the box and um
17:20like it just sort of came to my uh mind
17:22when I was uh like developing these
17:25tools kind of like
17:27just to like Lazy Max, right? Like just
17:30to be as lazy as possible and see like
17:33give as much to the uh
17:34AI as possible.
17:36So after like the formal verification
17:39actually found some breaks in the spec,
17:42it got fed to another agent and like I'd
17:44mentioned earlier, I had the um agent
17:46write a POC
17:48um in Foundry to see like oh can this
17:50actually happen
17:52from the forked block like from the
17:54Aave's test suite?
17:57Um so surprisingly all the tools were
18:00actually producing sound bugs. Um so all
18:03of the things well like they have high
18:06percentages for like what could actually
18:08happen in the code. So
18:10these things weren't like very noisy. It
18:12was actually very valuable input to like
18:15guide the uh smart contract audit
18:17itself. Um we know sometimes that if
18:20you're like a smart contract and you're
18:23going back and forth with Claude,
18:25you can waste hours being convinced that
18:27there's a bug somewhere where it's not.
18:29So this was actually pretty cool to see.
18:32Um, you could go to the next slide.
18:35And
18:37the caveat here is that like
18:40some bugs Well, many bugs are sound in
18:43the report, but it was actually like you
18:45wouldn't want to bring it to the client
18:47because it actually like doesn't mean
18:48much to the client.
18:50One very simple example here is like
18:52abuse of owner privilege. Like when
18:54you're working with a client, you're
18:55probably going to assume that they want
18:56their protocol to do well and um that
18:59you know they're going to be treating
19:01owner functions uh with like at least
19:04some responsibility. I'll let the OpSec
19:05guys talk more about how you can make
19:08sure those owner functions are working
19:10the way they should, but there wouldn't
19:11be any blatant abuse of owner privilege.
19:13So that's why some of the bugs were
19:15sound, but didn't actually count to
19:17being in the report itself.
19:19Another thing is like unrealistic
19:21deployment conditions. This one vault
19:23that was um
19:25being reviewed by these tools was an
19:28upgrade. Um, so you would assume like
19:30the proxies in place and some of these
19:31things are already in place. Um, and
19:34some things were sound
19:36in the context of what the agent saw,
19:38but they were actually just handled by
19:39the dependencies or elsewhere.
19:41Um, so
19:46if you could go to the previous slide
19:47actually.
19:48Thank you. So like yes, many were sound,
19:51but not all of them made it to the
19:53actual report because of some of the
19:55things we talk about. All right, two
19:56slides forward. Thank you, sir.
19:59Okay, so now we wanted to see like out
20:01of all the rules that the AI produced,
20:04how many of those
20:06rules
20:09actually matched what the human did
20:10versus like all of the rules that were
20:12in the human report? Okay, hold on. Let
20:14me say that one more time.
20:16So like the AI produced a bunch of
20:18rules, like way more rules than the
20:19human did. So of those rules that
20:22actually matched the human. So of like
20:25the formal verification tool produced
20:27100 rules, 10 of those were also
20:30produced by the human.
20:32That divided by all the human rules.
20:35Okay, I hope that makes sense. Next
20:36slide.
20:38So they actually performed fairly fairly
20:40low here. Um
20:43I think I have some ideas why. We'll see
20:45like once we actually check um some more
20:47results later, but on the left side you
20:50could see like all of the rules that the
20:52human wrote or at least groups of rules
20:55that the human wrote
20:56um and the rules that the um agents were
21:00actually able to come up with.
21:02And for time sake I'm going to go to the
21:05next slide.
21:07Um yeah, so this is just sort of you can
21:10go to the next one.
21:12Yeah, uh
21:13this is just a breakdown of runtime,
21:16token spent, and cost. And I think that
21:19this uh slide is a big takeaway.
21:23And it sort of shows why Auto Prover
21:26performed well in uh several different
21:29of the tests. Um probably because it's
21:32spending like almost a magnitude of 10
21:35more tokens than the harnesses we were
21:38able to develop. So that's probably a
21:40testament to Auto Prover being a more
21:43mature piece of software. Um and there's
21:45still being work to be done on the
21:47harnesses uh themselves.
21:50So what was unexpected to me as an
21:52auditor was just how long these provers
21:54take to run, like
21:5758 hours um is a long time.
22:01And yeah, you've got to
22:05hope your CEO is funding your tokens.
22:09Next slide.
22:10Okay, so is the slop or prop? Is like
22:15can we actually have AI power and
22:18experienced smart contract auditors
22:19access to formal verification tools?
22:22So if we look at the numbers, it's
22:24tending towards slop.
22:26But honestly,
22:28as an auditor that worked on it, I am
22:30tending towards definitely a value prop
22:32here.
22:33Um, at the very least,
22:36as an auditor, you have like access to
22:40sort of start to learn this new
22:42specialized skill, like at the very
22:44least. Um, on the next slide.
22:47Uh, actually, could you go to the slide
22:49after this?
22:52Okay, I'll go here and then I'll go
22:54back. Um,
22:56I think the biggest thing why I think
22:58it's prop is that you get valuable
23:00results for edge cases that can just
23:02sort of further guide your manual
23:04review. Um, so
23:07we saw that like the soundness scored
23:09really high. Um, so you could look at
23:11some of those bugs and then like see, oh
23:13okay, like how bad is this really?
23:16Um,
23:17another really cool thing you could do
23:19with this is if you have like the formal
23:22verification running sort of maybe
23:24alongside like your manual review,
23:26um,
23:28maybe you give the report to the client
23:30and they come back uh and they fix a
23:32small part, you could like
23:34run your specs again and make sure that
23:36nothing else broke.
23:38Um,
23:40I do think that like the results alone
23:42do not suggest that like LLMs could
23:44drive a proper standalone formal
23:46verification engagement right now or at
23:48least that that could be driven by a
23:50sole smart contract auditor. I think
23:52it's still a very, very specialized
23:55thing that's going to need like practice
23:57um and expertise. Could you go to the
23:59previous slide actually?
24:01Thank you. Uh, so this sort of goes to
24:03the shortcomings. It's it's hard to
24:04learn. Um, the results that we got are
24:07actually dependent on the harness, so
24:09you could tell me to just prompt better.
24:11Okay, that's sort of like the next step.
24:13We'll work on it together. Um, results
24:16are on the whim of the model itself. So
24:18you can see a real screenshot that
24:21Claude gave me. Like this is probably
24:23the worst thing you could see after like
24:2559 hours of proving and hundreds of
24:28dollars spent on tokens. Um, luckily,
24:32like
24:33it was in the conversation history, so
24:35the fallout wasn't too bad, but like
24:38there's always this risk of like the
24:40models going rogue and just sort of
24:42like, you know, how much can you really
24:44trust them? Uh, we'll go forward to uh
24:48two slides or
24:50Yes. Okay. So like the next step
24:53uh would be like open sourcing some of
24:55these harnesses we developed. So once
24:58that's like out and shared with the
24:59community, like we can all sort of like
25:02take a crack at it and like tweaking the
25:03harnesses.
25:05Also the workflows um that we did were
25:09ultimately like prompt output prompt um
25:13in like a workflow instead of like a
25:15truly agentic solution. So eventually I
25:17would want to like develop towards that
25:19where you sort of had um either like one
25:22agent or a swarm of agents sort of
25:26like really really looping through all
25:29of these different stages instead of
25:30just like a prompting workflow.
25:33And then of course, there's a lot more
25:35work that should be done with like
25:36testing on multiple models and multiple
25:38code bases.
25:41Next slide.
25:42Okay.
25:43Well, thank you all for the time. Um,
25:46and I hope you learned something. I'll
25:47be around like today and tomorrow uh to
25:50talk more and I'm interested to hear
25:52your thoughts.
25:53Thank you.
Source and accuracy details
- Names, organizations, technical terms, code and numbers may be misrecognized.
- Source wording and disfluencies are preserved; no editorial corrections were applied.
- Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.
- A full caption-track export does not establish that every spoken word is represented.
- Automatic YouTube captions; not human-verified against the audio.
Source export SHA256: 6145dc50240426d431cfd4126b2d2574aca2809814c95bf00db010656e35c3e7