DSS 2026 / Day 1 / Pearl Hall
Can AI Audit Your Smart Contract? We Tested It on Real Audits.
Mohamad Mansouri
Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.
Transcript
0:01So I'm Mohamed Mansour from D-Dope.
0:03So I'm a security researcher researcher
0:05at D-Dope. At D-Dope we are a security
0:07company in Web 3 field and we do
0:10security audits. So that's why we are
0:12doing a research
0:13on
0:15how
0:16security how AI can do security audits.
0:19So basically the main question is that
0:23would you ship your code
0:25and deploy it on chain if it was only
0:27audited by AI?
0:30This is a main question. So how how many
0:31people are working on security in the
0:33room?
0:35So there's there's quite some people
0:36right working. So in order to answer
0:39this question so what we did actually is
0:42that in one line what we did is that we
0:44hid our audits
0:46and then we asked the AI to audit our
0:49the codes that we have audited before
0:51and generate reports and see how well
0:53they do.
0:54Right? So this is in one line what we
0:56are going to to show you the results of.
1:04So again we what we are doing here is to
1:06answer the question whether we can rely
1:08on AI to audit code
1:11and by this means that audit code to
1:13deploy without having human people
1:16auditing this code.
1:17And as a security company who have done
1:20a lot of audits and we have a lot of
1:22clients that we have engaged in security
1:25assessments of their protocols, we have
1:27a lot of data sets and what we wanted to
1:30do is to check this by
1:33by hiding our reports from the AI and
1:35asked it to audit itself this code at
1:37that version we audited and see how well
1:39it does.
1:42So here's how the experiment works in in
1:46specifically. So
1:47we chose eight of our past client audits
1:52and what we did is basically asked the
1:55AI to audit these different agents,
1:57different models. We'll talk about this
1:59and from the reports they generate, we
2:01will try to first match
2:03them with the reports to see how many of
2:04these are existing in our reports. And
2:06for the one who doesn't match, we will
2:09try to triage that by first doing some
2:12AI filtering. And for anything that
2:14looks serious, we're going to do manual
2:16revision and take a decision on it.
2:19Um, please.
2:22So from these eight
2:23eight audits,
2:25there are six audits are in Solidity
2:27with different code sizes.
2:29One in Move C chain and one in Talk for
2:33Ton chain.
2:34With different sizes, different types of
2:37different periods of engagement. In
2:39total, there's 61 bugs in these reports.
2:43And there's different amount of time
2:45spent on these to find them.
2:47Now let let me tell you that we have
2:50done this several times. So we've done
2:52this entire experiments on much more
2:54than eight audits in June. And this is
2:56the recent one we did lately in the last
2:58month. So we're going to present the
3:00recent one and also compare it to
3:02previously how was
3:03AI in
3:04four months, five months from now was
3:06doing.
3:07Please.
3:10These are the agents we are trying to
3:11test. So the ones in blue here are ones
3:13that we are internally building and
3:15developing and using in our internal
3:17procedures.
3:19We start from different We do different
3:21tiers. So we start from let's say an AI
3:24that is basically just a plain model,
3:27basically just a simple prompt
3:29that asks the model to to audit this
3:33code.
3:33And with the flagship
3:35version.
3:36And all the way to audit factories,
3:38which is like a very well, you know,
3:41pipelines of of using AI to to to find
3:45bugs. And there's like things in between
3:48like light skills, only using skills
3:49with lightweight with light workload so
3:52to make this fast. Also, there's like
3:54the public tools that are existing now,
3:57which basically relies on
3:59running security tools and then triaging
4:01the results with LLM. And there's the
4:03audit factories. We have two agents
4:05there.
4:06One is more lightweight and the one Zeus
4:08is the more, you know, heavy metal stuff
4:11that we really takes a lot of talking
4:13consumption but does everything from
4:16fuzzing to to understanding the protocol
4:18threat modeling and everything to find
4:20to find the bugs.
4:23Uh let's jump directly to the results.
4:26Do you have any can anyone give some
4:28expectation? What do they think?
4:31So we're going to measure the recall,
4:33how much the percentage of
4:37bugs
4:39that are in our reports that AI will
4:40found.
4:42>> Half.
4:43>> Half?
4:47Okay, let's see.
4:49So basically it's 45% if we took the
4:52average.
4:53So for every two bugs in our report,
4:56serious bugs, medium and above,
4:58there's a 50% chance that or almost 50%
5:02chance that on average an agent would
5:04find it. If we combine the results from
5:06all agents together,
5:08there's going to go up to 73. Now what's
5:10interesting is you can see from June
5:13it's was much much less, almost half of
5:15this result from June.
5:16So basically in four months the alarms
5:19went to a better version where increased
5:21this results.
5:22Uh but interestingly the
5:26the agents found all the critical bugs.
5:30So critical sorry high is wrong here. So
5:32only critical bugs.
5:34Uh this is only the result on the
5:36solidity codes. Now if we go over the
5:38entire eight
5:40eight eight projects with Sui and Ton in
5:43them, then it's going to be a bit bit
5:45less.
5:46But interesting from this slide we can
5:48see that
5:50the higher critical bugs, more severe
5:52bugs are most more likely to be found
5:55than medium bugs.
5:56In total for the eight eight audits,
5:59it's still 50% so like 29 bugs were
6:02found out of the 61 that we have in our
6:04reports.
6:07Now, if we look at the results per per
6:09per per per agent,
6:12so Zeus, which is like the heavy metal
6:16agent we have, done the best, but it
6:19also has like a highest token
6:20consumption around 7 750 like seven
6:24times or even more like 10 times more.
6:26And here I'm measuring the token
6:28consumption in dollar cost so that you
6:30can get a relative value of this.
6:33We have Argos at 55 %, our Hermes is at
6:3638 plain models as like doing a bit less
6:39than that.
6:43Now, the other question that you will
6:44might want to ask is basically, okay,
6:47this is how much agents I'm finding from
6:49the bugs we have in the reports.
6:51But what about the bugs that you don't
6:53have in the reports? Did they find new
6:55bugs that you missed?
6:57Well, yes, they did. So they are also
6:59finding bugs that we missed.
7:02Out of if we took the 10 most severe
7:05findings in their reports and we
7:07analyzed them, 4.5 of these 10 on
7:10average are ones that we have in our
7:12report and they reported
7:14there's almost on average 1.1 bug that
7:17is basically a new bug that we missed so
7:19medium and above and the others that are
7:21reporting basically are
7:23less severe bugs. So out of
7:26these eight projects, there was 16 bugs
7:30medium and above that we have missed.
7:33Now, none of these are critical.
7:35There's two of them are high and 14 as
7:38as medium.
7:40Okay?
7:41Now, one disclaimer here to say that all
7:45these
7:46projects that we run the audits on the
7:48experiments on are old projects.
7:52At the time we were not using AI. So all
7:54these results does not rely on the fact
7:56that currently we use AI. So basically
7:59the results will be much much different
8:01when you are actually integrating AI
8:02yourself when you are doing security
8:04audits as a human.
8:05So So comparing
8:07pure human review to
8:10pure AI review.
8:13Now this is a very important slide also
8:16for people interested in uh
8:19seeing the difference what did change
8:21from June to until this point with the
8:23new
8:24uh release. Just to give you a reference
8:26in June we were talking about Opus 4.1
8:294.8 for for Anthropic
8:32and GPT 5.1
8:35I think in June right? Now we're talking
8:37about Opus 5.5 and Sol 6.1
8:42uh and also like the Fable and Astra
8:44which are the flagships.
8:45Um
8:47here in the plane models this is very
8:50interesting here that you can see that
8:53the plane models in the current versions
8:56now Astra and Fable
8:58are doing much better than our previous
9:02agents the the full agents we have in
9:05June.
9:06So you have much better performance if
9:08you just have a single prompt with Astra
9:11and Fable
9:12now than whatever tool you were using in
9:15June.
9:16Right?
9:17Uh also uh another thing you can notice
9:20that previously
9:23uh we were having like double
9:26performance of plane models. Our agents
9:28were doing double of what the plane
9:30models were doing. Now it's it's a bit
9:33less than double.
9:34So the gap is closing but you still have
9:37better performance when you actually
9:40build pipelines over the agents.
9:43And uh one uh
9:45biggest improvement we have seen is the
9:46languages that are not
9:48uh were very common in June like they
9:51are learning more trained more on
9:55languages like Move and Ton than in
9:57June. So they are being trained on these
9:59new uh less common and famous languages.
10:05So um to answer the question so can AI
10:08audit your smart contracts? Now, based
10:11on these results,
10:12what I would conclude is no, not on
10:15their own. Now, if you are interested in
10:18in doing some sort of a bug bounty,
10:21well, it's very interesting. The the AI
10:23can do a lot of good stuff and find
10:26bugs. Because in bug bounties, what you
10:28are interested in is basically finding a
10:31bug in a protocol,
10:33right? Doesn't have to be a specific
10:35protocol. It doesn't have to be any
10:37specific bug.
10:39Any severe bug would give you money.
10:42Now, audits
10:43you don't you specifically want to find
10:46bugs in this report and all bugs that
10:48you can find.
10:49So your goal is to provide
10:52a security assessment of this protocol
10:54and like say this is a good posture for
10:56the protocol to be deployed.
10:58So that's why relying only on AI agents
11:02is not is not yet something that is a
11:04good decision yet.
11:05But of course, we as security auditors,
11:09we are using AI in our pipelines to do
11:12security audits which are is increasing
11:15a lot our efficiency, increasing a lot
11:17our
11:18coverage and
11:20the the overall performance.
11:25So
11:26thank you and this is the results for
11:28now and I'm happy to take questions.
11:31[applause]
11:36I have some time so I'll take some
11:37questions, right?
11:41>> Next question.
11:52>> That that's a great question. So
11:56basically
11:58in the pipelines
11:59you have you guide the LLM to look where
12:03to look and where to focus more.
12:06This is what what you get the best
12:08improvement. Of course, you also add
12:09tools,
12:11but you're also seen that
12:14like pipelines that basically used tools
12:16and then triage them. But our like heavy
12:19metal pipelines is basically more than
12:22that which is make it makes
12:24people
12:25make makes LLM redirect on specific
12:27parts of the code and do multiple rounds
12:29and do different
12:32instructions of how to look at this. So
12:35do you want to look for what types of
12:36bug?
12:37So some of them based on like invariance
12:40that you infer from the protocol and
12:41then ask the protocol to to to check.
12:43Now what is
12:45the recent models and flagship models is
12:47trying to do is basically do this in
12:48themselves. So they try to
12:52do some sub agents create some sub
12:54agents and give them these instructions.
12:56But you're still better
12:58as a human auditor with very high skills
13:00and experience in
13:01in auditing
13:03giving this instruction than AI yet. Now
13:05they might be more efficient in
13:07implementing these instructions but
13:09you're still better giving this
13:10instruction as a human. That's why you
13:12get you get this this
13:14you know
13:15better performance.
13:17Yes please.
13:19>> You were talking about your own order
13:21reports
13:24How do you ensure that both in terms of
13:28say the models including public reports
13:31in their training
13:32and maybe even inferring that from
13:35like public bug fixes?
13:38>> Yeah. So this is also a very good
13:41question and we do our best but we don't
13:43have a guarantee.
13:45What we do is basically of course we
13:46remove all the materials that can give
13:49him access directly like the git history
13:51and stuff like that.
13:53But like making sure that the training
13:55data have not seen our bug is something
13:57that is
13:58not very
14:00not very easy to do.
14:02Now one suspect we have that the
14:05additional performance and better
14:06performance we had is because we have
14:08trained we have did the experiment
14:10before. So they trained with the new
14:12models they've seen this then they are
14:14giving better performance. That's that
14:15that might be a limitation and there is.
14:17And then we have to assess it. This is
14:19an ongoing work. Some of these results
14:21are not finished yet. We are doing more
14:22and more. We're going to continue doing
14:24it and please subscribe to the to check
14:27our
14:28blog post to see how we're going to give
14:31more and more results about this. And
14:33yeah, it's it's a limitation. We don't
14:35We don't know that they are not trained
14:37on these bugs. Not all of these are
14:39public reports. So also that's a good
14:42thing. Not all of them, but some of
14:43them.
14:45Yes.
14:49What what type of false positive are you
14:51talking about?
14:52>> False positive how many results
14:58>> Yes. Now false positive could be like
15:00invalid bugs or bugs that are flagged as
15:03severe, but they are not. They are
15:05likely because there's there's like a
15:08line that's hard to put there.
15:10The line would be
15:12some sometimes they will find a bug, but
15:15when you assess it, okay, it's fine, but
15:17it has like very hard assumptions. You
15:19are assuming that the admin have not
15:21missed mistake for that bug to appear
15:23and this is unlikely to happen. So it's
15:25not that severe or that it's an
15:27informational. So if these are counted
15:29as false positive, it's very hard to put
15:31this line to count the false positive.
15:33But if we're talking about invalid
15:34issues like hallucinations or like
15:36assuming things that are not correct
15:37about the code, then this is very very
15:40low. And we have also done benchmarking
15:43of our benchmarking. So basically
15:45created hallucinations
15:47bugs and tried to see how much they can
15:50agree that they are bugs or something
15:52like that. So we have done experiments
15:54on this. We couldn't present everything,
15:55but in terms of invalid bugs, they're
15:58unlikely to to report invalid bugs. But
16:01yeah, informational bugs and stuff like
16:03that that they are exaggerating, this is
16:06this is
16:07possible.
16:08Exact numbers are not in here. We can
16:11put some some stuff online
16:13soon when we have more experiments.
16:16Okay.
16:17Thank you a lot.
Source and accuracy details
- Names, organizations, technical terms, code and numbers may be misrecognized.
- Source wording and disfluencies are preserved; no editorial corrections were applied.
- Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.
- A full caption-track export does not establish that every spoken word is represented.
- Automatic YouTube captions; not human-verified against the audio.
Source export SHA256: 04bca3d592131b1802b7461465c6e0ea2c63d51d6230df9dc858e5984281a8d9