DSS

DSS 2026 / Day 1 / Pearl Hall

Can AI Audit Your Smart Contract? We Tested It on Real Audits.

Mohamad Mansouri

16:21 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:01So I'm Mohamed Mansour from D-Dope.

0:03So I'm a security researcher researcher

0:05at D-Dope. At D-Dope we are a security

0:07company in Web 3 field and we do

0:10security audits. So that's why we are

0:12doing a research

0:13on

0:15how

0:16security how AI can do security audits.

0:19So basically the main question is that

0:23would you ship your code

0:25and deploy it on chain if it was only

0:27audited by AI?

0:30This is a main question. So how how many

0:31people are working on security in the

0:33room?

0:35So there's there's quite some people

0:36right working. So in order to answer

0:39this question so what we did actually is

0:42that in one line what we did is that we

0:44hid our audits

0:46and then we asked the AI to audit our

0:49the codes that we have audited before

0:51and generate reports and see how well

0:53they do.

0:54Right? So this is in one line what we

0:56are going to to show you the results of.

1:04So again we what we are doing here is to

1:06answer the question whether we can rely

1:08on AI to audit code

1:11and by this means that audit code to

1:13deploy without having human people

1:16auditing this code.

1:17And as a security company who have done

1:20a lot of audits and we have a lot of

1:22clients that we have engaged in security

1:25assessments of their protocols, we have

1:27a lot of data sets and what we wanted to

1:30do is to check this by

1:33by hiding our reports from the AI and

1:35asked it to audit itself this code at

1:37that version we audited and see how well

1:39it does.

1:42So here's how the experiment works in in

1:46specifically. So

1:47we chose eight of our past client audits

1:52and what we did is basically asked the

1:55AI to audit these different agents,

1:57different models. We'll talk about this

1:59and from the reports they generate, we

2:01will try to first match

2:03them with the reports to see how many of

2:04these are existing in our reports. And

2:06for the one who doesn't match, we will

2:09try to triage that by first doing some

2:12AI filtering. And for anything that

2:14looks serious, we're going to do manual

2:16revision and take a decision on it.

2:19Um, please.

2:22So from these eight

2:23eight audits,

2:25there are six audits are in Solidity

2:27with different code sizes.

2:29One in Move C chain and one in Talk for

2:33Ton chain.

2:34With different sizes, different types of

2:37different periods of engagement. In

2:39total, there's 61 bugs in these reports.

2:43And there's different amount of time

2:45spent on these to find them.

2:47Now let let me tell you that we have

2:50done this several times. So we've done

2:52this entire experiments on much more

2:54than eight audits in June. And this is

2:56the recent one we did lately in the last

2:58month. So we're going to present the

3:00recent one and also compare it to

3:02previously how was

3:03AI in

3:04four months, five months from now was

3:06doing.

3:07Please.

3:10These are the agents we are trying to

3:11test. So the ones in blue here are ones

3:13that we are internally building and

3:15developing and using in our internal

3:17procedures.

3:19We start from different We do different

3:21tiers. So we start from let's say an AI

3:24that is basically just a plain model,

3:27basically just a simple prompt

3:29that asks the model to to audit this

3:33code.

3:33And with the flagship

3:35version.

3:36And all the way to audit factories,

3:38which is like a very well, you know,

3:41pipelines of of using AI to to to find

3:45bugs. And there's like things in between

3:48like light skills, only using skills

3:49with lightweight with light workload so

3:52to make this fast. Also, there's like

3:54the public tools that are existing now,

3:57which basically relies on

3:59running security tools and then triaging

4:01the results with LLM. And there's the

4:03audit factories. We have two agents

4:05there.

4:06One is more lightweight and the one Zeus

4:08is the more, you know, heavy metal stuff

4:11that we really takes a lot of talking

4:13consumption but does everything from

4:16fuzzing to to understanding the protocol

4:18threat modeling and everything to find

4:20to find the bugs.

4:23Uh let's jump directly to the results.

4:26Do you have any can anyone give some

4:28expectation? What do they think?

4:31So we're going to measure the recall,

4:33how much the percentage of

4:37bugs

4:39that are in our reports that AI will

4:40found.

4:42>> Half.

4:43>> Half?

4:47Okay, let's see.

4:49So basically it's 45% if we took the

4:52average.

4:53So for every two bugs in our report,

4:56serious bugs, medium and above,

4:58there's a 50% chance that or almost 50%

5:02chance that on average an agent would

5:04find it. If we combine the results from

5:06all agents together,

5:08there's going to go up to 73. Now what's

5:10interesting is you can see from June

5:13it's was much much less, almost half of

5:15this result from June.

5:16So basically in four months the alarms

5:19went to a better version where increased

5:21this results.

5:22Uh but interestingly the

5:26the agents found all the critical bugs.

5:30So critical sorry high is wrong here. So

5:32only critical bugs.

5:34Uh this is only the result on the

5:36solidity codes. Now if we go over the

5:38entire eight

5:40eight eight projects with Sui and Ton in

5:43them, then it's going to be a bit bit

5:45less.

5:46But interesting from this slide we can

5:48see that

5:50the higher critical bugs, more severe

5:52bugs are most more likely to be found

5:55than medium bugs.

5:56In total for the eight eight audits,

5:59it's still 50% so like 29 bugs were

6:02found out of the 61 that we have in our

6:04reports.

6:07Now, if we look at the results per per

6:09per per per agent,

6:12so Zeus, which is like the heavy metal

6:16agent we have, done the best, but it

6:19also has like a highest token

6:20consumption around 7 750 like seven

6:24times or even more like 10 times more.

6:26And here I'm measuring the token

6:28consumption in dollar cost so that you

6:30can get a relative value of this.

6:33We have Argos at 55 %, our Hermes is at

6:3638 plain models as like doing a bit less

6:39than that.

6:43Now, the other question that you will

6:44might want to ask is basically, okay,

6:47this is how much agents I'm finding from

6:49the bugs we have in the reports.

6:51But what about the bugs that you don't

6:53have in the reports? Did they find new

6:55bugs that you missed?

6:57Well, yes, they did. So they are also

6:59finding bugs that we missed.

7:02Out of if we took the 10 most severe

7:05findings in their reports and we

7:07analyzed them, 4.5 of these 10 on

7:10average are ones that we have in our

7:12report and they reported

7:14there's almost on average 1.1 bug that

7:17is basically a new bug that we missed so

7:19medium and above and the others that are

7:21reporting basically are

7:23less severe bugs. So out of

7:26these eight projects, there was 16 bugs

7:30medium and above that we have missed.

7:33Now, none of these are critical.

7:35There's two of them are high and 14 as

7:38as medium.

7:40Okay?

7:41Now, one disclaimer here to say that all

7:45these

7:46projects that we run the audits on the

7:48experiments on are old projects.

7:52At the time we were not using AI. So all

7:54these results does not rely on the fact

7:56that currently we use AI. So basically

7:59the results will be much much different

8:01when you are actually integrating AI

8:02yourself when you are doing security

8:04audits as a human.

8:05So So comparing

8:07pure human review to

8:10pure AI review.

8:13Now this is a very important slide also

8:16for people interested in uh

8:19seeing the difference what did change

8:21from June to until this point with the

8:23new

8:24uh release. Just to give you a reference

8:26in June we were talking about Opus 4.1

8:294.8 for for Anthropic

8:32and GPT 5.1

8:35I think in June right? Now we're talking

8:37about Opus 5.5 and Sol 6.1

8:42uh and also like the Fable and Astra

8:44which are the flagships.

8:45Um

8:47here in the plane models this is very

8:50interesting here that you can see that

8:53the plane models in the current versions

8:56now Astra and Fable

8:58are doing much better than our previous

9:02agents the the full agents we have in

9:05June.

9:06So you have much better performance if

9:08you just have a single prompt with Astra

9:11and Fable

9:12now than whatever tool you were using in

9:15June.

9:16Right?

9:17Uh also uh another thing you can notice

9:20that previously

9:23uh we were having like double

9:26performance of plane models. Our agents

9:28were doing double of what the plane

9:30models were doing. Now it's it's a bit

9:33less than double.

9:34So the gap is closing but you still have

9:37better performance when you actually

9:40build pipelines over the agents.

9:43And uh one uh

9:45biggest improvement we have seen is the

9:46languages that are not

9:48uh were very common in June like they

9:51are learning more trained more on

9:55languages like Move and Ton than in

9:57June. So they are being trained on these

9:59new uh less common and famous languages.

10:05So um to answer the question so can AI

10:08audit your smart contracts? Now, based

10:11on these results,

10:12what I would conclude is no, not on

10:15their own. Now, if you are interested in

10:18in doing some sort of a bug bounty,

10:21well, it's very interesting. The the AI

10:23can do a lot of good stuff and find

10:26bugs. Because in bug bounties, what you

10:28are interested in is basically finding a

10:31bug in a protocol,

10:33right? Doesn't have to be a specific

10:35protocol. It doesn't have to be any

10:37specific bug.

10:39Any severe bug would give you money.

10:42Now, audits

10:43you don't you specifically want to find

10:46bugs in this report and all bugs that

10:48you can find.

10:49So your goal is to provide

10:52a security assessment of this protocol

10:54and like say this is a good posture for

10:56the protocol to be deployed.

10:58So that's why relying only on AI agents

11:02is not is not yet something that is a

11:04good decision yet.

11:05But of course, we as security auditors,

11:09we are using AI in our pipelines to do

11:12security audits which are is increasing

11:15a lot our efficiency, increasing a lot

11:17our

11:18coverage and

11:20the the overall performance.

11:25So

11:26thank you and this is the results for

11:28now and I'm happy to take questions.

11:31[applause]

11:36I have some time so I'll take some

11:37questions, right?

11:41>> Next question.

11:52>> That that's a great question. So

11:56basically

11:58in the pipelines

11:59you have you guide the LLM to look where

12:03to look and where to focus more.

12:06This is what what you get the best

12:08improvement. Of course, you also add

12:09tools,

12:11but you're also seen that

12:14like pipelines that basically used tools

12:16and then triage them. But our like heavy

12:19metal pipelines is basically more than

12:22that which is make it makes

12:24people

12:25make makes LLM redirect on specific

12:27parts of the code and do multiple rounds

12:29and do different

12:32instructions of how to look at this. So

12:35do you want to look for what types of

12:36bug?

12:37So some of them based on like invariance

12:40that you infer from the protocol and

12:41then ask the protocol to to to check.

12:43Now what is

12:45the recent models and flagship models is

12:47trying to do is basically do this in

12:48themselves. So they try to

12:52do some sub agents create some sub

12:54agents and give them these instructions.

12:56But you're still better

12:58as a human auditor with very high skills

13:00and experience in

13:01in auditing

13:03giving this instruction than AI yet. Now

13:05they might be more efficient in

13:07implementing these instructions but

13:09you're still better giving this

13:10instruction as a human. That's why you

13:12get you get this this

13:14you know

13:15better performance.

13:17Yes please.

13:19>> You were talking about your own order

13:21reports

13:24How do you ensure that both in terms of

13:28say the models including public reports

13:31in their training

13:32and maybe even inferring that from

13:35like public bug fixes?

13:38>> Yeah. So this is also a very good

13:41question and we do our best but we don't

13:43have a guarantee.

13:45What we do is basically of course we

13:46remove all the materials that can give

13:49him access directly like the git history

13:51and stuff like that.

13:53But like making sure that the training

13:55data have not seen our bug is something

13:57that is

13:58not very

14:00not very easy to do.

14:02Now one suspect we have that the

14:05additional performance and better

14:06performance we had is because we have

14:08trained we have did the experiment

14:10before. So they trained with the new

14:12models they've seen this then they are

14:14giving better performance. That's that

14:15that might be a limitation and there is.

14:17And then we have to assess it. This is

14:19an ongoing work. Some of these results

14:21are not finished yet. We are doing more

14:22and more. We're going to continue doing

14:24it and please subscribe to the to check

14:27our

14:28blog post to see how we're going to give

14:31more and more results about this. And

14:33yeah, it's it's a limitation. We don't

14:35We don't know that they are not trained

14:37on these bugs. Not all of these are

14:39public reports. So also that's a good

14:42thing. Not all of them, but some of

14:43them.

14:45Yes.

14:49What what type of false positive are you

14:51talking about?

14:52>> False positive how many results

14:58>> Yes. Now false positive could be like

15:00invalid bugs or bugs that are flagged as

15:03severe, but they are not. They are

15:05likely because there's there's like a

15:08line that's hard to put there.

15:10The line would be

15:12some sometimes they will find a bug, but

15:15when you assess it, okay, it's fine, but

15:17it has like very hard assumptions. You

15:19are assuming that the admin have not

15:21missed mistake for that bug to appear

15:23and this is unlikely to happen. So it's

15:25not that severe or that it's an

15:27informational. So if these are counted

15:29as false positive, it's very hard to put

15:31this line to count the false positive.

15:33But if we're talking about invalid

15:34issues like hallucinations or like

15:36assuming things that are not correct

15:37about the code, then this is very very

15:40low. And we have also done benchmarking

15:43of our benchmarking. So basically

15:45created hallucinations

15:47bugs and tried to see how much they can

15:50agree that they are bugs or something

15:52like that. So we have done experiments

15:54on this. We couldn't present everything,

15:55but in terms of invalid bugs, they're

15:58unlikely to to report invalid bugs. But

16:01yeah, informational bugs and stuff like

16:03that that they are exaggerating, this is

16:06this is

16:07possible.

16:08Exact numbers are not in here. We can

16:11put some some stuff online

16:13soon when we have more experiments.

16:16Okay.

16:17Thank you a lot.

Source and accuracy details

Source export SHA256: 04bca3d592131b1802b7461465c6e0ea2c63d51d6230df9dc858e5984281a8d9