DSS 2026 — Can AI Audit Your Smart Contract? We Tested It on Real Audits. Mohamad Mansouri Source video: https://www.youtube.com/watch?v=HvPK8VfCIG0 Automatic captions — not human-reviewed. Complete spoken-audio coverage is unverified. Source language tag: en (English). Source export SHA256: 04bca3d592131b1802b7461465c6e0ea2c63d51d6230df9dc858e5984281a8d9 [0:01] So I'm Mohamed Mansour from D-Dope. [0:03] So I'm a security researcher researcher [0:05] at D-Dope. At D-Dope we are a security [0:07] company in Web 3 field and we do [0:10] security audits. So that's why we are [0:12] doing a research [0:13] on [0:15] how [0:16] security how AI can do security audits. [0:19] So basically the main question is that [0:23] would you ship your code [0:25] and deploy it on chain if it was only [0:27] audited by AI? [0:30] This is a main question. So how how many [0:31] people are working on security in the [0:33] room? [0:35] So there's there's quite some people [0:36] right working. So in order to answer [0:39] this question so what we did actually is [0:42] that in one line what we did is that we [0:44] hid our audits [0:46] and then we asked the AI to audit our [0:49] the codes that we have audited before [0:51] and generate reports and see how well [0:53] they do. [0:54] Right? So this is in one line what we [0:56] are going to to show you the results of. [1:04] So again we what we are doing here is to [1:06] answer the question whether we can rely [1:08] on AI to audit code [1:11] and by this means that audit code to [1:13] deploy without having human people [1:16] auditing this code. [1:17] And as a security company who have done [1:20] a lot of audits and we have a lot of [1:22] clients that we have engaged in security [1:25] assessments of their protocols, we have [1:27] a lot of data sets and what we wanted to [1:30] do is to check this by [1:33] by hiding our reports from the AI and [1:35] asked it to audit itself this code at [1:37] that version we audited and see how well [1:39] it does. [1:42] So here's how the experiment works in in [1:46] specifically. So [1:47] we chose eight of our past client audits [1:52] and what we did is basically asked the [1:55] AI to audit these different agents, [1:57] different models. We'll talk about this [1:59] and from the reports they generate, we [2:01] will try to first match [2:03] them with the reports to see how many of [2:04] these are existing in our reports. And [2:06] for the one who doesn't match, we will [2:09] try to triage that by first doing some [2:12] AI filtering. And for anything that [2:14] looks serious, we're going to do manual [2:16] revision and take a decision on it. [2:19] Um, please. [2:22] So from these eight [2:23] eight audits, [2:25] there are six audits are in Solidity [2:27] with different code sizes. [2:29] One in Move C chain and one in Talk for [2:33] Ton chain. [2:34] With different sizes, different types of [2:37] different periods of engagement. In [2:39] total, there's 61 bugs in these reports. [2:43] And there's different amount of time [2:45] spent on these to find them. [2:47] Now let let me tell you that we have [2:50] done this several times. So we've done [2:52] this entire experiments on much more [2:54] than eight audits in June. And this is [2:56] the recent one we did lately in the last [2:58] month. So we're going to present the [3:00] recent one and also compare it to [3:02] previously how was [3:03] AI in [3:04] four months, five months from now was [3:06] doing. [3:07] Please. [3:10] These are the agents we are trying to [3:11] test. So the ones in blue here are ones [3:13] that we are internally building and [3:15] developing and using in our internal [3:17] procedures. [3:19] We start from different We do different [3:21] tiers. So we start from let's say an AI [3:24] that is basically just a plain model, [3:27] basically just a simple prompt [3:29] that asks the model to to audit this [3:33] code. [3:33] And with the flagship [3:35] version. [3:36] And all the way to audit factories, [3:38] which is like a very well, you know, [3:41] pipelines of of using AI to to to find [3:45] bugs. And there's like things in between [3:48] like light skills, only using skills [3:49] with lightweight with light workload so [3:52] to make this fast. Also, there's like [3:54] the public tools that are existing now, [3:57] which basically relies on [3:59] running security tools and then triaging [4:01] the results with LLM. And there's the [4:03] audit factories. We have two agents [4:05] there. [4:06] One is more lightweight and the one Zeus [4:08] is the more, you know, heavy metal stuff [4:11] that we really takes a lot of talking [4:13] consumption but does everything from [4:16] fuzzing to to understanding the protocol [4:18] threat modeling and everything to find [4:20] to find the bugs. [4:23] Uh let's jump directly to the results. [4:26] Do you have any can anyone give some [4:28] expectation? What do they think? [4:31] So we're going to measure the recall, [4:33] how much the percentage of [4:37] bugs [4:39] that are in our reports that AI will [4:40] found. [4:42] >> Half. [4:43] >> Half? [4:47] Okay, let's see. [4:49] So basically it's 45% if we took the [4:52] average. [4:53] So for every two bugs in our report, [4:56] serious bugs, medium and above, [4:58] there's a 50% chance that or almost 50% [5:02] chance that on average an agent would [5:04] find it. If we combine the results from [5:06] all agents together, [5:08] there's going to go up to 73. Now what's [5:10] interesting is you can see from June [5:13] it's was much much less, almost half of [5:15] this result from June. [5:16] So basically in four months the alarms [5:19] went to a better version where increased [5:21] this results. [5:22] Uh but interestingly the [5:26] the agents found all the critical bugs. [5:30] So critical sorry high is wrong here. So [5:32] only critical bugs. [5:34] Uh this is only the result on the [5:36] solidity codes. Now if we go over the [5:38] entire eight [5:40] eight eight projects with Sui and Ton in [5:43] them, then it's going to be a bit bit [5:45] less. [5:46] But interesting from this slide we can [5:48] see that [5:50] the higher critical bugs, more severe [5:52] bugs are most more likely to be found [5:55] than medium bugs. [5:56] In total for the eight eight audits, [5:59] it's still 50% so like 29 bugs were [6:02] found out of the 61 that we have in our [6:04] reports. [6:07] Now, if we look at the results per per [6:09] per per per agent, [6:12] so Zeus, which is like the heavy metal [6:16] agent we have, done the best, but it [6:19] also has like a highest token [6:20] consumption around 7 750 like seven [6:24] times or even more like 10 times more. [6:26] And here I'm measuring the token [6:28] consumption in dollar cost so that you [6:30] can get a relative value of this. [6:33] We have Argos at 55 %, our Hermes is at [6:36] 38 plain models as like doing a bit less [6:39] than that. [6:43] Now, the other question that you will [6:44] might want to ask is basically, okay, [6:47] this is how much agents I'm finding from [6:49] the bugs we have in the reports. [6:51] But what about the bugs that you don't [6:53] have in the reports? Did they find new [6:55] bugs that you missed? [6:57] Well, yes, they did. So they are also [6:59] finding bugs that we missed. [7:02] Out of if we took the 10 most severe [7:05] findings in their reports and we [7:07] analyzed them, 4.5 of these 10 on [7:10] average are ones that we have in our [7:12] report and they reported [7:14] there's almost on average 1.1 bug that [7:17] is basically a new bug that we missed so [7:19] medium and above and the others that are [7:21] reporting basically are [7:23] less severe bugs. So out of [7:26] these eight projects, there was 16 bugs [7:30] medium and above that we have missed. [7:33] Now, none of these are critical. [7:35] There's two of them are high and 14 as [7:38] as medium. [7:40] Okay? [7:41] Now, one disclaimer here to say that all [7:45] these [7:46] projects that we run the audits on the [7:48] experiments on are old projects. [7:52] At the time we were not using AI. So all [7:54] these results does not rely on the fact [7:56] that currently we use AI. So basically [7:59] the results will be much much different [8:01] when you are actually integrating AI [8:02] yourself when you are doing security [8:04] audits as a human. [8:05] So So comparing [8:07] pure human review to [8:10] pure AI review. [8:13] Now this is a very important slide also [8:16] for people interested in uh [8:19] seeing the difference what did change [8:21] from June to until this point with the [8:23] new [8:24] uh release. Just to give you a reference [8:26] in June we were talking about Opus 4.1 [8:29] 4.8 for for Anthropic [8:32] and GPT 5.1 [8:35] I think in June right? Now we're talking [8:37] about Opus 5.5 and Sol 6.1 [8:42] uh and also like the Fable and Astra [8:44] which are the flagships. [8:45] Um [8:47] here in the plane models this is very [8:50] interesting here that you can see that [8:53] the plane models in the current versions [8:56] now Astra and Fable [8:58] are doing much better than our previous [9:02] agents the the full agents we have in [9:05] June. [9:06] So you have much better performance if [9:08] you just have a single prompt with Astra [9:11] and Fable [9:12] now than whatever tool you were using in [9:15] June. [9:16] Right? [9:17] Uh also uh another thing you can notice [9:20] that previously [9:23] uh we were having like double [9:26] performance of plane models. Our agents [9:28] were doing double of what the plane [9:30] models were doing. Now it's it's a bit [9:33] less than double. [9:34] So the gap is closing but you still have [9:37] better performance when you actually [9:40] build pipelines over the agents. [9:43] And uh one uh [9:45] biggest improvement we have seen is the [9:46] languages that are not [9:48] uh were very common in June like they [9:51] are learning more trained more on [9:55] languages like Move and Ton than in [9:57] June. So they are being trained on these [9:59] new uh less common and famous languages. [10:05] So um to answer the question so can AI [10:08] audit your smart contracts? Now, based [10:11] on these results, [10:12] what I would conclude is no, not on [10:15] their own. Now, if you are interested in [10:18] in doing some sort of a bug bounty, [10:21] well, it's very interesting. The the AI [10:23] can do a lot of good stuff and find [10:26] bugs. Because in bug bounties, what you [10:28] are interested in is basically finding a [10:31] bug in a protocol, [10:33] right? Doesn't have to be a specific [10:35] protocol. It doesn't have to be any [10:37] specific bug. [10:39] Any severe bug would give you money. [10:42] Now, audits [10:43] you don't you specifically want to find [10:46] bugs in this report and all bugs that [10:48] you can find. [10:49] So your goal is to provide [10:52] a security assessment of this protocol [10:54] and like say this is a good posture for [10:56] the protocol to be deployed. [10:58] So that's why relying only on AI agents [11:02] is not is not yet something that is a [11:04] good decision yet. [11:05] But of course, we as security auditors, [11:09] we are using AI in our pipelines to do [11:12] security audits which are is increasing [11:15] a lot our efficiency, increasing a lot [11:17] our [11:18] coverage and [11:20] the the overall performance. [11:25] So [11:26] thank you and this is the results for [11:28] now and I'm happy to take questions. [11:31] [applause] [11:36] I have some time so I'll take some [11:37] questions, right? [11:41] >> Next question. [11:52] >> That that's a great question. So [11:56] basically [11:58] in the pipelines [11:59] you have you guide the LLM to look where [12:03] to look and where to focus more. [12:06] This is what what you get the best [12:08] improvement. Of course, you also add [12:09] tools, [12:11] but you're also seen that [12:14] like pipelines that basically used tools [12:16] and then triage them. But our like heavy [12:19] metal pipelines is basically more than [12:22] that which is make it makes [12:24] people [12:25] make makes LLM redirect on specific [12:27] parts of the code and do multiple rounds [12:29] and do different [12:32] instructions of how to look at this. So [12:35] do you want to look for what types of [12:36] bug? [12:37] So some of them based on like invariance [12:40] that you infer from the protocol and [12:41] then ask the protocol to to to check. [12:43] Now what is [12:45] the recent models and flagship models is [12:47] trying to do is basically do this in [12:48] themselves. So they try to [12:52] do some sub agents create some sub [12:54] agents and give them these instructions. [12:56] But you're still better [12:58] as a human auditor with very high skills [13:00] and experience in [13:01] in auditing [13:03] giving this instruction than AI yet. Now [13:05] they might be more efficient in [13:07] implementing these instructions but [13:09] you're still better giving this [13:10] instruction as a human. That's why you [13:12] get you get this this [13:14] you know [13:15] better performance. [13:17] Yes please. [13:19] >> You were talking about your own order [13:21] reports [13:24] How do you ensure that both in terms of [13:28] say the models including public reports [13:31] in their training [13:32] and maybe even inferring that from [13:35] like public bug fixes? [13:38] >> Yeah. So this is also a very good [13:41] question and we do our best but we don't [13:43] have a guarantee. [13:45] What we do is basically of course we [13:46] remove all the materials that can give [13:49] him access directly like the git history [13:51] and stuff like that. [13:53] But like making sure that the training [13:55] data have not seen our bug is something [13:57] that is [13:58] not very [14:00] not very easy to do. [14:02] Now one suspect we have that the [14:05] additional performance and better [14:06] performance we had is because we have [14:08] trained we have did the experiment [14:10] before. So they trained with the new [14:12] models they've seen this then they are [14:14] giving better performance. That's that [14:15] that might be a limitation and there is. [14:17] And then we have to assess it. This is [14:19] an ongoing work. Some of these results [14:21] are not finished yet. We are doing more [14:22] and more. We're going to continue doing [14:24] it and please subscribe to the to check [14:27] our [14:28] blog post to see how we're going to give [14:31] more and more results about this. And [14:33] yeah, it's it's a limitation. We don't [14:35] We don't know that they are not trained [14:37] on these bugs. Not all of these are [14:39] public reports. So also that's a good [14:42] thing. Not all of them, but some of [14:43] them. [14:45] Yes. [14:49] What what type of false positive are you [14:51] talking about? [14:52] >> False positive how many results [14:58] >> Yes. Now false positive could be like [15:00] invalid bugs or bugs that are flagged as [15:03] severe, but they are not. They are [15:05] likely because there's there's like a [15:08] line that's hard to put there. [15:10] The line would be [15:12] some sometimes they will find a bug, but [15:15] when you assess it, okay, it's fine, but [15:17] it has like very hard assumptions. You [15:19] are assuming that the admin have not [15:21] missed mistake for that bug to appear [15:23] and this is unlikely to happen. So it's [15:25] not that severe or that it's an [15:27] informational. So if these are counted [15:29] as false positive, it's very hard to put [15:31] this line to count the false positive. [15:33] But if we're talking about invalid [15:34] issues like hallucinations or like [15:36] assuming things that are not correct [15:37] about the code, then this is very very [15:40] low. And we have also done benchmarking [15:43] of our benchmarking. So basically [15:45] created hallucinations [15:47] bugs and tried to see how much they can [15:50] agree that they are bugs or something [15:52] like that. So we have done experiments [15:54] on this. We couldn't present everything, [15:55] but in terms of invalid bugs, they're [15:58] unlikely to to report invalid bugs. But [16:01] yeah, informational bugs and stuff like [16:03] that that they are exaggerating, this is [16:06] this is [16:07] possible. [16:08] Exact numbers are not in here. We can [16:11] put some some stuff online [16:13] soon when we have more experiments. [16:16] Okay. [16:17] Thank you a lot.