DSS 2026 — Run Early, Disclose Late: When to Unleash Your AI Auditor Nicolas Garcia Source video: https://www.youtube.com/watch?v=HPGE_zNsgug Automatic captions — not human-reviewed. Complete spoken-audio coverage is unverified. Source language tag: en (English). Source export SHA256: 7af65fdec4ae8ec77361096adb30654e32d03714b892a17f941b4c7550551f1a [0:00] Okay, guess now it's fine. [0:04] Okay, so at Open Zeppelin, we've been [0:07] using an AI auditor in pretty much all [0:11] of our audits because at this point, [0:14] it's undeniable that you can find really [0:17] good bugs using LLMs. [0:20] Please raise your hand if you also use [0:22] some sort of AI auditor. [0:26] Okay, many of you. Nice. [0:29] But the question is [0:32] when do you run it? At the beginning or [0:35] at the end of the engagement. So let's [0:38] start with the end. [0:42] So imagine a three-week audit. You [0:45] already finished the manual review. [0:48] So you have two days left. So you [0:51] decided to run the tool. [0:53] Boom. 206 issues, right? [0:58] Um, of course, what you do is you use a [1:01] set of agents to try to triage the [1:04] issues, but uh, some are going to be [1:09] dismissed, some are going to be [1:10] confirmed, but there are some others [1:12] that are going to be inconclusive. No [1:14] verdict, right? And you must verify [1:18] those and also the ones with verdict [1:20] because at the end of the day, you're [1:21] the one who make the last call. [1:27] Okay, so the deadline hit and there are [1:30] still 47 issues that you haven't review. [1:33] So this is of course a pretty unwanted [1:36] scenario, right? So I guess you may be [1:39] thinking, okay, let's run it uh at the [1:42] beginning on day one. [1:45] So there is a couple issues with that [1:48] approach. The first one is that triaging [1:50] issues on day one is pretty much slow [1:53] because you don't know the code yet, [1:55] right? So, it's going to be slow. Of [1:59] course, you're going to learn through [2:01] the issues, but only a small part of it [2:05] because you're not getting the full [2:06] picture. [2:08] And there is another issue which is I [2:11] guess worse than the first one and it's [2:14] the fact that it's going to frame the [2:16] way you audit because you're going to [2:18] tackle the code through the lenses of [2:21] the AI auditor right so what about the [2:24] issues that you could find yourself that [2:26] the agent will not found right you won't [2:30] never know because you don't measure [2:34] that right so I noticed that there's [2:38] some fields that they actually like [2:41] study this particular behavior and is [2:46] basically flight simulators, right? [2:49] Pilots that use um like some sort of [2:53] warning system, automated warning system [2:57] like they compare two groups, one with [3:00] the automated system and the other ones [3:03] without it. when I issue that hadn't had [3:07] a warning pop up, the ones that used the [3:11] warning system were not able to spot it [3:16] like the other group was able to do it, [3:18] right? So, yeah, it was pretty [3:21] interesting. And also [3:24] there was another uh interesting [3:28] research about radiologist, right? They [3:31] were told to look for nodles in some [3:34] long images. And in the experiment, the [3:37] researcher on the last scan add a [3:40] gorilla that was like 48 times bigger [3:42] than the nodles. And for some reason, 20 [3:45] out of 24 were not able to spot the [3:48] gorilla even though like the eye [3:49] tracking [3:51] system they use noticed that they like [3:56] watch right of it. Right? So the thing [3:59] is that they were asked to spot nautles [4:02] not gorillas right and this is what [4:05] happened when you have a list of [4:06] findings right you can miss the gorilla [4:10] and in this case the gorilla is the [4:11] issue that you're able to spot that the [4:14] llm will not spot right so yes the main [4:18] thing here is that you don't want to [4:20] bias the way you tackle the code because [4:24] if you you can miss the actual [4:28] original issues, let's say. [4:32] Okay. Yes. Basically, so you take your [4:39] manual approach, write your issues down [4:42] and then you continue to the triaging, [4:46] right? So this is like the order that I [4:48] suggest. [4:50] So yes, the question is beginning or end [4:53] is actually both, right? you run of [4:56] course at the beginning to have a sense [4:58] of the amount of issues that you need to [5:00] trash and allocate some time for it with [5:03] some buffer and [5:06] read uh late but when I'm saying late is [5:10] not exactly that you wait until the last [5:13] state of the audit because if you are [5:16] taking into consideration a really long [5:18] audit the things that you look at the [5:21] beginning are not going to be as fresh [5:24] at at the end of the engagement. So what [5:26] you could do okay is pretty much split [5:30] the approach right by components as the [5:33] LLMs to like sort the findings and then [5:39] you can do component by component pretty [5:43] much. So this is a quick formula to know [5:47] how much time to allocate. It's pretty [5:49] average. Like the ones with no verdict [5:53] times your average time for those issues [5:57] and the ones with verdicts plus like a [6:00] quick check because those already have [6:01] evidence from the LLMs [6:04] because remember that we triage them. [6:08] And yes, this is what I told you. Do not [6:11] wait until the end. [6:14] And that is pretty much it. Hopefully, [6:18] next time you find 200 issues, uh, it's [6:22] not going to be on the last day of the [6:24] audit. Thank you so much.