DSS

DSS 2026 / Day 1 / Pearl Hall

Run Early, Disclose Late: When to Unleash Your AI Auditor

Nicolas Garcia

6:28 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:00Okay, guess now it's fine.

0:04Okay, so at Open Zeppelin, we've been

0:07using an AI auditor in pretty much all

0:11of our audits because at this point,

0:14it's undeniable that you can find really

0:17good bugs using LLMs.

0:20Please raise your hand if you also use

0:22some sort of AI auditor.

0:26Okay, many of you. Nice.

0:29But the question is

0:32when do you run it? At the beginning or

0:35at the end of the engagement. So let's

0:38start with the end.

0:42So imagine a three-week audit. You

0:45already finished the manual review.

0:48So you have two days left. So you

0:51decided to run the tool.

0:53Boom. 206 issues, right?

0:58Um, of course, what you do is you use a

1:01set of agents to try to triage the

1:04issues, but uh, some are going to be

1:09dismissed, some are going to be

1:10confirmed, but there are some others

1:12that are going to be inconclusive. No

1:14verdict, right? And you must verify

1:18those and also the ones with verdict

1:20because at the end of the day, you're

1:21the one who make the last call.

1:27Okay, so the deadline hit and there are

1:30still 47 issues that you haven't review.

1:33So this is of course a pretty unwanted

1:36scenario, right? So I guess you may be

1:39thinking, okay, let's run it uh at the

1:42beginning on day one.

1:45So there is a couple issues with that

1:48approach. The first one is that triaging

1:50issues on day one is pretty much slow

1:53because you don't know the code yet,

1:55right? So, it's going to be slow. Of

1:59course, you're going to learn through

2:01the issues, but only a small part of it

2:05because you're not getting the full

2:06picture.

2:08And there is another issue which is I

2:11guess worse than the first one and it's

2:14the fact that it's going to frame the

2:16way you audit because you're going to

2:18tackle the code through the lenses of

2:21the AI auditor right so what about the

2:24issues that you could find yourself that

2:26the agent will not found right you won't

2:30never know because you don't measure

2:34that right so I noticed that there's

2:38some fields that they actually like

2:41study this particular behavior and is

2:46basically flight simulators, right?

2:49Pilots that use um like some sort of

2:53warning system, automated warning system

2:57like they compare two groups, one with

3:00the automated system and the other ones

3:03without it. when I issue that hadn't had

3:07a warning pop up, the ones that used the

3:11warning system were not able to spot it

3:16like the other group was able to do it,

3:18right? So, yeah, it was pretty

3:21interesting. And also

3:24there was another uh interesting

3:28research about radiologist, right? They

3:31were told to look for nodles in some

3:34long images. And in the experiment, the

3:37researcher on the last scan add a

3:40gorilla that was like 48 times bigger

3:42than the nodles. And for some reason, 20

3:45out of 24 were not able to spot the

3:48gorilla even though like the eye

3:49tracking

3:51system they use noticed that they like

3:56watch right of it. Right? So the thing

3:59is that they were asked to spot nautles

4:02not gorillas right and this is what

4:05happened when you have a list of

4:06findings right you can miss the gorilla

4:10and in this case the gorilla is the

4:11issue that you're able to spot that the

4:14llm will not spot right so yes the main

4:18thing here is that you don't want to

4:20bias the way you tackle the code because

4:24if you you can miss the actual

4:28original issues, let's say.

4:32Okay. Yes. Basically, so you take your

4:39manual approach, write your issues down

4:42and then you continue to the triaging,

4:46right? So this is like the order that I

4:48suggest.

4:50So yes, the question is beginning or end

4:53is actually both, right? you run of

4:56course at the beginning to have a sense

4:58of the amount of issues that you need to

5:00trash and allocate some time for it with

5:03some buffer and

5:06read uh late but when I'm saying late is

5:10not exactly that you wait until the last

5:13state of the audit because if you are

5:16taking into consideration a really long

5:18audit the things that you look at the

5:21beginning are not going to be as fresh

5:24at at the end of the engagement. So what

5:26you could do okay is pretty much split

5:30the approach right by components as the

5:33LLMs to like sort the findings and then

5:39you can do component by component pretty

5:43much. So this is a quick formula to know

5:47how much time to allocate. It's pretty

5:49average. Like the ones with no verdict

5:53times your average time for those issues

5:57and the ones with verdicts plus like a

6:00quick check because those already have

6:01evidence from the LLMs

6:04because remember that we triage them.

6:08And yes, this is what I told you. Do not

6:11wait until the end.

6:14And that is pretty much it. Hopefully,

6:18next time you find 200 issues, uh, it's

6:22not going to be on the last day of the

6:24audit. Thank you so much.

Source and accuracy details

Source export SHA256: 7af65fdec4ae8ec77361096adb30654e32d03714b892a17f941b4c7550551f1a