DSS 2026 / Day 1 / Pearl Hall
Run Early, Disclose Late: When to Unleash Your AI Auditor
Nicolas Garcia
Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.
Transcript
0:00Okay, guess now it's fine.
0:04Okay, so at Open Zeppelin, we've been
0:07using an AI auditor in pretty much all
0:11of our audits because at this point,
0:14it's undeniable that you can find really
0:17good bugs using LLMs.
0:20Please raise your hand if you also use
0:22some sort of AI auditor.
0:26Okay, many of you. Nice.
0:29But the question is
0:32when do you run it? At the beginning or
0:35at the end of the engagement. So let's
0:38start with the end.
0:42So imagine a three-week audit. You
0:45already finished the manual review.
0:48So you have two days left. So you
0:51decided to run the tool.
0:53Boom. 206 issues, right?
0:58Um, of course, what you do is you use a
1:01set of agents to try to triage the
1:04issues, but uh, some are going to be
1:09dismissed, some are going to be
1:10confirmed, but there are some others
1:12that are going to be inconclusive. No
1:14verdict, right? And you must verify
1:18those and also the ones with verdict
1:20because at the end of the day, you're
1:21the one who make the last call.
1:27Okay, so the deadline hit and there are
1:30still 47 issues that you haven't review.
1:33So this is of course a pretty unwanted
1:36scenario, right? So I guess you may be
1:39thinking, okay, let's run it uh at the
1:42beginning on day one.
1:45So there is a couple issues with that
1:48approach. The first one is that triaging
1:50issues on day one is pretty much slow
1:53because you don't know the code yet,
1:55right? So, it's going to be slow. Of
1:59course, you're going to learn through
2:01the issues, but only a small part of it
2:05because you're not getting the full
2:06picture.
2:08And there is another issue which is I
2:11guess worse than the first one and it's
2:14the fact that it's going to frame the
2:16way you audit because you're going to
2:18tackle the code through the lenses of
2:21the AI auditor right so what about the
2:24issues that you could find yourself that
2:26the agent will not found right you won't
2:30never know because you don't measure
2:34that right so I noticed that there's
2:38some fields that they actually like
2:41study this particular behavior and is
2:46basically flight simulators, right?
2:49Pilots that use um like some sort of
2:53warning system, automated warning system
2:57like they compare two groups, one with
3:00the automated system and the other ones
3:03without it. when I issue that hadn't had
3:07a warning pop up, the ones that used the
3:11warning system were not able to spot it
3:16like the other group was able to do it,
3:18right? So, yeah, it was pretty
3:21interesting. And also
3:24there was another uh interesting
3:28research about radiologist, right? They
3:31were told to look for nodles in some
3:34long images. And in the experiment, the
3:37researcher on the last scan add a
3:40gorilla that was like 48 times bigger
3:42than the nodles. And for some reason, 20
3:45out of 24 were not able to spot the
3:48gorilla even though like the eye
3:49tracking
3:51system they use noticed that they like
3:56watch right of it. Right? So the thing
3:59is that they were asked to spot nautles
4:02not gorillas right and this is what
4:05happened when you have a list of
4:06findings right you can miss the gorilla
4:10and in this case the gorilla is the
4:11issue that you're able to spot that the
4:14llm will not spot right so yes the main
4:18thing here is that you don't want to
4:20bias the way you tackle the code because
4:24if you you can miss the actual
4:28original issues, let's say.
4:32Okay. Yes. Basically, so you take your
4:39manual approach, write your issues down
4:42and then you continue to the triaging,
4:46right? So this is like the order that I
4:48suggest.
4:50So yes, the question is beginning or end
4:53is actually both, right? you run of
4:56course at the beginning to have a sense
4:58of the amount of issues that you need to
5:00trash and allocate some time for it with
5:03some buffer and
5:06read uh late but when I'm saying late is
5:10not exactly that you wait until the last
5:13state of the audit because if you are
5:16taking into consideration a really long
5:18audit the things that you look at the
5:21beginning are not going to be as fresh
5:24at at the end of the engagement. So what
5:26you could do okay is pretty much split
5:30the approach right by components as the
5:33LLMs to like sort the findings and then
5:39you can do component by component pretty
5:43much. So this is a quick formula to know
5:47how much time to allocate. It's pretty
5:49average. Like the ones with no verdict
5:53times your average time for those issues
5:57and the ones with verdicts plus like a
6:00quick check because those already have
6:01evidence from the LLMs
6:04because remember that we triage them.
6:08And yes, this is what I told you. Do not
6:11wait until the end.
6:14And that is pretty much it. Hopefully,
6:18next time you find 200 issues, uh, it's
6:22not going to be on the last day of the
6:24audit. Thank you so much.
Source and accuracy details
- Names, organizations, technical terms, code and numbers may be misrecognized.
- Source wording and disfluencies are preserved; no editorial corrections were applied.
- Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.
- A full caption-track export does not establish that every spoken word is represented.
- Automatic YouTube captions; not human-verified against the audio.
Source export SHA256: 7af65fdec4ae8ec77361096adb30654e32d03714b892a17f941b4c7550551f1a