DSS 2026 / Day 2 / Pearl Hall
AI-Native Auditing in Five Minutes: Questions, Evidence, Coverage
nisedo
Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.
Transcript
0:01Hi everyone.
0:02[clears throat] Um, Nicedo from Trail of
0:04Bits.
0:05Um,
0:07generally AI have become so good now
0:09that we all know we should use them
0:11during our audits.
0:13Uh, thankfully at Trail of Bits we have
0:14like almost unlimited tokens, so I burn
0:17I burnt a hell lot of tokens to know
0:19what works and what doesn't. And I will
0:22be presenting five techniques that I use
0:23heavily in all of our audit right now
0:25and are used as well by my um
0:27colleagues.
0:29First one is to start every of your
0:32agents from a threatmodel.md. This
0:34threat model is a basic markdown file at
0:37the root of your repo. Um, it should
0:38contain at least these four items.
0:41Um, it will first of all help your agent
0:43focus on what matters truly for your
0:45system. No more malicious admin can
0:47steal all the funds. Uh, this kind of
0:49findings are are really um not useful.
0:52And um this should evolve as well as
0:54your code. If you are a developer using
0:56this technique, of course, uh be sure to
0:58own this file and to make it evolve
1:01along with your uh codebase.
1:02My recommendation is to use the Code X
1:04Security Threat Model skill, which is
1:06part of the Code X Security um
1:09main plugin. Uh, it's a really good
1:11starting point, but as I said, you
1:12should really be honing this um file.
1:15Technique number two is to give your
1:17agents a narrower scope compared to the
1:20usual find bugs in this codebase with a
1:22um
1:23God knows how many line of code.
1:26Um, we divided all the system we are
1:29auditing at Trail of Bits into three
1:30different um division style, by file, by
1:33feature, by component. My favorite one
1:35is by file, but my colleagues have great
1:38results with by feature.
1:39The goal obviously is not to ask your
1:42agent to focus on one file in an
1:44isolated manner, but instead to review
1:47the entire system through a different
1:49scope every single time and through a
1:51different angle, sorry, every single
1:52time. So you will reach a great coverage
1:55with different angles. And obviously at
1:57the end you should have read and you
1:59should have on all your um
2:01all the scope of your audit.
2:05Technique number three is to run several
2:07models on the same scope. This is by far
2:09my favorite one because we realize that
2:11all model family have their own
2:13strengths and weaknesses and blind
2:14spots. And so it happened many times
2:17that the latest SOTA model from GPT was
2:20missing something that Claude 4.8 was
2:23finding, which is really weird. So
2:26obviously there is um a weakness to this
2:29technique is that it costs a lot of
2:30money. So be aware of your
2:33token budget. But if you can
2:35be sure to run at least the latest GPT
2:38and Claude and even more some open
2:40weight model because they um perform
2:43very well on certain type of of bug
2:45classes. So you should give the same
2:47scope, the same threat model, different
2:49families, and you will by far expand
2:51your coverage of the type of findings
2:53you could be finding in your audit.
2:56Technique number four is to consider
2:59every single valid findings as a
3:01starting point and not as the ending
3:02point. What I mean by that
3:05find a low, a medium, or a high and feed
3:07it to another agent with the goal to
3:10expand it to a higher severity finding
3:12by chaining multiple lows and medium for
3:15example or by increasing the maximum
3:18impact or by um reducing the constraint
3:21that the attacker has to have in order
3:23to make this valid finding.
3:25This work less now than with the
3:28previous model given that the model have
3:29become very good at not stopping at the
3:31first finding they um they they
3:33encounter and they usually push a little
3:34bit forward. But when it works and you
3:37end up with a critical starting from a
3:38low because you chain it with a few
3:40other lows and medium, trust me, it's
3:42amazing.
3:44Technique five and the last one, do not
3:47trust the coverage of your agents.
3:49Agents will tell you that they read file
3:51XYZ or that they cover the feature XYZ
3:54but the truth is they might have skipped
3:56some functions because they consider it
3:58not critical. They might have skipped
4:00some line of code. So we built the tool,
4:04agent cov, uh, which is uh, freely
4:06available and open source at Trail of
4:08Bits, uh, on our GitHub, which allow you
4:11to actually make sure of what your agent
4:13had covered. So, you could actually see
4:15what it has read, what it has missed,
4:17and then you can use some
4:19focused prompt on the gaps, uh, in order
4:22to make sure you actually reach full
4:24coverage on your audit.
4:26So, to sum up, threat model is very
4:29useful, narrow scope instead of broad
4:31coverage, uh, several model, but be
4:34aware of your token consumption,
4:36escalate all your valid findings and
4:39check what your agent actually read and
4:41do not trust it.
4:43Thank you.
Source and accuracy details
- Auto-generated YouTube captions; native source language tag preserved; not human-verified against audio.
- Names, organizations, technical terms, code and numbers may be misrecognized.
- Source wording and disfluencies are preserved; no editorial corrections were applied.
- Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.
- A full caption-track export does not establish that every spoken word is represented.
Source export SHA256: 3d89b07c971c430e09a9dd224e835475b785bb567d808ecf38d08b092be3b0f3