DSS

DSS 2026 / Day 2 / Pearl Hall

AI-Native Auditing in Five Minutes: Questions, Evidence, Coverage

nisedo

4:45 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:01Hi everyone.

0:02[clears throat] Um, Nicedo from Trail of

0:04Bits.

0:05Um,

0:07generally AI have become so good now

0:09that we all know we should use them

0:11during our audits.

0:13Uh, thankfully at Trail of Bits we have

0:14like almost unlimited tokens, so I burn

0:17I burnt a hell lot of tokens to know

0:19what works and what doesn't. And I will

0:22be presenting five techniques that I use

0:23heavily in all of our audit right now

0:25and are used as well by my um

0:27colleagues.

0:29First one is to start every of your

0:32agents from a threatmodel.md. This

0:34threat model is a basic markdown file at

0:37the root of your repo. Um, it should

0:38contain at least these four items.

0:41Um, it will first of all help your agent

0:43focus on what matters truly for your

0:45system. No more malicious admin can

0:47steal all the funds. Uh, this kind of

0:49findings are are really um not useful.

0:52And um this should evolve as well as

0:54your code. If you are a developer using

0:56this technique, of course, uh be sure to

0:58own this file and to make it evolve

1:01along with your uh codebase.

1:02My recommendation is to use the Code X

1:04Security Threat Model skill, which is

1:06part of the Code X Security um

1:09main plugin. Uh, it's a really good

1:11starting point, but as I said, you

1:12should really be honing this um file.

1:15Technique number two is to give your

1:17agents a narrower scope compared to the

1:20usual find bugs in this codebase with a

1:22um

1:23God knows how many line of code.

1:26Um, we divided all the system we are

1:29auditing at Trail of Bits into three

1:30different um division style, by file, by

1:33feature, by component. My favorite one

1:35is by file, but my colleagues have great

1:38results with by feature.

1:39The goal obviously is not to ask your

1:42agent to focus on one file in an

1:44isolated manner, but instead to review

1:47the entire system through a different

1:49scope every single time and through a

1:51different angle, sorry, every single

1:52time. So you will reach a great coverage

1:55with different angles. And obviously at

1:57the end you should have read and you

1:59should have on all your um

2:01all the scope of your audit.

2:05Technique number three is to run several

2:07models on the same scope. This is by far

2:09my favorite one because we realize that

2:11all model family have their own

2:13strengths and weaknesses and blind

2:14spots. And so it happened many times

2:17that the latest SOTA model from GPT was

2:20missing something that Claude 4.8 was

2:23finding, which is really weird. So

2:26obviously there is um a weakness to this

2:29technique is that it costs a lot of

2:30money. So be aware of your

2:33token budget. But if you can

2:35be sure to run at least the latest GPT

2:38and Claude and even more some open

2:40weight model because they um perform

2:43very well on certain type of of bug

2:45classes. So you should give the same

2:47scope, the same threat model, different

2:49families, and you will by far expand

2:51your coverage of the type of findings

2:53you could be finding in your audit.

2:56Technique number four is to consider

2:59every single valid findings as a

3:01starting point and not as the ending

3:02point. What I mean by that

3:05find a low, a medium, or a high and feed

3:07it to another agent with the goal to

3:10expand it to a higher severity finding

3:12by chaining multiple lows and medium for

3:15example or by increasing the maximum

3:18impact or by um reducing the constraint

3:21that the attacker has to have in order

3:23to make this valid finding.

3:25This work less now than with the

3:28previous model given that the model have

3:29become very good at not stopping at the

3:31first finding they um they they

3:33encounter and they usually push a little

3:34bit forward. But when it works and you

3:37end up with a critical starting from a

3:38low because you chain it with a few

3:40other lows and medium, trust me, it's

3:42amazing.

3:44Technique five and the last one, do not

3:47trust the coverage of your agents.

3:49Agents will tell you that they read file

3:51XYZ or that they cover the feature XYZ

3:54but the truth is they might have skipped

3:56some functions because they consider it

3:58not critical. They might have skipped

4:00some line of code. So we built the tool,

4:04agent cov, uh, which is uh, freely

4:06available and open source at Trail of

4:08Bits, uh, on our GitHub, which allow you

4:11to actually make sure of what your agent

4:13had covered. So, you could actually see

4:15what it has read, what it has missed,

4:17and then you can use some

4:19focused prompt on the gaps, uh, in order

4:22to make sure you actually reach full

4:24coverage on your audit.

4:26So, to sum up, threat model is very

4:29useful, narrow scope instead of broad

4:31coverage, uh, several model, but be

4:34aware of your token consumption,

4:36escalate all your valid findings and

4:39check what your agent actually read and

4:41do not trust it.

4:43Thank you.

Source and accuracy details

Source export SHA256: 3d89b07c971c430e09a9dd224e835475b785bb567d808ecf38d08b092be3b0f3