DSS

DSS 2026 / Day 1 / Amber Hall

Don’t Trust the First Model: A Methodology for Adversarial AI Audits

Christian Vari

12:58 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:01How many security researchers do we have

0:03in the room? Raise your hand, guys. Oh,

0:06great. I see a lot of you. So, from a

0:08security researcher to security

0:10researcher,

0:12please don't trust the first model. So,

0:16you know,

0:17LLM models always try to convince us

0:19that they went through all the code

0:20base, they checked everything, and that

0:23those this is the list of all the

0:25vulnerabilities that are in the code

0:27base.

0:28But the biggest issue that we have here

0:31is that they are overconfident. And at

0:34Ox Security, we developed a methodology

0:37for doing this adversarially. So, how to

0:39have models challenge each other in

0:42order to reach an agreement.

0:46My name is Christian Vari. I'm the the

0:48head of audit operation at Ox Security.

0:50Ox Security is

0:52spotting vulnerabilities since 2017. We

0:56have worked on more than 600

0:58engagements, and we have a pool of more

1:01than 50 senior security researchers.

1:06So,

1:08we usually ask a lot of things to LLMs,

1:10and we got usually good replies. So, we

1:13get a lot of info from them.

1:15But there is one thing that we usually

1:18don't know. We don't have the real

1:20coverage of what they've done, the real

1:22part that they checked. So,

1:25is true that probably they checked

1:28mostly of the things, but we don't have

1:30the observability for being sure about

1:32that. So, what if some files has not

1:35been read, if some parts were not been

1:39examined? What if they just keep it an

1:41entire file?

1:43We're not just thinking about, you know,

1:45smart contracts, but maybe we are

1:47auditing a full node or, you know,

1:51off-chain components that then move

1:53nodes. So, we have big parts, large

1:56amount of code. How can we

1:59be sure about coverage that they have

2:01have checked? And also another thing,

2:05not just, you know, the the file and the

2:07code base, but also the claims. So have

2:08they challenged really all the potential

2:11things that are in the code base?

2:13Have they challenged the assumptions?

2:16We usually just by, you know, running

2:19LLM or a model or something like that,

2:22we're usually not really aware of that.

2:25So we came up with, you know,

2:28an harness, a workflow. We have

2:30implemented it and its name is Hive, but

2:33in this talk I will just

2:35tell you the methodology. I mean, you

2:37can apply the same methodology.

2:39We

2:41define it this in five steps actually.

2:44So the first one is the scope. So

2:46as a security researcher is what we

2:48usually do. We prepare a scope, okay? So

2:51we have the commit hash of the code base

2:54that we will review. We have, you know,

2:56the architecture, we have the

2:58boundaries, you know, the the the

3:01preconditions and all the

3:03issues or, you know,

3:05things that are already known to the

3:07client. Then we have for each of this

3:10block a gate. So for the scope we have

3:12the gate that we know the scope. Then we

3:15have discover and this part actually is

3:18where we use a multiple amount of

3:20models. Usually we

3:23try to run all of them, so from

3:25different providers and different type

3:27of models in order to check the code

3:29base and spot all the possible

3:31vulnerabilities. The important thing for

3:33this is that at the end of this part of

3:35the process we have some very specified

3:40candidate of vulnerabilities. Then we

3:43next to the the gate open and we move to

3:46the next step that is actually the the

3:48duplication. And we

3:51actually perform some some execution of

3:54the models in order to get a single

3:58a single candidate. Then we do the

4:00validation. I'll explain you why that is

4:02one of the most important thing and we

4:04then move to the the human part because

4:07we believe that at the end of the

4:08process we still need

4:10human reviewer and this is just, you

4:12know, a suggestion, an a testable

4:14hypothesis of what the vulnerability

4:16will be.

4:18So that's what I was saying before. We

4:20are creating the scope but our tool

4:23should be able to ensure that we check

4:25all the all the files, all the paths and

4:28check all the potential precondition and

4:31and and also

4:34the external dependencies where

4:37something is built on.

4:39So for example here in the

4:41in in this in this matrix we are just

4:44checking, you know, that all the models

4:46checked all the file and this is

4:47important because sometimes we have a

4:50global coverage on all the models that

4:52are working on that but we are not

4:54ensuring that the each model have 100%

4:58of the coverage.

5:01Oh.

5:03Another important thing is that models

5:05should not be biased and this is really

5:08line with what we usually do with our

5:10human auditors. So at Ox Security we

5:13have our blinded approach where each

5:15auditor actually go through the code

5:16base

5:18independently, so without sharing

5:19insight with the other auditors. And we

5:22are trying to do the same thing applied

5:23to models. So the models have a shared

5:25input where they have, you know, the

5:27code base, the revision, the

5:29architecture, all this kind of stuff.

5:31And then also we keep separate all their

5:34work. So the candidate findings, the the

5:37path that the models went through in

5:39order to find the the vulnerabilities.

5:43Then we start with the discovery that is

5:45executed by multiple models and is

5:47actually performing two step. So we have

5:49a blind pass where actually the models

5:53go through the

5:54most known, you know, vulnerabilities

5:56glasses like I don't know, overflow,

5:59arithmetic, all this kind of stuff. And

6:01then we have an informed pass. So in all

6:03the the past engagement, we got a lot of

6:06knowledge about

6:08vulnerabilities and potential issues.

6:11So, we created a a dataset for that and

6:15we used that in the informed pass. And

6:17we actually execute both. The informed

6:19pass is also very specific depending on

6:21the on the technology that we are

6:24auditing.

6:26Another important thing is that at the

6:28end of the discovery phase, we have the

6:31claim that is testable and that is well

6:34defined. So, we have to know who is the

6:36attacker, so their role and his

6:38capabilities. What are the

6:39preconditions? So, probably

6:42you know

6:44that type of attacker have a

6:45precondition and depending on the

6:46precondition depends also the severity.

6:49The boundary, the invariance, if there

6:52are scoped invariance, actually it

6:54should be checked. The location

6:56obviously, so the code and the related

6:58code and the provenance, so which model

7:01found it and which phase. So, the

7:02important thing for us is keeping

7:05an audit log actually of what happened,

7:07who found something and the path that

7:09has been executed. This is very

7:11important especially on very very large

7:12code bases where we have, I don't know,

7:14a node that is depending on a big SDK

7:18and actually we have to scope the the

7:21whole path in order to pass it correctly

7:24to the model and also, you know,

7:26have

7:27a feasible type of cost because, you

7:30know, tokens are not free.

7:34Another part that is important, so after

7:37the the the discovery part, we have this

7:39list of of findings, potential findings.

7:43We have to duplicate them. And, you

7:45know,

7:46as a security researcher, everyone has a

7:48way for doing the duplication, okay?

7:52It's a very opinionated thing. And in

7:54our opinion, the best way is with the

7:56root cause, okay? So, here for example,

7:58we have two claims but they have the the

8:00same root cause and the same fix fix the

8:02same thing. And the model should be able

8:05also to separate the things that that

8:08makes each claim

8:10unique. So, for example, here it should

8:12be able to

8:14separate the part that are in common and

8:16merge them into just one and then also

8:19create another findings with the

8:20remaining part.

8:24It's not working.

8:27Okay. Well,

8:32just trying.

8:33By the way, even if it's not moving.

8:36Okay. Yeah, after that we have the

8:38validation. This is the most important

8:40part because

8:42actually we use this as hypothesis for

8:45the security researcher and we don't

8:47want to waste the time of security

8:49researcher. So, it's true that we have a

8:50lot of harnesses for generating

8:53findings,

8:54but there are a lot of false positive

8:56and bringing a lot of false positive to

8:59security researcher is actually a waste

9:00of time. You know, you have 200

9:02potential findings and the security

9:04researcher have to spend their time on

9:06double checking all these AI crap

9:09instead of checking anti-code. And this

9:11is, you know, is something that we

9:14cannot deal with. So, the idea here is

9:15to rerun all the the claims that have

9:19been found adversarially through other

9:21models. So,

9:23we create a fresh context with this with

9:25all the data that is required, all the

9:27dependencies and all the the the the

9:30architecture part that is needed and we

9:32actually let those new models try to

9:35execute them and validate it.

9:39Also, it's very important especially in

9:41large code base like I don't know

9:43blockchain node or this kind of stuff to

9:45give all the the the path to them, so

9:48the the the dependencies. So, imagine a

9:50big SDK or this kind of stuff. So, we

9:53have to give all the context needed

9:55trying to scope it to make, you know,

9:57the less amount to spend the less amount

9:59of tokens and this will be able actually

10:03our models to

10:06identify where a path is actually

10:08something that is reachable or not, for

10:10example, because it's very common, you

10:12know, with models when use them, that

10:13say, okay, we have this kind of

10:15vulnerabilities,

10:16just go through it and you can execute

10:18it. But then it's not really true

10:20because it may be in a dependency of,

10:22you know, depending on the on the

10:25execution,

10:27there is a difference because maybe it's

10:29gated, so like in this example.

10:32And one another thing that is important

10:35in our process is confidence. So I mean,

10:39we

10:40have traced all the the

10:43all the path that our model our

10:46methodology has going through, so we

10:48know which model found something and at

10:50which stage. And we have all the audit

10:52log of this. Having this, we have this

10:55part of the model that actually create a

10:57summary of that

10:59and also depending on a threshold, put

11:02the findings in a different way. And the

11:05important thing is that we are able then

11:07to pass all this information to the

11:08auditors. So if we have some models that

11:12are supporting it, some model that are

11:14refusing it, and you know, they're not

11:16really sure about that, actually we gave

11:19all their thinking to the security

11:22researcher, the human one, the the real

11:23one, in order to be able then to to

11:26judge this.

11:29At the end of the process, we have this

11:31last part that is

11:33you know, a sort of health health check.

11:36So sometimes, you know, our models, you

11:39know, we use all of them, can

11:43you know, give no verdict or giving no

11:45candidates or just not read the scope.

11:48So it's very important that your harness

11:51is able to identify that, to be aware of

11:53that because we are tracking it and then

11:56be able to rerun it. So if, I don't

11:58know, a part of the code base has not

12:00been checked or if a part of the

12:02architecture is still not checked or

12:06we are unsure about it, we have to

12:08iterate about that or at least we have

12:10to let the auditors, the human the human

12:12auditors be aware of it in order then

12:16that they can then just go through it

12:18manually and check it.

12:21So this is our model. Actually we

12:23implemented it through our tool that is

12:25called Hive and you can actually get

12:28more information on our website that is

12:30oxsecurity.io.

12:32Um

12:34um our tool is something that you can

12:38use if you reach us to us, but the

12:40methodology is something that you can

12:42they can use as you want because is

12:44something that is not really dependent

12:46on models or technology. And that's all.

12:49If you want to reach out for for trying

12:52it or just have any question, I'll be

12:54here. Thank you.

Source and accuracy details

Source export SHA256: 4c75d3f91c3ba7c842bc174543489f1688987480c08f1d0e4bfda9a7bbed6666