DSS 2026 / Day 1 / Amber Hall
Don’t Trust the First Model: A Methodology for Adversarial AI Audits
Christian Vari
Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.
Transcript
0:01How many security researchers do we have
0:03in the room? Raise your hand, guys. Oh,
0:06great. I see a lot of you. So, from a
0:08security researcher to security
0:10researcher,
0:12please don't trust the first model. So,
0:16you know,
0:17LLM models always try to convince us
0:19that they went through all the code
0:20base, they checked everything, and that
0:23those this is the list of all the
0:25vulnerabilities that are in the code
0:27base.
0:28But the biggest issue that we have here
0:31is that they are overconfident. And at
0:34Ox Security, we developed a methodology
0:37for doing this adversarially. So, how to
0:39have models challenge each other in
0:42order to reach an agreement.
0:46My name is Christian Vari. I'm the the
0:48head of audit operation at Ox Security.
0:50Ox Security is
0:52spotting vulnerabilities since 2017. We
0:56have worked on more than 600
0:58engagements, and we have a pool of more
1:01than 50 senior security researchers.
1:06So,
1:08we usually ask a lot of things to LLMs,
1:10and we got usually good replies. So, we
1:13get a lot of info from them.
1:15But there is one thing that we usually
1:18don't know. We don't have the real
1:20coverage of what they've done, the real
1:22part that they checked. So,
1:25is true that probably they checked
1:28mostly of the things, but we don't have
1:30the observability for being sure about
1:32that. So, what if some files has not
1:35been read, if some parts were not been
1:39examined? What if they just keep it an
1:41entire file?
1:43We're not just thinking about, you know,
1:45smart contracts, but maybe we are
1:47auditing a full node or, you know,
1:51off-chain components that then move
1:53nodes. So, we have big parts, large
1:56amount of code. How can we
1:59be sure about coverage that they have
2:01have checked? And also another thing,
2:05not just, you know, the the file and the
2:07code base, but also the claims. So have
2:08they challenged really all the potential
2:11things that are in the code base?
2:13Have they challenged the assumptions?
2:16We usually just by, you know, running
2:19LLM or a model or something like that,
2:22we're usually not really aware of that.
2:25So we came up with, you know,
2:28an harness, a workflow. We have
2:30implemented it and its name is Hive, but
2:33in this talk I will just
2:35tell you the methodology. I mean, you
2:37can apply the same methodology.
2:39We
2:41define it this in five steps actually.
2:44So the first one is the scope. So
2:46as a security researcher is what we
2:48usually do. We prepare a scope, okay? So
2:51we have the commit hash of the code base
2:54that we will review. We have, you know,
2:56the architecture, we have the
2:58boundaries, you know, the the the
3:01preconditions and all the
3:03issues or, you know,
3:05things that are already known to the
3:07client. Then we have for each of this
3:10block a gate. So for the scope we have
3:12the gate that we know the scope. Then we
3:15have discover and this part actually is
3:18where we use a multiple amount of
3:20models. Usually we
3:23try to run all of them, so from
3:25different providers and different type
3:27of models in order to check the code
3:29base and spot all the possible
3:31vulnerabilities. The important thing for
3:33this is that at the end of this part of
3:35the process we have some very specified
3:40candidate of vulnerabilities. Then we
3:43next to the the gate open and we move to
3:46the next step that is actually the the
3:48duplication. And we
3:51actually perform some some execution of
3:54the models in order to get a single
3:58a single candidate. Then we do the
4:00validation. I'll explain you why that is
4:02one of the most important thing and we
4:04then move to the the human part because
4:07we believe that at the end of the
4:08process we still need
4:10human reviewer and this is just, you
4:12know, a suggestion, an a testable
4:14hypothesis of what the vulnerability
4:16will be.
4:18So that's what I was saying before. We
4:20are creating the scope but our tool
4:23should be able to ensure that we check
4:25all the all the files, all the paths and
4:28check all the potential precondition and
4:31and and also
4:34the external dependencies where
4:37something is built on.
4:39So for example here in the
4:41in in this in this matrix we are just
4:44checking, you know, that all the models
4:46checked all the file and this is
4:47important because sometimes we have a
4:50global coverage on all the models that
4:52are working on that but we are not
4:54ensuring that the each model have 100%
4:58of the coverage.
5:01Oh.
5:03Another important thing is that models
5:05should not be biased and this is really
5:08line with what we usually do with our
5:10human auditors. So at Ox Security we
5:13have our blinded approach where each
5:15auditor actually go through the code
5:16base
5:18independently, so without sharing
5:19insight with the other auditors. And we
5:22are trying to do the same thing applied
5:23to models. So the models have a shared
5:25input where they have, you know, the
5:27code base, the revision, the
5:29architecture, all this kind of stuff.
5:31And then also we keep separate all their
5:34work. So the candidate findings, the the
5:37path that the models went through in
5:39order to find the the vulnerabilities.
5:43Then we start with the discovery that is
5:45executed by multiple models and is
5:47actually performing two step. So we have
5:49a blind pass where actually the models
5:53go through the
5:54most known, you know, vulnerabilities
5:56glasses like I don't know, overflow,
5:59arithmetic, all this kind of stuff. And
6:01then we have an informed pass. So in all
6:03the the past engagement, we got a lot of
6:06knowledge about
6:08vulnerabilities and potential issues.
6:11So, we created a a dataset for that and
6:15we used that in the informed pass. And
6:17we actually execute both. The informed
6:19pass is also very specific depending on
6:21the on the technology that we are
6:24auditing.
6:26Another important thing is that at the
6:28end of the discovery phase, we have the
6:31claim that is testable and that is well
6:34defined. So, we have to know who is the
6:36attacker, so their role and his
6:38capabilities. What are the
6:39preconditions? So, probably
6:42you know
6:44that type of attacker have a
6:45precondition and depending on the
6:46precondition depends also the severity.
6:49The boundary, the invariance, if there
6:52are scoped invariance, actually it
6:54should be checked. The location
6:56obviously, so the code and the related
6:58code and the provenance, so which model
7:01found it and which phase. So, the
7:02important thing for us is keeping
7:05an audit log actually of what happened,
7:07who found something and the path that
7:09has been executed. This is very
7:11important especially on very very large
7:12code bases where we have, I don't know,
7:14a node that is depending on a big SDK
7:18and actually we have to scope the the
7:21whole path in order to pass it correctly
7:24to the model and also, you know,
7:26have
7:27a feasible type of cost because, you
7:30know, tokens are not free.
7:34Another part that is important, so after
7:37the the the discovery part, we have this
7:39list of of findings, potential findings.
7:43We have to duplicate them. And, you
7:45know,
7:46as a security researcher, everyone has a
7:48way for doing the duplication, okay?
7:52It's a very opinionated thing. And in
7:54our opinion, the best way is with the
7:56root cause, okay? So, here for example,
7:58we have two claims but they have the the
8:00same root cause and the same fix fix the
8:02same thing. And the model should be able
8:05also to separate the things that that
8:08makes each claim
8:10unique. So, for example, here it should
8:12be able to
8:14separate the part that are in common and
8:16merge them into just one and then also
8:19create another findings with the
8:20remaining part.
8:24It's not working.
8:27Okay. Well,
8:32just trying.
8:33By the way, even if it's not moving.
8:36Okay. Yeah, after that we have the
8:38validation. This is the most important
8:40part because
8:42actually we use this as hypothesis for
8:45the security researcher and we don't
8:47want to waste the time of security
8:49researcher. So, it's true that we have a
8:50lot of harnesses for generating
8:53findings,
8:54but there are a lot of false positive
8:56and bringing a lot of false positive to
8:59security researcher is actually a waste
9:00of time. You know, you have 200
9:02potential findings and the security
9:04researcher have to spend their time on
9:06double checking all these AI crap
9:09instead of checking anti-code. And this
9:11is, you know, is something that we
9:14cannot deal with. So, the idea here is
9:15to rerun all the the claims that have
9:19been found adversarially through other
9:21models. So,
9:23we create a fresh context with this with
9:25all the data that is required, all the
9:27dependencies and all the the the the
9:30architecture part that is needed and we
9:32actually let those new models try to
9:35execute them and validate it.
9:39Also, it's very important especially in
9:41large code base like I don't know
9:43blockchain node or this kind of stuff to
9:45give all the the the path to them, so
9:48the the the dependencies. So, imagine a
9:50big SDK or this kind of stuff. So, we
9:53have to give all the context needed
9:55trying to scope it to make, you know,
9:57the less amount to spend the less amount
9:59of tokens and this will be able actually
10:03our models to
10:06identify where a path is actually
10:08something that is reachable or not, for
10:10example, because it's very common, you
10:12know, with models when use them, that
10:13say, okay, we have this kind of
10:15vulnerabilities,
10:16just go through it and you can execute
10:18it. But then it's not really true
10:20because it may be in a dependency of,
10:22you know, depending on the on the
10:25execution,
10:27there is a difference because maybe it's
10:29gated, so like in this example.
10:32And one another thing that is important
10:35in our process is confidence. So I mean,
10:39we
10:40have traced all the the
10:43all the path that our model our
10:46methodology has going through, so we
10:48know which model found something and at
10:50which stage. And we have all the audit
10:52log of this. Having this, we have this
10:55part of the model that actually create a
10:57summary of that
10:59and also depending on a threshold, put
11:02the findings in a different way. And the
11:05important thing is that we are able then
11:07to pass all this information to the
11:08auditors. So if we have some models that
11:12are supporting it, some model that are
11:14refusing it, and you know, they're not
11:16really sure about that, actually we gave
11:19all their thinking to the security
11:22researcher, the human one, the the real
11:23one, in order to be able then to to
11:26judge this.
11:29At the end of the process, we have this
11:31last part that is
11:33you know, a sort of health health check.
11:36So sometimes, you know, our models, you
11:39know, we use all of them, can
11:43you know, give no verdict or giving no
11:45candidates or just not read the scope.
11:48So it's very important that your harness
11:51is able to identify that, to be aware of
11:53that because we are tracking it and then
11:56be able to rerun it. So if, I don't
11:58know, a part of the code base has not
12:00been checked or if a part of the
12:02architecture is still not checked or
12:06we are unsure about it, we have to
12:08iterate about that or at least we have
12:10to let the auditors, the human the human
12:12auditors be aware of it in order then
12:16that they can then just go through it
12:18manually and check it.
12:21So this is our model. Actually we
12:23implemented it through our tool that is
12:25called Hive and you can actually get
12:28more information on our website that is
12:30oxsecurity.io.
12:32Um
12:34um our tool is something that you can
12:38use if you reach us to us, but the
12:40methodology is something that you can
12:42they can use as you want because is
12:44something that is not really dependent
12:46on models or technology. And that's all.
12:49If you want to reach out for for trying
12:52it or just have any question, I'll be
12:54here. Thank you.
Source and accuracy details
- Names, organizations, technical terms, code and numbers may be misrecognized.
- Source wording and disfluencies are preserved; no editorial corrections were applied.
- Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.
- A full caption-track export does not establish that every spoken word is represented.
- Automatic YouTube captions; not human-verified against the audio.
Source export SHA256: 4c75d3f91c3ba7c842bc174543489f1688987480c08f1d0e4bfda9a7bbed6666