DSS 2026 / Day 1 / Pearl Hall
Benchmarking Frontier Models: Which One Is the Best EVM Auditor?
Josef Gattermayer
Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.
Transcript
0:00So hello again.
0:02My name is Joseph and this is the longer
0:04talk, and we'll speak about
0:09about
0:10benchmarking of AI models.
0:13So let's get started.
0:16Um
0:17yeah, the timer.
0:20Okay, I'm starting.
0:22So yes,
0:23I'm co-founder of ACK free and today we
0:28will look at some benchmarks that we run
0:30in the past and also I did run the
0:32latest benchmarks overnight, so it is
0:34very fresh data that we'll present here
0:37today.
0:39We at ACK free, we are developing tools,
0:41we are doing audits, we are doing
0:43research and I'm also a stand professor
0:46Czech Technical University in Prague and
0:49we are happy we could be working with
0:50some of the industry leaders as for
0:53example Lydo here. Thank you guys.
0:56And so let's get started.
0:59The question is how to evaluate LLM
1:01auditors. It's not that easy
1:05because the
1:06web two security guys, they have
1:10more easy
1:12way of
1:14measuring if some attack succeeded
1:17because you can for example
1:19set a goal to the agent break the system
1:23or like get into this database. And when
1:25you set the goal, the agent is just
1:27working on that and that's all you need
1:29to settle. But our goal is open. We
1:32don't know how many bugs are in the
1:35contracts. So basically we cannot set at
1:38as easy goal. So that's the first
1:40problem. The second problem is of course
1:43data because when we have a
1:46when we have a project
1:48where we know the issues because it was
1:50audited,
1:51it's probably also public, which means
1:55the LLMs could have learned on it and
1:57now we are not evaluating how they
1:59perform in the future, but we are
2:02evaluating how they performed in the
2:03past. So it's very hard to finding some
2:06uh dataset that was not public, but was
2:09audited. We have some of the datasets,
2:11but it's of course a little bit limited.
2:13And of course we need to care about zero
2:14data retention when we are doing stuff
2:16like that, so we cannot just use any
2:18Chinese API, you know, that we don't
2:20trust.
2:22Uh
2:22so the evaluation part one is that we
2:25looked at individual models. Uh this is
2:28an older benchmark.
2:29Uh we took
2:31these uh
2:33these LLMs. We took just a very easy
2:36workflow with Codex and we just run uh
2:40that with some prompt, I don't know,
2:41find all bugs, make no mistakes
2:44uh over a codebase that was consisted of
2:47five unpublished audits. And we are
2:51looking for the recall because we knew
2:53how many uh
2:54criticals and highs are there. These are
2:57the only ones that we tracked. There are
2:5840 discovered by sorry, 36 discovered by
3:01human and four with the later AI
3:04findings. So the total recall was 40.
3:07And this is how the single models
3:10perform. So the human auditors
3:13discovered uh
3:15in the first revision, then then it was
3:16more 32 findings uh
3:20and the most performant
3:22uh LLM is Opus five that discovered 35
3:27and then it went slowly down. But this
3:31is not the chart that you should be
3:33looking at because it doesn't
3:35basically help you to prevent
3:38uh from the hacks. So more interesting
3:40is this chart where we looked at uh
3:44all the bugs. This is this axis
3:46basically uh that are present in the
3:49dataset. We looked at all the models and
3:52the humans and we looked who discovered
3:55what. And of course like the Opus
3:58discovered most of the issues, but is it
4:01sufficient for you that in your protocol
4:03you have something that discovers most
4:05of the issues? No. You need something
4:07that discovers all the issues. And here
4:10I must shout out to
4:14Kimi K Free because this guy
4:17the overall performance is very poor
4:19poor, but what he did is that this one
4:22issue
4:23discovered no one else but just this
4:25model. And this is of course like very
4:27interesting that that it's able to find
4:28a unique issue
4:30which was high, not a critical, but only
4:34this model discovered that it not even
4:35humans and not even Opus or someone
4:39else.
4:40So what does it tell us this benchmark?
4:44It helped us to form like two
4:46hypotheses.
4:48One of them is
4:50what happens if we just run the
4:52benchmark multiple times?
4:54And the second one is like maybe we need
4:57some kind of like mixture of models to
5:00be able to find the unique findings
5:02because this is what we are
5:04hunting for as well.
5:06So then we did a second benchmark.
5:09For this second benchmark
5:11the inspiration was this hack. I
5:13don't know if you're familiar with that,
5:14but it was like I think a month ago
5:18quite headline news that
5:21that there was an like internal
5:24experiment within OpenAI
5:27where they have put some
5:30goal to the agents.
5:33And
5:35what they did instead was that they
5:37hacked out of the sandbox and tried to
5:41or succeeded in hacking a hugging face
5:44which was the benchmark they were
5:46supposed to out perform. And
5:49the interesting thing about this is how
5:51the agents did it. And
5:54when you look at the report basically
5:57they the agents found a way how to
5:59communicate between each other. They
6:00were exchanging messages and they were
6:02like building on each other's
6:04findings and they even called themselves
6:08like a collective and
6:10it was pretty scary,
6:12but I like that. And so we said okay,
6:15maybe we could replicate something like
6:17this and see how how it performs.
6:20So we did this quick
6:23tool. We call it swarm.
6:25We just open sourced it yesterday, but
6:27it's still like quite early internal
6:29tool. And it's kind of like tool that
6:31help us to orchestrate us and experiment
6:33like this.
6:35So basically you can put any prompt, you
6:38can put any model, you can put any
6:39workflow.
6:41Every
6:43agent has is dedicated container
6:46and what works here like
6:47deterministically is that these
6:49containers with agents are sharing the
6:51evidence between each other and it's
6:53also running on subscriptions, which
6:55means like it doesn't
6:57ruin us.
6:58And with this tool we looked again at
7:01some benchmarking.
7:03And
7:06this time we took a smaller
7:09set, but it was just like
7:11one code base, but very, let's say big
7:13and complicated code base.
7:15And we took six models and
7:18because it is more recent, so I'll will
7:22just point out that we were using the
7:24latest DeepSeek Flash model.
7:26And this time the workflow
7:30was that the swarm, that's the our tool,
7:33was running inside of that the harnesses
7:36that are native to each agent. So we are
7:38running three harnesses, those Cloud
7:39Cod, Codex and DeepSeek harness. And
7:43with this we are looking at the data
7:45set.
7:46In the human baseline there were
7:49four critical and high issues. So that
7:51this was the 100% recall.
7:54And this is how the experiment looked
7:57like.
7:58This is the time window that we
7:59dedicated to this
8:02phase and this is five hours time
8:05window. So basically each this box is
8:08one how we call it wave, and the wave is
8:11like one container, one LLM, and simple
8:15prompt like find all bugs, make no
8:17mistakes. When it finishes, it just runs
8:19again. But between these waves, it
8:21shares the outputs with all the other
8:23guys. So basically
8:25between the waves is a synchronization,
8:27so they are sharing the findings so they
8:29can build on on the findings. And yeah,
8:32we just look what happens.
8:34Also here quite interesting is that you
8:36can see something like
8:38kind of like performance of the of the
8:40models that you can see that for example
8:42with
8:43Luna
8:44we finished 35 waves.
8:47Opus finished only six waves, but it
8:50basically doesn't tell anything how
8:51efficient the model is. It just tells
8:52like how fast it claimed that it has
8:56scanned all the codebase.
8:58And here's the result. So first of all,
9:02these are the findings when we looked
9:04just at the wave zero. So this is like
9:06apples to apples to the previous
9:08experiment. So here are the humans that
9:10discovered four issues.
9:12Opus discovered three. DeepSeek, which
9:14is quite a surprise, discovered three as
9:16well. And the rest is
9:19at two issues. These are the
9:22wave zero.
9:25Then we again look at this, which is the
9:28uniqueness of findings. So here we can
9:31see that the
9:33H2 issues discovered only by humans, and
9:37H H1 issue, which is kind of like
9:40problematic, was in the first wave
9:42discovered only by Opus and DeepSeek.
9:48But this is not what this experiment is
9:49about. This is just like the first wave.
9:52So now we look how it performed
9:55next. So when we run
9:58the more waves
9:59we we did run up to 32 waves of Luna,
10:03but we can see that the major
10:05discoveries happened within the first
10:08three waves. [snorts] And we can see the
10:10for example if Opus it the next runs
10:13didn't bring any value.
10:16With
10:18DeepSeek also
10:20not but for example
10:23GPT six Soul which is the daybreak model
10:26it was productive
10:29in three waves which means basically
10:32this tells when you want to max out the
10:34performance of GPT six Soul you have to
10:36run the same prompt three times
10:39after each other.
10:41And
10:43the next interesting thing is of course
10:47who first discovered the
10:50harder findings. So here we can see that
10:52it wave zero basically the discovery
10:55looks like this and then the single
10:57models were sharing the knowledge but
10:59the question is like who was the first
11:01model that discovered the
11:05harder findings and in this case it was
11:09in wave one
11:11this guy which is the GPT six Soul. So
11:15even if this guy didn't have the highest
11:18recall in the first wave it discovered
11:20the unique issue that Opus missed and
11:22they discovered it in the second wave.
11:25And then as they shared shared the
11:28findings everyone did catch up and in
11:30the end everyone had everything.
11:34And
11:36then the question is like
11:38if the theory that I presented if the
11:40sharing of information between the waves
11:43helped. So in this case it did not bring
11:46any new finding. So we cannot tell for
11:50sure but
11:52it helped the weaker models basically to
11:55catch up and here are two cases
11:58basically that
11:59helped so GLM discovered something which
12:03then DeepSeek finished as finding and
12:06the same here Kimmy was building on
12:08something that was presented in early
12:11waves which confirmed the findings. So,
12:14on this data set, we cannot say for sure
12:17that the replication of the hugging face
12:20experiment did bring the value, but if
12:23we would not be using the
12:25best two models for these guys, it would
12:28definitely help.
12:31And
12:33yeah, how did it end? So,
12:35the recall after the first wave was 75%.
12:40After the always combined, it was 100%.
12:42So, we can tell that at this
12:44benchmark, we
12:46were able to discover everything by the
12:49AI agent workflow.
12:51Just a comment on this, we did not play
12:54anyhow with the prompt engineering. The
12:55prompt was like very stupid. So, this
12:57was like brute force testing. And on
13:00this experiment, we spent about
13:031 million tokens, so it's not that much,
13:06but it's also not very efficient. It's
13:08kind of like brute force testing.
13:12Yeah, and if you would like to
13:16learn more about it, so here is the
13:18block post where we are comparing the
13:20models. And here is the interesting
13:23open AI experiment. And also, if you
13:26would like to try the tool that we just
13:28open sourced, it's in our GitHub since
13:32yesterday.
13:34And
13:35that's pretty much it. Thank you very
13:37much.
Source and accuracy details
- Names, organizations, technical terms, code and numbers may be misrecognized.
- Source wording and disfluencies are preserved; no editorial corrections were applied.
- Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.
- A full caption-track export does not establish that every spoken word is represented.
- Automatic YouTube captions; not human-verified against the audio.
Source export SHA256: a79e9e696ef3997696d418cad49f0951ffd59bec1b70a2dbe0907f7b2d84be05