DSS

DSS 2026 / Day 1 / Pearl Hall

Benchmarking Frontier Models: Which One Is the Best EVM Auditor?

Josef Gattermayer

13:40 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:00So hello again.

0:02My name is Joseph and this is the longer

0:04talk, and we'll speak about

0:09about

0:10benchmarking of AI models.

0:13So let's get started.

0:16Um

0:17yeah, the timer.

0:20Okay, I'm starting.

0:22So yes,

0:23I'm co-founder of ACK free and today we

0:28will look at some benchmarks that we run

0:30in the past and also I did run the

0:32latest benchmarks overnight, so it is

0:34very fresh data that we'll present here

0:37today.

0:39We at ACK free, we are developing tools,

0:41we are doing audits, we are doing

0:43research and I'm also a stand professor

0:46Czech Technical University in Prague and

0:49we are happy we could be working with

0:50some of the industry leaders as for

0:53example Lydo here. Thank you guys.

0:56And so let's get started.

0:59The question is how to evaluate LLM

1:01auditors. It's not that easy

1:05because the

1:06web two security guys, they have

1:10more easy

1:12way of

1:14measuring if some attack succeeded

1:17because you can for example

1:19set a goal to the agent break the system

1:23or like get into this database. And when

1:25you set the goal, the agent is just

1:27working on that and that's all you need

1:29to settle. But our goal is open. We

1:32don't know how many bugs are in the

1:35contracts. So basically we cannot set at

1:38as easy goal. So that's the first

1:40problem. The second problem is of course

1:43data because when we have a

1:46when we have a project

1:48where we know the issues because it was

1:50audited,

1:51it's probably also public, which means

1:55the LLMs could have learned on it and

1:57now we are not evaluating how they

1:59perform in the future, but we are

2:02evaluating how they performed in the

2:03past. So it's very hard to finding some

2:06uh dataset that was not public, but was

2:09audited. We have some of the datasets,

2:11but it's of course a little bit limited.

2:13And of course we need to care about zero

2:14data retention when we are doing stuff

2:16like that, so we cannot just use any

2:18Chinese API, you know, that we don't

2:20trust.

2:22Uh

2:22so the evaluation part one is that we

2:25looked at individual models. Uh this is

2:28an older benchmark.

2:29Uh we took

2:31these uh

2:33these LLMs. We took just a very easy

2:36workflow with Codex and we just run uh

2:40that with some prompt, I don't know,

2:41find all bugs, make no mistakes

2:44uh over a codebase that was consisted of

2:47five unpublished audits. And we are

2:51looking for the recall because we knew

2:53how many uh

2:54criticals and highs are there. These are

2:57the only ones that we tracked. There are

2:5840 discovered by sorry, 36 discovered by

3:01human and four with the later AI

3:04findings. So the total recall was 40.

3:07And this is how the single models

3:10perform. So the human auditors

3:13discovered uh

3:15in the first revision, then then it was

3:16more 32 findings uh

3:20and the most performant

3:22uh LLM is Opus five that discovered 35

3:27and then it went slowly down. But this

3:31is not the chart that you should be

3:33looking at because it doesn't

3:35basically help you to prevent

3:38uh from the hacks. So more interesting

3:40is this chart where we looked at uh

3:44all the bugs. This is this axis

3:46basically uh that are present in the

3:49dataset. We looked at all the models and

3:52the humans and we looked who discovered

3:55what. And of course like the Opus

3:58discovered most of the issues, but is it

4:01sufficient for you that in your protocol

4:03you have something that discovers most

4:05of the issues? No. You need something

4:07that discovers all the issues. And here

4:10I must shout out to

4:14Kimi K Free because this guy

4:17the overall performance is very poor

4:19poor, but what he did is that this one

4:22issue

4:23discovered no one else but just this

4:25model. And this is of course like very

4:27interesting that that it's able to find

4:28a unique issue

4:30which was high, not a critical, but only

4:34this model discovered that it not even

4:35humans and not even Opus or someone

4:39else.

4:40So what does it tell us this benchmark?

4:44It helped us to form like two

4:46hypotheses.

4:48One of them is

4:50what happens if we just run the

4:52benchmark multiple times?

4:54And the second one is like maybe we need

4:57some kind of like mixture of models to

5:00be able to find the unique findings

5:02because this is what we are

5:04hunting for as well.

5:06So then we did a second benchmark.

5:09For this second benchmark

5:11the inspiration was this hack. I

5:13don't know if you're familiar with that,

5:14but it was like I think a month ago

5:18quite headline news that

5:21that there was an like internal

5:24experiment within OpenAI

5:27where they have put some

5:30goal to the agents.

5:33And

5:35what they did instead was that they

5:37hacked out of the sandbox and tried to

5:41or succeeded in hacking a hugging face

5:44which was the benchmark they were

5:46supposed to out perform. And

5:49the interesting thing about this is how

5:51the agents did it. And

5:54when you look at the report basically

5:57they the agents found a way how to

5:59communicate between each other. They

6:00were exchanging messages and they were

6:02like building on each other's

6:04findings and they even called themselves

6:08like a collective and

6:10it was pretty scary,

6:12but I like that. And so we said okay,

6:15maybe we could replicate something like

6:17this and see how how it performs.

6:20So we did this quick

6:23tool. We call it swarm.

6:25We just open sourced it yesterday, but

6:27it's still like quite early internal

6:29tool. And it's kind of like tool that

6:31help us to orchestrate us and experiment

6:33like this.

6:35So basically you can put any prompt, you

6:38can put any model, you can put any

6:39workflow.

6:41Every

6:43agent has is dedicated container

6:46and what works here like

6:47deterministically is that these

6:49containers with agents are sharing the

6:51evidence between each other and it's

6:53also running on subscriptions, which

6:55means like it doesn't

6:57ruin us.

6:58And with this tool we looked again at

7:01some benchmarking.

7:03And

7:06this time we took a smaller

7:09set, but it was just like

7:11one code base, but very, let's say big

7:13and complicated code base.

7:15And we took six models and

7:18because it is more recent, so I'll will

7:22just point out that we were using the

7:24latest DeepSeek Flash model.

7:26And this time the workflow

7:30was that the swarm, that's the our tool,

7:33was running inside of that the harnesses

7:36that are native to each agent. So we are

7:38running three harnesses, those Cloud

7:39Cod, Codex and DeepSeek harness. And

7:43with this we are looking at the data

7:45set.

7:46In the human baseline there were

7:49four critical and high issues. So that

7:51this was the 100% recall.

7:54And this is how the experiment looked

7:57like.

7:58This is the time window that we

7:59dedicated to this

8:02phase and this is five hours time

8:05window. So basically each this box is

8:08one how we call it wave, and the wave is

8:11like one container, one LLM, and simple

8:15prompt like find all bugs, make no

8:17mistakes. When it finishes, it just runs

8:19again. But between these waves, it

8:21shares the outputs with all the other

8:23guys. So basically

8:25between the waves is a synchronization,

8:27so they are sharing the findings so they

8:29can build on on the findings. And yeah,

8:32we just look what happens.

8:34Also here quite interesting is that you

8:36can see something like

8:38kind of like performance of the of the

8:40models that you can see that for example

8:42with

8:43Luna

8:44we finished 35 waves.

8:47Opus finished only six waves, but it

8:50basically doesn't tell anything how

8:51efficient the model is. It just tells

8:52like how fast it claimed that it has

8:56scanned all the codebase.

8:58And here's the result. So first of all,

9:02these are the findings when we looked

9:04just at the wave zero. So this is like

9:06apples to apples to the previous

9:08experiment. So here are the humans that

9:10discovered four issues.

9:12Opus discovered three. DeepSeek, which

9:14is quite a surprise, discovered three as

9:16well. And the rest is

9:19at two issues. These are the

9:22wave zero.

9:25Then we again look at this, which is the

9:28uniqueness of findings. So here we can

9:31see that the

9:33H2 issues discovered only by humans, and

9:37H H1 issue, which is kind of like

9:40problematic, was in the first wave

9:42discovered only by Opus and DeepSeek.

9:48But this is not what this experiment is

9:49about. This is just like the first wave.

9:52So now we look how it performed

9:55next. So when we run

9:58the more waves

9:59we we did run up to 32 waves of Luna,

10:03but we can see that the major

10:05discoveries happened within the first

10:08three waves. [snorts] And we can see the

10:10for example if Opus it the next runs

10:13didn't bring any value.

10:16With

10:18DeepSeek also

10:20not but for example

10:23GPT six Soul which is the daybreak model

10:26it was productive

10:29in three waves which means basically

10:32this tells when you want to max out the

10:34performance of GPT six Soul you have to

10:36run the same prompt three times

10:39after each other.

10:41And

10:43the next interesting thing is of course

10:47who first discovered the

10:50harder findings. So here we can see that

10:52it wave zero basically the discovery

10:55looks like this and then the single

10:57models were sharing the knowledge but

10:59the question is like who was the first

11:01model that discovered the

11:05harder findings and in this case it was

11:09in wave one

11:11this guy which is the GPT six Soul. So

11:15even if this guy didn't have the highest

11:18recall in the first wave it discovered

11:20the unique issue that Opus missed and

11:22they discovered it in the second wave.

11:25And then as they shared shared the

11:28findings everyone did catch up and in

11:30the end everyone had everything.

11:34And

11:36then the question is like

11:38if the theory that I presented if the

11:40sharing of information between the waves

11:43helped. So in this case it did not bring

11:46any new finding. So we cannot tell for

11:50sure but

11:52it helped the weaker models basically to

11:55catch up and here are two cases

11:58basically that

11:59helped so GLM discovered something which

12:03then DeepSeek finished as finding and

12:06the same here Kimmy was building on

12:08something that was presented in early

12:11waves which confirmed the findings. So,

12:14on this data set, we cannot say for sure

12:17that the replication of the hugging face

12:20experiment did bring the value, but if

12:23we would not be using the

12:25best two models for these guys, it would

12:28definitely help.

12:31And

12:33yeah, how did it end? So,

12:35the recall after the first wave was 75%.

12:40After the always combined, it was 100%.

12:42So, we can tell that at this

12:44benchmark, we

12:46were able to discover everything by the

12:49AI agent workflow.

12:51Just a comment on this, we did not play

12:54anyhow with the prompt engineering. The

12:55prompt was like very stupid. So, this

12:57was like brute force testing. And on

13:00this experiment, we spent about

13:031 million tokens, so it's not that much,

13:06but it's also not very efficient. It's

13:08kind of like brute force testing.

13:12Yeah, and if you would like to

13:16learn more about it, so here is the

13:18block post where we are comparing the

13:20models. And here is the interesting

13:23open AI experiment. And also, if you

13:26would like to try the tool that we just

13:28open sourced, it's in our GitHub since

13:32yesterday.

13:34And

13:35that's pretty much it. Thank you very

13:37much.

Source and accuracy details

Source export SHA256: a79e9e696ef3997696d418cad49f0951ffd59bec1b70a2dbe0907f7b2d84be05