DSS 2026 — Benchmarking Frontier Models: Which One Is the Best EVM Auditor? Josef Gattermayer Source video: https://www.youtube.com/watch?v=SlDHTNd7cW8 Automatic captions — not human-reviewed. Complete spoken-audio coverage is unverified. Source language tag: en (English). Source export SHA256: a79e9e696ef3997696d418cad49f0951ffd59bec1b70a2dbe0907f7b2d84be05 [0:00] So hello again. [0:02] My name is Joseph and this is the longer [0:04] talk, and we'll speak about [0:09] about [0:10] benchmarking of AI models. [0:13] So let's get started. [0:16] Um [0:17] yeah, the timer. [0:20] Okay, I'm starting. [0:22] So yes, [0:23] I'm co-founder of ACK free and today we [0:28] will look at some benchmarks that we run [0:30] in the past and also I did run the [0:32] latest benchmarks overnight, so it is [0:34] very fresh data that we'll present here [0:37] today. [0:39] We at ACK free, we are developing tools, [0:41] we are doing audits, we are doing [0:43] research and I'm also a stand professor [0:46] Czech Technical University in Prague and [0:49] we are happy we could be working with [0:50] some of the industry leaders as for [0:53] example Lydo here. Thank you guys. [0:56] And so let's get started. [0:59] The question is how to evaluate LLM [1:01] auditors. It's not that easy [1:05] because the [1:06] web two security guys, they have [1:10] more easy [1:12] way of [1:14] measuring if some attack succeeded [1:17] because you can for example [1:19] set a goal to the agent break the system [1:23] or like get into this database. And when [1:25] you set the goal, the agent is just [1:27] working on that and that's all you need [1:29] to settle. But our goal is open. We [1:32] don't know how many bugs are in the [1:35] contracts. So basically we cannot set at [1:38] as easy goal. So that's the first [1:40] problem. The second problem is of course [1:43] data because when we have a [1:46] when we have a project [1:48] where we know the issues because it was [1:50] audited, [1:51] it's probably also public, which means [1:55] the LLMs could have learned on it and [1:57] now we are not evaluating how they [1:59] perform in the future, but we are [2:02] evaluating how they performed in the [2:03] past. So it's very hard to finding some [2:06] uh dataset that was not public, but was [2:09] audited. We have some of the datasets, [2:11] but it's of course a little bit limited. [2:13] And of course we need to care about zero [2:14] data retention when we are doing stuff [2:16] like that, so we cannot just use any [2:18] Chinese API, you know, that we don't [2:20] trust. [2:22] Uh [2:22] so the evaluation part one is that we [2:25] looked at individual models. Uh this is [2:28] an older benchmark. [2:29] Uh we took [2:31] these uh [2:33] these LLMs. We took just a very easy [2:36] workflow with Codex and we just run uh [2:40] that with some prompt, I don't know, [2:41] find all bugs, make no mistakes [2:44] uh over a codebase that was consisted of [2:47] five unpublished audits. And we are [2:51] looking for the recall because we knew [2:53] how many uh [2:54] criticals and highs are there. These are [2:57] the only ones that we tracked. There are [2:58] 40 discovered by sorry, 36 discovered by [3:01] human and four with the later AI [3:04] findings. So the total recall was 40. [3:07] And this is how the single models [3:10] perform. So the human auditors [3:13] discovered uh [3:15] in the first revision, then then it was [3:16] more 32 findings uh [3:20] and the most performant [3:22] uh LLM is Opus five that discovered 35 [3:27] and then it went slowly down. But this [3:31] is not the chart that you should be [3:33] looking at because it doesn't [3:35] basically help you to prevent [3:38] uh from the hacks. So more interesting [3:40] is this chart where we looked at uh [3:44] all the bugs. This is this axis [3:46] basically uh that are present in the [3:49] dataset. We looked at all the models and [3:52] the humans and we looked who discovered [3:55] what. And of course like the Opus [3:58] discovered most of the issues, but is it [4:01] sufficient for you that in your protocol [4:03] you have something that discovers most [4:05] of the issues? No. You need something [4:07] that discovers all the issues. And here [4:10] I must shout out to [4:14] Kimi K Free because this guy [4:17] the overall performance is very poor [4:19] poor, but what he did is that this one [4:22] issue [4:23] discovered no one else but just this [4:25] model. And this is of course like very [4:27] interesting that that it's able to find [4:28] a unique issue [4:30] which was high, not a critical, but only [4:34] this model discovered that it not even [4:35] humans and not even Opus or someone [4:39] else. [4:40] So what does it tell us this benchmark? [4:44] It helped us to form like two [4:46] hypotheses. [4:48] One of them is [4:50] what happens if we just run the [4:52] benchmark multiple times? [4:54] And the second one is like maybe we need [4:57] some kind of like mixture of models to [5:00] be able to find the unique findings [5:02] because this is what we are [5:04] hunting for as well. [5:06] So then we did a second benchmark. [5:09] For this second benchmark [5:11] the inspiration was this hack. I [5:13] don't know if you're familiar with that, [5:14] but it was like I think a month ago [5:18] quite headline news that [5:21] that there was an like internal [5:24] experiment within OpenAI [5:27] where they have put some [5:30] goal to the agents. [5:33] And [5:35] what they did instead was that they [5:37] hacked out of the sandbox and tried to [5:41] or succeeded in hacking a hugging face [5:44] which was the benchmark they were [5:46] supposed to out perform. And [5:49] the interesting thing about this is how [5:51] the agents did it. And [5:54] when you look at the report basically [5:57] they the agents found a way how to [5:59] communicate between each other. They [6:00] were exchanging messages and they were [6:02] like building on each other's [6:04] findings and they even called themselves [6:08] like a collective and [6:10] it was pretty scary, [6:12] but I like that. And so we said okay, [6:15] maybe we could replicate something like [6:17] this and see how how it performs. [6:20] So we did this quick [6:23] tool. We call it swarm. [6:25] We just open sourced it yesterday, but [6:27] it's still like quite early internal [6:29] tool. And it's kind of like tool that [6:31] help us to orchestrate us and experiment [6:33] like this. [6:35] So basically you can put any prompt, you [6:38] can put any model, you can put any [6:39] workflow. [6:41] Every [6:43] agent has is dedicated container [6:46] and what works here like [6:47] deterministically is that these [6:49] containers with agents are sharing the [6:51] evidence between each other and it's [6:53] also running on subscriptions, which [6:55] means like it doesn't [6:57] ruin us. [6:58] And with this tool we looked again at [7:01] some benchmarking. [7:03] And [7:06] this time we took a smaller [7:09] set, but it was just like [7:11] one code base, but very, let's say big [7:13] and complicated code base. [7:15] And we took six models and [7:18] because it is more recent, so I'll will [7:22] just point out that we were using the [7:24] latest DeepSeek Flash model. [7:26] And this time the workflow [7:30] was that the swarm, that's the our tool, [7:33] was running inside of that the harnesses [7:36] that are native to each agent. So we are [7:38] running three harnesses, those Cloud [7:39] Cod, Codex and DeepSeek harness. And [7:43] with this we are looking at the data [7:45] set. [7:46] In the human baseline there were [7:49] four critical and high issues. So that [7:51] this was the 100% recall. [7:54] And this is how the experiment looked [7:57] like. [7:58] This is the time window that we [7:59] dedicated to this [8:02] phase and this is five hours time [8:05] window. So basically each this box is [8:08] one how we call it wave, and the wave is [8:11] like one container, one LLM, and simple [8:15] prompt like find all bugs, make no [8:17] mistakes. When it finishes, it just runs [8:19] again. But between these waves, it [8:21] shares the outputs with all the other [8:23] guys. So basically [8:25] between the waves is a synchronization, [8:27] so they are sharing the findings so they [8:29] can build on on the findings. And yeah, [8:32] we just look what happens. [8:34] Also here quite interesting is that you [8:36] can see something like [8:38] kind of like performance of the of the [8:40] models that you can see that for example [8:42] with [8:43] Luna [8:44] we finished 35 waves. [8:47] Opus finished only six waves, but it [8:50] basically doesn't tell anything how [8:51] efficient the model is. It just tells [8:52] like how fast it claimed that it has [8:56] scanned all the codebase. [8:58] And here's the result. So first of all, [9:02] these are the findings when we looked [9:04] just at the wave zero. So this is like [9:06] apples to apples to the previous [9:08] experiment. So here are the humans that [9:10] discovered four issues. [9:12] Opus discovered three. DeepSeek, which [9:14] is quite a surprise, discovered three as [9:16] well. And the rest is [9:19] at two issues. These are the [9:22] wave zero. [9:25] Then we again look at this, which is the [9:28] uniqueness of findings. So here we can [9:31] see that the [9:33] H2 issues discovered only by humans, and [9:37] H H1 issue, which is kind of like [9:40] problematic, was in the first wave [9:42] discovered only by Opus and DeepSeek. [9:48] But this is not what this experiment is [9:49] about. This is just like the first wave. [9:52] So now we look how it performed [9:55] next. So when we run [9:58] the more waves [9:59] we we did run up to 32 waves of Luna, [10:03] but we can see that the major [10:05] discoveries happened within the first [10:08] three waves. [snorts] And we can see the [10:10] for example if Opus it the next runs [10:13] didn't bring any value. [10:16] With [10:18] DeepSeek also [10:20] not but for example [10:23] GPT six Soul which is the daybreak model [10:26] it was productive [10:29] in three waves which means basically [10:32] this tells when you want to max out the [10:34] performance of GPT six Soul you have to [10:36] run the same prompt three times [10:39] after each other. [10:41] And [10:43] the next interesting thing is of course [10:47] who first discovered the [10:50] harder findings. So here we can see that [10:52] it wave zero basically the discovery [10:55] looks like this and then the single [10:57] models were sharing the knowledge but [10:59] the question is like who was the first [11:01] model that discovered the [11:05] harder findings and in this case it was [11:09] in wave one [11:11] this guy which is the GPT six Soul. So [11:15] even if this guy didn't have the highest [11:18] recall in the first wave it discovered [11:20] the unique issue that Opus missed and [11:22] they discovered it in the second wave. [11:25] And then as they shared shared the [11:28] findings everyone did catch up and in [11:30] the end everyone had everything. [11:34] And [11:36] then the question is like [11:38] if the theory that I presented if the [11:40] sharing of information between the waves [11:43] helped. So in this case it did not bring [11:46] any new finding. So we cannot tell for [11:50] sure but [11:52] it helped the weaker models basically to [11:55] catch up and here are two cases [11:58] basically that [11:59] helped so GLM discovered something which [12:03] then DeepSeek finished as finding and [12:06] the same here Kimmy was building on [12:08] something that was presented in early [12:11] waves which confirmed the findings. So, [12:14] on this data set, we cannot say for sure [12:17] that the replication of the hugging face [12:20] experiment did bring the value, but if [12:23] we would not be using the [12:25] best two models for these guys, it would [12:28] definitely help. [12:31] And [12:33] yeah, how did it end? So, [12:35] the recall after the first wave was 75%. [12:40] After the always combined, it was 100%. [12:42] So, we can tell that at this [12:44] benchmark, we [12:46] were able to discover everything by the [12:49] AI agent workflow. [12:51] Just a comment on this, we did not play [12:54] anyhow with the prompt engineering. The [12:55] prompt was like very stupid. So, this [12:57] was like brute force testing. And on [13:00] this experiment, we spent about [13:03] 1 million tokens, so it's not that much, [13:06] but it's also not very efficient. It's [13:08] kind of like brute force testing. [13:12] Yeah, and if you would like to [13:16] learn more about it, so here is the [13:18] block post where we are comparing the [13:20] models. And here is the interesting [13:23] open AI experiment. And also, if you [13:26] would like to try the tool that we just [13:28] open sourced, it's in our GitHub since [13:32] yesterday. [13:34] And [13:35] that's pretty much it. Thank you very [13:37] much.