DSS 2026 — Benchmarking Security Intelligence: Why Web3 AI Security Eval Is Broken Suryansh Singh Source video: https://www.youtube.com/watch?v=Dm7fzCWJ0Oo Automatic captions — not human-reviewed. Complete spoken-audio coverage is unverified. Source language tag: en (English). Source export SHA256: 225c0f75f801f5b95a7d9e41db6eda603bcd31620a076cad71e679b947959745 [0:02] Hi guys. I'm [0:07] Hi guys. I'm Suryansh. I'm a security [0:10] researcher at OpenZeppelin. [0:12] Some of you may know me by my pseudonym [0:14] Zero X Monsoon. [0:16] And this is a talk on benchmarking like [0:19] your AI security agents. [0:22] We've heard a lot about them like in the [0:23] last 20 minutes on how to use them. This [0:26] talk is mostly about [0:28] um [0:29] how to actually measure the performance [0:31] of let's say new frontier models or [0:34] AI agents that are in the wild or one [0:37] you're building. [0:46] Okay. So [0:47] first of all, [0:49] why do you even need [0:51] an E WILE or your own [0:54] benchmark when there are so many [0:55] security public benchmarks? [0:57] Well, basically because most of them are [1:01] very broad and general. [1:03] You probably need something that's like [1:06] more domain specific for you and your [1:08] work. [1:09] Secondly, you need something that's [1:12] hard [1:13] and using which you can measure [1:15] improvements in any AI agents, security [1:17] agents, or anything that you're building [1:20] on your newer frontier models. [1:23] So [1:25] this year like one [1:28] Web 3 security benchmark got really [1:30] popular. [1:32] But it had some problems. First of all, [1:35] it uses it used like public audit [1:38] contests at as targets [1:40] which were probably already in the [1:43] training set of the models it was [1:44] testing. [1:45] Secondly, it used the answer key [1:49] of basically all the issues that were [1:50] found in that in those public contests [1:53] and anything new that the models [1:56] actually found were not graded. [1:59] So [2:00] let's about how to get uh like how to [2:02] get your evals wrong. Basically, [2:05] um [2:07] uh like if you pick the wrong targets, [2:10] you won't have a good eval. [2:12] And examples of wrong targets are [2:15] basically using public repos or public [2:17] audit contests because that risks [2:18] contamination. If you're only using like [2:21] one repo for testing every time you run [2:24] your AI agent on new models, [2:26] uh you risk like overfitting. [2:29] And [2:30] if you're synthetically injecting bugs [2:33] into let's say a code base, then you're [2:35] basically pattern matching what you're [2:37] injecting or if you're depending on [2:40] on an AI model to do that, you're [2:43] basically just testing like what it [2:45] tries to find or what it tries to inject [2:49] and then find itself again. [2:51] Do not use like smaller toy contracts or [2:53] CTFs because [2:55] the models now are really strong and [2:58] they'll just find all of the bugs and [3:00] you won't have a good eval. [3:02] You can also have log like wrong answer [3:04] keys. If your ground truth is fixed, [3:07] basically meaning uh every time [3:10] something unknown comes up whenever you [3:11] run your AI agent on on the eval, if [3:14] you're [3:16] not adding new issues to it, then that's [3:18] a problem. [3:19] Uh give um [3:22] like if you're only testing for the [3:24] correct issues, then also it's not a [3:26] very good one because you need to see [3:27] how much noise it's generating and have [3:29] a separate score for it or like have [3:31] penalties for that. And um [3:35] don't count all the issues it has fined [3:36] like way by severity or like have like a [3:40] better metric for it. [3:42] You could also give it wrong conditions [3:44] like basically if you're giving it not [3:47] enough access to tools and stuff, [3:50] then it's probably not performing at at [3:52] its best. [3:54] Uh but don't give it too much [3:55] information like hints and stuff because [3:58] then you're not really measuring the [3:59] performance like bug hunting [4:00] capabilities of the model of the [4:02] security agent. [4:05] So what a good one does, so just to go [4:07] through it really quickly, have a lot of [4:09] private code bases that you've audited. [4:12] Please take permissions from a client [4:14] before using their code bases for evals. [4:17] But this is probably the best thing you [4:18] could do. [4:19] Secondly, every time you run the agent [4:22] and new issues come up that you don't [4:23] know about, that they are not known [4:25] false positives or known like true [4:27] positives, triage them and add them to [4:29] either the [4:31] ground truth or like maintain a set of [4:33] like known false positives. [4:36] Give it all the tools that it could [4:38] probably use. Maybe give it any they [4:41] would the internet access give it [4:43] internet access if you're testing for [4:45] something specific. [4:46] And then benchmark your matching agent. [4:50] Give it like tweak the ground truth [4:52] issues a bit and see how it's [4:54] performing. [4:55] And [4:57] that's it, man. [4:59] Just to start gather a few past [5:01] engagements of good code bases of [5:03] various sizes. [5:04] Use them, seed the answer key with the [5:07] confirmed findings. [5:09] And then run the tool [5:11] with the repo, the tests, maybe the [5:12] internet. [5:14] And then triage everything it reports [5:15] that's not on the answer keys. Add it to [5:18] either the known issues and or all or [5:21] the list of false positives. [5:24] And that's how your evals will keep on [5:26] growing and it would be like domain [5:28] specific because the public benchmarks [5:31] they only like [5:33] test against known findings. But what [5:35] you need to do is you need to know how [5:37] well your agent performs on like unknown [5:40] code bases. Thank you.