DSS 2026 / Day 1 / Pearl Hall
Benchmarking Security Intelligence: Why Web3 AI Security Eval Is Broken
Suryansh Singh
Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.
Transcript
0:02Hi guys. I'm
0:07Hi guys. I'm Suryansh. I'm a security
0:10researcher at OpenZeppelin.
0:12Some of you may know me by my pseudonym
0:14Zero X Monsoon.
0:16And this is a talk on benchmarking like
0:19your AI security agents.
0:22We've heard a lot about them like in the
0:23last 20 minutes on how to use them. This
0:26talk is mostly about
0:28um
0:29how to actually measure the performance
0:31of let's say new frontier models or
0:34AI agents that are in the wild or one
0:37you're building.
0:46Okay. So
0:47first of all,
0:49why do you even need
0:51an E WILE or your own
0:54benchmark when there are so many
0:55security public benchmarks?
0:57Well, basically because most of them are
1:01very broad and general.
1:03You probably need something that's like
1:06more domain specific for you and your
1:08work.
1:09Secondly, you need something that's
1:12hard
1:13and using which you can measure
1:15improvements in any AI agents, security
1:17agents, or anything that you're building
1:20on your newer frontier models.
1:23So
1:25this year like one
1:28Web 3 security benchmark got really
1:30popular.
1:32But it had some problems. First of all,
1:35it uses it used like public audit
1:38contests at as targets
1:40which were probably already in the
1:43training set of the models it was
1:44testing.
1:45Secondly, it used the answer key
1:49of basically all the issues that were
1:50found in that in those public contests
1:53and anything new that the models
1:56actually found were not graded.
1:59So
2:00let's about how to get uh like how to
2:02get your evals wrong. Basically,
2:05um
2:07uh like if you pick the wrong targets,
2:10you won't have a good eval.
2:12And examples of wrong targets are
2:15basically using public repos or public
2:17audit contests because that risks
2:18contamination. If you're only using like
2:21one repo for testing every time you run
2:24your AI agent on new models,
2:26uh you risk like overfitting.
2:29And
2:30if you're synthetically injecting bugs
2:33into let's say a code base, then you're
2:35basically pattern matching what you're
2:37injecting or if you're depending on
2:40on an AI model to do that, you're
2:43basically just testing like what it
2:45tries to find or what it tries to inject
2:49and then find itself again.
2:51Do not use like smaller toy contracts or
2:53CTFs because
2:55the models now are really strong and
2:58they'll just find all of the bugs and
3:00you won't have a good eval.
3:02You can also have log like wrong answer
3:04keys. If your ground truth is fixed,
3:07basically meaning uh every time
3:10something unknown comes up whenever you
3:11run your AI agent on on the eval, if
3:14you're
3:16not adding new issues to it, then that's
3:18a problem.
3:19Uh give um
3:22like if you're only testing for the
3:24correct issues, then also it's not a
3:26very good one because you need to see
3:27how much noise it's generating and have
3:29a separate score for it or like have
3:31penalties for that. And um
3:35don't count all the issues it has fined
3:36like way by severity or like have like a
3:40better metric for it.
3:42You could also give it wrong conditions
3:44like basically if you're giving it not
3:47enough access to tools and stuff,
3:50then it's probably not performing at at
3:52its best.
3:54Uh but don't give it too much
3:55information like hints and stuff because
3:58then you're not really measuring the
3:59performance like bug hunting
4:00capabilities of the model of the
4:02security agent.
4:05So what a good one does, so just to go
4:07through it really quickly, have a lot of
4:09private code bases that you've audited.
4:12Please take permissions from a client
4:14before using their code bases for evals.
4:17But this is probably the best thing you
4:18could do.
4:19Secondly, every time you run the agent
4:22and new issues come up that you don't
4:23know about, that they are not known
4:25false positives or known like true
4:27positives, triage them and add them to
4:29either the
4:31ground truth or like maintain a set of
4:33like known false positives.
4:36Give it all the tools that it could
4:38probably use. Maybe give it any they
4:41would the internet access give it
4:43internet access if you're testing for
4:45something specific.
4:46And then benchmark your matching agent.
4:50Give it like tweak the ground truth
4:52issues a bit and see how it's
4:54performing.
4:55And
4:57that's it, man.
4:59Just to start gather a few past
5:01engagements of good code bases of
5:03various sizes.
5:04Use them, seed the answer key with the
5:07confirmed findings.
5:09And then run the tool
5:11with the repo, the tests, maybe the
5:12internet.
5:14And then triage everything it reports
5:15that's not on the answer keys. Add it to
5:18either the known issues and or all or
5:21the list of false positives.
5:24And that's how your evals will keep on
5:26growing and it would be like domain
5:28specific because the public benchmarks
5:31they only like
5:33test against known findings. But what
5:35you need to do is you need to know how
5:37well your agent performs on like unknown
5:40code bases. Thank you.
Source and accuracy details
- Names, organizations, technical terms, code and numbers may be misrecognized.
- Source wording and disfluencies are preserved; no editorial corrections were applied.
- Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.
- A full caption-track export does not establish that every spoken word is represented.
- Automatic YouTube captions; not human-verified against the audio.
Source export SHA256: 225c0f75f801f5b95a7d9e41db6eda603bcd31620a076cad71e679b947959745