DSS

DSS 2026 / Day 1 / Pearl Hall

Benchmarking Security Intelligence: Why Web3 AI Security Eval Is Broken

Suryansh Singh

5:44 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:02Hi guys. I'm

0:07Hi guys. I'm Suryansh. I'm a security

0:10researcher at OpenZeppelin.

0:12Some of you may know me by my pseudonym

0:14Zero X Monsoon.

0:16And this is a talk on benchmarking like

0:19your AI security agents.

0:22We've heard a lot about them like in the

0:23last 20 minutes on how to use them. This

0:26talk is mostly about

0:28um

0:29how to actually measure the performance

0:31of let's say new frontier models or

0:34AI agents that are in the wild or one

0:37you're building.

0:46Okay. So

0:47first of all,

0:49why do you even need

0:51an E WILE or your own

0:54benchmark when there are so many

0:55security public benchmarks?

0:57Well, basically because most of them are

1:01very broad and general.

1:03You probably need something that's like

1:06more domain specific for you and your

1:08work.

1:09Secondly, you need something that's

1:12hard

1:13and using which you can measure

1:15improvements in any AI agents, security

1:17agents, or anything that you're building

1:20on your newer frontier models.

1:23So

1:25this year like one

1:28Web 3 security benchmark got really

1:30popular.

1:32But it had some problems. First of all,

1:35it uses it used like public audit

1:38contests at as targets

1:40which were probably already in the

1:43training set of the models it was

1:44testing.

1:45Secondly, it used the answer key

1:49of basically all the issues that were

1:50found in that in those public contests

1:53and anything new that the models

1:56actually found were not graded.

1:59So

2:00let's about how to get uh like how to

2:02get your evals wrong. Basically,

2:05um

2:07uh like if you pick the wrong targets,

2:10you won't have a good eval.

2:12And examples of wrong targets are

2:15basically using public repos or public

2:17audit contests because that risks

2:18contamination. If you're only using like

2:21one repo for testing every time you run

2:24your AI agent on new models,

2:26uh you risk like overfitting.

2:29And

2:30if you're synthetically injecting bugs

2:33into let's say a code base, then you're

2:35basically pattern matching what you're

2:37injecting or if you're depending on

2:40on an AI model to do that, you're

2:43basically just testing like what it

2:45tries to find or what it tries to inject

2:49and then find itself again.

2:51Do not use like smaller toy contracts or

2:53CTFs because

2:55the models now are really strong and

2:58they'll just find all of the bugs and

3:00you won't have a good eval.

3:02You can also have log like wrong answer

3:04keys. If your ground truth is fixed,

3:07basically meaning uh every time

3:10something unknown comes up whenever you

3:11run your AI agent on on the eval, if

3:14you're

3:16not adding new issues to it, then that's

3:18a problem.

3:19Uh give um

3:22like if you're only testing for the

3:24correct issues, then also it's not a

3:26very good one because you need to see

3:27how much noise it's generating and have

3:29a separate score for it or like have

3:31penalties for that. And um

3:35don't count all the issues it has fined

3:36like way by severity or like have like a

3:40better metric for it.

3:42You could also give it wrong conditions

3:44like basically if you're giving it not

3:47enough access to tools and stuff,

3:50then it's probably not performing at at

3:52its best.

3:54Uh but don't give it too much

3:55information like hints and stuff because

3:58then you're not really measuring the

3:59performance like bug hunting

4:00capabilities of the model of the

4:02security agent.

4:05So what a good one does, so just to go

4:07through it really quickly, have a lot of

4:09private code bases that you've audited.

4:12Please take permissions from a client

4:14before using their code bases for evals.

4:17But this is probably the best thing you

4:18could do.

4:19Secondly, every time you run the agent

4:22and new issues come up that you don't

4:23know about, that they are not known

4:25false positives or known like true

4:27positives, triage them and add them to

4:29either the

4:31ground truth or like maintain a set of

4:33like known false positives.

4:36Give it all the tools that it could

4:38probably use. Maybe give it any they

4:41would the internet access give it

4:43internet access if you're testing for

4:45something specific.

4:46And then benchmark your matching agent.

4:50Give it like tweak the ground truth

4:52issues a bit and see how it's

4:54performing.

4:55And

4:57that's it, man.

4:59Just to start gather a few past

5:01engagements of good code bases of

5:03various sizes.

5:04Use them, seed the answer key with the

5:07confirmed findings.

5:09And then run the tool

5:11with the repo, the tests, maybe the

5:12internet.

5:14And then triage everything it reports

5:15that's not on the answer keys. Add it to

5:18either the known issues and or all or

5:21the list of false positives.

5:24And that's how your evals will keep on

5:26growing and it would be like domain

5:28specific because the public benchmarks

5:31they only like

5:33test against known findings. But what

5:35you need to do is you need to know how

5:37well your agent performs on like unknown

5:40code bases. Thank you.

Source and accuracy details

Source export SHA256: 225c0f75f801f5b95a7d9e41db6eda603bcd31620a076cad71e679b947959745