DSS

DSS 2026 / Day 1 / Pearl Hall

Fuzzing with Coding Agents: What Works in Practice

Josef Gattermayer

6:34 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:00Never mind. So we'll be spoiling a

0:01little bit. So yeah, nice to meet you.

0:02I'm Joseph.

0:04And I'm from ACK three. And now I'll

0:07speak a little bit about fuzzing with AI

0:10agents. So we at ACK we are auditing.

0:15We also are building Wake, which is the

0:17testing and fuzzing framework. And we

0:20are happy to be working for some of the

0:22top tier protocols and I'm also a

0:23professor at the Czech Technical

0:25University in Prague. Thus and let's get

0:28to the point. So here is some of the

0:30guys who are doing formal verification.

0:31So I'll just remind one thing.

0:33With fuzzing

0:35you have black box fuzzing, which means

0:37basically the fuzzer has no information

0:39about the code that is being tested. But

0:43this is the fuzzing that we are not

0:45using anymore.

0:47For years there is something that we at

0:50we call it manually guided fuzzing

0:53which basically means that you're

0:54writing some kind of like test suite

0:56that will help the fuzzer to get

0:58oriented in the code base and be more

1:00efficient not to scan the whole state

1:02space, but just some places where you

1:05think the bugs can occur. So this is

1:07just for explanation.

1:09And

1:12then we have a nice benchmark this

1:14called the

1:15those and here we tested some of the

1:20most known fuzzers.

1:23And it looks like this.

1:25When we did the pure black box fast

1:28testing, which means the fuzzers have no

1:30information about the code that is being

1:32tested.

1:34We can see that on this exercise where

1:36is up to 100

1:3853

1:39let's say bugs.

1:41The fuzzers reached to somewhere like 70

1:43to 75 and we tested Wake, we tested

1:46Foundry, we tested Echidna. Important is

1:49of course like during the time how

1:51efficient the fuzzers are.

1:54They are almost even. There is no much

1:56like difference.

1:58This is just like argument for someone

2:00because is written in Python. So you can

2:02see that it's not about the language

2:04that the fuzzer is implemented because

2:07even like in Python with some Rust

2:10internals Wake is performing on the

2:12black box fastest thing really good.

2:15But this is not what's the talk about.

2:17The talk is about this and this is what

2:19happens when you let AI agent write the

2:23manually guided fastest, which means

2:25basically they are giving specific

2:27instructions to the to the fuzzer how

2:30the fuzzer should approach to the task.

2:34And here what it and did you can see

2:37that basically

2:39it didn't discover like 70 out of 150

2:42issues, but it discovered all issues and

2:45not in 8 hours, but it discovered within

2:471 hour. And how did it do?

2:51Basically the AI agent has read the

2:54source code of the Maze. So we cannot

2:57call it anymore black box or gray box

2:59fastest thing. This is more close to

3:01white box fastest thing. So it reads the

3:04source code of the

3:05of the Maze and it writes an efficient

3:08fuzzing path. In Wake it's done by

3:12it's done done by combination of flows

3:16and environment testing. And in this

3:19path it discovers all the bugs very

3:21efficiently.

3:24Then we did one more test

3:26which is less synthetic. And the test is

3:29based on SC fuzz bench

3:33which basically contains known bugs and

3:36known human written fastest. And we

3:39tested on two projects

3:41with the same approach.

3:43So we let the AI agent write fastest in

3:47Wake using the manually guided fuzzing

3:49principles

3:51without exposing the actually fastest

3:53that were written by humans. So the AI

3:55agent

3:57knew the source code but didn't know any

4:00fastest. And it wrote the fastest. If

4:03the fastest failed, there was a loop

4:04that basically was repairing the fastest

4:07thing suite and it was going in this

4:09loop until it delivered. So, how did it

4:12go?

4:15In this set of contracts that we are

4:17testing, there are

4:2012 known bugs, but only six of them are

4:24discoverable

4:25so far by fuzzing, which means the

4:28human-written fast tests have recall six

4:32of 12 bugs.

4:34With AI, we were able to reproduce that.

4:37So basically,

4:39we delivered the same

4:41fastest as the human-written fastest.

4:44But there's a catch that basically the

4:48results are not that consistent, which

4:50means that in single fastest thing

4:54run,

4:55the highest recall was four out of 12.

4:59So there had to be more

5:01fastest thing rounds.

5:04What is the reason?

5:05The reason is that

5:08when AI is writing

5:10fastest or tests, the one of the things

5:13that are very hard for it to understand

5:16are time constraints, which means like

5:19what happens after something happens. It

5:22can be like

5:23race conditions in initialization of

5:25proxy or in that case it was like

5:29test scenario that was missing that

5:30should happen after payment has started.

5:33So basically it tests the system like

5:36how it is, but not how it changes in

5:38time by default. So this was something

5:40that the fastest were

5:43the AI generated fastest were struggling

5:44with.

5:45But overall like we get to very close

5:49results as humans.

5:51So is this something that could be used

5:53as a replacement of human work? We still

5:55think no because the very edge case

5:59is missed and this is exactly what needs

6:01to be

6:03done by some kind of like narratives,

6:06but to catch like 80-90%

6:09of the bugs, this works really well.

6:13If you would like to read more about the

6:16more about this test suite, so here is

6:18the complete source codes of that.

6:20There's also a link to actual version of

6:23Wake and

6:26yeah, I finished in time, so

6:28next speaker can follow. Thank you very

6:31much.

Source and accuracy details

Source export SHA256: e562ec00cf10b8315a723839f6aa23c8f9666ba364fdb4a50d35041de4379dd6