DSS 2026 / Day 1 / Pearl Hall
Fuzzing with Coding Agents: What Works in Practice
Josef Gattermayer
Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.
Transcript
0:00Never mind. So we'll be spoiling a
0:01little bit. So yeah, nice to meet you.
0:02I'm Joseph.
0:04And I'm from ACK three. And now I'll
0:07speak a little bit about fuzzing with AI
0:10agents. So we at ACK we are auditing.
0:15We also are building Wake, which is the
0:17testing and fuzzing framework. And we
0:20are happy to be working for some of the
0:22top tier protocols and I'm also a
0:23professor at the Czech Technical
0:25University in Prague. Thus and let's get
0:28to the point. So here is some of the
0:30guys who are doing formal verification.
0:31So I'll just remind one thing.
0:33With fuzzing
0:35you have black box fuzzing, which means
0:37basically the fuzzer has no information
0:39about the code that is being tested. But
0:43this is the fuzzing that we are not
0:45using anymore.
0:47For years there is something that we at
0:50we call it manually guided fuzzing
0:53which basically means that you're
0:54writing some kind of like test suite
0:56that will help the fuzzer to get
0:58oriented in the code base and be more
1:00efficient not to scan the whole state
1:02space, but just some places where you
1:05think the bugs can occur. So this is
1:07just for explanation.
1:09And
1:12then we have a nice benchmark this
1:14called the
1:15those and here we tested some of the
1:20most known fuzzers.
1:23And it looks like this.
1:25When we did the pure black box fast
1:28testing, which means the fuzzers have no
1:30information about the code that is being
1:32tested.
1:34We can see that on this exercise where
1:36is up to 100
1:3853
1:39let's say bugs.
1:41The fuzzers reached to somewhere like 70
1:43to 75 and we tested Wake, we tested
1:46Foundry, we tested Echidna. Important is
1:49of course like during the time how
1:51efficient the fuzzers are.
1:54They are almost even. There is no much
1:56like difference.
1:58This is just like argument for someone
2:00because is written in Python. So you can
2:02see that it's not about the language
2:04that the fuzzer is implemented because
2:07even like in Python with some Rust
2:10internals Wake is performing on the
2:12black box fastest thing really good.
2:15But this is not what's the talk about.
2:17The talk is about this and this is what
2:19happens when you let AI agent write the
2:23manually guided fastest, which means
2:25basically they are giving specific
2:27instructions to the to the fuzzer how
2:30the fuzzer should approach to the task.
2:34And here what it and did you can see
2:37that basically
2:39it didn't discover like 70 out of 150
2:42issues, but it discovered all issues and
2:45not in 8 hours, but it discovered within
2:471 hour. And how did it do?
2:51Basically the AI agent has read the
2:54source code of the Maze. So we cannot
2:57call it anymore black box or gray box
2:59fastest thing. This is more close to
3:01white box fastest thing. So it reads the
3:04source code of the
3:05of the Maze and it writes an efficient
3:08fuzzing path. In Wake it's done by
3:12it's done done by combination of flows
3:16and environment testing. And in this
3:19path it discovers all the bugs very
3:21efficiently.
3:24Then we did one more test
3:26which is less synthetic. And the test is
3:29based on SC fuzz bench
3:33which basically contains known bugs and
3:36known human written fastest. And we
3:39tested on two projects
3:41with the same approach.
3:43So we let the AI agent write fastest in
3:47Wake using the manually guided fuzzing
3:49principles
3:51without exposing the actually fastest
3:53that were written by humans. So the AI
3:55agent
3:57knew the source code but didn't know any
4:00fastest. And it wrote the fastest. If
4:03the fastest failed, there was a loop
4:04that basically was repairing the fastest
4:07thing suite and it was going in this
4:09loop until it delivered. So, how did it
4:12go?
4:15In this set of contracts that we are
4:17testing, there are
4:2012 known bugs, but only six of them are
4:24discoverable
4:25so far by fuzzing, which means the
4:28human-written fast tests have recall six
4:32of 12 bugs.
4:34With AI, we were able to reproduce that.
4:37So basically,
4:39we delivered the same
4:41fastest as the human-written fastest.
4:44But there's a catch that basically the
4:48results are not that consistent, which
4:50means that in single fastest thing
4:54run,
4:55the highest recall was four out of 12.
4:59So there had to be more
5:01fastest thing rounds.
5:04What is the reason?
5:05The reason is that
5:08when AI is writing
5:10fastest or tests, the one of the things
5:13that are very hard for it to understand
5:16are time constraints, which means like
5:19what happens after something happens. It
5:22can be like
5:23race conditions in initialization of
5:25proxy or in that case it was like
5:29test scenario that was missing that
5:30should happen after payment has started.
5:33So basically it tests the system like
5:36how it is, but not how it changes in
5:38time by default. So this was something
5:40that the fastest were
5:43the AI generated fastest were struggling
5:44with.
5:45But overall like we get to very close
5:49results as humans.
5:51So is this something that could be used
5:53as a replacement of human work? We still
5:55think no because the very edge case
5:59is missed and this is exactly what needs
6:01to be
6:03done by some kind of like narratives,
6:06but to catch like 80-90%
6:09of the bugs, this works really well.
6:13If you would like to read more about the
6:16more about this test suite, so here is
6:18the complete source codes of that.
6:20There's also a link to actual version of
6:23Wake and
6:26yeah, I finished in time, so
6:28next speaker can follow. Thank you very
6:31much.
Source and accuracy details
- Names, organizations, technical terms, code and numbers may be misrecognized.
- Source wording and disfluencies are preserved; no editorial corrections were applied.
- Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.
- A full caption-track export does not establish that every spoken word is represented.
- Automatic YouTube captions; not human-verified against the audio.
Source export SHA256: e562ec00cf10b8315a723839f6aa23c8f9666ba364fdb4a50d35041de4379dd6