DSS 2026 — Fuzzing with Coding Agents: What Works in Practice Josef Gattermayer Source video: https://www.youtube.com/watch?v=AVdpxgNfQFg Automatic captions — not human-reviewed. Complete spoken-audio coverage is unverified. Source language tag: en (English). Source export SHA256: e562ec00cf10b8315a723839f6aa23c8f9666ba364fdb4a50d35041de4379dd6 [0:00] Never mind. So we'll be spoiling a [0:01] little bit. So yeah, nice to meet you. [0:02] I'm Joseph. [0:04] And I'm from ACK three. And now I'll [0:07] speak a little bit about fuzzing with AI [0:10] agents. So we at ACK we are auditing. [0:15] We also are building Wake, which is the [0:17] testing and fuzzing framework. And we [0:20] are happy to be working for some of the [0:22] top tier protocols and I'm also a [0:23] professor at the Czech Technical [0:25] University in Prague. Thus and let's get [0:28] to the point. So here is some of the [0:30] guys who are doing formal verification. [0:31] So I'll just remind one thing. [0:33] With fuzzing [0:35] you have black box fuzzing, which means [0:37] basically the fuzzer has no information [0:39] about the code that is being tested. But [0:43] this is the fuzzing that we are not [0:45] using anymore. [0:47] For years there is something that we at [0:50] we call it manually guided fuzzing [0:53] which basically means that you're [0:54] writing some kind of like test suite [0:56] that will help the fuzzer to get [0:58] oriented in the code base and be more [1:00] efficient not to scan the whole state [1:02] space, but just some places where you [1:05] think the bugs can occur. So this is [1:07] just for explanation. [1:09] And [1:12] then we have a nice benchmark this [1:14] called the [1:15] those and here we tested some of the [1:20] most known fuzzers. [1:23] And it looks like this. [1:25] When we did the pure black box fast [1:28] testing, which means the fuzzers have no [1:30] information about the code that is being [1:32] tested. [1:34] We can see that on this exercise where [1:36] is up to 100 [1:38] 53 [1:39] let's say bugs. [1:41] The fuzzers reached to somewhere like 70 [1:43] to 75 and we tested Wake, we tested [1:46] Foundry, we tested Echidna. Important is [1:49] of course like during the time how [1:51] efficient the fuzzers are. [1:54] They are almost even. There is no much [1:56] like difference. [1:58] This is just like argument for someone [2:00] because is written in Python. So you can [2:02] see that it's not about the language [2:04] that the fuzzer is implemented because [2:07] even like in Python with some Rust [2:10] internals Wake is performing on the [2:12] black box fastest thing really good. [2:15] But this is not what's the talk about. [2:17] The talk is about this and this is what [2:19] happens when you let AI agent write the [2:23] manually guided fastest, which means [2:25] basically they are giving specific [2:27] instructions to the to the fuzzer how [2:30] the fuzzer should approach to the task. [2:34] And here what it and did you can see [2:37] that basically [2:39] it didn't discover like 70 out of 150 [2:42] issues, but it discovered all issues and [2:45] not in 8 hours, but it discovered within [2:47] 1 hour. And how did it do? [2:51] Basically the AI agent has read the [2:54] source code of the Maze. So we cannot [2:57] call it anymore black box or gray box [2:59] fastest thing. This is more close to [3:01] white box fastest thing. So it reads the [3:04] source code of the [3:05] of the Maze and it writes an efficient [3:08] fuzzing path. In Wake it's done by [3:12] it's done done by combination of flows [3:16] and environment testing. And in this [3:19] path it discovers all the bugs very [3:21] efficiently. [3:24] Then we did one more test [3:26] which is less synthetic. And the test is [3:29] based on SC fuzz bench [3:33] which basically contains known bugs and [3:36] known human written fastest. And we [3:39] tested on two projects [3:41] with the same approach. [3:43] So we let the AI agent write fastest in [3:47] Wake using the manually guided fuzzing [3:49] principles [3:51] without exposing the actually fastest [3:53] that were written by humans. So the AI [3:55] agent [3:57] knew the source code but didn't know any [4:00] fastest. And it wrote the fastest. If [4:03] the fastest failed, there was a loop [4:04] that basically was repairing the fastest [4:07] thing suite and it was going in this [4:09] loop until it delivered. So, how did it [4:12] go? [4:15] In this set of contracts that we are [4:17] testing, there are [4:20] 12 known bugs, but only six of them are [4:24] discoverable [4:25] so far by fuzzing, which means the [4:28] human-written fast tests have recall six [4:32] of 12 bugs. [4:34] With AI, we were able to reproduce that. [4:37] So basically, [4:39] we delivered the same [4:41] fastest as the human-written fastest. [4:44] But there's a catch that basically the [4:48] results are not that consistent, which [4:50] means that in single fastest thing [4:54] run, [4:55] the highest recall was four out of 12. [4:59] So there had to be more [5:01] fastest thing rounds. [5:04] What is the reason? [5:05] The reason is that [5:08] when AI is writing [5:10] fastest or tests, the one of the things [5:13] that are very hard for it to understand [5:16] are time constraints, which means like [5:19] what happens after something happens. It [5:22] can be like [5:23] race conditions in initialization of [5:25] proxy or in that case it was like [5:29] test scenario that was missing that [5:30] should happen after payment has started. [5:33] So basically it tests the system like [5:36] how it is, but not how it changes in [5:38] time by default. So this was something [5:40] that the fastest were [5:43] the AI generated fastest were struggling [5:44] with. [5:45] But overall like we get to very close [5:49] results as humans. [5:51] So is this something that could be used [5:53] as a replacement of human work? We still [5:55] think no because the very edge case [5:59] is missed and this is exactly what needs [6:01] to be [6:03] done by some kind of like narratives, [6:06] but to catch like 80-90% [6:09] of the bugs, this works really well. [6:13] If you would like to read more about the [6:16] more about this test suite, so here is [6:18] the complete source codes of that. [6:20] There's also a link to actual version of [6:23] Wake and [6:26] yeah, I finished in time, so [6:28] next speaker can follow. Thank you very [6:31] much.