DSS 2026 — Why a Boring LLM Pipeline Can Beat Agentic Complexity Thomas Marques Source video: https://www.youtube.com/watch?v=chhkbuKSBFg Automatic captions — not human-reviewed. Complete spoken-audio coverage is unverified. Source language tag: en (English). Source export SHA256: 9ba0590fd33376942e4977354eb92577461541e11c3a09bd1bc0c8ad6708e6f1 [0:01] So, hi, everyone. [0:03] So, I'm going to talk about the lessons [0:05] we we got from building the [0:08] so we took some interesting and [0:10] counterintuitive [0:12] technical decision that actually works [0:14] well. [0:15] So, just to put everything into [0:16] perspective, the [0:19] is part of the auto prover tool suite. [0:22] So, you have like the auto fuser, auto [0:23] prover that are basically inferring the [0:25] specs and generating it automatically [0:28] for you. [0:28] And the auditor which is more classical [0:30] and more what [0:32] everyone is doing like you provide the [0:34] code and get the [0:36] get get a report with the [0:39] with the [0:40] the vulnerabilities in the in it. [0:44] So, let's just define a bit of the [0:48] the terms we are going to use. So, [0:49] what's an agent? I think I'm not going [0:51] to spend too much time on it, but [0:52] basically it's autonomous system that [0:54] have a [0:55] a bunch of tools that it can cause to [0:58] basically resolve your your what you're [1:00] prompted. [1:01] So, code and Codex are example of that. [1:04] And what we use is that what I call AI [1:07] pipeline. [1:08] So, it's basically the most naive thing [1:11] you can think of. [1:12] You provide the context, so it can be [1:13] the code base. You set find bugs and [1:16] actually it works pretty well and it's [1:17] what we do. [1:19] So, it's deterministic. You do a bunch [1:21] of calls one after another and [1:24] yeah. [1:26] So, it's how how it works. We split the [1:28] context into shards, then we do several [1:30] passes with stream others most of the [1:32] time the three best currently available. [1:34] And then at the end we just the finding [1:36] and and get the final report. [1:38] So, what we bet on when we do [1:41] AI pipeline instead of agentic system is [1:43] that the token will be spent more on [1:44] thinking about the bugs instead of [1:47] actually navigating and thinking of the [1:50] of the next tool call we are going to [1:51] do, right? [1:52] Because when you have an agentic system [1:54] the system is not only thinking about [1:56] the vulnerabilities, but it spends [1:58] tokens on actually which file to browse, [2:01] which CMD to run, [2:03] which I don't know, variable to to grab, [2:05] right? [2:06] So that's the bet we are doing we are we [2:09] are we are taking here. [2:12] When we are doing this kind of systems, [2:15] what's cool is that we can use the batch [2:16] API. [2:18] So the price is reduced by 50%. [2:21] Why is that? It's that because [2:25] when we use the batch API, so the OpenAI [2:27] or Anthropic batch API, we are not [2:29] privatized. Like they have 24 hours to [2:31] to resolve your your your your your [2:33] call. So the price is divided by two. [2:36] So what we can do for example is that if [2:38] you have like 100 findings to check, you [2:40] can [2:41] into a single batch [2:43] check if those [clears throat] findings [2:44] are real. [2:46] So you do only a single cash prompt. So [2:48] you pay $1.25 for that and then you can [2:51] basically read the the cash for 5 cents. [2:54] I mean that's crazy cheap. [2:56] That's that's really [2:58] basically free. And then you can [3:01] basically [3:02] do calls at a really cheap rate. [3:05] So that's what we do. [3:08] And [3:09] some stuff that [3:12] can break the the the the system is that [3:14] imperfect shards. So if the context is [3:15] not well defined, then we have more [3:17] false positive of course because [3:19] the standard have all the context it [3:20] needs to [3:21] to [3:23] to to know if a bug is real or not. And [3:25] yeah, some [3:27] some LLMs that used to be good [3:30] are not that good in this [3:32] type of task, right? Because most of the [3:34] time they are built for agentic work. [3:38] So we benchmarked it with the [3:41] auto tools that are in the same price [3:44] range. [3:45] And it's performed really well. [3:47] And with a crazy cheap price actually, [3:51] that's the price basically for a [3:53] classic Solidity code base. [3:55] It's language agnostic, so you can run [3:57] you can run any code base on it. [3:59] But [4:00] and we are thinking of of even reducing [4:03] the price. We have some stuff in the [4:05] that are coming on that will even reduce [4:07] the price. So [4:08] yeah, our goal is mostly to to to allow [4:11] everyone to run it in their CI, to make [4:13] it affordable to run it in the CI on on [4:15] some commits. [4:17] So here it is. So you can check [4:21] our tools for Autoprover or the Auditor [4:23] on app.set .com. [4:25] You can also scan this QR code if you're [4:27] an enterprise. You [4:29] We might review your your your your your [4:32] I mean, you can get some free credits if [4:33] you want. [4:35] So just apply and we might come back to [4:37] you for free credits. [4:39] Yeah, times up. Thank you so much. [4:42] >> [applause]