DSS 2026 / Day 2 / Pearl Hall
Why a Boring LLM Pipeline Can Beat Agentic Complexity
Thomas Marques
Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.
Transcript
0:01So, hi, everyone.
0:03So, I'm going to talk about the lessons
0:05we we got from building the
0:08so we took some interesting and
0:10counterintuitive
0:12technical decision that actually works
0:14well.
0:15So, just to put everything into
0:16perspective, the
0:19is part of the auto prover tool suite.
0:22So, you have like the auto fuser, auto
0:23prover that are basically inferring the
0:25specs and generating it automatically
0:28for you.
0:28And the auditor which is more classical
0:30and more what
0:32everyone is doing like you provide the
0:34code and get the
0:36get get a report with the
0:39with the
0:40the vulnerabilities in the in it.
0:44So, let's just define a bit of the
0:48the terms we are going to use. So,
0:49what's an agent? I think I'm not going
0:51to spend too much time on it, but
0:52basically it's autonomous system that
0:54have a
0:55a bunch of tools that it can cause to
0:58basically resolve your your what you're
1:00prompted.
1:01So, code and Codex are example of that.
1:04And what we use is that what I call AI
1:07pipeline.
1:08So, it's basically the most naive thing
1:11you can think of.
1:12You provide the context, so it can be
1:13the code base. You set find bugs and
1:16actually it works pretty well and it's
1:17what we do.
1:19So, it's deterministic. You do a bunch
1:21of calls one after another and
1:24yeah.
1:26So, it's how how it works. We split the
1:28context into shards, then we do several
1:30passes with stream others most of the
1:32time the three best currently available.
1:34And then at the end we just the finding
1:36and and get the final report.
1:38So, what we bet on when we do
1:41AI pipeline instead of agentic system is
1:43that the token will be spent more on
1:44thinking about the bugs instead of
1:47actually navigating and thinking of the
1:50of the next tool call we are going to
1:51do, right?
1:52Because when you have an agentic system
1:54the system is not only thinking about
1:56the vulnerabilities, but it spends
1:58tokens on actually which file to browse,
2:01which CMD to run,
2:03which I don't know, variable to to grab,
2:05right?
2:06So that's the bet we are doing we are we
2:09are we are taking here.
2:12When we are doing this kind of systems,
2:15what's cool is that we can use the batch
2:16API.
2:18So the price is reduced by 50%.
2:21Why is that? It's that because
2:25when we use the batch API, so the OpenAI
2:27or Anthropic batch API, we are not
2:29privatized. Like they have 24 hours to
2:31to resolve your your your your your
2:33call. So the price is divided by two.
2:36So what we can do for example is that if
2:38you have like 100 findings to check, you
2:40can
2:41into a single batch
2:43check if those [clears throat] findings
2:44are real.
2:46So you do only a single cash prompt. So
2:48you pay $1.25 for that and then you can
2:51basically read the the cash for 5 cents.
2:54I mean that's crazy cheap.
2:56That's that's really
2:58basically free. And then you can
3:01basically
3:02do calls at a really cheap rate.
3:05So that's what we do.
3:08And
3:09some stuff that
3:12can break the the the the system is that
3:14imperfect shards. So if the context is
3:15not well defined, then we have more
3:17false positive of course because
3:19the standard have all the context it
3:20needs to
3:21to
3:23to to know if a bug is real or not. And
3:25yeah, some
3:27some LLMs that used to be good
3:30are not that good in this
3:32type of task, right? Because most of the
3:34time they are built for agentic work.
3:38So we benchmarked it with the
3:41auto tools that are in the same price
3:44range.
3:45And it's performed really well.
3:47And with a crazy cheap price actually,
3:51that's the price basically for a
3:53classic Solidity code base.
3:55It's language agnostic, so you can run
3:57you can run any code base on it.
3:59But
4:00and we are thinking of of even reducing
4:03the price. We have some stuff in the
4:05that are coming on that will even reduce
4:07the price. So
4:08yeah, our goal is mostly to to to allow
4:11everyone to run it in their CI, to make
4:13it affordable to run it in the CI on on
4:15some commits.
4:17So here it is. So you can check
4:21our tools for Autoprover or the Auditor
4:23on app.set .com.
4:25You can also scan this QR code if you're
4:27an enterprise. You
4:29We might review your your your your your
4:32I mean, you can get some free credits if
4:33you want.
4:35So just apply and we might come back to
4:37you for free credits.
4:39Yeah, times up. Thank you so much.
4:42>> [applause]
Source and accuracy details
- Auto-generated YouTube captions; native source language tag preserved; not human-verified against audio.
- Names, organizations, technical terms, code and numbers may be misrecognized.
- Source wording and disfluencies are preserved; no editorial corrections were applied.
- Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.
- A full caption-track export does not establish that every spoken word is represented.
Source export SHA256: 9ba0590fd33376942e4977354eb92577461541e11c3a09bd1bc0c8ad6708e6f1