DSS

DSS 2026 / Day 2 / Pearl Hall

Why a Boring LLM Pipeline Can Beat Agentic Complexity

Thomas Marques

4:44 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:01So, hi, everyone.

0:03So, I'm going to talk about the lessons

0:05we we got from building the

0:08so we took some interesting and

0:10counterintuitive

0:12technical decision that actually works

0:14well.

0:15So, just to put everything into

0:16perspective, the

0:19is part of the auto prover tool suite.

0:22So, you have like the auto fuser, auto

0:23prover that are basically inferring the

0:25specs and generating it automatically

0:28for you.

0:28And the auditor which is more classical

0:30and more what

0:32everyone is doing like you provide the

0:34code and get the

0:36get get a report with the

0:39with the

0:40the vulnerabilities in the in it.

0:44So, let's just define a bit of the

0:48the terms we are going to use. So,

0:49what's an agent? I think I'm not going

0:51to spend too much time on it, but

0:52basically it's autonomous system that

0:54have a

0:55a bunch of tools that it can cause to

0:58basically resolve your your what you're

1:00prompted.

1:01So, code and Codex are example of that.

1:04And what we use is that what I call AI

1:07pipeline.

1:08So, it's basically the most naive thing

1:11you can think of.

1:12You provide the context, so it can be

1:13the code base. You set find bugs and

1:16actually it works pretty well and it's

1:17what we do.

1:19So, it's deterministic. You do a bunch

1:21of calls one after another and

1:24yeah.

1:26So, it's how how it works. We split the

1:28context into shards, then we do several

1:30passes with stream others most of the

1:32time the three best currently available.

1:34And then at the end we just the finding

1:36and and get the final report.

1:38So, what we bet on when we do

1:41AI pipeline instead of agentic system is

1:43that the token will be spent more on

1:44thinking about the bugs instead of

1:47actually navigating and thinking of the

1:50of the next tool call we are going to

1:51do, right?

1:52Because when you have an agentic system

1:54the system is not only thinking about

1:56the vulnerabilities, but it spends

1:58tokens on actually which file to browse,

2:01which CMD to run,

2:03which I don't know, variable to to grab,

2:05right?

2:06So that's the bet we are doing we are we

2:09are we are taking here.

2:12When we are doing this kind of systems,

2:15what's cool is that we can use the batch

2:16API.

2:18So the price is reduced by 50%.

2:21Why is that? It's that because

2:25when we use the batch API, so the OpenAI

2:27or Anthropic batch API, we are not

2:29privatized. Like they have 24 hours to

2:31to resolve your your your your your

2:33call. So the price is divided by two.

2:36So what we can do for example is that if

2:38you have like 100 findings to check, you

2:40can

2:41into a single batch

2:43check if those [clears throat] findings

2:44are real.

2:46So you do only a single cash prompt. So

2:48you pay $1.25 for that and then you can

2:51basically read the the cash for 5 cents.

2:54I mean that's crazy cheap.

2:56That's that's really

2:58basically free. And then you can

3:01basically

3:02do calls at a really cheap rate.

3:05So that's what we do.

3:08And

3:09some stuff that

3:12can break the the the the system is that

3:14imperfect shards. So if the context is

3:15not well defined, then we have more

3:17false positive of course because

3:19the standard have all the context it

3:20needs to

3:21to

3:23to to know if a bug is real or not. And

3:25yeah, some

3:27some LLMs that used to be good

3:30are not that good in this

3:32type of task, right? Because most of the

3:34time they are built for agentic work.

3:38So we benchmarked it with the

3:41auto tools that are in the same price

3:44range.

3:45And it's performed really well.

3:47And with a crazy cheap price actually,

3:51that's the price basically for a

3:53classic Solidity code base.

3:55It's language agnostic, so you can run

3:57you can run any code base on it.

3:59But

4:00and we are thinking of of even reducing

4:03the price. We have some stuff in the

4:05that are coming on that will even reduce

4:07the price. So

4:08yeah, our goal is mostly to to to allow

4:11everyone to run it in their CI, to make

4:13it affordable to run it in the CI on on

4:15some commits.

4:17So here it is. So you can check

4:21our tools for Autoprover or the Auditor

4:23on app.set .com.

4:25You can also scan this QR code if you're

4:27an enterprise. You

4:29We might review your your your your your

4:32I mean, you can get some free credits if

4:33you want.

4:35So just apply and we might come back to

4:37you for free credits.

4:39Yeah, times up. Thank you so much.

4:42>> [applause]

Source and accuracy details

Source export SHA256: 9ba0590fd33376942e4977354eb92577461541e11c3a09bd1bc0c8ad6708e6f1