DSS

DSS 2026 / Day 1 / Pearl Hall

Certiflai: orchestrating swarms for certifications

George Kobakhidze

17:07 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:00Okay, perfect.

0:02Hi everyone.

0:03So today we're going to be talking about

0:06orchestrating swarms.

0:08Um, specifically for certifications that

0:11was

0:13the use case I took for this, but I

0:14think this can be applied actually quite

0:16a bit across the board for security work

0:19and it's quite quite um

0:21appropriate especially nowadays.

0:25Yes, so a little butterfly certify

0:28butterfly.

0:30So a little bit about me. My name is

0:32George. I work with Consensus Diligence.

0:35Uh,

0:37in particular I work quite a bit kind of

0:40in my own spare time outside of audits

0:42and security work on standards and

0:44frameworks.

0:45Some of those are BSSC standards, uh,

0:49EEA, the old school OG Ethereum

0:52Enterprise Alliance and most recently as

0:55well seal frameworks though haven't done

0:57as much work there as I have with the

0:58other two. Special shout out to BSSC.

1:01They've recently released

1:03a second version of their smart contract

1:05security standard, which is actually

1:08going to be a big target of my

1:09presentation today. And as part of that

1:12release, they've also

1:14added audit guidelines for each control

1:17in the standards, which was actually

1:18very very important for my work.

1:21And then in some of the other time that

1:23I that I have, I also work quite a bit

1:25on the AI security internally diligence

1:27and most recently I've been working a

1:28lot on vulnerability mining to actually

1:31well mine vulnerabilities using agents

1:34and swarms and AI and whatnot.

1:37So why standards?

1:39Um, this is not a very popular I would

1:42say topic in our industry sadly, but it

1:44is actually quite crucial. If we take a

1:47look at some of the other

1:49mission critical industries out there in

1:51the world, we'll see that all of them

1:52have very strict standards that

1:56their suppliers such as their software

1:58suppliers or they themselves to abide

1:59by.

2:00Um, are across aviation, energy,

2:03healthcare, banking, you know, they all

2:05have actually fairly strict standards

2:08and uh this allows them to have common

2:11solutions to common problems. This

2:13allows them to have common interfaces to

2:15share expertise that they can build upon

2:18uh

2:18amongst their employees or partners.

2:20It's it's very much um

2:23a very useful thing.

2:24Within Ethereum, we don't really have

2:26all that much success across standards

2:29in general. We do of course have a lot

2:30of success across technical standards.

2:33You know, uh we can talk about the

2:34actual very infrastructure level and

2:37kind of foundation level standards, you

2:38know, network standards, you know,

2:39Ethereum itself is a specification. It

2:42is a standard.

2:43Uh but we can talk a little bit more

2:44higher level like even token standards

2:46like ERC -20, tremendously successful um

2:50technical standard for common, you know,

2:52use cases and communication for tokens.

2:54But anything that has has to do with

2:56processes, we really really aren't very

2:59good there. And as we can see lately, a

3:01lot of processes

3:03are the places where things get

3:04exploited. And in fact, just recently,

3:07BSCC also released a uh

3:10transaction lifecycle uh framework and

3:12standard that potentially could have

3:13helped with a lot of the signature-based

3:15exploits we've we've seen lately.

3:18Uh speaking about standards, we can we

3:20of course have to talk about

3:21certification as that is the uh the the

3:24the main topic here. Um

3:26it's very quite it's very simple. So for

3:28us, we can define it as just a an

3:31attestation to standards compliance.

3:34Uh it is very much

3:36uh based on the reputation of the

3:38certifier, I would say. So how much you

3:40trust a certification depends on how

3:42much you trust the certifier. So

3:43reputation is very important here. And

3:45in fact, this is going to be a big

3:46problem for

3:48doing stuff like certifying things with

3:50agents because of course reputation is

3:52quite finicky there.

3:54For Web3 specifically, uh certification

3:57and standards are very hard to scale and

4:00make work kind of as I mentioned

4:01earlier.

4:02So, the challenges there are

4:05and that's also shared with other

4:07mission-critical industries. It's it's

4:08very expensive to do these

4:09certifications. You need expertise. You

4:11need very very uh knowledgeable people

4:14and their time is simply expensive. They

4:16simply do not like to do that.

4:18And oftentimes all of this is helped by,

4:20frankly, the government. It's helped by

4:23uh government bodies that subsidize

4:24these things and enforce standards and

4:28frameworks to be uh to be done within

4:30certain companies and industries. Uh we

4:34of course are famously not very good

4:36with governments. Uh you know lately we

4:38have a lot of institutions coming into

4:40the space, so maybe that's going to

4:41change. But that certainly has been the

4:44case up until very very recently. And

4:46people simply aren't excited about

4:48performing standards or certifications

4:50for processes across the board. And I

4:51understand it. Even I am not really into

4:54doing all that. I like to contribute to

4:55them, certainly get a lot of value out

4:57of it.

4:58But actually applying them is not so

5:00fun.

5:01Agents, however, don't have a concept of

5:03fun. They like to do menial work. They

5:06thrive with instructions, especially

5:09when they're within an isolated kind of

5:12independent control. And that is

5:13precisely what certifications are. And

5:16of course we can parallelize our agents

5:19and we can do hundreds of thousands of

5:21hours within just a few hours if you

5:23have enough compute to do that. So, it

5:25could be quite quite interesting.

5:28Uh very quickly I want to touch uh a

5:30little bit about swarms um because of

5:34course that's what we're going to be

5:35talking about today. So, if we think of

5:37an agent as a singular item, a swarm is

5:40a plural of that.

5:41That's that's really quite simple. Uh of

5:43course it's not actually as simple as

5:46that. Nowadays many agents that we use

5:48are actually swarms underneath them. For

5:50example, our favorite Cloud Code and

5:53Codex are actually systems of agents, so

5:55they're technically swarms. And they are

5:57incredibly valuable and actually very

6:00useful and we'll see how just how much

6:02they kick ass later.

6:04Uh they are what's called a form of a

6:06central orchestrator type of an agent or

6:08or swarm.

6:09There are a bunch of others and like to

6:11give a special shout out specifically to

6:13LangGraph. It's something that we're

6:14going to look at later in the

6:16presentation, but that is what allows us

6:18to actually create much more controlled

6:20swarms where we can provide proper

6:23instructions and actually control what

6:25the agents do and do not do.

6:28All of this is of course very very new.

6:30All of these swarm technologies are

6:31really just coming up this year, maybe a

6:33little bit last year.

6:35So it's it's all bound to change soon

6:37enough of course, but the point is there

6:39are trade-offs to the swarm architecture

6:42that you're going to pick.

6:44So certifications are many isolated

6:47tasks

6:48to be checked and controlled together

6:50and swarms are many agents that like to

6:52do these tasks. Clearly there is some

6:54fit, so I decided to see if it's

6:56actually going to work.

6:59What I did was a series of runs, roughly

7:0244 45 runs. There is the 45th one on my

7:04laptop right now

7:07across a couple of different targets, a

7:10couple of different standards, and a

7:11couple of different modes. So I picked

7:13very very simple standards. I picked ERC

7:1620 actually and we'll we'll see just how

7:18simple it can be or how complex it can

7:20get. And I picked a subsection of the

7:22SSC

7:24smart contract security standard,

7:26its development section, and then I

7:27picked its full standard for the sake of

7:30well seeing how it's going to work with

7:31the bigger corpus.

7:33As far as the modes go, I actually

7:35picked just a singular agent. Again,

7:38it's a bit incorrect because

7:40I use Cloud Code and Codex and that's

7:42technically a swarm underneath, but for

7:44all intents and purposes it was very

7:46very simple. I picked sub agents, so

7:48several Cloud Code models and agents

7:50working together through a loose

7:52orchestration of .md files and prompts

7:55and actually built something out more

7:57custom

7:58Certify which ends up using LangGraph,

8:02custom VMs,

8:04controlled environments with actually a

8:06little bit of a human review there which

8:08could be a little bit of a cheat code,

8:09but we'll see why that's critical for

8:12certifications. And as for targets, I

8:14used Uniswap v2, very small, very

8:17battle-tested um

8:19codebase, and Morpho Blue, uh likewise

8:21small enough and a battle-tested

8:23codebase.

8:25So as far as the actual results, um

8:29happy to say that swarm that I've made

8:32with CertiK actually did test

8:36kind of the best. It actually only

8:38tested as good as Cloud Code itself

8:42without any additional prompts. Uh it's

8:44kind of a testament to just how powerful

8:47these native agents are. Um but as far

8:51as actual verifiability, which is

8:52critical for certifications, we'll see

8:54that it tested even better. Um

8:58an important note is I've tried

8:59different models for my swarm, and it is

9:02a just a night and day difference. In

9:05fact, uh the swarms with Sonnet uh

9:08tested the worst across the board. As

9:11far as the actual rubric here, uh

9:13I cheated a little bit, and I used

9:15agents as well to verify my reports. I

9:17used Fable to go through all the reports

9:19together in a blind test and to go

9:22through a rubric of verifiability, uh

9:25correctness, and if there is actual

9:27argument that follows these reports and

9:29stuff like that.

9:32As far as the first model goes, so the

9:34first uh single agent one prompt, yeah,

9:36that really did incredibly well. It is a

9:39very efficient way to actually certify

9:43things. It's a very efficient way to

9:45review things. Um I think a lot of the

9:48uh people here in the room who use a lot

9:50of custom agents to perform

9:53vulnerability mining and auditing can

9:55attest to just the sheer power of these

9:57native kind of wrappers like Cloud Code

9:59and Codex. They are very, very, very

10:01good. Uh

10:03it it scored the best across the board,

10:06and it was incredibly cheap. Now 30

10:08cents is a little bit of an outlier. I

10:11used uh Codex and that one uh 61 on a

10:15low thinking. Uh I mean, it was still a

10:17good model, still scored very well, and

10:19it was very very cheap. The problem was

10:22that when you give a very simple prompt,

10:24and it was indeed literally find the

10:26certification in your folder, find the

10:28target and apply it, it was one

10:30sentence, the output, of course, looked

10:33very very different uh for each each

10:36run.

10:37So the output was different. Oftentimes

10:39they actually disagreed with each other

10:40because they interpreted the

10:42certifications differently, um because

10:44the certifications often didn't have

10:46actual guidelines.

10:48Uh sometimes they were too lenient. They

10:50read it something as passable, and

10:52sometimes they read it as not passable,

10:54which of course you kind of need to

10:56actually have a decision there.

10:58Uh they

11:00interestingly did not follow directions

11:03for some quite strict things like

11:06actually editing the targets, like the

11:08code, like the Uniswap code and whatnot.

11:09They ended up sometimes editing it as

11:11well, as of course agents sometimes go

11:13rogue. And um they really sucked up with

11:17evidence. They didn't have enough

11:19evidence around their decisions, and

11:21that

11:21course is a really big problem for

11:23certifications. But all in all, it's

11:26very good.

11:28Uh the second mode where I've tried to

11:32kind of paralyze things myself and do it

11:34all through loosely orchestration loose

11:36orchestration of .md files. In fact, I

11:39had something like 17 .md files that

11:42gave guidelines and said how to create

11:44verifier agents and how to do this and

11:47that and kind of create a hierarchy of

11:49sub agents. That was a catastrophe. That

11:52did not work well whatsoever, and I

11:54think again people who have experience

11:58with reviewing agents will know that

12:01if you

12:03if you go about it wrong, it can go

12:05really wrong. And that's exactly what

12:07happened here.

12:08So the good stuff, yeah, the workflows

12:11worked. They were good. They were

12:13getting in right way and they were

12:14feeding the right context and it was

12:16fairly structured and indeed it was

12:18parallel. In fact, it was so parallel

12:20that I ended up running something like

12:2140 agents at the same time at one point.

12:23The bad thing is

12:25that was 40 agents times the initial

12:27context times token cost and I ended up

12:30ballooning my token cost to hundreds of

12:31dollars for each run and that ended up

12:35going on for hours and hours and hours.

12:37Now, if I had infinite token compute, I

12:40guess this could have ended up working

12:42at some point. But the problem is that

12:46it is just incredibly inefficient. And

12:47if you think about it, if you had just a

12:50few people and if you had 100 controls

12:52to go through, would you rather have two

12:54people go through all 100 controls or

12:56would you would you rather have 100

12:58people go through two controls each?

13:00And the answer is actually quite

13:02obvious. You would take the first one

13:05because you would need to have 100

13:07different people in the second case go

13:08through the same context, the same

13:10target every time. It's just

13:11inefficient. So it makes sense

13:14why this works like that in the agent

13:15world too.

13:17And indeed it was quite bad. Another

13:20really bad thing here that happened was

13:21that

13:23once there was a failure at any one

13:25layer, it cascaded across the board.

13:28There was a specific instance with a DNS

13:30failure. It was a tooling error, but

13:32regardless that cascaded across all the

13:34sub agents that are underneath it, the

13:35verifiers, the re-reproducers and it

13:38just it was not a good time.

13:42So certainly something to think about.

13:44In fact, just recently Anthropic put out

13:46research and put out an article saying

13:49begging really people to stop using sub

13:52agents for a lot of the work that a

13:55single agent could do.

13:57So I took both of those learnings and I

14:00sat a little bit with my

14:02agent personnel and we came up with

14:05Certifly, which is a LangGraph based

14:08swarm, meaning it actually orchestrates

14:11in a very direct manner and controls

14:13what agents can and cannot do,

14:15connected with an actual uh little

14:17Python script, program that didn't let

14:20agents to actually write the report

14:21themselves, but picked out from the

14:23agents output and wrote the report.

14:26Um I only settled on two agents this

14:29time. I mean, I did sometimes run a few

14:31instances of that in parallel, but

14:33really I only had two agents, a producer

14:35and a reviewer.

14:37Had a custom VM that would actually be

14:39running tooling and

14:41um like fuzzers and like static

14:43analyzers that would produce very

14:46determined deterministic, excuse me,

14:48output.

14:49And all in all, it worked into something

14:51like this.

14:52Uh yeah, as I mentioned, agents only

14:54proposed reviews and the actual code was

14:57what decided whether or not this is

14:59going to make it into the report.

15:02Uh an important thing here is that

15:04certifications are not just a report.

15:06Certification is a lot more. And with

15:08agents, it is very easy to

15:11um not actually have any of that work to

15:14be done and to kind of hallucinate the

15:16answer. So one of the things you could

15:18have is you could have a report and you

15:19could even have evidence without report,

15:21but it doesn't necessarily mean the

15:23agent went through that review.

15:25So how I did about went about this is I

15:28had some visibility for all processes. I

15:30had some verification checks like hash

15:33files uh or hashes of files that were

15:36were checked, audit trails to make sure

15:37that agents actually read it. Had

15:39canaries where I placed uh incorrect or

15:43things that should trip up the agents

15:45um certifications in the code to see

15:47that actually showed up. And then

15:49finally actually had the human in the

15:50loop to attest. And how I did that was

15:54for every verdict of every control, I

15:56provided a certain amount of metadata

15:59that has to come with the verdict so

16:01that somebody else could go along and

16:02certify that. And that could be a human,

16:05could be not necessarily human. In my

16:06case, I did a little bit of a UI here

16:09where I had an actual evidence kind of

16:11portal where a human could come in and

16:13verify this. And the idea here is that

16:15maybe

16:17it's not agents that need to verify and

16:18test a test everything. Maybe it's still

16:21humans, but agents and swarms can help

16:23us get that information quickly.

16:26In any case, it ranked as best as the

16:29the best agent,

16:31but it also had all the evidence

16:33alongside with it, with the reports,

16:35with the

16:37tooling, and with everything that goes

16:39alongside it.

16:42So yeah,

16:43be very very careful creating swarms cuz

16:46of course they can spiral out of

16:48control.

16:49And note that in the future you might be

16:51creating swarms for other swarms. So

16:54specifically in the case with

16:55certifications, we might be creating

16:57certifications for other agents to kind

16:59of have that reputation network of code.

17:04That's it.

Source and accuracy details

Source export SHA256: 8e957fc1f619122af2aab4f83b9dcd3d34e983fd98233a52af2ec891e39051c4