DSS 2026 / Day 1 / Pearl Hall
Certiflai: orchestrating swarms for certifications
George Kobakhidze
Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.
Transcript
0:00Okay, perfect.
0:02Hi everyone.
0:03So today we're going to be talking about
0:06orchestrating swarms.
0:08Um, specifically for certifications that
0:11was
0:13the use case I took for this, but I
0:14think this can be applied actually quite
0:16a bit across the board for security work
0:19and it's quite quite um
0:21appropriate especially nowadays.
0:25Yes, so a little butterfly certify
0:28butterfly.
0:30So a little bit about me. My name is
0:32George. I work with Consensus Diligence.
0:35Uh,
0:37in particular I work quite a bit kind of
0:40in my own spare time outside of audits
0:42and security work on standards and
0:44frameworks.
0:45Some of those are BSSC standards, uh,
0:49EEA, the old school OG Ethereum
0:52Enterprise Alliance and most recently as
0:55well seal frameworks though haven't done
0:57as much work there as I have with the
0:58other two. Special shout out to BSSC.
1:01They've recently released
1:03a second version of their smart contract
1:05security standard, which is actually
1:08going to be a big target of my
1:09presentation today. And as part of that
1:12release, they've also
1:14added audit guidelines for each control
1:17in the standards, which was actually
1:18very very important for my work.
1:21And then in some of the other time that
1:23I that I have, I also work quite a bit
1:25on the AI security internally diligence
1:27and most recently I've been working a
1:28lot on vulnerability mining to actually
1:31well mine vulnerabilities using agents
1:34and swarms and AI and whatnot.
1:37So why standards?
1:39Um, this is not a very popular I would
1:42say topic in our industry sadly, but it
1:44is actually quite crucial. If we take a
1:47look at some of the other
1:49mission critical industries out there in
1:51the world, we'll see that all of them
1:52have very strict standards that
1:56their suppliers such as their software
1:58suppliers or they themselves to abide
1:59by.
2:00Um, are across aviation, energy,
2:03healthcare, banking, you know, they all
2:05have actually fairly strict standards
2:08and uh this allows them to have common
2:11solutions to common problems. This
2:13allows them to have common interfaces to
2:15share expertise that they can build upon
2:18uh
2:18amongst their employees or partners.
2:20It's it's very much um
2:23a very useful thing.
2:24Within Ethereum, we don't really have
2:26all that much success across standards
2:29in general. We do of course have a lot
2:30of success across technical standards.
2:33You know, uh we can talk about the
2:34actual very infrastructure level and
2:37kind of foundation level standards, you
2:38know, network standards, you know,
2:39Ethereum itself is a specification. It
2:42is a standard.
2:43Uh but we can talk a little bit more
2:44higher level like even token standards
2:46like ERC -20, tremendously successful um
2:50technical standard for common, you know,
2:52use cases and communication for tokens.
2:54But anything that has has to do with
2:56processes, we really really aren't very
2:59good there. And as we can see lately, a
3:01lot of processes
3:03are the places where things get
3:04exploited. And in fact, just recently,
3:07BSCC also released a uh
3:10transaction lifecycle uh framework and
3:12standard that potentially could have
3:13helped with a lot of the signature-based
3:15exploits we've we've seen lately.
3:18Uh speaking about standards, we can we
3:20of course have to talk about
3:21certification as that is the uh the the
3:24the main topic here. Um
3:26it's very quite it's very simple. So for
3:28us, we can define it as just a an
3:31attestation to standards compliance.
3:34Uh it is very much
3:36uh based on the reputation of the
3:38certifier, I would say. So how much you
3:40trust a certification depends on how
3:42much you trust the certifier. So
3:43reputation is very important here. And
3:45in fact, this is going to be a big
3:46problem for
3:48doing stuff like certifying things with
3:50agents because of course reputation is
3:52quite finicky there.
3:54For Web3 specifically, uh certification
3:57and standards are very hard to scale and
4:00make work kind of as I mentioned
4:01earlier.
4:02So, the challenges there are
4:05and that's also shared with other
4:07mission-critical industries. It's it's
4:08very expensive to do these
4:09certifications. You need expertise. You
4:11need very very uh knowledgeable people
4:14and their time is simply expensive. They
4:16simply do not like to do that.
4:18And oftentimes all of this is helped by,
4:20frankly, the government. It's helped by
4:23uh government bodies that subsidize
4:24these things and enforce standards and
4:28frameworks to be uh to be done within
4:30certain companies and industries. Uh we
4:34of course are famously not very good
4:36with governments. Uh you know lately we
4:38have a lot of institutions coming into
4:40the space, so maybe that's going to
4:41change. But that certainly has been the
4:44case up until very very recently. And
4:46people simply aren't excited about
4:48performing standards or certifications
4:50for processes across the board. And I
4:51understand it. Even I am not really into
4:54doing all that. I like to contribute to
4:55them, certainly get a lot of value out
4:57of it.
4:58But actually applying them is not so
5:00fun.
5:01Agents, however, don't have a concept of
5:03fun. They like to do menial work. They
5:06thrive with instructions, especially
5:09when they're within an isolated kind of
5:12independent control. And that is
5:13precisely what certifications are. And
5:16of course we can parallelize our agents
5:19and we can do hundreds of thousands of
5:21hours within just a few hours if you
5:23have enough compute to do that. So, it
5:25could be quite quite interesting.
5:28Uh very quickly I want to touch uh a
5:30little bit about swarms um because of
5:34course that's what we're going to be
5:35talking about today. So, if we think of
5:37an agent as a singular item, a swarm is
5:40a plural of that.
5:41That's that's really quite simple. Uh of
5:43course it's not actually as simple as
5:46that. Nowadays many agents that we use
5:48are actually swarms underneath them. For
5:50example, our favorite Cloud Code and
5:53Codex are actually systems of agents, so
5:55they're technically swarms. And they are
5:57incredibly valuable and actually very
6:00useful and we'll see how just how much
6:02they kick ass later.
6:04Uh they are what's called a form of a
6:06central orchestrator type of an agent or
6:08or swarm.
6:09There are a bunch of others and like to
6:11give a special shout out specifically to
6:13LangGraph. It's something that we're
6:14going to look at later in the
6:16presentation, but that is what allows us
6:18to actually create much more controlled
6:20swarms where we can provide proper
6:23instructions and actually control what
6:25the agents do and do not do.
6:28All of this is of course very very new.
6:30All of these swarm technologies are
6:31really just coming up this year, maybe a
6:33little bit last year.
6:35So it's it's all bound to change soon
6:37enough of course, but the point is there
6:39are trade-offs to the swarm architecture
6:42that you're going to pick.
6:44So certifications are many isolated
6:47tasks
6:48to be checked and controlled together
6:50and swarms are many agents that like to
6:52do these tasks. Clearly there is some
6:54fit, so I decided to see if it's
6:56actually going to work.
6:59What I did was a series of runs, roughly
7:0244 45 runs. There is the 45th one on my
7:04laptop right now
7:07across a couple of different targets, a
7:10couple of different standards, and a
7:11couple of different modes. So I picked
7:13very very simple standards. I picked ERC
7:1620 actually and we'll we'll see just how
7:18simple it can be or how complex it can
7:20get. And I picked a subsection of the
7:22SSC
7:24smart contract security standard,
7:26its development section, and then I
7:27picked its full standard for the sake of
7:30well seeing how it's going to work with
7:31the bigger corpus.
7:33As far as the modes go, I actually
7:35picked just a singular agent. Again,
7:38it's a bit incorrect because
7:40I use Cloud Code and Codex and that's
7:42technically a swarm underneath, but for
7:44all intents and purposes it was very
7:46very simple. I picked sub agents, so
7:48several Cloud Code models and agents
7:50working together through a loose
7:52orchestration of .md files and prompts
7:55and actually built something out more
7:57custom
7:58Certify which ends up using LangGraph,
8:02custom VMs,
8:04controlled environments with actually a
8:06little bit of a human review there which
8:08could be a little bit of a cheat code,
8:09but we'll see why that's critical for
8:12certifications. And as for targets, I
8:14used Uniswap v2, very small, very
8:17battle-tested um
8:19codebase, and Morpho Blue, uh likewise
8:21small enough and a battle-tested
8:23codebase.
8:25So as far as the actual results, um
8:29happy to say that swarm that I've made
8:32with CertiK actually did test
8:36kind of the best. It actually only
8:38tested as good as Cloud Code itself
8:42without any additional prompts. Uh it's
8:44kind of a testament to just how powerful
8:47these native agents are. Um but as far
8:51as actual verifiability, which is
8:52critical for certifications, we'll see
8:54that it tested even better. Um
8:58an important note is I've tried
8:59different models for my swarm, and it is
9:02a just a night and day difference. In
9:05fact, uh the swarms with Sonnet uh
9:08tested the worst across the board. As
9:11far as the actual rubric here, uh
9:13I cheated a little bit, and I used
9:15agents as well to verify my reports. I
9:17used Fable to go through all the reports
9:19together in a blind test and to go
9:22through a rubric of verifiability, uh
9:25correctness, and if there is actual
9:27argument that follows these reports and
9:29stuff like that.
9:32As far as the first model goes, so the
9:34first uh single agent one prompt, yeah,
9:36that really did incredibly well. It is a
9:39very efficient way to actually certify
9:43things. It's a very efficient way to
9:45review things. Um I think a lot of the
9:48uh people here in the room who use a lot
9:50of custom agents to perform
9:53vulnerability mining and auditing can
9:55attest to just the sheer power of these
9:57native kind of wrappers like Cloud Code
9:59and Codex. They are very, very, very
10:01good. Uh
10:03it it scored the best across the board,
10:06and it was incredibly cheap. Now 30
10:08cents is a little bit of an outlier. I
10:11used uh Codex and that one uh 61 on a
10:15low thinking. Uh I mean, it was still a
10:17good model, still scored very well, and
10:19it was very very cheap. The problem was
10:22that when you give a very simple prompt,
10:24and it was indeed literally find the
10:26certification in your folder, find the
10:28target and apply it, it was one
10:30sentence, the output, of course, looked
10:33very very different uh for each each
10:36run.
10:37So the output was different. Oftentimes
10:39they actually disagreed with each other
10:40because they interpreted the
10:42certifications differently, um because
10:44the certifications often didn't have
10:46actual guidelines.
10:48Uh sometimes they were too lenient. They
10:50read it something as passable, and
10:52sometimes they read it as not passable,
10:54which of course you kind of need to
10:56actually have a decision there.
10:58Uh they
11:00interestingly did not follow directions
11:03for some quite strict things like
11:06actually editing the targets, like the
11:08code, like the Uniswap code and whatnot.
11:09They ended up sometimes editing it as
11:11well, as of course agents sometimes go
11:13rogue. And um they really sucked up with
11:17evidence. They didn't have enough
11:19evidence around their decisions, and
11:21that
11:21course is a really big problem for
11:23certifications. But all in all, it's
11:26very good.
11:28Uh the second mode where I've tried to
11:32kind of paralyze things myself and do it
11:34all through loosely orchestration loose
11:36orchestration of .md files. In fact, I
11:39had something like 17 .md files that
11:42gave guidelines and said how to create
11:44verifier agents and how to do this and
11:47that and kind of create a hierarchy of
11:49sub agents. That was a catastrophe. That
11:52did not work well whatsoever, and I
11:54think again people who have experience
11:58with reviewing agents will know that
12:01if you
12:03if you go about it wrong, it can go
12:05really wrong. And that's exactly what
12:07happened here.
12:08So the good stuff, yeah, the workflows
12:11worked. They were good. They were
12:13getting in right way and they were
12:14feeding the right context and it was
12:16fairly structured and indeed it was
12:18parallel. In fact, it was so parallel
12:20that I ended up running something like
12:2140 agents at the same time at one point.
12:23The bad thing is
12:25that was 40 agents times the initial
12:27context times token cost and I ended up
12:30ballooning my token cost to hundreds of
12:31dollars for each run and that ended up
12:35going on for hours and hours and hours.
12:37Now, if I had infinite token compute, I
12:40guess this could have ended up working
12:42at some point. But the problem is that
12:46it is just incredibly inefficient. And
12:47if you think about it, if you had just a
12:50few people and if you had 100 controls
12:52to go through, would you rather have two
12:54people go through all 100 controls or
12:56would you would you rather have 100
12:58people go through two controls each?
13:00And the answer is actually quite
13:02obvious. You would take the first one
13:05because you would need to have 100
13:07different people in the second case go
13:08through the same context, the same
13:10target every time. It's just
13:11inefficient. So it makes sense
13:14why this works like that in the agent
13:15world too.
13:17And indeed it was quite bad. Another
13:20really bad thing here that happened was
13:21that
13:23once there was a failure at any one
13:25layer, it cascaded across the board.
13:28There was a specific instance with a DNS
13:30failure. It was a tooling error, but
13:32regardless that cascaded across all the
13:34sub agents that are underneath it, the
13:35verifiers, the re-reproducers and it
13:38just it was not a good time.
13:42So certainly something to think about.
13:44In fact, just recently Anthropic put out
13:46research and put out an article saying
13:49begging really people to stop using sub
13:52agents for a lot of the work that a
13:55single agent could do.
13:57So I took both of those learnings and I
14:00sat a little bit with my
14:02agent personnel and we came up with
14:05Certifly, which is a LangGraph based
14:08swarm, meaning it actually orchestrates
14:11in a very direct manner and controls
14:13what agents can and cannot do,
14:15connected with an actual uh little
14:17Python script, program that didn't let
14:20agents to actually write the report
14:21themselves, but picked out from the
14:23agents output and wrote the report.
14:26Um I only settled on two agents this
14:29time. I mean, I did sometimes run a few
14:31instances of that in parallel, but
14:33really I only had two agents, a producer
14:35and a reviewer.
14:37Had a custom VM that would actually be
14:39running tooling and
14:41um like fuzzers and like static
14:43analyzers that would produce very
14:46determined deterministic, excuse me,
14:48output.
14:49And all in all, it worked into something
14:51like this.
14:52Uh yeah, as I mentioned, agents only
14:54proposed reviews and the actual code was
14:57what decided whether or not this is
14:59going to make it into the report.
15:02Uh an important thing here is that
15:04certifications are not just a report.
15:06Certification is a lot more. And with
15:08agents, it is very easy to
15:11um not actually have any of that work to
15:14be done and to kind of hallucinate the
15:16answer. So one of the things you could
15:18have is you could have a report and you
15:19could even have evidence without report,
15:21but it doesn't necessarily mean the
15:23agent went through that review.
15:25So how I did about went about this is I
15:28had some visibility for all processes. I
15:30had some verification checks like hash
15:33files uh or hashes of files that were
15:36were checked, audit trails to make sure
15:37that agents actually read it. Had
15:39canaries where I placed uh incorrect or
15:43things that should trip up the agents
15:45um certifications in the code to see
15:47that actually showed up. And then
15:49finally actually had the human in the
15:50loop to attest. And how I did that was
15:54for every verdict of every control, I
15:56provided a certain amount of metadata
15:59that has to come with the verdict so
16:01that somebody else could go along and
16:02certify that. And that could be a human,
16:05could be not necessarily human. In my
16:06case, I did a little bit of a UI here
16:09where I had an actual evidence kind of
16:11portal where a human could come in and
16:13verify this. And the idea here is that
16:15maybe
16:17it's not agents that need to verify and
16:18test a test everything. Maybe it's still
16:21humans, but agents and swarms can help
16:23us get that information quickly.
16:26In any case, it ranked as best as the
16:29the best agent,
16:31but it also had all the evidence
16:33alongside with it, with the reports,
16:35with the
16:37tooling, and with everything that goes
16:39alongside it.
16:42So yeah,
16:43be very very careful creating swarms cuz
16:46of course they can spiral out of
16:48control.
16:49And note that in the future you might be
16:51creating swarms for other swarms. So
16:54specifically in the case with
16:55certifications, we might be creating
16:57certifications for other agents to kind
16:59of have that reputation network of code.
17:04That's it.
Source and accuracy details
- Names, organizations, technical terms, code and numbers may be misrecognized.
- Source wording and disfluencies are preserved; no editorial corrections were applied.
- Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.
- A full caption-track export does not establish that every spoken word is represented.
- Automatic YouTube captions; not human-verified against the audio.
Source export SHA256: 8e957fc1f619122af2aab4f83b9dcd3d34e983fd98233a52af2ec891e39051c4