DSS 2026 — Certiflai: orchestrating swarms for certifications George Kobakhidze Source video: https://www.youtube.com/watch?v=Xgfle-pNSMg Automatic captions — not human-reviewed. Complete spoken-audio coverage is unverified. Source language tag: en (English). Source export SHA256: 8e957fc1f619122af2aab4f83b9dcd3d34e983fd98233a52af2ec891e39051c4 [0:00] Okay, perfect. [0:02] Hi everyone. [0:03] So today we're going to be talking about [0:06] orchestrating swarms. [0:08] Um, specifically for certifications that [0:11] was [0:13] the use case I took for this, but I [0:14] think this can be applied actually quite [0:16] a bit across the board for security work [0:19] and it's quite quite um [0:21] appropriate especially nowadays. [0:25] Yes, so a little butterfly certify [0:28] butterfly. [0:30] So a little bit about me. My name is [0:32] George. I work with Consensus Diligence. [0:35] Uh, [0:37] in particular I work quite a bit kind of [0:40] in my own spare time outside of audits [0:42] and security work on standards and [0:44] frameworks. [0:45] Some of those are BSSC standards, uh, [0:49] EEA, the old school OG Ethereum [0:52] Enterprise Alliance and most recently as [0:55] well seal frameworks though haven't done [0:57] as much work there as I have with the [0:58] other two. Special shout out to BSSC. [1:01] They've recently released [1:03] a second version of their smart contract [1:05] security standard, which is actually [1:08] going to be a big target of my [1:09] presentation today. And as part of that [1:12] release, they've also [1:14] added audit guidelines for each control [1:17] in the standards, which was actually [1:18] very very important for my work. [1:21] And then in some of the other time that [1:23] I that I have, I also work quite a bit [1:25] on the AI security internally diligence [1:27] and most recently I've been working a [1:28] lot on vulnerability mining to actually [1:31] well mine vulnerabilities using agents [1:34] and swarms and AI and whatnot. [1:37] So why standards? [1:39] Um, this is not a very popular I would [1:42] say topic in our industry sadly, but it [1:44] is actually quite crucial. If we take a [1:47] look at some of the other [1:49] mission critical industries out there in [1:51] the world, we'll see that all of them [1:52] have very strict standards that [1:56] their suppliers such as their software [1:58] suppliers or they themselves to abide [1:59] by. [2:00] Um, are across aviation, energy, [2:03] healthcare, banking, you know, they all [2:05] have actually fairly strict standards [2:08] and uh this allows them to have common [2:11] solutions to common problems. This [2:13] allows them to have common interfaces to [2:15] share expertise that they can build upon [2:18] uh [2:18] amongst their employees or partners. [2:20] It's it's very much um [2:23] a very useful thing. [2:24] Within Ethereum, we don't really have [2:26] all that much success across standards [2:29] in general. We do of course have a lot [2:30] of success across technical standards. [2:33] You know, uh we can talk about the [2:34] actual very infrastructure level and [2:37] kind of foundation level standards, you [2:38] know, network standards, you know, [2:39] Ethereum itself is a specification. It [2:42] is a standard. [2:43] Uh but we can talk a little bit more [2:44] higher level like even token standards [2:46] like ERC -20, tremendously successful um [2:50] technical standard for common, you know, [2:52] use cases and communication for tokens. [2:54] But anything that has has to do with [2:56] processes, we really really aren't very [2:59] good there. And as we can see lately, a [3:01] lot of processes [3:03] are the places where things get [3:04] exploited. And in fact, just recently, [3:07] BSCC also released a uh [3:10] transaction lifecycle uh framework and [3:12] standard that potentially could have [3:13] helped with a lot of the signature-based [3:15] exploits we've we've seen lately. [3:18] Uh speaking about standards, we can we [3:20] of course have to talk about [3:21] certification as that is the uh the the [3:24] the main topic here. Um [3:26] it's very quite it's very simple. So for [3:28] us, we can define it as just a an [3:31] attestation to standards compliance. [3:34] Uh it is very much [3:36] uh based on the reputation of the [3:38] certifier, I would say. So how much you [3:40] trust a certification depends on how [3:42] much you trust the certifier. So [3:43] reputation is very important here. And [3:45] in fact, this is going to be a big [3:46] problem for [3:48] doing stuff like certifying things with [3:50] agents because of course reputation is [3:52] quite finicky there. [3:54] For Web3 specifically, uh certification [3:57] and standards are very hard to scale and [4:00] make work kind of as I mentioned [4:01] earlier. [4:02] So, the challenges there are [4:05] and that's also shared with other [4:07] mission-critical industries. It's it's [4:08] very expensive to do these [4:09] certifications. You need expertise. You [4:11] need very very uh knowledgeable people [4:14] and their time is simply expensive. They [4:16] simply do not like to do that. [4:18] And oftentimes all of this is helped by, [4:20] frankly, the government. It's helped by [4:23] uh government bodies that subsidize [4:24] these things and enforce standards and [4:28] frameworks to be uh to be done within [4:30] certain companies and industries. Uh we [4:34] of course are famously not very good [4:36] with governments. Uh you know lately we [4:38] have a lot of institutions coming into [4:40] the space, so maybe that's going to [4:41] change. But that certainly has been the [4:44] case up until very very recently. And [4:46] people simply aren't excited about [4:48] performing standards or certifications [4:50] for processes across the board. And I [4:51] understand it. Even I am not really into [4:54] doing all that. I like to contribute to [4:55] them, certainly get a lot of value out [4:57] of it. [4:58] But actually applying them is not so [5:00] fun. [5:01] Agents, however, don't have a concept of [5:03] fun. They like to do menial work. They [5:06] thrive with instructions, especially [5:09] when they're within an isolated kind of [5:12] independent control. And that is [5:13] precisely what certifications are. And [5:16] of course we can parallelize our agents [5:19] and we can do hundreds of thousands of [5:21] hours within just a few hours if you [5:23] have enough compute to do that. So, it [5:25] could be quite quite interesting. [5:28] Uh very quickly I want to touch uh a [5:30] little bit about swarms um because of [5:34] course that's what we're going to be [5:35] talking about today. So, if we think of [5:37] an agent as a singular item, a swarm is [5:40] a plural of that. [5:41] That's that's really quite simple. Uh of [5:43] course it's not actually as simple as [5:46] that. Nowadays many agents that we use [5:48] are actually swarms underneath them. For [5:50] example, our favorite Cloud Code and [5:53] Codex are actually systems of agents, so [5:55] they're technically swarms. And they are [5:57] incredibly valuable and actually very [6:00] useful and we'll see how just how much [6:02] they kick ass later. [6:04] Uh they are what's called a form of a [6:06] central orchestrator type of an agent or [6:08] or swarm. [6:09] There are a bunch of others and like to [6:11] give a special shout out specifically to [6:13] LangGraph. It's something that we're [6:14] going to look at later in the [6:16] presentation, but that is what allows us [6:18] to actually create much more controlled [6:20] swarms where we can provide proper [6:23] instructions and actually control what [6:25] the agents do and do not do. [6:28] All of this is of course very very new. [6:30] All of these swarm technologies are [6:31] really just coming up this year, maybe a [6:33] little bit last year. [6:35] So it's it's all bound to change soon [6:37] enough of course, but the point is there [6:39] are trade-offs to the swarm architecture [6:42] that you're going to pick. [6:44] So certifications are many isolated [6:47] tasks [6:48] to be checked and controlled together [6:50] and swarms are many agents that like to [6:52] do these tasks. Clearly there is some [6:54] fit, so I decided to see if it's [6:56] actually going to work. [6:59] What I did was a series of runs, roughly [7:02] 44 45 runs. There is the 45th one on my [7:04] laptop right now [7:07] across a couple of different targets, a [7:10] couple of different standards, and a [7:11] couple of different modes. So I picked [7:13] very very simple standards. I picked ERC [7:16] 20 actually and we'll we'll see just how [7:18] simple it can be or how complex it can [7:20] get. And I picked a subsection of the [7:22] SSC [7:24] smart contract security standard, [7:26] its development section, and then I [7:27] picked its full standard for the sake of [7:30] well seeing how it's going to work with [7:31] the bigger corpus. [7:33] As far as the modes go, I actually [7:35] picked just a singular agent. Again, [7:38] it's a bit incorrect because [7:40] I use Cloud Code and Codex and that's [7:42] technically a swarm underneath, but for [7:44] all intents and purposes it was very [7:46] very simple. I picked sub agents, so [7:48] several Cloud Code models and agents [7:50] working together through a loose [7:52] orchestration of .md files and prompts [7:55] and actually built something out more [7:57] custom [7:58] Certify which ends up using LangGraph, [8:02] custom VMs, [8:04] controlled environments with actually a [8:06] little bit of a human review there which [8:08] could be a little bit of a cheat code, [8:09] but we'll see why that's critical for [8:12] certifications. And as for targets, I [8:14] used Uniswap v2, very small, very [8:17] battle-tested um [8:19] codebase, and Morpho Blue, uh likewise [8:21] small enough and a battle-tested [8:23] codebase. [8:25] So as far as the actual results, um [8:29] happy to say that swarm that I've made [8:32] with CertiK actually did test [8:36] kind of the best. It actually only [8:38] tested as good as Cloud Code itself [8:42] without any additional prompts. Uh it's [8:44] kind of a testament to just how powerful [8:47] these native agents are. Um but as far [8:51] as actual verifiability, which is [8:52] critical for certifications, we'll see [8:54] that it tested even better. Um [8:58] an important note is I've tried [8:59] different models for my swarm, and it is [9:02] a just a night and day difference. In [9:05] fact, uh the swarms with Sonnet uh [9:08] tested the worst across the board. As [9:11] far as the actual rubric here, uh [9:13] I cheated a little bit, and I used [9:15] agents as well to verify my reports. I [9:17] used Fable to go through all the reports [9:19] together in a blind test and to go [9:22] through a rubric of verifiability, uh [9:25] correctness, and if there is actual [9:27] argument that follows these reports and [9:29] stuff like that. [9:32] As far as the first model goes, so the [9:34] first uh single agent one prompt, yeah, [9:36] that really did incredibly well. It is a [9:39] very efficient way to actually certify [9:43] things. It's a very efficient way to [9:45] review things. Um I think a lot of the [9:48] uh people here in the room who use a lot [9:50] of custom agents to perform [9:53] vulnerability mining and auditing can [9:55] attest to just the sheer power of these [9:57] native kind of wrappers like Cloud Code [9:59] and Codex. They are very, very, very [10:01] good. Uh [10:03] it it scored the best across the board, [10:06] and it was incredibly cheap. Now 30 [10:08] cents is a little bit of an outlier. I [10:11] used uh Codex and that one uh 61 on a [10:15] low thinking. Uh I mean, it was still a [10:17] good model, still scored very well, and [10:19] it was very very cheap. The problem was [10:22] that when you give a very simple prompt, [10:24] and it was indeed literally find the [10:26] certification in your folder, find the [10:28] target and apply it, it was one [10:30] sentence, the output, of course, looked [10:33] very very different uh for each each [10:36] run. [10:37] So the output was different. Oftentimes [10:39] they actually disagreed with each other [10:40] because they interpreted the [10:42] certifications differently, um because [10:44] the certifications often didn't have [10:46] actual guidelines. [10:48] Uh sometimes they were too lenient. They [10:50] read it something as passable, and [10:52] sometimes they read it as not passable, [10:54] which of course you kind of need to [10:56] actually have a decision there. [10:58] Uh they [11:00] interestingly did not follow directions [11:03] for some quite strict things like [11:06] actually editing the targets, like the [11:08] code, like the Uniswap code and whatnot. [11:09] They ended up sometimes editing it as [11:11] well, as of course agents sometimes go [11:13] rogue. And um they really sucked up with [11:17] evidence. They didn't have enough [11:19] evidence around their decisions, and [11:21] that [11:21] course is a really big problem for [11:23] certifications. But all in all, it's [11:26] very good. [11:28] Uh the second mode where I've tried to [11:32] kind of paralyze things myself and do it [11:34] all through loosely orchestration loose [11:36] orchestration of .md files. In fact, I [11:39] had something like 17 .md files that [11:42] gave guidelines and said how to create [11:44] verifier agents and how to do this and [11:47] that and kind of create a hierarchy of [11:49] sub agents. That was a catastrophe. That [11:52] did not work well whatsoever, and I [11:54] think again people who have experience [11:58] with reviewing agents will know that [12:01] if you [12:03] if you go about it wrong, it can go [12:05] really wrong. And that's exactly what [12:07] happened here. [12:08] So the good stuff, yeah, the workflows [12:11] worked. They were good. They were [12:13] getting in right way and they were [12:14] feeding the right context and it was [12:16] fairly structured and indeed it was [12:18] parallel. In fact, it was so parallel [12:20] that I ended up running something like [12:21] 40 agents at the same time at one point. [12:23] The bad thing is [12:25] that was 40 agents times the initial [12:27] context times token cost and I ended up [12:30] ballooning my token cost to hundreds of [12:31] dollars for each run and that ended up [12:35] going on for hours and hours and hours. [12:37] Now, if I had infinite token compute, I [12:40] guess this could have ended up working [12:42] at some point. But the problem is that [12:46] it is just incredibly inefficient. And [12:47] if you think about it, if you had just a [12:50] few people and if you had 100 controls [12:52] to go through, would you rather have two [12:54] people go through all 100 controls or [12:56] would you would you rather have 100 [12:58] people go through two controls each? [13:00] And the answer is actually quite [13:02] obvious. You would take the first one [13:05] because you would need to have 100 [13:07] different people in the second case go [13:08] through the same context, the same [13:10] target every time. It's just [13:11] inefficient. So it makes sense [13:14] why this works like that in the agent [13:15] world too. [13:17] And indeed it was quite bad. Another [13:20] really bad thing here that happened was [13:21] that [13:23] once there was a failure at any one [13:25] layer, it cascaded across the board. [13:28] There was a specific instance with a DNS [13:30] failure. It was a tooling error, but [13:32] regardless that cascaded across all the [13:34] sub agents that are underneath it, the [13:35] verifiers, the re-reproducers and it [13:38] just it was not a good time. [13:42] So certainly something to think about. [13:44] In fact, just recently Anthropic put out [13:46] research and put out an article saying [13:49] begging really people to stop using sub [13:52] agents for a lot of the work that a [13:55] single agent could do. [13:57] So I took both of those learnings and I [14:00] sat a little bit with my [14:02] agent personnel and we came up with [14:05] Certifly, which is a LangGraph based [14:08] swarm, meaning it actually orchestrates [14:11] in a very direct manner and controls [14:13] what agents can and cannot do, [14:15] connected with an actual uh little [14:17] Python script, program that didn't let [14:20] agents to actually write the report [14:21] themselves, but picked out from the [14:23] agents output and wrote the report. [14:26] Um I only settled on two agents this [14:29] time. I mean, I did sometimes run a few [14:31] instances of that in parallel, but [14:33] really I only had two agents, a producer [14:35] and a reviewer. [14:37] Had a custom VM that would actually be [14:39] running tooling and [14:41] um like fuzzers and like static [14:43] analyzers that would produce very [14:46] determined deterministic, excuse me, [14:48] output. [14:49] And all in all, it worked into something [14:51] like this. [14:52] Uh yeah, as I mentioned, agents only [14:54] proposed reviews and the actual code was [14:57] what decided whether or not this is [14:59] going to make it into the report. [15:02] Uh an important thing here is that [15:04] certifications are not just a report. [15:06] Certification is a lot more. And with [15:08] agents, it is very easy to [15:11] um not actually have any of that work to [15:14] be done and to kind of hallucinate the [15:16] answer. So one of the things you could [15:18] have is you could have a report and you [15:19] could even have evidence without report, [15:21] but it doesn't necessarily mean the [15:23] agent went through that review. [15:25] So how I did about went about this is I [15:28] had some visibility for all processes. I [15:30] had some verification checks like hash [15:33] files uh or hashes of files that were [15:36] were checked, audit trails to make sure [15:37] that agents actually read it. Had [15:39] canaries where I placed uh incorrect or [15:43] things that should trip up the agents [15:45] um certifications in the code to see [15:47] that actually showed up. And then [15:49] finally actually had the human in the [15:50] loop to attest. And how I did that was [15:54] for every verdict of every control, I [15:56] provided a certain amount of metadata [15:59] that has to come with the verdict so [16:01] that somebody else could go along and [16:02] certify that. And that could be a human, [16:05] could be not necessarily human. In my [16:06] case, I did a little bit of a UI here [16:09] where I had an actual evidence kind of [16:11] portal where a human could come in and [16:13] verify this. And the idea here is that [16:15] maybe [16:17] it's not agents that need to verify and [16:18] test a test everything. Maybe it's still [16:21] humans, but agents and swarms can help [16:23] us get that information quickly. [16:26] In any case, it ranked as best as the [16:29] the best agent, [16:31] but it also had all the evidence [16:33] alongside with it, with the reports, [16:35] with the [16:37] tooling, and with everything that goes [16:39] alongside it. [16:42] So yeah, [16:43] be very very careful creating swarms cuz [16:46] of course they can spiral out of [16:48] control. [16:49] And note that in the future you might be [16:51] creating swarms for other swarms. So [16:54] specifically in the case with [16:55] certifications, we might be creating [16:57] certifications for other agents to kind [16:59] of have that reputation network of code. [17:04] That's it.