DSS 2026 — We Gave AI the Keys to Our CI — Then We Tried to Hack It Gustavo Dutra Source video: https://www.youtube.com/watch?v=y6CclXkTXms Automatic captions — not human-reviewed. Complete spoken-audio coverage is unverified. Source language tag: en (English). Source export SHA256: 7b6362702082d47a11cf321783f21ddea16bf0a99a5322fc9e8c00cd3b9c4e4f [0:07] Hello everyone. [0:09] I am Gustavo, and [0:11] this is the talk that I'm going to do. [0:17] Yeah, so I'm going to talk about how we [0:20] built our current agentic systems at the [0:23] security engineering team at Polygon to [0:25] keep up with the engineering and how we [0:27] attacked them to [0:29] uncover some vulnerabilities. [0:32] Yep, so I'm Gustavo, senior security [0:34] engineer at Polygon Labs. I have a heavy [0:37] engineering background like offensive [0:38] security engineer, and today I build and [0:41] break a lot of agents, and this is [0:44] basically my [0:46] all my work. [0:49] So how we begin all the agentic work [0:52] that we are doing today there? [0:55] Our first problem was triaging bug [0:58] bounties. So we built this this guy [1:01] called [1:02] And we were getting like [1:05] 200 something reports every month since [1:07] the beginning of the year because of AI. [1:09] Everyone is using AI reporting a lot of [1:11] things. [1:14] This are weird. Yeah. So [1:17] we have today three models that judges [1:20] every report that come, and the findings [1:22] go to their own team to handle, accept, [1:25] and do the fix if needed, and a human [1:28] makes the final call whether the [1:30] vulnerability gets accepted or not. [1:34] So we needed to keep up with the [1:36] engineering team and not only with the [1:38] bug bounty reports. [1:41] And the engineering was scaling pretty [1:43] well and pretty quick. They moved from [1:47] just having cloud or Codex or anything [1:49] on their CLI to cloud [1:52] agents. [1:54] Their work today is mostly done from [1:57] Slack. [1:58] They [2:00] 200+ [2:02] pull requests done every week, which is [2:05] almost impossible to keep up and audit [2:07] manually. [2:09] And they AR review all the pull [2:11] requests, but we still have some humans [2:13] that does the final call on them. [2:16] So about this, we have a blog post on [2:19] the Polygon Engineering blog from Dom [2:22] Felker, which is the responsible for [2:24] scaling AI with the engineering. [2:26] So the security team also needed to [2:28] scale. [2:30] And our one agent [2:33] that we had on the beginning of the year [2:35] became a fleet of agents. We have Janus [2:37] for bug bounty reports, triaging. [2:41] We have Spectre. There is our internal [2:43] red team agent that hunts for [2:46] vulnerabilities every day, every time, [2:48] and validates these bugs and propose a [2:51] fix if they are found. [2:53] We have Up Auditor, which is a [2:57] self-served [2:59] audit [3:00] app [3:01] that can trigger security audits from [3:04] Slack and [3:06] build threat models every day and also [3:09] audit some some some repositories that [3:12] are critical every single day. [3:14] And from that, we went to the 17 [3:17] security agents that we have today [3:19] running every day at Polygon Labs [3:22] and doing threat models, [3:25] alerting from triaging alerts for us, [3:29] doing machine checks, like we have we [3:31] own every machine at Polygon and we [3:33] check for patches and everything, and [3:36] also doing security audits, not only [3:38] from Up Auditor, but a lot of those [3:40] agents are doing security audits on [3:42] other repositories. [3:44] And we to make this secure, we build for [3:48] a compromised agent. [3:50] Inputs they are data. We never input [3:52] commands on the agents. [3:54] We only have write access where is [3:57] strictly needed. [3:59] Sandbox everything, of course. [4:01] And dangerous actions also need to be [4:04] triggered on the final for by humans. [4:07] And we don't let the or at least we try [4:10] to not let the agents socially engineer [4:13] the employees of Polygon Labs, so we [4:15] don't tag people or carry links from the [4:18] output of the agents. [4:21] Uh [4:23] So yeah, we assume that every agent is [4:26] compromised. We threat model them and [4:28] then we attack them ourselves. [4:31] And how we attacked the agents? [4:35] Before uh [4:38] firing the first payload to any agent, [4:40] we map every read and write that we [4:43] have. We threat model them with our own [4:46] tailor skills that we have in house. [4:49] Uh [4:50] we fire the payloads that we tailor for [4:53] each agent. [4:55] And for this we use prompt full some AI [4:58] bring [4:59] libraries and things that we gather [5:01] every day from our own agents. [5:06] So what the models did well when we [5:08] attacked them? They checked it, decided [5:11] sources. They spotted a lot of [5:13] injections. [5:14] They asked it before rights and [5:18] some agents detected and cited attacks [5:21] that were made and logged by other [5:23] agents. [5:24] There was one special attack that we did [5:26] and got through. We [5:30] in did a injection of this footer on a [5:34] message that was sent to a agent. [5:36] And [5:38] Wait. [5:40] So it got this text and it it bypassed [5:46] some checks and went through the other [5:47] agents and this could be used to [5:50] poison not to attack this agent [5:52] specifically, but also to poison the [5:54] other agents that we have in fleet and [5:56] scale for uh [5:58] more critical attack. [6:01] Uh and [6:02] how we fix this? We fix this not on the [6:05] model because the models usually detect [6:07] well the [6:09] uh prompt injection injections and [6:11] things that we have today, Opus, GLM, [6:13] the other models they are like pretty [6:15] good on that, but we need to fix this on [6:18] the the infrastructure and on the [6:20] pipeline. So we tag data [6:23] uh and the agents now know uh that this [6:25] output is coming from another agent and [6:27] not from a human or trusted source. [6:31] And I brought finally uh [6:34] case study [6:36] of uh attack that we did on internal [6:38] agent from the engineering team. [6:41] This is Geo. [6:42] It was [6:44] kind of the first uh [6:46] cloud solution that we did where you [6:49] input uh [6:52] uh a prompt in English and the output is [6:55] a pull request. [6:56] Uh [6:57] it had its own sandbox and a web app and [7:00] you could trigger them from from Geo on [7:03] Slack. [7:04] Every session is hoping to the whole [7:06] work because we wanted everything full [7:08] transparent, so every engineer could see [7:10] what everyone is prompting. [7:14] And it was the first version of what we [7:16] use today that is referenced on that [7:18] blog post from Don Felker. Today it's on [7:20] Slack as well, but we don't use this bot [7:23] anymore, but it was fun when we were [7:25] using it. [7:27] And to attack Geo, we needed to ask uh [7:30] different question. [7:32] Every session was holding its own Git [7:34] credential [7:35] and the question was can I become an [7:38] engineer and push code without anyone [7:40] noticing because everything was [7:42] transparent. So [7:45] uh of course I can write code, but can I [7:47] write code impersonating another [7:49] engineer and not being detected? [7:51] And yes, we could because every session [7:55] also has a shell on that on that system. [7:58] So we opened it uh on the web browser or [8:01] on the on the Slack and every session [8:05] new newly created session made a shell [8:08] on [8:09] to do the pull request. [8:11] And this is what we attacked. So the [8:13] five steps that we did to get the shell [8:16] from other people [8:18] were list every running session, which [8:20] was pretty easy because everything was [8:23] transparent, get auto added as a member, [8:25] which also happened [8:28] instantaneously, [8:30] and subscribed over the web socket that [8:32] was exposed. [8:37] And we when we subscribed to the web [8:39] socket, we received [8:42] terminal token that was decrypted and we [8:46] could open the shell that was accessed [8:49] to access that machine directly and not [8:51] through the the solution that they made. [8:57] And so this is the chain that we did on [9:01] HTTP requests. We subscribed it, we [9:05] opened the the open socket session, and [9:08] from that we were able to just SSH on [9:10] the machine and write anything and [9:13] everything that was written there were [9:15] was [9:16] opening a pull request on the name of [9:18] the engineer that was using the [9:20] solution. [9:21] We can play the demo video, but it's the [9:23] same thing but showing exactly the steps [9:27] and getting the shell and doing some [9:29] nasty stuff there. [9:31] So with this attack and this the [9:34] analysis of the scenario in Geo and [9:36] everything that we learned today [9:39] was the model refusals are not security [9:42] boundary. [9:45] Most of the models today [9:47] refuses a lot of prompt injections [9:50] in general, but there is a lot of other [9:52] security boundaries like that SSH token [9:55] or other things that we can do to like [9:58] access the agent work for from other [10:01] channels. [10:02] And the agent output is never to be [10:04] trusted. [10:07] The worst bug on that was the plane [10:09] access control, not something on the [10:10] agent itself. [10:12] And the attack [10:14] uh against the the agents are done every [10:17] day after any change. So [10:20] as I said, we have 17 agents only on the [10:23] security teams, not counting all the [10:25] other agents that the engineering are [10:27] using. And every time we see a change on [10:29] the code that handles that agent, we [10:32] fire the same attacks to check again uh [10:35] if they are vulnerable to this kind of [10:37] attack, not only prompt injections, but [10:39] also attacks on other channels as well. [10:46] And that's it. Thank you. [10:49] If you have any questions, you can [10:51] contact me at [10:53] on Telegram and that's it.