DSS 2026 / Day 1 / Amber Hall
We Gave AI the Keys to Our CI — Then We Tried to Hack It
Gustavo Dutra
Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.
Transcript
0:07Hello everyone.
0:09I am Gustavo, and
0:11this is the talk that I'm going to do.
0:17Yeah, so I'm going to talk about how we
0:20built our current agentic systems at the
0:23security engineering team at Polygon to
0:25keep up with the engineering and how we
0:27attacked them to
0:29uncover some vulnerabilities.
0:32Yep, so I'm Gustavo, senior security
0:34engineer at Polygon Labs. I have a heavy
0:37engineering background like offensive
0:38security engineer, and today I build and
0:41break a lot of agents, and this is
0:44basically my
0:46all my work.
0:49So how we begin all the agentic work
0:52that we are doing today there?
0:55Our first problem was triaging bug
0:58bounties. So we built this this guy
1:01called
1:02And we were getting like
1:05200 something reports every month since
1:07the beginning of the year because of AI.
1:09Everyone is using AI reporting a lot of
1:11things.
1:14This are weird. Yeah. So
1:17we have today three models that judges
1:20every report that come, and the findings
1:22go to their own team to handle, accept,
1:25and do the fix if needed, and a human
1:28makes the final call whether the
1:30vulnerability gets accepted or not.
1:34So we needed to keep up with the
1:36engineering team and not only with the
1:38bug bounty reports.
1:41And the engineering was scaling pretty
1:43well and pretty quick. They moved from
1:47just having cloud or Codex or anything
1:49on their CLI to cloud
1:52agents.
1:54Their work today is mostly done from
1:57Slack.
1:58They
2:00200+
2:02pull requests done every week, which is
2:05almost impossible to keep up and audit
2:07manually.
2:09And they AR review all the pull
2:11requests, but we still have some humans
2:13that does the final call on them.
2:16So about this, we have a blog post on
2:19the Polygon Engineering blog from Dom
2:22Felker, which is the responsible for
2:24scaling AI with the engineering.
2:26So the security team also needed to
2:28scale.
2:30And our one agent
2:33that we had on the beginning of the year
2:35became a fleet of agents. We have Janus
2:37for bug bounty reports, triaging.
2:41We have Spectre. There is our internal
2:43red team agent that hunts for
2:46vulnerabilities every day, every time,
2:48and validates these bugs and propose a
2:51fix if they are found.
2:53We have Up Auditor, which is a
2:57self-served
2:59audit
3:00app
3:01that can trigger security audits from
3:04Slack and
3:06build threat models every day and also
3:09audit some some some repositories that
3:12are critical every single day.
3:14And from that, we went to the 17
3:17security agents that we have today
3:19running every day at Polygon Labs
3:22and doing threat models,
3:25alerting from triaging alerts for us,
3:29doing machine checks, like we have we
3:31own every machine at Polygon and we
3:33check for patches and everything, and
3:36also doing security audits, not only
3:38from Up Auditor, but a lot of those
3:40agents are doing security audits on
3:42other repositories.
3:44And we to make this secure, we build for
3:48a compromised agent.
3:50Inputs they are data. We never input
3:52commands on the agents.
3:54We only have write access where is
3:57strictly needed.
3:59Sandbox everything, of course.
4:01And dangerous actions also need to be
4:04triggered on the final for by humans.
4:07And we don't let the or at least we try
4:10to not let the agents socially engineer
4:13the employees of Polygon Labs, so we
4:15don't tag people or carry links from the
4:18output of the agents.
4:21Uh
4:23So yeah, we assume that every agent is
4:26compromised. We threat model them and
4:28then we attack them ourselves.
4:31And how we attacked the agents?
4:35Before uh
4:38firing the first payload to any agent,
4:40we map every read and write that we
4:43have. We threat model them with our own
4:46tailor skills that we have in house.
4:49Uh
4:50we fire the payloads that we tailor for
4:53each agent.
4:55And for this we use prompt full some AI
4:58bring
4:59libraries and things that we gather
5:01every day from our own agents.
5:06So what the models did well when we
5:08attacked them? They checked it, decided
5:11sources. They spotted a lot of
5:13injections.
5:14They asked it before rights and
5:18some agents detected and cited attacks
5:21that were made and logged by other
5:23agents.
5:24There was one special attack that we did
5:26and got through. We
5:30in did a injection of this footer on a
5:34message that was sent to a agent.
5:36And
5:38Wait.
5:40So it got this text and it it bypassed
5:46some checks and went through the other
5:47agents and this could be used to
5:50poison not to attack this agent
5:52specifically, but also to poison the
5:54other agents that we have in fleet and
5:56scale for uh
5:58more critical attack.
6:01Uh and
6:02how we fix this? We fix this not on the
6:05model because the models usually detect
6:07well the
6:09uh prompt injection injections and
6:11things that we have today, Opus, GLM,
6:13the other models they are like pretty
6:15good on that, but we need to fix this on
6:18the the infrastructure and on the
6:20pipeline. So we tag data
6:23uh and the agents now know uh that this
6:25output is coming from another agent and
6:27not from a human or trusted source.
6:31And I brought finally uh
6:34case study
6:36of uh attack that we did on internal
6:38agent from the engineering team.
6:41This is Geo.
6:42It was
6:44kind of the first uh
6:46cloud solution that we did where you
6:49input uh
6:52uh a prompt in English and the output is
6:55a pull request.
6:56Uh
6:57it had its own sandbox and a web app and
7:00you could trigger them from from Geo on
7:03Slack.
7:04Every session is hoping to the whole
7:06work because we wanted everything full
7:08transparent, so every engineer could see
7:10what everyone is prompting.
7:14And it was the first version of what we
7:16use today that is referenced on that
7:18blog post from Don Felker. Today it's on
7:20Slack as well, but we don't use this bot
7:23anymore, but it was fun when we were
7:25using it.
7:27And to attack Geo, we needed to ask uh
7:30different question.
7:32Every session was holding its own Git
7:34credential
7:35and the question was can I become an
7:38engineer and push code without anyone
7:40noticing because everything was
7:42transparent. So
7:45uh of course I can write code, but can I
7:47write code impersonating another
7:49engineer and not being detected?
7:51And yes, we could because every session
7:55also has a shell on that on that system.
7:58So we opened it uh on the web browser or
8:01on the on the Slack and every session
8:05new newly created session made a shell
8:08on
8:09to do the pull request.
8:11And this is what we attacked. So the
8:13five steps that we did to get the shell
8:16from other people
8:18were list every running session, which
8:20was pretty easy because everything was
8:23transparent, get auto added as a member,
8:25which also happened
8:28instantaneously,
8:30and subscribed over the web socket that
8:32was exposed.
8:37And we when we subscribed to the web
8:39socket, we received
8:42terminal token that was decrypted and we
8:46could open the shell that was accessed
8:49to access that machine directly and not
8:51through the the solution that they made.
8:57And so this is the chain that we did on
9:01HTTP requests. We subscribed it, we
9:05opened the the open socket session, and
9:08from that we were able to just SSH on
9:10the machine and write anything and
9:13everything that was written there were
9:15was
9:16opening a pull request on the name of
9:18the engineer that was using the
9:20solution.
9:21We can play the demo video, but it's the
9:23same thing but showing exactly the steps
9:27and getting the shell and doing some
9:29nasty stuff there.
9:31So with this attack and this the
9:34analysis of the scenario in Geo and
9:36everything that we learned today
9:39was the model refusals are not security
9:42boundary.
9:45Most of the models today
9:47refuses a lot of prompt injections
9:50in general, but there is a lot of other
9:52security boundaries like that SSH token
9:55or other things that we can do to like
9:58access the agent work for from other
10:01channels.
10:02And the agent output is never to be
10:04trusted.
10:07The worst bug on that was the plane
10:09access control, not something on the
10:10agent itself.
10:12And the attack
10:14uh against the the agents are done every
10:17day after any change. So
10:20as I said, we have 17 agents only on the
10:23security teams, not counting all the
10:25other agents that the engineering are
10:27using. And every time we see a change on
10:29the code that handles that agent, we
10:32fire the same attacks to check again uh
10:35if they are vulnerable to this kind of
10:37attack, not only prompt injections, but
10:39also attacks on other channels as well.
10:46And that's it. Thank you.
10:49If you have any questions, you can
10:51contact me at
10:53on Telegram and that's it.
Source and accuracy details
- Names, organizations, technical terms, code and numbers may be misrecognized.
- Source wording and disfluencies are preserved; no editorial corrections were applied.
- Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.
- A full caption-track export does not establish that every spoken word is represented.
- Automatic YouTube captions; not human-verified against the audio.
Source export SHA256: 7b6362702082d47a11cf321783f21ddea16bf0a99a5322fc9e8c00cd3b9c4e4f