DSS

DSS 2026 / Day 1 / Amber Hall

We Gave AI the Keys to Our CI — Then We Tried to Hack It

Gustavo Dutra

10:58 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:07Hello everyone.

0:09I am Gustavo, and

0:11this is the talk that I'm going to do.

0:17Yeah, so I'm going to talk about how we

0:20built our current agentic systems at the

0:23security engineering team at Polygon to

0:25keep up with the engineering and how we

0:27attacked them to

0:29uncover some vulnerabilities.

0:32Yep, so I'm Gustavo, senior security

0:34engineer at Polygon Labs. I have a heavy

0:37engineering background like offensive

0:38security engineer, and today I build and

0:41break a lot of agents, and this is

0:44basically my

0:46all my work.

0:49So how we begin all the agentic work

0:52that we are doing today there?

0:55Our first problem was triaging bug

0:58bounties. So we built this this guy

1:01called

1:02And we were getting like

1:05200 something reports every month since

1:07the beginning of the year because of AI.

1:09Everyone is using AI reporting a lot of

1:11things.

1:14This are weird. Yeah. So

1:17we have today three models that judges

1:20every report that come, and the findings

1:22go to their own team to handle, accept,

1:25and do the fix if needed, and a human

1:28makes the final call whether the

1:30vulnerability gets accepted or not.

1:34So we needed to keep up with the

1:36engineering team and not only with the

1:38bug bounty reports.

1:41And the engineering was scaling pretty

1:43well and pretty quick. They moved from

1:47just having cloud or Codex or anything

1:49on their CLI to cloud

1:52agents.

1:54Their work today is mostly done from

1:57Slack.

1:58They

2:00200+

2:02pull requests done every week, which is

2:05almost impossible to keep up and audit

2:07manually.

2:09And they AR review all the pull

2:11requests, but we still have some humans

2:13that does the final call on them.

2:16So about this, we have a blog post on

2:19the Polygon Engineering blog from Dom

2:22Felker, which is the responsible for

2:24scaling AI with the engineering.

2:26So the security team also needed to

2:28scale.

2:30And our one agent

2:33that we had on the beginning of the year

2:35became a fleet of agents. We have Janus

2:37for bug bounty reports, triaging.

2:41We have Spectre. There is our internal

2:43red team agent that hunts for

2:46vulnerabilities every day, every time,

2:48and validates these bugs and propose a

2:51fix if they are found.

2:53We have Up Auditor, which is a

2:57self-served

2:59audit

3:00app

3:01that can trigger security audits from

3:04Slack and

3:06build threat models every day and also

3:09audit some some some repositories that

3:12are critical every single day.

3:14And from that, we went to the 17

3:17security agents that we have today

3:19running every day at Polygon Labs

3:22and doing threat models,

3:25alerting from triaging alerts for us,

3:29doing machine checks, like we have we

3:31own every machine at Polygon and we

3:33check for patches and everything, and

3:36also doing security audits, not only

3:38from Up Auditor, but a lot of those

3:40agents are doing security audits on

3:42other repositories.

3:44And we to make this secure, we build for

3:48a compromised agent.

3:50Inputs they are data. We never input

3:52commands on the agents.

3:54We only have write access where is

3:57strictly needed.

3:59Sandbox everything, of course.

4:01And dangerous actions also need to be

4:04triggered on the final for by humans.

4:07And we don't let the or at least we try

4:10to not let the agents socially engineer

4:13the employees of Polygon Labs, so we

4:15don't tag people or carry links from the

4:18output of the agents.

4:21Uh

4:23So yeah, we assume that every agent is

4:26compromised. We threat model them and

4:28then we attack them ourselves.

4:31And how we attacked the agents?

4:35Before uh

4:38firing the first payload to any agent,

4:40we map every read and write that we

4:43have. We threat model them with our own

4:46tailor skills that we have in house.

4:49Uh

4:50we fire the payloads that we tailor for

4:53each agent.

4:55And for this we use prompt full some AI

4:58bring

4:59libraries and things that we gather

5:01every day from our own agents.

5:06So what the models did well when we

5:08attacked them? They checked it, decided

5:11sources. They spotted a lot of

5:13injections.

5:14They asked it before rights and

5:18some agents detected and cited attacks

5:21that were made and logged by other

5:23agents.

5:24There was one special attack that we did

5:26and got through. We

5:30in did a injection of this footer on a

5:34message that was sent to a agent.

5:36And

5:38Wait.

5:40So it got this text and it it bypassed

5:46some checks and went through the other

5:47agents and this could be used to

5:50poison not to attack this agent

5:52specifically, but also to poison the

5:54other agents that we have in fleet and

5:56scale for uh

5:58more critical attack.

6:01Uh and

6:02how we fix this? We fix this not on the

6:05model because the models usually detect

6:07well the

6:09uh prompt injection injections and

6:11things that we have today, Opus, GLM,

6:13the other models they are like pretty

6:15good on that, but we need to fix this on

6:18the the infrastructure and on the

6:20pipeline. So we tag data

6:23uh and the agents now know uh that this

6:25output is coming from another agent and

6:27not from a human or trusted source.

6:31And I brought finally uh

6:34case study

6:36of uh attack that we did on internal

6:38agent from the engineering team.

6:41This is Geo.

6:42It was

6:44kind of the first uh

6:46cloud solution that we did where you

6:49input uh

6:52uh a prompt in English and the output is

6:55a pull request.

6:56Uh

6:57it had its own sandbox and a web app and

7:00you could trigger them from from Geo on

7:03Slack.

7:04Every session is hoping to the whole

7:06work because we wanted everything full

7:08transparent, so every engineer could see

7:10what everyone is prompting.

7:14And it was the first version of what we

7:16use today that is referenced on that

7:18blog post from Don Felker. Today it's on

7:20Slack as well, but we don't use this bot

7:23anymore, but it was fun when we were

7:25using it.

7:27And to attack Geo, we needed to ask uh

7:30different question.

7:32Every session was holding its own Git

7:34credential

7:35and the question was can I become an

7:38engineer and push code without anyone

7:40noticing because everything was

7:42transparent. So

7:45uh of course I can write code, but can I

7:47write code impersonating another

7:49engineer and not being detected?

7:51And yes, we could because every session

7:55also has a shell on that on that system.

7:58So we opened it uh on the web browser or

8:01on the on the Slack and every session

8:05new newly created session made a shell

8:08on

8:09to do the pull request.

8:11And this is what we attacked. So the

8:13five steps that we did to get the shell

8:16from other people

8:18were list every running session, which

8:20was pretty easy because everything was

8:23transparent, get auto added as a member,

8:25which also happened

8:28instantaneously,

8:30and subscribed over the web socket that

8:32was exposed.

8:37And we when we subscribed to the web

8:39socket, we received

8:42terminal token that was decrypted and we

8:46could open the shell that was accessed

8:49to access that machine directly and not

8:51through the the solution that they made.

8:57And so this is the chain that we did on

9:01HTTP requests. We subscribed it, we

9:05opened the the open socket session, and

9:08from that we were able to just SSH on

9:10the machine and write anything and

9:13everything that was written there were

9:15was

9:16opening a pull request on the name of

9:18the engineer that was using the

9:20solution.

9:21We can play the demo video, but it's the

9:23same thing but showing exactly the steps

9:27and getting the shell and doing some

9:29nasty stuff there.

9:31So with this attack and this the

9:34analysis of the scenario in Geo and

9:36everything that we learned today

9:39was the model refusals are not security

9:42boundary.

9:45Most of the models today

9:47refuses a lot of prompt injections

9:50in general, but there is a lot of other

9:52security boundaries like that SSH token

9:55or other things that we can do to like

9:58access the agent work for from other

10:01channels.

10:02And the agent output is never to be

10:04trusted.

10:07The worst bug on that was the plane

10:09access control, not something on the

10:10agent itself.

10:12And the attack

10:14uh against the the agents are done every

10:17day after any change. So

10:20as I said, we have 17 agents only on the

10:23security teams, not counting all the

10:25other agents that the engineering are

10:27using. And every time we see a change on

10:29the code that handles that agent, we

10:32fire the same attacks to check again uh

10:35if they are vulnerable to this kind of

10:37attack, not only prompt injections, but

10:39also attacks on other channels as well.

10:46And that's it. Thank you.

10:49If you have any questions, you can

10:51contact me at

10:53on Telegram and that's it.

Source and accuracy details

Source export SHA256: 7b6362702082d47a11cf321783f21ddea16bf0a99a5322fc9e8c00cd3b9c4e4f