DSS

DSS 2026 / Day 2 / Amber Hall

A Bug's Life: Why We Ran Our Own Audit Competition

Nikita Belenkov

25:43 · Recording

Watch on YouTube ↗Open recording viewerOriginal programme entry
Download plain textDownload timestamped JSON

Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.

Transcript

0:01Hi, good morning. I'm Nikita. I am I

0:04work at Anza. And yeah, we're going to

0:06talk about some bug bounty stuff,

0:09mainly a bug bounty competition that we

0:10ran for AlpenGlow, which is the new

0:13consensus protocol that recently went

0:15live on Solana, which is very exciting.

0:18Yeah, let's kick right off. Yeah, so I'm

0:21Nikita. I do mainly security, anything

0:24and anything related to security, and

0:26this talk was written with me by Colin,

0:29who's another security engineer.

0:30And yeah, at Anza, we engineer Solana's

0:33core protocol. So we work on Agave

0:35mainly, which is the main

0:39basically the main client along with

0:41Firedancer.

0:42And we worked on AlpenGlow, which is the

0:44new consensus protocol. We work on

0:45on-chain programs and a bunch of Solana

0:48SDK stuff like Pinocchio, Solana Care,

0:50Kudama.

0:52Yeah, I'll try to answer basically the

0:53six questions in whatever 30 minutes I

0:56have.

0:58Looking at what is AlpenGlow, why we did

0:59it in house, how did we set it up?

1:03Slightly controversial having a burn

1:04wall or a paywall for submissions.

1:08Agentic triaging of the bugs. What did

1:10the competition show and like what we

1:13ended up learning from it and taking

1:14from our normal bug bounty program to

1:17not completely drown in submissions.

1:21Yeah, so what is AlpenGlow? AlpenGlow is

1:23the new consensus protocol. It's built

1:25on BLS

1:27certificates instead of normal votes. It

1:30replaced Tower BFT along with proof of

1:32history.

1:33It went live

1:35on 21st of October.

1:37We went through migration.

1:39And yeah, so the big change was that it

1:41has a much, much, much lower finality.

1:44The votes are not sent on chain. They

1:47are aggregated as BLS certificates and

1:49then that allows you to do it much

1:51quicker. As I said, there's no proof of

1:52history and we also have a higher

1:54adversarial

1:57basically adversarial percentage that

1:58you can sustain. Uh in Tower BFT you can

2:01sustain 33% adversarial. Here we can do

2:0320 + 20, which is 20% adversarial and

2:0620% offline stake, which is much much

2:09nicer. It's been under development for a

2:12while, and this is kind of talking about

2:14the final stages of how we got it over

2:18basically over the end and

2:20uh yeah.

2:22Uh this is a

2:24[sighs] uh

2:25vibe coded

2:27a vision of how the migration looked

2:29like from our Grafana tools.

2:31[laughter and gasps]

2:32So we went from on the 21st of October

2:34the finality drop from about 10 seconds

2:37down to 0.4 seconds, which was very

2:40exciting at the same time, but also very

2:42uneventful, which is kind of what you

2:43want for a migration.

2:45Um

2:46yeah, we got about 25% faster finality,

2:49which was very very cool.

2:51Um why competition for this? Why was

2:54important? I mean this is quite a It was

2:56This was the biggest change that we've

2:58had on Solana on par with basically the

3:00Ethereum merge.

3:02So we needed to really get

3:03get this stuff right, and we did a a lot

3:07of audits. We worked with OtterSec,

3:08Neodyme, Asymmetric. We did a lot of

3:10fuzzing. I think Arnau will talk about

3:13some of the fuzzing later.

3:14Um

3:16and uh we did We had a special cluster

3:18doing public migrations. We did nine

3:20migration rehearsals,

3:22and uh yeah, and then the final step was

3:24the The way our bug bounty operates is

3:26that the master code is in scope. So

3:28when Otter and Glow wasn't development

3:30it was out of scope, so we needed to

3:31bring it back in. And we were like,

3:33okay, what would be a really

3:35Do we just put it

3:37straight into scope and then see what

3:38happens or this kind of competition

3:40style actually makes a lot of sense?

3:42And then we started looking into what

3:44are the other options and what we could

3:45do. There is Immunefi, there's Sherlock,

3:47there's a lot of cantinas, a lot of

3:49different providers. We've been always

3:51been running our own bug bounty and

3:52GHSAs. We've been self-hosting it,

3:54self-triaging it. So running it in-house

3:56was already an somewhat obvious choice.

3:59Uh a competition does come with its a

4:01bit more scale and more the promotion

4:04side, which that's what we lacked. But

4:07we know, we went, you know what, let's

4:08let's try it. What do we got to lose?

4:10The big ones being we define our own

4:12rules, we control the data, which is

4:14having the biggest one, trusting an

4:16external third party with quite

4:18sensitive findings as a

4:20becomes a an interesting thing uh which

4:23we ideally wanted to avoid.

4:25And then I mean saying that inherently

4:27running it on Gnosis says you trust

4:29GitHub for it. So it's it depends.

4:32Um

4:33yeah, it's triaged by the people who

4:35know the code very well and we keep the

4:37intake. Um yeah, this was the actual

4:39competition was 50,000 SOL. That was the

4:42the full price pool. It was a 12-month

4:44locked SOL. The competition ran for two

4:46weeks. It required POCs. Uh basically

4:49the scope was quite

4:51different. It required it was basically

4:54anything and anything related to Alpine

4:55Global in scope.

4:57Because it's it's like a diff migration

4:59where it's there is the whole validator

5:02and the consensus only part of it. The

5:04most interesting bugs are were either in

5:06migration itself just because it's such

5:07a fragile piece of technology.

5:10And um

5:12it was much less fragile at the time of

5:13migration, but still it's there's just

5:15so many moving parts and so much uh

5:19so much you had to like plan for. But it

5:22was basically anything that touched uh

5:24touched Alpine Global. And that worked

5:26pretty well.

5:27Um the this one was quite interesting. I

5:30don't think I've seen much of this being

5:32done. This is basically what happens if

5:33you take the bug bounty program and you

5:36put it to a competition. Instead of

5:37having a frozen commit, you end up with

5:39a moving master. So

5:42at the time of your submission of your

5:45entry, you evaluate if that bug is live

5:48or not. So there were a lot of cases

5:50where people found the bugs, but at the

5:52time they came to submit, they were

5:53actually not live anymore or we already

5:55publicly disclosed them or they already

5:57got fixed.

5:59It is a bit tough on the researcher. On

6:02the other hand, we also have the ability

6:04to get our fixes reviewed as well

6:06because we had a case where some of the

6:07fixes one of the fixes wasn't

6:09sufficient. So this was

6:11exactly kind of what we wanted.

6:13On the downside that some people had to

6:16restart a lot of work

6:18which is um

6:20just a decision we had to make.

6:22And to make people's life easier we had

6:24a live tracker on on GitHub of the

6:28issues that we are aware about. Though

6:30we aware so that people don't double

6:32burn or double submit and just end up

6:35submitting and we're like oh we know

6:36about this is getting fixed.

6:39Uh yeah this is pretty standards. It was

6:41um

6:42depending on the severity the pool has

6:44more unlock. The difference was that we

6:46don't share the whole pool. The

6:49vulnerabilities paid out at an

6:51evaluation. So for example a DOS I think

6:53was 300 until 500 or 800.

6:58And then the the pool acts as a upper

7:00cap versus the whole pool being shared.

7:03So this is

7:04it's it's it's like a Frankenstein's

7:05monster between competition and bug

7:08bounty basically. [laughter]

7:11Uh but a very nice one.

7:13Uh this one was also quite interesting.

7:15So we did not allow duplicates but we

7:19also did not pay the first submission.

7:22So we paid the first submission who

7:24proved the highest severity.

7:26So we had

7:27uh not this exact case but we had a case

7:29where

7:30the fourth submission actually won

7:32because the initial baseline was a DOS

7:35but they were actually able to prove a

7:37highest severity.

7:38The idea for this was that

7:40we didn't want to people to spray a

7:42bunch of findings across of across like

7:46surfaces that might contain bugs and

7:49hope that they hit something

7:51while

7:52we were encouraging more

7:54much more developed attack scenarios

7:57that may be chain multiple bugs.

7:59Uh and we had one case where this

8:02actually worked. So I'm pretty bullish

8:05on this approach.

8:07Um

8:09yeah I mean, this is the has been the

8:11bane of our existence is um,

8:13with the with the new models or with the

8:15frontier models, not even with the

8:16frontier, with any LLMs, you basically

8:19can pay 10 bucks and spray 50 findings.

8:23Uh, cuz the cost to write a report is

8:25basically near zero. I mean, I put

8:27asterisks on both of them. Cost to write

8:30a bad report or a not necessarily a

8:32well-thought report is near zero.

8:34They're still like I mean, most people

8:36in this room are very good researchers

8:38and people really put effort into it.

8:41Um, is this a pointer? Yeah.

8:44Which means that the triage cost doesn't

8:46really change unless you start doing

8:48agentic triage, but then still the cost

8:50is of triage is so so much higher than

8:53cost of submission. So you have to you

8:54basically have to come up with ways. How

8:56do you make sure the submissions you

8:57receive are actually valid and valid

9:01[clears throat]

9:02like

9:03duplicates are good, uh novel

9:05submissions are good. It's just like

9:06things are actually reachable and

9:07actually have impact.

9:09And uh you basically end up with this

9:12options where the most basic one being

9:15the the working POC, which is basically

9:18all the submissions had.

9:19The problem like it it stops speculative

9:22claims uh and it does take time to

9:25build, but with LLMs it's pretty

9:26straightforward. Being like

9:282FA on GitHub and minimum account age

9:31does help a bit. So we had this three we

9:34had. This one

9:36we we ended up I'll talk a bit more, but

9:39yeah, we ended up requiring a

9:41half-a-soul burn versus a refundable

9:43deposit.

9:45And it helps with some like low-effort

9:47filings.

9:48This is I think the most I'm pretty

9:51bullish on this where I would say it's

9:53like a step beyond proof of concept

9:55where it's a proof of impact where you

9:59have some kind of infrastructure where

10:01you could provably

10:03show that your POC actually shows

10:06impact.

10:07Because the problem with the POC is POC

10:09is only as good as what it shows. You

10:11ask like you ask an LLM, it can write to

10:14you any POC you want. It can prove

10:16whatever.

10:17It's just like you start

10:19mocking different data structures. You

10:21start mocking different areas of the

10:23validator.

10:25And like, oh yes, this is you have I

10:27don't know, network exhaustion. Oh yeah,

10:29but you actually don't have any of the

10:31data center filtering or you don't have

10:34any of the other protections that we

10:36have. You're like, okay, yeah, if you

10:38take that into account, it doesn't work.

10:39[gasps]

10:40I think this is possible to do. The

10:42problem is you kind of need specialized

10:44tooling of how do you define

10:46a correct exploit? How do you define

10:48this impact? I think you can do this for

10:50really big programs like a I think it is

10:53possible. It's just

10:55it becomes a bit

10:57it's tough. Like it's not an easy

10:58definition to do and also you have to

11:00cover so so many cases from like runtime

11:02issues to networking to like

11:05um

11:07yeah, to like consensus

11:08like mapping. It's it's a bit tough, but

11:10I think this is what we really need to

11:12get to.

11:13Um yeah, the next one is like doing KYC

11:16upfront. I don't think this is a good

11:17solution.

11:18It's kind of comes together with a

11:20refundable deposit. If you want to do

11:21the deposit,

11:23as far as I understand, you basically

11:24have to do KYC at most points.

11:27This really stops people submitting. I

11:28think we should only do KYC at payout,

11:31but that's like personal opinion. The

11:33next ones are like invite only. Yeah, I

11:36mean that basically becomes a pseudo

11:37audit, pseudo like you kind of want new

11:41people to look into this, get new pairs

11:42of eyes. And last one is doing like

11:44penalties and bans and stops repeat

11:47offenders, but like drawing the line of

11:49where the penalties and bans go and then

11:51and then you wake up to like a lot of

11:53emails people asking to get unbanned and

11:56like it's it becomes it's a lot of

11:58management. I 'm not sure it really

11:59leads to very good results.

12:02Like

12:03you probably should ban like really

12:04really misbehaving people, but like it's

12:06it's tough.

12:08Um

12:09yeah, but I mean fundamentally like

12:10drawing back to crypto, you either do

12:12proof of work or you do proof of stake.

12:15And there's many more, but like in our

12:16context where proof of work becomes have

12:18you actually done the work? Like can you

12:20show the vulnerability is real? And I

12:22think I mean that is the solution. And

12:25we want like doing some kind of this

12:27proof of impact showing that this stuff

12:29is real in a maybe it's not necessarily

12:32ZK proof, but like some kind of setup

12:35where it's hard to dispute versus

12:38requiring a proof of concept. Everyone

12:40writes a proof of concept in a very

12:42different way and it takes a while to

12:44triage.

12:45So maybe some like unified tooling for

12:47proof of work could be the future

12:48solution. Proof of stake is what we did.

12:51Um

12:53it's the best solution we have at the

12:55time.

12:57I'm not

12:59I think eventually

13:01the best way is you don't charge people,

13:03but that is just what like that's the

13:07best solution currently we have. And it

13:10only works if it hurts more than the

13:12spam pays. And it fundamentally the

13:14amount of payouts we do

13:16the

13:18half a soul is not a huge huge

13:20commitment.

13:21Uh but yeah the that's actually how it

13:23worked.

13:25This we had a I I coded like a

13:27submission portal

13:28when up and going went live uh when the

13:31before the competition.

13:32It required a burn of half a soul, so we

13:35didn't do custodial. So we didn't care

13:37about taking the actual payment for it.

13:39Like the the only purpose of this was to

13:42add a layer where someone thinks twice

13:45if this

13:46think is valid or they will write up a

13:48better report just because there is some

13:50stake and money at stake. That also

13:52means you don't have to deal with refund

13:53and disputes.

13:54And um

13:56yeah it's just like and it also

13:58technically deflationary, which is nice.

14:01[laughter]

14:02Uh and then you basically submit the

14:04transaction hash, we verify it and then

14:07it opens a GitHub GHSA for you. And this

14:10actually worked really well.

14:12Uh

14:13this is what we actually ended up doing

14:14for the triage. So the the green, the

14:16green ones are agentic, and then this

14:19one is human.

14:20It was

14:21This was very much a proof of concept

14:23trial approach where

14:26uh we used a tool called Tracecat.

14:28Tracecat is like a agentic SO like SOAR

14:33tool that allows you It's mainly made

14:35for like web two

14:38triaging of large-scale alerts, but

14:42it actually worked pretty well. It

14:43allowed us somewhere to host the agents,

14:45do this um in a in a nice sustainable

14:48way. We mainly used GPT 5.6 which which

14:52was pretty good. Yeah, it basically

14:53pulls in the pulls in the the claim,

14:56checks the severity, checks the format,

14:58checks uh some some basic checks that

15:00it's like

15:01sane as a submission. Then we do some

15:04evaluation. We run a um

15:07no internet sandbox checking the POC,

15:10matching the claim against uh Alpine

15:13Glow Agave Master at the time because we

15:15also wanted to check if these issues

15:17have already been fixed.

15:19Uh the next one was This is like where

15:22the crème de la crème of this triage

15:23worked was

15:25an agent that was like the whole job was

15:27to convince that this is invalid. Is it

15:30reachable? Is it breaching the 20 plus

15:3220 model? Is it intended? Is it like

15:35Is it just made up? And it did it did a

15:37very good job. And then you kind of

15:39draft it, you check duplicates, and you

15:42rank by similarity, and then you

15:43basically hand it off to a person to be

15:44like, does this make sense?

15:46Final severity payouts, rejection,

15:48whatever. We were very specific that we

15:51did not want

15:53to auto close issues or auto

15:57like decide things because this is still

15:59like a supervised approach

16:01um just because

16:03this was our first attempt, but it

16:05worked very well.

16:06Um

16:07yeah, this is a timeline like lifeline

16:10for of a of a of a report. It went from

16:14like the bug line landing, then gets

16:16pulled from GitHub, case opened, it like

16:19agentic starts, pulls in POC, pulls in

16:23um

16:24pulls in source code. So this one

16:26uh

16:27then

16:28does like more deeper analysis, does the

16:30disproving. In the case of this bug, it

16:33came in as a critical, it went down to

16:35as a low.

16:37And this was actually the first of nine

16:39duplicates, I think.

16:41And then the human kind of then the next

16:43ones got bundled up to this one and um

16:48in the end it was information.

16:50It went through nearly all the stages.

16:52Um yeah, this is the the some fun stats.

16:56It we had 338 submissions over two

16:59weeks. 67% the agent and human agreed,

17:03which is pretty good.

17:05It was nine minutes to an actual agentic

17:07triage. It took humans about 14 hours,

17:10which is still pretty good.

17:12Um

17:13yeah, it worked for about 90% of the

17:15real bugs.

17:16Uh it saved us a lot of time. It

17:19rejected uh all 21 reports that

17:22breached the security model. It was very

17:24good at working out which ones did not

17:26follow the 20 plus 20. That was very

17:28good.

17:29Uh it wasn't so good at

17:31it was like valid bug, but the impact

17:33calculations it was not get good at, but

17:36this is very easy to like this is just

17:38future solving we can do. This is the

17:40competition itself. So we had 115

17:42researchers, 27 findings, valid

17:45findings, and we paid out 27795

17:48SOL.

17:49Uh in terms of like timeline, I'll move

17:51to the to the other side. Um like nearly

17:55within 10 minutes we got six valid

17:56submissions. Within the first hour we

17:58got 12 out of the 27. So nearly like I

18:01think we kind of exhausted the search

18:03space within like a few days. So these

18:05little

18:06like dashes are the winning reports.

18:09And then in the in the whole second week

18:11we only got 15 submissions,

18:13uh which was pretty interesting. Um

18:21and there we go. Uh these uh timings

18:23between different reports. So

18:26we kind of

18:27we didn't

18:28we didn't expect that we will get so so

18:30many duplicates within such a close time

18:32frame where we had a few reports within

18:35seconds of each other

18:37at the time of the opening. And about 58

18:40of the 85 submissions we had were

18:43duplicates.

18:45Um

18:46and nearly all of the duplicates were

18:48within 10 minutes of the first one. So

18:50it was very much like race to the

18:51bottom.

18:52And this is something we're going to

18:53reconsider when we do it next time

18:55because it becomes a bit unfair where

18:57like if you are the submission is within

18:595 seconds

19:01it's like it's this is not a skill-based

19:04competition. It's just like how quickly

19:06can you vibe code a submission thing

19:08which is not ideal.

19:10Um

19:12the other question is would we do it

19:13again? Yes. We would keep the portal and

19:17we actually ended up adopting the burn

19:18for our standing bounty.

19:21Oh

19:22one sec.

19:23Um the next time I think spending more

19:26time on this like

19:28basically feeding back the triage

19:29findings and the human verdicts back

19:31into our like agentic triage I think is

19:33the solution. It worked extremely well

19:36and that is

19:37that is the way we will have to deal

19:39with scale.

19:40I think this advanced POC creation

19:43I'm pretty bullish on it. It requires

19:45defining it very well, but I think

19:47especially for consensus bugs

19:49I think there's a way we can define a

19:52nice system which

19:54not necessarily forces the researcher.

19:56It gives the researcher tools to

19:58actually validate their findings in a

19:59very nice way versus them trying to like

20:02reverse engineer how to do POCs and

20:05which things that actually supposed to

20:06plug in.

20:08And

20:09that would also allow us to

20:11quickly validate these things and

20:13potentially

20:15do this whole process quicker and save

20:17people burns for issues that are not

20:19valid.

20:20And the duplicate policy

20:23is very much similar one. I think we

20:25probably would allow duplicates next

20:27round just because it becomes a bit

20:29I didn't realize how much of a race to

20:31the bottom it will be. So that's just a

20:33learning and I think better

20:34communication with researchers.

20:37We did not do the best of the job

20:40communicating to people partially

20:41because GitHub does not have an API to

20:44write comments and it's just a nightmare

20:46to manage in notifications.

20:48And but there's a lot of easy tooling

20:51you can do and these are pretty easy

20:54things we can improve on.

20:57Yeah, so we ended up migrating from

21:01to our main bug bounty program that is

21:03now also requiring half a soul burn

21:06but we also have a white list for

21:08because our

21:10researchers like at Asymmetric at

21:12Neodyme

21:14they submit their findings also through

21:16the bounty program.

21:18They ended up you end up like not

21:20requiring it and I think there is a

21:22potential where you expand it

21:25to potentially new participants you

21:26require burn for a few for not new you

21:29might not require burn but it's it's

21:31it's kind of experimentation stage.

21:35Yeah, this is actually some stats from

21:38we for about a month that's this thing

21:39has been running.

21:43This so the first two are

21:47the breakdown of the types of

21:49submissions. So the first one is the the

21:51January to

21:541st of January basically the year coming

21:57to 2nd October when we turned it off. We

21:59had about 50% were invalid findings, 31

22:02were duplicates and 19% were novel.

22:04With the burn wall

22:0720 we went from 50% invalid to 22%

22:09invalid which is an excellent reduction

22:13and then we went from 31 invalid to 52%

22:16duplicates. So the actual like amount of

22:18reports didn't go down as much as I

22:20thought.

22:21We went down only like 37% submissions

22:24but I think it'll actually go up because

22:26this is only a month of data.

22:28The the most biting one, I think, the

22:3052% unique researchers reduction.

22:34It does limit it to people who can

22:36afford this burn, and most commonly it

22:38also limits to um

22:41researchers who have been paid out

22:43before in this program.

22:44I think that's the thing. But the

22:46interesting thing that the actual novel

22:48findings, there was basically no change.

22:50I think it was something like 10%.

22:52So that means even though we got so 37%

22:55less findings and 50% less

22:57participating, the actual amount of new

22:59issues found hasn't really changed.

23:02So it kind of implies that this 50% were

23:06basically finding duplicates or invalid

23:08findings.

23:09Uh and we now get about 40% reports per

23:13researcher, which kind of

23:15you start drawing parallels that they're

23:16spending more time looking at the

23:18codebase and validating these findings

23:19and finding new things, which is

23:21kind of what you want. And really the if

23:25you look at if you put it in the context

23:26of the the payouts, our largest like our

23:30if you find a loss of funds in Solana,

23:33uh it's about starts at about 6.5000 SOL

23:36and goes up to

23:3825,000 SOL, I think. So these are quite

23:40large payouts. So even without average

23:43payout, half a SOL is like 0.2% of what

23:47you receive.

23:48Um these are some interesting These are

23:51like our weekly submission rates

23:53for the last year, and I tried to map

23:55them to like different models coming

23:57out.

23:59Um there is really no major parallel, I

24:02think.

24:02[laughter]

24:03All the models come out so quickly. I

24:05mean the summer has been very, very

24:07busy, and I think that's where the

24:09really interesting models came out.

24:11And then October at the end there is

24:13when

24:13we turned on the paywall.

24:15And it's definitely been going steadily

24:18up. And that's just kind of how working

24:23out how do you deal with this and is a I

24:25think a problem that everyone has. And

24:27then if we extend it to like 2025 when

24:32we didn't have these

24:34models,

24:35well, they weren't as good.

24:37So we got six times more submissions

24:41than in 2025 than we did in 2026 until

24:44the paywall.

24:45Um we got we now have three times more

24:48researchers, nearly all new.

24:50And this [clears throat] one is fun. We

24:51have 10 times more EM dashes per

24:54thousand words.

24:55Like this line is EM dashes per thousand

24:58words. And then very interestingly in

25:01January it skyrockets.

25:04[laughter] And then also six times more

25:06markdown tables.

25:08Um

25:10the the thing the line going down I

25:12assume is more the models are using less

25:14EM dashes than

25:17not model usage, but this was an

25:19interesting But yeah, it basically

25:21exploded.

25:22Um but yeah, that's that's basically it.

25:25Thank you very much.

25:26Yeah, basically huge shout out to also

25:28to the Consensus Team Foundation and

25:30everyone who did the triage for

25:32competition. Without them this would not

25:34be possible. And yeah, I'm I'm Belenkov

25:36on Twitter and Telegram and our

25:39submission. Yeah, thank you very much

25:40guys for listening.

25:41>> [applause]

Source and accuracy details

Source export SHA256: f1e445104ea22b3425a3abc52c55c16eb0a8828d0302bf78dba0ed8ed348a826