DSS 2026 / Day 2 / Amber Hall
A Bug's Life: Why We Ran Our Own Audit Competition
Nikita Belenkov
Source: YouTube automatic caption track · Exported 2026-11-16. Timestamps link to the source video. Cue end times were not supplied.
Transcript
0:01Hi, good morning. I'm Nikita. I am I
0:04work at Anza. And yeah, we're going to
0:06talk about some bug bounty stuff,
0:09mainly a bug bounty competition that we
0:10ran for AlpenGlow, which is the new
0:13consensus protocol that recently went
0:15live on Solana, which is very exciting.
0:18Yeah, let's kick right off. Yeah, so I'm
0:21Nikita. I do mainly security, anything
0:24and anything related to security, and
0:26this talk was written with me by Colin,
0:29who's another security engineer.
0:30And yeah, at Anza, we engineer Solana's
0:33core protocol. So we work on Agave
0:35mainly, which is the main
0:39basically the main client along with
0:41Firedancer.
0:42And we worked on AlpenGlow, which is the
0:44new consensus protocol. We work on
0:45on-chain programs and a bunch of Solana
0:48SDK stuff like Pinocchio, Solana Care,
0:50Kudama.
0:52Yeah, I'll try to answer basically the
0:53six questions in whatever 30 minutes I
0:56have.
0:58Looking at what is AlpenGlow, why we did
0:59it in house, how did we set it up?
1:03Slightly controversial having a burn
1:04wall or a paywall for submissions.
1:08Agentic triaging of the bugs. What did
1:10the competition show and like what we
1:13ended up learning from it and taking
1:14from our normal bug bounty program to
1:17not completely drown in submissions.
1:21Yeah, so what is AlpenGlow? AlpenGlow is
1:23the new consensus protocol. It's built
1:25on BLS
1:27certificates instead of normal votes. It
1:30replaced Tower BFT along with proof of
1:32history.
1:33It went live
1:35on 21st of October.
1:37We went through migration.
1:39And yeah, so the big change was that it
1:41has a much, much, much lower finality.
1:44The votes are not sent on chain. They
1:47are aggregated as BLS certificates and
1:49then that allows you to do it much
1:51quicker. As I said, there's no proof of
1:52history and we also have a higher
1:54adversarial
1:57basically adversarial percentage that
1:58you can sustain. Uh in Tower BFT you can
2:01sustain 33% adversarial. Here we can do
2:0320 + 20, which is 20% adversarial and
2:0620% offline stake, which is much much
2:09nicer. It's been under development for a
2:12while, and this is kind of talking about
2:14the final stages of how we got it over
2:18basically over the end and
2:20uh yeah.
2:22Uh this is a
2:24[sighs] uh
2:25vibe coded
2:27a vision of how the migration looked
2:29like from our Grafana tools.
2:31[laughter and gasps]
2:32So we went from on the 21st of October
2:34the finality drop from about 10 seconds
2:37down to 0.4 seconds, which was very
2:40exciting at the same time, but also very
2:42uneventful, which is kind of what you
2:43want for a migration.
2:45Um
2:46yeah, we got about 25% faster finality,
2:49which was very very cool.
2:51Um why competition for this? Why was
2:54important? I mean this is quite a It was
2:56This was the biggest change that we've
2:58had on Solana on par with basically the
3:00Ethereum merge.
3:02So we needed to really get
3:03get this stuff right, and we did a a lot
3:07of audits. We worked with OtterSec,
3:08Neodyme, Asymmetric. We did a lot of
3:10fuzzing. I think Arnau will talk about
3:13some of the fuzzing later.
3:14Um
3:16and uh we did We had a special cluster
3:18doing public migrations. We did nine
3:20migration rehearsals,
3:22and uh yeah, and then the final step was
3:24the The way our bug bounty operates is
3:26that the master code is in scope. So
3:28when Otter and Glow wasn't development
3:30it was out of scope, so we needed to
3:31bring it back in. And we were like,
3:33okay, what would be a really
3:35Do we just put it
3:37straight into scope and then see what
3:38happens or this kind of competition
3:40style actually makes a lot of sense?
3:42And then we started looking into what
3:44are the other options and what we could
3:45do. There is Immunefi, there's Sherlock,
3:47there's a lot of cantinas, a lot of
3:49different providers. We've been always
3:51been running our own bug bounty and
3:52GHSAs. We've been self-hosting it,
3:54self-triaging it. So running it in-house
3:56was already an somewhat obvious choice.
3:59Uh a competition does come with its a
4:01bit more scale and more the promotion
4:04side, which that's what we lacked. But
4:07we know, we went, you know what, let's
4:08let's try it. What do we got to lose?
4:10The big ones being we define our own
4:12rules, we control the data, which is
4:14having the biggest one, trusting an
4:16external third party with quite
4:18sensitive findings as a
4:20becomes a an interesting thing uh which
4:23we ideally wanted to avoid.
4:25And then I mean saying that inherently
4:27running it on Gnosis says you trust
4:29GitHub for it. So it's it depends.
4:32Um
4:33yeah, it's triaged by the people who
4:35know the code very well and we keep the
4:37intake. Um yeah, this was the actual
4:39competition was 50,000 SOL. That was the
4:42the full price pool. It was a 12-month
4:44locked SOL. The competition ran for two
4:46weeks. It required POCs. Uh basically
4:49the scope was quite
4:51different. It required it was basically
4:54anything and anything related to Alpine
4:55Global in scope.
4:57Because it's it's like a diff migration
4:59where it's there is the whole validator
5:02and the consensus only part of it. The
5:04most interesting bugs are were either in
5:06migration itself just because it's such
5:07a fragile piece of technology.
5:10And um
5:12it was much less fragile at the time of
5:13migration, but still it's there's just
5:15so many moving parts and so much uh
5:19so much you had to like plan for. But it
5:22was basically anything that touched uh
5:24touched Alpine Global. And that worked
5:26pretty well.
5:27Um the this one was quite interesting. I
5:30don't think I've seen much of this being
5:32done. This is basically what happens if
5:33you take the bug bounty program and you
5:36put it to a competition. Instead of
5:37having a frozen commit, you end up with
5:39a moving master. So
5:42at the time of your submission of your
5:45entry, you evaluate if that bug is live
5:48or not. So there were a lot of cases
5:50where people found the bugs, but at the
5:52time they came to submit, they were
5:53actually not live anymore or we already
5:55publicly disclosed them or they already
5:57got fixed.
5:59It is a bit tough on the researcher. On
6:02the other hand, we also have the ability
6:04to get our fixes reviewed as well
6:06because we had a case where some of the
6:07fixes one of the fixes wasn't
6:09sufficient. So this was
6:11exactly kind of what we wanted.
6:13On the downside that some people had to
6:16restart a lot of work
6:18which is um
6:20just a decision we had to make.
6:22And to make people's life easier we had
6:24a live tracker on on GitHub of the
6:28issues that we are aware about. Though
6:30we aware so that people don't double
6:32burn or double submit and just end up
6:35submitting and we're like oh we know
6:36about this is getting fixed.
6:39Uh yeah this is pretty standards. It was
6:41um
6:42depending on the severity the pool has
6:44more unlock. The difference was that we
6:46don't share the whole pool. The
6:49vulnerabilities paid out at an
6:51evaluation. So for example a DOS I think
6:53was 300 until 500 or 800.
6:58And then the the pool acts as a upper
7:00cap versus the whole pool being shared.
7:03So this is
7:04it's it's it's like a Frankenstein's
7:05monster between competition and bug
7:08bounty basically. [laughter]
7:11Uh but a very nice one.
7:13Uh this one was also quite interesting.
7:15So we did not allow duplicates but we
7:19also did not pay the first submission.
7:22So we paid the first submission who
7:24proved the highest severity.
7:26So we had
7:27uh not this exact case but we had a case
7:29where
7:30the fourth submission actually won
7:32because the initial baseline was a DOS
7:35but they were actually able to prove a
7:37highest severity.
7:38The idea for this was that
7:40we didn't want to people to spray a
7:42bunch of findings across of across like
7:46surfaces that might contain bugs and
7:49hope that they hit something
7:51while
7:52we were encouraging more
7:54much more developed attack scenarios
7:57that may be chain multiple bugs.
7:59Uh and we had one case where this
8:02actually worked. So I'm pretty bullish
8:05on this approach.
8:07Um
8:09yeah I mean, this is the has been the
8:11bane of our existence is um,
8:13with the with the new models or with the
8:15frontier models, not even with the
8:16frontier, with any LLMs, you basically
8:19can pay 10 bucks and spray 50 findings.
8:23Uh, cuz the cost to write a report is
8:25basically near zero. I mean, I put
8:27asterisks on both of them. Cost to write
8:30a bad report or a not necessarily a
8:32well-thought report is near zero.
8:34They're still like I mean, most people
8:36in this room are very good researchers
8:38and people really put effort into it.
8:41Um, is this a pointer? Yeah.
8:44Which means that the triage cost doesn't
8:46really change unless you start doing
8:48agentic triage, but then still the cost
8:50is of triage is so so much higher than
8:53cost of submission. So you have to you
8:54basically have to come up with ways. How
8:56do you make sure the submissions you
8:57receive are actually valid and valid
9:01[clears throat]
9:02like
9:03duplicates are good, uh novel
9:05submissions are good. It's just like
9:06things are actually reachable and
9:07actually have impact.
9:09And uh you basically end up with this
9:12options where the most basic one being
9:15the the working POC, which is basically
9:18all the submissions had.
9:19The problem like it it stops speculative
9:22claims uh and it does take time to
9:25build, but with LLMs it's pretty
9:26straightforward. Being like
9:282FA on GitHub and minimum account age
9:31does help a bit. So we had this three we
9:34had. This one
9:36we we ended up I'll talk a bit more, but
9:39yeah, we ended up requiring a
9:41half-a-soul burn versus a refundable
9:43deposit.
9:45And it helps with some like low-effort
9:47filings.
9:48This is I think the most I'm pretty
9:51bullish on this where I would say it's
9:53like a step beyond proof of concept
9:55where it's a proof of impact where you
9:59have some kind of infrastructure where
10:01you could provably
10:03show that your POC actually shows
10:06impact.
10:07Because the problem with the POC is POC
10:09is only as good as what it shows. You
10:11ask like you ask an LLM, it can write to
10:14you any POC you want. It can prove
10:16whatever.
10:17It's just like you start
10:19mocking different data structures. You
10:21start mocking different areas of the
10:23validator.
10:25And like, oh yes, this is you have I
10:27don't know, network exhaustion. Oh yeah,
10:29but you actually don't have any of the
10:31data center filtering or you don't have
10:34any of the other protections that we
10:36have. You're like, okay, yeah, if you
10:38take that into account, it doesn't work.
10:39[gasps]
10:40I think this is possible to do. The
10:42problem is you kind of need specialized
10:44tooling of how do you define
10:46a correct exploit? How do you define
10:48this impact? I think you can do this for
10:50really big programs like a I think it is
10:53possible. It's just
10:55it becomes a bit
10:57it's tough. Like it's not an easy
10:58definition to do and also you have to
11:00cover so so many cases from like runtime
11:02issues to networking to like
11:05um
11:07yeah, to like consensus
11:08like mapping. It's it's a bit tough, but
11:10I think this is what we really need to
11:12get to.
11:13Um yeah, the next one is like doing KYC
11:16upfront. I don't think this is a good
11:17solution.
11:18It's kind of comes together with a
11:20refundable deposit. If you want to do
11:21the deposit,
11:23as far as I understand, you basically
11:24have to do KYC at most points.
11:27This really stops people submitting. I
11:28think we should only do KYC at payout,
11:31but that's like personal opinion. The
11:33next ones are like invite only. Yeah, I
11:36mean that basically becomes a pseudo
11:37audit, pseudo like you kind of want new
11:41people to look into this, get new pairs
11:42of eyes. And last one is doing like
11:44penalties and bans and stops repeat
11:47offenders, but like drawing the line of
11:49where the penalties and bans go and then
11:51and then you wake up to like a lot of
11:53emails people asking to get unbanned and
11:56like it's it becomes it's a lot of
11:58management. I 'm not sure it really
11:59leads to very good results.
12:02Like
12:03you probably should ban like really
12:04really misbehaving people, but like it's
12:06it's tough.
12:08Um
12:09yeah, but I mean fundamentally like
12:10drawing back to crypto, you either do
12:12proof of work or you do proof of stake.
12:15And there's many more, but like in our
12:16context where proof of work becomes have
12:18you actually done the work? Like can you
12:20show the vulnerability is real? And I
12:22think I mean that is the solution. And
12:25we want like doing some kind of this
12:27proof of impact showing that this stuff
12:29is real in a maybe it's not necessarily
12:32ZK proof, but like some kind of setup
12:35where it's hard to dispute versus
12:38requiring a proof of concept. Everyone
12:40writes a proof of concept in a very
12:42different way and it takes a while to
12:44triage.
12:45So maybe some like unified tooling for
12:47proof of work could be the future
12:48solution. Proof of stake is what we did.
12:51Um
12:53it's the best solution we have at the
12:55time.
12:57I'm not
12:59I think eventually
13:01the best way is you don't charge people,
13:03but that is just what like that's the
13:07best solution currently we have. And it
13:10only works if it hurts more than the
13:12spam pays. And it fundamentally the
13:14amount of payouts we do
13:16the
13:18half a soul is not a huge huge
13:20commitment.
13:21Uh but yeah the that's actually how it
13:23worked.
13:25This we had a I I coded like a
13:27submission portal
13:28when up and going went live uh when the
13:31before the competition.
13:32It required a burn of half a soul, so we
13:35didn't do custodial. So we didn't care
13:37about taking the actual payment for it.
13:39Like the the only purpose of this was to
13:42add a layer where someone thinks twice
13:45if this
13:46think is valid or they will write up a
13:48better report just because there is some
13:50stake and money at stake. That also
13:52means you don't have to deal with refund
13:53and disputes.
13:54And um
13:56yeah it's just like and it also
13:58technically deflationary, which is nice.
14:01[laughter]
14:02Uh and then you basically submit the
14:04transaction hash, we verify it and then
14:07it opens a GitHub GHSA for you. And this
14:10actually worked really well.
14:12Uh
14:13this is what we actually ended up doing
14:14for the triage. So the the green, the
14:16green ones are agentic, and then this
14:19one is human.
14:20It was
14:21This was very much a proof of concept
14:23trial approach where
14:26uh we used a tool called Tracecat.
14:28Tracecat is like a agentic SO like SOAR
14:33tool that allows you It's mainly made
14:35for like web two
14:38triaging of large-scale alerts, but
14:42it actually worked pretty well. It
14:43allowed us somewhere to host the agents,
14:45do this um in a in a nice sustainable
14:48way. We mainly used GPT 5.6 which which
14:52was pretty good. Yeah, it basically
14:53pulls in the pulls in the the claim,
14:56checks the severity, checks the format,
14:58checks uh some some basic checks that
15:00it's like
15:01sane as a submission. Then we do some
15:04evaluation. We run a um
15:07no internet sandbox checking the POC,
15:10matching the claim against uh Alpine
15:13Glow Agave Master at the time because we
15:15also wanted to check if these issues
15:17have already been fixed.
15:19Uh the next one was This is like where
15:22the crème de la crème of this triage
15:23worked was
15:25an agent that was like the whole job was
15:27to convince that this is invalid. Is it
15:30reachable? Is it breaching the 20 plus
15:3220 model? Is it intended? Is it like
15:35Is it just made up? And it did it did a
15:37very good job. And then you kind of
15:39draft it, you check duplicates, and you
15:42rank by similarity, and then you
15:43basically hand it off to a person to be
15:44like, does this make sense?
15:46Final severity payouts, rejection,
15:48whatever. We were very specific that we
15:51did not want
15:53to auto close issues or auto
15:57like decide things because this is still
15:59like a supervised approach
16:01um just because
16:03this was our first attempt, but it
16:05worked very well.
16:06Um
16:07yeah, this is a timeline like lifeline
16:10for of a of a of a report. It went from
16:14like the bug line landing, then gets
16:16pulled from GitHub, case opened, it like
16:19agentic starts, pulls in POC, pulls in
16:23um
16:24pulls in source code. So this one
16:26uh
16:27then
16:28does like more deeper analysis, does the
16:30disproving. In the case of this bug, it
16:33came in as a critical, it went down to
16:35as a low.
16:37And this was actually the first of nine
16:39duplicates, I think.
16:41And then the human kind of then the next
16:43ones got bundled up to this one and um
16:48in the end it was information.
16:50It went through nearly all the stages.
16:52Um yeah, this is the the some fun stats.
16:56It we had 338 submissions over two
16:59weeks. 67% the agent and human agreed,
17:03which is pretty good.
17:05It was nine minutes to an actual agentic
17:07triage. It took humans about 14 hours,
17:10which is still pretty good.
17:12Um
17:13yeah, it worked for about 90% of the
17:15real bugs.
17:16Uh it saved us a lot of time. It
17:19rejected uh all 21 reports that
17:22breached the security model. It was very
17:24good at working out which ones did not
17:26follow the 20 plus 20. That was very
17:28good.
17:29Uh it wasn't so good at
17:31it was like valid bug, but the impact
17:33calculations it was not get good at, but
17:36this is very easy to like this is just
17:38future solving we can do. This is the
17:40competition itself. So we had 115
17:42researchers, 27 findings, valid
17:45findings, and we paid out 27795
17:48SOL.
17:49Uh in terms of like timeline, I'll move
17:51to the to the other side. Um like nearly
17:55within 10 minutes we got six valid
17:56submissions. Within the first hour we
17:58got 12 out of the 27. So nearly like I
18:01think we kind of exhausted the search
18:03space within like a few days. So these
18:05little
18:06like dashes are the winning reports.
18:09And then in the in the whole second week
18:11we only got 15 submissions,
18:13uh which was pretty interesting. Um
18:21and there we go. Uh these uh timings
18:23between different reports. So
18:26we kind of
18:27we didn't
18:28we didn't expect that we will get so so
18:30many duplicates within such a close time
18:32frame where we had a few reports within
18:35seconds of each other
18:37at the time of the opening. And about 58
18:40of the 85 submissions we had were
18:43duplicates.
18:45Um
18:46and nearly all of the duplicates were
18:48within 10 minutes of the first one. So
18:50it was very much like race to the
18:51bottom.
18:52And this is something we're going to
18:53reconsider when we do it next time
18:55because it becomes a bit unfair where
18:57like if you are the submission is within
18:595 seconds
19:01it's like it's this is not a skill-based
19:04competition. It's just like how quickly
19:06can you vibe code a submission thing
19:08which is not ideal.
19:10Um
19:12the other question is would we do it
19:13again? Yes. We would keep the portal and
19:17we actually ended up adopting the burn
19:18for our standing bounty.
19:21Oh
19:22one sec.
19:23Um the next time I think spending more
19:26time on this like
19:28basically feeding back the triage
19:29findings and the human verdicts back
19:31into our like agentic triage I think is
19:33the solution. It worked extremely well
19:36and that is
19:37that is the way we will have to deal
19:39with scale.
19:40I think this advanced POC creation
19:43I'm pretty bullish on it. It requires
19:45defining it very well, but I think
19:47especially for consensus bugs
19:49I think there's a way we can define a
19:52nice system which
19:54not necessarily forces the researcher.
19:56It gives the researcher tools to
19:58actually validate their findings in a
19:59very nice way versus them trying to like
20:02reverse engineer how to do POCs and
20:05which things that actually supposed to
20:06plug in.
20:08And
20:09that would also allow us to
20:11quickly validate these things and
20:13potentially
20:15do this whole process quicker and save
20:17people burns for issues that are not
20:19valid.
20:20And the duplicate policy
20:23is very much similar one. I think we
20:25probably would allow duplicates next
20:27round just because it becomes a bit
20:29I didn't realize how much of a race to
20:31the bottom it will be. So that's just a
20:33learning and I think better
20:34communication with researchers.
20:37We did not do the best of the job
20:40communicating to people partially
20:41because GitHub does not have an API to
20:44write comments and it's just a nightmare
20:46to manage in notifications.
20:48And but there's a lot of easy tooling
20:51you can do and these are pretty easy
20:54things we can improve on.
20:57Yeah, so we ended up migrating from
21:01to our main bug bounty program that is
21:03now also requiring half a soul burn
21:06but we also have a white list for
21:08because our
21:10researchers like at Asymmetric at
21:12Neodyme
21:14they submit their findings also through
21:16the bounty program.
21:18They ended up you end up like not
21:20requiring it and I think there is a
21:22potential where you expand it
21:25to potentially new participants you
21:26require burn for a few for not new you
21:29might not require burn but it's it's
21:31it's kind of experimentation stage.
21:35Yeah, this is actually some stats from
21:38we for about a month that's this thing
21:39has been running.
21:43This so the first two are
21:47the breakdown of the types of
21:49submissions. So the first one is the the
21:51January to
21:541st of January basically the year coming
21:57to 2nd October when we turned it off. We
21:59had about 50% were invalid findings, 31
22:02were duplicates and 19% were novel.
22:04With the burn wall
22:0720 we went from 50% invalid to 22%
22:09invalid which is an excellent reduction
22:13and then we went from 31 invalid to 52%
22:16duplicates. So the actual like amount of
22:18reports didn't go down as much as I
22:20thought.
22:21We went down only like 37% submissions
22:24but I think it'll actually go up because
22:26this is only a month of data.
22:28The the most biting one, I think, the
22:3052% unique researchers reduction.
22:34It does limit it to people who can
22:36afford this burn, and most commonly it
22:38also limits to um
22:41researchers who have been paid out
22:43before in this program.
22:44I think that's the thing. But the
22:46interesting thing that the actual novel
22:48findings, there was basically no change.
22:50I think it was something like 10%.
22:52So that means even though we got so 37%
22:55less findings and 50% less
22:57participating, the actual amount of new
22:59issues found hasn't really changed.
23:02So it kind of implies that this 50% were
23:06basically finding duplicates or invalid
23:08findings.
23:09Uh and we now get about 40% reports per
23:13researcher, which kind of
23:15you start drawing parallels that they're
23:16spending more time looking at the
23:18codebase and validating these findings
23:19and finding new things, which is
23:21kind of what you want. And really the if
23:25you look at if you put it in the context
23:26of the the payouts, our largest like our
23:30if you find a loss of funds in Solana,
23:33uh it's about starts at about 6.5000 SOL
23:36and goes up to
23:3825,000 SOL, I think. So these are quite
23:40large payouts. So even without average
23:43payout, half a SOL is like 0.2% of what
23:47you receive.
23:48Um these are some interesting These are
23:51like our weekly submission rates
23:53for the last year, and I tried to map
23:55them to like different models coming
23:57out.
23:59Um there is really no major parallel, I
24:02think.
24:02[laughter]
24:03All the models come out so quickly. I
24:05mean the summer has been very, very
24:07busy, and I think that's where the
24:09really interesting models came out.
24:11And then October at the end there is
24:13when
24:13we turned on the paywall.
24:15And it's definitely been going steadily
24:18up. And that's just kind of how working
24:23out how do you deal with this and is a I
24:25think a problem that everyone has. And
24:27then if we extend it to like 2025 when
24:32we didn't have these
24:34models,
24:35well, they weren't as good.
24:37So we got six times more submissions
24:41than in 2025 than we did in 2026 until
24:44the paywall.
24:45Um we got we now have three times more
24:48researchers, nearly all new.
24:50And this [clears throat] one is fun. We
24:51have 10 times more EM dashes per
24:54thousand words.
24:55Like this line is EM dashes per thousand
24:58words. And then very interestingly in
25:01January it skyrockets.
25:04[laughter] And then also six times more
25:06markdown tables.
25:08Um
25:10the the thing the line going down I
25:12assume is more the models are using less
25:14EM dashes than
25:17not model usage, but this was an
25:19interesting But yeah, it basically
25:21exploded.
25:22Um but yeah, that's that's basically it.
25:25Thank you very much.
25:26Yeah, basically huge shout out to also
25:28to the Consensus Team Foundation and
25:30everyone who did the triage for
25:32competition. Without them this would not
25:34be possible. And yeah, I'm I'm Belenkov
25:36on Twitter and Telegram and our
25:39submission. Yeah, thank you very much
25:40guys for listening.
25:41>> [applause]
Source and accuracy details
- Names, organizations, technical terms, code and numbers may be misrecognized.
- Source wording and disfluencies are preserved; no editorial corrections were applied.
- Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.
- A full caption-track export does not establish that every spoken word is represented.
- Automatic YouTube captions; not human-verified against the audio.
Source export SHA256: f1e445104ea22b3425a3abc52c55c16eb0a8828d0302bf78dba0ed8ed348a826