{
  "schemaVersion": 1,
  "videoId": "HvPK8VfCIG0",
  "videoUrl": "https://www.youtube.com/watch?v=HvPK8VfCIG0",
  "title": "Can AI Audit Your Smart Contract? We Tested It on Real Audits.",
  "speakerCredit": "Mohamad Mansouri",
  "programmeTalkId": "mohamad-mansouri",
  "language": "en",
  "captionType": "auto-generated",
  "humanAccuracyReviewed": false,
  "spokenAudioCompletenessVerified": false,
  "cueEndTimesAvailable": false,
  "source": {
    "provider": "YouTube",
    "method": "native browser transcript export",
    "sha256": "04bca3d592131b1802b7461465c6e0ea2c63d51d6230df9dc858e5984281a8d9",
    "exportedAt": "2026-11-16T07:37:28.738563+00:00"
  },
  "qualityCaveats": [
    "Names, organizations, technical terms, code and numbers may be misrecognized.",
    "Source wording and disfluencies are preserved; no editorial corrections were applied.",
    "Export contains whole-second cue starts only; cue ends and subsecond timing are unavailable.",
    "A full caption-track export does not establish that every spoken word is represented.",
    "Automatic YouTube captions; not human-verified against the audio."
  ],
  "timingValidation": {
    "first_cue_start_seconds": 1,
    "last_cue_start_seconds": 977,
    "video_duration_seconds": 980,
    "seconds_after_last_cue_start": 3,
    "cue_start_gaps_over_30_seconds": [],
    "boundary_check": "pass",
    "criterion": "first cue <=15 seconds; last cue start within 30 seconds of player duration, allowing 1-second precision",
    "meaning": "Timing plausibility only; silence/applause or source caption omissions require audio review."
  },
  "segments": [
    {
      "index": 0,
      "start_seconds": 1,
      "source_timestamp": "0:01",
      "end_seconds": null,
      "text": "So I'm Mohamed Mansour from D-Dope."
    },
    {
      "index": 1,
      "start_seconds": 3,
      "source_timestamp": "0:03",
      "end_seconds": null,
      "text": "So I'm a security researcher researcher"
    },
    {
      "index": 2,
      "start_seconds": 5,
      "source_timestamp": "0:05",
      "end_seconds": null,
      "text": "at D-Dope. At D-Dope we are a security"
    },
    {
      "index": 3,
      "start_seconds": 7,
      "source_timestamp": "0:07",
      "end_seconds": null,
      "text": "company in Web 3 field and we do"
    },
    {
      "index": 4,
      "start_seconds": 10,
      "source_timestamp": "0:10",
      "end_seconds": null,
      "text": "security audits. So that's why we are"
    },
    {
      "index": 5,
      "start_seconds": 12,
      "source_timestamp": "0:12",
      "end_seconds": null,
      "text": "doing a research"
    },
    {
      "index": 6,
      "start_seconds": 13,
      "source_timestamp": "0:13",
      "end_seconds": null,
      "text": "on"
    },
    {
      "index": 7,
      "start_seconds": 15,
      "source_timestamp": "0:15",
      "end_seconds": null,
      "text": "how"
    },
    {
      "index": 8,
      "start_seconds": 16,
      "source_timestamp": "0:16",
      "end_seconds": null,
      "text": "security how AI can do security audits."
    },
    {
      "index": 9,
      "start_seconds": 19,
      "source_timestamp": "0:19",
      "end_seconds": null,
      "text": "So basically the main question is that"
    },
    {
      "index": 10,
      "start_seconds": 23,
      "source_timestamp": "0:23",
      "end_seconds": null,
      "text": "would you ship your code"
    },
    {
      "index": 11,
      "start_seconds": 25,
      "source_timestamp": "0:25",
      "end_seconds": null,
      "text": "and deploy it on chain if it was only"
    },
    {
      "index": 12,
      "start_seconds": 27,
      "source_timestamp": "0:27",
      "end_seconds": null,
      "text": "audited by AI?"
    },
    {
      "index": 13,
      "start_seconds": 30,
      "source_timestamp": "0:30",
      "end_seconds": null,
      "text": "This is a main question. So how how many"
    },
    {
      "index": 14,
      "start_seconds": 31,
      "source_timestamp": "0:31",
      "end_seconds": null,
      "text": "people are working on security in the"
    },
    {
      "index": 15,
      "start_seconds": 33,
      "source_timestamp": "0:33",
      "end_seconds": null,
      "text": "room?"
    },
    {
      "index": 16,
      "start_seconds": 35,
      "source_timestamp": "0:35",
      "end_seconds": null,
      "text": "So there's there's quite some people"
    },
    {
      "index": 17,
      "start_seconds": 36,
      "source_timestamp": "0:36",
      "end_seconds": null,
      "text": "right working. So in order to answer"
    },
    {
      "index": 18,
      "start_seconds": 39,
      "source_timestamp": "0:39",
      "end_seconds": null,
      "text": "this question so what we did actually is"
    },
    {
      "index": 19,
      "start_seconds": 42,
      "source_timestamp": "0:42",
      "end_seconds": null,
      "text": "that in one line what we did is that we"
    },
    {
      "index": 20,
      "start_seconds": 44,
      "source_timestamp": "0:44",
      "end_seconds": null,
      "text": "hid our audits"
    },
    {
      "index": 21,
      "start_seconds": 46,
      "source_timestamp": "0:46",
      "end_seconds": null,
      "text": "and then we asked the AI to audit our"
    },
    {
      "index": 22,
      "start_seconds": 49,
      "source_timestamp": "0:49",
      "end_seconds": null,
      "text": "the codes that we have audited before"
    },
    {
      "index": 23,
      "start_seconds": 51,
      "source_timestamp": "0:51",
      "end_seconds": null,
      "text": "and generate reports and see how well"
    },
    {
      "index": 24,
      "start_seconds": 53,
      "source_timestamp": "0:53",
      "end_seconds": null,
      "text": "they do."
    },
    {
      "index": 25,
      "start_seconds": 54,
      "source_timestamp": "0:54",
      "end_seconds": null,
      "text": "Right? So this is in one line what we"
    },
    {
      "index": 26,
      "start_seconds": 56,
      "source_timestamp": "0:56",
      "end_seconds": null,
      "text": "are going to to show you the results of."
    },
    {
      "index": 27,
      "start_seconds": 64,
      "source_timestamp": "1:04",
      "end_seconds": null,
      "text": "So again we what we are doing here is to"
    },
    {
      "index": 28,
      "start_seconds": 66,
      "source_timestamp": "1:06",
      "end_seconds": null,
      "text": "answer the question whether we can rely"
    },
    {
      "index": 29,
      "start_seconds": 68,
      "source_timestamp": "1:08",
      "end_seconds": null,
      "text": "on AI to audit code"
    },
    {
      "index": 30,
      "start_seconds": 71,
      "source_timestamp": "1:11",
      "end_seconds": null,
      "text": "and by this means that audit code to"
    },
    {
      "index": 31,
      "start_seconds": 73,
      "source_timestamp": "1:13",
      "end_seconds": null,
      "text": "deploy without having human people"
    },
    {
      "index": 32,
      "start_seconds": 76,
      "source_timestamp": "1:16",
      "end_seconds": null,
      "text": "auditing this code."
    },
    {
      "index": 33,
      "start_seconds": 77,
      "source_timestamp": "1:17",
      "end_seconds": null,
      "text": "And as a security company who have done"
    },
    {
      "index": 34,
      "start_seconds": 80,
      "source_timestamp": "1:20",
      "end_seconds": null,
      "text": "a lot of audits and we have a lot of"
    },
    {
      "index": 35,
      "start_seconds": 82,
      "source_timestamp": "1:22",
      "end_seconds": null,
      "text": "clients that we have engaged in security"
    },
    {
      "index": 36,
      "start_seconds": 85,
      "source_timestamp": "1:25",
      "end_seconds": null,
      "text": "assessments of their protocols, we have"
    },
    {
      "index": 37,
      "start_seconds": 87,
      "source_timestamp": "1:27",
      "end_seconds": null,
      "text": "a lot of data sets and what we wanted to"
    },
    {
      "index": 38,
      "start_seconds": 90,
      "source_timestamp": "1:30",
      "end_seconds": null,
      "text": "do is to check this by"
    },
    {
      "index": 39,
      "start_seconds": 93,
      "source_timestamp": "1:33",
      "end_seconds": null,
      "text": "by hiding our reports from the AI and"
    },
    {
      "index": 40,
      "start_seconds": 95,
      "source_timestamp": "1:35",
      "end_seconds": null,
      "text": "asked it to audit itself this code at"
    },
    {
      "index": 41,
      "start_seconds": 97,
      "source_timestamp": "1:37",
      "end_seconds": null,
      "text": "that version we audited and see how well"
    },
    {
      "index": 42,
      "start_seconds": 99,
      "source_timestamp": "1:39",
      "end_seconds": null,
      "text": "it does."
    },
    {
      "index": 43,
      "start_seconds": 102,
      "source_timestamp": "1:42",
      "end_seconds": null,
      "text": "So here's how the experiment works in in"
    },
    {
      "index": 44,
      "start_seconds": 106,
      "source_timestamp": "1:46",
      "end_seconds": null,
      "text": "specifically. So"
    },
    {
      "index": 45,
      "start_seconds": 107,
      "source_timestamp": "1:47",
      "end_seconds": null,
      "text": "we chose eight of our past client audits"
    },
    {
      "index": 46,
      "start_seconds": 112,
      "source_timestamp": "1:52",
      "end_seconds": null,
      "text": "and what we did is basically asked the"
    },
    {
      "index": 47,
      "start_seconds": 115,
      "source_timestamp": "1:55",
      "end_seconds": null,
      "text": "AI to audit these different agents,"
    },
    {
      "index": 48,
      "start_seconds": 117,
      "source_timestamp": "1:57",
      "end_seconds": null,
      "text": "different models. We'll talk about this"
    },
    {
      "index": 49,
      "start_seconds": 119,
      "source_timestamp": "1:59",
      "end_seconds": null,
      "text": "and from the reports they generate, we"
    },
    {
      "index": 50,
      "start_seconds": 121,
      "source_timestamp": "2:01",
      "end_seconds": null,
      "text": "will try to first match"
    },
    {
      "index": 51,
      "start_seconds": 123,
      "source_timestamp": "2:03",
      "end_seconds": null,
      "text": "them with the reports to see how many of"
    },
    {
      "index": 52,
      "start_seconds": 124,
      "source_timestamp": "2:04",
      "end_seconds": null,
      "text": "these are existing in our reports. And"
    },
    {
      "index": 53,
      "start_seconds": 126,
      "source_timestamp": "2:06",
      "end_seconds": null,
      "text": "for the one who doesn't match, we will"
    },
    {
      "index": 54,
      "start_seconds": 129,
      "source_timestamp": "2:09",
      "end_seconds": null,
      "text": "try to triage that by first doing some"
    },
    {
      "index": 55,
      "start_seconds": 132,
      "source_timestamp": "2:12",
      "end_seconds": null,
      "text": "AI filtering. And for anything that"
    },
    {
      "index": 56,
      "start_seconds": 134,
      "source_timestamp": "2:14",
      "end_seconds": null,
      "text": "looks serious, we're going to do manual"
    },
    {
      "index": 57,
      "start_seconds": 136,
      "source_timestamp": "2:16",
      "end_seconds": null,
      "text": "revision and take a decision on it."
    },
    {
      "index": 58,
      "start_seconds": 139,
      "source_timestamp": "2:19",
      "end_seconds": null,
      "text": "Um, please."
    },
    {
      "index": 59,
      "start_seconds": 142,
      "source_timestamp": "2:22",
      "end_seconds": null,
      "text": "So from these eight"
    },
    {
      "index": 60,
      "start_seconds": 143,
      "source_timestamp": "2:23",
      "end_seconds": null,
      "text": "eight audits,"
    },
    {
      "index": 61,
      "start_seconds": 145,
      "source_timestamp": "2:25",
      "end_seconds": null,
      "text": "there are six audits are in Solidity"
    },
    {
      "index": 62,
      "start_seconds": 147,
      "source_timestamp": "2:27",
      "end_seconds": null,
      "text": "with different code sizes."
    },
    {
      "index": 63,
      "start_seconds": 149,
      "source_timestamp": "2:29",
      "end_seconds": null,
      "text": "One in Move C chain and one in Talk for"
    },
    {
      "index": 64,
      "start_seconds": 153,
      "source_timestamp": "2:33",
      "end_seconds": null,
      "text": "Ton chain."
    },
    {
      "index": 65,
      "start_seconds": 154,
      "source_timestamp": "2:34",
      "end_seconds": null,
      "text": "With different sizes, different types of"
    },
    {
      "index": 66,
      "start_seconds": 157,
      "source_timestamp": "2:37",
      "end_seconds": null,
      "text": "different periods of engagement. In"
    },
    {
      "index": 67,
      "start_seconds": 159,
      "source_timestamp": "2:39",
      "end_seconds": null,
      "text": "total, there's 61 bugs in these reports."
    },
    {
      "index": 68,
      "start_seconds": 163,
      "source_timestamp": "2:43",
      "end_seconds": null,
      "text": "And there's different amount of time"
    },
    {
      "index": 69,
      "start_seconds": 165,
      "source_timestamp": "2:45",
      "end_seconds": null,
      "text": "spent on these to find them."
    },
    {
      "index": 70,
      "start_seconds": 167,
      "source_timestamp": "2:47",
      "end_seconds": null,
      "text": "Now let let me tell you that we have"
    },
    {
      "index": 71,
      "start_seconds": 170,
      "source_timestamp": "2:50",
      "end_seconds": null,
      "text": "done this several times. So we've done"
    },
    {
      "index": 72,
      "start_seconds": 172,
      "source_timestamp": "2:52",
      "end_seconds": null,
      "text": "this entire experiments on much more"
    },
    {
      "index": 73,
      "start_seconds": 174,
      "source_timestamp": "2:54",
      "end_seconds": null,
      "text": "than eight audits in June. And this is"
    },
    {
      "index": 74,
      "start_seconds": 176,
      "source_timestamp": "2:56",
      "end_seconds": null,
      "text": "the recent one we did lately in the last"
    },
    {
      "index": 75,
      "start_seconds": 178,
      "source_timestamp": "2:58",
      "end_seconds": null,
      "text": "month. So we're going to present the"
    },
    {
      "index": 76,
      "start_seconds": 180,
      "source_timestamp": "3:00",
      "end_seconds": null,
      "text": "recent one and also compare it to"
    },
    {
      "index": 77,
      "start_seconds": 182,
      "source_timestamp": "3:02",
      "end_seconds": null,
      "text": "previously how was"
    },
    {
      "index": 78,
      "start_seconds": 183,
      "source_timestamp": "3:03",
      "end_seconds": null,
      "text": "AI in"
    },
    {
      "index": 79,
      "start_seconds": 184,
      "source_timestamp": "3:04",
      "end_seconds": null,
      "text": "four months, five months from now was"
    },
    {
      "index": 80,
      "start_seconds": 186,
      "source_timestamp": "3:06",
      "end_seconds": null,
      "text": "doing."
    },
    {
      "index": 81,
      "start_seconds": 187,
      "source_timestamp": "3:07",
      "end_seconds": null,
      "text": "Please."
    },
    {
      "index": 82,
      "start_seconds": 190,
      "source_timestamp": "3:10",
      "end_seconds": null,
      "text": "These are the agents we are trying to"
    },
    {
      "index": 83,
      "start_seconds": 191,
      "source_timestamp": "3:11",
      "end_seconds": null,
      "text": "test. So the ones in blue here are ones"
    },
    {
      "index": 84,
      "start_seconds": 193,
      "source_timestamp": "3:13",
      "end_seconds": null,
      "text": "that we are internally building and"
    },
    {
      "index": 85,
      "start_seconds": 195,
      "source_timestamp": "3:15",
      "end_seconds": null,
      "text": "developing and using in our internal"
    },
    {
      "index": 86,
      "start_seconds": 197,
      "source_timestamp": "3:17",
      "end_seconds": null,
      "text": "procedures."
    },
    {
      "index": 87,
      "start_seconds": 199,
      "source_timestamp": "3:19",
      "end_seconds": null,
      "text": "We start from different We do different"
    },
    {
      "index": 88,
      "start_seconds": 201,
      "source_timestamp": "3:21",
      "end_seconds": null,
      "text": "tiers. So we start from let's say an AI"
    },
    {
      "index": 89,
      "start_seconds": 204,
      "source_timestamp": "3:24",
      "end_seconds": null,
      "text": "that is basically just a plain model,"
    },
    {
      "index": 90,
      "start_seconds": 207,
      "source_timestamp": "3:27",
      "end_seconds": null,
      "text": "basically just a simple prompt"
    },
    {
      "index": 91,
      "start_seconds": 209,
      "source_timestamp": "3:29",
      "end_seconds": null,
      "text": "that asks the model to to audit this"
    },
    {
      "index": 92,
      "start_seconds": 213,
      "source_timestamp": "3:33",
      "end_seconds": null,
      "text": "code."
    },
    {
      "index": 93,
      "start_seconds": 213,
      "source_timestamp": "3:33",
      "end_seconds": null,
      "text": "And with the flagship"
    },
    {
      "index": 94,
      "start_seconds": 215,
      "source_timestamp": "3:35",
      "end_seconds": null,
      "text": "version."
    },
    {
      "index": 95,
      "start_seconds": 216,
      "source_timestamp": "3:36",
      "end_seconds": null,
      "text": "And all the way to audit factories,"
    },
    {
      "index": 96,
      "start_seconds": 218,
      "source_timestamp": "3:38",
      "end_seconds": null,
      "text": "which is like a very well, you know,"
    },
    {
      "index": 97,
      "start_seconds": 221,
      "source_timestamp": "3:41",
      "end_seconds": null,
      "text": "pipelines of of using AI to to to find"
    },
    {
      "index": 98,
      "start_seconds": 225,
      "source_timestamp": "3:45",
      "end_seconds": null,
      "text": "bugs. And there's like things in between"
    },
    {
      "index": 99,
      "start_seconds": 228,
      "source_timestamp": "3:48",
      "end_seconds": null,
      "text": "like light skills, only using skills"
    },
    {
      "index": 100,
      "start_seconds": 229,
      "source_timestamp": "3:49",
      "end_seconds": null,
      "text": "with lightweight with light workload so"
    },
    {
      "index": 101,
      "start_seconds": 232,
      "source_timestamp": "3:52",
      "end_seconds": null,
      "text": "to make this fast. Also, there's like"
    },
    {
      "index": 102,
      "start_seconds": 234,
      "source_timestamp": "3:54",
      "end_seconds": null,
      "text": "the public tools that are existing now,"
    },
    {
      "index": 103,
      "start_seconds": 237,
      "source_timestamp": "3:57",
      "end_seconds": null,
      "text": "which basically relies on"
    },
    {
      "index": 104,
      "start_seconds": 239,
      "source_timestamp": "3:59",
      "end_seconds": null,
      "text": "running security tools and then triaging"
    },
    {
      "index": 105,
      "start_seconds": 241,
      "source_timestamp": "4:01",
      "end_seconds": null,
      "text": "the results with LLM. And there's the"
    },
    {
      "index": 106,
      "start_seconds": 243,
      "source_timestamp": "4:03",
      "end_seconds": null,
      "text": "audit factories. We have two agents"
    },
    {
      "index": 107,
      "start_seconds": 245,
      "source_timestamp": "4:05",
      "end_seconds": null,
      "text": "there."
    },
    {
      "index": 108,
      "start_seconds": 246,
      "source_timestamp": "4:06",
      "end_seconds": null,
      "text": "One is more lightweight and the one Zeus"
    },
    {
      "index": 109,
      "start_seconds": 248,
      "source_timestamp": "4:08",
      "end_seconds": null,
      "text": "is the more, you know, heavy metal stuff"
    },
    {
      "index": 110,
      "start_seconds": 251,
      "source_timestamp": "4:11",
      "end_seconds": null,
      "text": "that we really takes a lot of talking"
    },
    {
      "index": 111,
      "start_seconds": 253,
      "source_timestamp": "4:13",
      "end_seconds": null,
      "text": "consumption but does everything from"
    },
    {
      "index": 112,
      "start_seconds": 256,
      "source_timestamp": "4:16",
      "end_seconds": null,
      "text": "fuzzing to to understanding the protocol"
    },
    {
      "index": 113,
      "start_seconds": 258,
      "source_timestamp": "4:18",
      "end_seconds": null,
      "text": "threat modeling and everything to find"
    },
    {
      "index": 114,
      "start_seconds": 260,
      "source_timestamp": "4:20",
      "end_seconds": null,
      "text": "to find the bugs."
    },
    {
      "index": 115,
      "start_seconds": 263,
      "source_timestamp": "4:23",
      "end_seconds": null,
      "text": "Uh let's jump directly to the results."
    },
    {
      "index": 116,
      "start_seconds": 266,
      "source_timestamp": "4:26",
      "end_seconds": null,
      "text": "Do you have any can anyone give some"
    },
    {
      "index": 117,
      "start_seconds": 268,
      "source_timestamp": "4:28",
      "end_seconds": null,
      "text": "expectation? What do they think?"
    },
    {
      "index": 118,
      "start_seconds": 271,
      "source_timestamp": "4:31",
      "end_seconds": null,
      "text": "So we're going to measure the recall,"
    },
    {
      "index": 119,
      "start_seconds": 273,
      "source_timestamp": "4:33",
      "end_seconds": null,
      "text": "how much the percentage of"
    },
    {
      "index": 120,
      "start_seconds": 277,
      "source_timestamp": "4:37",
      "end_seconds": null,
      "text": "bugs"
    },
    {
      "index": 121,
      "start_seconds": 279,
      "source_timestamp": "4:39",
      "end_seconds": null,
      "text": "that are in our reports that AI will"
    },
    {
      "index": 122,
      "start_seconds": 280,
      "source_timestamp": "4:40",
      "end_seconds": null,
      "text": "found."
    },
    {
      "index": 123,
      "start_seconds": 282,
      "source_timestamp": "4:42",
      "end_seconds": null,
      "text": ">> Half."
    },
    {
      "index": 124,
      "start_seconds": 283,
      "source_timestamp": "4:43",
      "end_seconds": null,
      "text": ">> Half?"
    },
    {
      "index": 125,
      "start_seconds": 287,
      "source_timestamp": "4:47",
      "end_seconds": null,
      "text": "Okay, let's see."
    },
    {
      "index": 126,
      "start_seconds": 289,
      "source_timestamp": "4:49",
      "end_seconds": null,
      "text": "So basically it's 45% if we took the"
    },
    {
      "index": 127,
      "start_seconds": 292,
      "source_timestamp": "4:52",
      "end_seconds": null,
      "text": "average."
    },
    {
      "index": 128,
      "start_seconds": 293,
      "source_timestamp": "4:53",
      "end_seconds": null,
      "text": "So for every two bugs in our report,"
    },
    {
      "index": 129,
      "start_seconds": 296,
      "source_timestamp": "4:56",
      "end_seconds": null,
      "text": "serious bugs, medium and above,"
    },
    {
      "index": 130,
      "start_seconds": 298,
      "source_timestamp": "4:58",
      "end_seconds": null,
      "text": "there's a 50% chance that or almost 50%"
    },
    {
      "index": 131,
      "start_seconds": 302,
      "source_timestamp": "5:02",
      "end_seconds": null,
      "text": "chance that on average an agent would"
    },
    {
      "index": 132,
      "start_seconds": 304,
      "source_timestamp": "5:04",
      "end_seconds": null,
      "text": "find it. If we combine the results from"
    },
    {
      "index": 133,
      "start_seconds": 306,
      "source_timestamp": "5:06",
      "end_seconds": null,
      "text": "all agents together,"
    },
    {
      "index": 134,
      "start_seconds": 308,
      "source_timestamp": "5:08",
      "end_seconds": null,
      "text": "there's going to go up to 73. Now what's"
    },
    {
      "index": 135,
      "start_seconds": 310,
      "source_timestamp": "5:10",
      "end_seconds": null,
      "text": "interesting is you can see from June"
    },
    {
      "index": 136,
      "start_seconds": 313,
      "source_timestamp": "5:13",
      "end_seconds": null,
      "text": "it's was much much less, almost half of"
    },
    {
      "index": 137,
      "start_seconds": 315,
      "source_timestamp": "5:15",
      "end_seconds": null,
      "text": "this result from June."
    },
    {
      "index": 138,
      "start_seconds": 316,
      "source_timestamp": "5:16",
      "end_seconds": null,
      "text": "So basically in four months the alarms"
    },
    {
      "index": 139,
      "start_seconds": 319,
      "source_timestamp": "5:19",
      "end_seconds": null,
      "text": "went to a better version where increased"
    },
    {
      "index": 140,
      "start_seconds": 321,
      "source_timestamp": "5:21",
      "end_seconds": null,
      "text": "this results."
    },
    {
      "index": 141,
      "start_seconds": 322,
      "source_timestamp": "5:22",
      "end_seconds": null,
      "text": "Uh but interestingly the"
    },
    {
      "index": 142,
      "start_seconds": 326,
      "source_timestamp": "5:26",
      "end_seconds": null,
      "text": "the agents found all the critical bugs."
    },
    {
      "index": 143,
      "start_seconds": 330,
      "source_timestamp": "5:30",
      "end_seconds": null,
      "text": "So critical sorry high is wrong here. So"
    },
    {
      "index": 144,
      "start_seconds": 332,
      "source_timestamp": "5:32",
      "end_seconds": null,
      "text": "only critical bugs."
    },
    {
      "index": 145,
      "start_seconds": 334,
      "source_timestamp": "5:34",
      "end_seconds": null,
      "text": "Uh this is only the result on the"
    },
    {
      "index": 146,
      "start_seconds": 336,
      "source_timestamp": "5:36",
      "end_seconds": null,
      "text": "solidity codes. Now if we go over the"
    },
    {
      "index": 147,
      "start_seconds": 338,
      "source_timestamp": "5:38",
      "end_seconds": null,
      "text": "entire eight"
    },
    {
      "index": 148,
      "start_seconds": 340,
      "source_timestamp": "5:40",
      "end_seconds": null,
      "text": "eight eight projects with Sui and Ton in"
    },
    {
      "index": 149,
      "start_seconds": 343,
      "source_timestamp": "5:43",
      "end_seconds": null,
      "text": "them, then it's going to be a bit bit"
    },
    {
      "index": 150,
      "start_seconds": 345,
      "source_timestamp": "5:45",
      "end_seconds": null,
      "text": "less."
    },
    {
      "index": 151,
      "start_seconds": 346,
      "source_timestamp": "5:46",
      "end_seconds": null,
      "text": "But interesting from this slide we can"
    },
    {
      "index": 152,
      "start_seconds": 348,
      "source_timestamp": "5:48",
      "end_seconds": null,
      "text": "see that"
    },
    {
      "index": 153,
      "start_seconds": 350,
      "source_timestamp": "5:50",
      "end_seconds": null,
      "text": "the higher critical bugs, more severe"
    },
    {
      "index": 154,
      "start_seconds": 352,
      "source_timestamp": "5:52",
      "end_seconds": null,
      "text": "bugs are most more likely to be found"
    },
    {
      "index": 155,
      "start_seconds": 355,
      "source_timestamp": "5:55",
      "end_seconds": null,
      "text": "than medium bugs."
    },
    {
      "index": 156,
      "start_seconds": 356,
      "source_timestamp": "5:56",
      "end_seconds": null,
      "text": "In total for the eight eight audits,"
    },
    {
      "index": 157,
      "start_seconds": 359,
      "source_timestamp": "5:59",
      "end_seconds": null,
      "text": "it's still 50% so like 29 bugs were"
    },
    {
      "index": 158,
      "start_seconds": 362,
      "source_timestamp": "6:02",
      "end_seconds": null,
      "text": "found out of the 61 that we have in our"
    },
    {
      "index": 159,
      "start_seconds": 364,
      "source_timestamp": "6:04",
      "end_seconds": null,
      "text": "reports."
    },
    {
      "index": 160,
      "start_seconds": 367,
      "source_timestamp": "6:07",
      "end_seconds": null,
      "text": "Now, if we look at the results per per"
    },
    {
      "index": 161,
      "start_seconds": 369,
      "source_timestamp": "6:09",
      "end_seconds": null,
      "text": "per per per agent,"
    },
    {
      "index": 162,
      "start_seconds": 372,
      "source_timestamp": "6:12",
      "end_seconds": null,
      "text": "so Zeus, which is like the heavy metal"
    },
    {
      "index": 163,
      "start_seconds": 376,
      "source_timestamp": "6:16",
      "end_seconds": null,
      "text": "agent we have, done the best, but it"
    },
    {
      "index": 164,
      "start_seconds": 379,
      "source_timestamp": "6:19",
      "end_seconds": null,
      "text": "also has like a highest token"
    },
    {
      "index": 165,
      "start_seconds": 380,
      "source_timestamp": "6:20",
      "end_seconds": null,
      "text": "consumption around 7 750 like seven"
    },
    {
      "index": 166,
      "start_seconds": 384,
      "source_timestamp": "6:24",
      "end_seconds": null,
      "text": "times or even more like 10 times more."
    },
    {
      "index": 167,
      "start_seconds": 386,
      "source_timestamp": "6:26",
      "end_seconds": null,
      "text": "And here I'm measuring the token"
    },
    {
      "index": 168,
      "start_seconds": 388,
      "source_timestamp": "6:28",
      "end_seconds": null,
      "text": "consumption in dollar cost so that you"
    },
    {
      "index": 169,
      "start_seconds": 390,
      "source_timestamp": "6:30",
      "end_seconds": null,
      "text": "can get a relative value of this."
    },
    {
      "index": 170,
      "start_seconds": 393,
      "source_timestamp": "6:33",
      "end_seconds": null,
      "text": "We have Argos at 55 %, our Hermes is at"
    },
    {
      "index": 171,
      "start_seconds": 396,
      "source_timestamp": "6:36",
      "end_seconds": null,
      "text": "38 plain models as like doing a bit less"
    },
    {
      "index": 172,
      "start_seconds": 399,
      "source_timestamp": "6:39",
      "end_seconds": null,
      "text": "than that."
    },
    {
      "index": 173,
      "start_seconds": 403,
      "source_timestamp": "6:43",
      "end_seconds": null,
      "text": "Now, the other question that you will"
    },
    {
      "index": 174,
      "start_seconds": 404,
      "source_timestamp": "6:44",
      "end_seconds": null,
      "text": "might want to ask is basically, okay,"
    },
    {
      "index": 175,
      "start_seconds": 407,
      "source_timestamp": "6:47",
      "end_seconds": null,
      "text": "this is how much agents I'm finding from"
    },
    {
      "index": 176,
      "start_seconds": 409,
      "source_timestamp": "6:49",
      "end_seconds": null,
      "text": "the bugs we have in the reports."
    },
    {
      "index": 177,
      "start_seconds": 411,
      "source_timestamp": "6:51",
      "end_seconds": null,
      "text": "But what about the bugs that you don't"
    },
    {
      "index": 178,
      "start_seconds": 413,
      "source_timestamp": "6:53",
      "end_seconds": null,
      "text": "have in the reports? Did they find new"
    },
    {
      "index": 179,
      "start_seconds": 415,
      "source_timestamp": "6:55",
      "end_seconds": null,
      "text": "bugs that you missed?"
    },
    {
      "index": 180,
      "start_seconds": 417,
      "source_timestamp": "6:57",
      "end_seconds": null,
      "text": "Well, yes, they did. So they are also"
    },
    {
      "index": 181,
      "start_seconds": 419,
      "source_timestamp": "6:59",
      "end_seconds": null,
      "text": "finding bugs that we missed."
    },
    {
      "index": 182,
      "start_seconds": 422,
      "source_timestamp": "7:02",
      "end_seconds": null,
      "text": "Out of if we took the 10 most severe"
    },
    {
      "index": 183,
      "start_seconds": 425,
      "source_timestamp": "7:05",
      "end_seconds": null,
      "text": "findings in their reports and we"
    },
    {
      "index": 184,
      "start_seconds": 427,
      "source_timestamp": "7:07",
      "end_seconds": null,
      "text": "analyzed them, 4.5 of these 10 on"
    },
    {
      "index": 185,
      "start_seconds": 430,
      "source_timestamp": "7:10",
      "end_seconds": null,
      "text": "average are ones that we have in our"
    },
    {
      "index": 186,
      "start_seconds": 432,
      "source_timestamp": "7:12",
      "end_seconds": null,
      "text": "report and they reported"
    },
    {
      "index": 187,
      "start_seconds": 434,
      "source_timestamp": "7:14",
      "end_seconds": null,
      "text": "there's almost on average 1.1 bug that"
    },
    {
      "index": 188,
      "start_seconds": 437,
      "source_timestamp": "7:17",
      "end_seconds": null,
      "text": "is basically a new bug that we missed so"
    },
    {
      "index": 189,
      "start_seconds": 439,
      "source_timestamp": "7:19",
      "end_seconds": null,
      "text": "medium and above and the others that are"
    },
    {
      "index": 190,
      "start_seconds": 441,
      "source_timestamp": "7:21",
      "end_seconds": null,
      "text": "reporting basically are"
    },
    {
      "index": 191,
      "start_seconds": 443,
      "source_timestamp": "7:23",
      "end_seconds": null,
      "text": "less severe bugs. So out of"
    },
    {
      "index": 192,
      "start_seconds": 446,
      "source_timestamp": "7:26",
      "end_seconds": null,
      "text": "these eight projects, there was 16 bugs"
    },
    {
      "index": 193,
      "start_seconds": 450,
      "source_timestamp": "7:30",
      "end_seconds": null,
      "text": "medium and above that we have missed."
    },
    {
      "index": 194,
      "start_seconds": 453,
      "source_timestamp": "7:33",
      "end_seconds": null,
      "text": "Now, none of these are critical."
    },
    {
      "index": 195,
      "start_seconds": 455,
      "source_timestamp": "7:35",
      "end_seconds": null,
      "text": "There's two of them are high and 14 as"
    },
    {
      "index": 196,
      "start_seconds": 458,
      "source_timestamp": "7:38",
      "end_seconds": null,
      "text": "as medium."
    },
    {
      "index": 197,
      "start_seconds": 460,
      "source_timestamp": "7:40",
      "end_seconds": null,
      "text": "Okay?"
    },
    {
      "index": 198,
      "start_seconds": 461,
      "source_timestamp": "7:41",
      "end_seconds": null,
      "text": "Now, one disclaimer here to say that all"
    },
    {
      "index": 199,
      "start_seconds": 465,
      "source_timestamp": "7:45",
      "end_seconds": null,
      "text": "these"
    },
    {
      "index": 200,
      "start_seconds": 466,
      "source_timestamp": "7:46",
      "end_seconds": null,
      "text": "projects that we run the audits on the"
    },
    {
      "index": 201,
      "start_seconds": 468,
      "source_timestamp": "7:48",
      "end_seconds": null,
      "text": "experiments on are old projects."
    },
    {
      "index": 202,
      "start_seconds": 472,
      "source_timestamp": "7:52",
      "end_seconds": null,
      "text": "At the time we were not using AI. So all"
    },
    {
      "index": 203,
      "start_seconds": 474,
      "source_timestamp": "7:54",
      "end_seconds": null,
      "text": "these results does not rely on the fact"
    },
    {
      "index": 204,
      "start_seconds": 476,
      "source_timestamp": "7:56",
      "end_seconds": null,
      "text": "that currently we use AI. So basically"
    },
    {
      "index": 205,
      "start_seconds": 479,
      "source_timestamp": "7:59",
      "end_seconds": null,
      "text": "the results will be much much different"
    },
    {
      "index": 206,
      "start_seconds": 481,
      "source_timestamp": "8:01",
      "end_seconds": null,
      "text": "when you are actually integrating AI"
    },
    {
      "index": 207,
      "start_seconds": 482,
      "source_timestamp": "8:02",
      "end_seconds": null,
      "text": "yourself when you are doing security"
    },
    {
      "index": 208,
      "start_seconds": 484,
      "source_timestamp": "8:04",
      "end_seconds": null,
      "text": "audits as a human."
    },
    {
      "index": 209,
      "start_seconds": 485,
      "source_timestamp": "8:05",
      "end_seconds": null,
      "text": "So So comparing"
    },
    {
      "index": 210,
      "start_seconds": 487,
      "source_timestamp": "8:07",
      "end_seconds": null,
      "text": "pure human review to"
    },
    {
      "index": 211,
      "start_seconds": 490,
      "source_timestamp": "8:10",
      "end_seconds": null,
      "text": "pure AI review."
    },
    {
      "index": 212,
      "start_seconds": 493,
      "source_timestamp": "8:13",
      "end_seconds": null,
      "text": "Now this is a very important slide also"
    },
    {
      "index": 213,
      "start_seconds": 496,
      "source_timestamp": "8:16",
      "end_seconds": null,
      "text": "for people interested in uh"
    },
    {
      "index": 214,
      "start_seconds": 499,
      "source_timestamp": "8:19",
      "end_seconds": null,
      "text": "seeing the difference what did change"
    },
    {
      "index": 215,
      "start_seconds": 501,
      "source_timestamp": "8:21",
      "end_seconds": null,
      "text": "from June to until this point with the"
    },
    {
      "index": 216,
      "start_seconds": 503,
      "source_timestamp": "8:23",
      "end_seconds": null,
      "text": "new"
    },
    {
      "index": 217,
      "start_seconds": 504,
      "source_timestamp": "8:24",
      "end_seconds": null,
      "text": "uh release. Just to give you a reference"
    },
    {
      "index": 218,
      "start_seconds": 506,
      "source_timestamp": "8:26",
      "end_seconds": null,
      "text": "in June we were talking about Opus 4.1"
    },
    {
      "index": 219,
      "start_seconds": 509,
      "source_timestamp": "8:29",
      "end_seconds": null,
      "text": "4.8 for for Anthropic"
    },
    {
      "index": 220,
      "start_seconds": 512,
      "source_timestamp": "8:32",
      "end_seconds": null,
      "text": "and GPT 5.1"
    },
    {
      "index": 221,
      "start_seconds": 515,
      "source_timestamp": "8:35",
      "end_seconds": null,
      "text": "I think in June right? Now we're talking"
    },
    {
      "index": 222,
      "start_seconds": 517,
      "source_timestamp": "8:37",
      "end_seconds": null,
      "text": "about Opus 5.5 and Sol 6.1"
    },
    {
      "index": 223,
      "start_seconds": 522,
      "source_timestamp": "8:42",
      "end_seconds": null,
      "text": "uh and also like the Fable and Astra"
    },
    {
      "index": 224,
      "start_seconds": 524,
      "source_timestamp": "8:44",
      "end_seconds": null,
      "text": "which are the flagships."
    },
    {
      "index": 225,
      "start_seconds": 525,
      "source_timestamp": "8:45",
      "end_seconds": null,
      "text": "Um"
    },
    {
      "index": 226,
      "start_seconds": 527,
      "source_timestamp": "8:47",
      "end_seconds": null,
      "text": "here in the plane models this is very"
    },
    {
      "index": 227,
      "start_seconds": 530,
      "source_timestamp": "8:50",
      "end_seconds": null,
      "text": "interesting here that you can see that"
    },
    {
      "index": 228,
      "start_seconds": 533,
      "source_timestamp": "8:53",
      "end_seconds": null,
      "text": "the plane models in the current versions"
    },
    {
      "index": 229,
      "start_seconds": 536,
      "source_timestamp": "8:56",
      "end_seconds": null,
      "text": "now Astra and Fable"
    },
    {
      "index": 230,
      "start_seconds": 538,
      "source_timestamp": "8:58",
      "end_seconds": null,
      "text": "are doing much better than our previous"
    },
    {
      "index": 231,
      "start_seconds": 542,
      "source_timestamp": "9:02",
      "end_seconds": null,
      "text": "agents the the full agents we have in"
    },
    {
      "index": 232,
      "start_seconds": 545,
      "source_timestamp": "9:05",
      "end_seconds": null,
      "text": "June."
    },
    {
      "index": 233,
      "start_seconds": 546,
      "source_timestamp": "9:06",
      "end_seconds": null,
      "text": "So you have much better performance if"
    },
    {
      "index": 234,
      "start_seconds": 548,
      "source_timestamp": "9:08",
      "end_seconds": null,
      "text": "you just have a single prompt with Astra"
    },
    {
      "index": 235,
      "start_seconds": 551,
      "source_timestamp": "9:11",
      "end_seconds": null,
      "text": "and Fable"
    },
    {
      "index": 236,
      "start_seconds": 552,
      "source_timestamp": "9:12",
      "end_seconds": null,
      "text": "now than whatever tool you were using in"
    },
    {
      "index": 237,
      "start_seconds": 555,
      "source_timestamp": "9:15",
      "end_seconds": null,
      "text": "June."
    },
    {
      "index": 238,
      "start_seconds": 556,
      "source_timestamp": "9:16",
      "end_seconds": null,
      "text": "Right?"
    },
    {
      "index": 239,
      "start_seconds": 557,
      "source_timestamp": "9:17",
      "end_seconds": null,
      "text": "Uh also uh another thing you can notice"
    },
    {
      "index": 240,
      "start_seconds": 560,
      "source_timestamp": "9:20",
      "end_seconds": null,
      "text": "that previously"
    },
    {
      "index": 241,
      "start_seconds": 563,
      "source_timestamp": "9:23",
      "end_seconds": null,
      "text": "uh we were having like double"
    },
    {
      "index": 242,
      "start_seconds": 566,
      "source_timestamp": "9:26",
      "end_seconds": null,
      "text": "performance of plane models. Our agents"
    },
    {
      "index": 243,
      "start_seconds": 568,
      "source_timestamp": "9:28",
      "end_seconds": null,
      "text": "were doing double of what the plane"
    },
    {
      "index": 244,
      "start_seconds": 570,
      "source_timestamp": "9:30",
      "end_seconds": null,
      "text": "models were doing. Now it's it's a bit"
    },
    {
      "index": 245,
      "start_seconds": 573,
      "source_timestamp": "9:33",
      "end_seconds": null,
      "text": "less than double."
    },
    {
      "index": 246,
      "start_seconds": 574,
      "source_timestamp": "9:34",
      "end_seconds": null,
      "text": "So the gap is closing but you still have"
    },
    {
      "index": 247,
      "start_seconds": 577,
      "source_timestamp": "9:37",
      "end_seconds": null,
      "text": "better performance when you actually"
    },
    {
      "index": 248,
      "start_seconds": 580,
      "source_timestamp": "9:40",
      "end_seconds": null,
      "text": "build pipelines over the agents."
    },
    {
      "index": 249,
      "start_seconds": 583,
      "source_timestamp": "9:43",
      "end_seconds": null,
      "text": "And uh one uh"
    },
    {
      "index": 250,
      "start_seconds": 585,
      "source_timestamp": "9:45",
      "end_seconds": null,
      "text": "biggest improvement we have seen is the"
    },
    {
      "index": 251,
      "start_seconds": 586,
      "source_timestamp": "9:46",
      "end_seconds": null,
      "text": "languages that are not"
    },
    {
      "index": 252,
      "start_seconds": 588,
      "source_timestamp": "9:48",
      "end_seconds": null,
      "text": "uh were very common in June like they"
    },
    {
      "index": 253,
      "start_seconds": 591,
      "source_timestamp": "9:51",
      "end_seconds": null,
      "text": "are learning more trained more on"
    },
    {
      "index": 254,
      "start_seconds": 595,
      "source_timestamp": "9:55",
      "end_seconds": null,
      "text": "languages like Move and Ton than in"
    },
    {
      "index": 255,
      "start_seconds": 597,
      "source_timestamp": "9:57",
      "end_seconds": null,
      "text": "June. So they are being trained on these"
    },
    {
      "index": 256,
      "start_seconds": 599,
      "source_timestamp": "9:59",
      "end_seconds": null,
      "text": "new uh less common and famous languages."
    },
    {
      "index": 257,
      "start_seconds": 605,
      "source_timestamp": "10:05",
      "end_seconds": null,
      "text": "So um to answer the question so can AI"
    },
    {
      "index": 258,
      "start_seconds": 608,
      "source_timestamp": "10:08",
      "end_seconds": null,
      "text": "audit your smart contracts? Now, based"
    },
    {
      "index": 259,
      "start_seconds": 611,
      "source_timestamp": "10:11",
      "end_seconds": null,
      "text": "on these results,"
    },
    {
      "index": 260,
      "start_seconds": 612,
      "source_timestamp": "10:12",
      "end_seconds": null,
      "text": "what I would conclude is no, not on"
    },
    {
      "index": 261,
      "start_seconds": 615,
      "source_timestamp": "10:15",
      "end_seconds": null,
      "text": "their own. Now, if you are interested in"
    },
    {
      "index": 262,
      "start_seconds": 618,
      "source_timestamp": "10:18",
      "end_seconds": null,
      "text": "in doing some sort of a bug bounty,"
    },
    {
      "index": 263,
      "start_seconds": 621,
      "source_timestamp": "10:21",
      "end_seconds": null,
      "text": "well, it's very interesting. The the AI"
    },
    {
      "index": 264,
      "start_seconds": 623,
      "source_timestamp": "10:23",
      "end_seconds": null,
      "text": "can do a lot of good stuff and find"
    },
    {
      "index": 265,
      "start_seconds": 626,
      "source_timestamp": "10:26",
      "end_seconds": null,
      "text": "bugs. Because in bug bounties, what you"
    },
    {
      "index": 266,
      "start_seconds": 628,
      "source_timestamp": "10:28",
      "end_seconds": null,
      "text": "are interested in is basically finding a"
    },
    {
      "index": 267,
      "start_seconds": 631,
      "source_timestamp": "10:31",
      "end_seconds": null,
      "text": "bug in a protocol,"
    },
    {
      "index": 268,
      "start_seconds": 633,
      "source_timestamp": "10:33",
      "end_seconds": null,
      "text": "right? Doesn't have to be a specific"
    },
    {
      "index": 269,
      "start_seconds": 635,
      "source_timestamp": "10:35",
      "end_seconds": null,
      "text": "protocol. It doesn't have to be any"
    },
    {
      "index": 270,
      "start_seconds": 637,
      "source_timestamp": "10:37",
      "end_seconds": null,
      "text": "specific bug."
    },
    {
      "index": 271,
      "start_seconds": 639,
      "source_timestamp": "10:39",
      "end_seconds": null,
      "text": "Any severe bug would give you money."
    },
    {
      "index": 272,
      "start_seconds": 642,
      "source_timestamp": "10:42",
      "end_seconds": null,
      "text": "Now, audits"
    },
    {
      "index": 273,
      "start_seconds": 643,
      "source_timestamp": "10:43",
      "end_seconds": null,
      "text": "you don't you specifically want to find"
    },
    {
      "index": 274,
      "start_seconds": 646,
      "source_timestamp": "10:46",
      "end_seconds": null,
      "text": "bugs in this report and all bugs that"
    },
    {
      "index": 275,
      "start_seconds": 648,
      "source_timestamp": "10:48",
      "end_seconds": null,
      "text": "you can find."
    },
    {
      "index": 276,
      "start_seconds": 649,
      "source_timestamp": "10:49",
      "end_seconds": null,
      "text": "So your goal is to provide"
    },
    {
      "index": 277,
      "start_seconds": 652,
      "source_timestamp": "10:52",
      "end_seconds": null,
      "text": "a security assessment of this protocol"
    },
    {
      "index": 278,
      "start_seconds": 654,
      "source_timestamp": "10:54",
      "end_seconds": null,
      "text": "and like say this is a good posture for"
    },
    {
      "index": 279,
      "start_seconds": 656,
      "source_timestamp": "10:56",
      "end_seconds": null,
      "text": "the protocol to be deployed."
    },
    {
      "index": 280,
      "start_seconds": 658,
      "source_timestamp": "10:58",
      "end_seconds": null,
      "text": "So that's why relying only on AI agents"
    },
    {
      "index": 281,
      "start_seconds": 662,
      "source_timestamp": "11:02",
      "end_seconds": null,
      "text": "is not is not yet something that is a"
    },
    {
      "index": 282,
      "start_seconds": 664,
      "source_timestamp": "11:04",
      "end_seconds": null,
      "text": "good decision yet."
    },
    {
      "index": 283,
      "start_seconds": 665,
      "source_timestamp": "11:05",
      "end_seconds": null,
      "text": "But of course, we as security auditors,"
    },
    {
      "index": 284,
      "start_seconds": 669,
      "source_timestamp": "11:09",
      "end_seconds": null,
      "text": "we are using AI in our pipelines to do"
    },
    {
      "index": 285,
      "start_seconds": 672,
      "source_timestamp": "11:12",
      "end_seconds": null,
      "text": "security audits which are is increasing"
    },
    {
      "index": 286,
      "start_seconds": 675,
      "source_timestamp": "11:15",
      "end_seconds": null,
      "text": "a lot our efficiency, increasing a lot"
    },
    {
      "index": 287,
      "start_seconds": 677,
      "source_timestamp": "11:17",
      "end_seconds": null,
      "text": "our"
    },
    {
      "index": 288,
      "start_seconds": 678,
      "source_timestamp": "11:18",
      "end_seconds": null,
      "text": "coverage and"
    },
    {
      "index": 289,
      "start_seconds": 680,
      "source_timestamp": "11:20",
      "end_seconds": null,
      "text": "the the overall performance."
    },
    {
      "index": 290,
      "start_seconds": 685,
      "source_timestamp": "11:25",
      "end_seconds": null,
      "text": "So"
    },
    {
      "index": 291,
      "start_seconds": 686,
      "source_timestamp": "11:26",
      "end_seconds": null,
      "text": "thank you and this is the results for"
    },
    {
      "index": 292,
      "start_seconds": 688,
      "source_timestamp": "11:28",
      "end_seconds": null,
      "text": "now and I'm happy to take questions."
    },
    {
      "index": 293,
      "start_seconds": 691,
      "source_timestamp": "11:31",
      "end_seconds": null,
      "text": "[applause]"
    },
    {
      "index": 294,
      "start_seconds": 696,
      "source_timestamp": "11:36",
      "end_seconds": null,
      "text": "I have some time so I'll take some"
    },
    {
      "index": 295,
      "start_seconds": 697,
      "source_timestamp": "11:37",
      "end_seconds": null,
      "text": "questions, right?"
    },
    {
      "index": 296,
      "start_seconds": 701,
      "source_timestamp": "11:41",
      "end_seconds": null,
      "text": ">> Next question."
    },
    {
      "index": 297,
      "start_seconds": 712,
      "source_timestamp": "11:52",
      "end_seconds": null,
      "text": ">> That that's a great question. So"
    },
    {
      "index": 298,
      "start_seconds": 716,
      "source_timestamp": "11:56",
      "end_seconds": null,
      "text": "basically"
    },
    {
      "index": 299,
      "start_seconds": 718,
      "source_timestamp": "11:58",
      "end_seconds": null,
      "text": "in the pipelines"
    },
    {
      "index": 300,
      "start_seconds": 719,
      "source_timestamp": "11:59",
      "end_seconds": null,
      "text": "you have you guide the LLM to look where"
    },
    {
      "index": 301,
      "start_seconds": 723,
      "source_timestamp": "12:03",
      "end_seconds": null,
      "text": "to look and where to focus more."
    },
    {
      "index": 302,
      "start_seconds": 726,
      "source_timestamp": "12:06",
      "end_seconds": null,
      "text": "This is what what you get the best"
    },
    {
      "index": 303,
      "start_seconds": 728,
      "source_timestamp": "12:08",
      "end_seconds": null,
      "text": "improvement. Of course, you also add"
    },
    {
      "index": 304,
      "start_seconds": 729,
      "source_timestamp": "12:09",
      "end_seconds": null,
      "text": "tools,"
    },
    {
      "index": 305,
      "start_seconds": 731,
      "source_timestamp": "12:11",
      "end_seconds": null,
      "text": "but you're also seen that"
    },
    {
      "index": 306,
      "start_seconds": 734,
      "source_timestamp": "12:14",
      "end_seconds": null,
      "text": "like pipelines that basically used tools"
    },
    {
      "index": 307,
      "start_seconds": 736,
      "source_timestamp": "12:16",
      "end_seconds": null,
      "text": "and then triage them. But our like heavy"
    },
    {
      "index": 308,
      "start_seconds": 739,
      "source_timestamp": "12:19",
      "end_seconds": null,
      "text": "metal pipelines is basically more than"
    },
    {
      "index": 309,
      "start_seconds": 742,
      "source_timestamp": "12:22",
      "end_seconds": null,
      "text": "that which is make it makes"
    },
    {
      "index": 310,
      "start_seconds": 744,
      "source_timestamp": "12:24",
      "end_seconds": null,
      "text": "people"
    },
    {
      "index": 311,
      "start_seconds": 745,
      "source_timestamp": "12:25",
      "end_seconds": null,
      "text": "make makes LLM redirect on specific"
    },
    {
      "index": 312,
      "start_seconds": 747,
      "source_timestamp": "12:27",
      "end_seconds": null,
      "text": "parts of the code and do multiple rounds"
    },
    {
      "index": 313,
      "start_seconds": 749,
      "source_timestamp": "12:29",
      "end_seconds": null,
      "text": "and do different"
    },
    {
      "index": 314,
      "start_seconds": 752,
      "source_timestamp": "12:32",
      "end_seconds": null,
      "text": "instructions of how to look at this. So"
    },
    {
      "index": 315,
      "start_seconds": 755,
      "source_timestamp": "12:35",
      "end_seconds": null,
      "text": "do you want to look for what types of"
    },
    {
      "index": 316,
      "start_seconds": 756,
      "source_timestamp": "12:36",
      "end_seconds": null,
      "text": "bug?"
    },
    {
      "index": 317,
      "start_seconds": 757,
      "source_timestamp": "12:37",
      "end_seconds": null,
      "text": "So some of them based on like invariance"
    },
    {
      "index": 318,
      "start_seconds": 760,
      "source_timestamp": "12:40",
      "end_seconds": null,
      "text": "that you infer from the protocol and"
    },
    {
      "index": 319,
      "start_seconds": 761,
      "source_timestamp": "12:41",
      "end_seconds": null,
      "text": "then ask the protocol to to to check."
    },
    {
      "index": 320,
      "start_seconds": 763,
      "source_timestamp": "12:43",
      "end_seconds": null,
      "text": "Now what is"
    },
    {
      "index": 321,
      "start_seconds": 765,
      "source_timestamp": "12:45",
      "end_seconds": null,
      "text": "the recent models and flagship models is"
    },
    {
      "index": 322,
      "start_seconds": 767,
      "source_timestamp": "12:47",
      "end_seconds": null,
      "text": "trying to do is basically do this in"
    },
    {
      "index": 323,
      "start_seconds": 768,
      "source_timestamp": "12:48",
      "end_seconds": null,
      "text": "themselves. So they try to"
    },
    {
      "index": 324,
      "start_seconds": 772,
      "source_timestamp": "12:52",
      "end_seconds": null,
      "text": "do some sub agents create some sub"
    },
    {
      "index": 325,
      "start_seconds": 774,
      "source_timestamp": "12:54",
      "end_seconds": null,
      "text": "agents and give them these instructions."
    },
    {
      "index": 326,
      "start_seconds": 776,
      "source_timestamp": "12:56",
      "end_seconds": null,
      "text": "But you're still better"
    },
    {
      "index": 327,
      "start_seconds": 778,
      "source_timestamp": "12:58",
      "end_seconds": null,
      "text": "as a human auditor with very high skills"
    },
    {
      "index": 328,
      "start_seconds": 780,
      "source_timestamp": "13:00",
      "end_seconds": null,
      "text": "and experience in"
    },
    {
      "index": 329,
      "start_seconds": 781,
      "source_timestamp": "13:01",
      "end_seconds": null,
      "text": "in auditing"
    },
    {
      "index": 330,
      "start_seconds": 783,
      "source_timestamp": "13:03",
      "end_seconds": null,
      "text": "giving this instruction than AI yet. Now"
    },
    {
      "index": 331,
      "start_seconds": 785,
      "source_timestamp": "13:05",
      "end_seconds": null,
      "text": "they might be more efficient in"
    },
    {
      "index": 332,
      "start_seconds": 787,
      "source_timestamp": "13:07",
      "end_seconds": null,
      "text": "implementing these instructions but"
    },
    {
      "index": 333,
      "start_seconds": 789,
      "source_timestamp": "13:09",
      "end_seconds": null,
      "text": "you're still better giving this"
    },
    {
      "index": 334,
      "start_seconds": 790,
      "source_timestamp": "13:10",
      "end_seconds": null,
      "text": "instruction as a human. That's why you"
    },
    {
      "index": 335,
      "start_seconds": 792,
      "source_timestamp": "13:12",
      "end_seconds": null,
      "text": "get you get this this"
    },
    {
      "index": 336,
      "start_seconds": 794,
      "source_timestamp": "13:14",
      "end_seconds": null,
      "text": "you know"
    },
    {
      "index": 337,
      "start_seconds": 795,
      "source_timestamp": "13:15",
      "end_seconds": null,
      "text": "better performance."
    },
    {
      "index": 338,
      "start_seconds": 797,
      "source_timestamp": "13:17",
      "end_seconds": null,
      "text": "Yes please."
    },
    {
      "index": 339,
      "start_seconds": 799,
      "source_timestamp": "13:19",
      "end_seconds": null,
      "text": ">> You were talking about your own order"
    },
    {
      "index": 340,
      "start_seconds": 801,
      "source_timestamp": "13:21",
      "end_seconds": null,
      "text": "reports"
    },
    {
      "index": 341,
      "start_seconds": 804,
      "source_timestamp": "13:24",
      "end_seconds": null,
      "text": "How do you ensure that both in terms of"
    },
    {
      "index": 342,
      "start_seconds": 808,
      "source_timestamp": "13:28",
      "end_seconds": null,
      "text": "say the models including public reports"
    },
    {
      "index": 343,
      "start_seconds": 811,
      "source_timestamp": "13:31",
      "end_seconds": null,
      "text": "in their training"
    },
    {
      "index": 344,
      "start_seconds": 812,
      "source_timestamp": "13:32",
      "end_seconds": null,
      "text": "and maybe even inferring that from"
    },
    {
      "index": 345,
      "start_seconds": 815,
      "source_timestamp": "13:35",
      "end_seconds": null,
      "text": "like public bug fixes?"
    },
    {
      "index": 346,
      "start_seconds": 818,
      "source_timestamp": "13:38",
      "end_seconds": null,
      "text": ">> Yeah. So this is also a very good"
    },
    {
      "index": 347,
      "start_seconds": 821,
      "source_timestamp": "13:41",
      "end_seconds": null,
      "text": "question and we do our best but we don't"
    },
    {
      "index": 348,
      "start_seconds": 823,
      "source_timestamp": "13:43",
      "end_seconds": null,
      "text": "have a guarantee."
    },
    {
      "index": 349,
      "start_seconds": 825,
      "source_timestamp": "13:45",
      "end_seconds": null,
      "text": "What we do is basically of course we"
    },
    {
      "index": 350,
      "start_seconds": 826,
      "source_timestamp": "13:46",
      "end_seconds": null,
      "text": "remove all the materials that can give"
    },
    {
      "index": 351,
      "start_seconds": 829,
      "source_timestamp": "13:49",
      "end_seconds": null,
      "text": "him access directly like the git history"
    },
    {
      "index": 352,
      "start_seconds": 831,
      "source_timestamp": "13:51",
      "end_seconds": null,
      "text": "and stuff like that."
    },
    {
      "index": 353,
      "start_seconds": 833,
      "source_timestamp": "13:53",
      "end_seconds": null,
      "text": "But like making sure that the training"
    },
    {
      "index": 354,
      "start_seconds": 835,
      "source_timestamp": "13:55",
      "end_seconds": null,
      "text": "data have not seen our bug is something"
    },
    {
      "index": 355,
      "start_seconds": 837,
      "source_timestamp": "13:57",
      "end_seconds": null,
      "text": "that is"
    },
    {
      "index": 356,
      "start_seconds": 838,
      "source_timestamp": "13:58",
      "end_seconds": null,
      "text": "not very"
    },
    {
      "index": 357,
      "start_seconds": 840,
      "source_timestamp": "14:00",
      "end_seconds": null,
      "text": "not very easy to do."
    },
    {
      "index": 358,
      "start_seconds": 842,
      "source_timestamp": "14:02",
      "end_seconds": null,
      "text": "Now one suspect we have that the"
    },
    {
      "index": 359,
      "start_seconds": 845,
      "source_timestamp": "14:05",
      "end_seconds": null,
      "text": "additional performance and better"
    },
    {
      "index": 360,
      "start_seconds": 846,
      "source_timestamp": "14:06",
      "end_seconds": null,
      "text": "performance we had is because we have"
    },
    {
      "index": 361,
      "start_seconds": 848,
      "source_timestamp": "14:08",
      "end_seconds": null,
      "text": "trained we have did the experiment"
    },
    {
      "index": 362,
      "start_seconds": 850,
      "source_timestamp": "14:10",
      "end_seconds": null,
      "text": "before. So they trained with the new"
    },
    {
      "index": 363,
      "start_seconds": 852,
      "source_timestamp": "14:12",
      "end_seconds": null,
      "text": "models they've seen this then they are"
    },
    {
      "index": 364,
      "start_seconds": 854,
      "source_timestamp": "14:14",
      "end_seconds": null,
      "text": "giving better performance. That's that"
    },
    {
      "index": 365,
      "start_seconds": 855,
      "source_timestamp": "14:15",
      "end_seconds": null,
      "text": "that might be a limitation and there is."
    },
    {
      "index": 366,
      "start_seconds": 857,
      "source_timestamp": "14:17",
      "end_seconds": null,
      "text": "And then we have to assess it. This is"
    },
    {
      "index": 367,
      "start_seconds": 859,
      "source_timestamp": "14:19",
      "end_seconds": null,
      "text": "an ongoing work. Some of these results"
    },
    {
      "index": 368,
      "start_seconds": 861,
      "source_timestamp": "14:21",
      "end_seconds": null,
      "text": "are not finished yet. We are doing more"
    },
    {
      "index": 369,
      "start_seconds": 862,
      "source_timestamp": "14:22",
      "end_seconds": null,
      "text": "and more. We're going to continue doing"
    },
    {
      "index": 370,
      "start_seconds": 864,
      "source_timestamp": "14:24",
      "end_seconds": null,
      "text": "it and please subscribe to the to check"
    },
    {
      "index": 371,
      "start_seconds": 867,
      "source_timestamp": "14:27",
      "end_seconds": null,
      "text": "our"
    },
    {
      "index": 372,
      "start_seconds": 868,
      "source_timestamp": "14:28",
      "end_seconds": null,
      "text": "blog post to see how we're going to give"
    },
    {
      "index": 373,
      "start_seconds": 871,
      "source_timestamp": "14:31",
      "end_seconds": null,
      "text": "more and more results about this. And"
    },
    {
      "index": 374,
      "start_seconds": 873,
      "source_timestamp": "14:33",
      "end_seconds": null,
      "text": "yeah, it's it's a limitation. We don't"
    },
    {
      "index": 375,
      "start_seconds": 875,
      "source_timestamp": "14:35",
      "end_seconds": null,
      "text": "We don't know that they are not trained"
    },
    {
      "index": 376,
      "start_seconds": 877,
      "source_timestamp": "14:37",
      "end_seconds": null,
      "text": "on these bugs. Not all of these are"
    },
    {
      "index": 377,
      "start_seconds": 879,
      "source_timestamp": "14:39",
      "end_seconds": null,
      "text": "public reports. So also that's a good"
    },
    {
      "index": 378,
      "start_seconds": 882,
      "source_timestamp": "14:42",
      "end_seconds": null,
      "text": "thing. Not all of them, but some of"
    },
    {
      "index": 379,
      "start_seconds": 883,
      "source_timestamp": "14:43",
      "end_seconds": null,
      "text": "them."
    },
    {
      "index": 380,
      "start_seconds": 885,
      "source_timestamp": "14:45",
      "end_seconds": null,
      "text": "Yes."
    },
    {
      "index": 381,
      "start_seconds": 889,
      "source_timestamp": "14:49",
      "end_seconds": null,
      "text": "What what type of false positive are you"
    },
    {
      "index": 382,
      "start_seconds": 891,
      "source_timestamp": "14:51",
      "end_seconds": null,
      "text": "talking about?"
    },
    {
      "index": 383,
      "start_seconds": 892,
      "source_timestamp": "14:52",
      "end_seconds": null,
      "text": ">> False positive how many results"
    },
    {
      "index": 384,
      "start_seconds": 898,
      "source_timestamp": "14:58",
      "end_seconds": null,
      "text": ">> Yes. Now false positive could be like"
    },
    {
      "index": 385,
      "start_seconds": 900,
      "source_timestamp": "15:00",
      "end_seconds": null,
      "text": "invalid bugs or bugs that are flagged as"
    },
    {
      "index": 386,
      "start_seconds": 903,
      "source_timestamp": "15:03",
      "end_seconds": null,
      "text": "severe, but they are not. They are"
    },
    {
      "index": 387,
      "start_seconds": 905,
      "source_timestamp": "15:05",
      "end_seconds": null,
      "text": "likely because there's there's like a"
    },
    {
      "index": 388,
      "start_seconds": 908,
      "source_timestamp": "15:08",
      "end_seconds": null,
      "text": "line that's hard to put there."
    },
    {
      "index": 389,
      "start_seconds": 910,
      "source_timestamp": "15:10",
      "end_seconds": null,
      "text": "The line would be"
    },
    {
      "index": 390,
      "start_seconds": 912,
      "source_timestamp": "15:12",
      "end_seconds": null,
      "text": "some sometimes they will find a bug, but"
    },
    {
      "index": 391,
      "start_seconds": 915,
      "source_timestamp": "15:15",
      "end_seconds": null,
      "text": "when you assess it, okay, it's fine, but"
    },
    {
      "index": 392,
      "start_seconds": 917,
      "source_timestamp": "15:17",
      "end_seconds": null,
      "text": "it has like very hard assumptions. You"
    },
    {
      "index": 393,
      "start_seconds": 919,
      "source_timestamp": "15:19",
      "end_seconds": null,
      "text": "are assuming that the admin have not"
    },
    {
      "index": 394,
      "start_seconds": 921,
      "source_timestamp": "15:21",
      "end_seconds": null,
      "text": "missed mistake for that bug to appear"
    },
    {
      "index": 395,
      "start_seconds": 923,
      "source_timestamp": "15:23",
      "end_seconds": null,
      "text": "and this is unlikely to happen. So it's"
    },
    {
      "index": 396,
      "start_seconds": 925,
      "source_timestamp": "15:25",
      "end_seconds": null,
      "text": "not that severe or that it's an"
    },
    {
      "index": 397,
      "start_seconds": 927,
      "source_timestamp": "15:27",
      "end_seconds": null,
      "text": "informational. So if these are counted"
    },
    {
      "index": 398,
      "start_seconds": 929,
      "source_timestamp": "15:29",
      "end_seconds": null,
      "text": "as false positive, it's very hard to put"
    },
    {
      "index": 399,
      "start_seconds": 931,
      "source_timestamp": "15:31",
      "end_seconds": null,
      "text": "this line to count the false positive."
    },
    {
      "index": 400,
      "start_seconds": 933,
      "source_timestamp": "15:33",
      "end_seconds": null,
      "text": "But if we're talking about invalid"
    },
    {
      "index": 401,
      "start_seconds": 934,
      "source_timestamp": "15:34",
      "end_seconds": null,
      "text": "issues like hallucinations or like"
    },
    {
      "index": 402,
      "start_seconds": 936,
      "source_timestamp": "15:36",
      "end_seconds": null,
      "text": "assuming things that are not correct"
    },
    {
      "index": 403,
      "start_seconds": 937,
      "source_timestamp": "15:37",
      "end_seconds": null,
      "text": "about the code, then this is very very"
    },
    {
      "index": 404,
      "start_seconds": 940,
      "source_timestamp": "15:40",
      "end_seconds": null,
      "text": "low. And we have also done benchmarking"
    },
    {
      "index": 405,
      "start_seconds": 943,
      "source_timestamp": "15:43",
      "end_seconds": null,
      "text": "of our benchmarking. So basically"
    },
    {
      "index": 406,
      "start_seconds": 945,
      "source_timestamp": "15:45",
      "end_seconds": null,
      "text": "created hallucinations"
    },
    {
      "index": 407,
      "start_seconds": 947,
      "source_timestamp": "15:47",
      "end_seconds": null,
      "text": "bugs and tried to see how much they can"
    },
    {
      "index": 408,
      "start_seconds": 950,
      "source_timestamp": "15:50",
      "end_seconds": null,
      "text": "agree that they are bugs or something"
    },
    {
      "index": 409,
      "start_seconds": 952,
      "source_timestamp": "15:52",
      "end_seconds": null,
      "text": "like that. So we have done experiments"
    },
    {
      "index": 410,
      "start_seconds": 954,
      "source_timestamp": "15:54",
      "end_seconds": null,
      "text": "on this. We couldn't present everything,"
    },
    {
      "index": 411,
      "start_seconds": 955,
      "source_timestamp": "15:55",
      "end_seconds": null,
      "text": "but in terms of invalid bugs, they're"
    },
    {
      "index": 412,
      "start_seconds": 958,
      "source_timestamp": "15:58",
      "end_seconds": null,
      "text": "unlikely to to report invalid bugs. But"
    },
    {
      "index": 413,
      "start_seconds": 961,
      "source_timestamp": "16:01",
      "end_seconds": null,
      "text": "yeah, informational bugs and stuff like"
    },
    {
      "index": 414,
      "start_seconds": 963,
      "source_timestamp": "16:03",
      "end_seconds": null,
      "text": "that that they are exaggerating, this is"
    },
    {
      "index": 415,
      "start_seconds": 966,
      "source_timestamp": "16:06",
      "end_seconds": null,
      "text": "this is"
    },
    {
      "index": 416,
      "start_seconds": 967,
      "source_timestamp": "16:07",
      "end_seconds": null,
      "text": "possible."
    },
    {
      "index": 417,
      "start_seconds": 968,
      "source_timestamp": "16:08",
      "end_seconds": null,
      "text": "Exact numbers are not in here. We can"
    },
    {
      "index": 418,
      "start_seconds": 971,
      "source_timestamp": "16:11",
      "end_seconds": null,
      "text": "put some some stuff online"
    },
    {
      "index": 419,
      "start_seconds": 973,
      "source_timestamp": "16:13",
      "end_seconds": null,
      "text": "soon when we have more experiments."
    },
    {
      "index": 420,
      "start_seconds": 976,
      "source_timestamp": "16:16",
      "end_seconds": null,
      "text": "Okay."
    },
    {
      "index": 421,
      "start_seconds": 977,
      "source_timestamp": "16:17",
      "end_seconds": null,
      "text": "Thank you a lot."
    }
  ]
}
