0:00.000–0:03.300
So OpenAI are finally introducing their newest models,
0:03.460–0:08.120
GPT 5.6 Sol, GPT 5.6 Terra, and the GPT 5.6 Luna.
0:08.320–0:11.120
Now, this is really interesting because this comes at a time
0:11.120–0:14.640
where GPT 5.6 isn't being released publicly for us.
0:14.760–0:17.540
And I'll dive more into that later, so let's dive into this.
0:18.720–0:21.060
They are announcing three new models.
0:21.260–0:25.120
GPT 5.6 Sol is going to be the largest model, the Frontier model,
0:25.260–0:27.480
or I guess you could say all of your complex tasks.
0:27.480–0:31.620
GPT 5.6 Terra is going to be the balanced and efficient model for everyday use.
0:31.800–0:37.140
And then, of course, you will have GPT 5.6 Luna for your fast slash high volume tasks.
0:37.280–0:39.980
So essentially, they went from max to medium to mini.
0:40.160–0:43.940
And essentially, you can think of this as OpenAI changing their names
0:43.940–0:49.920
from max to mini and medium to the style of Fable, Opus, Sonnet, and Haiku.
0:50.160–0:54.300
Okay, so you can basically think of this as Opus for GPT 5.6 Sol,
0:54.300–0:59.000
Sonnet for GPT 5.6 Terra, and Haiku for GPT 5.6 Luna in terms of the sizing.
0:59.160–1:02.200
Now, one model that people actually kept missing in this announcement,
1:02.400–1:04.120
which is unfortunately buried in there,
1:04.420–1:07.320
is that they're actually announcing a new max reasoning effort
1:07.320–1:09.560
to give Sol the most time to reason deeply.
1:09.740–1:13.660
So for the max model, they're essentially introducing a new Ultra mode
1:13.660–1:16.180
that goes beyond the capabilities of a single agent
1:16.180–1:19.260
by leveraging sub-agents to accelerate complex work.
1:19.260–1:23.120
They're calling this the GPT 5.6 Sol Ultra.
1:23.320–1:25.280
So rather than just one model spinning up,
1:25.440–1:29.380
it spins up sub-agents to accelerate more complex tasks,
1:29.380–1:31.460
which I find to be super interesting.
1:31.620–1:34.220
Now, the first benchmark that we're going to be looking at is rather interesting
1:34.220–1:39.360
because this is a section that talks about the fact that GPT 5.6 Sol
1:39.360–1:41.980
is actually beating Claude Mythos 5.
1:42.100–1:43.620
Now, if you don't know what Terminal Bench is,
1:43.680–1:45.700
I'm not going to spend too much time diving into, you know,
1:45.780–1:47.520
benchmark maxing and all of that crazy stuff.
1:47.520–1:52.280
But essentially, Terminal Bench basically is how well do AI agents use a real command line?
1:52.340–1:53.620
Instead of just asking the model,
1:53.740–1:56.240
can the model answer a question or can it write a code patch?
1:56.280–1:58.820
It asks, can the agent sit inside a terminal, run the commands,
1:58.940–2:02.900
edit the file, install dependencies, debug errors, and finish a real task end-to-end?
2:03.020–2:04.440
So this was built by Stanford.
2:04.780–2:08.700
And the reason that this benchmark is one of the first ones for GPT 5.6 Sol
2:08.700–2:10.460
is because when we actually look at this,
2:10.520–2:15.000
on the left-hand side, you can see that GPT 5.6 Sol Ultra and GPT 5.6 Sol
2:15.000–2:18.960
actually does beat Claude Mythos, even though it's a very small amount.
2:19.040–2:20.920
It actually does surpass Claude Mythos,
2:21.020–2:23.040
which was, of course, this scary, scary model.
2:23.140–2:27.100
And it really does surpass Claude Fable 5 by a real big amount.
2:27.200–2:28.780
So this is super, super surprising
2:28.780–2:32.220
because even the GPT 5.6 Terra model surpasses Claude Fable 5,
2:32.300–2:35.040
which means if we're looking at where opening and focusing,
2:35.300–2:37.700
it does mean, and I'm going to show you guys why later on in this video,
2:37.800–2:40.460
why this is not just like one benchmark with a small improvement,
2:40.600–2:42.900
why this is even bigger when you look at the other factors.
2:42.900–2:46.440
And yes, I would argue that this chart is probably one of the most confusing charts
2:46.440–2:47.800
just because of the way things are.
2:47.960–2:49.200
But let's dive into the next one.
2:49.320–2:51.500
So the next one that we have here is Exploit Bench.
2:51.580–2:54.740
And this one, this benchmark is super interesting for a variety of reasons.
2:54.740–2:57.800
So I think the reason that most people were focused on this benchmark
2:57.800–3:02.100
is because number one, GPT 5.6 is competitive with Mythos Preview.
3:02.460–3:05.860
And it's actually basically so much cheaper than Mythos Preview
3:05.860–3:07.680
or Mythos 5, whatever you want to call it.
3:07.820–3:10.140
And I think this is really, really important to know
3:10.140–3:13.040
because GPT 5.6, and I'm going to show you guys why later,
3:13.160–3:15.600
is shaping up to be a model that I think a lot of people
3:15.600–3:17.060
are actually going to switch to and use
3:17.060–3:20.200
because it seems to be a lot more cost-effective than Mythos 5
3:20.200–3:24.040
and a lot more smarter in many of the areas where you'd want to use it.
3:24.180–3:26.120
So GPT 5.6, Sol, Terra, and Luna,
3:26.500–3:28.900
they all demonstrate strong improvements in cyber capabilities
3:28.900–3:30.260
as they increase their reasoning.
3:30.400–3:32.860
And the Exploit Bench benchmark is substantially testing
3:32.860–3:36.400
how capable AI agents are at real software exploitation,
3:36.620–3:37.500
not just coding trivia.
3:37.500–3:40.300
So the Exploit Bench basically gives an agent a real patch vulnerability
3:40.300–3:43.580
and asks, can you work your way from I found the bug to can I exploit it?
3:43.640–3:45.660
And so that is what the benchmark is doing.
3:45.840–3:48.840
Now, I do want to say that I will dive into later, okay,
3:48.980–3:51.560
how all of the model restrictions and stuff is going to go on.
3:51.800–3:55.040
But I do find it interesting that if you take a look at where Mythos 5 is
3:55.040–3:56.420
and where GPT 5.6 is,
3:56.480–3:59.120
it stops right just under that Mythos 5 level.
3:59.260–4:01.240
Very, very interesting if you ask me.
4:01.320–4:04.520
Now, of course, I think this does show that OpenAI are very much still in the race
4:04.520–4:07.800
and poised to lead if they continue developing models at this stage.
4:07.880–4:09.400
And once again, if we look at Exploit Gym,
4:09.680–4:11.620
we don't actually have the Fable benchmarks here,
4:11.820–4:15.940
but this basically just asked if the AI agent can turn known bugs into working attacks.
4:16.100–4:18.260
And essentially here, you can see the quality.
4:18.400–4:20.080
Now, of course, this is a little bit concerning,
4:20.280–4:22.480
but at the end of the day, this is where AI is headed.
4:22.620–4:24.360
Now, we're going to get into some more interesting things
4:24.360–4:27.380
because I think there is some stuff that people did miss in the areas, okay?
4:27.380–4:31.680
And I'm talking about the areas of the actual papers called the system card.
4:31.800–4:34.320
Now, something that I found to be pretty ridiculous
4:34.320–4:37.900
is that one of the things that you're measuring is the hallucinations.
4:38.040–4:43.520
So you can see here that there are still an absurd level of hallucinations in AI models.
4:43.680–4:46.580
Now, this isn't just to dog on OpenAI's newest announcement,
4:46.820–4:51.460
but measuring the model on cases users already flagged as, you know, factual error prone,
4:51.600–4:52.800
this is what this benchmark is doing.
4:52.880–4:54.500
So the data set here is basically,
4:54.640–4:57.020
here are conversations where previous models already messed up.
4:57.020–4:59.360
Now, test the new models on those same cursed cases.
4:59.480–5:03.280
And OpenAI describes this as not really representative of ordinary production traffic.
5:03.440–5:05.440
And of course, hallucinations remain a hard problem,
5:05.440–5:09.940
but it does really show you how models are still managing to hallucinate.
5:10.040–5:13.000
And the fact that, like, when we previously thought that with more models,
5:13.280–5:14.960
with more compute, it would solve this issue,
5:15.040–5:18.560
it doesn't really show us that there's any meaningful improvements in these models.
5:18.640–5:22.820
In fact, you could argue that the newer models actually hallucinate even more, okay?
5:23.100–5:24.340
Which is a little bit concerning.
5:24.340–5:27.320
And the thing I would say here is that what this, you know,
5:27.380–5:31.400
information should be relaying back to you is that you still need to ground your answer.
5:31.520–5:36.140
So if you're going to be using GPT-5.6 Sol, GPT-5.6 Terra, any of these models,
5:36.220–5:39.900
in fact, any AI model, this means you need to be still checking your models
5:39.900–5:41.200
because the world is messy.
5:41.320–5:42.260
There are long tail facts.
5:42.380–5:43.160
There are stale data.
5:43.300–5:44.440
There's ambiguous questions.
5:44.820–5:47.480
There are fake sources, change details, niche topics.
5:47.680–5:50.200
These are all of the areas where hallucinations happen.
5:50.340–5:51.300
So think about it like this.
5:51.300–5:55.300
The model can be good for reasoning but can still reason from a false premise.
5:55.560–5:56.740
And you have to understand that, like,
5:56.780–5:58.580
this is something that is probably going to persist
5:58.580–6:00.860
until we develop new architectures to solve them.
6:00.900–6:02.960
Because the way how LLMs are built inherently,
6:03.240–6:05.600
hallucinations are essentially a part of the design.
6:05.720–6:07.180
So you have to understand that, like,
6:07.240–6:08.600
if you're trying to do market research,
6:08.600–6:10.600
if you're trying to do medical advice, factual claims,
6:10.760–6:12.460
ground your data, ground it in data,
6:12.600–6:15.340
have it, you know, with citations and have it checks, okay?
6:15.400–6:17.640
Because this is something that I think a lot of people do miss.
6:17.640–6:20.840
And a lot of people just take what they're getting from the AI at face value.
6:21.040–6:22.760
So ground your AIs.
6:22.940–6:24.300
Ensure they're working from docs.
6:24.400–6:26.100
Ensure if they're saying something, it's completely cited
6:26.100–6:28.040
because a lot of people are going to miss this.
6:28.180–6:31.340
Now, you might be wondering, okay, well, this model seems smart.
6:31.440–6:33.540
But Fable 5 was designed to work.
6:33.640–6:35.460
And this is a model that actually looks like
6:35.460–6:37.340
it was just designed for benchmark maxing.
6:37.420–6:39.720
Now, this is where we start to get into a really weird area
6:39.720–6:40.600
because essentially,
6:40.860–6:42.860
METER, the company that essentially measures
6:42.860–6:45.460
how long a human task AI can do by itself,
6:45.460–6:49.000
actually struggles to measure just how good GPT 5.6 is
6:49.000–6:50.620
because the model was acting strangely.
6:50.820–6:54.240
The problem is that they tried to estimate GPT 5.6 time horizons,
6:54.340–6:55.680
but the estimate got super messy.
6:55.820–6:56.320
And why was this?
6:56.400–6:58.040
Well, GPT 5.6 had some runs,
6:58.140–7:00.080
looked to it like it was cheating, okay?
7:00.460–7:01.960
And gaming the benchmark.
7:02.220–7:04.060
Now, not necessarily malicious like a human villain,
7:04.180–7:05.940
but doing things like making the benchmark pass
7:05.940–7:07.540
without really solving the intended task.
7:07.600–7:10.140
So they decided different ways to count those weird runs.
7:10.260–7:12.460
And they looked at, you know, many different runs.
7:12.460–7:14.620
And, you know, counting cheating attempts as failures
7:14.620–7:15.820
is around 11 hours.
7:16.160–7:18.040
And counting some failed cheating attempts
7:18.040–7:20.780
as legitimate failures as success puts it at 270 hours.
7:20.880–7:22.200
And so their entire range for this,
7:22.240–7:24.000
if you're wondering how long can this thing
7:24.000–7:24.840
autonomously work for,
7:24.940–7:28.740
the funny part is the range is from 13 hours to 11,400 hours.
7:28.840–7:30.980
So their best guess is 71 hours,
7:31.100–7:32.300
but the uncertainty is so huge
7:32.300–7:33.780
that it could be way lower or way higher,
7:33.880–7:35.760
which is one of the times where they say
7:35.760–7:37.920
this is the first time that the error bars
7:37.920–7:39.580
literally break the chart, which is insane.
7:39.580–7:41.740
So, I mean, this is something that it's probably good
7:41.740–7:44.040
at long software slash research tasks,
7:44.180–7:47.120
but Mita cannot confidently say exactly how good
7:47.120–7:49.520
because the model is beyond the benchmarks measuring range,
7:49.580–7:50.620
which means we're going to have to get
7:50.620–7:51.320
a new benchmark for this.
7:51.380–7:52.700
Now, something that you guys are going to want
7:52.700–7:53.880
to pay attention to is pricing
7:53.880–7:55.500
because everyone, and I know everyone,
7:55.720–7:58.320
is, you know, right now feeling the credit crunch
7:58.320–8:00.560
because these constant models are being updated
8:00.560–8:02.160
in terms of the pricing and usage.
8:02.420–8:03.540
So pricing is going to be something
8:03.540–8:04.920
that you have to pay attention to.
8:05.040–8:08.900
And right now, it looks like that TPT 5.6 Soul
8:08.900–8:12.200
is going to be a model that is far more cost-effective.
8:12.200–8:13.700
The same for Terra
8:13.700–8:16.180
and the same for the other variants of the model.
8:16.320–8:18.480
Now, I think that this has been a long time coming
8:18.480–8:20.180
because Anthropic has been charging
8:20.180–8:22.360
very, very expensive prices for their models.
8:22.500–8:24.860
And unfortunately, sometimes those models
8:24.860–8:27.460
don't tend to give you back the best responses
8:27.460–8:29.280
during peak times of traffic.
8:29.420–8:30.900
I promise you guys, you're not crazy.
8:31.260–8:32.660
Sometimes there is degraded performance
8:32.660–8:34.400
and Anthropic doesn't really say anything.
8:34.520–8:37.120
So you can kind of feel like you're being stolen from,
8:37.120–8:38.660
in a sense, if you're paying for a service
8:38.660–8:41.120
and it just degrades without any real notice.
8:41.320–8:43.060
And this is something that I think is really important
8:43.060–8:44.640
because if you are going to be building systems,
8:44.740–8:46.440
you need reliable models and reliable systems.
8:46.640–8:48.320
So with the GPT 5.6 being nearly,
8:48.660–8:49.920
you know, I guess you could say 40%,
8:49.920–8:51.720
around 40% cheaper than these models,
8:51.820–8:53.040
I would say that this is something
8:53.040–8:54.080
that you need to pay attention to.
8:54.200–8:56.620
And maybe think about putting future systems
8:56.620–8:58.320
into the GPT 6 ecosystem
8:58.320–9:00.080
because one thing that OpenAI is doing
9:00.080–9:02.280
is they are going to be able to have cheaper models
9:02.280–9:03.500
because they're building their own chips,
9:03.800–9:05.080
building their own full stack.
9:05.080–9:06.280
So this is going to be something
9:06.280–9:07.600
that you should be paying attention to.
9:07.680–9:09.060
Now, one thing that I'm excited for,
9:09.100–9:10.740
and this is something that was also left out,
9:10.800–9:12.060
something that most people didn't see,
9:12.320–9:14.980
is the fact that GPT 5.6 Sol on Cerebrus
9:14.980–9:18.720
is actually coming up with 750 tokens per second in July.
9:18.840–9:21.360
Now, I don't think people realize just how fast this is
9:21.360–9:22.940
because you need to understand
9:22.940–9:25.600
that Cerebrus inference is going to be powering
9:25.600–9:26.940
the next wave of AI.
9:27.280–9:28.000
And when that happens,
9:28.100–9:30.360
we are going to have this crazy,
9:30.540–9:32.320
crazy intelligence explosion, guys.
9:32.560–9:33.980
And what I mean by intelligence explosion is,
9:33.980–9:36.300
you're going to be able to do a lot of stuff with that.
9:36.440–9:37.860
So what this is, okay,
9:37.880–9:39.000
if you don't know what Cerebrus is,
9:39.040–9:41.660
it's basically a chip that is designed for LLM inference.
9:41.740–9:42.720
And this is how quick it is.
9:42.780–9:44.240
So right now the video is paused,
9:44.320–9:45.260
but I'm going to show you guys
9:45.260–9:47.680
because I want you to understand just how quick it is.
9:47.680–9:48.680
Because the moment I hit play,
9:48.980–9:50.260
you're probably not going to even realize
9:50.260–9:51.300
what happens in the first second.
9:51.420–9:52.980
So this is basically the prompt,
9:53.140–9:54.260
implement a test in Python.
9:54.520–9:56.820
On the left, you can see a model was done already.
9:56.940–9:59.700
It was already finished coding in just like three seconds
9:59.700–10:02.580
because that was at 2,500 tokens per second.
10:02.580–10:05.400
But on the right, you've got Llama for Maverick
10:05.400–10:07.360
on an NVIDIA GPU being served
10:07.360–10:09.000
with how traditional LLMs work.
10:09.160–10:10.260
So think about it like this, guys.
10:10.400–10:12.520
Imagine you are using GPT 5.6
10:12.520–10:14.180
and you're able to get that model
10:14.180–10:17.780
and it's able to be served for 750 tokens per second.
10:17.880–10:21.160
You're going to get all of your responses so, so quickly.
10:21.340–10:22.660
So this is something that I think
10:22.660–10:23.840
most people are underestimating.
10:24.180–10:25.900
If OpenAI does manage to secure this,
10:26.140–10:27.360
this is going to be something that I think
10:27.360–10:30.520
is going to be completely valuable to all of the users.
10:30.520–10:32.780
So I would say look out for this when it does announce
10:32.780–10:34.020
because this is going to be something
10:34.020–10:35.100
that is super, super useful.
10:35.280–10:36.120
If you're tired of your model,
10:36.440–10:37.640
you know, waiting for your models to think
10:37.640–10:39.560
and think and think and burn through tokens,
10:39.880–10:42.200
Cerebrus inference with GPT 5.6 SOL,
10:42.280–10:43.800
that is going to be absolutely insane.
10:43.980–10:45.080
Now, guys, this is the time
10:45.080–10:46.300
where we need to get into the rollout
10:46.300–10:49.240
because the rollout is really interesting, okay?
10:49.480–10:50.680
The rollout, for now,
10:50.880–10:54.420
you don't have access to GPT 5.6 SOL, Terra or Luna.
10:54.560–10:55.360
And it says, for now,
10:55.400–10:56.840
at the request of the United States government,
10:56.980–10:58.360
they're starting with a limited preview
10:58.360–11:01.300
among a small group of trusted partners in Codex and the API.
11:01.460–11:03.560
So this is something that is pretty crazy, guys.
11:03.680–11:05.820
Right now, there is a model that is really smart
11:05.820–11:08.060
and it is not even allowed to be, you know,
11:08.180–11:09.360
given out for public use.
11:09.440–11:12.100
And I could make this video genuinely 40 minutes long
11:12.100–11:13.600
talking about all of the different ways
11:13.600–11:15.640
that, you know, things have transpired
11:15.640–11:16.440
and what's going to happen.
11:16.700–11:17.600
But I will upload a video
11:17.600–11:18.540
within the next couple of hours
11:18.540–11:19.220
explaining everything
11:19.220–11:20.400
because I don't want it to be too long.
11:20.500–11:22.160
But the gist of this is the fact
11:22.160–11:23.220
that like these models
11:23.220–11:25.360
now have cyber capabilities
11:25.360–11:28.020
that are apparently meaning that it's too dangerous
11:28.020–11:29.800
for it to be released to the public
11:29.800–11:32.360
out of fears of people using those systems
11:32.360–11:33.380
to jailbreak them
11:33.380–11:35.460
and then hack critical infrastructure.
11:35.900–11:36.700
Now, I've got to be honest,
11:36.740–11:38.000
I can't argue with that
11:38.000–11:39.260
because that makes sense.
11:39.380–11:41.800
But I will say that this is going to pose
11:41.800–11:43.600
some interesting consequences
11:43.600–11:45.360
for the AI industry as a whole.
11:45.460–11:47.520
And I think you guys need to pay attention to this
11:47.520–11:48.140
because if you aren't,
11:48.200–11:49.300
you're not going to be in the best position
11:49.300–11:51.400
to benefit when AI does change
11:51.400–11:53.160
in terms of how the access gets distributed.
11:53.160–11:55.500
So currently it says at their requests,
11:55.620–11:56.920
we're starting with a limited preview
11:56.920–11:58.800
for a small group of trusted partners
11:58.800–12:00.840
whose participation has been shared
12:00.840–12:01.480
with the government
12:01.480–12:02.920
before releasing more broadly.
12:03.080–12:03.920
And during this preview,
12:04.180–12:05.040
we will continue testing
12:05.040–12:06.640
and coordinating closely with our partners
12:06.640–12:08.520
as we work towards broader availability.
12:08.840–12:09.860
And so they're saying that,
12:09.960–12:11.580
look, right now we're just releasing this
12:11.580–12:12.560
to a bunch of companies.
12:12.560–12:14.820
That way we know who exactly is using this
12:14.820–12:16.960
and we can ensure that there's complete safety.
12:17.140–12:18.500
Now, OpenAI says in this
12:18.500–12:20.020
that they don't believe that this kind of,
12:20.260–12:21.020
you know, government access
12:21.020–12:22.580
should be the long term by default.
12:22.580–12:23.840
You know, it keeps the best tools
12:23.840–12:25.420
from the users, developers and enterprises
12:25.420–12:26.200
and cyber defenders
12:26.200–12:27.640
and global partners who need them.
12:27.700–12:29.380
But they're taking this short term step
12:29.380–12:31.540
because they believe it is the strongest path
12:31.540–12:33.500
to broader availability in the coming weeks
12:33.500–12:35.660
while they keep the administration at bay
12:35.660–12:36.400
and work with them
12:36.400–12:38.320
to develop the cyber executive order framework
12:38.320–12:39.480
and a repeatable process
12:39.480–12:40.680
for future model releases.
12:40.680–12:43.300
So this is why earlier I said,
12:43.680–12:45.500
are models, are AI companies
12:45.500–12:47.240
now going to do benchmark minimizing?
12:47.540–12:48.920
So the entire thing started
12:48.920–12:50.160
when Mythos was released
12:50.160–12:52.040
and Anthropik decided to fear monger
12:52.040–12:52.900
and basically say,
12:53.180–12:54.480
our AI is so good
12:54.480–12:55.620
and go with the marketing angle.
12:55.700–12:56.160
It's so good.
12:56.240–12:57.580
It can hack everyone and anything.
12:57.960–12:59.120
And the US government decided,
12:59.500–13:00.760
wait a minute, if that's so good,
13:00.840–13:01.980
maybe we shouldn't, you know,
13:02.040–13:03.200
allow this just to be free.
13:03.280–13:04.520
We're going to actually regulate this.
13:04.640–13:05.940
So now you can see
13:05.940–13:07.400
that the Mythos preview line,
13:07.660–13:08.700
in fact, let me get another image
13:08.700–13:10.000
because this is a little bit hard to see.
13:10.120–13:10.680
And this tweet here
13:10.680–13:11.980
basically explains it really well.
13:12.020–13:13.740
It says, goodbye, benchmark maxing.
13:13.860–13:14.900
Hello, benchmark minimizing.
13:15.200–13:16.200
Mythos is the new bar
13:16.200–13:17.920
and you must be very careful
13:17.920–13:18.520
to not pass it.
13:18.520–13:19.580
So essentially you can see right here
13:19.580–13:20.420
on exploit bench,
13:20.740–13:21.900
we have Opus 4.8.
13:21.960–13:22.440
That's fine.
13:22.500–13:23.740
But the Mythos level model,
13:23.820–13:25.920
we can see that GPT 5.6 Sol
13:25.920–13:27.780
is actually right underneath there.
13:27.860–13:28.920
Now, some would argue
13:28.920–13:29.920
that this is on purpose
13:29.920–13:31.080
because they don't want their model
13:31.080–13:31.940
to be, you know,
13:32.080–13:33.080
too better than Mythos
13:33.080–13:34.240
on the exploit bench
13:34.240–13:34.880
because if it is,
13:35.140–13:35.800
they know that it's going
13:35.800–13:36.820
to get regulated to hell.
13:36.920–13:37.620
And then, of course,
13:37.660–13:38.400
that just results
13:38.400–13:39.460
in a wider range of issues
13:39.460–13:40.340
because no one's going
13:40.340–13:42.000
to have access to front-end models
13:42.000–13:43.160
and OpenAI is just put
13:43.160–13:43.940
in a rough position.
13:44.280–13:45.780
I mean, it really is going
13:45.780–13:46.700
to be interesting to see
13:46.700–13:48.660
if models start benchmark minimizing
13:48.660–13:49.540
to say that, look,
13:49.600–13:50.500
this model isn't better
13:50.500–13:51.200
than Mythos 5.
13:51.480–13:52.880
There's no need to regulate it.
13:52.900–13:53.980
It will be very interesting.
13:54.140–13:55.820
Now, Sam Altman made a statement
13:55.820–13:56.740
and he says,
13:56.840–13:57.400
good news.
13:57.540–13:58.860
Of course, these models are smart.
13:58.980–13:59.760
You know, we're launching
13:59.760–14:01.340
GPT 5.6, yada, yada, yada.
14:01.660–14:02.740
But he said, bad news.
14:02.800–14:03.440
At the request
14:03.440–14:04.440
of the United States government,
14:04.600–14:05.740
you know, it's launching today
14:05.740–14:06.540
in limited preview
14:06.540–14:07.560
instead of the open access
14:07.560–14:08.280
we were planning on.
14:08.360–14:09.740
We are working with the government
14:09.740–14:10.900
to get general availability
14:10.900–14:12.420
as fast as we can.
14:12.420–14:13.560
And I think it is quite
14:13.560–14:15.000
a reasonable rollout models,
14:15.240–14:16.440
especially as they reach
14:16.440–14:17.340
significant new levels
14:17.340–14:17.960
of capability.
14:18.260–14:18.800
In this way,
14:18.900–14:20.240
it fits with our long-held strategy
14:20.240–14:21.220
of iterative deployment.
14:21.360–14:22.500
But this isn't quite the process
14:22.500–14:23.340
we think is optimal.
14:23.500–14:23.960
And they said,
14:24.020–14:24.660
they're going to be working
14:24.660–14:25.180
with the government
14:25.180–14:26.500
to basically get something
14:26.500–14:27.880
that works with their safeguards.
14:28.000–14:29.140
So at the end of the day,
14:29.200–14:29.600
right now,
14:29.600–14:30.200
they are saying that,
14:30.280–14:31.180
look, they are working
14:31.180–14:31.600
with the government
14:31.600–14:32.220
to get this model
14:32.220–14:33.380
in our hands as quick as possible,
14:33.380–14:34.120
but they don't know
14:34.120–14:35.200
how long that could take.
14:35.260–14:36.420
Which means that the future
14:36.420–14:37.900
development of AI model releases
14:37.900–14:38.740
is going to be
14:38.740–14:39.620
particularly interesting
14:39.620–14:40.680
because we won't always
14:40.680–14:41.800
have access to the frontier,
14:42.160–14:43.420
but perhaps we will know
14:43.420–14:43.980
about the frontier
14:43.980–14:45.140
and what is being developed.
14:45.320–14:46.280
Now, if you're wondering
14:46.280–14:47.580
about the actual,
14:47.740–14:48.020
you know,
14:48.240–14:49.000
I guess you could say
14:49.000–14:50.000
deep research
14:50.000–14:50.820
in terms of looking
14:50.820–14:51.720
into what the models
14:51.720–14:52.920
are actually capable of,
14:53.020–14:54.380
GPT 5.6 is being treated
14:54.380–14:55.720
as a high-risk capability
14:55.720–14:56.960
in both cybersecurity
14:56.960–14:58.060
and biological
14:58.060–14:59.120
and chemical domains.
14:59.520–15:00.760
And even for the cheaper
15:00.760–15:02.300
Terra and faster Luna versions,
15:02.660–15:03.580
OpenAI said this is
15:03.580–15:04.520
the first time
15:04.520–15:06.100
that smaller and faster models
15:06.100–15:06.880
in a family
15:06.880–15:08.820
received a high designation
15:08.820–15:11.000
in any tracked danger category.
15:11.100–15:12.020
So this is super interesting.
15:12.160–15:12.500
And then,
15:12.600–15:13.100
not just that,
15:13.140–15:14.280
but in the cybersecurity area,
15:14.680–15:15.700
GPT 5.6 Sol
15:15.700–15:16.820
saturated
15:16.820–15:18.020
OpenAI's internal
15:18.020–15:18.820
cyber challenge
15:18.820–15:20.440
set at 96.7%,
15:20.440–15:21.400
putting it above
15:21.400–15:22.520
the high threshold.
15:22.880–15:24.300
And external cyber testers
15:24.300–15:25.400
found that high-impact
15:25.400–15:26.000
zero days,
15:26.320–15:27.240
including one where
15:27.240–15:28.320
read-only users
15:28.320–15:29.960
could modify and delete data
15:29.960–15:31.680
in a widely deployed database.
15:32.040–15:32.200
So,
15:32.480–15:34.040
GPT 5.6 is clearly,
15:34.200–15:34.480
you know,
15:34.620–15:35.620
pretty dangerous
15:35.620–15:36.880
when it comes to cybersecurity.
15:37.200–15:37.320
Now,
15:37.540–15:37.780
guys,
15:37.780–15:39.080
I'm not trying to fear-monger
15:39.080–15:39.760
so the US government
15:39.760–15:40.520
can regulate the model.
15:40.840–15:42.000
That stuff is already happening.
15:42.320–15:43.780
I think what this is showing you
15:43.780–15:44.760
is that these models
15:44.760–15:45.520
are essentially
15:45.520–15:46.800
passing the threshold
15:46.800–15:47.860
for where they can now,
15:48.060–15:48.420
you know,
15:48.500–15:50.000
have those zero-day vulnerabilities
15:50.000–15:51.440
that actually affect,
15:51.600–15:52.660
maybe not critical infrastructure,
15:52.780–15:53.260
but seriously,
15:53.440–15:54.480
the economy and companies
15:54.480–15:55.480
and that kind of stuff,
15:55.500–15:56.240
which is pretty crazy.
15:56.560–15:56.700
Now,
15:56.900–15:57.720
if you're wondering about the bio,
15:58.000–15:58.560
the bio result
15:58.560–15:59.600
was just as revealing.
15:59.980–16:00.500
High-threshold
16:00.500–16:01.420
bio-evaluations
16:01.420–16:02.220
crossed the line,
16:02.520–16:03.500
while zero out of three
16:03.500–16:04.700
critical bio-devaluations
16:04.700–16:05.240
crossed it.
16:05.240–16:06.680
On virology troubleshooting,
16:06.820–16:08.660
GPT 5.6 scored 55%,
16:08.660–16:09.220
far above
16:09.220–16:11.360
the 31.0%
16:11.360–16:12.620
expert performance threshold,
16:12.920–16:13.580
and Secure Bio
16:13.580–16:14.920
found that GPT 5.6
16:14.920–16:15.680
reached new highs
16:15.680–16:17.300
on several expert bio tests,
16:17.560–16:19.540
including 68.4%
16:19.540–16:21.020
on human pathogen capabilities
16:21.020–16:22.400
and 68.3%
16:22.400–16:23.640
on world-class bio.
16:23.760–16:24.100
And the thing
16:24.100–16:24.680
that I want you guys
16:24.680–16:25.240
to understand here
16:25.240–16:26.300
is that while these models
16:26.300–16:27.540
seem like they're plateauing,
16:27.600–16:27.820
because,
16:28.120–16:28.480
of course,
16:28.820–16:29.080
maybe,
16:29.180–16:29.480
I don't know,
16:29.580–16:30.300
maybe you might not be
16:30.300–16:31.020
a bio-researcher,
16:31.080–16:31.560
maybe you are.
16:31.620–16:32.380
Unless you are in
16:32.380–16:33.220
these specific fields
16:33.220–16:33.980
like cybersecurity,
16:34.540–16:34.700
you know,
16:34.760–16:35.440
bio-research,
16:35.700–16:36.600
you aren't really
16:36.600–16:37.260
going to see
16:37.260–16:38.140
those big changes,
16:38.420–16:39.140
and these are where
16:39.140–16:39.900
the big changes
16:39.900–16:41.000
are actually occurring.
16:41.260–16:41.480
So,
16:41.600–16:42.360
it's really,
16:42.660–16:43.420
really interesting
16:43.420–16:43.900
to see
16:43.900–16:45.020
just how crazy it is.
16:45.260–16:45.820
And this is something
16:45.820–16:46.360
that nobody,
16:46.420–16:47.180
I didn't see anyone
16:47.180–16:47.820
talking about this
16:47.820–16:48.440
on the timeline.
16:48.600–16:49.860
GPT 5.6 Sol's
16:49.860–16:51.100
unsettling agent behavior.
16:51.240–16:51.380
So,
16:51.480–16:52.100
the agent behavior
16:52.100–16:52.640
is arguably
16:52.640–16:53.600
the most unsettling.
16:53.720–16:54.680
GPT 5.6 Sol
16:54.680–16:55.380
more often
16:55.380–16:56.700
goes beyond user intent
16:56.700–16:57.280
when coding,
16:57.660–16:58.460
including deleting
16:58.460–16:59.900
the wrong virtual machines,
17:00.020–17:00.800
claiming unfinished
17:00.800–17:01.740
research is verified,
17:01.740–17:02.560
and moving
17:02.560–17:03.540
cached credentials
17:03.540–17:04.660
without permission.
17:05.060–17:05.680
And Mita found
17:05.680–17:06.900
that GPT 5.6 Sol
17:06.900–17:07.980
sometimes tried
17:07.980–17:08.900
to game the test
17:08.900–17:09.500
instead of just
17:09.500–17:10.140
doing the task.
17:10.220–17:10.960
The bench row result
17:10.960–17:11.880
couldn't be treated
17:11.880–17:13.060
as a clean result
17:13.060–17:14.000
of raw capability.
17:14.200–17:14.520
They're basically
17:14.520–17:15.000
trying to say that,
17:15.100–17:15.160
look,
17:15.220–17:16.380
this model was just
17:16.380–17:17.480
acting very strange
17:17.480–17:18.680
and we don't even know
17:18.680–17:19.340
if we can trust
17:19.340–17:20.000
the model's results
17:20.000–17:20.660
because it didn't even
17:20.660–17:21.740
want to participate properly.
17:21.900–17:22.220
And so,
17:22.300–17:23.140
what we have here
17:23.140–17:23.820
is a situation
17:23.820–17:24.620
where nobody's
17:24.620–17:25.140
really talking about
17:25.140–17:25.540
the fact that
17:25.540–17:26.000
these models
17:26.000–17:26.840
aren't even behaving.
17:27.280–17:27.760
And I find that
17:27.760–17:28.180
super,
17:28.380–17:28.960
super interesting
17:28.960–17:30.120
because you have
17:30.120–17:30.560
to understand
17:30.560–17:31.180
that these models
17:31.180–17:32.620
are really black boxes.
17:32.840–17:32.980
So,
17:33.220–17:33.500
I mean,
17:33.740–17:34.060
honestly,
17:34.240–17:34.740
things are starting
17:34.740–17:35.360
to get into
17:35.360–17:36.040
the weird zone.
17:36.100–17:36.480
But let me know
17:36.480–17:37.100
what you guys think
17:37.100–17:37.760
about this model.
17:37.920–17:38.200
And so,
17:38.260–17:38.640
it's going to be
17:38.640–17:39.200
really interesting
17:39.200–17:40.660
to see where things
17:40.660–17:41.560
go from the feature
17:41.560–17:41.940
from here.
0:00.000–0:03.300
So OpenAI are finally introducing their newest models,
(此句尚無繁中翻譯)
0:03.460–0:08.120
GPT 5.6 Sol, GPT 5.6 Terra, and the GPT 5.6 Luna.
(此句尚無繁中翻譯)
0:08.320–0:11.120
Now, this is really interesting because this comes at a time
(此句尚無繁中翻譯)
0:11.120–0:14.640
where GPT 5.6 isn't being released publicly for us.
(此句尚無繁中翻譯)
0:14.760–0:17.540
And I'll dive more into that later, so let's dive into this.
(此句尚無繁中翻譯)
0:18.720–0:21.060
They are announcing three new models.
(此句尚無繁中翻譯)
0:21.260–0:25.120
GPT 5.6 Sol is going to be the largest model, the Frontier model,
(此句尚無繁中翻譯)
0:25.260–0:27.480
or I guess you could say all of your complex tasks.
(此句尚無繁中翻譯)
0:27.480–0:31.620
GPT 5.6 Terra is going to be the balanced and efficient model for everyday use.
(此句尚無繁中翻譯)
0:31.800–0:37.140
And then, of course, you will have GPT 5.6 Luna for your fast slash high volume tasks.
(此句尚無繁中翻譯)
0:37.280–0:39.980
So essentially, they went from max to medium to mini.
(此句尚無繁中翻譯)
0:40.160–0:43.940
And essentially, you can think of this as OpenAI changing their names
(此句尚無繁中翻譯)
0:43.940–0:49.920
from max to mini and medium to the style of Fable, Opus, Sonnet, and Haiku.
(此句尚無繁中翻譯)
0:50.160–0:54.300
Okay, so you can basically think of this as Opus for GPT 5.6 Sol,
(此句尚無繁中翻譯)
0:54.300–0:59.000
Sonnet for GPT 5.6 Terra, and Haiku for GPT 5.6 Luna in terms of the sizing.
(此句尚無繁中翻譯)
0:59.160–1:02.200
Now, one model that people actually kept missing in this announcement,
(此句尚無繁中翻譯)
1:02.400–1:04.120
which is unfortunately buried in there,
(此句尚無繁中翻譯)
1:04.420–1:07.320
is that they're actually announcing a new max reasoning effort
(此句尚無繁中翻譯)
1:07.320–1:09.560
to give Sol the most time to reason deeply.
(此句尚無繁中翻譯)
1:09.740–1:13.660
So for the max model, they're essentially introducing a new Ultra mode
(此句尚無繁中翻譯)
1:13.660–1:16.180
that goes beyond the capabilities of a single agent
(此句尚無繁中翻譯)
1:16.180–1:19.260
by leveraging sub-agents to accelerate complex work.
(此句尚無繁中翻譯)
1:19.260–1:23.120
They're calling this the GPT 5.6 Sol Ultra.
(此句尚無繁中翻譯)
1:23.320–1:25.280
So rather than just one model spinning up,
(此句尚無繁中翻譯)
1:25.440–1:29.380
it spins up sub-agents to accelerate more complex tasks,
(此句尚無繁中翻譯)
1:29.380–1:31.460
which I find to be super interesting.
(此句尚無繁中翻譯)
1:31.620–1:34.220
Now, the first benchmark that we're going to be looking at is rather interesting
(此句尚無繁中翻譯)
1:34.220–1:39.360
because this is a section that talks about the fact that GPT 5.6 Sol
(此句尚無繁中翻譯)
1:39.360–1:41.980
is actually beating Claude Mythos 5.
(此句尚無繁中翻譯)
1:42.100–1:43.620
Now, if you don't know what Terminal Bench is,
(此句尚無繁中翻譯)
1:43.680–1:45.700
I'm not going to spend too much time diving into, you know,
(此句尚無繁中翻譯)
1:45.780–1:47.520
benchmark maxing and all of that crazy stuff.
(此句尚無繁中翻譯)
1:47.520–1:52.280
But essentially, Terminal Bench basically is how well do AI agents use a real command line?
(此句尚無繁中翻譯)
1:52.340–1:53.620
Instead of just asking the model,
(此句尚無繁中翻譯)
1:53.740–1:56.240
can the model answer a question or can it write a code patch?
(此句尚無繁中翻譯)
1:56.280–1:58.820
It asks, can the agent sit inside a terminal, run the commands,
(此句尚無繁中翻譯)
1:58.940–2:02.900
edit the file, install dependencies, debug errors, and finish a real task end-to-end?
(此句尚無繁中翻譯)
2:03.020–2:04.440
So this was built by Stanford.
(此句尚無繁中翻譯)
2:04.780–2:08.700
And the reason that this benchmark is one of the first ones for GPT 5.6 Sol
(此句尚無繁中翻譯)
2:08.700–2:10.460
is because when we actually look at this,
(此句尚無繁中翻譯)
2:10.520–2:15.000
on the left-hand side, you can see that GPT 5.6 Sol Ultra and GPT 5.6 Sol
(此句尚無繁中翻譯)
2:15.000–2:18.960
actually does beat Claude Mythos, even though it's a very small amount.
(此句尚無繁中翻譯)
2:19.040–2:20.920
It actually does surpass Claude Mythos,
(此句尚無繁中翻譯)
2:21.020–2:23.040
which was, of course, this scary, scary model.
(此句尚無繁中翻譯)
2:23.140–2:27.100
And it really does surpass Claude Fable 5 by a real big amount.
(此句尚無繁中翻譯)
2:27.200–2:28.780
So this is super, super surprising
(此句尚無繁中翻譯)
2:28.780–2:32.220
because even the GPT 5.6 Terra model surpasses Claude Fable 5,
(此句尚無繁中翻譯)
2:32.300–2:35.040
which means if we're looking at where opening and focusing,
(此句尚無繁中翻譯)
2:35.300–2:37.700
it does mean, and I'm going to show you guys why later on in this video,
(此句尚無繁中翻譯)
2:37.800–2:40.460
why this is not just like one benchmark with a small improvement,
(此句尚無繁中翻譯)
2:40.600–2:42.900
why this is even bigger when you look at the other factors.
(此句尚無繁中翻譯)
2:42.900–2:46.440
And yes, I would argue that this chart is probably one of the most confusing charts
(此句尚無繁中翻譯)
2:46.440–2:47.800
just because of the way things are.
(此句尚無繁中翻譯)
2:47.960–2:49.200
But let's dive into the next one.
(此句尚無繁中翻譯)
2:49.320–2:51.500
So the next one that we have here is Exploit Bench.
(此句尚無繁中翻譯)
2:51.580–2:54.740
And this one, this benchmark is super interesting for a variety of reasons.
(此句尚無繁中翻譯)
2:54.740–2:57.800
So I think the reason that most people were focused on this benchmark
(此句尚無繁中翻譯)
2:57.800–3:02.100
is because number one, GPT 5.6 is competitive with Mythos Preview.
(此句尚無繁中翻譯)
3:02.460–3:05.860
And it's actually basically so much cheaper than Mythos Preview
(此句尚無繁中翻譯)
3:05.860–3:07.680
or Mythos 5, whatever you want to call it.
(此句尚無繁中翻譯)
3:07.820–3:10.140
And I think this is really, really important to know
(此句尚無繁中翻譯)
3:10.140–3:13.040
because GPT 5.6, and I'm going to show you guys why later,
(此句尚無繁中翻譯)
3:13.160–3:15.600
is shaping up to be a model that I think a lot of people
(此句尚無繁中翻譯)
3:15.600–3:17.060
are actually going to switch to and use
(此句尚無繁中翻譯)
3:17.060–3:20.200
because it seems to be a lot more cost-effective than Mythos 5
(此句尚無繁中翻譯)
3:20.200–3:24.040
and a lot more smarter in many of the areas where you'd want to use it.
(此句尚無繁中翻譯)
3:24.180–3:26.120
So GPT 5.6, Sol, Terra, and Luna,
(此句尚無繁中翻譯)
3:26.500–3:28.900
they all demonstrate strong improvements in cyber capabilities
(此句尚無繁中翻譯)
3:28.900–3:30.260
as they increase their reasoning.
(此句尚無繁中翻譯)
3:30.400–3:32.860
And the Exploit Bench benchmark is substantially testing
(此句尚無繁中翻譯)
3:32.860–3:36.400
how capable AI agents are at real software exploitation,
(此句尚無繁中翻譯)
3:36.620–3:37.500
not just coding trivia.
(此句尚無繁中翻譯)
3:37.500–3:40.300
So the Exploit Bench basically gives an agent a real patch vulnerability
(此句尚無繁中翻譯)
3:40.300–3:43.580
and asks, can you work your way from I found the bug to can I exploit it?
(此句尚無繁中翻譯)
3:43.640–3:45.660
And so that is what the benchmark is doing.
(此句尚無繁中翻譯)
3:45.840–3:48.840
Now, I do want to say that I will dive into later, okay,
(此句尚無繁中翻譯)
3:48.980–3:51.560
how all of the model restrictions and stuff is going to go on.
(此句尚無繁中翻譯)
3:51.800–3:55.040
But I do find it interesting that if you take a look at where Mythos 5 is
(此句尚無繁中翻譯)
3:55.040–3:56.420
and where GPT 5.6 is,
(此句尚無繁中翻譯)
3:56.480–3:59.120
it stops right just under that Mythos 5 level.
(此句尚無繁中翻譯)
3:59.260–4:01.240
Very, very interesting if you ask me.
(此句尚無繁中翻譯)
4:01.320–4:04.520
Now, of course, I think this does show that OpenAI are very much still in the race
(此句尚無繁中翻譯)
4:04.520–4:07.800
and poised to lead if they continue developing models at this stage.
(此句尚無繁中翻譯)
4:07.880–4:09.400
And once again, if we look at Exploit Gym,
(此句尚無繁中翻譯)
4:09.680–4:11.620
we don't actually have the Fable benchmarks here,
(此句尚無繁中翻譯)
4:11.820–4:15.940
but this basically just asked if the AI agent can turn known bugs into working attacks.
(此句尚無繁中翻譯)
4:16.100–4:18.260
And essentially here, you can see the quality.
(此句尚無繁中翻譯)
4:18.400–4:20.080
Now, of course, this is a little bit concerning,
(此句尚無繁中翻譯)
4:20.280–4:22.480
but at the end of the day, this is where AI is headed.
(此句尚無繁中翻譯)
4:22.620–4:24.360
Now, we're going to get into some more interesting things
(此句尚無繁中翻譯)
4:24.360–4:27.380
because I think there is some stuff that people did miss in the areas, okay?
(此句尚無繁中翻譯)
4:27.380–4:31.680
And I'm talking about the areas of the actual papers called the system card.
(此句尚無繁中翻譯)
4:31.800–4:34.320
Now, something that I found to be pretty ridiculous
(此句尚無繁中翻譯)
4:34.320–4:37.900
is that one of the things that you're measuring is the hallucinations.
(此句尚無繁中翻譯)
4:38.040–4:43.520
So you can see here that there are still an absurd level of hallucinations in AI models.
(此句尚無繁中翻譯)
4:43.680–4:46.580
Now, this isn't just to dog on OpenAI's newest announcement,
(此句尚無繁中翻譯)
4:46.820–4:51.460
but measuring the model on cases users already flagged as, you know, factual error prone,
(此句尚無繁中翻譯)
4:51.600–4:52.800
this is what this benchmark is doing.
(此句尚無繁中翻譯)
4:52.880–4:54.500
So the data set here is basically,
(此句尚無繁中翻譯)
4:54.640–4:57.020
here are conversations where previous models already messed up.
(此句尚無繁中翻譯)
4:57.020–4:59.360
Now, test the new models on those same cursed cases.
(此句尚無繁中翻譯)
4:59.480–5:03.280
And OpenAI describes this as not really representative of ordinary production traffic.
(此句尚無繁中翻譯)
5:03.440–5:05.440
And of course, hallucinations remain a hard problem,
(此句尚無繁中翻譯)
5:05.440–5:09.940
but it does really show you how models are still managing to hallucinate.
(此句尚無繁中翻譯)
5:10.040–5:13.000
And the fact that, like, when we previously thought that with more models,
(此句尚無繁中翻譯)
5:13.280–5:14.960
with more compute, it would solve this issue,
(此句尚無繁中翻譯)
5:15.040–5:18.560
it doesn't really show us that there's any meaningful improvements in these models.
(此句尚無繁中翻譯)
5:18.640–5:22.820
In fact, you could argue that the newer models actually hallucinate even more, okay?
(此句尚無繁中翻譯)
5:23.100–5:24.340
Which is a little bit concerning.
(此句尚無繁中翻譯)
5:24.340–5:27.320
And the thing I would say here is that what this, you know,
(此句尚無繁中翻譯)
5:27.380–5:31.400
information should be relaying back to you is that you still need to ground your answer.
(此句尚無繁中翻譯)
5:31.520–5:36.140
So if you're going to be using GPT-5.6 Sol, GPT-5.6 Terra, any of these models,
(此句尚無繁中翻譯)
5:36.220–5:39.900
in fact, any AI model, this means you need to be still checking your models
(此句尚無繁中翻譯)
5:39.900–5:41.200
because the world is messy.
(此句尚無繁中翻譯)
5:41.320–5:42.260
There are long tail facts.
(此句尚無繁中翻譯)
5:42.380–5:43.160
There are stale data.
(此句尚無繁中翻譯)
5:43.300–5:44.440
There's ambiguous questions.
(此句尚無繁中翻譯)
5:44.820–5:47.480
There are fake sources, change details, niche topics.
(此句尚無繁中翻譯)
5:47.680–5:50.200
These are all of the areas where hallucinations happen.
(此句尚無繁中翻譯)
5:50.340–5:51.300
So think about it like this.
(此句尚無繁中翻譯)
5:51.300–5:55.300
The model can be good for reasoning but can still reason from a false premise.
(此句尚無繁中翻譯)
5:55.560–5:56.740
And you have to understand that, like,
(此句尚無繁中翻譯)
5:56.780–5:58.580
this is something that is probably going to persist
(此句尚無繁中翻譯)
5:58.580–6:00.860
until we develop new architectures to solve them.
(此句尚無繁中翻譯)
6:00.900–6:02.960
Because the way how LLMs are built inherently,
(此句尚無繁中翻譯)
6:03.240–6:05.600
hallucinations are essentially a part of the design.
(此句尚無繁中翻譯)
6:05.720–6:07.180
So you have to understand that, like,
(此句尚無繁中翻譯)
6:07.240–6:08.600
if you're trying to do market research,
(此句尚無繁中翻譯)
6:08.600–6:10.600
if you're trying to do medical advice, factual claims,
(此句尚無繁中翻譯)
6:10.760–6:12.460
ground your data, ground it in data,
(此句尚無繁中翻譯)
6:12.600–6:15.340
have it, you know, with citations and have it checks, okay?
(此句尚無繁中翻譯)
6:15.400–6:17.640
Because this is something that I think a lot of people do miss.
(此句尚無繁中翻譯)
6:17.640–6:20.840
And a lot of people just take what they're getting from the AI at face value.
(此句尚無繁中翻譯)
6:21.040–6:22.760
So ground your AIs.
(此句尚無繁中翻譯)
6:22.940–6:24.300
Ensure they're working from docs.
(此句尚無繁中翻譯)
6:24.400–6:26.100
Ensure if they're saying something, it's completely cited
(此句尚無繁中翻譯)
6:26.100–6:28.040
because a lot of people are going to miss this.
(此句尚無繁中翻譯)
6:28.180–6:31.340
Now, you might be wondering, okay, well, this model seems smart.
(此句尚無繁中翻譯)
6:31.440–6:33.540
But Fable 5 was designed to work.
(此句尚無繁中翻譯)
6:33.640–6:35.460
And this is a model that actually looks like
(此句尚無繁中翻譯)
6:35.460–6:37.340
it was just designed for benchmark maxing.
(此句尚無繁中翻譯)
6:37.420–6:39.720
Now, this is where we start to get into a really weird area
(此句尚無繁中翻譯)
6:39.720–6:40.600
because essentially,
(此句尚無繁中翻譯)
6:40.860–6:42.860
METER, the company that essentially measures
(此句尚無繁中翻譯)
6:42.860–6:45.460
how long a human task AI can do by itself,
(此句尚無繁中翻譯)
6:45.460–6:49.000
actually struggles to measure just how good GPT 5.6 is
(此句尚無繁中翻譯)
6:49.000–6:50.620
because the model was acting strangely.
(此句尚無繁中翻譯)
6:50.820–6:54.240
The problem is that they tried to estimate GPT 5.6 time horizons,
(此句尚無繁中翻譯)
6:54.340–6:55.680
but the estimate got super messy.
(此句尚無繁中翻譯)
6:55.820–6:56.320
And why was this?
(此句尚無繁中翻譯)
6:56.400–6:58.040
Well, GPT 5.6 had some runs,
(此句尚無繁中翻譯)
6:58.140–7:00.080
looked to it like it was cheating, okay?
(此句尚無繁中翻譯)
7:00.460–7:01.960
And gaming the benchmark.
(此句尚無繁中翻譯)
7:02.220–7:04.060
Now, not necessarily malicious like a human villain,
(此句尚無繁中翻譯)
7:04.180–7:05.940
but doing things like making the benchmark pass
(此句尚無繁中翻譯)
7:05.940–7:07.540
without really solving the intended task.
(此句尚無繁中翻譯)
7:07.600–7:10.140
So they decided different ways to count those weird runs.
(此句尚無繁中翻譯)
7:10.260–7:12.460
And they looked at, you know, many different runs.
(此句尚無繁中翻譯)
7:12.460–7:14.620
And, you know, counting cheating attempts as failures
(此句尚無繁中翻譯)
7:14.620–7:15.820
is around 11 hours.
(此句尚無繁中翻譯)
7:16.160–7:18.040
And counting some failed cheating attempts
(此句尚無繁中翻譯)
7:18.040–7:20.780
as legitimate failures as success puts it at 270 hours.
(此句尚無繁中翻譯)
7:20.880–7:22.200
And so their entire range for this,
(此句尚無繁中翻譯)
7:22.240–7:24.000
if you're wondering how long can this thing
(此句尚無繁中翻譯)
7:24.000–7:24.840
autonomously work for,
(此句尚無繁中翻譯)
7:24.940–7:28.740
the funny part is the range is from 13 hours to 11,400 hours.
(此句尚無繁中翻譯)
7:28.840–7:30.980
So their best guess is 71 hours,
(此句尚無繁中翻譯)
7:31.100–7:32.300
but the uncertainty is so huge
(此句尚無繁中翻譯)
7:32.300–7:33.780
that it could be way lower or way higher,
(此句尚無繁中翻譯)
7:33.880–7:35.760
which is one of the times where they say
(此句尚無繁中翻譯)
7:35.760–7:37.920
this is the first time that the error bars
(此句尚無繁中翻譯)
7:37.920–7:39.580
literally break the chart, which is insane.
(此句尚無繁中翻譯)
7:39.580–7:41.740
So, I mean, this is something that it's probably good
(此句尚無繁中翻譯)
7:41.740–7:44.040
at long software slash research tasks,
(此句尚無繁中翻譯)
7:44.180–7:47.120
but Mita cannot confidently say exactly how good
(此句尚無繁中翻譯)
7:47.120–7:49.520
because the model is beyond the benchmarks measuring range,
(此句尚無繁中翻譯)
7:49.580–7:50.620
which means we're going to have to get
(此句尚無繁中翻譯)
7:50.620–7:51.320
a new benchmark for this.
(此句尚無繁中翻譯)
7:51.380–7:52.700
Now, something that you guys are going to want
(此句尚無繁中翻譯)
7:52.700–7:53.880
to pay attention to is pricing
(此句尚無繁中翻譯)
7:53.880–7:55.500
because everyone, and I know everyone,
(此句尚無繁中翻譯)
7:55.720–7:58.320
is, you know, right now feeling the credit crunch
(此句尚無繁中翻譯)
7:58.320–8:00.560
because these constant models are being updated
(此句尚無繁中翻譯)
8:00.560–8:02.160
in terms of the pricing and usage.
(此句尚無繁中翻譯)
8:02.420–8:03.540
So pricing is going to be something
(此句尚無繁中翻譯)
8:03.540–8:04.920
that you have to pay attention to.
(此句尚無繁中翻譯)
8:05.040–8:08.900
And right now, it looks like that TPT 5.6 Soul
(此句尚無繁中翻譯)
8:08.900–8:12.200
is going to be a model that is far more cost-effective.
(此句尚無繁中翻譯)
8:12.200–8:13.700
The same for Terra
(此句尚無繁中翻譯)
8:13.700–8:16.180
and the same for the other variants of the model.
(此句尚無繁中翻譯)
8:16.320–8:18.480
Now, I think that this has been a long time coming
(此句尚無繁中翻譯)
8:18.480–8:20.180
because Anthropic has been charging
(此句尚無繁中翻譯)
8:20.180–8:22.360
very, very expensive prices for their models.
(此句尚無繁中翻譯)
8:22.500–8:24.860
And unfortunately, sometimes those models
(此句尚無繁中翻譯)
8:24.860–8:27.460
don't tend to give you back the best responses
(此句尚無繁中翻譯)
8:27.460–8:29.280
during peak times of traffic.
(此句尚無繁中翻譯)
8:29.420–8:30.900
I promise you guys, you're not crazy.
(此句尚無繁中翻譯)
8:31.260–8:32.660
Sometimes there is degraded performance
(此句尚無繁中翻譯)
8:32.660–8:34.400
and Anthropic doesn't really say anything.
(此句尚無繁中翻譯)
8:34.520–8:37.120
So you can kind of feel like you're being stolen from,
(此句尚無繁中翻譯)
8:37.120–8:38.660
in a sense, if you're paying for a service
(此句尚無繁中翻譯)
8:38.660–8:41.120
and it just degrades without any real notice.
(此句尚無繁中翻譯)
8:41.320–8:43.060
And this is something that I think is really important
(此句尚無繁中翻譯)
8:43.060–8:44.640
because if you are going to be building systems,
(此句尚無繁中翻譯)
8:44.740–8:46.440
you need reliable models and reliable systems.
(此句尚無繁中翻譯)
8:46.640–8:48.320
So with the GPT 5.6 being nearly,
(此句尚無繁中翻譯)
8:48.660–8:49.920
you know, I guess you could say 40%,
(此句尚無繁中翻譯)
8:49.920–8:51.720
around 40% cheaper than these models,
(此句尚無繁中翻譯)
8:51.820–8:53.040
I would say that this is something
(此句尚無繁中翻譯)
8:53.040–8:54.080
that you need to pay attention to.
(此句尚無繁中翻譯)
8:54.200–8:56.620
And maybe think about putting future systems
(此句尚無繁中翻譯)
8:56.620–8:58.320
into the GPT 6 ecosystem
(此句尚無繁中翻譯)
8:58.320–9:00.080
because one thing that OpenAI is doing
(此句尚無繁中翻譯)
9:00.080–9:02.280
is they are going to be able to have cheaper models
(此句尚無繁中翻譯)
9:02.280–9:03.500
because they're building their own chips,
(此句尚無繁中翻譯)
9:03.800–9:05.080
building their own full stack.
(此句尚無繁中翻譯)
9:05.080–9:06.280
So this is going to be something
(此句尚無繁中翻譯)
9:06.280–9:07.600
that you should be paying attention to.
(此句尚無繁中翻譯)
9:07.680–9:09.060
Now, one thing that I'm excited for,
(此句尚無繁中翻譯)
9:09.100–9:10.740
and this is something that was also left out,
(此句尚無繁中翻譯)
9:10.800–9:12.060
something that most people didn't see,
(此句尚無繁中翻譯)
9:12.320–9:14.980
is the fact that GPT 5.6 Sol on Cerebrus
(此句尚無繁中翻譯)
9:14.980–9:18.720
is actually coming up with 750 tokens per second in July.
(此句尚無繁中翻譯)
9:18.840–9:21.360
Now, I don't think people realize just how fast this is
(此句尚無繁中翻譯)
9:21.360–9:22.940
because you need to understand
(此句尚無繁中翻譯)
9:22.940–9:25.600
that Cerebrus inference is going to be powering
(此句尚無繁中翻譯)
9:25.600–9:26.940
the next wave of AI.
(此句尚無繁中翻譯)
9:27.280–9:28.000
And when that happens,
(此句尚無繁中翻譯)
9:28.100–9:30.360
we are going to have this crazy,
(此句尚無繁中翻譯)
9:30.540–9:32.320
crazy intelligence explosion, guys.
(此句尚無繁中翻譯)
9:32.560–9:33.980
And what I mean by intelligence explosion is,
(此句尚無繁中翻譯)
9:33.980–9:36.300
you're going to be able to do a lot of stuff with that.
(此句尚無繁中翻譯)
9:36.440–9:37.860
So what this is, okay,
(此句尚無繁中翻譯)
9:37.880–9:39.000
if you don't know what Cerebrus is,
(此句尚無繁中翻譯)
9:39.040–9:41.660
it's basically a chip that is designed for LLM inference.
(此句尚無繁中翻譯)
9:41.740–9:42.720
And this is how quick it is.
(此句尚無繁中翻譯)
9:42.780–9:44.240
So right now the video is paused,
(此句尚無繁中翻譯)
9:44.320–9:45.260
but I'm going to show you guys
(此句尚無繁中翻譯)
9:45.260–9:47.680
because I want you to understand just how quick it is.
(此句尚無繁中翻譯)
9:47.680–9:48.680
Because the moment I hit play,
(此句尚無繁中翻譯)
9:48.980–9:50.260
you're probably not going to even realize
(此句尚無繁中翻譯)
9:50.260–9:51.300
what happens in the first second.
(此句尚無繁中翻譯)
9:51.420–9:52.980
So this is basically the prompt,
(此句尚無繁中翻譯)
9:53.140–9:54.260
implement a test in Python.
(此句尚無繁中翻譯)
9:54.520–9:56.820
On the left, you can see a model was done already.
(此句尚無繁中翻譯)
9:56.940–9:59.700
It was already finished coding in just like three seconds
(此句尚無繁中翻譯)
9:59.700–10:02.580
because that was at 2,500 tokens per second.
(此句尚無繁中翻譯)
10:02.580–10:05.400
But on the right, you've got Llama for Maverick
(此句尚無繁中翻譯)
10:05.400–10:07.360
on an NVIDIA GPU being served
(此句尚無繁中翻譯)
10:07.360–10:09.000
with how traditional LLMs work.
(此句尚無繁中翻譯)
10:09.160–10:10.260
So think about it like this, guys.
(此句尚無繁中翻譯)
10:10.400–10:12.520
Imagine you are using GPT 5.6
(此句尚無繁中翻譯)
10:12.520–10:14.180
and you're able to get that model
(此句尚無繁中翻譯)
10:14.180–10:17.780
and it's able to be served for 750 tokens per second.
(此句尚無繁中翻譯)
10:17.880–10:21.160
You're going to get all of your responses so, so quickly.
(此句尚無繁中翻譯)
10:21.340–10:22.660
So this is something that I think
(此句尚無繁中翻譯)
10:22.660–10:23.840
most people are underestimating.
(此句尚無繁中翻譯)
10:24.180–10:25.900
If OpenAI does manage to secure this,
(此句尚無繁中翻譯)
10:26.140–10:27.360
this is going to be something that I think
(此句尚無繁中翻譯)
10:27.360–10:30.520
is going to be completely valuable to all of the users.
(此句尚無繁中翻譯)
10:30.520–10:32.780
So I would say look out for this when it does announce
(此句尚無繁中翻譯)
10:32.780–10:34.020
because this is going to be something
(此句尚無繁中翻譯)
10:34.020–10:35.100
that is super, super useful.
(此句尚無繁中翻譯)
10:35.280–10:36.120
If you're tired of your model,
(此句尚無繁中翻譯)
10:36.440–10:37.640
you know, waiting for your models to think
(此句尚無繁中翻譯)
10:37.640–10:39.560
and think and think and burn through tokens,
(此句尚無繁中翻譯)
10:39.880–10:42.200
Cerebrus inference with GPT 5.6 SOL,
(此句尚無繁中翻譯)
10:42.280–10:43.800
that is going to be absolutely insane.
(此句尚無繁中翻譯)
10:43.980–10:45.080
Now, guys, this is the time
(此句尚無繁中翻譯)
10:45.080–10:46.300
where we need to get into the rollout
(此句尚無繁中翻譯)
10:46.300–10:49.240
because the rollout is really interesting, okay?
(此句尚無繁中翻譯)
10:49.480–10:50.680
The rollout, for now,
(此句尚無繁中翻譯)
10:50.880–10:54.420
you don't have access to GPT 5.6 SOL, Terra or Luna.
(此句尚無繁中翻譯)
10:54.560–10:55.360
And it says, for now,
(此句尚無繁中翻譯)
10:55.400–10:56.840
at the request of the United States government,
(此句尚無繁中翻譯)
10:56.980–10:58.360
they're starting with a limited preview
(此句尚無繁中翻譯)
10:58.360–11:01.300
among a small group of trusted partners in Codex and the API.
(此句尚無繁中翻譯)
11:01.460–11:03.560
So this is something that is pretty crazy, guys.
(此句尚無繁中翻譯)
11:03.680–11:05.820
Right now, there is a model that is really smart
(此句尚無繁中翻譯)
11:05.820–11:08.060
and it is not even allowed to be, you know,
(此句尚無繁中翻譯)
11:08.180–11:09.360
given out for public use.
(此句尚無繁中翻譯)
11:09.440–11:12.100
And I could make this video genuinely 40 minutes long
(此句尚無繁中翻譯)
11:12.100–11:13.600
talking about all of the different ways
(此句尚無繁中翻譯)
11:13.600–11:15.640
that, you know, things have transpired
(此句尚無繁中翻譯)
11:15.640–11:16.440
and what's going to happen.
(此句尚無繁中翻譯)
11:16.700–11:17.600
But I will upload a video
(此句尚無繁中翻譯)
11:17.600–11:18.540
within the next couple of hours
(此句尚無繁中翻譯)
11:18.540–11:19.220
explaining everything
(此句尚無繁中翻譯)
11:19.220–11:20.400
because I don't want it to be too long.
(此句尚無繁中翻譯)
11:20.500–11:22.160
But the gist of this is the fact
(此句尚無繁中翻譯)
11:22.160–11:23.220
that like these models
(此句尚無繁中翻譯)
11:23.220–11:25.360
now have cyber capabilities
(此句尚無繁中翻譯)
11:25.360–11:28.020
that are apparently meaning that it's too dangerous
(此句尚無繁中翻譯)
11:28.020–11:29.800
for it to be released to the public
(此句尚無繁中翻譯)
11:29.800–11:32.360
out of fears of people using those systems
(此句尚無繁中翻譯)
11:32.360–11:33.380
to jailbreak them
(此句尚無繁中翻譯)
11:33.380–11:35.460
and then hack critical infrastructure.
(此句尚無繁中翻譯)
11:35.900–11:36.700
Now, I've got to be honest,
(此句尚無繁中翻譯)
11:36.740–11:38.000
I can't argue with that
(此句尚無繁中翻譯)
11:38.000–11:39.260
because that makes sense.
(此句尚無繁中翻譯)
11:39.380–11:41.800
But I will say that this is going to pose
(此句尚無繁中翻譯)
11:41.800–11:43.600
some interesting consequences
(此句尚無繁中翻譯)
11:43.600–11:45.360
for the AI industry as a whole.
(此句尚無繁中翻譯)
11:45.460–11:47.520
And I think you guys need to pay attention to this
(此句尚無繁中翻譯)
11:47.520–11:48.140
because if you aren't,
(此句尚無繁中翻譯)
11:48.200–11:49.300
you're not going to be in the best position
(此句尚無繁中翻譯)
11:49.300–11:51.400
to benefit when AI does change
(此句尚無繁中翻譯)
11:51.400–11:53.160
in terms of how the access gets distributed.
(此句尚無繁中翻譯)
11:53.160–11:55.500
So currently it says at their requests,
(此句尚無繁中翻譯)
11:55.620–11:56.920
we're starting with a limited preview
(此句尚無繁中翻譯)
11:56.920–11:58.800
for a small group of trusted partners
(此句尚無繁中翻譯)
11:58.800–12:00.840
whose participation has been shared
(此句尚無繁中翻譯)
12:00.840–12:01.480
with the government
(此句尚無繁中翻譯)
12:01.480–12:02.920
before releasing more broadly.
(此句尚無繁中翻譯)
12:03.080–12:03.920
And during this preview,
(此句尚無繁中翻譯)
12:04.180–12:05.040
we will continue testing
(此句尚無繁中翻譯)
12:05.040–12:06.640
and coordinating closely with our partners
(此句尚無繁中翻譯)
12:06.640–12:08.520
as we work towards broader availability.
(此句尚無繁中翻譯)
12:08.840–12:09.860
And so they're saying that,
(此句尚無繁中翻譯)
12:09.960–12:11.580
look, right now we're just releasing this
(此句尚無繁中翻譯)
12:11.580–12:12.560
to a bunch of companies.
(此句尚無繁中翻譯)
12:12.560–12:14.820
That way we know who exactly is using this
(此句尚無繁中翻譯)
12:14.820–12:16.960
and we can ensure that there's complete safety.
(此句尚無繁中翻譯)
12:17.140–12:18.500
Now, OpenAI says in this
(此句尚無繁中翻譯)
12:18.500–12:20.020
that they don't believe that this kind of,
(此句尚無繁中翻譯)
12:20.260–12:21.020
you know, government access
(此句尚無繁中翻譯)
12:21.020–12:22.580
should be the long term by default.
(此句尚無繁中翻譯)
12:22.580–12:23.840
You know, it keeps the best tools
(此句尚無繁中翻譯)
12:23.840–12:25.420
from the users, developers and enterprises
(此句尚無繁中翻譯)
12:25.420–12:26.200
and cyber defenders
(此句尚無繁中翻譯)
12:26.200–12:27.640
and global partners who need them.
(此句尚無繁中翻譯)
12:27.700–12:29.380
But they're taking this short term step
(此句尚無繁中翻譯)
12:29.380–12:31.540
because they believe it is the strongest path
(此句尚無繁中翻譯)
12:31.540–12:33.500
to broader availability in the coming weeks
(此句尚無繁中翻譯)
12:33.500–12:35.660
while they keep the administration at bay
(此句尚無繁中翻譯)
12:35.660–12:36.400
and work with them
(此句尚無繁中翻譯)
12:36.400–12:38.320
to develop the cyber executive order framework
(此句尚無繁中翻譯)
12:38.320–12:39.480
and a repeatable process
(此句尚無繁中翻譯)
12:39.480–12:40.680
for future model releases.
(此句尚無繁中翻譯)
12:40.680–12:43.300
So this is why earlier I said,
(此句尚無繁中翻譯)
12:43.680–12:45.500
are models, are AI companies
(此句尚無繁中翻譯)
12:45.500–12:47.240
now going to do benchmark minimizing?
(此句尚無繁中翻譯)
12:47.540–12:48.920
So the entire thing started
(此句尚無繁中翻譯)
12:48.920–12:50.160
when Mythos was released
(此句尚無繁中翻譯)
12:50.160–12:52.040
and Anthropik decided to fear monger
(此句尚無繁中翻譯)
12:52.040–12:52.900
and basically say,
(此句尚無繁中翻譯)
12:53.180–12:54.480
our AI is so good
(此句尚無繁中翻譯)
12:54.480–12:55.620
and go with the marketing angle.
(此句尚無繁中翻譯)
12:55.700–12:56.160
It's so good.
(此句尚無繁中翻譯)
12:56.240–12:57.580
It can hack everyone and anything.
(此句尚無繁中翻譯)
12:57.960–12:59.120
And the US government decided,
(此句尚無繁中翻譯)
12:59.500–13:00.760
wait a minute, if that's so good,
(此句尚無繁中翻譯)
13:00.840–13:01.980
maybe we shouldn't, you know,
(此句尚無繁中翻譯)
13:02.040–13:03.200
allow this just to be free.
(此句尚無繁中翻譯)
13:03.280–13:04.520
We're going to actually regulate this.
(此句尚無繁中翻譯)
13:04.640–13:05.940
So now you can see
(此句尚無繁中翻譯)
13:05.940–13:07.400
that the Mythos preview line,
(此句尚無繁中翻譯)
13:07.660–13:08.700
in fact, let me get another image
(此句尚無繁中翻譯)
13:08.700–13:10.000
because this is a little bit hard to see.
(此句尚無繁中翻譯)
13:10.120–13:10.680
And this tweet here
(此句尚無繁中翻譯)
13:10.680–13:11.980
basically explains it really well.
(此句尚無繁中翻譯)
13:12.020–13:13.740
It says, goodbye, benchmark maxing.
(此句尚無繁中翻譯)
13:13.860–13:14.900
Hello, benchmark minimizing.
(此句尚無繁中翻譯)
13:15.200–13:16.200
Mythos is the new bar
(此句尚無繁中翻譯)
13:16.200–13:17.920
and you must be very careful
(此句尚無繁中翻譯)
13:17.920–13:18.520
to not pass it.
(此句尚無繁中翻譯)
13:18.520–13:19.580
So essentially you can see right here
(此句尚無繁中翻譯)
13:19.580–13:20.420
on exploit bench,
(此句尚無繁中翻譯)
13:20.740–13:21.900
we have Opus 4.8.
(此句尚無繁中翻譯)
13:21.960–13:22.440
That's fine.
(此句尚無繁中翻譯)
13:22.500–13:23.740
But the Mythos level model,
(此句尚無繁中翻譯)
13:23.820–13:25.920
we can see that GPT 5.6 Sol
(此句尚無繁中翻譯)
13:25.920–13:27.780
is actually right underneath there.
(此句尚無繁中翻譯)
13:27.860–13:28.920
Now, some would argue
(此句尚無繁中翻譯)
13:28.920–13:29.920
that this is on purpose
(此句尚無繁中翻譯)
13:29.920–13:31.080
because they don't want their model
(此句尚無繁中翻譯)
13:31.080–13:31.940
to be, you know,
(此句尚無繁中翻譯)
13:32.080–13:33.080
too better than Mythos
(此句尚無繁中翻譯)
13:33.080–13:34.240
on the exploit bench
(此句尚無繁中翻譯)
13:34.240–13:34.880
because if it is,
(此句尚無繁中翻譯)
13:35.140–13:35.800
they know that it's going
(此句尚無繁中翻譯)
13:35.800–13:36.820
to get regulated to hell.
(此句尚無繁中翻譯)
13:36.920–13:37.620
And then, of course,
(此句尚無繁中翻譯)
13:37.660–13:38.400
that just results
(此句尚無繁中翻譯)
13:38.400–13:39.460
in a wider range of issues
(此句尚無繁中翻譯)
13:39.460–13:40.340
because no one's going
(此句尚無繁中翻譯)
13:40.340–13:42.000
to have access to front-end models
(此句尚無繁中翻譯)
13:42.000–13:43.160
and OpenAI is just put
(此句尚無繁中翻譯)
13:43.160–13:43.940
in a rough position.
(此句尚無繁中翻譯)
13:44.280–13:45.780
I mean, it really is going
(此句尚無繁中翻譯)
13:45.780–13:46.700
to be interesting to see
(此句尚無繁中翻譯)
13:46.700–13:48.660
if models start benchmark minimizing
(此句尚無繁中翻譯)
13:48.660–13:49.540
to say that, look,
(此句尚無繁中翻譯)
13:49.600–13:50.500
this model isn't better
(此句尚無繁中翻譯)
13:50.500–13:51.200
than Mythos 5.
(此句尚無繁中翻譯)
13:51.480–13:52.880
There's no need to regulate it.
(此句尚無繁中翻譯)
13:52.900–13:53.980
It will be very interesting.
(此句尚無繁中翻譯)
13:54.140–13:55.820
Now, Sam Altman made a statement
(此句尚無繁中翻譯)
13:55.820–13:56.740
and he says,
(此句尚無繁中翻譯)
13:56.840–13:57.400
good news.
(此句尚無繁中翻譯)
13:57.540–13:58.860
Of course, these models are smart.
(此句尚無繁中翻譯)
13:58.980–13:59.760
You know, we're launching
(此句尚無繁中翻譯)
13:59.760–14:01.340
GPT 5.6, yada, yada, yada.
(此句尚無繁中翻譯)
14:01.660–14:02.740
But he said, bad news.
(此句尚無繁中翻譯)
14:02.800–14:03.440
At the request
(此句尚無繁中翻譯)
14:03.440–14:04.440
of the United States government,
(此句尚無繁中翻譯)
14:04.600–14:05.740
you know, it's launching today
(此句尚無繁中翻譯)
14:05.740–14:06.540
in limited preview
(此句尚無繁中翻譯)
14:06.540–14:07.560
instead of the open access
(此句尚無繁中翻譯)
14:07.560–14:08.280
we were planning on.
(此句尚無繁中翻譯)
14:08.360–14:09.740
We are working with the government
(此句尚無繁中翻譯)
14:09.740–14:10.900
to get general availability
(此句尚無繁中翻譯)
14:10.900–14:12.420
as fast as we can.
(此句尚無繁中翻譯)
14:12.420–14:13.560
And I think it is quite
(此句尚無繁中翻譯)
14:13.560–14:15.000
a reasonable rollout models,
(此句尚無繁中翻譯)
14:15.240–14:16.440
especially as they reach
(此句尚無繁中翻譯)
14:16.440–14:17.340
significant new levels
(此句尚無繁中翻譯)
14:17.340–14:17.960
of capability.
(此句尚無繁中翻譯)
14:18.260–14:18.800
In this way,
(此句尚無繁中翻譯)
14:18.900–14:20.240
it fits with our long-held strategy
(此句尚無繁中翻譯)
14:20.240–14:21.220
of iterative deployment.
(此句尚無繁中翻譯)
14:21.360–14:22.500
But this isn't quite the process
(此句尚無繁中翻譯)
14:22.500–14:23.340
we think is optimal.
(此句尚無繁中翻譯)
14:23.500–14:23.960
And they said,
(此句尚無繁中翻譯)
14:24.020–14:24.660
they're going to be working
(此句尚無繁中翻譯)
14:24.660–14:25.180
with the government
(此句尚無繁中翻譯)
14:25.180–14:26.500
to basically get something
(此句尚無繁中翻譯)
14:26.500–14:27.880
that works with their safeguards.
(此句尚無繁中翻譯)
14:28.000–14:29.140
So at the end of the day,
(此句尚無繁中翻譯)
14:29.200–14:29.600
right now,
(此句尚無繁中翻譯)
14:29.600–14:30.200
they are saying that,
(此句尚無繁中翻譯)
14:30.280–14:31.180
look, they are working
(此句尚無繁中翻譯)
14:31.180–14:31.600
with the government
(此句尚無繁中翻譯)
14:31.600–14:32.220
to get this model
(此句尚無繁中翻譯)
14:32.220–14:33.380
in our hands as quick as possible,
(此句尚無繁中翻譯)
14:33.380–14:34.120
but they don't know
(此句尚無繁中翻譯)
14:34.120–14:35.200
how long that could take.
(此句尚無繁中翻譯)
14:35.260–14:36.420
Which means that the future
(此句尚無繁中翻譯)
14:36.420–14:37.900
development of AI model releases
(此句尚無繁中翻譯)
14:37.900–14:38.740
is going to be
(此句尚無繁中翻譯)
14:38.740–14:39.620
particularly interesting
(此句尚無繁中翻譯)
14:39.620–14:40.680
because we won't always
(此句尚無繁中翻譯)
14:40.680–14:41.800
have access to the frontier,
(此句尚無繁中翻譯)
14:42.160–14:43.420
but perhaps we will know
(此句尚無繁中翻譯)
14:43.420–14:43.980
about the frontier
(此句尚無繁中翻譯)
14:43.980–14:45.140
and what is being developed.
(此句尚無繁中翻譯)
14:45.320–14:46.280
Now, if you're wondering
(此句尚無繁中翻譯)
14:46.280–14:47.580
about the actual,
(此句尚無繁中翻譯)
14:47.740–14:48.020
you know,
(此句尚無繁中翻譯)
14:48.240–14:49.000
I guess you could say
(此句尚無繁中翻譯)
14:49.000–14:50.000
deep research
(此句尚無繁中翻譯)
14:50.000–14:50.820
in terms of looking
(此句尚無繁中翻譯)
14:50.820–14:51.720
into what the models
(此句尚無繁中翻譯)
14:51.720–14:52.920
are actually capable of,
(此句尚無繁中翻譯)
14:53.020–14:54.380
GPT 5.6 is being treated
(此句尚無繁中翻譯)
14:54.380–14:55.720
as a high-risk capability
(此句尚無繁中翻譯)
14:55.720–14:56.960
in both cybersecurity
(此句尚無繁中翻譯)
14:56.960–14:58.060
and biological
(此句尚無繁中翻譯)
14:58.060–14:59.120
and chemical domains.
(此句尚無繁中翻譯)
14:59.520–15:00.760
And even for the cheaper
(此句尚無繁中翻譯)
15:00.760–15:02.300
Terra and faster Luna versions,
(此句尚無繁中翻譯)
15:02.660–15:03.580
OpenAI said this is
(此句尚無繁中翻譯)
15:03.580–15:04.520
the first time
(此句尚無繁中翻譯)
15:04.520–15:06.100
that smaller and faster models
(此句尚無繁中翻譯)
15:06.100–15:06.880
in a family
(此句尚無繁中翻譯)
15:06.880–15:08.820
received a high designation
(此句尚無繁中翻譯)
15:08.820–15:11.000
in any tracked danger category.
(此句尚無繁中翻譯)
15:11.100–15:12.020
So this is super interesting.
(此句尚無繁中翻譯)
15:12.160–15:12.500
And then,
(此句尚無繁中翻譯)
15:12.600–15:13.100
not just that,
(此句尚無繁中翻譯)
15:13.140–15:14.280
but in the cybersecurity area,
(此句尚無繁中翻譯)
15:14.680–15:15.700
GPT 5.6 Sol
(此句尚無繁中翻譯)
15:15.700–15:16.820
saturated
(此句尚無繁中翻譯)
15:16.820–15:18.020
OpenAI's internal
(此句尚無繁中翻譯)
15:18.020–15:18.820
cyber challenge
(此句尚無繁中翻譯)
15:18.820–15:20.440
set at 96.7%,
(此句尚無繁中翻譯)
15:20.440–15:21.400
putting it above
(此句尚無繁中翻譯)
15:21.400–15:22.520
the high threshold.
(此句尚無繁中翻譯)
15:22.880–15:24.300
And external cyber testers
(此句尚無繁中翻譯)
15:24.300–15:25.400
found that high-impact
(此句尚無繁中翻譯)
15:25.400–15:26.000
zero days,
(此句尚無繁中翻譯)
15:26.320–15:27.240
including one where
(此句尚無繁中翻譯)
15:27.240–15:28.320
read-only users
(此句尚無繁中翻譯)
15:28.320–15:29.960
could modify and delete data
(此句尚無繁中翻譯)
15:29.960–15:31.680
in a widely deployed database.
(此句尚無繁中翻譯)
15:32.040–15:32.200
So,
(此句尚無繁中翻譯)
15:32.480–15:34.040
GPT 5.6 is clearly,
(此句尚無繁中翻譯)
15:34.200–15:34.480
you know,
(此句尚無繁中翻譯)
15:34.620–15:35.620
pretty dangerous
(此句尚無繁中翻譯)
15:35.620–15:36.880
when it comes to cybersecurity.
(此句尚無繁中翻譯)
15:37.200–15:37.320
Now,
(此句尚無繁中翻譯)
15:37.540–15:37.780
guys,
(此句尚無繁中翻譯)
15:37.780–15:39.080
I'm not trying to fear-monger
(此句尚無繁中翻譯)
15:39.080–15:39.760
so the US government
(此句尚無繁中翻譯)
15:39.760–15:40.520
can regulate the model.
(此句尚無繁中翻譯)
15:40.840–15:42.000
That stuff is already happening.
(此句尚無繁中翻譯)
15:42.320–15:43.780
I think what this is showing you
(此句尚無繁中翻譯)
15:43.780–15:44.760
is that these models
(此句尚無繁中翻譯)
15:44.760–15:45.520
are essentially
(此句尚無繁中翻譯)
15:45.520–15:46.800
passing the threshold
(此句尚無繁中翻譯)
15:46.800–15:47.860
for where they can now,
(此句尚無繁中翻譯)
15:48.060–15:48.420
you know,
(此句尚無繁中翻譯)
15:48.500–15:50.000
have those zero-day vulnerabilities
(此句尚無繁中翻譯)
15:50.000–15:51.440
that actually affect,
(此句尚無繁中翻譯)
15:51.600–15:52.660
maybe not critical infrastructure,
(此句尚無繁中翻譯)
15:52.780–15:53.260
but seriously,
(此句尚無繁中翻譯)
15:53.440–15:54.480
the economy and companies
(此句尚無繁中翻譯)
15:54.480–15:55.480
and that kind of stuff,
(此句尚無繁中翻譯)
15:55.500–15:56.240
which is pretty crazy.
(此句尚無繁中翻譯)
15:56.560–15:56.700
Now,
(此句尚無繁中翻譯)
15:56.900–15:57.720
if you're wondering about the bio,
(此句尚無繁中翻譯)
15:58.000–15:58.560
the bio result
(此句尚無繁中翻譯)
15:58.560–15:59.600
was just as revealing.
(此句尚無繁中翻譯)
15:59.980–16:00.500
High-threshold
(此句尚無繁中翻譯)
16:00.500–16:01.420
bio-evaluations
(此句尚無繁中翻譯)
16:01.420–16:02.220
crossed the line,
(此句尚無繁中翻譯)
16:02.520–16:03.500
while zero out of three
(此句尚無繁中翻譯)
16:03.500–16:04.700
critical bio-devaluations
(此句尚無繁中翻譯)
16:04.700–16:05.240
crossed it.
(此句尚無繁中翻譯)
16:05.240–16:06.680
On virology troubleshooting,
(此句尚無繁中翻譯)
16:06.820–16:08.660
GPT 5.6 scored 55%,
(此句尚無繁中翻譯)
16:08.660–16:09.220
far above
(此句尚無繁中翻譯)
16:09.220–16:11.360
the 31.0%
(此句尚無繁中翻譯)
16:11.360–16:12.620
expert performance threshold,
(此句尚無繁中翻譯)
16:12.920–16:13.580
and Secure Bio
(此句尚無繁中翻譯)
16:13.580–16:14.920
found that GPT 5.6
(此句尚無繁中翻譯)
16:14.920–16:15.680
reached new highs
(此句尚無繁中翻譯)
16:15.680–16:17.300
on several expert bio tests,
(此句尚無繁中翻譯)
16:17.560–16:19.540
including 68.4%
(此句尚無繁中翻譯)
16:19.540–16:21.020
on human pathogen capabilities
(此句尚無繁中翻譯)
16:21.020–16:22.400
and 68.3%
(此句尚無繁中翻譯)
16:22.400–16:23.640
on world-class bio.
(此句尚無繁中翻譯)
16:23.760–16:24.100
And the thing
(此句尚無繁中翻譯)
16:24.100–16:24.680
that I want you guys
(此句尚無繁中翻譯)
16:24.680–16:25.240
to understand here
(此句尚無繁中翻譯)
16:25.240–16:26.300
is that while these models
(此句尚無繁中翻譯)
16:26.300–16:27.540
seem like they're plateauing,
(此句尚無繁中翻譯)
16:27.600–16:27.820
because,
(此句尚無繁中翻譯)
16:28.120–16:28.480
of course,
(此句尚無繁中翻譯)
16:28.820–16:29.080
maybe,
(此句尚無繁中翻譯)
16:29.180–16:29.480
I don't know,
(此句尚無繁中翻譯)
16:29.580–16:30.300
maybe you might not be
(此句尚無繁中翻譯)
16:30.300–16:31.020
a bio-researcher,
(此句尚無繁中翻譯)
16:31.080–16:31.560
maybe you are.
(此句尚無繁中翻譯)
16:31.620–16:32.380
Unless you are in
(此句尚無繁中翻譯)
16:32.380–16:33.220
these specific fields
(此句尚無繁中翻譯)
16:33.220–16:33.980
like cybersecurity,
(此句尚無繁中翻譯)
16:34.540–16:34.700
you know,
(此句尚無繁中翻譯)
16:34.760–16:35.440
bio-research,
(此句尚無繁中翻譯)
16:35.700–16:36.600
you aren't really
(此句尚無繁中翻譯)
16:36.600–16:37.260
going to see
(此句尚無繁中翻譯)
16:37.260–16:38.140
those big changes,
(此句尚無繁中翻譯)
16:38.420–16:39.140
and these are where
(此句尚無繁中翻譯)
16:39.140–16:39.900
the big changes
(此句尚無繁中翻譯)
16:39.900–16:41.000
are actually occurring.
(此句尚無繁中翻譯)
16:41.260–16:41.480
So,
(此句尚無繁中翻譯)
16:41.600–16:42.360
it's really,
(此句尚無繁中翻譯)
16:42.660–16:43.420
really interesting
(此句尚無繁中翻譯)
16:43.420–16:43.900
to see
(此句尚無繁中翻譯)
16:43.900–16:45.020
just how crazy it is.
(此句尚無繁中翻譯)
16:45.260–16:45.820
And this is something
(此句尚無繁中翻譯)
16:45.820–16:46.360
that nobody,
(此句尚無繁中翻譯)
16:46.420–16:47.180
I didn't see anyone
(此句尚無繁中翻譯)
16:47.180–16:47.820
talking about this
(此句尚無繁中翻譯)
16:47.820–16:48.440
on the timeline.
(此句尚無繁中翻譯)
16:48.600–16:49.860
GPT 5.6 Sol's
(此句尚無繁中翻譯)
16:49.860–16:51.100
unsettling agent behavior.
(此句尚無繁中翻譯)
16:51.240–16:51.380
So,
(此句尚無繁中翻譯)
16:51.480–16:52.100
the agent behavior
(此句尚無繁中翻譯)
16:52.100–16:52.640
is arguably
(此句尚無繁中翻譯)
16:52.640–16:53.600
the most unsettling.
(此句尚無繁中翻譯)
16:53.720–16:54.680
GPT 5.6 Sol
(此句尚無繁中翻譯)
16:54.680–16:55.380
more often
(此句尚無繁中翻譯)
16:55.380–16:56.700
goes beyond user intent
(此句尚無繁中翻譯)
16:56.700–16:57.280
when coding,
(此句尚無繁中翻譯)
16:57.660–16:58.460
including deleting
(此句尚無繁中翻譯)
16:58.460–16:59.900
the wrong virtual machines,
(此句尚無繁中翻譯)
17:00.020–17:00.800
claiming unfinished
(此句尚無繁中翻譯)
17:00.800–17:01.740
research is verified,
(此句尚無繁中翻譯)
17:01.740–17:02.560
and moving
(此句尚無繁中翻譯)
17:02.560–17:03.540
cached credentials
(此句尚無繁中翻譯)
17:03.540–17:04.660
without permission.
(此句尚無繁中翻譯)
17:05.060–17:05.680
And Mita found
(此句尚無繁中翻譯)
17:05.680–17:06.900
that GPT 5.6 Sol
(此句尚無繁中翻譯)
17:06.900–17:07.980
sometimes tried
(此句尚無繁中翻譯)
17:07.980–17:08.900
to game the test
(此句尚無繁中翻譯)
17:08.900–17:09.500
instead of just
(此句尚無繁中翻譯)
17:09.500–17:10.140
doing the task.
(此句尚無繁中翻譯)
17:10.220–17:10.960
The bench row result
(此句尚無繁中翻譯)
17:10.960–17:11.880
couldn't be treated
(此句尚無繁中翻譯)
17:11.880–17:13.060
as a clean result
(此句尚無繁中翻譯)
17:13.060–17:14.000
of raw capability.
(此句尚無繁中翻譯)
17:14.200–17:14.520
They're basically
(此句尚無繁中翻譯)
17:14.520–17:15.000
trying to say that,
(此句尚無繁中翻譯)
17:15.100–17:15.160
look,
(此句尚無繁中翻譯)
17:15.220–17:16.380
this model was just
(此句尚無繁中翻譯)
17:16.380–17:17.480
acting very strange
(此句尚無繁中翻譯)
17:17.480–17:18.680
and we don't even know
(此句尚無繁中翻譯)
17:18.680–17:19.340
if we can trust
(此句尚無繁中翻譯)
17:19.340–17:20.000
the model's results
(此句尚無繁中翻譯)
17:20.000–17:20.660
because it didn't even
(此句尚無繁中翻譯)
17:20.660–17:21.740
want to participate properly.
(此句尚無繁中翻譯)
17:21.900–17:22.220
And so,
(此句尚無繁中翻譯)
17:22.300–17:23.140
what we have here
(此句尚無繁中翻譯)
17:23.140–17:23.820
is a situation
(此句尚無繁中翻譯)
17:23.820–17:24.620
where nobody's
(此句尚無繁中翻譯)
17:24.620–17:25.140
really talking about
(此句尚無繁中翻譯)
17:25.140–17:25.540
the fact that
(此句尚無繁中翻譯)
17:25.540–17:26.000
these models
(此句尚無繁中翻譯)
17:26.000–17:26.840
aren't even behaving.
(此句尚無繁中翻譯)
17:27.280–17:27.760
And I find that
(此句尚無繁中翻譯)
17:27.760–17:28.180
super,
(此句尚無繁中翻譯)
17:28.380–17:28.960
super interesting
(此句尚無繁中翻譯)
17:28.960–17:30.120
because you have
(此句尚無繁中翻譯)
17:30.120–17:30.560
to understand
(此句尚無繁中翻譯)
17:30.560–17:31.180
that these models
(此句尚無繁中翻譯)
17:31.180–17:32.620
are really black boxes.
(此句尚無繁中翻譯)
17:32.840–17:32.980
So,
(此句尚無繁中翻譯)
17:33.220–17:33.500
I mean,
(此句尚無繁中翻譯)
17:33.740–17:34.060
honestly,
(此句尚無繁中翻譯)
17:34.240–17:34.740
things are starting
(此句尚無繁中翻譯)
17:34.740–17:35.360
to get into
(此句尚無繁中翻譯)
17:35.360–17:36.040
the weird zone.
(此句尚無繁中翻譯)
17:36.100–17:36.480
But let me know
(此句尚無繁中翻譯)
17:36.480–17:37.100
what you guys think
(此句尚無繁中翻譯)
17:37.100–17:37.760
about this model.
(此句尚無繁中翻譯)
17:37.920–17:38.200
And so,
(此句尚無繁中翻譯)
17:38.260–17:38.640
it's going to be
(此句尚無繁中翻譯)
17:38.640–17:39.200
really interesting
(此句尚無繁中翻譯)
17:39.200–17:40.660
to see where things
(此句尚無繁中翻譯)
17:40.660–17:41.560
go from the feature
(此句尚無繁中翻譯)
17:41.560–17:41.940
from here.
(此句尚無繁中翻譯)