0:00.000–0:01.710
When sharing this paper,
0:01.710–0:07.573
Google's Subham Sahu wrote that for an agent to achieve recursive self-improvement,
0:07.573–0:09.120
it must learn to dream.
0:09.640–0:11.707
History is the event of its dreaming.
0:12.220–0:15.007
Well, this sounds like science fiction, but
0:15.007–0:20.280
what this paper actually does is more practical than the term dreaming might imply.
0:20.800–0:22.180
Why do humans dream?
0:22.800–0:25.800
Because what happened during the day is already over,
0:26.220–0:30.320
but at night, the brain replays and recombines those experiences.
0:30.940–0:33.740
We don't need to actually relive the same day,
0:34.140–0:37.273
but we might discover new connections in these replays,
0:37.273–0:40.007
or even change how we make decisions the next day.
0:40.007–0:42.022
Now, a research team from Google and
0:42.022–0:45.880
Google DeepMind has brought a very similar idea into AI.
0:46.240–0:48.056
The paper is called Dream RSI,
0:48.056–0:51.760
Recursive Self-Improvement Based on Evolutionary Worlds.
0:51.760–0:55.826
The system it proposes explores a real environment during the day,
0:55.826–0:59.020
writing actual code, compiling, and running tests.
0:59.460–1:00.660
But in the next round,
1:00.660–1:03.460
it doesn't repeat all those expensive processes.
1:03.460–1:08.100
Instead, it turns past explorations into a dream world that can be revisited.
1:08.300–1:12.690
In this world, it can ask, what if I hadn't kept digging down this path,
1:12.690–1:14.960
but had changed direction earlier?
1:15.160–1:19.160
What if, instead of running four processes at once, I ran eight?
1:19.160–1:22.760
If there has been no improvement for several consecutive steps,
1:22.760–1:24.427
should I have stopped sooner?
1:24.427–1:27.840
It doesn't regenerate the code or rerun all the experiments.
1:28.560–1:30.760
Many results have already happened and
1:30.760–1:33.227
are embedded in the history of exploration.
1:33.320–1:37.152
What the AI does is try different exploration methods over and
1:37.152–1:38.920
over within that past world.
1:39.620–1:43.960
Then it brings the better strategies found in the dream back to reality.
1:44.280–1:47.080
New strategies produce new exploration history.
1:47.080–1:49.080
New history becomes new dreams, and
1:49.080–1:52.480
new dreams modify the exploration method for the next round.
1:52.480–1:54.940
This is why the authors call it RSI.
1:57.203–2:02.028
But the really interesting question here isn't the dreaming metaphor itself,
2:02.028–2:04.733
but what actually changes after it wakes up?
2:07.026–2:12.179
And this is exactly where Dream RSI sets itself apart from AlphaCode,
2:12.179–2:14.756
Godel Machines, and AI Scientist.
2:15.196–2:16.407
In the last episode,
2:16.407–2:19.896
we asked what counts as true recursive self-improvement,
2:20.196–2:21.416
and in this episode,
2:21.416–2:24.716
a real system has emerged that we can actually examine.
2:24.956–2:27.889
All right, let's put the dreams aside for a moment and
2:27.889–2:30.023
look at what it's doing during the day.
2:30.023–2:34.356
Many automated discovery systems already look like they are self-evolving,
2:34.416–2:37.949
Like a coding agent that keeps generating candidate programs,
2:38.276–2:40.056
an evaluator that scores them,
2:40.336–2:44.269
and then the system keeping the better results to continue iterating.
2:44.269–2:48.069
But there are actually two completely different jobs mixed up here.
2:48.416–2:50.936
Imagine a research team with two people.
2:51.436–2:54.525
One performs the experiments, writes the code,
2:54.525–2:56.456
runs tests, and gets results,
2:56.456–2:59.789
while the other doesn't necessarily get their hands dirty.
3:00.316–3:03.116
They are responsible for deciding what to do next,
3:03.116–3:05.316
such as which path is worth continuing,
3:05.316–3:08.376
which should be stopped, and where resources should
3:08.376–3:13.976
be invested. Systems like AlphaEvo make it very easy to get the feeling that as the program gets
3:13.976–3:16.247
stronger generation by generation,
3:16.247–3:20.056
it seems like the AI is already improving itself. But if we zoom
3:20.056–3:22.246
out a little, we see a problem. While
3:22.246–3:25.896
the underlying code can change and which results to keep can
3:25.896–3:29.939
change, the second type of work, how to explore the next steps,
3:29.939–3:32.116
is often still pre-designed. For
3:32.116–3:34.758
For example, from which node to continue,
3:34.758–3:37.476
how many branches to open simultaneously,
3:37.476–3:40.676
and after how many attempts to abandon a direction.
3:40.676–3:43.809
These rules usually don't evolve along with the system.
3:43.809–3:45.496
What about Dream RSI?
3:45.496–3:50.736
It specifically targets the person responsible for deciding what to do next.
3:50.736–3:53.306
Dream RSI separates this layer, and
3:53.306–3:57.676
its real-world exploration continuously grows a discovery
3:57.676–3:58.676
tree.
3:58.676–4:03.516
Every node on the tree corresponds to a real attempt that has taken place, such as what
4:03.516–4:06.116
solution the model generated in what context,
4:06.116–4:08.583
whether the code could run, and what results
4:08.583–4:10.276
the evaluator gave.
4:10.276–4:11.943
All of this has been recorded.
4:11.943–4:16.896
The outer exploration strategy only needs to answer a few very simple questions, like
4:16.896–4:19.096
from which node should we continue now?
4:19.096–4:21.116
Should we start a new path?
4:21.116–4:24.116
How many directions should we pursue in parallel?
4:24.116–4:25.249
When should we stop?
4:25.249–4:26.516
Let's look at figure 2,
4:26.516–4:30.516
which is the most important figure for understanding the entire paper.
4:30.516–4:35.249
You can think of this tree in the middle as the maze that the AI actually walked through
4:35.249–4:35.916
yesterday.
4:35.916–4:38.983
Every node that is lit up is a place it actually visited,
4:38.983–4:40.916
and the results are already known.
4:40.916–4:42.164
So the question is,
4:42.164–4:46.976
if I want to compare two new exploration strategies today, must I make
4:46.976–4:50.116
the AI walk through the maze twice from the beginning?
4:50.116–4:53.416
In the past, that was basically the only way.
4:53.416–4:58.136
Because you could only know if a new strategy was good by actually running it again.
4:58.136–5:01.736
But Dream RSI's answer is, not necessarily.
5:01.736–5:04.536
Since it has already walked through it yesterday,
5:04.536–5:07.069
and what happened at each node has been saved,
5:07.069–5:11.196
a new strategy can completely walk through this historical tree first.
5:11.196–5:14.422
It might go left first, it might turn right sooner,
5:14.422–5:16.496
or it might just take three steps
5:16.496–5:17.696
and stop.
5:17.696–5:20.730
As long as he visits nodes that have truly occurred,
5:20.730–5:22.776
the simulator can directly reveal
5:22.776–5:25.916
the actual results from that time.
5:25.916–5:29.271
The key shift here isn't just remembering history,
5:29.271–5:31.456
but rather turning history from a
5:31.456–5:34.496
log into a re-enterable event.
5:34.496–5:37.836
Now, there is a common misunderstanding here.
5:37.836–5:41.385
Dream RSI does not train a new world model first and
5:41.385–5:43.836
then have it approximate reality.
5:43.836–5:49.176
For Dream RSI, the history that has already played out serves as the playback simulator
5:49.176–5:50.176
itself.
5:50.176–5:52.509
Thus, its dreams are quite conservative.
5:52.509–5:55.376
It isn't predicting events that haven't happened,
5:55.376–5:57.976
but re-entering events that truly took place.
5:57.976–6:01.356
Strictly speaking, this world has a clear boundary.
6:01.356–6:04.736
It can only dream of places it has actually experienced.
6:04.736–6:08.936
It cannot know the results of unexplored branches out of thin air.
6:08.936–6:12.496
This is precisely both its advantage and its limitation.
6:12.496–6:17.363
The advantage is that real executions that have already occurred don't require paying
6:17.363–6:19.696
for expensive online exploration again,
6:19.696–6:23.429
supporting a large volume of rapid offline strategy comparisons.
6:23.429–6:27.296
The limitation is that if the first round of real exploration is poor,
6:27.296–6:28.896
with almost no high-quality
6:28.896–6:30.363
paths in the history tree,
6:30.363–6:34.096
the subsequent dreams won't conjure up a good event out of nothing.
6:34.096–6:35.943
So the way it truly loops is this.
6:35.943–6:40.316
Reality provides a portion of the world. Dreams in this
6:40.316–6:43.362
world find better exploration strategies, and
6:43.362–6:46.256
new strategies return to reality to traverse
6:46.256–6:50.196
new nodes, which then expand the next round of dreams.
6:50.196–6:53.936
This is the evolving world mentioned in the paper's title.
6:53.936–6:57.136
The world isn't built once and left unchanged.
6:57.136–7:00.345
It expands continuously alongside real exploration,
7:00.345–7:02.856
and the exploration strategy evolves with
7:02.856–7:03.856
the world.
7:03.856–7:10.076
This is much more interesting than simply summarizing historical experience into a few sentences.
7:10.076–7:14.476
The authors actually tested another, seemingly simpler method.
7:14.476–7:17.876
Since we already know which directions succeeded or failed,
7:17.876–7:19.676
why not let a model summarize the
7:19.676–7:23.543
lessons and inject these suggestions into the next round of prompts?
7:24.611–7:26.841
However, the results weren't good.
7:27.461–7:28.781
Let's look at Figure 5.
7:29.281–7:32.745
After adding the guidance, performance actually worsened,
7:32.745–7:35.461
regardless of whether RSI was originally used.
7:35.701–7:39.364
This is worth pondering, because remembering the past and
7:39.364–7:41.981
utilizing the past are not the same thing.
7:42.141–7:50.301
Compressing history into a single best thing to do is equivalent to imposing a predetermined shared direction on all subsequent exploration.
7:50.301–7:52.438
It could be helpful, but it might also
7:52.438–7:54.644
constrain the search space too early,
7:54.644–7:58.781
excluding many unconventional paths that should have been preserved.
7:59.061–8:00.394
Dream RSI is different.
8:00.394–8:04.128
It doesn't tell the AI up front which experience is most important.
8:04.201–8:06.001
It simply preserves that world,
8:06.001–8:09.468
allowing different strategies to explore it for themselves.
8:09.468–8:14.534
I think the most interesting takeaway from this paper isn't that AI can learn from history,
8:14.921–8:21.361
but rather that perhaps truly valuable experience isn't about summarizing the past into a single instruction,
8:21.841–8:25.041
but about preserving a past that can be re-experienced.
8:25.201–8:28.131
If Dream RSI only ends up saving a few API calls,
8:28.131–8:30.761
then it's not actually that interesting.
8:31.301–8:36.768
What's truly worth looking at is whether it changes the actual exploration process that follows.
8:36.768–8:40.434
The authors conducted experiments in algorithmic engineering,
8:40.434–8:42.568
mathematical optimization, and GPU.
8:42.568–8:43.768
Kernel engineering.
8:43.768–8:45.434
There are a lot of numbers, but
8:45.434–8:47.701
I think only a few are worth remembering.
8:47.701–8:48.901
The first set is cost.
8:48.901–8:51.168
In the Lowell algorithm discovery task,
8:51.168–8:55.921
one of the most striking figures in the paper is a maximum call gap of about 162 times.
8:56.181–8:59.048
However, this figure cannot be taken in isolation.
8:59.141–9:04.121
Dream RSI uses Gemini 1.5 Flash, while Simpletext uses GPT-40.
9:04.121–9:08.388
A much cleaner comparison is the head-to-head control between Dream RSI and
9:08.388–9:11.521
recursive fixed exploration using the same base model.
9:11.561–9:14.494
However, I am more concerned with another question.
9:14.561–9:17.881
If we search so much less, will the results get worse?
9:18.041–9:21.161
The paper does not show this kind of simple trade-off.
9:21.541–9:25.808
On some tasks, it achieves comparable performance with fewer generations,
9:25.861–9:26.794
while on others,
9:26.794–9:30.261
it achieves even higher performance within a similar budget.
9:30.261–9:34.941
Curil BCH is a great example to illustrate these two different types of gains.
9:34.941–9:40.381
VGG16 and LayerNorm show a better sense of when enough is enough,
9:40.381–9:44.314
essentially achieving similar performance with fewer generations.
9:45.360–9:48.480
Meanwhile, Conv and CX show a better sense of when
9:48.480–9:53.350
not to stop, meaning they push performance further within a similar budget.
9:54.130–9:58.810
This leads to what I consider the most elegant behavioral chart in the entire paper.
9:59.050–10:00.983
Let's look at this chart together.
10:01.270–10:09.330
The strategy learned by Dream RSI isn't a simple rule like always do less or always branch out more.
10:09.330–10:12.295
When performance improvements are steady early on,
10:12.295–10:14.250
it reduces the number of attempts.
10:14.250–10:19.510
But when performance hits a plateau, it expands its exploration again.
10:19.510–10:23.810
In other words, it has learned a dynamic exploration rhythm.
10:23.810–10:26.410
Don't waste your budget casting a wide net when
10:26.410–10:28.410
things are going well, but don't get
10:28.410–10:30.677
stuck on the same path when you hit a wall.
10:30.677–10:34.222
This is more important than the 162 token count because
10:34.222–10:36.710
it shows that RSI ultimately changes
10:36.710–10:39.741
not just how much compute a single task uses, but
10:39.741–10:42.090
how compute is allocated for the next
10:42.090–10:43.490
exploration.
10:43.490–10:48.157
This is why the authors place it within the context of recursive self-improvement.
10:48.157–10:51.490
However, we must clearly define the boundaries.
10:51.490–10:54.223
The underlying coding agent remains unchanged,
10:54.223–10:56.557
the model remains the same, the evaluator
10:56.557–10:59.610
is untouched, and the task objectives are fixed.
10:59.610–11:04.870
This RSI only modifies the outer strategy for how to continue seeking better results.
11:04.870–11:10.450
Therefore, if we interpret this as Google's AI starting to rewrite its own brain, that
11:10.450–11:12.530
would be a complete exaggeration.
11:12.530–11:15.954
Conversely, if we just view it as a cost-saving scheduler,
11:15.954–11:19.098
that underestimates what this paper truly achieves,
11:19.098–11:23.710
as the scheduling rules themselves have now become subjects for improvement.
11:24.010–11:28.170
In fact, someone on X has already asked the most critical question.
11:28.770–11:30.650
Would you call this true RSI?
11:30.930–11:34.910
This is also the question I wanted to ask most after reading this paper.
11:34.910–11:42.238
Since we spent our entire last episode trying to answer what counts as true recursive self-improvement,
11:42.238–11:45.819
we can bring back the L1 to L5 chart from last time and
11:45.819–11:48.650
use it as a yardstick to measure this RSI.
11:49.150–11:54.217
One of the most important viewpoints from that paper was that performance improvement and
11:54.217–11:57.150
recursive self-improvement are not the same thing.
11:57.150–12:00.151
A system can perform tasks increasingly well, but
12:00.151–12:04.392
if the method of self-improvement is hard-coded from the outside,
12:04.392–12:07.270
it is still one step away from true recursion.
12:07.450–12:09.357
Taking AlphaDev as an example,
12:09.357–12:14.930
its most intuitive ability is to continuously search for better programs and algorithms.
12:15.330–12:18.310
The main object of improvement is the result itself.
12:18.830–12:21.497
You get better at finding better answers, while
12:21.497–12:24.030
Darwin-inspired agents take a step further.
12:24.030–12:25.630
It doesn't just change code.
12:25.630–12:29.630
It allows the implementation of the coding agent itself to be modified.
12:29.630–12:32.830
So the object of improvement is no longer just the answer,
12:32.830–12:35.097
but the agent that generates the answer.
12:35.097–12:37.430
However, in that RSI paper from last time,
12:37.430–12:40.563
rules regarding which candidate solutions are kept and
12:40.563–12:47.310
which candidate solutions the next generation evolves from were still not part of the self-modification loop.
12:48.030–12:50.290
In other words, the agent can change,
12:50.610–12:55.010
but the rules determining where the next generation of agents should come from
12:55.010–12:57.010
have not yet entered the recursion.
12:57.010–12:59.950
Dream RSI happens to tread exactly on this boundary.
13:00.230–13:03.497
What Dream RSI leaves outside the self-modification loop
13:03.497–13:07.897
is precisely the mechanism of candidate solution maintenance and selection,
13:07.897–13:11.030
essentially where the next generation continues from.
13:11.030–13:13.763
What Dream RSI does is precisely turn this layer,
13:13.763–13:17.163
which nodes continue to branch, how many branches to create,
13:17.163–13:18.097
and when to stop,
13:18.097–13:20.830
into an object that can be replaced and improved.
13:20.830–13:23.697
It didn't modify the underlying agent, but instead
13:23.697–13:25.097
changed the outer layer,
13:25.097–13:28.430
that is, how the next round of improvement should be sought,
13:28.430–13:31.897
how many branches to continue opening from historical nodes,
13:31.897–13:32.830
and when to stop.
13:32.830–13:36.630
These elements, which originally belonged to the system designer,
13:36.630–13:39.563
have been transformed into an exploration strategy
13:39.563–13:41.897
that can be evaluated, modified, and then
13:41.897–13:43.630
carried over to the next round.
13:43.630–13:47.030
The key is that this new strategy doesn't just stay in a dream.
13:47.030–13:50.363
It directly governs the real exploration of the next round.
13:50.363–13:54.097
With a fixed strategy, each round can still continue to explore and
13:54.097–13:54.963
grow new nodes.
13:54.963–13:59.897
The real difference is that Dream RSI lets the new strategy learned in the previous round
13:59.897–14:03.790
decide where the tree grows next and how resources are allocated.
14:04.010–14:08.070
And the new exploration history feeds back into the replay simulator,
14:08.490–14:11.623
which in turn modifies the next version of the strategy.
14:11.623–14:16.366
So, if we look at it through the L1-L5 framework from that paper last time,
14:16.366–14:17.590
Dream RSI already
14:17.590–14:20.670
exhibits a very clear L5 characteristic.
14:20.670–14:25.337
The object being modified now includes the method used to generate the next round of
14:25.337–14:26.603
improvements itself.
14:26.603–14:28.950
But I must add a very important caveat.
14:28.950–14:31.330
It is an extremely narrow L5.
14:31.330–14:34.526
It does not autonomously modify the underlying model,
14:34.526–14:36.610
change the evaluator, redefine the
14:36.610–14:40.250
task, or decide which capability I should improve next.
14:40.250–14:46.050
It performs recursive improvement on the exploration mechanism within the boundaries of fixed goals,
14:46.050–14:49.117
fixed evaluation rules, and a fixed underlying agent.
14:49.117–14:50.250
In the last episode,
14:50.250–14:53.583
we actually distinguished between two types of recursion.
14:53.583–14:54.983
The first is very simple.
14:54.983–14:58.783
Is the mechanism you modified actually used again in the next round?
14:58.783–15:00.917
This is called structural recursion.
15:00.917–15:02.983
Dream RSI is very clear on this point.
15:02.983–15:07.717
The new exploration strategy enters the next round of online exploration directly.
15:07.717–15:09.783
But the harder one is the second type.
15:09.783–15:13.717
does using it actually make the successor system better at improving?
15:13.717–15:15.383
This is effective recursion.
15:15.383–15:18.117
Dream RSI provides a fairly clean contrast here.
15:18.117–15:21.183
It uses the same underlying model, the same evaluator,
15:21.183–15:24.850
and the same initial strategy as the fixed exploration baseline.
15:25.581–15:31.801
by whether the subsequent exploration strategy has been improved through dreaming. So Dream RSI's
15:31.801–15:34.698
L5 has two dimensions. It is very narrow in scope,
15:34.698–15:37.241
but in terms of the clarity of its recursive
15:37.241–15:39.533
evidence, it is not weak. Therefore,
15:39.533–15:42.861
a more accurate statement is not that true general RSI
15:42.861–15:46.808
has arrived, but rather that we have finally seen a very clean,
15:46.808–15:49.161
local example where the improvement
15:49.161–15:54.801
This also makes its relationship with AV training particularly interesting.
15:54.801–15:57.091
particularly interesting. Last time,
15:57.091–16:00.421
we discussed how AV training precipitates the results of
16:00.421–16:04.754
post-training experiments into a set of persistent research strategies and
16:04.754–16:05.821
exploration logs,
16:05.821–16:10.561
which are then modified by the meta-agent to guide the subsequent executive agents.
16:10.941–16:14.181
Doesn't that sound already very close to Dream RSI?
16:14.541–16:18.874
Neither is satisfied with just modifying the final model or the final answer.
16:18.874–16:23.741
They have begun to make the research process itself something that can be accumulated,
16:23.741–16:25.208
modified, and inherited.
16:25.208–16:28.361
However, the way both utilize history is not the same.
16:28.581–16:34.221
AVIL training is more like letting a meta-agent read past evidence to directly diagnose problems
16:34.221–16:37.673
and then modify the policy. But it adds an extra step.
16:37.673–16:40.341
Instead of just inferring a new strategy
16:40.341–16:44.773
based on history, it generates multiple candidate exploration strategies,
16:44.773–16:46.341
puts them into a historical
16:46.341–16:48.505
replay world to actually run them, and
16:48.505–16:51.481
uses the results to choose which one enters reality.
16:51.701–16:55.483
See, this is more like modifying a plan after a post-mortem review,
16:55.483–16:56.901
whereas the other is more
16:56.901–17:00.701
like putting multiple new plans back into a sandbox for simulation.
17:00.701–17:01.834
This brings us right
17:01.834–17:02.901
back to the figure.
17:02.901–17:06.501
The authors actually specifically tried the more direct path of
17:06.501–17:09.301
summarizing history into semantic guidance, but
17:09.301–17:12.234
the results were worse. Therefore, the most special
17:12.234–17:15.301
thing about JRSI is not just that it preserves history,
17:15.301–17:17.434
but that it lets candidate strategies
17:17.434–17:19.234
re-experience history and then
17:19.234–17:22.768
uses the outcomes of that experience to make a choice. Perhaps a
17:22.768–17:28.101
truly powerful self-improving system in the future will not choose just one of these, but might
17:28.101–17:32.101
simultaneously distill experience into high-level strategies while
17:32.101–17:33.834
retaining a sufficiently rich
17:33.834–17:35.368
set of historical events so
17:35.368–17:38.701
these strategies can be retested at any time. At that point,
17:38.701–17:42.634
memory will no longer just be the model knowing what it has done before,
17:42.634–17:45.301
but will start to become a past that can be rerun.
17:45.301–17:48.761
Speaking of which, there is one final question.
17:49.461–17:52.530
Why did the authors choose to call this dream?
17:52.530–17:55.841
As we mentioned earlier, Dream RSI does not train
17:55.841–17:58.443
a world model in the traditional sense, but
17:58.443–18:01.841
it borrows the same intuition. Real-world interaction
18:01.841–18:03.037
is too expensive, so
18:03.037–18:07.961
can we turn past experiences into an internal environment for continued learning?
18:09.001–18:11.476
Dream RSI borrows this line of thinking, but
18:11.476–18:14.821
the object it simulates is different. It does not attempt to
18:14.821–18:17.354
learn a complete external world, but instead
18:17.354–18:20.021
preserves how it explored this world in the past
18:20.021–18:22.621
and what happened during each exploration. So
18:22.621–18:25.288
I think the truly interesting part of this paper
18:25.288–18:28.321
is that the world here has quietly changed its target.
18:28.561–18:30.494
In the past, we were more concerned
18:30.494–18:32.294
with how the world would change.
18:32.294–18:34.181
But Dream RSI starts to care about
18:34.181–18:36.781
how I personally sought out changes in the past
18:36.781–18:40.114
and what would happen if I used a different way of searching.
18:40.114–18:42.941
Of course, today's Dream RSI is still very far
18:42.941–18:44.741
from being a superintelligence
18:44.741–18:46.941
that can improve itself indefinitely.
18:46.941–18:49.941
Its model hasn't changed, its goals haven't changed,
18:49.941–18:52.874
and the rules evaluating it haven't changed either.
18:52.874–18:57.074
Even the dreams it can have are limited to the past it has truly experienced.
18:57.761–19:02.361
But something that used to be set in stone has started to shift.
19:03.021–19:05.554
It's not about what the next answer should be,
19:05.741–19:09.261
but rather how I should find a way to improve myself next time.
19:09.261–19:11.375
When the answer to this question also
19:11.375–19:14.581
starts to be modified and verified by the machine itself
19:14.581–19:16.114
to be used in the next round,
19:16.341–19:20.721
the most noteworthy part of recursive self-improvement may have just begun.
19:20.721–19:24.002
Alright, that's all for today's content. Please like,
19:24.002–19:26.141
share, and subscribe to my channel.
19:26.301–19:28.301
If you'd like to support me further,
19:28.301–19:31.501
feel free to join my membership. The cost of a cup of coffee
19:31.501–19:33.768
each month not only supports me, but also
19:33.768–19:36.768
helps me continue creating more content related to AI
19:36.768–19:39.000
and technology. Alright, thanks for watching, and
0:00.000–0:01.710
When sharing this paper,
0:01.710–0:07.573
Google's Subham Sahu wrote that for an agent to achieve recursive self-improvement,
0:07.573–0:09.120
it must learn to dream.
0:09.640–0:11.707
History is the event of its dreaming.
0:12.220–0:15.007
Well, this sounds like science fiction, but
0:15.007–0:20.280
what this paper actually does is more practical than the term dreaming might imply.
0:20.800–0:22.180
Why do humans dream?
0:22.800–0:25.800
Because what happened during the day is already over,
0:26.220–0:30.320
but at night, the brain replays and recombines those experiences.
0:30.940–0:33.740
We don't need to actually relive the same day,
0:34.140–0:37.273
but we might discover new connections in these replays,
0:37.273–0:40.007
or even change how we make decisions the next day.
0:40.007–0:42.022
Now, a research team from Google and
0:42.022–0:45.880
Google DeepMind has brought a very similar idea into AI.
0:46.240–0:48.056
The paper is called Dream RSI,
0:48.056–0:51.760
Recursive Self-Improvement Based on Evolutionary Worlds.
0:51.760–0:55.826
The system it proposes explores a real environment during the day,
0:55.826–0:59.020
writing actual code, compiling, and running tests.
0:59.460–1:00.660
But in the next round,
1:00.660–1:03.460
it doesn't repeat all those expensive processes.
1:03.460–1:08.100
Instead, it turns past explorations into a dream world that can be revisited.
1:08.300–1:12.690
In this world, it can ask, what if I hadn't kept digging down this path,
1:12.690–1:14.960
but had changed direction earlier?
1:15.160–1:19.160
What if, instead of running four processes at once, I ran eight?
1:19.160–1:22.760
If there has been no improvement for several consecutive steps,
1:22.760–1:24.427
should I have stopped sooner?
1:24.427–1:27.840
It doesn't regenerate the code or rerun all the experiments.
1:28.560–1:30.760
Many results have already happened and
1:30.760–1:33.227
are embedded in the history of exploration.
1:33.320–1:37.152
What the AI does is try different exploration methods over and
1:37.152–1:38.920
over within that past world.
1:39.620–1:43.960
Then it brings the better strategies found in the dream back to reality.
1:44.280–1:47.080
New strategies produce new exploration history.
1:47.080–1:49.080
New history becomes new dreams, and
1:49.080–1:52.480
new dreams modify the exploration method for the next round.
1:52.480–1:54.940
This is why the authors call it RSI.
1:57.203–2:02.028
But the really interesting question here isn't the dreaming metaphor itself,
2:02.028–2:04.733
but what actually changes after it wakes up?
2:07.026–2:12.179
And this is exactly where Dream RSI sets itself apart from AlphaCode,
2:12.179–2:14.756
Godel Machines, and AI Scientist.
2:15.196–2:16.407
In the last episode,
2:16.407–2:19.896
we asked what counts as true recursive self-improvement,
2:20.196–2:21.416
and in this episode,
2:21.416–2:24.716
a real system has emerged that we can actually examine.
2:24.956–2:27.889
All right, let's put the dreams aside for a moment and
2:27.889–2:30.023
look at what it's doing during the day.
2:30.023–2:34.356
Many automated discovery systems already look like they are self-evolving,
2:34.416–2:37.949
Like a coding agent that keeps generating candidate programs,
2:38.276–2:40.056
an evaluator that scores them,
2:40.336–2:44.269
and then the system keeping the better results to continue iterating.
2:44.269–2:48.069
But there are actually two completely different jobs mixed up here.
2:48.416–2:50.936
Imagine a research team with two people.
2:51.436–2:54.525
One performs the experiments, writes the code,
2:54.525–2:56.456
runs tests, and gets results,
2:56.456–2:59.789
while the other doesn't necessarily get their hands dirty.
3:00.316–3:03.116
They are responsible for deciding what to do next,
3:03.116–3:05.316
such as which path is worth continuing,
3:05.316–3:08.376
which should be stopped, and where resources should
3:08.376–3:13.976
be invested. Systems like AlphaEvo make it very easy to get the feeling that as the program gets
3:13.976–3:16.247
stronger generation by generation,
3:16.247–3:20.056
it seems like the AI is already improving itself. But if we zoom
3:20.056–3:22.246
out a little, we see a problem. While
3:22.246–3:25.896
the underlying code can change and which results to keep can
3:25.896–3:29.939
change, the second type of work, how to explore the next steps,
3:29.939–3:32.116
is often still pre-designed. For
3:32.116–3:34.758
For example, from which node to continue,
3:34.758–3:37.476
how many branches to open simultaneously,
3:37.476–3:40.676
and after how many attempts to abandon a direction.
3:40.676–3:43.809
These rules usually don't evolve along with the system.
3:43.809–3:45.496
What about Dream RSI?
3:45.496–3:50.736
It specifically targets the person responsible for deciding what to do next.
3:50.736–3:53.306
Dream RSI separates this layer, and
3:53.306–3:57.676
its real-world exploration continuously grows a discovery
3:57.676–3:58.676
tree.
3:58.676–4:03.516
Every node on the tree corresponds to a real attempt that has taken place, such as what
4:03.516–4:06.116
solution the model generated in what context,
4:06.116–4:08.583
whether the code could run, and what results
4:08.583–4:10.276
the evaluator gave.
4:10.276–4:11.943
All of this has been recorded.
4:11.943–4:16.896
The outer exploration strategy only needs to answer a few very simple questions, like
4:16.896–4:19.096
from which node should we continue now?
4:19.096–4:21.116
Should we start a new path?
4:21.116–4:24.116
How many directions should we pursue in parallel?
4:24.116–4:25.249
When should we stop?
4:25.249–4:26.516
Let's look at figure 2,
4:26.516–4:30.516
which is the most important figure for understanding the entire paper.
4:30.516–4:35.249
You can think of this tree in the middle as the maze that the AI actually walked through
4:35.249–4:35.916
yesterday.
4:35.916–4:38.983
Every node that is lit up is a place it actually visited,
4:38.983–4:40.916
and the results are already known.
4:40.916–4:42.164
So the question is,
4:42.164–4:46.976
if I want to compare two new exploration strategies today, must I make
4:46.976–4:50.116
the AI walk through the maze twice from the beginning?
4:50.116–4:53.416
In the past, that was basically the only way.
4:53.416–4:58.136
Because you could only know if a new strategy was good by actually running it again.
4:58.136–5:01.736
But Dream RSI's answer is, not necessarily.
5:01.736–5:04.536
Since it has already walked through it yesterday,
5:04.536–5:07.069
and what happened at each node has been saved,
5:07.069–5:11.196
a new strategy can completely walk through this historical tree first.
5:11.196–5:14.422
It might go left first, it might turn right sooner,
5:14.422–5:16.496
or it might just take three steps
5:16.496–5:17.696
and stop.
5:17.696–5:20.730
As long as he visits nodes that have truly occurred,
5:20.730–5:22.776
the simulator can directly reveal
5:22.776–5:25.916
the actual results from that time.
5:25.916–5:29.271
The key shift here isn't just remembering history,
5:29.271–5:31.456
but rather turning history from a
5:31.456–5:34.496
log into a re-enterable event.
5:34.496–5:37.836
Now, there is a common misunderstanding here.
5:37.836–5:41.385
Dream RSI does not train a new world model first and
5:41.385–5:43.836
then have it approximate reality.
5:43.836–5:49.176
For Dream RSI, the history that has already played out serves as the playback simulator
5:49.176–5:50.176
itself.
5:50.176–5:52.509
Thus, its dreams are quite conservative.
5:52.509–5:55.376
It isn't predicting events that haven't happened,
5:55.376–5:57.976
but re-entering events that truly took place.
5:57.976–6:01.356
Strictly speaking, this world has a clear boundary.
6:01.356–6:04.736
It can only dream of places it has actually experienced.
6:04.736–6:08.936
It cannot know the results of unexplored branches out of thin air.
6:08.936–6:12.496
This is precisely both its advantage and its limitation.
6:12.496–6:17.363
The advantage is that real executions that have already occurred don't require paying
6:17.363–6:19.696
for expensive online exploration again,
6:19.696–6:23.429
supporting a large volume of rapid offline strategy comparisons.
6:23.429–6:27.296
The limitation is that if the first round of real exploration is poor,
6:27.296–6:28.896
with almost no high-quality
6:28.896–6:30.363
paths in the history tree,
6:30.363–6:34.096
the subsequent dreams won't conjure up a good event out of nothing.
6:34.096–6:35.943
So the way it truly loops is this.
6:35.943–6:40.316
Reality provides a portion of the world. Dreams in this
6:40.316–6:43.362
world find better exploration strategies, and
6:43.362–6:46.256
new strategies return to reality to traverse
6:46.256–6:50.196
new nodes, which then expand the next round of dreams.
6:50.196–6:53.936
This is the evolving world mentioned in the paper's title.
6:53.936–6:57.136
The world isn't built once and left unchanged.
6:57.136–7:00.345
It expands continuously alongside real exploration,
7:00.345–7:02.856
and the exploration strategy evolves with
7:02.856–7:03.856
the world.
7:03.856–7:10.076
This is much more interesting than simply summarizing historical experience into a few sentences.
7:10.076–7:14.476
The authors actually tested another, seemingly simpler method.
7:14.476–7:17.876
Since we already know which directions succeeded or failed,
7:17.876–7:19.676
why not let a model summarize the
7:19.676–7:23.543
lessons and inject these suggestions into the next round of prompts?
7:24.611–7:26.841
However, the results weren't good.
7:27.461–7:28.781
Let's look at Figure 5.
7:29.281–7:32.745
After adding the guidance, performance actually worsened,
7:32.745–7:35.461
regardless of whether RSI was originally used.
7:35.701–7:39.364
This is worth pondering, because remembering the past and
7:39.364–7:41.981
utilizing the past are not the same thing.
7:42.141–7:50.301
Compressing history into a single best thing to do is equivalent to imposing a predetermined shared direction on all subsequent exploration.
7:50.301–7:52.438
It could be helpful, but it might also
7:52.438–7:54.644
constrain the search space too early,
7:54.644–7:58.781
excluding many unconventional paths that should have been preserved.
7:59.061–8:00.394
Dream RSI is different.
8:00.394–8:04.128
It doesn't tell the AI up front which experience is most important.
8:04.201–8:06.001
It simply preserves that world,
8:06.001–8:09.468
allowing different strategies to explore it for themselves.
8:09.468–8:14.534
I think the most interesting takeaway from this paper isn't that AI can learn from history,
8:14.921–8:21.361
but rather that perhaps truly valuable experience isn't about summarizing the past into a single instruction,
8:21.841–8:25.041
but about preserving a past that can be re-experienced.
8:25.201–8:28.131
If Dream RSI only ends up saving a few API calls,
8:28.131–8:30.761
then it's not actually that interesting.
8:31.301–8:36.768
What's truly worth looking at is whether it changes the actual exploration process that follows.
8:36.768–8:40.434
The authors conducted experiments in algorithmic engineering,
8:40.434–8:42.568
mathematical optimization, and GPU.
8:42.568–8:43.768
Kernel engineering.
8:43.768–8:45.434
There are a lot of numbers, but
8:45.434–8:47.701
I think only a few are worth remembering.
8:47.701–8:48.901
The first set is cost.
8:48.901–8:51.168
In the Lowell algorithm discovery task,
8:51.168–8:55.921
one of the most striking figures in the paper is a maximum call gap of about 162 times.
8:56.181–8:59.048
However, this figure cannot be taken in isolation.
8:59.141–9:04.121
Dream RSI uses Gemini 1.5 Flash, while Simpletext uses GPT-40.
9:04.121–9:08.388
A much cleaner comparison is the head-to-head control between Dream RSI and
9:08.388–9:11.521
recursive fixed exploration using the same base model.
9:11.561–9:14.494
However, I am more concerned with another question.
9:14.561–9:17.881
If we search so much less, will the results get worse?
9:18.041–9:21.161
The paper does not show this kind of simple trade-off.
9:21.541–9:25.808
On some tasks, it achieves comparable performance with fewer generations,
9:25.861–9:26.794
while on others,
9:26.794–9:30.261
it achieves even higher performance within a similar budget.
9:30.261–9:34.941
Curil BCH is a great example to illustrate these two different types of gains.
9:34.941–9:40.381
VGG16 and LayerNorm show a better sense of when enough is enough,
9:40.381–9:44.314
essentially achieving similar performance with fewer generations.
9:45.360–9:48.480
Meanwhile, Conv and CX show a better sense of when
9:48.480–9:53.350
not to stop, meaning they push performance further within a similar budget.
9:54.130–9:58.810
This leads to what I consider the most elegant behavioral chart in the entire paper.
9:59.050–10:00.983
Let's look at this chart together.
10:01.270–10:09.330
The strategy learned by Dream RSI isn't a simple rule like always do less or always branch out more.
10:09.330–10:12.295
When performance improvements are steady early on,
10:12.295–10:14.250
it reduces the number of attempts.
10:14.250–10:19.510
But when performance hits a plateau, it expands its exploration again.
10:19.510–10:23.810
In other words, it has learned a dynamic exploration rhythm.
10:23.810–10:26.410
Don't waste your budget casting a wide net when
10:26.410–10:28.410
things are going well, but don't get
10:28.410–10:30.677
stuck on the same path when you hit a wall.
10:30.677–10:34.222
This is more important than the 162 token count because
10:34.222–10:36.710
it shows that RSI ultimately changes
10:36.710–10:39.741
not just how much compute a single task uses, but
10:39.741–10:42.090
how compute is allocated for the next
10:42.090–10:43.490
exploration.
10:43.490–10:48.157
This is why the authors place it within the context of recursive self-improvement.
10:48.157–10:51.490
However, we must clearly define the boundaries.
10:51.490–10:54.223
The underlying coding agent remains unchanged,
10:54.223–10:56.557
the model remains the same, the evaluator
10:56.557–10:59.610
is untouched, and the task objectives are fixed.
10:59.610–11:04.870
This RSI only modifies the outer strategy for how to continue seeking better results.
11:04.870–11:10.450
Therefore, if we interpret this as Google's AI starting to rewrite its own brain, that
11:10.450–11:12.530
would be a complete exaggeration.
11:12.530–11:15.954
Conversely, if we just view it as a cost-saving scheduler,
11:15.954–11:19.098
that underestimates what this paper truly achieves,
11:19.098–11:23.710
as the scheduling rules themselves have now become subjects for improvement.
11:24.010–11:28.170
In fact, someone on X has already asked the most critical question.
11:28.770–11:30.650
Would you call this true RSI?
11:30.930–11:34.910
This is also the question I wanted to ask most after reading this paper.
11:34.910–11:42.238
Since we spent our entire last episode trying to answer what counts as true recursive self-improvement,
11:42.238–11:45.819
we can bring back the L1 to L5 chart from last time and
11:45.819–11:48.650
use it as a yardstick to measure this RSI.
11:49.150–11:54.217
One of the most important viewpoints from that paper was that performance improvement and
11:54.217–11:57.150
recursive self-improvement are not the same thing.
11:57.150–12:00.151
A system can perform tasks increasingly well, but
12:00.151–12:04.392
if the method of self-improvement is hard-coded from the outside,
12:04.392–12:07.270
it is still one step away from true recursion.
12:07.450–12:09.357
Taking AlphaDev as an example,
12:09.357–12:14.930
its most intuitive ability is to continuously search for better programs and algorithms.
12:15.330–12:18.310
The main object of improvement is the result itself.
12:18.830–12:21.497
You get better at finding better answers, while
12:21.497–12:24.030
Darwin-inspired agents take a step further.
12:24.030–12:25.630
It doesn't just change code.
12:25.630–12:29.630
It allows the implementation of the coding agent itself to be modified.
12:29.630–12:32.830
So the object of improvement is no longer just the answer,
12:32.830–12:35.097
but the agent that generates the answer.
12:35.097–12:37.430
However, in that RSI paper from last time,
12:37.430–12:40.563
rules regarding which candidate solutions are kept and
12:40.563–12:47.310
which candidate solutions the next generation evolves from were still not part of the self-modification loop.
12:48.030–12:50.290
In other words, the agent can change,
12:50.610–12:55.010
but the rules determining where the next generation of agents should come from
12:55.010–12:57.010
have not yet entered the recursion.
12:57.010–12:59.950
Dream RSI happens to tread exactly on this boundary.
13:00.230–13:03.497
What Dream RSI leaves outside the self-modification loop
13:03.497–13:07.897
is precisely the mechanism of candidate solution maintenance and selection,
13:07.897–13:11.030
essentially where the next generation continues from.
13:11.030–13:13.763
What Dream RSI does is precisely turn this layer,
13:13.763–13:17.163
which nodes continue to branch, how many branches to create,
13:17.163–13:18.097
and when to stop,
13:18.097–13:20.830
into an object that can be replaced and improved.
13:20.830–13:23.697
It didn't modify the underlying agent, but instead
13:23.697–13:25.097
changed the outer layer,
13:25.097–13:28.430
that is, how the next round of improvement should be sought,
13:28.430–13:31.897
how many branches to continue opening from historical nodes,
13:31.897–13:32.830
and when to stop.
13:32.830–13:36.630
These elements, which originally belonged to the system designer,
13:36.630–13:39.563
have been transformed into an exploration strategy
13:39.563–13:41.897
that can be evaluated, modified, and then
13:41.897–13:43.630
carried over to the next round.
13:43.630–13:47.030
The key is that this new strategy doesn't just stay in a dream.
13:47.030–13:50.363
It directly governs the real exploration of the next round.
13:50.363–13:54.097
With a fixed strategy, each round can still continue to explore and
13:54.097–13:54.963
grow new nodes.
13:54.963–13:59.897
The real difference is that Dream RSI lets the new strategy learned in the previous round
13:59.897–14:03.790
decide where the tree grows next and how resources are allocated.
14:04.010–14:08.070
And the new exploration history feeds back into the replay simulator,
14:08.490–14:11.623
which in turn modifies the next version of the strategy.
14:11.623–14:16.366
So, if we look at it through the L1-L5 framework from that paper last time,
14:16.366–14:17.590
Dream RSI already
14:17.590–14:20.670
exhibits a very clear L5 characteristic.
14:20.670–14:25.337
The object being modified now includes the method used to generate the next round of
14:25.337–14:26.603
improvements itself.
14:26.603–14:28.950
But I must add a very important caveat.
14:28.950–14:31.330
It is an extremely narrow L5.
14:31.330–14:34.526
It does not autonomously modify the underlying model,
14:34.526–14:36.610
change the evaluator, redefine the
14:36.610–14:40.250
task, or decide which capability I should improve next.
14:40.250–14:46.050
It performs recursive improvement on the exploration mechanism within the boundaries of fixed goals,
14:46.050–14:49.117
fixed evaluation rules, and a fixed underlying agent.
14:49.117–14:50.250
In the last episode,
14:50.250–14:53.583
we actually distinguished between two types of recursion.
14:53.583–14:54.983
The first is very simple.
14:54.983–14:58.783
Is the mechanism you modified actually used again in the next round?
14:58.783–15:00.917
This is called structural recursion.
15:00.917–15:02.983
Dream RSI is very clear on this point.
15:02.983–15:07.717
The new exploration strategy enters the next round of online exploration directly.
15:07.717–15:09.783
But the harder one is the second type.
15:09.783–15:13.717
does using it actually make the successor system better at improving?
15:13.717–15:15.383
This is effective recursion.
15:15.383–15:18.117
Dream RSI provides a fairly clean contrast here.
15:18.117–15:21.183
It uses the same underlying model, the same evaluator,
15:21.183–15:24.850
and the same initial strategy as the fixed exploration baseline.
15:25.581–15:31.801
by whether the subsequent exploration strategy has been improved through dreaming. So Dream RSI's
15:31.801–15:34.698
L5 has two dimensions. It is very narrow in scope,
15:34.698–15:37.241
but in terms of the clarity of its recursive
15:37.241–15:39.533
evidence, it is not weak. Therefore,
15:39.533–15:42.861
a more accurate statement is not that true general RSI
15:42.861–15:46.808
has arrived, but rather that we have finally seen a very clean,
15:46.808–15:49.161
local example where the improvement
15:49.161–15:54.801
This also makes its relationship with AV training particularly interesting.
15:54.801–15:57.091
particularly interesting. Last time,
15:57.091–16:00.421
we discussed how AV training precipitates the results of
16:00.421–16:04.754
post-training experiments into a set of persistent research strategies and
16:04.754–16:05.821
exploration logs,
16:05.821–16:10.561
which are then modified by the meta-agent to guide the subsequent executive agents.
16:10.941–16:14.181
Doesn't that sound already very close to Dream RSI?
16:14.541–16:18.874
Neither is satisfied with just modifying the final model or the final answer.
16:18.874–16:23.741
They have begun to make the research process itself something that can be accumulated,
16:23.741–16:25.208
modified, and inherited.
16:25.208–16:28.361
However, the way both utilize history is not the same.
16:28.581–16:34.221
AVIL training is more like letting a meta-agent read past evidence to directly diagnose problems
16:34.221–16:37.673
and then modify the policy. But it adds an extra step.
16:37.673–16:40.341
Instead of just inferring a new strategy
16:40.341–16:44.773
based on history, it generates multiple candidate exploration strategies,
16:44.773–16:46.341
puts them into a historical
16:46.341–16:48.505
replay world to actually run them, and
16:48.505–16:51.481
uses the results to choose which one enters reality.
16:51.701–16:55.483
See, this is more like modifying a plan after a post-mortem review,
16:55.483–16:56.901
whereas the other is more
16:56.901–17:00.701
like putting multiple new plans back into a sandbox for simulation.
17:00.701–17:01.834
This brings us right
17:01.834–17:02.901
back to the figure.
17:02.901–17:06.501
The authors actually specifically tried the more direct path of
17:06.501–17:09.301
summarizing history into semantic guidance, but
17:09.301–17:12.234
the results were worse. Therefore, the most special
17:12.234–17:15.301
thing about JRSI is not just that it preserves history,
17:15.301–17:17.434
but that it lets candidate strategies
17:17.434–17:19.234
re-experience history and then
17:19.234–17:22.768
uses the outcomes of that experience to make a choice. Perhaps a
17:22.768–17:28.101
truly powerful self-improving system in the future will not choose just one of these, but might
17:28.101–17:32.101
simultaneously distill experience into high-level strategies while
17:32.101–17:33.834
retaining a sufficiently rich
17:33.834–17:35.368
set of historical events so
17:35.368–17:38.701
these strategies can be retested at any time. At that point,
17:38.701–17:42.634
memory will no longer just be the model knowing what it has done before,
17:42.634–17:45.301
but will start to become a past that can be rerun.
17:45.301–17:48.761
Speaking of which, there is one final question.
17:49.461–17:52.530
Why did the authors choose to call this dream?
17:52.530–17:55.841
As we mentioned earlier, Dream RSI does not train
17:55.841–17:58.443
a world model in the traditional sense, but
17:58.443–18:01.841
it borrows the same intuition. Real-world interaction
18:01.841–18:03.037
is too expensive, so
18:03.037–18:07.961
can we turn past experiences into an internal environment for continued learning?
18:09.001–18:11.476
Dream RSI borrows this line of thinking, but
18:11.476–18:14.821
the object it simulates is different. It does not attempt to
18:14.821–18:17.354
learn a complete external world, but instead
18:17.354–18:20.021
preserves how it explored this world in the past
18:20.021–18:22.621
and what happened during each exploration. So
18:22.621–18:25.288
I think the truly interesting part of this paper
18:25.288–18:28.321
is that the world here has quietly changed its target.
18:28.561–18:30.494
In the past, we were more concerned
18:30.494–18:32.294
with how the world would change.
18:32.294–18:34.181
But Dream RSI starts to care about
18:34.181–18:36.781
how I personally sought out changes in the past
18:36.781–18:40.114
and what would happen if I used a different way of searching.
18:40.114–18:42.941
Of course, today's Dream RSI is still very far
18:42.941–18:44.741
from being a superintelligence
18:44.741–18:46.941
that can improve itself indefinitely.
18:46.941–18:49.941
Its model hasn't changed, its goals haven't changed,
18:49.941–18:52.874
and the rules evaluating it haven't changed either.
18:52.874–18:57.074
Even the dreams it can have are limited to the past it has truly experienced.
18:57.761–19:02.361
But something that used to be set in stone has started to shift.
19:03.021–19:05.554
It's not about what the next answer should be,
19:05.741–19:09.261
but rather how I should find a way to improve myself next time.
19:09.261–19:11.375
When the answer to this question also
19:11.375–19:14.581
starts to be modified and verified by the machine itself
19:14.581–19:16.114
to be used in the next round,
19:16.341–19:20.721
the most noteworthy part of recursive self-improvement may have just begun.
19:20.721–19:24.002
Alright, that's all for today's content. Please like,
19:24.002–19:26.141
share, and subscribe to my channel.
19:26.301–19:28.301
If you'd like to support me further,
19:28.301–19:31.501
feel free to join my membership. The cost of a cup of coffee
19:31.501–19:33.768
each month not only supports me, but also
19:33.768–19:36.768
helps me continue creating more content related to AI
19:36.768–19:39.000
and technology. Alright, thanks for watching, and
0:00.000–0:01.710
When sharing this paper,
When sharing this paper,
0:01.710–0:07.573
Google's Subham Sahu wrote that for an agent to achieve recursive self-improvement,
Google's Subham Sahu wrote that for an agent to achieve recursive self-improvement,
0:07.573–0:09.120
it must learn to dream.
it must learn to dream.
0:09.640–0:11.707
History is the event of its dreaming.
History is the event of its dreaming.
0:12.220–0:15.007
Well, this sounds like science fiction, but
Well, this sounds like science fiction, but
0:15.007–0:20.280
what this paper actually does is more practical than the term dreaming might imply.
what this paper actually does is more practical than the term dreaming might imply.
0:20.800–0:22.180
Why do humans dream?
Why do humans dream?
0:22.800–0:25.800
Because what happened during the day is already over,
Because what happened during the day is already over,
0:26.220–0:30.320
but at night, the brain replays and recombines those experiences.
but at night, the brain replays and recombines those experiences.
0:30.940–0:33.740
We don't need to actually relive the same day,
We don't need to actually relive the same day,
0:34.140–0:37.273
but we might discover new connections in these replays,
but we might discover new connections in these replays,
0:37.273–0:40.007
or even change how we make decisions the next day.
or even change how we make decisions the next day.
0:40.007–0:42.022
Now, a research team from Google and
Now, a research team from Google and
0:42.022–0:45.880
Google DeepMind has brought a very similar idea into AI.
Google DeepMind has brought a very similar idea into AI.
0:46.240–0:48.056
The paper is called Dream RSI,
The paper is called Dream RSI,
0:48.056–0:51.760
Recursive Self-Improvement Based on Evolutionary Worlds.
Recursive Self-Improvement Based on Evolutionary Worlds.
0:51.760–0:55.826
The system it proposes explores a real environment during the day,
The system it proposes explores a real environment during the day,
0:55.826–0:59.020
writing actual code, compiling, and running tests.
writing actual code, compiling, and running tests.
0:59.460–1:00.660
But in the next round,
But in the next round,
1:00.660–1:03.460
it doesn't repeat all those expensive processes.
it doesn't repeat all those expensive processes.
1:03.460–1:08.100
Instead, it turns past explorations into a dream world that can be revisited.
Instead, it turns past explorations into a dream world that can be revisited.
1:08.300–1:12.690
In this world, it can ask, what if I hadn't kept digging down this path,
In this world, it can ask, what if I hadn't kept digging down this path,
1:12.690–1:14.960
but had changed direction earlier?
but had changed direction earlier?
1:15.160–1:19.160
What if, instead of running four processes at once, I ran eight?
What if, instead of running four processes at once, I ran eight?
1:19.160–1:22.760
If there has been no improvement for several consecutive steps,
If there has been no improvement for several consecutive steps,
1:22.760–1:24.427
should I have stopped sooner?
should I have stopped sooner?
1:24.427–1:27.840
It doesn't regenerate the code or rerun all the experiments.
It doesn't regenerate the code or rerun all the experiments.
1:28.560–1:30.760
Many results have already happened and
Many results have already happened and
1:30.760–1:33.227
are embedded in the history of exploration.
are embedded in the history of exploration.
1:33.320–1:37.152
What the AI does is try different exploration methods over and
What the AI does is try different exploration methods over and
1:37.152–1:38.920
over within that past world.
over within that past world.
1:39.620–1:43.960
Then it brings the better strategies found in the dream back to reality.
Then it brings the better strategies found in the dream back to reality.
1:44.280–1:47.080
New strategies produce new exploration history.
New strategies produce new exploration history.
1:47.080–1:49.080
New history becomes new dreams, and
New history becomes new dreams, and
1:49.080–1:52.480
new dreams modify the exploration method for the next round.
new dreams modify the exploration method for the next round.
1:52.480–1:54.940
This is why the authors call it RSI.
This is why the authors call it RSI.
1:57.203–2:02.028
But the really interesting question here isn't the dreaming metaphor itself,
But the really interesting question here isn't the dreaming metaphor itself,
2:02.028–2:04.733
but what actually changes after it wakes up?
but what actually changes after it wakes up?
2:07.026–2:12.179
And this is exactly where Dream RSI sets itself apart from AlphaCode,
And this is exactly where Dream RSI sets itself apart from AlphaCode,
2:12.179–2:14.756
Godel Machines, and AI Scientist.
Godel Machines, and AI Scientist.
2:15.196–2:16.407
In the last episode,
In the last episode,
2:16.407–2:19.896
we asked what counts as true recursive self-improvement,
we asked what counts as true recursive self-improvement,
2:20.196–2:21.416
and in this episode,
and in this episode,
2:21.416–2:24.716
a real system has emerged that we can actually examine.
a real system has emerged that we can actually examine.
2:24.956–2:27.889
All right, let's put the dreams aside for a moment and
All right, let's put the dreams aside for a moment and
2:27.889–2:30.023
look at what it's doing during the day.
look at what it's doing during the day.
2:30.023–2:34.356
Many automated discovery systems already look like they are self-evolving,
Many automated discovery systems already look like they are self-evolving,
2:34.416–2:37.949
Like a coding agent that keeps generating candidate programs,
Like a coding agent that keeps generating candidate programs,
2:38.276–2:40.056
an evaluator that scores them,
an evaluator that scores them,
2:40.336–2:44.269
and then the system keeping the better results to continue iterating.
and then the system keeping the better results to continue iterating.
2:44.269–2:48.069
But there are actually two completely different jobs mixed up here.
But there are actually two completely different jobs mixed up here.
2:48.416–2:50.936
Imagine a research team with two people.
Imagine a research team with two people.
2:51.436–2:54.525
One performs the experiments, writes the code,
One performs the experiments, writes the code,
2:54.525–2:56.456
runs tests, and gets results,
runs tests, and gets results,
2:56.456–2:59.789
while the other doesn't necessarily get their hands dirty.
while the other doesn't necessarily get their hands dirty.
3:00.316–3:03.116
They are responsible for deciding what to do next,
They are responsible for deciding what to do next,
3:03.116–3:05.316
such as which path is worth continuing,
such as which path is worth continuing,
3:05.316–3:08.376
which should be stopped, and where resources should
which should be stopped, and where resources should
3:08.376–3:13.976
be invested. Systems like AlphaEvo make it very easy to get the feeling that as the program gets
be invested. Systems like AlphaEvo make it very easy to get the feeling that as the program gets
3:13.976–3:16.247
stronger generation by generation,
stronger generation by generation,
3:16.247–3:20.056
it seems like the AI is already improving itself. But if we zoom
it seems like the AI is already improving itself. But if we zoom
3:20.056–3:22.246
out a little, we see a problem. While
out a little, we see a problem. While
3:22.246–3:25.896
the underlying code can change and which results to keep can
the underlying code can change and which results to keep can
3:25.896–3:29.939
change, the second type of work, how to explore the next steps,
change, the second type of work, how to explore the next steps,
3:29.939–3:32.116
is often still pre-designed. For
is often still pre-designed. For
3:32.116–3:34.758
For example, from which node to continue,
For example, from which node to continue,
3:34.758–3:37.476
how many branches to open simultaneously,
how many branches to open simultaneously,
3:37.476–3:40.676
and after how many attempts to abandon a direction.
and after how many attempts to abandon a direction.
3:40.676–3:43.809
These rules usually don't evolve along with the system.
These rules usually don't evolve along with the system.
3:43.809–3:45.496
What about Dream RSI?
What about Dream RSI?
3:45.496–3:50.736
It specifically targets the person responsible for deciding what to do next.
It specifically targets the person responsible for deciding what to do next.
3:50.736–3:53.306
Dream RSI separates this layer, and
Dream RSI separates this layer, and
3:53.306–3:57.676
its real-world exploration continuously grows a discovery
its real-world exploration continuously grows a discovery
3:57.676–3:58.676
tree.
tree.
3:58.676–4:03.516
Every node on the tree corresponds to a real attempt that has taken place, such as what
Every node on the tree corresponds to a real attempt that has taken place, such as what
4:03.516–4:06.116
solution the model generated in what context,
solution the model generated in what context,
4:06.116–4:08.583
whether the code could run, and what results
whether the code could run, and what results
4:08.583–4:10.276
the evaluator gave.
the evaluator gave.
4:10.276–4:11.943
All of this has been recorded.
All of this has been recorded.
4:11.943–4:16.896
The outer exploration strategy only needs to answer a few very simple questions, like
The outer exploration strategy only needs to answer a few very simple questions, like
4:16.896–4:19.096
from which node should we continue now?
from which node should we continue now?
4:19.096–4:21.116
Should we start a new path?
Should we start a new path?
4:21.116–4:24.116
How many directions should we pursue in parallel?
How many directions should we pursue in parallel?
4:24.116–4:25.249
When should we stop?
When should we stop?
4:25.249–4:26.516
Let's look at figure 2,
Let's look at figure 2,
4:26.516–4:30.516
which is the most important figure for understanding the entire paper.
which is the most important figure for understanding the entire paper.
4:30.516–4:35.249
You can think of this tree in the middle as the maze that the AI actually walked through
You can think of this tree in the middle as the maze that the AI actually walked through
4:35.249–4:35.916
yesterday.
yesterday.
4:35.916–4:38.983
Every node that is lit up is a place it actually visited,
Every node that is lit up is a place it actually visited,
4:38.983–4:40.916
and the results are already known.
and the results are already known.
4:40.916–4:42.164
So the question is,
So the question is,
4:42.164–4:46.976
if I want to compare two new exploration strategies today, must I make
if I want to compare two new exploration strategies today, must I make
4:46.976–4:50.116
the AI walk through the maze twice from the beginning?
the AI walk through the maze twice from the beginning?
4:50.116–4:53.416
In the past, that was basically the only way.
In the past, that was basically the only way.
4:53.416–4:58.136
Because you could only know if a new strategy was good by actually running it again.
Because you could only know if a new strategy was good by actually running it again.
4:58.136–5:01.736
But Dream RSI's answer is, not necessarily.
But Dream RSI's answer is, not necessarily.
5:01.736–5:04.536
Since it has already walked through it yesterday,
Since it has already walked through it yesterday,
5:04.536–5:07.069
and what happened at each node has been saved,
and what happened at each node has been saved,
5:07.069–5:11.196
a new strategy can completely walk through this historical tree first.
a new strategy can completely walk through this historical tree first.
5:11.196–5:14.422
It might go left first, it might turn right sooner,
It might go left first, it might turn right sooner,
5:14.422–5:16.496
or it might just take three steps
or it might just take three steps
5:16.496–5:17.696
and stop.
and stop.
5:17.696–5:20.730
As long as he visits nodes that have truly occurred,
As long as he visits nodes that have truly occurred,
5:20.730–5:22.776
the simulator can directly reveal
the simulator can directly reveal
5:22.776–5:25.916
the actual results from that time.
the actual results from that time.
5:25.916–5:29.271
The key shift here isn't just remembering history,
The key shift here isn't just remembering history,
5:29.271–5:31.456
but rather turning history from a
but rather turning history from a
5:31.456–5:34.496
log into a re-enterable event.
log into a re-enterable event.
5:34.496–5:37.836
Now, there is a common misunderstanding here.
Now, there is a common misunderstanding here.
5:37.836–5:41.385
Dream RSI does not train a new world model first and
Dream RSI does not train a new world model first and
5:41.385–5:43.836
then have it approximate reality.
then have it approximate reality.
5:43.836–5:49.176
For Dream RSI, the history that has already played out serves as the playback simulator
For Dream RSI, the history that has already played out serves as the playback simulator
5:49.176–5:50.176
itself.
itself.
5:50.176–5:52.509
Thus, its dreams are quite conservative.
Thus, its dreams are quite conservative.
5:52.509–5:55.376
It isn't predicting events that haven't happened,
It isn't predicting events that haven't happened,
5:55.376–5:57.976
but re-entering events that truly took place.
but re-entering events that truly took place.
5:57.976–6:01.356
Strictly speaking, this world has a clear boundary.
Strictly speaking, this world has a clear boundary.
6:01.356–6:04.736
It can only dream of places it has actually experienced.
It can only dream of places it has actually experienced.
6:04.736–6:08.936
It cannot know the results of unexplored branches out of thin air.
It cannot know the results of unexplored branches out of thin air.
6:08.936–6:12.496
This is precisely both its advantage and its limitation.
This is precisely both its advantage and its limitation.
6:12.496–6:17.363
The advantage is that real executions that have already occurred don't require paying
The advantage is that real executions that have already occurred don't require paying
6:17.363–6:19.696
for expensive online exploration again,
for expensive online exploration again,
6:19.696–6:23.429
supporting a large volume of rapid offline strategy comparisons.
supporting a large volume of rapid offline strategy comparisons.
6:23.429–6:27.296
The limitation is that if the first round of real exploration is poor,
The limitation is that if the first round of real exploration is poor,
6:27.296–6:28.896
with almost no high-quality
with almost no high-quality
6:28.896–6:30.363
paths in the history tree,
paths in the history tree,
6:30.363–6:34.096
the subsequent dreams won't conjure up a good event out of nothing.
the subsequent dreams won't conjure up a good event out of nothing.
6:34.096–6:35.943
So the way it truly loops is this.
So the way it truly loops is this.
6:35.943–6:40.316
Reality provides a portion of the world. Dreams in this
Reality provides a portion of the world. Dreams in this
6:40.316–6:43.362
world find better exploration strategies, and
world find better exploration strategies, and
6:43.362–6:46.256
new strategies return to reality to traverse
new strategies return to reality to traverse
6:46.256–6:50.196
new nodes, which then expand the next round of dreams.
new nodes, which then expand the next round of dreams.
6:50.196–6:53.936
This is the evolving world mentioned in the paper's title.
This is the evolving world mentioned in the paper's title.
6:53.936–6:57.136
The world isn't built once and left unchanged.
The world isn't built once and left unchanged.
6:57.136–7:00.345
It expands continuously alongside real exploration,
It expands continuously alongside real exploration,
7:00.345–7:02.856
and the exploration strategy evolves with
and the exploration strategy evolves with
7:02.856–7:03.856
the world.
the world.
7:03.856–7:10.076
This is much more interesting than simply summarizing historical experience into a few sentences.
This is much more interesting than simply summarizing historical experience into a few sentences.
7:10.076–7:14.476
The authors actually tested another, seemingly simpler method.
The authors actually tested another, seemingly simpler method.
7:14.476–7:17.876
Since we already know which directions succeeded or failed,
Since we already know which directions succeeded or failed,
7:17.876–7:19.676
why not let a model summarize the
why not let a model summarize the
7:19.676–7:23.543
lessons and inject these suggestions into the next round of prompts?
lessons and inject these suggestions into the next round of prompts?
7:24.611–7:26.841
However, the results weren't good.
However, the results weren't good.
7:27.461–7:28.781
Let's look at Figure 5.
Let's look at Figure 5.
7:29.281–7:32.745
After adding the guidance, performance actually worsened,
After adding the guidance, performance actually worsened,
7:32.745–7:35.461
regardless of whether RSI was originally used.
regardless of whether RSI was originally used.
7:35.701–7:39.364
This is worth pondering, because remembering the past and
This is worth pondering, because remembering the past and
7:39.364–7:41.981
utilizing the past are not the same thing.
utilizing the past are not the same thing.
7:42.141–7:50.301
Compressing history into a single best thing to do is equivalent to imposing a predetermined shared direction on all subsequent exploration.
Compressing history into a single best thing to do is equivalent to imposing a predetermined shared direction on all subsequent exploration.
7:50.301–7:52.438
It could be helpful, but it might also
It could be helpful, but it might also
7:52.438–7:54.644
constrain the search space too early,
constrain the search space too early,
7:54.644–7:58.781
excluding many unconventional paths that should have been preserved.
excluding many unconventional paths that should have been preserved.
7:59.061–8:00.394
Dream RSI is different.
Dream RSI is different.
8:00.394–8:04.128
It doesn't tell the AI up front which experience is most important.
It doesn't tell the AI up front which experience is most important.
8:04.201–8:06.001
It simply preserves that world,
It simply preserves that world,
8:06.001–8:09.468
allowing different strategies to explore it for themselves.
allowing different strategies to explore it for themselves.
8:09.468–8:14.534
I think the most interesting takeaway from this paper isn't that AI can learn from history,
I think the most interesting takeaway from this paper isn't that AI can learn from history,
8:14.921–8:21.361
but rather that perhaps truly valuable experience isn't about summarizing the past into a single instruction,
but rather that perhaps truly valuable experience isn't about summarizing the past into a single instruction,
8:21.841–8:25.041
but about preserving a past that can be re-experienced.
but about preserving a past that can be re-experienced.
8:25.201–8:28.131
If Dream RSI only ends up saving a few API calls,
If Dream RSI only ends up saving a few API calls,
8:28.131–8:30.761
then it's not actually that interesting.
then it's not actually that interesting.
8:31.301–8:36.768
What's truly worth looking at is whether it changes the actual exploration process that follows.
What's truly worth looking at is whether it changes the actual exploration process that follows.
8:36.768–8:40.434
The authors conducted experiments in algorithmic engineering,
The authors conducted experiments in algorithmic engineering,
8:40.434–8:42.568
mathematical optimization, and GPU.
mathematical optimization, and GPU.
8:42.568–8:43.768
Kernel engineering.
Kernel engineering.
8:43.768–8:45.434
There are a lot of numbers, but
There are a lot of numbers, but
8:45.434–8:47.701
I think only a few are worth remembering.
I think only a few are worth remembering.
8:47.701–8:48.901
The first set is cost.
The first set is cost.
8:48.901–8:51.168
In the Lowell algorithm discovery task,
In the Lowell algorithm discovery task,
8:51.168–8:55.921
one of the most striking figures in the paper is a maximum call gap of about 162 times.
one of the most striking figures in the paper is a maximum call gap of about 162 times.
8:56.181–8:59.048
However, this figure cannot be taken in isolation.
However, this figure cannot be taken in isolation.
8:59.141–9:04.121
Dream RSI uses Gemini 1.5 Flash, while Simpletext uses GPT-40.
Dream RSI uses Gemini 1.5 Flash, while Simpletext uses GPT-40.
9:04.121–9:08.388
A much cleaner comparison is the head-to-head control between Dream RSI and
A much cleaner comparison is the head-to-head control between Dream RSI and
9:08.388–9:11.521
recursive fixed exploration using the same base model.
recursive fixed exploration using the same base model.
9:11.561–9:14.494
However, I am more concerned with another question.
However, I am more concerned with another question.
9:14.561–9:17.881
If we search so much less, will the results get worse?
If we search so much less, will the results get worse?
9:18.041–9:21.161
The paper does not show this kind of simple trade-off.
The paper does not show this kind of simple trade-off.
9:21.541–9:25.808
On some tasks, it achieves comparable performance with fewer generations,
On some tasks, it achieves comparable performance with fewer generations,
9:25.861–9:26.794
while on others,
while on others,
9:26.794–9:30.261
it achieves even higher performance within a similar budget.
it achieves even higher performance within a similar budget.
9:30.261–9:34.941
Curil BCH is a great example to illustrate these two different types of gains.
Curil BCH is a great example to illustrate these two different types of gains.
9:34.941–9:40.381
VGG16 and LayerNorm show a better sense of when enough is enough,
VGG16 and LayerNorm show a better sense of when enough is enough,
9:40.381–9:44.314
essentially achieving similar performance with fewer generations.
essentially achieving similar performance with fewer generations.
9:45.360–9:48.480
Meanwhile, Conv and CX show a better sense of when
Meanwhile, Conv and CX show a better sense of when
9:48.480–9:53.350
not to stop, meaning they push performance further within a similar budget.
not to stop, meaning they push performance further within a similar budget.
9:54.130–9:58.810
This leads to what I consider the most elegant behavioral chart in the entire paper.
This leads to what I consider the most elegant behavioral chart in the entire paper.
9:59.050–10:00.983
Let's look at this chart together.
Let's look at this chart together.
10:01.270–10:09.330
The strategy learned by Dream RSI isn't a simple rule like always do less or always branch out more.
The strategy learned by Dream RSI isn't a simple rule like always do less or always branch out more.
10:09.330–10:12.295
When performance improvements are steady early on,
When performance improvements are steady early on,
10:12.295–10:14.250
it reduces the number of attempts.
it reduces the number of attempts.
10:14.250–10:19.510
But when performance hits a plateau, it expands its exploration again.
But when performance hits a plateau, it expands its exploration again.
10:19.510–10:23.810
In other words, it has learned a dynamic exploration rhythm.
In other words, it has learned a dynamic exploration rhythm.
10:23.810–10:26.410
Don't waste your budget casting a wide net when
Don't waste your budget casting a wide net when
10:26.410–10:28.410
things are going well, but don't get
things are going well, but don't get
10:28.410–10:30.677
stuck on the same path when you hit a wall.
stuck on the same path when you hit a wall.
10:30.677–10:34.222
This is more important than the 162 token count because
This is more important than the 162 token count because
10:34.222–10:36.710
it shows that RSI ultimately changes
it shows that RSI ultimately changes
10:36.710–10:39.741
not just how much compute a single task uses, but
not just how much compute a single task uses, but
10:39.741–10:42.090
how compute is allocated for the next
how compute is allocated for the next
10:42.090–10:43.490
exploration.
exploration.
10:43.490–10:48.157
This is why the authors place it within the context of recursive self-improvement.
This is why the authors place it within the context of recursive self-improvement.
10:48.157–10:51.490
However, we must clearly define the boundaries.
However, we must clearly define the boundaries.
10:51.490–10:54.223
The underlying coding agent remains unchanged,
The underlying coding agent remains unchanged,
10:54.223–10:56.557
the model remains the same, the evaluator
the model remains the same, the evaluator
10:56.557–10:59.610
is untouched, and the task objectives are fixed.
is untouched, and the task objectives are fixed.
10:59.610–11:04.870
This RSI only modifies the outer strategy for how to continue seeking better results.
This RSI only modifies the outer strategy for how to continue seeking better results.
11:04.870–11:10.450
Therefore, if we interpret this as Google's AI starting to rewrite its own brain, that
Therefore, if we interpret this as Google's AI starting to rewrite its own brain, that
11:10.450–11:12.530
would be a complete exaggeration.
would be a complete exaggeration.
11:12.530–11:15.954
Conversely, if we just view it as a cost-saving scheduler,
Conversely, if we just view it as a cost-saving scheduler,
11:15.954–11:19.098
that underestimates what this paper truly achieves,
that underestimates what this paper truly achieves,
11:19.098–11:23.710
as the scheduling rules themselves have now become subjects for improvement.
as the scheduling rules themselves have now become subjects for improvement.
11:24.010–11:28.170
In fact, someone on X has already asked the most critical question.
In fact, someone on X has already asked the most critical question.
11:28.770–11:30.650
Would you call this true RSI?
Would you call this true RSI?
11:30.930–11:34.910
This is also the question I wanted to ask most after reading this paper.
This is also the question I wanted to ask most after reading this paper.
11:34.910–11:42.238
Since we spent our entire last episode trying to answer what counts as true recursive self-improvement,
Since we spent our entire last episode trying to answer what counts as true recursive self-improvement,
11:42.238–11:45.819
we can bring back the L1 to L5 chart from last time and
we can bring back the L1 to L5 chart from last time and
11:45.819–11:48.650
use it as a yardstick to measure this RSI.
use it as a yardstick to measure this RSI.
11:49.150–11:54.217
One of the most important viewpoints from that paper was that performance improvement and
One of the most important viewpoints from that paper was that performance improvement and
11:54.217–11:57.150
recursive self-improvement are not the same thing.
recursive self-improvement are not the same thing.
11:57.150–12:00.151
A system can perform tasks increasingly well, but
A system can perform tasks increasingly well, but
12:00.151–12:04.392
if the method of self-improvement is hard-coded from the outside,
if the method of self-improvement is hard-coded from the outside,
12:04.392–12:07.270
it is still one step away from true recursion.
it is still one step away from true recursion.
12:07.450–12:09.357
Taking AlphaDev as an example,
Taking AlphaDev as an example,
12:09.357–12:14.930
its most intuitive ability is to continuously search for better programs and algorithms.
its most intuitive ability is to continuously search for better programs and algorithms.
12:15.330–12:18.310
The main object of improvement is the result itself.
The main object of improvement is the result itself.
12:18.830–12:21.497
You get better at finding better answers, while
You get better at finding better answers, while
12:21.497–12:24.030
Darwin-inspired agents take a step further.
Darwin-inspired agents take a step further.
12:24.030–12:25.630
It doesn't just change code.
It doesn't just change code.
12:25.630–12:29.630
It allows the implementation of the coding agent itself to be modified.
It allows the implementation of the coding agent itself to be modified.
12:29.630–12:32.830
So the object of improvement is no longer just the answer,
So the object of improvement is no longer just the answer,
12:32.830–12:35.097
but the agent that generates the answer.
but the agent that generates the answer.
12:35.097–12:37.430
However, in that RSI paper from last time,
However, in that RSI paper from last time,
12:37.430–12:40.563
rules regarding which candidate solutions are kept and
rules regarding which candidate solutions are kept and
12:40.563–12:47.310
which candidate solutions the next generation evolves from were still not part of the self-modification loop.
which candidate solutions the next generation evolves from were still not part of the self-modification loop.
12:48.030–12:50.290
In other words, the agent can change,
In other words, the agent can change,
12:50.610–12:55.010
but the rules determining where the next generation of agents should come from
but the rules determining where the next generation of agents should come from
12:55.010–12:57.010
have not yet entered the recursion.
have not yet entered the recursion.
12:57.010–12:59.950
Dream RSI happens to tread exactly on this boundary.
Dream RSI happens to tread exactly on this boundary.
13:00.230–13:03.497
What Dream RSI leaves outside the self-modification loop
What Dream RSI leaves outside the self-modification loop
13:03.497–13:07.897
is precisely the mechanism of candidate solution maintenance and selection,
is precisely the mechanism of candidate solution maintenance and selection,
13:07.897–13:11.030
essentially where the next generation continues from.
essentially where the next generation continues from.
13:11.030–13:13.763
What Dream RSI does is precisely turn this layer,
What Dream RSI does is precisely turn this layer,
13:13.763–13:17.163
which nodes continue to branch, how many branches to create,
which nodes continue to branch, how many branches to create,
13:17.163–13:18.097
and when to stop,
and when to stop,
13:18.097–13:20.830
into an object that can be replaced and improved.
into an object that can be replaced and improved.
13:20.830–13:23.697
It didn't modify the underlying agent, but instead
It didn't modify the underlying agent, but instead
13:23.697–13:25.097
changed the outer layer,
changed the outer layer,
13:25.097–13:28.430
that is, how the next round of improvement should be sought,
that is, how the next round of improvement should be sought,
13:28.430–13:31.897
how many branches to continue opening from historical nodes,
how many branches to continue opening from historical nodes,
13:31.897–13:32.830
and when to stop.
and when to stop.
13:32.830–13:36.630
These elements, which originally belonged to the system designer,
These elements, which originally belonged to the system designer,
13:36.630–13:39.563
have been transformed into an exploration strategy
have been transformed into an exploration strategy
13:39.563–13:41.897
that can be evaluated, modified, and then
that can be evaluated, modified, and then
13:41.897–13:43.630
carried over to the next round.
carried over to the next round.
13:43.630–13:47.030
The key is that this new strategy doesn't just stay in a dream.
The key is that this new strategy doesn't just stay in a dream.
13:47.030–13:50.363
It directly governs the real exploration of the next round.
It directly governs the real exploration of the next round.
13:50.363–13:54.097
With a fixed strategy, each round can still continue to explore and
With a fixed strategy, each round can still continue to explore and
13:54.097–13:54.963
grow new nodes.
grow new nodes.
13:54.963–13:59.897
The real difference is that Dream RSI lets the new strategy learned in the previous round
The real difference is that Dream RSI lets the new strategy learned in the previous round
13:59.897–14:03.790
decide where the tree grows next and how resources are allocated.
decide where the tree grows next and how resources are allocated.
14:04.010–14:08.070
And the new exploration history feeds back into the replay simulator,
And the new exploration history feeds back into the replay simulator,
14:08.490–14:11.623
which in turn modifies the next version of the strategy.
which in turn modifies the next version of the strategy.
14:11.623–14:16.366
So, if we look at it through the L1-L5 framework from that paper last time,
So, if we look at it through the L1-L5 framework from that paper last time,
14:16.366–14:17.590
Dream RSI already
Dream RSI already
14:17.590–14:20.670
exhibits a very clear L5 characteristic.
exhibits a very clear L5 characteristic.
14:20.670–14:25.337
The object being modified now includes the method used to generate the next round of
The object being modified now includes the method used to generate the next round of
14:25.337–14:26.603
improvements itself.
improvements itself.
14:26.603–14:28.950
But I must add a very important caveat.
But I must add a very important caveat.
14:28.950–14:31.330
It is an extremely narrow L5.
It is an extremely narrow L5.
14:31.330–14:34.526
It does not autonomously modify the underlying model,
It does not autonomously modify the underlying model,
14:34.526–14:36.610
change the evaluator, redefine the
change the evaluator, redefine the
14:36.610–14:40.250
task, or decide which capability I should improve next.
task, or decide which capability I should improve next.
14:40.250–14:46.050
It performs recursive improvement on the exploration mechanism within the boundaries of fixed goals,
It performs recursive improvement on the exploration mechanism within the boundaries of fixed goals,
14:46.050–14:49.117
fixed evaluation rules, and a fixed underlying agent.
fixed evaluation rules, and a fixed underlying agent.
14:49.117–14:50.250
In the last episode,
In the last episode,
14:50.250–14:53.583
we actually distinguished between two types of recursion.
we actually distinguished between two types of recursion.
14:53.583–14:54.983
The first is very simple.
The first is very simple.
14:54.983–14:58.783
Is the mechanism you modified actually used again in the next round?
Is the mechanism you modified actually used again in the next round?
14:58.783–15:00.917
This is called structural recursion.
This is called structural recursion.
15:00.917–15:02.983
Dream RSI is very clear on this point.
Dream RSI is very clear on this point.
15:02.983–15:07.717
The new exploration strategy enters the next round of online exploration directly.
The new exploration strategy enters the next round of online exploration directly.
15:07.717–15:09.783
But the harder one is the second type.
But the harder one is the second type.
15:09.783–15:13.717
does using it actually make the successor system better at improving?
does using it actually make the successor system better at improving?
15:13.717–15:15.383
This is effective recursion.
This is effective recursion.
15:15.383–15:18.117
Dream RSI provides a fairly clean contrast here.
Dream RSI provides a fairly clean contrast here.
15:18.117–15:21.183
It uses the same underlying model, the same evaluator,
It uses the same underlying model, the same evaluator,
15:21.183–15:24.850
and the same initial strategy as the fixed exploration baseline.
and the same initial strategy as the fixed exploration baseline.
15:25.581–15:31.801
by whether the subsequent exploration strategy has been improved through dreaming. So Dream RSI's
by whether the subsequent exploration strategy has been improved through dreaming. So Dream RSI's
15:31.801–15:34.698
L5 has two dimensions. It is very narrow in scope,
L5 has two dimensions. It is very narrow in scope,
15:34.698–15:37.241
but in terms of the clarity of its recursive
but in terms of the clarity of its recursive
15:37.241–15:39.533
evidence, it is not weak. Therefore,
evidence, it is not weak. Therefore,
15:39.533–15:42.861
a more accurate statement is not that true general RSI
a more accurate statement is not that true general RSI
15:42.861–15:46.808
has arrived, but rather that we have finally seen a very clean,
has arrived, but rather that we have finally seen a very clean,
15:46.808–15:49.161
local example where the improvement
local example where the improvement
15:49.161–15:54.801
This also makes its relationship with AV training particularly interesting.
This also makes its relationship with AV training particularly interesting.
15:54.801–15:57.091
particularly interesting. Last time,
particularly interesting. Last time,
15:57.091–16:00.421
we discussed how AV training precipitates the results of
we discussed how AV training precipitates the results of
16:00.421–16:04.754
post-training experiments into a set of persistent research strategies and
post-training experiments into a set of persistent research strategies and
16:04.754–16:05.821
exploration logs,
exploration logs,
16:05.821–16:10.561
which are then modified by the meta-agent to guide the subsequent executive agents.
which are then modified by the meta-agent to guide the subsequent executive agents.
16:10.941–16:14.181
Doesn't that sound already very close to Dream RSI?
Doesn't that sound already very close to Dream RSI?
16:14.541–16:18.874
Neither is satisfied with just modifying the final model or the final answer.
Neither is satisfied with just modifying the final model or the final answer.
16:18.874–16:23.741
They have begun to make the research process itself something that can be accumulated,
They have begun to make the research process itself something that can be accumulated,
16:23.741–16:25.208
modified, and inherited.
modified, and inherited.
16:25.208–16:28.361
However, the way both utilize history is not the same.
However, the way both utilize history is not the same.
16:28.581–16:34.221
AVIL training is more like letting a meta-agent read past evidence to directly diagnose problems
AVIL training is more like letting a meta-agent read past evidence to directly diagnose problems
16:34.221–16:37.673
and then modify the policy. But it adds an extra step.
and then modify the policy. But it adds an extra step.
16:37.673–16:40.341
Instead of just inferring a new strategy
Instead of just inferring a new strategy
16:40.341–16:44.773
based on history, it generates multiple candidate exploration strategies,
based on history, it generates multiple candidate exploration strategies,
16:44.773–16:46.341
puts them into a historical
puts them into a historical
16:46.341–16:48.505
replay world to actually run them, and
replay world to actually run them, and
16:48.505–16:51.481
uses the results to choose which one enters reality.
uses the results to choose which one enters reality.
16:51.701–16:55.483
See, this is more like modifying a plan after a post-mortem review,
See, this is more like modifying a plan after a post-mortem review,
16:55.483–16:56.901
whereas the other is more
whereas the other is more
16:56.901–17:00.701
like putting multiple new plans back into a sandbox for simulation.
like putting multiple new plans back into a sandbox for simulation.
17:00.701–17:01.834
This brings us right
This brings us right
17:01.834–17:02.901
back to the figure.
back to the figure.
17:02.901–17:06.501
The authors actually specifically tried the more direct path of
The authors actually specifically tried the more direct path of
17:06.501–17:09.301
summarizing history into semantic guidance, but
summarizing history into semantic guidance, but
17:09.301–17:12.234
the results were worse. Therefore, the most special
the results were worse. Therefore, the most special
17:12.234–17:15.301
thing about JRSI is not just that it preserves history,
thing about JRSI is not just that it preserves history,
17:15.301–17:17.434
but that it lets candidate strategies
but that it lets candidate strategies
17:17.434–17:19.234
re-experience history and then
re-experience history and then
17:19.234–17:22.768
uses the outcomes of that experience to make a choice. Perhaps a
uses the outcomes of that experience to make a choice. Perhaps a
17:22.768–17:28.101
truly powerful self-improving system in the future will not choose just one of these, but might
truly powerful self-improving system in the future will not choose just one of these, but might
17:28.101–17:32.101
simultaneously distill experience into high-level strategies while
simultaneously distill experience into high-level strategies while
17:32.101–17:33.834
retaining a sufficiently rich
retaining a sufficiently rich
17:33.834–17:35.368
set of historical events so
set of historical events so
17:35.368–17:38.701
these strategies can be retested at any time. At that point,
these strategies can be retested at any time. At that point,
17:38.701–17:42.634
memory will no longer just be the model knowing what it has done before,
memory will no longer just be the model knowing what it has done before,
17:42.634–17:45.301
but will start to become a past that can be rerun.
but will start to become a past that can be rerun.
17:45.301–17:48.761
Speaking of which, there is one final question.
Speaking of which, there is one final question.
17:49.461–17:52.530
Why did the authors choose to call this dream?
Why did the authors choose to call this dream?
17:52.530–17:55.841
As we mentioned earlier, Dream RSI does not train
As we mentioned earlier, Dream RSI does not train
17:55.841–17:58.443
a world model in the traditional sense, but
a world model in the traditional sense, but
17:58.443–18:01.841
it borrows the same intuition. Real-world interaction
it borrows the same intuition. Real-world interaction
18:01.841–18:03.037
is too expensive, so
is too expensive, so
18:03.037–18:07.961
can we turn past experiences into an internal environment for continued learning?
can we turn past experiences into an internal environment for continued learning?
18:09.001–18:11.476
Dream RSI borrows this line of thinking, but
Dream RSI borrows this line of thinking, but
18:11.476–18:14.821
the object it simulates is different. It does not attempt to
the object it simulates is different. It does not attempt to
18:14.821–18:17.354
learn a complete external world, but instead
learn a complete external world, but instead
18:17.354–18:20.021
preserves how it explored this world in the past
preserves how it explored this world in the past
18:20.021–18:22.621
and what happened during each exploration. So
and what happened during each exploration. So
18:22.621–18:25.288
I think the truly interesting part of this paper
I think the truly interesting part of this paper
18:25.288–18:28.321
is that the world here has quietly changed its target.
is that the world here has quietly changed its target.
18:28.561–18:30.494
In the past, we were more concerned
In the past, we were more concerned
18:30.494–18:32.294
with how the world would change.
with how the world would change.
18:32.294–18:34.181
But Dream RSI starts to care about
But Dream RSI starts to care about
18:34.181–18:36.781
how I personally sought out changes in the past
how I personally sought out changes in the past
18:36.781–18:40.114
and what would happen if I used a different way of searching.
and what would happen if I used a different way of searching.
18:40.114–18:42.941
Of course, today's Dream RSI is still very far
Of course, today's Dream RSI is still very far
18:42.941–18:44.741
from being a superintelligence
from being a superintelligence
18:44.741–18:46.941
that can improve itself indefinitely.
that can improve itself indefinitely.
18:46.941–18:49.941
Its model hasn't changed, its goals haven't changed,
Its model hasn't changed, its goals haven't changed,
18:49.941–18:52.874
and the rules evaluating it haven't changed either.
and the rules evaluating it haven't changed either.
18:52.874–18:57.074
Even the dreams it can have are limited to the past it has truly experienced.
Even the dreams it can have are limited to the past it has truly experienced.
18:57.761–19:02.361
But something that used to be set in stone has started to shift.
But something that used to be set in stone has started to shift.
19:03.021–19:05.554
It's not about what the next answer should be,
It's not about what the next answer should be,
19:05.741–19:09.261
but rather how I should find a way to improve myself next time.
but rather how I should find a way to improve myself next time.
19:09.261–19:11.375
When the answer to this question also
When the answer to this question also
19:11.375–19:14.581
starts to be modified and verified by the machine itself
starts to be modified and verified by the machine itself
19:14.581–19:16.114
to be used in the next round,
to be used in the next round,
19:16.341–19:20.721
the most noteworthy part of recursive self-improvement may have just begun.
the most noteworthy part of recursive self-improvement may have just begun.
19:20.721–19:24.002
Alright, that's all for today's content. Please like,
Alright, that's all for today's content. Please like,
19:24.002–19:26.141
share, and subscribe to my channel.
share, and subscribe to my channel.
19:26.301–19:28.301
If you'd like to support me further,
If you'd like to support me further,
19:28.301–19:31.501
feel free to join my membership. The cost of a cup of coffee
feel free to join my membership. The cost of a cup of coffee
19:31.501–19:33.768
each month not only supports me, but also
each month not only supports me, but also
19:33.768–19:36.768
helps me continue creating more content related to AI
helps me continue creating more content related to AI
19:36.768–19:39.000
and technology. Alright, thanks for watching, and
and technology. Alright, thanks for watching, and
影片筆記：谷歌这次，真的不一样了
<!-- VIDEONOTE_PUBLICATION_WARNINGS_BEGIN -->
⚠️ 全文快速校讀發現待確認修正
本筆記已完成可閱讀的逐字稿與時間跳轉；下列為待處理問題，建議播放原片確認。
- 待修正：逐字稿語言與宣告語言不一致（已保留來源字幕/Whisper 內容並繼續產生筆記；請確認影片語言。；若語音辨識輸出英文，也會被語言檢查標記，不會覆蓋日文底稿。）
- 待修正：逐字稿來源選擇未完全通過，仍先生成筆記（no_trustworthy_transcript_source；原始逐字稿保留；請進入精進模式確認來源。）
- 待修正：⚠️ 全文快速校讀：已套用暫定修正，請確認（segment=18 已先套用暫定修正：Many results have already happened and are hung on the history of exploration. → Many results have already happened and are embedded in the history of exploration.；segment=46 已先套用暫定修正：What about JRSI? → What about Dream RSI?）
- 待修正：逐字稿可先使用，但有待校正清單（segment=211 rapid_semantic_review: 全文速讀上下文判斷；尚未回聽或交叉比對；segment=120 rapid_semantic_review: 通常對比實驗會使用 'Simple baseline' 或 'Standard text' 等，且 'GPT-40' 並非現有公開模型版本（應為 GPT-4o 或 GPT-4）。）
- 待修正：逐字稿品質檢核未完全通過（{'stage': 'canonical_transcript_quality', 'action': 'published_with_warning', 'signals': ['transcript_selection_not_resolved']}）
<!-- VIDEONOTE_PUBLICATION_WARNINGS_END -->
一句話總結
Google DeepMind 發表的《Dream RSI》論文提出了一種基於「演化世界」的遞歸自我改進（RSI）機制，通過將「現實探索」與「夢境策略測試」分離，利用歷史數據構建回放模擬器來優化探索策略本身，而非僅優化代碼結果，展現了狹義但結構性的 L5 遞歸自我改進特徵。
核心重點
- Dream RSI 的核心定義：
- 全稱為 Recursive Self-Improvement Based on Evolutionary Worlds（基於演化世界的遞歸自我改進）。
- 核心在於將 AI 的「夢境」比喻為對過去探索歷史的重播與重組，而非預測未來。
- 將「實際執行代碼」與「決定下一步探索策略」分為兩層：前者在現實中生成歷史數據（探索樹），後者則在「夢境世界」（歷史數據構成的模擬器）中測試不同的探索策略。
- 與傳統方法的區別：
- 傳統自動化發現系統（如 AlphaCode、Godel Machines、AI Scientist）通常預先設計「如何探索下一步」的規則，未隨系統演化。
- Dream RSI 明確分離「執行層」與「策略層」，專注於優化「決定下一步做什麼」的層面，並通過持續增長的「發現樹」（Discovery Tree）記錄每個節點的真實嘗試結果。
- 歷史數據作為「可進入的事件」：
- Dream RSI 不訓練新的世界模型來近似現實，而是直接利用已發生的歷史記錄作為回放模擬器。
- 歷史從「日誌」轉變為「可進入的事件」（Re-enterable event）。新策略可以直接在歷史樹上「重走」，只要訪問真實發生過的節點，模擬器即可直接揭示當時的結果。
- 這種「保守的夢境」避免了將歷史壓縮為單一指令所帶來的搜索空間限制，允許不同的探索策略在歷史樹上進行比較。
- 實驗結果與動態節奏：
- 在算法工程、數學優化及 GPU 核心工程等任務中，Dream RSI 通過動態調整探索節奏（在進展順利時減少嘗試，在遇到瓶頸時擴大探索），在節省計算成本的同時提升或保持性能。
- 與「總結歷史」方法對比，Dream RSI 不預先告訴 AI 哪段經驗最重要，而是保留整個世界，允許不同策略自行探索。
- RSI 的邊界與 L5 特徵：
- Dream RSI 僅修改外層策略（如何繼續尋找更好結果的規則），底層編碼代理、模型、評估器及任務目標均不變。
- 這被定義為極度狹窄的 L5 特徵：被修改的對象包含了生成下一輪改進的方法本身（探索機制），但未修改底層模型或任務定義。
- 區分了 結構性遞歸（新策略用於下一輪線上探索）與 有效性遞歸（新策略是否讓系統改進能力變強）。
詳細大綱
一、 Dream RSI 的核心機制：從「夢境」到「現實」的循環
- 人類夢境的啟示：人類在睡眠中重播並重組白天的經驗，以發現新連結或改變決策，而非重複經歷當天。
- Dream RSI 的定義：
- 白天（現實）：系統在真實環境中寫代碼、編譯、運行測試，生成實際的探索歷史。
- 夜晚（夢境）：將過去的探索轉化為可重訪的「夢境世界」。
- 夢境中的操作：不重新生成代碼或重跑實驗，而是在已發生的歷史中嘗試不同的探索路徑（例如：改變方向、調整並行進程數量、決定何時停止）。
- 循環機制：夢境中發現的更好策略被帶回現實，產生新的探索歷史，新歷史成為新的夢境基礎。
二、 與現有系統（AlphaCode, Godel Machines, AI Scientist）的區別
- 傳統自動化發現系統的缺陷：
- 混淆了兩種工作：「執行實驗/寫代碼」與「決定下一步策略」。
- 雖然代碼和結果會迭代，但「如何探索下一步」的規則（如從哪個節點繼續、開啟多少分支、何時放棄）通常是預先設計的，並未隨系統演化。
- Dream RSI 的改進：
- 明確分離「執行層」與「策略層」。
- 專注於優化「決定下一步做什麼」的層面。
- 通過持續增長的「發現樹」（Discovery Tree）記錄每個節點的真實嘗試結果。
三、 歷史數據作為「可進入的事件」而非「日誌」
- 探索樹（Figure 2 解析）：
- 樹中的節點對應真實發生的嘗試（模型生成的解決方案、代碼是否可運行、評估器給出的結果）。
- 外層探索策略只需回答簡單問題：從哪個節點繼續？是否開啟新路徑？並行方向數量？何時停止？
- 關鍵轉變：
- 傳統方法需重新運行新策略才能評估其好壞。
- Dream RSI 允許新策略直接在歷史樹上「重走」，只要訪問真實發生過的節點，模擬器即可直接揭示當時的結果。
- 歷史從「日誌」轉變為「可進入的事件」（Re-enterable event）。
- 保守的夢境：
- Dream RSI 不訓練新的世界模型來近似現實。
- 歷史記錄本身就是回放模擬器。
- 夢境僅限於實際經歷過的區域，無法預測未探索分支的結果。
- 優勢：無需支付昂貴的在線探索成本，支持大量快速的離線策略比較。
- 限制：若第一輪現實探索質量差，歷史樹中缺乏高質量路徑，後續夢境無法憑空創造好結果。
四、 與「總結歷史」方法的對比（Figure 5 分析）
- 另一種嘗試：讓模型總結成功/失敗的方向，並將建議注入下一輪提示（Prompt）。
- 結果：性能反而惡化（無論是否使用 RSI）。
- 原因分析：
- 將歷史壓縮為單一最佳建議，等同於強加預先決定的共同方向。
- 可能過早限制搜索空間，排除非常規路徑。
- Dream RSI 的優勢：
- 不預先告訴 AI 哪段經驗最重要。
- 保留整個世界，允許不同策略自行探索。
- 核心價值不在於節省 API 調用，而在於改變隨後的實際探索過程。
五、 實驗結果與行為特徵
- 測試領域：算法工程、數學優化、GPU 核心工程（Kernel engineering）。
- 成本與性能 trade-off：
- 成本節省：在 Lowell 算法發現任務中，最大調用差距約 162 倍（註：需查證具體模型對比，Dream RSI 使用 Gemini 1.5 Flash，Simpletext 使用 GPT-40）。
- 公平比較：Dream RSI 與使用相同基礎模型的遞歸固定探索進行對比。
- 性能表現：
- 部分任務（如 VGG16, LayerNorm）：用更少的生成次數達到相似性能（知道「足夠」是什麼時候）。
- 部分任務（如 Conv, CX）：在相似預算內推高性能（知道「何時不該停止」）。
- 動態探索節奏：
- 早期性能穩步提升時：減少嘗試次數。
- 性能遇到瓶頸時：擴大探索範圍。
- 核心意義：不僅節省單個任務的計算量，更改變了「計算資源如何分配給下一次探索」。
六、 邊界定義與 RSI 的爭議
- RSI 的邊界：
- 底層編碼代理（Coding agent）不變。
- 模型不變。
- 評估器（Evaluator）未動。
- 任務目標固定。
- 僅修改外層策略：如何繼續尋找更好結果的規則。
- 誤區澄清：
- 這不是 Google AI 重寫自己的大腦（Exaggeration）。
- 也不僅僅是節省成本的調度器（Underestimation），因為調度規則本身已成為改進對象。
- 真正的 RSI 定義（L1-L5 框架回顧）：
- 性能提升不等於遞歸自我改進。
- 若自我改進的方法由外部硬編碼，則距離真正的遞歸還有一步之遙。
- AlphaDev 的例子：主要改進對象是結果本身（找到更好的答案）。
- 達爾文式代理（Darwin-inspired agents）：進一步修改編碼代理本身的實現，改進對象不再僅限於答案。
七、 Dream RSI 的 L5 特徵與兩種遞歸
- Dream RSI 的核心突破：納入選擇機制：
- 過去 RSI 論文未將「保留哪些候選解」及「下一代演化規則」納入自我修改迴圈。
- Dream RSI 將這些原本屬於系統設計師的層面（節點分支、分支數量、停止時機）轉化為可被替換和改進的對象。
- 新策略直接統治下一輪的真實探索與資源分配，形成反饋迴路：新探索歷史回饋至回放模擬器，修改下一版策略。
- Dream RSI 的 L5 特徵與局限性：
- 符合 L1-L5 框架中的 L5 特徵：被修改的對象包含了生成下一輪改進的方法本身。
- 極度狹窄的 L5：
- 不自主修改底層模型。
- 不改變評估器。
- 不重新定義任務。
- 不決定應改進哪種能力。
- 僅在固定目標、固定評估規則、固定底層代理的邊界內，對探索機制進行遞歸改進。
- 兩種遞歸類型的區分：
- 結構性遞歸 (Structural Recursion)：修改後的機制是否在下一輪中被再次使用？Dream RSI 明確符合此點（新探索策略直接進入下一輪線上探索）。
- 有效性遞歸 (Effective Recursion)：使用該機制是否使後繼系統在改進能力上變得更強？
- Dream RSI 通過使用相同的底層模型、評估器和初始策略作為固定探索基準，提供了關於有效性遞歸的清晰對比證據。
八、 與 AV 訓練 (AV training) 的對比
- 共同點：都不滿足於僅修改最終模型或答案，而是將研究過程本身作為可累積、修改和繼承的對象。
- 差異點：
- AV 訓練：類似「死後診斷」，由元代理閱讀過去證據直接診斷問題並修改策略，增加額外步驟。
- Dream RSI：類似「沙盒模擬」，生成多個候選探索策略，放入歷史回放世界實際運行，利用結果選擇進入現實的策略。
- 實驗結果：作者嘗試將歷史總結為語義指導（直接路徑），但結果較差；因此 Dream RSI 的特殊性在於讓候選策略重新體驗歷史並根據結果做選擇。
九、 「Dream」一詞的含義與未來展望
- 為何稱為 Dream：
- 並非傳統意義上訓練世界模型。
- 借鑒直覺：現實世界互動成本過高，將過去經驗轉化為內部環境以進行持續學習。
- 模擬對象不同：不學習完整的外部世界，而是保留過去探索該世界的方式及每次探索發生的情況。
- 研究焦點的轉移：從關注「世界如何變化」轉向關注「我過去如何尋求變化」以及「若使用不同搜索方式會發生什麼」。
- 當前局限與意義：
- 模型、目標、評估規則均未改變。
- 夢境受限於過去真實經歷的範圍。
- 意義：遞歸自我改進的顯著部分始於「改進方法本身」開始由機器自身修改、驗證並用於下一輪。
工具 / 模型 / 名詞整理
- Google DeepMind
- Dream RSI (論文名稱/系統名稱)
- RSI (Recursive Self-Improvement，遞歸自我改進)
- AlphaCode
- Godel Machines
- AI Scientist
- AlphaEvo
- AlphaDev
- Gemini 1.5 Flash (Dream RSI 使用的模型)
- GPT-40 (Simpletext 使用的模型，疑點：通常為 GPT-4 或 GPT-4o，需查證)
- Simpletext
- Lowell (算法發現任務名稱)
- VGG16 (實驗任務/模型)
- LayerNorm (實驗任務/組件)
- Conv (實驗任務/模型)
- CX (實驗任務/模型)
- X (社交媒體平台，原 Twitter)
- L1 to L5 (評估框架等級)
- AV 訓練 (AV training，與 Dream RSI 對比的方法)
- 元代理 (Meta-agent)
- 執行代理 (Executive agents)
- 回放模擬器 (Replay simulator)
- 歷史回放世界 (Historical replay world)
- 達爾文式代理 (Darwin-inspired agents)
操作流程整理
- 現實探索（白天）：
- 系統在真實環境中寫代碼、編譯、運行測試。
- 生成實際的探索歷史，構建「發現樹」（Discovery Tree）。
- 記錄每個節點的真實嘗試結果（模型生成的解決方案、代碼是否可運行、評估器給出的結果）。
- 夢境模擬（夜晚）：
- 將過去的探索轉化為可重訪的「夢境世界」。
- 不重新生成代碼或重跑實驗。
- 在已發生的歷史中嘗試不同的探索路徑（改變方向、調整並行進程數量、決定何時停止）。
- 新策略直接在歷史樹上「重走」，訪問真實發生過的節點以揭示當時的結果。
- 策略評估與選擇：
- 比較不同探索策略在歷史回放世界中的表現。
- 選擇表現較好的策略。
- 循環反饋：
- 夢境中發現的更好策略被帶回現實。
- 產生新的探索歷史。
- 新歷史成為新的夢境基礎，形成遞歸改進迴路。
值得注意的限制或風險
- 保守的夢境限制：
- Dream RSI 不訓練新的世界模型來近似現實，歷史記錄本身就是回放模擬器。
- 夢境僅限於實際經歷過的區域，無法預測未探索分支的結果。
- 若第一輪現實探索質量差，歷史樹中缺乏高質量路徑，後續夢境無法憑空創造好結果。
- 極度狹窄的 L5 特徵：
- 僅修改外層策略（如何繼續尋找更好結果的規則）。
- 底層編碼代理、模型、評估器及任務目標均不變。
- 不自主修改底層模型、不改變評估器、不重新定義任務、不決定應改進哪種能力。
- 與「總結歷史」方法的對比風險：
- 若將歷史壓縮為單一最佳建議並注入下一輪提示，可能過早限制搜索空間，排除非常規路徑，導致性能惡化。
逐字稿辨識疑點
- GPT-40：逐字稿中提到 "Simpletext uses GPT-40"。常見模型版本為 GPT-4 或 GPT-4o，"GPT-40" 可能為口誤或聽寫錯誤，需查證。
- Lowell 算法發現任務：逐字稿提到 "Lowell algorithm discovery task"。需確認 "Lowell" 是否為特定內部代號或拼寫錯誤（如 Lowell 可能是人名或特定項目名，但也可能是其他名稱的誤聽）。
- Conv 和 CX：在實驗結果部分提到 "Conv and CX"。需確認 "CX" 是否為特定模型或任務的縮寫，或為聽寫錯誤。
- Curil BCH：逐字稿中提到 "Curil BCH is a great example"。此名稱在常見機器學習模型中較少見，可能為 "Curie"、"BCH" 或其他模型/任務名稱的誤聽，需查證。
- Darwin-inspired agents：講者提到 "Darwin-inspired agents"，需確認這是否為論文中的特定術語或泛指。
- AVIL training：逐字稿中出現一次「AVIL training」，其餘均為「AV training」，疑為口誤或聽寫錯誤。
- JRSI：逐字稿中出現一次「JRSI」，上下文均指代
尚未產生學習筆記
請在 Telegram 指令最後加上「學習」，例如：videonote 網址 英文 雙語 學習