WEBVTT

00:00:00.000 --> 00:00:06.720
So look at this, on this day I saved 91 million tokens because of cash read, and in the past week I've saved over 300 million tokens because of it.

00:00:06.840 --> 00:00:12.680
Now don't freak out, this isn't anything that you have to go change, this is happening automatically if you are using Claude Code or Claude.

00:00:12.880 --> 00:00:17.460
And I know that the concept of prompt caching might seem a little bit overwhelming, but today I'm going to make it as simple as possible,

00:00:17.720 --> 00:00:22.420
and only really tell you what you need to know in order to make sure that you are saving your session limits and saving tokens.

00:00:22.780 --> 00:00:26.900
I'll also give you guys this entire token dashboard for free so you can actually start tracking your tokens a little bit better.

00:00:26.900 --> 00:00:31.180
Anyway, so let's talk about prompt caching, why your sessions burn out, and how to stop it.

00:00:31.300 --> 00:00:33.020
So what does caching actually cost you?

00:00:33.360 --> 00:00:37.180
Well, cached tokens only cost you 10% of normal input.

00:00:37.360 --> 00:00:39.720
So all the tokens that are getting cached are saving you a ton of money.

00:00:40.000 --> 00:00:44.560
So if we go back to this example, on this day when I had 91 million tokens cached,

00:00:45.060 --> 00:00:48.600
that costed me only as if I was processing about 9 million of those tokens.

00:00:48.820 --> 00:00:51.260
The cache window on a Cloud subscription is an hour,

00:00:51.420 --> 00:00:54.780
meaning if you're working with Cloud Code and you don't touch it for an hour,

00:00:54.780 --> 00:00:56.120
and then you send another message,

00:00:56.420 --> 00:00:58.800
everything in that session gets uncashed.

00:00:58.960 --> 00:01:01.200
So if you leave a session sitting for an hour or longer,

00:01:01.620 --> 00:01:03.180
then you're gonna pay more for it.

00:01:03.300 --> 00:01:05.400
And if you're using Claude via API or subagents,

00:01:05.500 --> 00:01:08.180
then the TTL or the time to live is only five minutes.

00:01:08.340 --> 00:01:09.160
You can change that,

00:01:09.200 --> 00:01:10.120
but it's just a little bit more expensive.

00:01:10.260 --> 00:01:11.440
You could bump it up to an hour if you want.

00:01:11.440 --> 00:01:13.560
But for Claude code inside of your terminal

00:01:13.560 --> 00:01:15.780
or your extension, whatever it is, that's an hour.

00:01:16.020 --> 00:01:17.700
And now here's a quote from Thorik from Anthropic.

00:01:18.100 --> 00:01:19.500
He said that we actually run alerts

00:01:19.500 --> 00:01:20.620
on our prompt cache hit rate

00:01:20.620 --> 00:01:22.500
and declare SEVs if they're too low.

00:01:22.640 --> 00:01:23.560
So basically them saying,

00:01:23.560 --> 00:01:25.240
we take this stuff really, really seriously.

00:01:25.780 --> 00:01:27.900
And if we see that the hit rate isn't very high

00:01:27.900 --> 00:01:30.240
for users' cloud code caching,

00:01:30.520 --> 00:01:32.180
then we do something about it immediately.

00:01:32.480 --> 00:01:33.320
And that's very nice of them,

00:01:33.380 --> 00:01:35.420
but also, of course, it benefits themselves

00:01:35.420 --> 00:01:37.440
because with a high cache hit rate,

00:01:37.800 --> 00:01:38.720
cloud code feels faster,

00:01:39.180 --> 00:01:40.420
their serving cost is lower,

00:01:40.780 --> 00:01:42.500
subscription limits feel more generous,

00:01:42.700 --> 00:01:44.000
you know, because you're using less,

00:01:44.300 --> 00:01:46.160
and long coding sessions stay practical.

00:01:46.560 --> 00:01:48.200
And then if you have low cache hit rate,

00:01:48.340 --> 00:01:49.140
this is what happens.

00:01:49.180 --> 00:01:51.220
And obviously, it's just a lose-lose for everybody.

00:01:51.220 --> 00:01:56.900
and that's why I said like prompt caching can get very very complex and if you want to check out

00:01:56.900 --> 00:02:01.960
more then I'll link this article right here which Thorik really goes into some depth here but if

00:02:01.960 --> 00:02:05.240
you read this at least when I did I was like okay this is a little bit overwhelming I have a feeling

00:02:05.240 --> 00:02:09.440
I don't actually need to know all of this but I do need to know at least a little bit at least you

00:02:09.440 --> 00:02:14.020
know the 80 20 of prompt caching so that I can get the most out of my session limits and that's

00:02:14.020 --> 00:02:18.280
what I'm going to break down today so let's take a look at an example of how this actually grows

00:02:18.280 --> 00:02:24.040
so by default when you shoot off a message to claude there's going to be some information that

00:02:24.040 --> 00:02:27.240
needs to be cached right away and actually let me just switch back to one of thorax graphics real

00:02:27.240 --> 00:02:32.220
quick you can see here that we have the base system instructions get globally cached we have

00:02:32.220 --> 00:02:37.460
tools like read write bash grab glob globally cached we have per memory or sorry per project

00:02:37.460 --> 00:02:42.280
things like claude.md and memory and that gets cached per project we've got session state and

00:02:42.280 --> 00:02:45.620
and then we have user messages which grow each turn.

00:02:45.620 --> 00:02:48.480
So now that we take this into context

00:02:48.480 --> 00:02:49.660
and we flip back over here,

00:02:49.660 --> 00:02:50.660
this is what it looks like.

00:02:50.660 --> 00:02:52.920
This is an example where we have four turns.

00:02:52.920 --> 00:02:54.140
So on turn one, there's no cache.

00:02:54.140 --> 00:02:55.940
Basically we're matching on a prefix.

00:02:55.940 --> 00:02:58.840
So don't really have to worry about what that means,

00:02:58.840 --> 00:03:00.060
but I might mention that later.

00:03:00.060 --> 00:03:02.300
So anyways, on turn one, there's nothing, right?

00:03:02.300 --> 00:03:03.840
We're opening up a fresh session.

00:03:03.840 --> 00:03:06.360
We load in the system prompt, the project context,

00:03:06.360 --> 00:03:08.160
and we shoot off our first message.

00:03:08.160 --> 00:03:11.860
And all of this is kind of in this like brown highlight border,

00:03:11.860 --> 00:03:13.860
which means that this is new

00:03:13.860 --> 00:03:15.360
and it has to be fully processed

00:03:15.360 --> 00:03:17.800
and it's being written to the cache here.

00:03:17.800 --> 00:03:19.880
So before I continue down this graphic,

00:03:19.880 --> 00:03:21.500
in this dashboard you can see

00:03:21.500 --> 00:03:23.880
that we have the difference between cache create

00:03:23.880 --> 00:03:25.100
and cache read.

00:03:25.100 --> 00:03:27.420
So on these days you can see what are my input tokens,

00:03:27.420 --> 00:03:29.580
my output tokens and my cache create.

00:03:29.580 --> 00:03:31.740
And then over here you can see my daily cache reads.

00:03:31.740 --> 00:03:33.220
And just a quick explanation,

00:03:33.220 --> 00:03:35.860
a cache create is writing something into cache

00:03:35.860 --> 00:03:36.940
for the first time.

00:03:36.940 --> 00:03:39.740
It's a one-time cost and it pays off the next turn

00:03:39.740 --> 00:03:41.340
unless of course everything gets uncached

00:03:41.340 --> 00:03:43.700
and the cache read is tokens that Claude reused

00:03:43.700 --> 00:03:45.560
from a cache like your Claude.md

00:03:45.560 --> 00:03:46.300
or some of the files

00:03:46.300 --> 00:03:48.720
or some of the global system instructions.

00:03:48.720 --> 00:03:50.900
And these are the things that are 10 times cheaper

00:03:50.900 --> 00:03:52.120
than fresh input.

00:03:52.120 --> 00:03:53.960
So anyways, on turn two,

00:03:53.960 --> 00:03:57.320
given that we're within that one hour TTL window,

00:03:57.320 --> 00:04:00.140
everything here is already in context, so it's cached.

00:04:00.140 --> 00:04:02.240
And then all that Claude actually has to process

00:04:02.240 --> 00:04:05.360
for the first time is reply one and message two,

00:04:05.360 --> 00:04:06.440
and it caches that.

00:04:06.440 --> 00:04:08.880
So then down here in turn three, all of that's cached,

00:04:08.880 --> 00:04:11.400
and we are bumping up a reply and a message

00:04:11.400 --> 00:04:13.680
and those are the things that only get processed each time.

00:04:13.840 --> 00:04:15.380
But if we waited an hour

00:04:15.380 --> 00:04:16.740
and then we sent another message

00:04:16.740 --> 00:04:18.640
or if we changed the system prompt,

00:04:18.960 --> 00:04:20.660
then everything from the very beginning

00:04:20.660 --> 00:04:22.320
has to get fully recached.

00:04:22.580 --> 00:04:25.600
So imagine if you were on message like 16

00:04:25.600 --> 00:04:27.340
and you're way, way, way over here on the right

00:04:27.340 --> 00:04:29.680
and you change the system prompt or you wait an hour,

00:04:30.340 --> 00:04:31.940
then everything getting recached

00:04:31.940 --> 00:04:34.720
is going to be a pretty expensive move that you just made.

00:04:34.960 --> 00:04:36.920
So anyways, once again, we have the system layer,

00:04:37.020 --> 00:04:38.440
the project layer and the conversation layer,

00:04:38.440 --> 00:04:43.700
the system layer has instructions, tool definitions, output style, and here's where it might break.

00:04:44.000 --> 00:04:49.480
the project level or the project layer has Cloud.md, memory, and rules, and then here's when that might break.

00:04:49.760 --> 00:04:52.980
And then we have, of course, the conversation, which is just like the replies and the messages,

00:04:53.320 --> 00:04:56.660
which gets recached every time, but that's how it should be.

00:04:56.800 --> 00:04:59.700
So here's where there's been some confusion among the community.

00:04:59.840 --> 00:05:05.500
So how long does the cache snapshot live, which is kind of called the TTL, the time to live?

00:05:05.500 --> 00:05:10.860
so on your cloud subscription you have an hour by default because it uses your subscription

00:05:10.860 --> 00:05:15.580
but if you go over that weekly limit and you are now playing in your extra usage territory where

00:05:15.580 --> 00:05:21.860
you are paying per token api then by default that will be five minutes which is very dangerous if

00:05:21.860 --> 00:05:25.960
you're managing multiple sessions and you're constantly recaching everything because five

00:05:25.960 --> 00:05:29.380
minutes is passing you got to be careful about that and people were kind of suspicious i don't

00:05:29.380 --> 00:05:32.880
know if you remember like a month or so ago when everyone was complaining about their cloud

00:05:32.880 --> 00:05:38.080
subscriptions, how quick they were eating it up. People thought maybe that they switched the cache

00:05:38.080 --> 00:05:42.860
TTL from an hour to five minutes without like saying anything to anybody. It turns out they

00:05:42.860 --> 00:05:47.340
didn't. So it is an hour, but that's just like, you know, there was a lot of confusion around that.

00:05:47.340 --> 00:05:52.360
And I get why, because honestly, it's not super clear. Like if you're on an API, you have five

00:05:52.360 --> 00:05:57.040
minutes by default, but you can increase the cost and you can do an hour. And then your sub agents

00:05:57.040 --> 00:06:00.740
on any plan are going to be five minutes. And for some reason, all of this is documented about

00:06:00.740 --> 00:06:03.080
ClawedCode and the API, which are two very different things.

00:06:03.520 --> 00:06:07.860
But the Clawed.ai, like on the web, we don't know exactly how that works.

00:06:08.160 --> 00:06:10.460
At least I haven't found documentation on that exact.

00:06:10.740 --> 00:06:15.280
I'm assuming it's the same as your subscription, but I don't know 100% for fact.

00:06:15.500 --> 00:06:18.220
Anyways, three habits that cover 95% of people.

00:06:18.700 --> 00:06:19.540
Don't pause too long.

00:06:19.540 --> 00:06:24.900
So if you've gone over an hour on a session, just hand it off to a new session.

00:06:25.120 --> 00:06:26.920
Obviously, start fresh when you switch tasks.

00:06:27.280 --> 00:06:30.340
So do a slash compact, which will break the cache, or do a slash clear.

00:06:30.340 --> 00:06:32.340
或是用我的SessionHandoffSkill

00:06:32.340 --> 00:06:34.340
我會加上的作品

00:06:34.340 --> 00:06:36.340
所以我會加上的TokenDashboard

00:06:36.340 --> 00:06:37.340
GithubRepo和這個Skill

00:06:37.340 --> 00:06:39.340
會在我的FreeschoolCommunity

00:06:39.340 --> 00:06:42.340
但基本上,我會說

00:06:42.340 --> 00:06:43.340
我會有這個Project

00:06:43.340 --> 00:06:45.340
幫助我建立這個HTML

00:06:45.340 --> 00:06:47.340
它們有205,000TOKENS

00:06:47.340 --> 00:06:49.340
如果我來做SessionHandoff

00:06:49.340 --> 00:06:51.340
這基本上整理我們做的

00:06:51.340 --> 00:06:53.340
所有的重要招待

00:06:53.340 --> 00:06:54.340
所有的開招待

00:06:54.340 --> 00:06:55.340
所有的開招待

00:06:55.340 --> 00:06:58.340
然後我可以直接

00:06:58.340 --> 00:07:03.540
that summary do a slash clear and then keep going and it feels like i haven't actually lost anything

00:07:03.540 --> 00:07:07.780
so that has been basically my replacement for doing slash compact i've just enjoyed doing this

00:07:07.780 --> 00:07:11.540
better and sometimes the compact takes a long time this typically doesn't take anywhere over

00:07:11.540 --> 00:07:16.260
a minute there you go so that is my session handoff i do a slash copy and then i just go

00:07:16.260 --> 00:07:20.420
ahead and clear that paste it in hit enter and now i'm basically right back where i was and then

00:07:20.420 --> 00:07:24.660
this last one is for if you're using claude chat specifically if you're going to be pasting big

00:07:24.660 --> 00:07:28.180
documents in there you're probably better off doing a project because like i said i don't

00:07:28.180 --> 00:07:31.620
I don't know exactly how the caching works in Cloud Chat,

00:07:31.620 --> 00:07:35.180
but we do have some confidence in saying that projects,

00:07:35.180 --> 00:07:37.220
those files are cached a little bit differently

00:07:37.220 --> 00:07:40.260
and probably more optimized for storing a bunch of documents

00:07:40.260 --> 00:07:42.260
compared to just dropping them into your Cloud Chat.

00:07:42.645 --> 00:07:43.745
So keep it alive, keep it focused,

00:07:43.745 --> 00:07:45.685
and start fresh when you switch.

00:07:45.685 --> 00:07:46.805
Now there's a few other things

00:07:46.805 --> 00:07:48.505
that were a little bit confusing to me

00:07:48.505 --> 00:07:50.885
as far as like what breaks the cache.

00:07:50.885 --> 00:07:53.185
So the first one is if you switch the model.

00:07:53.185 --> 00:07:55.205
So you know if you're in here

00:07:55.205 --> 00:07:56.445
and you're talking to Cloud,

00:07:56.445 --> 00:07:57.645
hello,hello,hello

00:07:57.645 --> 00:07:59.325
and then you go in here and you do a slash model

00:07:59.325 --> 00:08:00.825
and you actually switch the model

00:08:00.825 --> 00:08:02.325
that's going to recache everything

00:08:02.325 --> 00:08:03.405
because if you remember earlier

00:08:03.405 --> 00:08:04.565
I said it's prefix matching

00:08:04.565 --> 00:08:06.165
which I'm not going to dive into right now

00:08:06.165 --> 00:08:07.665
but if you switch the model

00:08:07.665 --> 00:08:09.965
then you are switching essentially the prefix

00:08:09.965 --> 00:08:11.825
and it can't match on that same cache

00:08:11.825 --> 00:08:13.205
so if you switch the model

00:08:13.205 --> 00:08:15.065
you are recaching everything

00:08:15.065 --> 00:08:16.585
now I do want to apologize for something here

00:08:16.585 --> 00:08:19.725
because if you do model opus plan

00:08:19.725 --> 00:08:21.165
which is something that I've shown before

00:08:21.165 --> 00:08:23.265
in like token hacks videos

00:08:23.265 --> 00:08:25.385
this basically means it uses opus for plan mode

00:08:25.385 --> 00:08:28.525
and then it switches to Sonnet for the execution.

00:08:28.525 --> 00:08:30.365
But if you do that, just keep in mind,

00:08:30.365 --> 00:08:31.545
that's actually gonna break the cache

00:08:31.545 --> 00:08:33.505
because you're switching model halfway through.

00:08:33.505 --> 00:08:35.925
So right here, you can see each model has its own cache.

00:08:35.925 --> 00:08:38.065
Switching with model means the next request

00:08:38.065 --> 00:08:40.705
reads the entire conversation history with no cache hits,

00:08:40.705 --> 00:08:42.485
even though the context is identical.

00:08:42.485 --> 00:08:44.705
The Opus plan model setting resolves to Opus

00:08:44.705 --> 00:08:46.585
during plan mode and Sonnet during execution.

00:08:46.585 --> 00:08:48.645
So each plan toggle is a model switch

00:08:48.645 --> 00:08:50.305
and starts a fresh cache.

00:08:50.305 --> 00:08:52.845
So it's very interesting because typically the point of that

00:08:52.845 --> 00:08:54.025
is to save your session limit.

00:08:54.025 --> 00:08:55.985
and I think ultimately in the long run it does,

00:08:55.985 --> 00:08:57.105
but it is important to understand

00:08:57.105 --> 00:08:59.265
that doing that does reset the cache.

00:08:59.265 --> 00:09:02.745
Now what you can do is you can edit your Cloud.MD

00:09:02.745 --> 00:09:03.745
and you can do that mid-session

00:09:03.745 --> 00:09:05.465
because the edit actually doesn't apply

00:09:05.465 --> 00:09:08.825
until you restart that session, so the cache stays safe.

00:09:08.825 --> 00:09:11.105
And then of course the Cloud.AI projects caching,

00:09:11.105 --> 00:09:12.345
it's not exactly documented,

00:09:12.345 --> 00:09:15.145
but pretty confident that it does help to drop docs

00:09:15.145 --> 00:09:16.705
in projects rather than in the chat.

00:09:16.705 --> 00:09:18.785
But anyways, this token dashboard, like I said,

00:09:18.785 --> 00:09:21.945
is very helpful to just be able to understand,

00:09:21.945 --> 00:09:24.025
get a little bit more visibility into your tokens.

00:09:24.025 --> 00:09:27.465
这个组织的组织的组织

00:09:27.465 --> 00:09:29.465
如果你转过去的电话

00:09:29.465 --> 00:09:31.225
你的电话会会更改变

00:09:31.225 --> 00:09:33.625
比较多于你的主要是用的

00:09:33.625 --> 00:09:34.585
很简单的组织

00:09:34.585 --> 00:09:35.665
它是一组织的

00:09:35.665 --> 00:09:36.985
你会在我的社会里面的

00:09:36.985 --> 00:09:37.865
在这里的组织

00:09:37.865 --> 00:09:38.825
你会在教授

00:09:38.825 --> 00:09:40.665
你会在教授的组织

00:09:40.665 --> 00:09:42.425
然后你会在那里的组织

00:09:42.425 --> 00:09:43.625
然后你会在那里的组织

00:09:43.625 --> 00:09:44.265
你会在这些织织的织织

00:09:44.265 --> 00:09:45.465
就要给你一个组织的织织

00:09:45.465 --> 00:09:47.465
说说这个是一织织的织织

00:09:47.465 --> 00:09:48.985
就像这个织织织的织织

00:09:48.985 --> 00:09:50.425
就像它们的织织织

00:09:50.425 --> 00:09:52.705
and it will pull in all of your past sessions.

00:09:52.805 --> 00:09:54.405
So it's not like you're gonna start fresh

00:09:54.405 --> 00:09:57.125
as soon as you link in this repo,

00:09:57.265 --> 00:09:58.885
it will read your past files

00:09:58.885 --> 00:09:59.985
and it will pull in your tokens.

00:10:00.305 --> 00:10:01.585
And then of course, I will also include

00:10:01.585 --> 00:10:02.645
that session handoff skill

00:10:02.645 --> 00:10:03.985
that I just mentioned to you guys.

00:10:04.045 --> 00:10:05.125
So I know this one was super quick.

00:10:05.505 --> 00:10:06.485
Hopefully this one was helpful though.

00:10:06.785 --> 00:10:07.925
It's just important.

00:10:08.105 --> 00:10:10.685
Like I said, when I hear about stuff like this,

00:10:10.685 --> 00:10:12.225
I love to understand it to the point

00:10:12.225 --> 00:10:13.225
where I know how to use it

00:10:13.225 --> 00:10:14.625
and I know what's going on under the hood.

00:10:14.965 --> 00:10:17.565
But truthfully, if I looked at some of these other articles,

00:10:17.785 --> 00:10:19.545
like how in-depth they go

00:10:19.545 --> 00:10:21.385
and how much nuance there is.

00:10:21.625 --> 00:10:22.725
Most of the stuff right now,

00:10:22.805 --> 00:10:23.985
I just don't need to know

00:10:23.985 --> 00:10:26.585
because I'm not using the API in this way super heavily.

00:10:26.765 --> 00:10:28.165
So the reason I wanted to throw that out there

00:10:28.165 --> 00:10:29.665
is because it's important to stay updated

00:10:29.665 --> 00:10:30.605
and follow things,

00:10:30.845 --> 00:10:31.705
but just understand

00:10:31.705 --> 00:10:34.465
what do you really need to know at its core.

00:10:34.705 --> 00:10:35.585
So if you guys enjoyed the video

00:10:35.585 --> 00:10:36.265
or you learned something new,

00:10:36.505 --> 00:10:37.025
please give it a like,

00:10:37.065 --> 00:10:37.705
it helps me out a ton.

00:10:37.925 --> 00:10:38.485
And as always,

00:10:38.545 --> 00:10:40.205
I appreciate you guys making it to the end of the video

00:10:40.205 --> 00:10:41.405
and I'll see you on the next one.

00:10:42.005 --> 00:10:42.365
Thanks guys.