So look at this, on this day I saved 91 million tokens because of cash read, and in the past week I've saved over 300 million tokens because of it.
Now don't freak out, this isn't anything that you have to go change, this is happening automatically if you are using Claude Code or Claude.
And I know that the concept of prompt caching might seem a little bit overwhelming, but today I'm going to make it as simple as possible,
and only really tell you what you need to know in order to make sure that you are saving your session limits and saving tokens.
I'll also give you guys this entire token dashboard for free so you can actually start tracking your tokens a little bit better.
Anyway, so let's talk about prompt caching, why your sessions burn out, and how to stop it.
So what does caching actually cost you?
Well, cached tokens only cost you 10% of normal input.
So all the tokens that are getting cached are saving you a ton of money.
So if we go back to this example, on this day when I had 91 million tokens cached,
that costed me only as if I was processing about 9 million of those tokens.
The cache window on a Cloud subscription is an hour,
meaning if you're working with Cloud Code and you don't touch it for an hour,
and then you send another message,
everything in that session gets uncashed.
So if you leave a session sitting for an hour or longer,
then you're gonna pay more for it.
And if you're using Claude via API or subagents,
then the TTL or the time to live is only five minutes.
You can change that,
but it's just a little bit more expensive.
You could bump it up to an hour if you want.
But for Claude code inside of your terminal
or your extension, whatever it is, that's an hour.
And now here's a quote from Thorik from Anthropic.
He said that we actually run alerts
on our prompt cache hit rate
and declare SEVs if they're too low.
So basically them saying,
we take this stuff really, really seriously.
And if we see that the hit rate isn't very high
for users' cloud code caching,
then we do something about it immediately.
And that's very nice of them,
but also, of course, it benefits themselves
because with a high cache hit rate,
cloud code feels faster,
their serving cost is lower,
subscription limits feel more generous,
you know, because you're using less,
and long coding sessions stay practical.
And then if you have low cache hit rate,
this is what happens.
And obviously, it's just a lose-lose for everybody.
and that's why I said like prompt caching can get very very complex and if you want to check out
more then I'll link this article right here which Thorik really goes into some depth here but if
you read this at least when I did I was like okay this is a little bit overwhelming I have a feeling
I don't actually need to know all of this but I do need to know at least a little bit at least you
know the 80 20 of prompt caching so that I can get the most out of my session limits and that's
what I'm going to break down today so let's take a look at an example of how this actually grows
so by default when you shoot off a message to claude there's going to be some information that
needs to be cached right away and actually let me just switch back to one of thorax graphics real
quick you can see here that we have the base system instructions get globally cached we have
tools like read write bash grab glob globally cached we have per memory or sorry per project
things like claude.md and memory and that gets cached per project we've got session state and
and then we have user messages which grow each turn.
So now that we take this into context
and we flip back over here,
this is what it looks like.
This is an example where we have four turns.
So on turn one, there's no cache.
Basically we're matching on a prefix.
So don't really have to worry about what that means,
but I might mention that later.
So anyways, on turn one, there's nothing, right?
We're opening up a fresh session.
We load in the system prompt, the project context,
and we shoot off our first message.
And all of this is kind of in this like brown highlight border,
which means that this is new
and it has to be fully processed
and it's being written to the cache here.
So before I continue down this graphic,
in this dashboard you can see
that we have the difference between cache create
and cache read.
So on these days you can see what are my input tokens,
my output tokens and my cache create.
And then over here you can see my daily cache reads.
And just a quick explanation,
a cache create is writing something into cache
for the first time.
It's a one-time cost and it pays off the next turn
unless of course everything gets uncached
and the cache read is tokens that Claude reused
from a cache like your Claude.md
or some of the files
or some of the global system instructions.
And these are the things that are 10 times cheaper
than fresh input.
So anyways, on turn two,
given that we're within that one hour TTL window,
everything here is already in context, so it's cached.
And then all that Claude actually has to process
for the first time is reply one and message two,
and it caches that.
So then down here in turn three, all of that's cached,
and we are bumping up a reply and a message
and those are the things that only get processed each time.
But if we waited an hour
and then we sent another message
or if we changed the system prompt,
then everything from the very beginning
has to get fully recached.
So imagine if you were on message like 16
and you're way, way, way over here on the right
and you change the system prompt or you wait an hour,
then everything getting recached
is going to be a pretty expensive move that you just made.
So anyways, once again, we have the system layer,
the project layer and the conversation layer,
the system layer has instructions, tool definitions, output style, and here's where it might break.
the project level or the project layer has Cloud.md, memory, and rules, and then here's when that might break.
And then we have, of course, the conversation, which is just like the replies and the messages,
which gets recached every time, but that's how it should be.
So here's where there's been some confusion among the community.
So how long does the cache snapshot live, which is kind of called the TTL, the time to live?
so on your cloud subscription you have an hour by default because it uses your subscription
but if you go over that weekly limit and you are now playing in your extra usage territory where
you are paying per token api then by default that will be five minutes which is very dangerous if
you're managing multiple sessions and you're constantly recaching everything because five
minutes is passing you got to be careful about that and people were kind of suspicious i don't
know if you remember like a month or so ago when everyone was complaining about their cloud
subscriptions, how quick they were eating it up. People thought maybe that they switched the cache
TTL from an hour to five minutes without like saying anything to anybody. It turns out they
didn't. So it is an hour, but that's just like, you know, there was a lot of confusion around that.
And I get why, because honestly, it's not super clear. Like if you're on an API, you have five
minutes by default, but you can increase the cost and you can do an hour. And then your sub agents
on any plan are going to be five minutes. And for some reason, all of this is documented about
ClawedCode and the API, which are two very different things.
But the Clawed.ai, like on the web, we don't know exactly how that works.
At least I haven't found documentation on that exact.
I'm assuming it's the same as your subscription, but I don't know 100% for fact.
Anyways, three habits that cover 95% of people.
Don't pause too long.
So if you've gone over an hour on a session, just hand it off to a new session.
Obviously, start fresh when you switch tasks.
So do a slash compact, which will break the cache, or do a slash clear.
或是用我的SessionHandoffSkill
我會加上的作品
所以我會加上的TokenDashboard
GithubRepo和這個Skill
會在我的FreeschoolCommunity
但基本上,我會說
我會有這個Project
幫助我建立這個HTML
它們有205,000TOKENS
如果我來做SessionHandoff
這基本上整理我們做的
所有的重要招待
所有的開招待
所有的開招待
然後我可以直接
that summary do a slash clear and then keep going and it feels like i haven't actually lost anything
so that has been basically my replacement for doing slash compact i've just enjoyed doing this
better and sometimes the compact takes a long time this typically doesn't take anywhere over
a minute there you go so that is my session handoff i do a slash copy and then i just go
ahead and clear that paste it in hit enter and now i'm basically right back where i was and then
this last one is for if you're using claude chat specifically if you're going to be pasting big
documents in there you're probably better off doing a project because like i said i don't
I don't know exactly how the caching works in Cloud Chat,
but we do have some confidence in saying that projects,
those files are cached a little bit differently
and probably more optimized for storing a bunch of documents
compared to just dropping them into your Cloud Chat.
So keep it alive, keep it focused,
and start fresh when you switch.
Now there's a few other things
that were a little bit confusing to me
as far as like what breaks the cache.
So the first one is if you switch the model.
So you know if you're in here
and you're talking to Cloud,
hello,hello,hello
and then you go in here and you do a slash model
and you actually switch the model
that's going to recache everything
because if you remember earlier
I said it's prefix matching
which I'm not going to dive into right now
but if you switch the model
then you are switching essentially the prefix
and it can't match on that same cache
so if you switch the model
you are recaching everything
now I do want to apologize for something here
because if you do model opus plan
which is something that I've shown before
in like token hacks videos
this basically means it uses opus for plan mode
and then it switches to Sonnet for the execution.
But if you do that, just keep in mind,
that's actually gonna break the cache
because you're switching model halfway through.
So right here, you can see each model has its own cache.
Switching with model means the next request
reads the entire conversation history with no cache hits,
even though the context is identical.
The Opus plan model setting resolves to Opus
during plan mode and Sonnet during execution.
So each plan toggle is a model switch
and starts a fresh cache.
So it's very interesting because typically the point of that
is to save your session limit.
and I think ultimately in the long run it does,
but it is important to understand
that doing that does reset the cache.
Now what you can do is you can edit your Cloud.MD
and you can do that mid-session
because the edit actually doesn't apply
until you restart that session, so the cache stays safe.
And then of course the Cloud.AI projects caching,
it's not exactly documented,
but pretty confident that it does help to drop docs
in projects rather than in the chat.
But anyways, this token dashboard, like I said,
is very helpful to just be able to understand,
get a little bit more visibility into your tokens.
这个组织的组织的组织
如果你转过去的电话
你的电话会会更改变
比较多于你的主要是用的
很简单的组织
它是一组织的
你会在我的社会里面的
在这里的组织
你会在教授
你会在教授的组织
然后你会在那里的组织
然后你会在那里的组织
你会在这些织织的织织
就要给你一个组织的织织
说说这个是一织织的织织
就像这个织织织的织织
就像它们的织织织
and it will pull in all of your past sessions.
So it's not like you're gonna start fresh
as soon as you link in this repo,
it will read your past files
and it will pull in your tokens.
And then of course, I will also include
that session handoff skill
that I just mentioned to you guys.
So I know this one was super quick.
Hopefully this one was helpful though.
It's just important.
Like I said, when I hear about stuff like this,
I love to understand it to the point
where I know how to use it
and I know what's going on under the hood.
But truthfully, if I looked at some of these other articles,
like how in-depth they go
and how much nuance there is.
Most of the stuff right now,
I just don't need to know
because I'm not using the API in this way super heavily.
So the reason I wanted to throw that out there
is because it's important to stay updated
and follow things,
but just understand
what do you really need to know at its core.
So if you guys enjoyed the video
or you learned something new,
please give it a like,
it helps me out a ton.
And as always,
I appreciate you guys making it to the end of the video
and I'll see you on the next one.
Thanks guys.