Asks the provider to cache the start of the request on its side, so a later request with the same start reads it from the cache at a lower price and latency (RFC 0008). Set it with CompletionOptions.withPromptCache.
This is not org.llm4s.llmconnect.caching.CachingLLMClient, which keeps whole answers on the client and returns one without calling the provider. A prompt cache still calls the provider every time and changes what the call costs, not what it returns.
Anthropic is sent cache_control breakpoints: on the last system block, the last tool and, for PromptCache.Scope.Conversation, the conversation so far, each with PromptCache.Ttl. Bedrock Converse is sent cachePoint blocks in the same three places, for the Claude and Amazon Nova models that take them. OpenAI, Gemini and Vertex AI cache on their own and report it NotNeeded; other providers report it Dropped. A prefix shorter than the model's minimum cacheable length is not cached and the provider says nothing, so when a request asked for a prompt cache and nothing was read from or written to it, the completion reports Dropped.
'''Warm the cache before a fan-out.''' The cache is written when a response starts, so requests sent together all miss it and each pays to write it. Send one request first and wait for it, then send the rest.
@Stable, not @Experimental as RFC 0008 first wrote it: the @Stable CompletionOptions exposes it, as it exposes Thinking (decisions 14 and 16).
Value parameters
- scope
-
what to cache: the system prompt, the tools, the conversation so far
- ttl
-
how long the provider keeps it
Attributes
- Companion
- object
- Source
- PromptCache.scala
- Graph
-
- Supertypes
