PromptCache

org.llm4s.llmconnect.model.PromptCache
See thePromptCache companion object
@Stable
final case class PromptCache

Asks the provider to cache the start of the request on its side, so a later request with the same start reads it from the cache at a lower price and latency (RFC 0008). Set it with CompletionOptions.withPromptCache.

This is not org.llm4s.llmconnect.caching.CachingLLMClient, which keeps whole answers on the client and returns one without calling the provider. A prompt cache still calls the provider every time and changes what the call costs, not what it returns.

Anthropic is sent cache_control breakpoints: on the last system block, the last tool and, for PromptCache.Scope.Conversation, the conversation so far, each with PromptCache.Ttl. Bedrock Converse is sent cachePoint blocks in the same three places, for the Claude and Amazon Nova models that take them. OpenAI, Gemini and Vertex AI cache on their own and report it NotNeeded; other providers report it Dropped. A prefix shorter than the model's minimum cacheable length is not cached and the provider says nothing, so when a request asked for a prompt cache and nothing was read from or written to it, the completion reports Dropped.

'''Warm the cache before a fan-out.''' The cache is written when a response starts, so requests sent together all miss it and each pays to write it. Send one request first and wait for it, then send the rest.

@Stable, not @Experimental as RFC 0008 first wrote it: the @Stable CompletionOptions exposes it, as it exposes Thinking (decisions 14 and 16).

Value parameters

scope

what to cache: the system prompt, the tools, the conversation so far

ttl

how long the provider keeps it

Attributes

Companion
object
Source
PromptCache.scala
Graph
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

Members list

Value members

Concrete methods

def withScope(scope: Set[Scope]): PromptCache

Sets what to cache.

Sets what to cache.

Attributes

Source
PromptCache.scala
def withTtl(ttl: Ttl): PromptCache

Sets how long the provider keeps it.

Sets how long the provider keeps it.

Attributes

Source
PromptCache.scala

Inherited methods

Attributes

Inherited from:
Product

Attributes

Inherited from:
Product