# LLM4S 0.5.1 documentation
> The getting-started pages, the user guide, the cookbook and the Java and Kotlin guides of https://llm4s.org, release 0.5.1, in navigation order.
A Scala block under a page marked `not-compile-checked` is followed by `(not compile-checked)`: other blocks are compiled against the release.
---
Source: https://llm4s.org/getting-started/
# Getting Started
Get up and running with LLM4S in minutes.
This section will guide you through:
1. **[Installation](installation)** - Set up LLM4S in your project
2. **[First Example](first-example)** - Write your first LLM-powered program
3. **[Configuration](configuration)** - Configure providers and API keys
4. **[Next Steps](next-steps)** - Choose your learning path
5. **[Ollama Quick Start](ollama-quickstart)** - Local LLM development (no API keys needed)
6. **[Gradle Integration](gradle)** - Use LLM4S from a Gradle (Java or Kotlin) build
7. **[Scala CLI Quick Start](scala-cli)** - A first LLM call from a single file, with no sbt project
8. **[Maven Quick Start](maven)** - A complete `pom.xml` that runs a first call from Java with `llm4s-java-api`
9. **[JBang Quick Start](jbang)** - A first LLM call from one Java file, with no build tool
10. **[Mill Quick Start](mill)** - A `build.mill` project with a first call and a test that needs no API key
11. **From Java, Spring Boot or Kotlin** - the [Java guide](/guide/java), the [Spring Boot starter](/guide/spring-boot)
and [Kotlin coroutines](installation#for-kotlin-coroutines-and-flow) (`llm4s-kotlin-api`
0.5.1 is not on Maven Central yet; that section says how to build it)
12. **[First-Run Troubleshooting](troubleshooting)** - Symptoms, causes and fixes for the problems a first run most often hits
13. **[Using LLM4S from Scala 2.13](scala-2-13)** - Call LLM4S from a Scala 2.13 project through the TASTy reader
The current release is **0.5.1**, for Scala 3.7.x on JDK 21.
## Quick Start
The fastest way to get started:
```bash
# 1. Use the starter template, on the current release
sbt new llm4s/llm4s.g8 --llm4s_version=0.5.1
# 2. Set the API key - llm4s-openai reads it for the template's openai-main section
export OPENAI_API_KEY=sk-...
# 3. Run your first program
sbt run
```
Check that the generated `build.sbt` lists a provider module (such as `llm4s-openai`) next to `llm4s-core`: since
0.5.0 `llm4s-core` ships none. The template update for 0.5.x ([llm4s.g8#6](https://github.com/llm4s/llm4s.g8/pull/6))
adds `llm4s-ollama` and `llm4s-openai` and defaults to a local Ollama section (choose OpenAI with
`LLM4S_PROVIDER=openai-main`); an older template lists `llm4s-core` only, so add the provider yourself
([Installation](installation#quick-start-with-the-starter-kit) shows how).
llm4s reads providers from named sections in `application.conf`, not from variables of its own;
see [Configuration](configuration#named-provider-sections) to add or switch providers.
## What You'll Learn
By the end of this section, you'll:
- ✅ Have LLM4S installed and configured
- ✅ Understand Result-based error handling
- ✅ Know how to make basic LLM calls
- ✅ Be able to configure multiple providers
- ✅ Know where to go next based on your goals
## Prerequisites
Before starting, you should have:
- **JDK 21**
- **Scala 3.7.x** (llm4s is built with 3.7.1)
- **sbt 1.10.6+**
- An API key from OpenAI, Anthropic, Azure OpenAI or another provider, or [Ollama](ollama-quickstart) installed
## Time to Complete
- **Quick Start**: ~5 minutes (with starter kit)
- **Full Tutorial**: ~30 minutes
## Need Help?
If you get stuck:
- Check the [troubleshooting guides](configuration#troubleshooting)
- Ask in [Discord](https://discord.gg/4uvTPn6qww)
- Browse [examples](/examples/)
---
**Ready?** [Start with installation →](installation)
---
Source: https://llm4s.org/getting-started/installation
# Installation
Get LLM4S up and running in minutes.
---
## Prerequisites
Before installing LLM4S, ensure you have:
- **Java Development Kit (JDK) 21**
- **Scala 3.7.x** (llm4s 0.5.1 is built with 3.7.1)
- Scala 2.13: call-only, via the TASTy reader; see [Using LLM4S from Scala 2.13](scala-2-13)
- **sbt 1.10.6** or higher
- An API key from at least one LLM provider (OpenAI, Anthropic, Azure OpenAI, ...), or a local [Ollama](ollama-quickstart) server, which needs none
### Verify Prerequisites
```bash
# Check Java version
java -version # Should show 21
# Check Scala version
scala -version # 3.7.1
# Check SBT version
sbt version # 1.10.6 or higher
```
---
## Add LLM4S to Your Project
> **Artifact coordinates changed in 0.4.0.** Every published module now carries an
> `llm4s-` prefix (`core` → `llm4s-core`, `workspaceClient` → `llm4s-workspace-client`,
> and so on). Releases up to and including `0.3.4` remain available under the old names.
> See the [migration guide](/reference/migration#artifact-coordinate-rename-v040) for the
> full old → new table.
### SBT
Add LLM4S to your `build.sbt`:
```scala
// Scala 3
ThisBuild / scalaVersion := "3.7.1"
libraryDependencies ++= Seq(
"org.llm4s" %% "llm4s-core" % "0.5.1",
"org.llm4s" %% "llm4s-openai" % "0.5.1" // or the provider module you use
)
```
`llm4s-core` holds no provider client: add the module of each provider you call (`llm4s-openai`,
`llm4s-anthropic`, `llm4s-gemini`, `llm4s-ollama`, ...; see [Optional Dependencies](#optional-dependencies)).
Without one, loading a provider section fails because no module registers that provider. The Maven and Gradle
snippets below show `llm4s-core` only; add the provider module the same way, with the `_3` suffix
(`llm4s-openai_3`).
### Maven
```xml
org.llm4s
llm4s-core_3
0.5.1
```
For Java, the [Maven quick start](maven) has a complete `pom.xml` for `llm4s-java-api_3`.
### Gradle (Kotlin DSL)
Gradle does **not** resolve Scala cross-version suffixes automatically — you must append the `_3` suffix explicitly (LLM4S is Scala 3 only):
```kotlin
// build.gradle.kts
repositories { mavenCentral() }
dependencies {
implementation("org.llm4s:llm4s-core_3:0.5.1")
}
// Pin the Scala 3 library (scala3-library_3 only; scala-library stays at 2.13.x)
configurations.all {
resolutionStrategy.eachDependency {
if (requested.group == "org.scala-lang" && requested.name == "scala3-library_3") {
useVersion("3.7.1")
}
}
}
```
### Gradle (Groovy DSL)
```groovy
// build.gradle
repositories { mavenCentral() }
dependencies {
implementation 'org.llm4s:llm4s-core_3:0.5.1'
}
configurations.all {
resolutionStrategy.eachDependency { details ->
if (details.requested.group == 'org.scala-lang' && details.requested.name == 'scala3-library_3') {
details.useVersion '3.7.1'
}
}
}
```
> **Using Spring Boot, Ktor, or encountering dependency conflicts?**
> See the full [Gradle integration guide](gradle) for dependency exclusion recipes
> and the [dependency conflicts reference](/reference/dependency-conflicts).
### Scala CLI
No build at all: declare the dependency in the file.
```scala
//> using scala 3.7.1
//> using jvm 21
//> using dep org.llm4s::llm4s-core:0.5.1
//> using dep org.llm4s::llm4s-openai:0.5.1
```
Add the provider module you use next to `llm4s-core` (`llm4s-openai` above). The
[Scala CLI quick start](scala-cli) has a complete file that makes a first call.
### Mill
In `build.mill` (Mill 1.x), `::` adds the Scala suffix as `%%` does in sbt:
```scala
val llm4sVersion = "0.5.1"
object app extends ScalaModule {
def scalaVersion = "3.7.1"
def mvnDeps = Seq(
mvn"org.llm4s::llm4s-core:$llm4sVersion",
mvn"org.llm4s::llm4s-openai:$llm4sVersion"
)
}
```
The [Mill quick start](mill) has a complete project: a first call, its configuration, and a test that needs no API key.
### Multi-Module Project
If you have a multi-module project:
```scala
lazy val myProject = (project in file("."))
.settings(
name := "my-llm-project",
scalaVersion := "3.7.1",
libraryDependencies ++= Seq(
"org.llm4s" %% "llm4s-core" % "0.5.1",
"org.llm4s" %% "llm4s-openai" % "0.5.1"
)
)
```
### Align versions with the BOM
A project that uses several llm4s modules (a provider, `llm4s-rag`, `llm4s-observability`) can state the
version once, in the bill of materials `llm4s-bom`, and leave it off every module. The BOM contains no code;
it supplies the versions released together. Explicit overrides and dependency conflict resolution can still
select different versions, so check the resolved dependency graph.
Maven and Gradle have no notion of Scala cross-versioning, so the modules are written with the Scala binary
suffix (`llm4s-core_3`), and the BOM itself has none (`llm4s-bom`).
**Maven** (import scope):
```xml
org.llm4s
llm4s-bom
0.5.1
pom
import
org.llm4s
llm4s-core_3
org.llm4s
llm4s-openai_3
```
**Gradle** (Kotlin DSL; in Groovy write `implementation platform("org.llm4s:llm4s-bom:...")`):
```kotlin
dependencies {
implementation(platform("org.llm4s:llm4s-bom:0.5.1"))
implementation("org.llm4s:llm4s-core_3")
implementation("org.llm4s:llm4s-openai_3")
}
```
Gradle `platform(...)` treats the BOM versions as recommendations that other dependencies can upgrade.
For strict version pinning in an application, use `enforcedPlatform(...)` instead. Publishing libraries
should generally keep `platform(...)` to avoid forcing their versions on consumers. See the
[Gradle platform documentation](https://docs.gradle.org/current/userguide/platforms.html#sub:bom_import).
**Mill** imports it with `bomMvnDeps`; the [Mill quick start](mill#aligning-versions-with-llm4s-bom) shows how.
**sbt** has no BOM import, so the BOM is optional there: keep one `val` for the version, as the sections
below do (`%%` adds the `_3` suffix itself):
```scala
val llm4sVersion = "0.5.1"
libraryDependencies ++= Seq(
"org.llm4s" %% "llm4s-core" % llm4sVersion,
"org.llm4s" %% "llm4s-openai" % llm4sVersion
)
```
### Unreleased changes (`main`)
No snapshots are published: releases are cut from tags only, and nothing publishes between them. To try
what is on `main`, build it and publish it to your local repository:
```bash
git clone https://github.com/llm4s/llm4s && cd llm4s
sbt publishLocal # for sbt; use `sbt publishM2` for Maven or Gradle (mavenLocal())
sbt version # prints the version it published, for example 0.5.1+12-abc12345-SNAPSHOT
```
Then depend on that version, for example
`libraryDependencies += "org.llm4s" %% "llm4s-core" % "0.5.1+12-abc12345-SNAPSHOT"` (use the one `sbt version`
printed). This guide describes the latest release; `main` may have modules and configuration that it does not
have yet.
---
## Quick Start with the Starter Kit
The fastest way to a working sbt project is the **llm4s.g8** template:
```bash
sbt new llm4s/llm4s.g8 --llm4s_version=0.5.1
# Answer the remaining prompts (name, package, scala_version, ...) or accept the defaults
cd
sbt run
```
Pass `--llm4s_version` (or type the version at its prompt) so the project starts on
0.5.1, then check the generated `build.sbt`: it needs `llm4s-core` **and at least one
provider module** at that version, because since 0.5.0 `llm4s-core` ships no provider. The template is being
moved to 0.5.x ([llm4s.g8#6](https://github.com/llm4s/llm4s.g8/pull/6)), which adds `llm4s-ollama` and
`llm4s-openai` itself; a template from before that defaults to llm4s `0.4.0` and lists `llm4s-core` only, so add
the provider you use, for example `"org.llm4s" %% "llm4s-openai" % "0.5.1"`.
The generated project has an sbt build, a logging backend, an `application.conf` with
[named provider sections](configuration#named-provider-sections), a small sample program and its tests.
[View the starter kit →](https://github.com/llm4s/llm4s.g8)
---
## Optional Dependencies
> `llm4s-core` is the spine: types, configuration, the `LLMClient` interface and the tool API. Providers, the agent
> runtime and everything below are separate modules, released at the same version. Check [Maven Central](https://central.sonatype.com/namespace/org.llm4s) for available artifacts.
### For RAG, vector stores, chunking, reranking and document extraction
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-rag" % llm4sVersion
```
Brings `llm4s-knowledgegraph` with it, along with Tika, POI, PDFBox, jsoup and the AWS S3
client — the document-extraction and loader dependencies that `llm4s-core` no longer carries.
Package names are unchanged, so existing `org.llm4s.rag.*` imports keep working; see the
[migration note](../reference/migration#slice-1-llm4s-rag-and-llm4s-knowledgegraph).
To keep the vectors in an Elasticsearch or OpenSearch cluster you already run, add the
Elasticsearch store module (Experimental), which holds both `ElasticsearchVectorStore` and
`OpenSearchVectorStore`. It talks to the REST API through `llm4s-core`'s HTTP client, so it brings no
client library; see [Elasticsearch](../guide/vector-store#elasticsearch) and
[OpenSearch](../guide/vector-store#opensearch).
```scala
// same version as llm4s-rag
libraryDependencies += "org.llm4s" %% "llm4s-vectorstore-elasticsearch" % llm4sVersion
```
To keep them in Redis (Redis Stack, or Redis 8 and later) or Valkey (with `valkey-search`), add the Redis store
(Experimental). It brings the Jedis client; see [Redis and Valkey](../guide/vector-store#redis-and-valkey).
```scala
// same version as llm4s-rag
libraryDependencies += "org.llm4s" %% "llm4s-vectorstore-redis" % llm4sVersion
```
### For agent memory
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-memory" % llm4sVersion
// only if you store memories in Postgres/pgvector
libraryDependencies += "org.llm4s" %% "llm4s-memory-postgres" % llm4sVersion
```
`llm4s-memory` carries `MemoryStore`, the memory managers, and the in-memory and SQLite-backed
stores; it adds sqlite-jdbc to your classpath and nothing else. `PostgresMemoryStore` lives in
`llm4s-memory-postgres`, which brings HikariCP and the Postgres JDBC driver — the two
dependencies `llm4s-core` no longer carries. Package names are unchanged, so existing
`org.llm4s.agent.memory.*` imports keep working; see the
[migration note](../reference/migration#slice-2-llm4s-memory-and-llm4s-memory-postgres).
### For MCP (Model Context Protocol)
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-mcp" % llm4sVersion
```
Carries the MCP client, server, transports (stdio, HTTP, SSE) and `MCPToolRegistry`. It adds no
third-party dependency of its own. Package names are unchanged, so existing `org.llm4s.mcp.*`
imports keep working; see the
[migration note](../reference/migration#slice-3-llm4s-mcp).
### For speech (STT / TTS)
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-speech" % llm4sVersion
```
Carries speech-to-text (Vosk for offline recognition, Whisper), text-to-speech (Tacotron 2),
and the audio IO, conversion and validation helpers. **This is the module that brings Vosk**,
a 25 MB dependency that used to sit on every `llm4s-core` user's classpath — which is much of
the point of the split. Package names are unchanged; see the
[migration note](../reference/migration#slice-3-llm4s-speech).
### For Ollama (local models)
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-ollama" % llm4sVersion
```
Carries the Ollama chat client, the Ollama embedding provider and Ollama model listing. It adds
no third-party dependency of its own. Adding it is all the registration there is: the module
declares itself to the provider registry, so `provider = "ollama"` and
`EMBEDDING_MODEL=ollama/` resolve with no code change. Without it, both fail with an error
saying `ollama` is not registered and naming the providers that are. Package names are
unchanged; see the
[migration note](../reference/migration#slice-5-llm4s-ollama).
### For Gemini and Vertex AI
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-gemini" % llm4sVersion
```
Carries both of Google's chat providers: the Gemini API client and model listing, and the
Vertex AI client with its OAuth authentication. It adds no third-party dependency of its own -
Vertex AI authentication is implemented without a Google SDK. Adding it is all the registration
there is: the module declares itself to the provider registry, so `provider = "gemini"` (or
`"google"`) and `provider = "vertexai"` (or `"vertex"`) resolve with no code change. Without it,
they fail with an error saying the provider is not registered and naming the providers that
are. Package names are unchanged; see the
[migration note](../reference/migration#slice-5-llm4s-gemini).
### For Anthropic
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-anthropic" % llm4sVersion
```
Carries the Anthropic Claude chat client and Anthropic model listing. **This is the module that
brings the Anthropic Java SDK** (`com.anthropic:anthropic-java`), which used to sit on every
`llm4s-core` user's classpath. Adding it is all the registration there is: the module declares
itself to the provider registry, so `provider = "anthropic"` resolves with no code change.
Without it, it fails with an error saying the provider is not registered and naming the
providers that are. Package names are unchanged; see the
[migration note](../reference/migration#slice-5-llm4s-anthropic).
### For OpenAI, Azure OpenAI and Requesty
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-openai" % llm4sVersion
```
Carries the three providers that share `OpenAIClient` - OpenAI, Azure OpenAI and Requesty - the
OpenAI embedding provider, and OpenAI and Requesty model listing. **This is the module that
brings OpenAI's Java SDK** (`com.openai:openai-java`, with OkHttp, Jackson and the Kotlin
standard library), which serves Azure OpenAI as well. It replaced Microsoft's deprecated
`com.azure:azure-ai-openai`, which used to sit on every `llm4s-core` user's classpath;
`llm4s-core` depends on no vendor SDK. Adding it is
all the registration there is: the module declares itself to the provider registry, so
`provider = "openai"`, `"azure"` and `"requesty"`, and `EMBEDDING_MODEL=openai/`, resolve
with no code change. Without it, they fail with an error saying the provider is not registered
and naming the providers that are.
OpenRouter, DeepSeek and Z.ai speak the OpenAI wire format with no SDK, and are in
`llm4s-openai-compatible` (below), which this module depends on for `OpenAIConfig`.
`llm4s-rag`'s `RAGConfig.default` embeds with `openai`, so a RAG pipeline built from the default
needs this module too, or `.withEmbeddings(...)` naming a provider you do ship. Package names are
unchanged; see the [migration note](../reference/migration#slice-5-llm4s-openai).
### For Azure OpenAI with Microsoft Entra ID
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-azure-identity" % llm4sVersion
```
Keyless authentication for Azure OpenAI ([#2274](https://github.com/llm4s/llm4s/issues/2274)): an
`azure` section with `authentication = "entra-id"` takes Microsoft Entra ID tokens from Azure's
`DefaultAzureCredential` (a managed identity, workload identity or `az login`) instead of an API key,
and `AzureEntraId` turns any `TokenCredential` into the tokens a config or embedding provider takes.
It brings Azure's `azure-identity` (MIT, with `azure-core`, MSAL4J, Netty and Reactor), which is why
it is not part of `llm4s-openai`. See
[Enterprise authentication](../guide/providers#azure-openai-with-microsoft-entra-id).
### For DeepSeek, Z.ai, OpenRouter, Mistral, Cohere and any OpenAI-compatible endpoint
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-openai-compatible" % llm4sVersion
```
One SDK-free client for the OpenAI `/chat/completions` API, and the providers that run on it:
DeepSeek (`provider = "deepseek"`), Z.ai (`"zai"`), OpenRouter (`"openrouter"`), Mistral
(`"mistral"`), Cohere (`"cohere"`, through Cohere's OpenAI-compatibility API), xAI (`"xai"`), and a generic
`"openai-compatible"` provider for any other endpoint that speaks the API - Groq, Together,
Fireworks, NVIDIA NIM, OrcaRouter, a vLLM, LM Studio or llama.cpp server, an internal
gateway - configured with a `baseUrl` and a `model`, and no code. See
[OpenAI-compatible endpoints](../guide/providers#openai-compatible-endpoints) for a
configuration recipe per provider. The module
brings no dependency beyond `llm4s-core`, and adding it registers the providers. Package names
are unchanged; see the
[migration note](../reference/migration#slice-5-llm4s-openai-compatible), and for Mistral and
Cohere [this one](../reference/migration#slice-5-mistral-cohere-and-voyage-leave-core-core-ships-no-provider).
### For AWS Bedrock
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-bedrock" % llm4sVersion
```
The AWS Bedrock chat provider (`provider = "bedrock"`), over the Converse and ConverseStream APIs,
so one client reaches Claude, Llama, Mistral, Nova and Titan models. It brings the AWS SDK for
Java v2 `bedrockruntime` artifact (Apache-2.0, the same SDK release train `llm4s-rag` uses for S3);
`llm4s-core` gains no dependency. A section needs a `region` and a `model`:
```hocon
llm4s.providers.claude-bedrock {
provider = "bedrock"
model = "anthropic.claude-3-5-sonnet-20241022-v2:0"
region = ${?AWS_REGION}
}
```
Credentials come from the AWS default credential chain (environment variables, `~/.aws`, an
instance or task role), or from a `profile`, or from explicit `accessKeyId` / `secretAccessKey`
(plus `sessionToken` for temporary credentials).
### For Voyage AI embeddings
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-voyage" % llm4sVersion
```
The Voyage AI embedding provider (`EMBEDDING_MODEL=voyage/`), with its
`llm4s.embeddings.voyage` config block. It is the first community provider module under
`modules/providers/`, and brings no dependency beyond `llm4s-core`.
### For Jina AI embeddings
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-jina" % llm4sVersion
```
The Jina AI embedding provider (`EMBEDDING_MODEL=jina/`), with its
`llm4s.embeddings.jina` config block and a typed `JinaTask` (`retrieval.query`,
`retrieval.passage`, ...). A community provider module under `modules/providers/`; it brings no
dependency beyond `llm4s-core`.
### For Cohere embeddings
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-cohere" % llm4sVersion
```
The Cohere embedding provider (`EMBEDDING_MODEL=cohere/`), on Cohere's native `/v2/embed`,
with its `llm4s.embeddings.cohere` config block and a typed `CohereInputType` (`search_document`,
`search_query`, ...). Cohere chat is a different module: `llm4s-openai-compatible`. A community
provider module under `modules/providers/`; it brings no dependency beyond `llm4s-core`.
### For the Jev decision model
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-jev" % llm4sVersion
```
A typed client for TypeSafe's Jev decision model: send a state and typed questions (Noul, Choice, Score), get typed answers.
It is not a chat provider and returns no `Completion`. Configured under `llm4s.jev`, with the key in `TYPESAFE_API_KEY`. A
community module under `modules/providers/`; it brings no dependency beyond `llm4s-core`. See
[Jev decision model](../guide/jev).
### For IBM watsonx.ai
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-watsonx" % llm4sVersion
```
The IBM watsonx.ai chat provider (`provider = "watsonx"`), brings no dependency beyond
`llm4s-core`. See [IBM watsonx.ai](../guide/providers#ibm-watsonxai).
### For Java, Spring Boot, cats-effect and ZIO
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-java-api" % llm4sVersion // Java facade
libraryDependencies += "org.llm4s" %% "llm4s-java-rag" % llm4sVersion // Java facade for RAG (Experimental)
libraryDependencies += "org.llm4s" %% "llm4s-reactor" % llm4sVersion // Reactor Flux over the Java facade (Experimental)
libraryDependencies += "org.llm4s" %% "llm4s-spring-boot-starter" % llm4sVersion // Spring Boot auto-configuration
libraryDependencies += "org.llm4s" %% "llm4s-effect" % llm4sVersion // cats-effect 3 / fs2
libraryDependencies += "org.llm4s" %% "llm4s-zio" % llm4sVersion // ZIO 2 / ZIO Streams
libraryDependencies += "org.llm4s" %% "llm4s-pekko" % llm4sVersion // Apache Pekko Streams
```
`llm4s-java-api` is the Java-friendly facade (a runnable Gradle sample is in
[`modules/samples/gradle-java`](https://github.com/llm4s/llm4s/tree/main/modules/samples/gradle-java), and a Quarkus
application with a CDI producer in [`modules/samples/quarkus-java`](https://github.com/llm4s/llm4s/tree/main/modules/samples/quarkus-java),
see [Quarkus](../guide/quarkus)) and the base of the Spring starter, which adds the
`llm4s.*` properties, an `LLM4STemplate` and an Actuator health indicator. `llm4s-java-rag` puts `llm4s-rag`'s
pipeline behind one Java class, `JRag` ([Java RAG](../guide/java-rag)); it is a separate artifact so that a Java
facade user who only calls a model does not receive document parsing and the database drivers. `llm4s-reactor` turns
a streamed reply into a Reactor `Flux` ([Java Reactor](../guide/java-reactor)), apart from the facade so that it
does not bring `reactor-core` to an application without Reactor. `llm4s-effect`,
`llm4s-zio` and `llm4s-pekko` wrap `LLMClient` and `Agent` for their effect and streaming systems.
### For Kotlin (coroutines and Flow)
`llm4s-kotlin-api` is the Kotlin coroutine API over the Java facade: `LLMClientKt` and `AgentKt` with suspend
functions, and `Flow`s of an agent turn's events. It is a Kotlin artifact, so it has **no** `_3` suffix, and it is
released at the same version as every other llm4s module, so the BOM manages it too:
> **`llm4s-kotlin-api` 0.5.1 is not on Maven Central yet.** A fault in the release
> workflow left it out of the 0.5.0 and 0.5.1 releases; the fix is pending
> ([#1976](https://github.com/llm4s/llm4s/pull/1976)). Until a release includes it, the coordinates below do not
> resolve (the BOM lists it, but every other entry of the BOM resolves). Build it locally instead, as described
> below the snippet; it then depends on the released `llm4s-java-api_3`.
```kotlin
// build.gradle.kts
dependencies {
implementation(platform("org.llm4s:llm4s-bom:0.5.1"))
implementation("org.llm4s:llm4s-kotlin-api") // brings llm4s-java-api_3 and kotlinx-coroutines-core
}
```
Without the BOM, write the version on it: `implementation("org.llm4s:llm4s-kotlin-api:0.5.1")`.
It runs on JDK 21, like the Java facade it wraps. The module is built by Gradle (`modules/kotlin-api`), not sbt, so
`sbt publishLocal` does not build it. To build it, check out the release tag (`git checkout v0.5.1`)
and run `gradle publishToMavenLocal -Pllm4sVersion=0.5.1` in `modules/kotlin-api`,
then add `mavenLocal()` to your repositories; it depends on `llm4s-java-api_3` at that version, which is on Maven
Central. Without `-Pllm4sVersion` it builds at `0.1.0-SNAPSHOT`; to try it from `main`, run `sbt publishM2` first and
pass the version sbt published.
Its entry point is `org.llm4s.kotlin.Llm4s`: `createDefaultClient()` builds an `LLMClientKt` from `application.conf`,
and `wrapClient(JLlmClient)` wraps one configured in code with the Java facade's builders (see the Java guide's
[Configure in code](../guide/java#configure-in-code)); `createAgent` and `wrapAgent` give an `AgentKt`. A Kotlin DSL
declares tools (`tools { tool(...) { ... } }`), completion options (`completionOptions { ... }`) and clients
(`Llm4s.openAi { ... }`); see the Java guide's [Kotlin DSL](../guide/java#the-kotlin-dsl).
### For image generation and vision
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-image" % llm4sVersion
```
Carries image generation (`org.llm4s.imagegeneration` — Stable Diffusion, Stability AI,
Hugging Face, OpenAI) and image processing (`org.llm4s.imageprocessing` — the OpenAI and
Anthropic vision clients plus a local `javax.imageio` processor). It adds no third-party
dependency of its own, and brings `llm4s-media` with it.
Package names are unchanged, so existing `org.llm4s.imagegeneration.*` and
`org.llm4s.imageprocessing.*` imports keep working — but note that image *formats* did change,
in `llm4s-media`. See the [migration note](../reference/migration#slice-3-llm4s-image).
### For media types (`MediaType`, `MediaCategory`)
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-media" % llm4sVersion
```
The shared vocabulary the multimodal modules speak: `MediaType` (MIME string, canonical
extension, category, and lookups by extension, path or MIME type) and `MediaCategory`. It has
no dependencies at all and does no I/O — deciding what a file is from its bytes needs Tika and
lives in `llm4s-rag`.
You will usually get it transitively, from `llm4s-core` or `llm4s-rag`; declare it directly
only if you name these types in your own signatures. It replaces three overlapping image-format
types that used to ship in `llm4s-core`, which is a source break — see the
[migration note](../reference/migration#slice-3-llm4s-media).
### For agents (`Agent`, guardrails, handoffs, graphs)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-agent" % llm4sVersion // same version as llm4s-core
```
The agent runtime - `org.llm4s.agent` (`Agent`, `AgentResult`, guardrails, handoffs,
the graph runtime) and the console assistant, `org.llm4s.assistant` - lives here.
Agent memory is the separate `llm4s-memory`; the ready-made tools are `llm4s-agent-tools`.
`llm4s-core` keeps what the agent is built on: `LLMClient`, the tool API and the tracing
contract. Package names are unchanged. See the
[migration note](../reference/migration#slice-7-llm4s-agent---the-agent-runtime-leaves-core).
### For durable agent threads (SQLite checkpointer)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-agent-checkpoint-sqlite" % llm4sVersion // same version as llm4s-agent
```
`SqliteCheckpointer` (`org.llm4s.agent.graph.sqlite`) keeps graph and agent threads - checkpoints, pending
writes, event logs and run claims - in one SQLite file, so a run survives a restart and `recover` continues it
in a new process. It brings the SQLite JDBC driver (`org.xerial:sqlite-jdbc`), which `llm4s-agent` itself does
not depend on. See [Durable Checkpointers](../guide/agents/durable-checkpointers#the-sqlite-store).
### For agent threads shared across machines (PostgreSQL checkpointer)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-agent-checkpoint-postgres" % llm4sVersion // same version as llm4s-agent
```
`PostgresCheckpointer` (`org.llm4s.agent.graph.postgres`) keeps the same threads in a PostgreSQL schema that
runtimes on several machines share, with claims and retention judged by the database's clock. It brings the
PostgreSQL JDBC driver and HikariCP. See [Durable Checkpointers](../guide/agents/durable-checkpointers#the-postgresql-store).
### For semantic memory in agents
```scala
libraryDependencies += "org.llm4s" %% "llm4s-agent-memory" % llm4sVersion // same version as llm4s-agent
```
`SemanticMemoryMiddleware` (`org.llm4s.agent.harness.memory`) recalls a tenant's memories before each turn and
records the turn after it, on an `llm4s-memory` store. It depends on `llm4s-agent` and `llm4s-memory`. See
[Semantic Memory Middleware](../guide/agents/memory#semantic-memory-middleware).
### For built-in tools (web search, HTTP, filesystem, shell)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-agent-tools" % llm4sVersion // same version as llm4s-core
```
`BuiltinTools` and everything under `org.llm4s.toolapi.builtin` - DateTime, Calculator, UUID,
JSON, file access, HTTP, shell, and the Brave, DuckDuckGo and Exa search tools - live here, with
`ToolsConfigLoader` and the `llm4s.tools` settings the search tools read. `llm4s-core` keeps the
tool API they implement (`ToolFunction`, `ToolRegistry`). Package names are unchanged. See the
[migration note](../reference/migration#slice-7-llm4s-agent-tools---the-built-in-tools-leave-core).
### For tracing and metrics (Langfuse, OpenTelemetry, in-process trace store, Prometheus)
```scala
// same version as llm4s-core
libraryDependencies ++= Seq(
"org.llm4s" %% "llm4s-observability" % llm4sVersion, // TRACING_MODE=langfuse
"org.llm4s" %% "llm4s-observability-otel" % llm4sVersion, // TRACING_MODE=opentelemetry
"org.llm4s" %% "llm4s-observability-prometheus" % llm4sVersion, // llm4s.metrics.enabled = true
"org.llm4s" %% "llm4s-observability-micrometer" % llm4sVersion // MicrometerMetrics over your MeterRegistry
)
```
`llm4s-core` builds only the `console` (default) and `noop` tracers. `llm4s-observability`
registers Langfuse and carries the in-process trace collector and store
(`TraceCollectorTracing`, `org.llm4s.trace.store`, `org.llm4s.trace.model`) and `CostTracker`,
with no third-party dependency; `llm4s-observability-otel` registers OpenTelemetry and brings the
OpenTelemetry SDK. Adding the dependency is all the registration there is, and the
`LANGFUSE_*` / `OTEL_*` variables keep their names. `llm4s-observability-prometheus` carries
`PrometheusMetrics`, the `/metrics` endpoint and `MetricsConfigLoader`, with the Prometheus client
and HTTP server; core keeps only the `MetricsCollector` interface every client accepts. Package
names are unchanged. See the migration notes for
[Langfuse and the collector](../reference/migration#slice-6-llm4s-observability---langfuse-the-trace-collector-and-costtracker-leave-core)
and for [Prometheus](../reference/migration#slice-6-llm4s-observability-prometheus---prometheus-leaves-core).
`llm4s-observability-micrometer` is the Micrometer counterpart: a `MetricsCollector` that records into a `MeterRegistry`
you already have, with the same series names as the Prometheus module. It depends on `llm4s-core` and `micrometer-core`.
See [Micrometer metrics](../guide/observability/micrometer).
### For Neo4j (knowledge graph store)
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-knowledgegraph-neo4j" % llm4sVersion
```
`Neo4jGraphStore`, a graph store for `llm4s-knowledgegraph` backed by Neo4j. It depends on `llm4s-core`
and `llm4s-knowledgegraph` and brings the Neo4j driver.
### Structured output from a case class
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-schema-derivation" % llm4sVersion
```
`SchemaOf` and `client.completeStructuredOf[A]`: the JSON schema of `completeStructured`, derived from a case class
with `derives`. It is experimental, and depends on `llm4s-core` only. See [Derived Schemas](../guide/derived-schemas).
### Testing your own code (the testkit)
```scala
// same version as llm4s-core; test scope only
libraryDependencies += "org.llm4s" %% "llm4s-testkit" % llm4sVersion % Test
```
A scriptable fake `LLMClient` for the tests of code that calls llm4s: canned replies, scripted tool calls,
injected errors and recorded requests, with no network and no API key. It depends on `llm4s-core` only. See
[Testing with the Testkit](../guide/testing-with-the-testkit).
### For provider authors (testing a provider module)
```scala
// same version as llm4s-core; test scope only
libraryDependencies += "org.llm4s" %% "llm4s-provider-testkit" % llm4sVersion % Test
```
The checks a provider module's own `Llm4sModuleSpec` makes: discovery, sole ownership of its ids,
explicit registration and the config-to-client round trip. It requires JDK 21: its interruption checks
and its local test server run on virtual threads. See
[Writing a provider](../guide/writing-a-provider#testing).
### For checkpoint store authors (testing a `Checkpointer`)
```scala
// same version as llm4s-agent; test scope only
libraryDependencies += "org.llm4s" %% "llm4s-agent-testkit" % llm4sVersion % Test
```
`CheckpointerContract`, the suite every `Checkpointer` must pass: commits, event numbering, compaction, run-claim
leases and fencing, and two runtimes contending over one store. It requires JDK 21, as `llm4s-agent` does. See
[Durable Checkpointers](../guide/agents/durable-checkpointers#testing-it-with-the-contract-suite).
### For Workspace (Containerized Execution)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-workspace-client" % "0.5.1"
```
And install Docker:
```bash
# macOS
brew install docker
# Ubuntu/Debian
sudo apt-get install docker.io
# Verify
docker --version
```
`llm4s-workspace-client` brings `llm4s-workspace-shared`, the wire protocol the client and the runner image
speak, so you depend on it directly only if you implement the runner side.
---
## Provider Setup
llm4s reads provider configuration from named sections in your `application.conf`, not from
environment variables of its own. Keep secrets in environment variables and bind them into the
section with `${?VAR}`:
```hocon
# src/main/resources/application.conf
llm4s {
providers {
provider = "openai-main" # the default: the name of a section below
openai-main {
provider = "openai"
model = "gpt-4o-mini"
}
}
}
```
```bash
export OPENAI_API_KEY=sk-...
```
The section needs no `apiKey` line: `llm4s-openai` binds `OPENAI_API_KEY` to OpenAI's shared
key, which a section without a key of its own uses (see
[API keys](configuration#api-keys) for a second account). Nothing reads `LLM_MODEL`. The [Configuration guide](configuration#named-provider-sections)
shows sections for every provider, how to [switch between them](configuration#switching-providers),
and which [environment variables llm4s reads](configuration#environment-variables-llm4s-reads)
without any binding.
If you keep your variables in a `.env` file (add it to `.gitignore`!), load it before running:
```bash
source .env
sbt run
```
### Get API Keys
#### OpenAI
1. Go to [platform.openai.com](https://platform.openai.com)
2. Sign up or log in
3. Navigate to **API Keys**
4. Click **Create new secret key**
5. Copy the key (starts with `sk-`)
#### Anthropic
1. Go to [console.anthropic.com](https://console.anthropic.com)
2. Sign up or log in
3. Navigate to **API Keys**
4. Click **Create Key**
5. Copy the key (starts with `sk-ant-`)
#### Azure OpenAI
1. Create an Azure account
2. Navigate to **Azure OpenAI Service**
3. Create a resource
4. Deploy a model (e.g., gpt-4o)
5. Copy the **API Key** and **Endpoint**
#### Ollama (Local)
1. Install Ollama: [ollama.com](https://ollama.com)
2. Pull a model: `ollama pull llama2`
3. Start server: `ollama serve`
4. No API key needed!
---
## Verify Installation
Create a simple test file `VerifyInstall.scala`:
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model.{ Conversation, UserMessage }
import org.llm4s.model.ModelRegistryService
object VerifyInstall extends App {
println("Testing LLM4S installation...")
val result = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
response <- client.complete(Conversation(Seq(UserMessage("Say 'LLM4S is working!'"))))
} yield response
result match {
case Right(completion) =>
println("✅ Success!")
println(s"Response: ${completion.content}")
case Left(error) =>
println("❌ Error:")
println(error)
}
}
```
Run it:
```bash
sbt run
```
Expected output:
```
Testing LLM4S installation...
✅ Success!
Response: LLM4S is working!
```
---
## Troubleshooting
### "missing required fields: apiKey"
**Problem**: the provider section has no API key.
**Solution**:
1. Check the vendor's variable is set in the shell that starts the JVM: `echo $OPENAI_API_KEY`
(if you use a `.env` file, `source .env` first), and that the provider's module
(`llm4s-openai`) is a dependency - it is what binds the variable
2. If the section sets its own `apiKey = ${?SOME_VAR}`, check that variable too
3. Check the error names the section you meant to load - only that section is validated, so
if it names another, `llm4s.providers.provider` points somewhere you did not expect
### "Configured provider '...' was not found" or an unknown provider id
**Problem**: `llm4s.providers.provider` names a section that does not exist, or a section's
`provider = "..."` id is not registered.
**Solution**: make `llm4s.providers.provider` match a section name exactly, and add the module
for the provider id (for example `llm4s-openai` for `provider = "openai"`; see
[Optional Dependencies](#optional-dependencies)). Model names go in the section's `model`
without a provider prefix: `model = "gpt-4o"` for OpenAI, not `openai/gpt-4o`.
### Compilation Errors
**Problem**: Scala version mismatch.
**Solution**:
```bash
# Clean and recompile
sbt clean
sbt compile
```
### Dependency Resolution Issues
**Problem**: Can't resolve LLM4S dependency.
**Solution**:
- Releases are on Maven Central, so no additional resolver is needed
- With Maven or Gradle, write the Scala suffix yourself: `llm4s-core_3`, not `llm4s-core` (the BOM, `llm4s-bom`,
has none)
- No snapshots are published: to use `main`, build it locally (see [Unreleased changes](#unreleased-changes-main))
- `llm4s-kotlin-api` is not on Maven Central for 0.5.1 yet (see
[For Kotlin](#for-kotlin-coroutines-and-flow))
---
## Next Steps
Now that LLM4S is installed:
1. **[Write your first program →](first-example)** - Create a simple LLM application
2. **[Configure providers →](configuration)** - Set up multiple LLM providers
3. **[Explore examples →](/examples/)** - Browse 69 working examples
---
## Additional Resources
- **GitHub Repository**: [llm4s/llm4s](https://github.com/llm4s/llm4s)
- **Starter Kit**: [llm4s.g8](https://github.com/llm4s/llm4s.g8)
- **Discord Community**: [Join us](https://discord.gg/4uvTPn6qww)
- **API Reference**: [Core API](/api/)
---
**Installation complete!** Ready to [write your first program →](first-example)
---
Source: https://llm4s.org/getting-started/first-example
# Your First LLM4S Program
Build your first LLM-powered application in minutes.
---
## Hello, LLM!
Let's start with the simplest possible LLM4S program - a "Hello World" that asks the LLM a question.
### Create the File
The examples use llm4s 0.5.1 on Scala 3.7.x and JDK 21. Add `llm4s-core` and the
provider module you call, here `llm4s-openai` (see [Installation](installation)), to `build.sbt`:
```scala
val llm4sVersion = "0.5.1"
libraryDependencies ++= Seq(
"org.llm4s" %% "llm4s-core" % llm4sVersion,
"org.llm4s" %% "llm4s-openai" % llm4sVersion,
"org.llm4s" %% "llm4s-agent" % llm4sVersion // only for the agent examples further down
)
```
Then create `HelloLLM.scala`.
It reads the provider from a named section in `src/main/resources/application.conf`:
```hocon
llm4s {
providers {
provider = "openai-main" # the default: the name of a section below
openai-main {
provider = "openai"
model = "gpt-4o-mini"
}
}
}
```
The [Configuration guide](configuration#named-provider-sections) has sections for Anthropic,
Gemini, Azure and Ollama; the program below does not change when you switch.
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.{LLMClient, LLMConnect}
import org.llm4s.llmconnect.model.{ Conversation, UserMessage }
import org.llm4s.model.ModelRegistryService
// 1. Core Logic: Depends only on the injected client, not configuration
class HelloLLM(client: LLMClient) {
def sayHello(): Unit = {
val result = client.complete(Conversation(Seq(UserMessage("What is Scala?"))))
result match {
case Right(completion) =>
println(s"Response: ${completion.content}")
case Left(error) =>
println(s"Error: $error")
}
}
}
// 2. Configuration Boundary: The application entry point
object Main extends App {
val startup = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
} yield new HelloLLM(client).sayHello()
startup.left.foreach(err => println(s"Startup Error: $err"))
}
```
### Run It
```bash
# llm4s-openai binds OPENAI_API_KEY to OpenAI's shared key, which the section uses
export OPENAI_API_KEY=sk-...
sbt run
```
### Expected Output
```
Response: Scala is a high-level programming language that combines
object-oriented and functional programming paradigms. It runs on the
JVM and is known for its strong type system and concurrency support.
```
---
## Understanding the Code
Let's break down what's happening:
### 1. The Configuration Boundary
```scala
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
```
In LLM4S, we follow a strict configuration boundary. The entry point (`Main`) builds the client:
- Loads the typed config of the section `llm4s.providers.provider` names in `application.conf`
- Selects the appropriate provider (OpenAI, Anthropic, etc.)
- Injects the `LLMClient` into your core logic (`HelloLLM`).
### 2. Complete with Messages
```scala
val response = client.complete(
Conversation(List(UserMessage("What is Scala?"))),
CompletionOptions()
)
```
- **conversation**: The conversation messages (User, Assistant, System), wrapped in `Conversation`
- **options**: Per-request completion settings; the model is selected when the client is configured
- Returns `Result[Completion]`
### 3. Result Handling
```scala
result match {
case Right(completion) => // Success
case Left(error) => // Error
}
```
LLM4S uses **Result types** instead of exceptions:
- `Right(value)` = success
- `Left(error)` = failure
This makes error handling **explicit** and **type-safe**.
---
## Adding Conversation Context
Let's make our program more interesting with a multi-turn conversation:
```scala
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model._
class ConversationExample(client: LLMClient) {
def run(): Unit = {
val result = client.complete(
Conversation(Seq(
SystemMessage("You are a helpful programming tutor."),
UserMessage("What is Scala?"),
AssistantMessage("Scala is a high-level programming language..."),
UserMessage("How does it compare to Java?")
))
)
result.fold(
error => println(s"Error: $error"),
completion => println(s"Response: ${completion.content}")
)
}
}
```
### Message Types
LLM4S provides three message types:
| Type | Purpose | Example |
|------|---------|---------|
| `SystemMessage` | Set LLM behavior | "You are a helpful assistant" |
| `UserMessage` | User's input | "What is Scala?" |
| `AssistantMessage` | LLM's previous responses | "Scala is a programming language..." |
---
## Adding Tool Calling
Now let's give the LLM access to a tool:
```scala
import org.llm4s.llmconnect.LLMClient
import org.llm4s.toolapi.{ Schema, ToolBuilder, ToolRegistry }
import org.llm4s.agent.Agent
class ToolExample(client: LLMClient) {
// Define a simple tool
def getWeather(location: String): String = {
s"The weather in $location is sunny and 72°F"
}
private val weatherSchema = Schema
.`object`[Map[String, Any]]("Weather query parameters")
.withProperty(Schema.property("location", Schema.string("The city to look up")))
def run(): Unit = {
val result = for {
weatherTool <- ToolBuilder[Map[String, Any], String](
"get_weather",
"Get current weather for a location",
weatherSchema
).withHandler(extractor => extractor.getString("location").map(getWeather)).buildSafe()
agent <- Agent.builder("weather-agent", client)
.withTools(new ToolRegistry(Seq(weatherTool)))
.build()
result <- agent.run("What's the weather in Paris?")
} yield result
result.fold(
error => println(s"Error: $error"),
r => println(s"Final response: ${r.answer.getOrElse(r.status.toString)}")
)
}
}
```
The agent will:
1. Understand you're asking about weather
2. Call the `get_weather` tool with "Paris"
3. Use the tool result to formulate a response
---
## Streaming Responses
For real-time output, use streaming:
```scala
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model._
class StreamingExample(client: LLMClient) {
def run(): Unit = {
val result = client.streamComplete(
Conversation(Seq(UserMessage("Write a short poem about Scala"))),
onChunk = chunk => chunk.content.foreach(print) // Print each token as it arrives
)
result.fold(
error => println(s"Error: $error"),
_ => println("\nDone!")
)
}
}
```
Output appears token-by-token in real-time, like ChatGPT!
---
## Error Handling Patterns
### Basic Error Handling
```scala
import org.llm4s.error.{ AuthenticationError, NetworkError, ValidationError }
import org.llm4s.llmconnect.model._
// Assuming client is injected
val result = client.complete(Conversation(Seq(UserMessage("Hello"))))
result match {
case Right(completion) =>
// Success
println(completion.content)
case Left(error) =>
// Handle different error types
error match {
case e: NetworkError => println(s"Network issue: ${e.message}")
case e: AuthenticationError => println(s"Auth failed: ${e.message}")
case e: ValidationError => println(s"Invalid input: ${e.message}")
case _ => println(s"Error: $error")
}
}
```
### Using fold
```scala
result.fold(
error => println(s"Failed: $error"),
completion => println(s"Success: ${completion.content}")
)
```
### Mapping Results
```scala
val content: Result[String] = result.map(_.content)
val uppercased: Result[String] = content.map(_.toUpperCase)
```
---
## Configuration Options
### Using Different Models
```scala
import org.llm4s.config.Llm4sConfig
// The model belongs to the provider section, so to use another model load the section that names it,
// for example one with provider = "openai" and model = "gpt-4-turbo"
val gpt4Turbo = Llm4sConfig.provider("openai-gpt4-turbo")
```
### Provider-Specific Settings
The model and provider come from the section in `application.conf`; per-request settings such
as temperature and token limits go in `CompletionOptions`:
```hocon
openai-main {
provider = "openai"
model = "gpt-4o"
}
```
```scala
client.complete(conversation, CompletionOptions(temperature = 0.7, maxTokens = Some(1000)))
```
---
## Complete Working Example
Here's a complete example combining everything we've learned:
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.{LLMClient, LLMConnect}
import org.llm4s.llmconnect.model._
import org.llm4s.toolapi.{ Schema, ToolBuilder, ToolRegistry }
import org.llm4s.agent.Agent
class ComprehensiveAgent(client: LLMClient) {
def calculate(expression: String): String = {
// Simple calculator (use proper eval in production!)
s"Result: ${expression} = 42"
}
private val calcSchema = Schema
.`object`[Map[String, Any]]("Calculator parameters")
.withProperty(Schema.property("expression", Schema.string("The expression to evaluate")))
def run(): Unit = {
println("🚀 Starting LLM4S Example...")
// Tools belong to the agent, so give them to the builder
val result = for {
calcTool <- ToolBuilder[Map[String, Any], String](
"calculate",
"Evaluate a mathematical expression",
calcSchema
).withHandler(extractor => extractor.getString("expression").map(calculate)).buildSafe()
agent <- Agent.builder("calc-agent", client)
.withTools(new ToolRegistry(Seq(calcTool)))
.build()
result <- agent.run("What is 6 times 7? Please use the calculator.")
} yield result
result match {
case Right(r) =>
println(s"✅ Success!")
println(s"Response: ${r.answer.getOrElse(r.status.toString)}")
println(s"Messages exchanged: ${r.messages.length}")
case Left(error) =>
println(s"❌ Error: $error")
System.exit(1)
}
}
}
// Application Entry Point
object ComprehensiveMain extends App {
val startup = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given org.llm4s.model.ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
} yield new ComprehensiveAgent(client).run()
startup.left.foreach(err => println(s"Startup Error: $err"))
}
```
---
## Common Patterns
### Pattern 1: Simple Query
```scala
// Assuming client is injected
val response = client.complete(
Conversation(Seq(UserMessage("Your question here")))
)
```
### Pattern 2: With System Prompt
```scala
// Assuming client is injected
val response = client.complete(
Conversation(Seq(
SystemMessage("You are an expert in..."),
UserMessage("Question")
))
)
```
### Pattern 3: Agent with Tools
```scala
// Assuming client is injected
val result = for {
agent <- Agent.builder("assistant", client).withTools(new ToolRegistry(myTools)).build()
result <- agent.run("User query")
} yield result
```
### Pattern 4: Streaming
```scala
// Assuming client is injected
val completion = client.streamComplete(
Conversation(Seq(UserMessage("Your question here"))),
onChunk = chunk => chunk.content.foreach(print)
)
```
---
## Performance Optimization
### 1. Reuse Client Instances
Don't create a new client for every request:
```scala
// Assuming client: LLMClient is injected into your class
// ✅ Good: Reuse existing injected client
(1 to 10).foreach { i =>
client.complete(Conversation(Seq(UserMessage(s"Question $i"))))
}
// ❌ Bad: Creating new client inside the loop (wasteful and violates boundaries)
(1 to 10).foreach { i =>
for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
badClient <- LLMConnect.getClient(providerConfig) // Don't do this!
response <- badClient.complete(
Conversation(Seq(UserMessage(s"Q$i")))
)
} yield response
}
```
### 2. Use Streaming for Long Responses
Streaming gets you the first token faster and improves perceived latency:
```scala
import org.llm4s.llmconnect.model._
// Assuming client is injected
val streamResult = client.streamComplete(
Conversation(Seq(UserMessage("Write a long essay about Scala"))),
onChunk = chunk => chunk.content.foreach(print) // Prints as tokens arrive
)
streamResult match {
case Right(completion) => println(s"\nCompleted with ${completion.usage.map(_.totalTokens).getOrElse(0)} tokens")
case Left(error) => println(s"Error: $error")
}
```
**When to stream:**
- User-facing applications (chat interfaces)
- Long-form content generation
- When you need to show progress
**When not to stream:**
- Short queries (< 100 tokens)
- When you need the full response before processing
- Background batch jobs
### 3. Parallelize Independent Requests
Process multiple unrelated queries concurrently:
```scala
import scala.concurrent.{Future, ExecutionContext, Await}
import scala.concurrent.duration._
import ExecutionContext.Implicits.global
// Assuming client is injected
val queries = List(
"What is Scala?",
"What is functional programming?",
"What is the JVM?"
)
// Run all queries in parallel
val futures = queries.map { query =>
Future {
client.complete(Conversation(Seq(UserMessage(query))))
}
}
val results = Await.result(Future.sequence(futures), 30.seconds)
results.foreach {
case Right(response) => println(response.content)
case Left(error) => println(s"Error: $error")
}
```
### 4. Set Appropriate Timeouts
Each client has a default request and stream timeout (two minutes for a completion, five or ten for a
stream), and a provider section can change them:
```hocon
llm4s.providers.my-openai {
provider = "openai"
model = "gpt-4o"
timeouts { request = 3m, stream = 15m }
}
```
See [Timeouts](configuration#timeouts) for the keys, the defaults per provider and what each timeout
bounds. For long-form generation, prefer streaming (above).
### 5. Use Cheaper Models for Development
Change the section's `model`, or point `llm4s.providers.provider` at another section (see
[Switching providers](configuration#switching-providers)):
```hocon
llm4s.providers {
provider = "ollama-local" # development: free, local
ollama-local {
provider = "ollama"
model = "llama3.2"
baseUrl = "http://localhost:11434"
}
openai-main { # production: select with provider = "openai-main"
provider = "openai"
model = "gpt-4o-mini" # much cheaper than gpt-4o
}
}
```
Only the section being loaded is validated, so `OPENAI_API_KEY` need not be set while
`ollama-local` is the default - only where `openai-main` is selected.
### 6. Batch Embeddings
When generating embeddings for RAG:
```scala
import org.llm4s.llmconnect.model.EmbeddingRequest
// ✅ Good: Batch processing
val documents: List[String] = loadDocuments() // say, a thousand documents
val batchSize = 100
val allEmbeddings = documents.grouped(batchSize).flatMap { batch =>
embedder.embed(EmbeddingRequest(batch, embeddingModel)) match {
case Right(response) => response.embeddings
case Left(error) =>
println(s"Batch failed: $error")
List.empty
}
}.toList
// ❌ Bad: One at a time (slow, expensive)
val embeddings = documents.map { doc =>
embedder.embed(EmbeddingRequest(List(doc), embeddingModel))
}
```
### 7. Monitor Token Usage
Track costs in production:
```scala
val response = client.complete(Conversation(Seq(UserMessage("Hello"))))
response match {
case Right(completion) =>
// Note: usage returns Option[TokenUsage], so these are Option[Int]
println(s"Prompt tokens: ${completion.usage.map(_.promptTokens)}")
println(s"Completion tokens: ${completion.usage.map(_.completionTokens)}")
println(s"Total tokens: ${completion.usage.map(_.totalTokens)}")
case Left(error) => println(s"Error: $error")
}
```
## Running the Examples
**Note:** Use the same `Main` pattern shown above to run the `ConversationExample`, `ToolExample`, and `StreamingExample` classes.
---
## Next Steps
Great job! You've written your first LLM4S programs. Now explore:
1. **[Configuration →](configuration)** - Learn about advanced configuration
2. **[Agent Framework →](/examples/#agent-examples)** - Build sophisticated agents
3. **[Tool Calling →](/examples/#tool-examples)** - Create custom tools
4. **[Examples →](/examples/)** - Browse 69 working examples
---
## Learning Resources
- **[Basic Examples](/examples/basic)** - More beginner-friendly examples
- **[User Guide](/guide/)** - Comprehensive feature guide
- **[API Reference](/api/)** - Detailed API documentation
- **[Discord Community](https://discord.gg/4uvTPn6qww)** - Get help and share ideas
---
**Ready to dive deeper?** [Configure your environment →](configuration)
---
Source: https://llm4s.org/getting-started/configuration
# Configuration Guide
Configure LLM4S for multiple providers, environments, and use cases.
---
## How configuration is loaded
`Llm4sConfig` reads [Typesafe Config](https://github.com/lightbend/config) through PureConfig,
with this precedence (highest first):
1. **JVM system properties** - `-Dllm4s.providers.provider=claude`
2. **Your `application.conf`** - `src/main/resources/application.conf`, or a file named with
`-Dconfig.file=/path/to/app.conf` (or `-Dconfig.resource=prod.conf` for another resource)
3. **`reference.conf`** - the defaults shipped in each llm4s module on the classpath
Environment variables are **not** a layer of their own. llm4s reads one only where a `${?VAR}`
substitution names it: in a module's `reference.conf` (tracing, embeddings, tools, and each
provider module's vendor API key - see
[Environment variables llm4s reads](#environment-variables-llm4s-reads)), or in your own
`application.conf`. Each provider module binds its vendor's conventional key variable -
`llm4s-openai` binds `OPENAI_API_KEY`, `llm4s-anthropic` binds `ANTHROPIC_API_KEY` - to a shared
key under `llm4s.credentials` (see [API keys](#api-keys)). Nothing reads `LLM_MODEL`.
`${?VAR}` means "the value of `VAR` if it is set, otherwise leave the key unset". A second line
for the same key overrides the first only when the variable is set, which gives you a default
with an environment override:
```hocon
model = "gpt-4o-mini"
model = ${?OPENAI_MODEL} # used only when OPENAI_MODEL is set
```
---
## Named provider sections
Chat providers are configured as named sections under `llm4s.providers`. You choose each
section's name; `provider` inside it says which provider it is, and `llm4s.providers.provider`
names the section `Llm4sConfig.defaultProvider()` loads.
**1. Add llm4s and the module for your provider:**
```scala
// build.sbt
libraryDependencies ++= Seq(
"org.llm4s" %% "llm4s-core" % llm4sVersion,
"org.llm4s" %% "llm4s-openai" % llm4sVersion // the module for your provider
)
```
| `provider = ...` | Module |
|---|---|
| `openai`, `azure`, `requesty` | `llm4s-openai` |
| `anthropic` | `llm4s-anthropic` |
| `gemini`, `vertexai` | `llm4s-gemini` |
| `ollama` | `llm4s-ollama` |
| `bedrock` | `llm4s-bedrock` |
| `deepseek`, `zai`, `openrouter`, `mistral`, `cohere`, `xai`, `openai-compatible` | `llm4s-openai-compatible` |
Before 0.5.0, every provider shipped inside `llm4s-core`, so the `llm4s-core` dependency alone was enough.
**2. Add a section to `src/main/resources/application.conf`:**
```hocon
llm4s {
providers {
provider = "openai-main" # the default: the name of a section below
openai-main {
provider = "openai"
model = "gpt-4o-mini"
}
}
}
```
**3. Set your vendor's API-key variable:**
```bash
export OPENAI_API_KEY=sk-...
```
The section needs no `apiKey` line: `llm4s-openai` binds `OPENAI_API_KEY` to OpenAI's shared
key, `llm4s.credentials.openai.apiKey`, and a section without a key of its own uses that one.
See [API keys](#api-keys) for a second account or a variable of your own.
**4. Load it at the edge of your application:**
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model.{ Conversation, UserMessage }
import org.llm4s.model.ModelRegistryService
val result = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
response <- client.complete(Conversation(Seq(UserMessage("What is Scala?"))))
} yield response.content
```
If `OPENAI_API_KEY` is unset, `defaultProvider()` returns a `ConfigurationError` naming both
places a key can come from:
`apiKey: set OPENAI_API_KEY, or set apiKey under llm4s.providers.openai-main in application.conf`.
This exact configuration is exercised by `DocumentedProviderConfigSpec` in `modules/openai`.
Other `Llm4sConfig` calls read the same sections:
```scala
Llm4sConfig.provider("openai-main") // any section, by name
Llm4sConfig.defaultProviderName() // the value of llm4s.providers.provider
Llm4sConfig.providers() // every section, validated - fails if any one is invalid
Llm4sConfig.providerConfigs() // every section, each loaded or failed on its own
Llm4sConfig.listModels() // model discovery for the default section
Llm4sConfig.listModels("openai-main") // ... or for a named one
```
### API keys
Credentials belong to a vendor; sections belong to a use. A client's key is, in order:
1. its own `apiKey` - in its chat section `llm4s.providers.`, its embeddings block
`llm4s.embeddings.` or the reranker block `llm4s.rerank.cohere`;
2. otherwise its vendor's shared key, `llm4s.credentials..apiKey`, which the provider
module binds to the vendor's conventional variable;
3. otherwise a `ConfigurationError` naming both.
An alias resolves to its canonical provider's key: a `provider = "google"` section uses
`llm4s.credentials.gemini`. When a key is resolved, llm4s logs at INFO where it came from - for
example `llm4s.providers.openai-main: API key from llm4s.credentials.openai.apiKey` - and never
the value.
| Provider id | Shared key bound to | Module |
|---|---|---|
| `openai` | `OPENAI_API_KEY` | `llm4s-openai` |
| `azure` | `AZURE_OPENAI_API_KEY` | `llm4s-openai` |
| `requesty` | `REQUESTY_API_KEY` | `llm4s-openai` |
| `anthropic` | `ANTHROPIC_API_KEY` | `llm4s-anthropic` |
| `gemini` (alias `google`) | `GOOGLE_API_KEY`, else `GEMINI_API_KEY` | `llm4s-gemini` |
| `deepseek` | `DEEPSEEK_API_KEY` | `llm4s-openai-compatible` |
| `zai` | `ZAI_API_KEY` | `llm4s-openai-compatible` |
| `openrouter` | `OPENROUTER_API_KEY` | `llm4s-openai-compatible` |
| `mistral` | `MISTRAL_API_KEY` | `llm4s-openai-compatible` |
| `xai` | `XAI_API_KEY` | `llm4s-openai-compatible` |
| `cohere` | `COHERE_API_KEY` (chat, reranker and embeddings) | `llm4s-openai-compatible`, `llm4s-rag`, `llm4s-cohere` |
| `voyage` | `VOYAGE_API_KEY` | `llm4s-voyage` |
| `jina` | `JINA_API_KEY` | `llm4s-jina` |
| `typesafe` | `TYPESAFE_API_KEY` (the Jev decision client; not a chat or embeddings provider) | `llm4s-jev` |
`ollama` takes no key, the generic `openai-compatible` provider has no vendor, `vertexai`
authenticates with OAuth2 (Application Default Credentials, or a service-account file named by
its `apiKey`), and `bedrock` authenticates with AWS credentials (the AWS default credential chain,
a `profile`, or explicit keys), so none of them has a shared key.
A section for a **second account** sets its own key, which wins over the shared one:
```hocon
llm4s.providers {
openai-main { # uses OPENAI_API_KEY
provider = "openai"
model = "gpt-4o-mini"
}
openai-batch { # billed to another account
provider = "openai"
model = "gpt-4o-mini"
apiKey = ${?OPENAI_BATCH_API_KEY}
}
}
```
The same form reads a variable of your own choosing - `apiKey = ${?MY_TEAM_OPENAI_KEY}`. Note that
if `OPENAI_BATCH_API_KEY` is unset, `openai-batch` falls back to the shared `OPENAI_API_KEY`
and bills the default account. In production, give every section its own `apiKey`; the
config-policy `prod` preset flags any section that does not (see
[Production deployment](../PRODUCTION_DEPLOYMENT)).
### Workload identity (SPIFFE)
A section can authenticate with a workload identity token - typically a SPIFFE JWT-SVID that
[`spiffe-helper`](https://github.com/spiffe/spiffe-helper) keeps fresh in a file - instead of an
API key. Put an `auth` block in the section, with `identityTokenFile` (re-read on every exchange,
so rotation is picked up) and the keys the provider needs. A relative `identityTokenFile` is
resolved against the process's working directory when the configuration is loaded, so prefer an
absolute path. The path must name a regular file of at most 64 KiB (a symbolic link to one is
followed, as Kubernetes projected volumes rotate tokens that way); anything else fails the exchange
with an `AuthenticationError` that names the path but never quotes the file. For `openai-compatible`
and `openai` llm4s reads the file itself; for `anthropic` the Anthropic SDK reads it. Keep the file
readable only by the workload where the platform allows it. A section sets `apiKey` or `auth`, never both, and a section with `auth` may not set
an `Authorization` header either: the exchanged token is the request's bearer. With `auth`, the
shared `llm4s.credentials..apiKey` is not used.
**Databricks model serving** (the generic `openai-compatible` provider; the SVID is exchanged at
the workspace's RFC 8693 endpoint, the token cached until shortly before it expires, and refreshed
once if a request is rejected with 401):
```hocon
databricks-main {
provider = "openai-compatible"
baseUrl = "https://.cloud.databricks.com/serving-endpoints"
model = ""
auth {
identityTokenFile = "/var/run/secrets/spiffe/databricks"
tokenUrl = "https://.cloud.databricks.com/oidc/v1/token"
clientId = ${?DATABRICKS_CLIENT_ID} # the service principal, for a service-principal federation policy
scope = "all-apis"
}
}
```
The optional `audience` key sets the RFC 8693 `audience` parameter, for a token endpoint that needs one.
`tokenUrl` and `baseUrl` must be `https` (plain `http` only for a loopback host). The exchanged
token is sent as the bearer of every request to `baseUrl`, a host you choose, so no host is
allow-listed: make sure `tokenUrl` and `baseUrl` belong to the same trusted service. The token is cached for the
reply's `expires_in`; a reply without one is taken to live until the access token's `exp` claim if
it is a JWT, and otherwise for 5 minutes.
**OpenAI** (the OpenAI SDK's workload identity federation):
```hocon
openai-wif {
provider = "openai"
model = "gpt-4o-mini"
auth {
identityTokenFile = "/var/run/secrets/spiffe/openai"
identityProviderId = ${?OPENAI_IDENTITY_PROVIDER_ID}
serviceAccountId = ${?OPENAI_SERVICE_ACCOUNT_ID}
}
}
```
**Anthropic** (the Anthropic SDK's workload identity federation; `identityTokenFile` only, not a
literal `identityToken`):
```hocon
anthropic-wif {
provider = "anthropic"
model = "claude-sonnet-4-5"
auth {
identityTokenFile = "/var/run/secrets/spiffe/anthropic"
federationRuleId = ${?ANTHROPIC_FEDERATION_RULE_ID}
organizationId = ${?ANTHROPIC_ORGANIZATION_ID}
serviceAccountId = ${?ANTHROPIC_SERVICE_ACCOUNT_ID}
workspaceId = ${?ANTHROPIC_WORKSPACE_ID}
}
}
```
A `${?VAR}` that is unset leaves its key out, and a missing required key is reported when the
section is loaded. With `auth`, `baseUrl` must be the vendor's own API host over `https`, because the
exchanged token is a vendor credential sent with every request (plain `http` is accepted only for a
loopback host, for tests): for OpenAI `api.openai.com` or a data-residency host (`us.`, `eu.` or
`ae.api.openai.com`), for Anthropic `api.anthropic.com`, where the SDK also posts the identity token
(`/v1/oauth/token`). A proxy or gateway `baseUrl` needs an `apiKey` instead.
The Anthropic SDK reads `identityTokenFile` itself, so llm4s's own checks on a token file it reads (a regular file
of at most 64 KiB, read with a bounded read) do not guard the SDK's reads. Building the client applies them once: a
path that is a directory, a FIFO, a device or a file larger than 64 KiB is refused then, as a `ConfigurationError`
naming `workloadIdentity.identityTokenFile`. A path that does not exist yet is accepted, because an identity agent
may write it after the application starts. Point it at the file the agent keeps fresh.
Request each SVID for the audience its relying party expects (`jwt_audience` in `spiffe-helper`).
Other providers reject an `auth` block. The `prod` config-policy preset's `ownApiKey` rule accepts
a section that authenticates this way.
Model listing works for an `openai-compatible` section with `auth` (it exchanges once). For
`openai` and `anthropic` sections with `auth` it is not supported - their SDKs perform the exchange
only inside a client - and the lister returns a `ConfigurationError` saying so; list models from a
section with an `apiKey`.
### Section keys
| Key | Meaning |
|---|---|
| `provider` | Required. The provider id, e.g. `openai`, `anthropic`, `ollama` (table above) |
| `model` | Required. The model name as the provider spells it, e.g. `gpt-4o-mini` |
| `apiKey` | The API key; required by every cloud provider. Optional in the section when the vendor's shared key is set ([API keys](#api-keys)) |
| `baseUrl` | Overrides the provider's default endpoint (`ollama`: `http://localhost:11434`); **required** for `openai-compatible`. Checked when the section is loaded: an absolute `http` or `https` URL with a host, or a `ConfigurationError` naming the key (a blank value counts as unset) |
| `headers` | Extra HTTP headers; sent by generic `openai-compatible` endpoints and by model listing |
| `timeouts` | How long a request and a stream may take: `timeouts { request = 3m, stream = 15m }`. Every provider that makes HTTP calls reads it; see [Timeouts](#timeouts) |
| `organization` | OpenAI, Requesty and OpenRouter: the OpenAI organisation id, sent as `OpenAI-Organization` |
| `endpoint`, `apiVersion` | Azure OpenAI: the resource endpoint (required) and API version |
| `project`, `location` | Vertex AI: the GCP project id (required) and region (default `us-central1`) |
| `region`, `profile`, `accessKeyId`, `secretAccessKey`, `sessionToken` | AWS Bedrock: the AWS region (required, never defaulted); a shared-config profile; or explicit credentials, with a session token for temporary ones. With none of the credential keys the AWS default credential chain is used. `baseUrl` overrides the endpoint |
| `contextWindow`, `reserveCompletion`, `registryProvider`, `streamUsage` | Generic `openai-compatible` endpoints ([details](../guide/providers#openai-compatible-endpoints)) |
The first six keys are shared by every provider. The rest belong to the providers named, which
declare them ([provider-specific keys](../guide/providers#provider-specific-keys)); in a section for
any other provider such a key is ignored with a warning naming it, as a misspelt key is.
Each provider module's `reference.conf` has a commented example section, and the
[provider guide](../guide/providers) covers each provider in detail.
### Timeouts
Every provider section, and every embedding section, accepts an optional `timeouts` block:
```hocon
llm4s.providers.my-openai {
provider = "openai"
model = "gpt-4o"
timeouts {
request = 3m # a call that returns one response: a completion, an embedding
stream = 15m # a streamed completion
}
}
llm4s.embeddings.openai.timeouts.request = 30s
```
Each value is a duration (`30s`, `2m`, `1500ms`, `1h`) and is optional: **a value you leave out keeps
that client's own default, so a section without the block behaves exactly as it did before.** A value
must be positive and finite; `0s`, a negative value, `Inf` and anything that is not a duration are
refused when the section is loaded, naming the key (`llm4s.providers.my-openai.timeouts.request`). A
misspelt key inside the block (`reqest = 3m`) is refused too, rather than silently leaving the default
in force. In an embedding section only `request` has a meaning, since an embedding call does not stream.
The usual precedence applies: `-D` system properties, then `application.conf`, then a module's
`reference.conf`; to take a value from the environment, bind it yourself with
`timeouts.request = ${?MY_REQUEST_TIMEOUT}`, which leaves the default in force while the variable is unset.
| Provider | `request` default | `stream` default | What the timeout bounds |
|---|---|---|---|
| `openai-compatible`, `deepseek`, `zai`, `openrouter`, `mistral`, `cohere` (chat) | 2 minutes | 5 minutes | the wait for the response to begin |
| `gemini`, `vertexai`, `ollama`, `watsonx` (chat) | 2 minutes | 10 minutes | the wait for the response to begin |
| `openai`, `azure`, `requesty`, `anthropic` | the SDK's own default | the SDK's own default | the whole call, including a streamed body |
| `bedrock` | the AWS SDK's own default | no limit | the whole call, including retries and a streamed body |
| Embeddings: `openai`, `ollama` | 2 minutes | n/a | the wait for the response to begin |
| Embeddings: `voyage`, `jina`, `cohere` | 2 minutes | n/a | the wait for the response to begin |
Five things to know:
- **The HTTP-based clients bound the wait for the response to begin**, not the time a stream may then
run. A stream whose server has begun answering is not cut by `stream`.
- **The OpenAI, Azure, Requesty and Anthropic clients use their vendor's SDK**, which bounds the whole
call and retries a call that fails or times out **twice** by default (llm4s does not change that). A
`request = 30s` there can therefore take up to three attempts and their backoff before the call
fails. `request` and `stream` are independent: a short `request` does not cut a stream.
- **Bedrock uses the AWS SDK.** `request` becomes the SDK's API-call timeout, which covers its retries,
so a `request = 30s` call fails within 30 seconds; `stream` is a deadline on the whole `ConverseStream`
call, so unlike the HTTP-based clients it does cut a stream that is still running. Either expiry is a
`TimeoutError`.
- **The workload identity token exchange of an `openai-compatible` section with `auth`** (see
[Workload identity](#workload-identity-spiffe)) follows that section's `timeouts.request`, so a
slow token endpoint is given up on when a slow completion would be; with `request` unset it uses
the request default, 2 minutes.
- **Model listing, the Vertex AI token request and the watsonx IAM token exchange keep their own fixed
timeouts**, which the block does not change. That includes the exchange model listing makes for an
`openai-compatible` section with `auth`.
### Examples for other providers
```hocon
llm4s {
providers {
provider = "claude"
claude { # key from ANTHROPIC_API_KEY
provider = "anthropic"
model = "claude-sonnet-4-20250514"
}
gemini-main { # key from GOOGLE_API_KEY or GEMINI_API_KEY
provider = "gemini"
model = "gemini-2.0-flash"
}
azure-main { # key from AZURE_OPENAI_API_KEY
provider = "azure"
model = "gpt-4o"
endpoint = ${?AZURE_API_BASE} # https://.openai.azure.com
apiVersion = ${?AZURE_API_VERSION} # optional
}
ollama-local {
provider = "ollama"
model = "llama3.2"
baseUrl = ${?OLLAMA_BASE_URL} # optional override; the default is http://localhost:11434
}
}
}
```
**Only the section you load is validated.** A section with no key available, or
whose provider module is not on the classpath, fails when it is loaded - by
`provider("")`, or by `defaultProvider()` when it is the default - and not otherwise.
With the example above and `ANTHROPIC_API_KEY` alone set, `defaultProvider()` loads `claude`
while `provider("gemini-main")` fails with a `ConfigurationError` naming `GOOGLE_API_KEY`,
`GEMINI_API_KEY` and `gemini-main`'s `apiKey`. The exception is `Llm4sConfig.providers()`, which returns every section and so
validates them all; `Llm4sConfig.providerConfigs()` instead reports each section's error
separately. (Up to 0.4.1 every section was validated on every load.)
---
## Corporate networks: a proxy and a custom CA
Behind an outbound proxy, or behind TLS inspection that re-signs traffic with a corporate CA, the first call fails with
`connection timed out` or `PKIX path building failed`. One `llm4s.http` block fixes both for every provider client: the
JDK client in core (the OpenAI-compatible providers, Ollama, Gemini, Vertex AI, watsonx.ai, the embedding providers
and the model listers), the `openai-java` SDK (OpenAI, Azure OpenAI), the `anthropic-java` SDK and the AWS SDK (Bedrock)
([#2273](https://github.com/llm4s/llm4s/issues/2273)).
```hocon
llm4s.http {
proxy {
url = "http://proxy.corp:3128" # bound to HTTPS_PROXY; an http proxy, with its port
username = ${?LLM4S_PROXY_USER} # Basic proxy authentication, if the proxy asks for it
password = ${?LLM4S_PROXY_PASSWORD}
nonProxyHosts = ["localhost", "127.*", "*.internal"] # bound to NO_PROXY; reached directly
}
tls {
trustStore = "/etc/pki/corp-ca.p12" # PKCS12 or JKS; trusted in addition to the JDK's CAs
trustStorePassword = ${?LLM4S_TRUSTSTORE_PASSWORD}
keyStore = "/etc/pki/client.p12" # optional: a client certificate, for mTLS gateways
keyStorePassword = ${?LLM4S_KEYSTORE_PASSWORD}
}
}
```
- Core's `reference.conf` binds `HTTPS_PROXY`, `NO_PROXY`, `LLM4S_PROXY_USER`, `LLM4S_PROXY_PASSWORD`,
`LLM4S_TRUSTSTORE`, `LLM4S_TRUSTSTORE_PASSWORD`, `LLM4S_KEYSTORE` and `LLM4S_KEYSTORE_PASSWORD`, so in a container
the variables alone are enough. A `user:password@` in the proxy URL is read as the credentials too.
- `nonProxyHosts` entries are a host (`ollama.local`), a domain suffix (`*.internal` or `.internal`), a prefix
(`10.*`) or `*`. The default is the loopback addresses, so a local Ollama is reached directly; setting the list
replaces that default. CIDR ranges are not supported.
- The trust store **adds** CAs: the public CAs of the JDK stay trusted.
- An unknown key, a proxy URL that is not `http://host:port`, or a store that cannot be opened (missing, corrupt, wrong
password) fails the load with a `ConfigurationError` naming the key. Passwords never appear in an error, a log line
or a `toString`.
A section that needs a different route - a gateway reached through another proxy, or directly - sets its own `http`
block, whose keys take precedence over `llm4s.http` one by one:
```hocon
llm4s.providers.internal-gateway {
provider = "openai-compatible"
model = "llama-3-70b"
baseUrl = "https://llm.internal.corp/v1"
http.proxy.url = "" # no proxy for this section
}
```
An HTTP client of your own - a tool, a vector store, an MCP transport - takes the same settings through
`Llm4sHttpClient.create`:
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.http.Llm4sHttpClient
val routedClient = Llm4sConfig.httpNetwork().map(network => Llm4sHttpClient.create(network))
```
Java builders take the same settings with `proxy`, `proxyCredentials`, `nonProxyHosts`, `trustStore` and `keyStore`
(see [the Java guide](../guide/java#behind-a-proxy-or-a-corporate-ca)); the Spring Boot starter binds the
`llm4s.http.*` properties (`llm4s.http.proxy.url`, `llm4s.http.proxy.non-proxy-hosts`, `llm4s.http.tls.trust-store`,
...).
What is not covered:
- **PAC files and Kerberos/NTLM proxy authentication.** Configure the JVM instead: `-Djava.net.useSystemProxies=true`
for the operating system's proxy settings, and the JDK's `jdk.http.auth.*` properties for the schemes it supports.
- **Basic proxy authentication over an `https` tunnel in the JDK client.** The JDK disables Basic for `CONNECT` by
default; start the JVM with `-Djdk.http.auth.tunneling.disabledSchemes=""` to allow it. The `openai-java`,
`anthropic-java` and AWS SDK clients authenticate the tunnel without it.
- **Bedrock's credential lookups** (instance metadata, STS) use the AWS SDK's own clients and are not routed through
`llm4s.http`; the AWS SDK reads `HTTPS_PROXY` for those itself.
- **Clients you build yourself** with `Llm4sHttpClient.create()` keep the JVM's defaults; pass the settings as above.
---
## Switching providers
Your code calls `Llm4sConfig.defaultProvider()` and does not change. To use a different
provider, point `llm4s.providers.provider` at another section:
- **Edit the file**: `provider = "claude"`.
- **Bind it to a variable of your own**:
```hocon
llm4s.providers {
provider = "openai-main"
provider = ${?LLM4S_PROVIDER} # your binding; any variable name works
}
```
then `export LLM4S_PROVIDER=claude`.
- **Override it for one run**: `sbt -Dllm4s.providers.provider=claude run`, or
`java -Dllm4s.providers.provider=claude -jar app.jar`.
- **Load a section by name** instead of the default: `Llm4sConfig.provider("claude")`. This is
how an application uses several providers at once.
For per-environment setups (local Ollama in development, a cloud provider in production), keep
one file per environment - `application.conf` plus `prod.conf` that starts with
`include "application.conf"` - and pick one with `-Dconfig.resource=prod.conf`. This is a
matter of taste, not a requirement: a section an environment has no key for does no harm
there unless it is loaded.
---
## Using a config your application already has
`Llm4sConfig`'s methods read the process-wide `application.conf`. When your application already
holds its configuration as a Typesafe `Config` - Play's `Configuration.underlying`, a Pekko
`ActorSystem.settings.config`, one config per tenant, or one parsed per test case - pass it to
`Llm4sConfig.from`. It returns an `Llm4sSettings` with the same methods (`defaultProvider()`,
`provider(name)`, `providers()`, `embeddings()`, `tracing()`, `modelRegistryService()`, ...):
```scala
import com.typesafe.config.{ Config, ConfigFactory }
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.{ LLMClient, LLMConnect }
import org.llm4s.types.Result
// In a Play or Pekko app this is the Config the framework already loaded.
val appConfig: Config = ConfigFactory.parseString(
"""llm4s.providers {
| provider = "openai-main"
| openai-main { provider = "openai", model = "gpt-4o-mini" }
|}""".stripMargin
)
val llmClient: Result[LLMClient] = for {
settings <- Llm4sConfig.from(appConfig)
registry <- settings.modelRegistryService()
cfg <- settings.defaultProvider()
client <- LLMConnect.getClient(cfg)(using registry)
} yield client
```
The given config takes the place of `application.conf` and nothing else changes, exactly as
`ConfigFactory.load(config)` would: JVM system properties still override it, every llm4s
module's `reference.conf` still sits beneath it - so the shared `llm4s.credentials..apiKey`
fallback (`OPENAI_API_KEY` above) and each module's defaults apply - and `${?VAR}`
substitutions are resolved against the process environment. A config that is already loaded
and resolved is accepted as is. Two `Llm4sSettings` in one JVM never see each other's
configuration. `from` returns a `Left(ConfigurationError)` when the config cannot be resolved,
such as a `${VAR}` substitution with no value.
`Llm4sConfig.from` and `Llm4sSettings` are `@Experimental`: their shape may still change before
1.0.
---
## Running the samples
The repository's samples read `modules/samples/src/main/resources/application.conf`, which
defines one section, `ollama-local`, as the default, and binds three variables of its own:
| Variable | Bound by the samples' `application.conf` to |
|---|---|
| `LLM4S_PROVIDER` | `llm4s.providers.provider` - which section to use |
| `OLLAMA_MODEL` | `llm4s.providers.ollama-local.model` (default `llama3:latest`) |
| `OLLAMA_BASE_URL` | `llm4s.providers.ollama-local.baseUrl` (default `http://localhost:11434`) |
These are the samples' bindings, not the library's: your own application reads them only if
its `application.conf` binds them.
With the default section, the model must be in your Ollama install: run `ollama pull llama3`
once, or name a model you already have (`ollama list`) with `export OLLAMA_MODEL=`.
To run a sample against another provider, add a section to
`modules/samples/src/main/resources/application.local.conf` - git-ignored and included by the
samples' `application.conf` - and select it:
```hocon
# modules/samples/src/main/resources/application.local.conf
llm4s.providers {
openai-main {
provider = "openai"
model = "gpt-4o-mini"
}
}
```
```bash
export OPENAI_API_KEY=sk-...
export LLM4S_PROVIDER=openai-main
sbt "samples/runMain org.llm4s.samples.basic.BasicLLMCallingExample"
```
The chat-tui sample is the exception: `ChatTuiConfig` reads `LLM_MODEL=/` and
the matching API-key variable itself, and falls back to `Llm4sConfig.defaultProvider()` when
`LLM_MODEL` is unset.
---
## Embeddings Configuration
LLM4S supports multiple embedding providers for RAG (Retrieval-Augmented Generation) workflows.
### Unified Format (Recommended)
Embeddings are configured separately from chat providers, under `llm4s.embeddings`. Select
the provider and model with `llm4s.embeddings.model` in `provider/model-name` form, which core's
`reference.conf` binds to `EMBEDDING_MODEL`:
```bash
# OpenAI embeddings (the same OPENAI_API_KEY the chat sections use)
EMBEDDING_MODEL=openai/text-embedding-3-small
OPENAI_API_KEY=sk-...
# Voyage AI embeddings
EMBEDDING_MODEL=voyage/voyage-3
VOYAGE_API_KEY=pa-...
# Jina AI embeddings
EMBEDDING_MODEL=jina/jina-embeddings-v3
JINA_API_KEY=jina_...
# Cohere embeddings
EMBEDDING_MODEL=cohere/embed-english-v3.0
COHERE_API_KEY=...
# Ollama embeddings (local, no API key needed)
EMBEDDING_MODEL=ollama/nomic-embed-text
```
The key follows the same [order](#api-keys) as a chat section's: `llm4s.embeddings..apiKey`
if you set it, otherwise the vendor's shared `llm4s.credentials..apiKey`. So one
`OPENAI_API_KEY` serves both OpenAI chat and OpenAI embeddings. To bill embeddings to another
account, give the block its own key:
```hocon
llm4s.embeddings.openai.apiKey = ${?OPENAI_EMBEDDINGS_API_KEY}
```
Each embedding provider comes from its module: `openai` from `llm4s-openai`, `voyage` from
`llm4s-voyage`, `jina` from `llm4s-jina`, `cohere` from `llm4s-cohere` and `ollama` from `llm4s-ollama` (in `0.4.1` and earlier, `openai` and `voyage` are
inside `llm4s-core`). Without the module, the provider id fails with an error naming the
registered embedding providers.
Default base URLs are used automatically:
- OpenAI: `https://api.openai.com/v1`
- Voyage: `https://api.voyageai.com/v1`
- Jina: `https://api.jina.ai/v1`
- Cohere: `https://api.cohere.com` (the API root; the provider posts to `/v2/embed`)
- Ollama: `http://localhost:11434`
Override base URLs if needed - each module's `reference.conf` binds a variable for it:
```bash
OPENAI_EMBEDDING_BASE_URL=https://custom.openai.com/v1 # llm4s-openai
VOYAGE_EMBEDDING_BASE_URL=https://custom.voyage.ai/v1 # llm4s-voyage
JINA_EMBEDDING_BASE_URL=https://custom.jina.ai/v1 # llm4s-jina
COHERE_EMBEDDING_BASE_URL=https://custom.cohere.com # llm4s-cohere
OLLAMA_EMBEDDING_BASE_URL=http://embeddings-host:11434 # llm4s-ollama
```
### Available Models
**OpenAI:**
- `text-embedding-3-small` - Fast, cost-effective (1536 dimensions)
- `text-embedding-3-large` - Higher quality (3072 dimensions)
- `text-embedding-ada-002` - Legacy model (1536 dimensions)
**Voyage AI:**
- `voyage-3` - General purpose
- `voyage-3-large` - Higher quality
- `voyage-code-2` - Code-optimized
Voyage embeds queries and documents differently. Every request is sent with an `input_type` of `document`
or `query`, taken from the request's `purpose` (see [Queries and documents](#queries-and-documents) below).
Requests that do not say get `document`.
**Jina AI:**
- `jina-embeddings-v3` - Multilingual, 8192-token context (1024 dimensions)
- `jina-embeddings-v4` - (2048 dimensions)
Jina embeds queries and documents differently. The provider built from `EMBEDDING_MODEL` follows each
request's `purpose` (see [Queries and documents](#queries-and-documents) below): a document is sent with the
`retrieval.passage` task and a query with `retrieval.query`. To send one task whatever the request says, for
example `text-matching`, build the provider with the typed `JinaTask` setting:
`JinaEmbeddingProvider.fromConfig(config, JinaTask.TextMatching)`. An explicit task wins over the purpose.
**Cohere:**
- `embed-v4.0` - (1536 dimensions)
- `embed-english-v3.0`, `embed-multilingual-v3.0` - (1024 dimensions)
- `embed-english-light-v3.0`, `embed-multilingual-light-v3.0` - (384 dimensions)
Cohere embeds queries and documents differently. The provider built from `EMBEDDING_MODEL` follows each
request's `purpose` (see [Queries and documents](#queries-and-documents) below): a document is sent with
`input_type` `search_document` and a query with `search_query`. To send one input type whatever the request
says, for example `classification`, build the provider with the typed `CohereInputType` setting:
`CohereEmbeddingProvider.fromConfig(config, CohereInputType.Classification)`. An explicit input type wins over
the purpose. Texts are sent in requests of at most 96, Cohere's limit, and the vectors keep the model's default
size.
**Ollama (local):**
- `nomic-embed-text` - General purpose (768 dimensions)
- `mxbai-embed-large` - Higher quality (1024 dimensions)
- `all-minilm` - Lightweight (384 dimensions)
**Install Ollama embedding model:**
```bash
ollama pull nomic-embed-text
```
### Legacy Format (Still Supported)
The legacy format using `EMBEDDING_PROVIDER` is still supported for backward compatibility:
```bash
# OpenAI
EMBEDDING_PROVIDER=openai
OPENAI_EMBEDDING_BASE_URL=https://api.openai.com/v1
OPENAI_EMBEDDING_MODEL=text-embedding-3-small
OPENAI_API_KEY=sk-...
# Voyage AI
EMBEDDING_PROVIDER=voyage
VOYAGE_EMBEDDING_BASE_URL=https://api.voyageai.com/v1
VOYAGE_EMBEDDING_MODEL=voyage-3
VOYAGE_API_KEY=pa-...
# Ollama
EMBEDDING_PROVIDER=ollama
OLLAMA_EMBEDDING_BASE_URL=http://localhost:11434
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
```
### Using Embeddings in Code
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.EmbeddingClient
import org.llm4s.llmconnect.config.EmbeddingModelConfig
import org.llm4s.agent.memory.LLMEmbeddingService
import org.llm4s.model.ModelRegistryService
// Core logic depends on injected service
class RAGService(embeddingService: LLMEmbeddingService) {
def processDocuments(docs: Seq[String]): Unit = {
// Use embeddingService...
}
}
// Configuration boundary
object RAGApplication extends App {
val startup = for {
// 1. Load config
embeddingConfig <- Llm4sConfig.embeddings()
(provider, cfg) = embeddingConfig
model <- Llm4sConfig.textEmbeddingModel()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
// 2. Build dependencies
client <- EmbeddingClient.from(provider, cfg)
// 3. Create service
service = LLMEmbeddingService(
client,
EmbeddingModelConfig(model.modelName, model.dimensions)
)
} yield new RAGService(service)
startup.fold(
err => println(s"Startup failed: $err"),
service => println("RAG Service started successfully")
)
}
```
### Queries and documents
Several embedding models embed a search query differently from a document, so a query only finds what was
indexed if the two were embedded on matching sides. An `EmbeddingRequest` says which it is with its `purpose`:
```scala
import org.llm4s.llmconnect.model.{ EmbeddingRequest, InputPurpose }
val indexing = EmbeddingRequest(chunks, model) // a document: the default
val searching = EmbeddingRequest(Seq(question), model, InputPurpose.Query) // a query
```
Each provider maps the purpose onto its own parameter:
| Provider | `Document` | `Query` |
|---|---|---|
| Voyage | `input_type` `document` | `input_type` `query` |
| Jina | `task` `retrieval.passage` | `task` `retrieval.query` |
| Cohere | `input_type` `search_document` | `input_type` `search_query` |
| OpenAI, Ollama | nothing sent: these models embed both alike | nothing sent |
`RAG` and the benchmark `RAGPipeline` already do this: what they ingest is embedded as documents and the
question they answer as a query. Code that does not say keeps embedding documents.
- **An explicit setting wins.** A Jina task or a Cohere input type passed to the provider's `fromConfig` is
sent for every request, whatever its purpose, because it can be one the purpose cannot express
(`text-matching`, `classification`, `clustering`).
- **Voyage now sends `input_type`.** Before, it sent none, which Voyage treats as a plain embedding. Documents
indexed before this change were embedded that way; they still work against queries, and re-indexing makes
the two sides match exactly.
- **Caching keeps the two apart.** `CachedEmbeddingClient`'s key includes the request's purpose, so a
query and a document with the same text never share an entry (see the [caching guide](../guide/caching)).
### System Properties (Alternative)
When running via sbt, use system properties for reliable configuration:
```bash
# Using unified format (recommended)
sbt -Dllm4s.embeddings.model=ollama/nomic-embed-text \
"run"
# Or with legacy format
sbt -Dllm4s.embeddings.provider=ollama \
-Dllm4s.embeddings.ollama.model=nomic-embed-text \
"run"
```
---
## Tracing Configuration
### Console Tracing (Development)
```bash
TRACING_MODE=console
```
Console tracing is the default. It prints each event to stdout as metadata only - event type, model, ids, tool
names, durations, token counts and costs - and withholds queries, model output, tool input/output, custom event data
and error messages (`Query: [withheld: 42 chars]`). To print those too, while developing:
```bash
LLM4S_TRACING_CONSOLE_CAPTURE_CONTENT=true # llm4s.tracing.console.captureContent = true
```
Keep it off wherever stdout is collected into logs; see [Data protection](../guide/data-protection).
### Langfuse (Production)
Langfuse tracing needs the `llm4s-observability` module on the classpath:
```scala
libraryDependencies += "org.llm4s" %% "llm4s-observability" % ""
```
```bash
TRACING_MODE=langfuse
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_URL=https://cloud.langfuse.com
```
That module's `reference.conf` binds the `LANGFUSE_*` variables under
`llm4s.tracing.langfuse`, so they can equally be set there in `application.conf`. Both keys are
required: without one, `Tracing.fromSettings` returns a `ConfigurationError` naming
`llm4s.tracing.langfuse.publicKey (LANGFUSE_PUBLIC_KEY)` and/or `secretKey`, and
`Tracing.create` logs it and traces nothing. Without the module, `TRACING_MODE=langfuse` gives
an error naming the `llm4s-observability` artifact.
Get keys from [Langfuse](https://langfuse.com):
1. Sign up at langfuse.com
2. Create a project
3. Navigate to Settings → API Keys
4. Copy public and secret keys
### OpenTelemetry
OpenTelemetry provides distributed tracing for observability across your infrastructure.
**Setup:**
```bash
TRACING_MODE=opentelemetry
OTEL_SERVICE_NAME=llm4s-agent
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
```
OpenTelemetry tracing needs the `llm4s-observability-otel` module on the classpath.
**Settings** (bound in core's `reference.conf`):
| Config key | Variable | Default | Description |
|----------|----------|---------|-------------|
| `llm4s.tracing.mode` | `TRACING_MODE` | `console` | Must be `opentelemetry` |
| `llm4s.tracing.opentelemetry.serviceName` | `OTEL_SERVICE_NAME` | `llm4s-agent` | Service name for trace identification |
| `llm4s.tracing.opentelemetry.endpoint` | `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4317` | OpenTelemetry Collector endpoint |
| `llm4s.tracing.opentelemetry.headers` | (none) | (empty) | Headers sent with every export |
| `llm4s.tracing.opentelemetry.captureContent` | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` | `false` | Record prompts and completions, redacted, as `gen_ai.input.messages` / `gen_ai.output.messages` |
llm4s builds the OTLP exporter itself, so the OpenTelemetry SDK's own
`OTEL_EXPORTER_OTLP_HEADERS` variable is **not** read. Set headers in `application.conf`,
binding secrets as usual:
```hocon
llm4s.tracing.opentelemetry.headers {
Authorization = ${?OTEL_AUTH_HEADER} # e.g. "Bearer "
}
```
**Example Configuration:**
```bash
# Local Jaeger (via Docker)
TRACING_MODE=opentelemetry
OTEL_SERVICE_NAME=llm4s-agent
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
# Cloud Provider with Authentication
TRACING_MODE=opentelemetry
OTEL_SERVICE_NAME=llm4s-runner
OTEL_EXPORTER_OTLP_ENDPOINT=https://otel-collector.example.com:4317
OTEL_AUTH_HEADER="Bearer sk-otel-token" # bound by the headers block above
```
**Spans Generated:**
```
Span: LLM Completion
├─ gen_ai.request.model: gpt-4o
├─ gen_ai.usage.input_tokens: 150
└─ gen_ai.usage.output_tokens: 50
Span: Tool Execution: web_search
├─ tool.name: web_search
├─ tool.input: "Scala 3 features"
└─ duration_ms: 1234
Span: Token Usage - completion
├─ operation: completion
├─ gen_ai.usage.total_tokens: 200
└─ cost.usd: 0.0042
Span: Agent State Updated
├─ status: RUNNING
├─ message_count: 5
└─ log_count: 12
```
**Setting Up Jaeger (Local Development):**
```bash
# Start Jaeger all-in-one
docker run -d \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14268:14268 \
-p 14250:14250 \
-p 9411:9411 \
jaegertracing/all-in-one:latest
# Configure LLM4S
export TRACING_MODE=opentelemetry
export OTEL_SERVICE_NAME=llm4s-dev
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
# View traces at http://localhost:16686
```
**Code Usage:**
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.trace.{ TraceEvent, Tracing }
// Build the tracer that llm4s.tracing.mode selects. fromSettings fails if the mode's
// backend - here llm4s-observability-otel - is not on the classpath; Tracing.create
// would log that and fall back to NoOpTracing instead.
val result = for {
settings <- Llm4sConfig.tracing()
tracer <- Tracing.fromSettings(settings)
_ <- tracer.traceEvent(TraceEvent.AgentInitialized("query", Vector("tool1")))
_ <- tracer.traceTokenUsage(usage, "gpt-4o", "completion")
_ = tracer.shutdown() // always shut down to flush pending spans
} yield ()
```
### Other tracing backends
`console` and `none` are built into `llm4s-core`; every other mode is served by a
`org.llm4s.trace.spi.TracingBackend` that a module registers in
`META-INF/services/org.llm4s.trace.spi.TracingBackend` - `langfuse` by `llm4s-observability`,
`opentelemetry` by `llm4s-observability-otel`. A mode llm4s has no name for, such as
`TRACING_MODE=datadog`, is read as `TracingMode.Named("datadog")` and goes to whichever backend
declares that mode. With no such backend, `Tracing.create` logs an error listing the available
modes and traces nothing. For `langfuse` and `opentelemetry` the error also names the module to
add, since llm4s knows which of its own modules serves them:
```text
Tracing mode 'opentelemetry' is configured but no TracingBackend for it is on the classpath.
Add the 'org.llm4s' %% 'llm4s-observability-otel' dependency. Available modes: console, noop.
```
That hint is only a message: dispatch never consults it, and a third-party mode needs no entry in
it. See the
[observability guide](../guide/observability/#adding-a-tracing-backend) to write one.
### Disable Tracing
```bash
TRACING_MODE=none
```
---
## Environment variables llm4s reads
These are the variables bound by a `${?VAR}` in some llm4s module's `reference.conf`, so they
work without any configuration of your own. Each sets the config key beside it, which you can
also set in `application.conf` or with `-D`.
| Variable | Config key | Bound by |
|---|---|---|
| `TRACING_MODE` | `llm4s.tracing.mode` (`console` - the default - or `none`, both built in; `langfuse`, `opentelemetry`, or the mode of any other `TracingBackend` on the classpath) | `llm4s-core` |
| `LANGFUSE_URL`, `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY`, `LANGFUSE_ENV`, `LANGFUSE_RELEASE`, `LANGFUSE_VERSION` | `llm4s.tracing.langfuse.*` | `llm4s-observability` |
| `OTEL_SERVICE_NAME`, `OTEL_EXPORTER_OTLP_ENDPOINT` | `llm4s.tracing.opentelemetry.serviceName`, `.endpoint` | `llm4s-observability-otel` |
| `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` | `llm4s.tracing.opentelemetry.captureContent` (`false` by default: spans carry no prompt or completion content) | `llm4s-observability-otel` |
| `EMBEDDING_MODEL` (or legacy `EMBEDDING_PROVIDER`) | `llm4s.embeddings.model` (`.provider`) | `llm4s-core` |
| `LLM4S_TRACING_CONSOLE_CAPTURE_CONTENT` | `llm4s.tracing.console.captureContent` (`false` by default: console tracing prints no prompt or completion content) | `llm4s-core` |
| `LLM4S_EXCHANGE_LOGGING_ENABLED`, `LLM4S_EXCHANGE_LOGGING_DIR` | `llm4s.exchangeLogging.*` | `llm4s-core` |
| `HTTPS_PROXY`, `NO_PROXY`, `LLM4S_PROXY_USER`, `LLM4S_PROXY_PASSWORD` | `llm4s.http.proxy.url`, `.nonProxyHosts`, `.username`, `.password` ([Corporate networks](#corporate-networks-a-proxy-and-a-custom-ca)) | `llm4s-core` |
| `LLM4S_TRUSTSTORE`, `LLM4S_TRUSTSTORE_PASSWORD`, `LLM4S_KEYSTORE`, `LLM4S_KEYSTORE_PASSWORD` | `llm4s.http.tls.trustStore`, `.trustStorePassword`, `.keyStore`, `.keyStorePassword` | `llm4s-core` |
| `LLM4S_MODEL_REGISTRY_RESOURCE`, `LLM4S_MODEL_REGISTRY_FILE`, `LLM4S_MODEL_REGISTRY_URL` | `llm4s.modelRegistry.*` | `llm4s-core` |
| `WORKSPACE_DIR`, `WORKSPACE_IMAGE`, `WORKSPACE_PORT` | `llm4s.workspace.*` | `llm4s-core` |
| `BRAVE_SEARCH_API_KEY`, `EXA_API_KEY` and the other `BRAVE_*`, `EXA_*` variables, `DUCK_DUCK_GO_SEARCH_API_URL` | `llm4s.tools.*` | `llm4s-core` |
| `OPENAI_API_KEY` | `llm4s.credentials.openai.apiKey` (OpenAI chat sections and embeddings) | `llm4s-openai` |
| `AZURE_OPENAI_API_KEY` | `llm4s.credentials.azure.apiKey` | `llm4s-openai` |
| `REQUESTY_API_KEY` | `llm4s.credentials.requesty.apiKey` | `llm4s-openai` |
| `ANTHROPIC_API_KEY` | `llm4s.credentials.anthropic.apiKey` | `llm4s-anthropic` |
| `GOOGLE_API_KEY`, else `GEMINI_API_KEY` | `llm4s.credentials.gemini.apiKey` | `llm4s-gemini` |
| `DEEPSEEK_API_KEY`, `ZAI_API_KEY`, `OPENROUTER_API_KEY`, `MISTRAL_API_KEY`, `XAI_API_KEY` | `llm4s.credentials..apiKey` | `llm4s-openai-compatible` |
| `COHERE_API_KEY` | `llm4s.credentials.cohere.apiKey` (Cohere chat, the Cohere reranker and Cohere embeddings) | `llm4s-openai-compatible`, `llm4s-rag`, `llm4s-cohere` |
| `VOYAGE_API_KEY` | `llm4s.credentials.voyage.apiKey` | `llm4s-voyage` |
| `JINA_API_KEY` | `llm4s.credentials.jina.apiKey` | `llm4s-jina` |
| `TYPESAFE_API_KEY` | `llm4s.credentials.typesafe.apiKey` (the Jev decision client) | `llm4s-jev` |
| `TYPESAFE_BASE_URL`, `TYPESAFE_DEFAULT_MODEL` | `llm4s.jev.baseUrl`, `llm4s.jev.model` | `llm4s-jev` |
| `OPENAI_EMBEDDING_BASE_URL`, `OPENAI_EMBEDDING_MODEL` | `llm4s.embeddings.openai.*` | `llm4s-openai` |
| `VOYAGE_EMBEDDING_BASE_URL`, `VOYAGE_EMBEDDING_MODEL` | `llm4s.embeddings.voyage.*` | `llm4s-voyage` |
| `JINA_EMBEDDING_BASE_URL`, `JINA_EMBEDDING_MODEL` | `llm4s.embeddings.jina.*` | `llm4s-jina` |
| `COHERE_EMBEDDING_BASE_URL`, `COHERE_EMBEDDING_MODEL` | `llm4s.embeddings.cohere.*` | `llm4s-cohere` |
| `OLLAMA_EMBEDDING_BASE_URL`, `OLLAMA_EMBEDDING_MODEL` | `llm4s.embeddings.ollama.*` | `llm4s-ollama` |
| `RERANK_PROVIDER`, `COHERE_RERANK_BASE_URL`, `COHERE_RERANK_MODEL` | `llm4s.rerank.*`, read by `RerankerConfigLoader` | `llm4s-rag` |
| `LLM4S_CHECKPOINTER` | `llm4s.agent.checkpointer.kind` (`memory` - the default, built in - `sqlite`, `postgres`, or the kind of any other `CheckpointerBackend` on the classpath) | `llm4s-agent` |
| `LLM4S_RUNTIME_LABEL` | `llm4s.agent.runtime.claims.label` (names the runtime holding a thread's claim in the logs) | `llm4s-agent` |
| `LLM4S_WORKER_LABEL` | `llm4s.agent.worker.label` (names an `AgentWorker` in its queue leases and logs) | `llm4s-agent` |
| `LLM4S_CHECKPOINTER_SQLITE_PATH` | `llm4s.agent.checkpointer.sqlite.path` | `llm4s-agent-checkpoint-sqlite` |
| `LLM4S_CHECKPOINTER_POSTGRES_URL`, `LLM4S_CHECKPOINTER_POSTGRES_USER`, `LLM4S_CHECKPOINTER_POSTGRES_PASSWORD` | `llm4s.agent.checkpointer.postgres.url`, `.user`, `.password` | `llm4s-agent-checkpoint-postgres` |
| `PGVECTOR_HOST`, `PGVECTOR_PORT`, `PGVECTOR_DATABASE`, `PGVECTOR_USER`, `PGVECTOR_PASSWORD`, `PGVECTOR_TABLE`, ... | `llm4s.rag.permissions.pg.*` | `llm4s-rag` |
| `ELASTICSEARCH_URL`, `ELASTICSEARCH_INDEX`, `ELASTICSEARCH_API_KEY`, `ELASTICSEARCH_USERNAME`, `ELASTICSEARCH_PASSWORD` | `llm4s.vectorstore.elasticsearch.*`, read by `ElasticsearchVectorStoreConfigLoader` | `llm4s-vectorstore-elasticsearch` |
| `OPENSEARCH_URL`, `OPENSEARCH_INDEX`, `OPENSEARCH_USERNAME`, `OPENSEARCH_PASSWORD`, `OPENSEARCH_TOKEN` | `llm4s.vectorstore.opensearch.*`, read by `OpenSearchVectorStoreConfigLoader` | `llm4s-vectorstore-elasticsearch` |
| `REDIS_HOST`, `REDIS_PORT`, `REDIS_TLS`, `REDIS_USERNAME`, `REDIS_PASSWORD`, `REDIS_VECTOR_INDEX` | `llm4s.vectorstore.redis.*`, read by `RedisVectorStoreConfigLoader` | `llm4s-vectorstore-redis` |
**Not read by llm4s** unless your `application.conf` binds them: `LLM_MODEL`, `LLM4S_PROVIDER`,
every chat provider's endpoint and model variable (`AZURE_API_BASE`, `AZURE_API_VERSION`,
`OPENAI_BASE_URL`, `OLLAMA_BASE_URL`, ...), `AZURE_API_KEY` (the Azure key is
`AZURE_OPENAI_API_KEY`, the openai SDK's name), `CO_API_KEY`, anything for the generic
`openai-compatible` provider, and `OTEL_EXPORTER_OTLP_HEADERS`. A shared key sets only `apiKey`:
`baseUrl`, `endpoint`, `model` and `apiVersion` always come from the section. `LLM_MODEL` was
removed with legacy single-provider loading in [#903](https://github.com/llm4s/llm4s/issues/903).
The embeddings sample's variables - `EMBEDDING_INPUT_PATH`, `EMBEDDING_INPUT_PATHS`, `EMBEDDING_QUERY`, `CHUNK_SIZE`,
`CHUNK_OVERLAP`, `CHUNKING_ENABLED`, `MAX_ROWS_PER_FILE`, `TOP_DIMS_PER_ROW`, `GLOBAL_TOPK`, `SHOW_GLOBAL_TOP`, `COLOR`,
`TABLE_WIDTH`, `ENABLE_EXPERIMENTAL_STUBS` and `LLM4S_EMBEDDINGS_LOCAL_{IMAGE,AUDIO,VIDEO}_MODEL` - are bound by the
samples' own `application.conf` under `llm4s.samples.embeddings`, not by a library module
([#2265](https://github.com/llm4s/llm4s/issues/2265)).
The example agent service's variables - `PORT`, `LLM4S_SERVICE_MODEL`, `LLM4S_SERVICE_TOOL_DELAY` and
`LLM4S_SERVICE_METRICS` - are bound by `modules/deploy-service`'s own `reference.conf` (unpublished) under
`llm4s.deploy-service`; see [`deploy/README.md`](https://github.com/llm4s/llm4s/blob/main/deploy/README.md).
---
## Environment-specific configuration
Keep the provider structure in HOCON and inject secrets from the environment. One
`application.conf` can serve every environment, since only the selected section's variables need
to be set, or you can keep one file per environment:
```hocon
# src/main/resources/application.conf - development
llm4s {
providers {
provider = "ollama-local"
ollama-local {
provider = "ollama"
model = "llama3.2"
baseUrl = "http://localhost:11434"
}
}
tracing.mode = "console"
}
```
```hocon
# src/main/resources/prod.conf - production
include "application.conf"
llm4s {
providers = null # drop the development sections
providers {
provider = "claude"
claude {
provider = "anthropic"
model = "claude-sonnet-4-20250514"
apiKey = ${?ANTHROPIC_API_KEY} # explicit in production: which account this bills
}
}
tracing.mode = "langfuse" # llm4s-observability; LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY from the environment
}
```
```bash
java -Dconfig.resource=prod.conf -jar app.jar
```
`providers = null` matters: without it the included `ollama-local` section would stay, and be
validated, in production.
---
## Configuration Best Practices
### ✅ DO
1. **Keep secrets in environment variables** - the vendor's own variable (`OPENAI_API_KEY`), or
one you bind with `apiKey = ${?VAR}` in a section; in production, give each section its own
2. **Keep structure in `application.conf`**: provider sections, models, defaults
3. **Add `.env` files to `.gitignore`** if you use them to export variables
4. **Use different configs** for dev/staging/prod
5. **Validate configuration** at startup
```scala
import org.llm4s.config.Llm4sConfig
object ApplicationBoundary {
def validateAndStart(): Unit = {
// Fail fast at startup if config is invalid
Llm4sConfig.defaultProvider().fold(
error => {
Console.err.println(s"FATAL: Configuration error: ${error.formatted}")
System.exit(1)
},
config => {
println(s"Configuration loaded for model: ${config.model}")
// startApplication(config)
}
)
}
}
```
### ❌ DON'T
1. **Don't hardcode API keys** in source code or `application.conf`
2. **Don't commit .env files** to version control
3. **Don't use System.getenv() directly** (use Llm4sConfig)
4. **Don't mix production keys** in development
5. **Don't skip validation** - fail fast on bad config
---
## Validation Example
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.model.ModelRegistryService
object ValidateConfig extends App {
println("Validating LLM4S configuration...")
// Perform validation at the application edge
val validationResult = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
} yield (providerConfig, client)
validationResult match {
case Right((config, client)) =>
println(s"✅ Model: ${config.model}")
println(s"✅ Provider: ${config.getClass.getSimpleName}")
println(s"✅ Client ready: ${client.getClass.getSimpleName}")
println("✅ Configuration valid!")
case Left(error) =>
Console.err.println(s"❌ Configuration error: ${error.formatted}")
System.exit(1)
}
}
```
---
## Troubleshooting
### Problem: "missing required fields: apiKey"
**Symptoms:**
```
ConfigurationError: Provider 'openai-main' (provider = openai) is missing required fields:
- apiKey: set OPENAI_API_KEY, or set apiKey under llm4s.providers.openai-main in application.conf
```
**Root causes:**
1. Neither the vendor's variable (`OPENAI_API_KEY`) nor a section `apiKey` is set in this process
2. The provider's module is missing, so nothing binds its variable (see the table in
[Named provider sections](#named-provider-sections))
3. The section binds its own variable (`apiKey = ${?OTHER_VAR}`) that is unset, and the
vendor's variable is unset too
4. `.env` file not loaded in the shell that starts the JVM
The error names the section it is about, and only a section being loaded is validated. If it
names a section you did not ask for, check `llm4s.providers.provider` (and any
`-Dllm4s.providers.provider` override): that is the section `defaultProvider()` loads. The one
call that validates every section is `Llm4sConfig.providers()`.
**Debug steps:**
```bash
# Is the vendor's variable set in this shell?
echo $OPENAI_API_KEY
# Does the section set a key of its own? (an apiKey line wins over OPENAI_API_KEY)
grep -n apiKey src/main/resources/application.conf
```
### Problem: "Configured provider '...' was not found"
**Symptoms:**
```
ConfigurationError: Configured provider 'openai-main' was not found
```
**Fix:** `llm4s.providers.provider` (or the name passed to `Llm4sConfig.provider(...)`) must
match a section name exactly. Check for a typo, and for an `LLM4S_PROVIDER`-style override
selecting a section this file does not define. If `llm4s.providers.provider` is absent
altogether, `defaultProvider()` fails with "No default provider configured under
llm4s.providers.provider" - set it, or call `Llm4sConfig.provider("")`.
### Problem: "Unknown provider" / provider not registered
**Symptoms:** an error naming `llm4s.providers..provider` and listing the registered
providers.
**Fix:** the section's `provider = "..."` id comes from a module that is not on the classpath.
Add it (see the table in [Named provider sections](#named-provider-sections)), or fix the id's
spelling.
### Problem: "Configuration not loading from application.conf"
**Symptoms:** `defaultProvider()` behaves as if your sections were not there.
**Debug:**
```scala
import com.typesafe.config.ConfigFactory
val config = ConfigFactory.load()
println(config.hasPath("llm4s.providers.provider")) // Should be true
println(config.getString("llm4s.providers.provider")) // Should name your section
```
**Common causes:**
- application.conf not in `src/main/resources/` (or not on the runtime classpath)
- `-Dconfig.file` / `-Dconfig.resource` pointing somewhere else
- HOCON syntax error (missing quotes, wrong nesting)
- A `-D` system property overriding the key (system properties have the highest precedence)
### Problem: "Provider mismatch"
**Symptoms:**
```
import org.llm4s.error.AuthenticationError
Left(AuthenticationError("Invalid API key"))
```
**Root cause:** the section's own `apiKey` binds a variable holding another provider's key, or
the vendor's variable holds the wrong key. The INFO line `llm4s.providers.: API key from
` says which place the key came from.
```hocon
# ❌ Wrong: an OpenAI section reading the Anthropic key
openai-main {
provider = "openai"
model = "gpt-4o"
apiKey = ${?ANTHROPIC_API_KEY}
}
```
### Problem: "Rate limiting / 429 errors"
**Symptoms:**
```
import org.llm4s.error.RateLimitError
Left(RateLimitError("Rate limit exceeded"))
```
**Solutions:**
1. **Add retry logic:**
```scala
import scala.concurrent.duration._
import org.llm4s.error.RateLimitError
import org.llm4s.types.Result
// Note: This example uses Thread.sleep for simplicity.
// In production, prefer scala.concurrent or cats-effect for non-blocking delays.
def retryWithBackoff[A](op: => Result[A], maxRetries: Int = 3): Result[A] = {
(1 to maxRetries).foldLeft(op) { (result, attempt) =>
result match {
case Left(_: RateLimitError) if attempt < maxRetries =>
Thread.sleep(Math.pow(2, attempt).toLong * 1000) // Exponential backoff
op
case other => other
}
}
}
```
2. **Use a cheaper model for testing** - change the section's `model`:
```hocon
openai-main {
provider = "openai"
model = "gpt-4o-mini" # development; gpt-4o in production
}
```
3. **Switch to Ollama for development** - point `llm4s.providers.provider` at an
`ollama-local` section (see [Switching providers](#switching-providers)); free, no rate limits.
### Problem: "Slow responses or timeouts"
**Symptoms:** Requests take 30+ seconds or timeout.
**Solutions:**
1. **Raise (or lower) the timeouts.** Each client has a default (see [Timeouts](#timeouts)):
two minutes for a completion, and five or ten minutes for a stream. A section's
`timeouts { request = 5m, stream = 20m }` block changes them per provider.
2. **Use streaming for long responses:**
```scala
// Non-streaming: waits for full response
client.complete(Conversation(Seq(UserMessage("Your query"))))
// Streaming: get tokens as they arrive
client.streamComplete(
Conversation(Seq(UserMessage("Your query"))),
onChunk = chunk => chunk.content.foreach(print)
)
```
3. **Reduce response length:**
```scala
client.complete(
Conversation(Seq(UserMessage("Your query"))),
CompletionOptions(maxTokens = Some(500)) // Limit response length
)
```
### Problem: "OutOfMemoryError with embeddings"
**Symptoms:** JVM crashes when processing large document collections.
**Solutions:**
1. **Batch embeddings instead of loading all at once:**
```scala
import org.llm4s.llmconnect.model.EmbeddingRequest
val documents: List[String] = loadDocuments()
val batchSize = 100
val embeddings = documents.grouped(batchSize).flatMap { batch =>
embedder.embed(EmbeddingRequest(batch, embeddingModel)) match {
case Right(response) => response.embeddings
case Left(err) =>
println(s"Batch failed: $err")
List.empty
}
}.toList
```
2. **Increase JVM heap:**
```bash
export SBT_OPTS="-Xmx4G -Xss4M"
sbt run
```
3. **Use a local embedding model (smaller memory footprint):**
```bash
EMBEDDING_MODEL=ollama/nomic-embed-text
```
---
## Next Steps
Configuration complete! Now you can:
1. **[Explore features →](next-steps)** - Dive into agents, tools, and more
2. **[Browse examples →](/examples/)** - See configuration in action
3. **[User guide →](../guide/basic-usage)** - Learn core concepts
4. **[Observability →](../guide/observability)** - Set up tracing
---
## Additional Resources
- **[Llm4sConfig API](/api/#4-explicit-configuration)** - Detailed API documentation
- **[Environment variables llm4s reads](#environment-variables-llm4s-reads)** - Complete list
- **[Production Guide](../PRODUCTION_DEPLOYMENT)** - Production best practices
- **[Discord Community](https://discord.gg/4uvTPn6qww)** - Get help with configuration
---
**Ready to build?** [Explore what's next →](next-steps)
---
Source: https://llm4s.org/getting-started/next-steps
# Next Steps
You've completed the getting started guide! Here's where to go next.
---
## 🎉 Congratulations!
You've successfully:
✅ Installed LLM4S
✅ Written your first LLM program
✅ Configured providers and API keys
✅ Understood Result-based error handling
Now let's explore what you can build with LLM4S!
---
## Learning Paths
Choose your path based on what you want to build:
### 🤖 Path 1: Build Agents
**Best for:** Interactive applications, chatbots, assistants
**What you'll learn:**
- Agent framework basics
- Multi-turn conversations
- Tool calling and integration
- Conversation state management
**Start here:**
1. [Agent Framework Guide](/examples/#agent-examples)
2. [Single-Step Agent Example](/examples/#single-step)
3. [Multi-Turn Conversations](/guide/agents/#multi-turn-conversations)
4. [Tools in the Agents Guide](/guide/agents/#agent-with-tools)
**Example project ideas:**
- Customer support chatbot
- Code review assistant
- Research assistant with web search
- Interactive game master
---
### 🛠️ Path 2: Tool Integration
**Best for:** LLMs that interact with external systems
**What you'll learn:**
- Defining custom tools
- Tool parameter schemas
- Model Context Protocol (MCP)
- Tool error handling
**Start here:**
1. [Built-in Tools Guide](/guide/builtin-tools)
2. [Weather Tool Example](/examples/#weather)
3. [MCP Tool Example](/examples/#mcp-tool)
4. [Multi-Tool Example](/examples/#multi-tool)
**Example project ideas:**
- Database query assistant
- API integration agent
- File system navigator
- Task automation system
---
### 💬 Path 3: Conversational AI
**Best for:** Chat applications, dialogue systems
**What you'll learn:**
- Context window management
- Conversation persistence
- History pruning strategies
- Streaming responses
**Start here:**
1. [Multi-Turn Conversations](/guide/agents/#multi-turn-conversations)
2. [Context Window Pruning](/guide/context-window-pruning)
3. [Streaming Events Guide](/guide/agents/streaming)
4. [Long Conversation Example](/examples/#long-conversation)
**Example project ideas:**
- Slack bot
- Discord integration
- Customer service chat
- Educational tutor
---
### 🔍 Path 4: RAG & Knowledge
**Best for:** Question answering, document search, knowledge bases
**What you'll learn:**
- Vector embeddings
- Semantic search
- Document processing
- Retrieval-augmented generation
**Start here:**
1. [Vector Store Guide](/guide/vector-store)
2. [RAG for Enterprise](/guide/patterns/rag-enterprise)
3. [Embedding Example](/examples/#embedding-example)
4. [Vector Search](/guide/vector-store#search)
**Example project ideas:**
- Documentation Q&A system
- PDF analyzer
- Knowledge base search
- Code search engine
---
### 📊 Path 5: Production Systems
**Best for:** Deploying LLM apps to production
**What you'll learn:**
- Error handling patterns
- Observability and tracing
- Performance optimization
- Security best practices
**Start here:**
1. [Production Monitoring](/guide/patterns/production-monitoring)
2. [Observability Guide](../guide/observability)
3. [Error Handling](/guide/error-handling)
4. [Security Guide](/guide/patterns/security-best-practices)
**Example project ideas:**
- Scalable API service
- Multi-tenant SaaS application
- Enterprise integration
- Monitoring dashboard
---
## Quick Reference: Key Features
The snippets in this section assume a `client: LLMClient` (built as in [your first program](first-example)) and,
for agents, a `tools: ToolRegistry`.
### Agents
Build sophisticated multi-turn agents with automatic tool calling.
```scala
import org.llm4s.agent.Agent
val agent = Agent.builder("assistant", client).withTools(tools).build() // Result[Agent]
val result = agent.flatMap(_.run("Your query"))
```
[Learn more →](/examples/#agent-examples)
---
### Tool Calling
Give LLMs access to external functions and APIs.
```scala
import org.llm4s.toolapi.{ Schema, ToolBuilder }
def search(query: String): String = s"Results for $query" // your implementation
val searchSchema = Schema
.`object`[Map[String, Any]]("Search parameters")
.withProperty(Schema.property("query", Schema.string("What to search for")))
val tool = ToolBuilder[Map[String, Any], String]("search", "Search the web", searchSchema)
.withHandler(params => params.getString("query").map(search))
.buildSafe() // Result[ToolFunction[Map[String, Any], String]]
```
[Learn more →](/guide/builtin-tools)
---
### Multi-Turn Conversations
Functional conversation management without mutation.
```scala
import org.llm4s.agent.Agent
val turns = for {
agent <- Agent.builder("assistant", client).withTools(tools).build()
result1 <- agent.run("First question")
result2 <- agent.continueConversation(result1, "Next question")
} yield result2
```
[Learn more →](/guide/agents/#multi-turn-conversations)
---
### Context Management
Automatically manage token windows and prune history.
```scala
import org.llm4s.agent.{ ContextWindowConfig, PruningStrategy }
val config = ContextWindowConfig(
maxMessages = Some(20),
pruningStrategy = PruningStrategy.OldestFirst
)
```
[Learn more →](/guide/context-window-pruning)
---
### Streaming
Get real-time token-by-token responses.
```scala
import org.llm4s.llmconnect.model.{ Conversation, UserMessage }
val conversation = Conversation(Seq(UserMessage("Tell me a story")))
client.streamComplete(conversation, onChunk = chunk => chunk.content.foreach(print))
```
[Learn more →](/guide/agents/streaming)
---
### Observability
Trace LLM calls with Langfuse integration.
```bash
# Automatic tracing when configured (with llm4s-observability on the classpath)
export TRACING_MODE=langfuse
```
[Learn more →](../guide/observability)
---
### Embeddings
Create and search vector embeddings.
`EmbeddingClient` (in `llm4s-core`) turns text into vectors through an embedding provider module, and
`llm4s-rag` indexes and searches them in a vector store.
[Learn more →](/guide/vector-store)
---
### MCP Integration
Connect to external Model Context Protocol servers.
```scala
import org.llm4s.mcp.{ MCPServerConfig, MCPToolRegistry }
// llm4s-mcp: an MCPToolRegistry is a ToolRegistry, so an agent takes it like any other
val tools = new MCPToolRegistry(Seq(MCPServerConfig.stdio("files", Seq("npx", "my-mcp-server"))))
```
[Learn more →](/examples/#mcp-examples)
---
## Example Gallery
Browse the working examples organized by category:
### Basic Examples
- [Basic LLM Calling](/examples/#basic-llm-calling)
- [Streaming Responses](/examples/#streaming)
- [Multi-Provider Setup](/guide/basic-usage#multi-provider-pattern)
- [Ollama (Local Models)](/examples/#ollama)
- [Tracing Integration](https://github.com/llm4s/llm4s/blob/main/modules/samples/src/main/scala/org/llm4s/samples/basic/BasicLLMCallingWithTrace.scala)
[View all basic examples →](/examples/#basic-examples)
### Agent Examples
- [Single-Step Agent](/examples/#single-step)
- [Multi-Step Agent](/examples/#multi-step)
- [Multi-Turn Conversations](/examples/#multi-turn)
- [Long Conversations](/examples/#long-conversation)
- [Conversation Persistence](/examples/#persistence)
- [MCP Agent](https://github.com/llm4s/llm4s/blob/main/modules/samples/src/main/scala/org/llm4s/samples/agent/MCPAgentExample.scala)
[View all agent examples →](/examples/#agent-examples)
### Tool Examples
- [Weather Tool](/examples/#weather)
- [Multi-Tool Agent](/examples/#multi-tool)
- [Tool Error Messages](https://github.com/llm4s/llm4s/blob/main/modules/samples/src/main/scala/org/llm4s/samples/toolapi/ErrorMessageDemonstration.scala)
- [MCP Tools](/examples/#mcp-tool)
[View all tool examples →](/examples/#tool-examples)
### Context Management Examples
- [Context Pipeline](/examples/#context-pipeline)
- [Token Windows](https://github.com/llm4s/llm4s/blob/main/modules/samples/src/main/scala/org/llm4s/samples/context/tokens/TokenWindowExample.scala)
- [History Digest](https://github.com/llm4s/llm4s/blob/main/modules/samples/src/main/scala/org/llm4s/samples/context/HistoryDigestExample.scala)
- [Compression](https://github.com/llm4s/llm4s/blob/main/modules/samples/src/main/scala/org/llm4s/samples/context/DeterministicCompressionExample.scala)
- [Tool Externalization](https://github.com/llm4s/llm4s/blob/main/modules/samples/src/main/scala/org/llm4s/samples/context/ToolExternalizationExample.scala)
[View all context examples →](/examples/#context-management)
### More Examples
- [Embeddings](/examples/#embeddings)
- [MCP Integration](/examples/#mcp-examples)
- [Streaming](/examples/#streaming-examples)
[Browse all examples →](/examples/)
---
## Common Recipes
### Recipe 1: Simple Q&A Bot
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model._
import org.llm4s.model.ModelRegistryService
def askQuestion(question: String): String = {
val result = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
response <- client.complete(
Conversation(
Seq(
SystemMessage("You are a helpful Q&A assistant."),
UserMessage(question)
)
)
)
} yield response.content
result.fold(_ => "Sorry, I couldn't process that question.", identity)
}
```
### Recipe 2: Agent with Custom Tools
```scala
import org.llm4s.agent.Agent
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.model.ModelRegistryService
import org.llm4s.toolapi.{ Schema, ToolBuilder, ToolRegistry }
def getCurrentTime(): String =
java.time.LocalDateTime.now().toString
val result = for {
timeTool <- ToolBuilder[Map[String, Any], String](
"get_time",
"Get current date and time",
Schema.`object`[Map[String, Any]]("No parameters")
).withHandler(_ => Right(getCurrentTime())).buildSafe()
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
agent <- Agent.builder("time-agent", client).withTools(new ToolRegistry(Seq(timeTool))).build()
result <- agent.run("What time is it?")
} yield result.answer
```
### Recipe 3: Streaming Chat
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model.{ Conversation, UserMessage }
import org.llm4s.model.ModelRegistryService
def streamChat(message: String): Unit = {
val result = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
_ <- client.streamComplete(
Conversation(Seq(UserMessage(message))),
onChunk = chunk => chunk.content.foreach(print)
)
} yield println()
result.left.foreach(error => println(s"Error: $error"))
}
```
### Recipe 4: Multi-Turn with Pruning
```scala
import org.llm4s.agent.{ Agent, ContextWindowConfig, PruningStrategy }
import org.llm4s.agent.graph.middleware.ContextWindowMiddleware
val config = ContextWindowConfig(
maxMessages = Some(20),
preserveSystemMessage = true,
pruningStrategy = PruningStrategy.OldestFirst
)
// Pruning is middleware on the agent: it trims what each model call is sent,
// and the thread keeps the full history (client and tools as in the quick reference above)
val result = for {
agent <- Agent.builder("assistant", client)
.withTools(tools)
.withMiddleware(ContextWindowMiddleware(config))
.build()
first <- agent.run("First question") // Turn 1
second <- agent.continueConversation(first, "Follow-up question") // Turn 2, pruned when sent
} yield second
```
---
## Troubleshooting
### Common Issues
**Problem: API key errors**
- Check the vendor's variable is set in the shell that runs your app: `echo $OPENAI_API_KEY` - a
section without an `apiKey` of its own uses it ([API keys](configuration#api-keys))
- If the section sets its own `apiKey = ${?SOME_VAR}`, that wins: check `SOME_VAR` instead
- Check key starts with correct prefix (`sk-` for OpenAI, `sk-ant-` for Anthropic)
**Problem: Model not found**
- Verify `model` in the provider section is the provider's own model name (e.g. `gpt-4o-mini`, no `openai/` prefix)
- Check `llm4s.providers.provider` names the section you meant
- Check provider supports that model
- Try a different model
**Problem: Slow responses**
- Use streaming for real-time feedback
- Consider using a faster model (gpt-3.5-turbo, claude-haiku)
- Check your internet connection
**Problem: Token limit errors**
- Implement context window pruning
- Use shorter system prompts
- Summarize conversation history
[Full troubleshooting guide →](/reference/troubleshooting)
---
## Community & Support
### Get Help
- **Discord**: [Join our community](https://discord.gg/4uvTPn6qww) - Active community for questions
- **GitHub Issues**: [Report bugs](https://github.com/llm4s/llm4s/issues) - Bug reports and feature requests
- **Documentation**: Browse the [user guide](../guide/basic-usage) - Comprehensive guides
- **Examples**: Check [working examples](/examples/) - runnable code samples
### Stay Updated
- **GitHub**: [Star the repo](https://github.com/llm4s/llm4s) - Get notified of updates
- **Roadmap**: [View the roadmap](/reference/roadmap) - See what's coming
- **Changelog**: [Release notes](https://github.com/llm4s/llm4s/releases) - Track changes
### Contribute
- **Starter Kit**: Use [llm4s.g8](https://github.com/llm4s/llm4s.g8) to scaffold projects
- **Share Examples**: Post your projects in Discord
- **Contribute**: See the [contributing guide](/reference/contributing)
---
## Recommended Learning Order
### Week 1: Fundamentals
1. ✅ Complete Getting Started (you are here!)
2. Read [Basic Usage Guide](../guide/basic-usage)
3. Try [Basic Examples](/examples/#basic-examples)
4. Experiment with different providers
### Week 2: Agents & Tools
1. Read [Agent Framework](/examples/#agent-examples)
2. Build a simple agent with one tool
3. Try [Tool Examples](/examples/#tool-examples)
4. Add multiple tools
### Week 3: Advanced Patterns
1. Implement [Multi-Turn Conversations](/guide/agents/#multi-turn-conversations)
2. Add [Context Window Pruning](/guide/context-window-pruning)
3. Set up [Observability](../guide/observability)
4. Try [Long Conversation Example](/examples/#long-conversation)
### Week 4: Production
1. Read [Production Monitoring](/guide/patterns/production-monitoring)
2. Implement error handling
3. Add monitoring and tracing
4. Deploy your first production agent
---
## Quick Links
### Documentation
- [User Guide](../guide/basic-usage) - Feature guides
- [API Reference](/api/) - API docs
- [Advanced Topics](/advanced/) - Production topics
### Examples
- [All Examples](/examples/) - Browse all examples
- [Basic](/examples/#basic-examples) - Getting started
- [Agents](/examples/#agent-examples) - Agent patterns
- [Tools](/examples/#tool-examples) - Tool integration
### Reference
- [Configuration](/getting-started/configuration) - Setup guide
- [Migration Guide](/reference/migration) - Version upgrades
- [Roadmap](/reference/roadmap) - Future plans
---
## What to Build?
Need inspiration? Here are some project ideas:
**Beginner Projects:**
- Simple Q&A bot
- Code explainer
- Translation service
- Writing assistant
**Intermediate Projects:**
- Multi-tool research agent
- Database query interface
- API integration bot
- Document summarizer
**Advanced Projects:**
- Multi-agent system
- RAG-powered knowledge base
- Production chatbot service
- Custom tool ecosystem
---
## Ready to Build?
Pick your learning path and start building:
---
**Happy building with LLM4S!** 🚀
Questions? [Join our Discord](https://discord.gg/4uvTPn6qww)
---
Source: https://llm4s.org/getting-started/ollama-quickstart
# Ollama Quick Start Guide
Run LLM4S with Ollama for **free, local LLM inference** - no API keys required!
---
## Why Ollama?
**Ollama** is the easiest way to run large language models locally on your machine:
- ✅ **100% Free** - No API costs or rate limits
- ✅ **Private** - Your data never leaves your machine
- ✅ **Fast** - Low latency for local inference
- ✅ **Offline** - Works without internet connection
- ✅ **Multiple Models** - Easy model switching (llama2, mistral, phi, etc.)
Perfect for **development**, **testing**, and **production** workloads where privacy matters.
---
## Prerequisites
- **JDK 21**
- **Scala 3.7.x** (llm4s 0.5.1 is built with 3.7.1)
- **sbt 1.10.6+**
- **4-8GB RAM** (depending on model size)
No API keys needed! 🎉
Ollama support ships in its own artifact, alongside `llm4s-core`:
```scala
val llm4sVersion = "0.5.1"
libraryDependencies ++= Seq(
"org.llm4s" %% "llm4s-core" % llm4sVersion,
"org.llm4s" %% "llm4s-ollama" % llm4sVersion,
"org.llm4s" %% "llm4s-agent" % llm4sVersion // only for the agent under "Tool Calling"
)
```
Adding `llm4s-ollama` registers the provider; there is nothing else to wire up. Before 0.5.0, Ollama was
part of `llm4s-core`. See the
[installation guide](installation#for-ollama-local-models).
---
## Step 1: Install Ollama
### macOS / Linux
```bash
curl -fsSL https://ollama.com/install.sh | sh
```
### Windows
Download the installer from [ollama.com/download](https://ollama.com/download)
Or use PowerShell:
```powershell
# Download and install Ollama
winget install Ollama.Ollama
```
### Verify Installation
```bash
ollama --version
# Should output: ollama version is 0.x.x
```
---
## Step 2: Start Ollama Server
### Start the Server
```bash
ollama serve
```
The server will start on **http://localhost:11434** by default.
> On Windows/macOS, Ollama may start automatically as a background service. Check your system tray/menu bar.
### Verify Server is Running
```bash
curl http://localhost:11434
# Should output: Ollama is running
```
---
## Step 3: Pull a Model
Ollama models are pulled on-demand. Let's start with **Mistral 7B** (fast and capable):
```bash
ollama pull mistral
```
### Available Models
| Model | Size | RAM Required | Best For | Pull Command |
|-------|------|--------------|----------|--------------|
| **mistral** | 4.1GB | 8GB | General purpose, fast | `ollama pull mistral` |
| **llama2** | 3.8GB | 8GB | Good balance | `ollama pull llama2` |
| **phi** | 1.6GB | 4GB | Lightweight, fast | `ollama pull phi` |
| **neural-chat** | 4.1GB | 8GB | Conversational | `ollama pull neural-chat` |
| **codellama** | 3.8GB | 8GB | Code generation | `ollama pull codellama` |
| **llama3.2** | 2.0GB | 8GB | Latest Llama | `ollama pull llama3.2` |
| **gemma2** | 5.4GB | 8GB | Google's model | `ollama pull gemma2` |
> **Recommendation**: Start with `mistral` for the best balance of speed and quality.
### List Downloaded Models
```bash
ollama list
```
---
## Step 4: Configure LLM4S
LLM4S reads providers from named sections in your `application.conf`, not from environment
variables such as `LLM_MODEL` - nothing in the library reads those. See the
[Configuration Guide](configuration#named-provider-sections) for the full picture.
### Add the dependencies
```scala
// build.sbt
libraryDependencies ++= Seq(
"org.llm4s" %% "llm4s-core" % llm4sVersion,
"org.llm4s" %% "llm4s-ollama" % llm4sVersion // the Ollama provider
)
```
Before 0.5.0 the Ollama provider shipped inside `llm4s-core`, so `llm4s-core` alone was enough.
### Add an `ollama-local` section
Create `src/main/resources/application.conf`:
```hocon
llm4s {
providers {
provider = "ollama-local" # the default: the name of a section below
ollama-local {
provider = "ollama"
model = "mistral"
# baseUrl defaults to http://localhost:11434, where `ollama serve` listens
}
}
}
```
No API key is needed. If you want to take the model or server from the environment, bind the
variables yourself - the names are yours to choose:
```hocon
ollama-local {
provider = "ollama"
model = "mistral"
model = ${?OLLAMA_MODEL} # optional override from the environment
baseUrl = ${?OLLAMA_BASE_URL} # unset: the default, http://localhost:11434
}
```
---
## Step 5: Write Your First LLM4S + Ollama App
Create `HelloOllama.scala`:
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model._
import org.llm4s.model.ModelRegistryService
object HelloOllama extends App {
// Create a conversation with system and user messages
val conversation = Conversation(Seq(
SystemMessage("You are a helpful AI assistant."),
UserMessage("Explain what Scala is in one sentence.")
))
// Load the default provider section and make the request
val result = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
completion <- client.complete(conversation)
} yield completion
result match {
case Right(completion) =>
println(s"Response from ${completion.model}:")
println(completion.message.content)
case Left(error) =>
Console.err.println(s"Error: ${error.formatted}")
}
}
```
### Run It!
```bash
sbt run
```
### Example Output
```
✓ Response from mistral:
Scala is a statically-typed programming language that combines
object-oriented and functional programming paradigms, running on
the Java Virtual Machine (JVM).
```
---
## Step 6: Try Different Models
You can easily switch models by changing `model` in the `ollama-local` section (after
`ollama pull `):
```hocon
ollama-local {
provider = "ollama"
model = "llama3.2" # or "phi3", "codellama", ...
baseUrl = "http://localhost:11434"
}
```
If you added the `model = ${?OLLAMA_MODEL}` binding from Step 4, you can switch from the shell
instead - `export OLLAMA_MODEL=llama3.2` - because your own `application.conf` reads it.
Then run your program again without code changes!
---
## Step 6a: Write Your First LLM4S + Ollama/Llama3.2 App
Llama 3.2 is Meta's latest with an impressive 128K context window. Perfect for processing large documents and long conversations.
For the code, use the same Scala example from **Step 5** - simply change the model in the
`ollama-local` section of `application.conf` (run `ollama pull llama3.2` first):
```hocon
ollama-local {
provider = "ollama"
model = "llama3.2"
baseUrl = "http://localhost:11434"
}
```
Then run:
```bash
sbt run
```
> **Why Llama 3.2?** Latest Llama model with 128K context window. Excellent for RAG applications, long-form content generation, and processing large documents. Available in 1B, 3B, 8B, 70B, and 405B sizes.
---
## Step 6b: Write Your First LLM4S + Ollama/Phi3 App
Phi3 is Microsoft's efficient model, even smaller than Phi. Ideal for ultra-low-latency applications and edge deployment.
For the code, use the same Scala example from **Step 5** - simply change the model in the
`ollama-local` section of `application.conf` (run `ollama pull phi3` first):
```hocon
ollama-local {
provider = "ollama"
model = "phi3"
baseUrl = "http://localhost:11434"
}
```
Then run:
```bash
sbt run
```
> **Why Phi3?** Microsoft's compact model optimized for efficiency. Smaller than Phi (1.4GB) with competitive quality. Perfect for resource-constrained environments and real-time applications requiring minimal latency.
---
## Step 6c: Write Your First LLM4S + Ollama/CodeLlama App
CodeLlama is purpose-built for code generation and understanding. Create `HelloCodeLlama.scala`:
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model._
import org.llm4s.model.ModelRegistryService
object HelloCodeLlama extends App {
// Create a conversation asking for code
val conversation = Conversation(Seq(
SystemMessage("You are an expert Scala developer. Write clean, idiomatic code."),
UserMessage("Write a simple Scala function that reverses a list.")
))
// Load the default provider section and make the request
val result = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
completion <- client.complete(conversation)
} yield completion
result match {
case Right(completion) =>
println(s"Code suggestion from ${completion.model}:")
println(completion.message.content)
case Left(error) =>
println(s"Error: ${error.formatted}")
}
}
```
### Configure for CodeLlama
Set the model in the `ollama-local` section of `application.conf` (run `ollama pull codellama`
first):
```hocon
ollama-local {
provider = "ollama"
model = "codellama"
baseUrl = "http://localhost:11434"
}
```
### Run It!
```bash
sbt run
```
### Example Output
```
✓ Code suggestion from codellama:
def reverseList[T](list: List[T]): List[T] = {
list.reverse
}
// Or for manual reversal:
def reverseList[T](list: List[T]): List[T] = {
def helper(acc: List[T], remaining: List[T]): List[T] = {
if (remaining.isEmpty) acc
else helper(remaining.head :: acc, remaining.tail)
}
helper(Nil, list)
}
```
> **Why CodeLlama?** CodeLlama is specialized for code-related tasks. Use it for code generation, refactoring suggestions, and explaining code. 16K context window perfect for larger code files.
---
## Streaming Responses
Get real-time token streaming (like ChatGPT):
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model._
import org.llm4s.model.ModelRegistryService
object StreamingOllama extends App {
val conversation = Conversation(Seq(
SystemMessage("You are a concise assistant."),
UserMessage("Write a haiku about Scala programming.")
))
val result = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
completion <- client.streamComplete(
conversation,
CompletionOptions(),
chunk => chunk.content.foreach(print) // Print tokens as they arrive
)
} yield completion
result match {
case Right(completion) =>
println("\n--- Streaming complete! ---")
println(s"Total content: ${completion.message.content}")
case Left(error) =>
Console.err.println(s"Error: ${error.formatted}")
}
}
```
---
## Tool Calling with Ollama
Ollama supports tool calling (function calling) with compatible models, and `llm4s-ollama`'s native client sends the tools and reads the calls (streamed or not) itself - no `/v1` OpenAI-compatible workaround is needed.
Only models whose Ollama page lists the **tools** capability can do this (`llama3.1` can; the original `llama3`
does not). Sending tools to any other model fails, and the client reports it instead of dropping the tools:
```text
Invalid tools: Ollama model 'llama3:latest' does not support tool calling (the server said: registry.ollama.ai/library/llama3:latest does not support tools). Use a model whose Ollama page lists the 'tools' capability, or send no tools.
```
Pull a tool-capable model (`ollama pull llama3.1`) and set `OLLAMA_MODEL` to it:
```scala
import org.llm4s.agent.Agent
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.model.ModelRegistryService
import org.llm4s.toolapi._
import upickle.default._
object OllamaTools extends App {
// Define result type
case class WeatherResult(forecast: String)
implicit val weatherResultRW: ReadWriter[WeatherResult] = macroRW
// Define a weather tool with proper schema
val weatherSchema = Schema
.`object`[Map[String, Any]]("Weather parameters")
.withProperty(
Schema.property("location", Schema.string("City or location name"))
)
val getWeather = ToolBuilder[Map[String, Any], WeatherResult](
"get_weather",
"Get the current weather in a location",
weatherSchema
).withHandler { extractor =>
extractor.getString("location").map { location =>
// Mock implementation
WeatherResult(s"Weather in $location: Sunny, 72F")
}
}.buildSafe()
val result = for {
weatherTool <- getWeather
tools = new ToolRegistry(Seq(weatherTool))
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
agent <- Agent.builder("weather-agent", client).withTools(tools).build()
result <- agent.run("What's the weather in San Francisco?")
} yield result
result match {
case Right(r) =>
println("Final response:")
println(r.answer.getOrElse(s"Run ended: ${r.status}"))
case Left(error) =>
Console.err.println(s"Error: ${error.formatted}")
}
}
```
---
## Model Comparison
Performance comparison running on Apple M1 Mac:
| Model | Speed (tokens/sec) | Quality | Memory | Best Use Case |
|-------|-------------------|---------|--------|---------------|
| **mistral** | ~40 | ⭐⭐⭐⭐ | 8GB | General purpose, great balance |
| **llama2** | ~35 | ⭐⭐⭐⭐ | 8GB | Conversational, creative |
| **phi** | ~80 | ⭐⭐⭐ | 4GB | Quick tests, development |
| **codellama** | ~35 | ⭐⭐⭐⭐ | 8GB | Code generation |
| **neural-chat** | ~40 | ⭐⭐⭐⭐ | 8GB | Dialogue, chat apps |
> Performance varies by hardware. These are approximate values on M1 MacBook Pro 16GB RAM.
---
## Configuration Options
### Temperature Control
```scala
CompletionOptions(
temperature = 0.7, // Higher = more creative (0.0-2.0)
maxTokens = Some(1000),
topP = 0.9
)
```
### Context Length
Ollama models have different context windows:
- **mistral**: 8k tokens
- **llama2**: 4k tokens
- **llama3.2**: 128k tokens
- **codellama**: 16k tokens
- **gemma2**: 8k tokens
---
## Troubleshooting
### "Connection refused" error
**Problem**: Ollama server not running
**Solution**:
```bash
ollama serve
```
### "Model not found" error
**Problem**: Model not pulled
**Solution**:
```bash
ollama pull mistral
```
### Slow inference
**Problem**: Not enough RAM or CPU
**Solutions**:
- Use a smaller model: `ollama pull phi`
- Close other applications
- Check if running on GPU (M-series Mac, CUDA GPU)
### Model comparison
```bash
# List all models with sizes
ollama list
# Delete a model to free space
ollama rm llama2
```
---
## Running the Examples
Try the built-in LLM4S Ollama samples. The samples' own
`modules/samples/src/main/resources/application.conf` already defaults to an `ollama-local`
section (model `llama3:latest`, server `http://localhost:11434`), and binds `OLLAMA_MODEL` and
`OLLAMA_BASE_URL` to override it - those two variables are that file's bindings, not something
the library reads. See [Running the samples](configuration#running-the-samples).
The default model has to be pulled first; to use a model you already have instead, name it
with `OLLAMA_MODEL` (`ollama list` shows what is installed):
```bash
# Either pull the samples' default model once...
ollama pull llama3
# ...or override the samples' ollama-local section
export OLLAMA_MODEL=mistral
export OLLAMA_BASE_URL=http://localhost:11434
# Run basic Ollama example
sbt "samples/runMain org.llm4s.samples.basic.OllamaExample"
# Run Ollama streaming example
sbt "samples/runMain org.llm4s.samples.basic.OllamaStreamingExample"
# Run Ollama streaming example with raw provider exchange logging
sbt "samples/runMain org.llm4s.samples.basic.OllamaStreamingExample /tmp/my-provider-exchanges"
# Run tool calling example (works with any provider)
sbt "samples/runMain org.llm4s.samples.toolapi.BuiltinToolsExample"
```
The logging-enabled streaming example writes one timestamped JSONL file per run to the directory you provide. Each entry captures the raw request and response exchange, including accumulated streaming payloads for streaming-capable providers.
---
## Production Deployment
### Docker Compose Setup
```yaml
# docker-compose.yml
version: '3.8'
services:
ollama:
image: ollama/ollama:latest
ports:
- "11434:11434"
volumes:
- ollama-data:/root/.ollama
mem_limit: 8g
llm4s-app:
build: .
environment:
# Read by the app's own application.conf binding: baseUrl = ${?OLLAMA_BASE_URL}
- OLLAMA_BASE_URL=http://ollama:11434
depends_on:
- ollama
volumes:
ollama-data:
```
This assumes the app's `ollama-local` section binds the server address, as in Step 4:
`baseUrl = ${?OLLAMA_BASE_URL}`. Without the variable the section uses `http://localhost:11434`.
### Pre-pull Models
```bash
# Pull models into Docker volume
docker-compose exec ollama ollama pull mistral
docker-compose exec ollama ollama pull llama2
```
---
## Ollama vs Cloud Providers
| Feature | Ollama | OpenAI | Anthropic |
|---------|--------|--------|-----------|
| **Cost** | Free | $0.01-0.06/1K tokens | $0.003-0.015/1K tokens |
| **Privacy** | 100% local | Cloud-based | Cloud-based |
| **Speed** | Depends on hardware | Fast (API) | Fast (API) |
| **Offline** | ✅ Yes | ❌ No | ❌ No |
| **Model quality** | Good (7B-70B) | Excellent (GPT-4) | Excellent (Claude) |
| **Setup** | 5 minutes | Instant | Instant |
**Use Ollama for**:
- Development and testing
- Privacy-sensitive applications
- Cost-conscious deployments
- Offline environments
**Use cloud providers for**:
- Highest quality responses
- Scale without hardware limits
- Latest model capabilities
---
## Next Steps
- [Configuration Guide](configuration) - Named provider sections and advanced settings
- [First Example](first-example) - Build more complex agents
- [Tool Calling](../examples/) - Add custom tools
- [RAG with Ollama](../guide/vector-store) - Retrieval-augmented generation
---
## Resources
- [Ollama Official Site](https://ollama.com)
- [Ollama GitHub](https://github.com/ollama/ollama)
- [Ollama Model Library](https://ollama.com/library)
- [LLM4S Discord](https://discord.gg/4uvTPn6qww) - Get help from the community
---
> 💡 Pro Tip
>
> Use Ollama for development and testing, then switch to a cloud provider for production by
> pointing `llm4s.providers.provider` at another section - add the provider's module
> (e.g. `llm4s-openai`) and a section such as:
>
> ```hocon
> openai-main {
> provider = "openai"
> model = "gpt-4o"
> }
> ```
>
> then set `provider = "openai-main"` (or `-Dllm4s.providers.provider=openai-main`). Only the
> selected section is validated, so `OPENAI_API_KEY` is needed only where `openai-main` is used.
> See [Switching providers](configuration#switching-providers).
>
> Your code stays exactly the same! 🎉
---
---
Source: https://llm4s.org/getting-started/gradle
# Gradle Integration
Add LLM4S to a Gradle project (Java, Kotlin, or Kotlin/Ktor).
---
## Artifact names
LLM4S is published for **Scala 3 only**, and Gradle does **not** resolve Scala cross-version suffixes automatically. The coordinate is `org.llm4s:llm4s-core_3` - you must write the `_3` suffix yourself, unlike sbt where `%%` adds it.
> Releases up to `0.3.4` used unprefixed names (`core_3`); from `0.4.0` every artifact carries an `llm4s-` prefix. See the [migration guide](/reference/migration#artifact-coordinate-rename-v040).
> From `0.5.0` the provider clients, agent runtime and other features are separate artifacts (`llm4s-openai_3`, `llm4s-anthropic_3`, `llm4s-agent_3`, ...): add the modules you use alongside `llm4s-core_3`, or align them with the [BOM](installation#align-versions-with-the-bom). Up to `0.4.x` everything shipped in `llm4s-core`. The dependency notes below say which version each one applies to.
---
## Kotlin DSL (`build.gradle.kts`)
```kotlin
repositories {
mavenCentral()
}
val llm4sVersion = "0.5.1"
dependencies {
// Scala 3 artifacts - note the explicit _3 suffix
implementation("org.llm4s:llm4s-core_3:$llm4sVersion")
implementation("org.llm4s:llm4s-openai_3:$llm4sVersion") // the provider module you use
}
// Pin the Scala 3 library to avoid binary incompatibility from transitive deps.
// Match scala3-library_3 only: scala-library stays at 2.13.x and has no 3.x version.
configurations.all {
resolutionStrategy.eachDependency {
if (requested.group == "org.scala-lang" && requested.name == "scala3-library_3") {
useVersion("3.7.1")
}
}
}
```
---
## Groovy DSL (`build.gradle`)
```groovy
repositories {
mavenCentral()
}
def llm4sVersion = '0.5.1'
dependencies {
implementation "org.llm4s:llm4s-core_3:${llm4sVersion}"
implementation "org.llm4s:llm4s-openai_3:${llm4sVersion}" // the provider module you use
}
configurations.all {
resolutionStrategy.eachDependency { details ->
if (details.requested.group == 'org.scala-lang' && details.requested.name == 'scala3-library_3') {
details.useVersion '3.7.1'
}
}
}
```
---
## Dependency exclusion recipes
### Logback conflict (Spring Boot 3.x, `0.4.x` only)
`llm4s-core` `0.4.x` declares `logback-classic` as a compile dependency. Spring Boot 3.2.x defaults to `1.4.x`, and Gradle picks the highest version, so you may see _"multiple SLF4J bindings"_ warnings or a changed log format. From `0.5.0` the split modules depend on `slf4j-api` only (logback is a test dependency), so this recipe is not needed.
**Kotlin DSL:**
```kotlin
implementation("org.llm4s:llm4s-core_3:$llm4sVersion") {
exclude(group = "ch.qos.logback", module = "logback-classic")
}
```
**Groovy DSL:**
```groovy
implementation("org.llm4s:llm4s-core_3:${llm4sVersion}") {
exclude group: 'ch.qos.logback', module: 'logback-classic'
}
```
Then declare your preferred Logback version directly:
```kotlin
runtimeOnly("ch.qos.logback:logback-classic:1.4.14") // or let Spring BOM manage it
```
---
### Azure SDK exclusion (non-Azure projects, `0.4.x` only)
`llm4s-core` `0.4.x` depends on `com.azure:azure-ai-openai`, which transitively pulls the Azure SDK jars. If you do not use the Azure provider, exclude it:
**Kotlin DSL:**
```kotlin
implementation("org.llm4s:llm4s-core_3:$llm4sVersion") {
exclude(group = "com.azure", module = "azure-ai-openai")
}
```
**Groovy DSL:**
```groovy
implementation("org.llm4s:llm4s-core_3:${llm4sVersion}") {
exclude group: 'com.azure', module: 'azure-ai-openai'
}
```
The split modules on `main` have no Azure dependency.
---
### OkHttp and Kotlin standard library alignment
The OpenAI and Anthropic SDKs (`openai-java`, `anthropic-java`) both use OkHttp 4.12 and Kotlin standard library 1.9.x. If your build (for example a Ktor project) uses other versions, Gradle resolves to the highest on the classpath. That is normally harmless; if you need to pin them, use a constraint rather than an exclusion, because removing OkHttp from the SDKs breaks them:
```kotlin
dependencies {
constraints {
implementation("com.squareup.okhttp3:okhttp:4.12.0")
}
}
```
Use `./gradlew dependencyInsight --dependency okhttp --configuration runtimeClasspath` to see which version wins and why.
---
## Spring Boot BOM alignment
If you use the Spring Boot dependency-management plugin, align the BOM with your Spring Boot version. With `llm4s-core` `0.4.x`, also exclude the llm4s logback dependency:
```kotlin
plugins {
id("org.springframework.boot") version "3.2.5"
id("io.spring.dependency-management") version "1.1.5"
}
dependencies {
implementation("org.llm4s:llm4s-core_3:$llm4sVersion") {
exclude(group = "ch.qos.logback", module = "logback-classic")
}
// Spring Boot BOM manages logback version automatically
}
```
---
## Ktor setup
```kotlin
val llm4sVersion = "0.5.1"
val ktorVersion = "2.3.12"
dependencies {
implementation("io.ktor:ktor-server-core:$ktorVersion")
implementation("io.ktor:ktor-server-netty:$ktorVersion")
implementation("org.llm4s:llm4s-core_3:$llm4sVersion")
implementation("org.llm4s:llm4s-openai_3:$llm4sVersion")
// With 0.4.x, exclude logback-classic from llm4s-core here (see above): Ktor manages its own logging backend
}
```
---
## Java sample
[`modules/samples/gradle-java`](https://github.com/llm4s/llm4s/tree/main/modules/samples/gradle-java) is a
standalone Gradle project that calls llm4s from Java through `llm4s-java-api`: a client, a question, a
conversation, and `LlmResult` error handling, with no Scala types in the code. Its README says how to build and
run it. CI builds it on every pull request, so it stays current.
## Reference files
The `modules/gradle-demo/` directory in the llm4s repository contains:
- `build.gradle.kts` - Kotlin DSL reference: `llm4s-core` plus a provider module
- `build.gradle` - Groovy DSL equivalent
- `settings.gradle.kts` - Minimal settings file
These are standalone reference files, not part of the sbt build. Copy them directly into your project as a starting point. The same directory also holds `ConversationTemplates` and `GradleSnippets`, small Scala helpers with their own tests.
---
## Troubleshooting
### `Could not resolve org.llm4s:...`
Gradle could not find the artifact. Use the prefixed name and the explicit Scala suffix: `org.llm4s:llm4s-core_3`. There is no Scala 2.13 artifact.
### `Multiple SLF4J bindings found`
Two SLF4J implementations are on the classpath. With `llm4s-core` `0.4.x`, exclude `logback-classic` (see [Logback conflict](#logback-conflict-spring-boot-3x-04x-only)) and ensure only one binding is declared.
### `Binary incompatible Scala library versions`
Add the `resolutionStrategy` block shown above to pin `org.scala-lang:scala3-library_3` to `3.7.1`. Pin that artifact only: Scala 3 runs on the Scala 2.13 `scala-library` (`2.13.x`), which has no `3.x` release, so a rule that matches the whole `org.scala-lang` group fails with `Could not find org.scala-lang:scala-library:3.7.1`.
---
See the [dependency-conflicts reference](/reference/dependency-conflicts) for a full conflict matrix.
---
Source: https://llm4s.org/getting-started/testing-guide
# Testing LLM4S Applications
Learn how to test LLM-powered applications effectively without spending money on API calls.
---
## Testing Strategy
Testing LLM applications requires a different approach than traditional software. You need to balance:
1. **Speed** - Tests should run fast
2. **Cost** - Avoid expensive API calls in CI/CD
3. **Determinism** - LLM responses vary, so test behaviors not exact outputs
4. **Coverage** - Test error paths, timeouts, and edge cases
---
## Unit Testing with a Scripted Client
For pure logic tests, give your code a scripted client instead of a real one. `llm4s-testkit` provides
`ScriptedLLMClient`: it answers from a script you write, records every request your code sent, and never touches the
network. Add it as a test dependency:
```scala
libraryDependencies += "org.llm4s" %% "llm4s-testkit" % llm4sVersion % Test
```
[Testing with the Testkit](../guide/testing-with-the-testkit) covers the whole API; the examples below show the common
cases.
### Canned replies
```scala
import org.llm4s.error.NetworkError
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model._
import org.llm4s.testkit.{ Reply, ScriptedLLMClient }
import org.llm4s.types.Result
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers
class WeatherAgentSpec extends AnyFlatSpec with Matchers {
// Example code under test: it takes any LLMClient
class WeatherAgent(client: LLMClient) {
def run(query: String): Result[Completion] =
client.complete(Conversation(Seq(UserMessage(query))))
}
"WeatherAgent" should "return the model's answer" in {
val client = ScriptedLLMClient.always(
Reply
.text("The weather in London is 15°C and cloudy.")
.withUsage(TokenUsage(promptTokens = 10, completionTokens = 15, totalTokens = 25))
)
val result = new WeatherAgent(client).run("What's the weather in London?")
result.map(_.content) shouldBe Right("The weather in London is 15°C and cloudy.")
client.callCount shouldBe 1
client.lastCall.flatMap(_.lastUserText) shouldBe Some("What's the weather in London?")
}
it should "pass a provider error on" in {
val client = ScriptedLLMClient.always(
Reply.failure(NetworkError("Connection timeout", None, "https://api.example.com"))
)
new WeatherAgent(client).run("What's the weather?") match {
case Left(_: NetworkError) => succeed
case other => fail(s"Expected NetworkError but got: $other")
}
}
}
```
`streamComplete` replays the same reply as chunks (the text in pieces, then any tool calls, then a chunk with the
finish reason), so code that streams is tested with the same script.
### A reply chosen by the request
`ScriptedLLMClient.respondingTo` picks the reply from the conversation, so the answer does not depend on how many calls
came before. A request no rule matches fails with an `UnscriptedCallError` that shows the last message, instead of
hanging or throwing:
```scala
import org.llm4s.llmconnect.model._
import org.llm4s.testkit.{ Reply, ScriptedLLMClient }
def lastUser(conversation: Conversation): String =
conversation.messages.collect { case u: UserMessage => u.content }.lastOption.getOrElse("")
val client = ScriptedLLMClient.respondingTo {
case c if lastUser(c) == "What is 2+2?" => Reply.text("2+2 equals 4")
case c if lastUser(c) == "What is the capital of France?" => Reply.text("The capital of France is Paris")
}
val answer = client.complete(Conversation(Seq(UserMessage("What is 2+2?")))) // Right, "2+2 equals 4"
val unexpected = client.complete(Conversation(Seq(UserMessage("Something else")))) // Left(UnscriptedCallError)
```
---
## Integration Testing with Ollama
For integration tests, use Ollama to avoid API costs:
### Setup
```bash
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# Pull a small, fast model
ollama pull llama3.2
# Start server
ollama serve
```
### Test Configuration
Put a named provider section in your test resources and select it as the default, with
`llm4s-ollama` as a test dependency. See [Named provider sections](configuration#named-provider-sections).
```hocon
# src/test/resources/application.conf
llm4s {
providers {
provider = "ollama-test"
ollama-test {
provider = "ollama"
model = "llama3.2"
baseUrl = "http://localhost:11434" # optional: this is the default
}
}
}
```
### Integration Test Example
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model._
import org.llm4s.model.ModelRegistryService
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers
class LLMIntegrationSpec extends AnyFlatSpec with Matchers {
// Only run if Ollama is available
def ollamaAvailable: Boolean = {
try {
val url = new java.net.URL("http://localhost:11434")
val connection = url.openConnection()
connection.setConnectTimeout(1000)
connection.connect()
true
} catch {
case _: Exception => false
}
}
"LLMClient" should "complete basic requests" in {
assume(ollamaAvailable, "Ollama server not available")
val result = for {
config <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(config)
response <- client.complete(
Conversation(Seq(UserMessage("Say 'hello' and nothing else")))
)
} yield response
result match {
case Right(response) =>
response.content.toLowerCase should include("hello")
case Left(error) =>
fail(s"Request failed: $error")
}
}
it should "handle streaming responses" in {
assume(ollamaAvailable, "Ollama server not available")
var chunks = List.empty[StreamedChunk]
val result = for {
config <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(config)
completion <- client.streamComplete(
Conversation(Seq(UserMessage("Count: 1, 2, 3"))),
onChunk = chunk => chunks = chunks :+ chunk
)
} yield completion
result match {
case Right(completion) =>
chunks should not be empty
chunks.last.finishReason should be(Some("stop"))
case Left(error) =>
fail(s"Streaming failed: $error")
}
}
}
```
---
## Testing Error Handling
Always test error paths. `Reply.failure` makes the client return the real `LLMError` types, so your code sees exactly
what a provider would give it:
```scala
import org.llm4s.agent.Agent
import org.llm4s.agent.graph.GraphError
import org.llm4s.error.{ AuthenticationError, NetworkError, RateLimitError }
import org.llm4s.testkit.{ Reply, ScriptedLLMClient }
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers
class ErrorHandlingSpec extends AnyFlatSpec with Matchers {
private def runWith(reply: Reply) =
Agent.builder("test-agent", ScriptedLLMClient.always(reply)).build().flatMap(_.run("test query"))
"Agent" should "handle rate limiting" in {
// A provider error ends the run as GraphError.NodeFailed, carrying the client's error as its cause
runWith(Reply.failure(RateLimitError("Rate limit exceeded"))) match {
case Left(GraphError.NodeFailed(_, _, _: RateLimitError)) => succeed
case other => fail(s"Expected RateLimitError but got: $other")
}
}
it should "handle authentication errors" in {
runWith(Reply.failure(AuthenticationError("openai", "Invalid API key", "401"))) match {
case Left(GraphError.NodeFailed(_, _, _: AuthenticationError)) => succeed
case other => fail(s"Expected AuthenticationError but got: $other")
}
}
it should "handle network timeouts" in {
// No need to sleep: the error a timed-out call returns is what the code under test sees
runWith(Reply.failure(NetworkError("Request timeout", None, "https://api.example.com"))) match {
case Left(GraphError.NodeFailed(_, _, _: NetworkError)) => succeed
case other => fail(s"Expected NetworkError but got: $other")
}
}
}
```
---
## Testing Tool Calling
Script the model's tool call and then its answer with `ScriptedLLMClient.sequence`: the first call gets the first
reply, the second call the second. The agent runs your real tool in between, and the recorded calls show what it sent
back to the model:
```scala
import org.llm4s.agent.Agent
import org.llm4s.llmconnect.model._
import org.llm4s.testkit.{ Reply, ScriptedLLMClient }
import org.llm4s.toolapi.{ Schema, ToolBuilder, ToolRegistry }
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers
class ToolCallingSpec extends AnyFlatSpec with Matchers {
"Agent" should "invoke the weather tool with the model's arguments" in {
var capturedCity: Option[String] = None
// A tool that records the arguments it was called with
val weatherTool = ToolBuilder[Map[String, Any], String](
"get_weather",
"Get weather for a city",
Schema
.`object`[Map[String, Any]]("Weather query parameters")
.withProperty(Schema.property("city", Schema.string("City name")))
).withHandler { extractor =>
extractor.getString("city").map { city =>
capturedCity = Some(city)
s"Weather in $city: 20°C"
}
}.buildSafe()
val client = ScriptedLLMClient.sequence(
Reply.toolCall("get_weather", """{"city": "London"}"""),
Reply.text("It is 20°C in London.")
)
val outcome = for {
tool <- weatherTool
agent <- Agent.builder("test-agent", client).withTools(new ToolRegistry(Seq(tool))).build()
_ <- agent.run("What's the weather in London?")
} yield ()
outcome.isRight shouldBe true
capturedCity shouldBe Some("London")
client.callCount shouldBe 2
// the second request carries the tool's result back to the model
client.calls(1).messages.collect { case t: ToolMessage => t.content } shouldBe Seq("Weather in London: 20°C")
}
}
```
---
## Testing RAG Applications
Test document retrieval and answer generation separately:
```scala
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers
class RAGSpec extends AnyFlatSpec with Matchers {
"VectorStore" should "retrieve relevant documents" in {
val documents = List(
"Scala is a functional programming language",
"Python is a dynamically typed language",
"Java runs on the JVM"
)
// Note: This is conceptual pseudocode showing testing patterns. `InMemoryVectorStore`, `embedder` and
// `RAGPipeline` below are illustrative stand-ins, not llm4s classes with these signatures. The llm4s-rag
// module has real stores (SQLiteVectorStore, PgVectorStore, QdrantVectorStore).
val vectorStore = new InMemoryVectorStore() // Pseudocode - use your vector store
documents.foreach(doc => vectorStore.add(doc, embedder.embed(doc))) // embedder is conceptual
val results = vectorStore.search("functional programming", topK = 1)
results.head should include("Scala")
}
"RAG pipeline" should "include context in LLM prompt" in {
// a scripted client from llm4s-testkit: it records the prompt the pipeline sent
val client = ScriptedLLMClient.always(Reply.text("Based on the context, Scala is functional."))
val rag = new RAGPipeline(client, vectorStore, embedder)
rag.query("What is Scala?")
client.lastCall.flatMap(_.lastUserText).get should include("context")
client.lastCall.flatMap(_.lastUserText).get should include("functional")
}
}
```
---
## CI/CD Testing Strategy
### Fast CI Pipeline
```yaml
# .github/workflows/test.yml
name: Test
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Scala
uses: olafurpg/setup-scala@v14
with:
java-version: 21
- name: Install Ollama
run: |
curl -fsSL https://ollama.com/install.sh | sh
ollama serve &
sleep 5
ollama pull llama3.2
- name: Run unit tests (fast)
run: sbt "testOnly *UnitSpec"
- name: Run integration tests (with Ollama)
# The provider comes from src/test/resources/application.conf (the ollama-test section
# above); no environment variable selects it.
run: sbt "testOnly *IntegrationSpec"
# Skip expensive tests in CI
- name: Run full test suite
run: sbt test
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
```
### Test Categorization
```scala
// Tag tests by speed/cost
import org.scalatest.Tag
import org.scalatest.flatspec.AnyFlatSpec
object UnitTest extends Tag("UnitTest")
object IntegrationTest extends Tag("IntegrationTest")
object ExpensiveTest extends Tag("ExpensiveTest")
class FastSpec extends AnyFlatSpec {
"Fast unit test" should "run in CI" taggedAs UnitTest in {
// Mock-based test
}
}
class SlowSpec extends AnyFlatSpec {
"Expensive test" should "run manually" taggedAs ExpensiveTest in {
// Uses real OpenAI API
}
}
```
Run specific test categories:
```bash
# Fast tests only
sbt "testOnly * -- -n UnitTest"
# Everything except expensive tests
sbt "testOnly * -- -l ExpensiveTest"
```
---
## Best Practices
1. ✅ **Mock by default**: Use mock clients for unit tests
2. ✅ **Ollama for integration**: Free and fast enough for CI
3. ✅ **Test behaviors, not outputs**: LLM responses vary, so test that tools are called, documents are retrieved, etc.
4. ✅ **Use deterministic models when possible**: Set temperature=0 for more predictable outputs
5. ✅ **Separate concerns**: Test tool logic independently from LLM integration
6. ✅ **Tag expensive tests**: Don't run them in every CI build
7. ✅ **Use smaller models in CI**: llama3.2 is fast and free via Ollama
---
## Example Test Suite Structure
```
src/test/scala/
├── unit/
│ ├── ToolSpec.scala # Pure logic tests (mocked)
│ ├── ConfigSpec.scala # Configuration parsing
│ └── ErrorHandlingSpec.scala # Error path tests
├── integration/
│ ├── LLMClientSpec.scala # Real LLM calls (Ollama)
│ ├── AgentSpec.scala # End-to-end agent tests
│ └── RAGSpec.scala # RAG pipeline tests
└── resources/
└── application.conf # Test config (Ollama)
```
---
## Next Steps
- [Configuration Guide](configuration) - Configure test environments
- [Examples](/examples/) - See tests in action
- [CI/CD Best Practices](/advanced/ci-cd) - Production testing strategies
---
## Additional Resources
- [ScalaTest Documentation](https://www.scalatest.org/)
- [Ollama Models](https://ollama.com/library) - Available test models
- [Testing Guide](/reference/testing-guide) - Advanced testing patterns
---
Source: https://llm4s.org/getting-started/scala-cli
# Scala CLI Quick Start
A first LLM call from a single file, with no sbt project.
---
[Scala CLI](https://scala-cli.virtuslab.org/) runs a Scala file with its dependencies declared in the
file itself, so you can try LLM4S without creating a build. This page is the shortest path from nothing
to an answer; when you want a real project, use the [starter kit](installation#quick-start-with-the-starter-kit)
or [add LLM4S to your build](installation#add-llm4s-to-your-project).
## Before you start
- **Scala CLI**, installed as described on its [install page](https://scala-cli.virtuslab.org/install).
- **A JDK** is optional: the script's `//> using jvm 21` line makes Scala CLI use JDK 21, and download it the first
time if you do not have it. LLM4S needs JDK 21 (see [Installation](installation#prerequisites)).
- **An API key** for OpenAI, or a local [Ollama](ollama-quickstart) (no key).
## The files
Make a directory with these files. The same files are in
[`modules/samples/scala-cli`](https://github.com/llm4s/llm4s/tree/main/modules/samples/scala-cli).
```
hello-llm4s/
hello.scala
logback.xml
resources/application.conf
resources/ollama.conf (only for the no-key Ollama run)
```
`hello.scala`:
(not compile-checked)
```scala
//> using scala 3.7.1
//> using jvm 21
//> using dep org.llm4s::llm4s-core:0.5.1
//> using dep org.llm4s::llm4s-openai:0.5.1
//> using dep org.llm4s::llm4s-ollama:0.5.1
//> using dep ch.qos.logback:logback-classic:1.5.34
//> using resourceDir resources
//> using javaOpt -Dlogback.configurationFile=logback.xml
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model.{ Conversation, UserMessage }
import org.llm4s.model.ModelRegistryService
@main def hello(): Unit = {
val answer = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
completion <- client.complete(Conversation(Seq(UserMessage("In one sentence, what is Scala?"))))
} yield completion.content
answer.fold(error => println(s"Error: ${error.formatted}"), text => println(text))
}
```
`resources/application.conf` names the provider the script uses: the section that `provider` names. The
[Configuration guide](configuration#named-provider-sections) covers the other providers:
```hocon
llm4s {
providers {
provider = "openai-main"
openai-main {
provider = "openai"
model = "gpt-4o-mini"
apiKey = ${?OPENAI_API_KEY}
}
}
}
```
`resources/ollama.conf` is the no-key alternative, a local [Ollama](ollama-quickstart) (it needs `ollama serve`
and `ollama pull llama3`). It is a separate file so that each run reads only its own provider:
```hocon
llm4s {
providers {
provider = "ollama-local"
ollama-local {
provider = "ollama"
model = "llama3"
baseUrl = "http://localhost:11434"
baseUrl = ${?OLLAMA_BASE_URL}
}
}
}
```
`logback.xml` keeps the library's own logging out of your first answer (without it, logback prints
every debug message):
```xml
System.err
%d{HH:mm:ss} %-5level %logger{20} - %msg%n
```
## Run it
Run from inside the directory: the `logback.xml` path in the script is relative to where you run it.
```bash
cd hello-llm4s
export OPENAI_API_KEY=sk-...
scala-cli run .
```
With a local Ollama instead, no key needed (`-Dconfig.resource` tells the configuration library to read
`ollama.conf` in place of `application.conf`):
```bash
scala-cli run . --java-opt -Dconfig.resource=ollama.conf
```
The first run downloads the dependencies, which takes a while. You should see one sentence, for example:
```
Scala is a language that blends object-oriented and functional programming on the JVM.
```
The wording depends on the model.
## What the script does
1. `Llm4sConfig.defaultProvider()` reads the section that `llm4s.providers.provider` names. This is the one place
configuration is read.
2. `LLMConnect.getClient` builds the client for that provider.
3. `client.complete` sends the conversation. Every step returns a `Result`, so the `for` stops at the first
error and the last line prints it instead of throwing.
## If something goes wrong
**`Error: ConfigurationError: Provider 'openai-main' (provider = openai) is missing required fields:`** followed by
`- apiKey: set OPENAI_API_KEY, or set apiKey under llm4s.providers.openai-main in application.conf`: the key is not set
in the shell that runs `scala-cli`. Check `echo $OPENAI_API_KEY`, or use the Ollama command above.
**`Error: NetworkError: POST http://localhost:11434/api/chat: connection failed`** (the Ollama run): no Ollama server
is listening. Start it with `ollama serve`, or point `OLLAMA_BASE_URL` at the one you have.
**A wall of `DEBUG` lines before the answer**: `logback.xml` was not found. Run from the directory that
contains it, and check the name and the `javaOpt` line.
**The version**: the `//> using dep` lines pin the release. Change the number on every `org.llm4s` line together
to move to another one: the modules are released at the same version.
## The dependencies
`llm4s-core` holds no provider client, so the script adds the provider modules it uses, `llm4s-openai` and
`llm4s-ollama`; keep only the one you use. `llm4s-core` brings no logging backend either, so the script adds
logback (see the [migration guide](/reference/migration#llm4s-no-longer-brings-a-logging-backend)). The logback
line is a Java dependency (a single colon, unlike the `::` Scala form); without a backend SLF4J prints a single
"no SLF4J providers were found" warning and `logback.xml` has no effect.
The `apiKey = ${?OPENAI_API_KEY}` line in the section is optional: `llm4s-openai` binds `OPENAI_API_KEY` itself,
and a section's own `apiKey` takes precedence over it.
Up to 0.4.x, `llm4s-core` included the provider clients and a logging backend, so a script for 0.4.x needs only the
`llm4s-core` line, and the `apiKey` line in the section.
## Next steps
- [First Example](first-example): the same call in a project, with conversation context, tools and streaming
- [Configuration](configuration): all providers and how keys are found
- [Ollama Quick Start](ollama-quickstart): local models
---
Source: https://llm4s.org/getting-started/maven
# Maven Quick Start
A complete `pom.xml` that compiles and runs a first call from Java, with `llm4s-java-api`.
---
The project is in
[`modules/samples/maven-java`](https://github.com/llm4s/llm4s/tree/main/modules/samples/maven-java), and CI compiles
it against the library on every pull request. For Gradle, see [Gradle Integration](gradle); for the Java API itself,
the [Java guide](/guide/java).
## Before you start
- **JDK 21 or newer** and **Maven 3.9 or newer**. `llm4s-java-api` is JDK 21 bytecode, so the project sets
`maven.compiler.release` to 21.
- **An API key** for OpenAI, or a local [Ollama](ollama-quickstart) (no key).
## The coordinate
LLM4S is published for **Scala 3 only**, and Maven does **not** add the Scala cross-version suffix: the artifact is
`org.llm4s:llm4s-java-api_3`, and you write the `_3` yourself. Its POM brings `llm4s-core`, `llm4s-agent` and the
provider modules, so it is the one llm4s dependency a Java project needs.
## The pom.xml
```xml
4.0.0
org.llm4s.samples
llm4s-maven-sample
1
21
UTF-8
0.5.1
3.7.1
org.scala-lang
scala3-library_3
${scala3.version}
com.fasterxml.jackson
jackson-bom
2.19.4
pom
import
org.jetbrains.kotlin
kotlin-bom
1.9.10
pom
import
com.typesafe
config
1.4.9
org.slf4j
slf4j-api
2.0.17
com.google.errorprone
error_prone_annotations
2.50.0
io.swagger.core.v3
swagger-annotations
2.2.50
org.llm4s
llm4s-java-api_3
${llm4s.version}
ch.qos.logback
logback-classic
1.5.34
runtime
org.apache.maven.plugins
maven-enforcer-plugin
3.5.0
enforce-convergence
enforce
org.codehaus.mojo
exec-maven-plugin
3.5.0
org.llm4s.samples.HelloLLM4S
```
What each part is for:
- **`llm4s-java-api_3`** is the library. Change `llm4s.version` to move to another release; every llm4s artifact is
released at the same version.
- **`logback-classic`** is the logging backend. llm4s logs through SLF4J and ships none; without one SLF4J prints a
single "no SLF4J providers were found" warning and drops the logs.
- **`scala3-library_3`** in `dependencyManagement` pins the Scala 3 library. Several llm4s modules may ask for
different micro-versions, and Maven's nearest-wins rule would pick one silently. Pin `scala3-library_3` by name,
never the whole `org.scala-lang` group: Scala 3 runs on the Scala 2.13 `scala-library`, which has no 3.x release.
The [dependency conflicts reference](/reference/dependency-conflicts) has the matrix.
- **`maven-enforcer-plugin` with `dependencyConvergence`** fails the build when two dependencies need different
versions of one library, so a conflict shows up at build time and not as a `NoSuchMethodError` at run time.
When it fails, it names the library and both paths: add a `dependencyManagement` entry for it, as for
`scala3-library_3`, or an exclusion.
- **The other `dependencyManagement` entries** settle the conflicts the enforcer finds today: the OpenAI and
Anthropic SDKs ask for different Jackson, Kotlin, error-prone and Swagger versions, and pureconfig for an older
Typesafe Config and SLF4J than llm4s. Each pin is the newest version any of them asks for; the Jackson and Kotlin
BOMs pin every artifact of those families at once.
- **`exec-maven-plugin`** gives `mvn exec:java`, which runs the `main` class.
## The code and the configuration
`src/main/java/org/llm4s/samples/HelloLLM4S.java` creates a client from the default provider, asks one question
and prints the answer or the error. Every call returns an `LlmResult`, so there is no Scala `Either` in the code:
```java
LlmResult created = Llm4s.createDefaultClient();
if (created.isFailure()) {
System.err.println("Could not create a client: " + created.getError().getMessage());
return 1;
}
try (JLlmClient client = created.get()) {
LlmResult answer = client.complete("In one sentence, what is Scala?");
// ... print answer.get(), or answer.getError().getMessage()
}
```
`src/main/resources/application.conf` names the provider, as a named section; the
[Configuration guide](configuration#named-provider-sections) covers the others:
```hocon
llm4s {
providers {
provider = "openai-main"
openai-main {
provider = "openai"
model = "gpt-4o-mini"
}
}
}
```
## Run it
```bash
export OPENAI_API_KEY=sk-...
mvn -q compile exec:java
```
You should see one sentence; the wording depends on the model. With a local Ollama, change the section to
`provider = "ollama"` with a `model` you have pulled, as in the [Ollama quick start](ollama-quickstart).
## If something goes wrong
**`Could not find artifact org.llm4s:llm4s-java-api:jar`** (no `_3`): add the suffix, `llm4s-java-api_3`.
**`Could not create a client: ... apiKey: set OPENAI_API_KEY`**: the key is not set in the shell that runs `mvn`.
**`Rule 0: ...DependencyConvergence failed`**: see the enforcer entry above.
**`class file has wrong version 65.0`** or similar: Maven is running on a JDK older than 21. Check `mvn -v`.
## Next steps
- [Java guide](/guide/java): conversations, options, agents and errors from Java
- [Gradle Integration](gradle): the same project in Gradle
- [Quarkus](/guide/quarkus): a Quarkus application built the same way, with a CDI producer and a REST endpoint
- [Configuration](configuration): all providers and how keys are found
---
Source: https://llm4s.org/getting-started/jbang
# JBang Quick Start
A first LLM call from one Java file, with no build tool and no project.
---
[JBang](https://www.jbang.dev/) runs a Java file with its dependencies declared in the file itself, so you can try
LLM4S from Java without creating a project. `llm4s-java-api` is on Maven Central from `0.5.0`. When you want a real
project, use [Maven](maven) or [Gradle](gradle).
## Before you start
- **JBang**, installed as described on its [download page](https://www.jbang.dev/download/).
- **JDK 21 or newer**: the file's `//JAVA 21` line makes JBang use JDK 21, and download it the first time if you
do not have it. LLM4S needs JDK 21 (see [Installation](installation#prerequisites)).
- **A local [Ollama](ollama-quickstart)**, which needs no key: run `ollama serve` and `ollama pull llama3`.
## The files
Make a directory with these three files. The same files are in
[`modules/samples/jbang`](https://github.com/llm4s/llm4s/tree/main/modules/samples/jbang).
`HelloLlm4s.java`:
```java
///usr/bin/env jbang "$0" "$@" ; exit $?
//JAVA 21
//DEPS org.llm4s:llm4s-java-api_3:0.5.1
//DEPS ch.qos.logback:logback-classic:1.5.34
//FILES application.conf
//FILES logback.xml
import org.llm4s.javaapi.JLlmClient;
import org.llm4s.javaapi.Llm4s;
import org.llm4s.javaapi.LlmResult;
public class HelloLlm4s {
public static void main(String[] args) {
LlmResult created = Llm4s.createDefaultClient();
if (created.isFailure()) {
System.err.println("Could not create a client: " + created.getError().getMessage());
System.exit(1);
}
try (JLlmClient client = created.get()) {
LlmResult answer = client.complete("In one sentence, what is Scala?");
if (answer.isFailure()) {
System.err.println("The call failed: " + answer.getError().getMessage());
System.exit(1);
}
System.out.println(answer.get());
} catch (Exception e) {
System.err.println("Closing the client failed: " + e.getMessage());
System.exit(1);
}
}
}
```
The `//DEPS` line is the whole dependency: Maven does not add the Scala suffix, so the artifact is
`llm4s-java-api_3`, and its POM brings `llm4s-core`, `llm4s-agent` and the provider modules, including Ollama.
The second `//DEPS` line is the logging backend: llm4s logs through SLF4J and ships none, so without it SLF4J prints
a "no SLF4J providers were found" warning and drops the logs. `//FILES` puts a file on the classpath.
`application.conf` names the provider, as a named section; the
[Configuration guide](configuration#named-provider-sections) covers the others:
```hocon
llm4s {
providers {
provider = "ollama-local"
ollama-local {
provider = "ollama"
model = "llama3"
baseUrl = "http://localhost:11434"
baseUrl = ${?OLLAMA_BASE_URL}
}
}
}
```
`logback.xml` keeps the library's own logging out of your first answer (without it, logback prints every debug
message):
```xml
System.err
%d{HH:mm:ss} %-5level %logger{20} - %msg%n
```
## Run it
From the directory that holds the three files:
```bash
jbang HelloLlm4s.java
```
The first run downloads the dependencies, which takes a while. You should see one sentence; the wording depends on
the model. To compile without running, use `jbang build HelloLlm4s.java`.
For OpenAI instead of Ollama, change the section in `application.conf` to `provider = "openai"` with
`model = "gpt-4o-mini"`, and set `OPENAI_API_KEY` in the shell.
## If something goes wrong
**`Could not create a client: ...`**: the message names what is missing, for example a provider key. Check
`application.conf` and the environment.
**`The call failed: ... connection failed`**: no Ollama server is listening. Start it with `ollama serve`, or point
`OLLAMA_BASE_URL` at the one you have.
**A wall of `DEBUG` lines before the answer**: `logback.xml` is not beside the file, or `//FILES logback.xml` is
missing.
**The version**: change the number in the `//DEPS org.llm4s` line to move to another release.
## Trying the code on `main`
`llm4s-java-api` ships with `0.5.0`. Before a change is released, publish it to your local Maven repository from a
checkout of this repository, as the [Gradle sample](https://github.com/llm4s/llm4s/tree/main/modules/samples/gradle-java)
does:
```bash
sbt 'set ThisBuild / version := "0.1.0-SNAPSHOT"' publishM2
```
Then change the `//DEPS` line to version `0.1.0-SNAPSHOT` and add `//REPOS mavenlocal,mavencentral` above it.
## Next steps
- [Maven Quick Start](maven): the same call in a `pom.xml`
- [Java guide](/guide/java): conversations, options, agents and errors from Java
- [Configuration](configuration): all providers and how keys are found
---
Source: https://llm4s.org/getting-started/troubleshooting
# First-Run Troubleshooting
The problems people most often meet when an LLM4S program does not work the first time, each with the symptom, the
cause and the fix. The message texts below are the ones the code produces (the class that builds each is named), so
you can search this page for the text you see.
Each of these messages ends with a link to its entry here, such as `See https://llm4s.org/getting-started/troubleshooting#missing-api-key`, and a
configuration error also with the doctor's command. The anchors (`#missing-api-key`) are fixed: they do not change
when an entry is renumbered or its heading reworded.
Two tools help before you read on:
- **The doctor** checks your JDK, configuration, default provider, provider module, API key and (with `--local`) a
local model server in one run, and says what to change. See [Doctor](/reference/doctor).
- **The rest of the reference** has [Troubleshooting / FAQ](/reference/troubleshooting) for later problems (agents,
streaming, RAG, tracing) and [Configuration: Troubleshooting](configuration#troubleshooting).
## Contents
1. [The API key is missing](#missing-api-key)
2. [No default provider configured](#no-default-provider)
3. [Configured provider was not found](#provider-not-found)
4. [Provider is not registered](#provider-not-registered)
5. [application.conf is not picked up](#application-conf-not-loaded)
6. [LLM_MODEL has no effect](#llm-model-ignored)
7. [The code does not compile: no ModelRegistryService](#no-model-registry-service)
8. [Ollama is not running](#ollama-not-running)
9. [Ollama does not have the model](#ollama-model-missing)
10. [UnsupportedClassVersionError or a JDK that is too old](#jdk-too-old)
11. [Authentication failed (401 or 403)](#authentication-failed)
12. [Rate limited (429)](#rate-limited)
13. [Timeouts](#timeouts)
14. [Model not found (404)](#model-not-found)
15. [Behind a proxy or TLS inspection](#corporate-network)
## 1. The API key is missing {#missing-api-key}
**Symptom** (a `ConfigurationError`, from `Llm4sConfig.defaultProvider()` or `Llm4sConfig.provider(...)`):
```text
Provider 'openai-main' (provider = openai) is missing required fields:
- apiKey: set OPENAI_API_KEY, or set apiKey under llm4s.providers.openai-main in application.conf
See https://llm4s.org/getting-started/troubleshooting#missing-api-key
Run the doctor to check your whole setup: sbt "configPolicy/runMain org.llm4s.configpolicy.DoctorCli --config /path/to/application.conf"
```
An embeddings block without a key fails the same way, with `Missing openai embeddings apiKey: set OPENAI_API_KEY, ...`.
**Cause.** Neither the vendor's variable nor an `apiKey` in the section is available to the JVM. The variable named
is the one the provider's module binds (`llm4s-openai` binds `OPENAI_API_KEY`); a section that sets its own `apiKey`
uses that instead. A variable exported in one terminal is not seen by an IDE run configuration started elsewhere.
**Fix.** Export the variable in the shell that starts the JVM (or set it in the IDE run configuration), or set
`apiKey` in the section. The error names the section it is about: if it is not the one you meant, check
`llm4s.providers.provider`. See [API keys](configuration#api-keys).
## 2. No default provider configured {#no-default-provider}
**Symptom** (a `ConfigurationError`):
```text
No default provider configured under llm4s.providers.provider
See https://llm4s.org/getting-started/troubleshooting#no-default-provider
Run the doctor to check your whole setup: sbt "configPolicy/runMain org.llm4s.configpolicy.DoctorCli --config /path/to/application.conf"
```
**Cause.** `Llm4sConfig.defaultProvider()` loads the section that `llm4s.providers.provider` names, and the key is
not set, usually because no `application.conf` defines it.
**Fix.** Add the key to your `application.conf` (see [Named provider sections](configuration#named-provider-sections)),
or load a section by name with `Llm4sConfig.provider("")`.
## 3. Configured provider was not found {#provider-not-found}
**Symptom** (a `ConfigurationError`):
```text
Configured provider 'openai-main' was not found
See https://llm4s.org/getting-started/troubleshooting#provider-not-found
Run the doctor to check your whole setup: sbt "configPolicy/runMain org.llm4s.configpolicy.DoctorCli --config /path/to/application.conf"
```
**Cause.** `llm4s.providers.provider` (or the name you passed to `Llm4sConfig.provider(...)`) names a section that
does not exist, for example through a typo, or a `-Dllm4s.providers.provider=...` override.
**Fix.** Make the name match a section under `llm4s.providers` exactly.
## 4. Provider is not registered {#provider-not-registered}
**Symptom** (a `ConfigurationError`, from `LLMConnect.getClient`):
```text
Provider 'openai' is not registered. Registered providers: none. If you expected 'openai', add the dependency that supplies it, or register it explicitly with ProviderRegistry.of(...). Discovery scanned 0 modules; 0 failed.
See https://llm4s.org/getting-started/troubleshooting#provider-not-registered
Run the doctor to check your whole setup: sbt "configPolicy/runMain org.llm4s.configpolicy.DoctorCli --config /path/to/application.conf"
```
When the id came from a config section, the message also says which key it came from. The first line ends with a
sentence about classpath discovery, such as `Discovery scanned 0 modules; 0 failed.`
**Cause.** Since 0.5.0 `llm4s-core` ships no provider: each one is its own module (`llm4s-openai`, `llm4s-anthropic`,
`llm4s-ollama`, ...). Either the module is not a dependency of your build, or you build a fat jar and the
`META-INF/services` entries of the modules were dropped or overwritten when the jars were merged (`Discovery scanned 0 modules` in the
message points to this).
**Fix.** Add the provider's module next to `llm4s-core` (the table in
[Named provider sections](configuration#named-provider-sections) lists them), or, for a fat jar, merge the
`META-INF/services` files in your assembly settings or register the provider explicitly with
`ProviderRegistry.of(...)` / `.withProvider(...)`.
## 5. application.conf is not picked up {#application-conf-not-loaded}
**Symptom.** `Llm4sConfig` fails as if your sections were not there (problem 2 or 3 above), although you wrote them,
or it cannot read the block at all (a `ConfigurationError`):
```text
Failed to load raw providers config via PureConfig:
See https://llm4s.org/getting-started/troubleshooting#application-conf-not-loaded
Run the doctor to check your whole setup: sbt "configPolicy/runMain org.llm4s.configpolicy.DoctorCli --config /path/to/application.conf"
```
A section that cannot be read on its own fails only a load of that section, with
`Failed to read llm4s.providers.: ...` and the same two lines.
**Cause.** The usual reasons are that the file is not on the runtime classpath (it must be in `src/main/resources/`,
not at the project root), that `-Dconfig.file` or `-Dconfig.resource` points at another file, that the HOCON has a
syntax error, or that a `-D` system property overrides the key. Precedence is system properties, then
`application.conf`, then each module's `reference.conf`.
**Fix.** Check the file's location and run the [doctor](/reference/doctor) with `--config `, which reports an
unreadable configuration and where it is broken. See [How configuration is loaded](configuration#how-configuration-is-loaded).
## 6. LLM_MODEL has no effect {#llm-model-ignored}
**Symptom.** You set `LLM_MODEL=openai/gpt-4o` (as older guides and blog posts say) and the program still uses
another model, or fails with problem 2.
**Cause.** Nothing in the library reads `LLM_MODEL` any more. Providers are named sections in `application.conf`.
**Fix.** Define a section and select it, as in [Named provider sections](configuration#named-provider-sections). The
chat-tui sample is the exception: it reads `LLM_MODEL` itself.
## 7. The code does not compile: no ModelRegistryService {#no-model-registry-service}
**Symptom.** A Scala compile error at a `LLMConnect.getClient(...)` call: the compiler reports that no given instance
of `org.llm4s.model.ModelRegistryService` was found.
**Cause.** `LLMConnect.getClient` takes a `ModelRegistryService` (and a `ProviderRegistry`) as `using` parameters.
The `ProviderRegistry` has a default; the model registry service you provide.
**Fix.** Load it with `Llm4sConfig.modelRegistryService()` and bind it as a `given` before calling `getClient`,
as in [First Example](first-example).
## 8. Ollama is not running {#ollama-not-running}
**Symptom** (a `NetworkError` from a chat call, or the doctor's "Local server" step failing):
```text
POST http://localhost:11434/api/chat: connection failed: ConnectException. Start the server with `ollama serve`, and check the section's baseUrl. See https://llm4s.org/getting-started/troubleshooting#ollama-not-running
```
The doctor's step says `GET http://localhost:11434/api/tags: connection failed: ConnectException` instead.
**Cause.** Nothing is listening at the section's `baseUrl` (default `http://localhost:11434`). The text after
`connection failed:` is the Java exception's message, or its class name when it has none. `unknown host: ...` instead
means the host name in `baseUrl` does not resolve.
**Fix.** Start the server with `ollama serve` (on macOS and Windows the app may need to be started), and check the
section's `baseUrl` (the samples bind it to `OLLAMA_BASE_URL`). `curl http://localhost:11434` answers
`Ollama is running` when it is up. See [Ollama Quick Start](ollama-quickstart).
## 9. Ollama does not have the model {#ollama-model-missing}
**Symptom** (a `ServiceError` with HTTP 404 from a chat call):
```text
Service error from ollama: model "llama3" not found, try pulling it first (HTTP 404). Pull it with `ollama pull llama3`, or set the section's model to one `ollama list` shows. See https://llm4s.org/getting-started/troubleshooting#ollama-model-missing
```
or, from the doctor run with `--local`:
```text
http://localhost:11434 answered, but it does not have 'llama3' (it has: ...)
fix: run `ollama pull llama3`, or set the model to one it has
```
**Cause.** The section's `model` is not among the models the server has downloaded. The text before `(HTTP 404)` is
the server's own.
**Fix.** Pull it (`ollama pull `), or change `model` in the section to a model `ollama list` shows. The
samples' default is `llama3:latest`, which needs `ollama pull llama3` once, or `OLLAMA_MODEL` set to another.
## 10. UnsupportedClassVersionError or a JDK that is too old {#jdk-too-old}
**Symptom.** The JVM refuses to load LLM4S classes with `java.lang.UnsupportedClassVersionError` (that text is the
JVM's, not LLM4S's), or the doctor reports:
```text
Java is too old: LLM4S needs 21 or newer
```
**Cause.** Every LLM4S artifact is compiled for JDK 21 and needs at least that. `java -version`, `JAVA_HOME`, and the
JDK your IDE and sbt use can each name a different one.
**Fix.** Run sbt, your application and your IDE with JDK 21 or newer (set `JAVA_HOME`, or choose the JDK in the IDE).
See the [v1 scope](/reference/v1-scope) for the support statement.
## 11. Authentication failed (401 or 403) {#authentication-failed}
**Symptom** (an `AuthenticationError`):
```text
Authentication failed for openai: . Check that the key is this provider's, and that its project or organisation has access to the model. See https://llm4s.org/getting-started/troubleshooting#authentication-failed
```
A provider that answers 403 through an SDK that does not type it (Anthropic) gives a `ServiceError`,
`Service error from anthropic: ... (HTTP 403)`, with the same sentence and link.
**Cause.** The provider answered with HTTP 401 or 403: the key is wrong, expired, revoked or has no access to the
model. The most common slip is a key from another vendor: a section whose own `apiKey` binds a variable holding a
different provider's key wins over the shared `OPENAI_API_KEY`.
**Fix.** Check which key the section uses: the INFO line `llm4s.providers.: API key from ` says where it
came from (never the value). See [Provider mismatch](configuration#problem-provider-mismatch).
## 12. Rate limited (429) {#rate-limited}
**Symptom** (a `RateLimitError`):
```text
Rate limited by openai. Retry after 20s. Lower the request concurrency, or wrap the client in ReliableClient to retry and throttle. See https://llm4s.org/getting-started/troubleshooting#rate-limited
```
`Retry after 20s` appears only when the provider sent a `Retry-After` header.
**Cause.** The provider answered HTTP 429: too many requests or tokens for your plan.
**Fix.** Wait for the delay and retry, slow the caller down, or raise the plan's limit. A `RateLimitError` is a
recoverable error, and `ReliableClient` can retry and throttle for you (see the [fallback cookbook](/examples/cookbook/fallback) and
[Rate limiting / 429 errors](configuration#problem-rate-limiting--429-errors)).
## 13. Timeouts {#timeouts}
**Symptom** (a `TimeoutError`; these are the messages of the JDK-based HTTP client):
```text
POST https://example.com/v1/chat/completions: request timed out after 2 minutes
POST https://example.com/v1/chat/completions: connection timed out after 10 seconds
```
**Cause.** The response did not begin in time (`request timed out`), or the TCP connection could not be made
(`connection timed out`), often a wrong `baseUrl`, a proxy or a firewall. A slow local model on a CPU can also exceed
the default.
**Fix.** Check the endpoint is reachable - behind an outbound proxy, configure it ([problem 15](#corporate-network)) -
and if the model is just slow raise the timeout in the section's
`timeouts` block (`timeouts { request = 5m, stream = 20m }`). The defaults differ by provider, and the OpenAI,
Azure, Requesty and Anthropic clients use their SDK's own timeouts, which retry; see [Timeouts](configuration#timeouts).
## 14. Model not found (404) {#model-not-found}
**Symptom** (a `ServiceError` with HTTP 404):
```text
Service error from openai: The model `gpt-5-mini-typo` does not exist or you do not have access to it. (HTTP 404). Check the section's model name and baseUrl; Llm4sConfig.listModels() lists the models the provider offers. See https://llm4s.org/getting-started/troubleshooting#model-not-found
```
**Cause.** The provider has no model of that name, or the section's `baseUrl` points at the wrong path, so the
endpoint itself is not found. A local Ollama that has not pulled the model answers this way too; see
[problem 9](#ollama-model-missing).
**Fix.** Check the section's `model` against the provider's model list: `Llm4sConfig.listModels()` lists the models
of the default section's provider, and `Llm4sConfig.listModels("")` those of a named section. If the model is
right, check `baseUrl` (an OpenAI-compatible server's usually ends in `/v1`).
## 15. Behind a proxy or TLS inspection {#corporate-network}
**Symptom** (a `TimeoutError` or a `NetworkError`):
```text
POST https://api.openai.com/v1/chat/completions: connection timed out after 10 seconds
POST https://api.openai.com/v1/chat/completions: I/O error: PKIX path building failed: unable to find valid certification path to requested target
```
**Cause.** The network reaches the internet only through an outbound HTTP proxy, which llm4s is not using, or a proxy
inspects TLS and re-signs traffic with a corporate CA the JVM does not trust. A proxy that needs a login answers
`407 Proxy Authentication Required`.
**Fix.** Set `HTTPS_PROXY` (and `NO_PROXY` for hosts reached directly), or the `llm4s.http.proxy` block, with
`LLM4S_PROXY_USER` / `LLM4S_PROXY_PASSWORD` for an authenticating proxy; point `llm4s.http.tls.trustStore` (or
`LLM4S_TRUSTSTORE`) at a PKCS12 or JKS store holding the corporate CA. Every provider client honours them; see
[Corporate networks](configuration#corporate-networks-a-proxy-and-a-custom-ca), which also lists what is not covered
(PAC files, Kerberos/NTLM).
---
Still stuck? Run the [doctor](/reference/doctor), then ask on [Discord](https://discord.gg/4uvTPn6qww) with the full
message.
---
Source: https://llm4s.org/getting-started/scala-2-13
# Using LLM4S from Scala 2.13
Scala 2.13: call-only, via the TASTy reader. A Spark job, a Play application or an Akka or Pekko service can call LLM4S's Scala 3 artifacts today.
1. TOC
---
LLM4S is published for Scala 3 only: there is no `_2.13` artifact, and a cross-build is not planned before 1.0
([#874](https://github.com/llm4s/llm4s/issues/874)). But the Scala 2.13 compiler can read Scala 3 libraries through
its TASTy reader, and a project on Scala 2.13.16 or later that does so can *call* LLM4S: load a provider, complete
and stream, build options, run an agent and run tools. What it cannot do is *extend* LLM4S with Scala types (define a
tool with `ToolBuilder`, ask for structured output into a case class, implement `LLMClient`); for the first two,
`llm4s-java-api` is the route, and it works from Scala 2.13 through the TASTy reader as well.
These are not promises written from reading the code: each statement below is a probe, a small Scala 2.13 program
that CI compiles and runs against the current build, on Scala 2.13.16 and the latest 2.13, and that fails the build
when its outcome changes ([what is checked](#how-this-is-checked)).
## Setup
(not compile-checked)
```scala
// build.sbt of a Scala 2.13 project, which reads LLM4S through the TASTy reader
scalaVersion := "2.13.16" // or later: the TASTy reader of 2.13.16 reads the Scala 3.7 artifacts
scalacOptions += "-Ytasty-reader"
libraryDependencies ++= Seq(
("org.llm4s" %% "llm4s-core" % "0.5.1").cross(CrossVersion.for2_13Use3),
("org.llm4s" %% "llm4s-ollama" % "0.5.1").cross(CrossVersion.for2_13Use3), // one provider module per provider
("org.llm4s" %% "llm4s-agent" % "0.5.1").cross(CrossVersion.for2_13Use3) // if you run agents
)
```
`CrossVersion.for2_13Use3` makes `%%` pick the `_3` artifacts. Two rules keep the classpath consistent:
- **Do not add the `_2.13` build of a library LLM4S uses as `_3`** (cats, uPickle, pureconfig) next to it: the two
builds share class names, and whichever loads first wins. Let LLM4S bring the `_3` ones.
- **`scala-library` must be 2.13.16 or later at run time**, the version LLM4S's Scala 3 standard library is built
on. The [Spark guide](../guide/spark) covers runtimes that pin an older one.
## Calling LLM4S
From Scala 2.13, pass LLM4S's `using` parameters as an explicit second argument list:
(not compile-checked)
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model.{Conversation, UserMessage}
import org.llm4s.llmconnect.spi.ProviderRegistry
val registry = ProviderRegistry.default
val reply = for {
models <- Llm4sConfig.modelRegistryService()
config <- Llm4sConfig.defaultProvider()(registry)
client <- LLMConnect.getClient(config)(models, registry)
completion <- client.complete(Conversation(Seq(UserMessage("Hi"))))
} yield completion.content
```
Configuration is the same `application.conf` as from Scala 3 ([Configuration](configuration)).
## What works
| From Scala 2.13 | Works | Probe |
|---|---|---|
| `Conversation`, messages, `Result` | yes | `A_conversation` |
| `Llm4sConfig` + `LLMConnect.getClient` + `complete` | yes | `B_config_client` |
| `streamComplete` with a Scala 2.13 callback | yes | `C_streaming` |
| `CompletionOptions` builders, `ReasoningEffort` and the other enumerations | yes | `D_options_enums` |
| Running tools through a `ToolRegistry` you were given | yes | `E2_tool_registry` |
| `Agent.builder(...).build()`, `run`, `continueConversation` | yes | `G_agent` |
| Defining a tool with `llm4s-java-api`'s `JTool.builder`, and an agent calling it | yes | `J_java_api_tool` |
| Structured output into a Java record with `llm4s-java-api`'s `JLlmClient.complete(prompt, Record.class)` | yes | `K_java_api_structured` |
| Defining a tool with `ToolBuilder` | **no** | `E_tools` |
| `completeStructured[A]` | **no** | `F_structured` |
| Implementing `LLMClient` | **no** | `H_subclass_client` |
| Naming an agent id: `ThreadId("...")` | **no** | `I_opaque_ids` |
## The four caveats and their workarounds
### 1. Defining a tool with `ToolBuilder`
`ToolBuilder` takes a uPickle `ReadWriter` for the tool's result, and the TASTy reader cannot read uPickle's Scala 3
type classes, whether summoned or written by hand. Use `llm4s-java-api`'s tool builder, which needs no codec, and the
agent it builds:
(not compile-checked)
```scala
// add ("org.llm4s" %% "llm4s-java-api" % ...).cross(CrossVersion.for2_13Use3)
import org.llm4s.javaapi.{JTool, JToolArgs, Llm4s}
val add = JTool
.builder("add", "Adds two integers")
.param("a", classOf[Integer], "the first")
.param("b", classOf[Integer], "the second")
.handler((args: JToolArgs) => Integer.valueOf(args.getInt("a") + args.getInt("b")))
.build()
.get()
val client = Llm4s.createDefaultClient().get()
val answer = Llm4s.createAgentWithTools(client, java.util.List.of(add)).get().run("What is 2 + 3?")
```
A Scala 2.13 lambda converts to the Java `JToolHandler` interface. Tools that LLM4S already defines (the built-in
tools of `llm4s-agent-tools`) are ordinary values, and an agent or a `ToolRegistry` runs them from Scala 2.13 as it
runs any other.
### 2. Structured output with `completeStructured`
`completeStructured[A]` takes a uPickle `Reader[A]`, for the same reason out of reach. `llm4s-java-api`'s
`JLlmClient.complete(prompt, Record.class)` derives the schema from a **Java record** and reads the answer into it.
A Scala case class is not a record, so declare the record in a Java source file of the same project, which sbt
compiles alongside the Scala:
```java
// src/main/java/City.java
public record City(String name, String country, int population) {}
```
(not compile-checked)
```scala
import org.llm4s.javaapi.Llm4s
val city: City = Llm4s.createDefaultClient().get().complete("The largest city in Japan", classOf[City]).get()
```
`JTool` and `JLlmClient.complete(prompt, Record.class)` are on `main` and not yet in a release: until the next one,
[build LLM4S locally](installation#unreleased-changes-main) to use them.
### 3. Implementing `LLMClient`
Implementing the trait means the compiler checks every member, including `completeStructured`'s `Reader`, so a
Scala 2.13 class cannot extend it. Calling the members of a client LLM4S gives you works. For a test double, use
`llm4s-java-api`'s `FakeLlmClient` (`FakeLlmClient.replying("first", "second").client()`), which the structured-output
probe uses from Scala 2.13. A custom provider is written in Scala 3, in its own module
([writing a provider](../guide/writing-a-provider)), and called from Scala 2.13 like the built-in ones.
### 4. Naming an agent id
The agent runtime's ids (`ThreadId`, `RunId`, ...) are top-level opaque types whose companions the TASTy reader of
Scala 2.13 does not see, so `ThreadId("saved-1")` does not compile. Pass back the ids LLM4S returns
(`result.threadId`, or `agent.continueConversation(result, ...)`); through `llm4s-java-api` the ids are plain
`String`s (`JAgentResult.threadId()`).
## How this is checked
The probes live in
[`docs/design/scala-2.13-interop/probes`](https://github.com/llm4s/llm4s/tree/main/docs/design/scala-2.13-interop/probes),
and [`expected.md`](https://github.com/llm4s/llm4s/blob/main/docs/design/scala-2.13-interop/expected.md) beside them
holds each one's expected outcome. The *Scala 2.13 probes* workflow publishes the current build to the runner's local
repository, runs every probe on Scala 2.13.16 and the latest Scala 2.13 release, and fails when an outcome differs from the table in
either direction: a change that breaks one of the calls above fails, and so does one that fixes a caveat until this
page and the table say so. It runs nightly and on every change to the frozen modules, `llm4s-java-api` or the probes.
To run them yourself against a release, with [Scala CLI](https://scala-cli.virtuslab.org/):
```bash
cd docs/design/scala-2.13-interop
./run-probes.sh # the latest release, on 2.13.16 and the latest 2.13
```
The measurements and the reasons behind each caveat are in the design record,
[Calling LLM4S from Scala 2.13](https://github.com/llm4s/llm4s/blob/main/docs/design/scala-2.13-interop.md).
---
Source: https://llm4s.org/getting-started/mill
# Mill Quick Start
A first LLM call, and a test of it that needs no API key, in a [Mill](https://mill-build.org) project.
---
Mill 1.x writes its build in Scala 3, in a `build.mill` file, and LLM4S is published for Scala 3, so a Mill project
depends on LLM4S like on any other Scala library. This page is a complete project: the build, a program that asks a
model one question, its configuration, and a unit test that runs without a model.
## Before you start
- **Nothing to install for Mill**: the project carries Mill's launcher script, `./mill`, which downloads the Mill
version the build names the first time it runs. It downloads JDK 21 as well (the `mill-jvm-version` line), the JDK
LLM4S needs (see [Installation](installation#prerequisites)).
- **An API key** for OpenAI, or a local [Ollama](ollama-quickstart) (no key). The test needs neither.
## The files
Make a directory with these files. The same files are in
[`modules/samples/mill`](https://github.com/llm4s/llm4s/tree/main/modules/samples/mill), where CI builds them and runs
the test against the latest release on Maven Central.
```
hello-llm4s/
mill (the launcher, see below)
build.mill
app/src/Hello.scala
app/resources/application.conf
app/resources/logback.xml
app/test/src/HelloTests.scala
```
Mill's `ScalaModule` reads sources from `/src`, resources (the files put on the classpath) from
`/resources`, and a test module's sources from `/test/src`. The module here is `app`.
### The launcher
Download it from Maven Central and make it executable (on Windows, download `mill-dist-1.3.1-mill.bat` from the same
directory and save it as `mill.bat`):
```bash
curl -fL https://repo1.maven.org/maven2/com/lihaoyi/mill-dist/1.3.1/mill-dist-1.3.1-mill.sh -o mill
chmod +x mill
```
### build.mill
(not compile-checked)
```scala
//| mill-version: 1.3.1
//| mill-jvm-version: 21
package build
import mill.*, scalalib.*
val llm4sVersion = "0.5.1"
object app extends ScalaModule {
def scalaVersion = "3.7.1"
def mvnDeps = Seq(
mvn"org.llm4s::llm4s-core:$llm4sVersion",
mvn"org.llm4s::llm4s-openai:$llm4sVersion",
mvn"org.llm4s::llm4s-ollama:$llm4sVersion",
mvn"ch.qos.logback:logback-classic:1.5.34"
)
object test extends ScalaTests, TestModule.Munit {
def mvnDeps = Seq(
mvn"org.scalameta::munit:1.3.6",
mvn"org.llm4s::llm4s-testkit:$llm4sVersion"
)
}
}
```
- The `//|` lines at the top are Mill's header: the Mill version, and the JDK Mill runs and compiles with.
- `::` in a `mvn"..."` dependency adds the Scala binary suffix (`llm4s-core_3`), as `%%` does in sbt; a Java library
such as logback takes a single `:`.
- `llm4s-core` holds no provider client, so the build adds the provider modules the program can use,
`llm4s-openai` and `llm4s-ollama`; keep only the one you use. It brings no logging backend either, so the build
adds logback, and `app/resources/logback.xml` configures it.
- Every llm4s module is released at the same version, so `llm4sVersion` names it once.
### app/src/Hello.scala
(not compile-checked)
```scala
package hello
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.{ LLMClient, LLMConnect }
import org.llm4s.llmconnect.model.{ Conversation, UserMessage }
import org.llm4s.model.ModelRegistryService
import org.llm4s.types.Result
object Hello {
/** Asks `client` one question. Taking the client as a parameter is what lets the test script it. */
def ask(client: LLMClient, question: String): Result[String] =
client.complete(Conversation(Seq(UserMessage(question)))).map(_.content)
def main(args: Array[String]): Unit = {
val answer = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
text <- ask(client, "In one sentence, what is Scala?")
} yield text
answer.fold(error => println(s"Error: ${error.formatted}"), text => println(text))
}
}
```
1. `Llm4sConfig.defaultProvider()` reads the section of `application.conf` that `llm4s.providers.provider` names.
This is the one place configuration is read.
2. `LLMConnect.getClient` builds the client for that provider.
3. `client.complete` sends the conversation. Every step returns a `Result`, so the `for` stops at the first error
and the last line prints it instead of throwing.
### app/resources/application.conf
Mill puts `app/resources` on the classpath, which is where the configuration library looks for `application.conf`.
`provider` names the section the program uses; the [Configuration guide](configuration#named-provider-sections)
covers the other providers:
```hocon
llm4s {
providers {
provider = "openai-main"
provider = ${?LLM4S_PROVIDER}
openai-main {
provider = "openai"
model = "gpt-4o-mini"
}
ollama-local {
provider = "ollama"
model = "llama3"
baseUrl = "http://localhost:11434"
baseUrl = ${?OLLAMA_BASE_URL}
}
}
}
```
The OpenAI section needs no `apiKey` line: `llm4s-openai` reads `OPENAI_API_KEY` itself. The second `provider` line
lets the environment pick the other section (`LLM4S_PROVIDER=ollama-local`). Only the section being loaded is
checked, so the Ollama run does not need an OpenAI key.
### app/resources/logback.xml
It keeps the library's own logging out of your first answer (without it, logback prints every debug message):
```xml
System.err
%d{HH:mm:ss} %-5level %logger{20} - %msg%n
```
### app/test/src/HelloTests.scala
`ScriptedLLMClient`, from `llm4s-testkit`, is an `LLMClient` that answers from a script, so the test runs with no
model, no network and no key (see [Testing with the testkit](../guide/testing-with-the-testkit)):
(not compile-checked)
```scala
package hello
import org.llm4s.testkit.{ Reply, ScriptedLLMClient }
class HelloTests extends munit.FunSuite {
test("ask returns the model's answer") {
val client = ScriptedLLMClient.always(Reply.text("Scala is a JVM language."))
assertEquals(Hello.ask(client, "What is Scala?"), Right("Scala is a JVM language."))
}
test("ask sends the question as the user's message") {
val client = ScriptedLLMClient.always(Reply.text("ok"))
Hello.ask(client, "What is Scala?")
assertEquals(client.callCount, 1)
assertEquals(client.lastCall.flatMap(_.lastUserText), Some("What is Scala?"))
}
test("ask returns an error rather than throwing") {
val client = ScriptedLLMClient.sequence() // no reply scripted: the call fails
assert(Hello.ask(client, "What is Scala?").isLeft)
}
}
```
## Run it
```bash
cd hello-llm4s
./mill app.test # no key, no model
export OPENAI_API_KEY=sk-...
./mill app.run
LLM4S_PROVIDER=ollama-local ./mill app.run # a local Ollama instead, no key
```
The first run downloads Mill, the JDK and the dependencies, which takes a while. `app.run` should print one
sentence, for example:
```
Scala is a language that blends object-oriented and functional programming on the JVM.
```
The wording depends on the model.
Mill runs tasks in a background daemon that it starts on the first command and reuses, which makes later commands
fast. A program that reads from the terminal (a chat loop, say) works under `./mill app.run` too: since Mill 1.1 the
daemon passes the terminal through. Older guides use `./mill -i app.run` for this; in Mill 1.x `-i` (`--interactive`)
only means `--no-daemon`, which runs without the daemon and is what you want in CI.
## If something goes wrong
**`Error: ConfigurationError: Provider 'openai-main' (provider = openai) is missing required fields:`** followed by
`- apiKey: set OPENAI_API_KEY, or set apiKey under llm4s.providers.openai-main in application.conf`: the key is not set
in the shell that runs `./mill`. Check `echo $OPENAI_API_KEY`, or use the Ollama command above.
**`Error: NetworkError: POST http://localhost:11434/api/chat: connection failed`** (the Ollama run): no Ollama server
is listening. Start it with `ollama serve`, or point `OLLAMA_BASE_URL` at the one you have.
**A wall of `DEBUG` lines before the answer**: `logback.xml` is not on the classpath. Check that it is in
`app/resources`.
More first-run problems are in [First-Run Troubleshooting](troubleshooting).
## Optional extras
### Other llm4s modules
Add them to `mvnDeps` with the same `llm4sVersion`. For example `llm4s-schema-derivation`, which derives the JSON
schema of a structured answer from a case class with `derives SchemaOf` (see [Derived Schemas](../guide/derived-schemas)):
(not compile-checked)
```scala
mvn"org.llm4s::llm4s-schema-derivation:$llm4sVersion"
```
[Installation](installation#optional-dependencies) lists every module; its sbt lines translate one for one
(`"org.llm4s" %% "llm4s-rag" % llm4sVersion` is `mvn"org.llm4s::llm4s-rag:$llm4sVersion"`).
### Aligning versions with llm4s-bom
Instead of `llm4sVersion`, Mill can import the bill of materials, `llm4s-bom`, and take every llm4s module's version
from it. The BOM is a Maven artifact with no Scala suffix, so it takes a single `:`; the modules then name no version:
(not compile-checked)
```scala
object app extends ScalaModule {
def scalaVersion = "3.7.1"
def bomMvnDeps = Seq(mvn"org.llm4s:llm4s-bom:0.5.1")
def mvnDeps = Seq(
mvn"org.llm4s::llm4s-core",
mvn"org.llm4s::llm4s-openai"
)
}
```
See [Align versions with the BOM](installation#align-versions-with-the-bom) for what the BOM does and does not pin.
## Next steps
- [First Example](first-example): conversation context, tools and streaming
- [Configuration](configuration): all providers and how keys are found
- [Testing with the testkit](../guide/testing-with-the-testkit): scripting tool calls and multi-turn conversations
- [Ollama Quick Start](ollama-quickstart): local models
---
Source: https://llm4s.org/ai-assistants
# Using llm4s with AI coding assistants
Give your coding assistant the current llm4s documentation and the ten rules it most often gets wrong.
---
Assistants were trained mostly on older llm4s, or on none. They invent `Llm4sConfig` overloads that no longer exist,
read `LLM_MODEL`, call a provider from `llm4s-core` (which ships none), throw instead of returning `Result`, or reach
for `ConfigFactory.load()`. Two things fix most of it: the documentation in a form assistants read, and a short rules
file in your project.
## The documentation, as files
Both files are generated from this site's pages whenever the site is built, so they are never hand-edited copies.
| File | Contents |
|------|----------|
| [`/llms.txt`](/llms.txt) | A curated index ([llmstxt.org](https://llmstxt.org/)): the release version, then every page of the getting-started pages, the user guide, the cookbook and the Java and Kotlin guides, in navigation order, each with its one-line description. |
| [`/llms-full.txt`](/llms-full.txt) | The full text of those same pages in one file. Every Scala block comes from a page whose snippets are compiled against the release (`DocSnippets`); a block on any other page is preceded by `(not compile-checked)`. |
| [`/llm4s-rules.md`](/llm4s-rules.md) | The rules block below, as a file to drop into your project. |
`/llms.txt` follows the latest release. A project pinned to an older minor release can use the copy kept for it,
`/llms//llms.txt` (with `llms-full.txt` and `llm4s-rules.md` beside it), for the latest minor and the one
before it.
## Point your assistant at llm4s
Put the rules file in the place your tool reads, and, for a tool that fetches URLs, give it `llms.txt`.
| Tool | Where the rules go | The documentation |
|------|--------------------|-------------------|
| Claude Code | `CLAUDE.md` (or `AGENTS.md`, imported from it) | Ask it to read `https://llm4s.org/llms-full.txt`, or add the URL to your `CLAUDE.md` |
| Codex | `AGENTS.md` | Name `https://llm4s.org/llms.txt` in `AGENTS.md` |
| GitHub Copilot | `.github/copilot-instructions.md` | Reference `https://llm4s.org/llms.txt` in a chat with `#fetch` |
| Cursor | a rule in `.cursor/rules/` | Add `https://llm4s.org/llms-full.txt` under Settings, Indexing and Docs |
| Metals with an MCP client | `AGENTS.md` or your client's instructions file | Whatever your client offers for URLs |
For example, to start a project with the rules:
```bash
curl -fsSL https://llm4s.org/llm4s-rules.md >> AGENTS.md
```
## The rules
This block is the content of `/llm4s-rules.md`. Its Scala examples are compiled with the rest of the documentation,
so they match the current release.
## Rules for writing llm4s code
llm4s is Scala 3 on JDK 21. Follow these when you write or change code that uses it.
### 1. Return `Result[A]`; never throw
`Result[A]` is `Either[LLMError, A]`. A failing call is a `Left`, so compose with `for`, `map` and `fold`, and convert a
`Try` with `.toResult`.
```scala
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model.{ Conversation, UserMessage }
import org.llm4s.types.Result
def answer(client: LLMClient, question: String): Result[String] =
client.complete(Conversation(Seq(UserMessage(question)))).map(_.content)
```
### 2. Load configuration with `Llm4sConfig`, once, at the edge of the application
Core code receives the client it needs as a parameter.
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.{ LLMClient, LLMConnect }
import org.llm4s.model.ModelRegistryService
import org.llm4s.types.Result
def startup(): Result[LLMClient] = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
} yield client
```
### 3. Name the provider in `application.conf`, not in code
A chat provider is a named section under `llm4s.providers`, and `llm4s.providers.provider` names the default. Load the
default with `Llm4sConfig.defaultProvider()`.
```hocon
llm4s {
providers {
provider = "openai-main"
openai-main {
provider = "openai"
model = "gpt-4o-mini"
}
}
}
```
### 4. A provider is its own dependency
`llm4s-core` ships no provider. Add the module of each provider you call (`llm4s-openai`, `llm4s-anthropic`,
`llm4s-gemini`, `llm4s-ollama`, `llm4s-openai-compatible`, ...) next to it, and other features likewise (`llm4s-agent`,
`llm4s-rag`, `llm4s-agent-tools`).
### 5. Bring a `ModelRegistryService` into scope as a `given`
Do it before `LLMConnect.getClient`, as in rule 2.
### 6. Never read `LLM_MODEL`
Nothing in the library reads it. The model is the `model` key of the provider section.
### 7. Never use `sys.env`, `System.getenv`, `ConfigFactory.load()` or `ConfigSource.default`
An API key comes from `llm4s.credentials..apiKey`, which each provider module binds to the vendor's usual
variable (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, ...), or from the section's own `apiKey`. Environment variables reach
the configuration only through `${?VAR}` in a `.conf` file.
### 8. Build `CompletionOptions` with its `with` methods
Start from `CompletionOptions()` and chain `.withTemperature(...)`, `.withMaxTokens(...)`. Do not call `new` or `.copy`.
```scala
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model.{ CompletionOptions, Conversation }
def ask(client: LLMClient, conversation: Conversation) = {
val options = CompletionOptions().withTemperature(0.2).withMaxTokens(512)
client.complete(conversation, options)
}
```
### 9. A duration is a `FiniteDuration`, not a number of milliseconds
Write `30.seconds`, never `timeoutMs = 30000`.
```scala
import scala.concurrent.duration._
import org.llm4s.toolapi.ToolExecutionConfig
val toolConfig = ToolExecutionConfig.default.withTimeout(30.seconds)
```
### 10. Use the current artifact names and the latest release version
The group is `org.llm4s`, every artifact is `llm4s-` and Scala 3 only (`%%` in sbt, the `_3` suffix elsewhere);
`llm4s-core` is not `core`, and the Java API is `llm4s-java-api`. A project using several modules can import the
`llm4s-bom`. Take the version from the [installation page](/getting-started/installation), not from memory.
## Does it help?
The rules and `llms.txt` target the failures above, which are the ones that make a first attempt not compile. The
five prompts to test them with are: "make a first call to OpenAI", "add a tool", "stream a reply", "RAG over a
folder" and "call llm4s from Java". Give each to your assistant with and without the rules and `llms.txt`, and count
the outputs that compile.
---
Source: https://llm4s.org/guide/
# User Guide
Comprehensive guides for LLM4S features. They describe the current release, 0.5.1
(Scala 3.7.x, JDK 21); new to LLM4S? Start with [Getting Started](/getting-started/), and upgrading from 0.4.x,
with [Upgrading from 0.4.1 to 0.5.0](/migrations/0-4-1-to-0-5-0).
Since 0.5.0 each feature below is its own module next to `llm4s-core` (the agent runtime in `llm4s-agent`, RAG in
`llm4s-rag`, each provider in its own module, and so on); each guide names the module it needs, and
[Installation](/getting-started/installation#optional-dependencies) lists them all.
## Available Guides
### Fundamentals
- **[Basic Usage](basic-usage)** - Get started with LLM calls, client creation, and error handling
- **[Providers](providers)** - Overview of supported providers and how to configure them
- **[Java](java)** - Call a model from Java with `llm4s-java-api`: a client, a conversation and failures you can read without Scala
- **[Java RAG](java-rag)** - Ingest documents and ask questions from Java with `llm4s-java-rag`: answers with their sources
- **[Java Reactor](java-reactor)** - Stream a reply as a Reactor `Flux` with `llm4s-reactor`, for Spring WebFlux and other reactive pipelines
- **[Spring Boot](spring-boot)** - `llm4s-spring-boot-starter`: properties, an `LLM4STemplate` bean, asynchronous calls and a health indicator
- **[Kotlin](/getting-started/installation#for-kotlin-coroutines-and-flow)** - `llm4s-kotlin-api`: suspend functions and `Flow`s over the Java API (Experimental; not on Maven Central for 0.5.1 yet, so built locally for now)
- **[Error Handling](error-handling)** - Work with `Result[A]` and `LLMError`: pattern matching, for-comprehensions, every error type, recovery and testing
- **[Testing with the Testkit](testing-with-the-testkit)** - Script an `LLMClient` in your tests: canned replies, tool calls, injected errors, recorded requests
- **[Structured Output](structured-output)** - `completeStructured`: typed replies from a schema, which providers enforce it, and what a bad reply looks like
- **[Writing a Provider](writing-a-provider)** - Publish your own provider module against the `llm4s-core` SPI
- **[Comparing LLM4S](comparison)** - A sourced comparison with LangChain4j, Spring AI, Koog and Embabel, and a [migration guide](migrating-from-langchain4j)
- **[Caching](caching)** - Cache embeddings (exact) and model responses (semantic): configuration, keys, TTL, eviction and what the cache reports
- **[Multimodal Content](multimodal)** - Send images and documents in a user message or a tool result, and check before the request whether the provider takes them
- **[JSON Libraries](json-libraries)** - Use circe, play-json or zio-json with LLM4S: converting at the boundary, structured output and tools
### Agent Framework
- **[Agents Overview](agents/)** - Build LLM-powered agents with tools, guardrails, and multi-turn conversations
- **[Guardrails](agents/guardrails)** - Input/output validation for safety and quality
- **[Memory System](agents/memory)** - Persistent context and knowledge across conversations
- **[Handoffs](agents/handoffs)** - Agent-to-agent delegation for specialist routing
- **[Streaming Events](agents/streaming)** - Real-time execution feedback for responsive UIs
- **[Context Window Pruning](context-window-pruning)** - Keep a long conversation within the model's context window
- **[Built-in Tools](builtin-tools)** - Calculator, date and time, UUID, JSON, files, HTTP, shell and web search tools, the bundles that hold them, and what each one can do
- **[Build Your Own Tool](custom-tools)** - Write a tool with `ToolBuilder`: the `Schema` DSL, reading arguments, returning errors, registering it with an agent and testing it
- **[Derived Schemas](derived-schemas)** - Write the type once: `llm4s-schema-derivation` builds the `completeStructured` schema from a case class with `derives`.
### RAG & Semantic Search
- **[Embeddings](embeddings)** - Choose a provider and model, create an `EmbeddingClient`, embed documents and queries, and keep dimensions in step with the store
- **[Vector Store](vector-store)** - Complete RAG toolkit for semantic search and retrieval
- **Vector Backends**: SQLite (in-memory/file), PostgreSQL/pgvector, Qdrant
- **Keyword Backends**: SQLite FTS5, PostgreSQL native full-text search
- **Hybrid Search**: BM25 keyword + vector fusion with RRF strategy
- **Reranking**: Cohere cross-encoder for result refinement
- **Document Chunking**: Sentence-aware + simple chunking strategies
- **[Query Transformers](rag-query-transformers)** - Rewrite the user's query before retrieval: `LLMQueryRewriter`, `IdentityTransformer`, chaining, error behaviour and custom transformers
- **[RAG Evaluation](rag-evaluation)** - Measure and improve RAG quality
- **RAGAS Metrics**: Faithfulness, answer relevancy, context precision/recall
- **Benchmarking Harness**: Compare chunking, fusion, and embedding strategies
- **Optimization Workflow**: Data-driven RAG improvement
- **[Permission-Based RAG](permission-based-rag)** - Enterprise access control for RAG
- **Hierarchical Collections**: Organize documents by tenant, team, or project
- **Two-Level Permissions**: Collection-level `queryableBy` + document-level `readableBy`
- **Pattern Queries**: `*`, `path/*`, `path/**` for flexible collection scoping
- **Principal Management**: Map users/groups to efficient integer IDs
### Multimodal Capabilities
- **[Image Generation](image-generation)** - Generate images with DALL-E and other providers
- **[Speech](speech)** - Speech-to-text (STT) and text-to-speech (TTS)
- **[Jev decision model](jev)** - Ask TypeSafe's Jev typed questions (Noul, Choice, Score) and get typed answers: a decision client, not a chat model
### Effect Systems
- **[cats-effect](cats-effect)** - `LLMClientIO` and `AgentIO` for cats-effect `IO` and fs2 streaming
- **[ZIO](zio)** - `LLMClientZ` and `AgentZ` for ZIO 2 and ZIO Streams
- **[Apache Pekko](pekko)** - `LLMClientPekko` and `AgentPekko` for Pekko Streams
- **[Spark and Databricks](spark)** - enrich a DataFrame from a Spark job: a client per partition, `ReliableClient`, errors as data
### Java
- **[Java Threading and Cancellation](java-threading-and-cancellation)** - Which thread a call blocks, sharing one client, virtual threads, interrupts and timeouts for `llm4s-java-api`
### Observability
- **[Monitoring](observability/)** - Production monitoring for LLM4S applications
- **Tracing**: Langfuse and OpenTelemetry integration
- **Logging**: Structured JSON logging with SLF4J/Logback
- **[Provider Exchange Logging](observability/provider-exchange-logging)**: Capture raw provider requests and responses for debugging
- **Health Checks**: Startup validation and readiness probes
- **Cost Monitoring**: Token usage tracking and budget awareness
## Feature Coverage via Examples
For features not yet documented as dedicated guides, see our **[Examples Gallery](/examples/)** which includes 69 working examples:
| Feature | Examples Section |
|---------|------------------|
| Basic LLM Calling | [Basic Examples](/examples/#basic-examples) |
| Multi-Turn Conversations | [Context Management Examples](/examples/#context-management-examples) |
| Agent Framework | [Agent Examples](/examples/#agent-examples) |
| Tool Calling | [Tool Examples](/examples/#tool-examples) |
| Guardrails & Safety | [Guardrails Examples](/examples/#guardrails-examples) |
| Agent Handoffs | [Handoff Examples](/examples/#handoff-examples) |
| Memory System | [Memory Examples](/examples/#memory-examples) |
| Streaming | [Streaming Examples](/examples/#streaming-examples) |
| Embeddings & RAG | [Embeddings Examples](/examples/#embeddings-examples) |
| MCP Integration | [MCP Examples](/examples/#mcp-examples) |
| Observability | [Monitoring Guide](observability/) |
## Design Documents
For in-depth technical documentation, see our [design documents](/reference/#design-documents):
- [Agent Framework Roadmap](https://github.com/llm4s/llm4s/blob/main/docs/design/agent-framework-roadmap.md)
- [Phase 1.1: Conversations](https://github.com/llm4s/llm4s/blob/main/docs/design/phase-1.1-functional-conversation-management.md)
- [Phase 1.2: Guardrails](https://github.com/llm4s/llm4s/blob/main/docs/design/phase-1.2-guardrails-framework.md)
- [Phase 1.3: Handoffs](https://github.com/llm4s/llm4s/blob/main/docs/design/phase-1.3-handoff-mechanism.md)
## Getting Help
- Browse [examples](/examples/) for working code samples
- Check the [Scaladoc](/scaladoc/) for API documentation
- Join our [Discord community](https://discord.gg/4uvTPn6qww) for support
---
Source: https://llm4s.org/guide/basic-usage
# Basic Usage Guide
Learn the fundamentals of LLM4S: creating clients, making LLM calls, and handling results.
---
## Getting Started with LLM Calls
LLM4S makes it simple to integrate Large Language Models into your Scala applications. The core workflow is:
1. **Configure** a named provider section in your `application.conf`
2. **Create** an LLM client
3. **Send** messages to the LLM
4. **Handle** the result (success or error)
Let's walk through each step.
---
## Simple Client Creation Examples
### Minimal Example
The simplest way to get an LLM response:
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.{LLMClient, LLMConnect}
import org.llm4s.llmconnect.model.{Conversation, UserMessage}
import org.llm4s.model.ModelRegistryService
object SimpleExample extends App {
// Step 1: Load the configured default named provider
val startup = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
} yield {
// Step 2: Create a simple message
val conversation = Conversation(Seq(UserMessage("What is Scala?")))
// Step 3: Get a response
val response = client.complete(conversation)
// Step 4: Handle the result
response match {
case Right(completion) =>
println(s"Response: ${completion.content}")
case Left(error) =>
println(s"Error: $error")
}
}
// Execute and report any startup errors
startup.left.foreach(err => println(s"Startup Error: $err"))
}
```
The fragments in the rest of this guide use the imports of this example, plus `org.llm4s.types.Result`
(and `org.llm4s.error.LLMError` where they name it), and a `client` built as above.
**Before running, configure a provider** in `src/main/resources/application.conf`:
```hocon
llm4s {
providers {
provider = "openai-main" # the default: the name of a section below
openai-main {
provider = "openai"
model = "gpt-4o-mini"
}
}
}
```
and export the vendor's key, which `llm4s-openai` binds for any OpenAI section without an
`apiKey` of its own:
```bash
export OPENAI_API_KEY=sk-proj-...
```
For Anthropic, use a section with `provider = "anthropic"`, export `ANTHROPIC_API_KEY`, and name
the section in `provider`. llm4s reads no `LLM_MODEL`. See
[Named provider sections](../getting-started/configuration#named-provider-sections) and
[API keys](../getting-started/configuration#api-keys).
### Multi-Provider Pattern
LLM4S resolves whichever named provider you configure as the default:
```scala
// Written against the LLMClient interface, not a provider
def processWithAnyProvider(client: LLMClient): Result[Completion] =
client.complete(Conversation(Seq(UserMessage("What is Scala?"))))
val startup = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
} yield processWithAnyProvider(client)
```
The same code works with any configured named provider without modifications: change
`llm4s.providers.provider`, or load another section with `Llm4sConfig.provider("")`. See
[Switching providers](../getting-started/configuration#switching-providers).
### With Explicit Model Selection
The model comes from the section's `model`; to use another one, load a section that names it with
`Llm4sConfig.provider("")`. To adjust the generation settings of one call, pass `CompletionOptions`:
```scala
val startup = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
} yield {
val conversation = Conversation(Seq(UserMessage("Tell me about Scala")))
// Adjust generation settings while keeping the configured default provider
val response = client.complete(
conversation,
CompletionOptions(maxTokens = Some(512))
)
response.map(completion => println(completion.content))
}
```
---
## Understanding Result Types and Error Handling
### The Result Type
In LLM4S, operations return `Result[A]` instead of throwing exceptions. This is a type alias for `Either[LLMError, A]`:
```scala
type Result[+A] = Either[LLMError, A]
```
This approach provides:
- **Type-safe error handling** - Errors are part of the type signature
- **No surprise exceptions** - All failures are explicit
- **Composable operations** - Easy to chain operations with `for` comprehensions
### Pattern Matching on Results
The most common approach is pattern matching:
```scala
val result: Result[Completion] = client.complete(conversation)
result match {
case Right(completion) =>
// Success! Access the response
println(s"Content: ${completion.content}")
println(s"Model: ${completion.model}")
case Left(error) =>
// Handle the error
println(s"Error: ${error.message}")
println(s"Type: ${error.getClass.getSimpleName}")
}
```
### LLM Errors
All LLM operations return an `LLMError` on failure. It is an open trait in `org.llm4s.error`
with a `message`, an optional `code` and a `context` map, and each subtype is either
recoverable (worth retrying) or not:
```scala
import org.llm4s.error._
def handle(error: LLMError): Unit = error match {
case _: AuthenticationError => // the key was rejected: fix the credentials
case _: RateLimitError => // wait, then retry
case _: NetworkError => // transient: retry
case _: ConfigurationError => // missing or invalid configuration
case other => println(other.formatted)
}
// true for a RecoverableError (RateLimitError, NetworkError, TimeoutError, ...), false otherwise
def retryable(error: LLMError): Boolean = LLMError.isRecoverable(error)
```
`isRecoverable` answers for every error. One that carries neither marker (`EmbeddingError`, `RerankError`,
a custom `LLMError` that does not say) is not recoverable.
See the [Error Handling guide](error-handling) for every error type, when it is raised, and how
to handle, convert and test them.
### Using For-Comprehensions
For cleaner code, use Scala's `for` comprehensions:
```scala
val result = for {
// Configure and create client
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
// Make the LLM call — complete takes a Conversation and optional CompletionOptions
completion <- client.complete(
Conversation(Seq(UserMessage("Hello, LLM!")))
)
} yield {
// All operations succeeded - work with the completion
completion.content
}
// Execute
result match {
case Right(content) => println(s"Success: $content")
case Left(error) => println(s"Failed: ${error.message}")
}
```
This is much cleaner than nested pattern matches!
### Converting from Try to Result
If you have code using `Try`, convert it with the `TryOps` extension:
```scala
import org.llm4s.types.TryOps
import scala.util.Try
val tryValue = Try("123".toInt)
val result = tryValue.toResult // Result[Int]
```
---
## Message Management
### User Messages
Send simple user queries:
```scala
val userMsg = UserMessage("What is Scala?")
val response = client.complete(Conversation(Seq(userMsg)))
```
### Assistant Messages
Include assistant responses for multi-turn conversations:
```scala
val conversation = Conversation(Seq(
UserMessage("What is Scala?"),
AssistantMessage("Scala is a functional programming language..."),
UserMessage("Tell me more about its type system")
))
val response = client.complete(conversation)
```
### System Messages
Prepend a `SystemMessage` to the conversation to set context and behavior:
```scala
val conversation = Conversation(Seq(
SystemMessage("You are a Scala expert. Answer concisely."),
UserMessage("What is a monad?")
))
val response = client.complete(conversation)
```
---
## Configuration Methods
### application.conf (Recommended)
Providers are named sections under `llm4s.providers` in your `src/main/resources/application.conf`,
with secrets bound from the environment by `${?VAR}`:
```hocon
llm4s {
providers {
provider = "openai-main"
openai-main {
provider = "openai"
model = "gpt-4o"
}
}
}
```
```bash
export OPENAI_API_KEY=sk-...
```
### Environment Variables
llm4s reads an environment variable only where a `${?VAR}` binding names it - in your own
`application.conf`, or in a module's `reference.conf`. A few settings are bound for you,
for example:
```bash
export OPENAI_API_KEY=sk-... # llm4s.credentials.openai.apiKey
export TRACING_MODE=console # llm4s.tracing.mode
export EMBEDDING_MODEL=openai/text-embedding-3-small # llm4s.embeddings.model
```
`LLM_MODEL` is **not** among them. See
[Environment variables llm4s reads](../getting-started/configuration#environment-variables-llm4s-reads).
### System Properties
JVM system properties override `application.conf`, key for key:
```bash
java -Dllm4s.providers.provider=claude \
-Dllm4s.providers.openai-main.model=gpt-4o-mini \
-jar app.jar
```
---
## Common Patterns
### Handling Invalid Configuration
Always check startup results:
```scala
val startup = for {
config <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(config)
} yield client
startup match {
case Right(client) =>
// Use the client
val result = client.complete(Conversation(Seq(UserMessage("Hello!"))))
case Left(error) =>
// Configuration failed - log and exit
System.err.println(s"Startup failed: ${error.message}")
System.exit(1)
}
```
### Retrying Failed Calls
For transient failures, implement retry logic:
```scala
def retryWithBackoff[T](maxRetries: Int = 3)(
operation: () => Result[T]
): Result[T] = {
(1 until maxRetries).foldLeft(operation()) { (acc, attempt) =>
if (acc.isRight) acc
else {
Thread.sleep(math.pow(2, attempt.toDouble).toLong * 100)
operation()
}
}
}
// Usage
val response = retryWithBackoff(3) { () =>
client.complete(Conversation(Seq(UserMessage("Hello!"))))
}
```
### Logging Errors
Use SLF4J for consistent logging:
```scala
import org.slf4j.LoggerFactory
val logger = LoggerFactory.getLogger(getClass)
val response = client.complete(Conversation(Seq(UserMessage("Hello!"))))
response match {
case Right(completion) =>
logger.info(s"Got response: ${completion.content}")
case Left(error) =>
// LLMError is not a Throwable, so pass only the message string
logger.error(s"LLM call failed: ${error.message}")
}
```
---
## Troubleshooting
### "Invalid API Key"
- Verify `OPENAI_API_KEY` (or the variable the section's own `apiKey` binds, which wins) is set and correct
- Check the API key has the right permissions on the provider's dashboard
- Ensure no extra whitespace in the key
### "Model not found"
- Verify the model name matches the provider's API
- Check [MODEL_METADATA.md](/MODEL_METADATA.md) for available models
- Some models may not be available in your region or account
### "Configuration not found"
- Ensure `application.conf` (in `src/main/resources`) has a section under `llm4s.providers` and
that `llm4s.providers.provider` names it - `LLM_MODEL` is not read
- Ensure the vendor's variable (or the one the section's own `apiKey` binds) is exported in the
shell that starts the JVM
- Check the error names the section you meant to load: `defaultProvider()` loads the one
`llm4s.providers.provider` names, and only that section is validated
### "Connection timeout"
- Verify internet connectivity
- Check if the provider's service is operational
- Try using a different network or VPN
- Raise the provider's timeout in configuration if needed: a section's `timeouts { request = 5m }` block (see [Timeouts](../getting-started/configuration#timeouts))
---
Source: https://llm4s.org/guide/providers
# Providers Guide
LLM4S supports multiple LLM providers out of the box. Choose your provider, configure it, and start building.
---
## Supported Providers
LLM4S supports these LLM providers, plus any endpoint that speaks the OpenAI chat-completions API. What each one
supports with LLM4S - chat, streaming, tool calling, embeddings, API key or workload identity, live smoke suites - is in
the [compatibility matrix](../reference/compatibility-matrix), generated from the provider modules' code and checked in
CI, so this table does not repeat it:
| Provider | `provider` id | Type | Models | Setup |
|----------|---------------|------|--------|-------|
| **OpenAI** | `openai` | Cloud | GPT and o-series models | Medium |
| **Anthropic** | `anthropic` | Cloud | Claude models | Medium |
| **Google Gemini** | `gemini` | Cloud | Gemini models, with a free tier | Medium |
| **Google Vertex AI** | `vertexai` | Cloud Enterprise | Gemini models in your own GCP project | Medium |
| **Azure OpenAI** | `azure` | Cloud Enterprise | OpenAI models in your Azure deployment | Hard |
| **DeepSeek** | `deepseek` | Cloud | DeepSeek models | Easy |
| **OpenRouter** | `openrouter` | Cloud gateway | Many vendors' models behind one key | Easy |
| **Z.ai** | `zai` | Cloud | GLM models | Easy |
| **OpenAI-compatible** | `openai-compatible` | Cloud or local | Groq, Together, Fireworks, NVIDIA NIM, OrcaRouter; vLLM, LM Studio, llama.cpp - config only ([recipes](#openai-compatible-endpoints)) | Easy |
| **Mistral** | `mistral` | Cloud | Mistral and Magistral models | Easy |
| **Cohere** | `cohere` | Cloud | Command models | Easy |
| **xAI** | `xai` | Cloud | Grok models | Easy |
| **Ollama** | `ollama` | Local | Open models on your machine, no API key | Easy |
| **IBM watsonx.ai** | `watsonx` | Cloud Enterprise | Granite, Llama and Mistral models under IBM governance | Medium |
| **Amazon Bedrock** | `bedrock` | Cloud Enterprise | Models on AWS, through the Converse API (`llm4s-bedrock`) | Medium |
| **Requesty** | `requesty` | Cloud gateway | Many vendors' models behind one key | Easy |
Missing a vendor? See [Writing a Provider](writing-a-provider) to publish your own provider module.
---
## Provider Selection
### How It Works
Each provider you use is a **named section** under `llm4s.providers` in your own
`application.conf`. `llm4s.providers.provider` names the default section, which
`Llm4sConfig.defaultProvider()` loads; `Llm4sConfig.provider("")` loads any section by name.
A section's API key comes from its own `apiKey` when it sets one, and otherwise from its
vendor's shared key, `llm4s.credentials..apiKey`, which each provider module binds to
the vendor's conventional variable (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, ...). So with the
variable exported, a section needs only `provider` and `model`. The library reads no `LLM_MODEL`.
See [API keys](../getting-started/configuration#api-keys) for the full table and for a
second account.
```hocon
# src/main/resources/application.conf
llm4s {
providers {
provider = "openai-main" # the default: the name of a section below
openai-main { # key from OPENAI_API_KEY
provider = "openai"
model = "gpt-4o-mini"
}
claude { # key from ANTHROPIC_API_KEY
provider = "anthropic"
model = "claude-sonnet-4-20250514"
}
}
}
```
```scala
import org.llm4s.config.Llm4sConfig
val default = Llm4sConfig.defaultProvider() // openai-main
val claude = Llm4sConfig.provider("claude") // any section, by name
```
Only the section being loaded is validated, so a section with no key available - or whose
provider module is not on the classpath - fails only when it is loaded: with the file above and
just `OPENAI_API_KEY` set, `defaultProvider()` succeeds and `provider("claude")` fails with an
error naming `ANTHROPIC_API_KEY` and `claude`'s `apiKey`. See
[Named provider sections](../getting-started/configuration#named-provider-sections) and
[Switching providers](../getting-started/configuration#switching-providers) for the full story.
### Available Models
See [MODEL_METADATA.md](/MODEL_METADATA.md) for the complete model list. Quick reference:
**OpenAI:** `gpt-4o`, `gpt-4-turbo`, `gpt-3.5-turbo`
**Anthropic:** `claude-opus-4-6`, `claude-sonnet-4-5-latest`, `claude-haiku-3-5`
**Google Gemini:** `gemini-2.0-flash`, `gemini-1.5-pro`, `gemini-1.5-flash`
**DeepSeek:** `deepseek-chat`, `deepseek-reasoner`
**Mistral:** `mistral-large-latest`, `mistral-small-latest`, `magistral-medium-latest`
**Cohere:** `command-a-03-2025`
**Ollama:** `mistral`, `llama2`, `neural-chat`, `nomic-embed-text` (100+ models)
---
## OpenAI
From the release after `0.4.1`, OpenAI support - chat, embeddings, and the Azure OpenAI and
Requesty providers that share its client - ships in its own artifact, alongside `llm4s-core`;
adding it registers the providers:
```scala
libraryDependencies += "org.llm4s" %% "llm4s-openai" % llm4sVersion
```
In `0.4.1` and earlier it is part of `llm4s-core`. See the
[installation guide](../getting-started/installation#for-openai-azure-openai-and-requesty).
### Setup
1. **Get an API key** from [platform.openai.com/api-keys](https://platform.openai.com/api-keys)
2. **Add a named section** to `application.conf` and select it as the default (below).
3. **Export the key** - `llm4s-openai` binds it to `llm4s.credentials.openai.apiKey`, which the
section uses:
```bash
export OPENAI_API_KEY=sk-proj-...
```
### Configuration
In `application.conf`:
```hocon
llm4s {
providers {
provider = "openai-main"
openai-main {
provider = "openai"
model = "gpt-4o"
# apiKey = ${?OPENAI_BATCH_API_KEY} # optional; default: OPENAI_API_KEY
organization = ${?OPENAI_ORGANIZATION} # optional, for multi-workspace accounts
# baseUrl = "https://my-proxy.example.com/v1" # optional; default https://api.openai.com/v1
}
}
}
```
Without an `apiKey` line the section uses `OPENAI_API_KEY`. Set `apiKey` for a second account
or a variable name of your own; it wins over the shared key. `OPENAI_ORGANIZATION` is read only
because the section binds it.
### Available Models
- **Latest:** `gpt-4o`, `gpt-4o-mini`
- **Reasoning:** the gpt-5 family (`gpt-5`, `gpt-5-mini`, `gpt-5.1`, ...) and the o-series (`o3`, `o4-mini`)
- **Turbo:** `gpt-4-turbo`
- **Legacy:** `gpt-3.5-turbo`
### Reasoning models
`CompletionOptions.reasoning` is sent as `reasoning_effort` to OpenAI's reasoning models:
```scala
import org.llm4s.llmconnect.model.{ CompletionOptions, ReasoningEffort }
// client: LLMClient, conversation: Conversation
client.complete(conversation, CompletionOptions().withReasoning(ReasoningEffort.High))
```
| `ReasoningEffort` | `reasoning_effort` |
|---|---|
| `Low` / `Medium` / `High` | `low` / `medium` / `high` |
| `None` | not sent: the model's own default applies |
Which models are reasoning models comes from the model registry's `supports_reasoning` flag
(the same metadata llm4s uses for context windows and costs), so the o-series and the gpt-5
family get it, and `gpt-4o`, `gpt-4.1` and older models never do: OpenAI rejects the parameter
for them, so for those models `reasoning` is ignored, as `CompletionOptions` documents. A model
newer than the bundled metadata is recognised by its name (`o...`, `gpt-5` onwards,
`gpt-oss`). For Requesty, a routed id such as `openai/o4-mini` resolves the same way; a
non-OpenAI model behind Requesty is not sent `reasoning_effort`. A fine-tuned model is judged by
the model it was trained from: `ft:o4-mini-2025-04-16:my-org:my-suffix:abc123` is treated as
`o4-mini`, and a fine-tune of `gpt-4o-mini` as `gpt-4o-mini`.
Requests to a reasoning model also follow OpenAI's rules for them: `maxTokens` is sent as
`max_completion_tokens` (which counts reasoning tokens too) rather than `max_tokens`, and
`temperature`, `topP`, `presencePenalty` and `frequencyPenalty` are not sent, since those models
reject non-default values. (`gpt-oss` models keep their sampling parameters.)
The reasoning tokens a response reports (`completion_tokens_details.reasoning_tokens`) are
returned as `TokenUsage.thinkingTokens`, for streamed and non-streamed calls. They are part of
`completionTokens`, not in addition to it. OpenAI's chat-completions API does not return the
reasoning text itself, so `Completion.thinking` stays empty.
### Streaming and token usage
Streaming requests set `stream_options.include_usage`, so OpenAI reports token usage on a final
chunk with no choices, and a streamed `Completion` carries `usage` and `estimatedCost` like a
non-streamed one. The final chunk produces no `StreamedChunk`.
### Costs
See [OpenAI Pricing](https://openai.com/pricing). Generally:
- `gpt-4o`: $2.50-$10 per 1M input tokens
- `gpt-3.5-turbo`: $0.50-$1.50 per 1M input tokens
### Tips
- Use `gpt-4o-mini` for cost-effective applications
- Use `o1` for complex reasoning and math
- Batching API available for high-volume use
- Vision support in `gpt-4o`
---
## Anthropic
From the release after `0.4.1`, Anthropic support ships in its own artifact, alongside
`llm4s-core`; adding it registers the provider:
```scala
libraryDependencies += "org.llm4s" %% "llm4s-anthropic" % llm4sVersion
```
In `0.4.1` and earlier it is part of `llm4s-core`. See the
[installation guide](../getting-started/installation#for-anthropic).
### Setup
1. **Get an API key** from [console.anthropic.com](https://console.anthropic.com/account/keys)
2. **Add a named section** to `application.conf` and select it as the default (below).
3. **Export the key** - `llm4s-anthropic` binds it to `llm4s.credentials.anthropic.apiKey`:
```bash
export ANTHROPIC_API_KEY=sk-ant-...
```
### Configuration
In `application.conf`:
```hocon
llm4s {
providers {
provider = "anthropic-main"
anthropic-main {
provider = "anthropic"
model = "claude-opus-4-6"
# apiKey = ${?ANTHROPIC_TEAM_API_KEY} # optional; default: ANTHROPIC_API_KEY
# baseUrl = "https://my-proxy.example.com" # optional; default https://api.anthropic.com
}
}
}
```
### Available Models
- **Best Quality:** `claude-opus-4-6` (200K context)
- **Balanced:** `claude-sonnet-4-5-latest` (200K context)
- **Fast:** `claude-haiku-3-5` (200K context)
### Costs
- `claude-opus-4-6`: $3-$15 per 1M input tokens
- `claude-sonnet`: $3-$15 per 1M input tokens
- `claude-haiku`: $0.80-$4 per 1M input tokens
Claude models generally score higher on reasoning benchmarks.
### Tips
- All Claude models have 200K context window
- Exceptional at writing and analysis tasks
- Excellent vision capabilities
- Supports prompt caching for repeated queries
### Web search
Set `webSearch = true` in an `anthropic` section and every request carries Anthropic's server-side web search
tool (`web_search_20250305`). The model then decides per request whether to search, and the sources its answer
cites arrive on `Completion.citations` (see [Citations and grounding sources](#citations-and-grounding-sources)):
```hocon
llm4s.providers.claude-search {
provider = "anthropic"
model = "claude-sonnet-4-5"
webSearch = true
}
```
It is off by default. Anthropic bills each search on top of tokens, and an organization administrator has to
allow web search in the Console first. The tool is sent beside your own tools; Anthropic runs the search itself,
so it never reaches a `ToolRegistry` and the completion's `toolCalls` do not include it.
---
## Google Gemini
From the release after `0.4.1`, Gemini (and Vertex AI) support ships in its own artifact,
alongside `llm4s-core`; adding it registers both providers:
```scala
libraryDependencies += "org.llm4s" %% "llm4s-gemini" % llm4sVersion
```
In `0.4.1` and earlier they are part of `llm4s-core`. See the
[installation guide](../getting-started/installation#for-gemini-and-vertex-ai).
### Setup
1. **Get an API key** from [aistudio.google.com/apikey](https://aistudio.google.com/apikey)
- Free tier available (60 requests per minute)
2. **Add a named section** to `application.conf` and select it as the default (below).
3. **Export the key** - `llm4s-gemini` binds `GOOGLE_API_KEY` and `GEMINI_API_KEY` to
`llm4s.credentials.gemini.apiKey`, preferring `GOOGLE_API_KEY` when both are set, as Google's
SDKs do. A `provider = "google"` section uses the same key.
```bash
export GEMINI_API_KEY=your-api-key
```
### Configuration
In `application.conf`:
```hocon
llm4s {
providers {
provider = "gemini-main"
gemini-main {
provider = "gemini" # "google" is accepted as an alias
model = "gemini-2.0-flash"
# apiKey = ${?GEMINI_TEAM_API_KEY} # optional; default: GOOGLE_API_KEY or GEMINI_API_KEY
# baseUrl = "..." # optional; default https://generativelanguage.googleapis.com/v1beta
}
}
}
```
### Available Models
- **Latest:** `gemini-2.0-flash` (1M context)
- **Advanced:** `gemini-1.5-pro` (1M context)
- **Fast:** `gemini-1.5-flash` (1M context)
### Costs
- **Free tier:** 60 requests/minute, 2M free tokens/month
- **Paid:** Pay as you go (~$0.075-$1.50 per 1M input tokens)
Great for cost-conscious projects and high-volume applications.
### Tips
- Free tier perfect for development and testing
- 1M context window for processing large documents
- Very fast inference latency
- Strong code generation capabilities
---
### Google Search grounding
Set `googleSearch = true` in a `gemini` or `vertexai` section and every request carries Gemini's Google
Search tool (`google_search`; `googleSearch` on Vertex AI). The model then decides per request whether to
search, and the sources behind a grounded answer arrive on `Completion.citations` (see
[Citations and grounding sources](#citations-and-grounding-sources)):
```hocon
llm4s.providers.gemini-search {
provider = "gemini"
model = "gemini-2.5-flash"
googleSearch = true
}
```
It is off by default. Google bills grounded requests separately, and the citation URLs it returns are
`vertexaisearch.cloud.google.com` redirects, not the pages themselves. The tool is sent beside any function
declarations; a model that does not support search together with function calling rejects that request.
## Google Vertex AI
Vertex AI serves the same Gemini models from a GCP project, authenticated with OAuth2 rather
than an API key. It ships in `llm4s-gemini` beside the Gemini API provider, so the dependency
above registers both.
### Setup
1. **Pick a GCP project** with the Vertex AI API enabled, and a region that serves your model.
2. **Authenticate** with Application Default Credentials (`gcloud auth application-default login`),
or point `apiKey` at a service-account JSON file. Without `apiKey`, credentials come from
`GOOGLE_APPLICATION_CREDENTIALS`, then gcloud's `application_default_credentials.json`, then the
GCE metadata server. The file is located as Google's client libraries do: in `$CLOUDSDK_CONFIG`
when that is set (any OS), else `%APPDATA%\gcloud` on Windows, else `~/.config/gcloud`.
### Configuration
```hocon
llm4s {
providers {
provider = "vertex-main"
vertex-main {
provider = "vertexai" # "vertex" is accepted as an alias
model = "gemini-2.0-flash"
project = ${?VERTEXAI_PROJECT}
location = "europe-west4" # optional
apiKey = ${?GOOGLE_APPLICATION_CREDENTIALS} # optional: path to a credential file
}
}
}
```
| Key | Required | Meaning |
|-----|----------|---------|
| `project` | yes | The GCP project ID that owns your Vertex AI resources |
| `location` | no | The GCP region; requests go to `https://-aiplatform.googleapis.com/v1`. Default `us-central1` |
| `apiKey` | no | A path to a service-account credential file - not an API key |
Because Vertex AI has no API key, it has no shared `llm4s.credentials` key: nothing is bound for
it, and `GOOGLE_API_KEY` / `GEMINI_API_KEY` apply to the Gemini API provider only.
`project` and `location` are Vertex AI's own keys ([provider-specific keys](#provider-specific-keys)).
Before [#1215](https://github.com/llm4s/llm4s/issues/1215) Vertex read the project from `endpoint`
and the region from `organization`. Those still work as deprecated aliases, with a warning each
time the config loads; rename them. Setting both a key and a different value under its old name
is an error. See the [migration note](../reference/migration#vertex-ai-endpoint-becomes-project-organization-becomes-location).
---
## Provider-specific keys
Every named section shares the built-in fields - `provider`, `model`, `baseUrl`, `apiKey` and
`headers` - and a provider may declare keys of its own. Its section of this guide lists them:
| Provider | Its own keys |
|---|---|
| OpenAI, Requesty, OpenRouter | `organization` |
| Azure OpenAI | `endpoint` (required), `apiVersion`, `authentication`, `clientId` |
| Vertex AI | `project` (required), `location` |
| Generic `openai-compatible` | `contextWindow`, `reserveCompletion`, `registryProvider`, `streamUsage` |
A required one that is missing fails the config with its name, the section and what it means; a
key that is neither built-in nor declared by the section's provider is ignored with a warning
naming it, so a misspelt key - or `organization` in an Anthropic section - shows up in the log
rather than silently doing nothing. `organization`, `endpoint`, `apiVersion`, `contextWindow` and
`reserveCompletion` were built-in fields that every section carried until
[#1133](https://github.com/llm4s/llm4s/issues/1133); the HOCON for the providers that use them is
unchanged.
---
## IBM watsonx.ai
> **Beta, never run against the live service.** `llm4s-watsonx` calls the watsonx.ai chat API
> (`/ml/v1/text/chat` and `/ml/v1/text/chat_stream`), which replaces the "Infer text" endpoints IBM
> [deprecated in February 2026](https://www.ibm.com/docs/en/software-hub/5.3.x?topic=new-watsonxai).
> There is no watsonx account to test with, so nothing here has been checked against the real service. The
> request and response shapes are the ones IBM's public pages report, cross-checked against IBM's own
> open-source clients (the Node.js SDK and IBM's LangChain integration; the pinned sources are listed in
> `WatsonXClient`'s Scaladoc): the token-limit field, the encoding of tool-call arguments, the finish reasons
> and the stream's usage chunk are all shown there. IBM's
> [tool-calling guide](https://www.ibm.com/docs/SSYOK8/wsj/analyze-data/fm-api-chat-tools.html) shows the
> same tool request (`tool_choice_option: "auto"`) and tool-call reply, and a test pins its example. That proves IBM's clients send them, not that the
> service accepts them today. The module is Beta and its API is not frozen. The `@Cloud` suite
> `WatsonXAssumptionProbeSpec` (modules/it) checks each of those details against a real account and prints
> which held; running it is still open in [#1314](https://github.com/llm4s/llm4s/issues/1314). Until it has
> been run, treat tool calling and streaming as unconfirmed.
IBM's enterprise AI platform, serving Granite, Llama and Mistral models. It lives in its own module,
`llm4s-watsonx`; adding the dependency registers the `watsonx` provider.
```scala
libraryDependencies += "org.llm4s" %% "llm4s-watsonx" % llm4sVersion
```
```hocon
llm4s.providers.watsonx-main {
provider = "watsonx"
model = "ibm/granite-4-h-small" # any chat model your watsonx.ai region serves
projectId = ${?WATSONX_PROJECT_ID} # or spaceId for a deployment space
# baseUrl = "https://eu-de.ml.cloud.ibm.com" # default https://us-south.ml.cloud.ibm.com
# apiVersion = "2026-09-25" # the `version` query parameter; default as shown
}
```
`WATSONX_API_KEY` supplies the IBM Cloud API key, which the client exchanges for an IAM bearer token
and refreshes before it expires. The conversation goes as structured `messages` with roles
(`system`, `user`, `assistant`, `tool`), so user content is data and cannot forge a turn.
Behaviour to know about:
- **Tools.** `CompletionOptions.tools` are sent as `tools` with `tool_choice_option: "auto"`, a reply's
`tool_calls` (whole, or streamed as `delta` pieces) become `ToolCall`s, and a tool result goes back as a
`tool` message with its `tool_call_id`. Streamed calls are reported once, whole, after the last delta.
Whether a model supports tools depends on the model; a model that does not is answered by the service,
and that error is returned as the HTTP status maps it. A call the service sends without an `id` is
given a generated one.
- **Sampling options.** `temperature` is always sent, `maxTokens` (as `max_tokens`) and `topP` (when not
1.0) when set. `presencePenalty` and `frequencyPenalty` go as `presence_penalty` and `frequency_penalty`
when not 0.0, their default, so leaving them alone sends nothing new; the service checks their range.
- **Structured output.** `responseFormat` goes as `response_format`: `ResponseFormat.Json` as
`{"type": "json_object"}`, and `ResponseFormat.JsonSchema(schema, name, strict)` as
`{"type": "json_schema", "json_schema": {"name": ..., "schema": ..., "strict": ...}}`, the shapes IBM's
Python SDK and LangChain integration send (IBM's Node.js SDK names only `json_object`). JSON mode needs
the prompt to ask for JSON too; the client adds no instruction. Whether a model honours either format
depends on the model. Like the rest of this section, both are unconfirmed until the probe suite runs.
- **Reasoning.** `CompletionOptions.reasoning` goes as `reasoning_effort` (`Low`, `Medium` and `High` as
`low`, `medium` and `high`) to OpenAI's gpt-oss models (`openai/gpt-oss-120b`, `openai/gpt-oss-20b` and
their variants), the models IBM's own clients send it to. The chat API has no value that turns reasoning
off, so `ReasoningEffort.None` goes as `low`, and a warning is logged once per process. Any other model
is sent nothing for `reasoning`, again with a warning logged once per process: IBM's Model Gateway
rejects the field for models without it, and whether the chat API does is unconfirmed.
`include_reasoning` is not sent; the service defaults it to true.
- **Thinking.** A reply's `reasoning_content` (or `reasoning`) is read as thinking from any model:
`Completion.thinking` on `complete`, and `StreamedChunk.thinkingDelta` on a stream, with the whole
text on the final completion. It is not sent back on later turns. The chat API reports no reasoning
token count, so `TokenUsage.thinkingTokens` stays empty (IBM's recorded replies suggest the
reasoning is counted in `completion_tokens`). `budgetTokens` has no counterpart and is ignored. Granite models that put their
reasoning inside the content between tags (`...`) are not split: that text stays in
`content`. The field names come from IBM's SDKs and from replies IBM's Java SDK recorded from
`openai/gpt-oss-120b`; like the rest of this section, they are unconfirmed until the probe suite runs.
- **Abnormal endings.** A stream that ends without a `finish_reason`, or whose `finish_reason` is
`error`, `cancelled` or `time_limit`, returns `Left(ServiceError)` naming the reason, not the
partial text (chunks already passed to `onChunk` were delivered). `stop`, `length`, `tool_calls`
and unknown reasons are normal stops. `complete` applies the same rule to `choices[0].finish_reason`.
- **Stream error events.** An `event: error` frame ends the stream with the error IBM sends in its `data:`
line: its `status_code` picks the error as an HTTP status would (`AuthenticationError` for 401 and 403,
`RateLimitError` for 429, `ValidationError` for 400, `ServiceError` otherwise, and `ServiceError(502)`
without one), and the message is IBM's error `code` and `message`, with the API key and bearer token removed.
- **API version.** Every request carries `version=`. The default, `2026-09-25`, is the date
IBM's Python SDK (`ibm-watsonx-ai` 1.8.0) sends; set `apiVersion` in the section to pin another.
- **URLs and ids.** `baseUrl` and `iamUrl` must be `https` (plain `http` only for `localhost`,
`127.0.0.1` and `::1`), because the IAM request carries the API key. The API key is trimmed. Set
`projectId` or `spaceId`, not both.
- **Environment variables.** Only `WATSONX_API_KEY` is bound automatically. `WATSONX_PROJECT_ID` and
the other variables in the example are just `${?VAR}` substitutions you write in your own config.
---
## Azure OpenAI
Azure OpenAI ships in `llm4s-openai` with OpenAI itself, from the release after `0.4.1`; see
[OpenAI](#openai) above. It runs on OpenAI's Java SDK (`com.openai:openai-java`), which covers
Azure too: requests go to `/openai/deployments//chat/completions` with an
`api-key` header and the `api-version` query parameter. `apiVersion` accepts the wire form
(`2024-10-21`) or the constant name (`V2024_10_21`); an endpoint ending in `/openai/v1` uses
Azure's unified v1 API.
### Setup
1. **Create resource** in [Azure Portal](https://portal.azure.com)
2. **Deploy model** (e.g., gpt-4o) to get deployment name
3. **Get credentials** from Azure Portal → Keys & Endpoint
4. **Add a named section** to `application.conf` (below) and export the key, which
`llm4s-openai` binds to `llm4s.credentials.azure.apiKey` under the openai SDK's name, and the
endpoint the section binds:
```bash
export AZURE_OPENAI_API_KEY=your-azure-key
export AZURE_API_BASE=https://your-resource.openai.azure.com
```
### Configuration
In `application.conf`:
```hocon
llm4s {
providers {
provider = "azure-main"
azure-main {
provider = "azure"
model = "gpt-4o" # your deployment name
endpoint = ${?AZURE_API_BASE} # required: your resource's endpoint
# apiKey = ${?AZURE_TEAM_API_KEY} # optional; default: AZURE_OPENAI_API_KEY
apiVersion = "2024-10-21" # optional; default V2025_01_01_PREVIEW
}
}
}
```
| Key | Required | Meaning |
|-----|----------|---------|
| `model` | yes | The deployment name you chose when deploying the model |
| `endpoint` | yes | The Azure OpenAI resource endpoint |
| `apiKey` | no | The resource key; default `AZURE_OPENAI_API_KEY` |
| `apiVersion` | no | Azure API version, wire form or constant name |
| `authentication` | no | `api-key` (default) or `entra-id`: keyless [Microsoft Entra ID](#azure-openai-with-microsoft-entra-id) |
| `clientId` | no | With `entra-id`, the client id of a user-assigned managed identity |
### Available Models
Same as OpenAI (via Azure deployment). Choose models when deploying:
- `gpt-4o`
- `gpt-4-turbo`
- `gpt-35-turbo`
### Reasoning deployments
`reasoning_effort` works as for [OpenAI](#reasoning-models), with one difference: the model
llm4s sees is the **deployment name**, which need not name the model behind it. So:
- a deployment named after a model the registry knows (`o4-mini`, `gpt-5-mini`, `gpt-4o`) is
treated as that model, and one named after an Azure fine-tuned model
(`o4-mini-2025-04-16.ft-...`) as its base model;
- any other deployment (`prod-reasoner`, or a fine-tune deployed under a name of your own) gets `reasoning_effort` whenever you ask for a
reasoning effort, since asking is the only sign it is a reasoning model. The request then
also follows the reasoning-model rules: `max_completion_tokens`, and no sampling parameters.
Asking for reasoning on such a deployment that is not a reasoning model gets an error from
Azure; leave `reasoning` unset (or `ReasoningEffort.None`) for it.
### Streamed token usage
Streamed completions report token usage when `apiVersion` is `2024-09-01-preview` or later
(including the default `2025-01-01-preview`, the GA `2024-10-21` and the unified v1 API): only
those versions accept `stream_options`, so it is not sent for older ones, such as
`2024-02-15-preview`, and their streams report no usage.
### Costs
Similar to OpenAI but often bundled with enterprise agreements.
### Tips
- Use for VPC-isolated workloads
- Enterprise support available
- Same API as OpenAI (easy migration)
- Reserve capacity for predictable costs
---
## DeepSeek
From the release after `0.4.1`, DeepSeek ships in `llm4s-openai-compatible`, with Z.ai,
OpenRouter and the generic [OpenAI-compatible](#openai-compatible-endpoints) provider; adding it
registers them:
```scala
libraryDependencies += "org.llm4s" %% "llm4s-openai-compatible" % llm4sVersion
```
In `0.4.1` and earlier it is part of `llm4s-core`. See the
[installation guide](../getting-started/installation#for-deepseek-zai-openrouter-mistral-cohere-and-any-openai-compatible-endpoint).
### Setup
1. **Get API key** from [platform.deepseek.com](https://platform.deepseek.com/api_keys)
2. **Export the key** - `llm4s-openai-compatible` binds it to `llm4s.credentials.deepseek.apiKey`:
```bash
export DEEPSEEK_API_KEY=sk-...
```
### Configuration
In `application.conf`:
```hocon
llm4s {
providers {
provider = "deepseek-main"
deepseek-main {
provider = "deepseek"
model = "deepseek-chat"
# apiKey defaults to DEEPSEEK_API_KEY
# baseUrl defaults to https://api.deepseek.com
}
}
}
```
### Available Models
- **Chat:** `deepseek-chat` (best for general use)
- **Reasoning:** `deepseek-reasoner` (extended thinking). Its chain of thought
(`reasoning_content`) is returned on the message as `AssistantMessage.thinking` (and so
`Completion.thinking`), streamed as thinking deltas, and sent back as `reasoning_content` in later
requests, which DeepSeek's thinking mode requires once tools are involved
(see [Thinking in conversation history](#thinking-in-conversation-history)).
### Costs
Very competitive: ~$0.14-$0.28 per 1M input tokens
### Tips
- Excellent cost/performance ratio
- Reasoning model rivals GPT-4o
- Good for translations and multilingual tasks
- Supports very long contexts
---
## OpenRouter and Z.ai
Both ship in `llm4s-openai-compatible` from the release after `0.4.1`, as DeepSeek does.
```hocon
llm4s {
providers {
openrouter-main {
provider = "openrouter"
model = "anthropic/claude-3.5-sonnet"
# apiKey defaults to OPENROUTER_API_KEY
# baseUrl defaults to https://openrouter.ai/api/v1
}
zai-main {
provider = "zai"
model = "GLM-4.7"
# apiKey defaults to ZAI_API_KEY
# baseUrl defaults to https://api.z.ai/api/paas/v4
}
}
}
```
Both sections are shown together for brevity. Only the section you load is validated, so only
its key - `OPENROUTER_API_KEY` or `ZAI_API_KEY` - needs to be set.
OpenRouter maps `CompletionOptions.reasoning` onto the underlying model: a thinking budget for
Claude models, `reasoning_effort` for OpenAI o-series models, nothing for the rest.
Z.ai maps it onto what the configured GLM model documents. Thinking is on by default, and
`reasoning_effort` defaults to `max` on the models that accept it. The mapping only ever rises with
the effort, and `High` sends `max`, Z.ai's maximum and its default, so no level reasons more than
`High`:
| Model | `ReasoningEffort.None` | `Low` | `Medium` | `High` |
|---|---|---|---|---|
| GLM-5.3, GLM-5.3-Flash, GLM-5.3-FlashX | `reasoning_effort: low` | `low` | `high` | `max` |
| GLM-5.2 | `reasoning_effort: none` | `low` | `medium` | `max` |
| GLM-5.1, GLM-5, GLM-4.7, GLM-4.6, GLM-4.5 (and variants) | `thinking.type: disabled` | not sent | not sent | not sent |
| any other model | not sent | not sent | not sent | not sent |
GLM-5.3 always thinks and rejects `thinking.type: disabled`, so `None` gets the lowest effort it
accepts, as Z.ai advises; it still produces, and bills, thinking tokens, and the first such request
in a process logs a warning naming the model. GLM-5.3 accepts only `low`, `high` and `max`. GLM-5.2
accepts `none`, `low`, `medium`, `high` and `max`, but Z.ai currently runs `low` and `medium` as
`high` on it; llm4s still sends the level requested. The GLM-4.x and earlier GLM-5 models document
no `reasoning_effort`. With `reasoning` unset, neither field is sent. A GLM thinking
model's `reasoning_content` on Z.ai, and a model's `reasoning` and `reasoning_details` on OpenRouter,
are returned on the message and sent back (see [Thinking in conversation history](#thinking-in-conversation-history)).
---
## OpenAI-compatible endpoints
Any server that speaks the OpenAI `/chat/completions` API works with
`provider = "openai-compatible"` - hosted APIs such as Groq, Together, Fireworks, xAI or NVIDIA
NIM, local servers such as vLLM, LM Studio or llama.cpp, or an internal gateway. It needs no code
and no module of its own, only `llm4s-openai-compatible`, which ships in the release after `0.4.1`
(see the [installation guide](../getting-started/installation#for-deepseek-zai-openrouter-mistral-cohere-and-any-openai-compatible-endpoint)):
```scala
libraryDependencies += "org.llm4s" %% "llm4s-openai-compatible" % llm4sVersion
```
Each named section is one endpoint, so several can sit side by side. The
[recipes](#hosted-apis) below are sections you can paste into your `application.conf` under
`llm4s.providers`.
### Configuration keys
| Key | Required | Meaning |
|-----|----------|---------|
| `baseUrl` | yes | Requests go to `/chat/completions`, and model listing to `/models`. A trailing `/` is dropped |
| `model` | yes | Sent as-is in every request |
| `apiKey` | no | Sent as `Authorization: Bearer `; with none, no `Authorization` header is sent |
| `contextWindow` | no | The model's context window; overrides the model registry. With none, the registry's window for `model` when it has one of at least 8192 (see [where the context window comes from](#where-the-context-window-comes-from)), else 8192, which is deliberately small |
| `reserveCompletion` | no | Tokens held back for the reply. Default 2048, or a quarter of a smaller window |
| `registryProvider` | no | The [model registry](../MODEL_METADATA.md) provider whose entry for `model` gives the context window when `contextWindow` is not set: `groq`, `together_ai`, `fireworks_ai`, `xai`, `perplexity`, ... Inferred from the `baseUrl` host for those five hosted APIs; naming one here switches the inference off |
| `headers` | no | Extra headers sent on every request; values are redacted when the config is printed |
| `streamUsage` | no | Whether a streaming request asks for token usage with `stream_options.include_usage`. Default `true`; set `false` for an endpoint that rejects the field (see [streamed token usage](#what-the-generic-path-does-and-does-not-do)) |
`baseUrl` is everything before `/chat/completions` in the vendor's endpoint URL. For most vendors
that is the host plus `/v1`, but some add a prefix - Groq's is `/openai/v1`, Fireworks'
`/inference/v1` - and Perplexity's Sonar API has no `/v1` at all.
### Where the context window comes from
The generic provider cannot ask an arbitrary endpoint how large the model's window is, so it takes it from
the first of these that has an answer:
1. the section's `contextWindow`;
2. the [model registry](../MODEL_METADATA.md)'s entry for `model` under the `registryProvider` the section names;
3. when the section names none, the registry's entry under the provider inferred from the `baseUrl` host, for
exactly these hosts: `api.groq.com` (`groq`), `api.together.xyz` and `api.together.ai` (`together_ai`),
`api.fireworks.ai` (`fireworks_ai`), `api.x.ai` (`xai`) and `api.perplexity.ai` (`perplexity`);
4. 8192.
```hocon
internal-gateway {
provider = "openai-compatible"
baseUrl = "https://llm-gateway.internal.example/v1" # not a known host, so name the registry provider
model = "llama-3.1-8b-instant"
registryProvider = "groq" # the window comes from groq/llama-3.1-8b-instant
}
```
- **An explicit `registryProvider` is an instruction, not a hint.** It switches the host inference off, so a
`registryProvider = "perplexity"` on a Groq URL looks in Perplexity's entries only.
- **The lookup is strict.** Only the named provider's own entry for exactly that model id counts. A partial
name does not match, and neither does another provider's entry for the same name, so a model the registry
does not list under that provider gets the default, never a neighbour's window.
- **The host is matched exactly, never by substring.** `https://api.groq.com.evil.example/v1`,
`https://api.groq.com@evil.example/v1` and `https://evil.example/api.groq.com` all get no registry window.
NVIDIA NIM is not in the table because the registry's three `nvidia_nim` entries carry no chat model with a
window; `registryProvider = "nvidia_nim"` is still accepted.
- **The registry can only enlarge the window.** A registry window below 8192 is ignored and the default used:
many entries, most of Fireworks' and Perplexity's older ones among them, carry a 4096 placeholder for input,
output and total alike (`fireworks_ai/.../minimax-m1-80k` is one), which would cut an 80k-context model's prompt
budget in half and reject a `reserveCompletion` that fits 8192. A model whose window really is smaller sets
`contextWindow`. The skipped entry is logged like a miss.
- **Only the window is taken.** `reserveCompletion` keeps its own rule, a quarter of the window up to 2048,
because a registry entry's output limit can be as large as the whole window.
- **The registry does not know every model.** It has no input limit for the Together recipe's model, and does
not list xAI's `grok-4.7` or the NVIDIA NIM model at all, so those recipes keep their `contextWindow`. A miss
under an explicit `registryProvider` is logged as a warning, and under an inferred one at info; a hit is
logged at info with the entry it used.
### Selecting it, and environment variables
`Llm4sConfig` resolves providers from named sections only, and reads no `LLM_MODEL`. The generic
provider has no vendor, so - unlike `openai` or `anthropic` - it has no shared
`llm4s.credentials` key and no variable is bound for it. Every value lives in the named section,
and you bind environment variables to it with HOCON substitutions, under names you choose. The recipes use each vendor's own documented variable name, such as
`GROQ_API_KEY`:
```hocon
llm4s {
providers {
# the default named provider; LLM4S_PROVIDER overrides it when set
provider = "groq-main"
provider = ${?LLM4S_PROVIDER}
groq-main {
provider = "openai-compatible"
baseUrl = "https://api.groq.com/openai/v1"
model = "openai/gpt-oss-120b"
model = ${?GROQ_MODEL} # optional override
apiKey = ${?GROQ_API_KEY}
}
}
}
```
`${?VAR}` leaves the key unset when the variable is missing. For `apiKey` that is not an error
when the config loads, because the key is optional: the request goes out without an
`Authorization` header and fails with a 401 (see [Troubleshooting](#troubleshooting-openai-compatible-endpoints)).
For an endpoint chosen entirely from the environment, the conventional names are
`OPENAI_COMPATIBLE_BASE_URL` (required: the generic provider has no default endpoint) and
`OPENAI_COMPATIBLE_API_KEY` (optional), which `OpenAICompatibleConfigKeys` names:
```hocon
llm4s.providers {
provider = "compatible-env"
compatible-env {
provider = "openai-compatible"
baseUrl = ${?OPENAI_COMPATIBLE_BASE_URL}
model = ${?OPENAI_COMPATIBLE_MODEL}
apiKey = ${?OPENAI_COMPATIBLE_API_KEY}
}
}
```
With `OPENAI_COMPATIBLE_BASE_URL` or the model variable unset, that section fails to load, so
keep it in the `application.conf` of deployments that set them.
The `LLM_MODEL=openai-compatible/` shorthand is read, as `LLM_MODEL=deepseek/` and
the other providers' are, by the tools that read `LLM_MODEL` themselves: the chat-tui sample,
which then takes the endpoint from `OPENAI_COMPATIBLE_BASE_URL` and the key from
`OPENAI_COMPATIBLE_API_KEY`, and the config-policy env check, which checks
`OPENAI_COMPATIBLE_BASE_URL` rather than `OPENAI_BASE_URL` as the endpoint. Only the first `/`
separates the provider, so `LLM_MODEL=openai-compatible/openai/gpt-oss-120b` names Groq's
`openai/gpt-oss-120b`.
Then load it by name, or as the default:
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.config.ProviderConfig
import org.llm4s.types.Result
val groq: Result[ProviderConfig] = Llm4sConfig.provider("groq-main")
val dflt: Result[ProviderConfig] = Llm4sConfig.defaultProvider()
```
### What the generic path does, and does not do
It sends the plain chat-completions format and reads the plain reply: text, **streaming**,
**tool calling** (streamed and not), `response_format` for JSON output, and token usage. Every
request carries `temperature` and `top_p`; `max_tokens` is sent when set, and
`presence_penalty` and `frequency_penalty` only when non-zero. It never sends `stop` or
`tool_choice`. Whether a feature works also depends on the endpoint and the
model: tool calling on a local server usually needs a server flag, as the recipes note.
**Streamed token usage.** A streaming request asks for usage with
`"stream_options": {"include_usage": true}`, which is how OpenAI - and servers that follow it,
such as vLLM, Ollama's `/v1` and Perplexity's Router - are told to report it; usage is read from
whichever event carries it, including a final event with empty `choices`. An endpoint that
rejects unknown fields may refuse the request with a 400 or 422 naming `stream_options`. Turn it
off with `streamUsage = false` in the section (it takes `true` or `false`, and HOCON's
`yes`/`no`/`on`/`off`; anything else is a config error naming the key and section):
```hocon
strict-gateway {
provider = "openai-compatible"
baseUrl = "https://llm.example/v1"
model = "my-model"
streamUsage = false
}
```
or, building the config in code, with
`OpenAICompatibleConfig.fromValues(..., streamUsage = false)`.
A non-streaming request fails with a timeout after **two minutes** without a response, and a
streaming one after five minutes without one. Both are fixed for now; configurable timeouts are
[#712](https://github.com/llm4s/llm4s/issues/712). A slow local model answering a long prompt
can take longer than two minutes: stream it instead.
What it does **not** do:
- **No reasoning configuration.** `CompletionOptions.reasoning` is ignored; no
`reasoning_effort` or similar field is sent, so a reasoning model runs at its vendor's default.
- **No reasoning output.** A reply's `reasoning_content` or `reasoning` field is not read, so
`Completion.thinking` is empty and reasoning tokens are not reported separately.
- **No provider-specific fields.** Anything outside the standard reply - citations, search
results, annotations - is dropped; `Completion` has no field for it.
- **Text only.** Messages are sent as text.
- **No cost estimate** unless the model name is one llm4s's model registry knows.
A provider that needs any of these gets its own **dialect in `llm4s-openai-compatible`**: an
`OpenAICompatibleDialect` for how it departs from the standard format (reasoning parameters,
where its thinking is, extra response decoding) and a `ProviderDescriptor` to register it, as
DeepSeek, Z.ai, OpenRouter, Mistral and Cohere have.
[Adding an OpenAI-compatible provider](https://github.com/llm4s/llm4s/blob/main/CONTRIBUTING.md#adding-an-openai-compatible-provider-a-dialect)
in the contributing guide walks through the hooks, a worked example, registration, the tests to
write and the docs to update; see also
[Adding a provider](../reference/migration#adding-a-provider) for the descriptor.
### Hosted APIs
The values below were checked against each vendor's API documentation in September 2026 (links
under [Provider docs](#provider-docs)). Model catalogues change often: if a model is retired,
pick a current one from the vendor's list and keep the rest of the section.
#### Groq
```hocon
groq-main {
provider = "openai-compatible"
baseUrl = "https://api.groq.com/openai/v1"
model = "openai/gpt-oss-120b"
apiKey = ${?GROQ_API_KEY}
reserveCompletion = 8192
}
```
- The context window, 131072 for this model, comes from the model registry, because `api.groq.com` is
a known host. Set `contextWindow` to override it, or for a model the registry does not list.
- Model ids are Groq's own, including the `openai/` prefix here. `llama-3.3-70b-versatile`,
which earlier versions of this guide used, was shut down for free and developer tiers on
16 August 2026.
- `gpt-oss-120b` is a reasoning model. Groq returns its reasoning in a `reasoning` field, which
is not read, and the effort cannot be set.
- Groq does not support `logprobs`, `logit_bias`, `messages[].name` or `n` > 1; the generic path
sends none of them. A `temperature` of 0 is converted to 1e-8.
#### Together AI
```hocon
together-main {
provider = "openai-compatible"
baseUrl = "https://api.together.ai/v1"
model = "meta-llama/Llama-3.3-70B-Instruct-Turbo"
apiKey = ${?TOGETHER_API_KEY}
contextWindow = 131072
}
```
- Keep `contextWindow`: the registry lists this model with no input limit, so without it the provider would
use 8192. (For a Together model the registry does give a limit for, it is taken automatically.)
- Model ids are the `/` API strings from Together's serverless model list.
- For a reasoning model on Together, the same reasoning limits apply as for Groq above.
#### Fireworks AI
```hocon
fireworks-main {
provider = "openai-compatible"
baseUrl = "https://api.fireworks.ai/inference/v1"
model = "accounts/fireworks/models/gpt-oss-120b"
apiKey = ${?FIREWORKS_API_KEY}
}
```
- The context window, 131072 for this model, comes from the model registry. Set `contextWindow` to
override it, or for a model the registry does not list.
- Model ids are full paths, `accounts/fireworks/models/`. Not every model in the catalogue
is available serverless: check its model page before using it with a plain API key.
- If the prompt plus `max_tokens` exceeds the model's window, Fireworks lowers `max_tokens`
rather than failing (its default `context_length_exceeded_behavior` is `truncate`).
- Fireworks sends usage on the last chunk of a stream, so streamed token usage is reported.
`reasoning_content` is not read.
#### xAI (Grok)
xAI has a provider of its own, [`provider = "xai"`](#xai-grok), which reads Grok's reasoning and
sends `CompletionOptions.reasoning`; prefer it. The generic path still works, for chat, streaming
and tools only:
```hocon
xai-generic {
provider = "openai-compatible"
baseUrl = "https://api.x.ai/v1"
model = "grok-4.7"
apiKey = ${?XAI_API_KEY}
contextWindow = 500000
}
```
- Keep `contextWindow`: the model registry does not list `grok-4.7`, so without it the provider would use
8192. (For an xAI model the registry does list, such as `grok-2`, it is taken automatically.)
- On this path Grok's reasoning output is not read and the effort cannot be set, and xAI rejects
`presencePenalty` and `frequencyPenalty` on reasoning models: leave them at 0.
#### NVIDIA NIM (hosted)
```hocon
nim-cloud {
provider = "openai-compatible"
baseUrl = "https://integrate.api.nvidia.com/v1"
model = "meta/llama-3.3-70b-instruct"
apiKey = ${?NVIDIA_API_KEY}
contextWindow = 128000
}
```
- Keep `contextWindow`: the model registry has no usable chat entry for NVIDIA NIM, and
`integrate.api.nvidia.com` is not one of the hosts the registry provider is inferred from.
- Model ids are `/`, as in NVIDIA's API catalogue. For a NIM you run
yourself, see [NVIDIA NIM (self-hosted)](#nvidia-nim-self-hosted).
#### Perplexity
> Perplexity has replaced **Sonar Chat Completions** with its Agent API, and states that Sonar
> "will be supported until September 27, 2026". The Agent API uses the OpenAI **Responses**
> format (`/v1/responses`), which the generic provider does not speak. Check Sonar's status
> before relying on the section below.
Sonar, while it runs:
```hocon
perplexity-sonar {
provider = "openai-compatible"
baseUrl = "https://api.perplexity.ai" # no /v1: requests go to /chat/completions
model = "sonar-pro"
apiKey = ${?PERPLEXITY_API_KEY}
}
```
- The context window, 200000 for `sonar-pro`, comes from the model registry.
- **Citations and search results are not surfaced.** Sonar returns them in top-level
`citations` and `search_results` fields, and `Completion` has no field for them, so you get
the answer text - with its `[1]`-style markers - and not the sources they point to.
- Sonar's tool-calling support is undocumented; do not rely on it.
Perplexity's **Router API** speaks Chat Completions and serves open-weight models, without
Sonar's web search. It is in private preview (access by request to Perplexity), and publishes
no context windows, so set `contextWindow` for the model you pick:
```hocon
perplexity-router {
provider = "openai-compatible"
baseUrl = "https://api.perplexity.ai/router/v1"
model = "perplexity/glm-5.3-flash"
apiKey = ${?PERPLEXITY_API_KEY}
}
```
The Router reports token usage on a stream only when asked with `stream_options`, which the
generic provider sends, so streamed completions carry usage.
#### OrcaRouter
[OrcaRouter](https://www.orcarouter.ai) is an OpenAI-compatible gateway in front of several
vendors' models:
```hocon
orcarouter-main {
provider = "openai-compatible"
baseUrl = "https://api.orcarouter.ai/v1"
model = "openai/gpt-4o-mini"
apiKey = ${?ORCAROUTER_API_KEY} # keys start with sk-orca-
}
```
- OrcaRouter documents no environment variable name; `ORCAROUTER_API_KEY` is only this guide's
choice. It publishes no per-model context window either: set `contextWindow` to the
upstream model's.
- Model ids are provider-prefixed (`openai/`, `anthropic/`, `google/`, ...). OrcaRouter
accepts a reasoning effort as a model-name suffix (`-minimal`, `-low`, `-medium`, `-high`,
`-max`), and since `model` is sent as-is, that is the one way to choose an effort through the
generic path. The `reasoning_content` it passes through is not read.
### Local servers
A local server usually takes no API key: leave `apiKey` out and no `Authorization` header is
sent. Set it only if you started the server with one. Set `contextWindow` to the context the
server was started with, which may be smaller than the model's maximum.
#### vLLM
```bash
vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto
```
```hocon
local-vllm {
provider = "openai-compatible"
baseUrl = "http://localhost:8000/v1"
model = "NousResearch/Meta-Llama-3-8B-Instruct" # --served-model-name, if you set one
# apiKey = ${?VLLM_API_KEY} # only if started with --api-key
}
```
- The served model name defaults to the `--model` argument.
- Tool calling needs `--enable-auto-tool-choice` and a `--tool-call-parser` for the model's
family; without them the model never calls a tool.
#### LM Studio
Start the server from LM Studio's Developer page (default port 1234), then:
```hocon
lmstudio {
provider = "openai-compatible"
baseUrl = "http://localhost:1234/v1"
model = ""
# apiKey = ${?LM_API_TOKEN} # only with "Require Authentication" on (LM Studio 0.4.0+)
}
```
- Use the identifier LM Studio shows for the loaded model, or list them with
`Llm4sConfig.listModels("lmstudio")`.
#### llama.cpp (`llama-server`)
```bash
llama-server -hf /: --alias local-model -c 16384
```
```hocon
llama-cpp {
provider = "openai-compatible"
baseUrl = "http://localhost:8080/v1"
model = "local-model" # the --alias
contextWindow = 16384 # the -c / --ctx-size the server was started with
# apiKey = ${?LLAMA_API_KEY} # only if started with --api-key
}
```
- `llama-server` listens on `127.0.0.1:8080` by default. Without `--alias` the model id is the
model file's path; `--alias` gives it a stable name.
- Tool calling uses the Jinja chat template (`--jinja`, on by default), and some models need a
`--chat-template-file` override to get a tool-capable template.
#### NVIDIA NIM (self-hosted)
```hocon
nim-local {
provider = "openai-compatible"
baseUrl = "http://localhost:8000/v1"
model = "meta/llama-3.1-8b-instruct" # the id GET /v1/models reports
# no apiKey: NVIDIA's examples for a running NIM send none
}
```
- `NGC_API_KEY` is for pulling the container and model, not for requests: do not put it in
`apiKey`.
- The model id is the `id` from `GET /v1/models`; for a model-free NIM it is
`NIM_SERVED_MODEL_NAME`.
#### Ollama (`/v1`)
[`llm4s-ollama`](#ollama-local-models) (`provider = "ollama"`) is the first-class route to
Ollama: it uses Ollama's native `/api/chat` API, supports **tool calling** (tools are sent, tool
calls are read - whole or streamed - and tool results go back as `role: tool`) and also provides
Ollama embeddings. Tool calling needs a model whose Ollama page lists the **tools** capability: for any
other model Ollama answers HTTP 400 (`... does not support tools`), which the client reports as an
`Invalid tools: Ollama model '' does not support tool calling ...` validation error rather than
sending the request again without the tools. Use the generic provider on Ollama's OpenAI-compatible `/v1` endpoint instead
when Ollama is behind a gateway that exposes only the OpenAI API, or when you prefer that wire format:
```hocon
ollama-openai {
provider = "openai-compatible"
baseUrl = "http://localhost:11434/v1"
model = "gpt-oss:20b"
# no apiKey: local Ollama needs none
}
```
- The OpenAI API cannot set Ollama's context size. To change it, create a model from a
`Modelfile` with `PARAMETER num_ctx `, use that model's name, and set `contextWindow` to
the same size.
- Ollama's cloud API works the same way, with `baseUrl = "https://ollama.com/v1"` and
`apiKey = ${?OLLAMA_API_KEY}`.
- A reasoning model's thinking is not read on this route.
### An internal gateway
A gateway that authenticates with its own header rather than a bearer token takes it in
`headers`:
```hocon
internal-gateway {
provider = "openai-compatible"
baseUrl = "https://llm-gateway.internal.example/v1"
model = "gpt-4o-mini"
headers {
X-Team = "search"
X-Gateway-Token = ${?GATEWAY_TOKEN}
}
}
```
### Config policy
If you gate configs with `llm4s-config-policy`, its `dev` preset allows `openai-compatible` and
its `prod` preset does not: since the provider can point anywhere, a production config naming it
fails with `[allowedProviders] Provider 'openai-compatible' is not allowed` until you allow it.
Allow it explicitly, and pin the models and the endpoint:
```scala
import org.llm4s.configpolicy.{ CatalogEnvironment, ConfigPolicy }
val policy = ConfigPolicy.prodSafeDefaults
.withAllowedProviders("openai", "anthropic", "azure", "gemini", "deepseek", "openai-compatible")
.withAllowedModelPatterns(
"^openai/gpt-4o(-mini)?$", // ...and the other preset patterns you still need
"^openai-compatible/openai/gpt-oss-120b$"
)
.withRequiredBaseUrlPattern(
CatalogEnvironment.Prod,
"openai-compatible",
"https://api\\.groq\\.com/openai/v1"
)
.withMaxContextWindow(CatalogEnvironment.Prod, "openai-compatible", 131072)
```
- `withAllowedProviders` and `withAllowedModelPatterns` **replace** the preset's lists; repeat
the entries you want to keep.
- The `prod` preset also requires every section whose provider needs a key to set its own
`apiKey` (rule `ownApiKey`, `withOwnApiKeyRequired`). The generic provider's `apiKey` is
optional, so it is not checked; the rule is about sections that would otherwise inherit a
vendor's shared key.
- Model patterns match `/`, so a Groq model is
`openai-compatible/openai/gpt-oss-120b`. Model and base-URL patterns are regular expressions
that must match the **whole** value, so `^` and `$` are optional: `openai/gpt-4o` allows
neither `openai/gpt-4o-mini` nor a dated snapshot such as `openai/gpt-4o-2024-08-06` (write
`openai/gpt-4o(-.*)?` for those). End a base-URL pin that allows paths with `/.*`, never a
bare `.*`: `https://api\.groq\.com.*` also accepts `https://api.groq.com.evil.example/v1`.
- `withRequiredBaseUrlPattern(env, provider, pattern)` pins one provider's endpoint and
**replaces** the environment-wide pin for that provider. The environment-wide form,
`withRequiredBaseUrlPattern(env, pattern)`, is checked against every provider without a pin of
its own, not only `openai-compatible`; if you use it, name every endpoint in it (for example
`https://(api\.openai\.com|api\.groq\.com/openai)/v1`).
- Context caps work the same way. The `prod` preset caps each of its providers at its current
models' window (openai and azure 128000, anthropic 200000, gemini 1048576, deepseek 131072) and
any other provider, `openai-compatible` included, at an environment-wide 1048576; `dev` caps
everything at 1048576. So Groq, Together or Fireworks at 131072 and xAI at 500000 pass as they
are. Set a per-provider cap with `withMaxContextWindow(env, provider, max)`, as above, to hold a
provider tighter, or to allow a model above 1048576. A section over its cap fails with
`contextWindow ... exceeds `. A window taken from the model registry counts too: a Groq
section that sets no `contextWindow` resolves to 131072 and is checked at that.
- Provider names in a per-provider pin or cap are canonicalised like provider ids (`google` is
`gemini`). One that names no registered provider and is not in `allowedProviders` is an
`[unknownProvider]` violation, so a typo cannot leave a provider unpinned.
See `modules/config-policy/README.md` for the rest of the module.
### Troubleshooting OpenAI-compatible endpoints
**404 on every request - wrong `baseUrl`.** The client appends `/chat/completions` to `baseUrl`
and nothing else, so `baseUrl` must stop exactly where the vendor's path to
`/chat/completions` begins. Missing `/v1` (`http://localhost:8000` for vLLM) and a doubled one
(`.../v1/v1/chat/completions`, from a `baseUrl` of `.../v1/chat/completions` or with `/v1` added
twice) both return 404. Vendor paths differ: `/openai/v1` for Groq, `/inference/v1` for
Fireworks, no `/v1` at all for Perplexity Sonar. Model listing (`/models`) fails the
same way, which makes it a quick check.
**401 or 403 - authentication.** Surfaced as an `AuthenticationError`. Usually the key is not
reaching the request: `apiKey = ${?GROQ_API_KEY}` with the variable unset leaves `apiKey` out,
and the request goes without an `Authorization` header. Check the variable is exported in the
process that runs your app. Otherwise the key belongs to another vendor or project, or - for
NVIDIA - is the `NGC_API_KEY` used for pulling containers. For a local server started with
`--api-key`, set the same key as `apiKey`.
**Model not found.** Vendors answer an unknown model with a 404 or a 400. Model ids must match
exactly, prefix included (`openai/gpt-oss-120b` on Groq, `accounts/fireworks/models/...` on
Fireworks), and hosted catalogues retire models regularly. List what the endpoint serves, with
the section's key and headers:
```scala
import org.llm4s.config.Llm4sConfig
Llm4sConfig.listModels("groq-main") // GET /models
```
**No API key for a local server.** Leave `apiKey` out - do not set a placeholder. OpenAI's SDKs
require some key, which is why vendor examples show `api_key="ollama"` or `"not-used"`; llm4s
does not, and without one it sends no `Authorization` header at all. A 401 from a local server
means it was started with a key (`--api-key` for vLLM and `llama-server`, "Require
Authentication" in LM Studio), and `apiKey` must then match it.
**Missing `baseUrl`.** A section without one fails to load with
`baseUrl: set it in application.conf under llm4s.providers..baseUrl (e.g. http://localhost:8000/v1; to
read it from OPENAI_COMPATIBLE_BASE_URL, add baseUrl = ${?OPENAI_COMPATIBLE_BASE_URL} to the section)`.
`Llm4sConfig` reads no variable for it by itself: set `baseUrl` in the section, from a variable
if you like - `baseUrl = ${?OPENAI_COMPATIBLE_BASE_URL}` binds the conventional one. If the
section binds a variable and still fails this way, the variable is not set in the process that
runs your app.
**Streaming fails with a 400 or 422 naming `stream_options`, while `complete` works.** The
endpoint rejects fields it does not know. Set `streamUsage = false` in the section (see
[streamed token usage](#what-the-generic-path-does-and-does-not-do)); the stream then carries
usage only if the server sends it unasked.
**Replies cut short, or context errors.** Without a `contextWindow`, the generic provider uses the model
registry's window when it knows the model (see
[where the context window comes from](#where-the-context-window-comes-from)), and otherwise 8192. Set it to the
model's real window, or the server's configured context for a local server.
### Provider docs
The sources for the values above:
- Groq: [OpenAI compatibility](https://console.groq.com/docs/openai),
[models](https://console.groq.com/docs/models),
[deprecations](https://console.groq.com/docs/deprecations),
[reasoning](https://console.groq.com/docs/reasoning)
- Together AI: [OpenAI compatibility](https://docs.together.ai/docs/openai-api-compatibility),
[serverless models](https://docs.together.ai/docs/serverless-models)
- Fireworks AI: [OpenAI compatibility](https://docs.fireworks.ai/tools-sdks/openai-compatibility),
[quickstart](https://docs.fireworks.ai/getting-started/quickstart),
[gpt-oss-120b](https://fireworks.ai/models/fireworks/gpt-oss-120b)
- xAI: [quickstart](https://docs.x.ai/developers/quickstart),
[Chat Completions (legacy)](https://docs.x.ai/developers/model-capabilities/legacy/chat-completions),
[models](https://docs.x.ai/developers/models),
[reasoning](https://docs.x.ai/developers/model-capabilities/text/reasoning)
- NVIDIA NIM: [hosted LLM APIs](https://docs.api.nvidia.com/nim/reference/llm-apis),
[Llama 3.3 70B Instruct](https://docs.api.nvidia.com/nim/reference/meta-llama-3_3-70b-instruct),
[self-hosted quickstart](https://docs.nvidia.com/nim/large-language-models/latest/get-started/quickstart.html),
[self-hosted API reference](https://docs.nvidia.com/nim/large-language-models/latest/reference/api-reference.html)
- Perplexity: [Sonar OpenAI compatibility](https://docs.perplexity.ai/docs/sonar/openai-compatibility),
[Sonar Pro](https://docs.perplexity.ai/docs/sonar/models/sonar-pro),
[migrating from Sonar](https://docs.perplexity.ai/docs/agent-api/migrate-from-sonar/overview),
[Router API](https://docs.perplexity.ai/docs/router/quickstart),
[Router models](https://docs.perplexity.ai/docs/router/models)
- OrcaRouter: [quickstart](https://docs.orcarouter.ai/getting-started/quickstart),
[reasoning](https://docs.orcarouter.ai/advanced/reasoning)
- vLLM: [OpenAI-compatible server](https://docs.vllm.ai/en/latest/serving/online_serving/openai_compatible_server/),
[`vllm serve`](https://docs.vllm.ai/en/latest/cli/serve/),
[tool calling](https://docs.vllm.ai/en/latest/features/tool_calling/)
- LM Studio: [OpenAI compatibility](https://lmstudio.ai/docs/developer/openai-compat),
[authentication](https://lmstudio.ai/docs/developer/core/authentication)
- llama.cpp: [`llama-server` README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
- Ollama: [OpenAI compatibility](https://docs.ollama.com/api/openai-compatibility)
---
## Mistral
From the release after `0.4.1`, Mistral ships in `llm4s-openai-compatible`: Mistral's API is
the OpenAI `/v1/chat/completions` format, so it runs on the shared client with a small dialect.
```scala
libraryDependencies += "org.llm4s" %% "llm4s-openai-compatible" % llm4sVersion
```
In `0.4.1` and earlier it is part of `llm4s-core`, and does not stream
([#925](https://github.com/llm4s/llm4s/issues/925)).
### Setup
1. **Get API key** from [console.mistral.ai](https://console.mistral.ai/api-keys)
2. **Export the key** - `llm4s-openai-compatible` binds it to `llm4s.credentials.mistral.apiKey`:
```bash
export MISTRAL_API_KEY=your-key
```
### Configuration
```hocon
llm4s {
providers {
provider = "mistral-main"
mistral-main {
provider = "mistral"
model = "mistral-large-latest"
# apiKey defaults to MISTRAL_API_KEY
# baseUrl defaults to https://api.mistral.ai - the API root; requests go to
# /v1/chat/completions, and a baseUrl already ending in /v1 is used as it is
}
}
}
```
### What it supports
Chat, **streaming** (text, tool calls and token usage), **tool calling**, structured output
(`ResponseFormat.Json` and `ResponseFormat.JsonSchema`), model listing and exchange logging.
- **Tool-call ids.** Mistral accepts only nine-letter-or-digit ids. Its own pass through; any
other id in the conversation (from another provider, say) is mapped to a stable nine-character
one, the same for the call and its answer.
- **Reasoning models** (Magistral) return their reasoning as `Completion.thinking`, and stream it
as thinking deltas.
- `CompletionOptions.reasoning` is not sent: only some Mistral models accept `reasoning_effort`.
---
## Cohere
From the release after `0.4.1`, Cohere ships in `llm4s-openai-compatible` and calls Cohere's
[OpenAI-compatibility API](https://docs.cohere.com/docs/compatibility-api)
(`https://api.cohere.ai/compatibility/v1`) rather than its native v2 `/v2/chat`:
```scala
libraryDependencies += "org.llm4s" %% "llm4s-openai-compatible" % llm4sVersion
```
In `0.4.1` and earlier it is part of `llm4s-core`, calls `/v2/chat`, sends text only and does
not stream ([#925](https://github.com/llm4s/llm4s/issues/925)).
### Setup
1. **Get API key** from [dashboard.cohere.com](https://dashboard.cohere.com/api-keys)
2. **Export the key** - bound to `llm4s.credentials.cohere.apiKey`, which the Cohere reranker in
`llm4s-rag` shares, so one variable serves both. (Cohere's Python SDK prefers `CO_API_KEY`;
llm4s reads `COHERE_API_KEY` only, the name it has always used.)
```bash
export COHERE_API_KEY=your-key
```
### Configuration
```hocon
llm4s {
providers {
provider = "cohere-main"
cohere-main {
provider = "cohere"
model = "command-a-03-2025"
# apiKey defaults to COHERE_API_KEY
# baseUrl defaults to https://api.cohere.ai/compatibility/v1
}
}
}
```
A `baseUrl` naming Cohere's native API root, as configs written for `0.4.1` do
(`https://api.cohere.com`, with or without `/v1` or `/v2`), is mapped to
`/compatibility/v1` and logged; a proxy root is treated the same way. A `baseUrl` already
ending in `/compatibility/v1` is used as it is.
### What it supports
Chat, **streaming** (text, tool calls and token usage), **tool calling**, structured output and
exchange logging. System messages are sent under the `developer` role, and a JSON schema as
`{"type": "json_object", "schema": ...}`, as Cohere's compatibility API documents.
`CompletionOptions.reasoning` is not sent. Cohere's reranker (`llm4s-rag`) is separate and
unchanged.
### Available Models
`command-a-03-2025` and the other Command models your key can use; see
[Cohere's model list](https://docs.cohere.com/docs/models).
---
## xAI (Grok)
xAI's Grok models, through xAI's OpenAI-compatible Chat Completions API
(`https://api.x.ai/v1/chat/completions`). The provider ships in `llm4s-openai-compatible` as a
dialect on the shared client ([#1023](https://github.com/llm4s/llm4s/issues/1023)), and is Beta:
```scala
libraryDependencies += "org.llm4s" %% "llm4s-openai-compatible" % llm4sVersion
```
### Setup
1. **Get API key** from [console.x.ai](https://console.x.ai)
2. **Export the key** - `llm4s-openai-compatible` binds it to `llm4s.credentials.xai.apiKey`:
```bash
export XAI_API_KEY=xai-...
```
### Configuration
```hocon
llm4s {
providers {
provider = "xai-main"
xai-main {
provider = "xai"
model = "grok-4.7"
# apiKey defaults to XAI_API_KEY
# baseUrl defaults to https://api.x.ai/v1
}
}
}
```
The context window comes from the model registry when it lists the model, and otherwise from
xAI's published windows: 500k for `grok-4.5` to `grok-4.7`, 1M for `grok-4.3` and `grok-4.20`,
256k for the other Grok 4 models and 131072 for older ones.
### What it supports
Chat, **streaming** (text, tool calls and token usage), **tool calling**, images in user messages,
structured output, model listing (`/v1/models`) and exchange logging.
- **Reasoning output.** A reply's `reasoning_content`, where the model returns it (Grok 3 mini
does), is `Completion.thinking`, and streams as thinking deltas. Grok 4.5 and later keep their
reasoning encrypted and return it only on xAI's Responses API, so on Chat Completions they
return none. Earlier turns' thinking is not sent back: Chat Completions has no field for it.
- **Reasoning tokens** are `TokenUsage.thinkingTokens`, from
`usage.completion_tokens_details.reasoning_tokens`. xAI counts them outside `completion_tokens`
and inside `total_tokens`.
- **Reasoning effort.** `CompletionOptions.reasoning` is sent as `reasoning_effort` where the model
takes it. xAI's reasoning models cannot stop reasoning, so `None` sends the lowest level, and the
first such request in a process logs a warning:
| Model | `ReasoningEffort.None` | `Low` | `Medium` | `High` |
|---|---|---|---|---|
| `grok-4.5`, `grok-4.6`, `grok-4.7` (and their `-latest` aliases) | `low` | `low` | `medium` | `high` |
| `grok-3-mini` (and variants) | `low` | `low` | `high` | `high` |
| any other model | not sent | not sent | not sent | not sent |
`high` is the default on Grok 4.5 and later; `xhigh` (4.6 and later) is not reachable from
`CompletionOptions`. `grok-4` rejects the parameter, and on `grok-4.20-multi-agent` it sets the
number of agents, so neither is sent it. With `reasoning` unset nothing is sent.
- **Penalties.** xAI rejects `presence_penalty`, `frequency_penalty` and `stop` on reasoning models.
llm4s never sends `stop`; a non-zero `presencePenalty` or `frequencyPenalty` is left out of a
request to a reasoning model (logged once), and sent as given to a `-non-reasoning` model.
- xAI calls Chat Completions its **legacy** endpoint; new features ship on its Responses API, which
this provider does not speak.
### Available Models
`grok-4.7` and the other models your key can use; see [xAI's model list](https://docs.x.ai/developers/models).
---
## Ollama (Local Models)
From the release after `0.4.1`, Ollama support - chat and embeddings - ships in its own
artifact, alongside `llm4s-core`:
```scala
libraryDependencies += "org.llm4s" %% "llm4s-ollama" % llm4sVersion
```
In `0.4.1` and earlier it is part of `llm4s-core`. See the
[installation guide](../getting-started/installation#for-ollama-local-models).
### Setup
1. **Install Ollama** from [ollama.ai](https://ollama.ai)
2. **Pull a model:**
```bash
ollama pull mistral # Downloads model
ollama serve # Runs on http://localhost:11434
```
3. **Add a named section** to `application.conf` and select it as the default. No API key is
needed, and `baseUrl` defaults to `http://localhost:11434`, where `ollama serve` listens.
### Configuration
In `application.conf`:
```hocon
llm4s {
providers {
provider = "ollama-local"
ollama-local {
provider = "ollama"
model = "mistral"
# baseUrl = "http://gpu-box:11434" # optional: the default is http://localhost:11434
# baseUrl = ${?OLLAMA_BASE_URL} # optional: let OLLAMA_BASE_URL override the default
}
}
}
```
### Structured output
`llm4s-ollama` honours `CompletionOptions.responseFormat` (and so `completeStructured`) through
the `format` field of `/api/chat`, for streaming and non-streaming calls alike:
`ResponseFormat.Json` sends `"format": "json"`, and `ResponseFormat.JsonSchema` sends the schema
object, which needs Ollama 0.5 or later. `JsonSchema.name` and `strict` have no Ollama equivalent
and are ignored. With no `responseFormat`, no `format` field is sent.
### Available Models
100+ models available:
- **Small:** `phi`, `neural-chat` (~4GB)
- **Medium:** `mistral`, `llama2` (~13GB)
- **Large:** `llama2-70b` (~40GB)
- **Specialized:** `neural-chat`, `orca`, `wizard-math`
Run `ollama list` to see installed models.
### Costs
Free! Just compute (CPU or GPU needed).
### Tips
- Perfect for development and testing
- Works offline (no internet needed)
- Use GPU for faster inference
- Ideal for sensitive data (runs locally)
---
## Thinking in conversation history
A model's reasoning is returned on the message it produced, as `AssistantMessage.thinking` - a
sequence of `ThinkingBlock`s - and `Completion.thinking` is that message's thinking text. Because it
lives on the message, it stays in the conversation (and in an agent's thread) like the content and
tool calls do, and each client sends it back where its provider takes it:
| Provider | Read from | Sent back as |
|---|---|---|
| Anthropic | `thinking` blocks (each with its `signature`) and `redacted_thinking` | the same blocks, unchanged and first in the assistant turn |
| Bedrock (Converse) | `reasoningContent` (text with `signature`, or `redactedContent`) | the same blocks, unchanged and first in the assistant turn |
| Ollama | `message.thinking` | `thinking` on the assistant message |
| DeepSeek, Z.ai | `reasoning_content` | `reasoning_content` |
| OpenRouter | `reasoning` / `thinking`, and `reasoning_details` | `reasoning`, and `reasoning_details` unchanged (same items, order and fields) |
| Mistral (Magistral) | thinking chunks in `content` | a thinking chunk before the text chunk |
| Gemini API, Vertex AI | a `thoughtSignature` on a part: the `functionCall` part, or the last text part (and thought summaries, `thought: true`, as text) | the signature, on the same part ([below](#gemini-and-vertex-ai-thought-signatures)) |
| OpenAI, Azure, Cohere, generic `openai-compatible` | - | not sent: the API has no field for it |
| IBM watsonx.ai | `reasoning_content` (or `reasoning`) | not sent: the chat API's assistant message has no field for it |
Anthropic and Bedrock require the signed blocks back when a thinking model's turn ends in tool
calls, and reject a thinking block without a signature, so unsigned thinking - from another
provider earlier in the conversation, or set by hand with `withThinking(text)` - is left out of
their requests. The Anthropic client sends tool calls and their results as `tool_use` and
`tool_result` blocks for the same reason: the thinking has to sit in the turn that made the calls.
Both clients send a call only when its result is in the run of tool messages straight after it, as
those APIs require; a result anywhere else (after a user message, say) goes as
`[Tool result for ]: ...` text, and its call is left out.
OpenRouter returns `reasoning_details` when the underlying model's reasoning is signed, summarised
or encrypted (Claude, Gemini, OpenAI reasoning models), and
[requires the whole sequence back unchanged](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens#preserving-reasoning)
when a conversation continues after tool calls. Each item is kept, with every field, as a
`ThinkingBlock.Opaque("openrouter", json)` block after the reasoning text - streamed items are
joined by `index` first - and only the OpenRouter client sends them back; every other client ignores
opaque blocks that are not its own.
### Gemini and Vertex AI thought signatures
A thinking Gemini model attaches an opaque, base64 `thoughtSignature` to a part of its turn and expects it
back on the same part. Google's
[Vertex AI guide](https://cloud.google.com/vertex-ai/generative-ai/docs/thought-signatures) states the rules
these clients follow:
- A response with `functionCall` parts needs its signature back: Gemini 3 models answer HTTP 400 when a
required signature is missing. With parallel calls only the first `functionCall` part carries one; across
sequential steps each step's first call does. The part goes back "exactly as it was returned".
- A response without function calls may carry a signature on its last part (streaming can deliver it on a part
with empty text). Sending it back is recommended, and leaving it out is not an error.
- A part with a signature is never merged with one without.
Each signature is kept as a sealed `ThinkingBlock.Opaque` block of the client's provider id (`gemini` or
`vertexai`) and goes back only to the provider and model that produced it. Unlike Anthropic's prefix rule,
the binding is to that origin alone: Google asks for signatures to be preserved when history is modified or
trimmed, and Gemini 3 answers HTTP 400 when the current turn's function-call signature is missing, so
pruning or compressing earlier turns leaves them in place. What unseals one is a change of provider or
model, or an edit to the message that carries it. A function call's signature is stored against the tool
call's id with the `functionCall` payload exactly as Gemini returned it, and the part is replayed verbatim
(its optional `id` kept, `args` present or absent as received); a populated `functionCall.id` becomes the
tool call's own id and the matching `functionResponse` echoes it. A text part's signature records its
character offset in the message content, and the content is split there when the turn is sent, so a signed
part is never merged with an unsigned one (if the content no longer fits, the text signature is left out).
Parts are rebuilt as text first, then function calls, as these clients have always sent them.
Google's documentation does not say whether a signature from the Gemini API validates on Vertex AI, so the two
clients are separate signing authorities: a conversation moved from one to the other keeps its text and
drops the signatures, never sending a foreign one. A signature on an image part, or on a thought-summary
part, is not kept, since these clients neither request nor send those parts.
Signed, redacted and opaque thinking is *sealed*: it is valid only in the conversation it was produced in.
[Anthropic](https://platform.claude.com/docs/en/build-with-claude/preserved-thinking) validates a
thinking block against everything sent before it - the top-level system prompt, the tools and every
earlier message - and rejects it (400) once any of those changes; a block must also come back
unchanged, beside the content and tool calls it came with. Bedrock documents its reasoning signature
as a hash of all the messages in the conversation. llm4s enforces both halves with two rules
(`hasSealedThinking` reports which state a message is in):
- **The message itself.** `withContent` and `withToolCalls`, given a changed value, *unseal* the
thinking: they drop redacted and opaque blocks and signatures and keep the reasoning text. An agent's
`afterAgent` answer rewrite and a tool-call edit go through those setters, and so do the
Anthropic and Bedrock clients when they leave out an unpaired call.
- **Everything before it.** When Anthropic, Bedrock or OpenRouter returns sealed thinking, the client records a
fingerprint of the request on the message (`AssistantMessage.thinkingBinding`): every system
message, the tools, the response format and every earlier message in order, system messages in
their positions (OpenRouter sends them inline, so moving one changes what came before), as sent.
When the message is
sent again, the client replays its sealed thinking only if the conversation before it still has
that fingerprint, and sends it unsealed otherwise. The check runs at the point of sending, so it
covers every rewrite of the history - `ContextPruning` and the agent's context-window middleware,
`TokenWindow` trimming, the context compressors (`DeterministicCompressor`, `LLMCompressor`,
`HistoryCompressor`, `ToolOutputCompressor`, `ContextManager`), a handoff's view of the thread,
memory or RAG context inserted before existing turns, a changed system prompt or tool set, or a
hand edit - without any of them having to know about thinking.
- **Who produced it.** The fingerprint also covers the provider id and model that produced the
thinking, and the client checks it against the provider and model it is about to call. A
signature or reasoning item belongs to its producer, so a conversation produced by one client
and continued with another - Bedrock then Anthropic, or one OpenRouter model then another - is
sent unsealed, never with a foreign signature. Every client binds to the provider and model it is
configured with, never the model a response reports: Anthropic and Bedrock may report an alias's
resolved snapshot, and a router such as `openrouter/auto` (or OpenRouter's model fallbacks)
reports the model it chose for that request. The provider that resolved the alias or chose the
route is the one the thinking goes back to, and OpenRouter
[requires](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens#preserving-reasoning-blocks)
the complete `reasoning_details` sequence on a tool-call continuation, so routed turns keep it.
Changing the configured model unseals every earlier turn. The endpoint is not part of it: a proxy or regional endpoint in
front of the same provider changes nothing the provider checks.
### Moving between Anthropic and Bedrock
A conversation that moves between the Anthropic API and Bedrock (a failover, say) has its signed thinking
unsealed: the text is kept, the signatures and redacted blocks are dropped, and nothing is rejected. Anthropic
[documents](https://platform.claude.com/docs/en/build-with-claude/thinking#thinking-encryption) that signature
values are compatible across platforms (the Claude API, Amazon Bedrock and Google Cloud), but that is only one
of the conditions for a replayed block to be accepted. The API also checks the model that produced it
(each model reads its own thinking blocks and those of a fixed set of other models) and that everything sent
before it is unchanged, and rejects the request with a 400 or drops the block when either fails. The two
clients name the same model differently (`claude-sonnet-4-5-20250929` against
`anthropic.claude-sonnet-4-5-20250929-v1:0`) and serialise the system prompt and tools through different
APIs, so this library cannot show that the prefix a block was signed over is the prefix the other platform
sees. Losing the reasoning context on a failover is safe; a rejected request is not, so each remains its own
signing authority. Treating them as one would need a mapping between the two model ids and a live check that
a signature survives the move.
### Limits that remain
- **OpenRouter routers and fallbacks.** OpenRouter's documentation says dynamic routers (`openrouter/auto`,
`openrouter/free`) omit the reasoning field, so there is nothing to replay for them, and that the
`reasoning_details` sequence must match the original output with nothing rearranged or modified. It does
not say what happens to the details when a fallback sends the request to a different model, so they are
sent back unchanged, as its documentation asks.
- **Editing a tool call on a turn that is still in progress** (a human-in-the-loop edit) unseals that turn's
thinking, because the signature covers the tool call. Anthropic's manual extended thinking requires the
final assistant turn of a thinking-enabled request to begin with a thinking block, so such a request can be
rejected; adaptive thinking drops that requirement. Gemini 3 answers 400 for a missing function-call
signature in the current turn.
Earlier turns whose history is unchanged keep their sealed thinking: Anthropic recommends passing
all thinking blocks back, keeps them in context on newer models, and accepts any unbroken run of the
original blocks. A change unseals every turn after it and none before it, so the replayed blocks never
have a gap. Unsealed thinking keeps its text, which Anthropic and Bedrock leave out and other
providers still receive. Sealed thinking with no binding - built by hand with `withThinking` - is
not replayed. To keep signed thinking, append to the conversation and leave what is already there as
it is. Thinking also counts toward token estimates (`ConversationTokenCounter`), since most providers
resend it.
```scala
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model.{ Completion, CompletionOptions, Conversation, ToolCall, ToolMessage }
import org.llm4s.types.Result
def toolTurn(
client: LLMClient,
conversation: Conversation,
options: CompletionOptions,
runTools: Seq[ToolCall] => Seq[ToolMessage] // your tool execution
): Result[Completion] =
for {
first <- client.complete(conversation, options) // a tool-call turn, with thinking
results = runTools(first.message.toolCalls)
next = conversation.addMessage(first.message).addMessages(results) // keeps the thinking
answer <- client.complete(next, options) // the provider gets it back
} yield answer
```
---
## Citations and grounding sources
A model that searches the web by itself reports the sources behind its answer. llm4s returns them on
`Completion.citations`, a `List[Citation]` that is empty for every completion that has none:
```scala
import org.llm4s.llmconnect.model.Completion
def sources(completion: Completion): List[String] =
completion.citations.map(c => s"${c.title.getOrElse(c.url)} <${c.url}>")
```
A `Citation` has a `url`, always, and these fields, each `None` when the provider did not send it:
| Field | Meaning |
|---|---|
| `title` | the source's title |
| `citedText` | a passage of the source that came with the citation (OpenRouter's `content`); it describes the source, not the answer |
| `startIndex`, `endIndex` | where the inline citation sits, as the provider reports it: positions in `Completion.content`. OpenAI documents them as the first and last character "of the URL citation in the message"; no provider documents them as the span of prose the source supports |
Take the two indices as the location of the citation, not of the supported claim, and check the
provider's convention before cutting `content` with them (whether `end_index` is inclusive is the
provider's call). `hasSpan` says whether both are present.
| Provider | Citations |
|---|---|
| OpenAI (search models for Chat Completions), Azure, Requesty | Read from the message's `url_citation` annotations (`url`, `title`, `start_index`, `end_index`). Search models (`gpt-5-search-api`; the `*-search-preview` models were retired on 2026-07-23) return them with no request option; Azure and Requesty return them only if the deployment does. |
| OpenRouter (`:online` models, or the web plugin) | The same annotations, plus `content` as `citedText`. |
| DeepSeek, Z.ai, Mistral, Cohere, a generic `openai-compatible` endpoint | Read if the reply carries standard `url_citation` annotations; none of them is documented to. |
| Gemini, Vertex AI | With `googleSearch = true` in the section (see [Google Search grounding](#google-search-grounding)): one citation per source in the candidate's `groundingMetadata.groundingChunks` - `url` and `title`, and for a Vertex AI Search / RAG source (`retrievedContext`) its text as `citedText`. No indices: Gemini's `groundingSupports` are UTF-8 byte offsets of the supported text, not where a citation sits. Without the setting, empty. Read from `complete` and from a stream. |
| Anthropic | With `webSearch = true` in the section (see [Web search](#web-search)): one citation per source a text block cites - for a `web_search_result_location` its `url`, `title` and `cited_text`, for a `search_result_location` its `source` as `url`. Document citations (`char_location`, `page_location`, `content_block_location`) name a document, not a URL, and are skipped. No indices: Anthropic attaches a citation to a text block, not to a position in it. Read from `complete` and from a stream (`citations_delta`). |
| Ollama, Bedrock, watsonx | Empty. |
Things to know:
- **Streaming.** For OpenAI and OpenRouter, a completion assembled from streamed chunks has no citations:
neither documents where a stream carries them, so streamed chunks are not read. Gemini and Vertex AI send
`groundingMetadata` in a streamed chunk (usually the last), and the streamed completion carries it. Anthropic
streams each citation as a `citations_delta`, and the streamed completion carries them. (A model that does not
stream natively is answered with one ordinary call, and that completion keeps its citations.)
- **A citation never fails a reply.** One without a `url` is dropped, a field of the wrong type is read as
absent, and the answer is returned either way. The OpenAI client treats an `annotations` list the SDK
cannot parse at all as no citations; the OpenAI-compatible client keeps the entries it can read.
- **Not read:** Perplexity's top-level `citations` list (its provider is tracked in #1026).
- The shapes come from the OpenAI Java SDK's `url_citation` types and OpenRouter's documentation. No live
provider was called while writing this.
---
## Enterprise authentication
Many organisations forbid long-lived API keys for cloud models and require the platform's own
identity instead: a managed identity, a workload identity or an IAM role. Each of the three clouds
works that way in llm4s, with no key in `application.conf`.
| Cloud | Provider | How it authenticates without a key |
|---|---|---|
| Microsoft Azure | `azure` (`llm4s-openai` + `llm4s-azure-identity`) | `authentication = "entra-id"`: Microsoft Entra ID tokens from `DefaultAzureCredential` |
| AWS | `bedrock` (`llm4s-bedrock`) | The AWS default credentials chain: environment, `~/.aws` profile, an instance, task or IRSA role |
| Google Cloud | `vertexai` (`llm4s-gemini`) | Application Default Credentials: `GOOGLE_APPLICATION_CREDENTIALS`, gcloud's login, the metadata server |
### Azure OpenAI with Microsoft Entra ID
Azure Policy can disable key ("local") authentication on an Azure OpenAI resource. Such a
resource takes Microsoft Entra ID tokens instead, and so does llm4s
([#2274](https://github.com/llm4s/llm4s/issues/2274)). Add `llm4s-azure-identity` beside
`llm4s-openai`:
```scala
// build.sbt - same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-azure-identity" % llm4sVersion
```
and set `authentication = "entra-id"` in the section, with no `apiKey`:
```hocon
llm4s.providers {
provider = "azure-prod"
azure-prod {
provider = "azure"
endpoint = "https://my-resource.openai.azure.com"
model = "gpt-4o" # the deployment name
authentication = "entra-id"
# clientId = ${?AZURE_CLIENT_ID} # a user-assigned managed identity
}
}
```
The token comes from Azure's `DefaultAzureCredential`, which tries, in turn, environment
credentials (`AZURE_TENANT_ID`, `AZURE_CLIENT_ID` and a secret or certificate), workload identity
on AKS, a managed identity, and a developer's `az login`. It is requested for the
`https://cognitiveservices.azure.com/.default` scope, cached, and fetched again five minutes before
it expires, so no request carries an expired token; one the service rejects with a 401 is
dropped, and the next request fetches a new one. The identity needs the **Cognitive Services
OpenAI User** role on the resource.
Such a section takes no API key: setting `apiKey` as well is an error, and the shared
`AZURE_OPENAI_API_KEY` is not used. Without `llm4s-azure-identity` on the classpath, loading the
section fails with a configuration error naming it. `llm4s-azure-identity` brings Azure's
`azure-identity` (with `azure-core`, MSAL4J, Netty and Reactor), which is why it is a module of its
own: `llm4s-openai` alone puts no Azure SDK on your classpath.
In code, any `AccessTokenProvider` authenticates a config, and `AzureEntraId` makes one from Azure's
`DefaultAzureCredential` or from any other `TokenCredential`:
```scala
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.auth.azure.AzureEntraId
import org.llm4s.llmconnect.config.AzureConfig
import org.llm4s.model.ModelRegistryService
def azureClient(using ModelRegistryService) =
val tokens = AzureEntraId.defaultCredential() // or AzureEntraId.tokenProvider(anyTokenCredential)
LLMConnect.getClient(AzureConfig.keyless("https://my-resource.openai.azure.com", "gpt-4o", tokens))
```
Embeddings take the same tokens: `OpenAIEmbeddingProvider.fromConfig(config, tokens)`, with the
resource's v1 API as the base URL (`https://my-resource.openai.azure.com/openai/v1`) and the
embedding deployment's name as the model. From Java, `Llm4s.azureOpenAi().entraId()` (or
`.entraId(clientId)`, or `.tokenProvider(tokens)`) replaces `.apiKey(...)`. With the Spring Boot
starter, the properties are the section's keys: `llm4s.providers.azure-prod.authentication=entra-id`
and `llm4s.providers.azure-prod.client-id=...`.
### AWS Bedrock with IAM roles
A `bedrock` section that sets no `accessKeyId` uses the AWS SDK's default credentials chain: the
`AWS_*` environment variables, a `profile` in `~/.aws`, and the role of the EC2 instance, ECS task
or EKS service account (IRSA) the application runs as. See the
[installation guide](../getting-started/installation#for-aws-bedrock).
### Vertex AI with Application Default Credentials
A `vertexai` section with no `apiKey` uses Application Default Credentials:
`GOOGLE_APPLICATION_CREDENTIALS`, then gcloud's `application_default_credentials.json`, then the
metadata server of the GCE, GKE or Cloud Run workload. See [Google Vertex AI](#google-vertex-ai).
---
## API Key Management
### Security Best Practices
**Never commit API keys!**
1. **Keep keys in environment variables.** The vendor's own variable is enough - each provider
module binds it to the vendor's shared key:
```hocon
openai-main { provider = "openai", model = "gpt-4o" }
```
```bash
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
```
For a second account, or in production where each section should say which account it bills,
bind the section's own key: `apiKey = ${?OPENAI_BATCH_API_KEY}`. The config-policy `prod`
preset flags sections without one. llm4s logs at INFO which path each key came from, never
the value.
2. **Use .env file** (add to `.gitignore`):
```bash
# .env (NOT committed to git)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
```
3. **Use CI/CD secrets:**
```yaml
# GitHub Actions
- uses: actions/setup-java@v3
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
```
4. **Rotate keys regularly** on provider dashboards
### Using Keys Safely in Code
**Good:**
```scala
import org.llm4s.config.Llm4sConfig
// Keys come from the vendor's variable or the section's own ${?VAR} binding - never hardcoded
val providerConfig = Llm4sConfig.defaultProvider()
```
**Bad:**
```scala
// ❌ Never do this!
val key = "sk-proj-abc123..." // Hardcoded
sys.env.get("OPENAI_API_KEY") // Outside config boundary
```
---
## Base URL Customization
### When to Use Custom Base URLs
- **Reverse proxy** or load balancer
- **VPC endpoint** for security
- **Azure OpenAI** or self-hosted setup
- **Provider migration** (e.g., from OpenAI to similar API)
### Setting Custom URLs
A base URL is the `baseUrl` key of the named section (`endpoint` for Azure OpenAI). No
`_BASE_URL` variable is read by itself; bind one if you want the environment to
override the URL:
```hocon
llm4s {
providers {
provider = "openai-main"
openai-main {
provider = "openai"
model = "gpt-4o"
baseUrl = "https://proxy.example.com/openai"
baseUrl = ${?OPENAI_BASE_URL} # optional: overrides the line above when set
}
}
}
```
In HOCON the later line wins, and `${?VAR}` with the variable unset leaves the earlier value in
place. Defaults when `baseUrl` is omitted:
| Provider | Default `baseUrl` |
|----------|-------------------|
| `openai` | `https://api.openai.com/v1` |
| `anthropic` | `https://api.anthropic.com` |
| `gemini` | `https://generativelanguage.googleapis.com/v1beta` |
| `deepseek` | `https://api.deepseek.com` |
| `mistral` | `https://api.mistral.ai` |
| `cohere` | `https://api.cohere.ai/compatibility/v1` (a native root is mapped to `/compatibility/v1`) |
| `openrouter` | `https://openrouter.ai/api/v1` |
| `zai` | `https://api.z.ai/api/paas/v4` |
| `requesty` | `https://router.requesty.ai/v1` |
| `ollama` | `http://localhost:11434` |
| `openai-compatible` | none - required |
---
## Provider Comparison Table
| Feature | OpenAI | Anthropic | Gemini | Azure | DeepSeek | Cohere | Ollama | OpenAI-compatible |
|---------|--------|-----------|--------|-------|----------|--------|--------|-------------------|
| **Setup Difficulty** | Easy | Easy | Easy | Hard | Easy | Easy | Medium | Easy |
| **API Key Required** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | Hosted ✅, local ❌ |
| **Free Tier** | Limited | Limited | ✅ Generous | ❌ | Limited | Limited | ✅ | Endpoint-specific |
| **Local Option** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ vLLM, LM Studio, llama.cpp |
| **Context Window** | 128K | 200K | 1M | 128K | 4K-32K | 8K | Model-specific | Set in config (default 8K) |
| **Vision Support** | ✅ | ✅ | ✅ | ✅ | ⚠️ Limited | ❌ | Model-specific | ❌ Text only |
| **Function Calling** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ If the model supports it | ✅ If the endpoint supports it |
| **Reasoning Models** | ✅ o1 | ❌ | ❌ | ✅ (via OpenAI) | ✅ deepseek-reasoner | ❌ | ❌ | ⚠️ Run, but reasoning not configured or read |
| **Enterprise Support** | ✅ | ✅ | ✅ | ✅ | ⚠️ | ✅ | N/A | Endpoint-specific |
| **Cost (Budget)** | Medium | Medium | 🏆 Low | High | 🏆 Very Low | Low | Free | Endpoint-specific |
| **Speed** | Fast | Medium | 🏆 Very Fast | Medium | Fast | Medium | Varies | Endpoint-specific |
| **Reliability** | 🏆 Enterprise | 🏆 Enterprise | Good | 🏆 Enterprise | Good | Good | Local | Endpoint-specific |
### Which Provider Should I Use?
- **Getting started?** → Try **Gemini** (free tier) or **Ollama** (local)
- **Production API?** → **OpenAI** (most stable) or **Anthropic** (best reasoning)
- **Cost-conscious?** → **DeepSeek** or **Ollama**
- **Enterprise?** → **Azure OpenAI** or **Anthropic**
- **Private data?** → **Ollama** (runs locally)
- **xAI (Grok)?** → [**xAI**](#xai-grok)
- **Groq, Together, Fireworks, NVIDIA NIM, or a local vLLM, LM Studio or llama.cpp server?** → the generic [**OpenAI-compatible**](#openai-compatible-endpoints) provider
- **Reasoning tasks?** → **Anthropic Claude** or **DeepSeek reasoner**
- **Vision/multimodal?** → **OpenAI GPT-4o** or **Anthropic Claude**
---
## Multiple Providers in One App
Configure one named section per provider, then load the default or any section by name:
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.model.ModelRegistryService
// conversation: the Conversation to send
for {
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
primaryConfig <- Llm4sConfig.defaultProvider() // llm4s.providers.provider
fallbackConfig <- Llm4sConfig.provider("claude") // llm4s.providers.claude
primary <- LLMConnect.getClient(primaryConfig)
fallback <- LLMConnect.getClient(fallbackConfig)
} yield primary.complete(conversation).orElse(fallback.complete(conversation))
```
Each section is validated when it is loaded, so here both must have a key available;
a problem in either surfaces as the `Left` of this `for`. See
[Switching providers](../getting-started/configuration#switching-providers).
This enables:
- **Fallback logic** - Use OpenAI, fall back to Anthropic
- **A/B testing** - Compare provider outputs
- **Cost optimization** - Use cheapest available provider
---
## Troubleshooting
### "Invalid API Key"
```bash
# Verify the vendor's variable is set
echo $OPENAI_API_KEY
# ...and that the section sets no apiKey of its own, which would win over it.
# The INFO log line "llm4s.providers.: API key from " says which was used.
# Check key format (starts with correct prefix)
# OpenAI: sk-proj-* or sk-*
# Anthropic: sk-ant-*
# Gemini: Should be long alphanumeric
```
### "Connection refused"
For local providers (Ollama):
```bash
# Check if Ollama is running
curl http://localhost:11434/api/tags
# Start Ollama
ollama serve
```
### "Model not found"
Check the `model` key of the section you selected, and that its `provider` is the one serving
that model (`model = "gpt-4o"` under `provider = "openai"`; no `openai/` prefix):
```bash
# Check available models
# OpenAI: https://platform.openai.com/docs/models
# Anthropic: https://docs.anthropic.com/claude/reference/models
```
### "Rate limit exceeded"
Use provider-specific strategies:
- OpenAI: Wait before retrying, use batching API
- Gemini: Upgrade from free tier
- Ollama: Increase system resources or use GPU
---
Source: https://llm4s.org/guide/vector-store
# Vector Store
Low-level vector storage abstraction for RAG and semantic search.
---
## Overview
The `VectorStore` trait provides a backend-agnostic interface for storing and searching vector embeddings. This is the foundation layer for building RAG (Retrieval-Augmented Generation) applications.
**Key Features:**
- Backend-agnostic API supporting multiple vector databases
- Type-safe error handling with `Result[A]`
- Metadata filtering with composable DSL
- Batch operations for efficient bulk processing
- Built-in statistics and monitoring
**Current Backends:**
- **SQLite** - File-based or in-memory storage (default)
- **pgvector** - PostgreSQL with pgvector extension (production-ready)
- **Qdrant** - Cloud-native vector database via REST API
- **Elasticsearch** - `dense_vector` kNN search in a cluster you already run, in the separate module `llm4s-vectorstore-elasticsearch` (Experimental); see [Elasticsearch](#elasticsearch)
- **OpenSearch** - `knn_vector` search with the k-NN plugin, in the same module (Experimental); see [OpenSearch](#opensearch)
- **Redis and Valkey** - KNN search over a vector index in the Redis (Redis Stack, Redis 8+) or Valkey (`valkey-search`) you already run, in the separate module `llm4s-vectorstore-redis` (Experimental); see [Redis and Valkey](#redis-and-valkey)
**Keyword Backends (for hybrid search):**
- **SQLite FTS5** - File-based or in-memory BM25 search (default)
- **PostgreSQL** - Native full-text search with tsvector/tsquery (production-ready)
**Planned Backends:**
- Milvus
- Pinecone
---
## Quick Start
### In-Memory Store
```scala
import org.llm4s.vectorstore._
// Create an in-memory store
val store = VectorStoreFactory.inMemory().fold(
e => throw new RuntimeException(s"Failed: ${e.formatted}"),
identity
)
// Store a vector
val record = VectorRecord(
id = "doc-1",
embedding = Array(0.1f, 0.2f, 0.3f),
content = Some("Hello world"),
metadata = Map("type" -> "greeting")
)
store.upsert(record)
// Search for similar vectors
val results = store.search(
queryVector = Array(0.1f, 0.2f, 0.3f),
topK = 5
)
// A Result: Right(hits), or Left(error)
results.foreach(_.foreach { scored =>
println(s"${scored.record.id}: ${scored.score}")
})
// Clean up
store.close()
```
### File-Based Store
```scala
import org.llm4s.vectorstore._
// Create a persistent store
val store = VectorStoreFactory.sqlite("/path/to/vectors.db").fold(
e => throw new RuntimeException(s"Failed: ${e.formatted}"),
identity
)
// Use the store...
store.close()
```
### PostgreSQL with pgvector
```scala
import org.llm4s.vectorstore._
// Local PostgreSQL with defaults
val store = VectorStoreFactory.pgvector().fold(
e => throw new RuntimeException(s"Failed: ${e.formatted}"),
identity
)
// Or with explicit connection settings (a Result, like every factory method)
val store2 = VectorStoreFactory.pgvector(
connectionString = "jdbc:postgresql://localhost:5432/mydb",
user = "postgres",
password = "secret",
tableName = "embeddings"
)
// Create HNSW index for faster search (optional)
store.asInstanceOf[PgVectorStore].createHnswIndex()
store.close()
```
**Setup:** Requires PostgreSQL with pgvector extension:
```sql
CREATE EXTENSION IF NOT EXISTS vector;
```
### Sharing your application's connection pool
The factories above create a HikariCP pool and close it with the store. An application that already has a pool - a
doobie `HikariTransactor`, a Play or Slick pool, or a `javax.sql.DataSource` a platform team hands out with credential
rotation, metrics and TLS configured - can pass it instead. Every Postgres store and index takes a plain
`javax.sql.DataSource` that it **does not own**: closing the store leaves the pool open, and the pool's owner closes it.
| Store | Over a caller-owned `DataSource` |
| ----- | -------------------------------- |
| `PgVectorStore` | `PgVectorStore.create(dataSource, tableName)` or `PgVectorStore(dataSource, tableName)` |
| `PgKeywordIndex` | `PgKeywordIndex(dataSource, tableName)` or `PgKeywordIndex(dataSource, tableName, language)` |
| `HybridSearcher` | `HybridSearcher.pgvectorShared(dataSource, vectorTableName, keywordTableName)` (optionally a `FusionStrategy`) |
| `PgSearchIndex` | `PgSearchIndex.fromDataSource(dataSource, vectorTableName)` |
| `PostgresMemoryStore` | `PostgresMemoryStore.fromDataSource(dataSource, tableName, embeddingService)` |
With doobie, the transactor's `kernel` is the `HikariDataSource` behind it. Build the store inside the transactor's
`Resource`, so the pool outlives every store that borrows it:
```scala
import cats.effect.{ IO, Resource }
import doobie.hikari.HikariTransactor
import doobie.util.ExecutionContexts
import org.llm4s.vectorstore.{ PgVectorStore, VectorRecord }
// The application's pool. Its Resource closes it - no LLM4S store does.
val transactor: Resource[IO, HikariTransactor[IO]] =
for {
connectEc <- ExecutionContexts.fixedThreadPool[IO](16)
xa <- HikariTransactor.newHikariTransactor[IO](
"org.postgresql.Driver",
"jdbc:postgresql://localhost:5432/mydb",
"postgres",
"secret",
connectEc
)
} yield xa
// The store borrows `xa.kernel`; closing the store leaves the pool open.
val vectorStore: Resource[IO, PgVectorStore] =
transactor.flatMap { xa =>
Resource.make(
IO.blocking(PgVectorStore.create(xa.kernel, "embeddings"))
.flatMap(result => IO.fromEither(result.left.map(e => new RuntimeException(e.formatted))))
)(store => IO.blocking(store.close()))
}
// Store calls are blocking JDBC: run them on the blocking pool.
val program: IO[Long] =
vectorStore.use { store =>
IO.blocking {
for {
_ <- store.upsert(VectorRecord("doc-1", Array(0.1f, 0.2f, 0.3f), Some("Scala programming")))
count <- store.count()
} yield count
}.flatMap(result => IO.fromEither(result.left.map(e => new RuntimeException(e.formatted))))
}
```
Every store call is blocking JDBC. From cats-effect code run it with `IO.blocking` (or `Sync[F].blocking`), from ZIO
with `ZIO.attemptBlocking`, so it never occupies a compute thread. Any `DataSource` works, not only HikariCP's: pgjdbc's
`PGSimpleDataSource` opens a connection per call, which is fine for tests and scripts. Hikari stays an implementation
detail of the config-based factories, and no public signature names it.
### Qdrant
```scala
import org.llm4s.vectorstore._
// Local Qdrant (docker run -p 6333:6333 qdrant/qdrant)
val store = VectorStoreFactory.qdrant().fold(
e => throw new RuntimeException(s"Failed: ${e.formatted}"),
identity
)
// Or Qdrant Cloud
val cloudStore = VectorStoreFactory.qdrantCloud(
cloudUrl = "https://your-cluster.qdrant.io",
apiKey = "your-api-key",
collectionName = "my_vectors"
)
store.close()
```
### Elasticsearch
`ElasticsearchVectorStore` keeps the vectors in one index of an Elasticsearch cluster, through the REST API and
`llm4s-core`'s HTTP client: no Elasticsearch client library is added. It lives in its own module (Experimental):
```scala
libraryDependencies += "org.llm4s" %% "llm4s-vectorstore-elasticsearch" % llm4sVersion
```
```scala
import org.llm4s.vectorstore.elasticsearch._
// Local single node with security off:
// docker run -p 9200:9200 -e discovery.type=single-node -e xpack.security.enabled=false \
// docker.elastic.co/elasticsearch/elasticsearch:8.15.3
val local = ElasticsearchVectorStore.local(index = "my-vectors")
// A secured cluster: an API key (its `encoded` value) or a username and password
val encodedKey = "your-encoded-api-key" // from your secret store
val config = ElasticsearchConfig("https://es.example.com:9200")
.withIndex("my-vectors")
.withAuth(ElasticsearchAuth.ApiKey(encodedKey))
.withSimilarity(ElasticsearchSimilarity.Cosine) // or DotProduct, L2Norm, MaxInnerProduct
.withDimensions(1536) // unset: the first record's size
val store = ElasticsearchVectorStore(config)
```
Or from configuration, at the app edge:
```hocon
llm4s.vectorstore.elasticsearch {
baseUrl = "https://es.example.com:9200" # ELASTICSEARCH_URL; default http://localhost:9200
index = "my-vectors" # ELASTICSEARCH_INDEX; default llm4s-vectors
apiKey = ${?ELASTICSEARCH_API_KEY} # or username / password (ELASTICSEARCH_USERNAME, _PASSWORD)
similarity = cosine # cosine | dot_product | l2_norm | max_inner_product
dimensions = 1536
# numCandidates = 100 # kNN candidates per shard; default max(100, 10 * topK)
# refreshOnWrite = true # writes wait for a refresh, so a search sees them at once
# timeout = 30 seconds
}
```
```scala
import org.llm4s.config.ElasticsearchVectorStoreConfigLoader
val store = for {
config <- ElasticsearchVectorStoreConfigLoader.default()
store <- ElasticsearchVectorStore(config)
} yield store
```
How it maps onto Elasticsearch:
- **Index**: created on the first upsert when it does not exist, with a `dense_vector` field (`embedding`, indexed
for kNN with the configured similarity and size), `content` as `text`, `id` as `keyword`, and every metadata value
as a `keyword` under `metadata` (a dynamic template). `clear()` deletes the index; the next upsert creates it again.
Reads of an index that does not exist yet answer as an empty store.
- **Writes**: `upsert` and `upsertBatch` are one `_bulk` request, and the document `_id` is the record ID, so an
upsert replaces. A record whose size differs from the configured `dimensions` is refused before anything is sent;
a record Elasticsearch rejects is reported with its ID and reason. `deleteByPrefix` and `deleteByFilter` are
`_delete_by_query` and return the number deleted.
- **Search**: an approximate kNN search with the metadata filter inside the `knn` clause, so `topK` results come back
when that many records match. Elasticsearch's `_score` is clamped to `[0, 1]`: it already lies there for `cosine`
and `l2_norm` (and for `dot_product` over unit vectors), while `max_inner_product` can score above 1.
- **Filters**: `Equals` is a `term` query, `In` a `terms` query, `HasKey` an `exists` query, `Contains` a `wildcard`
query (`*substring*`, matched literally), and `And`, `Or` and `Not` are `bool` queries.
- **Errors**: a 401 or 403 is an `AuthenticationError`, a 429 a `RateLimitError`, any other failing status a
`ServiceError` with the status and the redacted body; configured credentials are masked wherever a body echoes them.
Limits to know: `list` pages with `from` and `size`, so `offset + limit` cannot exceed the index's
`max_result_window` (10,000 by default); `Contains` is a leading-wildcard query, which a cluster with
`search.allow_expensive_queries: false` refuses; a metadata key containing a dot is mapped as a nested object path,
so keys `a` and `a.b` cannot both be used. The base URL must be `https` when credentials are configured, unless the
host is loopback. `ElasticsearchVectorStoreSpec` in `modules/it` (`@Docker` tier, `ELASTICSEARCH_TEST_URL`) runs every
operation against a real cluster.
### OpenSearch
`OpenSearchVectorStore` keeps the vectors in one index of an OpenSearch cluster with the k-NN plugin (bundled with
the standard distribution), through the REST API: no OpenSearch client library is added. It is in the same module as
the Elasticsearch store, `llm4s-vectorstore-elasticsearch` (Experimental), and shares its code: OpenSearch kept
Elasticsearch 7.10's document, `_bulk` and query DSL APIs, so records, writes, reads, deletes, `stats` and every
metadata filter behave as described for [Elasticsearch](#elasticsearch). Only the vector field, the kNN query, the
scores and the authentication differ, and those are what `OpenSearchVectorStore` changes.
```scala
import org.llm4s.vectorstore.opensearch._
// Local single node with the security plugin off:
// docker run -p 9200:9200 -e discovery.type=single-node -e DISABLE_SECURITY_PLUGIN=true \
// -e DISABLE_INSTALL_DEMO_CONFIG=true opensearchproject/opensearch:2.19.1
val local = OpenSearchVectorStore.local(index = "my-vectors")
// A secured cluster: a username and password, or a bearer token (JWT)
val password = "your-password" // from your secret store
val config = OpenSearchConfig("https://search.example.com:9200")
.withIndex("my-vectors")
.withAuth(OpenSearchAuth.Basic("llm4s", password))
.withSpaceType(OpenSearchSpaceType.CosineSimil) // or InnerProduct, L2
.withEngine(OpenSearchEngine.Lucene) // or Faiss, Nmslib (deprecated)
.withDimensions(1536) // unset: the first record's size
val store = OpenSearchVectorStore(config)
```
Or from configuration, at the app edge:
```hocon
llm4s.vectorstore.opensearch {
baseUrl = "https://search.example.com:9200" # OPENSEARCH_URL; default http://localhost:9200
index = "my-vectors" # OPENSEARCH_INDEX; default llm4s-vectors
username = llm4s # OPENSEARCH_USERNAME
password = ${?OPENSEARCH_PASSWORD} # or token = ${?OPENSEARCH_TOKEN}, not both
spaceType = cosinesimil # cosinesimil | innerproduct | l2
engine = lucene # lucene | faiss | nmslib
dimensions = 1536
# m = 16 # HNSW links per node; default: the engine's
# efConstruction = 100 # default: the engine's
# numCandidates = 100 # the query's k; see below
# refreshOnWrite = true
# timeout = 30 seconds
}
```
```scala
import org.llm4s.config.OpenSearchVectorStoreConfigLoader
val store = for {
config <- OpenSearchVectorStoreConfigLoader.default()
store <- OpenSearchVectorStore(config)
} yield store
```
How it maps onto OpenSearch:
- **Index**: created on the first upsert with `index.knn: true` and a `knn_vector` field (`embedding`) of the
configured `dimension`, indexed with `method` `hnsw`, the configured `space_type` and `engine`, and `m` and
`ef_construction` when set. The other fields are mapped as for Elasticsearch.
- **Search**: a `knn` query, `{"knn": {"embedding": {"vector": [...], "k": k}}}`. On Lucene (OpenSearch 2.4+) and
Faiss (2.9+) the metadata filter goes inside it as `filter` - efficient filtering, so `topK` results come back when
that many records match, and `k` defaults to `topK`. NMSLIB has no efficient filtering, so there the `knn` query is
the `must` clause of a `bool` query whose `filter` clause drops non-matching neighbours afterwards; `k` then
defaults to `max(100, 10 * topK)`, and a selective filter can still return fewer than `topK` - raise
`numCandidates` (the `k` sent) if it does. `k` is never below `topK` and at most 10,000.
- **Scores**: OpenSearch scores each space differently; the store maps them onto `[0, 1]` like Elasticsearch's.
`cosinesimil` is already `(1 + cosine) / 2` on every engine and is kept; `innerproduct`'s `dot + 1` (or
`1 / (1 - dot)` for a negative dot) becomes `(1 + dot) / 2`, which lies in `[0, 1]` for unit vectors and is clamped
otherwise; `l2`'s `1 / (1 + distance²)` is kept.
- **Authentication**: HTTP basic (the security plugin's internal users, also on Amazon OpenSearch Service with
fine-grained access control) or `Authorization: Bearer` (its JWT backend). AWS Signature Version 4 (IAM) request
signing is not supported; use an internal user, or a signing proxy in front of the domain.
- **Errors**: as for Elasticsearch, with `opensearch` as the provider of the error.
Engine notes: Faiss supports `cosinesimil` from OpenSearch 2.19 and normalises the vectors it indexes; NMSLIB is
deprecated by OpenSearch, and a newer cluster may refuse to create an NMSLIB index, so use it only for an index that
already has it. The `knn_vector` dimension limit is 16,000. `OpenSearchVectorStoreSpec` in `modules/it` (`@Docker`
tier, `OPENSEARCH_TEST_URL`) runs every operation against a real cluster, on each engine.
### Redis and Valkey
`RedisVectorStore` keeps the vectors as hashes in a Redis server with the Redis Query Engine (Redis Stack, or Redis 8
and later, where search is built in) or a Valkey server with the `valkey-search` module (the `valkey-bundle` image),
and searches them through a vector index. Redis speaks RESP rather than HTTP, so the module brings a client library,
[Jedis](https://github.com/redis/jedis) (MIT); it stays out of the store's API. It lives in its own module
(Experimental):
```scala
libraryDependencies += "org.llm4s" %% "llm4s-vectorstore-redis" % llm4sVersion
```
```scala
import org.llm4s.vectorstore.redis._
// Local server without authentication:
// docker run -p 6379:6379 redis/redis-stack-server:7.4.0-v8 (or valkey/valkey-bundle:9.0.6)
val local = RedisVectorStore.local(index = "my-vectors")
// A remote server: TLS, an ACL user and a password
val password = "your-password" // from your secret store
val config = RedisConfig("redis.example.com", port = 6380)
.withTls(true)
.withUsername("app")
.withPassword(password)
.withIndex("my-vectors")
.withDistanceMetric(RedisDistanceMetric.Cosine) // or InnerProduct, L2
.withAlgorithm(RedisVectorAlgorithm.Hnsw) // or Flat
.withDimensions(1536) // unset: the first record's size
val store = RedisVectorStore(config)
```
Or from configuration, at the app edge:
```hocon
llm4s.vectorstore.redis {
host = "redis.example.com" # REDIS_HOST; default localhost
port = 6380 # REDIS_PORT; default 6379
tls = true # REDIS_TLS; default false
username = ${?REDIS_USERNAME} # unset: the default user
password = ${?REDIS_PASSWORD} # unset: no authentication
index = "my-vectors" # REDIS_VECTOR_INDEX; default llm4s-vectors
distanceMetric = COSINE # COSINE | IP | L2
algorithm = HNSW # HNSW | FLAT
dimensions = 1536
# keyPrefix = "my-vectors:" # default ":"
# database = 0
# timeout = 10 seconds
}
```
```scala
import org.llm4s.config.RedisVectorStoreConfigLoader
val store = for {
config <- RedisVectorStoreConfigLoader.default()
store <- RedisVectorStore(config)
} yield store
```
How it maps onto Redis:
- **Records**: each record is a hash at ``, holding the ID, the content, the metadata as a JSON object,
the vector as little-endian `FLOAT32` bytes, a write sequence number, and the metadata again as encoded tags.
`upsert` and `upsertBatch` are one pipeline of `DEL` and `HSET` per record, so an upsert replaces the whole record.
A record whose size differs from the index's is refused before anything is sent.
- **Index**: created on the first upsert when it does not exist (`FT.CREATE ... ON HASH PREFIX 1 `), with
the `VECTOR` field (`embedding`: the configured algorithm, distance and size), a sortable `NUMERIC` field for write
order, and two `TAG` fields for metadata. An index that exists is used as it is. Reads of an index that does not
exist yet answer as an empty store. `clear()` drops the index and deletes the store's keys; the next upsert creates
it again.
- **Search**: a KNN query, `()=>[KNN k @embedding $BLOB]`, so the filter is a pre-filter and `topK` results come
back when that many records match. Redis reports a distance; the score is `1 - distance / 2` for `COSINE` and `IP`
(that is `(1 + cosine) / 2`) and `1 / (1 + distance)` for `L2`, clamped to `[0, 1]`.
- **Filters**: any metadata value can be filtered on; they are all strings. The schema of a search index is fixed and
metadata keys are not, so every entry is one tag of a `TAG` field, encoded (hex bytes) so that no value can clash
with the tag separator, case folding, whitespace trimming or query escaping. `Equals` and `In` match entries,
`HasKey` matches keys, `Contains` is a wildcard query (`w'...'`), and `And`, `Or` and `Not` combine clauses. An
empty `In` matches nothing and is answered without a query.
- **Deletes**: `delete` and `deleteBatch` are one `DEL`. `deleteByPrefix` walks the store's keys with `SCAN`;
`deleteByFilter` deletes the matching records a page at a time. Both return the number deleted.
- **Errors**: a connection failure is a `NetworkError` (a `TimeoutError` when the server does not answer in time),
refused credentials (`NOAUTH`, `WRONGPASS`, `NOPERM`) an `AuthenticationError`, and any other error reply a
`ProcessingError` with the server's message.
Valkey: the `valkey-search` in Valkey 9.0 (`valkey-bundle:9.0`) answers only vector queries. On it `upsert`, `get`,
`getBatch`, `delete`, `deleteBatch`, `deleteByPrefix`, `clear` and `search` with every filter but `Contains` work (the
index is created without `SORTABLE`, which it refuses), while `count` and `stats` over an index, `list` and
`deleteByFilter` are refused with the server's message, as is `Contains`, which has no wildcard query there.
Limits to know: `Contains` needs the Redis Query Engine 2.6 or later. A wildcard expands to at most
`search-max-prefix-expansions` distinct values (200 by default), so `Contains` over a key with more distinct values than that may miss records. A search reply holds at most
`search-max-search-results` records (10,000 by default), which bounds `topK` and `list`'s `offset + limit`. The store
connects to one standalone server, not a cluster, and the Redis Query Engine indexes only database 0 on most
deployments. `deleteByPrefix` and `clear()` scan the keyspace. Without `tls` the password crosses the network in clear
text: keep that to a trusted network. `RedisVectorStoreSpec` and `ValkeyVectorStoreSpec` in `modules/it` (`@Docker`
tier, `REDIS_TEST_URL` and `VALKEY_TEST_URL`) run every operation against real servers.
---
## Core Concepts
### VectorRecord
A `VectorRecord` represents a single entry in the vector store:
```scala
final case class VectorRecord(
id: String, // Unique identifier
embedding: Array[Float], // Vector embedding
content: Option[String] = None, // Optional text content
metadata: Map[String, String] = Map.empty // Key-value metadata
)
```
**Creating Records:**
```scala
// With explicit ID
val record1 = VectorRecord(
id = "doc-123",
embedding = Array(0.1f, 0.2f, 0.3f),
content = Some("Document text"),
metadata = Map("source" -> "wiki", "lang" -> "en")
)
// With auto-generated ID
val record2 = VectorRecord.create(
embedding = Array(0.1f, 0.2f, 0.3f),
content = Some("Another document")
)
// Add metadata fluently
val record3 = VectorRecord("id", Array(1.0f))
.withMetadata("key1", "value1")
.withMetadata(Map("key2" -> "value2", "key3" -> "value3"))
```
### ScoredRecord
Search results include similarity scores:
```scala
final case class ScoredRecord(
record: VectorRecord,
score: Double // 0.0 to 1.0, higher is more similar
)
```
---
## Operations
### CRUD Operations
The fragments from here to the end of the page use `store`, a `VectorStore` such as the one in the Quick Start,
`record` and `records` (a `VectorRecord` and a `Seq[VectorRecord]`), `ids` (a `Seq[String]`) and `embeddingVector`,
a query embedding (an `Array[Float]`).
```scala
// Single record operations
store.upsert(record) // Insert or replace
store.get("doc-id") // Retrieve by ID
store.delete("doc-id") // Delete by ID
// Batch operations (more efficient)
store.upsertBatch(records) // Insert/replace multiple
store.getBatch(ids) // Retrieve multiple
store.deleteBatch(ids) // Delete multiple
// Clear all records
store.clear()
```
### Search
```scala
// Basic search
val results = store.search(
queryVector = embeddingVector,
topK = 10
)
// Search with metadata filter
val filter = MetadataFilter.Equals("type", "document")
val filtered = store.search(
queryVector = embeddingVector,
topK = 10,
filter = Some(filter)
)
```
### Listing and Pagination
```scala
// List all records
val all = store.list()
// Paginate results
val page1 = store.list(limit = 10, offset = 0)
val page2 = store.list(limit = 10, offset = 10)
// List with filter
val docs = store.list(filter = Some(MetadataFilter.Equals("type", "doc")))
```
### Statistics
```scala
val stats = store.stats()
stats.foreach { s =>
println(s"Total records: ${s.totalRecords}")
println(s"Dimensions: ${s.dimensions}")
println(s"Size: ${s.formattedSize}")
}
```
---
## Metadata Filtering
The `MetadataFilter` DSL allows composing complex filters:
### Basic Filters
```scala
import org.llm4s.vectorstore.MetadataFilter._
// Exact match
val byType = Equals("type", "document")
// Contains substring
val byContent = Contains("summary", "Scala")
// Has key (any value)
val hasAuthor = HasKey("author")
// Value in set
val byLang = In("lang", Set("en", "es", "fr"))
```
### Combining Filters
```scala
// AND - both conditions must match
val andFilter = Equals("type", "doc").and(Equals("lang", "en"))
// OR - either condition can match
val orFilter = Equals("type", "doc").or(Equals("type", "article"))
// NOT - negate a filter
val notFilter = !Equals("archived", "true")
// Complex combinations
val complex = Equals("type", "doc")
.and(Equals("lang", "en").or(Equals("lang", "es")))
.and(!Equals("draft", "true"))
```
### Using Filters
With `byType` and `complex` from the filters above:
```scala
// In search
store.search(embeddingVector, topK = 10, filter = Some(byType))
// In list
store.list(filter = Some(complex))
// In count
store.count(filter = Some(byType))
// Delete by filter
store.deleteByFilter(Equals("archived", "true"))
```
---
## Factory Configuration
### Using VectorStoreFactory
`VectorStoreFactory` covers the backends inside `llm4s-rag` (SQLite, pgvector, Qdrant). A backend in a module of its
own, such as [Elasticsearch](#elasticsearch), [OpenSearch](#opensearch) or [Redis](#redis-and-valkey), is built with its own constructor, and any `VectorStore` can be passed
wherever one is taken.
```scala
import org.llm4s.vectorstore._
// In-memory (default)
val memStore = VectorStoreFactory.inMemory()
// File-based SQLite
val fileStore = VectorStoreFactory.sqlite("/path/to/db.sqlite")
// From provider name
val store = VectorStoreFactory.create("sqlite", path = Some("/path/to/db.sqlite"))
// From config object
val config = VectorStoreFactory.Config.sqlite("/path/to/db.sqlite")
val configStore = VectorStoreFactory.create(config)
```
### Configuration Options
```scala
// Default in-memory config
val defaultConfig = VectorStoreFactory.Config.default
// SQLite file config
val sqliteConfig = VectorStoreFactory.Config.sqlite("/path/to/vectors.db")
// In-memory config
val memConfig = VectorStoreFactory.Config.inMemory
// pgvector config
val pgConfig = VectorStoreFactory.Config.pgvector(
connectionString = "jdbc:postgresql://localhost:5432/postgres",
tableName = "vectors"
)
// Qdrant config
val qdrantConfig = VectorStoreFactory.Config.qdrant(
collectionName = "vectors",
port = 6333
)
// Qdrant Cloud config
val qdrantCloudConfig = VectorStoreFactory.Config.qdrantCloud(
cloudUrl = "https://xxx.qdrant.io",
apiKey = "your-api-key",
collectionName = "vectors"
)
// With options
val withOptions = VectorStoreFactory.Config()
.withSQLite("/path/to/db.sqlite")
.withOption("cache_size", "10000")
```
---
## Hybrid Search
Hybrid search combines vector similarity (semantic) with BM25 keyword matching for better retrieval quality. LLM4S provides a `HybridSearcher` that fuses results from both search types.
### Quick Start
```scala
import org.llm4s.vectorstore._
val embedding = Array(0.1f, 0.2f, 0.3f)
val results = for {
// Create stores
vectorStore <- VectorStoreFactory.inMemory()
keywordIndex <- KeywordIndex.inMemory()
// Create hybrid searcher
searcher = HybridSearcher(vectorStore, keywordIndex)
// Index documents in both stores
_ <- vectorStore.upsert(VectorRecord("doc-1", embedding, Some("Scala programming language")))
_ <- keywordIndex.index(KeywordDocument("doc-1", "Scala programming language"))
// Search with both vector and keyword
found <- searcher.search(
queryEmbedding = embedding,
queryText = "Scala",
topK = 10
)
_ = searcher.close()
} yield found
results.foreach(_.foreach { r =>
println(s"${r.id}: ${r.score} (vector: ${r.vectorScore}, keyword: ${r.keywordScore})")
})
```
### Fusion Strategies
Choose how to combine vector and keyword scores, given a `searcher: HybridSearcher` and a query `embedding` as in
the Quick Start:
```scala
import org.llm4s.vectorstore.FusionStrategy._
// Reciprocal Rank Fusion (default) - rank-based, robust to score differences
val rrfResults = searcher.search(embedding, "query", strategy = RRF(k = 60))
// Weighted Score - normalized scores with configurable weights
val weightedResults = searcher.search(
embedding, "query",
strategy = WeightedScore(vectorWeight = 0.7, keywordWeight = 0.3)
)
// Single-mode search
val vectorOnly = searcher.search(embedding, "query", strategy = VectorOnly)
val keywordOnly = searcher.search(embedding, "query", strategy = KeywordOnly)
```
**When to use each strategy:**
| Strategy | Best For |
|----------|----------|
| RRF (default) | General use, heterogeneous sources |
| WeightedScore | When you know relative importance |
| VectorOnly | Semantic/conceptual queries |
| KeywordOnly | Exact term matching, names, codes |
#### How WeightedScore combines the two channels
Vector similarity and BM25 scores live on different scales, so each channel is first rescaled on its own: its best
hit becomes `1.0`, its weakest hit becomes `0.1`, and everything between is spread linearly. A chunk the channel did
not return at all scores `0.0` for that channel. The final score is the weighted average of the two channel scores,
so it lies in `[0, 1]`. If a channel returned a single hit, or all its hits tie, each of them scores `1.0`.
The `0.1` floor is deliberate. With plain min-max the weakest hit would map to `0.0`, the same as a miss, so a
relevant chunk that happened to rank last in one channel would contribute nothing from it.
```scala
import org.llm4s.vectorstore._
// Only the vector channel counts (vector weight 1.0), and the keyword channel finds nothing.
val hits = for {
store <- VectorStoreFactory.inMemory()
keywords <- KeywordIndex.inMemory()
_ <- store.upsert(VectorRecord("strong", Array(1f, 0f), Some("strong")))
_ <- store.upsert(VectorRecord("weak", Array(0.6f, 0.8f), Some("weak")))
found <- HybridSearcher(store, keywords).search(
Array(1f, 0f),
"no keyword matches this",
topK = 5,
strategy = FusionStrategy.WeightedScore(vectorWeight = 1.0, keywordWeight = 0.0)
)
} yield found.map(hit => hit.id -> hit.score)
// Right(Seq("strong" -> 1.0, "weak" -> 0.1)): the weak hit is not 0.0
```
#### Invalid weights
`WeightedScore` needs finite, non-negative weights that are not both zero and whose sum is finite. Anything else
(`WeightedScore(0.0, 0.0)`, a negative, `NaN` or infinite weight) throws `IllegalArgumentException` from the
constructor, as does `RAGConfig.withWeightedScore`, which builds one. That is deliberate: weights written in code are
a programming error, like any `require`-guarded case class, and the chainable `RAGConfig` builders return a
`RAGConfig`, so they cannot return a `Left`.
If the weights come from user input, check them first so a bad value becomes a `Left` instead of an exception.
Letting the constructor do the checking keeps the rules in one place:
```scala
import org.llm4s.types.{ Result, TryOps }
import org.llm4s.vectorstore.FusionStrategy
import scala.util.Try
def weightedScore(vector: Double, keyword: Double): Result[FusionStrategy] =
Try(FusionStrategy.WeightedScore(vector, keyword)).toResult
```
### BM25 Keyword Index
The `KeywordIndex` provides BM25-scored full-text search using SQLite FTS5:
```scala
import org.llm4s.vectorstore._
val results = for {
// Create keyword index
index <- KeywordIndex.inMemory()
// Index documents
_ <- index.index(KeywordDocument("doc-1", "Scala is a programming language"))
_ <- index.index(KeywordDocument("doc-2", "Python is also popular"))
// Search with highlights
found <- index.searchWithHighlights("programming", topK = 5)
_ = index.close()
} yield found
results.foreach(_.foreach { r =>
println(s"${r.id}: ${r.score}")
r.highlights.foreach(h => println(s" ...${h}..."))
})
```
### Factory Methods
Each returns a `Result[HybridSearcher]`:
```scala
import org.llm4s.vectorstore._
// In-memory (development)
val memSearcher = HybridSearcher.inMemory()
// File-based SQLite
val fileSearcher = HybridSearcher.sqlite(
vectorDbPath = "/path/to/vectors.db",
keywordDbPath = "/path/to/keywords.db"
)
// From configuration
val config = HybridSearcher.Config()
.withVectorStore(
VectorStoreFactory.Config.pgvector(
connectionString = "jdbc:postgresql://localhost:5432/mydb",
tableName = "vectors"
)
)
.withRRF(k = 60)
val configSearcher = HybridSearcher(config)
```
### PostgreSQL Native Hybrid Search
For production deployments, you can run fully PostgreSQL-based hybrid search using pgvector for vector similarity and PostgreSQL native full-text search (tsvector/tsquery) for BM25-like keyword matching. This eliminates the need for a separate SQLite database.
**Requirements:** PostgreSQL 16+ with pgvector extension (PostgreSQL 18+ recommended for best performance)
```scala
import org.llm4s.vectorstore._
val embedding = Array(0.1f, 0.2f, 0.3f)
val results = for {
// Create fully PostgreSQL-based hybrid searcher with shared connection pool
searcher <- HybridSearcher.pgvectorShared(
connectionString = "jdbc:postgresql://localhost:5432/mydb",
user = "postgres",
password = "secret",
vectorTableName = "vectors",
keywordTableName = "documents"
)
// Index documents (both vector and keyword stores share the same PostgreSQL database)
_ <- searcher.vectorStore.upsert(VectorRecord("doc-1", embedding, Some("Scala programming")))
_ <- searcher.keywordIndex.index(KeywordDocument("doc-1", "Scala programming"))
// Search using both vector similarity and PostgreSQL full-text search
found <- searcher.search(embedding, "Scala", topK = 10)
_ = searcher.close()
} yield found
```
**PostgreSQL Keyword Index:**
The `PgKeywordIndex` uses PostgreSQL native full-text search capabilities:
- **tsvector** for tokenized document representation
- **ts_rank_cd** for BM25-like relevance scoring
- **websearch_to_tsquery** for natural search syntax (supports `"exact phrases"`, `OR`, `-exclude`)
- **ts_headline** for highlighted snippets
- **GIN indexes** for fast full-text and metadata queries
```scala
import org.llm4s.vectorstore._
val filtered = for {
// Create standalone PostgreSQL keyword index
keywordIndex <- KeywordIndex.postgres(
connectionString = "jdbc:postgresql://localhost:5432/mydb",
user = "postgres",
password = "secret",
tableName = "documents" // Creates documents_keyword table
)
// Index with metadata
_ <- keywordIndex.index(KeywordDocument(
id = "doc-1",
content = "PostgreSQL provides powerful full-text search",
metadata = Map("lang" -> "en", "type" -> "tutorial")
))
// Search with highlights
highlighted <- keywordIndex.searchWithHighlights("full-text search", topK = 5)
_ = highlighted.foreach { r =>
println(s"${r.id}: ${r.score}")
r.highlights.foreach(h => println(s" ${h}")) // Contains highlighted terms
}
// Filter by metadata
found <- keywordIndex.search(
"PostgreSQL",
topK = 10,
filter = Some(MetadataFilter.Equals("lang", "en"))
)
_ = keywordIndex.close()
} yield found
```
**RAG Configuration:**
Use `withPgHybrid()` to configure fully PostgreSQL-based hybrid search in the RAG pipeline:
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.model.ModelRegistryService
import org.llm4s.rag._
val config = RAGConfig()
.withEmbeddings("openai")
.withPgHybrid(
connectionString = "jdbc:postgresql://localhost:5432/mydb",
user = "postgres",
password = "secret",
vectorTableName = "vectors",
keywordTableName = "documents"
)
.withRRF(60) // Reciprocal Rank Fusion
val results = for {
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
// The embedding provider's settings come from llm4s.embeddings
rag <- RAG.build(config, _ => Llm4sConfig.embeddings().map(_._2))
// Documents are stored in PostgreSQL for both vector and keyword search
_ <- rag.ingestText("PostgreSQL hybrid search combines vector and full-text.", "doc-1")
found <- rag.query("database search")
} yield found
```
**Benefits of PostgreSQL Native Hybrid:**
- Single database for all RAG storage (vectors, keywords, metadata)
- Shared connection pool for efficiency
- Native PostgreSQL transactions and ACID guarantees
- Simplified deployment and backup strategy
- Full-text search with stemming, stop words, and language support
---
## Reranking
Reranking improves retrieval quality by re-scoring initial search results using a more powerful model (like Cohere's cross-encoder). This is particularly useful when you retrieve many candidates and want to refine the ranking.
### Quick Start
```scala
import org.llm4s.vectorstore._
import org.llm4s.reranker._
val embedding = Array(0.1f, 0.2f, 0.3f) // the query's embedding
val reranker = RerankerFactory.cohere(apiKey = "your-cohere-api-key")
val results = for {
// Create hybrid searcher (index documents into it as in Hybrid Search)
searcher <- HybridSearcher.inMemory()
// Search with reranking
found <- searcher.searchWithReranking(
queryEmbedding = embedding,
queryText = "What is Scala?",
topK = 5, // Final results to return
rerankTopK = 50, // Candidates to rerank
reranker = Some(reranker)
)
} yield found
results.foreach(_.foreach { r =>
println(s"${r.id}: ${r.score}")
})
```
### Reranker Options
```scala
import org.llm4s.reranker._
// Cohere reranker (recommended for production)
val cohereReranker = RerankerFactory.cohere(
apiKey = "your-api-key",
model = "rerank-english-v3.0", // or rerank-multilingual-v3.0
baseUrl = "https://api.cohere.com"
)
// Passthrough reranker (no-op, preserves original order)
val passthrough = RerankerFactory.passthrough
// From typed config
val config = RerankProviderConfig(
apiKey = "your-api-key",
model = "rerank-english-v3.0",
baseUrl = "https://api.cohere.com"
)
val fromConfig = RerankerFactory.fromConfig(Some(config))
```
### LLM Reranker
`LLMReranker` uses any configured chat model as the judge, so you can rerank without a Cohere key. Build it directly or through `RerankerFactory.llm`:
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.model.ModelRegistryService
import org.llm4s.reranker.{ LLMReranker, RerankerFactory }
val reranker = for {
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
providerCfg <- Llm4sConfig.defaultProvider()
client <- LLMConnect.getClient(providerCfg)
} yield LLMReranker(client) // same as RerankerFactory.llm(client)
// Optional knobs (both have defaults):
// RerankerFactory.llm(client, batchSize = 5, systemPrompt = Some("..."))
```
How it scores:
- **Batches.** Documents go to the model `batchSize` at a time (default `LLMReranker.DEFAULT_BATCH_SIZE` = 10), one model call per batch. A smaller batch means more calls but shorter prompts.
- **1,000-unit cut.** Each document is truncated to its first 1,000 UTF-16 code units (the Scala `String` length), stopping one short rather than splitting a surrogate pair, before it goes into the prompt. That is 1,000 characters for most text, but emoji-heavy text can be cut after roughly 500 characters. Put the most relevant text first in long chunks.
- **Scores.** The model must answer with a JSON array of scores. A number, or a numeric string such as `"0.9"`, is clamped to `[0, 1]`; `null`, booleans and other non-string values score 0.5. A string that is not a number fails the parse for the whole batch, so the entire batch gets neutral scores: `[0.9, "unknown"]` gives both documents 0.5, not 0.9 and 0.5. A short array is padded with 0.5; a long one is cut to the batch size.
- **Failed batches.** If a batch's call or parse fails, every document in that batch gets a neutral 0.5 and reranking continues with the next batch. Cancellation is the exception: it is returned as an error.
- **`systemPrompt`.** Replaces the default judging instructions. Keep asking for "ONLY a JSON array of scores" so replies stay parseable.
**LLM vs Cohere:** the LLM reranker reuses the model you already have and needs no extra service, but every batch is a full chat completion. That makes it slower and usually more expensive per query than Cohere's dedicated rerank endpoint, which scores all candidates in one request. Prefer Cohere for latency-sensitive or high-volume retrieval, and the LLM reranker when you can't add another provider or want the judging criteria in your own prompt.
### Reranking in the RAG pipeline
`RAGConfig` selects the strategy with `RerankingStrategy`:
| Strategy | Builder | Notes |
|---|---|---|
| `RerankingStrategy.None` | (default) | Results keep their fused retrieval order |
| `RerankingStrategy.Cohere(model)` | `.withCohereReranking(model = "rerank-english-v3.0")` | Needs a Cohere reranker config. The model comes from that `RerankProviderConfig`; the `model` argument is currently not used to choose it |
| `RerankingStrategy.LLM` | `.withLLMReranking` | Needs an LLM client: add `.withLLM(client)` when building RAG |
`.withRerankTopK(k)` sets how many retrieved candidates are reranked.
### Direct Reranking API
```scala
import org.llm4s.reranker._
val reranker = RerankerFactory.cohere(apiKey = "xxx")
val request = RerankRequest(
query = "What is Scala?",
documents = Seq(
"Scala is a programming language",
"Python is popular for ML",
"Scala runs on the JVM"
),
topK = Some(2)
)
val response = reranker.rerank(request)
response.foreach { r =>
r.results.foreach { result =>
println(s"[${result.index}] ${result.score}: ${result.document}")
}
}
```
### Configuration
Load the reranker from configuration at the edge of your application with
`RerankerConfigLoader` (in `llm4s-rag`), and hand it to `RAG.build`:
```scala
import org.llm4s.config.{ Llm4sConfig, RerankerConfigLoader }
import org.llm4s.model.ModelRegistryService
import org.llm4s.rag.{ RAG, RAGConfig }
val config = RAGConfig().withEmbeddings("openai")
for {
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
reranker <- RerankerConfigLoader.default() // Option[RerankProviderConfig]; None when disabled
rag <- RAG.build(
config,
_ => Llm4sConfig.embeddings().map(_._2),
resolveRerankerConfig = () => Right(reranker)
)
} yield rag
```
It reads `llm4s.rerank`. The Cohere key is `llm4s.rerank.cohere.apiKey` when set, and otherwise
Cohere's shared `llm4s.credentials.cohere.apiKey`, bound to `COHERE_API_KEY` - the key a Cohere
chat section uses too, so one variable serves both. Set `llm4s.rerank.cohere.apiKey` only to bill
reranking to another account. (Before `RerankerConfigLoader`, nothing read `llm4s.rerank`,
although `reference.conf` bound it.)
| Environment Variable | Description | Default |
|---------------------|-------------|---------|
| `RERANK_PROVIDER` | Provider: cohere, none | none |
| `COHERE_API_KEY` | Cohere API key, shared with Cohere chat (`llm4s.credentials.cohere.apiKey`) | - |
| `COHERE_RERANK_MODEL` | Model name | rerank-english-v3.0 |
| `COHERE_RERANK_BASE_URL` | API base URL | https://api.cohere.com |
---
## Document Chunking
Document chunking splits text into manageable pieces for embedding and retrieval. LLM4S provides multiple chunking strategies optimized for different content types. In the examples, `documentText` is the text to split.
### Quick Start
```scala
import org.llm4s.chunking._
// Create a sentence-aware chunker (recommended)
val chunker = ChunkerFactory.sentence()
// Chunk a document
val chunks = chunker.chunk(documentText, ChunkingConfig(
targetSize = 800, // Target chunk size in characters
maxSize = 1200, // Longer sentences are split at whitespace; a single longer word stays whole
overlap = 150 // Overlap between consecutive chunks
))
chunks.foreach { chunk =>
println(s"[${chunk.index}] ${chunk.content.take(50)}...")
}
```
### Chunking Strategies
```scala
import org.llm4s.chunking._
// Sentence-aware chunking (recommended for most text)
// Respects sentence boundaries for semantic coherence
val sentenceChunker = ChunkerFactory.sentence()
// Simple character-based chunking
// Fast but may split mid-sentence
val simpleChunker = ChunkerFactory.simple()
// Auto-detect based on content
// Detects markdown and chooses appropriate strategy
val autoChunker = ChunkerFactory.auto(documentText)
// By strategy name
val chunker = ChunkerFactory.create("sentence") // or "simple"
```
### Configuration Presets
```scala
import org.llm4s.chunking._
// Default: 800 char target, 150 overlap
val defaultConfig = ChunkingConfig.default
// Small chunks: 400 char target, 75 overlap
// Better for precise retrieval
val smallConfig = ChunkingConfig.small
// Large chunks: 1500 char target, 250 overlap
// Better for broader context
val largeConfig = ChunkingConfig.large
// No overlap
val noOverlapConfig = ChunkingConfig.noOverlap
// Custom configuration
val customConfig = ChunkingConfig(
targetSize = 600,
maxSize = 900,
overlap = 100,
minChunkSize = 50,
preserveCodeBlocks = true,
preserveHeadings = true
)
```
### With Source Metadata
```scala
import org.llm4s.chunking._
val chunker = ChunkerFactory.sentence()
// Chunks include source file in metadata
val chunks = chunker.chunkWithSource(
text = documentText,
sourceFile = "docs/guide.md",
config = ChunkingConfig.default
)
chunks.foreach { chunk =>
println(s"From: ${chunk.metadata.sourceFile.getOrElse("unknown")}")
println(s"Content: ${chunk.content.take(50)}...")
}
```
### Invalid Chunking Settings
`ChunkingConfig` checks its values when you build it, so a chunker never receives a configuration it cannot handle:
| Rule | Failing example |
|------|-----------------|
| `targetSize` is positive | `ChunkingConfig(targetSize = 0)` |
| `maxSize >= targetSize` | `ChunkingConfig(targetSize = 800, maxSize = 400)` |
| `0 <= overlap < targetSize` | `ChunkingConfig(targetSize = 800, overlap = 800)` |
| `minChunkSize` is not negative | `ChunkingConfig(minChunkSize = -1)` |
A value that breaks a rule throws `IllegalArgumentException` (the message names the rule, for example
`overlap must be >= 0 and < targetSize`). `RAGConfig.withChunking(strategy, size, overlap)` builds a
`ChunkingConfig`, so it throws too. Every configuration that *can* be built is safe: no chunker throws for it, and the
largest overlap allowed (`targetSize - 1`) still moves each chunk one character forward.
When the sizes come from user input, check them first. To get a `Left(ValidationError)` straight from the numbers, use
`ChunkingUtils.chunkTextValidated`; to build a config, let the constructor decide:
```scala
import org.llm4s.chunking.ChunkingConfig
import org.llm4s.llmconnect.utils.ChunkingUtils
import org.llm4s.types.{ Result, TryOps }
import scala.util.Try
val text = "one two three four five six seven eight nine ten"
// Left(ValidationError) with field "overlap" (or "size"), not an exception
val chunks = ChunkingUtils.chunkTextValidated(text, size = 20, overlap = 20)
def chunkingConfig(size: Int, overlap: Int): Result[ChunkingConfig] =
Try(
ChunkingConfig(
targetSize = size,
maxSize = math.min(size.toLong * 3 / 2, Int.MaxValue.toLong).toInt, // Long: no overflow for a large size
overlap = overlap
)
).toResult
```
### Chunking Best Practices
| Content Type | Recommended Strategy | Config |
|--------------|---------------------|--------|
| Prose/articles | `sentence` | `default` |
| Technical docs | `sentence` | `large` |
| Code files | `simple` | Custom with no overlap |
| Q&A pairs | `sentence` | `small` |
| Mixed content | `auto` | `default` |
**Tips:**
- Use overlap for context continuity in retrieval
- Smaller chunks improve retrieval precision but may lose context
- Larger chunks preserve context but may dilute relevance
- Sentence chunking prevents mid-thought splits
---
## Integration with RAG Pipeline
### Complete RAG Example
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.{ EmbeddingClient, LLMConnect }
import org.llm4s.llmconnect.config.{ EmbeddingModelConfig, ModelDimensionRegistry }
import org.llm4s.llmconnect.model.{ Completion, Conversation, EmbeddingRequest, InputPurpose, UserMessage }
import org.llm4s.model.ModelRegistryService
import org.llm4s.types.Result
import org.llm4s.vectorstore._
val documents = Seq(
"Scala is a programming language",
"LLM4S provides LLM integration",
"Vector stores enable semantic search"
)
val query = "What is Scala?"
val response: Result[Completion] = for {
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
// 1. Create embedding client and vector store
settings <- Llm4sConfig.embeddings()
providerId = settings._1
embeddingConfig = settings._2
dimensions <- ModelDimensionRegistry.getDimension(providerId, embeddingConfig.model)
model = EmbeddingModelConfig(embeddingConfig.model, dimensions)
embeddingClient <- EmbeddingClient.from(providerId, embeddingConfig)
vectorStore <- VectorStoreFactory.inMemory()
// 2. Ingest documents: one request, one vector per document, in order
embedded <- embeddingClient.embed(EmbeddingRequest(documents, model))
records = documents.zip(embedded.embeddings).zipWithIndex.map { case ((doc, vector), idx) =>
VectorRecord(id = s"doc-$idx", embedding = vector.map(_.toFloat).toArray, content = Some(doc))
}
_ <- vectorStore.upsertBatch(records)
// 3. Query with retrieval
queried <- embeddingClient.embed(EmbeddingRequest(Seq(query), model, InputPurpose.Query))
relevant <- vectorStore.search(queried.embeddings.flatten.map(_.toFloat).toArray, topK = 3)
// 4. Augment prompt with context
context = relevant.map(_.record.content.getOrElse("")).mkString("\n")
prompt = s"""Based on the following context:
$context
Answer this question: $query"""
// 5. Generate response
providerConfig <- Llm4sConfig.defaultProvider()
llm <- LLMConnect.getClient(providerConfig)
completion <- llm.complete(Conversation(Seq(UserMessage(prompt))))
} yield completion
```
---
## Best Practices
### Resource Management
Always close stores when done:
```scala
import org.llm4s.vectorstore._
val count = VectorStoreFactory.inMemory().flatMap { store =>
// Use scala.util.Using for automatic cleanup
scala.util.Using.resource(new java.io.Closeable {
def close(): Unit = store.close()
}) { _ =>
store.count() // Use the store
}
}
```
### Batch Operations
Use batch operations for efficiency:
```scala
// Good - single batch call
store.upsertBatch(records)
// Less efficient - individual calls
records.foreach(store.upsert)
```
### Error Handling
All operations return `Result[A]`:
```scala
store.search(embeddingVector, topK = 10) match {
case Right(results) =>
results.foreach(r => println(r.score))
case Left(error) =>
println(s"Search failed: ${error.formatted}")
}
// Or use for-comprehension
for {
results <- store.search(embeddingVector, topK = 10)
count <- store.count()
} yield (results, count)
```
### Metadata Design
Design metadata for your filtering needs:
```scala
// Good - filterable metadata
VectorRecord(
id = "doc-1",
embedding = embeddingVector,
metadata = Map(
"type" -> "article",
"source" -> "wikipedia",
"lang" -> "en",
"year" -> "2024"
)
)
// Then filter efficiently
store.search(
embeddingVector,
topK = 10,
filter = Some(Equals("type", "article").and(Equals("lang", "en")))
)
```
---
## Performance Considerations
### SQLite Backend
The SQLite backend is suitable for:
- Development and testing
- Small to medium datasets (~100K vectors)
- Single-machine deployments
- Scenarios where simplicity is preferred
**Limitations:**
- Vector similarity computed in Scala (not hardware-accelerated)
- All candidate vectors loaded into memory during search
- No built-in sharding or replication
### pgvector Backend
PostgreSQL with pgvector is ideal for:
- Production workloads with existing PostgreSQL infrastructure
- Medium to large datasets (millions of vectors)
- Teams familiar with PostgreSQL operations
- Applications requiring ACID transactions
**Features:**
- HNSW indexing for fast approximate nearest neighbor search
- Connection pooling with HikariCP
- Native vector operations in PostgreSQL
- Excellent SQL tooling and monitoring
**Setup:**
```bash
# Enable pgvector extension
psql -c "CREATE EXTENSION IF NOT EXISTS vector;"
```
### Qdrant Backend
Qdrant is recommended for:
- High-performance production workloads
- Large-scale deployments (billions of vectors)
- Cloud-native architectures
- Teams wanting managed vector database service
**Features:**
- Cloud-native architecture with horizontal scaling
- REST and gRPC APIs
- Rich filtering on payload fields
- Snapshot and backup capabilities
- Managed cloud offering available
**Setup:**
```bash
# Local development with Docker
docker run -p 6333:6333 qdrant/qdrant
# Or use Qdrant Cloud for production
```
### Choosing a Backend
| Requirement | Recommended Backend |
|-------------|---------------------|
| Development/testing | SQLite (in-memory) |
| Single-machine production | SQLite (file-based) |
| Existing PostgreSQL | pgvector |
| Existing Elasticsearch or OpenSearch cluster | `ElasticsearchVectorStore` / `OpenSearchVectorStore` |
| Full PostgreSQL hybrid RAG | pgvector + PgKeywordIndex |
| High-scale production | Qdrant |
| Managed service | Qdrant Cloud |
**Note:** For hybrid search with PostgreSQL, use `HybridSearcher.pgvectorShared()` or `RAGConfig.withPgHybrid()` to run both vector and keyword search against a single PostgreSQL database. This provides simplified operations, shared connection pooling, and ACID guarantees.
---
## API Reference
### VectorStore Trait
```scala
trait VectorStore {
def upsert(record: VectorRecord): Result[Unit]
def upsertBatch(records: Seq[VectorRecord]): Result[Unit]
def search(queryVector: Array[Float], topK: Int = 10,
filter: Option[MetadataFilter] = None): Result[Seq[ScoredRecord]]
def get(id: String): Result[Option[VectorRecord]]
def getBatch(ids: Seq[String]): Result[Seq[VectorRecord]]
def delete(id: String): Result[Unit]
def deleteBatch(ids: Seq[String]): Result[Unit]
def deleteByFilter(filter: MetadataFilter): Result[Long]
def count(filter: Option[MetadataFilter] = None): Result[Long]
def list(limit: Int = 100, offset: Int = 0,
filter: Option[MetadataFilter] = None): Result[Seq[VectorRecord]]
def clear(): Result[Unit]
def stats(): Result[VectorStoreStats]
def close(): Unit
}
```
---
## Measuring RAG Quality
Once you have a RAG pipeline, use the evaluation framework to measure and improve retrieval quality.
### Quick Evaluation
Given an `LLMClient` to judge with (`client`), an `EmbeddingClient` (`embedder`) and its `EmbeddingModelConfig`
(`embeddingModel`):
```scala
import org.llm4s.rag.evaluation._
val evaluator = new RAGASEvaluator(client, embedder, embeddingModel)
val sample = EvalSample(
question = "What is Scala?",
answer = "Scala is a programming language that runs on the JVM.", // your pipeline's answer
contexts = Seq("Scala is a programming language", "Scala runs on the JVM"), // the chunks it retrieved
groundTruth = Some("Scala is a programming language.")
)
val metrics = evaluator.evaluate(sample)
metrics.foreach { m =>
println(s"RAGAS Score: ${m.ragasScore}")
m.getMetric("faithfulness").foreach(f => println(s"Faithfulness: ${f.score}"))
}
```
### Benchmarking Different Configurations
Compare chunking strategies, fusion methods, and embedding providers:
```bash
# Compare fusion strategies
sbt "samples/runMain org.llm4s.samples.rag.BenchmarkExample --suite fusion"
# Compare chunking strategies
sbt "samples/runMain org.llm4s.samples.rag.BenchmarkExample --suite chunking"
```
**See the full [RAG Evaluation Guide](rag-evaluation) for:**
- RAGAS metrics explained (faithfulness, relevancy, precision, recall)
- Running benchmarks programmatically
- Optimization workflow and best practices
- Actual benchmark results and recommendations
---
## Next Steps
- **[RAG Evaluation Guide](rag-evaluation)** - Measure and improve RAG quality
- **[Embeddings Configuration](../getting-started/configuration#embeddings-configuration)** - Configure embedding providers
- **[Examples Gallery](/examples/#embeddings-examples)** - See RAG examples in action
- **[RAG in a Box](https://github.com/llm4s/rag_in_a_box)** - Production-ready RAG server with REST API, admin UI, and deployment options
---
Source: https://llm4s.org/guide/permission-based-rag
# Permission-Based RAG
Enterprise-grade access control for RAG applications.
---
## Overview
Permission-based RAG extends LLM4S with enterprise-grade access control for document retrieval. It enables you to:
- **Organize documents in collections** with hierarchical structure (e.g., `confluence/EN/archive`)
- **Control who can search** with collection-level `queryableBy` permissions
- **Fine-grained document access** with document-level `readableBy` permissions
- **Map users and groups** to efficient integer IDs for fast database queries
- **Query with patterns** like `*` (all), `confluence/*` (immediate children), `confluence/**` (all descendants)
**Key Features:**
- Two-level permission model (collection + document)
- Permission inheritance (children can only restrict, never loosen)
- Pattern-based collection queries
- PostgreSQL backend with pgvector for efficient vector search
- Seamless integration with the existing RAG API
---
## Quick Start
### Prerequisites
You need PostgreSQL with pgvector extension:
```bash
# Using Docker
docker run -d --name pgvector -p 5432:5432 \
-e POSTGRES_PASSWORD=postgres \
pgvector/pgvector:pg16
# Or enable on existing PostgreSQL
psql -c "CREATE EXTENSION IF NOT EXISTS vector;"
```
### Minimal Example
(not compile-checked)
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.model.ModelRegistryService
import org.llm4s.rag.RAG
import org.llm4s.rag.RAG.RAGConfigOps
import org.llm4s.rag.permissions._
import org.llm4s.rag.permissions.pg.PgSearchIndex
// build needs the extension import above and a model registry; ProviderRegistry.default is a given.
given ModelRegistryService = Llm4sConfig.modelRegistryService().toOption.get
// 1. Create and initialize the SearchIndex
val searchIndex = PgSearchIndex.fromJdbcUrl(
jdbcUrl = "jdbc:postgresql://localhost:5432/postgres",
user = "postgres",
password = "postgres",
vectorTableName = "rag_vectors"
).getOrElse(throw new RuntimeException("Failed to create SearchIndex"))
searchIndex.initializeSchema().getOrElse(throw new RuntimeException("Failed to init schema"))
// 2. Create a collection
val collectionPath = CollectionPath.unsafe("my-docs")
searchIndex.collections.create(CollectionConfig.publicLeaf(collectionPath))
// 3. Build RAG with permission support
val rag = RAG.builder()
.withEmbeddings("openai")
.withSearchIndex(searchIndex)
.build
.getOrElse(throw new RuntimeException("Failed to build RAG"))
// 4. Ingest a document
rag.ingestWithPermissions(
collectionPath = collectionPath,
documentId = "doc-1",
content = "LLM4S is a framework for building LLM-powered applications in Scala.",
metadata = Map("source" -> "readme")
)
// 5. Query with permissions
val results = rag.queryWithPermissions(
auth = UserAuthorization.Admin, // Admin bypasses all checks
collectionPattern = CollectionPattern.All,
queryText = "What is LLM4S?"
)
results.foreach { r =>
r.foreach(result => println(s"${result.id}: ${result.content.take(50)}..."))
}
// 6. Cleanup
rag.close()
searchIndex.close()
```
---
## Core Concepts
### Principal IDs
Principals represent users and groups. They're mapped to integers for efficient database queries:
(not compile-checked)
```scala
import org.llm4s.rag.permissions._
// Users have positive IDs
val userId = PrincipalId.user(42) // Creates PrincipalId(42)
userId.isUser // true
userId.isGroup // false
// Groups have negative IDs
val groupId = PrincipalId.group(5) // Creates PrincipalId(-5)
groupId.isUser // false
groupId.isGroup // true
// Create from raw value
PrincipalId.fromRaw(42) // Right(PrincipalId(42))
PrincipalId.fromRaw(-5) // Right(PrincipalId(-5))
PrincipalId.fromRaw(0) // Left(error) - zero not allowed
```
### External Principals
External principals are human-readable identifiers that get mapped to integer IDs:
(not compile-checked)
```scala
import org.llm4s.rag.permissions._
// User by email
val user = ExternalPrincipal.User("john@example.com")
user.externalId // "user:john@example.com"
// Group by name
val group = ExternalPrincipal.Group("engineering")
group.externalId // "group:engineering"
// Parse from string
ExternalPrincipal.parse("user:john@example.com") // Right(User("john@example.com"))
ExternalPrincipal.parse("group:admins") // Right(Group("admins"))
ExternalPrincipal.parse("invalid") // Left(error)
```
### Collection Paths
Collections use hierarchical paths with forward-slash separators:
(not compile-checked)
```scala
import org.llm4s.rag.permissions._
// Create with validation
val path = CollectionPath.create("confluence/EN/archive")
// Right(CollectionPath(Seq("confluence", "EN", "archive")))
// Path properties
path.foreach { p =>
p.value // "confluence/EN/archive"
p.depth // 3
p.name // "archive" (final segment)
p.isRoot // false
p.parent // Some(CollectionPath("confluence/EN"))
}
// Hierarchy relationships
val parent = CollectionPath.unsafe("confluence")
val child = CollectionPath.unsafe("confluence/EN")
val grandchild = CollectionPath.unsafe("confluence/EN/archive")
child.isChildOf(parent) // true
grandchild.isChildOf(parent) // false (not direct child)
grandchild.isDescendantOf(parent) // true
// Invalid paths are rejected
CollectionPath.create("path with spaces") // Left(error)
CollectionPath.create("") // Left(error)
```
### Collection Patterns
Patterns filter which collections to search:
(not compile-checked)
```scala
import org.llm4s.rag.permissions._
// Match all collections
CollectionPattern.All
// Match exact path
CollectionPattern.Exact(CollectionPath.unsafe("confluence/EN"))
// Match immediate children only
CollectionPattern.ImmediateChildren(CollectionPath.unsafe("confluence"))
// Matches: confluence/EN, confluence/DE
// Does NOT match: confluence/EN/archive
// Match all descendants
CollectionPattern.AllDescendants(CollectionPath.unsafe("confluence"))
// Matches: confluence, confluence/EN, confluence/EN/archive, confluence/DE
// Parse from string
CollectionPattern.parse("*") // All
CollectionPattern.parse("confluence") // Exact
CollectionPattern.parse("confluence/*") // ImmediateChildren
CollectionPattern.parse("confluence/**") // AllDescendants
```
### User Authorization
Authorization context contains the user's principal IDs:
(not compile-checked)
```scala
import org.llm4s.rag.permissions._
// Create for a user with group memberships
val auth = UserAuthorization.forUser(
userId = PrincipalId.user(42),
groups = Set(PrincipalId.group(1), PrincipalId.group(2))
)
auth.principalIds // Set(PrincipalId(42), PrincipalId(-1), PrincipalId(-2))
auth.isAdmin // false
// Admin bypasses all permission checks
val adminAuth = UserAuthorization.Admin
adminAuth.isAdmin // true
// Anonymous user (no permissions)
val anonAuth = UserAuthorization.Anonymous
anonAuth.principalIds.isEmpty // true
// Check if user includes a specific principal
auth.includes(PrincipalId.user(42)) // true
auth.includes(PrincipalId.group(1)) // true
auth.includes(PrincipalId.user(99)) // false
```
---
## Collections
### Creating Collections
Collections are created through the `CollectionStore`:
(not compile-checked)
```scala
import org.llm4s.rag.permissions._
val collections = searchIndex.collections
// Public leaf collection (anyone can query, can contain documents)
val publicConfig = CollectionConfig.publicLeaf(
CollectionPath.unsafe("public-docs")
)
collections.create(publicConfig)
// Restricted collection (only specific groups can query)
val restrictedConfig = CollectionConfig.restrictedLeaf(
path = CollectionPath.unsafe("internal"),
queryableBy = Set(PrincipalId.group(1)) // Only engineering group
)
collections.create(restrictedConfig)
// Parent collection (can have sub-collections, not documents)
val parentConfig = CollectionConfig.publicParent(
CollectionPath.unsafe("confluence")
)
collections.create(parentConfig)
// Fluent configuration
val config = CollectionConfig(CollectionPath.unsafe("my-collection"))
.withQueryableBy(PrincipalId.group(5))
.withQueryableBy(Set(PrincipalId.user(1), PrincipalId.user(2)))
.withMetadata("description", "My documents")
.asLeaf
```
### Hierarchical Structure
Collections form a tree structure:
```
SearchIndex
├── confluence (parent, queryableBy: [employees])
│ ├── confluence/EN (leaf, inherits + can restrict)
│ ├── confluence/DE (leaf, inherits)
│ └── confluence/archive (leaf, inherits)
├── internal (parent, queryableBy: [admins])
│ └── internal/hr (leaf, queryableBy: [hr-team])
└── public (leaf, queryableBy: []) ← empty = public
```
**Rules:**
- Parent collections cannot contain documents (only sub-collections)
- Leaf collections can contain documents
- When you create a child, the parent automatically becomes a non-leaf
- Permissions are inherited: children can only restrict, never loosen
(not compile-checked)
```scala
// Create parent first
collections.create(CollectionConfig.publicParent(CollectionPath.unsafe("confluence")))
// Create child - parent automatically becomes non-leaf
collections.create(CollectionConfig.publicLeaf(CollectionPath.unsafe("confluence/EN")))
// Or use ensureExists to create parent hierarchy automatically
collections.ensureExists(CollectionConfig.publicLeaf(
CollectionPath.unsafe("deep/nested/collection")
))
// Creates: deep (parent), deep/nested (parent), deep/nested/collection (leaf)
```
### Permission Inheritance
Permissions flow down the hierarchy:
(not compile-checked)
```scala
val principals = searchIndex.principals
// Create groups
val employees = principals.getOrCreate(ExternalPrincipal.Group("employees")).toOption.get
val managers = principals.getOrCreate(ExternalPrincipal.Group("managers")).toOption.get
// Parent restricted to employees
collections.create(CollectionConfig(
path = CollectionPath.unsafe("company"),
queryableBy = Set(employees),
isLeaf = false
))
// Child further restricted to managers only
collections.create(CollectionConfig(
path = CollectionPath.unsafe("company/executive"),
queryableBy = Set(managers),
isLeaf = true
))
// Get effective permissions (considers inheritance)
collections.getEffectivePermissions(CollectionPath.unsafe("company/executive"))
// Returns intersection: only managers who are also employees
// Cannot loosen permissions
collections.create(CollectionConfig(
path = CollectionPath.unsafe("company/public"),
queryableBy = Set.empty, // Trying to make public
isLeaf = true
))
// ERROR: Cannot make collection public when parent is restricted
```
### Querying Collections
(not compile-checked)
```scala
val collections = searchIndex.collections
// Get a specific collection
collections.get(CollectionPath.unsafe("confluence/EN"))
// Right(Some(Collection(...)))
// List all collections matching a pattern
collections.list(CollectionPattern.AllDescendants(CollectionPath.unsafe("confluence")))
// Seq(confluence, confluence/EN, confluence/DE, ...)
// Find accessible collections for a user
val userAuth = UserAuthorization.forUser(userId, groups = Set(employeesGroup))
collections.findAccessible(userAuth, CollectionPattern.All)
// Only returns collections the user can access
// List children of a collection
collections.listChildren(CollectionPath.unsafe("confluence"))
// Seq(confluence/EN, confluence/DE, ...)
// Check if user can query a collection
collections.canQuery(CollectionPath.unsafe("internal"), userAuth)
// Right(true) or Right(false)
// Get collection statistics
collections.stats(CollectionPath.unsafe("confluence/EN"))
// Right(CollectionStats(documentCount = 42, chunkCount = 156, subCollectionCount = 0))
```
---
## Documents
### Ingesting with Permissions
Use `ingestWithPermissions` to add documents with access control:
(not compile-checked)
```scala
// Public document (inherits collection permissions)
rag.ingestWithPermissions(
collectionPath = CollectionPath.unsafe("public-docs"),
documentId = "doc-1",
content = "This document is visible to anyone who can query the collection.",
metadata = Map("source" -> "wiki", "author" -> "john")
)
// Restricted document (only specific users can read)
rag.ingestWithPermissions(
collectionPath = CollectionPath.unsafe("public-docs"),
documentId = "confidential-doc",
content = "This is confidential information.",
metadata = Map("classification" -> "confidential"),
readableBy = Set(PrincipalId.user(42), PrincipalId.group(1))
)
```
### Two-Level Permission Model
Permissions are checked at two levels:
1. **Collection-level (`queryableBy`)**: Controls who can search the collection
2. **Document-level (`readableBy`)**: Fine-grained access within a collection
```
Collection: "hr-docs" (queryableBy: [hr-group])
├── doc-1 (readableBy: []) ← Anyone in hr-group can read
├── doc-2 (readableBy: [hr-managers]) ← Only HR managers
└── doc-3 (readableBy: [ceo]) ← Only CEO can read
```
**Query Flow:**
1. Filter collections by user's principals (`queryableBy` check)
2. Vector search within accessible collections
3. Filter results by `readableBy` check
4. Return permission-filtered results
### Deleting Documents
(not compile-checked)
```scala
// Delete a specific document
rag.deleteFromCollection(
collectionPath = CollectionPath.unsafe("public-docs"),
documentId = "doc-1"
)
// Clear all documents in a collection (via SearchIndex)
searchIndex.clearCollection(CollectionPath.unsafe("public-docs"))
```
---
## Querying
### Permission-Filtered Search
(not compile-checked)
```scala
// Query with user permissions
val userAuth = UserAuthorization.forUser(
userId = PrincipalId.user(42),
groups = Set(PrincipalId.group(1))
)
val results = rag.queryWithPermissions(
auth = userAuth,
collectionPattern = CollectionPattern.All,
queryText = "What is the vacation policy?",
topK = Some(10)
)
results.foreach { searchResults =>
searchResults.foreach { result =>
println(s"[${result.score}] ${result.id}")
println(s" Content: ${result.content.take(100)}...")
println(s" Metadata: ${result.metadata}")
}
}
```
### Query with Answer Generation
(not compile-checked)
```scala
// Requires LLM client to be configured
val rag = RAG.builder()
.withEmbeddings("openai")
.withSearchIndex(searchIndex)
.withLLM(llmClient) // Add LLM for answer generation
.build
.getOrElse(???)
val answer = rag.queryWithPermissionsAndAnswer(
auth = userAuth,
collectionPattern = CollectionPattern.AllDescendants(CollectionPath.unsafe("hr")),
question = "How many vacation days do employees get?",
topK = Some(5)
)
answer.foreach { result =>
println(s"Answer: ${result.answer}")
println(s"Based on ${result.contexts.size} sources")
}
```
### Collection Patterns in Queries
(not compile-checked)
```scala
// Search all collections
rag.queryWithPermissions(auth, CollectionPattern.All, "query")
// Search exact collection
rag.queryWithPermissions(
auth,
CollectionPattern.Exact(CollectionPath.unsafe("hr/policies")),
"query"
)
// Search immediate children of a collection
rag.queryWithPermissions(
auth,
CollectionPattern.ImmediateChildren(CollectionPath.unsafe("confluence")),
"query"
) // Searches: confluence/EN, confluence/DE, but NOT confluence/EN/archive
// Search all descendants
rag.queryWithPermissions(
auth,
CollectionPattern.AllDescendants(CollectionPath.unsafe("confluence")),
"query"
) // Searches: confluence, confluence/EN, confluence/EN/archive, etc.
```
---
## Principal Management
### Creating and Looking Up Principals
(not compile-checked)
```scala
val principals = searchIndex.principals
// Create or get existing principal
val john = principals.getOrCreate(ExternalPrincipal.User("john@example.com"))
val engineering = principals.getOrCreate(ExternalPrincipal.Group("engineering"))
// Batch create
val mapping = principals.getOrCreateBatch(Seq(
ExternalPrincipal.User("alice@example.com"),
ExternalPrincipal.User("bob@example.com"),
ExternalPrincipal.Group("hr")
))
// Lookup without creating
principals.lookup(ExternalPrincipal.User("john@example.com"))
// Right(Some(PrincipalId(42)))
principals.lookup(ExternalPrincipal.User("unknown@example.com"))
// Right(None)
// Reverse lookup
principals.getExternalId(PrincipalId(42))
// Right(Some(ExternalPrincipal.User("john@example.com")))
// List all users
principals.list("user", limit = 100, offset = 0)
// Count groups
principals.count("group")
```
---
## Schema Management
### Initializing the Schema
The schema must be initialized before use:
(not compile-checked)
```scala
val searchIndex = PgSearchIndex.fromJdbcUrl(
jdbcUrl = "jdbc:postgresql://localhost:5432/postgres",
user = "postgres",
password = "postgres",
vectorTableName = "rag_vectors"
).getOrElse(???)
// Initialize permission tables and indexes
searchIndex.initializeSchema()
```
This creates:
- `llm4s_principals` table for user/group ID mapping
- `llm4s_collections` table for collection hierarchy
- Extends your vectors table with `collection_id` and `readable_by` columns
- GIN indexes for efficient array containment queries
### Migration from Non-Permission RAG
If you have existing vectors without permissions:
(not compile-checked)
```scala
import org.llm4s.rag.permissions.pg.PgSchemaManager
import java.sql.DriverManager
val conn = DriverManager.getConnection(jdbcUrl, user, password)
// Run full migration
PgSchemaManager.runFullMigration(conn, "vectors").foreach { stats =>
println(s"Created ${stats.tablesCreated} tables")
println(s"Created ${stats.indexesCreated} indexes")
println(s"Migrated ${stats.vectorsMigrated} vectors to default collection")
}
conn.close()
```
This:
1. Creates permission tables
2. Adds `collection_id` and `readable_by` columns to your vectors table
3. Creates a "default" public collection
4. Migrates existing vectors to the default collection
---
## Complete Example
Here's a complete example showing a multi-tenant document system:
(not compile-checked)
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.model.ModelRegistryService
import org.llm4s.rag.RAG
import org.llm4s.rag.RAG.RAGConfigOps
import org.llm4s.rag.permissions._
import org.llm4s.rag.permissions.pg.PgSearchIndex
object MultiTenantRAGExample extends App {
given ModelRegistryService = Llm4sConfig.modelRegistryService().toOption.get
// 1. Setup
val searchIndex = PgSearchIndex.fromJdbcUrl(
"jdbc:postgresql://localhost:5432/postgres",
"postgres", "postgres", "vectors"
).getOrElse(throw new RuntimeException("Failed to create SearchIndex"))
searchIndex.initializeSchema()
val principals = searchIndex.principals
val collections = searchIndex.collections
// 2. Create users and groups
val john = principals.getOrCreate(ExternalPrincipal.User("john@acme.com")).toOption.get
val alice = principals.getOrCreate(ExternalPrincipal.User("alice@acme.com")).toOption.get
val engineering = principals.getOrCreate(ExternalPrincipal.Group("engineering")).toOption.get
val hr = principals.getOrCreate(ExternalPrincipal.Group("hr")).toOption.get
// 3. Create collection hierarchy
// acme (parent, all employees)
// ├── acme/engineering (engineering group)
// │ └── acme/engineering/secrets (restricted)
// └── acme/hr (hr group)
collections.create(CollectionConfig(
path = CollectionPath.unsafe("acme"),
queryableBy = Set(engineering, hr), // All employees
isLeaf = false
))
collections.create(CollectionConfig.restrictedLeaf(
CollectionPath.unsafe("acme/engineering"),
Set(engineering)
))
collections.create(CollectionConfig.restrictedLeaf(
CollectionPath.unsafe("acme/hr"),
Set(hr)
))
// 4. Build RAG
val rag = RAG.builder()
.withEmbeddings("openai")
.withSearchIndex(searchIndex)
.build
.getOrElse(throw new RuntimeException("Failed to build RAG"))
// 5. Ingest documents
rag.ingestWithPermissions(
CollectionPath.unsafe("acme/engineering"),
"api-docs",
"Our REST API uses OAuth 2.0 for authentication...",
metadata = Map("type" -> "documentation")
)
rag.ingestWithPermissions(
CollectionPath.unsafe("acme/hr"),
"vacation-policy",
"Employees receive 20 days of paid vacation per year...",
metadata = Map("type" -> "policy")
)
// Secret document only Alice can read
rag.ingestWithPermissions(
CollectionPath.unsafe("acme/hr"),
"salary-data",
"Confidential salary information...",
readableBy = Set(alice) // Only Alice can read
)
// 6. Query as different users
val johnAuth = UserAuthorization.forUser(john, Set(engineering))
val aliceAuth = UserAuthorization.forUser(alice, Set(hr))
println("=== John (Engineering) searching 'vacation' ===")
rag.queryWithPermissions(johnAuth, CollectionPattern.All, "vacation policy").foreach { results =>
if (results.isEmpty) println("No results (no access to HR docs)")
else results.foreach(r => println(s"Found: ${r.id}"))
}
println("\n=== Alice (HR) searching 'vacation' ===")
rag.queryWithPermissions(aliceAuth, CollectionPattern.All, "vacation policy").foreach { results =>
results.foreach(r => println(s"Found: ${r.id}"))
}
println("\n=== Alice searching 'salary' ===")
rag.queryWithPermissions(aliceAuth, CollectionPattern.All, "salary").foreach { results =>
results.foreach(r => println(s"Found: ${r.id} (secret doc visible to Alice)"))
}
// 7. Cleanup
rag.close()
searchIndex.close()
}
```
---
## Best Practices
### Collection Design
1. **Use meaningful hierarchy**: `{org}/{department}/{project}` or `{source}/{category}`
2. **Keep paths short**: Deep nesting increases permission check overhead
3. **Plan for growth**: Use parent collections for organizational units
4. **Document your structure**: Create a diagram of your collection hierarchy
### Permission Design
1. **Prefer groups over users**: Easier to manage at scale
2. **Use empty `queryableBy` for public**: More efficient than listing all users
3. **Reserve `readableBy` for exceptions**: Most docs should inherit from collection
4. **Validate early**: Check permissions before expensive operations
### Performance
1. **Use batch operations**: `getOrCreateBatch` for multiple principals
2. **Create indexes**: The schema manager creates GIN indexes automatically
3. **Limit pattern scope**: Use specific patterns instead of `All` when possible
4. **Monitor collection sizes**: Large collections may need partitioning
---
## API Reference
### SearchIndex Trait
(not compile-checked)
```scala
trait SearchIndex {
def principals: PrincipalStore
def collections: CollectionStore
def query(
auth: UserAuthorization,
collectionPattern: CollectionPattern,
queryVector: Array[Float],
topK: Int = 10,
additionalFilter: Option[MetadataFilter] = None
): Result[Seq[ScoredRecord]]
def ingest(
collectionPath: CollectionPath,
documentId: String,
chunks: Seq[ChunkWithEmbedding],
metadata: Map[String, String] = Map.empty,
readableBy: Set[PrincipalId] = Set.empty
): Result[Int]
def deleteDocument(collectionPath: CollectionPath, documentId: String): Result[Long]
def clearCollection(collectionPath: CollectionPath): Result[Long]
def stats(collectionPath: CollectionPath): Result[CollectionStats]
def initializeSchema(): Result[Unit]
def close(): Unit
}
```
### RAG Permission Methods
(not compile-checked)
```scala
class RAG {
def searchIndex: Option[SearchIndex]
def hasPermissions: Boolean
def queryWithPermissions(
auth: UserAuthorization,
collectionPattern: CollectionPattern,
queryText: String,
topK: Option[Int] = None
): Result[Seq[RAGSearchResult]]
def queryWithPermissionsAndAnswer(
auth: UserAuthorization,
collectionPattern: CollectionPattern,
question: String,
topK: Option[Int] = None
): Result[RAGAnswerResult]
def ingestWithPermissions(
collectionPath: CollectionPath,
documentId: String,
content: String,
metadata: Map[String, String] = Map.empty,
readableBy: Set[PrincipalId] = Set.empty
): Result[Int]
def deleteFromCollection(
collectionPath: CollectionPath,
documentId: String
): Result[Long]
}
```
---
## Troubleshooting
### Common Issues
**"Collection not found" error**
- Ensure the collection exists before ingesting
- Use `ensureExists` to create parent hierarchy automatically
**"Cannot make collection public when parent is restricted"**
- Child collections cannot have looser permissions than parents
- Either make the parent public or restrict the child to a subset of parent's principals
**"SearchIndex required" error**
- Call `.withSearchIndex(index)` when building RAG
- Ensure the SearchIndex is created and initialized
**Empty search results**
- Check that the user has access to the collection (`canQuery`)
- Verify documents have matching `readableBy` permissions
- Confirm the collection pattern includes the target collection
### Debugging Tips
(not compile-checked)
```scala
// Check user's accessible collections
collections.findAccessible(userAuth, CollectionPattern.All).foreach { accessible =>
println(s"User can access: ${accessible.map(_.path.value)}")
}
// Check effective permissions
collections.getEffectivePermissions(path).foreach { perms =>
if (perms.isEmpty) println("Collection is public")
else println(s"Required principals: ${perms.map(_.value)}")
}
// Verify collection exists and is a leaf
collections.get(path).foreach {
case Some(c) => println(s"Collection ${c.path.value}: isLeaf=${c.isLeaf}")
case None => println("Collection not found")
}
```
---
## Next Steps
- **[Vector Store Guide](vector-store)** - Learn about the underlying vector storage
- **[RAG Evaluation Guide](rag-evaluation)** - Measure and improve RAG quality
- **[Examples Gallery](/examples/#embeddings-examples)** - See more RAG examples
---
Source: https://llm4s.org/guide/error-handling
# Error Handling with Result
How to work with `Result[A]` and `LLMError` in practice.
---
LLM4S does not throw exceptions for failures it expects: a missing API key, a rate limit, a timeout
and a rejected request all come back as values. If you know Java or Python, this replaces
`try`/`catch` with something the compiler checks. This page shows how to use it day to day.
Every snippet after section 1 (which only shows definitions) is compiled and run by
[`ErrorHandlingGuideSpec`](https://github.com/llm4s/llm4s/blob/main/modules/core/src/test/scala/org/llm4s/error/ErrorHandlingGuideSpec.scala),
so the names and calls in it match the current API, and each snippet shows the imports it needs. If you
change one, change the other.
## 1. What is `Result[A]`?
```scala
type Result[+A] = Either[LLMError, A]
```
A call either succeeds with a `Right(value)` or fails with a `Left(error)`, where `error` is an
`LLMError`. There is no third outcome: nothing is thrown past you, and a `null` is never returned in
place of an error.
Two imports come up. The type alias is in `org.llm4s.types`; the helper object (`Result.success`,
`Result.traverse` and friends, used in [section 8](#8-combining-results)) is `org.llm4s.Result`:
```scala
import org.llm4s.Result // the helper object
import org.llm4s.types.{ Result, TryOps } // the type alias, and Try-to-Result syntax
```
## 2. The basic pattern
Match on the result. `Right` holds the value and `Left` holds the error:
```scala
import org.llm4s.llmconnect.model.Completion
import org.llm4s.types.Result
def basicPattern(result: Result[Completion]): String =
result match {
case Right(completion) => s"Response: ${completion.content}"
case Left(error) => s"Error: ${error.message}"
}
```
A completion's text is `completion.content`. `error.message` is for people;
`error.formatted` adds the error's type, code and context, which is what to log.
## 3. Chaining with for-comprehensions
Most programs make several calls that can each fail. A `for` comprehension runs them in order and
stops at the first `Left`, which becomes the result. The calls after it do not run:
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model.Conversation
import org.llm4s.model.ModelRegistryService
import org.llm4s.types.Result
def chained(): Result[String] =
for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
conversation <- Conversation.userOnly("What is Scala?")
completion <- client.complete(conversation)
} yield completion.content
```
Loading the configuration, building the client, building the conversation (which validates it) and
making the call are four steps that can each fail, and you handle them once, at the end.
`Llm4sConfig.defaultProvider()` reads the section `llm4s.providers.provider` names; see
[Configuration](../getting-started/configuration#named-provider-sections).
## 4. Error types and when each is raised
Every error is an `LLMError`, carrying a `message`, an optional `code` and a `context` map. The types in
`org.llm4s.error` are each also marked as one of two kinds:
- **Recoverable** (`RecoverableError`): the call may succeed if tried again, perhaps after you do
something first (wait, re-read a record, correct the request).
- **Non-recoverable** (`NonRecoverableError`): trying again will not help; something has to change.
`LLMError.isRecoverable(error)` tells you which. It answers for every error: one that carries neither marker,
as some errors from other modules do (see [errors defined by other modules](#errors-defined-by-other-modules)),
is not recoverable, since nothing says a retry can help.
| Error | Recoverable | Where the library raises it |
|---|---|---|
| `AuthenticationError` | no | A provider rejects the credentials: HTTP 401 or 403, Anthropic's `UnauthorizedException`, Bedrock's access-denied or missing-credentials exceptions, Gemini's invalid-key 400. Also when Vertex AI or watsonx cannot obtain a token, and from `DefaultErrorMapper` for an exception whose message names HTTP status 401 (`HTTP 401`, `401 Unauthorized`, `status code: 401`). |
| `ConfigurationError` | no | Configuration is missing or invalid: no provider section, no API key, a provider id that no module on the classpath registers, an embedding model with no known dimensions, or a block a config loader cannot read (providers, embeddings, tracing, metrics, tools, RAG, speech). |
| `ValidationError` | no | A request or value is rejected: an invalid message or conversation, a model the model registry does not know, a guardrail rejecting input or output (the built-in guardrails reject with one), HTTP 400 (on field `request`), Anthropic's or Bedrock's invalid-request exceptions, or a response body that cannot be parsed. |
| `InvalidInputError` | no | Only `llm4s-image`'s image processing (`LocalImageProcessor`, saving an image): an unreadable path, a bad resize or crop, a path traversal. It carries the `field`, the `value` and the `reason`. |
| `RateLimitError` | yes | A provider's rate limit: HTTP 429, or Anthropic's or Bedrock's throttling exception. Also `ReliableClient`'s own limiter, and `DefaultErrorMapper` for an exception whose message names HTTP status 429 (`HTTP 429`, `429 Too Many Requests`, `status code: 429`). It carries `retryAfter` when the provider says how long to wait. |
| `ServiceError` | yes | Any other non-2xx status from a provider (through `HttpErrorMapper` or `Llm4sHttpClient`, and from Bedrock and watsonx), or a call rejected by an open circuit breaker (`ReliableClient` or `ErrorRecovery.CircuitBreaker`, status 503). It carries `httpStatus`; see the note below. |
| `NetworkError` | yes | A connection fails, a host is unknown or I/O breaks, in llm4s's HTTP client or the Bedrock client; `DefaultErrorMapper`, which the OpenAI and Anthropic clients use for I/O failures, gives one for a socket timeout or a refused connection. Also a URL refused by the SSRF check, and a failed `llm4s-rag` URL, web-crawl or S3 load, including a non-2xx answer to `UrlLoader`. |
| `TimeoutError` | yes | A connect, request or socket timeout elapses in llm4s's own HTTP client, or a `ReliableClient` deadline passes. The vendor-SDK clients map timeouts themselves: OpenAI and Bedrock report a `NetworkError`, and Anthropic maps its exceptions through `DefaultErrorMapper`. |
| `APIError` | yes | Only `llm4s-image`'s vision clients (OpenAI, Anthropic, Gemini), through `LLMError.apiCallFailed`, when the vision API call fails or returns no text; it carries the provider and, optionally, a status code. Image generation has its own errors (see below). |
| `ExecutionError` | yes | Only `ErrorRecovery.recoverWithBackoff`, when it runs out of attempts ([section 9](#9-recovering-from-failures)). Tool and MCP failures are other types; see the note below. |
| `SystemError` | yes | Not raised by the library; available for your own code, for an unexpected failure that may be transient. |
| `OptimisticLockFailure` | yes | Only `llm4s-memory-postgres`'s `PostgresMemoryStore`, when another writer updated the same memory record first; re-read it and try again. |
| `CancelledError` | no | The thread was interrupted. Interruption is how llm4s cancels work, and a cancelled call is never retried. |
| `ProcessingError` | no | A storage, parsing or processing step fails: a vector-store, keyword-index or memory-store operation, RAG document loading, extraction or permissions, knowledge-graph storage, extraction or queries, speech audio processing, image encoding or saving. |
| `NotFoundError` | no | A memory store (in-memory, SQLite, Postgres) is asked to update a memory it does not hold, or `VectorStoreFactory` is given an unknown backend name. |
| `ContextError` | no | Context compression fails: `LLMCompressor`'s LLM call fails, or `ToolOutputCompressor` cannot store an artifact or parse JSON content. |
| `TokenizerError` | no | `ConversationTokenCounter` has no tokenizer for the requested id. |
| `SimpleError` | no | The MCP client and its transports (`llm4s-mcp`): connection, session, JSON-RPC and server-process failures. |
| `UnknownError` | no | An unexpected exception, wrapped: the fallback of `DefaultErrorMapper` (and so of `toResult` and `toLLMError`), an unexpected failure in llm4s's HTTP client, or a tracing backend that fails to start or to export. |
The table lists the `LLMError` types in `org.llm4s.error`, the package that is the source of truth for this core
set. Other modules define errors of their own, below.
**Same name, different type.** A failed tool call is not an `org.llm4s.error.ExecutionError`.
`ToolRegistry.execute`, and the MCP tool registry, return a `ToolCallError` (`org.llm4s.toolapi`), whose
case for a tool that threw is `ToolCallError.ExecutionError`. `ToolCallError` is not an `LLMError`, and the
agent hands it back to the model as the tool's result instead of failing the run. A graph's tool loop fails
with `GraphError.ToolFailed`, described below.
**`ServiceError` and its status.** The marker says a `ServiceError` is recoverable, but a 404 is not
going to fix itself. When it matters, look at `httpStatus`: `error.isRecoverableStatus` (a method of
`ServiceError`, no import needed) is true for 5xx, 429 and 408. The library's automatic retries retry a
`ServiceError` only when it is; see the next paragraph.
**Recoverable is not the same as retried automatically.** Every retry in the library -
`recoverWithBackoff`, `ReliableClient`'s `RetryPolicy`, `LLMClientRetry` and the agent graph's default node
retry - uses one rule, `RetryPolicy.isRetryable`: it resends the identical request with nobody in the loop,
so it retries a recoverable error unless repeating the request cannot help. That leaves out two recoverable
cases: a `ServiceError` or an `APIError` with a client-error status (any 4xx but 408 and 429), because the
request itself is wrong, and an `OptimisticLockFailure`, because you must re-read the record first. An
`APIError` with no status is retried. A non-recoverable error is never retried, and neither is an error that
carries no marker.
**Which status becomes which error.** The status mapping in the table is `HttpErrorMapper`'s: 401 and
403 to `AuthenticationError`, 429 to `RateLimitError`, 400 to `ValidationError`, any other non-2xx to
`ServiceError`. The OpenAI, Azure, Requesty, Gemini, Vertex AI, Ollama, watsonx and OpenAI-compatible
clients use it; Gemini first turns a 400 that reports an invalid API key into an `AuthenticationError`.
The Anthropic client maps its SDK's exceptions instead: unauthorized (401) to `AuthenticationError`, rate
limited to `RateLimitError` (with no `retryAfter`), invalid data to a `ValidationError` on field `input`,
and anything else, including other HTTP statuses, through `DefaultErrorMapper`
([section 7](#7-turning-exceptions-into-errors)), usually to an `UnknownError`. The Bedrock client maps the
AWS SDK's exceptions: throttling or an exceeded quota to `RateLimitError`, an invalid request to
`ValidationError`, access denied, a 401 or 403, or missing credentials to `AuthenticationError`, any other
service status to `ServiceError`, and any other client failure to `NetworkError`.
**The next step, in the message.** A 401 or 403, a 429 or a 404 from `HttpErrorMapper` or the Anthropic client ends
its message with one sentence saying what to do and a link to its entry in
[First-Run Troubleshooting](/getting-started/troubleshooting), for example
`... See https://llm4s.org/getting-started/troubleshooting#rate-limited`. The first-run configuration errors (a
missing API key, no default provider, an unknown section, a provider that is not registered, an unreadable
`application.conf`) end with their link and the [doctor](/reference/doctor)'s command. Only the text changes: match on
the error's type and fields, never on its message.
### Errors defined by other modules
Some modules add their own `LLMError` subtypes. The errors below carry **neither** marker, so
`LLMError.isRecoverable` reports them as not recoverable and the library's automatic retries leave them alone.
For each, the right answer depends on more than the type (a status inside a string `code`, a flag, a wrapped
cause, which of several failures it reports), so none carries a marker:
| Module | Package | Errors |
|---|---|---|
| `llm4s-core` | `org.llm4s.llmconnect.model` | `EmbeddingError`: how `EmbeddingClient.embed` and the embedding providers (OpenAI, Ollama, Voyage, Cohere, Jina) report a failure other than cancellation, except that the Cohere provider reports a 429 as a `RateLimitError` |
| `llm4s-rag` | `org.llm4s.rag.evaluation`, `org.llm4s.reranker` | `EvaluationError` from RAGAS evaluation and the RAG benchmark tools; `RerankError` from the Cohere and LLM rerankers (`Reranker.rerank`) |
| `llm4s-speech` | `org.llm4s.speech.stt`, `.tts` | `STTError.ProcessingFailed` from the speech-to-text clients (an empty transcription, no recognisable speech, an unparseable response; its `retryable` flag says `true`); `TTSError.SynthesisFailed` from the text-to-speech clients (an empty audio body) |
The rest of those families are marked:
| Module | Recoverable | Non-recoverable |
|---|---|---|
| `llm4s-speech` (`org.llm4s.speech.stt`, `.tts`, `.io`) | `STTError.EngineNotAvailable`, `TTSError.EngineNotAvailable` | `STTError.UnsupportedFormat`, `STTError.InvalidInput`, `WavFileGenerator.WavError`, `AudioIO.AudioIOError` |
So are two whole families: `GraphError` in `llm4s-agent`
(`org.llm4s.agent.graph`, from the graph runtime; every case is non-recoverable except `DeadlineExceeded` and
`ThreadBusy`)
and `ImageGenerationError` in `llm4s-image` (`org.llm4s.imagegeneration`, from the image-generation
clients; a `ServiceError` there is recoverable only for a transient status). The image module reuses the
names `AuthenticationError`, `RateLimitError`, `ServiceError`, `ValidationError` and `UnknownError`, so
import those by package instead of with a wildcard next to `org.llm4s.error._`. Its vision clients
(`org.llm4s.imageprocessing`) return the core errors in the table above.
## 5. Handling specific error types
Match on the type to react differently to each failure. Put the specific cases first and finish with
a catch-all:
```scala
import org.llm4s.error._
import org.llm4s.types.Result
def describe(result: Result[String]): String =
result match {
case Right(text) => s"ok: $text"
case Left(e: RateLimitError) =>
s"wait ${e.retryDelay.getOrElse(RateLimitError.DefaultRetryDelay)}, then retry"
case Left(e: AuthenticationError) => s"fix the credentials for ${e.provider}"
case Left(e: RecoverableError) => s"transient, may succeed on retry: ${e.message}"
case Left(e) => s"not retried (permanent, or not marked either way): ${e.message}"
}
```
`RateLimitError.retryDelay` is the delay the provider asked for, or 30 seconds when it did not say.
Two things to know:
- **`LLMError` is not sealed**, so the compiler cannot tell you that a match is complete. Always end
with a `case Left(e)`.
- **A custom error must say what kind it is.** If you define your own error type, mix in
`RecoverableError` or `NonRecoverableError` as well as `LLMError`.
`LLMError.isRecoverable` reports a type that is neither as not recoverable, so a transient failure that
forgets its marker is never retried.
```scala
import org.llm4s.error.{ LLMError, NonRecoverableError }
final case class VendorError(message: String) extends LLMError with NonRecoverableError
```
## 6. Converting to exceptions (when you must)
Sometimes a framework expects an exception: a test setup, a `main` that should crash on bad
configuration, an API you do not control. Convert at the edge, in one place, and keep the error's
details:
```scala
import org.llm4s.types.Result
def orThrow[A](result: Result[A]): A =
result.fold(error => throw new RuntimeException(error.formatted), identity)
```
`error.formatted` carries the error's type and context, so the stack trace says what went wrong.
Prefer `fold` to `result.getOrElse(throw new RuntimeException("LLM call failed"))`. The second
compiles, but it discards the error, so the exception cannot say what went wrong. And do not use
`result.getOrElse(default)` unless a silent fallback is what you want: a failure becomes the default
with no trace.
Do not throw from library code you write: return a `Result` and let the caller decide.
## 7. Turning exceptions into errors
The other direction matters just as much, because the JDK and other libraries throw. Wrap them at
the boundary so the rest of your code only sees `Result`:
```scala
import org.llm4s.Result
import org.llm4s.error.NotFoundError
import org.llm4s.error.ThrowableOps._
import org.llm4s.types.{ OptionOps, TryOps }
import scala.util.Try
Try("123".toInt).toResult // Right(123)
Try("abc".toInt).toResult // Left(...), the exception mapped to an LLMError
Result.safely(1 / 0) // Left(...), a throwing block captured
new IllegalStateException("boom").toLLMError // a Throwable as an LLMError
Option.empty[String].toResult(NotFoundError("no such key", "model")) // Left(NotFoundError)
```
`toResult` and `toLLMError` map an exception through `DefaultErrorMapper`: an interrupt becomes a
`CancelledError`, a socket timeout or a refused connection a `NetworkError`, an exception whose message
names HTTP status 401 or 429 an `AuthenticationError` (code `401`, with the redacted message) or a
`RateLimitError`, and anything else an `UnknownError` that keeps the exception. A message names a status
only in an HTTP context, at a word boundary - after `HTTP`, `status`, `status code`, `error code` or
`response code`, before its reason phrase (`401 Unauthorized`, `429 Too Many Requests`), or leading the
message as `openai-java` and `anthropic-java` write it (`401: `) - so `Index 4012 out of bounds` or
`port 14290` is an `UnknownError`. A JSON-style `"status": 429` or `http_status=401` counts too. The SDK
shape has no other HTTP marker, so a message that merely *starts* with `401: ` or `429: ` is classified as
well. Only the first 4 KiB and the last 1 KiB of a message are scanned. Pass your own `ErrorMapper` for a finer mapping. `Try` (and so
`Result.safely`) never captures an `InterruptedException`, which Scala treats as fatal, so let an
interrupt propagate or map it yourself with `toLLMError`.
To make an error yourself, use the type's smart constructor:
```scala
import org.llm4s.error.{ ConfigurationError, NotFoundError, ValidationError }
ValidationError("model", "must not be empty")
ConfigurationError("no provider configured", List("llm4s.providers.provider"))
NotFoundError("no such key", "model")
```
## 8. Combining results
When you have a list of things to do, `Result` has helpers so you do not write the loop:
```scala
import org.llm4s.Result
import org.llm4s.types.{ Result, TryOps }
import scala.util.Try
def parseAll(inputs: List[String]): Result[List[Int]] =
Result.traverse(inputs)(s => Try(s.toInt).toResult)
parseAll(List("1", "2", "3")) // Right(List(1, 2, 3))
parseAll(List("1", "x", "y")) // Left(...), the first failure
```
- `Result.traverse` stops at the first failure: it does not call the function on the elements after
it, so side effects and expensive work stop there too. `Result.sequence` returns the first failure
in a list of results that have already been computed.
- `Result.validateAll(items)(check)` runs every check and returns **all** the failures as a
`Left(List[LLMError])`: use it when you want to report every problem at once.
- `Result.combine(a, b)` joins two independent results into a tuple.
## 9. Recovering from failures
A recoverable error is worth a retry. `ErrorRecovery.recoverWithBackoff` calls an operation up to
`maxAttempts` times in all, waiting between attempts. Here `client` is any `LLMClient` and `conversation` a `Conversation`, built as in
[section 3](#3-chaining-with-for-comprehensions):
```scala
import org.llm4s.error.ErrorRecovery
import scala.concurrent.duration._
val result = ErrorRecovery.recoverWithBackoff(
() => client.complete(conversation),
maxAttempts = 3,
baseDelay = 1.second
)
```
It retries what every retry in the library retries (`RetryPolicy.isRetryable`; see
[section 4](#4-error-types-and-when-each-is-raised)), each error on its own schedule. The delays are not
exponential:
| Error | Wait before the next attempt |
|---|---|
| `RateLimitError` | Its `retryDelay`: the provider's `retryAfter` when it gave one, else 30 seconds (`RateLimitError.DefaultRetryDelay`). `baseDelay` is not used. |
| `ServiceError` with a 5xx, 429 or 408 status | The provider's `retryAfter` when it gave one, else `baseDelay` times the number of the attempt that failed: `baseDelay`, then `2 * baseDelay`, and so on. |
| `TimeoutError` | `baseDelay`, every time. |
| `NetworkError`, `ExecutionError`, `SystemError`, and an `APIError` with no status or a 5xx, 429 or 408 one | `baseDelay` times the number of the attempt that failed, as for a `ServiceError` without a hint. |
Every other error comes back unchanged straight away, whichever attempt it happens on: a
`ValidationError` on the last attempt, after a retried timeout, is still a `ValidationError`. That
includes a `ServiceError` or an `APIError` with any other status and an `OptimisticLockFailure`. A
`CancelledError` is never retried or wrapped, and an interrupt during a wait returns one.
When the last attempt fails with a retried error, you get an `ExecutionError` whose
message gives the number of attempts and the last error's message, and whose `operation` is the last
error's `formatted` text; the original error's type is not kept. `ExecutionError` is itself a
`RecoverableError` that the rule retries, so an outer `recoverWithBackoff` or `RetryPolicy` will retry it. The operation
always runs at least once, even if `maxAttempts` is below 1.
To stop calling a service that keeps failing, wrap the call in an `ErrorRecovery.CircuitBreaker`.
After `failureThreshold` consecutive failures (a success resets the count) it opens, and further calls fail fast with a `ServiceError`
without reaching the service, until `recoveryTimeout` has passed and a single probe call is allowed.
For production use, `ReliableClient` wraps a client with retry (its own `RetryPolicy`, with
configurable backoff), a circuit breaker, and an optional deadline and rate limit, in one place. See [Error Recovery](patterns/error-recovery) for retry strategies, fallbacks and
graceful degradation.
## 10. Testing code that returns Result
Check the `Right` and the `Left` the same way you would any value. With ScalaTest's `EitherValues`,
`.value` unwraps a `Right` and `.left.value` unwraps a `Left`, failing the test with a clear message
if it is the other one:
```scala
import org.llm4s.error.ValidationError
import org.llm4s.types.Result
import org.scalatest.EitherValues
import org.scalatest.matchers.should.Matchers
import org.scalatest.wordspec.AnyWordSpec
class ResultSpec extends AnyWordSpec with Matchers with EitherValues {
"a Result" should {
"unwrap with EitherValues, and check the error type" in {
val ok: Result[String] = Right("expected")
val failed: Result[String] = Left(ValidationError("model", "empty"))
ok.value shouldBe "expected"
ok.map(_.toUpperCase) shouldBe Right("EXPECTED")
failed.left.value shouldBe a[ValidationError]
failed.left.value.message should include("empty")
}
}
}
```
To test code that calls an LLM without a network, give it a client that returns what you choose.
See the [Testing Guide](../getting-started/testing-guide).
## Rules of thumb
- Return `Result` from your own functions; do not throw.
- Convert exceptions to errors where they enter your code, and errors to exceptions only where a
framework forces you to, in one place.
- Match on specific types first, then on `RecoverableError`, then a catch-all. `LLMError.isRecoverable`
asks the same question of any error.
- Retry only what is recoverable, with a limit and a delay; never retry a `CancelledError`. The
library's own retries also leave out a client-error status and an `OptimisticLockFailure`.
- Log `error.formatted`, show `error.message`, and never put an API key in either.
## See also
- [Basic Usage](basic-usage): your first calls and the `Result` type
- [Error Recovery](patterns/error-recovery): retries, circuit breakers and fallbacks
- [Configuration](../getting-started/configuration): named provider sections and API keys
- [Testing Guide](../getting-started/testing-guide): testing with mock clients
---
Source: https://llm4s.org/guide/structured-output
# Structured Output with completeStructured
Ask a model for a typed value instead of free text, and know what you get back when it does not comply.
---
## 1. What it does
`LLMClient.completeStructured[A]` sends a conversation, asks the provider for JSON that matches a schema,
and returns the reply as a value of type `A`:
(not compile-checked)
```scala
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model.{ Conversation, UserMessage }
import org.llm4s.toolapi.Schema
import org.llm4s.types.Result
import upickle.default.{ macroRW, ReadWriter }
final case class Invoice(vendor: String, amount: Double, currency: String)
object Invoice { implicit val rw: ReadWriter[Invoice] = macroRW }
val invoiceSchema =
Schema
.`object`[Invoice]("An invoice extracted from text")
.withRequiredField("vendor", Schema.string("Name of the vendor or supplier"))
.withRequiredField("amount", Schema.number("Total invoice amount as a decimal number"))
.withRequiredField("currency", Schema.string("ISO 4217 currency code, e.g. USD, EUR, GBP"))
def extractInvoice(client: LLMClient, text: String): Result[Invoice] =
client.completeStructured[Invoice](
Conversation(Seq(UserMessage(s"Extract the invoice from this text:\n$text"))),
invoiceSchema
)
```
The snippets below build on these definitions (`Invoice`, `invoiceSchema` and `extractInvoice`) and import
what else they use.
The result is a `Result[Invoice]`: `Right(Invoice("Acme Supplies Ltd", 1250.0, "GBP"))` for a reply of
`{"vendor":"Acme Supplies Ltd","amount":1250.0,"currency":"GBP"}`. Other replies that can be deserialised
as `Invoice` also return `Right`; replies that cannot be deserialised return `Left` (section 7).
A runnable version that builds the client from configuration is `StructuredOutputExample`:
```bash
sbt "samples/runMain org.llm4s.samples.basic.StructuredOutputExample"
```
Use it for extraction, classification, routing and any step whose output a program, not a person, reads.
Streaming is not part of it: `completeStructured` makes one non-streaming `complete` call.
## 2. Describing the type
Two things describe `A`, and they have to agree:
- **The schema**, an `ObjectSchema[A]` built with `Schema.\`object\``, `withRequiredField` and `withProperty`.
It is what the provider is shown. The same builders define tool parameters; see
[Agents and tools](agents/).
- **A uPickle reader**, an implicit `upickle.default.Reader[A]` (a `ReadWriter` from `macroRW` is the usual way).
It is what turns the reply into `A`.
Nothing checks that the two match, and some mismatches are never reported. A field the case class has but
the schema lacks fails to parse when the reply omits it (section 7), unless the field has a default, which
the reader fills in silently. A field the schema has but the case class lacks is worse: the provider returns
it, the reader ignores it as an extra key, and the result is `Right(A)` with that value dropped. Keep the
schema and the case class in step by hand, and test the pair on a sample reply.
The schema is a closed object: `additionalProperties` is `false` unless you pass `true` to `ObjectSchema`.
`llm4s-schema-derivation` removes the need to keep the two in step: `client.completeStructuredOf[A](conversation)`
derives the schema from `A` (`derives SchemaOf`); see [Derived Schemas](derived-schemas).
### Sum types
When the answer is one of several shapes ("refund this amount", "escalate to this team" or "reply with this
text"), make `A` an enum or sealed trait whose cases have fields, deriving `SchemaOf` and `ReadWriter`, and call
`completeStructuredOf[A]`. The derived schema is a JSON Schema `anyOf` of one object per case, each tagged with
the discriminator uPickle writes (`"$type"`, or the trait's `@upickle.implicits.key`), so the reply reads back into
the right case. Because OpenAI's strict mode refuses an `anyOf` at the root, the request asks for
`{"value": }` and the reply is unwrapped before it is read; the caller only sees `A`. The request is a
`ResponseFormat.JsonSchema` that core's `ObjectSchema` could not express, sent the same way to every provider
(section 8). [Derived Schemas: Sum types](derived-schemas#sum-types) has the example, compiled, and the exact
shape of the schema.
## 3. Fenced and prose-wrapped replies
Models often wrap JSON in a markdown fence or write a sentence around it. Before parsing, the reply is
normalised: a surrounding code fence is stripped, and if the rest is still not JSON, the first balanced
`{...}` or `[...]` that parses as JSON is taken from it. Both of these give the same `Invoice`:
````text
```json
{"vendor":"Acme Supplies Ltd","amount":1250.0,"currency":"GBP"}
```
Sure! Here is the invoice you asked for: {"vendor":"Acme Supplies Ltd","amount":1250.0,"currency":"GBP"}. Let me
know if you need anything else.
````
The recovery is bounded: it tries at most 16 places where a `{` or `[` starts, so a long reply full of braces
that are not JSON is not searched forever. If it finds nothing, the trimmed text goes on to the parser, which
reports the failure.
That function, `LLMClient.extractJson`, is internal to llm4s (`private[llmconnect]`). You cannot call it, and
code that calls `complete` and parses the reply itself gets no recovery (section 6).
## 4. What is sent, and why every property is required
`completeStructured` turns your schema into JSON Schema with `strict = true`, wraps it in
`ResponseFormat.JsonSchema(schema)` and sets that as the `responseFormat` of the call.
`strict = true` lists **every** property under `required`, including a property you declared optional:
```scala
import org.llm4s.toolapi.Schema
val note =
Schema
.`object`[Map[String, String]]("A note")
.withRequiredField("vendor", Schema.string("Vendor"))
.withProperty(Schema.property("note", Schema.string("Free-text note"), required = false))
note.toJsonSchema(strict = true)("required") // ["vendor", "note"]
note.toJsonSchema(strict = false)("required") // ["vendor"]
```
So `required = false` does nothing through `completeStructured`: the provider is told `note` is required, and a
model that follows the schema always includes it. If you need a value that may be absent, make the field's
type allow it (`Schema.nullable(...)`, with an `Option` in the case class) instead of leaving the field out.
The `name` and `strict` fields of the format are the defaults, `"response"` and `true`. They are not parameters
of `completeStructured`; section 6 shows how to set them.
## 5. Options
`completeStructured` keeps every option you pass and replaces only `responseFormat`:
(not compile-checked)
```scala
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model.{ CompletionOptions, Conversation, UserMessage }
import org.llm4s.types.Result
def deterministic(client: LLMClient, text: String): Result[Invoice] =
client.completeStructured[Invoice](
Conversation(Seq(UserMessage(text))),
invoiceSchema,
CompletionOptions().withTemperature(0.0).withMaxTokens(256)
)
```
Temperature, token limit and the rest are passed to `complete` as given. The provider client may still
adjust them for the model, as it does for any `complete` call: OpenAI reasoning models, for example, omit
sampling parameters, and an option the model does not support can be dropped. A `responseFormat` you set
yourself is overwritten by the schema's, so `completeStructured` is not a way to request plain JSON mode; call `complete`
with `ResponseFormat.Json` for that.
## 6. Choosing the schema name or strictness
For a name other than `"response"`, or `strict = false`, call `complete` with your own format and read the
reply yourself:
(not compile-checked)
```scala
import org.llm4s.error.ValidationError
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model.{ CompletionOptions, Conversation, ResponseFormat, UserMessage }
import org.llm4s.types.Result
def withCustomFormat(client: LLMClient, text: String): Result[Invoice] = {
val format = ResponseFormat.JsonSchema(
invoiceSchema.toJsonSchema(strict = false),
name = "invoice",
strict = false
)
for {
completion <- client.complete(Conversation(Seq(UserMessage(text))), CompletionOptions().withResponseFormat(format))
// No fence or prose recovery on this path: the reply has to be bare JSON.
invoice <- scala.util.Try(upickle.default.read[Invoice](completion.content)).toEither.left.map(e =>
ValidationError("invoice", e.getMessage)
)
} yield invoice
}
```
`name` and `strict` only mean something to providers that have an equivalent (section 8). Where there is none,
as with Ollama, they are ignored.
## 7. When the reply is not usable
A reply that cannot become an `A` is a `ValidationError` whose `field` is `"structured_output"`:
| The reply | `error.message` contains |
|---|---|
| text with no JSON in it | `Response is not valid JSON` |
| the JSON value `null` | `Response does not match expected schema: got JSON null` |
| JSON of the wrong shape (an array for an object, a missing field, a wrong type) | `Response does not match expected schema` |
The reply is deserialised, not validated against the schema. A missing field or a wrong type fails because
the uPickle reader for `A` fails on it, but a constraint the reader does not check is not checked at all: an
enum, a numeric or string bound, or `additionalProperties = false` (extra keys are ignored). A reply that
breaks one of those still comes back as `Right(A)`, which matters most where the provider does not enforce
the schema (section 8). Check such constraints on the result yourself.
A failed provider call is a different error, returned unchanged (a `NetworkError`, `RateLimitError` and so on;
see [Error Handling](error-handling)). Telling the two apart is a pattern match:
(not compile-checked)
```scala
import org.llm4s.error.ValidationError
import org.llm4s.types.Result
def describe(result: Result[Invoice]): String = result match {
case Right(invoice) => s"ok: ${invoice.vendor}"
case Left(error: ValidationError) if error.field == "structured_output" => s"model reply rejected: ${error.message}"
case Left(error) => s"call failed: ${error.message}"
}
```
A rejected reply is often worth one retry. A failed call is handled by whatever retry you already use
(a `ReliableClient` retries transient provider errors, but never a `ValidationError`):
(not compile-checked)
```scala
import org.llm4s.error.ValidationError
import org.llm4s.llmconnect.LLMClient
import org.llm4s.types.Result
def retryOnce(client: LLMClient, text: String): Result[Invoice] =
extractInvoice(client, text) match {
case Left(error: ValidationError) if error.field == "structured_output" => extractInvoice(client, text)
case other => other
}
```
`ValidationError`s are not marked recoverable on their own, so decide per call site whether a retry is
reasonable. A model that refuses, or answers in prose again, will fail again.
## 8. What each provider does with the schema
The format is a request, not a guarantee. This is what each client does with `ResponseFormat.JsonSchema`,
taken from its code:
| Provider | What is sent | Enforced? |
|---|---|---|
| OpenAI, Azure OpenAI | `response_format` of type `json_schema`, with your `name` and `strict` | Yes: the provider constrains generation to the schema |
| Requesty | The same OpenAI `json_schema` shape, with your `name` and `strict` | Depends on the routed model: Requesty is a multi-provider router, so enforcement is up to the backend model or fallback it selects |
| openai-compatible: generic, DeepSeek, Z.ai, OpenRouter, Mistral | The same OpenAI `json_schema` shape, with your `name` and `strict` | The server decides: a server that ignores `response_format` returns unconstrained text |
| Cohere (through its OpenAI-compatibility API) | `response_format` of type `json_object` carrying the `schema`; `name` and `strict` are not sent | Cohere's own mechanism |
| Gemini, Vertex AI | `responseMimeType: application/json` and `responseSchema` set to the schema | Yes, by Gemini |
| Ollama | The schema as the top-level `format` field of `/api/chat` | Yes, on Ollama 0.5 or later; an older server rejects or ignores it. `name` and `strict` are ignored |
| Anthropic | **Nothing is enforced.** The schema is appended to the system prompt as an instruction to answer with JSON only | **No**: best effort. Expect section 7 errors more often |
| watsonx (Beta) | `response_format` of type `json_schema`, with your `name` and `strict` (`ResponseFormat.Json` as `json_object`) | Depends on the model; not yet confirmed against the live service ([#1314](https://github.com/llm4s/llm4s/issues/1314)) |
| AWS Bedrock | **Nothing.** This client does not read `responseFormat` | **No**: the model sees only your conversation |
For the last row, `completeStructured` still parses whatever comes back, so a model that happens to answer
in JSON works and one that does not gives a `ValidationError`. Ask for JSON in the prompt yourself when you
use that provider.
## 9. Limits and gotchas
- **The model registry can switch the schema off.** The OpenAI, Anthropic, Gemini, Vertex AI and Bedrock
clients check the options against the model registry first. When the registry records that a model does not
support response schemas, the schema is dropped without an error (and with it Anthropic's prompt
instruction), and `completeStructured` then parses an unconstrained reply.
- **Every property is required** (section 4), whatever you declared.
- **Integers above 2^53.** A uPickle `Long` field reads a number such as 9007199254740993 exactly, because
`completeStructured` reads `A` from the reply's text. A reader that starts from `ujson.Value` (the circe,
play-json and zio-json bridges) and every tool argument get the rounded double instead, so send large identifiers as
strings there; see [Numbers](json-libraries#numbers).
- **One call, no streaming, no repair.** There is no automatic retry inside `completeStructured`.
- **The content goes to the provider.** The conversation and the schema are sent to whichever provider the client
talks to, like any other completion.
- **The schema is JSON Schema, not your case class.** A type uPickle can read but `ObjectSchema` cannot describe
(a map with free keys, a recursive type) needs a schema that matches what the model should produce, and a
reader that accepts it. A sum type is described by `completeStructuredOf` ([Sum types](#sum-types)).
## Related
- [Error Handling](error-handling): every `LLMError`, and how to combine results.
- [Providers](providers): configuring each provider.
- `StructuredOutputExample` in `modules/samples`: the runnable version of section 1.
---
Source: https://llm4s.org/guide/spring-boot
# Using LLM4S with Spring Boot
Add `llm4s-spring-boot-starter` and inject an `LLM4STemplate`: the client, an executor for asynchronous calls and a health indicator are configured from your `application.properties`.
---
## What the starter gives you
The starter is Spring Boot auto-configuration on top of the [Java API](java). When it is on the classpath it registers:
| Bean | What it is | Replace it by |
|---|---|---|
| `llm4sClient` | the `JLlmClient`, built from the `llm4s.*` properties; closed when the context closes | defining a bean of type `JLlmClient` |
| `llm4sTaskExecutor` | the `ExecutorService` behind `completeAsync` and the health probe | defining a bean named `llm4sTaskExecutor` |
| `llm4sTemplate` | the `LLM4STemplate` you inject | defining a bean of type `LLM4STemplate` |
| `llmHealthIndicator` | an Actuator `HealthIndicator`, only when Spring Boot Actuator is on the classpath | defining a bean of type `LlmHealthIndicator` (`LlmBoot4HealthIndicator` on Spring Boot 4) |
| `llm4sMetricsCollector` | a `MicrometerMetrics` over your `MeterRegistry`, only when `llm4s-observability-micrometer` is on the classpath and there is a `MeterRegistry` bean; the client reports every call to it (see [Metrics](#metrics)) | defining a bean of type `MetricsCollector` |
Set `llm4s.enabled=false` to switch all of them off; the beans are also registered when `llm4s.enabled` is absent.
**Spring Boot versions.** One artifact supports Spring Boot 3.x from 3.3 on and Spring Boot 4.x from 4.0 on. It is
built against 3.3.6, and CI runs its whole test suite again against the latest 3.x release (3.5.16), against the 4.0
line (4.0.8) and against the latest 4.x release (4.1.1), so both ends of each range are tested on every change; the
Spring Boot 4 runs test the published classes, not a Boot 4 build of them. The one difference is the health indicator:
Spring Boot 4 moved `HealthIndicator` to `org.springframework.boot.health.contributor`, so there the bean
`llmHealthIndicator` is an `LlmBoot4HealthIndicator`, reporting exactly what `LlmHealthIndicator` reports on 3.x.
> `llm4s-spring-boot-starter` is on Maven Central from `0.5.0` on, together with the Java API it is built on; the
> current release is 0.5.1. It needs **JDK 21**, like the Java API (see
> [Scala and JDK support](/reference/v1-scope#scala-and-jdk-support)); whether some artifacts get a lower floor before
> 1.0 is tracked in [#1493](https://github.com/llm4s/llm4s/issues/1493).
## Add the dependency
The artifact is `org.llm4s:llm4s-spring-boot-starter_3`. The `_3` is the Scala binary version (LLM4S is Scala 3 only), and
Maven and Gradle need it spelled out.
**Maven**
```xml
org.llm4s
llm4s-spring-boot-starter_3
0.5.1
```
**Gradle (Kotlin DSL)**
```kotlin
dependencies {
implementation("org.llm4s:llm4s-spring-boot-starter_3:0.5.1")
}
```
It brings the Java API, so the [dependency notes of the Java guide](java#add-the-dependency) apply: the provider modules,
Scala's standard library, and no logging backend. For the health indicator, add `spring-boot-starter-actuator` yourself:
the starter declares it as `provided`, so it is not on your classpath unless you add it.
## Configure it
The starter reads Spring properties under the `llm4s` prefix. It does **not** read the HOCON `application.conf` that the
[Java guide](java#configure-a-provider) uses: here the provider, model and key are properties.
```properties
llm4s.provider=openai
llm4s.model=gpt-4o
llm4s.api-key=${OPENAI_API_KEY}
```
The same in YAML:
```yaml
llm4s:
provider: openai
model: gpt-4o
api-key: ${OPENAI_API_KEY}
```
`llm4s.provider` and `llm4s.model` are required, and so is `llm4s.api-key` for OpenAI and Anthropic. (These flat keys serve
OpenAI, Anthropic and Ollama; every other provider is configured under `llm4s.providers`, see [Any provider](#any-provider).) A missing one stops
the application context from starting, with a message that names the property. Put the key in an environment variable or a
secrets store and refer to it with a placeholder, as above; do not commit it.
### Properties
| Property | Default | Meaning |
|---|---|---|
| `llm4s.enabled` | `true` | `false` registers none of the beans |
| `llm4s.provider` | empty | required without `llm4s.providers.*`: `openai`, `anthropic` or `ollama` |
| `llm4s.model` | empty | required: the model name as the provider spells it |
| `llm4s.api-key` | empty | required for `openai` and `anthropic`; not used by `ollama` |
| `llm4s.base-url` | empty | overrides the provider's default URL (below) |
| `llm4s.organization` | empty | OpenAI only: the organisation id |
| `llm4s.context-window` | `128000` | the model's context window in tokens; must be positive |
| `llm4s.reserve-completion` | `4096` | tokens kept for the reply; at least 0 and less than the context window |
| `llm4s.async.max-threads` | `16` | worker threads of the default executor |
| `llm4s.async.queue-capacity` | `1000` | calls queued before `completeAsync` fails its future |
| `llm4s.health.probe` | `false` | opt in to a real provider call per health check |
| `llm4s.health.probe-ttl` | `60s` | how long a probe result is reused |
| `llm4s.health.probe-timeout` | `10s` | a slower probe is cancelled and reports `DOWN` |
The starter ships Spring Boot configuration metadata (`META-INF/spring-configuration-metadata.json`), so IntelliJ
IDEA and the Spring tools for VS Code and Eclipse complete these keys, show their defaults and flag a misspelt one in
`application.properties` and `application.yml`. Inside a `llm4s.providers.` section they complete nothing: its keys
depend on the provider, and are listed under [Any provider](#any-provider).
### Which providers
The flat `llm4s.provider` key supports three providers:
| `llm4s.provider` | API key | Default `llm4s.base-url` |
|---|---|---|
| `openai` | required | `https://api.openai.com/v1` |
| `anthropic` | required | `https://api.anthropic.com` |
| `ollama` | not used | `http://localhost:11434` |
Any other value stops the context with `Unknown provider: ''. The flat llm4s.provider key supports openai, anthropic
and ollama; configure any other provider under llm4s.providers.`, followed by the providers on the classpath.
### Any provider
Every other provider (Gemini, Azure OpenAI, Mistral, OpenRouter, a self-hosted OpenAI-compatible server, or a provider
module you add) is configured under `llm4s.providers`. These properties mirror the `llm4s.providers` block of llm4s's own
HOCON configuration, so any provider on the classpath works, with the extra keys it declares. `llm4s.providers.provider`
names the section the starter uses:
```properties
llm4s.providers.provider=gemini-main
llm4s.providers.gemini-main.provider=gemini
llm4s.providers.gemini-main.model=gemini-2.0-flash
llm4s.providers.gemini-main.api-key=${GOOGLE_API_KEY}
```
Azure OpenAI takes its resource endpoint and API version as keys of the section:
```properties
llm4s.providers.provider=azure-main
llm4s.providers.azure-main.provider=azure
llm4s.providers.azure-main.model=my-deployment
llm4s.providers.azure-main.api-key=${AZURE_OPENAI_API_KEY}
llm4s.providers.azure-main.endpoint=https://my-resource.openai.azure.com
llm4s.providers.azure-main.api-version=2024-02-01
```
A server that speaks the OpenAI chat-completions format needs only its URL and model; the generic `openai-compatible`
provider also reads `context-window`, `reserve-completion` and `headers.*`:
```properties
llm4s.providers.provider=local
llm4s.providers.local.provider=openai-compatible
llm4s.providers.local.model=my-model
llm4s.providers.local.base-url=http://localhost:8000/v1
llm4s.providers.local.context-window=32000
llm4s.providers.local.headers.X-Team-Id=team-7
```
- `kebab-case` and `snake_case` names in a section are read as the `camelCase` names of the HOCON block (`api-key` is
`apiKey`, `api-version` is `apiVersion`); section and header names are kept as written.
- A section with no `api-key` uses the vendor's shared `llm4s.credentials..api-key`, then the vendor's
conventional environment variable (`GOOGLE_API_KEY`, `MISTRAL_API_KEY`, ...).
- An `LLM4S_PROVIDERS_*` environment variable cannot add a setting; it only overrides one already declared in a property
source (Spring's relaxed binding). Pass a secret with a placeholder (`api-key=${MY_KEY}`), or leave
`api-key` out and set the vendor's environment variable, which reaches the section through `llm4s.credentials`.
- As soon as any `llm4s.providers.*` property is set, the block is used and the flat `llm4s.model`, `llm4s.api-key`, ...
are ignored. With none, the flat keys are read exactly as before.
- An unknown provider id in a section stops the context with llm4s's own message, which names the registered providers
and the dependency to add.
- Behind an outbound proxy or a corporate CA, the `llm4s.http.*` properties - `llm4s.http.proxy.url`,
`.username`, `.password`, `.non-proxy-hosts` (a list), `llm4s.http.tls.trust-store`, `.trust-store-password`,
`.key-store`, `.key-store-password` - apply to the client in both forms, the flat keys and `llm4s.providers.*`, as
[`llm4s.http`](../getting-started/configuration#corporate-networks-a-proxy-and-a-custom-ca) does in HOCON. A
section's own `llm4s.providers..http.*` overrides them for that section.
You can also define your own `JLlmClient` bean, for example from an `application.conf` as the Java guide describes. The
starter's client backs off, and the template and the health indicator use yours; `llm4s.provider` and the other properties
are then not needed:
```java
import org.llm4s.javaapi.JLlmClient;
import org.llm4s.javaapi.Llm4s;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class OtherProviderConfig {
@Bean
JLlmClient llm4sClient() {
return Llm4s.createDefaultClient().get();
}
}
```
`get()` throws an `LlmException` when the `application.conf` is wrong, which stops the context with its message.
## Use the template
`LLM4STemplate` is the bean to inject. It wraps the client: `complete` blocks the calling thread and returns the reply text,
`tryComplete` blocks and returns an `LlmResult` instead of throwing, and `completeAsync` does not block.
```java
import org.llm4s.spring.LLM4STemplate;
import org.springframework.stereotype.Service;
@Service
public class SummaryService {
private final LLM4STemplate llm;
public SummaryService(LLM4STemplate llm) {
this.llm = llm;
}
public String summarise(String text) {
return llm.complete("Summarise in one sentence: " + text);
}
}
```
Each of the three takes a plain `String` or a `Conversation` (built with the Java API's `ConversationBuilder`); `complete` also
has a form that takes `CompletionOptions`.
### When a call fails
`complete` throws `LlmException` when the call fails. `tryComplete` returns the failure as a value:
```java
try {
String answer = llm.complete("What is 2+2?");
System.out.println(answer);
} catch (LlmException e) {
System.err.println(e.getMessage());
}
LlmResult result = llm.tryComplete("What is 2+2?");
```
`LlmException` and `LlmResult` are in `org.llm4s.javaapi` and are described in the [Java guide](java#handling-a-failure).
### Asynchronous calls
`completeAsync` returns a `CompletableFuture` straight away. The blocking provider call runs on the
`llm4sTaskExecutor` bean and the future completes from there:
```java
CompletableFuture reply = llm.completeAsync("Translate to French: " + text);
reply.thenAccept(answer -> System.out.println(answer));
```
It is [`JLlmClient.completeAsync`](java#asynchronous-calls) on that executor. A failed call completes the future
exceptionally with the `LlmException`. `reply.cancel(true)` interrupts the provider call. The
default executor is a bounded pool of daemon threads named `llm4s-async-N`, sized by `llm4s.async.*`; when its queue is full the
returned future fails with an `LlmException` of kind `OTHER` whose cause is the `RejectedExecutionException`, and the call does
not block. It is shut down with `shutdownNow` when the context closes, which interrupts calls still running. A
`llm4sTaskExecutor` bean of your own replaces it and can be any `ExecutorService`, a `ForkJoinPool` included: the future
interrupts the thread running its call itself, rather than relying on the executor's `Future.cancel(true)`.
### A web endpoint
A controller calls the service like any other Spring bean. This one is not compiled with the other snippets, because it needs
`spring-boot-starter-web`, which the starter does not depend on:
```java
// illustrative: needs spring-boot-starter-web, which the starter does not depend on
@RestController
class SummaryController {
private final SummaryService summaries;
SummaryController(SummaryService summaries) {
this.summaries = summaries;
}
@PostMapping("/summaries")
String summarise(@RequestBody String text) {
return summaries.summarise(text);
}
}
```
## Health
With Actuator on the classpath the starter adds a health indicator, the bean `llmHealthIndicator`. By default it makes **no
provider call**: it reports `UP` with the details `provider`, `model` and `probe=disabled`, which means "configured", not "reachable". The API key
is never reported.
With `llm4s.health.probe=true` it sends a one-token completion on the executor, waits at most `llm4s.health.probe-timeout`, and
reuses the outcome (a failure too) for `llm4s.health.probe-ttl`, so health polling does not call the provider each time. A
failed or timed-out probe is `DOWN` with `probe=failed` or `probe=timeout` and an `error` message that has the API key and
token-like strings removed. Every credential of the resolved provider is removed, wherever it was set (a Spring property,
`application.conf`, a `reference.conf` default or an environment variable) and however deep it sits in the provider's
config: Bedrock's access key id, secret key and session token, a workload-identity token, and every `headers` value,
whatever the header is called, along with the token after an auth scheme such as `Bearer`. A health check whose thread is interrupted while it waits (Actuator shutting down, say) is
`DOWN` with `probe=cancelled` and the interrupt flag left set; `health()` never throws `InterruptedException`. A cancelled
probe is not cached: it says something about that thread, not the provider, so the next check probes again. A probe is a
billed call: keep the TTL generous.
## Metrics
Add `llm4s-observability-micrometer` next to `spring-boot-starter-actuator`, and the client records request, duration,
token, cost and error meters (`llm4s.requests`, `llm4s.request.duration`, `llm4s.tokens`, ...) in the application's
`MeterRegistry`, with no code: they are on `/actuator/metrics` and go wherever your registry exports (Prometheus,
Datadog, CloudWatch, OTLP). The starter declares the module `provided`, as it does Actuator, so nothing Micrometer reaches
your classpath unless you add it.
```xml
org.llm4s
llm4s-observability-micrometer_3
0.5.1
```
A `MetricsCollector` bean of your own replaces the Micrometer one, and the client reports to every `MetricsCollector` bean
in the context, so a cost tracker can run next to it. The meters, their tags and the matching Prometheus series are in
[Micrometer metrics](observability/micrometer).
## Testing code that uses the template
There is no supported fake client for tests yet ([#1497](https://github.com/llm4s/llm4s/issues/1497)). What works today: a test
that does not need the model can run with `llm4s.enabled=false`, so no client is built and no key is needed; put the template
behind an interface of your own, which that test replaces. And because the starter's beans back off for a bean of the same
type, a test configuration can supply its own `JLlmClient`.
## Where next
- [Java](java): the client, conversations and failures the template is built on.
- [Providers](providers) and [configuration](../getting-started/configuration): every provider, for the HOCON route.
- The starter's own [README](https://github.com/llm4s/llm4s/blob/main/modules/spring-boot-starter/README.md).
---
Source: https://llm4s.org/guide/image-generation
# Image Generation with LLM4S
The LLM4S library provides a powerful and flexible API for generating images using different providers like Stable Diffusion (via a local WebUI) and the Hugging Face Inference API.
---
## Core Concepts
The API is centered around the `ImageGenerationClient` trait, which defines the common interface for interacting with different image generation services. You can get a pre-configured client using the `ImageGeneration` factory object.
Key models include:
- `ImageGenerationConfig`: A trait for provider-specific configurations (`StableDiffusionConfig`, `HuggingFaceConfig`).
- `ImageGenerationOptions`: A class to control generation parameters like image size, seed, and negative prompts.
- `GeneratedImage`: A case class representing the generated image, including its data and metadata.
---
## Using the Stable Diffusion Client
This client connects to an instance of the [AUTOMATIC1111/stable-diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui). You must have this running locally or on a remote server.
### Setup
First, ensure your `build.sbt` includes the LLM4S dependencies. Image generation is in `llm4s-image`.
(not compile-checked)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-core" % "0.5.1"
libraryDependencies += "org.llm4s" %% "llm4s-image" % "0.5.1"
```
### Example Usage
Here's how to create a client and generate an image:
(not compile-checked)
```scala
import org.llm4s.imagegeneration._
import java.nio.file.Paths
// 1. Create a client for a local Stable Diffusion server
val sdClient = ImageGeneration.stableDiffusionClient(
baseUrl = "http://localhost:7860" // Default URL
)
// 2. Define a prompt and generation options
val prompt = "A photorealistic portrait of a majestic lion"
val options = ImageGenerationOptions(
size = ImageSize.Square512,
negativePrompt = Some("cartoon, drawing, sketch, blurry")
)
// 3. Generate the image
println("Generating image with Stable Diffusion...")
sdClient.generateImage(prompt, options) match {
case Right(image) =>
println("Image generated successfully!")
// Save the image
val path = Paths.get("stable_diffusion_lion.png")
image.saveToFile(path)
println(s"Image saved to ${path.toAbsolutePath}")
case Left(error) =>
println(s"Error generating image: ${error.message}")
}
```
---
## Using the Hugging Face Client
This client uses the Hugging Face Inference API, which requires an API token.
### Setup
You will need a Hugging Face account and an API token with write permissions.
### Example Usage
The process is very similar. The main difference is the configuration.
(not compile-checked)
```scala
import org.llm4s.imagegeneration._
import java.nio.file.Paths
// 1. Get your API token (it's best to use an environment variable)
val hfApiKey = sys.env.getOrElse("HF_API_TOKEN", "your_hf_api_token_here")
if (hfApiKey == "your_hf_api_token_here") {
println("Please set the HF_API_TOKEN environment variable.")
} else {
// 2. Create a Hugging Face client
val hfClient = ImageGeneration.huggingFaceClient(
apiKey = hfApiKey,
model = "runwayml/stable-diffusion-v1-5" // You can choose other models
)
// 3. Define a prompt
val prompt = "A cute robot reading a book, sci-fi concept art"
// 4. Generate the image
println("Generating image with Hugging Face...")
hfClient.generateImage(prompt) match {
case Right(image) =>
println("Image generated successfully!")
val path = Paths.get("hugging_face_robot.png")
image.saveToFile(path)
println(s"Image saved to ${path.toAbsolutePath}")
case Left(error) =>
println(s"Error generating image: ${error.message}")
}
}
```
---
## Customizing Generation
The `ImageGenerationOptions` class allows you to fine-tune the generation process:
| Option | Type | Description |
|--------|------|-------------|
| `size` | `ImageSize` | The dimensions of the output image (e.g., `ImageSize.Square512`) |
| `format` | `ImageMediaType` | The image format, from `org.llm4s.media` (`MediaType.Png` or `MediaType.Jpeg`) |
| `seed` | `Option[Long]` | A specific seed for reproducible results |
| `guidanceScale` | `Double` | How strictly the model should follow the prompt |
| `inferenceSteps` | `Int` | The number of steps in the diffusion process |
| `negativePrompt` | `Option[String]` | A prompt describing what you *don't* want to see |
---
## See Also
- [Examples](/examples/) - Working code examples
- [API Reference](/api/llm-client) - Complete API documentation
---
Source: https://llm4s.org/guide/speech
# LLM4S Speech Module
A comprehensive speech recognition and text-to-speech synthesis module for the LLM4S project, built with Scala and functional programming principles.
---
## Features
### Speech Recognition (STT)
- **Vosk**: Lightweight, offline speech recognition engine
- **Whisper**: High-accuracy transcription via CLI integration
- **Audio Preprocessing**: Resampling, channel conversion, silence trimming
- **Multiple Input Formats**: File, bytes, and stream audio support
- **Cloud**: `OpenAISTTClient` (Whisper API) and `AzureSTTClient` (Azure AI Speech), see [Cloud providers](#cloud-providers)
### Text-to-Speech (TTS)
- **Tacotron2**: Neural speech synthesis via CLI integration
- **Cloud**: `OpenAITTSClient`, `ElevenLabsTTSClient` and `AzureTTSClient`, see [Cloud providers](#cloud-providers)
- **Voice Customization**: Language, speaking rate, pitch, volume control
- **Output Formats**: WAV and raw PCM16 audio support
- **Cross-platform**: Works on Windows, Linux, and macOS
---
## Architecture
The module follows functional programming principles with:
- **Result Types**: `Either[LLMError, T]` for error handling
- **Pure Functions**: Immutable audio transformations
- **ADTs**: Algebraic Data Types for type-safe modeling
- **Composition**: Functional composition for audio processing pipelines
---
## Quick Start
### Basic Usage
(not compile-checked)
```scala
import org.llm4s.speech._
import org.llm4s.speech.stt.{WhisperSpeechToText, STTOptions}
import org.llm4s.speech.tts.{Tacotron2TextToSpeech, TTSOptions}
import org.llm4s.speech.util.PlatformCommands
// Speech Recognition
val stt = new WhisperSpeechToText(PlatformCommands.mockSuccess)
val audioInput = AudioInput.FileAudio(Paths.get("audio.wav"))
val options = STTOptions(language = Some("en"))
val result = stt.transcribe(audioInput, options)
// Text-to-Speech
val tts = new Tacotron2TextToSpeech(PlatformCommands.echo)
val ttsOptions = TTSOptions(
voice = Some("en-female"),
language = Some("en"),
speakingRate = Some(1.2)
)
val audio = tts.synthesize("Hello, world!", ttsOptions)
```
### Audio Preprocessing
(not compile-checked)
```scala
import org.llm4s.speech.processing.AudioPreprocessing
val audioBytes = // ... your audio data
val audioMeta = AudioMeta(sampleRate = 44100, numChannels = 2, bitDepth = 16)
// Convert to mono, resample to 16kHz for STT
val processed = AudioPreprocessing.standardizeForSTT(
audioBytes,
audioMeta,
targetRate = 16000
)
```
`AudioPreprocessing.resamplePcm16` (the resampling step of `standardizeForSTT`) checks its arguments before it
converts anything. The target rate and the source rate must be between 1 and 768000 Hz, the channel count between 1
and 64, and the bit depth a multiple of 8 from 8 to 32; anything else is a `Left(ValidationError)` naming the field
(`targetRate`, `source.sampleRate`, `source.numChannels` or `source.bitDepth`). An output above 256 MiB (about 46
minutes of 48 kHz mono) is a `ValidationError` on `targetRate` too, checked before anything is allocated: resample a
longer recording in pieces. The output has exactly `round(frames * targetRate / sourceRate)` frames, so empty input
gives empty output and a trailing partial frame is ignored; a converter that delivers noticeably fewer frames than
that is a `ProcessingError`, not a silently padded success. Java Sound's converter delays the signal by a fraction of a millisecond, and the length fit drops the
matching tail, so the source's final fraction of a millisecond is not in the output.
---
## Configuration
### Vosk Configuration
(not compile-checked)
```scala
import org.llm4s.speech.stt.VoskSpeechToText
// Use default English model
val stt = new VoskSpeechToText()
// Use custom model path
val stt = new VoskSpeechToText(modelPath = Some("/path/to/vosk-model"))
```
### Environment Variables
```bash
# Vosk Model Path (optional)
VOSK_MODEL_PATH=/path/to/vosk-model
```
---
## File Structure
```
src/main/scala/org/llm4s/speech/
├── Audio.scala # Core audio data structures
├── stt/ # Speech-to-Text implementations
│ ├── SpeechToText.scala # STT trait interface
│ ├── VoskSpeechToText.scala # Vosk integration
│ └── WhisperSpeechToText.scala # Whisper CLI integration
├── tts/ # Text-to-Speech implementations
│ ├── TextToSpeech.scala # TTS trait interface
│ └── Tacotron2TextToSpeech.scala # Tacotron2 CLI integration
├── processing/ # Audio preprocessing utilities
│ └── AudioPreprocessing.scala # Audio transformation functions
├── io/ # Audio I/O operations
│ └── AudioIO.scala # File saving utilities
└── util/ # Cross-platform utilities
└── PlatformCommands.scala # OS-agnostic command helpers
```
---
## Cross-Platform Support
The `PlatformCommands` utility automatically provides the right commands:
| Platform | Echo | File Reader | Directory Listing |
|----------|------|-------------|-------------------|
| Windows | `cmd /c echo` | `cmd /c type` | `cmd /c dir` |
| POSIX | `echo` | `cat` | `ls` |
---
## External Tools
### Whisper
- **Installation**: `pip install openai-whisper`
- **Usage**: The module integrates with Whisper CLI for transcription
- **Models**: Supports various model sizes (tiny, base, small, medium, large)
### Tacotron2
- **Installation**: Requires Tacotron2 CLI tool
- **Usage**: The module integrates with Tacotron2 CLI for synthesis
- **Features**: Voice customization, language support, audio output
---
## Cloud providers
Three cloud TTS clients and two cloud STT clients live in `org.llm4s.speech.tts.provider` and
`org.llm4s.speech.stt.provider`. They are selected like chat models, with a `provider/model`
string, and built through `SpeechProviderSelector`:
| Setting | Values |
|---|---|
| `SPEECH_TTS_MODEL` | `openai/tts-1`, `openai/tts-1-hd`, `elevenlabs/`, `azure/` (e.g. `azure/en-US-JennyNeural`) |
| `SPEECH_TTS_VOICE` | optional voice override (OpenAI default `alloy`) |
| `SPEECH_STT_MODEL` | `openai/whisper-1`, `azure/` (e.g. `azure/en-US`) |
| `OPENAI_API_KEY` | OpenAI TTS and STT |
| `ELEVENLABS_API_KEY` | ElevenLabs TTS (optional `ELEVENLABS_MODEL_ID`, default `eleven_multilingual_v2`) |
| `AZURE_SPEECH_KEY`, `AZURE_SPEECH_REGION` | Azure TTS and STT |
Only the selected provider's credentials are needed. The same keys exist as HOCON under
`llm4s.speech.*` (see this module's `reference.conf`); `OPENAI_SPEECH_BASE_URL`,
`ELEVENLABS_BASE_URL`, `AZURE_SPEECH_TTS_BASE_URL` and `AZURE_SPEECH_STT_BASE_URL` override the
endpoints.
(not compile-checked)
```scala
import org.llm4s.speech.SpeechProviderSelector
import org.llm4s.speech.tts.TTSOptions
import org.llm4s.speech.stt.STTOptions
import org.llm4s.speech.AudioInput
for {
tts <- SpeechProviderSelector.tts() // reads llm4s.speech.tts / SPEECH_TTS_MODEL
audio <- tts.synthesize("Hello from LLM4S")
stt <- SpeechProviderSelector.stt() // reads llm4s.speech.stt / SPEECH_STT_MODEL
text <- stt.transcribe(AudioInput.FileAudio(java.nio.file.Paths.get("hello.wav")), STTOptions())
} yield text.text
```
To pick the model in code instead of `SPEECH_*_MODEL`, use
`SpeechConfigLoader.tts("openai/tts-1")` / `.stt("openai/whisper-1")` (credentials still come from
config) and `SpeechProviderSelector.getTTSClient` / `getSTTClient`.
Behaviour worth knowing:
- **Audio is raw PCM.** The TTS clients request raw 24 kHz, 16-bit, mono PCM from each service, so
`GeneratedAudio.data` is headerless PCM with an accurate `AudioMeta`, the same shape Tacotron2
produces. Write a playable file with `WavFileGenerator.saveAsWav(audio, path)`. The default
`GeneratedAudio.format` is labelled `AudioFormat.WavPcm16` but, as above, the bytes carry no WAV header.
- **MP3 is opt-in.** `TTSOptions(outputFormat = AudioFormat.Mp3)` makes the OpenAI, ElevenLabs and Azure
clients request the service's MP3 (`response_format=mp3`, `output_format=mp3_44100_128`,
`audio-24khz-48kbitrate-mono-mp3`) and return its bytes untouched: no decoding, no resampling.
`GeneratedAudio.format` is `AudioFormat.Mp3`, so the PCM helpers (`WavFileGenerator.saveAsWav`,
`AudioIO.saveWav` / `saveRawPcm16`, `AudioPreprocessing.standardizeForSTT(audio, rate)`) refuse it with a
`ValidationError`; save it with `AudioIO.saveMp3(audio, path)`. `AudioMeta` carries the requested
sample rate and channel count as nominal labels (not verified against the service) and `bitDepth = 0`: MP3 has
no sample width, so do not compute a duration from the byte count. Tacotron2 produces PCM only and refuses MP3. The default stays PCM.
- **OpenAI TTS limits are checked locally**: text over 4096 characters and a `speakingRate` outside
0.25 to 4.0 are a `ValidationError` before any request is sent. Split long text yourself; the clients
do not chunk it.
- **STT input is WAV.** `BytesAudio` and `StreamAudio` are treated as WAV data, as for Whisper and Vosk.
Azure's REST endpoint recognises about 60 seconds of audio per request.
- **Errors map like the chat providers**: HTTP 401/403 is `AuthenticationError`, 429 is
`RateLimitError` (with `Retry-After`), 400 is `ValidationError`, other statuses are `ServiceError`,
and a timeout or refused connection is `TimeoutError` / `NetworkError`.
- The clients take an `Llm4sHttpClient`, so tests inject a stub and never touch the network.
---
## Error Handling
The module uses `Result[T]` (alias for `Either[LLMError, T]`) for robust error handling:
(not compile-checked)
```scala
val result: Result[Transcription] = stt.transcribe(audioInput, options)
result match {
case Right(transcription) =>
println(s"Transcript: ${transcription.text}")
case Left(error) =>
println(s"Error: ${error.formatted}")
}
```
---
## Testing
The module includes comprehensive tests that work cross-platform:
```bash
# Run all tests
sbt test
# Run specific test suites
sbt "testOnly org.llm4s.speech.*"
# Run the current supported Scala build
sbt test
```
The cloud clients are covered without a network by `speech/test` (stubbed HTTP layer, including
`CloudSpeechProviderIntegrationSpec`). Live-API smoke suites, one per provider, are in
`modules/it` (`org.llm4s.speech.*`, tier `@Cloud`): run them with `sbt testSmoke` and
`OPENAI_API_KEY`, `ELEVENLABS_API_KEY`, `AZURE_SPEECH_KEY` / `AZURE_SPEECH_REGION` set. They make
billed calls; a suite whose key is missing is skipped, or fails under `LLM4S_IT_STRICT=true`.
---
## See Also
- [Examples](/examples/) - Working code examples
- [API Reference](/api/llm-client) - Complete API documentation
---
Source: https://llm4s.org/guide/builtin-tools
# Built-in Tools
Ready-made tools for agents: a calculator, the date and time, UUIDs, JSON, files, HTTP, a shell and web search.
---
The built-in tools live in the `llm4s-agent-tools` module (see
[installation](../getting-started/installation#for-built-in-tools-web-search-http-filesystem-shell)). They depend
on `llm4s-core` only, not on the agent runtime, so they work with plain tool calling through a `ToolRegistry` as well
as with an `Agent`. Some of them can read your files, call your network or run programs, so decide which ones a model
gets before you register anything: [section 6](#6-safety-what-each-tool-can-do) says what each can do.
What this page says is checked by tests. The bundle tables, the parameter table, the defaults and the safety section
are asserted against the real code by
[`BuiltinToolsGuideSpec`](https://github.com/llm4s/llm4s/blob/main/modules/agent-tools/src/test/scala/org/llm4s/toolapi/builtin/BuiltinToolsGuideSpec.scala).
The snippets are compiled and run there too, with configuration passed in explicitly where a snippet would read
your environment, except the agent snippet, which needs both `llm4s-agent` and `llm4s-agent-tools` and so is run by
[`BuiltinToolsGuideAgentSpec`](https://github.com/llm4s/llm4s/blob/main/modules/samples/src/test/scala/org/llm4s/samples/toolapi/BuiltinToolsGuideAgentSpec.scala)
in `samples`. The last snippet, which refers to a tool of yours, is the one exception. If you change the page, change
the specs.
## 1. Pick a bundle
`BuiltinTools` builds a set of tools for you. Every method returns a `Result`, because building a tool can fail.
| Bundle | Tools it contains | Reach |
|--------|-------------------|-------|
| `BuiltinTools.coreSafe` | `get_current_datetime`, `calculator`, `generate_uuid`, `json_tool` | None: no files, no network, no processes |
| `BuiltinTools.withHttpSafe()` | the core tools, plus `http_request` | Read-only HTTP |
| `BuiltinTools.withFilesSafe()` | the above, plus `read_file`, `list_directory`, `file_info` | Read-only HTTP and files |
| `BuiltinTools.developmentSafe()` | the above, plus `write_file`, `shell_command` | Files (read and write) and a shell |
| `BuiltinTools.customSafe(...)` | the core tools, plus a tool for each configuration you pass | Exactly what you configure |
Two things are in no bundle: the web search tools ([section 4](#4-web-search)) and `WeatherTool`
([section 7](#7-things-to-know)). You add them yourself.
Start from the smallest bundle that does the job. `customSafe` is the one to use in production, because it forces
you to say what each tool may touch:
(not compile-checked)
```scala
import org.llm4s.toolapi.builtin.BuiltinTools
import org.llm4s.toolapi.builtin.filesystem.FileConfig
import org.llm4s.toolapi.builtin.http.HttpConfig
val tools = BuiltinTools.customSafe(
fileConfig = Some(FileConfig(allowedPaths = Some(Seq("/srv/agent-data")))),
httpConfig = Some(HttpConfig.restricted(Seq("api.example.com")))
) // Result[Seq[ToolFunction[_, _]]]
```
## 2. Register the tools
### With a `ToolRegistry`
(not compile-checked)
```scala
import org.llm4s.toolapi.{ ToolCallRequest, ToolRegistry }
import org.llm4s.toolapi.builtin.BuiltinTools
BuiltinTools.coreSafe.foreach { tools =>
val registry = new ToolRegistry(tools)
registry.execute(ToolCallRequest("calculator", ujson.Obj("operation" -> "multiply", "a" -> 6, "b" -> 7))) match {
case Right(json) => println(json("result")) // 42.0
case Left(error) => println(error.getFormattedMessage)
}
}
```
`registry.getOpenAITools()` gives the tool definitions in the format every client takes. `execute` returns
`Either[ToolCallError, ujson.Value]`: a refused or failed call is a `Left` you can read, never an exception. For
example, dividing by zero comes back as `Left(...)` whose `getFormattedMessage` includes "Division by zero".
### With an `Agent`
(not compile-checked)
```scala
import org.llm4s.agent.Agent
import org.llm4s.toolapi.ToolRegistry
import org.llm4s.toolapi.builtin.BuiltinTools
// `client` is an LLMClient: see Basic Usage for how to create one
val result = for {
tools <- BuiltinTools.coreSafe
agent <- Agent.builder("calculator-agent", client).withTools(new ToolRegistry(tools)).build()
state <- agent.run("What is 15% of 850?")
} yield state
result.map(_.answer) // Right(Some("...")) when the run completed
```
`answer` is `None` when the run did not complete (a guardrail blocked it, or it hit the step limit): read `status`
to see why. The runnable version is
[`BuiltinToolsAgentExample`](https://github.com/llm4s/llm4s/blob/main/modules/samples/src/main/scala/org/llm4s/samples/agent/BuiltinToolsAgentExample.scala).
## 3. The tools
The names are what the model sees, and the parameters are what it can send. A parameter in **bold** is required:
a call without it comes back as a `Left` naming it. The others have defaults.
| Tool | Bundle | Parameters | What it returns |
|------|--------|------------|-----------------|
| `get_current_datetime` | core | `timezone` (default UTC), `format` (`iso` or `human`: English text; anything else is an error) | `datetime`, `timezone`, `timestamp`, `iso8601`, `components` |
| `calculator` | core | **`operation`** (`add`, `subtract`, `multiply`, `divide`, `power`, `sqrt`, `percentage`, `abs`, `min`, `max`, `modulo`), **`a`**, `b` (needed by the two-operand operations) | `expression`, `result`, `formatted` |
| `generate_uuid` | core | `count` (1 to 10, default 1), `format` (`standard` or `compact`) | `uuids`: each with `uuid`, `version`, `variant` |
| `json_tool` | core | **`operation`** (`parse`, `format`, `query`, `validate`), **`json`**, `path` (for `query`) | `success`, `result`, `formatted` |
| `http_request` | http | **`url`**, `method`, `headers`, `body`, `content_type` | `statusCode`, `headers`, `body`, `contentType`, `truncated`, and more |
| `read_file` | files | **`path`**, `max_lines`, `encoding` | `path`, `content`, `size`, `lines`, `truncated` |
| `list_directory` | files | **`path`**, `max_entries` (at most 100), `include_hidden` | `entries`, `totalFiles`, `totalDirectories`, `truncated` |
| `file_info` | files | **`path`** | `exists`, `size`, `sizeHuman`, timestamps, permissions, `isSymlink`, `extension` |
| `write_file` | development | **`path`**, **`content`**, `append`, `encoding` | `path`, `bytesWritten`, `created`, `appended` |
| `shell_command` | development | **`command`** | `exitCode`, `stdout`, `stderr`, `truncated`, `timedOut` |
| `duckduckgo_search` | none | **`search_query`** | `abstract_`, `answer`, `relatedTopics`, and more |
| `brave_web_search` | none | **`search_query`** | web results (`brave_image_search`, `brave_video_search` and `brave_news_search` exist too) |
| `exa_search` | none | **`query`** | results with `title`, `url`, `text`, `highlights`, and more |
`json_tool` queries use dot notation for objects and brackets for arrays: `data.users[0].name`. A path is read to the end: an empty segment (`a..b`), an index that is not a whole number (`a[x]`, `a[-1]`) or an index too large for an `Int` is an error that names the text it stopped at, not a silently shortened query. Every operation refuses a document nested more than 512 levels deep.
## 4. Web search
Three search tools are not in any bundle, because each calls a third party and two of them need a key. Their settings
are read from `llm4s.tools.*` with `ToolsConfigLoader`, then passed to the tool:
(not compile-checked)
```scala
import org.llm4s.config.ToolsConfigLoader
import org.llm4s.toolapi.ToolRegistry
import org.llm4s.toolapi.builtin.BuiltinTools
import org.llm4s.toolapi.builtin.search.DuckDuckGoSearchTool
val registry = for {
config <- ToolsConfigLoader.loadDuckDuckGoSearchTool()
search <- DuckDuckGoSearchTool.create(config)
core <- BuiltinTools.coreSafe
} yield new ToolRegistry(core :+ search)
```
The Brave and Exa tools are built the same way:
(not compile-checked)
```scala
import org.llm4s.config.ToolsConfigLoader
import org.llm4s.toolapi.builtin.search.{ BraveSearchCategory, BraveSearchTool, BraveWebSearchResult, ExaSearchTool }
val brave = ToolsConfigLoader
.loadBraveSearchTool()
.flatMap(config => BraveSearchTool.create[BraveWebSearchResult](config, BraveSearchCategory.Web))
val exa = ToolsConfigLoader.loadExaSearchTool().flatMap(config => ExaSearchTool.create(config))
```
Brave also has image, video and news categories. These are the keys, as `llm4s-agent-tools` ships them in its
`reference.conf` (set them in your `application.conf`, or let the environment variable fill them):
| Tool | Key under `llm4s.tools` | Default | Environment variable |
|------|-------------------------|---------|----------------------|
| DuckDuckGo | `duckduckgo.apiUrl` | `https://api.duckduckgo.com/` | `DUCK_DUCK_GO_SEARCH_API_URL` |
| Brave | `brave.apiKey` | none: required | `BRAVE_SEARCH_API_KEY` |
| | `brave.apiUrl` | `https://api.search.brave.com/res/v1` | `BRAVE_SEARCH_API_URL` |
| | `brave.count` | `5` | `BRAVE_SEARCH_COUNT` |
| | `brave.safeSearch` | `moderate` | `BRAVE_SAFE_SEARCH` |
| Exa | `exa.apiKey` | none: required | `EXA_API_KEY` |
| | `exa.apiUrl` | `https://api.exa.ai` | `EXA_API_URL` |
| | `exa.numResults` | `10` | `EXA_NUM_RESULTS` |
| | `exa.searchType` | `auto` (or `neural`, `fast`, `deep`) | `EXA_SEARCH_TYPE` |
| | `exa.maxCharacters` | `500` | `EXA_MAX_CHARACTERS` |
`ExaSearchTool.create` rejects an empty API key when it builds the tool, so a missing key is an error at start-up and
not in the middle of a conversation. DuckDuckGo needs no key and is best for definitions, facts and quick lookups,
not for general web search. Whatever the model searches for is sent to the search provider.
## 5. Configure the tools
Each tool that can reach outside the process takes a configuration. The defaults below are what you get when you do
not pass one.
### Files: `FileConfig` and `WriteConfig`
| Setting | `FileConfig` (read tools) | `WriteConfig` (`write_file`) |
|---------|---------------------------|------------------------------|
| `allowedPaths` | `None`: any path outside `blockedPaths` | **required**: no default |
| `blockedPaths` | `FileConfig.DefaultBlockedPaths`: the platform's system and credential locations (wins over `allowedPaths`; see below) | not a setting |
| `maxFileSize` | 1 MB | 10 MB |
| `followSymlinks` | `false` | not a setting |
| `allowOverwrite` | not a setting | `false` |
| `createDirectories` | not a setting | `true` |
(not compile-checked)
```scala
import org.llm4s.toolapi.builtin.BuiltinTools
import org.llm4s.toolapi.builtin.filesystem.{ FileConfig, WriteConfig }
val tools = BuiltinTools.customSafe(
fileConfig = Some(FileConfig(allowedPaths = Some(Seq("/srv/agent-data")), maxFileSize = 256 * 1024)),
writeConfig = Some(WriteConfig(allowedPaths = Seq("/srv/agent-data/out")))
)
```
A path is judged by where it really is: symbolic links are resolved, then that real location is compared with each
allowed or blocked entry (resolved the same way), one path component at a time. The file tools remove `..` as text
first and then open the location they judged. `FileConfig.isPathAllowed`, and the shell tool's path policy, judge the
path as given, and an operating system applies a `..` that comes after a link in one of two ways: POSIX (Linux, macOS)
follows the link first, so `link/..` is the parent of the link's target, while Windows removes `..` as text first, so
`link/..` is the directory holding the link. The check reads the path both ways, on every platform, and allows it
only when both locations are allowed. So with `data/l -> data/a/b`, `data/l/../x.txt` is allowed (`data/a/x.txt` or
`data/x.txt`), and `data/l/../../x.txt` is refused even on Linux, where it means `data/x.txt`, because on Windows it
is `x.txt` beside `data`; spell such a path without the `..` after the link. `/srv/agent-data`
covers `/srv/agent-data/notes.txt` but not `/srv/agent-data-secret`, and a link inside an allowed directory that leads
outside it is refused, for reading, listing and writing alike. A link that cannot be resolved (a dangling link) is
refused. `followSymlinks = false` (the default) additionally refuses a path that is itself a link when reading or
listing; it does not decide whether a link may lead out of the allowed area, which is always refused. The tools open
the resolved path; a directory swapped for a link between the check and the open is a race this narrows and does not
close. A hard link inside an allowed directory to a file elsewhere is not contained: no path check can tell it from the
file itself. On macOS `/var` is a link to `/private/var`, so the default blocklist's `/var` also blocks the temporary
directories under `/private/var/folders`.
The default blocklist, `FileConfig.DefaultBlockedPaths`, depends on the platform the JVM runs on:
| Platform | System locations | Under the user's home, besides the common entries |
|----------|------------------|----------------------------------------------------|
| Linux and other POSIX | `/etc`, `/var`, `/sys`, `/proc`, `/dev`, `/boot`, `/root`, `/run` | `.local/share/keyrings` |
| macOS | `/etc`, `/var`, `/sys`, `/proc`, `/dev`, `/private/etc`, `/private/var`, `/Library/Keychains` | `Library/Keychains` |
| Windows | `C:\Windows`, `C:\ProgramData\Microsoft\Crypto`, `C:\ProgramData\ssh` | `_netrc`, `NTUSER.DAT`, and under `AppData`: `Roaming\Microsoft\Credentials`, `Local\Microsoft\Credentials`, `Roaming\Microsoft\Protect`, `Roaming\Microsoft\Crypto`, `Roaming\gcloud`, `Roaming\gnupg`, `Roaming\GitHub CLI`, `Roaming\postgresql` |
The common entries under the home directory (`user.home`) are `.ssh`, `.gnupg`, `.aws`, `.azure`, `.config/gcloud`,
`.kube`, `.docker`, `.config/gh`, `.netrc`, `.git-credentials`, `.pgpass`, `.password-store`, `.npmrc`, `.pypirc`,
`.m2/settings.xml`, `.m2/settings-security.xml`, `.ivy2/.credentials` and `.sbt/.credentials`. A blocked entry is
matched without regard to case (`/ETC/hosts` is inside `/etc`), as a case-insensitive file system such as the macOS
and Windows defaults requires. A Windows installed on a drive other than `C:` is not covered: add its directory, as
in `FileConfig(blockedPaths = FileConfig.DefaultBlockedPaths :+ "D:\\Windows")`. A blocklist cannot name every
sensitive file, so it is defence in depth: set `allowedPaths` to the directories the agent needs.
`developmentSafe(workingDirectory, fileAllowedPaths)` reads only inside `workingDirectory` when you give one, and
anywhere outside the blocklist when you do not. It writes inside `fileAllowedPaths` (default `/tmp`) and the working
directory, and it **does** allow overwriting.
The default blocklist includes `/var`, which on macOS is where the system temporary directory lives
(`/var/folders/...`): a file there is refused even if you put its directory in `allowedPaths`.
### HTTP: `HttpConfig`
| Setting | Default |
|---------|---------|
| `allowedMethods` | `GET`, `HEAD` |
| `allowedDomains` | `None`: any domain that is not blocked |
| `blockedDomains` | `localhost`, `127.0.0.1`, `0.0.0.0`, `::1`, the cloud metadata hosts and `169.254.169.254` |
| `blockInternalIPs` | `true` |
| `followRedirects` | `false` |
| `redirectSafeHeaders` | `Accept`, `Accept-Language`, `Accept-Encoding`, `User-Agent`, `Content-Type` (only with a re-sent body): the only caller-set headers a redirect carries to another origin |
| `timeout` | 30 seconds, for the whole call: connecting, every redirect hop and reading the body |
| `maxResponseSize` | 10 MB: at most this many bytes of the body are read; the rest is never read, and `truncated` is `true` |
`HttpConfig.restricted(Seq("api.example.com"))` allows only those domains. `HttpConfig.withWriteMethods()` adds
`POST`, `PUT`, `DELETE` and the rest, and `HttpConfig.unsafe` also turns the address checks off: use that only in a
sandbox.
### Shell: `ShellConfig`
| Setting | Default |
|---------|---------|
| `allowedCommands` | none: an empty list allows nothing |
| `workingDirectory` | the process's own |
| `timeout` | 30 seconds |
| `maxOutputSize` | 100,000 characters |
| `environment` | none: extra variables, set on top of the environment the command receives |
| `inheritedEnvironment` | `PATH`, `LANG`, `LC_ALL`, `TERM` and `SystemRoot` (those that exist); `development()` passes the whole environment |
| `pathPolicy` | none: the file-like arguments of a command are not checked |
`ShellConfig.readOnly()` allows `ls`, `cat`, `head`, `tail`, `pwd`, `echo`, `wc`, `date`, `whoami`, `which` and
`file`. `ShellConfig.readOnlyWithin(policy)` is the same list with the working directory and every file-like argument
held to a `FileConfig` by its `isPathAllowed` (the rule above, with a `..` after a link read both the POSIX and the
Windows way). `ShellConfig.development()` adds `git`, `sbt`, `make`, `npm`, `grep`, `find`, `cp`, `mv`, `rm` and more:
read the next section before you use it.
## 6. Safety: what each tool can do
A tool call is a request from a model, and a model can be wrong, confused or steered by text it read. Pick the
bundle by what you would be willing to let it do unsupervised.
| Bundle | A model given it can |
|--------|----------------------|
| `coreSafe` | compute, read the clock, make UUIDs and parse the JSON it is given. Nothing leaves the process. |
| `withHttpSafe()` | all of the above, and fetch pages from the public internet with `GET` and `HEAD`: your server's IP address is the one the remote site sees. |
| `withFilesSafe()` | all of the above, and read files: by default any the process can read outside the blocklist (`/etc`, `/var`, ...), so set `allowedPaths`. |
| `developmentSafe()` | all of the above, and write files, and run any program on the allowlist, which for `development()` is most of a developer's toolbox. |
What the controls do, and where they stop:
- **HTTP** refuses methods outside `allowedMethods`, a scheme other than `http` and `https`, any domain outside
`allowedDomains`, `localhost`, loopback, the cloud metadata addresses, and private ranges such as `10.x`, `172.16.x`
and `192.168.x` (and their IPv6 counterparts, such as unique-local `fc00::/7` and IPv6 forms that carry a private
IPv4 address, and every other range the IANA special-purpose registries mark as not globally reachable, such as
`240.0.0.0/4`, `192.0.0.0/24` and IPv6 outside `2000::/3`), all before sending a request. Redirects are not followed
unless you turn that on; when they are, each hop is checked again, and from the first hop that leaves the original
scheme, host and port (so a downgrade from `https` to `http` counts), and on every hop after it, only the headers on
`redirectSafeHeaders` go with the request: every other header the model set, such as `X-Api-Key` or
`X-Auth-Token`, is dropped, and a credential header (`Authorization`, `Cookie`, a name that redaction treats as
sensitive, plurals such as `X-Api-Keys` included, or one ending in `token` or `key`) is dropped even if you list it. A hop that drops the body (a `301` or `302` after a `POST`, `PUT`, `PATCH` or
`DELETE`), on either origin, sends no `Content-Type`, neither a caller-set header nor the tool's own `content_type`. An IPv6 address
inside a global prefix is not judged by the IPv4 address its interface identifier may embed (ISATAP-style
`...:5efe:a.b.c.d`), so such an address is allowed whatever IPv4 address it names. `timeout` bounds the whole call, so a
server that answers a byte at a time cannot hold the tool past it. A `TIMEOUT` releases the caller, not the
request: one already sent may still be delivered and acted on (a `POST` is not rolled back), and a DNS lookup cannot
be interrupted (`InetAddress.getAllByName` waits for the system resolver), so the tool's worker thread may outlive
the deadline by up to the resolver's own timeout, typically a few seconds per attempt as `/etc/resolv.conf` or the
OS sets it ([#1734](https://github.com/llm4s/llm4s/issues/1734)). The address check resolves the host name, and
the connection then resolves it again, so a domain whose DNS answer changes in between (DNS rebinding) can pass the check with a public address and connect to a private one. The request still
goes out from your network, so do not give it to a model that handles untrusted text next to credentials or
internal services the server can reach; where that matters, also block private ranges at the network level, for
example with an egress proxy or firewall.
- **Files**: `allowedPaths` and `blockedPaths` judge the real location of a path (links resolved, compared one
component at a time), so a link cannot lead out of an allowed directory and a sibling that shares a name prefix is
not inside it. Treat them as a containment check on the file tools, not as a sandbox: the check and the open are
two steps, which this narrows and does not close, and the shell tool is a separate matter (below). Allow one
directory made for the agent, and when the files matter, run the process as a user that cannot read anything
else, or in a container.
- **Shell**: the command is split into words and started directly, **without a shell**. `&&`, `;`, `|`, `>` and
`$VAR` reach the program as ordinary text and are not interpreted, and only the first word is checked against
`allowedCommands`. That stops a command from chaining into another one. It does not limit what an allowed
program does:
- `readOnly()` is an allowlist of program names, not read-only execution. Its programs only read in ordinary use,
but most options are passed through unchecked: `date -s` sets the clock when the process is allowed to. The
options that write a file, run a program or read one the command does not name are refused (`file -C`, `-m`,
`-M` and `-f`, `date -f` and `-r`, `wc --files0-from`, and, for an allowlist that adds `sort`, `sort -o`,
`--output`, `-T`, `--temporary-directory`, `--compress-program` and `--files0-from`, plus on Windows, where
`sort` may be `sort.exe`, `sort -t` and the `/O` and `/T` switches) in every spelling: anywhere in a cluster of
short options, with the value attached or not (`-bC`, `-Mmagic`, `-roout`), but not inside the value of
`sort -t` (`sort -to` sets the separator to `o`), under any abbreviation of the long form
(`date --fil`), with or without `=value`, after a `--`, which an option taking an argument can consume, and
whatever the case of the program's name or a Windows `.exe` (`FILE -C`, `file.exe -C`)
([#1723](https://github.com/llm4s/llm4s/issues/1723)).
`cat`, `head` and `tail` can read any file the process can read, **so the file settings above do not apply to the
shell** unless you use `ShellConfig.readOnlyWithin(policy)`, which holds the working directory and each file-like
argument to that rule, judged as the program will hand it to the OS (a `..` after a link is read both as POSIX
applies it, at the link target's parent, and as Windows does, at the directory holding the link, and both must
be allowed). A value attached to a flag is checked the same way, since the program may open it: the value after
`=` of a long option (`grep --file=x`) and every tail of a short-option cluster (`grep -fx`, `grep -ifx`), so a
link out of the allowed directory, a blocked file or `..` cannot be passed that way
([#1723](https://github.com/llm4s/llm4s/issues/1723)). Which options take a value is not modelled, so text that
happens to name such a file (`grep -e..`) is refused too. A value that is one plain path component (no `/` or
`\`, not `.` or `..`, nothing Windows reads specially such as `:`, a trailing `.` or space, or a device name)
costs one lookup, whether anything is in the working directory under that name: if nothing is, it cannot be a
link, and the policy is applied to its spelling with no further lookup, which is exactly the full check's verdict;
if something is, it is resolved in full. No limit on the length of a file name is assumed. The cost of the checks
is bounded per command: an argument longer than 4096 characters is refused, each distinct path is checked once,
and a command whose checks would take more than 20000 file-system lookups (one per plain component, two per
component of a path resolved in full) is refused as too costly to check, rather than holding the agent for
minutes before the command starts (the shell `timeout` covers only the command's run). A hard
link to a file outside passes, as it does for the file tools. A command that walks directories itself (`ls -R`,
`grep -r`, `find`) is checked only at the path it starts from.
- On Windows, a character a program may receive as a different ASCII character is refused, with or without a path
policy ([#1766](https://github.com/llm4s/llm4s/issues/1766)). A program that reads its command line in an ANSI
code page gets each argument converted with "best fit", so `..／..／secret` (U+FF0F) reaches it as `../../secret`,
`¥` and `₩` as `\` on the Japanese and Korean code pages, and `-ｏ` as `-o`, past checks that read the argument
as one plain name or no option. An argument, the working directory and an `environment` value are refused when
they hold a character a common ANSI code page (874, 932, 936, 949, 950, 1250 to 1258) best-fits to an ASCII
character other than a letter or digit, any best-fitted character in an argument starting with `-` or `/`, or a
character the host's own ANSI code page cannot hold. Accented letters such as `é` stay allowed in an operand when
that code page holds them. POSIX hosts are unchanged.
- On a Windows host whose ANSI code page is double-byte (932, 936, 950), an argument holding a character whose
second byte is 0x5C, the byte of `\` (`表` and `十` on 932, `功` on 950), is refused when the JDK puts it in quotes
([#1807](https://github.com/llm4s/llm4s/issues/1807)): `a 表` reaches the command line as `"a 表"`, whose bytes
end `5C 22`, and a program that splits its command line byte by byte reads `\"` as a literal quote and splits the
arguments after it differently. The JDK quotes an argument holding a space or whitespace character, `"`, `<`, `>`,
`&`, `|` or `^`; `表.txt` stays allowed.
- `development()` is not a sandbox: `sbt`, `make`, `npm`, `git`, `find` and `env` can run arbitrary programs, so a
model given it can do anything the process can.
- The started program receives a scrubbed environment: only `PATH`, `LANG`, `LC_ALL`, `TERM` and `SystemRoot`
(those that exist) plus whatever you put in `environment`, so provider API keys in your process's environment do
not reach it. `development()` passes the whole environment through, because build tools need it.
- **Search** sends the query text to the search provider.
A reasonable starting point for an agent that answers questions about some documents is `customSafe` with
`fileConfig` set to one dedicated directory, no `writeConfig`, no `shellConfig`, and `httpConfig` set to the domains it
needs. Add the shell only for a developer's own machine.
## 7. Things to know
- **Failures are values.** A refused call returns a `Left`. The message names the reason: for example `Access denied:
path '...' is not allowed`, `HTTP method 'POST' is not allowed. Allowed: GET, HEAD`, `SSRF_BLOCKED: domain '...' is
not allowed` or `Command 'rm' is not allowed`. The registry hands the message back to the model, so it can try
something else.
- **`WeatherTool` is a demo.** It lives in `org.llm4s.toolapi.tools`, returns the same fixed reading (sunny, 22.5)
for every location and calls no weather service. Use it to see tool calling work, not for weather.
- **The shell lists are POSIX programs.** `readOnly()` and `development()` name programs such as `ls` and `cat`, so
on Windows give `ShellConfig` a list of your own.
- **A call with a missing parameter is a `Left`.** For example `calculator` without `a` comes back as an error that
names the parameter, and nothing runs.
## 8. Your own tools
The built-in tools are built with `ToolBuilder`, the same way you build yours, and they register in the same
`ToolRegistry`. Build your tool, then add it to the sequence:
(not compile-checked)
```scala
// myTool: a ToolFunction you built with ToolBuilder
val registry = BuiltinTools.coreSafe.map(core => new ToolRegistry(core :+ myTool))
```
[Build Your Own Tool](custom-tools) walks through writing one: the `Schema` DSL, reading arguments, returning
errors and testing it. See [Agents](agents/#agent-with-tools) for how a tool is used in a run, and
[Error Handling](error-handling) for working with the `Result` these methods return.
## Related
- [Agents](agents/): running tools in an agent loop, with guardrails
- [Error Handling](error-handling): `Result[A]` and `LLMError`
- [`BuiltinToolsExample`](https://github.com/llm4s/llm4s/blob/main/modules/samples/src/main/scala/org/llm4s/samples/toolapi/BuiltinToolsExample.scala):
every bundle in a runnable sample
---
Source: https://llm4s.org/guide/context-window-pruning
# Context Window Pruning Strategies
Complete guide to LLM4S context window management and pruning strategies.
## Overview
As conversations grow, token counts increase, leading to:
- **Higher costs** - Most LLM providers charge per token
- **Slower responses** - Longer context = slower processing
- **Context window limits** - Models have maximum input token limits
LLM4S provides context window management via `ContextWindowConfig` and `ContextWindowMiddleware`, with multiple pruning strategies to keep conversations within budget while preserving important context. The middleware trims what is sent to the model on each call; the thread keeps its full history.
Two rules hold for every strategy: the current turn (the latest user message onward) is never pruned, so the request always starts with a user message and may exceed the budget; and a tool call is never separated from its result. The agent's system prompt is outside the budget.
## Configuration Basics
(not compile-checked)
```scala
import org.llm4s.agent.{ Agent, ContextWindowConfig, PruningStrategy }
import org.llm4s.agent.graph.middleware.ContextWindowMiddleware
val contextConfig = ContextWindowConfig(
maxTokens = Some(4096), // Keep max 4K tokens
maxMessages = Some(20), // OR max 20 messages
preserveSystemMessage = true, // Always keep system prompt
minRecentTurns = 3, // Keep last 3 user/assistant pairs
pruningStrategy = PruningStrategy.OldestFirst // How to prune
)
val agent = Agent.builder("assistant", client)
.withMiddleware(new ContextWindowMiddleware(contextConfig))
.build()
agent.flatMap(_.runMultiTurn("...", Seq(...)))
```
## Pruning Strategies
### 1. OldestFirst (Recommended for most use cases)
**How it works:** Removes the oldest messages first while preserving the system message and most recent turns.
**Best for:**
- ✅ General-purpose multi-turn conversations
- ✅ FAQ-bot with rolling history
- ✅ Customer support chats
- ✅ When you want to keep latest context fresh
**Cost:**
- 🟢 Good - Removes old/repetitive context first
- 🟢 Predictable token savings
**Quality:**
- 🟡 Medium - May lose context from early turns
- ✅ Recent context stays intact
**Example:**
(not compile-checked)
```scala
val config = ContextWindowConfig(
maxMessages = Some(15),
pruningStrategy = PruningStrategy.OldestFirst
)
// Conversation with 30 messages → last 15 messages kept
// System message is always preserved
```
**Trade-offs:**
| Aspect | Rating | Notes |
|--------|--------|-------|
| Token efficiency | ⭐⭐⭐⭐ | Removes metadata/greetings first |
| Context continuity | ⭐⭐⭐ | Recent context preserved |
| Implementation | ⭐⭐⭐⭐⭐ | Simple FIFO removal |
| Use case fit | Broad | Works for most scenarios |
---
### 2. MiddleOut (For complex reasoning)
**How it works:** Keeps the start (system + initial setup) and end (recent exchanges), removes middle messages.
**Best for:**
- ✅ Complex problem-solving requiring both initial context and recent progress
- ✅ Long code reviews or technical documents
- ✅ When you need to preserve both "setup" and "current state"
- ✅ Scientific or analytical conversations
**Cost:**
- 🟡 Medium - Loses middle context but keeps important bookends
- 🟡 Less predictable than OldestFirst
**Quality:**
- 🟡 Medium - Loses intermediate steps
- ✅ Good - Keeps initial and recent context
**Example:**
(not compile-checked)
```scala
val config = ContextWindowConfig(
maxMessages = Some(20),
pruningStrategy = PruningStrategy.MiddleOut
)
// Conversation:
// [System] [Initial Context] [Step 1] [Step 2] ... [Step N] [Recent Q&A]
// ↑ ↑
// These middle steps are pruned; keeps ends
```
**Trade-offs:**
| Aspect | Rating | Notes |
|--------|--------|-------|
| Token efficiency | ⭐⭐⭐ | Medium removal |
| Context continuity | ⭐⭐⭐⭐ | Preserves bookends |
| Implementation | ⭐⭐⭐⭐ | Straightforward middle removal |
| Use case fit | Specialized | Good for reasoning chains |
---
### 3. RecentTurnsOnly (For strict turn-based systems)
**How it works:** Keeps only the last N complete conversation turns (user message + assistant response + tool calls).
**Best for:**
- ✅ Turn-based games or dialogues
- ✅ Systems that value latest N interactions equally
- ✅ When conversation structure matters more than time
- ✅ Predictable conversation patterns
**Cost:**
- 🟢 Good - Focused token reduction
- ✅ Very predictable
**Quality:**
- 🟡 Medium - Only recent exchanges (may need earlier context)
- ✅ Good - Full recent turns preserved
**Example:**
(not compile-checked)
```scala
val config = ContextWindowConfig(
pruningStrategy = PruningStrategy.RecentTurnsOnly(5)
)
// Keeps exactly the last 5 user/assistant turn pairs
// A "turn" = [User message] → [Assistant response] + optional [Tool messages]
// Turn 1: User asks → Assistant responds (+ tool calls)
// Turn 2: User asks → Assistant responds (+ tool calls)
// ...
// Turn 5: User asks → Assistant responds (+ tool calls)
// Turns 1-N: Pruned
```
**Trade-offs:**
| Aspect | Rating | Notes |
|--------|--------|-------|
| Token efficiency | ⭐⭐⭐⭐ | Removes entire turn groups |
| Context continuity | ⭐⭐ | May lose early turns |
| Implementation | ⭐⭐⭐⭐ | Turn-based filtering |
| Use case fit | Specific | Best for turn-based systems |
---
### 4. Custom (Maximum flexibility)
**How it works:** You provide a pure function that decides which messages to keep.
**Best for:**
- ✅ Domain-specific pruning logic
- ✅ Marketing messages removal
- ✅ Metadata-first removal
- ✅ Complex importance scoring
- ✅ Hybrid strategies
**Cost:**
- 🟢 Excellent - You decide what's expensive
- ⭐ Depends on your logic
**Quality:**
- 🟢 Excellent - You decide what matters
- ⭐ Depends on your logic
**Example:**
(not compile-checked)
```scala
// Remove verbose debug messages, keep concise ones
val pruneDebugMessages: Seq[Message] => Seq[Message] = { messages =>
messages.filter { msg =>
!msg.content.toLowerCase.contains("[debug]") ||
msg.content.length < 100 // Keep short debug messages
}
}
val config = ContextWindowConfig(
maxMessages = Some(50),
pruningStrategy = PruningStrategy.Custom(pruneDebugMessages)
)
```
**Advanced Example (importance scoring):**
(not compile-checked)
```scala
val scoreImportance: Seq[Message] => Seq[Message] = { messages =>
messages.map { msg =>
val score = msg.role match {
case MessageRole.System => 100 // Always keep
case MessageRole.User => 80 // Keep user messages
case MessageRole.Assistant => 70 // Keep responses
case MessageRole.Tool => 40 // Lower priority
case _ => 0
}
(msg, score)
}
.sortBy(_._2) // Sort by importance
.dropWhile(_ => tokenCount > limit) // Remove until under limit
.map(_._1)
}
```
**Trade-offs:**
| Aspect | Rating | Notes |
|--------|--------|-------|
| Token efficiency | ⭐⭐⭐⭐⭐ | Complete control |
| Context continuity | ⭐⭐⭐⭐⭐ | Complete control |
| Implementation | ⭐⭐ | Requires custom code |
| Use case fit | Very specific | Best for special requirements |
---
### 5. AdaptiveWindowing (Intelligent auto-tuning)
**How it works:** Automatically calculates the optimal context window based on:
- Model's context size
- Model tier (small, medium, large, extra-large)
- Token pricing (input vs output cost)
- Cost sensitivity preference
**Best for:**
- ✅ Multi-model deployments (switch models without config changes)
- ✅ Cost-conscious production systems
- ✅ Quality-sensitive applications
- ✅ Hybrid cost/quality optimization
- ✅ When you want "set it and forget it"
**Cost:**
- 🟢 Excellent - Optimizes based on actual pricing
- ✅ Adapts to model changes
**Quality:**
- 🟢 Good - Uses 70-80% of context window
- ✅ Leaves room for model reasoning
**Example:**
(not compile-checked)
```scala
import org.llm4s.model.ModelMetadata
import org.llm4s.agent.PruningStrategy
// From ModelMetadata (automatically available)
val strategy = PruningStrategy.AdaptiveWindowing(
contextWindowSize = 128_000, // gpt-4o context
inputCostPerToken = Some(0.000005), // $5 per 1M input tokens
outputCostPerToken = Some(0.000015), // $15 per 1M output tokens
preserveMinTurns = 3,
costSensitivity = 0.5 // Balanced (0=quality, 1=cost)
)
val config = ContextWindowConfig(
pruningStrategy = strategy
)
println(strategy.calculateOptimalWindow) // Output: 89,600 tokens (70% * 128K)
println(strategy.explanation) // "AdaptiveWindowing: using 89,600 tokens (70% of 128K large model)"
```
**Model tier adjustments:**
| Model Size | Context | Multiplier | Adjusted Window | Use Case |
|-----------|---------|-----------|-----------------|----------|
| Small | 8K | 60% | 4.8K | Budget-conscious |
| Medium | 100K | 70% | 70K | Balanced |
| Large | 200K | 75% | 150K | Quality-focused |
| Extra Large | 1M+ | 80% | 800K+ | Complex reasoning |
**Cost sensitivity examples:**
(not compile-checked)
```scala
// 🟡 Balanced (default)
// Sensitivity = 0.5
// Minimizes cost moderately while preserving quality
val balanced = PruningStrategy.AdaptiveWindowing(
contextWindowSize = 128_000,
inputCostPerToken = Some(0.000005),
outputCostPerToken = Some(0.000015),
costSensitivity = 0.5
)
// 🟢 Quality-focused
// Sensitivity = 0.1
// Use most of the available context, cost matters less
val qualityFocused = PruningStrategy.AdaptiveWindowing(
contextWindowSize = 128_000,
costSensitivity = 0.1 // Low cost sensitivity
)
// 💰 Cost-optimized
// Sensitivity = 0.9
// Aggressively minimize token usage
val costOptimized = PruningStrategy.AdaptiveWindowing(
contextWindowSize = 128_000,
costSensitivity = 0.9 // High cost sensitivity
)
```
**Trade-offs:**
| Aspect | Rating | Notes |
|--------|--------|-------|
| Token efficiency | ⭐⭐⭐⭐ | Auto-optimized |
| Context continuity | ⭐⭐⭐⭐ | Preserves minimum turns |
| Implementation | ⭐⭐⭐⭐⭐ | Recommended for production |
| Use case fit | Universal | Works everywhere |
---
## Strategy Comparison Matrix
| Strategy | Pros | Cons | Cost | Quality | When to Use |
|----------|------|------|------|---------|------------|
| **OldestFirst** | Simple, predictable | May lose early context | ⭐⭐⭐⭐ | ⭐⭐⭐ | Default choice |
| **MiddleOut** | Preserves bookends | Loses intermediate steps | ⭐⭐⭐ | ⭐⭐⭐⭐ | Complex reasoning |
| **RecentTurnsOnly** | Consistent turn count | Loses older turns | ⭐⭐⭐⭐ | ⭐⭐ | Turn-based systems |
| **Custom** | Maximum flexibility | Requires implementation | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | Special needs |
| **AdaptiveWindowing** | Auto-optimized, scales | Needs model metadata | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ | Production systems |
---
## Configuration Patterns
### Pattern 1: Token-based Limiting (Recommended)
(not compile-checked)
```scala
val config = ContextWindowConfig(
maxTokens = Some(4096),
pruningStrategy = PruningStrategy.OldestFirst
)
// More precise - based on actual token usage
// Handles variable-length messages correctly
```
### Pattern 2: Message-based Limiting (Simpler)
(not compile-checked)
```scala
val config = ContextWindowConfig(
maxMessages = Some(20),
pruningStrategy = PruningStrategy.OldestFirst
)
// Simpler to understand
// Less precise but easier to reason about
```
### Pattern 3: Hybrid (Both limits)
(not compile-checked)
```scala
val config = ContextWindowConfig(
maxTokens = Some(4096),
maxMessages = Some(50),
pruningStrategy = PruningStrategy.OldestFirst
)
// Enforces BOTH limits
// Prunes when either limit exceeded
```
### Pattern 4: Preserve Minimum Context
(not compile-checked)
```scala
val config = ContextWindowConfig(
maxMessages = Some(15),
preserveSystemMessage = true,
minRecentTurns = 3, // Always keep last 3 turns
pruningStrategy = PruningStrategy.RecentTurnsOnly(5)
)
// Guarantees minimum context is always available
// Even if other limits would be exceeded
```
### Pattern 5: Adaptive Mode (Multi-model)
(not compile-checked)
```scala
val strategy = if (isExpensiveModel) {
PruningStrategy.AdaptiveWindowing(
contextWindowSize = modelSize,
inputCostPerToken = Some(0.000010),
costSensitivity = 0.8 // Aggressive cost optimization
)
} else {
PruningStrategy.AdaptiveWindowing(
contextWindowSize = modelSize,
inputCostPerToken = Some(0.000001),
costSensitivity = 0.3 // More quality-focused
)
}
val config = ContextWindowConfig(
pruningStrategy = strategy
)
```
---
## Cost Analysis
### Calculate conversation cost
(not compile-checked)
```scala
val tokenCounter = ConversationTokenCounter.forModel("gpt-4o").get
val conversationTokens = tokenCounter.countConversation(conversation)
// Cost calculation
val inputCostPerToken = 0.000005 // $5 per 1M tokens
val totalCost = conversationTokens * inputCostPerToken
println(s"Conversation: $conversationTokens tokens = \$$totalCost")
```
### Compare strategies
(not compile-checked)
```scala
// Without pruning: 10,000 tokens = $0.05
// With OldestFirst (keeping 50 messages): ~3,000 tokens = $0.015 (70% savings)
// With AdaptiveWindowing: ~4,096 tokens = $0.020 (60% savings)
// For 100 conversations per day:
// Without: $5.00/day
// With pruning: $1.50-2.00/day (savings: $3-3.50/day)
```
---
## Best Practices
### ✅ DO
1. **Use token-based limits** - More accurate for variable-length conversations
(not compile-checked)
```scala
maxTokens = Some(4096) // Better than maxMessages
```
2. **Preserve system message** - Keeps instructions intact
(not compile-checked)
```scala
preserveSystemMessage = true // Default, always set this
```
3. **Use AdaptiveWindowing for production** - Handles model changes automatically
(not compile-checked)
```scala
PruningStrategy.AdaptiveWindowing(contextWindowSize, costPerToken)
```
4. **Monitor pruning events** - Log when and what gets removed
(not compile-checked)
```scala
// See next section on observability
```
5. **Test with real conversations** - Benchmark different strategies
### ❌ DON'T
1. **Set maxTokens too low** - Breaks context coherence
(not compile-checked)
```scala
maxTokens = Some(256) // Too small!
```
2. **Forget about output budget** - Leave room for model responses
(not compile-checked)
```scala
// If context is 100K, use ~70K for input (leave 30K for output)
maxTokens = Some(70_000)
```
3. **Use token limits without a tokenizer** - Falls back to word count estimation
(not compile-checked)
```scala
// Integrate ConversationTokenCounter for accuracy
```
4. **Hard-code window sizes** - Use AdaptiveWindowing instead
(not compile-checked)
```scala
// Bad: Fixed 4K window works only for specific models
// Good: AdaptiveWindowing scales with model changes
```
---
## Observability & Monitoring
### Inspect pruning results
Pruning happens in `ContextWindowMiddleware`, on each model call, and never changes the thread.
The thread's `AgentResult.messages` is the full history; to see what was sent, wrap the model call
with a small middleware of your own, placed after the context window middleware with `runsAfter`:
(not compile-checked)
```scala
import org.llm4s.agent.graph.middleware.{ AgentMiddleware, MiddlewareId, ModelContext, ModelRequest, ModelResponse }
class LogSentMessages(logger: Logger) extends AgentMiddleware {
val id = MiddlewareId("log-sent-messages")
override def runsAfter = Set(MiddlewareId("context-window"))
override def wrapModelCall(request: ModelRequest, context: ModelContext)(
next: ModelRequest => Result[ModelResponse]
): Result[ModelResponse] = {
logger.info(s"Sending ${request.messages.length} messages")
next(request)
}
}
```
### Log pruning performance
`ContextWindowMiddleware` takes an optional token counter (`new ContextWindowMiddleware(config, tokenCounter)`),
and `AgentResult.usage` reports the tokens the provider actually billed across the thread, which is
the number to compare with and without pruning:
(not compile-checked)
```scala
logger.info(s"Thread usage: ${result.usage}")
```
---
## Troubleshooting
### Issue: Conversation keeps growing despite pruning
**Cause:** the agent was built without a `ContextWindowMiddleware`
**Solution:**
(not compile-checked)
```scala
// ❌ Wrong: pruning not enabled
Agent.builder("assistant", client).build()
// ✅ Right: pruning enabled
Agent.builder("assistant", client)
.withMiddleware(new ContextWindowMiddleware(config)) // Must add it!
.build()
```
### Issue: Important context is being removed
**Cause:** Too aggressive pruning strategy
**Solution:**
(not compile-checked)
```scala
// Increase limits
val config = ContextWindowConfig(
maxTokens = Some(8192), // Was 4096
pruningStrategy = PruningStrategy.MiddleOut // Preserves bookends
)
// Or use Custom strategy to protect important messages
```
### Issue: Conversations are too short
**Cause:** minRecentTurns or preserveSystemMessage settings
**Solution:**
(not compile-checked)
```scala
val config = ContextWindowConfig(
maxTokens = Some(8192),
minRecentTurns = 1, // Fewer forced turns
preserveSystemMessage = false, // Allow system to be pruned if needed
pruningStrategy = PruningStrategy.OldestFirst
)
```
---
## Advanced Examples
### Example 1: Cost-conscious chatbot
(not compile-checked)
```scala
val config = ContextWindowConfig(
pruningStrategy = PruningStrategy.AdaptiveWindowing(
contextWindowSize = 8000, // Smaller model
inputCostPerToken = Some(0.00001), // Expensive
outputCostPerToken = Some(0.00003),
costSensitivity = 0.9 // Minimize cost
)
)
```
### Example 2: Research assistant (preserve all context)
(not compile-checked)
```scala
val config = ContextWindowConfig(
maxTokens = Some(200_000), // Large window
pruningStrategy = PruningStrategy.MiddleOut, // Keep setup + recent
preserveSystemMessage = true
)
```
### Example 3: Code review assistant
(not compile-checked)
```scala
val config = ContextWindowConfig(
pruningStrategy = PruningStrategy.Custom { messages =>
messages.filter { msg =>
// Keep code snippets and review comments
msg.role == MessageRole.System || // Keep instructions
msg.content.contains("```") || // Keep code blocks
msg.content.contains("review") || // Keep review comments
messages.indexOf(msg) >= messages.length - 10 // Keep recent
}
}
)
```
### Example 4: Turn-based game
(not compile-checked)
```scala
val config = ContextWindowConfig(
pruningStrategy = PruningStrategy.RecentTurnsOnly(10), // Last 10 turns
preserveSystemMessage = true,
minRecentTurns = 1 // Always keep last turn
)
```
---
## Summary
| Need | Strategy | Reason |
|------|----------|--------|
| I just want a working solution | **OldestFirst** | Simplest, most reliable |
| I'm building a production system | **AdaptiveWindowing** | Auto-scales with models |
| I have complex reasoning | **MiddleOut** | Preserves bookends |
| I need precise control | **Custom** | Maximum flexibility |
| I have turn-based logic | **RecentTurnsOnly** | Natural for turns |
Choose **AdaptiveWindowing** for production systems that work across multiple models. Choose **OldestFirst** for everything else.
---
Source: https://llm4s.org/guide/agents/
# Agent Framework
Build sophisticated AI agents with tools, guardrails, memory, and multi-agent coordination.
---
## Overview
The LLM4S Agent Framework provides a production-ready foundation for building LLM-powered agents with:
- **Tool Calling** - Type-safe tools with automatic schema generation
- **Guardrails** - Input/output validation for safety and quality
- **Memory** - Short and long-term context with semantic search
- **Handoffs** - Agent-to-agent delegation for specialist routing
- **Streaming** - Real-time events for responsive UIs (`agent.stream`, [streaming guide](streaming))
- **Graphs** - Typed multi-agent workflows on `GraphBuilder`: parallel nodes, joins and checkpoints ([recipe](../../examples/cookbook/multi-agent-graph.html))
## Quick Start
The agent runtime is the `llm4s-agent` module, alongside `llm4s-core` and a provider module:
(not compile-checked)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-agent" % llm4sVersion
```
### Basic Agent
An agent is built once - tools, guardrails, handoffs and middleware belong to it - and run by thread.
(not compile-checked)
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.agent.Agent
import org.llm4s.model.ModelRegistryService
// Create an agent and run a query
val result = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
agent <- Agent.builder("assistant", client).build()
result <- agent.run("What is the capital of France?")
} yield result
result match {
case Right(r) => println(r.answer.getOrElse(s"Run ended: ${r.status}"))
case Left(error) => println(s"Error: $error")
}
```
`Agent.builder(id, client)` takes the agent's id (letters, digits, `_` and `-`, up to 52) and an
`LLMClient`. `build()` returns a `Result[Agent]` and refuses an invalid agent: clashing tool names,
a bad or duplicate handoff id, or `withMaxSteps` below 1.
### Agent with Tools
(not compile-checked)
```scala
import org.llm4s.agent.Agent
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.model.ModelRegistryService
import org.llm4s.toolapi.{ SafeParameterExtractor, Schema, ToolBuilder, ToolRegistry }
// Define a tool
def getWeather(params: SafeParameterExtractor): Either[String, String] =
params.getString("location").map(location => s"The weather in $location is sunny, 72F")
val weatherSchema = Schema
.`object`[Map[String, Any]]("Weather query")
.withProperty(Schema.property("location", Schema.string("The city to get the weather for")))
// Give the agent its tools when you build it
val result = for {
weatherTool <- ToolBuilder[Map[String, Any], String](
"get_weather",
"Get current weather for a location",
weatherSchema
).withHandler(getWeather).buildSafe()
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
agent <- Agent.builder("assistant", client)
.withTools(new ToolRegistry(Seq(weatherTool)))
.build()
result <- agent.run("What's the weather in Paris?")
} yield result
```
`withTools` takes a `ToolRegistry` (an `MCPToolRegistry` too) or a `ToolSet` of `AgentTool`s.
### Multi-Turn Conversations
A conversation is carried by its thread. `continueConversation` runs the next turn on the thread
of a previous result:
(not compile-checked)
```scala
val result = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
agent <- Agent.builder("assistant", client).build()
// First turn
result1 <- agent.run("Tell me about Scala")
// Follow-up (preserves context)
result2 <- agent.continueConversation(result1, "How does it compare to Java?")
// Another follow-up
result3 <- agent.continueConversation(result2, "What about performance?")
} yield result3
```
To name the thread yourself, use `agent.run(threadId, query)`; `agent.runMultiTurn(first, followUps)`
runs several turns on one thread and stops at the first result that is not `Completed`. A `Left` carries no
thread id, so when a turn of `run(query)` or `runMultiTurn` fails or is cancelled, its thread is forgotten once
the turn has ended; name the thread to recover such a turn.
### Handling the Result
`run` returns `Result[AgentResult]`. `Left` is a failure: a provider, tool or middleware error is a
`GraphError`, and a thread that is busy or still has pending work is refused. A run that ended in a
defined way is `Right`, with a status:
(not compile-checked)
```scala
result.map { r =>
r.status match {
case AgentStatus.Completed(answer) => println(answer)
case AgentStatus.Blocked(guardrail, why) => println(s"Blocked by $guardrail: $why")
case AgentStatus.StepLimitReached => println("Hit the step limit")
case AgentStatus.TokenLimitReached => println("Hit the run's token budget")
case AgentStatus.Suspended(approvals, qs, asked, held) => println("Waiting for a person")
}
}
```
`AgentResult` also carries `threadId`, `runId`, `activeAgent`, `messages` (the thread's full
history, never a system prompt) and `usage`.
### A Typed Answer
`withOutput(schema)` gives the agent a response format: a `SchemaDefinition[O]`, with a uPickle
`ReadWriter[O]` in scope. Write the schema by hand, or derive it with `llm4s-schema-derivation`:
`withOutput(SchemaOf[Invoice].definition)`. The system prompt states the schema on every model call,
and a call that offers no tools also carries it as its `responseFormat`. A call that offers tools carries
no format, because several providers cannot combine one with tool calls. `decodeOutput[O]` reads the answer:
(not compile-checked)
```scala
case class Invoice(customer: String, total: Double) derives ReadWriter
val invoiceSchema: ObjectSchema[Invoice] = ??? // or SchemaOf[Invoice].definition
for {
agent <- Agent.builder("extractor", client).withOutput(invoiceSchema).build()
result <- agent.run("Bill Acme 12.50")
invoice <- result.decodeOutput[Invoice]
} yield invoice
```
A final answer must be JSON that matches the schema and reads as `O`; it is stored as that JSON alone.
An invalid answer is sent back to the model once, with its problems, in the same step; neither is stored.
A second invalid answer ends the turn with `Left(GraphError.OutputInvalid(errors))`: the turn is removed
from the thread, as an output guardrail's block removes it, both calls stay in `usage`, `recover` has
nothing to continue, and the next turn runs from before it. The format is the agent's own: a handoff
target answers in its own format, or freely.
---
## Safety Defaults
- **Agent step limit**: an agent makes at most `maxSteps` model calls per turn, `Agent.DefaultMaxSteps` (50) unless you call `withMaxSteps(n)`. The count is shared by every agent a turn hands off to; at the limit the result is `AgentStatus.StepLimitReached`. The limit means the same through `run`, `start`, `stream`, `continueConversation` and `runMultiTurn`: each turn gets its own `maxSteps` model calls.
- **Run token budget**: off by default. `RunConfig().withBudgets(RunBudgets.default.withMaxTokens(n))` bounds the input and output tokens one run (`run`, `recover` or `resume`) may use, counted from the thread's usage when the run started. Before each model call - the loop's own, and every call a middleware makes through `next` or `ModelContext.accounted` - the run's tokens are compared with it, and at the budget the turn ends `AgentStatus.TokenLimitReached` without calling the model. A call in flight is never interrupted, so a run can overshoot the budget by one model call. A completion that reports no usage is charged an estimate of the whole request (messages, tool definitions and response schema) and the whole answer, never zero. A retrying client (`ReliableClient`) is one call: its failed attempts carry no usage and are not counted, so set `maxTokens` with retries in mind. Each run brings its own budget, like its `timeout`; a task reads the run's deadline as `RunContext.deadline`.
- **HTTPTool methods**: `HttpConfig()` defaults to `GET` and `HEAD` only. Use `HttpConfig.withWriteMethods()` or `HttpConfig().withAllMethods` to allow write methods.
---
## Core Concepts
### Threads and Results
A conversation lives in a thread on a `GraphRuntime` (in memory unless you call `withRuntime`). The
thread holds the messages, the active agent and the usage; it holds no tools, guardrails or agents,
which belong to the `Agent`. `AgentResult` is a value to read, not something to pass back in.
- `agent.start(threadId, query)` returns an `AgentRun` at once, with `await()` and `cancel()`;
`run` is `start` followed by `await`.
- A run that failed or was cancelled leaves its thread recoverable: `agent.recover(threadId)`
re-runs only the work that did not finish.
- A run that suspended for a person (`AgentStatus.Suspended`) - an approval, a tool's or a middleware's
question, or a static breakpoint - continues with `agent.resume(threadId, answers)`, building each
answer with `result.approve(id)`, `result.reject(id, reason)`, `result.edit(id, arguments)`,
`result.reply(id, value)` or `result.proceed(id)`; see [Human Review](human-review).
- A thread stays in the runtime until `agent.forget(threadId)` removes it - one-shot
`agent.run(query)` threads too, which on the default in-memory runtime means they stay in memory.
Forget a conversation you will not continue: `agent.run(query).flatMap(r => agent.forget(r.threadId).map(_ => r))`.
`forget` refuses a thread that a run still holds - in this runtime, or by a live claim in any runtime sharing
the store (`ThreadBusy`) - or that belongs to another tenant (`TenantMismatch`); an unknown thread is `Right(())`.
### Suspended turns from Java and Kotlin
The Java facade (`llm4s-java-api`) returns every turn as a `JAgentResult`, read without Scala types:
`answer()` is an `Optional`, `messages()` a `java.util.List`, `usage()` a
`JUsageSummary`, and `status()` a `JAgentStatus` whose `kind()` is the Java enum `AgentStatusKind` -
`COMPLETED`, `BLOCKED`, `STEP_LIMIT_REACHED`, `TOKEN_LIMIT_REACHED` or `SUSPENDED` - with `answer()`, `guardrail()` and
`reason()` as `Optional`s (see the [Java guide](../java#an-agent-turn)). A `SUSPENDED`
status's `pending()` is a `java.util.List`: the turn's approvals, then its tool
questions, middleware questions and breakpoints; it is empty for any other status, and
`JAgent.pending(result)` is a shortcut for it. Each `PendingInterrupt` has `id()` and `kind()` (the Java
enum `InterruptKind`: `APPROVAL`, `QUESTION`, `MIDDLEWARE_QUESTION` or `BREAKPOINT`). The rest are
`Optional`s, present only for the kinds they belong to: `toolName()` and `argumentsJson()` when a tool
call waits (always for an approval or a tool question), `reason()` for an approval, `questionJson()` -
the question as JSON - for a tool's or a middleware's question, `middleware()` for a middleware
question, and `node()` and `phase()` (the Java enum `BreakpointPhase`) for a breakpoint. Answer each
pending item with `Answer.approve(id)`, `reject(id, reason)`, `edit(id, argumentsJson)`,
`reply(id, json)` or `proceed(id)`, then call `agent.resume(threadId, answers)`, which blocks like `run`
and returns an `LlmResult`.
You can answer only some of them: the result is `SUSPENDED` again, with the rest still pending.
```java
LlmResult turn = agent.run("Deploy the release");
while (turn.get().status().kind() == AgentStatusKind.SUSPENDED) {
List answers = new ArrayList<>();
for (PendingInterrupt p : turn.get().status().pending()) {
switch (p.kind()) {
case APPROVAL -> answers.add(Answer.approve(p.id())); // or reject / edit
case QUESTION, MIDDLEWARE_QUESTION -> answers.add(Answer.reply(p.id(), "{\"ok\":true}"));
case BREAKPOINT -> answers.add(Answer.proceed(p.id()));
}
}
turn = agent.resume(turn.get().threadId(), answers);
}
System.out.println(turn.get().answer().orElse("(" + turn.get().status().kind() + ")"));
```
`agent.recover(threadId)` continues a turn that failed or was cancelled, and also returns an
`LlmResult`. A failed result means one of these: a malformed answer, an empty answers
list (`InvalidResume`), an answer to an id the thread is not waiting for, a thread that is not suspended (`resume`), or a thread with nothing to
recover. `resume` and `recover` handle an interrupt the way `run` and `continueConversation` do: interrupting the
thread blocked in the call cancels the turn, and the call returns a `CancelledError` with the interrupt flag still
set once the turn has ended, leaving the thread for `recover` (see
[Java Threading and Cancellation](../java-threading-and-cancellation#interrupting-the-caller-cancels-an-agent-turn)).
To cancel a turn without interrupting a thread, use `streamResume` or `streamRecover` and cancel the stream (see
[Streaming Events](streaming#java-and-kotlin)).
The Kotlin API reuses these types: every `AgentKt` turn returns a `JAgentResult`, and `when` over
`status().kind()` covers it. `AgentKt.pending(result)` is the same `List`.
`agent.resume(threadId, answers)` and `agent.recover(threadId)` are `suspend` functions that return
the `JAgentResult` or throw `LLMException`. Every `AgentKt` suspend function that runs a turn - `run`,
`continueConversation`, `resume` and `recover` - runs it as the matching flow does (`stream`,
`streamResume`, `streamRecover`), so cancelling the caller - a cancelled scope, `withTimeout` -
cancels the turn. The call throws `CancellationException` once the turn has ended, and the
conversation thread is no longer busy: it is left for `recover`, which finishes the cancelled turn.
If the turn had already completed when the cancellation arrived, the call still throws
`CancellationException`, but the turn's result is committed to the thread: `recover` then has nothing to
recover and throws `LLMException` ("no incomplete execution"), and the next turn continues from that result.
The one-shot `run(query)` is the exception, as in Java and Scala: nothing it throws carries its random thread
id, so once a failed or cancelled turn has ended it forgets the thread. A conversation you named for `stream` is
yours to forget: `agent.forget(threadId)` needs no result, succeeds for an unknown thread and throws
`LLMException` for one whose turn is still running. A cancelled call waits up to 5 seconds
for the turn to end; a provider that ignores the interrupt for longer is left to finish on its own, and the
thread stays busy (`ThreadBusy`) until it does.
```kotlin
var turn = agent.run("Deploy the release")
while (turn.status().kind() == AgentStatusKind.SUSPENDED) {
val answers = turn.status().pending().map { p ->
when (p.kind()) {
InterruptKind.APPROVAL -> Answer.approve(p.id())
InterruptKind.QUESTION, InterruptKind.MIDDLEWARE_QUESTION -> Answer.reply(p.id(), """{"ok":true}""")
InterruptKind.BREAKPOINT -> Answer.proceed(p.id())
}
}
turn = agent.resume(turn.threadId(), answers)
}
```
### Agent Lifecycle
```
Query
|
v
input node (input guardrails) ---> Blocked (nothing stored)
|
v
model <---------- tool results
|
+-- tool calls --> call-tool (parallel, bounded) --> collect --> model
+-- handoff -----> the target agent's model
|
v
finish node (output guardrails) ---> Completed | Blocked (turn removed)
```
### Parallel Tool Calls
The tool calls of one model message run in parallel, bounded by `RunBudgets.maxConcurrency`,
which you set through the `RunConfig` you pass to `run`.
`ToolExecutionStrategy` remains in core as a `ToolRegistry` feature; the agent no longer takes one.
### Tool Side Effects
**Every tool call gets exactly one result.** The agent, not the tool, writes the result the model
sees, for every outcome: the tool's content on success, and an `{"error": ...}` result for a tool
error, a thrown exception, invalid arguments, an unknown tool, a denial, a rejected approval and
every call of a refused handoff batch. The model is called again only once every call of the batch
has its result, so a call waiting for approval or an answer holds the model back, and a partial
`resume` sends nothing to it. Before each model call the agent checks the request, as every
`wrapModelCall` middleware left it, and refuses one in which a tool call does not have exactly one
result straight after its message; a model message whose call ids repeat, or are blank, is refused
before it is stored, and `history` you import must keep the same rule. The OpenAI and Anthropic
clients are checked against this contract in both wire formats (`assertOneToolResultPerCall` in
`llm4s-provider-testkit`).
**A call cut short gets no result until it runs again.** When a turn is cancelled, times out, or
fails on a tool's `Fatal` or a store error mid-batch, the calls that finished keep their results
and the unfinished ones get none: the thread is left for `recover`, which runs only the unfinished
calls and never repeats a finished one. `start` refuses the thread until then (`IncompleteRun`).
Until [#1702](https://github.com/llm4s/llm4s/issues/1702) adds `updateState`, there is no way to
give an unfinished call a result of your own: a call that misbehaves every time it runs - it hangs
until cancelled, or fails with `Fatal` - leaves the thread two choices, `recover` (run it again) or
`forget` (drop the thread and its history).
**Tools run at least once, so pass the idempotency key on.** A call can run again: a retrying
`wrapToolCall` middleware, `recover` after a cancellation, failure or crash, and an approval each
run it once more, and a call cut short may already have done its work. With a store shared between
processes, a call can even run in two runs at the same time: a worker paused past its claim's lease
can still be running a tool when another runtime takes the thread over and runs the same call (see
[Fencing](durable-checkpointers#fencing-a-run-that-lost-its-claim-stops-and-records-nothing)). `ToolContext.idempotencyKey`
is the same in every run of one call, so a tool with an external side effect passes it to the
system it calls, which then does the work once:
(not compile-checked)
```scala
import org.llm4s.agent.graph.tool.{AgentTool, ToolContent, ToolOutcome}
val charge = AgentTool(chargeSpec) { (args, context) =>
payments.charge(args.amount, idempotencyKey = context.idempotencyKey.value) match
case Right(receipt) => ToolOutcome.Success(ToolContent.Json(ujson.Str(receipt.id)))
case Left(error) => ToolOutcome.Error(error.message)
}
```
A plain `ToolFunction` - one built with `ToolBuilder`, a built-in tool, an MCP tool - reads the same call as
`params.call`, a `ToolCallInfo` with the call's `toolCallId` and `idempotencyKey`. It is `None` when the function runs
outside an agent, through `ToolRegistry.execute`, which has no call to describe:
(not compile-checked)
```scala
import org.llm4s.toolapi.{ Schema, ToolBuilder }
val charge = ToolBuilder[Map[String, Any], String]("charge", "Charges a card", chargeSchema)
.withHandler { params =>
for
amount <- params.getDouble("amount")
receipt <- payments.charge(amount, idempotencyKey = params.call.map(_.idempotencyKey))
yield receipt.id
}
.buildSafe()
```
Two tools pass it on by themselves. `HTTPTool` adds `Idempotency-Key: ` to a POST, PUT, PATCH or DELETE it
sends as an agent's call, unless the call's own headers set one. An MCP tool sends it to its server in the
`tools/call` request's `_meta`, as `llm4s.org/idempotencyKey` (with `llm4s.org/toolCallId`), never among the
arguments; an llm4s `MCPServer` hands it to its tool as `params.call`, and another server can read it there or
ignore it.
A tool defined from Java (`llm4s-java-api`) reads the same call from its `JToolArgs`: `idempotencyKey()` and
`toolCallId()` are `Optional`s, empty outside an agent run. A `JTool.builder` lambda reads them from its
`args`, and a `@Tool` method from a parameter of type `JToolArgs`, which the model does not see (see the Java guide's
[Tools](../java#tools)):
```java
LlmResult refund = JTool.builder("refund", "Refunds a charge once")
.param("charge", String.class, "The charge to refund")
.handler(args -> billing.refund(args.getString("charge"), args.idempotencyKey().orElse(null)))
.build();
```
A tool declared with the Kotlin DSL (`llm4s-kotlin-api`) reads it as `args.idempotencyKey`, `null` outside an agent
run ([The Kotlin DSL](../java#the-kotlin-dsl)):
```kotlin
tool("refund", "Refunds a charge once") {
param("charge", "The charge to refund")
handler { args -> billing.refund(args.get("charge"), args.idempotencyKey) }
}
```
The key is 64 hexadecimal characters, derived from the thread, where on it the model call that
issued the tool call ran (its checkpoint, superstep and task), and the call's id, and recorded
with the call. A call of a later model request gets a new key even when the provider reuses its
call id (`call_0` in every turn) and the turns share one `RunConfig`, and so does a call from a
model request made again because the first response was never stored. The boundaries of the guarantee:
- The key de-duplicates only where the external system honours it. Without one, a side effect can
happen more than once; the runtime never promises exactly-once execution.
- A tool sees the key only when it reads it: `ToolContext.idempotencyKey` in an `AgentTool`,
`params.call` in a `ToolFunction`, `idempotencyKey()` in a Java tool's `JToolArgs`, `idempotencyKey` in a
Kotlin tool's `ToolArgs`. One that ignores it runs at least once like any other.
- An approved or edited call keeps the key it was proposed with, and a call that asked a question
keeps it when it resumes. Make a tool's first run, before approval or an answer, free of side
effects.
- The agent stores each call's result as soon as the call finishes (the runtime's `Sync`
durability), and a stored result is never run again. A process that dies between a tool's side
effect and that store runs the call again on `recover`, with the same key. A graph run under
`Async` or `OnExit` durability can lose more after its last durable commit: finished calls run
again with their keys, and a lost model response is asked for again, so its calls get new keys.
- A stale worker and its successor send the same key for a call both run, so a key-honouring system
does the work once. A system that can fence writers can also refuse the stale one outright:
`context.run.position.fencingToken` is the run's claim token, which grows with every claim on the
thread, so the system keeps the highest token it has seen and refuses a lower one.
**Guarded effects: replay without repeating.** For a target that does not take an idempotency key,
such as a file, a sandbox or a store, perform the effect through `ReplayGuard`. The guard records an
intent before the effect and its outcome after, in the thread's own store, under the run's claim. A
call that runs again then gets its original outcome back instead of repeating the work. The target
declares one `ReplaySupport`, and each has its own guard:
- `ReplayGuard.transactional` is for a target that records the key with the outcome in the same
transaction as the effect, and implements `AppliedKeys`. The effect and `forget` are given the
run's claim token, and the target refuses a token older than the newest it has seen for the
key's `scope` (its thread). So a run that lost its claim can neither apply, nor read back, nor
forget anything once its successor has touched the target.
- `ReplayGuard.verifiable` is for a target that can read a digest (`StateCheck`) of what the effect
touches. If the effect began but its outcome was not recorded, the guard checks the target. It
returns the presumed outcome if the effect landed, runs the effect again if it did not, and
refuses if another thread changed the target in between. The effect must be a compare-and-set
against `before`, so that a stalled run's write cannot land on its successor's.
- `ReplayGuard.unverifiable` is for everything else. If the effect began but its outcome was not
recorded, the guard refuses to run it again ("it may already have run"), unless the tool is
idempotent.
(not compile-checked)
```scala
import org.llm4s.agent.graph.{GraphError, ReplayGuard, StateCheck}
val append = AgentTool(appendSpec) { (args, context) =>
val key = context.effectKey("file.append", args.path)
ReplayGuard.verifiable[Int](context.run, key)(Right(StateCheck.of(read(args.path)))) {
val before = read(args.path)
val after = before + args.text
Right(ReplayGuard.Plan(StateCheck.of(before), StateCheck.of(after), after.length) {
// under a lock every writer takes: write only if the file is still what the plan read
writeIfUnchanged(args.path, StateCheck.of(before), after).map(_ => after.length)
})
} match
case Right(length) => ToolOutcome.Success(ToolContent.Json(ujson.Num(length)))
case Left(r: GraphError.EffectReplayRefused) => ToolOutcome.Error(r.message)
case Left(other) => ToolOutcome.Fatal(other) // outcome unknown: recover replays it
}
```
What each guard does when the run crashed at each point, and why, is in design record
[§4.24](https://github.com/llm4s/llm4s/blob/main/docs/design/typed-agent-runtime/4.24-stage-3-slice-c4-replay-of-external-effects.md). The
rules a caller relies on:
- **Build each key with `effectKey`.** In a tool, `context.effectKey(operation, discriminator*)`
derives the key from the call's idempotency key. In a hook or node, `RunContext.effectKey` derives
it from the task's `taskKey`. Give each effect of a task its own operation or discriminator.
- **A fork's re-run of a call is a new effect.** The key includes the thread.
- **The evidence is kept exactly as long as the task can run again.** That is until the commit that
finishes the task, however long the thread waits. `prune` never removes it, and `deleteThread`
removes it with the thread.
- **A Block does not undo an external effect.** A guardrail that blocks a turn restores every state
key the turn wrote, its messages included (§4.22), but a file the turn wrote outside thread state, a
command it ran or a record it stored stays as it is. The turn's replay evidence goes with its
closing commit, so nothing replays it either.
### Harness File Backends
The harness keeps an agent's files in a `FileBackend` (`org.llm4s.agent.harness`): thread state
(`StateFileBackend`, the default), a long-term store (`StoreFileBackend`, for `/memories/`), several at
once (`CompositeBackend`), files the application ships, read-only (`StaticFileBackend`: mount it at `/app`
and point `SkillsConfig.withSources` and `InstructionsConfig.withPaths` there, since on state files only a tool
call writes a file), a directory of the host (`HostFileBackend`), or one chosen per run
(`FileBackend.resolved`). Each backend declares the locks its mutations hold, replays its external
effects through `ReplayGuard`, and enforces its own size limits.
`HostFileBackend.open(root, lockDirectory)` exposes a trusted directory as `/`
([design §4.40](https://github.com/llm4s/llm4s/blob/main/docs/design/typed-agent-runtime/4.40-host-file-backend.md)).
It is the root's only writer: it holds an OS lock on the root until `close()`, and any other open of
the same root, or of one inside or around it, in this process or another, is refused with
`HostRootInUse`. Give every process on the host one `lockDirectory`, outside every root, on a local
file system that nothing prunes. It refuses symbolic links, files with other hard links, mounts beneath
the root and volumes that make 8.3 short names; on Windows, which reports no hard-link counts, pass
`allowUncheckedIdentity = true` only for a root with no hard links. It is not a security boundary:
never give it a root an untrusted process can write, and run untrusted commands in the workspace
sandbox.
The backend holds the root's lock until `close()`, so give it a managed lifetime: one that closes it
however the work ends. Until it is closed, every other open of the root is refused.
(not compile-checked)
```scala
import org.llm4s.agent.graph.tool.ToolSet
import org.llm4s.agent.harness.{FileTools, HostFileBackend}
import java.nio.file.Path
import scala.util.Using
HostFileBackend.open(Path.of("/srv/project"), Path.of("/var/lib/myapp/llm4s-locks")).flatMap { opened =>
Using.resource(opened) { project => // closed when the block ends, whether it succeeds or fails
for {
tools <- ToolSet.of(FileTools(project)*)
agent <- Agent.builder("coder", client).withTools(tools).build()
result <- agent.run("Summarise README.md")
} yield result
}
}
```
An application that keeps an agent for longer keeps the backend beside it, and closes it when it shuts
the agent down.
### Harness File Tools
`FileTools(backend)` gives the model seven tools over any `FileBackend`: `ls`, `read_file`,
`write_file`, `edit_file`, `delete`, `glob` and `grep` ([design §4.32](https://github.com/llm4s/llm4s/blob/main/docs/design/typed-agent-runtime/4.32-stage-3-slice-f1-filesystem-tools.md)).
(not compile-checked)
```scala
import org.llm4s.agent.graph.tool.ToolSet
import org.llm4s.agent.harness.{FileToolLimits, FileTools, StateFileBackend}
for {
tools <- ToolSet.of(FileTools(StateFileBackend(), FileToolLimits(maxResultChars = 20000))*)
agent <- Agent.builder("coder", client).withTools(tools).build()
} yield agent
```
- **Paths are absolute** (`/src/main.scala`); `.`, `..` and empty segments are refused.
- **`read_file`** numbers lines from 1 and pages with `offset` (the line to start at) and `limit`
(default and at most `readLimit`, 2000), and says how many lines the file has. **`edit_file`**
replaces `old_string` with `new_string`, and refuses when `old_string` is absent, or occurs more
than once without `replace_all`.
- **Bounded results.** Lines are cut at `maxLineChars`; `ls`, `glob` and `grep` stop at `maxResults`
and say so; every result a tool returns, an error's included, is at most `maxResultChars` characters (an outcome the loop or a middleware gives instead, such as a denial, is theirs).
- **Order in one message.** Mutations take the backend's locks, so two edits of one file in one
message apply in order, the second against the first's result. Reads take no lock: a read in the
same message as a write may not see it (over thread state it sees the file as before the message).
- **Permissions.** Each tool declares its resources: `Read` of the path for `ls` and `read_file`,
`Write` for `write_file` and `edit_file`, `Delete` for `delete`. `glob` and `grep` declare the
envelope of what they may read (`/src/**`, `/src/*.md`), which a `PermissionMiddleware` decides as
a whole, so a deny rule beneath the searched path denies the search. `ls` shows the names, not the
contents, of entries beneath a path it may read.
- **Block and replay.** An output Block restores state files with the rest of the turn; writes to a
store are not undone. A store mutation that runs again after a crash replays its recorded outcome.
- **Not with the host file tools.** `llm4s-agent-tools`' `ReadFileTool` and `WriteFileTool` have the
names `read_file` and `write_file`, and a `ToolSet` refuses duplicate names: use one suite or the
other.
- `FileToolsContract` (`llm4s-agent-testkit`) runs the tools over your own `FileBackend`.
### Workspace Sandbox
`llm4s-workspace-client` puts a containerised workspace behind both backend SPIs: `WorkspaceFileBackend`
for its files and `WorkspaceSandboxBackend` for its commands (`org.llm4s.agent.harness.workspace`). The
`execute` tool (`ExecuteTool`, in `llm4s-agent`) runs a command in any `SandboxBackend`:
(not compile-checked)
```scala
import org.llm4s.agent.harness.ExecuteTool
import org.llm4s.agent.harness.workspace.{WorkspaceFileBackend, WorkspaceSandboxBackend}
val workspace = new ContainerisedWorkspace(dir, image, port) // started with startContainer()
for
files <- WorkspaceFileBackend("ws1", workspace) // a FileBackend for the file tools
sandbox <- WorkspaceSandboxBackend("ws1", workspace)
agent <- Agent.builder("coder", client)
.withTools(ToolSet.of(ExecuteTool(sandbox)).toOption.get)
.build()
yield agent
```
- **Permissions.** An `execute` call touches `(Execute, )` and is `destructive` and
`openWorld`, so `PermissionRule.allow(Operation.Execute, "sbt *")` lets the agent run builds and
nothing else. The runner refuses anything its command policy forbids as well
([Workspace sandbox](../../reference/workspace-sandbox.html)), and enforces `allowedPaths`,
`readOnlyPaths` and its size limits itself.
- **Order.** A file mutation and a command both lock the whole workspace (`files/workspace/`), so a
batch's edits and commands in one workspace apply one after another, in call order. Two paths that name
one file through a link are ordered too.
- **Replay.** File mutations are `Verifiable`: each is a compare-and-set against the file's hash, which
the runner applies atomically, so a replayed edit that landed returns its first result and one whose
file changed meanwhile is refused. Commands are `Unverifiable`: a completed command replays its stored
result, and one that may have run is refused, never run again.
- **Effects stay.** What a command or a file write did to the workspace is outside thread state: a Block
that retracts the turn does not undo it. The runner never runs a file change beside a command it started,
but a process a command leaves running, or a writer outside the container, is outside the compare-and-set.
- **Tenants.** One workspace is one tenant's. An application serving several builds a pair per tenant
and picks it with `FileBackend.resolved` and
`SandboxBackend.resolved(_ => Set(WorkspaceLocks.AllWorkspaces))`.
### Tool Permissions, Timeouts and Locks
An `AgentToolSpec` can declare what a call touches, how long one run may take, and which locks it
holds, each from the call's decoded arguments:
(not compile-checked)
```scala
import org.llm4s.agent.graph.middleware.{PermissionMiddleware, PermissionRule}
import org.llm4s.agent.graph.tool.{LockKey, Operation, ToolResource}
import scala.concurrent.duration.*
val writeSpec = baseSpec
.withResources(args => Vector(ToolResource(Operation.Write, args.path)))
.withLocks(args => Set(LockKey("workspace", workspaceId, args.path)))
.withTimeout(30.seconds)
val agent = Agent.builder("coder", client)
.withTools(tools)
.withMiddleware( // register it first, so it wraps every other middleware
new PermissionMiddleware(Seq(
PermissionRule.deny(Operation.Write, "/etc/**"),
PermissionRule.ask(Operation.Write, "/deploy/**"),
PermissionRule.allow(Operation.Write, "/src/**"),
PermissionRule.allow(Operation.Invoke, "*")
))
)
.build()
```
**Permissions are deny-if-unmatched.** For each resource the first rule with its operation whose
`Glob` matches the target decides, and a resource no rule matches is denied. A call runs only if
every resource is allowed: a denial is the error result `Denied: `, an `Ask`
suspends the call for approval. A tool that declares no resources touches `(Invoke, )`,
so it needs an `Invoke` rule. Resources are the call's whole envelope: a tool that asks a question and lets
the answer choose its target declares a resource covering every target it may pick (`/out/**`) and
refuses an answer outside it, because the check runs on the declaration, not on what the tool does. Such an envelope
is checked for every target in it: a deny rule reaching any part of it denies the call, and it is
allowed only when allow rules cover all of it. Approval is one flag per call: once a reviewer approves the call,
every `Ask` passes. An edited call is checked again on its new arguments, so a deny rule still
refuses it. Rules see the arguments after the schema check and decoding. In a `Glob`, `*` and `?`
stay within one `/`-separated segment and `**` crosses them. An `Execute` rule limits which
commands the agent may ask to run, not what a command does once it runs: rely on the sandbox for
that.
**A timeout bounds each run of the tool.** A run past it is interrupted and the model sees the
error `Tool '' timed out after ms`; the run continues. The tool runs on a thread of its
own, and the call waits for that thread to finish, so **block interruptibly**: a tool that ignores
its interrupt holds its whole batch past the timeout. A timed-out call may have done part of its
work: it keeps its idempotency key if it runs again (a retrying middleware, `recover`), but if the
model calls the tool again, that is a new call with a new key. Cancelling the run still cancels the
call, which records nothing.
**Locks order conflicting calls of one batch.** Two `LockKey`s conflict when one is a prefix of
the other, segment by segment: `workspace/w1` conflicts with `workspace/w1/src/a.scala`, and
`a/b` does not conflict with `a/bc`. When a model message has several calls, a call whose locks
conflict with an earlier call's waits for it and runs in a later superstep, in call order, seeing
what the earlier call committed. So a check against thread state (a quota, a file's content) is
exact, and two calls writing one key no longer lose one of the writes. Calls without conflicting
locks still run in parallel. Each further round costs two supersteps of `RunBudgets.maxSupersteps`.
A call that waits for approval holds back the calls queued behind it, so a reviewer can see a
batch's approvals in more than one round. Locks order calls within one batch only, not across
threads. An approval edit may narrow a call's locks (`/prod` to `/prod/app`) but not take a lock
its proposed arguments did not cover: such an edit is refused, the call does not run, and the model
sees why.
### Todos
`TodoMiddleware` gives an agent a plan: a `write_todos` tool that replaces the thread's todo list,
and a prompt section that shows the list to the model on every model step.
(not compile-checked)
```scala
import org.llm4s.agent.harness.{HarnessEvents, TodoMiddleware}
for {
agent <- Agent.builder("researcher", client)
.withMiddleware(new TodoMiddleware(maxTodos = 20, maxTodoChars = 200))
.build() // Result[Agent]
run <- agent.stream(threadId, "Write a report on ...") {
case HarnessEvents.TodosProgress(p) =>
p.todos.foreach(t => println(s"[${t.status.wire}] ${t.content}"))
case _ => ()
}
result <- run.await()
} yield result
```
- **The model sends the whole list** - each item's `content`, `status` (`pending`, `in_progress`,
`done`) and, to keep an item, its `id` - and it replaces the plan. The tool assigns ids: an item
without one keeps the id of an item with the same content, or gets a new one, and an id is never
reused within a thread. An unknown id, an id given twice, a list over `maxTodos` (default 50) or an
item over `maxTodoChars` code points (default 200) is refused as an error the model sees.
- **The plan is thread state**, in `TodoMiddleware.key` (a `TodoList`): it is checkpointed, read
by the next turn, and restored by an output Block with the rest of the turn. Two `write_todos`
calls in one message run in turn (the tool locks `todos`), the second seeing the first's list.
- **Events.** Each change emits a durable `HarnessEvents.TodosUpdated` with ids and statuses only,
and a live `HarnessEvents.TodosProgress` with the whole list.
- **The plan guides the model only.** It changes no route and no permission: a
`PermissionMiddleware` needs `PermissionRule.allow(Operation.Invoke, "write_todos")` for the tool.
- **Adding it to an existing agent reshapes its graph** (it registers the `todos` key, and the
tool's lock registers `tool-queue`), so threads checkpointed before are refused with
`RestoreRejected`: finish or `forget` them first.
### Skills
`SkillsMiddleware` gives an agent [Agent Skills](https://agentskills.io/): folders holding a
`SKILL.md` whose frontmatter names and describes the skill. It finds them through a `FileBackend`
and lists each skill's name, description and path in one prompt section; the model reads a skill's
`SKILL.md`, and its other files, with `read_file` when a task matches it.
(not compile-checked)
```scala
import org.llm4s.agent.harness.{SkillsMiddleware, VPath}
for {
user <- VPath("/user/skills")
project <- VPath("/project/skills")
agent <- Agent.builder("coder", client)
.withMiddleware(new SkillsMiddleware(Vector(user, project), backend))
.build() // Result[Agent]
} yield agent
```
- **Give the agent `FileTools(backend)` over the same backend** (or the composite it is mounted in),
so the paths in the section are the ones `read_file` takes.
- **Read once per run**, on its first model call, and cached for the run. A skill added or edited
during a run shows from the next run; a recovered or resumed run reads again. A later source's
skill replaces an earlier one of the same name, so list the user's skills before the project's.
- **Bounded.** At most `maxSkills` (default 20) skills, each description cut at
`maxDescriptionChars` (256), the section at most `maxSectionTokens` (1500), at most `maxScanned`
(100) folders examined, and only each `SKILL.md`'s frontmatter read.
- **The frontmatter is a restricted YAML subset**, read without a YAML library: `key: value` scalars,
quoted strings, flat lists and a one-level `metadata` map. A block scalar (`description: |`), a
nested value or a name that does not match its folder makes the skill malformed.
- **Problems never fail the run.** A malformed, unlisted, shadowed or unreadable skill is skipped and
named by path in a durable `HarnessEvents.DocumentsReported`. A missing source is skipped.
- `allowed-tools` is parsed onto `Skill.allowedTools`, not enforced.
### Instruction Memory
`InstructionMemoryMiddleware` shows standing instructions kept in `AGENTS.md` files to the model, as
one prompt section: the files at the paths given, read through a `FileBackend` in order (a later
file refines an earlier one), each under a heading naming its path.
(not compile-checked)
```scala
import org.llm4s.agent.harness.{InstructionMemoryMiddleware, VPath}
for {
user <- VPath("/user/AGENTS.md")
project <- VPath("/project/AGENTS.md")
agent <- Agent.builder("coder", client)
.withMiddleware(new InstructionMemoryMiddleware(Vector(user, project), backend))
.build() // Result[Agent]
} yield agent
```
- **Read once per run**, as skills are: an edit during a run shows from the next run, and a recovered
or resumed run reads again. A missing file is skipped.
- **Editable when permitted.** The agent edits a file with `edit_file` (`FileTools(backend)`) over the same backend when the
backend accepts the write and the permission rules allow `Write` on its path; deny it to keep a
scope read-only.
- **Bounded.** Each file at most `maxFileTokens` (default 1000), the section at most
`maxTotalTokens` (2000); a file over either is cut and marked, files past a full section are left
out, and each is named by path in a durable `HarnessEvents.DocumentsReported`.
- Instruction memory is a few whole files the agent always sees; [semantic memory](memory) is facts
recalled per query.
### Harness Profile
`HarnessProfile` assembles the pieces above into one value, installed with `withHarness`:
(not compile-checked)
```scala
import org.llm4s.agent.graph.middleware.PermissionRule
import org.llm4s.agent.graph.tool.Operation
import org.llm4s.agent.harness.{HarnessProfile, PromptMode}
import org.llm4s.config.HarnessConfigLoader
for {
profile <- HarnessConfigLoader.load() // HarnessProfile.deepAgents, with llm4s.agent.harness applied
agent <- Agent.builder("researcher", client)
.withHarness(
profile
.withSystemPrompt(PromptMode.Suffix("Write the report to /report.md."))
.withPermissions(Vector(
PermissionRule.allow(Operation.Read, "/**"),
PermissionRule.allow(Operation.Write, "/report.md"),
PermissionRule.allow(Operation.Invoke, "write_todos")
))
)
.build() // Result[Agent]
} yield agent
```
- **`HarnessProfile.deepAgents`** turns everything on at its defaults: the file tools over a
`StateFileBackend`, todos, summarisation (the model-free digest compressor), offload of large tool
results, skills under `/skills` and instructions in `/AGENTS.md`. `HarnessProfile()` is the file
tools alone. Change either with the `with*` setters: `withBackend`, `withSandbox` (adds `execute`),
`withTools(ToolSelection.Only(...))`, `withTodos(None)`, `withToolTimeout`, `withRetention`, ...
- **Subagents.** `deepAgents` also offers the general-purpose subagent through the `task` tool;
`HarnessProfile()` offers none. See [Subagents in a harness](delegation#subagents-in-a-harness).
- **The order is fixed**, outermost first: permissions, subagent delegation, summarisation, offload,
skills, instructions, todos, then the builder's own middleware. With no permission rules there is no
`PermissionMiddleware`; with rules, a call no rule allows is denied.
- **The prompt** is the profile's (plan, act and verify; context; tool safety), `PromptMode.Replace`
or `Suffix`. `withSystemPrompt` beside a harness is refused.
- **Retention.** As each turn is admitted, the thread is pruned to its latest 20 checkpoints (each is
a full snapshot of the files, plan and summary), so a finished turn stays whole until the next one
starts, and a refused turn changes nothing; `withRetention(RetentionPolicy.keepAll)` keeps
everything. `RunConfig.withRetention` does the same for any run.
- **Defaults live in `llm4s-agent`'s `reference.conf`**, under `llm4s.agent.harness`; override them in
`application.conf`. Backends, a sandbox, permission rules and a model compressor are set in code.
- **Middleware order is checked.** A middleware that must run outside another says so
(`AgentMiddleware.placement`), and `build()` refuses the wrong order: a `SummarizationMiddleware`
inside a `ToolOutputOffload`, or a `SemanticMemoryMiddleware` inside guardrail middleware.
---
## Features
### [Guardrails](guardrails)
Validate inputs and outputs for safety. Guardrails are middleware on the agent:
(not compile-checked)
```scala
import org.llm4s.agent.graph.middleware.GuardrailMiddleware
import org.llm4s.agent.guardrails.builtin._
val agent = Agent.builder("assistant", client)
.withMiddleware(
new GuardrailMiddleware(
input = Seq(new LengthCheck(1, 10000), new ProfanityFilter()),
output = Seq(new JSONValidator())
)
)
.build()
```
A blocked run ends `AgentStatus.Blocked(guardrail, reason)`, the blocked turn removed from the thread, which stays usable.
[Learn more about guardrails →](guardrails)
### [Memory System](memory)
Persistent context across conversations:
(not compile-checked)
```scala
import org.llm4s.agent.memory._
val result = for {
manager <- SimpleMemoryManager.empty
m1 <- manager.recordUserFact("Prefers Scala", Some("user-1"), Some(0.9))
context <- m1.getRelevantContext("programming preferences")
} yield context
```
`SemanticMemoryMiddleware`, in `llm4s-agent-memory`, recalls a tenant's memories before each turn
and records the turn after it, safely across `recover`:
[Semantic Memory Middleware](memory#semantic-memory-middleware).
[Learn more about memory →](memory)
### [Handoffs](handoffs)
Delegate to specialist agents:
(not compile-checked)
```scala
import org.llm4s.agent.Handoff
val physics = Agent.builder("physics", client)
.withSystemPrompt("You are a physicist")
val agent = Agent.builder("triage", client)
.withHandoffs(Handoff.to("physics", physics, "Physics expertise required"))
.build()
```
[Learn more about handoffs →](handoffs)
### [Human Review](human-review)
Pause a turn for a person: tool approvals, questions from tools and from middleware - to review or edit
an answer, or for missing information - and static breakpoints before or after an agent's model calls,
tool calls and final answer. Answer any subset of what a turn waits for; the rest stays pending.
### [Streaming Events](streaming)
`Agent.builder(...).withStreaming()` streams the model's answer, and
`agent.stream(threadId, query)(listener)` delivers every event of the turn - text deltas, tool
calls, model calls, handoffs, guardrail blocks - as a `StreamEvent`, matched with `AgentEvents`.
Durable events carry no message content; content is live-only. `AgentIO.stream` and `AgentZ.stream`
give the same events as fs2 and ZIO streams.
[Learn more about streaming →](streaming)
### [Durable Checkpointers](durable-checkpointers)
Every run claims its thread in the `Checkpointer` before it starts, renews the claim while it runs, and fences
every commit with the claim's token. Runtimes sharing one store - in one process or many - are refused a live
run's thread with `ThreadBusy`, can `recover` a run whose process died once its claim has expired, and a run that
lost its claim records nothing more. `SqliteCheckpointer` (`llm4s-agent-checkpoint-sqlite`) keeps threads in a
SQLite file, so a run survives a restart and `recover` continues it in the next process. `CheckpointerContract` in
`llm4s-agent-testkit` tests a store of your own.
[Learn more about durable checkpointers →](durable-checkpointers)
### [Running agents on several machines](multiple-machines)
Several instances of one agent service share threads through one PostgreSQL database: runs are submitted to a durable
queue and executed by whichever instance's `AgentWorker` takes them, a run whose machine dies is recovered by
another, any instance can follow or cancel any run, and a message for a busy thread is rejected, queued or
interrupts the running turn, as the caller chooses.
[Learn more about running on several machines →](multiple-machines)
---
## Built-in Tools
The `llm4s-agent-tools` module ships ready-made tools - a calculator, the date and time, UUIDs, JSON, files, HTTP, a
shell and web search - in bundles you register with an agent. They work with plain tool calling through
`ToolRegistry` too, without an `Agent`. The [Built-in Tools guide](/guide/builtin-tools) lists every tool and its
parameters, and says what each bundle lets a model do: read its safety section before you give a model files, the
network or a shell. For the module itself, see
[installation](/getting-started/installation#for-built-in-tools-web-search-http-filesystem-shell).
(not compile-checked)
```scala
import org.llm4s.agent.Agent
import org.llm4s.toolapi.ToolRegistry
import org.llm4s.toolapi.builtin.BuiltinTools
val result = for {
tools <- BuiltinTools.coreSafe // date/time, calculator, UUID, JSON: no files, network or processes
agent <- Agent.builder("assistant", client).withTools(new ToolRegistry(tools)).build()
state <- agent.run("What is 15% of 850?")
} yield state
```
The bundles, smallest to largest: `coreSafe`, `withHttpSafe()` (adds read-only HTTP), `withFilesSafe()` (adds
read-only files) and `developmentSafe()` (adds file writing and a shell). `customSafe(...)` builds exactly the tools
whose configuration you pass.
---
## Context Window Management
Handle long conversations with `ContextWindowMiddleware`. It prunes what is sent to the model on
each call; the thread keeps its full history:
(not compile-checked)
```scala
import org.llm4s.agent.{ContextWindowConfig, PruningStrategy}
import org.llm4s.agent.graph.middleware.ContextWindowMiddleware
val config = ContextWindowConfig(
maxMessages = Some(20),
preserveSystemMessage = true,
minRecentTurns = 2,
pruningStrategy = PruningStrategy.OldestFirst
)
val agent = Agent.builder("assistant", client)
.withMiddleware(new ContextWindowMiddleware(config))
.build()
```
The current turn (the latest user message onward) is never pruned, a tool call is never separated
from its result, and the request always starts with a user message - so a request may exceed the
budget. The system prompt is outside the budget.
**Pruning Strategies:**
| Strategy | Behavior |
|----------|----------|
| `OldestFirst` | Remove oldest messages first (FIFO) |
| `MiddleOut` | Keep first and last messages, remove middle |
| `RecentTurnsOnly(n)` | Keep only the last N conversation turns |
| `Custom(fn)` | User-defined pruning function |
### Summarisation
`SummarizationMiddleware` (`org.llm4s.agent.harness`) replaces the older part of each request with a summary
instead of dropping it. When the outgoing request is over its trigger, every message but the last `keepRecent`
is folded into the summary, which goes into that request and is kept in the thread for the next ones; when it
fires again, the summary is extended, not recomputed. The stored history is never edited.
(not compile-checked)
```scala
import org.llm4s.agent.harness.{SummarizationMiddleware, SummaryCompressor, SummaryTrigger}
val agent = Agent.builder("assistant", client)
.withMiddleware(
new SummarizationMiddleware(
trigger = SummaryTrigger.Tokens(60000),
keepRecent = 10,
compressor = SummaryCompressor.model(summariserClient),
maxSummaryTokens = 2000
)
)
.build()
```
`SummaryCompressor.model(client)` asks a model for the summary; its calls are counted in the thread's usage.
`SummaryCompressor.digest(counter)` uses core's `HistoryCompressor`, with no model call. The summary never exceeds
`maxSummaryTokens`, a tool call is never separated from its results, and an output Block discards a blocked turn's
summary. Use summarisation or `ContextWindowMiddleware`, not both.
### Large tool results
`ToolOutputOffload` keeps a tool result over `thresholdTokens` out of the conversation: the model sees a header,
the first lines, and a path such as `/large_tool_results/`. The whole result (up to `maxResultBytes`) is kept
in the thread, oldest evicted first beyond `maxTotalBytes` (4 MiB by default). Mount `OffloadBackend` so file tools
can read it:
(not compile-checked)
```scala
import org.llm4s.agent.graph.tool.ToolSet
import org.llm4s.agent.harness.{CompositeBackend, FileTools, OffloadBackend, StateFileBackend, ToolOutputOffload}
for {
files <- CompositeBackend(StateFileBackend(), ToolOutputOffload.Mount -> OffloadBackend)
tools <- ToolSet.of(FileTools(files)*)
agent <- Agent.builder("assistant", client)
.withTools(tools)
.withMiddleware(new ToolOutputOffload(thresholdTokens = 20000))
.build()
} yield agent
```
The mount is read-only. Only text is offloaded: images and documents in a result are passed on unchanged. With
summarisation, results the summary covers are dropped; register `SummarizationMiddleware` before
`ToolOutputOffload`. `ToolOutputOffload` also caps every tool result the model is sent at `thresholdTokens`,
including errors and denials that no tool produced.
---
## Conversation Persistence
A thread lives in the runtime. To keep a conversation across processes, save the result's messages
and import them as the `history` of a new thread:
(not compile-checked)
```scala
import upickle.default.{read, write}
// Save the messages of a completed run
Files.writeString(path, write(result.messages))
// Later: load them and continue on a new thread
val saved = read[Vector[Message]](Files.readString(path))
val next = agent.run(ThreadId(java.util.UUID.randomUUID().toString), "Continue our conversation", RunConfig(), saved)
```
`history` is accepted only on a thread that does not exist yet, and may not contain system messages
(prompts belong to the agents). Save only runs that completed; `history` in which a tool call does
not have exactly one result straight after its message - one that ends mid tool call, say - is
refused.
---
## Reasoning Modes
Enable extended thinking for complex problems:
(not compile-checked)
```scala
import org.llm4s.llmconnect.model.{CompletionOptions, ReasoningEffort}
val options = CompletionOptions()
.withReasoning(ReasoningEffort.High) // None, Low, Medium, High
.withMaxTokens(4096)
// Use with agent
Agent.builder("assistant", client).withCompletionOptions(options).build()
```
Supported by OpenAI o1/o3 and Anthropic Claude models.
---
## Examples
| Example | Description |
|---------|-------------|
| [SingleStepAgentExample](/examples/#single-step) | A run with one tool call |
| [MultiStepAgentExample](/examples/#multi-step) | Complete execution flow |
| [MultiTurnConversationExample](/examples/#multi-turn) | Functional multi-turn API |
| [LongConversationExample](/examples/#long-conversation) | Context window pruning |
| [ConversationPersistenceExample](/examples/#persistence) | Save and resume |
| [BuiltinToolsAgentExample](/examples/#agent-examples) | Built-in tools |
| [DeepResearchExample](/examples/#deep-research) | The harness on state files: plan, offload, summary, memory, recover after a stop |
| [CodingWorkspaceExample](/examples/#coding-workspace) | The harness in a workspace: permissions, approval of `execute`, resume |
| [ParallelDeepResearchExample](/examples/#parallel-deep-research) | Typed subagents in parallel: bounds, a child's approval through the lead, nested streams, recover after a stop |
[Browse all examples →](/examples/)
---
## Design Documents
For in-depth technical details:
- [Agent Framework Roadmap](/design/agent-framework-roadmap) - Strategic direction
- [Phase 1.1: Conversations](/design/phase-1.1-functional-conversation-management) - Conversation API
- [Phase 1.2: Guardrails](/design/phase-1.2-guardrails-framework) - Validation framework
- [Phase 1.3: Handoffs](/design/phase-1.3-handoff-mechanism) - Agent delegation
- [Phase 1.4: Memory](/design/phase-1.4-memory-system) - Memory architecture
- [Typed Agent Runtime §4.13](https://github.com/llm4s/llm4s/blob/main/docs/design/typed-agent-runtime/4.13-stage-1-slice-2-the-agent-loop-on-the-graph-runtime.md) - How `Agent` runs on the graph runtime
- [Phase 2.1: Streaming](/design/phase-2.1-streaming-events) - Event system
---
## Next Steps
1. **[Guardrails Guide](guardrails)** - Input/output validation
2. **[Memory Guide](memory)** - Persistent context
3. **[Handoffs Guide](handoffs)** - Agent delegation
4. **[Streaming Guide](streaming)** - Real-time events
5. **[Durable Checkpointers](durable-checkpointers)** - Shared stores, claims and recovery
6. **[Running agents on several machines](multiple-machines)** - One service, several instances, one Postgres
7. **[Examples Gallery](/examples/)** - Working code samples
---
Source: https://llm4s.org/guide/agents/guardrails
# Guardrails
Validate agent inputs and outputs for safety, quality, and compliance.
---
## Overview
Guardrails are validation functions that run before (input) and after (output) agent processing. They help ensure:
- **Safety** - Block harmful or inappropriate content
- **Quality** - Enforce response standards
- **Compliance** - Meet business requirements
- **Security** - Detect prompt injection and PII
Guardrails belong to the agent, not to a run: give them to the builder in a `GuardrailMiddleware`.
(not compile-checked)
```scala
import org.llm4s.agent.Agent
import org.llm4s.agent.graph.middleware.GuardrailMiddleware
val agent = Agent.builder("assistant", client)
.withTools(tools)
.withMiddleware(
GuardrailMiddleware(
input = Seq(...), // Validate each query before the LLM call
output = Seq(...) // Validate each final answer
)
)
.build()
```
A guardrail that refuses does not make `run` return a `Left`: the turn ends with
`AgentStatus.Blocked(guardrail, reason)` - the first failing guardrail's name, and every failure's
error - and the thread stays usable, so the next `run` or `continueConversation` on it works. The
blocked turn is never kept: an input block stores nothing for that turn, and an output block removes
the whole turn - the query, any tool calls and results, and the answer - so `result.messages` is the
conversation as it was before the turn and the model never sees the blocked answer again. `usage`
still counts the blocked turn's model calls. A guardrail that transforms (`PIIMasker`) changes the
stored query or answer instead.
Underneath, a block is the graph runtime's *Block*: the run ends as a finished failure
(`RunResult.Failed` with a `GuardrailBlocked` error, the thread's checkpoint `Failed`), which
`recover` has nothing to continue and `start` accepts like a completed thread. A `Left` from another
middleware's `beforeAgent` or `afterAgent` blocks the same way, but is returned as that `Left`.
A list of guardrails runs completely: each guardrail, in order, on the value the previous one
returned, **even after one has failed**. That is why the block carries every failure's error, and why a
guardrail that costs a call - an LLM judge - runs on every turn that reaches its list. To stop at the first
failure, use [`CompositeGuardrail.sequential`](#sequential-short-circuit) (adapted with `InputGuardrail.from`,
see [Using a composite in an agent](#using-a-composite-in-an-agent)); the block then names the composite as the
guardrail. Running the whole list is the settled default ([#1521](https://github.com/llm4s/llm4s/issues/1521)): it
reports every reason a value was refused in one block, and an early stop is one composite away, while the opposite
default would hide the other failures with no way to ask for them.
A block sends one durable `agent.guardrail_blocked` event, naming the first failing guardrail and its phase
(`Input` or `Output`), and no event comes from a guardrail that passed or that ran after the first failure.
The run then ends with the kernel's `RunFailed`; `RunCompleted` is never sent for a blocked turn. An input
block comes before any model call. An output block comes after the turn's `ModelCallCompleted` events, since
the answer it refuses has been produced. See [Streaming](streaming) for the event stream.
Guardrails on the root agent guard its whole handoff family: they apply to every turn's query and
every final answer, whichever agent is active after a handoff. See
[Guardrails and Middleware Across Handoffs](handoffs#guardrails-and-middleware-across-handoffs).
---
## Built-in Guardrails
### Simple Validators
These guardrails run locally without LLM calls:
| Guardrail | Purpose | Example |
|-----------|---------|---------|
| `LengthCheck` | Enforce min/max length | `new LengthCheck(1, 10000)` |
| `ProfanityFilter` | Block profane content | `new ProfanityFilter()` |
| `JSONValidator` | Ensure valid JSON output | `new JSONValidator()` |
| `RegexValidator` | Pattern matching | `new RegexValidator("\\d{3}-\\d{4}")` |
| `ToneValidator` | Simple tone detection | `new ToneValidator(Tone.Professional)` |
| `PIIDetector` | Detect PII (email, SSN, etc.) | `new PIIDetector()` |
| `PIIMasker` | Mask detected PII | `new PIIMasker()` |
| `PromptInjectionDetector` | Detect injection attempts | `new PromptInjectionDetector()` |
### LLM-as-Judge Guardrails
These use an LLM to evaluate subjective qualities:
| Guardrail | Purpose | Example |
|-----------|---------|---------|
| `LLMSafetyGuardrail` | Content safety check | `new LLMSafetyGuardrail(client)` |
| `LLMFactualityGuardrail` | Verify factual accuracy | `new LLMFactualityGuardrail(client)` |
| `LLMQualityGuardrail` | Assess response quality | `new LLMQualityGuardrail(client)` |
| `LLMToneGuardrail` | Validate tone compliance | `new LLMToneGuardrail(client, "professional")` |
Each judge scores the content and passes it when the score reaches its `threshold`. The threshold must be between 0.0 and 1.0: for any other value, or NaN, `validate` returns a `ValidationError` on the field `threshold` without calling the judge.
### RAG-Specific Guardrails
For retrieval-augmented generation:
| Guardrail | Purpose |
|-----------|---------|
| `GroundingGuardrail` | Verify answers are grounded in retrieved context |
| `ContextRelevanceGuardrail` | Check context relevance to query |
| `SourceAttributionGuardrail` | Ensure sources are cited |
| `TopicBoundaryGuardrail` | Prevent off-topic responses |
---
## Basic Usage
### Input Validation
(not compile-checked)
```scala
import org.llm4s.agent.{ Agent, AgentStatus }
import org.llm4s.agent.graph.middleware.GuardrailMiddleware
import org.llm4s.agent.guardrails.builtin._
val result = for {
agent <- Agent.builder("assistant", client)
.withTools(tools)
.withMiddleware(
GuardrailMiddleware(
input = Seq(
new LengthCheck(min = 1, max = 10000),
new ProfanityFilter(),
new PromptInjectionDetector()
),
output = Nil
)
)
.build()
result <- agent.run(userInput)
} yield result
result match {
case Right(r) =>
r.status match {
case AgentStatus.Blocked(guardrail, reason) => println(s"Input rejected by $guardrail: $reason")
case AgentStatus.Completed(answer) => println(answer)
case other => println(s"Run ended: $other")
}
case Left(error) =>
println(s"Run failed: ${error.message}")
}
```
### Output Validation
(not compile-checked)
```scala
val agent = Agent.builder("assistant", client)
.withTools(tools)
.withMiddleware(GuardrailMiddleware(input = Nil, output = Seq(new JSONValidator(), new PIIMasker())))
.build()
val result = agent.flatMap(_.run("Generate a JSON response with user data"))
```
### Combined Input/Output
(not compile-checked)
```scala
val agent = Agent.builder("assistant", client)
.withTools(tools)
.withMiddleware(
GuardrailMiddleware(
input = Seq(
new LengthCheck(1, 5000),
new ProfanityFilter()
),
output = Seq(
new LLMSafetyGuardrail(client),
new ToneValidator(Set(Tone.Professional))
)
)
)
.build()
val result = agent.flatMap(_.run(userInput))
```
---
## LLM-as-Judge Examples
### Safety Check
(not compile-checked)
```scala
import org.llm4s.agent.guardrails.builtin.LLMSafetyGuardrail
val safetyGuardrail = new LLMSafetyGuardrail(client)
val agent = Agent.builder("storyteller", client)
.withMiddleware(GuardrailMiddleware(input = Nil, output = Seq(safetyGuardrail)))
.build()
agent.flatMap(_.run("Write a story"))
```
### Factuality Check
Verify responses are grounded in source documents:
(not compile-checked)
```scala
import org.llm4s.agent.guardrails.builtin.LLMFactualityGuardrail
val factualityGuardrail = LLMFactualityGuardrail.strict(
client = client,
referenceContext = "The capital of France is Paris. Paris has a population of 2.1 million."
)
val agent = Agent.builder("geography", client)
.withMiddleware(GuardrailMiddleware(input = Nil, output = Seq(factualityGuardrail)))
.build()
agent.flatMap(_.run("What is the capital of France?"))
```
### Tone Validation
(not compile-checked)
```scala
import org.llm4s.agent.guardrails.builtin.LLMToneGuardrail
val toneGuardrail = new LLMToneGuardrail(
llmClient = client,
allowedTones = Set("professional", "helpful")
)
val agent = Agent.builder("support", client)
.withMiddleware(GuardrailMiddleware(input = Nil, output = Seq(toneGuardrail)))
.build()
agent.flatMap(_.run("Help with customer complaint"))
```
---
## Composite Guardrails
Combine multiple guardrails with different strategies:
### All Must Pass (AND)
(not compile-checked)
```scala
import org.llm4s.agent.guardrails.CompositeGuardrail
val strictValidation = CompositeGuardrail.all(Seq(
new LengthCheck(1, 5000),
new ProfanityFilter(),
new PIIDetector()
))
// All guardrails must pass for input to be accepted
```
### Any Must Pass (OR)
(not compile-checked)
```scala
val flexibleValidation = CompositeGuardrail.any(Seq(
new RegexValidator("^[A-Z].*"), // Starts with capital
new RegexValidator("^\\d.*") // Starts with digit
))
// At least one guardrail must pass
```
### Sequential (Short-Circuit)
(not compile-checked)
```scala
val sequentialValidation = CompositeGuardrail.sequential(Seq(
new LengthCheck(1, 10000), // Check length first
new ProfanityFilter(), // Then profanity
new PIIDetector() // Then PII
))
// Stops at first failure, more efficient
```
### Using a composite in an agent
A composite is a `Guardrail[String]`, while `GuardrailMiddleware` takes `InputGuardrail`s and
`OutputGuardrail`s, so adapt it with `InputGuardrail.from` or `OutputGuardrail.from`. A cast does not work: it
throws a `ClassCastException`.
(not compile-checked)
```scala
import org.llm4s.agent.graph.middleware.GuardrailMiddleware
import org.llm4s.agent.guardrails.InputGuardrail
val agent = Agent
.builder("assistant", client)
.withMiddleware(GuardrailMiddleware(Seq(InputGuardrail.from(sequentialValidation)), Seq.empty))
.build()
```
`from` keeps the guardrail's name and description and passes the hook's context on, so an LLM judge inside the
composite has its model calls counted in the thread's usage (a hand-written wrapper that overrides only
`validate(value)` loses that). An `InputGuardrail` passed to `InputGuardrail.from` comes back unchanged.
`all` runs every guardrail and reports every failure; `any` stops at the first guardrail that passes, so
those after it do not run; `sequential` stops at the first failure.
---
## Custom Guardrails
### Basic Custom Guardrail
(not compile-checked)
```scala
import org.llm4s.agent.guardrails.InputGuardrail
import org.llm4s.error.ValidationError
import org.llm4s.types.Result
class KeywordRequirementGuardrail(requiredKeywords: Set[String]) extends InputGuardrail {
val name: String = "keyword-requirement"
def validate(value: String): Result[String] = {
val found = requiredKeywords.filter(kw => value.toLowerCase.contains(kw.toLowerCase))
if (found.nonEmpty) {
Right(value)
} else {
Left(ValidationError.invalid(
"input",
s"Input must contain at least one of: ${requiredKeywords.mkString(", ")}"
))
}
}
}
// Usage
val guardrail = new KeywordRequirementGuardrail(Set("scala", "java", "kotlin"))
```
### Custom Output Guardrail
(not compile-checked)
```scala
import org.llm4s.agent.guardrails.OutputGuardrail
import org.llm4s.error.ValidationError
import org.llm4s.types.Result
class MaxSentenceCountGuardrail(maxSentences: Int) extends OutputGuardrail {
val name: String = "max-sentence-count"
def validate(value: String): Result[String] = {
val sentenceCount = value.split("[.!?]+").length
if (sentenceCount <= maxSentences) {
Right(value)
} else {
Left(ValidationError.invalid(
"output",
s"Response has $sentenceCount sentences, max allowed is $maxSentences"
))
}
}
}
```
### Custom LLM-Based Guardrail
(not compile-checked)
```scala
import org.llm4s.agent.guardrails.LLMGuardrail
import org.llm4s.llmconnect.LLMClient
class CustomLLMGuardrail(client: LLMClient) extends LLMGuardrail {
val name: String = "custom-llm-check"
val llmClient: LLMClient = client
val evaluationPrompt: String = "Rate if this content is appropriate for a children's website (0-1)"
override val threshold = 0.8
}
```
---
## RAG Guardrails
### Basic RAG Setup
(not compile-checked)
```scala
import org.llm4s.agent.guardrails.rag._
val ragGuardrails = RAGGuardrails.standard(client)
val agent = Agent.builder("rag", client)
.withTools(tools)
.withMiddleware(GuardrailMiddleware(ragGuardrails.inputGuardrails, ragGuardrails.outputGuardrails))
.build()
agent.flatMap(_.run(question))
```
### Preset Configurations
(not compile-checked)
```scala
// Minimal - basic safety only
val minimal = RAGGuardrails.minimal()
// Standard - balanced for production
val standard = RAGGuardrails.standard(client)
// Strict - maximum safety
val strict = RAGGuardrails.strict(client)
// Monitoring - warn mode, doesn't block
val monitoring = RAGGuardrails.monitoring(client)
```
### Individual RAG Guardrails
(not compile-checked)
```scala
// Verify answer is grounded in retrieved context
val grounding = new GroundingGuardrail(
client = client,
retrievedContext = retrievedDocuments
)
// Check retrieved context is relevant to query
val relevance = new ContextRelevanceGuardrail(
client = client,
query = userQuery
)
// Ensure sources are properly cited
val attribution = new SourceAttributionGuardrail(
client = client,
sourceDocuments = sources
)
// Prevent off-topic responses
val topicBoundary = new TopicBoundaryGuardrail(
client = client,
allowedTopics = Set("programming", "software engineering")
)
```
---
## PII Detection and Masking
### Detect PII
(not compile-checked)
```scala
import org.llm4s.agent.guardrails.builtin.PIIDetector
val piiDetector = new PIIDetector()
// Detects: emails, SSNs, credit cards, phone numbers, etc.
val agent = Agent.builder("assistant", client)
.withTools(tools)
.withMiddleware(GuardrailMiddleware(input = Seq(piiDetector), output = Nil))
.build()
agent.flatMap(_.run(userInput))
```
### Mask PII in Output
(not compile-checked)
```scala
import org.llm4s.agent.guardrails.builtin.PIIMasker
val piiMasker = new PIIMasker()
// Replaces PII with [REDACTED_EMAIL], [REDACTED_SSN], etc. - the stored answer is the masked one
val agent = Agent.builder("assistant", client)
.withTools(tools)
.withMiddleware(GuardrailMiddleware(input = Nil, output = Seq(piiMasker)))
.build()
agent.flatMap(_.run("Get user details"))
```
### Supported PII Types
| Type | Pattern | Masked As |
|------|---------|-----------|
| Email | `user@domain.com` | `[REDACTED_EMAIL]` |
| SSN | `123-45-6789` | `[REDACTED_SSN]` |
| Credit Card | `4111-1111-1111-1111`, `3782 822463 10005` (15-digit American Express) | `[REDACTED_CARD]` |
| Phone | `(555) 123-4567`, `+44 20 7946 0958` | `[REDACTED_PHONE]` |
| IP Address | `192.168.1.1` | `[REDACTED_IP]` |
A phone number is a US number (with an optional `+1` or `1` prefix) or an international number that starts with `+`
and has 8 to 15 digits, with spaces, dashes, dots or parentheses between them. Digits without a `+` are not treated
as an international number. A card number is recognised by its shape (a Visa, MasterCard, Amex or Discover prefix and
the right number of digits); the check digit is not validated. When two types match overlapping text (a plain card
number is also account-shaped, and a phone number can run into the email address after it), the overlapping text is
masked once, under the type whose match starts first. Detection does not merge them: `detect` returns one match per
type, so `summarize` counts that text once for each type (`123456789` gives `SSN -> 1, Passport -> 1` with
`PIIType.all`), and `PIIDetector`'s block message lists both types. A finding is one match of one type, so the sum of
the counts is how often the patterns fired, not how many stretches of PII the text holds.
---
## Prompt Injection Protection
(not compile-checked)
```scala
import org.llm4s.agent.guardrails.builtin.PromptInjectionDetector
val injectionDetector = new PromptInjectionDetector()
val agent = Agent.builder("assistant", client)
.withTools(tools)
.withMiddleware(GuardrailMiddleware(input = Seq(injectionDetector), output = Nil))
.build()
agent.flatMap(_.run(userInput))
// Detects patterns like:
// - "Ignore previous instructions..."
// - "System: You are now..."
// - "---\nNew instructions:"
// - Base64 encoded payloads
```
---
## Error Handling
### Blocked Turns
A block is an outcome, not an error: `run` returns `Right` with `AgentStatus.Blocked`, the blocked
turn absent from `result.messages`, and the thread can take another turn.
(not compile-checked)
```scala
agent.run(query) match {
case Right(result) =>
result.status match {
case AgentStatus.Blocked(guardrail, reason) =>
println(s"Guardrail '$guardrail' blocked the turn: $reason")
// Take appropriate action (rephrase, notify user, log); agent.continueConversation(result, ...) still works
case AgentStatus.Completed(answer) =>
println(answer)
case other =>
println(s"Run ended: $other")
}
case Left(error) =>
// a provider error, a tool's failure, a blank query, or another middleware's beforeAgent/afterAgent Left
println(s"Run failed: ${error.message}")
}
```
### Validation Mode
Control how guardrail results are handled:
(not compile-checked)
```scala
import org.llm4s.agent.guardrails.ValidationMode
// Block on failure (default)
val blocking = ValidationMode.Block
// Warn only, continue processing
val warn = ValidationMode.Warn
// Log and continue
val log = ValidationMode.Log
```
---
## Best Practices
### 1. Layer Your Guardrails
(not compile-checked)
```scala
// Fast, local checks first
val input = Seq(
new LengthCheck(1, 10000), // Cheapest first
new ProfanityFilter(), // Still fast
new PromptInjectionDetector(), // Pattern matching
new PIIDetector() // More complex but local
)
// LLM checks for output only (expensive)
val output = Seq(
new JSONValidator(), // Fast, local
new LLMSafetyGuardrail(client) // Expensive, last
)
val agent = Agent.builder("assistant", client)
.withMiddleware(GuardrailMiddleware(input, output))
.build()
```
### 2. Use Appropriate Guardrails for Each Use Case
| Use Case | Recommended Guardrails |
|----------|------------------------|
| Customer support | `ProfanityFilter`, `ToneValidator`, `LLMSafetyGuardrail` |
| Code generation | `LengthCheck`, `JSONValidator` (for structured output) |
| RAG application | `GroundingGuardrail`, `SourceAttributionGuardrail` |
| Content moderation | `PIIDetector`, `ProfanityFilter`, `LLMSafetyGuardrail` |
| Form processing | `RegexValidator`, `LengthCheck` |
### 3. Test Your Guardrails
(not compile-checked)
```scala
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers
class GuardrailSpec extends AnyFlatSpec with Matchers {
"LengthCheck" should "reject empty input" in {
val guardrail = new LengthCheck(min = 1, max = 100)
guardrail.validate("") shouldBe a[Left[_, _]]
}
it should "accept valid input" in {
val guardrail = new LengthCheck(min = 1, max = 100)
guardrail.validate("Hello") shouldBe Right("Hello")
}
}
```
---
## Examples
| Example | Description |
|---------|-------------|
| [BasicInputValidationExample](/examples/#basic) | Length and profanity checks |
| [JSONOutputValidationExample](/examples/#guardrails-examples) | JSON output validation |
| [LLMJudgeGuardrailExample](/examples/#guardrails-examples) | LLM-as-Judge patterns |
| [CompositeGuardrailExample](/examples/#composite) | Combining guardrails |
| [CustomGuardrailExample](/examples/#custom) | Building custom validators |
| [FactualityGuardrailExample](/examples/#guardrails-examples) | RAG factuality checking |
[Browse all examples →](/examples/)
---
## Next Steps
- [Memory Guide](memory) - Persistent context across conversations
- [Handoffs Guide](handoffs) - Agent-to-agent delegation
- [Streaming Guide](streaming) - Real-time execution events
---
Source: https://llm4s.org/guide/agents/memory
# Memory System
Persistent context and knowledge for agents across conversations.
---
## Overview
The LLM4S Memory System provides:
- **Short-term memory** - Conversation context within a session
- **Long-term memory** - Persistent facts and knowledge
- **Semantic search** - Find relevant context using embeddings
- **Entity tracking** - Remember information about people, places, things
- **Multiple backends** - In-memory, SQLite, vector stores
The snippets in this guide are kept in `MemoryGuideSpec`
(`modules/memory/src/test/scala/org/llm4s/agent/memory/MemoryGuideSpec.scala`), which compiles
them and checks what the text around them says. Snippets that need a live provider (an embedding
client, an agent) take it as a parameter, so they are compiled but not run against one. The
Postgres store lives in another module and is not covered there.
---
## Dependency
(not compile-checked)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-memory" % llm4sVersion
```
The Postgres backend is a separate artifact - see
[Postgres Memory Store](../../reference/postgres-memory-store). Package names are unchanged
either way, so `org.llm4s.agent.memory.*` imports keep working.
---
## Quick Start
(not compile-checked)
```scala
import org.llm4s.agent.memory._
val result = for {
// Record a user fact
m1 <- SimpleMemoryManager.empty.recordUserFact(
fact = "Prefers Scala over Java",
userId = Some("user-123"),
importance = Some(0.9)
)
// Record entity knowledge
m2 <- m1.recordEntityFact(
entityId = EntityId("anthropic"),
entityName = "Anthropic",
fact = "AI company that created Claude",
importance = Some(0.8)
)
// Get relevant context for a query
context <- m2.getRelevantContext("Tell me about Scala programming")
} yield context
result match {
case Right(ctx) => println(s"Context: $ctx")
case Left(err) => println(s"Error: $err")
}
```
Every write returns a `Result` holding the *updated* manager, so chain the calls and keep using
the manager you get back. This prints:
```
# Retrieved Context
## User Preferences
- Prefers Scala over Java
```
The in-memory store searches by **keyword**: it splits the query into phrases - its whitespace-separated
pieces, each of whose words must appear next to each other and in order - and returns the memories that
contain at least one of them as whole words, best first. A memory's score is the share of the query's
distinct phrases it contains, not of its words. "Prefers Scala over Java" contains the phrase "scala" from
the query, so it is returned; the Anthropic fact contains none, so it is not. A word never matches inside
a longer one, so `i` matches the word "I" but not "Berlin", and `prefer` does not match "Prefers" - there
is no stemming. Words are split and compared exactly as the SQLite stores' FTS5 index (`unicode61`) does,
so both kinds of store find the same memories for a query
([#1594](https://github.com/llm4s/llm4s/issues/1594)):
- A word is a run of letters, digits and private-use characters in any script. Anything else separates
words: punctuation (`java?` is the word `java`), and also combining marks such as Devanagari vowel
signs, so "मुझे" is the two words "म" and "झ".
- Case is ignored, and so is the accent of a Latin letter that has exactly one (`ECOLE` matches
`école`). Other accents are kept: `αθηνα` does not match "Αθήνα", `живет` does not match "Живёт",
`мои` does not match "Мой", and `viet` does not match Vietnamese "Việt", whose `ệ` has two.
- A query word with punctuation inside is a phrase: `berlin-based` matches "Berlin-based" but not
"based in Berlin".
The two kinds of store rank the memories they find differently (the SQLite store uses BM25). To retrieve
by meaning, use a vector store (see [Vector Store](#vector-store)).
---
## Memory Types
| Type | Purpose | Example |
|------|---------|---------|
| `Conversation` | Chat history | "User asked about weather" |
| `UserFact` | User preferences/info | "Prefers dark mode" |
| `Entity` | Knowledge about entities | "Paris is capital of France" |
| `Knowledge` | External knowledge | "Scala 3 released in 2021" |
| `Task` | Task outcomes | "Generated report successfully" |
| `Custom(name)` | Application-specific | Any custom memory type |
---
## Memory Manager
The `MemoryManager` is the main interface for working with memory. `SimpleMemoryManager` is the
standard implementation; `LLMMemoryManager` adds LLM-powered consolidation and entity extraction
(see [Memory Consolidation](#memory-consolidation) and [Entity Extraction](#entity-extraction)).
(not compile-checked)
```scala
import org.llm4s.agent.memory._
// In-memory store, default configuration
val manager = SimpleMemoryManager.empty
// A store and a configuration of your own
val configured = SimpleMemoryManager.withStore(
store = InMemoryStore.empty,
config = MemoryManagerConfig(defaultImportance = 0.7)
)
```
### Configuration Options
`MemoryManagerConfig` is built in code (this module reads no configuration file or environment
variable): `MemoryManagerConfig(...)` with any of the options below, or a preset
(`MemoryManagerConfig.default`, `.testing`, `.fullFeatured`), changed with its `with*` setters
(`withAutoExtractEntities`, `withDefaultImportance`, `withContextTokenBudget`,
`withConsolidationConfig`). It has no public constructor or `copy`:
| Option | Default | Description |
|--------|---------|-------------|
| `defaultImportance` | `0.5` | Importance given to a memory when you do not pass one |
| `contextTokenBudget` | `2000` | Token budget of `getRelevantContext` when the call passes no `maxTokens` |
| `autoExtractEntities` | `false` | `recordMessage` and `recordConversation` also run `extractEntities` over each user and assistant message |
| `consolidationConfig` | `ConsolidationConfig.default` | Group size and error handling of `consolidateMemories` |
Both `SimpleMemoryManager` and `LLMMemoryManager` read every option. An explicit `maxTokens` always
wins over `contextTokenBudget`. `autoExtractEntities` only changes what `LLMMemoryManager` stores,
because `SimpleMemoryManager.extractEntities` extracts nothing. Extraction is best-effort: if the
call fails, the message stays recorded and a warning is logged. A cancelled call is not a failure:
`recordMessage` returns the `CancelledError` (and `recordConversation` stops there), and an
interrupted thread keeps its interrupt flag.
> **`autoExtractEntities` costs one extra LLM call per recorded message.** With
> `MemoryManagerConfig.fullFeatured`, or `autoExtractEntities = true`, an `LLMMemoryManager` sends one
> entity-extraction request for every user and assistant message it records (system and tool
> messages are not analysed), on top of your own calls. Recording a 20-turn conversation with
> `recordConversation` is up to 20 more requests, paid for and waited on before it returns. Leave it
> off (the default) and call `extractEntities` yourself where you want entities.
(not compile-checked)
```scala
val config = MemoryManagerConfig(contextTokenBudget = 500, autoExtractEntities = true)
val tighter = config.withContextTokenBudget(250)
```
Managers record only what you pass to the `record*` methods, and nothing consolidates unless you
call `consolidateMemories`: the `autoRecordMessages` and `consolidationEnabled` options, which no
manager ever read, were removed in 0.5.0 ([#1579](https://github.com/llm4s/llm4s/issues/1579)).
---
## Recording Memory
Every `record*` method returns `Result[MemoryManager]`: the manager with the memory added.
### User Facts
(not compile-checked)
```scala
// Record a user preference
val result = manager.recordUserFact(
fact = "Prefers functional programming",
userId = Some("user-123"),
importance = Some(0.9)
)
```
### Entity Knowledge
(not compile-checked)
```scala
// Record knowledge about an entity
val result = manager.recordEntityFact(
entityId = EntityId("scala-lang"),
entityName = "Scala",
fact = "Scala is a JVM language combining OOP and FP",
entityType = "language",
importance = Some(0.8)
)
```
### External Knowledge
(not compile-checked)
```scala
// Record external knowledge: `source` says where it came from
val result = manager.recordKnowledge(
content = "The latest LLM4S version is 0.5.0",
source = "release-notes",
metadata = Map("channel" -> "stable")
)
```
`recordKnowledge` takes no `importance`, and the memory it stores has none (the metadata you pass
is kept on it). Use `recordEntityFact` or `recordUserFact` when importance matters.
### Task Outcomes
(not compile-checked)
```scala
// Record a task result
val result = manager.recordTask(
description = "Generate the quarterly report",
outcome = "Report generated",
success = true,
importance = Some(0.6)
)
```
### Conversation Messages
(not compile-checked)
```scala
import org.llm4s.llmconnect.model._
// Record a conversation turn
val result = manager.recordMessage(
message = UserMessage("What's the weather in Paris?"),
conversationId = "conv-789"
)
// Record several messages at once
val result = manager.recordConversation(
messages = Seq(UserMessage("And in Rome?"), AssistantMessage("Sunny, 24 degrees.")),
conversationId = "conv-789"
)
```
The conversation id is a plain `String`. Each message is stored with its role (`user`,
`assistant`, `system` or `tool`).
---
## Retrieving Memory
Each retrieval method returns a `Result[String]`, ready to put in a prompt. When there is nothing
to report the string is empty.
### Relevant Context
Get the memories relevant to a query. With the in-memory store this is keyword matching; with a
vector store it is similarity of meaning:
(not compile-checked)
```scala
val context = manager.getRelevantContext(
query = "Tell me about Scala programming",
maxTokens = 1000,
filter = MemoryFilter.ByTypes(Set(MemoryType.UserFact, MemoryType.Knowledge))
)
```
The context starts with `# Retrieved Context` and groups the memories under `## Relevant
Knowledge`, `## Entity Information`, `## User Preferences`, `## Previous Context` and `## Past
Tasks`. `maxTokens` limits the length at about four characters per token (see
[Manage Context Token Budget](#3-manage-context-token-budget)).
### Conversation History
(not compile-checked)
```scala
val history = manager.getConversationContext(
conversationId = "conv-789",
maxMessages = 10
)
```
The result starts with `Previous conversation:` and has one `[role]: text` line per message.
### Entity Context
(not compile-checked)
```scala
val entityInfo = manager.getEntityContext(EntityId("anthropic"))
```
The result is `Known facts about :` followed by one `- fact` line per memory.
### User Context
(not compile-checked)
```scala
val userInfo = manager.getUserContext(Some("user-123"))
```
The result is `Known facts about the user:` followed by one `- fact` line per fact recorded for
that user. Pass `None` to include the facts of every user.
---
## Memory Stores
### In-Memory Store
Fast but volatile - loses data on restart. Each store is an immutable value: a write returns a
new store, and `SimpleMemoryManager` keeps track of it for you.
(not compile-checked)
```scala
import org.llm4s.agent.memory.InMemoryStore
val store = InMemoryStore.empty
val manager = SimpleMemoryManager.withStore(store)
```
### SQLite Store
Persistent local storage. Opening a store returns a `Result`, and a SQLite store holds one JDBC
connection, so close it when you are done:
(not compile-checked)
```scala
import org.llm4s.agent.memory.SQLiteMemoryStore
for {
// File-based (persistent)
file <- SQLiteMemoryStore("memory.db")
// In-memory SQLite (fast, volatile)
volatile <- SQLiteMemoryStore.inMemory()
} yield (file, volatile)
```
A SQLite store is not safe for concurrent use: give each thread its own instance, or synchronise
access yourself.
### Vector Store
Semantic search with embeddings:
(not compile-checked)
```scala
import org.llm4s.agent.memory._
import org.llm4s.llmconnect.EmbeddingClient
import org.llm4s.llmconnect.config.EmbeddingModelConfig
import org.llm4s.types.Result
// Wrap an embedding client
def embeddingServiceFor(embeddingClient: EmbeddingClient, modelConfig: EmbeddingModelConfig): EmbeddingService =
LLMEmbeddingService(embeddingClient, modelConfig)
// A vector store in a SQLite file (VectorMemoryStore.inMemory(embeddingService) keeps it in memory)
def vectorManager(embeddingService: EmbeddingService, path: String): Result[SimpleMemoryManager] =
VectorMemoryStore(path, embeddingService).map(store => SimpleMemoryManager.withStore(store))
```
For tests, `MockEmbeddingService(dimensions = 8)` produces deterministic vectors from a hash of
the text; its scores say nothing about meaning.
The store embeds what it keeps as a *document* and the text you search with as a *query*
(`EmbeddingService.embedQuery`). Models that embed the two differently (Voyage, Cohere, Jina) are
then searched correctly. `embedQuery` delegates to `embed` unless a service overrides it, so a
custom `EmbeddingService` written before it existed keeps working unchanged.
---
## Memory with Agents
### Semantic Memory Middleware
(not compile-checked)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-agent-memory" % llm4sVersion // same version as llm4s-agent
```
`SemanticMemoryMiddleware` (`org.llm4s.agent.harness.memory`) does the recall and the recording for
you, for every turn of an agent:
(not compile-checked)
```scala
import org.llm4s.agent.Agent
import org.llm4s.agent.graph.{ RunConfig, TenantId, ThreadId }
import org.llm4s.agent.harness.memory.{ MemoryRecall, SemanticMemoryMiddleware }
import org.llm4s.agent.memory.{ MemoryBounds, SQLiteMemoryStore, SimpleMemoryManager }
val memory = new SemanticMemoryMiddleware(
stores = tenant => SQLiteMemoryStore(s"memory-${tenant.fold("default")(_.value)}.db"),
manager = (store, _) => SimpleMemoryManager.withStore(store),
client = client, // the client an LLMMemoryManager would call
recall = Some(MemoryRecall(maxRecallTokens = 512)),
bounds = MemoryBounds(maxMemories = 500)
)
for {
agent <- Agent.builder("assistant", client).withMiddleware(memory).build()
result <- agent.run(ThreadId("t-1"), "What did I say my favourite colour was?",
RunConfig().withTenantId(TenantId("alice")))
} yield result.answer
```
- **Per tenant.** `stores` is asked once for each tenant (`RunConfig.tenantId`; `None` for runs
without one) and must return a store of that tenant's alone: every thread of a tenant recalls what
the tenant's earlier threads recorded, and no tenant reads another's. A resolver that returns one
store instance for two tenants is refused.
- **Recall** runs `getRelevantContext` on each turn's input and shows the result to every model
call of the turn as a system-prompt section, cut to `maxRecallTokens` (four characters a token).
- **Recording** stores each turn's question and answer through the manager, each memory cut to
`MemoryRecording.maxRecordChars` (default 2000). With an `LLMMemoryManager`, its own model calls
(entity extraction, summaries) go through the run's accounted client, so they count in
`result.usage`.
- **Consolidation** (`consolidate = Some(MemoryConsolidation(olderThan = 1.day, minCount = 10))`)
replaces groups of old memories with a summary after each turn. It needs a
`TransactionalMemoryStore`: `InMemoryStore`, `SQLiteMemoryStore` or `PostgresMemoryStore`.
- **Bounded.** Each write evicts the memories of lowest importance, oldest first, beyond
`bounds` (default 1000 memories, 4 MiB), whether or not consolidation is on.
- **Safe across a crash.** Each recording and each consolidated group is one replayable effect of
the agent runtime: after a crash, `agent.recover` neither records a turn twice nor brings back a
memory a later consolidation removed. A store that is not transactional records too, but a
recording interrupted mid-write is skipped on recovery rather than repeated.
- **Failures never cost the answer.** A store that cannot be searched or written is skipped, and
announced with a durable `MemoryEvents.Skipped` (the operation and the error's type, no content);
`MemoryEvents.Recorded` and `MemoryEvents.Consolidated` carry counts only.
- **Register it before your guardrails** (or pass `runsBefore = Set(MiddlewareId("guardrails"))`),
so an answer a guardrail blocks is never recorded. A recording that already applied is not undone
by a later Block.
- **Adding it to an existing agent** registers the `semantic-memory-recalled` key, which starts
empty: a thread at rest continues under the agent that added it, and, since the recall is
cleared at the end of each turn, under an agent that removed it again. A thread in flight
(suspended, or after a crash) is refused with `RestoreRejected` by the changed agent: resume or
recover it first.
### Injecting Context
To do it by hand instead, without the middleware:
Put the relevant memory in the system prompt, run the agent, then record the turn. The memory half
of this is runnable as is; `runAgent` stands for however you build and call your agent (see the
[Agents guide](index)), taking the system prompt and returning the messages of the turn:
(not compile-checked)
```scala
import org.llm4s.agent.memory._
import org.llm4s.llmconnect.model.Message
import org.llm4s.types.Result
def answerWithMemory(
manager: MemoryManager,
userQuery: String,
conversationId: String
)(runAgent: String => Result[Seq[Message]]): Result[MemoryManager] =
for {
// Get relevant context for the query
context <- manager.getRelevantContext(userQuery)
// Build a system prompt that carries the context
systemPrompt = s"""You are a helpful assistant.
|
|Relevant context from memory:
|$context""".stripMargin
// Run your agent with that prompt and keep what was said
messages <- runAgent(systemPrompt)
updated <- manager.recordConversation(messages, conversationId)
} yield updated
```
### Recording the Conversation
Managers record only what you pass to the `record*` methods, so record each turn yourself, as
above, with `recordMessage` or `recordConversation`.
---
## Semantic Search
### With Embeddings
(not compile-checked)
```scala
import org.llm4s.agent.memory._
import scala.util.Using
def semanticSearch(embeddingService: EmbeddingService): Result[Seq[ScoredMemory]] =
VectorMemoryStore.inMemory(embeddingService).flatMap { store =>
Using.resource(new AutoCloseable { override def close(): Unit = store.close() }) { _ =>
for {
m1 <- SimpleMemoryManager.withStore(store).recordKnowledge("Paris is the capital of France", "geography")
m2 <- m1.recordKnowledge("Berlin is the capital of Germany", "geography")
m3 <- m2.recordKnowledge("Rome is the capital of Italy", "geography")
results <- m3.store.search("European capitals", topK = 2)
} yield results
}
}
```
The results are `ScoredMemory` values, best first. With a real embedding model they are the
memories closest in meaning to the query.
### Search with Filters
Filters are built from the `MemoryFilter` cases and combined with `&&`, `||` and `!`:
(not compile-checked)
```scala
val filter = MemoryFilter.ByType(MemoryType.Knowledge) &&
MemoryFilter.MinImportance(0.7) &&
MemoryFilter.ByMetadata("user_id", "user-123")
val results = store.search(
query = "programming languages",
topK = 5,
filter = filter
)
```
The other cases are `ByTypes`, `HasMetadata`, `MetadataContains`, `ByEntity`, `ByConversation`,
`ByTimeRange`, `ContentContains`, `Custom` (any `Memory => Boolean`), `All` and `None`.
---
## Memory Consolidation
Consolidation summarises old memories into fewer, shorter ones. Only `LLMMemoryManager` does it:
`SimpleMemoryManager.consolidateMemories` returns the manager unchanged.
(not compile-checked)
```scala
import org.llm4s.agent.memory._
import org.llm4s.llmconnect.LLMClient
import org.llm4s.types.Result
import java.time.Instant
import java.time.temporal.ChronoUnit
// An LLM client is needed to write the summaries
def llmManager(client: LLMClient): LLMMemoryManager =
LLMMemoryManager(
config = MemoryManagerConfig.default,
store = InMemoryStore.empty,
client = client
)
// Consolidate memories older than a week, in groups of at least ten
def consolidate(manager: MemoryManager): Result[MemoryManager] =
manager.consolidateMemories(
olderThan = Instant.now().minus(7, ChronoUnit.DAYS),
minCount = 10
)
```
`minCount` applies to each group of old memories (same type and context) separately, not to their
total. Nothing consolidates on its own: call `consolidateMemories` when you want it (for example
from a scheduled job).
---
## Entity Extraction
Extract entities from text using an LLM. Only `LLMMemoryManager` does it: on
`SimpleMemoryManager`, `extractEntities` returns the manager unchanged.
(not compile-checked)
```scala
import org.llm4s.agent.memory._
import org.llm4s.types.Result
def extractEntities(manager: MemoryManager): Result[String] =
for {
m1 <- manager.extractEntities(
text = "Anthropic is an AI company that created Claude",
conversationId = Some("conv-789")
)
// The extracted entities are ordinary entity memories
info <- m1.getEntityContext(EntityId.fromName("Anthropic"))
} yield info
```
`extractEntities` returns the manager holding the new entity memories, not a list of ids. Read
them back with `getEntityContext`; `EntityId.fromName` turns a name into the id used for it
(lower case, spaces replaced by `_`). To mine every recorded user and assistant message instead,
set `MemoryManagerConfig(autoExtractEntities = true)`: `recordMessage` then calls `extractEntities`
for you, with the message's conversation id.
---
## Memory Statistics
(not compile-checked)
```scala
val result = manager.stats.map { stats =>
s"""Total memories: ${stats.totalMemories}
|By type: ${stats.byType}
|Oldest: ${stats.oldestMemory}
|Newest: ${stats.newestMemory}""".stripMargin
}
```
`stats` is a `Result[MemoryStats]`. It also has `entityCount`, `conversationCount` (the number of
distinct conversations) and `embeddedCount`. `byType` leaves out types with no memories, and
`oldestMemory` and `newestMemory` are `None` when there are none.
---
## Persistence Patterns
### Save and Load
(not compile-checked)
```scala
import scala.util.Using
// Both opened stores close even when a Result is Left.
val result = SQLiteMemoryStore("memory.db").flatMap { store =>
Using.resource(new AutoCloseable { override def close(): Unit = store.close() }) { _ =>
SimpleMemoryManager.withStore(store).recordUserFact("Likes Scala", Some("user-1")).map(_ => ())
}
}.flatMap { _ =>
SQLiteMemoryStore("memory.db").flatMap { reopened =>
Using.resource(new AutoCloseable { override def close(): Unit = reopened.close() }) { _ =>
SimpleMemoryManager.withStore(reopened).getUserContext(Some("user-1"))
}
}
}
// Right("Known facts about the user:\n- Likes Scala")
```
### Cross-Session Memory
(not compile-checked)
```scala
import scala.util.Using
class PersistentMemory(dbPath: String) {
// Open the store, run something against a manager on it, and close the store again
def withManager[A](use: MemoryManager => Result[A]): Result[A] =
SQLiteMemoryStore(dbPath).flatMap { store =>
Using.resource(new AutoCloseable { override def close(): Unit = store.close() }) { _ =>
use(SimpleMemoryManager.withStore(store))
}
}
}
val memory = new PersistentMemory("memory.db")
// Session 1
val result1 = memory.withManager(_.recordUserFact("Likes Scala", Some("user-1")))
// Session 2 (later)
val result2 = memory.withManager(_.getUserContext(Some("user-1")))
// context includes "Likes Scala" from session 1
```
---
## Best Practices
### 1. Set Appropriate Importance
(not compile-checked)
```scala
for {
// High importance - core user preferences
m1 <- manager.recordUserFact("Primary programming language is Scala", importance = Some(0.9))
// Medium importance - useful but not critical
m2 <- m1.recordEntityFact(EntityId("scaladays"), "ScalaDays", "Attended in 2024", importance = Some(0.6))
// Low importance - ephemeral information
m3 <- m2.recordTask("Run the tests", "Passed", success = true, importance = Some(0.3))
} yield m3
```
### 2. Use Specific Memory Types
Don't use generic `Knowledge` for everything: specific types are grouped under their own
heading in the retrieved context and can be filtered with `MemoryFilter.ByType`.
- `recordUserFact(...)` for user preferences
- `recordEntityFact(...)` for entity-specific information
- `recordKnowledge(...)` for external knowledge
- `recordTask(...)` for task outcomes
### 3. Manage Context Token Budget
(not compile-checked)
```scala
// About four characters per token: the context is cut to fit
val context = manager.getRelevantContext(query, maxTokens = 500)
```
`getRelevantContext` stops adding memories once the next one would take the text past
`maxTokens * 4` characters; the section headings and the `# Retrieved Context` line count towards
that. A heading is written only with at least one memory under it, so a `maxTokens` too small for
any memory gives an empty string. Without `maxTokens` the budget is the manager's
`MemoryManagerConfig.contextTokenBudget`, 2000 tokens unless you set it.
### 4. Clean Up Old Memories
(not compile-checked)
```scala
// Consolidate old memories periodically (LLMMemoryManager only)
consolidate(manager)
// Delete old, low-importance memories (there is no "maximum importance" filter: use `Custom`)
store.deleteMatching(
MemoryFilter.Custom(_.importance.exists(_ <= 0.3)) &&
MemoryFilter.ByTimeRange(before = Some(thirtyDaysAgo))
)
```
`deleteMatching` returns the store without the matching memories.
---
## Examples
| Example | Description |
|---------|-------------|
| [BasicMemoryExample](/examples/#memory-examples) | Getting started with memory |
| [ConversationMemoryExample](/examples/#memory-examples) | Conversation history management |
| [MemoryWithAgentExample](/examples/#memory-examples) | Integrating memory with agents |
| [SQLiteMemoryExample](/examples/#memory-examples) | Persistent SQLite storage |
| [VectorMemoryExample](/examples/#memory-examples) | Semantic search with embeddings |
[Browse all examples →](/examples/)
---
## Next Steps
- [Handoffs Guide](handoffs) - Agent-to-agent delegation
- [Streaming Guide](streaming) - Real-time execution events
- [Guardrails Guide](guardrails) - Input/output validation
---
Source: https://llm4s.org/guide/agents/handoffs
# Agent Handoffs
Delegate queries to specialist agents for domain expertise.
---
## Overview
Handoffs enable LLM-driven agent-to-agent delegation. When a primary agent determines that a query requires specialist expertise, it can hand off the conversation to another agent.
A handoff is a route inside one graph. The root agent and every agent reachable through its handoffs are compiled together, by agent id, and one thread runs across them: the thread records which agent is active, and the next turn starts with that agent.
**Key Benefits:**
- **Specialization** - Route queries to domain experts
- **Modularity** - Build focused, maintainable agents
- **Scalability** - Add specialists without modifying the main agent
- **Context Control** - Choose what context to preserve
---
## Quick Start
```scala
import org.llm4s.agent.{Agent, Handoff}
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.model.ModelRegistryService
val result = for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(providerConfig)
// The specialist is a builder, not yet a built agent
mathAgent = Agent.builder("math", client)
.withSystemPrompt("You are a math expert. Solve problems step-by-step.")
// The main agent lists it as a handoff target
mainAgent <- Agent.builder("main", client)
.withHandoffs(Handoff.to("math", mathAgent, "Math expertise required"))
.build()
result <- mainAgent.run("What is the integral of x^2?")
} yield result
```
The handoff id must equal the target builder's agent id (`"math"` above): the graph is keyed by agent id.
---
## Handoff Configuration
### Basic Handoff
```scala
import org.llm4s.agent.Handoff
val handoff = Handoff.to(
"specialist", // equals the target builder's id
specialistBuilder,
"Domain expertise required"
)
```
`Handoff.of(id, builder, reason)` returns a `Result[Handoff]` instead of throwing for an invalid id
(`[a-zA-Z0-9_-]{1,52}`).
### Handing Back (Cycles)
A target that must hand back to an agent that is already in the graph names it by id, with
`Handoff.toId`, because two builders cannot each hold the other:
```scala
val specialist = Agent.builder("specialist", client)
.withHandoffs(Handoff.toId("triage", "Not a specialist question"))
val triage = Agent.builder("triage", client)
.withHandoffs(Handoff.to("specialist", specialist, "Needs the specialist"))
.build()
```
`build()` refuses an id that no builder in the family defines, and one id reached with two
different builders. A self-handoff is refused too.
### Without Context
```scala
// The specialist is sent the user's last question, then the transfer onwards
val handoff = Handoff.to("specialist", specialistBuilder, "Fresh analysis needed", preserveContext = false)
```
`Handoff.of(id, builder, reason, preserveContext = false)` takes it too, and, for an agent defined
elsewhere in the family, `Handoff.toId("specialist", Some("Fresh analysis needed"), preserveContext = false)`.
---
## Handoff Options
| Option | Default | Description |
|--------|---------|-------------|
| `id` | Required | The target agent's id; the tool is `handoff_to_` |
| `target` | by `to`/`of`: the builder | The target's builder; `None` (`toId`) names an agent defined elsewhere in the family |
| `transferReason` | `None` | Description shown to the LLM for routing |
| `preserveContext` | `true` | `false` sends the target only the last user message before the transfer, then the transfer onwards |
`transferSystemMessage` no longer exists: each agent sends its own system prompt, and a prompt is
never stored in the thread.
---
## Multiple Handoffs
Provide multiple specialists for intelligent routing:
```scala
val mainAgent = Agent.builder("main", client)
.withHandoffs(
Handoff.to("math", mathAgent, "Mathematical calculations and proofs"),
Handoff.to("code", codeAgent, "Programming and code review"),
Handoff.to("legal", legalAgent, "Legal questions and contracts"),
Handoff.to("medical", medicalAgent, "Health and medical information")
)
.build()
```
The LLM sees descriptions and chooses the appropriate specialist.
---
## How Handoffs Work
### 1. Tools are Generated
Each handoff target is offered to the model as a tool named `handoff_to_`, described by the transfer reason.
### 2. LLM Decides
The LLM can choose to:
- Answer directly (no handoff)
- Call a regular tool
- Hand off by calling a handoff tool
A handoff must be the **only** tool call in its message. A handoff mixed with other tool calls, or two handoffs in one message, is refused: every call in the message gets an error result explaining the rule, and the same agent's model is asked again.
### 3. The Transfer
The loop stores the assistant message, appends a tool result `Transferred to `, makes the target the active agent and continues in the target's model node. The target sends its own system prompt and offers its own tools and handoffs. The handoff counts as a step, and the step limit is shared by the whole turn: it is the current agent's `maxSteps` against the turn's total step count.
### 4. Response Returns
The target's answer is the run's result. `AgentResult.activeAgent` names the agent that answered, and the next turn starts with it, so a conversation that moved to a specialist stays there until the specialist hands it back.
---
## Specialist Agent Patterns
### Domain Expert
```scala
val physicsAgent = Agent.builder("physics", client)
.withSystemPrompt(
"""You are a physics expert with deep knowledge of:
|- Classical mechanics
|- Quantum mechanics
|- Thermodynamics
|- Electromagnetism
|
|Provide detailed explanations with equations when helpful.""".stripMargin
)
val mainAgent = Agent.builder("main", client)
.withHandoffs(Handoff.to("physics", physicsAgent, "Physics questions requiring expert explanation"))
.build()
mainAgent.flatMap(_.run("Explain quantum entanglement"))
```
### Tool Specialist
Tools belong to the agent that uses them:
```scala
val dataAgent = Agent.builder("data", client)
.withSystemPrompt("You are a data analysis specialist.")
.withTools(new ToolRegistry(Seq(queryDatabaseTool, generateChartTool, exportCSVTool)))
val mainAgent = Agent.builder("main", client)
.withTools(new ToolRegistry(basicTools))
.withHandoffs(Handoff.to("data", dataAgent, "Data analysis with database access"))
.build()
mainAgent.flatMap(_.run("Analyze our Q4 sales data"))
```
### Customer Support Triage
```scala
val billingAgent = Agent.builder("billing", client)
.withSystemPrompt("You handle billing and payment issues.")
val technicalAgent = Agent.builder("technical", client)
.withSystemPrompt("You solve technical problems.")
val salesAgent = Agent.builder("sales", client)
.withSystemPrompt("You help with purchases and upgrades.")
val triageAgent = Agent.builder("triage", client)
.withSystemPrompt("You are a customer support triage agent. Route queries to the appropriate specialist.")
.withHandoffs(
Handoff.to("billing", billingAgent, "Billing, payments, and subscription issues"),
Handoff.to("technical", technicalAgent, "Technical problems and account access"),
Handoff.to("sales", salesAgent, "Purchases, upgrades, and pricing questions")
)
.build()
triageAgent.flatMap(_.run("I can't log into my account and my payment failed"))
```
---
## Guardrails and Middleware Across Handoffs
The root agent's run-boundary middleware guards the whole family. Its `beforeAgent` hooks - input
guardrails among them - run on every turn's query, even when a specialist is active, before the
specialist's own; its `afterAgent` hooks - output guardrails - run on every final answer, after the
answering specialist's own. So guardrails given to the root keep applying after a handoff:
```scala
import org.llm4s.agent.graph.middleware.GuardrailMiddleware
import org.llm4s.agent.guardrails.builtin.{ PIIMasker, ProfanityFilter }
val triage = Agent.builder("triage", client)
.withMiddleware(GuardrailMiddleware(input = Seq(new ProfanityFilter()), output = Seq(new PIIMasker())))
.withHandoffs(Handoff.to("billing", billingAgent, "Billing questions"))
.build()
// a query to billing two turns later still passes ProfanityFilter; billing's answers still pass PIIMasker
```
A specialist's own boundary middleware applies only while it is active, inside the root's. A block
from either ends the turn `Blocked`; an output block removes the whole turn, a handoff made in it
included, so the next turn starts with the agent that was active before it. Model and tool wrappers (`wrapModelCall`, `wrapToolCall`, such as
`ContextWindowMiddleware` and `ApprovalMiddleware`) are per agent: give them to each agent that
needs them. Do not give the root's guardrails to its specialists too: they would run twice.
---
## Handling Handoff Results
```scala
mainAgent.run(query) match {
case Right(result) =>
result.status match {
case AgentStatus.Completed(answer) =>
println(s"${result.activeAgent} answered: $answer")
case AgentStatus.StepLimitReached =>
println("Hit the step limit")
case other =>
println(s"Run ended: $other")
}
case Left(error) =>
println(s"Error: $error")
}
```
A handoff is not a status: the run continues in the target and ends as any run does.
A handoff is reported by the durable `AgentEvents.HandedOff(from, to)` event on the agent's event
stream (the old `HandoffStarted` / `HandoffCompleted`); see [Streaming Events](streaming).
---
## Context Preservation Examples
### Full Context (Default)
```scala
// Specialist sees the entire conversation
Handoff.to("specialist", specialist, "Needs the specialist")
// User: "I'm building a Scala app"
// Assistant: "Great! What kind of app?"
// User: "A REST API with database access"
//
// Specialist sees all messages above, then its own reply
```
### Fresh Context
```scala
// Specialist starts from the last question
Handoff.toId("specialist", Some("Independent review"), preserveContext = false)
// User: "I'm building a Scala app"
// Assistant: "Great! What kind of app?"
// User: "A REST API with database access"
//
// Specialist is sent: "A REST API with database access", then the transfer
```
The stored history is the same either way: `preserveContext = false` changes only what the target is sent, and the transfers are recorded by the loop, so a non-root agent is never sent the full history by mistake.
---
## Handoffs vs Graphs
| Use Case | Approach |
|----------|----------|
| 2-3 specialists, LLM-driven routing | **Handoffs** |
| Complex multi-agent workflows | **Graph (`GraphBuilder`)** |
| Dynamic specialist selection | **Handoffs** |
| Parallel agent execution | **Graph** |
| Simple delegation | **Handoffs** |
| Type-safe data flow | **Graph** |
To hand a bounded task to another agent on a thread of its own, keeping control and getting back only its typed
report, use [delegation](delegation) instead.
For complex workflows, see the [multi-agent graph recipe](../../examples/cookbook/multi-agent-graph.html): agents as nodes of a typed graph, run in parallel and joined.
---
## Best Practices
### 1. Clear Transfer Reasons
```scala
// Good - specific and actionable
Handoff.to("db", agent, "Database queries and SQL optimization")
// Bad - vague
Handoff.to("db", agent, "Technical stuff")
```
### 2. Focused Specialists
```scala
// Good - focused specialist
val sqlAgent = Agent.builder("sql", client)
.withSystemPrompt("You are a SQL expert. Optimize queries and explain execution plans.")
// Bad - too broad
val everythingAgent = Agent.builder("everything", client)
.withSystemPrompt("You know everything about databases, APIs, UI, and infrastructure.")
```
### 3. Changing a Family Changes Its Threads
The graph's version covers each agent's id, tools, middleware, handoffs and system prompt. If you
rebuild the root without a handoff that a stored thread's active agent was, `run` refuses that
thread rather than routing to a missing node.
### 4. Don't Overuse Handoffs
```scala
// Good - meaningful specialization
.withHandoffs(
Handoff.to("math", mathAgent, "Complex mathematical calculations"),
Handoff.to("legal", legalAgent, "Legal analysis and compliance")
)
// Bad - too granular
.withHandoffs(
Handoff.to("addition", additionAgent, "Adding numbers"),
Handoff.to("subtraction", subtractionAgent, "Subtracting numbers")
)
```
---
## Examples
| Example | Description |
|---------|-------------|
| [SimpleTriageHandoffExample](/examples/#handoff-examples) | Basic query routing |
| [MathSpecialistHandoffExample](/examples/#handoff-examples) | Math specialist delegation |
| [ContextPreservationExample](/examples/#handoff-examples) | Context preservation patterns |
[Browse all examples →](/examples/)
---
## Next Steps
- [Streaming Guide](streaming) - Real-time execution events
- [Memory Guide](memory) - Persistent context
- [Guardrails Guide](guardrails) - Input/output validation
---
Source: https://llm4s.org/guide/agents/streaming
# Streaming Events
Watch an agent's turn as it happens: the answer token by token, each tool call, each model call.
---
## Streaming a turn
`AgentBuilder.withStreaming()` makes every model call of the agent stream its answer (the default
is off: the agent calls `complete`). `agent.stream(threadId, query)(listener)` runs a turn and hands
every event of it to `listener`, which receives the runtime's `StreamEvent`. Match the agent's events
with the extractors in `AgentEvents`; they need no `ujson`:
(not compile-checked)
```scala
import org.llm4s.agent.Agent
import org.llm4s.agent.events.AgentEvents
import org.llm4s.agent.graph.ThreadId
for
agent <- Agent.builder("assistant", client).withSystemPrompt("You are concise.").withStreaming().build()
run <- agent.stream(ThreadId("t1"), "Explain monads in three sentences.") {
case AgentEvents.TextDelta(d) => print(d.text)
case AgentEvents.ToolCallStarted(c) => println(s"\n[${c.tool}]")
case AgentEvents.ModelCallCompleted(m) => println(s"\n(${m.usage})")
case _ => ()
}
result <- run.await()
yield result
```
`stream` returns the `AgentRun` at once; `await()` gives the `AgentResult` as for `run`, and returns
only once the listener has returned from the run's last event - so everything the listener collected
is there when `await` returns. It waits at most 5 seconds for a listener that is
still busy, then logs a WARN and returns.
`streamResume(threadId, answers)` and `streamRecover(threadId)` are the streaming siblings of
`startResume` and `startRecover`. A listener runs on its own dispatcher thread, so a slow listener
never slows the run. Runnable version: `StreamingAgentExample` in `modules/samples`.
## The events
Every event is an `EventType` in `org.llm4s.agent.events.AgentEvents`, named `agent.`,
version 1.
**Durable** events are committed with the task that emits them, so they are stored, replayed, and
never duplicated by a retry or a `recover`. They carry no message content.
| Event | Fields | Sent |
|---|---|---|
| `ModelCallCompleted` | `agent`, `model`, `attempts`, `toolCalls`, `usage`, `estimatedCost` | after each provider call returns a completion - every attempt a retrying wrapper makes, numbered in `attempts`, and (`attempts` 0) each call a hook makes through `ModelContext.accounted` |
| `ToolExecuted` | `agent`, `toolCallId`, `tool`, `duration`, `outcome` (`Succeeded`, `Errored`, `Denied`, `Rejected`, `NeedsApproval`, `Asked`) | when a tool call's outcome is recorded |
| `HandedOff` | `from`, `to` | when the model routes a handoff |
| `GuardrailBlocked` | `guardrail`, `phase` (`Input`, `Output`) | when a guardrail blocks a turn |
**Live** events are for watching. They carry content, have no `seq`, and are never stored or replayed.
| Event | Fields | Sent |
|---|---|---|
| `ModelCallStarted` | `agent`, `attempt` | before each attempt of a model call |
| `TextDelta` | `attempt`, `text` | per chunk of the answer (needs `withStreaming()`) |
| `ThinkingDelta` | `attempt`, `text` | per chunk of the model's reasoning (needs `withStreaming()`) |
| `ToolCallStarted` | `toolCallId`, `tool`, `arguments` | after the arguments validate, before middleware runs |
| `ToolCallResult` | `toolCallId`, `content`, `isError` | when the call's result is recorded |
Notes on the durable events:
- `attempts` is 0 when a model middleware answered without calling the model.
- `ToolExecuted.tool` names the tool only when the agent has it; a call to a name the model invented
is recorded as `""` (the live `ToolCallResult` still carries the call). Each non-handoff
call of a batch that mixes a handoff with other calls runs nothing and is reported as `Errored`.
- A tool call that is approved or edited produces two `agent.tool_executed` events for one call id,
across two runs: `NeedsApproval` when it suspends, then the final outcome after `resume`.
- `agent.guardrail_blocked` fires only for a guardrail's block. Other middleware refusals (a blank
query or answer, a custom `Left`) emit none.
- The kernel's own events (`RunStarted`, `RunCompleted`, `RunFailed`, ...) arrive in the same
stream, as `StreamEvent.Durable`.
## Durable vs live, and the content rule
Durable events hold identifiers, names, usage, durations and outcomes, never message content:
no assistant text, thinking, tool arguments, tool results or guardrail reasons. Content travels
only in live events, and in `AgentRunEnded.messages` at the end of a traced run.
The reason is the guardrail Block guarantee. When an output guardrail blocks a turn, the runtime
removes the blocked turn from the thread. If the log held the content, a Block would also have to
redact the log. Replaying a run therefore shows its structure (which model calls, which tools, how
long, with what usage), and a blocked answer is in no stored event.
## Attempts
`attempt` counts the calls of the innermost model function within one task, from 1. A model
wrapper that retries or falls back (`wrapModelCall`) calls it again, so a higher `attempt` on the
same task means the text streamed so far for that task is discarded: clear what you have shown and
start again. A task that `recover` runs again starts at attempt 1 under a new task id.
## Subagents' events
An agent with subagents sends `SubagentEvents.TaskStarted` (live) as each child starts and `SubagentEvents.TaskEnded`
(durable, no content) when its task ends. With `nested = true` on `stream*` or `AgentRun.subscribe`, the listener also
receives each child's events as `StreamEvent.Nested(child, event)`, matched with `EventHandlers.onChild`; see
[Delegation: following children](delegation#following-children).
## `stream*` vs `subscribe`
- `Agent.stream`, `streamResume` and `streamRecover` subscribe your listener during admission,
before the run thread starts, so it sees every durable and every live event of the run, from the
claim on.
- `AgentRun.subscribe(capacity = 1024)(listener)` attaches later, to a run you started with
`start`. It replays the run's durable events from its start, then delivers new ones, but it
misses live events sent before it attached. Live events sent after it returns are not lost
while it replays: they are held and delivered in their place among the durable events. Up to
`capacity` are held; beyond that they are dropped and counted, as for a full queue, and the
listener gets a `LiveGap(dropped)` where they would have been.
Both are run-scoped: a listener sees only this run's events, even when other runs share the thread,
and ends after the run's terminal event (`RunCompleted`, `RunSuspended`, `RunFailed`,
`RunCancelled`, `RunTimedOut`), or after a `Disconnected` (`Lagging`, `ListenerFailed`, or - for
`subscribe`, which replays - `ReplayFailed`). A run that crashes without a terminal event (or whose
terminal commit fails) ends its stream as soon as the listener has returned from the run's last
event: as the run ends, a marker is queued behind its last event, and reaching it ends
the subscription - also for a `subscribe` made after the run ended, once its replay is done. There
is no fixed delay. Either way, `await` waits for the listener as described above.
## Falling behind
Each listener has a bounded buffer (1024 events for `stream*`) for durable events and another of the
same size for live ones, so a burst of live events never costs it a durable one:
- If the live buffer is full, live events are dropped, and the listener gets
`StreamEvent.LiveGap(n)` with the number lost, so a UI can note that text is missing. The final
answer is still in the `AgentResult`.
- If the durable buffer is full - the listener is that many durable events behind - it is
disconnected with
`StreamEvent.Disconnected(lastSeq, DisconnectReason.Lagging)`. Durable events are in the log, so
you can resubscribe from where you were with `GraphRuntime.subscribe(threadId, afterSeq = lastSeq)`.
That needs the agent's runtime: build the agent `withRuntime(runtime)` and subscribe on that
`runtime`. The subscription is thread-scoped, not run-scoped - it delivers every later run on the
thread too - and does not end itself: cancel it when you are done.
A listener that throws is disconnected with `ListenerFailed`; the run is unaffected.
## Replay
Durable events outlive the run. `GraphRuntime.subscribe(threadId, afterSeq = 0)` replays a thread's
events from the start, which gives the structure of every run without any content. See
`EventCollectionExample`.
### Tenants
A subscription reads the thread, so it is checked like `history`: pass the caller's `RunConfig`, and a
thread of another tenant is refused with `GraphError.TenantMismatch` before anything is delivered.
`TracingSubscriber.attach` takes the same `config`. The default, `RunConfig()`, is the caller with no
tenant, which is right only for threads whose runs set none.
(not compile-checked)
```scala
runtime.subscribe(threadId, config = RunConfig().withTenantId(TenantId("acme")))(listener)
```
A thread can change hands under its id after you subscribe: one that does not exist yet is created by
whichever tenant runs on it first, and one deleted through another runtime can be created again before
your subscription learns of the delete. So a subscription screens each event against your tenant, and
ends with `Disconnected(lastSeq, TenantMismatch)` - before the first event of another tenant's thread,
delivering none of it. `RunHandle.subscribe` and `AgentRun.subscribe` belong to a run, and use its
tenant.
## Subscriptions across runtimes
Several `GraphRuntime`s, in one process or several, can share one checkpoint store
([Durable Checkpointers](durable-checkpointers)). A `GraphRuntime.subscribe` on any of them delivers
the durable events of every commit on the thread, whichever runtime made it, with the same
durable guarantees: in `seq` order, without gap or duplicate, and only after the commit. How those
events interleave with live ones is the exception: see [Ordering across runtimes](#ordering-across-runtimes).
- The subscription's own runtime hands it that runtime's commits as they land. Other runtimes' commits
it reads from the store, on a second virtual thread (`llm4s-watch-`) that runs while the
subscription is live. A store that is told of commits delivers them at once: `InMemoryCheckpointer`,
shared between runtimes in one process. A store that is not is read every `pollInterval`, 250 ms by
default; the SQLite store is read this way.
- Only durable events cross runtimes. Live progress is never stored, so another runtime's
`TextDelta`s and other live events never arrive.
- Another runtime's events wait in the store until the listener has room for them, so they never
make a subscriber lag. The subscription's own runtime's commits are still delivered in their place.
A commit that finds other runtimes' events not yet read reads them first. If they do not fit, the
subscriber lags as usual, and you resubscribe from `lastSeq`.
- If a read of the store fails, the subscription delivers what it has queued and then ends with
`Disconnected(lastSeq, ReplayFailed(error))`. Closing the store under it ends it this way too.
Resubscribe with `afterSeq = lastSeq`, for example after a restart.
- If the thread is deleted (`GraphRuntime.deleteThread`, `Agent.forget`), through this runtime or another,
the subscription delivers what it has queued and then ends with `Disconnected(lastSeq, ThreadDeleted)`:
at once in the deleting runtime, and in another runtime at its next read of the store (its next poll, or,
with the watch disabled, its runtime's next commit on the thread). It never delivers a new thread created
under the same id as a continuation of the deleted one; subscribe again from `afterSeq = 0` to follow
that one. A subscription in another runtime that has delivered no event yet cannot be told from a new
one, and follows the new thread instead of ending - if it is your tenant's; one of another tenant ends
the subscription with `TenantMismatch` (see [Tenants](#tenants)). A subscription that falls behind while its thread is
deleted ends with `ThreadDeleted`, not `Lagging`: resubscribing from `lastSeq` would find the log gone.
(not compile-checked)
```scala
import org.llm4s.agent.graph.*
import scala.concurrent.duration.*
// a process that only watches threads whose runs execute elsewhere, over the same SQLite file
val watcher = GraphRuntime(store, watch = WatchPolicy(pollInterval = 100.millis))
watcher.subscribe(threadId) {
case StreamEvent.Durable(record) => println(s"${record.seq} ${record.event}")
case _ => ()
}
```
`WatchPolicy.disabled` turns the watch off, for a runtime that knows it is the store's only user. Its
subscriptions then see other runtimes' commits only by subscribing again, which replays the log.
`Agent.stream*` and `AgentRun.subscribe`, and the fs2, ZIO, Java and Kotlin streams built on them, follow
a run that their own runtime started, so they work as before. To follow a thread whose runs execute in
another process, subscribe with `GraphRuntime.subscribe` on a runtime over the same store.
### Ordering across runtimes
Concurrently active subscriptions to a thread that stay caught up see one interleaving of durable and live events within one runtime: every event handed to several of them has the same relative placement for each. That guarantee does not cover a subscription that attaches later, which misses earlier live events, or a slower one, which can lose live events (`LiveGap`) or be disconnected as `Lagging`. Durable `seq` order is preserved for every subscription. The guarantee holds when nothing outside this runtime commits to the thread. A commit made by another runtime or process reaches each subscription through its own store watch, through a hand-over's gap read, or, for a subscription still in
its initial replay, directly from that replay, so two subscriptions can place it differently; an external commit allows that, but does not necessarily make them differ.
If you need one order across runtimes, order only `StreamEvent.Durable(record)` by `record.seq` (or
collect those events and sort by `seq`). There is no durable-only subscription option, so filter in the
listener. Ignore `StreamEvent.Live` and `StreamEvent.LiveGap`: live events have no `seq` and are not
stored, so they cannot be placed in a cross-runtime order. Treat `StreamEvent.Disconnected` as the end
of the subscription, not as a record: it carries `lastSeq` and a reason, not an `EventRecord`. The structural fix, one ordered per-thread log shared by all subscriptions, is recorded as future
work in the design record ([#1920](https://github.com/llm4s/llm4s/issues/1920)).
## fs2, ZIO and Pekko Streams
`llm4s-effect`, `llm4s-zio` and `llm4s-pekko` expose the same stream as a value:
(not compile-checked)
```scala
// cats-effect / fs2: Stream[IO, AgentStreamItem]
agentIO.stream(threadId, "Explain monads").evalMap {
case AgentStreamItem.Event(AgentEvents.TextDelta(d)) => IO.print(d.text)
case AgentStreamItem.Event(_) => IO.unit
case AgentStreamItem.Done(result) => IO.println(s"\n${result.status}")
}.compile.drain
// ZIO: ZStream[Any, LLMError, AgentStreamItem]
agentZ.stream(threadId, "Explain monads").runForeach { /* same cases */ }
// Apache Pekko Streams: Source[AgentStreamItem, NotUsed]
agentPekko.stream(threadId, "Explain monads").runForeach { /* same cases */ }
```
`AgentStreamItem` is `Event(StreamEvent)` or `Done(AgentResult)`, one enum per module. The stream
ends after `Done`; a `Left` from admission or from the run fails the stream with that error.
A run that ends without a terminal event (a crash) still ends the stream, as soon as the run's last
event has been delivered, and the stream then fails with the run's error. `streamResume` and `streamRecover` exist on all three. A Pekko stream fails with an `LLMException` carrying the error; see [Apache Pekko Integration](../pekko).
Interrupting the stream or stopping early (`take(n)`, `head`) cancels the turn.
A slow consumer does not: the stream's buffer never holds up the run's subscription. Durable events
are kept; live events beyond the buffer's 256 are dropped, and the consumer receives one
`StreamEvent.LiveGap(n)` with their count where they were dropped, then the rest of the run and
`Done`. A `Disconnected` from the runtime fails the stream, and so does a consumer that has stopped
draining altogether: once more than `Agent.StreamCapacity` (1024) durable events wait for it, as the
runtime would disconnect a listener that far behind. Samples: `AgentStreamIOExample`,
`AgentStreamZIOExample`.
### Typed streams of one run
`events(run, eventType)` on `AgentIO`, `AgentZ` and `AgentPekko` is a stream of one event type's
decoded payloads, for a run you already hold (from `agent.start`); `events(run, handlers)` turns
several types into one ADT of yours:
(not compile-checked)
```scala
enum Progress:
case Text(delta: String)
case Checked(path: String)
case Gap(lost: Int)
val handlers = EventHandlers
.on(AgentEvents.TextDelta)(d => Progress.Text(d.text))
.on(Checked)(c => Progress.Checked(c.path))
.otherwise { case StreamEvent.LiveGap(n) => Progress.Gap(n); case _ => Progress.Gap(0) }
agentIO.events(run, Checked) // Stream[IO, Checked]
agentIO.events(run, handlers) // Stream[IO, Progress]
agentZ.events(run, handlers) // ZStream[Any, LLMError, Progress]
agentPekko.events(run, handlers) // Source[Progress, Future[Done]]
```
Merge typed streams with the library's own combinators. Such a stream observes the run, through
`AgentRun.subscribe`: it replays the run's durable events from its start, then follows it live, and
misses live events sent before it started. It ends with the run; it does not raise the run's own
error (await the run for that), and stopping it early cancels its subscription, never the run.
Handlers run on the stream, so a slow or throwing handler slows or fails the stream only.
## Java and Kotlin
The Java facade (`llm4s-java-api`) and the Kotlin API (`modules/kotlin-api`) stream a turn the same
way, with the same buffer and the same rules: a slow consumer loses live events and gets one
`StreamEvent.LiveGap(n)`, never cancels the run; a run that ends without a terminal event still
ends the stream with the run's error; and cancelling the stream cancels the run and returns once it
has ended, so the thread can be recovered at once. Both take the thread id as a `String`, as fs2 and
ZIO take a `ThreadId`; `streamResume` and `streamRecover` exist on both. To read what a suspended
turn waits for (`result.status().pending()`, or `JAgent.pending` and `AgentKt.pending`), or to resume or recover without a stream, see
[Suspended turns from Java and Kotlin](index#suspended-turns-from-java-and-kotlin).
**Java.** `JAgent.stream(threadId, query, listener)` returns an `LlmResult` at once - a
failed result for a refused start, and the listener hears nothing. An `AgentStreamListener` gets
each event through `onEvent`, then exactly one of `onComplete(JAgentResult)` or
`onError(LlmException)`, all on the stream's own thread. Only `onEvent` is abstract, so a lambda is
a listener; `StreamEvents.decode` reads an event as an `Optional`:
```java
// streaming = true: the agent's model calls stream, so the turn carries text deltas
JAgent agent = Llm4s.createAgent(client, ToolRegistry.empty(), true);
LlmResult started = agent.stream(threadId, "Explain monads", event -> {
StreamEvents.decode(AgentEvents.TextDelta(), event).ifPresent(d -> System.out.print(d.text()));
if (event instanceof StreamEvent.LiveGap) System.out.print("[...]");
});
LlmResult result = started.get().await(); // or started.get().cancel()
```
`AgentStream.await()` returns once the listener has returned from its last call, with the same
outcome. `cancel()` cancels the turn and returns once it has ended, or after 5 seconds with a WARN if it
has not (see [Threading and cancellation](../java-threading-and-cancellation)); the listener receives at most
the event already being delivered, then `onError` with the cancellation - or `onComplete`, when the
turn had already ended; an interrupt of the thread calling `cancel()` does not cut that wait short,
and is kept. A listener that throws from `onEvent` - or sets its own thread's interrupt flag -
cancels the turn, and `onError` receives what it threw, or a `CancelledError`. `Llm4s.createAgent(client)` and `createAgent(client, tools)` build an agent
that calls the model's `complete`, so its stream carries no text deltas; pass `streaming = true`
for them. `Llm4s.wrapAgent(agent)` takes an agent built with `Agent.builder`, for middleware, a
system prompt or a runtime. `streamResume` takes a `List`, built with `Answer.approve(id)`,
`reject(id, reason)`, `edit(id, argumentsJson)` or `reply(id, json)`. The Java sample in
`modules/samples/gradle-java` streams a turn's text.
**Kotlin.** `AgentKt.stream(threadId, query)` is a cold `Flow`: each collection
runs the turn, emitting `AgentStreamItem.Event(event)` for each event, then
`AgentStreamItem.Done(result)`, a `JAgentResult`. A refused start or a failed run throws `LLMException`. Cancelling
the collection - its scope, `take(n)`, `withTimeout` - cancels the turn; starting and cancelling it
run on `Dispatchers.IO`. A turn cancelled by anything but the collection fails it with
`LLMException`, as in fs2 and ZIO. So does a delivery that dies of a fatal error (an
`OutOfMemoryError`, a `LinkageError`): the Java listener then hears no terminal callback, and the flow -
or `run`, `continueConversation`, `resume` or `recover` - ends with the facade's report of it instead of
suspending.
```kotlin
val agent = Llm4s.createAgent(client, ToolRegistry.empty(), streaming = true)
agent.stream(threadId, "Explain monads").collect { item ->
when (item) {
is AgentStreamItem.Event -> StreamEvents.decode(AgentEvents.TextDelta(), item.event).ifPresent { print(it.text()) }
is AgentStreamItem.Done -> println("\n${item.result.status().kind()}")
}
}
```
The flow's channel holds 64 items; past that, the stream's 256-event buffer fills and live events
are dropped.
## Your own graphs
The same machinery serves graphs that are not agents. An `EventType[A]` names a payload once:
(not compile-checked)
```scala
val Checked = EventType[Checked]("myapp.checked", 1) // name matches [a-z0-9_.]{1,64}
Checked.emit(context, value) // durable: stored with the task's commit
Checked.progress(context, value) // live: sent now, never stored
listener { case Checked(v) => ... } // None for another name, version or an undecodable payload
```
`RunContext.progress(name, version, payload)` is the untyped form. A tool emits through its
context's run: `Checked.progress(context.run, value)`.
Progress is sent only while the task's run holds the thread. A node that keeps its `RunContext`, or
a thread of its own, past the end of its run sends nothing: once the run has released the thread, or
found its claim lost, `progress` is a no-op, so it never reaches a later run's subscribers.
### Typed handlers
`EventHandlers` packages typed handlers with their event types, so a selection of mixed types is a
value, and each handler gets its own payload type:
(not compile-checked)
```scala
val handlers = EventHandlers
.on(AgentEvents.TextDelta)(d => print(d.text))
.on(Checked)(c => println(s"checked ${c.path}"))
.otherwise {
case StreamEvent.LiveGap(n) => println(s"[$n live events lost]")
case _ => ()
}
agent.stream(threadId, query)(handlers) // also run.subscribe()(handlers), runtime.subscribe(...)(handlers)
```
An event is looked up once by name and version and its payload decoded once. Events no handler
takes - kernel events, `LiveGap`, `Disconnected`, other names or versions, a payload that does not
decode - go to `otherwise`, or are skipped without it. A second `on` for the same name and version
replaces the first. `handlers.dispatch(event)` returns a handler's result, for use in your own code.
Live events are at least once across recovery: a tool that `recover` runs again sends its progress
again, to the recovered run's stream. Durable events are not repeated in the log - a failed attempt's
are discarded with its task - but a handler with an effect of its own should key it by something
stable carried in the payload, such as the tool call's `IdempotencyKey`, never by run id.
## Limits
- **Kernel failure messages.** `TaskFailed` and `RunFailed` store an error message. If a guardrail's
or tool's error quotes content (a guardrail reason that echoes the user's text), that text reaches
the log through the kernel event, not through an `agent.*` event.
- **Live events are not replayed**, and a late `subscribe` misses earlier ones. A subscription
receives every live event sent after `subscribe` returns, so `GraphRuntime.subscribe` followed by
`start` sees all of the run's - unless more arrive than its `capacity` holds, while it replays or
before its listener drains them: those are dropped and reported as a `LiveGap` with their count.
- **Live events stay in their runtime.** A subscription receives another runtime's durable events,
but never its live ones, which are not stored. With a polled store, such as SQLite, another
runtime's commits arrive up to `WatchPolicy.pollInterval` after they land.
See also the [observability guide](../observability/), the
[migration note](../../reference/migration.html), and
[Typed Agent Runtime §4.14](https://github.com/llm4s/llm4s/blob/main/docs/design/typed-agent-runtime/4.14-stage-1-slice-3-events-and-tracing.md).
---
Source: https://llm4s.org/guide/agents/durable-checkpointers
# Durable Checkpointers
Share one checkpoint store between runtimes and processes, recover a run whose process died, inspect, fork and
correct a thread's checkpoint history, prune it, and test a store of your own against the contract every store
must meet.
---
## What a checkpointer holds
Every agent turn and every graph run executes on a `GraphRuntime`, which stores its threads in a
`Checkpointer` (`org.llm4s.agent.graph`). For each thread the store keeps:
- its checkpoint history, as versioned JSON: each checkpoint is the graph's state and what was left to run at
one superstep boundary, and the latest says what happens next;
- the pending writes of tasks that finished in the latest checkpoint's superstep, so `recover` does not run them
again;
- the durable event log, numbered `1, 2, 3, ...` per thread;
- the **claim** of the run that holds the thread, if one does.
Two stores ship with LLM4S:
- `InMemoryCheckpointer`, the one `GraphRuntime.inMemory()` and `Agent.builder` use. Its threads live as long as
its instance.
- `SqliteCheckpointer`, in `llm4s-agent-checkpoint-sqlite`: one SQLite file that survives a restart and that
processes on one host can share. See [The SQLite store](#the-sqlite-store).
Both meet the same contract, described below.
## Runs claim their thread
Several `GraphRuntime`s may share one store, in one process or many. A thread may still run only one run at a
time, and the store is what enforces it. Before a run does anything, it asks the store for a **claim**: a lease
on the thread with an expiry and a **fencing token**. While the run executes, it renews the claim, and it
releases the claim when it ends.
(not compile-checked)
```scala
import org.llm4s.agent.graph.*
import scala.concurrent.duration.*
val store = InMemoryCheckpointer()
// two runtimes over one store - in a real deployment, one per process
val a = GraphRuntime(store, claims = ClaimPolicy(ttl = 30.seconds, renewEvery = 10.seconds))
val b = GraphRuntime(store)
```
`ClaimPolicy` sets how long a claim lasts (`ttl`, 30 seconds by default) and how often a running run renews it
(`renewEvery`, 10 seconds by default, counted from the moment the claim is granted; it must be shorter than `ttl`).
The store judges expiry by its own clock, and the run judges its own lease too (see
[Fencing](#fencing-a-run-that-lost-its-claim-stops-and-records-nothing) below).
When the store has a single clock - a database server's, or one process's - the runtimes need not agree on the
time. An embedded store that each process opens itself, such as SQLite on a shared file, judges expiry by the
clock of whichever process grants or renews the claim, so clock skew between hosts shortens or lengthens the
effective `ttl`. Fencing (below) still keeps every write safe; skew costs liveness - a live run taken over early,
a dead one held longer - and can repeat side effects. Keep hosts' clocks synchronised.
`ClaimPolicy` can also carry a `label` naming the runtime to an operator - a host, pod or process name:
`ClaimPolicy.default.withLabel("worker-7")`. The runtime then logs the run id with it, at INFO, whenever it is
granted a claim (`Run 'r1' (runtime 'worker-7') claimed thread 't' with token 1`), and in the WARN lines for a
failed renewal or a lost claim, so the run a `ThreadBusy` names can be traced to the runtime holding it. Without a
label the grant is logged at DEBUG. The label goes nowhere else: not to the store, and never into an error another
caller - another tenant's included - gets.
While one run holds the thread, every other `start`, `recover`, `resume`, `deleteThread` or `updateState` on it,
and every `fork` onto it as the target, from any runtime, is refused with `GraphError.ThreadBusy`, which names the
holding run:
(not compile-checked)
```scala
b.recover(threadId, graph) match
case Left(GraphError.ThreadBusy(_, latestCheckpoint, GraphError.BusyHolder.Run(holder))) =>
// run `holder` is live: retry once it has suspended or completed
case other => ()
```
A refusal up front accepts and discards nothing, so `ThreadBusy` is a `RecoverableError`: the caller retries, and generic retry
code (`RetryPolicy`, `ErrorRecovery`) does too. `holder` is a `GraphError.BusyHolder`:
- `Run(runId)`, the holding run, when it is of the caller's tenant: a run of the same runtime with the caller's
tenant, or a claim the store recorded for the caller's tenant - each claim records its run's
`RunConfig.tenantId` - and the thread's latest checkpoint, if it has one, is the caller's tenant's too;
- `Unnamed`, when another run holds the thread but cannot be named: its claim is another tenant's, or recorded no
tenant (a claim a SQLite file held when it was upgraded to schema 5), or a change's write was refused as stale
naming only the claim that replaced it;
- `Lapsed`, when it is the call's own claim that was lost - a `fork`, `updateState` or `deleteThread` whose lease
passed, or whose store named no current claim, before its write was known to land - and no other run is known
to hold the thread. A write not yet sent was never applied, but one already in the store when the loss was found
may still land, fenced by the token: read the thread before retrying. A retry can still be refused, by the
call's own claim until its release lands or by a run that took the thread over meanwhile.
## Recovering a run whose process died
A process that dies stops renewing its claims. Once a claim has expired, another runtime can take the thread
over: `recover` there continues the run from its last checkpoint, reusing the pending writes of the tasks that
had finished, so they are not run again.
(not compile-checked)
```scala
// process B, after process A died mid-run: refused until A's claim expires, then admitted
b.recover(threadId, graph).flatMap(_.await())
```
So `recover` tells a live run from a dead one by its claim. It does not wait: until the dead run's claim
expires - at most `ttl` after its process stopped renewing - `recover` is refused with `ThreadBusy`, and you
retry. `resume` and `start` are refused the same way while another runtime holds the thread.
A `recover` whose claim replaced another run's expired, unreleased claim records that run's id on its
`RunEvent.RunRecovered` event, as `takenOverFrom`, so the thread's log shows the takeover. That run may have been
paused rather than dead, and still executing. It is recorded even when it is the recovering run's own id - a
`RunId` reused with `RunConfig.withRunId` - since the expired claim was still another, earlier run's;
`takenOverFrom` is `None` when the thread was free. It is `None` too when the claim replaced was another tenant's,
or recorded no tenant: the log belongs to the thread's tenant, and never names another tenant's run. A thread's
id can be claimed by another tenant's run - one admitted before the thread existed, say - and such a claim can lapse
unreleased; the owner's recovery then replaces it without naming it.
So a run of another tenant is never named to the thread's tenant: not by `ThreadBusy`, also in the moment between
its claim and its first checkpoint, not by `takenOverFrom`, and not by `GraphError.ClaimLost`, whose `current` is
only the fencing token of the claim that replaced the run's, a per-thread counter that names no run or tenant.
`recover` reads the dead run's pending writes again once its own claim is granted, so a task that finished in
the last moments before the takeover is reused, not run again.
## Fencing: a run that lost its claim stops and records nothing
A process that was only paused - a long GC, a frozen VM - may wake up after another runtime took its thread over.
Every commit a run makes carries its claim's token, and the store refuses any commit whose token is not the
thread's current claim, with `GraphError.StaleClaim`. Such a run therefore cannot record anything after its
successor started, and the thread's state and event log are its successor's.
A run does not wait for a refused commit to learn that its claim is gone. It judges its own lease: the lease ends
`ttl` after the moment, by the runtime's monotonic clock (`System.nanoTime`), just before it sent the request that
granted or last renewed the claim. The store started its own `ttl` no earlier than that, so the run's lease ends no
later than the store's - provided the store judges expiry by one clock that does not run faster than the runtime's
monotonic clock and is not stepped forward. A constant offset between the two clocks does not matter. A store clock
that runs fast or is stepped forward (an NTP step on `Clock.systemUTC()`), or a store whose processes each judge
expiry by their own skewed clock (SQLite shared between processes), can expire the claim while the run still judges
it live, so another runtime can take the thread over early. The run then learns of it from its next refused renewal
or commit - within `renewEvery` - and stops; fencing keeps every write safe, but tasks it was already running can
repeat side effects (see [Limits](#limits)). A heartbeat that never waits on the store checks the lease,
so a run whose renewals keep failing, or hang - its store unreachable, or partitioned from it - still notices. A
renewal that fails, or a store clock that throws, is retried at the next beat while the lease lasts.
The run finds its claim lost when its lease passes with no renewal landing, when a renewal is refused with
`StaleClaim`, or when a commit is. It then stops at once, in every durability mode:
- its running tasks are interrupted, as `cancel` interrupts them;
- it commits nothing more - not even `RunFailed` - and drops what it had not yet written: the queue of
`Durability.Async` and the buffer of `Durability.OnExit`. From the moment the loss is found, a commit the run
has not yet begun is skipped, even one of its outcome while it is finishing. A commit begins, or is skipped, in
one atomic step with that check, so the only commits of the run that can reach the store after the loss are ones
that had already begun when it was found. An `Async` commit already in the store is waited for
at most `ttl`; one still there then is left to finish on its own. It is fenced by the run's token, so it lands
only if nobody has claimed the thread since. If it lands, it lands after the run has ended: its events are an
orphan of the run, not part of it. They are in the thread's log, and reach the thread's subscribers in sequence
and before any later run's events, but never the run's own listeners or its tracing (see [Limits](#limits)).
The same holds for a commit that has landed but that this runtime has not yet handed to its subscribers, because
it is still reading, from the store, events committed elsewhere that one of them has not seen yet: a commit
reaches every subscription of the thread at once, or none, so all of them, and a replay, agree on it;
- it releases its claim, which is a no-op if another run holds the thread. It does not wait for a renewal that
may still be in the store, and the release runs on a thread of its own, waited for only until `2 × ttl` after
the loss, so the run's result and the thread's release in this runtime never wait on a store that hangs; a
release that never lands lets the claim lapse after `ttl`;
- it ends with `GraphError.ClaimLost(threadId, token, current)`, where `current` is the thread's current claim
when the store named one;
- its end is bounded whatever it was doing: `await()` returns, and the thread is free in this runtime, within
`2 × ttl` of the loss being found, plus only local work that never calls the store. The loss is found when the
lease passes - at most `ttl` after the last renewal that landed was sent - or when a renewal or commit is refused,
so the bound is at most `3 × ttl` after the last renewal that landed. If the run has not begun to end by
`2 × ttl` after the loss - its thread stuck in a store call that ignores interrupts, such as a `Sync` commit,
`OnExit`'s exit commit, or the commit of its outcome, which no stop interrupts - it is ended without its thread:
it fails with `ClaimLost`, its thread is freed in this runtime and its claim released as above. When the stuck
call does return, it changes nothing: the run has its one result, and its thread is not freed again. A write it
makes that lands then is an orphan, as above. The same bound holds while the run is admitted: if the commit
that claims the thread for `start`, `recover` or `resume` hangs in the store and the claim is found lost
meanwhile, the call returns `ClaimLost` within `2 × ttl` of the loss, with the thread freed and the claim
released as above. The run never starts. If that commit lands later, it is fenced by the token: once the
release has landed, or another run has taken the thread over, the store refuses it. But one that lands first -
the store still holds the claim, since nobody took it over - leaves the admission's `Running` checkpoint and its
`RunStarted` (or `RunRecovered` or `RunResumed`) event in the thread although the call returned `ClaimLost`, and
no run continues it: `recover` does, once the claim is released. A claim's commit that the store refuses as stale - the claim lapsed, or another runtime took it over,
before it landed - is a loss found then: the call returns `ClaimLost`, never `CheckpointWriteFailed`, with the
claim released and the thread freed. If the heartbeat declared the loss meanwhile, the call still returns one
`ClaimLost`, naming the current claim the first of the two to find the loss named.
`fork`, `updateState` and `deleteThread` are bounded the same way while they hold their claim: if it is found
lost while the change waits on the store - its next call queued behind a hung renewal in a store that serialises
its calls, say - the call returns `ThreadBusy` naming no checkpoint within `2 × ttl` of the loss, with the thread
freed in this runtime and the claim released as above. A change is not a run, so it never returns `ClaimLost`:
retry it once the thread is free. A change's write - its commit, or the store delete - that has not begun when
the loss is found never reaches the store, and the change returns `ThreadBusy` as for a commit the store refuses
as stale. A write the store refuses as stale is itself a loss found then, as a run's is: the change returns
`ThreadBusy` within `2 × ttl` of it, and gives its claim back as above, without waiting on the store. One already
in the store may still land, fenced by the token. A `deleteThread` that lands after its
call was given up on ends this runtime's subscriptions to the thread at once only if the call has not yet freed
the thread; otherwise they learn of the delete from the store, as another runtime's do (see
[Deleting a thread](#deleting-a-thread)).
`ClaimLost` is not a store outage (`CheckpointWriteFailed`). After it, `recover` may be attempted once the thread
is free:
- while a successor holds the thread, `recover` is refused with `ThreadBusy`: retry later;
- if a successor took over and finished the work, the thread's state shows it, and `recover` reports
`NothingToRecover` (or the thread is suspended or completed, as the successor left it);
- if the lease merely lapsed - renewals failed until it passed, and nobody took the thread over - the run released
its claim and its last durable checkpoint is still `Running`, so `recover` continues it as after a crash.
### Limits
- A stale worker can still act until it notices, and while its tasks see their interrupt. The loss is noticed as
the run's own lease passes, or as soon as a renewal or commit is refused - within `renewEvery` of a takeover that a
store clock running ahead allowed early. A task already
running then may finish an external side effect before it checks its interrupt, and a tool that ignores
interrupts runs to its end. Such a side effect can happen in both the stale run and its successor, concurrently.
That is the at-least-once boundary every run has: make such tools idempotent. A tool or node that writes to a
system able to fence can pass the claim's token on: it is `context.position.fencingToken` in a node and
`context.run.position.fencingToken` in a tool, and it grows with every claim, so the system can refuse a token
lower than the highest it has seen.
- Until the loss is noticed, such a run's live progress events (`StreamEvent.Live`, which are never stored) still
reach the subscribers of its own runtime; from then on its tasks send none, even a task that ignores the interrupt
and outlives the run. Its durable events are refused or dropped with its commits. A live event carries its run's
claim token (`StreamEvent.Live.token`), so a subscription that has been handed a newer claim's commit drops an
older claim's progress, and a run's observer and its run-scoped streams never deliver another claim's, even under
the same `RunId`.
- A `ClaimLost` run's own event stream - `AgentRun.subscribe`, `Agent.stream`, its observer and its tracing - ends
at the run's end, which never waits on a hung store. An `Async` write the run abandoned may still land after
that; it is then visible only through the thread's log - a thread subscription (`GraphRuntime.subscribe`) or a
replay - never as one of the run's events, even to a run-scoped subscription made later that replays the run.
That holds for the next run on the thread too, even if it reuses the lost run's `RunId`: a run's own events are
those committed under its claim's fencing token (`EventRecord.token`), so a late write that lands while it is being
admitted is never delivered to its observer, its stream or its tracing.
- A run that has begun committing its outcome finishes that commit: a loss found during it does not interrupt it.
The commit is still fenced, so it lands only if nobody has taken the thread over. The run's end does not wait for
it past `2 × ttl` of the loss: if the commit returns first the run ends with its outcome (or `ClaimLost`, if the
store refused it), and otherwise with `ClaimLost`.
- Commits on one thread are written one at a time, in this runtime, so that subscribers see them in order. A commit
hung in the store holds up later commits on its thread - including a new run's claim commit there, until the
store returns, which the store's own bound on a call limits (see [When the store hangs](#when-the-store-hangs)) -
but never another thread's. A subscription's read of the store that hangs holds up only that
subscription, and, for a read of events another runtime committed before this runtime's commit, that thread's
commits: never another thread's runs, their admission or release, nor a lost run's bounded end.
- A process that dies holds its threads for up to `ttl`; so does a store that fails to release a claim - except to
its own runtime's `deleteThread`, which tries the release once more first, so `Agent.run(query)` can still forget
the thread of a failed turn. A run that ends with its claim still held waits up to `ttl` for a renewal still in
the store, so that none lands after its release; if the renewal is still there then, the run releases on a thread
of its own without waiting, and its result is set at once. A run whose claim was lost does not wait for its
renewal at all. `fork`, `updateState` and a `deleteThread` that did not delete release the claim they hold the
same way, so a renewal hung in a store that serialises its calls cannot keep them from returning.
## When the store hangs
Every store must return from each call within a bound it documents, and fail a call that reaches it, changing
nothing ([#1919](https://github.com/llm4s/llm4s/issues/1919)). `SqliteCheckpointer`'s is twice its `busyTimeout`,
plus the time its own SQL takes (see [The SQLite store](#the-sqlite-store)); `InMemoryCheckpointer` never waits on
anything but its own brief lock. A store's bound should be well inside `ttl - renewEvery`, the time a renewal has
to land, so that a slow call never costs a run its claim.
The runtime bounds two waits itself, both by `2 × ttl`, so that a store that breaks the contract cannot hold a
run's result or its thread:
- **A run that lost its claim** ends within `2 × ttl` of the loss, whatever it was doing, with `ClaimLost` (see
[Fencing](#fencing-a-run-that-lost-its-claim-stops-and-records-nothing)). So does an admission, with `ClaimLost`,
and a `fork`, `updateState` or `deleteThread`, with `ThreadBusy`.
- **A run's closing wait while it holds its claim.** An `Async` run waits for its queued commits before it returns,
and an `OnExit` run makes its one exit commit. If a commit stays in the store for `2 × ttl` with none returning,
the run ends with `GraphError.CommitTimedOut(threadId, after)`, without waiting for it: `await()` returns, its
thread is free in this runtime, and its claim is released on a thread of its own, which nothing waits for - a
store stuck on a commit may serialise the release behind it. A drain that keeps making progress is never cut
short, however long it takes in all. The commit is an orphan, as a lost run's late write is: it is fenced by the
run's claim, so once the release lands the store refuses it, but it can land first. Its events then reach the
thread's subscribers, never the run's own listeners or its tracing. So after `CommitTimedOut`, read the thread:
its latest checkpoint is either the run's last durable one, still `Running` - `recover` continues the run - or the
run's outcome.
The bound is twice `ttl` rather than a setting of its own. A store that keeps claims alive already returns well
inside `ttl`, since a renewal must land within `ttl - renewEvery` and a store that serialises its calls queues
renewals behind its commits; the runtime's bound is a backstop for a store that breaks the contract, and the same
`2 × ttl` that bounds a lost run's end.
Every other call made while the claim holds - an admission's claim commit, a `Sync` run's commits, its outcome's
included, a change's write, and the release of a claim that was not lost - is waited for as long as it takes:
only the store's bound limits it. Nor can the runtime free what a call stuck in the store holds itself. While the
call lasts:
- **its thread stays alive.** A run thread, or the virtual thread on which an admission, `fork`, `updateState` or
`deleteThread` makes its store calls, stays inside the call after the runtime has given it up - and an `Async`
run's writer thread with it. Only the run's result, its slot and its claim are freed;
- **it holds up the thread's later commits.** Commits on one thread are written one at a time in a runtime, so a
commit, or a hand-over's read of events another runtime committed, holds up every later commit on that thread
in that runtime until it returns - a new run's claim commit included - but never another thread's;
- **it can hold up subscriptions.** A `GraphRuntime.subscribe` made while a run's claim commit is in the store, on
a thread with nothing of that run handed over yet - an admission given up on after a lost claim included - waits
for that commit to return, or to be cancelled, before it delivers its replay; a run-scoped subscription
(`RunHandle.subscribe`, `AgentRun.subscribe`) never does. And with the store watch disabled
(`WatchPolicy.disabled`), a replay made while a write a run gave up on is still in the store stops at the first
event another runtime committed after what this runtime has handed over; it gets the rest when this runtime
next commits on the thread, and a replay made once the write has returned has them all.
## Deleting a thread
`GraphRuntime.deleteThread(threadId, config)` (`Agent.forget`) claims the thread, checks its tenant under that
claim, and then has the store delete it with the claim's token, renewing the claim while the store works, as a run
does. The store's delete is fenced like a commit: it
deletes only while the token is the thread's current claim, atomically. So a runtime that stalls past its claim's
`ttl` between the claim and the delete - a long GC pause, a slow store - while another runtime takes the thread over
deletes nothing of that run: the call returns `ThreadBusy` (or `TenantMismatch`, if the run is another tenant's).
A deleted thread's id names a new thread, but the new thread does not reuse the old event numbers: it numbers its
events on past the deleted thread's, one number skipped. A subscription follows a thread, not its id, so a
subscription to the deleted thread ends with `Disconnected(lastSeq, DisconnectReason.ThreadDeleted)` once it has
delivered what it had queued, and never delivers the new thread's events as if they continued the old log:
- in the deleting runtime, every subscription made before `deleteThread` returns - before the delete, or while it
is in flight - ends at once;
- in another runtime sharing the store, a subscription ends when it next reads the store: when its watch next polls,
or, with `WatchPolicy.disabled`, only when its own runtime next commits on the thread;
- in another runtime, a subscription that has delivered no event yet - still at `afterSeq = 0`, as one to a thread
that never numbered an event always is - cannot be told from a new subscription, so it follows the new thread
from its first event instead of ending - if the new thread is its tenant's; otherwise it ends with
`TenantMismatch` before that event ([Tenants](streaming#tenants)).
Subscribe again from `afterSeq = 0` to follow the new thread from its first event.
There is no unfenced delete: to delete a thread outside a runtime, claim it and delete with the claim's token.
### Limits
- A deleted thread that numbered any event leaves a small record in the store, a tombstone: its next event number and
when it was deleted, nothing of its data. It is what lets a reader of the deleted log, in any process, be told that
the thread was deleted, and a new thread of the id be numbered past it. `prune` never removes it, so a store grows by
one per deleted thread - `Agent.run(query)` forgets the thread of every failed or cancelled one-shot turn, and child
retention deletes children - until a [sweep](#sweeping-the-whole-store) with a `tombstoneAge` reclaims the old ones.
Deleting the same id again, with no event since, adds nothing.
## Subscriptions across runtimes
A subscription sees every commit on its thread, made through any runtime or process that shares the store, not
only through its own: durable events in `seq` order, without gap or duplicate, as for its own runtime's commits.
(Where another runtime's commit falls among this runtime's live events is not fixed: it depends on whether the
subscription's initial replay, its store watch or a hand-over's gap read delivers it first; see
[Ordering across runtimes](streaming#ordering-across-runtimes).) Each live subscription
watches the store with `Checkpointer.awaitEventsAfter`. A store that is told of commits, such as PostgreSQL's (by
`LISTEN`/`NOTIFY`, [below](#the-postgresql-store)), returns at once. A store that is not, such as SQLite, is polled every `WatchPolicy.pollInterval` (250 ms by default):
(not compile-checked)
```scala
val runtime = GraphRuntime(store, watch = WatchPolicy(pollInterval = 100.millis))
```
Only durable events cross runtimes; live progress is never stored. A thread deleted through any runtime ends its
subscriptions with `ThreadDeleted`, with the exceptions [Deleting a thread](#deleting-a-thread) lists. See
[Subscriptions across runtimes](streaming#subscriptions-across-runtimes) for the delivery and back-pressure rules.
## The SQLite store
(not compile-checked)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-agent-checkpoint-sqlite" % llm4sVersion // same version as llm4s-agent
```
`SqliteCheckpointer` keeps every thread - its latest checkpoint, pending writes, event log and claim - in one
SQLite database file. Open it with a path, hand it to a `GraphRuntime` (or to `Agent.builder` through
`withRuntime`), and close it when the process is done with it:
(not compile-checked)
```scala
import org.llm4s.agent.graph.*
import org.llm4s.agent.graph.sqlite.SqliteCheckpointer
import java.nio.file.Path
for
store <- SqliteCheckpointer.open(Path.of("/var/lib/my-app/runs.db"))
runtime = GraphRuntime(store)
handle <- runtime.start(threadId, graph, input)
result <- handle.await()
yield result
```
`open` creates the file if it does not exist (its directory must), and brings it to the current schema version.
Any number of stores may open one file at the same moment, also a new one: each waits up to `busyTimeout` for the
others. The path may hold any character the file system allows, `?` and spaces included. It must be on a local
disk: a Windows UNC path (`\\server\share\runs.db`) becomes a `file://server/share/runs.db` URI, which the
native libraries bundled with sqlite-jdbc refuse (they are built without `SQLITE_ALLOW_URI_AUTHORITY`), and WAL
does not work on a network file system anyway (see [Sharing one file between processes](#sharing-one-file-between-processes)).
An open interrupted while it waits for a busy file returns a `CancelledError` and closes its connection.
`SqliteCheckpointer.open(path, clock)` and `open(path, clock, config)` take the clock claims expire by and a
`SqliteCheckpointerConfig`.
### Restart and recovery
Everything a run commits is in the file, so a process that stops - crashes, is killed, or is redeployed - loses
only the work in flight. A new process opens the same file and continues:
(not compile-checked)
```scala
// after a restart: the same file, a new store, a new runtime, the same graph
for
store <- SqliteCheckpointer.open(Path.of("/var/lib/my-app/runs.db"))
events <- store.eventsAfter(threadId, 0L, 100) // the stopped run's event log, as it was committed
handle <- GraphRuntime(store).recover(threadId, graph)
result <- handle.await()
yield result
```
`recover` reuses the pending writes of the tasks that had completed, so they are not run again; only the tasks
that were running or not yet started run. A run that was killed still holds its claim until the claim expires
(`ttl`, 30 seconds by default), so `recover` is refused with `ThreadBusy` until then; retry it. A run that
failed - a node returned `NodeResult.Fail` - released its claim as it ended, and `recover` is admitted at once.
Event numbers continue where the stopped run's log ended: none is reused, and a subscriber that had seen events
`1..n` resubscribes with `afterSeq = n`.
The sample `org.llm4s.samples.durable.SqliteRestartRecoveryExample` shows the whole sequence:
`sbt "samples/runMain org.llm4s.samples.durable.SqliteRestartRecoveryExample"`.
### Sharing one file between processes
Several stores - in one process or in several on the same host - can open one file, each with its own connection,
and the runtimes over them exclude each other through claims exactly as runtimes over one store do. Every write is
one `BEGIN IMMEDIATE` transaction, in which the claim's token and the thread's version are checked and the commit
applied, so two stores cannot both pass the checks.
- **WAL mode.** The store switches the file to SQLite's write-ahead log. Readers - `latest`, `eventsAfter`, a
subscriber's replay - never wait for a writer, and a writer never waits for readers. WAL needs memory shared
between the processes that open the file, so keep the file on a local disk, not on a network file system.
- **One writer at a time.** SQLite admits one writing transaction per file. The store's writes are short, so a
write that finds another in progress waits for it - up to `SqliteCheckpointerConfig.busyTimeout`, 5 seconds by
default - and only then fails, with a `ProcessingError` and nothing changed. Raise it if many processes write to
one file at once: `SqliteCheckpointer.open(path, clock, SqliteCheckpointerConfig(busyTimeout = 15.seconds))`.
- **Writes take turns within a process.** SQLite does not queue a waiting writer. It sleeps and tries again, and
gets the lock only if it wakes between two of the other connection's transactions. A store beside one that
commits back to back could therefore wait out its whole `busyTimeout`, even though each write was short. The
stores in one JVM that open the same file take turns through a fair lock for the file, first come, first served,
before each write. A write waits for the writes queued ahead of it, up to `busyTimeout`, and then up to
`busyTimeout` again for SQLite's lock if another process holds it. Writes from different processes are still
ordered by SQLite's polling, so a process that writes steadily can starve another process's write. Raise
`busyTimeout` there. A write interrupted while it waits for its turn returns a `CancelledError`.
- **Every call is bounded.** A call waits for the store's own earlier calls, and a write then for its turn at the
file, up to `busyTimeout` from the call's start, and a write then up to `busyTimeout` again for SQLite's lock: so
every call returns within `2 × busyTimeout`, plus the time its own SQL takes, failing with a `ProcessingError` and
nothing changed once it reaches a limit. That SQL itself cannot be timed out: sqlite-jdbc's native calls ignore
interrupts, so a disk that stalls inside a write holds that call, and the store's later calls fail once they have
waited `busyTimeout` for it. With the default 5 seconds and the default `ttl` of 30 seconds, the bound is well
inside `ttl - renewEvery`; keep it so if you raise either (see [When the store hangs](#when-the-store-hangs)).
- **Durable commits.** `synchronous = FULL`: a commit that returned survives a crash or a power loss.
- **Calls within one store are serialised.** A store is one connection; it is safe to share between threads, and
a busy runtime gains from a store of its own rather than from sharing another runtime's.
- **Virtual threads.** sqlite-jdbc makes its native calls inside `synchronized` methods, so on JDK 21 a call from
a virtual thread - as the runtime's tasks are - pins its carrier thread for as long as the call lasts: up to
`busyTimeout` while it waits for another connection's write. Calls within one store are serialised, so each store
pins at most one carrier at a time; keep `busyTimeout` short if many stores in one process write to busy files.
### Watching the file
SQLite cannot tell one connection that another has committed, so a subscription over a `SqliteCheckpointer` sees the
commits of other stores on the file by reading the event log, once per `WatchPolicy.pollInterval`. Each read is a
short indexed query, and in WAL mode it never waits for another connection's writer. It can wait for its own store,
though: the calls on one `SqliteCheckpointer` share its one connection and are serialised behind its lock, so a
watch's read waits while that store is committing - including a commit that is itself waiting, up to
`busyTimeout`, for another connection's write. A process's own runtime still delivers its own commits at once. Each live subscription reads once per interval, so where many subscriptions watch one file, raise
the interval. A subscription whose store is closed under it ends with `Disconnected(lastSeq, ReplayFailed(...))`,
from which the next process resubscribes.
### Clocks
A claim's expiry is judged by the clock of the store that grants or renews it - `Clock.systemUTC()` unless you pass
one. Stores on one host share that host's clock. Processes on different hosts cannot share one SQLite file safely
anyway (see WAL above), but if host clocks still differ - containers with skewed clocks, a clock step - a claim
lasts longer or shorter than `ttl` as other processes see it. Fencing keeps every write safe regardless. A store
clock running ahead lets another process take over a live run early; that run then finds its commits or renewals
refused and stops, but tasks it was already running can repeat side effects (see
[Limits](#limits)). A store clock running behind holds a dead run's thread longer. A run's own lease, judged by its
monotonic clock, does not protect against a store clock that runs ahead: the run learns of the takeover only from its
next refused renewal or commit.
### Schema versions
The file records its schema version: 2 since the fenced delete (which added `first_seq`, where a deleted thread's id
continues its event numbering), 3 and 4 for effect records, and 5 since claims record their tenant (`claim_tenant`,
with `claim_tenant_token`, the token of the claim the tenant belongs to), 6 for child threads, and 7 since each
checkpoint and event records when the store committed it, which retention judges age by (rows from before 7 are judged
by their producer's timestamps), and 8 for the thread registry (see [Finding threads](#finding-threads)), filled from
each thread's latest checkpoint when a version-7 file is opened, and 9 for tombstone reclamation (see
[Sweeping the whole store](#sweeping-the-whole-store)), which dates the tombstones a version-8 file holds from the
upgrade. A claim a version-4 file holds when it is
upgraded has no tenant recorded, so it matches no tenant: its holder is named to nobody, not even its own tenant,
and a recovery that takes it over records no `takenOverFrom`. Such a claim is no older than its `ttl`, so this lasts
at most that long. Stop every process running an older build before opening its file with this one: an
older process that already has the file open is not refused, and until it stops its deletes are not fenced and drop
the record a deleted thread's id numbers on from. Opening a file at an older version upgrades it in one transaction, and
opening a file written by a newer build is refused with a `ProcessingError`, so an older process never misreads
it. Checkpoints inside keep their own format version, as with every store, and are migrated when read.
### What it stores
Only data: checkpoints, pending writes and events as JSON, never closures or class names, so a file can be read by
any build that knows the graph. Its tables are prefixed `llm4s_checkpoint_`, so the file can hold other tables
too. It keeps every checkpoint of a thread's history, as the next section describes: `history` pages through it
newest first by an index, so a long history costs a page, not a scan, and `prune` and `deleteThread` remove from it
in one transaction each, safely beside other stores writing the file. `prune` reads the thread's whole history and
event log inside its transaction, so on a very long log it holds the file for a while and other stores' writes wait,
up to `busyTimeout`; prune regularly, or off-peak, rather than once after months.
## The PostgreSQL store
`PostgresCheckpointer`, in `llm4s-agent-checkpoint-postgres` (`org.llm4s.agent.graph.postgres`), keeps threads in a
PostgreSQL schema that runtimes on any number of machines share: the store for one agent service on several machines.
[Running agents on several machines](multiple-machines) puts it together with the run queue, workers, cancel,
tracing, metrics and retention into one service.
(not compile-checked)
```scala
import org.llm4s.agent.graph.GraphRuntime
import org.llm4s.agent.graph.postgres.{ PostgresCheckpointer, PostgresCheckpointerConfig }
for
store <- PostgresCheckpointer.open(
"jdbc:postgresql://db:5432/app",
user,
password,
PostgresCheckpointerConfig.default.withSchema("agents")
)
runtime = GraphRuntime(store)
// ...
yield store.close()
```
`open` builds a HikariCP pool of its own; `PostgresCheckpointer.create(dataSource, ownsDataSource, config)` uses a
`DataSource` the application already has, and closes it with the store only if `ownsDataSource`.
- **One schema, created and migrated on open.** The tables live in the configured schema (`llm4s` by default), so
the store can share a database. Opening creates the schema if it is missing and brings it to the current version in
one transaction under an advisory lock, so instances starting together migrate it once; a schema already current
is only read, so the application's role needs no `CREATE` once it exists.
- **The database decides time.** Claim expiry and retention age are judged by the server's clock, never a host's, so
skew between machines changes nothing.
- **Every write locks the thread's row** (`SELECT ... FOR UPDATE`) and checks the claim, the fence and the parent
checkpoint before it writes; two rows are always locked in the order of their ids, so fenced commits never
deadlock. Fencing tokens come from one sequence for the schema.
- **Every call returns within `connectionTimeout + callTimeout`** (5 + 5 seconds by default): the call timeout is set
on the server for each transaction and enforced on the client, which discards a connection whose call overran it.
A caller-supplied pool's own connection timeout stands in for `connectionTimeout`; set a driver socket timeout on it.
- **Subscriptions are told of commits** by `LISTEN`/`NOTIFY`. A commit that stores events, and a delete, notifies the
schema's channel (`llm4s_checkpoint_`) inside its transaction, so only a commit that lands is announced;
the payload is a hash of the thread id, nothing of its content. Each store opens one more connection, outside its
pool, on its first watch (`ApplicationName` `llm4s-agent-checkpoint-postgres-listener`), and keeps it until
`close`: count it when sizing `max_connections`. Over a `DataSource` that is not a HikariCP pool with a JDBC URL,
that connection is taken from the `DataSource` instead, and held. A commit through any store over the schema wakes
a waiting subscription within a second on a healthy database, whatever its `pollInterval`.
- **A lost listening connection degrades to polling.** While it is down - before it first opens, after the server or
network drops it, or once its heartbeat (a `SELECT 1` whenever `callTimeout` passes quietly) fails - each watch
reads once per `WatchPolicy.pollInterval`, as with SQLite, and the store reconnects in the background. A
notification is only a wake-up: the watch always reads the log, so a missed or repeated one loses nothing. Each
watch still reads once per `pollInterval` while it waits, so where many subscriptions watch one database, a longer
`pollInterval` cuts reads without slowing delivery.
### Upgrades
A release that changes neither the store's schema nor the checkpoint format can replace instances one at a time. A
release that changes either needs every instance stopped before the first new one starts: an old instance that meets
the newer schema refuses to open with `GraphError.UnsupportedStoreSchema`, and writes nothing.
[Running agents on several machines](multiple-machines#upgrades) covers the rest of a rolling upgrade: draining,
and runs in flight at an older graph version.
## Choosing the store in application.conf
`CheckpointerConfigLoader.open()` (`org.llm4s.config`, in `llm4s-agent`) opens the store `llm4s.agent.checkpointer.kind`
names, and `RuntimeConfigLoader.runtime(checkpointer)` builds a `GraphRuntime` over it with the `ClaimPolicy` and
`WatchPolicy` from `llm4s.agent.runtime`. Changing store is then a dependency and a setting, with no code:
```hocon
llm4s.agent {
checkpointer {
kind = "postgres" # memory (the default) | sqlite | postgres; or LLM4S_CHECKPOINTER
postgres {
url = "jdbc:postgresql://db:5432/app" # or LLM4S_CHECKPOINTER_POSTGRES_URL
user = "agents" # or LLM4S_CHECKPOINTER_POSTGRES_USER
# password: LLM4S_CHECKPOINTER_POSTGRES_PASSWORD
schema = "agents"
callTimeout = 5 seconds
connectionTimeout = 5 seconds
maxPoolSize = 10
}
sqlite {
path = "/var/lib/app/threads.db" # or LLM4S_CHECKPOINTER_SQLITE_PATH
busyTimeout = 5 seconds
}
}
runtime {
claims { ttl = 30 seconds, renewEvery = 10 seconds } # label: LLM4S_RUNTIME_LABEL
watch { enabled = true, pollInterval = 250 milliseconds }
}
}
```
(not compile-checked)
```scala
import org.llm4s.agent.service.RunSubmitter
import org.llm4s.config.{ CheckpointerConfigLoader, RunQueueConfigLoader, RuntimeConfigLoader }
for
store <- CheckpointerConfigLoader.open() // a CheckpointStore: close it when the application stops
runtime <- RuntimeConfigLoader.runtime(store.checkpointer)
agent <- Agent.builder("assistant", client).withRuntime(runtime).build()
queue <- store.runQueue // the store's own run queue, on the same connection or pool
settings <- RunQueueConfigLoader.load() // llm4s.agent.queue
submitter = new RunSubmitter(queue, store.checkpointer, settings)
yield (store, agent, submitter)
```
- **`memory`** is built into `llm4s-agent` and is the default; **`sqlite`** needs `llm4s-agent-checkpoint-sqlite` and
**`postgres`** `llm4s-agent-checkpoint-postgres`. Each store module ships its block's defaults in its own
`reference.conf`. A kind whose module is not on the classpath fails only `open`, with an error naming the module.
- **The store's run queue comes with it.** `store.runQueue` (a `Result[RunQueue]`) is the queue in the same database
as the checkpoints, on the checkpointer's own connection or pool, so `llm4s.agent.queue` holds only the queue's own
settings, as above. All three built-in kinds have one; a store whose backend gave it none answers a
`ConfigurationError`.
- **A busy thread is the submission's choice** (`policy` on `submitter.submit`): `BusyThreadPolicy.Reject`, the
default, refuses it with `ThreadBusy` while the thread has an open entry or a run holds it; `Enqueue` waits behind
them; `Interrupt` stops the thread's running queued run at its next superstep boundary - its model call cut, tool
calls already started finished and kept, the rest answered "interrupted" - and runs next, ahead of the queue.
- **Cancel from any machine** with `submitter.cancel(tenant, CancelTarget.Entry(ticket.entryId))` (or
`CancelTarget.Thread(threadId)`): an entry not yet started closes at once; a running one is stopped by its worker at
the worker's next lease renewal, its thread settled so the next entry starts normally, and its ticket closes
`Cancelled` - or with the run's own outcome if it finished first. `submitter.deleteThread(runtime, tenant, threadId,
timeout)` cancels the thread's entries, waits for them to close, then deletes it.
- **Only the selected kind's block is read**, so a half-written `postgres` block does not matter while `kind = sqlite`.
- **Credentials stay out of logs.** No setting's value reaches a log line or an error; set the password through its
variable rather than in a checked-in file.
- **A store of your own** registers a `CheckpointerBackend` (`org.llm4s.agent.graph.spi`): a `class` with a public
no-arg constructor, listed in `META-INF/services/org.llm4s.agent.graph.spi.CheckpointerBackend`, that opens its store
from its block `llm4s.agent.checkpointer.`, handed to it as a `CheckpointerSettings`.
`CheckpointerBackends.of(...)` registers backends explicitly, for a shaded jar whose services files were lost.
## Checkpoint history, fork and updateState
A store keeps every checkpoint a thread's runs commit, not only the latest. `history` lists them newest first, a
page at a time, and `checkpoint` reads one by id. Each is data; restore one with the graph to read its typed
state:
(not compile-checked)
```scala
val latestPage = runtime.history(threadId) // up to 100, newest first
val nextPage = runtime.history(threadId, before = Some(latestPage.toOption.get.last.id))
val state = runtime.checkpoint(threadId, id).flatMap(c => graph.restore(c.snapshot)).map(_.state)
```
Checkpoint ids are `/`, where `n` counts along the thread, so an id never repeats on a thread even
when one `RunId` serves several runs.
**`fork`** starts a new thread from any checkpoint of the history. The new thread's first checkpoint carries
that checkpoint's state, scheduled work, parked interrupts and status, and its event log begins with a
`RunEvent.ThreadForked`. Continue it as its status says: `start` after `Completed` or `Failed`, `resume` after
`Suspended`, `recover` after `Running`. The source thread is only read and is left unchanged.
(not compile-checked)
```scala
runtime.fork(threadId, checkpointId, ThreadId("what-if")).flatMap { _ =>
runtime.start(ThreadId("what-if"), graph, "a different question")
}
```
A fork never copies pending writes, so `recover` on a fork of a `Running` checkpoint runs that checkpoint's tasks
again, in the new thread. The target must be a new thread id (`ThreadExists` otherwise).
**`updateState`** applies a typed `StateUpdate` to the thread's latest checkpoint and commits the result as a new
checkpoint, recorded with a `RunEvent.StateUpdated`. Use it to correct a thread's state, for example before
`resume`. It is optimistic: you name the checkpoint you read, and it is refused with `CheckpointConflict` if the
thread has moved on since. Only the state changes - the status, scheduled tasks and parked interrupts stay - so
the same `start`, `recover` or `resume` continues the thread afterwards.
(not compile-checked)
```scala
for
latest <- runtime.history(threadId, limit = 1).flatMap(_.headOption.toRight(ValidationError("thread", "no checkpoint yet")))
updated <- runtime.updateState(threadId, graph, latest.id, StateUpdate.update(notes, "corrected"), asNode = Some(writer))
handle <- runtime.resume(threadId, graph, answers)
yield handle
```
With `asNode`, the update must stay within that node's declared write set, as if the node had returned it. The
thread is held, with a claim in the store, while it is updated, so a live run refuses it with `ThreadBusy`. If
that claim lapses, or another runtime takes it over, before the update's commit lands, the store refuses the commit
and `updateState` returns `ThreadBusy` too, with nothing applied: retry it. So does a claim found lost before the
commit begins: the commit is never sent. And if the claim is found lost while the update waits on a store that
hangs, `updateState` returns `ThreadBusy` within `2 × ttl` of the loss rather than waiting (see
[Fencing](#fencing-a-run-that-lost-its-claim-stops-and-records-nothing)). `fork` does the same for its target.
Neither is a run, so neither returns `ClaimLost`. Pending
writes of tasks that already completed are carried over, so `recover` still does not run them again. They apply
after your update when the thread is recovered, so a carried write to a key you edited can overwrite your edit on
`recover`. To be sure an edit stands, update a `Running` thread only in keys its completed tasks did not write.
Tool idempotency keys are derived from the thread, the checkpoint id, the superstep, the task and the call id.
`updateState` moves the thread on by one superstep, and checkpoint ids count along the thread, so no later call
reproduces an earlier key, whatever `RunId` you reuse.
A fork copies the checkpoint's snapshot as it is. Once [#1739](https://github.com/llm4s/llm4s/issues/1739) lands,
tool calls already issued at the forked checkpoint record their idempotency keys in the snapshot (in their
`ToolTask`, `ApprovalRequest` or `ToolQuestionRequest`), so in the new thread they keep the source thread's keys;
only model calls the fork makes afterwards derive new keys, since the key includes the thread. This is intended: a
fork that re-runs a call the source already issued - `recover` of a `Running` checkpoint, or `resume` of a parked
approval or question - is de-duplicated against the source's call by its key instead of performing the side effect
a second time.
> `forget` (`deleteThread`) starts a thread id afresh. Reusing both the thread id and a `RunId` after it repeats
> the earlier checkpoint ids and supersteps, and so the earlier tool idempotency keys. Use a fresh `RunId` (the
> default) for runs on a forgotten thread id.
### A guardrail Block leaves no trace in the history
When a node blocks a run (`NodeResult.Block`, as a guardrail does), the run's closing `Failed` checkpoint
**retracts its turn**: every checkpoint after the thread's last `Completed` or `Failed` one is removed from the
history in the same commit - including the checkpoints of earlier runs of the same turn, such as one that
suspended for review. Nothing of the blocked turn stays readable through those checkpoints with `history`,
`checkpoint` or `fork`. Events carry no message content and stay in the log.
The retraction removes the turn's *earlier* checkpoints. The closing `Failed` checkpoint itself - the thread's
latest, which `history`, `checkpoint`, `fork` and the next `start` read - holds whatever the blocking node's own
`Block` update leaves in the state. `Agent`'s output guardrail commits `MessageUpdate.RemoveTurn` there, and an
input Block blocks before the loop stores the turn, so with `Agent` nothing of the turn remains. **A custom blocking node must remove the content in its `Block` update** the same way, or the closing
checkpoint keeps it.
One window remains before the guard runs. The model step commits the answer, and the output guardrail runs in the
next superstep, so between those two commits `history`, `checkpoint` and `fork` can read the unguarded answer. The
Block retracts it from this thread afterwards, but a fork made in that window is another thread and keeps its copy;
the fork's copy of the open turn is guarded when the fork continues that turn.
## Retention
The history grows with every superstep. `prune` removes what a `RetentionPolicy` does not keep, by age or by
size, alongside `compactEvents`:
(not compile-checked)
```scala
runtime.prune(threadId, RetentionPolicy(maxAge = Some(30.days), maxCheckpoints = Some(50), maxEvents = Some(10000)))
```
- `maxAge` removes checkpoints and events older than it, by the store's clock;
- `maxCheckpoints` keeps the newest checkpoints, and `maxEvents` the newest events.
The latest checkpoint and its pending writes are always kept, so pruning never changes what the thread does next.
Pruning a checkpoint also removes the events recorded before the oldest checkpoint kept; `maxAge` and `maxEvents`
may remove more events than that, including those of checkpoints still kept, the latest checkpoint's among them.
Removing events raises the replay floor (`ReplayUnavailable`) as `compactEvents` does. `prune` needs no claim, so it
can run while a run executes; call it on a schedule that suits you.
`maxAge` compares the store's clock with the time the store committed each checkpoint and event, which it records by
that same clock - never with the `createdAt` and event timestamps the committing runtime stamped by its own, which stay
as the producer's record. So a host whose clock is skewed makes nothing too old or too young. The SQLite and in-memory
stores judge by the clock they were opened with; the PostgreSQL store by the database server's.
### Sweeping the whole store
`prune` works on one thread. `Checkpointer.sweep` applies a policy to every thread the store holds, a bounded page at a
time, and reclaims the tombstones of threads deleted long enough ago (design 4.53):
(not compile-checked)
```scala
val policy = SweepPolicy(
retention = RetentionPolicy(maxAge = Some(30.days), maxCheckpoints = Some(50)),
tombstoneAge = Some(7.days),
batchSize = 100
)
@annotation.tailrec
def sweepAll(after: Option[ThreadId]): Result[Unit] =
checkpointer.sweep(policy, after, SweepGuard.none) match
case Right(page) if page.next.isDefined => sweepAll(page.next)
case other => other.map(_ => ())
```
- Each call visits at most `batchSize` threads (1 to 1000), tombstones included, and returns the cursor of the next
page, so every call stays within the store's bound however many threads it holds. The PostgreSQL store makes a call
in up to three short transactions, so one returns within three times its bound.
- Each thread is pruned exactly as `prune(threadId, policy.retention)` would prune it, by the store's commit times.
- A tombstone is reclaimed once the store deleted the thread more than `tombstoneAge` ago, by its clock, and nothing was
stored under the id since (a claim counts until it has lapsed for `tombstoneAge` too).
- A reader of a reclaimed log is still refused with `ThreadDeleted`: reclaiming raises the store's reclamation mark,
and a new thread of any id without a tombstone numbers its events from it, past every reclaimed thread's. So once a
store has reclaimed a tombstone, new threads no longer number from `1`. Paging `children` after a reclaimed child
refuses that cursor.
- A `SweepGuard` names the threads something outside the store still depends on: each keeps its tombstone, and the
newest checkpoint each of its named `RunId`s committed, whatever the policy. A run queue's open entries are what a
guard will name (Stage 5, rule 4); `SweepGuard.none` protects nothing.
A sweep needs no claim. Run one at a time per store, on a schedule: two at once are safe, only wasteful.
## Finding threads
Every store keeps a thread registry (design 4.52): a tenant's root threads can be listed and searched, and the
threads a run left unfinished found, without knowing their ids. A thread's **metadata** is a small map of strings for
finding it again - not for content - set by the `start` that brings it and replaced with `updateMetadata`:
(not compile-checked)
```scala
val config = RunConfig().withTenantId(TenantId("acme")).withMetadata(Map("customer" -> "c-42", "channel" -> "web"))
runtime.start(threadId, graph, input, config)
runtime.updateMetadata(threadId, ThreadMetadata("customer" -> "c-42", "channel" -> "email"), config)
val page = store.threads(
ThreadQuery(Some(TenantId("acme")))
.withStatus(ThreadStatus.Suspended)
.withMetadata(ThreadMetadata("customer" -> "c-42"))
)
page.threads.foreach(t => println(s"${t.threadId.value} ${t.status} ${t.updatedAt}"))
val next = page.next.map(cursor => store.threads(ThreadQuery(Some(TenantId("acme"))).withAfter(cursor)))
```
- **Tenant-scoped.** Every query names one tenant (`None` finds only threads of no tenant) and never sees another's.
- **Newest first** by `updatedAt`, when the store last committed a checkpoint or metadata to the thread, by its clock;
paged by a cursor, at most `ThreadQuery.MaxLimit` (500) a page. A thread changed while you page moves to the front:
a later page of the same pass does not list it again, and the next pass (or `updatedSince`) finds it.
- **Filters**: status, agent (the graph id; an `Agent`'s is its id), metadata entries (all must match) and
`updatedSince`.
- **Root threads only.** A child thread is read through `children`; a deleted thread is not listed.
- **Status** is judged when the store is read: `Running` while a live claim holds the thread, wherever its run is;
otherwise `Idle` (the latest checkpoint is incomplete: a run stopped - crashed, cancelled or drained - and `recover`
would continue it), `Suspended`, `Completed` or `Failed`; `Deleted` while a cascade delete is under way.
`store.incompleteThreads(tenantId, after, limit)` lists the `Idle` threads oldest first, for an operator to look at.
It is not how a service recovers work: a cancelled run, a drained one and a crashed one all look alike here, and only
a run queue's entry says which should run again.
Metadata is limited to 32 entries, keys of `A-Z a-z 0-9 _ . : / -` up to 64 characters and values up to 256
(`ThreadMetadata.of` returns a `ValidationError` past them). The SQLite and PostgreSQL stores index every query, so a
page costs the same however many threads the store holds.
## Writing a store
A store implements `Checkpointer`: `claim`, `renew` and `release` for claims; `commit`, `latest`, `history`,
`checkpoint`, `eventsAfter`, `compactEvents`, `prune` and `deleteThread` for the data. `awaitEventsAfter`, which
subscriptions use to watch the store, has a default that polls `eventsAfter`. The Scaladoc of `Checkpointer` states
the contract. In short:
- grant at most one live claim per thread, by the store's clock, and replace an expired one;
- record each claim's tenant (`ClaimRequest.tenantId`, returned as `RunClaim.tenantId`), and name a claim's holder
to a later claim - as `BusyHolder.Run` in its `ThreadBusy`, or as its `RunClaim.takenOverFrom` - only when that
claim is of the same tenant (`None` and `Some` differ); otherwise refuse with `BusyHolder.Unnamed` and leave
`takenOverFrom` empty. A claim whose tenant the store cannot tell - stored before it recorded tenants - is of no
tenant;
- give every claim a token greater than every token issued for that thread before, also after `deleteThread`;
- apply a `Commit` entirely or not at all, and only with the current claim's token (`StaleClaim`, checked first).
With a `Commit.fence` of `(other, otherToken)`, apply it only if `otherToken` is also `other`'s current claim, in
that same lock or transaction, the thread's own claim checked first (`StaleClaim` for `other` otherwise, writing
nothing);
- delete a thread only with the current claim's token, checked in the same lock or transaction as the delete
(`StaleClaim` otherwise, deleting nothing); a thread with no claim is refused too. With a `fence` of
`(other, otherToken)`, delete only if `otherToken` is also `other`'s current claim, in that same lock or
transaction, the thread's own claim checked first (`StaleClaim` for `other` otherwise, deleting nothing);
- keep, after `deleteThread`, where the thread's event numbering ended - a tombstone with no data, for a thread that
numbered any event - and number the next thread of the id from one past the deleted thread's next number;
refuse `eventsAfter`/`awaitEventsAfter` after a number of the deleted thread with `ThreadDeleted`
(`Checkpointer.deleted(afterSeq, firstSeq)` is the rule), and read from `0`, or from just before the current
thread's first event, from that first event. A store that wakes `awaitEventsAfter` on commits wakes it on a
delete too;
- accept a new checkpoint only over the latest (`CheckpointConflict`) and with an id not already in the history,
and pending writes only for it;
- keep every superseded checkpoint in the history until `prune`, a commit's `retractTurn` or `deleteThread`
removes it, and keep pending writes for the latest only;
- on `retractTurn`, remove every checkpoint after the newest `Completed` or `Failed` one before adding the new one;
- number events contiguously inside the commit, and never reuse a number, also across `deleteThread`;
- record on each event the token of the commit that stored it (`EventRecord.committed(threadId, seq, commit.token,
draft)`): a run's subscriptions tell its events from an earlier run's of the same `RunId` by it;
- release only with the current token, and treat any other as a no-op;
- return from `awaitEventsAfter` what `eventsAfter` would: at once if there is something, and otherwise within about
its `timeout`. A store that can be told of commits, such as a database with a notification channel, overrides it
to return as soon as one lands. It must then wake for commits made through every store over the same storage;
- tolerate the reading thread being interrupted at any point. Cancelling a subscription interrupts its watch, which
may be in the middle of `eventsAfter` or `awaitEventsAfter`: that one call may fail (`CancelledError`), but the
store must stay usable by every other thread. A driver whose I/O goes through a
`java.nio.channels.InterruptibleChannel` (a `FileChannel` or `SocketChannel`) closes the channel when the thread is
interrupted mid-read, so a store sharing one such connection between threads must read on a connection of its own
for the watch, or replace a connection an interrupt closed;
- keep each thread's metadata, replaced only by a commit carrying `Commit.metadata` and removed by `deleteThread`, and
answer `threads` and `incompleteThreads` from the latest checkpoint (its tenant, graph id, status and `deleting`),
the claim judged by your clock, and when you last committed a checkpoint or metadata to the thread - tenant-scoped,
root threads only, by an index, within your bound;
- return from every call within a bound you document - a configured timeout - whatever the storage is doing:
waiting for a lock, another connection's write, the store's own earlier calls or the network. A call that reaches
it fails, changing nothing; `awaitEventsAfter` returns within its `timeout` plus the bound. Say which calls, if
any, you cannot bound, and why. Keep the bound well inside `ttl - renewEvery` (see
[When the store hangs](#when-the-store-hangs)).
### Testing it with the contract suite
(not compile-checked)
```scala
// same version as llm4s-agent; test scope only
libraryDependencies += "org.llm4s" %% "llm4s-agent-testkit" % llm4sVersion % Test
```
`CheckpointerContract` is the contract as a ScalaTest suite. Mix it into a spec and say how to make a store:
(not compile-checked)
```scala
import org.llm4s.agent.graph.Checkpointer
import org.llm4s.agent.testkit.CheckpointerContract
import org.scalatest.flatspec.AnyFlatSpec
class MyCheckpointerContractSpec extends AnyFlatSpec with CheckpointerContract:
protected def newCheckpointer(clock: java.time.Clock): Checkpointer =
MyCheckpointer.open(freshDatabase(), clock)
```
It runs every case against your store: commits (also fenced by another thread's claim), conflicts, event numbering, compaction, `deleteThread` (fenced by
the current token and by the fence thread's, and a recreated thread numbered on past the deleted one, whose log a reader cannot continue), round
trips, the checkpoint history (paging, reads by id, unique ids, retraction and pruning by age, count and event
count), claims and fencing (also from many threads at once), each claim's tenant (a holder named only to a claim of
its own tenant, also across a reopen), and two or more `GraphRuntime`s contending over one
store - a live run refused elsewhere, renewal, takeover after expiry without re-running completed tasks, a stale
run's commits refused, only one of several runtimes admitted to recover a thread, `updateState` and `fork` fenced
against a live run, a blocked turn leaving nothing in the history, and another tenant's claim on a thread never
revealing its run to the thread's tenant through `ThreadBusy`, `RunRecovered.takenOverFrom` or `ClaimLost`. It also checks change notification:
`awaitEventsAfter` itself, and subscriptions in one runtime that receive another runtime's commits in order, mid-run,
across a restart, and only until they are cancelled or the thread is deleted. `InMemoryCheckpointer` passes it.
Four hooks fit it to a durable store:
(not compile-checked)
```scala
class MyCheckpointerContractSpec extends AnyFlatSpec with CheckpointerContract:
protected def newCheckpointer(clock: java.time.Clock): Checkpointer = MyCheckpointer.open(freshDatabase(), clock)
// close the store and open the same storage again, as a restarted process would
override protected def reopen(store: Checkpointer, clock: ManualClock): Checkpointer =
store match
case mine: MyCheckpointer =>
mine.close()
MyCheckpointer.open(mine.location, clock)
case other => other
```
- **`reopen`** runs the cases that check a live claim, its token and the fencing of a stale holder survive a
restart. It defaults to the same instance.
- **`sibling`** opens another store over the same storage, as another process would, while the first stays open:
`MyCheckpointer.open(mine.location, clock)`. The change-notification cases put each runtime on a store of its own
through it. It defaults to the same instance.
- **`boundedStore`** gives the case that checks the bound on every call a store configured with a short one, the
bound it documents, and a way to make the storage busy from outside, as another process holding a write would:
`CheckpointerContract.BoundedStore(store, bound, busy)`. The case makes many calls at once while the storage is
busy, and fails if one takes longer than `bound` plus `boundGrace` (500 ms). It defaults to `None`, which skips the
case: right for a store whose storage nothing outside it can hold, as `InMemoryCheckpointer`'s.
- **`advanceStoreClock`** is how the suite expires claims. By default it moves the `ManualClock` it handed to
`newCheckpointer`, so the store must judge expiry by that `clock`. A store that can only use a clock of its own,
such as a database server's `now()`, overrides it to make claims expire as if that clock had moved - by moving
every stored expiry back by `by`, for example - and sets `exactExpiry = false`, so that `expiresAt` is checked by
its effect rather than against the `ManualClock`.
## See also
- [Running agents on several machines](multiple-machines): one agent service on several instances over the
PostgreSQL store - the run queue, workers, cancel, busy-thread policy, tracing, metrics, retention and upgrades.
- [Streaming events](streaming): subscribing to a thread's events, which a store numbers and replays.
- `docs/design/typed-agent-runtime/`: §4.16 (`4.16-stage-2-slice-1-the-durable-checkpointer-contract.md`), the
design of claims and fencing; §4.19 (`4.19-stage-2-slice-3-checkpoint-history-fork-and-retention.md`), history,
fork, `updateState` and retention; and §4.21 (`4.21-stage-2-slice-6-live-subscriptions-across-runtimes.md`),
subscriptions across runtimes.
- [Migration guide](../../reference/migration#stage-2-migration-durable-execution): what changed for code that
implemented `Checkpointer` or built a `Commit`.
---
Source: https://llm4s.org/guide/agents/human-review
# Human Review
Pause an agent's turn for a person: to approve a tool call, answer a question, review an answer, or
look at what a step did before the next one runs.
---
## Four ways a turn waits
A turn that waits for a person ends `AgentStatus.Suspended`. Everything it waits for is an
*interrupt* with its own `InterruptId`, in one of four lists:
| List | Raised by | Answer with |
|---|---|---|
| `approvals` | a tool's `ToolOutcome.NeedsApproval`, or a tool wrapper such as `ApprovalMiddleware` | `result.approve(id)`, `reject(id, reason)`, `edit(id, arguments)` |
| `questions` | a tool extending `AgentTool.Asking[A, Q, Ans]` | `result.reply(id, answer)`, the tool's `Ans` |
| `middlewareQuestions` | a middleware extending `AgentMiddleware.Asking[Q, Ans]` | `result.reply(id, answer)`, the middleware's `Ans` |
| `breakpoints` | a static breakpoint on one of the agent's nodes | `result.proceed(id)` |
`agent.resume(threadId, answers)` takes any non-empty subset of them. The ones you do not answer stay
pending, so the result can be `Suspended` again with what is left: this is incremental approval. An
unknown id is refused and the thread is unchanged, as is an approval decision, a middleware answer or a
breakpoint answer that does not decode as the asker's type. A tool question's answer is different: the
tool decodes it, so one that does not decode as the tool's `Ans` is accepted, and becomes that call's
error result (`Invalid answer for '': ...`), which the model sees. While a thread waits, `run` and
`recover` on it are refused with `PendingInterrupts`.
(not compile-checked)
```scala
agent.run(threadId, "Deploy the release").flatMap { result =>
result.status match
case AgentStatus.Suspended(approvals, questions, asked, held) =>
val answers = approvals.map((id, _) => result.approve(id)) ++ held.map((id, _) => result.proceed(id))
agent.resume(threadId, answers.toMap) // questions and middleware questions stay pending
case _ => Right(result)
}
```
The waiting thread is persisted before `run` returns, under every durability mode, so another process
with the same agent and checkpointer can resume it.
## Static breakpoints
A breakpoint holds a task of one of the agent's nodes, before it runs or after it ran:
(not compile-checked)
```scala
val agent = Agent
.builder("assistant", client)
.withTools(tools)
.withInterruptBefore(AgentNode.Tool) // each tool call, before it runs
.withInterruptAfter(AgentNode.Model) // each model call, after its message is stored
.build()
```
| `AgentNode` | Node | Before | After |
|---|---|---|---|
| `Model` | `/model` | the model is not called yet | the model's message is stored; its tool calls, or the answer's guarding, wait |
| `Tool` | `/call-tool` | the call has not run | its result is recorded; the next model call waits for it |
| `Finish` | `/finish` | the answer is stored; its `afterAgent` hooks (guardrails) have not run | the answer is guarded; the turn ends once answered |
Each held task is an interrupt of its own. The tool calls of one message are held one per call, so a
reviewer can let them through one at a time. `AgentStatus.Suspended.breakpoints` lists them as
`BreakpointRequest(node, phase, call)`; `call` is the tool call, for a call held before it runs.
`result.proceed(id)` continues one: a task held before it runs is run, and the same breakpoint does not
hold it again; a task held after it ran is not run again - what follows it is released. A breakpoint
takes no other answer.
A step continued after a review runs at another node than the step itself: a model call after a
`wrapModelCall` question at `/asked//wrapModelCall`; a tool call after an approval or a
question at `/approval`, `/ask/` or `/asked//wrapToolCall`; an answer
guarded again after an `afterAgent` question at `/asked//afterAgent` (the root's, for a
root middleware asking about a handoff target's answer). A breakpoint *after* `Model`, `Tool` or
`Finish` holds those nodes too, so nothing the step does escapes it: a model message the gate let
through still waits before its tool calls run, and a result recorded after an approval still waits
before the model sees it. `BreakpointRequest.node` names the node the step ran at. A breakpoint
*before* holds only the step's own node: a continued step was just reviewed.
Each task held after it ran is released in a superstep of its own, which replays its routes without
running the node again. That superstep counts against the run's `RunBudgets.maxSupersteps`, so a run
with many held steps may need a larger budget.
Breakpoints are not part of the agent's graph version, like streaming: a thread an agent holds can be
continued by the same agent built without them. A run can add its own with a `RunConfig`, naming the
node ids: `agent.run(threadId, query, RunConfig(interruptBefore = Set(NodeId("assistant/model"))))`. A
node the agent does not have is refused with a `ValidationError` before the thread is touched.
### On any graph
Breakpoints belong to the graph runtime, not only to agents. A `RunConfig`'s `interruptBefore` and
`interruptAfter` name nodes of any compiled graph; each `start`, `resume` and `recover` brings its own.
A held task is parked like a task that returned `NodeResult.Suspend`, keyed by its task id, and the run
pauses at the end of the superstep - its siblings run and commit. `RunResult.Suspended` reports it as a
`PendingInterrupt` with `breakpoint = Some(BreakpointPhase.Before)` (its `question` is the task's input)
or `Some(BreakpointPhase.After)` (the node's update is committed; its routes, static edges and join
arrivals wait). Answer it with `Breakpoint.proceed`:
(not compile-checked)
```scala
val config = RunConfig(interruptAfter = Set(NodeId("plan")))
for
held <- runtime.start(threadId, graph, input, config).flatMap(_.await())
resumed <- runtime.resume(threadId, graph, Map(InterruptId("0.0") -> Breakpoint.proceed), config)
done <- resumed.await()
yield done
```
Cancelling a run is unaffected: a cancelled run leaves its thread for `recover`, which holds a task the
breakpoint would hold, and runs a continuation that already passed its breakpoint without holding it
again.
## Questions from middleware
A middleware that needs a person - to review an answer, to edit it, or to supply missing information -
extends `AgentMiddleware.Asking[Q, Ans]`, as an asking tool extends `AgentTool.Asking`. A hook asks by
returning `ask(question)` (from `beforeAgent`, `afterAgent` or `wrapModelCall`) or
`askAbout(question)` (from `wrapToolCall`), and reads its answer with `answered(context)` when it runs
again:
(not compile-checked)
```scala
final case class Need(what: String) derives ReadWriter
final case class Info(value: String) derives ReadWriter
final class Region extends AgentMiddleware.Asking[Need, Info]:
val id = MiddlewareId("region")
override def beforeAgent(input: String, context: ModelContext): Result[BoundaryResult] =
if input.contains("region") then Right(BoundaryResult(input))
else answered(context).fold(ask(Need("region")))(r => Right(BoundaryResult(s"$input (region ${r.answer.value})")))
```
The turn suspends with a `MiddlewareQuestionRequest` - the agent and middleware that asked, the hook,
and the question as JSON (read it with `ToolLoop.middlewareQuestion[Need](request)`). Once answered
with `result.reply(id, Info("eu"))`, the asking task runs again from its start, and the whole stack of
that hook runs again from the outermost middleware; nothing about a stack's position is stored. What
runs again depends on the hook:
| Hook | While it waits | Once answered |
|---|---|---|
| `beforeAgent` | nothing of the turn is stored | the turn's input runs again |
| `wrapModelCall` | nothing of the model call is stored, but a call the wrapper made before asking is counted in usage | the model step runs again: a wrapper that asked after calling `next` calls the model again, unless it returns a completion of its own |
| `wrapToolCall` | the call has no result | the call runs again: its arguments are checked again, with the approval it had |
| `afterAgent` | the answer is stored, the turn has no outcome yet | the answer is guarded again |
Answers given earlier in the same stack run are kept while it runs again, so two asking middleware in
one stack each ask once, and a tool call's answer is kept through a later approval or tool question.
Because the stack runs again, a middleware outside the asking one runs twice and must not depend on
running once. A tool wrapper's questions are one per call, so the calls of one message can be answered
one at a time. A middleware that asks while the tool continues after its own question gets an error
result instead, since running the call again would lose the tool's answer.
### Reviewing what guardrails refuse
`GuardrailReviewMiddleware(input, output)` runs guardrails as `GuardrailMiddleware` does, but a refusal
asks a reviewer instead of blocking. The question is a `GuardrailReview(phase, guardrail, reason,
text)`; the answer a `GuardrailVerdict`: `Allow` lets the text through, `Edit(text)` replaces it, and
`Block` blocks the turn as `GuardrailMiddleware` would have (`AgentStatus.Blocked`). A verdict holds for
the text and phase it was given for, whatever the refusal's wording: once given, the guardrails do not
run on that text again, so an LLM judge whose reason varies from call to call is asked about once and is
not called again.
## From Java and Kotlin
`JAgentStatus.pending()` lists every interrupt as a `PendingInterrupt`, approvals first, then tool
questions, middleware questions and breakpoints. `kind()` is the Java enum `InterruptKind`: `APPROVAL`,
`QUESTION`, `MIDDLEWARE_QUESTION` or `BREAKPOINT`. A field that only some kinds have is an `Optional`:
`toolName()` and `argumentsJson()` when a tool call waits; `reason()` for an approval; `questionJson()`
for a question; `middleware()` for a middleware question; `node()` and `phase()` (the Java enum
`BreakpointPhase`, `BEFORE` or `AFTER`) for a breakpoint. `Answer.proceed(id)` continues a breakpoint;
`Answer.reply(id, json)` answers either kind of question. An agent with breakpoints or asking
middleware is built with `Agent.builder` and wrapped with `Llm4s.wrapAgent`. See
[Suspended turns from Java and Kotlin](index#suspended-turns-from-java-and-kotlin).
---
Source: https://llm4s.org/guide/agents/delegation
# Delegation to subagents
Hand a bounded task to another agent and get back only its typed report.
---
## Overview
A **subagent** is a `TypedAgent[I, O]`: an agent that takes a typed input `I` and gives back a typed report `O`. A
parent offers its subagents with `AgentBuilder.withSubagents`, and its model delegates through the `task` tools. Each
delegated task runs on a **child thread** of the parent's thread, with a fresh conversation and only the child's own
tools and middleware, plus what its [inheritance](#inheritance) grants. Only typed data crosses: the parent sends `I`, which becomes the child's first user message, and
gets back `O`. The child's messages never enter the parent's thread, and the parent's never enter the child's.
The API is `@Experimental` (design [§4.44](https://github.com/llm4s/llm4s/blob/main/docs/design/typed-agent-runtime/4.44-stage-4-slice-d1-typed-agent-and-task.md), [§4.45](https://github.com/llm4s/llm4s/blob/main/docs/design/typed-agent-runtime/4.45-stage-4-slice-d2-inheritance.md)).
## Defining subagents
(not compile-checked)
```scala
import org.llm4s.agent.Agent
import org.llm4s.agent.delegation.{ DelegationConfig, TypedAgent }
import org.llm4s.schema.SchemaOf
final case class ResearchQuestion(topic: String) derives ReadWriter, SchemaOf
final case class Findings(summary: String, sources: Vector[String]) derives ReadWriter, SchemaOf
// a typed subagent: its own tool, task_researcher, whose arguments are ResearchQuestion's schema
val researcher = TypedAgent("researcher", "Researches one topic and cites its sources",
SchemaOf[ResearchQuestion].definition, SchemaOf[Findings].definition)
.withSystemPrompt("You research one topic. Answer with your findings.")
.withTools(researchTools)
// a text subagent: offered through the shared `task` tool, {subagent_type, description}
val general = TypedAgent.text("general-purpose", "Any self-contained task")
for
lead <- Agent.builder("lead", client)
.withSubagents(researcher, general)
.withDelegation(DelegationConfig().withMaxConcurrentChildren(2))
.build()
result <- lead.run(ThreadId("brief-1"), "Write a brief on tidal power")
yield result.answer
```
- **`task`** takes `{subagent_type, description}` and covers every subagent made with `TypedAgent.text` (or whose
input is a `TaskRequest`), so prompts written for the Deep Agents `task` tool carry over.
- **`task_`** is the tool of each other subagent; its arguments are the subagent's input schema.
- A cacheable prompt section lists each subagent's id, tool and description.
A text subagent's report is its answer. A typed subagent answers in `O`'s schema (`AgentBuilder.withOutput`), and its
report is that answer decoded. The child uses the parent's client and completion options unless `withModel` sets
others: a model is not a capability, so it is inherited.
## Inheritance
A child takes nothing of its parent's unless granted. `withInheritance` says what it takes:
(not compile-checked)
```scala
import org.llm4s.agent.delegation.{ Inheritance, TaskBudget, ToolGrant }
import scala.concurrent.duration.*
researcher.withInheritance(
Inheritance()
.withTools(ToolGrant.Only(Set("fetch_page"))) // or ToolGrant.AllExcept(...), ToolGrant.None (default)
.withMiddleware(MiddlewareId("semantic-memory")) // the parent's instance: shared memory, same tenant
.withBudget(TaskBudget().withMaxSteps(20).withTimeout(2.minutes).withMaxTokens(50_000))
.withDelegate(false) // its own subagents only when true, to maxDepth
)
```
- **Tools and middleware** are the parent's instances, run against the child's thread. A tool over a
`StateFileBackend` sees the child's own (empty) files; one over a host, workspace or store backend shares the
parent's, so give parallel children distinct paths. A name the parent does not have is refused at `build()`. The
task tools are never granted as tools: `withDelegate(true)` lets a child offer its own `withSubagents`, and only
while its depth is below `maxDepth`.
- **Identity is never granted.** Tenant, principal, tracing, the runtime and the checkpointer are always the parent's.
- **Budgets** are capped by what the parent has left when the task is called: steps by the parent turn's remaining
steps (`RunContext.remainingSteps`; the parent's `maxSteps` when the budget sets none), tokens by the parent run's
`maxTokens` less its usage, time by the parent's deadline, concurrency by the parent's `maxConcurrency`. A parent
with no steps or tokens left starts no child: the call is refused (`TaskOutcome.NotStarted`) naming only the
subagent. A parent past its deadline fails the call with its own `DeadlineExceeded`. Each limit is the child's
whole life: a child recovered after a crash has its grant less what it used, and its time runs from its first start.
Parallel children each get the same remainder, so together they can overshoot the parent's tokens; the parent's
next model call then stops at `TokenLimitReached`.
`Agent.delegation` is the `DelegationPlan`: for each subagent its tools (own or granted), middleware in order, the
ancestors' permission rules, its model and whether it is inherited, its budget, and whether it may delegate. Log it
with `toString`.
## Bounds
`DelegationConfig` bounds delegation, each limit on its own:
| Setting | Default | What it bounds |
|---|---|---|
| `maxConcurrentChildren` | 4 | children one parent run has live at once; a call over it waits |
| `maxTasksPerTurn` | 16 | task calls per parent turn, by their order in the turn's messages, a handoff's agents' included; a call over it is a tool error |
| `maxDepth` | 1 | how deep delegation nests |
| `maxReportBytes` | 64 KiB | every task tool message, at least 64 bytes: a larger report is a tool error, and an error's reason is cut to fit |
| `childRetention` | 64 | settled child threads each parent thread keeps |
Task calls in one model message run in parallel, as any tool calls do, bounded by the run's `maxConcurrency` and then
by `maxConcurrentChildren`.
## Subagents in a harness
A `HarnessProfile` carries subagents too, as `subagents: SubagentConfig` ([design §4.48](https://github.com/llm4s/llm4s/blob/main/docs/design/typed-agent-runtime/4.48-stage-4-slice-p1-harness-subagents.md)).
`HarnessProfile()` has `SubagentConfig.none`: no subagents, because delegation is never granted silently.
`HarnessProfile.deepAgents` turns on the **general-purpose subagent**, `general-purpose`, which the model calls through
`task` like any `TypedAgent.text` subagent:
- Its own harness is the minimal profile: the file tools over its own `StateFileBackend`. Its files are its own; only
its report comes back.
- Its grant is every tool of the parent except the file tools and `execute`, and no delegation. The preset computes it
from the parent's actual tools when the agent is built, so a `deepAgents` agent with or without a sandbox builds.
`Agent.delegation` shows it.
(not compile-checked)
```scala
import org.llm4s.agent.harness.{GeneralPurposeConfig, HarnessProfile, SubagentConfig}
val profile = HarnessProfile.deepAgents
// specialists only, no general-purpose subagent
val specialists = profile.withSubagents(SubagentConfig.none.withSpecialists(researcher, critic))
// change the general-purpose subagent: its model, description, prompt or grant
val cheaper = profile.withSubagents(
profile.subagents.withGeneralPurpose(GeneralPurposeConfig().withModel(smallClient, CompletionOptions()))
)
// the bounds
val wider = profile.withSubagents(profile.subagents.withDelegation(DelegationConfig().withMaxConcurrentChildren(8)))
```
- The profile's subagents come first, then any added with `AgentBuilder.withSubagents`. `SubagentConfig.delegation`
bounds them all; `withDelegation` beside a harness is refused.
- The `delegation` middleware runs inside the harness's permissions, so with permission rules `task` needs an
`Invoke` rule (`task:general-purpose`).
- The `deepAgents` prompt gains a part on delegating: independent, bounded work, a self-contained description, parallel
tasks in one message, and checking a report before relying on it.
- `llm4s.agent.harness.subagents` in `reference.conf` holds `generalPurpose.enabled` and the `DelegationConfig` bounds,
which `HarnessConfigLoader` reads. Specialists and models are code.
## When a child does not report
A child that is blocked by a guardrail, reaches its step limit, gives a typed answer that fails its schema twice, or
fails, gives no report. With `ChildFailure.Report`, the default, the parent model sees a tool error naming the
subagent, the child thread and the reason, and can compose without it. With `withFailure(ChildFailure.Fail)` the
parent run fails instead, and the parent's `recover` continues the child from its own checkpoint.
A task call runs again on `recover`. It never starts its child twice: a completed child's report is read back from
the child thread, and a child that was still running continues from its checkpoint.
## Approvals and questions through the parent
A child that stops for an approval, a tool's or a middleware's question, or a breakpoint suspends its parent too. The
parent's other calls in the batch finish first. The parent's `AgentStatus.Suspended` lists each child interrupt in
`delegated`, under a composite id `~` (a grandchild's is `p~c~g`), with the
`ChildRef` of the child that waits and its request as the child gives it (`ChildInterrupt`). A delegated interrupt is
never also in `questions`.
Answer them with the usual helpers, by the composite id, and resume the parent:
(not compile-checked)
```scala
agent.run(threadId, "research tides").flatMap { result =>
result.status match
case AgentStatus.Suspended(_, _, _, _, delegated) =>
val answers = delegated.map {
case (id, DelegatedRequest(_, _: ChildInterrupt.Approval)) => result.approve(id)
case (id, DelegatedRequest(_, _: ChildInterrupt.Breakpoint)) => result.proceed(id)
// a tool's or a middleware's question: reply with its answer type (here a tool's `UnitChoice`)
case (id, DelegatedRequest(_, _)) => result.reply(id, UnitChoice("km"))
}
agent.resume(threadId, answers.toMap)
case _ => Right(result)
}
```
`approve`, `reject`, `edit`, `reply` and `proceed` encode the answer the child expects; `Agent.resume` routes it to
the child through its `task` call, which resumes the child and reports to the parent model once the child ends. A
rejection reaches the child's model as at top level. Answer some and the rest stay pending: the call asks again, under
a new parent interrupt id. A composite id that is not pending is refused with `GraphError.InvalidResume`. A parent
that stops after its child resumed never resumes the child twice: its `recover` reads the child first.
## Permissions
Each task call's resource is `(Invoke, task:)`, so a `PermissionMiddleware` on the parent can allow, ask or deny
each subagent:
(not compile-checked)
```scala
new PermissionMiddleware(Seq(PermissionRule.allow(Operation.Invoke, "task:researcher")))
```
Permissions only narrow. A child's call runs only if every ancestor's `PermissionMiddleware` rules allow it, and then
the child's own: a `Deny` at any level denies, and an `Ask` at any level asks once, the approval covering every level.
So rules on the parent are the whole policy for the subagents below it too: allow there what a child's tools touch.
## Inspecting children
`Agent.children(threadId)` lists the child threads a thread's tasks ran on, each with its origin and status. Read a
child's history through the runtime by its id. A child thread cannot be run, resumed or forgotten directly; forgetting
its parent forgets it.
## Following children
Every `task` call sends the parent's live `SubagentEvents.TaskStarted` as its child starts, and commits one durable
`SubagentEvents.TaskEnded` with its result: the child, a status label (`completed`, `blocked:`,
`step_limit_reached`, `token_limit_reached`, `output_invalid`, `failed`; a call waiting on a suspended child commits none until the child settles), the child's whole usage and its
duration. It carries no request, report or reason, so it is the event to count, time and cost subagents by.
A subscription made with `nested = true` also receives every event of each child, as `StreamEvent.Nested(child,
event)`, after the child's `TaskStarted` and before its `TaskEnded`; a grandchild's arrive nested twice. Match them
with `EventHandlers.onChild`, which gives the innermost child:
(not compile-checked)
```scala
agent.stream(threadId, "Research these three topics", nested = true)(
EventHandlers
.on(SubagentEvents.TaskStarted)(s => println(s"-> ${s.child.agent}"))
.onChild(AgentEvents.TextDelta)((child, d) => print(s"[${child.agent}] ${d.text}"))
.on(SubagentEvents.TaskEnded)(e => println(s"<- ${e.child.agent}: ${e.status} in ${e.duration}"))
)
```
A child's text deltas arrive only if the subagent streams (`TypedAgent.withStreaming()`). Nested events are live: they
take the subscription's live slots, so a slow listener loses them as a `LiveGap`, and a child this process cannot
watch - one another process holds, or one that had already finished when a replay reattached to it - shows as one
`Nested(child, LiveGap(0))`. Its durable events are kept: `agent.subscribeChild(ref)` replays and follows one child's
own log, tenant-checked like any read of a thread. The cats-effect, ZIO and Pekko modules offer the merged stream as
`nestedEvents(run, handlers)`.
A subagent traces to its parent's `Tracing`; its `AgentRunEnded` names the parent thread and run
(`parentThreadId`, `parentRunId`), which OpenTelemetry puts on the `invoke_agent ` span and Langfuse in the
run's metadata. The child's run is in its parent's trace, under the `execute_tool task` span of the call that started
it - in Langfuse too, with no trace of its own - and stays there when a later run of the parent recovers or resumes it:
its `RunOrigin.trace` records that call's span (design 4.58).
## Running a subagent alone
`typedAgent.bind(client, runtime)` compiles a definition (pass `options` for non-default `CompletionOptions`; `withModel` overrides both); `BoundAgent.run(input)` runs it once on a new top-level
thread and returns its `TaskOutcome`, for tests and one-off calls. A graph node calls `bound.run(threadId, input)`
instead, with a `threadId` derived from its own replay-stable key (one that identifies the input too): a thread
already there is reattached, never run twice - a settled one returns its stored outcome with no model call, a failed
or crashed one is recovered without repeating completed tool effects, a suspended one returns `Suspended` again, and
one another caller is running is waited on. `bound.resume(outcome.child, answers)` continues
a run that came back `Suspended`, taking the same answers as `Agent.resume`, and `bound.recover(outcome.child)` one that
came back `Refused(TaskRefusal.Failed(_))`; both return a `TaskOutcome` too. Only a standalone run's ref is accepted: a
delegated child is continued through its parent.
## A worked example
[`ParallelDeepResearchExample`](/examples/#parallel-deep-research) puts these together: a lead on
`HarnessProfile.deepAgents` with the general-purpose subagent off, a typed `researcher` granted the lead's
`fetch_page` and a tool-less `critic`, `maxConcurrentChildren = 2`, and permission rules that ask before a fetch from
an unknown site, wherever in the tree it is made. Its acceptance suite, `ParallelDeepResearchAcceptanceSpec`, runs it
on a scripted model through a parallel batch, a crash and `recover`, a child's failure, a child's approval, a cancel,
the merged stream and `forget` ([design §4.49](https://github.com/llm4s/llm4s/blob/main/docs/design/typed-agent-runtime/4.49-stage-4-slice-p2-exit-example.md)).
## Delegation or handoffs
A [handoff](handoffs) moves the conversation to another agent of the same family, which then owns it. Delegation
starts another thread, and the parent keeps control and sees only the report. Use handoffs for conversational routing,
delegation for bounded, independent work, especially work that can run in parallel.
---
Source: https://llm4s.org/guide/agents/multiple-machines
# Running agents on several machines
Run one agent service as several instances that share threads through one PostgreSQL database: any instance can
start, follow, cancel or delete any thread, a run whose machine dies is picked up by another, and one database is the
whole deployment.
---
## The shape of it
llm4s ships the *service kernel*, not an HTTP server: everything that touches the store's guarantees - claims,
fencing, the run queue, the worker, retention - is in `llm4s-agent` (`org.llm4s.agent.service`) and the store modules,
and your service puts its own endpoints, authentication and protocol in front of it. Each instance runs:
- a **store** over the shared database (`PostgresCheckpointer`, `llm4s-agent-checkpoint-postgres`), which holds every
thread's checkpoints and events and the **run queue**;
- a **`RunSubmitter`** that your endpoints call to queue a run, follow its events, cancel it or delete a thread;
- an **`AgentWorker`** that takes queued runs, runs them on your agents, recovers runs whose worker died, runs the
retention sweep and drains on shutdown.
```
HTTP / gRPC / your protocol
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ instance A │ │ instance B │ │ instance C │
│ RunSubmitter │ │ RunSubmitter │ │ RunSubmitter │
│ AgentWorker │ │ AgentWorker │ │ AgentWorker │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
└────────────────┼────────────────┘
PostgreSQL (one schema):
checkpoints, events, claims, run queue
```
There is no Redis, broker or coordinator: instances never talk to each other, only to the database, and the database's
clock decides every claim, lease and retention age, so clock skew between machines changes nothing.
A complete, runnable example is the [example agent service](#the-example-deployment); the stage's exit test runs three
of it against one Postgres and kills one mid-run.
## The store
Every instance opens the same schema. With `llm4s-agent-checkpoint-postgres` on the classpath, configuration alone
chooses it:
```hocon
llm4s.agent.checkpointer {
kind = "postgres" # or LLM4S_CHECKPOINTER
postgres {
url = "jdbc:postgresql://db:5432/app" # or LLM4S_CHECKPOINTER_POSTGRES_URL
user = "agents" # or LLM4S_CHECKPOINTER_POSTGRES_USER
# password: LLM4S_CHECKPOINTER_POSTGRES_PASSWORD, never in a checked-in file
schema = "agents"
callTimeout = 5 seconds
connectionTimeout = 5 seconds
maxPoolSize = 10
}
}
```
(not compile-checked)
```scala
for
store <- CheckpointerConfigLoader.open() // close it when the process stops
runtime <- RuntimeConfigLoader.runtime(store.checkpointer)
queue <- store.runQueue // the queue lives in the same schema
yield (store, runtime, queue)
```
- Instances that start together migrate the schema once, under an advisory lock.
- Every store call returns within `connectionTimeout + callTimeout`. Keep that well inside the claim's
`ttl - renewEvery` (below), so a slow database makes a run lose its claim cleanly rather than hang.
- Each instance holds up to `maxPoolSize` connections plus one listening connection (commits are announced with
`LISTEN`/`NOTIFY`, so a run committed on one machine reaches a subscriber on another at once). Size the database's
`max_connections` for every instance.
The store's details - fencing, notification, the registry - are in
[Durable Checkpointers](durable-checkpointers#the-postgresql-store).
## The queue and the workers
A run is **submitted** to the queue and **executed** by whichever instance's worker takes it. The queue entry, not the
instance that accepted the request, owns the run: it records each attempt's `RunId` before the attempt touches the
thread, so a run is attempted at least once and applied at most once, however many workers try it.
(not compile-checked)
```scala
// every instance
for
agent <- Agent.builder("assistant", client).withRuntime(runtime).build()
settings <- RunQueueConfigLoader.load() // llm4s.agent.queue
workerCfg <- WorkerConfigLoader.load() // llm4s.agent.worker
worker <- AgentWorker(queue, store.checkpointer, Seq(QueuedAgent(agent)), workerCfg, settings, metrics)
submitter = new RunSubmitter(queue, store.checkpointer, settings, metrics)
yield
worker.start()
sys.addShutdownHook { worker.shutdown(); store.close() } // drain, then close
submitter
// an endpoint, on any instance
submitter.submit(ThreadId("t-42"), AgentId("assistant"), QueuedInput.query("Summarise the report"), Some(tenant))
// -> Ticket(entryId, threadId, position, existing)
```
- **Every instance runs the same agents under the same ids.** A worker takes only entries for the agent ids it was
built with; a typed agent goes in as `QueuedAgent.typed(boundAgent)`.
- **Following a run** is `submitter.follow(tenant, entryId, after, limit, wait)`, a page at a time from a thread
sequence number: it waits for the next commit on whichever machine makes it, and resuming from the last number seen
repeats nothing. `submitter.await(tenant, entryId, timeout)` waits for the entry to close.
- **A machine that dies mid-run** leaves its entry's lease and its thread's claim to expire. Another worker then takes
the entry, records a new attempt and recovers the thread from its last checkpoint; a follower sees the recovered
run's events continue the same log, with no gap and no repeat. Recovery takes about the longer of the lease
and claim `ttl`s.
- **Attempts are bounded** by `llm4s.agent.queue.maxAttempts`; an entry that runs out is closed `failed` (a dead
letter), and so is one no worker's graph version can run for `unrunnableAfter`.
- **Submission keys** make a retried request idempotent: the same `submissionKey` from the same tenant returns the
first entry's ticket.
```hocon
llm4s.agent {
runtime.claims { ttl = 30 seconds, renewEvery = 10 seconds } # label: LLM4S_RUNTIME_LABEL
queue {
maxAttempts = 5
ticketLifetime = 24 hours # how long a closed entry stays readable
lease { ttl = 30 seconds, renewEvery = 10 seconds }
}
worker {
concurrency = 4 # runs one worker has at once
pollInterval = 1 second
drain { grace = 30 seconds, exitTimeout = 10 seconds }
label = ${?LLM4S_WORKER_LABEL} # the pod or host name: it names the lease holder in logs
}
}
```
**Shutdown drains.** `worker.shutdown()` stops taking entries, lets running runs reach rest within `drain.grace`,
then cancels the rest and gives their entries back, so another instance recovers them at once instead of waiting for
the lease to lapse. Give the process `grace + exitTimeout` plus a margin to stop (Kubernetes'
`terminationGracePeriodSeconds`).
**Readiness is `worker.health()`**: the worker is running, its loop is alive and the store answers. Keep liveness on
the process alone: restarting instances because the database is down turns an outage into a crash loop.
## Cancel and the busy-thread policy
**Cancel from any machine** with `submitter.cancel(tenant, CancelTarget.Entry(entryId), note)`, or
`CancelTarget.Thread(threadId)` for every open entry of a thread. An entry not yet started closes at once; a running
one is stopped by the worker holding it, on whichever machine, when it next renews the entry's lease (so within
`lease.renewEvery`): the run is interrupted, its thread settled, and the entry closes `cancelled` - or with the run's
own outcome if it finished first. `submitter.deleteThread(runtime, tenant, threadId, timeout)` cancels the thread's
entries, waits for them to close, and then deletes it.
**A message for a busy thread** - one with an open entry or a live run - follows the submission's `policy`:
| `BusyThreadPolicy` | What happens |
|---|---|
| `Reject` (default) | Refused with `GraphError.ThreadBusy`; the caller retries later. |
| `Enqueue` | Queued behind the thread's entries; runs after them, in submission order. |
| `Interrupt` | Queued ahead of them. The running queued run stops at its next superstep boundary: the model call in flight is cut, tool calls already started finish and keep their results, unstarted ones get a synthetic "interrupted" result, so the history stays valid for the provider. Its entry closes `interrupted` and the new message runs next. |
A run started directly (`agent.run`, outside the queue) is never interrupted or adopted by a worker: it is its
owner's, who calls `recover` after a crash. Pending approvals or questions are not a busy thread: answer them with a
`QueuedInput.Resume`.
## Tracing
A run is **one trace across every machine that executes it.** Pass the caller's position as the submission's
`traceContext` (for HTTP, parse the request's W3C `traceparent` header with `TraceContext.parse`); the entry stores
it, and every attempt continues it - the first, and a recovery on another machine after a crash. Each checkpoint
records the trace it belongs to (`Checkpoint.trace`), and span ids are derived from the trace id and the run's
identity, so two machines name the same spans without coordinating.
Spans are emitted by the agent's tracing: build the agent `withTracing(tracing)` with a backend from
`llm4s-observability-otel` (`TRACING_MODE=opentelemetry`) or `llm4s-observability` (Langfuse). Delegated child runs
sit under the tool call that started them. No message content goes into a span attribute.
## Metrics
Give the worker and a `MeteredCheckpointer` the same `MetricsCollector` (`PrometheusMetrics` from
`llm4s-observability-prometheus`, or `MicrometerMetrics` from `llm4s-observability-micrometer`):
(not compile-checked)
```scala
val metrics = new PrometheusMetrics(registry)
val checkpointer = MeteredCheckpointer(store.checkpointer, metrics) // runs, claims, commits, child tasks
val runtime = RuntimeConfigLoader.runtime(checkpointer)
// AgentWorker(..., metrics) and new RunSubmitter(..., metrics): the queue
```
| Area | Metrics (`llm4s.agent.*`; Prometheus writes `_` for `.`) |
|---|---|
| Runs | `runs.started`, `runs.ended` (by outcome), `run.duration` |
| Claims | `claims` (acquired, taken over, busy, failed), `claim.renewals` (renewed, lost, failed) |
| Commits | `commit.duration` |
| Queue | `queue.wait`, `queue.depth`, `queue.oldest.seconds`, `queue.attempts`, `queue.plans`, `queue.deferrals`, `queue.lease.expiries`, `queue.lease.lapses`, `queue.lease.losses`, `queue.closed` (by outcome and reason; dead letters are `failed` / `attempts_exhausted`) |
| Child tasks | `tasks.started`, `tasks.ended`, `task.duration` |
| Retention | `sweep.passes`, `sweep.threads.examined`, `sweep.tombstones.reclaimed`, `sweep.entries.reclaimed` |
| Shutdown | `worker.drained` |
Labels take values from fixed sets (and a graph's id), never thread ids or content. Alert on `queue.oldest.seconds`
(work waiting with no worker taking it), `queue.lease.expiries` (machines dying mid-run) and dead letters.
## Retention
The database stays bounded only if something removes old data. The worker runs the store-wide retention sweep on a
schedule, one instance at a time:
```hocon
llm4s.agent.worker.sweep {
enabled = true
every = 5 minutes
batchSize = 100
tombstoneAge = 7 days # unset: tombstones are kept
retention { maxAge = 30 days, maxCheckpoints = 20, maxEvents = 10000 } # unset keys remove nothing
}
```
- Each pass prunes every thread by `retention`, reclaims the tombstones of threads deleted more than `tombstoneAge`
ago, and reclaims closed queue entries older than `llm4s.agent.queue.ticketLifetime`.
- A thread with an open queue entry is still pruned, but keeps the newest checkpoint each of the entry's attempts
committed and its tombstone, so a run in flight can always be recovered once.
- **A reader of a reclaimed thread** - a client resuming an old event stream - is answered `GraphError.ThreadDeleted`,
before and after reclamation, and also once a new thread reuses the id: it is never shown the new thread's events
as a continuation of the old one. Its correct action is to read from sequence `0` to follow whatever is under the id
now. A ticket whose entry was reclaimed answers `QueueError.TicketExpired`.
Ages are judged by the database's clock against the time it committed each row, never a producer's timestamp.
## The example deployment
`modules/deploy-service` is an example agent service on this kernel (unpublished; copy it rather than depend on it):
submit a run, follow it over Server-Sent Events from a sequence number, cancel it, read and list threads, with
readiness, Prometheus metrics and SIGTERM draining. `deploy/overlays/postgres` runs three replicas of it against one
Postgres, with a PodDisruptionBudget. [`deploy/README.md`](https://github.com/llm4s/llm4s/blob/main/deploy/README.md)
has the endpoints, the environment variables and the `kubectl` steps.
Its exit test, `MultiMachineExitSpec` in `modules/it` (`sbt testIntegration`), starts three processes of it on one
schema and checks, through their HTTP endpoints: a run submitted through one instance, executed by another, streamed
live through a second and replayed from any sequence number through a third; the executing instance killed with
`kill -9` and the run recovered elsewhere with an unbroken stream, in one trace; `Enqueue`, `Interrupt` and a
cross-machine cancel; and a reclaimed tombstone answering `ThreadDeleted`.
## Upgrades
Whether instances can be replaced one at a time depends on what the release changes:
- **Same store schema and checkpoint format: rolling upgrade.** Replace instances one at a time. Each old instance
drains on SIGTERM and gives back what it could not finish; new instances take it. If the agent's graph changed, a
run in flight at the old graph version is passed over by new workers until an old one takes it, and closed `failed`
if none does within `llm4s.agent.worker.unrunnableAfter` (1 hour by default): keep at least one old instance until
its runs drain.
- **A new store schema version or checkpoint format: full stop.** Stop every instance, then start the new release.
The first new instance migrates the schema; an old instance that meets the newer schema refuses to open with
`GraphError.UnsupportedStoreSchema` and writes nothing, so a mixed fleet fails safe rather than corrupting threads.
The release notes and the [migration guide](../../reference/migration) say when a release changes either.
Expand-then-contract migrations, which would let schema changes roll too, are planned with the 1.0 compatibility
baseline.
## Limits
- **One database.** The SPI is open to another store, but PostgreSQL is the only multi-machine store shipped.
- **No HTTP server product.** Endpoints, authentication, Agent Protocol, A2A and AG-UI belong to the service you build
on the kernel; the example service shows one shape and authenticates nothing (its `X-Tenant-Id` header is not auth).
- **Not yet:** a shared Postgres long-term memory store, cron and webhooks, versioned assistants, checkpoint encryption
at rest, background subagents. See the [parity matrix](../../reference/agent-platform-parity) for where each went.
## See also
- [Durable Checkpointers](durable-checkpointers): claims, fencing, recovery, the PostgreSQL store, retention and the
thread registry.
- [Streaming events](streaming): the event log a follower reads.
- [Agent platform parity](../../reference/agent-platform-parity): llm4s against LangGraph Platform and Deep Agents.
- `docs/design/typed-agent-runtime/` 4.50-4.61: the Stage 5 design records.
---
Source: https://llm4s.org/guide/patterns/
# Real-World Application Patterns
Learn production-grade patterns for building scalable, reliable LLM applications with LLM4S. These guides cover architecture decisions, implementation strategies, and best practices based on real-world deployments.
## Table of Contents
1. **[Multi-Agent Orchestration](./multi-agent-orchestration)** - Design patterns for agent-to-agent communication, delegation, and failure handling
2. **[RAG for Enterprise](./rag-enterprise)** - Production strategies for document management, hybrid search, and quality assurance
3. **[Production Monitoring](./production-monitoring)** - Comprehensive observability, alerting, and cost tracking in production
4. **[Scaling Strategies](./scaling-strategies)** - Techniques for handling high throughput, caching, and distributed execution
5. **[Error Recovery](./error-recovery)** - Resilience patterns including retries, circuit breakers, and graceful degradation
6. **[Security Best Practices](./security-best-practices)** - API key management, input validation, and audit logging
---
## Getting Started
Choose a pattern based on your current challenge:
| Challenge | Guide | Key Topics |
|-----------|-------|-----------|
| Multiple agents need to work together | [Multi-Agent Orchestration](./multi-agent-orchestration) | Delegation, handoffs, context passing |
| Building with documents and retrieval | [RAG for Enterprise](./rag-enterprise) | Ingestion, search, quality |
| Need production visibility | [Production Monitoring](./production-monitoring) | Metrics, alerts, dashboards |
| Handling high request volume | [Scaling Strategies](./scaling-strategies) | Caching, batching, distribution |
| Dealing with failures | [Error Recovery](./error-recovery) | Retries, fallbacks, circuit breakers |
| Protecting sensitive data | [Security Best Practices](./security-best-practices) | Secrets, validation, audit |
---
## Architecture Decision Trees
### How Should Agents Communicate?
```
Need agents to coordinate?
├─ Synchronous response required
│ └─> Direct delegation (see handoff pattern)
├─ Fire-and-forget updates
│ └─> Event-based communication
└─> Complex multi-step workflows
└─> Orchestrator agent pattern
```
### Which Search Strategy?
```
Choosing RAG search approach?
├─ Fast, simple queries
│ └─> Vector search only
├─ Mixed structured/unstructured
│ └─> Hybrid search (keyword + vector)
├─ Complex business logic
│ └─> Multi-stage retrieval
└─> Real-time freshness critical
└─> Time-aware search with reranking
```
### Error Recovery Strategy?
```
API call failed, what now?
├─ Transient error (timeout, rate limit)
│ └─> Exponential backoff retry
├─ Model unavailable
│ └─> Fallback to alternate model
├─ Entire provider down
│ └─> Circuit breaker + cached response
└─> Recoverable from degraded mode
└─> Graceful degradation
```
---
## Pattern Comparison Matrix
| Pattern | Complexity | Resilience | Performance | When to Use |
|---------|-----------|------------|-------------|------------|
| **Synchronous delegation** | Low | Medium | High | Simple sequential workflows |
| **Event-based agents** | Medium | High | High | Loosely coupled systems |
| **Vector-only search** | Low | High | Very High | Semantic similarity priority |
| **Hybrid search** | Medium | High | Medium | Balanced search needs |
| **Exponential backoff** | Low | Medium | Good | Transient failures |
| **Circuit breaker** | High | Very High | Good | Preventing cascade failures |
| **Caching layer** | Medium | Medium | Excellent | High-volume, repeated queries |
---
## Common Use Cases
### E-Commerce Product Search Agent
- Pattern: [RAG for Enterprise](./rag-enterprise) + [Scaling Strategies](./scaling-strategies)
- Why: Needs fast search over large catalog with cost control
- Key: Hybrid search + caching + batch processing
### Multi-Department Support System
- Pattern: [Multi-Agent Orchestration](./multi-agent-orchestration) + [Error Recovery](./error-recovery)
- Why: Different agents for different departments with fallback
- Key: Agent delegation + circuit breaker pattern
### Financial Analysis Platform
- Pattern: [Security Best Practices](./security-best-practices) + [Production Monitoring](./production-monitoring)
- Why: Sensitive data + audit requirements + high availability
- Key: API key vaults + detailed logging + alerts
### Document Intelligence System
- Pattern: [RAG for Enterprise](./rag-enterprise) + [Production Monitoring](./production-monitoring)
- Why: Large documents + need quality metrics
- Key: Chunking strategy + grounding scores + observability
---
## Quick Links
- **Run Samples**: `sbt "samples/runMain org.llm4s.samples.agent.MultiStepAgentExample"`
- **View Sample Code**: [modules/samples/src/main/scala/org/llm4s/samples/agent](https://github.com/llm4s/llm4s/tree/main/modules/samples/src/main/scala/org/llm4s/samples/agent)
- **Report Issues**: [GitHub Issues](https://github.com/llm4s/llm4s/issues)
- **Discuss Patterns**: [GitHub Discussions](https://github.com/llm4s/llm4s/discussions)
---
## Feedback & Contributions
These patterns evolve based on community experience. If you've implemented a pattern not covered here, please share it in [Discussions](https://github.com/llm4s/llm4s/discussions) or contribute a guide!
---
**Last Updated:** February 2026
**Status:** Stable - Production Ready
---
Source: https://llm4s.org/guide/patterns/multi-agent-orchestration
# Multi-Agent Orchestration Patterns
> **Note:** Code examples in this guide are illustrative pseudocode showing recommended patterns. For working examples using the actual LLM4S API, see [modules/samples](https://github.com/llm4s/llm4s/tree/main/modules/samples).
Learn how to design systems where multiple agents collaborate, delegate tasks, and handle complex workflows. This guide covers agent communication patterns, handoff mechanisms, and failure recovery strategies.
## Overview
Multi-agent systems allow you to decompose complex tasks into specialized agents. Instead of one agent doing everything, each agent focuses on specific domain expertise:
- **Routing Agent**: Routes requests to appropriate specialists
- **Domain Specialists**: Domain-specific knowledge (finance, legal, technical)
- **Aggregator Agent**: Combines results from multiple agents
- **Quality Assurance Agent**: Validates outputs before returning to user
---
## Pattern 1: Sequential Delegation
### Use Case
Simple workflows where tasks need to execute in order, each building on previous results.
### Implementation
```text
import org.llm4s.agents._
import org.llm4s.Result
object SequentialDelegation {
// Agent for research
val researchAgent = Agent(
name = "Research Agent",
model = openaiClient,
systemPrompt = "You are a research specialist. Find relevant information."
)
// Agent for analysis
val analysisAgent = Agent(
name = "Analysis Agent",
model = openaiClient,
systemPrompt = "You are an analyst. Analyze provided information critically."
)
// Agent for summarization
val summaryAgent = Agent(
name = "Summary Agent",
model = openaiClient,
systemPrompt = "You are a technical writer. Create clear, concise summaries."
)
// Sequential workflow
def processQuery(query: String): Result[String] = for {
// Step 1: Research
researchResult <- researchAgent.run(query)
researchText = researchResult.message
// Step 2: Analyze findings
analysisResult <- analysisAgent.run(
s"Analyze this research: $researchText"
)
analysisText = analysisResult.message
// Step 3: Summarize
summaryResult <- summaryAgent.run(
s"Create a summary of this analysis: $analysisText"
)
} yield summaryResult.message
}
```
### Advantages
- ✅ Simple to understand and implement
- ✅ Clear data flow and dependencies
- ✅ Easy to debug individual steps
### Disadvantages
- ❌ Slower due to sequential execution
- ❌ Single point of failure (one agent fails, whole pipeline fails)
- ❌ No parallelization opportunities
---
## Pattern 2: Parallel Delegation with Aggregation
### Use Case
When multiple independent tasks can run in parallel, then results are combined.
The [multi-agent graph recipe](../../examples/cookbook/multi-agent-graph.html) is a worked, tested version of
this pattern: two specialist agents run in one superstep of a graph, and an editor agent combines their views behind a
static join; cancelling the graph run cancels the agent turns it is waiting on.
### Implementation
```text
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
object ParallelDelegation {
val financialAgent = Agent(
name = "Financial Agent",
model = openaiClient,
systemPrompt = "You are a financial analyst."
)
val technicalAgent = Agent(
name = "Technical Agent",
model = openaiClient,
systemPrompt = "You are a technical expert."
)
val marketAgent = Agent(
name = "Market Agent",
model = openaiClient,
systemPrompt = "You are a market researcher."
)
val aggregatorAgent = Agent(
name = "Aggregator",
model = openaiClient,
systemPrompt = "Synthesize viewpoints from multiple experts."
)
def analyzeCompany(company: String): Result[String] = {
// Run three agents in parallel
val financialFuture = Future(
financialAgent.run(s"Analyze finances of $company")
)
val technicalFuture = Future(
technicalAgent.run(s"Analyze tech stack of $company")
)
val marketFuture = Future(
marketAgent.run(s"Analyze market position of $company")
)
// Wait for all results
val allResults = for {
fin <- financialFuture
tech <- technicalFuture
market <- marketFuture
} yield (fin, tech, market)
// Convert Future to Result and aggregate
allResults.toResult.flatMap { case (fin, tech, market) =>
aggregatorAgent.run(
s"""
|Financial Analysis: ${fin.message}
|Technical Analysis: ${tech.message}
|Market Analysis: ${market.message}
|
|Provide a comprehensive assessment combining all three perspectives.
""".stripMargin
).map(_.message)
}
}
}
```
### Advantages
- ✅ Parallel execution reduces total time
- ✅ Independent agents can scale separately
- ✅ Natural load distribution
### Disadvantages
- ❌ Increased complexity
- ❌ Harder to debug concurrent issues
- ❌ Requires aggregation logic
---
## Pattern 3: Handoff Mechanism
### Use Case
Transferring control from one agent to another when current agent reaches its limits or recognizes need for specialist.
### Implementation
```text
import org.llm4s.agents.handoff._
object HandoffPattern {
// Routing agent that delegates to specialists
val routingAgent = Agent(
name = "Routing Agent",
model = openaiClient,
systemPrompt = """
You are a routing agent. Analyze the request and determine
which specialist agent should handle it:
- Technical questions → technical_specialist
- Account issues → account_specialist
- Billing issues → billing_specialist
- Escalated issues → escalation_team
"""
)
// Specialist agents
val technicalSpecialist = Agent(
name = "Technical Specialist",
model = openaiClient,
systemPrompt = "Solve technical problems with deep expertise."
)
val accountSpecialist = Agent(
name = "Account Specialist",
model = openaiClient,
systemPrompt = "Help with account management and user issues."
)
val billingSpecialist = Agent(
name = "Billing Specialist",
model = openaiClient,
systemPrompt = "Resolve billing and payment issues."
)
// Handoff decision
sealed trait HandoffTarget
case object Technical extends HandoffTarget
case object Account extends HandoffTarget
case object Billing extends HandoffTarget
case object EscalationTeam extends HandoffTarget
def routeAndHandle(userRequest: String): Result[String] = for {
// Step 1: Route to determine target
routing <- routingAgent.run(
s"Request: $userRequest\nDetermine target: technical_specialist, account_specialist, billing_specialist, or escalation_team"
)
// Step 2: Extract routing decision
target = routing.message match {
case s if s.contains("technical_specialist") => Technical
case s if s.contains("account_specialist") => Account
case s if s.contains("billing_specialist") => Billing
case _ => EscalationTeam
}
// Step 3: Hand off to appropriate specialist
response <- target match {
case Technical => technicalSpecialist.run(userRequest)
case Account => accountSpecialist.run(userRequest)
case Billing => billingSpecialist.run(userRequest)
case EscalationTeam =>
Result.failure(
"This issue requires human escalation. Ticket created."
)
}
} yield response.message
}
```
### Advantages
- ✅ Automatic routing based on request type
- ✅ Specialists have focused expertise
- ✅ Scales to many agent types
- ✅ Built on LLM4S handoff support
### Disadvantages
- ❌ Routing errors can send request to wrong agent
- ❌ Context loss during handoff
- ❌ Overhead of routing step
---
## Pattern 4: Hierarchical Agent Teams
### Use Case
Complex systems with multiple management levels (team leads, managers, executives) with clear authority chains.
### Implementation
```text
object HierarchicalTeams {
// Team leads (specialists)
val engineeringLead = Agent(name = "Engineering Lead")
val designLead = Agent(name = "Design Lead")
val productLead = Agent(name = "Product Lead")
// Manager (coordinates across teams)
val projectManager = Agent(
name = "Project Manager",
systemPrompt = """You coordinate across engineering, design, and product teams.
You can delegate to leads and synthesize their feedback."""
)
// Executive (makes final decisions)
val directorAgent = Agent(
name = "Director",
systemPrompt = """You review project status and make strategic decisions.
You receive summaries from the project manager."""
)
def planNewFeature(featureDescription: String): Result[String] = for {
// Level 1: Team leads provide input
engineeringPlan <- engineeringLead.run(
s"Plan engineering approach for: $featureDescription"
)
designPlan <- designLead.run(
s"Plan design approach for: $featureDescription"
)
productPlan <- productLead.run(
s"Plan product approach for: $featureDescription"
)
// Level 2: Manager synthesizes feedback
managerSummary <- projectManager.run(
s"""Coordinate feedback on feature:
|Engineering: ${engineeringPlan.message}
|Design: ${designPlan.message}
|Product: ${productPlan.message}
|
|Create a comprehensive plan addressing all perspectives.""".stripMargin
)
// Level 3: Executive makes decision
executiveDecision <- directorAgent.run(
s"Review this plan and approve/reject: ${managerSummary.message}"
)
} yield executiveDecision.message
}
```
### Advantages
- ✅ Clear organization mirroring real teams
- ✅ Scalable to deep hierarchies
- ✅ Natural delegation and authority
- ✅ Easy to understand and maintain
### Disadvantages
- ❌ More agents = more cost
- ❌ Deeper hierarchies = slower execution
- ❌ Potential for bottlenecks at higher levels
---
## Pattern 5: Context Management in Handoffs
### Use Case
Ensuring context is preserved and relevant when transferring between agents.
### Implementation
```text
import org.llm4s.agents.memory._
object ContextPreservation {
case class HandoffContext(
conversationHistory: List[String],
relevantDocuments: List[String],
userProfile: Option[String],
constraints: List[String]
)
def handoffWithContext(
fromAgent: Agent,
toAgent: Agent,
context: HandoffContext
): Result[String] = {
// Build rich context for receiving agent
val contextPrompt = s"""
|## Conversation Context
|${context.conversationHistory.mkString("\n")}
|
|## Relevant Information
|${context.relevantDocuments.mkString("\n")}
|
|## User Context
|${context.userProfile.getOrElse("No profile available")}
|
|## Constraints
|${context.constraints.mkString("\n")}
""".stripMargin
// Continue conversation with full context
toAgent.run(contextPrompt).map(_.message)
}
// Practical example: Customer support escalation
def supportEscalation(
userMessage: String,
previousInteraction: String
): Result[String] = {
val context = HandoffContext(
conversationHistory = List(previousInteraction, userMessage),
relevantDocuments = List("Knowledge base article on billing"),
userProfile = Some("Premium customer, high lifetime value"),
constraints = List("No refunds without authorization")
)
val level1Agent = Agent(name = "Level 1 Support")
val level2Agent = Agent(name = "Level 2 Support")
// First attempt
level1Agent.run(userMessage).flatMap { result =>
if (result.message.contains("escalate")) {
// Escalate with full context
handoffWithContext(level1Agent, level2Agent, context)
} else {
Result.success(result.message)
}
}
}
}
```
### Advantages
- ✅ Prevents context loss during handoffs
- ✅ Receiving agent has full information
- ✅ Better quality responses in second agent
- ✅ Improved user experience
### Disadvantages
- ❌ Larger context = higher token cost
- ❌ Must carefully select relevant information
- ❌ Requires memory/context management system
---
## Failure Handling in Multi-Agent Systems
### Circuit Breaker for Agents
```text
object AgentCircuitBreaker {
case class CircuitBreakerState(
isOpen: Boolean = false,
failureCount: Int = 0,
successCount: Int = 0,
lastFailureTime: Long = 0
)
def runWithCircuitBreaker(
agent: Agent,
message: String,
state: CircuitBreakerState
): Result[String] = {
// Check if circuit is open
if (state.isOpen) {
val timeSinceFailure = System.currentTimeMillis() - state.lastFailureTime
if (timeSinceFailure < 60000) { // 1 minute window
return Result.failure("Circuit breaker is open, agent temporarily unavailable")
}
}
// Try to execute agent
agent.run(message) match {
case Result.Success(response) =>
Result.success(response.message)
case Result.Failure(error) =>
val newFailureCount = state.failureCount + 1
if (newFailureCount >= 5) { // Open after 5 failures
Result.failure(
s"Agent failed 5 times, circuit breaker opened: $error"
)
} else {
Result.failure(error)
}
}
}
}
```
### Fallback Agents
```text
object FallbackAgents {
def runWithFallback(
primaryAgent: Agent,
fallbackAgent: Agent,
message: String
): Result[String] = {
primaryAgent.run(message) match {
case Result.Success(response) => Result.success(response.message)
case Result.Failure(error) =>
println(s"Primary agent failed: $error. Using fallback...")
fallbackAgent.run(message).map(_.message)
}
}
}
```
---
## Best Practices
### ✅ Do's
- **Clear responsibilities**: Each agent should have a well-defined role
- **Preserve context**: Pass relevant information during handoffs
- **Monitor agent health**: Track success/failure rates
- **Implement timeouts**: Prevent agents from hanging indefinitely
- **Log all transitions**: Track which agent handled what for auditing
- **Test agent chains**: Verify entire workflows work end-to-end
- **Handle failures gracefully**: Always have fallback paths
### ❌ Don'ts
- **Don't create circular dependencies**: Avoid Agent A calling Agent B calling Agent A
- **Don't lose context**: Never handoff without relevant information
- **Don't ignore timeouts**: Always set execution time limits
- **Don't forget to validate**: Verify outputs before passing between agents
- **Don't create too many levels**: Deep hierarchies become hard to manage
- **Don't forget cost tracking**: Monitor token usage across agents
- **Don't hardcode routing**: Make routing decisions learnable
---
## Performance Characteristics
| Pattern | Latency | Cost | Complexity |
|---------|---------|------|-----------|
| Sequential | High | Low | Low |
| Parallel | Low | Medium | Medium |
| Handoff | Medium | Medium | Medium |
| Hierarchical | Very High | High | High |
| Context Preservation | Medium | High | Medium |
---
## Common Pitfalls & Solutions
| Pitfall | Problem | Solution |
|---------|---------|----------|
| Context loss | Agent doesn't know conversation history | Explicit context passing |
| Routing errors | Request goes to wrong agent | Validate routing decision + add fallback |
| Infinite loops | Agent A → B → A → ... | Graph cycle detection, timeout guards |
| Cost explosion | Too many agents or deep chains | Monitor tokens, limit depth |
| Slow execution | Sequential chains take too long | Parallelize when possible |
---
## Example: Complete Support System
```text
object CompleteSupportSystem {
// Routing agent determines issue type
val routingAgent = Agent(name = "Support Router")
// Specialist agents for different issue types
val technicalAgent = Agent(name = "Technical Support")
val billingAgent = Agent(name = "Billing Support")
val accountAgent = Agent(name = "Account Support")
// Escalation agent for complex issues
val escalationAgent = Agent(name = "Escalation Team")
def handleSupportRequest(userMessage: String): Result[String] = {
// Step 1: Route the request
routingAgent.run(userMessage).flatMap { routing =>
val target = routing.message
// Step 2: Handle based on routing decision
(if (target.contains("technical")) {
technicalAgent.run(userMessage)
} else if (target.contains("billing")) {
billingAgent.run(userMessage)
} else if (target.contains("account")) {
accountAgent.run(userMessage)
} else {
escalationAgent.run(userMessage)
}).flatMap { response =>
// Step 3: Check if further escalation needed
if (response.message.contains("escalate")) {
escalationAgent.run(s"${response.message}\n\nPlease escalate to human team.")
} else {
Result.success(response.message)
}
}
}
}
}
```
---
## See Also
- [Error Recovery Patterns](./error-recovery) - Handle agent failures
- [Production Monitoring](./production-monitoring) - Monitor multi-agent systems
- [Scaling Strategies](./scaling-strategies) - Scale agent deployments
- [Agent Framework Documentation](../agents/)
---
**Last Updated:** February 2026
**Status:** Production Ready
---
Source: https://llm4s.org/guide/patterns/rag-enterprise
# RAG for Enterprise
> **Note:** Code examples in this guide are illustrative pseudocode showing recommended patterns. For working examples using the actual LLM4S API, see [modules/samples](https://github.com/llm4s/llm4s/tree/main/modules/samples).
Learn production-ready strategies for building Retrieval-Augmented Generation (RAG) systems at scale. This guide covers document ingestion, hybrid search, cost optimization, and quality assurance.
## Overview
Enterprise RAG requires careful attention to several challenges:
- **Document Volume**: Processing thousands or millions of documents
- **Freshness**: Keeping retrieved information current
- **Quality**: Ensuring search results are relevant and accurate
- **Cost**: Minimizing embedding and LLM API costs
- **Compliance**: Handling sensitive data and audit trails
---
## Document Ingestion at Scale
### Chunking Strategies
#### 1. Fixed-Size Chunks with Overlap
```text
object ChunkingStrategies {
case class Chunk(
text: String,
source: String,
chunkIndex: Int,
metadata: Map[String, String]
)
def fixedSizeChunks(
text: String,
chunkSize: Int = 512,
overlapSize: Int = 50,
source: String
): List[Chunk] = {
val chunks = scala.collection.mutable.ListBuffer[Chunk]()
var startIdx = 0
var chunkIndex = 0
while (startIdx < text.length) {
val endIdx = Math.min(startIdx + chunkSize, text.length)
val chunk = text.substring(startIdx, endIdx)
chunks += Chunk(
text = chunk,
source = source,
chunkIndex = chunkIndex,
metadata = Map(
"startIdx" -> startIdx.toString,
"endIdx" -> endIdx.toString
)
)
startIdx = endIdx - overlapSize
chunkIndex += 1
}
chunks.toList
}
// Example: Fixed 512-token chunks with 50-token overlap
val document = "Your long document text here..."
val chunks = fixedSizeChunks(document, 512, 50, "documents/whitepaper.pdf")
}
```
#### 2. Semantic Chunking
```text
object SemanticChunking {
case class SemanticChunk(
text: String,
topic: String,
source: String,
metadata: Map[String, String]
)
def semanticChunks(
text: String,
source: String,
embeddingClient: EmbeddingClient
): Result[List[SemanticChunk]] = {
// Split by paragraphs first
val paragraphs = text.split("\n\n").toList
val chunks = scala.collection.mutable.ListBuffer[SemanticChunk]()
var currentChunk = ""
var currentTopic = ""
paragraphs.foreach { paragraph =>
currentChunk += "\n" + paragraph
// Check if chunk is getting too large
if (currentChunk.length > 1000) {
// Create chunk
chunks += SemanticChunk(
text = currentChunk,
topic = currentTopic,
source = source,
metadata = Map("type" -> "semantic")
)
currentChunk = ""
}
}
// Add remaining chunk
if (currentChunk.nonEmpty) {
chunks += SemanticChunk(
text = currentChunk,
topic = currentTopic,
source = source,
metadata = Map("type" -> "semantic")
)
}
Result.success(chunks.toList)
}
}
```
#### 3. Document-Aware Chunking
```text
object DocumentAwareChunking {
sealed trait DocumentType
case object ResearchPaper extends DocumentType
case object LegalDocument extends DocumentType
case object ProductSpec extends DocumentType
case object WebPage extends DocumentType
def chunkByType(
text: String,
docType: DocumentType,
source: String
): List[String] = docType match {
// Research papers: chunk by sections
case ResearchPaper =>
text.split("(?=^## |^# )").toList
// Legal documents: chunk by clauses
case LegalDocument =>
text.split("(?=\\d+\\.|[A-Z][A-Z ]+\\s*\\n)").toList
// Product specs: chunk by features
case ProductSpec =>
text.split("(?=### |^Features|^Specifications)").toList
// Web pages: chunk by sections
case WebPage =>
text.split("(?=
accFuture.flatMap { processedCount =>
Future {
// Embed batch
val embeddings = batch.map { doc =>
job.embeddingClient.embed(doc)
}
// Store in vector database
embeddings.foreach { embedding =>
job.vectorStore.store(embedding)
}
processedCount + batch.length
}.flatMap(identity)
}
}
results.map(count => Result.success(count))
}
// With progress tracking
case class IngestionProgress(
processed: Int,
total: Int,
rate: Double // docs/second
) {
def percentage: Double = (processed.toDouble / total) * 100
}
def ingestWithProgress(
job: IngestionJob,
onProgress: IngestionProgress => Unit
)(implicit ec: ExecutionContext): Future[Result[Int]] = {
val batches = job.documents.grouped(job.batchSize).toList
val total = job.documents.length
val startTime = System.currentTimeMillis()
val results = batches.zipWithIndex.foldLeft(Future.successful(0)) {
case (accFuture, (batch, batchIdx)) =>
accFuture.flatMap { processedCount =>
Future {
// Process batch...
val newProcessedCount = processedCount + batch.length
// Report progress
val elapsed = (System.currentTimeMillis() - startTime) / 1000.0
val rate = newProcessedCount / elapsed
onProgress(IngestionProgress(newProcessedCount, total, rate))
newProcessedCount
}
}
}
results.map(count => Result.success(count))
}
}
```
---
## Hybrid Search Implementation
### Vector + Keyword Search
```text
object HybridSearch {
case class SearchResult(
content: String,
source: String,
vectorScore: Double,
keywordScore: Double,
hybridScore: Double // weighted combination
)
def hybridSearch(
query: String,
vectorStore: VectorStore,
keywordIndex: KeywordIndex,
alpha: Double = 0.7 // weight for vector score
): Result[List[SearchResult]] = for {
// Vector search
vectorResults <- vectorStore.search(query, topK = 10)
// Keyword search
keywordResults <- keywordIndex.search(query, topK = 10)
// Normalize scores
normalizedVector = vectorResults.map(r =>
r.copy(score = r.score / vectorResults.map(_.score).max)
)
normalizedKeyword = keywordResults.map(r =>
r.copy(score = r.score / keywordResults.map(_.score).max)
)
// Combine results
combined = combineResults(
normalizedVector,
normalizedKeyword,
alpha
)
} yield combined
private def combineResults(
vectorResults: List[SearchResult],
keywordResults: List[SearchResult],
alpha: Double
): List[SearchResult] = {
val resultMap = scala.collection.mutable.Map[String, SearchResult]()
// Add vector results
vectorResults.foreach { r =>
resultMap(r.source) = r.copy(
hybridScore = alpha * r.vectorScore
)
}
// Add keyword results
keywordResults.foreach { r =>
if (resultMap.contains(r.source)) {
val existing = resultMap(r.source)
resultMap(r.source) = existing.copy(
hybridScore = existing.hybridScore + (1 - alpha) * r.keywordScore
)
} else {
resultMap(r.source) = r.copy(
hybridScore = (1 - alpha) * r.keywordScore
)
}
}
resultMap.values.toList.sortBy(_.hybridScore).reverse
}
}
```
### Multi-Stage Retrieval
```text
object MultiStageRetrieval {
def multiStageRetrieval(
query: String,
vectorStore: VectorStore,
reranker: RerankerModel,
grounding: GroundingSystem
): Result[List[RankedResult]] = for {
// Stage 1: Broad retrieval
candidates <- vectorStore.search(query, topK = 100)
// Stage 2: Re-ranking with smaller model
reranked <- Future(
reranker.rank(query, candidates)
).toResult
topK = reranked.take(10)
// Stage 3: Grounding check
grounded <- Future(
topK.filter(r => grounding.isWellGrounded(r.content, query))
).toResult
} yield grounded
case class RankedResult(
content: String,
source: String,
score: Double,
rerankerScore: Option[Double] = None,
isGrounded: Boolean = false
)
}
```
---
## Cost Optimization
### Embedding Caching
```text
object EmbeddingCache {
import scala.collection.mutable
class SmartEmbeddingCache(embeddingClient: EmbeddingClient) {
private val cache = mutable.Map[String, Vector[Double]]()
private var hitRate = 0.0
private var totalQueries = 0
def embed(text: String): Vector[Double] = {
totalQueries += 1
cache.get(text) match {
case Some(embedding) =>
hitRate = (hitRate * (totalQueries - 1) + 1) / totalQueries
embedding
case None =>
hitRate = (hitRate * (totalQueries - 1)) / totalQueries
val embedding = embeddingClient.embed(text)
cache(text) = embedding
embedding
}
}
def stats(): CacheStats = CacheStats(
hitRate = hitRate,
cachedItems = cache.size,
estimatedSavings = cache.size * 0.0002 // estimate ~$0.0002 per embedding
)
}
case class CacheStats(
hitRate: Double,
cachedItems: Int,
estimatedSavings: Double
)
}
```
### Token Cost Tracking
```text
object CostTracking {
case class CostBreakdown(
embeddings: Double,
retrievals: Double,
llmCalls: Double,
total: Double
)
case class PerQueryCost(
query: String,
embeddingCost: Double,
retrievalCost: Double,
llmCost: Double
) {
def total: Double = embeddingCost + retrievalCost + llmCost
}
class RAGCostTracker {
private val queryLogs = scala.collection.mutable.ListBuffer[PerQueryCost]()
def logQuery(
query: String,
embeddingTokens: Int,
retrievalTime: Long,
llmInputTokens: Int,
llmOutputTokens: Int
): Unit = {
val embeddingCost = embeddingTokens * 0.00002 // OpenAI pricing
val retrievalCost = (retrievalTime / 1000.0) * 0.001 // $0.001 per second
val llmCost = (llmInputTokens * 0.000003 + llmOutputTokens * 0.000006) // GPT-4 pricing
queryLogs += PerQueryCost(
query = query,
embeddingCost = embeddingCost,
retrievalCost = retrievalCost,
llmCost = llmCost
)
}
def getCostSummary(): CostBreakdown = {
val totalEmbeddings = queryLogs.map(_.embeddingCost).sum
val totalRetrievals = queryLogs.map(_.retrievalCost).sum
val totalLLM = queryLogs.map(_.llmCost).sum
CostBreakdown(
embeddings = totalEmbeddings,
retrievals = totalRetrievals,
llmCalls = totalLLM,
total = totalEmbeddings + totalRetrievals + totalLLM
)
}
def averageCostPerQuery(): Double = {
if (queryLogs.isEmpty) 0.0
else getCostSummary().total / queryLogs.length
}
}
}
```
### Query Optimization
```text
object QueryOptimization {
def optimizeQuery(
userQuery: String,
optimizationAgent: Agent
): Result[String] = {
// Use LLM to reformulate query for better retrieval
optimizationAgent.run(
s"""Reformulate this search query to maximize relevance in a RAG system:
|Original: "$userQuery"
|
|Requirements:
|1. Add semantic keywords
|2. Specify document types if relevant
|3. Include related concepts
|4. Keep under 20 words
|
|Optimized query:""".stripMargin
).map(_.message)
}
def multiQueryRetrieval(
userQuery: String,
optimizationAgent: Agent,
vectorStore: VectorStore
): Result[List[SearchResult]] = {
// Generate multiple search variants
optimizationAgent.run(
s"""Generate 3 alternative search queries for: "$userQuery"
|Format: Query 1: ..., Query 2: ..., Query 3: ...""".stripMargin
).flatMap { result =>
val queries = result.message.split(",").map(_.trim).toList
// Search with each variant
val results = queries.flatMap { q =>
vectorStore.search(q, topK = 5)
}
Result.success(results)
}
}
}
```
---
## Quality Assurance
### Grounding System
```text
object Grounding {
case class GroundingScore(
score: Double, // 0.0 to 1.0
reasoning: String,
isHallucination: Boolean
)
class GroundingChecker(
groundingAgent: Agent,
sourceDocuments: List[String]
) {
def check(
claim: String,
retrievedContext: String
): Result[GroundingScore] = {
groundingAgent.run(
s"""Evaluate if this claim is grounded in the provided context:
|
|Claim: "$claim"
|
|Context:
|$retrievedContext
|
|Score (0.0-1.0): [provide number]
|Is hallucination: [yes/no]
|Reasoning: [brief explanation]""".stripMargin
).map { response =>
// Parse response to extract score
val score = extractScore(response.message)
val isHallucination = response.message.contains("yes")
GroundingScore(
score = score,
reasoning = response.message,
isHallucination = isHallucination
)
}
}
private def extractScore(response: String): Double = {
val pattern = """Score.*?(\d+\.\d+)""".r
pattern.findFirstMatchIn(response)
.map(_.group(1).toDouble)
.getOrElse(0.0)
}
}
}
```
### Relevance Metrics
```text
object RelevanceMetrics {
case class RelevanceMetrics(
mrr: Double, // Mean Reciprocal Rank
ndcg: Double, // Normalized Discounted Cumulative Gain
precision: Double, // Precision@10
recall: Double // Recall
)
def evaluateRetrieval(
query: String,
retrievedResults: List[SearchResult],
relevantDocuments: List[String]
): RelevanceMetrics = {
val mrr = calculateMRR(retrievedResults, relevantDocuments)
val ndcg = calculateNDCG(retrievedResults, relevantDocuments)
val precision = calculatePrecision(retrievedResults, relevantDocuments)
val recall = calculateRecall(retrievedResults, relevantDocuments)
RelevanceMetrics(
mrr = mrr,
ndcg = ndcg,
precision = precision,
recall = recall
)
}
private def calculateMRR(
retrieved: List[SearchResult],
relevant: List[String]
): Double = {
retrieved.zipWithIndex.collectFirst {
case (result, idx) if relevant.contains(result.source) => 1.0 / (idx + 1)
}.getOrElse(0.0)
}
private def calculateNDCG(
retrieved: List[SearchResult],
relevant: List[String]
): Double = {
val dcg = retrieved.zipWithIndex.map { case (result, idx) =>
val relevance = if (relevant.contains(result.source)) 1.0 else 0.0
relevance / Math.log(idx + 2) // log base 2
}.sum
val idcg = Math.min(relevant.size, retrieved.size)
if (idcg == 0) 0.0 else dcg / idcg
}
private def calculatePrecision(
retrieved: List[SearchResult],
relevant: List[String]
): Double = {
if (retrieved.isEmpty) 0.0
else {
val correct = retrieved.count(r => relevant.contains(r.source))
correct.toDouble / Math.min(10, retrieved.size) // Precision@10
}
}
private def calculateRecall(
retrieved: List[SearchResult],
relevant: List[String]
): Double = {
if (relevant.isEmpty) 0.0
else {
val correct = retrieved.count(r => relevant.contains(r.source))
correct.toDouble / relevant.size
}
}
}
```
---
## Production RAG Pipeline
```text
object ProductionRAG {
case class RAGConfig(
embeddingBatchSize: Int = 50,
vectorSearchTopK: Int = 10,
hybridAlpha: Double = 0.7,
groundingThreshold: Double = 0.7,
costTrackingEnabled: Boolean = true
)
class ProductionRAGPipeline(
config: RAGConfig,
embeddingClient: EmbeddingClient,
vectorStore: VectorStore,
groundingChecker: GroundingChecker,
costTracker: CostTracker
) {
def answerQuestion(
question: String,
agent: Agent
): Result[String] = for {
// Step 1: Search for relevant documents
startTime <- Result.success(System.currentTimeMillis())
searchResults <- vectorStore.search(question, config.vectorSearchTopK)
context = searchResults.map(_.content).mkString("\n\n")
// Step 2: Generate answer with context
result <- agent.run(
s"""Answer this question using the provided context.
|If the answer is not in the context, say "I don't have enough information".
|
|Question: $question
|
|Context:
|$context""".stripMargin
)
answer <- result.answer.toRight(ValidationError("rag", s"the agent ended with ${result.status}"))
// Step 3: Check grounding
groundingScore <- groundingChecker.check(answer, context)
finalAnswer <- if (groundingScore.score >= config.groundingThreshold) {
Result.success(answer)
} else {
Result.failure(
s"Answer not sufficiently grounded (score: ${groundingScore.score}). " +
s"Retrieved context may be insufficient."
)
}
// Step 4: Track costs
elapsed <- Result.success(System.currentTimeMillis() - startTime)
_ <- if (config.costTrackingEnabled) {
costTracker.logQuery(
question = question,
embeddingTokens = question.split(" ").length * 2,
retrievalTime = elapsed,
llmInputTokens = (question + context).split(" ").length,
llmOutputTokens = finalAnswer.split(" ").length
)
Result.success(())
} else {
Result.success(())
}
} yield finalAnswer
}
}
```
---
## Best Practices
### ✅ Do's
- **Version your embeddings**: Track which embedding model was used
- **Monitor relevance**: Track NDCG and MRR over time
- **Batch process documents**: Reduce API costs with batch operations
- **Use hybrid search**: Combine vector and keyword for better results
- **Track groundedness**: Ensure retrieved context supports answers
- **Set TTLs on cache**: Refresh stale embeddings periodically
- **A/B test retrieval**: Compare different chunking strategies
### ❌ Don'ts
- **Don't ignore chunk quality**: Poor chunks hurt relevance
- **Don't forget deduplication**: Avoid indexing same document twice
- **Don't hardcode thresholds**: Make grounding scores configurable
- **Don't skip validation**: Always verify retrieval quality
- **Don't neglect cost tracking**: Monitor and optimize spending
- **Don't lose metadata**: Track source, date, version in embeddings
- **Don't assume embedding stability**: Embeddings can change between models
---
## See Also
- [Production Monitoring](./production-monitoring) - Monitor RAG system health
- [Error Recovery](./error-recovery) - Handle retrieval failures
- [Scaling Strategies](./scaling-strategies) - Scale RAG to large datasets
---
**Last Updated:** February 2026
**Status:** Production Ready
---
Source: https://llm4s.org/guide/patterns/production-monitoring
# Production Monitoring
> **Note:** Code examples in this guide are illustrative pseudocode showing recommended patterns. For working examples using the actual LLM4S API, see [modules/samples](https://github.com/llm4s/llm4s/tree/main/modules/samples).
Comprehensive strategies for observing, debugging, and optimizing LLM applications in production. This guide covers metrics collection, alerting, cost tracking, and debugging techniques.
## Overview
Production monitoring for LLM applications requires tracking multiple dimensions:
- **Performance**: Latency, throughput, error rates
- **Quality**: Output quality, grounding scores, user satisfaction
- **Cost**: Token usage, API costs, ROI by feature
- **Reliability**: Uptime, error recovery, circuit breaker status
---
## Key Metrics to Track
### Performance Metrics
```text
object PerformanceMetrics {
case class PerformanceSnapshot(
p50Latency: Long, // milliseconds
p95Latency: Long,
p99Latency: Long,
avgLatency: Double,
throughput: Double, // requests/second
errorRate: Double // percentage
)
class LatencyTracker {
private val latencies = scala.collection.mutable.ListBuffer[Long]()
def recordLatency(ms: Long): Unit = {
latencies += ms
}
def snapshot(): PerformanceSnapshot = {
val sorted = latencies.sorted
val p50 = sorted((sorted.length * 0.5).toInt)
val p95 = sorted((sorted.length * 0.95).toInt)
val p99 = sorted((sorted.length * 0.99).toInt)
val avg = sorted.sum.toDouble / sorted.length
PerformanceSnapshot(
p50Latency = p50,
p95Latency = p95,
p99Latency = p99,
avgLatency = avg,
throughput = calculateThroughput(),
errorRate = calculateErrorRate()
)
}
private def calculateThroughput(): Double = {
// requests per second
latencies.length.toDouble / (latencies.sum.toDouble / 1000.0)
}
private def calculateErrorRate(): Double = {
// Calculate percentage of requests that failed
0.0 // implementation
}
}
}
```
### Quality Metrics
```text
object QualityMetrics {
case class QualityScore(
groundingScore: Double, // 0.0-1.0
relevanceScore: Double, // 0.0-1.0
userSatisfaction: Double, // 1-5 stars
hallucination: Boolean,
timeToFirstToken: Long // ms
) {
def overallQuality: Double =
(groundingScore + relevanceScore + userSatisfaction / 5.0) / 3.0
}
class QualityMonitor(groundingChecker: GroundingChecker) {
private val scores = scala.collection.mutable.ListBuffer[QualityScore]()
def recordScore(score: QualityScore): Unit = {
scores += score
}
def getAverageQuality(): Double = {
if (scores.isEmpty) 0.0
else scores.map(_.overallQuality).sum / scores.length
}
def getQualityByModel(model: String): Double = {
// Track quality per model to detect regressions
0.0
}
}
}
```
### Cost Metrics
```text
object CostMetrics {
case class CostSnapshot(
totalTokensUsed: Long,
totalCost: Double,
costPerRequest: Double,
costByProvider: Map[String, Double],
costTrend: List[Double] // last 7 days
)
class CostMonitor {
private val dailyCosts = scala.collection.mutable.Map[String, Double]()
private val providerCosts = scala.collection.mutable.Map[String, Double]()
def recordAPICall(
provider: String,
inputTokens: Int,
outputTokens: Int,
cost: Double
): Unit = {
val today = java.time.LocalDate.now().toString
dailyCosts(today) = dailyCosts.getOrElse(today, 0.0) + cost
providerCosts(provider) = providerCosts.getOrElse(provider, 0.0) + cost
}
def getDailyBudgetStatus(dailyBudget: Double): String = {
val today = java.time.LocalDate.now().toString
val spent = dailyCosts.getOrElse(today, 0.0)
val percentUsed = (spent / dailyBudget) * 100
s"Daily budget: $$${spent}/ $$${dailyBudget} (${percentUsed.toInt}%)"
}
def predictMonthlySpend(): Double = {
val currentDay = java.time.LocalDate.now().getDayOfMonth
val spentToDate = dailyCosts.values.sum
(spentToDate / currentDay) * 30
}
}
}
```
---
## Alerting Strategy
### Alert Rules
```text
object AlertRules {
sealed trait AlertSeverity
case object Critical extends AlertSeverity
case object Warning extends AlertSeverity
case object Info extends AlertSeverity
case class Alert(
severity: AlertSeverity,
title: String,
message: String,
affectedComponent: String,
timestamp: Long = System.currentTimeMillis()
)
class AlertManager {
private val alertHandlers = scala.collection.mutable.ListBuffer[
Alert => Unit
]()
def registerHandler(handler: Alert => Unit): Unit = {
alertHandlers += handler
}
def raiseAlert(alert: Alert): Unit = {
alertHandlers.foreach(_(alert))
}
}
// Alert rules
def setupAlerts(alertManager: AlertManager): Unit = {
// High error rate
if (errorRate > 0.05) { // > 5%
alertManager.raiseAlert(Alert(
severity = Critical,
title = "High error rate detected",
message = s"Error rate is ${(errorRate * 100).toInt}%, threshold is 5%",
affectedComponent = "LLM API"
))
}
// High latency
if (p99Latency > 30000) { // > 30 seconds
alertManager.raiseAlert(Alert(
severity = Warning,
title = "High p99 latency",
message = s"P99 latency is ${p99Latency}ms",
affectedComponent = "API Gateway"
))
}
// Daily spend exceeded
if (dailySpend > dailyBudget * 1.1) { // > 110% of budget
alertManager.raiseAlert(Alert(
severity = Warning,
title = "Daily budget exceeded",
message = s"Spent $$${dailySpend}, budget is $$${dailyBudget}",
affectedComponent = "Cost Tracking"
))
}
// Low grounding score
if (averageGroundingScore < 0.7) {
alertManager.raiseAlert(Alert(
severity = Warning,
title = "Low grounding score",
message = s"Average grounding score is ${averageGroundingScore}, target is 0.8+",
affectedComponent = "RAG Quality"
))
}
}
}
```
---
## Debugging in Production
### Request Tracing
```text
object RequestTracing {
case class TraceSpan(
spanId: String,
traceId: String,
operationName: String,
startTime: Long,
endTime: Long,
attributes: Map[String, String],
events: List[TraceEvent],
status: String
) {
def duration: Long = endTime - startTime
}
case class TraceEvent(
timestamp: Long,
name: String,
attributes: Map[String, String]
)
class RequestTracer {
private val traces = scala.collection.mutable.Map[String, TraceSpan]()
def startSpan(
traceId: String,
operationName: String
): String = {
val spanId = java.util.UUID.randomUUID().toString
traces(spanId) = TraceSpan(
spanId = spanId,
traceId = traceId,
operationName = operationName,
startTime = System.currentTimeMillis(),
endTime = 0,
attributes = Map(),
events = List(),
status = "UNSET"
)
spanId
}
def addEvent(
spanId: String,
eventName: String,
attributes: Map[String, String]
): Unit = {
traces.get(spanId).foreach { span =>
val event = TraceEvent(System.currentTimeMillis(), eventName, attributes)
traces(spanId) = span.copy(
events = span.events :+ event
)
}
}
def endSpan(
spanId: String,
status: String = "OK"
): Unit = {
traces.get(spanId).foreach { span =>
traces(spanId) = span.copy(
endTime = System.currentTimeMillis(),
status = status
)
}
}
def getTrace(traceId: String): List[TraceSpan] = {
traces.values.filter(_.traceId == traceId).toList
}
}
}
```
### Debug Logging
```text
object DebugLogging {
import org.slf4j.LoggerFactory
val logger = LoggerFactory.getLogger("LLM4S")
def debugRAGQuery(
query: String,
results: List[SearchResult],
retrievalTime: Long
): Unit = {
logger.debug(s"RAG Query: $query")
logger.debug(s"Retrieved ${results.length} documents in ${retrievalTime}ms")
results.take(3).foreach { result =>
logger.debug(s" - ${result.source} (score: ${result.score})")
}
}
def debugAgentExecution(
agentName: String,
input: String,
output: String,
tokens: (Int, Int), // (input, output)
latency: Long
): Unit = {
logger.debug(s"Agent: $agentName")
logger.debug(s"Input tokens: ${tokens._1}, Output tokens: ${tokens._2}")
logger.debug(s"Latency: ${latency}ms")
logger.debug(s"Output length: ${output.length} chars")
}
}
```
---
## Cost Tracking in Detail
### Per-Feature Cost Breakdown
```text
object FeatureCostTracking {
case class FeatureCost(
featureName: String,
usage: Long, // number of times used
totalCost: Double,
costPerUse: Double,
roi: Double // revenue generated / cost
)
class FeatureCostMonitor {
private val features = scala.collection.mutable.Map[String, FeatureCost]()
def recordFeatureUsage(
featureName: String,
tokensUsed: Int,
costIncurred: Double,
revenueGenerated: Double = 0.0
): Unit = {
features.get(featureName) match {
case Some(existing) =>
features(featureName) = existing.copy(
usage = existing.usage + 1,
totalCost = existing.totalCost + costIncurred,
costPerUse = (existing.totalCost + costIncurred) / (existing.usage + 1),
roi = revenueGenerated / costIncurred
)
case None =>
features(featureName) = FeatureCost(
featureName = featureName,
usage = 1,
totalCost = costIncurred,
costPerUse = costIncurred,
roi = revenueGenerated / costIncurred
)
}
}
def getLeastProfitableFeatures(topN: Int = 5): List[FeatureCost] = {
features.values.toList
.sortBy(_.roi)
.take(topN)
}
def getMostCostlyFeatures(topN: Int = 5): List[FeatureCost] = {
features.values.toList
.sortBy(_.totalCost)
.reverse
.take(topN)
}
}
}
```
---
## Production Dashboard Queries
### BigQuery Examples
```sql
-- Daily API costs by provider
SELECT
DATE(timestamp) as date,
provider,
COUNT(*) as requests,
SUM(input_tokens + output_tokens) as total_tokens,
SUM(cost) as daily_cost
FROM llm_calls
GROUP BY date, provider
ORDER BY date DESC, daily_cost DESC;
-- Latency percentiles
SELECT
PERCENTILE_CONT(latency_ms, 0.50) as p50,
PERCENTILE_CONT(latency_ms, 0.95) as p95,
PERCENTILE_CONT(latency_ms, 0.99) as p99,
AVG(latency_ms) as avg
FROM llm_calls
WHERE timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR);
-- Agent success rates
SELECT
agent_name,
COUNT(*) as executions,
COUNTIF(status = 'SUCCESS') as successes,
ROUND(COUNTIF(status = 'SUCCESS') / COUNT(*) * 100, 2) as success_rate
FROM agent_executions
GROUP BY agent_name;
-- Grounding score distribution
SELECT
grounding_score,
COUNT(*) as count,
ROUND(COUNT(*) / SUM(COUNT(*)) OVER () * 100, 2) as percentage
FROM rag_outputs
GROUP BY grounding_score
ORDER BY grounding_score DESC;
```
---
## Best Practices
### ✅ Do's
- **Track at request level**: Include latency, tokens, cost in every call
- **Set alert thresholds**: Define what "bad" looks like
- **Monitor trends**: Track metrics over days/weeks, not just seconds
- **Tag your spans**: Add metadata for easy filtering
- **Sample logs intelligently**: Don't log every request, but log errors
- **Archive metrics**: Keep historical data for trend analysis
- **Test your alerts**: Verify alerts actually fire
- **Correlate metrics**: Link errors to cost spikes, latency to quality
### ❌ Don'ts
- **Don't ignore errors**: Every error is a signal
- **Don't forget context**: Without tags, logs are useless
- **Don't alert on everything**: Alert fatigue leads to ignored alerts
- **Don't assume correlation**: High cost doesn't always mean issues
- **Don't hardcode thresholds**: Make them configurable
- **Don't lose audit trails**: Keep complete history for compliance
- **Don't mix concerns**: Separate performance from cost from quality metrics
---
## See Also
- [Cost Tracking & Optimization](../issues/#11) - Detailed cost analysis
- [Error Recovery](./error-recovery) - Handle failures
- [Security Best Practices](./security-best-practices) - Audit logging
---
**Last Updated:** February 2026
**Status:** Production Ready
---
Source: https://llm4s.org/guide/patterns/scaling-strategies
# Scaling Strategies
> **Note:** Code examples in this guide are illustrative pseudocode showing recommended patterns. For working examples using the actual LLM4S API, see [modules/samples](https://github.com/llm4s/llm4s/tree/main/modules/samples).
Production patterns for handling high throughput, reducing latency, and distributing load. This guide covers caching, rate limiting, batch processing, and distributed execution.
## Caching Strategies
### Request-Level Caching
```text
object RequestCaching {
import scala.collection.mutable
import scala.concurrent.duration._
case class CachedResponse(
result: String,
timestamp: Long,
ttl: Duration
) {
def isExpired: Boolean =
System.currentTimeMillis() - timestamp > ttl.toMillis
}
class RequestCache(maxSize: Int = 1000) {
private val cache = mutable.LinkedHashMap[String, CachedResponse]()
def get(key: String): Option[String] = {
cache.get(key).flatMap { cached =>
if (cached.isExpired) {
cache.remove(key)
None
} else {
Some(cached.result)
}
}
}
def put(key: String, value: String, ttl: Duration): Unit = {
if (cache.size >= maxSize) {
cache.remove(cache.head._1) // Remove oldest
}
cache(key) = CachedResponse(value, System.currentTimeMillis(), ttl)
}
def stats(): CacheStats = CacheStats(
size = cache.size,
hits = hitCount,
misses = missCount,
hitRate = hitCount.toDouble / (hitCount + missCount)
)
}
case class CacheStats(size: Int, hits: Long, misses: Long, hitRate: Double)
}
```
### Distributed Cache (Redis)
```text
object DistributedCache {
import redis.clients.jedis.Jedis
class RedisCache(host: String = "localhost", port: Int = 6379) {
private val jedis = new Jedis(host, port)
def get(key: String): Option[String] = {
Option(jedis.get(key))
}
def put(key: String, value: String, ttlSeconds: Int): Unit = {
jedis.setex(key, ttlSeconds, value)
}
def invalidate(keyPattern: String): Long = {
val keys = jedis.keys(keyPattern)
if (keys.isEmpty) 0L
else jedis.del(keys.toArray: _*)
}
}
// Usage with LLM4S
def answerWithCaching(
query: String,
agent: Agent,
cache: RedisCache
): Result[String] = {
val cacheKey = s"query:${query.hashCode}"
cache.get(cacheKey) match {
case Some(cachedAnswer) =>
println(s"Cache hit for: $query")
Result.success(cachedAnswer)
case None =>
agent.run(query).map { response =>
cache.put(cacheKey, response.message, ttlSeconds = 3600)
response.message
}
}
}
}
```
---
## Rate Limiting
### Token Bucket Algorithm
```text
object RateLimiting {
import scala.concurrent.duration._
class TokenBucket(
capacity: Int,
refillRate: Int, // tokens per second
refillInterval: Duration = 1.second
) {
private var tokens = capacity.toDouble
private var lastRefillTime = System.currentTimeMillis()
def tryConsume(amount: Int = 1): Boolean = {
refill()
if (tokens >= amount) {
tokens -= amount
true
} else {
false
}
}
def availableTokens: Int = tokens.toInt
private def refill(): Unit = {
val now = System.currentTimeMillis()
val timeSinceLastRefill = now - lastRefillTime
val tokensToAdd = (timeSinceLastRefill / refillInterval.toMillis) * refillRate
tokens = Math.min(capacity, tokens + tokensToAdd)
lastRefillTime = now
}
}
// Usage
val rateLimiter = new TokenBucket(
capacity = 1000,
refillRate = 100 // 100 tokens per second = 100 requests/sec
)
def limitedLLMCall(
query: String,
agent: Agent
): Result[String] = {
if (rateLimiter.tryConsume()) {
agent.run(query)
} else {
Result.failure(s"Rate limit exceeded. Available tokens: ${rateLimiter.availableTokens}")
}
}
}
```
### Per-User Rate Limiting
```text
object PerUserRateLimiting {
import scala.collection.mutable
class UserRateLimiter {
private val buckets = mutable.Map[String, TokenBucket]()
def getLimiter(userId: String): TokenBucket = {
buckets.getOrElseUpdate(userId, new TokenBucket(100, 10))
}
def tryConsume(userId: String): Boolean = {
getLimiter(userId).tryConsume()
}
def getStatus(userId: String): String = {
val limiter = getLimiter(userId)
s"User $userId has ${limiter.availableTokens} tokens available"
}
}
}
```
---
## Batch Processing
### Embedding Batching
```text
object EmbeddingBatching {
import scala.concurrent.Future
case class EmbeddingJob(
text: String,
callback: Vector[Double] => Unit
)
class BatchEmbeddingProcessor(
embeddingClient: EmbeddingClient,
batchSize: Int = 50,
batchTimeoutMs: Int = 1000
) {
private var batch = scala.collection.mutable.ListBuffer[EmbeddingJob]()
private var lastBatchTime = System.currentTimeMillis()
def embed(text: String): Future[Vector[Double]] = {
val promise = scala.concurrent.Promise[Vector[Double]]()
synchronized {
batch += EmbeddingJob(text, embedding => {
promise.success(embedding)
})
if (batch.size >= batchSize ||
System.currentTimeMillis() - lastBatchTime > batchTimeoutMs) {
processBatch()
}
}
promise.future
}
private def processBatch(): Unit = {
if (batch.nonEmpty) {
val texts = batch.map(_.text).toList
val callbacks = batch.map(_.callback).toList
// Embed all at once
val embeddings = embeddingClient.embedBatch(texts)
// Call callbacks
embeddings.zip(callbacks).foreach { case (emb, callback) =>
callback(emb)
}
batch.clear()
lastBatchTime = System.currentTimeMillis()
}
}
}
}
```
---
## Distributed Execution
### Worker Pool Pattern
```text
object WorkerPool {
import scala.concurrent.{ExecutionContext, Future}
import java.util.concurrent.{Executors, ThreadPoolExecutor}
class AgentWorkerPool(numWorkers: Int) {
private implicit val ec = ExecutionContext.fromExecutor(
Executors.newFixedThreadPool(numWorkers)
)
def processInParallel[T](
tasks: List[String],
processor: String => Result[T]
): Future[List[Result[T]]] = {
val futures = tasks.map(task =>
Future {
processor(task)
}
)
Future.sequence(futures)
}
}
// Usage: Process multiple queries in parallel
def processMultipleQueries(
queries: List[String],
agent: Agent
): Future[List[Result[String]]] = {
val pool = new AgentWorkerPool(numWorkers = 4)
pool.processInParallel(queries) { query =>
agent.run(query).map(_.message)
}
}
}
```
### Load Balancing Across Models
```text
object LoadBalancing {
case class ModelLoad(
modelName: String,
currentLoad: Int,
capacity: Int
) {
def utilization: Double = currentLoad.toDouble / capacity
}
class LoadBalancer(models: List[LlmClient]) {
private val loads = scala.collection.mutable.Map[String, Int]()
models.foreach(m => loads(m.modelName) = 0)
def selectLeastLoaded(): LlmClient = {
models.minBy(m => loads(m.modelName))
}
def recordRequest(modelName: String): Unit = {
loads(modelName) = loads.getOrElse(modelName, 0) + 1
}
def recordCompletion(modelName: String): Unit = {
loads(modelName) = Math.max(0, loads.getOrElse(modelName, 1) - 1)
}
def execute(query: String): Result[String] = {
val model = selectLeastLoaded()
recordRequest(model.modelName)
try {
model.generate(query)
} finally {
recordCompletion(model.modelName)
}
}
}
}
```
---
## Queue-Based Processing
### Task Queue
```text
object TaskQueue {
import scala.concurrent.{Future, Promise}
import scala.collection.mutable
sealed trait TaskStatus
case object Pending extends TaskStatus
case object Processing extends TaskStatus
case object Completed extends TaskStatus
case object Failed extends TaskStatus
case class Task(
id: String,
query: String,
status: TaskStatus = Pending,
result: Option[String] = None
)
class TaskProcessor(agent: Agent, numWorkers: Int = 3) {
private val queue = mutable.Queue[Task]()
private val tasks = mutable.Map[String, Task]()
private var running = true
def submitTask(query: String): String = {
val taskId = java.util.UUID.randomUUID().toString
val task = Task(taskId, query)
queue.enqueue(task)
tasks(taskId) = task
taskId
}
def getStatus(taskId: String): Option[TaskStatus] = {
tasks.get(taskId).map(_.status)
}
def getResult(taskId: String): Option[String] = {
tasks.get(taskId).flatMap(_.result)
}
def start(): Future[Unit] = {
Future {
while (running) {
if (queue.nonEmpty) {
val task = queue.dequeue()
processTask(task)
}
Thread.sleep(100)
}
}(scala.concurrent.ExecutionContext.global)
}
private def processTask(task: Task): Unit = {
tasks(task.id) = task.copy(status = Processing)
agent.run(task.query) match {
case Result.Success(response) =>
tasks(task.id) = task.copy(
status = Completed,
result = Some(response.message)
)
case Result.Failure(error) =>
tasks(task.id) = task.copy(
status = Failed,
result = Some(s"Error: $error")
)
}
}
}
}
```
---
## Performance Optimization
### Inference Optimization
```text
object InferenceOptimization {
// Use smaller models for simple tasks
def selectModelByComplexity(query: String): LlmClient = {
if (query.length < 100) {
fastModel // faster, cheaper
} else {
powerfulModel // slower, more capable
}
}
// Parallel generation with early stopping
def generateWithTimeout(
agent: Agent,
query: String,
timeoutMs: Long = 5000
): Result[String] = {
val future = scala.concurrent.Future {
agent.run(query)
}(scala.concurrent.ExecutionContext.global)
scala.concurrent.Await.result(
future,
scala.concurrent.duration.Duration(timeoutMs, "ms")
)
}
// Speculative decoding (decode multiple possibilities in parallel)
def speculativeDecoding(
agent: Agent,
query: String
): Result[String] = {
// Generate with draft model
val draft = draftAgent.run(query)
// Verify with main model
val verified = agent.run(draft.message)
verified
}
}
```
---
## Scaling Checklist
### ✅ Before scaling
- [ ] Current implementation handles 10x load?
- [ ] Identified bottleneck (CPU, memory, I/O)?
- [ ] Have baseline metrics?
- [ ] Cost estimates for 10x scale?
- [ ] Cache strategy defined?
- [ ] Rate limiting in place?
- [ ] Error handling for overload?
### ✅ Scaling implementation
- [ ] Horizontal scaling set up?
- [ ] Load balancing configured?
- [ ] Database/cache scaled?
- [ ] Monitoring alarms set?
- [ ] Gradual rollout planned (10% → 25% → 50% → 100%)?
- [ ] Rollback plan ready?
- [ ] Performance verified at scale?
---
## See Also
- [Production Monitoring](./production-monitoring) - Monitor scaled systems
- [Error Recovery](./error-recovery) - Handle failures at scale
- [Security Best Practices](./security-best-practices) - Secure at scale
---
**Last Updated:** February 2026
**Status:** Production Ready
---
Source: https://llm4s.org/guide/patterns/error-recovery
# Error Recovery Patterns
> **Note:** Code examples in this guide are illustrative pseudocode showing recommended patterns. For working examples using the actual LLM4S API, see [modules/samples](https://github.com/llm4s/llm4s/tree/main/modules/samples).
Resilience strategies for handling failures, recovering from errors, and gracefully degrading functionality. This guide covers retries, circuit breakers, fallbacks, and failure scenarios.
## Retry Strategies
### Exponential Backoff with Jitter
```text
object RetryStrategies {
import scala.concurrent.{Future, ExecutionContext}
import scala.concurrent.duration._
import scala.util.{Try, Success, Failure}
case class RetryConfig(
maxAttempts: Int = 5,
initialDelayMs: Long = 100,
maxDelayMs: Long = 32000,
multiplier: Double = 2.0,
jitterFactor: Double = 0.1
)
def exponentialBackoffRetry[T](
config: RetryConfig,
operation: () => Result[T]
)(implicit ec: ExecutionContext): Future[Result[T]] = {
def retry(attempt: Int): Future[Result[T]] = {
operation() match {
case success @ Result.Success(_) =>
Future.successful(success)
case failure @ Result.Failure(_) if attempt >= config.maxAttempts =>
Future.successful(failure)
case failure @ Result.Failure(error) =>
// Calculate delay with exponential backoff and jitter
val exponentialDelay =
Math.min(
config.maxDelayMs,
config.initialDelayMs * Math.pow(config.multiplier, attempt - 1)
).toLong
val jitter = (Math.random() - 0.5) * 2 * exponentialDelay * config.jitterFactor
val totalDelay = Math.max(0, exponentialDelay + jitter.toLong)
println(s"Attempt $attempt failed: $error. Retrying in ${totalDelay}ms...")
// Wait and retry
akka.pattern.after(totalDelay.millis,
scala.concurrent.ExecutionContext.Implicits.global
)(retry(attempt + 1))
}
}
retry(1)
}
}
```
### Retry with Timeout
```text
object TimeoutRetry {
def retryWithTimeout[T](
operation: () => Result[T],
timeoutMs: Long = 30000,
maxRetries: Int = 3
): Result[T] = {
var lastError: String = ""
for (attempt <- 1 to maxRetries) {
try {
val future = scala.concurrent.Future {
operation()
}(scala.concurrent.ExecutionContext.global)
val result = scala.concurrent.Await.result(
future,
scala.concurrent.duration.Duration(timeoutMs, "ms")
)
return result
} catch {
case e: scala.concurrent.TimeoutException =>
lastError = s"Timeout after ${timeoutMs}ms"
if (attempt < maxRetries) {
Thread.sleep(1000L * attempt) // Increasing delay
}
case e: Exception =>
lastError = e.getMessage
if (attempt < maxRetries) {
Thread.sleep(1000L * attempt)
}
}
}
Result.failure(s"Failed after $maxRetries attempts: $lastError")
}
}
```
---
## Circuit Breaker Pattern
### Implementation
```text
object CircuitBreaker {
import scala.concurrent.duration._
sealed trait CircuitState
case object Closed extends CircuitState // Operating normally
case object Open extends CircuitState // Failing, reject requests
case object HalfOpen extends CircuitState // Testing if recovered
case class CircuitBreakerConfig(
failureThreshold: Int = 5, // failures before opening
successThreshold: Int = 2, // successes in half-open before closing
timeout: Duration = 60.seconds // time before trying half-open
)
class CircuitBreakerImpl[T](config: CircuitBreakerConfig) {
private var state: CircuitState = Closed
private var failureCount = 0
private var successCount = 0
private var lastFailureTime = System.currentTimeMillis()
def execute[A](operation: () => Result[A]): Result[A] = {
state match {
case Closed =>
executeInClosedState(operation)
case Open =>
executeInOpenState(operation)
case HalfOpen =>
executeInHalfOpenState(operation)
}
}
private def executeInClosedState[A](
operation: () => Result[A]
): Result[A] = {
operation() match {
case success @ Result.Success(_) =>
failureCount = 0
success
case failure @ Result.Failure(error) =>
failureCount += 1
if (failureCount >= config.failureThreshold) {
state = Open
println(s"Circuit breaker opened after $failureCount failures")
}
failure
}
}
private def executeInOpenState[A](
operation: () => Result[A]
): Result[A] = {
val timeSinceFailure = System.currentTimeMillis() - lastFailureTime
if (timeSinceFailure > config.timeout.toMillis) {
state = HalfOpen
successCount = 0
println("Circuit breaker half-open, testing recovery...")
executeInHalfOpenState(operation)
} else {
Result.failure(
s"Circuit breaker is OPEN. Service unavailable. Retry in " +
s"${config.timeout.toMillis - timeSinceFailure}ms"
)
}
}
private def executeInHalfOpenState[A](
operation: () => Result[A]
): Result[A] = {
operation() match {
case success @ Result.Success(_) =>
successCount += 1
if (successCount >= config.successThreshold) {
state = Closed
failureCount = 0
println("Circuit breaker closed, service recovered")
}
success
case failure @ Result.Failure(error) =>
state = Open
lastFailureTime = System.currentTimeMillis()
println("Circuit breaker reopened")
failure
}
}
}
}
```
### Usage with Agents
```text
object CircuitBreakerAgent {
def safeAgentCall(
agent: Agent,
query: String,
breaker: CircuitBreaker
): Result[String] = {
breaker.execute { () =>
agent.run(query).map(_.message)
}
}
}
```
---
## Fallback Strategies
### Model Fallback
```text
object ModelFallback {
case class ModelFallbackConfig(
primaryModel: LlmClient,
secondaryModel: LlmClient,
tertiaryModel: LlmClient
)
def runWithFallback(
query: String,
config: ModelFallbackConfig
): Result[String] = {
println("Attempting primary model...")
config.primaryModel.generate(query) match {
case success @ Result.Success(_) => success
case Result.Failure(error1) =>
println(s"Primary failed: $error1. Trying secondary...")
config.secondaryModel.generate(query) match {
case success @ Result.Success(_) => success
case Result.Failure(error2) =>
println(s"Secondary failed: $error2. Trying tertiary...")
config.tertiaryModel.generate(query) match {
case success @ Result.Success(_) => success
case Result.Failure(error3) =>
Result.failure(
s"All models failed. Primary: $error1, Secondary: $error2, Tertiary: $error3"
)
}
}
}
}
// Cost-aware fallback: use cheaper models first
def costAwareFallback(
query: String,
models: List[(LlmClient, Double)] // (model, costPer1kTokens)
): Result[String] = {
val sortedByPrice = models.sortBy(_._2) // Cheapest first
sortedByPrice.foldLeft[Result[String]](
Result.failure("No models available")
) { case (acc, (model, price)) =>
acc match {
case success @ Result.Success(_) => success
case Result.Failure(_) =>
println(s"Trying ${model.modelName} (price: $$${price}/1k tokens)")
model.generate(query)
}
}
}
}
```
### Cached Fallback
```text
object CachedFallback {
def runWithCacheFallback(
query: String,
agent: Agent,
cache: Cache[String],
circuitBreaker: CircuitBreaker
): Result[String] = {
// Try circuit breaker with cache fallback
circuitBreaker.execute { () =>
agent.run(query)
} match {
case success @ Result.Success(_) =>
// Cache the successful response
cache.put(query, success.message, ttl = 3600)
success
case Result.Failure(error) =>
// Fall back to cache
cache.get(query) match {
case Some(cachedResponse) =>
println(s"Service failed, using cached response")
Result.success(cachedResponse)
case None =>
println(s"Service failed and no cache available")
Result.failure(error)
}
}
}
}
```
---
## Graceful Degradation
### Feature Degradation
```text
object GracefulDegradation {
case class RAGResult(
answer: String,
sources: List[String],
groundingScore: Double,
degradedMode: Boolean = false
)
def answerWithDegradation(
query: String,
ragSystem: RAGSystem,
agent: Agent
): RAGResult = {
// Try full RAG pipeline
try {
val sources = ragSystem.retrieve(query)
val context = sources.map(_.content).mkString("\n")
val answer = agent
.run(s"Answer this question using the context:\n$context\n\nQuestion: $query")
.toOption
.flatMap(_.answer) // Some only for a Completed turn
.getOrElse("")
val groundingScore = ragSystem.checkGrounding(answer, sources)
RAGResult(
answer = answer,
sources = sources.map(_.url),
groundingScore = groundingScore,
degradedMode = false
)
} catch {
case e: Exception =>
println(s"RAG pipeline failed: ${e.getMessage}. Using degraded mode.")
// Fall back to simple generation without RAG
val answer = agent.run(query).toOption.flatMap(_.answer).getOrElse("")
RAGResult(
answer = answer,
sources = List(), // No sources
groundingScore = 0.0, // Unknown grounding
degradedMode = true
)
}
}
// Progressive degradation - disable expensive features
def progressiveDegradation(
operationCost: Double,
costBudgetRemaining: Double
): List[String] = {
val disabledFeatures = scala.collection.mutable.ListBuffer[String]()
if (operationCost > costBudgetRemaining * 0.5) {
disabledFeatures += "grounding_check" // Disable grounding check
}
if (operationCost > costBudgetRemaining * 0.7) {
disabledFeatures += "reranking" // Disable reranking
}
if (operationCost > costBudgetRemaining * 0.9) {
disabledFeatures += "multi_stage_retrieval" // Use single-stage
}
disabledFeatures.toList
}
}
```
---
## Failure Scenarios & Responses
### Timeout Handling
```text
object TimeoutHandling {
def handleTimeout[T](
operation: String,
timeoutMs: Long
): Result[String] = {
Result.failure(
s"$operation timed out after ${timeoutMs}ms. " +
s"This may indicate the service is overloaded or network is slow. " +
s"Please try again in a moment."
)
}
}
```
### Rate Limit Handling
```text
object RateLimitHandling {
def handleRateLimit(
retryAfterSeconds: Int
): Result[String] = {
Result.failure(
s"Rate limit exceeded. Please retry after $retryAfterSeconds seconds. " +
s"Consider implementing exponential backoff and jitter in your client."
)
}
}
```
### Provider Outage Handling
```text
object OutageHandling {
def handleProviderOutage(
provider: String,
fallbackAvailable: Boolean
): Result[String] = {
if (fallbackAvailable) {
Result.failure(
s"$provider is temporarily unavailable. Using fallback provider."
)
} else {
Result.failure(
s"$provider is temporarily unavailable and no fallback is configured. " +
s"Please try again later."
)
}
}
}
```
---
## Best Practices
### ✅ Do's
- **Retry transient errors only**: Don't retry 401 authentication errors
- **Use exponential backoff**: Prevents overwhelming failed services
- **Add jitter**: Prevents thundering herd problems
- **Circuit break early**: Don't wait for timeouts
- **Log all retries**: For debugging and monitoring
- **Have multiple fallbacks**: Don't rely on single fallback
- **Test failure paths**: Ensure graceful degradation works
- **Set sensible timeouts**: Not too short (false failures), not too long (user wait)
### ❌ Don'ts
- **Don't retry forever**: Set max attempt limit
- **Don't hardcode delays**: Make retry config configurable
- **Don't lose context**: Include error details in logs
- **Don't assume fixed backoff works**: Different errors need different strategies
- **Don't ignore context limits**: LLM calls may fail near context window limit
- **Don't forget to reset state**: Circuit breaker needs periodic recovery attempts
- **Don't cascade failures**: Use circuit breakers to isolate failures
---
## Error Recovery Checklist
### ✅ Implementation checklist
- [ ] All API calls wrapped with retry logic?
- [ ] Circuit breaker implemented for external services?
- [ ] Fallback models/providers defined?
- [ ] Timeout configured appropriately?
- [ ] Error messages are user-friendly?
- [ ] Errors are logged with full context?
- [ ] Monitoring alerts for error rates?
- [ ] Graceful degradation tested?
- [ ] Failure scenarios documented?
---
## See Also
- [Production Monitoring](./production-monitoring) - Monitor error rates
- [Scaling Strategies](./scaling-strategies) - Handle failures under load
- [Multi-Agent Orchestration](./multi-agent-orchestration) - Fallback agents
---
**Last Updated:** February 2026
**Status:** Production Ready
---
Source: https://llm4s.org/guide/patterns/security-best-practices
# Security Best Practices
> **Note:** Code examples in this guide are illustrative pseudocode showing recommended patterns. For working examples using the actual LLM4S API, see [modules/samples](https://github.com/llm4s/llm4s/tree/main/modules/samples).
Comprehensive security patterns for protecting sensitive data, managing credentials, and maintaining audit trails in production LLM systems.
## API Key Management
> **Note:** Code examples in this guide are illustrative pseudocode showing recommended patterns. For working examples using the actual LLM4S API, see [modules/samples](https://github.com/llm4s/llm4s/tree/main/modules/samples).
### Configuration-Based Access
```text
object SecureKeyManagement {
import org.llm4s.config.Llm4sConfig
def loadApiKey(provider: String): Result[String] = {
Llm4sConfig.provider(provider).apiKey match {
case Some(key) if key.nonEmpty => Result.success(key)
case _ => Result.failure(
s"API key for provider '$provider' not configured. " +
s"Set it in your configuration file or environment."
)
}
}
// Safe logging (don't log keys!)
def logApiUsage(key: String, provider: String): Unit = {
val maskedKey = key.take(5) + "..." + key.takeRight(5)
println(s"Using provider: $provider with key: $maskedKey")
}
}
```
### Key Vault Integration
```text
object KeyVault {
import com.azure.security.keyvault.secrets._
class SecureKeyManager(vaultUri: String) {
private val client = new SecretClientBuilder()
.vaultUrl(vaultUri)
.buildClient()
def getKey(keyName: String): Result[String] = {
try {
val secret = client.getSecret(keyName)
Result.success(secret.getValue)
} catch {
case e: Exception =>
Result.failure(s"Failed to retrieve key from vault: ${e.getMessage}")
}
}
}
// Usage
def getOpenAIKey(): Result[String] = {
val vaultManager = new SecureKeyManager("https://myvault.vault.azure.net/")
vaultManager.getKey("openai-api-key")
}
}
```
### Key Rotation
```text
object KeyRotation {
case class KeyMetadata(
key: String,
provider: String,
createdAt: Long,
rotatedAt: Long,
expiresAt: Long
) {
def isExpired: Boolean = System.currentTimeMillis() > expiresAt
def daysSinceRotation: Long =
(System.currentTimeMillis() - rotatedAt) / (1000 * 60 * 60 * 24)
}
class KeyRotationManager {
private var activeKey: KeyMetadata = _
private var rotationScheduler: Option[java.util.Timer] = None
def rotateKey(newKey: String, provider: String): Unit = {
val now = System.currentTimeMillis()
val thirtyDaysFromNow = now + (30 * 24 * 60 * 60 * 1000)
activeKey = KeyMetadata(
key = newKey,
provider = provider,
createdAt = now,
rotatedAt = now,
expiresAt = thirtyDaysFromNow
)
println(s"Key rotated for $provider. Valid until ${new java.util.Date(thirtyDaysFromNow)}")
}
def scheduleRotation(intervalDays: Int = 30): Unit = {
val timer = new java.util.Timer()
val delayMs = intervalDays * 24 * 60 * 60 * 1000L
timer.scheduleAtFixedRate(
new java.util.TimerTask {
def run(): Unit = {
println("Time to rotate API keys!")
// In production: fetch new key from vault and rotate
}
},
delayMs,
delayMs
)
rotationScheduler = Some(timer)
}
}
}
```
---
## Input Validation
### Query Validation
```text
object InputValidation {
case class ValidationRule(
name: String,
validate: String => Boolean,
errorMessage: String
)
class InputValidator {
private val rules = scala.collection.mutable.ListBuffer[ValidationRule]()
def addRule(rule: ValidationRule): Unit = {
rules += rule
}
def validate(input: String): Result[String] = {
rules.find(!_.validate(input)) match {
case Some(failedRule) =>
Result.failure(failedRule.errorMessage)
case None =>
Result.success(input)
}
}
}
// Common validation rules
def setupCommonValidations(): InputValidator = {
val validator = new InputValidator()
// Length check
validator.addRule(ValidationRule(
name = "length",
validate = _.length <= 10000,
errorMessage = "Input exceeds maximum length of 10000 characters"
))
// No injection patterns
validator.addRule(ValidationRule(
name = "no_injection",
validate = !containsSQLInjectionPattern(_),
errorMessage = "Input contains potentially harmful patterns"
))
// No excessive special characters
validator.addRule(ValidationRule(
name = "no_spam",
validate = countSpecialChars(_) < 0.5,
errorMessage = "Input contains too many special characters"
))
validator
}
private def containsSQLInjectionPattern(input: String): Boolean = {
val sqlPatterns = List("';", "--", "/*", "*/", "xp_", "sp_")
sqlPatterns.exists(input.toLowerCase.contains(_))
}
private def countSpecialChars(input: String): Double = {
val specialCount = input.count(!_.isLetterOrDigit)
specialCount.toDouble / input.length
}
}
```
### Prompt Injection Protection
```text
object PromptInjectionProtection {
def detectPromptInjection(userInput: String): Boolean = {
val injectionIndicators = List(
"ignore all previous instructions",
"forget everything",
"system prompt",
"you are now",
"instead please",
"role play as",
"pretend you are"
)
val lowerInput = userInput.toLowerCase
injectionIndicators.exists(lowerInput.contains(_))
}
def sanitizePrompt(userInput: String): Result[String] = {
if (detectPromptInjection(userInput)) {
Result.failure(
"Your input was flagged as potentially malicious. " +
"Please rephrase your request without system instructions."
)
} else {
Result.success(userInput)
}
}
// Use prompt templating to separate user input
def safePromptTemplate(
userQuery: String,
systemPrompt: String
): String = {
s"""$systemPrompt
User Query:
[BEGIN USER INPUT]
$userQuery
[END USER INPUT]
Please answer based only on the user's question above."""
}
}
```
---
## Output Sanitization
### PII Redaction
Use the built-in `PIIMasker` and `PIIDetector` guardrails rather than hand-written patterns: they run before a query
reaches the model and before an answer is returned, and are covered by tests. What they detect, how to attach them and
their limits are in [Data Protection: personal data in prompts](../data-protection#personal-data-in-prompts).
---
## Audit Logging
### Comprehensive Audit Trail
```text
object AuditLogging {
import org.slf4j.LoggerFactory
import java.time.Instant
val auditLogger = LoggerFactory.getLogger("AUDIT")
case class AuditEvent(
timestamp: Instant,
userId: String,
action: String,
resource: String,
details: Map[String, String],
success: Boolean,
errorMessage: Option[String] = None
)
def logAPICall(
userId: String,
provider: String,
model: String,
inputTokens: Int,
outputTokens: Int
): Unit = {
val event = AuditEvent(
timestamp = Instant.now(),
userId = userId,
action = "API_CALL",
resource = s"$provider/$model",
details = Map(
"input_tokens" -> inputTokens.toString,
"output_tokens" -> outputTokens.toString
),
success = true
)
auditLogger.info(
s"${event.timestamp} | User: ${event.userId} | " +
s"Action: ${event.action} | Resource: ${event.resource}"
)
}
def logAccessToSensitiveData(
userId: String,
dataType: String,
recordCount: Int,
accessReason: String
): Unit = {
auditLogger.info(
s"${Instant.now()} | User: $userId | " +
s"Accessed: $dataType ($recordCount records) | " +
s"Reason: $accessReason"
)
}
def logAuthenticationAttempt(
userId: String,
success: Boolean,
method: String
): Unit = {
auditLogger.info(
s"${Instant.now()} | User: $userId | " +
s"Authentication: ${if (success) "SUCCESS" else "FAILED"} via $method"
)
}
}
```
### Compliance Logging
```text
object ComplianceLogging {
case class DataProcessingLog(
timestamp: Long,
userId: String,
dataCategory: String,
operation: String, // PROCESS, STORE, SHARE, DELETE
retentionDays: Int,
userConsent: Boolean
)
class ComplianceLogger {
private val logs = scala.collection.mutable.ListBuffer[DataProcessingLog]()
def logDataProcessing(
userId: String,
dataCategory: String,
operation: String,
retentionDays: Int
): Unit = {
logs += DataProcessingLog(
timestamp = System.currentTimeMillis(),
userId = userId,
dataCategory = dataCategory,
operation = operation,
retentionDays = retentionDays,
userConsent = true
)
}
def getComplianceReport(days: Int = 30): String = {
val cutoff = System.currentTimeMillis() - (days * 24 * 60 * 60 * 1000L)
val recent = logs.filter(_.timestamp > cutoff)
val operationCounts = recent.groupBy(_.operation).mapValues(_.length)
val dataCategoryCounts = recent.groupBy(_.dataCategory).mapValues(_.length)
s"""Compliance Report (last $days days):
|Operations: $operationCounts
|Data Categories: $dataCategoryCounts
|Total Records Processed: ${recent.length}""".stripMargin
}
}
}
```
---
## Data Retention
### Automatic Data Expiration
Agent thread history is pruned with a `RetentionPolicy` (`maxAge`, `maxCheckpoints`, `maxEvents`) and a store sweep,
and a user's threads, memories and indexed documents are erased with `Agent.forget`, `MemoryStore.deleteMatching` and
`VectorStore.deleteByFilter`. See [Data Protection: deleting a user's data](../data-protection#deleting-a-users-data).
---
## Best Practices
### ✅ Do's
- **Never log API keys**: Even in debug mode
- **Rotate keys regularly**: At least every 30 days
- **Use key vaults**: Don't hardcode secrets
- **Validate all input**: Both user queries and system inputs
- **Sanitize all output**: Remove PII before returning
- **Keep audit trails**: For compliance and debugging
- **Use HTTPS only**: For all API communication
- **Implement rate limiting**: To prevent abuse
- **Monitor suspicious patterns**: Unusual access or usage
- **Use least privilege**: Grant minimal required permissions
### ❌ Don'ts
- **Don't store keys in code**: Use environment variables or vaults
- **Don't trust user input**: Always validate and sanitize
- **Don't output sensitive data**: Check for PII before returning
- **Don't log full queries**: Log only hashed or redacted version
- **Don't skip authentication**: Verify user identity
- **Don't use weak passwords**: Enforce strong credential policies
- **Don't ignore security updates**: Keep dependencies patched
- **Don't expose error details**: Log full errors, show generic messages to users
- **Don't test in production**: Use staging for security tests
- **Don't assume perimeter security**: Defense in depth required
---
## Security Checklist
### ✅ Before Production
- [ ] API keys loaded from secure vault, not code
- [ ] HTTPS/TLS enforced for all API calls
- [ ] Input validation on all user-facing endpoints
- [ ] Output sanitization removes PII
- [ ] Audit logging enabled for sensitive operations
- [ ] Rate limiting configured
- [ ] Authentication required for all endpoints
- [ ] Data retention policy defined and enforced
- [ ] Error messages don't expose system details
- [ ] Secrets not logged anywhere
- [ ] Security tests included in test suite
- [ ] Penetration testing scheduled
- [ ] Incident response plan documented
---
## See Also
- [Production Monitoring](./production-monitoring) - Monitor security events
- [Multi-Agent Orchestration](./multi-agent-orchestration) - Secure agent communication
- [OWASP Top 10](https://owasp.org/www-project-top-ten/) - Web application security
---
**Last Updated:** February 2026
**Status:** Production Ready
---
Source: https://llm4s.org/guide/writing-a-provider
# Writing a Provider
How to publish your own LLM4S provider module - a chat client, an embedding provider, or both -
against `llm4s-core`, without changing LLM4S itself.
---
## Before you start
`llm4s-core` ships no provider. OpenAI, Anthropic, Ollama, Voyage and the rest are separate
modules that register themselves through the same public SPI you will use, so a provider you
publish as `my-llm4s-acme` works exactly like one built into this repository: users add the
dependency, write a config section, and `LLMConnect.getClient` finds it.
Check first whether you need a module at all:
- **The vendor speaks the OpenAI `/chat/completions` API.** The generic `openai-compatible`
provider in `llm4s-openai-compatible` may already cover it with config alone - see
[OpenAI-compatible endpoints](providers#openai-compatible-endpoints). If the vendor bends the
format (tool-call ids, system role, reasoning fields), the fix is an
`OpenAICompatibleDialect` plus a descriptor in that module, contributed upstream - Mistral and
Cohere are the examples. Do not write another copy of the chat-completions client.
- **Anything else** gets a module of its own, which is what this page covers.
Three rules hold throughout:
1. **Return `Result[A]`, never throw.** `type Result[+A] = Either[LLMError, A]`. Wrap calls that
can throw with `Try(...).toResult` (`import org.llm4s.types.TryOps`).
2. **Never read the environment.** No `sys.env`, `System.getenv` or `ConfigFactory.load()` in
provider code. Configuration arrives typed, through the descriptor; environment variables are
bound in your module's `reference.conf`.
3. **Never edit `llm4s-core` to add a provider.** Everything below is an extension point.
## Worked examples
Read one of these alongside this page:
| Module | Shape | Start here when |
|--------|-------|-----------------|
| [`modules/providers/voyage`](https://github.com/llm4s/llm4s/tree/main/modules/providers/voyage) | Embeddings only, HTTP, no SDK | You only need embeddings - the smallest template |
| [`modules/ollama`](https://github.com/llm4s/llm4s/tree/main/modules/ollama) | Chat + embeddings, HTTP, no SDK, model lister | You need a chat client |
| [`modules/openai-compatible`](https://github.com/llm4s/llm4s/tree/main/modules/openai-compatible) | One client, many dialects | The vendor is OpenAI-compatible |
## Module layout
```
my-llm4s-acme/
├── build.sbt # "org.llm4s" %% "llm4s-core", and "llm4s-provider-testkit" % Test
└── src/
├── main/
│ ├── resources/
│ │ ├── META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule
│ │ └── reference.conf
│ └── scala/com/acme/llm4s/
│ ├── AcmeConfig.scala # your ProviderConfig
│ ├── AcmeProvider.scala # your ProviderDescriptor
│ ├── AcmeClient.scala # your LLMClient
│ └── Llm4sAcmeModule.scala # your Llm4sProviderModule
└── test/scala/com/acme/llm4s/
└── Llm4sAcmeModuleSpec.scala
```
Use your own package. The in-repo modules keep `org.llm4s.*` for source compatibility with
earlier releases; a third-party module has no reason to, and anything `private[llm4s]` stays
out of reach either way (see [Stability](#stability)).
## Registration
### The module class
`org.llm4s.llmconnect.spi.Llm4sProviderModule` is the unit of discovery: one per artifact, listing
whatever it supplies. Both members default to `Nil`.
(not compile-checked)
```scala
trait Llm4sProviderModule:
def chatProviders: Seq[ProviderDescriptor] = Nil
def embeddingProviders: Seq[EmbeddingProviderDescriptor] = Nil
```
Implement it as a **`class` with a public no-arg constructor, not an `object`** -
`java.util.ServiceLoader` instantiates the class it is given, and an `object` has no such
constructor:
(not compile-checked)
```scala
package com.acme.llm4s
import org.llm4s.llmconnect.spi.{ EmbeddingProviderDescriptor, Llm4sProviderModule, ProviderDescriptor }
final class Llm4sAcmeModule extends Llm4sProviderModule:
override def chatProviders: Seq[ProviderDescriptor] = Seq(AcmeProvider)
override def embeddingProviders: Seq[EmbeddingProviderDescriptor] = Seq(AcmeEmbeddings)
```
### The services file
`src/main/resources/META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule`, one line:
```
com.acme.llm4s.Llm4sAcmeModule
```
`ProviderRegistry.discover()` (and `ProviderRegistry.default`, its cached result) now finds the
module whenever the jar is on the classpath. Tell users who build fat jars to *merge* services
files (`MergeStrategy.filterDistinctLines` in sbt-assembly, `ServicesResourceTransformer` in
maven-shade); where that is impossible they register explicitly:
(not compile-checked)
```scala
given ProviderRegistry = ProviderRegistry.ofModules(new Llm4sAcmeModule)
// or extend another registry: ProviderRegistry.default.withModule(new Llm4sAcmeModule)
```
If two modules supply the same id, the last registered wins and `registry.report.collisions`
records it.
### `reference.conf`: binding the vendor key
Credentials belong to a vendor, keyed by provider id. Your module's `reference.conf` binds the
vendor's conventional variable to `llm4s.credentials..apiKey`; any chat section or embeddings
block without an `apiKey` of its own falls back to it:
```hocon
# my-llm4s-acme defaults. HOCON merges this with every other reference.conf on the classpath.
llm4s {
credentials {
acme {
apiKey = ${?ACME_API_KEY}
}
}
# Embeddings are keyed by provider id, so their env bindings can live here too.
embeddings {
acme {
baseUrl = ${?ACME_EMBEDDING_BASE_URL}
model = ${?ACME_EMBEDDING_MODEL}
}
}
}
```
Do not put defaults (base URL, default model) here - those are code, on the config spec. Chat
sections are keyed by the *user's* instance name, which a `reference.conf` cannot know.
## The chat descriptor
`org.llm4s.llmconnect.spi.ProviderDescriptor` is the whole chat extension point:
(not compile-checked)
```scala
trait ProviderDescriptor:
def id: ProviderId
def aliases: Set[String] = Set.empty
def configSpec: ProviderConfigSpec
def features: ProviderFeatures = ProviderFeatures.default
def modelLister: Option[ProviderModelLister] = None
def buildConfig(providerName: String, section: ProvidersConfigModel.NamedProviderConfig)(using
ContextWindowResolver
): Result[ProviderConfig]
def buildClient(config: ProviderConfig, options: LlmClientOptions)(using
ModelRegistryService
): Result[LLMClient]
```
`buildConfig` receives a section that has **already been validated** against `configSpec`:
required fields are present, the API key has fallen back to `llm4s.credentials..apiKey`, and
`section.extras` holds exactly your declared extra keys, defaults applied. `buildClient` must
check the config's type rather than cast - `ProviderDescriptor.expectConfig` does that.
### Workload identity
A provider that can authenticate with a workload identity token (a SPIFFE JWT-SVID, say) declares
the keys it reads from a section's `auth` block in `ProviderConfigSpec.authExtras`. Core parses
`identityTokenFile` / `identityToken` itself and hands the rest to `buildConfig` as
`section.auth`; read a key with `ProviderDescriptor.requireAuthExtra`. A section sets `apiKey` or
`auth`, never both, and a section with `auth` never consults the shared credential.
`TokenExchange.provider` gives an RFC 8693 exchange with caching and single-flight refresh, and
`FakeTokenExchangeServer` and `TestJwt` in the testkit test it without a network. An SDK that does
its own federation takes the identity token's file path instead (see `AnthropicProvider`).
The configuration side is `@Stable` ([#1818](https://github.com/llm4s/llm4s/issues/1818)): `AuthConfig`,
`IdentitySource`, `TokenExchangeConfig`, `AccessToken` and the `AccessTokenProvider` trait, which a Stable
`NamedProviderConfig`, `ProviderDescriptor.requireAuthExtra` and `OpenAICompatibleClient.Credential` carry. The
machinery behind them - `TokenExchange`, `CachingAccessTokenProvider` and `IdentityTokenSource` - is still
`@Experimental`, so keep it out of your own Stable signatures. An `AccessTokenProvider` should return a failure as a
`Left`; `OpenAICompatibleClient` and `CachingAccessTokenProvider` turn one that throws into an `AuthenticationError`.
### `ProviderConfigSpec` and `ProviderConfigKey`
The spec says what a section needs. The fields a provider author uses:
| Field | Meaning |
|-------|---------|
| `requiresApiKey` | the section must resolve an `apiKey` (its own, or the shared credential) |
| `requiresBaseUrl` | the section must set `baseUrl`; only when there is no `defaultBaseUrl` (Ollama) |
| `defaultBaseUrl` | used when the section omits `baseUrl` - the single source for your default endpoint |
| `baseUrlExample`, `baseUrlEnv` | shown in the missing-`baseUrl` message |
| `apiKeyEnv` | the variables your `reference.conf` binds, named in the missing-key error |
| `extras` | your provider-specific keys, as `ProviderConfigKey`s |
| `authExtras` | the keys you read from a section's `auth` block, as `ProviderConfigKey`s; empty (the default) means the provider rejects `auth` |
`ProviderConfigSpec.apiKeyAndDefaultBaseUrl(defaultBaseUrl, apiKeyEnv)` builds the common shape.
Anything beyond the built-in fields (`provider`, `model`, `baseUrl`, `apiKey`, `headers`, `timeouts` -
`ProviderConfigSpec.BuiltinKeys`) is declared as an extra rather than smuggled through a built-in
field:
(not compile-checked)
```scala
ProviderConfigKey.required("project", "the project that owns your Acme deployment")
ProviderConfigKey.optional("region", "the Acme region serving the model", default = Some("eu-west"))
```
`ProviderConfigKey` also takes `env` (a variable to suggest in the missing-key message) and
`deprecatedAliases` (old names, accepted with a warning), as named arguments or afterwards with
`withEnv` / `withDeprecatedAliases`. Undeclared keys in a section are
reported as unknown and not passed on. Values are strings; parse numbers or booleans in
`buildConfig` and return a `ConfigurationError` for a malformed one.
Read the values with `section.extra("region")` (an `Option[String]`) or
`ProviderDescriptor.requireExtra(providerName, section, "region")` (a `Result[String]`).
### The descriptor
(not compile-checked)
```scala
package com.acme.llm4s
import org.llm4s.config.ProvidersConfigModel.NamedProviderConfig
import org.llm4s.llmconnect.{ LLMClient, LlmClientOptions }
import org.llm4s.llmconnect.config.{ ContextWindowResolver, ProviderConfig }
import org.llm4s.llmconnect.spi.*
import org.llm4s.model.ModelRegistryService
import org.llm4s.types.ProviderModelTypes.ProviderId
import org.llm4s.types.Result
object AcmeProvider extends ProviderDescriptor:
val id: ProviderId = ProviderId("acme")
override val aliases: Set[String] = Set("acme-ai")
val configSpec: ProviderConfigSpec = ProviderConfigSpec(
requiresApiKey = true,
defaultBaseUrl = Some("https://api.acme.example/v1"),
apiKeyEnv = Seq("ACME_API_KEY"), // what reference.conf binds
extras = Seq(
ProviderConfigKey.optional("region", "the Acme region serving the model", default = Some("eu-west"))
)
)
// Say so if the client cannot stream or call tools; the default claims both.
override val features: ProviderFeatures = ProviderFeatures(streaming = true, toolCalling = false)
def buildConfig(providerName: String, section: NamedProviderConfig)(using
ContextWindowResolver
): Result[ProviderConfig] =
for
apiKey <- ProviderDescriptor.requireApiKey(providerName, section)
baseUrl <- ProviderDescriptor.resolveBaseUrl(providerName, section, configSpec)
region <- ProviderDescriptor.requireExtra(providerName, section, "region")
config <- AcmeConfig.fromValues(apiKey, section.model.asString, baseUrl, region)
yield config
def buildClient(config: ProviderConfig, options: LlmClientOptions)(using
ModelRegistryService
): Result[LLMClient] =
ProviderDescriptor.expectConfig[AcmeConfig](id, config).map(new AcmeClient(_, options))
```
### Your `ProviderConfig`
`org.llm4s.llmconnect.config.ProviderConfig` is an open trait: `providerId`, `model`,
`contextWindow`, `reserveCompletion`, `endpointUrl` and `withModel`. Resolve the window from the
model registry with `ContextWindowResolver` (the `using` parameter `buildConfig` receives), and
keep the key out of `toString`:
(not compile-checked)
```scala
import org.llm4s.error.ConfigurationError
final case class AcmeConfig(
apiKey: String,
model: String,
baseUrl: String,
region: String,
contextWindow: Int,
reserveCompletion: Int
) extends ProviderConfig:
val providerId: ProviderId = ProviderId("acme")
def endpointUrl: Option[String] = Some(baseUrl)
def withModel(model: String): AcmeConfig = copy(model = model)
override def toString: String = s"AcmeConfig(***, $model, $baseUrl, $region)"
object AcmeConfig:
def fromValues(apiKey: String, model: String, baseUrl: String, region: String)(using
resolver: ContextWindowResolver
): Result[AcmeConfig] =
if apiKey.trim.isEmpty then Left(ConfigurationError("Acme apiKey must be non-empty", List("apiKey")))
else
val (window, reserve) = resolver.resolve(
lookupProviders = Seq("acme"),
modelName = model,
defaultContextWindow = 128000,
defaultReserve = 4096,
fallbackResolver = _ => (128000, 4096) // when the registry does not know the model
)
Right(AcmeConfig(apiKey, model, baseUrl, region, window, reserve))
```
## What the user writes
With `my-llm4s-acme` on the classpath and `ACME_API_KEY` set, a section needs only `provider`
and `model`; extras sit beside the built-in fields:
```hocon
llm4s {
providers {
provider = "acme-main"
acme-main {
provider = "acme"
model = "acme-large"
region = "us-east" # your ProviderConfigKey; omit it to get "eu-west"
}
acme-team { # a second account sets its own key, which wins
provider = "acme"
model = "acme-small"
apiKey = ${?ACME_TEAM_API_KEY}
}
}
}
```
(not compile-checked)
```scala
for
config <- Llm4sConfig.defaultProvider()
client <- LLMConnect.getClient(config)
yield client
```
## Shared plumbing for clients
These live in `llm4s-core` and are part of the provider-author SPI. Use them rather than
re-implementing their behaviour, so your provider reports metrics, errors and costs the way
every other provider does.
### `BaseLifecycleLLMClient` and `MetricsRecording`
`org.llm4s.llmconnect.BaseLifecycleLLMClient` (extends `LLMClient with MetricsRecording`) gives
an idempotent `close()`, a closed-state check, and metrics around each call. You supply:
(not compile-checked)
```scala
protected def metrics: MetricsCollector // from MetricsRecording; use options.metrics
protected def clientDescription: String // "Acme client for model x", for the closed error
protected def providerName: String // metrics label, e.g. "acme"
protected def modelName: String
protected def releaseResources(): Unit = () // called once, from close()
```
Wrap `complete` and `streamComplete` bodies in
`completeWithMetrics(operation: => Result[Completion]): Result[Completion]`, which fails a closed
client, times the call, and records tokens and `Completion.estimatedCost` on success. For
non-`Completion` calls, `MetricsRecording.withMetrics(provider, model, operation, extractUsage,
extractCost)` does the same.
### `ProviderExchangeRecorder` and `ProviderExchangeLogging`
`LlmClientOptions.exchangeLogging` is a `ProviderExchangeLogging` (`Disabled`, or
`Enabled(sink)`) - the user's opt-in to capture raw request/response bodies. Honour it by calling,
once per request:
(not compile-checked)
```scala
ProviderExchangeRecorder.record(
exchangeLogging = options.exchangeLogging,
provider = providerName,
model = Some(config.model),
startedAt = startedAt, // java.time.Instant taken before the call
requestBody = requestText,
responseBody = Some(responseText),
result = result // the Result you are about to return
)
```
It is a no-op when disabled and never fails your call if the sink throws. Do not put credentials
in the request body you record; send them in headers.
### `Llm4sHttpClient`
`org.llm4s.http.Llm4sHttpClient` is the JDK-backed HTTP client (`Llm4sHttpClient.create()`), with
`get`, `post`, `postBytes`, `postMultipart`, `put`, `delete`, `postRaw` and `postStream`. Take one
as a constructor parameter so tests can inject a stub.
(not compile-checked)
```scala
def post(
url: String,
headers: Map[String, String] = Map.empty,
body: String = "",
timeout: FiniteDuration = 10.seconds
): Result[HttpResponse]
```
Every method returns a `Result` and never throws for a transport failure, so there is no
`try`/`catch` to write:
| Failure | `Left` |
|---|---|
| request or connection timed out | `TimeoutError` (carries the timeout) |
| connection refused, unknown host, other I/O error | `NetworkError` |
| invalid URL, header or timeout; unreadable multipart file | `ValidationError` |
| thread interrupted (the interrupt flag is kept) | `CancelledError` |
A non-2xx status is **not** an error at this layer: it is a `Right` response for you to inspect
(`HttpResponse.ensureSuccess`, or `HttpErrorMapper` below). Timeouts are
`scala.concurrent.duration.FiniteDuration`; the default is 10 seconds, and 10 minutes for
`postStream`. `HttpResponse`, `HttpRawResponse` and `StreamingHttpResponse` all carry `headers`
(lower-case keys); `response.header("Retry-After")` looks one up case-insensitively. A
`StreamingHttpResponse`'s body is yours to close, on an error status too.
### Honouring the `timeouts` block
Every provider section, and every embedding section, may carry `timeouts { request = ..., stream = ... }`.
You do not read the block: the loader parses it into `NamedProviderConfig.timeouts` (checked to be positive
and finite) and applies it to the config your descriptor builds, by calling `ProviderConfig.withTimeouts`.
To honour it, carry a `ProviderTimeouts` in your config type and pass it on when you call the HTTP client:
(not compile-checked)
```scala
final case class AcmeConfig(
apiKey: String,
model: String,
baseUrl: String,
override val timeouts: ProviderTimeouts = ProviderTimeouts.default
) extends ProviderConfig:
// ... providerId, contextWindow and the rest ...
override def withTimeouts(timeouts: ProviderTimeouts): AcmeConfig = copy(timeouts = timeouts)
// in the client: your own default stays when the section sets none
httpClient.post(url, headers, body, timeout = config.timeouts.requestOr(2.minutes))
httpClient.postStream(url, headers, body, timeout = config.timeouts.streamOr(10.minutes))
```
A config that does not override `withTimeouts` ignores the block, since the default returns `this`; a
provider with no timeouts to offer needs to do nothing. An embedding provider reads
`EmbeddingProviderConfig.timeouts.requestOr(...)`, and only `request` has a meaning there.
Methods added to the trait after 1.0 will have default implementations, so a test double that
implements it keeps compiling.
### `HttpErrorMapper`
(not compile-checked)
```scala
HttpErrorMapper.mapHttpError(
statusCode: Int,
body: String,
provider: String,
headers: Map[String, Seq[String]] = Map.empty
): Result[Nothing]
```
Maps a non-2xx response to the standard error types - 401/403 `AuthenticationError`, 429
`RateLimitError`, 400 `ValidationError`, anything else `ServiceError` - pulling a message out of
common JSON error shapes, redacted and truncated. Retry and fallback logic keys off these types,
so use it rather than inventing your own. Pass the response's `headers`: a `Retry-After`
(delta-seconds or an HTTP-date) becomes the error's `retryAfter`, a `FiniteDuration` - on a 429's
`RateLimitError`, or a 503's (any other status's) `ServiceError` - so retries wait as long as the
provider asked rather than a guessed backoff.
### `CostEstimator`
(not compile-checked)
```scala
CostEstimator.estimate(model: String, usage: TokenUsage)(using ModelRegistryService): Option[Double]
CostEstimator.estimateFromMetadata(metadata: Option[ModelMetadata], usage: TokenUsage): Option[Double]
```
Prices a call from the model registry, including cached and reasoning tokens. Put the result in
`Completion.estimatedCost`; `completeWithMetrics` records it.
### `RequestTransformer` and `TransformationResult`
`org.llm4s.model.RequestTransformer` applies what the model registry knows about a model -
temperature range, disallowed parameters, no tools, no system messages, no structured output,
no native streaming - before you encode a request:
(not compile-checked)
```scala
RequestTransformer.default(service: ModelRegistryService): RequestTransformer
RequestTransformer.withOverrides(overrides: Map[String, ModelCapabilities], service): RequestTransformer
RequestTransformer.adjusted(service)(adjust: (String, ModelCapabilities) => ModelCapabilities): RequestTransformer
```
Use `adjusted` for vendor rules the registry does not record: you receive the model id and the
registry's capabilities and return the ones to apply. `llm4s-openai`'s `OpenAIModelRules` is the
example - the o-series take temperature 1 only and no system message, whatever the registry says:
(not compile-checked)
```scala
RequestTransformer.adjusted(service) { (modelId, caps) =>
if modelId.startsWith("acme-reasoner") then caps.withSupportsSystemMessages(false)
else caps
}
```
`TransformationResult.transform(modelId, options, messages, transformer, dropUnsupported)` runs
the option and message transforms in one call and returns the transformed `options`, `messages`
and `requiresFakeStreaming`.
### `EmbeddingProvider`
(not compile-checked)
```scala
trait EmbeddingProvider:
def embed(request: EmbeddingRequest): Result[EmbeddingResponse]
def embedMultimodal(request: MultimediaEmbeddingRequest): Result[EmbeddingResponse] // defaults to a 501 error
```
What an `EmbeddingProviderDescriptor.build` returns. Report failures as `EmbeddingError(code,
message, provider)` - but not a cancellation: an interrupted `Llm4sHttpClient` call returns `CancelledError`, and
mapping it into an `EmbeddingError` (as `left.map(err => EmbeddingError(...))` does) hides it. Pass it through, as the
Voyage, Jina, Ollama and OpenAI providers do, and see [Cancellation](#cancellation).
`request.purpose` is an `InputPurpose`, `Document` (the default) or `Query`: whether the texts are to be
indexed or are a search query. If your vendor's API embeds the two differently, map it onto the vendor's own
parameter in `embed` (Voyage and Cohere send `input_type`, Jina sends `task`), and let a typed setting of the
provider, if it has one, win over it, because that is a deliberate choice and can say more than two values
can. If your models embed both alike, ignore it and do not send a field for it. Test the exact wire field
for both purposes against a local server, as `VoyageAIInputTypeSpec` does.
### Streaming: `SSEParser`, `StreamingAccumulator`, `StreamingToolArgumentParser`
- `SSEParser.createStreamingParser()` returns an incremental parser: `addChunk(text)`, then drain
`nextEvent()` / `hasEvents`, and `flush()` at end of stream. Each `SSEEvent` has `data`,
`event`, `id` and `retry`. `SSEParser.parseEvent` / `parseStream` handle whole strings.
- `StreamingAccumulator.create()` folds `StreamedChunk`s into a final `Completion`:
`addChunk(chunk)`, `updateTokens(prompt, completion)` (or `updateTokensWithThinking`) when usage
arrives, then `toCompletion: Result[Completion]`. It keys tool calls by id, so **every
tool-call chunk must carry its call's id** - if the wire format identifies continuations only
by index, map index to id yourself; a chunk with an empty id is skipped. It is mutable: one per
request.
- Pass each streamed argument fragment to the accumulator as-is, as `ujson.Str(raw)` (`ujson.Obj()`
for an empty one): the accumulator joins the fragments and parses the whole. Do not parse a
fragment on its own - a fragment that happens to be valid JSON (`":"`, `"Paris"`, `1`) would lose
its quotes or be dropped. `StreamingToolArgumentParser.parse(raw: String): ujson.Value` is for a
reassembled argument string (`{}` for empty, the parsed JSON when complete, the raw string
otherwise); `StreamingAccumulator` uses it on the joined arguments.
### `ProviderModelLister`
(not compile-checked)
```scala
trait ProviderModelLister:
def listModels(config: NamedProviderConfig, httpClient: Llm4sHttpClient): Result[List[DiscoveredModel]]
```
Return one from `ProviderDescriptor.modelLister` if the vendor has a model-listing endpoint. If
it serves the OpenAI `/models` shape, `ProviderModelListers.openAICompatible(provider,
defaultBaseUrl, ...)` in `llm4s-openai-compatible` builds one for you (add that dependency);
its `sectionHeaders` parameter derives request headers from the section, e.g. from one of your
extras.
### Redaction
llm4s's own credential redaction (`org.llm4s.util.Redaction`) is internal and not part of the
SPI. `HttpErrorMapper` already runs provider error bodies through it, so error text you build
with it is redacted; beyond that, keep credentials out of anything you log.
## A minimal client
(not compile-checked)
```scala
final class AcmeClient(
config: AcmeConfig,
clientOptions: LlmClientOptions,
http: Llm4sHttpClient = Llm4sHttpClient.create()
)(using registry: ModelRegistryService)
extends BaseLifecycleLLMClient:
protected val metrics: MetricsCollector = clientOptions.metrics
protected def clientDescription: String = s"Acme client for model ${config.model}"
protected def providerName: String = "acme"
protected def modelName: String = config.model
private val transformer = RequestTransformer.default(registry)
private val headers = Map("Authorization" -> s"Bearer ${config.apiKey}", "Content-Type" -> "application/json")
def complete(conversation: Conversation, options: CompletionOptions): Result[Completion] =
completeWithMetrics {
TransformationResult
.transform(config.model, options, conversation.messages, transformer)
.flatMap(t => send(AcmeWire.encode(config.model, t.messages, t.options)))
}
def streamComplete(
conversation: Conversation,
options: CompletionOptions,
onChunk: StreamedChunk => Unit
): Result[Completion] =
completeWithMetrics(...) // postStream + SSEParser + StreamingAccumulator, as Ollama's client does
def getContextWindow(): Int = config.contextWindow
def getReserveCompletion(): Int = config.reserveCompletion
private def send(requestText: String): Result[Completion] =
val startedAt = Instant.now()
val response = http.post(s"${config.baseUrl}/chat", headers, requestText, timeout = 120.seconds)
val result = response.flatMap { r =>
if r.statusCode / 100 == 2 then AcmeWire.decode(r.body).map(withCost)
else HttpErrorMapper.mapHttpError(r.statusCode, r.body, providerName, r.headers)
}
ProviderExchangeRecorder.record(
clientOptions.exchangeLogging, providerName, Some(config.model), startedAt,
requestText, response.toOption.map(_.body), result
)
result
private def withCost(c: Completion): Completion =
c.withEstimatedCost(c.usage.flatMap(CostEstimator.estimate(config.model, _)))
```
`AcmeWire` stands for your own request encoding and response decoding.
### Content parts
A `UserMessage` and a `ToolMessage` carry `parts: Vector[ContentPart]`: `Text`, `Image` and `Document`, each image
or document from a `MediaSource` (`Url`, `Inline` bytes, or a `ProviderFile` uploaded to a named provider). Your
encoder sends every part your API takes, in each role it takes it, and fails the request **before sending it** with
`UnsupportedContentPartError(providerName, part.kind, role.name)` for any other: never drop a part, and never turn
one into text. Send a `ProviderFile` only when its `provider` is your provider id. A message with exactly one text
part should encode as your API's plain text form, as before; `message.content` is the text view. The matrix of what
each llm4s provider sends is in [Multimodal content](multimodal).
## The embedding descriptor
`org.llm4s.llmconnect.spi.EmbeddingProviderDescriptor` is the embedding half. Its config lives at
`llm4s.embeddings.` and is selected with `EMBEDDING_MODEL=/`. Usually you declare
an `EmbeddingConfigSpec` and implement `build`; the default `buildConfig` resolves model, base URL
and key against the spec.
(not compile-checked)
```scala
object AcmeEmbeddings extends EmbeddingProviderDescriptor:
val id: ProviderId = ProviderId("acme") // chat and embedding ids are separate namespaces
override val configSpec: EmbeddingConfigSpec = EmbeddingConfigSpec(
requiresApiKey = true,
defaultBaseUrl = Some("https://api.acme.example/v1"),
defaultModel = Some("acme-embed-1"),
apiKeyEnv = Seq("ACME_API_KEY"),
modelEnv = Some("ACME_EMBEDDING_MODEL")
)
// What ModelDimensionRegistry answers from; override dimensionsOf for model-name variants.
override val modelDimensions: Map[String, Int] = Map("acme-embed-1" -> 1024)
def build(config: EmbeddingProviderConfig): Result[EmbeddingProvider] =
Right(AcmeEmbeddingProvider(config, Llm4sHttpClient.create()))
```
`EmbeddingConfigSpec.defaultApiKey` is for a provider that takes a key but needs no real one
(local Ollama). Override `buildConfig(section, modelOverride)` only when the config cannot be
expressed with the spec.
## Testing: `Llm4sModuleSpec`
Every provider module in this repository proves its registration in one spec, and yours should
too - it replaces the exhaustivity check the compiler gave when providers were a closed `enum`.
It shows that the module is discovered, that it is the only module supplying its ids, that it can
be registered explicitly, that each descriptor gets from config to client, and that your
`reference.conf` binds the variable `apiKeyEnv` names.
Those checks ship as **`llm4s-provider-testkit`**, the same ones the in-repo provider modules
run. Add it in test scope; it brings ScalaTest with it:
(not compile-checked)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-provider-testkit" % llm4sVersion % Test
```
The testkit **requires JDK 21**: `assertCancelsWhenInterrupted` and its siblings run the call on a
virtual thread (`Thread.ofVirtual`), and `LocalProviderTestServer` answers on
`Executors.newVirtualThreadPerTaskExecutor`, both JDK 21 APIs. The build that runs your
`Llm4sModuleSpec` needs a JDK 21 toolchain, whatever your provider module itself targets
(the library's own minimum JDK is settled in
[#1493](https://github.com/llm4s/llm4s/issues/1493)).
It has these parts, all in `org.llm4s.testkit`:
| | What it gives you |
|---|---|
| `ProviderModuleChecks` | The checks, as assertions: `assertModule` (= `assertDiscovered` + `assertSoleSupplier` + `assertRegistrableWith`), `assertBuildsClient` / `buildClient`, `assertRefusesForeignConfig`, `assertStreams`, `assertCancelsWhenInterrupted`, `assertCancelsStreamWhenInterrupted`, `assertOneToolResultPerCall`, `assertContentParts` (each `ContentPartCase` your client declares reaches a local server with its payload; every other fails with `UnsupportedContentPartError` and sends nothing), `assertBuildsEmbeddingProvider`, `assertCredentialBindings`, `assertEmbeddingCredentialBindings`. Mix the trait into a spec of any ScalaTest style, or call the companion object. A failure points at the line in your spec. |
| `ProviderTestConfig` | `loadSection`, `loadProvider` and `loadEmbeddings`: config loaded as an application loads it, from a HOCON string over every `reference.conf` on the classpath, with `${?VAR}` resolved against a `Map` you pass - never the real environment, so an exported `ACME_API_KEY` on your machine cannot make a test pass that fails in CI. |
| `CredentialsRoundTrip` | `chatSectionKey`, `chatBindings`, `embeddingsKey`, `embeddingBindings`: which key a section or embeddings block with no `apiKey` of its own ends up with, for cases the assertions do not cover - an alias, two variables in precedence order, a variable that must *not* be picked up. |
| `LocalProviderTestServer` | `withServer(path)(handler)(baseUrl => ...)`, `sendJsonResponse`, `sendSseResponse`, `holdOpen` / `streamThenHold` (requests that never finish, released when `withServer` ends), OpenAI-format bodies and an Anthropic-format `anthropicMessage`: the JDK's HTTP server on an ephemeral port, to point a client at. `ToolResultContract` (with `ToolMessageFormat`) reads a request body for the tool-result contract below. |
(not compile-checked)
```scala
package com.acme.llm4s
import org.llm4s.config.ProvidersConfigModel.NamedProviderConfig
import org.llm4s.llmconnect.spi.ProviderRegistry
import org.llm4s.testkit.{ CredentialsRoundTrip, ProviderModuleChecks, ProviderTestConfig }
import org.llm4s.testkit.LocalProviderTestServer.{ sendSseResponse, withServer }
import org.llm4s.types.ProviderModelTypes.*
import org.scalatest.matchers.should.Matchers
import org.scalatest.wordspec.AnyWordSpec
class Llm4sAcmeModuleSpec extends AnyWordSpec with Matchers with ProviderModuleChecks:
private val section = NamedProviderConfig(
provider = AcmeProvider.id, model = ModelName("acme-large"), baseUrl = None,
apiKey = Some(ApiKey("test-key")),
extras = Map("region" -> "eu-west") // built in code, so no validation fills the default
)
private val streamBody = "data: {\"delta\":\"Hi\"}\n\ndata: [DONE]\n\n" // Acme's wire format
"the my-llm4s-acme services entry" should {
// Discovered by ProviderRegistry.discover(), every id and alias resolving to your
// descriptors; no other module supplying "acme"; and ProviderRegistry.ofModules works.
"register the module" in assertModule(new Llm4sAcmeModule)
}
"AcmeProvider" should {
"build an AcmeClient from a section, and refuse another provider's config" in {
assertBuildsClient(AcmeProvider, section) shouldBe an[AcmeClient]
assertRefusesForeignConfig(AcmeProvider)
}
"really stream" in {
withServer("/v1/chat")(exchange => sendSseResponse(exchange, streamBody)) { baseUrl =>
assertStreams(assertBuildsClient(AcmeProvider, section.withBaseUrl(BaseUrl(baseUrl))))
}
}
"load from a named section as an application does, extras defaulted" in {
given ProviderRegistry = ProviderRegistry.default
ProviderTestConfig
.loadProvider(
"acme-main",
"""llm4s.providers.acme-main { provider = "acme", model = "acme-large" }""",
Map("ACME_API_KEY" -> "test-key")
)
.map(_.asInstanceOf[AcmeConfig].region) shouldBe Right("eu-west")
}
}
"the my-llm4s-acme reference.conf" should {
"bind ACME_API_KEY to llm4s.credentials.acme.apiKey, for chat and embeddings" in {
assertCredentialBindings(AcmeProvider)
assertEmbeddingCredentialBindings(AcmeEmbeddings, "acme-embed-1")
}
"give the acme-ai alias the same key" in {
given ProviderRegistry = ProviderRegistry.default
CredentialsRoundTrip.chatSectionKey("acme-ai", Map("ACME_API_KEY" -> "k")) shouldBe Right(Some("k"))
}
}
```
`assertCredentialBindings` loads a section with no `apiKey` once per variable in `apiKeyEnv`,
with only that variable set, so the shared credential is the only place a key can come from; when
it fails it names the `reference.conf` line that is missing. Test the client itself against
`LocalProviderTestServer` too, including that an error status maps to the right `LLMError`.
### Cancellation
A provider must return `Left(CancelledError)`, with the interrupt flag kept, when its thread is
interrupted. It must not throw `InterruptedException` or clear the flag. A client that extends
`BaseLifecycleLLMClient` gets this from `completeWithMetrics`; `Llm4sHttpClient` already returns
`CancelledError` for an interrupted request or stream read. Your module spec should run
`assertCancelsWhenInterrupted` and `assertCancelsStreamWhenInterrupted` against
`LocalProviderTestServer.holdOpen` and `streamThenHold`; an embedding provider runs
`assertEmbeddingCancelsWhenInterrupted`, and any other client call that returns a `Result` (a reranker, a speech
or image client) `assertCallCancelsWhenInterrupted("name")(call)`. A client that does not extend `BaseLifecycleLLMClient` can use the public helpers `CancelledError.attempt`, `CancelledError.whenInterrupted`, `CancelledError.fromThrowable` and `CancelledError.isCancellation`. `fromThrowable` and `isCancellation` only classify: they never set the flag, so if your code catches an `InterruptedException` itself, restore the flag with `Thread.currentThread().interrupt()`. A bare `InterruptedIOException`, such as OkHttp's call timeout, is not a cancellation unless the thread is interrupted or an `InterruptedException` lies beneath it.
### Tool results
An agent's tool loop sends a conversation in which every tool call has exactly one result, straight
after the assistant message that made it, and relies on your client to send it that way: OpenAI's
format puts one `role: "tool"` message per call after the assistant message, Anthropic's one
`tool_result` block per call at the start of the next user turn. A chat provider whose API takes
tools runs `assertOneToolResultPerCall(format, response, path)(baseUrl => client)`: it sends each
of `ToolResultContract.cases` - a parallel batch with error, rejection and denial results, an
unknown tool, provider call ids reused across turns, a handoff, a refused handoff batch - to a
`LocalProviderTestServer` answering with `response`, and fails on any
`ToolResultContract.violations` in the body the server received, and when the body carries fewer
calls or native results than the conversation: a client that drops a call it cannot pair, or sends
a result as plain text, would hide a broken conversation rather than send it. `ToolMessageFormat`
is `OpenAIChat` or `AnthropicMessages`; a provider with another wire format checks its own
encoding against `ToolResultContract.cases` the same way.
(not compile-checked)
```scala
"send exactly one tool result per call" in {
assertOneToolResultPerCall(ToolMessageFormat.OpenAIChat, LocalProviderTestServer.openAICompletion("done")) {
baseUrl => assertBuildsClient(AcmeProvider, section.withBaseUrl(BaseUrl(baseUrl)))
}
}
```
## Stability
`llm4s-core` reaches a binary-compatibility baseline (MiMa) at a pre-1.0 release, once the designs are final, and freezes at 1.0. The
provider-author SPI described on this page is **public and frozen at 1.0**: binary-compatible
across all 1.x releases, so a provider compiled against 1.0 keeps working. That covers:
- the registration SPI in `org.llm4s.llmconnect.spi` - `Llm4sProviderModule`,
`ProviderDescriptor`, `ProviderConfigSpec`, `ProviderConfigKey`, `ProviderFeatures`,
`EmbeddingProviderDescriptor`, `EmbeddingConfigSpec`, `EmbeddingProviderSection`,
`ProviderRegistry`;
- `ProviderConfig`, `NamedProviderConfig`, `ContextWindowResolver`, `LlmClientOptions`;
- `BaseLifecycleLLMClient`, `MetricsRecording`, `ProviderExchangeRecorder`,
`ProviderExchangeLogging`, `HttpErrorMapper`, `CostEstimator`, `EmbeddingProvider`,
`StreamingAccumulator`, `SSEParser`, `StreamingToolArgumentParser`, `Llm4sHttpClient`,
`ProviderModelLister`, `RequestTransformer` and `TransformationResult`.
`llm4s-provider-testkit` is **Beta**, not part of the frozen SPI: it is a test-scope dependency,
so a change to it can break your tests but never your users, and it may gain checks in a minor
release (with a migration note).
Anything `private[llm4s]` - `ProviderResultOps`, for example - is internal, may change in any
release, and cannot be reached from your package anyway. Do not work around that by declaring
your code in `org.llm4s`.
The OpenAI-format helpers - `ResponseFormatMapper`, `ToolCallDeserializer` and
`StandardToolCallDeserializer`, `OpenAICompatibleClient` and its dialects - ship in
`llm4s-openai-compatible`, not in core. Depending on that module for them puts it on your users'
classpath; for an OpenAI-compatible vendor, contribute a dialect there instead.
## Checklist
- [ ] `Llm4sProviderModule` as a `class` with a no-arg constructor, named in
`META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule`
- [ ] `reference.conf` binds the vendor variable to `llm4s.credentials..apiKey`, and
`apiKeyEnv` names the same variable
- [ ] defaults (base URL, model) on the spec, not in `reference.conf`
- [ ] provider-specific keys declared as `ProviderConfigKey` extras
- [ ] `ProviderFeatures` honest about streaming and tool calling
- [ ] client built on `BaseLifecycleLLMClient`, errors via `HttpErrorMapper`, cost via
`CostEstimator`, exchanges via `ProviderExchangeRecorder`
- [ ] no exceptions escape, no environment reads
- [ ] `Llm4sModuleSpec`, on `llm4s-provider-testkit`, covering discovery, sole ownership,
explicit registration, the config-to-client round trip and the credential binding
- [ ] a chat provider that takes tools runs `assertOneToolResultPerCall`
---
Source: https://llm4s.org/guide/cats-effect
# cats-effect Integration
The `llm4s-effect` module wraps the synchronous `LLMClient` and `Agent` APIs in
[cats-effect](https://typelevel.org/cats-effect/) `IO` (or any `Async[F]` type),
keeping the compute thread pool free while long-running LLM calls block on the
dedicated blocking pool.
## Dependency
(not compile-checked)
```scala
// build.sbt
libraryDependencies += "org.llm4s" %% "llm4s-effect" % ""
```
## LLMClientIO
`LLMClientIO[F[_]]` is the cats-effect wrapper for `LLMClient`.
### Acquire from the environment
Use `LLMClientIO.resource[F]` to load provider config from the environment,
build the client, and release it on scope exit:
(not compile-checked)
```scala
import cats.effect.{IO, IOApp}
import org.llm4s.effect.cats.LLMClientIO
import org.llm4s.llmconnect.model.{Conversation, UserMessage}
object MyApp extends IOApp.Simple {
def run: IO[Unit] =
LLMClientIO.resource[IO].use { client =>
for {
completion <- client.complete(Conversation(Seq(UserMessage("What is 2 + 2?"))))
_ <- IO.println(completion.content)
} yield ()
}
}
```
### Wrap an existing client
If you already hold an `LLMClient` (e.g. constructed manually):
(not compile-checked)
```scala
import org.llm4s.effect.cats.LLMClientIO
val wrapped = LLMClientIO[IO](existingClient)
```
### Streaming
`streamComplete` returns an `fs2.Stream[F, StreamedChunk]` that delivers chunks incrementally.
The blocking provider call runs on an interruptible blocking thread and feeds a bounded queue,
so a slow consumer applies backpressure to the provider thread, and stopping early (`take`,
fiber cancellation) interrupts the call. If the call fails mid-stream, chunks already received
are emitted first and the stream then fails with `LLMException`:
(not compile-checked)
```scala
client
.streamComplete(conversation)
.evalMap(chunk => IO.print(chunk.content.getOrElse("")))
.compile
.drain
```
## AgentIO
`AgentIO[F[_]]` wraps an `Agent`, shifting the blocking agent loop to the blocking pool. `client.agent(id)(configure)` builds the agent from `Agent.builder(id, client)` with `configure` applied, so tools, guardrails, handoffs and middleware are set there; a builder that does not build fails the effect with its error.
(not compile-checked)
```scala
for {
agentIO <- client.agent("assistant")(_.withTools(myTools).withSystemPrompt("You are concise."))
result <- agentIO.run("Summarise this")
_ <- IO.println(result.answer)
} yield ()
```
### Multi-turn conversations
(not compile-checked)
```scala
for {
s1 <- agentIO.run("What's the weather in Paris?")
s2 <- agentIO.continueConversation(s1, "And London?")
} yield s2
```
## Cancellation
Cancellation is by thread interrupt, matching the llm4s core contract. `LLMClientIO.complete`, `AgentIO` and the streaming
methods all run the provider call on an interruptible blocking thread, so cancelling the fiber (or
a timeout) interrupts the call instead of waiting for it to finish.
## Error handling
All `LLMError` values are raised as `LLMException` in the `F` error channel:
(not compile-checked)
```scala
import org.llm4s.effect.cats.LLMException
client.complete(conversation).handleErrorWith {
case e: LLMException => IO.println(s"LLM error: ${e.error.message}")
case t => IO.raiseError(t)
}
```
## Serving an agent over HTTP
[Serving an Agent over HTTP](http-reference-apps) is a tested http4s (Ember) app over `AgentIO[IO]`: `POST /chat`
answers as JSON and `POST /chat/stream` streams the answer as server-sent events, with errors mapped to HTTP statuses
and the run cancelled when the client disconnects. Run it with `sbt "examplesHttp/run --fake"`. The same guide has a
Tapir variant: the two endpoints as Tapir definitions with typed error outputs, served by `tapir-http4s-server` over
the same `AgentIO[IO]`, with generated OpenAPI documentation and a Swagger UI at `/docs`.
## Environment variables
The variables llm4s reads, such as `OPENAI_API_KEY`, are listed in
[Environment variables llm4s reads](../getting-started/configuration#environment-variables-llm4s-reads).
The provider and model are chosen by a named section in `application.conf`; see
[Named provider sections](../getting-started/configuration#named-provider-sections).
### Differences from `Agent`
`AgentIO` is a deliberately thin wrapper over `Agent.start` and `AgentRun.await`. It exposes `run`,
`continueConversation`, `recover` and `resume`, each taking a `RunConfig` and returning an `F[AgentResult]`.
It does not expose a named `ThreadId` on `run` (each `run` starts a new thread, and
`continueConversation` continues the thread of a previous result) or `history`; for those, build the
`Agent` yourself and call `agent.start(...)` inside `IO.blocking`.
Cancelling the fiber cancels the run and returns once its turn has ended, so the thread can be
recovered at once with `recover`.
Errors arrive as `LLMException` in the effect's error channel: a provider, tool or middleware failure
is a `GraphError` (a provider error is `GraphError.NodeFailed(cause)`; an exception thrown by user
code arrives as a `NodeFailed` carrying the original). A guardrail block, the step limit and a
suspension for approval are not failures: they are the `AgentResult`'s `status`.
Tool calls are not a failure of the effect either. When the model calls a tool with arguments that do not
fit the tool's schema, the agent hands a structured error result (`{"error": ...}`) back to the model so it
can correct itself; the run continues and only the step limit or a provider error ends it. If you
need to see those results, read the `ToolMessage`s in `AgentResult.messages`.
---
Source: https://llm4s.org/guide/zio
# ZIO Integration
The `llm4s-zio` module wraps the synchronous `LLMClient` and `Agent` APIs in
[ZIO](https://zio.dev) effects, shifting blocking LLM calls to ZIO's blocking
thread pool and surfacing `LLMError` directly in the ZIO error channel.
## Dependency
(not compile-checked)
```scala
// build.sbt
libraryDependencies += "org.llm4s" %% "llm4s-zio" % ""
```
## LLMClientZ
`LLMClientZ` is the ZIO wrapper for `LLMClient`.
### Acquire via ZLayer
Use `LLMClientZ.layer` to load provider config from the environment, build the
client, and release it on scope exit:
(not compile-checked)
```scala
import org.llm4s.llmconnect.model.{Conversation, UserMessage}
import org.llm4s.toolapi.ToolRegistry
import org.llm4s.zio.LLMClientZ
import zio.{ZIO, ZIOAppDefault}
object MyApp extends ZIOAppDefault {
def run: ZIO[Any, Any, Any] =
(for {
client <- ZIO.service[LLMClientZ]
c <- client.complete(Conversation(Seq(UserMessage("What is 2 + 2?"))))
_ <- ZIO.debug(c.content)
} yield ()).provide(LLMClientZ.layer)
}
```
### Wrap an existing client
(not compile-checked)
```scala
import org.llm4s.zio.LLMClientZ
val wrapped: LLMClientZ = LLMClientZ(existingClient)
```
### Streaming
`streamComplete` returns a `ZStream[Any, LLMError, StreamedChunk]` that delivers chunks
incrementally. The blocking provider call runs on an interruptible blocking thread and feeds a
bounded queue, so a slow consumer applies backpressure, and stopping early or interrupting the
fiber interrupts the call. If the call fails mid-stream, chunks already received are emitted
first and the stream then fails with the `LLMError`:
(not compile-checked)
```scala
client
.streamComplete(conversation)
.map(_.content.getOrElse(""))
.runCollect
.map(_.mkString)
```
## AgentZ
`AgentZ` wraps an `Agent`, shifting the blocking agent loop to ZIO's blocking pool. `client.agent(id)(configure)` builds the agent from `Agent.builder(id, client)` with `configure` applied, so tools, guardrails, handoffs and middleware are set there; a builder that does not build fails with its `LLMError`.
`LLMError` is the native error type — no wrapping needed.
(not compile-checked)
```scala
for {
agentZ <- client.agent("assistant")(_.withTools(myTools).withSystemPrompt("You are concise."))
result <- agentZ.run("Summarise this")
_ <- ZIO.debug(result.answer.toString)
} yield ()
```
### Multi-turn conversations
(not compile-checked)
```scala
for {
s1 <- agentZ.run("What's the weather in Paris?")
s2 <- agentZ.continueConversation(s1, "And London?")
} yield s2
```
## Cancellation
Cancellation is by thread interrupt, matching the llm4s core contract. `LLMClientZ.complete`, `AgentZ` and the streaming
methods all run the provider call on an interruptible blocking thread, so cancelling the fiber (or
a timeout) interrupts the call instead of waiting for it to finish.
## Serving an agent over HTTP
[Serving an Agent over HTTP](http-reference-apps) includes a tested ZIO HTTP app over `AgentZ`: `POST /chat` answers
as JSON and `POST /chat/stream` streams `AgentZ.stream` as server-sent events, with errors mapped to HTTP statuses and
the run cancelled when the client disconnects. Run it with
`sbt "examplesHttp/runMain org.llm4s.examples.http.ziohttp.ZioChatServerMain --fake"`.
## Error handling
`LLMError` flows naturally in the ZIO error channel:
(not compile-checked)
```scala
client.complete(conversation).catchAll { err =>
ZIO.debug(s"LLM error: ${err.message}") *> ZIO.fail(err)
}
```
## Environment variables
The variables llm4s reads, such as `OPENAI_API_KEY`, are listed in
[Environment variables llm4s reads](../getting-started/configuration#environment-variables-llm4s-reads).
The provider and model are chosen by a named section in `application.conf`; see
[Named provider sections](../getting-started/configuration#named-provider-sections).
### Differences from `Agent`
`AgentZ` is a deliberately thin wrapper over `Agent.start` and `AgentRun.await`. It exposes `run`,
`continueConversation`, `recover` and `resume`, each taking a `RunConfig` and returning
`ZIO[Any, LLMError, AgentResult]`. It does not expose a named `ThreadId` on `run` (each `run` starts a
new thread, and `continueConversation` continues the thread of a previous result) or `history`; for
those, build the `Agent` yourself and call `agent.start(...)` inside `ZIO.attemptBlocking`.
Interrupting the fiber (or a timeout) cancels the run and returns once its turn has ended, so the
thread can be recovered at once with `recover`.
Errors arrive in the `LLMError` channel: a provider, tool or middleware failure is a `GraphError`
(a provider error is `GraphError.NodeFailed(cause)`; an exception thrown by user code arrives as a
`NodeFailed` carrying the original). A guardrail block, the step limit and a suspension for approval
are not failures: they are the `AgentResult`'s `status`.
Tool calls are not a failure of the effect either. When the model calls a tool with arguments that do not
fit the tool's schema, the agent hands a structured error result (`{"error": ...}`) back to the model so it
can correct itself; the run continues and only the step limit or a provider error ends it. If you
need to see those results, read the `ToolMessage`s in `AgentResult.messages`.
---
Source: https://llm4s.org/guide/caching
# Caching Responses and Embeddings
Avoid paying twice for the same work: an embedding cache for exact repeats, and a semantic cache for model responses.
---
## 1. What is available
Both caches live in `llm4s-core`, package `org.llm4s.llmconnect.caching`. With their default backends both keep their entries in memory, inside the object you create, and lose them when the process ends. The completion cache always works that way; the embedding cache takes its storage as an `EmbeddingCache`, and one of your own (see [Your own storage](#your-own-storage)) can keep vectors in a shared or persistent store instead.
| | Embedding cache | Completion (semantic) cache |
|---|---|---|
| Class | `CachedEmbeddingClient` with an `EmbeddingCache`, by default `InMemoryEmbeddingCache` | `CachingLLMClient`, which wraps any `LLMClient` |
| A hit needs | the same text, the same model name and the same `InputPurpose` (document or query) | an embedding at least `similarityThreshold` similar, the same `CompletionOptions`, and an entry no older than the TTL |
| Configured with | `InMemoryEmbeddingCache(maxSize = 10000, ttl = None)` | `CacheConfig.create(similarityThreshold, ttl, maxSize = 1000)` |
| Eviction | least recently used, plus the optional TTL | least recently used, plus the TTL |
| Seeing what happened | `cacheStats` | `TraceEvent.CacheHit` and `TraceEvent.CacheMiss` sent to your `Tracing` |
There is no configuration-file key for either cache. You construct them in code, and the examples below do exactly that.
## 2. Choosing between them
**The embedding cache is exact, so it is safe.** The same text embedded with the same model for the same purpose gives the same vector, so a cached vector is as good as a fresh one. Use it whenever you embed the same texts more than once: re-indexing documents, repeated queries in a RAG service, evaluation runs.
**The completion cache is approximate, so it can be wrong.** It answers a new question with the stored answer to an earlier, *similar* one. That saves a model call, but:
- It costs one embedding call on every `complete()`, hit or miss, because the prompt has to be embedded to be compared. The cache pays off only if a model call is worth more than that embedding call and your hit rate is high enough.
- A hit can be a false hit: two prompts the embedding model considers close may need different answers.
- A cached answer replaces a fresh sample. If you want a different answer each time, or an answer that depends on the current time, the user or fresh data, do not cache it.
## 3. Caching embeddings
Wrap an `EmbeddingClient` in a `CachedEmbeddingClient` and call `embed` on the wrapper:
(not compile-checked)
```scala
import org.llm4s.llmconnect.caching.{ CachedEmbeddingClient, InMemoryEmbeddingCache }
import org.llm4s.llmconnect.config.EmbeddingModelConfig
import org.llm4s.llmconnect.model.EmbeddingRequest
import scala.concurrent.duration._
val cache = new InMemoryEmbeddingCache[Seq[Double]](maxSize = 10000, ttl = Some(1.hour))
val cached = new CachedEmbeddingClient(base, cache)
val model = EmbeddingModelConfig("text-embedding-3-small", 1536)
cached.embed(EmbeddingRequest(Seq("hello", "world"), model)) // two misses, sent as one batched call
cached.embed(EmbeddingRequest(Seq("hello", "again"), model)) // "hello" is served from the cache
cached.cacheStats
```
Here `base` is the `EmbeddingClient` you would use without a cache. A request without a purpose embeds documents (`InputPurpose.Document`), which is what indexing wants. To cache repeated search queries, say so on the request, so that the query is embedded as one and cached apart from a document with the same text:
(not compile-checked)
```scala
import org.llm4s.llmconnect.caching.{ CachedEmbeddingClient, InMemoryEmbeddingCache }
import org.llm4s.llmconnect.config.EmbeddingModelConfig
import org.llm4s.llmconnect.model.{ EmbeddingRequest, InputPurpose }
val queries = new CachedEmbeddingClient(base, new InMemoryEmbeddingCache[Seq[Double]]())
val model = EmbeddingModelConfig("text-embedding-3-small", 1536)
queries.embed(EmbeddingRequest(Seq("what is llm4s?"), model, InputPurpose.Query)) // a miss
queries.embed(EmbeddingRequest(Seq("what is llm4s?"), model, InputPurpose.Query)) // a hit
queries.embed(EmbeddingRequest(Seq("what is llm4s?"), model)) // a miss: a document
```
The wrapper:
- looks every input text up first, and sends only the texts it does not have to the base client, in **one** batched request, then returns the vectors in the original input order;
- sends a text that is repeated inside one request once;
- never caches a failure: an error from the base client is returned as it is, and the next call tries again;
- is **not** an `EmbeddingClient`. It has the same `embed` method, but it cannot be passed where an `EmbeddingClient` is required, so call it directly.
`cacheStats` returns `CacheStats(size, hits, misses, totalRequests, hitRatePercent)`. Every looked-up text counts, so the two calls above give 3 misses and 1 hit (25%), and a text repeated inside one request counts once per occurrence. With the default `InMemoryEmbeddingCache`, `clearCache()` empties the cache and resets the statistics. With a custom `EmbeddingCache`, it delegates to that backend's `clear()` implementation; the trait default does nothing, and a backend may clear entries without resetting statistics.
### The in-memory cache
`InMemoryEmbeddingCache(maxSize = 10000, ttl = None)`:
- **Size.** At more than `maxSize` entries the least recently used one is evicted. Reading an entry counts as using it.
- **TTL.** With `ttl = Some(d)` an entry is expired once it is *older* than `d`. An entry exactly `d` old is still valid. Expiry is lazy: an expired entry is removed when it is read, and that read counts as a miss.
- **Threads.** It is safe to share between threads.
### What the key contains
The key function is given three things: the text, the model name and the request's `InputPurpose`. Some providers (Voyage, Jina and Cohere among them) embed a query and a document with the same text differently, so the purpose has to be part of the key.
The default, `CacheKeyGenerator.embeddingKey`, is the SHA-256 of those three as 64 hex characters. Each part is prefixed with its length before hashing, so the encoding is unambiguous: no text or model name, whatever characters it contains, can produce another request's key. The text `a:b` with model `c` and the text `a` with model `b:c` have different keys, and so do a query for model `m` and a document for a model named `m#query`.
The key holds the model *name* only. It does not include the embedding dimension or the provider, so one cache shared by clients that use the same model name for different things would mix their vectors. If that matters to you, or you want to keep tenants apart in one cache, pass your own key function as the third argument. `CacheKeyGenerator.sha256` takes any number of parts and encodes them the same unambiguous way, so a tenant can simply be one more part:
(not compile-checked)
```scala
import org.llm4s.llmconnect.caching.{ CacheKeyGenerator, CachedEmbeddingClient, InMemoryEmbeddingCache }
import org.llm4s.llmconnect.config.EmbeddingModelConfig
import org.llm4s.llmconnect.model.{ EmbeddingRequest, InputPurpose }
// The tenant is one more part of the key; sha256 keeps every part apart from the others.
def tenantKey(tenant: String)(text: String, model: String, purpose: InputPurpose): String =
CacheKeyGenerator.sha256(tenant, model, purpose.toString, text)
val cache = new InMemoryEmbeddingCache[Seq[Double]]()
val cached = new CachedEmbeddingClient(base, cache, tenantKey("tenant-a"))
cached.embed(EmbeddingRequest(Seq("hello"), EmbeddingModelConfig("text-embedding-3-small", 1536)))
```
### Your own storage
`EmbeddingCache[A]` is a small trait: `get`, `put`, `stats`, and an optional `clear`. Implement it to keep vectors somewhere else (a shared map, Redis, a database). Expiry and eviction are then your backend's job:
(not compile-checked)
```scala
import org.llm4s.llmconnect.caching.{ CacheStats, EmbeddingCache }
import java.util.concurrent.ConcurrentHashMap
import java.util.concurrent.atomic.AtomicInteger
/** A backend of your own: any store that can get and put a vector by key. */
class MapEmbeddingCache extends EmbeddingCache[Seq[Double]] {
private val store = new ConcurrentHashMap[String, Seq[Double]]()
private val hits = new AtomicInteger(0)
private val misses = new AtomicInteger(0)
def get(key: String): Option[Seq[Double]] = {
val found = Option(store.get(key))
if (found.isDefined) hits.incrementAndGet() else misses.incrementAndGet()
found
}
def put(key: String, embedding: Seq[Double]): Unit = { store.put(key, embedding); () }
override def clear(): Unit = store.clear()
def stats(): CacheStats = {
val (h, m) = (hits.get().toLong, misses.get().toLong)
CacheStats(store.size(), h, m, h + m, if (h + m == 0) 0.0 else 100.0 * h / (h + m))
}
}
```
### Limits
- **No request coalescing.** The cache is checked and then filled in separate steps. If several threads miss the same text at the same moment, each one calls the base client; the last write wins. Four concurrent requests for the same new text made four provider calls.
## 4. Caching model responses
### `CacheConfig`
Build a `CacheConfig` with `CacheConfig.create`. It returns a `Result`, because the values are checked:
(not compile-checked)
```scala
import org.llm4s.llmconnect.caching.CacheConfig
import scala.concurrent.duration._
CacheConfig.create(
similarityThreshold = 0.95, // cosine similarity, 0.0 to 1.0
ttl = 5.minutes, // an older entry is ignored
maxSize = 100 // least recently used entry is evicted beyond this
)
```
| Field | Type | Meaning |
|---|---|---|
| `similarityThreshold` | `Double`, 0.0 to 1.0 inclusive (`NaN` is rejected) | the minimum cosine similarity between the new prompt's embedding and a stored one for a hit |
| `ttl` | `FiniteDuration`, positive | an entry older than this is ignored |
| `maxSize` | `Int`, positive, default `1000` | the number of entries kept; beyond it the least recently used is evicted |
`create` reports every violated constraint at once, in the message of the `ValidationError` it returns. There is no public constructor: `CacheConfig(...)` without `create` does not compile.
### The caching client
`CachingLLMClient` takes the client to wrap, an `EmbeddingClient` and the embedding model it should use, the `CacheConfig`, and a `Tracing`. This is the shape the [`SemanticCachingSample`](https://github.com/llm4s/llm4s/blob/main/modules/samples/src/main/scala/org/llm4s/samples/cache/SemanticCachingSample.scala) uses:
(not compile-checked)
```scala
import org.llm4s.llmconnect.caching.CachingLLMClient
import org.llm4s.llmconnect.config.EmbeddingModelConfig
new CachingLLMClient(
baseClient = baseClient,
embeddingClient = embeddingClient,
embeddingModel = EmbeddingModelConfig("text-embedding-3-small", 1536),
config = cacheConfig,
tracing = tracing
)
```
It is an `LLMClient`, so it goes anywhere the client you wrap would go. The sample shows it end to end against a real provider: it needs an OpenAI key (see [running the samples](../getting-started/configuration#running-the-samples)), then
```
sbt "samples/runMain org.llm4s.samples.cache.SemanticCachingSample"
```
### How a lookup works
On each `complete(conversation, options)`:
1. The cache builds a prompt from the **user and system messages** only, as lines of `role: content`, and embeds it with your `EmbeddingClient`. That is one embedding call, and the prompt text goes to your embedding provider.
2. It compares that embedding with every stored one by cosine similarity. A stored entry is a candidate when its similarity is **at or above** `similarityThreshold`.
3. A candidate is a hit when its `CompletionOptions` are **equal** to the request's (temperature, tools and the rest) and it is **within the TTL**. If several qualify, the most similar wins.
4. On a hit the stored `Completion` is returned and the entry counts as recently used. It is the same object each time, so do not mutate it.
5. On a miss the base client is called, a successful result is stored, and it is returned.
### What is, and is not, compared
Only user and system messages are embedded. Assistant messages and tool messages are left out on purpose, so that tool arguments and tool results never reach the embedding provider. The consequence is that **two conversations that differ only in their assistant or tool turns look identical to the cache**:
```text
[user: "what is 2+2", assistant: "thinking", tool: "4"]
[user: "what is 2+2", assistant: "other", tool: "5"] -> served the first one's answer
```
For a multi-turn agent run this can return an answer that does not match the tool results in front of it. Cache the single-shot, self-contained calls, not the steps of an agent loop.
### Choosing the threshold
A high threshold gives few false hits and few hits; a low one the opposite. The right value depends on the embedding model and on your questions, so measure it on real queries. The sample uses `0.95`.
Do not use `1.0` to mean "exact match". Similarity is computed in floating point, and even an identical prompt can come out a hair below `1.0`: with an embedding of `(0.3, 0.3, 0.3)` the same prompt asked twice was a miss at a threshold of `1.0`. If you want exact matching, use the exact embedding cache above, or a threshold just under `1.0`.
### TTL, size and memory
- An entry older than `ttl` is skipped (one exactly `ttl` old is still used, as in the embedding cache), but it is **not removed**. It stays in memory until the least-recently-used rule evicts it, so `maxSize`, not `ttl`, is what bounds memory.
- The lookup compares the new embedding with every stored entry, so its cost grows with `maxSize`. Keep it modest.
- Use the `clock` constructor parameter to control time in tests. It is only consulted for the TTL:
(not compile-checked)
```scala
import org.llm4s.llmconnect.caching.CachingLLMClient
import org.llm4s.llmconnect.config.EmbeddingModelConfig
// The clock only decides whether an entry is still within its TTL. Tests pass a fixed, movable one.
new CachingLLMClient(
baseClient,
embeddingClient,
EmbeddingModelConfig("text-embedding-3-small", 1536),
cacheConfig,
tracing,
clock
)
```
### What is never cached
- **Streaming.** `streamComplete` goes straight to the base client and neither reads nor fills the cache.
- **Failures.** If the base client returns an error, the error is returned and nothing is stored.
- **When the embedding call fails.** If embedding the prompt fails or returns nothing, the request goes to the base client uncached. The only trace of it is a warning in the log: no trace event is emitted, so a cache whose embedding service is down looks like a cache that always misses.
### Sharing, privacy and threads
- The entries belong to that `CachingLLMClient` instance. Everyone who uses the instance shares them; nothing is keyed by user or tenant. Do not share one instance between users whose answers must not mix, and give each its own.
- The entries live in memory, and a restart empties the cache.
- It is safe to call from several threads. The model calls themselves are not serialised, and like the embedding cache it does not coalesce: several threads asking the same new question at once all call the model (four concurrent identical requests made four model calls).
- The completion cache has no way to inspect or clear it. To start again, create a new instance.
## 5. Seeing what the cache did
The completion cache reports each decision to the `Tracing` you give it. A hit sends `TraceEvent.CacheHit(similarity, threshold, timestamp)`; a miss sends `TraceEvent.CacheMiss(reason, timestamp)`, where the reason says why:
| `reason` | `reason.value` | Meaning |
|---|---|---|
| `LowSimilarity` | `low_similarity` | no stored entry reached the threshold (this includes an empty cache) |
| `TtlExpired` | `ttl_expired` | entries reached the threshold, and some had the same options, but none was within the TTL |
| `OptionsMismatch` | `options_mismatch` | entries reached the threshold, but none had the same `CompletionOptions` |
A `Tracing` that records them, and a way to print one:
(not compile-checked)
```scala
import org.llm4s.llmconnect.model.{ Completion, TokenUsage }
import org.llm4s.trace.{ TraceEvent, Tracing }
import org.llm4s.types.Result
import scala.collection.mutable.ListBuffer
def collectingTracing(): (Tracing, () => List[TraceEvent]) = {
val events = ListBuffer.empty[TraceEvent]
val tracing = new Tracing {
override def traceEvent(event: TraceEvent): Result[Unit] = { events.synchronized(events += event); Right(()) }
override def traceToolCall(toolName: String, input: String, output: String): Result[Unit] = Right(())
override def traceError(error: Throwable, context: String): Result[Unit] = Right(())
override def traceCompletion(completion: Completion, model: String): Result[Unit] = Right(())
override def traceTokenUsage(usage: TokenUsage, model: String, operation: String): Result[Unit] = Right(())
}
(tracing, () => events.synchronized(events.toList))
}
def describe(event: TraceEvent): String = event match {
case TraceEvent.CacheHit(similarity, threshold, _) => f"hit, similarity $similarity%.3f >= $threshold%.3f"
case TraceEvent.CacheMiss(reason, _) => s"miss: ${reason.value}"
case other => other.eventType
}
```
Asking the same question twice then gives `miss: low_similarity` followed by `hit, similarity 1.000 >= 0.900` (for a threshold of `0.900`). The bundled tracing backends already handle both events: `ConsoleTracing` prints `CACHE HIT` (with the similarity and threshold) and `CACHE MISS` (with the reason), the Langfuse backend records a `Cache Hit` or `Cache Miss` span with the same values, and the OpenTelemetry backend records a `Cache Hit` or `Cache Miss` event with `similarity`, `threshold` and `reason` attributes.
The embedding cache has no trace events; read its `cacheStats` instead (section 3).
## 6. Where caching does not belong
- Calls that should return a fresh, varied or personalised answer.
- Calls with side effects, such as the steps of an agent that runs tools.
- Anything where a wrong answer from a near-miss prompt costs more than the model call saved.
- Several users on one `CachingLLMClient`, when their answers must stay separate.
## See also
- [Error Handling](error-handling): the `Result` and `LLMError` types every call here returns.
- [Vector Store](vector-store): where the vectors of an indexed corpus live, which a cache does not replace.
- [`SemanticCachingSample`](https://github.com/llm4s/llm4s/blob/main/modules/samples/src/main/scala/org/llm4s/samples/cache/SemanticCachingSample.scala): the cache against a real provider.
---
Source: https://llm4s.org/guide/json-libraries
# Using LLM4S with circe, play-json and zio-json
LLM4S reads and writes JSON with uPickle and its `ujson` AST. This page shows how an application that already uses circe, play-json or zio-json passes its own types in and gets them back, without writing a second set of codecs.
---
## Where ujson shows in the API
Three places are where your JSON library meets LLM4S:
| Where | What LLM4S takes or returns |
|---|---|
| `LLMClient.completeStructured[A]` | an implicit `upickle.default.Reader[A]` for the type you want back |
| Tool handlers and results | `SafeParameterExtractor.params` is a `ujson.Value`; `ToolFunction.execute` takes and returns a `ujson.Value`; a tool's result type needs a uPickle `ReadWriter` (use `ujson.Value` itself to return a document you built) |
| Agent interrupts and events | `Agent.streamResume` takes `Map[InterruptId, ujson.Value]`; `AgentResult.edit` takes the edited `ujson.Value`; the `ToolCallStarted` event carries its `arguments` as a `ujson.Value` |
So there are two jobs: turn a `ujson.Value` into your library's document (and back), and turn your library's decoder into the `Reader` that `completeStructured` asks for.
The examples use these types: a nested object, an `Option`, a collection and the three numeric kinds.
(not compile-checked)
```scala
final case class Address(city: String, zip: Option[String])
final case class Order(id: Int, total: Double, address: Address, tags: List[String], note: Option[String])
val addressSchema: ObjectSchema[Address] = Schema
.`object`[Address]("Where to deliver")
.withRequiredField("city", Schema.string("The city"))
.withProperty(PropertyDefinition("zip", Schema.string("The postal code"), required = false))
val orderSchema: ObjectSchema[Order] = Schema
.`object`[Order]("An order")
.withRequiredField("id", Schema.integer("The order number"))
.withRequiredField("total", Schema.number("The total"))
.withRequiredField("address", addressSchema)
.withRequiredField("tags", Schema.array("Labels", Schema.string("A label")))
.withProperty(PropertyDefinition("note", Schema.string("A note"), required = false))
```
## What to add
Nothing here goes into LLM4S itself; add the library you use to your own build. uPickle's `ujson-circe` is the bridge for circe. **There is no Scala 3 build of `ujson-play`** (on Maven Central it exists for Scala 2.11 to 2.13 only, checked on 2026-10-08), and zio-json has no bridge at all, so those two cross as text. The versions below are the ones this page was run with.
```sbt
// circe: ujson-circe is uPickle's own bridge, built for Scala 3. Keep its version equal to the ujson that LLM4S uses.
libraryDependencies ++= Seq(
"io.circe" %% "circe-core" % "0.14.17",
"com.lihaoyi" %% "ujson-circe" % "4.4.3"
)
// play-json (no bridge needed: it crosses as text)
libraryDependencies += "org.playframework" %% "play-json" % "3.0.6"
// zio-json (no bridge needed: it crosses as text)
libraryDependencies += "dev.zio" %% "zio-json" % "1.0.0"
```
**zio-json 1.1.0 needs a Scala 3.9 compiler.** It depends on `scala3-library` 3.9.0, and a project compiled with Scala 3.7.1 (the version LLM4S is built with) cannot read that library: the compiler stops with "Forward incompatible TASTy file ... produced by Scala 3.9.0". Use 1.0.0 (built against Scala 3.3) unless your project is on Scala 3.9 or later. zio-json 1.0.0 also needs zio 2.1.26 or newer, which sbt resolves for you.
## circe
### Converting a document
`ujson.transform` visits a `ujson.Value` with circe's visitor and builds a `Json`, and `CirceJson.transform` goes the other way. Nothing is rendered to text on the way.
(not compile-checked)
```scala
import io.circe.Json
import ujson.circe.CirceJson
def toCirce(value: ujson.Value): Json = ujson.transform(value, CirceJson)
def fromCirce(json: Json): ujson.Value = CirceJson.transform(json, ujson.Value)
```
### Using your Decoder with completeStructured
`completeStructured` wants a uPickle `Reader[A]`. A `Reader` can be built from the `ujson.Value` reader, so any circe `Decoder` becomes one. Here the codecs come from circe's own derivation:
(not compile-checked)
```scala
import io.circe.{ Decoder, Encoder }
import io.circe.syntax._
given Decoder[Address] = Decoder.derived
given Decoder[Order] = Decoder.derived
given Encoder[Address] = Encoder.AsObject.derived
given Encoder[Order] = Encoder.AsObject.derived
```
(not compile-checked)
```scala
import upickle.default.{ reader, Reader }
def circeReader[A](using decoder: Decoder[A]): Reader[A] =
reader[ujson.Value].map { value =>
decoder.decodeJson(toCirce(value)).fold(error => throw upickle.core.Abort(error.getMessage), identity)
}
```
Then pass it where the `Reader` goes:
(not compile-checked)
```scala
def orderViaCirce(client: LLMClient, conversation: Conversation): Result[Order] =
client.completeStructured[Order](conversation, orderSchema)(using circeReader[Order])
```
`Option` fields can be `null` or absent, and nested objects decode as circe decodes them. The schema you pass to `completeStructured` is still an LLM4S `ObjectSchema` (circe does not derive it; deriving one from a case class is tracked in [#1472](https://github.com/llm4s/llm4s/issues/1472)), and it reaches the provider exactly as with a uPickle `Reader`.
**When decoding fails.** The `Reader` throws `upickle.core.Abort` with the decoder's message. `completeStructured` catches anything thrown while reading and returns `Left(ValidationError("structured_output", "Response does not match expected schema: ..."))`, the same error the uPickle path gives, so your error handling does not change (see [Error Handling](error-handling)). A reply that is JSON `null` is refused and never comes back as `Right(null)`.
### Tools that use circe
A tool handler receives its arguments as `extractor.params`, a `ujson.Value`. Decode it with circe, and return a `ujson.Value` built from your result's encoder:
(not compile-checked)
```scala
final case class AddArgs(a: Int, b: Int)
final case class AddResult(sum: Long)
given Decoder[AddArgs] = Decoder.derived
given Encoder[AddResult] = Encoder.AsObject.derived
val addSchema = Schema
.`object`[Map[String, Any]]("Two integers")
.withRequiredField("a", Schema.integer("The first"))
.withRequiredField("b", Schema.integer("The second"))
val addTool = ToolBuilder[Map[String, Any], ujson.Value]("add", "Adds two integers", addSchema)
.withHandler { extractor =>
toCirce(extractor.params).as[AddArgs] match {
case Right(args) => Right(fromCirce(AddResult(args.a.toLong + args.b).asJson))
case Left(error) => Left(error.getMessage)
}
}
.buildSafe()
```
A value that does not fit the case class (`"a": "two"`, or a missing field) is returned to the model as a tool error, not thrown. See [Built-in Tools](builtin-tools) for how tool errors reach the model.
### Answers and edits to an interrupted run
Wherever LLM4S takes a `ujson.Value` (the `answers` of `Agent.streamResume`, the `arguments` of `AgentResult.edit`), build it from a circe `Json` with the same conversion:
(not compile-checked)
```scala
val answer: ujson.Value = fromCirce(Json.obj("approved" -> Json.fromBoolean(true)))
```
## play-json
play-json reads and writes text, so the document crosses as a string: `value.render()` is compact JSON, and `PlayJson.parse` reads it. This costs one copy of the document each way; it is fine for tool arguments and structured answers, which are small.
(not compile-checked)
```scala
import play.api.libs.json.{ JsError, JsSuccess, Json as PlayJson, JsValue, Reads }
def toPlay(value: ujson.Value): JsValue = PlayJson.parse(value.render())
def fromPlay(json: JsValue): ujson.Value = ujson.read(PlayJson.stringify(json))
def playReader[A](using reads: Reads[A]): Reader[A] =
reader[ujson.Value].map { value =>
reads.reads(toPlay(value)) match {
case JsSuccess(decoded, _) => decoded
case JsError(errors) => throw upickle.core.Abort(errors.toString)
}
}
given Reads[Address] = PlayJson.reads[Address]
given Reads[Order] = PlayJson.reads[Order]
```
(not compile-checked)
```scala
def orderViaPlay(client: LLMClient, conversation: Conversation): Result[Order] =
client.completeStructured[Order](conversation, orderSchema)(using playReader[Order])
```
A document that does not fit the `Reads` gives the same `ValidationError("structured_output", ...)` as above.
## zio-json
zio-json decodes straight from text, so the `Reader` renders the document once and decodes it. To hand a zio-json value to LLM4S, encode it to text and read it back as a `ujson.Value`. Note that zio-json leaves a `None` field out of the document it writes, where circe writes `null`; every one of the three libraries reads both forms back as `None`, so a model may answer either way.
(not compile-checked)
```scala
import zio.json.{ DeriveJsonDecoder, DeriveJsonEncoder, JsonDecoder, JsonEncoder }
import zio.json.{ DecoderOps, EncoderOps }
given JsonDecoder[Address] = DeriveJsonDecoder.gen[Address]
given JsonDecoder[Order] = DeriveJsonDecoder.gen[Order]
given JsonEncoder[Address] = DeriveJsonEncoder.gen[Address]
given JsonEncoder[Order] = DeriveJsonEncoder.gen[Order]
def zioReader[A](using decoder: JsonDecoder[A]): Reader[A] =
reader[ujson.Value].map { value =>
value.render().fromJson[A].fold(message => throw upickle.core.Abort(message), identity)
}
def fromZio[A](value: A)(using encoder: JsonEncoder[A]): ujson.Value = ujson.read(value.toJson)
```
(not compile-checked)
```scala
def orderViaZio(client: LLMClient, conversation: Conversation): Result[Order] =
client.completeStructured[Order](conversation, orderSchema)(using zioReader[Order])
```
## Numbers
A `ujson` number is a double. Doubles and ordinary integers (up to 2^53) round trip unchanged, but **a whole number above 2^53 is rounded to the nearest double as soon as it is parsed into the `ujson` AST**. That is `ujson`'s own parse, so it is not a property of circe, play-json or zio-json:
(not compile-checked)
```scala
val big = 9007199254740993L // 2^53 + 1: the first integer a double cannot hold
val parsed: Long = ujson.read(s"""{"id":$big}""")("id").num.toLong
```
The first integer a double cannot hold, 2^53 + 1 (9007199254740993), comes back as 9007199254740992.
What this means for structured output depends on the reader:
- **uPickle readers are exact.** `completeStructured` reads `A` from the reply's text, so a uPickle `Long` field receives 9007199254740993 unchanged.
- **The bridges on this page round.** `circeReader`, `playReader` and `zioReader` start from `reader[ujson.Value]`, so a circe `Decoder[Long]` (and its play-json and zio-json counterparts) reads the already-rounded value.
- **Tool arguments round too.** A tool call's arguments reach a handler as a `ujson.Value`. `SafeParameterExtractor.getInt` refuses any number outside the `Int` range, so a typed integer parameter is never silently wrong; a large number read from the raw JSON, though, has already been rounded.
So if a model has to return or receive large identifiers (snowflake ids, database keys above 9 quadrillion) through a bridge or a tool, make them strings in the schema and the case class, and convert in your own code. See [#1597](https://github.com/llm4s/llm4s/issues/1597).
## What this page does not cover
- **A bridge module.** No `llm4s-circe` module is published. The conversion is two lines and the `Reader` is five, as shown above, so a module would add a dependency and a maintenance promise for little. Its main value would be one place for the error mapping, and only circe has a Scala 3 bridge, so a module could not treat the three libraries alike. If you would use one, say so on [#1458](https://github.com/llm4s/llm4s/issues/1458).
- **Deriving tool or response schemas** from your case classes. The schemas are still built with LLM4S's `Schema` (issue [#1472](https://github.com/llm4s/llm4s/issues/1472)).
- **Streaming events.** The event fields that are `ujson.Value`s, such as the `arguments` of `ToolCallStarted`, convert the same way.
## How this page is checked
Every Scala block on this page is a snippet of `JsonLibrariesGuideSpec` in the `llm4s-samples` tests, which compiles and runs them against the real libraries at the versions in the `sbt` block above, using a scripted client in place of a model. The same spec fails if a block here differs from its snippet or a version here differs from the build's.
---
Source: https://llm4s.org/guide/rag-evaluation
# RAG Evaluation & Benchmarking
Systematically measure and improve your RAG pipeline quality using RAGAS metrics and the benchmarking harness.
---
## Overview
Building a RAG (Retrieval-Augmented Generation) system is just the beginning. The real challenge is knowing whether your retrieval is actually helping your LLM produce better answers. LLM4S provides two complementary tools:
1. **RAGAS Evaluator** - Measure retrieval and generation quality with industry-standard metrics
2. **Benchmarking Harness** - Systematically compare different RAG configurations
**Key Benefits:**
- Data-driven optimization of chunking strategies
- Objective comparison of fusion methods (RRF vs weighted vs vector-only)
- Embedding provider evaluation
- Reproducible experiments with JSON reports
---
## RAGAS Metrics Explained
[RAGAS](https://docs.ragas.io/) (Retrieval Augmented Generation Assessment) provides four core metrics:
### Faithfulness (0.0 - 1.0)
Measures whether the generated answer is grounded in the retrieved context.
- **1.0**: Answer fully supported by context - no hallucination
- **0.0**: Answer contains claims not in the retrieved documents
```
Question: "What year was Scala released?"
Context: "Scala was first released in 2004."
Answer: "Scala was released in 2004."
Faithfulness: 1.0 (fully grounded)
```
### Answer Relevancy (0.0 - 1.0)
Measures how relevant the answer is to the question asked.
- **1.0**: Answer directly addresses the question
- **0.0**: Answer is completely off-topic
```
Question: "What is Scala?"
Answer: "Scala is a programming language."
Relevancy: ~0.85 (relevant but could be more complete)
```
### Context Precision (0.0 - 1.0)
Measures whether the retrieved chunks are relevant to answering the question.
- **1.0**: All retrieved chunks contribute to the answer
- **0.0**: None of the retrieved chunks are relevant
*High precision = no wasted context tokens*
### Context Recall (0.0 - 1.0)
Measures whether all information needed to answer was retrieved.
- **1.0**: All necessary information was retrieved
- **0.0**: Critical information was missed
*High recall = comprehensive retrieval*
### Overall RAGAS Score
The harmonic mean of all metrics, providing a single quality score. This is what the benchmarking harness uses for ranking.
---
## Quick Start: Evaluate Your RAG
### Basic Evaluation
(not compile-checked)
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.rag.evaluation._
import org.llm4s.llmconnect.LLMConnect
// Initialize
val llm = Llm4sConfig
.provider()
.flatMap(LLMConnect.getClient)
.fold(_ => ???, identity)
val evaluator = new RAGASEvaluator(llm)
// Create an evaluation sample
val sample = EvalSample(
question = "What is Scala?",
answer = "Scala is a programming language that runs on the JVM.",
contexts = Seq("Scala is a functional and object-oriented programming language."),
groundTruth = Some("Scala is a programming language.")
)
// Evaluate
val result = evaluator.evaluate(sample)
result.foreach { metrics =>
println(s"Faithfulness: ${metrics.faithfulness}")
println(s"Answer Relevancy: ${metrics.answerRelevancy}")
println(s"Context Precision: ${metrics.contextPrecision}")
println(s"Context Recall: ${metrics.contextRecall}")
println(s"Overall RAGAS: ${metrics.ragasScore}")
}
```
### Batch Evaluation
(not compile-checked)
```scala
val samples = Seq(sample1, sample2, sample3)
val results = evaluator.evaluateBatch(samples)
results.foreach { allMetrics =>
val avgRagas = allMetrics.map(_.ragasScore).sum / allMetrics.size
println(s"Average RAGAS Score: $avgRagas")
}
```
---
## Benchmarking Harness
The benchmarking harness lets you systematically compare different RAG configurations to find what works best for your data.
### Running Pre-Built Benchmarks
```bash
# Download test dataset
./scripts/download-datasets.sh ragbench
# Configure providers (see the note below), then:
export OPENAI_API_KEY=sk-...
export LLM4S_PROVIDER=openai-main # bound by the samples' application.conf
export EMBEDDING_MODEL=openai/text-embedding-3-small # bound by llm4s-core's reference.conf
# Run fusion strategy comparison
sbt "samples/runMain org.llm4s.samples.rag.BenchmarkExample --suite fusion"
# Run chunking strategy comparison
sbt "samples/runMain org.llm4s.samples.rag.BenchmarkExample --suite chunking"
# Run embedding provider comparison
sbt "samples/runMain org.llm4s.samples.rag.BenchmarkExample --suite embedding"
# Quick test (5 samples)
sbt "samples/runMain org.llm4s.samples.rag.BenchmarkExample --quick"
```
The samples read `modules/samples/src/main/resources/application.conf`, whose only provider
section is `ollama-local`, so add an OpenAI chat section to
`modules/samples/src/main/resources/application.local.conf` (not committed; included by
`application.conf`). Neither it nor OpenAI embeddings need an `apiKey` line: both use
`OPENAI_API_KEY`, which `llm4s-openai` binds to OpenAI's shared key.
```hocon
llm4s {
providers {
openai-main {
provider = "openai"
model = "gpt-4o"
}
}
}
```
See [Running the samples](../getting-started/configuration#running-the-samples).
### Understanding the Output
```
======================================================================
BENCHMARK RESULTS: fusion-comparison
Compare different search fusion strategies
======================================================================
Duration: 327.83s
Experiments: 6 (6 passed, 0 failed)
RANKINGS BY RAGAS SCORE
----------------------------------------------------------------------
Rank Config RAGAS Faith Relevancy
----------------------------------------------------------------------
1 rrf-20 0.911 1.000 0.844
2 rrf-60 0.894 1.000 0.841
3 weighted-70v 0.893 1.000 0.841
4 weighted-50v 0.892 1.000 0.836
5 vector-only 0.884 0.960 0.845
6 keyword-only 0.809 N/A 0.809
======================================================================
WINNER: rrf-20 (RAGAS: 0.911)
======================================================================
```
---
## What Each Benchmark Tests
### Fusion Strategy Benchmark
Compares how to combine vector (semantic) and keyword (BM25) search results.
| Strategy | Description | When to Use |
|----------|-------------|-------------|
| `rrf-20` | Reciprocal Rank Fusion, k=20 | Best overall performance |
| `rrf-60` | Reciprocal Rank Fusion, k=60 | More conservative ranking |
| `weighted-70v` | 70% vector + 30% keyword | Semantic-focused |
| `weighted-50v` | 50% vector + 50% keyword | Balanced |
| `vector-only` | Pure embedding similarity | Conceptual queries |
| `keyword-only` | Pure BM25 | Exact term matching |
**Typical Finding:** Hybrid search (RRF) outperforms single-method search by 3-5%.
### Chunking Strategy Benchmark
Compares different document splitting approaches.
| Strategy | Target Size | Best For |
|----------|-------------|----------|
| `sentence-small` | 400 chars | Focused factual queries |
| `sentence-default` | 800 chars | General use |
| `sentence-large` | 1200 chars | Complex reasoning |
| `markdown-default` | 800 chars | Structured documents |
| `simple-default` | 800 chars | Baseline comparison |
**Typical Finding:** Sentence-aware chunking with smaller chunks improves precision.
### Embedding Provider Benchmark
Compares embedding quality across providers.
| Provider | Model | Dimensions | Notes |
|----------|-------|------------|-------|
| OpenAI | text-embedding-3-small | 1536 | Good balance of cost/quality |
| OpenAI | text-embedding-3-large | 3072 | Highest quality |
| Voyage | voyage-3 | 1024 | Optimized for retrieval |
| Ollama | nomic-embed-text | 768 | Free, local |
---
## Programmatic Benchmarking
### Custom Benchmark Suite
(not compile-checked)
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.rag.benchmark._
import org.llm4s.llmconnect.{ EmbeddingClient, LLMConnect }
// Initialize clients
val llm = Llm4sConfig
.provider()
.flatMap(LLMConnect.getClient)
.fold(_ => ???, identity)
val embed = Llm4sConfig
.embeddings()
.flatMap { case (provider, cfg) => EmbeddingClient.from(provider, cfg) }
.fold(_ => ???, identity)
val runner = new BenchmarkRunner(llm, embed)
// Define custom experiments
val experiments = Seq(
RAGExperimentConfig(
name = "my-config-1",
chunkingStrategy = ChunkerFactory.Strategy.Sentence,
fusionStrategy = FusionStrategy.RRF(30),
topK = 5
),
RAGExperimentConfig(
name = "my-config-2",
chunkingStrategy = ChunkerFactory.Strategy.Markdown,
fusionStrategy = FusionStrategy.WeightedScore(0.8, 0.2),
topK = 10
)
)
// Create suite
val suite = BenchmarkSuite(
name = "my-comparison",
description = "Custom experiment comparison",
experiments = experiments,
datasetPath = "path/to/dataset.jsonl"
)
// Run and report
val results = runner.runSuite(suite)
println(BenchmarkReport.console(results))
BenchmarkReport.save(results, "data/results/my-comparison.json") // returns Result[Unit]
```
### Custom Dataset Format
Create a JSONL file with your own test data:
```jsonl
{"question": "What is Scala?", "documents": ["Scala is a programming language..."], "answer": "Scala is a JVM language."}
{"question": "How does pattern matching work?", "documents": ["Pattern matching in Scala..."], "answer": "Pattern matching uses case expressions."}
```
---
## Improving Your RAG with Benchmarks
### Optimization Workflow
```
1. BASELINE
Run benchmark with default settings
Record baseline RAGAS score
2. HYPOTHESIS
"Smaller chunks might improve precision"
3. EXPERIMENT
Run chunking benchmark
Compare sentence-small vs sentence-default
4. ANALYZE
Did RAGAS score improve?
Which sub-metrics changed?
5. ITERATE
Apply winning config
Test new hypothesis
```
### Common Optimization Patterns
#### Low Faithfulness Score
**Problem:** LLM generating claims not in context.
**Solutions:**
- Increase `topK` to retrieve more context
- Switch to larger chunk sizes
- Add reranking to improve context quality
- Use more explicit RAG prompts
#### Low Context Precision
**Problem:** Retrieving irrelevant chunks.
**Solutions:**
- Try smaller chunk sizes for tighter matches
- Use hybrid search (RRF) to combine semantic + keyword
- Add metadata filtering
- Tune embedding model
#### Low Context Recall
**Problem:** Missing relevant information.
**Solutions:**
- Increase `topK` retrieval count
- Use larger chunk overlap
- Add query expansion
- Try different embedding models
#### Low Answer Relevancy
**Problem:** Answers don't address the question directly.
**Solutions:**
- Improve RAG prompt template
- Use query rewriting
- Check if context contains the answer
---
## Interpreting Results
### What's a Good RAGAS Score?
| Score Range | Interpretation |
|-------------|----------------|
| 0.90+ | Excellent - production ready |
| 0.80-0.90 | Good - suitable for most use cases |
| 0.70-0.80 | Acceptable - room for improvement |
| 0.60-0.70 | Fair - needs optimization |
| <0.60 | Poor - significant issues |
### Comparing Configurations
When comparing configurations:
1. **Statistical Significance**: Small differences (<0.02) may be noise
2. **Trade-offs**: Higher precision often means lower recall
3. **Domain Specificity**: Best config varies by data type
4. **Cost Considerations**: More retrieval = more embedding calls
---
## Benchmark Results Reference
Based on actual benchmarks with RAGBench dataset:
### Fusion Strategy Results
| Config | RAGAS | Faithfulness | Relevancy |
|--------|-------|--------------|-----------|
| rrf-20 | 0.911 | 1.000 | 0.844 |
| rrf-60 | 0.894 | 1.000 | 0.841 |
| weighted-70v | 0.893 | 1.000 | 0.841 |
| weighted-50v | 0.892 | 1.000 | 0.836 |
| vector-only | 0.884 | 0.960 | 0.845 |
| keyword-only | 0.809 | N/A | 0.809 |
**Key Insight:** Hybrid search (RRF) outperforms pure vector or keyword search.
### Chunking Strategy Results
| Config | RAGAS | Faithfulness | Relevancy |
|--------|-------|--------------|-----------|
| sentence-small | 0.910 | 1.000 | 0.840 |
| sentence-large | 0.893 | 1.000 | 0.838 |
| sentence-default | 0.892 | 1.000 | 0.834 |
| markdown-default | 0.885 | 0.960 | 0.847 |
| simple-default | 0.882 | 0.960 | 0.833 |
**Key Insight:** Smaller, sentence-aware chunks perform best for focused queries.
---
## Best Practices
### 1. Start with Baselines
Always run benchmarks with default settings first to establish a baseline.
### 2. Change One Variable at a Time
When optimizing, change only one configuration at a time to understand impact.
### 3. Use Representative Data
Your benchmark dataset should reflect real-world queries your system will handle.
### 4. Consider the Full Picture
Don't optimize for one metric at the expense of others. Balance is key.
### 5. Monitor in Production
Benchmark scores are offline metrics. Monitor real user satisfaction in production.
### 6. Re-benchmark After Changes
When you update your document corpus, re-run benchmarks to verify quality.
---
## API Reference
### RAGASEvaluator
(not compile-checked)
```scala
class RAGASEvaluator(llmClient: LLMClient) {
def evaluate(sample: EvalSample): Result[RAGASMetrics]
def evaluateBatch(samples: Seq[EvalSample]): Result[Seq[RAGASMetrics]]
}
```
### EvalSample
(not compile-checked)
```scala
final case class EvalSample(
question: String,
answer: String,
contexts: Seq[String],
groundTruth: Option[String] = None
)
```
### RAGASMetrics
(not compile-checked)
```scala
final case class RAGASMetrics(
faithfulness: Option[Double],
answerRelevancy: Option[Double],
contextPrecision: Option[Double],
contextRecall: Option[Double]
) {
def ragasScore: Double // Harmonic mean of available metrics
}
```
### BenchmarkRunner
(not compile-checked)
```scala
class BenchmarkRunner(llmClient: LLMClient, embeddingClient: EmbeddingClient) {
def runSuite(suite: BenchmarkSuite): BenchmarkResults
def runExperiment(config: RAGExperimentConfig, dataset: Dataset): ExperimentResult
}
```
---
## Next Steps
- **[Vector Store Guide](vector-store)** - Configure vector backends
- **[Benchmark Results](../rag-benchmark-results.md)** - Detailed benchmark data
- **[Examples Gallery](/examples/)** - RAG implementation examples
---
Source: https://llm4s.org/guide/java
# Using LLM4S from Java
Call a model from Java with `llm4s-java-api`: a client, a prompt, a conversation, and failures you can read without Scala.
---
## What `llm4s-java-api` is
LLM4S is written in Scala. Its core API returns Scala types (`Either`, `Option`, `Seq`) that are awkward to use from Java.
`llm4s-java-api` is a small layer in front of it for Java callers: `Llm4s` creates a client, `JLlmClient` calls the
model, `ConversationBuilder` builds a conversation, and every call returns an `LlmResult`, a result type modelled on
`java.util.Optional` and `CompletableFuture`. A failed call is a value you check, not an exception you have to catch.
It does not hide every Scala type. `Conversation` and `ProviderConfig` are Scala classes that still appear in its
signatures, as does `LLMError` behind `LlmException.error()`, though [its kind](#handling-a-failure) reads as a Java
enum, and so does core's `CompletionOptions`, in an overload that [`JCompletionOptions`](#completion-options)
makes unnecessary. [What is not here yet](#what-is-not-here-yet) says which of them gets in
your way.
The values it hands you to read - `JCompletion`, `JAgentResult`, `JMessage`, `PendingInterrupt`, `JEmbeddings` and the
rest - are Java classes whose constructors are package-private: you receive them, you do not build them, so a later
release can add a field without breaking code compiled against this one. The classes that do the work come from a
factory: `Llm4s` makes clients and agents, `JCompletionOptions.builder()` and `ConversationBuilder.create()` start a
builder, `LlmResult.success` and `failure` make a result, and the `Answer` factories an answer. Their constructors are
not for you either: each takes a first parameter of a type only the package can name, and throws
`UnsupportedOperationException` when called from anywhere else. To call a Scala `LLMClient` you built yourself - a
`ReliableClient` around a provider's client, your own implementation, a test double - wrap it with
`Llm4s.wrapClient(client)`; a Scala `EmbeddingClient`, with `Llm4s.wrapEmbeddingClient(client, model)`.
> `llm4s-java-api` is on Maven Central from `0.5.0` on; the current release is 0.5.1.
> The [`gradle-java` sample](https://github.com/llm4s/llm4s/tree/main/modules/samples/gradle-java) is a complete Java
> project you can copy. It needs **JDK 21**: it depends on `llm4s-agent`, which is compiled for JDK 21 and starts
> virtual threads (see [Scala and JDK support](/reference/v1-scope#scala-and-jdk-support)). Whether some artifacts get a
> lower floor before 1.0 is tracked in [#1493](https://github.com/llm4s/llm4s/issues/1493).
## Add the dependency
The artifact is `org.llm4s:llm4s-java-api_3`. The `_3` is the Scala binary version, and LLM4S is Scala 3 only. sbt adds
it for you (`%%`); Maven and Gradle do not, so write it yourself.
**Maven**
```xml
org.llm4s
llm4s-java-api_3
0.5.1
```
**Gradle (Kotlin DSL)**
```kotlin
dependencies {
implementation("org.llm4s:llm4s-java-api_3:0.5.1")
}
```
**sbt**
(not compile-checked)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-java-api" % "0.5.1"
```
What arrives with it:
- `llm4s-core`, `llm4s-agent` and the OpenAI, Anthropic, Ollama, Gemini and OpenAI-compatible provider modules, so you do
not add a provider artifact. Making providers opt-in is tracked in
[#1496](https://github.com/llm4s/llm4s/issues/1496).
- Scala's standard library (`scala3-library_3` and `scala-library`). If your build pins or excludes it, see the
[Gradle guide](../getting-started/gradle) for the recipes that keep the two Scala libraries aligned.
- No logging backend. LLM4S logs through SLF4J; add one (for example Logback), or SLF4J prints a warning and drops the
logs.
## Configure a provider
Java code does not build a provider configuration. It comes from an `application.conf` on your classpath, the same file
Scala users write: a named section per provider, and one section named as the default.
```hocon
# src/main/resources/application.conf
llm4s {
providers {
provider = "openai-main" # the default: the name of a section below
openai-main {
provider = "openai"
model = "gpt-4o-mini"
}
}
}
```
With `OPENAI_API_KEY` set in the environment, that is the whole configuration: the OpenAI module binds the variable to
the shared credential for `openai`, so the section needs only `provider` and `model`. Other providers work the same way;
`anthropic` reads `ANTHROPIC_API_KEY`, and `ollama` needs no key but does need a `baseUrl`:
```hocon
llm4s {
providers {
provider = "claude"
claude { # key from ANTHROPIC_API_KEY
provider = "anthropic"
model = "claude-sonnet-4-20250514"
}
ollama-local {
provider = "ollama"
model = "llama3.2"
baseUrl = "http://localhost:11434"
}
}
}
```
The [configuration guide](../getting-started/configuration) lists every key and every provider. Nothing in the library
reads a `LLM_MODEL` variable; the model is the `model` key of the section.
### Configure in code
To build a client in code instead of from a file, `Llm4s` has a builder for each provider this module brings:
`openAi()`, `anthropic()`, `gemini()`, `ollama()`, `openAiCompatible()` (vLLM, LM Studio, LiteLLM, a gateway: any
server that speaks OpenAI's chat-completions API) and `azureOpenAi()`. Each takes plain Java values and builds an
`LlmResult`:
```java
LlmResult openai = Llm4s.openAi()
.apiKey(apiKey)
.model("gpt-4o-mini")
.build();
LlmResult local = Llm4s.ollama()
.model("llama3.2")
.build();
LlmResult gateway = Llm4s.openAiCompatible()
.baseUrl("http://localhost:8000/v1")
.model("qwen2.5-coder")
.header("X-Team-Id", "platform")
.build();
```
What each provider needs is required - an API key and a model, for Ollama the model alone, for an OpenAI-compatible
server its base URL and model, for Azure the endpoint, key and deployment - and everything else has the default
`application.conf` would give it: a builder makes the same config as a section with the same values. That means the
provider's own base URL (`http://localhost:11434` for Ollama), and the context window and reply reserve the model
catalogue lists for the model; for an OpenAI-compatible server, the catalogue's window for the provider its base URL's
host names (Groq, Together, Fireworks, xAI, Perplexity), else 8192 tokens. `baseUrl`, `contextWindow`,
`reserveCompletion`, `requestTimeout` and `streamTimeout` (each a `java.time.Duration`, at most about 292 years)
override them, as do `organization` for OpenAI, `apiVersion` for Azure and `header` for an OpenAI-compatible server. A
context window set without a reserve keeps a quarter of itself for the reply, up to the reserve the provider would
otherwise use, as a section's `contextWindow` does. `metrics(MetricsCollector)` makes the client report every call, as
[`createClient(config, metrics)`](#recording-metrics) does.
`build()` sends no request and does not throw. A required field left out - or set to `null` or a blank string, as
`System.getenv` returns for a variable that is not set - is a failed result of kind `VALIDATION` whose message names
it (`Field 'apiKey' is required`), as is a value out of range, such as a context window of zero. A builder is
immutable, so one can be kept and forked: `base.model("gpt-4o")` returns a new builder and leaves `base` as it was.
Its `toString` prints the API key as `***`, an OpenAI-compatible server's headers by name only, and a base URL or
endpoint without its userinfo, query or fragment, so a builder can be logged.
From Kotlin, build the client the same way and hand it to `llm4s-kotlin-api`'s `Llm4s.wrapClient`, which returns
an `LLMClientKt` with suspend functions, `Flow` streaming and `LLMException`
([#1850](https://github.com/llm4s/llm4s/issues/1850)). `wrapClient` takes any `JLlmClient`, so a client from
`createClient(ProviderConfig)` works the same way:
```kotlin
import org.llm4s.kotlin.Llm4s
import org.llm4s.javaapi.Llm4s as JLlm4s
val client = Llm4s.wrapClient(
JLlm4s.ollama()
.model("llama3.2")
.build()
.get(), // throws the facade's LlmException on a VALIDATION failure; check isSuccess to handle it
)
client.use { println(it.complete("What is 2 + 2?")) } // complete is a suspend function: call it in a coroutine
```
For any other provider - one from its own module, such as `llm4s-bedrock` - `Llm4s.createClient(ProviderConfig)` takes
the provider's config object. OpenAI, Anthropic and Ollama have one with a Java entry point too, which takes the
essentials and fills in the rest (the default base URL, and a context window and completion reserve guessed from the
model name):
```java
LlmResult openai = Llm4s.createClient(OpenAIConfig.apply(apiKey, "gpt-4o-mini"));
LlmResult anthropic = Llm4s.createClient(AnthropicConfig.apply(apiKey, "claude-sonnet-4-20250514"));
LlmResult local = Llm4s.createClient(OllamaConfig.apply("llama3.2", "http://localhost:11434"));
```
(The imports are listed under [The snippets' imports](#the-snippets-imports).) Reading the key is up to you; creating the client sends no
request. The `with` methods of each config (`withContextWindow`, `withReserveCompletion`, and for OpenAI
`withBaseUrl` and `withOrganization`) change one field at a time; some take a Scala `Option`, which the builders above
avoid.
### Recording metrics
`Llm4s.createClient(ProviderConfig, MetricsCollector)` builds a client that reports every call (its outcome, duration,
tokens and cost) to a `MetricsCollector`:
```java
LlmResult measured = Llm4s.createClient(OllamaConfig.apply("llama3.2", "http://localhost:11434"), metrics);
```
For `metrics`, use `new MicrometerMetrics(meterRegistry)` from `llm4s-observability-micrometer` to record into the
application's Micrometer `MeterRegistry` (see [Micrometer metrics](observability/micrometer)), or `PrometheusMetrics`
from `llm4s-observability-prometheus`. A null collector records nothing. The Spring Boot starter does this for you when
the application has a `MeterRegistry`.
A [client builder](#configure-in-code) takes the collector too: `Llm4s.ollama().model("llama3.2").metrics(metrics).build()`.
### Behind a proxy or a corporate CA
Every client builder takes the proxy and TLS settings that `llm4s.http` gives a configured provider
([Corporate networks](../getting-started/configuration#corporate-networks-a-proxy-and-a-custom-ca)): an HTTP proxy,
its Basic credentials, the hosts reached directly, a trust store whose CAs are trusted in addition to the JDK's, and,
for a gateway that requires mutual TLS, a key store with the client certificate (`keyStore(path, password)`):
```java
LlmResult corporate = Llm4s.openAi()
.apiKey(apiKey)
.model("gpt-4o")
.proxy("http://proxy.corp:3128")
.proxyCredentials(proxyUser, proxyPassword)
.nonProxyHosts("localhost", "*.internal")
.trustStore(java.nio.file.Path.of("/etc/pki/corp-ca.p12"), trustStorePassword)
.build();
```
A proxy URL the builder cannot use, or a store it cannot open, makes `build()` return a failed result of kind
`VALIDATION` naming the field; `toString` prints every password as `***`.
## Your first call
```java
import java.io.PrintStream;
import org.llm4s.javaapi.JLlmClient;
import org.llm4s.javaapi.Llm4s;
import org.llm4s.javaapi.LlmResult;
public final class HelloLlm4s {
private HelloLlm4s() {}
public static void main(String[] args) {
System.exit(run(System.out, System.err));
}
static int run(PrintStream out, PrintStream err) {
LlmResult created = Llm4s.createDefaultClient();
if (created.isFailure()) {
err.println("Could not create a client: " + created.getError().getMessage());
return 1;
}
// JLlmClient is AutoCloseable: it holds an HTTP client, so close it.
try (JLlmClient client = created.get()) {
LlmResult answer = client.complete("What is a monad? Answer in one sentence.");
answer
.ifSuccess(text -> out.println(text))
.ifFailure(error -> err.println("The call failed: " + error.getMessage()));
return answer.isSuccess() ? 0 : 1;
}
}
}
```
`Llm4s.createDefaultClient()` loads the default provider section and builds a client from it. It does not throw: a
missing section, a missing API key or an unknown provider is a failed `LlmResult`, and its message says what to fix (for
a missing key it names the environment variable and the section). `JLlmClient` is `AutoCloseable` and closing it closes
the underlying client, so use try-with-resources.
`ifSuccess` and `ifFailure` each take a lambda and return the result, so they chain. The `run` method is separate from
`main` so that a test can pass its own streams.
The [`gradle-java` sample](https://github.com/llm4s/llm4s/tree/main/modules/samples/gradle-java) is this program in a
complete Gradle project.
### The snippets' imports
The snippets below leave out their imports. Together they use these:
```java
import java.util.List;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.Flow;
import org.llm4s.javaapi.CompletionStream;
import org.llm4s.javaapi.ConversationBuilder;
import org.llm4s.javaapi.FakeLlmClient;
import org.llm4s.javaapi.JCompletion;
import org.llm4s.javaapi.JCompletionOptions;
import org.llm4s.javaapi.JEmbeddingClient;
import org.llm4s.javaapi.JEmbeddingPurpose;
import org.llm4s.javaapi.JEmbeddings;
import org.llm4s.javaapi.JLlmClient;
import org.llm4s.javaapi.JReasoningEffort;
import org.llm4s.javaapi.JTool;
import org.llm4s.javaapi.JToolArgs;
import org.llm4s.javaapi.JToolCall;
import org.llm4s.javaapi.Llm4s;
import org.llm4s.javaapi.LlmErrorKind;
import org.llm4s.javaapi.LlmException;
import org.llm4s.javaapi.LlmResult;
import org.llm4s.javaapi.Param;
import org.llm4s.javaapi.Tool;
import org.llm4s.llmconnect.config.AnthropicConfig;
import org.llm4s.llmconnect.config.OllamaConfig;
import org.llm4s.llmconnect.config.OpenAIConfig;
import org.llm4s.llmconnect.model.Conversation;
import org.llm4s.metrics.MetricsCollector;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
```
## A conversation
A conversation is a list of messages. `ConversationBuilder` builds one without Scala syntax:
```java
Conversation conversation = ConversationBuilder.create()
.system("You answer in one short sentence.")
.user("What is a monad?")
.build();
LlmResult answer = client.complete(conversation);
System.out.println(answer.get());
```
The builder has `system`, `user` and `assistant`, and messages are sent in the order you add them. It is immutable: each
method returns a new builder, so one can be shared and extended. A `null` message throws `NullPointerException` at once.
## Completion options
`JCompletionOptions` sets how the model answers one request, built with a builder and passed with the conversation:
```java
JCompletionOptions options = JCompletionOptions.builder()
.temperature(0.2)
.maxTokens(512)
.reasoning(JReasoningEffort.MEDIUM)
.build();
LlmResult answer = client.complete(conversation, options);
System.out.println(answer.get());
```
The builder has `temperature`, `topP`, `maxTokens`, `presencePenalty`, `frequencyPenalty`, `reasoning` (the Java enum
`JReasoningEffort`: `NONE`, `LOW`, `MEDIUM` or `HIGH`) and `budgetTokens`, an explicit thinking budget that overrides
the one the reasoning level implies. The providers that take a budget are Anthropic, and OpenRouter for Claude models:
both raise a budget below `1024` to `1024` and keep it below the token limit, and a budget set with no reasoning level
still turns thinking on. A setting you leave out keeps the library's default:
temperature `0.7`, top-p `1.0`, no penalties, and no token limit, reasoning level or thinking budget. The built options
have an accessor of the same name for each setting. `maxTokens()` and `budgetTokens()` return an `OptionalInt` and
`reasoning()` an `Optional`, empty when unset. Their setters also take an `OptionalInt` or `Optional`,
so an empty one clears the value.
The builder is immutable like `ConversationBuilder`: each setter returns a new builder, and `options.toBuilder()` starts
one from existing options. A value no provider accepts throws `IllegalArgumentException` at once: a negative or
non-finite temperature, a top-p outside `0` to `1`, a non-finite penalty, or a token count below `1`. A `null` throws
`NullPointerException`. A value only some models reject, such as a temperature above `1` for some models, is left to
the provider. A model that does not reason ignores `reasoning`. Tools and response formats are not options here yet.
## The whole reply
`complete` returns the reply's text. `completion` takes the same arguments - a query, a conversation, or a conversation
and `JCompletionOptions` - and returns the whole reply as a `JCompletion`: the model that answered, the tokens it used,
its estimated cost and the tool calls it asked for, alongside the text:
```java
JCompletion reply = client.completion(conversation).get();
System.out.println(reply.model() + ": " + reply.content());
reply.usage().ifPresent(usage ->
System.out.println(usage.promptTokens() + " tokens in, " + usage.completionTokens() + " out"));
reply.estimatedCost().ifPresent(cost -> System.out.println("about $" + cost.toPlainString()));
for (JToolCall call : reply.toolCalls()) {
System.out.println("wants " + call.name() + " " + call.argumentsJson());
}
```
`model()` is the model as the provider names it, which can be more specific than the one you configured. `usage()` is a
`JTokenUsage` with `promptTokens()`, `completionTokens()`, `totalTokens()` and `thinkingTokens()`, and the prompt-cache
counts `cachedTokens()` (input read from the provider's prompt cache, billed at the cheaper cache-read rate) and
`cacheCreationTokens()` (input written into it, typically billed above the normal input rate) - each an `int`, zero when
the model reported none - or empty when the provider reported no usage. `estimatedCost()` is a `java.math.BigDecimal` in USD, empty
when the cost is not known; it is the figure an agent adds to its `usage().totalCost()`. `toolCalls()` lists `JToolCall`s,
the same type an agent's messages use, with the arguments as JSON text; `thinking()` is the model's reasoning text when
it reported one. A `JCompletion` is a value with `equals`, `hashCode` and a `toString` that prints the full text.
The method is `completion`, a noun, because it returns the reply rather than performs a request for its text; `complete`
keeps returning a `String`. There is no `completion` overload for core's Scala `CompletionOptions`, so
`client.completion(conversation, null)` compiles, and returns a failed result like any other `null` argument.
## Streaming the reply
`stream` sends the same arguments as `complete` - a query, a conversation, or a conversation and `JCompletionOptions` -
with a `CompletionStreamListener`, and returns at once. The listener receives each piece of the reply's text as it
arrives (`onPartial`), then the whole reply (`onComplete`, a `JCompletion`) or the error (`onError`); only `onPartial` is
abstract, so a lambda or a method reference is a listener:
```java
JCompletion reply;
try (CompletionStream stream = client.stream(conversation, System.out::print).get()) {
reply = stream.await().get();
}
System.out.println();
Flow.Publisher pieces = client.streamAsPublisher(conversation);
```
The returned `CompletionStream` awaits the reply (`await()`, the same outcome the listener heard) and cancels the call
(`cancel()`, or `close()`, so try-with-resources stops a call its block leaves). Cancelling interrupts the provider's
thread, which is how llm4s providers stop, and the listener then hears `onError` with a `CANCELLED` error. Listener calls
are made one at a time, never on your thread; one that throws stops the call, and `onError` receives what it threw.
`streamAsPublisher` returns the text as a `java.util.concurrent.Flow.Publisher` instead: it is in the JDK, so
Reactor (`JdkFlowAdapter`), RxJava and Mutiny adapt it without another llm4s dependency, and `llm4s-reactor` returns
it as a `Flux` ([Java Reactor](java-reactor)). Each subscription makes its own
call. A subscriber slower than the model holds the provider's thread rather than losing text, and cancelling the
subscription cancels the call. A failed call ends with `onError` (an `LlmException`) after every piece that streamed
before the failure. Streaming yields text only; tool-call deltas are not streamed.
## Structured output
Pass a record class after the query, or after the conversation (and options), and `complete` returns an instance of
it instead of text. The record's components become a JSON Schema that is sent as the structured-output format, and
the answer is read back into the record:
```java
record City(String name, String country, int population) {}
City city = client.complete("The largest city in Japan, with its population", City.class).get();
System.out.println(city.name() + ", " + city.country() + ": " + city.population());
```
A component can be a `String`, an `int`, `long`, `short` or `byte` (boxed or not), a `double` or `float`, a
`boolean`, an enum (the model answers with a constant's name), another record, a `java.util.List` of any of these, or an
`Optional` of any of these. `Optional` is the only way to let the model answer `null` or leave a field out; every other
field is required. A `Map`, an array, a raw `List`, an interface or a record that contains itself is refused before
any call, and an answer that does not fit - a missing field, the wrong JSON type, a fraction for an `int`, an unknown
enum name - is a failed result whose message names the field, such as `field 'previous[0].city': expected a string,
got a number`. Numbers pass through a `double`, so ask for an identifier above 2^53 as a `String`.
How firmly the schema binds the model is the provider's: OpenAI-style providers send it as a native JSON Schema,
Anthropic adds it to the system prompt, and watsonx and Bedrock ignore it, so the answer is still checked when it comes
back. See [Structured output](structured-output) for each provider.
## Asynchronous calls
Every blocking call has an `*Async` variant that returns a `CompletableFuture` at once and makes the call on another
thread: `completeAsync` and `completionAsync` on `JLlmClient` (with the same arguments as `complete` and `completion`,
a record class for structured output included), `runAsync`, `continueConversationAsync`, `resumeAsync` and
`recoverAsync` on `JAgent`, and `embedAsync` on `JEmbeddingClient`. Fan several calls out and wait for all of them
with `allOf`, or bound one with `orTimeout`:
```java
CompletableFuture first = client.completeAsync("Summarise the first report");
CompletableFuture second = client.completeAsync("Summarise the second report");
CompletableFuture turn = agent.runAsync("Plan the release");
CompletableFuture.allOf(first, second, turn).join();
System.out.println(first.join() + " / " + second.join() + " / " + turn.join().answer().orElse(""));
CompletableFuture answer = client.completeAsync("A long question").orTimeout(30, TimeUnit.SECONDS);
```
A failed call completes its future exceptionally with the `LlmException` that the blocking call's `LlmResult` would
hold, of the same `LlmErrorKind`, so `join()` throws a `CompletionException` and `get()` an `ExecutionException` whose
cause is that `LlmException`. A future you cancel throws `CancellationException`, as any `CompletableFuture` does.
`cancel(true)` interrupts the call in flight, which is how llm4s providers stop; an agent turn cancelled this way ends
as the blocking call interrupted does: cancelled, the conversation of a continued, resumed or recovered turn left for
`recover`. Completing the future yourself - `orTimeout`,
`completeOnTimeout`, `complete` - interrupts the call too, since its answer can no longer be delivered. `cancel(false)`
lets the call finish and discards its answer. Cancel the future the method returned: a future derived from it with
`thenApply` does not pass its cancellation back.
By default each call runs on a new virtual thread. A `JLlmClient` has an executor of its own for that, which
`close()` shuts down: a call still running is then interrupted, and one made after it fails with kind `OTHER`.
`withAsyncExecutor` runs the calls on an executor of yours instead - a bounded pool, a container's managed executor:
```java
ExecutorService pool = Executors.newFixedThreadPool(8);
JLlmClient pooled = client.withAsyncExecutor(pool); // shares client's connection; never shuts pool down
```
The pooled client shares the connection of the client it came from, so closing either closes it, and it never shuts
the pool down. `JAgent` and `JEmbeddingClient` have `withAsyncExecutor` too. Any executor works, a `ForkJoinPool`
included: cancelling a future interrupts the thread running its call whatever the executor. An executor that refuses
a call - shut down, or its queue full - fails that call's future at once with kind `OTHER`, the executor's
`RejectedExecutionException` its cause. The [threading guide](java-threading-and-cancellation) says which thread runs
what.
## Reading a result
Every `complete` call returns an `LlmResult`, and every `completion` call an `LlmResult`. There
are several ways to take the value out, depending on how you want to treat a failure:
```java
LlmResult result = client.complete("What is 2+2?");
String text = result.get(); // the value, or throws LlmException
String orNull = result.getOrNull(); // the value, or null when the call failed
Optional optional = result.toOptional(); // Optional.empty() when the call failed
LlmResult length = result.map(String::length);
CompletableFuture future = result.toCompletableFuture();
```
`isSuccess()` and `isFailure()` test it, `getError()` returns the `LlmException` of a failure (or `null`), and `map`
transforms a success and passes a failure through. `toCompletableFuture()` returns a future that is already complete: it
makes a result fit an API that expects a future, and does not make the call asynchronous (see
[Asynchronous calls](#asynchronous-calls)).
## Handling a failure
A call that fails does not throw. `JLlmClient` turns a failure of the provider, the network or the library into a failed
`LlmResult`, and a `null` argument into a failed result too. Only `get()` throws, and it throws `LlmException`, an
unchecked exception that says what went wrong in Java types:
```java
try {
String text = client.complete("What is 2+2?").get();
System.out.println(text);
} catch (LlmException e) {
System.err.println(e.getMessage());
switch (e.getKind()) {
case AUTHENTICATION, CONFIGURATION -> System.err.println("check the API key and the provider section");
case RATE_LIMIT -> System.err.println("rate limited; wait "
+ e.getRetryAfter().map(d -> d.toSeconds() + " s").orElse("a while"));
case SERVICE -> System.err.println("the provider answered HTTP "
+ (e.getStatusCode().isPresent() ? e.getStatusCode().getAsInt() : "?"));
default -> { }
}
if (e.isRecoverable()) {
System.err.println("a retry may succeed");
}
}
```
`getKind()` is the Java enum `LlmErrorKind`:
| Kind | The failure | Recoverable |
|---|---|---|
| `AUTHENTICATION` | the provider rejected the API key | no, unless it is a provider's `401` or `403` reported as a service error |
| `RATE_LIMIT` | the provider, or a local limiter, refused the call for now | yes, unless it is an embedding provider's error |
| `TIMEOUT` | the call did not finish in time | yes |
| `NETWORK` | the provider could not be reached | yes |
| `SERVICE` | the provider answered with an error | yes, unless it is an embedding provider's error |
| `VALIDATION` | the request is wrong: a missing or invalid argument | no, unless it is a provider's `400` reported as a service error |
| `CONFIGURATION` | a setting is missing or invalid | no |
| `CANCELLED` | the call was cancelled, for example by an interrupt | no |
| `OTHER` | anything else, including an error from another llm4s module or your own code | depends on the error |
A provider's error response with status `400`, `401`/`403` or `429` has the kind that status means (`VALIDATION`,
`AUTHENTICATION`, `RATE_LIMIT`) even when a provider client reported it as a generic service error. Such an error keeps
the recoverability of its class: a `ServiceError` or an `APIError` is recoverable whatever its status, so a `403`
reported as one reads as `AUTHENTICATION` and `isRecoverable()` is `true`. The table's column is what the kind's own
error classes say; to decide whether to try again, always call `isRecoverable()` rather than infer it from the kind.
Every error class in `org.llm4s.error` has a kind, and a test fails when a new one is added without one.
An [embedding](#embeddings) provider's error response has the kind of its HTTP status in the same way, but llm4s does
not call any embedding error recoverable, so `isRecoverable()` is `false` for it whatever its kind.
`isRecoverable()` says whether the call may succeed if tried again, perhaps after you do something first: wait out a rate
limit, or correct a request the provider rejected. `getRetryAfter()` is the `Optional` the provider asked you
to wait (its `Retry-After` header), empty when it did not say, and `getStatusCode()` the `OptionalInt` HTTP status of a
provider's error response. If the error carries a `Throwable`, it is the exception's `getCause()`.
`e.error()` is still the llm4s error itself, a Scala `LLMError` whose `message()` and `formatted()` (the message plus its
code and context) are plain methods, and whose classes - `RateLimitError`, `ServiceError` and the rest - you can test
with `instanceof` for a detail the kind does not carry. The [error handling guide](error-handling) has the full list
and the recovery tools, written for Scala.
If the thread blocked in `complete` is interrupted, the call returns a failed result whose error is a `CancelledError`,
with the thread's interrupt flag still set. `InterruptedException` is never thrown, so the method does not declare it
and Java will not let you write `catch (InterruptedException e)` around the call; test the result for a
`CancelledError` instead. The [threading and cancellation guide](java-threading-and-cancellation) has the details.
## An agent turn
`Llm4s.createAgent(client)` wraps a client in a `JAgent`. `run` and `continueConversation` block, like `complete`, and
return an `LlmResult`. A `JAgentResult` is read with JDK types and this module's own: `answer()` is an
`Optional`, `messages()` a `java.util.List`, and `status().kind()` the Java enum `AgentStatusKind`, so
a `switch` covers every way a turn ends:
```java
JAgent agent = Llm4s.createAgent(client);
JAgentResult result = agent.run("What is 2+2?").get();
switch (result.status().kind()) {
case COMPLETED -> System.out.println(result.answer().orElseThrow());
case BLOCKED -> System.out.println("Blocked by " + result.status().guardrail().orElseThrow());
case STEP_LIMIT_REACHED -> System.out.println("Hit the step limit");
case TOKEN_LIMIT_REACHED -> System.out.println("Hit the token budget");
case SUSPENDED -> System.out.println("Waiting for " + result.status().pending().size() + " answers");
}
for (JMessage message : result.messages()) {
System.out.println(message.role() + ": " + message.content());
}
JUsageSummary usage = result.usage();
System.out.println(usage.inputTokens() + " tokens in, " + usage.outputTokens() + " out");
JAgentResult next = agent.continueConversation(result, "And 3+3?").get();
```
Each status's data has its own accessor, empty for the other kinds: `answer()` for `COMPLETED`, `guardrail()` and
`reason()` for `BLOCKED`, and `pending()` for `SUSPENDED` - the approvals and questions the turn waits for, which the
agent guide's [suspended turns](agents/#suspended-turns-from-java-and-kotlin) section answers. A `JMessage` has a
`role()` (the Java enum `JMessageRole`), its `content()`, the `toolCalls()` an assistant message asked for, with their
arguments as JSON text (an object, as a model sends them; a call built with a `ujson.Str` renders as a JSON string
literal), and the `toolCallId()` a tool message answers. `usage()` counts tokens as `long`s, the cost as a
`java.math.BigDecimal` (two usages are equal when their costs are numerically equal, whatever the scale), and
`byModel()` breaks both down per model. No accessor returns a Scala or `ujson` type.
These types are values, but not `Serializable`. Their `toString` prints the full text - a message's content, a tool
call's arguments, an answer or a guardrail's reason - as the Scala types do, so mind what you log.
## Tools
An agent can call tools you write in Java, in either of two styles. Annotate the public methods of a class with `@Tool`,
and each parameter with `@Param` (its `name` is what the model passes; without one, the compiled name is used, which
needs `javac -parameters`):
```java
class WeatherTools {
@Tool(description = "The current weather in a city")
public String weather(@Param(name = "city", description = "City name") String city) {
return "Sunny in " + city;
}
}
```
or build one from a lambda with `JTool.builder`, then give the tools to `Llm4s.createAgentWithTools`:
```java
List weather = JTool.fromAnnotated(new WeatherTools()).get();
LlmResult add = JTool.builder("add", "Adds two numbers")
.param("a", long.class, "First")
.param("b", long.class, "Second")
.handler(args -> args.getLong("a") + args.getLong("b"))
.build();
JAgent agent = Llm4s.createAgentWithTools(client, List.of(weather.get(0), add.get())).get();
JAgentResult result = agent.run("What is the weather in Paris?").get();
```
A parameter is a `String`, an `int`, `long`, `short` or `byte` (boxed or not), a `double` or `float`, a `boolean`, or an
enum (the model passes a constant's name); its JSON Schema is derived from the type, and `optionalParam` (or
`@Param(required = false)`, with a boxed type) lets the model leave it out. The model's arguments are read into those Java
values before your code runs, with no JSON library: a missing or mistyped argument is a tool error the model sees, and so
is an exception your method or lambda throws - the agent keeps running and the model can try again. A result is passed to
the model as text if it is a `String`; a number, boolean, enum, `Map` or `Collection` becomes JSON. A tool that cannot be
built (no handler, an unsupported parameter type, a parameter declared twice) and an agent given two tools with one name
are failed results, not exceptions.
An agent runs a tool at least once, not exactly once: a retry, an approval, an answered question or a `recover` can run
the same call again (see [Tool side effects](agents/#tool-side-effects)). A tool with an external side effect therefore
passes the call's idempotency key to the system it reaches, which then does the work once. A lambda reads it from its
arguments, as `args.idempotencyKey()`; an annotated method declares a `JToolArgs` parameter, which the model does not
see, and reads it there:
```java
@Tool(description = "Charges a card once")
public String charge(@Param(name = "cents", description = "Amount in cents") long cents, JToolArgs call) {
return billing.charge(cents, call.idempotencyKey().orElse(null)); // empty outside an agent run
}
```
```java
LlmResult refund = JTool.builder("refund", "Refunds a charge once")
.param("charge", String.class, "The charge to refund")
.handler(args -> billing.refund(args.getString("charge"), args.idempotencyKey().orElse(null)))
.build();
```
`idempotencyKey()` and `toolCallId()` (the id the model gave the call) are `Optional`s: present for every call
an agent runs, the key the same in every run of one call, and empty when the tool runs outside an agent.
## The Kotlin DSL
`llm4s-kotlin-api` declares the same tools, options and clients with a Kotlin DSL
([#1495](https://github.com/llm4s/llm4s/issues/1495)). Each block builds the Java facade's object - a `JTool`, a
`JCompletionOptions`, a provider's client builder - so it behaves exactly as the Java code above, and a failure throws
`LLMException` (an option out of range, `IllegalArgumentException`) instead of returning a failed `LlmResult`:
```kotlin
import org.llm4s.javaapi.JReasoningEffort
import org.llm4s.kotlin.Llm4s
import org.llm4s.kotlin.completionOptions
import org.llm4s.kotlin.tools
import kotlin.time.Duration.Companion.seconds
val client = Llm4s.openAi { // also anthropic, gemini, ollama, openAiCompatible, azureOpenAi
apiKey = System.getenv("OPENAI_API_KEY")
model = "gpt-4o-mini"
requestTimeout = 30.seconds
}
val reply = client.complete("Name a colour", completionOptions {
temperature = 0.2
maxTokens = 200
reasoning = JReasoningEffort.LOW
})
val agent = Llm4s.createAgent(client, tools {
tool("add", "Adds two numbers") {
param("a", "First")
param("b", "Second")
handler { args -> args.get("a") + args.get("b") }
}
tool("refund", "Refunds a charge once") {
param("charge", "The charge to refund")
handler { args -> billing.refund(args.get("charge"), args.idempotencyKey) }
}
})
val result = agent.run("What is 2 + 3?") // a suspend function: call it in a coroutine
```
A client block takes the fields of its Java builder (see [Configure in code](#configure-in-code)), with a
`kotlin.time.Duration` for each timeout; `completionOptions(base) { ... }` starts from existing options. `tool` declares
one tool and `tools` a list of them, to which `add` appends one built with `JTool.builder` or `JTool.fromAnnotated`.
`param` and `optionalParam` take the types `JTool.builder` does - `String`, `Int`, `Long`, `Short`, `Byte`,
`Double`, `Float`, `Boolean` or an enum - and the handler reads them with `args.get(name)`, or `args.getOrNull(name)`
for an optional one the model may leave out. `args.idempotencyKey` and `args.toolCallId` describe the call
([#1740](https://github.com/llm4s/llm4s/issues/1740)), `null` outside an agent run; `args.java` is the `JToolArgs`
underneath.
The handler is a `suspend` function. The agent runs a tool on a thread of its own, which waits for the handler in a
coroutine of its own (`runBlocking`), not in the caller's. Cancelling the coroutine that runs the turn - the caller of
`run`, a collector of `stream`, a `withTimeout` - cancels the turn, which interrupts that thread and so cancels the
handler: a suspended `delay`, `withContext` or other cancellable call throws `CancellationException`.
## Embeddings
An embedding model turns a text into a vector, a `float[]`, and texts that mean similar things get vectors that point
in similar directions: the basis of semantic search. `Llm4s.createDefaultEmbeddingClient()` creates a
`JEmbeddingClient` for the model `application.conf` names, as `createDefaultClient()` does for chat. The model is
`provider/model` under `llm4s.embeddings`, or the `EMBEDDING_MODEL` environment variable:
```hocon
llm4s {
embeddings {
model = "openai/text-embedding-3-small" # or set EMBEDDING_MODEL
}
}
```
The key comes from the vendor's variable, `OPENAI_API_KEY` here, as a chat section's does. The embedding providers
`llm4s-java-api` brings are `openai` and `ollama`; Voyage, Jina and Cohere come with their own modules
(`llm4s-voyage`, `llm4s-jina`, `llm4s-cohere`). Creating the client sends no request, and does not throw:
```java
LlmResult created = Llm4s.createDefaultEmbeddingClient();
created.ifFailure(error -> System.err.println("Could not create an embedding client: " + error.getMessage()));
JEmbeddingClient embedder = created.getOrNull(); // null when it failed
```
No model configured, a provider that is not on the classpath or a missing key is a failed result of kind
`CONFIGURATION`, whose message says what to set. `embedder.model()` is the model as configured, and
`embedder.dimensions()` an `OptionalInt`: its vector length as the provider's module declares it, or empty for a model
the module does not list, such as an Ollama model you pulled yourself - that model still works, and its vectors are as
long as the provider makes them.
To embed with a model other than the one `application.conf` names - two models side by side, say -
`Llm4s.createEmbeddingClient` takes it as `provider/model`. The provider's key, base URL and timeouts still come from
configuration, as for the default client, so `llm4s.embeddings.model` need not be set at all:
```java
JEmbeddingClient large = Llm4s.createEmbeddingClient("openai/text-embedding-3-large").get();
JEmbeddingClient local = Llm4s.createEmbeddingClient("ollama/bge-m3").get(); // dimensions() is empty
```
A `null` model is a failed result of kind `VALIDATION`; one not in the form `provider/model`, a provider that is not on
the classpath or a missing key, one of kind `CONFIGURATION`.
`embed` takes a `java.util.List` and returns a `JEmbeddings`: one vector per text, in the order of the texts,
from a single request. `JEmbeddings.cosineSimilarity` compares two of them:
```java
JEmbeddings embeddings = embedder.embed(List.of(
"The cat sat on the mat.",
"A kitten was sitting on the rug.")).get();
List vectors = embeddings.vectors();
double similarity = JEmbeddings.cosineSimilarity(vectors.get(0), vectors.get(1));
System.out.println(embeddings.model() + ", " + embeddings.dimensions() + " dimensions: similarity " + similarity);
```
`model()` is the model as the provider named it in its reply, and `dimensions()` the length of every vector.
`vectors()` returns a new unmodifiable list of new arrays on each call, so changing one changes nothing; call it once and
keep the list. A `JEmbeddings` is a value, with `equals` and `hashCode` over its vectors' components. Core's vectors
are `double`s; these are their nearest `float`s, the precision vector stores keep.
`cosineSimilarity` goes from `-1` (opposite) through `0` (unrelated) to `1` (the same direction). A zero vector has no
direction, so its similarity to anything is `0`. Two vectors of different lengths, which come from different models,
throw `IllegalArgumentException`, and a `null` throws `NullPointerException`.
Several embedding models embed a search query differently from the documents it searches, and comparing a query
embedded as a document quietly finds worse matches. Say which side a text is on with `JEmbeddingPurpose`:
```java
List documents = embedder.embed(texts, JEmbeddingPurpose.DOCUMENT).get().vectors();
float[] query = embedder.embed(List.of("Where did the cat sit?"), JEmbeddingPurpose.QUERY).get().vectors().get(0);
int best = 0;
for (int i = 1; i < documents.size(); i++) {
if (JEmbeddings.cosineSimilarity(query, documents.get(i)) > JEmbeddings.cosineSimilarity(query, documents.get(best))) {
best = i;
}
}
System.out.println("closest: " + texts.get(best));
```
`embed(texts)` without a purpose embeds documents. Voyage and Cohere send the purpose as `input_type` and Jina as
`task`; OpenAI and Ollama embed both alike and ignore it, so code that says which side it is on works with any of them.
Like `complete`, `embed` blocks and never throws: an empty list returns no vectors without a request, a `null` list,
text or purpose is a failed result of kind `VALIDATION`, and an interrupt one of kind `CANCELLED`. A provider's error
response has the kind of its HTTP status - `401` reads as `AUTHENTICATION`, `429` as `RATE_LIMIT`, `400` as
`VALIDATION`, any other as `SERVICE`, with `getStatusCode()`. A provider that cannot be reached is `NETWORK`, and one
that does not answer in time `TIMEOUT`; any other failure with no status, such as a reply that cannot be read, is
`OTHER`. To keep the vectors in a store, search them and answer questions from what they find, see
[Java RAG](java-rag) (`llm4s-java-rag`).
## Testing your code
For a complete runnable example, see the [Java offline ticket classifier](../examples/#java-offline-example).
It injects a `JLlmClient`, handles unknown replies and provider failures, and checks recorded prompts in its test.
`FakeLlmClient` is a fake model for your own tests: it answers from a script, records every prompt your code sent, and
needs no network or API key. Its `client()` is a real `JLlmClient`, so the code under test needs no interface or mock -
give it the fake's client wherever it takes one:
```java
class CalculatorTest {
@Test
void answersFromTheModel() {
FakeLlmClient fake = FakeLlmClient.replying("4");
String answer = new Calculator(fake.client()).ask("What is 2+2?");
assertEquals("4", answer);
assertEquals(List.of("What is 2+2?"), fake.prompts());
}
@Test
void reportsAProviderFailure() {
FakeLlmClient fake = FakeLlmClient.builder()
.failWith(LlmErrorKind.RATE_LIMIT, "slow down")
.build();
LlmResult result = fake.client().complete("What is 2+2?");
assertTrue(result.isFailure());
assertEquals(LlmErrorKind.RATE_LIMIT, result.getError().getKind());
}
}
```
`Calculator` stands for your own class: here, one whose `ask` returns `client.complete(question).getOrNull()`.
`FakeLlmClient.replying("a", "b")` answers the first call with `a` and the second with `b`; `builder()` mixes replies
(`reply`) and failures (`failWith`, with the `LlmErrorKind` that `getKind()` then reports). A call beyond the script is a
failed result whose message names the call and its prompt, so a test that makes more calls than it scripted says why.
`callCount()`, `prompts()` (the last user message of each call, in order) and `lastPrompt()` show what your code sent.
The fake is in `llm4s-java-api` itself, so a test needs no other dependency. It answers `complete`, `completion` and
structured output; for an agent with tools from Scala, `llm4s-testkit`'s `ScriptedLLMClient` scripts tool calls too
([Testing with the Testkit](testing-with-the-testkit)), and `Llm4s.wrapClient` turns it into a `JLlmClient`.
## What is not here yet
`llm4s-java-api` covers a client, a conversation, the whole reply, the kind of a failure and embeddings. These are not
available from Java yet, each with the issue that tracks it:
| You may expect | State today |
|---|---|
| Agents beyond a turn | [An agent turn](#an-agent-turn) reads a result; the agent guide covers [streaming a turn](agents/streaming#java-and-kotlin) and [suspended turns](agents/#suspended-turns-from-java-and-kotlin) from Java. Tools are defined from Java in [Tools](#tools) |
## Where next
- [Java RAG](java-rag): ingest documents and ask questions about them, with `llm4s-java-rag`.
- [Java Reactor](java-reactor): a streamed reply as a Reactor `Flux`, with `llm4s-reactor`.
- [Spring Boot](spring-boot): auto-configuration, a template bean and a health indicator on top of this module.
- [Quarkus](quarkus): the client as a CDI bean, configured from `application.properties`, behind a REST endpoint.
- [Basic usage](basic-usage) and [providers](providers): the same concepts in Scala, and every provider's configuration.
- [Gradle integration](../getting-started/gradle): dependency recipes for Gradle builds.
- The [`gradle-java` sample](https://github.com/llm4s/llm4s/tree/main/modules/samples/gradle-java): a Java project you
can run.
---
Source: https://llm4s.org/guide/java-threading-and-cancellation
# Java Threading and Cancellation
Which thread a call blocks, whether one client can be shared, how to run calls on virtual threads and what an
interrupt does, for code that uses `llm4s-java-api`.
---
Every statement on this page is checked by a test in
[`ThreadingModelSpec`](https://github.com/llm4s/llm4s/blob/main/modules/java-api/src/test/scala/org/llm4s/javaapi/ThreadingModelSpec.scala)
unless it says otherwise, and was observed on JDK 21. Virtual threads need JDK 21.
The Java snippets below are compiled by
[`ThreadingGuideSnippets.java`](https://github.com/llm4s/llm4s/blob/main/modules/java-api/src/test/java/org/llm4s/javaapi/ThreadingGuideSnippets.java).
## Which calls block
`JLlmClient.complete(...)`, `JAgent.run(...)`, `JAgent.continueConversation(...)`, `JAgent.resume(...)`,
`JAgent.recover(...)` and `JEmbeddingClient.embed(...)` block the calling thread until the answer is back. Each has an
`*Async` variant - `completeAsync`, `completionAsync`, `runAsync`, `continueConversationAsync`, `resumeAsync`,
`recoverAsync`, `embedAsync` - that returns a `CompletableFuture` at once and makes the same call on another thread
(see [Asynchronous calls](java#asynchronous-calls)). `JAgent.stream`, `streamResume` and `streamRecover` return an
`AgentStream` at once too. `LlmResult.toCompletableFuture()` does not make a call asynchronous: it returns a future that
is already complete, so `cancel(true)` on it returns `false` and changes nothing.
To run a call off your own thread, use its `*Async` variant:
```java
CompletableFuture answer = client.completeAsync("Summarise this.");
String text = answer.join(); // waits on your thread; answer.cancel(true) interrupts the call
```
The call runs on a new virtual thread, named `llm4s-async-N`, of an executor the client owns and `close()` shuts down,
or on the executor that `withAsyncExecutor` names. `JAgent` and `JEmbeddingClient`, which have no `close()`, start a
virtual thread of their own for each call unless `withAsyncExecutor` names an executor.
## Which thread runs the call
| Call | The provider call runs on |
|------|---------------------------|
| `JLlmClient.complete` | your thread, whether it is a platform thread or a virtual thread |
| `JLlmClient.completeAsync` | a virtual thread of the client's own executor, or a thread of the executor `withAsyncExecutor` names |
| `JAgent.run` | a virtual thread of the library's, never yours: your thread waits in `run`, and the model call runs on a virtual thread of its own |
The library drives an agent run from a virtual thread it starts with `Thread.ofVirtual()`
([`RunHandle.scala`](https://github.com/llm4s/llm4s/blob/main/modules/agent/src/main/scala/org/llm4s/agent/graph/RunHandle.scala)),
and runs each task of the run, the model call included, on a virtual thread of its own, forked with the Ox
structured-concurrency library: the `TaskExecutor` in
[`GraphBuilder.scala`](https://github.com/llm4s/llm4s/blob/main/modules/agent/src/main/scala/org/llm4s/agent/graph/GraphBuilder.scala)
documents that "the default runs them concurrently on virtual threads in an Ox scope". Virtual threads are always
daemon threads. The test checks the thread the model call runs on: not the caller, virtual and a daemon.
On JDK 21 that thread had an empty name (observed, not pinned by a test), so look for the run thread, named
`llm4s-run-`, in a thread dump instead.
## Sharing one client or agent
One `JLlmClient` and one `JAgent` can be used by many threads at once. The tests start 50 platform threads on one
client and 32 virtual threads on one agent, hold every call inside the client until all of them are in flight
together, and check that each thread gets the answer to its own query.
## Virtual threads
Calls from virtual threads work, and a call does not hold on to the virtual thread's carrier thread while it waits
for the provider. The test starts more virtual threads than the machine has cores (at least 64), has a local server
that answers only once every request has arrived, and checks that all of them were served at once through
`JLlmClient` and the real Ollama client. A call that pinned its carrier thread could not have served more requests
than there are carriers, so the server would never have answered. A one-off run of the same check with the
scheduler limited to two carrier threads (`-Djdk.virtualThreadScheduler.parallelism=2
-Djdk.virtualThreadScheduler.maxPoolSize=2`) served 16 concurrent requests too; the committed test does not set
those options.
This was verified for the facade, the Ollama provider and the JDK's `HttpClient`. The other providers were not
tested. The OpenAI and Anthropic providers use their vendor SDKs, and this page makes no claim about them.
## Interrupting a call
Interrupt the thread that is blocked in a call and the call returns a failed result whose error is a
`CancelledError`, with the thread's interrupt flag still set, so your own code (and any executor) still sees the
interruption. It behaves the same on a platform thread and on a virtual thread.
```java
LlmResult result = client.complete("Long question");
if (result.isFailure() && result.getError().error() instanceof CancelledError) {
// the thread was interrupted; its interrupt flag is still set
}
```
`cancel(true)` on the future of an `*Async` call interrupts the thread running it, which cancels the provider call in
the same way; the future knows that thread, so this works on any executor, a `ForkJoinPool` included, and the
executor's thread is handed back with its interrupt flag cleared. Completing the future yourself (`orTimeout`,
`completeOnTimeout`) interrupts the call too. `Future.cancel(true)` on a task you submitted yourself interrupts the call
as well, on an executor whose `cancel(true)` interrupts. The Spring Boot starter's `completeAsync` is
`JLlmClient.completeAsync` on its `llm4sTaskExecutor`.
### `InterruptedException` is never thrown, so Java cannot catch it
Neither `JLlmClient.complete` nor `JAgent.run` throws `InterruptedException`. An interrupt comes back as the
`CancelledError` result above, with the flag set, and an `InterruptedException` that a custom `LLMClient` lets escape
is turned into the same `CancelledError`, with the flag restored. The methods therefore declare no checked exception,
and the Java compiler rejects `catch (InterruptedException e)` around either call:
```
error: exception InterruptedException is never thrown in body of corresponding try statement
```
(This message was produced with JDK 21; the test checks the declared exception list, which is what makes the
compiler behave this way.) Test the result instead of catching an exception the call never throws:
```java
LlmResult result = client.complete("hi");
if (result.isFailure() && result.getError().error() instanceof CancelledError) {
// interrupted while blocked: Thread.currentThread().isInterrupted() is still true, so a
// loop or an executor further up the stack sees the interruption too
return null;
}
return result.getOrNull();
```
The flag is still set when the result comes back, so there is nothing to restore; clear it with `Thread.interrupted()`
only if you mean to carry on. The test runs this snippet against a call interrupted on a real provider's request
path and against a custom client that throws `InterruptedException`, and checks the flag in both cases.
### Interrupting the caller cancels an agent turn
All four blocking `JAgent` calls - `run`, `continueConversation`, `resume` and `recover` - cancel the turn they are
waiting on when the thread blocked in them is interrupted (since [#1330](https://github.com/llm4s/llm4s/issues/1330);
before, they stopped only the wait and the turn carried on in the background). The call returns a failed result
whose error is a `CancelledError`, with the interrupt flag still set; the turn's model call is interrupted, and the
conversation's thread is left for `recover`. The call returns once the cancelled turn has ended, so `recover` can
follow at once. It waits up to 5 seconds for the turn to end (with tracing, delivering the ended turn's last trace
events can add to that): a provider that ignores the interrupt for longer is logged at WARN and left to finish on its
own, and until it does the thread is busy (`ThreadBusy`). A thread that is already
interrupted when it calls one of them starts no turn. To stop a turn from Java, interrupt the thread that called it -
for example `cancel(true)` on the future of `runAsync`, `continueConversationAsync`, `resumeAsync` or `recoverAsync`,
which ends the turn as an interrupted blocking call does.
A turn that had already begun committing its outcome when the interrupt came cannot be cancelled: the call returns
that outcome - a completed or suspended turn - with the interrupt flag still set. `run(query)` starts a conversation
whose id a failed result does not carry, so a turn of it that fails or is cancelled is forgotten once it has ended.
A conversation you named, for `stream` say, is forgotten with `forget(threadId)`, which needs no result.
To cancel a turn without interrupting any thread, start it with `JAgent.stream`, `streamResume` or `streamRecover`
and call `cancel()` on the returned `AgentStream`. `cancel()` stops the turn and leaves the thread for `streamRecover`
or `recover`. Like the blocking calls, it waits up to 5 seconds for the turn to end; if the turn has not ended by then
(a provider that ignores the interrupt), it logs a WARN and returns, leaving that turn to finish on its own.
Kotlin matches Java: every `AgentKt` suspend function that runs a turn - `run`, `continueConversation`, `resume` and
`recover` - cancels the turn when the calling coroutine is cancelled (see
[Suspended turns from Java and Kotlin](agents/#suspended-turns-from-java-and-kotlin)), waiting up to 5 seconds for the
turn to end through `AgentStream.cancel()`, and the one-shot `run(query)` forgets the thread of a turn that failed or
was cancelled once it has ended; a conversation you named, for `stream` say, is forgotten with the suspend
`forget(threadId)`, which throws `LLMException` for a thread whose turn is still running. One difference remains, as coroutine cancellation requires: a Kotlin cancellation
that loses the race to a turn's commit still throws, though the turn's result is committed.
`ThreadingModelSpec` checks `run`; `JAgentPendingSpec` checks `continueConversation`, `resume`, `recover` and an
already-interrupted caller; `JAgentStreamSpec` checks `AgentStream.cancel()` and its 5-second bound; the blocking
calls' 5-second bound, the commit that beats a cancel and the forgotten one-shot and `runMultiTurn` threads are
checked by `AgentRunCancellationSpec` in `llm4s-agent`, and the Kotlin functions - their bound and `run`'s forgotten
thread included - by `AgentKtStreamTest`, `AgentKtPendingTest` and `AgentKtTest`.
## Timeouts
A call that gets no answer ends with a `TimeoutError`. The limit is set by each provider client: two minutes for a
non-streaming call in the Ollama
([`OllamaClient.scala`](https://github.com/llm4s/llm4s/blob/main/modules/ollama/src/main/scala/org/llm4s/llmconnect/provider/OllamaClient.scala)),
Gemini and Vertex AI clients and in the OpenAI-compatible clients, which allow five minutes for a streaming call
([`OpenAICompatibleClient.scala`](https://github.com/llm4s/llm4s/blob/main/modules/openai-compatible/src/main/scala/org/llm4s/llmconnect/provider/OpenAICompatibleClient.scala)).
Against a server that never answered, the Ollama client returned `TimeoutError: request timed out after 120
seconds`. That run took two minutes, so no test pins it. The OpenAI and Anthropic limits come from their SDKs and
were not checked. The limit cannot be configured in this release; [#712](https://github.com/llm4s/llm4s/issues/712)
tracks making it configurable.
## Shutting down
`JLlmClient` implements `AutoCloseable` and `close()` closes the provider client, so use it in a
try-with-resources block. It also shuts down the client's own executor of `*Async` calls, interrupting a call still
running; an executor you supplied with `withAsyncExecutor` is yours to shut down. The agent's run and task threads are virtual and so are daemon threads, and its graph writer
thread is a daemon thread
([`GraphRuntime.scala`](https://github.com/llm4s/llm4s/blob/main/modules/agent/src/main/scala/org/llm4s/agent/graph/GraphRuntime.scala)).
In a one-off check on JDK 21, a call through the Ollama provider left only daemon threads (the JDK
`HttpClient`'s selector and workers), and none were left after `close()`. No test pins that, because the set of
threads in a shared test JVM is not predictable. Forgetting to close a client therefore should not keep the JVM from
exiting, but close it anyway.
---
Source: https://llm4s.org/guide/observability/
# Monitoring LLM4S Applications
Observability and monitoring for LLM4S applications in production, covering latency, provider reliability, token usage, and cost visibility.
---
## Overview
Monitoring LLM applications differs from traditional service monitoring. Key concerns include:
- **Latency variability** - LLM calls range from milliseconds to minutes depending on model and prompt size
- **Provider reliability** - External API dependencies with rate limits and outages
- **Token usage & cost** - Direct correlation between usage and spend
- **Trace debugging** - Understanding multi-turn conversations and tool executions
- **Quality signals** - Beyond uptime: response relevance, hallucinations, guardrail triggers
LLM4S provides tracing infrastructure through multiple backends. Production monitoring typically combines tracing with your existing logging and alerting stack.
---
## Logging in Production
LLM4S logs through the SLF4J API and does not bring a logging backend: add one to your application, such as `libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.5.34"`. Without one, SLF4J logs nothing and prints a single warning. Configure the backend for structured JSON output in production environments.
### Logback Configuration
```xml
```
### What Gets Logged
At `INFO` level, LLM4S logs:
- Client initialization and shutdown
- Tracing backend connection status
- Configuration validation results
At `DEBUG` level (not recommended for production):
- Request/response payloads
- Token counts per request
- Tracing event details
### Capturing Provider Requests and Responses
To see the exact request body a client sent and the exact body that came back, opt in to
[Provider Exchange Logging](provider-exchange-logging): each completed call is handed to a sink, and a
ready-made sink appends JSON Lines to a file. It is off by default, and what it writes contains your
prompts and the model's answers, so read its privacy notes before enabling it.
---
## Tracing for Observability
LLM4S supports five tracing backends:
| Mode | Use Case | Configuration | Module |
|------|----------|---------------|--------|
| `langfuse` | Production LLM observability | `TRACING_MODE=langfuse` | `llm4s-observability` |
| `opentelemetry` | Integration with existing APM | `TRACING_MODE=opentelemetry` | `llm4s-observability-otel` |
| `console` (default) | Local development/debugging; metadata only unless `llm4s.tracing.console.captureContent = true` | `TRACING_MODE=console` | `llm4s-core` |
| `noop` | Disabled | `TRACING_MODE=noop` | `llm4s-core` |
| `collector` | In-process queryable store | Programmatic (see below) | `llm4s-observability` |
What each mode records, and which settings keep personal data out of traces, is in
[Data protection](../data-protection#logging-and-tracing-defaults).
All backends implement the `Tracing` trait and can be composed with `TracingComposer.combine()`.
`llm4s-core` holds the contract - `Tracing`, `TraceEvent`, `TracingComposer`, `TracingMode`, the
`TracingBackend` SPI - and builds only `console` and `noop` itself. Every other mode is a
`TracingMode.Named` (`TracingMode.Named("langfuse")`, `TracingMode.Named("opentelemetry")`) served
by a backend its module registers, so adding the dependency is all it takes:
(not compile-checked)
```scala
libraryDependencies ++= Seq(
"org.llm4s" %% "llm4s-observability" % llm4sVersion, // Langfuse, trace collector/store, CostTracker
"org.llm4s" %% "llm4s-observability-otel" % llm4sVersion // OpenTelemetry
)
```
On `0.4.1` and earlier, Langfuse, the collector and `CostTracker` ship inside `llm4s-core`.
### Configuration
Core reads only `llm4s.tracing.mode`, and hands the selected mode's block to its backend as
`TracingSettings.extras`. Each block, with its variable bindings, ships in the `reference.conf`
of the module that reads it - `langfuse` in `llm4s-observability`, `opentelemetry` in
`llm4s-observability-otel` - so the keys and variables below are unchanged, and apply once the
module is on the classpath:
```hocon
llm4s {
tracing {
mode = ${?TRACING_MODE}
langfuse {
url = ${?LANGFUSE_URL}
publicKey = ${?LANGFUSE_PUBLIC_KEY}
secretKey = ${?LANGFUSE_SECRET_KEY}
}
opentelemetry {
serviceName = ${?OTEL_SERVICE_NAME}
endpoint = ${?OTEL_EXPORTER_OTLP_ENDPOINT}
captureContent = false # ${?OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT}
}
}
}
```
### OpenTelemetry Integration
For teams with existing APM infrastructure (Jaeger, Grafana Tempo, Datadog), the `opentelemetry` mode (alias `otel`) exports traces via OTLP:
```bash
TRACING_MODE=opentelemetry
OTEL_SERVICE_NAME=my-llm-service
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
```
This requires the `llm4s-observability-otel` module:
(not compile-checked)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-observability-otel" % llm4sVersion
```
The module registers itself for `opentelemetry` through a `META-INF/services` entry, so the
dependency is all it takes. Its backend builds an `OpenTelemetryConfig` from the
`llm4s.tracing.opentelemetry` block with `OpenTelemetryConfig.fromExtras`.
#### GenAI semantic conventions
Model calls follow the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/),
version `1.37.0` (`OpenTelemetryTracing.SemconvVersion`, also the tracer's schema URL). The GenAI conventions are
still in Development status, so an attribute name may change when llm4s moves to a later version.
A plain or streamed completion is traced by wrapping the client in `TracingLLMClient`; an `EmbeddingClient` with a
tracer traces its own calls:
(not compile-checked)
```scala
import org.llm4s.trace.TracingLLMClient
val traced = TracingLLMClient(client, tracing, providerConfig) // request model and server.address from the config
val embeddings = EmbeddingClient.from("openai", embeddingConfig).map(_.withTracing(tracing))
```
| Span | Kind | Attributes |
|------|------|------------|
| `chat ` (a completion, plain or streamed) | `CLIENT` | `gen_ai.operation.name` = `chat`, `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.response.id`, `gen_ai.response.finish_reasons` (streamed), `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`, `gen_ai.request.temperature`, `gen_ai.request.top_p`, `gen_ai.request.max_tokens` (when set), `gen_ai.request.presence_penalty`, `gen_ai.request.frequency_penalty`, `server.address`, `llm4s.model_call.tool_calls`, `llm4s.model_call.streamed`; with `captureContent`, `gen_ai.input.messages` and `gen_ai.output.messages` |
| `embeddings ` | `CLIENT` | `gen_ai.operation.name` = `embeddings`, `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `operation` (the client's label), `input_count` |
| `chat ` / `embeddings ` of a failed call | `CLIENT`, status `ERROR` (the error message, redacted) | `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.request.model`, `error.type` (the `LLMError`'s type, e.g. `RateLimitError`) |
| `chat ` (an agent's model call) | `CLIENT` | `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.agent.name`, `gen_ai.conversation.id`, `gen_ai.usage.*`, `llm4s.model_call.attempt`, `llm4s.model_call.tool_calls`, `cost.usd` |
| `invoke_agent ` | `INTERNAL` | `gen_ai.operation.name`, `gen_ai.provider.name` (the run's first model call's), `gen_ai.agent.name`, `gen_ai.conversation.id`, `gen_ai.usage.*`, thread, run, status |
| `execute_tool ` | `INTERNAL` | `gen_ai.operation.name`, `gen_ai.tool.name`, `gen_ai.tool.call.id`; with `captureContent` (for `ToolExecuted`), `tool.input` and `tool.output` |
`gen_ai.provider.name` is the conventions' well-known value where there is one (`gemini` is `gcp.gemini`, `vertexai`
`gcp.vertex_ai`, `bedrock` `aws.bedrock`, `azure` `azure.ai.openai`, `watsonx` `ibm.watsonx.ai`, `mistral`
`mistral_ai`, `xai` `x_ai`), else the llm4s provider id (`openai`, `anthropic`, `ollama`, ...). A client reports its
provider through `LLMClient.providerId`.
**Message content is opt-in.** Prompts and completions often hold personal or confidential data, so by default no span
carries them, nor an agent's query or a tool's input and output. Set `llm4s.tracing.opentelemetry.captureContent = true`
(or `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true`) to record them as `gen_ai.input.messages` /
`gen_ai.output.messages`, in the conventions' JSON form; every piece of text passes through llm4s' secret redaction
first and is cut to 1000 characters.
`TokenUsageRecorded` remains an `INTERNAL` span `Token Usage - `: it records usage, not a call. GenAI
metrics (call duration and token usage) are not spans: [`llm4s-observability-micrometer`](micrometer) records them.
### Adding a Tracing Backend
A backend is a `TracingBackend` - the extension point `llm4s-observability` and
`llm4s-observability-otel` use - so adding one needs no change to llm4s:
(not compile-checked)
```scala
import org.llm4s.error.ConfigurationError
import org.llm4s.llmconnect.config.TracingSettings
import org.llm4s.trace.{ Tracing, TracingMode }
import org.llm4s.trace.spi.TracingBackend
import org.llm4s.types.Result
// A class with a public no-arg constructor - not an object: ServiceLoader instantiates it
final class DatadogTracingBackend extends TracingBackend:
val mode: TracingMode = TracingMode.Named("datadog")
def create(settings: TracingSettings): Result[Tracing] =
settings.extras.get("apiKey") match
case Some(key) => Right(new DatadogTracing(key, settings.extras.getOrElse("site", "datadoghq.com")))
case None => Left(ConfigurationError("llm4s.tracing.datadog.apiKey is not set"))
```
The backend's settings live under `llm4s.tracing.`, and it reads them from
`settings.extras`: that block for the selected mode, flattened to strings keyed by path within it
(`site`, `tags.team`). Only the selected mode's block is read. Ship the defaults and variable
bindings in the backend module's own `reference.conf`, which HOCON merges with the application's:
```hocon
llm4s.tracing.datadog {
site = "datadoghq.com"
site = ${?DD_SITE}
apiKey = ${?DD_API_KEY}
}
```
Declare it in `src/main/resources/META-INF/services/org.llm4s.trace.spi.TracingBackend`:
```
com.example.DatadogTracingBackend
```
`TRACING_MODE=datadog` then selects it. `Tracing.fromSettings(settings)` returns a
`ConfigurationError` when no backend serves the configured mode, and the backend's own error when
`create` fails; `Tracing.create(settings)` logs either and falls back to `NoOpTracing`. Where
services files do not survive packaging (some shaded jars), register the backend explicitly:
`Tracing.fromSettings(settings, TracingBackends.of(new DatadogTracingBackend))`.
---
## In-Process Trace Collection
`TraceCollectorTracing` + `InMemoryTraceStore` provide a fully queryable trace store
that runs entirely within the JVM — no external service required. They ship in
`llm4s-observability` (`org.llm4s.trace`, `org.llm4s.trace.store`, `org.llm4s.trace.model`). Recorded spans can be
filtered, paginated, and serialized to JSON.
This is the primary backend for **unit testing** and for **in-process analytics** (latency
breakdowns, cache hit rates, token cost per span kind).
### Quick Start
(not compile-checked)
```scala
import org.llm4s.trace._
import org.llm4s.trace.store._
import org.llm4s.trace.model._
val store = InMemoryTraceStore()
// apply returns Result[TraceCollectorTracing]; InMemoryTraceStore never fails
val tracer = TraceCollectorTracing(store).getOrElse(sys.error("tracing init failed"))
// give the tracer to the agent: every run it makes is traced
val result = for {
agent <- Agent.builder("assistant", client).withTools(tools).withTracing(tracer).build()
result <- agent.run("query")
} yield result
// retrieve all spans; an agent's runs are recorded as `graph.*` custom events,
// spans named `custom:graph.run_started`, `custom:graph.task_completed`, ...
val spans = store.getSpans(tracer.traceId)
```
### Querying Traces
`TraceStore.queryTraces` accepts a `TraceQuery` with optional filters and cursor-based
pagination. All filters combine with AND semantics.
(not compile-checked)
```scala
import java.time.Instant
// recent traces
val page = store.queryTraces(
TraceQuery.withTimeRange(Instant.now.minusSeconds(3600), Instant.now)
)
// failed traces only
val errors = store.queryTraces(TraceQuery.withStatus(SpanStatus.Error(""))).traces
// by metadata tag (e.g. experiment grouping)
val traceIds = store.searchByMetadata("experiment", "v2")
// combined filter with pagination
val q = TraceQuery(
startTimeFrom = Some(Instant.now.minusSeconds(3600)),
status = Some(SpanStatus.Error("")),
limit = 10
)
val first = store.queryTraces(q)
if (first.hasNext) {
val second = store.queryTraces(q.copy(cursor = first.nextCursor))
}
```
### Span Analytics
Every `Span` carries `startTime`, `endTime`, `kind`, `status`, and typed `attributes`,
making it straightforward to compute aggregates without an external system.
(not compile-checked)
```scala
val allSpans = store.getSpans(tracer.traceId)
// total milliseconds spent in LLM calls
val llmMs = allSpans
.filter(_.kind == SpanKind.LlmCall)
.flatMap(s => s.endTime.map(e => e.toEpochMilli - s.startTime.toEpochMilli))
.sum
// prompt tokens across the run
val promptTokens = allSpans
.filter(_.kind == SpanKind.LlmCall)
.flatMap(_.attributes.get("prompt_tokens").flatMap(_.asLong))
.sum
// semantic cache hit rate
val cacheSpans = allSpans.filter(_.kind == SpanKind.Cache)
val hitRate = if (cacheSpans.isEmpty) 0.0
else cacheSpans.count(
_.attributes.get("hit").flatMap(_.asBoolean).contains(true)
).toDouble / cacheSpans.size
```
### Deterministic Agent Testing
Wire `TraceCollectorTracing` in ScalaTest specs to assert on recorded spans without
mocking external services or parsing console output.
(not compile-checked)
```scala
import org.llm4s.trace.model.SpanKind
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers
import org.scalatest.BeforeAndAfterEach
class AgentBehaviourSpec extends AnyFlatSpec with Matchers with BeforeAndAfterEach {
val store = InMemoryTraceStore()
val tracer = TraceCollectorTracing(store).getOrElse(fail("tracing init failed"))
override def afterEach(): Unit = store.clear()
// built once per spec, with a scripted client and the tracer
def agent = Agent.builder("assistant", client).withTools(tools).withTracing(tracer).build()
.getOrElse(fail("agent build failed"))
/** One span per graph task the agent's `call-tool` node completed: one per tool call. */
def toolCalls = store.getSpans(tracer.traceId).filter { s =>
s.name == "custom:graph.task_completed" &&
s.attributes.get("nodeId").flatMap(_.asString).exists(_.endsWith("/call-tool"))
}
"agent" should "call a tool exactly once" in {
agent.run("what is 6 * 7?") shouldBe a[Right[_, _]]
toolCalls should have size 1
}
"agent" should "record no failed task on a valid query" in {
agent.run("hello") shouldBe a[Right[_, _]]
store.getSpans(tracer.traceId).map(_.name) should not contain "custom:graph.task_failed"
}
}
```
### Composing with External Backends
`TracingComposer.combine()` fans events out to multiple backends simultaneously.
Use this to keep a local in-process snapshot while also forwarding to Langfuse or OpenTelemetry.
(not compile-checked)
```scala
val store = InMemoryTraceStore()
val collector = TraceCollectorTracing(store).getOrElse(sys.error("tracing init failed"))
// LangfuseConfigLoader reads llm4s.tracing.langfuse whatever TRACING_MODE selects
val langfuse = LangfuseTracing.from(LangfuseConfigLoader.default().getOrElse(sys.error("bad langfuse config")))
val tracer = TracingComposer.combine(collector, langfuse)
val agent = Agent.builder("assistant", client).withTools(tools).withTracing(tracer).build()
agent.flatMap(_.run("query"))
// local span queries still work
val graphSpans = store.getSpans(collector.traceId).filter(_.name.startsWith("custom:graph."))
```
### Span JSON Round-Trip
`TraceModelJson` serialises and deserialises every span type losslessly via `ujson`.
Use this to write spans to disk, ship them to a custom HTTP endpoint, or reload them
for offline analysis.
(not compile-checked)
```scala
import org.llm4s.trace.model.TraceModelJson._
val json = span.toJson // ujson.Value
val parsed = TraceModelJson.parseSpan(json) // Result[Span]
parsed match {
case Right(s) => println(s.name)
case Left(error) => println(s"Parse error: ${error.field} — ${error.reason}")
}
```
---
## Langfuse Monitoring Workflow
Langfuse provides purpose-built LLM observability. When configured, LLM4S automatically traces:
### What Gets Captured
| Event Type | Description | Data Included |
|------------|-------------|---------------|
| **Traces** | Top-level request lifecycle | Query, final output, duration |
| **Generations** | Each LLM call | Model, prompt, completion, tokens, latency |
| **Spans** | Tool executions, retrieval ops | Tool name, input/output, duration |
| **Events** | Custom markers | User-defined metadata |
### Trace Structure Example
A typical RAG query produces this hierarchy:
```
Trace: "RAG Query Processing"
├── Span: "Document Retrieval" (200ms)
│ └── Event: "Retrieved 5 documents"
├── Generation: "Context Synthesis" (1,200ms)
│ ├── Model: gpt-4o
│ ├── Prompt tokens: 1,234
│ └── Completion tokens: 456
└── Event: "Response Delivered"
```
### What an agent run sends
An agent built with `Agent.builder(...).withTracing(tracing)` sends, for each run:
- `graph.*` custom events for the runtime's own events (run, task and checkpoint lifecycle), and
`agent.*` custom events for the agent's durable events (`agent.model_call_completed`,
`agent.tool_executed`, `agent.handed_off`, `agent.guardrail_blocked`), which carry no message content;
- a `TokenUsageRecorded` for each model call that reports usage;
- one `TraceEvent.AgentRunEnded` when the run ends: thread id, run id, the active agent, a status
(`completed`, `suspended`, `step_limit_reached`, `blocked:`, `cancelled`, `timed_out`,
`failed`), this turn's messages (from its user message on; empty when blocked, cancelled, timed
out or failed) and the run's own `UsageSummary` - the model calls this run made, summed from its
`agent.model_call_completed` events, not the thread's total, so per-run figures add up.
A run that crashes without a terminal event is traced as `ErrorOccurred`, with a WARN, and has no
`AgentRunEnded`; it is traced as soon as the run's last event has been, with no fixed delay.
Message content reaches tracing only in `AgentRunEnded.messages`; a blocked turn's content is not in
it. (The kernel's own `TracingSubscriber`, used for graphs that are not agents,
names agent events `graph.custom`; `withTracing` on an agent names them `agent.*`.)
Every one of these events is placed in one trace (`TraceEvent.span`, design 4.58): the run is a span, each event a
span under it - a tool call's spanning its duration - and a subagent's run is a span under the `task` tool call that
started it, whichever machine runs it. A run continues an upstream trace when given its context:
(not compile-checked)
```scala
import org.llm4s.trace.TraceContext
val upstream = TraceContext.parse(request.header("traceparent")).toOption
agent.run(threadId, query, RunConfig().withTraceContext(upstream), Nil)
```
The run's span then takes that context as its remote parent and keeps its trace id. A `resume` or `recover` given
no context continues the trace its work started in, which the thread's checkpoints record.
| Backend | What `AgentRunEnded` shows |
|---------|----------------------------|
| Langfuse | One trace per top-level run (trace id = the run's 32-hex trace id, the OpenTelemetry one; `run_id` in its metadata), grouped in a session per thread (session id = thread id); input is the first user message, output the last assistant message; metadata holds the agent, status and usage. The run is an `invoke_agent ` observation over its model calls (generations `chat `), tool calls (spans `execute_tool `), other events and one span per message; a subagent's run creates no trace and nests under the `execute_tool task` that started it |
| OpenTelemetry | An `INTERNAL` span `invoke_agent ` with `gen_ai.operation.name`, `gen_ai.agent.name`, `gen_ai.conversation.id`, thread, run, status (`ERROR` when failed, cancelled or timed out), message count, `gen_ai.usage.*` totals and, for a subagent's run, `llm4s.parent.thread_id` / `llm4s.parent.run_id`; it is the parent of the run's `chat ` (`CLIENT`, per-call `gen_ai.usage.*`) and `execute_tool ` (`INTERNAL`) spans and its other events, and a subagent's `invoke_agent` is under its `execute_tool task`. No attribute carries message content |
| `TraceCollector` | A `SpanKind.AgentCall` span with the same attributes, token totals as `input_tokens`/`output_tokens`, and no cost; like every event of the run it is stored in the run's own trace, with its span id, parent and start |
| Console | A multi-line "Agent Run Ended" block: agent, status, thread, run, message count, token totals |
### Environment Setup
Langfuse is served by `llm4s-observability`:
(not compile-checked)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-observability" % llm4sVersion
```
```bash
TRACING_MODE=langfuse
LANGFUSE_PUBLIC_KEY=
LANGFUSE_SECRET_KEY=
LANGFUSE_URL=https://cloud.langfuse.com # or self-hosted URL
```
Both keys are required. With either missing, `Tracing.fromSettings` returns a
`ConfigurationError` naming what is unset - e.g. `llm4s.tracing.langfuse.publicKey
(LANGFUSE_PUBLIC_KEY)` - and `Tracing.create` logs that error and falls back to `NoOpTracing`.
Without `llm4s-observability` on the classpath, the error names the artifact to add instead.
To read the Langfuse settings whatever `TRACING_MODE` selects (for example to combine Langfuse with
another tracer), use `LangfuseConfigLoader.default()` from `org.llm4s.config`.
### Tracing API
LLM4S exposes a `Tracing` trait for custom instrumentation:
(not compile-checked)
```scala
import org.llm4s.trace.{Tracing, TraceEvent}
import scala.concurrent.duration.*
// Trace custom events
tracing.traceEvent(TraceEvent.CustomEvent("cache_hit", ujson.Obj("key" -> "query_123")))
// Trace token usage explicitly
tracing.traceTokenUsage(usage, model = "gpt-4o", operation = "completion")
// Trace a finished agent run (an agent built withTracing does this itself, once per run)
tracing.traceEvent(TraceEvent.AgentRunEnded(threadId, runId, agent, status, messages, usage))
// Trace costs
tracing.traceCost(
costUsd = 0.0234,
model = "gpt-4o",
operation = "completion",
tokenCount = 1500,
costType = "total"
)
// Trace RAG operations
tracing.traceRAGOperation(
operation = "search",
duration = 150.millis,
embeddingTokens = Some(128),
llmPromptTokens = None,
llmCompletionTokens = None,
totalCostUsd = Some(0.0001)
)
```
---
## Metrics
Tracing records what happened inside one request. Metrics count what happens across all of them: requests, tokens, cost, errors and latency, per provider and model. They are cheap enough to alert on, and they answer "is a provider failing right now?" faster than a trace search.
The contract is `MetricsCollector` in `llm4s-core` (`@Stable`): `observeRequest`, `addTokens` and `recordCost`, plus calls for retries, circuit-breaker transitions, errors and image generation, and the generic `increment`, `observe` and `set` through which other modules record metrics of their own ([below](#agent-runtime-metrics)). Every chat client calls the collector it was built with, and collectors are expected not to throw. The default is `MetricsCollector.noop`, so metrics cost nothing until you pass one in. `llm4s-observability-prometheus` provides the implementation, `PrometheusMetrics`, and the HTTP endpoint that serves it, `PrometheusEndpoint`. If your application already has a Micrometer `MeterRegistry` (Spring Boot Actuator, Quarkus), use `llm4s-observability-micrometer` instead: its `MicrometerMetrics` records the same series into that registry, and the Spring Boot starter wires it with no code. See [Micrometer metrics](micrometer).
### Add the module
(not compile-checked)
```scala
libraryDependencies += "org.llm4s" %% "llm4s-observability-prometheus" % llm4sVersion
```
On `0.4.1` and earlier this code ships inside `llm4s-core`; the [migration note](../../reference/migration#slice-6-llm4s-observability-prometheus---prometheus-leaves-core) has the details. Package names are unchanged.
### Turn it on from configuration
Metrics are off unless asked for, because turning them on starts an HTTP server. To turn them on:
```hocon
llm4s {
metrics {
enabled = true
prometheus {
enabled = true
port = 9090
}
}
}
```
The module's `reference.conf` ships `enabled = false`, `prometheus.enabled = true` and `port = 9090`, so setting only `enabled = true` is enough: the Prometheus backend is used on port `9090`. `MetricsConfigLoader.default()` reads the block and returns `Result[(MetricsCollector, Option[PrometheusEndpoint])]`:
| `llm4s.metrics` | Collector | Endpoint |
|---|---|---|
| `enabled = false` (the default) | `MetricsCollector.noop` | none, no server is started |
| `enabled = true`, `prometheus.enabled = false` | `MetricsCollector.noop` | none |
| `enabled = true` | `PrometheusMetrics` | started on `prometheus.port`; `Left(ConfigurationError)` if the port is taken |
Pass the collector to the client when you build it:
(not compile-checked)
```scala
import org.llm4s.config.{Llm4sConfig, MetricsConfigLoader}
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.model.ModelRegistryService
val application = for {
providerConfig <- Llm4sConfig.defaultProvider()
registryService <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registryService
configured <- MetricsConfigLoader.default()
(metrics, endpoint) = configured
client <- LLMConnect.getClient(providerConfig, metrics).left.map { error =>
endpoint.foreach(_.stop()) // release the server if client creation fails
error
}
} yield (client, endpoint)
```
On success, retain the returned endpoint alongside the client and call `endpoint.foreach(_.stop())` during application shutdown. The collector and endpoint from the loader share the same registry.
`LlmClientOptions(metrics = ...)` is the same thing through the options overload of `LLMConnect.getClient`. The endpoint that `MetricsConfigLoader` starts stays up until you call `PrometheusEndpoint.stop()`, which is safe to call twice.
### Or build the pieces yourself
If you already own a `PrometheusRegistry`, create the collector on it and start the endpoint on the same registry:
(not compile-checked)
```scala
import io.prometheus.metrics.model.registry.PrometheusRegistry
import org.llm4s.metrics.{ PrometheusEndpoint, PrometheusMetrics }
val registry = new PrometheusRegistry()
val metrics = new PrometheusMetrics(registry)
val endpoint: Option[PrometheusEndpoint] =
PrometheusEndpoint.start(9090, registry) match { // Result[PrometheusEndpoint]; serves /metrics
case Right(ep) => Some(ep)
case Left(error) =>
// the port could not be bound: report this as a startup failure rather than
// running with metrics configured but nothing to scrape
None
}
// retain the handle: endpoint.foreach(_.stop()) at application shutdown (safe to call twice)
```
### What a call records
- **Every call**, successful or not, records one request with its latency and its outcome. A failure is classified into an error kind (`rate_limit`, `timeout`, `authentication`, `network`, `validation`, `service_error`, `execution_error`, `cancelled` or `unknown`) from the error the client returned.
- **Tokens and cost are recorded only on success**, and only when the completion carries them: tokens when the provider reported usage, cost when the client attached an estimated cost. A call with neither adds nothing to those series.
- **A streamed call is recorded once, when the stream has finished**, with the total latency, not as it goes.
- **Embedding calls are not recorded, and no RAG- or reranking-specific series exists.** The chat calls that RAG and the LLM reranker make through an instrumented client are recorded as ordinary requests - `RAG`'s answer generation and `LLMReranker`'s scoring both go through the client's `complete` - so that traffic does show up in the request series under the provider and model. The Cohere reranker calls Cohere's API directly, not through an LLM client, and is not recorded. Image generation is recorded through `InstrumentedImageGenerationClient`, which wraps an image client.
### The series
| Series | Type | Labels |
|---|---|---|
| `llm4s_requests_total` | counter | `provider`, `model`, `status` (`success` or `error_`) |
| `llm4s_tokens_total` | counter | `provider`, `model`, `type` (`input` or `output`) |
| `llm4s_cost_usd_total` | counter | `provider`, `model` |
| `llm4s_errors_total` | counter | `provider`, `error_type` |
| `llm4s_request_duration_seconds` | histogram | `provider`, `model`; buckets 0.1, 0.5, 1, 2, 5, 10, 30, 60 and 120 seconds |
| `llm4s_image_generations_total` | counter | `provider`, `model`, `operation`, `status` |
| `llm4s_images_generated_total` | counter | `provider`, `model` |
| `llm4s_image_generation_duration_seconds` | histogram | `provider`, `model`, `operation` |
| `llm4s_image_generation_cost_usd_total` | counter | `provider`, `model` |
| `llm4s_image_generation_errors_total` | counter | `provider`, `model`, `operation`, `error_type` |
After one successful call and one rate-limited call to `gpt-4o`, the chat series on `/metrics` look like this (comment lines left out; the second call took 90 seconds):
```text
llm4s_cost_usd_total{model="gpt-4o",provider="openai"} 0.0021
llm4s_errors_total{error_type="rate_limit",provider="openai"} 1.0
llm4s_request_duration_seconds_bucket{model="gpt-4o",provider="openai",le="0.1"} 0
llm4s_request_duration_seconds_bucket{model="gpt-4o",provider="openai",le="0.5"} 0
llm4s_request_duration_seconds_bucket{model="gpt-4o",provider="openai",le="1.0"} 0
llm4s_request_duration_seconds_bucket{model="gpt-4o",provider="openai",le="2.0"} 1
llm4s_request_duration_seconds_bucket{model="gpt-4o",provider="openai",le="5.0"} 1
llm4s_request_duration_seconds_bucket{model="gpt-4o",provider="openai",le="10.0"} 1
llm4s_request_duration_seconds_bucket{model="gpt-4o",provider="openai",le="30.0"} 1
llm4s_request_duration_seconds_bucket{model="gpt-4o",provider="openai",le="60.0"} 1
llm4s_request_duration_seconds_bucket{model="gpt-4o",provider="openai",le="120.0"} 2
llm4s_request_duration_seconds_bucket{model="gpt-4o",provider="openai",le="+Inf"} 2
llm4s_request_duration_seconds_count{model="gpt-4o",provider="openai"} 2
llm4s_request_duration_seconds_sum{model="gpt-4o",provider="openai"} 91.5
llm4s_requests_total{model="gpt-4o",provider="openai",status="error_rate_limit"} 1.0
llm4s_requests_total{model="gpt-4o",provider="openai",status="success"} 1.0
llm4s_tokens_total{model="gpt-4o",provider="openai",type="input"} 120.0
llm4s_tokens_total{model="gpt-4o",provider="openai",type="output"} 30.0
```
### Scrape it and alert on it
Point Prometheus at the endpoint:
```yaml
scrape_configs:
- job_name: 'llm4s'
static_configs:
- targets: ['localhost:9090']
```
Queries to start from (standard PromQL over the series above):
```promql
# Requests per second, by provider
sum by (provider) (rate(llm4s_requests_total[5m]))
# Share of requests that failed
sum(rate(llm4s_requests_total{status!="success"}[5m])) / sum(rate(llm4s_requests_total[5m]))
# 95th percentile latency, by provider
histogram_quantile(0.95, sum by (le, provider) (rate(llm4s_request_duration_seconds_bucket[5m])))
# Rate-limit errors per second
sum(rate(llm4s_errors_total{error_type="rate_limit"}[5m]))
# Estimated spend over the last hour, by model
sum by (model) (increase(llm4s_cost_usd_total[1h]))
```
### Send metrics to more than one place
`MetricsCollector.compose` forwards every `MetricsCollector` method, the two image-generation ones (`observeImageGeneration` and `recordImageGenerationCost`) included, to each collector it is given, so Prometheus can run next to `CostTracker` (in `llm4s-observability`) or a collector of your own, and a composed collector handed to `InstrumentedImageGenerationClient` records image metrics in every child. A collector that throws does not stop the others:
(not compile-checked)
```scala
import org.llm4s.metrics.MetricsCollector
val both = MetricsCollector.compose(metrics, myCollector)
// LLMConnect.getClient(providerConfig, both)
```
### Agent runtime metrics
The agent runtime (`llm4s-agent`) reports runs, claims, commits, child tasks and its run queue through the same
collector, as generic `Metric`s (`MetricsCollector.increment`, `observe` and `set`), so `PrometheusMetrics` and
`MicrometerMetrics` export them without depending on `llm4s-agent` ([design 4.59](https://github.com/llm4s/llm4s/blob/main/docs/design/typed-agent-runtime/4.59-stage-5-slice-o2-run-claim-queue-and-child-task-metrics.md)).
Two things record them: a `MeteredCheckpointer` wrapped around the store every runtime and worker of the process
shares, and an `AgentWorker` given the collector.
(not compile-checked)
```scala
import org.llm4s.agent.metrics.MeteredCheckpointer
val metered = MeteredCheckpointer(store.checkpointer, metrics) // every GraphRuntime and the worker use `metered`
// AgentWorker(queue, metered, agents, metrics = metrics)
```
| Metric (Prometheus series) | Type | Labels | Recorded by |
|---|---|---|---|
| `llm4s_agent_runs_started_total` | counter | `graph`, `trigger` (`start`, `resume`, `recover`) | checkpointer |
| `llm4s_agent_runs_ended_total` | counter | `graph`, `status` (`completed`, `suspended`, `failed`, `cancelled`, `timed_out`) | checkpointer |
| `llm4s_agent_run_duration_seconds` | histogram | `graph`, `status` | checkpointer |
| `llm4s_agent_claims_total` | counter | `result` (`acquired`, `taken_over`, `busy`, `failed`) | checkpointer |
| `llm4s_agent_claim_renewals_total` | counter | `result` (`renewed`, `lost`, `failed`) | checkpointer |
| `llm4s_agent_commit_duration_seconds` | histogram | `result` (`ok`, `conflict`, `claim_lost`, `failed`) | checkpointer |
| `llm4s_agent_tasks_started_total` | counter | `agent` | checkpointer |
| `llm4s_agent_tasks_ended_total` | counter | `agent`, `status` | checkpointer |
| `llm4s_agent_task_duration_seconds` | histogram | `agent`, `status` | checkpointer |
| `llm4s_agent_queue_depth` | gauge | `agent`, `state` (`waiting`, `leased`) | worker |
| `llm4s_agent_queue_oldest_seconds` | gauge | `agent` | worker |
| `llm4s_agent_queue_wait_seconds` | histogram | `agent` | worker |
| `llm4s_agent_queue_attempts_total` | counter | `agent`, `result` (`recorded`, `exhausted`, `thread_deleted`) | worker |
| `llm4s_agent_queue_plans_total` | counter | `agent`, `plan` (`start`, `recover`, `close`) | worker |
| `llm4s_agent_queue_deferrals_total` | counter | `agent`, `reason` (`busy`, `unrunnable`) | worker |
| `llm4s_agent_queue_lease_expiries_total` | counter | `agent` | worker |
| `llm4s_agent_queue_lease_lapses_total` | counter | `agent` | worker |
| `llm4s_agent_queue_lease_losses_total` | counter | `agent`, `reason` (`refused`, `unrenewed`) | worker |
| `llm4s_agent_queue_closed_total` | counter | `agent`, `outcome`, `reason` | worker |
| `llm4s_agent_sweep_passes_total` | counter | `result` (`done`, `skipped`, `failed`) | worker |
| `llm4s_agent_sweep_threads_examined_total`, `..._tombstones_reclaimed_total`, `..._entries_reclaimed_total` | counter | none | worker |
| `llm4s_agent_worker_drained_total` | counter | `result` (`finished`, `cancelled`, `abandoned`) | worker |
Dead letters are `llm4s_agent_queue_closed_total{outcome="failed",reason="attempts_exhausted"}`. The queue gauges are
store-wide: every worker of a fleet reports the same numbers, every `llm4s.agent.worker.metrics.queueDepthEvery`
(15 seconds), so aggregate them with `max`. `graph` and `agent` are ids your code fixes; no thread, run, entry or
tenant id is ever a label, and no metric carries message content. Every backend caps a label at
`MetricsCollector.MaxLabelValues` (256) distinct values and records the rest as `other`.
### Limits
- **Retries, circuit-breaker transitions and `ReliableClient`'s error events have no series of their own.** `ReliableClient` takes its own `collector` argument and reports those three kinds of event to it, but `PrometheusMetrics` does not implement them. The request series still see every attempt: `ReliableClient` calls the wrapped client once per attempt, and a metrics-enabled client records each of those calls, so a call that fails twice and then succeeds adds three requests - two `error_*`, one `success` - and three latency observations. Request and error rates are per attempt, not per logical call.
- **The latency buckets are fixed** (0.1 to 120 seconds); there is no way to configure them.
- **Counters live in the process.** A restart starts them from zero, which is what `rate()` and `increase()` expect.
- **The endpoint has no authentication and no TLS**: llm4s starts it with only a port and a registry. Keep it off the public network and let your network policy decide who can scrape it.
- **Labels are provider and model.** The number of series grows with the number of distinct models you call, not with traffic.
---
## Health Checks
### Startup Validation
Use `client.validate()` during application startup to fail fast on misconfiguration:
(not compile-checked)
```scala
val clientResult = for {
config <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
client <- LLMConnect.getClient(config)
_ <- client.validate()
} yield client
clientResult match {
case Left(error) =>
logger.error(s"LLM client validation failed: $error")
System.exit(1)
case Right(client) =>
logger.info("LLM client ready")
}
```
### Readiness Probes
For Kubernetes deployments, include LLM connectivity in readiness checks:
(not compile-checked)
```scala
// Integrate with your framework's health check mechanism
def isLLMReady(): Boolean =
client.validate().isRight
```
### Liveness vs Readiness
- **Liveness**: Application process is healthy (standard JVM checks)
- **Readiness**: Can serve LLM requests (includes `client.validate()`)
Separating these prevents pod restarts during temporary provider outages.
---
## Provider Reliability Signals
### Failures to Monitor
| Signal | Meaning | Action |
|--------|---------|--------|
| `429 Too Many Requests` | Rate limited | Back off, check quotas |
| `503 Service Unavailable` | Provider outage | Failover or queue |
| Connection timeout | Network issue | Retry with backoff |
| Response timeout | Slow generation | Increase timeout or reduce prompt |
### Logging Provider Failures
(not compile-checked)
```scala
import org.llm4s.error._
def logProviderFailure(error: LLMError): Unit = error match {
case rle: RateLimitError =>
logger.warn(s"Rate limited: ${rle.message}, retry after: ${rle.retryAfter.fold("unknown")(_.toString)}")
case ServiceError(message, httpStatus, provider, requestId, retryAfter) =>
logger.error(s"Provider API error [$httpStatus]: $message")
case NetworkError(message, cause, endpoint) =>
logger.error(
s"Network failure from $endpoint: $message" +
cause.map(c => s" - ${c.getMessage}").getOrElse("")
)
case timeout: TimeoutError =>
logger.warn(s"Request timeout: ${timeout.message}")
case other =>
logger.error(s"LLM error: $other")
}
```
### Alerting Thresholds
Recommended alert conditions for LLM services:
| Metric | Warning | Critical |
|--------|---------|----------|
| Error rate | > 1% | > 5% |
| P95 latency | > 10s | > 30s |
| Rate limit events | > 10/min | > 50/min |
| Daily token spend | > 80% budget | > 95% budget |
---
## Token & Cost Monitoring
### Token Usage Tracking
LLM4S traces token usage through the tracing infrastructure. With Langfuse, usage appears automatically in the dashboard.
For programmatic access:
(not compile-checked)
```scala
completion.usage match {
case Some(usage) =>
logger.info(
s"Tokens - prompt: ${usage.promptTokens}, " +
s"completion: ${usage.completionTokens}, " +
s"total: ${usage.totalTokens}"
)
// Trace for aggregation
tracing.traceTokenUsage(usage, model, "completion")
case None =>
logger.warn("Token usage not available from provider")
}
```
### Cost Estimation
Pricing varies by provider and changes over time. For cost tracking:
- **Langfuse** automatically calculates costs when model pricing is configured in its dashboard
- **Provider billing APIs** offer authoritative usage and spend data
- **Manual tracking** can use `tracing.traceCost()` with your own pricing logic
(not compile-checked)
```scala
// Track cost through tracing (pricing logic is your responsibility)
tracing.traceCost(
costUsd = estimatedCost,
model = "gpt-4o",
operation = "completion",
tokenCount = usage.totalTokens,
costType = "total"
)
```
### Budget Awareness
Use context budget methods to prevent runaway costs:
(not compile-checked)
```scala
import org.llm4s.agent.{Agent, ContextWindowConfig}
import org.llm4s.agent.graph.middleware.ContextWindowMiddleware
import org.llm4s.types.HeadroomPercent
// Get available tokens considering model limits and safety margin
val budget = client.getContextBudget(HeadroomPercent.Standard)
val config = ContextWindowConfig(maxTokens = Some(budget))
// ContextWindowMiddleware uses a default token counter (words * 1.3)
// or accepts a custom tokenCounter function for more accurate estimation
val agent = Agent.builder("assistant", client)
.withMiddleware(new ContextWindowMiddleware(config))
.build()
```
---
## Production Checklist
Before deploying, verify monitoring coverage:
### Tracing
- [ ] `TRACING_MODE` set to `langfuse` or `opentelemetry`
- [ ] Tracing credentials configured and tested
- [ ] Traces visible in dashboard (test with sample request)
### Metrics
- [ ] `llm4s.metrics.enabled = true`, and Prometheus scrapes the `/metrics` endpoint
- [ ] The endpoint is reachable only from your monitoring network
- [ ] Error-ratio and latency alerts use `llm4s_requests_total` and `llm4s_request_duration_seconds`
### Logging
- [ ] Structured JSON logging enabled
- [ ] Log levels appropriate (`INFO` for `org.llm4s`)
- [ ] Logs shipping to aggregation system
### Alerting
- [ ] Error rate alerts configured
- [ ] Latency (P95/P99) alerts configured
- [ ] Rate limit event alerts configured
- [ ] Cost/budget alerts configured
### Health
- [ ] `client.validate()` called on startup
- [ ] Readiness probe includes LLM connectivity
- [ ] Graceful degradation for provider outages
---
## Known Limitations
Current monitoring limitations in LLM4S:
- **Prometheus metrics cover LLM and image-generation calls, not traces** (see [Metrics](#metrics)) - `PrometheusMetrics` (`llm4s-observability-prometheus`) records requests, tokens, cost, errors and latency through `MetricsCollector`; for span-level data, read `InMemoryTraceStore` and push to your Prometheus client, or implement a custom `TraceStore` that writes directly to a metrics registry
- **No automatic cost aggregation** - Langfuse provides this via its dashboard; for in-process aggregation, sum `cost_usd` attributes from `SpanKind.Rag` and `SpanKind.LlmCall` spans in `InMemoryTraceStore`
- **No real-time streaming metrics** - Streaming completions are traced on completion, not in-flight
- **Guardrail metrics require custom tracing** - Add `traceEvent` calls in guardrail implementations; the resulting spans are then queryable via `InMemoryTraceStore`
These are tracked in the [Production Readiness Roadmap](../../reference/roadmap).
---
## Related Documentation
- [Provider Exchange Logging](provider-exchange-logging) - Capture raw provider requests and responses for debugging
- [In-Process Tracing Use Cases](enhanced-tracing-use-cases.md) - Full catalogue of `TraceCollectorTracing` + `InMemoryTraceStore` scenarios
- [Langfuse Workflow Patterns](../../langfuse-workflow-patterns.md) - Detailed trace event sequences
- [Configuration Guide](../../getting-started/configuration) - Complete configuration reference
- [Roadmap](../../reference/roadmap) - Planned observability improvements
---
Source: https://llm4s.org/guide/observability/provider-exchange-logging
# Provider Exchange Logging
Capture the raw request and response of each provider call, to a file or to your own code, to debug what was sent and what came back.
---
> **Privacy warning.** An exchange holds the request and response bodies as the provider saw them: your
> prompts, the conversation history, tool arguments and results, any documents you retrieved into the
> prompt, and the model's answers. They routinely contain personal data and business data. The file sink
> redacts credentials by pattern, on a best-effort basis, and cannot recognise personal data; **a sink
> you write receives the bodies untouched.** Treat everything this feature writes as sensitive: keep it
> off unless you are debugging, protect and delete what it writes, and do not share it unreviewed.
> See [Redaction and truncation](#redaction-and-truncation) and [Using it safely](#using-it-safely).
## What it is
Exchange logging records, for each call a client makes to a provider, one `ProviderExchange`: the JSON
body it sent, the body it got back (or the error), the timing and the outcome. It answers "what exactly
did the provider receive, and what exactly did it say?", which tracing does not: tracing records what an
agent did, not the wire.
It is **off by default** and **opt in per client**. You turn it on with a `ProviderExchangeLogging`
setting carried by `LlmClientOptions`, and the setting names a `ProviderExchangeSink` that receives each
completed exchange. LLM4S ships a sink that appends JSON Lines to a file; you can write your own.
## What is recorded
A `ProviderExchange` has these fields:
| Field | Type | What it holds |
|-------|------|---------------|
| `exchangeId` | `String` | A random UUID for this exchange |
| `provider` | `String` | The client's provider name, such as `deepseek` |
| `model` | `Option[String]` | The model the client is **configured** with, not the model the response reports (a router can serve another) |
| `requestId`, `correlationId` | `Option[String]` | Reserved: no built-in client sets them, so they are always `None` |
| `startedAt`, `completedAt` | `Instant` | When the call began and when it finished |
| `duration` | `FiniteDuration` | The time between the two |
| `outcome` | `ProviderExchangeOutcome` | `Success` or `Error`; see the limitations for `Cancelled` |
| `requestBody` | `String` | The JSON body sent to the provider |
| `responseBody` | `Option[String]` | The body received: for a streamed call, the raw server-sent events; for an HTTP error, the error body; `None` when no response arrived |
| `errorMessage` | `Option[String]` | The error's message, when the call failed |
Headers are not recorded, so an API key that travels in a header is not in an exchange.
These clients record both `complete` and `streamComplete`: OpenAI, Azure and Requesty
(`llm4s-openai`); Anthropic; Gemini and Vertex AI; Ollama; the OpenAI-compatible clients (DeepSeek, Z.ai,
OpenRouter, Mistral, Cohere and the generic `openai-compatible` provider); Bedrock; and watsonx.
Embedding, image and speech clients do not record exchanges.
## Turning it on
### From configuration
```hocon
llm4s {
exchangeLogging {
enabled = true
dir = "exchange-logs"
}
}
```
The two keys have variables bound in `reference.conf`: `LLM4S_EXCHANGE_LOGGING_ENABLED` and
`LLM4S_EXCHANGE_LOGGING_DIR`. `enabled` defaults to `false`, and there is no default directory:
`enabled = true` without a `dir` is a `ConfigurationError` ("Provider exchange logging is enabled but
llm4s.exchangeLogging.dir is missing").
**Nothing reads this section on its own.** `LLMConnect.getClient(config)` uses
`LlmClientOptions.default`, which has logging disabled. Read the setting with
`Llm4sConfig.exchangeLogging()` and hand it to the client:
(not compile-checked)
```scala
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.{ LLMClient, LLMConnect, LlmClientOptions }
import org.llm4s.model.ModelRegistryService
import org.llm4s.types.Result
def fromConfiguration(): Result[LLMClient] =
for {
providerConfig <- Llm4sConfig.defaultProvider()
registry <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registry
exchangeLogging <- Llm4sConfig.exchangeLogging()
client <- LLMConnect.getClient(providerConfig, LlmClientOptions(exchangeLogging = exchangeLogging))
} yield client
```
When logging is enabled, `Llm4sConfig.exchangeLogging()` creates the directory if it is missing and the
file for this run immediately, so a directory that cannot be written is reported at start-up, not at the
first call.
### From code
`ProviderExchangeSink.createRunScopedJsonl` creates the file sink; `ProviderExchangeLogging.enabled` wraps
a sink in the setting:
(not compile-checked)
```scala
import java.nio.file.Path
import org.llm4s.llmconnect.{ ProviderExchangeLogging, ProviderExchangeSink }
import org.llm4s.types.Result
def jsonlLogging(dir: Path): Result[ProviderExchangeLogging] =
ProviderExchangeSink.createRunScopedJsonl(dir).map(ProviderExchangeLogging.enabled)
```
Any sink works the same way: wrap it in `ProviderExchangeLogging.enabled` and give it to the client in
`LlmClientOptions`:
(not compile-checked)
```scala
import org.llm4s.llmconnect.{ LLMClient, LLMConnect, LlmClientOptions, ProviderExchangeLogging, ProviderExchangeSink }
import org.llm4s.llmconnect.config.ProviderConfig
import org.llm4s.model.ModelRegistryService
import org.llm4s.types.Result
def clientFor(providerConfig: ProviderConfig, sink: ProviderExchangeSink)(using
ModelRegistryService
): Result[LLMClient] =
LLMConnect.getClient(providerConfig, LlmClientOptions(exchangeLogging = ProviderExchangeLogging.enabled(sink)))
```
## The JSON Lines file
`createRunScopedJsonl` creates one file for the run, named from the time it was created:
`provider-exchanges-2026-10-07T10-20-30Z.jsonl`. If that name is taken (two runs in the same second), it
adds a number: `...Z-2.jsonl`, `...Z-3.jsonl`. It creates missing directories, and returns a
`ConfigurationError` if the path is a file. The file exists, empty, as soon as the sink is created; each
exchange is appended as one line when the call completes. A line, from a real run against a local test
server that answered 401:
```json
{"exchange_id":"bd1e7d34-3c95-44e5-a9be-428bf3ca85f1","provider":"deepseek","model":"deepseek-chat","request_id":"","correlation_id":"","started_at":"2026-10-07T10:27:19.668236Z","completed_at":"2026-10-07T10:27:19.676649Z","duration_ms":"8","outcome":"Error","request_body":"{\"model\":\"deepseek-chat\",\"messages\":[{\"role\":\"user\",\"content\":\"will fail\"}],\"temperature\":0.7,\"top_p\":1}","response_body":"{\"error\":\"Unauthorized\"}","error_message":"Authentication failed for deepseek: Unauthorized. Check that the key is this provider's, and that its project or organisation has access to the model. See https://llm4s.org/getting-started/troubleshooting#authentication-failed"}
```
The keys are always these twelve, in this order. A value that is absent is an empty string, not a missing
key or `null`. `duration_ms` is a JSON **string** (`"8"`), so read it as one. `request_body`,
`response_body` and `error_message` are the text of the exchange, escaped into a JSON string, after the
redaction and truncation below. The sink does not rotate or limit the file, and does not set permissions
on it.
## Redaction and truncation
The file sink changes `request_body`, `response_body` and `error_message` before writing them:
- **Redaction.** It replaces, with `[REDACTED]`: an `Authorization`, `Proxy-Authorization`, `Cookie` or
`Set-Cookie` header, from after the `:` and the whitespace after it to the end of its line (for a
cookie, every name and value of it; see below for where a cookie header's value ends inside a string),
bearer tokens, URL query
parameters with sensitive names (`api_key`, `token`, `password` and similar), JSON fields with sensitive
names (also when the JSON sits inside a prompt or response string with escaped quotes,
`\"api_key\": \"...\"`, or is single-quoted, or is cut off before its closing quote, as a truncated
payload leaves it), numbers under such a key (including exponent forms such as `1e10`, and inside a
string too; the number is written back as the string `"[REDACTED]"`, so the JSON still parses, or as
`'[REDACTED]'` under a single-quoted key), a double-quoted value under a single-quoted key, as Python's
`repr` writes a string that holds a `'` (`{'password': "it's"}`, and `\"it's\"` inside a string), where
it reads as a value of the dict (followed by `,` and the next key, by `}`, or cut off), arrays
and objects under such a key (`{"token": ["..."]}`, `{"credentials": {"user": "...", "pass": "..."}}`:
every leaf under the key is replaced, however deep - strings, numbers and, outside a string, bare
words - also inside a string and when the key or the leaves are single-quoted, as a Python dict is,
`{'token': ['...']}`; the brackets, the keys of nested objects, `true`, `false` and `null` are kept, so
the JSON still parses and keeps its shape; inside a string a single-quoted container ends where the
string does, so a `'token': [` that a message merely mentions does not take the fields after it, nor
the prose after an apostrophe in it (`it's`) unless escaped JSON (`\"`) follows it, and an unclosed
`'password': '` there ends where the string ends rather than taking the rest of the document - but only
where no `'` that could close it follows anywhere in the input, and not where that end would leave the
value empty (`'password': '",`); otherwise it runs to the next `'`, or to the end of the input, as it
does outside a string, so a credential that holds a `"` (`'Qx"]]9secret'`) is redacted whole; a field
under a single-quoted key is also read in the input as it was given, so that a double-quoted value before
it that runs over its key - a `"password": "` that a Python dict's message only mentions, `{'note': 'see
"password": " here', 'apiKey': [...]}` - cannot hide it: what that reading replaces is added where
nothing else was replaced, and every other replacement is made as before; outside
any string, the bare words after an unclosed `'token': [` are replaced to the end of the input, since nothing tells them from leaves; under
a single-quoted key a single-quoted leaf that stands where a value does is replaced whatever it holds,
a connection string `'postgres://u:...@h/db'` or a `'k=v'` token included, and so is a `\"`-quoted leaf
there, `\"{'token': [\\\"...\\\"]}\"`, when its own `\"` closes it where a value does; any other `"`
under a single-quoted key inside a string ends the container, since it may be the end of a string
inside it; a Python literal is also read as Python writes it, wherever a dict key `'password':` or a
keyword argument `token=[...]` / `password='...'` stands: each string, number and word of the value is
replaced, its quotes, brackets and the dict's keys kept, a string ending only at its own quote written the
way the opening one is - so `{'password': "x'y", 1: 2}` and a value before any next key or none, as a dict
cut off after its `,` has it, a `pytest` assertion diff whose lines start with `E` and its `+`/`-` markers,
a notebook's `text/plain` line array in an `.ipynb` file, `str(dict)` inside a `repr`
(`{'body': '{\'password\': "it\'s..."}'}`), Python's `\'` inside a JSON string
(`"{'password': 'a\\'b\"c'}"`), `Settings(token=['a', 'b'])`, a call under a sensitive key
(`Settings(token=SecretStr('...'))`, `api_key=module.Secret(value='...')`) and a `\"` in an earlier leaf are
redacted; a `"` of a credential in a single-quoted string is replaced with it (`'[REDACTED]'`), while a
`"` that delimits a JSON string, at any depth, is never replaced, so the JSON still parses, and what this
reading replaces is added only where no other replacement is), `key=value` pairs and quoted
`KEY="value"` / `KEY='value'` assignments outside a query string (for example `password=...`,
`spring.datasource.password=...`, `PASSWORD="..."`), also with the quotes escaped by backslashes, as a
logfmt line inside a JSON string writes them (`{"message": "login password=\"...\" ok"}`, `\'...\'`, and
`\\\"...\\\"` one string deeper: the content is replaced and the escaped quotes kept, so the JSON still
parses; an escaped quote never closed ends where the enclosing string or the line does; a value in quotes behind
an even run of backslashes, `password=\\'p&ss=...\\'` inside a JSON string, is redacted up to the same quote
written the same way, `&word=` included), also with `=` and `&`
written as the JSON escapes Gson's default HTML-safe output uses (`password\u003d...\u0026user\u003d...`) and in
a log written into a JSON string, where the key follows the escape of a line break or a tab
(`...\ntoken\u003d...`, `\r\nAuthorization=...`, also one string deeper); a value that holds a literal `&` with no
other pair after it is redacted on both sides of the `&` (`password=p&ssW0rd` becomes
`password=[REDACTED]&[REDACTED]`, while `password=abc&user=bob` keeps `user=bob`); a `Bearer` or `Basic` scheme
word right before a pair does not hide the pair's value (`Bearer password=...` becomes
`[REDACTED]=[REDACTED]`), `key: value` header lines (for example
`x-api-key: ...`), and strings shaped like known provider API keys (for example `sk-` keys). A quoted
value is redacted whole, escaped quotes included. A key is sensitive when its whole name, lower-cased
with `_` and `-` dropped, is `token`, `authorization`, `proxyauthorization`, `credential(s)`,
`cookie(s)` or `setcookie`, or ends in `apikey`, `secret`, `password`, `passwd`, `privatekey`,
`accesstoken`, `refreshtoken`, `idtoken`, `authtoken`, `sessiontoken`, `securitytoken` or
`bearertoken`, or the plural `secrets`, `passwords` or `apikeys`: `client_secret`, `x-api-key`, `api-key`,
`x-goog-api-key`, `x-auth-token`, `secrets`, `api_keys`, `db_passwords`,
`x-amz-security-token`, `refresh_token`, `db_password`, `Proxy-Authorization`, `Cookie` and
`Set-Cookie` are redacted, as a header line, a JSON field or header map (escaped too) and a
`key=value` pair; `max_tokens`, `prompt_tokens`, `token_count`, `next_page_token`, `cookie_policy` and
`cookie_consent` are not, because the match is never on a substring; nor is `tokens`, a count in every
provider exchange. URL query parameters are also
matched against a second, substring list (`key`, `token`, `secret`, `password`, `auth` and similar), so
`?x_api_key_id=` is redacted too, and a third, matched whole, of parameters that carry a short-lived
credential: `sig`, `signature`, `X-Amz-Signature`, `X-Goog-Signature`, `code` (an OAuth authorization code),
`session`, `sessionid` and `jsessionid` (also as the matrix parameter `;jsessionid=`, and in a URL's fragment,
`#code=...`, and after an HTML-escaped separator, `?sv=...&sig=...`); a field `"code": 400`
is left, since only a query parameter's key is read so. The password of a URL's userinfo is redacted wherever the URL is
(`https://alice:[REDACTED]@gw.example.com/v1`, also with JSON's `\/`), every URL of a list joined by `,` or
`;` included; the user and the host are kept. A query value written right before JSON
(`?token=...,{"api_key": "..."}`, also with the quotes escaped) does not hide the field after it.
- **Where a cookie header's value ends.** Outside any string (a raw request dump, a log line), at the end
of its line, quotes, apostrophes and backslashes included: `Cookie: sid="abc"; x=y` becomes
`Cookie: [REDACTED]`. Inside a string that opened on the same line - a JSON string, a header in a JSON
array, an HTTP exchange logged inside JSON, a Python repr - earlier: at the escape of a line break
(`\r`, `\n`, `\u000a`, `\u000d`, and `\\r`, `\\n` before the next header in JSON inside a JSON
string) or, in a double-quoted string, at the unescaped `"` that ends the string. So
`{"log":"GET / HTTP/1.1\r\nCookie: sid=...\r\nAccept: */*"}` becomes
`{"log":"GET / HTTP/1.1\r\nCookie: [REDACTED]\r\nAccept: */*"}`, and
`["Cookie: sid=...", "Accept: */*"]` keeps its second element; the JSON still parses. An escaped quote
(`sid=\"abc\"`) is part of the value. Ending at the quote is a heuristic for well-formed JSON
strings, where the first unescaped `"` ends the string: when that quote is not followed by what follows
the end of a JSON string (`}`, `]`, or `,` and a value, then the end of the line; or `:` and a value,
where the header is an object's key), the line is not well-formed JSON - a stray quote earlier on it,
`size 5" Cookie: sid="abc"; x=y`, or quoted cookie values in a log line - and the value runs to the
end of the line, quotes and all, as outside a string: `msg="Cookie: theme="dark" sid=abc"` becomes
`msg="Cookie: [REDACTED]`. Cookie headers are redacted last, after every other pattern, so they only
add to what is redacted: a cookie value that takes a quote of a malformed line cannot change how the
field patterns read the quotes around it, and a `password="..."` or `{'password': "..."}` field later
on that line or the next is still redacted. An `Authorization` or `Proxy-Authorization` header's value
always runs to the end of its line, inside a string too, taking the rest of that string and the fields
after it on the same line: a Digest credential holds quoted strings and commas, so a quote is no sure
end of it.
- **Truncation.** It keeps the first 1000 characters, after redaction, and appends
`... [truncated, N chars omitted]` with the count it dropped. Redact the full text before you cut it, as
this sink and every other llm4s call site do: redaction of text that is already cut off is weaker. An
unclosed single-quoted value inside a raw (unescaped) double-quoted string, holding a `"` followed by what
follows a string's end and with no `'` after it that could close it, ends at that `"`, so the part after
it is shown - `msg="{'password': 'Qx"]]9secretPW` cut off there becomes
`msg="{'password': '[REDACTED]"]]9secretPW`. A closing `'` with a letter or digit on both sides
(`'Qx"]]9SECRETPW'it"`) reads as an apostrophe, not as the end of the value, so the same happens there.
Both are limits you should know about:
- **The file is not a full record.** A prompt or an answer longer than 1000 characters is cut, so a long
conversation or a long completion is only partly in the file. For complete bodies, write your own sink.
- **Redaction recognises patterns, not meaning.** It does not look for personal data: an email address or
a phone number is written as it is. Text you put in a prompt is inside a JSON string in the request
body, and a credential there is not guaranteed to match a pattern: a secret that is not under a key (a
key pasted into a prompt as prose), and JSON escaped twice (JSON inside a string inside a string,
`\\\"api_key\\\": ...`) are not redacted, by decision ([#1576](https://github.com/llm4s/llm4s/issues/1576)):
a pattern cannot know what a secret is, and only one level of escaping is recognised. A URL query value
that holds a `'` before `,`, `)`, `;` or `:` (`?token=ab',cd`) is redacted only up to that quote, since it
cannot be told from the quote that ends a string, as in `fetch('...?token=ab')`. Do not rely on
redaction to make the file safe to share.
A sink you write is **not** redacted or truncated: it is handed the exchange exactly as the client built
it, including the full bodies. If your sink writes bodies anywhere, redaction is your job.
## Writing your own sink
A sink implements one method:
(not compile-checked)
```scala
trait ProviderExchangeSink:
def record(exchange: ProviderExchange): Unit
```
The client calls it once the call has finished, inline, on the thread that made the call (the tests behind
this guide check that for the OpenAI-compatible clients), so keep it fast. If it throws, the client discards the exception: the call still succeeds, and nothing logs the failure, so a
broken sink is silent. Catch and report your own errors.
A sink that keeps the metadata and never touches a body is the safest way to see what your application
sends and how long it takes:
(not compile-checked)
```scala
import org.llm4s.llmconnect.{ ProviderExchange, ProviderExchangeSink }
final class MetadataOnlySink(out: String => Unit) extends ProviderExchangeSink:
def record(exchange: ProviderExchange): Unit =
out(s"${exchange.provider} ${exchange.model.getOrElse("-")} ${exchange.outcome} ${exchange.duration.toMillis}ms")
```
With `out = println`, a call prints `deepseek deepseek-chat Success 180ms`. To keep bodies only when
something went wrong, wrap another sink:
(not compile-checked)
```scala
import org.llm4s.llmconnect.{ ProviderExchange, ProviderExchangeOutcome, ProviderExchangeSink }
final class ErrorsOnlySink(delegate: ProviderExchangeSink) extends ProviderExchangeSink:
def record(exchange: ProviderExchange): Unit =
if exchange.outcome == ProviderExchangeOutcome.Error then delegate.record(exchange)
```
## Using it safely
- **Keep it off in production** unless you are chasing a specific problem, and turn it off again after.
For production monitoring use [tracing and metrics](index), which record what an agent did and what
it cost rather than the provider's wire format.
- **Protect the directory.** The sink creates the file with the process's default permissions; restrict
access to the directory yourself, and delete old files, because nothing rotates or expires them.
- **Keep the logs out of version control.** The samples' default directory,
`.devlog/logs/provider-exchanges`, is not in the repository's `.gitignore`; if you enable logging while
running the samples, add the directory to yours before you commit.
- **Mind the cost.** The file sink opens, appends and closes the file for each exchange, on the thread
making the call.
## Known limitations
- The file sink truncates bodies to 1000 characters; see above.
- Redaction is pattern-based and best effort; it does not detect personal data. A secret that is not
under a key, and JSON escaped twice, are not redacted.
- `requestId` and `correlationId` are never filled by the built-in clients.
- `ProviderExchangeOutcome` declares `Cancelled`, but the recorder produces only `Success` and `Error`: a
cancelled call is recorded as an `Error` carrying the cancellation message.
- A sink that throws fails silently.
- Embedding, image and speech clients do not record exchanges.
- Nothing enables logging from configuration automatically; you pass the setting to the client.
- `duration_ms` in the file is a string.
## Related
- [Monitoring](index): tracing, logging and health checks for production
- [Writing a Provider](../writing-a-provider): how a provider records its exchanges with
`ProviderExchangeRecorder`
- [Configuration](../../getting-started/configuration): the full configuration reference
---
Source: https://llm4s.org/guide/observability/micrometer
# Micrometer metrics
Record LLM4S request, token, cost and error metrics in a Micrometer `MeterRegistry` you already have, so a Spring Boot,
Quarkus or plain JVM service shows them next to its own metrics.
---
## When to use it
Use `llm4s-observability-micrometer` when your application already ships metrics through Micrometer (to Prometheus,
Datadog, CloudWatch, OpenTelemetry or anything else a registry exports to) and you want the LLM metrics in the same
place. If you only want a Prometheus endpoint and have no Micrometer registry, use
[`llm4s-observability-prometheus`](index) instead.
The two modules are independent: neither depends on the other, and `llm4s-core` depends on neither. Core defines only
the `MetricsCollector` contract that every client accepts.
## Install
(not compile-checked)
```scala
// same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-observability-micrometer" % llm4sVersion
```
It depends on `llm4s-core` and `micrometer-core`. The Micrometer jars from 1.14 to 1.17 are class-file version 52, which any
JDK can load, so the module does not raise the JDK you need. The module is built against Micrometer 1.17 and uses only the stable instrument API
(`Counter`, `Timer`, `MeterRegistry`); your application's own Micrometer version is the one that runs.
## Quick start
Create a `MicrometerMetrics` over your registry and give it to the client:
(not compile-checked)
```scala
import io.micrometer.core.instrument.MeterRegistry
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.metrics.MicrometerMetrics
import org.llm4s.model.ModelRegistryService
def clientWithMetrics(meterRegistry: MeterRegistry) =
for {
providerConfig <- Llm4sConfig.defaultProvider()
registryService <- Llm4sConfig.modelRegistryService()
given ModelRegistryService = registryService
metrics = new MicrometerMetrics(meterRegistry)
client <- LLMConnect.getClient(providerConfig, metrics)
} yield client
```
Every call through that client now records into `meterRegistry`. To read a value back, for example in a test:
(not compile-checked)
```scala
val calls = meterRegistry.find("llm4s.requests").tag("status", "success").counter().count()
```
## What is recorded
| Meter | Type | Tags | Prometheus series |
|---|---|---|---|
| `llm4s.requests` | counter | `provider`, `model`, `status` | `llm4s_requests_total` |
| `llm4s.request.duration` | timer | `provider`, `model` | `llm4s_request_duration_seconds` |
| `llm4s.tokens` | counter | `provider`, `model`, `type` | `llm4s_tokens_total` |
| `llm4s.cost.usd` | counter | `provider`, `model` | `llm4s_cost_usd_total` |
| `llm4s.errors` | counter | `provider`, `error_type` | `llm4s_errors_total` |
| `llm4s.reliability.errors` | counter | `provider`, `error_type` | `llm4s_reliability_errors_total` |
| `llm4s.retries` | counter | `provider` | `llm4s_retries_total` |
| `llm4s.circuit.breaker.transitions` | counter | `provider`, `state` | `llm4s_circuit_breaker_transitions_total` |
| `llm4s.image.generations` | counter | `provider`, `model`, `operation`, `status` | `llm4s_image_generations_total` |
| `llm4s.images.generated` | counter | `provider`, `model` | `llm4s_images_generated_total` |
| `llm4s.image.generation.duration` | timer | `provider`, `model`, `operation` | `llm4s_image_generation_duration_seconds` |
| `llm4s.image.generation.cost.usd` | counter | `provider`, `model` | `llm4s_image_generation_cost_usd_total` |
| `llm4s.image.generation.errors` | counter | `provider`, `model`, `operation`, `error_type` | `llm4s_image_generation_errors_total` |
The last column is what a Micrometer `PrometheusMeterRegistry` exposes for the meter. Tag values:
- `status` is `success` or `error_