# LLM4S > LLM4S (Large Language Models for Scala) is a type-safe Scala 3 framework for LLM-powered applications: multi-provider chat and embeddings, tools, agents, RAG and observability, with Java and Kotlin APIs. Version: 0.5.1 Full text of every page below: https://llm4s.org/llms-full.txt. Rules for code written with a coding assistant: https://llm4s.org/llm4s-rules.md (https://llm4s.org/ai-assistants). ## Getting started - [Getting Started](https://llm4s.org/getting-started/): Get up and running with LLM4S in minutes. - [Installation](https://llm4s.org/getting-started/installation): Get LLM4S up and running in minutes. - [First Example](https://llm4s.org/getting-started/first-example): Build your first LLM-powered application in minutes. - [Configuration](https://llm4s.org/getting-started/configuration): Configure LLM4S for multiple providers, environments, and use cases. - [Next Steps](https://llm4s.org/getting-started/next-steps): You've completed the getting started guide! Here's where to go next. - [Ollama Quick Start](https://llm4s.org/getting-started/ollama-quickstart): Run LLM4S with Ollama for **free, local LLM inference** - no API keys required! - [Gradle Integration](https://llm4s.org/getting-started/gradle): Add LLM4S to a Gradle project (Java, Kotlin, or Kotlin/Ktor). - [Testing Guide](https://llm4s.org/getting-started/testing-guide): Learn how to test LLM-powered applications effectively without spending money on API calls. - [Scala CLI Quick Start](https://llm4s.org/getting-started/scala-cli): A first LLM call from a single file, with no sbt project. - [Maven Quick Start](https://llm4s.org/getting-started/maven): A complete pom.xml that compiles and runs a first call from Java, with llm4s-java-api. - [JBang Quick Start](https://llm4s.org/getting-started/jbang): A first LLM call from one Java file, with no build tool and no project. - [First-Run Troubleshooting](https://llm4s.org/getting-started/troubleshooting): The problems people most often meet when an LLM4S program does not work the first time, each with the symptom, the cause and the fix. The message texts below are the ones the code produces (the cla... - [Using LLM4S from Scala 2.13](https://llm4s.org/getting-started/scala-2-13): Scala 2.13: call-only, via the TASTy reader. A Spark job, a Play application or an Akka or Pekko service can call LLM4S's Scala 3 artifacts today. - [Mill Quick Start](https://llm4s.org/getting-started/mill): A first LLM call, and a test of it that needs no API key, in a Mill project. - [AI Coding Assistants](https://llm4s.org/ai-assistants): Point Copilot, Claude Code, Cursor and Codex at llm4s: llms.txt, llms-full.txt and a rules file that keeps generated code compiling. ## User guide (including Java and Kotlin) - [User Guide](https://llm4s.org/guide/): Comprehensive guides for LLM4S features. They describe the current release, 0.5.1 (Scala 3.7.x, JDK 21); new to LLM4S? Start with Getting Started, and upgrading from 0.4.x, with Upgrading from 0.4.... - [Basic Usage](https://llm4s.org/guide/basic-usage): Learn the fundamentals of LLM4S: creating clients, making LLM calls, and handling results. - [Providers](https://llm4s.org/guide/providers): LLM4S supports multiple LLM providers out of the box. Choose your provider, configure it, and start building. - [Vector Store](https://llm4s.org/guide/vector-store): Low-level vector storage abstraction for RAG and semantic search. - [Permission-Based RAG](https://llm4s.org/guide/permission-based-rag): Enterprise-grade access control for RAG applications. - [Error Handling](https://llm4s.org/guide/error-handling): How to work with Result[A] and LLMError in practice. - [Structured Output](https://llm4s.org/guide/structured-output): Ask a model for a typed value instead of free text, and know what you get back when it does not comply. - [Spring Boot](https://llm4s.org/guide/spring-boot): Add llm4s-spring-boot-starter and inject an LLM4STemplate: the client, an executor for asynchronous calls and a health indicator are configured from your application.properties. - [Image Generation](https://llm4s.org/guide/image-generation): The LLM4S library provides a powerful and flexible API for generating images using different providers like Stable Diffusion (via a local WebUI) and the Hugging Face Inference API. - [Speech (STT/TTS)](https://llm4s.org/guide/speech): A comprehensive speech recognition and text-to-speech synthesis module for the LLM4S project, built with Scala and functional programming principles. - [Built-in Tools](https://llm4s.org/guide/builtin-tools): Ready-made tools for agents: a calculator, the date and time, UUIDs, JSON, files, HTTP, a shell and web search. - [Context Window Pruning](https://llm4s.org/guide/context-window-pruning): Complete guide to LLM4S context window management and pruning strategies. - [Agents](https://llm4s.org/guide/agents/): Build sophisticated AI agents with tools, guardrails, memory, and multi-agent coordination. - [Guardrails](https://llm4s.org/guide/agents/guardrails): Validate agent inputs and outputs for safety, quality, and compliance. - [Memory System](https://llm4s.org/guide/agents/memory): Persistent context and knowledge for agents across conversations. - [Handoffs](https://llm4s.org/guide/agents/handoffs): Delegate queries to specialist agents for domain expertise. - [Streaming Events](https://llm4s.org/guide/agents/streaming): Watch an agent's turn as it happens: the answer token by token, each tool call, each model call. - [Durable Checkpointers](https://llm4s.org/guide/agents/durable-checkpointers): Share one checkpoint store between runtimes and processes, recover a run whose process died, inspect, fork and correct a thread's checkpoint history, prune it, and test a store of your own against ... - [Human Review](https://llm4s.org/guide/agents/human-review): Pause an agent's turn for a person: to approve a tool call, answer a question, review an answer, or look at what a step did before the next one runs. - [Delegation](https://llm4s.org/guide/agents/delegation): Hand a bounded task to another agent and get back only its typed report. - [Running agents on several machines](https://llm4s.org/guide/agents/multiple-machines): Run one agent service as several instances that share threads through one PostgreSQL database: any instance can start, follow, cancel or delete any thread, a run whose machine dies is picked up by ... - [Real-World Application Patterns](https://llm4s.org/guide/patterns/): Learn production-grade patterns for building scalable, reliable LLM applications with LLM4S. These guides cover architecture decisions, implementation strategies, and best practices based on real-w... - [Multi-Agent Orchestration](https://llm4s.org/guide/patterns/multi-agent-orchestration): Learn how to design systems where multiple agents collaborate, delegate tasks, and handle complex workflows. This guide covers agent communication patterns, handoff mechanisms, and failure recovery... - [RAG for Enterprise](https://llm4s.org/guide/patterns/rag-enterprise): Learn production-ready strategies for building Retrieval-Augmented Generation (RAG) systems at scale. This guide covers document ingestion, hybrid search, cost optimization, and quality assurance. - [Production Monitoring](https://llm4s.org/guide/patterns/production-monitoring): Comprehensive strategies for observing, debugging, and optimizing LLM applications in production. This guide covers metrics collection, alerting, cost tracking, and debugging techniques. - [Scaling Strategies](https://llm4s.org/guide/patterns/scaling-strategies): Production patterns for handling high throughput, reducing latency, and distributing load. This guide covers caching, rate limiting, batch processing, and distributed execution. - [Error Recovery](https://llm4s.org/guide/patterns/error-recovery): Resilience strategies for handling failures, recovering from errors, and gracefully degrading functionality. This guide covers retries, circuit breakers, fallbacks, and failure scenarios. - [Security Best Practices](https://llm4s.org/guide/patterns/security-best-practices): Comprehensive security patterns for protecting sensitive data, managing credentials, and maintaining audit trails in production LLM systems. - [Writing a Provider](https://llm4s.org/guide/writing-a-provider): How to publish your own LLM4S provider module - a chat client, an embedding provider, or both - against llm4s-core, without changing LLM4S itself. - [cats-effect Integration](https://llm4s.org/guide/cats-effect): The llm4s-effect module wraps the synchronous LLMClient and Agent APIs in cats-effect IO (or any Async[F] type), keeping the compute thread pool free while long-running LLM calls block on the dedic... - [ZIO Integration](https://llm4s.org/guide/zio): The llm4s-zio module wraps the synchronous LLMClient and Agent APIs in ZIO effects, shifting blocking LLM calls to ZIO's blocking thread pool and surfacing LLMError directly in the ZIO error channel. - [Caching](https://llm4s.org/guide/caching): Avoid paying twice for the same work: an embedding cache for exact repeats, and a semantic cache for model responses. - [JSON Libraries](https://llm4s.org/guide/json-libraries): LLM4S reads and writes JSON with uPickle and its ujson AST. This page shows how an application that already uses circe, play-json or zio-json passes its own types in and gets them back, without wri... - [RAG Evaluation](https://llm4s.org/guide/rag-evaluation): Systematically measure and improve your RAG pipeline quality using RAGAS metrics and the benchmarking harness. - [Java](https://llm4s.org/guide/java): Call a model from Java with llm4s-java-api: a client, a prompt, a conversation, and failures you can read without Scala. - [Java Threading and Cancellation](https://llm4s.org/guide/java-threading-and-cancellation): Which thread a call blocks, whether one client can be shared, how to run calls on virtual threads and what an interrupt does, for code that uses llm4s-java-api. - [Monitoring](https://llm4s.org/guide/observability/): Observability and monitoring for LLM4S applications in production, covering latency, provider reliability, token usage, and cost visibility. - [Provider Exchange Logging](https://llm4s.org/guide/observability/provider-exchange-logging): Capture the raw request and response of each provider call, to a file or to your own code, to debug what was sent and what came back. - [Micrometer metrics](https://llm4s.org/guide/observability/micrometer): Record LLM4S request, token, cost and error metrics in a Micrometer MeterRegistry you already have, so a Spring Boot, Quarkus or plain JVM service shows them next to its own metrics. - [RAG Query Transformers](https://llm4s.org/guide/rag-query-transformers): Rewrite the user's query before it is embedded and sent to the vector store, improving retrieval quality for vague or conversational questions. - [Apache Pekko Integration](https://llm4s.org/guide/pekko): The llm4s-pekko module (Beta) exposes LLMClient.streamComplete and an Agent run as Apache Pekko Streams Sources, so LLM output plugs into a Pekko, Pekko HTTP or Play pipeline with the backpressure ... - [Jev decision model](https://llm4s.org/guide/jev): llm4s-jev is a client for Jev, TypeSafe's System One model. Jev is **not a chat model**. You send it a *state* and named, typed *questions*, and it answers each question with a typed answer your co... - [Comparing LLM4S](https://llm4s.org/guide/comparison): This page compares LLM4S with LangChain4j, Spring AI, Koog and Embabel, so that you can decide quickly whether it fits your project. It states facts and trade-offs, not rankings. - [Coming from LangChain4j or Spring AI](https://llm4s.org/guide/migrating-from-langchain4j): If you know LangChain4j or Spring AI, this page maps the concepts you already use to LLM4S, with working code for the five things most programs do: call a model, get typed output, give the model a ... - [Derived Schemas](https://llm4s.org/guide/derived-schemas): Write the type once and let llm4s-schema-derivation build the JSON schema for structured output, and typed tools, from it. - [Testing with the Testkit](https://llm4s.org/guide/testing-with-the-testkit): llm4s-testkit is a scriptable fake LLMClient for the tests of **your** code. It answers from a script you write, records what your code sent, and never touches the network, so a test that calls an ... - [Multimodal content](https://llm4s.org/guide/multimodal): Send images and documents to a model in a user message or a tool result, and know before the request goes out whether the provider can take them. - [Java RAG](https://llm4s.org/guide/java-rag): Ingest documents and ask questions about them from Java with llm4s-java-rag: answers come with the chunks they were drawn from. - [Spark and Databricks](https://llm4s.org/guide/spark): Enrich a DataFrame with LLM calls from a Spark job: one client per partition, retries and a rate limit from ReliableClient, and failed calls kept as data instead of failing the stage. - [Embeddings](https://llm4s.org/guide/embeddings): Turn text into vectors: pick a provider and a model, create an EmbeddingClient, and embed documents and queries. - [Custom Tools](https://llm4s.org/guide/custom-tools): Write a tool with ToolBuilder: declare its parameters, read them safely, return a result or an error, and test it without a model. - [Java Reactor](https://llm4s.org/guide/java-reactor): Stream a model's reply as a Reactor Flux with llm4s-reactor, for Spring WebFlux and other reactive pipelines. - [Serving an Agent over HTTP](https://llm4s.org/guide/http-reference-apps): modules/examples-http holds small, tested reference apps that expose an llm4s agent from a Scala HTTP stack. Each serves the same two endpoints, so they can be compared: - [Quarkus](https://llm4s.org/guide/quarkus): Call a model from a Quarkus application: a CDI producer for the JLlmClient, its settings in application.properties, and a REST endpoint that keeps the blocking call off the event loop. - [Data Protection](https://llm4s.org/guide/data-protection): What personal data llm4s moves, to where, what it keeps, and what it logs by default: the facts a records-of-processing entry, a DPIA or a data-residency review needs. - [Modular RAG Example](https://llm4s.org/guide/modular-rag-example): This guide introduces a practical, modular Retrieval-Augmented Generation (RAG) example for LLM4S. ## Cookbook - [Cookbook](https://llm4s.org/examples/cookbook): Complete programs for the things you would actually build, each on one page: the problem, the program, how to run it, what to change for a real provider, and the pitfalls. Every recipe is a small S... - [Classify text into an enum](https://llm4s.org/examples/cookbook/structured-output): Turn a customer message into a Ticket whose category is one of a fixed set. - [Extract structured data from an email](https://llm4s.org/examples/cookbook/email-extraction): Read an order change out of a free-text email into a case class with a list of line items. - [Summarise a long document](https://llm4s.org/examples/cookbook/summarise): Cut a document into parts, summarise each part, then summarise the summaries (map-reduce). - [Answer questions over a folder of files](https://llm4s.org/examples/cookbook/folder-qa): Ingest every file in a folder into the RAG pipeline, retrieve the passages for a question, and answer from them. - [An agent that calls two tools](https://llm4s.org/examples/cookbook/tool-calling): Give an agent an exchange-rate tool of its own and the built-in calculator, and let the model chain them. - [A guardrailed chatbot](https://llm4s.org/examples/cookbook/guardrails): Reject a bad request before the model sees it, and an over-long answer before the user does. - [Stream tokens to the terminal](https://llm4s.org/examples/cookbook/streaming): Print each piece of the reply as the model produces it, and get the whole completion at the end. - [Fall back between providers](https://llm4s.org/examples/cookbook/fallback): Retry a provider with ReliableClient, and ask a second provider when the first still fails. - [Cache repeated calls](https://llm4s.org/examples/cookbook/caching): Wrap a client in CachingLLMClient so that a question asked again is answered without a model call. - [Evaluate an answer with a judge](https://llm4s.org/examples/cookbook/judge): Ask a model a question, then have a judge model grade the answer against a reference, from 1 to 5. - [Answer questions with keyword search](https://llm4s.org/examples/cookbook/document-qa): Chunk some text, put it in an in-memory keyword index (BM25), retrieve the best passage and answer from it. - [Remember facts between turns](https://llm4s.org/examples/cookbook/memory): Record what a user says, and put the relevant facts in the system prompt when the next question arrives. - [Several agents in one graph](https://llm4s.org/examples/cookbook/multi-agent-graph): Ask two specialist agents in parallel, then let an editor agent combine their views once both have answered.