User Guide

Comprehensive guides for LLM4S features. They describe the current release, 0.5.1 (Scala 3.7.x, JDK 21); new to LLM4S? Start with Getting Started, and upgrading from 0.4.x, with Upgrading from 0.4.1 to 0.5.0.

Since 0.5.0 each feature below is its own module next to llm4s-core (the agent runtime in llm4s-agent, RAG in llm4s-rag, each provider in its own module, and so on); each guide names the module it needs, and Installation lists them all.

Available Guides

Fundamentals

  • Basic Usage - Get started with LLM calls, client creation, and error handling
  • Providers - Overview of supported providers and how to configure them
  • Java - Call a model from Java with llm4s-java-api: a client, a conversation and failures you can read without Scala
  • Java AI services - Declare an annotated interface and let llm4s-java-api implement it: prompt templates, typed answers, memory and tools (experimental)
  • Java RAG - Ingest documents and ask questions from Java with llm4s-java-rag: answers with their sources
  • Moving a Java app from LangChain4j or Spring AI - Each LangChain4j and Spring AI concept in llm4s-java-api terms, with compiled before-and-after Java, provider configuration and the gaps
  • Java Reactor - Stream a reply as a Reactor Flux<String> with llm4s-reactor, for Spring WebFlux and other reactive pipelines
  • Spring Boot - llm4s-spring-boot-starter: properties, an LLM4STemplate bean, asynchronous calls and a health indicator
  • Kotlin - llm4s-kotlin-api: suspend functions and Flows over the Java API (Experimental; not on Maven Central for 0.5.1 yet, so built locally for now)
  • Error Handling - Work with Result[A] and LLMError: pattern matching, for-comprehensions, every error type, recovery and testing
  • Testing with the Testkit - Script an LLMClient in your tests: canned replies, tool calls, injected errors, recorded requests
  • Typed Prompts - prompt"..." and system"...": values rendered by a type class checked at compile time, and untrusted input fenced
  • Structured Output - completeStructured: typed replies from a schema, which providers enforce it, and what a bad reply looks like
  • Writing a Provider - Publish your own provider module against the llm4s-core SPI
  • Comparing LLM4S - A sourced comparison with LangChain4j, Spring AI, Koog and Embabel, and a migration guide
  • Coming from sttp-ai or openai-scala-client - Your Scala client code next to the same thing in LLM4S, compiled: completions, typed output, tools, streaming, providers, and when to stay on your client
  • Caching - Cache embeddings (exact) and model responses (semantic): configuration, keys, TTL, eviction and what the cache reports
  • Multimodal Content - Send images and documents in a user message or a tool result, and check before the request whether the provider takes them
  • JSON Libraries - Use circe, play-json, zio-json or jsoniter-scala with LLM4S: converting at the boundary, structured output and tools
  • Replacing a Python Sidecar - Move the LLM logic of a Python sidecar into your Scala service: Python-served models (vLLM, TGI, llama.cpp) as an OpenAI-compatible section, Python-only functions as agent tools over MCP, and a staged migration

Agent Framework

  • Agents Overview - Build LLM-powered agents with tools, guardrails, and multi-turn conversations
    • Guardrails - Input/output validation for safety and quality
    • Memory System - Persistent context and knowledge across conversations
    • Handoffs - Agent-to-agent delegation for specialist routing
    • Streaming Events - Real-time execution feedback for responsive UIs
  • Context Window Pruning - Keep a long conversation within the model’s context window
  • Built-in Tools - Calculator, date and time, UUID, JSON, files, HTTP, shell and web search tools, the bundles that hold them, and what each one can do
  • Build Your Own Tool - Write a tool with ToolBuilder: the Schema DSL, reading arguments, returning errors, registering it with an agent and testing it
  • Derived Schemas - Write the type once: llm4s-schema-derivation builds the completeStructured schema from a case class with derives.
  • Tapir Endpoints as Tools - llm4s-tapir turns the tapir endpoints a service already has into tools, with each schema derived from its endpoint

RAG & Semantic Search

  • Embeddings - Choose a provider and model, create an EmbeddingClient, embed documents and queries, and keep dimensions in step with the store
  • Vector Store - Complete RAG toolkit for semantic search and retrieval
    • Vector Backends: SQLite (in-memory/file), PostgreSQL/pgvector, Qdrant
    • Keyword Backends: SQLite FTS5, PostgreSQL native full-text search
    • Hybrid Search: BM25 keyword + vector fusion with RRF strategy
    • Reranking: Cohere cross-encoder for result refinement
    • Document Chunking: Sentence-aware + simple chunking strategies

  • Query Transformers - Rewrite the user’s query before retrieval: LLMQueryRewriter, IdentityTransformer, chaining, error behaviour and custom transformers

  • RAG Evaluation - Measure and improve RAG quality
    • RAGAS Metrics: Faithfulness, answer relevancy, context precision/recall
    • Benchmarking Harness: Compare chunking, fusion, and embedding strategies
    • Optimization Workflow: Data-driven RAG improvement
  • Permission-Based RAG - Enterprise access control for RAG
    • Hierarchical Collections: Organize documents by tenant, team, or project
    • Two-Level Permissions: Collection-level queryableBy + document-level readableBy
    • Pattern Queries: *, path/*, path/** for flexible collection scoping
    • Principal Management: Map users/groups to efficient integer IDs

Multimodal Capabilities

  • Image Generation - Generate images with DALL-E and other providers
  • Speech - Speech-to-text (STT) and text-to-speech (TTS)
  • Jev decision model - Ask TypeSafe’s Jev typed questions (Noul, Choice, Score) and get typed answers: a decision client, not a chat model

Effect Systems

  • Pick your stack - One program in plain Result, cats-effect, ZIO and Pekko side by side: the semantics every bridge shares, and the differences
  • cats-effect - LLMClientIO and AgentIO for cats-effect IO and fs2 streaming
  • ZIO - LLMClientZ and AgentZ for ZIO 2 and ZIO Streams
  • Apache Pekko - LLMClientPekko and AgentPekko for Pekko Streams
  • Spark and Databricks - enrich a DataFrame from a Spark job: a client per partition, ReliableClient, errors as data
  • Apache Flink - enrich a Flink 2.x stream from Scala 3: AsyncDataStream, a client per task built in open(), dead letters as a side output
  • Kafka - enrich a Kafka topic with fs2-kafka: bounded concurrency, retries, a dead-letter topic, offsets committed after the produce

Java

Observability

  • Monitoring - Production monitoring for LLM4S applications
    • Tracing: Langfuse and OpenTelemetry integration
    • Logging: Structured JSON logging with SLF4J/Logback
    • Provider Exchange Logging: Capture raw provider requests and responses for debugging
    • Health Checks: Startup validation and readiness probes
    • Cost Monitoring: Token usage tracking and budget awareness

Feature Coverage via Examples

For features not yet documented as dedicated guides, see our Examples Gallery which includes 69 working examples:

Feature Examples Section
Basic LLM Calling Basic Examples
Multi-Turn Conversations Context Management Examples
Agent Framework Agent Examples
Tool Calling Tool Examples
Guardrails & Safety Guardrails Examples
Agent Handoffs Handoff Examples
Memory System Memory Examples
Streaming Streaming Examples
Embeddings & RAG Embeddings Examples
MCP Integration MCP Examples
Observability Monitoring Guide

Design Documents

For in-depth technical documentation, see our design documents:

Getting Help

Table of contents