Skip to main content Link Menu Expand (external link) Document Search Copy Copied

Why LLM4S

AI features belong in the systems you already trust

Your services, data pipelines and operations run on the JVM. LLM4S lets you add agents, tools and retrieval there, in typed Scala, instead of standing up and securing a second stack beside them.

Get started Read the comparison

The JVM

Why build LLM applications on the JVM

Most LLM tooling is Python-first, while many production systems in finance, telecom, data platforms and enterprise software are built on Scala and the JVM.

  • No Python sidecar

    The LLM code runs in the same process, build and deployment as the rest of your service: one language to review, one artifact to ship, one runtime to patch.

  • Operations you already have

    Your logging, metrics, tracing, secrets handling, CI and security review apply unchanged. LLM4S reports through OpenTelemetry, Prometheus and Micrometer.

  • Concurrency that scales

    LLM calls are slow I/O. On JDK 21, LLM4S runs tools on virtual threads and integrates with cats-effect, ZIO, Pekko Streams and Reactor for streaming and cancellation.

Scala

Why Scala suits LLM work

Model output is untrusted, provider calls fail in many ways, and agents are state machines. Scala's type system makes each of those explicit.

  • Failures as values. A Result[A] with typed error classes makes rate limits, timeouts, refused requests and unreadable answers part of the signature, not a surprise in production.
  • Types as schemas. Case classes describe what you expect back from a model and what a tool accepts; derives SchemaOf turns them into the JSON schema the provider enforces (Experimental).
  • Immutable conversations and agent state. A run is data you can serialise, checkpoint, replay and test.
  • Effect systems if you want them. AgentIO for cats-effect and AgentZ for ZIO, or plain Result if you do not.

What you get

What LLM4S gives a team

  • providers

    Multi-provider through one SPI

    Sixteen chat providers on main: OpenAI, Azure OpenAI, Anthropic, Gemini, Vertex AI, Ollama, Mistral, Cohere, DeepSeek, Z.ai, xAI, OpenRouter, Requesty, Amazon Bedrock, IBM watsonx.ai and any OpenAI-compatible endpoint, plus five embedding providers. Each is a module found on the classpath.

    Providers

  • agents

    Agents with guardrails

    Input and output guardrails, including LLM-as-judge, handoffs, delegation, human review, streaming events and a typed graph runtime with durable checkpoints in SQLite or PostgreSQL.

    Agents

  • rag

    RAG and memory

    Loaders, chunking, pgvector, Qdrant, SQLite and Elasticsearch stores, hybrid search, reranking, permission-aware retrieval, GraphRAG and RAGAS-style evaluation.

    Vector stores

  • mcp

    Tools and MCP

    A tool API with typed tools, built-in tools, and Model Context Protocol client and server with stdio, SSE and Streamable HTTP transports, bearer-token auth and public-bind protection.

    Tools

  • observability

    Observability and cost

    Tracing through OpenTelemetry or Langfuse, metrics through Prometheus or Micrometer, a trace collector, cost tracking and provider exchange logging.

    Observability

  • interop

    Java, Kotlin and frameworks

    A Java facade, a Kotlin coroutine API (Experimental), a Spring Boot starter, a Quarkus guide and sample, and cats-effect, ZIO, Pekko and Reactor modules.

    Java guide

How it differs

Compared with Python-first stacks and other JVM libraries

Facts and trade-offs, not rankings. The full, sourced table is on Comparing LLM4S, last verified 2026-10-11 against LangChain4j, Spring AI, Koog and Embabel.

Versus a Python-first stack

  • Runs inside your JVM services: no second runtime, no service boundary to secure between your code and the LLM logic.
  • Type-checked request and answer shapes instead of dictionaries.
  • The same build, dependency scanning (SBOMs per module) and deployment pipeline as the rest of your code.

Versus other JVM libraries

  • Written for Scala 3, with errors as Result[A] values; the others are Java or Kotlin libraries.
  • A provider is a small published interface, ProviderDescriptor: adding one is adding a dependency.
  • Guardrails and handoffs in the same library as the client, usable on an agent run.
  • AgentIO and AgentZ for cats-effect and ZIO users.

Where it is not the right fit yet. LLM4S is pre-1.0 and Scala 3 only, and needs JDK 21. Its Java facade is incomplete (defining tools, token streaming and structured output from Java are open issues), its Spring Boot starter wires three providers, and LangChain4j and Spring AI each list far more vector stores. If you need those today, the comparison says so plainly.

Use cases

Where it fits

  • Customer and internal assistants

    Assistants inside existing Spring Boot, Pekko or plain JVM services, with guardrails on input and output and traces in your observability stack.

  • Knowledge search and Q&A

    RAG over documents, tickets and wikis with hybrid search, reranking and permission-aware retrieval, evaluated in CI.

  • Document and data extraction

    Turning emails, invoices and reports into typed records inside Spark or streaming pipelines.

  • Long-running agent workflows

    Multi-step agents with approvals and checkpoints that resume after a restart.

  • Developer and platform tooling

    Issue triage, code agents and MCP servers that expose internal systems to AI clients.

  • Multimodal products

    Speech, image generation and vision behind the same typed client.

For companies

Adopting LLM4S in your organisation

What a team evaluating LLM4S for production can check today.

  • Modular artifacts

    Depend on llm4s-core and add only the providers and integrations you use. llm4s-bom pins every artifact to one version. The core holds no vendor SDK.

    1.0 scope

  • A compatibility promise

    Every package has a tier: Frozen at 1.0, Beta or Experimental. Frozen modules will be binary compatible across 1.x, enforced by MiMa from a pre-1.0 baseline, with deprecation before removal.

    Compatibility policy

  • Security work in the open

    A published threat model, API-key redaction, a sandboxed workspace for tool execution, MCP auth hardening, a config-policy CI gate, Dependabot and a CycloneDX SBOM for every module.

    Security reference

  • MIT licence

    Permissive licensing with no field-of-use restrictions, and a DCO sign-off on every contribution.

    Licence

  • Open governance

    Governance, maintainers and release process are documented in the repository, and the roadmap is public.

    Governance

  • Sustain what you depend on

    If LLM4S matters to your business, sponsoring the foundation funds maintenance, security hardening and release engineering. Tell others you use it in ADOPTERS.md.

    Support the foundation

Start a pilot List your organisation Talk to the maintainers

Support the foundation

LLM4S is community-led and maintained by a small core team, mostly in volunteer time. Your support pays for maintainer time, contributor onboarding, documentation, release engineering, CI, security hardening, the weekly Dev Hour and internships. Every contribution is handled in public through Open Collective.

Support on Open Collective How funding is used

  • CompaniesSponsor the project your JVM services depend on, or talk to us about partnering.
  • Grants and institutionsWe welcome grants and institutional support for open infrastructure.
  • UniversitiesPartner with us on internships and mentored student projects.