Providers Guide

LLM4S supports multiple LLM providers out of the box. Choose your provider, configure it, and start building.

Table of contents

  1. Supported Providers
  2. Provider Selection
    1. How It Works
    2. Available Models
  3. OpenAI
    1. Setup
    2. Configuration
    3. Available Models
    4. Reasoning models
    5. Streaming and token usage
    6. Costs
    7. Tips
  4. Anthropic
    1. Setup
    2. Configuration
    3. Available Models
    4. Costs
    5. Tips
    6. Web search
  5. Google Gemini
    1. Setup
    2. Configuration
    3. Available Models
    4. Costs
    5. Tips
    6. Google Search grounding
  6. Google Vertex AI
    1. Setup
    2. Configuration
  7. Provider-specific keys
  8. IBM watsonx.ai
  9. Perplexity
  10. Azure OpenAI
    1. Setup
    2. Configuration
    3. Available Models
    4. Reasoning deployments
    5. Streamed token usage
    6. Costs
    7. Tips
  11. DeepSeek
    1. Setup
    2. Configuration
    3. Available Models
    4. Costs
    5. Tips
  12. OpenRouter and Z.ai
  13. OpenAI-compatible endpoints
    1. Configuration keys
    2. Where the context window comes from
    3. Selecting it, and environment variables
    4. What the generic path does, and does not do
    5. Hosted APIs
      1. Groq
      2. Together AI
      3. Fireworks AI
      4. xAI (Grok)
      5. NVIDIA NIM (hosted)
      6. Perplexity
      7. OrcaRouter
    6. Local servers
      1. vLLM
      2. LM Studio
      3. llama.cpp (llama-server)
      4. NVIDIA NIM (self-hosted)
      5. Ollama (/v1)
    7. An internal gateway
    8. Config policy
    9. Troubleshooting OpenAI-compatible endpoints
    10. Provider docs
  14. Mistral
    1. Setup
    2. Configuration
    3. What it supports
  15. Cohere
    1. Setup
    2. Configuration
    3. What it supports
    4. Available Models
  16. xAI (Grok)
    1. Setup
    2. Configuration
    3. What it supports
    4. Available Models
  17. Ollama (Local Models)
    1. Setup
    2. Configuration
    3. Structured output
    4. Available Models
    5. Costs
    6. Tips
  18. Thinking in conversation history
    1. Gemini and Vertex AI thought signatures
    2. Moving between Anthropic and Bedrock
    3. Limits that remain
  19. Citations and grounding sources
  20. Enterprise authentication
    1. Azure OpenAI with Microsoft Entra ID
    2. AWS Bedrock with IAM roles
    3. Vertex AI with Application Default Credentials
  21. API Key Management
    1. Security Best Practices
    2. Using Keys Safely in Code
  22. Base URL Customization
    1. When to Use Custom Base URLs
    2. Setting Custom URLs
  23. Provider Comparison Table
    1. Which Provider Should I Use?
  24. Multiple Providers in One App
  25. Troubleshooting
    1. “Invalid API Key”
    2. “Connection refused”
    3. “Model not found”
    4. “Rate limit exceeded”

Supported Providers

LLM4S supports these LLM providers, plus any endpoint that speaks the OpenAI chat-completions API. What each one supports with LLM4S - chat, streaming, tool calling, embeddings, API key or workload identity, live smoke suites - is in the compatibility matrix, generated from the provider modules’ code and checked in CI, so this table does not repeat it:

Provider provider id Type Models Setup
OpenAI openai Cloud GPT and o-series models Medium
Anthropic anthropic Cloud Claude models Medium
Google Gemini gemini Cloud Gemini models, with a free tier Medium
Google Vertex AI vertexai Cloud Enterprise Gemini models in your own GCP project Medium
Azure OpenAI azure Cloud Enterprise OpenAI models in your Azure deployment Hard
DeepSeek deepseek Cloud DeepSeek models Easy
OpenRouter openrouter Cloud gateway Many vendors’ models behind one key Easy
Z.ai zai Cloud GLM models Easy
OpenAI-compatible openai-compatible Cloud or local Groq, Together, Fireworks, NVIDIA NIM, OrcaRouter; vLLM, LM Studio, llama.cpp - config only (recipes) Easy
Mistral mistral Cloud Mistral and Magistral models Easy
Cohere cohere Cloud Command models Easy
xAI xai Cloud Grok models Easy
Ollama ollama Local Open models on your machine, no API key Easy
IBM watsonx.ai watsonx Cloud Enterprise Granite, Llama and Mistral models under IBM governance Medium
Perplexity perplexity Cloud Agent API: web-search answers with citations, from Perplexity’s and other vendors’ models (llm4s-perplexity) Easy
Amazon Bedrock bedrock Cloud Enterprise Models on AWS, through the Converse API (llm4s-bedrock) Medium
Requesty requesty Cloud gateway Many vendors’ models behind one key Easy

Missing a vendor? See Writing a Provider to publish your own provider module.

Provider Selection

How It Works

Each provider you use is a named section under llm4s.providers in your own application.conf. llm4s.providers.provider names the default section, which Llm4sConfig.defaultProvider() loads; Llm4sConfig.provider("<name>") loads any section by name. A section’s API key comes from its own apiKey when it sets one, and otherwise from its vendor’s shared key, llm4s.credentials.<provider>.apiKey, which each provider module binds to the vendor’s conventional variable (OPENAI_API_KEY, ANTHROPIC_API_KEY, …). So with the variable exported, a section needs only provider and model. The library reads no LLM_MODEL. See API keys for the full table and for a second account.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# src/main/resources/application.conf
llm4s {
  providers {
    provider = "openai-main"          # the default: the name of a section below

    openai-main {                     # key from OPENAI_API_KEY
      provider = "openai"
      model    = "gpt-4o-mini"
    }

    claude {                          # key from ANTHROPIC_API_KEY
      provider = "anthropic"
      model    = "claude-sonnet-4-20250514"
    }
  }
}

1
2
3
4

import org.llm4s.config.Llm4sConfig

val default = Llm4sConfig.defaultProvider()   // openai-main
val claude  = Llm4sConfig.provider("claude")  // any section, by name

Only the section being loaded is validated, so a section with no key available - or whose provider module is not on the classpath - fails only when it is loaded: with the file above and just OPENAI_API_KEY set, defaultProvider() succeeds and provider("claude") fails with an error naming ANTHROPIC_API_KEY and claude’s apiKey. See Named provider sections and Switching providers for the full story.

Available Models

See MODEL_METADATA.md for the complete model list. Quick reference:

OpenAI: gpt-4o, gpt-4-turbo, gpt-3.5-turbo

Anthropic: claude-opus-4-6, claude-sonnet-4-5-latest, claude-haiku-3-5

Google Gemini: gemini-2.0-flash, gemini-1.5-pro, gemini-1.5-flash

DeepSeek: deepseek-chat, deepseek-reasoner

Mistral: mistral-large-latest, mistral-small-latest, magistral-medium-latest

Cohere: command-a-03-2025

Ollama: mistral, llama2, neural-chat, nomic-embed-text (100+ models)

OpenAI

From the release after 0.4.1, OpenAI support - chat, embeddings, and the Azure OpenAI and Requesty providers that share its client - ships in its own artifact, alongside llm4s-core; adding it registers the providers:

1

libraryDependencies += "org.llm4s" %% "llm4s-openai" % llm4sVersion

In 0.4.1 and earlier it is part of llm4s-core. See the installation guide.

Setup

  1. Get an API key from platform.openai.com/api-keys
  2. Add a named section to application.conf and select it as the default (below).
  3. Export the key - llm4s-openai binds it to llm4s.credentials.openai.apiKey, which the section uses:
1

export OPENAI_API_KEY=sk-proj-...

Configuration

In application.conf:

1
2
3
4
5
6
7
8
9
10
11
12
13

llm4s {
  providers {
    provider = "openai-main"

    openai-main {
      provider     = "openai"
      model        = "gpt-4o"
      # apiKey     = ${?OPENAI_BATCH_API_KEY}  # optional; default: OPENAI_API_KEY
      organization = ${?OPENAI_ORGANIZATION}   # optional, for multi-workspace accounts
      # baseUrl    = "https://my-proxy.example.com/v1"  # optional; default https://api.openai.com/v1
    }
  }
}

Without an apiKey line the section uses OPENAI_API_KEY. Set apiKey for a second account or a variable name of your own; it wins over the shared key. OPENAI_ORGANIZATION is read only because the section binds it.

Available Models

  • Latest: gpt-4o, gpt-4o-mini
  • Reasoning: the gpt-5 family (gpt-5, gpt-5-mini, gpt-5.1, …) and the o-series (o3, o4-mini)
  • Turbo: gpt-4-turbo
  • Legacy: gpt-3.5-turbo

Reasoning models

CompletionOptions.reasoning is sent as reasoning_effort to OpenAI’s reasoning models:

1
2
3
4

import org.llm4s.llmconnect.model.{ CompletionOptions, ReasoningEffort }

// client: LLMClient, conversation: Conversation
client.complete(conversation, CompletionOptions().withReasoning(ReasoningEffort.High))
ReasoningEffort reasoning_effort
Low / Medium / High low / medium / high
None not sent: the model’s own default applies

Which models are reasoning models comes from the model registry’s supports_reasoning flag (the same metadata llm4s uses for context windows and costs), so the o-series and the gpt-5 family get it, and gpt-4o, gpt-4.1 and older models never do: OpenAI rejects the parameter for them, so for those models reasoning is ignored, as CompletionOptions documents. A model newer than the bundled metadata is recognised by its name (o<n>..., gpt-5 onwards, gpt-oss). For Requesty, a routed id such as openai/o4-mini resolves the same way; a non-OpenAI model behind Requesty is not sent reasoning_effort. A fine-tuned model is judged by the model it was trained from: ft:o4-mini-2025-04-16:my-org:my-suffix:abc123 is treated as o4-mini, and a fine-tune of gpt-4o-mini as gpt-4o-mini.

Requests to a reasoning model also follow OpenAI’s rules for them: maxTokens is sent as max_completion_tokens (which counts reasoning tokens too) rather than max_tokens, and temperature, topP, presencePenalty and frequencyPenalty are not sent, since those models reject non-default values. (gpt-oss models keep their sampling parameters.)

The reasoning tokens a response reports (completion_tokens_details.reasoning_tokens) are returned as TokenUsage.thinkingTokens, for streamed and non-streamed calls. They are part of completionTokens, not in addition to it. OpenAI’s chat-completions API does not return the reasoning text itself, so Completion.thinking stays empty.

Streaming and token usage

Streaming requests set stream_options.include_usage, so OpenAI reports token usage on a final chunk with no choices, and a streamed Completion carries usage and estimatedCost like a non-streamed one. The final chunk produces no StreamedChunk.

Costs

See OpenAI Pricing. Generally:

  • gpt-4o: $2.50-$10 per 1M input tokens
  • gpt-3.5-turbo: $0.50-$1.50 per 1M input tokens

Tips

  • Use gpt-4o-mini for cost-effective applications
  • Use o1 for complex reasoning and math
  • Batching API available for high-volume use
  • Vision support in gpt-4o

Anthropic

From the release after 0.4.1, Anthropic support ships in its own artifact, alongside llm4s-core; adding it registers the provider:

1

libraryDependencies += "org.llm4s" %% "llm4s-anthropic" % llm4sVersion

In 0.4.1 and earlier it is part of llm4s-core. See the installation guide.

Setup

  1. Get an API key from console.anthropic.com
  2. Add a named section to application.conf and select it as the default (below).
  3. Export the key - llm4s-anthropic binds it to llm4s.credentials.anthropic.apiKey:
1

export ANTHROPIC_API_KEY=sk-ant-...

Configuration

In application.conf:

1
2
3
4
5
6
7
8
9
10
11
12

llm4s {
  providers {
    provider = "anthropic-main"

    anthropic-main {
      provider = "anthropic"
      model    = "claude-opus-4-6"
      # apiKey = ${?ANTHROPIC_TEAM_API_KEY}   # optional; default: ANTHROPIC_API_KEY
      # baseUrl = "https://my-proxy.example.com"  # optional; default https://api.anthropic.com
    }
  }
}

Available Models

  • Best Quality: claude-opus-4-6 (200K context)
  • Balanced: claude-sonnet-4-5-latest (200K context)
  • Fast: claude-haiku-3-5 (200K context)

Costs

  • claude-opus-4-6: $3-$15 per 1M input tokens
  • claude-sonnet: $3-$15 per 1M input tokens
  • claude-haiku: $0.80-$4 per 1M input tokens

Claude models generally score higher on reasoning benchmarks.

Tips

  • All Claude models have 200K context window
  • Exceptional at writing and analysis tasks
  • Excellent vision capabilities
  • Supports prompt caching for repeated queries

Set webSearch = true in an anthropic section and every request carries Anthropic’s server-side web search tool (web_search_20250305). The model then decides per request whether to search, and the sources its answer cites arrive on Completion.citations (see Citations and grounding sources):

1
2
3
4
5

llm4s.providers.claude-search {
  provider  = "anthropic"
  model     = "claude-sonnet-4-5"
  webSearch = true
}

It is off by default. Anthropic bills each search on top of tokens, and an organization administrator has to allow web search in the Console first. The tool is sent beside your own tools; Anthropic runs the search itself, so it never reaches a ToolRegistry and the completion’s toolCalls do not include it.

Google Gemini

From the release after 0.4.1, Gemini (and Vertex AI) support ships in its own artifact, alongside llm4s-core; adding it registers both providers:

1

libraryDependencies += "org.llm4s" %% "llm4s-gemini" % llm4sVersion

In 0.4.1 and earlier they are part of llm4s-core. See the installation guide.

Setup

  1. Get an API key from aistudio.google.com/apikey
    • Free tier available (60 requests per minute)
  2. Add a named section to application.conf and select it as the default (below).
  3. Export the key - llm4s-gemini binds GOOGLE_API_KEY and GEMINI_API_KEY to llm4s.credentials.gemini.apiKey, preferring GOOGLE_API_KEY when both are set, as Google’s SDKs do. A provider = "google" section uses the same key.
1

export GEMINI_API_KEY=your-api-key

Configuration

In application.conf:

1
2
3
4
5
6
7
8
9
10
11
12

llm4s {
  providers {
    provider = "gemini-main"

    gemini-main {
      provider = "gemini"            # "google" is accepted as an alias
      model    = "gemini-2.0-flash"
      # apiKey = ${?GEMINI_TEAM_API_KEY}   # optional; default: GOOGLE_API_KEY or GEMINI_API_KEY
      # baseUrl = "..."              # optional; default https://generativelanguage.googleapis.com/v1beta
    }
  }
}

Available Models

  • Latest: gemini-2.0-flash (1M context)
  • Advanced: gemini-1.5-pro (1M context)
  • Fast: gemini-1.5-flash (1M context)

Costs

  • Free tier: 60 requests/minute, 2M free tokens/month
  • Paid: Pay as you go (~$0.075-$1.50 per 1M input tokens)

Great for cost-conscious projects and high-volume applications.

Tips

  • Free tier perfect for development and testing
  • 1M context window for processing large documents
  • Very fast inference latency
  • Strong code generation capabilities

Google Search grounding

Set googleSearch = true in a gemini or vertexai section and every request carries Gemini’s Google Search tool (google_search; googleSearch on Vertex AI). The model then decides per request whether to search, and the sources behind a grounded answer arrive on Completion.citations (see Citations and grounding sources):

1
2
3
4
5

llm4s.providers.gemini-search {
  provider     = "gemini"
  model        = "gemini-2.5-flash"
  googleSearch = true
}

It is off by default. Google bills grounded requests separately, and the citation URLs it returns are vertexaisearch.cloud.google.com redirects, not the pages themselves. The tool is sent beside any function declarations; a model that does not support search together with function calling rejects that request.

Google Vertex AI

Vertex AI serves the same Gemini models from a GCP project, authenticated with OAuth2 rather than an API key. It ships in llm4s-gemini beside the Gemini API provider, so the dependency above registers both.

Setup

  1. Pick a GCP project with the Vertex AI API enabled, and a region that serves your model.
  2. Authenticate with Application Default Credentials (gcloud auth application-default login), or point apiKey at a service-account JSON file. Without apiKey, credentials come from GOOGLE_APPLICATION_CREDENTIALS, then gcloud’s application_default_credentials.json, then the GCE metadata server. The file is located as Google’s client libraries do: in $CLOUDSDK_CONFIG when that is set (any OS), else %APPDATA%\gcloud on Windows, else ~/.config/gcloud.

Configuration 

1
2
3
4
5
6
7
8
9
10
11
12
13

llm4s {
  providers {
    provider = "vertex-main"

    vertex-main {
      provider = "vertexai"          # "vertex" is accepted as an alias
      model    = "gemini-2.0-flash"
      project  = ${?VERTEXAI_PROJECT}
      location = "europe-west4"      # optional
      apiKey   = ${?GOOGLE_APPLICATION_CREDENTIALS}  # optional: path to a credential file
    }
  }
}
Key Required Meaning
project yes The GCP project ID that owns your Vertex AI resources
location no The GCP region; requests go to https://<location>-aiplatform.googleapis.com/v1. Default us-central1
apiKey no A path to a service-account credential file - not an API key

Because Vertex AI has no API key, it has no shared llm4s.credentials key: nothing is bound for it, and GOOGLE_API_KEY / GEMINI_API_KEY apply to the Gemini API provider only.

project and location are Vertex AI’s own keys (provider-specific keys). Before #1215 Vertex read the project from endpoint and the region from organization. Those still work as deprecated aliases, with a warning each time the config loads; rename them. Setting both a key and a different value under its old name is an error. See the migration note.

Provider-specific keys

Every named section shares the built-in fields - provider, model, baseUrl, apiKey and headers - and a provider may declare keys of its own. Its section of this guide lists them:

Provider Its own keys
OpenAI, Requesty, OpenRouter organization
Azure OpenAI endpoint (required), apiVersion, authentication, clientId
Vertex AI project (required), location
Generic openai-compatible contextWindow, reserveCompletion, registryProvider, streamUsage
Perplexity webSearch, contextWindow, reserveCompletion

A required one that is missing fails the config with its name, the section and what it means; a key that is neither built-in nor declared by the section’s provider is ignored with a warning naming it, so a misspelt key - or organization in an Anthropic section - shows up in the log rather than silently doing nothing. organization, endpoint, apiVersion, contextWindow and reserveCompletion were built-in fields that every section carried until #1133; the HOCON for the providers that use them is unchanged.

IBM watsonx.ai

Beta, never run against the live service. llm4s-watsonx calls the watsonx.ai chat API (/ml/v1/text/chat and /ml/v1/text/chat_stream), which replaces the “Infer text” endpoints IBM deprecated in February 2026. There is no watsonx account to test with, so nothing here has been checked against the real service. The request and response shapes are the ones IBM’s public pages report, cross-checked against IBM’s own open-source clients (the Node.js SDK and IBM’s LangChain integration; the pinned sources are listed in WatsonXClient’s Scaladoc): the token-limit field, the encoding of tool-call arguments, the finish reasons and the stream’s usage chunk are all shown there. IBM’s tool-calling guide shows the same tool request (tool_choice_option: "auto") and tool-call reply, and a test pins its example. WatsonXChatOverHttpSpec runs the client through the real HTTP stack against a local server: tool calls whole and streamed, structured messages with no role-marker prompt, and the error mapping and redaction for a wrong key. That proves IBM’s clients send these shapes, not that the service accepts them today. The module is Beta and its API is not frozen. The @Cloud suite WatsonXAssumptionProbeSpec (modules/it) checks each of those details against a real account and prints which held; running it is still open in #1314. Until it has been run, treat tool calling and streaming as unconfirmed.

IBM’s enterprise AI platform, serving Granite, Llama and Mistral models. It lives in its own module, llm4s-watsonx; adding the dependency registers the watsonx provider.

1

libraryDependencies += "org.llm4s" %% "llm4s-watsonx" % llm4sVersion

1
2
3
4
5
6
7

llm4s.providers.watsonx-main {
  provider  = "watsonx"
  model     = "ibm/granite-4-h-small"          # any chat model your watsonx.ai region serves
  projectId = ${?WATSONX_PROJECT_ID}          # or spaceId for a deployment space
  # baseUrl = "https://eu-de.ml.cloud.ibm.com" # default https://us-south.ml.cloud.ibm.com
  # apiVersion = "2026-09-25"                  # the `version` query parameter; default as shown
}

WATSONX_API_KEY supplies the IBM Cloud API key, which the client exchanges for an IAM bearer token and refreshes before it expires. The conversation goes as structured messages with roles (system, user, assistant, tool), so user content is data and cannot forge a turn.

Behaviour to know about:

  • Tools. CompletionOptions.tools are sent as tools with tool_choice_option: "auto", a reply’s tool_calls (whole, or streamed as delta pieces) become ToolCalls, and a tool result goes back as a tool message with its tool_call_id. Streamed calls are reported once, whole, after the last delta. Whether a model supports tools depends on the model; a model that does not is answered by the service, and that error is returned as the HTTP status maps it. A call the service sends without an id is given a generated one.
  • Sampling options. temperature is always sent, maxTokens (as max_tokens) and topP (when not 1.0) when set. presencePenalty and frequencyPenalty go as presence_penalty and frequency_penalty when not 0.0, their default, so leaving them alone sends nothing new; the service checks their range.
  • Structured output. responseFormat goes as response_format: ResponseFormat.Json as {"type": "json_object"}, and ResponseFormat.JsonSchema(schema, name, strict) as {"type": "json_schema", "json_schema": {"name": ..., "schema": ..., "strict": ...}}, the shapes IBM’s Python SDK and LangChain integration send (IBM’s Node.js SDK names only json_object). JSON mode needs the prompt to ask for JSON too; the client adds no instruction. Whether a model honours either format depends on the model. Like the rest of this section, both are unconfirmed until the probe suite runs.
  • Reasoning. CompletionOptions.reasoning goes as reasoning_effort (Low, Medium and High as low, medium and high) to OpenAI’s gpt-oss models (openai/gpt-oss-120b, openai/gpt-oss-20b and their variants), the models IBM’s own clients send it to. The chat API has no value that turns reasoning off, so ReasoningEffort.None goes as low, and a warning is logged once per process. Any other model is sent nothing for reasoning, again with a warning logged once per process: IBM’s Model Gateway rejects the field for models without it, and whether the chat API does is unconfirmed. include_reasoning is not sent; the service defaults it to true.
  • Thinking. A reply’s reasoning_content (or reasoning) is read as thinking from any model: Completion.thinking on complete, and StreamedChunk.thinkingDelta on a stream, with the whole text on the final completion. It is not sent back on later turns. The chat API reports no reasoning token count, so TokenUsage.thinkingTokens stays empty (IBM’s recorded replies suggest the reasoning is counted in completion_tokens). budgetTokens has no counterpart and is ignored. Granite models that put their reasoning inside the content between tags (<think>...</think>) are not split: that text stays in content. The field names come from IBM’s SDKs and from replies IBM’s Java SDK recorded from openai/gpt-oss-120b; like the rest of this section, they are unconfirmed until the probe suite runs.
  • Abnormal endings. A stream that ends without a finish_reason, or whose finish_reason is error, cancelled or time_limit, returns Left(ServiceError) naming the reason, not the partial text (chunks already passed to onChunk were delivered). stop, length, tool_calls and unknown reasons are normal stops. complete applies the same rule to choices[0].finish_reason.
  • Stream error events. An event: error frame ends the stream with the error IBM sends in its data: line: its status_code picks the error as an HTTP status would (AuthenticationError for 401 and 403, RateLimitError for 429, ValidationError for 400, ServiceError otherwise, and ServiceError(502) without one), and the message is IBM’s error code and message, with the API key and bearer token removed.
  • API version. Every request carries version=<apiVersion>. The default, 2026-09-25, is the date IBM’s Python SDK (ibm-watsonx-ai 1.8.0) sends; set apiVersion in the section to pin another.
  • URLs and ids. baseUrl and iamUrl must be https (plain http only for localhost, 127.0.0.1 and ::1), because the IAM request carries the API key. The API key is trimmed. Set projectId or spaceId, not both.
  • Environment variables. Only WATSONX_API_KEY is bound automatically. WATSONX_PROJECT_ID and the other variables in the example are just ${?VAR} substitutions you write in your own config.

Perplexity

llm4s-perplexity calls Perplexity’s Agent API (POST https://api.perplexity.ai/v1/agent), which replaced Sonar Chat Completions: Perplexity ended Sonar’s support on 27 September

  1. The Agent API speaks the OpenAI Responses format, so it has its own client rather than the generic openai-compatible one. Adding the dependency registers the perplexity provider.
1

libraryDependencies += "org.llm4s" %% "llm4s-perplexity" % llm4sVersion

1
2
3
4
5
6

llm4s.providers.perplexity-main {
  provider = "perplexity"
  model    = "perplexity/sonar"     # any Agent API model: perplexity/sonar, openai/gpt-5.6-sol, anthropic/claude-sonnet-5-5, ...
  # apiKey defaults to PERPLEXITY_API_KEY (llm4s.credentials.perplexity.apiKey)
  # webSearch = false               # default true
}
Key Required Meaning
model yes An Agent API model id, provider/model (the list)
apiKey no Default PERPLEXITY_API_KEY
webSearch no Offer the model Perplexity’s web_search tool on every request; default true. Each search is billed
contextWindow no The model’s window in tokens; default the model registry’s entry for the model’s vendor, else 128000
reserveCompletion no Tokens held back for the reply; default 4096
  • Citations. Completion.citations holds the url_citation annotations on the answer, then each web search result (its URL, title and snippet as citedText). The answer’s [web:N] markers point at those results. Both complete and a stream carry them.
  • Usage and cost. Cached and reasoning tokens are read from the usage details, and the cost Perplexity reports for the request (usage.cost.total_cost, USD, searches included) is Completion.estimatedCost.
  • Tools. Your tools are sent as function tools beside web_search, and a reply’s function_call items become ToolCalls. A call’s thought_signature, which some models attach, is kept and sent back with it.
  • Request mapping. System messages become instructions. ReasoningEffort is sent as reasoning.effort (None as minimal). A ResponseFormat.JsonSchema is sent as response_format; plain ResponseFormat.Json is refused, since the Agent API takes only a schema. An anthropic/ model with no maxTokens is sent reserveCompletion as max_output_tokens, which the Agent API requires for those models. Penalties have no counterpart and are not sent. Only text content is sent.
  • Not supported yet: effort levels (effort), profiles, model fallback chains, the other built-in tools (fetch_url, finance_search, …), background runs and previous_response_id.
  • No live call was made while writing this module. The @Cloud suite PerplexitySmokeSpec runs it against the service with PERPLEXITY_API_KEY.

Azure OpenAI

Azure OpenAI ships in llm4s-openai with OpenAI itself, from the release after 0.4.1; see OpenAI above. It runs on OpenAI’s Java SDK (com.openai:openai-java), which covers Azure too: requests go to <endpoint>/openai/deployments/<deployment>/chat/completions with an api-key header and the api-version query parameter. apiVersion accepts the wire form (2024-10-21) or the constant name (V2024_10_21); an endpoint ending in /openai/v1 uses Azure’s unified v1 API.

Setup

  1. Create resource in Azure Portal
  2. Deploy model (e.g., gpt-4o) to get deployment name
  3. Get credentials from Azure Portal → Keys & Endpoint
  4. Add a named section to application.conf (below) and export the key, which llm4s-openai binds to llm4s.credentials.azure.apiKey under the openai SDK’s name, and the endpoint the section binds:
1
2

export AZURE_OPENAI_API_KEY=your-azure-key
export AZURE_API_BASE=https://your-resource.openai.azure.com

Configuration

In application.conf:

1
2
3
4
5
6
7
8
9
10
11
12
13

llm4s {
  providers {
    provider = "azure-main"

    azure-main {
      provider   = "azure"
      model      = "gpt-4o"                  # your deployment name
      endpoint   = ${?AZURE_API_BASE}        # required: your resource's endpoint
      # apiKey   = ${?AZURE_TEAM_API_KEY}    # optional; default: AZURE_OPENAI_API_KEY
      apiVersion = "2024-10-21"              # optional; default V2025_01_01_PREVIEW
    }
  }
}
Key Required Meaning
model yes The deployment name you chose when deploying the model
endpoint yes The Azure OpenAI resource endpoint
apiKey no The resource key; default AZURE_OPENAI_API_KEY
apiVersion no Azure API version, wire form or constant name
authentication no api-key (default) or entra-id: keyless Microsoft Entra ID
clientId no With entra-id, the client id of a user-assigned managed identity

Available Models

Same as OpenAI (via Azure deployment). Choose models when deploying:

  • gpt-4o
  • gpt-4-turbo
  • gpt-35-turbo

Reasoning deployments

reasoning_effort works as for OpenAI, with one difference: the model llm4s sees is the deployment name, which need not name the model behind it. So:

  • a deployment named after a model the registry knows (o4-mini, gpt-5-mini, gpt-4o) is treated as that model, and one named after an Azure fine-tuned model (o4-mini-2025-04-16.ft-...) as its base model;
  • any other deployment (prod-reasoner, or a fine-tune deployed under a name of your own) gets reasoning_effort whenever you ask for a reasoning effort, since asking is the only sign it is a reasoning model. The request then also follows the reasoning-model rules: max_completion_tokens, and no sampling parameters. Asking for reasoning on such a deployment that is not a reasoning model gets an error from Azure; leave reasoning unset (or ReasoningEffort.None) for it.

Streamed token usage

Streamed completions report token usage when apiVersion is 2024-09-01-preview or later (including the default 2025-01-01-preview, the GA 2024-10-21 and the unified v1 API): only those versions accept stream_options, so it is not sent for older ones, such as 2024-02-15-preview, and their streams report no usage.

Costs

Similar to OpenAI but often bundled with enterprise agreements.

Tips

  • Use for VPC-isolated workloads
  • Enterprise support available
  • Same API as OpenAI (easy migration)
  • Reserve capacity for predictable costs

DeepSeek

From the release after 0.4.1, DeepSeek ships in llm4s-openai-compatible, with Z.ai, OpenRouter and the generic OpenAI-compatible provider; adding it registers them:

1

libraryDependencies += "org.llm4s" %% "llm4s-openai-compatible" % llm4sVersion

In 0.4.1 and earlier it is part of llm4s-core. See the installation guide.

Setup

  1. Get API key from platform.deepseek.com
  2. Export the key - llm4s-openai-compatible binds it to llm4s.credentials.deepseek.apiKey:
1

export DEEPSEEK_API_KEY=sk-...

Configuration

In application.conf:

1
2
3
4
5
6
7
8
9
10
11
12

llm4s {
  providers {
    provider = "deepseek-main"

    deepseek-main {
      provider = "deepseek"
      model = "deepseek-chat"
      # apiKey defaults to DEEPSEEK_API_KEY
      # baseUrl defaults to https://api.deepseek.com
    }
  }
}

Available Models

  • Chat: deepseek-chat (best for general use)
  • Reasoning: deepseek-reasoner (extended thinking). Its chain of thought (reasoning_content) is returned on the message as AssistantMessage.thinking (and so Completion.thinking), streamed as thinking deltas, and sent back as reasoning_content in later requests, which DeepSeek’s thinking mode requires once tools are involved (see Thinking in conversation history).

Costs

Very competitive: ~$0.14-$0.28 per 1M input tokens

Tips

  • Excellent cost/performance ratio
  • Reasoning model rivals GPT-4o
  • Good for translations and multilingual tasks
  • Supports very long contexts

OpenRouter and Z.ai

Both ship in llm4s-openai-compatible from the release after 0.4.1, as DeepSeek does.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

llm4s {
  providers {
    openrouter-main {
      provider = "openrouter"
      model = "anthropic/claude-3.5-sonnet"
      # apiKey defaults to OPENROUTER_API_KEY
      # baseUrl defaults to https://openrouter.ai/api/v1
    }

    zai-main {
      provider = "zai"
      model = "GLM-4.7"
      # apiKey defaults to ZAI_API_KEY
      # baseUrl defaults to https://api.z.ai/api/paas/v4
    }
  }
}

Both sections are shown together for brevity. Only the section you load is validated, so only its key - OPENROUTER_API_KEY or ZAI_API_KEY - needs to be set.

OpenRouter maps CompletionOptions.reasoning onto the underlying model: a thinking budget for Claude models, reasoning_effort for OpenAI o-series models, nothing for the rest.

Z.ai maps it onto what the configured GLM model documents. Thinking is on by default, and reasoning_effort defaults to max on the models that accept it. The mapping only ever rises with the effort, and High sends max, Z.ai’s maximum and its default, so no level reasons more than High:

Model ReasoningEffort.None Low Medium High
GLM-5.3, GLM-5.3-Flash, GLM-5.3-FlashX reasoning_effort: low low high max
GLM-5.2 reasoning_effort: none low medium max
GLM-5.1, GLM-5, GLM-4.7, GLM-4.6, GLM-4.5 (and variants) thinking.type: disabled not sent not sent not sent
any other model not sent not sent not sent not sent

GLM-5.3 always thinks and rejects thinking.type: disabled, so None gets the lowest effort it accepts, as Z.ai advises; it still produces, and bills, thinking tokens, and the first such request in a process logs a warning naming the model. GLM-5.3 accepts only low, high and max. GLM-5.2 accepts none, low, medium, high and max, but Z.ai currently runs low and medium as high on it; llm4s still sends the level requested. The GLM-4.x and earlier GLM-5 models document no reasoning_effort. With reasoning unset, neither field is sent. A GLM thinking model’s reasoning_content on Z.ai, and a model’s reasoning and reasoning_details on OpenRouter, are returned on the message and sent back (see Thinking in conversation history).

OpenAI-compatible endpoints

Any server that speaks the OpenAI /chat/completions API works with provider = "openai-compatible" - hosted APIs such as Groq, Together, Fireworks, xAI or NVIDIA NIM, local servers such as vLLM, LM Studio or llama.cpp, or an internal gateway. It needs no code and no module of its own, only llm4s-openai-compatible, which ships in the release after 0.4.1 (see the installation guide):

1

libraryDependencies += "org.llm4s" %% "llm4s-openai-compatible" % llm4sVersion

Each named section is one endpoint, so several can sit side by side. The recipes below are sections you can paste into your application.conf under llm4s.providers.

Configuration keys

Key Required Meaning
baseUrl yes Requests go to <baseUrl>/chat/completions, and model listing to <baseUrl>/models. A trailing / is dropped
model yes Sent as-is in every request
apiKey no Sent as Authorization: Bearer <key>; with none, no Authorization header is sent
contextWindow no The model’s context window; overrides the model registry. With none, the registry’s window for model when it has one of at least 8192 (see where the context window comes from), else 8192, which is deliberately small
reserveCompletion no Tokens held back for the reply. Default 2048, or a quarter of a smaller window
registryProvider no The model registry provider whose entry for model gives the context window when contextWindow is not set: groq, together_ai, fireworks_ai, xai, perplexity, … Inferred from the baseUrl host for those five hosted APIs; naming one here switches the inference off
headers no Extra headers sent on every request; values are redacted when the config is printed
streamUsage no Whether a streaming request asks for token usage with stream_options.include_usage. Default true; set false for an endpoint that rejects the field (see streamed token usage)

baseUrl is everything before /chat/completions in the vendor’s endpoint URL. For most vendors that is the host plus /v1, but some add a prefix - Groq’s is /openai/v1, Fireworks’ /inference/v1 - and Perplexity’s Sonar API has no /v1 at all.

Where the context window comes from

The generic provider cannot ask an arbitrary endpoint how large the model’s window is, so it takes it from the first of these that has an answer:

  1. the section’s contextWindow;
  2. the model registry’s entry for model under the registryProvider the section names;
  3. when the section names none, the registry’s entry under the provider inferred from the baseUrl host, for exactly these hosts: api.groq.com (groq), api.together.xyz and api.together.ai (together_ai), api.fireworks.ai (fireworks_ai), api.x.ai (xai) and api.perplexity.ai (perplexity);
  4. 8192.
1
2
3
4
5
6

internal-gateway {
  provider = "openai-compatible"
  baseUrl = "https://llm-gateway.internal.example/v1"   # not a known host, so name the registry provider
  model = "llama-3.1-8b-instant"
  registryProvider = "groq"                             # the window comes from groq/llama-3.1-8b-instant
}
  • An explicit registryProvider is an instruction, not a hint. It switches the host inference off, so a registryProvider = "perplexity" on a Groq URL looks in Perplexity’s entries only.
  • The lookup is strict. Only the named provider’s own entry for exactly that model id counts. A partial name does not match, and neither does another provider’s entry for the same name, so a model the registry does not list under that provider gets the default, never a neighbour’s window.
  • The host is matched exactly, never by substring. https://api.groq.com.evil.example/v1, https://api.groq.com@evil.example/v1 and https://evil.example/api.groq.com all get no registry window. NVIDIA NIM is not in the table because the registry’s three nvidia_nim entries carry no chat model with a window; registryProvider = "nvidia_nim" is still accepted.
  • The registry can only enlarge the window. A registry window below 8192 is ignored and the default used: many entries, most of Fireworks’ and Perplexity’s older ones among them, carry a 4096 placeholder for input, output and total alike (fireworks_ai/.../minimax-m1-80k is one), which would cut an 80k-context model’s prompt budget in half and reject a reserveCompletion that fits 8192. A model whose window really is smaller sets contextWindow. The skipped entry is logged like a miss.
  • Only the window is taken. reserveCompletion keeps its own rule, a quarter of the window up to 2048, because a registry entry’s output limit can be as large as the whole window.
  • The registry does not know every model. It has no input limit for the Together recipe’s model, and does not list xAI’s grok-4.7 or the NVIDIA NIM model at all, so those recipes keep their contextWindow. A miss under an explicit registryProvider is logged as a warning, and under an inferred one at info; a hit is logged at info with the entry it used.

Selecting it, and environment variables

Llm4sConfig resolves providers from named sections only, and reads no LLM_MODEL. The generic provider has no vendor, so - unlike openai or anthropic - it has no shared llm4s.credentials key and no variable is bound for it. Every value lives in the named section, and you bind environment variables to it with HOCON substitutions, under names you choose. The recipes use each vendor’s own documented variable name, such as GROQ_API_KEY:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

llm4s {
  providers {
    # the default named provider; LLM4S_PROVIDER overrides it when set
    provider = "groq-main"
    provider = ${?LLM4S_PROVIDER}

    groq-main {
      provider = "openai-compatible"
      baseUrl = "https://api.groq.com/openai/v1"
      model = "openai/gpt-oss-120b"
      model = ${?GROQ_MODEL}          # optional override
      apiKey = ${?GROQ_API_KEY}
    }
  }
}

${?VAR} leaves the key unset when the variable is missing. For apiKey that is not an error when the config loads, because the key is optional: the request goes out without an Authorization header and fails with a 401 (see Troubleshooting).

For an endpoint chosen entirely from the environment, the conventional names are OPENAI_COMPATIBLE_BASE_URL (required: the generic provider has no default endpoint) and OPENAI_COMPATIBLE_API_KEY (optional), which OpenAICompatibleConfigKeys names:

1
2
3
4
5
6
7
8
9

llm4s.providers {
  provider = "compatible-env"
  compatible-env {
    provider = "openai-compatible"
    baseUrl = ${?OPENAI_COMPATIBLE_BASE_URL}
    model = ${?OPENAI_COMPATIBLE_MODEL}
    apiKey = ${?OPENAI_COMPATIBLE_API_KEY}
  }
}

With OPENAI_COMPATIBLE_BASE_URL or the model variable unset, that section fails to load, so keep it in the application.conf of deployments that set them.

The LLM_MODEL=openai-compatible/<model> shorthand is read, as LLM_MODEL=deepseek/<model> and the other providers’ are, by the tools that read LLM_MODEL themselves: the chat-tui sample, which then takes the endpoint from OPENAI_COMPATIBLE_BASE_URL and the key from OPENAI_COMPATIBLE_API_KEY, and the config-policy env check, which checks OPENAI_COMPATIBLE_BASE_URL rather than OPENAI_BASE_URL as the endpoint. Only the first / separates the provider, so LLM_MODEL=openai-compatible/openai/gpt-oss-120b names Groq’s openai/gpt-oss-120b.

Then load it by name, or as the default:

1
2
3
4
5
6

import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.config.ProviderConfig
import org.llm4s.types.Result

val groq: Result[ProviderConfig]  = Llm4sConfig.provider("groq-main")
val dflt: Result[ProviderConfig]  = Llm4sConfig.defaultProvider()

What the generic path does, and does not do

It sends the plain chat-completions format and reads the plain reply: text, streaming, tool calling (streamed and not), response_format for JSON output, and token usage. Every request carries temperature and top_p; max_tokens is sent when set, and presence_penalty and frequency_penalty only when non-zero. It never sends stop or tool_choice. Whether a feature works also depends on the endpoint and the model: tool calling on a local server usually needs a server flag, as the recipes note.

Streamed token usage. A streaming request asks for usage with "stream_options": {"include_usage": true}, which is how OpenAI - and servers that follow it, such as vLLM, Ollama’s /v1 and Perplexity’s Orchestrator - are told to report it; usage is read from whichever event carries it, including a final event with empty choices. An endpoint that rejects unknown fields may refuse the request with a 400 or 422 naming stream_options. Turn it off with streamUsage = false in the section (it takes true or false, and HOCON’s yes/no/on/off; anything else is a config error naming the key and section):

1
2
3
4
5
6

strict-gateway {
  provider = "openai-compatible"
  baseUrl = "https://llm.example/v1"
  model = "my-model"
  streamUsage = false
}

or, building the config in code, with OpenAICompatibleConfig.fromValues(..., streamUsage = false).

A non-streaming request fails with a timeout after two minutes without a response, and a streaming one after five minutes without one. Both are fixed for now; configurable timeouts are #712. A slow local model answering a long prompt can take longer than two minutes: stream it instead.

What it does not do:

  • No reasoning configuration. CompletionOptions.reasoning is ignored; no reasoning_effort or similar field is sent, so a reasoning model runs at its vendor’s default.
  • No reasoning output. A reply’s reasoning_content or reasoning field is not read, so Completion.thinking is empty and reasoning tokens are not reported separately.
  • No provider-specific fields. Anything outside the standard reply - citations, search results, annotations - is dropped; Completion has no field for it.
  • Text only. Messages are sent as text.
  • No cost estimate unless the model name is one llm4s’s model registry knows.

A provider that needs any of these gets its own dialect in llm4s-openai-compatible: an OpenAICompatibleDialect for how it departs from the standard format (reasoning parameters, where its thinking is, extra response decoding) and a ProviderDescriptor to register it, as DeepSeek, Z.ai, OpenRouter, Mistral and Cohere have. Adding an OpenAI-compatible provider in the contributing guide walks through the hooks, a worked example, registration, the tests to write and the docs to update; see also Adding a provider for the descriptor.

Hosted APIs

The values below were checked against each vendor’s API documentation in September 2026 (links under Provider docs). Model catalogues change often: if a model is retired, pick a current one from the vendor’s list and keep the rest of the section.

Groq 

1
2
3
4
5
6
7

groq-main {
  provider = "openai-compatible"
  baseUrl = "https://api.groq.com/openai/v1"
  model = "openai/gpt-oss-120b"
  apiKey = ${?GROQ_API_KEY}
  reserveCompletion = 8192
}
  • The context window, 131072 for this model, comes from the model registry, because api.groq.com is a known host. Set contextWindow to override it, or for a model the registry does not list.
  • Model ids are Groq’s own, including the openai/ prefix here. llama-3.3-70b-versatile, which earlier versions of this guide used, was shut down for free and developer tiers on 16 August 2026.
  • gpt-oss-120b is a reasoning model. Groq returns its reasoning in a reasoning field, which is not read, and the effort cannot be set.
  • Groq does not support logprobs, logit_bias, messages[].name or n > 1; the generic path sends none of them. A temperature of 0 is converted to 1e-8.

Together AI 

1
2
3
4
5
6
7

together-main {
  provider = "openai-compatible"
  baseUrl = "https://api.together.ai/v1"
  model = "meta-llama/Llama-3.3-70B-Instruct-Turbo"
  apiKey = ${?TOGETHER_API_KEY}
  contextWindow = 131072
}
  • Keep contextWindow: the registry lists this model with no input limit, so without it the provider would use 8192. (For a Together model the registry does give a limit for, it is taken automatically.)
  • Model ids are the <organisation>/<model> API strings from Together’s serverless model list.
  • For a reasoning model on Together, the same reasoning limits apply as for Groq above.

Fireworks AI 

1
2
3
4
5
6

fireworks-main {
  provider = "openai-compatible"
  baseUrl = "https://api.fireworks.ai/inference/v1"
  model = "accounts/fireworks/models/gpt-oss-120b"
  apiKey = ${?FIREWORKS_API_KEY}
}
  • The context window, 131072 for this model, comes from the model registry. Set contextWindow to override it, or for a model the registry does not list.
  • Model ids are full paths, accounts/fireworks/models/<name>. Not every model in the catalogue is available serverless: check its model page before using it with a plain API key.
  • If the prompt plus max_tokens exceeds the model’s window, Fireworks lowers max_tokens rather than failing (its default context_length_exceeded_behavior is truncate).
  • Fireworks sends usage on the last chunk of a stream, so streamed token usage is reported. reasoning_content is not read.

xAI (Grok)

xAI has a provider of its own, provider = "xai", which reads Grok’s reasoning and sends CompletionOptions.reasoning; prefer it. The generic path still works, for chat, streaming and tools only:

1
2
3
4
5
6
7

xai-generic {
  provider = "openai-compatible"
  baseUrl = "https://api.x.ai/v1"
  model = "grok-4.7"
  apiKey = ${?XAI_API_KEY}
  contextWindow = 500000
}
  • Keep contextWindow: the model registry does not list grok-4.7, so without it the provider would use
    1. (For an xAI model the registry does list, such as grok-2, it is taken automatically.)
  • On this path Grok’s reasoning output is not read and the effort cannot be set, and xAI rejects presencePenalty and frequencyPenalty on reasoning models: leave them at 0.

NVIDIA NIM (hosted) 

1
2
3
4
5
6
7

nim-cloud {
  provider = "openai-compatible"
  baseUrl = "https://integrate.api.nvidia.com/v1"
  model = "meta/llama-3.3-70b-instruct"
  apiKey = ${?NVIDIA_API_KEY}
  contextWindow = 128000
}
  • Keep contextWindow: the model registry has no usable chat entry for NVIDIA NIM, and integrate.api.nvidia.com is not one of the hosts the registry provider is inferred from.
  • Model ids are <publisher>/<model>, as in NVIDIA’s API catalogue. For a NIM you run yourself, see NVIDIA NIM (self-hosted).

Perplexity

For Perplexity’s search-grounded answers use the perplexity provider: its Agent API speaks the OpenAI Responses format, which this provider does not. Sonar Chat Completions, which this provider could reach, lost Perplexity’s support on 27 September 2026.

Perplexity’s Orchestrator API (formerly the Router API) speaks Chat Completions and serves open-weight models, without the Agent API’s web search. It is in private preview (access by request to Perplexity), and publishes no context windows, so set contextWindow for the model you pick:

1
2
3
4
5
6

perplexity-orchestrator {
  provider = "openai-compatible"
  baseUrl = "https://api.perplexity.ai/orchestrator/v1"   # the old /router/v1 still works
  model = "perplexity/kimi-k3"
  apiKey = ${?PERPLEXITY_API_KEY}
}

The Orchestrator reports token usage on a stream only when asked with stream_options, which the generic provider sends, so streamed completions carry usage.

OrcaRouter

OrcaRouter is an OpenAI-compatible gateway in front of several vendors’ models:

1
2
3
4
5
6

orcarouter-main {
  provider = "openai-compatible"
  baseUrl = "https://api.orcarouter.ai/v1"
  model = "openai/gpt-4o-mini"
  apiKey = ${?ORCAROUTER_API_KEY}   # keys start with sk-orca-
}
  • OrcaRouter documents no environment variable name; ORCAROUTER_API_KEY is only this guide’s choice. It publishes no per-model context window either: set contextWindow to the upstream model’s.
  • Model ids are provider-prefixed (openai/, anthropic/, google/, …). OrcaRouter accepts a reasoning effort as a model-name suffix (-minimal, -low, -medium, -high, -max), and since model is sent as-is, that is the one way to choose an effort through the generic path. The reasoning_content it passes through is not read.

Local servers

A local server usually takes no API key: leave apiKey out and no Authorization header is sent. Set it only if you started the server with one. Set contextWindow to the context the server was started with, which may be smaller than the model’s maximum.

vLLM 

1

vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto

1
2
3
4
5
6

local-vllm {
  provider = "openai-compatible"
  baseUrl = "http://localhost:8000/v1"
  model = "NousResearch/Meta-Llama-3-8B-Instruct"   # --served-model-name, if you set one
  # apiKey = ${?VLLM_API_KEY}                        # only if started with --api-key
}
  • The served model name defaults to the --model argument.
  • Tool calling needs --enable-auto-tool-choice and a --tool-call-parser for the model’s family; without them the model never calls a tool.

LM Studio

Start the server from LM Studio’s Developer page (default port 1234), then:

1
2
3
4
5
6

lmstudio {
  provider = "openai-compatible"
  baseUrl = "http://localhost:1234/v1"
  model = "<the model identifier LM Studio shows>"
  # apiKey = ${?LM_API_TOKEN}   # only with "Require Authentication" on (LM Studio 0.4.0+)
}
  • Use the identifier LM Studio shows for the loaded model, or list them with Llm4sConfig.listModels("lmstudio").

llama.cpp (llama-server) 

1

llama-server -hf <user>/<model>:<tag> --alias local-model -c 16384 --port 8080

1
2
3
4
5
6
7

llama-cpp {
  provider = "openai-compatible"
  baseUrl = "http://localhost:8080/v1"
  model = "local-model"          # the --alias
  contextWindow = 16384          # the -c / --ctx-size the server was started with
  # apiKey = ${?LLAMA_API_KEY}   # only if started with --api-key
}
  • llama-server listens on 127.0.0.1, on port 9931 by default in current builds; the command above pins 8080 with --port. Without --alias the model id is the model file’s path; --alias gives it a stable name.
  • Tool calling uses the Jinja chat template (--jinja, on by default), and some models need a --chat-template-file override to get a tool-capable template.

NVIDIA NIM (self-hosted) 

1
2
3
4
5
6

nim-local {
  provider = "openai-compatible"
  baseUrl = "http://localhost:8000/v1"
  model = "meta/llama-3.1-8b-instruct"   # the id GET /v1/models reports
  # no apiKey: NVIDIA's examples for a running NIM send none
}
  • NGC_API_KEY is for pulling the container and model, not for requests: do not put it in apiKey.
  • The model id is the id from GET /v1/models; for a model-free NIM it is NIM_SERVED_MODEL_NAME.

Ollama (/v1)

llm4s-ollama (provider = "ollama") is the first-class route to Ollama: it uses Ollama’s native /api/chat API, supports tool calling (tools are sent, tool calls are read - whole or streamed - and tool results go back as role: tool) and also provides Ollama embeddings. Tool calling needs a model whose Ollama page lists the tools capability: for any other model Ollama answers HTTP 400 (... does not support tools), which the client reports as an Invalid tools: Ollama model '<model>' does not support tool calling ... validation error rather than sending the request again without the tools. Use the generic provider on Ollama’s OpenAI-compatible /v1 endpoint instead when Ollama is behind a gateway that exposes only the OpenAI API, or when you prefer that wire format:

1
2
3
4
5
6

ollama-openai {
  provider = "openai-compatible"
  baseUrl = "http://localhost:11434/v1"
  model = "gpt-oss:20b"
  # no apiKey: local Ollama needs none
}
  • The OpenAI API cannot set Ollama’s context size. To change it, create a model from a Modelfile with PARAMETER num_ctx <size>, use that model’s name, and set contextWindow to the same size.
  • Ollama’s cloud API works the same way, with baseUrl = "https://ollama.com/v1" and apiKey = ${?OLLAMA_API_KEY}.
  • A reasoning model’s thinking is not read on this route.

An internal gateway

A gateway that authenticates with its own header rather than a bearer token takes it in headers:

1
2
3
4
5
6
7
8
9

internal-gateway {
  provider = "openai-compatible"
  baseUrl = "https://llm-gateway.internal.example/v1"
  model = "gpt-4o-mini"
  headers {
    X-Team = "search"
    X-Gateway-Token = ${?GATEWAY_TOKEN}
  }
}

Config policy

If you gate configs with llm4s-config-policy, its dev preset allows openai-compatible and its prod preset does not: since the provider can point anywhere, a production config naming it fails with [allowedProviders] Provider 'openai-compatible' is not allowed until you allow it. Allow it explicitly, and pin the models and the endpoint:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

import org.llm4s.configpolicy.{ CatalogEnvironment, ConfigPolicy }

val policy = ConfigPolicy.prodSafeDefaults
  .withAllowedProviders("openai", "anthropic", "azure", "gemini", "deepseek", "openai-compatible")
  .withAllowedModelPatterns(
    "^openai/gpt-4o(-mini)?$", // ...and the other preset patterns you still need
    "^openai-compatible/openai/gpt-oss-120b$"
  )
  .withRequiredBaseUrlPattern(
    CatalogEnvironment.Prod,
    "openai-compatible",
    "https://api\\.groq\\.com/openai/v1"
  )
  .withMaxContextWindow(CatalogEnvironment.Prod, "openai-compatible", 131072)
  • withAllowedProviders and withAllowedModelPatterns replace the preset’s lists; repeat the entries you want to keep.
  • The prod preset also requires every section whose provider needs a key to set its own apiKey (rule ownApiKey, withOwnApiKeyRequired). The generic provider’s apiKey is optional, so it is not checked; the rule is about sections that would otherwise inherit a vendor’s shared key.
  • Model patterns match <provider>/<model>, so a Groq model is openai-compatible/openai/gpt-oss-120b. Model and base-URL patterns are regular expressions that must match the whole value, so ^ and $ are optional: openai/gpt-4o allows neither openai/gpt-4o-mini nor a dated snapshot such as openai/gpt-4o-2024-08-06 (write openai/gpt-4o(-.*)? for those). End a base-URL pin that allows paths with /.*, never a bare .*: https://api\.groq\.com.* also accepts https://api.groq.com.evil.example/v1.
  • withRequiredBaseUrlPattern(env, provider, pattern) pins one provider’s endpoint and replaces the environment-wide pin for that provider. The environment-wide form, withRequiredBaseUrlPattern(env, pattern), is checked against every provider without a pin of its own, not only openai-compatible; if you use it, name every endpoint in it (for example https://(api\.openai\.com|api\.groq\.com/openai)/v1).
  • Context caps work the same way. The prod preset caps each of its providers at its current models’ window (openai and azure 128000, anthropic 200000, gemini 1048576, deepseek 131072) and any other provider, openai-compatible included, at an environment-wide 1048576; dev caps everything at 1048576. So Groq, Together or Fireworks at 131072 and xAI at 500000 pass as they are. Set a per-provider cap with withMaxContextWindow(env, provider, max), as above, to hold a provider tighter, or to allow a model above 1048576. A section over its cap fails with contextWindow ... exceeds <max>. A window taken from the model registry counts too: a Groq section that sets no contextWindow resolves to 131072 and is checked at that.
  • Provider names in a per-provider pin or cap are canonicalised like provider ids (google is gemini). One that names no registered provider and is not in allowedProviders is an [unknownProvider] violation, so a typo cannot leave a provider unpinned.

See modules/config-policy/README.md for the rest of the module.

Troubleshooting OpenAI-compatible endpoints

404 on every request - wrong baseUrl. The client appends /chat/completions to baseUrl and nothing else, so baseUrl must stop exactly where the vendor’s path to /chat/completions begins. Missing /v1 (http://localhost:8000 for vLLM) and a doubled one (.../v1/v1/chat/completions, from a baseUrl of .../v1/chat/completions or with /v1 added twice) both return 404. Vendor paths differ: /openai/v1 for Groq, /inference/v1 for Fireworks, no /v1 at all for Perplexity Sonar. Model listing (<baseUrl>/models) fails the same way, which makes it a quick check.

401 or 403 - authentication. Surfaced as an AuthenticationError. Usually the key is not reaching the request: apiKey = ${?GROQ_API_KEY} with the variable unset leaves apiKey out, and the request goes without an Authorization header. Check the variable is exported in the process that runs your app. Otherwise the key belongs to another vendor or project, or - for NVIDIA - is the NGC_API_KEY used for pulling containers. For a local server started with --api-key, set the same key as apiKey.

Model not found. Vendors answer an unknown model with a 404 or a 400. Model ids must match exactly, prefix included (openai/gpt-oss-120b on Groq, accounts/fireworks/models/... on Fireworks), and hosted catalogues retire models regularly. List what the endpoint serves, with the section’s key and headers:

1
2
3

import org.llm4s.config.Llm4sConfig

Llm4sConfig.listModels("groq-main")   // GET <baseUrl>/models

No API key for a local server. Leave apiKey out - do not set a placeholder. OpenAI’s SDKs require some key, which is why vendor examples show api_key="ollama" or "not-used"; llm4s does not, and without one it sends no Authorization header at all. A 401 from a local server means it was started with a key (--api-key for vLLM and llama-server, “Require Authentication” in LM Studio), and apiKey must then match it.

Missing baseUrl. A section without one fails to load with baseUrl: set it in application.conf under llm4s.providers.<name>.baseUrl (e.g. http://localhost:8000/v1; to read it from OPENAI_COMPATIBLE_BASE_URL, add baseUrl = ${?OPENAI_COMPATIBLE_BASE_URL} to the section). Llm4sConfig reads no variable for it by itself: set baseUrl in the section, from a variable if you like - baseUrl = ${?OPENAI_COMPATIBLE_BASE_URL} binds the conventional one. If the section binds a variable and still fails this way, the variable is not set in the process that runs your app.

Streaming fails with a 400 or 422 naming stream_options, while complete works. The endpoint rejects fields it does not know. Set streamUsage = false in the section (see streamed token usage); the stream then carries usage only if the server sends it unasked.

Replies cut short, or context errors. Without a contextWindow, the generic provider uses the model registry’s window when it knows the model (see where the context window comes from), and otherwise 8192. Set it to the model’s real window, or the server’s configured context for a local server.

Provider docs

The sources for the values above:

Mistral

From the release after 0.4.1, Mistral ships in llm4s-openai-compatible: Mistral’s API is the OpenAI /v1/chat/completions format, so it runs on the shared client with a small dialect.

1

libraryDependencies += "org.llm4s" %% "llm4s-openai-compatible" % llm4sVersion

In 0.4.1 and earlier it is part of llm4s-core, and does not stream (#925).

Setup

  1. Get API key from console.mistral.ai
  2. Export the key - llm4s-openai-compatible binds it to llm4s.credentials.mistral.apiKey:
1

export MISTRAL_API_KEY=your-key

Configuration 

1
2
3
4
5
6
7
8
9
10
11
12
13

llm4s {
  providers {
    provider = "mistral-main"

    mistral-main {
      provider = "mistral"
      model = "mistral-large-latest"
      # apiKey defaults to MISTRAL_API_KEY
      # baseUrl defaults to https://api.mistral.ai - the API root; requests go to
      # <baseUrl>/v1/chat/completions, and a baseUrl already ending in /v1 is used as it is
    }
  }
}

What it supports

Chat, streaming (text, tool calls and token usage), tool calling, structured output (ResponseFormat.Json and ResponseFormat.JsonSchema), model listing and exchange logging.

  • Tool-call ids. Mistral accepts only nine-letter-or-digit ids. Its own pass through; any other id in the conversation (from another provider, say) is mapped to a stable nine-character one, the same for the call and its answer.
  • Reasoning models (Magistral) return their reasoning as Completion.thinking, and stream it as thinking deltas.
  • CompletionOptions.reasoning is not sent: only some Mistral models accept reasoning_effort.

Cohere

From the release after 0.4.1, Cohere ships in llm4s-openai-compatible and calls Cohere’s OpenAI-compatibility API (https://api.cohere.ai/compatibility/v1) rather than its native v2 /v2/chat:

1

libraryDependencies += "org.llm4s" %% "llm4s-openai-compatible" % llm4sVersion

In 0.4.1 and earlier it is part of llm4s-core, calls /v2/chat, sends text only and does not stream (#925).

Setup

  1. Get API key from dashboard.cohere.com
  2. Export the key - bound to llm4s.credentials.cohere.apiKey, which the Cohere reranker in llm4s-rag shares, so one variable serves both. (Cohere’s Python SDK prefers CO_API_KEY; llm4s reads COHERE_API_KEY only, the name it has always used.)
1

export COHERE_API_KEY=your-key

Configuration 

1
2
3
4
5
6
7
8
9
10
11
12

llm4s {
  providers {
    provider = "cohere-main"

    cohere-main {
      provider = "cohere"
      model = "command-a-03-2025"
      # apiKey defaults to COHERE_API_KEY
      # baseUrl defaults to https://api.cohere.ai/compatibility/v1
    }
  }
}

A baseUrl naming Cohere’s native API root, as configs written for 0.4.1 do (https://api.cohere.com, with or without /v1 or /v2), is mapped to <root>/compatibility/v1 and logged; a proxy root is treated the same way. A baseUrl already ending in /compatibility/v1 is used as it is.

What it supports

Chat, streaming (text, tool calls and token usage), tool calling, structured output and exchange logging. System messages are sent under the developer role, and a JSON schema as {"type": "json_object", "schema": ...}, as Cohere’s compatibility API documents. CompletionOptions.reasoning is not sent. Cohere’s reranker (llm4s-rag) is separate and unchanged.

Available Models

command-a-03-2025 and the other Command models your key can use; see Cohere’s model list.

xAI (Grok)

xAI’s Grok models, through xAI’s OpenAI-compatible Chat Completions API (https://api.x.ai/v1/chat/completions). The provider ships in llm4s-openai-compatible as a dialect on the shared client (#1023), and is Beta:

1

libraryDependencies += "org.llm4s" %% "llm4s-openai-compatible" % llm4sVersion

Setup

  1. Get API key from console.x.ai
  2. Export the key - llm4s-openai-compatible binds it to llm4s.credentials.xai.apiKey:
1

export XAI_API_KEY=xai-...

Configuration 

1
2
3
4
5
6
7
8
9
10
11
12

llm4s {
  providers {
    provider = "xai-main"

    xai-main {
      provider = "xai"
      model = "grok-4.7"
      # apiKey defaults to XAI_API_KEY
      # baseUrl defaults to https://api.x.ai/v1
    }
  }
}

The context window comes from the model registry when it lists the model, and otherwise from xAI’s published windows: 500k for grok-4.5 to grok-4.7, 1M for grok-4.3 and grok-4.20, 256k for the other Grok 4 models and 131072 for older ones.

What it supports

Chat, streaming (text, tool calls and token usage), tool calling, images in user messages, structured output, model listing (/v1/models) and exchange logging.

  • Reasoning output. A reply’s reasoning_content, where the model returns it (Grok 3 mini does), is Completion.thinking, and streams as thinking deltas. Grok 4.5 and later keep their reasoning encrypted and return it only on xAI’s Responses API, so on Chat Completions they return none. Earlier turns’ thinking is not sent back: Chat Completions has no field for it.
  • Reasoning tokens are TokenUsage.thinkingTokens, from usage.completion_tokens_details.reasoning_tokens. xAI counts them outside completion_tokens and inside total_tokens.
  • Reasoning effort. CompletionOptions.reasoning is sent as reasoning_effort where the model takes it. xAI’s reasoning models cannot stop reasoning, so None sends the lowest level, and the first such request in a process logs a warning:
Model ReasoningEffort.None Low Medium High
grok-4.5, grok-4.6, grok-4.7 (and their -latest aliases) low low medium high
grok-3-mini (and variants) low low high high
any other model not sent not sent not sent not sent

high is the default on Grok 4.5 and later; xhigh (4.6 and later) is not reachable from CompletionOptions. grok-4 rejects the parameter, and on grok-4.20-multi-agent it sets the number of agents, so neither is sent it. With reasoning unset nothing is sent.

  • Penalties. xAI rejects presence_penalty, frequency_penalty and stop on reasoning models. llm4s never sends stop; a non-zero presencePenalty or frequencyPenalty is left out of a request to a reasoning model (logged once), and sent as given to a -non-reasoning model.
  • xAI calls Chat Completions its legacy endpoint; new features ship on its Responses API, which this provider does not speak.

Available Models

grok-4.7 and the other models your key can use; see xAI’s model list.

Ollama (Local Models)

From the release after 0.4.1, Ollama support - chat and embeddings - ships in its own artifact, alongside llm4s-core:

1

libraryDependencies += "org.llm4s" %% "llm4s-ollama" % llm4sVersion

In 0.4.1 and earlier it is part of llm4s-core. See the installation guide.

Setup

  1. Install Ollama from ollama.ai
  2. Pull a model:
1
2

ollama pull mistral        # Downloads model
ollama serve               # Runs on http://localhost:11434
  1. Add a named section to application.conf and select it as the default. No API key is needed, and baseUrl defaults to http://localhost:11434, where ollama serve listens.

Configuration

In application.conf:

1
2
3
4
5
6
7
8
9
10
11
12

llm4s {
  providers {
    provider = "ollama-local"

    ollama-local {
      provider = "ollama"
      model    = "mistral"
      # baseUrl = "http://gpu-box:11434"  # optional: the default is http://localhost:11434
      # baseUrl = ${?OLLAMA_BASE_URL}       # optional: let OLLAMA_BASE_URL override the default
    }
  }
}

Structured output

llm4s-ollama honours CompletionOptions.responseFormat (and so completeStructured) through the format field of /api/chat, for streaming and non-streaming calls alike: ResponseFormat.Json sends "format": "json", and ResponseFormat.JsonSchema sends the schema object, which needs Ollama 0.5 or later. JsonSchema.name and strict have no Ollama equivalent and are ignored. With no responseFormat, no format field is sent.

Available Models

100+ models available:

  • Small: phi, neural-chat (~4GB)
  • Medium: mistral, llama2 (~13GB)
  • Large: llama2-70b (~40GB)
  • Specialized: neural-chat, orca, wizard-math

Run ollama list to see installed models.

Costs

Free! Just compute (CPU or GPU needed).

Tips

  • Perfect for development and testing
  • Works offline (no internet needed)
  • Use GPU for faster inference
  • Ideal for sensitive data (runs locally)

Thinking in conversation history

A model’s reasoning is returned on the message it produced, as AssistantMessage.thinking - a sequence of ThinkingBlocks - and Completion.thinking is that message’s thinking text. Because it lives on the message, it stays in the conversation (and in an agent’s thread) like the content and tool calls do, and each client sends it back where its provider takes it:

Provider Read from Sent back as
Anthropic thinking blocks (each with its signature) and redacted_thinking the same blocks, unchanged and first in the assistant turn
Bedrock (Converse) reasoningContent (text with signature, or redactedContent) the same blocks, unchanged and first in the assistant turn
Ollama message.thinking thinking on the assistant message
DeepSeek, Z.ai reasoning_content reasoning_content
OpenRouter reasoning / thinking, and reasoning_details reasoning, and reasoning_details unchanged (same items, order and fields)
Mistral (Magistral) thinking chunks in content a thinking chunk before the text chunk
Gemini API, Vertex AI a thoughtSignature on a part: the functionCall part, or the last text part (and thought summaries, thought: true, as text) the signature, on the same part (below)
OpenAI, Azure, Cohere, generic openai-compatible - not sent: the API has no field for it
IBM watsonx.ai reasoning_content (or reasoning) not sent: the chat API’s assistant message has no field for it

Anthropic and Bedrock require the signed blocks back when a thinking model’s turn ends in tool calls, and reject a thinking block without a signature, so unsigned thinking - from another provider earlier in the conversation, or set by hand with withThinking(text) - is left out of their requests. The Anthropic client sends tool calls and their results as tool_use and tool_result blocks for the same reason: the thinking has to sit in the turn that made the calls. Both clients send a call only when its result is in the run of tool messages straight after it, as those APIs require; a result anywhere else (after a user message, say) goes as [Tool result for <id>]: ... text, and its call is left out.

OpenRouter returns reasoning_details when the underlying model’s reasoning is signed, summarised or encrypted (Claude, Gemini, OpenAI reasoning models), and requires the whole sequence back unchanged when a conversation continues after tool calls. Each item is kept, with every field, as a ThinkingBlock.Opaque("openrouter", json) block after the reasoning text - streamed items are joined by index first - and only the OpenRouter client sends them back; every other client ignores opaque blocks that are not its own.

Gemini and Vertex AI thought signatures

A thinking Gemini model attaches an opaque, base64 thoughtSignature to a part of its turn and expects it back on the same part. Google’s Vertex AI guide states the rules these clients follow:

  • A response with functionCall parts needs its signature back: Gemini 3 models answer HTTP 400 when a required signature is missing. With parallel calls only the first functionCall part carries one; across sequential steps each step’s first call does. The part goes back “exactly as it was returned”.
  • A response without function calls may carry a signature on its last part (streaming can deliver it on a part with empty text). Sending it back is recommended, and leaving it out is not an error.
  • A part with a signature is never merged with one without.

Each signature is kept as a sealed ThinkingBlock.Opaque block of the client’s provider id (gemini or vertexai) and goes back only to the provider and model that produced it. Unlike Anthropic’s prefix rule, the binding is to that origin alone: Google asks for signatures to be preserved when history is modified or trimmed, and Gemini 3 answers HTTP 400 when the current turn’s function-call signature is missing, so pruning or compressing earlier turns leaves them in place. What unseals one is a change of provider or model, or an edit to the message that carries it. A function call’s signature is stored against the tool call’s id with the functionCall payload exactly as Gemini returned it, and the part is replayed verbatim (its optional id kept, args present or absent as received); a populated functionCall.id becomes the tool call’s own id and the matching functionResponse echoes it. A text part’s signature records its character offset in the message content, and the content is split there when the turn is sent, so a signed part is never merged with an unsigned one (if the content no longer fits, the text signature is left out). Parts are rebuilt as text first, then function calls, as these clients have always sent them.

Google’s documentation does not say whether a signature from the Gemini API validates on Vertex AI, so the two clients are separate signing authorities: a conversation moved from one to the other keeps its text and drops the signatures, never sending a foreign one. A signature on an image part, or on a thought-summary part, is not kept, since these clients neither request nor send those parts.

Signed, redacted and opaque thinking is sealed: it is valid only in the conversation it was produced in. Anthropic validates a thinking block against everything sent before it - the top-level system prompt, the tools and every earlier message - and rejects it (400) once any of those changes; a block must also come back unchanged, beside the content and tool calls it came with. Bedrock documents its reasoning signature as a hash of all the messages in the conversation. llm4s enforces both halves with two rules (hasSealedThinking reports which state a message is in):

  • The message itself. withContent and withToolCalls, given a changed value, unseal the thinking: they drop redacted and opaque blocks and signatures and keep the reasoning text. An agent’s afterAgent answer rewrite and a tool-call edit go through those setters, and so do the Anthropic and Bedrock clients when they leave out an unpaired call.
  • Everything before it. When Anthropic, Bedrock or OpenRouter returns sealed thinking, the client records a fingerprint of the request on the message (AssistantMessage.thinkingBinding): every system message, the tools, the response format and every earlier message in order, system messages in their positions (OpenRouter sends them inline, so moving one changes what came before), as sent. When the message is sent again, the client replays its sealed thinking only if the conversation before it still has that fingerprint, and sends it unsealed otherwise. The check runs at the point of sending, so it covers every rewrite of the history - ContextPruning and the agent’s context-window middleware, TokenWindow trimming, the context compressors (DeterministicCompressor, LLMCompressor, HistoryCompressor, ToolOutputCompressor, ContextManager), a handoff’s view of the thread, memory or RAG context inserted before existing turns, a changed system prompt or tool set, or a hand edit - without any of them having to know about thinking.
  • Who produced it. The fingerprint also covers the provider id and model that produced the thinking, and the client checks it against the provider and model it is about to call. A signature or reasoning item belongs to its producer, so a conversation produced by one client and continued with another - Bedrock then Anthropic, or one OpenRouter model then another - is sent unsealed, never with a foreign signature. Every client binds to the provider and model it is configured with, never the model a response reports: Anthropic and Bedrock may report an alias’s resolved snapshot, and a router such as openrouter/auto (or OpenRouter’s model fallbacks) reports the model it chose for that request. The provider that resolved the alias or chose the route is the one the thinking goes back to, and OpenRouter requires the complete reasoning_details sequence on a tool-call continuation, so routed turns keep it. Changing the configured model unseals every earlier turn. The endpoint is not part of it: a proxy or regional endpoint in front of the same provider changes nothing the provider checks.

Moving between Anthropic and Bedrock

A conversation that moves between the Anthropic API and Bedrock (a failover, say) has its signed thinking unsealed: the text is kept, the signatures and redacted blocks are dropped, and nothing is rejected. Anthropic documents that signature values are compatible across platforms (the Claude API, Amazon Bedrock and Google Cloud), but that is only one of the conditions for a replayed block to be accepted. The API also checks the model that produced it (each model reads its own thinking blocks and those of a fixed set of other models) and that everything sent before it is unchanged, and rejects the request with a 400 or drops the block when either fails. The two clients name the same model differently (claude-sonnet-4-5-20250929 against anthropic.claude-sonnet-4-5-20250929-v1:0) and serialise the system prompt and tools through different APIs, so this library cannot show that the prefix a block was signed over is the prefix the other platform sees. Losing the reasoning context on a failover is safe; a rejected request is not, so each remains its own signing authority. Treating them as one would need a mapping between the two model ids and a live check that a signature survives the move.

Limits that remain

  • OpenRouter routers and fallbacks. OpenRouter’s documentation says dynamic routers (openrouter/auto, openrouter/free) omit the reasoning field, so there is nothing to replay for them, and that the reasoning_details sequence must match the original output with nothing rearranged or modified. It does not say what happens to the details when a fallback sends the request to a different model, so they are sent back unchanged, as its documentation asks.
  • Editing a tool call on a turn that is still in progress (a human-in-the-loop edit) unseals that turn’s thinking, because the signature covers the tool call. Anthropic’s manual extended thinking requires the final assistant turn of a thinking-enabled request to begin with a thinking block, so such a request can be rejected; adaptive thinking drops that requirement. Gemini 3 answers 400 for a missing function-call signature in the current turn.

Earlier turns whose history is unchanged keep their sealed thinking: Anthropic recommends passing all thinking blocks back, keeps them in context on newer models, and accepts any unbroken run of the original blocks. A change unseals every turn after it and none before it, so the replayed blocks never have a gap. Unsealed thinking keeps its text, which Anthropic and Bedrock leave out and other providers still receive. Sealed thinking with no binding - built by hand with withThinking - is not replayed. To keep signed thinking, append to the conversation and leave what is already there as it is. Thinking also counts toward token estimates (ConversationTokenCounter), since most providers resend it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model.{ Completion, CompletionOptions, Conversation, ToolCall, ToolMessage }
import org.llm4s.types.Result

def toolTurn(
  client: LLMClient,
  conversation: Conversation,
  options: CompletionOptions,
  runTools: Seq[ToolCall] => Seq[ToolMessage]          // your tool execution
): Result[Completion] =
  for {
    first <- client.complete(conversation, options)   // a tool-call turn, with thinking
    results = runTools(first.message.toolCalls)
    next    = conversation.addMessage(first.message).addMessages(results) // keeps the thinking
    answer <- client.complete(next, options)           // the provider gets it back
  } yield answer

Citations and grounding sources

A model that searches the web by itself reports the sources behind its answer. llm4s returns them on Completion.citations, a List[Citation] that is empty for every completion that has none:

1
2
3
4

import org.llm4s.llmconnect.model.Completion

def sources(completion: Completion): List[String] =
  completion.citations.map(c => s"${c.title.getOrElse(c.url)} <${c.url}>")

A Citation has a url, always, and these fields, each None when the provider did not send it:

Field Meaning
title the source’s title
citedText a passage of the source that came with the citation (OpenRouter’s content); it describes the source, not the answer
startIndex, endIndex where the inline citation sits, as the provider reports it: positions in Completion.content. OpenAI documents them as the first and last character “of the URL citation in the message”; no provider documents them as the span of prose the source supports

Take the two indices as the location of the citation, not of the supported claim, and check the provider’s convention before cutting content with them (whether end_index is inclusive is the provider’s call). hasSpan says whether both are present.

Provider Citations
OpenAI (search models for Chat Completions), Azure, Requesty Read from the message’s url_citation annotations (url, title, start_index, end_index). Search models (gpt-5-search-api; the *-search-preview models were retired on 2026-07-23) return them with no request option; Azure and Requesty return them only if the deployment does.
OpenRouter (:online models, or the web plugin) The same annotations, plus content as citedText.
DeepSeek, Z.ai, Mistral, Cohere, a generic openai-compatible endpoint Read if the reply carries standard url_citation annotations; none of them is documented to.
Gemini, Vertex AI With googleSearch = true in the section (see Google Search grounding): one citation per source in the candidate’s groundingMetadata.groundingChunks - url and title, and for a Vertex AI Search / RAG source (retrievedContext) its text as citedText. No indices: Gemini’s groundingSupports are UTF-8 byte offsets of the supported text, not where a citation sits. Without the setting, empty. Read from complete and from a stream.
Anthropic With webSearch = true in the section (see Web search): one citation per source a text block cites - for a web_search_result_location its url, title and cited_text, for a search_result_location its source as url. Document citations (char_location, page_location, content_block_location) name a document, not a URL, and are skipped. No indices: Anthropic attaches a citation to a text block, not to a position in it. Read from complete and from a stream (citations_delta).
Perplexity The url_citation annotations on the answer, then each web_search result (url, title, the snippet as citedText) not already cited; see Perplexity. Read from complete and from a stream.
Ollama, Bedrock, watsonx Empty.

Things to know:

  • Streaming. For OpenAI and OpenRouter, a completion assembled from streamed chunks has no citations: neither documents where a stream carries them, so streamed chunks are not read. Gemini and Vertex AI send groundingMetadata in a streamed chunk (usually the last), and the streamed completion carries it. Anthropic streams each citation as a citations_delta, and the streamed completion carries them. (A model that does not stream natively is answered with one ordinary call, and that completion keeps its citations.)
  • A citation never fails a reply. One without a url is dropped, a field of the wrong type is read as absent, and the answer is returned either way. The OpenAI client treats an annotations list the SDK cannot parse at all as no citations; the OpenAI-compatible client keeps the entries it can read.
  • The shapes come from the OpenAI Java SDK’s url_citation types and OpenRouter’s documentation. No live provider was called while writing this.

Enterprise authentication

Many organisations forbid long-lived API keys for cloud models and require the platform’s own identity instead: a managed identity, a workload identity or an IAM role. Each of the three clouds works that way in llm4s, with no key in application.conf.

Cloud Provider How it authenticates without a key
Microsoft Azure azure (llm4s-openai + llm4s-azure-identity) authentication = "entra-id": Microsoft Entra ID tokens from DefaultAzureCredential
AWS bedrock (llm4s-bedrock) The AWS default credentials chain: environment, ~/.aws profile, an instance, task or IRSA role
Google Cloud vertexai (llm4s-gemini) Application Default Credentials: GOOGLE_APPLICATION_CREDENTIALS, gcloud’s login, the metadata server

Azure OpenAI with Microsoft Entra ID

Azure Policy can disable key (“local”) authentication on an Azure OpenAI resource. Such a resource takes Microsoft Entra ID tokens instead, and so does llm4s (#2274). Add llm4s-azure-identity beside llm4s-openai:

1
2

// build.sbt - same version as llm4s-core
libraryDependencies += "org.llm4s" %% "llm4s-azure-identity" % llm4sVersion

and set authentication = "entra-id" in the section, with no apiKey:

1
2
3
4
5
6
7
8
9
10

llm4s.providers {
  provider = "azure-prod"
  azure-prod {
    provider       = "azure"
    endpoint       = "https://my-resource.openai.azure.com"
    model          = "gpt-4o"              # the deployment name
    authentication = "entra-id"
    # clientId     = ${?AZURE_CLIENT_ID}   # a user-assigned managed identity
  }
}

The token comes from Azure’s DefaultAzureCredential, which tries, in turn, environment credentials (AZURE_TENANT_ID, AZURE_CLIENT_ID and a secret or certificate), workload identity on AKS, a managed identity, and a developer’s az login. It is requested for the https://cognitiveservices.azure.com/.default scope, cached, and fetched again five minutes before it expires, so no request carries an expired token; one the service rejects with a 401 is dropped, and the next request fetches a new one. The identity needs the Cognitive Services OpenAI User role on the resource.

Such a section takes no API key: setting apiKey as well is an error, and the shared AZURE_OPENAI_API_KEY is not used. Without llm4s-azure-identity on the classpath, loading the section fails with a configuration error naming it. llm4s-azure-identity brings Azure’s azure-identity (with azure-core, MSAL4J, Netty and Reactor), which is why it is a module of its own: llm4s-openai alone puts no Azure SDK on your classpath.

In code, any AccessTokenProvider authenticates a config, and AzureEntraId makes one from Azure’s DefaultAzureCredential or from any other TokenCredential:

1
2
3
4
5
6
7
8

import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.auth.azure.AzureEntraId
import org.llm4s.llmconnect.config.AzureConfig
import org.llm4s.model.ModelRegistryService

def azureClient(using ModelRegistryService) =
  val tokens = AzureEntraId.defaultCredential() // or AzureEntraId.tokenProvider(anyTokenCredential)
  LLMConnect.getClient(AzureConfig.keyless("https://my-resource.openai.azure.com", "gpt-4o", tokens))

Embeddings take the same tokens: OpenAIEmbeddingProvider.fromConfig(config, tokens), with the resource’s v1 API as the base URL (https://my-resource.openai.azure.com/openai/v1) and the embedding deployment’s name as the model. From Java, Llm4s.azureOpenAi().entraId() (or .entraId(clientId), or .tokenProvider(tokens)) replaces .apiKey(...). With the Spring Boot starter, the properties are the section’s keys: llm4s.providers.azure-prod.authentication=entra-id and llm4s.providers.azure-prod.client-id=....

AWS Bedrock with IAM roles

A bedrock section that sets no accessKeyId uses the AWS SDK’s default credentials chain: the AWS_* environment variables, a profile in ~/.aws, and the role of the EC2 instance, ECS task or EKS service account (IRSA) the application runs as. See the installation guide.

Vertex AI with Application Default Credentials

A vertexai section with no apiKey uses Application Default Credentials: GOOGLE_APPLICATION_CREDENTIALS, then gcloud’s application_default_credentials.json, then the metadata server of the GCE, GKE or Cloud Run workload. See Google Vertex AI.

API Key Management

Security Best Practices

Never commit API keys!

  1. Keep keys in environment variables. The vendor’s own variable is enough - each provider module binds it to the vendor’s shared key:
1

openai-main { provider = "openai", model = "gpt-4o" }

1
2

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...

For a second account, or in production where each section should say which account it bills, bind the section’s own key: apiKey = ${?OPENAI_BATCH_API_KEY}. The config-policy prod preset flags sections without one. llm4s logs at INFO which path each key came from, never the value.

  1. Use .env file (add to .gitignore):
1
2
3

# .env (NOT committed to git)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
  1. Use CI/CD secrets:
1
2
3
4

# GitHub Actions
- uses: actions/setup-java@v3
  env:
    OPENAI_API_KEY: $
  1. Rotate keys regularly on provider dashboards

Using Keys Safely in Code

Good:

1
2
3
4

import org.llm4s.config.Llm4sConfig

// Keys come from the vendor's variable or the section's own ${?VAR} binding - never hardcoded
val providerConfig = Llm4sConfig.defaultProvider()

Bad:

1
2
3

// ❌ Never do this!
val key = "sk-proj-abc123..."  // Hardcoded
sys.env.get("OPENAI_API_KEY")  // Outside config boundary

Base URL Customization

When to Use Custom Base URLs

  • Reverse proxy or load balancer
  • VPC endpoint for security
  • Azure OpenAI or self-hosted setup
  • Provider migration (e.g., from OpenAI to similar API)

Setting Custom URLs

A base URL is the baseUrl key of the named section (endpoint for Azure OpenAI). No <PROVIDER>_BASE_URL variable is read by itself; bind one if you want the environment to override the URL:

1
2
3
4
5
6
7
8
9
10
11
12

llm4s {
  providers {
    provider = "openai-main"

    openai-main {
      provider = "openai"
      model    = "gpt-4o"
      baseUrl  = "https://proxy.example.com/openai"
      baseUrl  = ${?OPENAI_BASE_URL}   # optional: overrides the line above when set
    }
  }
}

In HOCON the later line wins, and ${?VAR} with the variable unset leaves the earlier value in place. Defaults when baseUrl is omitted:

Provider Default baseUrl
openai https://api.openai.com/v1
anthropic https://api.anthropic.com
gemini https://generativelanguage.googleapis.com/v1beta
deepseek https://api.deepseek.com
mistral https://api.mistral.ai
cohere https://api.cohere.ai/compatibility/v1 (a native root is mapped to <root>/compatibility/v1)
openrouter https://openrouter.ai/api/v1
zai https://api.z.ai/api/paas/v4
requesty https://router.requesty.ai/v1
ollama http://localhost:11434
openai-compatible none - required

Provider Comparison Table

Feature OpenAI Anthropic Gemini Azure DeepSeek Cohere Ollama OpenAI-compatible
Setup Difficulty Easy Easy Easy Hard Easy Easy Medium Easy
API Key Required ✅ ✅ ✅ ✅ ✅ ✅ ❌ Hosted ✅, local ❌
Free Tier Limited Limited ✅ Generous ❌ Limited Limited ✅ Endpoint-specific
Local Option ❌ ❌ ❌ ❌ ❌ ❌ ✅ ✅ vLLM, LM Studio, llama.cpp
Context Window 128K 200K 1M 128K 4K-32K 8K Model-specific Set in config (default 8K)
Vision Support ✅ ✅ ✅ ✅ ⚠️ Limited ❌ Model-specific ❌ Text only
Function Calling ✅ ✅ ✅ ✅ ✅ ✅ ✅ If the model supports it ✅ If the endpoint supports it
Reasoning Models ✅ o1 ❌ ❌ ✅ (via OpenAI) ✅ deepseek-reasoner ❌ ❌ ⚠️ Run, but reasoning not configured or read
Enterprise Support ✅ ✅ ✅ ✅ ⚠️ ✅ N/A Endpoint-specific
Cost (Budget) Medium Medium 🏆 Low High 🏆 Very Low Low Free Endpoint-specific
Speed Fast Medium 🏆 Very Fast Medium Fast Medium Varies Endpoint-specific
Reliability 🏆 Enterprise 🏆 Enterprise Good 🏆 Enterprise Good Good Local Endpoint-specific

Which Provider Should I Use?

  • Getting started? → Try Gemini (free tier) or Ollama (local)
  • Production API? → OpenAI (most stable) or Anthropic (best reasoning)
  • Cost-conscious? → DeepSeek or Ollama
  • Enterprise? → Azure OpenAI or Anthropic
  • Private data? → Ollama (runs locally)
  • xAI (Grok)? → xAI
  • Groq, Together, Fireworks, NVIDIA NIM, or a local vLLM, LM Studio or llama.cpp server? → the generic OpenAI-compatible provider
  • Reasoning tasks? → Anthropic Claude or DeepSeek reasoner
  • Vision/multimodal? → OpenAI GPT-4o or Anthropic Claude

Multiple Providers in One App

Configure one named section per provider, then load the default or any section by name:

1
2
3
4
5
6
7
8
9
10
11
12
13

import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.model.ModelRegistryService

// conversation: the Conversation to send
for {
  registry <- Llm4sConfig.modelRegistryService()
  given ModelRegistryService = registry
  primaryConfig  <- Llm4sConfig.defaultProvider()      // llm4s.providers.provider
  fallbackConfig <- Llm4sConfig.provider("claude")     // llm4s.providers.claude
  primary  <- LLMConnect.getClient(primaryConfig)
  fallback <- LLMConnect.getClient(fallbackConfig)
} yield primary.complete(conversation).orElse(fallback.complete(conversation))

Each section is validated when it is loaded, so here both must have a key available; a problem in either surfaces as the Left of this for. See Switching providers.

This enables:

  • Fallback logic - Use OpenAI, fall back to Anthropic
  • A/B testing - Compare provider outputs
  • Cost optimization - Use cheapest available provider

Troubleshooting

“Invalid API Key” 

1
2
3
4
5
6
7
8
9

# Verify the vendor's variable is set
echo $OPENAI_API_KEY
# ...and that the section sets no apiKey of its own, which would win over it.
# The INFO log line "llm4s.providers.<name>: API key from <path>" says which was used.

# Check key format (starts with correct prefix)
# OpenAI: sk-proj-* or sk-*
# Anthropic: sk-ant-*
# Gemini: Should be long alphanumeric

“Connection refused”

For local providers (Ollama):

1
2
3
4
5

# Check if Ollama is running
curl http://localhost:11434/api/tags

# Start Ollama
ollama serve

“Model not found”

Check the model key of the section you selected, and that its provider is the one serving that model (model = "gpt-4o" under provider = "openai"; no openai/ prefix):

1
2
3

# Check available models
# OpenAI: https://platform.openai.com/docs/models
# Anthropic: https://docs.anthropic.com/claude/reference/models

“Rate limit exceeded”

Use provider-specific strategies:

  • OpenAI: Wait before retrying, use batching API
  • Gemini: Upgrade from free tier
  • Ollama: Increase system resources or use GPU