org.llm4s.llmconnect.provider

@Stable

LLMClient implementation for Anthropic Claude models.

Uses the official Anthropic Java SDK (AnthropicOkHttpClient) for all API calls. SDK exceptions are mapped to the appropriate org.llm4s.error.LLMError subtypes before being returned.

== Message format adaptations ==

The Anthropic Messages API differs from the OpenAI convention in several ways that this client handles transparently:

Default system prompt: if the conversation contains no SystemMessage, the client injects "You are Claude, a helpful AI assistant." automatically. Supply an explicit SystemMessage to override this.
Tool calls as content blocks: an AssistantMessage's tool calls are sent as tool_use blocks after its text, and each ToolMessage answering one as a tool_result block, consecutive results sharing one user turn. A tool call with no ToolMessage in the conversation is left out (Anthropic rejects an unanswered tool_use), and a ToolMessage whose call is not in the conversation - after pruning, say - is sent as user text prefixed "[Tool result for <toolCallId>]: ", followed by its media blocks.
Content parts: user messages and tool results send text; PNG, JPEG, GIF and WebP images by URL or inline; PDFs by URL or inline; and inline plain text as a text document. Any other part - a provider file, another image or document type, plain text by URL - fails the request with UnsupportedContentPartError before anything is sent.
Thinking replayed: an AssistantMessage's signed thinking blocks and redacted thinking go back first in its turn, unchanged and in order, as extended thinking with tool use requires. Unsigned thinking (from another provider) is left out, since Anthropic rejects a thinking block without a signature. Returned sealed thinking is bound to the request, and replayed only while the conversation before it is unchanged (Anthropic rejects a block whose earlier history, system prompt or tools changed); otherwise it goes unsealed (see ThinkingReplay).
Schema sanitisation: OpenAI-specific fields (strict, additionalProperties) are stripped from tool schemas before sending, because Anthropic's API rejects them.

== Extended thinking ==

When CompletionOptions.reasoning is set, a thinking block is added to the request. The response's thinking and redacted_thinking blocks - streamed or not - become the returned message's org.llm4s.llmconnect.model.AssistantMessage.thinking, each thinking block with its signature, so that a later request can send them back. The token budget is clamped to [1024, maxTokens - 1] to satisfy the Anthropic API constraint; the effective budget may therefore differ from what was requested.

maxTokens defaults to 2048 when not set in CompletionOptions because the Anthropic API requires the field.

Value parameters

config: AnthropicConfig carrying the API key, model name, and base URL.
metrics: Receives per-call latency and token-usage events. Defaults to MetricsCollector.noop.

Attributes

Companion: object
Supertypes: trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: AnthropicClient.type

@Stable

Registration for the Anthropic Claude API.

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: AnthropicProvider.type

@Stable

Registration for Azure OpenAI deployments.

Azure reuses OpenAIClient but not OpenAIConfig: its endpoint is per-deployment and it requires an apiVersion, which AzureConfig carries.

Both are provider-specific keys of the section - endpoint (required) and apiVersion (default org.llm4s.llmconnect.config.AzureConfig.DEFAULT_API_VERSION) - rather than fields of NamedProviderConfig, which they were until #1133; the HOCON is unchanged:

azure-main {
 provider   = "azure"
 model      = "gpt-4o"
 endpoint   = "https://my-resource.openai.azure.com/"
 apiVersion = "2025-01-01-preview"
}

A resource that forbids keys authenticates with Microsoft Entra ID instead (#2274): authentication = "entra-id" takes tokens from Azure's DefaultAzureCredential - a managed identity, workload identity, the Azure CLI's login - through llm4s-azure-identity, which must then be on the classpath. Such a section takes no apiKey, its own or the shared AZURE_OPENAI_API_KEY; clientId picks a user-assigned managed identity:

azure-prod {
 provider       = "azure"
 model          = "gpt-4o"
 endpoint       = "https://my-resource.openai.azure.com"
 authentication = "entra-id"
 clientId       = ${?AZURE_CLIENT_ID}
}

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: AzureProvider.type

org.llm4s.llmconnect.LLMClient for the AWS Bedrock Converse and ConverseStream APIs.

Reaches the models Bedrock hosts - Anthropic Claude, Meta Llama, Mistral, Amazon Nova and Titan, and others - through Bedrock's one Converse interface.

== Authentication ==

Explicit org.llm4s.llmconnect.config.BedrockCredentials when configured (with a session token for temporary credentials); otherwise the config's profile; otherwise the AWS default credential chain (environment variables, ~/.aws/credentials, EC2 instance profile, ECS task role, ...).

== Streaming ==

streamComplete uses ConverseStream, which the AWS SDK offers on its asynchronous client. Its callbacks run on SDK threads; they only enqueue events, and the calling thread drains the queue, so onChunk runs on the caller's thread, in order, and interrupting the caller cancels the request and returns Left(CancelledError).

== Timeouts ==

The section's timeouts.request becomes the SDK client's API-call timeout for Converse; without it the AWS SDK keeps its defaults. timeouts.stream bounds a ConverseStream call from the request to the end of the stream; without it a stream has no limit. Either expiry is a TimeoutError.

== Errors ==

ThrottlingException and ServiceQuotaExceededException become RateLimitError, ValidationException ValidationError, AccessDeniedException (and any 401/403) AuthenticationError, other service errors ServiceError, and a failure to reach Bedrock at all NetworkError.

Value parameters

config: region, model, credentials and endpoint.
exchangeLogging: optional provider exchange logging.
metrics: receives per-call latency and token-usage events.

Attributes

Companion: object
Supertypes: trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: BedrockClient.type

Registration for AWS Bedrock, through the Converse and ConverseStream APIs.

A section names the AWS region (required, never defaulted: a model is only available in some regions, and a silent default would send traffic to the wrong one). With no credentials it authenticates through the AWS default credential chain; it can instead name a profile, or set accessKeyId and secretAccessKey (plus sessionToken for temporary credentials). Bedrock has no single API key, so there is no llm4s.credentials.bedrock block.

claude-bedrock {
 provider = "bedrock"
 model    = "anthropic.claude-3-5-sonnet-20241022-v2:0"
 region   = ${?AWS_REGION}
}

baseUrl is an optional endpoint override (a VPC endpoint, or a test server).

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: BedrockProvider.type

@Experimental

Cohere client, over Cohere's OpenAI-compatibility API (https://api.cohere.ai/compatibility/v1/chat/completions).

An OpenAICompatibleClient with CohereDialect, so it has everything the shared client has: completion, streaming (including streamed tool calls and usage), tool calling, structured output and one exchange recorded per call. Until #1132 it was a separate client for the native v2 /v2/chat API that sent text only and whose streamComplete returned a Left (#925). The Compatibility API supports chat, streaming, tools and response_format, so everything that client did carries over, and the rest comes with the shared client rather than a second SSE parser for Cohere's native event format.

Requests go to <baseUrl>/chat/completions, where baseUrl is mapped through org.llm4s.llmconnect.config.CohereConfig.compatibilityBaseUrl, so a config naming the native root still works.

Value parameters

config: Cohere configuration: API key, model, base URL and context settings.
exchangeLogging: where raw request/response exchanges are recorded, if anywhere.
metrics: receives per-call latency and token-usage events.

Attributes

Companion: object
Supertypes: class OpenAICompatibleClient

trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

@Experimental

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: CohereClient.type

Embedding provider implementation for Cohere's native embed API (POST <baseUrl>/v2/embed).

This is not an OpenAI-compatible endpoint: it takes texts, an input_type and the embedding_types to return, and answers with the vectors keyed by type ({"embeddings": {"float": [[...]]}}). That is why it is a module of its own and not a dialect in llm4s-openai-compatible, where Cohere's chat client lives.

Texts are sent in requests of at most CohereEmbeddingProvider.MaxTextsPerRequest (Cohere's limit); the vectors come back in input order and the billed tokens of every request are summed into the response's usage. The first failing request fails the whole call.

Requires a Cohere API key (COHERE_API_KEY) in the provider configuration. The vectors have the model's default size; no output_dimension is sent.

Cohere supplies chat through llm4s-openai-compatible, so this is an org.llm4s.llmconnect.spi.EmbeddingProviderDescriptor only, registered by Llm4sCohereModule.

Attributes

See also: EmbeddingProvider for the common embedding interface
Supertypes: trait EmbeddingProviderDescriptor

class Object

trait Matchable

class Any
Self type: CohereEmbeddingProvider.type

The Cohere input_type: which side of retrieval a text is on.

Cohere's v3 and v4 models embed a query and a document differently, so a mismatch between how texts were indexed and how a query is embedded quietly degrades retrieval. A request says which it is with its org.llm4s.llmconnect.model.InputPurpose, and the provider maps it onto SearchQuery or SearchDocument by itself, not guessed from the number of inputs or encoded in the model name. An input type the purpose cannot express (Classification, Clustering) is a typed setting of the provider: pass it to CohereEmbeddingProvider.fromConfig, where it takes precedence over the purpose of every request.

Value parameters

wireName: the value sent as input_type in the request body

Attributes

Companion: object
Supertypes: trait Enum

trait Serializable

trait Product

trait Equals

class Object

trait Matchable

class Any
Show all

Attributes

Companion: enum
Supertypes: trait Sum

trait Mirror

class Object

trait Matchable

class Any
Self type: CohereInputType.type

@Experimental

Registration for the Cohere API, through its OpenAI-compatibility endpoint.

Streaming is supported - the default org.llm4s.llmconnect.spi.ProviderFeatures - since Cohere moved onto the shared OpenAI-compatible client (#925, #1132).

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: CohereProvider.type

@Stable

Centralized cost estimation for LLM completions.

This provides a single source of truth for estimating completion costs based on token usage and model pricing information. It integrates with the ModelRegistryService to look up pricing data and applies it to usage statistics.

The estimator:

Uses existing ModelPricing logic (no duplication)
Returns None if pricing is unavailable
Preserves precision of micro-cost values
Works uniformly across all providers

Example usage:

 val usage = TokenUsage(promptTokens = 100, completionTokens = 50, totalTokens = 150)
 val cost = CostEstimator.estimate("gpt-4o", usage)
 // cost: Some(0.0015) for gpt-4o pricing

Attributes

Supertypes: class Object

trait Matchable

class Any
Self type: CostEstimator.type

@Stable

DeepSeek LLM client implementation using the OpenAI-compatible API.

Provides access to DeepSeek models including DeepSeek-Chat (V3) and DeepSeek-Reasoner (R1) for advanced reasoning tasks.

An OpenAICompatibleClient with DeepSeekDialect: the standard chat-completions format plus DeepSeek's User-Agent, and reasoning_content - the reasoner's chain of thought - read back as Completion.thinking and as streamed thinking deltas.

Value parameters

config: DeepSeek configuration containing API key, model, base URL, and context settings
exchangeLogging: where raw request/response exchanges are recorded, if anywhere
metrics: MetricsCollector for recording request metrics

Attributes

Companion: object
Supertypes: class OpenAICompatibleClient

trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: DeepSeekClient.type

@Stable

Registration for the DeepSeek API.

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: DeepSeekProvider.type

@Stable

Text embedding provider interface for generating vector representations.

Provides a unified interface for different embedding services (OpenAI, VoyageAI, Ollama). Each implementation handles provider-specific API calls and response formats.

Text content is the primary input; multimedia content (images, audio) should be processed through the FileEmbedder façade in llm4s-rag, which handles content extraction before embedding.

== Query or document ==

request.purpose says whether the texts are documents to index (org.llm4s.llmconnect.model.InputPurpose.Document, the default) or queries to run against them (org.llm4s.llmconnect.model.InputPurpose.Query). A provider whose API embeds the two differently maps it onto the vendor's own parameter (Voyage and Cohere input_type, Jina task); a provider whose models embed both alike ignores it.

== Usage Example ==

val provider: EmbeddingProvider = OpenAIEmbeddingProvider.fromConfig(config)
val request = EmbeddingRequest(
 input = Seq("Hello world", "How are you?"),
 model = EmbeddingModelConfig("text-embedding-3-small", dimensions = 1536)
)
val result: Result[EmbeddingResponse] = provider.embed(request)

Attributes

See also: OpenAIEmbeddingProvider for OpenAI text-embedding models (llm4s-openai)

VoyageAIEmbeddingProvider for VoyageAI embedding models (llm4s-voyage)

OllamaEmbeddingProvider for local Ollama embedding models (llm4s-ollama)
Supertypes: class Object

trait Matchable

class Any

@Stable

LLMClient implementation for Google Gemini models.

Calls the Google Generative AI REST API directly using org.llm4s.http.Llm4sHttpClient.

== Message format ==

Gemini uses a different conversation structure from OpenAI:

Roles are "user" and "model" (not "user" and "assistant").
SystemMessage values are sent as a separate systemInstruction field, not inside the contents array.
Tool results (ToolMessage) are sent as functionResponse parts inside a "user" turn, keyed by function name (not tool-call ID). The function name is resolved from an in-request map built while processing the preceding AssistantMessage.

== Tool call IDs ==

The Gemini API does not return an ID with function-call responses. This client generates a random UUID for each tool call so that the llm4s ToolCall / ToolMessage pairing convention is preserved. These IDs are synthetic and are not round-tripped to Gemini.

== Authentication ==

The API key is appended as a ?key= query parameter on every request (Google's API requires this; it is not sent as a header). The full URL is not logged; only the base URL and model are emitted at DEBUG level.

== Schema sanitisation ==

OpenAI-specific fields (strict, additionalProperties) are stripped from tool schemas before sending, because Gemini's API rejects them.

Value parameters

config: GeminiConfig with API key, model, and base URL.
metrics: Receives per-call latency and token-usage events. Defaults to MetricsCollector.noop.

Attributes

Companion: object
Supertypes: trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: GeminiClient.type

@Stable

Registration for the Google Gemini API.

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: GeminiProvider.type

@Stable

Shared HTTP status-code to org.llm4s.error.LLMError mapping used by all HTTP-based LLM provider clients.

Centralises the duplicated pattern of converting non-2xx responses into typed Result errors. Provider-specific error details are extracted from the JSON response body when possible and truncated to a safe length.

Attributes

Supertypes: class Object

trait Matchable

class Any
Self type: HttpErrorMapper.type

Embedding provider implementation for the Jina AI embedding API.

Generates text embeddings by posting batched input to Jina's <baseUrl>/embeddings endpoint (the default base URL is https://api.jina.ai/v1), with a JinaTask as the task field: the one the provider was built with if there is one, otherwise the one the request's org.llm4s.llmconnect.model.InputPurpose stands for (JinaTask.forPurpose). All texts go in one HTTP call.

Requires a valid Jina AI API key (JINA_API_KEY) in the provider configuration.

Jina supplies embeddings and no chat client, so it is an org.llm4s.llmconnect.spi.EmbeddingProviderDescriptor only, registered by Llm4sJinaModule.

Attributes

See also: EmbeddingProvider for the common embedding interface
Supertypes: trait EmbeddingProviderDescriptor

class Object

trait Matchable

class Any
Self type: JinaEmbeddingProvider.type

The Jina AI embedding task: which LoRA adapter jina-embeddings-v3 applies to the input.

Jina embeds queries and documents differently, so the right task depends on what the caller is embedding. A request says so with its org.llm4s.llmconnect.model.InputPurpose, and the provider maps it onto RetrievalQuery or RetrievalPassage by itself. A task the purpose cannot express (TextMatching, Classification, Separation) is a typed setting of the provider, not part of the model name: pass it to JinaEmbeddingProvider.fromConfig, where it takes precedence over the purpose of every request.

Value parameters

wireName: the value sent as task in the request body

Attributes

Companion: object
Supertypes: trait Enum

trait Serializable

trait Product

trait Equals

class Object

trait Matchable

class Any
Show all

Attributes

Companion: enum
Supertypes: trait Sum

trait Mirror

class Object

trait Matchable

class Any
Self type: JinaTask.type

@Stable

Registers llm4s-anthropic's provider: the Anthropic Claude API (id anthropic).

Anthropic supplies a chat client only, so this module overrides chatProviders and leaves embeddingProviders empty.

Declared in this module's META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule, so adding the llm4s-anthropic dependency is all it takes for ProviderRegistry.default to resolve provider = "anthropic". Where discovery cannot run - a fat jar whose services files were overwritten rather than merged - register it explicitly:

given ProviderRegistry = ProviderRegistry.ofModules(new Llm4sAnthropicModule)

A class rather than an object because java.util.ServiceLoader instantiates the class named in the services file through its public no-arg constructor.

Attributes

Supertypes: trait Llm4sProviderModule

class Object

trait Matchable

class Any

Registers llm4s-bedrock's one provider: AWS Bedrock, under the id bedrock. Bedrock supplies a chat client and no embedding provider here, so this module overrides only chatProviders.

Declared in this module's META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule, so adding the llm4s-bedrock dependency is all it takes for ProviderRegistry.default to resolve provider = "bedrock". Where discovery cannot run - a fat jar whose services files were overwritten rather than merged - register it explicitly:

given ProviderRegistry = ProviderRegistry.ofModules(new Llm4sBedrockModule)

A class rather than an object because java.util.ServiceLoader instantiates the class named in the services file through its public no-arg constructor.

Attributes

Supertypes: trait Llm4sProviderModule

class Object

trait Matchable

class Any

Registers llm4s-cohere's one provider: the Cohere embedding provider, under the id cohere. Cohere's chat client is a dialect in llm4s-openai-compatible (under the same id: chat and embedding ids are separate namespaces), so this module overrides only embeddingProviders.

Declared in this module's META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule, so adding the llm4s-cohere dependency is all it takes for ProviderRegistry.default to resolve EMBEDDING_MODEL=cohere/<model>. Where discovery cannot run - a fat jar whose services files were overwritten rather than merged - register it explicitly:

given ProviderRegistry = ProviderRegistry.ofModules(new Llm4sCohereModule)

A class rather than an object because java.util.ServiceLoader instantiates the class named in the services file through its public no-arg constructor.

Attributes

Supertypes: trait Llm4sProviderModule

class Object

trait Matchable

class Any

@Stable

Registers llm4s-gemini's providers: the Google Gemini API (id gemini, alias google) and Google Cloud Vertex AI (id vertexai, alias vertex).

Vertex AI ships here rather than in a module of its own because it only calls Google's publishers/google (Gemini) models, in the same JSON format as the Gemini API. It differs in endpoint and authentication, not in model family, and adds no dependency.

Declared in this module's META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule, so adding the llm4s-gemini dependency is all it takes for ProviderRegistry.default to resolve provider = "gemini" and provider = "vertexai". Where discovery cannot run - a fat jar whose services files were overwritten rather than merged - register it explicitly:

given ProviderRegistry = ProviderRegistry.ofModules(new Llm4sGeminiModule)

A class rather than an object because java.util.ServiceLoader instantiates the class named in the services file through its public no-arg constructor.

Attributes

Supertypes: trait Llm4sProviderModule

class Object

trait Matchable

class Any

Registers llm4s-jina's one provider: the Jina AI embedding provider, under the id jina. Jina supplies embeddings and no chat client, so this module overrides only embeddingProviders.

Declared in this module's META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule, so adding the llm4s-jina dependency is all it takes for ProviderRegistry.default to resolve EMBEDDING_MODEL=jina/<model>. Where discovery cannot run - a fat jar whose services files were overwritten rather than merged - register it explicitly:

given ProviderRegistry = ProviderRegistry.ofModules(new Llm4sJinaModule)

A class rather than an object because java.util.ServiceLoader instantiates the class named in the services file through its public no-arg constructor.

Attributes

Supertypes: trait Llm4sProviderModule

class Object

trait Matchable

class Any

@Stable

Registers llm4s-ollama's providers: the Ollama chat client and the Ollama embedding provider, both under the id ollama.

Declared in this module's META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule, so adding the llm4s-ollama dependency is all it takes for ProviderRegistry.default to resolve provider = "ollama" and EMBEDDING_MODEL=ollama/<model>. Where discovery cannot run - a fat jar whose services files were overwritten rather than merged - register it explicitly:

given ProviderRegistry = ProviderRegistry.ofModules(new Llm4sOllamaModule)

A class rather than an object because java.util.ServiceLoader instantiates the class named in the services file through its public no-arg constructor.

Attributes

Supertypes: trait Llm4sProviderModule

class Object

trait Matchable

class Any

@Stable

Registers llm4s-openai-compatible's providers: the generic openai-compatible provider, for any endpoint speaking the OpenAI /chat/completions API, DeepSeek (deepseek), Z.ai (zai), OpenRouter (openrouter), Mistral (mistral), Cohere (cohere, through its OpenAI-compatibility API) and xAI (xai), each running on OpenAICompatibleClient with its own OpenAICompatibleDialect.

Declared in this module's META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule, so adding the llm4s-openai-compatible dependency is all it takes for ProviderRegistry.default to resolve these ids. Where discovery cannot run - a fat jar whose services files were overwritten rather than merged - register it explicitly:

given ProviderRegistry = ProviderRegistry.ofModules(new Llm4sOpenAICompatibleModule)

A class rather than an object because java.util.ServiceLoader instantiates the class named in the services file through its public no-arg constructor.

Attributes

Supertypes: trait Llm4sProviderModule

class Object

trait Matchable

class Any

@Stable

Registers llm4s-openai's providers: the three that share OpenAIClient - OpenAI (id openai), Azure OpenAI (azure) and Requesty (requesty) - and the OpenAI embedding provider (openai).

OpenRouter, DeepSeek and Z.ai speak the same wire format without an SDK, so they are llm4s-openai-compatible's, on its shared OpenAICompatibleClient (#1132). This module depends on that one for OpenAIConfig.

Declared in this module's META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule, so adding the llm4s-openai dependency is all it takes for ProviderRegistry.default to resolve provider = "openai", "azure" and "requesty", and EMBEDDING_MODEL=openai/<model>. Where discovery cannot run - a fat jar whose services files were overwritten rather than merged - register it explicitly:

given ProviderRegistry = ProviderRegistry.ofModules(new Llm4sOpenAIModule)

A class rather than an object because java.util.ServiceLoader instantiates the class named in the services file through its public no-arg constructor.

Attributes

Supertypes: trait Llm4sProviderModule

class Object

trait Matchable

class Any

Registers llm4s-perplexity's provider: the Perplexity Agent API chat client, under the id perplexity.

Declared in this module's META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule, so adding the llm4s-perplexity dependency is all it takes for ProviderRegistry.default to resolve provider = "perplexity". Where discovery cannot run - a fat jar whose services files were overwritten rather than merged - register it explicitly:

given ProviderRegistry = ProviderRegistry.ofModules(new Llm4sPerplexityModule)

A class rather than an object because java.util.ServiceLoader instantiates the class named in the services file through its public no-arg constructor.

Attributes

Supertypes: trait Llm4sProviderModule

class Object

trait Matchable

class Any

Registers llm4s-voyage's one provider: the Voyage AI embedding provider, under the id voyage (alias voyageai). Voyage supplies embeddings and no chat client, so this module overrides only embeddingProviders.

Declared in this module's META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule, so adding the llm4s-voyage dependency is all it takes for ProviderRegistry.default to resolve EMBEDDING_MODEL=voyage/<model>. Where discovery cannot run - a fat jar whose services files were overwritten rather than merged - register it explicitly:

given ProviderRegistry = ProviderRegistry.ofModules(new Llm4sVoyageModule)

A class rather than an object because java.util.ServiceLoader instantiates the class named in the services file through its public no-arg constructor.

Attributes

Supertypes: trait Llm4sProviderModule

class Object

trait Matchable

class Any

Registers llm4s-watsonx's provider: the IBM watsonx.ai chat client, under the id watsonx.

'''Beta - never run against the live service.''' The client calls the watsonx.ai chat API (/ml/v1/text/chat and /ml/v1/text/chat_stream), with tool calling; its request and response shapes are cross-checked against IBM's own open-source clients but have not been checked against a real account (https://github.com/llm4s/llm4s/issues/1314). The module's API is not frozen.

Declared in this module's META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule, so adding the llm4s-watsonx dependency is all it takes for ProviderRegistry.default to resolve provider = "watsonx". Where discovery cannot run - a fat jar whose services files were overwritten rather than merged - register it explicitly:

given ProviderRegistry = ProviderRegistry.ofModules(new Llm4sWatsonXModule)

A class rather than an object because java.util.ServiceLoader instantiates the class named in the services file through its public no-arg constructor.

Attributes

Supertypes: trait Llm4sProviderModule

class Object

trait Matchable

class Any

@Stable

Helper trait for recording metrics consistently across all provider clients.

Extracts the common pattern of timing requests, observing outcomes, recording tokens, and reading costs from completion results.

Attributes

Supertypes: class Object

trait Matchable

class Any
Known subtypes: trait BaseLifecycleLLMClient

class AnthropicClient

class BedrockClient

class GeminiClient

class OllamaClient

class OpenAIClient

class OpenAICompatibleClient

class CohereClient

class DeepSeekClient

class MistralClient

class OpenRouterClient

class XaiClient

class ZaiClient

class PerplexityClient

class VertexAIClient

class WatsonXClient
Show all

@Experimental

Mistral AI client, over Mistral's OpenAI-format /v1/chat/completions API.

An OpenAICompatibleClient with MistralDialect, so it has everything the shared client has: completion, streaming (including streamed tool calls and usage), tool calling, response_format (JSON object and JSON schema) and one exchange recorded per call. Until #1132 it was a separate non-streaming client whose streamComplete returned a Left (#925) and which refused tool messages.

Requests go to <baseUrl>/v1/chat/completions, where baseUrl is the API root (https://api.mistral.ai by default); see org.llm4s.llmconnect.config.MistralConfig.apiBaseUrl.

Value parameters

config: Mistral configuration: API key, model, base URL and context settings.
exchangeLogging: where raw request/response exchanges are recorded, if anywhere.
metrics: receives per-call latency and token-usage events.

Attributes

Companion: object
Supertypes: class OpenAICompatibleClient

trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

@Experimental

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: MistralClient.type

@Experimental

Registration for the Mistral AI API.

Streaming is supported - the default org.llm4s.llmconnect.spi.ProviderFeatures - since Mistral moved onto the shared OpenAI-compatible client (#925, #1132).

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: MistralProvider.type

@Stable

LLMClient implementation for locally-hosted Ollama models.

Connects to an Ollama server via its HTTP chat API (/api/chat). All Ollama-specific protocol details (JSON-lines streaming, token-count field names) are handled internally.

== Tool calling ==

Tools in CompletionOptions.tools are sent as the tools field ({type: "function", function: {name, description, parameters}}). A reply's message.tool_calls - whole, or streamed - becomes ToolCalls. Ollama's native API sends no call ids, so the client synthesizes them (call_<12 hex>_<index>, a fresh prefix per reply or stream); an id the server does send is kept. An assistant turn's tool calls are sent back as Ollama's native history records them: each with its id, its position in the turn as function.index (Ollama reads an omitted index as 0, so parallel calls would collide) and object arguments. A ToolMessage is sent as role: tool with the tool_call_id it answers and, when that call is in the conversation, its tool_name. A malformed tool_calls entry is a ProcessingError. A model without the tools capability makes Ollama answer HTTP 400 (... does not support tools); that is a org.llm4s.error.ValidationError on tools naming the model, and the request is not retried without its tools. Whether the model calls tools at all depends on the model.

== Thinking ==

A reply's message.thinking becomes the returned message's AssistantMessage.thinking (and so Completion.thinking); streamed, each line's thinking is a StreamedChunk.thinkingDelta and the accumulated text is the message's thinking. An assistant message's thinking text is sent back as its thinking, as Ollama's tool-calling guide asks: a thinking model's follow-up request after a tool call carries its earlier reasoning with the content and tool calls (#1381).

== Structured output ==

CompletionOptions.responseFormat is honoured through the top-level format field of /api/chat, on both the streaming and non-streaming paths:

None sends no format key.
ResponseFormat.Json sends "format": "json" (JSON mode).
ResponseFormat.JsonSchema sends "format": <schema object> (structured outputs, which requires Ollama 0.5 or later; older servers reject or ignore the object). The name and strict parameters have no Ollama equivalent and are ignored.

== Streaming ==

Token counts (prompt_eval_count, eval_count) are only present in the final JSON-lines chunk (done: true). The accumulator updates its count at that point; chunks before the final one report zero tokens.

== Timeouts ==

Non-streaming requests time out after 120 seconds; streaming requests after 600 seconds. A section's timeouts { request = ..., stream = ... } block overrides either (OllamaConfig.timeouts).

Value parameters

config: Ollama configuration containing the model name and base URL.
metrics: Receives per-call latency and token-usage events. Defaults to MetricsCollector.noop.

Attributes

Companion: object
Supertypes: trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: OllamaClient.type

@Stable

Embedding provider implementation for Ollama, a local model inference server.

Generates text embeddings by calling the Ollama /api/embeddings HTTP endpoint. Each input text is embedded individually (one HTTP request per text) because the Ollama embedding API accepts a single prompt per call. Results are collected and returned as an org.llm4s.llmconnect.model.EmbeddingResponse.

No API key is required when Ollama runs locally, though one can be supplied for remote or authenticated deployments.

Ollama's entry in the embedding half of the provider SPI.

It shares the ollama id with OllamaProvider's chat client - the two namespaces are separate - so EMBEDDING_MODEL=ollama/nomic-embed-text and a named chat section with provider = "ollama" name the same provider.

Attributes

Supertypes: trait EmbeddingProviderDescriptor

class Object

trait Matchable

class Any
Self type: OllamaEmbeddingProvider.type

@Stable

Registration for a locally-running Ollama instance.

A section's baseUrl defaults to http://localhost:11434, where ollama serve listens and where Ollama's own clients look, so a local section needs only provider and model (#1991). For another host, set baseUrl, or bind the conventional variable in the section: baseUrl = ${?OLLAMA_BASE_URL}. It needs no API key — access is controlled at the network level.

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: OllamaProvider.type

@Stable

LLMClient implementation for OpenAI, Azure OpenAI and Requesty, built on OpenAI's official Java SDK (com.openai:openai-java).

Handles message conversion between llm4s format and the chat-completions format, completion requests, streaming responses, and tool calling (function calling). OpenAI and Requesty are reached through an org.llm4s.llmconnect.config.OpenAIConfig; Azure OpenAI through an org.llm4s.llmconnect.config.AzureConfig, using the SDK's Azure support (an api-key header, the deployment in the path and an api-version query parameter).

Until #1132 this client ran on Microsoft's com.azure:azure-ai-openai SDK, which Microsoft has deprecated in favour of openai-java.

== Reasoning ==

CompletionOptions.reasoning is sent as reasoning_effort (Low, Medium and High as low, medium and high; None sends nothing, leaving the model's default) to OpenAI reasoning models - the o-series and the gpt-5 family, as the model registry flags them - and never to other models, which reject it. On Azure a deployment name the registry cannot resolve gets it whenever it is asked for. Reasoning models get max_completion_tokens rather than max_tokens, and no temperature, top_p or penalties. The reasoning tokens a response reports (completion_tokens_details.reasoning_tokens) become TokenUsage.thinkingTokens, streamed or not; they are part of completionTokens, not added to it. See OpenAIReasoning.

== Streamed usage ==

Streaming requests set stream_options.include_usage, so the service reports token usage on a final chunk with no choices, and a streamed Completion carries usage and estimatedCost (Azure from api-version 2024-09-01-preview on).

For Anthropic Claude models with extended thinking, use AnthropicClient (in llm4s-anthropic), which has full support for the thinking parameter with budget_tokens.

Value parameters

config: provider configuration containing context window and reserve completion settings
metrics: metrics collector for observability (default: noop)
model: the model identifier (e.g., "gpt-4o"); for Azure, the deployment name
provider: the provider this client serves - openai, azure or requesty - which labels its metrics, exchange log and errors
transport: the SDK calls this client makes

Attributes

Companion: object
Supertypes: trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

Factory methods for creating OpenAIClient instances.

Provides safe construction of OpenAI clients with error handling via Result type.

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: OpenAIClient.type

@Stable

An org.llm4s.llmconnect.LLMClient for any endpoint speaking the OpenAI /chat/completions API, with no SDK.

This is the one implementation behind the generic openai-compatible provider and behind DeepSeekClient, ZaiClient and OpenRouterClient, which used to be three near-identical copies of it (#1132). What differs between providers is an OpenAICompatibleDialect; everything else is here:

complete: POST <baseUrl>/chat/completions, Bearer auth when there is a key, HttpErrorMapper for non-2xx replies, cost estimation, and a two-minute timeout (OpenAICompatibleClient.RequestTimeout);
streamComplete: the same with "stream": true, read as server-sent events until [DONE], each delta fanned out to one chunk per tool call, with a five-minute timeout (OpenAICompatibleClient.StreamTimeout). A stream that ends without a finish_reason or [DONE], or carries an error event, is a Left: what was delivered is not a finished reply;
request building: role mapping, sampling parameters, tools and response_format;
one provider exchange recorded per call, success or failure; and the response body closed on every path, success or failure.

Value parameters

dialect: how this provider departs from the standard format.
exchangeLogging: where raw request/response exchanges are recorded, if anywhere.
metrics: receives per-call latency and token-usage events.
settings: where to send requests and what to report.
suppliedHttpClient: the HTTP client to send requests through, which the caller keeps owning (#2341); by default one built from settings.network, closed with this client.

Attributes

Companion: object
Supertypes: trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all
Known subtypes: class CohereClient

class DeepSeekClient

class MistralClient

class OpenRouterClient

class XaiClient

class ZaiClient
Show all

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: OpenAICompatibleClient.type

@Stable

The places where an OpenAI-compatible provider departs from the plain /chat/completions wire format.

OpenAICompatibleClient holds everything the providers share - request building, the HTTP round trip, SSE streaming, tool-call fan-out, error mapping, exchange logging - written once. A dialect answers the handful of questions on which they differ, and every answer defaults to the standard format, so a provider overrides only where it departs from it:

'''Request:''' extra headers; the role system messages go under (systemRole); how message text is encoded (encodeContent); which parts of a user message with media are sent, and how (encodeUserPart); whether an assistant turn with no text still sends content (alwaysSendAssistantContent) or is sent at all (sendEmptyAssistantTurns); the tool-call ids the provider accepts (encodeToolCallId); the response_format shape (encodeResponseFormat); whether a stream asks for usage (streamUsageOption); any reasoning fields (addReasoning); and whether, and where, an assistant turn's earlier thinking goes back (encodeThinking).
'''Response:''' how content is read back (decodeContent); where the model's thinking is (thinking), and any provider-specific replay data beside it (thinkingDetails, decodeThinkingDetails); where its reasoning-token count is (reasoningTokens); and how a non-streaming tool_calls array is parsed (parseToolCalls).

The generic openai-compatible provider uses OpenAICompatibleDialect.standard, which overrides only headers and streamed usage, both from config. DeepSeek, Z.ai, OpenRouter, Mistral, Cohere and xAI each override two to six members. A new OpenAI-compatible provider whose differences fit these members is a dialect and a descriptor, not a client; one whose differences do not should extend this trait rather than fork OpenAICompatibleClient (#1132).

Attributes

Companion: object
Supertypes: class Object

trait Matchable

class Any

Attributes

Companion: trait
Supertypes: class Object

trait Matchable

class Any
Self type: OpenAICompatibleDialect.type

@Stable

Registration for the generic openai-compatible provider: any endpoint speaking the OpenAI /chat/completions API, with no module and no code.

A named provider section with provider = "openai-compatible" needs a baseUrl and a model; apiKey and headers are optional (local servers need no key). It also accepts four provider-specific keys: contextWindow and reserveCompletion (whole numbers; the model's window cannot be known from an arbitrary endpoint, so these default to org.llm4s.llmconnect.config.OpenAICompatibleConfig.DEFAULT_CONTEXT_WINDOW and a quarter of the window up to org.llm4s.llmconnect.config.OpenAICompatibleConfig.DEFAULT_RESERVE_COMPLETION), registryProvider, and streamUsage (default true), which controls whether a streaming request asks for token usage with stream_options.include_usage; set it to false for a server that rejects that field.

'''Where the context window comes from.''' In this order: the section's contextWindow; else the model registry's entry for model under the registryProvider the section names (groq, together_ai, fireworks_ai, xai, perplexity, ...); else, when the section names none, the entry under the provider inferred from the baseUrl host; else the default. An explicit registryProvider is an instruction, so it switches the inference off. The host is inferred only by an exact match on the URL's host (api.groq.com, api.together.xyz, api.together.ai, api.fireworks.ai, api.x.ai, api.perplexity.ai), never by a substring, so a look-alike host such as api.groq.com.evil.example gets no registry window. Only the window is taken from the registry: reserveCompletion keeps its own rule (a quarter of the window, up to 2048) because a registry entry's output limit can be as large as the whole window. The lookup is strict - only the named provider's own entry for exactly that model id counts, never a substring match or another provider's entry for the same name - and a model the registry does not know, or lists with no input limit, gets the default. A registry window below the default is ignored too: many entries (most of Fireworks' older and newer models, Perplexity's retired ones) carry a 4096 placeholder for input, output and total alike, which would shrink an 80k-context model's prompt budget below the default and reject a reserveCompletion that fits it. The registry can therefore only enlarge the window, never shrink it; a model whose window really is smaller sets contextWindow. Several sections can use it side by side:

llm4s.providers {
 databricks {
   provider = "openai-compatible"
   baseUrl  = "https://ws.cloud.databricks.com/serving-endpoints"
   model    = "databricks-meta-llama-3-3-70b-instruct"
   auth {   // workload identity: no apiKey
     identityTokenFile = "/var/run/secrets/spiffe/svid.jwt"
     tokenUrl          = "https://ws.cloud.databricks.com/oidc/v1/token"
     clientId          = "<service principal UUID>"
     scope             = "all-apis"
   }
 }
 groq-main {
   provider = "openai-compatible"
   baseUrl  = "https://api.groq.com/openai/v1"
   model    = "openai/gpt-oss-120b"
   apiKey   = ${?GROQ_API_KEY}
   contextWindow = 131072
 }
 local-vllm {
   provider = "openai-compatible"
   baseUrl  = "http://localhost:8000/v1"
   model    = "Qwen/Qwen2.5-7B-Instruct"
 }
}

The client is OpenAICompatibleClient with the standard dialect: no reasoning parameters and no provider-specific decoding. A provider that needs those gets its own OpenAICompatibleDialect and descriptor in this module.

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: OpenAICompatibleProvider.type

@Stable

OpenAI embedding provider implementation.

Provides text embeddings using OpenAI's embedding API (text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002). Supports batch embedding of multiple texts in a single request.

== Supported Models ==

text-embedding-3-small - Efficient, lower cost (recommended)
text-embedding-3-large - Higher quality, higher cost
text-embedding-ada-002 - Legacy model

== Token Usage == The response includes token usage information when available from the API.

OpenAI's entry in the embedding half of the provider SPI.

The object that builds the provider is also its org.llm4s.llmconnect.spi.EmbeddingProviderDescriptor, so there is one place per provider rather than a descriptor shadowing a factory.

Attributes

See also: EmbeddingProvider for the provider interface

org.llm4s.llmconnect.config.EmbeddingProviderConfig for configuration
Supertypes: trait EmbeddingProviderDescriptor

class Object

trait Matchable

class Any
Self type: OpenAIEmbeddingProvider.type

@Stable

Registration for the OpenAI API.

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: OpenAIProvider.type

@Stable

org.llm4s.llmconnect.LLMClient implementation for the OpenRouter unified model gateway.

Sends requests to the OpenRouter REST API using the OpenAI-compatible /chat/completions endpoint. Accepts OpenAIConfig — there is no separate OpenRouterConfig; LLMConnect detects OpenRouter by checking whether baseUrl contains "openrouter.ai" and routes accordingly.

An OpenAICompatibleClient with OpenRouterDialect.

== Required headers ==

OpenRouter's usage policy requires two additional headers on every request. This client sends them automatically:

HTTP-Referer: https://github.com/llm4s/llm4s
X-Title: LLM4S

== Reasoning / extended thinking ==

Model type is detected by substring matching on the lower-cased model name:

Names containing "claude" or "anthropic" → Anthropic-style thinking object (type: "enabled", budget_tokens).
Names containing "o1", "o3", or "o4" → OpenAI-style reasoning_effort string parameter.
All other models → reasoning configuration is silently omitted.

The thinking budget is clamped to [1024, maxTokens - 1] for Anthropic models, matching the Anthropic API constraint.

== Thinking content ==

Extended thinking text is extracted from whichever field the model populates: message.thinking, message.reasoning, or choice.thinking (checked in that order).

When the underlying model returns signed, summarised or encrypted reasoning (Claude, Gemini, OpenAI reasoning models), OpenRouter also returns it as message.reasoning_details, and requires the whole sequence back unchanged when a conversation continues after tool calls. Each item is kept as a sealed org.llm4s.llmconnect.model.ThinkingBlock.Opaque block beside the reasoning text, bound to the request it answered, and sent back as reasoning_details with the same order and fields while the conversation before it is unchanged; once it changes (pruned, compressed, edited) the details are dropped and only the reasoning text is sent.

Value parameters

config: OpenAIConfig whose baseUrl must contain "openrouter.ai"; carries the API key and model name. A config with workloadIdentity or a blank apiKey is refused with an IllegalArgumentException.
exchangeLogging: where raw request/response exchanges are recorded, if anywhere
metrics: Receives per-call latency and token-usage events. Defaults to MetricsCollector.noop.

Attributes

Companion: object
Supertypes: class OpenAICompatibleClient

trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: OpenRouterClient.type

@Stable

Registration for OpenRouter.

OpenRouter speaks the OpenAI wire format and so reuses OpenAIConfig, but it has its own client: OpenRouterClient adds the referer/title headers the service expects, reasoning parameters and its own tool-call parsing.

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: OpenRouterProvider.type

@Experimental

LLMClient for Perplexity's Agent API: POST <baseUrl>/v1/agent, Perplexity's canonical path for its OpenAI Responses-format endpoint (/v1/responses is its alias).

The request and response shapes are those of Perplexity's API reference ("Create Agent Response", https://docs.perplexity.ai/api-reference/agent-post) and its Agent API guides, read 2026-11-05. Nothing here has been run against the live service; the @Cloud suite PerplexitySmokeSpec in modules/it does that with a key.

== Request ==

The model goes in model, in the Agent API's provider/model form. System messages are joined, in order, into instructions; every other message becomes an input item: a user or assistant turn a message item, a tool call a function_call item (its arguments a JSON string) and a ToolMessage a function_call_output item carrying the call's call_id. Only text parts are sent; any other part is refused before anything is sent.

temperature is always sent, top_p when not 1.0 and max_output_tokens when set. The Agent API rejects an anthropic/ model without max_output_tokens, so such a request that sets no limit is sent the config's reserveCompletion. CompletionOptions.reasoning is sent as reasoning.effort (ReasoningEffort.None as minimal, the least the API takes; budgetTokens has no counterpart). The penalties have no counterpart in the Agent API and are not sent. A ResponseFormat.JsonSchema is sent as response_format ({"type":"json_schema","json_schema":{"name","schema","strict"}}); the Agent API takes no plain JSON mode, so ResponseFormat.Json is refused before anything is sent.

== Tools and web search ==

With webSearch on (the default) every request offers Perplexity's web_search tool, which Perplexity runs itself and bills per search; the model decides whether to search. CompletionOptions.tools are sent as function tools. A function_call item in the reply becomes a ToolCall whose id is the item's call_id. A model may attach a thought_signature to a call, which Perplexity says to send back with it: it is kept as a sealed ThinkingBlock.Opaque block of provider perplexity and replayed on the call it came with, only to the same provider and model (see ThinkingReplay.bindOrigin).

== Reply ==

The answer is the text of the message items' output_text parts. Completion.citations holds the url_citation annotations on that text, then each result of the search_results items (URL, title, and the snippet as citedText) whose URL is not already there: the answer's [web:N] markers point at those results. A response whose status is failed, or anything but completed or incomplete, is an error; incomplete (a limit reached) is returned as an answer cut short, with finish reason length on a stream. The usage's input_tokens_details and output_tokens_details.reasoning_tokens fill the cached and thinking token counts, and the cost Perplexity reports (usage.cost.total_cost, in USD) is the completion's estimatedCost. The Agent API returns no reasoning text, so a completion has no thinking text.

== Streaming ==

stream: true returns server-sent events, each a JSON object discriminated by type. Each response.output_text.delta is passed on as a content chunk; the stream's response.completed event carries the whole response, which gives the tool calls (sent as whole calls after the text), citations and usage. A response.failed event, or a stream that ends without response.completed, is an error.

Value parameters

config: model, key, endpoint and web-search setting.
exchangeLogging: optional provider exchange logging.
httpClient: the HTTP client; by default one built from the config's network settings.
metrics: receives per-call latency and token-usage events.

Attributes

Companion: object
Supertypes: trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

@Experimental

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: PerplexityClient.type

Registration for Perplexity's Agent API: provider = "perplexity".

The Agent API (POST https://api.perplexity.ai/v1/agent) replaced Sonar Chat Completions, whose support Perplexity ended on 27 September 2026. It speaks the OpenAI Responses format, so it is a client of its own rather than an openai-compatible dialect. A section names an Agent API model in its provider/model form and takes the key as apiKey, or from llm4s.credentials.perplexity.apiKey, which PERPLEXITY_API_KEY binds:

perplexity-main {
 provider = "perplexity"
 model    = "perplexity/sonar"
}

Extras: webSearch (default true) offers the model Perplexity's web_search tool on every request; contextWindow and reserveCompletion override the window, which otherwise comes from the model registry entry of the model's own vendor (anthropic/claude-sonnet-5-5 looks up anthropic's claude-sonnet-5-5), else org.llm4s.llmconnect.config.PerplexityConfig.DEFAULT_CONTEXT_WINDOW.

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: PerplexityProvider.type

@Stable

Records completed provider exchanges to the configured logging sink.

Attributes

Supertypes: class Object

trait Matchable

class Any
Self type: ProviderExchangeRecorder.type

@Stable

Registration for Requesty.

Requesty is an OpenAI-compatible router: it reuses both OpenAIConfig and OpenAIClient, and differs only in its default base URL.

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: RequestyProvider.type

@Stable

Provides Google Cloud OAuth2 access tokens for Vertex AI via ADC.

Token resolution order:

GOOGLE_ACCESS_TOKEN environment variable — direct token; useful for CI and testing.
JSON credential file resolved from credentialFilePath, then GOOGLE_APPLICATION_CREDENTIALS, then gcloud's well-known application_default_credentials.json, found as Google's client libraries find it: in $CLOUDSDK_CONFIG when that is set (any OS), else %APPDATA%\gcloud on Windows, else ~/.config/gcloud. APPDATA and CLOUDSDK_CONFIG are resolved at the config edge (org.llm4s.config.VertexAIAdcConfigLoader) and arrive as wellKnownCredentialPath, not through envReader. Supports both authorized_user (refresh-token flow) and service_account (RS256 JWT assertion flow, no extra dependencies).
GCE / GKE metadata server — Workload Identity path for production deployments.

Tokens are cached until one minute before expiry to minimise round-trips.

== Config-boundary exemption ==

This class reads GOOGLE_ACCESS_TOKEN and GOOGLE_APPLICATION_CREDENTIALS directly (hence the scalafix:off NoSystemGetenv at the top of the file), rather than receiving them through org.llm4s.config.Llm4sConfig like other settings. This is deliberate: Application Default Credentials is a runtime discovery protocol whose inputs (a possibly-rotated access token, the ambient credentials path, and the GCE/GKE metadata server) must be resolved lazily at token-fetch time, not frozen into static config at startup. The reads are funnelled through the injectable envReader so the class stays fully testable without touching the real environment.

Value parameters

credentialFilePath: Optional explicit path to a Google JSON credential file.
envReader: Environment variable reader; injectable for testing.
fileReader: File reader; injectable for testing.
httpClient: HTTP client for token-endpoint calls.
wellKnownCredentialPath: Location of gcloud's well-known credential file, as resolved by the config loader; None skips that lookup.

Attributes

Companion: object
Supertypes: class Object

trait Matchable

class Any

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: VertexAIAuthProvider.type

@Stable

LLMClient implementation for Google Cloud Vertex AI.

Calls the Vertex AI REST API (aiplatform.googleapis.com) using the same Gemini-compatible JSON format as GeminiClient, but with:

A region-scoped endpoint: https://{location}-aiplatform.googleapis.com/v1/projects/{project}/...
OAuth2 bearer-token auth via VertexAIAuthProvider (ADC) instead of an API-key query param.

== Authentication ==

Tokens are obtained and cached by VertexAIAuthProvider, which supports authorized_user refresh-token credentials (from gcloud auth application-default login), service_account JWT credentials, and the GCE/GKE metadata server (Workload Identity). Set GOOGLE_ACCESS_TOKEN as an escape hatch for testing.

== Message format ==

Identical to GeminiClient:

Roles are "user" and "model".
System messages go into systemInstruction.
Tool results are sent as functionResponse parts keyed by function name.
Synthetic UUIDs are generated for tool-call IDs (Vertex AI does not return them).

Value parameters

config: VertexAIConfig with project, location, model, and credential path.
exchangeLogging: Controls whether raw request/response bodies are recorded.
httpClient: HTTP client (injectable for testing).
metrics: Receives per-call latency and token-usage events.

Attributes

Companion: object
Supertypes: trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: VertexAIClient.type

@Stable

Registration for Google Cloud Vertex AI.

A section names the GCP project (required) and location (default us-central1), both provider-specific keys, and may set apiKey to a path to a service-account credential file - Vertex authenticates with OAuth2, not an API key. With no apiKey it uses Application Default Credentials. It therefore has no llm4s.credentials.vertexai block: there is no vendor API key to share.

vertex-main {
 provider = "vertexai"
 model    = "gemini-2.0-flash"
 project  = "my-gcp-project"
 location = "europe-west4"
}

Until #1215 the project was read from endpoint and the location from organization. Both still work as deprecated aliases, with a warning, and will be removed in a later release.

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: VertexAIProvider.type

Embedding provider implementation for the Voyage AI embedding API.

Generates text embeddings by posting batched input to the Voyage AI <baseUrl>/embeddings endpoint (the default base URL is https://api.voyageai.com/v1). Unlike Ollama, Voyage accepts multiple inputs in a single request, so all texts are sent in one HTTP call.

Requires a valid Voyage AI API key in the provider configuration.

Voyage's entry in the embedding half of the provider SPI.

Voyage is the case that shapes the SPI: it supplies embeddings and no chat client at all, which is why org.llm4s.llmconnect.spi.EmbeddingProviderDescriptor is a separate trait rather than a method on the chat descriptor.

It lives in llm4s-voyage and is registered by Llm4sVoyageModule; it was built into llm4s-core until #1132, package unchanged.

Attributes

See also: EmbeddingProvider for the common embedding interface
Supertypes: trait EmbeddingProviderDescriptor

class Object

trait Matchable

class Any
Self type: VoyageAIEmbeddingProvider.type

LLMClient for IBM watsonx.ai chat (POST /ml/v1/text/chat and /ml/v1/text/chat_stream).

'''Beta - never run against the live service.''' There is no watsonx account behind this project, so nothing here has been checked against the real service (see https://github.com/llm4s/llm4s/issues/1314). The request and response shapes are those IBM's public pages report, cross-checked against IBM's own open-source client code (see ''Evidence'' below). The module's API is not frozen. It replaces the text-generation endpoints (/ml/v1/text/generation and /generation_stream), which IBM's February 2026 release notes deprecate without giving a removal date.

== Evidence ==

A detail is ''SDK-evidenced'' when IBM's own code shows it. That proves IBM's client sends it, which is strong evidence the service accepts it, but it is not a live check. The sources, pinned:

NODE: https://github.com/IBM/watsonx-ai-node-sdk at 47a4a0c (2026-08-05), IBM's Node.js SDK;
LCIBM: https://github.com/langchain-ai/langchain-ibm at 6c32b1d (2026-10-05), IBM's LangChain integration, libs/ibm/langchain_ibm/chat_models.py;
PY: IBM's Python SDK, ibm-watsonx-ai 1.8.0 on PyPI;
JAVA: https://github.com/IBM/watsonx-ai-java-sdk at 02246d5 (2026-10-22), IBM's Java SDK, whose test resources include replies recorded from the service (modules/watsonx-ai/src/test/...);
DOCS: IBM's documentation page "Building agent-driven workflows with the chat API" (https://www.ibm.com/docs/SSYOK8/wsj/analyze-data/fm-api-chat-tools.html, read 2026-10-30), whose curl examples post to /ml/v1/text/chat?version=... with tool_choice_option: "auto", and whose quoted reply has finish_reason: "tool_calls" and a call {"id","type":"function","function":{"name","arguments"}} with arguments a JSON string. WatsonXDocumentedShapesSpec pins those shapes.

The @Cloud suite WatsonXAssumptionProbeSpec (modules/it) re-checks each of these against a real account and prints which held.

== Authentication ==

The IBM Cloud API key is exchanged at the IAM endpoint for a bearer token that lives an hour. The token is cached and refreshed lazily, five minutes before it expires.

== Request format ==

The conversation is sent as structured messages with roles (system, user, assistant, tool), so content is data and cannot forge a turn: there is no prompt string with role markers, and no stop sequences standing in for them. The model goes in model_id, the project or space in project_id or space_id, and the API version in the version query parameter (WatsonXConfig.apiVersion, by default the date PY sends: org.llm4s.llmconnect.config.WatsonXConfig.DEFAULT_API_VERSION). temperature is always sent, max_tokens when set (SDK-evidenced: NODE types/vml_v1.ts:790-797, a deprecated alias of max_completion_tokens), and top_p when not 1.0.

== Tools ==

CompletionOptions.tools are sent as tools (type: function, function: {name, description, parameters}; the OpenAI strict flag is dropped) with tool_choice_option: "auto". Tool calls in a reply (message.tool_calls, whole, or streamed as delta.tool_calls pieces merged by index) become ToolCalls. An assistant turn's calls go back as tool_calls, and a ToolMessage as a tool message with its tool_call_id. arguments is a JSON string on the wire (SDK-evidenced: NODE types/messages.ts:19-24) and a JSON object in a ToolCall. A call without an id gets a generated one, defensively: the SDK's type makes id required (NODE types/messages.ts:29-30), and a stream sends it on a call's first piece only (LCIBM chat_models.py:324). Whether the model calls tools, or supports them at all, depends on the model.

== Penalties and response format ==

presencePenalty and frequencyPenalty are sent as presence_penalty and frequency_penalty when not 0.0, their default, so a request that leaves them alone is unchanged (SDK-evidenced: NODE vml_v1.ts:2808,2815 for textChat and 3032,3039 for textChatStream; PY foundation_models/schema/_api.py:270,273; LCIBM chat_models.py:939,969,1664,1670). Their range is the service's to check.

responseFormat is sent as response_format (WatsonXClient.encodeResponseFormat): ResponseFormat.Json as {"type":"json_object"} (NODE types/vml_v1.ts:3283-3310; PY schema/_api.py:225; LCIBM chat_models.py:2147-2149), and ResponseFormat.JsonSchema as {"type":"json_schema","json_schema":{"name","schema","strict"}} (PY schema/_api.py:226,231-234,255-257; LCIBM chat_models.py:2160-2182). The Node SDK's type names only json_object but takes any string, so json_schema rests on the Python SDK and IBM's LangChain integration. IBM's SDK notes that JSON mode needs the prompt to ask for JSON as well (NODE types/vml_v1.ts:3284-3292); this client adds no instruction. Whether a model honours either format depends on the model.

== Reasoning ==

CompletionOptions.reasoning is sent as reasoning_effort (low, medium or high; SDK-evidenced: NODE types/vml_v1.ts:860-864 and vml_v1.ts:2828,3053 for textChat and textChatStream; PY schema/_api.py:289; LCIBM chat_models.py:1010), but only to the models known to take it (WatsonXClient.takesReasoningEffort): OpenAI's gpt-oss models, openai/gpt-oss-*, which IBM's clients send it to (JAVA ChatServiceThinkingTest; LCIBM chat_models.py:657-690). Any other model is sent nothing, and a warning is logged once per process: IBM's Model Gateway rejects the parameter for models without it (JAVA ModelGatewayChatServiceIT), and whether the chat API does is unchecked. The chat API has no value that turns reasoning off, so ReasoningEffort.None is sent as low, the least, with a warning logged once per process (WatsonXClient.reasoningEffortValue). include_reasoning is not sent: it defaults to true (NODE types/vml_v1.ts:858).

A reply's reasoning is read as thinking, from any model: message.reasoning_content on complete (NODE types/vml_v1.ts:3383-3384) and delta.reasoning_content on a stream, passed on as StreamedChunk.thinkingDelta (JAVA gpt_oss_thinking_response.txt and gpt_oss_thinking_streaming_response.txt, recorded replies of openai/gpt-oss-120b); reasoning, which IBM's LangChain integration also reads (LCIBM chat_models.py:150-153, 332-335), when reasoning_content is absent. Thinking is not sent back on a later turn: the chat API's assistant message has no field for it. Reasoning tokens are not reported: the chat API's usage has only prompt_tokens, completion_tokens and total_tokens (NODE types/vml_v1.ts:3496-3503, and the recorded replies above), so TokenUsage.thinkingTokens stays empty. budgetTokens has no counterpart and is ignored.

== Finish reasons and stream endings ==

A stream must end with a choice that carries a finish_reason. One that ends without it, or whose reason is in WatsonXClient.ErrorFinishReasons, is a Left(ServiceError) naming the reason; text received so far is not returned as a success. Every other reason (stop, length, tool_calls, unknown values) is a normal stop. complete applies the same rule to choices[0].finish_reason (a missing one is fine). The usage arrives in a final chunk that may have no choices (SDK-evidenced: LCIBM chat_models.py:379-383). Streamed tool calls are reported once, whole, after the last delta, never as fragments.

An event: error frame ends the stream with IBM's error: its data: line holds IBM's error JSON, whose status_code picks the LLMError as an HTTP status would (401/403 AuthenticationError, 429 RateLimitError, 400 ValidationError, otherwise ServiceError; none is ServiceError(502)) and whose first errors entry's code and message, redacted, are the message (SDK-evidenced: PY foundation_models/inference/fm_model_inference.py:683-689 raises on that frame).

== Timeouts ==

A chat call times out after two minutes and a streamed one after ten, unless the section's timeouts { request = ..., stream = ... } block sets either (WatsonXConfig.timeouts). The IAM token exchange keeps its own 30 seconds.

Value parameters

config: model, credentials, project or space and endpoints.
exchangeLogging: optional provider exchange logging.
httpClient: used for both the model calls and the IAM exchange.
metrics: receives per-call latency and token-usage events.

Attributes

Companion: object
Supertypes: trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: WatsonXClient.type

Registration for IBM watsonx.ai.

'''Beta - never run against the live service.''' The client calls the watsonx.ai chat API (/ml/v1/text/chat and /ml/v1/text/chat_stream), with tool calling; its request and response shapes are cross-checked against IBM's own open-source clients but have not been checked against a real account (https://github.com/llm4s/llm4s/issues/1314). The module's API is not frozen.

A section names the model and the projectId (or spaceId) inference runs in, and takes the IBM Cloud API key as apiKey - or from llm4s.credentials.watsonx.apiKey, which WATSONX_API_KEY binds. baseUrl selects the region and defaults to Dallas (us-south).

watsonx-main {
 provider  = "watsonx"
 model     = "ibm/granite-4-h-small"
 projectId = ${?WATSONX_PROJECT_ID}
}

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: WatsonXProvider.type

@Experimental

xAI (Grok) client, over xAI's OpenAI-compatible Chat Completions API (https://api.x.ai/v1/chat/completions).

An OpenAICompatibleClient with XaiDialect, so it has everything the shared client has: completion, streaming (including streamed tool calls and usage), tool calling, structured output and one exchange recorded per call. What the dialect adds is reasoning: a reply's reasoning_content is read as thinking, its reasoning_tokens as thinking tokens, and CompletionOptions.reasoning is sent as reasoning_effort to the models that take it.

xAI calls Chat Completions its legacy endpoint; new features ship on its Responses API, which this client does not speak.

Value parameters

config: xAI configuration: API key, model, base URL and context settings.
exchangeLogging: where raw request/response exchanges are recorded, if anywhere.
metrics: receives per-call latency and token-usage events.

Attributes

Companion: object
Supertypes: class OpenAICompatibleClient

trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

@Experimental

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: XaiClient.type

@Experimental

Registration for the xAI (Grok) API, through its OpenAI-compatible Chat Completions endpoint (#1023). The key defaults to XAI_API_KEY, bound to llm4s.credentials.xai.apiKey, and the base URL to org.llm4s.llmconnect.config.XaiConfig.DEFAULT_BASE_URL.

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: XaiProvider.type

@Stable

LLM client for the Z.ai API.

Z.ai uses an OpenAI-compatible /chat/completions endpoint with one important difference: message content is always an array of typed objects ([{"type":"text","text":"..."}]) rather than a plain string. This applies to user, system, assistant, and tool messages alike. Sending a plain string causes a rejection from the Z.ai API.

Both non-streaming (complete) and streaming (streamComplete) are supported. Tool calling follows the standard OpenAI function-calling format.

An OpenAICompatibleClient with ZaiDialect.

Value parameters

config: Z.ai connection configuration (API key, model, base URL, context window)
exchangeLogging: where raw request/response exchanges are recorded, if anywhere
metrics: records per-call latency and token-usage events; use org.llm4s.metrics.MetricsCollector.noop when metrics are not needed

Attributes

Companion: object
Supertypes: class OpenAICompatibleClient

trait BaseLifecycleLLMClient

trait MetricsRecording

trait LLMClient

trait AutoCloseable

class Object

trait Matchable

class Any
Show all

Attributes

Companion: class
Supertypes: class Object

trait Matchable

class Any
Self type: ZaiClient.type

@Stable

Registration for the Z.ai GLM API.

Attributes

Supertypes: trait ProviderDescriptor

class Object

trait Matchable

class Any
Self type: ZaiProvider.type

org.llm4s.llmconnect.provider

Members list

Type members

Classlikes

Value parameters

Attributes

Attributes

Attributes

Attributes

Value parameters

Attributes

Attributes

Attributes

Value parameters

Attributes

Attributes

Attributes

Value parameters

Attributes

Attributes

Attributes

Attributes

Value parameters

Attributes

Attributes

Attributes

Attributes

Value parameters

Attributes

Attributes

Attributes

Attributes

Attributes

Value parameters

Attributes

Attributes

Attributes

Attributes

Attributes

Attributes

Attributes

Attributes

Attributes

Attributes

Attributes

Attributes

Attributes

Attributes

Value parameters

Attributes

Attributes

Attributes

Value parameters

Attributes

Attributes

Attributes

Attributes

Value parameters

Attributes

Attributes

Value parameters

Attributes

Attributes

Attributes

Attributes

Attributes

Attributes

Attributes

Value parameters

Attributes

Attributes

Attributes

Value parameters

Attributes

Attributes

Attributes

Attributes

Attributes

Value parameters

Attributes