LLMClient implementation for locally-hosted Ollama models.
Connects to an Ollama server via its HTTP chat API (/api/chat). All Ollama-specific protocol details (JSON-lines streaming, token-count field names) are handled internally.
== Tool calling ==
Tools in CompletionOptions.tools are sent as the tools field ({type: "function", function: {name, description, parameters}}). A reply's message.tool_calls - whole, or streamed - becomes ToolCalls. Ollama's native API sends no call ids, so the client synthesizes them (call_<12 hex>_<index>, a fresh prefix per reply or stream); an id the server does send is kept. An assistant turn's tool calls are sent back as Ollama's native history records them: each with its id, its position in the turn as function.index (Ollama reads an omitted index as 0, so parallel calls would collide) and object arguments. A ToolMessage is sent as role: tool with the tool_call_id it answers and, when that call is in the conversation, its tool_name. A malformed tool_calls entry is a ProcessingError. A model without the tools capability makes Ollama answer HTTP 400 (... does not support tools); that is a org.llm4s.error.ValidationError on tools naming the model, and the request is not retried without its tools. Whether the model calls tools at all depends on the model.
== Thinking ==
A reply's message.thinking becomes the returned message's AssistantMessage.thinking (and so Completion.thinking); streamed, each line's thinking is a StreamedChunk.thinkingDelta and the accumulated text is the message's thinking. An assistant message's thinking text is sent back as its thinking, as Ollama's tool-calling guide asks: a thinking model's follow-up request after a tool call carries its earlier reasoning with the content and tool calls (#1381).
== Structured output ==
CompletionOptions.responseFormat is honoured through the top-level format field of /api/chat, on both the streaming and non-streaming paths:
ResponseFormat.JsonSchema sends "format": <schema object> (structured outputs, which requires Ollama 0.5 or later; older servers reject or ignore the object). The name and strict parameters have no Ollama equivalent and are ignored.
== Streaming ==
Token counts (prompt_eval_count, eval_count) are only present in the final JSON-lines chunk (done: true). The accumulator updates its count at that point; chunks before the final one report zero tokens.
== Timeouts ==
Non-streaming requests time out after 120 seconds; streaming requests after 600 seconds. A section's timeouts { request = ..., stream = ... } block overrides either (OllamaConfig.timeouts).
Value parameters
config
Ollama configuration containing the model name and base URL.
metrics
Receives per-call latency and token-usage events. Defaults to MetricsCollector.noop.
Executes a blocking completion request and returns the full response.
Executes a blocking completion request and returns the full response.
Sends the conversation to the LLM and waits for the complete response. Use when you need the entire response at once or when streaming is not required.
Value parameters
conversation
conversation history including system, user, assistant, and tool messages
options
configuration including temperature, max tokens, tools, etc. (default: CompletionOptions())
Attributes
Returns
Right(Completion) with the model's response, or Left(LLMError) on failure
Returns the maximum context window size supported by this model in tokens.
Returns the maximum context window size supported by this model in tokens.
The context window is the total tokens (prompt + completion) the model can process in a single request, including all conversation messages and the generated response.
Attributes
Returns
total context window size in tokens (e.g., 4096, 8192, 128000)
Returns the number of tokens reserved for the model's completion response.
Returns the number of tokens reserved for the model's completion response.
This value is subtracted from the context window when calculating available tokens for prompts. Corresponds to the max_tokens or completion token limit configured for the model.
Executes a streaming completion request, invoking a callback for each chunk as it arrives.
Executes a streaming completion request, invoking a callback for each chunk as it arrives.
Streams the response incrementally, calling onChunk for each token/chunk received. Enables real-time display of responses. Returns the final accumulated completion on success.
Value parameters
conversation
conversation history including system, user, assistant, and tool messages
onChunk
callback invoked for each chunk; called synchronously, avoid blocking operations
options
configuration including temperature, max tokens, tools, etc. (default: CompletionOptions())
Attributes
Returns
Right(Completion) with the complete accumulated response, or Left(LLMError) on failure
Releases resources and closes connections to the LLM provider.
Releases resources and closes connections to the LLM provider.
Call when the client is no longer needed. After calling close(), the client should not be used. Default implementation is a no-op; override if managing resources like connections or thread pools.
Sends the conversation and parses the response into a typed value using the provided schema.
Sends the conversation and parses the response into a typed value using the provided schema.
Sets ResponseFormat.JsonSchema on the options. OpenAI, Azure OpenAI and Gemini enforce the schema at generation time, as does Ollama 0.5 or later through its format field. Requesty and the OpenAI-compatible providers (including Cohere) send it as response_format, but whether it is enforced is up to the server - for Requesty, a router, the backend model it routes to: one that ignores the field returns unconstrained text. Anthropic falls back to a best-effort system-prompt instruction, which is not schema-enforced. Clients that do not read responseFormat (watsonx, Bedrock) send no schema at all. Because models may wrap JSON in markdown code fences or surround it with prose, the response is normalised (fence stripped, first balanced {...} or [...] extracted) before being deserialised with uPickle into the expected type A. A is read from that text, so a Long field receives an integer above 2^53 exactly; a reader that goes through ujson.Value (such as a bridge to another JSON library) sees it rounded to the nearest double.
The reply is '''not''' validated against the schema: it is only deserialised. A constraint the reader does not check - an enum, a numeric or string bound, additionalProperties = false (extra keys are ignored) - can be violated by a reply that still returns Right(A). Check such constraints on the result yourself.
The schema is derived with strict = true, which lists '''every''' property as required, including a property declared optional with required = false. Only responseFormat is overridden: every other option you pass is forwarded unchanged to complete, where the provider client may adjust or drop options the model does not support, as for any other complete call. name and strict on the format are left at their defaults ("response" and true); call complete with your own ResponseFormat.JsonSchema to set them.
Type parameters
A
target type; must have a corresponding upickle.default.Reader[A]
implicit uPickle reader for deserialising the JSON into A
schema
JSON-Schema description of the expected response object
Attributes
Returns
Right(A) on success. Left(ValidationError) with field structured_output when the reply is not JSON, is JSON null, or cannot be deserialised as A; any other Left is the provider call's own error, returned unchanged
Validates that the client is open, executes the operation, and records standard completion metrics (latency, token usage, estimated cost).
Validates that the client is open, executes the operation, and records standard completion metrics (latency, token usage, estimated cost).
Use this in complete and streamComplete implementations to avoid repeating the lifecycle-check + metrics-wrapping boilerplate.
An interrupted call - one that throws InterruptedException, or fails while the thread is interrupted - is returned as Left(CancelledError) with the interrupt flag kept, whatever the provider SDK did with it. A call made with the flag already set returns Left(CancelledError) at once, without running operation: an SDK that ignores the flag would otherwise send the (billed) request anyway.
Value parameters
operation
The provider-specific completion logic to execute. Called only when the client is open and the thread is not interrupted.
Attributes
Returns
The completion result with metrics recorded as a side-effect.
Validates client configuration and connectivity to the LLM provider.
Validates client configuration and connectivity to the LLM provider.
May perform checks such as verifying API credentials, testing connectivity, and validating configuration. Default implementation returns success; override for provider-specific validation.
Attributes
Returns
Right(()) if validation succeeds, Left(LLMError) with details on failure
protected def withMetrics[A](provider: String, model: String, operation: => Result[A], extractUsage: A => Option[TokenUsage], extractCost: A => Option[Double]): Result[A]
Executes operation and records metrics for the call.
Executes operation and records metrics for the call.
Latency and outcome (success or classified error) are recorded for every call regardless of result. Token counts and cost are recorded only on success — a Left result emits an Outcome.Error (org.llm4s.metrics.Outcome) event whose kind is derived from the org.llm4s.error.LLMError subtype via ErrorKind.fromLLMError.
Value parameters
extractCost
Extracts the pre-computed cost (USD) from a successful result; return None to skip cost recording.
extractUsage
Extracts prompt/completion token counts from a successful result; return None to skip token recording.
model
Model identifier forwarded to the collector.
operation
The LLM call to time and observe.
provider
Provider label forwarded to the collector (e.g. "openai").