RFC 0009: The model call contract
- Status: Proposed
- Author(s): @Somay-kousis
- Tracking issue: #2613
- Pull request: #2614
- Line references:
file:linecitations are to main atd928be4f4(2026-11-10), when this was written.
Summary
Right(completion) from any LLMClient should mean one thing: the provider said the model’s turn ended, and
completion says how. Today it does not. A tool call whose arguments were cut comes back as Right with {} and
can run (run). A stream that was cut, or that carried an error, came back as Right from the OpenAI-compatible client
until #2550 (run). By the code, a cut stream still does from Anthropic, Gemini, Vertex AI and Ollama, and an error
event mid-stream does from Ollama (read). This RFC writes the contract
down and makes every chat provider keep it. It has five parts, shipped as the PRs in Delivery.
- Part 1: A typed finish reason.
CompletiongainsfinishReason: FinishReason(Stop,ToolCalls,Length,Other(raw)).StreamedChunk.finishReasonchanges fromOption[String]toOption[FinishReason]. Each client maps its provider’s reason; none hardcodes one. A content-filter finish is not a reason but aLeft(ContentFilteredError), from every provider, as #2664 made it for Gemini and Vertex AI. - Part 2: Stream endings. A stream is
Rightonly if the provider sent its end marker and no error. #2550 did this for the OpenAI-compatible client; Bedrock, watsonx and Perplexity already did. The rest follow. - Part 3: Tool call arguments. Arguments that are not a complete JSON object are a
Left, never{}, never dropped. A reply that ended onLengthwith a tool call in it is aLeft. - Part 4: Retries. A layer that retries
streamCompleteretries only while no chunk has reached the caller. - Part 5: A conformance suite in
llm4s-provider-testkitthat every chat provider module must pass: clean end, cut by limit, stream cut short, error event, cut tool call.
Error endings stay Left, as they are today where they are detected. FinishReason has no Error case.
Motivation
Evidence is labelled: (run) reproduced on main with a local HTTP server replaying the bytes described; (live) captured from a real provider and replayed through the llm4s clients; (read) from reading the code on main, not run; (bytecode) from the SDK jar the build resolves, not run.
A caller can’t tell a finished answer from a cut one, and a cut tool call can run. Reproduced on main on
2026-11-05 (run). The first three rows are tests in OpenAICompatibleStreamEndingSpec (#2550) and the last in
ReliableClientStreamReplaySpec (#2562, merged); the complete and agent rows use the inputs shown and become tests in PR 3:
| Input from provider | Expected | Got |
|---|---|---|
Stream ends, no finish_reason, no [DONE] |
Left |
Right, content “The answer is” |
| Stream cut after the tool name, before the arguments | Left |
Right, purge_files({}) |
Stream carries an {"error":...} event |
Left |
Right, content “par” |
complete with arguments {"older_than_days": 3 and finish_reason: length |
not {} |
purge_files({}) |
An agent receives that {} call for a tool whose parameters are all optional |
tool not run | tool ran once, run Completed |
ReliableClient retries after a partial stream |
caller sees “Hello” | caller saw “HelHello”, completion says “Hello” |
The first three rows are fixed for the OpenAI-compatible client by #2550 (merged as a2edf440b). The last three are
not fixed anywhere. The complete row is the ordinary case of a model reaching max_tokens inside a tool call. It was
replayed from a local server; which real providers send that reply as HTTP 200 is not known yet (Groq answers
complete with HTTP 400 for it, see below).
Real providers send these replies. Captured from Groq (openai/gpt-oss-20b) and Gemini (gemini-3.5-flash) and
replayed through the llm4s clients (live; captured with plain HTTP calls on 2026-11-05, so no llm4s request reached a
live server; the table describes each response, and the same request reproduces it):
| Case | Real server sends | llm4s main | after #2550 |
|---|---|---|---|
| Groq tool stream, token limit | HTTP 200, an error event with status_code 400, no finish_reason, no [DONE] |
Right, 0 chars, no tools |
Left, not retried |
| Groq text stream, token limit | length, [DONE] |
Right, 0 chars |
Right, 0 chars |
Gemini complete, MAX_TOKENS before any text |
content with no parts, 531 thought tokens |
Left(UnknownError: key not found: parts) |
not touched |
| Gemini stream, same | MAX_TOKENS, no text |
Right, 0 chars |
not touched |
Two things follow. A reply can honestly be Right("") because the model spent its whole budget thinking, and
Completion cannot say so. And the same Gemini situation is an UnknownError from complete and an empty success
from streamComplete. A caller can’t write one handler for both. (#2563 and #2664 have since fixed the Gemini rows:
both paths now give an empty Right.)
Completion has no finish reason. Completion (Completion.scala:27-37) has none. Only StreamedChunk has one,
as a raw provider string (Completion.scala:219). StreamingAccumulator records it (StreamingAccumulator.scala:82)
and drops it: toCompletion builds the Completion without it and always returns Right
(StreamingAccumulator.scala:138-173). Tracing shows the gap. TracingLLMClient reports finishReasons from the
stream’s chunks and None for every complete call (TracingLLMClient.scala:42), so the OpenTelemetry attribute
gen_ai.response.finish_reasons (OpenTelemetryTracing.scala:223) is empty for non-streamed calls.
The repo already holds this rule in places. Bedrock fails a stream that ended without messageStop, with the
comment that “a truncated tool call is worse” (BedrockClient.scala:383-386). watsonx fails a stream with no
finish_reason or an error-class one (WatsonXClient.scala:475-490). Perplexity fails a stream without
response.completed (PerplexityClient.scala:216-217). #2550 brought the OpenAI-compatible client in line. The
other clients, and the tool-argument path, do not hold it.
No provider is held to any of this. ProviderModuleChecks.assertStreams passes when a client returns Right and
delivered one chunk (ProviderModuleChecks.scala:233-238). Nothing checks how a stream ended, what the finish reason
was, or what a cut tool call becomes.
Consumers waiting on a typed reason. #2278 (Spring AI bridge) needs a finish reason in ChatResponse
metadata; today there is none to map from a non-streamed Completion.
Where each client stands today
Read on main d928be4f4. “Ending enforced” means a stream with no end marker is a Left.
| Client | complete reads the reason |
Stream reason | Ending enforced | Bad or cut tool arguments |
|---|---|---|---|---|
OpenAIClient (OpenAI, Azure, Requesty; openai-java 4.69.3) |
No (OpenAIClient.scala:665-692) |
Raw string (:364). Fake streaming hardcodes "stop" (:440) |
Not in llm4s (:298-334). The SDK stops at [DONE] and throws SseException on an error payload (bytecode). A stream that ends with neither: unverified |
complete drops the call silently (:733-752). Stream: the accumulator, below |
OpenAICompatibleClient (DeepSeek, Mistral, Cohere, OpenRouter, xAI, Z.ai, generic) |
No (OpenAICompatibleClient.scala:539-565) |
Raw string (:647) |
Yes, since #2550 (:239-243, :253-256) |
complete: {} through lenientToolCalls (OpenAICompatibleDialect.scala:191, :231-239), pinned by OpenAICompatibleDialectToolCallsSpec.scala:26-28. OpenRouter overrides it and fails instead (OpenRouterClient.scala:285-286, ToolCallDeserializer.scala:50-53). Stream: the accumulator |
AnthropicClient (anthropic-java 2.65.0) |
No, stop_reason is never read (AnthropicClient.scala:725-773) |
Hardcoded "stop" on message_stop (:485-494); message_delta’s stop_reason is not passed on |
Not in llm4s. The SDK throws SseException on an error event (bytecode). A stream that ends without message_stop: unverified |
complete: Left on unreadable input (:793-806). Stream: a tool block starts as ujson.Obj() (:477), then the accumulator |
GeminiClient, VertexAIClient (no-parts and filter cases since fixed by #2563, #2664) |
No. A candidate with no parts throws, giving UnknownError (GeminiClient.scala:459, VertexAIClient.scala:401) (live) |
Read (GeminiClient.scala:520, VertexAIClient.scala:455), but a chunk with no parts is dropped whole by the surrounding Try (GeminiClient.scala:513, :526; VertexAIClient.scala:452, :459), so MAX_TOKENS with no text is lost (live) |
No (GeminiClient.scala:249, VertexAIClient.scala:238). An unreadable chunk is skipped (GeminiClient.scala:240, VertexAIClient.scala:229) |
Arguments are native JSON in the envelope; too deep is refused (#1562). A cut function call: untested (Gemini returned 429 before capture) |
OllamaClient |
No, done_reason is not read (OllamaClient.scala:367-380) |
Made up from done: "tool_calls" or "stop" (:238, :251), so a length end reads as "stop" |
No (:291). A {"error":...} line is not checked for (read) |
Left for anything but a JSON object (:534-564) |
BedrockClient |
Only into the exchange log (BedrockClient.scala:702) |
Raw stopReason (:288, :375-378) |
Yes (:383-386) |
complete: Left only when too deep (:862-870). Stream: a tool starts as ujson.Obj() (:368), then the accumulator. A max_tokens stop after a tool start gives {} (read) |
WatsonXClient |
Checks error-class reasons (WatsonXClient.scala:576, :488-490), does not return the reason |
Normalised string (:642-655) |
Yes (:475-485) |
Left unless a JSON object; absent or blank is {} (:767-777) |
PerplexityClient |
Accepts incomplete (PerplexityClient.scala:459, :505) and does not report it |
Made up from status: tool_calls, length or stop (:224) |
Yes (:216-217) |
Left unless a JSON object; absent or blank is {} (:520-531) |
The shared accumulator. Every streamed tool call goes through StreamingToolArgumentParser.parse
(StreamingToolArgumentParser.scala:18-19): empty arguments become {}, unparseable ones stay a raw ujson.Str.
So a stream cut after the tool name gives {} (run), whichever client produced the chunks.
If nothing changes, an agent on DeepSeek, Mistral, Cohere, xAI, Z.ai or a generic endpoint can run a tool with
arguments the model never finished writing, and report the run Completed (run). A caller on any provider can’t tell
a cut answer from a whole one without parsing a provider string from a stream, and can’t do it at all for complete.
Detailed design
Part 1: A typed finish reason
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
package org.llm4s.llmconnect.model
/** Why the model's turn ended, as the provider reported it, mapped to one vocabulary. */
@Stable
enum FinishReason derives ReadWriter {
/** The model ended its turn: a natural stop, or a stop sequence. */
case Stop
/** The model asked for one or more tool calls, each with complete arguments. */
case ToolCalls
/** An output limit was reached: max tokens, or the context window. The content is cut short. */
case Length
/** A reason this client has no mapping for, kept as the provider sent it. */
case Other(raw: String)
}
There is no Error case. An error ending is a Left, in every client. A Right with an error inside it is the
thing this RFC removes.
Completion, a growth-prone @Stable type, gains the field under the RFC 0002 pattern:
1
2
3
4
5
6
7
8
9
10
11
12
13
final case class Completion private (
id: String,
created: Instant,
content: String,
model: String,
message: AssistantMessage,
toolCalls: List[ToolCall],
usage: Option[TokenUsage],
estimatedCost: Option[Double],
citations: List[Citation],
finishReason: FinishReason // new; FinishReason.Stop in apply (open question 1)
)
def withFinishReason(finishReason: FinishReason): Completion
The previous apply stays as an overload without defaults that forwards, as CLAUDE.md requires for a new field.
StreamedChunk.finishReason becomes Option[FinishReason], with withFinishReason(FinishReason) and
withFinishReason(Option[FinishReason]). A stream and a complete then speak one vocabulary.
StreamingAccumulator keeps the last reason it saw and puts it on the Completion it builds.
Mapping. Each client maps its own provider’s values, in its own module. No shared mapper trait.
FinishReason |
Chat completions (OpenAI, compatible, watsonx) | Anthropic stop_reason |
Gemini finishReason |
Bedrock stopReason |
Ollama done_reason |
Perplexity status |
|---|---|---|---|---|---|---|
Stop |
stop |
end_turn, stop_sequence |
STOP with no function call |
end_turn, stop_sequence |
stop |
completed, no call |
ToolCalls |
tool_calls, function_call |
tool_use |
STOP with a function call |
tool_use |
stop with a call |
completed, with a call |
Length |
length |
max_tokens, model_context_window_exceeded |
MAX_TOKENS |
max_tokens, model_context_window_exceeded |
length |
incomplete |
Left(ContentFilteredError) |
content_filter |
refusal |
SAFETY, RECITATION, BLOCKLIST, PROHIBITED_CONTENT, SPII and the rest #2664 lists (done) |
guardrail_intervened, content_filtered |
none | none |
other Left |
error (and watsonx’s cancelled, time_limit, as today) |
an error event |
any other finish (MALFORMED_FUNCTION_CALL, OTHER, …) with no text and no call: ValidationError (done in #2664). With text it is Right, Other(raw) |
an exception from the SDK | an error line |
failed |
The vendor values in this table come from the vendors’ documentation, except those the clients already read
(WatsonXClient.scala:648-655, Groq’s length and Gemini’s MAX_TOKENS (live)). Each mapping PR checks its row
against captured replies. Any other value is Other(raw), so a new provider value never fails a call.
Gemini, specifically. A candidate with a finishReason and no parts is a reply, not a malformed one. #2563 and
#2664 (merged) made both paths of Gemini and Vertex AI return Right with empty content for it when it finished
STOP, MAX_TOKENS or CONTINUATION, instead of UnknownError (live); a content-filter finish is a
ContentFilteredError, and any other finish with no text and no call a ValidationError. PR 2 adds
finishReason = Length for MAX_TOKENS.
Where it is read.
TracingLLMClientreports theCompletion’s reason forcompleteas well as for streams, soTraceEvent.CompletionReceived.finishReasons(TraceEvent.scala:109) and the OpenTelemetry attribute stop being empty for non-streamed calls.finishReasonsstaysSeq[String], filled with the mapped name (stop,length,tool_calls, orOther’s raw value).- Agent runs.
AgentEvents.ModelCallCompleted(AgentEventPayloads.scala:33-41) gainsfinishReason: Option[FinishReason] = None, filled from the call’sCompletion. A stored event written before the field existed still reads, asNone. This follows RFC 0008’sadjustmentson the same event. The agent’s own behaviour on aLengthtext answer does not change here (open question 6).
Part 2: Stream endings
Rules for every streamComplete:
- The end marker is required. The stream is
Rightonly if the provider sent its end marker:finish_reasonor[DONE](chat completions; a server may send only one),message_stop(Anthropic),messageStop(Bedrock), a candidatefinishReason(Gemini),done: true(Ollama),response.completed(Perplexity). A stream that ends without it is aLeft: a retryableNetworkError, as #2550 does, because a reconnect can succeed. watsonx and Perplexity already returnServiceError(502)here (WatsonXClient.scala:477-482,PerplexityClient.scala:216-217); this RFC does not change them, and the suite accepts either. - An error ends the call. An error event, or an error-class reason, is a
Leftcarrying the provider’s error, even after chunks were delivered. Reading stops there. - The reason is the provider’s. The
Completion’sfinishReasonis the mapped provider value. A client never writes"stop"because the stream ended. - Chunks already delivered stay delivered. A
Leftafter chunks does not unsend them. The caller learns the reply was incomplete from theLeft.
What each client needs:
| Client | Rule 1 | Rule 2 | Rule 3 |
|---|---|---|---|
OpenAICompatibleClient |
Done (#2550) | Done (#2550) | PR 1 |
BedrockClient |
Done (BedrockClient.scala:383-386) |
Done (:382) |
PR 2 |
WatsonXClient |
Done (WatsonXClient.scala:475-485) |
Done (:365-414, event: error) |
PR 1 |
PerplexityClient |
Done (PerplexityClient.scala:216-217) |
Done (:201-205) |
PR 1 |
OpenAIClient |
Unverified for a stream that ends with neither [DONE] nor a reason. PR 4 checks it in llm4s rather than relying on the SDK |
SDK throws (bytecode); covered by the suite in PR 5 | PR 1 (fake streaming stops hardcoding "stop") |
AnthropicClient |
Missing in llm4s; PR 4 requires message_stop |
SDK throws (bytecode); covered by the suite | PR 2 (read message_delta.stop_reason) |
GeminiClient, VertexAIClient |
Missing (GeminiClient.scala:249); PR 4 |
Unverified: llm4s has no handling for an error object mid-stream. PR 4 adds it | PR 2 |
OllamaClient |
Missing (OllamaClient.scala:291); PR 4 requires done: true |
Missing (read); PR 4 | PR 2 (read done_reason) |
Part 3: Tool call arguments
Rules for complete and streamComplete alike:
- Never
{}for a broken call. A tool call whose arguments are not a complete JSON object is aLeft, using the malformed-tool-callProcessingErrorthe Anthropic, Ollama, watsonx and Perplexity clients already return. It is never replaced by{}, never dropped, and never passed on as a raw string. - Empty means “no arguments” only on a clean end. Absent or blank arguments are
{}when the reply ended withToolCallsorStop: a call to a tool with no parameters. watsonx and Perplexity already read it this way (WatsonXClient.scala:770-772,PerplexityClient.scala:522-524). - A tool call in a
Lengthreply is aLeft. The model was cut while writing calls. A later call may be missing, or a call’s arguments may be a valid prefix of what it meant. Running any of them acts on intent the model did not finish stating.
Changes:
OpenAICompatibleDialect.lenientToolCallsandparseToolCallsreturnResult[Seq[ToolCall]], with aLeftfor unparseable or too-deep arguments. The arguments stay unparsed when too deep, so #1562’s protection is kept: the difference is that the call fails instead of becoming{}.OpenAIClient.extractToolCallsreturns aLeftinstead of dropping the call (OpenAIClient.scala:733-752). Today a reply whose only call was cut comes back as a text answer with no tool call, and the caller can’t see why.StreamingToolArgumentParser.parsereturnsResult[ujson.Value].StreamingAccumulator.toCompletion, which already returnsResult, returns theLeftfor any call that fails rules 1 to 3. Every streaming client gets the rule from the accumulator, including Anthropic and Bedrock, whose streamed calls start asujson.Obj().
A spec that pins today’s behaviour changes. OpenAICompatibleDialectToolCallsSpec.scala:26-28 asserts that
{not json becomes {}, and :31-35 and :44-45 assert the same for arguments nested too deeply; :37-43 accepts a 512-deep array, which rule 1
makes a Left (it is not an object). The reason to
change them: they encode the defect this part removes, and an agent run on 2026-11-05 ran a tool on that {} (run).
The depth limit itself stays tested. The new assertions are that each case is a Left and that nothing deeper than
512 levels is parsed. This needs the module owner’s agreement before the PR.
Part 4: Retries
Rule. A layer that retries streamComplete retries only while no chunk has reached the caller’s onChunk.
After the first chunk, the failure is returned as is.
LLMClientRetry.streamCompleteWithRetry keeps this rule with its chunkEmitted flag
(LLMClientRetry.scala:121-133). ReliableClient.streamComplete does not: it passes the caller’s onChunk straight
to each attempt (ReliableClient.scala:89-97), so a retry after a partial stream delivers the start again. On main,
the caller saw “HelHello” while the Completion said “Hello” (run). The fix is #2562 (merged). This RFC
states the rule so that every retrying layer, now and later, is held to it.
Part 2 makes this rule matter more. Once a cut stream is a Left(NetworkError), which is retryable, more streams reach
the retry path.
Part 5: The conformance suite
llm4s-provider-testkit gains a check that every chat provider module runs from its Llm4s<Name>ModuleSpec, next
to assertStreams:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
enum ContractCase {
case CleanText // Right, FinishReason.Stop, the text
case TextCutByLimit // Right, FinishReason.Length, the partial text
case ToolCall // Right, FinishReason.ToolCalls, arguments a JSON object
case NoArgumentToolCall // Right, arguments {}
case ToolCallCutByLimit // Left; no ToolCall in any Right
case MalformedArguments // Left
case StreamEndsEarly // stream only: Left
case ErrorEvent // stream only: Left
case ContentFiltered // Left(ContentFilteredError), where the provider has a filter
}
def assertModelCallContract(
client: LLMClient,
server: RecordedReplies, // serves the module's captured bytes for each case
cases: Map[ContractCase, Applicability] // Applies, or NotApplicable(reason): no case is left out silently
)(using Position): Assertion
Each module supplies the bytes its provider sends for each case, captured from the real service where possible, and
served by the testkit’s local server (LocalProviderTestServer already serves chat-completions and Anthropic
shapes). The check runs complete and streamComplete for each case, and for the stream also checks that the chunks
delivered match the Completion. The Groq and Gemini captures behind the live evidence above are the first fixtures.
Until a case lands, every provider declares it NotApplicable("PR n"). A case a provider cannot produce is declared NotApplicable with a reason, so a gap shows in the spec instead of
being missed. Whether the SDK clients can point their SDK at the local server needs checking per client: OpenAI and
Anthropic take a base URL; Bedrock’s endpoint override is unverified.
Not changed
- Agent statuses. A
Lengthtext answer still ends an agent runCompleted. The event carries the reason (Part 1); whether the run should end differently is open question 6. RequestAdjustment(RFC 0008). It describes the request as sent.finishReasondescribes how the reply ended. Neither replaces the other.- Usage on a failed call. A billed call that ends in
Leftreports no usage. That is a separate problem (read) and not part of this contract.
Alternatives
- Do nothing per provider; let callers read
StreamedChunk.finishReason. It works only for streams, only with a raw string per provider, and not at all forcomplete. It also leaves the cut tool call running. finishReason: Option[String]onCompletion. Smallest change. Callers would still match on provider strings ("length","max_tokens","MAX_TOKENS"), which drift and never fail a match. RFC 0008 chose an enum over a string forRequestAdjustment.optionfor the same reason.- An
Errorcase inFinishReason. It would let a client returnRightfor an error ending. That is the bug. Lengthas aLeftfor text too. Simpler to state (“Rightmeans a natural end”), but it throws away a usable partial answer and a usable usage report, and Groq and Gemini sendlengthandMAX_TOKENSas ordinary replies (live). A caller that wants a strict rule checksfinishReason.- Drop a cut tool call and return the rest. This is what
OpenAIClientdoes today. The caller gets a text answer with no sign that the model asked for a tool. - Make a cut tool call
{}and let the tool’s schema reject it. It fails for any tool whose parameters are all optional, which is the 2026-11-05 run. - Fix stream endings in the accumulator for everyone. The accumulator can’t know a provider’s end marker, and a stream can carry a reason on a chunk and still be cut afterwards. Each client knows its own protocol. The accumulator does take the tool-argument rule (Part 3), which is protocol-free.
- Retry with de-duplication of chunks instead of no retry after the first chunk. Chunks of two attempts don’t align: a model can phrase the second attempt differently. Not retrying is the only rule a caller can rely on.
Compatibility impact
- Modules and tiers:
llm4s-core(Frozen at 1.0):Completion,StreamedChunk,StreamingAccumulator,StreamingToolArgumentParserandTracingLLMClientchange;FinishReasonis new and@Stable, becauseCompletionis@StableandstabilityTierCheck(#1818) refuses a Stable signature that names an Experimental type; RFC 0008 settled its new types the same way (#2532).Other(raw)is what lets a case be added later: a new provider value maps toOtheruntil it gets its own case.llm4s-openai-compatible:OpenAICompatibleDialect.parseToolCallsandlenientToolCallschange type, which affects anyone who wrote a dialect.llm4s-agent:ModelCallCompletedgains a defaulted field.llm4s-provider-testkit: a new check.llm4s-testkit(ScriptedLLMClient.scala:177) andllm4s-java-api(FakeLlmClients.scala:71) set typed reasons. Provider modules change internally. The Java facade does not expose a finish reason today (no reference inmodules/java-api/src/mainbeyondFakeLlmClients), so it gains no API. - Binary: there is no baseline yet (#1281), so no MiMa filter is needed.
- Source:
- Code that reads
StreamedChunk.finishReasonas aStringbreaks. The samples that print it keep compiling. - A silent case:
chunk.finishReason.contains("stop")still compiles, becauseOption.containswidens its argument, and is now alwaysfalse. The migration note calls this out. Main has no such comparison outside tests (searched); 75 references in 32 test files need updating. StreamingToolArgumentParser.parseand the dialect methods returnResult.- A positional pattern match on
Completionchanges arity, as with any growth-pattern field.
- Code that reads
- Behaviour: calls that returned
Rightcan now returnLeft:- a stream that ends without its end marker (Anthropic, Gemini, Vertex AI, Ollama, and OpenAI if its SDK does not already fail it);
- an error event or line on Gemini, Vertex AI or Ollama streams;
- any tool call with unparseable arguments, or in a
Lengthreply, on every client. OnOpenAIClientsuch a call was dropped; on the OpenAI-compatible clients it was{}; - a content-filter finish on OpenAI, the OpenAI-compatible clients, Anthropic and Bedrock (
content_filter,refusal, guardrail reasons): aContentFilteredError, as on Gemini and Vertex AI since #2664, where today it is a finishedCompletionwith whatever content arrived.
- Wire formats: a
ModelCallCompletedwritten before the change reads withfinishReason = None. No checkpoint or config key changes.
Migration
1
2
3
4
5
6
7
8
9
10
// Before: a provider string, streams only
client.streamComplete(conversation, options, chunk => chunk.finishReason.foreach(r => if (r == "length") warn()))
// After: one type, on the Completion too
client.complete(conversation, options).map { c =>
c.finishReason match {
case FinishReason.Length => warn() // content is cut short
case FinishReason.Stop | FinishReason.ToolCalls | FinishReason.Other(_) => ()
}
}
Check any comparison of a finish reason against a string literal: it compiles and is now always false.
1
2
3
4
// Before: compiles, was true on a normal end
chunk.finishReason.contains("stop")
// After
chunk.finishReason.contains(FinishReason.Stop)
A custom OpenAICompatibleDialect that overrides parseToolCalls returns Result[Seq[ToolCall]] and returns a
Left for arguments it can’t parse, instead of {}.
Code that relied on a cut stream or a cut tool call coming back as Right gets a Left. For a stream it is a
retryable NetworkError; for a tool call it is a ProcessingError. Raising maxTokens is the usual fix for the
second. Code that read the partial content of a filtered OpenAI, Anthropic or Bedrock reply matches
Left(ContentFilteredError) instead; the text a stream already delivered stays delivered.
The CHANGELOG gets Added (FinishReason, Completion.finishReason), Changed (StreamedChunk.finishReason,
the tool-argument rule, stream endings per client, content filtering on the other providers) entries, and lines under
### Breaking changes for StreamedChunk.finishReason, StreamingToolArgumentParser.parse, the dialect methods and
content filtering. ContentFilteredError’s Scaladoc, written for Gemini today, becomes provider-neutral. docs/reference/migration.md gets one section. docs/guide/writing-a-provider.md states
the contract and the conformance check.
Delivery
Already done or in progress, no RFC needed:
- #2550 (merged, a2edf440b). The OpenAI-compatible client fails a stream with no
finish_reasonor[DONE], or with an error event. ReliableClientre-delivery (#2562, merged). No retry after the first chunk.- Gemini and Vertex AI (#2563, #2664, merged). A candidate with no
partsis an empty answer, and a content-filter finish or a blocked prompt aContentFilteredError, on both paths.
Under this RFC, each PR with its own review:
- PR 1 (Part 1, core).
FinishReason,Completion.finishReason, typedStreamedChunk.finishReason, the accumulator carrying it,TracingLLMClientreading it forcomplete. Mapped for the chat-completions clients (OpenAI-compatible, OpenAI including fake streaming, watsonx) and Perplexity. The conformance suite lands here with theCleanText,TextCutByLimitandToolCallcases. TypingStreamedChunk.finishReasontouches every producer (Anthropic, Gemini and Vertex AI, Ollama, Bedrock,ScriptedLLMClient,FakeLlmClients), so PR 1 changes them all, but the clients PR 2 maps writeOther(raw)(orOther("")with no raw value) on both paths until then. No release between PR 1 and PR 2 reports aStopthe provider didn’t send. - PR 2 (Part 1, the rest). Anthropic (
stop_reason,message_delta), Gemini and Vertex AI (MAX_TOKENS), Ollama (done_reason), Bedrock. OpenAI’s and the compatible clients’content_filter, Anthropic’srefusaland Bedrock’s guardrail reasons becomeContentFilteredError; #2664 left them a finishedCompletion. Each adds its fixtures to the suite, and PR 2 adds theContentFilteredcase. - PR 3 (Part 3). The tool-argument rules, in the accumulator, the dialect and
OpenAIClient. The spec change above, agreed first. AddsNoArgumentToolCall,ToolCallCutByLimitandMalformedArgumentsto the suite. - PR 4 (Part 2). End markers and error events for Anthropic, OpenAI, Gemini, Vertex AI and Ollama. Adds
StreamEndsEarlyandErrorEventto the suite. - PR 5 (Part 5).
assertStreamsis replaced by the full check in every chat provider’s module spec, anddocs/guide/writing-a-provider.mddocuments it. - PR 6 (agent).
ModelCallCompleted.finishReason.
Sequencing with RFC 0008. #2532 (RFC 0008 PR 1) added Completion.adjustments and #2617 (PR 2)
ModelCallCompleted.adjustments, both merged. PR 1 here adds another field to Completion, keeping one forwarding apply
overload per the growth pattern, and PR 6 one to ModelCallCompleted beside adjustments. The two fit together: RFC 0008 reports what was changed in the request, and this
RFC reports how the reply ended. The Gemini case links them. A reply that ended on Length with empty content after
531 thought tokens is the case where a caller wants both: finishReason = Length says the budget ran out, and
Thinking.Budget (RFC 0008, not merged yet) and the MaxTokens adjustment say what to change. RFC 0008 is not blocked by this one, nor this one
by it.
This RFC’s own PR also adds its row to docs/rfcs/README.md.
Open questions
- Default in
Completion.apply.FinishReason.Stopkeeps every existing construction site compiling, but a client that forgets to set the reason then reportsStop, which the conformance suite has to catch. The other choice is no default, which breaks every test that builds aCompletion. Maintainers decide. - A
completereply with no reason. watsonx accepts a missingfinish_reason(WatsonXClient.scala:576). Should the client inferToolCallsorStopfrom the body, which is whole, or reportOther("")? - The error for a cut tool call. The existing
ProcessingError, or a new error that carries the partialCompletionso a caller can log or show what arrived? A new error type is a public addition to core. - Where the chat-completions mapping lives. It is shared by
llm4s-openai-compatible,llm4s-openai(which depends on it) andllm4s-watsonx(which depends only on core). In core as a wire-format helper, or a small copy in watsonx? - Unreadable chunks. Gemini and Vertex AI skip a data line that is not JSON (
GeminiClient.scala:240). Is that aLeftunder rule 2, or a tolerated keep-alive? There is no evidence either way yet. - Agent behaviour on
Length. Should an agent run whose last answer was cut endCompleted, or shouldAgentStatussay so? This touches the agent runtime’s owners and could be its own RFC. - Content filter: decided. #2664 made it a
Left(ContentFilteredError)for Gemini and Vertex AI, whether or not text came first, and neutral forReliableClient’s breaker. This RFC takes that rule for every provider, soFinishReasonhas noContentFiltercase. - A Scalafix rule in
modules/scalafix-rulesfor the silentcontains("stop")case, or the migration note only?