Structured Output with completeStructured
Ask a model for a typed value instead of free text, and know what you get back when it does not comply.
Table of contents
- 1. What it does
- 2. Describing the type
- 3. Fenced and prose-wrapped replies
- 4. What is sent, and why every property is required
- 5. Options
- 6. Choosing the schema name or strictness
- 7. When the reply is not usable
- 8. What each provider does with the schema
- 9. Limits and gotchas
- 10. Streaming structured output
- Related
1. What it does
LLMClient.completeStructured[A] sends a conversation, asks the provider for JSON that matches a schema,
and returns the reply as a value of type A:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model.{ Conversation, UserMessage }
import org.llm4s.toolapi.Schema
import org.llm4s.types.Result
import upickle.default.{ macroRW, ReadWriter }
final case class Invoice(vendor: String, amount: Double, currency: String)
object Invoice { implicit val rw: ReadWriter[Invoice] = macroRW }
val invoiceSchema =
Schema
.`object`[Invoice]("An invoice extracted from text")
.withRequiredField("vendor", Schema.string("Name of the vendor or supplier"))
.withRequiredField("amount", Schema.number("Total invoice amount as a decimal number"))
.withRequiredField("currency", Schema.string("ISO 4217 currency code, e.g. USD, EUR, GBP"))
def extractInvoice(client: LLMClient, text: String): Result[Invoice] =
client.completeStructured[Invoice](
Conversation(Seq(UserMessage(s"Extract the invoice from this text:\n$text"))),
invoiceSchema
)
The snippets below build on these definitions (Invoice, invoiceSchema and extractInvoice) and import
what else they use.
The result is a Result[Invoice]: Right(Invoice("Acme Supplies Ltd", 1250.0, "GBP")) for a reply of
{"vendor":"Acme Supplies Ltd","amount":1250.0,"currency":"GBP"}. Other replies that can be deserialised
as Invoice also return Right; replies that cannot be deserialised return Left (section 7).
A runnable version that builds the client from configuration is StructuredOutputExample:
1
sbt "samples/runMain org.llm4s.samples.basic.StructuredOutputExample"
Use it for extraction, classification, routing and any step whose output a program, not a person, reads.
completeStructured makes one non-streaming complete call; to show the value while the model writes it, see
Streaming structured output.
2. Describing the type
Two things describe A, and they have to agree:
- The schema, an
ObjectSchema[A]built withSchema.\object`,withRequiredFieldandwithProperty`. It is what the provider is shown. The same builders define tool parameters; see Agents and tools. - A uPickle reader, an implicit
upickle.default.Reader[A](aReadWriterfrommacroRWis the usual way). It is what turns the reply intoA.
Nothing checks that the two match, and some mismatches are never reported. A field the case class has but
the schema lacks fails to parse when the reply omits it (section 7), unless the field has a default, which
the reader fills in silently. A field the schema has but the case class lacks is worse: the provider returns
it, the reader ignores it as an extra key, and the result is Right(A) with that value dropped. Keep the
schema and the case class in step by hand, and test the pair on a sample reply.
The schema is a closed object: additionalProperties is false unless you pass true to ObjectSchema.
llm4s-schema-derivation removes the need to keep the two in step: client.completeStructuredOf[A](conversation)
derives the schema from A (derives SchemaOf); see Derived Schemas.
Sum types
When the answer is one of several shapes (“refund this amount”, “escalate to this team” or “reply with this
text”), make A an enum or sealed trait whose cases have fields, deriving SchemaOf and ReadWriter, and call
completeStructuredOf[A]. The derived schema is a JSON Schema anyOf of one object per case, each tagged with
the discriminator uPickle writes ("$type", or the trait’s @upickle.implicits.key), so the reply reads back into
the right case. Because OpenAI’s strict mode refuses an anyOf at the root, the request asks for
{"value": <the case>} and the reply is unwrapped before it is read; the caller only sees A. The request is a
ResponseFormat.JsonSchema that core’s ObjectSchema could not express, sent the same way to every provider
(section 8). Derived Schemas: Sum types has the example, compiled, and the exact
shape of the schema.
3. Fenced and prose-wrapped replies
Models often wrap JSON in a markdown fence or write a sentence around it. Before parsing, the reply is
normalised: a surrounding code fence is stripped, and if the rest is still not JSON, the first balanced
{...} or [...] that parses as JSON is taken from it. Both of these give the same Invoice:
1
2
3
4
5
6
```json
{"vendor":"Acme Supplies Ltd","amount":1250.0,"currency":"GBP"}
```
Sure! Here is the invoice you asked for: {"vendor":"Acme Supplies Ltd","amount":1250.0,"currency":"GBP"}. Let me
know if you need anything else.
The recovery is bounded: it tries at most 16 places where a { or [ starts, so a long reply full of braces
that are not JSON is not searched forever. If it finds nothing, the trimmed text goes on to the parser, which
reports the failure.
That function, LLMClient.extractJson, is internal to llm4s (private[llmconnect]). You cannot call it, and
code that calls complete and parses the reply itself gets no recovery (section 6).
4. What is sent, and why every property is required
completeStructured turns your schema into JSON Schema with strict = true, wraps it in
ResponseFormat.JsonSchema(schema) and sets that as the responseFormat of the call.
strict = true lists every property under required, including a property you declared optional:
1
2
3
4
5
6
7
8
9
10
import org.llm4s.toolapi.Schema
val note =
Schema
.`object`[Map[String, String]]("A note")
.withRequiredField("vendor", Schema.string("Vendor"))
.withProperty(Schema.property("note", Schema.string("Free-text note"), required = false))
note.toJsonSchema(strict = true)("required") // ["vendor", "note"]
note.toJsonSchema(strict = false)("required") // ["vendor"]
So required = false does nothing through completeStructured: the provider is told note is required, and a
model that follows the schema always includes it. If you need a value that may be absent, make the field’s
type allow it (Schema.nullable(...), with an Option in the case class) instead of leaving the field out.
The name and strict fields of the format are the defaults, "response" and true. They are not parameters
of completeStructured; section 6 shows how to set them.
5. Options
completeStructured keeps every option you pass and replaces only responseFormat:
1
2
3
4
5
6
7
8
9
10
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model.{ CompletionOptions, Conversation, UserMessage }
import org.llm4s.types.Result
def deterministic(client: LLMClient, text: String): Result[Invoice] =
client.completeStructured[Invoice](
Conversation(Seq(UserMessage(text))),
invoiceSchema,
CompletionOptions().withTemperature(0.0).withMaxTokens(256)
)
Temperature, token limit and the rest are passed to complete as given. The provider client may still
adjust them for the model, as it does for any complete call: OpenAI reasoning models, for example, omit
sampling parameters, and an option the model does not support can be dropped. A responseFormat you set
yourself is overwritten by the schema’s, so completeStructured is not a way to request plain JSON mode; call complete
with ResponseFormat.Json for that.
6. Choosing the schema name or strictness
For a name other than "response", or strict = false, call complete with your own format and read the
reply yourself:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import org.llm4s.error.ValidationError
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model.{ CompletionOptions, Conversation, ResponseFormat, UserMessage }
import org.llm4s.types.Result
def withCustomFormat(client: LLMClient, text: String): Result[Invoice] = {
val format = ResponseFormat.JsonSchema(
invoiceSchema.toJsonSchema(strict = false),
name = "invoice",
strict = false
)
for {
completion <- client.complete(Conversation(Seq(UserMessage(text))), CompletionOptions().withResponseFormat(format))
// No fence or prose recovery on this path: the reply has to be bare JSON.
invoice <- scala.util.Try(upickle.default.read[Invoice](completion.content)).toEither.left.map(e =>
ValidationError("invoice", e.getMessage)
)
} yield invoice
}
name and strict only mean something to providers that have an equivalent (section 8). Where there is none,
as with Ollama, they are ignored.
7. When the reply is not usable
A reply that cannot become an A is a ValidationError whose field is "structured_output":
| The reply | error.message contains |
|---|---|
| text with no JSON in it | Response is not valid JSON |
the JSON value null |
Response does not match expected schema: got JSON null |
| JSON of the wrong shape (an array for an object, a missing field, a wrong type) | Response does not match expected schema, then the JSON path ($['amount']) and the reader’s reason |
The reply is deserialised, not validated against the schema. A missing field or a wrong type fails because
the uPickle reader for A fails on it, but a constraint the reader does not check is not checked at all: an
enum, a numeric or string bound, or additionalProperties = false (extra keys are ignored). A reply that
breaks one of those still comes back as Right(A), which matters most where the provider does not enforce
the schema (section 8). Check such constraints on the result yourself.
A failed provider call is a different error, returned unchanged (a NetworkError, RateLimitError and so on;
see Error Handling). Telling the two apart is a pattern match:
1
2
3
4
5
6
7
8
import org.llm4s.error.ValidationError
import org.llm4s.types.Result
def describe(result: Result[Invoice]): String = result match {
case Right(invoice) => s"ok: ${invoice.vendor}"
case Left(error: ValidationError) if error.field == "structured_output" => s"model reply rejected: ${error.message}"
case Left(error) => s"call failed: ${error.message}"
}
A rejected reply is often worth one retry. A failed call is handled by whatever retry you already use
(a ReliableClient retries transient provider errors, but never a ValidationError):
1
2
3
4
5
6
7
8
9
import org.llm4s.error.ValidationError
import org.llm4s.llmconnect.LLMClient
import org.llm4s.types.Result
def retryOnce(client: LLMClient, text: String): Result[Invoice] =
extractInvoice(client, text) match {
case Left(error: ValidationError) if error.field == "structured_output" => extractInvoice(client, text)
case other => other
}
ValidationErrors are not marked recoverable on their own, so decide per call site whether a retry is
reasonable. A model that refuses, or answers in prose again, will fail again.
8. What each provider does with the schema
The format is a request, not a guarantee. This is what each client does with ResponseFormat.JsonSchema,
taken from its code:
| Provider | What is sent | Enforced? |
|---|---|---|
| OpenAI, Azure OpenAI | response_format of type json_schema, with your name and strict |
Yes: the provider constrains generation to the schema |
| Requesty | The same OpenAI json_schema shape, with your name and strict |
Depends on the routed model: Requesty is a multi-provider router, so enforcement is up to the backend model or fallback it selects |
| openai-compatible: generic, DeepSeek, Z.ai, OpenRouter, Mistral | The same OpenAI json_schema shape, with your name and strict |
The server decides: a server that ignores response_format returns unconstrained text |
| Cohere (through its OpenAI-compatibility API) | response_format of type json_object carrying the schema; name and strict are not sent |
Cohere’s own mechanism |
| Gemini, Vertex AI | responseMimeType: application/json and responseSchema set to the schema |
Yes, by Gemini |
| Ollama | The schema as the top-level format field of /api/chat |
Yes, on Ollama 0.5 or later; an older server rejects or ignores it. name and strict are ignored |
| Anthropic | Nothing is enforced. The schema is appended to the system prompt as an instruction to answer with JSON only | No: best effort. Expect section 7 errors more often |
| watsonx (Beta) | response_format of type json_schema, with your name and strict (ResponseFormat.Json as json_object) |
Depends on the model; not yet confirmed against the live service (#1314) |
| AWS Bedrock | Nothing. This client does not read responseFormat |
No: the model sees only your conversation |
For the last row, completeStructured still parses whatever comes back, so a model that happens to answer
in JSON works and one that does not gives a ValidationError. Ask for JSON in the prompt yourself when you
use that provider.
9. Limits and gotchas
- The model registry can switch the schema off. The OpenAI, Anthropic, Gemini, Vertex AI and Bedrock
clients check the options against the model registry first. When the registry records that a model does not
support response schemas, the schema is dropped without an error (and with it Anthropic’s prompt
instruction), and
completeStructuredthen parses an unconstrained reply. - Every property is required (section 4), whatever you declared.
- Integers above 2^53. A uPickle
Longfield reads a number such as 9007199254740993 exactly, becausecompleteStructuredreadsAfrom the reply’s text. A reader that starts fromujson.Value(the circe, play-json, zio-json and jsoniter-scala bridges) and every tool argument get the rounded double instead, so send large identifiers as strings there; see Numbers. - One call, no repair. There is no automatic retry inside
completeStructured. It does not stream;streamStructuredOfdoes (section 10). - The content goes to the provider. The conversation and the schema are sent to whichever provider the client talks to, like any other completion.
- The schema is JSON Schema, not your case class. A type uPickle can read but
ObjectSchemacannot describe (a map with free keys, a recursive type) needs a schema that matches what the model should produce, and a reader that accepts it. A sum type is described bycompleteStructuredOf(Sum types).
10. Streaming structured output
completeStructured shows nothing until the whole reply has arrived. For a long object, such as an invoice with
thirty line items, llm4s-schema-derivation’s client.streamStructuredOf[A] streams it instead: it sends the request
completeStructuredOf[A] sends, through streamComplete, and calls your function with the value so far each time it
changes. The result is the finished value, read exactly as completeStructuredOf reads it, with the same errors.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model.Conversation
import org.llm4s.schema.*
import org.llm4s.types.Result
import upickle.default.ReadWriter
case class LineItem(sku: String, quantity: Int, price: BigDecimal) derives SchemaOf, ReadWriter
case class Order(vendor: String, currency: String, items: List[LineItem]) derives SchemaOf, ReadWriter
def extractOrder(client: LLMClient, conversation: Conversation): Result[Order] =
client.streamStructuredOf[Order](conversation) { p =>
p.vendor.foreach(vendor => println(s"vendor: $vendor")) // p.vendor: Option[String]
println(s"rows: ${p.items.getOrElse(Nil).size}") // p.items: Option[List[(sku: Option[String], ...)]]
}
The value passed to the function is a named tuple with Order’s field names, so p.vendor compiles and p.vendr
does not. Each field is None until the model starts writing it, then holds what has arrived:
| Field type | Partial value |
|---|---|
String |
the text so far |
| a case class | its own named tuple of Options |
List, Vector, Seq |
the elements so far; the last one partial when it is a string or a case class |
Option[T] |
Some(None) once the model writes null, else T’s partial value in a Some |
anything else: numbers, Boolean, BigDecimal, enums, sum types, Set |
the value, only once complete |
Each value extends the one before it: a string only grows, a list only gains elements, and a number is shown only
once complete, never 1 while 12.5 is still arriving. A value the model writes with the wrong type (a number for a
string) is left out, and the result is then the ValidationError of section 7. Text before the first {, such as a
code fence, and thinking deltas are not parsed. The function runs on the thread that streams, so keep it quick.
To name the partial type in a signature, take it from its instance:
1
2
3
4
val orders = summon[PartialOf[Order]]
def render(p: orders.Out): String =
s"${p.vendor.getOrElse("...")}: ${p.items.getOrElse(Nil).size} rows"
A must be a case class with fields; a sum type (which completeStructuredOf accepts) does not compile, and nor does
a field type that has neither a partial form nor a uPickle Reader. Field access by name needs Scala 3.7’s default
source level. The reply is parsed again on each chunk with PartialJson, which is public if you need it on its own,
so the cost grows with the square of the reply’s length: fine for structured output, slow for many megabytes.
StreamingStructuredOutputExample in modules/samples prints a growing invoice from a scripted stream, with no API key:
1
sbt "samples/runMain org.llm4s.samples.basic.StreamingStructuredOutputExample"
Related
- Error Handling: every
LLMError, and how to combine results. - Providers: configuring each provider.
StructuredOutputExampleinmodules/samples: the runnable version of section 1.