Serving an Agent over HTTP
modules/examples-http holds small, tested reference apps that expose an llm4s agent from a Scala HTTP stack. Each
serves the same two endpoints, so they can be compared:
| Endpoint | Request | Response |
|---|---|---|
POST /chat |
{"message": "..."} |
{"threadId", "status", "answer"} as JSON, or an error status |
POST /chat/stream |
{"message": "..."} |
server-sent events of the answer as it is generated |
| App | Built on | Status |
|---|---|---|
| http4s (Ember) | llm4s-effect: AgentIO[IO], an fs2 Stream |
available |
| ZIO HTTP (Netty) | llm4s-zio: AgentZ, ZStream |
available |
| Tapir, served by http4s (Ember) | endpoint definitions interpreted by tapir-http4s-server, over AgentIO[IO]; OpenAPI docs at /docs |
available |
| Play 3 (Pekko HTTP) | llm4s-pekko: AgentPekko, a Pekko Source; Guice wiring |
available |
The module is not published: it is code to read and copy. Its HTTP dependencies stay in it. The apps share what does
not depend on the stack: the request and event JSON (ChatProtocol), the error mapping (HttpErrors), the command
line (ServerSettings) and the agent’s configuration (ChatAgent).
Run it
With the fake client, which needs no model and no API key (it answers “You said: …”):
1
sbt "examplesHttp/run --fake"
With a model: the app loads the provider at startup through Llm4sConfig, from the named section
llm4s.providers.provider names. The module’s application.conf uses the samples’ default, a local Ollama
(ollama pull llama3 first, or set OLLAMA_MODEL), and has openai-main and anthropic-main sections that use
OPENAI_API_KEY and ANTHROPIC_API_KEY:
1
2
sbt examplesHttp/run
LLM4S_PROVIDER=openai-main sbt examplesHttp/run # with OPENAI_API_KEY set
run starts the http4s app; the ZIO HTTP and Tapir apps take the same arguments through runMain:
1
2
3
4
sbt "examplesHttp/runMain org.llm4s.examples.http.ziohttp.ZioChatServerMain --fake"
LLM4S_PROVIDER=openai-main sbt "examplesHttp/runMain org.llm4s.examples.http.ziohttp.ZioChatServerMain"
sbt "examplesHttp/runMain org.llm4s.examples.http.tapir.TapirChatServerMain --fake"
sbt "examplesHttp/runMain org.llm4s.examples.http.playframework.PlayChatServerMain --fake"
--port N and --host H change the address (default 127.0.0.1:8080). Then:
1
2
curl -s localhost:8080/chat -d '{"message": "What is Scala?"}'
curl -N localhost:8080/chat/stream -d '{"message": "What is Scala?"}'
The stream
POST /chat/stream answers 200 with Content-Type: text/event-stream once the request is valid, then sends:
| Event | Data |
|---|---|
delta |
{"attempt", "text"}: the next piece of the answer. A higher attempt means the model call was retried; discard the earlier attempt’s text. |
tool_call |
{"id", "tool"}: the agent is calling a tool. |
done |
the /chat response body; always the last event of a run that ends. |
error |
the error body below, plus status (and retryAfterSeconds): the run failed. |
A : keep-alive comment is sent every 15 seconds while the run is quiet.
The http4s app (ChatEvents) turns AgentIO.stream - the run’s events, then its result - into ServerSentEvents:
1
2
3
4
5
6
7
8
9
10
agent
.stream(ThreadId(UUID.randomUUID().toString), message)
.collect {
case AgentStreamItem.Event(AgentEvents.TextDelta(d)) => event("delta", ChatProtocol.delta(d.attempt, d.text))
case AgentStreamItem.Event(AgentEvents.ToolCallStarted(t)) =>
event("tool_call", ChatProtocol.toolCall(t.toolCallId, t.tool))
case AgentStreamItem.Done(result) => event("done", ChatProtocol.result(result))
}
.handleErrorWith(error => Stream.emit(event("error", ChatProtocol.streamError(HttpErrors.of(llmError(error))))))
.mergeHaltL(Stream.awakeEvery[IO](heartbeat).as(ServerSentEvent(comment = Some("keep-alive"))))
The ZIO HTTP app does the same with AgentZ.stream, writing each event in the SSE format itself, so the keep-alive can
be a comment:
1
2
3
4
5
6
7
8
9
10
agent
.stream(ThreadId(UUID.randomUUID().toString), message)
.collect {
case AgentStreamItem.Event(AgentEvents.TextDelta(d)) => event("delta", ChatProtocol.delta(d.attempt, d.text))
case AgentStreamItem.Event(AgentEvents.ToolCallStarted(t)) =>
event("tool_call", ChatProtocol.toolCall(t.toolCallId, t.tool))
case AgentStreamItem.Done(result) => event("done", ChatProtocol.result(result))
}
.catchAllCause(cause => ZStream.succeed(event("error", ChatProtocol.streamError(HttpErrors.of(llmError(cause))))))
.mergeHaltLeft(ZStream.fromSchedule(Schedule.spaced(heartbeat)).as(": keep-alive\n\n"))
The response is that stream as a chunked body (Body.fromCharSequenceStreamChunked) with
Content-Type: text/event-stream.
The Tapir app serves the http4s app’s stream (ChatEvents), encoded with http4s’s ServerSentEvent.encoder, as a
Tapir streaming body. It does not use tapir-http4s-server’s serverSentEventsBody, because sttp’s ServerSentEvent
has no comment field for the keep-alive. See The Tapir app.
Both agents are built with withStreaming(), so its model calls stream and their text reaches the run’s events as
AgentEvents.TextDelta.
Cancellation
When the client disconnects, the run is cancelled and the provider call interrupted. The server learns of the
disconnect when a write fails; the heartbeat makes a write happen at least every 15 seconds, even while the model is
silent. The failed write interrupts the response stream, and interrupting AgentIO.stream cancels the run
(AgentRun.cancel), which interrupts its model call. Http4sChatServerSpec shows it: a provider call that parks after
its first chunk is interrupted once the test’s socket closes.
The Tapir app hands the same stream to Ember through tapir-http4s-server, so a disconnect cancels the run in the same
way; TapirChatServerSpec makes the same check.
In the Play app, Play’s Pekko HTTP server cancels the response’s Source when the connection closes, and cancelling
AgentPekko.stream cancels the run in the same way; PlayChatServerSpec makes the same check.
In the ZIO HTTP app, the server interrupts the response stream when the connection closes, or when a write fails;
interrupting AgentZ.stream cancels the run in the same way. ZioChatServerSpec makes the same check.
POST /chat runs the turn with AgentIO.run or AgentZ.run, each of which cancels the turn when its fiber is
cancelled. Ember does not watch a connection while it computes a response, so with the http4s app the turn of a client
that leaves /chat runs to the end, and so does the Play app’s, whose AgentPekko.run is a Future, which cannot be
cancelled; use /chat/stream when that matters, or bound the run with a timeout
(RunConfig().withBudgets(RunBudgets().withTimeout(...))).
The Tapir app
ChatEndpoints defines the two endpoints as values, independent of any server, with the errors as a typed output:
ErrorResponse (the status, Retry-After in seconds, the error body), one oneOf variant per status HttpErrors.of
can choose, so the documentation lists each one.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
val chat: PublicEndpoint[String, ErrorResponse, ujson.Value, Any] =
endpoint.post
.in("chat")
.in(requestBody)
.out(jsonBody(summon[Schema[ChatResponse]]).description("The turn's result."))
.errorOut(oneOf[ErrorResponse](variant(StatusCode.BadRequest), ErrorStatuses.tail.map(variant)*))
val chatStream: PublicEndpoint[String, ErrorResponse, Stream[IO, Byte], Fs2Streams[IO]] =
endpoint.post
.in("chat" / "stream")
.in(requestBody)
.out(header(HeaderNames.CacheControl, "no-cache"))
.out(streamTextBody(Fs2Streams[IO])(CodecFormat.TextEventStream(), Some(StandardCharsets.UTF_8)))
.errorOut(oneOf[ErrorResponse](variant(StatusCode.BadRequest)))
The 429 variant adds the Retry-After header. The request body is read as text and checked by
ChatProtocol.message, so an invalid body gets the shared 400 error body instead of Tapir’s own decode-failure
response. The bodies are the shared protocol’s ujson values; the case classes in ChatSchemas (derives Schema)
only describe them for the documentation.
TapirChatRoutes gives the endpoints their logic, the same as the http4s app’s, and interprets them with
Http4sServerInterpreter:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
val chat = ChatEndpoints.chat.serverLogic[IO] { body =>
ChatProtocol.message(body) match {
case Left(invalid) => IO.pure(Left(ErrorResponse.of(HttpErrors.of(invalid))))
case Right(message) =>
agent.run(message).attempt.map {
case Right(result) => Right(ChatProtocol.result(result))
case Left(error) => Left(ErrorResponse.of(HttpErrors.of(ChatEvents.llmError(error))))
}
}
}
val chatStream = ChatEndpoints.chatStream.serverLogicPure[IO] { body =>
ChatProtocol.message(body).left.map(invalid => ErrorResponse.of(HttpErrors.of(invalid)))
.map(message => ChatEvents(agent, message, heartbeat).through(ServerSentEvent.encoder[IO]))
}
val routes = Http4sServerInterpreter[IO]().toRoutes(List(chat, chatStream) ++ docs)
It is served on the http4s app’s Ember server. Tapir 1.11.20 is the last release built on the cats-effect (3.5.7), fs2 (3.11.0) and http4s (0.23.30) versions the module uses; a later Tapir needs cats-effect 3.6.
The OpenAPI documentation
The same endpoint values are interpreted as an OpenAPI document (OpenAPIDocsInterpreter), so the documentation
cannot drift from what is served. With the Tapir app running:
http://localhost:8080/docs- the Swagger UI, where both endpoints can be tried;http://localhost:8080/docs/openapi.yaml- the OpenAPI 3.1 document.
1
2
sbt "examplesHttp/runMain org.llm4s.examples.http.tapir.TapirChatServerMain --fake"
curl -s localhost:8080/docs/openapi.yaml
The document has both endpoints, the request, answer and error body schemas, and every error status with its
description; the 429 response documents Retry-After. To watch the stream’s events as they arrive, use curl -N
rather than the Swagger UI.
The Play app
modules/examples-http/src/main/scala/org/llm4s/examples/http/playframework is a Play 3 app on Scala 3, over
llm4s-pekko. It uses Play as a library - play-guice and play-pekko-http-server - without the Play sbt
plugin: the routes are a SIRD router (ChatRouter) instead of a compiled conf/routes file, and PlayChatServerMain
starts the server itself (PekkoHttpServer.fromApplication). In an application built with the plugin, the module, the
controller and the configuration below are the same; only the router and the start differ.
1
2
sbt "examplesHttp/runMain org.llm4s.examples.http.playframework.PlayChatServerMain --fake"
LLM4S_PROVIDER=openai-main sbt "examplesHttp/runMain org.llm4s.examples.http.playframework.PlayChatServerMain"
Versions
Play 3.0.12 is built on Pekko 1.0.3 and Pekko HTTP 1.0.1; llm4s-pekko is built on Pekko 1.7.1. Pekko refuses to
start with mixed versions of its modules, so the module pins every Pekko artifact Play brings (pekko-actor,
pekko-actor-typed, pekko-slf4j, pekko-serialization-jackson, pekko-protobuf-v3, pekko-stream) to 1.7.1 with
dependencyOverrides, which Pekko’s binary compatibility within 1.x allows. An application on Play 3.0 does the same;
Pekko HTTP keeps its own version. Play is Apache-2.0 licensed, and only this unpublished module depends on it.
Wiring with Guice
ChatModule, a Play Module enabled in application.conf, binds the LLMClient and the agent as eager
singletons, so a provider section that cannot be loaded stops the application from starting instead of failing its
first request:
1
2
bind[LLMClient].toProvider[LLMClientProvider].eagerly(),
bind[AgentPekko].toProvider[AgentPekkoProvider].eagerly()
LLMClientProvider loads the client and registers an ApplicationLifecycle stop hook that closes it when the
application stops:
1
lifecycle.addStopHook(() => Future.fromTry(Try(client.close())))
Configuration
The llm4s settings live in Play’s own application.conf, beside Play’s keys. The client is loaded from the
Configuration Play injects, with Llm4sConfig.from, not from a second read of the classpath:
1
2
3
4
settings <- Llm4sConfig.from(configuration.underlying)
registry <- settings.modelRegistryService()
provider <- settings.defaultProvider()
client <- LLMConnect.getClient(provider)(using registry)
The precedence is the library’s: JVM system properties (-Dllm4s.providers.provider=openai-main) override Play’s
configuration, which overrides each llm4s module’s reference.conf - so llm4s.credentials.<provider>.apiKey, bound
to the vendor’s variable such as OPENAI_API_KEY, applies without a line in application.conf. A key a test sets with
GuiceApplicationBuilder().configure(...) is part of Play’s configuration, so it reaches llm4s too. The module’s
application.conf adds Play’s keys to the shared llm4s block:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
play {
modules.enabled += "org.llm4s.examples.http.playframework.ChatModule"
http.router = "org.llm4s.examples.http.playframework.ChatRouter"
}
examples.chat {
heartbeat = 15 seconds
blocking-dispatcher {
executor = "thread-pool-executor"
throughput = 1
thread-pool-executor {
fixed-pool-size = 16
}
}
}
(Production mode also needs play.http.secret.key, at least 32 bytes; the app signs nothing, so it sets a fixed one
that APPLICATION_SECRET overrides.)
Not blocking Play’s dispatcher
A provider call blocks its thread. POST /chat waits for its run with AgentPekko.run on ChatExecutionContext, a
CustomExecutionContext over examples.chat.blocking-dispatcher, so Play’s default dispatcher, which serves every
request, is never held:
1
2
3
4
agent
.run(message)(using blocking)
.map(result => Ok(ChatProtocol.result(result).render()).as(ContentTypes.JSON))
.recover { case error => failed(HttpErrors.of(PlayChatEvents.llmError(error))) }
POST /chat/stream needs no dispatcher of its own: AgentPekko.stream runs the turn on a thread of its own. The
controller answers the stream with Ok.chunked:
1
2
3
Ok.chunked(PlayChatEvents(agent, message, heartbeat))
.as(ContentTypes.EVENT_STREAM)
.withHeaders(HeaderNames.CACHE_CONTROL -> "no-cache")
PlayChatEvents writes each event in the SSE format itself, so the keep-alive can be a comment (Play’s
EventSource.Event has no comment), and merges in the heartbeat until the run ends:
1
2
3
4
5
6
7
8
9
10
11
12
val run = agent
.stream(ThreadId(UUID.randomUUID().toString), message)
.collect {
case AgentStreamItem.Event(AgentEvents.TextDelta(d)) => event("delta", ChatProtocol.delta(d.attempt, d.text))
case AgentStreamItem.Event(AgentEvents.ToolCallStarted(t)) =>
event("tool_call", ChatProtocol.toolCall(t.toolCallId, t.tool))
case AgentStreamItem.Done(result) => event("done", ChatProtocol.result(result))
}
.recover { case error => event("error", ChatProtocol.streamError(HttpErrors.of(llmError(error)))) }
run
.merge(Source.tick(heartbeat, heartbeat, KeepAlive), eagerComplete = true)
.mapMaterializedValue(_ => NotUsed)
Errors map to the same statuses as the other apps (HttpErrors.of, below); a failed Future or stream carries its
LLMError in llm4s-pekko’s LLMException.
Errors
An error is answered as {"error": {"type", "message", "recoverable"}}. HttpErrors.of chooses the status from the
error that failed the run, unwrapping the agent runtime’s GraphError.NodeFailed:
| Error | Status |
|---|---|
a body that is not {"message": "<non-blank>"}, ValidationError, InvalidInputError |
400 |
RateLimitError |
429, with Retry-After: the provider’s hint, else RateLimitError.DefaultRetryDelay |
TimeoutError |
504 |
AuthenticationError |
502: the provider refused the server’s credential, which the caller cannot fix |
ConfigurationError |
500 |
any other RecoverableError (network, provider 5xx) |
503 |
| anything else | 502 |
The message is the library’s, which can name the provider; a public service may want to log it and send less.
Testing
The tests need no API key and no network beyond the loopback. ChatRoutesSpec calls the routes in memory against a
ScriptedLLMClient from llm4s-testkit: the JSON answer, the order of the streamed
events, each error’s status, and the stream’s error event. Http4sChatServerSpec starts the server on an ephemeral
port and calls it over real sockets, including the disconnect above. ZioChatRoutesSpec (with Routes.runZIO) and
ZioChatServerSpec check the ZIO HTTP app the same way, and TapirChatRoutesSpec and TapirChatServerSpec the Tapir
app. ChatEndpointsSpec checks the generated OpenAPI document: both endpoints, their schemas, every error status, and
Retry-After on the 429 response.
The Play app is tested with Play’s test helpers. ChatControllerSpec sends FakeRequests through route to an
application built by GuiceApplicationBuilder with the agent overridden: the same checks as ChatRoutesSpec.
PlayChatServerSpec starts the app on Play’s server on an ephemeral port, including the disconnect above, and
ChatModuleSpec checks the wiring: the client from Play’s configuration, the start that fails on a missing provider
section, the blocking dispatcher, and the stop hook that closes the client. PlayGuideSpec checks that every line of
this page’s Play snippets is in the app’s sources.
1
sbt examplesHttp/test
What a real service adds
These apps leave out authentication, rate limiting, request size limits, conversation history (each request starts a new thread in the agent’s in-memory store) and deployment. See Production Deployment for closing the client and the rest of running llm4s in production.