org.llm4s.testkit

Members list

Type members

Classlikes

enum ContentPartCase(val role: MessageRole)

One content part a chat client may or may not send, in the role it is sent in. A provider module's spec names the cases its client supports to ProviderModuleChecks.assertContentParts, which checks that each of those reaches the server and that every other case is refused with an org.llm4s.error.UnsupportedContentPartError before anything is sent.

One content part a chat client may or may not send, in the role it is sent in. A provider module's spec names the cases its client supports to ProviderModuleChecks.assertContentParts, which checks that each of those reaches the server and that every other case is refused with an org.llm4s.error.UnsupportedContentPartError before anything is sent.

Attributes

Source
ContentPartContract.scala
Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

The content-part contract: a chat client sends every part its provider takes, and fails the request with an org.llm4s.error.UnsupportedContentPartError naming any other, before anything is sent - never dropping a part or sending it as text. ProviderModuleChecks.assertContentParts runs it.

The content-part contract: a chat client sends every part its provider takes, and fails the request with an org.llm4s.error.UnsupportedContentPartError naming any other, before anything is sent - never dropping a part or sending it as text. ProviderModuleChecks.assertContentParts runs it.

Attributes

Source
ContentPartContract.scala
Supertypes
class Object
trait Matchable
class Any
Self type

End-to-end check that a credential a provider echoes back in its error body never reaches the LLMError message. A client spec builds the client with key as its credential, points it at the local server assertNotEchoed starts, and makes one call.

End-to-end check that a credential a provider echoes back in its error body never reaches the LLMError message. A client spec builds the client with key as its credential, points it at the local server assertNotEchoed starts, and makes one call.

Attributes

Source
CredentialEchoChecks.scala
Supertypes
class Object
trait Matchable
class Any
Self type

Proves a provider module's reference.conf binds what its descriptor declares.

Proves a provider module's reference.conf binds what its descriptor declares.

A descriptor's configSpec.apiKeyEnv names the variables its module binds to llm4s.credentials.<id>.apiKey, and the binding itself is a reference.conf line: nothing but a test keeps the two in step. These functions load a section or embeddings block that sets no apiKey of its own - so the shared credential is the only place a key can come from - against every reference.conf on the test classpath and an injected environment (the real one switched off).

ProviderModuleChecks.assertCredentialBindings wraps chatBindings as an assertion; use these directly for the cases it does not cover - an alias, two variables in precedence order, a provider that must ''not'' pick up a neighbour's variable.

Attributes

Source
CredentialsRoundTrip.scala
Supertypes
class Object
trait Matchable
class Any
Self type

A local identity provider plus a protected API, for testing workload-identity auth without a network: an RFC 8693 token endpoint (Databricks' /oidc/v1/token), Anthropic's jwt-bearer grant, and OpenAI-format and Anthropic-format endpoints (each streaming when the request asks) that accept only the most recently issued token. Tokens are t1, t2, ... in issue order.

A local identity provider plus a protected API, for testing workload-identity auth without a network: an RFC 8693 token endpoint (Databricks' /oidc/v1/token), Anthropic's jwt-bearer grant, and OpenAI-format and Anthropic-format endpoints (each streaming when the request asks) that accept only the most recently issued token. Tokens are t1, t2, ... in issue order.

Attributes

Companion
object
Source
FakeTokenExchangeServer.scala
Supertypes
class Object
trait Matchable
class Any

Attributes

Companion
class
Source
FakeTokenExchangeServer.scala
Supertypes
class Object
trait Matchable
class Any
Self type

A local HTTP server to point a provider client at, so a test exercises the client's real request and response handling - including streaming - without a network or an API key.

A local HTTP server to point a provider client at, so a test exercises the client's real request and response handling - including streaming - without a network or an API key.

It is the JDK's com.sun.net.httpserver.HttpServer on an ephemeral port, so it adds no dependency.

withServer("/v1/chat")(exchange => sendSseResponse(exchange, body)) { baseUrl =>
 val client = assertBuildsClient(AcmeProvider, section.copy(baseUrl = Some(BaseUrl(baseUrl))))
 assertStreams(client)
}

holdOpen and streamThenHold are handlers that never finish answering, for checking that a client's call can be interrupted (see ProviderModuleChecks.assertCancelsWhenInterrupted). Handlers run on virtual threads, so this needs JDK 21 (see docs/guide/writing-a-provider.md).

Attributes

Source
LocalProviderTestServer.scala
Supertypes
class Object
trait Matchable
class Any
Self type
trait ProviderModuleChecks extends Assertions

The checks every provider module's Llm4s<Name>ModuleSpec makes, as assertions.

The checks every provider module's Llm4s<Name>ModuleSpec makes, as assertions.

A provider module is a dependency, not an edit to llm4s: its Llm4sProviderModule is found through META-INF/services, and a mistake there - a typo in the services file, an object where a class is needed, a second module claiming the same id, a reference.conf binding the wrong variable - fails at a user's runtime, not at your compile time. These checks catch each of those in your own build. They are the ones every provider module in the llm4s repository runs.

Mix the trait into a spec of any ScalaTest style, or call the companion object's methods. A failed check fails the test at the call site with a message saying what was wrong:

class Llm4sAcmeModuleSpec extends AnyWordSpec with Matchers with ProviderModuleChecks:

 "llm4s-acme" should {
   "register itself through META-INF/services" in assertModule(new Llm4sAcmeModule)

   "build a client from a section" in {
     assertBuildsClient(AcmeProvider, AcmeProvider.section(...))
     assertRefusesForeignConfig(AcmeProvider)
   }

   "bind ACME_API_KEY" in assertCredentialBindings(AcmeProvider)
 }

Chat providers also run assertCancelsWhenInterrupted and assertCancelsStreamWhenInterrupted against LocalProviderTestServer.holdOpen and LocalProviderTestServer.streamThenHold. Those checks run the call on a virtual thread, and the server answers on one, so the testkit needs JDK 21 whatever the provider module itself targets.

Modules are compared by class and descriptors by equality (normally reference equality on an object), so pass the same descriptor instances your module lists.

Attributes

Companion
object
Source
ProviderModuleChecks.scala
Supertypes
trait Assertions
trait TripleEquals
trait TripleEqualsSupport
class Object
trait Matchable
class Any
Show all
Known subtypes

The checks, for a spec that would rather call them than mix them in.

The checks, for a spec that would rather call them than mix them in.

Attributes

Companion
trait
Source
ProviderModuleChecks.scala
Supertypes
trait Assertions
trait TripleEquals
trait TripleEqualsSupport
class Object
trait Matchable
class Any
Show all
Self type

Loads provider configuration the way an application does, with the environment supplied by the test.

Loads provider configuration the way an application does, with the environment supplied by the test.

Each method reads hocon - what the application would write in its application.conf - over every reference.conf on the test classpath, your module's included, and resolves ${?VAR} substitutions against env alone. The real process environment is switched off, so a developer machine with ACME_API_KEY exported cannot make a test pass that fails in CI, and a test can prove your reference.conf binds a variable without setting it.

given ProviderRegistry = ProviderRegistry.default

ProviderTestConfig.loadProvider(
 "acme-main",
 """llm4s.providers.acme-main { provider = "acme", model = "acme-large" }""",
 Map("ACME_API_KEY" -> "test-key")
) // Right(AcmeConfig(...)), the key taken from llm4s.credentials.acme.apiKey

The provider a section names is resolved through the ProviderRegistry in scope: ProviderRegistry.default for what discovery finds on the classpath, or ProviderRegistry.ofModules(new MyModule) to pin it.

Attributes

Source
ProviderTestConfig.scala
Supertypes
class Object
trait Matchable
class Any
Self type
final case class RecordedCall

One request a ScriptedLLMClient received, kept so a test can check what the code under test sent.

One request a ScriptedLLMClient received, kept so a test can check what the code under test sent.

Value parameters

conversation

the conversation the code sent

options

the completion options the code sent

streamed

true when the request came through streamComplete

Attributes

Companion
object
Source
RecordedCall.scala
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object RecordedCall

Attributes

Companion
class
Source
RecordedCall.scala
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
final case class Reply

What one scripted call to a ScriptedLLMClient returns: a text answer, one or more tool calls, or a failure.

What one scripted call to a ScriptedLLMClient returns: a text answer, one or more tool calls, or a failure.

Build one with Reply.text, Reply.toolCall, Reply.toolCalls or Reply.failure, and adjust it with the with* methods:

Reply.text("Paris").withUsage(TokenUsage(promptTokens = 12, completionTokens = 1, totalTokens = 13))
Reply.toolCall("get_weather", """{"city": "Paris"}""")
Reply.failure(RateLimitError("openai"))

A reply that carries a failure is returned as a Left; every other reply becomes a Completion.

Tool-call arguments are written as a JSON string, so the kit's own factories need no JSON library in the test. Reply.toolCalls and withToolCalls take core's ToolCall as it is, for a test that already builds them.

Value parameters

content

the text of the answer; empty for a reply that only calls tools

failure

the error to return instead of a completion

model

the model name reported in the completion

toolCalls

the tool calls the model makes; a call whose id is empty is given the id call-<call number>-<position> when the client returns it

usage

the token usage reported in the completion

Attributes

Companion
object
Source
Reply.scala
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object Reply

Attributes

Companion
class
Source
Reply.scala
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
Reply.type
final class ScriptedLLMClient extends LLMClient

An org.llm4s.llmconnect.LLMClient that answers from a script, for testing code that calls LLM4S without a model, a network or an API key.

An org.llm4s.llmconnect.LLMClient that answers from a script, for testing code that calls LLM4S without a model, a network or an API key.

Two ways to script it:

  • ScriptedLLMClient.sequence answers the first call with the first reply, the second call with the second, and so on. This suits an agent loop: a tool call, then the final answer.
  • ScriptedLLMClient.respondingTo chooses the reply from the conversation, so the answer does not depend on how many calls came before.
val client = ScriptedLLMClient.sequence(
 Reply.toolCall("get_weather", """{"city": "Paris"}"""),
 Reply.text("It is sunny in Paris.")
)
// ... run the code under test with `client` ...
client.callCount          // 2
client.calls.head.lastUserText

A reply the code under test never asked for usually means it took a wrong branch. assertExhausted returns a Left naming every reply of a ScriptedLLMClient.sequence script that no call used, so a test (or a fixture of any test framework) can fail on it; unusedReplies and remaining give the same information as values.

Every request is recorded (calls), whether it came through complete or streamComplete. A call the script has no reply for does not hang or throw: it returns a Left(UnscriptedCallError) that names the call and shows the last message, so a test that makes more calls than it scripted fails with a message that says why.

streamComplete replays the same reply as chunks: the text in pieces of at most chunkSize characters, then each tool call, then a final chunk that carries the finish reason (stop, or tool_calls when the reply calls tools).

The client is thread-safe: calls from several threads are recorded one at a time and each gets its own position in the script. It reports a context window of getContextWindow tokens and reserves getReserveCompletion for the answer, which withContextWindow and withReserveCompletion change.

Attributes

Companion
object
Source
ScriptedLLMClient.scala
Supertypes
trait LLMClient
class Object
trait Matchable
class Any

Attributes

Companion
class
Source
ScriptedLLMClient.scala
Supertypes
class Object
trait Matchable
class Any
Self type
object TestJwt

Signed JWTs for tests that need a realistic subject token, using the JDK only.

Signed JWTs for tests that need a realistic subject token, using the JDK only.

Attributes

Source
TestJwt.scala
Supertypes
class Object
trait Matchable
class Any
Self type
TestJwt.type

A chat API's wire format for tool calls and their results, as the tool-result contract reads a request body.

A chat API's wire format for tool calls and their results, as the tool-result contract reads a request body.

Attributes

Source
ToolResultContract.scala
Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
final case class ToolResultCase(name: String, conversation: Conversation)

One conversation the contract sends: a shape of tool calls and results an agent produces.

One conversation the contract sends: a shape of tool calls and results an agent produces.

Attributes

Source
ToolResultContract.scala
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

The tool-result contract (design §5.3 of the typed agent runtime): every tool call a request carries has exactly one result, straight after the message that made it, in the provider's own format - and no result answers a call that is not there. OpenAI rejects a request that breaks it; Anthropic rejects one too, which is why an Anthropic client must never be handed such a conversation, rather than drop or rewrite what does not pair.

The tool-result contract (design §5.3 of the typed agent runtime): every tool call a request carries has exactly one result, straight after the message that made it, in the provider's own format - and no result answers a call that is not there. OpenAI rejects a request that breaks it; Anthropic rejects one too, which is why an Anthropic client must never be handed such a conversation, rather than drop or rewrite what does not pair.

violations reads a request body; cases are the conversations an agent's tool loop sends after every outcome of a call - success, a tool error, a rejection or denial, an unknown tool, a refused handoff batch, a handoff - including provider call ids reused across turns. ProviderModuleChecks.assertOneToolResultPerCall sends each case through a client and checks what reaches the server.

Attributes

Source
ToolResultContract.scala
Supertypes
class Object
trait Matchable
class Any
Self type

The error a ScriptedLLMClient returns for a call its script has no reply for: a call past the end of a ScriptedLLMClient.sequence, or a conversation no case of ScriptedLLMClient.respondingTo matches.

The error a ScriptedLLMClient returns for a call its script has no reply for: a call past the end of a ScriptedLLMClient.sequence, or a conversation no case of ScriptedLLMClient.respondingTo matches.

It is a type of its own, so a test can tell "the code made a call I did not script" apart from every error a real provider returns, and from the errors a test scripts with Reply.failure:

client.complete(conversation, options) match {
 case Left(e: UnscriptedCallError) => e.callNumber   // 3
 case other                        => ...
}

It is a NonRecoverableError, so a ReliableClient around the scripted client does not retry it.

Value parameters

callNumber

which call had no reply, counting from 1

lastMessage

the last message of the conversation, as role: "text" cut to 80 characters; None when the conversation had no messages

reason

why there was none, for example the script has 2 replies

Attributes

Companion
object
Source
UnscriptedCallError.scala
Supertypes
trait LLMError
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

Attributes

Companion
class
Source
UnscriptedCallError.scala
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type