org.llm4s.testkit
Members list
Type members
Classlikes
One content part a chat client may or may not send, in the role it is sent in. A provider module's spec names the cases its client supports to ProviderModuleChecks.assertContentParts, which checks that each of those reaches the server and that every other case is refused with an org.llm4s.error.UnsupportedContentPartError before anything is sent.
One content part a chat client may or may not send, in the role it is sent in. A provider module's spec names the cases its client supports to ProviderModuleChecks.assertContentParts, which checks that each of those reaches the server and that every other case is refused with an org.llm4s.error.UnsupportedContentPartError before anything is sent.
Attributes
- Source
- ContentPartContract.scala
- Supertypes
-
trait Enumtrait Serializabletrait Producttrait Equalsclass Objecttrait Matchableclass AnyShow all
The content-part contract: a chat client sends every part its provider takes, and fails the request with an org.llm4s.error.UnsupportedContentPartError naming any other, before anything is sent - never dropping a part or sending it as text. ProviderModuleChecks.assertContentParts runs it.
The content-part contract: a chat client sends every part its provider takes, and fails the request with an org.llm4s.error.UnsupportedContentPartError naming any other, before anything is sent - never dropping a part or sending it as text. ProviderModuleChecks.assertContentParts runs it.
Attributes
- Source
- ContentPartContract.scala
- Supertypes
- Self type
-
ContentPartContract.type
End-to-end check that a credential a provider echoes back in its error body never reaches the LLMError message. A client spec builds the client with key as its credential, points it at the local server assertNotEchoed starts, and makes one call.
End-to-end check that a credential a provider echoes back in its error body never reaches the LLMError message. A client spec builds the client with key as its credential, points it at the local server assertNotEchoed starts, and makes one call.
Attributes
- Source
- CredentialEchoChecks.scala
- Supertypes
- Self type
-
CredentialEchoChecks.type
Proves a provider module's reference.conf binds what its descriptor declares.
Proves a provider module's reference.conf binds what its descriptor declares.
A descriptor's configSpec.apiKeyEnv names the variables its module binds to llm4s.credentials.<id>.apiKey, and the binding itself is a reference.conf line: nothing but a test keeps the two in step. These functions load a section or embeddings block that sets no apiKey of its own - so the shared credential is the only place a key can come from - against every reference.conf on the test classpath and an injected environment (the real one switched off).
ProviderModuleChecks.assertCredentialBindings wraps chatBindings as an assertion; use these directly for the cases it does not cover - an alias, two variables in precedence order, a provider that must ''not'' pick up a neighbour's variable.
Attributes
- Source
- CredentialsRoundTrip.scala
- Supertypes
- Self type
-
CredentialsRoundTrip.type
A local identity provider plus a protected API, for testing workload-identity auth without a network: an RFC 8693 token endpoint (Databricks' /oidc/v1/token), Anthropic's jwt-bearer grant, and OpenAI-format and Anthropic-format endpoints (each streaming when the request asks) that accept only the most recently issued token. Tokens are t1, t2, ... in issue order.
A local identity provider plus a protected API, for testing workload-identity auth without a network: an RFC 8693 token endpoint (Databricks' /oidc/v1/token), Anthropic's jwt-bearer grant, and OpenAI-format and Anthropic-format endpoints (each streaming when the request asks) that accept only the most recently issued token. Tokens are t1, t2, ... in issue order.
Attributes
- Companion
- object
- Source
- FakeTokenExchangeServer.scala
- Supertypes
Attributes
- Companion
- class
- Source
- FakeTokenExchangeServer.scala
- Supertypes
- Self type
A local HTTP server to point a provider client at, so a test exercises the client's real request and response handling - including streaming - without a network or an API key.
A local HTTP server to point a provider client at, so a test exercises the client's real request and response handling - including streaming - without a network or an API key.
It is the JDK's com.sun.net.httpserver.HttpServer on an ephemeral port, so it adds no dependency.
withServer("/v1/chat")(exchange => sendSseResponse(exchange, body)) { baseUrl =>
val client = assertBuildsClient(AcmeProvider, section.copy(baseUrl = Some(BaseUrl(baseUrl))))
assertStreams(client)
}
holdOpen and streamThenHold are handlers that never finish answering, for checking that a client's call can be interrupted (see ProviderModuleChecks.assertCancelsWhenInterrupted). Handlers run on virtual threads, so this needs JDK 21 (see docs/guide/writing-a-provider.md).
Attributes
- Source
- LocalProviderTestServer.scala
- Supertypes
- Self type
The checks every provider module's Llm4s<Name>ModuleSpec makes, as assertions.
The checks every provider module's Llm4s<Name>ModuleSpec makes, as assertions.
A provider module is a dependency, not an edit to llm4s: its Llm4sProviderModule is found through META-INF/services, and a mistake there - a typo in the services file, an object where a class is needed, a second module claiming the same id, a reference.conf binding the wrong variable - fails at a user's runtime, not at your compile time. These checks catch each of those in your own build. They are the ones every provider module in the llm4s repository runs.
Mix the trait into a spec of any ScalaTest style, or call the companion object's methods. A failed check fails the test at the call site with a message saying what was wrong:
class Llm4sAcmeModuleSpec extends AnyWordSpec with Matchers with ProviderModuleChecks:
"llm4s-acme" should {
"register itself through META-INF/services" in assertModule(new Llm4sAcmeModule)
"build a client from a section" in {
assertBuildsClient(AcmeProvider, AcmeProvider.section(...))
assertRefusesForeignConfig(AcmeProvider)
}
"bind ACME_API_KEY" in assertCredentialBindings(AcmeProvider)
}
Chat providers also run assertCancelsWhenInterrupted and assertCancelsStreamWhenInterrupted against LocalProviderTestServer.holdOpen and LocalProviderTestServer.streamThenHold. Those checks run the call on a virtual thread, and the server answers on one, so the testkit needs JDK 21 whatever the provider module itself targets.
Modules are compared by class and descriptors by equality (normally reference equality on an object), so pass the same descriptor instances your module lists.
Attributes
- Companion
- object
- Source
- ProviderModuleChecks.scala
- Supertypes
-
trait Assertionstrait TripleEqualstrait TripleEqualsSupportclass Objecttrait Matchableclass AnyShow all
- Known subtypes
-
object ProviderModuleChecks
The checks, for a spec that would rather call them than mix them in.
The checks, for a spec that would rather call them than mix them in.
Attributes
- Companion
- trait
- Source
- ProviderModuleChecks.scala
- Supertypes
-
trait ProviderModuleCheckstrait Assertionstrait TripleEqualstrait TripleEqualsSupportclass Objecttrait Matchableclass AnyShow all
- Self type
-
ProviderModuleChecks.type
Loads provider configuration the way an application does, with the environment supplied by the test.
Loads provider configuration the way an application does, with the environment supplied by the test.
Each method reads hocon - what the application would write in its application.conf - over every reference.conf on the test classpath, your module's included, and resolves ${?VAR} substitutions against env alone. The real process environment is switched off, so a developer machine with ACME_API_KEY exported cannot make a test pass that fails in CI, and a test can prove your reference.conf binds a variable without setting it.
given ProviderRegistry = ProviderRegistry.default
ProviderTestConfig.loadProvider(
"acme-main",
"""llm4s.providers.acme-main { provider = "acme", model = "acme-large" }""",
Map("ACME_API_KEY" -> "test-key")
) // Right(AcmeConfig(...)), the key taken from llm4s.credentials.acme.apiKey
The provider a section names is resolved through the ProviderRegistry in scope: ProviderRegistry.default for what discovery finds on the classpath, or ProviderRegistry.ofModules(new MyModule) to pin it.
Attributes
- Source
- ProviderTestConfig.scala
- Supertypes
- Self type
-
ProviderTestConfig.type
One request a ScriptedLLMClient received, kept so a test can check what the code under test sent.
One request a ScriptedLLMClient received, kept so a test can check what the code under test sent.
Value parameters
- conversation
-
the conversation the code sent
- options
-
the completion options the code sent
- streamed
-
true when the request came through
streamComplete
Attributes
- Companion
- object
- Source
- RecordedCall.scala
- Supertypes
Attributes
- Companion
- class
- Source
- RecordedCall.scala
- Supertypes
- Self type
-
RecordedCall.type
What one scripted call to a ScriptedLLMClient returns: a text answer, one or more tool calls, or a failure.
What one scripted call to a ScriptedLLMClient returns: a text answer, one or more tool calls, or a failure.
Build one with Reply.text, Reply.toolCall, Reply.toolCalls or Reply.failure, and adjust it with the with* methods:
Reply.text("Paris").withUsage(TokenUsage(promptTokens = 12, completionTokens = 1, totalTokens = 13))
Reply.toolCall("get_weather", """{"city": "Paris"}""")
Reply.failure(RateLimitError("openai"))
A reply that carries a failure is returned as a Left; every other reply becomes a Completion.
Tool-call arguments are written as a JSON string, so the kit's own factories need no JSON library in the test. Reply.toolCalls and withToolCalls take core's ToolCall as it is, for a test that already builds them.
Value parameters
- content
-
the text of the answer; empty for a reply that only calls tools
- failure
-
the error to return instead of a completion
- model
-
the model name reported in the completion
- toolCalls
-
the tool calls the model makes; a call whose
idis empty is given the idcall-<call number>-<position>when the client returns it - usage
-
the token usage reported in the completion
Attributes
- Companion
- object
- Source
- Reply.scala
- Supertypes
An org.llm4s.llmconnect.LLMClient that answers from a script, for testing code that calls LLM4S without a model, a network or an API key.
An org.llm4s.llmconnect.LLMClient that answers from a script, for testing code that calls LLM4S without a model, a network or an API key.
Two ways to script it:
- ScriptedLLMClient.sequence answers the first call with the first reply, the second call with the second, and so on. This suits an agent loop: a tool call, then the final answer.
- ScriptedLLMClient.respondingTo chooses the reply from the conversation, so the answer does not depend on how many calls came before.
val client = ScriptedLLMClient.sequence(
Reply.toolCall("get_weather", """{"city": "Paris"}"""),
Reply.text("It is sunny in Paris.")
)
// ... run the code under test with `client` ...
client.callCount // 2
client.calls.head.lastUserText
A reply the code under test never asked for usually means it took a wrong branch. assertExhausted returns a Left naming every reply of a ScriptedLLMClient.sequence script that no call used, so a test (or a fixture of any test framework) can fail on it; unusedReplies and remaining give the same information as values.
Every request is recorded (calls), whether it came through complete or streamComplete. A call the script has no reply for does not hang or throw: it returns a Left(UnscriptedCallError) that names the call and shows the last message, so a test that makes more calls than it scripted fails with a message that says why.
streamComplete replays the same reply as chunks: the text in pieces of at most chunkSize characters, then each tool call, then a final chunk that carries the finish reason (stop, or tool_calls when the reply calls tools).
The client is thread-safe: calls from several threads are recorded one at a time and each gets its own position in the script. It reports a context window of getContextWindow tokens and reserves getReserveCompletion for the answer, which withContextWindow and withReserveCompletion change.
Attributes
- Companion
- object
- Source
- ScriptedLLMClient.scala
- Supertypes
Attributes
- Companion
- class
- Source
- ScriptedLLMClient.scala
- Supertypes
- Self type
-
ScriptedLLMClient.type
Signed JWTs for tests that need a realistic subject token, using the JDK only.
Signed JWTs for tests that need a realistic subject token, using the JDK only.
Attributes
- Source
- TestJwt.scala
- Supertypes
- Self type
-
TestJwt.type
A chat API's wire format for tool calls and their results, as the tool-result contract reads a request body.
A chat API's wire format for tool calls and their results, as the tool-result contract reads a request body.
Attributes
- Source
- ToolResultContract.scala
- Supertypes
-
trait Enumtrait Serializabletrait Producttrait Equalsclass Objecttrait Matchableclass AnyShow all
One conversation the contract sends: a shape of tool calls and results an agent produces.
One conversation the contract sends: a shape of tool calls and results an agent produces.
Attributes
- Source
- ToolResultContract.scala
- Supertypes
The tool-result contract (design §5.3 of the typed agent runtime): every tool call a request carries has exactly one result, straight after the message that made it, in the provider's own format - and no result answers a call that is not there. OpenAI rejects a request that breaks it; Anthropic rejects one too, which is why an Anthropic client must never be handed such a conversation, rather than drop or rewrite what does not pair.
The tool-result contract (design §5.3 of the typed agent runtime): every tool call a request carries has exactly one result, straight after the message that made it, in the provider's own format - and no result answers a call that is not there. OpenAI rejects a request that breaks it; Anthropic rejects one too, which is why an Anthropic client must never be handed such a conversation, rather than drop or rewrite what does not pair.
violations reads a request body; cases are the conversations an agent's tool loop sends after every outcome of a call - success, a tool error, a rejection or denial, an unknown tool, a refused handoff batch, a handoff - including provider call ids reused across turns. ProviderModuleChecks.assertOneToolResultPerCall sends each case through a client and checks what reaches the server.
Attributes
- Source
- ToolResultContract.scala
- Supertypes
- Self type
-
ToolResultContract.type
The error a ScriptedLLMClient returns for a call its script has no reply for: a call past the end of a ScriptedLLMClient.sequence, or a conversation no case of ScriptedLLMClient.respondingTo matches.
The error a ScriptedLLMClient returns for a call its script has no reply for: a call past the end of a ScriptedLLMClient.sequence, or a conversation no case of ScriptedLLMClient.respondingTo matches.
It is a type of its own, so a test can tell "the code made a call I did not script" apart from every error a real provider returns, and from the errors a test scripts with Reply.failure:
client.complete(conversation, options) match {
case Left(e: UnscriptedCallError) => e.callNumber // 3
case other => ...
}
It is a NonRecoverableError, so a ReliableClient around the scripted client does not retry it.
Value parameters
- callNumber
-
which call had no reply, counting from 1
- lastMessage
-
the last message of the conversation, as
role: "text"cut to 80 characters;Nonewhen the conversation had no messages - reason
-
why there was none, for example
the script has 2 replies
Attributes
- Companion
- object
- Source
- UnscriptedCallError.scala
- Supertypes
-
trait NonRecoverableErrortrait LLMErrortrait Serializabletrait Producttrait Equalsclass Objecttrait Matchableclass AnyShow all
Attributes
- Companion
- class
- Source
- UnscriptedCallError.scala
- Supertypes
- Self type
-
UnscriptedCallError.type
