Testing LLM4S Applications

Learn how to test LLM-powered applications effectively without spending money on API calls.

Table of contents

  1. Testing Strategy
  2. Unit Testing with Mock Clients
    1. Basic Mock
    2. Parameterized Mock
  3. Integration Testing with Ollama
    1. Setup
    2. Test Configuration
    3. Integration Test Example
  4. Testing Error Handling
  5. Testing Tool Calling
  6. Testing RAG Applications
  7. CI/CD Testing Strategy
    1. Fast CI Pipeline
    2. Test Categorization
  8. Best Practices
  9. Example Test Suite Structure
  10. Next Steps
  11. Additional Resources

Testing Strategy

Testing LLM applications requires a different approach than traditional software. You need to balance:

  1. Speed - Tests should run fast
  2. Cost - Avoid expensive API calls in CI/CD
  3. Determinism - LLM responses vary, so test behaviors not exact outputs
  4. Coverage - Test error paths, timeouts, and edge cases

Unit Testing with Mock Clients

For pure logic tests, mock the LLM client:

For a ready-made scriptable client - canned replies, tool calls, injected errors and recorded requests - see Testing with the Testkit (llm4s-testkit). The hand-written mocks below still work for a one-off.

Basic Mock 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97

import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model._
import org.llm4s.types.Result
import org.llm4s.error.NetworkError
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers

class WeatherAgentSpec extends AnyFlatSpec with Matchers {

  // Mock client that returns canned responses
  class MockLLMClient extends LLMClient {
    override def getContextWindow(): Int     = 128000
    override def getReserveCompletion(): Int = 4096

    override def complete(
      conversation: Conversation,
      options: CompletionOptions = CompletionOptions()
    ): Result[Completion] = {
      Right(Completion(
        id = "mock-1",
        created = System.currentTimeMillis(),
        content = "The weather in London is 15°C and cloudy.",
        model = "mock-model",
        message = AssistantMessage("The weather in London is 15°C and cloudy."),
        usage = Some(TokenUsage(promptTokens = 10, completionTokens = 15, totalTokens = 25))
      ))
    }

    override def streamComplete(
      conversation: Conversation,
      options: CompletionOptions = CompletionOptions(),
      onChunk: StreamedChunk => Unit
    ): Result[Completion] = {
      val chunks = List(
        StreamedChunk(id = "mock-1", content = Some("The weather"), finishReason = None),
        StreamedChunk(id = "mock-1", content = Some(" is sunny"), finishReason = Some("stop"))
      )
      chunks.foreach(onChunk)
      Right(Completion(
        id = "mock-1",
        created = System.currentTimeMillis(),
        content = "The weather is sunny",
        model = "mock-model",
        message = AssistantMessage("The weather is sunny")
      ))
    }
  }

  // Example agent that uses the LLM client
  class WeatherAgent(client: LLMClient) {
    def run(query: String): Result[Completion] = {
      client.complete(Conversation(Seq(UserMessage(query))))
    }
  }

  "WeatherAgent" should "extract city from user query" in {
    val agent = new WeatherAgent(new MockLLMClient)
    val result = agent.run("What's the weather in London?")
    
    result match {
      case Right(response) =>
        response.content should include("London")
        response.content should include("°C")
      case Left(error) =>
        fail(s"Expected success but got: $error")
    }
  }

  it should "handle errors gracefully" in {
    class FailingMockClient extends LLMClient {
      override def getContextWindow(): Int     = 128000
      override def getReserveCompletion(): Int = 4096

      override def complete(
        conversation: Conversation,
        options: CompletionOptions = CompletionOptions()
      ): Result[Completion] = {
        Left(NetworkError("Connection timeout", None, "https://api.example.com"))
      }
      override def streamComplete(
        conversation: Conversation,
        options: CompletionOptions = CompletionOptions(),
        onChunk: StreamedChunk => Unit
      ): Result[Completion] = {
        Left(NetworkError("Connection timeout", None, "https://api.example.com"))
      }
    }

    val agent = new WeatherAgent(new FailingMockClient)
    val result = agent.run("What's the weather?")
    
    result match {
      case Left(_: NetworkError) => succeed
      case other => fail(s"Expected NetworkError but got: $other")
    }
  }
}

Parameterized Mock

For more complex scenarios:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

import org.llm4s.error.ValidationError
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model._
import org.llm4s.types.Result

class ConfigurableMockClient(responses: Map[String, String]) extends LLMClient {
  override def getContextWindow(): Int     = 128000
  override def getReserveCompletion(): Int = 4096

  override def complete(
    conversation: Conversation,
    options: CompletionOptions = CompletionOptions()
  ): Result[Completion] = {
    val userMessage = conversation.messages
      .collectFirst { case UserMessage(content) => content }
      .getOrElse("")
    
    responses.get(userMessage) match {
      case Some(responseText) =>
        Right(Completion(
          id = "mock-1",
          created = System.currentTimeMillis(),
          content = responseText,
          model = "mock-model",
          message = AssistantMessage(responseText)
        ))
      case None =>
        Left(ValidationError("prompt", s"No mock response for: $userMessage"))
    }
  }

  override def streamComplete(
    conversation: Conversation,
    options: CompletionOptions = CompletionOptions(),
    onChunk: StreamedChunk => Unit
  ): Result[Completion] = {
    complete(conversation, options).map { completion =>
      onChunk(StreamedChunk(id = "mock-1", content = Some(completion.content), finishReason = Some("stop")))
      completion
    }
  }
}

// Usage in tests
val mockResponses = Map(
  "What is 2+2?" -> "2+2 equals 4",
  "What is the capital of France?" -> "The capital of France is Paris"
)

val client = new ConfigurableMockClient(mockResponses)

Integration Testing with Ollama

For integration tests, use Ollama to avoid API costs:

Setup 

1
2
3
4
5
6
7
8

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a small, fast model
ollama pull llama3.2

# Start server
ollama serve

Test Configuration

Put a named provider section in your test resources and select it as the default, with llm4s-ollama as a test dependency (in v0.4.1 and earlier the Ollama provider is inside llm4s-core). See Named provider sections.

1
2
3
4
5
6
7
8
9
10
11
12

# src/test/resources/application.conf
llm4s {
  providers {
    provider = "ollama-test"

    ollama-test {
      provider = "ollama"
      model    = "llama3.2"
      baseUrl  = "http://localhost:11434"   # required for Ollama
    }
  }
}

Integration Test Example 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model._
import org.llm4s.model.ModelRegistryService
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers

class LLMIntegrationSpec extends AnyFlatSpec with Matchers {

  // Only run if Ollama is available
  def ollamaAvailable: Boolean = {
    try {
      val url = new java.net.URL("http://localhost:11434")
      val connection = url.openConnection()
      connection.setConnectTimeout(1000)
      connection.connect()
      true
    } catch {
      case _: Exception => false
    }
  }

  "LLMClient" should "complete basic requests" in {
    assume(ollamaAvailable, "Ollama server not available")

    val result = for {
      config <- Llm4sConfig.defaultProvider()
      registry <- Llm4sConfig.modelRegistryService()
      given ModelRegistryService = registry
      client <- LLMConnect.getClient(config)
      response <- client.complete(
        Conversation(Seq(UserMessage("Say 'hello' and nothing else")))
      )
    } yield response

    result match {
      case Right(response) =>
        response.content.toLowerCase should include("hello")
      case Left(error) =>
        fail(s"Request failed: $error")
    }
  }

  it should "handle streaming responses" in {
    assume(ollamaAvailable, "Ollama server not available")

    var chunks = List.empty[StreamedChunk]
    val result = for {
      config <- Llm4sConfig.defaultProvider()
      registry <- Llm4sConfig.modelRegistryService()
      given ModelRegistryService = registry
      client <- LLMConnect.getClient(config)
      completion <- client.streamComplete(
        Conversation(Seq(UserMessage("Count: 1, 2, 3"))),
        onChunk = chunk => chunks = chunks :+ chunk
      )
    } yield completion

    result match {
      case Right(completion) =>
        chunks should not be empty
        chunks.last.finishReason should be(Some("stop"))
      case Left(error) =>
        fail(s"Streaming failed: $error")
    }
  }
}

Testing Error Handling

Always test error paths:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94

import org.llm4s.agent.Agent
import org.llm4s.agent.graph.GraphError
import org.llm4s.error.{RateLimitError, AuthenticationError, NetworkError}
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model._
import org.llm4s.types.Result
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers

class ErrorHandlingSpec extends AnyFlatSpec with Matchers {

  "Agent" should "handle rate limiting" in {
    class RateLimitedClient extends LLMClient {
      override def getContextWindow(): Int     = 128000
      override def getReserveCompletion(): Int = 4096

      override def complete(
        conversation: Conversation,
        options: CompletionOptions = CompletionOptions()
      ): Result[Completion] = {
        Left(RateLimitError("Rate limit exceeded"))
      }
      override def streamComplete(
        conversation: Conversation,
        options: CompletionOptions = CompletionOptions(),
        onChunk: StreamedChunk => Unit
      ): Result[Completion] = {
        Left(RateLimitError("Rate limit exceeded"))
      }
    }

    // A provider error ends the run as GraphError.NodeFailed, carrying the client's error as its cause
    val result = Agent.builder("test-agent", new RateLimitedClient).build().flatMap(_.run("test query"))

    result match {
      case Left(GraphError.NodeFailed(_, _, _: RateLimitError)) => succeed
      case other => fail(s"Expected RateLimitError but got: $other")
    }
  }

  it should "handle authentication errors" in {
    class UnauthorizedClient extends LLMClient {
      override def getContextWindow(): Int     = 128000
      override def getReserveCompletion(): Int = 4096

      override def complete(
        conversation: Conversation,
        options: CompletionOptions = CompletionOptions()
      ): Result[Completion] = {
        Left(AuthenticationError("openai", "Invalid API key", "401"))
      }
      override def streamComplete(
        conversation: Conversation,
        options: CompletionOptions = CompletionOptions(),
        onChunk: StreamedChunk => Unit
      ): Result[Completion] = {
        Left(AuthenticationError("openai", "Invalid API key", "401"))
      }
    }

    val result = Agent.builder("test-agent", new UnauthorizedClient).build().flatMap(_.run("test"))

    result.isLeft shouldBe true
  }

  it should "handle network timeouts" in {
    class TimeoutClient extends LLMClient {
      override def getContextWindow(): Int     = 128000
      override def getReserveCompletion(): Int = 4096

      override def complete(
        conversation: Conversation,
        options: CompletionOptions = CompletionOptions()
      ): Result[Completion] = {
        Thread.sleep(5000)  // Simulate timeout
        Left(NetworkError("Request timeout", None, "https://api.example.com"))
      }
      override def streamComplete(
        conversation: Conversation,
        options: CompletionOptions = CompletionOptions(),
        onChunk: StreamedChunk => Unit
      ): Result[Completion] = {
        Left(NetworkError("Request timeout", None, "https://api.example.com"))
      }
    }

    val result = Agent.builder("test-agent", new TimeoutClient).build().flatMap(_.run("test"))

    result match {
      case Left(GraphError.NodeFailed(_, _, _: NetworkError)) => succeed
      case other => fail(s"Expected NetworkError but got: $other")
    }
  }
}

Testing Tool Calling

Test that tools are invoked correctly.

Note: This example uses simplified Tool API for clarity. In production, use ToolBuilder and ToolFunction from the org.llm4s.toolapi package. See the Tools documentation for actual API.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75

import org.llm4s.agent.Agent
import org.llm4s.llmconnect.LLMClient
import org.llm4s.llmconnect.model._
import org.llm4s.toolapi.{ Schema, ToolBuilder, ToolRegistry }
import org.llm4s.types.Result
import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers

class ToolCallingSpec extends AnyFlatSpec with Matchers {

  "Agent" should "invoke weather tool" in {
    var toolWasCalled = false
    var capturedCity: Option[String] = None

    // A tool that records the arguments it was called with
    val weatherTool = ToolBuilder[Map[String, Any], String](
      "get_weather",
      "Get weather for a city",
      Schema
        .`object`[Map[String, Any]]("Weather query parameters")
        .withProperty(Schema.property("city", Schema.string("City name")))
    ).withHandler { extractor =>
      toolWasCalled = true
      extractor.getString("city").map { city =>
        capturedCity = Some(city)
        s"Weather in $city: 20°C"
      }
    }.buildSafe()

    // Mock client that calls the tool
    class ToolCallingMock extends LLMClient {
      override def getContextWindow(): Int     = 128000
      override def getReserveCompletion(): Int = 4096

      override def complete(
        conversation: Conversation,
        options: CompletionOptions = CompletionOptions()
      ): Result[Completion] = {
        val message = if (conversation.messages.exists(_.isInstanceOf[ToolMessage])) {
          AssistantMessage("Weather in London: 20°C")
        } else {
          AssistantMessage(
            contentOpt = None,
            toolCalls = Seq(ToolCall("call_1", "get_weather", ujson.Obj("city" -> "London")))
          )
        }
        Right(Completion(
          id = "mock-1",
          created = System.currentTimeMillis(),
          content = message.content,
          model = "mock-model",
          message = message,
          toolCalls = message.toolCalls.toList
        ))
      }
      override def streamComplete(
        conversation: Conversation,
        options: CompletionOptions = CompletionOptions(),
        onChunk: StreamedChunk => Unit
      ): Result[Completion] = {
        complete(conversation, options)
      }
    }

    val outcome = for {
      tool  <- weatherTool
      agent <- Agent.builder("test-agent", new ToolCallingMock).withTools(new ToolRegistry(Seq(tool))).build()
      _     <- agent.run("What's the weather in London?")
    } yield ()

    outcome.isRight shouldBe true
    toolWasCalled shouldBe true
    capturedCity shouldBe Some("London")
  }
}

Testing RAG Applications

Test document retrieval and answer generation separately:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

import org.scalatest.flatspec.AnyFlatSpec
import org.scalatest.matchers.should.Matchers

class RAGSpec extends AnyFlatSpec with Matchers {

  "VectorStore" should "retrieve relevant documents" in {
    val documents = List(
      "Scala is a functional programming language",
      "Python is a dynamically typed language",
      "Java runs on the JVM"
    )

    // Note: This is conceptual pseudocode showing testing patterns. `InMemoryVectorStore`, `embedder` and
    // `RAGPipeline` below are illustrative stand-ins, not llm4s classes with these signatures. The llm4s-rag
    // module has real stores (SQLiteVectorStore, PgVectorStore, QdrantVectorStore).
    val vectorStore = new InMemoryVectorStore()  // Pseudocode - use your vector store
    documents.foreach(doc => vectorStore.add(doc, embedder.embed(doc)))  // embedder is conceptual

    val results = vectorStore.search("functional programming", topK = 1)
    
    results.head should include("Scala")
  }

  "RAG pipeline" should "include context in LLM prompt" in {
    class RAGMockClient extends LLMClient {
      override def getContextWindow(): Int     = 128000
      override def getReserveCompletion(): Int = 4096

      var lastPrompt: Option[String] = None

      override def complete(
        conversation: Conversation,
        options: CompletionOptions = CompletionOptions()
      ): Result[Completion] = {
        lastPrompt = conversation.messages
          .collectFirst { case UserMessage(content) => content }
        Right(Completion(
          id = "mock-1",
          created = System.currentTimeMillis(),
          content = "Based on the context, Scala is functional.",
          model = "mock-model",
          message = AssistantMessage("Based on the context, Scala is functional.")
        ))
      }

      override def streamComplete(
        conversation: Conversation,
        options: CompletionOptions = CompletionOptions(),
        onChunk: StreamedChunk => Unit
      ): Result[Completion] = {
        complete(conversation, options)
      }
    }

    val mockClient = new RAGMockClient
    val rag = new RAGPipeline(mockClient, vectorStore, embedder)
    rag.query("What is Scala?")

    mockClient.lastPrompt.get should include("context")
    mockClient.lastPrompt.get should include("functional")
  }
}

CI/CD Testing Strategy

Fast CI Pipeline 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# .github/workflows/test.yml
name: Test

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Scala
        uses: olafurpg/setup-scala@v14
        with:
          java-version: 21
      
      - name: Install Ollama
        run: |
          curl -fsSL https://ollama.com/install.sh | sh
          ollama serve &
          sleep 5
          ollama pull llama3.2
      
      - name: Run unit tests (fast)
        run: sbt "testOnly *UnitSpec"
      
      - name: Run integration tests (with Ollama)
        # The provider comes from src/test/resources/application.conf (the ollama-test section
        # above); no environment variable selects it.
        run: sbt "testOnly *IntegrationSpec"
      
      # Skip expensive tests in CI
      - name: Run full test suite
        run: sbt test
        if: github.event_name == 'push' && github.ref == 'refs/heads/main'

Test Categorization 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

// Tag tests by speed/cost
import org.scalatest.Tag
import org.scalatest.flatspec.AnyFlatSpec

object UnitTest extends Tag("UnitTest")
object IntegrationTest extends Tag("IntegrationTest")
object ExpensiveTest extends Tag("ExpensiveTest")

class FastSpec extends AnyFlatSpec {
  "Fast unit test" should "run in CI" taggedAs UnitTest in {
    // Mock-based test
  }
}

class SlowSpec extends AnyFlatSpec {
  "Expensive test" should "run manually" taggedAs ExpensiveTest in {
    // Uses real OpenAI API
  }
}

Run specific test categories:

1
2
3
4
5

# Fast tests only
sbt "testOnly * -- -n UnitTest"

# Everything except expensive tests
sbt "testOnly * -- -l ExpensiveTest"

Best Practices

  1. ✅ Mock by default: Use mock clients for unit tests
  2. ✅ Ollama for integration: Free and fast enough for CI
  3. ✅ Test behaviors, not outputs: LLM responses vary, so test that tools are called, documents are retrieved, etc.
  4. ✅ Use deterministic models when possible: Set temperature=0 for more predictable outputs
  5. ✅ Separate concerns: Test tool logic independently from LLM integration
  6. ✅ Tag expensive tests: Don’t run them in every CI build
  7. ✅ Use smaller models in CI: llama3.2 is fast and free via Ollama

Example Test Suite Structure 

1
2
3
4
5
6
7
8
9
10
11

src/test/scala/
├── unit/
│   ├── ToolSpec.scala           # Pure logic tests (mocked)
│   ├── ConfigSpec.scala         # Configuration parsing
│   └── ErrorHandlingSpec.scala  # Error path tests
├── integration/
│   ├── LLMClientSpec.scala      # Real LLM calls (Ollama)
│   ├── AgentSpec.scala          # End-to-end agent tests
│   └── RAGSpec.scala            # RAG pipeline tests
└── resources/
    └── application.conf         # Test config (Ollama)

Next Steps

Additional Resources