Enhanced Tracing: New Use Cases

This document lists the concrete use cases unlocked by the add-enhanced-tracing implementation (TraceCollectorTracing, InMemoryTraceStore, TraceStore, TraceModel). All of these were impossible before — the previous tracing infrastructure (ConsoleTracing, LangfuseTracing, NoOpTracing) was write-only: events were emitted outward with no return path.

Background

The existing Tracing trait and its implementations are pure fire-and-forget sinks. Once a TraceEvent is emitted, the data is gone from the process. There was no way to read back, query, or act on trace data within the JVM.

The new TraceCollectorTracing + TraceStore layer adds structured, queryable, in-process trace storage as a standard Tracing implementation — composable with any existing backend via TracingComposer.combine().

These types ship in llm4s-observability ("org.llm4s" %% "llm4s-observability"), with Langfuse; the Tracing contract, ConsoleTracing and NoOpTracing stay in llm4s-core. Package names are unchanged. On 0.4.1 and earlier they are part of llm4s-core.

Use Cases

1. In-Process Trace Retrieval by ID

InMemoryTraceStore.getTrace(traceId) and getSpans(traceId) return structured data after agent execution completes.

1
2
3
4
5
6

val store = InMemoryTraceStore()
val spans: Result[List[Span]] = for {
  tracer <- TraceCollectorTracing(store)
  agent  <- Agent.builder("assistant", client).withTools(tools).withTracing(tracer).build()
  _      <- agent.run("query")
} yield store.getSpans(tracer.traceId) // the run's `graph.*` events, as `custom:graph.*` spans

2. Deterministic Agent Testing

Wire TraceCollectorTracing(InMemoryTraceStore()) in tests and assert directly on recorded spans — no mocking of external systems, no console output parsing.

1
2
3
4
5
6
7
8
9
10
11
12

val store  = InMemoryTraceStore()
val tracer = TraceCollectorTracing(store).getOrElse(fail("tracing init failed"))
val agent  = Agent.builder("assistant", client).withTools(tools).withTracing(tracer).build()
  .getOrElse(fail("agent build failed"))
agent.run("query") shouldBe a[Right[_, _]]

// one completed `call-tool` task per tool call the model made
val toolCalls = store.getSpans(tracer.traceId).filter { s =>
  s.name == "custom:graph.task_completed" &&
  s.attributes.get("nodeId").flatMap(_.asString).exists(_.endsWith("/call-tool"))
}
toolCalls should have size 2

3. Time-Range Queries

Retrieve all traces recorded within a time window.

1
2

val query = TraceQuery.withTimeRange(Instant.now.minusSeconds(3600), Instant.now)
val page  = store.queryTraces(query)

4. Status-Based Filtering

Surface failed traces across all runs without touching external systems.

1
2

val query  = TraceQuery.withStatus(SpanStatus.Error(""))
val errors = store.queryTraces(query).traces

5. Metadata-Based Trace Lookup (Experiment Grouping)

Tag traces at creation with arbitrary key-value metadata, then retrieve them by tag. Enables A/B experiment grouping entirely within the process.

1
2
3

val trace   = Trace.start(Map("experiment" -> "v2", "user_id" -> userId))
// ...
val results = store.searchByMetadata("experiment", "v2")  // List[TraceId]

6. Latency Measurement Per Span Kind

Every Span carries startTime and endTime. Filter by SpanKind and compute durations.

1
2
3
4

store.getSpans(traceId)
  .filter(_.kind == SpanKind.ToolCall)
  .flatMap(s => s.endTime.map(e => e.toEpochMilli - s.startTime.toEpochMilli))
  .sum  // total milliseconds spent in tool calls

7. Structured Error Attribution

SpanStatus.Error(message) makes failed spans first-class queryable objects, with their error message, timing, and all typed attributes.

1
2
3

store.getSpans(traceId)
  .collect { case s if s.status.isInstanceOf[SpanStatus.Error] => s }
  .map(s => s.attributes("error_type").asString)

8. Cache Hit Rate Measurement

Cache spans carry hit: Boolean, similarity, and threshold. Compute cache hit rate across a session. Directly supports the semantic caching feature.

1
2
3
4

val cacheSpans = store.getSpans(traceId).filter(_.kind == SpanKind.Cache)
val hitRate    = cacheSpans.count(
  _.attributes.get("hit").flatMap(_.asBoolean).contains(true)
).toDouble / cacheSpans.size

9. Per-Kind Cost and Token Breakdowns

LlmCall spans carry prompt_tokens, completion_tokens; Embedding spans carry input_count; Rag spans carry total_cost_usd. Sum them per operation type.

1
2
3
4

val totalPromptTokens = store.getSpans(traceId)
  .filter(_.kind == SpanKind.LlmCall)
  .flatMap(_.attributes.get("prompt_tokens").flatMap(_.asLong))
  .sum

10. Pluggable Storage Backend

TraceStore is a trait. Implement it backed by PostgreSQL, Redis, DynamoDB, or any store without modifying any tracing code. InMemoryTraceStore is the reference implementation.

1
2
3
4
5

class PostgresTraceStore(ds: DataSource) extends TraceStore {
  override def saveTrace(trace: Trace): Unit = // ...
  override def queryTraces(query: TraceQuery): TracePage = // ...
  // ...
}

11. Span and Trace JSON Round-Trip

TraceModelJson provides both serialization and deserialization for the first time. Enables writing spans to disk and reloading them, or forwarding structured span data to a custom HTTP backend.

1
2
3
4
5
6
7
8

val json   = span.toJson                      // ujson.Value
val parsed = TraceModelJson.parseSpan(json)   // Result[Span]

// Handle the result
parsed match {
  case Right(s)    => println(s.name)
  case Left(error) => println(s"Parse error: ${error.field} — ${error.reason}")
}

12. Trace Deletion and Lifecycle Management

store.deleteTrace(traceId) removes a trace and all its spans atomically. store.clear() wipes everything. Enables test fixture cleanup between cases and on-demand removal of sensitive traces.

1

override def afterEach(): Unit = store.clear()

13. Dataset Construction for LLM Evaluation

Query LlmCall spans filtered by metadata tag to collect structured (input, output, model, token count) records. This is a direct prerequisite for the datasets and evaluators planned in later phases of the Observability & Evaluation initiative.

1
2
3
4
5
6
7

store.searchByMetadata("dataset", "eval-set-v1")
  .flatMap(traceId => store.getSpans(traceId).filter(_.kind == SpanKind.LlmCall))
  .map(s => EvalRecord(
    model          = s.attributes("model").asString.get,
    completionId   = s.attributes("completion_id").asString.get,
    promptTokens   = s.attributes("prompt_tokens").asLong.get
  ))

14. Sub-Span Event Timeline

Span.events: List[SpanEvent] allows point-in-time annotations within a single span, each with a timestamp and typed attributes — without creating child spans.

1
2
3

span
  .withEvent(SpanEvent("cache-lookup-started"))
  .withEvent(SpanEvent("chunks-retrieved", Map("count" -> SpanValue.LongValue(5L))))