org.llm4s.agent.metrics

Members list

Type members

Classlikes

@Experimental
object AgentMetrics

The agent runtime's metrics (Stage 5 O2, design 4.59), declared as core Metrics and recorded through any MetricsCollector - PrometheusMetrics, MicrometerMetrics or a MetricsCollector.compose of them - so neither backend depends on llm4s-agent. Two things record them:

The agent runtime's metrics (Stage 5 O2, design 4.59), declared as core Metrics and recorded through any MetricsCollector - PrometheusMetrics, MicrometerMetrics or a MetricsCollector.compose of them - so neither backend depends on llm4s-agent. Two things record them:

  • a MeteredCheckpointer wrapped around the store the runtimes and workers share: runs, claims, commits and child tasks, read from what each call to the store did, so every run is counted once, by the process that ran it;
  • an AgentWorker given the collector: the run queue and the worker's own work.

Every label takes its values from a fixed set named on the metric, except graph (a graph's id) and agent (an agent's id), which the code that builds the agents fixes; backends cap every label's distinct values. No thread, run, entry or tenant id is ever a label, and no metric carries message content (Stage 5 rule 10).

The names below are Micrometer's; a Prometheus backend exposes llm4s.agent.runs.started as llm4s_agent_runs_started_total, a timer with the suffix _seconds, a gauge under its name with dots as underscores.

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
@Experimental
final class MeteredCheckpointer extends Checkpointer

A Checkpointer that records the agent runtime's run, claim, commit and child-task metrics (AgentMetrics, design 4.59) into metrics as it forwards every call to inner unchanged. Give it to every runtime and worker of a process in place of the store:

A Checkpointer that records the agent runtime's run, claim, commit and child-task metrics (AgentMetrics, design 4.59) into metrics as it forwards every call to inner unchanged. Give it to every runtime and worker of a process in place of the store:

val metrics = PrometheusMetrics.create()
for
 store  <- CheckpointerConfigLoader.open()           // llm4s.agent.checkpointer
 queue  <- store.runQueue
 metered = MeteredCheckpointer(store.checkpointer, metrics)
 agent  <- Agent.builder("assistant", client).withRuntime(new GraphRuntime(metered)).build()
 worker <- AgentWorker(queue, metered, Seq(QueuedAgent(agent)), metrics = metrics)
yield worker

It reads what each call did, so it needs no hook in the runtime and counts every run once, in the process whose runtime committed it:

Every other call is forwarded as it is - awaitEventsAfter included, so a store's change notification is kept. A metric that fails to record never changes a call's answer, and no metric carries a thread, run or tenant id or any content (Stage 5 rule 10).

Attributes

Companion
object
Supertypes
trait Checkpointer
class Object
trait Matchable
class Any
@Experimental

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type