AgentMetrics
The agent runtime's metrics (Stage 5 O2, design 4.59), declared as core Metrics and recorded through any MetricsCollector - PrometheusMetrics, MicrometerMetrics or a MetricsCollector.compose of them - so neither backend depends on llm4s-agent. Two things record them:
- a MeteredCheckpointer wrapped around the store the runtimes and workers share: runs, claims, commits and child tasks, read from what each call to the store did, so every run is counted once, by the process that ran it;
- an
AgentWorkergiven the collector: the run queue and the worker's own work.
Every label takes its values from a fixed set named on the metric, except graph (a graph's id) and agent (an agent's id), which the code that builds the agents fixes; backends cap every label's distinct values. No thread, run, entry or tenant id is ever a label, and no metric carries message content (Stage 5 rule 10).
The names below are Micrometer's; a Prometheus backend exposes llm4s.agent.runs.started as llm4s_agent_runs_started_total, a timer with the suffix _seconds, a gauge under its name with dots as underscores.
Attributes
- Graph
-
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
AgentMetrics.type
Members list
Value members
Concrete fields
Claim renewals: result - renewed, lost (another claim replaced it, or the thread was deleted) or failed.
Claim renewals: result - renewed, lost (another claim replaced it, or the thread was deleted) or failed.
Attributes
Claims asked of the store: result - acquired, taken_over (granted over an expired claim nobody released: a dead run's, or one whose process lost the store), busy (a live claim held the thread) or failed. A worker's probe of a thread's claim and its sweep's lock are claims too.
Claims asked of the store: result - acquired, taken_over (granted over an expired claim nobody released: a dead run's, or one whose process lost the store), busy (a live claim held the thread) or failed. A worker's probe of a thread's claim and its sweep's lock are claims too.
Attributes
Commits to the store and how long each took: result - ok, conflict (the parent checkpoint was not the thread's latest), claim_lost (the claim, or the fencing claim, was no longer current) or failed.
Commits to the store and how long each took: result - ok, conflict (the parent checkpoint was not the thread's latest), claim_lost (the claim, or the fencing claim, was no longer current) or failed.
Attributes
Runs a draining worker had, by how they ended: result (finished within the grace period, cancelled and exited, abandoned - not exited, their claims and leases left to expire).
Runs a draining worker had, by how they ended: result (finished within the grace period, cancelled and exited, abandoned - not exited, their claims and leases left to expire).
Attributes
Attempts a worker asked to record: agent, result (recorded, exhausted, thread_deleted).
Attempts a worker asked to record: agent, result (recorded, exhausted, thread_deleted).
Attributes
Entries closed: agent, outcome (done, failed, cancelled, interrupted) and reason: for failed, attempts_exhausted (a dead letter), run_failed or unrunnable; for cancelled, requested or thread_deleted; otherwise none.
Entries closed: agent, outcome (done, failed, cancelled, interrupted) and reason: for failed, attempts_exhausted (a dead letter), run_failed or unrunnable; for cancelled, requested or thread_deleted; otherwise none.
Attributes
Entries a worker gave back without an attempt: agent, reason (busy - a live claim held the thread - or unrunnable - its graph could not take the thread on).
Entries a worker gave back without an attempt: agent, reason (busy - a live claim held the thread - or unrunnable - its graph could not take the thread on).
Attributes
Leases a worker let lapse on purpose, so the entry is tried again a lease later - after a failed run, or a store error reading its thread: agent.
Leases a worker let lapse on purpose, so the entry is tried again a lease later - after a failed run, or a store error reading its thread: agent.
Attributes
Takes that replaced an expired lease (LeasedEntry.expiredLease): agent. An expired lease is a dead or cut-off worker's, or one a worker let lapse after a failed run (QueueLapses).
Takes that replaced an expired lease (LeasedEntry.expiredLease): agent. An expired lease is a dead or cut-off worker's, or one a worker let lapse after a failed run (QueueLapses).
Attributes
Leases a worker lost while it ran the entry, cancelling the run: agent, reason (refused - another worker took the entry or it closed - or unrenewed - no renewal landed for a whole ttl).
Leases a worker lost while it ran the entry, cancelling the run: agent, reason (refused - another worker took the entry or it closed - or unrenewed - no renewal landed for a whole ttl).
Attributes
How long the oldest waiting entry has been open, in seconds, by the store's clock; 0 when none waits: agent.
How long the oldest waiting entry has been open, in seconds, by the store's clock; 0 when none waits: agent.
Attributes
Open entries now, as a worker last sampled them for its agents: agent, state (waiting, leased). Every worker of the fleet reports the same store-wide numbers; aggregate them with max, not sum.
Open entries now, as a worker last sampled them for its agents: agent, state (waiting, leased). Every worker of the fleet reports the same store-wide numbers; aggregate them with max, not sum.
Attributes
What a recorded attempt did (AttemptPlan): agent, plan (start, recover - the input was admitted and its run is continued, the recovery of a dead worker's run - or close).
What a recorded attempt did (AttemptPlan): agent, plan (start, recover - the input was admitted and its run is continued, the recovery of a dead worker's run - or close).
Attributes
How long entries waited from submission to their first recorded attempt, both by the store's clock: agent. A deferral to a busy thread, or an entry queued behind its thread's running one, waits on.
How long entries waited from submission to their first recorded attempt, both by the store's clock: agent. A deferral to a busy thread, or an entry queued behind its thread's running one, waits on.
Attributes
How long runs took, from the event that began the run to its terminal event, both stamped by the process that ran it: graph, status.
How long runs took, from the event that began the run to its terminal event, both stamped by the process that ran it: graph, status.
Attributes
Runs that ended: graph, status (completed, suspended, failed, cancelled, timed_out). A run that stops without a terminal event - its process died, or it lost its claim - ends nothing here; its recovery starts a run.
Runs that ended: graph, status (completed, suspended, failed, cancelled, timed_out). A run that stops without a terminal event - its process died, or it lost its claim - ends nothing here; its recovery starts a run.
Attributes
Runs that began: graph, trigger (start, resume, recover). A recovery is a run continuing an incomplete one.
Runs that began: graph, trigger (start, resume, recover). A recovery is a run continuing an incomplete one.
Attributes
Closed queue entries past their ticket lifetime the retention sweep reclaimed.
Closed queue entries past their ticket lifetime the retention sweep reclaimed.
Attributes
Threads the retention sweep examined.
Threads the retention sweep examined.
Attributes
Retention sweep passes: result (done, skipped - another instance was sweeping - or failed).
Retention sweep passes: result (done, skipped - another instance was sweeping - or failed).
Attributes
Tombstones the retention sweep reclaimed.
Tombstones the retention sweep reclaimed.
Attributes
How long delegated tasks took, from the child's first admission to the call's end: agent, status.
How long delegated tasks took, from the child's first admission to the call's end: agent, status.
Attributes
Delegated tasks that ended (agent.task_ended): agent, status (completed, blocked, step_limit_reached, token_limit_reached, output_invalid, failed).
Delegated tasks that ended (agent.task_ended): agent, status (completed, blocked, step_limit_reached, token_limit_reached, output_invalid, failed).
Attributes
Delegated tasks whose child run was first admitted: agent (the child's agent).
Delegated tasks whose child run was first admitted: agent (the child's agent).