org.llm4s.agent.service

Members list

Type members

Classlikes

@Experimental
final case class Attempt(runId: RunId, startedAt: Instant, holder: String)

One attempt at an entry's run (Stage 5 rule 4): the RunId the attempt runs as, recorded durably before the attempt touches the thread, when the store recorded it, by its clock, and the lease holder that recorded it.

One attempt at an entry's run (Stage 5 rule 4): the RunId the attempt runs as, recorded durably before the attempt touches the thread, when the store recorded it, by its clock, and the lease holder that recorded it.

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental

What an attempt at a queue entry does to its thread (Stage 5 rule 4, design 4.55), decided after RunQueue.recordAttempt recorded the attempt's RunId and before the attempt touches the thread:

What an attempt at a queue entry does to its thread (Stage 5 rule 4, design 4.55), decided after RunQueue.recordAttempt recorded the attempt's RunId and before the attempt touches the thread:

  • Start: none of the entry's runs committed - its input was never admitted, an earlier attempt having died before its first commit - so the attempt admits it: run (or the typed agent's run) for a start, resume for a resume.
  • Recover: the attempt recovers the thread - its entry's admitted run is the thread's latest and incomplete (Running), so a worker died mid-run; or the entry is a queued recover, whose attempts recover the thread until one of them commits (the owner's explicit transfer of the thread's recovery to the queue).
  • Close: the entry's input was admitted and the thread has moved on - its run reached rest, or the thread moved on through a direct recover, which applied the input once - so the attempt closes the entry with outcome, running nothing. That includes an admitted entry (QueueEntry.admitted) whose checkpoints the lookup no longer finds: a direct run's blocked turn retracted them (Commit.retractTurn), so the input must not run again.

Attributes

Companion
object
Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental
object AttemptPlan

Attributes

Companion
enum
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
@Experimental

What RunQueue.recordAttempt did (Stage 5 rule 4).

What RunQueue.recordAttempt did (Stage 5 rule 4).

  • Recorded: the attempt's RunId is recorded, the last of entry.attempts; the attempt may now touch the thread, classifying it first (AttemptPlan).
  • Exhausted: the entry has made maxAttempts attempts, so none is recorded. The worker settles the thread if the entry's input was admitted, then closes the entry Failed(AttemptsExhausted) - a dead letter.
  • ThreadDeleted: the thread was deleted since the entry was submitted; the store closed the entry Cancelled(ThreadDeleted) and recorded nothing, so nothing runs.

Attributes

Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental

How a submission treats a thread that already has an open entry or a live run (design 4.55; W3 #2136 gives each its full behaviour). Reject, the default, refuses the submission with GraphError.ThreadBusy, as a direct run on a busy thread is refused (§5.3). Enqueue waits behind the thread's open entries, in submission order. Interrupt goes ahead of the thread's queued entries, which keep their order behind it; of two Interrupt entries the later goes first and the earlier is closed EntryOutcome.Interrupted before it runs (Stage 5 rule 5). Stopping the thread's running entry at its next safe point is W3's.

How a submission treats a thread that already has an open entry or a live run (design 4.55; W3 #2136 gives each its full behaviour). Reject, the default, refuses the submission with GraphError.ThreadBusy, as a direct run on a busy thread is refused (§5.3). Enqueue waits behind the thread's open entries, in submission order. Interrupt goes ahead of the thread's queued entries, which keep their order behind it; of two Interrupt entries the later goes first and the earlier is closed EntryOutcome.Interrupted before it runs (Stage 5 rule 5). Stopping the thread's running entry at its next safe point is W3's.

Attributes

Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental

Why an entry closed cancelled (design 4.55). ThreadDeleted: the thread was deleted after the entry was submitted, so the entry ran nothing (a deleted thread is never resurrected, rule 4). Requested: a cancel request stopped it (W3), with the requester's note.

Why an entry closed cancelled (design 4.55). ThreadDeleted: the thread was deleted after the entry was submitted, so the entry ran nothing (a deleted thread is never resurrected, rule 4). Requested: a cancel request stopped it (W3), with the requester's note.

Attributes

Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental
object EntryId

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
EntryId.type
@Experimental

How an entry closed: its terminal state (design 4.55).

How an entry closed: its terminal state (design 4.55).

  • Done: the entry's run reached rest - completed or suspended - or the thread had moved on past the entry's admitted input (rule 4), at checkpoint checkpointId with status, if the thread had one.
  • Failed: see FailureReason.
  • Cancelled: see CancelReason.
  • Interrupted: an Interrupt entry by superseded it - before it ran, for a queued Interrupt entry, or at its next safe point (W3). It is not retried.

Attributes

Companion
object
Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental
object EntryOutcome

Attributes

Companion
enum
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
@Experimental

Why an entry closed failed (design 4.55).

Why an entry closed failed (design 4.55).

  • AttemptsExhausted: attempts attempts ran and none finished (a dead letter). If the input was admitted, the worker that found the attempts exhausted settled the thread first, so later entries start.
  • RunFailed: the run ended in failure - a terminal Failed checkpoint, or a refusal such as NotSuspended for a resume of a thread that is not waiting - with the error's message.
  • Unrunnable: no worker could run the entry (no worker has its agent at a graph version that can restore its thread, W2), with why.

Attributes

Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental
final case class FollowedRun(entry: QueueEntry, events: Vector[EventRecord], next: Long, finished: Boolean)

A page of a queued run's events (RunSubmitter.follow): events, those of the entry's attempts among the thread's events after the cursor the caller passed, ascending; next, the cursor to pass next (the last thread event read, whoever's it was); and entry as it was read after the events, so its attempts cover them. finished is true once the entry is closed and every event up to its close (QueueEntry.closedAtSeq) has been read: nothing more will come, however busy the thread stays with later runs.

A page of a queued run's events (RunSubmitter.follow): events, those of the entry's attempts among the thread's events after the cursor the caller passed, ascending; next, the cursor to pass next (the last thread event read, whoever's it was); and entry as it was read after the events, so its attempts cover them. finished is true once the entry is closed and every event up to its close (QueueEntry.closedAtSeq) has been read: nothing more will come, however busy the thread stays with later runs.

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental
final class InMemoryRunQueue extends RunQueue

The RunQueue of an InMemoryCheckpointer (InMemoryCheckpointer.runQueue): entries in memory, under the checkpointer's own lock, leases judged and closes stamped by its clock. It shares the checkpointer's lock so the checkpointer's deleteThread marks its open entries, and its prune reads their runs, in one step. awaitClosed wakes on closes rather than polling. Its calls take only each other's brief turn at the lock, so each returns at once.

The RunQueue of an InMemoryCheckpointer (InMemoryCheckpointer.runQueue): entries in memory, under the checkpointer's own lock, leases judged and closes stamped by its clock. It shares the checkpointer's lock so the checkpointer's deleteThread marks its open entries, and its prune reads their runs, in one step. awaitClosed wakes on closes rather than polling. Its calls take only each other's brief turn at the lock, so each returns at once.

Attributes

Supertypes
trait RunQueue
class Object
trait Matchable
class Any
@Experimental
final case class Lease(entryId: EntryId, token: LeaseToken, holder: String, expiresAt: Instant)

A lease on an open entry, as RunQueue.take grants it: the worker holding it may record attempts at the entry, renew the lease and close the entry, for as long as no other lease replaced it - expiry, by the store's clock, only lets another worker take the entry over. token fences every call made under it.

A lease on an open entry, as RunQueue.take grants it: the worker holding it may record attempts at the entry, renew the lease and close the entry, for as long as no other lease replaced it - expiry, by the store's clock, only lets another worker take the entry over. token fences every call made under it.

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental
final case class LeasePolicy

How a worker holds its queue entries' leases (design 4.55): each lease lasts ttl from when the store grants or renews it, by the store's clock, and is renewed every renewEvery - with the run's claim, W2 - so an entry whose worker died comes back to the queue at most ttl after its last renewal. Keep renewEvery at most a third of ttl, as the default does.

How a worker holds its queue entries' leases (design 4.55): each lease lasts ttl from when the store grants or renews it, by the store's clock, and is renewed every renewEvery - with the run's claim, W2 - so an entry whose worker died comes back to the queue at most ttl after its last renewal. Keep renewEvery at most a third of ttl, as the default does.

Both durations must be positive and renewEvery shorter than ttl: apply and the with* setters throw IllegalArgumentException otherwise, and LeasePolicy.of returns a ValidationError.

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental
object LeasePolicy

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
@Experimental
object LeaseToken

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
LeaseToken.type
@Experimental
final case class LeasedEntry(entry: QueueEntry, lease: Lease)

An entry RunQueue.take leased, with its lease.

An entry RunQueue.take leased, with its lease.

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental
final case class QueueEntry

A run queue entry, as a store reads it (design 4.55): one submitted run on a thread, durable, with the attempts made at it and, once closed, its EntryOutcome.

A run queue entry, as a store reads it (design 4.55): one submitted run on a thread, durable, with the attempts made at it and, once closed, its EntryOutcome.

An entry is open until it is closed, and only an open entry is ever taken or reclaimed-proof: a closed one is kept for the ticket lifetime after closedAt, then reclaimed. While open, position is how many of the thread's open entries run before it (0: it runs next, or is running), and leaseHolder and leaseExpiresAt say which worker holds it, if one does - a lease that has expired by the store's clock may be taken over.

Value parameters

admitted

whether a run of one of attempts has committed a checkpoint on the thread - set by the store in that commit's own transaction, and kept when the checkpoint later leaves the history (a direct run's Commit.retractTurn removes it), so the input is known to be admitted even when the run-id lookup no longer finds it (AttemptPlan)

attempts

the attempts recorded, oldest first, each before it touched the thread

closedAtSeq

the number of the thread's last event when the entry closed (0 for none): every event of the entry's runs is at or below it, so following the entry stops there (RunSubmitter.follow)

maxAttempts

the most attempts the entry may make, fixed at submission

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental
object QueueEntry

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
QueueEntry.type
@Experimental
sealed trait QueueError extends LLMError

Errors a RunQueue answers with (design 4.55). Every case is non-recoverable: retrying the same call changes nothing.

Errors a RunQueue answers with (design 4.55). Every case is non-recoverable: retrying the same call changes nothing.

Attributes

Companion
object
Supertypes
trait LLMError
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
Known subtypes
@Experimental
object QueueError

Attributes

Companion
trait
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
QueueError.type
@Experimental

What a queued run does to its thread (design 4.55), in the JSON form the entry stores. The queue runs Agents and TypedAgent[I, O]s, the two entry points whose inputs have codecs; a raw CompiledGraph is not queued.

What a queued run does to its thread (design 4.55), in the JSON form the entry stores. The queue runs Agents and TypedAgent[I, O]s, the two entry points whose inputs have codecs; a raw CompiledGraph is not queued.

  • Start: a new turn with input - an AgentInput for an Agent (QueuedInput.query), an I for a TypedAgent[I, O] (QueuedInput.typed).
  • Resume: answers to the thread's pending interrupts, as Agent.resume takes them.
  • Recover: continue the thread's failed or interrupted run, as Agent.recover does. A queued recover is the thread owner's explicit transfer of the thread's recovery to the queue: its first attempt recovers the thread whatever ran on it before (rule 4 otherwise forbids a worker from adopting a run started outside the queue).

The input is the caller's to store: the queue keeps it in its entry until the entry is reclaimed, but no metric, trace attribute or log line carries it (Stage 5 rule 10).

Attributes

Companion
object
Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental
object QueuedInput

Attributes

Companion
enum
Supertypes
trait Sum
trait Mirror
class Object
trait Matchable
class Any
Self type
@Experimental
trait RunQueue

The durable run queue (Stage 5 W1, design 4.55): submitted runs, one entry each, that workers on any machine take, run and close. A store's queue lives in the same database as its checkpoints - InMemoryCheckpointer.runQueue, SqliteCheckpointer.runQueue, PostgresCheckpointer.runQueue - but is a separate SPI: no transaction spans an entry and a commit. Correctness rests on rule 4 instead - every attempt has its own RunId, recorded here before the attempt touches the thread, and an attempt classifies the thread by those ids (Checkpointer.latestOfRuns, AttemptPlan) - so an input is applied at most once however many attempts run.

The durable run queue (Stage 5 W1, design 4.55): submitted runs, one entry each, that workers on any machine take, run and close. A store's queue lives in the same database as its checkpoints - InMemoryCheckpointer.runQueue, SqliteCheckpointer.runQueue, PostgresCheckpointer.runQueue - but is a separate SPI: no transaction spans an entry and a commit. Correctness rests on rule 4 instead - every attempt has its own RunId, recorded here before the attempt touches the thread, and an attempt classifies the thread by those ids (Checkpointer.latestOfRuns, AttemptPlan) - so an input is applied at most once however many attempts run.

Implementations must be safe to call from several threads and processes, and must:

  • '''submit''' an entry atomically: look up its SubmissionKey, if any, among the tenant's entries and insert only if there is none, so a retried submit gets the first one's ticket (Ticket.existing) and a different submission under a held key is refused (QueueError.SubmissionKeyReused); refuse a Reject submission to a thread with an open entry with GraphError.ThreadBusy naming no holder; close every queued Interrupt entry of the thread that has recorded no attempt Interrupted(by = the new entry) when an Interrupt entry is submitted; and issue entry ids in submission order, never reused;
  • '''order''' each thread's open entries (rule 5): an entry that has recorded an attempt first, then Interrupt entries, then the rest, each by id; only a thread's first entry in that order - its head - is ever taken, and only while no other entry of the thread holds a live lease;
  • '''lease''' like a claim: take leases a head whose lease is absent or expired, by the store's clock, with a new LeaseToken greater than every one issued for the entry, and ends the expired lease of every other entry of the thread (a head an Interrupt entry displaced), so two workers never hold an actionable lease on one thread; a lease may be renewed, released, have attempts recorded under it and close its entry for as long as it is the entry's current one, expired or not, and every such call under another token is refused (QueueError.LeaseLost), as is any on a closed entry (QueueError.EntryClosed); taking is skip-locked where the store can, so workers do not contend for one entry;
  • '''record attempts''' durably before answering (recordAttempt): an attempt's RunId naming no other run on the thread - no checkpoint of the thread's history and no attempt of any entry of the thread has it - so the run-id lookup finds only the attempt's own run; no attempt past maxAttempts (AttemptStart.Exhausted); and none once the thread was deleted while the entry was open, closing the entry Cancelled(ThreadDeleted) instead (AttemptStart.ThreadDeleted);
  • '''learn of deletes''': the store's Checkpointer.deleteThread marks every open entry of the thread in the delete's own transaction (or under its lock), whatever the deleted thread numbered, so no such entry runs on a new thread of the id; an entry submitted after the delete is not marked. The store's Checkpointer.commit refuses a checkpoint whose run is an attempt a marked entry recorded - while the entry is open and after it closes, until it is reclaimed - so a worker whose claim outlived its lease never writes the deleted thread's input;
  • '''learn of admissions''': the store's Checkpointer.commit of a checkpoint whose run is an attempt of an open entry marks the entry QueueEntry.admitted in the commit's own transaction (or under its lock), and the mark stays when a later retraction removes the checkpoint, so AttemptPlan never starts an admitted input again;
  • '''keep the evidence''': while an entry is open, its store's Checkpointer.prune keeps the newest checkpoint each of the entry's recorded runs committed on its thread, so latestOfRuns never loses an admitted run;
  • '''close''' an entry once, with its EntryOutcome, the store's time and the thread's last event number then (QueueEntry.closedAtSeq), and keep it for the ticket lifetime; reclaim removes closed entries older than that, oldest first, and never an open one;
  • '''scope by tenant''' (rule 9): entry, awaitClosed and openEntries answer for the caller's tenant's entries only (None and Some differ), another tenant's being as unknown as a reclaimed one;
  • return from every call within a bound it documents, as Checkpointer does.

RunQueueContract in llm4s-agent-testkit is the executable form of this contract.

Attributes

Companion
object
Supertypes
class Object
trait Matchable
class Any
Known subtypes
@Experimental
object RunQueue

Attributes

Companion
trait
Supertypes
class Object
trait Matchable
class Any
Self type
RunQueue.type
@Experimental
final case class RunQueueSettings

The run queue's own settings (design 4.55), loaded from llm4s.agent.queue by RunQueueConfigLoader:

The run queue's own settings (design 4.55), loaded from llm4s.agent.queue by RunQueueConfigLoader:

  • maxAttempts: the most attempts an entry makes before it is closed Failed(AttemptsExhausted) (default RunQueueSettings.DefaultMaxAttempts). Fixed on each entry at submission, so every worker agrees on it.
  • ticketLifetime: how long a closed entry - its input, attempts and outcome - is kept after it closed, by the store's clock, before the retention sweep reclaims it (default RunQueueSettings.DefaultTicketLifetime). Waiting on or following a reclaimed ticket answers QueueError.TicketExpired.
  • lease: the LeasePolicy a worker leases entries with.

maxAttempts must be at least 1 and ticketLifetime positive: apply and the with* setters throw IllegalArgumentException otherwise, and RunQueueSettings.of returns a ValidationError.

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
@Experimental
final class RunSubmitter(val queue: RunQueue, val checkpointer: Checkpointer, val settings: RunQueueSettings)

The submission API (design 4.55): queue a run for a worker on any machine to take, wait for its ticket's terminal state, or follow its events, over a store's RunQueue and the Checkpointer of the same store. Every call is tenant-scoped (rule 9): a ticket is answered only to its own tenant.

The submission API (design 4.55): queue a run for a worker on any machine to take, wait for its ticket's terminal state, or follow its events, over a store's RunQueue and the Checkpointer of the same store. Every call is tenant-scoped (rule 9): a ticket is answered only to its own tenant.

Entries are submitted with settings.maxAttempts; the worker side (W2) takes them.

Attributes

Supertypes
class Object
trait Matchable
class Any
@Experimental
final case class Submission

A run to queue (design 4.55): input for thread threadId, run by the agent agentId for tenantId, treating a busy thread as policy says, with at most maxAttempts attempts (RunQueueSettings.maxAttempts when built by RunSubmitter). A submissionKey makes the submit idempotent: a second submit with the key of an entry the store still holds returns that entry's ticket, and a different submission under the key is refused (QueueError.SubmissionKeyReused).

A run to queue (design 4.55): input for thread threadId, run by the agent agentId for tenantId, treating a busy thread as policy says, with at most maxAttempts attempts (RunQueueSettings.maxAttempts when built by RunSubmitter). A submissionKey makes the submit idempotent: a second submit with the key of an entry the store still holds returns that entry's ticket, and a different submission under the key is refused (QueueError.SubmissionKeyReused).

apply and the with* setters throw IllegalArgumentException for maxAttempts below 1; Submission.of returns it as a ValidationError.

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental
object Submission

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
Submission.type
@Experimental
object SubmissionKey

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
@Experimental
final case class TakeRequest

What RunQueue.take may take: the head entry of a thread whose agent is one of agents, leased to holder for ttl from now, by the store's clock.

What RunQueue.take may take: the head entry of a thread whose agent is one of agents, leased to holder for ttl from now, by the store's clock.

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Experimental
object TakeRequest

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
@Experimental
final case class Ticket(entryId: EntryId, threadId: ThreadId, position: Option[Int], existing: Boolean)

What RunQueue.submit answers (design 4.55): the entry entryId made for the submission on threadId, and its position among the thread's open entries when the store answered (0: it runs next). existing is true when a submission key named an entry the store already held, so this submit made none; its position is then the entry's current one, and None once that entry has closed.

What RunQueue.submit answers (design 4.55): the entry entryId made for the submission on threadId, and its position among the thread's open entries when the store answered (0: it runs next). existing is true when a submission key named an entry the store already held, so this submit made none; its position is then the entry's current one, and None once that entry has closed.

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

Types

opaque type EntryId

A run queue entry's id (design 4.55): positive, issued by the store in submission order and never reused, so it is also the entry's place in the store-wide order of submissions. A Ticket names it; a caller keeps it to wait on or follow the entry's run, from any machine.

A run queue entry's id (design 4.55): positive, issued by the store in submission order and never reused, so it is also the entry's place in the store-wide order of submissions. A Ticket names it; a caller keeps it to wait on or follow the entry's run, from any machine.

Attributes

opaque type LeaseToken

The fencing token of a lease on a queue entry (Lease): strictly greater than every lease token the store issued for that entry before, so a worker whose lease was taken over after it expired can neither renew it, record an attempt under it nor close the entry.

The fencing token of a lease on a queue entry (Lease): strictly greater than every lease token the store issued for that entry before, so a worker whose lease was taken over after it expired can neither renew it, record an attempt under it nor close the entry.

Attributes

opaque type SubmissionKey

A caller's key for one submission (design 4.55): unique per tenant, so a caller that retries a submit whose answer it lost - a dropped response, an ambiguous timeout - gets the ticket of the entry its first try made rather than a second entry. 1 to 200 characters. The key is held for as long as its entry is (the ticket lifetime after it closes); once the entry is reclaimed the key names nothing and may be used again.

A caller's key for one submission (design 4.55): unique per tenant, so a caller that retries a submit whose answer it lost - a dropped response, an ambiguous timeout - gets the ticket of the entry its first try made rather than a second entry. 1 to 200 characters. The key is held for as long as its entry is (the ticket lifetime after it closes); once the entry is reclaimed the key names nothing and may be used again.

Attributes