PostgresCheckpointer

org.llm4s.agent.graph.postgres.PostgresCheckpointer
See thePostgresCheckpointer companion object
@Experimental
final class PostgresCheckpointer extends Checkpointer, AutoCloseable

A durable org.llm4s.agent.graph.Checkpointer in a PostgreSQL database: threads, their checkpoint histories, pending writes, event logs, effect records and run claims, shared by every runtime on every machine that opens a store over the same database and schema. It is the store for running one agent service on several machines (Stage 5, design 4.50).

Open it with PostgresCheckpointer.open, over a pool of its own, or PostgresCheckpointer.create, over a DataSource the caller already has; close it with close. It meets the Checkpointer contract and passes CheckpointerContract from llm4s-agent-testkit:

  • '''Atomic, fenced commits.''' Every call that writes is one transaction, which first locks the thread's row (SELECT ... FOR UPDATE) and then checks the claim's token, a Commit.fence's claim on another thread, and the parent checkpoint, before it allocates event numbers from the row and writes. A call that locks two threads' rows
    • a fenced commit or delete - locks them in the order of their ids, so two such calls never deadlock.
  • '''The database decides time.''' A claim's expiry and a retention policy's age are judged by the database server's clock (clock_timestamp(), read once the call holds its locks), never a host's: skew between the machines sharing the store changes nothing. Age is measured from the commit time the store stamps, by that clock, on every checkpoint and event row - never from the producer's Checkpoint.createdAt or event timestamp, which stay as the producer's record. The clock a runtime is given still schedules its renewals.
  • '''Fencing tokens''' come from one sequence for the whole schema, so they increase across restarts, releases, takeovers and deleteThread, and are never issued twice.
  • '''History, retention, deletes and children''' behave exactly as in the SQLite store: every superseded checkpoint is kept until prune, retractTurn or deleteThread removes it; a deleted thread leaves a tombstone of its event numbering; a parent's children are listed in the order their first commits landed.
  • '''The retention sweep''' (design 4.53) pages through the threads by primary key, prunes each and reclaims old tombstones, raising the schema's reclamation mark in the same transaction, under an advisory lock that every claim creating a row shares; see sweep.
  • '''Data only.''' Checkpoints, pending writes, events and effect records are stored as JSON text and decoded on read, so nothing executable is stored.
  • '''Change notification by LISTEN/NOTIFY''' (design 4.51). A commit that stores events, and a delete, NOTIFYs the schema's channel inside its transaction, so the notification is delivered only if it commits; its payload is a hash of the thread id, nothing of the thread's content. One dedicated connection per store, opened by the first awaitEventsAfter and kept until close, LISTENs and wakes the store's waiting calls, so a commit through any store over the same database and schema wakes them as soon as it is delivered: within a second, tested, on a healthy database. A notification is only a hint: a woken call reads the log again, so a missed or duplicated one loses nothing. While the listening connection is down - not yet open, lost, or found dead by its heartbeat - a call waits out its timeout and its caller reads again, polling under its WatchPolicy; the connection is reopened in the background and, once it listens, every waiting call reads again. A wait never depends on that connection, so every call keeps its bound.

Reads run in a REPEATABLE READ, read-only transaction, so what one call reads is one snapshot; they never wait for a writer.

'''Every call returns within a bound''', as the Checkpointer contract requires (#1919): the pool's connection timeout plus PostgresCheckpointerConfig.callTimeout, plus the time a rollback takes. The call timeout is set on the server for the transaction (statement_timeout, lock_timeout, idle_in_transaction_session_timeout) and enforced on the client as well: a call that reaches it has its connection aborted and discarded from the pool, so a stalled network cannot hold it either. A call that reaches the bound fails with a ProcessingError. Like any networked store, a call that times out while its COMMIT is in flight cannot know whether the commit landed; the claim's fencing keeps that safe, as a later commit on a stale view is refused.

'''Interrupts.''' An interrupt flag already set when a call begins decides nothing: it is cleared for the call and set again when the call returns. An interrupt that lands during a call (on a virtual thread, it closes the connection's socket) ends that call with CancelledError or a failed result, and the connection is discarded, so the store stays usable by every other thread.

Every failure of the database itself is a ProcessingError, and so is any call after close. Safe to call from any number of threads: each call takes a connection of its own from the pool.

Attributes

Companion
object
Graph
Supertypes
trait AutoCloseable
trait Checkpointer
class Object
trait Matchable
class Any

Members list

Value members

Concrete methods

override def awaitEventsAfter(threadId: ThreadId, afterSeq: Long, limit: Int, timeout: FiniteDuration): Result[Vector[EventRecord]]

eventsAfter, waiting for a commit when there is nothing to return yet (see Checkpointer.awaitEventsAfter). The call registers its wait before it reads, so a commit that lands after the read wakes it; a wake reads the log again and returns what it finds, or waits on for what is left of timeout if a notification brought nothing (a hash collision, or a commit that stored nothing past afterSeq). It waits only on its own timeout, never on the listening connection: while that is down, the call returns empty once timeout passes, and its caller's next call reads again, so it polls. It returns within timeout plus the bound of each read it makes.

eventsAfter, waiting for a commit when there is nothing to return yet (see Checkpointer.awaitEventsAfter). The call registers its wait before it reads, so a commit that lands after the read wakes it; a wake reads the log again and returns what it finds, or waits on for what is left of timeout if a notification brought nothing (a hash collision, or a commit that stored nothing past afterSeq). It waits only on its own timeout, never on the listening connection: while that is down, the call returns empty once timeout passes, and its caller's next call reads again, so it polls. It returns within timeout plus the bound of each read it makes.

Attributes

Definition Classes
def checkpoint(threadId: ThreadId, checkpointId: String): Result[Option[Checkpoint]]

Checkpoint checkpointId of the thread's history, latest included; None if it is not there.

Checkpoint checkpointId of the thread's history, latest included; None if it is not there.

Attributes

def children(parent: ThreadId, after: Option[ThreadId], limit: Int): Result[Vector[ChildThread]]

Up to limit of parent's direct children (design 4.41), oldest first: every thread whose latest checkpoint's Checkpoint.origin names parent, in the order each first committed a checkpoint with that origin, starting after child after when it is given, so a caller pages through them by passing the last id of each page. A thread deleteThread removed is not listed, nor is a thread whose latest checkpoint names another parent or none. An after the store does not know as a child of parent is Left(ValidationError); a child deleted since it was listed keeps its place as long as its tombstone does (deleteThread), so a page boundary survives retention deleting it. A thread with no children has none; limit below 1 reads none. Needs no claim, and reads no tenant: the runtime checks the parent's tenant (GraphRuntime.children).

Up to limit of parent's direct children (design 4.41), oldest first: every thread whose latest checkpoint's Checkpoint.origin names parent, in the order each first committed a checkpoint with that origin, starting after child after when it is given, so a caller pages through them by passing the last id of each page. A thread deleteThread removed is not listed, nor is a thread whose latest checkpoint names another parent or none. An after the store does not know as a child of parent is Left(ValidationError); a child deleted since it was listed keeps its place as long as its tombstone does (deleteThread), so a page boundary survives retention deleting it. A thread with no children has none; limit below 1 reads none. Needs no claim, and reads no tenant: the runtime checks the parent's tenant (GraphRuntime.children).

Attributes

def claim(threadId: ThreadId, request: ClaimRequest): Result[RunClaim]

Claims threadId for request.holder, recording request.tenantId with the claim, until request.ttl from now, by the store's clock, if no other claim on it is live, and returns the claim with a new FencingToken. A live claim - whoever holds it, the same holder included - is Left(GraphError.ThreadBusy(threadId, latest, holder)) with the thread's latest checkpoint id and nothing changed. holder is GraphError.BusyHolder.Run(holder) only when the live claim's tenant is request.tenantId (None and Some differ), and GraphError.BusyHolder.Unnamed otherwise - also for a claim that recorded no tenant, such as one persisted before claims carried one - so that a refusal never reveals another tenant's RunId; a store never returns GraphError.BusyHolder.Lapsed. A thread need not have a checkpoint to be claimed. A claim that replaces an expired one nobody released names that claim's holder as RunClaim.takenOverFrom, under the same rule: only if it was of request.tenantId, and None otherwise.

Claims threadId for request.holder, recording request.tenantId with the claim, until request.ttl from now, by the store's clock, if no other claim on it is live, and returns the claim with a new FencingToken. A live claim - whoever holds it, the same holder included - is Left(GraphError.ThreadBusy(threadId, latest, holder)) with the thread's latest checkpoint id and nothing changed. holder is GraphError.BusyHolder.Run(holder) only when the live claim's tenant is request.tenantId (None and Some differ), and GraphError.BusyHolder.Unnamed otherwise - also for a claim that recorded no tenant, such as one persisted before claims carried one - so that a refusal never reveals another tenant's RunId; a store never returns GraphError.BusyHolder.Lapsed. A thread need not have a checkpoint to be claimed. A claim that replaces an expired one nobody released names that claim's holder as RunClaim.takenOverFrom, under the same rule: only if it was of request.tenantId, and None otherwise.

Attributes

def close(): Unit

Closes the store, and its pool if it owns one. Later calls return a ProcessingError; closing again does nothing. A claim this store granted stays in the database until it is released through another store or expires.

Closes the store, and its pool if it owns one. Later calls return a ProcessingError; closing again does nothing. A claim this store granted stays in the database until it is released through another store or expires.

Attributes

def commit(threadId: ThreadId, commit: Commit): Result[Vector[EventRecord]]

Applies commit atomically, fenced by its token, and returns its events with their sequence numbers, each recording that token. A Commit.fence of (other, otherToken) fences it by other's claim too, checked in the same transaction or under the same lock as the write, after the thread's own claim: otherwise Left(GraphError.StaleClaim(other, otherToken, otherCurrent)) with nothing written.

Applies commit atomically, fenced by its token, and returns its events with their sequence numbers, each recording that token. A Commit.fence of (other, otherToken) fences it by other's claim too, checked in the same transaction or under the same lock as the write, after the thread's own claim: otherwise Left(GraphError.StaleClaim(other, otherToken, otherCurrent)) with nothing written.

Attributes

def compactEvents(threadId: ThreadId, beforeSeq: Long): Result[Unit]

Drops events with seq < beforeSeq; replay can then start no earlier than beforeSeq.

Drops events with seq < beforeSeq; replay can then start no earlier than beforeSeq.

Attributes

def deleteThread(threadId: ThreadId, token: FencingToken, fence: Option[(ThreadId, FencingToken)]): Result[Unit]

Deletes the thread in one transaction, which first locks its row (and a fence thread's, in id order) and checks that token is its current claim - so the check and the delete see one state of the thread. The row stays, emptied, as a tombstone once the thread has numbered an event, so that a new thread of the id numbers on past it.

Deletes the thread in one transaction, which first locks its row (and a fence thread's, in id order) and checks that token is its current claim - so the check and the delete see one state of the thread. The row stays, emptied, as a tombstone once the thread has numbered an event, so that a new thread of the id numbers on past it.

Attributes

def effect(threadId: ThreadId, key: EffectKey): Result[Option[EffectRecord]]

The thread's effect record under key (ReplayGuard, design §4.24); None if it has none, or the thread is unknown. Needs no claim.

The thread's effect record under key (ReplayGuard, design §4.24); None if it has none, or the thread is unknown. Needs no claim.

Attributes

def eventsAfter(threadId: ThreadId, afterSeq: Long, limit: Int): Result[Vector[EventRecord]]

Up to limit events with seq > afterSeq, ascending. afterSeq 0 reads from the thread's first event, as does the number just before it; one behind the replay floor is GraphError.ReplayUnavailable, and one inside the log of a deleted thread of the id (Checkpointer.deleted) is GraphError.ThreadDeleted.

Up to limit events with seq > afterSeq, ascending. afterSeq 0 reads from the thread's first event, as does the number just before it; one behind the replay floor is GraphError.ReplayUnavailable, and one inside the log of a deleted thread of the id (Checkpointer.deleted) is GraphError.ThreadDeleted.

Attributes

def history(threadId: ThreadId, before: Option[String], limit: Int): Result[Vector[Checkpoint]]

Up to limit of the thread's checkpoints, newest first - the latest is the first of the first page - starting after checkpoint before when it is given, so a caller pages through the history by passing the last id of each page. A before that is not in the history is Left(GraphError.CheckpointNotFound); a thread with no checkpoint has an empty history.

Up to limit of the thread's checkpoints, newest first - the latest is the first of the first page - starting after checkpoint before when it is given, so a caller pages through the history by passing the last id of each page. A before that is not in the history is Left(GraphError.CheckpointNotFound); a thread with no checkpoint has an empty history.

Attributes

def incompleteThreads(tenantId: Option[TenantId], after: Option[ThreadCursor], limit: Int): Result[ThreadPage]

The incomplete root threads of tenantId (design 4.52): those whose latest checkpoint is incomplete (CheckpointStatus.Running) and that no live claim holds, by the store's clock - ThreadStatus.Idle - oldest first by ThreadSummary.updatedAt (ties by thread id, ascending), starting after after, at most limit (1 to ThreadQuery.MaxLimit, ValidationError otherwise), with the next page's cursor when more match. A thread whose claim expired without a release is listed; one whose cascade delete has begun is not (ThreadStatus.Deleted).

The incomplete root threads of tenantId (design 4.52): those whose latest checkpoint is incomplete (CheckpointStatus.Running) and that no live claim holds, by the store's clock - ThreadStatus.Idle - oldest first by ThreadSummary.updatedAt (ties by thread id, ascending), starting after after, at most limit (1 to ThreadQuery.MaxLimit, ValidationError otherwise), with the next page's cursor when more match. A thread whose claim expired without a release is listed; one whose cascade delete has begun is not (ThreadStatus.Deleted).

It is for operators, not a worker's recovery query: a cancelled run, a drained one and a crashed one all look alike here, and only a run queue's entry says which should run again (Stage 5, rule 4). Tenant-scoped and bounded as threads is.

Attributes

def isClosed: Boolean

Whether close has been called.

Whether close has been called.

Attributes

def latest(threadId: ThreadId): Result[Option[StoredCheckpoint]]

The thread's latest checkpoint and its pending writes; None for a new thread.

The thread's latest checkpoint and its pending writes; None for a new thread.

Attributes

def latestOfRuns(threadId: ThreadId, runIds: Set[RunId]): Result[Option[Checkpoint]]

The run-id lookup (Stage 5 rule 4, design 4.55): the newest checkpoint in threadId's history whose Checkpoint.runId is one of runIds, or None if none of them committed one that is still kept - none of them was admitted, or the thread was deleted since. A run queue's attempt asks it with the RunIds its entry recorded, before it touches the thread, to learn whether the entry's input was admitted, whatever ran on the thread since: the thread's latest checkpoint alone cannot say, because a direct run after the attempt died would have replaced it. Newest is by place in the history, not by createdAt.

The run-id lookup (Stage 5 rule 4, design 4.55): the newest checkpoint in threadId's history whose Checkpoint.runId is one of runIds, or None if none of them committed one that is still kept - none of them was admitted, or the thread was deleted since. A run queue's attempt asks it with the RunIds its entry recorded, before it touches the thread, to learn whether the entry's input was admitted, whatever ran on the thread since: the thread's latest checkpoint alone cannot say, because a direct run after the attempt died would have replaced it. Newest is by place in the history, not by createdAt.

A store answers it by an index on the thread and run, within its bound, however long the history; an empty runIds is Right(None). While an open entry of the store's run queue names a run, prune keeps the newest checkpoint that run committed, so the lookup never loses the evidence an open entry depends on. Needs no claim.

Attributes

def prune(threadId: ThreadId, policy: RetentionPolicy): Result[Unit]

Removes what policy does not keep, in one transaction holding the thread's row, so it sees and changes one state of the thread; it takes no claim, as the contract says. Age is judged by the database's clock against the time, by that clock, each checkpoint and event was committed - never the producer's createdAt or timestamp, which a writer with a skewed clock would make too old or too young (rule 1).

Removes what policy does not keep, in one transaction holding the thread's row, so it sees and changes one state of the thread; it takes no claim, as the contract says. Age is judged by the database's clock against the time, by that clock, each checkpoint and event was committed - never the producer's createdAt or timestamp, which a writer with a skewed clock would make too old or too young (rule 1).

Attributes

def release(threadId: ThreadId, token: FencingToken): Result[Unit]

Ends the claim whose token is token, so the thread can be claimed again at once. A token that is not the thread's current claim changes nothing and is Right(()): a run that lost its claim must never release its successor's.

Ends the claim whose token is token, so the thread can be claimed again at once. A token that is not the thread's current claim changes nothing and is Right(()): a run that lost its claim must never release its successor's.

Attributes

def renew(threadId: ThreadId, token: FencingToken, ttl: FiniteDuration): Result[RunClaim]

Extends the claim whose token is token to ttl from now, by the store's clock, and returns it. Left(GraphError.StaleClaim) if token is not the thread's current claim (another run took it over, it was released, or the thread was deleted). An expired claim nobody replaced is renewed.

Extends the claim whose token is token to ttl from now, by the store's clock, and returns it. Left(GraphError.StaleClaim) if token is not the thread's current claim (another run took it over, it was released, or the thread was deleted). An expired claim nobody replaced is renewed.

Attributes

def schema: String

The schema this store's tables are in.

The schema this store's tables are in.

Attributes

def sweep(policy: SweepPolicy, after: Option[ThreadId], guard: SweepGuard): Result[SweepPage]

One page of the sweep (design 4.53), in up to three transactions, each bounded as every call is - so a sweep call returns within three times the store's bound: the page's ids read by the primary key; the guard asked outside any transaction; then, if the policy reclaims, the page's old tombstones reclaimed under the reclamation lock (taken exclusively, so no claim creates a row from the mark meanwhile), each row locked in id order as every call locks rows - skipping a row another call holds, which is in use and so no tombstone to reclaim, for a sweep holding the reclamation lock never waits on a live thread's commit while every claim of a new id waits on it - and removed only if it is still only an old tombstone; then the rest of the page pruned, the rows locked in that same order. Time is the database's.

One page of the sweep (design 4.53), in up to three transactions, each bounded as every call is - so a sweep call returns within three times the store's bound: the page's ids read by the primary key; the guard asked outside any transaction; then, if the policy reclaims, the page's old tombstones reclaimed under the reclamation lock (taken exclusively, so no claim creates a row from the mark meanwhile), each row locked in id order as every call locks rows - skipping a row another call holds, which is in use and so no tombstone to reclaim, for a sweep holding the reclamation lock never waits on a live thread's commit while every claim of a new id waits on it - and removed only if it is still only an old tombstone; then the rest of the page pruned, the rows locked in that same order. Time is the database's.

Attributes

The thread registry (design 4.52): a page of the root threads of query.tenantId that pass every filter of query (ThreadQuery.matches), newest first by ThreadSummary.updatedAt - when the store last committed a checkpoint or metadata to the thread, by its clock - ties broken by thread id, descending; starting after query.after, at most query.limit of them, with the cursor of the next page when more match (ThreadPage.of). Each thread's ThreadStatus is judged from its latest checkpoint and whether a live claim holds it, by the store's clock, when it is read.

The thread registry (design 4.52): a page of the root threads of query.tenantId that pass every filter of query (ThreadQuery.matches), newest first by ThreadSummary.updatedAt - when the store last committed a checkpoint or metadata to the thread, by its clock - ties broken by thread id, descending; starting after query.after, at most query.limit of them, with the cursor of the next page when more match (ThreadPage.of). Each thread's ThreadStatus is judged from its latest checkpoint and whether a live claim holds it, by the store's clock, when it is read.

Listed are only threads with a checkpoint whose latest checkpoint's tenant is query.tenantId (None and Some differ) and whose latest checkpoint names no parent (Checkpoint.origin): a child thread is read through children, and a thread deleteThread removed is never listed. Paging is by position: a thread changed while a caller pages moves to the front, so a later page of that pass does not list it again, and a pass from the start, or with updatedSince, finds it. Needs no claim, and reads no other tenant's thread: the tenant is the query's. A store answers it within its bound however many threads it holds, by an index.

Attributes

Concrete fields

lazy val runQueue: RunQueue

The store's run queue (design 4.55), in the same schema and over the same pool as its checkpoints: its open entries learn of their thread's deletion in deleteThread's own transaction, its leases expire by the same database clock as claims, and prune keeps the newest checkpoint of every run an open entry recorded. Closing the store closes it.

The store's run queue (design 4.55), in the same schema and over the same pool as its checkpoints: its open entries learn of their thread's deletion in deleteThread's own transaction, its leases expire by the same database clock as claims, and prune keeps the newest checkpoint of every run an open entry recorded. Closing the store closes it.

Attributes