Checkpointer

org.llm4s.agent.graph.Checkpointer
See theCheckpointer companion object
trait Checkpointer

Durable storage for graph threads: each thread's checkpoint history, the pending writes recorded against its latest checkpoint, its durable event log, and the claim of the run that holds it. One store owns all of them so that a commit is atomic and fenced. Several GraphRuntimes, in one process or many, may share a store: the claim is what keeps two of them from running one thread at once.

Implementations must be safe to call from several threads and processes, and must:

  • grant at most one live claim per thread (claim): a claim is live until its expiresAt, by the store's own clock, has passed; a request while one is live is refused with GraphError.ThreadBusy naming its holder, and an expired claim is replaced - taken over;
  • issue each claim a FencingToken strictly greater than every token issued for that thread before - across releases, takeovers and deleteThread - so a token is never reused;
  • apply a Commit entirely or not at all, and only if its token is the thread's current claim (GraphError.StaleClaim otherwise, checked first) - so a writer that lost its claim, even one whose token is the highest it has seen, records nothing after another run took over - and fence deleteThread by its token in the same way, atomically with the delete; a commit's Commit.fence and a delete's fence are another thread's claim, checked in that same atomic step;
  • accept a new checkpoint only when its parent is the thread's latest checkpoint id - the thread version the writer expects (GraphError.CheckpointConflict otherwise) - and its id is not already in the thread's history, and only pending writes naming the resulting latest checkpoint (GraphError.InvalidCommit);
  • keep every checkpoint a commit superseded in the thread's history, newest first in history and readable by id with checkpoint, until prune, a commit's retractTurn or deleteThread removes it - but keep pending writes for the latest checkpoint only;
  • number a thread's events consecutively in commit order, inside the commit, never reusing a number - including after compaction, pruning, a failed commit or deleteThread: a thread's first event is 1 unless a thread of the same id that numbered events was deleted, and then it is one past the deleted thread's next number n - n itself is skipped - so a new thread of the id numbers on past every event of the old one; once sweep has reclaimed a tombstone, a thread whose id has none starts at the store's reclamation mark instead of 1 - at or past the first number of every tombstone it reclaimed - so a reclaimed id's new thread still numbers on past the old one (design 4.53);
  • record on each event the token of the commit that stored it (EventRecord.token, EventRecord.committed), so a run's events can be told from those of an earlier run that had the same RunId;
  • refuse a read of the log after a number of a deleted thread of the id - from 1 up to the skipped n, exclusive (Checkpointer.deleted) - with GraphError.ThreadDeleted, from eventsAfter and awaitEventsAfter alike, also once sweep has reclaimed the deleted thread's tombstone, so a reader that was following the deleted thread never takes the new one for its continuation; a read from 0 reads the current thread from its first event, and a watch waiting in awaitEventsAfter from inside a log that is deleted gets that refusal within about its timeout (from its next call, for the polling default);
  • keep events until compactEvents or prune removes them, and report the earliest sequence still available when asked to replay from before it (GraphError.ReplayUnavailable);
  • keep a thread's effect records (EffectRecord), upserted by key by a commit's effects and remembered with the task they belong to (taskId, lineage), until a commit with a checkpoint releases them (Commit.releasesEffect, judged before that commit's own records are applied) or deleteThread removes them - never prune, never by age - and copy none of them anywhere; effect reads one;
  • store checkpoints, pending writes and effect records as data (Checkpoint.toJson, PendingWrite's ReadWriter, EffectRecord.toJson), so nothing executable survives a round trip;
  • keep what it stores apart from the values its callers hold: ujson values are mutable, so changing a checkpoint, pending write or event payload after committing it, or one a read or a commit returned, must change nothing stored or later read back;
  • return from awaitEventsAfter with what eventsAfter would, at once if the log already holds an event after afterSeq, and otherwise within about timeout - with the events of a commit made meanwhile through any store over the same storage, or empty;
  • tolerate the thread calling eventsAfter or awaitEventsAfter being interrupted at any point: cancelling a subscription interrupts its watch mid-read, so an interrupt must end that one call - with Left(CancelledError) or a failed result - and leave the store usable by every other thread and later call (see awaitEventsAfter);
  • list a thread's direct children (children) - the threads whose latest checkpoint's RunOrigin names it as parent - in the order they were first committed, paged, and never one deleteThread removed;
  • keep each thread's ThreadMetadata, replaced only by a commit carrying Commit.metadata and removed only by deleteThread, and list a tenant's root threads (threads) and its incomplete ones (incompleteThreads) - the thread registry (design 4.52) - by their latest checkpoint, their claim judged by the store's clock, and when the store last committed a checkpoint or metadata to them, by that clock, never one of another tenant, a child thread, one with no checkpoint, or one deleteThread removed;
  • answer latestOfRuns - the newest checkpoint any of a set of runs committed on a thread, the run-id lookup a run queue's entry classifies its attempts by (Stage 5 rule 4, design 4.55) - by an index on the run, and keep the checkpoint it answers with through prune while an open entry of the store's run queue names that run;
  • visit every thread it holds, tombstones included, through sweep, a bounded page at a time, pruning each as prune does and reclaiming old tombstones, but never what the sweep's SweepGuard protects;
  • return from every call within a bound it documents (#1919) - see below.

'''Every call returns within a bound.''' A store times out its own calls: it documents a bound, set by its configuration (such as SqliteCheckpointerConfig.busyTimeout), within which every call returns, whatever its storage is doing - waiting for a lock, for another connection's or process's write, for the store's own earlier calls, or for a network - and a call that reaches the bound fails, changing nothing. awaitEventsAfter returns within its timeout plus that bound. A store that cannot bound some call - a disk that stalls inside a write, say, which no driver can time out - documents which call, and why. Keep the bound well inside ClaimPolicy.ttl - ClaimPolicy.renewEvery: a renewal must land within that, and a store that serialises its calls queues a renewal behind a slow commit.

A GraphRuntime bounds what it can without the store: a run's result, its slot in the runtime and its claim are freed within twice ClaimPolicy.ttl of a lost claim being found (GraphError.ClaimLost), and of a closing commit - an Async run's drain, or an OnExit run's exit commit - sticking while the claim holds (GraphError.CommitTimedOut). Every other call made while a claim holds - an admission's claim commit, a Sync run's commits, its outcome's included, a change's write, and the release of a claim that was not lost - is waited for as long as it takes, and only the store's bound limits it. Nor can the runtime free what a call stuck in the store holds itself, so these last as long as the call does:

  • '''the calling thread.''' A run thread, or the virtual thread an admission, fork, updateState or deleteThread makes its store calls on, stays alive inside the call, and an Async run's writer thread with it, after the runtime has given the call up;
  • '''the thread's later commits.''' Commits on one thread are written one at a time in a runtime, so a commit - or a subscription's read of events another runtime committed, which a commit's hand-over makes under the same lock - holds up every later commit on that thread in that runtime, a new run's claim commit included, until it returns; never another thread's;
  • '''subscriptions behind a commit.''' A GraphRuntime.subscribe made while a run's claim commit is in the store, on a thread with nothing of the run handed over yet - a claim commit its admission gave up on after a lost claim included - waits for that commit to return, or for the subscription to be cancelled, before it delivers its replay; a run-scoped subscription never does. And with WatchPolicy.disabled, a replay made while a write a run gave up on is still in the store stops at the first event another runtime committed above what this runtime has handed over; it receives the rest when this runtime next commits on the thread, and a replay made once the write has returned has them all.

The holder of a claim may commit and renew it after expiresAt for as long as no other claim has replaced it: expiry only lets another run take the thread over. CheckpointerContract in llm4s-agent-testkit is the executable form of this contract; a store should pass it.

Attributes

Companion
object
Source
Checkpointer.scala
Graph
Supertypes
class Object
trait Matchable
class Any
Known subtypes

Members list

Value members

Abstract methods

def checkpoint(threadId: ThreadId, checkpointId: String): Result[Option[Checkpoint]]

Checkpoint checkpointId of the thread's history, latest included; None if it is not there.

Checkpoint checkpointId of the thread's history, latest included; None if it is not there.

Attributes

Source
Checkpointer.scala
def children(parent: ThreadId, after: Option[ThreadId], limit: Int): Result[Vector[ChildThread]]

Up to limit of parent's direct children (design 4.41), oldest first: every thread whose latest checkpoint's Checkpoint.origin names parent, in the order each first committed a checkpoint with that origin, starting after child after when it is given, so a caller pages through them by passing the last id of each page. A thread deleteThread removed is not listed, nor is a thread whose latest checkpoint names another parent or none. An after the store does not know as a child of parent is Left(ValidationError); a child deleted since it was listed keeps its place as long as its tombstone does (deleteThread), so a page boundary survives retention deleting it. A thread with no children has none; limit below 1 reads none. Needs no claim, and reads no tenant: the runtime checks the parent's tenant (GraphRuntime.children).

Up to limit of parent's direct children (design 4.41), oldest first: every thread whose latest checkpoint's Checkpoint.origin names parent, in the order each first committed a checkpoint with that origin, starting after child after when it is given, so a caller pages through them by passing the last id of each page. A thread deleteThread removed is not listed, nor is a thread whose latest checkpoint names another parent or none. An after the store does not know as a child of parent is Left(ValidationError); a child deleted since it was listed keeps its place as long as its tombstone does (deleteThread), so a page boundary survives retention deleting it. A thread with no children has none; limit below 1 reads none. Needs no claim, and reads no tenant: the runtime checks the parent's tenant (GraphRuntime.children).

Attributes

Source
Checkpointer.scala
def claim(threadId: ThreadId, request: ClaimRequest): Result[RunClaim]

Claims threadId for request.holder, recording request.tenantId with the claim, until request.ttl from now, by the store's clock, if no other claim on it is live, and returns the claim with a new FencingToken. A live claim - whoever holds it, the same holder included - is Left(GraphError.ThreadBusy(threadId, latest, holder)) with the thread's latest checkpoint id and nothing changed. holder is GraphError.BusyHolder.Run(holder) only when the live claim's tenant is request.tenantId (None and Some differ), and GraphError.BusyHolder.Unnamed otherwise - also for a claim that recorded no tenant, such as one persisted before claims carried one - so that a refusal never reveals another tenant's RunId; a store never returns GraphError.BusyHolder.Lapsed. A thread need not have a checkpoint to be claimed. A claim that replaces an expired one nobody released names that claim's holder as RunClaim.takenOverFrom, under the same rule: only if it was of request.tenantId, and None otherwise.

Claims threadId for request.holder, recording request.tenantId with the claim, until request.ttl from now, by the store's clock, if no other claim on it is live, and returns the claim with a new FencingToken. A live claim - whoever holds it, the same holder included - is Left(GraphError.ThreadBusy(threadId, latest, holder)) with the thread's latest checkpoint id and nothing changed. holder is GraphError.BusyHolder.Run(holder) only when the live claim's tenant is request.tenantId (None and Some differ), and GraphError.BusyHolder.Unnamed otherwise - also for a claim that recorded no tenant, such as one persisted before claims carried one - so that a refusal never reveals another tenant's RunId; a store never returns GraphError.BusyHolder.Lapsed. A thread need not have a checkpoint to be claimed. A claim that replaces an expired one nobody released names that claim's holder as RunClaim.takenOverFrom, under the same rule: only if it was of request.tenantId, and None otherwise.

Attributes

Source
Checkpointer.scala
def commit(threadId: ThreadId, commit: Commit): Result[Vector[EventRecord]]

Applies commit atomically, fenced by its token, and returns its events with their sequence numbers, each recording that token. A Commit.fence of (other, otherToken) fences it by other's claim too, checked in the same transaction or under the same lock as the write, after the thread's own claim: otherwise Left(GraphError.StaleClaim(other, otherToken, otherCurrent)) with nothing written.

Applies commit atomically, fenced by its token, and returns its events with their sequence numbers, each recording that token. A Commit.fence of (other, otherToken) fences it by other's claim too, checked in the same transaction or under the same lock as the write, after the thread's own claim: otherwise Left(GraphError.StaleClaim(other, otherToken, otherCurrent)) with nothing written.

Attributes

Source
Checkpointer.scala
def compactEvents(threadId: ThreadId, beforeSeq: Long): Result[Unit]

Drops events with seq < beforeSeq; replay can then start no earlier than beforeSeq.

Drops events with seq < beforeSeq; replay can then start no earlier than beforeSeq.

Attributes

Source
Checkpointer.scala
def deleteThread(threadId: ThreadId, token: FencingToken, fence: Option[(ThreadId, FencingToken)]): Result[Unit]

Removes everything stored for the thread - its checkpoint history, its pending writes, its event log, its effect records and its claim - so that its id names a new thread again; tokens issued after it are still greater than every token issued before. The store keeps only where the thread's event numbering ended - a tombstone, holding nothing of the thread's data, and nothing at all for a thread that numbered no event - so that a new thread of the id numbers its events on past the deleted thread's, one number skipped, and a read of the deleted log after the delete is GraphError.ThreadDeleted (see eventsAfter); a store that wakes awaitEventsAfter on commits wakes it on a delete too. Deleting a thread whose tombstone has numbered no event since does not move the numbering on again.

Removes everything stored for the thread - its checkpoint history, its pending writes, its event log, its effect records and its claim - so that its id names a new thread again; tokens issued after it are still greater than every token issued before. The store keeps only where the thread's event numbering ended - a tombstone, holding nothing of the thread's data, and nothing at all for a thread that numbered no event - so that a new thread of the id numbers its events on past the deleted thread's, one number skipped, and a read of the deleted log after the delete is GraphError.ThreadDeleted (see eventsAfter); a store that wakes awaitEventsAfter on commits wakes it on a delete too. Deleting a thread whose tombstone has numbered no event since does not move the numbering on again.

'''The tombstone is kept until sweep reclaims it.''' A deleted id keeps a small record - its next event number and when the store deleted it, nothing of the thread's data - which neither prune nor a later delete removes. It is what lets a reader of the deleted log, in any process, be told the thread was deleted, and a new thread of the id be numbered past it. A store whose threads are deleted often - every failed one-shot Agent.run forgets its thread, and child retention deletes children - grows by one such record per deleted thread that numbered an event until a sweep with a tombstoneAge reclaims the old ones; a reclaimed tombstone's readers are still told the thread was deleted, through the store's reclamation mark (design 4.53). Fenced like a commit: only while token is the thread's current claim, checked atomically with the delete, and otherwise Left(GraphError.StaleClaim) with nothing deleted - so a deleter that lost its claim (it expired and another run took the thread over, or it was released) deletes nothing of the run that holds the thread now. A thread with no claim, an unknown one included, is therefore refused too: claim it first. Use GraphRuntime.deleteThread, which claims the thread, so it refuses a thread with a live run in any runtime, and checks the tenant under its claim.

'''The fence.''' With fence = Some((other, otherToken)) the delete is also fenced by another thread's claim: it happens only if otherToken is other's current claim too, checked in the same atomic step as the thread's own claim and the delete (one transaction, or under one lock), and otherwise is Left(GraphError.StaleClaim(other, otherToken, otherCurrent)) with nothing deleted. The thread's own claim is checked first. GraphRuntime.deleteThread fences a cascade's delete of each descendant by the root's claim, and child retention each delete by the parent's (design 4.41), so a successor run that took the thread above over never finds a child it may reattach gone. None fences by the thread's own claim alone.

Attributes

Source
Checkpointer.scala

The thread's effect record under key (ReplayGuard, design §4.24); None if it has none, or the thread is unknown. Needs no claim.

The thread's effect record under key (ReplayGuard, design §4.24); None if it has none, or the thread is unknown. Needs no claim.

Attributes

Source
Checkpointer.scala
def eventsAfter(threadId: ThreadId, afterSeq: Long, limit: Int): Result[Vector[EventRecord]]

Up to limit events with seq > afterSeq, ascending. afterSeq 0 reads from the thread's first event, as does the number just before it; one behind the replay floor is GraphError.ReplayUnavailable, and one inside the log of a deleted thread of the id (Checkpointer.deleted) is GraphError.ThreadDeleted.

Up to limit events with seq > afterSeq, ascending. afterSeq 0 reads from the thread's first event, as does the number just before it; one behind the replay floor is GraphError.ReplayUnavailable, and one inside the log of a deleted thread of the id (Checkpointer.deleted) is GraphError.ThreadDeleted.

Attributes

Source
Checkpointer.scala
def history(threadId: ThreadId, before: Option[String], limit: Int): Result[Vector[Checkpoint]]

Up to limit of the thread's checkpoints, newest first - the latest is the first of the first page - starting after checkpoint before when it is given, so a caller pages through the history by passing the last id of each page. A before that is not in the history is Left(GraphError.CheckpointNotFound); a thread with no checkpoint has an empty history.

Up to limit of the thread's checkpoints, newest first - the latest is the first of the first page - starting after checkpoint before when it is given, so a caller pages through the history by passing the last id of each page. A before that is not in the history is Left(GraphError.CheckpointNotFound); a thread with no checkpoint has an empty history.

Attributes

Source
Checkpointer.scala

The incomplete root threads of tenantId (design 4.52): those whose latest checkpoint is incomplete (CheckpointStatus.Running) and that no live claim holds, by the store's clock - ThreadStatus.Idle - oldest first by ThreadSummary.updatedAt (ties by thread id, ascending), starting after after, at most limit (1 to ThreadQuery.MaxLimit, ValidationError otherwise), with the next page's cursor when more match. A thread whose claim expired without a release is listed; one whose cascade delete has begun is not (ThreadStatus.Deleted).

The incomplete root threads of tenantId (design 4.52): those whose latest checkpoint is incomplete (CheckpointStatus.Running) and that no live claim holds, by the store's clock - ThreadStatus.Idle - oldest first by ThreadSummary.updatedAt (ties by thread id, ascending), starting after after, at most limit (1 to ThreadQuery.MaxLimit, ValidationError otherwise), with the next page's cursor when more match. A thread whose claim expired without a release is listed; one whose cascade delete has begun is not (ThreadStatus.Deleted).

It is for operators, not a worker's recovery query: a cancelled run, a drained one and a crashed one all look alike here, and only a run queue's entry says which should run again (Stage 5, rule 4). Tenant-scoped and bounded as threads is.

Attributes

Source
Checkpointer.scala

The thread's latest checkpoint and its pending writes; None for a new thread.

The thread's latest checkpoint and its pending writes; None for a new thread.

Attributes

Source
Checkpointer.scala
def latestOfRuns(threadId: ThreadId, runIds: Set[RunId]): Result[Option[Checkpoint]]

The run-id lookup (Stage 5 rule 4, design 4.55): the newest checkpoint in threadId's history whose Checkpoint.runId is one of runIds, or None if none of them committed one that is still kept - none of them was admitted, or the thread was deleted since. A run queue's attempt asks it with the RunIds its entry recorded, before it touches the thread, to learn whether the entry's input was admitted, whatever ran on the thread since: the thread's latest checkpoint alone cannot say, because a direct run after the attempt died would have replaced it. Newest is by place in the history, not by createdAt.

The run-id lookup (Stage 5 rule 4, design 4.55): the newest checkpoint in threadId's history whose Checkpoint.runId is one of runIds, or None if none of them committed one that is still kept - none of them was admitted, or the thread was deleted since. A run queue's attempt asks it with the RunIds its entry recorded, before it touches the thread, to learn whether the entry's input was admitted, whatever ran on the thread since: the thread's latest checkpoint alone cannot say, because a direct run after the attempt died would have replaced it. Newest is by place in the history, not by createdAt.

A store answers it by an index on the thread and run, within its bound, however long the history; an empty runIds is Right(None). While an open entry of the store's run queue names a run, prune keeps the newest checkpoint that run committed, so the lookup never loses the evidence an open entry depends on. Needs no claim.

Attributes

Source
Checkpointer.scala
def prune(threadId: ThreadId, policy: RetentionPolicy): Result[Unit]

Removes the checkpoints and events of the thread that policy does not keep (see RetentionPolicy), by the store's clock: maxAge is judged against the time the store committed each checkpoint and event, by that same clock, never against the producer's Checkpoint.createdAt or EventRecord.timestamp, so a writer whose clock is skewed makes nothing too old or too young. The latest checkpoint and its pending writes are always kept. Pruning a checkpoint also removes the events before the oldest checkpoint kept; maxAge and maxEvents may remove more, including the latest checkpoint's. Removing events raises the replay floor as compactEvents does. Needs no claim: it touches nothing a run reads or writes. An unknown thread is Right(()).

Removes the checkpoints and events of the thread that policy does not keep (see RetentionPolicy), by the store's clock: maxAge is judged against the time the store committed each checkpoint and event, by that same clock, never against the producer's Checkpoint.createdAt or EventRecord.timestamp, so a writer whose clock is skewed makes nothing too old or too young. The latest checkpoint and its pending writes are always kept. Pruning a checkpoint also removes the events before the oldest checkpoint kept; maxAge and maxEvents may remove more, including the latest checkpoint's. Removing events raises the replay floor as compactEvents does. Needs no claim: it touches nothing a run reads or writes. An unknown thread is Right(()).

Attributes

Source
Checkpointer.scala
def release(threadId: ThreadId, token: FencingToken): Result[Unit]

Ends the claim whose token is token, so the thread can be claimed again at once. A token that is not the thread's current claim changes nothing and is Right(()): a run that lost its claim must never release its successor's.

Ends the claim whose token is token, so the thread can be claimed again at once. A token that is not the thread's current claim changes nothing and is Right(()): a run that lost its claim must never release its successor's.

Attributes

Source
Checkpointer.scala
def renew(threadId: ThreadId, token: FencingToken, ttl: FiniteDuration): Result[RunClaim]

Extends the claim whose token is token to ttl from now, by the store's clock, and returns it. Left(GraphError.StaleClaim) if token is not the thread's current claim (another run took it over, it was released, or the thread was deleted). An expired claim nobody replaced is renewed.

Extends the claim whose token is token to ttl from now, by the store's clock, and returns it. Left(GraphError.StaleClaim) if token is not the thread's current claim (another run took it over, it was released, or the thread was deleted). An expired claim nobody replaced is renewed.

Attributes

Source
Checkpointer.scala
def sweep(policy: SweepPolicy, after: Option[ThreadId], guard: SweepGuard): Result[SweepPage]

The store-wide retention sweep (design 4.53): one bounded page of the threads the store holds, tombstones included, in an order of the store's own that every call follows - the threads after after in it, whether or not the store still holds after, at most policy.batchSize of them - with the next page's cursor when more follow. A caller sweeps the whole store by calling again with each page's next until it is None; a thread created while a caller pages may be visited in that pass or only in the next.

The store-wide retention sweep (design 4.53): one bounded page of the threads the store holds, tombstones included, in an order of the store's own that every call follows - the threads after after in it, whether or not the store still holds after, at most policy.batchSize of them - with the next page's cursor when more follow. A caller sweeps the whole store by calling again with each page's next until it is None; a thread created while a caller pages may be visited in that pass or only in the next.

Before it changes anything of the page it asks guard which of the page's threads are protected (SweepGuard), and a Left from the guard ends the call with nothing changed. Then, for each thread of the page:

  • its history is pruned by policy.retention exactly as prune prunes it, judged by the store's clock against the store's commit times - except that, of a protected thread, the newest checkpoint each of its protected RunIds committed (Checkpoint.runId) is kept as well, whatever the policy, while the rest is pruned as usual and the events' floor follows the policy alone; and so, as prune does, is the newest checkpoint each run an open entry of the store's run queue recorded committed (design 4.55), whatever the guard answers;
  • its tombstone (deleteThread) is reclaimed - removed, with the place it kept among its parent's children - when policy.tombstoneAge is set, the store last deleted the thread more than that long ago by its clock, nothing has been stored under the id since, and the thread is not protected. Reclaiming raises the store's reclamation mark to the tombstone's first number, so a new thread of the id - or of any id without a tombstone - numbers its events on past the old thread's, and a read from inside the reclaimed log is still GraphError.ThreadDeleted (eventsAfter). Only paging children after a reclaimed child refuses its cursor.

Like prune, it needs no claim and changes nothing a run reads or writes; it reads no tenant, for it is the operator's, store-wide. Each call returns within the store's bound, whatever the store holds. Run one sweep at a time per store (a worker's schedule does: Stage 5, W2); two at once are safe, only wasteful.

Attributes

Source
Checkpointer.scala

The thread registry (design 4.52): a page of the root threads of query.tenantId that pass every filter of query (ThreadQuery.matches), newest first by ThreadSummary.updatedAt - when the store last committed a checkpoint or metadata to the thread, by its clock - ties broken by thread id, descending; starting after query.after, at most query.limit of them, with the cursor of the next page when more match (ThreadPage.of). Each thread's ThreadStatus is judged from its latest checkpoint and whether a live claim holds it, by the store's clock, when it is read.

The thread registry (design 4.52): a page of the root threads of query.tenantId that pass every filter of query (ThreadQuery.matches), newest first by ThreadSummary.updatedAt - when the store last committed a checkpoint or metadata to the thread, by its clock - ties broken by thread id, descending; starting after query.after, at most query.limit of them, with the cursor of the next page when more match (ThreadPage.of). Each thread's ThreadStatus is judged from its latest checkpoint and whether a live claim holds it, by the store's clock, when it is read.

Listed are only threads with a checkpoint whose latest checkpoint's tenant is query.tenantId (None and Some differ) and whose latest checkpoint names no parent (Checkpoint.origin): a child thread is read through children, and a thread deleteThread removed is never listed. Paging is by position: a thread changed while a caller pages moves to the front, so a later page of that pass does not list it again, and a pass from the start, or with updatedSince, finds it. Needs no claim, and reads no other tenant's thread: the tenant is the query's. A store answers it within its bound however many threads it holds, by an index.

Attributes

Source
Checkpointer.scala

Concrete methods

def awaitEventsAfter(threadId: ThreadId, afterSeq: Long, limit: Int, timeout: FiniteDuration): Result[Vector[EventRecord]]

eventsAfter, waiting for a commit when there is nothing to return yet: up to limit events with seq > afterSeq, ascending, at once if the log holds any; otherwise those of a commit that lands within timeout, or an empty Vector once it has passed. It is how a subscription learns of commits made through another GraphRuntime or process sharing the store (see WatchPolicy): each subscription calls it in a loop, from the last event it has queued.

eventsAfter, waiting for a commit when there is nothing to return yet: up to limit events with seq > afterSeq, ascending, at once if the log holds any; otherwise those of a commit that lands within timeout, or an empty Vector once it has passed. It is how a subscription learns of commits made through another GraphRuntime or process sharing the store (see WatchPolicy): each subscription calls it in a loop, from the last event it has queued.

The default polls. It reads once and, finding nothing, waits timeout and returns empty, so the caller's next call reads again: a commit is seen up to timeout after it lands, at the cost of one eventsAfter per timeout per subscription. A store that can be told of commits overrides it to return as soon as one lands, without reading meanwhile - InMemoryCheckpointer wakes on its own commits, and a database store can use its notification channel (PostgreSQL's LISTEN/NOTIFY). Such a store must wake for commits made through every store over the same storage, not only its own, and still return within about timeout when none lands.

Errors are eventsAfter's - GraphError.ReplayUnavailable when afterSeq is behind the replay floor. Interrupting the waiting thread ends the wait with Left(CancelledError), the thread's interrupt flag set again.

'''The reading thread may be interrupted at any point.''' A subscription's watch calls this on a thread of its own, and Subscription.cancel - or the subscription's end - interrupts that thread whether it is waiting or in the middle of a read, so the interrupt can land inside the store's own I/O (eventsAfter included). A store must take that as the end of this one call, never as damage to itself: release what the call held (a pooled connection, a lock), and stay usable by other threads and later calls. Beware drivers whose I/O goes through a java.nio.channels.InterruptibleChannel (a FileChannel or SocketChannel): an interrupt during a read or write closes the channel, with ClosedByInterruptException, so a store that shares one channel - one connection - between threads would lose it for all of them. Such a store reads on a connection of its own for the waiting call, or discards and replaces a connection an interrupt has closed.

Attributes

Source
Checkpointer.scala