GraphRuntime

org.llm4s.agent.graph.GraphRuntime
See theGraphRuntime companion object
final class GraphRuntime(checkpointer: Checkpointer, clock: Clock, claims: ClaimPolicy, watch: WatchPolicy)

Runs compiled graphs on durable threads.

A thread's latest checkpoint says what may happen next:

  • none, Completed or Failed: start runs the graph with a new input (on a finished thread, over its committed state). Failed is a run that ended in a failure that is its outcome, not an interruption: a node returned NodeResult.Block, as a guardrail does. The thread is usable and recover has nothing to continue;
  • Running - work was scheduled when the last run stopped: recover continues it with no new input, reusing every pending write so completed tasks are not run again, and running failed or unstarted tasks again, each with its node's full RetryPolicy;
  • Suspended: resume answers any non-empty subset of the parked interrupts; unanswered ones stay parked, and the run suspends again if nothing else can proceed.

Each call's RunConfig brings the run's static breakpoints; one on a node the graph does not have is refused with a ValidationError before the thread is read. A task they hold parks like a task that suspended, so the thread ends Suspended and resume answers it with Breakpoint.proceed.

Any other call is refused (GraphError.IncompleteRun, GraphError.PendingInterrupts, GraphError.NothingToRecover, GraphError.NotSuspended) without changing the thread. A thread belongs to the tenant on its latest checkpoint, and a call with another tenantId is refused with GraphError.TenantMismatch before anything else is reported - the thread's status, or that a run is live on it - and the error names only the caller's tenant, so it learns nothing about the thread.

A thread whose latest checkpoint records a RunOrigin is a child thread, started by a tool call of its parent's run (design 4.41). It is driven only through that run: start, recover, resume, fork, updateState and deleteThread on it are refused with GraphError.ChildThread, after the tenant check and before any claim, while history, checkpoint, subscribe and children read it as any thread. Deleting a thread deletes its children first, and child retention (RunConfig.withChildRetention) bounds how many settled children a thread keeps. Each run's terminal (Completed or Failed) checkpoint records how it settled (Checkpoint.outcome).

Each call is a new run, and claims the thread in two steps before it runs anything. First it asks the store for a RunClaim: a lease with an expiry and a FencingToken, which the run renews while it executes (see ClaimPolicy) and releases when it ends. While another run's claim is live - in this runtime or in any other runtime or process sharing the store - the call fails with GraphError.ThreadBusy naming that run, so recover elsewhere refuses a live run and takes over a dead one only once its claim has expired. Then it commits a checkpoint whose parent is the latest it read; if another run advanced the thread in between, it fails with ThreadBusy too. Either way its input or answers are neither accepted nor discarded. Every commit of the run carries its claim's token, and the store refuses one whose claim was taken over (GraphError.StaleClaim), so a run that lost its claim - a process paused past ttl - cannot record anything after its successor started. The run also judges its own lease, by its monotonic clock, so it learns of the loss even when no renewal reaches the store: once the lease passes with no renewal landing, or a renewal or commit is refused as stale, it stops at once - its tasks interrupted, as a cancel interrupts them - commits nothing more, and fails with GraphError.ClaimLost. A loss found once the run has begun committing its outcome sends no interrupt, but still skips every commit not yet begun; and whatever the run is doing, it ends within twice ttl of the loss: if it has not begun to by then - its thread stuck in a store call that ignores interrupts - it is ended without its thread, ClaimLost, and the thread freed. Admission is bounded the same way: a claim's commit still in the store twice ttl after the loss is given up, and the call fails ClaimLost - and if that commit then lands before the claim's release does, nobody having taken the thread over, it leaves its Running checkpoint and its RunStarted, RunRecovered or RunResumed event although no run continues it: recover does, once the release lands; one the store refuses as stale - the claim lapsed, or another run took it over, before it landed - fails ClaimLost too, never GraphError.CheckpointWriteFailed. So are fork, updateState and deleteThread, which hold a claim while they change the thread: a write not begun at the loss is never sent, and the call returns GraphError.ThreadBusy within twice ttl of it. A recover that takes over an expired claim records the run it took the thread from (RunEvent.RunRecovered). A call on a thread whose run is still executing in this runtime fails with ThreadBusy before reading the thread at all.

Admission - the checks above, restoring the thread and committing the claim - runs on the caller's thread, but for the read under the claim and the claim's commit, which run on llm4s-admit-<threadId> while the caller waits, and never throws: a Left means no run exists, and the thread is free again. A throwable from the store, the clock or the graph during admission is GraphError.RunCrashed, and an interrupt is CancelledError with the flag set. The run itself executes on a virtual thread owned by the runtime, named llm4s-run-<threadId>, behind the returned RunHandle; the thread stays busy until that run thread exits. An unexpected throwable escaping the run ends it with GraphError.RunCrashed.

RunHandle.cancel cancels the run by interrupting its run thread: every task is interrupted and joined, tasks that finished first keep their pending writes, an interrupted task records nothing, and the run ends Failed with GraphError.Cancelled - naming the last checkpoint, which a RunEvent.RunCancelled event is committed against. The checkpoint stays Running, so recover continues the run. A cancel that interrupts a superstep's commit ends the run the same way even if the store reports that commit as failed (as a JDBC store may, rather than throwing InterruptedException): the run is cancelled, not failed by its store, and the error names the last checkpoint that was durable. Once the run has begun committing its outcome - its completed or suspended checkpoint, or the RunFailed of a failed run - a cancel is ignored - it sends no interrupt, so that commit is not disturbed - and the run ends with its outcome. Interrupting a thread blocked in RunHandle.await does not cancel the run.

A run's closing wait is bounded while it holds its claim too (#1919). An Async run waiting for its queued commits, or an OnExit run making its exit commit, whose commit stays in the store for twice claims.ttl with none returning, ends GraphError.CommitTimedOut without waiting for it, through the same settling as a lost run: its result is set, its thread freed in this runtime, and its claim released on a detached thread. The commit is an orphan, fenced by the claim as a lost run's late write is: it lands only if the store applies it before the release, and the run's own listeners never receive it. A drain that keeps making progress is never cut short, however long it takes in all. Every other store call is waited for as long as it takes: every Checkpointer must return from each call within a bound it documents.

A run whose RunBudgets.timeout is set has a deadline, fixed when its claim commits; its tasks read it, by the runtime's clock, as RunContext.deadline. When it passes, the run stops the same way, but ends with GraphError.DeadlineExceeded and commits RunEvent.RunTimedOut. The first of cancel and expiry to be recorded wins, so a run reports exactly one. DeadlineExceeded is recoverable: recover with a new budget continues the run.

subscribe replays a thread's committed events after a sequence number and then delivers new ones as their commits succeed - only to a caller of the thread's tenant, as every other read of a thread is (see subscribe) - in ascending order with no gaps or duplicates, followed by live progress as it happens. Each subscription has its own dispatcher thread and a queue of capacity events, so a slow listener never holds up a run: one that falls behind by more than capacity durable events is disconnected (DisconnectReason.Lagging), and live events that do not fit are dropped and counted (StreamEvent.LiveGap). See EventHub. Live events are never replayed, but a subscription receives every one sent after subscribe returns, also while it is still replaying: so subscribing and then calling start loses none of the run's. One made after start returns can miss the run's first live events; an Observer passed to start, recover or resume is subscribed during admission, before the claim commits, and so sees every event of the run (RunHandle.observation).

A subscription also sees the durable events of commits made through another runtime or process sharing the checkpointer, with the same durable guarantees (ascending seq, no gap or duplicate): each live subscription watches the store (WatchPolicy, Checkpointer.awaitEventsAfter), so they arrive as soon as the store reports them - at once from a store that is told of commits, within pollInterval from one that is polled. Live progress events are never stored, so another runtime's do not arrive.

'''Ordering across runtimes.''' Concurrently active subscriptions to a thread that stay caught up see one interleaving of durable and live events within one runtime: every event handed to several of them has the same relative placement for each. A subscription that attaches later misses earlier live events, and a slower one can lose live events (StreamEvent.LiveGap) or be disconnected as Lagging; the guarantee does not cover those cases. A commit made elsewhere reaches each subscription through that subscription's own store watch, through a hand-over's gap read, or, while the subscription is still in its initial replay (one created after the commit included), directly from that replay, so its position among this runtime's live events can differ from one subscription to another; each still sees the durable events in seq order. The guarantee holds when nothing outside this runtime commits to the thread; an external commit allows subscriptions to place it differently, not necessarily make them differ. A consumer that needs one order across runtimes orders only StreamEvent.Durable records by EventRecord.seq. It ignores StreamEvent.Live and StreamEvent.LiveGap, which have no seq, and treats StreamEvent.Disconnected as the end of the subscription, not as a record: it carries lastSeq and a reason.

Attributes

Companion
object
Source
GraphRuntime.scala
Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Constructors

def this(checkpointer: Checkpointer, clock: Clock, claims: ClaimPolicy)

For callers that cannot use default arguments (Java, Kotlin): WatchPolicy.default.

For callers that cannot use default arguments (Java, Kotlin): WatchPolicy.default.

Attributes

Source
GraphRuntime.scala
def this(checkpointer: Checkpointer, clock: Clock)

For callers that cannot use default arguments (Java, Kotlin): ClaimPolicy.default and WatchPolicy.default.

For callers that cannot use default arguments (Java, Kotlin): ClaimPolicy.default and WatchPolicy.default.

Attributes

Source
GraphRuntime.scala
def this(checkpointer: Checkpointer)

For callers that cannot use default arguments (Java, Kotlin): the system clock, ClaimPolicy.default and WatchPolicy.default.

For callers that cannot use default arguments (Java, Kotlin): the system clock, ClaimPolicy.default and WatchPolicy.default.

Attributes

Source
GraphRuntime.scala

Concrete methods

def checkpoint(threadId: ThreadId, checkpointId: String, config: RunConfig): Result[Checkpoint]

Checkpoint checkpointId of threadId's history (see history), the latest included. GraphError.CheckpointNotFound if it is not there; GraphError.TenantMismatch for a thread of another tenant.

Checkpoint checkpointId of threadId's history (see history), the latest included. GraphError.CheckpointNotFound if it is not there; GraphError.TenantMismatch for a thread of another tenant.

Attributes

Source
GraphRuntime.scala
def children(threadId: ThreadId, after: Option[ThreadId], limit: Int, config: RunConfig): Result[Vector[ChildThread]]

Up to limit of threadId's direct children (design 4.41), oldest first, starting after child after when it is given: pass the last id of a page to read the next (Checkpointer.children). Tenant-checked as history is: a thread of another tenant is GraphError.TenantMismatch. A thread with no checkpoint has no children; limit must be positive (ValidationError). Allowed on a child thread, whose own children are listed. Needs no claim.

Up to limit of threadId's direct children (design 4.41), oldest first, starting after child after when it is given: pass the last id of a page to read the next (Checkpointer.children). Tenant-checked as history is: a thread of another tenant is GraphError.TenantMismatch. A thread with no checkpoint has no children; limit must be positive (ValidationError). Allowed on a child thread, whose own children are listed. Needs no claim.

Attributes

Source
GraphRuntime.scala
def deleteThread(threadId: ThreadId, config: RunConfig): Result[Unit]

Deletes threadId from the runtime's store - its checkpoint, pending writes, event log and claim - so its id names a new thread again, and before it every thread below it (children, design 4.41): its children's children before its children, each under its own claim, all of them held before any is deleted. Refused, with nothing deleted, as admission refuses a run: a thread of another tenant is GraphError.TenantMismatch, a child thread GraphError.ChildThread (delete its parent), and one whose run - or a run of any thread below it - is admitting or executing in this runtime, or holds a live claim in any runtime, is GraphError.ThreadBusy. A delete that fails part-way leaves the thread and fewer descendants, the thread marked Checkpoint.deleting so that every admission refuses it with GraphError.ThreadDeleting; deleting again finishes it. The thread is held exclusively, and claimed in the store under config.runId, while it is deleted, so no run starts on it meanwhile; and the store's delete is fenced by that claim's token, so if the claim lapses before the delete lands - this runtime stalled past claims.ttl - and another run takes the thread over, nothing of that run is deleted: the call is ThreadBusy, or TenantMismatch if the run is another tenant's. An unknown thread is Right(()). The claim is renewed while the delete runs and given back as fork's and updateState's are, if the delete does not happen. As theirs, a claim found lost bounds the call: a delete not yet begun then is never sent, and is ThreadBusy; if the call is still waiting on the store 2 × claims.ttl after the loss it returns ThreadBusy naming no checkpoint, with the thread freed, and a delete already in the store may land afterwards. The tenant is checked once more after the store claim is granted, so a thread another runtime created for another tenant in between is TenantMismatch, not deleted. A claim this runtime failed to give back when a run on the thread ended is given back first, once the tenant has been checked, so that it does not refuse the delete until it lapses.

Deletes threadId from the runtime's store - its checkpoint, pending writes, event log and claim - so its id names a new thread again, and before it every thread below it (children, design 4.41): its children's children before its children, each under its own claim, all of them held before any is deleted. Refused, with nothing deleted, as admission refuses a run: a thread of another tenant is GraphError.TenantMismatch, a child thread GraphError.ChildThread (delete its parent), and one whose run - or a run of any thread below it - is admitting or executing in this runtime, or holds a live claim in any runtime, is GraphError.ThreadBusy. A delete that fails part-way leaves the thread and fewer descendants, the thread marked Checkpoint.deleting so that every admission refuses it with GraphError.ThreadDeleting; deleting again finishes it. The thread is held exclusively, and claimed in the store under config.runId, while it is deleted, so no run starts on it meanwhile; and the store's delete is fenced by that claim's token, so if the claim lapses before the delete lands - this runtime stalled past claims.ttl - and another run takes the thread over, nothing of that run is deleted: the call is ThreadBusy, or TenantMismatch if the run is another tenant's. An unknown thread is Right(()). The claim is renewed while the delete runs and given back as fork's and updateState's are, if the delete does not happen. As theirs, a claim found lost bounds the call: a delete not yet begun then is never sent, and is ThreadBusy; if the call is still waiting on the store 2 × claims.ttl after the loss it returns ThreadBusy naming no checkpoint, with the thread freed, and a delete already in the store may land afterwards. The tenant is checked once more after the store claim is granted, so a thread another runtime created for another tenant in between is TenantMismatch, not deleted. A claim this runtime failed to give back when a run on the thread ended is given back first, once the tenant has been checked, so that it does not refuse the delete until it lapses.

A subscription follows a thread, not its id. A subscribe in this runtime made before the delete, or while it is in flight, ends with Disconnected(lastSeq, DisconnectReason.ThreadDeleted) once it has delivered what it had queued. One in another runtime sharing the store ends the same way when it next reads the store: when its watch next polls (WatchPolicy), or, with the watch disabled, only when its own runtime next commits on the thread. A subscription in another runtime that has delivered no event yet - it is still at 0 - cannot be told from a new one, and follows the new thread instead - if it is its tenant's: one of another tenant ends with Disconnected(lastSeq, DisconnectReason.TenantMismatch) before its first event (subscribe). None of them delivers a new thread created under the same id as a continuation of the deleted one: its events are numbered on past the deleted thread's (Checkpointer.deleteThread); subscribe again from 0 to follow it.

Attributes

Source
GraphRuntime.scala
def fork(source: ThreadId, checkpointId: String, target: ThreadId, config: RunConfig): Result[Checkpoint]

Starts thread target from checkpoint checkpointId of thread source, and returns the new thread's first checkpoint: the source checkpoint's state, scheduled work, parked interrupts and status, with no parent, written by config.runId and recorded with a RunEvent.ThreadForked event, the first of the new thread's log. Continue it as its status says: start after Completed or Failed, recover after Running - which runs the checkpoint's pending tasks again, in the new thread, because pending writes are never copied - and resume after Suspended. The source thread is only read, so it may be running meanwhile, and is left unchanged.

Starts thread target from checkpoint checkpointId of thread source, and returns the new thread's first checkpoint: the source checkpoint's state, scheduled work, parked interrupts and status, with no parent, written by config.runId and recorded with a RunEvent.ThreadForked event, the first of the new thread's log. Continue it as its status says: start after Completed or Failed, recover after Running - which runs the checkpoint's pending tasks again, in the new thread, because pending writes are never copied - and resume after Suspended. The source thread is only read, so it may be running meanwhile, and is left unchanged.

Refused, with nothing changed: a source of another tenant is GraphError.TenantMismatch, a checkpoint not in its history GraphError.CheckpointNotFound; a target that is the source is a ValidationError, one that exists GraphError.ThreadExists (or TenantMismatch if it is another tenant's), and one held by a run GraphError.ThreadBusy. The target is claimed in the store while it is created; a fork whose claim lapses, or is taken over, before its commit lands is ThreadBusy too, with nothing created - as is one whose claim is found lost before its commit begins, which is then never sent. Once the claim is found lost, the call waits on the store at most 2 × claims.ttl after the loss: past that it returns ThreadBusy naming no checkpoint, with the target freed, and a commit already in the store may land afterwards, fenced by the claim's token.

The snapshot is copied as it is, so tool calls already issued at the forked checkpoint keep the source thread's idempotency keys, which the snapshot records (once #1739 lands, in their ToolTask, ApprovalRequest or ToolQuestionRequest); only model calls the new thread makes afterwards derive new keys, since the key includes the thread. This is intended: a fork that re-runs a call the source already issued is de-duplicated against the source's call by its key, rather than performing the side effect a second time.

Attributes

Source
GraphRuntime.scala
def history(threadId: ThreadId, before: Option[String], limit: Int, config: RunConfig): Result[Vector[Checkpoint]]

Up to limit of threadId's checkpoints, newest first - the latest is the first of the first page

Up to limit of threadId's checkpoints, newest first - the latest is the first of the first page

  • starting after checkpoint before when it is given: pass the last id of a page to read the next. Each is data; read its typed state with graph.restore(checkpoint.snapshot). The history holds every checkpoint the thread's runs, updateState and fork committed, except those prune removed and the earlier checkpoints of a turn a guardrail blocked (NodeResult.Block retracts its turn). The retraction leaves the closing Failed checkpoint, which holds whatever the blocking node's own Block update leaves in the state: Agent removes the turn there with MessageUpdate.RemoveTurn, and a custom blocking node must remove the content in its Block update likewise. A thread of another tenant is GraphError.TenantMismatch; a before not in the history is GraphError.CheckpointNotFound; a new thread has an empty history. limit must be positive (ValidationError). Reading needs no claim, so it never waits for a run.

Attributes

Source
GraphRuntime.scala
def prune(threadId: ThreadId, policy: RetentionPolicy, config: RunConfig): Result[Unit]

Removes the checkpoints and events of threadId that policy does not keep, by the store's clock (Checkpointer.prune): never the latest checkpoint or its pending writes, so what the thread does next is unchanged. Pruning a checkpoint also removes earlier events; maxAge and maxEvents may remove more, including the latest checkpoint's. Removing events raises the replay floor (GraphError.ReplayUnavailable). A thread of another tenant is GraphError.TenantMismatch; an unknown thread is Right(()). It needs no claim, so it may run while a run executes.

Removes the checkpoints and events of threadId that policy does not keep, by the store's clock (Checkpointer.prune): never the latest checkpoint or its pending writes, so what the thread does next is unchanged. Pruning a checkpoint also removes earlier events; maxAge and maxEvents may remove more, including the latest checkpoint's. Removing events raises the replay floor (GraphError.ReplayUnavailable). A thread of another tenant is GraphError.TenantMismatch; an unknown thread is Right(()). It needs no claim, so it may run while a run executes.

Attributes

Source
GraphRuntime.scala
def recover[I, O](threadId: ThreadId, graph: CompiledGraph[I, O], config: RunConfig, durability: Durability, observer: Option[Observer]): Result[RunHandle[O]]

Continues the incomplete execution on threadId; see the class description.

Continues the incomplete execution on threadId; see the class description.

Attributes

Source
GraphRuntime.scala
def resume[I, O](threadId: ThreadId, graph: CompiledGraph[I, O], answers: Map[InterruptId, Value], config: RunConfig, durability: Durability, observer: Option[Observer]): Result[RunHandle[O]]

Answers some of the suspended thread's interrupts and continues; see the class description. Encode a typed answer with ResumeRef.answer.

Answers some of the suspended thread's interrupts and continues; see the class description. Encode a typed answer with ResumeRef.answer.

Attributes

Source
GraphRuntime.scala
def start[I, O](threadId: ThreadId, graph: CompiledGraph[I, O], input: I, config: RunConfig, durability: Durability, observer: Option[Observer]): Result[RunHandle[O]]

Starts a run on threadId with input; see the class description.

Starts a run on threadId with input; see the class description.

Attributes

Source
GraphRuntime.scala
def subscribe(threadId: ThreadId, afterSeq: Long, capacity: Int, config: RunConfig, nested: Boolean)(listener: StreamEvent => Unit): Result[Subscription]

Replays events with seq > afterSeq, then delivers new events until cancelled or disconnected. The subscription is scoped to the thread, so it delivers every later run on it, and its dispatcher (a virtual thread, parked while idle) lives until Subscription.cancel; see there for how cancel waits on a running listener. Commits made through this runtime are handed over as they land; those of another runtime or process sharing the checkpointer are read from the store by the subscription's watch (WatchPolicy: a second virtual thread, which cancel stops too), in seq order and with the same durable guarantees, and are queued only as they fit, so they never make it lag. Where such a commit falls among this runtime's live events is not fixed: it depends on which read reached this subscription first (its initial replay, its watch, or a hand-over's gap read), so two subscriptions can place it differently. The guarantee that caught-up subscriptions in one runtime see one live/durable interleaving holds while nothing outside the runtime commits to the thread; an external commit allows it to differ rather than making it differ. Where one order across runtimes matters, order only StreamEvent.Durable records by seq, ignoring StreamEvent.Live and StreamEvent.LiveGap, and end at StreamEvent.Disconnected (#1920). Live progress of another runtime's runs is not delivered: it is never stored. With WatchPolicy.disabled, another runtime's commits are seen only by subscribing again, which replays the log - all of it once no commit a run of this runtime gave up on is still in the store on the thread; until then - for as long as the store's bound on a call lets that write take (Checkpointer) - the replay stops at the first event above what this runtime has handed over. A failed read of the store ends the subscription, after what is queued, with DisconnectReason.ReplayFailed. Returns without replaying: replay runs on the subscription's dispatcher thread, the only thread listener is called on, and a failed replay ends it with DisconnectReason.ReplayFailed. The subscription receives every live event sent after this returns: one sent while it replays is held - up to capacity, the rest dropped and counted as a StreamEvent.LiveGap - and delivered after the durable events committed before it and before those committed after it, as live events are once the replay is done. A listener that throws is disconnected (DisconnectReason.ListenerFailed). capacity bounds durable and live events separately, so live events never crowd out durable ones, and must be at least two: a live event is queued only while two live slots are free, one being reserved for the StreamEvent.LiveGap marker that precedes it, so with one slot no live event could ever be accepted.

Replays events with seq > afterSeq, then delivers new events until cancelled or disconnected. The subscription is scoped to the thread, so it delivers every later run on it, and its dispatcher (a virtual thread, parked while idle) lives until Subscription.cancel; see there for how cancel waits on a running listener. Commits made through this runtime are handed over as they land; those of another runtime or process sharing the checkpointer are read from the store by the subscription's watch (WatchPolicy: a second virtual thread, which cancel stops too), in seq order and with the same durable guarantees, and are queued only as they fit, so they never make it lag. Where such a commit falls among this runtime's live events is not fixed: it depends on which read reached this subscription first (its initial replay, its watch, or a hand-over's gap read), so two subscriptions can place it differently. The guarantee that caught-up subscriptions in one runtime see one live/durable interleaving holds while nothing outside the runtime commits to the thread; an external commit allows it to differ rather than making it differ. Where one order across runtimes matters, order only StreamEvent.Durable records by seq, ignoring StreamEvent.Live and StreamEvent.LiveGap, and end at StreamEvent.Disconnected (#1920). Live progress of another runtime's runs is not delivered: it is never stored. With WatchPolicy.disabled, another runtime's commits are seen only by subscribing again, which replays the log - all of it once no commit a run of this runtime gave up on is still in the store on the thread; until then - for as long as the store's bound on a call lets that write take (Checkpointer) - the replay stops at the first event above what this runtime has handed over. A failed read of the store ends the subscription, after what is queued, with DisconnectReason.ReplayFailed. Returns without replaying: replay runs on the subscription's dispatcher thread, the only thread listener is called on, and a failed replay ends it with DisconnectReason.ReplayFailed. The subscription receives every live event sent after this returns: one sent while it replays is held - up to capacity, the rest dropped and counted as a StreamEvent.LiveGap - and delivered after the durable events committed before it and before those committed after it, as live events are once the replay is done. A listener that throws is disconnected (DisconnectReason.ListenerFailed). capacity bounds durable and live events separately, so live events never crowd out durable ones, and must be at least two: a live event is queued only while two live slots are free, one being reserved for the StreamEvent.LiveGap marker that precedes it, so with one slot no live event could ever be accepted.

A subscription reads the thread, so it is tenant-checked as history is: a thread whose latest checkpoint is another tenant's than config.tenantId - or, for a thread with no checkpoint yet, whose run admitting or executing in this runtime is - is refused with GraphError.TenantMismatch before the subscription joins. A read of the thread that fails is returned as it is; one that throws is GraphError.RunCrashed, as for history. A thread can still change hands under its id once subscribed: one with no checkpoint yet is created by whichever tenant runs first, and one deleted through another runtime can be created again before the subscription learns of the delete. So every event is screened before it is delivered - by the tenant the event names, for the events a run, a fork or a state update begins with, and otherwise, until the first event delivered has confirmed the thread as the caller's, by its latest checkpoint's, read again then - and an event of another tenant's thread ends the subscription before it with Disconnected(lastSeq, DisconnectReason.TenantMismatch). A thread with no checkpoint any more ends it with ThreadDeleted, and a failed read with ReplayFailed.

Attributes

Source
GraphRuntime.scala
def updateMetadata(threadId: ThreadId, metadata: ThreadMetadata, config: RunConfig): Result[Unit]

Replaces threadId's metadata with metadata (design 4.52), which the thread registry searches by (Checkpointer.threads); the thread's checkpoints and state are unchanged. It is a change, held and fenced as updateState is: the thread is held in this runtime and claimed in the store, and the commit carries the claim's token, so a run holding the thread, here or in another runtime, refuses it with GraphError.ThreadBusy, as does a claim that lapses before the commit lands. A thread of another tenant is GraphError.TenantMismatch; a child thread GraphError.ChildThread; a thread whose cascade delete has begun GraphError.ThreadDeleting; a thread with no checkpoint a ValidationError (start it first, with RunConfig.metadata). Every refusal leaves the thread unchanged.

Replaces threadId's metadata with metadata (design 4.52), which the thread registry searches by (Checkpointer.threads); the thread's checkpoints and state are unchanged. It is a change, held and fenced as updateState is: the thread is held in this runtime and claimed in the store, and the commit carries the claim's token, so a run holding the thread, here or in another runtime, refuses it with GraphError.ThreadBusy, as does a claim that lapses before the commit lands. A thread of another tenant is GraphError.TenantMismatch; a child thread GraphError.ChildThread; a thread whose cascade delete has begun GraphError.ThreadDeleting; a thread with no checkpoint a ValidationError (start it first, with RunConfig.metadata). Every refusal leaves the thread unchanged.

Attributes

Source
GraphRuntime.scala
def updateState[I, O](threadId: ThreadId, graph: CompiledGraph[I, O], expected: String, update: StateUpdate, asNode: Option[NodeRef[_]], config: RunConfig): Result[Checkpoint]

Applies update to the committed state of threadId's latest checkpoint, which must be expected - the version the caller read; GraphError.CheckpointConflict otherwise, so an update never lands on a state its caller has not seen - and commits the result as a new latest checkpoint, recorded with a RunEvent.StateUpdated event, and returns it. Use it to correct a thread's state, for example before resume.

Applies update to the committed state of threadId's latest checkpoint, which must be expected - the version the caller read; GraphError.CheckpointConflict otherwise, so an update never lands on a state its caller has not seen - and commits the result as a new latest checkpoint, recorded with a RunEvent.StateUpdated event, and returns it. Use it to correct a thread's state, for example before resume.

Only the state changes: the new checkpoint keeps the status, scheduled tasks, open joins and parked interrupts, so the same call - start, recover or resume - continues the thread afterwards. With asNode, the update is checked as if that node had returned it: every key must be in its declared write set (GraphError.UndeclaredWrite), and the event names the node. Without it, every key must be one the graph registers (GraphError.UnknownStateKey). The update is applied with each key's update function, like any node's (GraphError.StateUpdateFailed).

The pending writes of tasks that completed against expected are carried over to the new checkpoint, so recover still does not run them again; their updates apply after this one. The new checkpoint is one superstep on from expected, so the supersteps a thread runs keep increasing and no later model call derives a tool idempotency key an earlier one used.

The thread is held - in this runtime and by a claim in the store, so the commit is fenced - while it is updated, and a run holding it refuses the update with GraphError.ThreadBusy, as does a claim that lapses, or is taken over, before the update's commit lands, or that is found lost before the commit begins - the commit is then never sent. Once the claim is found lost, the call waits on the store at most 2 × claims.ttl after the loss: past that it returns ThreadBusy naming no checkpoint, with the thread freed, and a commit already in the store may land afterwards. A thread of another tenant is GraphError.TenantMismatch; a thread with no checkpoint GraphError.CheckpointNotFound. Every refusal leaves the thread unchanged.

Attributes

Source
GraphRuntime.scala