Runs compiled graphs on durable threads.
A thread's latest checkpoint says what may happen next:
- none,
CompletedorFailed:startruns the graph with a new input (on a finished thread, over its committed state).Failedis a run that ended in a failure that is its outcome, not an interruption: a node returned NodeResult.Block, as a guardrail does. The thread is usable andrecoverhas nothing to continue; Running- work was scheduled when the last run stopped:recovercontinues it with no new input, reusing every pending write so completed tasks are not run again, and running failed or unstarted tasks again, each with its node's full RetryPolicy;Suspended:resumeanswers any non-empty subset of the parked interrupts; unanswered ones stay parked, and the run suspends again if nothing else can proceed.
Each call's RunConfig brings the run's static breakpoints; one on a node the graph does not have is refused with a ValidationError before the thread is read. A task they hold parks like a task that suspended, so the thread ends Suspended and resume answers it with Breakpoint.proceed.
Any other call is refused (GraphError.IncompleteRun, GraphError.PendingInterrupts, GraphError.NothingToRecover, GraphError.NotSuspended) without changing the thread. A thread belongs to the tenant on its latest checkpoint, and a call with another tenantId is refused with GraphError.TenantMismatch before anything else is reported - the thread's status, or that a run is live on it - and the error names only the caller's tenant, so it learns nothing about the thread.
A thread whose latest checkpoint records a RunOrigin is a child thread, started by a tool call of its parent's run (design 4.41). It is driven only through that run: start, recover, resume, fork, updateState and deleteThread on it are refused with GraphError.ChildThread, after the tenant check and before any claim, while history, checkpoint, subscribe and children read it as any thread. Deleting a thread deletes its children first, and child retention (RunConfig.withChildRetention) bounds how many settled children a thread keeps. Each run's terminal (Completed or Failed) checkpoint records how it settled (Checkpoint.outcome).
Each call is a new run, and claims the thread in two steps before it runs anything. First it asks the store for a RunClaim: a lease with an expiry and a FencingToken, which the run renews while it executes (see ClaimPolicy) and releases when it ends. While another run's claim is live - in this runtime or in any other runtime or process sharing the store - the call fails with GraphError.ThreadBusy naming that run, so recover elsewhere refuses a live run and takes over a dead one only once its claim has expired. Then it commits a checkpoint whose parent is the latest it read; if another run advanced the thread in between, it fails with ThreadBusy too. Either way its input or answers are neither accepted nor discarded. Every commit of the run carries its claim's token, and the store refuses one whose claim was taken over (GraphError.StaleClaim), so a run that lost its claim - a process paused past ttl - cannot record anything after its successor started. The run also judges its own lease, by its monotonic clock, so it learns of the loss even when no renewal reaches the store: once the lease passes with no renewal landing, or a renewal or commit is refused as stale, it stops at once - its tasks interrupted, as a cancel interrupts them - commits nothing more, and fails with GraphError.ClaimLost. A loss found once the run has begun committing its outcome sends no interrupt, but still skips every commit not yet begun; and whatever the run is doing, it ends within twice ttl of the loss: if it has not begun to by then - its thread stuck in a store call that ignores interrupts - it is ended without its thread, ClaimLost, and the thread freed. Admission is bounded the same way: a claim's commit still in the store twice ttl after the loss is given up, and the call fails ClaimLost - and if that commit then lands before the claim's release does, nobody having taken the thread over, it leaves its Running checkpoint and its RunStarted, RunRecovered or RunResumed event although no run continues it: recover does, once the release lands; one the store refuses as stale - the claim lapsed, or another run took it over, before it landed - fails ClaimLost too, never GraphError.CheckpointWriteFailed. So are fork, updateState and deleteThread, which hold a claim while they change the thread: a write not begun at the loss is never sent, and the call returns GraphError.ThreadBusy within twice ttl of it. A recover that takes over an expired claim records the run it took the thread from (RunEvent.RunRecovered). A call on a thread whose run is still executing in this runtime fails with ThreadBusy before reading the thread at all.
Admission - the checks above, restoring the thread and committing the claim - runs on the caller's thread, but for the read under the claim and the claim's commit, which run on llm4s-admit-<threadId> while the caller waits, and never throws: a Left means no run exists, and the thread is free again. A throwable from the store, the clock or the graph during admission is GraphError.RunCrashed, and an interrupt is CancelledError with the flag set. The run itself executes on a virtual thread owned by the runtime, named llm4s-run-<threadId>, behind the returned RunHandle; the thread stays busy until that run thread exits. An unexpected throwable escaping the run ends it with GraphError.RunCrashed.
RunHandle.cancel cancels the run by interrupting its run thread: every task is interrupted and joined, tasks that finished first keep their pending writes, an interrupted task records nothing, and the run ends Failed with GraphError.Cancelled - naming the last checkpoint, which a RunEvent.RunCancelled event is committed against. The checkpoint stays Running, so recover continues the run. A cancel that interrupts a superstep's commit ends the run the same way even if the store reports that commit as failed (as a JDBC store may, rather than throwing InterruptedException): the run is cancelled, not failed by its store, and the error names the last checkpoint that was durable. Once the run has begun committing its outcome - its completed or suspended checkpoint, or the RunFailed of a failed run - a cancel is ignored - it sends no interrupt, so that commit is not disturbed - and the run ends with its outcome. Interrupting a thread blocked in RunHandle.await does not cancel the run.
A run's closing wait is bounded while it holds its claim too (#1919). An Async run waiting for its queued commits, or an OnExit run making its exit commit, whose commit stays in the store for twice claims.ttl with none returning, ends GraphError.CommitTimedOut without waiting for it, through the same settling as a lost run: its result is set, its thread freed in this runtime, and its claim released on a detached thread. The commit is an orphan, fenced by the claim as a lost run's late write is: it lands only if the store applies it before the release, and the run's own listeners never receive it. A drain that keeps making progress is never cut short, however long it takes in all. Every other store call is waited for as long as it takes: every Checkpointer must return from each call within a bound it documents.
A run whose RunBudgets.timeout is set has a deadline, fixed when its claim commits; its tasks read it, by the runtime's clock, as RunContext.deadline. When it passes, the run stops the same way, but ends with GraphError.DeadlineExceeded and commits RunEvent.RunTimedOut. The first of cancel and expiry to be recorded wins, so a run reports exactly one. DeadlineExceeded is recoverable: recover with a new budget continues the run.
subscribe replays a thread's committed events after a sequence number and then delivers new ones as their commits succeed - only to a caller of the thread's tenant, as every other read of a thread is (see subscribe) - in ascending order with no gaps or duplicates, followed by live progress as it happens. Each subscription has its own dispatcher thread and a queue of capacity events, so a slow listener never holds up a run: one that falls behind by more than capacity durable events is disconnected (DisconnectReason.Lagging), and live events that do not fit are dropped and counted (StreamEvent.LiveGap). See EventHub. Live events are never replayed, but a subscription receives every one sent after subscribe returns, also while it is still replaying: so subscribing and then calling start loses none of the run's. One made after start returns can miss the run's first live events; an Observer passed to start, recover or resume is subscribed during admission, before the claim commits, and so sees every event of the run (RunHandle.observation).
A subscription also sees the durable events of commits made through another runtime or process sharing the checkpointer, with the same durable guarantees (ascending seq, no gap or duplicate): each live subscription watches the store (WatchPolicy, Checkpointer.awaitEventsAfter), so they arrive as soon as the store reports them - at once from a store that is told of commits, within pollInterval from one that is polled. Live progress events are never stored, so another runtime's do not arrive.
'''Ordering across runtimes.''' Concurrently active subscriptions to a thread that stay caught up see one interleaving of durable and live events within one runtime: every event handed to several of them has the same relative placement for each. A subscription that attaches later misses earlier live events, and a slower one can lose live events (StreamEvent.LiveGap) or be disconnected as Lagging; the guarantee does not cover those cases. A commit made elsewhere reaches each subscription through that subscription's own store watch, through a hand-over's gap read, or, while the subscription is still in its initial replay (one created after the commit included), directly from that replay, so its position among this runtime's live events can differ from one subscription to another; each still sees the durable events in seq order. The guarantee holds when nothing outside this runtime commits to the thread; an external commit allows subscriptions to place it differently, not necessarily make them differ. A consumer that needs one order across runtimes orders only StreamEvent.Durable records by EventRecord.seq. It ignores StreamEvent.Live and StreamEvent.LiveGap, which have no seq, and treats StreamEvent.Disconnected as the end of the subscription, not as a record: it carries lastSeq and a reason.
Attributes
- Companion
- object
- Source
- GraphRuntime.scala
- Graph
-
- Supertypes
