Runs queued entries (Stage 5 W2, design 4.56): it takes entries from a store's RunQueue and runs them on the agents it was built with, whose runtimes are over the same store, at most settings.concurrency at once. Built with AgentWorker.apply, started with start, stopped with shutdown.
For each entry it takes it:
holds the entry's lease, renewing it every lease.renewEvery beside the run's claim, which the run's runtime renews; a renewal the queue refuses (another worker took the entry over, or it closed) cancels the run, which then writes nothing more, rather than run on beside the entry's next attempt;
defers to a live claim on the thread - a run there, queued or direct, on any machine - by giving the entry back (RunQueue.release) without recording an attempt, so deferring never spends maxAttempts;
leaves an entry whose thread its agent's graph cannot take on - another graph version's run in flight, during a rolling upgrade - for a worker that can, passing over it for a while, and closes it Failed(Unrunnable) once it has waited settings.unrunnableAfter by the store's clock;
records the attempt with a new RunId.random() before touching the thread (rule 4), then acts on AttemptPlan: starts the input, recovers the thread, or closes the entry from the thread's state;
closes the entry from the thread's state once its run has ended at rest, or - for a run that failed - leaves it to come back when its lease expires, for an attempt that recovers it, until its attempts run out;
on AttemptStart.Exhausted, settles the thread first if its input was admitted - each tool call left without a result gets a synthetic one and a Failed checkpoint is committed under the thread's claim - and closes the entry Failed(AttemptsExhausted).
A crashed worker's entries come back when their leases expire, by the store's clock, and the worker that takes one recovers its thread, once the crashed run's claim - on the same clock - has expired too.
With settings.sweepEvery set it also runs the store-wide retention sweep (Checkpointer.sweep, with a guard that keeps every open entry's runs and its thread's tombstone) and reclaims closed entries past their ticket lifetime, one instance of the fleet at a time: the sweep holds a store claim on a reserved thread id while it runs.
The worker's readiness, for a service's probes: it is running, its loop has beaten within a bound and its last take succeeded, and the store answers a read of the checkpoints and of the queue, each within the store's bound.
The worker's readiness, for a service's probes: it is running, its loop has beaten within a bound and its last take succeeded, and the store answers a read of the checkpoints and of the queue, each within the store's bound.
Drains and stops the worker: it stops taking entries, lets its running runs reach a suspension or their end within settings.drainGrace, then cancels the rest and waits up to settings.exitTimeout for them to exit. A cancelled run's lease is given back (or its entry closed from how it ended) only once the run has exited, and its claim is released by its runtime as it exits, so another worker recovers it at once without overlapping its side effects. A run that has not exited by then keeps its claim and lease: renewing the lease stops here, the claim's renewal stops when the process exits, and both expire after their ttl, as after a crash. Returns at once, with an empty report, on a worker never started or already stopped.
Drains and stops the worker: it stops taking entries, lets its running runs reach a suspension or their end within settings.drainGrace, then cancels the rest and waits up to settings.exitTimeout for them to exit. A cancelled run's lease is given back (or its entry closed from how it ended) only once the run has exited, and its claim is released by its runtime as it exits, so another worker recovers it at once without overlapping its side effects. A run that has not exited by then keeps its claim and lease: renewing the lease stops here, the claim's renewal stops when the process exits, and both expire after their ttl, as after a crash. Returns at once, with an empty report, on a worker never started or already stopped.