Feat: add bounded whole-run FIFO admission

docs/orchestrator.md

@@ -99,10 +99,13 @@ worker.submit(orchestration, args, config).wait()
 
 The current L2 backend is synchronous, so L2 `submit()` executes the existing
 blocking path and returns an already-completed handle. At L3 and above, graph
-callbacks remain serialized, but a later submit waits only for every dispatch
-in the prior run to cross its endpoint acceptance boundary. On A2A3 onboard,
-that boundary is after both device kernels are enqueued and before stream
-synchronization. Endpoints without an earlier signal fall back to completion.
+callbacks remain serialized, and what admits a later submit is a free pipeline
+slot rather than the prior run's acceptance: `begin_run` reserves a
+generation-safe lease before the callback is invoked and blocks there when the
+negotiated depth is already spent. Endpoint acceptance remains the launch fence
+a run's own dispatches advance — on A2A3 onboard, after both device kernels are
+enqueued and before stream synchronization, with endpoints lacking an earlier
+signal falling back to completion — but it no longer gates the next callback.
 Each run still owns its completion error, keepalives, and cleanup independently.
 
 Remote L3 submit adds two hidden pieces of metadata: final eligible worker-id
@@ -160,23 +163,28 @@ SubmitResult Orchestrator::submit_next_level(const CallableIdentity &callable,
         // NO_DEP: skip both
     }
 
-    // 4. Record fanin on self
-    s.fanin_count    = static_cast<int32_t>(producers.size());
-    s.fanin_released = 0;
-
-    // 5. Register with scope (holds slot open until scope_end releases ref)
+    // 4. Register with scope (holds slot open until scope_end releases ref)
     scope_.register_task(sid);          // increments s.fanout_total by 1
 
-    // 6. Attach fanout edges under each producer's mutex. Producers already
+    // 5. Attach fanout edges under each producer's mutex. Producers already
     //    completed do not count as live fanins; failed producers poison this
-    //    slot. Route an immediately READY slot through enqueue_ready().
-    attach_fanout_and_count_live_producers(sid, producers);
-    if (s.fanin_count == 0) {
-        s.state = TaskState::READY;
-        enqueue_ready(sid);
-    } else {
-        s.state = TaskState::PENDING;
+    //    slot.
+    int32_t live = attach_fanout_and_count_live_producers(sid, producers);
+
+    // 6. Publication: the one point where the slot leaves BUILDING. The count
+    //    and the transition are published together under fanout_mu — see
+    //    §8 "The BUILDING publication rule".
+    bool ready;
+    {
+        std::lock_guard<std::mutex> lk(s.fanout_mu);
+        s.fanin_count.store(live);
+        ready = s.fanin_released >= live;   // releases that landed while BUILDING
+        if (!s.state.compare_exchange_strong(BUILDING, ready ? READY : PENDING)) {
+            propagate_failure(sid);         // a producer claimed us mid-wiring
+            return {sid};
+        }
     }
+    if (ready) enqueue_ready(sid);          // outside the lock: takes runs_mu_
 
     // 7. Return handle
     return {sid};
@@ -556,13 +564,18 @@ insert, erase, and size operations across those threads.
 Each `TaskSlotState.state` progresses through:
 
 ```text
-FREE ──► PENDING ──► READY ──► RUNNING ──► COMPLETED ──► CONSUMED ──► FREE
-                             │               │
-                             └──────────────► FAILED ─────► CONSUMED ──► FREE
+FREE ─► BUILDING ─► PENDING ──► READY ──► RUNNING ──► COMPLETED ──► CONSUMED ──► FREE
+             │                │             │
+             └────────────────┴─────────────┴──► FAILED ─────► CONSUMED ──► FREE
 ```
 
 - **FREE**: slot in the ring pool, not allocated
-- **PENDING**: allocated; waiting on live fanin producers
+- **BUILDING**: allocated; submit owns it. Step 3 registers its outputs in the
+  TensorMap and step 5 appends it to its producers' fanout lists, so it is
+  observable from other threads well before its fanin and fanout counters are
+  final. No other thread may advance it — see
+  [the publication rule](#the-building-publication-rule) below
+- **PENDING**: published; waiting on live fanin producers
 - **READY**: all fanins satisfied; queued for Scheduler dispatch
 - **RUNNING**: dispatched to one or more endpoints
 - **COMPLETED**: worker(s) done; may still be referenced by fanout / scope
@@ -573,12 +586,86 @@ FREE ──► PENDING ──► READY ──► RUNNING ──► COMPLETED ─
 
 State transitions are driven by atomic operations:
 
-- Orch: FREE → PENDING at submit time
+- Orch: FREE → BUILDING at slot claim, BUILDING → PENDING/READY at publication
 - Scheduler: PENDING → READY → RUNNING during dispatch
-- Scheduler: RUNNING → COMPLETED or RUNNING/PENDING/READY → FAILED during
+- Scheduler: RUNNING → COMPLETED, or BUILDING/PENDING/READY → FAILED during
   completion / dependency poisoning
 - Scheduler/Orch cleanup: COMPLETED/FAILED → CONSUMED
 
+### The BUILDING publication rule
+
+Wiring a task to a producer has to happen under that producer's `fanout_mu` —
+otherwise a producer completing concurrently either misses the new consumer or
+releases one that has not counted it. That makes a half-built task reachable
+from another thread, and BUILDING is what tells that thread the counters are
+not final:
+
+- A **producer that fails** claims the slot through `claim_task_failure` and
+  **stops there**. It does not mark group members, poison consumers, or release
+  references — the submitting thread does, once its wiring is done. Running the
+  propagation from both sides releases every producer reference twice, which
+  reclaims a producer's output while the device is still reading it.
+- A **producer that completes** advances `fanin_released` against a
+  `fanin_count` submit has not published, and zero is a count any release
+  passes. Readiness is therefore one decision, not two readable facts:
+  `try_mark_ready` compares the pair and changes the state under `fanout_mu`,
+  and publication writes `fanin_count` together with the transition out of
+  BUILDING under the same lock. A producer that arrives first is re-evaluated
+  by the publication; one that arrives after reads a published count. Neither
+  can act on half of the pair, and exactly one enqueues the task.
+- **Dispatch** never claims a BUILDING slot: there are no final args or fanin
+  count to dispatch on.
+
+`cancel_unstarted_run` is the one caller that *does* propagate for a BUILDING
+slot, because submission is closed before it runs — a slot still BUILDING there
+is one whose submit threw part-way through wiring, and no submitting thread is
+left to publish it. The same applies one step later: a claim won from BUILDING
+sets `failure_propagation_pending`, and cancellation takes over any slot still
+carrying it. Without that, a submit that threw *after* being claimed would
+leave a FAILED slot cancellation skips, holding references the run's fence
+waits on forever.
+
+That debt is settled at one point, and it is not the start of the propagation.
+Everything up to the first reference release is ordered first and is
+idempotent — group marking skips already-terminal members, the run error is
+first-wins, the producer list is a copy — so an exception there leaves the debt
+for the next attempt to resume from. The releases that follow are not
+repeatable, which is why the debt is cleared immediately before them rather
+than after.
+
+### What a submit that throws leaves behind
+
+A submit charges its slot to the run before it can finish building it, so a
+throw part-way has to leave something the run's cancellation can fully reclaim.
+Two orderings carry that:
+
+- **The scope reference is registered before it is charged.** `fanout_total`
+  counts a release that `scope_end` will make; charging it before
+  `scope_->register_task` succeeded would leave a slot owing a release nothing
+  can make. Cancellation contributes only the terminal release, the threshold
+  is never reached, and the slot — and with it the run's task count and its
+  fence — never resolves.
+- **Both happen before anything that can throw**, and `fanout_total` is charged
+  before Step 2 publishes the slot's outputs, because a consumer that wires
+  onto it from that point on increments the same field.
+
+### The failure claim
+
+Every path that fails a task — a completion poisoning its consumers, a run
+cancelling its unstarted slots, and a submit that wires onto an already-failed
+producer — moves it to FAILED through `claim_task_failure`, and the winner is
+the only one that writes `failure_message`. A plain store could put a CONSUMED
+slot back to FAILED and then consume it, releasing its dependency references a
+second time.
+
+The claim, the message and the fanout snapshot are all taken under `fanout_mu`,
+so a task wiring itself onto a producer either registers before the claim and is
+poisoned by it, or observes FAILED — with its reason — and poisons itself. The
+same lock covers the ordinary terminal transition in `on_task_complete`: split
+from the snapshot, a consumer can wire itself in afterwards and never be
+released, or read COMPLETED, decline to count the fanin, and still be released,
+reaching READY one producer early.
+
 ### Fanout-release threshold
 
 Both paths that can trigger COMPLETED/FAILED → CONSUMED (the scheduler's

docs/scheduler.md

@@ -37,10 +37,19 @@ Both immediately-ready submissions and dependency-released consumers use
 this function. A directed task therefore cannot re-enter a shared
 NEXT_LEVEL queue after waiting in PENDING.
 
-Each `ReadyQueue` is a mutex-protected non-blocking FIFO. Root submission,
-worker completion, and stop requests notify the Scheduler condition variable;
-its wait predicate checks the completion FIFO, every ready queue, and the stop
-flag. Ready queues have no blocking pop or shutdown state.
+Each `ReadyQueue` is a mutex-protected non-blocking FIFO, partitioned by run:
+a task is enqueued under its own `run_id`, and the Scheduler pops only from the
+partition of the run that currently holds the FIFO head. That is what keeps two
+admitted runs from interleaving their device work while both are live. Root
+submission, worker completion, and stop requests notify the Scheduler condition
+variable; its wait predicate checks the completion FIFO, the dispatchable run's
+partitions, and the stop flag. Ready queues have no blocking pop or shutdown
+state.
+
+Popping a slot is not the same as owning it. A run whose graph callback throws
+fails and consumes its own unstarted slots, so the Scheduler claims each slot
+with an atomic `READY -> RUNNING` compare-exchange at the moment it dispatches;
+a lost claim means the cancelling path already owns that slot.
 
 ## 2. Scheduler loop
 

docs/task-flow.md

@@ -64,8 +64,9 @@ to C++:
 | `w3.submit_sub(handle, …)` dispatched to a SUB child | `LOCAL_PYTHON` | child resolves digest to a Python callable and calls `fn(args)` |
 
 All three paths share one mailbox wire format: `MAILBOX_OFF_CALLABLE` is
-reserved, and the 32-byte digest prefixes the args blob. The receiving child
-does the digest-to-slot resolve in its own address space.
+reserved, the run's generation-safe pipeline lease follows `CallConfig`, and
+the 32-byte digest prefixes the args blob. The receiving child does the
+digest-to-slot resolve in its own address space.
 
 The proposed remote L3 path keeps the same callable identity contract, but
 sends it in a versioned TASK frame. The remote endpoint resolves the digest
@@ -292,12 +293,52 @@ own AICore stream and retires it on every exit path, and no record of which
 image a stream last ran is load-bearing. The AICPU stream carries no such state
 and stays with its slot.
 
-This is dormant capacity at this layer. The ordinary synchronous entry point
-continues to use slot 0, and the chip child's mailbox loop passes no lease, so
-every production run is unleased. This contract does not enable a second
-mailbox frame, a second device execution, or cross-run publication overlap.
-Carrying a lease across the mailbox, and deciding when slot 1 may be leased at
-all, belong to whole-run admission.
+#### Whole-run FIFO admission
+
+L3 graph callbacks remain synchronous and serialized, and how many live run
+reservations native admission allows is the depth the child backends
+negotiated — not a constant. At the negotiated depth two:
+
+```text
+run N:     EXECUTING
+run N+1:   PREPARED
+run N+2:   blocked in begin_run before its graph callback
+```
+
+Where a backend publishes depth one — A5, or any runtime without a depth-two
+contract — the *second* submission is the one that blocks in `begin_run`, and
+there is no prepared successor at all. Code that relies on a later callback to
+unblock an earlier run deadlocks on such a backend.
+
+`begin_run` acquires a generation-safe lease before invoking the callback.
+The FIFO head may enter `EXECUTING` while its callback is still building, which
+preserves orchestration callbacks that submit device work and wait for L2
+communication before returning. A non-head run remains `BUILDING` or becomes
+`PREPARED` when graph construction closes; it cannot execute until every prior
+run is terminal. The scheduler observes only the ready-queue partition belonging
+to that single active FIFO head. A run's device effects therefore cannot
+interleave with another run even when both graphs contain ready tasks. TensorMap
+keys remain `(run_id, tensor_key)`, so adjacent runs may reuse the same tensor
+address without creating cross-run dependencies.
+
+The terminal transition releases the reservation and lease exactly once,
+wakes whichever submission was blocked on capacity, and activates the next prepared run. Empty
+runs take the same transition immediately. If graph construction fails, every
+unstarted slot is poisoned and consumed, its ready-queue partition is erased,
+and the lease is returned without dispatching device work.
+
+Each direct chip child publishes its runtime contract's `pipeline_depth` in
+the startup mailbox before `INIT_READY`. The parent configures admission to the
+minimum published depth. Backends without a depth-two contract therefore keep
+depth-one serial behavior instead of receiving an invalid slot-1 lease.
+
+Whole-run admission decides when a slot may be leased and carries the lease
+from `TaskSlot` through the chip mailbox into the runtime slot, so a production
+run now executes under the lease its run holds rather than unconditionally on
+slot 0. What this contract still does not enable is a second mailbox frame, a
+second device execution, or cross-run publication overlap: the endpoint remains
+a single synchronous round trip, and the scheduler dispatches only the run that
+holds the FIFO head and still owns its lease.
 
 Simulation implements the same depth, so the contract means the same thing on
 both platforms: its runner owns one arena bank and one retained temporary
@@ -382,6 +423,9 @@ Where the data goes after submit:
    Tags are consumed during the same submit call for dep inference and
    **never carried further**.
 3. `CallConfig` — copied into `slot.config` (parent heap, POD)
+4. `PipelineSlotLease` — copied from the owning run into
+   `slot.pipeline_lease`; local chip mailboxes forward `{slot_id, generation}`
+   to `ChipWorker::run_with_lease`.
 
 For the full submit mechanics (ring alloc, TensorMap lookup/insert, scope ref,
 fanout wiring), see [orchestrator.md](orchestrator.md).
@@ -390,7 +434,8 @@ fanout wiring), see [orchestrator.md](orchestrator.md).
 
 For local endpoints, after the Scheduler resolves the submitted NEXT_LEVEL
 target (or chooses an idle SUB worker), `LocalMailboxEndpoint` encodes
-`(callable digest, CallConfig, TaskArgs)` into the per-worker shm mailbox and
+`(callable digest, CallConfig, PipelineSlotLease, TaskArgs)` into the
+per-worker shm mailbox and
 the forked child decodes it. Remote NEXT_LEVEL dispatch through
 `RemoteL3Endpoint` serializes the same logical payload into a framed TASK
 request instead.