PuPr: publish and qualify SFT, OPD, async GRPO, and TITO recipes #72

Goal

Publish and release-qualify four maintained Cortex Training journeys:

1. Conversational SFT using the public causal_cross_entropy processor path
2. On-policy distillation (OPD)
3. Asynchronous GRPO
4. Token-in/token-out (TITO) agentic RL

A user should be able to run each journey from a clean clone using only public instructions and pinned dependencies. Reuse existing Cortex Training and PrimeRL capabilities; this issue does not propose a new renderer framework or a broad async SDK redesign.

Audit baseline: main at da2d7ba (September 25, 2026).

Current state

1. Make SFT use causal_cross_entropy

The SFT recipe currently enables its processing block only when fused_lm_head_token_chunk_size is configured:

• _uses_chunked_logprob_loss
• conditional context and processing submission in train.py

None of the four checked-in SFT configs sets that field, so the public examples currently submit processing=None and use the dedicated labels/model-loss path. Chunking should be an implementation optimization, not the switch that chooses SFT semantics.

• Make all public SFT runs submit:

  {
      "loss_fn": "causal_cross_entropy",
      "post": ["compute_logprobs"],
      "config": {},
  }

• Always construct next-token-aligned targets and a weighted loss_mask for this path.

• Preserve renderer supervision weights rather than reducing all positive weights to an indistinguishable labels-only contract.

• Keep fused_lm_head_token_chunk_size as an optional/required profile optimization where needed for memory, independent of loss selection.

• Add focused tests for renderer-weight to next-token-mask alignment, padding, truncation, ignored tokens, and emitted forward-backward payloads.

• Qualify at least one advertised LoRA and one FullFT configuration through train, save, reload, and deterministic before/after generation.

• Reconcile this work with PR #34 and remove unsafe DeepSpeed-owned batch fields through PR #65 or its successor.

The legacy labels-only path may remain for compatibility, but maintained public SFT recipes must not depend on it.

2. Add an OPD recipe

Create recipes/distillation/on_policy/ with a README, entry point, recipe.yaml, and one supported configuration.

The recipe should:

• Create a student training sub-job, student sampling sub-job, and frozen teacher log_probability sub-job.
• Generate rollouts from the student policy.
• Score the exact prompt and sampled token IDs with the teacher through the public forward operation; do not decode and re-tokenize completions.
• Align teacher scores with next-token predictions and the training loss_mask, then provide teacher_log_probs_shifted to the existing GRPO/OPD processor configuration.
• Expose documented teacher_tau and clipping settings.
• Keep teacher identity fixed for the run and synchronize updated student weights only to the student sampler.
• Fail closed on missing, mis-sized, misaligned, or non-finite teacher results.
• Log teacher contribution/KL metrics and support a teacher_tau=0 comparison run.
• Add CPU contract tests plus a short end-to-end validation on the advertised profile.

This journey requires byte-based CortexTrainingClient.forward() to be end-to-end compatible. The current limitation is documented in docs/reference/rest-api.md.

3. Add an async GRPO journey

Publish a separate async GRPO recipe using the existing PrimeRL orchestration path. Do not turn the synchronous SDK example into a new hand-built scheduler.

• Add a pinned public install and launcher/config for one supported model and environment.
• Document bounded in-flight rollout work, max_async_level, policy/weight versions, and stale-rollout limits.
• Include weight synchronization, evaluation, checkpoint, reconnect, and resume behavior.
• Surface rollout policy version and observed policy lag in logs/metrics.
• Add config/contract tests and a short run proving rollout/training overlap, bounded lag, successful optimization, weight sync, and resume.
• Retain the existing synchronous Math GRPO recipe as the simpler step-by-step SDK example.

4. Add a TITO agentic-RL journey

Create one public multi-turn tool-use recipe using PrimeRL's existing TITO client and the existing PrimeIntellect renderers package.

• Choose one public agentic environment and pin its version, renderer version, and model/profile.
• Document environment installation, tool schema, verifier/reward, budgets, termination, and run/evaluation commands.
• Preserve exact prompt and completion token IDs across turns; do not decode/re-encode trajectory prefixes.
• Carry tool events and server-provided action masks into training.
• Reject environments or templates that do not satisfy TITO's linear-history/token-extension requirements.
• Add prefix-extension and no-decode/re-encode contract tests.
• Validate a run with multiple tool turns, non-empty trainable action masks, an optimizer step, checkpoint save, and resume.

A new Cortex-owned renderer package is explicitly out of scope for this work.

5. Make recipes reproducible and maintained

The current setup requires separate installation of mutable tinker-cookbook@nightly, while recipe-only dependencies such as chz, datasets, cookbook/renderers, environments, and optional logging integrations are not represented by one reproducible environment.

• Add a recipes optional dependency group or an equivalent pinned environment and document one clean-install command.
• Replace mutable branch dependencies with immutable releases or commit pins.
• Ensure each journey has a README, runnable entry point, checked-in config, recipe.yaml, provenance, expected output/metrics, troubleshooting, and a non-null last_validated after qualification.
• Add missing metadata for Math GRPO; currently only conversational SFT has recipe.yaml.
• Update recipes/README.md, model compatibility links, and human/agent instructions.
• Add a cheap static recipe validation command and clean-environment smoke coverage.
• Repair the checked-in model-catalog baseline before using it as a gate: profiles/inference.json references missing examples/api/glm-sampling.json, which currently causes the catalog validator and most catalog tests to fail from the same root error.

Related dependency/setup tracking: #44 and #50.

Scope boundaries

• No new renderer framework.
• No broad async SDK redesign.
• No new OPD kernel primitive if the existing processor passes qualification.
• No additional recipe families or exhaustive model/configuration matrix.
• Service/runtime fixes should remain independently owned; this issue owns public recipes and their validation.

Definition of done

From a clean environment, a user can follow only this public repository's instructions to run each journey on its advertised profile. Each recipe has pinned dependencies, local contract tests, recorded validation evidence, and a release-owned smoke test.

Specifically:

• SFT demonstrably submits causal_cross_entropy and preserves intended supervision masks.
• OPD scores exact student rollout tokens with a frozen teacher and never syncs student weights into it.
• Async GRPO demonstrates rollout/training overlap with bounded, observable policy lag.
• TITO preserves exact multi-turn tool tokens and action masks through training and checkpoint resume.