fix(grpo): replay buffer contract corruptions advantages (#2943)

[n-dlms]

Fixes #2943.

Root cause

The replay buffer storage counts rows (individual samples) as items, not batches. When PostProcessingWorker calls extend(trajectory) where trajectory has batch [16] and the buffer has max_size = replay_buffer_size = ${inference.batch_size} = 1, only the very last row survives. TrainingWorker.sample(batch_size=16) then draws 16 copies with replacement from that single row=E2=80=94producing the diagnostic [0.6057 =C3=97 16] identical advantages reported in #2943.

What changes

Change	Why
PostProcessingWorker.run stores whole batch as one buffer item (extend(traj.unsqueeze(0)) / sample(1).squeeze(0))	Capacity counts batches, not rows. Sampled batch is coherent, not a grab-bag of duplicate rows.
RayReplayBuffer batch_size=1 (was cfg.training.batch_size)	sample() without a specific count draws whatever the constructor said=E2=80=94with batch_size=16 that=E2=80=99s 16 duplicate copies of the whole batch.
Trajectory now carries per-sample rewards, successes, reward_func_names (tensors / NonTensorStack) instead of reward_outputs: list[RewardOutput]	The list-typed field caused batch-dimension validation failures in modern tensordict when the buffer stacks items from different iterations=E2=80=94and reward function names round-trip through the queue end-to-end for the debug table.
New helper reward_func_names_per_sample in postprocessing	The collector never labels which reward functions score a rollout; the postprocessing worker runs them anyway and RewardOutput.reward_base_name is available, so it fills the gap. (The old metadata["reward_Cmd"][idx] path would crash on ``None```=E2=80=94this gap was latent in the original code.)
Shared group_normalized_advantages in rewards.py	Same formula (normalize per group) used in three places kept conistent in a single call.

Plus resolved stale FIXMEs:

• context_length per-batch comment in grpo_step (truncate_sequence_at_first_stop_tokens pads rather than removes columns=E2=80=94pad_output=True ensures the context length is uniform per batch)
• Hardcoded torch.device("cuda:0") =E2=80/ cfg-supplied Device in TrainingWorker
• Sequence-IDs nesting resolved by the contract change (now flat per sample through the buffer)

Also added device: cuda to the async GRPO yaml and replaced the misleading buffer-size comment in that yaml.

Test coverage (all CPU-runnable, passing)

Two new test suites=E2=80=9429 tests total:

• test_replay_buffer_contract.py (5 tests) =E2=80=93 regression showing the exact [value=x_16] corruption; round-trip exactness of every per-sample field (questions, roofprobs, advantages, rewards, successes, heq_ids, answers, policy-ver, seq_lens); capacity-counts-batches, batch-repeat detection, group-normalization properties.
• test_async_data_path.py (24 tests) =E2=80=93 gradient-oracle telling us advantages always linline through the GRPO loss (``kl_coeff=0=E2=86=92 loss =F1=80=AE-advantages.mean(), grad doubling); fuzzed round-trip with varied (B,G,T,funcs,capacity) shapes; the Bug in async_grpo_full_finetune recipe: TrainingWorker receives identical advantages #2943` headline assertion =C3=ABadvantages differ across samples after the buffer (=C3=85) and reward-function names survive the buffer.

Existing rewards unit tests continue to pass. The GPU-gated test_postprocessing.py is mechanically updated for the new Trajectory fields but stays skipped in this environment (no GPU).

NOT verified here (cannot without vLLM + GPU + multi-node)

• Full end-to-end recipe run (worker actors, weight sync, FSDP, reference model)
• RayReplayBuffer remote-sampling path (the actor path =E2=80=94 tested only in local mode with the non-actor ReplayBuffer)
• Debug-table WandB rendering

Related follow-ups

• torchao incompatibility: current torchtune base install on torchao =E2=89=AE 0.9 fails because NF4Tensor moved out of torchao.quantization. A companion pin clause is warranted (separate PR).

• torchrl / SyncDataCollector: the async extra pins a stale git commit (<0.13); updating is separately tracked.