Skip to content

Decide checkpoint phase-awareness from the checkpoint name only - #1076

Open
arpitjain099 wants to merge 1 commit into
meta-pytorch:masterfrom
arpitjain099:fix/checkpoint-phase-detection
Open

Decide checkpoint phase-awareness from the checkpoint name only#1076
arpitjain099 wants to merge 1 commit into
meta-pytorch:masterfrom
arpitjain099:fix/checkpoint-phase-detection

Conversation

@arpitjain099

Copy link
Copy Markdown

Summary:

CheckpointPath._populate_from_str picks between the phase-aware and phase-naive regex by testing for train_step / eval_step / predict_step anywhere in the path string, which includes the user-supplied dirpath. A phase-naive checkpoint that happens to live under a directory containing one of those substrings gets routed onto the phase-aware regex, and there are two ways that goes wrong:

  • Silent corruption. foo/eval_step_sweeps/epoch_2_step_1_acc=0.98 parses, but (?:_(\w+)=(...))? swallows _step_1_acc=0.98 as the metric, so the result has step={} and a metric named step_1_acc.
  • Hard failure. foo/train_step_dir/epoch_2_step_1 (no metric) matches neither branch of the phase-aware regex and raises ValueError: Attempted to parse malformed checkpoint path.

Both are reachable from ordinary use. CheckpointPath.path produces exactly these strings, so the serialize then parse round-trip breaks; __setstate__ calls _populate_from_str, so unpickling a CheckpointPath raises, which matters because this class is broadcast across ranks; and get_latest_checkpoint_path / get_checkpoint_dirpaths / CheckpointManager discovery break for anyone whose output directory is named something like eval_step_sweeps.

The dirpath has no bearing on whether the checkpoint itself is phase-aware, so the fix looks at the final path component only.

Test plan:

Added two entries to the existing valid_paths table in CheckpointPathTest::test_from_str, one per failure mode above. test_from_str fails before the change and the file passes after: pytest tests/utils/test_checkpoint.py gives 28 passed on torch 2.13 CPU, Python 3.12.

_populate_from_str ran the train_step/eval_step/predict_step substring
test over the whole path, including the caller's dirpath. A phase-naive
checkpoint under a directory whose name contains one of those substrings
was routed onto the phase-aware regex, which either drops the step into
the metric-name group or fails to match at all.

Signed-off-by: Arpit Jain <arpitjain099@gmail.com>
@meta-cla meta-cla Bot added the cla signed label Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant