Skip to content

feat(config): warn on CLI overrides introducing unknown top-level keys - #2979

Open
n-dlms wants to merge 1 commit into
meta-pytorch:mainfrom
n-dlms:feat/config-warn-unused-args
Open

feat(config): warn on CLI overrides introducing unknown top-level keys#2979
n-dlms wants to merge 1 commit into
meta-pytorch:mainfrom
n-dlms:feat/config-warn-unused-args

Conversation

@n-dlms

@n-dlms n-dlms commented Jul 30, 2026

Copy link
Copy Markdown

Summary

When a CLI override (e.g. foobbar=1) introduces a top-level key that is not present in the YAML config, the user has likely either misspelled a YAML key or passed a kwarg the recipe does not support. Previously the merge happened silently via OmegaConf.merge(yaml_conf, cli_conf) and the user would believe the kwarg was active when the recipe ignored it.

This commit adds a post-merge warning that lists each unknown top-level key and surfaces close YAML-key candidates via a simple difflib + substring heuristic in _closest_yaml_keys. The warning is emitted through log_rank_zero so it appears only on rank zero.

What changed

  • torchtune/config/_utils.py
    • _merge_yaml_and_cli_args: collects top-level keys from each CLI override, tracks ones not present in the YAML top-level keys, and emits a single aggregated warning via log_rank_zero before the merge.
    • _closest_yaml_keys (new helper): difflib.SequenceMatcher ratio + substring containment reward, returns up to max_suggestions (default 3) YAML keys that look similar to the offending key.
  • tests/torchtune/config/test_config_utils.py
    • test_merge_warns_on_unused_cli_key: two unknown keys both surface in the warning; existing keys do not.
    • test_merge_no_warning_when_all_keys_known: clean override set produces no warning output.
    • test_closest_yaml_keys: typo "batch_siz" matches "batch_size"; unrelated keys return empty list; max_suggestions is respected.

Why this is the right place

The merge happens in _merge_yaml_and_cli_args (called from TuneRecipeArgumentParser.parse_known_args). By the time we reach the merge, the YAML keys are known (vars(yaml_args)) and the CLI dotlist is fully parsed. Tracking top-level keys during the CLI loop avoids any extra pass and adds zero overhead when no warning fires.

The implementation intentionally uses log_rank_zero (not logger.warning) because:

  1. Torchtune's distributed recipes call _merge_yaml_and_cli_args on every rank; emitting the warning on all ranks would spam logs.
  2. The existing log_config uses the same helper for the same reason (issue Log config to output only on rank zero #2700).

Limitations

  • Only top-level keys are tracked. Sub-keys of an existing component (e.g. model.lora_rank when model._component_ exists in YAML) are not flagged because they would produce too many false positives during early recipe prototyping.
  • The difflib heuristic is intentionally simple; it is a did-you-mean hint, not a guaranteed typo detector. False positives are possible; the warning copy acknowledges this: "If this was intentional ... you can ignore this warning".

Verification

  • python3 -m py_compile torchtune/config/_utils.py tests/torchtune/config/test_config_utils.py passes
  • python3 -m ruff check torchtune/config/_utils.py tests/torchtune/config/test_config_utils.py passes
  • _closest_yaml_keys standalone validation (without torchao import) produces expected outputs:
    • closest_yaml_keys("batch_siz", {batch_size,epochs,batch,optimizer,lr_scheduler}) -> ["batch_size", "batch"]
    • closest_yaml_keys("zzz", {batch_size,epochs}) -> []
    • closest_yaml_keys("batch", {batch_size,batch,optimizer}, max_suggestions=1) -> ["batch"]
  • Full test suite blocked locally by missing torchao dependency (per issue install instructions); new tests follow the existing pattern of mocking dist.is_available and get_logger.

Backward compatibility

  • New behavior is opt-in by triggering a CLI override with a YAML-unknown key. Existing recipes and configs are unaffected.
  • No public API change.
  • The warning is emitted via get_logger("WARNING") and routed through log_rank_zero; users who have configured their root logger to ERROR will not see it.

Fixes #1646

When a CLI override (e.g. 'foobbar=1') introduces a top-level key
that is not present in the YAML config, the user has likely either
misspelled a YAML key or passed a kwarg the recipe does not
support. Previously the merge happened silently and the user would
believe the kwarg was active when in fact the recipe ignored it.

This commit adds a post-merge warning that lists each unknown
top-level key and surfaces close YAML-key candidates via a simple
difflib + substring heuristic in _closest_yaml_keys. The warning is
emitted through log_rank_zero so it only appears on rank zero.

Added tests in tests/torchtune/config/test_config_utils.py:
- test_merge_warns_on_unused_cli_key verifies the warning fires for
  unknown top-level keys and does not include existing keys.
- test_merge_no_warning_when_all_keys_known verifies no warning is
  emitted when all CLI overrides correspond to YAML keys.
- test_closest_yaml_keys verifies the heuristic surfaces typo
  matches, returns empty list for wildly different keys, and
  respects max_suggestions.

Fixes meta-pytorch#1646
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feat] Warn users about unused arguments

1 participant