feat: validate LAMMPS template revision variables before task execution - #366
feat: validate LAMMPS template revision variables before task execution#366SchrodingersCattt wants to merge 8 commits into
Conversation
Add pre-check logic in LmpTemplateTaskGroup.make_task() that scans substituted templates for unreplaced V_* revision placeholders. This catches undefined revision variables at submit time (fail fast) instead of waiting until LAMMPS execution on remote cluster fails, which can waste hours of GPU training + queue time. Changes: - Add find_unreplaced_variables() to detect residual V_* patterns - Add check_revisions_completeness() with two checks: 1. Post-substitution residual check (raises ValueError) 2. Unused revision key detection (emits warning for typos) - Update test_lmp_empty to expect ValueError (previously would silently pass templates with unreplaced variables to LAMMPS) - Add TestRevisionVariablePrecheck test class with 5 test cases Closes: template variable typo wastes 75min train + queue time issue
dosubot
Bot
added
size:M
for more information, see https://pre-commit.ci
dosubot
Bot
added
size:L
|
Warning Review limit reached
Next review available in: 5 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughLAMMPS template task creation now detects unresolved ChangesRevision Placeholder Validation
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
dpgen2/exploration/task/lmp_template_task_group.py (1)
314-320: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueConsider using word boundaries for unused key detection.
Checking
if key not in template_rawuses simple substring matching. IfrevisionsdefinesV_NSTEPS, but the template only uses a longer variable likeV_NSTEPS_1, the substring match will still evaluate toTrue, inadvertently suppressing the unused key warning forV_NSTEPS.Since this is just a warning, it's not critical, but leveraging word boundaries ensures accurate matching.
💡 Proposed fix using regular expressions
# Check 2: Unused revision keys (warning only) if template_raw and revision_keys: for key in revision_keys: - if key not in template_raw: + if not re.search(rf"(?<![A-Za-z0-9_]){re.escape(key)}(?![A-Za-z0-9_])", template_raw): warnings.warn( f"Revision key '{key}' is defined but does not appear in the "🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@dpgen2/exploration/task/lmp_template_task_group.py` around lines 314 - 320, Update the unused revision-key check in the loop over revision_keys to match complete variable names rather than raw substrings in template_raw. Use a word-boundary-aware regular-expression search so a key such as V_NSTEPS does not match V_NSTEPS_1, while preserving the existing warning behavior for genuinely absent keys.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@dpgen2/exploration/task/lmp_template_task_group.py`:
- Around line 107-116: Update the check_revisions_completeness call in the
revisions validation block to validate every substituted template in conts, not
only conts[0]. Flatten or otherwise combine conts before passing it to the check
so PLUMED templates in conts[1] are checked whenever self.plm_set is enabled,
while preserving the existing revision keys and template_raw arguments.
---
Nitpick comments:
In `@dpgen2/exploration/task/lmp_template_task_group.py`:
- Around line 314-320: Update the unused revision-key check in the loop over
revision_keys to match complete variable names rather than raw substrings in
template_raw. Use a word-boundary-aware regular-expression search so a key such
as V_NSTEPS does not match V_NSTEPS_1, while preserving the existing warning
behavior for genuinely absent keys.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ea5e019e-b0fd-4dba-9673-28774b4ebd37
📒 Files selected for processing (2)
dpgen2/exploration/task/lmp_template_task_group.pytests/exploration/test_lmp_templ_task_group.py
Address CodeRabbit review: check_revisions_completeness was only called with conts[0] (LAMMPS templates), missing conts[1] (PLUMED templates). Now flatten all template variants before validation so V_* placeholders in PLUMED templates are also caught. Add test_plumed_template_undefined_variable_raises test case.
The ValueError for 'no revisions but template has V_*' broke existing tests (test_submit.TestSubmitCmdStd) that legitimately use templates with V_* placeholders without providing revisions (e.g., customized- lmp-template workflows where substitution is handled externally). Change to warnings.warn() instead of raise ValueError for the empty-revisions case. The hard ValueError is still raised when revisions ARE provided but incomplete (the important fail-fast case). Update tests to expect UserWarning instead of ValueError.
This reverts commit edd20b1.
Avoid false positives when V_* appears in comments (e.g., '# TODO: add V_PRESS support later'). The regex now only scans non-comment portions of each line. Add _strip_lammps_comments() helper and test_commented_variables_not_flagged.
| template_raw += "\n" + "\n".join(self.plm_template) | ||
| # Flatten all template variants (LAMMPS + PLUMED) for validation | ||
| all_conts = [c for c_list in conts for c in c_list] | ||
| check_revisions_completeness( |
There was a problem hiding this comment.
P1 — Validate placeholder names before substring substitution. This check runs only after make_cont() has called revise_by_keys(), which uses plain str.replace. If revisions defines V_TEMP but the template contains an undefined V_TEMPERATURE, substitution turns it into 300ERATURE; the residual regex then sees no V_* token, so the fail-fast feature silently misses the typo. Compare raw placeholder tokens against the revision-key set before substitution, and make replacement token-aware (or otherwise handle overlapping defined keys) so prefix collisions cannot destroy the evidence; add regressions for both an undefined longer token and two defined overlapping tokens. This needs coordinated changes across validation and substitution, so a single-line suggestion would be incomplete.
Codex quota is about to reset, so I am using the remaining token budget to review this PR now.
Coding agent: Codex
Codex version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning effort: xhigh
| stripped = [] | ||
| for line in lines: | ||
| # LAMMPS comments start with # (not inside quotes for our purposes) | ||
| idx = line.find("#") |
There was a problem hiding this comment.
P2 — Preserve # characters inside quoted LAMMPS strings. LAMMPS only starts a comment at an unquoted #, but this strips at the first # unconditionally. For example, print "# target V_MISSING" is executable template content; this helper removes V_MISSING before scanning, so an undefined placeholder is not reported. Please use quote-aware comment stripping (covering both quote forms and escapes as supported by the template grammar) and add regression tests for quoted hashes. A safe patch requires a small parser rather than a localized one-line replacement.
Codex quota is about to reset, so I am using the remaining token budget to review this PR now.
Coding agent: Codex
Codex version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning effort: xhigh
njzjz-bot
left a comment
njzjz-bot
left a comment
There was a problem hiding this comment.
The fail-fast validation has two false-negative paths on the current head: raw substring substitution can erase an undefined longer placeholder before validation, and comment stripping incorrectly treats quoted # characters as comment starts. I left exact inline reproductions and the required test cases. The targeted test module could not be collected locally because the active environment lacks dflow; the reported pre-commit and documentation checks pass.
Codex quota is about to reset, so I am using the remaining token budget to review this PR now.
Coding agent: Codex
Codex version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning effort: xhigh
2 participants
Problem
template.lammpsmay contain revision placeholders likeV_PRESSthat are not defined in therevisionsconfig dict. Currently this is only discovered when LAMMPS actually runs on the remote cluster and fails — potentially wasting hours of GPU training time plus queue wait.Solution
Add pre-check logic in
LmpTemplateTaskGroup.make_task()that validates revision variable substitution before task submission (fail fast):Post-substitution residual check: After applying
revise_by_keys(), scan output templates for remainingV_[A-Z][A-Z0-9_]*patterns. If found, raiseValueErrorwith a clear message listing undefined variables and available revisions.Unused key warning: If a revision key is defined but never appears in the template, emit
warnings.warn()to catch typos.Empty revisions with V_ in template*: If no revisions provided but template contains
V_*variables, raiseValueError.Design decisions
V_prefix is a strong convention from dpgen v1/v2 (100% of tests/examples use it). The check only scans for this pattern.${VARNAME}syntax is correctly ignored.Tests
TestRevisionVariablePrechecktest_lmp_emptyto expectValueErrorAll 9 tests pass.
Summary by CodeRabbit