Skip to content

fix: don't force-delete workers when end_date moves to past - #5188

Draft
cursor[bot] wants to merge 2 commits into
masterfrom
cursor/critical-bug-management-cdc4
Draft

fix: don't force-delete workers when end_date moves to past#5188
cursor[bot] wants to merge 2 commits into
masterfrom
cursor/critical-bug-management-cdc4

Conversation

@cursor

@cursor cursor Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

handle_end_date_change_for_challenge called delete_workers whenever a challenge's end_date was moved into the past. That force-deleted the ECS service even when submissions were still queued/running/resuming, undoing the pending-aware cleanup contract from #5179.

Trigger scenario

  1. Challenge has active Fargate workers and pending submissions.
  2. Host/admin sets end_date into the past (early close or date correction).
  3. Old signal path immediately called delete_workers.
  4. In-flight evaluations were killed; remaining queue messages had no consumers until a later submission recreated the stack.

Changes

  • Route past-end_date changes through schedule_challenge_cleanup_soon, which schedules the pending-aware cleanup Lambda shortly after now (EventBridge rejects past at() times).
  • The Lambda still checks pending submission count before tearing down ECS resources.
  • Regression tests for the signal path and the new scheduler helper.

Test plan

  • pytest tests/unit/challenges/test_aws_utils.py::TestScheduleChallengeCleanupSoon tests/unit/challenges/test_aws_utils.py::TestHandleEndDateChange — 11 passed
  • black --check / isort --check on changed files
Open in Web View Automation 

Summary by CodeRabbit

  • Bug Fixes
    • Improved cleanup scheduling when a challenge’s end date is moved into the past.
    • Cleanup now runs through the pending-aware workflow instead of immediately removing workers.
    • Added safeguards for missing configuration and development environments.
    • Existing cleanup schedules are updated when available, or created when needed.

cursoragent and others added 2 commits August 6, 2026 11:10
handle_end_date_change_for_challenge called delete_workers whenever
end_date was set in the past, killing queued/running submissions.
Route through the pending-aware cleanup Lambda instead so the queue
can drain first (#5179).

Co-authored-by: Rishabh Jain <rishabhjain2018@gmail.com>
Co-authored-by: Rishabh Jain <rishabhjain2018@gmail.com>
@codecov

codecov Bot commented Aug 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.29%. Comparing base (b269a39) to head (dc756d4).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
apps/challenges/aws_utils.py 86.95% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5188      +/-   ##
==========================================
- Coverage   91.30%   91.29%   -0.02%     
==========================================
  Files         115      115              
  Lines        8996     9019      +23     
==========================================
+ Hits         8214     8234      +20     
- Misses        782      785       +3     
Flag Coverage Δ
backend 93.82% <87.50%> (-0.03%) ⬇️
frontend 87.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Accounts & Authentication 97.40% <ø> (ø)
Challenges Management 96.12% <87.50%> (-0.08%) ⬇️
Job Processing 90.19% <ø> (ø)
Participants & Teams 99.54% <ø> (ø)
Challenge Hosts 100.00% <ø> (ø)
Analytics 100.00% <ø> (ø)
Web Interface 100.00% <ø> (ø)
Frontend (Gulp) 87.54% <ø> (ø)
All Models 97.37% <100.00%> (ø)
All Views 100.00% <ø> (ø)
All Serializers 98.20% <ø> (ø)
Utility Functions 97.18% <86.95%> (-0.12%) ⬇️
Core Configuration 78.94% <ø> (ø)
Files with missing lines Coverage Δ
apps/challenges/models.py 96.12% <100.00%> (ø)
apps/challenges/aws_utils.py 98.50% <86.95%> (-0.26%) ⬇️
Files with missing lines Coverage Δ
apps/challenges/models.py 96.12% <100.00%> (ø)
apps/challenges/aws_utils.py 98.50% <86.95%> (-0.26%) ⬇️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b269a39...dc756d4. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@RishabhJain2018

Copy link
Copy Markdown
Member

@coderabbitai Please review this!

@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown

@RishabhJain2018 I will review the changes in #5188.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

The change adds near-term EventBridge scheduling for expired challenge cleanup. It updates or creates schedules, skips unsupported environments, and logs AWS failures. Past challenge end dates now use this scheduler instead of immediate worker deletion.

Changes

Challenge cleanup scheduling

Layer / File(s) Summary
Near-term cleanup scheduler
apps/challenges/aws_utils.py, tests/unit/challenges/test_aws_utils.py
Adds schedule_challenge_cleanup_soon with UTC scheduling, EventBridge update and creation paths, configuration checks, debug-mode handling, and AWS error logging.
Expired challenge cleanup integration
apps/challenges/models.py, tests/unit/challenges/test_aws_utils.py
Routes past end dates with workers to pending-aware cleanup scheduling and verifies that delete_workers is not called.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ChallengeModel
  participant EventBridge
  participant CleanupLambda
  ChallengeModel->>EventBridge: schedule cleanup approximately one minute later
  EventBridge->>CleanupLambda: invoke cleanup Lambda
Loading

Possibly related PRs

Suggested reviewers: rishabhjain2018

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main behavior change: avoiding force-deletion of workers when end_date moves into the past.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch cursor/critical-bug-management-cdc4

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
tests/unit/challenges/test_aws_utils.py (1)

5564-5650: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Test non-not-found scheduler failures.

A non-ResourceNotFoundException from update_schedule executes the re-raise path and the outer error handler, but no new test covers that behavior. Add a test that injects a different ClientError, verifies that create_schedule is not called, and confirms that the helper returns normally.

As per path instructions, test code must cover edge cases.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/challenges/test_aws_utils.py` around lines 5564 - 5650, Extend
TestScheduleChallengeCleanupSoon with a test for schedule_challenge_cleanup_soon
where update_schedule raises a ClientError using a non-ResourceNotFoundException
code. Assert create_schedule is not called and the helper returns normally,
covering the outer error-handling path without changing existing tests.

Sources: Path instructions, Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/unit/challenges/test_aws_utils.py`:
- Around line 5643-5648: Update test_schedule_soon_skipped_in_debug to patch
valid scheduler configuration and get_boto3_client, then assert the client
factory is not called after schedule_challenge_cleanup_soon runs. Preserve the
DEBUG settings patch and ensure the test fails if the debug guard is removed.

---

Nitpick comments:
In `@tests/unit/challenges/test_aws_utils.py`:
- Around line 5564-5650: Extend TestScheduleChallengeCleanupSoon with a test for
schedule_challenge_cleanup_soon where update_schedule raises a ClientError using
a non-ResourceNotFoundException code. Assert create_schedule is not called and
the helper returns normally, covering the outer error-handling path without
changing existing tests.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 179d099c-cd29-4c35-b44d-afb3ec7015c8

📥 Commits

Reviewing files that changed from the base of the PR and between b269a39 and dc756d4.

📒 Files selected for processing (3)
  • apps/challenges/aws_utils.py
  • apps/challenges/models.py
  • tests/unit/challenges/test_aws_utils.py

Comment on lines +5643 to +5648
@patch("challenges.aws_utils.settings", DEBUG=True)
def test_schedule_soon_skipped_in_debug(self, mock_settings):
challenge = MagicMock()
challenge.pk = 42

schedule_challenge_cleanup_soon(challenge)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Assert the debug skip behavior.

test_schedule_soon_skipped_in_debug has no assertion. Patch valid scheduler configuration and get_boto3_client, then assert that the client factory is not called. The current test can pass if the debug guard is removed.

As per path instructions, test code must assert behavior and must not silently pass.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/unit/challenges/test_aws_utils.py` around lines 5643 - 5648, Update
test_schedule_soon_skipped_in_debug to patch valid scheduler configuration and
get_boto3_client, then assert the client factory is not called after
schedule_challenge_cleanup_soon runs. Preserve the DEBUG settings patch and
ensure the test fails if the debug guard is removed.

Source: Path instructions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants