Repository navigation
feat(soak): drive and judge the steps of autoscaler mode Auto - #315
Conversation
The soak tester did not test mode Auto. Two instances in mode Auto get a load that gives a step, and the user paths of the autoscaler. The observer judges each step by the rules of the operator documentation. - internal/autoscale: the judges of the step size, the timeout of Applying, the cooldown, the budget and the back-off, and of restarts, recreates, handovers, shrinks and maxmemory above the limit. - observer: the judge of each autoscale instance, the counter redis_soak_autoscale_steps_total, and a window while a step is Applying. - mutator: seven autoscale kinds. Each kind tells the judge what it may do to the pods. A kind that found no step to disturb is skipped. - data: Load writes keys up to 97 percent of maxmemory. - deploy: the instances auto and auto-sent, and the right to delete the state ConfigMap. The full e2e profile leaves them out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4mmyo3sqLwf23m5FE2nBE
The README explains the step rate that the fixed rules of the operator allow, the user paths that may recreate or shrink a pod, and each judge. An autoscale kind also waits until the kubelet applied the limit of each pod spec, so that a lower limit cannot shrink a pod after the window. The tests cover the error paths of the load and of the kinds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4mmyo3sqLwf23m5FE2nBE
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8c2fbd39ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ct step - A step that stays Applying extended the window of a mutation at each round, so the mutation never timed out and the mutator stopped. After the step timeout of 18 minutes, the step keeps no window open. The finding autoscale_step_timeout stays. - The step rule gives one target. A step below it was no violation. - A user path that may shrink a pod also excuses autoscale_maxmemory in the next round, because the operator lowers maxmemory one reconcile later. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4mmyo3sqLwf23m5FE2nBE
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7bc0b24753
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…the lost state - A step in Applying with a lastStep that does not parse, or without lastStep, kept its window open for ever. The timeout then counts from the first round in Applying, 17 minutes. - autoscale_state_lost converged before the operator saw the loss, because the operator does not watch the ConfigMap and the instance did not change. The mutation now also waits until the operator wrote the state ConfigMap again, within its convergence timeout. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4mmyo3sqLwf23m5FE2nBE
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e980fdde1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
autoscale_limit and autoscale_memory_max excused a shrink also for a higher value. A lower pod limit in such a mutation was then no finding. A higher spec limit changes the template, so it may still recreate a pod. A memoryMax at or above the largest pod limit may change no pod. The largest limit is the largest applied pod limit, the effective limit or the spec limit. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4mmyo3sqLwf23m5FE2nBE
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f110dae3e3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The first round sees a generation, but no change of it. It gave the judge an allowance for all changes, so a restart, a recreate, a shrink or a handover in the next round of a step was no finding. This applies at each start of the tester and to each instance that it observes the first time. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4mmyo3sqLwf23m5FE2nBE
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aee7972a6b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- A Reverted step without a retryAfter that parses is an autoscale_step_rate violation. The back-off then counts from the revert that the tester saw, so a step in it is still a violation. - The judge autoscale_status reports an unknown phase, a lastStep that does not parse, is after the round, is before the last step or is missing in Applying or Reverted, and an effectiveMemory that is not a positive quantity. The size check skips such a step. - A lastStep in the future no longer moves the step timeout after the first round in Applying. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4mmyo3sqLwf23m5FE2nBE
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e28682e5ec
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… lastStep - A tester that starts at a Reverted step takes the back-off from retryAfter, or 1 hour after lastStep. A retryAfter that does not parse is an autoscale_step_rate violation. A Done step gives the cooldown from lastStep. - The judge follows each phase change, also without lastStep. The held step of a lost state has no lastStep, so its end kept the step active, and a later restart or handover was a false finding. - Only an adopted step in Applying counts its first result, because Held is a result. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4mmyo3sqLwf23m5FE2nBE
|
@codex review Generated by Claude Code |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f1030650ff
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…he killed pod - lastStep has a resolution of a minute. Applying after Done, Reverted or Held is a further step, also with the same lastStep, so its result, its size, its rate and its budget count. - A kill excused the recreate of each pod. The allowance names the killed pod, so the recreate or restart of another pod in the step is a finding. The handover of kill_master stays. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4mmyo3sqLwf23m5FE2nBE
|
@codex review Generated by Claude Code |
|
Codex Review: Didn't find any major issues. Swish! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Changes proposed on the PR:
auto(operator mode) andauto-sent(Sentinel mode) in mode Auto, because the soak tester did not test mode Auto. Both start at 128Mi with request equal to the limit,maxMemory35 percent withnoeviction,memoryMax384Mi. They need an operator with mode Auto (feat(autoscaler): mode Auto raises the redis memory limit in place #312 or later) and Kubernetes 1.33 or later. Against an older release candidate whose CRD rejects mode Auto, remove both instances.maxmemory, a master or replica kill while a step isApplying, and the user paths limit,memoryMax, mode and lost state. Each kind tells what it may do to the pods (recreate, hand over, shrink).memoryMax), the Applying timeout, cooldown, budget and back-off, and no restart, recreate, handover, shrink ormaxmemoryabove the limit without a user path. A violation is a finding (redis_soak_findings_total{invariant=autoscale_*}), so the existing alert fires.redis_soak_autoscale_steps_total{result}and a window while a step isApplying, becausemaxmemoryfollows one reconcile later.deleteon configmaps). The full e2e profile leaves the autoscale instances out. README: section "Autoscaler mode Auto".Notes: the operator constants are fixed, so the time cannot be compressed. A load to 97 percent gives a step in about 1 to 2 minutes when cooldown and budget allow; the budget of 6 steps in 24 hours gives about 1 step per hour on average. Not in this PR:
image_upgradeduring a step, an e2e profile (no CI change), dashboard panels, and provoking a revert orHeldstep on purpose.Tests: judge tests (one table per judge, each with a violation fixture), observer
TestAutoscaleStep,TestAutoscaleChaos,TestCollectRestarts, mutator autoscale tests, lossless cases,TestLoad, config tests,TestShippedAutoscale, and theTestShippedConfigsParseassertion. They do not compile on main. Not tested: a kind or e2e run on a cluster,make check-monitoring(needs Docker). The operator code is untouched, and no operator bug was found.🤖 Generated with Claude Code
https://claude.ai/code/session_01G4mmyo3sqLwf23m5FE2nBE
Generated by Claude Code