feat(soak): drive and judge the steps of autoscaler mode Auto

test/soak/README.md

@@ -24,7 +24,9 @@ window. It counts these events in `redis_soak_findings_total`, and
   [Mutations](#mutations));
 - a version change that fails unsafely (see [Server versions](#server-versions));
 - a reset that does not complete within two times its timeout;
-- a Ready replica that never completed a sync.
+- a Ready replica that never completed a sync;
+- a step of the autoscaler that breaks a rule of mode Auto, also inside a
+  window (see [Autoscaler mode Auto](#autoscaler-mode-auto)).
 
 These events are also not expected, and have their own alerts:
 
@@ -380,6 +382,94 @@ Adoption has these limits:
   invariants hold. Adoption does not run. This gap is shorter than the start of
   the tester, but it exists.
 
+### Autoscaler mode Auto
+
+An instance with `autoscale` in the config gets a judge of the steps of the
+memory autoscaler. Its template must set `redis.autoscaling.mode: Auto`,
+`maxMemory` with `noeviction`, and a memory request equal to the limit. The
+instances `auto` and `auto-sent` (with Sentinels) do this. They need
+Kubernetes 1.33 or later.
+
+The rules of the operator are constants, so the tester cannot make a step
+come sooner. The operator samples the master one time in each minute. A use
+of 95 percent of `maxmemory` gives a step at the next sample, if the instance
+is `Healthy`, the cooldown passed, and the budget of 6 steps in 24 hours
+allows it. Thus a load of 97 percent of `maxmemory` gives a step after
+about 1 to 2 minutes. Mode `Off` and a lost state start the budget again,
+and the user paths lower the limit below `memoryMax`. With the shipped
+weights, expect about one step in each hour. The counter
+`redis_soak_autoscale_steps_total` shows the real rate.
+
+The template sets `maxMemory.percent` to 35. Thus the load fits two times in
+the limit, plus the operator reserve of 32Mi, for a fork. The fill is 10
+percent of `maxmemory`, so that a user path can lower the limit to the
+spec limit.
+
+The kinds, and what each kind may do to the redis pods:
+
+| Kind | Change | May |
+|------|--------|-----|
+| `autoscale_load` | Write keys up to 97 percent of `maxmemory`, wait at most 3 minutes for a step, and delete the keys. | nothing |
+| `autoscale_kill_master` | The load, and a graceful delete of the master while the step is `Applying`. | recreate, hand over |
+| `autoscale_kill_replica` | The load, and a graceful delete of a replica while the step is `Applying`, as a node drain does. | recreate |
+| `autoscale_limit` | Set the memory limit and request of the spec to a value in `autoscale.limit`. | recreate, shrink |
+| `autoscale_memory_max` | Set `memoryMax` to a value in `autoscale.memoryMax`. | recreate, shrink |
+| `autoscale_mode` | Change the mode in the cycle `Auto`, `Recommend`, `Off`. | `Off`: recreate, shrink |
+| `autoscale_state_lost` | Delete the state ConfigMap `rfa-<name>`. | nothing |
+
+A kill kind that sees no step `Applying` within the 3 minutes is `skipped`.
+The other mutation kinds, a chaos action, a stopped operator and a change of
+the generation by hand may do all of it.
+
+An autoscale kind converges when no step is `Applying`, the rollout is
+complete, and no resize is pending. Each pod must also run the limit of its
+own spec, and at least the limit of the StatefulSet. `maxmemory` must follow,
+and the instance must be `Healthy`. Mode `Off` also needs an empty
+`status.autoscaling` and each pod on the spec limit. A kind that may not
+recreate a pod must lose no write.
+
+Each violation of a judge counts one time in `redis_soak_findings_total`,
+also inside a window, as `oom_killed` does:
+
+- `autoscale_step_size`: a step raises the limit to one target: 25 percent
+  more, rounded up to a multiple of 64Mi, and `memoryMax` at most. A smaller
+  or a larger step is a violation. Because the operator rounds up, a small
+  limit gets a larger step: 64Mi goes to 128Mi.
+- `autoscale_step_timeout`: a step leaves `Applying` within 18 minutes after
+  `lastStep`. This is the timeout of 15 minutes, the minute of `lastStep`, and
+  2 minutes for the reconcile that ends the step. After this time, the step
+  keeps no window open, so the window of a mutation can time out.
+- `autoscale_step_rate`: a step starts at least 5 minutes after the end of
+  the last step. At most 6 steps start in 24 hours. After a revert,
+  `retryAfter` is at least the back-off of 1 hour, which doubles at each
+  further revert up to 24 hours. No step starts before `retryAfter`, unless
+  the generation changed.
+- `autoscale_restart` and `autoscale_handover`: while a step is `Applying`
+  or `Held`, no redis container restarts, no redis pod is recreated, and the
+  master does not change.
+- `autoscale_shrink`: the applied limit of a pod does not decrease.
+- `autoscale_maxmemory`: the `maxmemory` of a pod is not above its limit.
+
+A user path that may do the change excuses it. The excuse covers the round
+of the user path and the next round, because a change can show one round
+late. For `autoscale_maxmemory`, a user path that may shrink a pod excuses
+it, because the operator lowers `maxmemory` one reconcile later.
+
+The status gives its times in full minutes. Each rate check uses the latest
+time that the minute allows, and a slack of 30 seconds for the clocks and the
+age of a read. Thus only a sure violation is a finding. The status does not
+tell an urgent step from a sustained one, so the cooldown check uses the
+urgent cooldown of 5 minutes for each step.
+
+A step keeps a convergence window open while it is `Applying`, and in the
+round of its end. The operator sets `maxmemory` one reconcile after the last
+resize, so `config` can fail for a short time. `redis_soak_autoscale_steps_total`
+counts each step at its start, `started`, and at its first end: `done`,
+`reverted` or `held`.
+
+The judges, the planner and the convergence of each kind have unit tests
+with fakes: `go test ./internal/autoscale/ ./internal/observer/ ./internal/mutator/`.
+
 ### Chaos
 
 The chaos lane (`chaos.kinds`) does one action at a time on all instances
@@ -453,7 +543,9 @@ script asserts these conditions from the metrics:
 - chaos: every action converged, and no action found a change in progress.
 
 The versions profile leaves out `redis-chain-big`. Its 256Mi of data make
-each step and each reset too slow for the kind job.
+each step and each reset too slow for the kind job. The full profile leaves
+out the instances with `autoscale`, because a step of the autoscaler takes
+minutes. No profile runs them.
 
 The versions profile runs 3600 seconds by default. The instance `downgrade`
 needs 16 steps in that time. This holds only if each stuck change logs that
@@ -493,6 +585,10 @@ together with the window: inside a window, it is the cost of the change.
 
 ## Limits
 
+- The autoscale judges read the status of the RedisFailover, not the
+  metrics of the operator. The tester does not change the image during a
+  step, and it does not cause a revert or a held step on purpose, for
+  example with a ResourceQuota.
 - The tester does not check two documented operator limits: until a redis
   pod restarts after a password change, its exporter and its pre-stop
   `SAVE` use the old password; and an operator that restarts between a

test/soak/cmd/soak/main.go

@@ -117,12 +117,14 @@ func run(configPath, listen string, log *slog.Logger) error {
 			wg.Go(func() { rd.Run(ctx) })
 			d = rd
 		}
-		wg.Go(func() { o.Run(ctx) })
 		var mu *mutator.Mutator
 		if len(in.Mutations.Kinds) > 0 {
 			mu = mutator.New(in, cfg, kube, rfs, o, d, a, lock, insts[in.Name], m, log)
+			// Before Run, because the observer reads it without a lock.
+			o.SetMutation(mu.Allowed)
 			wg.Go(func() { mu.Run(ctx) })
 		}
+		wg.Go(func() { o.Run(ctx) })
 		// An outage counts for the chaos action, or else the mutation, that
 		// ran when it started.
 		r.SetEvent(func() string {

test/soak/deploy/config.yaml

@@ -414,3 +414,41 @@ instances:
         sentinel_image_flip: 1
       forceDeleteProbability: 0.5
       sentinelImages: [valkey-9, redis-7.2]
+  # Mode Auto of the memory autoscaler: the observer judges each step. A load
+  # gives a step at the next sample of the operator. The budget of 6 steps in
+  # 24 hours and memoryMax stop the steps. Mode Off and a lost state start
+  # the budget again, and the user paths lower the limit. The full e2e
+  # profile leaves these instances out.
+  - name: auto
+    template: instances.yaml
+    maxMemoryPolicy: noeviction
+    data:
+      fill: {percent: 10}
+      ledger: {}
+    autoscale:
+      limit: {min: 128, max: 192}
+      memoryMax: {min: 256, max: 384}
+    mutations:
+      kinds:
+        autoscale_load: 3
+        autoscale_kill_master: 1
+        autoscale_kill_replica: 1
+        autoscale_limit: 1
+        autoscale_memory_max: 1
+        autoscale_mode: 1
+        autoscale_state_lost: 1
+  # The same with Sentinels, for a master kill in a step in both modes.
+  - name: auto-sent
+    template: instances.yaml
+    mode: sentinel
+    maxMemoryPolicy: noeviction
+    data:
+      fill: {percent: 10}
+      ledger: {}
+    autoscale: {}
+    mutations:
+      kinds:
+        autoscale_load: 2
+        autoscale_kill_master: 1
+        autoscale_kill_replica: 1
+        autoscale_mode: 1

test/soak/deploy/instances.yaml

@@ -414,3 +414,37 @@ spec:
     resources:
       requests: {cpu: 50m, memory: 64Mi}
       limits: {memory: 192Mi}
+---
+# Mode Auto needs a memory request equal to the limit. maxmemory is 35
+# percent of the limit, so a load of 97 percent of maxmemory fits two times
+# in the limit, for a fork.
+apiVersion: databases.spotahome.com/v1
+kind: RedisFailover
+metadata:
+  name: auto
+spec:
+  sentinel:
+    enabled: false
+  redis:
+    replicas: 3
+    resources:
+      requests: {cpu: 50m, memory: 128Mi}
+      limits: {memory: 128Mi}
+    maxMemory: {percent: 35, policy: noeviction}
+    autoscaling: {mode: Auto, memoryMax: 384Mi}
+---
+apiVersion: databases.spotahome.com/v1
+kind: RedisFailover
+metadata:
+  name: auto-sent
+spec:
+  sentinel:
+    enabled: true
+    replicas: 3
+  redis:
+    replicas: 3
+    resources:
+      requests: {cpu: 50m, memory: 128Mi}
+      limits: {memory: 128Mi}
+    maxMemory: {percent: 35, policy: noeviction}
+    autoscaling: {mode: Auto, memoryMax: 384Mi}

test/soak/deploy/rbac.yaml

@@ -91,6 +91,11 @@ rules:
   - apiGroups: [""]
     resources: ["services", "configmaps"]
     verbs: ["get"]
+  # autoscale_state_lost deletes the state ConfigMap rfa-<name> of the
+  # autoscaler.
+  - apiGroups: [""]
+    resources: ["configmaps"]
+    verbs: ["delete"]
   # The tester and auth_add create the auth Secrets. A reset deletes and
   # creates them again. password_rotate_offline_forget deletes the Secret
   # rfr-auth-applied-<name> of the operator.

test/soak/e2e/full.jq

@@ -1,7 +1,8 @@
 # The full profile: the instances of deploy/config.yaml without a server
-# version, with a short mutation interval. The seed picks each kind of each
-# instance within its first nine steps. An auth kind counts only when auth
-# is on or off as the kind needs.
+# version and without autoscale, with a short mutation interval. The seed
+# picks each kind of each instance within its first nine steps. An auth kind
+# counts only when auth is on or off as the kind needs. A step of the
+# autoscaler needs minutes, so the autoscale instances are left out.
 del(.chaos, .versions, .edges, .mutation.timeouts)
 | .mutation += {interval: "10s", jitter: "10s", seed: 2646446, stopAfter: $stopAfter}
-| .instances |= map(select(.chain == null and .version == null))
+| .instances |= map(select(.chain == null and .version == null and .autoscale == null))