feat(failover): wait for ui target health and turn on quota checks

README.md

@@ -56,7 +56,7 @@ The Region Switch plan in [`deployment/failover.yaml`](deployment/failover.yaml)
 
 **What the plan does when it deactivates a Region**, in this order:
 
-1. It scales up the ECS services of the Region that stays, all six in parallel, to twice the highest count they reached in the last 24 hours (15 minutes allowed).
+1. It scales up the ECS services of the Region that stays, all six in parallel, to twice the highest count they reached in the last 24 hours (15 minutes allowed). For ui, the only service behind the load balancer, the step also waits until the new tasks pass the target group's health check, so DNS moves only onto tasks the load balancer sends requests to. The other five are reached through ECS Service Connect and have no target group, so there is nothing more to wait for.
 2. It moves DNS. The plan's Route 53 health check for the Region goes unhealthy, so the application's name resolves only to the Region that stays (5 minutes allowed).
 3. It switches the catalog database over. The writer of the Aurora global database moves to the Region that stays (10 minutes allowed). This is a switchover, which loses no data.
 
@@ -96,9 +96,13 @@ Every step reads before it writes, so running the command again after a partial
 
 Activating the Region in the ARC console runs only the DNS step. It does not put the capacity back or move the writer, so use `make failback`.
 
+**Service quotas in the Region that stays.** The plan compares the two Regions' quotas for what its scaling steps launch (such as Fargate vCPUs) within 30 minutes of being created or updated, and then once a day. A quota that is lower in one Region shows on the plan's Service quota warnings tab in the ARC console. A warning doesn't stop an execution, but the scale-up can fail when it reaches the lower quota. The execution role can read quotas and not request them, so ARC never files an increase, which would open a support case, on its own. To let it, add `servicequotas:RequestServiceQuotaIncrease` and `servicequotas:GetRequestedServiceQuotaChange` to the role, and `iam:CreateServiceLinkedRole` in an account that has never requested a quota increase.
+
+**What plan evaluation shows.** ARC checks the plan against your resources several times an hour, and shows the result in the console and in `aws arc-region-switch get-plan-evaluation-status`. For this plan it reads `actionRequired`, with one warning naming `servicequotas:RequestServiceQuotaIncrease` and `servicequotas:GetRequestedServiceQuotaChange`, the two permissions the role leaves out. A warning that names any other permission is a gap in the role. Plan evaluation tests the reads a quota check needs for services on EC2 container instances even when every service runs on Fargate, as here, so the role has them too: `ecs:DescribeContainerInstances`, `ec2:DescribeInstances`, `ec2:DescribeLaunchTemplateVersions` and `autoscaling:DescribeLaunchConfigurations`. It also tests `ecs:DescribeTasks` on a task outside any cluster, so that grant covers every task in the two Regions, not only the two clusters' tasks. With these reads the role can see every EC2 instance, launch template and launch configuration in the account, including the user data in launch templates and launch configurations.
+
 **Measurement and reports.** The plan has a recovery time objective of 10 minutes. ARC measures it from the start of an execution until the application health alarms, the `journey-global-*` alarms of both Regions, are green, and writes a report of each execution (the step timeline, the alarms' states and the recovery time against the objective) to the stack's reports bucket, named in the output `ReportsBucketName`, under `executions/`. Reports expire after 90 days. The plan also lists the other journey alarms (`journey-lcl-*`, `journey-rmt-*` and `region-degraded`) as trigger alarms.
 
-**How long it takes, measured three times.** An operator-started deactivate of us-east-1 on a test deployment took 6 minutes 18 seconds, which ARC reported as the recovery time (objective: 10 minutes). The six scale-ups ran in parallel and ended after 65 seconds (ui) to 195 seconds (carts, the service that starts slowest, which sets the pace); moving DNS took 2 minutes, because the step waits for Route 53 to show the Region's health check as unhealthy; the switchover took 1 minute. That run had no failure to detect, so a failover that an alarm starts adds the 2 to 3 minutes the alarms need, and the time ARC takes from the alarms to the start of the plan, which is not published. Two failovers that the triggers started confirmed it: in the `catalog-recovery` run the plan started 2 minutes 52 seconds after the fault began (12 seconds after the last alarm condition was met) and took 6 minutes 48 seconds, so the four `journey-global-*` alarms were green again 8 minutes 18 seconds to 9 minutes 40 seconds after the fault began, and users of the DNS name saw errors for 6 to 7 minutes; the failover in the `cart-isolation` run took 5 minutes 57 seconds. `make failback` took 4 minutes: the activation 2 minutes, the capacity reset seconds, the switchover back 1 minute 19 seconds. Task size and start-up time set these numbers, so measure your own with `make ngrh-test TEST=catalog-recovery` or an operator-started deactivate.
+**How long it takes, measured three times.** An operator-started deactivate of us-east-1 on a test deployment took 6 minutes 18 seconds, which ARC reported as the recovery time (objective: 10 minutes). The six scale-ups ran in parallel and ended after 65 seconds (ui) to 195 seconds (carts, the service that starts slowest, which sets the pace); moving DNS took 2 minutes, because the step waits for Route 53 to show the Region's health check as unhealthy; the switchover took 1 minute. That run had no failure to detect, so a failover that an alarm starts adds the 2 to 3 minutes the alarms need, and the time ARC takes from the alarms to the start of the plan, which is not published. Two failovers that the triggers started confirmed it: in the `catalog-recovery` run the plan started 2 minutes 52 seconds after the fault began (12 seconds after the last alarm condition was met) and took 6 minutes 48 seconds, so the four `journey-global-*` alarms were green again 8 minutes 18 seconds to 9 minutes 40 seconds after the fault began, and users of the DNS name saw errors for 6 to 7 minutes; the failover in the `cart-isolation` run took 5 minutes 57 seconds. `make failback` took 4 minutes: the activation 2 minutes, the capacity reset seconds, the switchover back 1 minute 19 seconds. Task size and start-up time set these numbers, so measure your own with `make ngrh-test TEST=catalog-recovery` or an operator-started deactivate. These runs predate the scale-up waiting for ui's health check. The target group counts a new ui task healthy after five passing checks 30 seconds apart (its defaults), about two minutes after the task starts, so ui now finishes around when carts did and the scale-up can take a little longer than measured here.
 
 **What the switchover costs users.** The Region that gives up the database writer fails its catalog-backed journeys for about a minute: its catalog gets `connection refused` while the cluster there changes roles, then connects again on its own, without restarting a task. In the failover that Region is the one traffic has just left: the global journeys, which go through the DNS name as users do, stayed at 100%, but the Region's own canaries failed for that minute, and so did the other Region's canaries that test it. In the fail-back it is the Region that has been serving, and DNS already names both Regions again, so some requests to it failed for that minute. One bad minute does not meet an alarm's two-of-three rule, but two do: the canaries that test the Region from outside (`journey-rmt-*`) went into alarm for two minutes after the failover. That did not meet a trigger, because a trigger also needs the Region's own journey alarm (`journey-lcl-*`), and one bad minute did not raise it.
 

deployment/failover.yaml

@@ -158,6 +158,50 @@ Resources:
                 Action:
                   - "ecs:ListServices"
                 Resource: "*"
+              # The ui scaling step waits for target health: ARC lists the service's tasks, reads each task's IP
+              # address, and reads the health of those targets in the service's target group.
+              - Effect: Allow
+                Action:
+                  - "ecs:ListTasks"
+                  - "elasticloadbalancing:DescribeTargetHealth"
+                Resource: "*"
+              # Every task in the two Regions, not only the two clusters' tasks: plan evaluation tests this read on a
+              # task outside any cluster (task/sq-permission-probe) and reports it missing unless the grant covers it.
+              - Effect: Allow
+                Action:
+                  - "ecs:DescribeTasks"
+                Resource:
+                  - !Sub 'arn:aws:ecs:${PrimaryRegion}:${AWS::AccountId}:task/*'
+                  - !Sub 'arn:aws:ecs:${StandbyRegion}:${AWS::AccountId}:task/*'
+              # Service quota checks: ARC reads each scaled service's task definition and capacity providers to work
+              # out which quotas it uses, then reads those quotas in both Regions. Reading only: without
+              # servicequotas:RequestServiceQuotaIncrease ARC files no increase, and so opens no support case, in the
+              # account that deploys the sample.
+              - Effect: Allow
+                Action:
+                  - "ecs:DescribeTaskDefinition"
+                  - "ecs:DescribeCapacityProviders"
+                  - "servicequotas:GetServiceQuota"
+                Resource: "*"
+              # Plan evaluation also tests the reads a quota check needs for services on EC2 container instances
+              # (the instances, and an Auto Scaling group's launch template or launch configuration), although all six
+              # services here run on Fargate, and reports each one the role lacks. With them, its only warning names
+              # the two increase-request permissions the role leaves out. They let the role read every EC2 instance,
+              # launch template and launch configuration in the account, including the user data of launch templates
+              # and launch configurations. The EC2 and Auto Scaling actions take no resource ARN, so "*" is the only
+              # resource they accept.
+              - Effect: Allow
+                Action:
+                  - "ecs:DescribeContainerInstances"
+                Resource:
+                  - !Sub 'arn:aws:ecs:${PrimaryRegion}:${AWS::AccountId}:container-instance/*'
+                  - !Sub 'arn:aws:ecs:${StandbyRegion}:${AWS::AccountId}:container-instance/*'
+              - Effect: Allow
+                Action:
+                  - "ec2:DescribeInstances"
+                  - "ec2:DescribeLaunchTemplateVersions"
+                  - "autoscaling:DescribeLaunchConfigurations"
+                Resource: "*"
               - Effect: Allow
                 Action:
                   - "application-autoscaling:DescribeScalableTargets"
@@ -230,6 +274,11 @@ Resources:
       # ARC reports each execution against this objective: the time from the start of the execution until the
       # application health alarms below are green.
       RecoveryTimeObjectiveMinutes: 10
+      # ARC compares the two Regions' quotas for what the scaling steps launch (such as Fargate vCPUs), within 30
+      # minutes of a create or update and then daily, and lists a quota that is lower in one Region as a warning. Set
+      # here because the default depends on when the plan was created, and an update keeps whatever the plan had. The
+      # role can read quotas but not request them, so ARC never files an increase on its own.
+      ServiceQuotaChecksEnabled: true
       ReportConfiguration:
         ReportOutput:
           # The bucket and prefix ReportsBucket holds; the execution role may write under executions/.
@@ -534,6 +583,11 @@ Resources:
                           TargetPercent: 200
                           CapacityMonitoringApproach: containerInsightsMaxInLast24Hours
                           TimeoutMinutes: 15
+                          # Done only when the load balancer counts the new ui tasks healthy, not when they run, so
+                          # DNS never moves onto tasks the load balancer won't send requests to. ui is the only
+                          # service behind a target group; the others are reached through Service Connect, and ARC
+                          # skips the check for a service without one.
+                          WaitELBTargetGroupHealthy: enabled
             - Name: shift-dns-traffic-away
               ExecutionBlockType: Route53HealthCheck
               ExecutionBlockConfiguration: