Skip to content

feat(failover): wait for ui target health and turn on quota checks - #206

Merged
mplaws merged 2 commits into
mainfrom
feat/region-switch-quota-checks-target-health
Oct 28, 2026
Merged

mplaws merged 2 commits into
mainfrom
feat/region-switch-quota-checks-target-health

Conversation

@mplaws

@mplaws mplaws commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • The deactivate workflow's ui scale-up now waits until the load balancer counts the new ui tasks healthy (WaitELBTargetGroupHealthy: enabled on scale-ui), so DNS moves only onto tasks that receive requests. ui is the only service behind a target group. The other five use ECS Service Connect, and ARC skips the wait for a service without one.
  • The plan sets ServiceQuotaChecksEnabled: true explicitly, because the default depends on when a plan was created and an update without the field keeps the old setting. ARC compares the two Regions' quotas for what the scaling steps launch (Fargate vCPUs here) and lists a lower one as a warning.
  • The execution role gets the reads the wait and the quota check use, and the reads ARC's plan evaluation tests for the quota check (CloudTrail shows plan evaluation making these calls in the e2e runs of 2026-10-14, 10-19 and 10-23):
    • ecs:ListTasks and elasticloadbalancing:DescribeTargetHealth on *, and ecs:DescribeTasks on every task in the two Regions, because plan evaluation tests it on a task outside any cluster.
    • ecs:DescribeTaskDefinition, ecs:DescribeCapacityProviders and servicequotas:GetServiceQuota.
    • ecs:DescribeContainerInstances in the two Regions, and ec2:DescribeInstances, ec2:DescribeLaunchTemplateVersions and autoscaling:DescribeLaunchConfigurations on *, the only resource those three accept. These are for services on EC2 container instances; plan evaluation tests them although all six services run on Fargate. They let the role read every EC2 instance, launch template and launch configuration in the account, including the user data of launch templates and launch configurations.
  • The role cannot request quota increases, so ARC opens no AWS Support case in the account that deploys the sample. Plan evaluation keeps one warning, naming servicequotas:RequestServiceQuotaIncrease and servicequotas:GetRequestedServiceQuotaChange.
  • The README describes the wait, the quota checks and what plan evaluation shows, and says the measured failover timings predate the wait: a new ui task needs about two minutes to pass the health check.

Testing

  • pytest tests/: 1,860 passed, 13 of them new.
  • Reverting each part of the change makes at least one new test fail (12 reverts across the two commits).
  • The pr-validation cfn-lint command (cfn-lint 1.58.0), and checkov 3.3.30 with .checkov.yaml: no findings.
  • IAM Access Analyzer validate-policy on the role's policy: no findings. Policy simulation: the resource names plan evaluation tests and the clusters' tasks and container instances are allowed in the two Regions; the same names in another Region, ec2:DescribeInstanceAttribute, ec2:GetLaunchTemplateData, every write tested and the increase-request permissions are denied.

Not verified yet

  • The e2e deploys the plan, doesn't run it, and deletes it about 20 minutes later, before ARC's first quota comparison (within 30 minutes of creation). The ui wait first runs at the next failover of a long-lived deployment.
  • ARC does evaluate the plan while it exists. On this PR's e2e, its quota-check calls with the plan's role should all succeed in CloudTrail, and plan evaluation should read actionRequired with a quota-check warning naming only the two increase-request permissions. After merge, test2's plan should show the same; any other permission named there is a gap in this change.

The Region switch plan's scale-up of the Region that stays now waits,
for ui, until the load balancer counts the new tasks healthy, and the
plan compares the two Regions' service quotas.

- scale-ui sets WaitELBTargetGroupHealthy: enabled, so DNS moves only
  onto ui tasks the load balancer sends requests to. ui is the only
  service behind a target group. The other five are reached through
  ECS Service Connect, and ARC skips the wait for a service without
  one, so they leave it unset.
- The plan sets ServiceQuotaChecksEnabled: true. The default depends
  on when a plan was created, and an update without the field keeps
  the old setting, so the template says it.
- The execution role gets what the wait reads (ecs:ListTasks and
  elasticloadbalancing:DescribeTargetHealth, and ecs:DescribeTasks on
  the two clusters' tasks) and what the quota check reads
  (ecs:DescribeTaskDefinition, ecs:DescribeCapacityProviders and
  servicequotas:GetServiceQuota). It cannot request an increase, so
  ARC opens no support case in the account that deploys the sample.
- The README describes the wait and the quota checks, and says the
  measured failover timings predate the wait: a new ui task needs
  about two minutes to pass the health check.
- Eight new tests pin the wait to the services behind a load balancer,
  the role's new reads and their resources, the absence of the
  increase permissions, and the README's description of both.
ARC's plan evaluation calls each read a quota check might need with the
plan's role, on a resource named sq-permission-probe, and lists every
read the role can't use as a missing permission. CloudTrail shows these
calls for the e2e runs since 2026-10-14. The role now has all of them,
so the one warning plan evaluation keeps names the two increase-request
permissions the role leaves out.

- ecs:DescribeTasks covers every task in the two Regions instead of the
  two clusters' tasks, because plan evaluation tests it on a task
  outside any cluster.
- ecs:DescribeContainerInstances on every container instance in the two
  Regions, and ec2:DescribeInstances,
  ec2:DescribeLaunchTemplateVersions and
  autoscaling:DescribeLaunchConfigurations on "*", the only resource
  those three accept. They are the reads a quota check needs for
  services on EC2 container instances, and plan evaluation tests them
  although all six services run on Fargate. They let the role read
  every EC2 instance, launch template and launch configuration in the
  account, including the user data of launch templates and launch
  configurations.
- The README says what plan evaluation shows, which reads the role has
  for it and what they expose, and that the first increase request in
  an account also needs iam:CreateServiceLinkedRole.
- Tests pin each grant, check that its patterns match the names plan
  evaluation tests and the clusters' resources but not another Region's
  or account's, fail on any other EC2 or Auto Scaling action, and pin
  the README text.
@mplaws
mplaws merged commit 2f8d4e6 into main Oct 28, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants