Repository navigation
feat(failover): wait for ui target health and turn on quota checks - #206
Merged
Merged
Conversation
The Region switch plan's scale-up of the Region that stays now waits, for ui, until the load balancer counts the new tasks healthy, and the plan compares the two Regions' service quotas. - scale-ui sets WaitELBTargetGroupHealthy: enabled, so DNS moves only onto ui tasks the load balancer sends requests to. ui is the only service behind a target group. The other five are reached through ECS Service Connect, and ARC skips the wait for a service without one, so they leave it unset. - The plan sets ServiceQuotaChecksEnabled: true. The default depends on when a plan was created, and an update without the field keeps the old setting, so the template says it. - The execution role gets what the wait reads (ecs:ListTasks and elasticloadbalancing:DescribeTargetHealth, and ecs:DescribeTasks on the two clusters' tasks) and what the quota check reads (ecs:DescribeTaskDefinition, ecs:DescribeCapacityProviders and servicequotas:GetServiceQuota). It cannot request an increase, so ARC opens no support case in the account that deploys the sample. - The README describes the wait and the quota checks, and says the measured failover timings predate the wait: a new ui task needs about two minutes to pass the health check. - Eight new tests pin the wait to the services behind a load balancer, the role's new reads and their resources, the absence of the increase permissions, and the README's description of both.
ARC's plan evaluation calls each read a quota check might need with the plan's role, on a resource named sq-permission-probe, and lists every read the role can't use as a missing permission. CloudTrail shows these calls for the e2e runs since 2026-10-14. The role now has all of them, so the one warning plan evaluation keeps names the two increase-request permissions the role leaves out. - ecs:DescribeTasks covers every task in the two Regions instead of the two clusters' tasks, because plan evaluation tests it on a task outside any cluster. - ecs:DescribeContainerInstances on every container instance in the two Regions, and ec2:DescribeInstances, ec2:DescribeLaunchTemplateVersions and autoscaling:DescribeLaunchConfigurations on "*", the only resource those three accept. They are the reads a quota check needs for services on EC2 container instances, and plan evaluation tests them although all six services run on Fargate. They let the role read every EC2 instance, launch template and launch configuration in the account, including the user data of launch templates and launch configurations. - The README says what plan evaluation shows, which reads the role has for it and what they expose, and that the first increase request in an account also needs iam:CreateServiceLinkedRole. - Tests pin each grant, check that its patterns match the names plan evaluation tests and the clusters' resources but not another Region's or account's, fail on any other EC2 or Auto Scaling action, and pin the README text.
tomljmj
approved these changes
Oct 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
WaitELBTargetGroupHealthy: enabledonscale-ui), so DNS moves only onto tasks that receive requests. ui is the only service behind a target group. The other five use ECS Service Connect, and ARC skips the wait for a service without one.ServiceQuotaChecksEnabled: trueexplicitly, because the default depends on when a plan was created and an update without the field keeps the old setting. ARC compares the two Regions' quotas for what the scaling steps launch (Fargate vCPUs here) and lists a lower one as a warning.ecs:ListTasksandelasticloadbalancing:DescribeTargetHealthon*, andecs:DescribeTaskson every task in the two Regions, because plan evaluation tests it on a task outside any cluster.ecs:DescribeTaskDefinition,ecs:DescribeCapacityProvidersandservicequotas:GetServiceQuota.ecs:DescribeContainerInstancesin the two Regions, andec2:DescribeInstances,ec2:DescribeLaunchTemplateVersionsandautoscaling:DescribeLaunchConfigurationson*, the only resource those three accept. These are for services on EC2 container instances; plan evaluation tests them although all six services run on Fargate. They let the role read every EC2 instance, launch template and launch configuration in the account, including the user data of launch templates and launch configurations.servicequotas:RequestServiceQuotaIncreaseandservicequotas:GetRequestedServiceQuotaChange.Testing
pytest tests/: 1,860 passed, 13 of them new..checkov.yaml: no findings.ec2:DescribeInstanceAttribute,ec2:GetLaunchTemplateData, every write tested and the increase-request permissions are denied.Not verified yet
actionRequiredwith a quota-check warning naming only the two increase-request permissions. After merge, test2's plan should show the same; any other permission named there is a gap in this change.