migrate runtime secrets from ssm to aws secrets manager

[Mr-Rahul-Paul]

Proposed change

Resolves #5071

Move runtime secrets from SSM Parameter Store to AWS Secrets Manager. Non-secret settings stay in SSM, and local development continues to use .env files.

Migration flow

This adds two migration modes:

• prepare: creates the new Secrets Manager secrets but keeps ECS reading the existing SSM secrets. This gives maintainers time to add and check the real values.
• complete: checks that every required secret has an AWSCURRENT value, switches ECS to Secrets Manager, and removes the old secret-valued SSM parameters.

The rollout is: run prepare in staging, add and verify the secret values, run complete, check staging, and then repeat for production.

What changed

• Added Secrets Manager resources for runtime secrets.
• Kept the existing database secret as the only source for the database password.
• Added a Secrets Manager secret for the Redis password.
• Updated backend, frontend, and scheduled ECS tasks to accept both SSM and Secrets Manager valueFrom references.
• Added the required Secrets Manager and KMS permissions to ECS execution roles.
• Added bootstrap permissions for the new /<project>/<environment>/* secret names.
• Added a deployment check that blocks complete mode when a secret has no AWSCURRENT value.
• Kept non-secret settings in SSM and local .env behavior unchanged.

After complete, seeing both SSM and Secrets Manager references in an ECS task definition is expected: SSM holds normal configuration and Secrets Manager holds credentials.

Testing

Terraform formatting and documentation

terraform fmt -recursive infrastructure
pre-commit run terraform-docs-go --all-files
terraform fmt -check -recursive infrastructure

The affected module READMEs were regenerated, and the final format check returned no errors.

Terraform tests

make test-infrastructure

Output: 378 passed, 0 failed across 17 modules, including 10 passed, 0 failed for the bootstrap IAM tests. This covered both migration modes, generated secret references, state moves, and the staging/production IAM namespace and policy-size checks.

Live module validation

terraform -chdir=infrastructure/live init -backend=false -reconfigure -input=false
terraform -chdir=infrastructure/live validate

This confirmed that the updated module inputs and outputs connect correctly in the full live configuration without accessing the remote backend.

testing with LocalStack

I used LocalStack fixture (not included in this PR) and fake values only.

First, I created the migration resources in prepare mode:

tflocal -chdir=infrastructure/localstack init -backend=false -input=false
tflocal -chdir=infrastructure/localstack apply \
  -auto-approve \
  -input=false \
  -var='runtime_secrets_mode=prepare'

I then listed both stores:

awslocal ssm get-parameters-by-path \
  --path /nest/localstack \
  --recursive \
  --query 'Parameters[].{Name:Name,Type:Type}' \
  --output table

awslocal secretsmanager list-secrets \
  --query 'SecretList[].Name' \
  --output table

The SSM output contained the existing String and secret-valued SecureString parameters, while Secrets Manager contained the new runtime-secret entries. This confirmed that prepare keeps the old source available while creating the new one.

I added different fake values to each externally managed secret using:

awslocal secretsmanager put-secret-value \
  --secret-id /nest/localstack/GITHUB_TOKEN \
  --secret-string 'fake-local-github-token'

I repeated this for the other external secrets. Each response contained:

VersionStages:
  AWSCURRENT

I ran the deployment preflight against LocalStack:

AWS_ACCESS_KEY_ID=test \
AWS_SECRET_ACCESS_KEY=test \
AWS_DEFAULT_REGION=us-east-1 \
AWS_REGION=us-east-1 \
AWS_ENDPOINT_URL=http://localhost:4566 \
bash .github/scripts/verify-runtime-secrets.sh localstack false

It first rejected a missing value, then passed after all required fake values had an AWSCURRENT version. This confirmed that complete deployments are blocked when a secret has not been populated.

I then applied complete:

tflocal -chdir=infrastructure/localstack apply \
  -auto-approve \
  -input=false \
  -var='runtime_secrets_mode=complete'

The Terraform plan removed the secret-valued SSM resources. A second SSM listing contained only non-secret String configuration.

Finally, I inspected the generated ECS task definition:

task_definition_arn=$(tflocal -chdir=infrastructure/localstack output -raw task_definition_arn)

awslocal ecs describe-task-definition \
  --region us-east-1 \
  --task-definition "$task_definition_arn" \
  --query 'taskDefinition.containerDefinitions[0].secrets' \
  --output table

The output showed Secrets Manager ARNs for credentials, :password:: for the database password JSON key, and SSM ARNs only for non-secret configuration. This matches the intended final state from #5071.

Final static checks

bash -n .github/scripts/verify-runtime-secrets.sh
shellcheck .github/scripts/verify-runtime-secrets.sh
git diff --check

All three commands passed. I also confirmed that no Terraform state, credentials, tokens, or real secret values are tracked.

The real prepare → complete migration still needs to be run and checked in staging before production by the maintainers.

Checklist

• Required: I followed the contributing workflow
• Required: I verified that my code works as intended and resolves the issue as described
• Required: I ran all required checks and tests locally; all warnings addressed and failures resolved
• I used AI for code, documentation, tests, or communication related to this PR

[coderabbitai]

Summary by CodeRabbit

• New Features
  • Added phased runtime secret migration with prepare and complete modes.
  • Runtime secrets can now be securely supplied through AWS Secrets Manager, including database and Redis credentials.
  • Deployment workflows can verify that required secrets are ready before completion.
• Documentation
  • Added guidance for configuring runtime secret migration and recovery settings.
• Bug Fixes
  • Improved secret access permissions and support for supported secret naming formats.
• Tests
  • Added coverage for migration modes, secret configuration, permissions, and deployment validation.

Walkthrough

The PR introduces a staged migration from SSM runtime secrets to AWS Secrets Manager. It updates Terraform resources, ECS secret injection, IAM permissions, deployment validation, state migration, documentation, examples, and tests.

Changes

Runtime Secrets Migration

Layer / File(s)	Summary
Deploy validation and secret verification .github/scripts/verify-runtime-secrets.sh, .github/workflows/run-deploy.yaml	Validates the migration mode and checks that required Secrets Manager secrets have an AWSCURRENT version before complete-mode deployment.
Parameters module secret resources and outputs infrastructure/modules/parameters/*	Adds Secrets Manager resources, conditional SSM resources, state migrations, mode-dependent ECS secret mappings, and Secrets Manager ARN outputs.
Cache Redis secret migration infrastructure/modules/cache/*	Adds the Redis Secrets Manager secret, gates the legacy SSM parameter by mode, and tests both migration modes.
Database credential secret migration infrastructure/modules/database/*	Gates the legacy database password parameter, preserves state addresses, exposes the credentials secret ARN, and updates tests and documentation.
Live infrastructure runtime secret wiring infrastructure/live/*	Adds the runtime mode input and connects parameters, cache, database, services, and tasks to the new secret outputs.
Service container secrets and IAM access infrastructure/modules/service/*	Replaces parameters_arns with container_secrets and adds conditional Secrets Manager and KMS permissions.
Tasks container secrets and IAM access infrastructure/modules/tasks/*	Renames task secret mappings, propagates them through task modules, and adds conditional execution-role permissions and tests.

Bootstrap IAM Namespace Refactor

Layer / File(s)	Summary
Bootstrap IAM namespace expansion infrastructure/bootstrap/main.tf, infrastructure/bootstrap/tests/unit.tftest.hcl	Shortens IAM statement identifiers, expands Secrets Manager ARN matching, and tests staging and production policy coverage.

Estimated code review effort: 5 (Critical) | ~90+ minutes

Possibly related PRs

• OWASP/Nest#5122: Shares Terraform integration test changes in the parameters module.
• OWASP/Nest#5205: Shares bootstrap IAM policy and Secrets Manager ARN scope changes.
• OWASP/Nest#5230: Shares parameters module integration test changes.

Suggested reviewers: kasya, arkid15r

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the migration of runtime secrets from SSM to AWS Secrets Manager.
Description check	✅ Passed	The description directly explains the migration design, rollout modes, affected infrastructure, testing, and operational constraints.
Linked Issues check	✅ Passed	The changes satisfy the coding objectives for Secrets Manager migration, ECS injection, duplicate removal, IAM access, and staged rollout [#5071].
Out of Scope Changes check	✅ Passed	The changes remain focused on runtime secret migration, supporting IAM, deployment validation, documentation, and related tests.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Contribution validation failed:

• commit_sign_off: One or more commits are missing or have an invalid Signed-off-by trailer.
  • 9af3bed
  • b5626bd
  • 6fb0f8a
  • 9ca6a7f
  • a637a22
  • b25f82f
  • 5933e77
  • 7cf68d4
  • b98be19
  • b711bb8
  • 77b4f87
  • 73bf0a3
  • 42858a5
  • d6c361e
  • 502e243
  • e5d5418
  • c7b6c9d
  • 88d67f7
  • 3fec885
  • 89427d9
  • bdd5a26
  • bfefc74
  • 6e9899e
  • e8da19a
  • d316805
  • fe4ed52
  • 6eada6d
  • f8fce46
  • 3e6bcec

[Mr-Rahul-Paul]

I shortened the Management suffix to Mgmt because AWS managed policies have a 6,144-character limit.
shortening them was the smallest low-risk fix.

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

<sub>Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic</sub>

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

<sub>Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic</sub>

[github-actions]

Contribution validation failed:

• commit_sign_off: One or more commits are missing or have an invalid Signed-off-by trailer.
  • 0a20148
  • 948894e

[cubic-dev-ai]

All reported issues were addressed across 2 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[github-actions]

Contribution validation failed:

• commit_sign_off: One or more commits are missing or have an invalid Signed-off-by trailer.
  • e4bbcc8

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

infrastructure/modules/tasks/main.tf (1)

75-85: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Allow kms:Decrypt via SSM as well.

Both execution-role policies need KMS access for SecureString parameters, but the kms:ViaService condition only allows Secrets Manager. Add ssm.${var.aws_region}.amazonaws.com in:

• infrastructure/modules/tasks/main.tf
• infrastructure/modules/service/main.tf

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@infrastructure/modules/tasks/main.tf` around lines 75 - 85, Update the
kms:Decrypt policy condition in infrastructure/modules/tasks/main.tf lines 75-85
and infrastructure/modules/service/main.tf lines 276-293 to allow both Secrets
Manager and SSM via-service values, including
ssm.${var.aws_region}.amazonaws.com alongside the existing Secrets Manager
service.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@infrastructure/modules/tasks/main.tf`:
- Around line 75-85: Update the kms:Decrypt policy condition in
infrastructure/modules/tasks/main.tf lines 75-85 and
infrastructure/modules/service/main.tf lines 276-293 to allow both Secrets
Manager and SSM via-service values, including
ssm.${var.aws_region}.amazonaws.com alongside the existing Secrets Manager
service.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 20660a26-1af9-420d-a8fb-6f7bfb46b87e

📥 Commits

Reviewing files that changed from the base of the PR and between 6fc617f and 42ab0e3.

📒 Files selected for processing (17)

• .github/workflows/run-deploy.yaml
• infrastructure/bootstrap/main.tf
• infrastructure/live/README.md
• infrastructure/live/main.tf
• infrastructure/modules/cache/README.md
• infrastructure/modules/cache/main.tf
• infrastructure/modules/database/README.md
• infrastructure/modules/database/main.tf
• infrastructure/modules/parameters/README.md
• infrastructure/modules/parameters/main.tf
• infrastructure/modules/parameters/tests/parameters.tftest.hcl
• infrastructure/modules/service/README.md
• infrastructure/modules/service/main.tf
• infrastructure/modules/tasks/README.md
• infrastructure/modules/tasks/main.tf
• infrastructure/modules/tasks/modules/task/README.md
• infrastructure/modules/tasks/modules/task/main.tf

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Copilot]

Pull request overview

This pull request implements the infrastructure-side migration of runtime secrets from SSM Parameter Store (SecureString) to AWS Secrets Manager, while keeping non-secret configuration in SSM and preserving local .env behavior. It introduces a two-phase rollout (prepare → complete) and updates ECS services/tasks and IAM policies to support Secrets Manager injection when the migration is completed.

Changes:

• Added runtime_secrets_mode and Secrets Manager resources/outputs across modules to support a staged migration (prepare keeps SSM secret injection; complete switches to Secrets Manager and removes legacy SecureString parameters).
• Updated ECS service/task modules to accept mixed valueFrom references (SSM + Secrets Manager) and to conditionally grant Secrets Manager + KMS permissions.
• Added a deployment preflight check (verify-runtime-secrets.sh) and workflow gating to block complete deployments when required secrets lack an AWSCURRENT version.

Reviewed changes

Copilot reviewed 39 out of 39 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
infrastructure/modules/tasks/variables.tf	Rename container secret input and add Secrets Manager ARN allowlist input.
infrastructure/modules/tasks/tests/unit.tftest.hcl	Add IAM policy assertions for optional Secrets Manager/KMS statements.
infrastructure/modules/tasks/README.md	Regenerate docs for new/renamed module inputs.
infrastructure/modules/tasks/modules/task/variables.tf	Rename task-level container secret input.
infrastructure/modules/tasks/modules/task/tests/unit.tftest.hcl	Update unit tests to use renamed task input.
infrastructure/modules/tasks/modules/task/README.md	Regenerate docs for renamed task input.
infrastructure/modules/tasks/modules/task/main.tf	Switch ECS secrets to use the renamed input map.
infrastructure/modules/tasks/main.tf	Extend execution role policy to optionally allow Secrets Manager + KMS decrypt.
infrastructure/modules/service/variables.tf	Rename container secret input and add Secrets Manager ARN allowlist input.
infrastructure/modules/service/tests/unit.tftest.hcl	Add IAM policy assertions for optional Secrets Manager/KMS statements.
infrastructure/modules/service/README.md	Regenerate docs for new/renamed service inputs.
infrastructure/modules/service/main.tf	Switch ECS secrets to use renamed input map and extend execution role policy.
infrastructure/modules/parameters/variables.tf	Add inputs needed for Secrets Manager migration (KMS key, db/redis secret ARNs, mode).
infrastructure/modules/parameters/tests/unit.tftest.hcl	Add tests for prepare/complete behavior and updated resource addressing.
infrastructure/modules/parameters/tests/integration.tftest.hcl	Update integration coverage to assert Secrets Manager secrets and KMS key usage.
infrastructure/modules/parameters/README.md	Regenerate docs for new Secrets Manager resources and outputs.
infrastructure/modules/parameters/outputs.tf	Replace SSM-only outputs with mixed valueFrom outputs and SM allowlist outputs.
infrastructure/modules/parameters/main.tf	Create Secrets Manager secrets (external + generated), gate legacy SSM secrets by mode, add state moves.
infrastructure/modules/database/variables.tf	Add runtime_secrets_mode input to gate legacy SSM password parameter.
infrastructure/modules/database/tests/unit.tftest.hcl	Add tests for complete-mode removal of legacy SSM param and DB credentials secret config.
infrastructure/modules/database/tests/integration.tftest.hcl	Add integration test coverage for DB secret/SSM behavior across modes.
infrastructure/modules/database/README.md	Regenerate docs for new inputs/outputs related to Secrets Manager.
infrastructure/modules/database/outputs.tf	Expose DB credentials secret ARN and make legacy SSM password ARN nullable.
infrastructure/modules/database/main.tf	Gate legacy SSM password parameter by mode and add moved block for indexed addressing.
infrastructure/modules/cache/variables.tf	Add runtime secrets mode and recovery window inputs for Redis secret migration.
infrastructure/modules/cache/tests/unit.tftest.hcl	Add tests for complete-mode removal and Redis secret configuration.
infrastructure/modules/cache/tests/integration.tftest.hcl	Add integration test coverage for Redis secret/SSM behavior across modes.
infrastructure/modules/cache/README.md	Regenerate docs for Redis Secrets Manager resources and new outputs.
infrastructure/modules/cache/outputs.tf	Expose Redis secret ARN and make legacy SSM password ARN nullable.
infrastructure/modules/cache/main.tf	Gate legacy SSM Redis password parameter by mode and add Secrets Manager secret + version.
infrastructure/live/variables.tf	Add runtime_secrets_mode variable to live stack inputs.
infrastructure/live/terraform.staging.tfvars.example	Default staging example to runtime_secrets_mode = \"prepare\".
infrastructure/live/terraform.production.tfvars.example	Default production example to runtime_secrets_mode = \"prepare\".
infrastructure/live/README.md	Regenerate docs for new live stack input.
infrastructure/live/main.tf	Wire new parameters/secret outputs into backend/frontend/tasks, and pass mode into modules.
infrastructure/bootstrap/tests/unit.tftest.hcl	Add assertions for Secrets Manager namespace permissions.
infrastructure/bootstrap/main.tf	Extend Terraform bootstrap IAM policy to manage both legacy and /project/env/* Secrets Manager namespaces.
.github/workflows/run-deploy.yaml	Add runtime mode validation + preflight verification and pass mode into Terraform.
.github/scripts/verify-runtime-secrets.sh	Add Secrets Manager preflight to require AWSCURRENT versions before complete deploys.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

infrastructure/modules/parameters/main.tf (1)

16-47: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Keep DJANGO_ALGOLIA_APPLICATION_ID as SecureString.

aws_ssm_parameter.type cannot change for an existing Parameter Store parameter. An existing staging or production environment will fail on prepare/apply if this hierarchical parameter type is changed from SecureString to String.

Retain type = "SecureString" for aws_ssm_parameter.django_algolia_application_id and its matching test assertion, or run a separate delete-and-recreate migration.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@infrastructure/modules/parameters/main.tf` around lines 16 - 47, Keep
aws_ssm_parameter.django_algolia_application_id configured with type =
"SecureString" in infrastructure/modules/parameters/main.tf:16-47; do not change
the existing parameter type to String. Update or retain the matching assertion
in infrastructure/modules/parameters/tests/unit.tftest.hcl:157-162 to expect
SecureString.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@infrastructure/modules/service/tests/unit.tftest.hcl`:
- Around line 15-20: Update infrastructure/modules/service/tests/unit.tftest.hcl
lines 15-20 to provide both SSM and Secrets Manager entries in container_secrets
and assert decoded aws_ecs_task_definition.main.container_definitions contains
matching name/valueFrom mappings; update
infrastructure/modules/tasks/modules/task/tests/unit.tftest.hcl line 7 with a
non-empty mapping and assert aws_ecs_task_definition.task.container_definitions
preserves each name and reference; update
infrastructure/modules/tasks/tests/unit.tftest.hcl lines 6-20 with a non-empty
mapping and assert every affected nested scheduled-task definition receives it.

---

Outside diff comments:
In `@infrastructure/modules/parameters/main.tf`:
- Around line 16-47: Keep aws_ssm_parameter.django_algolia_application_id
configured with type = "SecureString" in
infrastructure/modules/parameters/main.tf:16-47; do not change the existing
parameter type to String. Update or retain the matching assertion in
infrastructure/modules/parameters/tests/unit.tftest.hcl:157-162 to expect
SecureString.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 97dd9fec-fd24-4ecf-8456-3d11f6880a43

📥 Commits

Reviewing files that changed from the base of the PR and between 42ab0e3 and 2d51487.

📒 Files selected for processing (26)

• .github/workflows/run-deploy.yaml
• infrastructure/bootstrap/main.tf
• infrastructure/bootstrap/tests/unit.tftest.hcl
• infrastructure/live/README.md
• infrastructure/live/main.tf
• infrastructure/modules/cache/README.md
• infrastructure/modules/cache/main.tf
• infrastructure/modules/cache/tests/integration.tftest.hcl
• infrastructure/modules/cache/tests/unit.tftest.hcl
• infrastructure/modules/database/README.md
• infrastructure/modules/database/main.tf
• infrastructure/modules/database/tests/integration.tftest.hcl
• infrastructure/modules/database/tests/unit.tftest.hcl
• infrastructure/modules/parameters/README.md
• infrastructure/modules/parameters/main.tf
• infrastructure/modules/parameters/tests/integration.tftest.hcl
• infrastructure/modules/parameters/tests/unit.tftest.hcl
• infrastructure/modules/service/README.md
• infrastructure/modules/service/main.tf
• infrastructure/modules/service/tests/unit.tftest.hcl
• infrastructure/modules/tasks/README.md
• infrastructure/modules/tasks/main.tf
• infrastructure/modules/tasks/modules/task/README.md
• infrastructure/modules/tasks/modules/task/main.tf
• infrastructure/modules/tasks/modules/task/tests/unit.tftest.hcl
• infrastructure/modules/tasks/tests/unit.tftest.hcl

[cubic-dev-ai]

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}