diff --git a/enhancements/kube-apiserver/kms-encryption-foundations.md b/enhancements/kube-apiserver/kms-encryption-foundations.md index a79a1faff4..87aa57dde7 100644 --- a/enhancements/kube-apiserver/kms-encryption-foundations.md +++ b/enhancements/kube-apiserver/kms-encryption-foundations.md @@ -224,7 +224,7 @@ To enable the apiservers to access the KMS plugin, the `/var/run/kmsplugin` dire type: AppRole appRole: secret: - name: vault-approle # Secret in openshift-config namespace with roleID and secretID keys + name: vault-approle # Secret in openshift-config namespace with role-id and secret-id keys transitMount: transit transitKey: my-encryption-key ``` @@ -717,6 +717,263 @@ These rollup conditions (`KMSPluginsDegraded`, and any future `KMSPluginsAvailab The plan is to extend the existing [`conditionController`](https://github.com/openshift/library-go/blob/master/pkg/operator/encryption/controllers/condition_controller.go) in library-go's encryption controllers, which already emits the `Encrypted` condition on the same operator CR. It sits in the right call path (operator CR → ClusterOperator) and runs on the informer set the rollup needs. If extending it turns out to be a poor fit (conflicting sync triggers, unrelated dependencies that make the rollup hard to reason about), a dedicated controller will be introduced instead. +#### KMS Plugin Image Verification and Certification + +Vendors must sign their images using cosign, and run the certification +test suite described in this document before Red Hat can officially support +the vendor's KMS plugin. + +##### Image Signature and Verification + +We advise vendors to sign their images with cosign using public key encryption. +OpenShift must be able to verify image signatures in air-gapped clusters, and +public key encryption is best suited for this scenario. + +###### Signing Images with cosign v3 + +cosign v3 defaults to storing signatures as OCI 1.1 referrers (which are not +visible as tags). To create signatures compatible with OpenShift, vendors must +use the `--new-bundle-format=false` and `--use-signing-config=false` flags to +force creation of legacy `.sig` tags. + +**Step-by-step signing process:** + +1. **Generate cosign key pair**: + ```bash + cosign generate-key-pair + # Creates cosign.key (private, keep secure) and cosign.pub (public, distribute to customers) + ``` + +2. **Build and push your KMS plugin image**: + ```bash + # Build the image + podman build -t registry.example.com/kms-plugin:v1.0.0 . + + # Push to registry + podman push registry.example.com/kms-plugin:v1.0.0 + ``` + +3. **Get the image digest**: + ```bash + DIGEST=$(skopeo inspect docker://registry.example.com/kms-plugin:v1.0.0 | jq -r '.Digest') + echo "Image digest: $DIGEST" + ``` + +4. **Sign the image with compatibility flags**: + ```bash + COSIGN_PASSWORD= cosign sign \ + --key cosign.key \ + --new-bundle-format=false \ + --use-signing-config=false \ + registry.example.com/kms-plugin@${DIGEST} \ + --yes + ``` + +5. **Verify the .sig tag was created**: + ```bash + # Check for signature tag (should show sha256-.sig) + skopeo list-tags docker://registry.example.com/kms-plugin | jq -r '.Tags[]' | grep ".sig" + ``` + +**Why these flags are required:** +- `--new-bundle-format=false`: Forces cosign to use the legacy tag-based signature format instead of OCI 1.1 referrers +- `--use-signing-config=false`: Required when using `--new-bundle-format=false` to disable TUF-provided signing configuration + +**Note:** These flags will be removed in cosign v4. When containers/image adds +OCI 1.1 referrers support in future OpenShift versions, vendors can use cosign +v3+ default behavior without compatibility flags. + +###### Customer Configuration + +Users must create a `ClusterImagePolicy` resource to configure OpenShift to +verify the image signature every time CRI-O pulls a plugin image. Users should +obtain the vendor's public key from the vendor's official documentation (e.g., +HashiCorp's documentation for the Vault KMS plugin). + +`ClusterImagePolicy` example: +```yaml +apiVersion: config.openshift.io/v1 +kind: ClusterImagePolicy +metadata: + name: vault-kms-plugin + labels: + config.openshift.io/type: kms +spec: + scopes: + - docker.io/hashicorp/vault-plugin-kms # this must match the kmsPluginImage in the APIServer config + policy: + rootOfTrust: + policyType: PublicKey + publicKey: + # base64-encoded cosign public key (cosign.pub) + keyData: LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFLi4uCi0tLS0tRU5EIFBVQkxJQyBLRVktLS0tLQo= + signedIdentity: + matchPolicy: MatchRepository +``` + +To generate the base64-encoded public key for the `keyData` field: +```bash +cat cosign.pub | base64 -w 0 +``` + +When vendors release images signed with a new key, users must create a new +`ClusterImagePolicy` resource with the updated public key. During the transition +period, both old and new `ClusterImagePolicy` resources must exist in the +cluster until the previous `ClusterImagePolicy` is no longer needed (i.e when +migration to the new plugin image is finished). + +Users should refer to the vendor's official documentation for the most current +public key to use in their `ClusterImagePolicy`. + +##### Certification + +KMS plugin certification follows the CSI driver certification model. A +dedicated test suite will be added to the `openshift-tests` binary, which +vendors can extract from an OpenShift release payload and run in a test cluster +with their KMS plugin installed. + +The certification test suite validates: +- KMS plugin compliance with the Kubernetes KMS v2 API +- Encrypt/decrypt operations under various conditions +- Plugin behavior during upgrades and migrations + +###### Extracting the Test Binary and Running the Test Suite + +The KMS encryption test suite may take several hours to finish and is marked as +disruptive (it may affect cluster stability during testing). Vendors are +advised to run it in a dedicated test cluster within a CI/CD environment where +test output is persisted. The test output is required to validate KMS plugin +compatibility with OpenShift. + +**Prerequisites:** +- An OpenShift cluster (version matching the target certification version) +- KMS plugin configured via OpenShift APIServer configuration API +- `oc` CLI tool installed and authenticated to the cluster +- At least 1GB of disk space for test binary extraction and test output + +**Step 1: Extract the openshift-tests binary from the release payload** + +```bash +# Set the release image you want to certify against +RELEASE_IMAGE="quay.io/openshift-release-dev/ocp-release:4.22.0-x86_64" + +# Get the tests image reference from the release payload +TESTS_IMAGE=$(oc adm release info ${RELEASE_IMAGE} --image-for=tests) +echo "Tests image: ${TESTS_IMAGE}" + +# Extract the openshift-tests binary +mkdir -p ./kms-certification +oc image extract ${TESTS_IMAGE} \ + --path /usr/bin/openshift-tests:./kms-certification \ + --confirm + +# Make the binary executable +chmod +x ./kms-certification/openshift-tests + +# Verify extraction +./kms-certification/openshift-tests version +``` + +**Step 2: Run the KMS certification test suite** + +```bash +# Set your kubeconfig (if not already set) +export KUBECONFIG=/path/to/your/kubeconfig + +# Run the full KMS test suite +# Note: This may take several hours and is disruptive to the cluster +./kms-certification/openshift-tests run openshift/kms \ + --junit-dir=./kms-certification/junit \ + -o ./kms-certification/kms-test-output.log + +# The exit code indicates success (0) or failure (non-zero) +echo "Test run exit code: $?" +``` + +**Step 3: Review test results** + +```bash +# Check the test output log +less ./kms-certification/kms-test-output.log + +# Review JUnit XML results (for CI integration) +ls -lh ./kms-certification/junit/ + +# Count passed/failed tests +grep -c "PASS:" ./kms-certification/kms-test-output.log +grep -c "FAIL:" ./kms-certification/kms-test-output.log +``` + +**Test Suite Configuration:** + +The `openshift/kms` test suite is configured with the following parameters: +- **Parallelism:** 1 (tests run sequentially to avoid conflicts) +- **Timeout:** 4 hours +- **Stability:** Disruptive (may affect cluster operations) +- **Scope:** Aggregates all KMS encryption tests across kube-apiserver, openshift-apiserver, and oauth-apiserver +- **Exclusions:** Automatically excludes tests marked as `[Flaky]` or `[Disabled:]` + +**What the Test Suite Validates:** + +The test suite performs comprehensive validation including: + +1. **KMS v2 API Compliance:** + - Status, Encrypt, and Decrypt gRPC method correctness + - Proper error handling and timeout behavior + - Health check responses + +2. **Encryption Operations:** + - Successful encryption and decryption of secrets and configmaps + - Concurrent encrypt/decrypt operations under load + - Round-trip data integrity verification + +3. **Migration Scenarios:** + - identity → KMS migration + - KMS → identity migration + - KMS → KMS migration + - aesgcm/aescbc ↔ KMS migrations + +4. **Operational Behaviors:** + - Plugin startup and initialization + - Graceful handling of KMS service unavailability + - API server restart with KMS encryption enabled + - Multi-plugin scenarios during migration + +5. **OpenShift Integration:** + - Sidecar container lifecycle + - Credential management from Secrets + - ConfigMap integration for CA bundles + - Pre-flight checker validation + +**Submitting Certification Results:** + +After successful completion, vendors should provide: +1. Full test output log (`kms-test-output.log`) +2. JUnit XML files from the `junit/` directory +3. OpenShift version and KMS plugin version tested +4. Any test failures with root cause analysis and remediation plans + +**Troubleshooting Common Issues:** + +```bash +# If tests fail to connect to the cluster +export KUBECONFIG=/path/to/correct/kubeconfig +oc whoami # Verify authentication + +# If tests timeout +# Increase timeout with: --timeout=6h +./kms-certification/openshift-tests run openshift/kms --timeout=6h + +# To run a subset of tests for debugging +# First, list the tests with --dry-run, then filter specific ones +./kms-certification/openshift-tests run openshift/kms \ + --dry-run | grep "specific-test-name" | \ + ./kms-certification/openshift-tests run -f - + +# For verbose output during test run +./kms-certification/openshift-tests run openshift/kms -v=4 +``` + ### Risks and Mitigations **Risk: KMS Plugin Unavailable During Controller Sync**