fix(cloudflare/state-store): treat JSON decode failures as unreadable entries - #1399
Open
its-rosetta wants to merge 5 commits into
Open
fix(cloudflare/state-store): treat JSON decode failures as unreadable entries#1399its-rosetta wants to merge 5 commits into
its-rosetta wants to merge 5 commits into
Conversation
… entries AES-CTR decryption with a wrong key usually does not throw in Web Crypto - it yields random plaintext bytes. The existing beta.45 key-rotation tolerance only catches crypto.subtle.decrypt failures, so JSON.parse of that garbage escapes the try block and, via Effect.orDie, kills the whole deploy with an unrecoverable defect instead of the intended return-undefined reconciliation path. Move the parse inside the try so wrong-key entries degrade the same way decrypt failures already do. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rong-key tolerance Move the AES-CTR entry encrypt/decrypt out of the Durable Object into EntryCodec.ts (plain Web Crypto, no behavior change) so the codec can be exercised in-process. The new test pins the failure mode this PR fixes: an entry written under a different key resolves `undefined` (logged) instead of escaping as a SyntaxError, and malformed/truncated entries never reject. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
sam-goodwin
force-pushed
the
fix/state-store-decode-tolerance
branch
from
September 1, 2026 21:49
9718ae1 to
e377688
Compare
…d secret's value The Secret provider PATCHes an existing secret with the program's value on every reconcile, including adoption. The state-store stack deploys with adopt(true), so any bootstrap that reaches it with a state that lacks the StateStoreEncryptionKeyValue row mints a fresh Random and overwrites the real encryption key — the 2.0.0-beta.45 rotation, and every entry in the store becomes unreadable. `preserveExistingValue` makes the stored value authoritative once the secret exists: it is only ever sent on create, and scopes/comment still reconcile. The state store's encryption-key secret opts in. Pinned by a live test that seeds the secret out-of-band, adopts it with a different program value, and reads the stored value back through a Worker, with a plain secret as the overwrite control. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…cryption key Each stack Durable Object records a SHA-256 fingerprint of its key on first use (`k:fingerprint`, outside every listed key space). If a later boot is handed a different key, get/set/getOutput/setOutput/getReplacedResources die with an explicit EncryptionKeyChangedError instead of decoding every entry as garbage, reporting it absent, and letting the next deploy overwrite the still-recoverable ciphertext. Listing and deleting stay available so a deliberate wipe is still possible. The first v8 boot of an existing store records its current key, so beta.45 leftovers keep degrading to "absent" exactly as before; any future rotation, by whatever path, fails loudly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t the decode fix The Durable Object only changes behavior once the deployed worker is upgraded, and the upgrade is gated on this version — without a bump the decode-tolerance fix and the rotation guard never reach an existing state store. Follows the precedent of every prior worker fix (alchemy-run#251, alchemy-run#468, alchemy-run#477). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
sam-goodwin
force-pushed
the
fix/state-store-decode-tolerance
branch
from
September 1, 2026 22:39
e377688 to
5f661ec
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
decryptEntryin the Cloudflare state store tolerates decryption failures (the 2.0.0-beta.45 key-rotation situation) by returningundefined— but only whencrypto.subtle.decryptitself throws. For AES-CTR, a wrong key typically does not throw: it yields random plaintext bytes.JSON.parseof those bytes sits outside thetry, so the resultingSyntaxErrorescapes intoEffect.orDieand kills the entire deploy with an unrecoverable defect, instead of taking the intended return-undefinedreconciliation path.We hit this in production against entries written around the beta.45 key rotation; we have been carrying this exact fix as a
patchedDependenciespatch since June and it has been load-bearing since.Fix
Move the
JSON.parseinside the existingtryso wrong-key garbage degrades identically to a decrypt failure (logged, entry treated as unreadable, engine reconciles). Two-line move plus comment; no behavior change for healthy entries.🤖 Generated with Claude Code
Tests (maintainer)
The AES-CTR entry codec is extracted from the Durable Object into
Cloudflare/StateStore/EntryCodec.ts(plain Web Crypto, no behavior change) so it can be unit-tested in-process.test/Cloudflare/StateStore/EntryCodec.test.tspins:encodeStatemarker formundefined(logged) instead of throwing theSyntaxErrorthis PR fixesundefined, never rejectEncryption-key rotation guarantee (maintainer)
Rolling out a new worker version is only safe if the
AlchemyStateStoreEncryptionKeysecret cannot rotate on the way — the beta.45 incident. The rotation mechanism is generic: theSecretprovider PATCHes an existing secret with whatever value the program holds, and the state-store stack is deployed withadopt(true). So any bootstrap that reaches the stack with a state that lacks (or, after this PR, cannot decrypt) theStateStoreEncryptionKeyValuerow mints a freshRandomand overwrites the real key. Two independent guards now close that:Cloudflare.SecretsStore.SecretgainspreserveExistingValue. Once a secret with that name exists, its value is never PATCHed again (scopes/comment still reconcile). The state store's encryption-key secret opts in, so a regeneratedRandom— stale local bootstrap state, a flaky serving probe, an unreadable state row — can no longer rotate the key. Pinned by a live test that seeds the secret out-of-band, adopts it with a different program value, and reads the stored value back through a Worker (plain secrets keep today's overwrite semantics as the control).k:fingerprint, outside every listed key space) on first use and refusesget/set/getOutput/setOutput/getReplacedResourceswith an explicitEncryptionKeyChangedErrorif a later boot is handed a different key. Listing and deleting stay available so a deliberate wipe is still possible. The first v8 boot of an existing store records its current key, so beta.45 leftovers keep degrading to "absent" exactly as before, but any future rotation fails loudly instead of reading everything as missing and then overwriting the still-recoverable ciphertext.STATE_STORE_VERSIONis bumped 7 → 8 in its own commit: the deployed worker is only upgraded when this version changes, so without the bump the fix never reaches an existing state store (same as #251, #468, #477). Drop that commit if the rollout should be batched with another change.