Skip to content

fix(cloudflare/state-store): treat JSON decode failures as unreadable entries - #1399

Open
its-rosetta wants to merge 5 commits into
alchemy-run:mainfrom
its-rosetta:fix/state-store-decode-tolerance
Open

fix(cloudflare/state-store): treat JSON decode failures as unreadable entries#1399
its-rosetta wants to merge 5 commits into
alchemy-run:mainfrom
its-rosetta:fix/state-store-decode-tolerance

Conversation

@its-rosetta

@its-rosetta its-rosetta commented Aug 28, 2026

Copy link
Copy Markdown

Problem

decryptEntry in the Cloudflare state store tolerates decryption failures (the 2.0.0-beta.45 key-rotation situation) by returning undefined — but only when crypto.subtle.decrypt itself throws. For AES-CTR, a wrong key typically does not throw: it yields random plaintext bytes. JSON.parse of those bytes sits outside the try, so the resulting SyntaxError escapes into Effect.orDie and kills the entire deploy with an unrecoverable defect, instead of taking the intended return-undefined reconciliation path.

We hit this in production against entries written around the beta.45 key rotation; we have been carrying this exact fix as a patchedDependencies patch since June and it has been load-bearing since.

Fix

Move the JSON.parse inside the existing try so wrong-key garbage degrades identically to a decrypt failure (logged, entry treated as unreadable, engine reconciles). Two-line move plus comment; no behavior change for healthy entries.

🤖 Generated with Claude Code

Tests (maintainer)

The AES-CTR entry codec is extracted from the Durable Object into Cloudflare/StateStore/EntryCodec.ts (plain Web Crypto, no behavior change) so it can be unit-tested in-process. test/Cloudflare/StateStore/EntryCodec.test.ts pins:

  • round-trip under a random nonce, stored as the encodeState marker form
  • an entry written under a different key resolves undefined (logged) instead of throwing the SyntaxError this PR fixes
  • malformed / truncated entries resolve undefined, never reject

Encryption-key rotation guarantee (maintainer)

Rolling out a new worker version is only safe if the AlchemyStateStoreEncryptionKey secret cannot rotate on the way — the beta.45 incident. The rotation mechanism is generic: the Secret provider PATCHes an existing secret with whatever value the program holds, and the state-store stack is deployed with adopt(true). So any bootstrap that reaches the stack with a state that lacks (or, after this PR, cannot decrypt) the StateStoreEncryptionKeyValue row mints a fresh Random and overwrites the real key. Two independent guards now close that:

  • Cloudflare.SecretsStore.Secret gains preserveExistingValue. Once a secret with that name exists, its value is never PATCHed again (scopes/comment still reconcile). The state store's encryption-key secret opts in, so a regenerated Random — stale local bootstrap state, a flaky serving probe, an unreadable state row — can no longer rotate the key. Pinned by a live test that seeds the secret out-of-band, adopts it with a different program value, and reads the stored value back through a Worker (plain secrets keep today's overwrite semantics as the control).
  • The Durable Object records a SHA-256 fingerprint of its key (k:fingerprint, outside every listed key space) on first use and refuses get/set/getOutput/setOutput/getReplacedResources with an explicit EncryptionKeyChangedError if a later boot is handed a different key. Listing and deleting stay available so a deliberate wipe is still possible. The first v8 boot of an existing store records its current key, so beta.45 leftovers keep degrading to "absent" exactly as before, but any future rotation fails loudly instead of reading everything as missing and then overwriting the still-recoverable ciphertext.

STATE_STORE_VERSION is bumped 7 → 8 in its own commit: the deployed worker is only upgraded when this version changes, so without the bump the fix never reaches an existing state store (same as #251, #468, #477). Drop that commit if the rollout should be batched with another change.

colelawrence and others added 2 commits September 1, 2026 14:46
… entries

AES-CTR decryption with a wrong key usually does not throw in Web Crypto -
it yields random plaintext bytes. The existing beta.45 key-rotation
tolerance only catches crypto.subtle.decrypt failures, so JSON.parse of
that garbage escapes the try block and, via Effect.orDie, kills the whole
deploy with an unrecoverable defect instead of the intended
return-undefined reconciliation path.

Move the parse inside the try so wrong-key entries degrade the same way
decrypt failures already do.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rong-key tolerance

Move the AES-CTR entry encrypt/decrypt out of the Durable Object into
EntryCodec.ts (plain Web Crypto, no behavior change) so the codec can be
exercised in-process. The new test pins the failure mode this PR fixes:
an entry written under a different key resolves `undefined` (logged)
instead of escaping as a SyntaxError, and malformed/truncated entries
never reject.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@sam-goodwin
sam-goodwin force-pushed the fix/state-store-decode-tolerance branch from 9718ae1 to e377688 Compare September 1, 2026 21:49
sam-goodwin and others added 3 commits September 1, 2026 15:38
…d secret's value

The Secret provider PATCHes an existing secret with the program's value
on every reconcile, including adoption. The state-store stack deploys with
adopt(true), so any bootstrap that reaches it with a state that lacks the
StateStoreEncryptionKeyValue row mints a fresh Random and overwrites the
real encryption key — the 2.0.0-beta.45 rotation, and every entry in the
store becomes unreadable.

`preserveExistingValue` makes the stored value authoritative once the
secret exists: it is only ever sent on create, and scopes/comment still
reconcile. The state store's encryption-key secret opts in. Pinned by a
live test that seeds the secret out-of-band, adopts it with a different
program value, and reads the stored value back through a Worker, with a
plain secret as the overwrite control.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…cryption key

Each stack Durable Object records a SHA-256 fingerprint of its key on first
use (`k:fingerprint`, outside every listed key space). If a later boot is
handed a different key, get/set/getOutput/setOutput/getReplacedResources
die with an explicit EncryptionKeyChangedError instead of decoding every
entry as garbage, reporting it absent, and letting the next deploy
overwrite the still-recoverable ciphertext. Listing and deleting stay
available so a deliberate wipe is still possible.

The first v8 boot of an existing store records its current key, so
beta.45 leftovers keep degrading to "absent" exactly as before; any
future rotation, by whatever path, fails loudly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t the decode fix

The Durable Object only changes behavior once the deployed worker is
upgraded, and the upgrade is gated on this version — without a bump the
decode-tolerance fix and the rotation guard never reach an existing state
store. Follows the precedent of every prior worker fix (alchemy-run#251, alchemy-run#468, alchemy-run#477).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@sam-goodwin
sam-goodwin force-pushed the fix/state-store-decode-tolerance branch from e377688 to 5f661ec Compare September 1, 2026 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants