Fix /team/sales staging outage: batch the Customer 360 per-customer query fan-out (~4,000 → ~35 D1 calls) - #152
Fix /team/sales staging outage: batch the Customer 360 per-customer query fan-out (~4,000 → ~35 D1 calls)#152PawSpaceIND wants to merge 1 commit into
Conversation
…uery fan-out Staging symptom: /team/sales rendered 'Too many API requests by single Worker invocation' and an empty customer list. Root cause: lib/customer-360.ts buildCustomer360() ran 8 D1 queries PER CUSTOMER (addresses, pets, bookings, food orders, coupons, cases, tickets, preferences) for up to 500 customers - ~4,000 subrequests in one GET, straight past Cloudflare's per-invocation cap. Both APIs the page calls (/api/customer-360 and /api/revenue-intelligence, whose refresh also builds the 360) died on real staging data, so the page showed nothing. Fix: batched section reads - ONE query per table per chunk of 80 customer IDs (D1's ~100-bound-parameter ceiling respected), grouped in JS. The old per-row ORDER BY / LIMIT semantics are reproduced exactly (default address first, newest bookings first with the 100/50-per-customer caps, food-order timeline merge, consent defaults, shared-phone dedup), so the output shape is byte-identical - every existing consumer (revenue-crm, revenue intelligence, promotion/marketing governance, daily revenue opportunities, AI context) passes unchanged. 300 customers now cost ~35 D1 calls instead of ~2,402; the fix also speeds up every one of those callers. tests/customer-360-fanout.test.mjs - 4 real-execution tests with a query-COUNTING D1 shim (batches counted as one request, mirroring D1): 300 customers stay under a 50-call budget; the real /api/customer-360 route with 500 customers stays under 70 calls end to end; ordering, per-customer limits (120 coupons -> newest 50), food-order merge, consent, LTV exclusions and dedup asserted field by field; cold-DB build still degrades to empty sections instead of crashing. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CE4fKzrWRZetcYKh8CaU3b
… it caught Another account's PR #152 fixed the same Customer 360 N+1 that #160 fixed here, so its code is superseded. Its TEST is not: porting it found a real defect in the merged fix. chunk 50 (what #160 shipped) 300 customers: 51 calls 500: 82 calls FAILS chunk 80 (what #152 used) 300 customers: 34 calls 500: 58 calls passes budget asserted by the test 300 customers: <= 50 500: <= 70 #160 removed the 8-queries-per-customer loop but sized the chunk a third too small, so main was making 82 D1 calls where 58 would do. ID_CHUNK is now 80, still inside D1's ~100 bound-parameter ceiling. Verified by reverting to 50 and watching the test fail with "got 51 D1 calls". The bind-parameter cap this repo already enforced would never have caught either defect. Eight narrow queries per customer are each perfectly legal; only a count of calls catches a fan-out. The two guards catch different things: bind cap -> ONE statement that is too wide call budget -> MANY statements that are individually fine That test also proves something #160 never did: it pins the preserved semantics field by field - default-address-first ordering, newest-first booking timeline, the per-customer LIMIT 50 on coupons, the food-order merge, consent defaults and duplicate detection. The refactor claimed byte-identical output and now has the evidence. tests/helpers/d1-harness.mjs collects both guards so every suite models the same database instead of each writing a more forgiving shim: - D1_MAX_BOUND_PARAMS, previously stranded in one suite - a call counter where batch() costs ONE subrequest as real D1 charges it - otherwise a budget would punish batching, the exact thing we want code to do - assertWithinBudget asserts a FLOOR as well as a ceiling, so a budget cannot pass because the work silently did nothing tests/unit-economics.test.mjs now imports the shared cap rather than defining it. Credit where due: the framing is theirs - assert the property that would actually break, not the shape of the code. Tests 1265 -> 1269. Lint unchanged at 23 errors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011Hon8zmG1pwVY6zwxtT5th
… it caught (#166) Another account's PR #152 fixed the same Customer 360 N+1 that #160 fixed here, so its code is superseded. Its TEST is not: porting it found a real defect in the merged fix. chunk 50 (what #160 shipped) 300 customers: 51 calls 500: 82 calls FAILS chunk 80 (what #152 used) 300 customers: 34 calls 500: 58 calls passes budget asserted by the test 300 customers: <= 50 500: <= 70 #160 removed the 8-queries-per-customer loop but sized the chunk a third too small, so main was making 82 D1 calls where 58 would do. ID_CHUNK is now 80, still inside D1's ~100 bound-parameter ceiling. Verified by reverting to 50 and watching the test fail with "got 51 D1 calls". The bind-parameter cap this repo already enforced would never have caught either defect. Eight narrow queries per customer are each perfectly legal; only a count of calls catches a fan-out. The two guards catch different things: bind cap -> ONE statement that is too wide call budget -> MANY statements that are individually fine That test also proves something #160 never did: it pins the preserved semantics field by field - default-address-first ordering, newest-first booking timeline, the per-customer LIMIT 50 on coupons, the food-order merge, consent defaults and duplicate detection. The refactor claimed byte-identical output and now has the evidence. tests/helpers/d1-harness.mjs collects both guards so every suite models the same database instead of each writing a more forgiving shim: - D1_MAX_BOUND_PARAMS, previously stranded in one suite - a call counter where batch() costs ONE subrequest as real D1 charges it - otherwise a budget would punish batching, the exact thing we want code to do - assertWithinBudget asserts a FLOOR as well as a ceiling, so a budget cannot pass because the work silently did nothing tests/unit-economics.test.mjs now imports the shared cap rather than defining it. Credit where due: the framing is theirs - assert the property that would actually break, not the shape of the code. Tests 1265 -> 1269. Lint unchanged at 23 errors. Claude-Session: https://claude.ai/code/session_011Hon8zmG1pwVY6zwxtT5th Co-authored-by: Claude <noreply@anthropic.com>
|
Closing — superseded, and the useful half is already on
Diffing this branch against current One thing carried forward in #158 rather than lost: Thanks to #166 for porting the budget test — the point it makes is the right one: a bind-parameter cap catches one statement that is too wide, but only a call count catches many statements that are each individually fine. Generated by Claude Code |
|
Closing this — the fix is superseded, but the test was taken and it caught a defect in the version that shipped.
#160 removed the fan-out but sized the chunk a third too small. Two other things from this PR that #160 lacked and now has:
Credit for the framing: assert the property that would actually break, not the shape of the code. That is now the shared harness guard, and the bind-cap check that had been stranded in a single suite was folded into it. Nothing further needed here. Generated by Claude Code |
The outage
Staging
/team/salesrenders "Too many API requests by single Worker invocation" and an empty customer list (screenshot from UAT). Root cause,lib/customer-360.ts:7:buildCustomer360()looped every customer (up to 500) and ran 8 D1 queries per customer (addresses, pets, bookings, food orders, coupons, cases, tickets, preferences) → ~4,000 subrequests in a single GET, far past Cloudflare's per-invocation cap./api/customer-360directly, and/api/revenue-intelligencewhoserefresh()also builds the full 360 on every GET. Result: the whole Sales worklist showed nothing.The fix
Batched section reads: one query per table per chunk of 80 customer IDs (respecting D1's ~100 bound-parameter ceiling), grouped in JS.
ORDER BY/LIMITsemantics are reproduced exactly in JS — default address first, newest bookings first with the 100/50-per-customer caps, food-order →pet_foodtimeline merge, consent defaults, LTV exclusions, shared-phone/email dedup — so the output shape is byte-identical.Tests
tests/customer-360-fanout.test.mjs— 4 real-execution tests with a query-counting D1 shim (batches counted as one request, mirroring D1), so the fix is pinned by an actual budget rather than inspection:/api/customer-360route with 500 customers stays under 70 calls end to end and returns all 500.Adult Dog Food × 2package label, LTV excludes cancelled, consent flags, open-ticket count, duplicate-candidate detection, and the per-customerLIMIT 50on coupons (120 stored → newest 50 returned).Full suite: all pre-existing customer-360 consumers (CRM stack, revenue stack, food hardening, AI intelligence, daily-revenue governance tests) pass against the batched rebuild — behavioral equivalence verified by the tests that already pinned those behaviors.
Files changed
lib/customer-360.ts— batchedfetchGroupedreads replacing the per-customer loop queriestests/customer-360-fanout.test.mjs— new (4 tests)Gates
npm run typecheck— 0 errorsnpm run lint— 0 errors, 14 warnings (all pre-existing baseline, none new)npm test— 1192/1192 pass (1188 existing + 4 new)After merge + deploy,
/team/saleson staging should load its worklist normally.🤖 Generated with Claude Code
https://claude.ai/code/session_01CE4fKzrWRZetcYKh8CaU3b
Generated by Claude Code