Skip to content

Fix /team/sales staging outage: batch the Customer 360 per-customer query fan-out (~4,000 → ~35 D1 calls) - #152

Closed
PawSpaceIND wants to merge 1 commit into
mainfrom
claude/customer-360-fanout
Closed

Fix /team/sales staging outage: batch the Customer 360 per-customer query fan-out (~4,000 → ~35 D1 calls)#152
PawSpaceIND wants to merge 1 commit into
mainfrom
claude/customer-360-fanout

Conversation

@PawSpaceIND

Copy link
Copy Markdown
Owner

The outage

Staging /team/sales renders "Too many API requests by single Worker invocation" and an empty customer list (screenshot from UAT). Root cause, lib/customer-360.ts:7:

  • buildCustomer360() looped every customer (up to 500) and ran 8 D1 queries per customer (addresses, pets, bookings, food orders, coupons, cases, tickets, preferences) → ~4,000 subrequests in a single GET, far past Cloudflare's per-invocation cap.
  • Both APIs the page calls die on real data: /api/customer-360 directly, and /api/revenue-intelligence whose refresh() also builds the full 360 on every GET. Result: the whole Sales worklist showed nothing.

The fix

Batched section reads: one query per table per chunk of 80 customer IDs (respecting D1's ~100 bound-parameter ceiling), grouped in JS.

  • 300 customers: ~2,402 D1 calls → ~35. 500 customers through the real route end-to-end: < 70, comfortably inside the cap.
  • The old per-row ORDER BY/LIMIT semantics are reproduced exactly in JS — default address first, newest bookings first with the 100/50-per-customer caps, food-order → pet_food timeline merge, consent defaults, LTV exclusions, shared-phone/email dedup — so the output shape is byte-identical.
  • Every existing consumer benefits automatically and passes unchanged: revenue-crm, revenue-intelligence, promotion & marketing governance, daily revenue opportunities, AI conversation context, customer business view.
  • Missing section tables still degrade to empty sections (guarded reads), never a crash.

Tests

tests/customer-360-fanout.test.mjs — 4 real-execution tests with a query-counting D1 shim (batches counted as one request, mirroring D1), so the fix is pinned by an actual budget rather than inspection:

  • 300 customers build under a 50-call budget (pre-fix: ~2,402) with all 300 records returned.
  • The real /api/customer-360 route with 500 customers stays under 70 calls end to end and returns all 500.
  • Field-by-field semantics: address ordering (default first), merged booking timeline newest-first, Adult Dog Food × 2 package label, LTV excludes cancelled, consent flags, open-ticket count, duplicate-candidate detection, and the per-customer LIMIT 50 on coupons (120 stored → newest 50 returned).
  • Cold-DB build with no section tables at all.

Full suite: all pre-existing customer-360 consumers (CRM stack, revenue stack, food hardening, AI intelligence, daily-revenue governance tests) pass against the batched rebuild — behavioral equivalence verified by the tests that already pinned those behaviors.

Files changed

  • lib/customer-360.ts — batched fetchGrouped reads replacing the per-customer loop queries
  • tests/customer-360-fanout.test.mjs — new (4 tests)

Gates

  • npm run typecheck — 0 errors
  • npm run lint — 0 errors, 14 warnings (all pre-existing baseline, none new)
  • npm test1192/1192 pass (1188 existing + 4 new)

After merge + deploy, /team/sales on staging should load its worklist normally.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CE4fKzrWRZetcYKh8CaU3b


Generated by Claude Code

…uery fan-out

Staging symptom: /team/sales rendered 'Too many API requests by single
Worker invocation' and an empty customer list. Root cause:
lib/customer-360.ts buildCustomer360() ran 8 D1 queries PER CUSTOMER
(addresses, pets, bookings, food orders, coupons, cases, tickets,
preferences) for up to 500 customers - ~4,000 subrequests in one GET,
straight past Cloudflare's per-invocation cap. Both APIs the page calls
(/api/customer-360 and /api/revenue-intelligence, whose refresh also
builds the 360) died on real staging data, so the page showed nothing.

Fix: batched section reads - ONE query per table per chunk of 80 customer
IDs (D1's ~100-bound-parameter ceiling respected), grouped in JS. The old
per-row ORDER BY / LIMIT semantics are reproduced exactly (default address
first, newest bookings first with the 100/50-per-customer caps, food-order
timeline merge, consent defaults, shared-phone dedup), so the output shape
is byte-identical - every existing consumer (revenue-crm, revenue
intelligence, promotion/marketing governance, daily revenue opportunities,
AI context) passes unchanged. 300 customers now cost ~35 D1 calls instead
of ~2,402; the fix also speeds up every one of those callers.

tests/customer-360-fanout.test.mjs - 4 real-execution tests with a
query-COUNTING D1 shim (batches counted as one request, mirroring D1):
300 customers stay under a 50-call budget; the real /api/customer-360
route with 500 customers stays under 70 calls end to end; ordering,
per-customer limits (120 coupons -> newest 50), food-order merge, consent,
LTV exclusions and dedup asserted field by field; cold-DB build still
degrades to empty sections instead of crashing.

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CE4fKzrWRZetcYKh8CaU3b
PawSpaceIND pushed a commit that referenced this pull request Aug 13, 2026
… it caught

Another account's PR #152 fixed the same Customer 360 N+1 that #160 fixed here, so
its code is superseded. Its TEST is not: porting it found a real defect in the
merged fix.

  chunk 50 (what #160 shipped)   300 customers: 51 calls   500: 82 calls   FAILS
  chunk 80 (what #152 used)      300 customers: 34 calls   500: 58 calls   passes
  budget asserted by the test    300 customers: <= 50      500: <= 70

#160 removed the 8-queries-per-customer loop but sized the chunk a third too
small, so main was making 82 D1 calls where 58 would do. ID_CHUNK is now 80,
still inside D1's ~100 bound-parameter ceiling. Verified by reverting to 50 and
watching the test fail with "got 51 D1 calls".

The bind-parameter cap this repo already enforced would never have caught either
defect. Eight narrow queries per customer are each perfectly legal; only a count
of calls catches a fan-out. The two guards catch different things:

  bind cap     -> ONE statement that is too wide
  call budget  -> MANY statements that are individually fine

That test also proves something #160 never did: it pins the preserved semantics
field by field - default-address-first ordering, newest-first booking timeline,
the per-customer LIMIT 50 on coupons, the food-order merge, consent defaults and
duplicate detection. The refactor claimed byte-identical output and now has the
evidence.

tests/helpers/d1-harness.mjs collects both guards so every suite models the same
database instead of each writing a more forgiving shim:

- D1_MAX_BOUND_PARAMS, previously stranded in one suite
- a call counter where batch() costs ONE subrequest as real D1 charges it -
  otherwise a budget would punish batching, the exact thing we want code to do
- assertWithinBudget asserts a FLOOR as well as a ceiling, so a budget cannot
  pass because the work silently did nothing

tests/unit-economics.test.mjs now imports the shared cap rather than defining it.

Credit where due: the framing is theirs - assert the property that would actually
break, not the shape of the code.

Tests 1265 -> 1269. Lint unchanged at 23 errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011Hon8zmG1pwVY6zwxtT5th
PawSpaceIND added a commit that referenced this pull request Aug 13, 2026
… it caught (#166)

Another account's PR #152 fixed the same Customer 360 N+1 that #160 fixed here, so
its code is superseded. Its TEST is not: porting it found a real defect in the
merged fix.

  chunk 50 (what #160 shipped)   300 customers: 51 calls   500: 82 calls   FAILS
  chunk 80 (what #152 used)      300 customers: 34 calls   500: 58 calls   passes
  budget asserted by the test    300 customers: <= 50      500: <= 70

#160 removed the 8-queries-per-customer loop but sized the chunk a third too
small, so main was making 82 D1 calls where 58 would do. ID_CHUNK is now 80,
still inside D1's ~100 bound-parameter ceiling. Verified by reverting to 50 and
watching the test fail with "got 51 D1 calls".

The bind-parameter cap this repo already enforced would never have caught either
defect. Eight narrow queries per customer are each perfectly legal; only a count
of calls catches a fan-out. The two guards catch different things:

  bind cap     -> ONE statement that is too wide
  call budget  -> MANY statements that are individually fine

That test also proves something #160 never did: it pins the preserved semantics
field by field - default-address-first ordering, newest-first booking timeline,
the per-customer LIMIT 50 on coupons, the food-order merge, consent defaults and
duplicate detection. The refactor claimed byte-identical output and now has the
evidence.

tests/helpers/d1-harness.mjs collects both guards so every suite models the same
database instead of each writing a more forgiving shim:

- D1_MAX_BOUND_PARAMS, previously stranded in one suite
- a call counter where batch() costs ONE subrequest as real D1 charges it -
  otherwise a budget would punish batching, the exact thing we want code to do
- assertWithinBudget asserts a FLOOR as well as a ceiling, so a budget cannot
  pass because the work silently did nothing

tests/unit-economics.test.mjs now imports the shared cap rather than defining it.

Credit where due: the framing is theirs - assert the property that would actually
break, not the shape of the code.

Tests 1265 -> 1269. Lint unchanged at 23 errors.


Claude-Session: https://claude.ai/code/session_011Hon8zmG1pwVY6zwxtT5th

Co-authored-by: Claude <noreply@anthropic.com>

Copy link
Copy Markdown
Owner Author

Closing — superseded, and the useful half is already on main.

Diffing this branch against current main leaves only the older implementation of the same fix, so there is nothing left to merge.

One thing carried forward in #158 rather than lost: main briefly held two chunk sizes in two modules — 80 in lib/customer-360.ts and 50 in lib/unit-economics.ts (#165). Both now go through one helper at one size, with a test that fails if a module declares its own again.

Thanks to #166 for porting the budget test — the point it makes is the right one: a bind-parameter cap catches one statement that is too wide, but only a call count catches many statements that are each individually fine.


Generated by Claude Code

Copy link
Copy Markdown
Owner Author

Closing this — the fix is superseded, but the test was taken and it caught a defect in the version that shipped.

#160 fixed the same buildCustomer360 N+1 (8 queries per customer × up to 1,000 customers, ~8,000 subrequests against Cloudflare's 1,000 ceiling) and merged first, so lib/customer-360.ts here is redundant.

tests/customer-360-fanout.test.mjs was not redundant. It is now on main via #166, and porting it immediately failed:

300 customers 500 customers Budget asserted here
chunk 50 — what #160 shipped 51 calls 82 calls ≤ 50 / ≤ 70 → failed both
chunk 80 — what this PR used 34 calls 58 calls passed

#160 removed the fan-out but sized the chunk a third too small. ID_CHUNK is now 80 on main, verified by reverting to 50 and watching this test fail with got 51 D1 calls.

Two other things from this PR that #160 lacked and now has:

  • The call-budget shim. The repo already enforced D1's bind-parameter cap, which would never have caught either defect — 8 narrow queries per customer are each perfectly legal. Only counting calls catches a fan-out. Both guards now live in tests/helpers/d1-harness.mjs, with batch() charged as one subrequest as real D1 charges it, and a floor asserted alongside every ceiling so a budget cannot pass on work that silently did nothing.
  • The behavioural proof. Kill the Customer 360 N+1 that was blocking every marketing campaign #160 claimed byte-identical output after the refactor and had no test for it. This PR's field-by-field assertions — default-address-first ordering, newest-first booking timeline, per-customer LIMIT 50 on coupons, food-order merge, consent defaults, duplicate detection — are what actually establish that.

Credit for the framing: assert the property that would actually break, not the shape of the code. That is now the shared harness guard, and the bind-cap check that had been stranded in a single suite was folded into it.

Nothing further needed here.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants