Skip to content

fix: stop sending Accept-Encoding to the Audible storefront (CloudFront 503s) - #507

Closed
topgunffx1 wants to merge 2 commits into
kikootwo:mainfrom
topgunffx1:fix/audible-accept-encoding-503
Closed

topgunffx1 wants to merge 2 commits into
kikootwo:mainfrom
topgunffx1:fix/audible-accept-encoding-503

Conversation

@topgunffx1

Copy link
Copy Markdown
Contributor

Linked issue

No open issue for this. Related to #450, where series search hit the same CloudFront 503s on the HTML /search page. That fix moved series search to the catalog API; this one finds the cause of the 503s on the storefront itself.

What changed

  1. Stop sending Accept-Encoding to the Audible storefront (scrape-resilience.ts). getBrowserHeaders now sets 'Accept-Encoding': false, so axios sends no Accept-Encoding header at all. Without the false, axios on Node adds its own gzip, compress, deflate, br.
  2. Keep a section's last good list (audible-refresh.processor.ts). persistSectionBooks no longer deletes a section's entries when the scrape returned nothing. A category that came back empty isn't counted in categoriesSynced.
  3. Catalog API fallback for categories (audible.service.ts). If the storefront returns 0 books for a node, getCategoryBooks reads /1.0/catalog/products?category_id=<id>&products_sort_by=BestSellers, filtered to the region's languages as searchByAuthorAsin already does. The storefront stays the primary source.

Docs are updated in integrations/audible.md (a Fixed Issues entry and the htmlClient notes) and features/home-sections.md.

Why

User category sections showed "No audiobooks yet" after the nightly refresh. The job log showed Request failed (503) 12 times per category (about 17 minutes each), then fetched 0 books. persistSectionBooks then wiped that category's previous entries, so one bad night emptied the section until a later night succeeded.

Root cause: Audible's CloudFront answers 503 to storefront requests that carry an Accept-Encoding header, whatever its value (even identity), and serves the same request without it. Measured with the same request within the same minute:

Request to /search?node=… Result
curl, no Accept-Encoding 200, 200
curl --compressed / Accept-Encoding: gzip / identity 503 on every request
axios, RMAB's getBrowserHeaders as they were 0 of 4 categories
axios, same headers with 'Accept-Encoding': false 4 of 4 categories (50 books each)

The user agents and Sec-Fetch-* headers made no difference. Changes 2 and 3 are safety nets in case Audible changes this behaviour.

How it was tested

  • Full npm test: 416 files passed (6,450 tests), 0 failed.
  • docker compose -f docker-compose.local.yml build readmeabook succeeded.
  • ESLint reports no problems on the changed lines. The pre-existing no-explicit-any hits in these files are unchanged.
  • Ran the built image and triggered Audible Data Refresh from Admin → Jobs:
    • Before: every category hit 12 503s (about 17 minutes each) and most saved 0 books. The refresh took about 50 minutes.
    • With changes 2 and 3 only: the same 503s, then the catalog fallback filled each category, and no section was wiped.
    • With all three changes: no 503s. Every category came from the storefront, the fallback wasn't used, and the whole refresh took 3 minutes.

New and updated tests:

  • scrape-resilience.test.ts: Accept-Encoding is false.
  • audible.service.test.ts: the catalog fallback triggers on an empty page and after a 503 storm, filters by language, pages through results, and returns [] if the catalog API fails too.
  • audible-refresh.processor.test.ts: an empty fetch keeps the previous entries. Two existing tests relied on wipe-on-empty and are updated.

Note: audible.service.ts was already about 1,060 lines, over the 400-line guideline. I kept the fallback next to getCategoryBooks rather than split the file in a bug-fix PR. I'm happy to move it into its own module if you'd prefer.

Checklist

  • Tests added or updated, and npm test passes
  • Docs in documentation/ updated in the compact format (bullets, tables, short code blocks), plus documentation/TABLEOFCONTENTS.md if a doc was added
  • No secrets, API keys, passwords or personal data in the code, logs or screenshots

🤖 Generated with Claude Code

topgunffx1 and others added 2 commits October 27, 2026 12:02
…efront

The nightly refresh scrapes /search?node=<id> for each user category. CloudFront
can 503 one node through all 12 retries (~17 min) while the catalog API answers
at once; the scrape then returns [] and persistSectionBooks wiped the section,
so one bad night left "No audiobooks yet" until a later night succeeded.

- getCategoryBooks: when the storefront yields 0 books, fall back to the catalog
  API (category_id, products_sort_by=BestSellers), filtered to the region's
  accepted languages like searchByAuthorAsin. The storefront stays primary.
- persistSectionBooks: an empty list keeps the previous entries (warn, no wipe);
  such a category isn't counted in categoriesSynced.
- Tests for both; two existing tests that relied on wipe-on-empty updated.
- Docs: integrations/audible.md, features/home-sections.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt 503s)

Audible's CloudFront answers 503 to storefront requests that carry an
Accept-Encoding header, whatever its value (even `identity`); the same request
without it gets 200. getBrowserHeaders sent `gzip, deflate, br`, and axios on
Node adds `gzip, compress, deflate, br` when the key is absent, so every
nightly-refresh page and series page went through the 503 retry storm.

Measured at the same moment with ReadMeABook's exact other headers:
with Accept-Encoding 0/4 category pages answered, with `false` 4/4 (50 books each).

Set 'Accept-Encoding': false so axios sends none. Pages arrive uncompressed
(~2.5 MB each), which is fine for a nightly job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kikootwo

kikootwo commented Nov 2, 2026

Copy link
Copy Markdown
Owner

@topgunffx1 thanks for this. The empty refresh wiping a home section is a real bug, and falling back to the catalog API is a good idea, so I'm bringing those two parts in on a branch of mine with you as co-author.

I'm leaving out the Accept-Encoding change: when I tested the storefront from here, the request without that header got the 503 and the one with it got a 200, so the 503s depend on more than that header, and turning compression off makes every page several times bigger.

I'll close this one when the other two parts land.

kikootwo added a commit that referenced this pull request Nov 2, 2026
…ne, and category refresh falls back to the catalog API (#535)

One failed nightly refresh could empty a home section: when the storefront answered 503 through every retry, the refresh deleted the section's rows anyway. A section whose fetch returns no books (counted before the language filter) now keeps its previous list, logs one warning naming it, and isn't counted as synced; this covers popular, new releases and categories. A category whose storefront scrape yields no books falls back to the catalog API's bestsellers for that category (sub-categories included), keeping only products in the region's language; the "Only show books in ..." setting then applies to those rows as to storefront rows. Request headers are unchanged.

Based on #507 by @topgunffx1 (the empty-refresh fix and the catalog fallback).

Co-authored-by: topgunffx1 <341746530+topgunffx1@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kikootwo

kikootwo commented Nov 2, 2026

Copy link
Copy Markdown
Owner

@topgunffx1 thanks again. It's in now as #535 with you as co-author: a home section keeps its books when a nightly refresh fetches none, and the category refresh falls back to the catalog API. It will be in the next release. Closing this one in favour of #535.

@kikootwo kikootwo closed this Nov 2, 2026
@kikootwo

kikootwo commented Nov 3, 2026

Copy link
Copy Markdown
Owner

@topgunffx1 this is out in v1.2.13, landed in #535: https://github.com/kikootwo/ReadMeABook/releases/tag/v1.2.13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants