Repository navigation
fix: stop sending Accept-Encoding to the Audible storefront (CloudFront 503s) - #507
topgunffx1 wants to merge 2 commits into
Conversation
…efront The nightly refresh scrapes /search?node=<id> for each user category. CloudFront can 503 one node through all 12 retries (~17 min) while the catalog API answers at once; the scrape then returns [] and persistSectionBooks wiped the section, so one bad night left "No audiobooks yet" until a later night succeeded. - getCategoryBooks: when the storefront yields 0 books, fall back to the catalog API (category_id, products_sort_by=BestSellers), filtered to the region's accepted languages like searchByAuthorAsin. The storefront stays primary. - persistSectionBooks: an empty list keeps the previous entries (warn, no wipe); such a category isn't counted in categoriesSynced. - Tests for both; two existing tests that relied on wipe-on-empty updated. - Docs: integrations/audible.md, features/home-sections.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt 503s) Audible's CloudFront answers 503 to storefront requests that carry an Accept-Encoding header, whatever its value (even `identity`); the same request without it gets 200. getBrowserHeaders sent `gzip, deflate, br`, and axios on Node adds `gzip, compress, deflate, br` when the key is absent, so every nightly-refresh page and series page went through the 503 retry storm. Measured at the same moment with ReadMeABook's exact other headers: with Accept-Encoding 0/4 category pages answered, with `false` 4/4 (50 books each). Set 'Accept-Encoding': false so axios sends none. Pages arrive uncompressed (~2.5 MB each), which is fine for a nightly job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@topgunffx1 thanks for this. The empty refresh wiping a home section is a real bug, and falling back to the catalog API is a good idea, so I'm bringing those two parts in on a branch of mine with you as co-author. I'm leaving out the Accept-Encoding change: when I tested the storefront from here, the request without that header got the 503 and the one with it got a 200, so the 503s depend on more than that header, and turning compression off makes every page several times bigger. I'll close this one when the other two parts land. |
…ne, and category refresh falls back to the catalog API (#535) One failed nightly refresh could empty a home section: when the storefront answered 503 through every retry, the refresh deleted the section's rows anyway. A section whose fetch returns no books (counted before the language filter) now keeps its previous list, logs one warning naming it, and isn't counted as synced; this covers popular, new releases and categories. A category whose storefront scrape yields no books falls back to the catalog API's bestsellers for that category (sub-categories included), keeping only products in the region's language; the "Only show books in ..." setting then applies to those rows as to storefront rows. Request headers are unchanged. Based on #507 by @topgunffx1 (the empty-refresh fix and the catalog fallback). Co-authored-by: topgunffx1 <341746530+topgunffx1@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@topgunffx1 thanks again. It's in now as #535 with you as co-author: a home section keeps its books when a nightly refresh fetches none, and the category refresh falls back to the catalog API. It will be in the next release. Closing this one in favour of #535. |
|
@topgunffx1 this is out in v1.2.13, landed in #535: https://github.com/kikootwo/ReadMeABook/releases/tag/v1.2.13 |
Linked issue
No open issue for this. Related to #450, where series search hit the same CloudFront 503s on the HTML
/searchpage. That fix moved series search to the catalog API; this one finds the cause of the 503s on the storefront itself.What changed
Accept-Encodingto the Audible storefront (scrape-resilience.ts).getBrowserHeadersnow sets'Accept-Encoding': false, so axios sends noAccept-Encodingheader at all. Without thefalse, axios on Node adds its owngzip, compress, deflate, br.audible-refresh.processor.ts).persistSectionBooksno longer deletes a section's entries when the scrape returned nothing. A category that came back empty isn't counted incategoriesSynced.audible.service.ts). If the storefront returns 0 books for a node,getCategoryBooksreads/1.0/catalog/products?category_id=<id>&products_sort_by=BestSellers, filtered to the region's languages assearchByAuthorAsinalready does. The storefront stays the primary source.Docs are updated in
integrations/audible.md(a Fixed Issues entry and thehtmlClientnotes) andfeatures/home-sections.md.Why
User category sections showed "No audiobooks yet" after the nightly refresh. The job log showed
Request failed (503)12 times per category (about 17 minutes each), thenfetched 0 books.persistSectionBooksthen wiped that category's previous entries, so one bad night emptied the section until a later night succeeded.Root cause: Audible's CloudFront answers 503 to storefront requests that carry an
Accept-Encodingheader, whatever its value (evenidentity), and serves the same request without it. Measured with the same request within the same minute:/search?node=…Accept-Encoding--compressed/Accept-Encoding: gzip/identitygetBrowserHeadersas they were'Accept-Encoding': falseThe user agents and
Sec-Fetch-*headers made no difference. Changes 2 and 3 are safety nets in case Audible changes this behaviour.How it was tested
npm test: 416 files passed (6,450 tests), 0 failed.docker compose -f docker-compose.local.yml build readmeabooksucceeded.no-explicit-anyhits in these files are unchanged.New and updated tests:
scrape-resilience.test.ts:Accept-Encodingisfalse.audible.service.test.ts: the catalog fallback triggers on an empty page and after a 503 storm, filters by language, pages through results, and returns[]if the catalog API fails too.audible-refresh.processor.test.ts: an empty fetch keeps the previous entries. Two existing tests relied on wipe-on-empty and are updated.Note:
audible.service.tswas already about 1,060 lines, over the 400-line guideline. I kept the fallback next togetCategoryBooksrather than split the file in a bug-fix PR. I'm happy to move it into its own module if you'd prefer.Checklist
npm testpassesdocumentation/updated in the compact format (bullets, tables, short code blocks), plusdocumentation/TABLEOFCONTENTS.mdif a doc was added🤖 Generated with Claude Code