Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions documentation/features/home-sections.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,14 @@ Users customize their home page by adding/removing/reordering sections. Each sec
- All section data stored in `AudibleCacheCategory` with reserved IDs: `__popular__` and `__new_releases__` for built-in sections
- Popular/new-releases use same wipe-and-populate pattern as user categories
- After built-in sections, queries DISTINCT categoryIds from `UserHomeSection`
- Per section: wipe `AudibleCacheCategory` rows, scrape, upsert `AudibleCache` metadata, insert ranked category entries
- Per section: scrape, wipe `AudibleCacheCategory` rows, upsert `AudibleCache` metadata, insert ranked category entries
- Empty scrape (e.g. 503 storm) → previous rows kept, warning logged; category not counted in `categoriesSynced`
- Batch cooldown between sections (10-20s random)
- Constants exported from `audible-refresh.processor.ts`: `POPULAR_CATEGORY_ID`, `NEW_RELEASES_CATEGORY_ID`

## AudibleService Methods
- `getCategories()`: Scrapes `{baseUrl}/categories`, returns `{id, name}[]`
- `getCategoryBooks(categoryId, limit)`: Scrapes `/search?node={id}&pageSize=50&sort=popularity-rank`, up to 200 results
- `getCategoryBooks(categoryId, limit)`: Scrapes `/search?node={id}&pageSize=50&sort=popularity-rank`, up to 200 results; storefront yields 0 books → catalog API fallback (`category_id={id}&products_sort_by=BestSellers`, region-language filtered)

## Frontend
- **Hooks:** `useHomeSections()`, `useCategoryAudiobooks()`, `useAudibleCategories()` in `src/lib/hooks/useHomeSections.ts`
Expand All @@ -48,7 +49,7 @@ Users customize their home page by adding/removing/reordering sections. Each sec
- 10 section limit per user (total)
- Category picker scraped live (no categories table)
- Top-level categories only (v1)
- Wipe-and-re-scrape per category during refresh
- Wipe-and-re-scrape per category during refresh, skipped when the scrape returns nothing (last good list kept)
- Deduplication of categories across users before scraping
- If category disappears, user sees empty section
- 10-color palette assigned by sort order
Expand Down
11 changes: 9 additions & 2 deletions documentation/integrations/audible.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Audiobook metadata for discovery, search, and detail pages. Split by access patt

## Architecture

- **Curated HTML (refresh job only):** the three methods called solely by `audible-refresh.processor.ts` (`getPopularAudiobooks`, `getNewReleases`, `getCategoryBooks`) scrape Audible's storefront HTML to inherit editorial curation. Beefed-up retry/backoff knobs (12 retries, 3-min jittered cap) handle 503 storms patiently on the nightly job without slowing healthy users.
- **Curated HTML (refresh job only):** the three methods called solely by `audible-refresh.processor.ts` (`getPopularAudiobooks`, `getNewReleases`, `getCategoryBooks`) scrape Audible's storefront HTML to inherit editorial curation. Beefed-up retry/backoff knobs (12 retries, 3-min jittered cap) handle 503 storms patiently on the nightly job without slowing healthy users. `getCategoryBooks` falls back to the catalog API (`category_id`, `products_sort_by=BestSellers`, region-language filtered) when the storefront yields 0 books; the refresh processor keeps a section's previous entries when a scrape returns nothing.
- **JSON catalog API (real-time):** `search`, `searchByAuthorAsin`, `getCategories` (categories listing), and `fetchAudibleDetailsFromApi` (per-ASIN fallback); series search (`audible-series-search.ts`, [own doc](./audible-series-search.md)). Same endpoint used by the official Audible mobile apps. No authentication, no API key, no user credentials, no special headers.
- **Audnexus (per-ASIN):** `getAudiobookDetails` and `getRuntime` prefer Audnexus, with catalog API fallback for `getAudiobookDetails`.
- **Series fill** (`audible-series-fill.ts`, `SeriesFill`, one per service): Audnexus result without a series → `series`/`seriesPart`/`seriesAsin` from `/1.0/catalog/products/{asin}` (same region), only when the chosen entry has a non-empty title, all three or none; every other field stays Audnexus. `getAudiobookDetails(asin, { fillSeries: false })` skips it (bulk-import scan, BookDate `matchToAudnexus`, organize tag lookup when the series is stored).
Expand All @@ -27,6 +27,7 @@ Audiobook metadata for discovery, search, and detail pages. Split by access patt
| Popular | `/adblbestsellers` | `pageSize=50`, `page=<n>` (omitted on first page) |
| New releases | `/newreleases` | `pageSize=50`, `page=<n>` (omitted on first page) |
| Category books | `/search` | `node=<categoryId>&pageSize=50&sort=popularity-rank&page=<n>` |
| Category books (fallback, catalog API) | `/1.0/catalog/products` | `category_id=<categoryId>&products_sort_by=BestSellers&num_results=50&page=<n>` (0-based) |

Parsed via cheerio. Selectors: `.productListItem` (popular/new releases), `.s-result-item, .productListItem` (categories).

Expand Down Expand Up @@ -136,7 +137,7 @@ Configurable Audible region for accurate metadata matching across international

**Per-region HTTP clients (on init):**
- `apiClient` — `baseURL=apiBaseUrl`, `Accept: application/json`, `User-Agent: ReadMeABook/1.0`, no language/ipRedirect params. Used for the real-time JSON catalog operations (search, series search via `fetch(url, config, { catalog: true })` = one attempt, author books, categories listing, per-ASIN details fallback).
- `htmlClient` — `baseURL=baseUrl`, rotating browser headers (`pickUserAgent` + `getBrowserHeaders`), default params `ipRedirectOverride=true` + `language=<audibleLocaleParam>`. Used by the nightly discovery refresh (`/adblbestsellers`, `/newreleases`, `/search?node=...`), by the series detail page (`audible-series.ts`), and by `getBaseUrl()`-based link generation.
- `htmlClient` — `baseURL=baseUrl`, rotating browser headers (`pickUserAgent` + `getBrowserHeaders`; `Accept-Encoding: false` — CloudFront 503s storefront requests carrying any Accept-Encoding, and `false` stops axios adding its default), default params `ipRedirectOverride=true` + `language=<audibleLocaleParam>`. Used by the nightly discovery refresh (`/adblbestsellers`, `/newreleases`, `/search?node=...`), by the series detail page (`audible-series.ts`), and by `getBaseUrl()`-based link generation.
- Audnexus calls include `region=<audnexusParam>`.

**Files:**
Expand Down Expand Up @@ -335,6 +336,12 @@ interface AuthorBooksResult {

## Fixed Issues

**Home category sections empty after the nightly refresh (2026-10-27)**
- **Problem:** "No audiobooks yet" on user category sections (and partial popular/category lists); job log `Request failed (503)` ×12 then `fetched 0 books`, ~17 min per category.
- **Root cause:** Audible's CloudFront answers 503 to storefront requests that carry an `Accept-Encoding` header (any value, even `identity`); the same request without it gets 200. `getBrowserHeaders` sent `gzip, deflate, br`, and axios on Node adds its own default when the key is absent. Verified with the same request at the same moment: with the header 0/4 categories answered, without it 4/4. `persistSectionBooks` then wiped the section's previous entries on the empty result.
- **Fix:** `getBrowserHeaders` sets `Accept-Encoding: false` (pages arrive uncompressed, ~2.5 MB each). Safety nets: `getCategoryBooks` falls back to the catalog API (`category_id`, `BestSellers`) on an empty storefront result; `persistSectionBooks` keeps previous entries when a scrape returns nothing.
- **Location:** `src/lib/utils/scrape-resilience.ts`, `src/lib/integrations/audible.service.ts`, `src/lib/processors/audible-refresh.processor.ts`; tests `tests/utils/scrape-resilience.test.ts`, `tests/integrations/audible.service.test.ts`, `tests/processors/audible-refresh.processor.test.ts`.

**Author pages missing books (#310 part 1, 2026-10-27)**
- **Problem:** Author pages (e.g. J.R.R. Tolkien) and entire-catalog watched authors stopped before the end of the author's catalog.
- **Root cause:** `searchByAuthorAsin` set `hasMore` only when the page had books left after the author-ASIN/language filters, so a page of namesakes or other languages ended paging; `scanAuthorBooks` (relevance scan) also stopped at such a page.
Expand Down
65 changes: 65 additions & 0 deletions src/lib/integrations/audible.service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -895,6 +895,71 @@ export class AudibleService {
logger.info(
`Category ${categoryId}: collected ${audiobooks.length} books across ${page - 1} pages`,
);

// CloudFront can 503 the storefront for one node through every retry while the
// catalog API answers at once; an empty storefront result falls back to it.
if (audiobooks.length === 0) {
return this.getCategoryBooksFromCatalog(categoryId, limit);
}

return audiobooks;
}

/**
* Catalog API fallback for getCategoryBooks: the node's bestsellers, in the region's
* accepted languages (the storefront filters language server-side, the catalog doesn't).
*/
private async getCategoryBooksFromCatalog(
categoryId: string,
limit: number,
): Promise<AudibleAudiobook[]> {
logger.warn(`Category ${categoryId}: storefront returned no books, trying the catalog API...`);

const langConfig = getLanguageForRegion(this.region);
const audiobooks: AudibleAudiobook[] = [];
const seenAsins = new Set<string>();
// Language filtering drops some of each page, so allow twice the storefront's page count
const maxPages = Math.ceil(limit / AUDIBLE_PAGE_SIZE) * 2;

for (let page = 0; page < maxPages && audiobooks.length < limit; page++) {
try {
const { data: response } = await this.fetchWithRetry(
'/1.0/catalog/products',
{
params: {
category_id: categoryId,
num_results: AUDIBLE_PAGE_SIZE,
page,
products_sort_by: 'BestSellers',
response_groups: CATALOG_RESPONSE_GROUPS,
},
},
5,
this.apiClient,
);

const envelope: CatalogProductsResponse = response.data;
const products = envelope.products ?? [];

for (const product of products) {
if (audiobooks.length >= limit) break;
if (!product.asin || seenAsins.has(product.asin)) continue;
if (!product.language || !isAcceptedLanguage(product.language, langConfig)) continue;
seenAsins.add(product.asin);
audiobooks.push(mapCatalogProduct(product));
}

if (products.length < AUDIBLE_PAGE_SIZE) break;
} catch (error) {
logger.error(`Catalog fallback failed for category ${categoryId} page ${page + 1}`, {
error: error instanceof Error ? error.message : String(error),
collectedSoFar: audiobooks.length,
});
break;
}
}

logger.info(`Category ${categoryId}: collected ${audiobooks.length} books from the catalog API`);
return audiobooks;
}

Expand Down
11 changes: 10 additions & 1 deletion src/lib/processors/audible-refresh.processor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,8 @@ export async function processAudibleRefresh(payload: AudibleRefreshPayload): Pro
books, catId, syncTime, thumbnailCache, logger, 'category book'
);

categoriesSynced++;
// An empty fetch keeps the previous list, so it doesn't count as synced
if (books.length > 0) categoriesSynced++;
logger.info(`Category ${catId}: saved ${saved} entries`);
} catch (error) {
logger.error(`Failed to refresh category ${catId}: ${error instanceof Error ? error.message : 'Unknown error'}`);
Expand Down Expand Up @@ -128,6 +129,7 @@ export async function processAudibleRefresh(payload: AudibleRefreshPayload): Pro
/**
* Wipe previous entries for a category, upsert book metadata into AudibleCache,
* and insert ranked entries into AudibleCacheCategory.
* An empty list leaves the previous entries in place (a failed scrape returns []).
* Returns the number of books successfully saved.
*/
async function persistSectionBooks(
Expand Down Expand Up @@ -157,6 +159,13 @@ async function persistSectionBooks(
);
}

// Scrapers return [] on failure (e.g. a CloudFront 503 storm). Wiping then would
// blank the section until the next good night, so keep the last good list instead.
if (dedupedBooks.length === 0) {
logger.warn(`No books fetched for ${categoryId}, keeping the previous entries`);
return 0;
}

// Wipe previous entries for this section
logger.info(`Clearing previous data for ${categoryId}...`);
await prisma.audibleCacheCategory.deleteMany({
Expand Down
7 changes: 5 additions & 2 deletions src/lib/utils/scrape-resilience.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,15 @@ export function pickUserAgent(): string {
}

/** Build a full set of realistic browser headers for the given UA */
export function getBrowserHeaders(userAgent: string): Record<string, string> {
export function getBrowserHeaders(userAgent: string): Record<string, string | false> {
return {
'User-Agent': userAgent,
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
'Accept-Language': 'en-US,en;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
// Audible's CloudFront 503s storefront requests that carry any Accept-Encoding
// (even `identity`) and serves the same request without it. `false` also stops
// axios adding its default `gzip, compress, deflate, br` on Node.
'Accept-Encoding': false,
'Connection': 'keep-alive',
'Sec-Fetch-Site': 'none',
'Sec-Fetch-Mode': 'navigate',
Expand Down
87 changes: 87 additions & 0 deletions tests/integrations/audible.service.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1420,6 +1420,93 @@ describe('AudibleService', () => {

expect(book.narrator).toBe('Alice, Bob, Carol, Dan');
});

describe('catalog API fallback', () => {
it('asks the catalog API for the node\'s bestsellers when the storefront page has no books', async () => {
htmlClientMock.get.mockResolvedValue(htmlResponse(makeHtmlPage([])));
apiClientMock.get.mockResolvedValue(
apiResponse(makeProductsResponse([makeProduct({ asin: 'BCATALOG01' })])),
);

const service = new AudibleService();
const books = await service.getCategoryBooks('CAT001', 200);

expect(apiClientMock.get.mock.calls[0][0]).toBe('/1.0/catalog/products');
const params = apiClientMock.get.mock.calls[0][1].params;
expect(params.category_id).toBe('CAT001');
expect(params.products_sort_by).toBe('BestSellers');
expect(params.page).toBe(0);
expect(params.num_results).toBe(50);
expect(books.map((b) => b.asin)).toEqual(['BCATALOG01']);
});

it('falls back after the storefront 503s through every retry (regression: section went blank)', async () => {
const error503 = Object.assign(new Error('Request failed with status code 503'), {
response: { status: 503 },
});
htmlClientMock.get.mockRejectedValue(error503);
apiClientMock.get.mockResolvedValue(
apiResponse(makeProductsResponse([makeProduct({ asin: 'BCATALOG02' })])),
);

const service = new AudibleService();
vi.spyOn(service as unknown as { delay: (ms: number) => Promise<void> }, 'delay').mockResolvedValue(undefined);
const books = await service.getCategoryBooks('CAT001', 200);

expect(htmlClientMock.get).toHaveBeenCalled();
expect(books.map((b) => b.asin)).toEqual(['BCATALOG02']);
});

it('keeps only the region\'s accepted languages', async () => {
htmlClientMock.get.mockResolvedValue(htmlResponse(makeHtmlPage([])));
apiClientMock.get.mockResolvedValue(
apiResponse(
makeProductsResponse([
makeProduct({ asin: 'BENGLISH01', language: 'english' }),
makeProduct({ asin: 'BSPANISH01', language: 'spanish' }),
makeProduct({ asin: 'BNOLANG001', language: undefined }),
]),
),
);

const service = new AudibleService();
const books = await service.getCategoryBooks('CAT001', 200);

expect(books.map((b) => b.asin)).toEqual(['BENGLISH01']);
});

it('pages until the limit and stops at a short page', async () => {
const fullPage = Array.from({ length: 50 }, (_, i) =>
makeProduct({ asin: `BFULL${String(i).padStart(5, '0')}` }),
);
const shortPage = [makeProduct({ asin: 'BSHORT0001' })];
htmlClientMock.get.mockResolvedValue(htmlResponse(makeHtmlPage([])));
apiClientMock.get
.mockResolvedValueOnce(apiResponse(makeProductsResponse(fullPage)))
.mockResolvedValueOnce(apiResponse(makeProductsResponse(shortPage)));

const service = new AudibleService();
const books = await service.getCategoryBooks('CAT001', 200);

expect(apiClientMock.get).toHaveBeenCalledTimes(2);
expect(apiClientMock.get.mock.calls[1][1].params.page).toBe(1);
expect(books).toHaveLength(51);
});

it('returns an empty list when the catalog API fails too', async () => {
const error503 = Object.assign(new Error('Service Unavailable'), {
response: { status: 503 },
});
htmlClientMock.get.mockResolvedValue(htmlResponse(makeHtmlPage([])));
apiClientMock.get.mockRejectedValue(error503);

const service = new AudibleService();
vi.spyOn(service as unknown as { delay: (ms: number) => Promise<void> }, 'delay').mockResolvedValue(undefined);
const books = await service.getCategoryBooks('CAT001', 200);

expect(books).toEqual([]);
});
});
});

// -------------------------------------------------------------------------
Expand Down
Loading