Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions .context/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -552,14 +552,15 @@ One row per failed scrape attempt. Queried for vendors with 3+ failures in 7 day

#### `provider_coins` — Provider Config

| Column | Type | Description |
| ----------- | ------- | ---------------------------------------- |
| `slug` | TEXT PK | Coin slug |
| `metal` | TEXT | `"gold"` \| `"silver"` \| `"platinum"` |
| `name` | TEXT | Display name |
| `weight_oz` | REAL | Troy ounces |
| `enabled` | INTEGER | 1 = active |
| `fbp_url` | TEXT | FindBullionPrices URL (legacy, nullable) |
| Column | Type | Description |
| ----------- | ------- | ---------------------------------------------------- |
| `slug` | TEXT PK | Coin slug |
| `metal` | TEXT | `"gold"` \| `"silver"` \| `"platinum"` |
| `name` | TEXT | Display name |
| `weight_oz` | REAL | Troy ounces |
| `enabled` | INTEGER | 1 = active |
| `fbp_url` | TEXT | FindBullionPrices URL (legacy, nullable) |
| `fbp_match` | TEXT | FBP slug-resolver keyword hints (STRK-334, nullable) |

#### `provider_vendors` — Provider Config

Expand Down
22 changes: 16 additions & 6 deletions .context/data-pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,15 @@ MintBuilder is the **first vendor with a first-party price feed** (offered by Mi
- **Key hygiene:** `MB_API_KEY` lives only in container env (compose declaration + `run-home.sh` cron re-export, STRK-230 pattern; Infisical `dev` is the source of record). `provider_vendors.url` keeps the human product-page URL — it is republished into public JSON (Buy links); the feed request URL is never logged and error messages are key-redacted.
- **Pending vendor ask:** an explicit availability field (or `qty_max: 0` on sold-out) would eliminate the residual per-item scrapes; `feedStockIsConfident()` in the feed client is the single consumption point when it arrives.

### JM Bullion via FindBullionPrices (STRK-334)

JM Bullion direct scraping is Webscale/reCAPTCHA-blocked, so `jmbullion` prices are gap-filled from **FindBullionPrices.com (FBP)**. `price-extract-vendor-jmbullion-fbp.js` `scrape(context)` fetches the coin's FBP product page and reads the embedded schema.org `ItemList`, picking the JM Bullion offer and recording `source: "fbp"`. Extraction is a **plain HTTPS `fetch` + JSON-LD parse** (`fbp-jsonld.js`, browser `User-Agent`, AbortController timeout) — no Firecrawl, no Byparr/CF-bypass. It runs on the same polite hourly retail cadence as every other vendor.

- **Fetch seam:** the network call is injected as `context.fetchFbpPage` (test double), falling back to the real `fetchFbpPage`. `scrape` never throws — misses return a failed result (`no-fbp-url`, `jm-not-listed`, or the fetch error).
- **Slug resolver:** `resolve-fbp-slugs.js` is a cold, on-demand sitemap resolver — it fetches FBP's product sitemap once, matches each coin via the stored `provider_coins.fbp_match` keyword hint, prefers the current-year slug (current-year ▸ random-year ▸ older-dated), and **fails closed** (host-allowlisted to `findbullionprices.com`; skips unless the candidate page's ItemList name contains every keyword).
- **Live JM re-enable is a post-deploy step** — the vendor module and FBP sourcing ship on this branch, but flipping JM back on in provider config happens after deploy.
- **Attribution:** the market footer carries a FindBullionPrices attribution link.

### STRK-32 Vendor Module Boundary

As of v3.35.6, `devops/pollers/shared/price-extract.js` is the orchestrator, not the
Expand All @@ -278,12 +287,13 @@ scrape through a standard Vendor module interface.

Migrated Vendors:

| Vendor | Module | Notes |
| -------------- | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `apmex` | `price-extract-vendor-apmex.js` | Owns APMEX Firecrawl-preferred config and delegates to shared generic scraping |
| `goldback` | `price-extract-vendor-goldback.js` | Owns the Goldback Phase 0 bypass predicate and delegates to shared generic scraping |
| `summitmetals` | `price-extract-vendor-summit.js` | Owns Summit cutoff patterns + qty-tier-first extract strategy (JSON-LD offer price untrusted — it is the 100+ bulk tier) |
| `mintbuilder` | `price-extract-vendor-mintbuilder.js` | **Feed-first (STRK-321/325)** — direct vendor API with page-scrape fallback; see #MintBuilder Direct API Feed (STRK-321 / STRK-325) |
| Vendor | Module | Notes |
| -------------- | --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `apmex` | `price-extract-vendor-apmex.js` | Owns APMEX Firecrawl-preferred config and delegates to shared generic scraping |
| `goldback` | `price-extract-vendor-goldback.js` | Owns the Goldback Phase 0 bypass predicate and delegates to shared generic scraping |
| `summitmetals` | `price-extract-vendor-summit.js` | Owns Summit cutoff patterns + qty-tier-first extract strategy (JSON-LD offer price untrusted — it is the 100+ bulk tier) |
| `mintbuilder` | `price-extract-vendor-mintbuilder.js` | **Feed-first (STRK-321/325)** — direct vendor API with page-scrape fallback; see #MintBuilder Direct API Feed (STRK-321 / STRK-325) |
| `jmbullion` | `price-extract-vendor-jmbullion-fbp.js` | **FBP-backed (STRK-334)** — JM direct is Webscale/reCAPTCHA-blocked, so prices are sourced from FindBullionPrices.com JSON-LD (`source: "fbp"`); consumes a `context.fetchFbpPage` seam. See #JM Bullion via FindBullionPrices (STRK-334) |

All not-yet-migrated Vendors still route through `price-extract-vendor-legacy.js`,
which preserves the pre-refactor generic scrape behavior. To migrate another Vendor,
Expand Down
1 change: 1 addition & 0 deletions .context/deep-dives/provider-database.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ CREATE TABLE IF NOT EXISTS provider_coins (
name TEXT NOT NULL, -- "American Silver Eagle"
weight_oz REAL NOT NULL, -- troy ounces (e.g. 1, 10)
fbp_url TEXT, -- FindBullionPrices URL (legacy, nullable)
fbp_match TEXT, -- FBP slug-resolver keyword hints (STRK-334, nullable)
notes TEXT, -- free-form notes
enabled INTEGER NOT NULL DEFAULT 1,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
Expand Down
2 changes: 1 addition & 1 deletion .context/deep-dives/vendor-quirks.md
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,7 @@ This section documents how each vendor's website is structured for scraping purp

- **Stack:** Next.js App Router with RSC streaming
- **Spot API:** `contentapi-managed.jmbullion.com/api/spot/summary` — unauthenticated, returns current spot prices.
- **Product pricing:** No dedicated product price API. Prices must be extracted from HTML scraping only.
- **Product pricing:** No dedicated product price API. JM direct scraping is Webscale/reCAPTCHA-blocked, so product prices are gap-filled from **FindBullionPrices.com** JSON-LD `ItemList` via `price-extract-vendor-jmbullion-fbp.js` (`source: "fbp"`, STRK-334) — a plain HTTPS fetch + JSON-LD parse, no CF-bypass. See the JM-via-FBP subsection in `.context/data-pipelines.md`.

### APMEX

Expand Down
2 changes: 2 additions & 0 deletions .context/reusable-patterns.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,8 @@ JMBullion marks some coins as "Presale" at year-start but still shows purchasabl

Preorder tolerance is resolved **per vendor** by `resolvePreorderTolerant()` (`devops/pollers/shared/price-extract-shared.js:106-110`, consumed at `:214`): a `preorderTolerant` field on the vendor descriptor wins, and `LEGACY_PREORDER_TOLERANT_PROVIDERS` (`:72`) is only the fallback. That legacy set holds **two** vendors — `jmbullion` and `monumentmetals`. There is no `PREORDER_TOLERANT_PROVIDERS` constant.

Poller-side (STRK-334), `jmbullion` prices are FBP-sourced rather than JM-direct scraped: `price-extract-vendor-jmbullion-fbp.js` follows the standard `MIGRATED_VENDOR_MAP` `scrape(context)` contract and consumes a `context.fetchFbpPage` seam (see `.context/data-pipelines.md`).

### Frontend Consumption

The frontend fetches `providers.json` once per sync and caches it in localStorage. The modal uses `retailProviders[slug][vendorId]` for deep-link URLs, falling back to `RETAIL_VENDOR_URLS[vendorId]`.
Expand Down
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [3.36.10] - 2026-08-16

### Added — STRK-334: Restore JM Bullion market pricing via FindBullionPrices gap-fill

- **FindBullionPrices.com JSON-LD gap-fill poller vendor** (STRK-334): JM
Bullion is blocked at the source by its Webscale/reCAPTCHA challenge, so its
market prices had gone stale. A new poller vendor module scrapes
FindBullionPrices.com's product pages — which expose clean JSON-LD ItemList
data with no bot challenge — and uses those listings to gap-fill JM Bullion
pricing back onto the market table. A year-preferring slug resolver picks the
correct year-specific FBP product URL (older slugs 302-redirect), and a new
`provider_coins.fbp_match` column records the resolved FBP mapping per coin.
- **Market footer FindBullionPrices attribution** (STRK-334): the market view
footer now carries a FindBullionPrices.com attribution link crediting the
gap-fill data source.
- Re-enabling JM Bullion as a live market vendor is a post-deploy step once the
gap-fill feed is publishing.

## [3.36.9] - 2026-08-16

### Added — STRK-345: v2 day-file backfill tool for new metals
Expand Down
2 changes: 1 addition & 1 deletion data/spot-history-2026.json

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions data/spot-history-bundle.js

Large diffs are not rendered by default.

Loading
Loading