Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 8 additions & 9 deletions .context/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -552,15 +552,14 @@ One row per failed scrape attempt. Queried for vendors with 3+ failures in 7 day

#### `provider_coins` — Provider Config

| Column | Type | Description |
| ----------- | ------- | ---------------------------------------------------- |
| `slug` | TEXT PK | Coin slug |
| `metal` | TEXT | `"gold"` \| `"silver"` \| `"platinum"` |
| `name` | TEXT | Display name |
| `weight_oz` | REAL | Troy ounces |
| `enabled` | INTEGER | 1 = active |
| `fbp_url` | TEXT | FindBullionPrices URL (legacy, nullable) |
| `fbp_match` | TEXT | FBP slug-resolver keyword hints (STRK-334, nullable) |
| Column | Type | Description |
| ----------- | ------- | ------------------------------------------------------- |
| `slug` | TEXT PK | Coin slug |
| `metal` | TEXT | `"gold"` \| `"silver"` \| `"platinum"` |
| `name` | TEXT | Display name |
| `weight_oz` | REAL | Troy ounces |
| `enabled` | INTEGER | 1 = active |
| `fbp_url` | TEXT | FindBullionPrices product URL, hand-assigned (nullable) |

#### `provider_vendors` — Provider Config

Expand Down
3 changes: 2 additions & 1 deletion .context/data-pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,8 @@ MintBuilder is the **first vendor with a first-party price feed** (offered by Mi
JM Bullion direct scraping is Webscale/reCAPTCHA-blocked, so `jmbullion` prices are gap-filled from **FindBullionPrices.com (FBP)**. `price-extract-vendor-jmbullion-fbp.js` `scrape(context)` fetches the coin's FBP product page and reads the embedded schema.org `ItemList`, picking the JM Bullion offer and recording `source: "fbp"`. Extraction is a **plain HTTPS `fetch` + JSON-LD parse** (`fbp-jsonld.js`, browser `User-Agent`, AbortController timeout) — no Firecrawl, no Byparr/CF-bypass. It runs on the same polite hourly retail cadence as every other vendor.

- **Fetch seam:** the network call is injected as `context.fetchFbpPage` (test double), falling back to the real `fetchFbpPage`. `scrape` never throws — misses return a failed result (`no-fbp-url`, `jm-not-listed`, or the fetch error).
- **Slug resolver:** `resolve-fbp-slugs.js` is a cold, on-demand sitemap resolver — it fetches FBP's product sitemap once, matches each coin via the stored `provider_coins.fbp_match` keyword hint, prefers the current-year slug (current-year ▸ random-year ▸ older-dated), and **fails closed** (host-allowlisted to `findbullionprices.com`; skips unless the candidate page's ItemList name contains every keyword).
- **URL source — direct hand-assignment:** each coin's `provider_coins.fbp_url` is set by hand to the exact FBP product page. An auto-resolver (`resolve-fbp-slugs.js` + an `fbp_match` keyword-hint column) shipped with STRK-334 but was removed as unused in STRK-346: FBP lists a single 1 oz coin alongside a `Tube-of-50`, `Monster-Box`, and `1-10-oz` fractional for the same coin+year, and positive-token matching (ranked by year only) can't prefer the single coin, so hand-assignment is both simpler and more correct. The vendor module reads `coin.fbp_url` directly and host-allowlists it to `findbullionprices.com` before fetching (SSRF fail-closed).
- **Annual refresh:** FBP's dated slugs roll each January (`2026-…` → `2027-…`). Prefer FBP **Random-Year** product URLs where they exist — those don't roll — otherwise re-point the affected `fbp_url` values by hand once a year.
- **Live JM re-enable is a post-deploy step** — the vendor module and FBP sourcing ship on this branch, but flipping JM back on in provider config happens after deploy.
- **Attribution:** the market footer carries a FindBullionPrices attribution link.

Expand Down
3 changes: 1 addition & 2 deletions .context/deep-dives/provider-database.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,7 @@ CREATE TABLE IF NOT EXISTS provider_coins (
metal TEXT NOT NULL, -- "gold", "silver", "platinum"
name TEXT NOT NULL, -- "American Silver Eagle"
weight_oz REAL NOT NULL, -- troy ounces (e.g. 1, 10)
fbp_url TEXT, -- FindBullionPrices URL (legacy, nullable)
fbp_match TEXT, -- FBP slug-resolver keyword hints (STRK-334, nullable)
fbp_url TEXT, -- FindBullionPrices product URL, hand-assigned (nullable)
notes TEXT, -- free-form notes
enabled INTEGER NOT NULL DEFAULT 1,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
Expand Down
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [3.36.11] - 2026-08-17

### Removed — STRK-346: Remove unused FBP slug resolver + provider_coins.fbp_match column

- **Dropped the dormant FindBullionPrices slug resolver** (STRK-346): STRK-334
shipped an on-demand year-preferring slug resolver (`resolve-fbp-slugs.js`) and
a `provider_coins.fbp_match` keyword-hint column to auto-populate each coin's
`fbp_url`, but go-live chose direct hand-assignment instead — the resolver's
positive-token matching can't disambiguate FBP's multi-variant products (single
coin vs tube vs monster-box vs fractional). The resolver never ran in the hot
path, so it and its `fbp_match` threading are removed as dead code. No runtime
behavior change: the JM Bullion vendor module still reads `coin.fbp_url`
directly. The inert `fbp_match` column is left in existing databases (libSQL
DROP COLUMN needs a table rebuild) but is no longer created or referenced.

---

## [3.36.10] - 2026-08-16

### Added — STRK-334: Restore JM Bullion market pricing via FindBullionPrices gap-fill
Expand Down
41 changes: 0 additions & 41 deletions devops/pollers/shared/__fixtures__/fbp-sitemap.xml

This file was deleted.

7 changes: 3 additions & 4 deletions devops/pollers/shared/price-extract-vendor-jmbullion-fbp.js
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,9 @@ import { parseFbpItemList, findVendorOffer, fetchFbpPage } from "./fbp-jsonld.js

const JM_SELLER_NAME = "JM Bullion";

// Exact apex host FBP pages are served from. Matches the allowlist
// resolve-fbp-slugs.js enforces when it writes fbp_url, so a poisoned or
// misconfigured provider_coins.fbp_url pointing at any other host fails closed
// here before we ever open a socket (SSRF defense-in-depth).
// Exact apex host FBP pages are served from. A poisoned or misconfigured
// provider_coins.fbp_url pointing at any other host fails closed here before we
// ever open a socket (SSRF defense-in-depth). fbp_url is hand-assigned per coin.
const FBP_HOST = "findbullionprices.com";

export const vendor = {
Expand Down
19 changes: 4 additions & 15 deletions devops/pollers/shared/provider-db.js
Original file line number Diff line number Diff line change
Expand Up @@ -77,12 +77,6 @@ export async function initProviderSchema(client) {
} catch {
// Column already exists — expected on subsequent runs
}
// STRK-334: Add fbp_match column if missing (migration for existing DBs)
try {
await client.execute("ALTER TABLE provider_coins ADD COLUMN fbp_match TEXT");
} catch {
// Column already exists — expected on subsequent runs
}
}

// ---------------------------------------------------------------------------
Expand All @@ -99,7 +93,7 @@ export async function initProviderSchema(client) {
*/
export async function getProviders(client) {
const coinsResult = await client.execute(
"SELECT slug, metal, name, weight_oz, fbp_url, fbp_match, notes, enabled FROM provider_coins ORDER BY slug"
"SELECT slug, metal, name, weight_oz, fbp_url, notes, enabled FROM provider_coins ORDER BY slug"
);
const vendorsResult = await client.execute(
"SELECT coin_slug, vendor_id, vendor_name, url, enabled, selector, hints, skip_bounds FROM provider_vendors ORDER BY coin_slug, vendor_id"
Expand Down Expand Up @@ -128,7 +122,6 @@ export async function getProviders(client) {
metal: row.metal,
weight_oz: row.weight_oz,
...(row.fbp_url ? { fbp_url: row.fbp_url } : {}),
...(row.fbp_match ? { fbp_match: row.fbp_match } : {}),
...(row.notes ? { notes: row.notes } : {}),
providers: vendorsByCoin.get(row.slug) || [],
};
Expand Down Expand Up @@ -168,15 +161,14 @@ export async function getProvidersByCoin(client, coinSlug) {
*/
export async function getAllCoins(client) {
const result = await client.execute(
"SELECT slug, metal, name, weight_oz, fbp_url, fbp_match, notes, enabled FROM provider_coins ORDER BY slug"
"SELECT slug, metal, name, weight_oz, fbp_url, notes, enabled FROM provider_coins ORDER BY slug"
);
return result.rows.map((row) => ({
slug: row.slug,
metal: row.metal,
name: row.name,
weight_oz: row.weight_oz,
fbp_url: row.fbp_url,
fbp_match: row.fbp_match,
notes: row.notes,
enabled: row.enabled === 1,
}));
Expand All @@ -196,21 +188,19 @@ export async function getAllCoins(client) {
* @param {string} coin.name
* @param {number} [coin.weight_oz=1.0]
* @param {string} [coin.fbp_url]
* @param {string} [coin.fbp_match]
* @param {string} [coin.notes]
* @param {boolean} [coin.enabled=true]
*/
export async function upsertCoin(client, coin) {
await client.execute({
sql: `
INSERT INTO provider_coins (slug, metal, name, weight_oz, fbp_url, fbp_match, notes, enabled, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, datetime('now'))
INSERT INTO provider_coins (slug, metal, name, weight_oz, fbp_url, notes, enabled, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, datetime('now'))
ON CONFLICT(slug) DO UPDATE SET
metal = excluded.metal,
name = excluded.name,
weight_oz = excluded.weight_oz,
fbp_url = excluded.fbp_url,
fbp_match = excluded.fbp_match,
notes = excluded.notes,
enabled = excluded.enabled,
updated_at = datetime('now')
Expand All @@ -221,7 +211,6 @@ export async function upsertCoin(client, coin) {
coin.name,
coin.weight_oz ?? 1.0,
coin.fbp_url ?? null,
coin.fbp_match ?? null,
coin.notes ?? null,
coin.enabled !== false ? 1 : 0,
],
Expand Down
29 changes: 14 additions & 15 deletions devops/pollers/shared/provider-db.test.mjs
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
#!/usr/bin/env node
/**
* TDD contract tests for the provider_coins fbp_match migration + round-trip
* (STRK-334, design C5 / Data Models).
* Contract tests for the provider_coins schema + coin round-trip.
*
* provider-db.js talks to a libSQL client (createClient from @libsql/client),
* whose native/better-sqlite3 stack cannot build in this repo's dev/CI
Expand All @@ -12,8 +11,9 @@
* build), so `initProviderSchema` / `upsertCoin` / `getAllCoins` run their
* real SQL against a genuine in-memory SQLite database.
*
* RED phase: the schema has no `fbp_match` column and neither the upsert nor
* getAllCoins carry it, so the round-trip assertion on `fbp_match` fails.
* fbp_url is hand-assigned per coin (STRK-334); the STRK-334 slug resolver and
* its fbp_match keyword-hint column were removed as unused (STRK-346), so the
* round-trip here asserts fbp_url only.
*
* Run with:
* node --test devops/pollers/shared/provider-db.test.mjs
Expand Down Expand Up @@ -54,19 +54,23 @@ function makeMemoryClient() {
};
}

test("provider_coins gains an fbp_match column after schema init", async () => {
test("provider_coins carries fbp_url but not fbp_match after schema init", async () => {
const client = makeMemoryClient();
await initProviderSchema(client);

const { rows } = await client.execute("PRAGMA table_info(provider_coins)");
const columns = rows.map((row) => row.name);
assert.ok(
columns.includes("fbp_match"),
`provider_coins should carry an fbp_match column; got: ${columns.join(", ")}`
columns.includes("fbp_url"),
`provider_coins should carry an fbp_url column; got: ${columns.join(", ")}`
);
assert.ok(
!columns.includes("fbp_match"),
`provider_coins should NOT carry fbp_match (removed in STRK-346); got: ${columns.join(", ")}`
);
});

test("upsertCoin + getAllCoins round-trip both fbp_match and fbp_url", async () => {
test("upsertCoin + getAllCoins round-trip fbp_url", async () => {
const client = makeMemoryClient();
await initProviderSchema(client);

Expand All @@ -76,19 +80,14 @@ test("upsertCoin + getAllCoins round-trip both fbp_match and fbp_url", async ()
name: "American Silver Eagle 1 oz",
weight_oz: 1,
fbp_url: "https://findbullionprices.com/p/2026-american-silver-eagle-1-oz-bu-coin/",
fbp_match: "american silver eagle 1 oz",
});

const coins = await getAllCoins(client);
const ase = coins.find((coin) => coin.slug === "ase");
assert.ok(ase, "the upserted coin must round-trip");
assert.equal(
ase.fbp_url,
"https://findbullionprices.com/p/2026-american-silver-eagle-1-oz-bu-coin/"
);
assert.equal(
ase.fbp_match,
"american silver eagle 1 oz",
"fbp_match must survive the upsert → getAllCoins round-trip"
"https://findbullionprices.com/p/2026-american-silver-eagle-1-oz-bu-coin/",
"fbp_url must survive the upsert → getAllCoins round-trip"
);
});
Loading
Loading