From be2720478efdf4c34b3f938f700388dc2bc69bd7 Mon Sep 17 00:00:00 2001 From: lbruton Date: Mon, 17 Aug 2026 17:11:40 -0500 Subject: [PATCH 1/2] =?UTF-8?q?v3.36.11=20=E2=80=94=20STRK-346:=20remove?= =?UTF-8?q?=20unused=20FBP=20slug=20resolver=20+=20provider=5Fcoins.fbp=5F?= =?UTF-8?q?match=20column?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .context/architecture.md | 17 +- .context/data-pipelines.md | 3 +- .context/deep-dives/provider-database.md | 3 +- CHANGELOG.md | 17 + .../shared/__fixtures__/fbp-sitemap.xml | 41 --- .../price-extract-vendor-jmbullion-fbp.js | 7 +- devops/pollers/shared/provider-db.js | 19 +- devops/pollers/shared/provider-db.test.mjs | 29 +- devops/pollers/shared/resolve-fbp-slugs.js | 334 ------------------ .../pollers/shared/resolve-fbp-slugs.test.mjs | 85 ----- js/about.js | 2 +- js/constants.js | 2 +- package-lock.json | 4 +- package.json | 2 +- sw.js | 2 +- version.json | 4 +- 16 files changed, 57 insertions(+), 514 deletions(-) delete mode 100644 devops/pollers/shared/__fixtures__/fbp-sitemap.xml delete mode 100644 devops/pollers/shared/resolve-fbp-slugs.js delete mode 100644 devops/pollers/shared/resolve-fbp-slugs.test.mjs diff --git a/.context/architecture.md b/.context/architecture.md index a947f576..9b372a9c 100644 --- a/.context/architecture.md +++ b/.context/architecture.md @@ -552,15 +552,14 @@ One row per failed scrape attempt. Queried for vendors with 3+ failures in 7 day #### `provider_coins` — Provider Config -| Column | Type | Description | -| ----------- | ------- | ---------------------------------------------------- | -| `slug` | TEXT PK | Coin slug | -| `metal` | TEXT | `"gold"` \| `"silver"` \| `"platinum"` | -| `name` | TEXT | Display name | -| `weight_oz` | REAL | Troy ounces | -| `enabled` | INTEGER | 1 = active | -| `fbp_url` | TEXT | FindBullionPrices URL (legacy, nullable) | -| `fbp_match` | TEXT | FBP slug-resolver keyword hints (STRK-334, nullable) | +| Column | Type | Description | +| ----------- | ------- | ------------------------------------------------------- | +| `slug` | TEXT PK | Coin slug | +| `metal` | TEXT | `"gold"` \| `"silver"` \| `"platinum"` | +| `name` | TEXT | Display name | +| `weight_oz` | REAL | Troy ounces | +| `enabled` | INTEGER | 1 = active | +| `fbp_url` | TEXT | FindBullionPrices product URL, hand-assigned (nullable) | #### `provider_vendors` — Provider Config diff --git a/.context/data-pipelines.md b/.context/data-pipelines.md index b7a994aa..b18b962d 100644 --- a/.context/data-pipelines.md +++ b/.context/data-pipelines.md @@ -273,7 +273,8 @@ MintBuilder is the **first vendor with a first-party price feed** (offered by Mi JM Bullion direct scraping is Webscale/reCAPTCHA-blocked, so `jmbullion` prices are gap-filled from **FindBullionPrices.com (FBP)**. `price-extract-vendor-jmbullion-fbp.js` `scrape(context)` fetches the coin's FBP product page and reads the embedded schema.org `ItemList`, picking the JM Bullion offer and recording `source: "fbp"`. Extraction is a **plain HTTPS `fetch` + JSON-LD parse** (`fbp-jsonld.js`, browser `User-Agent`, AbortController timeout) — no Firecrawl, no Byparr/CF-bypass. It runs on the same polite hourly retail cadence as every other vendor. - **Fetch seam:** the network call is injected as `context.fetchFbpPage` (test double), falling back to the real `fetchFbpPage`. `scrape` never throws — misses return a failed result (`no-fbp-url`, `jm-not-listed`, or the fetch error). -- **Slug resolver:** `resolve-fbp-slugs.js` is a cold, on-demand sitemap resolver — it fetches FBP's product sitemap once, matches each coin via the stored `provider_coins.fbp_match` keyword hint, prefers the current-year slug (current-year ▸ random-year ▸ older-dated), and **fails closed** (host-allowlisted to `findbullionprices.com`; skips unless the candidate page's ItemList name contains every keyword). +- **URL source — direct hand-assignment:** each coin's `provider_coins.fbp_url` is set by hand to the exact FBP product page. An auto-resolver (`resolve-fbp-slugs.js` + an `fbp_match` keyword-hint column) shipped with STRK-334 but was removed as unused in STRK-346: FBP lists a single 1 oz coin alongside a `Tube-of-50`, `Monster-Box`, and `1-10-oz` fractional for the same coin+year, and positive-token matching (ranked by year only) can't prefer the single coin, so hand-assignment is both simpler and more correct. The vendor module reads `coin.fbp_url` directly and host-allowlists it to `findbullionprices.com` before fetching (SSRF fail-closed). +- **Annual refresh:** FBP's dated slugs roll each January (`2026-…` → `2027-…`). Prefer FBP **Random-Year** product URLs where they exist — those don't roll — otherwise re-point the affected `fbp_url` values by hand once a year. - **Live JM re-enable is a post-deploy step** — the vendor module and FBP sourcing ship on this branch, but flipping JM back on in provider config happens after deploy. - **Attribution:** the market footer carries a FindBullionPrices attribution link. diff --git a/.context/deep-dives/provider-database.md b/.context/deep-dives/provider-database.md index 4d8b37b9..628d045d 100644 --- a/.context/deep-dives/provider-database.md +++ b/.context/deep-dives/provider-database.md @@ -33,8 +33,7 @@ CREATE TABLE IF NOT EXISTS provider_coins ( metal TEXT NOT NULL, -- "gold", "silver", "platinum" name TEXT NOT NULL, -- "American Silver Eagle" weight_oz REAL NOT NULL, -- troy ounces (e.g. 1, 10) - fbp_url TEXT, -- FindBullionPrices URL (legacy, nullable) - fbp_match TEXT, -- FBP slug-resolver keyword hints (STRK-334, nullable) + fbp_url TEXT, -- FindBullionPrices product URL, hand-assigned (nullable) notes TEXT, -- free-form notes enabled INTEGER NOT NULL DEFAULT 1, created_at TEXT NOT NULL DEFAULT (datetime('now')), diff --git a/CHANGELOG.md b/CHANGELOG.md index 0153290d..6135da55 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [3.36.11] - 2026-08-17 + +### Removed — STRK-346: Remove unused FBP slug resolver + provider_coins.fbp_match column + +- **Dropped the dormant FindBullionPrices slug resolver** (STRK-346): STRK-334 + shipped an on-demand year-preferring slug resolver (`resolve-fbp-slugs.js`) and + a `provider_coins.fbp_match` keyword-hint column to auto-populate each coin's + `fbp_url`, but go-live chose direct hand-assignment instead — the resolver's + positive-token matching can't disambiguate FBP's multi-variant products (single + coin vs tube vs monster-box vs fractional). The resolver never ran in the hot + path, so it and its `fbp_match` threading are removed as dead code. No runtime + behavior change: the JM Bullion vendor module still reads `coin.fbp_url` + directly. The inert `fbp_match` column is left in existing databases (libSQL + DROP COLUMN needs a table rebuild) but is no longer created or referenced. + +--- + ## [3.36.10] - 2026-08-16 ### Added — STRK-334: Restore JM Bullion market pricing via FindBullionPrices gap-fill diff --git a/devops/pollers/shared/__fixtures__/fbp-sitemap.xml b/devops/pollers/shared/__fixtures__/fbp-sitemap.xml deleted file mode 100644 index 4c577ae7..00000000 --- a/devops/pollers/shared/__fixtures__/fbp-sitemap.xml +++ /dev/null @@ -1,41 +0,0 @@ - - - - - https://findbullionprices.com/p/2024-american-silver-eagle-1-oz-bu-coin/ - 2024-01-15 - - - https://findbullionprices.com/p/american-silver-eagle-1-oz-random-year/ - 2026-06-01 - - - https://findbullionprices.com/p/2026-american-silver-eagle-1-oz-bu-coin/ - 2026-01-05 - - - https://findbullionprices.com/p/american-silver-eagle-1-oz/ - 2020-03-22 - - - https://findbullionprices.com/p/2026-american-gold-eagle-1-oz-bu-coin/ - 2026-01-05 - - - https://findbullionprices.com/p/2026-canadian-silver-maple-leaf-1-oz/ - 2026-01-05 - - - https://findbullionprices.com/about/ - 2026-01-01 - - diff --git a/devops/pollers/shared/price-extract-vendor-jmbullion-fbp.js b/devops/pollers/shared/price-extract-vendor-jmbullion-fbp.js index bfc6c5b3..cb1bf82c 100644 --- a/devops/pollers/shared/price-extract-vendor-jmbullion-fbp.js +++ b/devops/pollers/shared/price-extract-vendor-jmbullion-fbp.js @@ -17,10 +17,9 @@ import { parseFbpItemList, findVendorOffer, fetchFbpPage } from "./fbp-jsonld.js const JM_SELLER_NAME = "JM Bullion"; -// Exact apex host FBP pages are served from. Matches the allowlist -// resolve-fbp-slugs.js enforces when it writes fbp_url, so a poisoned or -// misconfigured provider_coins.fbp_url pointing at any other host fails closed -// here before we ever open a socket (SSRF defense-in-depth). +// Exact apex host FBP pages are served from. A poisoned or misconfigured +// provider_coins.fbp_url pointing at any other host fails closed here before we +// ever open a socket (SSRF defense-in-depth). fbp_url is hand-assigned per coin. const FBP_HOST = "findbullionprices.com"; export const vendor = { diff --git a/devops/pollers/shared/provider-db.js b/devops/pollers/shared/provider-db.js index c09f2610..44aaf831 100644 --- a/devops/pollers/shared/provider-db.js +++ b/devops/pollers/shared/provider-db.js @@ -77,12 +77,6 @@ export async function initProviderSchema(client) { } catch { // Column already exists — expected on subsequent runs } - // STRK-334: Add fbp_match column if missing (migration for existing DBs) - try { - await client.execute("ALTER TABLE provider_coins ADD COLUMN fbp_match TEXT"); - } catch { - // Column already exists — expected on subsequent runs - } } // --------------------------------------------------------------------------- @@ -99,7 +93,7 @@ export async function initProviderSchema(client) { */ export async function getProviders(client) { const coinsResult = await client.execute( - "SELECT slug, metal, name, weight_oz, fbp_url, fbp_match, notes, enabled FROM provider_coins ORDER BY slug" + "SELECT slug, metal, name, weight_oz, fbp_url, notes, enabled FROM provider_coins ORDER BY slug" ); const vendorsResult = await client.execute( "SELECT coin_slug, vendor_id, vendor_name, url, enabled, selector, hints, skip_bounds FROM provider_vendors ORDER BY coin_slug, vendor_id" @@ -128,7 +122,6 @@ export async function getProviders(client) { metal: row.metal, weight_oz: row.weight_oz, ...(row.fbp_url ? { fbp_url: row.fbp_url } : {}), - ...(row.fbp_match ? { fbp_match: row.fbp_match } : {}), ...(row.notes ? { notes: row.notes } : {}), providers: vendorsByCoin.get(row.slug) || [], }; @@ -168,7 +161,7 @@ export async function getProvidersByCoin(client, coinSlug) { */ export async function getAllCoins(client) { const result = await client.execute( - "SELECT slug, metal, name, weight_oz, fbp_url, fbp_match, notes, enabled FROM provider_coins ORDER BY slug" + "SELECT slug, metal, name, weight_oz, fbp_url, notes, enabled FROM provider_coins ORDER BY slug" ); return result.rows.map((row) => ({ slug: row.slug, @@ -176,7 +169,6 @@ export async function getAllCoins(client) { name: row.name, weight_oz: row.weight_oz, fbp_url: row.fbp_url, - fbp_match: row.fbp_match, notes: row.notes, enabled: row.enabled === 1, })); @@ -196,21 +188,19 @@ export async function getAllCoins(client) { * @param {string} coin.name * @param {number} [coin.weight_oz=1.0] * @param {string} [coin.fbp_url] - * @param {string} [coin.fbp_match] * @param {string} [coin.notes] * @param {boolean} [coin.enabled=true] */ export async function upsertCoin(client, coin) { await client.execute({ sql: ` - INSERT INTO provider_coins (slug, metal, name, weight_oz, fbp_url, fbp_match, notes, enabled, updated_at) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, datetime('now')) + INSERT INTO provider_coins (slug, metal, name, weight_oz, fbp_url, notes, enabled, updated_at) + VALUES (?, ?, ?, ?, ?, ?, ?, datetime('now')) ON CONFLICT(slug) DO UPDATE SET metal = excluded.metal, name = excluded.name, weight_oz = excluded.weight_oz, fbp_url = excluded.fbp_url, - fbp_match = excluded.fbp_match, notes = excluded.notes, enabled = excluded.enabled, updated_at = datetime('now') @@ -221,7 +211,6 @@ export async function upsertCoin(client, coin) { coin.name, coin.weight_oz ?? 1.0, coin.fbp_url ?? null, - coin.fbp_match ?? null, coin.notes ?? null, coin.enabled !== false ? 1 : 0, ], diff --git a/devops/pollers/shared/provider-db.test.mjs b/devops/pollers/shared/provider-db.test.mjs index 26e69387..c0db7066 100644 --- a/devops/pollers/shared/provider-db.test.mjs +++ b/devops/pollers/shared/provider-db.test.mjs @@ -1,7 +1,6 @@ #!/usr/bin/env node /** - * TDD contract tests for the provider_coins fbp_match migration + round-trip - * (STRK-334, design C5 / Data Models). + * Contract tests for the provider_coins schema + coin round-trip. * * provider-db.js talks to a libSQL client (createClient from @libsql/client), * whose native/better-sqlite3 stack cannot build in this repo's dev/CI @@ -12,8 +11,9 @@ * build), so `initProviderSchema` / `upsertCoin` / `getAllCoins` run their * real SQL against a genuine in-memory SQLite database. * - * RED phase: the schema has no `fbp_match` column and neither the upsert nor - * getAllCoins carry it, so the round-trip assertion on `fbp_match` fails. + * fbp_url is hand-assigned per coin (STRK-334); the STRK-334 slug resolver and + * its fbp_match keyword-hint column were removed as unused (STRK-346), so the + * round-trip here asserts fbp_url only. * * Run with: * node --test devops/pollers/shared/provider-db.test.mjs @@ -54,19 +54,23 @@ function makeMemoryClient() { }; } -test("provider_coins gains an fbp_match column after schema init", async () => { +test("provider_coins carries fbp_url but not fbp_match after schema init", async () => { const client = makeMemoryClient(); await initProviderSchema(client); const { rows } = await client.execute("PRAGMA table_info(provider_coins)"); const columns = rows.map((row) => row.name); assert.ok( - columns.includes("fbp_match"), - `provider_coins should carry an fbp_match column; got: ${columns.join(", ")}` + columns.includes("fbp_url"), + `provider_coins should carry an fbp_url column; got: ${columns.join(", ")}` + ); + assert.ok( + !columns.includes("fbp_match"), + `provider_coins should NOT carry fbp_match (removed in STRK-346); got: ${columns.join(", ")}` ); }); -test("upsertCoin + getAllCoins round-trip both fbp_match and fbp_url", async () => { +test("upsertCoin + getAllCoins round-trip fbp_url", async () => { const client = makeMemoryClient(); await initProviderSchema(client); @@ -76,7 +80,6 @@ test("upsertCoin + getAllCoins round-trip both fbp_match and fbp_url", async () name: "American Silver Eagle 1 oz", weight_oz: 1, fbp_url: "https://findbullionprices.com/p/2026-american-silver-eagle-1-oz-bu-coin/", - fbp_match: "american silver eagle 1 oz", }); const coins = await getAllCoins(client); @@ -84,11 +87,7 @@ test("upsertCoin + getAllCoins round-trip both fbp_match and fbp_url", async () assert.ok(ase, "the upserted coin must round-trip"); assert.equal( ase.fbp_url, - "https://findbullionprices.com/p/2026-american-silver-eagle-1-oz-bu-coin/" - ); - assert.equal( - ase.fbp_match, - "american silver eagle 1 oz", - "fbp_match must survive the upsert → getAllCoins round-trip" + "https://findbullionprices.com/p/2026-american-silver-eagle-1-oz-bu-coin/", + "fbp_url must survive the upsert → getAllCoins round-trip" ); }); diff --git a/devops/pollers/shared/resolve-fbp-slugs.js b/devops/pollers/shared/resolve-fbp-slugs.js deleted file mode 100644 index 3085d3d6..00000000 --- a/devops/pollers/shared/resolve-fbp-slugs.js +++ /dev/null @@ -1,334 +0,0 @@ -#!/usr/bin/env node -/** - * FindBullionPrices (FBP) year-preferring slug resolver (STRK-334, design C5). - * - * Cold path — run on demand. Fetches FBP's product sitemap ONCE, matches each - * coin to its product slug via the stored `fbp_match` keyword hint, prefers the - * current-year slug, verifies the candidate page's JSON-LD `name` before - * committing, and upserts the resolved `fbp_url` back to `provider_coins`. - * - * Fail closed: a coin is skipped (its existing `fbp_url` untouched) whenever no - * slug matches, the candidate fetch fails, or the candidate's ItemList name does - * not contain every keyword — never store an unverified URL. - * - * Pure functions (`extractSitemapLocs`, `matchCoinSlugs`, `rankByYear`) carry - * the ranking contract and are unit-tested in isolation from network I/O. - * - * CLI: - * node resolve-fbp-slugs.js [--dry-run] [COINS=ase,gae] - * COINS=ase node resolve-fbp-slugs.js --dry-run - */ - -import { realpathSync } from "node:fs"; -import { fileURLToPath } from "node:url"; - -import { fetchFbpPage } from "./fbp-jsonld.js"; -import { getAllCoins, upsertCoin, initProviderSchema } from "./provider-db.js"; - -const SITEMAP_URL = "https://findbullionprices.com/sitemap.xml"; - -const LOC_TAG = /\s*([\s\S]*?)\s*<\/loc>/gi; -const LD_JSON_BLOCK = - /]*type=["']application\/ld\+json["'][^>]*>([\s\S]*?)<\/script>/gi; - -/** - * Extract every `` URL from a sitemap XML string. - * - * @param {string} xml - Raw sitemap XML. - * @returns {string[]} Every `` value, in document order; `[]` when none. - */ -function extractSitemapLocs(xml) { - if (typeof xml !== "string" || xml.length === 0) { - return []; - } - const locs = []; - let match; - LOC_TAG.lastIndex = 0; - while ((match = LOC_TAG.exec(xml)) !== null) { - locs.push(match[1].trim()); - } - return locs; -} - -/** - * Normalize a candidate URL to its product slug words: take the last non-empty - * path segment, lowercase it, and turn hyphens into spaces. - * - * @param {string} loc - A product URL. - * @returns {string} The normalized slug (space-separated words). - */ -function normalizeSlug(loc) { - const segments = String(loc).split("/").filter(Boolean); - const basename = segments.length > 0 ? segments[segments.length - 1] : ""; - return basename.toLowerCase().replace(/-/g, " "); -} - -/** - * Split a keyword hint (space-separated string or array) into lowercase tokens. - * - * @param {string|string[]} keywords - * @returns {string[]} - */ -function toKeywordTokens(keywords) { - const raw = Array.isArray(keywords) ? keywords : String(keywords ?? "").split(/\s+/); - return raw.map((k) => String(k).toLowerCase().trim()).filter(Boolean); -} - -/** - * Keep only the URLs whose normalized product slug contains EVERY keyword. - * - * @param {string[]} locs - Candidate URLs (e.g. from `extractSitemapLocs`). - * @param {string|string[]} keywords - Space-separated string or token array. - * @returns {string[]} The matching subset of `locs`, in input order. - */ -function matchCoinSlugs(locs, keywords) { - const tokens = toKeywordTokens(keywords); - if (!Array.isArray(locs) || tokens.length === 0) { - return []; - } - return locs.filter((loc) => { - const slugWords = new Set(normalizeSlug(loc).split(/\s+/).filter(Boolean)); - return tokens.every((token) => - token - .split(/\s+/) - .filter(Boolean) - .every((word) => slugWords.has(word)) - ); - }); -} - -/** - * Classify a slug into a ranking tier for `rankByYear`. - * - * @param {string} loc - * @param {number} currentYear - * @returns {{ tier: number, year: number }} Lower `tier` ranks first; within the - * dated tier, higher `year` ranks first. - */ -function slugRank(loc, currentYear) { - const segments = String(loc).split("/").filter(Boolean); - const basename = (segments.length > 0 ? segments[segments.length - 1] : "").toLowerCase(); - if (basename.startsWith(`${currentYear}-`)) { - return { tier: 0, year: currentYear }; - } - if (basename.includes("random-year")) { - return { tier: 1, year: 0 }; - } - const yearMatch = basename.match(/^(\d{4})-/); - if (yearMatch) { - return { tier: 2, year: Number(yearMatch[1]) }; - } - return { tier: 3, year: 0 }; -} - -/** - * Order slugs by preference: current-year ▸ random-year ▸ older-dated - * (descending) ▸ undated. Does not mutate the input. - * - * @param {string[]} slugs - * @param {number} currentYear - * @returns {string[]} A new, preference-ordered array. - */ -function rankByYear(slugs, currentYear) { - if (!Array.isArray(slugs)) { - return []; - } - return [...slugs].sort((a, b) => { - const ra = slugRank(a, currentYear); - const rb = slugRank(b, currentYear); - if (ra.tier !== rb.tier) { - return ra.tier - rb.tier; - } - if (ra.tier === 2) { - return rb.year - ra.year; - } - return 0; - }); -} - -/** - * Read the first schema.org ItemList `name` from an FBP page's JSON-LD. - * - * @param {string} html - * @returns {string|null} The ItemList `name`, or `null` when absent/malformed. - */ -function extractItemListName(html) { - if (typeof html !== "string" || html.length === 0) { - return null; - } - let match; - LD_JSON_BLOCK.lastIndex = 0; - while ((match = LD_JSON_BLOCK.exec(html)) !== null) { - let parsed; - try { - parsed = JSON.parse(match[1]); - } catch { - continue; - } - if (parsed && parsed["@type"] === "ItemList" && typeof parsed.name === "string") { - return parsed.name; - } - } - return null; -} - -/** - * Verify a candidate page's ItemList name contains every keyword (fail closed). - * - * @param {string} html - The fetched candidate page HTML. - * @param {string|string[]} keywords - * @returns {boolean} `true` only when a name is present and matches all keywords. - */ -function verifyCandidateName(html, keywords) { - const name = extractItemListName(html); - if (!name) { - return false; - } - const normalized = name.toLowerCase(); - const tokens = toKeywordTokens(keywords); - return tokens.length > 0 && tokens.every((token) => normalized.includes(token)); -} - -/** - * Cold-path resolver entry point. - * - * Loads coins, fetches the FBP sitemap once, and for each coin carrying an - * `fbp_match` hint: match → rank → pick top → fetch + verify → upsert. Any coin - * that fails to match, fetch, or verify is skipped with its existing `fbp_url` - * left intact (fail closed). - * - * @param {object} [options] - * @param {boolean} [options.dryRun=false] - Print intended upserts, write nothing. - * @param {string[]|null} [options.coinsFilter=null] - Restrict to these slugs. - * @returns {Promise<{ resolved: number, skipped: number }>} - */ -async function main({ dryRun = false, coinsFilter = null } = {}) { - const { createSqldClient } = await import("./sqld-client.js"); - const client = createSqldClient(); - await initProviderSchema(client); - - const currentYear = new Date().getUTCFullYear(); - const allCoins = await getAllCoins(client); - const coins = - coinsFilter && coinsFilter.length > 0 - ? allCoins.filter((coin) => coinsFilter.includes(coin.slug)) - : allCoins; - - const sitemapXml = await fetchFbpPage(SITEMAP_URL); - const locs = extractSitemapLocs(sitemapXml); - console.log( - `[resolve-fbp-slugs] sitemap: ${locs.length} URLs; ${coins.length} coin(s) in scope` - ); - - let resolved = 0; - let skipped = 0; - - for (const coin of coins) { - if (!coin.fbp_match) { - continue; - } - - const matches = matchCoinSlugs(locs, coin.fbp_match); - const candidate = rankByYear(matches, currentYear)[0]; - if (!candidate) { - console.warn(`[resolve-fbp-slugs] SKIP ${coin.slug}: no slug matches "${coin.fbp_match}"`); - skipped += 1; - continue; - } - - let candidateHost; - try { - candidateHost = new URL(candidate).host; - } catch { - candidateHost = ""; - } - if (candidateHost !== "findbullionprices.com") { - console.warn(`[resolve-fbp-slugs] SKIP ${coin.slug}: candidate off-host (${candidate})`); - skipped += 1; - continue; - } - - let html; - try { - html = await fetchFbpPage(candidate); - } catch (err) { - console.warn( - `[resolve-fbp-slugs] SKIP ${coin.slug}: fetch failed (${candidate}): ${err.message}` - ); - skipped += 1; - continue; - } - - if (!verifyCandidateName(html, coin.fbp_match)) { - console.warn(`[resolve-fbp-slugs] SKIP ${coin.slug}: JSON-LD name mismatch (${candidate})`); - skipped += 1; - continue; - } - - if (dryRun) { - console.log(`[resolve-fbp-slugs] DRY-RUN ${coin.slug} → ${candidate}`); - resolved += 1; - continue; - } - - await upsertCoin(client, { ...coin, fbp_url: candidate }); - console.log(`[resolve-fbp-slugs] OK ${coin.slug} → ${candidate}`); - resolved += 1; - } - - console.log( - `[resolve-fbp-slugs] done: ${resolved} resolved, ${skipped} skipped${dryRun ? " (dry-run)" : ""}` - ); - return { resolved, skipped }; -} - -/** - * Parse CLI args into `main()` options. Supports `--dry-run` and a `COINS=` - * comma-separated slug filter (arg or environment variable). - * - * @param {string[]} argv - Typically `process.argv.slice(2)`. - * @param {NodeJS.ProcessEnv} [env=process.env] - * @returns {{ dryRun: boolean, coinsFilter: string[]|null }} - */ -function parseArgs(argv, env = process.env) { - let dryRun = false; - let coinsRaw = env.COINS ?? null; - for (const arg of argv) { - if (arg === "--dry-run") { - dryRun = true; - } else if (arg.startsWith("COINS=")) { - coinsRaw = arg.slice("COINS=".length); - } else { - throw new Error(`unrecognized argument: ${arg}`); - } - } - const coinsFilter = coinsRaw - ? coinsRaw - .split(",") - .map((slug) => slug.trim()) - .filter(Boolean) - : null; - return { dryRun, coinsFilter }; -} - -const isMain = - typeof process.argv[1] === "string" && - realpathSync(process.argv[1]) === fileURLToPath(import.meta.url); -if (isMain) { - try { - await main(parseArgs(process.argv.slice(2))); - } catch (err) { - console.error(`[resolve-fbp-slugs] fatal: ${err.message}`); - process.exitCode = 1; - } -} - -export { - extractSitemapLocs, - matchCoinSlugs, - rankByYear, - extractItemListName, - verifyCandidateName, - parseArgs, - main, -}; diff --git a/devops/pollers/shared/resolve-fbp-slugs.test.mjs b/devops/pollers/shared/resolve-fbp-slugs.test.mjs deleted file mode 100644 index 9accd1aa..00000000 --- a/devops/pollers/shared/resolve-fbp-slugs.test.mjs +++ /dev/null @@ -1,85 +0,0 @@ -#!/usr/bin/env node -/** - * TDD contract tests for the year-preferring FBP slug resolver's pure - * functions (STRK-334, design C5) — the core R1.1 / R1.2 behavior. - * - * RED phase: `./resolve-fbp-slugs.js` does not exist yet. - * - * Run with: - * node --test devops/pollers/shared/resolve-fbp-slugs.test.mjs - */ - -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { readFileSync } from "node:fs"; - -const SITEMAP_XML = readFileSync( - new URL("./__fixtures__/fbp-sitemap.xml", import.meta.url), - "utf8" -); - -const BASE = "https://findbullionprices.com/p/"; -const SLUG_2026 = `${BASE}2026-american-silver-eagle-1-oz-bu-coin/`; -const SLUG_RANDOM = `${BASE}american-silver-eagle-1-oz-random-year/`; -const SLUG_2024 = `${BASE}2024-american-silver-eagle-1-oz-bu-coin/`; -const SLUG_UNDATED = `${BASE}american-silver-eagle-1-oz/`; - -const importModule = () => import(new URL("./resolve-fbp-slugs.js", import.meta.url)); - -test("extractSitemapLocs returns every URL in the sitemap", async () => { - const { extractSitemapLocs } = await importModule(); - const locs = extractSitemapLocs(SITEMAP_XML); - - assert.ok(Array.isArray(locs)); - assert.equal(locs.length, 7, "the fixture sitemap declares seven entries"); - assert.ok(locs.includes(SLUG_2026)); - assert.ok(locs.includes(SLUG_UNDATED)); - assert.ok(locs.includes(`${BASE}2026-american-gold-eagle-1-oz-bu-coin/`)); - assert.ok(locs.includes("https://findbullionprices.com/about/")); -}); - -test("matchCoinSlugs keeps only slugs containing every keyword (hyphen-normalized)", async () => { - const { extractSitemapLocs, matchCoinSlugs } = await importModule(); - const locs = extractSitemapLocs(SITEMAP_XML); - - const matched = matchCoinSlugs(locs, "american silver eagle 1 oz"); - assert.deepEqual( - [...matched].sort(), - [SLUG_2024, SLUG_UNDATED, SLUG_RANDOM, SLUG_2026].sort(), - "only the four ASE 1 oz slugs match; gold eagle, maple, and /about/ are excluded" - ); -}); - -test("matchCoinSlugs matches whole words, not substrings (1 oz must not match 10 oz)", async () => { - const { matchCoinSlugs } = await importModule(); - - const oneOz = `${BASE}2026-american-silver-eagle-1-oz-bu-coin/`; - const tenOz = `${BASE}2026-american-silver-eagle-10-oz-bu-coin/`; - - const matched = matchCoinSlugs([oneOz, tenOz], "american silver eagle 1 oz"); - assert.deepEqual( - matched, - [oneOz], - "the '1 oz' hint matches the 1 oz slug but NOT the 10 oz slug (substring bug)" - ); -}); - -test("rankByYear orders current-year ▸ random-year ▸ older-dated ▸ undated", async () => { - const { rankByYear } = await importModule(); - - // Deliberately unsorted input to prove rankByYear does the ordering. - const ranked = rankByYear([SLUG_UNDATED, SLUG_2024, SLUG_2026, SLUG_RANDOM], 2026); - assert.deepEqual(ranked, [SLUG_2026, SLUG_RANDOM, SLUG_2024, SLUG_UNDATED]); - assert.equal(ranked[0], SLUG_2026, "the current-year slug is the top pick"); -}); - -test("rankByYear falls back to the random-year slug when no current-year slug exists", async () => { - const { rankByYear } = await importModule(); - - const ranked = rankByYear([SLUG_UNDATED, SLUG_2024, SLUG_RANDOM], 2026); - assert.equal( - ranked[0], - SLUG_RANDOM, - "with 2026 absent, the random-year slug outranks the older-dated and undated slugs" - ); -}); diff --git a/js/about.js b/js/about.js index ac816181..eac108c1 100644 --- a/js/about.js +++ b/js/about.js @@ -134,11 +134,11 @@ const setupWhatsNewPopupEvents = () => {}; const getEmbeddedWhatsNew = () => { return ` +
  • v3.36.11 – STRK-346: Housekeeping around the JM Bullion price feed: Purely under-the-hood maintenance with no visible change. When JM Bullion’s pricing was rerouted through FindBullionPrices.com, the first attempt included an automatic tool for guessing each coin’s source page — but real-world listings proved too ambiguous for it (a single coin sits beside its tube, monster box, and fractional versions), so those pages are now assigned by hand instead. This release removes the unused guessing tool and its leftover database field. Nothing about your holdings, prices, or the market comparison changes (STRK-346).
  • v3.36.10 – STRK-334: JM Bullion returns to the market comparison: JM Bullion’s own site had started blocking automated price checks — an anti-bot challenge that kept slamming the door on the price poller’s connection — so JM had quietly dropped out of the market price comparison. StakTrakr now sources JM Bullion’s price from FindBullionPrices.com, a public dealer-price aggregator that publishes clean, machine-readable data and explicitly permits its use, and republishes it as an ordinary JM Bullion vendor price. In the same honest spirit, the market footer now carries a small FindBullionPrices.com attribution and thank-you link alongside the other sourcing disclosures. JM prices reappear as the new feed comes online (STRK-334).
  • v3.36.9 – STRK-345: The copper price archive gets its missing months: Copper’s day-by-day price archive on the StakTrakr feed previously began the day copper went live — requests for any earlier day came back empty, even though the daily history existed in the app’s bundled data. A new maintenance tool now fills the archive back to late March 2026, matching the other four metals, using the same daily history that powers the long-range charts. Each backfilled day is honest about its resolution: one daily price point, clearly marked as a single sample rather than invented hourly data. Future metals get this backfill as a standard rollout step so the gap never recurs (STRK-345).
  • v3.36.8 – STRK-344: Copper’s 90-day chart backfills for existing users: The v3.36.7 fix gave copper its first week of hourly history, but on long-standing profiles the 90-day and 180-day views still ran out of road — the deep history was sitting in the app’s bundled seed data the whole time, and the code that hands seed history to charts only ever ran for brand-new profiles. That hand-off is now made per metal: any metal missing older history — like copper on a profile from before it existed — receives its bundled history automatically on the next load, with your live recorded prices always taking precedence. Between the two fixes, enabling a new metal now fills in its complete chart history on every profile, new or old (STRK-344).
  • v3.36.7 – STRK-343: Copper charts now fill in for existing users: If you enabled copper on a profile you’d been using for a while, its little price chart likely drew flat while gold and silver charted fine — the app decided it was “already caught up” by looking at your existing metals and never fetched copper’s first week of hourly history. That check is now made per metal: any metal missing recent history triggers the full seven-day pull, so copper’s chart fills itself in on the next price sync with no action needed. Fresh installs were never affected, and any future metal we add will inherit the fix automatically (STRK-343).
    • -
  • v3.36.6 – STRK-341: The Silver-to-Copper ratio joins the ratios toolkit: The ratio widget now tracks Ag:Cu — how many ounces of copper one ounce of silver buys (around 157 right now) — alongside the classic gold-to-silver ratio and the platinum-group pairs. It appears in the ratios panel and on the full /ratios/ page for everyone, and as a chip on the copper spot card for those who have switched copper on in Settings → Metal Order; if copper is off, no copper figure appears anywhere on your dashboard. The historical chart is honest about its sources: copper’s deep history before 2013 is monthly, so the early years show real monthly points rather than invented daily ones (STRK-341).
    • `; }; diff --git a/js/constants.js b/js/constants.js index acea8e6a..cfff328a 100644 --- a/js/constants.js +++ b/js/constants.js @@ -397,7 +397,7 @@ const CERT_LOOKUP_URLS = { * Updated: 2026-05-12 - STRK-66: Add ¼ Goldback denomination (Idaho, g0.25) */ -const APP_VERSION = "3.36.10"; +const APP_VERSION = "3.36.11"; /** * Numista metadata cache TTL: 30 days in milliseconds. diff --git a/package-lock.json b/package-lock.json index 2306cfdb..83e5f043 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "staktrakr", - "version": "3.36.10", + "version": "3.36.11", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "staktrakr", - "version": "3.36.10", + "version": "3.36.11", "license": "MIT", "devDependencies": { "@playwright/test": "^1.62.1", diff --git a/package.json b/package.json index fee2f362..bb53d315 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "staktrakr", "private": true, - "version": "3.36.10", + "version": "3.36.11", "license": "MIT", "type": "module", "description": "Precious metals inventory tracker", diff --git a/sw.js b/sw.js index 8de50de3..bbe9e6c6 100644 --- a/sw.js +++ b/sw.js @@ -25,7 +25,7 @@ const DEV_MODE = false; // Set to true during development — bypasses all cachi // base the relative "./" cache keys resolve against. const SW_SCOPE_PATH = new URL("./", self.location.href).pathname; -const CACHE_NAME = "staktrakr-v3.36.10-b1786926832"; +const CACHE_NAME = "staktrakr-v3.36.11-b1787004701"; // Offline fallback for navigation requests when all cache/network strategies fail const OFFLINE_HTML = diff --git a/version.json b/version.json index cca665f2..735c1001 100644 --- a/version.json +++ b/version.json @@ -1,5 +1,5 @@ { - "version": "3.36.10", - "releaseDate": "2026-08-16", + "version": "3.36.11", + "releaseDate": "2026-08-17", "releaseUrl": "https://github.com/lbruton/StakTrakr/releases/latest" } From 4db3721054ca4b4168682328425d87f035a29042 Mon Sep 17 00:00:00 2001 From: lbruton Date: Mon, 17 Aug 2026 17:37:23 -0500 Subject: [PATCH 2/2] fix: resolve PR review findings (PR #1469) --- js/about.js | 2 +- sw.js | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/js/about.js b/js/about.js index eac108c1..b2299a60 100644 --- a/js/about.js +++ b/js/about.js @@ -134,7 +134,7 @@ const setupWhatsNewPopupEvents = () => {}; const getEmbeddedWhatsNew = () => { return ` -
  • v3.36.11 – STRK-346: Housekeeping around the JM Bullion price feed: Purely under-the-hood maintenance with no visible change. When JM Bullion’s pricing was rerouted through FindBullionPrices.com, the first attempt included an automatic tool for guessing each coin’s source page — but real-world listings proved too ambiguous for it (a single coin sits beside its tube, monster box, and fractional versions), so those pages are now assigned by hand instead. This release removes the unused guessing tool and its leftover database field. Nothing about your holdings, prices, or the market comparison changes (STRK-346).
    • +
  • v3.36.11 – STRK-346: Housekeeping around the JM Bullion price feed: Purely under-the-hood maintenance with no visible change. When JM Bullion’s pricing was rerouted through FindBullionPrices.com, the first attempt included an automatic tool for guessing each coin’s source page — but real-world listings proved too ambiguous for it (a single coin sits beside its tube, monster box, and fractional versions), so those pages are now assigned by hand instead. This release removes the unused guessing tool and stops using the extra database field it relied on. Nothing about your holdings, prices, or the market comparison changes (STRK-346).
  • v3.36.10 – STRK-334: JM Bullion returns to the market comparison: JM Bullion’s own site had started blocking automated price checks — an anti-bot challenge that kept slamming the door on the price poller’s connection — so JM had quietly dropped out of the market price comparison. StakTrakr now sources JM Bullion’s price from FindBullionPrices.com, a public dealer-price aggregator that publishes clean, machine-readable data and explicitly permits its use, and republishes it as an ordinary JM Bullion vendor price. In the same honest spirit, the market footer now carries a small FindBullionPrices.com attribution and thank-you link alongside the other sourcing disclosures. JM prices reappear as the new feed comes online (STRK-334).
  • v3.36.9 – STRK-345: The copper price archive gets its missing months: Copper’s day-by-day price archive on the StakTrakr feed previously began the day copper went live — requests for any earlier day came back empty, even though the daily history existed in the app’s bundled data. A new maintenance tool now fills the archive back to late March 2026, matching the other four metals, using the same daily history that powers the long-range charts. Each backfilled day is honest about its resolution: one daily price point, clearly marked as a single sample rather than invented hourly data. Future metals get this backfill as a standard rollout step so the gap never recurs (STRK-345).
  • v3.36.8 – STRK-344: Copper’s 90-day chart backfills for existing users: The v3.36.7 fix gave copper its first week of hourly history, but on long-standing profiles the 90-day and 180-day views still ran out of road — the deep history was sitting in the app’s bundled seed data the whole time, and the code that hands seed history to charts only ever ran for brand-new profiles. That hand-off is now made per metal: any metal missing older history — like copper on a profile from before it existed — receives its bundled history automatically on the next load, with your live recorded prices always taking precedence. Between the two fixes, enabling a new metal now fills in its complete chart history on every profile, new or old (STRK-344).
    • diff --git a/sw.js b/sw.js index bbe9e6c6..7ad263ea 100644 --- a/sw.js +++ b/sw.js @@ -25,7 +25,7 @@ const DEV_MODE = false; // Set to true during development — bypasses all cachi // base the relative "./" cache keys resolve against. const SW_SCOPE_PATH = new URL("./", self.location.href).pathname; -const CACHE_NAME = "staktrakr-v3.36.11-b1787004701"; +const CACHE_NAME = "staktrakr-v3.36.11-b1787006243"; // Offline fallback for navigation requests when all cache/network strategies fail const OFFLINE_HTML =