ci(ai-gateway): keep /models.json in sync with models.dev#5204
Merged
Conversation
Adds a scheduled GitHub Action + script that verifies the /models.json catalog stays in sync with the upstream models.dev `neon` provider, failing on any model or capability drift. Also regenerates data.json to pick up qwen35's reasoning_options.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…rovider `npm run generate:models` fetches the models.dev neon provider (with all base_model-inherited values resolved) and regenerates data.json, applying the same transform /models.json uses (adds per-model provider, drops description/benchmarks/weights/links, points doc at the AI Gateway page). Pairs with the sync check, which now points users at this command.
…h committed shape
…atalog Regenerated src/app/models.json/data.json from the local models.dev neon provider (neon-catalog-sync-2026-07). Adds the legacy context_over_200k cost block and normalizes tier key order for gpt-5-4, gpt-5-4-mini, gemini-3-flash, and gemini-3-1-flash-lite. Sync check passes (36 models match).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #5203. Adds a drift check that keeps the Neon AI Gateway catalog served at
/models.jsonin sync with its upstream source of truth — theneonprovider in the models.dev API, which Neon maintains..github/workflows/models-sync-check.yml— scheduled (daily) +workflow_dispatchjob that runs the check and fails on any drift.src/scripts/check-models-sync.js— fetcheshttps://models.dev/api.json, extracts theneonprovider, and compares it to the committedsrc/app/models.json/data.json(what backs/models.json). It fails on:name,family,attachment,reasoning,reasoning_options,tool_call,temperature,structured_output,open_weights,knowledge, dates,modalities,limit,cost,status).providerconvenience field we add, anddescription/benchmarks/weights/links.package.json—npm run check:models-syncfor local runs (--citerse,--jsonmachine-readable;NEON_MODELS_URL=https://neon.com/models.jsoncompares the live endpoint instead of the repo file).data.jsonto pick up the latest models.dev Neon changes (addsqwen35-122b-a10b'sreasoning_options).Heads up — currently reports drift (expected)
The companion models.dev PR (anomalyco/models.dev#3019) that adds these 12 models and removes
gpt-5-5is merged but not yet deployed tomodels.dev/api.json. Until it redeploys, the check reports the expected delta:Once models.dev redeploys the merged
neonprovider, the daily job goes green (both catalogs derive from the same TOMLs). The workflow is schedule + manual only (not a PR gate), so it won't block merges.Test plan
node src/scripts/check-models-sync.jsexits1on drift,0when in sync;--jsonand--cimodes verified.eslintclean on the new script.