research: verifier head-to-head — re-feed vs true-KV replay (H100, vLLM 0.23)

[matteso1]

The GPU run behind the inference-receipt thesis (the $0.65 H100 experiment). A replay verifier compares a recomputed token signature to a commitment; the commitment is the decode-time logprob, but DiFR/TOPLOC/SVIP/VeriLLM all reconstruct by re-feeding the transcript — and re-feed (bulk prefill) != decode. A thaw verifier replays from the true decode-time KV (.thawkv) and matches by construction.

Fresh generations, Qwen2.5-7B, 1× H100, vLLM 0.23.0, 4,652 tokens, prefix caching off (re-feed = clean fresh prefill):

• false-reject: at τ=0.01 nats a re-feed verifier wrongly rejects 6.9% of genuine tokens; true-KV verifier 0%.
• forgery masking: at low-margin decision tokens, re-feed drift exceeds the top1–top2 gap 19.8% of the time (1 in 5) — a single-token swap hides inside the verifier's own noise; a true-KV verifier (drift ~0) catches it.

So re-feed verification pays a false-reject cost on genuine output and has a forgery-acceptance hole at the tokens that matter; thaw's true-KV inference receipt removes both.

benchmarks/verifier_headtohead.py + site/receipts/2026-06-25_h100_verifier_headtohead.json. Honest scope: true-KV arm sound by construction (decode-time logprob = commitment, .thawkv bit-identical per separate receipts); a deployed-verifier head-to-head + cross-model generality are the follow-ups.

Summary by CodeRabbit

• New Features
  • Added a new benchmark script to compare two verifier replay approaches on math problem prompts.
  • Now reports drift statistics, false-reject rates across tolerance settings, and token-masking rates in JSON output.
  • Added a new experiment receipt with hardware, version, and configuration details plus reproducible command information.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
site	Ready	Preview, Comment	Jun 25, 2026 3:27am

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3c2da30f-b167-4c53-aa95-6ac3cfea7063

📥 Commits

Reviewing files that changed from the base of the PR and between 627597d and 893f962.

📒 Files selected for processing (2)

• benchmarks/verifier_headtohead.py
• site/receipts/2026-06-25_h100_verifier_headtohead.json

---

📝 Walkthrough

Walkthrough

Adds a benchmark script that compares decode-time and re-feed verifier logprobs on fixed prompts, computes drift and false-reject metrics across tolerances, and writes a JSON receipt with the measured results.

Changes

Verifier head-to-head benchmark

Layer / File(s)	Summary
Benchmark scaffold and prompt set benchmarks/verifier_headtohead.py	Defines the embedded prompt set and logprob helper functions for token lookup and margin calculation.
Decode-time and re-feed capture benchmarks/verifier_headtohead.py	Parses CLI args, runs generation with prefix caching disabled, and re-feeds each transcript to collect prompt logprobs.
Drift and masking analysis benchmarks/verifier_headtohead.py	Aligns decode-time and re-feed logprobs, computes drift, margin, and masking metrics, and emits JSON results.
Experiment receipt site/receipts/2026-06-25_h100_verifier_headtohead.json	Adds the experiment metadata, drift statistics, false-reject sweep, forgery-masking metrics, scope text, and reproducer command.

Sequence Diagram(s)

sequenceDiagram
  participant main as main()
  participant llm as LLM
  participant tokenizer as tokenizer
  participant stdout as stdout
  participant jsonout as json-out

  main->>tokenizer: format PROMPTS with chat template
  main->>llm: generate completions with decode-time logprobs
  llm-->>main: committed token logprobs and top-5 alternatives
  main->>llm: re-feed each transcript as a single prefill
  llm-->>main: prompt-logprobs for each token
  main->>stdout: print results as JSON
  main->>jsonout: write JSON when requested

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐇 I hop through prompts and logprob dew,
Re-fed the transcript, line by line anew.
Drift stats twinkle, margins glow,
JSON receipts make the burrow know.
Thump-thump—truth and replay both show.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch research/verifier-headtohead

---

_{Comment @coderabbitai help to get the list of available commands.}