Small-LLM × novel-DSL benchmark: Qwen2.5-1.5B goes 0% → 85%

[piroz]

Completes step 2 from the post-PR-#3 plan: a public ablation bench that shows what noroshi buys on a 1B-class on-device model.

Headline

20 novel creative-coding tasks (distinct from the few-shot bank), driven through 5 ablation cells with Qwen2.5-1.5B-Instruct via local Ollama on an RTX 2060 Mobile:

Ablation	Valid	Success	Avg attempts	Avg latency
baseline (task only)	0/20	0%	1.0	492 ms
+ BNF grammar in prompt	1/20	5%	1.0	645 ms
+ few-shot examples	11/20	55%	1.0	443 ms
+ retry-with-feedback (×3)	15/20	75%	1.8	727 ms
+ best-of-3 with GrammarAwareRanker	17/20	85%	4.7	1,539 ms

A 1.5B model on a 4-year-old laptop GPU clears 85% of novel DSL tasks without any fine-tuning.

What each row buys

• 0 → 5%: grammar alone barely moves the needle on a model this size.
• 5 → 55%: derivation-first few-shot is the big lever — biggest single jump.
• 55 → 75%: retry-with-feedback recovers 4 of the 9 remaining misses by piping the validator's error back into the next prompt.
• 75 → 85%: best-of-3 + GrammarAwareRanker (PR Add GrammarAwareRanker for best-of-N self-consistency #3) clears 2 more cases where retry's correction didn't generalise.

Latency tradeoff: +rerank triples wall time for +10 pt. Right when the human-noticing threshold matters less than success rate (one-shot prompts), wrong for keystroke-level interactivity. Documented in the bench README.

Files

• examples/creative-coding-p5js/bench/tasks.ts — 20 tasks across static / repeat / tick / mouse / time.
• examples/creative-coding-p5js/bench/run.ts — env-configurable (endpoint, model, API key), prints per-cell trace and a summary table, writes results-<model>.json.
• examples/creative-coding-p5js/bench/results-qwen2.5_1.5b.json — frozen run, checked in so the README numbers are reproducible.
• examples/creative-coding-p5js/bench/README.md — methodology, full table, per-row commentary, caveats.
• README.md (root) — headline table mirrored, with a link to the full bench dir.

How to reproduce

ollama pull qwen2.5:1.5b
npx tsx examples/creative-coding-p5js/bench/run.ts

Other models / endpoints via env vars (NOROSHI_BENCH_MODEL, NOROSHI_BENCH_ENDPOINT, NOROSHI_BENCH_API_KEY).

Not in scope (follow-ups)

• Second-model run (e.g. llama3.2:1b, gemma-2-2b-it) so we can factor model-vs-pipeline.
• Best-of-N with N > 3 — diminishing returns at this size are likely but unmeasured.
• CRANE-style derivation/output delimiter split (option A in the research spike). The +few-shot row already includes derivation as Wang et al. originally proposed; CRANE goes further and we have a separate issue for it.