SkillForge

This code is based on the paper: WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution

Evolve Claude Code skills from measured agent experience. SkillForge runs an agent over scored tasks, mines the traces into a persistent knowledge wiki, proposes one skill change per iteration, and keeps the change only if it measurably improves the validation score. The output is a skill set you can install into .claude/skills/.

Note: this repo comes with its own skill to help you apply the tooling.

Each iteration:

1. Run all training tasks with the current skills.
2. A maintainer agent distills the traces into wiki pattern pages.
3. A proposer agent reads the wiki (and the log of past accepted/rejected edits) and makes exactly one skill change.
4. The change is scored on the validation tasks: better than the best so far → kept, otherwise rolled back. The wiki always keeps what was learned.

The task-solving agent never sees the wiki — skills have to carry the knowledge.

Requirements

• Python 3.11+ and pyyaml (pip install -e ., which also adds a skillforge command)
• The claude CLI on PATH, logged in (only for real runs; the smoke pack runs offline)

Try it (offline, free)

python -m skillforge evolve packs/smoke

The bundled smoke pack uses a simulated agent (runner.backend: fake), so the full loop runs in seconds with zero API spend. Then read the run report:

cat runs/smoke-*/report.md

Worked example: your own pack

A pack is a directory of tasks plus fixed train/val/test splits. Each task gives the agent a prompt and a workspace, then scores the result with a script.

Scaffold it, then duplicate tasks/t001/ for each additional task:

python -m skillforge init mypack

packs/mypack/
├── pack.yaml
├── config.yaml
└── tasks/
    ├── t001/
    │   ├── task.yaml
    │   ├── prompt.md      # what the agent is asked to do
    │   ├── setup.sh       # optional: prepares the workspace
    │   └── verify.sh      # scores the workspace afterwards
    ├── t002/ ...
    └── t003/ ...

pack.yaml — every task id goes in exactly one split:

name: mypack
description: Make failing pytest suites pass
splits:
  train: [t001]
  val:   [t002]
  test:  [t003]

config.yaml:

model: haiku
iterations: 8
runner:
  backend: claude
budget:
  run_usd: 20.0

A task, end to end. tasks/t001/task.yaml:

id: t001
prompt_file: prompt.md
setup: setup.sh
verify: verify.sh
timeout_sec: 300
tags: [pytest]

prompt.md:

tests/test_parser.py fails. Fix the bug in parser.py so the whole
test suite passes. Do not modify the tests.

setup.sh copies the task's fixture files into the fresh workspace ($SKILLFORGE_TASK_DIR points at the task directory):

#!/usr/bin/env bash
set -e
cp -r "$SKILLFORGE_TASK_DIR/fixtures/." .

verify.sh must print SCORE=<0.0..1.0> to stdout — this line is the entire scoring contract (no line or a non-zero exit without it counts as 0.0):

#!/usr/bin/env bash
if python3 -m pytest -q > /dev/null 2>&1; then
  echo "SCORE=1.0"
else
  echo "SCORE=0.0"
fi

A weak scorer is the most common way to get useless skills: if "the agent said it worked" scores 1.0, the loop learns to make the agent say that. Score the outcome, never the claim. Use more train/val tasks than this sketch (8+ / 8+) for results that generalize.

Run it:

python -m skillforge validate packs/mypack        # splits + scorer dry-run
python -m skillforge discriminate packs/mypack    # scorers tell solved from unsolved
python -m skillforge evolve   packs/mypack        # the evolution loop
python -m skillforge report   <run-id>            # re-print the report
python -m skillforge install  <run-id> --scope project   # → .claude/skills/

<run-id> is the directory name under runs/ that evolve prints. The report shows baseline vs final per task, the accept/reject history, cost, and whether the improvement is statistically significant. evolve exits non-zero if the final skills do not beat the no-skill baseline on the held-out test split.

Commands

Command	Purpose
init <name>	Scaffold a pack under packs/<name>
validate <pack>	Check split integrity, dry-run every scorer
discriminate <pack>	Run every scorer on a bare workspace and after the task's solve.sh; fails unless each scores <1.0 then 1.0
baseline <pack>	No-skill scores on val + test
evolve <pack> [--iterations K] [--resume <run-id>] [--seed <skill>]	Run the loop (resumable). --seed starts from an existing skill and requires the result to beat it, not just to beat no-skill
eval <run-id> <pack> [--split test] [--no-skills]	Score a skill set on a split
transfer <run-id> <pack> --model m1,m2	Score the skills under other models vs their no-skill baselines (flags negative transfer)
compare <run-id> <run-id> ...	Aggregate several runs: mean ± stdev of final score and delta
mine [--project <path>] [--since <date>]	Seed the wiki from past Claude Code sessions (hypotheses only)
install <run-id> [--scope project|user]	Copy the evolved skills into .claude/skills/
report <run-id>	Regenerate and print the run report

Everything a run produces lives in runs/<run-id>/: report.md, scores.csv, events.jsonl, and state/ (the evolving skills/ and wiki/ git repos plus the immutable traces under raw/).