Can an unfamiliar coding agent finish your quickstart using only what an ordinary developer can see? This project runs that journey, preserves the evidence, and produces a repeatable 0–100 audit.
The initial release deliberately tests one narrow question well: can the agent discover, install, build, debug, and validate one minimal application? It reports before it patches anything.
- Task completion and deterministic success checks
- Documentation navigation and search operations
- Unsupported assumptions and missing prerequisites
- Ambiguous, stale, or version-inconsistent instructions
- Failed code examples and sandbox command errors
- Evidence-backed documentation suggestions
- A JSON scorecard plus a readable Markdown report
Requirements: Python 3.11+, a running Docker daemon, and HF_TOKEN for the configured Hugging Face
Inference Provider model.
python -m venv .venv
source .venv/bin/activate
python -m pip install -e .
cp examples/smolagents-quickstart.toml my-audit.toml
readiness-audit validate my-audit.toml
readiness-audit run my-audit.toml --output audit-results/smolagentsThe command writes report.md and scorecard.json. An incomplete developer task exits with status
2, making audits suitable for release-candidate reporting without pretending that a probabilistic
agent is a unit test.
- Gradio Space — public portfolio interface and curated runner
- Audit Dataset — versioned scorecards and redacted artifacts
- Run-artifact Bucket — mutable per-run storage
- Project Collection — grouped Hub portfolio surface
The Space uses short-lived Hugging Face OAuth for live audits. Signed-in users authorize only Jobs and Inference Providers; Sandbox and inference usage are billed directly to their Hugging Face namespace. The public UI serves an explicitly labeled fixture to signed-out users.
Public runs offer a bounded set of tool-capable model/provider pairs and three journey personas. All
personas use the same sandbox-safe ToolCallingAgent; the persona changes the developer perspective,
not the security boundary. The default openai/gpt-oss-120b route uses Cerebras. At the rates shown
in the UI, a typical 18-step run is estimated at roughly $0.04–$0.10 for inference, plus the separate
time-based dedicated Sandbox charge. The selected CPU Basic flavor is currently $0.01/hour, so a
10-minute Sandbox adds roughly $0.0017. Actual cost varies with transcript and output length.
The curated Hugging Face journey remains the default one-click target. Signed-in users may also select Custom public guide and provide a public HTTPS documentation entry, a repository on a supported public Git host, a task, and paired success checks. Custom fields remain hidden for curated runs. The controller rejects local/private-network documentation destinations and redirect hops; custom repository code and validation commands execute only inside the user's dedicated Sandbox. Custom inputs and redacted results are published only after the user explicitly confirms they are public.
OAuth token responses remain in a short-lived server-side session. The browser receives only an opaque session identifier, avoiding browser cookie-size failures while keeping credentials out of Gradio event payloads and repository code.
There are two supported billing modes:
Owner-funded curated demo: create a dedicated fine-grained controller token with Jobs/Sandbox,
Inference Providers, and write access only to the audit Dataset. Add it to the Space as HF_TOKEN,
confirm the account has a positive compute credit balance, then set ENABLE_LIVE_AUDITS=1. Keep the
target allowlist, concurrency_limit=1, step cap, command timeout, and dedicated Sandbox backend.
User-funded audits (implemented by the public Space): the Space requests only jobs and
inference-api. Its server handler rejects signed-out calls, verifies the granted scopes and token
lifetime, then passes the short-lived token explicitly to both InferenceClientModel and
Sandbox.create(token=..., namespace=<signed-in-user>). The Sandbox API is the authority for prepaid
credit eligibility. Hugging Face bills Sandbox/Job and inference usage to that user.
forward_hf_token=False prevents repository code from receiving the token. One in-memory gate
prevents concurrent runs and enforces a ten-minute user cooldown.
Public result publication is deliberately separate. Set a dedicated RESULTS_PUBLISH_TOKEN Space
secret with write access only to the results Dataset; otherwise results remain in the browser session.
The user's OAuth token is never used to write into the project owner's Dataset.
The Space displays this persistence state before an audit so a successful run cannot silently appear
durable when publication is not configured.
Each publication validates the scorecard against the bundled schema, then writes the run artifacts,
the current schema, and an idempotent data/runs.jsonl Dataset Viewer index in one parent-guarded Hub
commit. Concurrent publications retry against the new parent rather than overwriting another run.
[target]
name = "Acme SDK quickstart"
docs_url = "https://docs.acme.dev"
repository_url = "https://github.com/acme/sdk.git"
task = "Build a minimal app that creates and retrieves one widget."
success_criteria = ["app.py exists", "the smoke test exits zero"]
validation_commands = ["test -f app.py", "python app.py"]
allowed_web_origins = ["https://docs.acme.dev"]
[runtime]
model_id = "openai/gpt-oss-120b"
inference_provider = "cerebras"
agent_persona = "new_developer"
max_steps = 18
docker_image = "python:3.12-bookworm"
sandbox_backend = "docker" # use "hf" for a dedicated Hugging Face Sandbox
hf_sandbox_flavor = "cpu-basic"Make each validation command correspond to the success criterion at the same index. Commands execute
inside /workspace/repo in the disposable container. The docs browser refuses URLs outside the
allowlist. The public configuration allowlists model/provider pairs verified against Hugging Face's
live router metadata during release. Each run establishes Sandbox billing eligibility before performing
a file-writing tool-protocol inference probe. The model supplies file content as a JSON array of source lines; the
controller validates the path and writes it without placing model-generated multiline content in a
shell command. Infrastructure failures are marked invalid and are not assigned readiness scores.
The MVP uses one smolagents ToolCallingAgent and eight narrowly scoped tools. This is intentional:
the agent may decide what to read and try, but every repository command crosses a Docker boundary.
The tool-calling agent does not execute model-generated Python on the host. A future CodeAgent backend
can use smolagents' Docker executor once remote tool transport and this project's threat model are
validated end to end.
Cloud execution preserves the same split. The Space/controller owns model and Hub credentials while
repository code runs in a dedicated Hugging Face Sandbox created with forward_hf_token=False.
For public runs, the controller holds the signed-in user's short-lived token only for the duration of
the request. Audited code never receives it. Dedicated Sandboxes are used instead of pooled
Sandboxes because repositories must be treated as mutually untrusted.
public docs ──► bounded docs browser ─┐
├─► journey recorder ─► deterministic rubric
git repository ─► Docker sandbox ────┘ │
├─ report.md
smolagents agent ─► choose next tool / classify evidence └─ scorecard.json
Scoring is ordinary Python in scoring.py, not another model
opinion. The agent classifies evidence; the rubric applies fixed weights. Task-completion points and a
successful CLI exit require deterministic validation commands; an agent-reported completion without
validation receives no completion points. See the
JSON Schema and methodology.
- Docker is mandatory; there is no host-shell fallback.
- Containers drop all Linux capabilities, use
no-new-privileges, a read-only root, CPU/memory/PID limits, and a disposable bind-mounted workspace. - Documentation navigation is origin-allowlisted, public-network-only, and response sizes are bounded.
- The sandbox currently has network access because real quickstarts install dependencies. Do not pass production credentials into it. Network policy and secret brokering belong on the roadmap.
- Treat browsed documentation and cloned repositories as untrusted prompt-injection sources.
- Every documentation redirect hop is checked against the configured origin allowlist.
Curated journeys state what the sandbox provides. Neutral infrastructure such as Bash, Git, curl, Python, and ordinary shell utilities is an auditor responsibility and must not become a false docs finding. Custom journeys may require the agent to discover and install product-specific prerequisites.
python -m pip install -e '.[dev]'
ruff check src tests
pytest -q- Record exact per-run token usage and show model, Sandbox, and total cost.
- Add an evaluator pass that adjudicates likely agent errors without changing deterministic scores.
- Add E2B, Modal, and Hugging Face Sandbox backends.
- Compare
llms.txt, docs, API references, release notes, and source for contradictions. - Draft patch files; keep PR creation human-approved.
- Only then add manager/specialist delegation behind the existing journey-persona comparison.
Apache-2.0 licensed.