Skip to content

Latest commit

 

History

History

ticket-triage

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 

README.md

Worked example — support ticket triage

A full run of the skill on a real-shaped problem, start to finish. This is what using it actually looks like.

The request

"We get 100+ support tickets a day. I want a multi-agent system: one agent labels them, one assigns priority, one detects duplicates."

Three agents, named up front. That is the tell.

Phase 0 — scope

Question Answer
Task raw tickets in → {label, priority} per ticket + a digest
Volume 12 in the fixture; ~100/day in reality. Interactive, not batch
Failure that hurts Under-prioritising. A cross-account data leak filed P2 sits in the backlog for a week. Over-prioritising just annoys someone
Ceilings Seconds, cents
Current pain Manual triage, inconsistent between people

Note how asymmetric the failure is. That single answer determines the whole design: the reviewer's only job is catching under-prioritisation.

Phase 1 — the ladder

Level Verdict
1 · Plain script Climb. Keyword rules can label some tickets, but "how severe is this really" is judgement a rule cannot encode
2 · Loop Climb. Schema validity is assertable (priority ∈ P0..P3); correctness of a priority is not. And a wrong priority is the failure that hurts
3 · Loop + reviewer STOP. A reviewer seeing tickets + assignments — but never the writer's reasoning — catches under-prioritisation
4 · Panel No. One defect class matters. One lens covers it
5 · Fan-out No. 100 short tickets is nowhere near a context window
6 · Durable No. Runs in seconds, on demand

Three "agents" collapsed into one writer. Labelling, prioritising and duplicate-detection all need the same context — the full ticket list. Splitting them means three models each seeing less than the one model would have, and duplicate-detection is impossible without seeing every ticket anyway. That is the Data Processing Inequality argument from references/evidence.md in miniature: each handoff can only lose information.

Phase 2 — the design

        ┌──────────────────────────────────────┐
        │ load  (fn)                           │
        │ read tickets.json, skip malformed    │
        └──────────────────┬───────────────────┘
                           ▼
        ┌──────────────────────────────────────┐ ◀────────┐
        │ triage                               │          │
        │ label + priority + dupes — THE writer│          │ FAIL
        └──────────────────┬───────────────────┘          │ (attempts < 3)
                           ▼                              │
        ┌──────────────────────────────────────┐          │
        │ review        read-only, clean context│ ─────────┘
        │ is anything under-prioritised?        │
        └──────────────────┬───────────────────┘
                           ▼ PASS / bounds hit
        ┌──────────────────────────────────────┐
        │ digest  (fn)   render by priority    │
        └──────────────────────────────────────┘
flowchart TD
    S([start]) --> load["load<br/>read tickets.json, skip malformed"]
    load --> triage("triage<br/>label + priority + duplicates — THE writer")
    triage --> review{{"review<br/>read-only, clean context<br/>anything under-prioritised?"}}
    review -->|"FAIL · attempts &lt; 3"| triage
    review -->|"PASS"| digest["digest<br/>render markdown by priority"]
    review -.->|"bounds hit"| park[/"park<br/>ships marked UNREVIEWED"/]
    digest --> E([done])
    park --> E

    classDef fixed fill:#e8eef2,stroke:#5b7183,color:#1d2b36
    classDef model fill:#dbeafe,stroke:#2563eb,color:#12244a
    classDef term  fill:#dcfce7,stroke:#15803d,color:#0a2e15
    classDef halt  fill:#fee2e2,stroke:#b91c1c,color:#3f0d0d
    class load,digest fixed
    class triage,review model
    class S,E term
    class park halt
Loading

Edit this diagram — opens mermaid.live with it already loaded.

State

Field Owner (only writer) Reducer
tickets load append
assignments triage only replace
verdict, feedback review only replace
attempts triage counter
tokens_spent any model call sum
errors any append
digest digest replace

Bounds

MAX_ATTEMPTS = 3 · BUDGET_TOKENS = 20_000, both in the loop condition · on exhaustion the digest ships marked UNREVIEWED rather than silently.

Cost — and why level 3 rather than 5

The number the skill makes you produce before building. It is only meaningful as a comparison, so here are all three options for the same 12 tickets:

Option Arithmetic Worst case Buys you
Level 2 — loop, no reviewer 1 × 2.5k ~2.5k —
Level 3 — chosen 2 rounds × (2.5k triage + 1.5k review) ~8k typical, 12k worst the leak gets caught
Level 5 — fan-out per ticket 12 × 2 × (400 + 300) ~17k nothing — and loses duplicate detection

Level 3 costs ~3× the bare loop and buys the one thing that matters: the leak gets caught. Level 5 costs more than level 3 and buys nothing here — the tickets fit in one context, so splitting them only loses cross-ticket information (you can't spot duplicates if each agent sees one ticket).

At 100 tickets/day this is roughly $0.05/day on current pricing.

Phase 3 — target

plain-code.md. Two model calls and one conditional edge do not justify a framework. ~200 lines of Python, no dependencies.

Phase 4 — verification (levels 1–3: assert behaviour, not topology)

All five assertions, run by --selftest:

# Assertion Result
1 Reviewer is read-only — snapshot assignments, run reviewer, compare asserted every round, inline
2 Both bounds are live — never-passing reviewer for the attempt cap, then the same run at a budget one round cannot afford stops at MAX_ATTEMPTS; then stops on spend before MAX_ATTEMPTS
3 Exhaustion terminal reachable and marked digest carries UNREVIEWED
4 Failure isolated — inject a malformed ticket skipped, other 12 complete
5 Counts add up — one assignment per ticket, no duplicate ids catches the reducer bug below

The run

demo run

models   : stub (deterministic)
tickets  : 12
attempts : 2   verdict: PASS: no under-prioritised critical tickets.
tokens   : 240  (budget 20000)
--------------------------------------------------------------
### P0
- `T-1044` [security] API returns other customers' data on /v2/orders
- `T-1049` [outage] Cannot log in at all since this morning

### P1
- `T-1042` [billing] Invoice shows wrong VAT for EU customers
...

attempts: 2 is the whole story. On the first pass the writer filed T-1044 — "API returns other customers' data" — as P2. Plausible, quiet, and wrong. The reviewer, seeing only the tickets and the assignments, flagged it as a cross-account exposure that must be P0. The second pass fixed it.

That is one reviewer catching a real security misclassification, for ~1.5k tokens.

verification

Run it yourself

python examples/ticket-triage/triage.py            # the digest
python examples/ticket-triage/triage.py --selftest # the five assertions

Runs offline with deterministic stub models — no API key. Set ANTHROPIC_API_KEY to use real models (sonnet writer, haiku reviewer, so the producer never grades itself).

What building this caught

Two real defects, both found by running the skill rather than reading it:

  1. A state-drift bug in this example. assignments is a list the design marks replace, but the first apply() helper inferred the reducer from the Python type — list means append. Round 2 stacked on round 1, so T-1044 appeared twice, once at P2 and once at P0. Reducers are now declared, never inferred, and a count assertion catches the whole class.
  2. The same footgun shipped in references/targets/plain-code.md, which demonstrated exactly that type-inferring apply(). Now fixed: the target declares an explicit APPEND set copied from the Phase 2 state table, shows the wrong version under a "never infer the reducer from the type" heading, and requires a count assertion as the guard.
  3. A decorative spend bound, from the other half of the same fix. Declaring reducers cured the append bug and quietly created a sum bug: tokens_spent was left out of APPEND, so it was replaced with the cost of the last call instead of summed. It sat at 240 across a 960-token run, and BUDGET_TOKENS = 20_000 could never bind at any scale. + implements two Phase 2 reducers — append for lists, sum for ints — and only one of them was declared. Invisible to every test, because the attempt cap always fired first. Assertion 2 now forces the spend cap specifically and asserts the attempt cap did not get there first; reverting the one-word fix fails it by name. The same omission was in plain-code.md, and is fixed there too.