CodePulse

Production incident detection, root-cause analysis and policy-controlled auto-recovery, built from scratch.

Telemetry pipeline · anomaly detection · trace-based RCA · evidence-cited AI investigator · guarded remediation · chaos engineering

Node.js 22+ (TypeScript, no build step) · Python 3 · Next.js 16 · OpenTelemetry OTLP · no runtime dependencies in the core (Claude SDK optional)

_{A connection leak gets injected into payments. CodePulse detects it, works out that the cause is payments even though the errors show up in ledger-db, restarts the leaking service, reclaims the leaked connections, checks that the system really recovered, and closes the incident. No human involved. The GIF runs at 2× speed.
▶ Full 2-minute walkthrough (MP4), which also covers the human-approval flow for a bad deploy and the audit log.}

---

Contents

• The 30-second version
• The problem
• What I did differently
• Architecture
• Walkthrough: one incident, end to end
• Chaos benchmark results
• Engineering deep dives
• Problems I hit while building it
• Running it
• Project layout
• Limitations and what's next

---

The 30-second version

Seven microservices (Node.js and Python) run a small e-commerce checkout:

loadgen ─▶ gateway ─▶ users (py)
              └────▶ orders ─▶ inventory (py)
                        ├───▶ payments ─▶ ledger-db (connection pool, 20 slots)
                        └───▶ notifications (fire-and-forget)

Every service exports OpenTelemetry (OTLP/HTTP JSON) traces, metrics and logs to the CodePulse core. When something breaks, CodePulse:

1. Detects the anomaly, typically within 6–10 s, using per-service statistical baselines rather than hand-set thresholds.
2. Groups every red service into one incident, so you get one page instead of an alert storm.
3. Finds the root cause from distributed traces, the dependency graph, deploy history, resource gauges and log signatures, and reports a confidence score.
4. Explains it with evidence. Every sentence in the narrative cites evidence (E1, E2…). If an LLM is configured, claims it makes without valid citations are dropped.
5. Picks a playbook and asks a policy engine whether it may run on its own, needs a human, or must not run at all.
6. Executes, verifies and rolls back if needed. A fix counts as successful only when the affected services are measured healthy for 3 consecutive buckets.
7. Writes it all down in a hash-chained audit log and a postmortem draft.
8. Measures itself. A built-in chaos lab injects real faults and scores detection time, RCA accuracy and recovery time.

---

The problem

Before writing any code I spent time on one question: why is incident response slow even at companies with excellent monitoring? In real postmortems, the time rarely goes into noticing that something is wrong. It goes into these four places:

Where MTTR actually goes	Why it happens	What existing tools do
Alert storms	One root failure turns every service above it red. On-call gets 12 pages for 1 problem.	Static thresholds per service, so each red service fires its own alert.
Symptom ≠ cause	Errors surface where the failure lands, not where it starts. A leak in payments shows up as "pool exhausted" errors in the database.	Dashboards show where it's red, and a human has to work backwards.
Diagnosis by archaeology	The on-call engineer opens 6 dashboards, greps logs, checks the deploy channel, and builds a timeline in their head.	The data is all there, but scattered across tools.
Fear of automation	Auto-remediation that is wrong is worse than none: it restarts the wrong thing, in a loop, at 3 am. So teams keep it off.	Runbook automation fires on a matching alert, with no notion of evidence, blast radius or verification.

The scenario from the brief makes this concrete: payments starts failing → is it payments? the database? the network? A good SRE answers that by following the failing requests downstream, checking who holds the database connections, and noticing that nothing was deployed. CodePulse automates that line of reasoning and shows its work.

---

What I did differently

1. RCA from traces, not from dashboards. CodePulse doesn't ask "which service has the highest error rate?" (in an outage, they all do). For every failing trace it finds the deepest failing span (the error origin). For slow traces it computes each service's exclusive self time and compares it with the pre-incident baseline. This separates the service that fails from the services that merely wait.

2. It can tell where errors surface from where they start. In the connection-leak scenario, 100% of failing traces originate in ledger-db ("pool exhausted"). A naive RCA blames the database. CodePulse also reads the pool's per-client lease gauges, sees that payments holds 20/20 connections and has kept one for over 10 s (normal queries take ~10 ms), and transfers the blame: the database is the victim, payments is the cause. The UI shows both: "errors surface in ledger-db, cause is upstream."

3. The LLM narrates; it doesn't decide. The AI investigator (Claude, optional) only receives the evidence pack the deterministic engine built, and must return structured JSON in which every claim cites evidence IDs. The output is validated: claims that cite unknown or no evidence are dropped and counted, and if the model names a different root cause than the ranking, that is flagged and the ranking wins. With no API key, a deterministic narrator produces the same structure, so the product works fully offline.

4. Remediation is guarded, verified and audited. A playbook never decides whether it may run; a pure-function policy engine does, using confidence floors, risk levels, rate limits, a loop guard, a change freeze, and a "humans only" rule for stateful services. Success means measured recovery, not "the command returned 200". Every proposal, approval, step and verification goes into a SHA-256 hash-chained audit log, so editing any past entry breaks verification.

5. Chaos experiments are blind. Fault injection talks straight to a service's admin API and is never written to anything the RCA engine reads. The engine only sees what a real system would see: a deploy going through the orchestrator, or a process exiting. RCA accuracy measured this way means something.

6. Built from first principles, wire-compatible with the standard. The telemetry SDK (Node and Python), OTLP receiver, quantile sketch, log-template miner, anomaly detector, RCA, policy engine and executor are all written here, in about 5,700 lines. The core has no runtime dependencies; the only package it can load is the official Anthropic SDK, and only when an API key is configured. Because the wire format is real OTLP/HTTP JSON and context propagation is real W3C traceparent, a trace starts in the Node gateway and continues through the Python services with no shared code.

---

Architecture

flowchart LR
  subgraph Services["Demo microservices (Node.js + Python)"]
    GW[gateway] --> US[users · py]
    GW --> OR[orders]
    OR --> IN[inventory · py]
    OR --> PA[payments]
    OR -.fire & forget.-> NO[notifications]
    PA --> DB[(ledger-db<br/>pool 20)]
  end

  Services -- "OTLP/HTTP JSON<br/>traces · metrics · logs" --> ING

  subgraph Core["CodePulse core (Node.js, zero deps)"]
    ING[OTLP receiver<br/>:4318] --> ST[Bucketed store<br/>RED aggregates · sketches<br/>spans · log templates · gauges]
    ST --> AD[Anomaly engine<br/>EWMA · Wilson · hysteresis]
    AD --> IM[Incident manager<br/>correlation · severity]
    IM --> RCA[RCA engine<br/>origin · self-time · graph<br/>changes · leaks · logs]
    RCA --> INV[Investigator<br/>deterministic / Claude<br/>citation-validated]
    RCA --> PB[Playbooks]
    PB --> POL{Policy engine}
    POL -- auto --> EX[Executor<br/>steps → verify → rollback]
    POL -- approval --> HUM((human))
    HUM --> EX
    EX --> ORC[Orchestrator<br/>local processes · k8s-shaped]
    EX --> AUD[(Hash-chained<br/>audit log)]
    CH[Chaos engine<br/>blind experiments] -. faults .-> Services
  end

  ORC -. restart · rollback · flags .-> Services
  Core -- "REST + SSE :7000" --> UI[Next.js command center]

Loading

The control loop runs once per 3 s bucket, after a 1.5 s grace period so late spans land in the right bucket:

1. Snapshot every service (requests, errors, p50/p95/p99, heartbeat) and feed the detectors.
2. Open, grow or resolve incidents. All concurrent anomalies join the open incident.
3. Re-run RCA for open incidents and refresh the narrative.
4. Once the hypothesis has been stable for 2 consecutive ticks, plan a playbook, ask the policy engine, and hand it to the executor.
5. While a fix is executing or has been verified, the RCA is frozen. Post-fix data says nothing about the cause.

---

Walkthrough: one incident, end to end

The setup. payments has a classic bug: its fraud-review branch returns early and never releases its ledger-db connection. Nothing fails at first; the pool just quietly fills up.

~10 s after the fault: the pool is full and ledger-db starts rejecting new sessions. Within two buckets the detectors on ledger-db, payments, orders and gateway all fire, and CodePulse opens one SEV1 incident instead of four alerts. The dependency graph marks payments as ROOT CAUSE, although ledger-db is the service throwing the errors.

The candidate ranking shows why. ledger-db has every symptom feature (origin, latency, frontier, saturation), but the leak evidence transfers its score to payments. The policy engine checks seven guardrails: the playbook is low-risk, confidence is 98%, the target is stateless, and the rate limit isn't hit, so it runs automatically:

1. restart_service payments: stops the leak (payments-ea8216 → payments-f75d37)
2. terminate_db_sessions ledger-db client=payments: reclaims the 20 leaked connections (the database equivalent of pg_terminate_backend)
3. Verify payments, ledger-db and gateway are healthy for 3 consecutive buckets

Every piece of evidence is concrete and numbered. Error signatures are template-mined: pool exhausted: <num>/<num> connections in use…. Errors that merely wrap a downstream error (orders failed: payments failed: ledger-db failed: …) are recognised as propagation and are not counted as independent evidence.

A sample failing trace crosses Node and Python services, and the failure is visible at the bottom of the waterfall: ledger-db POST /sessions, after ~500 ms of waiting in the pool queue.

The second scenario in the video is a bad deploy (orders 2.5.0). Rollback is a medium-risk playbook, so the policy engine routes it to a human:

After approval, CodePulse rolls back, verifies recovery, and every step lands in the audit log:

Each incident also produces a postmortem draft in Markdown (GET /api/incidents/:id/report): impact, timeline, root cause with cited reasoning, the candidate table, remediation steps with their outputs, and follow-ups such as "find the code path in payments that returns without releasing its connection; add idle_in_transaction_session_timeout on ledger-db".

---

Chaos benchmark results

npm run chaos:bench boots the whole platform with synthetic load, runs every experiment back to back, and writes docs/chaos-results.md. Approval-gated playbooks are approved by a simulated on-call after 5 s, so those numbers include a human in the loop.

Experiment	Detected (MTTD)	Root cause found	Confidence	Playbook	Approval	Recovered (MTTR)	Severity	Result
Connection leak in payments	9.8s	✅ payments / connection_pool_leak	98%	restart-leaking-client	auto	42.8s	SEV1	passed
Bad deploy of orders 2.5.0	7.5s	✅ orders / bad_deploy	98%	rollback-deploy	human	34.5s	SEV1	passed
Slow reservations in inventory	7.0s	✅ inventory / latency_degradation	97%	enable-degraded-mode	auto	25.1s	SEV1	passed
notifications process crash	7.3s	✅ notifications / service_down	98%	restart-crashed-service	auto	25.2s	SEV3	passed
Connection leak in payments	10.8s	✅ payments / connection_pool_leak	98%	restart-leaking-client	auto	28.9s	SEV1	passed
Bad deploy of orders 2.5.0	7.7s	✅ orders / bad_deploy	98%	rollback-deploy	human	34.6s	SEV1	passed
Slow reservations in inventory	7.6s	✅ inventory / latency_degradation	97%	enable-degraded-mode	auto	25.5s	SEV1	passed
notifications process crash	8.4s	✅ notifications / service_down	98%	restart-crashed-service	auto	23.5s	SEV3	passed
8/8 experiments recovered; root cause correct in 8/8.

MTTD and MTTR are measured from the moment the fault is injected, so they include the time the fault needs to become visible (the leak needs ~10 s to fill the pool). Detection itself takes 2 buckets (≈6 s) after the first bad bucket. The previous run of this benchmark scored 7/8, which is how I found bug #8 below.

Experiment	What it tests
Connection leak in payments	Symptom ≠ cause. Errors surface in ledger-db; the cause is a leaking client two hops up.
Bad deploy of orders 2.5.0	Change correlation and new log signatures; human-in-the-loop rollback for a medium-risk action.
Slow reservations in inventory	inventory never returns an error. orders times out waiting for it, so the errors appear one hop away. The fix is a reversible degraded-mode flag, not a restart.
notifications crash	Blast-radius awareness. The service is off the critical path, so CodePulse grades it SEV3 (checkout never broke) and simply restarts it.

---

Engineering deep dives

Telemetry pipeline

• SDK (packages/telemetry/index.ts, services/pykit/codepulse_kit.py): AsyncLocalStorage-based context, W3C traceparent in and out, cumulative counters, explicit-bucket histograms, observable gauges, and trace-correlated logs, all batched and exported as OTLP/HTTP JSON every second.
• Telemetry never blocks the app. Export queues are bounded; on overflow or when the collector is down, data is dropped and counted, never buffered without limit. An app that buffers unbounded telemetry while its collector is down is how one outage becomes two.
• Cumulative counters → rates (core/src/ingest/otlp.ts). A process restart resets a counter to 0, and a naive delta then gives a huge negative rate. Resets are detected from a decreasing value or a changed startTimeUnixNano.
• The receiver accepts both hex and base64 trace IDs. The OTLP JSON spec says hex, but some exporters send base64.
• RED metrics are derived from SERVER spans at ingest time (like the OTel Collector's span-metrics connector) and aggregated per 3 s bucket with a log-bucketed quantile sketch (core/src/store/sketch.ts, the idea behind DDSketch): ≤2% relative error, bounded memory, and sketches merge exactly, so 3 s buckets roll up into any window without keeping raw samples.
• Dependency edges come from CLIENT spans (peer.service), so a dead service still appears in the graph: its callers keep reporting failed calls to it.

Anomaly detection (core/src/analysis/anomaly.ts)

A z-score over an EWMA baseline breaks in four predictable ways. Each has a specific guard:

Failure mode	Guard
Baseline poisoning. During a long incident the EWMA learns the broken state and the alert clears itself.	The baseline is frozen while a signal is breaching or anomalous.
Zero variance. Stable traffic has σ≈0, so a 2 ms wobble scores z = 40.	σ is floored relative to the mean (max(σ, 0.3·μ, 15ms)).
Small samples. 1 error in 3 requests is a "33% error rate".	The error rate is scored by its Wilson lower bound, and buckets with < 8 requests are skipped.
Flapping. One noisy bucket opens and closes an incident.	k-of-n debounce (2 breaching buckets to open), hysteresis (3 healthy buckets to close).

There is also a practical-significance gate: 0.1% → 0.4% errors may be 6σ, but nobody should be paged for it. Availability is a heartbeat check, and a service that has never reported is "not onboarded yet", not "down" (see problems).

Root cause analysis (core/src/analysis/rca.ts)

Each candidate service gets independent, explainable features in [0, 1]:

Feature	Weight	Meaning
origin	3.0	Share of failing traces whose deepest error is in this service. A failing CLIENT span with no failing child (a timeout or refused connection) blames the callee.
latency	2.5	Share of the added per-request self time spent in this service, vs a 60 s pre-incident baseline.
leak	3.5	Holds ≥80% of another service's saturated pool, with leases far older than any query.
change	2.5	A deploy (1.0) or flag change (0.6) before onset, with a 5-minute half-life. Changes made by CodePulse itself are excluded.
down	4.0	No heartbeat at all.
newLogs	1.2	An error template that didn't exist before the incident and isn't a wrapper of another service's error.
frontier	1.0	Anomalous, but none of its own dependencies are.
saturation	1.0	Owns a resource pool at ≥90% utilisation.
onset	0.8	How early its anomaly started relative to the others.

score = Σ weight·feature, then a blame-transfer step (a pool owner exhausted by someone else's leak keeps 30% of its score and the holder gets 80% of it), then a softmax over candidates. Confidence is discounted when evidence is thin (few failing traces). The top candidate's feature signature picks the hypothesis: service_down, connection_pool_leak, bad_deploy, latency_degradation or error_spike.

AI investigator (core/src/investigator/narrator.ts)

• Model: claude-opus-5-5 via the official SDK, using structured outputs (output_config.format with a JSON schema), effort: medium, and server-side refusal fallbacks. It is only used when ANTHROPIC_API_KEY is set. Override the model with CODEPULSE_MODEL.
• It receives only the evidence pack (no raw logs, no free browsing) and is told to separate symptom from cause.
• validateLlmOutput drops claims without citations or with citations to non-existent evidence, counts them (shown in the UI), and flags disagreement with the deterministic ranking instead of adopting it.

Policy engine (core/src/remediation/policy.ts)

A pure function of (playbook, context), which makes it trivially unit-testable. Every decision is explainable and is shown check by check in the UI:

Check	Fails →
RCA confidence ≥ 35%	deny (keep investigating)
Playbook hasn't failed on this target in the last 30 min (loop guard)	deny
Playbook risk is low	needs approval
Confidence ≥ 60% for unattended execution	needs approval
Target is stateless	needs approval (humans only for stateful services)
≤ 3 automated actions on this target per hour	needs approval
No change freeze (CODEPULSE_FREEZE=1)	needs approval

Executor (core/src/remediation/executor.ts)

awaiting_approval ─▶ approved ─▶ executing ─▶ verifying ─▶ succeeded
       │                             │             │
       ▼                             ▼             ▼
   rejected                 failed (rollback)   rolled_back / failed

Steps run sequentially and their outputs are recorded. Verification waits one bucket for the fix to show up in telemetry, then requires every service in the playbook's verify list (always including the user-facing gateway) to be healthy for 3 consecutive buckets within the window. Otherwise the playbook's rollback steps run (for example, reverting the degraded-mode flag). CODEPULSE_DRY_RUN=1 simulates every action.

Orchestrator (core/src/orchestrator/local.ts)

A local process supervisor implementing a Kubernetes-shaped interface: restart = delete the pod, deploy = patch the image, rollback = previous ReplicaSet, kill = PodChaos. It deliberately has no restart policy: a crashed service stays down until CodePulse decides to heal it, which keeps the self-healing honest.

---

Problems I hit while building it

These are real bugs I found by running the system against itself. Each one changed the design.

1. Every boot was an outage. The first run opened a SEV2 "gateway is down" incident in the first 6 seconds: the engine starts before the services, so all seven had "no heartbeat". Fix: availability only counts for services that have reported at least once. A never-seen service is not onboarded yet; it isn't down. (Lesson: absence of data has more than one meaning.)

2. The RCA changed its mind halfway through the fix. The connection-leak incident was correctly diagnosed and auto-healed, but the postmortem was titled "ledger-db latency degradation". While the fix was being verified, RCA kept running on post-restart data, where ledger-db briefly looked like the slowest service, and overwrote the title. Fix: once a remediation is executing or verified, the RCA is frozen. Re-diagnosing the tail of an incident is how auto-remediation ends up "fixing" noise.

3. Every service got credit for the same log line. The "new error signature" signal fired for gateway, orders, payments and ledger-db, because each one logs the wrapped error (orders failed: payments failed: ledger-db failed: pool exhausted…). Fix: if a service's new template ends with another service's error message, it's propagation. Only the innermost service is credited.

4. CodePulse could have blamed its own fix. I caught this in review, before it fired. Change correlation looked at every change "around" the incident, including the feature flag CodePulse itself flips as a remediation, and a change after onset scored as "0 s before". Fix: only changes before onset count, and changes by the codepulse actor never count.

5. A latency fault that verification could never pass. I spotted this while designing the slow-inventory experiment. If the fault slowed every inventory route, then even after the degraded-mode flag took reservations off the checkout path, GET /products would stay slow, the gateway's p95 would never recover, and verification would correctly roll back a fix that did work for checkout. Fix: faults can be scoped to a route (lock contention on writes), which matches how this failure happens in practice. The lesson: what you verify has to match what the fix is supposed to fix.

6. "Approved by undefined", and a restart that "didn't change the instance". Both were ordering bugs: an update event fired before approvedBy was set, and the "before" status object was a live reference that the restart mutated. Small, but they're the details that make an audit trail trustworthy or not.

7. The double rollback. This was the most interesting one, and the chaos benchmark found it. After CodePulse rolled orders back from the bad 2.5.0, the freshly started process had a few slow requests (JIT warm-up, new connections). The latency detectors fired, a new incident opened, and RCA blamed the 2.5.0 deploy again: it was still inside the 10-minute change window, though it had already been rolled back. CodePulse proposed rolling back a service that was already on the good version. Because that playbook is medium risk it waited for approval, and the signals recovered while it waited, so the incident could never close. Three fixes: (a) only the live deploy can be blamed, so a deploy superseded by a later deploy or rollback before onset is ignored; (b) a pending approval expires if the signals recover before anyone approves it; (c) latency-only anomalies that start within 3 buckets of CodePulse's own restart or rollback are treated as cold start, not a new incident. Errors and outages still open incidents normally. (Lesson: an auto-remediation system has to model the effects of its own actions.)

8. A deploy that happened "after" the incident it caused. One benchmark run scored 7/8: a bad deploy was classified as a generic error_spike and got a restart instead of a rollback. The restart was correctly rejected by verification, so the guardrails held, but the diagnosis was wrong. The orchestrator stamped the deploy event when the new process became healthy. Errors start during the stop→start gap, so occasionally the anomaly's onset bucket closed before the deploy's timestamp and the change didn't count as "before onset". Fix: change events are stamped when the rollout starts (as a CD system records them), with one extra bucket of tolerance. The next run scored 8/8.

9. Wrong evidence wording. Two bugs here. "Callers timed out" was attached to a bad deploy because the first trace sampled happened to be incomplete; the label now requires a majority. "Deployed −3 s before the anomaly" appeared because onset is a bucket boundary and the deploy landed inside the bucket.

---

Running it

Requirements: Node.js ≥ 22.18 (TypeScript runs natively via type stripping, so there's no build step), Python ≥ 3.9 (standard library only), and macOS or Linux.

git clone https://github.com/amartya1523/CodePulse.git
cd CodePulse
npm install
npm install --prefix dashboard
npm run dev

npm run dev starts the core (which boots all seven services), the load generator, and the dashboard. Open http://localhost:3000, go to Chaos Lab and inject a fault. If port 3000 is taken, use DASHBOARD_PORT=3100 npm run dev.

Other commands:

npm test                    # 22 unit tests: detector, sketch, RCA, policy, audit chain, OTLP, LLM validation
npm run typecheck           # tsc --noEmit over core, services, scripts
npm run chaos:bench         # run every experiment, write docs/chaos-results.md (ROUNDS=3 for more)
npm run core                # core + services only (API :7000, OTLP :4318)

Environment variable	Effect
ANTHROPIC_API_KEY	Enables the Claude investigator (falls back to deterministic narration without it).
CODEPULSE_MODEL	Model for the investigator (default claude-opus-5-5).
CODEPULSE_DRY_RUN=1	Remediation steps are simulated, never executed.
CODEPULSE_FREEZE=1	Change freeze: nothing runs without approval.
CODEPULSE_BUCKET_MS	Detection bucket size (default 3000).
CODEPULSE_MANAGE_SERVICES=0	Don't spawn services; just ingest OTLP from whatever sends it.
LOADGEN_RPS	Synthetic load (default 25 req/s).

API

Endpoint
POST :4318/v1/{traces,metrics,logs}	OTLP/HTTP JSON receiver
GET /api/overview	Services, health, series, topology edges, incidents, change feed
GET /api/incidents · /api/incidents/:id	Incidents with RCA, evidence, narrative, runs, audit
GET /api/incidents/:id/report	Markdown postmortem
POST /api/runs/:id/approve · /reject	Human-in-the-loop decisions
GET /api/traces/:traceId	Spans and correlated logs
GET /api/chaos · POST /api/chaos/:id/run	Experiments and runs
GET /api/audit	Audit entries and chain-integrity check
GET /api/stream	Server-sent events (incidents, runs, anomalies, changes)
GET /metrics	Prometheus exposition of CodePulse's own view

---

Project layout

packages/telemetry/     OTLP SDK (tracer, meter, logger, exporter) + HTTP service kit
services/               7 demo services: gateway, orders, payments, ledger-db, notifications (Node)
                        users, inventory (Python) + pykit (Python SDK twin)
core/src/
  ingest/               OTLP/HTTP JSON receiver
  store/                bucketed telemetry store, quantile sketch
  analysis/             anomaly detection, log template mining, root cause analysis
  investigator/         deterministic + Claude narrators, citation validation
  incidents/            incident model, postmortem rendering
  remediation/          playbooks, policy engine, executor, hash-chained audit log
  orchestrator/         local process orchestrator (k8s-shaped interface)
  chaos/                blind chaos experiments
  api/                  REST, SSE, Prometheus
  engine.ts             the control loop
core/test/              unit tests (node:test)
dashboard/              Next.js 16 command center
loadgen/                synthetic traffic
scripts/                dev launcher, chaos benchmark

---

Limitations and what's next

I'd rather list these than have a reviewer find them:

• Storage is in memory with 15-minute retention. Everything goes through TelemetryStore, so a ClickHouse backend (spans and logs in MergeTree tables, RED aggregates as materialized views) is the natural next step.
• Single node, one open incident at a time. Concurrent anomalies join the open incident. Correlating unrelated simultaneous incidents by graph connectivity is the next refinement.
• Confidence is a softmax score, not a calibrated probability. Calibrating it needs many more labelled incidents than four chaos experiments, but the benchmark harness is where that data would come from.
• The orchestrator is local processes. The interface is Kubernetes-shaped, but no k8s implementation exists yet.
• The demo services are synthetic. The failure modes (leaked connections, bad deploys, slow dependencies, crashes) are the classic real ones, but real systems have messier telemetry.
• The OTLP receiver speaks HTTP/JSON, not protobuf or gRPC.

---

Built by Amartya Vikram Singh