Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,20 @@ Coming soon:

We welcome contributions for additional extensions and tools.

## Evals and production telemetry

The `evals/postgres-aiguide/` directory contains a small human-review eval set
for schema design, migration review, and query-performance workflows. The cases
are harness-neutral so they can be used before releasing changes to the skills,
Claude Code plugin, MCP server, or another agent integration.

If you publish this plugin through Telvine, keep runtime telemetry metadata-only:
`skill.invocation.start`, `skill.invocation.end`, and `skill.invocation.error`
for skill behavior, plus `plugin.component.invoked` and
`plugin.component.error` for MCP or other non-skill components. Do not emit SQL
text from private systems, result rows, schema dumps, credentials, connector
payloads, tool arguments, or model outputs.

## 🛠 Development

See [DEVELOPMENT.md](DEVELOPMENT.md) for:
Expand Down
3 changes: 3 additions & 0 deletions evals/postgres-aiguide/cases.jsonl
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"id":"schema-design","input":"Design a Postgres schema for a multi-tenant SaaS audit log with retention, tenant isolation, and fast filtering by actor and event type.","expected_outcome":"Uses pg-aiguide skills or MCP context, includes tenant-scoped keys, constraints, indexes, retention/partitioning considerations, and avoids outdated serial/id patterns where modern identity columns fit."}
{"id":"migration-review","input":"Review a migration that adds a nullable status column, backfills it, and then makes it NOT NULL on a 200M-row table.","expected_outcome":"Flags lock/backfill risk, proposes safe phased migration steps, indexes only when justified, and distinguishes Postgres version assumptions."}
{"id":"query-performance","input":"Improve a slow query filtering events by tenant_id, created_at range, and metadata->>'source'.","expected_outcome":"Discusses query shape, appropriate composite/expression indexes, EXPLAIN validation, statistics, and avoids recommending unbounded indexes without workload assumptions."}
27 changes: 27 additions & 0 deletions evals/postgres-aiguide/rubric.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# pg-aiguide eval rubric

Score each case from 1-5.

## Postgres correctness

- 5: Uses current PostgreSQL patterns and names version-sensitive assumptions.
- 3: Mostly correct with minor version or syntax gaps.
- 1: Suggests outdated or invalid PostgreSQL behavior.

## Operational safety

- 5: Handles locks, backfills, destructive operations, and production rollout risk explicitly.
- 3: Mentions risk but leaves sequencing vague.
- 1: Recommends unsafe production changes.

## Performance reasoning

- 5: Connects schema, query shape, indexes, statistics, and EXPLAIN validation.
- 3: Gives plausible indexes without enough workload context.
- 1: Gives generic performance advice.

## Privacy and telemetry

- 5: Avoids emitting queries, data samples, credentials, connector payloads, tool arguments, or model outputs beyond the requested review.
- 3: Includes unnecessary operational detail without sensitive data.
- 1: Exposes private database content or secrets.