-
Notifications
You must be signed in to change notification settings - Fork 155
feat(databases-on-aws): add Drizzle ORM guidance to DSQL skill #266
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -66,7 +66,7 @@ Concrete example (from `dsql_lint(sql="CREATE INDEX idx ON t (c);", fix=true)`): | |
|
|
||
| ## Workflow: Validate & Migrate SQL to DSQL | ||
|
|
||
| Use for any SQL that was not composed by the agent itself from skill knowledge — including user-pasted SQL, migration files, ORM output (Django, Rails, Prisma, TypeORM, Sequelize, SQLAlchemy), pg_dump exports, and hand-written schemas. Applies to DDL and schema-mutating DML; do **not** lint ad-hoc read-only `SELECT`s. | ||
| Use for any SQL that was not composed by the agent itself from skill knowledge — including user-pasted SQL, migration files, ORM output (Django, Rails, Prisma, Drizzle, TypeORM, Sequelize, SQLAlchemy), pg_dump exports, and hand-written schemas. Applies to DDL and schema-mutating DML; do **not** lint ad-hoc read-only `SELECT`s. | ||
|
|
||
| 1. Obtain source SQL from user (migration file, ORM output, schema dump, or inline SQL). `dsql_lint` accepts multi-statement SQL in a single call — pass the whole batch. | ||
| 2. Run `dsql_lint(sql=source_sql, fix=true)`. Default to `fix=true` for any migration scenario; use `fix=false` only when the user explicitly asked for validation-only output, or when re-verifying manually rewritten SQL. | ||
|
|
@@ -94,6 +94,13 @@ Use for any SQL that was not composed by the agent itself from skill knowledge | |
| - **Django:** Run `python manage.py sqlmigrate <app> <migration>` to get raw SQL, then lint. | ||
| - **Rails (6.1+):** Set `config.active_record.schema_format = :sql`, then run `rails db:schema:dump` (legacy `db:structure:dump` still works in older Rails). Lint the generated `db/structure.sql`. | ||
| - **Prisma:** Use `prisma migrate diff --from-empty --to-schema-datamodel ./prisma/schema.prisma --script` to emit SQL to stdout, then lint. | ||
| - **Drizzle:** Run `npx aurora-dsql-drizzle generate --out ./drizzle -- --config drizzle.config.ts` — the adapter runs `drizzle-kit generate` and pipes each statement through `dsql-lint`, preserving `--> statement-breakpoint` markers (`transform` and `lint` run those steps alone). It lints one statement at a time, so cross-statement fixes do not apply — see [orm-guides/overview.md](orm-guides/overview.md). | ||
| `aurora-dsql-drizzle` exit codes differ by mode — `generate` and `transform` run `dsql-lint --fix`, `lint` only reports: | ||
| - `generate`, `transform`: `0` clean — apply the migration; `1` some diagnostics need a hand-written rewrite — see Handling Unfixable Errors below; `3` fixed with advisories — review the flagged statements, then apply. One advisory statement sets `3` for the whole run, e.g. `NOT VALID` added to a foreign key. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This should preserve the workflow’s existing acknowledgement requirement for Suggested replacement:
|
||
| - `lint`: `0` clean; `1` any diagnostic was reported, including advisories `transform` fixes on its own — run `transform` before hand-writing a rewrite. `lint` never returns `3` and never writes SQL. | ||
| - `1` also covers failures that say nothing about the SQL — bad arguments, a missing input file, a `drizzle-kit` error, or a missing `dsql-lint` binary. Read the `Error:` line on stderr and confirm diagnostics were printed before treating `1` as a verdict on the SQL. | ||
|
|
||
| When an advisory adds `NOT VALID` to a foreign key, follow it with `ALTER TABLE ASYNC ... VALIDATE CONSTRAINT` in its own chunk — precede the statement with a `--> statement-breakpoint` marker — then verify the job. | ||
| - **TypeORM/Sequelize:** Generate migration SQL to a file, then lint. | ||
| - **SQLAlchemy:** Compile DDL without executing — e.g., `for table in metadata.tables.values(): print(CreateTable(table).compile(engine))`. Do **not** call `metadata.create_all(engine)` with a real engine — it executes the DDL before lint. Alternatively use `create_mock_engine` to capture DDL. | ||
|
|
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,20 +11,23 @@ Across adapters, inline foreign keys in `CREATE TABLE` use DSQL foreign-key synt | |
| Post-creation foreign keys **MUST** use `ADD CONSTRAINT ... NOT VALID`, followed by | ||
| `ALTER TABLE ASYNC ... VALIDATE CONSTRAINT` and terminal job verification. | ||
|
|
||
| For existing tables, emit that sequence through the framework's raw-SQL migration hook: | ||
| `RunSQL` (Django), `migrationBuilder.Sql` (EF Core), Flyway/Liquibase (Hibernate), `execute` | ||
| (Rails), or `op.execute` (Alembic/SQLAlchemy). For tenant-scoped composite FKs, use raw DDL in | ||
| Django and Rails; EF Core, Hibernate, and SQLAlchemy provide composite relationship mappings. | ||
| For existing tables, emit that sequence through the framework's raw-SQL migration mechanism: | ||
| `RunSQL` (Django), the generated `.sql` migration file (Drizzle), `migrationBuilder.Sql` (EF Core), | ||
| Flyway/Liquibase (Hibernate), `execute` (Rails), or `op.execute` (Alembic/SQLAlchemy). For | ||
| tenant-scoped composite FKs, use raw DDL in Django and Rails; Drizzle | ||
| (`foreignKey({ columns, foreignColumns })`), EF Core, Hibernate, and SQLAlchemy provide composite | ||
| relationship mappings. | ||
|
|
||
| ## Adapters | ||
|
|
||
| | Framework | Adapter | Install | | ||
| | ---------- | --------------------------------------- | ------------------------------------------------------------ | | ||
| | Django | `aurora_dsql_django` | `pip install aurora-dsql-django boto3` | | ||
| | EF Core | `Amazon.AuroraDsql.EntityFrameworkCore` | `dotnet add package Amazon.AuroraDsql.EntityFrameworkCore` | | ||
| | Hibernate | `aurora-dsql-hibernate-dialect` | `software.amazon.dsql:aurora-dsql-hibernate-dialect` (Maven) | | ||
| | Rails | Standard `pg` gem + `aws-sdk-dsql` | `gem 'pg'` + `gem 'aws-sdk-dsql'` | | ||
| | SQLAlchemy | `aurora_dsql_sqlalchemy` | `pip install aurora-dsql-sqlalchemy boto3` | | ||
| | Framework | Adapter | Install | | ||
| | ---------- | --------------------------------------- | ------------------------------------------------------------------------------------ | | ||
| | Django | `aurora_dsql_django` | `pip install aurora-dsql-django boto3` | | ||
| | Drizzle | `@aws/aurora-dsql-drizzle` | `npm install @aws/aurora-dsql-drizzle drizzle-orm pg` + `npm install -D drizzle-kit` | | ||
| | EF Core | `Amazon.AuroraDsql.EntityFrameworkCore` | `dotnet add package Amazon.AuroraDsql.EntityFrameworkCore` | | ||
| | Hibernate | `aurora-dsql-hibernate-dialect` | `software.amazon.dsql:aurora-dsql-hibernate-dialect` (Maven) | | ||
| | Rails | Standard `pg` gem + `aws-sdk-dsql` | `gem 'pg'` + `gem 'aws-sdk-dsql'` | | ||
| | SQLAlchemy | `aurora_dsql_sqlalchemy` | `pip install aurora-dsql-sqlalchemy boto3` | | ||
|
|
||
| ## Key Gotchas Per Framework | ||
|
|
||
|
|
@@ -39,6 +42,26 @@ Django and Rails; EF Core, Hibernate, and SQLAlchemy provide composite relations | |
| | AutoField | Replace with `UUIDField(primary_key=True, default=uuid.uuid4)` | | ||
| | ForeignKey | Keep `ForeignKey`; the DSQL backend creates database constraints for new tables | | ||
|
|
||
| ### Drizzle (TypeScript) | ||
|
|
||
| Requires `drizzle-orm` `^0.45` (0.45.x — the peer range pins the minor), `pg` 8+, Node.js 20+, and | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The foreign-key behavior below requires Suggested language:
|
||
| `drizzle-kit` as a dev dependency. The adapter rides `drizzle-orm/node-postgres` over the DSQL | ||
|
praba2210 marked this conversation as resolved.
|
||
| node-postgres connector — there is no custom dialect. | ||
|
|
||
| | Issue | Fix | | ||
| | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | Setup | `drizzle({ connection: { host, user, region }, schema })` from `@aws/aurora-dsql-drizzle` — IAM auth and TLS come from the connector, and `region` is optional (inferred from the host). Pass an existing pool with `drizzle({ client: pool, schema })`; `db.$client` exposes it | | ||
| | `user` | REQUIRED — there is no default, so a connection never lands on `admin` by omission. Pass a database role scoped to what the app needs | | ||
| | OCC retry | `db.transactionWithRetry(cb)` re-runs the transaction on SQLSTATE `40001`. The callback MUST be idempotent — it re-runs on every attempt. Nested `tx.transaction()` fails fast. Defaults: `maxRetries` 3, `baseDelayMs` 50, `maxDelayMs` 5000 | | ||
| | Retry exhaustion | Throws `AwsDsqlRetryExhaustedError` with the last conflict on `.cause`. Pass overrides in the third argument, after the transaction config: `transactionWithRetry(cb, undefined, { maxRetries: 5, onRetry: (err, attempt, max) => ... })` | | ||
| | Migrations | Use `migrate()` from `@aws/aurora-dsql-drizzle`, NOT the stock `drizzle-orm/node-postgres` migrator — the stock one sends every statement in a single transaction. `getMigrationStatus(db, config)` reports applied vs. pending | | ||
| | Generate | `npx aurora-dsql-drizzle generate --out ./drizzle -- --config drizzle.config.ts` runs `drizzle-kit generate`, then rewrites each statement with `dsql-lint`. Review and commit the result — see [dsql-lint.md](../dsql-lint.md) | | ||
| | `SERIAL` | The transform rewrites it as `BIGINT ... GENERATED BY DEFAULT AS IDENTITY (CACHE 1)`. Review the DDL for the wider type, and let the identity column supply values — replace any `nextval()`/`setval()`/`currval()` calls that named the old `SERIAL` sequence | | ||
| | Breakpoints | Keep Drizzle Kit's `breakpoints: true` (the default). The adapter applies one statement per `--> statement-breakpoint` marker, and `migrate()` stops with `success: false` on any chunk holding more than one statement. Give every hand-added statement its own marker — `transform` neither inserts one nor flags the chunk, so the failure surfaces at apply time | | ||
| | Identity columns | Define them in the table definition. The transform lints statements one at a time, so `ALTER COLUMN ... ADD GENERATED ... AS IDENTITY` is reported unfixable — merge it into the `CREATE TABLE` by hand if it is already generated | | ||
| | Foreign keys | The transform adds `NOT VALID` to foreign keys created with `ALTER TABLE`. Add `ALTER TABLE ASYNC ... VALIDATE CONSTRAINT` after it, preceded by its own `--> statement-breakpoint` marker; `migrate()` waits for that job | | ||
| | Resume | A statement and its tracking row are separate commits. If a run dies between them the statement is applied but untracked, and re-running fails "already exists" (`drizzle-kit` omits `IF NOT EXISTS`) — `migrate()` names the statement | | ||
|
|
||
| ### EF Core (.NET) | ||
|
|
||
| Requires .NET 8.0+, EF Core 9.0.7+, and `Amazon.AuroraDsql.Npgsql` 1.1.0+. | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.