dlt loads data in, dbt transforms it, and drt activates it back out — reverse ETL from your warehouse to the tools your team works in, as YAML in your own repo. No dashboard to babysit, no audience builder duplicating what dbt already does, no hosted runtime billing you per row. Your data goes straight from your warehouse to the destination — never through a drt-hosted intermediary.
dlt load → dbt transform → drt activate
pip install drt-core # core (DuckDB included)
drt init && drt run
|
Same DX as dbt. If you know |
CI-native. Exit codes and |
|
LLM-native. A built-in MCP server and Claude Code skills let AI tools author and run your syncs. |
Free and open source. Every connector, the CLI, the MCP server, and the sync engine — Apache 2.0, self-hosted, no lock-in. |
What's always free? All connectors, CLI, MCP server, and sync engine. See OPEN_CORE.md for the open core boundary.
No cloud accounts needed — DuckDB + httpbin.org, three commands.
pip install drt-core
mkdir my-drt-project && cd my-drt-project
drt init --template duckdb_to_restThat scaffolds a runnable syncs/duckdb_to_rest.yml. Seed a tiny
DuckDB table and run:
python -c "
import duckdb
c = duckdb.connect('warehouse.duckdb')
c.execute('''CREATE TABLE IF NOT EXISTS users AS SELECT * FROM (VALUES
(1, 'Alice', 'alice@example.com'),
(2, 'Bob', 'bob@example.com'),
(3, 'Carol', 'carol@example.com')
) t(id, name, email)''')
c.close()
"
drt run --dry-run # preview, no data sent
drt run # POST each row to httpbin.org
drt status # check resultsdrt init --template list # see all available templates
drt init --template postgres_to_slack
drt init --template duckdb_to_hubspotEach template prints next-steps for the env vars / source data it needs. See examples/ for the full collection (Discord, Google Sheets, GitHub Actions, MySQL, ClickHouse, BigQuery, …) and docs/connectors/ for per-connector reference.
For a guided wizard that walks you through profile + project setup:
drt init # interactive — picks a source, configures profile, scaffolds projectBoth flows produce the same project shape (drt_project.yml, syncs/,
.drt/). drt sources --detailed and drt destinations --detailed
print every connector's required env vars and a sample YAML stanza —
useful when hand-authoring beyond the templates.
drt init # initialize project
drt list # list sync definitions
drt sources # list available source connectors
drt destinations # list available destination connectors
drt run # run all syncs
drt run --select <name> # run a specific sync (globs work: 'users_*')
drt run --select tag:<tag> # run syncs matching a tag (repeat --select to union)
drt run --select destination:<type> # run syncs by destination type
drt run --exclude <name> # subtract syncs (same grammar as --select)
drt run --failed # re-run only syncs that failed last time
drt run --limit 10 # sampled run: send only N rows (watermark frozen)
drt run --fail-fast # stop scheduling after the first failure
drt run --vars 'lookback_days: 1' # override project vars: for this run
drt run --threads 4 # parallel sync execution
drt run --dry-run # dry run
drt run --verbose # show row-level error details
drt run --output json # structured JSON output for CI/scripting
drt run --log-format json # structured JSON logging to stderr
drt run --profile prd # override profile (or DRT_PROFILE env var)
drt run --cursor-value '…' # override watermark cursor for backfill
drt test # run post-sync validation tests
drt test --select <name> # test a specific sync
drt test --store-failures # sample up to N failing rows per failed test (masked)
drt build # run each sync and its tests in one pass
drt validate # validate sync YAML configs
drt status # show recent sync status
drt status --output json # JSON output for status
drt profile list # list credential profiles in ~/.drt/profiles.yml
drt profile show <name> # show a profile (secrets masked)
drt profile test <name> # verify a profile's source connectivity
drt profile add <name> # interactively add a profile
drt profile remove <name> # remove a profile
drt serve # start HTTP webhook endpoint
drt docs generate --format mermaid # print project DAG as Mermaid
drt docs generate --format dbt-exposures # print dbt exposure YAML for ref() syncs
drt deploy github-actions # scaffold a scheduled sync workflow (drt-action + secrets wired)
drt mcp run # start MCP server (requires drt-core[mcp])
drt --install-completion # install shell completion (bash/zsh/fish)
drt --show-completion # show completion scriptGenerate a Mermaid DAG from your local drt_project.yml and syncs/*.yml files:
drt docs generate --format mermaid > dag.mdgraph LR
subgraph Sources
src_bigquery_prod["bigquery_prod<br/><i>bigquery</i>"]
end
subgraph Syncs
sync_users_to_hubspot{{"users_to_hubspot<br/><i>upsert</i>"}}
sync_accounts_to_hubspot{{"accounts_to_hubspot<br/><i>upsert</i>"}}
end
subgraph Destinations
dst_hubspot_contacts["hubspot (contacts)<br/><i>hubspot</i>"]
end
src_bigquery_prod -->|extract| sync_users_to_hubspot
src_bigquery_prod -->|extract| sync_accounts_to_hubspot
sync_users_to_hubspot -->|load| dst_hubspot_contacts
sync_accounts_to_hubspot -->|load| dst_hubspot_contacts
sync_users_to_hubspot -.lookup.-> sync_accounts_to_hubspot
Shell completion is supported for bash, zsh, and fish:
# Recommended: auto-install for your current shell (idempotent)
drt --install-completion
# Or manually add to your shell config (run once from the target shell)
drt --show-completion >> ~/.bashrc # bash
drt --show-completion >> ~/.zshrc # zsh
drt --show-completion > ~/.config/fish/completions/drt.fish # fishNote:
--show-completionoutputs the script for your current shell. Run it from the shell you want to configure. The manual>>append is not idempotent — run it once only.
After installation, restart your shell and tab-complete commands and options.
Connect drt to Claude, Cursor, or any MCP-compatible client so you can run syncs, check status, and validate configs without leaving your AI environment.
pip install drt-core[mcp]
drt mcp runClaude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json):
{
"mcpServers": {
"drt": {
"command": "drt",
"args": ["mcp", "run"]
}
}
}Available MCP tools:
| Tool | What it does |
|---|---|
drt_list_syncs |
List all sync definitions |
drt_run_sync |
Run a sync (supports dry_run + compute_diff for --diff parity) |
drt_run_test |
Run post-sync validation tests (mirrors drt test) |
drt_get_status |
Get last run result(s) |
drt_get_history |
Get recent sync run history |
drt_validate |
Validate sync YAML configs |
drt_get_schema |
Return JSON Schema for config files |
drt_list_connectors |
List available sources and destinations |
drt_dlq |
Inspect a sync's Dead Letter Queue (depth + failed records) |
drt_retry |
Replay a sync's Dead Letter Queue (mirrors drt retry) |
drt_get_manifest |
Machine-readable sync catalog + lineage (drt docs JSON) |
drt_list_profiles |
List credential profiles (name + type, no secrets) |
drt_test_profile |
Connectivity check for a credential profile |
drt_doctor |
Environment diagnostics (mirrors drt doctor) |
drt_state_show |
Stored watermark + last-run state |
drt_state_reset |
Reset watermark / run state / tracked-mirror keys |
Install the official Claude Code skills to generate YAML, debug failures, and migrate from other tools — all from the chat interface.
/plugin marketplace add drt-hub/drt
/plugin install drt@drt-hubTip: Enable auto-update so you always get the latest skills when drt is updated:
/plugin→ Marketplaces → drt-hub → Enable auto-update
Copy the files from .claude/commands/ into your drt project's .claude/commands/ directory.
| Skill | Trigger | What it does |
|---|---|---|
/drt-create-sync |
"create a sync" | Generates valid sync YAML from your intent |
/drt-debug |
"sync failed" | Diagnoses a specific error and suggests fixes |
/drt-troubleshoot |
"drt isn't working" | Walks a full top-to-bottom diagnostic checklist |
/drt-init |
"set up drt" | Guides through project initialization |
/drt-migrate |
"migrate from Census" | Converts existing configs to drt YAML |
Per-connector reference: docs/connectors/ · Discoverable from the CLI:
drt sources --detailed/drt destinations --detailed
| Connector | Status | Install | Auth |
|---|---|---|---|
| BigQuery | ✅ v0.1 | pip install drt-core[bigquery] |
Application Default / Service Account Keyfile |
| DuckDB | ✅ v0.1 | (core) | File path |
| PostgreSQL | ✅ v0.1 | pip install drt-core[postgres] |
Password (env var) |
| Snowflake | ✅ v0.5 | pip install drt-core[snowflake] |
Password (env var) |
| SQLite | ✅ v0.4.2 | (core) | File path |
| Redshift | ✅ v0.3.4 | pip install drt-core[redshift] |
Password (env var) |
| ClickHouse | ✅ v0.4.3 | pip install drt-core[clickhouse] |
Password (env var) |
| MySQL | ✅ v0.5 | pip install drt-core[mysql] |
Password (env var) |
| Databricks | ✅ v0.6 | pip install drt-core[databricks] |
Access Token (env var) |
| Delta Lake | ✅ v0.8 | pip install drt-core[deltalake] |
Storage options (env var) |
| Iceberg | ✅ v0.8 | pip install drt-core[iceberg] |
Catalog properties (env var) |
| SQL Server | ✅ v0.6 | pip install drt-core[sqlserver] |
Password (env var) |
| REST API | ✅ v0.7 | (core) | Bearer / API Key / Basic / OAuth2 |
35 destinations — warehouses, SaaS APIs, cloud storage, files & webhooks (click to expand)
| Connector | Status | Install | Auth |
|---|---|---|---|
| REST API | ✅ v0.1 | (core) | Bearer / API Key / Basic / OAuth2 |
| Slack Incoming Webhook | ✅ v0.1 | (core) | Webhook URL |
| Discord Webhook | ✅ v0.4.2 | (core) | Webhook URL |
| GitHub Actions | ✅ v0.1 | (core) | Token (env var) |
| HubSpot | ✅ v0.1 | (core) | Token (env var) |
| Zendesk | ✅ v0.7 | (core) | Basic (email + API token) |
| Amplitude | ✅ v0.7 | (core) | Project API key (env var) |
| Klaviyo | ✅ v0.8 | (core) | Private API key (env var) |
| Mixpanel | ✅ v0.8 | (core) | Project token / service account |
| Google Ads | ✅ v0.6 | (core) | OAuth2 Client Credentials |
| Meta Conversions | ✅ next | (core) | Long-lived access token (env var) |
| Google Sheets | ✅ v0.4 | pip install drt-core[sheets] |
Service Account Keyfile |
| PostgreSQL (upsert) | ✅ v0.4 | pip install drt-core[postgres] |
Password (env var) |
| MySQL (upsert) | ✅ v0.4 | pip install drt-core[mysql] |
Password (env var) |
| ClickHouse | ✅ v0.5 | pip install drt-core[clickhouse] |
Password (env var) |
| Parquet file | ✅ v0.5 | pip install drt-core[parquet] |
File path |
| Amazon S3 | ✅ v0.7.9 | pip install drt-core[s3] |
AWS credential chain / env vars |
| Google Cloud Storage | ✅ v0.7.9 | pip install drt-core[gcs] |
Application Default / Service Account Keyfile |
| Azure Blob Storage | ✅ v0.7.9 | pip install drt-core[azure] |
Connection string env / DefaultAzureCredential |
| Microsoft Teams Webhook | ✅ v0.5 | (core) | Webhook URL |
| CSV / JSON / JSONL file | ✅ v0.5 | (core) | File path |
| Jira | ✅ v0.5 | (core) | Basic (email + API token) |
| Linear | ✅ v0.5 | (core) | API Key (env var) |
| SendGrid | ✅ v0.5 | (core) | API Key (env var) |
| Notion | ✅ v0.6 | (core) | Bearer Token (env var) |
| Airtable | ✅ v0.8 | (core) | Personal Access Token (env var) |
| Twilio SMS | ✅ v0.6 | (core) | Basic (Account SID + Auth Token) |
| Intercom | ✅ v0.6 | (core) | Bearer Token (env var) |
| Email SMTP | ✅ v0.6 | (core) | Username / Password (env var) |
| Salesforce Bulk API 2.0 | ✅ v0.6 | (core) | OAuth2 (username-password) |
| Staged Upload | ✅ v0.6 | (core) | Configurable per provider |
| Elasticsearch / OpenSearch | ✅ v0.7.9 | (core) | API key / HTTP Basic (env var) |
| Snowflake | ✅ v0.7 | pip install drt-core[snowflake] |
Password (env var) |
| Databricks Delta Lake | ✅ v0.7.9 | pip install drt-core[databricks] |
Personal Access Token (env var) |
| BigQuery | ✅ v0.8 | pip install drt-core[bigquery] |
ADC / Service Account keyfile |
| Connector | Status | Install |
|---|---|---|
| Dagster | ✅ v0.4 | pip install dagster-drt |
| Prefect | ✅ v0.6 | (core) |
| Airflow | ✅ v0.6 | (core) |
| dbt manifest reader | ✅ v0.4 | (core) |
Upcoming releases → ROADMAP.md (scope, themes, targets) Issue-level tracking → GitHub Milestones Looking to contribute? → Good First Issues
Shipped: now on v1.0.0 — the first stable release. The Source/Destination/StateManager Protocols are frozen (a breaking signature change now requires a MAJOR version), backed by VERSIONING.md's deprecation policy and a migration guide for connector/plugin authors (docs/migration/v0.x-to-v1.0.md) — no config/CLI changes for anyone else, the same drop-in-upgrade story every release has told since v0.4.0. A bounded hardening pass closed six real issues alongside the freeze itself: OIDC auth for drt serve's Pub/Sub push, a Klaviyo backfill flag to keep historical replays from re-triggering live customer flows, a durable per-invocation run_results.json artifact, correct google_ads rate-limit quota scoping, and two DLQ/destination-batch correctness fixes — plus a Databricks warehouse-backed DLQ bug caught live while verifying it all. Grouped milestones below; the full per-release changelog lives in CHANGELOG.md and GitHub Releases.
| Milestone | Highlights |
|---|---|
| v0.1–0.3 | Core CLI · BigQuery / DuckDB / Postgres sources · REST / Slack / HubSpot destinations · incremental sync · MCP server + AI Skills for Claude Code |
| v0.4–0.6 | 20+ destinations (Sheets, SQL upserts, Notion, Twilio, Salesforce…) · dagster-drt / Airflow / Prefect · drt test · parallel --threads · JSON Schema validation |
| v0.7 ✅ | Production Ready — graceful shutdown · sync history · atomic table swaps · drt doctor · opt-in telemetry · sync.mode: mirror · DLQ + drt profile |
| v0.7.9–0.7.11 ✅ | Cloud destinations (S3 / GCS / Azure / Databricks / BigQuery / Elasticsearch) · Delta Lake + Iceberg sources · tracked & scoped mirror · PII masking · VS Code extension |
| v0.8.0 ✅ | drt docs lineage site (safe-by-default labels, static SVG DAG, byte-identical) · project vars: · drt build · --fail-fast / --limit / --failed · selection v2 · incremental REST API · watermark.lag · drt deploy github-actions |
| v0.8.1 ✅ | sync.match_policy update_only / create_only (Postgres + HubSpot) · drt docs generate --inline (self-contained HTML for object-store hosting) · manifest schema v2 (run history · column facts · DLQ depth) · tracked-mirror no-CREATE-grant · Retry-After |
| v0.8.2 ✅ | drt docs --inline → single navigable HTML object — the whole catalog in one file with in-page nav, so it renders and navigates on an authenticated GCS / S3 object URL (completes v0.8.1's --inline) |
| v0.8.3 ✅ | --dry-run --diff previews mirror DELETEs (tracked / destination / scoped, read-only, labelled apart from a replace rebuild) · keyed destination lookup instead of a full scan · alerts.on_degraded (row-error %, duration SLA, empty source, DLQ depth) · drt test custom SQL query tests, severity: warn, --store-failures |
| v0.8.4 ✅ | Mirror symmetry closes out — strategy: tracked + scope composed together, on every SQL destination (Postgres/MySQL/Snowflake/ClickHouse/Databricks) · tracked mirror's state diff moves from Python to a server-side SQL join · query tagging (SQL comment + native BigQuery/Snowflake/Databricks tags) · streaming extraction · source-side retry · rate limiting v2 · --full-refresh + drt state show/reset |
| v0.8.5 ✅ | Databricks composite-key mirror fix — sync.mode: mirror + tracked/scope on a multi-column upsert_key now goes through MERGE instead of the tuple-IN anti-join Delta rejects |
| v0.9.0 ✅ | Engine Foundation — ADR 0005 remote state (GCS/S3 backends for state/history/DLQ, CI-safe, team-shared) · computed_fields · metadata_columns + run_id/sync_run_id correlation · REST API body_mode: batch · state:modified selection · sync.unit_tests · secret provider URIs (AWS/GCP Secret Manager, Vault) · drt serve real delivery contract (coalescing, 202 + poll, pluggable auth) · CLI/MCP parity gate, 11 gaps closed |
| v0.10.0 ✅ | Enterprise Boundary & Ecosystem — Protocol stability/freeze prep · OSS product and Enterprise extension boundaries · age-encrypted project secrets · QueryableDestination · entry-point plugins usable from YAML · dbt exposures · benchmark/real-I/O/PyO3 evidence · pluggable rate-limit coordination · dagster-drt v0.4.0 event-driven sensors · FileDestination and least-privilege Snowflake mirror fixes |
| v1.0.0 ✅ | Stable Release — Source/Destination/StateManager Protocol freeze (semver-guaranteed) · migration guide for connector/plugin authors · Development Status :: 5 - Production/Stable · hardening pass: drt serve --auth oidc, klaviyo.backfill, target/drt/run_results.json run artifact, google_ads.cloud_project_id, DLQ/destination-batch correctness fixes |
Next: v1.x Rust Engine
Community-maintained Dagster integration. Expose drt syncs as Dagster assets with full observability.
pip install dagster-drtfrom dagster import AssetExecutionContext, Definitions
from dagster_drt import drt_assets, DagsterDrtResource
@drt_assets(project_dir="path/to/drt-project")
def my_syncs(context: AssetExecutionContext, drt: DagsterDrtResource):
yield from drt.run(context=context)
defs = Definitions(
assets=[my_syncs],
resources={"drt": DagsterDrtResource(project_dir="path/to/drt-project")},
)For event-driven activation, build_drt_change_sensor() watches metadata-only
change signals from Delta Lake, Iceberg, Snowflake, or SQL Server. The same
resource also runs explicitly selected syncs from a plain Dagster @op, returns
a chainable DrtEventIterator for source row-count checks, and powers the
declarative DrtSyncComponent for defs.yaml projects.
See dagster-drt README for full API docs
(sensors, Components, @op, Translator, Pipes support, and dry-run config).
Run drt syncs straight from CI/CD with the official drt-hub/drt-action — no infrastructure, just a few lines of YAML. Trigger on a schedule, on every push, or right after dbt finishes.
- uses: drt-hub/drt-action@v1
with:
select: '*'
extras: postgres
env:
PG_PASSWORD: ${{ secrets.PG_PASSWORD }}Inputs cover sync selection (select), connector extras, profile, dry-run and threads; outputs expose status, succeeded, failed and duration-seconds (plus a step-summary table). Secrets are passed via env: and resolved by drt's *_env keys. See the action README for the secrets pattern and more examples (run-after-dbt, PR preview).
drt is designed to work alongside, not against, the modern data stack:
drt collects no telemetry by default. Opting in helps us understand which sources / destinations / sync modes are actually used, so we can prioritise.
drt config set telemetry.enabled true # opt in
drt config show-telemetry # preview the exact payload that would be sent
drt config set telemetry.enabled false # opt out
DO_NOT_TRACK=1 drt run # universal kill switch — overrides everythingWhen opted in, drt sends one sync_completed event per sync. The only properties we collect are these 9 fields: drt_version, python_version, os, source_type, destination_type, sync_mode, rows_synced, duration_seconds, status. The wire envelope additionally carries event, distinct_id (a per-machine random UUID at ~/.drt/.anonymous_id), timestamp, and api_key. Sync names, model SQL, destination URLs, credentials, and project paths are never transmitted — the allow-list is enforced at the function-signature level in drt/telemetry.py. By default events go to PostHog Cloud (EU region); override with DRT_TELEMETRY_ENDPOINT and DRT_TELEMETRY_API_KEY for self-hosted PostHog or a custom collector.
Note: drt itself never transmits your IP, but the receiving PostHog backend records the TCP source IP as
$ip. See docs/telemetry.md for details and how to disable / substitute the backend.
For full details see docs/telemetry.md.
We welcome contributions of all sizes — from typo fixes to new connectors. drt has a transparent contributor ladder so your work builds toward greater trust and responsibility over time.
- Get started: CONTRIBUTING.md — setup, workflow, and your first connector tutorial
- Pick something to work on: Good First Issues
- Understand how decisions are made: GOVERNANCE.md
- What's free vs. enterprise: OPEN_CORE.md
- Versioning & breaking changes: VERSIONING.md
Every contribution — code, docs, bug reports, ideas — makes drt better. Thank you 💜
Full credits by contribution type (emoji key)
drt is an independent open-source project and is not affiliated with, endorsed by, or sponsored by dbt Labs, dlt-hub, or any other company.
"dbt" is a registered trademark of dbt Labs, Inc. "dlt" is a project maintained by dlt-hub.
drt is designed to complement these tools as part of the modern data stack, but is a separate project with its own codebase and maintainers.
Apache 2.0 — see LICENSE.