feat(actions): Pluggable DQ Actions & Alerting

AGENTS.md

@@ -275,6 +275,13 @@ model: Any  # type: ignore[assignment] — mlflow has no stubs
 ```
 `Any` in `anomaly/` is a known legacy exception. New code outside that module must not introduce it.
 
+**Avoid `from __future__ import annotations`**. Making every annotation a lazy
+string hides circular dependencies and heavy-import problems that should fail loudly at import time. Keep
+annotations eager. **Quoted forward references** (e.g. `config: "ScoringConfig"`) and `if TYPE_CHECKING:`
+imports should generally be avoided too — they defer the same problem one annotation at a time. When you
+hit a circular import, prefer to restructure (move the shared type to a module both sides import, or break
+the dependency); reach for a `TYPE_CHECKING` import plus a quoted reference only as a last resort, or for a self-reference.
+
 ### Docstrings
 
 Use [Google Style Python Docstrings](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) for public functions, classes, and modules. See [Writing Docstrings](https://databrickslabs.github.io/dqx/docs/dev/contributing/#writing-docstrings) for full guidance.

README.md

@@ -15,6 +15,23 @@ Simplified Data Quality checking at Scale for PySpark Workloads on streaming and
 [![PyPI](https://img.shields.io/pypi/v/databricks-labs-dqx?label=pypi%20package&cacheSeconds=3600)](https://pypi.org/project/databricks-labs-dqx/) 
 ![PyPI Downloads](https://static.pepy.tech/personalized-badge/databricks-labs-dqx?period=month&units=international_system&left_color=grey&right_color=orange&left_text=PyPI%20downloads&cacheSeconds=3600)
 
+# ✨ Key capabilities
+
+- **Rule-based quality checks** — 80+ built-in checks (null, range, regex, referential, aggregate, geo, PII, and more) for row-level and column/dataset-level validation, with support for custom check functions.
+- **Code or config checks** — define checks programmatically as code or declaratively as YAML/JSON configuration.
+- **Check levels** — mark failed checks as warning or error.
+- **Custom reactions to failed checks** — drop, mark, or quarantine invalid data flexibly.
+- **Detailed failure info** — get detailed insights into why each check failed.
+- **AI-assisted rule generation** — LLM-driven rule suggestions from business descriptions, powered by DSPy and Databricks Model Serving.
+- **Data profiling & rule generation** — automatic statistics collection and quality rule candidate generation from existing data.
+- **ML row anomaly detection** — Isolation Forest–based row anomaly detection with SHAP explanations and AI-generated narratives.
+- **Data contracts** — generate quality rules from ODCS data contracts, including schema validation.
+- **Summary metrics & quality dashboard** — built-in and custom aggregate metrics (input/error/warning/valid row counts, per-check breakdowns) persisted to Delta tables, with a Lakeview quality dashboard for tracking and identifying data quality issues.
+- **Actions and alerting** — automatically send Slack, Microsoft Teams, or generic webhook alerts and/or fail the pipeline when summary metrics cross a threshold.
+- **Flexible checks storage** — save and load quality rules from YAML/JSON files, Unity Catalog tables, Volumes, or Lakebase (PostgreSQL).
+- **DQX Studio** — browser-based no-code UI for authoring, reviewing, running, and monitoring quality rules, deployed as a Databricks App.
+- **Data format agnostic & streaming support** — works with PySpark DataFrames and applies checks to both batch and Spark Structured Streaming (including Lakeflow Pipelines / DLT) using the same API.
+
 # 📖 Documentation
 
 The complete documentation is available at: [https://databrickslabs.github.io/dqx/](https://databrickslabs.github.io/dqx/)

demos/dqx_demo_alerting.py

@@ -0,0 +1,146 @@
+# Databricks notebook source
+# MAGIC %md
+# MAGIC # Alerting with DQX Actions
+# MAGIC
+# MAGIC This short demo shows how to react automatically to data quality problems using DQX **actions**. After checks run, DQX evaluates each action's optional *condition* against the observed summary metrics and, when it matches, fires the action — for example sending an alert.
+# MAGIC
+# MAGIC We use a `LogDQAlertDestination`, which writes the alert to the driver log and contacts no external system, so the demo runs end-to-end anywhere. We also show how to add a Slack destination: supply a Slack incoming-webhook URL in the `slack_webhook_url` widget and the demo will additionally deliver to Slack.
+# MAGIC
+# MAGIC See the [Actions and Alerting guide](https://databrickslabs.github.io/dqx/docs/guide/actions_and_alerts) for the full reference.
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Install DQX
+
+# COMMAND ----------
+
+dbutils.widgets.text("test_library_ref", "", "Test Library Ref")
+
+if dbutils.widgets.get("test_library_ref") != "":
+    %pip install '{dbutils.widgets.get("test_library_ref")}'
+else:
+    %pip install databricks-labs-dqx
+
+%restart_python
+
+# COMMAND ----------
+
+# Optional: provide a Slack incoming-webhook URL to also deliver alerts to Slack.
+# Leave empty to log alerts only (no external calls).
+dbutils.widgets.text("slack_webhook_url", "", "Slack Webhook URL")
+slack_webhook_url = dbutils.widgets.get("slack_webhook_url")
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Create a small dataset with some invalid rows
+# MAGIC
+# MAGIC Two rows have a null `name` or `age`, so an error-level check will flag them and drive our alert.
+
+# COMMAND ----------
+
+df = spark.createDataFrame(
+    [
+        (1, "Alice", 30),
+        (2, None, 25),  # null name -> error
+        (3, "Bob", None),  # null age -> error
+        (4, "Carol", 40),
+    ],
+    "id int, name string, age int",
+)
+display(df)
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Define the quality checks
+# MAGIC
+# MAGIC We require `name` and `age` to be non-null at the error level.
+
+# COMMAND ----------
+
+import yaml
+
+checks = yaml.safe_load(
+    """
+- criticality: error
+  check:
+    function: is_not_null
+    for_each_column:
+      - name
+      - age
+"""
+)
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Configure the alerting action
+# MAGIC
+# MAGIC Actions require a `DQMetricsObserver` because they evaluate the metrics it collects. Our action fires whenever at least one error row is observed (`error_row_count > 0`) and delivers the alert to the driver log — plus Slack if a webhook URL was provided.
+
+# COMMAND ----------
+
+from databricks.sdk import WorkspaceClient
+from databricks.labs.dqx.engine import DQEngine
+from databricks.labs.dqx.metrics_observer import DQMetricsObserver
+from databricks.labs.dqx.actions import (
+    DQAction,
+    DQAlert,
+    LogDQAlertDestination,
+    SlackDQAlertDestination,
+)
+
+# Always log alerts; add Slack only when a webhook URL is supplied.
+destinations = [LogDQAlertDestination(name="driver-log", level="warning")]
+if slack_webhook_url:
+    destinations.append(SlackDQAlertDestination(name="slack", webhook_url=slack_webhook_url))
+
+actions = [
+    DQAction(
+        condition="error_row_count > 0",
+        action=DQAlert(destinations=destinations),
+    )
+]
+
+observer = DQMetricsObserver(name="alerting_demo")
+engine = DQEngine(WorkspaceClient(), observer=observer, actions=actions)
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Apply the checks and evaluate the actions
+# MAGIC
+# MAGIC After applying checks we trigger a Spark action (`count()`) to materialize the observed metrics, then call `evaluate_actions`. The returned results describe which actions fired; the alert itself is written to the driver log (look for the `[DQX alert]` line above the cell output).
+
+# COMMAND ----------
+
+checked_df, observation = engine.apply_checks_by_metadata(df, checks)
+
+# Materialize the observation so metrics are populated.
+checked_df.count()
+
+results = engine.evaluate_actions(observation.get, input_location="in-memory demo dataframe")
+
+for result in results:
+    print(f"action={result.action_name} fired={result.fired} status={result.status.value}")
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Wiring a real Slack alert
+# MAGIC
+# MAGIC To deliver to Slack, create a [Slack incoming webhook](https://api.slack.com/messaging/webhooks) and pass its URL to the `slack_webhook_url` widget above (delivery is restricted to `hooks.slack.com`). In production, store the URL in a Databricks secret scope and reference it with `DQSecret` rather than passing a plaintext string:
+# MAGIC
+# MAGIC ```python
+# MAGIC from databricks.labs.dqx.config import DQSecret
+# MAGIC from databricks.labs.dqx.actions import SlackDQAlertDestination
+# MAGIC
+# MAGIC SlackDQAlertDestination(
+# MAGIC     name="slack",
+# MAGIC     webhook_url=DQSecret(scope="dq-secrets", key="slack-webhook-url"),
+# MAGIC )
+# MAGIC ```
+# MAGIC
+# MAGIC You can also attach actions directly to `apply_checks_and_save_in_table` so they fire automatically after each save (batch or streaming) — see the guide for details.

docs/dqx/docs/demos.mdx

@@ -12,6 +12,7 @@ Import the following notebooks in the Databricks workspace to try DQX out:
 * [DQX Intermediate Demo Notebook](https://github.com/databrickslabs/dqx/blob/v0.15.0/demos/dqx_intermediate_demo_library.py) - intermediate-level DQX usage as a library.
 * [DQX Comprehensive Demo Notebook](https://github.com/databrickslabs/dqx/blob/v0.15.0/demos/dqx_demo_library.py) - comprehensive DQX usage as a library.
 * [DQX Demo Notebook for Data Quality Summary Metrics](https://github.com/databrickslabs/dqx/blob/v0.15.0/demos/dqx_demo_summary_metrics.py) - demonstrates how to generate summary-level data quality metrics when validating data with DQX.
+* [DQX Demo Notebook for Actions and Alerting](https://github.com/databrickslabs/dqx/blob/v0.15.0/demos/dqx_demo_alerting.py) - demonstrates how to react to data quality problems by firing alerts (driver log, with optional Slack) when summary metrics cross a threshold.
 * [DQX Demo Notebook for Profiling and Applying Checks at Scale on Multiple Tables](https://github.com/databrickslabs/dqx/blob/v0.15.0/demos/dqx_multi_table_demo.py) - demonstrates how to use DQX as a library at scale to apply checks on multiple tables.
 * [DQX Demo Notebook for Row Anomaly Detection](https://github.com/databrickslabs/dqx/blob/v0.15.0/demos/dqx_row_anomaly_detection_demo.py) - comprehensive demo showing how to use DQX Row Anomaly Detection to detect unusual patterns in your data.
 * [DQX Demo Notebook for AI-assisted checks generation](https://github.com/databrickslabs/dqx/blob/v0.15.0/demos/dqx_demo_ai_assisted_checks_generation.py) - demonstrates how to generate DQX rules/checks with LLM using natural language.