[feature](data governance) Add data lineage documentation

docs/key-features/data-lineage.mdx

@@ -20,7 +20,7 @@ featureCard:
     - metadata
 ---
 
-> **TL;DR** Apache Doris extracts column-level lineage from the analyzed Nereids plan of every `INSERT`, `INSERT OVERWRITE`, and `CREATE TABLE AS SELECT`, including JOIN, FILTER, GROUP BY, WINDOW, and CASE WHEN dependencies. Events flow through a SPI to whichever lineage backend you ship: a custom emitter, an OpenLineage receiver, Apache Atlas, or DataHub. For table-level lineage from ad-hoc queries, the `audit_log` system table records the source tables and chosen materialized views per query.
+> **TL;DR** Apache Doris extracts column-level lineage from the analyzed Nereids plan of every supported `INSERT INTO ... SELECT`, `INSERT OVERWRITE TABLE ... SELECT`, and `CREATE TABLE AS SELECT`, including JOIN, FILTER, GROUP BY, WINDOW, and CASE WHEN dependencies. Events flow through a SPI to whichever lineage backend you ship: a custom emitter, an OpenLineage receiver, Apache Atlas, or DataHub. For table-level lineage from ad-hoc queries, the `audit_log` system table records the source tables and chosen materialized views per query.
 
 ![Apache Doris Data Lineage: Column-level lineage extracted from the Nereids plan and shipped to your governance tool through a pluggable SPI, plus table-level traces in the audit log.](/images/next/key-features/data-lineage.jpg)
 ## Why use Apache Doris data lineage? {#why}
@@ -42,53 +42,115 @@ Apache Doris data lineage is column-level lineage extraction built into the Nere
 **Key terms**
 
 - **`LineageInfo`**: the in-memory event format. Per output column, it holds the direct lineage type (`IDENTITY`, `TRANSFORMATION`, `AGGREGATION`) and the source expression, plus dataset-level indirect lineage (`JOIN`, `FILTER`, `GROUP_BY`, `SORT`) and per-column indirect lineage (`WINDOW`, `CONDITIONAL`).
-- **`LineagePlugin`**: SPI you implement to receive `LineageInfo` events. Discovered via `ServiceLoader` on the classpath or as an external jar in `$plugin_dir/lineage/`.
-- **`activate_lineage_plugin`**: FE config naming the plugins that should receive events. Empty by default, so the planner does no lineage work until you enable a plugin.
+- **`LineagePlugin`**: SPI you implement to receive `LineageInfo` events. Discovered via `ServiceLoader` on the classpath or from an external plugin directory under `$plugin_dir/lineage/`.
+- **`activate_lineage_plugin`**: FE config naming the discovered plugin factories to instantiate. An empty value disables name filtering and instantiates every discovered factory.
 - **`audit_log.queried_tables_and_views`**: an `ARRAY<STRING>` column in `__internal_schema.audit_log` that lists the catalog-qualified tables touched by each query.
 - **`audit_log.chosen_m_views`**: companion column listing the materialized views the optimizer rewrote into.
 
 ## How does Apache Doris data lineage work? {#how}
 
-Apache Doris hooks the Nereids planner before optimization, walks the analyzed plan to resolve each output column back to its sources, records the JOIN/FILTER/GROUP BY/WINDOW/CASE WHEN dependencies, and ships the event off the query thread to every registered `LineagePlugin`.
+Apache Doris hooks the Nereids planner before optimization, walks the analyzed plan to resolve each output column back to its sources, records the JOIN/FILTER/GROUP BY/WINDOW/CASE WHEN dependencies, and queues the event for the loaded `LineagePlugin` instances.
 
-1. **Hook the planner.** Each write command (`INSERT INTO`, `INSERT OVERWRITE`, `CTAS`) registers an analyze-plan hook. The hook grabs the analyzed `LogicalPlan` before optimization, so column references still carry their source slots and lineage matches the SQL the user wrote.
+1. **Hook the planner.** Each supported write command (`INSERT INTO ... SELECT`, `INSERT OVERWRITE TABLE ... SELECT`, CTAS) registers an analyze-plan hook. The hook grabs the analyzed `LogicalPlan` before optimization, so column references still carry their source slots and lineage matches the SQL the user wrote.
 2. **Extract direct lineage.** `LineageInfoExtractor` walks the plan and resolves every output column back to its source expressions. A pure column reference becomes `IDENTITY`. A scalar expression becomes `TRANSFORMATION`. An aggregate function becomes `AGGREGATION`. CTE consumer slots get pre-resolved to producer slots so the chain does not stop at a `WITH` clause boundary.
 3. **Extract indirect lineage.** JOIN keys, filter predicates, and GROUP BY columns are recorded as dataset-level indirect lineage, since they affect every output. `WindowExpression` partition keys and `CaseWhen`/`If`/`Coalesce` branches are recorded per output column, so you can trace why a single value took the path it took.
-4. **Submit asynchronously.** A single worker thread drains a bounded queue (`lineage_event_queue_size`, default 50000) and dispatches each event to every active plugin in order. Plugin work happens off the query thread, so a slow downstream system slows the queue, not the query the user is waiting on.
+4. **Submit asynchronously.** A single worker thread drains a bounded queue (`lineage_event_queue_size`, default 50000), evaluates each loaded plugin in order, and calls `exec()` only when its `eventFilter()` returns `true`. Plugin work happens off the query thread, so a slow downstream system slows the queue, not the query the user is waiting on.
 5. **Skip what does not matter.** `VALUES`-only inserts and writes targeting `__internal_schema` are filtered out before extraction. The optimizer does not pay for events nobody wants.
 
 For ad-hoc `SELECT` traffic that never lands in a write, the picture is simpler. The audit log already records the source tables and any matched materialized views per query, and you derive table-level lineage from there.
 
 ## Quick start {#quick-start}
 
+Run the example as an administrator who can change global variables and create databases and tables. It creates a dedicated database with one replica, runs a Join query, and then reads the recorded table-level lineage from the audit log.
+
 ```sql
 -- 1. Enable the audit plugin so query-level lineage is recorded.
 SET GLOBAL enable_audit_plugin = true;
 
--- 2. Run a query that joins two base tables.
-SELECT n.n_name, SUM(o.o_totalprice)
-FROM orders o JOIN nation n ON o.o_custkey = n.n_nationkey
-GROUP BY n.n_name;
+-- 2. Create the source tables.
+DROP DATABASE IF EXISTS lineage_audit_demo;
+CREATE DATABASE lineage_audit_demo;
+USE lineage_audit_demo;
+
+CREATE TABLE lineage_orders (
+    order_id BIGINT,
+    region_id BIGINT,
+    amount DECIMAL(18, 2)
+)
+DUPLICATE KEY(order_id)
+DISTRIBUTED BY HASH(order_id) BUCKETS 1
+PROPERTIES ("replication_num" = "1");
+
+CREATE TABLE lineage_regions (
+    region_id BIGINT,
+    region_name VARCHAR(32)
+)
+DUPLICATE KEY(region_id)
+DISTRIBUTED BY HASH(region_id) BUCKETS 1
+PROPERTIES ("replication_num" = "1");
+
+INSERT INTO lineage_orders VALUES
+    (1, 10, 100.00),
+    (2, 10, 50.00),
+    (3, 20, 80.00);
+
+INSERT INTO lineage_regions VALUES
+    (10, 'east'),
+    (20, 'west');
+
+-- 3. Run a query that joins the two source tables.
+SELECT r.region_name, SUM(o.amount) AS total_amount
+FROM lineage_orders o
+JOIN lineage_regions r ON o.region_id = r.region_id
+GROUP BY r.region_name
+ORDER BY r.region_name;
+```
 
--- 3. Read the lineage out of the audit log.
-SELECT time, queried_tables_and_views, chosen_m_views, stmt_id
-FROM   internal.__internal_schema.audit_log
-WHERE  user = CURRENT_USER() AND is_query = 1
-ORDER  BY time DESC LIMIT 5;
+The Join query returns:
+
+```text
++-------------+--------------+
+| region_name | total_amount |
++-------------+--------------+
+| east        |       150.00 |
+| west        |        80.00 |
++-------------+--------------+
 ```
 
-**Expected result**
+Audit records are written in batches and become visible asynchronously. After the Join query finishes, wait for the current `audit_plugin_max_batch_interval_sec` and then run the following query. The default maximum interval is 60 seconds.
 
+```sql
+SELECT time, queried_tables_and_views, chosen_m_views, stmt_id
+FROM internal.__internal_schema.audit_log
+WHERE user = TRIM(BOTH "'" FROM SUBSTRING_INDEX(CURRENT_USER(), '@', 1))
+  AND is_query = 1
+  AND array_contains(
+          queried_tables_and_views,
+          'internal.lineage_audit_demo.lineage_orders')
+  AND array_contains(
+          queried_tables_and_views,
+          'internal.lineage_audit_demo.lineage_regions')
+ORDER BY time DESC
+LIMIT 1;
 ```
-+---------------------+--------------------------------------------+----------------+
-| time                | queried_tables_and_views                   | chosen_m_views |
-+---------------------+--------------------------------------------+----------------+
-| 2026-05-09 10:14:22 | ["internal.tpch.orders","internal.tpch.    | []             |
-|                     |  nation"]                                  |                |
-+---------------------+--------------------------------------------+----------------+
+
+The audit query returns a row similar to the following. The timestamp and statement ID depend on the execution:
+
+```text
++-------------------------+-----------------------------------------------------------------------------------------------+----------------+---------+
+| time                    | queried_tables_and_views                                                                      | chosen_m_views | stmt_id |
++-------------------------+-----------------------------------------------------------------------------------------------+----------------+---------+
+| <event time>            | ["internal.lineage_audit_demo.lineage_orders", "internal.lineage_audit_demo.lineage_regions"] | []             | <ID>    |
++-------------------------+-----------------------------------------------------------------------------------------------+----------------+---------+
 ```
 
-The row holds catalog-qualified table names and any MV the optimizer rewrote into. That is enough to draw a table-level graph for ad-hoc queries. For column-level lineage on writes, implement `LineagePlugin`, drop the jar in `plugin_dir/lineage/`, and add the plugin name to `activate_lineage_plugin`. From then on, every `INSERT` and `CTAS` produces a `LineageInfo` event.
+The row holds catalog-qualified table names and any MV the optimizer rewrote into. That is enough to draw a table-level graph for ad-hoc queries. For column-level lineage on writes, implement `LineagePlugin`, place its plugin directory under `$plugin_dir/lineage/`, and add the plugin name to `activate_lineage_plugin`. From then on, each supported `INSERT` or CTAS produces a `LineageInfo` event.
+
+After verification, remove the example database:
+
+```sql
+DROP DATABASE lineage_audit_demo;
+```
 
 ## When should you use Apache Doris data lineage? {#when}
 
@@ -105,13 +167,14 @@ Apache Doris data lineage fits impact analysis, compliance trails, MV governance
 **Not a good fit**
 
 - A built-in lineage browser. Doris emits events. The visualization is your governance tool's job. If you want a graph today and have no governance stack, deploy DataHub or Marquez and point a plugin at it.
-- Lineage for pure `SELECT` queries that never land in a write. The Nereids extractor only fires on `INSERT`, `INSERT OVERWRITE`, and `CTAS`. For read-only traffic, use the `queried_tables_and_views` array in the audit log to get table-level coverage.
+- Lineage for pure `SELECT` queries that never land in a write. The Nereids extractor only fires on `INSERT INTO ... SELECT`, `INSERT OVERWRITE TABLE ... SELECT`, and CTAS. For read-only traffic, use the `queried_tables_and_views` array in the audit log to get table-level coverage.
 - Lineage across systems that never touch Doris. The plugin sees Doris-side queries. A Spark job that writes Parquet which Doris later reads needs Spark-side instrumentation (OpenLineage's Spark integration, for example), and the two streams meet in your catalog.
 - Source-of-truth metadata storage. Lineage events are emitted, not retained inside Doris. Persist them in your catalog of choice.
 - Operations on `__internal_schema` targets and `VALUES`-only inserts. These are filtered out by design and will not produce lineage events.
 
 ## Further reading {#further-reading}
 
+- [Data Lineage Technical Guide](../data-governance/data-lineage): supported statements, event semantics, FE configuration, examples, and troubleshooting.
 - [audit_log system table](../admin-manual/system-tables/internal_schema/audit_log): full column reference, including the `queried_tables_and_views` and `chosen_m_views` arrays you query for table-level lineage.
 - [Audit Log plugin](../admin-manual/audit-plugin): how to enable `enable_audit_plugin`, configure batch interval and SQL length, and exclude internal users.
 - [Async Materialized Views](../query-acceleration/materialized-view/async-materialized-view/overview): the rewrite engine whose decisions show up in `chosen_m_views` and whose base-table dependencies are tracked by `MTMVRelationManager`.

i18n/zh-CN/docusaurus-plugin-content-docs-community/current.json

@@ -43,6 +43,10 @@
     "message": "数据源扩展开发",
     "description": "The label for category Data Source Extension in sidebar community"
   },
+  "sidebar.community.category.Plugin Development": {
+    "message": "插件开发",
+    "description": "The label for category Plugin Development in sidebar community"
+  },
   "sidebar.community.category.Data Format Reference": {
     "message": "数据格式参考",
     "description": "The label for category Data Format Reference in sidebar community"