diff --git a/docs/enterprise/deployments-administration/scheduled-compaction.md b/docs/enterprise/deployments-administration/scheduled-compaction.md new file mode 100644 index 0000000000..3b322b1960 --- /dev/null +++ b/docs/enterprise/deployments-administration/scheduled-compaction.md @@ -0,0 +1,100 @@ +--- +keywords: [scheduled compaction, compaction cronjob, GreptimeDB Enterprise, Metasrv, plugin] +description: Configure scheduled compaction in GreptimeDB Enterprise and manage compaction jobs through the Metasrv HTTP API. +--- + +# Scheduled Compaction + +Scheduled Compaction is a GreptimeDB Enterprise feature that periodically submits regular compaction requests for all physical Regions that have a leader in a cluster. It runs on the leader Metasrv, uses a cron expression and an IANA time zone to determine the schedule, and stores recent job reports in the Metasrv KV backend. + +This feature complements GreptimeDB's automatic and [manual compaction](/user-guide/deployments-administration/manage-data/compaction.md#strict-window-compaction-strategy-swcs-and-manual-compaction). Use it when you want to compact all Regions regularly during a predictable maintenance window. + +## Configure the plugin + +Add the `compaction_cronjob` plugin to the Metasrv configuration: + +```toml +[[plugins]] +[plugins.compaction_cronjob] +enable = true +timezone = "Asia/Shanghai" +cron = "0 0 0 * * *" +max_concurrent_regions = 4 +compact_parallelism = 1 +manual_trigger_cooldown_secs = 3600 +max_history_jobs = 100 +``` + +The cron expression includes seconds. For example, `0 0 0 * * *` runs at midnight every day in the configured time zone. + +| Option | Default | Description | +| --- | --- | --- | +| `enable` | `false` | Enables scheduled compaction. | +| `timezone` | Local IANA time zone, or `UTC` if it cannot be detected | The time zone used to evaluate `cron`, for example `Asia/Shanghai`. | +| `cron` | `0 0 0 * * *` | The compaction schedule. | +| `max_concurrent_regions` | `4` | The maximum number of Region compaction requests submitted concurrently by one job. Must be greater than `0`. | +| `compact_parallelism` | `1` | The compaction parallelism passed to each Region request. Must be greater than `0`. | +| `manual_trigger_cooldown_secs` | `3600` | The minimum interval between a manual trigger and the most recent job, whether that job was scheduled or manually triggered. Set it to `0` to disable the cooldown. Scheduled jobs are not delayed by this option. | +| `max_history_jobs` | `100` | The maximum number of terminal job reports retained after history pruning runs when a job completes. Must be greater than `0`. | + +Only the leader Metasrv runs the scheduler. After leadership changes, the new leader continues scheduling future jobs from the configured cron expression. + +Before starting the scheduler, a new leader marks jobs left in `running` state as `cancelled` and writes a warning log. This status change does not cancel Region requests that were already submitted by the previous leader. Cancelled jobs are terminal and participate in history pruning the next time a job completes. + +## HTTP endpoints + +The plugin adds the following endpoints to the Metasrv HTTP server. Replace `` with the configured Metasrv HTTP address. + +### Trigger a job + +```bash +curl -X POST \ + "http:///admin/compaction_cronjob/trigger" +``` + +Send this request to the leader Metasrv. The request waits for the job to finish submitting Region compaction requests and returns its report. + +- Returns `409 Conflict` when sent to a follower Metasrv. +- Returns `429 Too Many Requests` when the manual trigger is still within `manual_trigger_cooldown_secs` of the most recent scheduled or manual job. +- Returns `500 Internal Server Error` for other failures. + +### Get a job + +```bash +curl --get \ + --data-urlencode "job_id=2026-06-16T00:00:00+00:00" \ + "http:///admin/compaction_cronjob/job" +``` + +The `job_id` is returned by the trigger and job-list endpoints. It is an RFC 3339 timestamp and should be URL encoded. The endpoint returns `404 Not Found` when the job does not exist. + +### List recent jobs + +```bash +curl --get \ + --data-urlencode "limit=20" \ + "http:///admin/compaction_cronjob/jobs" +``` + +Jobs are ordered by scheduled time in descending order. The optional `limit` parameter defaults to `20`. + +## Job report + +The trigger and query endpoints return JSON job reports. A completed successful job resembles: + +```json +{ + "job_id": "2026-06-16T00:00:00+00:00", + "scheduled_at": "2026-06-16T00:00:00Z", + "started_at": "2026-06-16T00:00:00.010Z", + "finished_at": "2026-06-16T00:00:01.010Z", + "status": "succeeded", + "execution_result": { + "total_regions": 8, + "submitted_regions": 8, + "failed_regions": [] + } +} +``` + +The `status` field is `"running"`, `"succeeded"`, `"partial_failed"`, or `"cancelled"`. A failed job uses `{"failed": ""}` as the `status` value. The execution result records how many Region requests were attempted and submitted, together with Region-level submission failures. diff --git a/docs/enterprise/overview.md b/docs/enterprise/overview.md index 507a77b978..2674d1878b 100644 --- a/docs/enterprise/overview.md +++ b/docs/enterprise/overview.md @@ -32,6 +32,7 @@ which are described in detail in the documentation in this section: user activity with detailed audit logs. - [Automatic region load balance](./autopilot/region-balancer.md): Auto balance datanodes workload by moving regions between them. +- [Scheduled Compaction](./deployments-administration/scheduled-compaction.md): Periodically submit regular compaction requests for all physical Regions, with HTTP endpoints for manual triggers and job queries. - [Elasticsearch query compatibility](./elasticsearch-compatible/overview.md): Use GreptimeDB backed Kibana for your logs. - [Greptime Enterprise Management Console](./console-ui.md): An enhanced version of our dashboard UI, carries more cluster management and monitoring features. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/enterprise/deployments-administration/scheduled-compaction.md b/i18n/zh/docusaurus-plugin-content-docs/current/enterprise/deployments-administration/scheduled-compaction.md new file mode 100644 index 0000000000..118cae3c7f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/enterprise/deployments-administration/scheduled-compaction.md @@ -0,0 +1,100 @@ +--- +keywords: [定时 Compaction, Compaction Cronjob, GreptimeDB 企业版, Metasrv, plugin] +description: 介绍如何在 GreptimeDB 企业版中配置定时 Compaction，并通过 Metasrv HTTP API 管理 Compaction 任务。 +--- + +# 定时 Compaction + +定时 Compaction 是 GreptimeDB 企业版功能，用于周期性地为集群中所有存在 leader 的物理 Region 提交 regular Compaction 请求。该功能运行在 leader Metasrv 上，根据 cron 表达式和 IANA 时区执行任务，并将最近的任务报告保存在 Metasrv KV backend 中。 + +该功能是 GreptimeDB 自动 Compaction 和[手动 Compaction](/user-guide/deployments-administration/manage-data/compaction.md)的补充。当你希望在固定的维护时间窗口内定期 Compaction 所有 Region 时，可以使用该功能。 + +## 配置 plugin + +在 Metasrv 配置文件中添加 `compaction_cronjob` plugin： + +```toml +[[plugins]] +[plugins.compaction_cronjob] +enable = true +timezone = "Asia/Shanghai" +cron = "0 0 0 * * *" +max_concurrent_regions = 4 +compact_parallelism = 1 +manual_trigger_cooldown_secs = 3600 +max_history_jobs = 100 +``` + +cron 表达式包含秒字段。例如，`0 0 0 * * *` 表示每天在所配置时区的午夜执行。 + +| 配置项 | 默认值 | 说明 | +| --- | --- | --- | +| `enable` | `false` | 是否启用定时 Compaction。 | +| `timezone` | 本地 IANA 时区；无法检测时使用 `UTC` | 计算 `cron` 时使用的时区，例如 `Asia/Shanghai`。 | +| `cron` | `0 0 0 * * *` | Compaction 执行计划。 | +| `max_concurrent_regions` | `4` | 单个任务并发提交 Region Compaction 请求的最大数量，必须大于 `0`。 | +| `compact_parallelism` | `1` | 传递给每个 Region Compaction 请求的并行度，必须大于 `0`。 | +| `manual_trigger_cooldown_secs` | `3600` | 人工触发与最近一次任务之间的最小间隔。最近的任务可以是定时触发或人工触发。设置为 `0` 可禁用冷却时间；该配置不会延迟定时任务。 | +| `max_history_jobs` | `100` | 后续任务完成并执行历史清理后，Metasrv KV backend 中最多保留的终态任务报告数量，必须大于 `0`。 | + +只有 leader Metasrv 会运行 scheduler。leader 发生切换后，新的 leader 会继续根据所配置的 cron 表达式调度后续任务。 + +新的 leader 会在启动 scheduler 前，将遗留在 `running` 状态的任务标记为 `cancelled`，并输出 warning 日志。该状态变更不会取消前一个 leader 已经提交的 Region 请求。`cancelled` 是终态，会在后续任务完成时按照 `max_history_jobs` 参与历史清理。 + +## HTTP endpoints + +该 plugin 会在 Metasrv HTTP server 上增加以下 endpoints。请将 `` 替换为 Metasrv 配置的 HTTP 地址。 + +### 触发任务 + +```bash +curl -X POST \ + "http:///admin/compaction_cronjob/trigger" +``` + +该请求必须发送到 leader Metasrv。请求会等待任务完成 Region Compaction 请求的提交，并返回任务报告。 + +- 请求发送到 follower Metasrv 时返回 `409 Conflict`。 +- 人工触发距离最近一次定时或人工任务不足 `manual_trigger_cooldown_secs` 时返回 `429 Too Many Requests`。 +- 其他错误返回 `500 Internal Server Error`。 + +### 查询指定任务 + +```bash +curl --get \ + --data-urlencode "job_id=2026-06-16T00:00:00+00:00" \ + "http:///admin/compaction_cronjob/job" +``` + +`job_id` 由触发任务和任务列表 endpoints 返回。它是 RFC 3339 时间戳，在请求中应进行 URL 编码。任务不存在时，该 endpoint 返回 `404 Not Found`。 + +### 查询最近任务 + +```bash +curl --get \ + --data-urlencode "limit=20" \ + "http:///admin/compaction_cronjob/jobs" +``` + +任务按计划执行时间倒序排列。可选参数 `limit` 的默认值为 `20`。 + +## 任务报告 + +触发和查询 endpoints 返回 JSON 格式的任务报告。以下是一个执行成功的任务报告： + +```json +{ + "job_id": "2026-06-16T00:00:00+00:00", + "scheduled_at": "2026-06-16T00:00:00Z", + "started_at": "2026-06-16T00:00:00.010Z", + "finished_at": "2026-06-16T00:00:01.010Z", + "status": "succeeded", + "execution_result": { + "total_regions": 8, + "submitted_regions": 8, + "failed_regions": [] + } +} +``` + +`status` 字段的取值可以是 `"running"`、`"succeeded"`、`"partial_failed"` 或 `"cancelled"`。失败任务的 `status` 值为 `{"failed": ""}`。`execution_result` 记录尝试提交和成功提交的 Region 请求数量，以及各 Region 的提交失败信息。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/enterprise/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/enterprise/overview.md index fe12fd0399..3283164fd3 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/enterprise/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/enterprise/overview.md @@ -29,6 +29,7 @@ GreptimeDB Enterprise 包括以下高级功能， - [部署 GreptimeDB](./deployments-administration/overview.md)：设置认证信息及其他关键配置后，将 GreptimeDB 部署在 Kubernetes 上并监控关键指标。 - [审计日志](./deployments-administration/monitoring/audit-logging.md)：记录数据库用户行为的日志。 - [自动分区平衡](./autopilot/region-balancer.md)：通过分区监控和迁移在 datanode 之间自动平衡负载。 +- [定时 Compaction](./deployments-administration/scheduled-compaction.md)：定期为所有物理 Region 提交 regular Compaction 请求，并提供用于人工触发和任务查询的 HTTP endpoints。 - [Elasticsearch 查询兼容性](./elasticsearch-compatible/overview.md)：在 Kibana 中以 GreptimeDB 作为后端。 - [Greptime 企业版管理控制台](./console-ui.md)：加强版本的管理界面，提供更多的集群管理和监控功能。 - [读副本](./read-replicas/overview.md)：专门运行复杂的查询操作的 datanode，避免影响实时写入。 diff --git a/sidebars.ts b/sidebars.ts index d5c6e2c25f..dd9d91f13c 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -481,6 +481,7 @@ const sidebars: SidebarsConfig = { ] }, 'enterprise/deployments-administration/backup', + 'enterprise/deployments-administration/scheduled-compaction', ], }, {