-
Notifications
You must be signed in to change notification settings - Fork 52
docs: add ee compaction cronjob introduction #2660
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
98 changes: 98 additions & 0 deletions
98
docs/enterprise/deployments-administration/scheduled-compaction.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| --- | ||
| keywords: [scheduled compaction, compaction cronjob, GreptimeDB Enterprise, Metasrv, plugin] | ||
| description: Configure scheduled compaction in GreptimeDB Enterprise and manage compaction jobs through the Metasrv HTTP API. | ||
| --- | ||
|
|
||
| # Scheduled Compaction | ||
|
|
||
| Scheduled Compaction is a GreptimeDB Enterprise feature that periodically submits regular compaction requests for all physical Regions that have a leader in a cluster. It runs on the leader Metasrv, uses a cron expression and an IANA time zone to determine the schedule, and stores recent job reports in the Metasrv KV backend. | ||
|
|
||
| This feature complements GreptimeDB's automatic and [manual compaction](/user-guide/deployments-administration/manage-data/compaction.md#strict-window-compaction-strategy-swcs-and-manual-compaction). Use it when you want to compact all Regions regularly during a predictable maintenance window. | ||
|
|
||
| ## Configure the plugin | ||
|
|
||
| Add the `compaction_cronjob` plugin to the Metasrv configuration: | ||
|
|
||
| ```toml | ||
| [[plugins]] | ||
| [plugins.compaction_cronjob] | ||
| enable = true | ||
| timezone = "Asia/Shanghai" | ||
| cron = "0 0 0 * * *" | ||
| max_concurrent_regions = 4 | ||
| compact_parallelism = 1 | ||
| manual_trigger_cooldown_secs = 3600 | ||
| max_history_jobs = 100 | ||
| ``` | ||
|
|
||
| The cron expression includes seconds. For example, `0 0 0 * * *` runs at midnight every day in the configured time zone. | ||
|
|
||
| | Option | Default | Description | | ||
| | --- | --- | --- | | ||
| | `enable` | `false` | Enables scheduled compaction. | | ||
| | `timezone` | Local IANA time zone, or `UTC` if it cannot be detected | The time zone used to evaluate `cron`, for example `Asia/Shanghai`. | | ||
| | `cron` | `0 0 0 * * *` | The compaction schedule. | | ||
| | `max_concurrent_regions` | `4` | The maximum number of Region compaction requests submitted concurrently by one job. Must be greater than `0`. | | ||
| | `compact_parallelism` | `1` | The compaction parallelism passed to each Region request. Must be greater than `0`. | | ||
| | `manual_trigger_cooldown_secs` | `3600` | The minimum interval between a manual trigger and the most recent job, whether that job was scheduled or manually triggered. Set it to `0` to disable the cooldown. Scheduled jobs are not delayed by this option. | | ||
| | `max_history_jobs` | `100` | The maximum number of completed job reports retained in the Metasrv KV backend. Must be greater than `0`. | | ||
|
|
||
| Only the leader Metasrv runs the scheduler. After leadership changes, the new leader continues scheduling future jobs from the configured cron expression. | ||
|
|
||
| ## HTTP endpoints | ||
|
|
||
| The plugin adds the following endpoints to the Metasrv HTTP server. Replace `<metasrv-http-address>` with the configured Metasrv HTTP address. | ||
|
|
||
| ### Trigger a job | ||
|
|
||
| ```bash | ||
| curl -X POST \ | ||
| "http://<metasrv-http-address>/admin/compaction_cronjob/trigger" | ||
| ``` | ||
|
|
||
| Send this request to the leader Metasrv. The request waits for the job to finish submitting Region compaction requests and returns its report. | ||
|
|
||
| - Returns `409 Conflict` when sent to a follower Metasrv. | ||
| - Returns `429 Too Many Requests` when the manual trigger is still within `manual_trigger_cooldown_secs` of the most recent scheduled or manual job. | ||
| - Returns `500 Internal Server Error` for other failures. | ||
|
|
||
| ### Get a job | ||
|
|
||
| ```bash | ||
| curl --get \ | ||
| --data-urlencode "job_id=2026-06-16T00:00:00+00:00" \ | ||
| "http://<metasrv-http-address>/admin/compaction_cronjob/job" | ||
| ``` | ||
|
|
||
| The `job_id` is returned by the trigger and job-list endpoints. It is an RFC 3339 timestamp and should be URL encoded. The endpoint returns `404 Not Found` when the job does not exist. | ||
|
|
||
| ### List recent jobs | ||
|
|
||
| ```bash | ||
| curl --get \ | ||
| --data-urlencode "limit=20" \ | ||
| "http://<metasrv-http-address>/admin/compaction_cronjob/jobs" | ||
| ``` | ||
|
|
||
| Jobs are ordered by scheduled time in descending order. The optional `limit` parameter defaults to `20`. | ||
|
|
||
| ## Job report | ||
|
|
||
| The trigger and query endpoints return JSON job reports. A completed successful job resembles: | ||
|
|
||
| ```json | ||
| { | ||
| "job_id": "2026-06-16T00:00:00+00:00", | ||
| "scheduled_at": "2026-06-16T00:00:00Z", | ||
| "started_at": "2026-06-16T00:00:00.010Z", | ||
| "finished_at": "2026-06-16T00:00:01.010Z", | ||
| "status": "succeeded", | ||
| "execution_result": { | ||
| "total_regions": 8, | ||
| "submitted_regions": 8, | ||
| "failed_regions": [] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| Possible job states are `running`, `succeeded`, and `partial_failed`. A failed job carries its reason as `{"failed": "<reason>"}`. The execution result records how many Region requests were attempted and submitted, together with Region-level submission failures. | ||
|
MichaelScofield marked this conversation as resolved.
Outdated
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
98 changes: 98 additions & 0 deletions
98
...tent-docs/current/enterprise/deployments-administration/scheduled-compaction.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| --- | ||
| keywords: [定时 Compaction, Compaction Cronjob, GreptimeDB 企业版, Metasrv, plugin] | ||
| description: 介绍如何在 GreptimeDB 企业版中配置定时 Compaction,并通过 Metasrv HTTP API 管理 Compaction 任务。 | ||
| --- | ||
|
|
||
| # 定时 Compaction | ||
|
|
||
| 定时 Compaction 是 GreptimeDB 企业版功能,用于周期性地为集群中所有存在 leader 的物理 Region 提交 regular Compaction 请求。该功能运行在 leader Metasrv 上,根据 cron 表达式和 IANA 时区执行任务,并将最近的任务报告保存在 Metasrv KV backend 中。 | ||
|
|
||
| 该功能是 GreptimeDB 自动 Compaction 和[手动 Compaction](/user-guide/deployments-administration/manage-data/compaction.md)的补充。当你希望在固定的维护时间窗口内定期 Compaction 所有 Region 时,可以使用该功能。 | ||
|
|
||
| ## 配置 plugin | ||
|
|
||
| 在 Metasrv 配置文件中添加 `compaction_cronjob` plugin: | ||
|
|
||
| ```toml | ||
| [[plugins]] | ||
| [plugins.compaction_cronjob] | ||
| enable = true | ||
| timezone = "Asia/Shanghai" | ||
| cron = "0 0 0 * * *" | ||
| max_concurrent_regions = 4 | ||
| compact_parallelism = 1 | ||
| manual_trigger_cooldown_secs = 3600 | ||
| max_history_jobs = 100 | ||
| ``` | ||
|
|
||
| cron 表达式包含秒字段。例如,`0 0 0 * * *` 表示每天在所配置时区的午夜执行。 | ||
|
|
||
| | 配置项 | 默认值 | 说明 | | ||
| | --- | --- | --- | | ||
| | `enable` | `false` | 是否启用定时 Compaction。 | | ||
| | `timezone` | 本地 IANA 时区;无法检测时使用 `UTC` | 计算 `cron` 时使用的时区,例如 `Asia/Shanghai`。 | | ||
| | `cron` | `0 0 0 * * *` | Compaction 执行计划。 | | ||
| | `max_concurrent_regions` | `4` | 单个任务并发提交 Region Compaction 请求的最大数量,必须大于 `0`。 | | ||
| | `compact_parallelism` | `1` | 传递给每个 Region Compaction 请求的并行度,必须大于 `0`。 | | ||
| | `manual_trigger_cooldown_secs` | `3600` | 人工触发与最近一次任务之间的最小间隔。最近的任务可以是定时触发或人工触发。设置为 `0` 可禁用冷却时间;该配置不会延迟定时任务。 | | ||
| | `max_history_jobs` | `100` | Metasrv KV backend 中最多保留的已完成任务报告数量,必须大于 `0`。 | | ||
|
|
||
| 只有 leader Metasrv 会运行 scheduler。leader 发生切换后,新的 leader 会继续根据所配置的 cron 表达式调度后续任务。 | ||
|
|
||
| ## HTTP endpoints | ||
|
|
||
| 该 plugin 会在 Metasrv HTTP server 上增加以下 endpoints。请将 `<metasrv-http-address>` 替换为 Metasrv 配置的 HTTP 地址。 | ||
|
|
||
| ### 触发任务 | ||
|
|
||
| ```bash | ||
| curl -X POST \ | ||
| "http://<metasrv-http-address>/admin/compaction_cronjob/trigger" | ||
| ``` | ||
|
|
||
| 该请求必须发送到 leader Metasrv。请求会等待任务完成 Region Compaction 请求的提交,并返回任务报告。 | ||
|
|
||
| - 请求发送到 follower Metasrv 时返回 `409 Conflict`。 | ||
| - 人工触发距离最近一次定时或人工任务不足 `manual_trigger_cooldown_secs` 时返回 `429 Too Many Requests`。 | ||
| - 其他错误返回 `500 Internal Server Error`。 | ||
|
|
||
| ### 查询指定任务 | ||
|
|
||
| ```bash | ||
| curl --get \ | ||
| --data-urlencode "job_id=2026-06-16T00:00:00+00:00" \ | ||
| "http://<metasrv-http-address>/admin/compaction_cronjob/job" | ||
| ``` | ||
|
|
||
| `job_id` 由触发任务和任务列表 endpoints 返回。它是 RFC 3339 时间戳,在请求中应进行 URL 编码。任务不存在时,该 endpoint 返回 `404 Not Found`。 | ||
|
|
||
| ### 查询最近任务 | ||
|
|
||
| ```bash | ||
| curl --get \ | ||
| --data-urlencode "limit=20" \ | ||
| "http://<metasrv-http-address>/admin/compaction_cronjob/jobs" | ||
| ``` | ||
|
|
||
| 任务按计划执行时间倒序排列。可选参数 `limit` 的默认值为 `20`。 | ||
|
|
||
| ## 任务报告 | ||
|
|
||
| 触发和查询 endpoints 返回 JSON 格式的任务报告。以下是一个执行成功的任务报告: | ||
|
|
||
| ```json | ||
| { | ||
| "job_id": "2026-06-16T00:00:00+00:00", | ||
| "scheduled_at": "2026-06-16T00:00:00Z", | ||
| "started_at": "2026-06-16T00:00:00.010Z", | ||
| "finished_at": "2026-06-16T00:00:01.010Z", | ||
| "status": "succeeded", | ||
| "execution_result": { | ||
| "total_regions": 8, | ||
| "submitted_regions": 8, | ||
| "failed_regions": [] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| 任务状态包括 `running`、`succeeded` 和 `partial_failed`。失败任务的状态包含失败原因,格式为 `{"failed": "<reason>"}`。`execution_result` 记录尝试提交和成功提交的 Region 请求数量,以及各 Region 的提交失败信息。 | ||
|
MichaelScofield marked this conversation as resolved.
Outdated
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.