add within group

[daviszhen]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #25144
#25110

What this PR does / why we need it:

• 新增 SQL WITHIN GROUP (ORDER BY ...) 语法及 AST、Binder、Explain 支持。

• GROUP_CONCAT 支持两种排序写法：

  • GROUP_CONCAT(v ORDER BY k)
  • GROUP_CONCAT(v) WITHIN GROUP (ORDER BY k)

• 新增 PERCENTILE_CONT 和 PERCENTILE_DISC 标量聚合，支持分组、ASC/DESC、NULL、数值类型及分布式合并。

• 百分位排序改为复用查询 ORDER BY 的 mosort.SortByVectors 实现。

• 增加聚合 spill、配置编码、资源释放和 DISTINCT group-concat spill 安全保护。

• 补充 parser、plan、executor、边界测试及 BVT 结果文件。

• 当前仍是 MVP：不支持百分位窗口函数、参数化 p、非数值排序列和 decimal256。

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[iamlinjunhong]

Requesting changes for four P1 blockers on head 6fc7078b7c01df729b9a08d24a25220de92cda2f:

1. The new aggregate IDs can be sent to previous-version CNs during a rolling upgrade without a capability gate.
2. Exact percentile state is unbounded for H0 and a single hot group, so normal large inputs can OOM despite SpillMem.
3. Making WITHIN reserved breaks SQL and schemas that previously used it as an unquoted identifier.
4. PERCENTILE_CONT declares an invalid/insufficient decimal result type for valid maximum-precision inputs and can overflow even at p=0.

I reviewed the full diff from merge-base a4b0ce286d182c24efd5a349620ae37016262301, including parser → binder → plan → compile → remote aggregate → merge/spill/flush/free. All 26 GitHub checks are complete with no failing conclusion. The added parser tests pass locally; a base/head parser probe also confirms the WITHIN compatibility regression. No P2/P3-only items are being used to block this review.

[aunjgr]

Reviewed exact head 6fc7078 after the successful CI rollup.

[P1] pkg/sql/colexec/aggexec/ordered_percentile.go:49: PERCENTILE_CONT always declares decimal128 scale input scale plus one. A valid DECIMAL(38,38) input therefore becomes the invalid DECIMAL(38,39), and flush multiplies the 38-scale unscaled value by 10; values such as 0.9 overflow even for percentile 0, which should simply return the input. Cap or otherwise choose a representable result precision/scale and test the maximum-scale boundary.

[P2] pkg/sql/parsers/dialect/mysql/mysql_sql.y:12524: GROUP_CONCAT accepts both its in-parentheses ORDER BY and WITHIN GROUP ORDER BY, then silently discards the first list in favor of the second. Reject the ambiguous double-order form instead of changing its meaning silently, and add parser coverage.

[daviszhen]

Requesting changes for four P1 blockers on head 6fc7078b7c01df729b9a08d24a25220de92cda2f:

1. The new aggregate IDs can be sent to previous-version CNs during a rolling upgrade without a capability gate.
2. Exact percentile state is unbounded for H0 and a single hot group, so normal large inputs can OOM despite SpillMem.
3. Making WITHIN reserved breaks SQL and schemas that previously used it as an unquoted identifier.
4. PERCENTILE_CONT declares an invalid/insufficient decimal result type for valid maximum-precision inputs and can overflow even at p=0.

I reviewed the full diff from merge-base a4b0ce286d182c24efd5a349620ae37016262301, including parser → binder → plan → compile → remote aggregate → merge/spill/flush/free. All 26 GitHub checks are complete with no failing conclusion. The added parser tests pass locally; a base/head parser probe also confirms the WITHIN compatibility regression. No P2/P3-only items are being used to block this review.

1. 远程执行能力门控

• 新增 ordered-set percentile 聚合 ID 的 MORPC 协议门控。
• merge main 后，保留 main 的 MORPCVersion11/12/13，将 percentile 能力顺延为 MORPCVersion14。
• remote aggregate / merge group 在协议版本不足时拒绝远程执行，避免新 CN 把 PERCENTILE_CONT/DISC 发给旧 CN 后报 unexpected aggID。

2. percentile 聚合内存无限增长

• 为 ordered percentile 增加聚合器内部 spill 机制。
• 覆盖 H0 聚合和单热点 group 场景，避免只依赖 generic group spill。
• 补充了 H0、大单组、generic spill 组合场景的回归测试。

3. WITHIN 关键字兼容性

• 将 WITHIN 从保留关键字改为上下文关键字/非保留用法。
• 保证已有 SQL 如 select within from t、create table t (within int) 不被破坏。
• 补充 parser 兼容性测试。

4. decimal 返回类型/溢出

• 修正 PERCENTILE_CONT 对 DECIMAL 输入的返回类型推导，避免生成非法 DECIMAL(38,39)。
• 处理最大 precision/scale 边界，避免 p=0 时仍因内部放大计算溢出。
• 补充 DECIMAL(38,38) 和 38 位整数边界测试。

5. GROUP_CONCAT 双 ORDER BY

• 对 group_concat(x order by y) within group (order by z) 这类双排序写法改为拒绝。
• 避免静默丢弃其中一个 ORDER BY 导致语义被悄悄改写。
• 补充 parser 覆盖。

[daviszhen]

Reviewed exact head 6fc7078 after the successful CI rollup.

[P1] pkg/sql/colexec/aggexec/ordered_percentile.go:49: PERCENTILE_CONT always declares decimal128 scale input scale plus one. A valid DECIMAL(38,38) input therefore becomes the invalid DECIMAL(38,39), and flush multiplies the 38-scale unscaled value by 10; values such as 0.9 overflow even for percentile 0, which should simply return the input. Cap or otherwise choose a representable result precision/scale and test the maximum-scale boundary.

[P2] pkg/sql/parsers/dialect/mysql/mysql_sql.y:12524: GROUP_CONCAT accepts both its in-parentheses ORDER BY and WITHIN GROUP ORDER BY, then silently discards the first list in favor of the second. Reject the ambiguous double-order form instead of changing its meaning silently, and add parser coverage.

1. 远程执行能力门控

• 新增 ordered-set percentile 聚合 ID 的 MORPC 协议门控。
• merge main 后，保留 main 的 MORPCVersion11/12/13，将 percentile 能力顺延为 MORPCVersion14。
• remote aggregate / merge group 在协议版本不足时拒绝远程执行，避免新 CN 把 PERCENTILE_CONT/DISC 发给旧 CN 后报 unexpected aggID。

2. percentile 聚合内存无限增长

• 为 ordered percentile 增加聚合器内部 spill 机制。
• 覆盖 H0 聚合和单热点 group 场景，避免只依赖 generic group spill。
• 补充了 H0、大单组、generic spill 组合场景的回归测试。

3. WITHIN 关键字兼容性

• 将 WITHIN 从保留关键字改为上下文关键字/非保留用法。
• 保证已有 SQL 如 select within from t、create table t (within int) 不被破坏。
• 补充 parser 兼容性测试。

4. decimal 返回类型/溢出

• 修正 PERCENTILE_CONT 对 DECIMAL 输入的返回类型推导，避免生成非法 DECIMAL(38,39)。
• 处理最大 precision/scale 边界，避免 p=0 时仍因内部放大计算溢出。
• 补充 DECIMAL(38,38) 和 38 位整数边界测试。

5. GROUP_CONCAT 双 ORDER BY

• 对 group_concat(x order by y) within group (order by z) 这类双排序写法改为拒绝。
• 避免静默丢弃其中一个 ORDER BY 导致语义被悄悄改写。
• 补充 parser 覆盖。

[iamlinjunhong]

Requesting changes on exact head 15d9b16e832431baa3cdedc892813aab2ded3892 for two P1 blockers:

1. Maximum-width DECIMAL inputs can still make PERCENTILE_CONT return a discrete rounded value instead of the required continuous interpolation.
2. The contextual WITHIN lexer still breaks existing identifier use when the next token is GROUP.

I read the previous review, all review threads, and the PR conversation. The author replies explicitly address the previous findings, and the protocol gate, bounded spill path, basic identifier cases, and double-ORDER rejection are present; the two blockers below remain on the current head.

Review scope: full diff from merge-base 3c7392a0f86b23712da71f31d7a9198a2c8591a9, including parser → binder → plan/type inference → compile/remote protocol → aggregate merge/spill/flush/free and tests. All 26 GitHub checks are complete with no failing conclusion. Local targeted tests could not start in the isolated worktree because the required cgo/libmo.dylib artifact is absent; that is a test prerequisite failure, not test evidence. Both blockers below are directly provable from the current code paths.

[daviszhen]

Requesting changes on exact head 15d9b16e832431baa3cdedc892813aab2ded3892 for two P1 blockers:

1. Maximum-width DECIMAL inputs can still make PERCENTILE_CONT return a discrete rounded value instead of the required continuous interpolation.
2. The contextual WITHIN lexer still breaks existing identifier use when the next token is GROUP.

I read the previous review, all review threads, and the PR conversation. The author replies explicitly address the previous findings, and the protocol gate, bounded spill path, basic identifier cases, and double-ORDER rejection are present; the two blockers below remain on the current head.

Review scope: full diff from merge-base 3c7392a0f86b23712da71f31d7a9198a2c8591a9, including parser → binder → plan/type inference → compile/remote protocol → aggregate merge/spill/flush/free and tests. All 26 GitHub checks are complete with no failing conclusion. Local targeted tests could not start in the isolated worktree because the required cgo/libmo.dylib artifact is absent; that is a test prerequisite failure, not test evidence. Both blockers below are directly provable from the current code paths.

1. 远程执行能力门控

• ordered-set percentile 新增独立 MORPC 能力门控。
• merge main 后，main 已占用 MORPCVersion14 给 utf8mb4 text MIN/MAX collation semantics，所以 ordered-set percentile 顺延为 MORPCVersion15。
• 远程执行时：
  • PERCENTILE_CONT/DISC 要求 v15；
  • GROUP_CONCAT ORDER BY 继续要求 v6；
  • collation-aware text MIN/MAX 继续要求 v14。

2. DECIMAL(38,*) 连续插值问题

• PERCENTILE_CONT 对最大宽度 DECIMAL(38,*) order 表达式改为拒绝，避免 0,1,p=0.5 被错误四舍五入成离散值。
• PERCENTILE_DISC 不受影响，仍支持 DECIMAL(38,*)。
• 增加 UT 和 BVT：
  • PERCENTILE_CONT + DECIMAL(38,0) 拒绝；
  • PERCENTILE_DISC + DECIMAL(38,0) 正常返回；
  • DECIMAL(37,0) 连续插值返回 0.5。

3. WITHIN 关键字兼容性

• scanner 只在完整识别到 WITHIN GROUP ( 时返回 WITHIN token。
• WITHIN /* comment */ GROUP (...) 支持。
• SELECT 1 AS within GROUP BY 1 继续把 within 当普通 identifier。
• 增加 parser 回归测试覆盖 alias 邻接 GROUP BY 和注释场景。

4. GROUP_CONCAT 双 ORDER BY

• group_concat(v order by x) within group (order by y) 改为报错。
• 避免静默丢弃其中一个排序条件造成语义变化。

5. bounded spill / 大 group 内存问题

• ordered percentile 增加聚合器内部 spill 逻辑。
• 覆盖 H0、大单组、generic spill 组合场景，避免只依赖 group spill 导致 OOM。