feat(writer): extend hoodie.meta.fields.mode to MoR + fail-fast on RLI/SI incompatibility - #19323
Draft
nsivabalan wants to merge 7 commits into
Draft
feat(writer): extend hoodie.meta.fields.mode to MoR + fail-fast on RLI/SI incompatibility#19323nsivabalan wants to merge 7 commits into
nsivabalan wants to merge 7 commits into
Conversation
…pulation on CoW tables
Introduces `hoodie.meta.fields.mode` — a comma-separated list of meta columns to populate
when `hoodie.populate.meta.fields=false`. Allowed tokens are `_hoodie_commit_time` and
`_hoodie_file_name`; any other token is rejected up-front. The mode is immutable at runtime
(settable only at table creation, via hudi-cli, or during upgrade).
Motivation: the existing all-or-nothing `populate.meta.fields=false` disables incremental
queries (needs `_hoodie_commit_time`) and file-level pruning / debugging lookups
(needs `_hoodie_file_name`). This lets a table opt into either or both without paying for
the remaining three meta columns.
Resulting modes:
- populate.meta.fields=true → ALL (default; mode list ignored)
- populate.meta.fields=false, mode="" → NONE
- populate.meta.fields=false, mode=<subset> → selective (COMMIT_TIME_ONLY /
FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME)
- populate.meta.fields=true, mode=<subset> → rejected at writer init
Scope in this patch:
- Spark writer path (Avro + Row): HoodieAvroParquetWriter, HoodieSparkParquetWriter,
HoodieRowCreateHandle take a Set<String> meta-fields mode and populate the columns
selectively; bloom filter / record-key index registration stays skipped when record key
is not populated.
- Incremental read path: CoW IncrementalRelationV1/V2 accept the new mode via
isCommitTimePopulated(); MoR incremental relations keep the strict populate=true guard
until the MoR log-write path is updated in a follow-up.
Fail-fast at writer init:
- Unknown token in the mode list (parser throws).
- populate.meta.fields=true combined with a non-empty mode (ambiguous).
- MERGE_ON_READ + non-empty mode (log-write path not yet wired; follow-up).
- Non-Spark engine + non-empty mode (Flink RowData / Java-client not yet wired; follow-up).
Runtime immutability:
- HoodieWriterUtils.validateTableConfig now explicitly rejects a null-on-disk → non-empty
transition on the mode property (existing loop only flagged the mismatch when the on-disk
value was non-null, which let this silent-drop path slip through).
Tests:
- TestHoodieMetaFieldsMode (hudi-hadoop-common) — accessor coverage for every combination
plus unknown-token rejection and whitespace tolerance.
- TestHoodieWriteConfigMetaFieldsMode (hudi-client-common) — builder + validate() coverage.
- TestMetaFieldsMode (hudi-spark) — end-to-end write / re-read for every mode plus the
three rejection paths (populate=true+mode, unknown token, MoR+mode).
…enum Addresses review feedback on apache#19205: - Fold (populateMetaFields boolean, comma-separated mode) into a single MetaFieldsMode enum (ALL / NONE / COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME). The enum is the sole argument passed through every writer factory and writer constructor. populate.meta.fields stays on the config surface for backward compat and is auto-derived from the enum. - On-disk representation is now the enum name (e.g. "COMMIT_TIME_ONLY") instead of a comma-separated token list. Existing tables without the property continue to fall back to ALL or NONE based on the legacy hoodie.populate.meta.fields boolean. - Drop _hoodie_commit_seqno population from the selective write path — only _hoodie_commit_time is needed for incremental queries. (Review comments on HoodieRowCreateHandle:193, HoodieSparkParquetWriter:100, HoodieAvroParquetWriter:103.) - Extend the IncrementalRelation error message to state that hoodie.meta.fields.mode is physical-storage and cannot be flipped by changing write options — existing tables must be recreated to change it. - StreamSync.initializeEmptyTable() was missing the mode pass-through — wired via setMetaFieldsModeFromString. All other non-test initTable call sites either use fromProperties() (which now routes the mode through) or intentionally don't touch meta fields (CLI table create, MDT, sample writes). Test coverage additions: - TestMetaFieldsMode (hudi-spark): end-to-end assertions read parquet back and verify which meta columns are populated / null per mode. Covers row-writer path (bulk_insert with row.writer.enable=true, default) AND non-row-writer path (insert with row.writer.enable=false). Adds inline-clustering coverage for all 5 modes verifying that clustered files preserve the mode's column population semantics. - TestHoodieStreamerMetaFieldsMode (hudi-utilities): parameterized end-to-end test through the HoodieStreamer entrypoint covering all 5 modes plus MoR+selective rejection.
Previously the populateMetaFields=false branch of HoodieDatasetBulkInsertHelper projected the five Hudi meta columns as Alias(Literal(UTF8String.EMPTY_UTF8, StringType), name). A non-null Literal derives a non-null Alias, so the resulting StructField had nullable=false. Under ALL mode this was harmless because HoodieRowCreateHandle.writeRow unconditionally fills every meta slot with a non-null value. Once selective / NONE modes started leaving the opted-out columns null on the row (per hoodie.meta.fields.mode), the non-nullable StructType led SparkToParquetSchemaConverter to declare the columns as `required binary` on disk, and any subsequent read failed with ParquetDecodingException: could not read bytes at offset 0. Switch the stub to Alias(Literal.create(null, StringType), name) so the projected StructField is nullable and the physical parquet column is written as OPTIONAL. Handle-level population (writeRow / writeRowNoMetaFields / writeRowSelectiveMetaFields) is unchanged — the stub value is never observed on disk; the fix only affects the parquet schema. Fixes all row-writer + clustering selective-mode failures in TestMetaFieldsMode (commitTimeOnly / fileNameOnly / commitTimeAndFileName / none, plus clustering variants). 18/18 tests pass locally.
…elds modes TableSchemaResolver.getTableSchema() and getTableSchema(String timestamp) decided whether to include the five Hudi meta columns in the projected schema based on populateMetaFields(). Under the selective modes introduced in this PR (COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME), populateMetaFields() returns false but the opted-in meta columns are still physically present in every Parquet file. The old logic silently stripped those columns from the read schema, so downstream consumers (notably the incremental relations, whose range filter projects _hoodie_commit_time) saw no meta columns at all and produced zero-row reads. Switch the decision to "include meta fields whenever the table's mode is not NONE." That preserves the previous behaviour for the two originally supported modes (ALL -> true, NONE -> false) and correctly turns the flag on for the three new selective modes. No schema change on disk; this only changes what the read side projects.
Adds a hudi-cli command to toggle hoodie.meta.fields.mode on an existing table: table set-meta-fields-mode --target-mode <MODE> [--force true|false] Where <MODE> is one of ALL, NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME. This is the sanctioned way to change the property outside of table creation — hoodie.meta.fields.mode is immutable at runtime because it is a physical-storage decision baked into files at write time. Safety guard: - On a table that already has commits, the command refuses to change the mode by default. Existing files were written under the current mode and are not rewritten by this command; new commits would be written under the new mode, producing mixed-mode files whose incremental / file-pruning semantics differ between old and new data. - Pass --force to override the guard. A warning is logged describing the specific data correctness impact. Behavior: - ALL and NONE are persisted implicitly (via populate.meta.fields=true/false); the hoodie.meta.fields.mode property is cleared when transitioning to ALL/NONE. - Selective modes (COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME) are persisted explicitly on hoodie.properties alongside populate.meta.fields=false. - No-op when target matches current mode. Test coverage (TestTableCommand): - All 5 modes settable on a fresh table. - Selective → ALL clears the property. - No-op when target matches current mode. - Rejects unknown enum values. - Refuses on a populated table without --force; the mode does not change. - Accepts on a populated table with --force; the mode changes. Follow-up to PR apache#19205 (hoodie.meta.fields.mode). PR-C (MoR support) is the next follow-up.
…remental-read paths Wires the selective meta-fields mode through the MoR path so MoR tables can opt into COMMIT_TIME_ONLY / FILE_NAME_ONLY / COMMIT_TIME_AND_FILE_NAME modes previously restricted to CoW. HoodieAppendHandle.populateMetadataFields now honors the mode when populateMetaFields=false: only the opted-in meta columns are set; the rest stay null on the log record. Log-compaction still skips commit_time by design (base record's commit_time is preserved). MergeOnReadIncrementalRelationV1/V2 gate on isCommitTimePopulated() instead of populateMetaFields(), matching the CoW read-path change from PR-A. Selective COMMIT_TIME modes on MoR are now accepted; NONE / FILE_NAME_ONLY still rejected with a clearer message. HoodieWriteConfig.validate() drops the MoR-selective-mode rejection guard added in PR-A now that MoR is supported. Tests: parameterized MoR bulk-insert + upsert coverage for all 5 modes, plus an end-to-end incremental-query round-trip under COMMIT_TIME_ONLY. Streamer test extended to accept all 5 modes on MERGE_ON_READ.
… mode The record-level index and secondary index both maintain their entries by _hoodie_record_key. Only MetaFieldsMode.ALL populates the record key column; every other mode (NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME) leaves it null on the base file. Prior to this change the writer would accept the combination and silently corrupt index maintenance — RLI would key on nulls, SI updates would land on phantom rows, and lookups would return incomplete results with no error surfaced. Add an explicit checkArgument in HoodieWriteConfig.Builder.validate() that rejects (RLI or SI) enabled with mode.isRecordKeyPopulated() == false. The guard reads the properties directly via ConfigProperty rather than through the accessor methods because validate() runs before the private HoodieWriteConfig constructor materializes the metadataConfig field, so writeConfig.isRecordLevelIndexEnabled() would NPE at that point. Tests: - TestHoodieWriteConfigMetaFieldsMode: parameterized rejection tests for all four non-ALL modes covering both RLI and secondary-index cases, plus a positive test that RLI is still allowed under ALL mode. - TestMetaFieldsMode: negative end-to-end tests confirming MoR incremental reads are rejected with a clear error message under FILE_NAME_ONLY and NONE modes (where _hoodie_commit_time is not populated).
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe the issue this Pull Request addresses
Phase 3 of 3 in the selective meta-fields series. Follow-up to #19205 (CoW writer + reader + config) and #19206 (
hudi-cli table set-meta-fields-mode). This PR extends the mode to Merge-on-Read tables and adds a fail-fast validation for indexes that depend on_hoodie_record_key.Without this PR, MoR tables silently ignore
hoodie.meta.fields.modeon the log-write path (base file honors it, log block does not — merge produces mixed-mode records), and enabling RLI or secondary index on a non-ALL mode silently corrupts index maintenance (index keyed on nulls, updates target phantom rows).Summary and Changelog
Two additive changes, one commit each:
1.
feat(writer): extend hoodie.meta.fields.mode to MoR log-write and incremental-read pathsHoodieAppendHandle.populateMetadataFields()now honors the mode whenpopulate.meta.fields=false. Under selective modes the append handle writes only the opted-in meta columns to log blocks, matching the base-file behavior from phase 1.MergeOnReadIncrementalRelationV1/V2(and CoWIncrementalRelationV1/V2) now gate ontableConfig.isCommitTimePopulated()instead ofpopulateMetaFields(). Tables inCOMMIT_TIME_ONLY/COMMIT_TIME_AND_FILE_NAMEnow serve incremental queries correctly;FILE_NAME_ONLY/NONEfail fast with an actionable error message that names the mode and points at remediation.TableSchemaResolverfrom phase 1 already emits meta columns whenever the mode is not NONE, so the log-block reader picks them up without further changes.2.
feat(writer): fail-fast when RLI/SI are enabled with a non-record-key modecheckArguments inHoodieWriteConfig.Builder.validate()reject the combination of (RLI or SI enabled) with any mode that leaves_hoodie_record_keyunpopulated (NONE, COMMIT_TIME_ONLY, FILE_NAME_ONLY, COMMIT_TIME_AND_FILE_NAME).HoodieMetadataConfig.isRecordLevelIndexEnabled()/isSecondaryIndexEnabled()exactly — including the metadata-enabled gate and (for SI) the column-non-empty gate. Inlined rather than delegating to the accessor becausevalidate()runs before theHoodieWriteConfiginstance'smetadataConfigfield is materialized.Impact
Behavior change (MoR tables only, opt-in): MoR tables that set
hoodie.meta.fields.modeexplicitly now respect that mode in log blocks. Previously the mode was silently dropped on the log-write path — this is a bug fix, not a compatibility break. Tables that never set the mode see no change.Behavior change (RLI/SI validation, all engines): Attempting to enable RLI or SI on a table with a non-ALL mode now fails at write-config build time with a clear error. Previously it would silently produce incomplete / phantom index entries. No impact on tables that use RLI/SI with the default (ALL) mode.
Read-side error: MoR incremental queries on
FILE_NAME_ONLY/NONEtables that previously returned zero rows silently now throw with an actionable message. This is the same guard already present for CoW in phase 1 — extended to MoR here.Risk Level
Low. All changes are additive and guarded by the mode enum introduced in phase 1. The MoR log-write path change only fires when the mode is set (opt-in), and the RLI/SI guard only fires when the user actively enables those indexes on a non-ALL mode — both cases previously produced silent data-correctness issues, so the fail-fast behavior is strictly safer.
Verification: new unit + integration tests covering all 5 modes on MoR (bulk-insert, upsert, incremental) plus parameterized RLI + SI rejection across all four non-ALL modes. All existing tests still pass (17 in
TestHoodieWriteConfigMetaFieldsMode, 32 inTestMetaFieldsMode, 10 inTestHoodieStreamerMetaFieldsMode).Documentation Update
None. The
hoodie.meta.fields.modeconfig was documented in phase 1 (#19205). This PR extends its scope from CoW to MoR and adds validation guards; no new user-facing config or feature.Contributor's checklist