feat(testtool): generate FSST reference data for other implementations

[CurtHagenlocher]

Adds ew-test-tool create_fsst_test_file, which writes one small Parquet file exercising the FSST encoding's corner cases, for use as reference data by other implementations.

Modelled on the ALP file in apache/parquet-testing#119: one file rather than a corpus, and every FSST column duplicated by a DELTA_LENGTH_BYTE_ARRAY column holding the same values — so a reader can bit-compare the two without needing any expected data shipped alongside the file. That predecessor was rejected partly for size, so this one is 182 KB uncompressed (80 KB with --compression zstd).

The corpus is synthetic and seeded, so the file is reproducible from source and carries no third-party licensing. Four row groups, each training its own symbol table (§1.4):

Row group	Rows	Coverage
urls	400	happy path — high-cardinality URLs over 14 pages sharing one table
nulls_and_empties	240	NULLs and zero-length values, def levels alongside FSST pages
escapes	200	bytes absent from the table — forces escape sequences
long_values	16	~400-byte values, few per page, so the offset array encoding differs

It verifies rather than asserts

The command reads the file back before reporting, because most of what makes it useful is chosen rather than declared: the writer picks the offset encoding per page, the trainer picks the symbol table, and §7.5 may decline FSST for a chunk outright. Two rounds of that feedback changed the corpus:

• The escape section produced no escapes at all. It put a rare code point in every value, so the trainer simply learned it as a symbol. Only every ninth value carries one now, and the file contains 181 escape sequences.
• A binary chunk fell back under §7.5 and silently vanished from the summary, which read as a bug in the summary rather than the writer declining. Fallback is now reported explicitly.

Sample output:

Row group 3 — long_values
  string_fsst       FSST    table of  131 symbols (longest  8, body 463 bytes), 4 data pages, offsets 4 PLAIN / 0 DELTA, 32 escapes
  string_fsst16     FSST_16 table of  421 symbols (longest  8, body 954 bytes), 4 data pages, offsets 4 PLAIN / 0 DELTA, no escapes

OK — every FSST column decodes bit-identically to its reference column.

Both offset encodings are covered (long_values is the row group where PLAIN beats DELTA), and both symbol table widths appear in every row group.

Known coverage gap

No FSST_16 column contains an escape sequence, so the file does not exercise FSST_16 escape decoding. That is not fixable from the writer: Clast.Fsst's 16-bit tables always contain all 256 single-byte symbols, so nothing is ever left to escape. It is also the one place the two variants are genuinely asymmetric, and the one part of the FSST_16 wire format still resting on inference rather than confirmed spec text.

Also

Grants Core internals to the TestTool — every other consuming project already had them — so the page walk can decompress a compressed values section and report page detail for --compression zstd files too.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds a new ew-test-tool create_fsst_test_file command that generates a small, reproducible Parquet file designed to exercise FSST encoding corner cases, then reads it back to verify FSST columns decode bit-identically to reference columns and to report what was actually written (table details, per-page offset encoding, escape counts). It also extends Core internals access to the TestTool so the command can walk/decompress pages when describing the output.

Changes:

• Add CLI command wiring and help text for create_fsst_test_file.
• Introduce FsstTestData to synthesize the corpus, write the file with FSST/FSST_16 columns plus reference columns, then verify and describe the encoded pages.
• Grant EngineeredWood.Parquet.TestTool access to EngineeredWood.Core internals via InternalsVisibleTo.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
src/EngineeredWood.Parquet.TestTool/Program.cs	Adds command dispatch + usage/help text for the new FSST test-file generator.
src/EngineeredWood.Parquet.TestTool/FsstTestData.cs	Implements corpus generation, Parquet writing options, verification, and page-walk reporting for FSST/FSST_16.
src/EngineeredWood.Core/EngineeredWood.Core.csproj	Adds InternalsVisibleTo for the TestTool so it can use Core internals needed for detailed page inspection.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.