Clast.Pcodec

A .NET implementation of the Pco numeric compression format described by Mortenson et al. (2025). Wire-compatible with the reference pcodec/pcodec Rust crate, so files produced by either implementation can be read by the other.

Targets net10.0, net8.0, and netstandard2.0. Performance is the primary goal; the net10.0 build is the priority and uses SIMD where it makes sense, with slower scalar fallbacks on the older targets.

Status

Encoder and decoder feature-parity with the Rust reference for the wrapped/standalone v4.1 format:

• Modes — Classic, IntMult, FloatMult, FloatQuant (Dict on the decode side).
• Delta encoding — NoOp, Consecutive, Lookback, Conv1.
• Auto-pick — per-chunk brute-force across detected mode candidates and all delta variants.
• Streaming — PcoReader<T> over ReadOnlySequence<byte>, PcoWriter<T> over IBufferWriter<byte>.
• Wrapped format — PcoWrappedDecoder<T> / PcoWrappedEncoder<T> for hosts that manage chunk/page framing themselves (Vortex, parquet's pco extension, etc.).
• Multi-chunk — transparent for inputs above the per-chunk format limit.
• Cross-validated — every encoder feature is tested by encoding through Clast.Pcodec and decoding through the Rust reference binary.

Quick start

using Clast.Pcodec;

// One-shot
byte[] encoded = Pco.Compress(values);              // auto mode/delta selection
double[] decoded = Pco.Decompress<double>(encoded);

// Streaming (no T[] allocation, pipelines-friendly)
using var writer = new PcoWriter<double>(bufferWriter);
writer.Write(batch1);
writer.Write(batch2);
writer.Dispose();   // flushes remaining + emits format terminator

using var reader = new PcoReader<double>(memory);
Span<double> dst = stackalloc double[256];
while (reader.ReadBatch(dst) is var n && n > 0)
    Process(dst.Slice(0, n));

// Wrapped format (host manages framing — Vortex, parquet, ClickHouse)
var encoder = new PcoWrappedEncoder<double>();
ReadOnlySpan<byte> header = encoder.Header;          // 1–2 bytes; persist alongside the array
WrappedChunkBytes wrapped = encoder.EncodeChunk(values);
//   wrapped.ChunkMeta — store once per chunk
//   wrapped.Page      — store once per page (1 page per chunk in this encoder)

var decoder = new PcoWrappedDecoder<double>(header);
ChunkHandle chunk = decoder.BeginChunk(wrapped.ChunkMeta);
double[] dst = new double[pageElementCount];
decoder.DecodePage(chunk, wrapped.Page, pageElementCount, dst);

Layout

src/Pcodec            Clast.Pcodec library
tests/Pcodec.Tests    xUnit tests (multi-targeted: net10.0 + net472)
bench/Pcodec.Bench    BenchmarkDotNet harness

License

Apache-2.0. See LICENSE.