fix: decode bytea hex from wal2json before writing to postgres

[siriusfreak]

Fixes #1089.

wal2json renders a bytea column as bare hex digits:

"columnnames":["id","payload"],"columntypes":["integer","bytea"],"columnvalues":[3,"deadbeef"]

filterRowColumnsForAction special-cases jsonb/json, ranges, tsvector and arrays but has no case for bytea, so that Go string is handed to pgx as the parameter value for a bytea column and its eight ASCII characters are stored as the column contents. \xdeadbeef on the source becomes \x6465616462656566 on the target.

Nothing else about the row changes, so row counts and keys still match and the corruption only surfaces when something parses the column. Snapshots are unaffected — pgx hands those values over as []byte — so a freshly loaded target looks correct and then degrades as replication runs, which makes this easy to miss.

The change

deserializeByteaValue on the write path, applied wherever serializeJSONBValue already is (the row values and the WHERE clause, so UPDATE and DELETE are covered as well as INSERT).

It accepts both the bare form wal2json emits and the postgres \x hex format, since pkg/transformers/encrypted_aes_siv_transformer.go documents the latter for this path. Values that are already []byte, as they are during snapshots, fall through untouched.

A value that is not valid hex is passed through rather than dropped, so an unexpected producer format degrades to the previous behaviour instead of failing the batch.

While here: decodeByteaHex in that transformer requires the \x prefix and returns errEncryptedAESSIVByteaNotHex without one, while wal2json emits the bare form — that path looks affected too, but it is a separate concern and I left it alone.

Verification

Unit tests cover both hex forms, an empty value, an already-decoded []byte, invalid and odd-length hex, nil, and a non-bytea column.

Also checked end to end against two live PostgreSQL instances with the repro from #1089, on INSERT, on UPDATE (which exercises the WHERE path), and on an empty bytea:

--- source ---                          --- target, patched ---
10 \x0102ff                             10 \x0102ff
11 \x7b2273747265616d6564223a747275657d  11 \x7b2273747265616d6564223a747275657d
12 \x                                    12 \x

Before the change those three arrive as \x303130326666, \x37623232... and an empty value respectively.

[siriusfreak]

LLM miss to disclose. Claude Code, Opus 5

[github-actions]

Coverage

Total: 59.5% (±0.0% vs main)

Coverage in packages changed by this PR:

Package	Coverage	Δ
pkg/wal/processor/postgres	84.4%	+0.2%

[kvch]

Thank you for the contribution. The issue you have identified is real. However, the fix doec not cover all of the problematic areas. I would prefer to decode at the wal2json boundary.

walDataDeserialiser is the only place that knows the value came from wal2json. Decoding there covers every downstream call site at once, including buildBulkDeleteSinglePK and buildBulkDeleteCompositePK in postgres_wal_dml_adapter_bulk.go, which still bind the raw hex today.

// wal_pg_listener.go, processWALEvent
if err := l.walDataDeserialiser(msg.Data, event.Data); err != nil {
    return fmt.Errorf("error unmarshaling wal data: %w", err)
}
decodeByteaColumns(event.Data)

wal.Data carries both Columns and Identity, so one pass handles insert values, the SET clause, the WHERE clause and both bulk-delete builders.

This also removes a hazard in the current placement. The transformer processor wraps the writer (stream.go:246), so deserializeByteaValue sees post-transform values. Six transformers declare ByteArrayDataType and return a plain string, so any even-length hex output gets halved:

column_transformers:
  payload:                    # bytea column
    name: literal_string
    parameters:
      literal: "00000000"     # stores 4 bytes, not 8

[siriusfreak]

Thanks — you're right on both counts, and I've moved it.

decodeByteaColumns(event.Data) now runs in processWALEvent right after walDataDeserialiser, walking Columns and Identity. That covers the SET clause, the WHERE clause and both bulk-delete builders in one pass instead of the two call sites I had patched.

The transformer hazard is the more important half and I had missed it entirely. Decoding after the transformer processor meant any ByteArrayDataType transformer returning even-length hex would have had its output halved — your literal_string: "00000000" storing 4 bytes is exactly right. At the boundary, transformers now see the same []byte the snapshot path gives them, which also lines up with what encrypted_aes_siv_transformer documents.

The writer-side change is gone; postgres_wal_dml_adapter.go is back to its original state. The diff is three files, all additive.

Verified against two live PostgreSQL instances after the move, including the identity path specifically: a table with a bytea PRIMARY KEY, where DELETE ... WHERE key = '\xcafe' and UPDATE ... WHERE key = '\xdeadbeef' both match and apply on the target. Before the fix those bound hex text. INSERT, UPDATE and DELETE all round-trip byte-identically.

Also rebased onto current main.

[kvch]

Thank you!