fix(schema): store msgpack columns as binary in every dialect - #1411
Open
snowyukitty wants to merge 2 commits into
Open
fix(schema): store msgpack columns as binary in every dialect#1411snowyukitty wants to merge 2 commits into
snowyukitty wants to merge 2 commits into
Conversation
added 2 commits
July 23, 2026 19:04
A `bun:",msgpack"` column could not be read back on any dialect. Two
independent causes had to be fixed together.
First, Table.newField discovers the SQL type from the Go field type, so a
msgpack column was created textual — varchar(255) on MySQL. The column
holds encoded bytes rather than the Go type's own representation, so it
needs a binary type whatever the field is declared as.
Second, appendMsgpack wrote through internal.HexEncoder, which hard-codes
PostgreSQL's '\x..' byte literal and never consults the dialect, while
every other value appender routes through Dialect.AppendBytes. SQLite and
MySQL spell byte literals X'..' and MSSQL uses 0x.., so the value was
stored as text and could not be decoded on the way back:
msgpack: unexpected code=5c decoding map length (0x5c = '\')
msgpack: unexpected code=78 decoding map length (0x78 = 'x')
The dialect needs the whole encoded value before it can wrap it, so the
output is staged in a pooled writer and handed to Dialect.AppendBytes.
The writer records whether the encoder wrote at all, because a
CustomEncoder that returns without writing is SQL NULL while a Marshaler
returning no bytes is an empty value, and buffer emptiness cannot tell
the two apart. HexEncoder tracked the same bit.
internal/dbtest/msgpack_test.go adds a round trip driven by testEachDB.
On master it fails for pg, pgx, mysql5, mysql8, mariadb, mssql2019 and
sqlite; with this change all seven pass, and the full dbtest suite gains
no new failures.
Fixes uptrace#1219
internal.HexEncoder was the only consumer of github.com/tmthrgd/go-hex and appendMsgpack was its only caller, so the file is now dead. The submodule go.mod files still carry the dependency as an indirect requirement; `make go_mod_tidy` sweeps those separately.
snowyukitty
force-pushed
the
fix/msgpack-dialect-bytes
branch
from
July 23, 2026 10:06
b2f5288 to
081da3f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1219.
The issue is filed against SQLite, but a round trip on every configured database
fails on
master. Abun:",msgpack"column cannot be read back on anydialect:
Those two bytes are the whole story: the value was stored as the text of a
PostgreSQL byte literal, so decoding it later trips over the
\or thex.There turned out to be two independent causes, and neither fix works without the
other.
1. The column was created as text
Table.newFielddiscovers the SQL type from the Go field type, which for amsgpack-tagged struct yields a textual column —
varchar(255)on MySQL. But amsgpack column stores the encoded bytes, not the Go type's own representation,
so it needs a binary type regardless of how the field is declared:
Without this, MySQL rejects a correct binary literal outright
(
Error 1366: Incorrect string value), and 255 characters would truncateanything larger anyway.
2. The value was written with PostgreSQL's literal syntax everywhere
appendMsgpackwrote throughinternal.HexEncoder, which hard-codes'\x..'and never consults the dialect — the only value appender that doesn't. SQLite and
MySQL spell byte literals
X'..'; MSSQL uses0x...The dialect needs the complete encoded value before it can wrap it, so the output
is now staged in a pooled writer and handed to
gen.Dialect().AppendBytes.The writer keeps one bit that
HexEncoderalso kept, and it earns its place:Marshaler.MarshalMsgpack()returningnil, nilWrite(nil)CustomEncoder.EncodeMsgpack()returning without writingNULLAn empty buffer cannot tell those apart, so "was the writer used at all" is
tracked explicitly rather than inferred.
3. Which leaves
internal/hex.godeadHexEncoderwas that file's entire contents andappendMsgpackwas its onlycaller, so the second commit removes it along with
github.com/tmthrgd/go-hex, of which it was the only importer.To be precise about scope: this drops the dependency from the root module, while
36 submodule
go.modfiles still carry it as// indirect. I left those alone sothe diff stays readable —
make go_mod_tidysweeps them, and I'm glad to addthat as a third commit. So this narrows #1385 to a mechanical tidy rather than
closing it. Please drop the second commit if you'd rather take one of the
existing dependency PRs; after the first commit the file is dead either way.
Verification
internal/dbtest/msgpack_test.goadds a round trip driven bytestEachDB, so itruns against everything in
docker-compose.yaml:masterI ran the entire
internal/dbtestsuite on both trees against the samecontainers and diffed the failure lists: seven failures fixed, none introduced.
Unit coverage lives in
schema/appendmsgpack_test.go— dialect syntax, appendingto an in-progress query, PostgreSQL output holding still, the unwritten-versus-
zero-length distinction, a failing encoder producing only an error marker,
zero-length outcomes routed through the dialect, and confirmation that the
bun:",msgpack"tag actually reaches this path viaField.AppendValue. Four ofthose fail on
master.TestMsgpackWriterResetexercises the staging writerdirectly rather than through the pool, since
sync.Poolmay hand back adifferent object and a test that assumes otherwise can pass for the wrong reason.
Also run:
go vet ./...andgo test -race(in agolang:1.25container),GOOS=linux GOARCH=386 go build ./..., andgolangci-lint runwith findingsidentical to
master.Performance
Staging costs one buffer, taken from a pool that drops writers grown past 64 KiB
rather than keeping them alive. On a representative struct with a 512-byte
payload, 14 interleaved rounds of 30,000 fixed iterations:
Paired median +10.5%, slower in 10 of 14 rounds; the medians coincide only
because my machine drifts, so the paired figure is the honest one. Twelve
allocations become one. I haven't isolated how much of the time is staging versus
the dialects using stdlib
encoding/hexwherego-hexhad SIMD — an attempt toseparate them was confounded by escape analysis, so I'd rather not guess.
Things worth knowing before you merge
DiscoveredSQLTypeonly, so an explicitbun:",msgpack,type:..."still winsand anyone who already worked around this is untouched; and since msgpack
couldn't be read back anywhere before, no existing textual column holds msgpack
that anything successfully reads today. Existing tables keep their old column
until their owner alters them. Happy to document that, or to gate the type
change, if you'd prefer.
sync.Poolmay return a differentobject or none, so any assertion about its contents can silently pass. I tested
reset()directly instead and left the ceiling as three lines of visible coderather than write a test that could lie.