Repository navigation
Bug on 32 bit systems? #56
Description
Debian packaging ran into a problem with pandoc on i386, described at jgm/pandoc#9233. I think I've traced it to base64. Here is a Dockerfile that reproduces the segfault:
FROM i386/debian:sid
RUN echo 'deb-src http://deb.debian.org/debian/ unstable main' >> /etc/apt/sources.list
RUN apt update
RUN apt install -y cabal-install libghc-base64-dev libghc-bytestring-dev
RUN printf 'import "base64" Data.ByteString.Base64 (encodeBase64)\n\
import qualified Data.ByteString as B\n\
main = B.readFile "/usr/bin/whoami" >>= print . encodeBase64\n'\
> test.hs
RUN runghc -XPackageImports test.hs
Activity
iliastsi commentedon Dec 6, 2023
I reproduced this with ghc-9.0 as well (i.e., use
debian:stableinstead ofdebian:sid), in case this helps. In addition, run thebase64testsuite on 32-bits fails as well (with segmentation fault).jgm commentedon Dec 6, 2023
Nothing special here about
/usr/bin/whoami- I just chose a binary I was sure would be available. If you useB.take nto trim down the size of the bytestring that is converted, you'll eventually stop getting the segfault. What I found is that:emilypi commentedon Dec 7, 2023
Hi John, I deleted all of my 32 bit code because I don't want to support it anymore. Do you have a dire need for it? Also, what version of
base64?emilypi commentedon Dec 7, 2023
Looking into it, I'd wager the memory boundaries for
peekgot a little stricter in GHC versions in the 9.x series which makes this line:base64/src/Data/ByteString/Base64/Internal/W64/Loop.hs
Line 51 in 7080ac0
A little less sound. I can have a fix out rather quickly, but I'd need test data. Can you provide the offending image so i can add it as a regression? I can just add that to the test suite and support i386 in CI without adding my word32-specific code back into the repo
jgm commentedon Dec 7, 2023
The Docker container above gives you everything you need to reproduce the problem. Any medium sized binary will work (I just use the
whoamiexecutable in the above test, but you could probably use any image).As noted above, I've been able to reproduce problems (but not deterministically) with even fairly short bytestring literals.
There is no urgency on my end, because I already switched back to using base64-bytestring in pandoc.
emilypi commentedon Dec 7, 2023
If performance isn't your bottleneck, that's certainly a solution. It uses the worst-case-in-every-case approach. That said, on the branch for #57 I can't get it to trigger, even with multiple 32-bit CI attempts, so maybe the fix diagnosis was spot on? I'll need a regression tho, so if you end up with something that triggers it more deterministically than not, I'm all ears. I could also just attempt to encode/decode something against
whoamiand the like to regress, but i have no reasonable means of testing on 32 bit systems.jgm commentedon Dec 7, 2023
With whoami, it never failed to segfault for me. I only got indeterministic results for very short bytestrings (e.g. the first 20 bytes).
Are there benchmarks of base64 vs bytestring-base64? I may go back if the problem is fixed.
emilypi commentedon Dec 7, 2023
Okay - thanks i'll focus on that and see what i can dredge up. My guess is that when peeking a 64 byte word off, we cross some memory barrier somewhere that gets tripped in 32 bit systems when there are fewer than 2 words left in the array. The fix should target that case, since prior to the fix, we were looking only if there were at least six bytes left, not necessarily a full word, and GHC was always lax enough to say "if we attempt to read a full word and it's partial, fill with
\NUL." It's actually probably saner to have this fix in regardless.Benchmarks exist here, and do compare
encodeBase64anddecodeBase64against thebase64-bytestringlibrary. I maintain both libraries, so I regress against my worst case in this package alot of the time :)Here's a sample of all recent output: https://github.com/emilypi/base64/blob/72cfd854ee3ba394e6dd7cfa0473d0fe542bf8ad/output.html.
The long short of it is that
base64gets away with an encode that's roughly 80%-100% faster, and decode is roughly 25%-40% faster for the typed loops, with negligible difference in the untyped loops. So, if you plan on encoding, and in particular encoding large numbers of bytes, I'd use this library. If you're primarily decoding, it's your pick. if you're only encoding small bytes, however, i'd honestly probably go with just thebase64-bytestringpackage just because that one has fewer dependencies and the difference you'll see is minimal at best.jgm commentedon Dec 7, 2023
OK, thanks, that's helpful.
By the way, the initial image I tested with is
test/lalune.jpgin the pandoc source distribution. Feel free to use that. But I think just about any image would work.emilypi commentedon Dec 8, 2023
After discussion with @kozross, I'm pretty sure the issue is as described. Merging and releasing
jgm commentedon Dec 8, 2023
I don't see a new release yet?
emilypi commentedon Jan 11, 2024
Got caught up in the holidays and forgot. It's up now: https://hackage.haskell.org/package/base64-1.0
emilypi commentedon Jan 11, 2024
Actually, keeping this open just to confirm.
dpwiz commentedon Apr 3, 2024
Don't know if that's related, but we've been hit by SIGBUS/unaligned access on 32bit armv7a device 😟
(base64-1.0 on GHC-8.10.7)
emilypi commentedon Apr 3, 2024
@dpwiz could you give more about the details? My suspicion would immediately be the encoding loop in that situation, but that's probably because we use 64-bit intrinsics in that particular hotloop. This library dropped support for 32 bit arches a few years ago.
dpwiz commentedon Apr 4, 2024
Oops... Nvm then (:
emilypi commentedon Apr 8, 2024
I'd still like to figure out how to solve it 😄
dpwiz commentedon Apr 15, 2024
Well, that was some android mobile running 32bit/v7 chip. Apparently those can't tolerate unaligned reads, so the app got killed.
I don't know much of the details, only that SIGBUS error code and the env description. Should be reproducible in a faithful QEMU I think.
emilypi commentedon Apr 15, 2024
I'll see if I can work it out - thanks for the lead @dpwiz :)
Add regression test for issue emilypi#56 (32-bit segfault in W64 loop)
emilypi commentedon Oct 13, 2026
Closing as "won't fix" because i don't support 32 bit systems anymore, and it's not reproducible post-#56 anyway.