Add HexSHA256File for computing the SHA256 checksum of a file - #6505
Conversation
be5e639 to
f306db2
Compare
Checksumming a file so far meant HexSHA256(InputTextFile(name)), which reads the whole file into memory, and reads it as text: a file whose name ends in '.gz' is silently decompressed, and on systems distinguishing text and binary mode the line endings are translated. Neither is what one wants when checking a downloaded file against a published checksum. HexSHA256File reads the file in binary and in chunks, so the digest describes the bytes on disk and the file size is not bounded by memory. Passing 'true' as second argument opts into the transparent decompression instead, since SyFopen offers it anyway. HexSHA256 on a stream now also hashes in chunks rather than calling ReadAll; the comment claiming the streams API cannot do this predates ReadAll gaining its length argument. Assistance from Claude Opus 5 via Claude Code: implementation, documentation and tests, reviewed by me. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
f306db2 to
479241a
Compare
|
I've not yet reviewed this AI generated PR myself. So you may wish to hold off a review until I did so to avoid potentially being exposed to slop. |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6505 +/- ##
==========================================
- Coverage 79.06% 79.04% -0.03%
==========================================
Files 685 685
Lines 293835 293882 +47
Branches 8666 8669 +3
==========================================
- Hits 232325 232295 -30
- Misses 59710 59783 +73
- Partials 1800 1804 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
I feel this is ready for review. |
ThomasBreuer
left a comment
There was a problem hiding this comment.
Looks good, two minor comments.
| if not args[2] in [ true, false ] then | ||
| ErrorNoReturn("<decompress> must be 'true' or 'false'"); | ||
| fi; | ||
| decompress := args[2]; |
There was a problem hiding this comment.
Perhaps assign decompress before the check for true/false, then the error message fits better.
| end); | ||
|
|
||
| InstallGlobalFunction( HexSHA256File, | ||
| function(args...) |
There was a problem hiding this comment.
| function(args...) | |
| function(filename, decompress...) |
and then simplify the code accordingly
lgoettgens
left a comment
There was a problem hiding this comment.
LGTM (up to Thomas' comments)
|
Honestly, part of me would prefer a 'binary read'. Also, maybe make that 'decompression' argument compulsory, it's a bit strange if this is the only function in GAP which transparently doesn't decompress, if everything else does -- at least if the argument is required it's obvious (and if someone is using this function they should probably think about if they do want decompression or not). |
Everything else in GAP decompresses a '.gz' file as it reads it, so a default of 'false' here is the one function that quietly does the opposite. Requiring the argument makes the choice visible at every call site, and it removes the optional-argument handling entirely. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claude replying here:
@ChrisJefferson — agreed, and for the reason you give: everything else in GAP @ThomasBreuer — both your comments dissolve rather than get applied, I hope On the binary read: I agree it is the better primitive and said so in the
|
|
OK, the real Max here now (I am sorry for letting loose the AI without checking its comment first, will do that next time): contrary to what it says, I'd really like to get this PR merged as-is (well, up to fixing bugs and other stuff you guys point out of course). We can look into adding a good "binary file access API" to GAP itself in the future, but I think this is a non-trivial task if we want something that (unlike IO) is not just a thin wrapper around POSIX and hence annoying to support on Windows (I still dream of a "native" MingW GAP) or worse, on future OSes. Can do that, find it interesting -- but's a lot more involved than this PR. If we ever add |
Checksumming a file means
HexSHA256(InputTextFile(name)), which reads thewhole file into memory and reads it as text: a
.gzname is silentlydecompressed, and where text and binary mode differ the line endings are
translated. Neither is what you want when checking a download against a
published checksum.
HexSHA256File( <filename>[, <decompress>] )reads in binary and in chunks.Passing
trueopts into the decompression instead, sinceSyFopenoffers itanyway.
HexSHA256on a stream now also hashes in chunks; the comment saying thestreams API could not do that predates
ReadAllgaining its length argument.Where the file reading belongs. I put it in the kernel because the library
has no binary-safe read — that is the missing primitive, and this is one
consumer of it. Exposing a general binary read instead would be more useful
and more design; happy to go that way.
make check: 0 failures in 314 files.Written with Claude Opus 5 via Claude Code; reviewed by me. The commit lists
the tool as co-author.