tarfs: compact Entry, reconstructing tar.Header on demand - #2433
Draft
markusthoemmes wants to merge 1 commit into
Draft
tarfs: compact Entry, reconstructing tar.Header on demand#2433markusthoemmes wants to merge 1 commit into
markusthoemmes wants to merge 1 commit into
Conversation
A tarfs index retains ~1KB per file: the stored tar.Header costs three time.Times and several always-zero fields, every entry carries a PAX map allocated to hold a single well-known checksum record, and hdr.FileInfo() boxes another allocation. Since the package cache keeps a tarfs index alive per cached package, this multiplies across every file of every package in long-running processes. Store a compact form instead: packed scalar fields, the APK-TOOLS.checksum.SHA1 record hex-decoded into a fixed [20]byte with remaining PAX records (xattrs, mostly) spilled to a nil-by-default map, and *Entry implementing fs.FileInfo itself. Entry.Header becomes a method that reconstructs the header on demand, which is a breaking change for consumers of the Entry.Header field. On a wolfi go-1.25 package (6339 files) this shrinks the index from 987 to 324 bytes per entry (6.0 to 2.0 MB), over half of which comes from replacing the per-entry PAX map with the decoded checksum field. An image built from this branch is byte-identical to one built from main in every blob (layers, manifests, configs), differing only in the vcs-revision annotations.
markusthoemmes
force-pushed
the
tarfs-compact
branch
from
August 25, 2026 10:13
add210d to
d2171c7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A tarfs index retains ~1KB per file: the stored tar.Header costs three time.Times and several always-zero fields, every entry carries a PAX map allocated to hold a single well-known checksum record, and hdr.FileInfo() boxes another allocation. Since the package cache keeps a tarfs index alive per cached package, this multiplies across every file of every package in long-running processes.
Store a compact form instead: packed scalar fields, the APK-TOOLS.checksum.SHA1 record hex-decoded into a fixed [20]byte with remaining PAX records (xattrs, mostly) spilled to a nil-by-default map, and *Entry implementing fs.FileInfo itself. Entry.Header becomes a method that reconstructs the header on demand, which is a breaking change for consumers of the Entry.Header field.
On a wolfi go-1.25 package (6339 files) this shrinks the index from 987 to 324 bytes per entry (6.0 to 2.0 MB), over half of which comes from replacing the per-entry PAX map with the decoded checksum field. An image built from this branch is byte-identical to one built from main in every blob (layers, manifests, configs), differing only in the vcs-revision annotations.