Repository navigation
APPS-4463: Improve DXDA parallelism for large single-part downloads - #110
Open
bluednanexus wants to merge 6 commits into
Open
bluednanexus wants to merge 6 commits into
bluednanexus wants to merge 6 commits into
Conversation
Kodem Security Scan Summary
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During the Symlinks 2.0 download investigation (APPS-4449), we confirmed a DXDA throughput bottleneck when a large file is represented by a single platform part. This is especially relevant for Symlinks 2.0 where, if there are no real multipart boundaries, the platform may expose one fake full-file part.
Solution
This PR addresses the large-part bottleneck by splitting oversized regular file parts into persisted sub-chunks and downloading those chunks in parallel, with a per-file in-flight cap to avoid overloading a single file. Progress is tracked at chunk level in the stats DB, so downloads remain fully resumable after interruptions. Once all chunks of a part complete, we run a finalize integrity check; if it fails, both the parent part and its chunk rows are reset for safe retry. This keeps throughput high on large files while preserving correctness and recovery behavior.
PR test summary (10 GB download validation)
Scope:
file-JBPx6X00Gk6PKj4y5PpY5JPB (/ingress/readable_random_10gb.bin): symlink file with a single 10 GB part.
file-JBP8y780vB5y7v8JFXFjV18J (/ingress/writable_10gb.bin): symlink-backed file with full parts metadata.
Measurement method:
Per-file active download window computed from stats DB as:
start = first completed chunk/part timestamp
end = last completed chunk/part timestamp
duration = end - start
Results:
file-JBPx6X00Gk6PKj4y5PpY5JPB:
Start: 2026-09-23 08:42:57 UTC
End: 2026-09-23 08:50:33 UTC
Duration: 456.57s (vs. 3,341s before implementation)
file-JBP8y780vB5y7v8JFXFjV18J:
Start: 2026-09-23 08:50:20 UTC
End: 2026-09-23 08:56:41 UTC
Duration: 380.868 s (vs 369s before implementation)
Conclusion:
Both download paths completed successfully under the new flow.
The regular single-part 10 GB case and the symlink-with-parts 10 GB case both showed stable completion with expected per-file timing captured from persisted progress state.