Skip to content

APPS-4463: Improve DXDA parallelism for large single-part downloads - #110

Open
bluednanexus wants to merge 6 commits into
masterfrom
APPS-4463
Open

bluednanexus wants to merge 6 commits into
masterfrom
APPS-4463

Conversation

@bluednanexus

@bluednanexus bluednanexus commented Sep 18, 2026 •

Copy link
Copy Markdown

During the Symlinks 2.0 download investigation (APPS-4449), we confirmed a DXDA throughput bottleneck when a large file is represented by a single platform part. This is especially relevant for Symlinks 2.0 where, if there are no real multipart boundaries, the platform may expose one fake full-file part.

Solution

This PR addresses the large-part bottleneck by splitting oversized regular file parts into persisted sub-chunks and downloading those chunks in parallel, with a per-file in-flight cap to avoid overloading a single file. Progress is tracked at chunk level in the stats DB, so downloads remain fully resumable after interruptions. Once all chunks of a part complete, we run a finalize integrity check; if it fails, both the parent part and its chunk rows are reset for safe retry. This keeps throughput high on large files while preserving correctness and recovery behavior.

PR test summary (10 GB download validation)

Scope:

  • file-JBPx6X00Gk6PKj4y5PpY5JPB (/ingress/readable_random_10gb.bin): symlink file with a single 10 GB part.

  • file-JBP8y780vB5y7v8JFXFjV18J (/ingress/writable_10gb.bin): symlink-backed file with full parts metadata.

Measurement method:

Per-file active download window computed from stats DB as:
start = first completed chunk/part timestamp
end = last completed chunk/part timestamp
duration = end - start

Results:

  • file-JBPx6X00Gk6PKj4y5PpY5JPB:
    Start: 2026-09-23 08:42:57 UTC
    End: 2026-09-23 08:50:33 UTC
    Duration: 456.57s (vs. 3,341s before implementation)

  • file-JBP8y780vB5y7v8JFXFjV18J:
    Start: 2026-09-23 08:50:20 UTC
    End: 2026-09-23 08:56:41 UTC
    Duration: 380.868 s (vs 369s before implementation)

Conclusion:

Both download paths completed successfully under the new flow.
The regular single-part 10 GB case and the symlink-with-parts 10 GB case both showed stable completion with expected per-file timing captured from persisted progress state.

@kodem-security

kodem-security Bot commented Sep 18, 2026 •

Copy link
Copy Markdown

Kodem Security Scan Summary

Status Check Issues by Severity Link
✅ Passed Code Weakness Security Report 🟧 M: 3    Link to Check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant