Repository navigation
Archiving: finish all traffic fetches before writing a repository #103
Description
Why
archive_repo currently writes views before fetching clones and snapshots, so a later API failure can leave an incomplete refresh.
Scope and acceptance criteria
Separate collection/preparation from publication for one repository: fetch all four endpoint payloads successfully before its first archive write. Preserve continuation to other repositories. This prevents writes on fetch failure, not a claim of an atomic six-file transaction; #14 still owns per-file atomic writes.
Starting points
- src/traffic_archive/cli.py archive_repo
- tests/test_cli.py
Validation
Inject a failure at each endpoint and compare all existing file bytes/directory entries; assert no new repository directory on failed collection and unchanged successful output.
For Python changes, run the focused tests and python -m pytest with the same environment. Use synthetic fixtures, mocked network calls and temporary files; preserve standard-library-only runtime dependencies and supported Python versions. Distinguish actual local results from CI and untested steps.
Contributing
Difficulty: intermediate. Read CONTRIBUTING.md, post a short plan and coordinate assignment before substantial work. Keep one problem per PR. Proposed paths are new files. Resolve interfaces/dependencies with cited issues before implementation; do not duplicate their scope. Never commit credentials or live traffic data.
Activity
rajshadow commentedon Oct 22, 2026
Hello there, I would be interested in claiming this. Here is my understanding of the issue and the proposed fix + validation. Open to adjusting the plan before implementation.
Current flow:
The issue is that if say, views and clones succeed but referrers call fails, we have an intermediate state w.r.t freshness of the various histories. The process would mark the repo as skipped while the archive has been partly updated.
Success criteria:
Proposal :
Reorganize the flow so that all the reads are attempted first, and upon successful fetch for all, perform the necessary metric level merge and save:
Test plan:
Out of scope:
HafidIdrissi commentedon Oct 22, 2026
Thanks @rajshadow — assigned to you. Your analysis, proposed ordering and test matrix match #103; no conflicting implementation is currently open. All four traffic fetches must succeed before the first archive write or creation of a new repository archive directory. If any fetch raises the handled API error, existing files/directory entries must remain byte-for-byte unchanged, an absent archive must remain absent, and the CLI must continue to the next repository with the existing summary/exit behavior.
Please preserve successful merge rules, the six output files, formats and totals. As you correctly noted, this is protection against collection failure, not an atomic six-file publication guarantee; write-failure transactions and #14's per-file atomicity remain separate. #125 is reviewed/approved but not merged yet and validates response containers in api.py; keep that scope separate and update your branch when it lands. Existing CLI changes in #77 also remain separate.
Use synthetic rows, mocked failures at each of the four endpoints, disposable existing archives and absent-output cases. Verify a later repository still succeeds and compare successful output against the existing behavior. Run focused CLI tests and python -m pytest with the same environment; retain supported Python versions and standard-library-only runtime dependencies. Report exact OS/Python, commands/results and skipped checks separately from CI. Open one focused PR with Closes #103.
fix(cli): finish traffic collection before writing archives
fix(cli): finish traffic collection before writing archives