Skip to content

docs: parallelize Read the Docs HTML build - #5888

Draft
njzjz-bot wants to merge 6 commits into
deepmodeling:masterfrom
njzjz:docs/sphinx-j2
Draft

docs: parallelize Read the Docs HTML build#5888
njzjz-bot wants to merge 6 commits into
deepmodeling:masterfrom
njzjz:docs/sphinx-j2

Conversation

@njzjz-bot

@njzjz-bot njzjz-bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Override only the Read the Docs HTML build command.
  • Run Sphinx with two workers via -j 2.
  • Preserve the existing HTML builder, language, doctree directory, source directory, and output directory.
  • Keep this change independent from the Linkcode experiment.

Performance measurement

This PR intentionally contains only the Sphinx worker-count change so its Read the Docs preview measures parallelism independently.

Baseline Read the Docs build 33680413:

  • total build: 1889 seconds
  • dependency installation: 303 seconds
  • Sphinx HTML: 1524 seconds
  • Sphinx worker count: 1

Both performance PRs use upstream commit 83e7245646185a2de8f7d6277edc4f64b4f500f0 as their base.

Testing

  • Parsed .readthedocs.yml with PyYAML
  • Ran the configured Sphinx arguments against a Sphinx 9.0.4 smoke project with -j 2
  • ruff check .
  • ruff format .
  • Repository commit hooks, including YAML validation and Prettier

Coding agent: Codex
Codex version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning effort: xhigh

Summary by CodeRabbit

  • Documentation
    • Improved Read the Docs HTML builds with explicit Sphinx settings.
    • Standardized English-language output and build directories.
    • Limited documentation builds to two workers for more predictable performance.

Override the Read the Docs HTML command to use two Sphinx workers while preserving the existing builder, language, doctree, and output settings.

Coding-Agent: Codex
Codex-Version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning-Effort: xhigh
@dosubot dosubot Bot added the Docs label Jul 21, 2026
@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: b0811fff-fac2-4366-bc43-6956763d9821

📥 Commits

Reviewing files that changed from the base of the PR and between 83e7245 and 4cf6179.

📒 Files selected for processing (1)
  • .readthedocs.yml

📝 Walkthrough

Walkthrough

The Read the Docs configuration adds an explicit Sphinx HTML build command using two workers, English language settings, doc as the source, doc/_build/doctrees for doctrees, and $READTHEDOCS_OUTPUT/html for output.

Changes

Read the Docs build

Layer / File(s) Summary
Configure the Sphinx HTML build
.readthedocs.yml
Adds explicit Sphinx options for concurrency, language, doctree storage, source input, and HTML output paths.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: parallelizing the Read the Docs HTML build.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.36%. Comparing base (cc689a7) to head (c43224b).
⚠️ Report is 16 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5888      +/-   ##
==========================================
- Coverage   79.47%   79.36%   -0.11%     
==========================================
  Files        1072     1085      +13     
  Lines      125041   126405    +1364     
  Branches     4536     4592      +56     
==========================================
+ Hits        99373   100319     +946     
- Misses      24044    24431     +387     
- Partials     1624     1655      +31     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@njzjz
njzjz marked this pull request as draft July 21, 2026 10:47
njzjz-bot added 3 commits July 25, 2026 09:56
Run the custom Sphinx command from the documentation directory so the source checkout does not shadow the installed package. Add a 45-minute timeout with a forced-kill grace period to turn stalled parallel builds into explicit failures.

Coding-Agent: Codex
Codex-Version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning-Effort: xhigh
dargs.sphinx.DargsDomain advertises parallel_read_safe but does not
implement merge_domaindata, so Sphinx refuses to combine the per-worker
inventories and -j aborts the build. The domain's only state is a flat
targetid -> (docname, objtype) map, so merging is a dict update
restricted to the documents the worker read.

Verified on a minimal two-page project using dargs directives: -j 2
fails with the same NotImplementedError without the shim and succeeds
with it.
@njzjz

njzjz commented Jul 27, 2026

Copy link
Copy Markdown
Member

Rebased onto master and made -j 2 actually work in eed8f9cfc.

The RTD build on this branch was failing at 655 s with:

NotImplementedError: merge_domaindata must be implemented in
<class 'dargs.sphinx.DargsDomain'> to be able to do parallel builds!

dargs.sphinx returns {"parallel_read_safe": True} from its setup, but DargsDomain inherits Domain.merge_domaindata, which raises. Sphinx then tears down the worker pool and the failure surfaces as a secondary KeyError in sphinx/util/parallel.py. doc/conf.py now installs a merge_domaindata on DargsDomain when it is still the base implementation; the domain's only state is a flat targetid -> (docname, objtype) map, so merging is a dict update restricted to the documents that worker read. Worth upstreaming to dargs, at which point the guard makes the shim a no-op.

Verified on a minimal two-page project using .. dargs:: directives: python -m sphinx -j 2 reproduces the exact NotImplementedError without the shim and reports build succeeded with it.

For context on why this matters beyond this PR: the docs/readthedocs.org:deepmd check is red on a large number of open PRs, and every one I sampled is the same 40-minute RTD wall-clock limit — builds 33625117, 33625500, 33626766 all ended at 2409-2411 s with the sphinx step still running. So this PR is the fix for that whole class of failures, not just a speed-up.

Coding-Agent: Codex
Codex-Version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning-Effort: xhigh
@njzjz

njzjz commented Aug 1, 2026

Copy link
Copy Markdown
Member

The build process is not sped up.

@njzjz-bot njzjz-bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed draft head 347ea8d8daa927302c7bf903df4d260c0d98c380 with three independent full subagent reviews. The Sphinx parallel-domain shim and Read the Docs custom-build syntax are sound, and the preview build succeeds. One P2 configuration issue remains inline: the command timeout starts too late and exceeds the platform's observed hard build limit, so it cannot provide the graceful termination described by the comment. Some C++ checks on this draft are currently failing or cancelled, with no evidence tying them to this documentation-only diff. Because this PR was opened by the active njzjz-bot account, this is a comment-only review rather than a self-request-changes event.

The quota is about to reset, so I am concentrating the remaining token budget on these reviews.

Coding agent: Codex
Codex version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning effort: xhigh

Comment thread .readthedocs.yml Outdated
Coding-Agent: Codex
Codex-Version: codex-cli 0.144.6
Model: gpt-5.6-sol
Reasoning-Effort: xhigh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants