Skip to content

[AutoDiff] docs: add AD design notes - #559

Closed
aornugent wants to merge 2 commits into
traitecoevo:developfrom
aornugent:ad-design
Closed

aornugent wants to merge 2 commits into
traitecoevo:developfrom
aornugent:ad-design

Conversation

@aornugent

Copy link
Copy Markdown
Collaborator

Summary

This is a design-only PR — it adds a roadmap for the plant SCM automatic-differentiation
work, with no code changes. It proposes how to land the exact trait/birth-rate gradients
prototyped in #553 with surgical changes to the existing plant components plus a small
generic surface added to odelia, rather than as a plant-private AD stack.

The prototype in #553 is validated (gradients to ~1e-13) and remains the specification and
the regression oracle throughout. This roadmap is about how to merge that capability with
lower long-term technical debt.

What's included

A set of design docs (under notes/):

  • README.md — orientation, reading order, and a glossary for readers new to the model
    or to AD.
  • ad-infrastructure-design.md — the design: three layers (odelia / plant / UX), the
    surgical changes, the two workflows (resident vs. invasion), and the four replay levels.
  • ad-r-interface.md — the R/C++ boundary (why XAD types are awkward through Rcpp, the
    "only doubles cross" invariant) plus user stories for each persona.
  • ad-issues.md — the work breakdown: scoped items across odelia / plant / R-boundary /
    prototypes, with dependencies and a critical-path build order.

Key design positions

  • odelia owns the generic AD mechanism (tape, reverse-mode Jacobian, the functional
    seam, IFT/analytic edges) it already almost provides; plant keeps only irreducible
    physiology. plant already LinkingTo: odelia.
  • Surgical, not parallel. The Patch is already an odelia::ode::Solver System, and
    SCM::run_mutant() is already the frozen-schedule replay — the gradients differentiate the
    runs we already have, not three bespoke *_emergent.cpp engines. The <T,E,S> third
    axis collapses to the existing <T,E> with the scalar carried by the strategy/environment.
  • One replay primitive — "record where the adaptive stepper/interpolator/quadrature
    placed their nodes, replay on them fixed with the active scalar" — which removes the
    stand_stage_history machinery and reuses basic_interpolator<S>/QK<S>.
  • Reverse mode for the metrics×traits Jacobian; forward mode stays plant-local at the
    leaf optimizer.

Relationship to #553

This does not replace #553 — it plans how to land its result. The recommendation is to not
merge the spike as-is
but to codesign the odelia surface first (with the spike's validated
Jacobians as the AD-vs-AD oracle), then rebuild the plant path onto it strategy by strategy.
The docs are explicit about scope and risk, including the TF24 leaf-optimizer cross-sensitivity
and the TF24f long-horizon stiffness limit.

Scope note

The design spans both plant and odelia; the odelia-side work items are scoped here for
coordination even though they land in traitecoevo/odelia. Feedback on the direction is
welcome before implementation begins.

🤖 Generated with Claude Code

Import the AD infrastructure design docs (design, work items,
R-interface, and README) into notes/ad-design.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UwX91k99M8mhKzb4SJa3ej
@aornugent aornugent changed the title docs: add AD design notes [AutoDiff] docs: add AD design notes Jul 4, 2026
@aornugent

Copy link
Copy Markdown
Collaborator Author

@dfalster - I've made a start on the odelia changes. PRs to follow.

@dfalster

dfalster commented Jul 5, 2026

Copy link
Copy Markdown
Member

Excellent plan @aornugent! This level of planning is exactly what's needed. In retrospect, I should have done this earlier, but it's the right detail to inject now. Appreciate you taking the lead on that. FYI, I'll have to take a break from development for the next two weeks. Very glad if you want to pursue this In the meantime.

Also, we've discovered some instability in TF24, which causes issues for automatic differentiation. For the AD work, we should assume that instability will be resolved by improvements in the TF24 model, so of now just work with cases where it performs adequately.

@dfalster

dfalster commented Jul 5, 2026

Copy link
Copy Markdown
Member

Happy for you to merge this.

Refresh notes/ad-design with the latest design docs from the plant-dev
workspace, including the new ad-record-replay.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B4FyyLScweWwLXYNKKmTfz
@aornugent aornugent closed this Jul 16, 2026
@aornugent
aornugent deleted the ad-design branch July 16, 2026 02:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants