Plan: make MIKE IO documentation excellent for AI agents - #995
Open
ecomodeller wants to merge 2 commits into
Open
Plan: make MIKE IO documentation excellent for AI agents#995ecomodeller wants to merge 2 commits into
ecomodeller wants to merge 2 commits into
Conversation
Documentation is currently written for humans reading a rendered website. The people writing MIKE IO scripts increasingly are not reading it — an agent is, and it gets whatever the model already believes about the API plus whatever it can reach. That produces scripts that fail on the first run, and the cost lands on the user, not on us. The plan targets that directly: measure what agents actually get wrong against a task suite, then fix only what the measurement shows. Agent guidance ships inside the wheel so it is always version-matched and reachable without a network, and its snippets are executed in CI so it cannot rot into confidently teaching an API that no longer exists. Lives outside roadmap/features/ so it stays out of the generated public roadmap.
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
roadmap/ai-agent-documentation.md— an execution plan, no code changes.Why
Our documentation assumes a human reading a rendered website. Increasingly the thing writing MIKE IO scripts is an agent, and it works from whatever the model already believes about the API plus whatever it can reach at the time. When those disagree with reality the script fails on the first run, and that cost lands on the user rather than on us. In advisory work the script is the deliverable, so this is not a niche concern.
Two things follow, and the plan is built around them.
Measure before writing. It is easy to write an agent guide against imagined failures. Phase 1 builds a task suite of realistic advisory work and scores generated scripts by whether they actually execute against
tests/testdata/, with a cheap rubric pass on top for idiom. Everything written later has to trace back to an observed failure — including the length budget, which is decided by comparing variants rather than by taste.Ship it in the wheel. Agent guidance goes in
src/mikeio/AGENTS.mdas package data. A file insite-packagesis always version-matched to the installed code and needs no network, which matters on the locked-down machines a lot of advisory work happens on. The docs site serves the same bytes for agents that fetch.The failure mode of any such file is rotting into confidently teaching an API that no longer exists — worse than having none, and shipped to every user. So every fenced Python block in it is executed by a test in CI: a recipe that breaks fails on the PR that broke it.
Also found while scoping
docs.ymltriggers on pull requests withpaths: ['docs/**'], but the push trigger also listssrc/mikeio/**. A PR touching only source therefore does not rebuild the docs — so a change that breaks a user-guide code block passes review and only fails after it is merged. Quarto executing the user guide is our correctness guarantee for the largest body of example code in the project, and it currently does not gate PRs. One-line fix, filed as its own issue.Separately,
generate_llms_txt.pyalready emits frontmatter descriptions intollms.txt, but only 2 of 19 pages have one, so the published index is a bare list of titles.Scope
Consumer agents only — agents writing user scripts. Agents working inside this repo are already served by
CLAUDE.md,CONTEXT.mdandadr/. No MCP server: everything it would serve is static text, and it would be a service to operate.Work items are filed as individual issues.