docs: diagnose why the witan-code inject-context block is not acted on

[blarghmatey]

Closes tk-diagnose-why-the-witan-code-inject-context-block-440e8d.

The previous re-measurement of PR #163 covered 1.5 days and 66 injections,
where the pre-existing rate predicts ~0.6 calls. It ruled out a large
improvement and nothing else. This re-runs it over 34 days and 100 sessions
and then answers the question the task was actually filed for: what stops
the block from being acted on.

Does the line work?

No detectable change, and the window bounds rather than settles it. Session
adoption is 3/51 → 6/100 (Fisher p = 1.00) and the per-injection ToolSearch
attempt rate the rewrite targeted is 6/586 → 8/951 (p = 0.79) — point
estimates, not a demonstrated decrease. The 95% interval on post-#163
adoption is [2.8%, 12.5%], so a large improvement is excluded and a small
one is not. Raw calls rose 5 → 17 (p = 0.18), entirely from two sessions
using the tools more.

Why not?

Not deferral. code_* and task_* arrive deferred in the same 97 of 100
sessions, at the same ToolSearch cost. 89 of those sessions pay it for
task_* and 6 for code_* (p < 0.001).

Substitution. Across 27,277 tool calls the Grep and Glob tools are used
zero times; all searching is Bash running rg/grep/find — 6,432
invocations in 94 of 100 sessions, 674 of them definition lookups, against
3 code_find_definition calls. So the block's "use them instead of grep"
names a behavior that does not occur, while the habit it means to displace
lives in a different tool and is endorsed by a standing system-prompt
instruction. The code_* tools that do get used (14 of 17 calls) are the
inventory and cross-repo ones rg cannot answer; code_callers and
code_impact, the payoff the block's own template names, were called zero
times in 951 injections.

Those two findings are the ones the diagnosis rests on, and neither depends
on the underpowered pre/post comparison.

The task's leading hypothesis — that witan-council's In-Flight Branch block
wins by naming a specific thing — is retired as confounded: 75 of 100
sessions began with a prompt already naming a task slug, and 53 of those
claimed. Among the 25 whose prompt named no task, block-only uptake is 5/25
(CI [8.9%, 39.1%]) — small enough to leave the rate loosely pinned, but
enough to show the headline 59% is not the block's own effect.

What this does not do

Rewrite the block. The task said not to spend the 600-char budget
(test_inject_context_block_stays_small) until the mechanism had an answer;
the answer is that the mechanism is not wording, so a better-worded
exhortation is still an exhortation. Follow-ups are filed separately.

Docs-only. just docs-check and just docs-build pass; nav wired into
docs/internals/index.md and zensical.toml.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ki2iWyNzEzD1c4tJM7WYPA

[Copilot]

🟡 Changes recommended

Central conclusions overstate the measured evidence, and the provided reproduction script omits decisive calculations.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Documents why the witan-code context block sees low adoption and proposes follow-up experiments.

Changes:

• Adds a 100-session uptake analysis.
• Documents substitution and deferral hypotheses.
• Adds the diagnosis to internal documentation navigation.

File summaries

File	Description
zensical.toml	Adds navigation entry.
docs/internals/index.md	Links the diagnosis.
docs/internals/design/witan-code-uptake-diagnosis.md	Adds findings, methodology, and reproduction steps.

Review details

Suppressed comments (4)

docs/internals/design/witan-code-uptake-diagnosis.md:29

• The task_* comparison shows that agents can pay the loading round trip, but it does not isolate deferral’s effect on code_*: 75/100 prompts already named a task and task operations have no Bash substitute. The proposed non-deferred experiment in item 2 is what can distinguish loading cost from substitution, so substitution should remain a hypothesis rather than the demonstrated binding constraint.

**Deferral is not the binding constraint.** `code_*` and `task_*` arrive
deferred in the same 97 of 100 sessions, at the same `ToolSearch` cost.
89 of those sessions pay that cost for `task_*` and 6 for `code_*`.

docs/internals/design/witan-code-uptake-diagnosis.md:34

• The later table identifies 6,432 as all Bash invocations using rg/grep/find/fd; only 674 are classified as definition lookups. Here, “those” reads as 6,432 definition lookups and overstates the directly substitutable behavior by nearly 10×.

runs 6,432 of those.

docs/internals/design/witan-code-uptake-diagnosis.md:154

• The 89% figure is the share of sessions issuing a task_* ToolSearch, not tool uptake across task_*, workflow_*, and memory_*; the table explicitly omits session counts for the latter two. Limit this claim to the measured task-search metric instead of attributing 89% uptake to all three families.

Deferral is free when nothing else can answer (`task_*`, `workflow_*`,
`memory_*`: 89% uptake) and decisive when something can, because the
substitute is already loaded, always works, and is named approvingly by a

docs/internals/design/witan-code-uptake-diagnosis.md:192

• This snippet cannot reproduce the report’s decisive substitution measurements: it never inspects Bash inputs, classifies rg/grep/find/fd or definition lookups, or computes their per-session counts. It also does not count ToolSearch queries for workflow/memory families. Include the omitted classifiers (and output for the low-frequency code_* tools) or avoid calling this the “whole rig,” because the 6,432, 674, 94/100, and family-comparison figures currently cannot be verified from it.

   Worth checking against `_resolve_branch`'s fall-through to main.

## Reproducing

• Files reviewed: 3/3 changed files
• Comments generated: 2
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.