fix(install): repair smart-explore runtime provisioning across IDEs

[rymalia]

Summary

Fix the shared installation path so smart-explore is actually functional after
installing claude-mem, and fail loudly when its runtime cannot be provisioned
instead of reporting success with silently broken tools.

Previously, the installer could report success while leaving tree-sitter-cli
without its platform executable. The MCP tools were registered and callable,
but structural parsing silently failed:

• smart_search scanned files but returned zero symbols
• smart_outline could not parse supported TypeScript files
• smart_unfold could not locate symbols

This affected fresh installations across IDEs, including the current 13.9.1
release.

Root cause

Three independent failures combined to make smart-explore appear operational
while returning empty results.

1. Plugin dependencies were installed with lifecycle scripts disabled.
   tree-sitter-cli requires its top-level install script to download the
   platform-specific executable, so only the npm package metadata was present.

2. On Windows, even when tree-sitter.exe was present, the runtime resolver
   checked only the extensionless tree-sitter name and discarded it, falling
   through to a bare-PATH lookup that usually fails.

3. Tree-sitter attempted to write compiled grammar artifacts under the user's
   cache directory, which managed environments may expose as read-only.

The parser caught Tree-sitter errors and returned an empty result, which hid
all three failures.

Changes

• Run only the required top-level tree-sitter-cli install script after the
  script-suppressed dependency installation.
• Validate the installed runtime with a bounded tree-sitter --version
  execution instead of checking only whether a file exists.
• Reject missing, non-executable, corrupt, or non-responsive binaries.
• Resolve tree-sitter.exe before the extensionless binary on Windows so a
  downloaded Windows executable is actually used.
• Install the Codex plugin directly with codex plugin add after registering
  the local marketplace, removing the predictable Git-only marketplace upgrade failure from local Codex installations.
• Write the Codex .install-version marker into the plugin cache only after
  runtime provisioning succeeds, so it is a truthful "runtime ready" signal
  rather than one seeded optimistically before installation.
• Set XDG_CACHE_HOME to a writable process-local directory for Tree-sitter
  queries.
• Add regression coverage for:
  • missing Tree-sitter executables
  • non-executable files
  • corrupt executables with invalid version output
  • Windows .exe resolution and bare-PATH fallback
  • local Codex marketplace installation and post-provision marker write
  • installation failure without runtime provisioning
  • parsing with an unwritable user cache
• Regenerate the MCP server bundle.
• Document npx claude-mem install --ide codex-cli in the README.

Scope and intentional behavior change

The runtime-provisioning and validation logic added here lives in the shared
installPluginDependencies / verifyCriticalModules path—not a Codex-only
branch. It runs for the standard npx claude-mem install flow and for repair
across supported IDEs, including Claude Code.

That is deliberate: the same silent-failure mode existed on every shared
installation path, so the fix and its validation apply everywhere.

• Before: installation completed when tree-sitter-cli lacked a working
  executable. Core memory remained functional, but smart-explore silently
  returned empty or unparseable results.
• After: a missing, non-executable, corrupt, non-responsive, or incorrectly
  installed Tree-sitter executable fails installation or repair.

smart-explore is the only feature that requires Tree-sitter; capture,
injection, recall, and the worker do not. This therefore converts some partial
successes into hard failures. That is intentional: a loud, correct failure is
preferable to reporting success with a broken installed feature, and uniform
behavior across IDEs is easier to support.

Newly blocked environments are platform-availability cases:

• musl/Alpine, where the downloaded Linux prebuilt requires glibc and cannot
  execute;
• platforms or architectures outside tree-sitter-cli's prebuilt matrix.

Deferred follow-up, not included here: if unsupported environments become a
significant support burden, Claude Code could warn and continue without
smart-explore while Codex remains strict. The paths are separable, so that can
remain an isolated policy change.

Also out of scope: parser.ts still degrades a runtime Tree-sitter query
failure to an empty result (#2910's third point). This PR fixes install-time
provisioning and validation so the binary is present and valid; surfacing a
runtime query failure in the tool result instead of returning empty is a
separate change.

Relationship to existing work

This change shares a root cause with an actively tracked cluster and supersedes
parts of two open PRs — flagged so it isn't reviewed in isolation.

• Root cause: v13.5.6: smart_search/smart_outline/smart_unfold return 0 symbols on clean Windows install - tree-sitter-cli binary never downloaded (--ignore-scripts), #1247's extensionless existsSync check still unfixed #2910 (smart-explore returns 0 symbols—tree-sitter-cli
  binary never downloaded under --ignore-scripts, plus the Windows
  extensionless resolver). Tracking issue tracking: plugin runtime deps missing in distributed bundles — zod external, tree-sitter-cli not downloaded, hooks crash #2964 consolidates a broader cluster
  (also zod / shell-quote / grammar deps); this PR addresses its
  tree-sitter slice; refs tracking: plugin runtime deps missing in distributed bundles — zod external, tree-sitter-cli not downloaded, hooks crash #2964 but does not resolve the full tracker.
• Overlaps fix(setup): restore tree-sitter CLI on clean Windows installs (closes #2910) #2918 (install path)—same installPluginDependencies
  (setup-runtime.ts) and getTreeSitterBin (parser.ts). Different strategy:
  fix(setup): restore tree-sitter CLI on clean Windows installs (closes #2910) #2918 drops --ignore-scripts globally; this PR keeps it and runs only
  tree-sitter-cli's vetted install script, avoiding re-introduction of the
  tree-sitter-swift postinstall hang that --ignore-scripts was added to
  prevent ([plan-04] Installer Failure Transparency — cross-IDE error taxonomy + 12×4 test matrix #2379). This PR adds --version validation fix(setup): restore tree-sitter CLI on clean Windows installs (closes #2910) #2918 lacks and folds in
  fix(setup): restore tree-sitter CLI on clean Windows installs (closes #2910) #2918's Windows tree-sitter.exe resolver fix, so the two are not competing
  half-fixes.
• Overlaps fix(codex): install local plugin cache during setup #3066 (Codex path)—same codex plugin add install and the same
  removal of the Git-only marketplace upgrade call. This PR is a superset:
  it adds tree-sitter runtime provisioning and writes .install-version into
  the Codex plugin cache after provisioning succeeds (rather than seeding the
  marketplace root before plugin add, so the marker cannot outlive a failed
  provision). Verified end-to-end that Codex supplies the cache root to the hook
  environment, so the cache-side marker is sufficient (see Verification). Credit
  to @Retengart for identifying the missing-marker case.

Verification

• Affected and installer-related tests pass (codex-cli-installer,
  install-non-tty, setup-runtime, parser, verify-critical-modules).
• Full TypeScript typecheck passes.
• Full build and bundle regeneration pass; the regenerated mcp-server.cjs
  contains the parser fix (Windows .exe resolution and process-local cache).
• Fresh isolated Claude Code installation completed successfully through the
  shared installPluginDependencies path and provisioned a working
  tree-sitter 0.26.9 executable.
• Verified end-to-end on a fresh local install + new Codex session: the
  installer wrote .install-version only to the Codex plugin cache (the
  marketplace root stayed marker-free), and the session did not emit
  "runtime not yet set up". Repeated with the Claude-cache marker temporarily
  removed to rule out the fallback path — the warning stayed absent, confirming
  Codex supplies the installed plugin root to the hook environment.
• Injectable Windows resolver tests (tree-sitter-bin.test.ts) cover
  tree-sitter.exe preference, bare-PATH fallback, and non-Windows resolution.
• isInstallCurrent regression test covers a same-version cache with a marker
  but an unusable tree-sitter-cli binary (the stale-cache repair path).
• The existing marketplace peer-dependency fallback was exercised and completed
  successfully.
• prepublishOnly runs the full build before publishing; the npm CLI is
  rebuilt from src/npx-cli/index.ts into dist/npx-cli/index.js. dist is
  included in the published package but intentionally gitignored, so no
  additional committed installer artifact is required.
• A fresh MCP process successfully completed smart_search (symbols found),
  smart_outline (structure returned), and smart_unfold (implementation
  returned).
• Worker restarted successfully and reports version 13.9.1.

Test-suite note

A full npm test run was attempted locally. Unrelated listener-based tests
cannot bind port 0 inside the managed workspace sandbox and time out during
setup. All tests covering the changed installation, parser, and platform paths
pass; CI should run the unrestricted full suite.

User impact

Codex users no longer receive a successful claude-mem installation with
silently broken smart-explore tools. Installation provisions and validates the
required runtime and fails clearly if it cannot.

Claude Code and other IDE users receive the same strict validation through the
shared install and repair paths. Environments without a working
tree-sitter-cli binary—musl/Alpine and architectures outside the prebuilt
matrix—now fail loudly instead of completing with smart-explore silently
broken.

This behavior is intentional; see "Scope and intentional behavior change."

[greptile-apps]

Greptile Summary

This PR makes smart-explore runtime setup explicit and validated. The main changes are:

• Provisions tree-sitter-cli after dependency install with scripts disabled.
• Validates the Tree-sitter executable with a bounded --version check.
• Fixes Windows resolution for package-local tree-sitter.exe.
• Runs Tree-sitter queries with a process-local writable cache directory.
• Installs the Codex plugin cache with codex plugin add and writes .install-version after runtime provisioning succeeds.
• Adds tests for installer, Codex, parser, and binary-resolution paths.

Confidence Score: 4/5

The changes appear merge-safe, with focused installer and parser updates backed by targeted regression coverage.

The touched paths have tests covering provisioning, validation, Windows binary resolution, Codex installation, and writable cache behavior.

T-Rex Logs

What T-Rex did

• T-Rex performed a base verifyCriticalModules check and confirmed PASS for the missing executable, non-executable (EACCES), and corrupt stdout scenarios, and verified that the tree-sitter CLI binary was not exported.
• T-Rex validated post-install behavior by triggering a head check that failed with Post-install check failed: unresolvable modules: tree-sitter-cli executable, then provisioned a working binary returning tree-sitter 0.99.1.
• T-Rex produced a proof for a posted P1 finding, and the review comment contains the validation details.
• T-Rex extended tests to include Windows resolver tests, but parser.test.ts failed because parseFile() returned [] under unwritable XDG_CACHE_HOME, despite Windows resolver tests passing.
• T-Rex captured codex install marker progression, showing before state with base run and missing cache markers, and after state with version 88.0.0, explicit marker checks, and a README instruction.

_{Ran code and verified through T-Rex}

Comments Outside Diff (1)

1. General comment

   Head parser still returns no TypeScript symbols under the unwritable-cache runtime scenario

   • Bug
     • The new targeted test smart file parser > extracts TypeScript symbols when the user cache is not writable fails on head: parsed.symbols.map(...) is [] instead of ["smartExploreProbe", "ProbeClass"]. This means the claimed end-to-end behavior that smart-file-read can parse supported TypeScript files after installation was not demonstrated in the actual runtime path.
   • Cause
     • The runtime query path in src/services/smart-file-read/parser.ts still degrades any tree-sitter query execution failure to an empty match map. In this environment the installed tree-sitter-cli binary cannot execute (GLIBC_2.39 requirement), so even with the new process-local XDG_CACHE_HOME, parseFile() silently returns a symbol-less folded view. This violates the requested contract that parsing succeeds under the bad user-cache setup.
   • Fix
     • Ensure the installed/provisioned tree-sitter runtime is executable on the supported target environments, or use a compatible bundled CLI/native path. Keep the process-local cache override, but add install/runtime validation that catches an unusable tree-sitter-cli binary before smart_search/smart_outline/smart_unfold degrade to empty symbol output.

   _{Ran code and verified through T-Rex}

_{Reviews (6): Last reviewed commit: "fix(install): repair stale-cache bypass ..." | Re-trigger Greptile}

[greptile-apps]

T-Rex pricing update — T-Rex was free through June 2026. Effective July 1, 2026, T-Rex adds 2 credits on top of the standard 1-credit review (3 total). T-Rex settings

[rymalia]

Thanks for the re-review. Both findings concern the explicitly documented, out-of-scope runtime silent-empty behavior rather than the changed installer paths.

On head 672f8199, the project's ubuntu-latest CI successfully ran the full bun test suite, including the unwritable-cache parser test. T-Rex's parse failure is caused by its Tree-sitter executable requiring unavailable GLIBC_2.39.

In the supported install flow, that executable fails the bounded tree-sitter --version validation and installation aborts. The T-Rex MCP harness bypassed installation and launched the server directly from the checkout, so it does not invalidate the installer guarantee.

We are intentionally retaining the parser assertion without a skip guard: silently passing when Tree-sitter cannot execute would conceal resolver and runtime regressions. Surfacing runtime query failures from MCP tools remains the separately documented follow-up from #2910 point 3.

[Retengart]

I prepared a clean sync branch for this PR since the current head is conflicting with main and I do not have UpdatePullRequest permission on this PR body.

Branch: https://github.com/Retengart/claude-mem/tree/pr3095-main-sync
Commit: f003fcb0 (Merge main into smart-explore runtime provisioning)

What it does:

• merges current origin/main into this PR head;
• resolves the only merge conflict in the generated plugin/scripts/mcp-server.cjs by rebuilding the generated bundles, not by hand-editing the bundle;
• preserves the fix(install): repair smart-explore runtime provisioning across IDEs #3095 source changes while incorporating the 13.9.2 provider/docs/test changes from main.

Verification on the prepared branch:

• npm run build
• npm run typecheck:root
• bun test tests/install-non-tty.test.ts tests/integration/codex-cli-installer.test.ts tests/setup-runtime.test.ts tests/services/smart-file-read/parser.test.ts tests/services/smart-file-read/tree-sitter-bin.test.ts tests/cli/verify-critical-modules.test.ts tests/gemini_provider.test.ts --timeout 15000

One important PR-body fix is still needed before merge: GitHub currently treats #2964 as a closing reference because the body contains the phrase should not close #2964 wholesale. Suggested replacement:

**tree-sitter slice**; refs #2964 but does not resolve the full tracker.

That keeps #2964 open as the broader tracker while this PR covers only the tree-sitter slice.

[greptile-apps]

Enable after provisioning
enableCodexPluginConfig() writes ~/.codex/config.toml before installCodexPluginCache() has installed the plugin cache and validated tree-sitter-cli. If plugin add or runtime provisioning fails, the installer returns failure but leaves claude-mem@claude-mem-local enabled in Codex config, so new Codex sessions can try to load a plugin cache that is missing or not runtime-ready. Move config enablement until after the cache install/provision step succeeds, or roll it back on failure.

Artifacts

Repro: focused Bun test that drives installCodexCli with isolated HOME and failing fake codex plugin add

• Contains supporting evidence from the run (text/typescript; charset=utf-8).

Repro: test output showing installer failure and enabled claude-mem@claude-mem-local in isolated config.toml

• Keeps the command output available without making the summary code-heavy.

_{Ran code and verified through T-Rex}

[rymalia]

Thanks @Retengart — resolved the conflict on our side (rebased onto main 13.9.2, regenerated mcp-server.cjs from source rather than hand-editing, same approach as your sync branch) and applied your #2964 body fix. PR is green/mergeable again. And thanks for the catch on the closing keyword - whoops.

[thedotmack]

Merged into the community-edge branch — claude-mem's bleeding-edge release line. Closing here since your change now lives on that branch. See the branch model (community-edge → core-dev → main): https://docs.claude-mem.ai/branches . Thanks for the contribution! 🙏