[Bug]: untracked files are invisible to the graph — discovery is git ls-files only, so a newly written file is never indexed, even by a full rebuild

[tbgihu]

Summary

File discovery is sourced entirely from git ls-files and git diff, so a file that
git has never been told about is invisible to the graph — not just until the next
build, but permanently, including through a full rebuild. For an AI assistant that
writes a new file and immediately asks the graph about it, this is the common case:
the assistant's own new code is the one thing the graph cannot see.

Where

code_review_graph/incremental.py, both discovery paths:

• Full build — collect_all_files() takes its candidate list from
  get_all_tracked_files(), which is git ls-files. Untracked files never become
  candidates, so full_build() never parses them. Worse, full_build()'s stale
  purge computes existing_files - current_abs from that same list, so an untracked
  file that somehow reached the graph is actively deleted from it on the next
  rebuild.
• Incremental update — incremental_update() gets its change set from
  get_changed_files(), which is git diff --name-status <base> --. git diff
  does not report untracked paths, so the file never enters all_files and is never
  parsed.

get_staged_and_unstaged() does see untracked files (git status --porcelain),
but it is only reached by detect-changes, update --brief and the review tools as
a fallback when get_changed_files() returns empty — never by the code that
actually writes nodes into the graph.

Reproduction

In any repo with at least one commit:

code-review-graph build                      # note the file count
printf 'def marker_fn():\n    return 1\n' > new_module.py   # do NOT git add
code-review-graph update                     # "No changes detected"
code-review-graph build                      # file count unchanged
# query_graph_tool(query_type="callers_of", target="marker_fn") -> not_found

git add -N new_module.py                     # intent-to-add, no content change
code-review-graph build                      # file count +1, marker_fn resolves

git add -N changes nothing about the file's content — only git's index — which
isolates the cause to discovery.

Impact

The MCP integration is wired to be live: a PostToolUse hook runs
code-review-graph update after every edit, and serve --auto-watch watches the
tree. Both are defeated for new files. The assistant is told the graph is current,
asks callers_of / references_to / file_summary about code it wrote seconds
ago, gets zero, and concludes the symbol does not exist.

This is the same failure #987 describes, approached from the other end: #987 makes
the empty result admit its uncertainty, which is a real improvement, but the result
is still empty. The proposal here is that it should not be empty.

Proposed fix

Make discovery tracked ∪ untracked-but-not-ignored, which is exactly
git ls-files ∪ git ls-files --others --exclude-standard. --exclude-standard
applies .gitignore, .git/info/exclude and core.excludesFile, so build output,
node_modules, virtualenvs and the tool's own .code-review-graph/ stay out — the
existing DEFAULT_IGNORE_PATTERNS pass then applies on top as it does today. Unlike
git status --porcelain, ls-files --others lists individual files rather than
collapsing a wholly-untracked directory to dir/, so no second walk is needed.

Two call sites:

• collect_all_files() — union the untracked list into candidates (keep the
  filesystem-walk fallback for when both are empty, i.e. not a git repo).
• get_changed_files() — union the untracked list into the git branch's result, so
  the incremental path sees new files. A file git has never seen is, by any useful
  definition, a change.

--recurse-submodules is not valid with --others, so the submodule option applies
only to the tracked call.

Deletion caveat — also fixed in the local patch

Once untracked files are indexed, deleting one is invisible to the incremental path
in the same way: git diff never names it (never tracked) and --others no longer
names it (gone), so remove_file_data() is not called and the node lingers until the
next full build, whose stale purge does clear it. serve --auto-watch catches it via
filesystem events, so this only affects hook-only setups (the PostToolUse hook
calling update, or any other caller of incremental_update() with no watcher
running).

Update: this is no longer an open gap in my local patch. incremental_update() now
resolves the graph's recorded file list against disk before its early return: for
every file the graph already knows about that this update's own diff/untracked scan
didn't name, it stats the path and calls the existing remove_file_data() if the
file is gone — the same mechanism the watcher's on_deleted uses, just triggered by
a filesystem stat instead of a filesystem event. Verified round-trip on a scratch
repo: create an untracked file → update → node appears; delete it, no rebuild →
update → node is gone. No extra git calls, no watcher required.

Environment

• code-review-graph 2.3.6 (installed as a uv tool), macOS 15, Python 3.13
• Verified still present on main as of today: collect_all_files() and
  get_changed_files() are unchanged in this respect in 2.3.8 and later.

[PushpakAg]

It seems like the issue is that untracked files aren't being indexed by the graph, even with a full rebuild. I'd check how collect_all_files() and incremental_update() handle file discovery, especially regarding untracked files.

[tirth8205]

@PushpakAg those are the two functions, yes. @tbgihu already named them in the report along with the exact lines and a fix, so there is nothing left to locate here.

Confirming the report against staging, since it deserves an actual answer rather than agreement. I ran the repro:

Full build: 1 files, 2 nodes, 1 edges     # committed seed.py only
<write new_module.py, do not git add>
Incremental: 0 files updated, 0 nodes, 0 edges
Full build: 1 files, 2 nodes, 1 edges     # file count unchanged
query callers_of marker_fn -> not_found

So it reproduces exactly as filed, including the part that makes it worse than a staleness bug: a full rebuild does not recover it either. --others and --exclude-standard appear nowhere in the codebase, so discovery really is tracked-only on both paths.

@tbgihu the proposed fix is the right one. git ls-files --others --exclude-standard unioned into collect_all_files() and get_changed_files(), with --recurse-submodules staying on the tracked call only, is what I would have asked for. Two things I want to see in the PR:

• The stale purge in full_build() has to consume the same union it discovers from, or an untracked file gets indexed by one build and deleted by the next.
• The deletion round-trip you describe in your update. Resolving the graph's recorded file list against disk before the early return is the right mechanism, but it is a per-update stat over every known file, so it needs a bound or a cheap short circuit on large graphs. Say in the PR what it costs on a 100k-node graph.

Please open it against staging. The local patch you describe sounds like the whole fix.

On the interaction with #987: that is the other half, and it is open. #987 withdraws the real-absence claim when the answer could be untracked source, which is right regardless of this issue, because a graph can always be behind. But as you say, the honest empty answer is still an empty answer. This issue is the one that makes it non-empty, and it is the more important of the two.

[PushpakAg]

Got it, thanks for confirming the issue and the proposed fix. I'll make sure the stale purge in full_build() uses the same union for consistency.