Skip to content

Latest commit

 

History

History
449 lines (315 loc) · 82.1 KB

PROGRESS.md

File metadata and controls

449 lines (315 loc) · 82.1 KB

Semantix Progress

This document records current state, gaps, priorities, near-term evidence, and released milestone history.

Current objective

2026-11-09 0.13.0 release scope: dense Arrow recall vectors convert in batches, candidate display highlights are calculated on first read, and empty exclusion rules skip frontend Vault enumeration. In an interleaved same-model comparison on a copied 644-note / 15,313-chunk index, hot API P50/P95 fell from 171/226 to 118/161 ms (fast), 632/780 to 576/727 ms (balanced), and 765/937 to 713/865 ms (high quality). All full responses matched across 12 real and 12 labeled synthetic queries per mode. Full Engine suite: 106 passed. Required plugin checks passed. Controlled Obsidian 1.15.0 whole-note timing used a 120 ms fixture delay, with zero new dispatches after cancellation. A larger first recall window changed ranking evidence and stays experimental. FP16 remains an independent candidate. Human real-Vault relevance and model-backed editor-to-paint latency remain open. Method, reproduction and raw timings.

2026-10-30 0.12.0 release scope and acceptance: version metadata was synchronized from 0.11.7 through npm run version -- minor. The numeric 0.12.0 tag triggers the repository's GitHub Release workflow; the remote workflow and published asset checks are authoritative for publication. Update the companion Engine to this release for vault_indexed_paths support; the installed real-Vault plugin remains a separate deployment step.

Publication verified: release commit 5ae25bbb62f0cf99f7befb181f4c6ff8247c1075 and numeric tag 0.12.0 were atomically pushed and checked against remote refs. GitHub CI run 40980721611 and Release run 40980721846 both completed successfully. Release 0.12.0 was published at 2026-10-30 08:37:40 Asia/Shanghai. All three downloaded assets matched the validated local SHA-256 values, and the downloaded manifest reports 0.12.0. Release notes include companion Engine upgrade instructions. Verification artifacts are under C:/Users/seeyo/Documents/Codex/2026-10-30/semantix-release-0.12.0. This publication did not install the release into the real Vault; the tag remains on the release code commit, with publication evidence recorded in a later documentation commit.

  • Index consistency: added Vault-scoped /index/paths with vault_indexed_paths capability, strict inventory response validation and failure visibility. Startup/reconnect/exclusion changes reconcile actual paths with successful mtime/size signatures persisted per Vault/engine address. Saves serialize; acknowledgments retain the submitted signature instead of accidentally certifying newer edits. No whole-Vault re-embedding on each health poll. First use without saved records may queue every eligible note.
  • Full rebuild/clear: rebuild updates valid files before bounded obsolete-path deletion, works when all local notes are absent/excluded, retains failed notes' old indexes and retries them, and reports cancelled, cleanup, FTS or connection failures as incomplete. Clear/rebuild/reconciliation coordinate with incremental writes within this plugin instance. Failed clear preserves pending work; successful clear drops only pre-boundary work, preserves later events and clears old cards. A later edit or startup/reconnect can index notes again. Older engines without a usable path inventory cannot complete this rebuild workflow and remain visibly unverified.
  • Exclusions/settings/context: one matcher serves indexing, concrete request exclusions and rendered cards; missing source files are also filtered before rendering, and confirmed deletes remove already displayed cards; queued work rechecks before read/submission. Auto-trigger off now blocks typing/cursor/selection updates while retaining file-switch and explicit scans. Search settings invalidate immediately and coalesce for 250ms before refreshing the retained scope. Insufficient text marks old cards stale, and source tooltips/accessible labels expose a captured excerpt. Repeated settings changes preserve an interrupted whole-note scope.
  • Discover: whole-note union preserves Related membership across final result limits, preventing reclassification into Discover. Shared 16/20 rerank budgets, model selection, ranking gates and weights are unchanged. No global whole-note semantic MMR or new relevance improvement is claimed; candidate coverage and human-rated extension usefulness remain open evidence work.
  • Host-found status repair: settings now request metrics from display() rather than relying on an unused onOpen() hook, and accepted index-status snapshots update cached counts. A late metrics response cannot overwrite a newer count or another endpoint. Status slots and storage descriptions update without rebuilding focused inputs; unavailable counts are not presented as zero/not-indexed.
  • Documentation: README, architecture, validation gates and ADR-0013 describe current behavior and limitations. Corrected token header, engine acquisition/auto-start prerequisites, source-anchored DB path and explicit insert-link behavior. Earlier audit and release entries below are historical evidence; real-Vault installation is separate from the release.

Current validation: focused consistency pytest 10/10; locked dependency sync and full engine pytest 97/97 passed without pytest warnings (7.84s). Plugin lint, production build, type checking restricted to Obsidian, indexing, new index-consistency/radar-UX, aggregate Radar/settings/adaptive, API-health, lifecycle, card-hover and release-contract checks passed. Existing unchanged service-manager results from the implementation pass remain applicable. New regression commands are included in both CI and release workflows. Backend tests use temporary databases and deterministic model seams; the later host-found fixes affect frontend code only, so the unchanged 97-test engine result is reused. No models were downloaded and no production index operation was issued.

Controlled host acceptance completed on Obsidian 1.14.5 in a disposable Semantix-Test-Vault, using a stdlib loopback engine on port 18753 with synthetic documents in memory. The human handled Obsidian's first-use trust prompt. The tested build's plugin assets matched the local build. Observed behavior: startup indexing; offline add/edit/rename and obsolete-path cleanup; correct first-display four-note index count; actual cursor/selection/typing with auto-trigger off produced no additional queries (2 before/2 after), while file switch and explicit scan still queried; a result-limit change retained whole-note scope; Private/** expanded to Private/Hidden.md, removed its card and reduced the index from four notes to three; a synthetic missing-source index record was not displayed and was cleaned up; whole-note partial progress and cancellation; insufficient text retained visibly stale cards, exposed the captured excerpt and sent no query; and card navigation opened the expected synthetic source. A deliberately delayed 100-part scan stopped after 40 requests, with the incomplete-result notice retained. This delay is fixture control, not a performance measurement.

The host run exposed and closed the initial-settings metric lifecycle defect and missing-source stale-card display gap above; focused regressions, lint/build/type checks and a host reload verified the repairs. The destructive-clear UI button was not invoked; deterministic Node tests cover successful/failed clear, in-flight writes and boundary events. Remaining evidence gaps: representative real-model/real-Vault relevance and latency, large-Vault reconciliation cost, changes preserving both file mtime and size, and independent-client/process coordination. Shared-shortlist Discover coverage and global whole-note semantic diversity remain explicit limitations rather than claimed improvements.

Host evidence and repeatable fixture: C:/Users/seeyo/Documents/Codex/2026-10-30/semantix-host-check/host-result.json, requests.jsonl and ACCEPTANCE.md. Cleanup was verified: the disposable Vault was normally closed, all owned fixture-engine processes stopped, port 18753 released, only its temporary Vault registration removed, original Vault open flags restored without overwriting unrelated registrations, and the original Dashboard - note window reopened. Synthetic files/backups remain available for inspection. The host run did not modify real-Vault notes or installed plugin files and preceded the release version synchronization below.

Release-specific local checks: the 0.12.0 production build, lint, Obsidian-only type check and release-contract check passed. Locked Engine dependency synchronization and all eight versioned smoke checks passed without pytest warnings. The unchanged full 97-test Engine result, Node behavioral regressions and host evidence above are reused; version synchronization changed metadata, the health version and its smoke assertion only. Release main.js/styles.css SHA-256 values are byte-identical to the accepted host assets; its manifest reports 0.11.7, while the release manifest reports 0.12.0.

Prior implementation and release records

2026-10-29 0.11.7 patch release preparation: package, plugin, engine, lockfile, README and smoke-test versions were synchronized through npm run version -- patch. This patch contains the defensive-programming convergence below. The versioned production build and release-contract check passed, as did all eight engine smoke tests and the indexing/process regressions. The preceding 95-test engine result, lint, ambient-type-restricted type check, radar, lifecycle and API-health results remain applicable to the unchanged implementation. The numeric 0.11.7 tag is the release trigger; GitHub Actions is authoritative for remote build and publication status. Real-Vault installation and Obsidian host acceptance remain unverified.

2026-10-29 defensive-programming convergence (local, unreleased): engine pytest now sets a session-temporary database path before collection, overrides inherited database paths, and provides the engine import root for independent focused runs. Session teardown releases shared application storage before deleting the temporary database and does not suppress cleanup failures. Removed the unreachable missing-jieba n-gram branch and its artificial import-failure test, unused indexing/storage forwarding or insertion methods, and SDK enumeration branches outside the declared LanceDB dependency contract. Feature construction now relies on aligned score arrays; existing pipeline validation still degrades the entire batch with an explicit warning on empty, short or overlong reranker output. A successful health request already refreshes backend activity, so the plugin no longer sends an extra ping immediately afterward. Vault scope, atomic replacement, model configuration, ranking formulas, recovery budgets, schema migration and maintenance retention are unchanged. Validation: 28 focused engine checks and the final full 95-test engine suite passed without pytest warnings; locked dependency sync, plugin lint/build, type checking without ambient Node types, API-health, plugin-lifecycle and aggregate radar regressions passed. The focused run initially exposed the missing standalone import path; the shared test setup now resolves it. No model downloads, real-Vault mutation or host UI acceptance are part of this work. Validation guidance explicitly reuses unchanged successful checks; existing CI and release gates remain in place.

2026-10-29 storage maintenance and heading-only indexing repair (local, unreleased): manual maintenance now honors the selected history retention instead of forcing zero days; both manual and scheduled optimization retain at least one day of verified versions and leave unverified files under LanceDB's normal age protection. Existing saved zero-day settings migrate to one day. A heading-only note with no body chunks is indexed through bounded overlapping pieces so its middle and tail remain available; ordinary body chunking is unchanged. Isolated temporary LanceDB checks cover current rows, FTS, vectors, Vault separation, a pre-replacement readable snapshot, bounded fallback chunks, and preservation after embedding failure. This safety window supports normal readers but does not guarantee arbitrarily long pinned snapshots or interprocess coordination. Final isolated checks used temporary databases and did not access a production Vault. An earlier unisolated suite connected to the default engine database; no file writes were observed from that run.

2026-10-28 writing-retrieval product pass (local, unreleased): the adaptive switch now invalidates stale searches and queues one refresh of the current Focus or whole-note scope in both directions, after the authoritative connection/preparation check finishes. A health capability advertises soft frequent-term discount; older/offline engines receive explicit enable_adaptive_filtering=false, and the settings drawer shows whether the saved preference is actually active. Connected supporting engines recalculate the Vault list once per connection/preference change before adaptive requests are enabled; failure stays visible and does not retry on every poll. Endpoint identity and generation guards discard stale preparation and health responses. Base/custom stopwords are described as lexical/highlight-only exclusions. Relative score traffic lights and newly emitted DEEP_ECHO were removed; older label codes remain display-compatible, while island/cross-folder copy now states the actual structural evidence. The sidebar shows the source file, heading and captured selection/line/paragraph/whole-note mode. No ranking weights or model configuration changed. Current checks: 90 engine tests, radar/settings/API-health regressions, lint and build passed; final host UI is unverified. A single frozen real-model run of the default balanced + adaptive=false path used 12 queries/28 synthetic notes in an isolated DB: Related returned all 11 strictly useful notes plus three irrelevant q06 cards, Discover returned none, and unrelated q12 returned none. This does not establish Discover extension quality or real-Vault relevance. Evidence: C:/Users/seeyo/Documents/Codex/2026-10-28/semantix-lexical-validation/default-balanced-20261028.json and .log.

2026-10-29 adaptive switch race closure (local, unreleased): a newer health check that exits disabled/loading/error/disconnected now invalidates an in-flight list calculation and settles the pending visible-scope refresh with adaptive=false. A recovered connection recomputes instead of accepting a result from before the failure, including a same-URL restart. Manual recalculation shares the generation/endpoint result guard with automatic preparation, so a late result from another endpoint cannot mark the current one ready. An older index-status snapshot cannot overwrite a newer manually accepted list, and an open settings drawer updates its visible words when the accepted list changes. Health responses use request order as well as endpoint identity; a late rejection or older same-endpoint response cannot erase newer capability evidence. Controlled concurrent Node regressions, plugin type checking, lint and build passed; the prior 90-test engine run and frozen model evidence remain applicable because engine and ranking code did not change in this closure. Obsidian host interaction remains unverified.

2026-10-28 lexical normalization/token follow-up (accepted into local implementation; unreleased): positive-strength rows normalize against the maximum raw BM25 among rows with strength at least their own; zero-strength rows never enter a denominator. Final lexical caps remain 0.25/0.5/1.0. Scoring accepts single ASCII identifiers only as complete whitespace-delimited indexed tokens, so query C supplies evidence for C but not atomic C++/C# tokens; the analyzer still recalls those collisions, and dedicated C++/C# queries remain deferred. Display keeps its two-character minimum. Adaptive extraction reuses full query identities (GPT4o, Python3, Go); existing snapshots acquire them on normal recalculation, with no automatic production mutation or index migration. Unconditional quarter/half semantic reservation was rejected after useful exact shortlist counts fell from 1 to 0 in 16/20-budget adversarial cases; radar_service.py is byte-identical to the frozen baseline and shortlist starvation remains unresolved. Full current engine verification passed 89 tests without pytest warnings in 8.93s, including the final single-letter boundary repair. Existing frontend checks remain applicable because frontend source did not change in this follow-up. Independent frozen-label real-model A/B used 12 queries, 28 synthetic notes, isolated databases, unchanged BGE/M3 and options: baseline and candidate both returned fast useful/irrelevant/missing 11/31/0 and balanced 11/3/0, with identical per-query cards and scores. This supports controlled nonregression, not relevance or speed gains; q06 still returns three irrelevant balanced cards. The final boundary repair follows that A/B but cannot affect its inputs: no query or indexed fixture token is a single ASCII letter; dedicated real-FTS regressions and the final full suite cover that branch. Evidence: C:/Users/seeyo/Documents/Codex/2026-10-28/semantix-lexical-validation. Acceptance is a bounded invariant repair; one ordinary informative match still restores full lexical credit. Production Vault/index, models, global weights, versions and deployment remain untouched; real-Vault/UI acceptance remains unverified.

2026-10-28 adaptive lexical evidence and Chinese compound refinement (unreleased; version unchanged): adaptive frequent terms remain in FTS and receive a per-candidate soft evidence discount; base/custom stopwords remain hard exclusions. After BM25 normalization, generic/function-like matches receive at most 0.25 lexical credit, adaptive content matches 0.5, and informative matches 1.0; the strongest matching term determines the factor so extra weak matches cannot dilute informative evidence. Evidence uses the selected row's indexed title/heading/child tokens, excluding directory names and expanded parent context. Raw BM25/RRF inputs remain unchanged. Chinese compound recovery now uses noun/name constituents and a bounded technical glossary rather than arbitrary 2+1 concatenation; highlight display remains separate from ranking evidence. UI copy, architecture, acceptance docs and ADR-0010 describe the distinction. Validation: 41 focused tests and all 73 engine tests passed without pytest warnings after locked GPU dependency sync; plugin lint, build, Obsidian-only type checking, radar/highlight/source regressions and whitespace checks passed. A separate read-only final reviewer accepted the current implementation. A deterministic lead probe rejected a generic-only candidate after its score changed from 0.4475 to 0.335, while retaining a topical candidate at 0.4175; this demonstrates the score contract, not real-model relevance. POS classification, the two-entry compound glossary, and full-strength credit from any informative match remain bounded heuristics. No production Vault/index, version, deployment or publication was changed; real-Vault relevance, host UI and latency acceptance remain unverified.

2026-10-28 0.11.6 sidebar follow-up (local commit, not published): the user's installed 0.11.5 saved layout contained one left and two right Semantix leaves, and the screenshot showed results only on the left. Radar UI updates previously targeted only the first loaded leaf. Results, progress, errors, cancellation and context reset now reach every loaded RadarView. Sidebar discovery uses saved view types across all workspace leaves, including deferred views; layout changes consolidate duplicates while retaining an already loaded panel in its existing position. Controlled lifecycle tests cover deferred reuse, later restored duplicates, reentrant layout events and unload during deferred loading; multi-panel radar regressions and build passed. The corrected main.js/styles.css were installed with SHA-256 verification; Obsidian was running at installation and requires plugin reload, after backing up plugin assets and workspace.json under the user's Codex backups directory. The saved workspace was not edited. Version metadata is synchronized to 0.11.6 through the patch-version script. The versioned production build, release-contract check and eight engine smoke tests passed; prior frontend lifecycle/radar/indexing/process/health, lint and type-check results remain applicable. Final host startup and visual acceptance remain unverified. The installed Vault build is the preceding 0.11.5 development patch; this local version commit does not install or publish 0.11.6.

2026-10-28 0.11.5 patch release published: the approved retrieval-quality changes include pinned M3 reranking with an explicit legacy-model option, coherent passage evidence and lower-value highlight filtering. npm run version -- patch synchronized plugin and engine metadata from 0.11.4 to 0.11.5. Versioned local checks passed: 65 engine tests without pytest warnings, lint, both plugin type-check/build paths, indexing/radar/highlight/source/process/lifecycle/API-health regressions, and the release contract. Existing frozen model/independent-review evidence remains applicable because release preparation changed version metadata and this status record only. Release commit 51638647e1258edcdacb7baf4e3a9cd0026ad33b and numeric tag 0.11.5 were atomically pushed and verified against remote refs. GitHub CI run 40653936160 and Release run 40653936204 both completed successfully. The published release contains main.js, manifest.json, and styles.css; all three downloaded SHA-256 values match the local final build, and the downloaded manifest reports 0.11.5. No real Vault installation or production-index change is part of this release.

2026-10-28 M3 reranker candidate (unreleased; no version/index/profile change): default shared reranker and pre-downloader now use official BAAI/bge-reranker-v2-m3 revision 953dc6f6f85a1b2dbfca4c34a2796e7dde08d41e, safetensors/no remote code and a 512-token pair limit. SEMANTIX_RERANKER_MODEL=BAAI/bge-reranker-base explicitly retains legacy resource compatibility. No weights or admission thresholds were tuned. On the frozen 27-query synthetic regression across four corpora, balanced useful/cards changed 39/50 to 42/45 (irrelevant cards 11 to 3), returning all four useful q01 recovery cards and removing the park result; fast was unchanged. This is a reused frozen regression, not an unseen holdout or real-Vault acceptance. Opposite-intent overtime and screen-versus-printer recommendations remain failure cases. Independent-process float32 M3-only CUDA checks used 2.271 GB allocated after load, peaking at 2.645/2.737 GB for 16/20 pairs of 505 tokens (0.623/0.735 seconds); embedding and other process memory are excluded. CPU four-thread short-pair smoke was 0.334 seconds and is not a long-input CPU throughput claim. Earlier 3.535 GB evidence included overlapping base/M3 lifetimes and is not a standalone M3 requirement. Health now exposes selected reranker identity/revision/status/device; semantic-ready ok messages explicitly disclose unavailable reranking. Permanent reranker failure stops automatic retries, and partial pre-download failure returns nonzero. Validation: locked dependency sync and all 65 engine tests passed without warnings; lint, build, plugin type checking, API-health and radar/highlight regressions passed. A separate metadata-rich independent probe retained all four useful cards while reducing balanced/high-quality totals from six to five; its opposite-intent overtime card still remained. The engine has not been installed into the real Vault or published.

2026-10-27 retrieval-quality candidate (unreleased, no version change): degree-word FTS filtering and display-only weak-modifier suppression preserve people/events, semantic input and complete shared expressions. Document aggregation keeps all final ranking evidence attached to its displayed passage. Independent synthetic evaluation rejected a separate content-only admission gate: it lost a useful paraphrase while retaining unrelated recommendations. A two-passage reranking experiment also failed to repair the known park-versus-recovery example, so the added inference budget was withdrawn. A fixed cosine-best versus RRF-best single-passage comparison supports retaining cosine-best for a fast-mode evidence correction, with no observed regression in the controlled exact-term sample. Balanced mode still rejects that useful recovery paraphrase, and same-person/different-event, opposite-intent, and low-information queries still produce unrelated cards. Independent passage-level comparison: original 12-query set balanced useful/total cards 16/22 to 16/21 (the misleading recovery card disappears), fast 21/63 to 22/62 (its passage becomes useful); separate 8-query holdout remains balanced 11/14 and fast 11/28; 3-query exact-term sample remains balanced 6/7 and fast 6/12. Independent 4-query stress remains balanced 6 useful/2 irrelevant and fast 8/8 after withdrawing the content gate. These are controlled synthetic findings, not real-Vault precision/recall acceptance; no broad quality improvement is claimed. Existing composite thresholds remain experimental heuristics, not calibrated probabilities. This candidate has not been installed into the user's Vault. Locked engine dependency sync and all 54 current engine tests passed without warnings.

2026-10-27 0.11.4 offline/repeated-sidebar release: local recovery no longer clears its failure budget after two minutes. Three automatic recovery attempts exhaust the budget and notify once; manual restart or a healthy connection permits recovery again. Pending incremental updates/deletes remain queued while offline, without file reads, submissions or progress churn, and resume on a healthy connection without releasing a full-index pause. Sidebar activation shares one pending operation, consolidates restored duplicate Semantix leaves, and cannot recreate a view after unload. Health responses after stop/unload cannot resume indexing. Validation: zero-warning lint, production build, plugin type checking without ambient Node types, lifecycle/indexing/process/health/radar regressions and release-contract checks passed. Controlled tests include extended offline time, stop-then-manual-restart, interrupted batches, retained revisions and concurrent sidebar opens. Read-only inspection of the user's saved Vault found Semantix 0.11.3 and one saved Semantix view; the intermittent duplicate-icon report was not reproduced in the live host. Version metadata is synchronized through the project patch-version script for 0.11.4. This build has not been installed into the user's Vault or accepted through a final Obsidian host UI run.

2026-10-07 0.11.3 patch release: publish the approved minimal-sufficient-implementation repairs, Semantix display-name cleanup, and the pending Moment dependency override. Version metadata is synchronized through npm run version -- patch. Local release validation covers lint, both plugin type checks/build, indexing/radar/process/health regressions, release-contract checks, and all 48 engine tests using temporary storage. Obsidian host UI and real-model relevance/latency remain unverified. Release commit/tag a259be4 / 0.11.3 were pushed and GitHub Release workflow 37646737934 completed successfully. The release is published with main.js, manifest.json, and styles.css; downloaded SHA-256 values match the local final build for all three assets.

2026-10-07 minimal sufficient implementation repair (unreleased; version remains 0.11.2): full indexing and incremental sync now share document preparation and batching (25 notes / 150,000-character target; a larger indivisible note is isolated). Both submit empty documents for atomic removal of old chunks. Incremental read failures no longer block healthy notes; failed batches remain queued, and acknowledgments preserve newer update/delete revisions. Full-index submission failure is reported separately from user cancellation. Permanent embedding initialization failure returns an explicit error health state, retains the failure message in the client/settings, and does not launch a duplicate sidecar. Stopword snapshots serialize per storage instance and atomically replace the disk file before updating memory; failures propagate to the API while preserving prior disk/memory snapshots. Ranking bonuses and RRF defaults now use the existing configuration without changing values. Removed the forwarding DatabaseService facade, unused internal aliases/helpers, and the unused global clear operation; persistent view IDs and old wire-label display compatibility remain intact. Architecture/testing docs now match token headers, implemented routes, global maintenance/metrics, current labels, and the desktop-only release boundary. Validation: all 48 engine tests passed without warnings after locked dependency sync, including real temporary LanceDB empty-document/Vault isolation, concurrent stopword persistence, write-failure/API behavior and configuration-effect checks. Zero-warning lint, both plugin type checks/production build, indexing/radar/highlight/source/process/health regressions, release-contract and whitespace checks passed. The indexing regression is included in CI and release gates. No production DB or real Vault was used; final Obsidian host UI, real-model relevance/latency, mobile and power-loss acceptance remain unverified.

2026-10-07 display-name cleanup (unreleased): the ribbon tooltip, native hover-preview source, and localized sidebar title now use Semantix without the legacy Radar name. Internal view identifiers and the radar icon remain compatible. Zero-warning lint, production build, plugin type checking without ambient Node types, radar/highlight regressions, and whitespace checks passed. Obsidian host hover behavior has not been reverified.

2026-10-03 dependency review fix (unreleased): override Obsidian's pinned transitive moment@2.29.4 with 2.31.0 for GHSA-4p3w-j4w9-5jqw and refresh only the Moment lockfile entry. A clean npm ci --ignore-scripts resolves moment@2.31.0; production dependency audit reports zero vulnerabilities. Full audit still reports eight other dependency findings (three moderate, five high), outside this fix. Zero-warning lint, production build, plugin type checking without ambient Node types, and release-contract checks passed. Runtime locale detection continues to use Obsidian's host-provided Moment through the external obsidian import; this override does not upgrade the host library. No Obsidian host QA or external review-bot rerun was performed.

2026-10-03 0.11.2 patch release: user requested publishing the retrieval correctness and interaction fixes below. Version metadata is synchronized through the repository patch-version script. Final-build host QA remains unverified; no manual QA result has been supplied. This release does not claim broad accuracy or latency gains. Local lint, both type checks/build, radar/process/health regressions, release-contract checks and all 41 engine tests passed for 0.11.2. Remote workflow and published-asset verification follows the tag push.

2026-10-02 retrieval and interaction optimization (unreleased; version remains 0.11.1):

  • Correct CrossEncoder output to raw logits before the existing normalization, including CPU fallback. Remove assignment to the read-only CrossEncoder device property. Successful reranking now selects only from its scored shortlist; failures explicitly report degradation. Ranking weights and thresholds remain unchanged.
  • Refill recall with unseen documents for at most four rounds per channel, preserving Vault and path exclusions. A failed optional refill retains collected candidates. FTS-only candidates receive measured cosine evidence; failure of all eligible channels returns an error. Candidate paths accompany snippets during reranking.
  • Invalidate pending Focus results immediately on edits/cursor changes, serialize transport requests and discard superseded waiters, deduplicate identical whole-note query parts, render explicitly partial scan results, and stop dispatching further parts after cancellation. Already running inference is not cancelled. Stabilization retains only current evidence; unchanged card payloads avoid rebuilding DOM.
  • Pass the matched source text to navigation and preview. Locate only a unique complete normalized match in the current Markdown; changed or ambiguous text reports that exact positioning is unavailable.

Current verification: all 41 engine tests passed after the CPU fallback fix; frontend radar/highlight/note-scan/source-location regressions, zero-warning lint, both build/type checks, and release-contract validation passed. Automated storage tests use real temporary LanceDB with deterministic vectors and cover refill limits, failure preservation and Vault isolation. No release or installation into the user's real Vault was performed.

Evidence and limits:

  • Read-only real-index inspection found 594 notes / 14,852 chunks. One note occupied 68 of the first 80 Docker FTS rows, leaving only five documents. A three-query intermediate real-model A/B probe expanded candidate counts from 5/39/58 to 45/56/77, with warm elapsed times increasing from 65.7/108.6/159.7 ms to 170.4/146.6/215.3 ms. These single warm measurements preceded the final path-enriched rerank input; they demonstrate recall/cost tradeoffs, not final-build latency or broad relevance gains.
  • Four synthetic real-model probes returned the expected topic for Docker, gardening and cooking, and no result for an unrelated orbital query. This is a controlled sanity check, not labeled real-Vault accuracy. Cached model loading reported unexpected position_ids buffers; automated pytest completed without that model-loading path.
  • In disposable Obsidian 1.13.7 with a controlled loopback backend, the QA report confirmed connected/degraded states, partial cards, Stop scan, cancellation, completed results and multiline source navigation to line 7. This host run preceded the final DOM reuse and duplicate-query refinements; those have automated coverage. The final-build host rerun was blocked by automatic approval review of the fixture-copy/start command (blocked by policy), so it remains unverified. A fresh visual review, final real-model latency distribution and a labeled cross-topic real-Vault evaluation remain open.

2026-09-30 0.11.1 review fixes: replace Node-type-dependent port probing with a minimal typed interface and narrow unknown errors before inspecting their code. Remove the unused shell-based Python version validator; Windows owned-process shutdown now executes taskkill with argument arrays and shell disabled, and background process windows are hidden. Markdown enumeration remains limited to full indexing with exclusion rules, and clipboard access remains a user-triggered Vault ID write. CI and release checks now reject lint warnings and exercise plugin types without ambient Node declarations plus process/health regressions. Current validation: zero-warning lint, both plugin type-check configurations, production build, radar/process/health regressions, release contract, and all 34 engine tests passed. A controlled real Windows/Python sidecar also verified startup, direct taskkill shutdown of the owned process, and listener cleanup; no Vault notes were used and the Obsidian UI was not reloaded for this patch. Community capability notices for local process management and Vault enumeration may remain; these are required by the documented workflows. The community review bot's new result remains external evidence.

2026-09-30 0.11.0 scope: release the engine-owned lexical filtering, per-card highlight evidence, truthful adaptive-stopword statistics, and display-only highlight quality refinements. The prior eight-query real-model comparison did not justify changing global ranking weights. Full backend tests (34), plugin lint/build, and radar/highlight regression passed before release. After backing up and loading the build in Obsidian 1.13.7, live recalculation reported 594 notes and a 179-note cutoff, with four adaptive words. A real query returned complete 树莓派/阿里云 terms; a generic query returned cards without weak-word highlights. The plugin remained connected with one Radar view after cleanup. This validates the revised UI/contract on the real Vault, not broad relevance gains.

2026-09-29 lexical filtering and highlighting: function-word filtering now lives in the engine; each request carries the adaptive-filter toggle and custom stopwords. An empty filtered query skips FTS while retaining the original semantic query. Cards carry filtered terms present in their own snippet, and the UI no longer selects query words by length. English token boundaries prevent substring highlights; retained cards without current evidence lose old highlights, and whole-note merges retain the winning card's evidence. Existing indexes need no rebuild. Older engines without matched_terms display plain snippets. Validation: 34 engine tests, frontend radar/highlight regression, lint, build, and whitespace checks passed with no test warnings. Adaptive document-frequency thresholds and ranking weights remain unchanged. Follow-up real-Vault acceptance verified request filtering settings, echoed context IDs, and backend-provided terms rendered as highlights in Obsidian 1.13.7 after backing up and reloading the approved build. A 593-document / 14,851-chunk snapshot independently reproduced exactly four adaptive words at the 30% threshold (178 documents). Eight exploratory real-model probes compared lexical filtering and weights: reducing lexical weight to 0.10 did not resolve generic-query noise, and raising reranker weight to 0.60 displaced a directly targeted note from the top four. Retain current weights pending labeled cross-topic evaluation. This is a small technical-topic probe set, not a relevance or latency benchmark.

Measure real-Vault relevance and latency for the retrieval changes before further ranking or performance tuning.

2026-09-24 0.10.0 scope: publish the sidecar lifecycle repair, lexical recall fixes, and sidebar/whole-note interaction work together. Plugin and engine versions are synchronized by npm run version -- minor; release checks verify both. Controlled Obsidian UI and automated checks support the interaction changes, while real-model relevance, end-to-end speed, touch interaction, and large-note latency remain follow-up evidence work.

2026-09-24 retrieval and interaction refinement: FTS tokens now include note title and heading ancestry as well as chunk text, so title-only queries can enter lexical recall after the affected notes are reindexed. Lexical normalization now retains positive evidence for a single hit or tied scores. Query stopword filtering now segments once instead of twice. The sidebar highlights every matched keyword consistently and rejects a response when no matching active Markdown view remains. Synthetic LanceDB regression, engine tests, plugin lint/build pass; real-Vault relevance and end-to-end latency gains are still unmeasured. A 300-note synthetic projection benchmark found no query speedup and surfaced future score-column compatibility warnings, so the column projection was reverted.

2026-09-24 sidebar follow-up: failed searches now show a retry action and mark retained cards stale; changing notes resets that state. Keyboard focus opens the contextual preview, touch pointers expose the link action, focus outlines and reduced-motion styles are present, and score dots are described as ranking strength rather than probability. Settings connection checks restore the button on failure and avoid hardcoded engine version/language. Whole-note scans search bounded text parts with progress, reject results if the note changes during scanning, and merge results after all parts succeed. A cached-model probe reproduced the old tail omission (512-token maximum; two long inputs differing only at the end had cosine 1.0). Controlled frontend tests cover tail inclusion, result merging and failure state. In a disposable Vault on Obsidian 1.13.7, a loopback synthetic service verified the visible failure/retry flow, keyboard focus preview, score explanation, scan progress, and seven distinct note queries including the tail marker; the result merged to one card. This is UI/contract evidence, not model relevance evidence. Real-model ranking, touch/mobile interaction, and large-note latency remain open.

2026-09-23 lifecycle repair and live diagnosis: the Vault had both obsidian-semantix 0.7.0 and semantix 0.9.3 enabled. Both register semantix-whisperer-view; the older plugin won and the current plugin failed during onload before engine startup. The older plugin also pointed to a missing D:\code\Semantix\backend. After disabling the old plugin and reloading the current one, the managed process listened on port 8000 and /health returned ok with engine 0.9.3 on CUDA. Disabling the current plugin removed the listener and process; enabling it restored them. The current plugin also now prefers the existing engine virtual environment over bare uv, reports startup errors, and only stops its own process on force restart. Previous-session orphans rely on the engine parent watchdog; old PID-file and broad port-kill code was removed.

2026-09-24 port handling: a non-Semantix response on the configured endpoint is treated as a conflict. If the configured local port is occupied, startup asks the OS for a free loopback port, saves the selected URL, and starts there. A real Node listener returned {"status":"ok"} on port 8000 during an Obsidian restart; the plugin saved port 53362 and that endpoint returned engine 0.9.3 ok on CUDA. Obsidian exit removed the managed listener; the temporary listener was stopped, the original localhost:8000 setting was restored, and a final launch again returned ok on port 8000. Only the current semantix plugin remains enabled.

Implemented baseline

  • Standard Obsidian Root-as-Plugin architecture with companion calculation engine under engine/.
  • Eliminated loose legacy facades and purged unused dependencies (markdown, beautifulsoup4).
  • Vault-scoped LanceDB storage, incremental file events, adaptive full-index batches, and immediate FTS rebuild.
  • Hybrid vector/FTS recall, document aggregation, Related ranking, and diverse Discover selection.
  • Header-aware chunking, BGE query prefixing, snippets, highlighting, and adaptive stopwords.
  • Search/context freshness guards and bounded sidecar recovery.
  • Obsidian-compatible release boundary: root metadata, direct plugin build output, exact-version tags, and three-file Release assets.
  • CI runs plugin lint/build/release validation and engine pytest via uv.
  • Closed P0 gaps: Vault scope enforcement (A1), non-destructive rebuild (A2), concurrent edit retention (A3), PID ownership verification (A5), truthful reranker degradation without synthetic scores (B4).
  • Closed P1 gaps: BM25 score preservation on overlapping chunks (B1), decoupled document & chunk RRF scoring (B2), Chinese word-level FTS via dedicated tokens column (B3), tag & link parity (B5), stopword query segmentation (B6), header ancestry tracking (B7), truthful indexing failure notification (C1), stale search invalidation on note switch (C2), result stabilizer card expiration and exclusivity (C3), non-destructive partial settings refresh (C4), native markdown link generation (C5), label translations for RELEVANT and SHARED_CONCEPT (C7).
  • Interaction & workflow enhancements: Native Page Preview (hover-link) integration, tab/split navigation, explicit insert-link action with Mod+Enter keyboard access, selection-first context querying, and a dedicated focus/selection scan command.
  • UI & interaction implementation: Radar cards show a title, insert-link button, two-line snippet with lexical highlighting, and at most one structural evidence badge. Heuristic score tiers/dots are no longer displayed. The custom hover preview shows source context; normal click opens the note and attempts reliable passage location, while Shift+click opens a new tab. Hover scheduling uses a 200ms delay; host interaction and layout quality still require Obsidian acceptance.
  • Settings page architecture & interaction overhaul: Consolidates settings into 4 cohesive cards (Recommendation & Interaction, Vault Index & Scope, Service Engine & Connectivity, Storage Maintenance & Diagnostics); Mode-driven conditional branching (Local sidecar vs Remote server); Dependent typing debounce control; Integrated stopwords and path exclusion drawers; Non-blocking two-step armed confirmation for destructive actions (rebuild and clear); Input focus preservation via slot-based updates.
  • Code Quality & Architecture Governance (2026-09-18):
    • Purged obsolete /search/semantic route, semanticSearch client method, and dead wire models.
    • Aligned core domain concept from legacy Whisperer to RadarEngine (src/core/radar.ts) and RadarView (src/ui/radar-view.ts), while preserving WHISPERER_VIEW_TYPE layout compatibility.
    • Aligned ranking configuration SSOT class name from RankingProfileV1 to RankingConfig.
    • Decoupled DatabaseService God-facade into explicit domain singletons (storage, index_service, radar_pipeline).
    • Standardized engine data directory resolution to prevent process CWD drift.
    • Unified infrastructure terminology across settings and i18n ("Semantix Engine" & "Sidecar" mode).
    • Flattened built plugin output directly to root (main.js, styles.css) aligning with Obsidian community plugin standards.

Correctness and Reliability Gaps (Resolved)

The table below records closure of the earlier enumerated audit findings. It is historical evidence for those specific contracts, not a claim that all current product or reliability gaps have been resolved. Current changes and outstanding limitations are recorded above and below.

Priority Gap Completion evidence Status
P0 An unavailable reranker produces zero logits that normalization treats as real evidence (B4). RerankerService.predict_scores returns None when loading/unavailable; FeatureBuilder falls back to sem_norm without synthetic logit. Verified in engine/tests/test_radar.py. Resolved
P0 Settings clear/rebuild actions omit vault_id, selecting all-Vault delete path (A1). ApiClient.clearIndex mandates vault_id; engine/main.py rejects unscoped clear requests with HTTP 400. Verified in engine/tests/test_smoke.py. Resolved
P0 Rebuild clears valid index before replacement succeeds (A2); sync acknowledgements discard in-flight edits (A3). Rebuild triggers incremental atomic note replacement without wiping valid documents; SyncManager tracks in-flight revision numbers and preserves edits. Verified in npm run build and src/core/sync.ts. Resolved
P0 PID-file cleanup checks only numeric PID without ownership verification (A5). Force restart now stops only the process handle created by this plugin instance; broad PID-file and port-based kills were removed. tests/service-manager.cjs covers the managed-process path. Resolved for managed processes; previous-session orphan handling relies on watchdog
P1 Overlapping vector + BM25 chunk discards lexical score (B1). _fuse_and_aggregate preserves fts_scores map and attributes lexical score correctly. Verified in engine/tests/test_radar.py. Resolved
P1 Represented chunk score compared against accumulated doc RRF (B2). best_chunk_rrf tracks chunk-level RRF independently from accumulated document score. Verified in engine/tests/test_radar.py. Resolved
P1 Chinese FTS lacks word-level segmentation in LanceDB (B3). Added fts_tokens column pre-tokenized via jieba.cut_for_search; queries segmented via jieba.cut. Verified in engine/tests/test_radar.py. Resolved
P1 Tag & link metadata disparity between index and query snapshot (B5). Context query normalizes tags (strips #, includes frontmatter) and exact-matches resolved links. Verified in engine/tests/test_radar.py and src/core/context.ts. Resolved
P1 Heading ancestry stack miscalculates level hierarchy (B7). split_into_chunks maintains level-aware heading stack popping <= level. Verified in engine/tests/test_chunker.py. Resolved
P1 Full indexing failure still reported as complete (C1). Tracks failed paths across batches, checks FTS rebuild result, and displays truthful status notice. Verified in src/core/sync.ts. Resolved
P1 Stale search responses render after file switch (C2). RadarEngine increments search counter on note switch and validates active view file path and echo context ID. Verified in src/core/radar.ts. Resolved
P1 Stabilizer score-margin retains expired cards and breaks stream exclusivity (C3). ResultStabilizer retains only cards present in the current response and refreshes their evidence; Discover strictly excludes Related items. Verified by frontend regression. Resolved
P1 Settings page full re-render interrupts user text input (C4). Status banner, vault index, and engine status update dedicated slots in-place; typing focus is preserved. Verified in src/settings.ts and npm run lint. Resolved
P1 Insert link uses simple wikilink and can target wrong file (C5). Calls app.fileManager.generateMarkdownLink(file, activeView.file.path). Verified in src/ui/radar-view.ts and npm run build. Resolved
P1 Missing label translations for RELEVANT and SHARED_CONCEPT (C7). Added i18n keys and switch branches in radar-view.ts and popover-preview.ts. Verified in npm run build. Resolved

The previously stale API and retrieval prose was removed during Harness consolidation. Wire facts now route to code/OpenAPI and ARCHITECTURE.md.

Open product decisions

Resolve Q1-Q2 in ARCHITECTURE.md before claiming mobile support or enforcing version compatibility. Q3 degradation behavior is recorded in ADR-0008 in DECISION.md.

P1 evidence work

  • Establish a fixed query corpus and record encode, recall, rerank, and total latency together with relevance outcomes.
  • Validate structured title/tag/header signals before adding or retuning weights.
  • Define and test the accepted mobile/private-remote support level.

Deferred until evidence exists

  • Knowledge-graph visualization.
  • Backend-owned distributed or long-lived indexing jobs.
  • New ranking abstractions or dependencies without a demonstrated limitation in the current pipeline.

Current limitations

  • Full indexing is frontend-driven; closing Obsidian resets in-memory progress.
  • First model load and large Vault indexing depend on local hardware and model cache state.
  • Filesystem reconciliation uses mtime/size signatures; same-stat external edits require explicit rebuild. Controlled disposable-host flows passed as recorded above; large-Vault reconciliation cost remains unmeasured.
  • Discover shares the reranked candidate shortlist with Related, and whole-note merging does not run global semantic MMR. Controlled regressions do not prove real-Vault extension usefulness.
  • Release tags alone do not prove remote publication, asset integrity or installation. Inspect the remote CI/Release result and three downloaded assets for the selected release before reporting publication success; host evidence remains scoped to the tested paths and build.
  • Release tags use the exact numeric manifest version; GitHub Actions builds the three Obsidian assets from that tag.

Next order

  1. Build a fixed real-Vault query corpus and measure relevance plus encode, recall, rerank, and total latency.
  2. Extend the accepted synthetic host flows to representative real-model inputs and measure large-note scan latency and large-Vault reconciliation cost separately.
  3. Resolve the open mobile and version-compatibility decisions before broadening support claims. Reranker degradation is already specified by ADR-0008.

2026-09-14–15 深度评估:架构、算法与交互

本节保留当时的检查结果和路径,属于历史记录;现行实现、已修复状态与剩余缺口以上方当前状态和 ARCHITECTURE.md 为准。

结论与证据范围

当前架构适合继续演进:本地 Sidecar、统一模型服务、Vault 字段隔离、文档原子替换、混合召回和双流排序的边界基本合理。当前不适合宣称“可靠索引、准确推荐、真实状态反馈”已经完整达成,也不建议直接扩大功能或增加新基础设施。

主要缺陷发生在边界衔接:客户端清空范围、异步确认与新事件、全量与增量索引、召回分数与排序特征、后端互斥结果与前端稳定器,以及连接状态与任务成功状态。底层局部正确不能替代端到端不变量。

本次依据当前工作区代码与未提交 diff;原有文档/发布布局修改保留。未修改生产实现。本次新增两个无真实模型、无真实笔记读写的审计复现脚本,并更新本文件。未启动或操作真实 Obsidian;交互结论来自实现追踪和模拟宿主执行,不代表视觉/键盘/触控实机验收。

当前验证:

  • cd backend; uv run pytest:22 passed,7.72s;没有本次运行的退出清理警告。原有用例主要覆盖局部逻辑、空查询 API、基础鉴权和少数存储行为。
  • cd frontend; npm run lint、npm run build、npm run check:release:均通过,发布元数据为 0.8.0。
  • cd backend; uv run python ../scripts/audit_20260914.py:复现后端问题;真实 LanceDB 操作只发生在临时目录。全库清空调用被 mock,未删除实际数据库。
  • node scripts/audit_20260914.cjs:转译并执行真实 TypeScript,模拟 Obsidian、网络和定时器,复现前端状态问题。
  • 审计脚本中的断言验证“当前问题存在”,不是修复验收门槛;修复后应改为相反的不变量断言。临时数据库的坏向量检查发生在提交前,不证明写入中断、进程崩溃或并发写入下的恢复能力。
  • 未建立真实模型相关性语料、延迟分位数、长时间运行、移动端或多窗口宿主测试。不能据此给准确率、性能提升率或主观总分。

A. 架构与数据完整性

A1 / P0:当前 Vault 的设置操作会清空所有 Vault。 frontend/src/settings.ts 两处调用 apiClient.clearIndex();api/client.ts 不默认使用自身 vaultId,缺参时请求无 scope;backend/main.py 将缺省 scope 解释为 ALL VAULTS。客户端与 API 两端均已模拟复现。二次确认只确认该请求,并不能纠正范围错误。应将普通客户端清空绑定当前 Vault,后端禁止普通清空接口隐式扩大范围。

A2 / P0:重建采用先清空再索引,破坏失败保留旧索引的端到端保证。 设置页先 clear,后调用 startFullIndexing;如果加载、编码、请求、取消或关闭失败,旧可用索引已经消失。底层 replace_documents 的原子提交无法保护已被上层清空的数据。最小方向是保留旧索引并逐篇替换,只在成功扫描确认后清除消失/排除路径;若产品要求全仓库快照切换,再考虑 staging generation,而非现在引入复杂任务系统。

A3 / P0:同步确认会丢失请求期间的新修改。 core/sync.ts 用 Set<path> 同时表示已发出的版本和新事件,响应成功后直接 pendingUpdates.delete(path)。复现:发送旧内容→同一路径再次 queueUpdate→旧响应成功→待更新数为 0。应分离 in-flight 快照与待处理队列,或以每路径 revision 确认;失败回队必须尊重更晚的删除/重建事件。

A4 / P1:全量、增量与停机期间的同步缺乏协调。 pause/resume 已定义但没有调用者,全量索引可以和 flush 同时写同一路径;暂停也需要等待正在执行的 flush。增量队列只驻内存,启动后没有离线变更对账,文件夹 rename/delete 被 TFile 条件排除;将文件改为非 md 也可能保留旧索引。重命名到排除目录时 queueUpdate 提前返回,旧路径删除未必启动定时器。建议先修单写入顺序与启动对账,再根据必要性决定是否持久化队列。

A5 / P0:PID 文件不等于进程所有权。 core/service-manager.ts 只检查 PID 为数字就强制结束进程树,未核对可执行文件、完整命令、创建时间或父进程。旧 PID 被复用或多个 Vault 共享 backendPath 时可能针对错误进程;端口兜底的 main:app 加 uv 匹配也过宽。模拟旧 PID=4242 时直接生成 taskkill 命令,所有进程操作均被 mock,未实际终止任何进程。最小安全方向是仅管理自己创建并确认身份的进程,明确区分外部连接和自有 Sidecar。

A6 / P1:外部启动的本地服务被误判为禁用。 默认 local + autoStart=false;checkConnection 在没有受管进程时直接 disabled 并返回,不调用健康接口。模拟中健康服务调用次数为 0。自动启动发现服务已存在时也未建立外部服务状态。连接可用性应独立于进程归属;启动/停止权限再由归属决定。

A7 / P1:失败、就绪与指标没有贯通到协议。 两路召回异常会返回 [];健康接口对 embedding 加载失败仍报 loading,对 reranker 和 storage 不报告能力;RadarSearchResponse 没有 degraded/error stage。metrics?vault_id 仅文档数按 Vault 统计,搜索次数、时间、数据库大小仍全局;维护和 FTS 重建也为全局。应明确全局资源与 Vault 指标边界,不能将数据库全局维护简单描述为行级隔离操作。

A8 / P1:版本和边界验证不完整。 Schema 只检查四个扩展字段存在,未验证关键字段类型、向量维度和持久化 embedding profile;同维度模型/预处理变更可能混用索引。API 接受任意 ranking_mode、无界 top_k 和文档规模,top_k=0 又被 or 4 覆盖。请求未统一约束 deadline;同步模型调用没有显式负载上限。先增加必要输入校验、profile 校验和实际能力状态,不需要泛化插件框架。

Vault 身份局限。 桌面 ID 为名称和路径的 32-bit hash;迁移/改名会产生新身份,移动端缺 basePath 时同名 Vault 得到相同身份。远程共享 token 并不构成每 Vault 的授权边界。Q1/Q2 仍须产品确认;本次没有更改移动/兼容政策。

B. 检索与排序算法

B1 / P1:重叠命中的 BM25 证据被丢弃。 retrieval_service.py 按 chunk key 优先保留向量行,FTS 同 key 的 _score 未合并。实测向量命中与 FTS 分数 12 同时存在,聚合后 lexical_score=0。RRF 排名仍有双路奖励,但最终 Related 的词面权重和 KEYWORD_MATCH 标签拿不到真实证据。应分别读取两路特征,不混用保留行。

B2 / P1:代表 chunk 与文档总分混用。 选择片段时,将新 chunk 的 RRF 与已累计的文档 RRF 比较。三个 chunk 的复现中,第 2 号 chunk 双路命中最强,仍返回第 0 号。后续重排、摘要与 MMR 向量因此可能对应错误证据。文档累积分数和最佳 chunk 分数应分开保留。

B3 / P1:中文 FTS 未形成词级检索。 当前存储调用 create_fts_index('text', replace=True),没有中文分词设置。当前安装版本/临时真实数据库的例子:文档“数据库系统设计与事务处理”,查询“数据库”命中 0,查询完整字符串命中 1。这只证明该受控案例,不是中文语料总体召回率。应在入库与查询端采用一致的中文词项处理并验证短词、专名、中英混合和停用词;先复用现有能力,再决定是否需要新增依赖。

B4 / P0(原有已知问题):reranker 未就绪被当作真实分数。 predict_scores 返回全零 logits,标准化后为 0.615572。Related 的 0.50 reranker 权重给没有重排证据的候选增加约 0.3078;会改变 Related 排名和 Discover 门槛。另有超过 rerank_limit 的尾部候选用 semantic 替代 rerank,和已重排候选直接混排,需明确评分策略。先解决 Q3 的 fail/degrade 决策,再校准分数。

B5 / P1:索引与查询元数据不对称。 索引通过 getFileContext 去除标签 # 且包含 frontmatter;查询的 ContextEngine 只读取 inline tag 并保留 #。模拟和特征复现中 #ai 对 ai 的 tag_overlap=0,YAML tags 缺失。路径匹配又无条件进行 basename 兜底:已解析的 A/shared.md 会将 B/shared.md 误认直接链接。应复用一份上下文提取逻辑,已解析链接用规范路径精确匹配,仅对明确未解析别名兜底。

B6 / P1:停用词的计算、开关和检索使用不一致。 计算用 jieba,查询过滤用空格 split,中文句内词很难命中;过滤后 tokens 为空又保留原查询。关闭 enableAdaptiveFiltering 只影响 UI 高亮,后台仍使用已存集合;customStopwords 未传到后端。30% 文档频率也不等同于模板噪声,专业 Vault 的主题词可能被删除。应先统一开关语义和词项处理,保留可解释、可撤销的过滤行为,再验证阈值。

B7 / P1:分块的结构假设有边界错误。 用数组长度代替真实 heading level,# root → ### first → ### second 被解析为 root > first > second。前端还先移除代码围栏,后端无法识别代码中的 # 为代码而非标题。超长父段为每个子块重复保存完整 parent_text,可增加存储/读出体积。应保存真实层级,保证清洗与 chunker 契约一致;是否压缩父块存储需测量后决定。

评分与召回的设计限制,尚未经过语料验证:

  • 每路限制的是 chunk 数量,再按文档聚合;长文可占满候选,45 不等于 45 篇笔记。文档 RRF 累加后又加 hit bonus,具有长度偏置,应验证文档级 recall 再调整。
  • sigmoid 的幂变换不是有标签数据上的概率校准;BM25 批次 min-max 随候选池变化,单候选或全部同分时词面分全为 0。当前 UI 不宜把分数理解成确定的“匹配概率”。
  • Related 没有最低相关度门槛,可能把最不差的候选呈现为“强相关”。应通过明确无答案查询测试是否需要拒答阈值。
  • Discover 与 Related 互斥、相关性 gate 和 MMR 方向合理,但“未链接”“跨目录”不足以证明意外启发;共享标签也没有实现注释所称的“稀有度”计算。
  • “扫描整篇”直接传全篇到一次 encode,精排 query 又截至约 300 字符,没有按章节覆盖/聚合。长笔记尾部主题覆盖未被验证。
  • MMR 候选数不超过 top_k 时直接返回原输入顺序;不影响集合覆盖,却不保证 Discover 展示按最终分数/多样性顺序排列。
  • 兼容 semantic 路径先 encode,再经 Radar 再 encode;rerank=false 仍进入 balanced 管线,min_similarity/query_vector 在 facade 中未使用。兼容成本已经有真实行为漂移,应修正契约或明确移除计划。

C. 交互与用户判断

C1 / P1:全量索引失败仍显示完成。 full flush 只检查 status=success,忽略 failed_paths;完成条件按已读文件计数;FTS rebuild 的 false 也被忽略。模拟一次文档完全失败加 FTS 失败,仍显示“全文索引已就绪”。索引状态至少区分读取、成功提交、失败、待重试、取消和全文索引准备;最后一批确认前不能将处理计数解释为成功计数。

C2 / P1:陈旧响应校验只覆盖“另一个请求已经开始”。 searchId 仅发请求时递增;切换到空笔记、关闭文件或新上下文未达到触发条件,旧请求仍有效。response.context_id 没有比较。模拟切换文件且返回错误 context_id 后仍触发渲染。应在上下文失效事件同步取消显示资格,再在响应前核对请求和上下文;不能依赖 120ms 后一定发出新请求。

C3 / P1:稳定器把旧证据变成长期结果,并破坏双流互斥。 已不在新候选中的高分卡片,仍需新查询分数超过旧查询分数加 margin 才替换;不同查询的分数不宜直接比较。复现旧卡超寿命后仍留存;两流分别稳定后,同一 B 同时出现在 Related/Discover。标签先保留旧两项也会阻止理由更新。建议稳定布局/动画而非长期保留失效候选,最终显示统一执行互斥与过期检查。

C4 / P1:设置页重新渲染打断输入。 saveSettings → checkConnection → updateAllViewStatus → refreshStatusDisplay → display → containerEl.empty();文本设置逐次 onChange 保存,连接响应与周期心跳都会整体重建 DOM。静态调用链成立,实际焦点影响仍需 Obsidian 验收。应局部刷新状态区域,避免销毁正在编辑的控件;索引健康也不能在非 active 时直接标“已更新”。

C5 / P1:插入链接可指向错误同名笔记。 卡片已有 path,但插入仅生成 [[item.title]];建议使用 Obsidian 的文件链接生成能力和当前源路径解析,明确插入目标编辑器。跳转仅用清洗摘要前 25 字在原始 Markdown 的单行查找,多行/格式转换/重复段落可能无法定位;chunk_index 也不是源行号。索引返回源位置或可靠块锚点更可验证。

C6 / P2:信息减法已经影响来源识别和操作可达性。 卡片不展示标题/路径,上下文 breadcrumb 是空实现,用户需悬停才知道来源。卡片 Enter/Space 打开已实现,但插入/复制仅随鼠标 hover 弹层出现,没有 focus 打开、Esc 关闭和焦点关系;移动/键盘无法等价操作。建议保留一行简短来源、Focus/Note 状态和显式操作入口,不必恢复大段元数据。

C7 / P2:局部细节增加误导。 两处 label resolver 没有 RELEVANT、SHARED_CONCEPT 映射,直接显示机器代码;Popover“完整上下文”仍使用同一个 snippet;Related relevance 与 Discover 扣除重复惩罚后的 score 使用同一“匹配度”展示;原生服务日志用中文字符串/emoji 判断状态,语言或日志格式改变会影响完成通知。另有固定 320px 弹层和估算高度,须做窄窗口、多窗口、缩放实测。

积极部分:双流适合低打扰写作场景;原生 details 渐进设置、键盘打开卡片、文本节点渲染、请求序号和 batch 间让出执行权均有价值。应保留这些设计,而不是推倒重做。索引/搜索/连接三类状态应可同时表达,避免一个 connected 标签覆盖任务失败。

D. 复杂度、性能与下一步验收

当前不需要微服务、消息队列、通用排名插件、知识图谱平台或新前端框架。继续使用现有模块,优先消除重复逻辑:两份上下文提取、两份标签翻译、多个 score normalizer 和无内部调用的兼容门面。markdown/beautifulsoup4 在业务代码中没有找到使用;依赖移除需检查锁文件和安装链后执行,本次未改依赖。

性能风险只做待测判断:当前没有显式向量 ANN 建索引,召回与周期 count 会随数据量增长;encode 按文档调用,网络 batch 不等于模型 batch;增量 flush 无全量路径的批次上限;并发搜索/索引/全局维护没有共同负载控制。不能仅凭这些代码结构宣布性能差,也不应在没测之前加缓存、ANN 或并发层。

建议按以下可验证顺序实施:

  1. 完整性闭环:两个 Vault、两个同名路径;重建失败/取消;请求期间编辑/删除/重建;受管与外部进程。验收:其他 Vault 不变,旧有效索引可用,最后事件最终生效,外部进程不被结束。
  2. 真实状态与上下文:模拟 embedding/reranker/FTS/网络失败;A→空 B 时 A 延迟返回;同一笔记跨流迁移。验收:失败可见,无伪成功,无陈旧内容和跨流重复。
  3. 算法证据修复:重叠 chunk 的两路分数、最佳 chunk、中文专名、frontmatter/inline tag、同名链接、跳级标题、围栏代码。先修事实错误,再调权重。
  4. 相关性基线:准备约 40–60 条人工标注查询,覆盖中英混合、专名、短焦点、长笔记尾部、无答案和重名文件;记录文档级 recall@K、Related nDCG@4、Discover 相关性/重复率/人工启发评价。做 vector、FTS、fusion、rerank、MMR 的消融;这是建议规模,不是已验证数据。
  5. 性能与交互验收:记录数据规模、chunk 数、硬件、冷/热模型状态以及 encode/recall/rerank/总延迟 P50/P95/P99;在可丢弃 Obsidian Vault 实测键盘、悬浮、切换、断线、取消、大笔记和多 Vault。按证据决定是否优化批次、ANN 或负载控制。

文档现有的 bearer token 描述与实现的 X-Semantix-Token 不一致;“前端校验 context_id”“maintenance 为 Vault scope”“可靠有界同步”“旧索引总能保留”等描述也应在对应修复/产品决策后校正。此次仅将已确认差距记入当前状态,不把期望行为改写成已实现行为。


Release History

[0.9.3] - 2026-09-18

🛡️ 进程模型类型去耦与审查全绿达成 (Pure Type Decoupling & Zero-Warning Review)

  • Node 宿主进程模型完全解耦:
    • 针对审查环境缺少 Node.js 内置模块类型定义导致的 'error' 类型污染(Buffer 与 ChildProcess 级联引发 40 余项 unsafe member/call 告警),彻底移除对 'child_process' 的外部类型导入及全局 Buffer 引用。
    • 在 service-manager.ts 中构建纯净原生的 ManagedProcess、ProcessStream、ExecSyncResult 接口,解耦宿主依赖。
    • 在 node-adapter.ts 中新增强类型 getElectronProcess() 适配函数,彻底规避浏览器 DOM 库中未定义的全局 process.env 与 process.pid 访问。
  • 声明式设置 API 规范对齐:
    • 在 SemantixSettingTab 中声明 getSettingDefinitions(),完全消除 obsidianmd/settings-tab/prefer-setting-definitions 警告。
  • 代码收敛与冗余清理:
    • 修复 radar-view.ts 中 Keymap.isModEvent 多余的 as UserEvent 类型断言并移除无用导入。
    • 全项目 npm run lint 达成 0 错误、0 警告 极致清洁状态。
  • 全链路版本与构建对齐:
    • 同步版本至 0.9.3(package.json、package-lock.json、manifest.json、versions.json、README.md、engine/pyproject.toml、engine/main.py、engine/tests/test_smoke.py、uv.lock)。
    • 通过 npm run lint、npm run build、npm run check:release 以及全部 30 项 engine pytest 测试。

[0.9.2] - 2026-09-18

🛡️ 官方审查合规与类型安全全量加固 (Review Compliance & Strict Type Safety)

  • Obsidian 审查规则全量合规:
    • 本地 ESLint 规范升级至 eslint-plugin-obsidianmd@^0.4.2 并启用 obsidianmd.configs.recommended,100% 本地复现 Obsidian 官方审查机器人的严格规则集。
    • DOM 规范: 全量迁移 DOM 构建至 Obsidian 推荐的原生 DOM 辅助方法(createDiv、createSpan、createEl),彻底消除 document.createElement 审查警告。
    • 全局上下文隔离: 彻底移除 globalThis 使用,严格通过 Electron Node 适配器安全调用。
    • 严格类型安全与空安全: 全面修复 @typescript-eslint/no-unsafe-* 系列报错(service-manager.ts 进程流数据、client.ts 响应模型转换、Frontmatter 标签提取、JSON 解析与 PID 校验),无任何忽略指令。
    • Promise 生命周期防护: 全面修复 @typescript-eslint/no-floating-promises 与 @typescript-eslint/no-misused-promises,对所有非阻塞异步调用显式标记 void 忽略,对 setTimeout 回调安全包裹,杜绝浮动 Promise 与竞态异常。
    • 对等依赖与类型声明: 安装 @types/picomatch 补齐类型,移除非必要类型断言与 @ts-expect-error 指令。
  • 全链路版本与构建对齐:
    • 同步版本至 0.9.2(package.json、package-lock.json、manifest.json、versions.json、README.md、engine/pyproject.toml、engine/main.py、engine/tests/test_smoke.py、uv.lock)。
    • 通过 npm run lint、npm run build、npm run check:release、npm ci 以及全部 30 项 engine pytest 测试。

[0.9.1] - 2026-09-18

🩹 依赖加固与审查发布 (Dependency Hardening & Review Release)

  • 对等依赖严格锁定与 CI 编译加固:
    • 精准锁定 @codemirror/view 版本为 "6.38.6",完全契合 obsidian@1.10.3 的 strict peerDependency 约束,根治 GitHub Actions CI 在全新容器中执行 npm ci 时抛出的 ERESOLVE 对等依赖冲突。
    • 同步全链路引擎版本标识(ENGINE_VERSION、pyproject.toml、uv.lock)至 0.9.1。
    • 触发正式 Release 自动化构建以对接 Obsidian 官方插件市场审查。

[0.9.0] - 2026-09-18

🚀 架构重构与规范化 (Architecture & Compliance)

  • 代码库结构与 Obsidian 发布规范对齐:
    • 构建产物扁平化至根目录(./main.js、./styles.css),完全符合 Obsidian 官方加载契约与社区插件发布要求。
    • manifest.json 明确标注 isDesktopOnly: true 与维护者主页。
    • 清理多余临时脚本及冗余构建输出,简化根目录层级。
  • 社区插件上架审查合规治理 (Review Compliance):
    • 修复 API 版本契约:manifest.json 与 versions.json 将 minAppVersion 提升至 1.7.2 对齐 workspace.revealLeaf,并保留运行时向下降级容错。
    • 移除 window.confirm(),全量替换为 Obsidian 原生 Modal 异步交互对话框 (FullIndexConfirmModal)。
    • 规范设置项输入框 Placeholder,统一遵循 Obsidian UI Sentence Case 规范,移除所有行内 eslint-disable 指令。
    • 清理生产代码中所有非必要的 console.log 调试日志,保留标准的 console.warn 与 console.error。
    • 全面使用 window.setTimeout 代替全局 setTimeout,确保在 Obsidian 多窗口/分离窗口(Popout Windows)中的生命周期兼容性。
    • 将 @codemirror/view 显式声明至 package.json 的生产依赖项中。
    • 重构 src/styles.css,通过提升 CSS 选择器特异性全量移除 !important 规则。
    • 重构 README.md,提供纯正规范的英文主文(涵盖功能、安装、本地引擎启动、索引与日常使用指南)并保留完整中文说明。
    • 解决 Linux CI 运行 pytest 时 PyTorch 线程析构导致的 SIGABRT 退出码 134 异常。
  • 领域概念统一 (Radar Domain):
    • 核心模块全面从早期原型的 Whisperer 重构统一为 Radar(RadarEngine、RadarView),并维持原有工作区视图布局完全向后兼容。
    • 领域服务解耦:移除上帝外观,显式导出 LanceDBStorage、IndexService、RadarPipeline 单例。
    • 修复引擎存储路径解析,消除了运行目录(CWD)漂移缺陷。
    • 统一配置单一真理源(SSOT)为 RankingConfig。

[0.8.0] - 2026-09-10

🚀 新功能与体验增强 (Features & UI Redesign)

  • 设置页信息架构全量重构 (Settings Architecture Redesign):
    • 弃用平铺大卡片布局,对齐 Obsidian 原生折叠与分组规范。
    • 核心划分为:状态概览 (Status Banner)、推荐体验 (Whisperer Flow)、索引范围 (Scope) 与折叠式 高级设置 (Advanced)。
    • 新增双流 Discover 打散系数(MMR $\lambda$)调节滑块,支持 0.1~0.9 动态平滑调整探索多样性。
  • 倒排索引即时构建 (Instant FTS Indexing):
    • 新增 POST /index/rebuild-fts 接口并在初次全量索引完成后自动触发,消除前 30 秒混合检索由于倒排未就绪而降级的冷启动延迟。
  • 全量索引自适应流控 (Adaptive Indexing Flow Control):
    • 引入双阈值自适应分片(≤25 篇且 ≤150k 字符),结合 requestIdleCallback 帧对齐主线程让渡,杜绝索引期间 Obsidian 界面掉帧卡顿。

🛡️ 进程治理与自愈机制 (Process Governance & Self-Healing)

  • Win32 精准宿主状态判定 (Win32 Host Suicide):
    • 采用 Windows 原生 GetExitCodeProcess 探测 Obsidian 父进程退出码(退出码 ≠ 259 即判定销毁),彻底解决句柄假存活导致的僵尸进程滞留问题。
  • 孤儿进程树治理 (PID Lockfile Management):
    • 写入 .semantix.pid 锁文件,启动前基于 PID 树深度回收历史残留孤儿进程。
  • 三振出局自愈状态机 (Self-Healing Circuit Breaker):
    • 遇到异常退出采取 3s / 6s / 15s 指数退避重试;连续失败 3 次触发熔断阻断无限重试;支持用户主动停止压制与控制面板一键重置重启。

[0.7.0] - 2026-04-11

🚀 新功能 (Features)

  • 智能关键词高亮 (Intelligent Keyword Highlighting):弃用暴力 N-gram 切分,全面接入浏览器原生 Intl.Segmenter API 实现语言感知分词。高亮结果从"碎片化噪音"跃迁为真正有语义的关键词。
  • 权威停用词库 (Built-in Stopwords):内嵌约 150+ 词的权威中文停用词典(涵盖虚词、代词、连词、副词),自动过滤"怎么"、"由于"、"但是"等无意义高亮词汇。
  • 启发式噪音过滤 (Adaptive Noise Filtering):后端新增基于文档频率 (DF) 的自适应噪音词识别引擎。
    • 自动统计仓库内所有词汇的出现频率,将在超过 80% 文档中出现的"大众脸"词汇动态加入停用词表。
    • 前端新增设置开关 "启发式噪音过滤",可一键开启/关闭。
    • 新增 "立即分析仓库噪音" 按钮,支持手动触发词频分析。
  • 新增 API 端点:POST /index/compute-stopwords,用于触发仓库词频分析并返回噪音词列表。
  • 状态同步增强:GET /index/status 响应新增 vault_stopwords 字段,前端自动同步并缓存。

🩹 修复 (Fixes)

  • 后端启动崩溃修复:修复 models.py 中 Dict 类型未导入导致的 NameError(后端启动即退出,Code: 1)。
  • MaintenanceRequest 模型补全:为 MaintenanceRequest 添加缺失的 vault_id 字段,修复 compute-stopwords 端点调用时的 AttributeError。
  • 前端进程管理加固:
    • 修复后端服务未启用时仍刷新面板骨架的问题。
    • 修复手动启动后端时提示异常退出的进程竞态问题(Process Pinning 机制)。
    • 修复测试后端连接无响应问题(引入 5s 硬超时)。
    • 修复跨平台端口清理逻辑(新增 Unix lsof/kill 支持)。

🚀 稳定性与生命周期 (Stability & Lifecycle)

  • 后端“看门狗”自杀机制 (Watchdog Suicide): 实现了一种高度稳健的后端进程管理方案。后端现在具备自我监控能力:
    • 父进程存活探测: 通过环境变量感知 Obsidian PID,一旦发现父进程异常消失,立即启动自我清理。
    • 心跳超时回收: 若 120 秒内未收到前端 Ping 信号,后端将自动执行优雅退出,防止资源泄露。
  • 静默喂狗机制: 插件心跳探测现在会自动向后端发送生命信号,对用户完全无感且资源占用极低。

[0.4.6] - 2026-04-10

🚀 新功能与优化 (Features & Optimizations)

  • 动态状态追踪 (Live Status Tracking): 引入基于单条动态 Notice 的进度追踪系统。在后端启动全周期(同步依赖、唤醒服务、加载模型)提供原生的右上角实时反馈。
  • 高精度日志解析:
    • 升级 ServiceManager 实时流式解析 stdout/stderr,捕捉耗时较长的“模型加载”或“下载”期并及时播报。
    • 增加了对模型下载进度(Downloading: XX%)的实时捕获。
    • 细化了 uv sync 的环境同步阶段(解析、准备、安装)反馈。
  • 状态实时同步 (Real-time Status Sync): 增强了设置面板与插件核心的状态订阅机制。设置面板现在能自动响应后台心跳探测,实时展示连接状态指示灯,并在页面顶部增加全局状态徽标。
  • 启动性能优化: 将后端启动过程改为非阻塞异步执行,避免在自动拉起时占用 Obsidian 的初始化时间。
  • 一键环境创建: 当后端项目缺少虚拟环境时,在设置页提供初始化按钮,执行 uv venv 和 uv sync。
  • 检索上下文增强: 在语义搜索请求中自动注入笔记标题、Tags 及文件路径,显著提升推荐准确度。
  • 并发检索版本控制: 通过闭包版本号校验,丢弃过时的异步请求,彻底解决高频操作下的结果跳闪现象。

🩹 修复与加固 (Fixes & Hardening)

  • 浮窗生命周期管理: 引入状态锁机制,彻底解决启动浮窗在服务就绪后因后续杂散日志而“复活”并驻留的问题。
  • 进程生命周期加固:
    • 将 Windows 进程清理逻辑改为同步执行 (execSync),确保 Obsidian 退出时后端进程及其子进程完全回收。
    • 智能特征匹配: 引入基于命令行指纹的进程校验,确保清理端口冲突时绝不误杀无关进程。
  • 平台兼容性: 为所有底层进程操作增加平台校验,确保在移动端环境下不触发无效调用。
  • UI 体验优化:
    • 修正了 ServiceManager 的语法结构错误。
    • 优化了设置项文案,将“测试自启动”改为“探测服务连接”。
    • 将所有侧边栏视图标题、设置项名称统一调整为 Sentence case。

🛠️ 架构与工程 (Engineering)

  • 文档体系重塑: 建立 docs/ 专项手册体系,实现技术手册与入门指南的完全解耦。
  • 样式系统重构: 实现了 styles.css 构建集成,全量弃用 JS 硬编码样式,完美适配原生暗色/亮色主题。
  • 构建链路优化: 升级 esbuild 配置支持多入口异步构建,自动压缩输出 CSS 产物。
  • CI/CD 修复: 解决了 npm ci 依赖同步问题,并建立了前端产物的自动打包流程。

[0.3.0] - 2026-04-09

🚀 新功能 (Features)

  • Hybrid Search (混合检索): 集成基于 LanceDB 的全文本搜索 (FTS) 与向量检索,支持查询时的动态权衡。
  • Glob 排除规则: 引入 picomatch 库,支持复杂的通配符路径过滤(如 **/node_modules/**)。
  • Hit-boost 聚合排名: 在后端实现文档级 Hit-boost 排序算法,显著优化搜索结果的宏观相关度。

🛠️ 架构与工程 (Engineering)

  • Vault 哈希隔离: 实现基于路径哈希的 vault_id 机制,确保不同 Obsidian 仓库之间的数据索引物理隔离。
  • 后端模型热加载: 实现 FastAPI 启动时的模型异步预热与健康状态反馈。

[0.1.0] - 2026-03-20

🏗️ 初始版本

  • 建立 Semantix 插件核心骨架,支持单路向量检索。
  • 实现基础的文件变更实时监听同步机制。
  • 侧边栏基础视图雏形。