[Multilingual Gap] Implement span-grounded semantic unitization and concept-dictionary validation #168

Buyer problem

The approved product requires multilingual evidence to enter a shared latent space while preserving exact native-language evidence. Current work provides immutable spans and bounded language/concept gates, but protected main does not yet turn real multilingual documents into validated semantic units and versioned concept mappings. Architecture support alone cannot justify a language-support claim.

Product outcome

Implement the production preprocessing vertical:

immutable text/layout evidence
→ Unicode-safe structural segmentation
→ language/script posterior
→ language-tailored sentence/word/morphology/dependency evidence
→ optional LLM semantic-unit proposal
→ deterministic span/schema/security validation
→ versioned shared-concept + native-lexical channels

The base algorithm must remain language-agnostic: language metadata may select a validated profile or tailoring, but missing/mixed/unresolved metadata cannot silently switch to an unrelated heuristic.

Existing related work

• evidence_core owns exact immutable source spans.
• feat(concept): language-profile gates with shared-space RMSE #53 defines profile and concept-alignment gates, not the complete pipeline.
• feat(evidence): keep embedded image URIs as positional non-lexical units #58 preserves embedded images as positional non-lexical evidence.
• feat(corpus): treat NFC/NFD bodies as split leakage #59 covers NFC/NFD split identity.
• feat(corpus): refuse TF-IDF and BM25 as inferential weights #63/feat(method): model template sources without inferential weights #75/feat(method): refuse default stopword deletion of report language #145–feat(method): refuse prompt boilerplate as unique content #152 separate retrieval/method effects from inferential content.
• docs(evidence): define language-agnostic semantic spans #83 is a design proposal for semantic spans and model-profile token limits, not executable production behavior.

Acceptance criteria

• Original source bytes/text and exact UTF-8 byte plus Unicode-scalar offsets remain immutable through every transformation.
• NFC is an analysis key without overwriting source text; hostile Unicode, invalid boundaries, mixed scripts, and normalization collisions are tested.
• Headings, paragraphs, lists, tables, captions, code blocks, DOM/layout blocks, and oversized units are segmented without silent truncation.
• Language-tailored profiles cover at least the declared initial release languages and explicitly report validated, calibrated, provisional, or unresolved status.
• Morphology, universal POS, dependency phrases, negation, modality, quantity, temporal expressions, and script/language evidence remain typed inputs rather than deletion heuristics.
• LLM proposals are optional, schema-bound, exact-span-grounded, and untrusted; document prompt injection cannot alter tools, policy, credentials, or concept authority.
• concept_dictionary is versioned and reviewable; known concepts, aliases, language expressions, unknown concepts, merges/splits, and provenance are persisted without losing native lexical evidence.
• Every semantic unit binds to source span(s), payload digest, profile/version, model/input role when applicable, concept-dictionary version, confidence/calibration, and review status.
• Base64 or embedded images preserve position and use separate OCR/object/tag evidence; binary payload never becomes lexical estimator text.
• Parallel and expert-annotated corpora report span F1, concept precision/recall, calibration/Brier score, inter-rater agreement, language alignment, and invariance evidence.
• Equivalent translated/revised/copied documents remain in the same evaluation partition.
• No default stopword list, TF-IDF, BM25, translation-first summary, or free-form LLM concept creation is used as production inferential authority.
• Production statement/branch coverage and public docs remain 100%; fuzz/property tests cover oversized, malformed, hostile, mixed-language, and code-switched inputs.

Initial release language claim

Choose a bounded first release profile and validate it completely—preferably Korean and English—before expanding to Japanese, Chinese, Vietnamese, Indonesian, French, German, Turkish, and long-tail languages. The architecture may be broad; the product claim must remain evidence-specific.

Authority

Use PRD §§3, 6, 14, ADR 0004, ADR 0012, and the research register. Update doctoring with the exact Unicode, segmentation, morphology, embedding/profile, multilingual-alignment, and measurement-invariance sources used by the implementation.