Skip to content

fix(semantic): preserve abstracts after list markers - #3718

Draft
ehz0ah wants to merge 1 commit into
volcengine:mainfrom
ehz0ah:fix/abstract-list-marker-truncation
Draft

fix(semantic): preserve abstracts after list markers#3718
ehz0ah wants to merge 1 commit into
volcengine:mainfrom
ehz0ah:fix/abstract-list-marker-truncation

Conversation

@ehz0ah

@ehz0ah ehz0ah commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Description

Fix generated abstract truncation so numbered-list markers such as 1. are not mistaken for sentence endings.

A long overview introduction beginning with OpenViking's own numbered format could previously truncate .abstract.md to exactly 1., silently completing semantic processing with an unusable summary. The new boundary scanner ignores numbered/lettered list markers, decimals, and common abbreviations while preserving CJK punctuation, paragraph boundaries, Markdown structure, tiny limits, and the existing long-first-sentence behavior.

Human Involvement

  • A human participated in the implementation or review loop
  • This PR was generated entirely by AI agents without human participation in the loop

Related Issue

Fixes #3709

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update

Changes Made

  • Detect contextual sentence and paragraph boundaries instead of treating every period-before-space as terminal.
  • Ignore numbered/hierarchical/lettered list markers, decimals, and conservative common abbreviations.
  • Add regression coverage through _normalize_overview_generation and the default 256-character abstract limit.

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this on the following platforms:
    • Linux
    • macOS
    • Windows
pytest tests/storage/test_semantic_processor_l0_l1.py
# 27 passed

broader semantic-processor suite
# 108 passed

ruff check openviking/storage/queuefs/semantic_processor.py tests/storage/test_semantic_processor_l0_l1.py
ruff format --check openviking/storage/queuefs/semantic_processor.py
git diff --check
# passed

The complete semantic-processor glob has one unrelated existing failure in the move/vector-store test caused by an unchanged path-lock mock.

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Screenshots (if applicable)

Not applicable.

Additional Notes

This PR intentionally does not change abstract_max_chars or overview_max_chars defaults.

Ignore list markers, decimals, and abbreviations when selecting truncation boundaries so generated abstracts cannot collapse to fragments such as 1.
@ehz0ah
ehz0ah force-pushed the fix/abstract-list-marker-truncation branch from f966aea to 08b6e43 Compare August 4, 2026 02:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

1 participant