fix(search): expose markdown frontmatter on chunks

[RerankerGuo]

Summary

• copy parsed Markdown frontmatter onto emitted FileChunk metadata
• preserve JSON-safe frontmatter values such as conversation_date for search filters
• add coverage showing chunk metadata and metadata search_filter plumbing

Refs #302

Verification

• .. [100%]
  2 passed in 1.02s
• check python ast.........................................................Passed
  check yaml...........................................(no files to check)Skipped
  check xml............................................(no files to check)Skipped
  check toml...........................................(no files to check)Skipped
  check json...........................................(no files to check)Skipped
  detect private key.......................................................Passed
  trim trailing whitespace.................................................Passed
  Add trailing commas......................................................Passed
  black....................................................................Passed
  flake8...................................................................Passed
  pylint...................................................................Passed
  Check package with Pyroma................................................Passed

[ployts]

Thanks for working on this! Exposing the markdown frontmatter on chunks is a great addition for search filtering.
To ensure backward compatibility and prevent any unexpected side effects for existing users, I suggest adding a configuration flag (e.g., enable_frontmatter_metadata or include_frontmatter_in_metadata) to toggle this behavior. Could we set it to False by default?
This would allow users to opt-in when needed while keeping the default behavior unchanged. Let me know your thoughts!

[RerankerGuo]

Updated per feedback. I added the include_frontmatter_in_metadata flag, set the default config value to false, and adjusted tests to cover both the backward-compatible default and the opt-in metadata behavior.

Verification:

• Targeted pytest: 2 passed
• Pre-commit on touched files: passed

[RerankerGuo]

Rebased onto current origin/main (38cf160) to clear the CONFLICTING state from the merge of #317, #321, #322, #326 and others.

Resolution notes:

• Kept all new tests from origin/main (tool_context_deduplicates…, seen_chunks_expire…, start_end_date_promoted…, invalid_date_is_ignored…, etc.) and preserved the include_frontmatter_in_metadata opt-in flag from the prior round of feedback.
• Dropped the duplicate test_search_step_passes_metadata_filter_to_store left over from the conflict and added it once after the dedup/expiry tests where it reads more naturally.
• Restored the trailing asyncio.run(run()) for the seen_chunks_expire_after_ttl test that the conflict had clipped, so pylint no longer flags an unused run.

Local verification before pushing:

• pytest tests/unit/test_search_step.py tests/unit/test_markdown_file_chunker.py tests/unit/test_config_parser.py → 37 passed
• pre-commit run --files <touched paths> → all hooks green (pylint 10.00/10)

[RerankerGuo]

Pushed a follow-up commit f654961 adding a complementary opt-in flag.

New: include_frontmatter_keys_in_metadata (default: []) on the markdown chunker config. When include_frontmatter_in_metadata: true and this list is non-empty, only the listed frontmatter keys land on FileChunk.metadata — other frontmatter fields stay out.

Why: the previous PR exposed all non-empty frontmatter, which is fine for short, well-curated frontmatter but starts leaking implementation metadata (name, description, unrelated tags) once a file's frontmatter grows. With the allow-list, a deployment can opt in to only conversation_date for the LME use case without every other field ending up in the search filter bag.

Behavior matrix (now covered by test_parse_frontmatter_metadata_keys_allowlist + test_default_config_keeps_frontmatter_chunk_metadata_opt_in):

include_frontmatter_in_metadata	include_frontmatter_keys_in_metadata	Result
false	any	empty metadata (back-compat default)
true	[] or unset	all non-empty frontmatter keys (matches the original PR)
true	["conversation_date"]	only conversation_date, others dropped
true	list with absent key	absent key is silently skipped

Local verification:

• pytest tests/unit/test_markdown_file_chunker.py tests/unit/test_config_parser.py → 26 passed
• pre-commit run --files reme/components/file_chunker/markdown_file_chunker.py reme/config/default.yaml tests/unit/test_markdown_file_chunker.py tests/unit/test_config_parser.py → all green (pylint 10.00/10)

No change to search.py, search_filter API, or any other step — purely additive at the markdown chunker boundary.

[RerankerGuo]

Heads-up: the branch head on the fork had been reverted to 5ef7feb (an older fixup! commit from before the include_frontmatter_in_metadata PR), which was making GitHub show stale 'this branch has conflicts that must be resolved' against the now-current origin/main. Just re-pushed the rebased-onto-eb471d7 chain that includes the include_frontmatter_keys_in_metadata follow-up so the PR is back in a clean MERGEABLE state.

Local verification on the restored chain:

• git rebase origin/main → clean, no conflicts.
• pytest tests/unit/test_search_step.py tests/unit/test_markdown_file_chunker.py tests/unit/test_config_parser.py → 38 passed.
• pre-commit run --files → all hooks green.

If anyone wonders where the earlier revert came from: it lined up with a local branch named rebase/fix/markdown-chunk-frontmatter-filter-api (5ef7feb) that pre-dates this work — likely a stray force-push from an earlier session bound the wrong ref. Worth a note so this doesn't bite future rebases.

[RerankerGuo]

Heads-up on the previous push: rebased onto origin/main (82971ac) again, this time over the diff from #325 (xyf2020, json/jsonl chunkers) and #331 (chore workflow) so the conflict on reme/config/default.yaml is gone and the chain is now 1edcc08 → 53381cf → 42d5447 → 79f448f → 8d1a49e on top of 82971ac.

Conflict resolution: kept #325's json / jsonl chunker entries and the include_frontmatter_in_metadata: false plus include_frontmatter_keys_in_metadata: [] lines under file_chunker.markdown. reindex-style jobs in default.yaml already reference the jsonl/index updates from #325, so I did not touch anything else in default.yaml.

Local verification:

• pytest tests/unit/test_search_step.py tests/unit/test_markdown_file_chunker.py tests/unit/test_config_parser.py → 38 passed.
• pre-commit run --files reme/config/default.yaml reme/components/file_chunker/markdown_file_chunker.py tests/unit/test_markdown_file_chunker.py tests/unit/test_config_parser.py tests/unit/test_search_step.py → all hooks green.
• CI re-running on the new head (check-pr-title already SUCCESS, unit tests + pre-commit in progress).

[ployts]

Thanks for the thorough follow-through, @RerankerGuo — this looks good to merge. 🎉

The backward-compat concerns are fully addressed:

• include_frontmatter_in_metadata defaults to false, so existing users see no behavior change.
• include_frontmatter_keys_in_metadata (default []) gives a clean opt-in path to expose only the specific frontmatter keys needed for search filters, avoiding the implementation-metadata leakage from the original approach.
• Rebased onto current origin/main (82971ac), conflicts on reme/config/default.yaml resolved while keeping #325's json/jsonl chunker entries, and tests cover both the default and opt-in paths (targeted pytest + pre-commit passing).

Approving — will merge. Thanks again for exposing frontmatter for search filtering, this is a nice addition. 🙏

[RerankerGuo]

Thanks @ployts for the review and the merge! 🙏

Quick recap of the collaboration loop on this one for future reference:

• v1 exposed frontmatter unconditionally → you flagged the backward-compat risk.
• v2 added include_frontmatter_in_metadata: false default and the opt-in flag.
• v3 added include_frontmatter_keys_in_metadata: [] so callers can ship only conversation_date to the search filter.

The allow-list landed as a follow-up precisely because your first-round comment pointed at the same surface area; happy to keep using that pattern (default-OFF + explicit opt-in) on future config-facing PRs.

Will leave #332 (docs follow-up) and #333 (jinli_lme.yaml lme_one_question job) sitting for now — they're complementary to #326 and shouldn't need changes from your side unless the LongMemEval direction tightens further.