optimize(llmcore): decouple trim_messages_history token estimator from cap calc (closes #750)

[Kailigithub]

What

Extract the legacy 3 chars/token heuristic from trim_messages_history() into a single named function estimate_context_tokens(). Default behaviour is byte-identical with the prior code (same cost()/3 formula), so this PR is observably a refactor — the only user-visible change is the debug log now reports both the char count (ctx) and a token estimate (tokens_est) side-by-side.

Why

Issue #750 documents that today's /3 heuristic lives inside trim_messages_history() and is wrong for CJK/code/JSON content. There is no single place to swap it without forking the trim path. The issue's "minimum change" plan is exactly this PR:

1. 把现有 / 3 逻辑抽成独立 estimator；
2. 默认 estimator 仍然使用现在的 heuristic；
3. 给 session/model 留一个覆盖 estimator 的入口；
4. debug 时把 estimate 和 provider 返回的真实 usage 一起记录。

How

# llmcore.py
DEFAULT_CHARS_PER_TOKEN = 3

def estimate_context_tokens(history, sess=None):
    # 1. sess.token_estimator(messages) -> int  (full override)
    # 2. sess.token_chars_per_token      float (multiplier override, default 3)
    # 3. fallback: int(sum(len(json.dumps(m, ensure_ascii=False)) for m in history) / cpt)
    ...

trim_messages_history() keeps its char-based gate (cost(history) <= context_win * 3) so all downstream consumers — frontends/cost_tracker.py, frontends/tui_v3.py, frontends/tuiapp_v2.py, the hub — keep reading STATS['ctx'] in chars exactly as before. The estimator is wired into STATS['tokens_est'] and the existing debug print.

Extension points (opt-in)

Hook	Type	Behaviour
sess.token_estimator(msgs)	callable	full override; takes the history list, returns int token estimate. Exception is swallowed, falls back to multiplier branch.
sess.token_chars_per_token	float (default 3)	multiplier on the message-char sum. None, 0, negative, or unparsable values fall back to the default.

Both hooks are bug-tolerant so a misconfigured session can never crash the trim path.

Tests

tests/test_llmcore_estimate_context_tokens.py — 18 cases, all passing under python3 -m unittest discover -s tests -p 'test_*.py':

Group	What it pins
EstimateContextTokensDefaultHeuristicTests	v1 int(char_count / 3) behaviour for empty / English / CJK / multi-message / non-serialisable histories.
EstimateContextTokensCharsPerTokenOverrideTests	token_chars_per_token accepts strings, falls back on garbage / zero / negative / None.
EstimateContextTokensCallableOverrideTests	token_estimator callable wins over the multiplier, swallows exceptions, clamps negatives.
TrimMessagesHistoryBehaviorPreservedTests	the trim gate is unchanged (small history skipped, large history cuts), and the per-session multiplier only affects tokens_est not the gate.

Out of scope (intentional)

• frontends/cost_tracker.py has its own copy of the *3 heuristic (context_window_chars()). The issue explicitly excludes this from the first PR; it can be a follow-up that points at DEFAULT_CHARS_PER_TOKEN.
• Switching the actual trim gate from chars to tokens. The issue's "first version even keeps current behaviour" plan is what this PR ships — the gate is still char-based.
• compress_history_tags()'s max_len=800 field truncation (also called out in the issue as a separate follow-up).

Risk

Low. STATS['ctx'] still reports the same char count, the trim gate uses the same condition, and the only new output is a debug-only ~N tokens est annotation in the trim log. All 18 tests pass.

Closes #750