Skip to content

Commit f7ca1df

Browse files
committed
MAP-6 steps 2-4: caching mapping table, cache-resource surface, cookbook
Adapters (the MAP-6 table, lm15-contract/changes/2026-09-01-caching-design.md): - OpenAI, both dialects: mode=off sends prompt_cache_options {mode: explicit} on the gpt-5.6+ class (real off switch for writes; receipt: 3k-token prefix, cache_write_tokens 0) and nothing on older models; prefix=stable renders the system prompt as the first marked input item (instructions cannot carry a breakpoint); prefix=history sends nothing (implicit mode already trails); retention=long raises on 5.6+, maps to 24h below; key maps; resource raises. Model class by gpt-<major>.<minor> parse, extensions as the override (stated). - Anthropic: prefix=history marks the last block of the last message (receipt: turn 2 reads 3275, writes 24); stable/auto mark the system block; key and resource raise. - Gemini: the hidden per-request cachedContents POST is gone (resolve_prompt_cache, _cached_content_ids, complete/stream overrides); prefix intents send nothing (automatic tier, MAP-6 A1); key/retention raise; resource sends cachedContent + the suffix after prefix_until_index, no system/tools (receipt: cachedContentTokenCount 3580 against the created object). Surface (provider-neutral; Gemini first implementer): - BaseProviderLM cache hooks/drivers: cache_create/get/list/delete/update, lm.cache(prefix) -> CachedPrefix (pure where no resource tier), async mirrors, LMRouter/AsyncLMRouter.cache, EndpointSupport.caches, ProviderLM protocol verbs; vet ops cache_op_build/cache_op_parse. - docs: MAP-6 in mapping-rules.md; cookbook 19 with live outputs. - 922 tests (17 new mapping/surface tests). Signed-off-by: Maxime Rivest <mrive052@gmail.com>
1 parent deedac7 commit f7ca1df

18 files changed

Lines changed: 889 additions & 187 deletions
Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
# Prompt caching
2+
3+
**Problem** — You send the same long beginning many times: a document
4+
with many questions, or a chat that grows. Every provider can reuse that
5+
work, but each asks differently, and one of them will bill you for cache
6+
writes you never read back. You want one line that does the right thing
7+
everywhere and shows you what happened.
8+
9+
## Recipe
10+
11+
Make the beginning reusable once, then ask against it. `router.cache`
12+
uses the best tier the provider has; `cached + question` builds the
13+
request with the boundary at the seam.
14+
15+
```python
16+
from lm15 import LMRouter, Request, Message
17+
18+
router = LMRouter()
19+
DOCUMENT = "Field notes. " + " ".join(f"Plot {i}: soil pH {5 + i % 4}.{i % 10}, moisture {30 + i % 50}%." for i in range(300))
20+
prefix = Request(model="claude-sonnet-4-5", system="You answer with one number.", messages=[Message.user(DOCUMENT)])
21+
22+
cached = router.cache(prefix)
23+
for question in ["What is the pH of plot 7?", "What is the moisture of plot 12?"]:
24+
response = router.complete(cached + question)
25+
print(response.text, response.usage.cache_write_tokens, response.usage.cache_read_tokens)
26+
```
27+
28+
```output
29+
…The pH of plot 7 is **8.7**. 4513 0
30+
…42 0 4513
31+
```
32+
33+
The first call wrote 4513 tokens to the cache; the second read them
34+
back. On Anthropic that is a mark on the document block.
35+
36+
Same code, Gemini. Here the beginning becomes a stored object with a
37+
lifetime and an id, and `cached + question` sends only the question.
38+
39+
```python
40+
prefix = Request(model="gemini-2.5-flash", system="You answer with one number.", messages=[Message.user(DOCUMENT)])
41+
cached = router.cache(prefix, ttl_seconds=300)
42+
print(cached.id, cached.resource.tokens, cached.expires_at)
43+
for question in ["What is the pH of plot 7?", "What is the moisture of plot 12?"]:
44+
response = router.complete(cached + question)
45+
print(response.text, response.usage.cache_read_tokens)
46+
router.lm("gemini:x").cache_delete(cached.id)
47+
```
48+
49+
```output
50+
cachedContents/veegzt3p… 5300 2026-09-02T12:13:40Z
51+
8.7 5300
52+
42 5300
53+
```
54+
55+
A chat that grows is a different intent: mark everything so far each
56+
turn, and the next turn reads it.
57+
58+
```python
59+
from lm15 import Config, CacheConfig
60+
61+
history = [Message.user(DOCUMENT + "\nRemember these notes.")]
62+
for turn in ["Which plot has the highest pH?", "And its moisture?"]:
63+
request = Request(model="claude-sonnet-4-5", messages=history, config=Config(cache=CacheConfig(prefix="history")))
64+
response = router.complete(request)
65+
print(response.text[:40], "| write", response.usage.cache_write_tokens, "read", response.usage.cache_read_tokens)
66+
history += [response.message, Message.user(turn)]
67+
```
68+
69+
```output
70+
I've recorded all 300 plots with their s | write 4512 read 0
71+
The plots with the highest pH are those | write 191 read 4512
72+
```
73+
74+
## How it works
75+
76+
Every provider caches in up to three tiers, measured live on 2026-09-01
77+
(the receipts are in the contract repository under `research/caching/`):
78+
79+
- **Automatic.** Nothing to send. Best-effort. OpenAI, Gemini, xAI, Groq
80+
and most compatible servers. Gemini hit 1 time in 10 in our runs.
81+
- **A mark on a block.** Anthropic, and OpenAI from gpt-5.6 on. Reliable
82+
above the model's minimum (1,024 to 4,096 tokens). Writes cost 1.25×,
83+
reads 0.1×.
84+
- **A stored object.** Gemini today. Reliable, billed per token-hour
85+
while it exists, pinned to one model, and it owns your system prompt
86+
and tools, so the request may not repeat them.
87+
88+
`CacheConfig` names an intent, and each adapter maps it to the best tier
89+
it has. `prefix="stable"` marks the end of system and tools;
90+
`prefix="history"` marks the last message; `prefix_until_index=N` marks
91+
message N. Providers with no marks fall back to their automatic tier.
92+
That fallback is allowed because it costs nothing and the outcome is
93+
visible in `Usage.cache_read_tokens`. Fields that name a specific
94+
mechanism raise where it does not exist: `retention="long"`, `key`,
95+
`resource`.
96+
97+
```python
98+
try:
99+
router.complete(Request(model="gemini-2.5-flash", messages=[Message.user("hi")], config=Config(cache=CacheConfig(key="thread-1"))))
100+
except Exception as e:
101+
print(type(e).__name__, "-", str(e)[:90])
102+
```
103+
104+
```output
105+
UnsupportedFeatureError - gemini: cache.key is not supported — GenerateContent has no cache affinity key; use cache.…
106+
```
107+
108+
lm15 never creates cache state behind your back. `router.cache` is the
109+
one call that may spend money on a stored object, and it returns the
110+
object with its token count and expiry so you can delete it. The full
111+
rule is MAP-6 in [mapping rules](../mapping-rules.md).
112+
113+
## Variations
114+
115+
- **The fan-out trap on gpt-5.6 and later.** With no `CacheConfig`,
116+
OpenAI's automatic mode marks the *latest* message. A document with
117+
many different questions then writes the whole prompt at 1.25× on
118+
every call and never reads it back (measured: five calls, five full
119+
writes, zero hits). `router.cache(prefix)` or `prefix="stable"` is the
120+
one-line fix. A growing chat is the case automatic mode was built for.
121+
- **Turning writes off.** `CacheConfig(mode="off")` sends OpenAI's real
122+
off switch on gpt-5.6+ and nothing elsewhere, where writes are free:
123+
124+
```python
125+
request = Request(model="gpt-5.6-sol", messages=[Message.user(DOCUMENT), Message.user("pH of plot 7?")], config=Config(cache=CacheConfig(mode="off")))
126+
r = router.complete(request); print(r.usage.cache_write_tokens, r.usage.cache_read_tokens)
127+
```
128+
129+
```output
130+
0 0
131+
```
132+
133+
- **Keep the beginning stable.** A hit needs the same model, tools,
134+
system prompt, and earlier messages. Changing the tool list is a miss
135+
on every provider. Put timestamps and user names at the end.
136+
- **A second process.** Hits are server-side; nothing to share. For a
137+
stored object, share `cached.id` and reattach with
138+
`lm.cache_get(id)`.
139+
- **Async.** `await router.cache(prefix)`, `await router.complete(cached + question)`.
140+
141+
## See also
142+
143+
- [03 — System prompts](03-system-prompts.md)
144+
- [02 — Conversations](02-conversations.md)
145+
- [18 — Provider passthrough](18-provider-passthrough.md) for a provider's own cache knobs
146+
- [Mapping rules](../mapping-rules.md), MAP-6
147+
- [Using the type system](../using-the-type-system.md)

‎docs/cookbooks/STYLE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Cookbook style guide
22

3-
The editorial constitution for the 18-recipe lm15 cookbook. Every recipe
3+
The editorial constitution for the 19-recipe lm15 cookbook. Every recipe
44
page MUST conform. When in doubt, this file wins over precedent in
55
older docs (the retired monolithic cookbooks were raw material to mine,
66
not style to copy).

‎docs/cookbooks/index.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,3 +53,5 @@ capture. Style rules live in [STYLE.md](STYLE.md).
5353
error tree, why retry policy is yours, and testing without a network.
5454
- [18 — Provider passthrough](18-provider-passthrough.md) — Reaching
5555
provider-only knobs without forking your code off lm15's types.
56+
- [19 — Prompt caching](19-prompt-caching.md) — One line that reuses a
57+
long beginning on every provider, and shows what it cost.

‎docs/index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ that materializes into the identical `Response`.
7777
[Providers & models](providers-and-models.md).
7878
- **Keys, rotating tokens, subscriptions:**
7979
[Authentication](authentication.md).
80-
- **"How do I do X?"** — eighteen [cookbook recipes](cookbooks/index.md),
80+
- **"How do I do X?"** — nineteen [cookbook recipes](cookbooks/index.md),
8181
from [first request](cookbooks/01-first-request.md) to
8282
[video generation](cookbooks/14-video-generation.md) and
8383
[live sessions](cookbooks/15-live-sessions.md).

‎docs/mapping-rules.md‎

Lines changed: 76 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,80 @@ and grok-4.6 spent 158 while accepting `thinking: {"type": "disabled"}`
114114
without effect. Reasoning tokens are billed output; an explicit off that
115115
silently does nothing charges the caller for what they disabled.
116116

117+
## MAP-6 — Caching: one model for every provider
118+
119+
Every provider caches prompt prefixes in up to three tiers, measured
120+
2026-09-01 across 13 providers (lm15-contract/research/caching/):
121+
122+
- **automatic** — nothing to send, best-effort, no user-visible state
123+
(OpenAI all classes, Gemini implicit, xAI, Groq, DeepSeek, Fireworks,
124+
vLLM, SGLang);
125+
- **breakpoint** — a mark on a block, guaranteed above a per-model
126+
minimum, 1.25x write price (Anthropic; OpenAI gpt-5.6 and later);
127+
- **resource** — a stored, named object with a lifetime and a storage
128+
price per token-hour, pinned to one model (Gemini `cachedContents`,
129+
Vertex; any future provider with the same shape).
130+
131+
`CacheConfig` names INTENTS. Each adapter maps an intent to the best tier
132+
it has; the outcome is always visible in `Usage.cache_read_tokens` /
133+
`cache_write_tokens`.
134+
135+
1. **No `config.cache`: send nothing.** Automatic tiers apply server-side.
136+
2. **`mode="off"`: send nothing, and disable cache WRITES where a switch
137+
exists** — OpenAI gpt-5.6+ `prompt_cache_options: {"mode":
138+
"explicit"}` with no marks. Pre-5.6 OpenAI models reject the option and
139+
write for free, so they get nothing (option 2, ratified 2026-09-01).
140+
3. **`mode="auto"` with no prefix: the cheapest safe instruction.**
141+
Anthropic marks the system block; every other provider sends nothing.
142+
Never the trailing marker: with a changing last message it wrote the
143+
full prefix at 1.25x on every call and read nothing (measured on both
144+
Anthropic top-level `cache_control` and OpenAI implicit mode).
145+
4. **Prefix intents are marks where marks exist, and fall back to the
146+
automatic tier where they do not.** `prefix="stable"` marks the end of
147+
system + tools (Anthropic: the system block; OpenAI: the system prompt
148+
is rendered as the first developer/system message with the mark,
149+
because top-level `instructions` cannot carry one). `prefix="history"`
150+
marks the last block of the last message (Anthropic); on OpenAI 5.6+
151+
implicit mode already does exactly that, so nothing is sent.
152+
`prefix_until_index=N` marks the last block of message N (Anthropic:
153+
any block; OpenAI: a text block, else RAISE). Providers without marks
154+
(Gemini, xAI, Groq, older OpenAI, compat servers with
155+
`cache_control="none"`) send nothing. The fallback is permitted by two
156+
conditions, both required: it spends nothing, and its outcome is
157+
observable in usage. It must not be extended to fields that fail
158+
either condition.
159+
5. **`retention="long"`** names a specific mechanism: Anthropic `ttl:
160+
"1h"` (2x write); OpenAI <5.6 `prompt_cache_retention: "24h"`;
161+
OpenAI 5.6+ (30m is the only value) and Gemini (lifetime belongs to
162+
the stored object) RAISE.
163+
6. **`key`** is a best-effort affinity hint: OpenAI and OpenRouter
164+
`prompt_cache_key`; Anthropic and Gemini RAISE.
165+
7. **`resource`** is a `CacheInfo.id` from the resource tier. The adapter
166+
references the object and sends only what the object does not hold —
167+
Gemini: `cachedContent` + the messages after `prefix_until_index`, no
168+
`systemInstruction`/`tools`/`toolConfig` (the server rejects them next
169+
to a cache). Providers without the tier RAISE.
170+
8. **The resource tier is a surface**, shaped like files: `cache_create
171+
(prefix: Request, ttl_seconds, label)`, `cache_get`, `cache_list`,
172+
`cache_delete`, `cache_update(id, ttl_seconds)`, pure hooks, async
173+
mirrors, the `cache` harness direction, `EndpointSupport.caches`.
174+
`lm.cache(prefix)` returns a `CachedPrefix`: on the resource tier it
175+
creates the object (one explicit, billed call); elsewhere it is pure.
176+
`cached + messages` builds the Request with the boundary at the seam.
177+
9. **No hidden network calls.** An adapter's `build_request` never
178+
creates cache state. (Removed 2026-09-02: the Gemini adapter's
179+
per-request `cachedContents` POST, which made a billed object per
180+
turn and reused none.)
181+
10. **Docs state the fan-out trap**: on OpenAI 5.6+ with no config, one
182+
document and many questions writes at 1.25x every time and never
183+
reads; `prefix="stable"` or `lm.cache(prefix)` is the one-line fix.
184+
A tools change is a miss everywhere.
185+
186+
**Why:** the caching design pass (lm15-contract/changes/2026-09-01-caching-design.md,
187+
research/caching/). Provider agnosticism is defined as: the same code
188+
runs everywhere, does the best thing the provider offers, and shows the
189+
result — not identical bytes saved everywhere.
190+
117191
---
118192

119193
History: MAP-1 and MAP-2 were implicit in the reference adapters; they were
@@ -124,4 +198,5 @@ gemini.max_output_tokens (MAP-2) — see
124198
after live vLLM/SGLang/ollama testing showed the multi-end merge losing usage.
125199
MAP-5 was written on 2026-09-01 after a reasoning-off audit found four
126200
adapters silently omitting the disable (see
127-
`lm15-contract/changes/2026-09-01-reasoning-off.md`).
201+
`lm15-contract/changes/2026-09-01-reasoning-off.md`). MAP-6 was written on
202+
2026-09-01/02 from the first design pass (`lm15-contract/playbooks/design-pass.md`).

‎lm15/features.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ class EndpointSupport:
4545
video: bool = False
4646
responses_api: bool = False
4747
models: bool = False
48+
caches: bool = False
4849

4950
# Escape hatch for endpoint names not yet promoted to typed booleans.
5051
extra: frozenset[str] = field(default_factory=frozenset)

‎lm15/protocols.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,9 @@
1010
from .features import EndpointSupport, ProviderManifest
1111
from .models import ModelInfo
1212
from .types import (
13+
CachedPrefix,
14+
CacheInfo,
15+
CachePage,
1316
VideoGenerationRequest,
1417
VideoJobInfo,
1518
VideoPart,
@@ -80,3 +83,9 @@ def video_status(self, video_id: str) -> VideoJobInfo: ...
8083
def video_result(self, video_id: str) -> VideoPart: ...
8184
def video_list(self, limit: int = 20, model: str | None = None) -> tuple[VideoJobInfo, ...]: ...
8285
def list_models(self) -> tuple[ModelInfo, ...]: ...
86+
def cache_create(self, prefix: Request, *, ttl_seconds: int | None = None, label: str | None = None) -> CacheInfo: ...
87+
def cache_get(self, cache_id: str) -> CacheInfo: ...
88+
def cache_list(self, limit: int = 20, cursor: str | None = None) -> CachePage: ...
89+
def cache_delete(self, cache_id: str) -> None: ...
90+
def cache_update(self, cache_id: str, *, ttl_seconds: int) -> CacheInfo: ...
91+
def cache(self, prefix: Request, *, ttl_seconds: int | None = None, label: str | None = None) -> CachedPrefix: ...

‎lm15/providers/anthropic.py‎

Lines changed: 30 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -453,13 +453,38 @@ def _payload(self, request: Request, stream: bool) -> dict[str, Any]:
453453

454454
messages = [self._message(m) for m in request.messages]
455455

456-
# Apply prefix caching if requested
457-
if use_cache and cache_cfg is not None and cache_cfg.prefix_until_index is not None:
458-
idx = min(cache_cfg.prefix_until_index, len(messages) - 1)
459-
if idx >= 0 and messages[idx].get("content"):
456+
# MAP-6 marks. prefix_until_index=N and prefix="history" (N = last
457+
# message) put cache_control on the last block of message N; the
458+
# explicit block form rather than the top-level automatic marker,
459+
# which walks backwards silently when the last block is ineligible.
460+
# prefix="stable" (and plain auto) mark the system block below.
461+
# key / resource name mechanisms the Messages API does not have.
462+
if use_cache and cache_cfg is not None:
463+
if cache_cfg.key is not None:
464+
raise UnsupportedFeatureError(
465+
"anthropic: cache.key is not supported — the Messages API has no "
466+
"cache affinity key (OpenAI's prompt_cache_key); marks on blocks "
467+
"are the mechanism (prefix / prefix_until_index)",
468+
provider=self.provider,
469+
)
470+
if cache_cfg.resource is not None:
471+
raise UnsupportedFeatureError(
472+
"anthropic: cache.resource is not supported — the Messages API has no "
473+
"stored-cache tier; it caches by marks on blocks",
474+
provider=self.provider,
475+
)
476+
idx = None
477+
if cache_cfg.prefix_until_index is not None:
478+
idx = min(cache_cfg.prefix_until_index, len(messages) - 1)
479+
elif cache_cfg.prefix == "history":
480+
idx = len(messages) - 1
481+
if idx is not None and idx >= 0 and messages[idx].get("content"):
460482
last_block = messages[idx]["content"][-1]
461483
if isinstance(last_block, dict):
462-
last_block.setdefault("cache_control", {"type": "ephemeral"})
484+
marker: dict[str, Any] = {"type": "ephemeral"}
485+
if long_cache:
486+
marker["ttl"] = "1h"
487+
last_block.setdefault("cache_control", marker)
463488

464489
thinking_budget = _reasoning_thinking_budget(request)
465490
payload: dict[str, Any] = {

0 commit comments

Comments
 (0)