|
| 1 | +# Prompt caching |
| 2 | + |
| 3 | +**Problem** — You send the same long beginning many times: a document |
| 4 | +with many questions, or a chat that grows. Every provider can reuse that |
| 5 | +work, but each asks differently, and one of them will bill you for cache |
| 6 | +writes you never read back. You want one line that does the right thing |
| 7 | +everywhere and shows you what happened. |
| 8 | + |
| 9 | +## Recipe |
| 10 | + |
| 11 | +Make the beginning reusable once, then ask against it. `router.cache` |
| 12 | +uses the best tier the provider has; `cached + question` builds the |
| 13 | +request with the boundary at the seam. |
| 14 | + |
| 15 | +```python |
| 16 | +from lm15 import LMRouter, Request, Message |
| 17 | + |
| 18 | +router = LMRouter() |
| 19 | +DOCUMENT = "Field notes. " + " ".join(f"Plot {i}: soil pH {5 + i % 4}.{i % 10}, moisture {30 + i % 50}%." for i in range(300)) |
| 20 | +prefix = Request(model="claude-sonnet-4-5", system="You answer with one number.", messages=[Message.user(DOCUMENT)]) |
| 21 | + |
| 22 | +cached = router.cache(prefix) |
| 23 | +for question in ["What is the pH of plot 7?", "What is the moisture of plot 12?"]: |
| 24 | + response = router.complete(cached + question) |
| 25 | + print(response.text, response.usage.cache_write_tokens, response.usage.cache_read_tokens) |
| 26 | +``` |
| 27 | + |
| 28 | +```output |
| 29 | +…The pH of plot 7 is **8.7**. 4513 0 |
| 30 | +…42 0 4513 |
| 31 | +``` |
| 32 | + |
| 33 | +The first call wrote 4513 tokens to the cache; the second read them |
| 34 | +back. On Anthropic that is a mark on the document block. |
| 35 | + |
| 36 | +Same code, Gemini. Here the beginning becomes a stored object with a |
| 37 | +lifetime and an id, and `cached + question` sends only the question. |
| 38 | + |
| 39 | +```python |
| 40 | +prefix = Request(model="gemini-2.5-flash", system="You answer with one number.", messages=[Message.user(DOCUMENT)]) |
| 41 | +cached = router.cache(prefix, ttl_seconds=300) |
| 42 | +print(cached.id, cached.resource.tokens, cached.expires_at) |
| 43 | +for question in ["What is the pH of plot 7?", "What is the moisture of plot 12?"]: |
| 44 | + response = router.complete(cached + question) |
| 45 | + print(response.text, response.usage.cache_read_tokens) |
| 46 | +router.lm("gemini:x").cache_delete(cached.id) |
| 47 | +``` |
| 48 | + |
| 49 | +```output |
| 50 | +cachedContents/veegzt3p… 5300 2026-09-02T12:13:40Z |
| 51 | +8.7 5300 |
| 52 | +42 5300 |
| 53 | +``` |
| 54 | + |
| 55 | +A chat that grows is a different intent: mark everything so far each |
| 56 | +turn, and the next turn reads it. |
| 57 | + |
| 58 | +```python |
| 59 | +from lm15 import Config, CacheConfig |
| 60 | + |
| 61 | +history = [Message.user(DOCUMENT + "\nRemember these notes.")] |
| 62 | +for turn in ["Which plot has the highest pH?", "And its moisture?"]: |
| 63 | + request = Request(model="claude-sonnet-4-5", messages=history, config=Config(cache=CacheConfig(prefix="history"))) |
| 64 | + response = router.complete(request) |
| 65 | + print(response.text[:40], "| write", response.usage.cache_write_tokens, "read", response.usage.cache_read_tokens) |
| 66 | + history += [response.message, Message.user(turn)] |
| 67 | +``` |
| 68 | + |
| 69 | +```output |
| 70 | +I've recorded all 300 plots with their s | write 4512 read 0 |
| 71 | +The plots with the highest pH are those | write 191 read 4512 |
| 72 | +``` |
| 73 | + |
| 74 | +## How it works |
| 75 | + |
| 76 | +Every provider caches in up to three tiers, measured live on 2026-09-01 |
| 77 | +(the receipts are in the contract repository under `research/caching/`): |
| 78 | + |
| 79 | +- **Automatic.** Nothing to send. Best-effort. OpenAI, Gemini, xAI, Groq |
| 80 | + and most compatible servers. Gemini hit 1 time in 10 in our runs. |
| 81 | +- **A mark on a block.** Anthropic, and OpenAI from gpt-5.6 on. Reliable |
| 82 | + above the model's minimum (1,024 to 4,096 tokens). Writes cost 1.25×, |
| 83 | + reads 0.1×. |
| 84 | +- **A stored object.** Gemini today. Reliable, billed per token-hour |
| 85 | + while it exists, pinned to one model, and it owns your system prompt |
| 86 | + and tools, so the request may not repeat them. |
| 87 | + |
| 88 | +`CacheConfig` names an intent, and each adapter maps it to the best tier |
| 89 | +it has. `prefix="stable"` marks the end of system and tools; |
| 90 | +`prefix="history"` marks the last message; `prefix_until_index=N` marks |
| 91 | +message N. Providers with no marks fall back to their automatic tier. |
| 92 | +That fallback is allowed because it costs nothing and the outcome is |
| 93 | +visible in `Usage.cache_read_tokens`. Fields that name a specific |
| 94 | +mechanism raise where it does not exist: `retention="long"`, `key`, |
| 95 | +`resource`. |
| 96 | + |
| 97 | +```python |
| 98 | +try: |
| 99 | + router.complete(Request(model="gemini-2.5-flash", messages=[Message.user("hi")], config=Config(cache=CacheConfig(key="thread-1")))) |
| 100 | +except Exception as e: |
| 101 | + print(type(e).__name__, "-", str(e)[:90]) |
| 102 | +``` |
| 103 | + |
| 104 | +```output |
| 105 | +UnsupportedFeatureError - gemini: cache.key is not supported — GenerateContent has no cache affinity key; use cache.… |
| 106 | +``` |
| 107 | + |
| 108 | +lm15 never creates cache state behind your back. `router.cache` is the |
| 109 | +one call that may spend money on a stored object, and it returns the |
| 110 | +object with its token count and expiry so you can delete it. The full |
| 111 | +rule is MAP-6 in [mapping rules](../mapping-rules.md). |
| 112 | + |
| 113 | +## Variations |
| 114 | + |
| 115 | +- **The fan-out trap on gpt-5.6 and later.** With no `CacheConfig`, |
| 116 | + OpenAI's automatic mode marks the *latest* message. A document with |
| 117 | + many different questions then writes the whole prompt at 1.25× on |
| 118 | + every call and never reads it back (measured: five calls, five full |
| 119 | + writes, zero hits). `router.cache(prefix)` or `prefix="stable"` is the |
| 120 | + one-line fix. A growing chat is the case automatic mode was built for. |
| 121 | +- **Turning writes off.** `CacheConfig(mode="off")` sends OpenAI's real |
| 122 | + off switch on gpt-5.6+ and nothing elsewhere, where writes are free: |
| 123 | + |
| 124 | + ```python |
| 125 | + request = Request(model="gpt-5.6-sol", messages=[Message.user(DOCUMENT), Message.user("pH of plot 7?")], config=Config(cache=CacheConfig(mode="off"))) |
| 126 | + r = router.complete(request); print(r.usage.cache_write_tokens, r.usage.cache_read_tokens) |
| 127 | + ``` |
| 128 | + |
| 129 | + ```output |
| 130 | + 0 0 |
| 131 | + ``` |
| 132 | + |
| 133 | +- **Keep the beginning stable.** A hit needs the same model, tools, |
| 134 | + system prompt, and earlier messages. Changing the tool list is a miss |
| 135 | + on every provider. Put timestamps and user names at the end. |
| 136 | +- **A second process.** Hits are server-side; nothing to share. For a |
| 137 | + stored object, share `cached.id` and reattach with |
| 138 | + `lm.cache_get(id)`. |
| 139 | +- **Async.** `await router.cache(prefix)`, `await router.complete(cached + question)`. |
| 140 | + |
| 141 | +## See also |
| 142 | + |
| 143 | +- [03 — System prompts](03-system-prompts.md) |
| 144 | +- [02 — Conversations](02-conversations.md) |
| 145 | +- [18 — Provider passthrough](18-provider-passthrough.md) for a provider's own cache knobs |
| 146 | +- [Mapping rules](../mapping-rules.md), MAP-6 |
| 147 | +- [Using the type system](../using-the-type-system.md) |
0 commit comments