feat(search): Korean morphological tokenizer on the workerd search API

[seokju-na]

Adds a Korean morphological tokenizer (garu-orama-tokenizer) to the docs search, running server-side on the Cloudflare Workers (workerd) runtime.

What it does

Korean is now tokenized by real morphological analysis instead of whitespace, so queries match across inflections and particles:

Query	Matches
먹다	먹었다, 먹었습니다, 먹지
학교	학교에서, 학교를, 학교가

English (and any other) search is unchanged — the Korean tokenizer is composed with Orama's default tokenizer as a union of both token sets.

The workerd challenge

garu-ko normally fetch()es (browser build) or fs.readFiles (node build) its WASM + model at runtime — neither works in workerd. So instead:

• The 341 KB WASM is imported as a pre-compiled module and instantiated with initSync({ module }). workerd allows instantiating a compiled WebAssembly.Module at request time; only compiling from bytes at runtime is banned, which this path avoids.
• The ~1 MB model is inlined as bytes via a small Vite plugin.
• garu-ko's non-exported internals are reached via Vite aliases; garu-ko and @orama/orama are pinned, with a build-time existence guard so a version bump that relocates files fails loudly.

All garu code is dynamically imported so its weight lands in a lazy chunk loaded only on the first search, not at Worker startup. A failed init resets its cache so a transient cold-start error can't permanently poison an isolate.

Verified (dev Miniflare + prod build)

• ✅ Korean morphological matching (먹다→먹었습니다, 학교→학교에서)
• ✅ English search unchanged
• ✅ Client bundle clean — WASM/model/tokenizer are server-only
• ✅ Worker ~2.4 MB gzipped (under Free 3 MB / Paid 10 MB limits)
• ✅ Reviewed adversarially (workerd correctness, wasm-bindgen singleton, tokenizer correctness, build robustness)

Note

First search per isolate does a one-time WASM instantiate + model parse (a few ms of CPU). Fine on a Paid plan (30 s CPU); on the Free plan (10 ms/request) this could risk error 1102 — the deploy config (workers_dev: false, observability) indicates Paid, so this should be a non-issue, but worth confirming.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HhaZ2zL7bAqT3c8mRWTSs9

---

Summary by cubic

Adds a Korean morphological tokenizer to the docs search on the workerd API so queries match stems across particles and inflections. English and other languages are unchanged.

• New Features

  • Compose garu-orama-tokenizer with Orama’s default tokenizer (union of tokens) for Korean-aware matching.
  • Runs server-side on Cloudflare Workers: WASM imported as a pre-compiled module and the model is inlined; both are dynamically loaded on first search.
  • Keeps client bundle clean; failed init clears its cache to avoid stuck isolates.

• Dependencies

  • Add garu-ko@0.9.7, garu-orama-tokenizer@0.4.1; pin @orama/orama@3.1.18.
  • Vite plugin to inline the model and aliases to garu-ko internals; build-time guard for expected files.
  • Add src/garu-runtime.d.ts for internal garu bindings.

^{Written for commit 06995e1. Summary will update on new commits.}

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	webview-bundle-website	06995e1	Commit Preview URL Branch Preview URL	Jul 01 2026, 01:56 PM

[cubic-dev-ai]

No issues found across 5 files

_{Re-trigger cubic}