Skip to content

Enable bf16 export for XNNPACK backend - #247

Open
JakeStevens wants to merge 5 commits into
huggingface:mainfrom
JakeStevens:bf16-support
Open

Enable bf16 export for XNNPACK backend#247
JakeStevens wants to merge 5 commits into
huggingface:mainfrom
JakeStevens:bf16-support

Conversation

@JakeStevens

Copy link
Copy Markdown
Collaborator

Three changes:

  1. custom_sdpa stops force-upcasting to fp32.
  2. A narrow fallback keeps older runtimes working.
  3. bf16 XNNPACK delegation is requested when the model is bf16 as XnnpackPartitioner(enable_bf16=...) is opt-in

Note: enable_bf16 first appears in executorch==1.4.0.dev20260801

This means we cannot test with a stable version until a future branch cut post 1.4 for ET

Test Plan

  1. Environment
conda create -n oe-bf16 python=3.11 -y
conda activate oe-bf16
  1. Install
python install_dev.py --dependency_stack nightly
  1. Verify the env
python -c "import executorch.version as v, torch, torchao, transformers; \
print(v.__version__, torch.__version__, torchao.__version__, transformers.__version__)"
# expect: 1.4.0.dev20260802+cpu 2.14.0.dev20260801+cpu 0.19.0.dev20260802+cpu 5.0.0rc0
  1. Export
optimum-cli export executorch \
    --model google/gemma-3-1b-it \
    --task text-generation \
    --recipe xnnpack \
    --use_custom_sdpa \
    --use_custom_kv_cache \
    --qlinear 8da4w \
    --qembedding 8w \
    --output_dir hf_gemma3_bf16 \
    --dtype bfloat16
  1. Check it runs
from transformers import AutoConfig, AutoTokenizer
from optimum.executorch import ExecuTorchModelForCausalLM

model_id = "google/gemma-3-1b-it"
model = ExecuTorchModelForCausalLM.from_pretrained(
    "hf_gemma3_bf16", config=AutoConfig.from_pretrained(model_id))
print(model.text_generation(
    tokenizer=AutoTokenizer.from_pretrained(model_id),
    prompt="Write a poem about machine learning.", max_seq_len=64))

Coherent output at ~11.9 tok/s: "The data flows, a river deep, / With patterns hidden, secrets sleep..."

GregoryComer and others added 4 commits July 22, 2026 11:05
ExecuTorch's custom_sdpa only gained f16/bf16 support in 1.4; before that it
asserts float32, so the previous commit's removal of the unconditional upcast
breaks half-dtype exports on older runtimes.

Attempt the native call and upcast only if the op actually rejects the dtype,
rather than comparing versions -- source builds report versions such as
1.4.0a0+<sha>, which PEP 440 sorts after every 1.4.0.devN nightly and would
therefore be misread as new enough. The recovery is deliberately narrow: it
requires a half input dtype and an assertion mentioning float32 (executorch
1.3.1 raises "Expected query to be float32 but got ..."), so every unrelated
assertion still propagates.
bf16 fully-connected delegation is opt-in in the XNNPACK backend (it
requires a new enough XNNPACK), so it must be explicitly requested via
XnnpackPartitioner(enable_bf16=...). Enable it only when the exported
model carries bf16 tensors (i.e. exported with --dtype bfloat16; a bf16
model always has bf16 tensors such as RMSNorm weights, even when linears
are quantized), leaving non-bf16 exports unchanged.
XnnpackPartitioner(enable_bf16=...) only exists in executorch
1.4.0.dev20260801 and later. The pinned nightly stack predated it
(dev20260714), so the flag added in the previous commit was accepted and
silently ignored: CI exercised the no-op path and every bf16 matmul stayed
off the delegate. Bump the nightly stack to dev20260802, which was
validated end to end (gemma-3-1b-it, bf16 + 8da4w + 8w, exports and
generates).

Also fail loudly when the installed executorch cannot delegate bf16, since
the downstream symptom is unhelpful. Detect the capability by probing the
partitioner rather than comparing versions: source builds report versions
such as 1.4.0a0+<sha>, and PEP 440 sorts .dev before a0, so a version check
would wave them through whether or not they carry the feature.

Two severities, because the failure modes differ:

  * with quantized linears, raise -- torchao's affine quant ops only lower
    inside an XNNPACK partition, so skipping bf16 linears strands them in
    the graph and to_executorch dies with "Missing out variants:
    torchao::..." far from the cause;
  * without them, warn -- the export still succeeds, it just leaves every
    bf16 operator on the portable kernels (312 vs 1094 delegated nodes on
    SmolLM2-135M), so raising would regress a working path.
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The try/except fallback around llama.custom_sdpa never fired where it
mattered:

- Under torch.export, dynamo wraps the meta kernel's AssertionError in a
  TorchRuntimeError before it reaches the call site, so exporting an
  f16/bf16 model against ExecuTorch < 1.4 hard-failed instead of falling
  back (gemma3/qwen3 float16 CI).
- In eager, the AOT op library's half kernel doesn't raise at all. It
  logs 'No temp allocator provided' (1.4) or 'Invalid arguments' (1.3.1)
  and returns garbage, so bf16 eager generation produced empty output
  (common CI).

Decide up front instead: probe the meta kernel once at import time for
half support, and only skip the fp32 upcast when we are building an
exported graph, which is the one place half dtypes are known to work
end to end.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants