Enable bf16 export for XNNPACK backend - #247
Open
JakeStevens wants to merge 5 commits into
Open
Conversation
ExecuTorch's custom_sdpa only gained f16/bf16 support in 1.4; before that it asserts float32, so the previous commit's removal of the unconditional upcast breaks half-dtype exports on older runtimes. Attempt the native call and upcast only if the op actually rejects the dtype, rather than comparing versions -- source builds report versions such as 1.4.0a0+<sha>, which PEP 440 sorts after every 1.4.0.devN nightly and would therefore be misread as new enough. The recovery is deliberately narrow: it requires a half input dtype and an assertion mentioning float32 (executorch 1.3.1 raises "Expected query to be float32 but got ..."), so every unrelated assertion still propagates.
bf16 fully-connected delegation is opt-in in the XNNPACK backend (it requires a new enough XNNPACK), so it must be explicitly requested via XnnpackPartitioner(enable_bf16=...). Enable it only when the exported model carries bf16 tensors (i.e. exported with --dtype bfloat16; a bf16 model always has bf16 tensors such as RMSNorm weights, even when linears are quantized), leaving non-bf16 exports unchanged.
XnnpackPartitioner(enable_bf16=...) only exists in executorch
1.4.0.dev20260801 and later. The pinned nightly stack predated it
(dev20260714), so the flag added in the previous commit was accepted and
silently ignored: CI exercised the no-op path and every bf16 matmul stayed
off the delegate. Bump the nightly stack to dev20260802, which was
validated end to end (gemma-3-1b-it, bf16 + 8da4w + 8w, exports and
generates).
Also fail loudly when the installed executorch cannot delegate bf16, since
the downstream symptom is unhelpful. Detect the capability by probing the
partitioner rather than comparing versions: source builds report versions
such as 1.4.0a0+<sha>, and PEP 440 sorts .dev before a0, so a version check
would wave them through whether or not they carry the feature.
Two severities, because the failure modes differ:
* with quantized linears, raise -- torchao's affine quant ops only lower
inside an XNNPACK partition, so skipping bf16 linears strands them in
the graph and to_executorch dies with "Missing out variants:
torchao::..." far from the cause;
* without them, warn -- the export still succeeds, it just leaves every
bf16 operator on the portable kernels (312 vs 1094 delegated nodes on
SmolLM2-135M), so raising would regress a working path.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
The try/except fallback around llama.custom_sdpa never fired where it mattered: - Under torch.export, dynamo wraps the meta kernel's AssertionError in a TorchRuntimeError before it reaches the call site, so exporting an f16/bf16 model against ExecuTorch < 1.4 hard-failed instead of falling back (gemma3/qwen3 float16 CI). - In eager, the AOT op library's half kernel doesn't raise at all. It logs 'No temp allocator provided' (1.4) or 'Invalid arguments' (1.3.1) and returns garbage, so bf16 eager generation produced empty output (common CI). Decide up front instead: probe the meta kernel once at import time for half support, and only skip the fp32 upcast when we are building an exported graph, which is the one place half dtypes are known to work end to end.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three changes:
Note: enable_bf16 first appears in executorch==1.4.0.dev20260801
This means we cannot test with a stable version until a future branch cut post 1.4 for ET
Test Plan
Coherent output at ~11.9 tok/s: "The data flows, a river deep, / With patterns hidden, secrets sleep..."