Add direct CANN FAI decode attention for Qwen3#755
Conversation
|
Warning Review limit reached
Next review available in: 15 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (72)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request restructures the Qwen3-14B decode layer to minimize critical-path latency. Key optimizations include consolidating projection tasks to SPMD, merging zero-fill seeds, implementing early dispatch and resolution flags, and splitting the out_proj tasks. Additionally, the custom flash-decoding attention implementation is replaced with a direct external CCE paged attention kernel (paged_attention_cce) matching vLLM's physical BSND cache layout. The review feedback highlights a critical precision requirement on Ascend hardware: when casting from FP32 to BF16 via pl.cast, the mode="rint" parameter should be explicitly specified to ensure standard round-to-nearest-even behavior and avoid potential precision regressions in the cache and query assemblies.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
7a7122d to
b7c2d5b
Compare
|
Addressed the numerical review feedback in b7c2d5b: the new Q/K/V FP32-to-BF16 conversions now use explicit mode="rint". The same update also restores Huawei attribution and adds the CANN OSLA v2 license and provenance notice. |
|
Latest CI triage (run 29183004763): Those jobs install PyPTO from current |
b7c2d5b to
a3dea66
Compare
ChaoWao
changed the title
a3dea66 to
1435da5
Compare
1fb03e2 to
ee40267
Compare
ChaoWao
changed the title
216e9fa to
ed4377c
Compare
- Call CANN FusedInferAttentionScore through pl.jit.extern while preserving the fused QKV, QK-norm, RoPE, projection, MLP, and FP32 LM-head paths. - Keep active-TND query/output and paged-BSND KV storage without a materialized cache layout conversion. - Specialize attention for B1 and true B16, and fingerprint the BSND, batch, and A2/A3 platform contract. - Vendor the required CANN kernel subset with runtime tiling and retain the four-scope prefill capacity boundary. - Keep host tuple returns compatible with current PyPTO specialization and allocate 2 GiB for the Qwen serving CI warmup.
ed4377c to
10d43e5
Compare
1 participant
Summary
FusedInferAttentionScoreand invoked throughpl.jit.extern.include_dirssupport are available in merged PyPTO #2009.Testing
python -m pytest -q tests/golden tests/contract: 189 passed.decode_fwd.py -p a2a3sim --smoke: all 37 generated functions compiled.b2f347fcompiled the complete serving HOST decode wrapper toDistributedCompiledProgram.task-submittasktask_20260713_013155_194847010745.Compatibility
Merge order
The non-required serving check currently fails before NPU execution because it intentionally tests pypto-serving main, whose old HOST tuple wrapper is incompatible with current PyPTO specialization. The companion fix and full hardware validation are in pypto-serving #69.