Integrate Qwen3 CANN FAI decode

[ChaoWao]

Summary

• Pin pypto-lib #755 and run Qwen3 decode through its direct CANN FAI CCEC kernel.
• Dispatch active-prefix tensor views for B1 or true B16 while preserving the public [B, 8] token ABI. B1 no longer replicates inactive KV rows, and serving performs no KV-cache layout conversion.
• Restrict this executor to A2/A3 onboard and reject B2-B15 before decode dispatch.
• Map warmup reads and writes to scratch slot 0 instead of the invalid -1 slot.
• Rebind the HOST decode output to its declared out parameter so current PyPTO can specialize all tuple return shapes.
• Cover true B16 with 16 distinct prompt histories, each compared against its own B1 result.
• Use a 2 GiB ring for CI's B16/4096-token warmup while retaining #753's manual scope boundaries and 16K task/dep pools.

Testing

• python -m pytest -q tests/test_batching.py tests/test_parallel.py tests/test_device_sampling_submission.py: 40 passed.
• Headers, English-only, ruff on changed files, workflow YAML parsing, and git diff --check: passed.
• Current PyPTO b2f347f compiled the complete HOST decode wrapper to DistributedCompiledProgram.
• Hardware golden under task-submit: task task_20260713_002105_18464897737, 8/8 expected tokens matched.
• CI-shape B16/4096-token warmup with 2 GiB heap and 16K pools: task task_20260713_013155_194847010745, full accuracy passed.
• True independent B16 under task-submit: task task_20260712_234748_24151917638, 16 requests matched their independent B1 references without deadlock.
• A 3338-token, full 40-layer B1 serving run completed all 19 decode steps without device deadlock: TPOT 31.2 ms, decode-kernel wall average 30.82 ms, device Effective 28.53 ms.
• At that B1 shape, L2 records show a 42.39 us median slowest-AIV span inside the mixed attention launch and a 166.12 us median scheduler-visible attention stage including tiling and dispatch.

Dependencies

• PyPTO #2009 is merged.
• Depends on pypto-lib #755.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a6e497a3-2d85-401d-96a8-748089ef3296

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Code Review

This pull request refactors the Qwen3-14B NPU runner and executor to support dynamic public batching (batch 1 or full capacity) for the fused decode kernel, replacing the previous fixed-batch padding approach. It also removes the standalone token embedding kernel, as embedding lookup and greedy sampling are now integrated directly into the fused decode kernel. Feedback is provided regarding a potential shape mismatch in _stage_decode_inputs where slicing token_ids with :width could cause issues if sampled_ids_width is greater than 1.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.