[gfx1250] Fuse A4W4 stage1 FP4 quantization

[XiaobingSuper]

Summary

• extend the grouped gfx1250 TDM GEMM1 epilogue to emit packed MXFP4 payloads and preshuffled E8M0 scales directly after SiLU, SwiGLU, or SiTUv2
• compute the E8M0 scale directly from the FP32 activation, convert once to BF16 for the native gfx1250 8-wide FP4 pack, and exchange packed BF16 dwords to halve cross-kgrp shuffles
• route supported bias-free A4W4 shapes through the fused epilogue and retain the standalone quant fallback for unsupported tile geometry or bias
• add coverage for all three activations and scale layouts with quant_wmma_rep 1, 4, and 8

Performance

Kimi-K3 TP4 grouped MoE shape on gfx1250: E=896, topk=16, model_dim=3584, inter_dim=768, SiTUv2, A4W4, default 64/64 tiles. Numbers are end-to-end operator latency, averaged across two independent runs of 20 warmups / 300 iterations each.

Tokens	Before (unfused)	After (fused)	Latency reduction
1	73.01 us	47.39 us	35.1%
2	84.26 us	59.86 us	29.0%
4	109.57 us	87.42 us	20.2%
8	140.69 us	118.90 us	15.5%
16	217.18 us	201.41 us	7.3%
32	338.73 us	324.80 us	4.1%
64	474.46 us	455.99 us	3.9%
128	622.69 us	599.87 us	3.7%
256	704.41 us	676.86 us	3.9%
1024	753.56 us	729.37 us	3.2%
4096	1396.36 us	1351.28 us	3.2%

Correctness is unchanged across the sweep (logits_diff 3.39e-06 to 3.52e-06, rel_l2 2.60e-03 to 2.65e-03).

Test plan

• 5 fused A4W4 activation/scale-layout pytest cases on gfx1250
• A8W4 fused-quant regression check
• Kimi-K3 tokens 1 through 4096 operator correctness and performance checks
• bias-enabled A4W4 fallback check
• Ruff, Black, Python syntax, and whitespace checks

Made with Cursor

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:triton-300x	Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4607 --add-label <label>

[Copilot]

Pull request overview

This PR extends the gfx1250 FlyDSL grouped TDM GEMM1 epilogue to optionally emit packed MXFP4 (A4W4) payloads plus preshuffled E8M0 scales directly after the activation (SiLU/SwiGLU/SiTUv2), and updates routing + tests to exercise the new fused path.

Changes:

• Add a gfx1250 native BF16→FP4 scaled-pack IR helper and wire it into the GEMM1 fused activation+quant epilogue (stage1_quant_out=2).
• Update gfx1250 grouped MoE routing to use the fused quant epilogue for bias-free, supported tile geometries; preserve fallback for other cases.
• Add pytest coverage for fused A4W4 across activations and scale layouts (quant_wmma_rep 1/4/8 via tile_m overrides).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
op_tests/test_flydsl_grouped_gemm_gfx1250.py	Adds fused-A4W4 activation and scale-layout test coverage for gfx1250.
aiter/ops/flydsl/kernels/quant_utils.py	Introduces v_cvt_scalef32_pk8_fp4_bf16 inline-asm helper for native FP4 packing.
aiter/ops/flydsl/kernels/mxfp4_preshuffle_gfx1250_tdm.py	Extends GEMM1 epilogue to support packed FP4 output + direct scale preshuffle.
aiter/ops/flydsl/grouped_moe_gfx1250.py	Adjusts routing/allocations so eligible A4W4 bias-free shapes use the fused epilogue.
aiter/ops/flydsl/batched_gemm_mxfp4.py	Updates public launcher docs/validation for stage1_quant_out values (now 0/1/2).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.