feat(cpu): add Qwen3.5 4B mobile CPU support - #691
Conversation
📝 WalkthroughWalkthroughQwen3.5 support now covers 0.8B and 4B ARM CPU workflows, with official runtime-contract checks, KAI quantization validation, converted-model descriptor checks, updated configurations, runner validation, documentation, and expanded tests. ChangesQwen3.5 runtime and conversion support
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
examples/qwen3_5/validate_converted_model.py (1)
215-239: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winShared uint32-offset boundary duplicated as an unlinked literal/constant across two files.
validate_converted_model.pyhardcodes1 << 32inline forcrosses_uint32_offset, whiletest_validators.pyindependently definesUINT32_LIMIT = 1 << 32to assert the same boundary — a single exported constant would prevent the two from silently drifting apart.
examples/qwen3_5/validate_converted_model.py#L215-L239: define a module-levelUINT32_LIMIT = 1 << 32constant and use it in thecrosses_uint32_offsetcomputation instead of the inline literal.examples/qwen3_5/test_validators.py#L32-L34: importUINT32_LIMITfromvalidate_converted_modelinstead of redefining it locally.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/qwen3_5/validate_converted_model.py` around lines 215 - 239, Centralize the uint32 boundary as a module-level UINT32_LIMIT in examples/qwen3_5/validate_converted_model.py:215-239 and use it for crosses_uint32_offset instead of the inline 1 << 32 literal. In examples/qwen3_5/test_validators.py:32-34, import UINT32_LIMIT from validate_converted_model and remove the local duplicate definition.examples/qwen3_5/test_validators.py (1)
158-214: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider testing the coverage-mismatch and tied-embedding-violation branches of
validate_kai_recipe_contract.Existing tests cover hint mismatches and
linear_impl_typeresolution well, but the "missing/unexpected linear" coverage check and the tied-embeddingreplace/renameenforcement (both critical safety checks invalidate_checkpoint.py) aren't exercised by a negative test here.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/qwen3_5/test_validators.py` around lines 158 - 214, Extend test_kai_recipe_contract_rejects_runtime_incompatible_hints to add negative cases for validate_kai_recipe_contract’s missing/unexpected-linear coverage validation and tied-embedding replace/rename enforcement. Mutate the loaded quantization or runtime configuration minimally to trigger each branch, assert AssertionError, and keep the existing hint and linear_impl_type checks unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/qwen3_5/validate_checkpoint.py`:
- Around line 314-322: Guard the runtime configuration validation around
resolve_model_size so a missing runtime_config["text_config"] produces the
tool’s descriptive AssertionError instead of a raw KeyError. Update the
validation block after the model_type check, preserving the existing
runtime/checkpoint size comparison for configurations that include text_config.
In `@mllm/models/qwen3_5/configuration_qwen3_5.hpp`:
- Around line 182-255: Add documentation comments for the exported helpers
hasOfficialLayerSchedule, hasOfficialCommonRuntimeContract,
isOfficialQwen35_08BRuntimeConfig, isOfficialQwen35_4BRuntimeConfig,
matchesOfficialRuntimeContract, modelNameForConfig, and
validateModelConfigMatch. Describe each function’s purpose, configuration input,
return value where applicable, and document that validateModelConfigMatch throws
std::invalid_argument for unsupported runtime configurations, missing
embeddings, incompatible embedding dtype, or invalid embedding size/shape.
---
Nitpick comments:
In `@examples/qwen3_5/test_validators.py`:
- Around line 158-214: Extend
test_kai_recipe_contract_rejects_runtime_incompatible_hints to add negative
cases for validate_kai_recipe_contract’s missing/unexpected-linear coverage
validation and tied-embedding replace/rename enforcement. Mutate the loaded
quantization or runtime configuration minimally to trigger each branch, assert
AssertionError, and keep the existing hint and linear_impl_type checks
unchanged.
In `@examples/qwen3_5/validate_converted_model.py`:
- Around line 215-239: Centralize the uint32 boundary as a module-level
UINT32_LIMIT in examples/qwen3_5/validate_converted_model.py:215-239 and use it
for crosses_uint32_offset instead of the inline 1 << 32 literal. In
examples/qwen3_5/test_validators.py:32-34, import UINT32_LIMIT from
validate_converted_model and remove the local duplicate definition.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 99584525-da62-43e6-916c-d6fabe8f80ac
📒 Files selected for processing (14)
README.mdexamples/qwen3_5/README.mdexamples/qwen3_5/config_0.8B_w4a32_kai.jsonexamples/qwen3_5/config_4B_w4a32_kai.jsonexamples/qwen3_5/main.cppexamples/qwen3_5/quant_cfg_4B_w4a32_kai.jsonexamples/qwen3_5/test_validators.pyexamples/qwen3_5/validate_checkpoint.pyexamples/qwen3_5/validate_converted_model.pymllm/models/qwen3_5/configuration_qwen3_5.hppmllm/models/qwen3_5/modeling_qwen3_5.hpppymllm/mobile/tests/test_convertor.pytests/cpu/CMakeLists.txttests/cpu/Qwen35ConfigTest.cpp
- validate_kai_recipe_contract now rejects a missing or non-object runtime text_config with a descriptive AssertionError instead of raising KeyError - document the exported Qwen3.5 runtime-contract helpers in configuration_qwen3_5.hpp, including validateModelConfigMatch's std::invalid_argument cases - centralize the unsigned 32-bit offset boundary as UINT32_LIMIT in validate_converted_model.py and import it in the tests - add negative coverage for missing/unexpected KAI Linear coverage, invalid tied-embedding entries, and malformed runtime text_config
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
mllm/models/qwen3_5/configuration_qwen3_5.hpp (1)
198-213: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winValidate every special-token ID in the runtime contract.
The contract description includes special-token IDs, but only
eos_token_idandim_end_token_idare checked.end_of_text_token_id,im_start_token_id, and both thinking IDs retain defaults because they are not parsed or validated, so an incompatible config can be accepted. Load and compare all supported token IDs.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@mllm/models/qwen3_5/configuration_qwen3_5.hpp` around lines 198 - 213, Extend hasOfficialCommonRuntimeContract to load and validate every supported special-token ID, including end_of_text_token_id, im_start_token_id, and both thinking IDs, alongside the existing eos_token_id and im_end_token_id checks. Ensure these IDs are parsed from the configuration before comparison so incompatible values cannot pass validation.
🧹 Nitpick comments (1)
examples/qwen3_5/test_validators.py (1)
216-235: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd docstrings to the new test methods.
Each new public test function needs a brief docstring describing its rejected contract case.
As per coding guidelines, public APIs, classes, and functions must have clear docstrings or comments explaining purpose, parameters, returns, and errors.
Also applies to: 237-260, 262-299, 301-321
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/qwen3_5/test_validators.py` around lines 216 - 235, Add brief docstrings to each newly added public test method, including test_kai_recipe_contract_rejects_missing_linear_coverage and the methods in the referenced ranges, describing the rejected contract case each test verifies. Keep the existing test logic unchanged.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@mllm/models/qwen3_5/configuration_qwen3_5.hpp`:
- Around line 198-213: Extend hasOfficialCommonRuntimeContract to load and
validate every supported special-token ID, including end_of_text_token_id,
im_start_token_id, and both thinking IDs, alongside the existing eos_token_id
and im_end_token_id checks. Ensure these IDs are parsed from the configuration
before comparison so incompatible values cannot pass validation.
---
Nitpick comments:
In `@examples/qwen3_5/test_validators.py`:
- Around line 216-235: Add brief docstrings to each newly added public test
method, including test_kai_recipe_contract_rejects_missing_linear_coverage and
the methods in the referenced ranges, describing the rejected contract case each
test verifies. Keep the existing test logic unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 8173d3c3-74d9-4433-b2f5-009f7a28f029
📒 Files selected for processing (4)
examples/qwen3_5/test_validators.pyexamples/qwen3_5/validate_checkpoint.pyexamples/qwen3_5/validate_converted_model.pymllm/models/qwen3_5/configuration_qwen3_5.hpp
🚧 Files skipped from review as they are similar to previous changes (1)
- examples/qwen3_5/validate_converted_model.py
2 participants
Summary
This adds Qwen3.5-4B text-only support to the existing mllm v2 mobile CPU
path. It reuses the Qwen3.5 GDN/full-attention runtime and KAI Linear path
already used by 0.8B; this PR does not add a separate 4B kernel or quantization
format.
Review map
config_4B_w4a32_kai.jsonandquant_cfg_4B_w4a32_kai.json4 GiB
runner documentation, and the supported-model table
Supported contract
The runner accepts only the official 0.8B and 4B runtime contracts and checks
the model embedding descriptor before constructing the model or its eager KV
cache.
Validation
The completed H20
final03gates are bound to an exact 1,865-entry sourcemanifest verified before and after the run.
Qwen/Qwen3.5-4Bpinned revision; 426 selected text tensors; 249/249 KAI targets--helpalso passed28.1.13356709), API 28,arm64-v8a; runner, five focused test binaries, and required shared libraries passed AArch64 ELF/dependency checksThe pinned V2 artifact was generated by the audited conversion run and
revalidated by
final03;final03did not regenerate it. The Android result iscross-build evidence, not device-execution evidence.
Exact evidence identities
b27f801c0bee0b880e5a0298611a3c2c70c7bae8final03source manifest SHA-256:d8fb8a6307e48c514ddb61c5b853a63224ff2e9fc9ab12b27844befc48187f6e851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a79b130ce333c44c15e7420f7dc3ce42badf374358aefe9be7acd38f5bd3d13e6Pixel 9 Pro XL gates
final03runner handled two identical one-token prompts in one process, producing the same token ID after each state reset; strict UTF-8 and no error markerBoth gates used the artifact-bound
final03device bundle. These are focusedcorrectness/safety gates, not numerical-parity or performance claims.
Quantization naming
The user-facing contract is W4A8: KAI Linear dynamically quantizes
activations to INT8 and uses packed INT4 weights while retaining FP32 operator
inputs and outputs.
The existing
w4a32_kaiconfig names, model filename convention, andw4a32_kai_pipelineidentifier remain unchanged for compatibility.Out of scope
Tracks #651.
Extends the Qwen3.5-0.8B CPU support merged in #690.
Summary by CodeRabbit
New Features
Documentation
Bug Fixes