feat(cpu): add Qwen3.5 4B mobile CPU support

README.md

@@ -105,6 +105,7 @@ The mllm framework integrates seamlessly with popular community frameworks' chec
 | [Qwen3-1.7B](https://github.com/QwenLM/Qwen3)                     | [✔️ w4a8](https://www.modelscope.cn/models/mllmTeam/Qwen3-1.7B-w4a8-i8mm-kai)  | [W4A16-SM8650](https://modelscope.cn/models/mllmTeam/Qwen3-1.7B-Qnn-AOT-SM8650/) | |
 | [Qwen3-4B](https://github.com/QwenLM/Qwen3)                      | [✔️ w4a8](https://www.modelscope.cn/models/mllmTeam/Qwen3-4B-w4a8-i8mm-kai)  |  | |
 | [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B)           | [✔️ w4a8](./examples/qwen3_5/README.md) | | |
+| [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B)               | [✔️ w4a8](./examples/qwen3_5/README.md) | | |
 | [DeepSeek-OCR](https://github.com/deepseek-ai/DeepSeek-OCR)       | [✔️ w4a8](https://www.modelscope.cn/models/mllmTeam/DeepSeek-OCR-w4a8-i8mm-kai)  |  | |
 | [SmolLM3](https://huggingface.co/blog/smollm3)| [✔️ w4a8](https://www.modelscope.cn/models/mllmTeam/SmolLM3-3B-w4a8-i8mm-kai)  |  | |
 | [Qwen2-VL-2B-Instruct](https://qwenlm.github.io/zh/blog/qwen2-vl/)|[✔️ w4a8](https://www.modelscope.cn/models/mllmTeam/Qwen2-VL-2B-Instruct-w4a32kai) || |

examples/qwen3_5/README.md

@@ -1,27 +1,47 @@
-# Qwen3.5 0.8B on ARM CPU
+# Qwen3.5 0.8B and 4B on ARM CPU
 
-This example runs the text tower of `Qwen/Qwen3.5-0.8B`. The checkpoint also
-contains a vision tower and an MTP layer; they are intentionally excluded from
-the CPU model file.
+This example runs the text towers of
+[`Qwen/Qwen3.5-0.8B`](https://huggingface.co/Qwen/Qwen3.5-0.8B) and
+[`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B). Both checkpoints
+also contain a vision tower and an MTP layer; those components are
+intentionally excluded from the CPU model file.
 
-The model uses six full-attention layers and eighteen Gated Delta Net (GDN)
-layers. Both the GDN recurrence and its depthwise-convolution history are
-stateful across prefill and decode. `Qwen3_5ForCausalLM::resetState()` clears
-those states together with the full-attention KV cache.
+| Model | Hidden size | Layers | GDN layers | Full-attention layers | GDN key/value heads |
+|-------|------------:|-------:|-----------:|----------------------:|--------------------:|
+| 0.8B  | 1024 | 24 | 18 | 6 | 16 / 16 |
+| 4B    | 2560 | 32 | 24 | 8 | 16 / 32 |
 
-## Convert the checkpoint
+Both mobile configurations limit the cache to 2048 tokens and support batch
+size 1. The GDN recurrence and its depthwise-convolution history are stateful
+across prefill and decode. `Qwen3_5ForCausalLM::resetState()` clears those
+states together with the full-attention KV cache.
 
-First verify that the checkpoint architecture, tensor shapes, and quantization
-coverage match this 0.8B CPU implementation:
+## Quantization
 
-```bash
-python examples/qwen3_5/validate_checkpoint.py \
-  /path/to/Qwen3.5-0.8B
-```
+The user-facing W4A8 configuration uses dynamic INT8 activations with INT4
+weights while retaining FP32 operator inputs and outputs. The existing
+`w4a32_kai` configuration names and conversion pipeline are retained for
+compatibility with mllm tooling.
+
+Every `nn::Linear`, including the small GDN `in_proj_a` and `in_proj_b` gates,
+is packed for the configured KAI runtime. Convolution weights, recurrent
+parameters, embeddings, and norms stay in float32. The tied embedding matrix is
+retained for token lookup and separately packed as `lm_head_out.weight` for
+KAI.
 
-Run the converter from the repository root:
+## Convert a checkpoint
+
+Run these commands from the repository root. The checkpoint audit verifies the
+official text architecture, tensor shapes, and quantization coverage before
+conversion.
+
+### Qwen3.5-0.8B
 
 ```bash
+python examples/qwen3_5/validate_checkpoint.py \
+  /path/to/Qwen3.5-0.8B \
+  --quant-config examples/qwen3_5/quant_cfg_0.8B_w4a32_kai.json
+
 python -m pymllm.mobile.utils.mllm_convertor \
   --input_path /path/to/Qwen3.5-0.8B \
   --output_path /path/to/qwen3.5-0.8b-w4a32-kai.mllm \
@@ -31,35 +51,71 @@ python -m pymllm.mobile.utils.mllm_convertor \
   --include_prefix model.language_model. \
   --format v2 \
   --verbose
+
+python examples/qwen3_5/validate_converted_model.py \
+  /path/to/qwen3.5-0.8b-w4a32-kai.mllm \
+  /path/to/Qwen3.5-0.8B \
+  --quant-config examples/qwen3_5/quant_cfg_0.8B_w4a32_kai.json \
+  --model-name Qwen3.5-0.8B
 ```
 
-The tied embedding matrix is retained for token lookup and separately packed as
-`lm_head_out.weight` for KAI. Every `nn::Linear`, including the small GDN
-`in_proj_a` and `in_proj_b` gates, is packed for the configured KAI runtime;
-convolution weights, recurrent parameters, embeddings, and norms stay in
-float32.
+### Qwen3.5-4B
 
-Linear uses dynamic INT8 activations with INT4 weights while retaining FP32
-operator inputs and outputs.
+```bash
+python examples/qwen3_5/validate_checkpoint.py \
+  /path/to/Qwen3.5-4B \
+  --quant-config examples/qwen3_5/quant_cfg_4B_w4a32_kai.json
 
-Audit the resulting V2 descriptors without loading the tensor data:
+python -m pymllm.mobile.utils.mllm_convertor \
+  --input_path /path/to/Qwen3.5-4B \
+  --output_path /path/to/qwen3.5-4b-w4a32-kai.mllm \
+  --model_name Qwen3.5-4B \
+  --cfg_path examples/qwen3_5/quant_cfg_4B_w4a32_kai.json \
+  --pipeline w4a32_kai_pipeline \
+  --include_prefix model.language_model. \
+  --format v2 \
+  --verbose
 
-```bash
 python examples/qwen3_5/validate_converted_model.py \
-  /path/to/qwen3.5-0.8b-w4a32-kai.mllm \
-  /path/to/Qwen3.5-0.8B
+  /path/to/qwen3.5-4b-w4a32-kai.mllm \
+  /path/to/Qwen3.5-4B \
+  --quant-config examples/qwen3_5/quant_cfg_4B_w4a32_kai.json \
+  --model-name Qwen3.5-4B
 ```
 
+Use the V2 model-file format for the 4B conversion. Its converted tensor data
+is expected to exceed 4 GiB. On the supported 64-bit host and arm64 targets,
+V2 stores descriptor sizes and offsets as 64-bit values. The converted-model
+audit checks every descriptor and walks the full offset chain without loading
+tensor payloads.
+
+The official 4B conversion peaked at about 29.3 GiB RSS and produced a
+4.9 GB model file. Use a host with at least 32 GiB of available memory and
+about 16 GB of free disk for the checkpoint, output, and working margin.
+Exact resource usage can vary by host and software environment.
+
 ## Run
 
+Select the model and matching configuration:
+
 ```bash
+# Qwen3.5-0.8B
 mllm-qwen3-5-runner \
   --model_path /path/to/qwen3.5-0.8b-w4a32-kai.mllm \
   --model_version v2 \
   --tokenizer_path /path/to/Qwen3.5-0.8B/tokenizer.json \
   --config_path examples/qwen3_5/config_0.8B_w4a32_kai.json \
   --prompt "Give a one-sentence introduction." \
   --max_new_tokens 32
+
+# Qwen3.5-4B
+mllm-qwen3-5-runner \
+  --model_path /path/to/qwen3.5-4b-w4a32-kai.mllm \
+  --model_version v2 \
+  --tokenizer_path /path/to/Qwen3.5-4B/tokenizer.json \
+  --config_path examples/qwen3_5/config_4B_w4a32_kai.json \
+  --prompt "Give a one-sentence introduction." \
+  --max_new_tokens 32
 ```
 
 Omit `--prompt` for the interactive loop. The CLI treats each prompt as an

examples/qwen3_5/config_0.8B_w4a32_kai.json

@@ -9,6 +9,7 @@
     "eos_token_id": 248044,
     "full_attention_interval": 4,
     "head_dim": 256,
+    "hidden_act": "silu",
     "hidden_size": 1024,
     "intermediate_size": 3584,
     "layer_types": [
@@ -42,6 +43,7 @@
     "linear_num_key_heads": 16,
     "linear_num_value_heads": 16,
     "linear_value_head_dim": 128,
+    "mamba_ssm_dtype": "float32",
     "max_position_embeddings": 262144,
     "num_attention_heads": 8,
     "num_hidden_layers": 24,

examples/qwen3_5/config_4B_w4a32_kai.json

@@ -0,0 +1,77 @@
+{
+  "architectures": [
+    "Qwen3_5ForConditionalGeneration"
+  ],
+  "model_type": "qwen3_5",
+  "text_config": {
+    "attention_bias": false,
+    "attn_output_gate": true,
+    "eos_token_id": 248044,
+    "full_attention_interval": 4,
+    "head_dim": 256,
+    "hidden_act": "silu",
+    "hidden_size": 2560,
+    "intermediate_size": 9216,
+    "layer_types": [
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention"
+    ],
+    "linear_conv_kernel_dim": 4,
+    "linear_key_head_dim": 128,
+    "linear_num_key_heads": 16,
+    "linear_num_value_heads": 32,
+    "linear_value_head_dim": 128,
+    "mamba_ssm_dtype": "float32",
+    "max_position_embeddings": 262144,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 32,
+    "num_key_value_heads": 4,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        11,
+        11,
+        10
+      ],
+      "partial_rotary_factor": 0.25,
+      "rope_theta": 10000000,
+      "rope_type": "default"
+    },
+    "tie_word_embeddings": true,
+    "vocab_size": 248320
+  },
+  "tie_word_embeddings": true,
+  "max_cache_length": 2048,
+  "linear_impl_type": "KaiLinear_f32_qai8dxp_qsi4c32p_mxk_nxk_qai8dxp1x8_qsi4c32p8x8_1x8x32"
+}

examples/qwen3_5/main.cpp

@@ -51,18 +51,20 @@ MLLM_MAIN({
     }
 
     auto cfg = mllm::models::qwen3_5::Qwen3_5Config(config_path.get());
-    auto tokenizer = mllm::models::qwen3_5::Qwen3_5Tokenizer(tokenizer_path.get());
-    auto model = mllm::models::qwen3_5::Qwen3_5ForCausalLM(cfg);
     int generation_limit = max_new_tokens.isSet() ? max_new_tokens.get() : 64;
     if (generation_limit <= 0 || generation_limit > cfg.max_cache_length) {
       throw std::invalid_argument("max_new_tokens must be between 1 and max_cache_length");
     }
     if (prompt.isSet() && prompt.get().empty()) { throw std::invalid_argument("prompt must not be empty"); }
 
-    fmt::print("Qwen3.5 0.8B: {} layers ({} full attention + {} GDN)\n", cfg.num_hidden_layers, cfg.numFullAttentionLayers(),
-               cfg.numGDNLayers());
-
     auto param = mllm::load(model_path.get(), file_version);
+    mllm::models::qwen3_5::validateModelConfigMatch(cfg, param);
+
+    auto tokenizer = mllm::models::qwen3_5::Qwen3_5Tokenizer(tokenizer_path.get());
+    auto model = mllm::models::qwen3_5::Qwen3_5ForCausalLM(cfg);
+    fmt::print("{}: {} layers ({} full attention + {} GDN)\n", mllm::models::qwen3_5::modelNameForConfig(cfg),
+               cfg.num_hidden_layers, cfg.numFullAttentionLayers(), cfg.numGDNLayers());
+
     model.load(param);
 
     fmt::print("\n{:*^60}\n", prompt.isSet() ? " Qwen3.5 One-shot CLI " : " Qwen3.5 Interactive CLI ");