upload checkpoint

README.md

@@ -1,3 +1,137 @@
----
-license: mit
----
+---
+language:
+- en
+library_name: transformers
+tags:
+- glm
+- MOE
+- pruning
+- compression
+license: mit
+name: cerebras/GLM-4.7-REAP-268B-A32B
+description: >
+  This model was obtained by uniformly pruning 25% of experts in GLM-4.7 using the REAP method.
+readme: >
+  https://huggingface.co/cerebras/GLM-4.7-REAP-268B-A32B/main/README.md
+license_link: https://huggingface.co/zai-org/GLM-4.7/blob/main/LICENSE
+pipeline_tag: text-generation
+base_model:
+- zai-org/GLM-4.7
+---
+
+<p align="center">
+  <em>𓌳 <strong>REAP</strong>𓌳  the Experts: Why Pruning Prevails for One-Shot MoE Compression</em><br>
+  <img src="https://i.imgur.com/rmzG3gg.png" alt="REAP" width="75%">
+</p>
+
+# GLM-4.7-REAP-268B-A32B
+
+## ✨ Highlights
+
+Introducing **GLM-4.7-REAP-268B-A32B**, a **memory-efficient compressed variant** of GLM-4.7 that maintains near-identical performance while being **25% lighter**.
+
+This model was created using **REAP (Router-weighted Expert Activation Pruning)**, a novel expert pruning method that selectively removes redundant experts while preserving the router's independent control over remaining experts. Key features include:
+
+- **Near-Lossless Performance**: Maintains almost identical accuracy on code generation, agentic coding, and function calling tasks compared to the full 355B model
+- **25% Memory Reduction**: Compressed from 355B to 268B parameters, significantly lowering deployment costs and memory requirements
+- **Preserved Capabilities**: Retains all core functionalities including code generation, agentic workflows, repository-scale understanding, and function calling
+- **Drop-in Compatibility**: Works with vanilla vLLM - no source modifications or custom patches required
+- **Optimized for Real-World Use**: Particularly effective for resource-constrained environments, local deployments, and academic research
+
+**For downstream low-bit quantization, we suggest using the [BF16 variant](https://huggingface.co/cerebras/GLM-4.7-REAP-268B-A32B).**
+
+---
+## 📋 Model Overview
+
+**GLM-4.7-REAP-268B-A32B** has the following specifications:
+
+- **Base Model**: GLM-4.7
+- **Compression Method**: REAP (Router-weighted Expert Activation Pruning)
+- **Compression Ratio**: 25% expert pruning
+- **Type**: Sparse Mixture-of-Experts (SMoE) Causal Language Model
+- **Number of Parameters**: 268B total, 32B activated per token
+- **Number of Layers**: 92
+- **Number of Attention Heads (GQA)**: 96 for Q and 8 for KV
+- **Number of Experts**: 120 (uniformly pruned from 160)
+- **Number of Activated Experts**: 8 per token
+- **Context Length**: 202,752 tokens
+- **License**: MIT
+
+---
+
+## 📊 Evaluations
+
+TBD for BF16 model. [Evalulation results available for the FP8 variant](https://huggingface.co/cerebras/GLM-4.7-REAP-268B-A32B-FP8#%F0%9F%93%8A-evaluations).
+
+For more details on the evaluation setup, refer to the [REAP arXiv preprint](https://arxiv.org/abs/2510.13999).
+
+---
+
+## 🚀 Deployment
+
+You can deploy the model directly using the **latest vLLM** (v0.11.0), no source modifications or custom patches required.
+
+```bash
+vllm serve cerebras/GLM-4.7-REAP-268B-A32B \
+    --tensor-parallel-size 8 \
+    --tool-call-parser glm45 \
+    --enable-auto-tool-choice \
+    --enable-expert-parallel
+```
+
+If you encounter insufficient memory when running this model, you might need to set a lower value for `--max-num-seqs` flag (e.g. set to 64).
+
+
+## 🧩 Model Creation
+
+This checkpoint was created by applying the **REAP (Router-weighted Expert Activation Pruning)** method uniformly across all Mixture-of-Experts (MoE) blocks of **GLM-4.7**, with a **25% pruning rate**.
+
+### How REAP Works
+
+REAP selects experts to prune based on a novel **saliency criterion** that considers both:
+- **Router gate values**: How frequently and strongly the router activates each expert
+- **Expert activation norms**: The magnitude of each expert's output contributions
+
+This dual consideration ensures that experts contributing minimally to the layer's output are pruned, while preserving those that play critical roles in the model's computations.
+
+### Key Advantages
+
+- **One-Shot Compression**: No fine-tuning required after pruning - the model is immediately ready for deployment
+- **Preserved Router Control**: Unlike expert merging methods, REAP maintains the router's independent, input-dependent control over remaining experts, avoiding "functional subspace collapse"
+- **Generative Task Superiority**: REAP significantly outperforms expert merging approaches on generative benchmarks (code generation, creative writing, mathematical reasoning) while maintaining competitive performance on discriminative tasks
+
+### Calibration
+
+The model was calibrated using a diverse mixture of domain-specific datasets including:
+- Code generation samples ([evol-codealpaca](https://huggingface.co/datasets/theblackcat102/evol-codealpaca-v1))
+- Function calling examples ([xlam-function-calling](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k))
+- Agentic multi-turn trajectories ([SWE-smith-trajectories](https://huggingface.co/datasets/SWE-bench/SWE-smith-trajectories))
+
+📚 For more details, refer to the following resources:
+
+- [🧾 arXiv Preprint](https://arxiv.org/abs/2510.13999)
+- [🧾 REAP Blog](https://www.cerebras.ai/blog/reap)
+- [💻 REAP Codebase (GitHub)](https://github.com/CerebrasResearch/reap)
+
+---
+
+## ⚖️ License
+
+This model is derived from
+**[`zai-org/GLM-4.7`](https://huggingface.co/zai-org/GLM-4.7)**
+and distributed under the **MIT license**.
+
+---
+
+## 🧾 Citation
+
+If you use this checkpoint, please cite the REAP paper:
+
+```bibtex
+@article{lasby-reap,
+  title={REAP the Experts: Why Pruning Prevails for One-Shot MoE compression},
+  author={Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan},
+  journal={arXiv preprint arXiv:2510.13999},
+  year={2025}
+}
+```

chat_template.jinja

@@ -0,0 +1,86 @@
+[gMASK]<sop>
+{%- if tools -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{{ tool | tojson(ensure_ascii=False) }}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) %}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {% set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{% for m in messages %}
+{%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set reasoning_content = '' %}
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- else %}
+    {%- if '</think>' in content %}
+        {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+        {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+    {%- endif %}
+{%- endif %}
+{%- if ((clear_thinking is defined and not clear_thinking) or loop.index0 > ns.last_user_index) and reasoning_content -%}
+{{ '<think>' + reasoning_content.strip() +  '</think>'}}
+{%- else -%}
+{{ '</think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{- '<tool_call>' + tc.name -}}
+{% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if m.content is string -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' }}
+{%- endif %}
+{{- '<tool_response>' }}
+{{- m.content }}
+{{- '</tool_response>' }}
+{%- else -%}
+<|observation|>{% for tr in m.content %}
+<tool_response>{{ tr.output if tr.output is defined else tr }}</tool_response>{% endfor -%}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '</think>' if (enable_thinking is defined and not enable_thinking) else '<think>' -}}
+{%- endif -%}

config.json

@@ -0,0 +1,43 @@
+{
+  "architectures": [
+    "Glm4MoeForCausalLM"
+  ],
+  "attention_bias": true,
+  "attention_dropout": 0.0,
+  "eos_token_id": [
+    151329,
+    151336,
+    151338
+  ],
+  "first_k_dense_replace": 3,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "max_position_embeddings": 202752,
+  "model_type": "glm4_moe",
+  "moe_intermediate_size": 1536,
+  "n_group": 1,
+  "n_routed_experts": 120,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 96,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 92,
+  "num_key_value_heads": 8,
+  "num_nextn_predict_layers": 0,
+  "pad_token_id": 151329,
+  "partial_rotary_factor": 0.5,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "routed_scaling_factor": 2.5,
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.55.0",
+  "use_cache": true,
+  "use_qk_norm": true,
+  "vocab_size": 151552
+}

model-00001-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ede6c6a0e9de3b7f67e6256ba08911318e8cb11fbef6b858817607f0c0ac554a
+size 5363662896

model-00002-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d6239e07c8745f600cc8fa2aecdadc6e34e296b0957dbb74ea00b5cce76607fa
+size 5354300984

model-00003-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60fcad3686cea0788d58ef8f6618a4679dd1b32e6b9cf2356e93f3096da1650b
+size 5354300984

model-00004-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2e88d666058abebeeec96ffe2437ef3960af68343f04c2c0e90d6a5af0bb306
+size 5364738088

model-00005-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:098d018a0099575ddbdc401454cfc0b815ef9f37d1124094aed8fb44994576a8
+size 5353071440

model-00006-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1cf2073694fa982cb5db204ffea925ce450f7381fa6fa470eb040c5d94dcc3df
+size 5354300952

model-00007-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e3db218d091de80f1a566c9bf5dfd64e8a94633205bc04c02681bad28e8b5b3
+size 5354300976

model-00008-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f020df07f5bc0524732077099dcedbbd472e451755cbb748000c5791062099b
+size 5354300984

model-00009-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c7f3421b15ccebcc80598af09e4b2fab8bfbc87301f9dd90d96a0eee740c4b1
+size 5354301160

model-00010-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e604cf2774246980a317b54958ee1af61116d6c3c2343e46d4643537ac50098
+size 5354301312

model-00011-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1bc7efe2718953ef60b829890788ab79a2a46f52ef9809a63138a818422c3429
+size 5354301320

model-00012-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35bc379a16b9d2928a0ebb59f2a9b7bbc09aae93531e4419c9c15416f76e4719
+size 5354301312

model-00013-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5cc32d4e390abc16cb7adf17b2abd613d6133a64c702cf5961af900eebf834d4
+size 5354301344

model-00014-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d833c7367750a4c669bc9c186c0f40c1a74e09b9d6d22dd96d4c8a483022410f
+size 5363508888

model-00015-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f46dbc7b61184d2842edf3b60898d8b097d44d0068897da6198a15b80979b9c6
+size 5354301272

model-00016-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a3a53658f561a1d13effa6204cc108b6ef72a438fe03ac9cb9cfb4126890a45
+size 5354301304

model-00017-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0340ebbd48bef6c01da36223594db9cb4d85616fd9edd5b013d91902b6670cdc
+size 5354301320

model-00018-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23be87625478ed570833863db47aedf016db87514d12500b732c750d726c1e48
+size 5354301312

model-00019-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c25ca828202d993c71bcec36792d149824a598797cfc499afd3c7f065acbae59
+size 5354301312

model-00020-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7f2f48ee4b803cc500912d401201a592b5da7eb14e709f321e4703927580afab
+size 5354301320

model-00021-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0681d7894d9cfaec8f1e150afd25bae19b508900d1cadd23dc00b875e1944f0d
+size 5354301312

model-00022-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6f78b9ca0308e3de0a96adcf01ab8e3105fb2867dd0f909d0bc02cf854c2a24d
+size 5354301320

model-00023-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bbc793310c41b9f92137c520341f8650b1b3703e798d46db7223cbe38378658d
+size 5349030376

model-00024-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3f9fc40d43cd9b1c79945205feb1b1ff54c2b1d8d1bb50c94e6483d028e4ba2
+size 5353051064

model-00025-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:88a46be217e6539c4a6c1b6521f6ddab78e4d4bec5a80fea53735403864a9a98
+size 5354301288

model-00026-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9eeb271e88637a083c34ccd1fb5d77fbe663d8a59ebbfcac9fbbc01f1b6d728a
+size 5354301312

model-00027-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5d7a146c292183e29db53d2b00d586ab6f8407e1323aaaf0da708d5cc64f7f1
+size 5354301312

model-00028-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91043910206957b011adf4f41312c8135119ac7d8aabd3cab55d3378f576f6ed
+size 5354301320

model-00029-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ecd6d24cabf8716cd7fac3faec581b754388cfd95d99914f31bcecd813ab389
+size 5354301312

model-00030-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:52a59db18bae58e6d1047f8cbc9691d2fdbff8181563d800a4cf77410b4b711a
+size 5354301312

model-00031-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d656d01405b685dff98cc32b803915e54270625404babb2d37fd79a5001cca4
+size 5354301320

model-00032-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9287bf6c56662d8a4c7a2efafbf6f720cbbe9c302affec19d274b0f566cc8b6c
+size 5354301344

model-00033-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74d4383d62d189b33e92925b36a5536feb06300803db4efbe5cb8953a515ce5d
+size 5363508888

model-00034-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07d6773470b989b7637570036884dab91f03b57a3e2be9255df7735ddacc6003
+size 5354301272

model-00035-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3dd45debdcbf8d94803847a8f70de910370f7a26028d3801aa3a435666d9c01c
+size 5354301304

model-00036-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd80b39bdfcc88b4eedb8dad8806894aa7d66c5936486ddda1e560f2fc3cdf64
+size 5354301312

model-00037-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5acf3e1720737b527a3e621988bda073ae8724122905bf02ce4bb5570467ea4b
+size 5354301320

model-00038-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e1ba43051565a9896c6fb283282df21d09e4a8a2ede3d0a5cae70a28e9544ed
+size 5354301312

model-00039-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:01ed5a3c07f07f842863a693ecd82c52b39d61546b884ef43a41039f82563d45
+size 5354301312

model-00040-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8fa3222b4e3e1bd4eea2863ffec0ff1fda7f8f3250c97a2db0466a59881d717
+size 5354301320

model-00041-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:377d568d9db28a1e17c87386acd2d2cd7309eaa6ba5890a9ad1ae9e672813e1b
+size 5354301320

model-00042-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2cb2db48589d7893d1f3d988988799198639dfd716a6101661d5268d9d1b875a
+size 5333301608

model-00043-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7e550ff1abc1873fcbbb8cf124b0d3eb131cc7a069c827280cf983bd096a6395
+size 5353051064

model-00044-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98d8d24e5958e8ace266dc7dcdf7bfb9b02d191ec5b7c008b3689622560d039b
+size 5354301288

model-00045-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09a7687e61faaebb4db6db2befc09bfd9e1f840a0f02d708f62e7ba7add5d134
+size 5354301312

model-00046-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d7aa431bd41d1309ff91d185976dea69c057f957021a418096dc6c3cdfdbca58
+size 5354301312

model-00047-of-00101.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ea18aaf3058d203df0e3f365221c5c69d4d3820ee50b2e9728dda13ec898fe9
+size 5354301320