Model save

README.md

@@ -0,0 +1,58 @@
+---
+base_model: Qwen/Qwen1.5-MoE-A2.7B
+library_name: transformers
+model_name: Qwen1.5-MOE-aux-free-sft-math7k-1e-3-gamma
+tags:
+- generated_from_trainer
+- trl
+- sft
+licence: license
+---
+
+# Model Card for Qwen1.5-MOE-aux-free-sft-math7k-1e-3-gamma
+
+This model is a fine-tuned version of [Qwen/Qwen1.5-MoE-A2.7B](https://huggingface.co/Qwen/Qwen1.5-MoE-A2.7B).
+It has been trained using [TRL](https://github.com/huggingface/trl).
+
+## Quick start
+
+```python
+from transformers import pipeline
+
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="HectorHe/Qwen1.5-MOE-aux-free-sft-math7k-1e-3-gamma", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
+
+## Training procedure
+
+[<img src="https://raw.githubusercontent.com/wandb/assets/main/wandb-github-badge-28.svg" alt="Visualize in Weights & Biases" width="150" height="24"/>](https://wandb.ai/hector_-carnegie-mellon-university/huggingface/runs/26r47xsq) 
+
+
+This model was trained with SFT.
+
+### Framework versions
+
+- TRL: 0.16.0.dev0
+- Transformers: 4.51.0
+- Pytorch: 2.6.0
+- Datasets: 4.0.0
+- Tokenizers: 0.21.4
+
+## Citations
+
+
+
+Cite TRL as:
+    
+```bibtex
+@misc{vonwerra2022trl,
+	title        = {{TRL: Transformer Reinforcement Learning}},
+	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallouédec},
+	year         = 2020,
+	journal      = {GitHub repository},
+	publisher    = {GitHub},
+	howpublished = {\url{https://github.com/huggingface/trl}}
+}
+```

all_results.json

@@ -0,0 +1,8 @@
+{
+    "total_flos": 2.0144468407196058e+17,
+    "train_loss": 0.325891209826913,
+    "train_runtime": 788.7208,
+    "train_samples": 6851,
+    "train_samples_per_second": 8.686,
+    "train_steps_per_second": 0.273
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "attn_implementation": "flash_attention_2",
+  "bos_token_id": 151643,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "transformers_version": "4.51.0",
+  "use_cache": false
+}

moe_bias_states.json

@@ -0,0 +1,1667 @@
+{
+  "metadata": {
+    "total_moe_layers": 24,
+    "save_timestamp": "2025-09-15T02:07:52.402156",
+    "model_type": "Qwen2MoeForCausalLM",
+    "pytorch_version": "2.6.0+cu124",
+    "description": "Auxiliary-loss-free MoE bias states saved during training"
+  },
+  "moe_bias_states": {
+    "model.layers.0.mlp": {
+      "bias_values": [
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.474609375,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.443359375,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.1.mlp": {
+      "bias_values": [
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.017822265625,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.490234375,
+        0.5,
+        -0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.2.mlp": {
+      "bias_values": [
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.458984375,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.3.mlp": {
+      "bias_values": [
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.310546875,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.419921875,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.4.mlp": {
+      "bias_values": [
+        0.5,
+        -0.5,
+        0.5,
+        0.490234375,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.5.mlp": {
+      "bias_values": [
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.6.mlp": {
+      "bias_values": [
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.2451171875,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.7.mlp": {
+      "bias_values": [
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.2333984375,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.8.mlp": {
+      "bias_values": [
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.474609375,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.9.mlp": {
+      "bias_values": [
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.474609375,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.318359375,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.419921875,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.10.mlp": {
+      "bias_values": [
+        -0.5,
+        0.1435546875,
+        -0.5,
+        -0.5,
+        -0.458984375,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.458984375,
+        -0.490234375,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.482421875,
+        0.5,
+        0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.11.mlp": {
+      "bias_values": [
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.490234375,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.474609375,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.326171875,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.12.mlp": {
+      "bias_values": [
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.458984375,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.490234375,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.404296875,
+        0.5,
+        0.5,
+        0.5,
+        -0.427734375,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.13.mlp": {
+      "bias_values": [
+        -0.5,
+        0.2119140625,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.412109375,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.451171875,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.388671875,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.14.mlp": {
+      "bias_values": [
+        0.5,
+        0.5,
+        0.5,
+        0.435546875,
+        -0.5,
+        -0.5,
+        -0.2412109375,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.2138671875,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.404296875,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.15.mlp": {
+      "bias_values": [
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.349609375,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.16.mlp": {
+      "bias_values": [
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.01385498046875,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.474609375,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.2431640625,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.17.mlp": {
+      "bias_values": [
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.373046875,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.1943359375,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.451171875,
+        -0.5,
+        -0.443359375,
+        -0.5,
+        0.5,
+        0.396484375,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.490234375,
+        0.5,
+        0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.18.mlp": {
+      "bias_values": [
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.419921875,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.474609375,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.451171875,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.443359375,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.19.mlp": {
+      "bias_values": [
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.388671875,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.474609375,
+        -0.5,
+        0.5,
+        -0.5,
+        0.271484375,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.20.mlp": {
+      "bias_values": [
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.466796875,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.490234375,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.482421875,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.396484375,
+        -0.5,
+        -0.490234375,
+        0.443359375,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.21.mlp": {
+      "bias_values": [
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.2060546875,
+        -0.5,
+        -0.380859375,
+        -0.5,
+        -0.5,
+        0.458984375,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.396484375,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.326171875,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.22.mlp": {
+      "bias_values": [
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.341796875,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.384765625,
+        -0.5,
+        -0.5,
+        -0.365234375,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    },
+    "model.layers.23.mlp": {
+      "bias_values": [
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.404296875,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.380859375,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.1572265625,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5,
+        -0.5,
+        0.5,
+        -0.5,
+        -0.5
+      ],
+      "bias_update_speed": 0.001,
+      "num_experts": 60,
+      "module_type": "Qwen2MoeSparseMoeBlock",
+      "device": "cuda:0",
+      "dtype": "torch.bfloat16"
+    }
+  }
+}

train_results.json

@@ -0,0 +1,8 @@
+{
+    "total_flos": 2.0144468407196058e+17,
+    "train_loss": 0.325891209826913,
+    "train_runtime": 788.7208,
+    "train_samples": 6851,
+    "train_samples_per_second": 8.686,
+    "train_steps_per_second": 0.273
+}

trainer_state.json

@@ -0,0 +1,1763 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 215,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.004651162790697674,
+      "grad_norm": 19.566913604736328,
+      "learning_rate": 0.0,
+      "loss": 0.842,
+      "mean_token_accuracy": 0.8234314918518066,
+      "step": 1
+    },
+    {
+      "epoch": 0.009302325581395349,
+      "grad_norm": 21.503646850585938,
+      "learning_rate": 4.5454545454545457e-07,
+      "loss": 0.8773,
+      "mean_token_accuracy": 0.8220880627632141,
+      "step": 2
+    },
+    {
+      "epoch": 0.013953488372093023,
+      "grad_norm": 19.072912216186523,
+      "learning_rate": 9.090909090909091e-07,
+      "loss": 0.8522,
+      "mean_token_accuracy": 0.832254946231842,
+      "step": 3
+    },
+    {
+      "epoch": 0.018604651162790697,
+      "grad_norm": 18.07330322265625,
+      "learning_rate": 1.3636363636363636e-06,
+      "loss": 0.8166,
+      "mean_token_accuracy": 0.8297423720359802,
+      "step": 4
+    },
+    {
+      "epoch": 0.023255813953488372,
+      "grad_norm": 21.246156692504883,
+      "learning_rate": 1.8181818181818183e-06,
+      "loss": 0.9312,
+      "mean_token_accuracy": 0.8097259998321533,
+      "step": 5
+    },
+    {
+      "epoch": 0.027906976744186046,
+      "grad_norm": 18.991504669189453,
+      "learning_rate": 2.2727272727272728e-06,
+      "loss": 0.7816,
+      "mean_token_accuracy": 0.8285066485404968,
+      "step": 6
+    },
+    {
+      "epoch": 0.03255813953488372,
+      "grad_norm": 16.567394256591797,
+      "learning_rate": 2.7272727272727272e-06,
+      "loss": 0.8123,
+      "mean_token_accuracy": 0.8268486261367798,
+      "step": 7
+    },
+    {
+      "epoch": 0.037209302325581395,
+      "grad_norm": 10.308907508850098,
+      "learning_rate": 3.181818181818182e-06,
+      "loss": 0.6385,
+      "mean_token_accuracy": 0.8507297039031982,
+      "step": 8
+    },
+    {
+      "epoch": 0.04186046511627907,
+      "grad_norm": 8.591865539550781,
+      "learning_rate": 3.6363636363636366e-06,
+      "loss": 0.5651,
+      "mean_token_accuracy": 0.863410234451294,
+      "step": 9
+    },
+    {
+      "epoch": 0.046511627906976744,
+      "grad_norm": 9.019761085510254,
+      "learning_rate": 4.0909090909090915e-06,
+      "loss": 0.4933,
+      "mean_token_accuracy": 0.8713976144790649,
+      "step": 10
+    },
+    {
+      "epoch": 0.05116279069767442,
+      "grad_norm": 9.10396671295166,
+      "learning_rate": 4.5454545454545455e-06,
+      "loss": 0.4211,
+      "mean_token_accuracy": 0.8915254473686218,
+      "step": 11
+    },
+    {
+      "epoch": 0.05581395348837209,
+      "grad_norm": 7.293844223022461,
+      "learning_rate": 5e-06,
+      "loss": 0.4411,
+      "mean_token_accuracy": 0.8812285661697388,
+      "step": 12
+    },
+    {
+      "epoch": 0.06046511627906977,
+      "grad_norm": 4.323971271514893,
+      "learning_rate": 5.4545454545454545e-06,
+      "loss": 0.4205,
+      "mean_token_accuracy": 0.8759526610374451,
+      "step": 13
+    },
+    {
+      "epoch": 0.06511627906976744,
+      "grad_norm": 5.9293413162231445,
+      "learning_rate": 5.90909090909091e-06,
+      "loss": 0.364,
+      "mean_token_accuracy": 0.8886061906814575,
+      "step": 14
+    },
+    {
+      "epoch": 0.06976744186046512,
+      "grad_norm": 3.369750738143921,
+      "learning_rate": 6.363636363636364e-06,
+      "loss": 0.3435,
+      "mean_token_accuracy": 0.8966382145881653,
+      "step": 15
+    },
+    {
+      "epoch": 0.07441860465116279,
+      "grad_norm": 3.6587915420532227,
+      "learning_rate": 6.818181818181818e-06,
+      "loss": 0.3801,
+      "mean_token_accuracy": 0.8860718607902527,
+      "step": 16
+    },
+    {
+      "epoch": 0.07906976744186046,
+      "grad_norm": 3.0630006790161133,
+      "learning_rate": 7.272727272727273e-06,
+      "loss": 0.333,
+      "mean_token_accuracy": 0.9021540880203247,
+      "step": 17
+    },
+    {
+      "epoch": 0.08372093023255814,
+      "grad_norm": 3.0884928703308105,
+      "learning_rate": 7.727272727272727e-06,
+      "loss": 0.3774,
+      "mean_token_accuracy": 0.8831896185874939,
+      "step": 18
+    },
+    {
+      "epoch": 0.08837209302325581,
+      "grad_norm": 3.343445301055908,
+      "learning_rate": 8.181818181818183e-06,
+      "loss": 0.3834,
+      "mean_token_accuracy": 0.884019136428833,
+      "step": 19
+    },
+    {
+      "epoch": 0.09302325581395349,
+      "grad_norm": 2.862471580505371,
+      "learning_rate": 8.636363636363637e-06,
+      "loss": 0.3523,
+      "mean_token_accuracy": 0.8919129371643066,
+      "step": 20
+    },
+    {
+      "epoch": 0.09767441860465116,
+      "grad_norm": 3.1519503593444824,
+      "learning_rate": 9.090909090909091e-06,
+      "loss": 0.3994,
+      "mean_token_accuracy": 0.8742711544036865,
+      "step": 21
+    },
+    {
+      "epoch": 0.10232558139534884,
+      "grad_norm": 3.08125376701355,
+      "learning_rate": 9.545454545454547e-06,
+      "loss": 0.3518,
+      "mean_token_accuracy": 0.8841822147369385,
+      "step": 22
+    },
+    {
+      "epoch": 0.10697674418604651,
+      "grad_norm": 3.0018293857574463,
+      "learning_rate": 1e-05,
+      "loss": 0.345,
+      "mean_token_accuracy": 0.8950249552726746,
+      "step": 23
+    },
+    {
+      "epoch": 0.11162790697674418,
+      "grad_norm": 3.267643451690674,
+      "learning_rate": 9.999403846557509e-06,
+      "loss": 0.3704,
+      "mean_token_accuracy": 0.8792986273765564,
+      "step": 24
+    },
+    {
+      "epoch": 0.11627906976744186,
+      "grad_norm": 3.051391839981079,
+      "learning_rate": 9.99761554418511e-06,
+      "loss": 0.3331,
+      "mean_token_accuracy": 0.896356999874115,
+      "step": 25
+    },
+    {
+      "epoch": 0.12093023255813953,
+      "grad_norm": 3.030799627304077,
+      "learning_rate": 9.99463556670619e-06,
+      "loss": 0.3339,
+      "mean_token_accuracy": 0.9024479985237122,
+      "step": 26
+    },
+    {
+      "epoch": 0.12558139534883722,
+      "grad_norm": 3.1301109790802,
+      "learning_rate": 9.990464703686895e-06,
+      "loss": 0.3652,
+      "mean_token_accuracy": 0.8962038159370422,
+      "step": 27
+    },
+    {
+      "epoch": 0.13023255813953488,
+      "grad_norm": 3.1041147708892822,
+      "learning_rate": 9.985104060226937e-06,
+      "loss": 0.3538,
+      "mean_token_accuracy": 0.8938086032867432,
+      "step": 28
+    },
+    {
+      "epoch": 0.13488372093023257,
+      "grad_norm": 3.011711835861206,
+      "learning_rate": 9.978555056666784e-06,
+      "loss": 0.3763,
+      "mean_token_accuracy": 0.878600537776947,
+      "step": 29
+    },
+    {
+      "epoch": 0.13953488372093023,
+      "grad_norm": 3.104379653930664,
+      "learning_rate": 9.97081942821133e-06,
+      "loss": 0.3706,
+      "mean_token_accuracy": 0.8877567648887634,
+      "step": 30
+    },
+    {
+      "epoch": 0.14418604651162792,
+      "grad_norm": 3.3601248264312744,
+      "learning_rate": 9.961899224470146e-06,
+      "loss": 0.3956,
+      "mean_token_accuracy": 0.8821022510528564,
+      "step": 31
+    },
+    {
+      "epoch": 0.14883720930232558,
+      "grad_norm": 3.0410044193267822,
+      "learning_rate": 9.95179680891442e-06,
+      "loss": 0.318,
+      "mean_token_accuracy": 0.9060280323028564,
+      "step": 32
+    },
+    {
+      "epoch": 0.15348837209302327,
+      "grad_norm": 3.7731399536132812,
+      "learning_rate": 9.940514858250736e-06,
+      "loss": 0.3565,
+      "mean_token_accuracy": 0.8926541209220886,
+      "step": 33
+    },
+    {
+      "epoch": 0.15813953488372093,
+      "grad_norm": 2.947474241256714,
+      "learning_rate": 9.928056361711854e-06,
+      "loss": 0.3035,
+      "mean_token_accuracy": 0.9058524370193481,
+      "step": 34
+    },
+    {
+      "epoch": 0.16279069767441862,
+      "grad_norm": 3.0079967975616455,
+      "learning_rate": 9.914424620264714e-06,
+      "loss": 0.3581,
+      "mean_token_accuracy": 0.8900263905525208,
+      "step": 35
+    },
+    {
+      "epoch": 0.16744186046511628,
+      "grad_norm": 3.0129306316375732,
+      "learning_rate": 9.899623245735798e-06,
+      "loss": 0.3283,
+      "mean_token_accuracy": 0.8980950713157654,
+      "step": 36
+    },
+    {
+      "epoch": 0.17209302325581396,
+      "grad_norm": 3.025678873062134,
+      "learning_rate": 9.883656159854166e-06,
+      "loss": 0.3179,
+      "mean_token_accuracy": 0.9024977087974548,
+      "step": 37
+    },
+    {
+      "epoch": 0.17674418604651163,
+      "grad_norm": 3.019639253616333,
+      "learning_rate": 9.866527593212355e-06,
+      "loss": 0.3314,
+      "mean_token_accuracy": 0.899016261100769,
+      "step": 38
+    },
+    {
+      "epoch": 0.1813953488372093,
+      "grad_norm": 2.9794199466705322,
+      "learning_rate": 9.848242084145462e-06,
+      "loss": 0.3199,
+      "mean_token_accuracy": 0.9022731184959412,
+      "step": 39
+    },
+    {
+      "epoch": 0.18604651162790697,
+      "grad_norm": 2.8634836673736572,
+      "learning_rate": 9.82880447752868e-06,
+      "loss": 0.3377,
+      "mean_token_accuracy": 0.8929729461669922,
+      "step": 40
+    },
+    {
+      "epoch": 0.19069767441860466,
+      "grad_norm": 3.023061990737915,
+      "learning_rate": 9.808219923493606e-06,
+      "loss": 0.3331,
+      "mean_token_accuracy": 0.8968125581741333,
+      "step": 41
+    },
+    {
+      "epoch": 0.19534883720930232,
+      "grad_norm": 3.201726198196411,
+      "learning_rate": 9.786493876063685e-06,
+      "loss": 0.3291,
+      "mean_token_accuracy": 0.8950858116149902,
+      "step": 42
+    },
+    {
+      "epoch": 0.2,
+      "grad_norm": 3.0741653442382812,
+      "learning_rate": 9.763632091709125e-06,
+      "loss": 0.31,
+      "mean_token_accuracy": 0.8963869214057922,
+      "step": 43
+    },
+    {
+      "epoch": 0.20465116279069767,
+      "grad_norm": 2.884732723236084,
+      "learning_rate": 9.739640627821678e-06,
+      "loss": 0.3381,
+      "mean_token_accuracy": 0.8979052901268005,
+      "step": 44
+    },
+    {
+      "epoch": 0.20930232558139536,
+      "grad_norm": 3.245725631713867,
+      "learning_rate": 9.714525841109697e-06,
+      "loss": 0.2951,
+      "mean_token_accuracy": 0.9035796523094177,
+      "step": 45
+    },
+    {
+      "epoch": 0.21395348837209302,
+      "grad_norm": 3.057401180267334,
+      "learning_rate": 9.68829438591387e-06,
+      "loss": 0.3034,
+      "mean_token_accuracy": 0.9002149701118469,
+      "step": 46
+    },
+    {
+      "epoch": 0.2186046511627907,
+      "grad_norm": 3.0023374557495117,
+      "learning_rate": 9.660953212444116e-06,
+      "loss": 0.3138,
+      "mean_token_accuracy": 0.8987898826599121,
+      "step": 47
+    },
+    {
+      "epoch": 0.22325581395348837,
+      "grad_norm": 2.846851110458374,
+      "learning_rate": 9.632509564938073e-06,
+      "loss": 0.3213,
+      "mean_token_accuracy": 0.8954131603240967,
+      "step": 48
+    },
+    {
+      "epoch": 0.22790697674418606,
+      "grad_norm": 2.9974007606506348,
+      "learning_rate": 9.60297097974169e-06,
+      "loss": 0.3149,
+      "mean_token_accuracy": 0.8978151082992554,
+      "step": 49
+    },
+    {
+      "epoch": 0.23255813953488372,
+      "grad_norm": 2.860105037689209,
+      "learning_rate": 9.572345283312407e-06,
+      "loss": 0.3152,
+      "mean_token_accuracy": 0.8984004259109497,
+      "step": 50
+    },
+    {
+      "epoch": 0.2372093023255814,
+      "grad_norm": 3.178861141204834,
+      "learning_rate": 9.540640590145496e-06,
+      "loss": 0.3362,
+      "mean_token_accuracy": 0.8821947574615479,
+      "step": 51
+    },
+    {
+      "epoch": 0.24186046511627907,
+      "grad_norm": 2.8864493370056152,
+      "learning_rate": 9.507865300624057e-06,
+      "loss": 0.338,
+      "mean_token_accuracy": 0.8844265937805176,
+      "step": 52
+    },
+    {
+      "epoch": 0.24651162790697675,
+      "grad_norm": 2.9779324531555176,
+      "learning_rate": 9.474028098793277e-06,
+      "loss": 0.3133,
+      "mean_token_accuracy": 0.8958209753036499,
+      "step": 53
+    },
+    {
+      "epoch": 0.25116279069767444,
+      "grad_norm": 2.81357479095459,
+      "learning_rate": 9.439137950059539e-06,
+      "loss": 0.3293,
+      "mean_token_accuracy": 0.8908188343048096,
+      "step": 54
+    },
+    {
+      "epoch": 0.2558139534883721,
+      "grad_norm": 2.7432124614715576,
+      "learning_rate": 9.403204098814965e-06,
+      "loss": 0.3289,
+      "mean_token_accuracy": 0.8898541927337646,
+      "step": 55
+    },
+    {
+      "epoch": 0.26046511627906976,
+      "grad_norm": 2.78106689453125,
+      "learning_rate": 9.366236065988053e-06,
+      "loss": 0.3222,
+      "mean_token_accuracy": 0.8953770995140076,
+      "step": 56
+    },
+    {
+      "epoch": 0.2651162790697674,
+      "grad_norm": 2.4186606407165527,
+      "learning_rate": 9.32824364652104e-06,
+      "loss": 0.2825,
+      "mean_token_accuracy": 0.908851146697998,
+      "step": 57
+    },
+    {
+      "epoch": 0.26976744186046514,
+      "grad_norm": 2.4946393966674805,
+      "learning_rate": 9.289236906774663e-06,
+      "loss": 0.3115,
+      "mean_token_accuracy": 0.8956378698348999,
+      "step": 58
+    },
+    {
+      "epoch": 0.2744186046511628,
+      "grad_norm": 2.7663369178771973,
+      "learning_rate": 9.249226181861e-06,
+      "loss": 0.2915,
+      "mean_token_accuracy": 0.9062834978103638,
+      "step": 59
+    },
+    {
+      "epoch": 0.27906976744186046,
+      "grad_norm": 2.6216726303100586,
+      "learning_rate": 9.208222072905113e-06,
+      "loss": 0.3069,
+      "mean_token_accuracy": 0.8927038908004761,
+      "step": 60
+    },
+    {
+      "epoch": 0.2837209302325581,
+      "grad_norm": 2.7621278762817383,
+      "learning_rate": 9.166235444236209e-06,
+      "loss": 0.3425,
+      "mean_token_accuracy": 0.8820893168449402,
+      "step": 61
+    },
+    {
+      "epoch": 0.28837209302325584,
+      "grad_norm": 2.748720407485962,
+      "learning_rate": 9.123277420509053e-06,
+      "loss": 0.3364,
+      "mean_token_accuracy": 0.8870337605476379,
+      "step": 62
+    },
+    {
+      "epoch": 0.2930232558139535,
+      "grad_norm": 2.5530190467834473,
+      "learning_rate": 9.079359383756411e-06,
+      "loss": 0.3133,
+      "mean_token_accuracy": 0.8994490504264832,
+      "step": 63
+    },
+    {
+      "epoch": 0.29767441860465116,
+      "grad_norm": 2.48974609375,
+      "learning_rate": 9.034492970373305e-06,
+      "loss": 0.3035,
+      "mean_token_accuracy": 0.8979023098945618,
+      "step": 64
+    },
+    {
+      "epoch": 0.3023255813953488,
+      "grad_norm": 2.6978116035461426,
+      "learning_rate": 8.988690068033864e-06,
+      "loss": 0.3339,
+      "mean_token_accuracy": 0.8893972635269165,
+      "step": 65
+    },
+    {
+      "epoch": 0.30697674418604654,
+      "grad_norm": 2.308871030807495,
+      "learning_rate": 8.941962812541604e-06,
+      "loss": 0.3016,
+      "mean_token_accuracy": 0.8937329649925232,
+      "step": 66
+    },
+    {
+      "epoch": 0.3116279069767442,
+      "grad_norm": 2.511657953262329,
+      "learning_rate": 8.894323584613951e-06,
+      "loss": 0.3194,
+      "mean_token_accuracy": 0.889618456363678,
+      "step": 67
+    },
+    {
+      "epoch": 0.31627906976744186,
+      "grad_norm": 2.389920949935913,
+      "learning_rate": 8.845785006601898e-06,
+      "loss": 0.281,
+      "mean_token_accuracy": 0.9037378430366516,
+      "step": 68
+    },
+    {
+      "epoch": 0.3209302325581395,
+      "grad_norm": 2.5016298294067383,
+      "learning_rate": 8.796359939145614e-06,
+      "loss": 0.3076,
+      "mean_token_accuracy": 0.8938754200935364,
+      "step": 69
+    },
+    {
+      "epoch": 0.32558139534883723,
+      "grad_norm": 2.682018518447876,
+      "learning_rate": 8.74606147776692e-06,
+      "loss": 0.3102,
+      "mean_token_accuracy": 0.8926771879196167,
+      "step": 70
+    },
+    {
+      "epoch": 0.3302325581395349,
+      "grad_norm": 2.5323574542999268,
+      "learning_rate": 8.694902949399555e-06,
+      "loss": 0.3095,
+      "mean_token_accuracy": 0.8985507488250732,
+      "step": 71
+    },
+    {
+      "epoch": 0.33488372093023255,
+      "grad_norm": 2.5909910202026367,
+      "learning_rate": 8.642897908858096e-06,
+      "loss": 0.3112,
+      "mean_token_accuracy": 0.8959133625030518,
+      "step": 72
+    },
+    {
+      "epoch": 0.3395348837209302,
+      "grad_norm": 2.635803461074829,
+      "learning_rate": 8.590060135246516e-06,
+      "loss": 0.3358,
+      "mean_token_accuracy": 0.8812984228134155,
+      "step": 73
+    },
+    {
+      "epoch": 0.34418604651162793,
+      "grad_norm": 2.2963192462921143,
+      "learning_rate": 8.53640362830732e-06,
+      "loss": 0.2899,
+      "mean_token_accuracy": 0.9005515575408936,
+      "step": 74
+    },
+    {
+      "epoch": 0.3488372093023256,
+      "grad_norm": 2.634748935699463,
+      "learning_rate": 8.481942604712209e-06,
+      "loss": 0.3187,
+      "mean_token_accuracy": 0.891135573387146,
+      "step": 75
+    },
+    {
+      "epoch": 0.35348837209302325,
+      "grad_norm": 2.4233202934265137,
+      "learning_rate": 8.426691494295269e-06,
+      "loss": 0.2874,
+      "mean_token_accuracy": 0.9031267762184143,
+      "step": 76
+    },
+    {
+      "epoch": 0.3581395348837209,
+      "grad_norm": 2.462113618850708,
+      "learning_rate": 8.370664936229688e-06,
+      "loss": 0.314,
+      "mean_token_accuracy": 0.8929266929626465,
+      "step": 77
+    },
+    {
+      "epoch": 0.3627906976744186,
+      "grad_norm": 2.456052780151367,
+      "learning_rate": 8.313877775149009e-06,
+      "loss": 0.2742,
+      "mean_token_accuracy": 0.9058647155761719,
+      "step": 78
+    },
+    {
+      "epoch": 0.3674418604651163,
+      "grad_norm": 2.792787790298462,
+      "learning_rate": 8.256345057213925e-06,
+      "loss": 0.3157,
+      "mean_token_accuracy": 0.8885141015052795,
+      "step": 79
+    },
+    {
+      "epoch": 0.37209302325581395,
+      "grad_norm": 2.2764811515808105,
+      "learning_rate": 8.198082026125707e-06,
+      "loss": 0.3285,
+      "mean_token_accuracy": 0.8876305222511292,
+      "step": 80
+    },
+    {
+      "epoch": 0.3767441860465116,
+      "grad_norm": 2.815459966659546,
+      "learning_rate": 8.139104119087265e-06,
+      "loss": 0.2839,
+      "mean_token_accuracy": 0.9014216065406799,
+      "step": 81
+    },
+    {
+      "epoch": 0.3813953488372093,
+      "grad_norm": 2.5127663612365723,
+      "learning_rate": 8.07942696271296e-06,
+      "loss": 0.2825,
+      "mean_token_accuracy": 0.9009503126144409,
+      "step": 82
+    },
+    {
+      "epoch": 0.386046511627907,
+      "grad_norm": 2.3554956912994385,
+      "learning_rate": 8.019066368888222e-06,
+      "loss": 0.2873,
+      "mean_token_accuracy": 0.902760922908783,
+      "step": 83
+    },
+    {
+      "epoch": 0.39069767441860465,
+      "grad_norm": 2.4911487102508545,
+      "learning_rate": 7.958038330580067e-06,
+      "loss": 0.3324,
+      "mean_token_accuracy": 0.883284866809845,
+      "step": 84
+    },
+    {
+      "epoch": 0.3953488372093023,
+      "grad_norm": 2.3755667209625244,
+      "learning_rate": 7.89635901759967e-06,
+      "loss": 0.3,
+      "mean_token_accuracy": 0.8920915126800537,
+      "step": 85
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 2.7483432292938232,
+      "learning_rate": 7.834044772318033e-06,
+      "loss": 0.3282,
+      "mean_token_accuracy": 0.8910906910896301,
+      "step": 86
+    },
+    {
+      "epoch": 0.4046511627906977,
+      "grad_norm": 2.4823901653289795,
+      "learning_rate": 7.77111210533597e-06,
+      "loss": 0.3,
+      "mean_token_accuracy": 0.8902961015701294,
+      "step": 87
+    },
+    {
+      "epoch": 0.40930232558139534,
+      "grad_norm": 2.6033935546875,
+      "learning_rate": 7.707577691109519e-06,
+      "loss": 0.3548,
+      "mean_token_accuracy": 0.880534827709198,
+      "step": 88
+    },
+    {
+      "epoch": 0.413953488372093,
+      "grad_norm": 2.27488374710083,
+      "learning_rate": 7.6434583635319e-06,
+      "loss": 0.3022,
+      "mean_token_accuracy": 0.9012537002563477,
+      "step": 89
+    },
+    {
+      "epoch": 0.4186046511627907,
+      "grad_norm": 2.287916660308838,
+      "learning_rate": 7.578771111473276e-06,
+      "loss": 0.2578,
+      "mean_token_accuracy": 0.9131473898887634,
+      "step": 90
+    },
+    {
+      "epoch": 0.4232558139534884,
+      "grad_norm": 2.3548431396484375,
+      "learning_rate": 7.513533074279427e-06,
+      "loss": 0.2677,
+      "mean_token_accuracy": 0.9082207083702087,
+      "step": 91
+    },
+    {
+      "epoch": 0.42790697674418604,
+      "grad_norm": 2.484837532043457,
+      "learning_rate": 7.4477615372305545e-06,
+      "loss": 0.3218,
+      "mean_token_accuracy": 0.8943215012550354,
+      "step": 92
+    },
+    {
+      "epoch": 0.4325581395348837,
+      "grad_norm": 2.419360637664795,
+      "learning_rate": 7.3814739269614265e-06,
+      "loss": 0.2945,
+      "mean_token_accuracy": 0.9002562165260315,
+      "step": 93
+    },
+    {
+      "epoch": 0.4372093023255814,
+      "grad_norm": 2.4345784187316895,
+      "learning_rate": 7.314687806844067e-06,
+      "loss": 0.3091,
+      "mean_token_accuracy": 0.8968407511711121,
+      "step": 94
+    },
+    {
+      "epoch": 0.4418604651162791,
+      "grad_norm": 2.479210376739502,
+      "learning_rate": 7.247420872334221e-06,
+      "loss": 0.3019,
+      "mean_token_accuracy": 0.8969621658325195,
+      "step": 95
+    },
+    {
+      "epoch": 0.44651162790697674,
+      "grad_norm": 2.2260143756866455,
+      "learning_rate": 7.179690946282808e-06,
+      "loss": 0.2833,
+      "mean_token_accuracy": 0.9025951027870178,
+      "step": 96
+    },
+    {
+      "epoch": 0.4511627906976744,
+      "grad_norm": 2.2926838397979736,
+      "learning_rate": 7.111515974213639e-06,
+      "loss": 0.2491,
+      "mean_token_accuracy": 0.9109234809875488,
+      "step": 97
+    },
+    {
+      "epoch": 0.4558139534883721,
+      "grad_norm": 2.386190414428711,
+      "learning_rate": 7.042914019568621e-06,
+      "loss": 0.3028,
+      "mean_token_accuracy": 0.8939149379730225,
+      "step": 98
+    },
+    {
+      "epoch": 0.4604651162790698,
+      "grad_norm": 2.2225522994995117,
+      "learning_rate": 6.973903258921719e-06,
+      "loss": 0.2564,
+      "mean_token_accuracy": 0.909423291683197,
+      "step": 99
+    },
+    {
+      "epoch": 0.46511627906976744,
+      "grad_norm": 2.3535642623901367,
+      "learning_rate": 6.904501977162949e-06,
+      "loss": 0.3274,
+      "mean_token_accuracy": 0.88499915599823,
+      "step": 100
+    },
+    {
+      "epoch": 0.4697674418604651,
+      "grad_norm": 2.362877607345581,
+      "learning_rate": 6.834728562653659e-06,
+      "loss": 0.309,
+      "mean_token_accuracy": 0.8906528353691101,
+      "step": 101
+    },
+    {
+      "epoch": 0.4744186046511628,
+      "grad_norm": 2.4089605808258057,
+      "learning_rate": 6.764601502354403e-06,
+      "loss": 0.3059,
+      "mean_token_accuracy": 0.8971619009971619,
+      "step": 102
+    },
+    {
+      "epoch": 0.4790697674418605,
+      "grad_norm": 2.2496674060821533,
+      "learning_rate": 6.6941393769266995e-06,
+      "loss": 0.2987,
+      "mean_token_accuracy": 0.8985577821731567,
+      "step": 103
+    },
+    {
+      "epoch": 0.48372093023255813,
+      "grad_norm": 2.0976436138153076,
+      "learning_rate": 6.6233608558099405e-06,
+      "loss": 0.2774,
+      "mean_token_accuracy": 0.9051709771156311,
+      "step": 104
+    },
+    {
+      "epoch": 0.4883720930232558,
+      "grad_norm": 2.2868094444274902,
+      "learning_rate": 6.552284692274803e-06,
+      "loss": 0.3088,
+      "mean_token_accuracy": 0.8942136764526367,
+      "step": 105
+    },
+    {
+      "epoch": 0.4930232558139535,
+      "grad_norm": 2.3689169883728027,
+      "learning_rate": 6.48092971845443e-06,
+      "loss": 0.2793,
+      "mean_token_accuracy": 0.9030190706253052,
+      "step": 106
+    },
+    {
+      "epoch": 0.49767441860465117,
+      "grad_norm": 2.390408515930176,
+      "learning_rate": 6.409314840354724e-06,
+      "loss": 0.3383,
+      "mean_token_accuracy": 0.8868411779403687,
+      "step": 107
+    },
+    {
+      "epoch": 0.5023255813953489,
+      "grad_norm": 2.522303342819214,
+      "learning_rate": 6.337459032845068e-06,
+      "loss": 0.3146,
+      "mean_token_accuracy": 0.8873773217201233,
+      "step": 108
+    },
+    {
+      "epoch": 0.5069767441860465,
+      "grad_norm": 2.418745279312134,
+      "learning_rate": 6.2653813346308e-06,
+      "loss": 0.2738,
+      "mean_token_accuracy": 0.9050701260566711,
+      "step": 109
+    },
+    {
+      "epoch": 0.5116279069767442,
+      "grad_norm": 2.2483389377593994,
+      "learning_rate": 6.193100843208772e-06,
+      "loss": 0.282,
+      "mean_token_accuracy": 0.8973640203475952,
+      "step": 110
+    },
+    {
+      "epoch": 0.5162790697674419,
+      "grad_norm": 2.3183794021606445,
+      "learning_rate": 6.120636709807334e-06,
+      "loss": 0.3109,
+      "mean_token_accuracy": 0.8886767029762268,
+      "step": 111
+    },
+    {
+      "epoch": 0.5209302325581395,
+      "grad_norm": 2.3985159397125244,
+      "learning_rate": 6.048008134312078e-06,
+      "loss": 0.2891,
+      "mean_token_accuracy": 0.9019830226898193,
+      "step": 112
+    },
+    {
+      "epoch": 0.5255813953488372,
+      "grad_norm": 2.401186943054199,
+      "learning_rate": 5.975234360178698e-06,
+      "loss": 0.3263,
+      "mean_token_accuracy": 0.8926165103912354,
+      "step": 113
+    },
+    {
+      "epoch": 0.5302325581395348,
+      "grad_norm": 2.2257766723632812,
+      "learning_rate": 5.902334669334287e-06,
+      "loss": 0.2991,
+      "mean_token_accuracy": 0.9015018939971924,
+      "step": 114
+    },
+    {
+      "epoch": 0.5348837209302325,
+      "grad_norm": 2.4873902797698975,
+      "learning_rate": 5.829328377068476e-06,
+      "loss": 0.2952,
+      "mean_token_accuracy": 0.8938588500022888,
+      "step": 115
+    },
+    {
+      "epoch": 0.5395348837209303,
+      "grad_norm": 2.5127971172332764,
+      "learning_rate": 5.756234826915686e-06,
+      "loss": 0.3015,
+      "mean_token_accuracy": 0.8990825414657593,
+      "step": 116
+    },
+    {
+      "epoch": 0.5441860465116279,
+      "grad_norm": 2.7055206298828125,
+      "learning_rate": 5.683073385529938e-06,
+      "loss": 0.3148,
+      "mean_token_accuracy": 0.8902867436408997,
+      "step": 117
+    },
+    {
+      "epoch": 0.5488372093023256,
+      "grad_norm": 2.184539794921875,
+      "learning_rate": 5.60986343755352e-06,
+      "loss": 0.2628,
+      "mean_token_accuracy": 0.907613217830658,
+      "step": 118
+    },
+    {
+      "epoch": 0.5534883720930233,
+      "grad_norm": 2.460629463195801,
+      "learning_rate": 5.536624380480878e-06,
+      "loss": 0.2526,
+      "mean_token_accuracy": 0.9117704629898071,
+      "step": 119
+    },
+    {
+      "epoch": 0.5581395348837209,
+      "grad_norm": 2.3159563541412354,
+      "learning_rate": 5.4633756195191235e-06,
+      "loss": 0.261,
+      "mean_token_accuracy": 0.90885990858078,
+      "step": 120
+    },
+    {
+      "epoch": 0.5627906976744186,
+      "grad_norm": 2.0507678985595703,
+      "learning_rate": 5.390136562446482e-06,
+      "loss": 0.2262,
+      "mean_token_accuracy": 0.9185211062431335,
+      "step": 121
+    },
+    {
+      "epoch": 0.5674418604651162,
+      "grad_norm": 2.456918954849243,
+      "learning_rate": 5.316926614470063e-06,
+      "loss": 0.3052,
+      "mean_token_accuracy": 0.8933428525924683,
+      "step": 122
+    },
+    {
+      "epoch": 0.5720930232558139,
+      "grad_norm": 2.683582067489624,
+      "learning_rate": 5.2437651730843165e-06,
+      "loss": 0.3406,
+      "mean_token_accuracy": 0.8779610991477966,
+      "step": 123
+    },
+    {
+      "epoch": 0.5767441860465117,
+      "grad_norm": 2.5005733966827393,
+      "learning_rate": 5.170671622931527e-06,
+      "loss": 0.3104,
+      "mean_token_accuracy": 0.8913931250572205,
+      "step": 124
+    },
+    {
+      "epoch": 0.5813953488372093,
+      "grad_norm": 2.073763370513916,
+      "learning_rate": 5.097665330665714e-06,
+      "loss": 0.2393,
+      "mean_token_accuracy": 0.9153239727020264,
+      "step": 125
+    },
+    {
+      "epoch": 0.586046511627907,
+      "grad_norm": 2.638117551803589,
+      "learning_rate": 5.024765639821305e-06,
+      "loss": 0.321,
+      "mean_token_accuracy": 0.8932949900627136,
+      "step": 126
+    },
+    {
+      "epoch": 0.5906976744186047,
+      "grad_norm": 2.077889919281006,
+      "learning_rate": 4.951991865687923e-06,
+      "loss": 0.2523,
+      "mean_token_accuracy": 0.916557788848877,
+      "step": 127
+    },
+    {
+      "epoch": 0.5953488372093023,
+      "grad_norm": 2.4227733612060547,
+      "learning_rate": 4.879363290192667e-06,
+      "loss": 0.2931,
+      "mean_token_accuracy": 0.9004586338996887,
+      "step": 128
+    },
+    {
+      "epoch": 0.6,
+      "grad_norm": 2.7623236179351807,
+      "learning_rate": 4.806899156791231e-06,
+      "loss": 0.359,
+      "mean_token_accuracy": 0.8809077143669128,
+      "step": 129
+    },
+    {
+      "epoch": 0.6046511627906976,
+      "grad_norm": 2.4066247940063477,
+      "learning_rate": 4.734618665369202e-06,
+      "loss": 0.2844,
+      "mean_token_accuracy": 0.8950888514518738,
+      "step": 130
+    },
+    {
+      "epoch": 0.6093023255813953,
+      "grad_norm": 2.34375,
+      "learning_rate": 4.662540967154934e-06,
+      "loss": 0.2673,
+      "mean_token_accuracy": 0.9051871299743652,
+      "step": 131
+    },
+    {
+      "epoch": 0.6139534883720931,
+      "grad_norm": 2.1489133834838867,
+      "learning_rate": 4.5906851596452765e-06,
+      "loss": 0.2721,
+      "mean_token_accuracy": 0.9027295112609863,
+      "step": 132
+    },
+    {
+      "epoch": 0.6186046511627907,
+      "grad_norm": 2.2944834232330322,
+      "learning_rate": 4.519070281545571e-06,
+      "loss": 0.3041,
+      "mean_token_accuracy": 0.8964738845825195,
+      "step": 133
+    },
+    {
+      "epoch": 0.6232558139534884,
+      "grad_norm": 2.323543071746826,
+      "learning_rate": 4.447715307725197e-06,
+      "loss": 0.3137,
+      "mean_token_accuracy": 0.8975229263305664,
+      "step": 134
+    },
+    {
+      "epoch": 0.627906976744186,
+      "grad_norm": 2.3029696941375732,
+      "learning_rate": 4.376639144190061e-06,
+      "loss": 0.3004,
+      "mean_token_accuracy": 0.8968156576156616,
+      "step": 135
+    },
+    {
+      "epoch": 0.6325581395348837,
+      "grad_norm": 2.538562536239624,
+      "learning_rate": 4.305860623073304e-06,
+      "loss": 0.2989,
+      "mean_token_accuracy": 0.8975074291229248,
+      "step": 136
+    },
+    {
+      "epoch": 0.6372093023255814,
+      "grad_norm": 2.164029836654663,
+      "learning_rate": 4.2353984976456e-06,
+      "loss": 0.2795,
+      "mean_token_accuracy": 0.9066439270973206,
+      "step": 137
+    },
+    {
+      "epoch": 0.641860465116279,
+      "grad_norm": 2.2238547801971436,
+      "learning_rate": 4.1652714373463435e-06,
+      "loss": 0.2878,
+      "mean_token_accuracy": 0.8984693884849548,
+      "step": 138
+    },
+    {
+      "epoch": 0.6465116279069767,
+      "grad_norm": 2.2204103469848633,
+      "learning_rate": 4.095498022837051e-06,
+      "loss": 0.2723,
+      "mean_token_accuracy": 0.9076555967330933,
+      "step": 139
+    },
+    {
+      "epoch": 0.6511627906976745,
+      "grad_norm": 2.327342987060547,
+      "learning_rate": 4.026096741078281e-06,
+      "loss": 0.3144,
+      "mean_token_accuracy": 0.8956175446510315,
+      "step": 140
+    },
+    {
+      "epoch": 0.6558139534883721,
+      "grad_norm": 2.408447027206421,
+      "learning_rate": 3.957085980431382e-06,
+      "loss": 0.3109,
+      "mean_token_accuracy": 0.8903638124465942,
+      "step": 141
+    },
+    {
+      "epoch": 0.6604651162790698,
+      "grad_norm": 2.17551851272583,
+      "learning_rate": 3.888484025786364e-06,
+      "loss": 0.2808,
+      "mean_token_accuracy": 0.9039127230644226,
+      "step": 142
+    },
+    {
+      "epoch": 0.6651162790697674,
+      "grad_norm": 2.148789405822754,
+      "learning_rate": 3.820309053717195e-06,
+      "loss": 0.2644,
+      "mean_token_accuracy": 0.9068344831466675,
+      "step": 143
+    },
+    {
+      "epoch": 0.6697674418604651,
+      "grad_norm": 2.4230973720550537,
+      "learning_rate": 3.75257912766578e-06,
+      "loss": 0.3521,
+      "mean_token_accuracy": 0.8838323354721069,
+      "step": 144
+    },
+    {
+      "epoch": 0.6744186046511628,
+      "grad_norm": 2.407576560974121,
+      "learning_rate": 3.6853121931559334e-06,
+      "loss": 0.3176,
+      "mean_token_accuracy": 0.8907623291015625,
+      "step": 145
+    },
+    {
+      "epoch": 0.6790697674418604,
+      "grad_norm": 2.257375478744507,
+      "learning_rate": 3.618526073038574e-06,
+      "loss": 0.3402,
+      "mean_token_accuracy": 0.877872109413147,
+      "step": 146
+    },
+    {
+      "epoch": 0.6837209302325581,
+      "grad_norm": 2.3605127334594727,
+      "learning_rate": 3.552238462769446e-06,
+      "loss": 0.2822,
+      "mean_token_accuracy": 0.898716390132904,
+      "step": 147
+    },
+    {
+      "epoch": 0.6883720930232559,
+      "grad_norm": 2.5195729732513428,
+      "learning_rate": 3.4864669257205745e-06,
+      "loss": 0.2726,
+      "mean_token_accuracy": 0.9063690900802612,
+      "step": 148
+    },
+    {
+      "epoch": 0.6930232558139535,
+      "grad_norm": 2.344093084335327,
+      "learning_rate": 3.4212288885267246e-06,
+      "loss": 0.3185,
+      "mean_token_accuracy": 0.8916053771972656,
+      "step": 149
+    },
+    {
+      "epoch": 0.6976744186046512,
+      "grad_norm": 2.4580070972442627,
+      "learning_rate": 3.3565416364681016e-06,
+      "loss": 0.2861,
+      "mean_token_accuracy": 0.8970394134521484,
+      "step": 150
+    },
+    {
+      "epoch": 0.7023255813953488,
+      "grad_norm": 2.1077418327331543,
+      "learning_rate": 3.2924223088904816e-06,
+      "loss": 0.2753,
+      "mean_token_accuracy": 0.9032756090164185,
+      "step": 151
+    },
+    {
+      "epoch": 0.7069767441860465,
+      "grad_norm": 2.571012258529663,
+      "learning_rate": 3.228887894664029e-06,
+      "loss": 0.3382,
+      "mean_token_accuracy": 0.8874310851097107,
+      "step": 152
+    },
+    {
+      "epoch": 0.7116279069767442,
+      "grad_norm": 2.415187358856201,
+      "learning_rate": 3.1659552276819693e-06,
+      "loss": 0.311,
+      "mean_token_accuracy": 0.8911898136138916,
+      "step": 153
+    },
+    {
+      "epoch": 0.7162790697674418,
+      "grad_norm": 2.24096941947937,
+      "learning_rate": 3.1036409824003324e-06,
+      "loss": 0.2866,
+      "mean_token_accuracy": 0.9001513123512268,
+      "step": 154
+    },
+    {
+      "epoch": 0.7209302325581395,
+      "grad_norm": 2.1161606311798096,
+      "learning_rate": 3.0419616694199327e-06,
+      "loss": 0.2648,
+      "mean_token_accuracy": 0.9077988862991333,
+      "step": 155
+    },
+    {
+      "epoch": 0.7255813953488373,
+      "grad_norm": 2.244520425796509,
+      "learning_rate": 2.98093363111178e-06,
+      "loss": 0.2427,
+      "mean_token_accuracy": 0.9150313138961792,
+      "step": 156
+    },
+    {
+      "epoch": 0.7302325581395349,
+      "grad_norm": 2.494776725769043,
+      "learning_rate": 2.92057303728704e-06,
+      "loss": 0.2976,
+      "mean_token_accuracy": 0.8998464941978455,
+      "step": 157
+    },
+    {
+      "epoch": 0.7348837209302326,
+      "grad_norm": 2.248281717300415,
+      "learning_rate": 2.860895880912735e-06,
+      "loss": 0.2849,
+      "mean_token_accuracy": 0.9054994583129883,
+      "step": 158
+    },
+    {
+      "epoch": 0.7395348837209302,
+      "grad_norm": 2.331533432006836,
+      "learning_rate": 2.801917973874294e-06,
+      "loss": 0.3037,
+      "mean_token_accuracy": 0.8930827379226685,
+      "step": 159
+    },
+    {
+      "epoch": 0.7441860465116279,
+      "grad_norm": 2.3448197841644287,
+      "learning_rate": 2.7436549427860766e-06,
+      "loss": 0.3222,
+      "mean_token_accuracy": 0.889302134513855,
+      "step": 160
+    },
+    {
+      "epoch": 0.7488372093023256,
+      "grad_norm": 2.071540355682373,
+      "learning_rate": 2.6861222248509926e-06,
+      "loss": 0.2476,
+      "mean_token_accuracy": 0.9146341681480408,
+      "step": 161
+    },
+    {
+      "epoch": 0.7534883720930232,
+      "grad_norm": 2.1529202461242676,
+      "learning_rate": 2.6293350637703123e-06,
+      "loss": 0.2813,
+      "mean_token_accuracy": 0.9099606871604919,
+      "step": 162
+    },
+    {
+      "epoch": 0.7581395348837209,
+      "grad_norm": 2.4522061347961426,
+      "learning_rate": 2.5733085057047325e-06,
+      "loss": 0.2871,
+      "mean_token_accuracy": 0.9062443971633911,
+      "step": 163
+    },
+    {
+      "epoch": 0.7627906976744186,
+      "grad_norm": 2.1827683448791504,
+      "learning_rate": 2.518057395287792e-06,
+      "loss": 0.2984,
+      "mean_token_accuracy": 0.8948549628257751,
+      "step": 164
+    },
+    {
+      "epoch": 0.7674418604651163,
+      "grad_norm": 2.5053515434265137,
+      "learning_rate": 2.463596371692681e-06,
+      "loss": 0.297,
+      "mean_token_accuracy": 0.8924633860588074,
+      "step": 165
+    },
+    {
+      "epoch": 0.772093023255814,
+      "grad_norm": 2.1398110389709473,
+      "learning_rate": 2.409939864753487e-06,
+      "loss": 0.2976,
+      "mean_token_accuracy": 0.8955926895141602,
+      "step": 166
+    },
+    {
+      "epoch": 0.7767441860465116,
+      "grad_norm": 2.19425892829895,
+      "learning_rate": 2.3571020911419067e-06,
+      "loss": 0.2816,
+      "mean_token_accuracy": 0.9085714221000671,
+      "step": 167
+    },
+    {
+      "epoch": 0.7813953488372093,
+      "grad_norm": 2.4098033905029297,
+      "learning_rate": 2.3050970506004463e-06,
+      "loss": 0.3113,
+      "mean_token_accuracy": 0.8950749635696411,
+      "step": 168
+    },
+    {
+      "epoch": 0.786046511627907,
+      "grad_norm": 2.4144301414489746,
+      "learning_rate": 2.2539385222330797e-06,
+      "loss": 0.2541,
+      "mean_token_accuracy": 0.90910804271698,
+      "step": 169
+    },
+    {
+      "epoch": 0.7906976744186046,
+      "grad_norm": 2.286418914794922,
+      "learning_rate": 2.203640060854387e-06,
+      "loss": 0.269,
+      "mean_token_accuracy": 0.9078757166862488,
+      "step": 170
+    },
+    {
+      "epoch": 0.7953488372093023,
+      "grad_norm": 2.281157970428467,
+      "learning_rate": 2.1542149933981014e-06,
+      "loss": 0.3069,
+      "mean_token_accuracy": 0.893151581287384,
+      "step": 171
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 2.220977544784546,
+      "learning_rate": 2.10567641538605e-06,
+      "loss": 0.279,
+      "mean_token_accuracy": 0.9098817110061646,
+      "step": 172
+    },
+    {
+      "epoch": 0.8046511627906977,
+      "grad_norm": 2.178304433822632,
+      "learning_rate": 2.058037187458398e-06,
+      "loss": 0.2903,
+      "mean_token_accuracy": 0.9047447443008423,
+      "step": 173
+    },
+    {
+      "epoch": 0.8093023255813954,
+      "grad_norm": 2.4263737201690674,
+      "learning_rate": 2.011309931966136e-06,
+      "loss": 0.2844,
+      "mean_token_accuracy": 0.9039999842643738,
+      "step": 174
+    },
+    {
+      "epoch": 0.813953488372093,
+      "grad_norm": 2.2288589477539062,
+      "learning_rate": 1.965507029626695e-06,
+      "loss": 0.2977,
+      "mean_token_accuracy": 0.8857610821723938,
+      "step": 175
+    },
+    {
+      "epoch": 0.8186046511627907,
+      "grad_norm": 2.2465760707855225,
+      "learning_rate": 1.920640616243589e-06,
+      "loss": 0.2632,
+      "mean_token_accuracy": 0.9077427387237549,
+      "step": 176
+    },
+    {
+      "epoch": 0.8232558139534883,
+      "grad_norm": 2.478376626968384,
+      "learning_rate": 1.8767225794909484e-06,
+      "loss": 0.321,
+      "mean_token_accuracy": 0.8919563889503479,
+      "step": 177
+    },
+    {
+      "epoch": 0.827906976744186,
+      "grad_norm": 2.138991594314575,
+      "learning_rate": 1.8337645557637929e-06,
+      "loss": 0.2744,
+      "mean_token_accuracy": 0.9032313227653503,
+      "step": 178
+    },
+    {
+      "epoch": 0.8325581395348837,
+      "grad_norm": 2.145294189453125,
+      "learning_rate": 1.7917779270948887e-06,
+      "loss": 0.2934,
+      "mean_token_accuracy": 0.8986738324165344,
+      "step": 179
+    },
+    {
+      "epoch": 0.8372093023255814,
+      "grad_norm": 2.3890199661254883,
+      "learning_rate": 1.7507738181390027e-06,
+      "loss": 0.2847,
+      "mean_token_accuracy": 0.9007092118263245,
+      "step": 180
+    },
+    {
+      "epoch": 0.8418604651162791,
+      "grad_norm": 2.3594157695770264,
+      "learning_rate": 1.7107630932253383e-06,
+      "loss": 0.2853,
+      "mean_token_accuracy": 0.8999655842781067,
+      "step": 181
+    },
+    {
+      "epoch": 0.8465116279069768,
+      "grad_norm": 2.2847506999969482,
+      "learning_rate": 1.6717563534789594e-06,
+      "loss": 0.2909,
+      "mean_token_accuracy": 0.9014893174171448,
+      "step": 182
+    },
+    {
+      "epoch": 0.8511627906976744,
+      "grad_norm": 2.3248329162597656,
+      "learning_rate": 1.6337639340119476e-06,
+      "loss": 0.2996,
+      "mean_token_accuracy": 0.8940504193305969,
+      "step": 183
+    },
+    {
+      "epoch": 0.8558139534883721,
+      "grad_norm": 2.390031576156616,
+      "learning_rate": 1.596795901185037e-06,
+      "loss": 0.305,
+      "mean_token_accuracy": 0.8964377641677856,
+      "step": 184
+    },
+    {
+      "epoch": 0.8604651162790697,
+      "grad_norm": 2.054699420928955,
+      "learning_rate": 1.5608620499404628e-06,
+      "loss": 0.2769,
+      "mean_token_accuracy": 0.9020813703536987,
+      "step": 185
+    },
+    {
+      "epoch": 0.8651162790697674,
+      "grad_norm": 2.3316433429718018,
+      "learning_rate": 1.5259719012067249e-06,
+      "loss": 0.3028,
+      "mean_token_accuracy": 0.8970876932144165,
+      "step": 186
+    },
+    {
+      "epoch": 0.8697674418604651,
+      "grad_norm": 2.2691869735717773,
+      "learning_rate": 1.4921346993759453e-06,
+      "loss": 0.27,
+      "mean_token_accuracy": 0.9035449028015137,
+      "step": 187
+    },
+    {
+      "epoch": 0.8744186046511628,
+      "grad_norm": 2.1666178703308105,
+      "learning_rate": 1.459359409854505e-06,
+      "loss": 0.2836,
+      "mean_token_accuracy": 0.9079626202583313,
+      "step": 188
+    },
+    {
+      "epoch": 0.8790697674418605,
+      "grad_norm": 2.2185068130493164,
+      "learning_rate": 1.4276547166875946e-06,
+      "loss": 0.2652,
+      "mean_token_accuracy": 0.9092705249786377,
+      "step": 189
+    },
+    {
+      "epoch": 0.8837209302325582,
+      "grad_norm": 2.47299861907959,
+      "learning_rate": 1.397029020258313e-06,
+      "loss": 0.2879,
+      "mean_token_accuracy": 0.896272599697113,
+      "step": 190
+    },
+    {
+      "epoch": 0.8883720930232558,
+      "grad_norm": 2.3769164085388184,
+      "learning_rate": 1.367490435061928e-06,
+      "loss": 0.3358,
+      "mean_token_accuracy": 0.8814517259597778,
+      "step": 191
+    },
+    {
+      "epoch": 0.8930232558139535,
+      "grad_norm": 2.21427845954895,
+      "learning_rate": 1.3390467875558855e-06,
+      "loss": 0.2465,
+      "mean_token_accuracy": 0.9141114950180054,
+      "step": 192
+    },
+    {
+      "epoch": 0.8976744186046511,
+      "grad_norm": 2.2213754653930664,
+      "learning_rate": 1.3117056140861317e-06,
+      "loss": 0.2912,
+      "mean_token_accuracy": 0.9031753540039062,
+      "step": 193
+    },
+    {
+      "epoch": 0.9023255813953488,
+      "grad_norm": 2.2283401489257812,
+      "learning_rate": 1.285474158890304e-06,
+      "loss": 0.2645,
+      "mean_token_accuracy": 0.9112547039985657,
+      "step": 194
+    },
+    {
+      "epoch": 0.9069767441860465,
+      "grad_norm": 2.1087920665740967,
+      "learning_rate": 1.2603593721783219e-06,
+      "loss": 0.2653,
+      "mean_token_accuracy": 0.9116804599761963,
+      "step": 195
+    },
+    {
+      "epoch": 0.9116279069767442,
+      "grad_norm": 2.2635421752929688,
+      "learning_rate": 1.2363679082908766e-06,
+      "loss": 0.2987,
+      "mean_token_accuracy": 0.8961791396141052,
+      "step": 196
+    },
+    {
+      "epoch": 0.9162790697674419,
+      "grad_norm": 2.3147056102752686,
+      "learning_rate": 1.2135061239363161e-06,
+      "loss": 0.2587,
+      "mean_token_accuracy": 0.9121925234794617,
+      "step": 197
+    },
+    {
+      "epoch": 0.9209302325581395,
+      "grad_norm": 2.262284755706787,
+      "learning_rate": 1.1917800765063954e-06,
+      "loss": 0.2704,
+      "mean_token_accuracy": 0.9092745780944824,
+      "step": 198
+    },
+    {
+      "epoch": 0.9255813953488372,
+      "grad_norm": 2.5080342292785645,
+      "learning_rate": 1.1711955224713209e-06,
+      "loss": 0.2932,
+      "mean_token_accuracy": 0.898510217666626,
+      "step": 199
+    },
+    {
+      "epoch": 0.9302325581395349,
+      "grad_norm": 2.2359910011291504,
+      "learning_rate": 1.1517579158545386e-06,
+      "loss": 0.2699,
+      "mean_token_accuracy": 0.9084086418151855,
+      "step": 200
+    },
+    {
+      "epoch": 0.9348837209302325,
+      "grad_norm": 2.4288928508758545,
+      "learning_rate": 1.1334724067876463e-06,
+      "loss": 0.3007,
+      "mean_token_accuracy": 0.8988877534866333,
+      "step": 201
+    },
+    {
+      "epoch": 0.9395348837209302,
+      "grad_norm": 2.267275810241699,
+      "learning_rate": 1.1163438401458358e-06,
+      "loss": 0.2668,
+      "mean_token_accuracy": 0.9053376913070679,
+      "step": 202
+    },
+    {
+      "epoch": 0.9441860465116279,
+      "grad_norm": 2.3959038257598877,
+      "learning_rate": 1.1003767542642021e-06,
+      "loss": 0.2833,
+      "mean_token_accuracy": 0.9051226377487183,
+      "step": 203
+    },
+    {
+      "epoch": 0.9488372093023256,
+      "grad_norm": 2.2642016410827637,
+      "learning_rate": 1.0855753797352868e-06,
+      "loss": 0.2544,
+      "mean_token_accuracy": 0.9085396528244019,
+      "step": 204
+    },
+    {
+      "epoch": 0.9534883720930233,
+      "grad_norm": 2.1275274753570557,
+      "learning_rate": 1.0719436382881466e-06,
+      "loss": 0.2614,
+      "mean_token_accuracy": 0.9060357809066772,
+      "step": 205
+    },
+    {
+      "epoch": 0.958139534883721,
+      "grad_norm": 2.3473360538482666,
+      "learning_rate": 1.0594851417492665e-06,
+      "loss": 0.2869,
+      "mean_token_accuracy": 0.9008370637893677,
+      "step": 206
+    },
+    {
+      "epoch": 0.9627906976744186,
+      "grad_norm": 2.4919135570526123,
+      "learning_rate": 1.0482031910855804e-06,
+      "loss": 0.3038,
+      "mean_token_accuracy": 0.8915042281150818,
+      "step": 207
+    },
+    {
+      "epoch": 0.9674418604651163,
+      "grad_norm": 2.309241533279419,
+      "learning_rate": 1.0381007755298547e-06,
+      "loss": 0.2765,
+      "mean_token_accuracy": 0.9052693843841553,
+      "step": 208
+    },
+    {
+      "epoch": 0.9720930232558139,
+      "grad_norm": 2.2533209323883057,
+      "learning_rate": 1.029180571788672e-06,
+      "loss": 0.2576,
+      "mean_token_accuracy": 0.9127118587493896,
+      "step": 209
+    },
+    {
+      "epoch": 0.9767441860465116,
+      "grad_norm": 2.08575439453125,
+      "learning_rate": 1.021444943333218e-06,
+      "loss": 0.2668,
+      "mean_token_accuracy": 0.908217191696167,
+      "step": 210
+    },
+    {
+      "epoch": 0.9813953488372092,
+      "grad_norm": 2.299942970275879,
+      "learning_rate": 1.0148959397730637e-06,
+      "loss": 0.2626,
+      "mean_token_accuracy": 0.9057932496070862,
+      "step": 211
+    },
+    {
+      "epoch": 0.986046511627907,
+      "grad_norm": 2.232316017150879,
+      "learning_rate": 1.0095352963131057e-06,
+      "loss": 0.2853,
+      "mean_token_accuracy": 0.9010600447654724,
+      "step": 212
+    },
+    {
+      "epoch": 0.9906976744186047,
+      "grad_norm": 2.287255048751831,
+      "learning_rate": 1.0053644332938118e-06,
+      "loss": 0.251,
+      "mean_token_accuracy": 0.9092245697975159,
+      "step": 213
+    },
+    {
+      "epoch": 0.9953488372093023,
+      "grad_norm": 2.229083776473999,
+      "learning_rate": 1.0023844558148912e-06,
+      "loss": 0.2969,
+      "mean_token_accuracy": 0.8984347581863403,
+      "step": 214
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 2.0628480911254883,
+      "learning_rate": 1.0005961534424925e-06,
+      "loss": 0.2309,
+      "mean_token_accuracy": 0.909231960773468,
+      "step": 215
+    },
+    {
+      "epoch": 1.0,
+      "step": 215,
+      "total_flos": 2.0144468407196058e+17,
+      "train_loss": 0.325891209826913,
+      "train_runtime": 788.7208,
+      "train_samples_per_second": 8.686,
+      "train_steps_per_second": 0.273
+    }
+  ],
+  "logging_steps": 1,
+  "max_steps": 215,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.0144468407196058e+17,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

training.log

@@ -580,3 +580,56 @@ weight_decay=0.0,
   )
   (lm_head): Linear(in_features=2048, out_features=151936, bias=False)
 )
+2025-09-15 02:07:52 - INFO - __main__ - *** Save model ***
+2025-09-15 02:07:52 - INFO - __main__ - 💾 Saving MoE bias states...
+2025-09-15 02:07:52 - INFO - __main__ - 🔍 Searching for MoE layers with bias states...
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.0.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.1.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.2.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.3.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.4.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.5.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.6.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.7.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.8.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.9.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.10.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.11.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.12.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.13.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.14.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.15.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.16.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.17.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.18.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.19.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.20.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.21.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.22.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - ✅ Saved bias from model.layers.23.mlp: 60 experts, update_speed=0.001000
+2025-09-15 02:07:52 - INFO - __main__ - 🎉 Successfully saved 24 MoE bias states to /tmp/data/Qwen1.5-MOE/aux_free_sft/math7k/1e-3-gamma/moe_bias_states.json
+2025-09-15 02:07:52 - INFO - __main__ - 📊 Bias States Summary:
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.0.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.1.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.2.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.3.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.4.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.5.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.6.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.7.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.8.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.9.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.10.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.11.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.12.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.13.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.14.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.15.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.16.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.17.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.18.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.19.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.20.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.21.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.22.mlp: 60 experts, range=[-0.5000, 0.5000]
+2025-09-15 02:07:52 - INFO - __main__ -   model.layers.23.mlp: 60 experts, range=[-0.5000, 0.5000]