yuxinlu1/Mellum2-12B-A2.5B-Claude-4.6-4.8-Opus-Thinking-GGUF

⚙️ Mellum2-12B-A2.5B-Reasoning-Distill (GGUF) ⚙️

🧑‍💻 A fast little coding brain — local AI for everyone

12B total params, only 2.5B active per token. This is a Mixture-of-Experts model, so it runs like a ~2.5B model but thinks like a 12B one. 🚀 Built on JetBrains' Mellum 2 (a from-scratch software-engineering model) and tuned on Claude Opus 4.6 / 4.7 / 4.8 reasoning traces — it reasons step-by-step in <think> blocks, then answers. 🧠💻 All local, all yours, no API, no cloud. And it's seriously fast.

---

⚡ Blazing fast — measured, not marketing 🏎️💨

~440 tokens/sec on a single RTX 5090 at Q4_K_M (--n-gpu-layers 99 -fa on) — and generation quality holds up: correct, coherent code and clean step-by-step reasoning. 🎯

You get big-model answers at small-model speed. Why so quick? It's a Mixture-of-Experts (only 2.5B of the 12B params fire per token) with a compact 98K vocab — so it generates several times faster than a dense model its size, with no draft / speculative model needed. 💚

Hardware	Quant	Generation speed (measured)
RTX 5090 (32 GB)	Q4_K_M	~440 tok/s ⚡

---

📦 Pick your size (GGUF quants)

Quant	Size	Vibe
🟢 Q2_K	5.0 GB	tiniest — runs almost anywhere
🔵 Q4_K_M	8.1 GB	the sweet spot 👌 (recommended)
🟣 Q6_K	10.9 GB	near-lossless
⚪ Q8_0	12.9 GB	basically full quality

💡 It's a Mixture-of-Experts: all 64 experts live on disk/VRAM (so size is for the whole 12B), but only 8 fire per token — that's why it's so quick.

---

🧮 "Will it fit?" — rough VRAM guide

Mellum2 has a tiny KV cache (GQA with just 4 KV heads, and sliding-window attention on 3 of every 4 layers) — so context is rarely the limiter. Pick the quant that fits your VRAM and you'll have plenty of room for long context (max is 131K). Rough numbers 🤓 (weights + ~2 GB overhead):

Your VRAM / unified mem	Best quant that fits	Context headroom
8 GB	🟢 Q2_K	comfy (long ctx still fits)
12 GB	🔵 Q4_K_M	lots
16 GB	🟣 Q6_K / ⚪ Q8_0	lots
24 GB+	⚪ Q8_0	up to 131K 🎉

💡 Apple Silicon / iGPUs with unified memory count too — same idea, just slower than a dGPU. 💡 Tight on room? Drop a quant or use a q4_0 KV cache for even more context.

---

🚀 How to run it (super easy)

Option A — llama.cpp (recommended) 🦙

1. Grab a quant above (e.g. …-Q4_K_M.gguf) and llama-server from llama.cpp.

   ⚠️ Needs a recent llama.cpp that supports the mellum2 architecture (a mid-2026 build or newer). Older builds fail with unknown architecture: 'mellum2'.

2. Run a server (Windows .bat shown — tweak --port, --ctx-size to taste):

@echo off
cd /d C:\llama.cpp
llama-server.exe ^
  -m C:\models\mellum2-claude-Q4_K_M.gguf ^
  --ctx-size 16384 ^
  --n-gpu-layers 99 ^
  --no-mmap ^
  -fa on ^
  --jinja --reasoning-format deepseek ^
  --temp 0.6 --top-p 0.95 --top-k 20 ^
  --host 0.0.0.0 --port 18080
pause

3. Open http://localhost:18080 and chat. 🎉 (Tip: bump --ctx-size — the KV cache is small, so go big.)

Option B — one-click apps 🖱️

Works in LM Studio, Jan, Ollama, etc. — just import the GGUF, pick your quant, go. 🐾 (Make sure the app ships a recent llama.cpp that knows mellum2.)

🧠 Thinking mode

This model thinks natively in <think> … </think> blocks. The chat template handles it automatically; the --reasoning-format deepseek flag tells llama.cpp to surface the reasoning cleanly. Recommended sampling: temp 0.6, top_p 0.95, top_k 20 (JetBrains' official settings for the Thinking model).

🐍 Prefer raw transformers? (click)

This repo ships GGUF (for llama.cpp). To run in raw transformers you need non-GGUF weights — point mid at the original JetBrains checkpoint (or your own merged fp16). Needs transformers >= 5.8 (the mellum architecture is built in — no trust_remote_code). It's a plain text CausalLM (not multimodal).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

mid = "JetBrains/Mellum2-12B-A2.5B-Thinking"   # GGUF won't load here — use non-GGUF weights
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype=torch.bfloat16,
                                             device_map="auto", attn_implementation="sdpa")

msgs = [{"role": "user", "content": "Write a Python function to check if a number is prime."}]
inputs = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True,
                                 enable_thinking=True)
inputs = inputs.to(model.device)
out = model.generate(inputs, max_new_tokens=512, temperature=0.6, top_p=0.95, top_k=20)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=False))

---

⚡ Why no MTP / draft model?

JetBrains' Mellum 2 uses Multi-Token Prediction as a training-time objective only — it isn't exported to the released weights, so there's no draft to ship. You don't need one: with just 2.5B active params and a compact vocab, it already generates extremely fast (~440 tok/s on a 5090 at Q4_K_M). 🏎️

---

🧩 What is this, exactly?

• Base: JetBrains/Mellum2-12B-A2.5B-Thinking — a from-scratch, software-engineering-focused Mixture-of-Experts model (12.15B total / 2.5B active, 28 layers, 64 experts with 8 active, 131K context). JetBrains already did SFT + RLVR on it; this is a light extra LoRA distillation pass on top.
• This fine-tune: a low-intensity QLoRA-style pass over the attention projections, distilling Claude Opus reasoning style into the model. It keeps Mellum 2's coding/agent strengths while nudging the reasoning voice toward Opus. 💡

---

⚠️ Good to know

• Coding-first: Mellum 2 is built for code generation, editing, debugging, tool-calls and agents. Great at programming + structured reasoning; it's not a general-knowledge encyclopedia.
• Reduced refusals: the distillation data omits safety hedging, so it refuses less than a typical aligned chat model. It is not safety-aligned — add your own guardrails for production. Use responsibly. 🙏
• The reasoning is stylistic synthetic CoT — great for structure, but double-check facts and numbers.
• English-centric (handles other languages, but English is strongest).

---

📚 Data & License

• Base model: JetBrains/Mellum2-12B-A2.5B-Thinking, released under Apache-2.0.
• Training data: built on the public, Apache-2.0 dataset angrygiraffe/claude-opus-4.6-4.7-reasoning-8.7k, augmented with additional Opus 4.8-generated reasoning samples I curated and mixed in.
• Personal/hobby project — shared as-is, no warranty. Have fun! 🐾✨