⚡ Qwen3-30B-A3B-Instruct-2507-NVFP4

NVFP4 quantization of Qwen3-30B-A3B-Instruct-2507 — 61GB → 16.9GB, ready for single-GPU deployment.

A high-quality NVFP4 (NVIDIA FP4) quantization of Qwen's updated Mixture-of-Experts instruct model (July 2025), calibrated on Italian-language data with full expert coverage. Designed for production inference with vLLM on NVIDIA Blackwell, Hopper, and Ada GPUs.

---

🏗️ Model Overview

🧬 Architecture	Qwen3-MoE — standard transformer with Mixture-of-Experts
📐 Parameters	30B total, 3B active per token (128 experts, top-8 routing)
🗜️ Quantization	NVFP4 (4-bit floating point weights and activations)
📦 Size	16.9 GB (from 61 GB BF16) — 72% reduction
🔧 Format	compressed-tensors — native vLLM support
📏 Context	262,144 tokens natively

---

🆕 What's New in 2507

This is the quantized version of the July 2025 update, which brings significant improvements over the original Qwen3-30B-A3B:

• Improved instruction following, logical reasoning, math, science, coding and tool usage
• Better long-tail knowledge coverage across multiple languages
• Enhanced alignment with user preferences in open-ended tasks
• Improved 256K long-context understanding

Note: This model supports only non-thinking mode and does not generate <think></think> blocks.

---

🚀 Quick Start

vLLM (recommended)

vllm serve Sophia-AI/Qwen3-30B-A3B-Instruct-2507-NVFP4

vLLM with Docker

docker run --gpus all \
    -p 8000:8000 \
    vllm/vllm-openai:latest \
    --model Sophia-AI/Qwen3-30B-A3B-Instruct-2507-NVFP4

Python (OpenAI-compatible API)

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

response = client.chat.completions.create(
    model="Sophia-AI/Qwen3-30B-A3B-Instruct-2507-NVFP4",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain mixture-of-experts architectures in simple terms."},
    ],
    max_tokens=512,
)
print(response.choices[0].message.content)

Python (Transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Sophia-AI/Qwen3-30B-A3B-Instruct-2507-NVFP4",
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
    "Sophia-AI/Qwen3-30B-A3B-Instruct-2507-NVFP4"
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What are the key improvements in Qwen3's July 2025 update?"},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

🔬 Quantization Details

Method

NVFP4 quantization using llmcompressor with the compressed-tensors format. Weights and activations are quantized to 4-bit NVIDIA floating point with per-group scales (group size 16).

Calibration

📊 Samples	512
📏 Sequence length	1024 tokens (p90-optimized)
🌍 Calibration language	Italian
🔀 MoE coverage	All 128 experts calibrated (moe_calibrate_all_experts=True)
⚙️ Pipeline	Basic (full GPU, no CPU offload)
🖥️ Hardware	2× NVIDIA B200 SXM (366 GB VRAM)
⏱️ Quantization time	~18 minutes
💾 Compression time	~14 minutes

Preserved Layers (not quantized)

The following layers are kept in their original BF16 precision to preserve model quality:

Pattern	Count	Reason
lm_head	1	Output projection — critical for token prediction
mlp.gate	48	MoE routing gates — low parameter count, high impact on expert selection

A total of 49 modules are preserved in original precision.

---

💻 Hardware Requirements

Setup	VRAM	Notes
1× RTX 4090 (24 GB)	~17 GB	✅ Fits with room for KV cache
1× RTX 5090 (32 GB)	~17 GB	✅ Comfortable fit
1× A100 (40/80 GB)	~17 GB	✅ Plenty of headroom
1× H100 (80 GB)	~17 GB	✅ Ideal for long contexts
1× B200 (192 GB)	~17 GB	✅ Maximum KV cache capacity

At only 16.9 GB, this model fits comfortably on consumer GPUs. NVFP4 inference requires NVIDIA GPUs with compute capability ≥ 8.9 (Ada Lovelace, Hopper, Blackwell).

---

🏛️ Architecture Notes

Qwen3-30B-A3B is a standard transformer with Mixture-of-Experts (MoE) feed-forward layers:

• 48 transformer layers, each with a MoE FFN block
• 128 experts per layer, with top-8 routing per token
• ~3B parameters active per token out of 30B total
• Standard multi-head attention (not hybrid like Qwen3-Next)
• 262,144 native context length

This architecture enables strong performance at a fraction of the compute cost of a dense 30B model, while maintaining the full capacity of 128 specialized experts.

---

⚠️ Important Notes

• 🎯 Calibration language — calibrated on Italian data. The model retains its full multilingual capabilities (100+ languages), but quantization quality may be slightly optimized for Italian and similar Romance languages.
• 📏 Sequence length — calibrated at 1024 tokens. The model supports up to 262K context but quantization statistics are optimized for this range.
• 🔧 vLLM recommended — compressed-tensors format is natively supported by vLLM. Other inference engines may require conversion.
• 🧠 Non-thinking mode only — this model does not generate <think></think> blocks. For reasoning mode, use the Thinking variant.
• 📊 Benchmarks — coming soon. Community evaluations welcome.

---

📜 License

This model inherits the Apache 2.0 license from the base model.

---

Quantized with ❤️ by Sophia AI
NVFP4 via llmcompressor • 128 experts fully calibrated • Ready for vLLM