Instructions to use tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF",
	filename="Qwen3.5-397B-A17B-heretic-smol-IQ2_XS.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS
# Run inference directly in the terminal:
llama-cli -hf tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS
# Run inference directly in the terminal:
llama-cli -hf tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS
# Run inference directly in the terminal:
./llama-cli -hf tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS

Use Docker

docker model run hf.co/tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS

LM Studio
Jan

vLLM

How to use tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS

Ollama
How to use tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF with Ollama:
```
ollama run hf.co/tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS
```

Unsloth Studio

How to use tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF to start chatting

How to use tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS

Run Hermes

hermes

Atomic Chat new
Docker Model Runner
How to use tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF with Docker Model Runner:
```
docker model run hf.co/tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS
```

Lemonade

How to use tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull tarruda/Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF:IQ2_XS

Run and chat with the model

lemonade run user.Qwen3.5-397B-A17B-heretic-smol-IQ2_XS-GGUF-IQ2_XS

List all available models

lemonade list

Any chance of IQ2_XXS? IQ2_XS is just slightly too big for Strix Halo.

by Cortex0833 - opened Mar 22

Discussion

Cortex0833

Mar 22

I'm really enjoying the Bartowski IQ2_XXS (holds up shockingly well) but I know it's only a matter of time before it attempts to nanny me.

tarruda

Owner Mar 22

IQ2_XS is just slightly too big for Strix Halo.

Why is that? Doesn't Strix Halo also have 128G unified memory? I think I read somewhere you can increase the amount of memory available for the GPU, which I also had to do with my M1 ultra (now I can allocate up to 125G to video!).

As for IQ2_XXS, yea I can do it eventually. Only recently I started playing with creating my own quants and I'm currently experimenting with variations of @ubergarm 's recipe!

Cortex0833

Mar 22

•

edited Mar 23

I'm pretty deep into optimizing this platform, but the most memory possible is 124 with Fedora Server. Practically, it's 122 on my current Fedora LXDE system. (This gives reliable Bluetooth conference speaker connection for a seamless voice assistant.) I did try terminating all unneeded services and running the XS quant, but it still blew up. Even if I could get it running there would be no room for context.

tarruda

Owner Mar 23

If I change the DOWN tensors to XS to XXS the size would change from 116130.32 MiB (2.46 BPW) to 112290.32 MiB (2.38 BPW), reducing approximately 4G.

TBH I have no idea if that will work but we can give it a shot. WDYT?

Cortex0833

Mar 23

That 4GB might be just enough, but it would be tight. Might be safer to go down further for Strix Halo. Although how certain is that calculation? Bartowski's IQ2_XXS is 107GB. https://huggingface.co/bartowski/Qwen_Qwen3.5-397B-A17B-GGUF/tree/main/Qwen_Qwen3.5-397B-A17B-IQ2_XXS

If you make it , I'll try it.

ubergarm

Mar 23

•

edited Mar 23

@tarruda

If I change the DOWN tensors to XS to XXS

The tradition is to keep ffn_down_exps one size larger than ffn_(gate|up)_exps for most recipes. My smol recipes keep them all the same. Always keep gate|up the same size as they are often fused with -muge at runtime or mainline folks fuse them when converting the safetensor to bf16 gguf statically (which i don't do and is causing some hiccups at the moment: https://github.com/ggml-org/llama.cpp/issues/20883).

To give these users a little extra head space for more kv-cache and all you could probably sacrifice and use this recipe:

./build/bin/llama-quantize \
    --tensor-type ffn_down_exps=iq2_xs \
    --tensor-type ffn_gate_exps=iq2_xxs \
    --tensor-type ffn_up_exps=iq2_xxs \
    --token-embedding-type q4_K \
    --output-tensor-type q6_K \
    --imatrix /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/imatrix-Qwen3.5-397B-A17B-BF16-mainline.gguf \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-BF16-00001-of-00017.gguf \
    /mnt/data/models/ubergarm/Qwen3.5-397B-A17B-GGUF/Qwen3.5-397B-A17B-smol-IQ2_XS.gguf \
    Q8_0 \
    128

You can run with --dry-run to see the estimated size before committing the resources to actually cook it.

You may run into issues with imatrix causing you grief going down as low as iq2_xxs though.

tarruda

Owner Mar 23

Will give that a shot, thanks @ubergarm !

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment