Instructions to use RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16")
model = AutoModelForMultimodalLM.from_pretrained("RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16

SGLang

How to use RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16 with Docker Model Runner:
```
docker model run hf.co/RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16
```

Great! Sucessfully Running on 8*48G 4090Ti: Avg generation throughput: 221.5 tokens/s, Running: 4 reqs

by nwzjk - opened 3 days ago

Discussion

nwzjk

3 days ago

https://huggingface.co/nwzjk/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16-W4A16-G128/blob/main/README.md

Jon-Nielsen

3 days ago

Thanks for posting your results with this model and quant. I've been enjoying this quant more than the official NVFP4 so far.

https://huggingface.co/nwzjk/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16-W4A16-G128/blob/main/README.md

I saw you changed the config file also. I'm interested in your motivation. Can you explain your changes?

One thing that eludes me with this quant is higher token generation speed; Having MTP enabled seems to cause rapid drift in accuracy and coherence. Specifically any kind of output where loss of precision is obvious such as tool calling is immediate.

nwzjk

3 days ago

•

edited 3 days ago

I saw you changed the config file also. I'm interested in your motivation

when i'm downloading the weights, I wonder if it can run sucessfully on 4090, so i run claude code to exploring it, cc analysis the weight cofigs and the vllm lib, and suggests the changes

Having MTP enabled seems to cause rapid drift in accuracy and coherence

not found, normally used with claude code. but the simultaneous use of Mamba cache, Prefix Caching, KV Offloading, and MTP in vLLM is currently unstable and may cause system crashes

Jon-Nielsen

3 days ago

I saw you changed the config file also. I'm interested in your motivation

when i'm downloading the weights, I wonder if it can run sucessfully on 4090, so i run claude code to exploring it, cc analysis the weight cofigs and the vllm lib, and suggests the changes

Having MTP enabled seems to cause rapid drift in accuracy and coherence

not found, normally used with claude code. but the simultaneous use of Mamba cache, Prefix Caching, KV Offloading, and MTP in vLLM is currently unstable and may cause system crashes

Thanks for the info! I see you have all of those enabled in your updated README. Is the version of vLLM you use not affected?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment