Upload folder using huggingface_hub

README.md

@@ -0,0 +1,140 @@
+---
+license: other
+license_name: nvidia-open-model-license
+license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license
+base_model: nvidia/Llama-4-Maverick-17B-128E-Eagle3
+tags:
+- speculative-decoding
+- eagle3
+- llama3
+- llama4
+- vllm
+- speculators
+---
+
+# Llama4-Maverick-Eagle3-Speculators
+
+## Model Description
+
+**⚠️ Development Reference Model**: This model has been converted as a reference for development on vLLM. Once development is complete, it can be served using:
+```bash
+vllm serve nm-testing/Llama4-Maverick-Eagle3-Speculators
+```
+
+This is a manually converted Eagle3 speculator model based on NVIDIA's Llama-4-Maverick-17B-128E-Eagle3, reformatted for compatibility with the [Speculators](https://github.com/neuralmagic/speculators) library and vLLM speculative decoding.
+
+### Development Status
+🚧 **Reference Implementation for vLLM Development**
+- This model serves as a reference implementation for vLLM Eagle3 support
+- Contains non-standard features (auxiliary hidden states) that require vLLM extensions
+- Once vLLM development is complete, will support direct serving
+
+### Key Features
+- **Architecture**: Eagle3 speculator with Llama3-based draft head
+- **Target Verifier**: Llama4 Maverick 17B (quantized w4a16)
+- **Vocabulary Size**: 202,048 tokens (unusually large for a draft model)
+- **Special Feature**: Uses auxiliary hidden states from verifier layers [1, 23, 44]
+
+## Configuration Details
+
+This model represents a unique hybrid configuration:
+- **Draft Model**: Llama3-based Eagle3 head (single transformer layer)
+- **Verifier Model**: `RedHatAI/Llama-4-Maverick-17B-128E-Instruct-quantized.w4a16`
+- **Architecture Class**: `Llama4ForConditionalGeneration` for verifier
+
+### Non-Standard Features
+
+This model includes several non-standard Eagle3 features preserved from the NVIDIA checkpoint:
+- Auxiliary hidden state layers from positions [1, 23, 44]
+- Custom layer normalization configurations
+- Large vocabulary matching the target model
+
+## Usage
+
+### With vLLM (After Development Complete)
+
+```bash
+# Once vLLM development is complete, serve directly:
+vllm serve nm-testing/Llama4-Maverick-Eagle3-Speculators
+```
+
+### With Speculators Library
+
+```python
+from speculators import SpeculatorModel
+from transformers import AutoModelForCausalLM
+
+# Load the speculator
+speculator = SpeculatorModel.from_pretrained("nm-testing/Llama4-Maverick-Eagle3-Speculators")
+
+# Load and attach the verifier
+verifier = AutoModelForCausalLM.from_pretrained(
+    "RedHatAI/Llama-4-Maverick-17B-128E-Instruct-quantized.w4a16",
+    trust_remote_code=True
+)
+speculator.attach_verifier(verifier)
+
+# Use for generation
+outputs = speculator.generate(input_ids, max_length=100)
+```
+
+## Configuration Structure
+
+The model uses the Speculators Eagle3 format with additional fields for NVIDIA-specific features:
+
+```json
+{
+  "speculators_model_type": "eagle3",
+  "architectures": ["Eagle3Speculator"],
+  "draft_vocab_size": 202048,
+  "transformer_layer_config": {
+    "rope_scaling": {
+      "rope_type": "llama3"  // Confirms Llama3 architecture
+    }
+  },
+  "eagle_aux_hidden_state_layer_ids": [1, 23, 44],
+  "use_aux_hidden_state": true
+}
+```
+
+## Performance Notes
+
+- **Vocabulary Size**: The 202K vocabulary is unusually large and may impact memory usage
+- **Auxiliary Hidden States**: May require custom Eagle3Speculator extensions for full functionality
+- **Acceptance Rate**: Expected ~2-3 tokens per forward pass based on NVIDIA benchmarks
+
+## Model Weights
+
+- **Format**: SafeTensors
+- **Precision**: bfloat16
+- **Size**: ~3.2GB
+
+## Citation
+
+If you use this model, please cite both the original NVIDIA model and the Speculators library:
+
+```bibtex
+@misc{nvidia2025llama4maverick,
+  title={Llama 4 Maverick 17B Eagle3},
+  author={NVIDIA Corporation},
+  year={2025},
+  publisher={Hugging Face}
+}
+
+@misc{speculators2024,
+  title={Speculators: A Unified Library for Speculative Decoding},
+  author={Neural Magic},
+  year={2024},
+  url={https://github.com/neuralmagic/speculators}
+}
+```
+
+## License
+
+This model is subject to the NVIDIA Open Model License. Please review the license terms before use.
+
+## Acknowledgments
+
+- Original model by NVIDIA Corporation
+- Conversion and formatting for Speculators/vLLM compatibility
+- Based on Eagle3 architecture with Llama3 draft head targeting Llama4 verifier

config.json

@@ -0,0 +1,84 @@
+{
+  "architectures": [
+    "Eagle3Speculator"
+  ],
+  "speculators_model_type": "eagle3",
+  "speculators_version": "0.1.0.dev42",
+  "draft_vocab_size": 202048,
+  "norm_before_residual": true,
+  "target_hidden_size": null,
+  "transformer_layer_config": {
+    "model_type": "llama",
+    "vocab_size": 202048,
+    "hidden_size": 5120,
+    "intermediate_size": 32768,
+    "num_hidden_layers": 1,
+    "num_attention_heads": 40,
+    "num_key_value_heads": 8,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "max_position_embeddings": 1048576,
+    "initializer_range": 0.02,
+    "rms_norm_eps": 1e-05,
+    "pretraining_tp": 1,
+    "use_cache": true,
+    "rope_theta": 500000.0,
+    "rope_scaling": {
+      "factor": 8.0,
+      "high_freq_factor": 4.0,
+      "low_freq_factor": 1.0,
+      "original_max_position_embeddings": 8192,
+      "rope_type": "llama3"
+    },
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "mlp_bias": false,
+    "tie_word_embeddings": false
+  },
+  "speculators_config": {
+    "algorithm": "eagle3",
+    "default_proposal_method": "greedy",
+    "proposal_methods": [
+      {
+        "proposal_type": "greedy",
+        "speculative_tokens": 3,
+        "verifier_accept_k": 1,
+        "accept_tolerance": 0.0
+      }
+    ],
+    "verifier": {
+      "name_or_path": "RedHatAI/Llama-4-Maverick-17B-128E-Instruct-quantized.w4a16",
+      "architectures": [
+        "Llama4ForConditionalGeneration"
+      ]
+    }
+  },
+  "torch_dtype": "bfloat16",
+  "eagle_aux_hidden_state_layer_ids": [
+    1,
+    23,
+    44
+  ],
+  "use_aux_hidden_state": true,
+  "use_input_layernorm_in_first_layer": true,
+  "use_mtp_layernorm": false,
+  "eagle_config": {
+    "eagle_aux_hidden_state_layer_ids": [
+      1,
+      23,
+      44
+    ],
+    "use_aux_hidden_state": true,
+    "use_input_layernorm_in_first_layer": true,
+    "use_last_layernorm": true,
+    "use_mtp_layernorm": false
+  },
+  "_comment": "Eagle3 head based on Llama3 architecture targeting Llama4 Maverick verifier",
+  "_conversion_notes": {
+    "source": "nvidia/Llama-4-Maverick-17B-128E-Eagle3",
+    "architecture_notes": "Eagle3 head uses Llama3 rope_type, targets Llama4 verifier",
+    "vocabulary_notes": "Large 202K vocabulary, same for draft and target",
+    "auxiliary_layers": "Uses hidden states from verifier layers 1, 23, 44",
+    "implementation_note": "May require Eagle3Speculator extensions for aux hidden states"
+  }
+}

metadata.json

@@ -0,0 +1,15 @@
+{
+  "conversion_tool": "create_final_eagle3_config.py",
+  "source_checkpoint": "nvidia/Llama-4-Maverick-17B-128E-Eagle3",
+  "format": "speculators-eagle3",
+  "architecture": "Llama3-based Eagle3 head",
+  "verifier": "Llama4 Maverick",
+  "notes": [
+    "Eagle3 head based on Llama3 architecture (rope_type: llama3)",
+    "Targets Llama4 Maverick verifier (Llama4ForConditionalGeneration)",
+    "Large vocabulary of 202,048 tokens",
+    "Uses auxiliary hidden states from layers 1, 23, 44",
+    "NVIDIA-specific fields preserved as extra configuration",
+    "May require Eagle3Speculator implementation extensions"
+  ]
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b09a7b8316252795759a0ef96e0420eedba7752323a1c02eb259c57536a20c90
+size 3432162640