Duplicate from douyamv/Gemma-4-31B-JANG_4M-CRACK-GGUF

Co-authored-by: douyamv <douyamv@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,46 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-jang-crack-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-jang-crack-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-jang-crack-Q8_0-00006-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-jang-crack-Q8_0-00002-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-jang-crack-Q8_0-00005-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-jang-crack-Q8_0-00001-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-jang-crack-Q8_0-00008-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-jang-crack-Q8_0-00009-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-jang-crack-Q8_0-00004-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-jang-crack-Q8_0-00003-of-00009.gguf filter=lfs diff=lfs merge=lfs -text
+gemma-4-31b-jang-crack-Q8_0-00007-of-00009.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,86 @@
+---
+language:
+  - en
+license: gemma
+tags:
+  - gemma4
+  - gguf
+  - quantized
+  - 31b
+base_model: google/gemma-4-31b-it
+pipeline_tag: text-generation
+---
+
+# Gemma-4-31B-JANG_4M-CRACK-GGUF
+
+GGUF quantizations of Gemma-4-31B-JANG_4M-CRACK for use with llama.cpp, LM Studio, Ollama, and other GGUF-compatible inference engines.
+
+## About the Model
+
+- **Base model:** [google/gemma-4-31b-it](https://huggingface.co/google/gemma-4-31b-it)
+- **Architecture:** Gemma 4 Dense Transformer (31B parameters, 60 layers)
+- **Features:** Hybrid Sliding/Global Attention, Vision + Audio multimodal
+- **Modification:** CRACK abliteration (refusal removal) + JANG v2 mixed-precision quantization
+
+## Why This Conversion?
+
+The original model uses **JANG v2 mixed-precision MLX quantization** (attention 8-bit + MLP 4-bit), which is only compatible with vMLX. Standard tools (llama.cpp, LM Studio, oMLX, mlx-lm) cannot load this format due to mixed per-layer bit widths.
+
+This repository provides standard GGUF quantizations that work everywhere.
+
+## Conversion Process
+
+```
+Original (JANG v2 MLX safetensors, ~18GB)
+    ↓ dequantize (attention 8-bit → f16, MLP 4-bit → f16)
+Intermediate (float16 safetensors, ~60GB)
+    ↓ convert_hf_to_gguf.py + quantize
+GGUF (various quantizations)
+```
+
+**Note:** Since the original was already quantized (avg 5.1 bits), the dequantized f16 intermediate is an approximation. Re-quantizing to GGUF introduces minimal additional quality loss since the attention layers were preserved at 8-bit in the original.
+
+## Available Quantizations
+
+| File | Quant | Size | Quality | Notes |
+|------|-------|------|---------|-------|
+| `gemma-4-31b-jang-crack-Q3_K_M.gguf` | Q3_K_M | ~14 GB | Acceptable | Minimum viable quality |
+| `gemma-4-31b-jang-crack-Q4_K_M.gguf` | Q4_K_M | ~18 GB | Good | Best size/quality balance |
+| `gemma-4-31b-jang-crack-Q5_K_M.gguf` | Q5_K_M | ~21 GB | Better | Recommended if RAM allows |
+| `gemma-4-31b-jang-crack-Q6_K.gguf` | Q6_K | ~25 GB | Very Good | High quality |
+| `gemma-4-31b-jang-crack-Q8_0.gguf` | Q8_0 | ~33 GB | Near lossless | Closest to original |
+
+## System Requirements
+
+| Quantization | Minimum RAM | Recommended |
+|-------------|------------|-------------|
+| Q3_K_M | 20 GB | 24 GB |
+| Q4_K_M | 24 GB | 32 GB |
+| Q5_K_M | 28 GB | 36 GB |
+| Q6_K | 32 GB | 40 GB |
+| Q8_0 | 40 GB | 48 GB |
+
+## Usage
+
+### LM Studio
+Download any `.gguf` file and open it in LM Studio.
+
+### llama.cpp
+```bash
+./llama-cli -m gemma-4-31b-jang-crack-Q4_K_M.gguf -p "Hello" -n 256
+```
+
+### Ollama
+```bash
+echo 'FROM ./gemma-4-31b-jang-crack-Q4_K_M.gguf' > Modelfile
+ollama create gemma4-crack -f Modelfile
+ollama run gemma4-crack
+```
+
+## License
+
+[Gemma License](https://ai.google.dev/gemma/terms)
+
+## Disclaimer
+
+This model has had safety guardrails removed. Use responsibly and in compliance with applicable laws.

gemma-4-31b-jang-crack-Q3_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2def38f79f60769ee9d3135c723b7e2add6745ab5653292cfb90e040b4067503
+size 15287102912

gemma-4-31b-jang-crack-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1fc8ee10f916da019dbb1d177854fa3b6421fbea1e93839a2061861308e1de7
+size 18687057344

gemma-4-31b-jang-crack-Q8_0-00001-of-00009.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3194e06834d6e4df8ad8b92701f4ccf26d1eef38b950262fcfd86bd2fa16aab0
+size 3935089728

gemma-4-31b-jang-crack-Q8_0-00002-of-00009.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3038a7dcb6f63544c613af743215327d9ccda8d73ce882e0ad5d22202f92a7c2
+size 3942979136

gemma-4-31b-jang-crack-Q8_0-00003-of-00009.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee63ba2abec0b3817322996305067aabc50360a43a1b68e0627a69e361a97261
+size 3989767648

gemma-4-31b-jang-crack-Q8_0-00004-of-00009.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45308b26548d60f8b966f1382bfa9aee2fc94b2470337715e5de931e9cc8f34e
+size 3884443136

gemma-4-31b-jang-crack-Q8_0-00005-of-00009.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7b26a9c315ffc7e155531092877204cc3ebaf461c5cb0d1cf91fe1f4ba8be96f
+size 3972244096

gemma-4-31b-jang-crack-Q8_0-00006-of-00009.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4fb201454d6561099fd02d7ae4d70f14edc6b17e232c3ddd363f5e6530c1ecd5
+size 3960481088

gemma-4-31b-jang-crack-Q8_0-00007-of-00009.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e294d61e9e911c9e7212f7be44e7b9d00a1202b0a445ea24d81e2d52d08be784
+size 3884486336

gemma-4-31b-jang-crack-Q8_0-00008-of-00009.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8ccb562b088d99ed36515a924b62ffec28217036ac21733646840725b6a3177
+size 3989767648

gemma-4-31b-jang-crack-Q8_0-00009-of-00009.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da198b3dadf0721bcba9d718759b7cb3f5cbd1bd6a01313c9f0b2ce207ab745e
+size 1076412160