24GB Models
Collection
Models optimized for 24GB VRAM • 7 items • Updated
This is currently a STATIC quant, because the imatrix tool seems to be broken with Gemma 4 (>100 ppl). I will update with an imatrix once I can verify correctness.
I made a custom imatrix dataset by slapping together random columns from some popular datasets on huggingface and formatting using the official jinja template. Comapred to the unstructured bartowski dataset, PPL went from multiple thousands to single digits, so I think it should be good now. Just in case, I mirrored the old static quant to https://huggingface.co/Beinsezii/gemma-4-26B-A4B-it-GGUF-6.52BPW-static
6.52 bpw, a mixture of Q6_K, Q5_K, and Q8_0
Fits ~75k F16 CTX + MMPROJ on a 24GiB GPU, or ~150k CTX without vision. Measured on a DE, not headless.
- Downloads last month
- 3,435
Hardware compatibility
Log In to add your hardware
We're not able to determine the quantization variants.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for Beinsezii/gemma-4-26B-A4B-it-GGUF-6.52BPW
Base model
google/gemma-4-26B-A4B-it