| --- |
| license: mit |
| quantized_by: Pomni |
| language: |
| - en |
| base_model: |
| - distil-whisper/distil-large-v2 |
| pipeline_tag: automatic-speech-recognition |
| datasets: |
| - mozilla-foundation/common_voice_13_0 |
| - facebook/voxpopuli |
| - LIUM/tedlium |
| - MLCommons/peoples_speech |
| - speechcolab/gigaspeech |
| - edinburghcstr/ami |
| tags: |
| - whisper.cpp |
| - ggml |
| - whisper |
| - audio |
| - speech |
| - voice |
| - distil |
| new_version: Pomni/distil-large-v3.5-ggml-allquants |
| --- |
| # Distil-Large-v2 quants |
| This is a repository of **GGML quants for [distil-large-v2](https://huggingface.co/distil-whisper/distil-large-v2)** (a Whisper-based transcription model), for use with [whisper.cpp](https://github.com/ggml-org/whisper.cpp). |
|
|
| If you are looking for a program to run this model with, then I would recommend [EasyWhisper UI](https://github.com/mehtabmahir/easy-whisper-ui), as it is user-friendly, has a GUI, and automates a lot of the hard stuff for you. |
| ## List of Quants |
| Clicking on a link will download the corresponding quant instantly. |
| | Link | Quant | Size | Notes |
| |:-----|:-----|--------:|:------| |
| | [GGML](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/resolve/main/ggml-distil-large-v2-f32.bin) | F32 | 3.03 GB | Likely overkill. | |
| | [GGML](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/resolve/main/ggml-distil-large-v2-f16.bin) | F16 | 1.52 GB | Performs better than Q8_0 for noisy audio and music. | |
| | [GGML](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/resolve/main/ggml-distil-large-v2-q8_0.bin) | Q8_0 | 818 MB | Sweet spot; superficial quality loss at nearly double the speed. | |
| | [GGML](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/resolve/main/ggml-distil-large-v2-q6_k.bin) | Q6_K | 637 MB | | |
| | [GGML](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/resolve/main/ggml-distil-large-v2-q5_k.bin) | Q5_K | 537 MB | | |
| | [GGML](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/resolve/main/ggml-distil-large-v2-q5_1.bin) | Q5_1 | 584 MB | | |
| | [GGML](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/resolve/main/ggml-distil-large-v2-q5_0.bin) | Q5_0 | 537 MB | Last "good" quant; anything below loses quality rapidly. | |
| | [GGML](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/resolve/main/ggml-distil-large-v2-q4_k.bin) | Q4_K | 444 MB | *Might* not have lost too much quality, but I'm not sure. | |
| | [GGML](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/resolve/main/ggml-distil-large-v2-q4_1.bin) | Q4_1 | 491 MB | | |
| | [GGML](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/resolve/main/ggml-distil-large-v2-q4_0.bin) | Q4_0 | 444 MB | | |
| | [GGML](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/resolve/main/ggml-distil-large-v2-q3_k.bin) | Q3_K | 345 MB | | |
| | [GGML](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/resolve/main/ggml-distil-large-v2-q2_k.bin) | Q2_K | 269 MB | Completely non-sensical output. | |
| |
| The F32 quant was taken from [distil-whisper/distil-large-v2/ggml-large-32-2.fp32.en.bin](https://huggingface.co/distil-whisper/distil-large-v2/blob/main/ggml-large-32-2.fp32.en.bin), and the F16 quant was taken from [distil-whisper/distil-large-v2/ggml-large-32-2.en.bin](https://huggingface.co/distil-whisper/distil-large-v2/blob/main/ggml-large-32-2.en.bin). |
| ## Questions you may have |
| ### Why do the "K-quants" not work for me? |
| My guess is that your GPU might be too old to recognize them, considering that I have gotten the same error on my GTX 1080. If you would like to run them regardless, you can try switching to CPU inference. |
| ### Are the K-quants "S", "M", or "L"? |
| The quantizer I was using was not specific about this, so I do not know about this either. |
| ### What program did you use to make these quants? |
| I used [whisper.cpp v1.7.6](https://github.com/ggml-org/whisper.cpp/releases/tag/v1.7.6) on Windows x64, leveraging CUDA 12.4.0. |
| ### One or multiple of the quants are not working for me. |
| [Open a new discussion](https://huggingface.co/Pomni/distil-large-v2-ggml-allquants/discussions/new) in the community tab about this, and I will look into the issue. |
