Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -24,23 +24,20 @@ language:
|
|
| 24 |
|
| 25 |
## TQ3_4S Release
|
| 26 |
|
| 27 |
-
This repository packages the model as a TurboQuant `TQ3_4S` GGUF for local deployment.
|
| 28 |
-
|
| 29 |
-
- **Text-only chat / coding:** use `Qwen3.6-27B-TQ3_4S.gguf` only.
|
| 30 |
-
- **Image-to-text / multimodal:** use `Qwen3.6-27B-TQ3_4S.gguf` together with `mmproj.gguf`.
|
| 31 |
|
| 32 |
## Runtime Compatibility
|
| 33 |
|
| 34 |
This quant requires a TurboQuant-capable runtime. For llama.cpp, use the `turbo-tan/llama.cpp-tq3` fork rather than stock upstream llama.cpp if you want native `TQ3_4S` support.
|
| 35 |
|
| 36 |
- TurboQuant runtime fork: [turbo-tan/llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3)
|
|
|
|
| 37 |
|
| 38 |
## Files
|
| 39 |
|
| 40 |
| File | Quant | Size |
|
| 41 |
| --- | --- | ---: |
|
| 42 |
-
| `Qwen3.6-27B-TQ3_4S.gguf` | TQ3_4S
|
| 43 |
-
| `mmproj.gguf` | Qwen3.6-27B vision projector | ~889 MB |
|
| 44 |
| `chat_template.jinja` | chat template | text |
|
| 45 |
| `thumbnail.png` | model card image | png |
|
| 46 |
|
|
@@ -66,44 +63,11 @@ Prompt processing:
|
|
| 66 |
|
| 67 |
- Use a TurboQuant-capable llama.cpp build for best performance.
|
| 68 |
- For llama.cpp, the intended runtime is the `turbo-tan/llama.cpp-tq3` fork.
|
| 69 |
-
-
|
| 70 |
-
- Image-to-text usage requires `mmproj.gguf`; pass it with `--mmproj mmproj.gguf` when using `llama-server` or other compatible llama.cpp tools.
|
| 71 |
- For llama.cpp chat usage, keep `--jinja` enabled so the bundled chat template is honored.
|
| 72 |
- Upstream guidance recommends keeping at least `128K` context when possible for reasoning-heavy workloads. On smaller local GPUs, reduce context as needed to fit memory.
|
| 73 |
- Upstream default sampling guidance differs between thinking and non-thinking mode; follow the official Qwen card if you are trying to reproduce base-model behavior.
|
| 74 |
|
| 75 |
-
|
| 76 |
-
## Text-Only vs Image-To-Text
|
| 77 |
-
|
| 78 |
-
### Text-only
|
| 79 |
-
|
| 80 |
-
For normal chat, coding, and text generation, load only the main model:
|
| 81 |
-
|
| 82 |
-
```bash
|
| 83 |
-
llama-server \
|
| 84 |
-
-m Qwen3.6-27B-TQ3_4S.gguf \
|
| 85 |
-
-ngl 99 -c 4096 -np 1 \
|
| 86 |
-
-ctk q4_0 -ctv tq3_0 -fa on \
|
| 87 |
-
--jinja --reasoning off --reasoning-budget 0
|
| 88 |
-
```
|
| 89 |
-
|
| 90 |
-
### Image-to-text
|
| 91 |
-
|
| 92 |
-
For vision/image prompts, also load the projector:
|
| 93 |
-
|
| 94 |
-
`mmproj.gguf` was smoke-tested with the `turbo-tan/llama.cpp-tq3` `llama-server` runtime on RTX 5060 Ti. The server loaded the projector as a Qwen-VL multimodal model and `/health` returned `ok`.
|
| 95 |
-
|
| 96 |
-
Validated smoke-test settings:
|
| 97 |
-
|
| 98 |
-
```bash
|
| 99 |
-
llama-server \
|
| 100 |
-
-m Qwen3.6-27B-TQ3_4S.gguf \
|
| 101 |
-
--mmproj mmproj.gguf \
|
| 102 |
-
-ngl 99 -c 2048 -np 1 \
|
| 103 |
-
-ctk q4_0 -ctv tq3_0 -fa on \
|
| 104 |
-
--jinja --reasoning off --reasoning-budget 0
|
| 105 |
-
```
|
| 106 |
-
|
| 107 |
## Recommended llama.cpp Settings
|
| 108 |
|
| 109 |
Default prompt-processing settings on 16 GB:
|
|
@@ -118,23 +82,11 @@ llama-bench \
|
|
| 118 |
-p 2048 -n 0 -r 3
|
| 119 |
```
|
| 120 |
|
| 121 |
-
Default
|
| 122 |
-
|
| 123 |
-
```bash
|
| 124 |
-
llama-server \
|
| 125 |
-
-m Qwen3.6-27B-TQ3_4S.gguf \
|
| 126 |
-
--host 127.0.0.1 --port 8080 \
|
| 127 |
-
-ngl 99 -c 4096 -np 1 \
|
| 128 |
-
-ctk q4_0 -ctv tq3_0 -fa on \
|
| 129 |
-
--jinja
|
| 130 |
-
```
|
| 131 |
-
|
| 132 |
-
Image-to-text server settings:
|
| 133 |
|
| 134 |
```bash
|
| 135 |
llama-server \
|
| 136 |
-m Qwen3.6-27B-TQ3_4S.gguf \
|
| 137 |
-
--mmproj mmproj.gguf \
|
| 138 |
--host 127.0.0.1 --port 8080 \
|
| 139 |
-ngl 99 -c 4096 -np 1 \
|
| 140 |
-ctk q4_0 -ctv tq3_0 -fa on \
|
|
|
|
| 24 |
|
| 25 |
## TQ3_4S Release
|
| 26 |
|
| 27 |
+
This repository packages the model as a TurboQuant `TQ3_4S` GGUF for local deployment.
|
|
|
|
|
|
|
|
|
|
| 28 |
|
| 29 |
## Runtime Compatibility
|
| 30 |
|
| 31 |
This quant requires a TurboQuant-capable runtime. For llama.cpp, use the `turbo-tan/llama.cpp-tq3` fork rather than stock upstream llama.cpp if you want native `TQ3_4S` support.
|
| 32 |
|
| 33 |
- TurboQuant runtime fork: [turbo-tan/llama.cpp-tq3](https://github.com/turbo-tan/llama.cpp-tq3)
|
| 34 |
+
- LM Studio setup: [docs/backend/LMStudio.md](https://github.com/turbo-tan/llama.cpp-tq3/blob/main/docs/backend/LMStudio.md)
|
| 35 |
|
| 36 |
## Files
|
| 37 |
|
| 38 |
| File | Quant | Size |
|
| 39 |
| --- | --- | ---: |
|
| 40 |
+
| `Qwen3.6-27B-TQ3_4S.gguf` | TQ3_4S | ~13.0 GB |
|
|
|
|
| 41 |
| `chat_template.jinja` | chat template | text |
|
| 42 |
| `thumbnail.png` | model card image | png |
|
| 43 |
|
|
|
|
| 63 |
|
| 64 |
- Use a TurboQuant-capable llama.cpp build for best performance.
|
| 65 |
- For llama.cpp, the intended runtime is the `turbo-tan/llama.cpp-tq3` fork.
|
| 66 |
+
- The upstream family is multimodal-capable, but the public 27B repos used here do not currently expose a separate GGUF `mmproj` artifact.
|
|
|
|
| 67 |
- For llama.cpp chat usage, keep `--jinja` enabled so the bundled chat template is honored.
|
| 68 |
- Upstream guidance recommends keeping at least `128K` context when possible for reasoning-heavy workloads. On smaller local GPUs, reduce context as needed to fit memory.
|
| 69 |
- Upstream default sampling guidance differs between thinking and non-thinking mode; follow the official Qwen card if you are trying to reproduce base-model behavior.
|
| 70 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 71 |
## Recommended llama.cpp Settings
|
| 72 |
|
| 73 |
Default prompt-processing settings on 16 GB:
|
|
|
|
| 82 |
-p 2048 -n 0 -r 3
|
| 83 |
```
|
| 84 |
|
| 85 |
+
Default chat/server settings:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 86 |
|
| 87 |
```bash
|
| 88 |
llama-server \
|
| 89 |
-m Qwen3.6-27B-TQ3_4S.gguf \
|
|
|
|
| 90 |
--host 127.0.0.1 --port 8080 \
|
| 91 |
-ngl 99 -c 4096 -np 1 \
|
| 92 |
-ctk q4_0 -ctv tq3_0 -fa on \
|