Instructions to use mmnga/DeepSeek-V3-bf16-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- llama-cpp-python
How to use mmnga/DeepSeek-V3-bf16-gguf with llama-cpp-python:
# !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained( repo_id="mmnga/DeepSeek-V3-bf16-gguf", filename="IQ1_M/DeepSeek-V3_IQ1_M-00001-of-00013.gguf", )
llm.create_chat_completion( messages = "No input example has been defined for this model task." )
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use mmnga/DeepSeek-V3-bf16-gguf with llama.cpp:
Install from brew
brew install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf mmnga/DeepSeek-V3-bf16-gguf:IQ1_M # Run inference directly in the terminal: llama-cli -hf mmnga/DeepSeek-V3-bf16-gguf:IQ1_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama-server -hf mmnga/DeepSeek-V3-bf16-gguf:IQ1_M # Run inference directly in the terminal: llama-cli -hf mmnga/DeepSeek-V3-bf16-gguf:IQ1_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf mmnga/DeepSeek-V3-bf16-gguf:IQ1_M # Run inference directly in the terminal: ./llama-cli -hf mmnga/DeepSeek-V3-bf16-gguf:IQ1_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf mmnga/DeepSeek-V3-bf16-gguf:IQ1_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf mmnga/DeepSeek-V3-bf16-gguf:IQ1_M
Use Docker
docker model run hf.co/mmnga/DeepSeek-V3-bf16-gguf:IQ1_M
- LM Studio
- Jan
- Ollama
How to use mmnga/DeepSeek-V3-bf16-gguf with Ollama:
ollama run hf.co/mmnga/DeepSeek-V3-bf16-gguf:IQ1_M
- Unsloth Studio
How to use mmnga/DeepSeek-V3-bf16-gguf with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for mmnga/DeepSeek-V3-bf16-gguf to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for mmnga/DeepSeek-V3-bf16-gguf to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for mmnga/DeepSeek-V3-bf16-gguf to start chatting
- Atomic Chat new
- Docker Model Runner
How to use mmnga/DeepSeek-V3-bf16-gguf with Docker Model Runner:
docker model run hf.co/mmnga/DeepSeek-V3-bf16-gguf:IQ1_M
- Lemonade
How to use mmnga/DeepSeek-V3-bf16-gguf with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull mmnga/DeepSeek-V3-bf16-gguf:IQ1_M
Run and chat with the model
lemonade run user.DeepSeek-V3-bf16-gguf-IQ1_M
List all available models
lemonade list
Add files using upload-large-folder tool
Browse files- .gitattributes +18 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00001-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00002-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00003-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00004-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00005-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00006-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00007-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00008-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00009-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00010-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00011-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00012-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00013-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00014-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00015-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00016-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00017-of-00018.gguf +3 -0
- IQ2_XS/DeepSeek-V3_IQ2_XS-00018-of-00018.gguf +3 -0
.gitattributes
CHANGED
|
@@ -83,3 +83,21 @@ IQ4_NL/DeepSeek-V3_IQ4_NL-00001-of-00035.gguf filter=lfs diff=lfs merge=lfs -tex
|
|
| 83 |
IQ4_NL/DeepSeek-V3_IQ4_NL-00031-of-00035.gguf filter=lfs diff=lfs merge=lfs -text
|
| 84 |
IQ4_NL/DeepSeek-V3_IQ4_NL-00032-of-00035.gguf filter=lfs diff=lfs merge=lfs -text
|
| 85 |
IQ4_NL/DeepSeek-V3_IQ4_NL-00034-of-00035.gguf filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 83 |
IQ4_NL/DeepSeek-V3_IQ4_NL-00031-of-00035.gguf filter=lfs diff=lfs merge=lfs -text
|
| 84 |
IQ4_NL/DeepSeek-V3_IQ4_NL-00032-of-00035.gguf filter=lfs diff=lfs merge=lfs -text
|
| 85 |
IQ4_NL/DeepSeek-V3_IQ4_NL-00034-of-00035.gguf filter=lfs diff=lfs merge=lfs -text
|
| 86 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00018-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 87 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00003-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 88 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00004-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 89 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00006-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 90 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00009-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 91 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00002-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 92 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00008-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 93 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00012-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 94 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00005-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 95 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00011-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 96 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00015-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 97 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00007-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 98 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00017-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 99 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00016-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 100 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00010-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 101 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00013-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 102 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00001-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
| 103 |
+
IQ2_XS/DeepSeek-V3_IQ2_XS-00014-of-00018.gguf filter=lfs diff=lfs merge=lfs -text
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00001-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d9256ac2feb64337526c5b5f76c62cd9ded34b77666bca96cbf2bb14d9d47b62
|
| 3 |
+
size 11816544544
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00002-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6d1ad8c318b810d8e804bc66014f4bbc743cccb0c002d595f3bb89e841132a4d
|
| 3 |
+
size 11090160544
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00003-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3d1218e6cedfe920908f94af16cf0aad848bde40c0942a726e7e87751dcf370e
|
| 3 |
+
size 11097529408
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00004-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8e37c872b14080c5662568f5eef680b132cd0904c2872bba5410aeadb76e52c4
|
| 3 |
+
size 11144284160
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00005-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:44e9835111d618fbe32a62ed44f2fdf282d7a7a7c404242cd5d39d8a920ec030
|
| 3 |
+
size 11090160608
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00006-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7574a7a3962752bbef5061c70fa976b242eefe2d78bdaebbf06f015a0734b6b5
|
| 3 |
+
size 11097529408
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00007-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:00f5f1a2cf8f48669d90b8401bbb90b0b0fd4a8fb60708f2c7ed934da2511843
|
| 3 |
+
size 11144284160
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00008-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:da33729c36ebca87e38162a12cba75a257ea40caa1b87a32809e80f688a5118b
|
| 3 |
+
size 11090160608
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00009-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:89d9b2808b43fddf30bd937ce464e62e211bd39b8a24a7758ff6259f1b5d9101
|
| 3 |
+
size 11097529408
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00010-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cdc89644f37f69d4becc8d767bc54b8987ce51b2e91852d42abf54c2b29d20ef
|
| 3 |
+
size 11144284160
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00011-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3e170c01f0ea3f334e7cbaddb0e733d1f0d1ba3f617da7a05b772c365415630a
|
| 3 |
+
size 11090160608
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00012-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2152a080713bfc4ce611274adb9acfbdf304b5618a62924403ae7d0cd2db26ba
|
| 3 |
+
size 11097529408
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00013-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5454d99d5ff41ef5e09b17c761960a836242bf1186641c21070f23bdc4c5c427
|
| 3 |
+
size 11144284160
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00014-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4f4b37c6bfac0b8d4763cdbf68fb56e9af3ac58d96a0deca2a01d0a7996364a9
|
| 3 |
+
size 11090160608
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00015-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4473e719c1bdbafa4141d46b427b526852fd1935428afaead567d6f743bc7878
|
| 3 |
+
size 11097529408
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00016-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:40880927d6cd7605bed744dd851e6e5699fec1037315e388a80b959fae3b6506
|
| 3 |
+
size 11144284160
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00017-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8529225cf4c5673a3b9d888aa55f208bfdbd4edd54fc1a35ffebe4c6ce5669e0
|
| 3 |
+
size 11090160608
|
IQ2_XS/DeepSeek-V3_IQ2_XS-00018-of-00018.gguf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4f2903a53c6759bd871938ad7e80c21b43ca766d9ac613228e1457080381a98b
|
| 3 |
+
size 5521703008
|