Instructions to use mradermacher/mathexpert7b-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mradermacher/mathexpert7b-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("mradermacher/mathexpert7b-GGUF", dtype="auto")

llama-cpp-python

How to use mradermacher/mathexpert7b-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="mradermacher/mathexpert7b-GGUF",
	filename="mathexpert7b.IQ4_XS.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use mradermacher/mathexpert7b-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf mradermacher/mathexpert7b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf mradermacher/mathexpert7b-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf mradermacher/mathexpert7b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf mradermacher/mathexpert7b-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf mradermacher/mathexpert7b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf mradermacher/mathexpert7b-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf mradermacher/mathexpert7b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf mradermacher/mathexpert7b-GGUF:Q4_K_M

Use Docker

docker model run hf.co/mradermacher/mathexpert7b-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use mradermacher/mathexpert7b-GGUF with Ollama:
```
ollama run hf.co/mradermacher/mathexpert7b-GGUF:Q4_K_M
```

Unsloth Studio

How to use mradermacher/mathexpert7b-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mradermacher/mathexpert7b-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for mradermacher/mathexpert7b-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for mradermacher/mathexpert7b-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use mradermacher/mathexpert7b-GGUF with Docker Model Runner:
```
docker model run hf.co/mradermacher/mathexpert7b-GGUF:Q4_K_M
```

Lemonade

How to use mradermacher/mathexpert7b-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull mradermacher/mathexpert7b-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.mathexpert7b-GGUF-Q4_K_M

List all available models

lemonade list

mradermacher commited on Dec 20, 2024

Commit

bcc88c6

verified ·

1 Parent(s): 4333c76

uploaded from nico1

Browse files

Files changed (13) hide show

.gitattributes +12 -0
mathexpert7b.IQ4_XS.gguf +3 -0
mathexpert7b.Q2_K.gguf +3 -0
mathexpert7b.Q3_K_L.gguf +3 -0
mathexpert7b.Q3_K_M.gguf +3 -0
mathexpert7b.Q3_K_S.gguf +3 -0
mathexpert7b.Q4_K_M.gguf +3 -0
mathexpert7b.Q4_K_S.gguf +3 -0
mathexpert7b.Q5_K_M.gguf +3 -0
mathexpert7b.Q5_K_S.gguf +3 -0
mathexpert7b.Q6_K.gguf +3 -0
mathexpert7b.Q8_0.gguf +3 -0
mathexpert7b.f16.gguf +3 -0

@@ -33,3 +33,15 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+mathexpert7b.IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
+mathexpert7b.Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+mathexpert7b.Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+mathexpert7b.Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+mathexpert7b.Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+mathexpert7b.Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+mathexpert7b.Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+mathexpert7b.Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+mathexpert7b.Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+mathexpert7b.Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+mathexpert7b.Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+mathexpert7b.f16.gguf filter=lfs diff=lfs merge=lfs -text

+version https://git-lfs.github.com/spec/v1
+oid sha256:2fb44cc50ba772dfacc15565158a0b8e10c6d321434d5dda8d4073c0d9d28d7e
+size 3944389216

+version https://git-lfs.github.com/spec/v1
+oid sha256:b3fdbe35745cd472e198d45b51710a48c218f3fa69431e6377b244ff89602a78
+size 2719242848

+version https://git-lfs.github.com/spec/v1
+oid sha256:f0c5f2d9e087bce91d23e18e729090cb341f6476f890efb1044a17b81b98edb2
+size 3822025312

+version https://git-lfs.github.com/spec/v1
+oid sha256:21318b4b4178dec1e50d4858ced43b66dd51f17fce1520594593202bb150dbc1
+size 3518986848

+version https://git-lfs.github.com/spec/v1
+oid sha256:d2086f01b55142bf228bdbfa3af03451c0a051f961bd216def2eaf74d88fa28d
+size 3164568160

+version https://git-lfs.github.com/spec/v1
+oid sha256:b6322887ca78c102347923f0b70b1329b4e2033e853b5fa3a7ee36cf7c43cc0c
+size 4368439904

+version https://git-lfs.github.com/spec/v1
+oid sha256:e1f180a331ce7dd170447fe4f7801c53e7de68ed142a8cde99ff3b4f9e34b2d3
+size 4140374624

+version https://git-lfs.github.com/spec/v1
+oid sha256:d20c69a5580b7c219871a2970068170a455740007e4ac71d98135e19097f8191
+size 5131410016

+version https://git-lfs.github.com/spec/v1
+oid sha256:5e886d7eea64cfc793955384dcdbb1f58a101eb28470682ddfb4b421297ed9ad
+size 4997716576

+version https://git-lfs.github.com/spec/v1
+oid sha256:5deac33d787292559f54b9f986a7c6b54b02b170319a58bd19440d23eddc751a
+size 5942065760

	@@ -0,0 +1,3 @@

	@@ -0,0 +1,3 @@

	@@ -0,0 +1,3 @@

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:cbc6db4d887af2fc71e66657fecc2968f32a00bb6d695e72115aadaf962bc2ce
+size 7695858272

+version https://git-lfs.github.com/spec/v1
+oid sha256:b78d0fb4c5a5189ae9bf92e1ce32470fb44cc6b87e1425f06701585f553db78f
+size 14484732512