Instructions to use raincandy-u/Qwen1.5-1.8B_llamafy with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use raincandy-u/Qwen1.5-1.8B_llamafy with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="raincandy-u/Qwen1.5-1.8B_llamafy")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("raincandy-u/Qwen1.5-1.8B_llamafy")
model = AutoModelForMultimodalLM.from_pretrained("raincandy-u/Qwen1.5-1.8B_llamafy")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use raincandy-u/Qwen1.5-1.8B_llamafy with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "raincandy-u/Qwen1.5-1.8B_llamafy"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "raincandy-u/Qwen1.5-1.8B_llamafy",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/raincandy-u/Qwen1.5-1.8B_llamafy

SGLang

How to use raincandy-u/Qwen1.5-1.8B_llamafy with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "raincandy-u/Qwen1.5-1.8B_llamafy" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "raincandy-u/Qwen1.5-1.8B_llamafy",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "raincandy-u/Qwen1.5-1.8B_llamafy" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "raincandy-u/Qwen1.5-1.8B_llamafy",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use raincandy-u/Qwen1.5-1.8B_llamafy with Docker Model Runner:
```
docker model run hf.co/raincandy-u/Qwen1.5-1.8B_llamafy
```

Llamafied Qwen

This is a conversion of the Qwen1.5-1.8B model, adapted to the LLama architecture, aiming to augment its generality and suitability for academic research and broader computational linguistics applications.

Disclaimer

This conversion of the Qwen model is intended for research and educational purposes only. It is important to note that the converted model may generate more unpredictable responses compared to its original version. The user assumes full responsibility for any outcomes or consequences arising from the use of this converted model.

Acknowledgments

Special thanks go to @Minami-su for developing the conversion script that made this possible.

Downloads last month: 3

Safetensors

Model size

2B params

Tensor type

BF16

Collection including raincandy-u/Qwen1.5-1.8B_llamafy

Llamafied Models

Collection

This is a collection of llamafied models - such as Qwen. • 5 items • Updated Apr 19, 2024 • 1