Instructions to use watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron")

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron")
model = AutoModelForMultimodalLM.from_pretrained("watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron

SGLang

How to use watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron with Docker Model Runner:
```
docker model run hf.co/watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron
```

Watashiha-Llama-2-13B-Ogiri-sft-neuron / README_en.md

watashihakobashi

Update README_en.md

ccdd91c verified over 2 years ago

preview code

raw

history blame contribute delete

2.59 kB

metadata

license: llama2
language:
  - ja
  - en

Model Overview

This model is a compiled version of Watashiha-Llama-2-13B-Ogiri-sft designed to run on AWS's inf2 instances.

The compilation was done following the instructions in this article: https://huggingface.co/docs/optimum-neuron/tutorials/llama2-13b-chatbot

License: LLAMA 2 COMMUNITY LICENSE]

How to Use

Launch an inf2.xlarge instance on AWS EC2. As downloading the model requires about 50GB, it is recommended to set the storage size to 256GB or more. Please use the following AMI: Deep Learning AMI Neuron PyTorch 1.13 (Ubuntu 20.04) 20240102
Execute the following command to activate the provided Python environment.

source /opt/aws_neuron_venv_pytorch/bin/activate

Install optimum.

pip install optimum[neuronx]

Once the above steps are completed, execute the provided source code.

from optimum.neuron import NeuronModelForCausalLM
from transformers import AutoTokenizer

model_name = "watashiha/Watashiha-Llama-2-13B-Ogiri-sft-neuron"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
model = NeuronModelForCausalLM.from_pretrained(model_name)

odai = "What happens when a clock is hungry?"
text = f"""
Below is a combination of instructions explaining the task and contextually relevant input. Write a response that appropriately fulfills the request.

Instructions:
The input sentence is a prompt for a comedy skit. Generate a funny punchline that aligns with the prompt.

Input:
{odai}

Response:
"""
text = text.lstrip()

token_ids = tokenizer.encode(text, return_tensors="pt")
input_len = token_ids.shape[1]
output_ids = model.generate(
    token_ids,
    max_length=input_len + 64,
    do_sample=True,
    top_p=0.9,
    top_k=50,
    temperature=0.8,
    pad_token_id=tokenizer.pad_token_id,
    eos_token_id=tokenizer.eos_token_id,
)
output = tokenizer.decode(output_ids.tolist()[0], skip_special_tokens=True)
print(output)
"""
Below is a combination of instructions explaining the task and contextually relevant input. Write a response that appropriately fulfills the request.

Instructions:
The input sentence is a prompt for a comedy skit. Generate a funny punchline that aligns with the prompt.

Input:
{odai}

Response:
It takes time to get back on top!
"""

Parameters for compilation

input_shapes

{
    "batch_size": 1,
    "sequence_length": 1024,
}

compiler_args

{
    "num_cores": 2,
    "auto_cast_type": 'bf16',
}