Instructions to use NbAiLab/borealis-27b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NbAiLab/borealis-27b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="NbAiLab/borealis-27b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("NbAiLab/borealis-27b")
model = AutoModelForMultimodalLM.from_pretrained("NbAiLab/borealis-27b")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use NbAiLab/borealis-27b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NbAiLab/borealis-27b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NbAiLab/borealis-27b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/NbAiLab/borealis-27b

SGLang

How to use NbAiLab/borealis-27b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NbAiLab/borealis-27b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NbAiLab/borealis-27b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NbAiLab/borealis-27b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NbAiLab/borealis-27b",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use NbAiLab/borealis-27b with Docker Model Runner:
```
docker model run hf.co/NbAiLab/borealis-27b
```

borealis-27b / signing /ca-chain.pem

versae

Squash Borealis release history

6aa5979 17 days ago

raw

history blame contribute delete

4.47 kB

	-----BEGIN CERTIFICATE-----
	MIIG1jCCBL6gAwIBAgIQSIcdqlEaPQ81vpaZKLkkvzANBgkqhkiG9w0BAQsFADB1
	MQswCQYDVQQGEwJHUjE3MDUGA1UECgwuSGVsbGVuaWMgQWNhZGVtaWMgYW5kIFJl
	c2VhcmNoIEluc3RpdHV0aW9ucyBDQTEtMCsGA1UEAwwkSEFSSUNBIENvZGUgU2ln
	bmluZyBSU0EgUm9vdCBDQSAyMDIxMB4XDTIxMDMxOTA5MjEzNFoXDTM2MDMxNTA5
	MjEzM1owaDELMAkGA1UEBhMCR1IxNzA1BgNVBAoMLkhlbGxlbmljIEFjYWRlbWlj
	IGFuZCBSZXNlYXJjaCBJbnN0aXR1dGlvbnMgQ0ExIDAeBgNVBAMMF0hBUklDQSBD
	b2RlIFNpZ25pbmcgUlNBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA
	31Aluoq9mggXHg7TLFGOUTVDufEJjAKMhSYM4vh98DxFUPCE0soUcvj4AcbfOq0E
	rXG93MW2o/8byyISWPDTu7CKSYzF2x67GEcjAlOvdQn5UZOvI2fw86727Q0GHZ77
	+EfBRkLcLHPEGeVs4mkchtLydKjv42/Lsqu9/5AcYWro03lYf1q++LOmaJjSscvY
	LdAPPnzm+Aa9FIZc3K3JeZ1dxNhT6fAheWeAO4JxnBT2k7Qa1oJqJ1u6t3RYiadj
	shRVAwDq6/WVCW0hysPB9GuQEFkRKgkpQjcvZqG2yCMIVy9cCRQjIQWG9LSnLezv
	wAan4sUIzzgoABiodCU7Gc6uJsshdjHUBiMxQzsbd5rKPZYPk+uiiXaN6V5LP079
	9treRIh6h3h111HITXE2rQDEBsdKrandn4oUTUz3VZMO6XLQA6xpPAinx7ISDdkO
	3VOkEYebMxDF/7lo1GSgrgPvFPRVNWhPxHdVrRkB8/ZKg90HyP0g339NkmDwhVqo
	ZK4EiL5J3033w+U+HNowegNKxn8b1g/vB6ZHlVBGCugsO+77M6Z8/ouk755Jr8TI
	gcyn/CKKadsb748oy/AvqmviFOM+ilORjZVWOFXy82QdjBDapfMOM6JqtSFUTvf5
	OmVXDdIHMvosCWurRYXbVKgTNgOXVgJRAaquQBahtc0CAwEAAaOCAW0wggFpMBIG
	A1UdEwEB/wQIMAYBAf8CAQAwHwYDVR0jBBgwFoAUtGQWSOj8WkszKYnrmUC5ILT2
	YRowXAYIKwYBBQUHAQEEUDBOMEwGCCsGAQUFBzAChkBodHRwOi8vcmVwby5oYXJp
	Y2EuZ3IvY2VydHMvSEFSSUNBLUNvZGVTaWduaW5nLVJvb3QtMjAyMS1SU0EuY2Vy
	MEQGA1UdIAQ9MDswOQYEVR0gADAxMC8GCCsGAQUFBwIBFiNodHRwOi8vcmVwby5o
	YXJpY2EuZ3IvZG9jdW1lbnRzL0NQUzATBgNVHSUEDDAKBggrBgEFBQcDAzBKBgNV
	HR8EQzBBMD+gPaA7hjlodHRwOi8vY3JsLmhhcmljYS5nci9IQVJJQ0EtQ29kZVNp
	Z25pbmctUm9vdC0yMDIxLVJTQS5jcmwwHQYDVR0OBBYEFKnsNRTrbmcFDsRtCQyg
	cgYhtMouMA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsFAAOCAgEAhcU6MtmA
	N9i3luIBF/JqB+CyCqlc9TFuu0d/Gi1PU86Mf6sqIqAObiDcM4J/uL2QWogUk5GZ
	b1bNCRBrmhPv9C8rFsbeNKJYdDEWtslXQrnnd9vpAKkurG9imNW4RE/jsPV28T2I
	iHWEYA0zGiP0qyAyEVvwSntEpFigaslhLFtVqf0uCGIvHRMfdD53WTXq37sInwcG
	/W2C0zcno3PUK8qtCvF7cO1jujGxsLG/h0blm+M6b09doO1iSq3SAO4kwue+7AvB
	97ppo4XKgcp1Kq6LGl4Rzac23KeMRkucwofNILRbIBwxdaeudozb6XIvgGeNPmFq
	71HdLzBS0i9Tyxsf4VjlJj1vazncSFxz56RDNyendlYR39bZrQ6FMr6W5/vt1cwe
	e7HlVnnQvdNMiFX/+uWH8kz29rP5q1NEQx8xjV7WnDqADnbTzGoUg9P3GR5Mv4m9
	tSktUu6WqserABWf8e0L3ClvUKcnCeG/lyXbmQWxRIGrWAMmUSP1/BVajITUrEDW
	qrrBmUlCoSpa0OZ7lfmlQ/QtaWvwrGRNgbLx7RgqlxsjeJtZUoBwC3nFNO6j80QB
	CQEdsxUWKYZmXTnSuiG63O/wON+TmWrM7SbxqbiZRIGw90aHT2bK+8CfMgSdxl1k
	JT844f3ByJb2qHA+z2lw0JqwVp07ORkrwwE=
	-----END CERTIFICATE-----
	-----BEGIN CERTIFICATE-----
	MIIFtjCCA56gAwIBAgIQFcKuKk2ZmmOM07oTGXYI9TANBgkqhkiG9w0BAQsFADB1
	MQswCQYDVQQGEwJHUjE3MDUGA1UECgwuSGVsbGVuaWMgQWNhZGVtaWMgYW5kIFJl
	c2VhcmNoIEluc3RpdHV0aW9ucyBDQTEtMCsGA1UEAwwkSEFSSUNBIENvZGUgU2ln
	bmluZyBSU0EgUm9vdCBDQSAyMDIxMB4XDTIxMDIxOTEwNTk1NFoXDTQ1MDIxMzEw
	NTk1M1owdTELMAkGA1UEBhMCR1IxNzA1BgNVBAoMLkhlbGxlbmljIEFjYWRlbWlj
	IGFuZCBSZXNlYXJjaCBJbnN0aXR1dGlvbnMgQ0ExLTArBgNVBAMMJEhBUklDQSBD
	b2RlIFNpZ25pbmcgUlNBIFJvb3QgQ0EgMjAyMTCCAiIwDQYJKoZIhvcNAQEBBQAD
	ggIPADCCAgoCggIBAIpq7qoKI1UMiwZC45VbVHhfxY4GLHw8Mb6vDamh7EogWAWd
	4miyu+tffyozufJVnG+qpB7tEL6DKRE25p4/+m17UeHVd6W9y2kOOyIglAwxZUAN
	Ca8QNXqb6nkIRKSLZ6krTcHn0Nen9rU6jdmjqXm4pGVcvPM+95+Z9rjDZWgtq4Mu
	3YWZBKn10VzVUUIBuZ9BtUsisgD0y2cQ72nEEK36lAZ2UBJXgq7FFK08fbud8XPh
	fPCucd5b9xLd94Dx3D7xYLrJGZdvdXSFTP05Q8NkbjSsE8EcZbZSQvFG6y/6XabP
	Dkmd93R8eA0GOiwS++JLJuRviyGBM0I5E+hCq1tV1Bu7N7YSkffzfbzS/+yxw9Wg
	zLIrx49dfkxSQkj7j/1akHMQ1alyVI1J2zhSqkZIDi/6ACzMcww2SyTOE2ympKSj
	1nyb6TgLxiTbTaxnIUlbQTdk5mBrH+0qL2AZMB7YPJwZQ4ffwAr05MpgiHrWo7nk
	JS554v7Byzy3989OWEz7w62rfarKiPsOOA0fnlw567gHxFAiTfiFf27sjPq2cU3e
	fZZpxN0+Ht4mkAosTRWVqaI+3D0Od3yNQShPuINRvj63j5A6cDGJWvqTU/xgyY11
	kO5aLx2EnwCp5sOGI6Ie3RLjoUZgG2e/UBUiknxKtIyPbpyVwizcOzogtryLAgMB
	AAGjQjBAMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFLRkFkjo/FpLMymJ65lA
	uSC09mEaMA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsFAAOCAgEAK6xZi+Y5
	RNz7rUqI4GT/q8Py0s5wLwPlb7LE9TbWU2uHqx2LmQfIMI5Hcv2huHXWF6EWvWRz
	kTBXrX39BUCGk9FJlrIL1HxkE4vTIUk4u/vozD4x8a57TCrfggajjgWEnH9wzsC7
	RRffXWUM5FBIB0SLpC9W2sZ69mCNpDjxnYcWMLLwX3pCVSBxz8zuALESR79CueSz
	tR0ZhLmYbeBptBVBteJQSMcf6pq3eZHd1dVTGfyuGGxp261ZKJtkmKc8x1WO0TAA
	z9QynmKxwyruNSLGu/B8u4O4AIn13BqX2TiaKVNhGaan9DtH/d1nlQ+OpGYsgpq3
	LXHm3674+GjBuw/qUKBFl7calRJvyLO2BWjaah4ONYWErHQMirT0dZ8ir7BUu7ae
	IvnZ69WKDcfc+JgxX576xpf0QRB104G2MV963YiFCK9wRwI3e+JK7F3y3SkSRMiK
	qt3SVXgXda9xaQ13cEqwH39C28dx3FjWGL1QxblyBMdne3xTYMpJGBW8QHOuK6gr
	rG8RRDnsgkh+Ecr71j+2wLm4BpN1vZMndxcNXKWe6sFbAC0MqDVgx+JvNRs9drXn
	rd10I0+GskfvyC/QhXw5ljehKykCvT6Hz6j2HXUyzzjjc7HuEIG1KrGIBlEeWjpI
	UfM2WWLfQmZZULdYvvV2QJoSFuYOqtdsoNU=
	-----END CERTIFICATE-----