Instructions to use upstage/Solar-Open-100B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use upstage/Solar-Open-100B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="upstage/Solar-Open-100B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("upstage/Solar-Open-100B")
model = AutoModelForCausalLM.from_pretrained("upstage/Solar-Open-100B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use upstage/Solar-Open-100B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "upstage/Solar-Open-100B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "upstage/Solar-Open-100B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/upstage/Solar-Open-100B

SGLang

How to use upstage/Solar-Open-100B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "upstage/Solar-Open-100B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "upstage/Solar-Open-100B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "upstage/Solar-Open-100B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "upstage/Solar-Open-100B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use upstage/Solar-Open-100B with Docker Model Runner:
```
docker model run hf.co/upstage/Solar-Open-100B
```

[Tip] Running Solar-Open-100B on vLLM - workaround for two compatibility issues

#25

by davi0600 - opened Feb 2

Discussion

davi0600

Feb 2

Following the issue reported in the last discussion, I found a tricky workaround that gets vLLM working with Solar-Open-100B for now.

Fix 1: ALLOWED_LAYER_TYPES ImportError

vllm/config/model.py, line 14:

# Before
from transformers.configuration_utils import ALLOWED_LAYER_TYPES

# After
from transformers.configuration_utils import ALLOWED_MLP_LAYER_TYPES
ALLOWED_LAYER_TYPES = ALLOWED_MLP_LAYER_TYPES

Fix 2: use_qk_norm AttributeError in SolarOpenDecoderLayer

vllm/model_executor/models/solar_open.py, in SolarOpenDecoderLayer class:

# Before
use_qk_norm=config.use_qk_norm,
# After
use_qk_norm=getattr(config, "use_qk_norm", False),

After these two patches, vllm serve upstage/Solar-Open-100B --tensor-parallel-size 4 loads and generates correctly.
These are hacky workarounds, not proper upstream fixes. But for anyone stuck right now, this should get you going.
Env: Upstage custom vLLM (0.12.1.dev1+solaropen) / transformers==5.0.0 / CUDA 12.8 / 4x GPU

hancheol

upstage org Feb 13

There was one more place to patch.

diff -ruN a/vllm/config/model.py b/vllm/config/model.py
--- a/vllm/config/model.py
+++ b/vllm/config/model.py
@@ -11,7 +11,8 @@
 from pydantic import ConfigDict, SkipValidation, field_validator, model_validator
 from pydantic.dataclasses import dataclass
 from safetensors.torch import _TYPES as _SAFETENSORS_TO_TORCH_DTYPE
-from transformers.configuration_utils import ALLOWED_LAYER_TYPES
+from transformers.configuration_utils import ALLOWED_MLP_LAYER_TYPES
+ALLOWED_LAYER_TYPES = ALLOWED_MLP_LAYER_TYPES
 import vllm.envs as envs
 from vllm.attention.backends.registry import AttentionBackendEnum
 from vllm.config.multimodal import MMCacheType, MMEncoderTPMode, MultiModalConfig
diff -ruN a/vllm/model_executor/models/solar_open.py b/vllm/model_executor/models/solar_open.py
--- a/vllm/model_executor/models/solar_open.py
+++ b/vllm/model_executor/models/solar_open.py
@@ -351,7 +351,7 @@
             cache_config=cache_config,
             quant_config=quant_config,
             prefix=f"{prefix}.self_attn",
-            use_qk_norm=config.use_qk_norm,
+            use_qk_norm=getattr(config, "use_qk_norm", False),
         )

         if (
diff -ruN a/vllm/transformers_utils/config.py b/vllm/transformers_utils/config.py
--- a/vllm/transformers_utils/config.py
+++ b/vllm/transformers_utils/config.py
@@ -15,7 +15,8 @@
 )
 from packaging.version import Version
 from transformers import GenerationConfig, PretrainedConfig
-from transformers.configuration_utils import ALLOWED_LAYER_TYPES
+from transformers.configuration_utils import ALLOWED_MLP_LAYER_TYPES
+ALLOWED_LAYER_TYPES = ALLOWED_MLP_LAYER_TYPES
 from transformers.models.auto.image_processing_auto import get_image_processor_config
 from transformers.models.auto.modeling_auto import (
     MODEL_FOR_CAUSAL_LM_MAPPING_NAMES,

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment