Instructions to use midorin-Linux/plamo-3-nict-8b-base-bnb-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use midorin-Linux/plamo-3-nict-8b-base-bnb-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="midorin-Linux/plamo-3-nict-8b-base-bnb-4bit", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("midorin-Linux/plamo-3-nict-8b-base-bnb-4bit", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use midorin-Linux/plamo-3-nict-8b-base-bnb-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "midorin-Linux/plamo-3-nict-8b-base-bnb-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "midorin-Linux/plamo-3-nict-8b-base-bnb-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/midorin-Linux/plamo-3-nict-8b-base-bnb-4bit

SGLang

How to use midorin-Linux/plamo-3-nict-8b-base-bnb-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "midorin-Linux/plamo-3-nict-8b-base-bnb-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "midorin-Linux/plamo-3-nict-8b-base-bnb-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "midorin-Linux/plamo-3-nict-8b-base-bnb-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "midorin-Linux/plamo-3-nict-8b-base-bnb-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use midorin-Linux/plamo-3-nict-8b-base-bnb-4bit with Docker Model Runner:
```
docker model run hf.co/midorin-Linux/plamo-3-nict-8b-base-bnb-4bit
```

midorin-Linux/plamo-3-nict-8b-base-bnb-4bit

This model is a bnb 4bit version of pfnet/plamo-3-nict-8b-base.

Base model: pfnet/plamo-3-nict-8b-base

Model Description

PLaMo 3 NICT 8B Base is a 8B model pre-trained on English and Japanese datasets, developed by Preferred Networks, Inc. collaborative with National Institute of Information and Communications Technology, NICT.

PLaMo 3 NICT models adapt a hybrid architecture with Sliding Window Attention (SWA) and Traditional Attetntion layers.

PLaMo 3 NICT 8B Base is released under PLaMo community license. Please check the following license and agree to this before downloading.

NOTE: This model has NOT been instruction-tuned for chat dialog or other downstream tasks.

For commercial users

Please check the PLaMo community license and contact us via the following form to use commercial purpose.

(EN/JA) https://forms.gle/mTL8tBLrMYXKNZD56

Usage

Requirements

python>=3.10.0
numpy>=1.26.4
numba>=0.60.0
torch>=2.6.0,<=2.9.0
transformers>=4.56.0,<=4.57.1

Use vLLM

We provide the implementation via vLLM plugin.

See more details to setup the plugin for PLaMo 3 models: https://github.com/pfnet-research/vllm-plamo3-plugin

import vllm
model = vllm.LLM("midorin-Linux/plamo-3-nict-8b-base-bnb-4bit", trust_remote_code=True)
response = model.generate(
    prompts=["The future of artificial intelligence technology is "],
    sampling_params=vllm.SamplingParams(
        n=1,
        max_tokens=32,
        top_k=50,
        top_p=0.95,
        temperature=1.0,
    ),
)[0]
generated_text = response.outputs[0].text
print(generated_text)

Load model directly

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("midorin-Linux/plamo-3-nict-8b-base-bnb-4bit", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("midorin-Linux/plamo-3-nict-8b-base-bnb-4bit", trust_remote_code=True)
text = "これからの人工知能技術は"
input_ids = tokenizer(text, return_tensors="pt").input_ids
generated_tokens = model.generate(
    inputs=input_ids,
    max_new_tokens=32,
    do_sample=True,
    top_k=50,
    top_p=0.95,
    temperature=1.0,
)[0]
generated_text = tokenizer.decode(generated_tokens)
print(generated_text)

Model Details

Model size: 8B
Trained tokens: 800B
Developed by:
- Preferred Networks, Inc.
- collaborative with National Institute of Information and Communications Technology, NICT
Model type: Causal decoder-only
Language(s): English, Japanese
License: PLaMo community license

Training Dataset

We trained PLaMo 3 NICT 8B Base in two phases, phase 1 with 875B tokens and phase 2 with 125B tokens. The percentage of datasets in each phase is shown in the following table.

	875B (phase 1)	125B (phase 2)	Tokens
English	45%	35%	437.5B
Japanese	30%	40%	312.5B
Code	15%	15%	150.0B
Other	10%	10%	100.0B

The training dataset also includes the following publicly available community resources:

Tokenizer

PLaMo 3 NICT 8B Base tokenizer is optimized by numba, which is JIT compiler for numerical functions. The tokenizer is trained on a subst of the datasets for model pre-training.

Bias, Risks, and Limitations

PLaMo 3 NICT 8B Base is a new technology that carries risks with use. Testing conducted to date has been in English and Japanese, and has not covered, nor could it cover all scenarios. For these reasons, as with all LLMs, PLaMo 3 NICT 8B Base's potential outputs cannot be predicted in advance, and the model may in some instances produce inaccurate, biased or other objectionable responses to user prompts. Therefore, before deploying any applications of PLaMo 3 NICT 8B Base, developers should perform safety testing and tuning tailored to their specific applications of the model.

AI policies for Preferred Networks, Inc. group

Built with PLaMo

Downloads last month: 36

Safetensors

Model size

8B params

Tensor type

F32

F16

Model tree for midorin-Linux/plamo-3-nict-8b-base-bnb-4bit

Base model

pfnet/plamo-3-nict-8b-base

Quantized

(2)

this model

Collection including midorin-Linux/plamo-3-nict-8b-base-bnb-4bit

PLaMo 3

Collection

Created for personal research using the PLaMo series. • 2 items • Updated 23 days ago