Instructions to use medalpaca/medalpaca-13b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use medalpaca/medalpaca-13b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="medalpaca/medalpaca-13b")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("medalpaca/medalpaca-13b")
model = AutoModelForCausalLM.from_pretrained("medalpaca/medalpaca-13b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use medalpaca/medalpaca-13b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "medalpaca/medalpaca-13b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "medalpaca/medalpaca-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/medalpaca/medalpaca-13b

SGLang

How to use medalpaca/medalpaca-13b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "medalpaca/medalpaca-13b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "medalpaca/medalpaca-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "medalpaca/medalpaca-13b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "medalpaca/medalpaca-13b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use medalpaca/medalpaca-13b with Docker Model Runner:
```
docker model run hf.co/medalpaca/medalpaca-13b
```

TypeError: forward() got an unexpected keyword argument 'token_type_ids'

by rsml - opened May 22, 2023

Discussion

rsml

May 22, 2023

Getting this error when trying to run the qa script from 'Model Usage'. Is there a workaround?

Trace:

Traceback (most recent call last):
File "/Users/rsml/Documents/medAlpaca/test.py", line 6, in
answer = qa_pipeline({"question": question, "context": context})
File "/Users/rsml/miniconda3/lib/python3.9/site-packages/transformers/pipelines/question_answering.py", line 390, in call
return super().call(examples[0], **kwargs)
File "/Users/rsml/miniconda3/lib/python3.9/site-packages/transformers/pipelines/base.py", line 1111, in call
return next(
File "/Users/rsml/miniconda3/lib/python3.9/site-packages/transformers/pipelines/pt_utils.py", line 124, in next
item = next(self.iterator)
File "/Users/rsml/miniconda3/lib/python3.9/site-packages/transformers/pipelines/pt_utils.py", line 266, in next
processed = self.infer(next(self.iterator), **self.params)
File "/Users/rsml/miniconda3/lib/python3.9/site-packages/transformers/pipelines/base.py", line 1025, in forward
model_outputs = self._forward(model_inputs, **forward_params)
File "/Users/rsml/miniconda3/lib/python3.9/site-packages/transformers/pipelines/question_answering.py", line 513, in _forward
output = self.model(**model_inputs)
File "/Users/rsml/miniconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
TypeError: forward() got an unexpected keyword argument 'token_type_ids'

kbressem

medalpaca org Jun 4, 2023

Seems like the pipeline does not support the model fully. Check out the inferer on the GitHub repo.
We are slowly working on rewriting the model, so this will probably not be fixed but rather addressed in the new version.

kbressem changed discussion status to closed Jun 4, 2023

rsml

Jun 6, 2023

Hi @kbressem , is there a MedAlpaca model that is supported by the pipeline that I can use now instead of this model?

rsml changed discussion status to open Jun 6, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment