Instructions to use SandLogicTechnologies/MediPhi-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use SandLogicTechnologies/MediPhi-Instruct-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="SandLogicTechnologies/MediPhi-Instruct-GGUF",
	filename="MediPhi-Instruct_IQ3_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use SandLogicTechnologies/MediPhi-Instruct-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M
# Run inference directly in the terminal:
llama-cli -hf SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M
# Run inference directly in the terminal:
llama-cli -hf SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M
# Run inference directly in the terminal:
./llama-cli -hf SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M

Use Docker

docker model run hf.co/SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M

LM Studio
Jan

vLLM

How to use SandLogicTechnologies/MediPhi-Instruct-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SandLogicTechnologies/MediPhi-Instruct-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SandLogicTechnologies/MediPhi-Instruct-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M

Ollama
How to use SandLogicTechnologies/MediPhi-Instruct-GGUF with Ollama:
```
ollama run hf.co/SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M
```

Unsloth Studio new

How to use SandLogicTechnologies/MediPhi-Instruct-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for SandLogicTechnologies/MediPhi-Instruct-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for SandLogicTechnologies/MediPhi-Instruct-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for SandLogicTechnologies/MediPhi-Instruct-GGUF to start chatting

Docker Model Runner
How to use SandLogicTechnologies/MediPhi-Instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M
```

Lemonade

How to use SandLogicTechnologies/MediPhi-Instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull SandLogicTechnologies/MediPhi-Instruct-GGUF:IQ3_M

Run and chat with the model

lemonade run user.MediPhi-Instruct-GGUF-IQ3_M

List all available models

lemonade list

MediPhi-Instruct

MediPhi-Instruct is a healthcare-focused instruction-tuned language model developed for medical reasoning, clinical knowledge understanding, and healthcare-oriented conversational tasks. The model is designed to assist with biomedical and clinical text processing while maintaining efficient inference capabilities for practical deployment scenarios.

It is optimized for instruction following, medical Q&A, summarization, and healthcare reasoning workflows, making it suitable for research, educational, and domain-specific assistant applications.

MediPhi-Instruct is particularly effective in scenarios involving clinical discussions, medical documentation, biomedical reasoning, and healthcare-related conversational systems.

Model Overview

Model Name: MediPhi-Instruct
Base Model: microsoft/MediPhi-Instruct
Architecture: Decoder-only Transformer
Parameter Count: 3.8B parameters
Modalities: Text
Primary Languages: English
Developer: Microsoft
License: MIT

Quantization Details

This repository provides various GGUF quantized versions of the MediPhi-Instruct model, optimized for efficient local inference using llama.cpp. Below are the details of the available I-Matrix (IQ) formats.

Quantization Formats (I-Quants)

IQ3_M

Size reduction of approx 75.70% (1.73 GB) compared to 16-bit (7.12 GB)
Aggressive 3-bit quantization optimized for maximum memory reduction and lightweight deployment
Suitable for low-memory systems and CPU-based inference environments
Enables practical deployment of larger healthcare-oriented models on constrained hardware
May reduce precision on complex clinical reasoning and detailed biomedical tasks

IQ4_XS

Size reduction of approx 72.89% (1.93 GB) compared to 16-bit (7.12 GB)
Balanced 4-bit quantization designed for efficient inference and stable generation quality
Provides a strong trade-off between memory usage, speed, and response consistency
Suitable for general-purpose healthcare assistants, summarization, and medical conversational workflows
Maintains reliable inference performance across most practical medical NLP tasks

IQ4_NL

Size reduction of approx 71.35% (2.04 GB) compared to 16-bit (7.12 GB)
Advanced 4-bit non-linear quantization focused on preserving output quality and reasoning capability
Better suited for structured medical reasoning, biomedical analysis, and detailed explanatory tasks
Typically provides improved consistency for healthcare-oriented workflows
Slightly higher computational overhead during inference compared to simpler formats

Training Overview

Pretraining

The model is trained on diverse biomedical and clinical text corpora to improve understanding of healthcare terminology, medical concepts, and domain-specific reasoning.

Training objectives include:

Large-scale language modeling
Biomedical knowledge representation
Clinical reasoning enhancement
Context-aware healthcare text understanding

Alignment and Optimization

Post-training refinement focuses on instruction-following and healthcare-oriented conversational reliability.

Optimization objectives include:

Instruction tuning for medical conversational tasks
Improved response consistency for healthcare workflows
Enhanced biomedical reasoning capability
Better contextual understanding of clinical language

Core Capabilities

Medical Question Answering Handles healthcare-related prompts and biomedical information requests.
Clinical Reasoning Supports structured reasoning across medical and healthcare-oriented discussions.
Healthcare Text Understanding Processes clinical documentation and biomedical terminology effectively.
Instruction Following Generates structured and context-aware responses for domain-specific tasks.
Efficient Local Inference Quantized variants enable practical deployment without requiring cloud infrastructure.

Example Usage

llama.cpp

./llama-cli \
  -m SandlogicTechnologies/MediPhi-Instruct_IQ4_NL.gguf \
  -p "Explain the clinical symptoms and treatment considerations for hypertension."

Recommended Use Cases

Medical and healthcare conversational assistants
Biomedical research and educational workflows
Clinical document summarization
Healthcare-oriented question answering systems
Medical terminology understanding and explanation
Research and experimentation in clinical AI applications

Acknowledgments

These quantized models are based on the original work by the Microsoft development team.

Special thanks to:

The Microsoft team for developing and releasing the MediPhi-Instruct model.
Georgi Gerganov and the llama.cpp open-source community for enabling efficient quantization and inference via the GGUF format.

Contact

For any inquiries or support, please contact us at support@sandlogic.com or visit our Website.

Downloads last month: 208

GGUF

Model size

4B params

Architecture

phi3

Hardware compatibility

3-bit

4-bit

Model tree for SandLogicTechnologies/MediPhi-Instruct-GGUF

Base model

microsoft/Phi-3.5-mini-instruct

Finetuned

microsoft/MediPhi-Instruct

Quantized

(12)

this model