Instructions to use Thox-ai/ThoxMini-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Thox-ai/ThoxMini-3B with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("unsloth/Llama-3.2-3B-Instruct")
model = PeftModel.from_pretrained(base_model, "Thox-ai/ThoxMini-3B")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Thox-ai/ThoxMini-3B with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Thox-ai/ThoxMini-3B:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Thox-ai/ThoxMini-3B:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Thox-ai/ThoxMini-3B:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Thox-ai/ThoxMini-3B:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Thox-ai/ThoxMini-3B:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Thox-ai/ThoxMini-3B:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Thox-ai/ThoxMini-3B:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Thox-ai/ThoxMini-3B:Q4_K_M

Use Docker

docker model run hf.co/Thox-ai/ThoxMini-3B:Q4_K_M

LM Studio
Jan

vLLM

How to use Thox-ai/ThoxMini-3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Thox-ai/ThoxMini-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Thox-ai/ThoxMini-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Thox-ai/ThoxMini-3B:Q4_K_M

Ollama
How to use Thox-ai/ThoxMini-3B with Ollama:
```
ollama run hf.co/Thox-ai/ThoxMini-3B:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use Thox-ai/ThoxMini-3B with Docker Model Runner:
```
docker model run hf.co/Thox-ai/ThoxMini-3B:Q4_K_M
```

Lemonade

How to use Thox-ai/ThoxMini-3B with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Thox-ai/ThoxMini-3B:Q4_K_M

Run and chat with the model

lemonade run user.ThoxMini-3B-Q4_K_M

List all available models

lemonade list

Atomic Chat

You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Access to this THOX model is granted manually by THOX.ai LLC.

ThoxMini-3B

Capability status (2026-09-29): Public weights. Device mapping is recorded in ttracx/thox-device-models and certified only as public + mapped + no blocked marketing claims. This is not a tok/s, AIME, or HIPAA certificate. ThoxNova names on this org are a model family, not a campaign SKU.

Your AI. Your Data. Your Rules.

THOX 'Mini' — Llama-3.2-3B-Instruct SFT LoRA with a bundled Q4_K_M GGUF for on-device use.

What this is

Base: meta-llama/Llama-3.2-3B-Instruct (Llama 3.2 Community License). Trained on the unsloth 4-bit re-upload.
LoRA r=16, α=32; 1,565-row SFT; ctx 8192; bundled thoxmini-3b-Q4_K_M.gguf.
Gated (Thox.ai LLC manual approval).
Naming note: Meta's Llama license expects derivative names to carry 'Llama' — flagged in thox-ollama-models.

Intended use

On-device / edge text generation within the THOX stack. Not a safety-aligned public assistant unless deployed behind THOX guardrails.

Prompt / chat template

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{system}<|eot_id|><|start_header_id|>user<|end_header_id|>

{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Usage

Transformers + PEFT

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "meta-llama/Llama-3.2-3B-Instruct"
adapter = "Thox-ai/ThoxMini-3B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, device_map="auto")
model = PeftModel.from_pretrained(model, adapter)

Bundled GGUF (llama.cpp / Ollama)

hf download Thox-ai/ThoxMini-3B --include '*.gguf' --local-dir ./ThoxMini-3B

Files

File	Size	SHA-256
`adapter_model.safetensors`	97.31 MB (97,307,544 B)	`df729d5105f0811ab03be2e6693e640344c77845c45005e06d4a7244cac2a83d`
`thoxmini-3b-Q4_K_M.gguf`	2.02 GB (2,019,373,952 B)	`3bcfb6e420667731de4f31845fde0e78557e28db8e4ed7ea897832de30776dfa`

Use with ThoxOS

Download the file once and run it fully offline on a ThoxOS host (or any Linux/macOS/Windows machine):

pip install -U "huggingface_hub[cli]"
hf download Thox-ai/ThoxMini-3B thoxmini-3b-Q4_K_M.gguf --local-dir ./ThoxMini-3B
llama-cli -m ./ThoxMini-3B/thoxmini-3b-Q4_K_M.gguf -p "Hello" -n 128

Ollama (pulls straight from the Hub):

ollama run hf.co/Thox-ai/ThoxMini-3B:Q4_K_M

This repo is gated: accept the terms on the Hub first, then run hf auth login (and link your Hugging Face account in Ollama for hf.co/ pulls).

A same-named entry also exists in the Ollama library: ollama.com/Thox-ai/thoxmini-3b (contents not re-verified in this pass; prefer the hf.co/ pull above for a pinned file).

LM Studio: search Thox-ai/ThoxMini-3B in the model browser (or lms get Thox-ai/ThoxMini-3B), pick thoxmini-3b-Q4_K_M.gguf, and load it.

Use with MeshStack

Serve the model as an OpenAI-compatible endpoint on any MeshStack node, then register that endpoint with your MeshStack coordinator (protocol openai, base URL http://<node-ip>:8080/v1; or protocol ollama at http://<node-ip>:11434 if you serve it with Ollama).

llama-server -m ./ThoxMini-3B/thoxmini-3b-Q4_K_M.gguf --host 0.0.0.0 --port 8080 -c 4096
curl http://<node-ip>:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Hello"}]}'

Device fit (by file size only)

thoxmini-3b-Q4_K_M.gguf (2.02 GB): Medium (<2.5 GB): devices with ≥4 GB RAM — recent phones, laptops, Apple Silicon Macs, MeshStack edge hosts.

RAM needed ≈ file size + context (KV cache) + runtime overhead. These tiers are derived from file size alone; no throughput, TOPS or battery figures are claimed here.

Licence and attribution

Llama 3.2 is licensed under the Llama 3.2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. Built with Llama. Use must comply with the Acceptable Use Policy. Open item (legal): the licence expects derivative model names to begin with "Llama".

Out of scope

Medical, legal, financial, safety-critical or other high-stakes decisions without human review.
Unattended control of physical devices or actuators without a confirmation step.
Any use prohibited by the upstream licence or acceptable-use policy.

Limitations and bias

Small and quantized models make factual mistakes and can produce confident but wrong output; verify important answers.
Behaviour and bias reflect the training data, which is mostly English web/instruction text; quality in other languages is lower unless stated above.
Quantization trades accuracy for size; lower-bit files degrade more.

THOX.ai LLC — Your AI. Your Data. Your Rules.™ · thox.ai · Contact: tommy@thox.ai

Downloads last month: 19

GGUF

Model size

3B params

Architecture

llama

Hardware compatibility

4-bit

Model tree for Thox-ai/ThoxMini-3B

Base model

meta-llama/Llama-3.2-3B-Instruct

Adapter

(872)

this model

Thox-ai
/

ThoxMini-3B