Instructions to use SandLogicTechnologies/gemma-4-12B-it-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use SandLogicTechnologies/gemma-4-12B-it-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M
# Run inference directly in the terminal:
llama cli -hf SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M
# Run inference directly in the terminal:
llama cli -hf SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M
# Run inference directly in the terminal:
./llama-cli -hf SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M

Use Docker

docker model run hf.co/SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M

LM Studio
Jan

vLLM

How to use SandLogicTechnologies/gemma-4-12B-it-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SandLogicTechnologies/gemma-4-12B-it-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SandLogicTechnologies/gemma-4-12B-it-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M

Ollama
How to use SandLogicTechnologies/gemma-4-12B-it-GGUF with Ollama:
```
ollama run hf.co/SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M
```

Unsloth Studio

How to use SandLogicTechnologies/gemma-4-12B-it-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for SandLogicTechnologies/gemma-4-12B-it-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for SandLogicTechnologies/gemma-4-12B-it-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for SandLogicTechnologies/gemma-4-12B-it-GGUF to start chatting

How to use SandLogicTechnologies/gemma-4-12B-it-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use SandLogicTechnologies/gemma-4-12B-it-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use SandLogicTechnologies/gemma-4-12B-it-GGUF with Docker Model Runner:
```
docker model run hf.co/SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M
```

Lemonade

How to use SandLogicTechnologies/gemma-4-12B-it-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M

Run and chat with the model

lemonade run user.gemma-4-12B-it-GGUF-IQ3_M

List all available models

lemonade list

Hermes Agent

How to use SandLogicTechnologies/gemma-4-12B-it-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default SandLogicTechnologies/gemma-4-12B-it-GGUF:IQ3_M

Run Hermes

hermes

Atomic Chat

Gemma-4-12B-IT

Gemma-4-12B-IT is an instruction-tuned multimodal language model developed by Google, designed to support reasoning, coding, conversational AI, and vision-language understanding workflows. This repository contains GGUF quantized variants of the model optimized for efficient local inference using llama.cpp.

The model combines strong instruction-following capabilities with native multimodal understanding, enabling users to process textual and visual information within a unified architecture. The quantized formats significantly reduce memory requirements while preserving robust multimodal reasoning and generation quality, allowing practical deployment on consumer-grade hardware.

Model Overview

Model Name: Gemma-4-12B-IT
Base Model: google/gemma-4-12B-it
Architecture: Unified encoder-free multimodal model
Parameter Count: 12 Billion
Context Window: 256K Tokens
Modalities: Text, Image
Primary Languages: Multilingual
Developer: Google
License: Gemma License

Quantization Formats

This repository provides various GGUF quantized versions of the Gemma-4-12B-IT model optimized for efficient local inference using llama.cpp. Below are the details of the available quantization formats.

IQ3_M

Size reduction of approx 75.95% (5.34 GB) compared to 16-bit (22.20 GB)
Aggressive 3-bit quantization optimized for maximum memory efficiency
Suitable for low-memory deployment environments and CPU-based inference
Enables practical execution of multimodal reasoning workloads on consumer hardware
Complex analytical reasoning, long-context processing, and fine-grained multimodal understanding may experience reduced output fidelity compared to higher-precision variants

IQ4_NL

Size reduction of approx 70.54% (6.54 GB) compared to 16-bit (22.20 GB)
Advanced 4-bit non-linear quantization designed to better preserve multimodal reasoning quality and generation stability
Better suited for analytical tasks, coding workflows, structured outputs, and multimodal understanding applications
Designed to reduce quantization loss compared to more aggressive formats
May require slightly increased computational overhead during inference

IQ4_XS

Size reduction of approx 71.94% (6.23 GB) compared to 16-bit (22.20 GB)
Balanced 4-bit quantization focused on efficient inference and dependable response quality
Provides a practical balance between memory efficiency, multimodal capability, and runtime performance
Suitable for conversational AI, image-aware assistants, coding workflows, and structured instruction-following tasks
Maintains stable generation quality across most practical local deployment scenarios

Q6_K

Size reduction of approx 58.96% (9.11 GB) compared to 16-bit (22.20 GB)
Higher-precision 6-bit K-Quant format designed to preserve reasoning capability and multimodal fidelity
Better suited for demanding workloads involving long-context understanding, detailed image analysis, and complex problem solving
Provides stronger output consistency and improved retention of the original model's capabilities compared to lower-bit formats
Requires higher memory resources but offers enhanced stability and generation quality

Training Background (Original Model)

Gemma-4-12B-IT is trained with an emphasis on multimodal understanding, instruction following, reasoning, and robust generation across a diverse range of downstream tasks.

Pretraining

Large-scale multimodal pretraining across textual and visual datasets
Focus on contextual understanding, cross-modal representation learning, and multilingual capability
Optimized for downstream conversational, analytical, and multimodal reasoning workloads

Instruction Tuning

Refined using instruction-following objectives and alignment datasets
Enhanced for structured responses and complex task execution
Improved consistency for reasoning, coding, multilingual interaction, and multimodal workflows

Key Capabilities

Multimodal Understanding Processes and integrates textual, visual information for comprehensive task understanding.
Instruction Following Produces structured, context-aware responses aligned with user intent.
Reasoning and Analysis Performs effectively on analytical tasks requiring multi-step problem solving.
Coding Assistance Supports code generation, explanation, and technical reasoning workflows.
Large Context Processing Handles long documents and extended interactions through a 256K token context window.
Efficient Local Deployment Quantized variants enable practical multimodal inference on consumer hardware.

Usage Example

Using llama.cpp

./llama-mtmd-cli \
  -m SandlogicTechnologies/Gemma-4-12B-IT_IQ4_NL.gguf \
  --mmproj SandlogicTechnologies/mmproj-BF16.gguf \
  --image image.png \
  -p "Analyze the diagram and explain the underlying concept step-by-step."

Recommended Usecases

Multimodal AI Assistants Build local assistants capable of understanding text, images inputs.
Reasoning and Analytical Workflows Generate structured outputs for problem solving and multi-step tasks.
Coding and Technical Assistance Support debugging, implementation guidance, and developer-oriented interactions.
Document and Visual Understanding Analyze visual content and long-form documents using multimodal reasoning.
Research and Experimentation Evaluate multimodal prompting strategies and local inference workflows.

Acknowledgments

These quantized models are based on the original work by the Google DeepMind development team.

Special thanks to:

The Google team for developing and releasing the Gemma-4-12B-IT model.
Georgi Gerganov and the llama.cpp open-source community for enabling efficient quantization and inference via the GGUF format.

Contact

For questions, feedback, or support, please reach out at support@sandlogic.com or visit https://www.sandlogic.com/

Downloads last month: 219

GGUF

Model size

12B params

Architecture

gemma4

Hardware compatibility

3-bit

4-bit

6-bit

Model tree for SandLogicTechnologies/gemma-4-12B-it-GGUF

Base model

google/gemma-4-12B

Finetuned

google/gemma-4-12B-it

Quantized

(302)

this model