InferForge — Qwen3.5-4B-Q4_K_M

A Q4_K_M GGUF quantization of Qwen3.5-4B, prepared by InferForge for efficient local inference with llama.cpp.

This build focuses on practical local inference on consumer hardware, with an emphasis on:

• Coding
• Technical analysis
• DevOps and infrastructure
• Kubernetes troubleshooting
• Distributed systems
• Agent and worker workloads
• Long-context inference
• Efficient GPU/CPU inference

Model Information

Property	Value
Base model	Qwen/Qwen3.5-4B
Parameters	~4B
Format	GGUF
Quantization	Q4_K_M
Inference engine	llama.cpp
Tested context	131,072 tokens (128K)
Primary target	Local inference
Maintainer	InferForge

Quantization

The original Qwen3.5-4B model was converted to GGUF and quantized locally using llama.cpp.

The resulting file is:

Qwen3.5-4B-Q4_K_M.gguf

The Q4_K_M format was selected as a practical balance between:

• Model quality
• Memory usage
• Generation speed
• Consumer GPU compatibility

Usage with llama.cpp

Interactive CLI

Run the model directly with llama-cli:

llama-cli \
  -m Qwen3.5-4B-Q4_K_M.gguf \
  -ngl 99 \
  -c 131072 \
  -n 1024

Docker

docker run --rm -it \
  --gpus all \
  --entrypoint /app/llama-cli \
  -v "$PWD:/models" \
  ghcr.io/ggml-org/llama.cpp:full-cuda \
  -m /models/Qwen3.5-4B-Q4_K_M.gguf \
  -ngl 99 \
  -c 131072 \
  -n 1024

Usage with llama-server

The model can also be exposed through an OpenAI-compatible API.

Native

llama-server \
  -m Qwen3.5-4B-Q4_K_M.gguf \
  -ngl 99 \
  -c 131072 \
  -n 1024 \
  --host 0.0.0.0 \
  --port 8080

Docker

docker run --rm -d \
  --name qwen35-4b \
  --gpus all \
  --entrypoint /app/llama-server \
  -p 30000:8080 \
  -v "$PWD:/models" \
  ghcr.io/ggml-org/llama.cpp:full-cuda \
  -m /models/Qwen3.5-4B-Q4_K_M.gguf \
  -ngl 99 \
  -c 131072 \
  -n 1024 \
  --host 0.0.0.0 \
  --port 8080

The OpenAI-compatible API is available at:

http://localhost:30000/v1

Example API Request

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.5-4B-Q4_K_M.gguf",
    "messages": [
      {
        "role": "user",
        "content": "Explain how Kubernetes Pods differ from Docker containers."
      }
    ],
    "temperature": 0.2,
    "max_tokens": 512
  }'

Context Length

This build has been tested with:

131072 tokens

or approximately:

128K context

Large context windows substantially increase memory requirements.

On GPUs with limited VRAM, llama.cpp may offload part of the workload to system RAM.

Users should benchmark different context sizes on their own hardware to determine the best performance/memory trade-off.

Limitations

• Performance varies across hardware and inference backends.
• Large context sizes require significantly more memory.
• Q4_K_M trades some numerical precision for lower memory usage and higher inference speed.
• Quantization may introduce quality differences compared with higher-precision versions.
• This repository contains a quantized derivative of the original model and is not a new base model.

File Integrity

SHA256:

025d2cb46dc4cb13b54b8d5c44c3cb7c46fc2cc6a0728b69762c9465ca4c03d2

Generate the checksum with:

sha256sum Qwen3.5-4B-Q4_K_M.gguf

Credits

This repository contains a GGUF quantization derived from:

Qwen3.5-4B

Please refer to the original Qwen model repository for the original model, architecture, training information, and licensing terms.

This is an independent InferForge quantization project and is not affiliated with or endorsed by Qwen.

About InferForge

InferForge is an independent model experimentation and inference project focused on practical and reproducible local AI infrastructure.

Areas of interest include:

• GGUF quantization
• Local LLM inference
• Consumer GPU optimization
• Inference benchmarking
• Long-context experimentation
• Efficient model serving
• Agent-oriented workloads
• Reproducible AI infrastructure

Additional models, quantizations, and benchmark results may be added to the InferForge collection over time.