Instructions to use gatilin/qwen3-vl-2b-referring with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use gatilin/qwen3-vl-2b-referring with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="gatilin/qwen3-vl-2b-referring")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("gatilin/qwen3-vl-2b-referring")
model = AutoModelForMultimodalLM.from_pretrained("gatilin/qwen3-vl-2b-referring", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use gatilin/qwen3-vl-2b-referring with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "gatilin/qwen3-vl-2b-referring"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gatilin/qwen3-vl-2b-referring",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/gatilin/qwen3-vl-2b-referring

SGLang

How to use gatilin/qwen3-vl-2b-referring with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "gatilin/qwen3-vl-2b-referring" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gatilin/qwen3-vl-2b-referring",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "gatilin/qwen3-vl-2b-referring" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gatilin/qwen3-vl-2b-referring",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use gatilin/qwen3-vl-2b-referring with Docker Model Runner:
```
docker model run hf.co/gatilin/qwen3-vl-2b-referring
```

Qwen3-VL-2B Referring (LoRA merged)

基于 Qwen3-VL-2B-Instruct 的 referring 判断模型：给定图片、若干候选检测框和一段文本描述，判断每个框内内容是否匹配描述，输出匹配置信度 [0, 1]。典型用法是接在开放词汇检测器（如 DINOx）之后做二次判定，例如 "检测图片中是否有人" → 检测出 person 框 → 本模型逐框确认。

权重形式

本仓库为 LoRA 合并后的完整模型（merged full model），可独立加载，无需底座。 LoRA 训练配置见 lora_adapter_config.json（r=64, alpha=64, 目标 q/k/v/o_proj，iteration 271950）。原始 adapter 基座为 Qwen/Qwen3-VL-2B-Instruct。

文件说明

文件	说明
`model.safetensors`	合并后完整权重（bf16, ~4GB）
`config.json` / `generation_config.json`	模型配置（Qwen3VL 架构）
`tokenizer.json` / `tokenizer_config.json` / `chat_template.jinja`	分词器与对话模板
`processor_config.json`	图像/视频处理器配置
`lora_adapter_config.json`	训练时的 LoRA adapter 配置（留档）
`infer_referring.py`	独立推理加载代码

快速使用

from infer_referring import ReferringPredictor

predictor = ReferringPredictor(model_dir=".")
confs = predictor.inference(
    "demo.jpg",
    bboxes=[[119, 136, 475, 1081], [10, 323, 217, 1025]],
    prompt="检测图片中是否有人",
)
# [0.9821, 0.9517]  每个框的匹配置信度，>0.5 视为匹配

输入格式（推理协议）

模型采用 全局上下文 + 框裁剪 双图输入：全局图缩放到 512 长边，候选框按比例裁剪后送入；输入序列为 [vision_start] global_patch [image_pad...] crop_patch [vision_end] prompt文本，仅对序列末位 logits 的前两个 token 做 softmax，取 index 1 作为匹配置信度。文本支持中文及任意语种。

训练信息

任务: referring 二分类（框内内容与文本描述是否匹配）
微调方式: LoRA（r=64, dropout=0, 目标模块 q_proj/k_proj/v_proj/o_proj）
训练步数: iteration 271950
基座: Qwen3-VL-2B-Instruct (bf16)

Downloads last month: 6

Safetensors

Model size

2B params

Tensor type

BF16

Model tree for gatilin/qwen3-vl-2b-referring

Base model

Qwen/Qwen3-VL-2B-Instruct

Adapter

(122)

this model

Collection including gatilin/qwen3-vl-2b-referring

Detection, Grounding & Segmentation

Collection

Detection, referring-expression grounding, and segmentation models, with weight packages distinguished from deployment-code repositories. • 6 items • Updated 8 days ago