VultronRetrieverPrime

VultronRetrieverPrime-Qwen3.5-8B-v2

VultronRetrieverPrime v2 is the second release of the flagship of the VultronRetriever family, a late-interaction (ColBERT-style) retriever that scores document pages directly from their rendered image: layout, tables, charts and text, across six languages. v2 moves from 320 to 4096-dimensional multi-vectors and runs its full-attention layers bidirectionally.

The family has three tiers: the 8B Prime (this model) for maximum accuracy, the 4.5B Core for the accuracy/footprint mid-point, and the 0.8B Flash for latency- and footprint-sensitive serving. Trained and evaluated on Vultr Cloud.

Highlights

  • ViDoRe V3: 67.46 mean nDCG@10 over the 8 public tasks.
  • MTEB: V1 93.35, V2 72.91, V3 67.46.
  • 8.4B parameters, 4096-dim, ~17 GB bf16, runs on one GPU.
  • Smaller vectors: V3 holds at 67.17 with 1024 dims and 66.40 with 256.
  • Six languages (en, fr, de, es, it, pt).

Results

Scores are nDCG ร—100 from mteb 2.26.3, using 4096-dim embeddings and up to 1,792 visual tokens per page.

ViDoRe V3 (nDCG@10)

Industrial Finance EN Finance FR Pharma Energy Physics Computer Science HR Mean
59.65 72.62 57.04 70.96 73.03 53.54 82.84 70.00 67.46

ViDoRe V1 (nDCG@5)

ArxivQA DocVQA InfoVQA ShiftProject SynAI SynEnergy SynGov SynHealth TabFQuAD TAT-DQA Mean
93.53 70.37 95.36 95.55 99.63 97.63 99.26 100.00 97.74 84.46 93.35

ViDoRe V2 (nDCG@5)

BioMedical Lectures ESG Reports HL ESG Reports Economics Reports Mean
70.98 79.24 73.08 68.35 72.91

Per-task mteb result files are in results/. The 2 private V3 tasks are scored by the ViDoRe maintainers.

Smaller embeddings

The token vectors can be shortened: keep the first N dimensions of each vector and re-normalize it. The index shrinks in proportion. Scores are nDCG ร—100 on the full benchmarks at each size.

Dim Index size V1 V2 V3 (8 public)
4096 1ร— 93.35 72.91 67.46
2048 1/2 93.28 72.85 67.37
1024 1/4 93.29 73.06 67.17
512 1/8 93.14 73.13 66.79
256 1/16 92.73 71.80 66.40

ViDoRe V3 per task (nDCG@10):

Dim Industrial Finance EN Finance FR Pharma Energy Physics Computer Science HR Mean
4096 59.65 72.62 57.04 70.96 73.03 53.54 82.84 70.00 67.46
2048 59.68 72.47 57.07 70.86 72.98 53.41 82.64 69.90 67.37
1024 59.15 72.43 56.99 70.57 72.64 53.33 82.72 69.54 67.17
512 58.63 72.06 57.02 70.08 72.28 52.92 82.52 68.77 66.79
256 58.23 71.41 56.87 69.22 72.22 52.48 82.21 68.56 66.40

ViDoRe leaderboard

Top 10 by mean nDCG@10 over the 8 public ViDoRe V3 tasks; V1, V2 and the 10-task V3 mean are shown alongside.

Model Params Dim V1 V2 V3 (8 public) V3 (all 10)
VultronRetrieverPrime-Qwen3.5-8B-v2 (this model) 8.4B 4096 93.35 72.91 67.46 pending
athrael-soju/ColQwen3.5-4.5B-v4 4.5B 2048 93.09 69.78 67.14 pending
tencent/EVIE-8B 8.4B 4096 92.90 74.12 66.45 66.24
Microsoft/microsoft-embedding-1-large โ€” 1024 โ€” โ€” 66.24 โ€”
tencent/EVIE-4.5B 4.5B 2048 92.86 72.92 65.88 65.70
webAI-Official/webAI-ColVec1.1-8b 8.4B 640 91.53 67.08 65.32 64.95
perplexity-ai/pplx-embed-v2-late-9b 8.4B 128 90.67 68.96 64.95 64.77
VultronRetrieverPrime-Qwen3.5-8B (v1) 8.4B 320 92.08 68.18 64.72 64.26
webAI-Official/webAI-ColVec1-9b 9.4B 2560 91.30 65.82 64.45 63.00
webAI-Official/webAI-ColVec1.1-4b 4.5B 640 90.91 66.60 64.24 63.90

V1/V2: mean nDCG@5. V3 (8 public): mean nDCG@10 over the 8 public ViDoRe V3 tasks. V3 (all 10) adds the 2 private tasks (Nuclear, Telecom), scored by the ViDoRe maintainers on held-out corpora. Other models' figures: MTEB results repository, 2026-11-18; "โ€”" = not reported on that benchmark.

Per-task nDCG@10 (ViDoRe V3)

* = best on that task among the models listed. Mean is over the 8 public tasks; Nuc and Tel are the private tasks.

Model CS Energy FinEn FinFr Hr Ind Nuc Pharma Phys Tel Mean
VultronRetrieverPrime-Qwen3.5-8B-v2 (this model) 82.8 73.0* 72.6* 57.0* 70.0 59.6* pending 71.0 53.5* pending 67.46
athrael-soju/ColQwen3.5-4.5B-v4 82.9* 72.1 72.5 56.1 70.2* 59.3 pending 70.8 53.3 pending 67.14
tencent/EVIE-8B 82.1 72.2 71.2 55.8 68.9 59.1 56.6* 70.3 52.0 74.3* 66.45
Microsoft/microsoft-embedding-1-large 82.6 72.5 69.4 54.5 68.5 59.3 โ€” 71.2* 51.9 โ€” 66.24
tencent/EVIE-4.5B 82.0 72.0 69.9 54.7 67.6 58.8 56.4 70.0 52.1 73.5 65.88
webAI-Official/webAI-ColVec1.1-8b 80.1 70.1 71.9 54.9 68.5 57.6 53.7 67.9 51.5 73.3 65.32
perplexity-ai/pplx-embed-v2-late-9b 79.5 71.7 71.6 53.9 67.7 56.2 54.5 68.9 50.1 73.6 64.95
VultronRetrieverPrime-Qwen3.5-8B (v1) 79.8 70.3 69.0 54.5 66.8 57.4 53.6 68.2 51.7 71.3 64.72
webAI-Official/webAI-ColVec1-9b 80.9 69.8 68.3 53.7 70.0 57.2 47.7 67.3 48.4 66.7 64.45
webAI-Official/webAI-ColVec1.1-4b 80.3 69.5 69.2 53.1 66.9 56.4 53.3 67.2 51.2 71.8 64.24

Intended use

  • Visual document retrieval / multimodal RAG over PDFs, scans, slides and reports, including pages with layout, tables, charts and figures.
  • Multilingual document collections (en, fr, de, es, it, pt).
  • Maximum-accuracy retrieval with 4096-dim multi-vectors.

Out of scope: text-only semantic search, where a single-vector dense embedder is cheaper; generative QA (this is a retriever; pair it with a reader/LLM).

Method

Per-token MaxSim scoring captures fine-grained matches against tables, figures, and layout that a single-vector embedder averages away.

  • Base: Qwen/Qwen3.5-9B (hybrid GatedDeltaNet + full-attention backbone).
  • Late-interaction retriever (ColQwen3_5): 4096-dim multi-vector embeddings, MaxSim scoring, image + text inputs.
  • Size: 8.4B parameters, the generative head dropped for retrieval (HF reports 8B).
  • 4096-dim head: trained and operated at 4096 dimensions.
  • Bidirectional attention: the eight full-attention layers attend in both directions; the GatedDeltaNet layers stay causal.
  • Hard negatives: mined with a retriever, then filtered with a reranker that removes negatives that are actually relevant.
  • Model merging: several full-weight checkpoints merged per-block into one checkpoint.
  • Trained at up to 1280 visual tokens, evaluated and deployed at 1792.

Training data

An enhanced, multilingual mixture of public and synthetic visual-document retrieval sources, spanning en, es, de, fr, it and pt. The training recipe and the assembled training dataset are not distributed in this repository.

Inputs and outputs

  • Input: document-page images (RGB) and/or text queries; pages encode at up to 1792 visual tokens.
  • Output: multi-vector embeddings, one 4096-dim vector per token (not a single pooled vector).
  • Scoring: late-interaction MaxSim between query-token and page-token vectors.

Usage

Use transformers>=5.2.0. Older versions ignore the config's is_causal: false, run the model with causal attention and give different embeddings, so Sentence Transformers won't load the model on them.

Sentence Transformers

This model can be used with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via the MultiVectorEncoder:

pip install "sentence-transformers[image]>=6.0.0"
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("vultr/VultronRetrieverPrime-Qwen3.5-8B-v2")

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
images = [
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
]

query_embeddings = model.encode_query(queries)
image_embeddings = model.encode_document(images)

# Diagonal should have higher scores
scores = model.similarity(query_embeddings, image_embeddings)
print(scores)

config.json carries dim=4096, so the projection is sized correctly at load. It also carries text_config.is_causal=false and retrieval_attention_contract="bidirectional".

vLLM

vLLM serves this model natively through its pooling runner (the ColQwen3_5 architecture), returning the per-token multi-vectors for late-interaction scoring. It requires a vLLM build that includes the projection-bias fix and explicit retrieval-attention metadata support. The handler reads retrieval_attention_contract="bidirectional" from this checkpoint and runs all eight full-attention layers bidirectionally; it must reject missing, conflicting, or unsupported contract metadata. The server uses the stock chat/image processor, so the ColQwen3.5 prompt contract is applied client-side: wrap each page image in the instruction template, append the query-augmentation tokens to each query, and set the visual-token budget through mm-processor-kwargs. Prefix caching and chunked prefill remain off for the GatedDeltaNet hybrid.

pip install vllm
import torch
from PIL import Image
from vllm import LLM
MODEL = "vultr/VultronRetrieverPrime-Qwen3.5-8B-v2"
MAX_PIXELS = 1792 * 32 * 32   # max_num_visual_tokens * (patch_size 16 * merge_size 2)^2
llm = LLM(
    model=MODEL,
    runner="pooling",
    dtype="bfloat16",
    enable_prefix_caching=False,
    enable_chunked_prefill=False,
    mm_processor_kwargs={"min_pixels": 65536, "max_pixels": MAX_PIXELS},
)
# ColQwen3.5 processor contract, applied client-side:
IMAGE_PROMPT = ("<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>"
                "Describe the image.<|im_end|><|endoftext|>")
def query_prompt(q): return q + "<|endoftext|>" * 10   # query augmentation
images  = [Image.open("page_0.png"), Image.open("page_1.png")]
queries = ["What was Q3 revenue?", "Summarize the safety findings."]
doc_out = llm.encode([{"prompt": IMAGE_PROMPT, "multi_modal_data": {"image": im}}
                      for im in images], pooling_task="token_embed")
qry_out = llm.encode([query_prompt(q) for q in queries], pooling_task="token_embed")
def mv(o):  # one [num_tokens, 4096] multi-vector per item, L2-normalized per token
    t = torch.as_tensor(o.outputs.data, dtype=torch.float32)
    return torch.nn.functional.normalize(t, p=2, dim=-1)
docs, qrys = [mv(o) for o in doc_out], [mv(o) for o in qry_out]
# late-interaction MaxSim: per query token take the best doc token, then sum
scores = [[(q @ d.T).max(dim=-1).values.sum().item() for d in docs] for q in qrys]
print(scores)  # scores[i][j] = relevance of query i to page j

To serve over HTTP instead:

vllm serve vultr/VultronRetrieverPrime-Qwen3.5-8B-v2 \
  --runner pooling \
  --no-enable-prefix-caching --no-enable-chunked-prefill \
  --mm-processor-kwargs '{"min_pixels": 65536, "max_pixels": 1835008}'

Apply the same image template and query augmentation in your client requests. See the upstream example examples/pooling/score/colqwen3_5_rerank_online.py for the full online rerank flow.

Limitations

  • Tuned for six languages (en, fr, de, es, it, pt); other languages are out of distribution.
  • Late-interaction multi-vector indexes are larger than single-vector dense indexes: the trade for per-token layout/table/figure sensitivity. At 4096 dimensions, v2's index is about 13ร— the size of v1's 320-dim index.

License

Apache 2.0, covering the contents of this repository: model weights, config, and evaluation results. Built on Qwen/Qwen3.5-9B (Apache 2.0); the upstream license and attribution are retained. The training recipe and the assembled training dataset are not distributed in this repository.

Citation

@misc{vultronretrieverprime2026v2,
  title  = {VultronRetrieverPrime-Qwen3.5-8B-v2: Late-Interaction Visual Document Retrieval},
  author = {Georgiou, Athos (athrael-soju)},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B-v2}}
}

Trained and evaluated on Vultr Cloud.

Downloads last month
39
Safetensors
Model size
8B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for vultr/VultronRetrieverPrime-Qwen3.5-8B-v2

Finetuned
Qwen/Qwen3.5-9B
Finetuned
(1,469)
this model