bnjmnmarie's picture
Update README.md
054ee3f verified
|
Raw History Blame Contribute Delete
1.64 kB
metadata
license: apache-2.0
base_model:
  - ornith-ai/Ornith-1.5-9B
tags:
  - autoround
  - llm-compressor

This is ornith-ai/Ornith-1.5-9B quantized with AutoRound. Model packed by llm-compressor.

Quantization:

  • INT data types
  • ~4.5 bits for linear layers
  • 8-bit LM head
  • BF16 MTP layers
  • 16-bit activations

The model is compatible with vLLM (tested: v0.26+). Made and Tested with an RTX Pro 6000 (Google Colab's G4). Currently under evaluation: Subscribe to be notified when I publish the results.

Instructions

vllm serve [this_model_id] \
  --tensor-parallel-size 1 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --trust-remote-code

Add --speculative-config '{"method":"mtp","num_speculative_tokens":X}' for MTP, where good values for X are between 2 and 6 (depending on your tasks and concurrency).