|
Download README.md from kaitchup/Ornith-1.5-9B-hum-quality: direct link, hf CLI and curl.
- Browser
- Download file 1.64 kB
-
https://huggingface.co/kaitchup/Ornith-1.5-9B-hum-quality/resolve/main/README.md
- Command line
-
hf download hf://kaitchup/Ornith-1.5-9B-hum-quality/README.md
-
curl -L -o README.md https://huggingface.co/kaitchup/Ornith-1.5-9B-hum-quality/resolve/main/README.md
1.64 kB
metadata
license: apache-2.0
base_model:
- ornith-ai/Ornith-1.5-9B
tags:
- autoround
- llm-compressor
This is ornith-ai/Ornith-1.5-9B quantized with AutoRound. Model packed by llm-compressor.
Quantization:
- INT data types
- ~4.5 bits for linear layers
- 8-bit LM head
- BF16 MTP layers
- 16-bit activations
The model is compatible with vLLM (tested: v0.26+). Made and Tested with an RTX Pro 6000 (Google Colab's G4). Currently under evaluation: Subscribe to be notified when I publish the results.
- Made by: The Kaitchup
Instructions
vllm serve [this_model_id] \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--reasoning-parser qwen3 \
--trust-remote-code
Add --speculative-config '{"method":"mtp","num_speculative_tokens":X}' for MTP, where good values for X are between 2 and 6 (depending on your tasks and concurrency).