--- license: apache-2.0 base_model: - ornith-ai/Ornith-1.5-9B tags: - autoround - llm-compressor ---
This is [ornith-ai/Ornith-1.5-9B](https://huggingface.co/ornith-ai/Ornith-1.5-9B) quantized with [AutoRound](https://github.com/intel/auto-round). Model packed by [llm-compressor](https://github.com/vllm-project/llm-compressor). Quantization: * INT data types * ~4.5 bits for linear layers * 8-bit LM head * BF16 MTP layers * 16-bit activations The model is compatible with vLLM (tested: v0.26+). Made and Tested with an RTX Pro 6000 (Google Colab's G4). Currently under evaluation: [Subscribe](https://kaitchup.substack.com/subscribe) to be notified when I publish the results. - **Made by:** [The Kaitchup](https://kaitchup.substack.com/) ## Instructions ``` vllm serve [this_model_id] \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xml \ --reasoning-parser qwen3 \ --trust-remote-code ``` Add --speculative-config '{"method":"mtp","num_speculative_tokens":X}' for MTP, where good values for X are between 2 and 6 (depending on your tasks and concurrency).