steampunque/HY-MT2-7B-MP-GGUF

Mixed Precision GGUF layer quantization of Hy-MT2-7B by tencent

Original model: https://huggingface.co/tencent/Hy-MT2-7B

The hybrid quant employs different quantization levels on a per layer basis to increased flexibility of trading off performance vs file size. Less parameter bits are used at deep layers and more bits at cortex layers to simultaneously optimize quantized size and model performance. An extended layer definition E quant Q6_E_H for the model is defined as follows:

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q6_K_888"], [1 ,"Q6_K_866"], [2 ,"Q6_K_666"], [3 ,"Q6_K_666"], [4 ,"Q5_K_666"], [5 ,"Q5_K_666"], [6 ,"Q5_K_666"], [7 ,"Q5_K_666"],
   [8 ,"Q5_K_666"], [9 ,"Q5_K_666"], [10,"Q5_K_666"], [11,"Q5_K_666"], [12,"Q5_K_666"], [13,"Q5_K_666"], [14,"Q5_K_666"], [15,"Q5_K_666"],
   [16,"Q6_K_666"], [17,"Q6_K_666"], [18,"Q6_K_666"], [19,"Q6_K_666"], [20,"Q6_K_666"], [21,"Q6_K_666"], [22,"Q6_K_666"], [23,"Q6_K_666"],
   [24,"Q6_K_866"], [25,"Q6_K_866"], [26,"Q6_K_866"], [27,"Q6_K_866"], [28,"Q6_K_888"], [29,"Q6_K_888"], [30,"Q6_K_888"], [31,"Q6_K_888"]
   ]'
   FLAGS="--token-embedding-type Q6_K --output-tensor-type Q6_K --layer-types-high"

The MP quant was optimized to ~ Q6_K bpw efficiency.

Comparison:

Quant	size	PPL	Comment
Q6_K	6.2e9	10.9	modified PPL
Q6_E_H	6.0e9	10.9	modified PPL

Usage:

Hy MT2 7B is an auto-id (it automatically detects source lang) machine translation model supporting 33 languages. See its model page for the list of languages it supports : https://huggingface.co/tencent/Hy-MT2-7B#supported-languages

Translation prompt format:

In order for the model to work correctly for translations, a prompt template can be used as follows:

User prompt: <|startoftext|>Translate the following text into {TL}. **Note that you should only output the translated result without any additional explanation.**\n\nText to be translated
Assistant prompt: <|extra_0|>

Where {TL} is from a set of language codes supported by the model, i.e {TL}=DE for tranlating english to german. The instructions for the model state that full language names instead of codes should be used, however empirical tests with the model show that it will produce identical translations when using either the full language name or the upper cased standard language code (not de, but DE). If using lower case 2 letter language codes the model will mix up german and dutch target, but using upper case 2 letter langauge codes was found to work across all tested languages.

Example using a downstream llama.cpp server with an autotokenizer implementing the above prompt template:

translate()

{
   phrase=$1
   SOURCE=$2
   TARGET=$3
   # use uppercase lang codes for Hy MT
   echo -n $(SL=${SOURCE^^} TL=${TARGET^^} lm "$phrase")
}

langs=(de es fr ru ja zh en)

phrase='The secret code to open the safe is "dead mens chest"'
SOURCE=en
echo "${SOURCE}: $phrase"
for TARGET in ${langs[@]}; do
   phrase=$(translate "$phrase" $SOURCE $TARGET)
   # Remove lead and trail \n delimiters
   phrase="${phrase#\\n}"
   phrase="${phrase%\\n}"
   echo "${TARGET}: $phrase"
   SOURCE=$TARGET
done

Running this script produces:

en: The secret code to open the safe is "dead mens chest"
de: Der geheime Code, um die Safetür zu öffnen, lautet „Dead Man’s Chest“.
es: El código secreto para abrir la puerta del seguro es “Dead Man’s Chest”.
fr: Le code secret pour ouvrir la porte du coffre-fort est « Dead Man’s Chest ».
ru: Секретный код для открытия дверцы сейфа — «Dead Man’s Chest».
ja: 金庫の扉を開けるための秘密コードは「Dead Man’s Chest」です。
zh: 打开保险箱门所需的密码是“Dead Man’s Chest”。
en: The password needed to open the safe door is “Dead Man’s Chest”.

Perplexity issue:

The model cannot compute valid perplexity on raw wikitext strings, which will result in very high values useless for comparative purposes. To work around this problem a modifed perplexity is computed by overwriting the beginning of the perplexity chunk contents with the following prefix:

  # chunk is a string of raw text to eval perplexity on
  injects='<|startoftext|><|extra_0|>'
  chunk="${injects}${chunk:${#injects}}"

logprobs are skipped over the beginning part of the perplexity prompt using a modified llama.cpp downstream server to compute the modified perplexity.

MT benchmarks for the model are given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link	Type	Size/e9 B	Notes
HY-MT2-7B.Q6_E_H.gguf	Q6_E_H	6.0e9 B	~0.2B smaller than Q6_K

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040