License: free for commercial and non-commercial use, with attribution. If you use this model, its outputs, or anything derived from it in a product, service, paper or dataset, you must credit "AletheiaAING" (https://huggingface.co/AletheiaAING) and keep this notice. This model is derived from Google Gemma and is also subject to the Gemma Terms of Use.
gemma-3-1b-csw-experiments (RL1, DPO, RL2)
Three training experiments of the Gemma-3-1B csw model in one repository: RL1 (rules only), DPO, and RL2 (rules + learned evaluator + AfriCOMET). They are research artifacts; for use, take gemma-3-1b-csw-best or the smaller gemma-3-0.8b-csw.
What the model does
It rewrites text in 37 African languages in two ways: it can code-switch (mix a second language into the base language the way native speakers do) and it can rewrite idiomatically (turn plain or literal text into lively, idiom-rich chat style).
| Task | What you give it | What you get | Prompt (Yoruba as an example) |
|---|---|---|---|
eng_hm |
English text | the same meaning in the base language with phrases of the switch language mixed in | <hm> <yor>+<eng>: I will explain the method we use. |
m_hm |
text in the base language | the same text, code-switched | <hm> <yor>+<eng>: รmi yรณรฒ แนฃร lร yรฉ แปฬnร tรญ a ล gbร . |
eng_ood |
English text | a lively, idiom-rich rewrite in the base language | <ood> <yor>: I will explain the method we use. |
m_ood |
text in the base language | an idiom-rich rewrite in the same language | <ood> <yor>: รmi yรณรฒ แนฃร lร yรฉ แปฬnร tรญ a ล gbร . |
hm_ood |
an already code-switched text | an idiom-rich rewrite that keeps the switching | <ood> <yor>+<eng>: รmi yรณรฒ explain แปฬnร tรญ a ล gbร . |
What do m_text, hm and ood mean? (in simple words)
m_text: a plain translation into the language you want, no mixing. This family of models does not dom_text(it was trained without translation tasks); for plain translation use a translation model. The 270M csw models and the datasetcsw-synth-samples8do havem_text.hm(human-style mixing): the text mixes two languages the way real people talk, for example Yoruba with English words in the middle of a sentence ("Ma explain fรบn แบน bรญ a แนฃe ล generate funds"). You choose the base language and the language to mix in.ood(idiom-rich style): the text is rewritten in a lively, everyday way, full of sayings, proverbs, comparisons and exclamations, the way a chatty native speaker talks, instead of plain textbook language.
In the task names, eng_ means the input is English, m_ means the input is already in the base language, and hm_ood means the input is already mixed (hm) and you want the idiom-rich rewrite of it.
Quick recipes (with the helper code below; ask calls the model):
# hm: English -> base language with a second language mixed in (switch language must be one of the listed pairs)
ask(build_prompt("eng_hm", "yor", "I will explain the method we use.", "eng"))
# hm: text already in the base language -> mixed
ask(build_prompt("m_hm", "yor", "รmi yรณรฒ แนฃร lร yรฉ แปฬnร tรญ a ล gbร .", "eng"))
# ood: English -> idiom-rich base language
ask(build_prompt("eng_ood", "yor", "I will explain the method we use."))
# ood: base-language text -> idiom-rich
ask(build_prompt("m_ood", "yor", "รmi yรณรฒ แนฃร lร yรฉ แปฬnร tรญ a ล gbร ."))
# hm -> ood: mixed text -> idiom-rich rewrite that keeps the mixing
ask(build_prompt("hm_ood", "yor", "รmi yรณรฒ explain แปฬnร tรญ a ล gbร .", "eng"))
Input format (read this before writing prompts).
- Send one user message and nothing else (no system prompt). The message is
<task-tag> <base>[+<switch>]: TEXT. <hm>asks for code-switching,<ood>for the idiomatic rewrite.- Languages are written as their three-letter code inside angle brackets:
<yor>, notYorubaand notyorwithout brackets. The codes are in the language table below. How the tags are tokenized: the nine tags<hm>,<ood>,<dev>,</dev>,<eng>,<fra>,<ara>,<sheng>and<gpe>are single tokens; the other language tags (<yor>,<hau>,<pcm>, ...) are ordinary text that the tokenizer splits into several pieces (for example<,yor,>). The model was trained exactly this way, so pass every tag as plain text as shown, with the tokenizer of this repository, and do not replace tags by token ids. - There are two spaces after the colon. Do not change the spacing or the tags.
+<switch>is required for<hm>prompts and forhm_ood; use only the switch languages listed for that base language (other pairs were not trained and come out unnatural). For<ood>prompts on plain base-language or English text, leave the switch language out.- The model always answers in the format below; if the structure is missing (no
<dev>block) treat the output as invalid and resample.
Output format. <base>[<switch>]<dev>device one | device two | ...</dev> text
The tags name the base language (and the switch language for code-switching). The <dev> block lists the devices: the switched phrases (code-switching) or the idioms and expressions used (idiomatic rewrite), copied exactly as they appear in the text. The text follows. Example:
<yor><eng><dev>explain | generate funds | members</dev> Ma explain fรบn แบน bรญ a แนฃe ล generate funds fรบn ร wแปn members wa. ...
The helper parse() below returns {"ok", "tags", "devices", "text"}; use text for display and ok to filter.
Examples (greedy output of this model family; prompts shortened with ...; unreviewed):
| Prompt | Output |
|---|---|
<hm> <pcm>+<hau>: You are informing me that your maize farm is unsuccessful and the crops are not growing. ... |
<pcm><hau><dev>wallahi | kai | gaskiya | insha Allah</dev> You dey tell me say your maize farm no work and the crops no dey grow, wallahi. You talk say you don dey pray and wait every day, but you no do any manual labor, kai! ... |
<hm> <yor>+<eng>: I will explain the method we use to generate funds for our members. ... |
<yor><eng><dev>explain | generate funds | members | contribute small amount daily | ...</dev> Ma explain fรบn แบน bรญ a แนฃe ล generate funds fรบn ร wแปn members wa. แบธnรฌ kแปฬแปkan ล contribute small amount ... |
<ood> <pcm>: Some young individuals are easily deceived into giving their remaining funds to fraudulent religious leaders ... |
<pcm><dev>na wash dem dey wash dem | prayer fit cook rice | money no dey fall from sky like rain</dev> Some young people na wash dem dey wash dem, dem just dey throw their last kobo give fake prophet because dem think say miracle go happen. ... |
Typical failure modes are visible here too: some devices are awkward or invented expressions (the Pidgin idiom example), and Pidgin outputs keep a lot of English.
Supported languages and switch languages
37 base languages. Write the language code from the second column in the prompt; the switch languages are the ones this model was trained with for that base language (the table is the one used by the whole csw 1B pipeline; it removed unrealistic partnerships). Languages marked as base languages can also appear as switch languages of other bases (for example eng, fra, ara, swh).
| Language | Code used in prompts | Switch languages (trained pairs) | Example prompt (code-switching) |
|---|---|---|---|
| Afrikaans | afr |
English eng |
<hm> <afr>+<eng>: TEXT |
| Amharic | amh |
English eng |
<hm> <amh>+<eng>: TEXT |
| Bambara | bam |
French fra |
<hm> <bam>+<fra>: TEXT |
| Cameroonian Pidgin | wes |
French fra |
<hm> <wes>+<fra>: TEXT |
| Chichewa | nya |
English eng, Tumbuka tum |
<hm> <nya>+<eng>: TEXT |
| Ewe | ewe |
French fra, English eng |
<hm> <ewe>+<fra>: TEXT |
| Fon | fon |
French fra, Yoruba yor |
<hm> <fon>+<fra>: TEXT |
| Fulah | ful |
French fra, English eng |
<hm> <ful>+<fra>: TEXT |
| Ghanaian Pidgin | gpe |
Twi twi, Ewe ewe, Ga gaa |
<hm> <gpe>+<twi>: TEXT |
| Hausa | hau |
English eng, Arabic ara |
<hm> <hau>+<eng>: TEXT |
| Igbo | ibo |
English eng, Nigerian Pidgin pcm |
<hm> <ibo>+<eng>: TEXT |
| Kikuyu | kik |
English eng, Swahili swh |
<hm> <kik>+<eng>: TEXT |
| Kinyarwanda | kin |
French fra, English eng, Swahili swh |
<hm> <kin>+<fra>: TEXT |
| Kirundi | run |
French fra, Swahili swh |
<hm> <run>+<fra>: TEXT |
| Kongo | kon |
French fra, Lingala lin |
<hm> <kon>+<fra>: TEXT |
| Lingala | lin |
French fra |
<hm> <lin>+<fra>: TEXT |
| Luganda | lug |
English eng, Swahili swh |
<hm> <lug>+<eng>: TEXT |
| Luo | luo |
English eng, Swahili swh |
<hm> <luo>+<eng>: TEXT |
| Malagasy | mlg |
French fra |
<hm> <mlg>+<fra>: TEXT |
| Nigerian Pidgin | pcm |
Yoruba yor, Igbo ibo, Hausa hau |
<hm> <pcm>+<yor>: TEXT |
| North Ndebele | nde |
English eng, Shona sna |
<hm> <nde>+<eng>: TEXT |
| Oromo | orm |
Amharic amh, English eng |
<hm> <orm>+<amh>: TEXT |
| Sepedi | nso |
English eng |
<hm> <nso>+<eng>: TEXT |
| Sesotho | sot |
English eng, Afrikaans afr |
<hm> <sot>+<eng>: TEXT |
| Setswana | tsn |
English eng, Afrikaans afr |
<hm> <tsn>+<eng>: TEXT |
| Shona | sna |
English eng |
<hm> <sna>+<eng>: TEXT |
| Somali | som |
Arabic ara, English eng |
<hm> <som>+<ara>: TEXT |
| Swahili | swh |
English eng, Arabic ara, Sheng sheng |
<hm> <swh>+<eng>: TEXT |
| Swati | ssw |
English eng, Zulu zul |
<hm> <ssw>+<eng>: TEXT |
| Tigrinya | tir |
Amharic amh, English eng |
<hm> <tir>+<amh>: TEXT |
| Tsonga | tso |
English eng, Zulu zul |
<hm> <tso>+<eng>: TEXT |
| Tumbuka | tum |
English eng, Chichewa nya |
<hm> <tum>+<eng>: TEXT |
| Twi | twi |
English eng, Ghanaian Pidgin gpe |
<hm> <twi>+<eng>: TEXT |
| Wolof | wol |
French fra |
<hm> <wol>+<fra>: TEXT |
| Xhosa | xho |
English eng |
<hm> <xho>+<eng>: TEXT |
| Yoruba | yor |
English eng, Nigerian Pidgin pcm |
<hm> <yor>+<eng>: TEXT |
| Zulu | zul |
English eng |
<hm> <zul>+<eng>: TEXT |
How to use it
The repository holds three complete models in subfolders. Each folder has the same files as a normal model (model.safetensors, tokenizer, config.json).
| Subfolder | Experiment | Full-test chrF++ mean |
|---|---|---|
rl1_rules_only/ |
RL1: GRPO from the SFT model with rules only (no learned evaluator) | 47.38 |
dpo/ |
DPO on 7,587 Gemini-Flash-corrected preference pairs, started from RL1 | 47.53 |
rl2_evaluator_africomet/ |
RL2: rules + learned evaluator + AfriCOMET-QE (the checkpoint with the best monitor reward, which is the final policy at step 400; its held-out chrF++ was 51.38 against 51.84 at the start, and it peaked at 52.1 around step 50-100) | 47.57 |
For comparison the SFT start scores 48.05 and the released best model (RL3) 47.83. None of these three beats the SFT model on the mean; RL2's evaluator reward rose while held-out quality fell (the policy drifted toward copying its input), which is why RL3 added rules against that. They are kept as research artifacts.
# pip install vllm transformers huggingface_hub (tested with vLLM 0.32, bf16 / FP8 on one GPU)
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
from huggingface_hub import snapshot_download
from csw_helpers import build_prompt, parse # helper file below (also in this repository)
repo = "AletheiaAING/gemma-3-1b-csw-experiments"
sub = "rl2_evaluator_africomet" # or "rl1_rules_only", "dpo"
path = snapshot_download(repo, allow_patterns=[f"{sub}/*"]) + "/" + sub
tok = AutoTokenizer.from_pretrained(path)
stop = [tok.convert_tokens_to_ids("<end_of_turn>"), tok.eos_token_id]
llm = LLM(model=path, dtype="bfloat16", max_model_len=2048)
def generate(prompts, temperature=0.0, n=1):
ids = [{"prompt_token_ids": tok(tok.apply_chat_template([{"role": "user", "content": p}], tokenize=False, add_generation_prompt=True), add_special_tokens=False)["input_ids"]} for p in prompts]
sp = SamplingParams(temperature=temperature, n=n, max_tokens=768, stop_token_ids=stop, skip_special_tokens=False)
return [[o.text.replace("<end_of_turn>", "").strip() for o in g.outputs] for g in llm.generate(ids, sp)]
prompt = build_prompt("eng_hm", "yor", "I will explain the method we use to generate funds for our members.", "eng") # build_prompt / parse: see below
print(parse(generate([prompt])[0][0]))
I have not tested loading with plain Transformers AutoModelForCausalLM; the vLLM path above is the one used for every number in this card.
Helper code (prompt builder and parser)
Save as csw_helpers.py. build_prompt checks the language pair against the table above, parse splits an answer into devices and text, and ask calls any OpenAI-compatible server (llama.cpp llama-server, vLLM vllm serve). Tested end to end with the Q8_0 GGUF on llama.cpp.
"""csw helpers: build a prompt, call a model served by llama.cpp / vLLM (OpenAI-compatible), parse the answer."""
import json, re, urllib.request
# base language -> switch languages the model was trained with (use only these pairs)
ALLOWED_SWITCH = {
"afr": ["eng"],
"amh": ["eng"],
"bam": ["fra"],
"ewe": ["fra", "eng"],
"fon": ["fra", "yor"],
"ful": ["fra", "eng"],
"gpe": ["twi", "ewe", "gaa"],
"hau": ["eng", "ara"],
"ibo": ["eng", "pcm"],
"kik": ["eng", "swh"],
"kin": ["fra", "eng", "swh"],
"kon": ["fra", "lin"],
"lin": ["fra"],
"lug": ["eng", "swh"],
"luo": ["eng", "swh"],
"mlg": ["fra"],
"nde": ["eng", "sna"],
"nso": ["eng"],
"nya": ["eng", "tum"],
"orm": ["amh", "eng"],
"pcm": ["yor", "ibo", "hau"],
"run": ["fra", "swh"],
"sna": ["eng"],
"som": ["ara", "eng"],
"sot": ["eng", "afr"],
"ssw": ["eng", "zul"],
"swh": ["eng", "ara", "sheng"],
"tir": ["amh", "eng"],
"tsn": ["eng", "afr"],
"tso": ["eng", "zul"],
"tum": ["eng", "nya"],
"twi": ["eng", "gpe"],
"wes": ["fra"],
"wol": ["fra"],
"xho": ["eng"],
"yor": ["eng", "pcm"],
"zul": ["eng"],
}
TASKS = ("eng_hm", "m_hm", "eng_ood", "m_ood", "hm_ood")
def build_prompt(task, base, text, switch=None):
"""text = English (eng_*), base-language text (m_*) or code-switched text (hm_ood).
The language is written as its three-letter code in angle brackets; there are TWO spaces after the colon."""
if base not in ALLOWED_SWITCH: raise ValueError(f"unsupported base language {base!r}")
if task in ("eng_hm", "m_hm", "hm_ood"):
switch = switch or ALLOWED_SWITCH[base][0]
if switch not in ALLOWED_SWITCH[base]: raise ValueError(f"{switch!r} is not a trained switch language of {base!r}: {ALLOWED_SWITCH[base]}")
if task in ("eng_hm", "m_hm"): return f"<hm> <{base}>+<{switch}>: {text}"
if task in ("eng_ood", "m_ood"): return f"<ood> <{base}>: {text}"
if task == "hm_ood": return f"<ood> <{base}>+<{switch}>: {text}"
raise ValueError(task)
_TAGS = re.compile(r"^((?:<(?!/?dev>)[a-z]{3,5}>)+)"); _DEV = re.compile(r"<dev>(.*?)</dev>", re.S)
def parse(completion):
"""'<yor><eng><dev>phrase one | phrase two</dev> text' -> {'ok', 'tags', 'devices', 'text'}"""
c = completion.replace("<end_of_turn>", "").strip(); t = _TAGS.match(c); rest = c[t.end():].lstrip() if t else c; d = _DEV.match(rest)
text = rest[d.end():].strip() if d else rest
return {"ok": bool(t and d and text), "tags": re.findall(r"<([a-z]{3,5})>", t.group(1)) if t else [], "devices": [x.strip() for x in d.group(1).split("|") if x.strip()] if d else [], "text": text}
def ask(prompt, url="http://127.0.0.1:8080/v1/chat/completions", temperature=0.0, max_tokens=600):
body = {"messages": [{"role": "user", "content": prompt}], "temperature": temperature, "max_tokens": max_tokens}
req = urllib.request.Request(url, json.dumps(body).encode(), {"Content-Type": "application/json"})
return json.loads(urllib.request.urlopen(req, timeout=600).read())["choices"][0]["message"]["content"]
if __name__ == "__main__":
out = parse(ask(build_prompt("eng_hm", "yor", "I will explain the method we use to generate funds for our members.", "eng")))
print(out["ok"], out["tags"], out["devices"]); print(out["text"])
Decoding: greedy or MBR-16
Greedy decoding is the default and what the numbers above use. Sampling 16 outputs (temperature 0.8) and keeping the one that agrees most with the others (minimum Bayes risk, chrF++ as the agreement measure) gave +1.17 chrF++ on the full test set (47.83 to 49.00 for RL3), better on all five tasks, at 16 times the generation cost. Choosing candidates with our learned quality evaluator did not help (46.62; MBR plus evaluator 48.46).
import re
from sacrebleu.metrics import CHRF
chrf = CHRF(word_order=2)
def mbr(prompt, n=16, temperature=0.8):
outs = [ask(prompt, temperature=temperature) for _ in range(n)] # ask() from csw_helpers (sampled)
outs = [o for o in outs if parse(o)["ok"]] or outs
texts = [parse(o)["text"] for o in outs]
score = [sum(chrf.sentence_score(t, [u]).score for j, u in enumerate(texts) if j != i) for i, t in enumerate(texts)]
return outs[max(range(len(outs)), key=score.__getitem__)]
Evaluation
Test set: 6,230 items in 34 of the 37 languages (held-out; never used for training), greedy decoding, chrF++ (word order 2) against one reference, mean over task x language cells. The reference is a rewrite produced by our data pipeline with language models, so chrF++ measures closeness to that style, not human-judged quality.
| Task | SFT start | RL1 (rules only) | RL2 (rules + evaluator) | RL3 (this release) | RL3 + MBR-16 |
|---|---|---|---|---|---|
| eng_hm | 53.80 | 52.91 | 53.43 | 53.76 | 54.66 |
| m_hm | 59.28 | 58.23 | 59.03 | 58.30 | 59.25 |
| eng_ood | 41.09 | 40.22 | 39.82 | 40.27 | 41.54 |
| m_ood | 44.17 | 44.17 | 44.06 | 44.65 | 45.91 |
| hm_ood | 41.93 | 41.36 | 41.50 | 42.16 | 43.66 |
| mean | 48.05 | 47.38 | 47.57 | 47.83 | 49.00 |
Format validity (one <dev> block and a non-empty text) is 0.97-1.00 on every task for RL3 (0.92 for the SFT start on the code-switching tasks); the listed devices appear in the text 94-96% of the time (86-95% for the SFT start).
RL3 is not better than the SFT start under greedy decoding overall (47.83 vs 48.05): better on m_ood (+0.5) and hm_ood (+0.2), equal on eng_hm, worse on m_hm (-1.0) and eng_ood (-0.8). Its advantages are the more reliable format and device use, and MBR decoding (+1.17).
Variants of the same checkpoint (same test set, greedy): unpruned bf16 47.83 | pruned bf16 47.90 | pruned FP8 47.77 | pruned Q8_0 GGUF 49.25 on a 120-prompt subset where the bf16 pruned model scores 49.43.
How it was made
- Base:
google/gemma-3-1b-it, trained by us into an English/African-language translation model (SFT, then RL), then extended with nine new language-tag tokens in unused vocabulary slots. - Supervised stage (SFT): 633,579 rows, two epochs, five csw tasks in 37 languages; no English-to-X translation rows in this stage.
- Reinforcement learning (GRPO, groups of 8, 200 steps, learning rate 1e-6): reward = a format gate, rule penalties (device use, switch-language share, language identification, length, repetition, English leakage, diacritics, copying the source), a learned quality evaluator (capped, and scaled down for copies or a wrong switch-language share) and AfriCOMET-QE for the code-switching tasks in the languages it covers. The group's anchor member is the MBR-16 pick instead of the greedy output. The checkpoint with the best held-out chrF++ (step 150) is the release (
RL3).
Limitations and risks
- Outputs are machine-generated and not reviewed by native speakers. Training data and references were produced and graded with language models; they can contain unnatural phrasing, invented idioms or proverbs, wrong tone marks and meaning drift. Do not use the outputs where correctness matters without human review.
- Quality varies by language and by pair. Low-resource languages are weaker, a few languages have very few test items, and Fulfulde and Bambara had almost no
m_hmprompts in the RL stage; per-language numbers are noisy. - Rewriting, not translation. It can drop or change content and add devices that are not in the source. Pidgin outputs keep much English because English words are normal Pidgin.
- The learned reward could be gamed. In one run the evaluator's score rose while quality fell (the policy drifted toward copying its input); the released run adds rules against that, but quality was measured only with chrF++ and the evaluator, not with human raters.
- No added safety training beyond what the base model had, and no systematic check for offensive or harmful output.
The csw 1B model family
| Repository | What it is |
|---|---|
AletheiaAING/gemma-3-0.8b-csw |
Q8_0 GGUF, pruned vocabulary (about 0.82B parameters), public. Smallest; runs on CPU with llama.cpp. |
AletheiaAING/gemma-3-1b-csw-best-pruned-fp8 |
best model, pruned vocabulary, FP8 weights for vLLM |
AletheiaAING/gemma-3-1b-csw-best-pruned |
best model, pruned vocabulary, bf16 |
AletheiaAING/gemma-3-1b-csw-best |
best model (RL3 best-chrF++ checkpoint), full vocabulary, bf16 |
AletheiaAING/gemma-3-1b-csw-sft |
the supervised model every RL run starts from |
AletheiaAING/gemma-3-1b-csw-experiments |
the RL1 (rules only), DPO and RL2 (evaluator) training experiments in one repository |
License and attribution
Commercial use is allowed, with attribution to AletheiaAING (see the notice at the top of this card). Because this model is derived from Google Gemma, the Gemma Terms of Use and prohibited use policy also apply. Training data and quality labels used the outputs of other models (a 31B instruction model for generation, Gemini 3.8 Flash as judge); check the terms of those services for your own use. A full data inventory is in the accompanying project report; if you need the exact sources and licences, ask the authors.