zmsali/qwen3-4b-bangla-agent

Qwen3-4B Bangla Agent

A Bangla-first assistant model for agentic use on CPU: it answers in Bangla, does grounded question answering over supplied documents (RAG) with citations, calls tools in Qwen3's native <tool_call> format from Bangla or English requests, and answers in Bangla after reading tool results.

It is a QLoRA fine-tune of Qwen/Qwen3-4B-Instruct-2507. The repository ships the LoRA adapter and merged GGUF files for llama.cpp, Ollama, LM Studio and Jan.

Base model	Qwen3-4B-Instruct-2507 (4.02B parameters, non-thinking)
Method	QLoRA, rank 16, all attention and MLP projections, 33.0M trainable parameters (0.8%)
Training data	~16.1k conversations: Bangla instructions, Bangla RAG, English and Bangla tool calling
Context	trained at 2,048 tokens; base supports far longer
Languages	Bangla (primary), English (tool schemas, tool results, mixed prompts)
Hardware	2× Tesla T4 on Kaggle, data-parallel, 7.1 hours
Formats	PEFT adapter · GGUF Q4_K_M (2.4 GiB) · GGUF Q8_0 (4.0 GiB)

Repository layout

adapter/            LoRA adapter (safetensors) + tokenizer + chat template
gguf/               qwen3-4b-bangla-agent-Q4_K_M.gguf   2.4 GiB   default for CPU
                    qwen3-4b-bangla-agent-Q8_0.gguf     4.0 GiB   near-lossless
train_log.json      full Trainer log history (loss, LR, per-source eval loss)
last-checkpoint/    final trainer checkpoint (adapter + optimizer + scheduler), for resuming

Quick start

llama.cpp (CPU, recommended)

--jinja is required: it enables the Qwen3 chat template embedded in the GGUF, which is what turns the model's <tool_call> output into structured tool calls on the OpenAI-compatible API.

llama-server -m qwen3-4b-bangla-agent-Q4_K_M.gguf --jinja -c 8192 --port 8080

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="x")

tools = [{"type": "function", "function": {
    "name": "get_weather",
    "description": "Get the current weather for a city.",
    "parameters": {"type": "object",
                   "properties": {"city": {"type": "string", "description": "City name in English"}},
                   "required": ["city"]}}}]

r = client.chat.completions.create(
    model="local", tools=tools,
    messages=[{"role": "user", "content": "আজ সিলেট শহরের আবহাওয়া কেমন?"}])
print(r.choices[0].message.tool_calls)
# -> get_weather(arguments='{"city": "Sylhet"}')

Feed the tool result back as a tool message and the model answers in Bangla:

messages += [r.choices[0].message,
             {"role": "tool", "tool_call_id": r.choices[0].message.tool_calls[0].id,
              "content": '{"city": "Sylhet", "temperature_c": 29, "condition": "light rain", "humidity": 88}'}]
r2 = client.chat.completions.create(model="local", tools=tools, messages=messages)
print(r2.choices[0].message.content)
# -> সিলেট শহরে এখন তাপমাত্রা ২৯°C, আবহাওয়া হালকা বৃষ্টি এবং আর্দ্রতা ৮৮%।

Q4_K_M runs at roughly 10 to 20 tokens per second on a modern laptop CPU. Use Q8_0 when you want to rule out quantization as the cause of an odd answer.

Transformers + PEFT (GPU)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base_id = "Qwen/Qwen3-4B-Instruct-2507"
tok = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "zmsali/qwen3-4b-bangla-agent", subfolder="adapter").eval()

messages = [{"role": "user", "content": "বাংলাদেশের স্বাধীনতা যুদ্ধ কত সালে হয়েছিল? দুই বাক্যে বলো।"}]
text = tok.apply_chat_template(messages, tools=None, tokenize=False, add_generation_prompt=True)
ids = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=200, do_sample=False)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))

Pass tools=[...] to apply_chat_template for tool calling; the template renders the tool schemas into the system turn exactly as they were rendered during training.

Ollama

FROM ./qwen3-4b-bangla-agent-Q4_K_M.gguf
PARAMETER temperature 0.3

Add the TEMPLATE block from ollama show --modelfile qwen3:4b (same tool-call format), then ollama create bangla-agent -f Modelfile.

Prompt formats the model was trained on

System prompt. Half of the training conversations had a system prompt and half had none, so both work. Examples used in training:

• তুমি একজন সহায়ক বাংলা এআই সহকারী। ব্যবহারকারীর প্রশ্নের সঠিক ও সংক্ষিপ্ত উত্তর বাংলায় দাও।
• You are a helpful assistant. Answer in Bangla unless the user writes in another language.

Tool calling. Standard Qwen3 / Hermes format, produced by the chat template from OpenAI-style tools:

<tool_call>
{"name": "get_weather", "arguments": {"city": "Sylhet"}}
</tool_call>

Tool results go back as role: "tool" messages. The model was trained on single calls, parallel calls, multi-turn tool loops, and on conversations where tools were offered but not needed.

RAG. Put retrieved passages in the user message, numbered with Bangla digits, and end with the question:

নিচের নথিগুলো ব্যবহার করে প্রশ্নের উত্তর দাও। শুধুমাত্র নথিতে থাকা তথ্যের ভিত্তিতে উত্তর দেবে;
নথিতে উত্তর না থাকলে স্পষ্টভাবে বলবে যে তথ্যটি পাওয়া যায়নি।

[নথি ১]
...passage...

[নথি ২]
...passage...

প্রশ্ন: ...

The model answers briefly, often with a citation (নথি ১ অনুযায়ী, …), and declines when the passages do not contain the answer (প্রদত্ত নথিতে এই প্রশ্নের উত্তর পাওয়া যায়নি।). For agentic RAG, expose retrieval as a tool named search_documents(query, top_k); the model was trained to call it.

Training data

All examples were rendered through the base model's own chat template with tools= where applicable, so the training text is byte-identical to inference text. Loss was computed only on assistant turns, including <tool_call> blocks and the closing <|im_end|>. Examples longer than 2,048 tokens were dropped. Approximately 16.1k conversations were used for training; 100 per source (40 for the smallest) were held out for evaluation.

Source	Approx. rows	Content	License
Bangla-Instruct	~7,100	Bangla instruction / response pairs, random subset (seed 3407)	MIT
squad_bn, rewritten as RAG	~2,200	1 to 3 Bangla passages per example with the answer passage at a random position; answers phrased with and without citations; ~10% with the gold passage removed and a "not found" reply; natively unanswerable questions kept as "not found"	CC-BY-NC-SA-4.0
hermes-function-calling-v1	~5,200	English tool calling, func_calling_singleturn, func_calling and glaive_func_calling configs; converted to structured tool_calls and tool messages	Apache-2.0
Synthetic Bangla tool calling	~1,500	Templated Bangla requests for get_weather, calculator, convert_currency, set_reminder (full call → result → Bangla answer loops) and get_news, web_search, search_documents (call only), with 1 to 3 distractor tools and English argument values	generated
Bangla, tools offered but not needed	~350	Bangla-Instruct rows with 2 to 4 tool schemas attached and a direct answer	MIT

Bangla is token-expensive in Qwen's vocabulary (about 1.5 characters per token); a Bangla-Instruct row averages ~1,000 tokens and a RAG row ~1,300.

Training procedure

Quantization	4-bit NF4, double quantization, fp16 compute (bitsandbytes)
LoRA	r = 16, α = 32, dropout 0.05, targets q_proj k_proj v_proj o_proj gate_proj up_proj down_proj
Trainable parameters	33.0M of 4.02B
Optimizer	paged AdamW 8-bit, weight decay 0, gradient clipping 1.0
Learning rate	1e-4, cosine schedule, 15 warmup steps
Batch	2 per GPU × 8 accumulation × 2 GPUs = 32 sequences per step, length-grouped
Steps / epochs	503 steps, 1 epoch
Sequence length	2,048, padded to the longest in each batch
Precision	fp16 mixed precision (T4 has no bf16), gradient checkpointing
Parallelism	PyTorch DDP via torchrun, one 4-bit model replica per GPU
Hardware / time	2× Tesla T4 (Kaggle), 7.06 h, ~590 tokens/s aggregate, 3.2e17 FLOPs
Framework	transformers 5.16, peft, bitsandbytes, datasets; no Unsloth or TRL
Final training loss	0.406 (mean over the run)

The merged GGUF files were produced by merging the adapter into fp16 base weights, converting with llama.cpp convert_hf_to_gguf.py --outtype f16, and quantizing with llama-quantize to Q4_K_M and Q8_0. Because the adapter was trained against 4-bit weights and merged into fp16 ones, there is a small, expected mismatch; Q8_0 is the closer of the two to the adapter as trained.

Evaluation

Held-out cross-entropy loss per data source, before training and at three points during it. Lower is better. The tools row is the one to watch for forgetting: it fell rather than rose, so Bangla training did not erode tool calling.

Source	Before training	Step 200	Step 400	Final (503)
instruct (Bangla)	0.901	0.494	0.479	0.478
notool (Bangla, tools offered)	0.722	0.514	0.497	0.497
rag (Bangla)	1.259	0.190	0.118	0.115
synth (Bangla tool calls)	0.660	0.055	0.028	0.026
tools (English, Hermes)	0.556	0.096	0.087	0.087

Qualitative checks, greedy decoding, run on the 4-bit base plus adapter after training. Verbatim outputs:

Prompt	Output
বাংলাদেশের স্বাধীনতা যুদ্ধ কত সালে হয়েছিল? দুই বাক্যে বলো।	বাংলাদেশের স্বাধীনতা যুদ্ধ ১৯৭১ সালে হয়েছিল। এটি ভারতের সাথে সমন্বিত হয়ে স্বাধীন দেশ হয়ে উঠেছিল।
RAG, one passage about the National Parliament House, question: who was the architect?	নথি ১ অনুযায়ী, জাতীয় সংসদ ভবনের স্থপতি ছিলেন লুই আই কান।
Same passage, question not answerable from it (construction cost)	পাওয়া যায়নি
আজ সিলেট শহরের আবহাওয়া কেমন? with get_weather and calculator offered	<tool_call>{"name": "get_weather", "arguments": {"city": "Sylhet"}}</tool_call>
After the tool returned 29 °C, light rain, 88% humidity, 9 km/h wind	সিলেট শহরে এখন তাপমাত্রা ২৯°C, আকাশ হালকা বৃষ্টি এবং আর্দ্রতা ৮৮%। বাতাসের গতি ঘণ্টায় ৯ কিলোমিটার।
বাংলাদেশের জাতীয় ফুল কী? with tools offered	বাংলাদেশের জাতীয় ফুল হলো বাংলাদেশ ফুল। … (wrong; correct answer is শাপলা)

No standardized Bangla benchmark scores are reported. The eval losses above are on held-out slices of the training distribution and are not comparable across models.

Limitations

• Factual recall in Bangla is weak, as expected for a 4B model, and the last example above shows a confident hallucination. Use the model with retrieval or a search tool for facts; do not rely on its memory.
• Bangla-Instruct is machine-generated and noisy. Some training responses are glossaries or truncated answers, and this shows in occasional awkward phrasing (the second sentence of the first example).
• Synthetic tool-calling data is narrow. Seven tools, templated phrasing, twenty cities. The model generalizes to unseen schemas because the Hermes data is diverse, but Bangla answers after tool results tend to echo the training templates.
• Standard Bangla only. Regional dialects (Chittagonian, Sylheti, Noakhali and others) were not in the training data.
• Non-thinking model. The base is the 2507 Instruct checkpoint and emits no reasoning traces. Do not set enable_thinking.
• Trained at 2,048 tokens. Longer contexts inherit the base model's behavior and were not tested.
• No safety tuning beyond the base model's. Qwen3's refusals carry over; nothing Bangla-specific was added.

Intended use

Local Bangla assistants and agents on CPU: document question answering over a private corpus, tool-using assistants (weather, calculators, reminders, search), and as a starting adapter for further Bangla fine-tuning. Not intended for medical, legal or financial advice, or for any use where a wrong Bangla fact stated confidently is harmful.

License and attribution

The base model and this adapter are released under Apache-2.0. Note that one training source, squad_bn, is licensed CC-BY-NC-SA-4.0; if you need a strictly commercial-clean model, retrain with the RAG source removed or replaced (the training scripts make this a one-line change). Bangla-Instruct is MIT and hermes-function-calling-v1 is Apache-2.0.

If you use this model, please also credit the data sources:

• Raihan and Zampieri, TigerLLM: A Family of Bangla Large Language Models, ACL 2025 (Bangla-Instruct)
• Bhattacharjee et al., BanglaBERT, Findings of NAACL 2022 (squad_bn)
• NousResearch, Hermes Function Calling v1

Reproduction

Training was done with three scripts run from a single Kaggle notebook: a data builder that renders and masks every source through the chat template, a torchrun DDP trainer that pushes full checkpoints to this repository every 100 steps for resumable training, and an export cell that merges and converts to GGUF. train_log.json and last-checkpoint/trainer_state.json contain the complete run history. Random seed 3407 throughout.

Citation

@misc{qwen3-4b-bangla-agent,
  author = {Shahjahan Ali},
  title  = {Qwen3-4B Bangla Agent: a Bangla tool-calling and RAG assistant for CPU},
  year   = {2026},
  url    = {https://huggingface.co/zmsali/qwen3-4b-bangla-agent}
}