---
language:
- hi
- en
pipeline_tag: text-generation
tags:
- bharatgen
- bilingual
- hindi
- english
- causal-lm
- transformers
license: other
---
# Param-1-5B
**Param-1-5B** is a bilingual (**English–Hindi**) large language model developed under the **Param-1** family. With **5 billion parameters**, this model extends the capabilities of **Param-1-2.9B** by incorporating **enhanced mathematical reasoning and code understanding/generation**.
The model is pretrained from scratch and designed to serve as a strong foundation for downstream tasks such as mathematical problem solving, and code-related understanding / generation.
---
## Key Highlights
* **5B parameter** dense Transformer model
* **Bilingual**: English and Hindi
* **Enhanced domains**: Math and Code (compared to Param-1-2.9B)
* Updated **dataset mixture and percentage ratios**
* Designed as a **pretrained (PT)** base model
---
## Model Inference
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
model_name = "bharatgenai/Param-1-5B"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
model_name,
trust_remote_code=True,
torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
device_map="auto"
)
prompt = "Your prompt here"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
output = model.generate(
**inputs,
max_new_tokens=300,
do_sample=True,
top_k=50,
top_p=0.95,
temperature=0.6,
eos_token_id=tokenizer.eos_token_id,
use_cache=False
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```
---
## Benchmarks
| Task | **Param-1-5B (PT)** |
| ----------------- | ------------------- |
| ARC Challenge | 55.1 (few) |
| ARC Easy | 75.6 (few) |
| HellaSwag | 76 (few) |
| HellaSwag Hi | 44.6 (few) |
| MMLU En | 50.8 (few) |
| MMLU Hi | 39.1 (few) |
| TriviaQA | 48.2 |
| TruthfulQA - Gen | 32.9 |
| TruthfulQA - MC1 | 27.1 |
| TruthfulQA - MC2 | 38.6 |
| PIQA | 79.6 |
| SuperGLUE - BoolQ | 72.7 |
| SuperGLUE - WiC | 50.5 |
| SuperGLUE - WSC | 48.1 |
| SuperGLUE - RTE | 65 |
---
## Model Architecture
* Architecture: Transformer (Decoder-only)
* Number of parameters: **~5B**
* Hidden size: `4096`
* Intermediate size: `14336`
* Number of layers: `16`
* Number of attention heads: `32`
* Number of key-value heads: `8`
* Maximum position embedding: `4096`
* Positional embeddings: Rotary (RoPE)
* Attention mechanism: Grouped-query attention
* Activation function: `silu`
* Precision: `bf16-mixed`
---
## Training Data
The model is pretrained on a large-scale bilingual corpus with a strong focus on **English and Hindi**, along with **dedicated Math and Code data**.
### Dataset Composition (Total ≈ **6.61T tokens**)
- **English Natural Language:** `3.6T`
- **Hindi Natural Language:** `2.77T`
- **Math & Code:** `238.4B`
- Math: **40%**
- Code: **60%**
Compared to **Param-1-2.9B**, **Param-1-5B** includes:
- A **larger overall training corpus**
- Explicit inclusion of **Math and Code datasets**
- Improved performance in **mathematical** and **code understanding/generation**
- Continued emphasis on **English–Hindi bilingual capability**
---
## Training Details
* Training framework: `NVIDIA NeMo`
* Training infrastructure: `Yotta's Shakti Cloud`
* Hardware: `NVIDIA H100 - 1024 GPUs`
* Total training tokens: `6.61 Trillion`
---
## Limitations
* This is a **pretrained base model** and may require fine-tuning for instruction-following or chat use cases.
* The model may reflect biases present in large-scale web and code data.
* Performance on low-resource Hindi domains may vary depending on task complexity.
---
## License
This model is released under the **BharatGen non-commercial license**.
Please refer to the [LICENSE](./LICENSE) file for detailed terms and conditions.
---
.png)
