| --- |
| language: |
| - hi |
| - en |
| pipeline_tag: text-generation |
| tags: |
| - bharatgen |
| - bilingual |
| - hindi |
| - english |
| - causal-lm |
| - transformers |
| license: other |
| --- |
| |
|
|
| <div align="center"> |
| <a href="https://bharatgen.com" target="_blank"> |
| <img src="https://huggingface.co/bharatgenai/Param-1-2.9B-Instruct/resolve/main/BharatGen%20Logo%20(1).png" width="60%" alt="BharatGen" /> |
| </a> |
| </div> |
| <hr> |
| <div align="center"> |
| <a href="https://bharatgen.com" target="_blank" style="margin: 4px;"> |
| <img alt="Homepage" src="https://img.shields.io/badge/Homepage-Visit-blue.svg" /> |
| </a> |
| <a href="./LICENSE" target="_blank" style="margin: 4px;"> |
| <img alt="License" src="https://img.shields.io/badge/License-yellow.svg" /> |
| </a> |
| </div> |
| |
| # Param-1-5B |
|
|
| **Param-1-5B** is a bilingual (**English–Hindi**) large language model developed under the **Param-1** family. With **5 billion parameters**, this model extends the capabilities of **Param-1-2.9B** by incorporating **enhanced mathematical reasoning and code understanding/generation**. |
|
|
| The model is pretrained from scratch and designed to serve as a strong foundation for downstream tasks such as mathematical problem solving, and code-related understanding / generation. |
|
|
| --- |
|
|
| ## Key Highlights |
|
|
| * **5B parameter** dense Transformer model |
| * **Bilingual**: English and Hindi |
| * **Enhanced domains**: Math and Code (compared to Param-1-2.9B) |
| * Updated **dataset mixture and percentage ratios** |
| * Designed as a **pretrained (PT)** base model |
|
|
| --- |
|
|
| ## Model Inference |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| import torch |
| |
| model_name = "bharatgenai/Param-1-5B" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=False) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_name, |
| trust_remote_code=True, |
| torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32, |
| device_map="auto" |
| ) |
| |
| prompt = "Your prompt here" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| |
| with torch.no_grad(): |
| output = model.generate( |
| **inputs, |
| max_new_tokens=300, |
| do_sample=True, |
| top_k=50, |
| top_p=0.95, |
| temperature=0.6, |
| eos_token_id=tokenizer.eos_token_id, |
| use_cache=False |
| ) |
| |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
|
|
| --- |
|
|
| ## Benchmarks |
|
|
| | Task | **Param-1-5B (PT)** | |
| | ----------------- | ------------------- | |
| | ARC Challenge | 55.1 (few) | |
| | ARC Easy | 75.6 (few) | |
| | HellaSwag | 76 (few) | |
| | HellaSwag Hi | 44.6 (few) | |
| | MMLU En | 50.8 (few) | |
| | MMLU Hi | 39.1 (few) | |
| | TriviaQA | 48.2 | |
| | TruthfulQA - Gen | 32.9 | |
| | TruthfulQA - MC1 | 27.1 | |
| | TruthfulQA - MC2 | 38.6 | |
| | PIQA | 79.6 | |
| | SuperGLUE - BoolQ | 72.7 | |
| | SuperGLUE - WiC | 50.5 | |
| | SuperGLUE - WSC | 48.1 | |
| | SuperGLUE - RTE | 65 | |
|
|
| --- |
|
|
| ## Model Architecture |
|
|
| * Architecture: Transformer (Decoder-only) |
| * Number of parameters: **~5B** |
| * Hidden size: `4096` |
| * Intermediate size: `14336` |
| * Number of layers: `16` |
| * Number of attention heads: `32` |
| * Number of key-value heads: `8` |
| * Maximum position embedding: `4096` |
| * Positional embeddings: Rotary (RoPE) |
| * Attention mechanism: Grouped-query attention |
| * Activation function: `silu` |
| * Precision: `bf16-mixed` |
|
|
| --- |
|
|
| ## Training Data |
|
|
| The model is pretrained on a large-scale bilingual corpus with a strong focus on **English and Hindi**, along with **dedicated Math and Code data**. |
|
|
| ### Dataset Composition (Total ≈ **6.61T tokens**) |
|
|
| - **English Natural Language:** `3.6T` |
| - **Hindi Natural Language:** `2.77T` |
| - **Math & Code:** `238.4B` |
| - Math: **40%** |
| - Code: **60%** |
|
|
| Compared to **Param-1-2.9B**, **Param-1-5B** includes: |
| - A **larger overall training corpus** |
| - Explicit inclusion of **Math and Code datasets** |
| - Improved performance in **mathematical** and **code understanding/generation** |
| - Continued emphasis on **English–Hindi bilingual capability** |
|
|
| --- |
|
|
| ## Training Details |
|
|
| * Training framework: `NVIDIA NeMo` |
| * Training infrastructure: `Yotta's Shakti Cloud` |
| * Hardware: `NVIDIA H100 - 1024 GPUs` |
| * Total training tokens: `6.61 Trillion` |
|
|
| --- |
|
|
| ## Limitations |
|
|
| * This is a **pretrained base model** and may require fine-tuning for instruction-following or chat use cases. |
| * The model may reflect biases present in large-scale web and code data. |
| * Performance on low-resource Hindi domains may vary depending on task complexity. |
|
|
| --- |
|
|
| ## License |
|
|
| This model is released under the **BharatGen non-commercial license**. |
|
|
| Please refer to the [LICENSE](./LICENSE) file for detailed terms and conditions. |
|
|
| --- |
|
|
