Ik45/fihris-embeddding-id-ar

Model Summary

This project provides a universal tokenizer pruning and multilingual continual pre-training pipeline built on top of sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. The pipeline performs:

1. Tokenizer Pruning — Removes unused tokens from a multilingual Unigram tokenizer to create a compact vocabulary optimized for specific language(s).
2. Embedding Pruning — Correctly resizes token embeddings using resize_token_embeddings() with ID remapping, avoiding the common pitfall of shape-mismatched state dict loading.
3. Multilingual Continual Pre-training — Further trains the pruned model on Indonesian and Arabic corpora using Masked Language Modeling (MLM).

The result is a lightweight, language-specific sentence embedding model with significantly reduced vocabulary size (from ~250K to ~39K tokens) while preserving semantic quality across target languages.

---

Intended Uses & Limitations

Intended Uses

• Semantic Search for Indonesian and Arabic text
• Text Clustering and similarity tasks
• Cross-lingual retrieval between Indonesian ↔ Arabic ↔ English
• Resource-constrained deployments where smaller vocabularies are preferred

Limitations

• The pruned vocabulary is optimized for the target language(s) selected during pruning; performance may degrade on languages outside the training distribution.
• Embedding quality is verified via cosine similarity comparison between original and pruned models, but some semantic drift is expected.
• The model is a sentence transformer (384-dim embeddings), not a generative model.

---

Training Details

Stage 1: Tokenizer Pruning

• Base Tokenizer: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (Unigram, 250K vocab)
• Pruning Method: Frequency-based token removal using FineWeb-2 corpus
• Supported Presets:
  • en_only — English only
  • id_only — Indonesian only
  • en_id — English + Indonesian
  • id_ar — Indonesian + Arabic
  • ar_only — Arabic only
  • en_id_ar — English + Indonesian + Arabic
• Metrics Tracked:
  • BPT (Bytes Per Token) — higher is more efficient
  • Perplexity (PPL) — lower is better
  • Embedding Cosine Similarity — measures semantic preservation

Stage 2: Continual Pre-training (MLM)

• Base Model: Ik45/fihris-embeddding-id-ar (derived from the pruned multilingual MiniLM)
• Training Data:
  • Arabic: ik4545/shamela-waqfeya-split (classical Arabic texts, ~3.57GB)
  • Indonesian: Custom modern Indonesian text corpus (indo_modern.txt)
• Training Objective: Masked Language Modeling (MLM)
  • Mask probability: 15%
  • 80% → [MASK], 10% → random token, 10% → unchanged
• Hyperparameters:
  • Epochs: 2
  • Batch size: 16
  • Learning rate: 1e-3
  • Warmup ratio: 0.1
  • Weight decay: 0.01
  • Max sequence length: 512
  • Mixed precision: FP16
• Data Collator: Custom DataCollatorForMLMOnTheFly with on-the-fly tokenization

Stage 3: MLM Head Reconstruction (for Embedding → MLM)

When converting a sentence transformer encoder to an MLM model:

• Encoder weights are copied from the pre-trained model
• Decoder weight is tied to the word embedding matrix
• Transform layer initialized as near-identity mapping
• LayerNorm and decoder bias initialized to standard values

---

Evaluation

Quality Verification Metrics

Language	BPT Original	BPT Pruned	PPL Original	PPL Pruned	CosSim
Target	~3.5	~4.2	~12.3	~14.1	>0.95
Other	~3.5	~2.8	~12.3	~18.5	~0.85

Values are illustrative; actual results depend on the selected language preset.

Evaluation Methodology

• BPT measured on held-out corpus samples
• PPL computed via forward pass on masked sequences
• Cosine Similarity between original and pruned model embeddings on identical sentences

---

How to Use

Quick Start — Sentence Embeddings

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("your-username/fihris-embedding-id-ar-pruned")
sentences = [
    "Ini adalah kalimat dalam bahasa Indonesia.",
    "هذه جملة باللغة العربية.",
    "This is a sentence in English."
]

embeddings = model.encode(sentences)
print(embeddings.shape)  # (3, 384)

Using with HuggingFace Transformers

from transformers import AutoTokenizer, AutoModel
import torch

tokenizer = AutoTokenizer.from_pretrained("your-username/fihris-embedding-id-ar-pruned")
model = AutoModel.from_pretrained("your-username/fihris-embedding-id-ar-pruned")

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output.last_hidden_state
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

texts = ["السلام عليكم", "Selamat pagi", "Good morning"]
encoded = tokenizer(texts, padding=True, truncation=True, max_length=128, return_tensors="pt")

with torch.no_grad():
    output = model(**encoded)

embeddings = mean_pooling(output, encoded["attention_mask"])

Running the Pruning Pipeline

# Configure and run the pruning notebook
UNI_MODEL_NAME = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
UNI_PRESET = "id_ar"  # Indonesian + Arabic
UNI_PRUNE_EMBEDDINGS = True
UNI_OUTPUT_DIR = "./pruned_output"

# The notebook will:
# 1. Load tokenizer and detect type (Unigram/BPE/WordPiece)
# 2. Compute token frequencies from FineWeb-2
# 3. Remove low-frequency tokens
# 4. Prune embeddings via resize_token_embeddings()
# 5. Verify quality with BPT, PPL, and CosSim
# 6. Save pruned tokenizer + model

---

Citation

If you use this model or pipeline, please cite:

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "http://arxiv.org/abs/1908.10084",
}

---

Acknowledgments

• Base model: sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
• Arabic corpus: ik4545/shamela-waqfeya-split (Shamela Waqfeya)
• Training corpora: HuggingFaceFW/fineweb and HuggingFaceFW/fineweb-2
• Framework: HuggingFace Transformers, Sentence-Transformers

---