gpt-mha-RoPE

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 6.0537

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 128
• eval_batch_size: 128
• seed: 20
• gradient_accumulation_steps: 4
• total_train_batch_size: 512
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 107
• training_steps: 1072
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
9.4939	0.0595	107	9.1958
7.6988	0.1190	214	7.5899
6.9868	0.1786	321	6.9357
6.6365	0.2381	428	6.5706
6.3897	0.2976	535	6.3402
6.2427	0.3571	642	6.2022
6.1614	0.4166	749	6.1189
6.1100	0.4762	856	6.0740
6.0978	0.5357	963	6.0566
6.0835	0.5952	1070	6.0537
6.0835	0.5963	1072	6.0537

Framework versions

• Transformers 5.5.4
• Pytorch 2.11.0+cu130
• Datasets 4.8.4
• Tokenizers 0.22.2