nikitastheo/phonebabylm-text-fas-eng-sequential_interleaved

Trained with train_clm.py, a Hugging Face Accelerate causal-LM training script (no Trainer).

Training details

  • Base config: gpt_base_config.json
  • Tokenizer: nikitastheo/phonelm-text-fas-eng-tokenizer
  • Max steps: 30000
  • Learning rate: 0.0001
  • LR scheduler: linear
  • Warmup steps: 3000
  • Batch size (per device): 32
  • Gradient accumulation steps: 1
  • Total train batch size: 32
  • Language switch steps: 10000
Downloads last month
2
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support