t5_small-xsum

google/t5-small-lm-adapt (LM-adapted T5 v1.1 small) fully fine-tuned on XSum with torchdistill.

It reproduces the supervised fine-tuning stage of GKD (Agarwal et al., ICLR 2024, Appendix A.2). It is the student initialization and no-distillation baseline for the T5-small knowledge distillation (KD) experiments: every KD config for T5-small starts from this checkpoint, and the distilled students are compared against it.

Model details

Base model google/t5-small-lm-adapt (77M parameters, T5 v1.1 encoder-decoder, 8 encoder + 8 decoder layers)
Fine-tuning Full-parameter supervised fine-tuning, cross-entropy on the reference summaries
Training data EdinburghNLP/xsum, first 99% of the train split
Language English
Lengths used in training input up to 1,024 tokens, summary up to 64 tokens
Input format summarize: <document>
Weights model.safetensors, float32
License Apache 2.0 (see License)

Usage

The model was trained on inputs prefixed with summarize: and evaluated with greedy decoding of up to 64 new tokens. Pass both explicitly: the bundled generation_config.json sets no length, so generate would otherwise stop at the library default of 20 tokens, and the tokenizer's model_max_length (512) is shorter than the 1,024 input tokens used in training.

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = 'yoshitomo-matsubara/t5_small-xsum'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id, dtype='bfloat16', device_map='auto')

document = 'The full text of a news article ...'
inputs = tokenizer(
    f'summarize: {document}', max_length=1024, truncation=True, return_tensors='pt'
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64, do_sample=False, num_beams=1)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For batches, pass a list of inputs with padding=True. Unlike a decoder-only model, T5 needs no left padding: the padded encoder positions are masked out, and the decoder starts from its own start token.

Use bf16 or fp32. T5 overflows in fp16.

Training data and split

EdinburghNLP/xsum has train (204,045 examples), validation (11,332) and test (11,334) splits. They are used as follows, with the datasets split syntax:

Split Use
train[:99%] Training
train[99%:] "Monitor" split: perplexity after each epoch, for monitoring only
validation Reported ROUGE (the paper's reporting split)
test Reported ROUGE

The paper reports the validation split and selects no checkpoint, so this run keeps the last checkpoint (ckpt_selection: 'last'). Neither reported split, nor the monitor split, is used for any decision. The training run did not pin a dataset revision.

Each example maps the document field, prefixed with summarize: (the T5 convention; the paper does not state its input format), to the summary field as the target.

Training procedure

The paper fine-tunes T5-small for 100,000 steps and gives no other SFT hyperparameters, so the run reuses those of its distillation runs (Table A.1: Adafactor, batch size 32, input/output length 1024/64, learning rate per Appendix A.3), with warmup and cooldown at the same proportions of the run (5% and the last 25%).

Dropout is the one deviation: T5 v1.1's default 0.1 instead of Table A.1's 0.0. With 0.0, plain fine-tuning overfits (T5-base: monitor perplexity 5.2 after epoch 1, 52 after epoch 8) and falls short of the paper's ROUGE-2.

Hyperparameter Value
Steps 100,000 optimizer steps (16 epochs, the last one partial)
Optimizer Adafactor (scale_parameter=False, relative_step=False, warmup_init=False), lr 1e-3
Schedule Warmup-stable-decay: linear warmup over 5,000 steps, constant, linear decay to 0 over the last 25,000 steps
Batch size 8 per device x 2 devices x gradient accumulation 2 = 32
Dropout 0.1
Gradient clipping none
Precision fp32 master weights, bf16 autocast
Hardware 2 GPUs, DDP, bf16 mixed precision (accelerate launch)
Training time ~34 min for the first epoch, ~9 h 5 min in total
Seed 42

The full torchdistill config, CLI arguments and per-step metrics are in logs/: config.yaml, args.json, metrics.json and trackio/summary.csv. To reproduce the run with torchdistill, use the command in the header of logs/config.yaml.

Evaluation

ROUGE F-measure (x100) of greedy decoding (up to 64 new tokens) on the released, last checkpoint:

Split ROUGE-1 ROUGE-2 ROUGE-L
Validation (11,332) 38.63 16.06 31.10
Test (11,334) 38.52 15.85 30.94

The paper reports a ROUGE-2 of 13.4 on the validation split after this stage. Both splits are held out from training and were not used to select the checkpoint. The scores come from the 2-process distributed evaluation of the training run.

Perplexity over the summary tokens of the monitor split, measured after each epoch:

Epoch Monitor perplexity
1 8.8971
2 7.8231
3 7.3308
4 7.1183
5 6.9253
6 6.8334
7 6.7924
8 6.7822
9 6.7498
10 6.7033
11 6.6901
12 6.6955
13 6.7240
14 6.7181
15 6.7818
16 (released) 6.8748

The monitor perplexity was lowest after epoch 11 (6.6901) and rose afterwards, but no checkpoint was selected on it (see above).

Limitations

  • XSum's reference summaries often contain information that is not in the article, and models fine-tuned on it are known to hallucinate such content (Maynez et al., 2020). Summaries may be fluent but unfaithful.
  • Trained on English BBC news articles; quality on other domains and languages is not measured.
  • Inputs longer than 1,024 tokens were truncated in training; summaries are single sentences of up to 64 tokens.
  • No downstream evaluation beyond ROUGE on XSum (e.g. factual-consistency metrics or human evaluation) has been run.
  • Intended for research, in particular as a reference point for knowledge-distillation experiments.

License

The weights are released under the Apache License 2.0 (LICENSE).

This model is a modification of google/t5-small-lm-adapt by Google, which is licensed under the Apache License 2.0. The modification is full-parameter fine-tuning on the data described above.

The training data, XSum, consists of BBC articles collected by Narayan et al. (2018) for research and carries no explicit license on the Hugging Face Hub. Users are responsible for making sure their use of the model is consistent with the terms of the underlying data.

Citation

If you use this model, please cite torchdistill, as well as GKD, T5 and XSum:

@inproceedings{matsubara2021torchdistill,
  title={torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation},
  author={Matsubara, Yoshitomo},
  booktitle={International Workshop on Reproducible Research in Pattern Recognition},
  pages={24--44},
  year={2021},
  organization={Springer}
}

@inproceedings{matsubara2023torchdistill,
  title={{torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP}},
  author={Matsubara, Yoshitomo},
  booktitle={Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)},
  publisher={Empirical Methods in Natural Language Processing},
  pages={153--164},
  year={2023}
}

@inproceedings{agarwal2024onpolicy,
  title={On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes},
  author={Rishabh Agarwal and Nino Vieillard and Yongchao Zhou and Piotr Stanczyk and Sabela Ramos Garea and Matthieu Geist and Olivier Bachem},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2024}
}

@article{raffel2020t5,
  title={Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
  author={Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
  journal={Journal of Machine Learning Research},
  volume={21},
  number={140},
  pages={1--67},
  year={2020}
}

@inproceedings{lester2021power,
  title={The Power of Scale for Parameter-Efficient Prompt Tuning},
  author={Brian Lester and Rami Al-Rfou and Noah Constant},
  booktitle={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  pages={3045--3059},
  year={2021}
}

@inproceedings{narayan2018xsum,
  title={Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization},
  author={Shashi Narayan and Shay B. Cohen and Mirella Lapata},
  booktitle={Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  pages={1797--1807},
  year={2018}
}
Downloads last month
16
Safetensors
Model size
77M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for yoshitomo-matsubara/t5_small-xsum

Finetuned
(3)
this model

Dataset used to train yoshitomo-matsubara/t5_small-xsum

Paper for yoshitomo-matsubara/t5_small-xsum

Evaluation results