Instructions to use yoshitomo-matsubara/t5_small-xsum with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use yoshitomo-matsubara/t5_small-xsum with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "summarization" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("summarization", model="yoshitomo-matsubara/t5_small-xsum")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("yoshitomo-matsubara/t5_small-xsum") model = AutoModelForSeq2SeqLM.from_pretrained("yoshitomo-matsubara/t5_small-xsum", device_map="auto") - Notebooks
- Google Colab
- Kaggle
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("yoshitomo-matsubara/t5_small-xsum")
model = AutoModelForSeq2SeqLM.from_pretrained("yoshitomo-matsubara/t5_small-xsum", device_map="auto")t5_small-xsum
google/t5-small-lm-adapt (LM-adapted T5 v1.1 small) fully fine-tuned on XSum with torchdistill.
It reproduces the supervised fine-tuning stage of GKD (Agarwal et al., ICLR 2024, Appendix A.2). It is the student initialization and no-distillation baseline for the T5-small knowledge distillation (KD) experiments: every KD config for T5-small starts from this checkpoint, and the distilled students are compared against it.
Model details
| Base model | google/t5-small-lm-adapt (77M parameters, T5 v1.1 encoder-decoder, 8 encoder + 8 decoder layers) |
| Fine-tuning | Full-parameter supervised fine-tuning, cross-entropy on the reference summaries |
| Training data | EdinburghNLP/xsum, first 99% of the train split |
| Language | English |
| Lengths used in training | input up to 1,024 tokens, summary up to 64 tokens |
| Input format | summarize: <document> |
| Weights | model.safetensors, float32 |
| License | Apache 2.0 (see License) |
Usage
The model was trained on inputs prefixed with summarize: and evaluated with greedy decoding of up to 64 new
tokens. Pass both explicitly: the bundled generation_config.json sets no length, so generate would otherwise stop
at the library default of 20 tokens, and the tokenizer's model_max_length (512) is shorter than the 1,024 input
tokens used in training.
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model_id = 'yoshitomo-matsubara/t5_small-xsum'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id, dtype='bfloat16', device_map='auto')
document = 'The full text of a news article ...'
inputs = tokenizer(
f'summarize: {document}', max_length=1024, truncation=True, return_tensors='pt'
).to(model.device)
outputs = model.generate(**inputs, max_new_tokens=64, do_sample=False, num_beams=1)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
For batches, pass a list of inputs with padding=True. Unlike a decoder-only model, T5 needs no left padding: the
padded encoder positions are masked out, and the decoder starts from its own start token.
Use bf16 or fp32. T5 overflows in fp16.
Training data and split
EdinburghNLP/xsum has train (204,045 examples), validation (11,332) and test (11,334) splits. They are used
as follows, with the datasets split syntax:
| Split | Use |
|---|---|
train[:99%] |
Training |
train[99%:] |
"Monitor" split: perplexity after each epoch, for monitoring only |
validation |
Reported ROUGE (the paper's reporting split) |
test |
Reported ROUGE |
The paper reports the validation split and selects no checkpoint, so this run keeps the last checkpoint
(ckpt_selection: 'last'). Neither reported split, nor the monitor split, is used for any decision. The training run
did not pin a dataset revision.
Each example maps the document field, prefixed with summarize: (the T5 convention; the paper does not state its
input format), to the summary field as the target.
Training procedure
The paper fine-tunes T5-small for 100,000 steps and gives no other SFT hyperparameters, so the run reuses those of its distillation runs (Table A.1: Adafactor, batch size 32, input/output length 1024/64, learning rate per Appendix A.3), with warmup and cooldown at the same proportions of the run (5% and the last 25%).
Dropout is the one deviation: T5 v1.1's default 0.1 instead of Table A.1's 0.0. With 0.0, plain fine-tuning overfits (T5-base: monitor perplexity 5.2 after epoch 1, 52 after epoch 8) and falls short of the paper's ROUGE-2.
| Hyperparameter | Value |
|---|---|
| Steps | 100,000 optimizer steps (16 epochs, the last one partial) |
| Optimizer | Adafactor (scale_parameter=False, relative_step=False, warmup_init=False), lr 1e-3 |
| Schedule | Warmup-stable-decay: linear warmup over 5,000 steps, constant, linear decay to 0 over the last 25,000 steps |
| Batch size | 8 per device x 2 devices x gradient accumulation 2 = 32 |
| Dropout | 0.1 |
| Gradient clipping | none |
| Precision | fp32 master weights, bf16 autocast |
| Hardware | 2 GPUs, DDP, bf16 mixed precision (accelerate launch) |
| Training time | ~34 min for the first epoch, ~9 h 5 min in total |
| Seed | 42 |
The full torchdistill config, CLI arguments and per-step metrics are in logs/: config.yaml,
args.json, metrics.json and trackio/summary.csv. To reproduce the run with torchdistill, use the command in the
header of logs/config.yaml.
Evaluation
ROUGE F-measure (x100) of greedy decoding (up to 64 new tokens) on the released, last checkpoint:
| Split | ROUGE-1 | ROUGE-2 | ROUGE-L |
|---|---|---|---|
| Validation (11,332) | 38.63 | 16.06 | 31.10 |
| Test (11,334) | 38.52 | 15.85 | 30.94 |
The paper reports a ROUGE-2 of 13.4 on the validation split after this stage. Both splits are held out from training and were not used to select the checkpoint. The scores come from the 2-process distributed evaluation of the training run.
Perplexity over the summary tokens of the monitor split, measured after each epoch:
| Epoch | Monitor perplexity |
|---|---|
| 1 | 8.8971 |
| 2 | 7.8231 |
| 3 | 7.3308 |
| 4 | 7.1183 |
| 5 | 6.9253 |
| 6 | 6.8334 |
| 7 | 6.7924 |
| 8 | 6.7822 |
| 9 | 6.7498 |
| 10 | 6.7033 |
| 11 | 6.6901 |
| 12 | 6.6955 |
| 13 | 6.7240 |
| 14 | 6.7181 |
| 15 | 6.7818 |
| 16 (released) | 6.8748 |
The monitor perplexity was lowest after epoch 11 (6.6901) and rose afterwards, but no checkpoint was selected on it (see above).
Limitations
- XSum's reference summaries often contain information that is not in the article, and models fine-tuned on it are known to hallucinate such content (Maynez et al., 2020). Summaries may be fluent but unfaithful.
- Trained on English BBC news articles; quality on other domains and languages is not measured.
- Inputs longer than 1,024 tokens were truncated in training; summaries are single sentences of up to 64 tokens.
- No downstream evaluation beyond ROUGE on XSum (e.g. factual-consistency metrics or human evaluation) has been run.
- Intended for research, in particular as a reference point for knowledge-distillation experiments.
License
The weights are released under the Apache License 2.0
(LICENSE).
This model is a modification of google/t5-small-lm-adapt by Google, which is licensed under the Apache License 2.0. The modification is full-parameter fine-tuning on the data described above.
The training data, XSum, consists of BBC articles collected by Narayan et al. (2018) for research and carries no explicit license on the Hugging Face Hub. Users are responsible for making sure their use of the model is consistent with the terms of the underlying data.
Citation
If you use this model, please cite torchdistill, as well as GKD, T5 and XSum:
@inproceedings{matsubara2021torchdistill,
title={torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation},
author={Matsubara, Yoshitomo},
booktitle={International Workshop on Reproducible Research in Pattern Recognition},
pages={24--44},
year={2021},
organization={Springer}
}
@inproceedings{matsubara2023torchdistill,
title={{torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP}},
author={Matsubara, Yoshitomo},
booktitle={Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)},
publisher={Empirical Methods in Natural Language Processing},
pages={153--164},
year={2023}
}
@inproceedings{agarwal2024onpolicy,
title={On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes},
author={Rishabh Agarwal and Nino Vieillard and Yongchao Zhou and Piotr Stanczyk and Sabela Ramos Garea and Matthieu Geist and Olivier Bachem},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024}
}
@article{raffel2020t5,
title={Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer},
author={Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu},
journal={Journal of Machine Learning Research},
volume={21},
number={140},
pages={1--67},
year={2020}
}
@inproceedings{lester2021power,
title={The Power of Scale for Parameter-Efficient Prompt Tuning},
author={Brian Lester and Rami Al-Rfou and Noah Constant},
booktitle={Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
pages={3045--3059},
year={2021}
}
@inproceedings{narayan2018xsum,
title={Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization},
author={Shashi Narayan and Shay B. Cohen and Mirella Lapata},
booktitle={Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
pages={1797--1807},
year={2018}
}
- Downloads last month
- 16
Model tree for yoshitomo-matsubara/t5_small-xsum
Base model
google/t5-small-lm-adaptDataset used to train yoshitomo-matsubara/t5_small-xsum
Paper for yoshitomo-matsubara/t5_small-xsum
Evaluation results
- ROUGE-1 on XSum (test split)test set self-reported38.520
- ROUGE-2 on XSum (test split)test set self-reported15.850
- ROUGE-L on XSum (test split)test set self-reported30.940
- ROUGE-1 on XSum (validation split)validation set self-reported38.630
- ROUGE-2 on XSum (validation split)validation set self-reported16.060
- ROUGE-L on XSum (validation split)validation set self-reported31.100
# Use a pipeline as a high-level helper # Warning: Pipeline type "summarization" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("summarization", model="yoshitomo-matsubara/t5_small-xsum")