LoRA adapters for RoBERTa-base on GLUE (a reproduction)
Five small sets of weights, one per GLUE task (SST-2, MRPC, CoLA, RTE, STS-B), trained with a
from-scratch implementation of Low-Rank Adaptation (Hu et al., 2021)
on top of a frozen roberta-base (FacebookAI/roberta-base).
Each task folder contains:
| File | Content |
|---|---|
<task>/adapter.safetensors |
LoRA A and B matrices for the query and value projections of all 12 attention layers (r = 8, alpha = 8; 294,912 parameters) and the trained classification head (592,130 parameters). |
<task>/config.json |
Base model, task, rank, alpha, target matrices, metric, maximum sequence length and the source run (seed, best epoch, dev score). |
The classification head is part of every adapter file. It is randomly initialised and trained
together with LoRA, and it is not part of the paper's trainable-parameter count. The weights are
stored under the module names of the Hugging Face RoBERTa model; they are not in the Hugging
Face PEFT format. Load them with the lora package of the code repository (snippet below).
Code, training configurations, all 170 runs and the full report: https://github.com/bartuonder/LoRA-Low-Rank-Adaptation-of-Large-Language-Models
Which runs these are
For every task the repository holds five seeds (0 to 4). The uploaded adapter is the run whose best-epoch dev score is the median of the five. On MRPC two seeds (1 and 2) tie at the median; seed 1 is uploaded. Per-task settings (learning rate, batch size, epochs) follow Table 9 of the paper: SST-2 5e-4 / 16 / 60, MRPC 4e-4 / 16 / 30, CoLA 4e-4 / 32 / 80, RTE 5e-4 / 32 / 80, STS-B 4e-4 / 16 / 40. Other settings: AdamW (weight decay 0.1, betas 0.9 and 0.98), linear schedule with 6 % warm-up, gradient clipping 1.0, maximum sequence length 512, bf16 autocast, LoRA dropout 0. The weights are the ones of the best dev epoch.
Results against the paper
GLUE dev (validation) split, median over 5 seeds (mean ± std next to it); published numbers are the RoBERTa-base LoRA row of Table 2 of the paper.
| Task | Metric | Paper (published) | Mine: mean ± std | Mine: median | Δ (median − paper) | Uploaded adapter (seed, best epoch, dev score) |
|---|---|---|---|---|---|---|
| SST-2 | Accuracy | 95.1 ± 0.2 | 94.8 ± 0.2 | 94.8 | -0.3 | seed 2, epoch 21, 94.84 |
| MRPC | Accuracy | 89.7 ± 0.7 | 89.2 ± 0.5 | 89.2 | -0.5 | seed 1, epoch 11, 89.22 |
| CoLA | Matthews corr. | 63.4 ± 1.2 | 63.0 ± 1.0 | 63.1 | -0.3 | seed 2, epoch 19, 63.07 |
| RTE | Accuracy | 86.6 ± 0.7 | 78.6 ± 0.5 | 78.7 | -7.9 | seed 3, epoch 56, 78.70 |
| STS-B | Pearson corr. | 91.5 ± 0.2 | 91.1 ± 0.1 | 91.0 | -0.5 | seed 3, epoch 31, 91.05 |
Every run is scored at its best dev epoch, as in the paper's protocol, so the dev numbers are selected on the split they are reported on.
Note on RTE (and MRPC, STS-B). According to Appendix D.1 of the paper, the LoRA modules for
MRPC, RTE and STS-B start from the best MNLI checkpoint. The adapters here start from the
pre-trained roberta-base instead, with no MNLI stage. This is the most likely reason for the
large RTE gap (78.7 against 86.6); it is a hypothesis, not a verified cause. Treat the RTE (and,
to a lesser degree, MRPC and STS-B) adapters as results from the plain pre-trained start.
Loading an adapter
Run from a clone of the code repository with src/ on the Python path (or after installing it).
import json
import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from lora.data import load_glue
from lora.inject import inject_lora
from lora.trainer import evaluate
task = "rte" # sst2, mrpc, cola, rte or stsb
root = snapshot_download("Bartu0nder/lora-roberta-base-glue")
cfg = json.load(open(f"{root}/{task}/config.json"))
tok = AutoTokenizer.from_pretrained(cfg["base_model"])
model = AutoModelForSequenceClassification.from_pretrained(
cfg["base_model"], num_labels=cfg["num_labels"], attn_implementation="sdpa")
inject_lora(model, targets=cfg["target_modules"], r=cfg["r"], alpha=cfg["alpha"])
state = load_file(f"{root}/{task}/adapter.safetensors") # LoRA A/B and classifier head
result = model.load_state_dict(state, strict=False)
assert not result.unexpected_keys
assert not [k for k in result.missing_keys if "lora_" in k or k.startswith("classifier.")]
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
_, dev = load_glue(task, tok, cfg["max_seq_len"]) # GLUE validation split
score = evaluate(model, dev, {"task": task, "metric": cfg["metric"], "precision": "bf16"},
device, tok.pad_token_id)
print(task, round(score, 2), "expected", cfg["dev_score"])
The snippet evaluates with bf16 autocast, the setting used in training, and reproduces the dev
scores recorded for these runs (dev_score in each config.json). Evaluating in fp32 instead
changes the small dev sets by a few examples: RTE gives 77.98 and CoLA 62.83 in that case; the
other three tasks are unchanged to two decimals.
License and citation
MIT. If you use these weights, please cite the original paper:
@article{hu2021lora,
title = {LoRA: Low-Rank Adaptation of Large Language Models},
author = {Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu},
journal = {arXiv preprint arXiv:2106.09685},
year = {2021}
}
Model tree for Bartu0nder/lora-roberta-base-glue
Base model
FacebookAI/roberta-base