LoRA adapters for RoBERTa-base on GLUE (a reproduction)

Five small sets of weights, one per GLUE task (SST-2, MRPC, CoLA, RTE, STS-B), trained with a from-scratch implementation of Low-Rank Adaptation (Hu et al., 2021) on top of a frozen roberta-base (FacebookAI/roberta-base).

Each task folder contains:

File Content
<task>/adapter.safetensors LoRA A and B matrices for the query and value projections of all 12 attention layers (r = 8, alpha = 8; 294,912 parameters) and the trained classification head (592,130 parameters).
<task>/config.json Base model, task, rank, alpha, target matrices, metric, maximum sequence length and the source run (seed, best epoch, dev score).

The classification head is part of every adapter file. It is randomly initialised and trained together with LoRA, and it is not part of the paper's trainable-parameter count. The weights are stored under the module names of the Hugging Face RoBERTa model; they are not in the Hugging Face PEFT format. Load them with the lora package of the code repository (snippet below).

Code, training configurations, all 170 runs and the full report: https://github.com/bartuonder/LoRA-Low-Rank-Adaptation-of-Large-Language-Models

Which runs these are

For every task the repository holds five seeds (0 to 4). The uploaded adapter is the run whose best-epoch dev score is the median of the five. On MRPC two seeds (1 and 2) tie at the median; seed 1 is uploaded. Per-task settings (learning rate, batch size, epochs) follow Table 9 of the paper: SST-2 5e-4 / 16 / 60, MRPC 4e-4 / 16 / 30, CoLA 4e-4 / 32 / 80, RTE 5e-4 / 32 / 80, STS-B 4e-4 / 16 / 40. Other settings: AdamW (weight decay 0.1, betas 0.9 and 0.98), linear schedule with 6 % warm-up, gradient clipping 1.0, maximum sequence length 512, bf16 autocast, LoRA dropout 0. The weights are the ones of the best dev epoch.

Results against the paper

GLUE dev (validation) split, median over 5 seeds (mean ± std next to it); published numbers are the RoBERTa-base LoRA row of Table 2 of the paper.

Task Metric Paper (published) Mine: mean ± std Mine: median Δ (median − paper) Uploaded adapter (seed, best epoch, dev score)
SST-2 Accuracy 95.1 ± 0.2 94.8 ± 0.2 94.8 -0.3 seed 2, epoch 21, 94.84
MRPC Accuracy 89.7 ± 0.7 89.2 ± 0.5 89.2 -0.5 seed 1, epoch 11, 89.22
CoLA Matthews corr. 63.4 ± 1.2 63.0 ± 1.0 63.1 -0.3 seed 2, epoch 19, 63.07
RTE Accuracy 86.6 ± 0.7 78.6 ± 0.5 78.7 -7.9 seed 3, epoch 56, 78.70
STS-B Pearson corr. 91.5 ± 0.2 91.1 ± 0.1 91.0 -0.5 seed 3, epoch 31, 91.05

Every run is scored at its best dev epoch, as in the paper's protocol, so the dev numbers are selected on the split they are reported on.

Note on RTE (and MRPC, STS-B). According to Appendix D.1 of the paper, the LoRA modules for MRPC, RTE and STS-B start from the best MNLI checkpoint. The adapters here start from the pre-trained roberta-base instead, with no MNLI stage. This is the most likely reason for the large RTE gap (78.7 against 86.6); it is a hypothesis, not a verified cause. Treat the RTE (and, to a lesser degree, MRPC and STS-B) adapters as results from the plain pre-trained start.

Loading an adapter

Run from a clone of the code repository with src/ on the Python path (or after installing it).

import json

import torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoModelForSequenceClassification, AutoTokenizer

from lora.data import load_glue
from lora.inject import inject_lora
from lora.trainer import evaluate

task = "rte"                                  # sst2, mrpc, cola, rte or stsb
root = snapshot_download("Bartu0nder/lora-roberta-base-glue")
cfg = json.load(open(f"{root}/{task}/config.json"))

tok = AutoTokenizer.from_pretrained(cfg["base_model"])
model = AutoModelForSequenceClassification.from_pretrained(
    cfg["base_model"], num_labels=cfg["num_labels"], attn_implementation="sdpa")
inject_lora(model, targets=cfg["target_modules"], r=cfg["r"], alpha=cfg["alpha"])

state = load_file(f"{root}/{task}/adapter.safetensors")      # LoRA A/B and classifier head
result = model.load_state_dict(state, strict=False)
assert not result.unexpected_keys
assert not [k for k in result.missing_keys if "lora_" in k or k.startswith("classifier.")]

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

_, dev = load_glue(task, tok, cfg["max_seq_len"])             # GLUE validation split
score = evaluate(model, dev, {"task": task, "metric": cfg["metric"], "precision": "bf16"},
                 device, tok.pad_token_id)
print(task, round(score, 2), "expected", cfg["dev_score"])

The snippet evaluates with bf16 autocast, the setting used in training, and reproduces the dev scores recorded for these runs (dev_score in each config.json). Evaluating in fp32 instead changes the small dev sets by a few examples: RTE gives 77.98 and CoLA 62.83 in that case; the other three tasks are unchanged to two decimals.

License and citation

MIT. If you use these weights, please cite the original paper:

@article{hu2021lora,
  title   = {LoRA: Low-Rank Adaptation of Large Language Models},
  author  = {Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu},
  journal = {arXiv preprint arXiv:2106.09685},
  year    = {2021}
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Bartu0nder/lora-roberta-base-glue

Adapter
(329)
this model

Dataset used to train Bartu0nder/lora-roberta-base-glue

Paper for Bartu0nder/lora-roberta-base-glue