---
license: cc-by-sa-3.0
datasets:
- databricks/databricks-dolly-15k
language:
- en
base_model:
- EleutherAI/pythia-410m
pipeline_tag: text-generation
tags:
- pythia
- 410m
- dolly
- instruction-following
- if
- it
- instruction
- following
- AI
- text-generation-inference
- lora
- instructpythia
- instructpythia410mlora-v1
- slm
- llm
- finetune
- pile
- eleutherai
- rextro111
- rextro
---
# Pythia-410M Dolly LoRA on Modal

A compact instruction-tuning project that fine-tunes the standard, non-deduplicated
[`EleutherAI/pythia-410m`](https://huggingface.co/EleutherAI/pythia-410m) model on
[`databricks/databricks-dolly-15k`](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
using 16-bit LoRA and Modal GPU compute.

## Project summary

| Item | Value |
|---|---|
| Base model | `EleutherAI/pythia-410m` |
| Dataset | `databricks/databricks-dolly-15k` |
| Training method | FP16 LoRA |
| Maximum sequence length | 768 tokens |
| Epochs | 3 |
| Training GPU | Modal A10 |
| Inference GPU | Modal T4 |
| Completed optimizer steps | 2,760 |
| Measured training time | 52 minutes 40 seconds |
| Training throughput | 13.962 samples/second |
| Final adapter path | `/data/outputs/pythia-410m-dolly-lora/final-adapter` |

The project saves checkpoints, the final PEFT adapter, tokenizer files, and training
metadata to a persistent Modal Volume named `pythia-410m-dolly-output`.

## Results

Observed test generations:

### Simple explanation

**Prompt**

```text
Explain artificial intelligence in simple terms.
```

**Response**

```text
Artificial Intelligence (AI) is the term for a new generation of computer
systems that are capable of performing tasks without human assistance. This can
be done by combining artificial neural networks with computational intelligence
algorithms to automate and improve the efficiency of our everyday life. AI
technology has already been used in areas such as robotics, video games,
healthcare, and manufacturing.
```

### Closed factual question

**Prompt**

```text
What is the capital of France?
```

**Response**

```text
Paris.
```

### Creative-writing limitation

**Prompt**

```text
Write a short poem about a robot exploring the Moon.
```

**Response**

```text
I love to explore, it's the best way to meet new people.
```

The model performs well on simple explanations and basic factual questions, but
creative constraint-following remains unreliable at this model size.

## Project layout

```text
pythia-410m-dolly/
├── train.py
├── test_model.py
├── README.md
└── .venv/
```

Persistent files inside the Modal Volume:

```text
/data/
├── cache/
│   └── huggingface/
└── outputs/
    └── pythia-410m-dolly-lora/
        ├── checkpoint-*/
        ├── final-adapter/
        │   ├── adapter_config.json
        │   ├── adapter_model.safetensors
        │   ├── tokenizer.json
        │   ├── tokenizer_config.json
        │   └── special_tokens_map.json
        ├── trainer_state.json
        └── training_summary.json
```

## Requirements

- Python 3.11 or newer
- A Modal account
- Modal CLI authentication
- Internet access for the initial package and model downloads

## Local environment

```bash
mkdir -p ~/pythia-410m-dolly
cd ~/pythia-410m-dolly

python3 -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip
pip install --upgrade modal

modal setup
```

## Training configuration

The training script uses approximately the following settings:

```python
MODEL_NAME = "EleutherAI/pythia-410m"
DATASET_NAME = "databricks/databricks-dolly-15k"

MAX_LENGTH = 768
NUM_TRAIN_EPOCHS = 3
LEARNING_RATE = 2e-4

PER_DEVICE_TRAIN_BATCH_SIZE = 4
GRADIENT_ACCUMULATION_STEPS = 4

LORA_R = 16
LORA_ALPHA = 32
LORA_DROPOUT = 0.05
```

The effective training batch size is 16 examples per optimizer step.

The LoRA adapter targets Pythia/GPT-NeoX projection modules:

```python
target_modules = [
    "query_key_value",
    "dense",
]
```

Training labels mask the instruction and context tokens with `-100`, so loss is
calculated only over response tokens.

## Run training

Check the script before launching a paid GPU container:

```bash
python -m py_compile train.py
```

Start a detached Modal run:

```bash
modal run --detach train.py
```

Modal's `--detach` option keeps the remote application running if the local
process exits or disconnects.

## Persistent storage

The same Modal Volume cannot be mounted at two different paths in one function.
This project mounts it once:

```python
VOLUME_ROOT = "/data"

volumes = {
    VOLUME_ROOT: output_volume,
}
```

Outputs and caches then use separate subdirectories:

```python
OUTPUT_DIR = "/data/outputs/pythia-410m-dolly-lora"
CACHE_DIR = "/data/cache/huggingface"
```

## Test the adapter on Modal

Run the default prompt:

```bash
modal run test_model.py
```

Run a custom prompt:

```bash
modal run test_model.py \
  --prompt "Explain artificial intelligence in simple terms."
```

Run deterministic greedy decoding:

```bash
modal run test_model.py \
  --prompt "What is the capital of France?" \
  --temperature 0 \
  --max-new-tokens 80
```

Run sampled decoding:

```bash
modal run test_model.py \
  --prompt "Write exactly four short lines about a robot on the Moon." \
  --temperature 0.45 \
  --top-p 0.85 \
  --top-k 30 \
  --repetition-penalty 1.12 \
  --max-new-tokens 100
```

At inference time, the script loads the original base model and then attaches the
saved LoRA weights:

```python
base_model = AutoModelForCausalLM.from_pretrained(
    "EleutherAI/pythia-410m",
    torch_dtype=torch.float16,
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER_DIR,
    is_trainable=False,
)
```

## Download the trained adapter

```bash
mkdir -p downloaded-output

modal volume get \
  pythia-410m-dolly-output \
  outputs/pythia-410m-dolly-lora/final-adapter \
  downloaded-output/
```

Inspect downloaded files:

```bash
find downloaded-output -type f -printf '%p  %k KB\n'
```

## Training metrics

The completed run reported:

```text
train_runtime: 3160.6921 seconds
train_samples_per_second: 13.962
train_steps: 2760
epoch: 3.0
```

Late-run training losses were mostly around `1.90` to `2.14`. Gradient norms
were generally stable, mostly near `0.7` to `1.3`, with no obvious sign of
gradient explosion in the supplied logs.

## Limitations

- The model is small and may ignore multi-part or creative instructions.
- It can generate inaccurate or outdated factual claims.
- Dolly 15K is English-only and does not make the model reliably multilingual.
- LoRA instruction tuning does not replace the base model's knowledge.
- The model has not undergone dedicated safety alignment.
- Outputs should be reviewed before use in consequential settings.

## Dataset and model licensing

- Pythia-410M is published under the Apache 2.0 license.
- Dolly 15K is published under CC BY-SA 3.0.
- Review both upstream licenses before redistributing a merged model, adapter, or
  derivative dataset.
- Final model license: CC BY-SA 3.0.

This repository should not imply endorsement by EleutherAI, Databricks,
Hugging Face, or Modal.

## References

- [EleutherAI Pythia-410M model card](https://huggingface.co/EleutherAI/pythia-410m)
- [Databricks Dolly 15K dataset card](https://huggingface.co/datasets/databricks/databricks-dolly-15k)
- [Modal `run` CLI documentation](https://modal.com/docs/cli/latest/run)
- [Modal Volume documentation](https://modal.com/docs/guide/volumes)
- [Modal GPU documentation](https://modal.com/docs/guide/gpu)
- [Hugging Face PEFT model documentation](https://huggingface.co/docs/peft/en/package_reference/peft_model)