Upload README.md with huggingface_hub

8f1355d verified 2 months ago

6.01 kB

---
library_name: pytorch
pipeline_tag: image-feature-extraction
license: cc-by-nc-sa-4.0
tags:
  - mri
  - brain-mri
  - medical-imaging
  - 3d
  - volumetric
  - self-supervised
  - vision-transformer
  - fomo300k
datasets:
  - FOMO-MRI/FOMO300K
---
# Dinov3D-Neuro: A 3D ViT-B/16^3 for Brain MRI trained using DINOv3-style SSL

> This model uses the **DINOv3 architecture and self-supervised
> training objective**, adapted to 3D volumetric input via the
> [`nrdg/dinov3d`](https://github.com/nrdg/dinov3d) codebase, itself a fork of
> [`facebookresearch/dinov3`](https://github.com/facebookresearch/dinov3). Dinov3D was **trained
> from scratch on brain MRI data and contains no weights from Meta's DINOv3 checkpoints**. It
> is not affiliated with, endorsed by, or sponsored by Meta.

---

## Model Details

### Model Description

A 3D Vision Transformer encoder pretrained with a DINOv3 self-supervised objective
(DINO + iBOT + KoLeo) with Gram anchoring added in later training phases, and a
final high resolution adaptation phase (In Progress!). The model was trained on single-channel
brain MRI volumes from the FOMO300K dataset of various contrasts (T1w, T2w, FLAIR, ...).
Given a preprocessed volume, the model returns a CLS token and a grid of patch tokens (792-dim each).

- **Developed by:** Asa Gilmore, Dr. Ariel Rokem.
- **Model type:** 3D Vision Transformer (self-supervised feature encoder)
- **Architecture:** DINOv3 ViT, adapted to 3D - See `vit3d_base` in dinov3d repo
- **Input:** Single-channel 3D volume, resampled to
  1 mm isotropic spacing and padded to a multiple of the patch size (16). Trained
  with 128^3 global / 48^3 local crops (base pretrain + Gram-anchoring phases), later adapted to
  192^3 global / 64^3 local crops in a high-resolution phase - see [Training Details](#training-details).
- **Output:** 792-dimensional embeddings: 1 CLS token + `(D/16)·(H/16)·(W/16)` patch tokens (no
  register tokens). `model.forward_features(x)` returns a dict including
  `x_norm_clstoken` (`B×792`) and `x_norm_patchtokens` (`B×N×792`).
- **License:** CC BY-NC-SA 4.0 (inherited from FOMO300K - see [Licensing](#licensing-and-attribution))

### Model Sources

- **Repository:** [`asagilmore/dinov3d`](https://github.com/nrdg/dinov3d) (training/modeling code)
- **Architecture reference:** [DINOv3](https://arxiv.org/abs/2508.10104) (Siméoni et al., 2025)

### Relationship to DINOv3

*Adapted code:* This model uses code adapted from
[`facebookresearch/dinov3`](https://github.com/facebookresearch/dinov3), which is licensed
under the [DINOv3 License](https://ai.meta.com/resources/models-and-libraries/dinov3-license).

**weights** are original work trained from scratch on brain MRI and are licensed separately
under CC BY-NC-SA 4.0.

**Modifications made for 3D:**

| Component | 2D DINOv3 | This model |
|---|---|---|
| Patch embedding | Conv2d, 16×16 | Conv3d, 16×16×16 (`in_chans=1`, single-channel MRI) |
| Positional encoding | 2-axis RoPE | 3-axis (D/H/W) axial RoPE |
| Register tokens | 4 | 0 |
| Global/local crops | 2D multi-crop | 3D multi-crop: 128^3 global / 48^3 local (base + Gram phases), later 192^3 / 64^3 (high-res phase); 8 local crops per global crop |
| Color augmentation | color jitter, RGB mean/std normalization | none - single-channel intensities are z-score normalized once during preprocessing instead |
| Objective terms | DINO + iBOT + Koleo + Gram anchoring | Same four terms, applied in stages - see [Training Details](#training-details) |

Data augmentation has also been reworked with domain specific augmentations for brain MRI,
more details on data augmentation can be found in the code repo.

---

## Uses

Evaluation and downstream adaptation is still in progress, check back later to
see more details.

## How to Get Started with the Model

This model is distributed as a raw PyTorch checkpoint plus the code in this repo, as
well as sharded checkpoints at the end of each training phase.

The teacher checkpoints can be loaded following the inference instructions in the repo,
and the sharded checkpoints can be used to restart training for fine-tuning experiments.

The teacher checkpoints can be found in the eval folder. We include all checkpoints captured
during training, but recommend using the latest one, unless doing experiments to
evaluate performance over training iterations.

The ckpt directory contains the sharded checkpoints, which we include at the end of
each training phase.

The layout of the checkpoints follows from the original dinov3 outputs, so their repo can
be used as a rough reference for layout.

---

## Training Details

### Training Data

**Dataset:** [FOMO-MRI/FOMO300K](https://huggingface.co/datasets/FOMO-MRI/FOMO300K)
(Cerri et al., 2026)

We filtered the fomo300k dataset to include only single channel anatomical scans.
The repo contains a fomo300k.json file listing all subjects used.

### Preprocessing

FOMO300K is distributed as NIfTI without co-registration or skull-stripping.

- Resampling: 1 mm^3 isotropic (`Spacingd`, bilinear/trilinear interpolation, `border` padding),
  applied at load time via `InferenceAugmentation3d` / the training data pipeline
- Padding: to the nearest multiple of patch size (16) per axis, filled with the volume's
  minimum intensity value (`DivisiblePadd(..., mode="minimum")`)
- Intensity normalization: per-volume z-score (`NormalizeIntensityd(nonzero=True,
  channel_wise=True)`), plus empty-signal filling (`SignalFillEmptyd`) - done once ahead of
  time when building the preprocessed dataset (`scripts/preproccess_fomo300k.py`), not
  per-forward-pass
- Augmentations (training only): 3D multi-crop (128^3 global / 48^3 local scaled up to 192^3/64^3
  in the high-res phase; crop scale ranges 0.32-1.0 global, 0.05–0.32 local; 8 local crops per
  sample), random flips. No color jitter (single-channel intensities, not RGB).

### Training Hyperparameters

All training hyperparameters can be found in the config.yaml file in this HF repo.

	---
	library_name: pytorch
	pipeline_tag: image-feature-extraction
	license: cc-by-nc-sa-4.0
	tags:
	- mri
	- brain-mri
	- medical-imaging
	- 3d
	- volumetric
	- self-supervised
	- vision-transformer
	- fomo300k
	datasets:
	- FOMO-MRI/FOMO300K
	---

	# Dinov3D-Neuro: A 3D ViT-B/16^3 for Brain MRI trained using DINOv3-style SSL

	> This model uses the **DINOv3 architecture and self-supervised
	> training objective**, adapted to 3D volumetric input via the
	> [`nrdg/dinov3d`](https://github.com/nrdg/dinov3d) codebase, itself a fork of
	> [`facebookresearch/dinov3`](https://github.com/facebookresearch/dinov3). Dinov3D was **trained
	> from scratch on brain MRI data and contains no weights from Meta's DINOv3 checkpoints**. It
	> is not affiliated with, endorsed by, or sponsored by Meta.

	---

	## Model Details

	### Model Description

	A 3D Vision Transformer encoder pretrained with a DINOv3 self-supervised objective
	(DINO + iBOT + KoLeo) with Gram anchoring added in later training phases, and a
	final high resolution adaptation phase (In Progress!). The model was trained on single-channel
	brain MRI volumes from the FOMO300K dataset of various contrasts (T1w, T2w, FLAIR, ...).
	Given a preprocessed volume, the model returns a CLS token and a grid of patch tokens (792-dim each).

	- Developed by: Asa Gilmore, Dr. Ariel Rokem.
	- Model type: 3D Vision Transformer (self-supervised feature encoder)
	- Architecture: DINOv3 ViT, adapted to 3D - See `vit3d_base` in dinov3d repo
	- Input: Single-channel 3D volume, resampled to
	1 mm isotropic spacing and padded to a multiple of the patch size (16). Trained
	with 128^3 global / 48^3 local crops (base pretrain + Gram-anchoring phases), later adapted to
	192^3 global / 64^3 local crops in a high-resolution phase - see [Training Details](#training-details).
	- Output: 792-dimensional embeddings: 1 CLS token + `(D/16)·(H/16)·(W/16)` patch tokens (no
	register tokens). `model.forward_features(x)` returns a dict including
	`x_norm_clstoken` (`B×792`) and `x_norm_patchtokens` (`B×N×792`).
	- License: CC BY-NC-SA 4.0 (inherited from FOMO300K - see [Licensing](#licensing-and-attribution))

	### Model Sources

	- Repository: [`asagilmore/dinov3d`](https://github.com/nrdg/dinov3d) (training/modeling code)
	- Architecture reference: [DINOv3](https://arxiv.org/abs/2508.10104) (Siméoni et al., 2025)

	### Relationship to DINOv3

	Adapted code: This model uses code adapted from
	[`facebookresearch/dinov3`](https://github.com/facebookresearch/dinov3), which is licensed
	under the [DINOv3 License](https://ai.meta.com/resources/models-and-libraries/dinov3-license).

	weights are original work trained from scratch on brain MRI and are licensed separately
	under CC BY-NC-SA 4.0.

	Modifications made for 3D:

	\| Component \| 2D DINOv3 \| This model \|
	\|---\|---\|---\|
	\| Patch embedding \| Conv2d, 16×16 \| Conv3d, 16×16×16 (`in_chans=1`, single-channel MRI) \|
	\| Positional encoding \| 2-axis RoPE \| 3-axis (D/H/W) axial RoPE \|
	\| Register tokens \| 4 \| 0 \|
	\| Global/local crops \| 2D multi-crop \| 3D multi-crop: 128^3 global / 48^3 local (base + Gram phases), later 192^3 / 64^3 (high-res phase); 8 local crops per global crop \|
	\| Color augmentation \| color jitter, RGB mean/std normalization \| none - single-channel intensities are z-score normalized once during preprocessing instead \|
	\| Objective terms \| DINO + iBOT + Koleo + Gram anchoring \| Same four terms, applied in stages - see [Training Details](#training-details) \|

	Data augmentation has also been reworked with domain specific augmentations for brain MRI,
	more details on data augmentation can be found in the code repo.

	---

	## Uses

	Evaluation and downstream adaptation is still in progress, check back later to
	see more details.

	## How to Get Started with the Model

	This model is distributed as a raw PyTorch checkpoint plus the code in this repo, as
	well as sharded checkpoints at the end of each training phase.

	The teacher checkpoints can be loaded following the inference instructions in the repo,
	and the sharded checkpoints can be used to restart training for fine-tuning experiments.

	The teacher checkpoints can be found in the eval folder. We include all checkpoints captured
	during training, but recommend using the latest one, unless doing experiments to
	evaluate performance over training iterations.

	The ckpt directory contains the sharded checkpoints, which we include at the end of
	each training phase.

	The layout of the checkpoints follows from the original dinov3 outputs, so their repo can
	be used as a rough reference for layout.

	---

	## Training Details

	### Training Data

	Dataset: [FOMO-MRI/FOMO300K](https://huggingface.co/datasets/FOMO-MRI/FOMO300K)
	(Cerri et al., 2026)

	We filtered the fomo300k dataset to include only single channel anatomical scans.
	The repo contains a fomo300k.json file listing all subjects used.

	### Preprocessing

	FOMO300K is distributed as NIfTI without co-registration or skull-stripping.

	- Resampling: 1 mm^3 isotropic (`Spacingd`, bilinear/trilinear interpolation, `border` padding),
	applied at load time via `InferenceAugmentation3d` / the training data pipeline
	- Padding: to the nearest multiple of patch size (16) per axis, filled with the volume's
	minimum intensity value (`DivisiblePadd(..., mode="minimum")`)
	- Intensity normalization: per-volume z-score (`NormalizeIntensityd(nonzero=True,
	channel_wise=True)`), plus empty-signal filling (`SignalFillEmptyd`) - done once ahead of
	time when building the preprocessed dataset (`scripts/preproccess_fomo300k.py`), not
	per-forward-pass
	- Augmentations (training only): 3D multi-crop (128^3 global / 48^3 local scaled up to 192^3/64^3
	in the high-res phase; crop scale ranges 0.32-1.0 global, 0.05–0.32 local; 8 local crops per
	sample), random flips. No color jitter (single-channel intensities, not RGB).

	### Training Hyperparameters

	All training hyperparameters can be found in the config.yaml file in this HF repo.