huggingbrain
/

Dinov3d-Neuro

+---
+library_name: pytorch            # raw PyTorch checkpoint + this repo's code, not a transformers/timm model — see §"How to Get Started". FILL_ME if you package it differently (e.g. wrap in a transformers custom-code repo).
+pipeline_tag: image-feature-extraction
+license: cc-by-nc-sa-4.0
+tags:
+  - mri
+  - brain-mri
+  - medical-imaging
+  - 3d
+  - volumetric
+  - self-supervised
+  - vision-transformer
+  - fomo300k
+datasets:
+  - FOMO-MRI/FOMO300K
+---
+# Dinov3D-Neuro: A 3D ViT-B/16³ for Brain MRI trained using DINOv3-style SSL
+> **Provenance statement.** This model uses the **DINOv3 architecture and self-supervised
+> training objective**, adapted to 3D volumetric input via the
+> [`asagilmore/dinov3d`](https://github.com/asagilmore/dinov3d) codebase, itself a fork of
+> [`facebookresearch/dinov3`](https://github.com/facebookresearch/dinov3). Dinov3D was **trained
+> from scratch on brain MRI data and contains no weights from Meta's DINOv3 checkpoints**. It
+> is not affiliated with, endorsed by, or sponsored by Meta.
+---
+## Model Details
+### Model Description
+A 3D Vision Transformer encoder pretrained with a DINOv3 self-supervised objective
+(DINO + iBOT + KoLeo) with Gram anchoring added in later training phases, and a
+final high resolution adaptation phase (In Progress!). The model was trained on single-channel
+brain MRI volumes from the FOMO300K dataset of various contrasts (T1w, T2w, FLAIR, ...).
+Given a preprocessed volume, the model returns a CLS token and a grid of patch tokens (792-dim each).
+- **Developed by:** Asa Gilmore, Dr. Ariel Rokem.
+- **Model type:** 3D Vision Transformer (self-supervised feature encoder)
+- **Architecture:** DINOv3 ViT, adapted to 3D — `vit3d_base`: embedding dim 792, 12 layers, 12
+  heads (head dim 66), MLP FFN (ratio 4), LayerScale init 1e-5, drop path rate 0.3, 3-axis RoPE
+  position embedding (base 100.0, `separate` per-axis coordinate normalization), **0 register
+  ("storage") tokens**, LayerNorm.
+- **Parameters:** ~93.7M (measured by instantiating `vit3d_base`)
+- **Input:** Single-channel ("grayscale") 3D volume, shape `1 × D × H × W`, resampled to
+  **1 mm isotropic** spacing and padded to a multiple of the patch size (16) per axis. Trained
+  with 128³ global / 48³ local crops (base pretrain + Gram-anchoring phases), later adapted to
+  192³ global / 64³ local crops in a high-resolution phase — see [Training Details](#training-details).
+- **Output:** 792-dimensional embeddings: 1 CLS token + `(D/16)·(H/16)·(W/16)` patch tokens (no
+  register tokens). `model.forward_features(x)` returns a dict including
+  `x_norm_clstoken` (`B×792`) and `x_norm_patchtokens` (`B×N×792`).
+- **Precision of released weights:** fp32 (as stored in the `.pth` checkpoint;
+  training itself used bf16 param dtype / fp32 reduce dtype under FSDP2 mixed precision).
+- **License:** CC BY-NC-SA 4.0 (inherited from FOMO300K — see [Licensing](#licensing-and-attribution)). FILL_ME — confirm this is the license you intend before publishing; it is not yet stated anywhere else in this repo.
+- **Trained from scratch:** Yes — no pretrained initialization of any kind
+### Model Sources
+- **Repository:** [`asagilmore/dinov3d`](https://github.com/asagilmore/dinov3d) (training/modeling code, this repo)
+- **Paper:** None — no technical report exists yet for this model. FILL_ME if one is written.
+- **Architecture reference:** [DINOv3](https://arxiv.org/abs/2508.10104) (Siméoni et al., 2025)
+### Relationship to DINOv3
+*Adapted code:* This model uses code adapted from
+[`facebookresearch/dinov3`](https://github.com/facebookresearch/dinov3), which is licensed
+under the [DINOv3 License](https://ai.meta.com/resources/models-and-libraries/dinov3-license).
+**weights** are original work trained from scratch on brain MRI and are licensed separately
+under CC BY-NC-SA 4.0.
+**Modifications made for 3D:**
+| Component | 2D DINOv3 | This model |
+|---|---|---|
+| Patch embedding | Conv2d, 16×16 | Conv3d, 16×16×16 (`in_chans=1`, single-channel MRI) |
+| Positional encoding | 2-axis RoPE | 3-axis (D/H/W) axial RoPE, base θ = 100.0, `separate` coordinate normalization |
+| Register tokens | 4 | 0 (`n_storage_tokens: 0`) |
+| Global/local crops | 2D multi-crop | 3D multi-crop: 128³ global / 48³ local (base + Gram phases), later 192³ / 64³ (high-res phase); 8 local crops per global crop |
+| Color augmentation | color jitter, RGB mean/std normalization | none — single-channel intensities are z-score normalized once during preprocessing instead |
+| Objective terms | DINO + iBOT + Koleo + Gram anchoring | Same four terms, applied in stages — see [Training Details](#training-details) |
+Data augmentation has also been reworked with domain specific augmentations for brain MRI,
+more details on data augmentation can be found in the code repo.
+---
+## Uses
+Evaluation and downstream adaptation is still in progress, check back later to
+see more details.
+## How to Get Started with the Model
+This model is distributed as a raw PyTorch checkpoint plus the code in this repo, as
+well as sharded checkpoints at the end of each training phase.
+The teacher checkpoints can be loaded following the inference instructions in the repo,
+and the sharded checkpoints can be used to restart training for fine-tuning experiments.
+The teacher checkpoints can be found in the eval folder. We include all checkpoints captured
+during training, but reccomend using the latest one, unless doing experiments to
+evaluate performance over training iterations.
+The ckpt directory contains the sharded checkpoints, which we include at the end of
+each training phase.
+---
+## Training Details
+### Training Data
+**Dataset:** [FOMO-MRI/FOMO300K](https://huggingface.co/datasets/FOMO-MRI/FOMO300K)
+(Cerri et al., 2026) — a large-scale heterogeneous corpus of brain MRI aggregated from public
+sources, spanning T1, MPRAGE, T2, T2*, FLAIR, SWI, T1c, PD, DWI, and ADC sequences, and
+including both clinical- and research-grade acquisitions. It is a superset of OpenMind and
+FOMO50K/FOMO60K.
+We filtered the fomo300k dataset to include only single channel anatomical scans.
+The repo contains a fomo300k.json file listing all subjects used.
+### Preprocessing
+FOMO300K is distributed as NIfTI without co-registration or skull-stripping.
+- Resampling: 1 mm³ isotropic (`Spacingd`, bilinear/trilinear interpolation, `border` padding),
+  applied at load time via `InferenceAugmentation3d` / the training data pipeline
+- Padding: to the nearest multiple of patch size (16) per axis, filled with the volume's
+  minimum intensity value (`DivisiblePadd(..., mode="minimum")`)
+- Intensity normalization: per-volume z-score (`NormalizeIntensityd(nonzero=True,
+  channel_wise=True)`), plus empty-signal filling (`SignalFillEmptyd`) — done once ahead of
+  time when building the preprocessed dataset (`scripts/preproccess_fomo300k.py`), not
+  per-forward-pass
+- Augmentations (training only): 3D multi-crop (128³ global / 48³ local scaled up to 192³/64³
+  in the high-res phase; crop scale ranges 0.32–1.0 global, 0.05–0.32 local; 8 local crops per
+  sample), random flips. No color jitter (single-channel intensities, not RGB).
+### Training Hyperparameters
+All training hyperparameters can be found in the config.yaml file in this HF repo.