antichronology/orthrus-4-track

Orthrus 4-track

10M-param Mamba RNA encoder, contrastive pretrained on 45M+ mature mRNA transcripts. 4-track one-hot input.

Model zoo

Repo	Tracks	Embed dim	Objective	Used in
antichronology/orthrus-4-track	4	512	contrastive	Nature Methods publication
antichronology/orthrus-6-track	6	512	contrastive	Nature Methods publication
antichronology/orthrus-small-6-track	6	256	contrastive	Nature Methods publication
antichronology/orthrus-mlm-6-track	6	512	contrastive + MLM	Nature Methods publication
quietflamingo/orthrus-base-4-track	4	256	contrastive	Pre-publication
quietflamingo/orthrus-large-4-track	4	512	contrastive	Pre-publication
quietflamingo/orthrus-large-6-track	6	512	contrastive	Pre-publication

Inference interface

Every Orthrus model exposes the same three inference methods plus a one-hot helper:

Method	Output shape	Notes
representation(x, lengths, channel_last=True)	(B, D)	Mean-pooled, padding-aware
representation_unpooled(x, channel_last=True)	(B, L, D)	Per-position hidden states
predict_tokens(x, lengths, channel_last=True)	(B, L, 4)	MLM logits over [A, C, G, T]. Available on MLM-pretrained repos (*-mlm-*); raises NotImplementedError on contrastive-only checkpoints.
seq_to_oh(seq)	(L, 4)	One-hot helper, ordering [A, C, G, T] (U is treated as T)

Environment setup

The model uses a Mamba state-space backbone, which requires CUDA. The recommended environment matches the Orthrus GitHub repo:

# Conda env with Python 3.10
mamba create -n orthrus python=3.10
mamba activate orthrus

# PyTorch + transformers + huggingface_hub
pip install 'torch>=2.2' 'transformers<4.46' 'huggingface_hub>=0.24' safetensors

# Mamba kernels (require CUDA; pin versions for the published checkpoints)
pip install causal-conv1d==1.2.0.post2 --no-build-isolation --no-cache-dir
pip install mamba-ssm==1.2.0.post1 --no-build-isolation --no-cache-dir

# GenomeKit, only if you want to build 6-track inputs from real transcripts
mamba install "genomekit>=6.0.0"
wget -O starter_build.sh https://raw.githubusercontent.com/deepgenomics/GenomeKit/main/starter/build.sh
chmod +x starter_build.sh
./starter_build.sh

Load the model

import torch
from transformers import AutoModel

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = AutoModel.from_pretrained(
    "antichronology/orthrus-4-track",
    trust_remote_code=True,
).to(device).eval()

Pooled representation (4-track input)

The 4-track input is just one-hot-encoded nucleotides:

sequence = (
    "TCATCTGGATTATACATATTTCGCAATGAAAGAGAGGAAGAAAAGGAAGCAGCAAAATATGTGGAGGCCCA"
    "ACAAAAGAGACTAGAAGCCTTATTCACTAAAATTCAGGAGGAATTTGAAGAACATGAAGTTACTTCCTCC"
)
oh = model.seq_to_oh(sequence).unsqueeze(0).to(device)  # (1, L, 4)
lengths = torch.tensor([oh.shape[1]], device=device)

with torch.no_grad():
    emb = model.representation(oh, lengths, channel_last=True)
# emb.shape == (1, D)

Un-pooled (per-position) representation

with torch.no_grad():
    hidden = model.representation_unpooled(oh, channel_last=True)
# hidden.shape == (1, L, D)
# Useful for: local scoring at a specific transcript position, attention
# probing, downstream sequence-tagging tasks.

Fine-tuning

Configuration files and training scripts for fine-tuning, linear probing, and homology-aware splitting live in the Orthrus GitHub repo. All fine-tuning data and pre-computed splits are mirrored on Zenodo.

Citation

@article{fradkinShi2026,
  title = {Orthrus: toward evolutionary and functional RNA foundation models},
  ISSN = {1548-7105},
  url = {http://dx.doi.org/10.1038/s41592-026-03064-3},
  DOI = {10.1038/s41592-026-03064-3},
  journal = {Nature Methods},
  publisher = {Springer Science and Business Media LLC},
  author = {Fradkin, Philip and Shi, Ruian "Ian" and Dalal, Taykhoom and Isaev, Keren and Frey, Brendan J. and Lee, Leo J. and Morris, Quaid and Wang, Bo},
  year = {2026},
  month = Apr
}

License

MIT