| --- |
| language: |
| - en |
| tags: |
| - ecology |
| - multimodal |
| - missing-modality |
| - masking |
| - species-distribution-modeling |
| - species-classification |
| library_name: pytorch |
| pipeline_tag: image-classification |
| datasets: |
| - MVRL/TaxaBench-8k |
| metrics: |
| - accuracy |
| - auc |
| license: mit |
| --- |
| |
| # MIAM: Modality Imbalance-Aware Masking for Multimodal Ecological Applications |
|
|
| This repository hosts released checkpoints for **MIAM** from the ICLR 2026 paper: |
|
|
| - Paper: https://openreview.net/forum?id=oljjAkgZN4 |
| - Project page: https://zbirobin.github.io/publications/miam/ |
| - Source code: https://github.com/zbirobin/MIAM |
|
|
| MIAM is a dynamic masking strategy for multimodal ecological learning. During training, it adapts masking probabilities using modality-specific performance and learning-speed signals to reduce modality imbalance and improve robustness to missing inputs. |
|
|
| ## Model details |
|
|
| - **Model family**: Multimodal ecological models trained with MIAM and masking baselines |
| - **Modalities covered**: |
| - GeoPlant (MaskSDM): satellite + environmental tabular + climate time series |
| - TaxaBench-8k: location + environmental tabular + natural image + audio + satellite |
| - SatBird: satellite + environmental tabular |
| - **Framework**: PyTorch |
|
|
| ## Available files |
|
|
| ### GeoPlant (MaskSDM) |
|
|
| - `geoplant_miam.pt` |
| - `geoplant_opm.pt` |
| - `geoplant_dropout.pt` |
| - `geoplant_constant.pt` |
| - `geoplant_dirichlet.pt` |
| - `geoplant_uniform.pt` |
| - `geoplant_pretraining_miam.pt` |
| - `geoplant_pretraining_opm.pt` |
| - `geoplant_pretraining_dropout.pt` |
| - `geoplant_pretraining_constant.pt` |
| - `geoplant_pretraining_dirichlet.pt` |
| - `geoplant_pretraining_uniform.pt` |
|
|
| ### TaxaBench-8k |
|
|
| - `taxabench_miam.pt` |
| - `taxabench_dirichlet.pt` |
| - `taxabench_dropout.pt` |
| - `taxabench_opm.pt` |
| - `taxabench_uniform.pt` |
| - `taxabench_embeds_loc_env_img_aud_sat.pt` |
| - `taxabench_num_species_10.csv` |
|
|
| ### SatBird |
|
|
| - `satbird_miam.pth` |
| - `satbird_opm.pth` |
| - `satbird_dropout.pth` |
| - `satbird_dirichlet.pth` |
|
|
| ## Quick start |
|
|
| ### Download with `hf` CLI |
|
|
| ```bash |
| hf download zbirobin/MIAM geoplant_miam.pt |
| hf download zbirobin/MIAM taxabench_miam.pt |
| hf download zbirobin/MIAM satbird_miam.pth |
| ``` |
|
|
| ### Download in Python |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| import torch |
| |
| ckpt_path = hf_hub_download(repo_id="zbirobin/MIAM", filename="geoplant_miam.pt") |
| state = torch.load(ckpt_path, map_location="cpu") |
| ``` |
|
|
| ## Intended use |
|
|
| - Research on multimodal ecological modeling |
| - Robustness to missing-modality settings |
| - Benchmark comparison of masking strategies |
| - Modality contribution analysis and interpretability studies |
|
|
| ## Out-of-scope use |
|
|
| - Safety-critical ecological decisions without domain validation |
| - Deployment outside the data distributions represented in GeoPlant, TaxaBench-8k, and SatBird |
| - Applications requiring calibrated uncertainty without additional validation |
|
|
| ## Training and evaluation data |
|
|
| - **GeoPlant**: species distribution modeling benchmark used in the paper |
| - **TaxaBench-8k**: multimodal species classification benchmark |
| - **SatBird-USA-summer**: bird species distribution benchmark |
|
|
| Please follow dataset licenses, terms of use, and any access restrictions from the original providers. |
|
|
| ## Limitations |
|
|
| - Transferability across regions/taxa/modalities may be limited |
|
|
| ## Reproducibility |
|
|
| Use the benchmark READMEs in the source repository for exact folder structure, commands, and evaluation scripts: |
|
|
| - `maskSDM/README.md` |
| - `taxabench/README.md` |
| - `satbird/README.md` |
|
|
| ## License |
|
|
| The model weights and associated files are licensed under the MIT License. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{ |
| zbinden2026miam, |
| title={{MIAM}: Modality Imbalance-Aware Masking for Multimodal Ecological Applications}, |
| author={Robin Zbinden and Wesley Monteith-Finas and Gencer Sumbul and Nina van Tiel and Chiara Vanalli and Devis Tuia}, |
| booktitle={International Conference on Learning Representations (ICLR)}, |
| year={2026}, |
| url={https://openreview.net/forum?id=oljjAkgZN4} |
| } |
| ``` |
|
|
| ## Contact |
|
|
| For issues and questions, please open a ticket in the source repository. |
