NeuroVFM

NeuroVFM is a volumetric visual foundation model for clinical neuroimaging developed by the MLiNS Lab at the University of Michigan and described in a Nature Medicine paper. It was trained with self-supervision on 5.24 million MRI and CT volumes from 566,915 studies collected over two decades of routine care at Michigan Medicine. The code is MIT-licensed on GitHub, and the pretrained weights are hosted on Hugging Face under CC-BY-NC-SA-4.0. The authors state it is a research tool only and not approved by the FDA.

What It Is

NeuroVFM is a stack of models for interpreting brain, head, neck, face and orbit MRI and CT studies. A 3D vision transformer encoder, pretrained with a method the authors call Vol-JEPA, turns each study into latent visual tokens. Those embeddings feed study-level diagnostic heads covering 74 MRI and 82 CT diagnoses, or a findings language model that generates preliminary findings from a study plus clinical context. The outputs can then be passed to an external reasoning model for tasks such as triage.

How the Workflow Runs

The Python pipeline loads an encoder, a diagnostic head and a vision-language model from Hugging Face. A study folder is preprocessed, embedded, and either classified into diagnoses with probabilities or used to generate findings text. The findings model pairs the frozen encoder with Qwen3-14B through a Perceiver-style connector. Repository scripts and configs cover Vol-JEPA pretraining, diagnostic head training with multiple instance learning, and findings model fine-tuning. FlashAttention-2 v2.6.3 is required for training and inference.

Reported Results

The paper reports a macro-averaged AUROC of about 92.7 on CT and 92.5 on MRI across 156 diagnostic tasks on a temporally held-out cohort, outperforming the baselines it compared against. In a blinded expert review, the authors report that NeuroVFM-generated findings had about half the key-finding error rate of GPT-5 (10% versus 20%). In a one-week prospective silent feasibility study of 1,155 studies, the NeuroVFM-based triage arm reached 92.6% balanced accuracy versus 71.2% for GPT-5, with 86.5% sensitivity for critical findings. The authors note it is not suited as an autonomous screening tool.

Access and Limits

Some weights require access approval with an institutional email. Raw patient imaging data is not public. The authors say the model is susceptible to bias from its training data and must not be used for clinical decision-making.