Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders

Katarina Trojachanec Dineva, Stefan Andonov, Ilinka Ivanoska, Ivan Kitanovski, Sasho Gramatikov, Monika Simjanoska Misheva, Kostadin Mishev

Abstract. Vision-enabled large language models (V-LLMs) are increasingly explored for clinical decision support, yet their reliability in neuroimaging-based reasoning remains under-evaluated. This benchmark evaluates 20 vision-enabled LLMs across MRI and CT neuroimaging tasks, including diagnosis identification, subtype classification, modality recognition, and anatomical plane detection. We assess accuracy, calibration, abstention behavior, robustness, and computational efficiency under a unified, reproducible prompting protocol, comparing zero-shot and few-shot approaches.

What is this benchmark?

Large-scale evaluation of 20 vision-enabled LLMs (proprietary and open-source)
Neuroimaging tasks spanning diagnosis, modality, sequence, and anatomical plane
Diseases include multiple sclerosis, stroke, and brain tumors
Safety-aware metrics: calibration (ECE, Brier), abstention, and error analysis
Efficiency and cost evaluation alongside predictive performance

Figure 1. Overview of the neuroimaging benchmark pipeline and evaluation framework.

Evaluation protocol

Figure 2. Three-phase evaluation design: exploratory calibration, scaled development, and final evaluation.

Phase 1: Prompt calibration on stratified subsets and temperature sweeps
Phase 2: Scaled development evaluation to select top-performing models
Phase 3: Final benchmarking using zero-shot and few-shot inference

Explore the Interactive Leaderboard →

Funding. This research is fully funded by the ChatMed Project, an EU-funded initiative focused on advancing trustworthy, transparent, and clinically grounded AI systems for medical decision support.

Citation

@article{dineva2025neurollm,
  title   = {Evaluation of Vision-Enabled Large Language Models for Clinical Reasoning in Neurological Disorders},
  author  = {Dineva, Katarina T. and Andonov, Stefan and Ivanoska, Ilinka and Kitanovski, Ivan and Gramatikov, Sasho and Simjanoska Misheva, Monika and Mishev, Kostadin},
  year    = {2025}
}