AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classification

• URL: http://arxiv.org/abs/2605.05573v1
• Date: Thu, 07 May 2026 01:36:27 GMT
• Title: AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classification
• Abstract summary: We introduce AstroAlertBench, a comprehensive benchmark designed to evaluate large language models (LLMs) for astronomical event review.<n>We use a pilot sample of 1,500 real-world alerts from the Zwicky Transient Facility (ZTF), a wide-field survey that scans the northern sky to detect transient astronomical events.<n>Our results reveal that high accuracy does not always align with model honesty'', defined as the ability to self-evaluate its reasoning.
• Score: 6.546448267229169
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern astronomical observatories generate a massive volume of multimodal data, creating a critical bottleneck for expert human review. While multimodal large language models (LLMs) have shown promise in interpreting complex visual and textual inputs, their ability to perform specialized scientific classification while providing interpretable reasoning remains understudied. We introduce AstroAlertBench, a comprehensive multimodal benchmark designed to evaluate LLM performance in astronomical event review along a three-stage logical chain: metadata grounding, scientific reasoning, and hierarchical classification over five categories. We use a pilot sample of 1,500 real-world alerts from the Zwicky Transient Facility (ZTF), a wide-field survey that scans the northern sky to detect transient astronomical events. On this dataset, we benchmark 13 frontier closed-source and open-weight LLMs that support visual input. Our results reveal that high accuracy does not always align with model ``honesty,'' defined as the ability to self-evaluate its reasoning, which affects its reliability as a real-world assistant. We further initialize a human-in-the-loop evaluation protocol as a precursor to future community-scale participation. Together, AstroAlertBench provides a framework for developing calibrated and interpretable astronomical assistants.

Related papers

• Hyrax: An Extensible Framework for Rapid ML Experimentation and Unsupervised Discovery in the Era of Rubin, Roman, and Euclid [26.715408970731332]
  We present Hyrax, an open-source, modular, GPU-enabled Python framework that supports the full machine learning lifecycle in astronomy.<n>We demonstrate Hyrax's versatility through five representative applications on real survey data.
  arXiv  Detail & Related papers  (2026-05-18T18:00:06Z)
• From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs [65.04549036809557]
  We introduce a benchmark built from pedestrian-perspective videos captured with synchronized stereo cameras, LiDAR, and IMU/GPS sensors.<n>This dataset provides metrically precise 3D information, enabling the automatic generation of spatial reasoning questions.<n> Evaluations reveal that the performance gains observed in structured indoor benchmarks vanish in open-world settings.
  arXiv  Detail & Related papers  (2025-12-22T18:58:12Z)
• Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks [108.15756345836901]
  We provide a comprehensive review of multimodal spatial reasoning tasks with large models.<n>We review advances in embodied AI, including vision-language navigation and action models.<n>We consider emerging modalities such as audio and egocentric video, which contribute to novel spatial understanding through new sensors.
  arXiv  Detail & Related papers  (2025-10-29T17:55:43Z)
• StarEmbed: Benchmarking Time Series Foundation Models on Astronomical Observations of Variable Stars [12.329789568475045]
  Time series foundation models (TSFMs) are increasingly being adopted as highly-capable general-purpose time series representation learners.<n>We introduce StarEmbed, the first public benchmark for rigorous and standardized evaluation of state-of-the-art TSFMs on stellar time series observations.<n>We benchmark on three scientifically-motivated downstream tasks: unsupervised clustering, supervised classification, and out-of-distribution source detection.
  arXiv  Detail & Related papers  (2025-10-07T17:53:56Z)
• AstroMMBench: A Benchmark for Evaluating Multimodal Large Language Models Capabilities in Astronomy [6.247581175023764]
  We introduce AstroMMBench, the first comprehensive benchmark to evaluate multimodal large language models (MLLMs) in astronomical image understanding.<n>AstroMMBench comprises 621 multiple-choice questions across six astrophysical subfields, curated and reviewed by 15 domain experts for quality and relevance.<n>Results show that Ovis2-34B achieved the highest overall accuracy (70.5%), demonstrating leading capabilities even compared to strong closed-source models.
  arXiv  Detail & Related papers  (2025-09-29T09:02:30Z)
• Radio Astronomy in the Era of Vision-Language Models: Prompt Sensitivity and Adaptation [5.711705587813085]
  Vision-Language Models (VLMs) are positioned as general-purpose AI systems capable of reasoning across domains.<n>We assess whether generic VLMs, presumed to lack exposure to astronomical corpora, can perform morphology-based classification of radio galaxies.
  arXiv  Detail & Related papers  (2025-08-31T14:31:47Z)
• Time-RA: Towards Time Series Reasoning for Anomaly with LLM Feedback [55.284574165467525]
  Time-series Reasoning for Anomaly (Time-RA) transforms classical time series anomaly detection into a generative, reasoning-intensive task.<n>Also, we introduce the first real-world multimodal benchmark dataset, RATs40K, explicitly annotated for anomaly reasoning.
  arXiv  Detail & Related papers  (2025-07-20T18:02:50Z)
• EarthMind: Leveraging Cross-Sensor Data for Advanced Earth Observation Interpretation with a Unified Multimodal LLM [103.7537991413311]
  Earth Observation (EO) data analysis is vital for monitoring environmental and human dynamics.<n>Recent Multimodal Large Language Models (MLLMs) show potential in EO understanding but remain restricted to single-sensor inputs.<n>We propose EarthMind, a unified vision-language framework that handles both single- and cross-sensor inputs.
  arXiv  Detail & Related papers  (2025-06-02T13:36:05Z)
• AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomy [39.94582666929051]
  We introduce AstroVisBench, the first benchmark for both scientific computing and visualization in the astronomy domain.<n>We present an evaluation of state-of-the-art language models, showing a significant gap in their ability to engage in astronomy research as useful assistants.
  arXiv  Detail & Related papers  (2025-05-26T21:49:18Z)
• TimeXL: Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop [79.5773512667468]
  TimeXL is a multi-modal prediction framework that integrates a prototype-based time series encoder with three collaborating Large Language Models.<n>A reflection LLM compares the predicted values against the ground truth, identifying textual inconsistencies or noise.<n>This closed-loop workflow-prediction, critique (reflect), and refinement-continuously boosts the framework's performance and interpretability.
  arXiv  Detail & Related papers  (2025-03-02T20:40:53Z)
• AstroM$^3$: A self-supervised multimodal model for astronomy [0.0]
  We propose AstroM$3$, a self-supervised pre-training approach that enables a model to learn from multiple modalities simultaneously. Specifically, we extend the CLIP (Contrastive Language-Image Pretraining) model to a trimodal setting, allowing the integration of time-series photometry data, spectra, and astrophysical metadata. Results demonstrate that CLIP pre-training improves classification performance for time-series photometry, where accuracy increases from 84.6% to 91.5%.
  arXiv  Detail & Related papers  (2024-11-13T18:20:29Z)
• Real-time gravitational-wave inference for binary neutron stars using machine learning [71.29593576787549]
  We present a machine learning framework that performs complete BNS inference in just one second without making any approximations. Our approach enhances multi-messenger observations by providing (i) accurate localization even before the merger; (ii) improved localization precision by $sim30%$ compared to approximate low-latency methods; and (iii) detailed information on luminosity distance, inclination, and masses.
  arXiv  Detail & Related papers  (2024-07-12T18:00:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.