Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraints

• URL: http://arxiv.org/abs/2604.15664v1
• Date: Fri, 17 Apr 2026 03:38:50 GMT
• Title: Stargazer: A Scalable Model-Fitting Benchmark Environment for AI Agents under Astrophysical Constraints
• Abstract summary: We introduce Stargazer, a scalable environment for evaluating AI agents on dynamic, iterative physics-grounded model-fitting tasks.<n>Stargazer comprises 120 tasks across three difficulty tiers, including 20 real archival cases.<n>Our evaluation of eight frontier agents reveals a gap between numerical optimization and adherence to physical constraints.
• Score: 48.80158223838359
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rise of autonomous AI agents suggests that dynamic benchmark environments with built-in feedback on scientifically grounded tasks are needed to evaluate the capabilities of these agents in research work. We introduce Stargazer, a scalable environment for evaluating AI agents on dynamic, iterative physics-grounded model-fitting tasks using inference on radial-velocity (RV) time series data. Stargazer comprises 120 tasks across three difficulty tiers, including 20 real archival cases, covering diverse scenarios ranging from high-SNR single-planet systems to complex multi-planetary configurations requiring involved low-SNR analysis. Our evaluation of eight frontier agents reveals a gap between numerical optimization and adherence to physical constraints: although agents often achieve a good statistical fit, they frequently fail to recover correct physical system parameters, a limitation that persists even when agents are equipped with vanilla skills. Furthermore, increasing test-time compute yields only marginal gains, with excessive token usage often reflecting recursive failure loops rather than meaningful exploration. Stargazer presents an opportunity to train, evaluate, scaffold, and scale strategies on a model-fitting problem of practical research relevance today. Our methodology to design a simulation-driven environment for AI agents presumably generalizes to many other model-fitting problems across scientific domains. Source code and the project website are available at https://github.com/Gudmorning2025/Stargazer and https://gudmorning2025.github.io/Stargazer, respectively.

Related papers

• MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agents [7.0922719251235655]
  MetaSpace is a novel framework designed to evaluate the spatial cognition of embodied agents.<n>We show that MetaSpace successfully detects 90,422 spatial cognition errors in state-of-the-art (SOTA) MLLM-driven agents.<n>Results indicate that all SOTA agents achieve average scores between 0.44 and 0.52, significantly lower than the human benchmark of 0.96.
  arXiv  Detail & Related papers  (2026-07-26T14:07:20Z)
• Benchmarking AI Agents for Addressing Scientific Challenges Across Scales [118.2204632627895]
  SciAgentArena is a systematic benchmark for evaluating AI agents in real-world scientific research scenarios.<n>It comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents.<n>We find that current agents can contribute effectively to well-specified data-analysis, particularly when the task structure and evaluation criteria are clear.
  arXiv  Detail & Related papers  (2026-06-10T22:55:30Z)
• gwBenchmarks: Stress-Testing LLM Agents on High-Precision Gravitational Wave Astronomy [2.316169407465118]
  Modern gravitational wave astronomy relies on modeling tasks that often require months of graduate-level effort.<n>We study whether state-of-the-art LLM coding agents can perform such end-to-end scientific modeling.<n>We introduce gwBenchmarks, a suite of eight tasks grounded in gravitational wave analytic calculations and numerical simulations collectively representing over $108$ core-hours of compute.
  arXiv  Detail & Related papers  (2026-05-11T21:47:22Z)
• Exploration and Exploitation Errors Are Measurable for Language Model Agents [52.95061000593404]
  Language Model (LM) agents are increasingly used in complex openended decision tasks.<n>A core requirement is the ability to both explore the problem space and exploit acquired knowledge effectively.<n>We design controllable environments inspired by practical embodied AI scenarios.
  arXiv  Detail & Related papers  (2026-04-14T17:59:57Z)
• Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning [62.499592503950026]
  Large language model (LLM) have empowered autonomous agents to perform complex tasks that require multi-turn interactions with tools and environments.<n>We propose Agent World Model (AWM), a fully synthetic environment generation pipeline.<n>We scale to 1,000 environments covering everyday scenarios, in which agents can interact with rich toolsets.
  arXiv  Detail & Related papers  (2026-02-10T18:55:41Z)
• Agentic AI in Remote Sensing: Foundations, Taxonomy, and Emerging Systems [9.388162021920206]
  This survey presents the first comprehensive review of agentic AI in remote sensing.<n>We introduce a unified taxonomy distinguishing between single-agent copilots and multi-agent systems.<n>We review emerging benchmarks that move the evaluation from pixel-level accuracy to trajectory-aware reasoning correctness.
  arXiv  Detail & Related papers  (2026-01-05T08:34:17Z)
• SciML Agents: Write the Solver, Not the Solution [69.5021018644143]
  We introduce two new datasets: a diagnostic dataset of adversarial "misleading" problems; and a large-scale benchmark of 1,000 diverse ODE tasks.<n>We evaluate open- and closed-source LLM models along two axes: (i) unguided versus guided prompting with domain-specific knowledge; and (ii) off-the-shelf versus fine-tuned variants.<n>Preliminary results indicate that careful prompting and fine-tuning can yield a specialized LLM agent capable of reliably solving simple ODE problems.
  arXiv  Detail & Related papers  (2025-09-12T02:53:57Z)
• STARec: An Efficient Agent Framework for Recommender Systems via Autonomous Deliberate Reasoning [54.28691219536054]
  We introduce STARec, a slow-thinking augmented agent framework that endows recommender systems with autonomous deliberative reasoning capabilities.<n>We develop anchored reinforcement training - a two-stage paradigm combining structured knowledge distillation from advanced reasoning models with preference-aligned reward shaping.<n>Experiments on MovieLens 1M and Amazon CDs benchmarks demonstrate that STARec achieves substantial performance gains compared with state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-08-26T08:47:58Z)
• Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning [29.605396813225386]
  We show how reinforcement learning can be used to train agents for multi-turn interactive tasks.<n>Our methodology offers a practical approach for training capable agents for multi-turn interactive tasks using open-weight models.
  arXiv  Detail & Related papers  (2025-08-05T14:30:47Z)
• Evaluating Temporal Observation-Based Causal Discovery Techniques Applied to Road Driver Behaviour [6.980076213134384]
  We evaluate 10 contemporary observational temporal causal discovery methods in the domain of autonomous driving. By evaluating these methods upon causal scenes drawn from real world datasets in addition to those generated synthetically we highlight where improvements need to be made. We discuss potential directions for future work that could help better tackle the difficulties currently experienced by state of the art techniques.
  arXiv  Detail & Related papers  (2023-01-31T19:55:24Z)
• Low-Thrust Orbital Transfer using Dynamics-Agnostic Reinforcement Learning [0.0]
  This study uses model-free Reinforcement Learning to train an agent on a constrained pericenter raising scenario for a low-thrust medium-Earth-orbit satellite. The trained agent is then used to design a trajectory and to autonomously control the satellite during the cruise.
  arXiv  Detail & Related papers  (2022-10-06T08:36:35Z)
• Exploring Dynamic Context for Multi-path Trajectory Prediction [33.66335553588001]
  We propose a novel framework, named Dynamic Context Network (DCENet) In our framework, the spatial context between agents is explored by using self-attention architectures. A set of future trajectories for each agent is predicted conditioned on the learned spatial-temporal context.
  arXiv  Detail & Related papers  (2020-10-30T13:39:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.