Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?

• URL: http://arxiv.org/abs/2605.19196v1
• Date: Mon, 18 May 2026 23:55:08 GMT
• Title: Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
• Abstract summary: We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
• Score: 61.49434544687523
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep research agents increasingly automate complex information-seeking tasks, producing evidence-grounded reports via multi-step reasoning, tool use, and synthesis. Their growing role demands scalable, reliable evaluation, positioning LLM-as-judge as a supervision paradigm for assessing factual accuracy, evidence use, and reasoning quality. Yet the reliability of these judges for deep research agents remains poorly understood, posing a critical meta-evaluation problem: before deploying LLM judges to supervise research agents, we must first evaluate the judges themselves. Existing meta-evaluations fall short in two ways: (1) reliance on coarse, subjective human-preference agreement; (2) focus on instruction-following or verifiable tasks, leaving open-ended agent executions unexplored. To address these gaps, we introduce REFLECT (REliable Fine-grained LLM judge Evaluation via Controlled inTervention), a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments. REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions on quality-screened agent execution traces. This yields verifiable, comprehensive, and fine-grained instances for validating the judge models. Our experiments show that current LLM judges remain unreliable: even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures, with especially poor performance on evidence verification. Together, our taxonomy and findings expose systematic judge limitations, reveal tradeoffs in cost and reliability, and offer actionable guidance for building more reliable evaluation pipelines for deep research agents.

Related papers

• Counsel: A Meta-Evaluation Dataset for Agentic Tasks [8.10022820678836]
  Counsel is the first public dataset of meta-evaluations for agentic tasks.<n>It consists of process-level critiques from open-weight LLMJs on two agent benchmarks.<n>Human annotators label critiques on each flagged error as "spot on", "correct location but poor reasoning", or "should not have flagged"
  arXiv  Detail & Related papers  (2026-06-19T17:30:56Z)
• From Blind Guess to Informed Judgment: Teaching LLMs to Evaluate Materials by Building Knowledge-Augmented Preference Signals [13.95417871252485]
  MaterEval produces evaluations that follow expert rules and provide supporting evidence, and a rule-removed blind guess.<n>We introduce a fast-slow reasoning scheme that decouples large-scale rapid screening from in-depth review on a small subset.<n>Results show that expert rules can be systematically transformed into learnable preference signals, enabling a low-cost and deployable evaluation module.
  arXiv  Detail & Related papers  (2026-05-28T08:09:35Z)
• CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems [19.083967725767387]
  CompliBench is a novel benchmark designed to evaluate the ability of LLM judges to detect and localize guideline violations in multi-turn dialogues.<n>We develop a scalable, automated data generation pipeline that simulates user-agent interactions.<n>Our controllable flaw injection process automatically yields precise ground-truth labels for the violated guideline and the exact conversation turn.
  arXiv  Detail & Related papers  (2026-04-14T05:42:41Z)
• DREAM: Deep Research Evaluation with Agentic Metrics [21.555357444628044]
  We propose DREAM (Deep Research Evaluation with Agentic Metrics), a framework that makes evaluation itself agentic.<n> DREAM structures assessment through an evaluation protocol combining query-agnostic metrics with adaptive metrics generated by a tool-calling agent.<n>Controlled evaluations demonstrate DREAM is significantly more sensitive to factual and temporal decay than existing benchmarks.
  arXiv  Detail & Related papers  (2026-02-21T19:14:31Z)
• TRACE: Trajectory-Aware Comprehensive Evaluation for Deep Research Agents [51.30998248590416]
  Trajectory-Aware Comprehensive Evaluation (TRACE) is a framework that holistically assesses the entire problem-solving trajectory.<n>Our contributions include the TRACE framework, its novel metrics, and the accompanying DeepResearch-Bench with controllable complexity.
  arXiv  Detail & Related papers  (2026-02-05T13:28:57Z)
• MAS-ProVe: Understanding the Process Verification of Multi-Agent Systems [59.20800753428596]
  We present MAS-ProVe, a systematic empirical study of process verification for multi-agent systems (MAS)<n>Our study spans three verification paradigms (LLM-as-a-Judge, reward models, and process reward models)<n>We find that process-level verification does not consistently improve performance and frequently exhibits high variance.
  arXiv  Detail & Related papers  (2026-02-03T03:30:36Z)
• FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights [63.32178443510396]
  We introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings.<n>Even the strongest agents achieve limited rediscovery success (50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning.
  arXiv  Detail & Related papers  (2026-02-02T23:21:13Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• Verifiability-First Agents: Provable Observability and Lightweight Audit Agents for Controlling Autonomous LLM Systems [0.0]
  We propose a Verifiability-First architecture that integrates run-time attestations of agent actions using cryptographic and symbolic methods.<n>We also embed Audit Agents that continuously verify intent versus behavior using constrained reasoning.<n>Our approach shifts the evaluation focus from how likely misalignment is to how quickly and reliably misalignment can be detected and remediated.
  arXiv  Detail & Related papers  (2025-12-19T06:12:43Z)
• Towards Evaluting Fake Reasoning Bias in Language Models [47.482898076525494]
  We show that models favor the surface structure of reasoning even when the logic is flawed.<n>We introduce THEATER, a benchmark that systematically investigates Fake Reasoning Bias (FRB)<n>We evaluate 17 advanced Large Language Models (LRMs) on both subjective DPO and factual datasets.
  arXiv  Detail & Related papers  (2025-07-18T09:06:10Z)
• CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards [72.44810390478229]
  CompassJudger-2 is a novel generalist judge model that overcomes limitations via a task-driven, multi-domain data curation strategy.<n> CompassJudger-2 achieves superior results across multiple judge and reward benchmarks.
  arXiv  Detail & Related papers  (2025-07-12T01:34:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.