ConfProBench: A Confidence Evaluation Benchmark for MLLM-Based Process Judges

• URL: http://arxiv.org/abs/2508.04576v1
• Date: Wed, 06 Aug 2025 16:00:19 GMT
• Title: ConfProBench: A Confidence Evaluation Benchmark for MLLM-Based Process Judges
• Abstract summary: We evaluate 14 state-of-the-art MLLMs, including both proprietary and open-source models.<n>We propose ConfProBench, the first comprehensive benchmark designed to systematically evaluate the reliability of step-level confidence scores generated by MPJs.
• Score: 15.47711837051754
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reasoning is a critical capability of multimodal large language models (MLLMs) for solving complex multimodal tasks, and judging the correctness of reasoning steps is crucial for improving this capability. Recently, MLLM-based process judges (MPJs) have been widely used to assess the correctness of reasoning steps in multimodal tasks. Therefore, evaluating MPJs is important for identifying their limitations and guiding future improvements. However, existing benchmarks for MPJs mainly focus on tasks such as step correctness classification and reasoning process search, while overlooking a key aspect: whether the confidence scores produced by MPJs at the step level are reliable. To address this gap, we propose ConfProBench, the first comprehensive benchmark designed to systematically evaluate the reliability of step-level confidence scores generated by MPJs. Our benchmark constructs three types of adversarially perturbed reasoning steps: Synonym Substitution, Syntactic Transformation, and Image Perturbation, to test the robustness of MPJ confidence under perturbations. In addition, we introduce three novel evaluation metrics: Confidence Robustness Score (CRS), Confidence Sensitivity Score (CSS), and Confidence Calibration Score (CCS), which evaluate robustness, sensitivity, and calibration, respectively. We evaluate 14 state-of-the-art MLLMs, including both proprietary and open-source models. Experiments reveal limitations in current MPJs' confidence performance and offer competitive baselines to support future research.

Related papers

• A Calibrated Reflection Approach for Enhancing Confidence Estimation in LLMs [3.9353006440879525]
  Calibrated Reflection is a framework that combines structured reasoning with distance-aware calibration technique.<n>We evaluate our framework on diverse datasets, including HelpSteer2, Llama T-REx, and a proprietary conversational dataset.
  arXiv  Detail & Related papers  (2026-09-03T22:52:01Z)
• Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs [60.80435138270317]
  We apply reinforcement learning with metacognitive feedback to the problem of faithful calibration.<n>Experiments show RLMF achieves generalizable, state-of-the-art FC on diverse tasks.<n> RLMF surpasses standard RL by up to 63% while enhancing models' ability to assess and express their own capability limits.
  arXiv  Detail & Related papers  (2026-06-30T17:56:01Z)
• Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate [2.1987782243719898]
  We investigate whether intrinsic confidence signals, token-level log-probabilities from decoding, can predict reasoning quality.<n>We find that early-token confidence, particularly within the first few generated tokens, is consistently the strongest predictor of reasoning quality.
  arXiv  Detail & Related papers  (2026-06-09T01:52:59Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• Process Supervision of Confidence Margin for Calibrated LLM Reasoning [52.373121066425455]
  Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability.<n>Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation.<n>We introduce Reinforcement Learning with Confidence Margin (textbfRLCM), a calibration-aware RL framework that jointly optimize correctness and confidence reliability.
  arXiv  Detail & Related papers  (2026-04-25T14:40:13Z)
• Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation [20.16938320120462]
  M-JudgeBench is a capability-oriented benchmark to comprehensively assess the judgment abilities of MLLMs.<n>Judge-MCTS is a data construction framework generating pairwise reasoning trajectories with various correctness and length.<n>Our work establishes a more principled foundation for evaluating MLLM-as-a-judge through M-JudgeBench and Judge-MCTS framework.
  arXiv  Detail & Related papers  (2026-02-28T08:49:22Z)
• Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations [49.84786015324238]
  Confidence estimation (CE) indicates how reliable the answers of large language models (LLMs) are, and can impact user trust and decision-making.<n>We present a comprehensive evaluation framework for CE that measures their confidence quality on three new aspects.<n>These include robustness of confidence against prompt perturbations, stability across semantic equivalent answers, and sensitivity to semantically different answers.
  arXiv  Detail & Related papers  (2026-01-12T23:16:50Z)
• Self-Evaluating LLMs for Multi-Step Tasks: Stepwise Confidence Estimation for Failure Detection [1.1087735229999818]
  Self-evaluating large language models (LLMs) provides meaningful confidence estimates in complex reasoning.<n>Stepwise evaluation generally outperforms holistic scoring in detecting potential errors.
  arXiv  Detail & Related papers  (2025-11-10T18:19:51Z)
• BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents [58.05949210993854]
  We investigate whether search agents have the ability to communicate their own confidence through verbalized confidence scores after long sequences of actions.<n>We propose Test-Time Scaling (TTS) methods that use confidence scores to determine answer quality, encourage the model to try again until reaching a satisfactory confidence level.
  arXiv  Detail & Related papers  (2025-10-27T15:58:51Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generation [63.49409574310576]
  Large language models (LLMs) exhibit overconfidence, assigning high confidence scores to incorrect predictions.<n>We introduce FineCE, a novel confidence estimation method that delivers accurate, fine-grained confidence scores during text generation.<n>Our code and all baselines used in the paper are available on GitHub.
  arXiv  Detail & Related papers  (2025-08-16T13:29:35Z)
• MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs [66.14178164421794]
  We introduce MetaFaith, a novel prompt-based calibration approach inspired by human metacognition.<n>We show that MetaFaith robustly improves faithful calibration across diverse models and task domains, enabling up to 61% improvement in faithfulness.
  arXiv  Detail & Related papers  (2025-05-30T17:54:08Z)
• MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibration [2.1824579248418017]
  We present MMBoundary, a novel framework that advances the knowledge boundary awareness of MLLMs through reasoning step confidence calibration.<n>In addition to supervised fine-tuning, we introduce a reinforcement learning stage with multiple reward functions for further aligning model knowledge.<n> Empirical results show that MMBoundary significantly outperforms existing methods across diverse domain datasets and metrics.
  arXiv  Detail & Related papers  (2025-05-29T08:14:40Z)
• ProJudge: A Multi-Modal Multi-Discipline Benchmark and Instruction-Tuning Dataset for MLLM-based Process Judges [13.957207630090064]
  We introduce ProJudgeBench, the first benchmark specifically designed for evaluating abilities of MLLM-based process judges.<n>ProJudgeBench comprises 2,400 test cases and 50,118 step-level labels, spanning four scientific disciplines.<n> Evaluation on ProJudgeBench reveals a significant performance gap between open-source and proprietary models.
  arXiv  Detail & Related papers  (2025-03-09T10:55:51Z)
• SteerConf: Steering LLMs for Confidence Elicitation [11.872504642312705]
  Large Language Models (LLMs) exhibit impressive performance across diverse domains but often suffer from overconfidence.<n>We propose SteerConf, a novel framework that systematically steers LLMs' confidence scores to improve their calibration and reliability.
  arXiv  Detail & Related papers  (2025-03-04T18:40:49Z)
• MCQA-Eval: Efficient Confidence Evaluation in NLG with Gold-Standard Correctness Labels [16.300463494913593]
  Large Language Models (LLMs) require robust confidence estimation.<n>McQCA-Eval is an evaluation framework for assessing confidence measures in Natural Language Generation.
  arXiv  Detail & Related papers  (2025-02-20T05:09:29Z)
• How Reliable are LLMs as Knowledge Bases? Re-thinking Facutality and Consistency [60.25969380388974]
  Large Language Models (LLMs) are increasingly explored as knowledge bases (KBs)<n>Current evaluation methods focus too narrowly on knowledge retention, overlooking other crucial criteria for reliable performance.<n>We propose new criteria and metrics to quantify factuality and consistency, leading to a final reliability score.
  arXiv  Detail & Related papers  (2024-07-18T15:20:18Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making.<n>We present a process-based benchmark MR-Ben that demands a meta-reasoning skill.<n>Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)
• UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questions [10.28688988951815]
  We introduce UBench, a new benchmark for evaluating the uncertainty of large language models (LLMs)<n>Unlike other benchmarks, UBench is based on confidence intervals. It encompasses 11,978 multiple-choice questions spanning knowledge, language, understanding, and reasoning capabilities.<n>Our analysis reveals several crucial insights: 1) Our confidence interval-based methods are highly effective for uncertainty quantification; 2) Regarding uncertainty, outstanding open-source models show competitive performance versus closed-source models; 3) CoT and RP prompts present potential ways to improve model reliability, while the influence of temperature changes follows no universal rule.
  arXiv  Detail & Related papers  (2024-06-18T16:50:38Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• When to Trust LLMs: Aligning Confidence with Response Quality [49.371218210305656]
  We propose CONfidence-Quality-ORDer-preserving alignment approach (CONQORD) It integrates quality reward and order-preserving alignment reward functions. Experiments demonstrate that CONQORD significantly improves the alignment performance between confidence and response accuracy.
  arXiv  Detail & Related papers  (2024-04-26T09:42:46Z)
• Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection [90.71323430635593]
  We propose a novel self-detection paradigm that considers the comprehensive answer space beyond LLM-generated answers. Building upon this paradigm, we introduce a two-step framework, which firstly instructs LLM to reflect and provide justifications for each candidate answer. This framework can be seamlessly integrated with existing approaches for superior self-detection.
  arXiv  Detail & Related papers  (2024-03-15T02:38:26Z)
• TrustScore: Reference-Free Evaluation of LLM Response Trustworthiness [58.721012475577716]
  Large Language Models (LLMs) have demonstrated impressive capabilities across various domains, prompting a surge in their practical applications. This paper introduces TrustScore, a framework based on the concept of Behavioral Consistency, which evaluates whether an LLMs response aligns with its intrinsic knowledge.
  arXiv  Detail & Related papers  (2024-02-19T21:12:14Z)
• Self-Evaluation Improves Selective Generation in Large Language Models [54.003992911447696]
  We reformulate open-ended generation tasks into token-level prediction tasks. We instruct an LLM to self-evaluate its answers. We benchmark a range of scoring methods based on self-evaluation.
  arXiv  Detail & Related papers  (2023-12-14T19:09:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.