Do Reasoning Representations Help Humans Evaluate LLM Outputs?

• URL: http://arxiv.org/abs/2609.09038v1
• Date: Tue, 08 Sep 2026 17:03:57 GMT
• Title: Do Reasoning Representations Help Humans Evaluate LLM Outputs?
• Abstract summary: We study reasoning representations as human-facing interfaces rather than proxies for model reasoning ability.<n>We conduct a controlled human study of six reasoning formats across tasks of varying complexity.<n>Our study shows a mismatch between perceived preference and support for human evaluation.
• Score: 7.900903824992017
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reasoning representations are increasingly used as explanations for large language model outputs. Yet they are typically evaluated with model-centric criteria, such as answer accuracy and faithfulness, leaving it unclear whether they help people evaluate model responses. In this work, we study reasoning representations as human-facing interfaces rather than proxies for model reasoning ability. We conduct a controlled human study of six reasoning formats across tasks of varying complexity, supported by a web-based framework that randomizes task domains, problem instances, and representation order. The study collects fine-grained judgments of structural understanding, error detection and localization, and trust calibration. Our study shows a mismatch between perceived preference and support for human evaluation. Participants prefer planning- and decomposition-based representations, but simpler chain-of-thought traces better support verification, trust, and interpretability. Preferred representations also introduce calibration risks, with more false alarms on correct traces and high trust despite low willingness to verify.

Related papers

• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Decoding the Critique Mechanism in Large Reasoning Models [50.821607345799386]
  Large Reasoning Models (LRMs) exhibit backtracking and self-verification mechanisms that enable them to revise intermediate steps and reach correct solutions.<n>We investigate how current LRMs recover from errors by inserting arithmetic mistakes in their intermediate reasoning steps.<n>We find that despite the error propagating through the chain-of-thought, the model still reaches the correct final answer.
  arXiv  Detail & Related papers  (2026-03-17T10:03:30Z)
• Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbations [1.8772057593980798]
  Large Language Models (LLMs) can produce verbalized self-explanations.<n>We evaluate how well humans and LLM judges can predict a model's answers to counterfactual follow-up questions.
  arXiv  Detail & Related papers  (2026-01-07T10:13:26Z)
• TRACE: A Framework for Analyzing and Enhancing Stepwise Reasoning in Vision-Language Models [9.607579442309639]
  We introduce TRACE, a framework for Transparent Reasoning And Consistency Evaluation.<n>At its core, TRACEleverages Auxiliary Reasoning Sets to decompose complex problems.<n>Our experiments show that consistency across ARS correlates with final-answer correctness.<n>TRACE defines confidence regions that distinguish reliable from unreliable reasoning paths.
  arXiv  Detail & Related papers  (2025-12-05T18:40:18Z)
• A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models [58.32070787537946]
  Chain-of-thought (CoT) reasoning enhances performance of large language models.<n>We present the first comprehensive study of CoT faithfulness in large vision-language models.
  arXiv  Detail & Related papers  (2025-05-29T18:55:05Z)
• The Third Pillar of Causal Analysis? A Measurement Perspective on Causal Representations [23.129188507631284]
  Causal reasoning and discovery often face challenges due to the complexity, noisiness, and high-dimensionality of real-world data.<n>What makes learned representations useful for causal downstream tasks and how to evaluate them are still not well understood.
  arXiv  Detail & Related papers  (2025-05-23T10:25:17Z)
• On The Role of Reasoning in the Identification of Subtle Stereotypes in Natural Language [0.03749861135832073]
  Large language models (LLMs) are trained on vast, uncurated datasets that contain various forms of biases and language reinforcing harmful stereotypes. It is essential to examine and address biases in language models, integrating fairness into their development to ensure that these models do not perpetuate social biases. This work firmly establishes reasoning as a critical component in automatic stereotype detection and is a first step towards stronger stereotype mitigation pipelines for LLMs.
  arXiv  Detail & Related papers  (2023-07-24T15:12:13Z)
• ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning [63.77667876176978]
  Large language models show improved downstream task interpretability when prompted to generate step-by-step reasoning to justify their final answers. These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness is difficult. We present ROS, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics.
  arXiv  Detail & Related papers  (2022-12-15T15:52:39Z)
• Explain, Edit, and Understand: Rethinking User Study Design for Evaluating Model Explanations [97.91630330328815]
  We conduct a crowdsourcing study, where participants interact with deception detection models that have been trained to distinguish between genuine and fake hotel reviews. We observe that for a linear bag-of-words model, participants with access to the feature coefficients during training are able to cause a larger reduction in model confidence in the testing phase when compared to the no-explanation control.
  arXiv  Detail & Related papers  (2021-12-17T18:29:56Z)
• Odd-One-Out Representation Learning [1.6822770693792826]
  We show that a weakly-supervised downstream task based on odd-one-out observations is suitable for model selection. We also show that a bespoke metric-learning VAE model which performs highly on this task also out-performs other standard unsupervised and a weakly-supervised disentanglement model.
  arXiv  Detail & Related papers  (2020-12-14T22:01:15Z)
• Are Visual Explanations Useful? A Case Study in Model-in-the-Loop Prediction [49.254162397086006]
  We study explanations based on visual saliency in an image-based age prediction task. We find that presenting model predictions improves human accuracy. However, explanations of various kinds fail to significantly alter human accuracy or trust in the model.
  arXiv  Detail & Related papers  (2020-07-23T20:39:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.