Logical Reasoning with Outcome Reward Models for Test-Time Scaling

• URL: http://arxiv.org/abs/2508.19903v1
• Date: Wed, 27 Aug 2025 14:08:43 GMT
• Title: Logical Reasoning with Outcome Reward Models for Test-Time Scaling
• Authors: Ramya Keerthy Thatikonda, Wray Buntine, Ehsan Shareghi,
• Abstract summary: We present a set of Outcome Reward Models (ORMs) for deductive reasoning.<n>To train the ORMs, we mainly generate data using Chain-of-Thought (CoT) with single and multiple samples.<n>We also propose a novel tactic to further expand the type of errors covered in the training dataset of the ORM.
• Score: 10.795521518273214
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), as it reflects their ability to derive valid conclusions from given premises. While the combination of test-time scaling with dedicated outcome or process reward models has opened up new avenues to enhance LLMs performance in complex reasoning tasks, this space is under-explored in deductive logical reasoning. We present a set of Outcome Reward Models (ORMs) for deductive reasoning. To train the ORMs we mainly generate data using Chain-of-Thought (CoT) with single and multiple samples. Additionally, we propose a novel tactic to further expand the type of errors covered in the training dataset of the ORM. In particular, we propose an echo generation technique that leverages LLMs' tendency to reflect incorrect assumptions made in prompts to extract additional training data, covering previously unexplored error types. While a standard CoT chain may contain errors likely to be made by the reasoner, the echo strategy deliberately steers the model toward incorrect reasoning. We show that ORMs trained on CoT and echo-augmented data demonstrate improved performance on the FOLIO, JustLogic, and ProverQA datasets across four different LLMs.

Related papers

• What Language Models Know But Don't Say: Non-Generative Prior Extraction for Generalization [5.663538370244175]
  We propose LoID, a deterministic method for extracting informative prior distributions for Bayesian logistic regression.<n>Rather than relying on generated text, we probe the model's confidence in opposing semantic directions through carefully constructed sentences.<n>We evaluate LoID on ten real-world datasets under synthetic out-of-distribution (OOD) settings.
  arXiv  Detail & Related papers  (2026-01-24T22:05:01Z)
• Learning to Reason in LLMs by Expectation Maximization [55.721496945401846]
  We formalize reasoning as a latent variable model and derive an expectation-maximization objective for learning to reason.<n>This view connects EM and modern reward-based optimization, and shows that the main challenge lies in designing a sampling distribution that generates rationales that justify correct answers.
  arXiv  Detail & Related papers  (2025-12-23T08:56:49Z)
• Making Mathematical Reasoning Adaptive [61.45161826629692]
  We propose the AdaR framework to enable adaptive reasoning in large language models (LLMs)<n>AdaR synthesizes logically equivalent queries by varying variable values, and trains models with RLVR on these data to penalize spurious logic.<n> Experimental results demonstrate that AdaR improves robustness and generalization, achieving substantial improvement in mathematical reasoning.
  arXiv  Detail & Related papers  (2025-10-06T09:30:05Z)
• Pushing LLMs to Their Logical Reasoning Bound: The Role of Data Reasoning Intensity [59.27594125465172]
  We introduce Data Reasoning Intensity (DRI), a novel metric that quantifies the latent logical reasoning complexity of samples.<n>We then introduce a re-cognizing optimization strategy that systematically enhances the logical reasoning intensity of training data.
  arXiv  Detail & Related papers  (2025-09-29T14:20:04Z)
• Audited Reasoning Refinement: Fine-Tuning Language Models via LLM-Guided Step-Wise Evaluation and Correction [1.41282143488996]
  Training a task-specific small reasoning model is challenging when direct human supervision or high-quality labels are scarce.<n>We propose Reason-Refine-then-Align (R2tA), which turns refined model rationales into supervision for training task-specific reasoning models.
  arXiv  Detail & Related papers  (2025-09-15T21:47:52Z)
• Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers [59.168391398830515]
  We evaluate 12 pre-trained LLMs and one specialized fact-verifier, using a collection of examples from 14 fact-checking benchmarks.<n>We highlight the importance of addressing annotation errors and ambiguity in datasets.<n> frontier LLMs with few-shot in-context examples, often overlooked in previous works, achieve top-tier performance.
  arXiv  Detail & Related papers  (2025-06-16T10:32:10Z)
• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models [69.798277882245]
  We introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance large language models' reasoning efficiency.<n>UPFT removes the need for labeled data or exhaustive sampling.<n> Experiments show that UPFT matches the performance of supervised methods.
  arXiv  Detail & Related papers  (2025-03-04T18:56:03Z)
• CounterBench: A Benchmark for Counterfactuals Reasoning in Large Language Models [5.409370027524351]
  We evaluate the performance of large language models (LLMs) in counterfactual reasoning.<n>We introduce a new benchmark dataset, CounterBench, comprising 1K counterfactual reasoning questions.
  arXiv  Detail & Related papers  (2025-02-16T06:19:37Z)
• Understanding Chain-of-Thought in LLMs through Information Theory [16.78730663293352]
  We formalize Chain-of-Thought (CoT) reasoning in Large Language Models (LLMs) through an information-theoretic lens.<n>Specifically, our framework quantifies the information-gain' at each reasoning step, enabling the identification of failure modes.<n>We demonstrate the efficacy of our approach through extensive experiments on toy arithmetic, GSM8K and PRM800k datasets.
  arXiv  Detail & Related papers  (2024-11-18T19:14:36Z)
• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph [70.79413606968814]
  We introduce Dynamic Evaluation of LLMs via Adaptive Reasoning Graph Evolvement (DARG) to dynamically extend current benchmarks with controlled complexity and diversity. Specifically, we first extract the reasoning graphs of data points in current benchmarks and then perturb the reasoning graphs to generate novel testing data. Such newly generated test samples can have different levels of complexity while maintaining linguistic diversity similar to the original benchmarks.
  arXiv  Detail & Related papers  (2024-06-25T04:27:53Z)
• Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction [75.25114727856861]
  Large language models (LLMs) tend to suffer from deterioration at the latter stage ofSupervised fine-tuning process. We introduce a simple disperse-then-merge framework to address the issue. Our framework outperforms various sophisticated methods such as data curation and training regularization on a series of standard knowledge and reasoning benchmarks.
  arXiv  Detail & Related papers  (2024-05-22T08:18:19Z)
• AS-ES Learning: Towards Efficient CoT Learning in Small Models [35.225382243612174]
  Chain-of-Thought (CoT) serves as a critical emerging ability in Large Language Models (LLMs) We propose a new training paradigm AS-ES (Abstractive Segments - Extractive Segments) learning, which exploits the inherent information in CoT for iterative generation. Experiments show that our methods surpass the direct seq2seq training on CoT-extensive tasks like MWP and PET summarization, without data augmentation or altering the model itself.
  arXiv  Detail & Related papers  (2024-03-04T12:13:59Z)
• Learning to Reduce: Optimal Representations of Structured Data in Prompting Large Language Models [42.16047343029512]
  Large Language Models (LLMs) have been widely used as general-purpose AI agents. We propose a framework, Learning to Reduce, that fine-tunes a language model to generate a reduced version of an input context. We show that our model achieves comparable accuracies in selecting the relevant evidence from an input context.
  arXiv  Detail & Related papers  (2024-02-22T00:41:23Z)
• Explaining Emergent In-Context Learning as Kernel Regression [61.57151500616111]
  Large language models (LLMs) have initiated a paradigm shift in transfer learning. In this paper, we investigate the reason why a transformer-based language model can accomplish in-context learning after pre-training. We find that during ICL, the attention and hidden features in LLMs match the behaviors of a kernel regression.
  arXiv  Detail & Related papers  (2023-05-22T06:45:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.