GeneralThinker: Domain-General Reasoning through Likelihood-Guided Answer-Conditioned Optimization

• URL: http://arxiv.org/abs/2605.27934v1
• Date: Wed, 27 May 2026 04:07:26 GMT
• Title: GeneralThinker: Domain-General Reasoning through Likelihood-Guided Answer-Conditioned Optimization
• Abstract summary: GeneralThinker is an on-policy framework that reformulates reasoning supervision as dense answer-conditioned optimization.<n>GeneralThinker evaluates generated reasoning trajectories using the likelihood of the ground-truth answer.<n>It constrains token-level updates through clipping and direction-preserving modulation.
• Score: 4.732347368043908
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning with verifiable rewards improves language model reasoning, but its reliance on domain-specific verifiers, sparse outcome rewards, and coarse-grained credit assignment limits its applicability. We introduce GeneralThinker, an on-policy framework that reformulates reasoning supervision as dense answer-conditioned optimization, enabling response-level evaluation and token-level credit assignment without domain-specific verifiers. GeneralThinker evaluates generated reasoning trajectories using the likelihood of the ground-truth answer and derives token-wise compatibility signals for fine-grained credit assignment. To stabilize optimization, it constrains token-level updates through clipping and direction-preserving modulation. Across 11 benchmarks spanning mathematics, STEM, and general reasoning, GeneralThinker achieves the best average performance. Further analyses show that uncontrolled token-level modulation can destabilize training, whereas controlled modulation makes fine-grained credit assignment consistently effective.

Related papers

• Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation [60.55792673956761]
  We propose textbfRubric-Conditioned Self-Distillation, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation.<n>We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance.
  arXiv  Detail & Related papers  (2026-06-17T17:54:04Z)
• Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards [73.44333771806282]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective paradigm for improving the reasoning capabilities of large language models.<n>We propose Correction-Oriented Policy Optimization (CIPO), a simple and effective extension to RLVR that converts on-policy failed trajectories into correction-oriented supervision.<n>CIPO improves learning effectiveness while explicitly enhancing the model's ability to correct its own errors.
  arXiv  Detail & Related papers  (2026-05-14T08:22:21Z)
• Verifiable Process Rewards for Agentic Reasoning [21.195739597726142]
  Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of large language models (LLMs)<n>Most existing approaches rely on sparse outcome-level feedback.<n>We propose Verifiable Process Rewards (VPR), a framework that converts such oracles into dense turn-level supervision for reinforcement learning.
  arXiv  Detail & Related papers  (2026-05-11T10:30:53Z)
• MAVEN: Multi-Agent Verification-Elaboration Network with In-Step Epistemic Auditing [18.35295672031847]
  MAVEN is a framework designed to transform LLMs into deliberate reasoners through explicit role-decoupling.<n>MAVEN consistently outperforms latent reasoning models such as GEMINI-3.1-Pro.<n>MAVEN is fully model-agnostic, serving as a strong and transferable reasoning booster.
  arXiv  Detail & Related papers  (2026-05-08T12:11:08Z)
• Segment-Aligned Policy Optimization for Multi-Modal Reasoning [55.29606572822562]
  We propose a novel reinforcement learning paradigm that treats coherent reasoning steps, rather than tokens or full sequences as fundamental units of policy update.<n>Experiments on representative reasoning benchmarks demonstrate that SAPO consistently outperforms token-level and sequence-level policy optimization methods.<n>Our work underscores the importance of aligning reinforcement learning updates with the structure of reasoning, paving the way for more efficient and semantically grounded policy optimization in complex reasoning tasks.
  arXiv  Detail & Related papers  (2026-05-02T08:47:45Z)
• Reducing Credit Assignment Variance via Counterfactual Reasoning Paths [11.514388061694268]
  Reinforcement learning for multi-step reasoning with large language models (LLMs) often relies on sparse terminal rewards.<n>We introduce a counterfactual comparison-based credit assignment framework, which samples multiple reasoning trajectories under the same input.<n>We construct an implicit process-level advantage estimator that transforms sparse terminal rewards into step-sensitive learning signals.
  arXiv  Detail & Related papers  (2026-04-20T13:33:54Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• Reinforcement Learning with Conditional Expectation Reward [11.862379897558556]
  Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective in enhancing the reasoning capabilities of large language models.<n>We propose Conditional Expectation Reward (CER), which leverages the large language model itself as an implicit verifier.<n>CER is defined as the expected likelihood of generating the reference answer conditioned on the generated answer.
  arXiv  Detail & Related papers  (2026-03-11T10:36:27Z)
• Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning [50.352417879912515]
  Large language models (LLMs) excel at complex tasks with advances in reasoning capabilities.<n>We propose Group Causal Counterfactual Policy Optimization to explicitly train LLMs to learn generalizable reasoning patterns.<n>We then construct token-level advantages from this reward and optimize the policy, encouraging LLMs to favor reasoning patterns that are process-valid and counterfactually robust.
  arXiv  Detail & Related papers  (2026-02-06T08:03:11Z)
• In-Token Rationality Optimization: Towards Accurate and Concise LLM Reasoning via Self-Feedback [38.915062716409686]
  InTRO is a new framework that enables both token-level exploration and self-feedback for accurate and concise reasoning.<n>InTRO consistently outperforms other baselines, raising solution accuracy by up to 20% relative to the base model.<n>Its chains of thought are notably more concise, exhibiting reduced verbosity.
  arXiv  Detail & Related papers  (2025-11-13T01:47:06Z)
• Optimal Baseline Corrections for Off-Policy Contextual Bandits [61.740094604552475]
  We aim to learn decision policies that optimize an unbiased offline estimate of an online reward metric. We propose a single framework built on their equivalence in learning scenarios. Our framework enables us to characterize the variance-optimal unbiased estimator and provide a closed-form solution for it.
  arXiv  Detail & Related papers  (2024-05-09T12:52:22Z)
• When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning [62.00672284480755]
  This paper aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving.
  arXiv  Detail & Related papers  (2023-02-15T04:14:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.