Test-Time Deep Thinking to Explore Implicit Rules

• URL: http://arxiv.org/abs/2605.24828v2
• Date: Sun, 31 May 2026 15:58:08 GMT
• Title: Test-Time Deep Thinking to Explore Implicit Rules
• Abstract summary: Test-Time Exploration (TTExplore) is a framework where a thinker component analyzes interaction history to infer implicit rules and guide an actor.<n>Experiments on five text-based embodied tasks show that TTExplore equipped with Exp-Thinker improves baseline agent performance by an average of $14$-$19$ points.
• Score: 80.74526536918196
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the continuous advancement of Large Language Models (LLMs), intelligent agents are becoming increasingly vital. However, these agents often fail in environments governed by implicit rules--hidden constraints that cannot be observed directly and must be inferred through interaction. This causes agents to fall into repetitive trial-and-error loops, ultimately leading to task failure. To address this challenge, we propose Test-Time Exploration (TTExplore), a framework where a thinker component analyzes interaction history to infer these implicit rules and guide an actor. Effective exploration in this setting critically depends on the reasoning ability of the thinker. However, evaluating deep reasoning trajectories is inherently unstable and difficult, which poses a major obstacle to effective training. To overcome this issue, we introduce a novel and stable reinforcement learning pipeline. The core idea is to use accurate task-level scores as indirect rewards to bypass the difficulty of evaluating intermediate reasoning, and to retain only a single thinking node per trajectory to alleviate reward sparsity. Using this pipeline, we train a specialized 7B model, Exp-Thinker. Experiments on five text-based embodied tasks show that TTExplore equipped with Exp-Thinker improves baseline agent performance by an average of $14$-$19$ points, demonstrating the effectiveness of explicitly reasoning about implicit rules.

Related papers

• ALIVE: Awakening LLM Reasoning via Adversarial Learning and Instructive Verbal Evaluation [4.265094703231012]
  We introduce textbfALIVE (emphAdrial Learning with Instructive Verbal Evaluation), a hands-free alignment framework.<n>By coupling adversarial learning with instructive verbal feedback, ALIVE enables models to internalize evaluative criteria directly from raw corpora.<n>With identical data and compute, ALIVE achieves markedly improved cross-domain generalization, and higher self-correction rates.
  arXiv  Detail & Related papers  (2026-02-05T09:20:23Z)
• Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Tasks [54.31998314008198]
  Large Language Models (LLMs) excel in reasoning tasks requiring a single correct answer, but they perform poorly in multi-solution tasks.<n>We attribute this limitation to textbfreasoning overconfidence: a tendency to express undue certainty in an incomplete solution set.<n>We propose the textbfcognitive-rigidity hypothesis, which posits that overconfidence arises when the reasoning process prematurely converges on a narrow set of thought paths.
  arXiv  Detail & Related papers  (2025-12-01T14:35:06Z)
• Thinking Before You Speak: A Proactive Test-time Scaling Approach [54.8205006555199]
  We implement our idea as a reasoning framework, named emphThinking Before You Speak (TBYS)<n>We design a pipeline for automatically collecting and filtering in-context examples for the generation of emphinsights.<n>Experiments on challenging mathematical datasets verify the effectiveness of TBYS.
  arXiv  Detail & Related papers  (2025-08-26T03:43:32Z)
• Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models [67.87579664988199]
  TON is a two-stage training strategy for vision-language models (VLMs)<n>It introduces a think-or-not format that serves as a cold start for selective reasoning.<n>TON can reduce the completion length by up to 90% compared to vanilla GRPO.
  arXiv  Detail & Related papers  (2025-05-22T16:13:29Z)
• Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment [54.62926010621013]
  We introduce a novel task, code reasoning, to provide a new perspective for the reasoning abilities of large language models.<n>We summarize three meta-benchmarks based on established forms of logical reasoning, and instantiate these into eight specific benchmark tasks.<n>We present a new pathway exploration pipeline inspired by human intricate problem-solving methods.
  arXiv  Detail & Related papers  (2025-02-17T10:39:58Z)
• Is Depth All You Need? An Exploration of Iterative Reasoning in LLMs [11.896234713853298]
  We investigate whether relevant knowledge that contributes directly to solving the given question can be activated from the initial reasoning path.<n>Our experiments reveal that increasing the diversity of initial reasoning paths can achieve comparable or superior performance.<n>We propose a simple yet effective method that enhances reasoning breadth by integrating contextual exploration with reduced sampling randomness.
  arXiv  Detail & Related papers  (2025-02-15T16:59:59Z)
• Towards Principled Unsupervised Multi-Agent Reinforcement Learning [49.533774397707056]
  We present a scalable, decentralized, trust-region policy search algorithm to address the problem in practical settings.<n>We show that optimizing for a specific objective, namely mixture entropy, provides an excellent trade-off between tractability and performances.
  arXiv  Detail & Related papers  (2025-02-12T12:51:36Z)
• Improving LLM Reasoning with Multi-Agent Tree-of-Thought Validator Agent [9.439315294704368]
  Tree of Thoughts (ToT) methods have shown potential in improving reasoning for complex question-answering tasks. A critical limitation in multi-agent reasoning is the 'Reasoner' agent's shallow exploration of reasoning paths. We introduce a novel approach combining ToT-based Reasoner agents with a Thought Validator agent. Our method demonstrates superior performance compared to existing techniques when evaluated on the GSM8K dataset.
  arXiv  Detail & Related papers  (2024-09-17T19:54:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.