Rotate2Think: Geometric Priming via Orthogonal Rotation to Improve Language Model Reasoning

• URL: http://arxiv.org/abs/2606.09873v1
• Date: Tue, 02 Jun 2026 18:05:57 GMT
• Title: Rotate2Think: Geometric Priming via Orthogonal Rotation to Improve Language Model Reasoning
• Abstract summary: Internal structure of representation space when reasoning is poorly understood.<n>We show that both input embeddings and thinking embeddings exhibit extremely high conicity.<n>We propose Rotate2Think, a training-free method that estimates this rotation from a small set of correctly solved examples.
• Score: 7.799141616169557
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning models achieve strong performance on challenging tasks by generating explicit intermediate reasoning traces before producing a final answer. Yet the internal structure of representation space when reasoning remains poorly understood: how do a model's hidden representations differ during thinking versus the embeddings of the input prompt, and can this structure be exploited to elicit stronger reasoning at inference time? We show that both input embeddings and thinking embeddings (mean-pooled last-layer hidden states over the prompt and reasoning trace, respectively) exhibit extremely high conicity, with all vectors clustering tightly around a single mean direction. Crucially, these mean input and thinking directions are non-collinear, with thinking embeddings occupying a geometrically distinct region of embedding space across many different models and benchmark tasks. This observation motivates casting the input-to-thinking transition as a rotation problem admitting a closed-form solution via orthogonal Procrustes analysis. We propose Rotate2Think, a training-free method that estimates this rotation from a small set of correctly solved examples and injects the resulting synthetic thinking vector between thinking delimiters at inference time, providing a geometric primer at the onset of the reasoning trace. Evaluated across multiple benchmarks and model families, Rotate2Think improves accuracy in 30 of 32 model-benchmark configurations across mathematics, science, and code tasks, and generalizes zero-shot to multimodal reasoning on MATH-Vision.

Related papers

• Forethought: Verifiable Reasoning from Neurosymbolic Primitive Programming [0.764671395172401]
  Forethought is a neurosymbolic reasoning system that treats reasoning as an explicit, verifiable program.<n>It improves base-model accuracy by about 30% relative and outperforms vanilla prompting, reinforcement learning scaffolds, and prompt-evolution methods.
  arXiv  Detail & Related papers  (2026-07-05T03:20:23Z)
• PRISM: A Dual View of LLM Reasoning through Semantic Flow and Latent Computation [15.91920027845529]
  PRISM (Probabilistic Reasoning Inspection through Semantic and Implicit Modeling) is a framework and diagnostic tool for jointly analyzing both levels.<n>It uncovers systematic patterns in the reasoning process, showing that failed trajectories are more likely to become trapped in unproductive verification loops.<n> PRISM makes these behaviors observable and analyzable rather than relying solely on final-task accuracy.
  arXiv  Detail & Related papers  (2026-03-24T03:31:53Z)
• PointCoT: A Multi-modal Benchmark for Explicit 3D Geometric Reasoning [82.55361351483005]
  We present PointCoT, a novel framework that empowers MLLMs with explicit Chain-of-Thought (CoT) reasoning for 3D data.<n>By leveraging a dual-stream multi-modal architecture, our method synergizes semantic appearance with geometric truth.
  arXiv  Detail & Related papers  (2026-02-27T11:47:45Z)
• Inference-Time Rethinking with Latent Thought Vectors for Math Reasoning [44.0190939401003]
  Inference-Time Rethinking is a generative framework that enables iterative self-correction.<n>We factorize reasoning into a continuous latent thought vector (what to reason about) and a decoder that verbalizes the trace conditioned on this vector.
  arXiv  Detail & Related papers  (2026-02-06T10:23:18Z)
• Addressing Overthinking in Large Vision-Language Models via Gated Perception-Reasoning Optimization [56.59356959631999]
  Gated Perception-Reasoning Optimization (GPRO) is a meta-reasoning controller that dynamically routes computation among three decision paths.<n>GPRO substantially improves both accuracy and efficiency, outperforming recent slow-thinking methods.
  arXiv  Detail & Related papers  (2026-01-07T23:05:17Z)
• Selection, Reflection and Self-Refinement: Revisit Reasoning Tasks via a Causal Lens [19.316594303998667]
  Reasoning tasks have long been regarded as rigorous benchmarks for assessing the capabilities of machine learning models.<n>We revisit reasoning tasks from a causal perspective, seeking to understand their behavior in latent space.<n>We introduce a framework, called SR$2$, that incorporates the estimated latent variables as feedback into the selection mechanism.
  arXiv  Detail & Related papers  (2025-10-09T13:45:31Z)
• REMA: A Unified Reasoning Manifold Framework for Interpreting Large Language Model [29.40036398095681]
  We define the Reasoning Manifold, a latent low-dimensional geometric structure formed by the internal representations corresponding to all correctly reasoned generations.<n>We build REMA, a framework that explains the origins of failures by quantitatively comparing the spatial relationships of internal model representations corresponding to both erroneous and correct reasoning samples.<n>Our experiments on diverse language and multimodal models and tasks demonstrate the low-dimensional nature of the reasoning manifold and the high separability between erroneous and correct reasoning representations.
  arXiv  Detail & Related papers  (2025-09-26T16:02:27Z)
• Fractional Reasoning via Latent Steering Vectors Improves Inference Time Compute [60.151643048803145]
  We propose Fractional Reasoning, a framework that enables continuous control over reasoning intensity at inference time.<n>Our method operates by extracting the latent steering vector associated with deeper reasoning and reapplying it with a tunable scaling factor.<n> Experiments on GSM8K, MATH500, and GPQA demonstrate that Fractional Reasoning consistently improves performance across diverse reasoning tasks and models.
  arXiv  Detail & Related papers  (2025-06-18T21:15:59Z)
• PixelThink: Towards Efficient Chain-of-Pixel Reasoning [70.32510083790069]
  PixelThink is a simple yet effective scheme that integrates externally estimated task difficulty and internally measured model uncertainty.<n>It learns to compress reasoning length in accordance with scene complexity and predictive confidence.<n> Experimental results demonstrate that the proposed approach improves both reasoning efficiency and overall segmentation performance.
  arXiv  Detail & Related papers  (2025-05-29T17:55:49Z)
• How do Transformers Learn Implicit Reasoning? [67.02072851088637]
  We study how implicit multi-hop reasoning emerges by training transformers from scratch in a controlled symbolic environment.<n>We find that training with atomic triples is not necessary but accelerates learning, and that second-hop generalization relies on query-level exposure to specific compositional structures.
  arXiv  Detail & Related papers  (2025-05-29T17:02:49Z)
• On the Diagram of Thought [20.805936414171892]
  Large Language Models (LLMs) excel at many tasks but often falter on complex problems that require structured, multi-step reasoning.<n>We introduce the Diagram of Thought (DoT), a new framework that enables a single LLM to build and navigate a mental map of its reasoning.
  arXiv  Detail & Related papers  (2024-09-16T07:01:41Z)
• Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanism [68.05754701230039]
  We construct a symbolic multi-step reasoning task to investigate the information propagation mechanisms in Transformer models.<n>We propose a random matrix-based algorithm to enhance the model's reasoning ability.
  arXiv  Detail & Related papers  (2024-05-24T07:41:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.