CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts

• URL: http://arxiv.org/abs/2606.31986v2
• Date: Fri, 03 Jul 2026 17:19:26 GMT
• Title: CoLT: Teaching Multi-Modal Models to Think with Chain of Latent Thoughts
• Abstract summary: Chain-of-thought (CoT) reasoning has enabled multi-modal large language models (MLLMs) to tackle complex visual reasoning tasks.<n>We propose CoLT, a novel framework that teaches multi-modal models to reason through a chain of latent thought representations.<n>Compared to text CoT methods, CoLT can notably reduce the inference time by 10.1$times$ and text decoding time by 22.6$times$.
• Score: 25.917787556467776
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chain-of-thought (CoT) reasoning has enabled multi-modal large language models (MLLMs) to tackle complex visual reasoning tasks by generating explicit intermediate reasoning steps in natural language. However, this text-based reasoning paradigm is inherently slow at inference time with even thousands of tokens and fundamentally constrained by the expressiveness of natural language. In this paper, we propose CoLT, (Chain of Latent Thoughts), a novel framework that teaches multi-modal models to reason through a chain of latent thought representations instead of verbose text tokens, which can perform thinking with as few as 3 steps. Naively forcing the model to think with latent states easily produces meaningless semantics and makes training unstable. To effectively regulate the latent reasoning process, we introduce a lightweight external decoder that provides step-level supervision for each latent reasoning step in two complementary directions: a forward mode that decodes latent thoughts into the textual reasoning of the next step, and a backward mode that aligns decoder hidden states with the model's latent thoughts given preceding textual context. We further incorporate internal supervision that encourages coherent step-by-step latent transitions. The decoder and internal supervision are removed during inference to maintain high efficiency of latent reasoning. Extensive experiments on eight benchmarks demonstrate that CoLT not only outperforms existing latent reasoning methods such as CODI and SIM-CoT, but also surpasses latent visual reasoning approaches that rely on auxiliary images with costly annotation requirements. Compared to text CoT methods, CoLT can notably reduce the inference time by 10.1$\times$ and text decoding time by 22.6$\times$. Code is released at https://github.com/hulianyuyy/CoLT.

Related papers

• A Model with No Head and Many Thoughts [8.93920041624494]
  We introduce Soft Latent Thinking, a method that replaces the LM head during reasoning with a lightweight projector.<n>We show that Soft Latent Thinking consistently improves pass@k across all k while reducing per-step compute during chain-of-thought.
  arXiv  Detail & Related papers  (2026-08-31T16:45:41Z)
• Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression [42.15219720606396]
  Large language models achieve high reasoning performance via explicit chain-of-thought and reinforcement learning.<n>Latent reasoning reduces this cost by shifting computation into a latent space.<n>We propose textbfDiscrete Latent Reasoning(DLR), the first method that converts continuous latent states into explicit discrete tokens.
  arXiv  Detail & Related papers  (2026-06-29T02:34:52Z)
• OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation [61.18260993245354]
  Chain-of-Thought (CoT) reasoning is a powerful driver of trajectory prediction in VLA-based autonomous driving.<n>We present OneVL, a unified VLA and World Model framework that routes reasoning through compact latent tokens supervised by dual auxiliary decoders.<n>OneVL becomes the first latent CoT method to surpass explicit CoT, delivering state-of-the-art accuracy at answer-only latency.
  arXiv  Detail & Related papers  (2026-04-20T16:37:22Z)
• SPOT: Span-level Pause-of-Thought for Efficient and Interpretable Latent Reasoning in Large Language Models [15.95627037350657]
  Explicit Chain-of-Thought (CoT) computation incurs high inference cost due to internalize token-level traces.<n>We propose SPOT, a flexible framework that compresses explicit CoT into compact latent pause tokens without enforcing a fixed response template.<n>Experiments on reasoning benchmarks demonstrate that SPOT improves accuracy by 2.3 points on average while reducing generated tokens by 37.5%.
  arXiv  Detail & Related papers  (2026-03-06T12:34:27Z)
• Latent Thoughts Tuning: Bridging Context and Reasoning with Fused Information in Latent Tokens [13.653741247835091]
  Latent Thoughts Tuning (LT-Tuning) is a framework that redefines how latent thoughts are constructed and deployed.<n>We introduce a Context-Prediction-Fusion mechanism that jointly leveraging contextual hidden states and predictive semantic guidance.<n>Our method outperforms existing latent reasoning baselines, effectively mitigating feature collapse and achieving robust reasoning accuracy.
  arXiv  Detail & Related papers  (2026-02-10T19:19:10Z)
• CoLT: Reasoning with Chain of Latent Tool Calls [31.228763375347608]
  Chain-of-Thought (CoT) is a critical technique in enhancing the reasoning ability of Large Language Models (LLMs)<n>We propose CoLT, a novel framework that implements latent reasoning as tool calls''
  arXiv  Detail & Related papers  (2026-02-04T06:12:53Z)
• Fast Thinking for Large Language Models [67.7238685892317]
  We introduce Latent Codebooks for Fast Thinking, a framework that uses concise CoT sketches only during training to learn a codebook of discrete strategy priors.<n>At inference, the model conditions on a handful of continuous thinking switches distilled from the codebook in a single pass, enabling strategy-level guidance without producing explicit reasoning tokens.
  arXiv  Detail & Related papers  (2025-09-28T04:19:48Z)
• A Survey on Latent Reasoning [100.54120559169735]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities.<n>CoT reasoning that verbalizes intermediate steps limits the model's expressive bandwidth.<n>Latent reasoning tackles this bottleneck by performing multi-step inference entirely in the model's continuous hidden state.
  arXiv  Detail & Related papers  (2025-07-08T17:29:07Z)
• Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space [62.54887038032942]
  We introduce Soft Thinking, a training-free method that emulates human-like "soft" reasoning by generating soft, abstract concept tokens.<n>These concept tokens are created by the probability-weighted mixture of token embeddings, which form the continuous concept space.<n>In essence, each generated concept token encapsulates multiple meanings from related discrete tokens, implicitly exploring various reasoning paths to converge.
  arXiv  Detail & Related papers  (2025-05-21T17:29:15Z)
• Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketching [64.74765550805024]
  Chain-of-Thought prompting elicits step-by-step problem solving, but often at the cost of excessive verbosity in intermediate outputs.<n>We propose Sketch-of-Thought (SoT), a prompting framework that integrates cognitively inspired reasoning paradigms with linguistic constraints.<n>SoT achieves token reductions of up to 84% with minimal accuracy loss across 18 reasoning datasets.
  arXiv  Detail & Related papers  (2025-03-07T06:57:17Z)
• Training Large Language Models to Reason in a Continuous Latent Space [71.0274000348354]
  We introduce a new paradigm called Coconut (Chain of Continuous Thought) to explore the potential of reasoning beyond language.<n>Instead of decoding this state into words, we feed it back to the model as the next input embedding directly in the continuous space.<n>This latent reasoning paradigm enables an advanced reasoning pattern, where continuous thoughts can encode multiple alternative next steps.
  arXiv  Detail & Related papers  (2024-12-09T18:55:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.