Sketch-in-Latents: Eliciting Unified Reasoning in MLLMs

• URL: http://arxiv.org/abs/2512.16584v1
• Date: Thu, 18 Dec 2025 14:29:41 GMT
• Title: Sketch-in-Latents: Eliciting Unified Reasoning in MLLMs
• Authors: Jintao Tong, Jiaqi Gu, Yujing Lou, Lubin Fan, Yixiong Zou, Yue Wu, Jieping Ye, Ruixuan Li,
• Abstract summary: Sketch-in-Latents is a novel paradigm for unified multi-modal reasoning.<n>It generates continuous visual embeddings, termed latent sketch tokens, as visual thoughts.<n>It achieves superior performance on vision-centric tasks while exhibiting strong generalization to diverse general multi-modal benchmarks.
• Score: 53.57402214935238
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Multimodal Large Language Models (MLLMs) excel at visual understanding tasks through text reasoning, they often fall short in scenarios requiring visual imagination. Unlike current works that take predefined external toolkits or generate images during thinking, however, humans can form flexible visual-text imagination and interactions during thinking without predefined toolkits, where one important reason is that humans construct the visual-text thinking process in a unified space inside the brain. Inspired by this capability, given that current MLLMs already encode visual and text information in the same feature space, we hold that visual tokens can be seamlessly inserted into the reasoning process carried by text tokens, where ideally, all visual imagination processes can be encoded by the latent features. To achieve this goal, we propose Sketch-in-Latents (SkiLa), a novel paradigm for unified multi-modal reasoning that expands the auto-regressive capabilities of MLLMs to natively generate continuous visual embeddings, termed latent sketch tokens, as visual thoughts. During multi-step reasoning, the model dynamically alternates between textual thinking mode for generating textual think tokens and visual sketching mode for generating latent sketch tokens. A latent visual semantics reconstruction mechanism is proposed to ensure these latent sketch tokens are semantically grounded. Extensive experiments demonstrate that SkiLa achieves superior performance on vision-centric tasks while exhibiting strong generalization to diverse general multi-modal benchmarks. Codes will be released at https://github.com/TungChintao/SkiLa.

Related papers

• Latent Sketchpad: Sketching Visual Thoughts to Elicit Multimodal Reasoning in MLLMs [80.2089647067782]
  We introduce Latent Sketchpad, a framework that equips Multimodal Large Language Models with an internal visual scratchpad.<n>Inspired by how humans use sketching as a form of visual thinking to develop and communicate ideas, we introduce Latent Sketchpad.<n>We evaluate the framework on our new dataset MazePlanning.
  arXiv  Detail & Related papers  (2025-10-28T15:26:20Z)
• Visual Jigsaw Post-Training Improves MLLMs [58.29961336087896]
  We introduce Visual Jigsaw, a generic self-supervised post-training framework designed to strengthen visual understanding in large language models (MLLMs)<n>Visual Jigsaw is formulated as a general ordering task: visual inputs are partitioned, shuffled, and the model must reconstruct the visual information by producing the correct permutation in natural language.<n>Extensive experiments demonstrate substantial improvements in fine-grained perception, temporal reasoning, and 3D spatial understanding.
  arXiv  Detail & Related papers  (2025-09-29T17:59:57Z)
• Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens [44.19323180593379]
  Vision-language models (VLMs) excel at multimodal understanding, yet their text-only decoding forces them to verbalize visual reasoning.<n>Recent attempts train VLMs to render explicit images, but the heavy image-generation pre-training often hinders the reasoning ability.<n>We present a Machine Mental Imagery framework, dubbed as Mirage, which augments VLM decoding with latent visual tokens alongside ordinary text.
  arXiv  Detail & Related papers  (2025-06-20T17:59:31Z)
• ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension [71.03445074045092]
  We propose ClawMachine, offering a new methodology that explicitly notates each entity using token collectives groups of visual tokens.<n>Our method unifies the prompt and answer of visual referential tasks without using additional syntax.<n>ClawMachine achieves superior performance on scene-level and referential understanding tasks with higher efficiency.
  arXiv  Detail & Related papers  (2024-06-17T08:39:16Z)
• Towards Semantic Equivalence of Tokenization in Multimodal LLM [149.11720372278273]
  Vision tokenization is essential for semantic alignment between vision and language.<n>This paper proposes a novel dynamic Semantic-Equivalent Vision Tokenizer (SeTok)<n>SeTok groups visual features into semantic units via a dynamic clustering algorithm.<n>The resulting vision tokens effectively preserve semantic integrity and capture both low-frequency and high-frequency visual features.
  arXiv  Detail & Related papers  (2024-06-07T17:55:43Z)
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization [52.935150075484074]
  We introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously.
  arXiv  Detail & Related papers  (2023-09-09T03:01:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.