Multimodal Latent Reasoning via Predictive Embeddings

• URL: http://arxiv.org/abs/2604.08065v1
• Date: Thu, 09 Apr 2026 10:27:32 GMT
• Title: Multimodal Latent Reasoning via Predictive Embeddings
• Abstract summary: Pearl is a framework that learns from expert tool-use trajectories entirely in the latent space.<n>Pearl is model-agnostic, simple to train, and naturally supports trajectories with multiple tool calls.
• Score: 43.40267514669565
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tool-augmented multimodal reasoning enables visual language models (VLMs) to improve perception by interacting with external tools (e.g., cropping, depth estimation). However, such approaches incur substantial inference overhead, require specialized supervision, and are prone to erroneous tool calls. We propose Pearl (Predictive Embedding Alignment for Reasoning in Latent space), a JEPA-inspired framework that learns from expert tool-use trajectories entirely in the latent space, eliminating the need for explicit tool invocation at inference time. Unlike reconstruction-based latent reasoning methods, which autoregressively generate latent tokens and suffer from training-inference mismatch and limited support for multi-step tool use, Pearl directly learns predictive embeddings from multimodal trajectories while preserving the standard vision-language generation pipeline: it is model-agnostic, simple to train, and naturally supports trajectories with multiple tool calls. Experiments across multiple perception benchmarks show that Pearl matches or outperforms standard supervised fine-tuning and reconstruction-based latent reasoning approaches. Furthermore, we provide empirical evidence that reconstruction-based methods primarily learn embeddings rather than image edits in latent space, motivating predictive embedding learning as a more principled alternative.

Related papers

• Think, Look, and Revise: Inconsistency-Aware Visual Self-Correction in MLLMs [35.17881745040865]
  Tool-augmented multimodal reasoning integrates external tools into multimodal large language models.<n>We propose ReVISE, a framework that equips MLLMs with verification and dynamic error recovery for tool-augmented reasoning.
  arXiv  Detail & Related papers  (2026-08-29T17:22:08Z)
• Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach [26.437913669133874]
  Post-training reinforcement learning algorithms are commonly used to align vision-language models with human intent.<n>Existing RL-based alignment methods are often resource-intensive and encounter mismatches between training objectives and inference-time distributions.<n>We propose a novel test-time alignment approach that leverages trajectory-guided structured sampling for dynamic inference-time refinement.
  arXiv  Detail & Related papers  (2026-08-04T06:47:08Z)
• Walk the Talk: Bridging the Reasoning-Action Gap for Thinking with Images via Multimodal Agentic Policy Optimization [89.68681087743876]
  Multimodal Large Language Models (MLLMs) have incentivized models to think with images'' by actively invoking visual tools during multi-turn reasoning.<n>The common Reinforcement Learning practice of relying on outcome-based rewards ignores the fact that textual plausibility often masks executive failure.<n>We introduce Multimodal Agentic Policy Optimization (MAPO), bridging the gap between textual reasoning and visual actions generated by models within their Multimodal Chain-of-Thought.
  arXiv  Detail & Related papers  (2026-04-08T07:48:07Z)
• Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought [73.39221516441624]
  Multimodal Chain-of-Thought (CoT) reasoning requires large vision-language models to construct reasoning trajectories.<n>Existing Reinforcement Learning with Verifiable Rewards (RLVR) methods treat CoT uniformly without distinguishing varying degrees of visual grounding.<n>We propose Perception-Exploration Policy Optimization (PEPO), which derives a perception prior to hidden state similarity and integrates it with token entropy.
  arXiv  Detail & Related papers  (2026-03-24T06:38:00Z)
• Perception-Aware Multimodal Spatial Reasoning from Monocular Images [57.42071289037214]
  spatial reasoning from monocular images is essential for autonomous driving.<n>Current Vision-Language Models (VLMs) still struggle with fine-grained geometric perception.<n>We propose a perception-aware multimodal reasoning framework that equipsVLMs with explicit object-centric grounding ability.
  arXiv  Detail & Related papers  (2026-03-07T02:05:12Z)
• ForgeryVCR: Visual-Centric Reasoning via Efficient Forensic Tools in MLLMs for Image Forgery Detection and Localization [62.03035862528452]
  ForgeryVCR is a framework that materializes imperceptible traces into explicit visual intermediates via Visual-Centric Reasoning.<n>ForgeryVCR achieves state-of-the-art (SOTA) performance in both detection and localization tasks.
  arXiv  Detail & Related papers  (2026-02-15T11:14:47Z)
• ReThinker: Scientific Reasoning by Rethinking with Guided Reflection and Confidence Control [44.113610704492224]
  We introduce ReThinker, a confidence-aware agentic framework that orchestrates retrieval, tool use, and multi-agent reasoning.<n> Experiments on HLE, GAIA, and XBench demonstrate that ReThinker consistently outperforms state-of-the-art foundation models.
  arXiv  Detail & Related papers  (2026-02-04T12:41:52Z)
• Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing [62.447497430479174]
  Drawing to reason in space is a novel paradigm that enables LVLMs to reason through elementary drawing operations in the visual space.<n>Our model, named VILASR, consistently outperforms existing methods across diverse spatial reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T17:41:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.