Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach

• URL: http://arxiv.org/abs/2608.03204v1
• Date: Tue, 04 Aug 2026 06:47:08 GMT
• Title: Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach
• Abstract summary: Post-training reinforcement learning algorithms are commonly used to align vision-language models with human intent.<n>Existing RL-based alignment methods are often resource-intensive and encounter mismatches between training objectives and inference-time distributions.<n>We propose a novel test-time alignment approach that leverages trajectory-guided structured sampling for dynamic inference-time refinement.
• Score: 26.437913669133874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Post-training reinforcement learning (RL) algorithms are commonly used to align large vision-language models (LVLMs) with human intent and the requirements of visual reasoning tasks. However, existing RL-based alignment methods are often resource-intensive and encounter mismatches between training objectives and inference-time distributions. To bridge this gap, we propose a novel test-time alignment approach that leverages trajectory-guided structured sampling for dynamic inference-time refinement, achieving better alignment with visual grounding and ensuring logical consistency. Our approach begins with curating a reasoning memory bank via a trajectory learning algorithm, which decomposes complex question solving into ordered sequences of predefined reasoning patterns. It subsequently accomplishes inference-time alignment by first collecting trajectories from reasoning memory bank to establish a global structural reasoning prior, and then using an iterative Markov Chain Monte Carlo (MCMC) algorithm for localized multi-objective refinement of the reasoning trace. Experiments across multiple multimodal reasoning datasets demonstrate that our approach significantly improves accuracy without incurring prohibitive inference overhead. These results establish trajectory-guided test-time sampling as a scalable and effective alternative to traditional post-training alignment, particularly for complex visual reasoning tasks.

Related papers

• When LLM Meets Tree Search: A Systematic View of Inference as Search in Large Language Models [92.71325249013535]
  Test-Time Scaling (TTS) has emerged as an important direction for improving reasoning by allocating inference-time compute to a fixed model prior.<n>This survey systematizes recent progress in tree-search-based reasoning, viewing inference as instance-specific optimization rather than decoding.
  arXiv  Detail & Related papers  (2026-08-31T07:49:56Z)
• Advantage-Guided Gate: Reshaping Open-Ended Reasoning for Vision-Based Spatial Intelligence [70.8951332710039]
  Multimodal large language models (MLLMs) have demonstrated significant potential in complex spatial scene understanding and reasoning tasks.<n>However, their open-ended reasoning process is prone to decision errors and error accumulation, leading to instability in answer quality.<n>We propose an advantage-guided gating framework that dynamically intervenes in and corrects deviations during the reasoning process.
  arXiv  Detail & Related papers  (2026-08-08T07:46:04Z)
• Rethinking Token-Level Policy Optimization for Multimodal Chain-of-Thought [73.39221516441624]
  Multimodal Chain-of-Thought (CoT) reasoning requires large vision-language models to construct reasoning trajectories.<n>Existing Reinforcement Learning with Verifiable Rewards (RLVR) methods treat CoT uniformly without distinguishing varying degrees of visual grounding.<n>We propose Perception-Exploration Policy Optimization (PEPO), which derives a perception prior to hidden state similarity and integrates it with token entropy.
  arXiv  Detail & Related papers  (2026-03-24T06:38:00Z)
• Insight-V++: Towards Advanced Long-Chain Visual Reasoning with Multimodal Large Language Models [65.4947731385794]
  We present a unified multi-agent visual reasoning framework that evolves from our foundational image-centric model, Insight-V.<n>We introduce two novel algorithms, ST-GRPO and J-GRPO, which enhance spatial-temporal reasoning and improve evaluative robustness.<n>Experiments on base models like LLaVA-NeXT and Qwen2.5-VL demonstrate significant performance gains across challenging image and video reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-18T15:28:07Z)
• StAR: Segment Anything Reasoner [11.958150552719296]
  We present Segment Anything Reasoner (StAR), a comprehensive framework that refines the design space from multiple perspectives.<n>StAR achieves significant gains over its base counterparts across extensive benchmarks.
  arXiv  Detail & Related papers  (2026-03-15T13:43:34Z)
• Learning Structured Reasoning via Tractable Trajectory Control [99.75278337895024]
  Ctrl-R is a framework for learning structured reasoning via tractable trajectory control.<n>We show that Ctrl-R enables effective exploration and internalization of previously unattainable reasoning patterns.
  arXiv  Detail & Related papers  (2026-03-02T09:18:19Z)
• Latent Chain-of-Thought for Visual Reasoning [53.541579327424046]
  Chain-of-thought (CoT) reasoning is critical for improving the interpretability and reliability of Large Vision-Language Models (LVLMs)<n>We reformulate reasoning in LVLMs as posterior inference and propose a scalable training algorithm based on amortized variational inference.<n>We empirically demonstrate that the proposed method enhances the state-of-the-art LVLMs on seven reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-27T23:10:06Z)
• Revisiting LLM Reasoning via Information Bottleneck [57.519119962528166]
  Large language models (LLMs) have recently demonstrated remarkable progress in reasoning capabilities through reinforcement learning with verifiable rewards (RLVR)<n>We present a theoretical characterization of LLM reasoning grounded in information bottleneck (IB) principle.<n>We propose IB-aware reasoning optimization (IBRO), a framework that encourages reasoning trajectories to be both informative about the final correct answer and generalizable.
  arXiv  Detail & Related papers  (2025-07-24T13:14:25Z)
• Latent Guided Sampling for Combinatorial Optimization [3.636090511738153]
  Recent Combinatorial Optimization methods leverage deep learning to learn solution strategies, trained via Supervised or Reinforcement Learning (RL)<n>While promising, these approaches often rely on task-specific augmentations, perform poorly on out-of-distribution instances, and lack robust inference mechanisms.<n>In this work, we propose LGS-Net, a novel latent space model that conditions on efficient problem instances, and introduce an efficient Neural inference method, Latent Guided Sampling (LGS)
  arXiv  Detail & Related papers  (2025-06-04T08:02:59Z)
• PRefLexOR: Preference-based Recursive Language Modeling for Exploratory Optimization of Reasoning and Agentic Thinking [0.0]
  PRefLexOR combines preference optimization with concepts from Reinforcement Learning to enable models to self-teach. We focus on applications in biological materials science and demonstrate the method in a variety of case studies.
  arXiv  Detail & Related papers  (2024-10-16T08:46:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.