StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning

• URL: http://arxiv.org/abs/2607.00465v1
• Date: Wed, 01 Jul 2026 05:34:07 GMT
• Title: StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning
• Abstract summary: Large Vision-Language Models (LVLMs) rely extensively on Visual Instruction Tuning (VIT) to elicit their multimodal reasoning capabilities.<n>Existing benchmarks evaluate LVLMs in isolated, single-turn scenarios.<n>We introduce a challenging, benchmark-agnostic evaluation mechanism based on the Balanced Latin Square to measure LVLMs' robustness.
• Score: 31.347649780181545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Vision-Language Models (LVLMs) rely extensively on Visual Instruction Tuning (VIT) to elicit their multimodal reasoning capabilities. However, we find a discrepancy: VIT often packs multiple language tasks about the same image for conversational, multi-turn training, whereas existing benchmarks evaluate LVLMs in isolated, single-turn scenarios. The models can suffer from visual attention decay and contextual overfitting during multi-turn training, making it hard for them to realize their full potential in the mismatched test phase. To close the gap, we propose learning with Stochastic Turn Depth (StochasT), which stochastically groups language tasks for the same image into clusters of varying sizes (turn depth) while preserving their organic order. Hence, while StochasT draws on Dropout and stochastic depth for ResNets, it does not actually drop anything to maximize the utility of the training data. Furthermore, we introduce a challenging, benchmark-agnostic evaluation mechanism based on the Balanced Latin Square to measure LVLMs' robustness under varying contextual dependencies. Extensive experiments demonstrate that StochasT effectively grants LVLMs strong, harmonized capabilities for both single-turn and multi-turn use cases.

Related papers

• Test-Time Hallucination Control in Large Vision-Language Models [11.243503200595937]
  Hallucination in large vision-language models (LVLMs) is a critical barrier to their reliability in real-world applications.<n>We propose Test-Time Hallucination Mitigation (TTH) method, a novel training-free method that addresses both limitations.<n>Extensive experiments across multiple LVLM families and diverse benchmarks demonstrate that TTH consistently improves accuracy and robustness.
  arXiv  Detail & Related papers  (2026-08-11T22:25:22Z)
• Self-Prophetic Decoding to Unlock Visual Search in LVLMs [108.77389957341586]
  Large Vision-Language Models (LVLMs) are rapidly evolving toward true multimodal reasoning.<n>LVLM visual search faces two key challenges: incompatibility among intrinsic capabilities after post-training, and interference in long multi-step reasoning contexts.<n>We introduce SeProD, a self-prophetic decoding framework that leverages intrinsic single-step capabilities to enable coherent multi-step reasoning.
  arXiv  Detail & Related papers  (2026-05-27T17:01:39Z)
• Many-Shot CoT-ICL: Making In-Context Learning Truly Learn [58.439517684779936]
  In-context learning (ICL) adapts large language models to new tasks by conditioning on demonstrations in the prompt without parameter updates.<n>We study many-shot chain-of-thought in-context learning (CoT-ICL) for reasoning and show that standard many-shot rules do not transfer.
  arXiv  Detail & Related papers  (2026-05-13T13:30:12Z)
• Learn to Think: Improving Multimodal Reasoning through Vision-Aware Self-Improvement Training [82.17582358979884]
  Post-training with explicit reasoning traces is common to improve the reasoning capabilities of Multimodal Large Language Models (MLLMs)<n>We propose VISTA, a vision-aware self-improvement training framework for enhancing the multimodal reasoning of MLLMs.
  arXiv  Detail & Related papers  (2026-05-12T10:44:35Z)
• FewMMBench: A Benchmark for Multimodal Few-Shot Learning [17.747746608503114]
  FewMMBench is a comprehensive benchmark designed to evaluate multimodal large language models (MLLMs) under few-shot conditions.<n>We evaluate 26 open-weight MLLMs from six model families across zero-shot, few-shot, and CoT-augmented few-shot settings.<n>Our findings reveal that instruction-tuned models exhibit strong zero-shot performance but benefit minimally, or even regress, with additional demonstrations or CoT reasoning.
  arXiv  Detail & Related papers  (2026-02-25T12:30:18Z)
• WeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokens [69.97021957331326]
  We propose Noisy Query Tokens, which learn a distributed representation space between the VLM and Diffusion Model via end-to-end optimization.<n>We also introduce a VAE branch with linear projection to recover fine-grained image details.
  arXiv  Detail & Related papers  (2025-12-02T09:02:20Z)
• Beyond Single Models: Mitigating Multimodal Hallucinations via Adaptive Token Ensemble Decoding [41.828387997311474]
  Large Vision-Language Models (LVLMs) have recently achieved impressive results in multimodal tasks such as image captioning and visual question answering.<n>They remain prone to object hallucination -- generating descriptions of nonexistent or misidentified objects.<n>We propose Adaptive Token Ensemble Decoding (ATED), a training-free, token-level ensemble framework that mitigates hallucination by aggregating predictions from multiple LVLMs during inference.
  arXiv  Detail & Related papers  (2025-10-21T06:11:24Z)
• Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting [70.83781268763215]
  Vision-language models (VLMs) have achieved impressive performance across diverse multimodal tasks by leveraging large-scale pre-training.<n>VLMs face unique challenges such as cross-modal feature drift, parameter interference due to shared architectures, and zero-shot capability erosion.<n>This survey aims to serve as a comprehensive and diagnostic reference for researchers developing lifelong vision-language systems.
  arXiv  Detail & Related papers  (2025-08-06T09:03:10Z)
• Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning [40.972648044298374]
  Multi-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks. They often lack interpretability and struggle with complex visual inputs. We introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs. We propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts.
  arXiv  Detail & Related papers  (2024-03-25T17:59:23Z)
• SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models [19.005364038603204]
  We introduce a novel fine-tuning paradigm named Self-Consistency Tuning (SC-Tune) SC-Tune features the synergistic learning of a cyclic describer-locator system. We demonstrate that SC-Tune significantly elevates performance across a spectrum of object-level vision-language benchmarks.
  arXiv  Detail & Related papers  (2024-03-20T03:00:21Z)
• Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models [76.410400238974]
  We propose TTA with feedback to rectify the model output and prevent the model from becoming blindly confident. A CLIP model is adopted as the reward model during TTA and provides feedback for the VLM. The proposed textitreinforcement learning with CLIP feedback(RLCF) framework is highly flexible and universal.
  arXiv  Detail & Related papers  (2023-05-29T11:03:59Z)
• Dense Contrastive Visual-Linguistic Pretraining [53.61233531733243]
  Several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. We propose unbiased Dense Contrastive Visual-Linguistic Pretraining to replace the region regression and classification with cross-modality region contrastive learning.
  arXiv  Detail & Related papers  (2021-09-24T07:20:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.