Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models

• URL: http://arxiv.org/abs/2606.27373v1
• Date: Thu, 25 Jun 2026 17:59:55 GMT
• Title: Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models
• Abstract summary: Current self-evolving LMMs struggle on vision-language understanding tasks such as image captioning and visual question answering.<n>We propose VISE, a purely unsupervised self-evolving framework that directly regularizes the model's visual conditioning policy.<n>VISE operates within a single model without specialist roles, external reward models, or annotations.
• Score: 54.097381112138
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, self-evolving large multimodal models (LMMs) have received attention for improving visual reasoning in a purely unsupervised setting. However, multi-role self-play and self-consistency reward schemes in existing self-evolving LMMs optimize answer agreement without ensuring the decoder attends to visual content, relying instead on statistical language priors to produce self consistent outputs. This leads to a persistent failure mode we term visual under-conditioning, where the decoder relies on language priors rather than the image during generation, manifesting as insufficient attention to visual tokens. As a result, current self-evolving LMMs struggle on vision--language understanding tasks such as image captioning and visual question answering. To address this, we propose VISE (Visual Invariance Self-Evolution), a purely unsupervised self-evolving framework that directly regularizes the model's visual conditioning policy through two complementary invariance-based rewards: a geometric invariance reward that enforces spatial consistency under known transformations, and a semantic invariance reward that penalizes evidence-agnostic generation by requiring the model to recognize the absence of evidence when predicted regions are perturbed. VISE operates within a single model without specialist roles, external reward models, or annotations, and is trained on raw unlabeled images. Experiments on 18 benchmarks demonstrate the efficacy of our approach. Using Qwen3-VL-2B as the base model, VISE achieves gains of $+16.85$ CIDEr on COCO and $+19.66$ CIDEr on TextCaps, reduces object hallucination by $5.0$ Chair-I points, and generalizes across four model families and scales. Our code and models are available at https://mbzuai-oryx.github.io/VISE

Related papers

• Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots [28.035029653746335]
  textbfCVPD (Contrastive Counterfactual Visual Process Distillation) is the first fully self-contained framework for dense, on-policy, token-level visual self-distillation for MLLMs.<n>CVPD identifies visual blind spots where zooming into a region changes and sharpens the model's answer distribution, while removing the same region leaves the full-image behavior largely unchanged.
  arXiv  Detail & Related papers  (2026-08-10T17:59:56Z)
• Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement [50.97806700110183]
  Failure-informed Image Self-Augmentation (textbfFISA) is a framework for MLLM self-improvement that constructs augmented images from the model's own failure cases.<n>Our method generates visually challenging yet answer-preserving image complications, verifies their utility through self-examination, and applies dual fidelity filtering to avoid semantic distortion.
  arXiv  Detail & Related papers  (2026-08-04T14:28:42Z)
• Ask, Solve, Generate: Self-Evolving Unified Multimodal Understanding and Generation via Self-Consistency Rewards [52.42920996842378]
  Most unified large multimodal models (LMMs) that support both visual understanding and image generation rely on curated post-training supervision.<n>We propose a self-evolving training framework with three internal roles: a Proposer that generates visual questions, a solver that answers and evaluates them, and a Generator that synthesizes images.<n>For image generation, we design a multi-scale internal evaluation scheme that combines question-answer fidelity scoring with cycle-consistent captioning.
  arXiv  Detail & Related papers  (2026-06-25T17:59:57Z)
• Adapting Vision-Language Models from Iconic to Inclusive for Multi-Label Recognition Without Labels [14.373553090930612]
  Vision-language models (VLMs) enable zero-shot recognition without labeled data.<n>These models often prioritize the most iconic object and omit other contextual positives.<n>We propose an unsupervised framework that adapts VLMs from iconic recognition toward inclusive understanding.
  arXiv  Detail & Related papers  (2026-06-10T03:41:50Z)
• reward-lens: A Mechanistic Interpretability Library for Reward Models [0.0]
  We present reward-lens, an open-source library that ports this toolkit to reward models.<n>The library provides a Reward Lens, component attribution, three-mode activation patching, a reward-hacking probe suite, TopK SAE feature attribution.<n>We validate on two production reward models across 695 RewardBench pairs.
  arXiv  Detail & Related papers  (2026-04-28T21:38:10Z)
• MAR-MAER: Metric-Aware and Ambiguity-Adaptive Autoregressive Image Generation [1.4552327135549117]
  We introduce MAR-MAER, an innovative hierarchical autoregressive framework.<n>It combines two main components: It is a metric-aware embedding regularization method, and it is a latent model used for handling ambiguous semantics.<n>Our method aligns the model's internal representations with human-preferred quality metrics, such as CLIPScore and HPSv2.<n>It achieves excellent performance in both metric consistency and semantic flexibility.
  arXiv  Detail & Related papers  (2026-04-02T10:19:46Z)
• Kelix Technical Report [86.64551727600104]
  We present Kelix, a fully discrete autoregressive unified model that closes the understanding gap between discrete and continuous visual representations.<n>Recent work has explored discrete visual tokenization to enable fully autoregressive multimodal modeling.
  arXiv  Detail & Related papers  (2026-02-10T14:48:26Z)
• EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewards [52.42920996842378]
  We propose a self-evolving framework, named EvoLMM, that instantiates two cooperative agents from a single backbone model.<n>This dynamic feedback encourages both the generation of informative queries and the refinement of structured reasoning.<n>Our code and models are available at https://github.com/mbzuai-oryx/EvoLMM.
  arXiv  Detail & Related papers  (2025-11-20T18:59:54Z)
• Prefilled responses enhance zero-shot detection of AI-generated images [2.6581858762749997]
  We explore pre-trained Vision-Language Models (VLMs) for zero-shot detection of AI-generated images.<n>We evaluate VLM performance on three benchmarks encompassing synthetic images of human faces, objects, and animals.<n>In particular, prefilling a VLM response with the task-aligned phrase "Let's examine the style and the synthesis artifacts" improves the Macro F1 scores of three widely used open-source VLMs by up to 24%.
  arXiv  Detail & Related papers  (2025-05-20T22:44:04Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.