DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Models

• URL: http://arxiv.org/abs/2603.06302v1
• Date: Fri, 06 Mar 2026 14:07:37 GMT
• Title: DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Models
• Abstract summary: We present DEX-AR (Dynamic Explainability for AutoRegressive models), a novel explainability method.<n>It generates both per-token and sequence-level 2D heatmaps highlighting image regions crucial for the model's textual responses.<n>Our evaluation on ImageNet, VQAv2, and PascalVOC, shows a consistent improvement in both perturbation-based metrics.
• Score: 27.64151438258739
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Vision-Language Models (VLMs) become increasingly sophisticated and widely used, it becomes more and more crucial to understand their decision-making process. Traditional explainability methods, designed for classification tasks, struggle with modern autoregressive VLMs due to their complex token-by-token generation process and intricate interactions between visual and textual modalities. We present DEX-AR (Dynamic Explainability for AutoRegressive models), a novel explainability method designed to address these challenges by generating both per-token and sequence-level 2D heatmaps highlighting image regions crucial for the model's textual responses. The proposed method offers to interpret autoregressive VLMs-including varying importance of layers and generated tokens-by computing layer-wise gradients with respect to attention maps during the token-by-token generation process. DEX-AR introduces two key innovations: a dynamic head filtering mechanism that identifies attention heads focused on visual information, and a sequence-level filtering approach that aggregates per-token explanations while distinguishing between visually-grounded and purely linguistic tokens. Our evaluation on ImageNet, VQAv2, and PascalVOC, shows a consistent improvement in both perturbation-based metrics, using a novel normalized perplexity measure, as well as segmentation-based metrics.

Related papers

• VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions [51.41587958253802]
  We introduce VISion On Request (VISOR), a method that reduces inference cost without discarding visual information.<n>VISOR improves efficiency by sparsifying the interaction between image and text tokens.<n>Experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results.
  arXiv  Detail & Related papers  (2026-03-24T17:58:17Z)
• Molecular Identifier Visual Prompt and Verifiable Reinforcement Learning for Chemical Reaction Diagram Parsing [52.825281124618535]
  Reaction diagram parsing (RxnDP) is critical for extracting chemical synthesis information from literature.<n>Recent Vision-Language Models (VLMs) have emerged as a promising paradigm to automate this complex visual reasoning task.<n>This work enhances VLM-based RxnDP from two complementary perspectives: prompting representation and learning paradigms.
  arXiv  Detail & Related papers  (2026-03-16T09:17:05Z)
• Self-Augmented Visual Contrastive Decoding [13.078702859025482]
  Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal capabilities.<n>LVLMs inherit the tendency to hallucinate from their underlying language models.<n>This study introduces a novel training-free decoding strategy that addresses these limitations.
  arXiv  Detail & Related papers  (2025-10-15T09:03:34Z)
• IAR2: Improving Autoregressive Visual Generation with Semantic-Detail Associated Token Prediction [77.06211178777939]
  IAR2 is an advanced autoregressive framework that enables a hierarchical semantic-detail synthesis process.<n>We show that IAR2 sets a new state-of-the-art for autoregressive image generation, achieving a FID of 1.50 on ImageNet.
  arXiv  Detail & Related papers  (2025-10-08T12:08:21Z)
• Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation [110.03631978640298]
  We present the first systematic investigation into the mechanisms of applying the next-token prediction paradigm to the visual domain.<n>We identify three key properties that hinder the learning of high-level visual semantics.<n>We show that these issues can be effectively addressed by introducing self-supervised objectives during training.
  arXiv  Detail & Related papers  (2025-09-18T17:47:40Z)
• HERO: Rethinking Visual Token Early Dropping in High-Resolution Large Vision-Language Models [60.028070589466445]
  We propose HERO, a framework that integrates content-adaptive token budget allocation with function-aware token selection.<n>This study provides both empirical insights and practical solutions toward efficient inference in HR-LVLMs.
  arXiv  Detail & Related papers  (2025-09-16T13:22:08Z)
• Harmonizing Visual Representations for Unified Multimodal Understanding and Generation [53.01486796503091]
  We present emphHarmon, a unified autoregressive framework that harmonizes understanding and generation tasks with a shared MAR encoder.<n>Harmon achieves state-of-the-art image generation results on the GenEval, MJHQ30K and WISE benchmarks.
  arXiv  Detail & Related papers  (2025-03-27T20:50:38Z)
• A Dual-Directional Context-Aware Test-Time Learning for Text Classification [17.38491257321053]
  We propose the Dynamic Bidirectional Elman Attention Network (DBEAN)<n>DBEAN combines bidirectional temporal modeling and self-attention.<n>It dynamically weights critical input segments and preserves computational efficiency.
  arXiv  Detail & Related papers  (2025-03-19T17:45:13Z)
• Dynamic Token Reduction during Generation for Vision Language Models [11.376359442815986]
  We introduce a dynamic pruning strategy tailored for Vision-Language Models (VLMs)<n>Our approach enables flexible adjustment of pruning rates based on the attention distribution.<n>Our experimental results demonstrate that our method not only reduces computational demands but also maintains the quality of responses.
  arXiv  Detail & Related papers  (2025-01-24T03:20:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.