Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts

• URL: http://arxiv.org/abs/2609.00293v1
• Date: Mon, 31 Aug 2026 19:37:07 GMT
• Title: Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts
• Abstract summary: We investigate how vision-language models handle context-memory conflicts.<n>Models tend to prefer in-context information about entities which appear in text, but prefer parametric information about entities which appear in images.
• Score: 16.43008975197273
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate how vision-language models (VLMs) handle context-memory conflicts; that is, situations in which the model is given information in context that differs from what was stored parametrically during training. We document asymmetric biases: models tend to prefer in-context information about entities which appear in text, but prefer parametric information about entities which appear in images. We relate this asymmetry to the late representational alignment across modalities, showing that the longer processing time associated with resolving visual entities prevents the suppression of the model's usual factual recall mechanism, thus resulting in more parametric answers. Chain-of-thought reasoning does not appear to resolve the gap, but increasing the amount of visual information in the context does show an effect. These results illustrate the complexity of ensuring consistent behavior as models become increasingly multimodal and retrieval-augmented.

Related papers

• Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge [72.54679781634037]
  We study how the context window shapes a model's mode of learning, shifting it between parametric internalization and contextualization.<n>We propose the Information Paradox, which hypothesizes that abundant relevant information in the training context can reduce the incentive to encode that information parametrically.
  arXiv  Detail & Related papers  (2026-08-12T16:13:05Z)
• The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context [76.36635039699112]
  Large language models (LLMs) are increasingly deployed in context-rich settings where task inputs are often accompanied by long, partially irrelevant context.<n>We find that state-of-the-art models often appear robust to task-irrelevant context at the aggregate level.<n>This aggregate stability, however, masks significant per-example instability.
  arXiv  Detail & Related papers  (2026-07-14T17:01:12Z)
• Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts [74.47786985522762]
  We identify a critical failure mode termed textual inertia, where models tend to blindly adhere to the erroneous text while neglecting conflicting visual evidence.<n>We propose the LogicGraph Perturbation Protocol that structurally injects perturbations into the reasoning chains of diverse LMMs.<n>Results reveal that models successfully self-correct in less than 10% of cases and predominantly succumb to blind textual error propagation.
  arXiv  Detail & Related papers  (2026-01-07T16:39:34Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• Counterfactual-Consistency Prompting for Relative Temporal Understanding in Large Language Models [24.586475741345616]
  We tackle the issue of temporal inconsistency in large language models (LLMs) by proposing a novel counterfactual prompting approach.<n>Our method generates counterfactual questions and enforces collective constraints, enhancing the model's consistency.<n>We evaluate our method on multiple datasets, demonstrating significant improvements in event ordering for explicit and implicit events and temporal commonsense understanding.
  arXiv  Detail & Related papers  (2025-02-17T04:37:07Z)
• Relation Rectification in Diffusion Model [64.84686527988809]
  We introduce a novel task termed Relation Rectification, aiming to refine the model to accurately represent a given relationship it initially fails to generate. We propose an innovative solution utilizing a Heterogeneous Graph Convolutional Network (HGCN) The lightweight HGCN adjusts the text embeddings generated by the text encoder, ensuring the accurate reflection of the textual relation in the embedding space.
  arXiv  Detail & Related papers  (2024-03-29T15:54:36Z)
• Cross-modal Contrastive Learning with Asymmetric Co-attention Network for Video Moment Retrieval [0.17590081165362778]
  Video moment retrieval is a challenging task requiring fine-grained interactions between video and text modalities. Recent work in image-text pretraining has demonstrated that most existing pretrained models suffer from information asymmetry due to the difference in length between visual and textual sequences. We question whether the same problem also exists in the video-text domain with an auxiliary need to preserve both spatial and temporal information.
  arXiv  Detail & Related papers  (2023-12-12T17:00:46Z)
• MTGER: Multi-view Temporal Graph Enhanced Temporal Reasoning over Time-Involved Document [26.26604509399347]
  MTGER is a novel framework for temporal reasoning over time-involved documents. It explicitly models the temporal relationships among facts by multi-view temporal graphs. We show that MTGER gives more consistent answers under question perturbations.
  arXiv  Detail & Related papers  (2023-11-08T16:41:37Z)
• Sparse Relational Reasoning with Object-Centric Representations [78.83747601814669]
  We investigate the composability of soft-rules learned by relational neural architectures when operating over object-centric representations. We find that increasing sparsity, especially on features, improves the performance of some models and leads to simpler relations.
  arXiv  Detail & Related papers  (2022-07-15T14:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.