Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language Models

• URL: http://arxiv.org/abs/2603.15940v1
• Date: Mon, 16 Mar 2026 21:46:59 GMT
• Title: Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language Models
• Abstract summary: Prior object-hiding attacks rely on suppressing or blocking region-specific representations.<n>We propose a new class of emphbackground-consistent object concealment attacks.<n>Our approach preserves token structure and attention flow, avoiding representational voids that trigger hallucination.
• Score: 3.8409038268900404
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) have recently shown remarkable capabilities in visual understanding and generation, but remain vulnerable to adversarial manipulations of visual content. Prior object-hiding attacks primarily rely on suppressing or blocking region-specific representations, often creating semantic gaps that inadvertently induce hallucination, where models invent plausible but incorrect objects. In this work, we demonstrate that hallucination arises not from object absence per se, but from semantic discontinuity introduced by such suppression-based attacks. We propose a new class of \emph{background-consistent object concealment} attacks, which hide target objects by re-encoding their visual representations to be statistically and semantically consistent with surrounding background regions. Crucially, our approach preserves token structure and attention flow, avoiding representational voids that trigger hallucination. We present a pixel-level optimization framework that enforces background-consistent re-encoding across multiple transformer layers while preserving global scene semantics. Extensive experiments on state-of-the-art vision-language models show that our method effectively conceals target objects while preserving up to $86\%$ of non-target objects and reducing grounded hallucination by up to $3\times$ compared to attention-suppression-based attacks.

Related papers

• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination [23.128973540926552]
  We find that both real and hallucinated objects receive equally strong visual attention in the model's mid-to-late layers.<n>We propose a training-free Detect-Mitigate framework comprising a Logit-Lens Consistency Check to detect hallucination.
  arXiv  Detail & Related papers  (2026-08-07T14:56:12Z)
• See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs [81.71384150441163]
  We propose Context-aware Attention Intervention (CAI) for large vision-language models (LVLMs)<n>CAI enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene.<n> experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation.
  arXiv  Detail & Related papers  (2026-06-29T06:35:47Z)
• Object Hallucination-Free Reinforcement Unlearning for Vision-Language Models [34.64078380392795]
  Vision-language models (VLMs) raise growing concerns about privacy, copyright, and bias, motivating machine unlearning to remove sensitive knowledge.<n>We propose HFRU, a reinforcement unlearning framework that operates on the vision encoder for deep semantic removal.
  arXiv  Detail & Related papers  (2026-05-08T17:19:53Z)
• Bridging Day and Night: Target-Class Hallucination Suppression in Unpaired Image Translation [37.38870850999494]
  Day-to-night unpaired image translation is challenging due to large appearance shifts and lack of direct pixel-level supervision.<n>Existing methods often introduce semantic hallucinations, where objects from target classes such as traffic signs and vehicles, as well as man-made light effects, are incorrectly synthesized.<n>We propose a novel framework that detects and suppresses hallucinations of target-class features during unpaired translation.
  arXiv  Detail & Related papers  (2026-02-17T06:37:00Z)
• What Makes "Good" Distractors for Object Hallucination Evaluation in Large Vision-Language Models? [95.46087552542998]
  This paper introduces the Hallucination searching-based Object Probing Evaluation benchmark.<n>It aims to generate the most misleading distractors that can trigger hallucination in Large Vision-Language Models.<n> Experimental results show that HOPE leads to a precision drop of at least 9% and up to 23% across various state-of-the-art LVLMs.
  arXiv  Detail & Related papers  (2025-08-03T03:11:48Z)
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs [98.27348724529257]
  We introduce ViCrit (Visual Caption Hallucination Critic), an RL proxy task that trains VLMs to localize a subtle, synthetic visual hallucination injected into paragraphs of human-written image captions.<n>Models trained with the ViCrit Task exhibit substantial gains across a variety of vision-language models benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T19:16:54Z)
• Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow [32.039946174953236]
  Large vision-language models show tremendous potential in understanding visual information through human languages.<n>They are prone to suffer from object hallucination, i.e., the generated image descriptions contain objects that do not exist in the image.<n>We propose Variational Information Bottleneck (VIB) to alleviate overconfidence by introducing hallucination noise.
  arXiv  Detail & Related papers  (2025-02-28T05:56:23Z)
• The Role of Background Information in Reducing Object Hallucination in Vision-Language Models: Insights from Cutoff API Prompting [7.813585591163111]
  This study analyzes success and failure cases of Attention-driven visual prompting in object hallucination.<n>It shows that preserving background context is crucial for mitigating object hallucination.
  arXiv  Detail & Related papers  (2025-02-21T11:26:21Z)
• Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities [5.602853217226167]
  We present Context-Aware Object Similarities (CAOS), a novel approach for evaluating object hallucination in large vision-language models (LVLMs)<n>CAOS integrates object statistics with semantic relationships between objects in captions and ground-truth data.<n>To address this, we further employ language model-based object recognition to detect potentially out-of-domain hallucinated objects.
  arXiv  Detail & Related papers  (2025-01-25T03:03:18Z)
• ZoomNeXt: A Unified Collaborative Pyramid Network for Camouflaged Object Detection [70.11264880907652]
  Recent object (COD) attempts to segment objects visually blended into their surroundings, which is extremely complex and difficult in real-world scenarios. We propose an effective unified collaborative pyramid network that mimics human behavior when observing vague images and camouflaged zooming in and out. Our framework consistently outperforms existing state-of-the-art methods in image and video COD benchmarks.
  arXiv  Detail & Related papers  (2023-10-31T06:11:23Z)
• Spotlight Attention: Robust Object-Centric Learning With a Spatial Locality Prior [88.9319150230121]
  Object-centric vision aims to construct an explicit representation of the objects in a scene. We incorporate a spatial-locality prior into state-of-the-art object-centric vision models. We obtain significant improvements in segmenting objects in both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-05-31T04:35:50Z)
• Attack to Fool and Explain Deep Networks [59.97135687719244]
  We counter-argue by providing evidence of human-meaningful patterns in adversarial perturbations. Our major contribution is a novel pragmatic adversarial attack that is subsequently transformed into a tool to interpret the visual models.
  arXiv  Detail & Related papers  (2021-06-20T03:07:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.