Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement

• URL: http://arxiv.org/abs/2608.03733v1
• Date: Tue, 04 Aug 2026 14:28:42 GMT
• Title: Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement
• Abstract summary: Failure-informed Image Self-Augmentation (textbfFISA) is a framework for MLLM self-improvement that constructs augmented images from the model's own failure cases.<n>Our method generates visually challenging yet answer-preserving image complications, verifies their utility through self-examination, and applies dual fidelity filtering to avoid semantic distortion.
• Score: 50.97806700110183
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal large language models (MLLMs) have achieved remarkable performance across vision-language tasks, but their progress depends heavily on large-scale, high-quality multimodal data that are costly to annotate. Self-augmentation offers a promising alternative by enabling models to expand their own training data without external supervision. However, existing MLLM self-augmentation methods are largely text-centric, while image augmentation remains underexplored and typically relies on generic or handcrafted transformations that are weakly aligned with the model's actual incapability. We propose Failure-informed Image Self-Augmentation (\textbf{FISA}), a framework for MLLM self-improvement that constructs augmented images from the model's own failure cases. Our method generates visually challenging yet answer-preserving image complications, verifies their utility through self-examination, and applies dual fidelity filtering to avoid semantic distortion. Experiments on visual question answering benchmarks show that the proposed method consistently improves performance across both in-distribution and out-of-distribution settings. Further experiments validate the compatibility of FISA with existing textual self-augmentation approaches, the superior data efficiency of the synthesized samples over generic image augmentation baselines, and the practical effectiveness of the proposed filtering strategy.

Related papers

• Enhanced Text-to-Image Generation by Fine-grained Multimodal Reasoning [13.412268665909826]
  Fine-grained Multimodal Reasoning (FiMR) is a framework that leverages visual question answering (VQA) to generate explicit, fine-grained feedback.<n>FiMR consistently outperforms image generation baselines, including reasoning-based methods.
  arXiv  Detail & Related papers  (2026-04-15T05:24:29Z)
• More Images, More Problems? A Controlled Analysis of VLM Failure Modes [80.64323947730905]
  Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities, yet their proficiency in understanding and reasoning over multiple images remains largely unexplored.<n>We introduce MIMIC, a new benchmark designed to rigorously evaluate the multi-image capabilities of LVLMs.
  arXiv  Detail & Related papers  (2026-01-12T18:45:13Z)
• ThinkFake: Reasoning in Multimodal Large Language Models for AI-Generated Image Detection [51.93101033997245]
  Increasing realism of AI-generated images has raised serious concerns about misinformation and privacy violations.<n>We propose ThinkFake, a novel reasoning-based and generalizable framework for AI-generated image detection.<n>We show that ThinkFake outperforms state-of-the-art methods on the GenImage benchmark and demonstrates strong zero-shot generalization on the challenging LOKI benchmark.
  arXiv  Detail & Related papers  (2025-09-24T07:34:09Z)
• A Visual Leap in CLIP Compositionality Reasoning through Generation of Counterfactual Sets [26.167194142428475]
  Vision-language models (VLMs) often struggle with compositional reasoning due to insufficient high-quality image-text data.<n>We propose a novel block-based diffusion approach that automatically generates counterfactual datasets without manual annotation.<n>Our approach achieves state-of-the-art results across multiple benchmarks while using substantially less training data than existing methods.
  arXiv  Detail & Related papers  (2025-07-07T06:47:10Z)
• Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Models [58.936893810674896]
  Face Anti-Spoofing (FAS) is essential for ensuring the security and reliability of facial recognition systems.<n>We introduce a multimodal large language model framework for FAS, termed Interpretable Face Anti-Spoofing (I-FAS)<n>We propose a Spoof-aware Captioning and Filtering (SCF) strategy to generate high-quality captions for FAS images.
  arXiv  Detail & Related papers  (2025-01-03T09:25:04Z)
• Image Augmentation Agent for Weakly Supervised Semantic Segmentation [32.40055370439922]
  Weakly-supervised semantic segmentation (WSSS) has achieved remarkable progress using only image-level labels.<n>We introduce a novel approach called Image Augmentation Agent (IAA) which shows that it is possible to enhance WSSS from data generation perspective.<n>IAA mainly design an augmentation agent that leverages large language models (LLMs) and diffusion models to automatically generate additional images for WSSS.
  arXiv  Detail & Related papers  (2024-12-29T11:32:55Z)
• Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution [43.07899102255169]
  We propose a novel multimodal self-evolution framework that enables the model to autonomously generate high-quality questions and answers.<n>First, we implement an image-driven self-questioning mechanism, allowing the model to create and evaluate questions based on image content.<n>Second, we introduce an answer self-enhancement technique, starting with image captioning to improve answer quality.
  arXiv  Detail & Related papers  (2024-12-20T08:06:00Z)
• MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling [64.09238330331195]
  We propose a novel Multi-Modal Auto-Regressive (MMAR) probabilistic modeling framework.<n>Unlike discretization line of method, MMAR takes in continuous-valued image tokens to avoid information loss in an efficient way.<n>We also propose a theoretically proven technique that addresses the numerical stability issue and a training strategy that balances the generation and understanding task goals.
  arXiv  Detail & Related papers  (2024-10-14T17:57:18Z)
• Enhancing Large Vision Language Models with Self-Training on Image Comprehension [131.14381425260706]
  We introduce Self-Training on Image (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference for image descriptions using unlabeled images. To further self-improve reasoning on the extracted visual information, we let the model reuse a small portion of existing instruction-tuning data.
  arXiv  Detail & Related papers  (2024-05-30T05:53:49Z)
• Debiasing Multimodal Large Language Models via Penalization of Language Priors [38.97645845493758]
  Multimodal Large Language Models (MLLMs) have become indispensable tools in computer vision and natural language processing.<n>Despite their advancements, our investigation reveals a noteworthy bias: the generated content is often driven more by the inherent priors of the underlying Large Language Models (LLMs) than by the input image.<n>We propose two simple, training-free strategies to rectify these biases and redirect the model's focus toward visual information.
  arXiv  Detail & Related papers  (2024-03-08T12:35:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.