Entity-Faithful Repair of Synthetic Supervision for Zero-Shot Image Captioning

• URL: http://arxiv.org/abs/2608.00994v1
• Date: Sun, 02 Aug 2026 04:43:04 GMT
• Title: Entity-Faithful Repair of Synthetic Supervision for Zero-Shot Image Captioning
• Abstract summary: Zero-shot image captioning aims to generate descriptions without annotated image-text pairs.<n>Recent approaches exploit text-to-image models to synthesize training data from text-only corpora.<n>ReCap is a plug-and-play framework that shifts synthetic data refinement from implicit global matching to explicit fine-grained realignment.
• Score: 5.694066916138962
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Zero-shot image captioning aims to generate image descriptions without annotated image-text pairs. Recent approaches exploit text-to-image models to synthesize training data from text-only corpora, but most focus on improving overall data quality. In contrast, we observe that synthetic image-text misalignment is often structured and fine-grained: pairs may remain globally plausible while containing missing entities or misgrounded attributes, thereby degrading supervision fidelity. As a result, methods based on global similarity for image rematching or regeneration may improve apparent plausibility, but cannot systematically repair entity-level misalignment. To address this issue, we propose ReCap, a plug-and-play framework that shifts synthetic data refinement from implicit global matching to explicit fine-grained realignment. Specifically, ReCap enforces entity-level correspondence by using detected image-supported entities to guide caption rewriting, yielding more faithful synthetic supervision. In addition, we introduce an adaptive dynamic weighted learning strategy to downweight unreliable synthetic pairs during training. As a general framework, ReCap can be integrated into existing synthetic-data pipelines. Extensive experiments show that ReCap consistently improves image-text consistency and achieves state-of-the-art performance on both in-domain and cross-domain zero-shot image captioning benchmarks.

Related papers

• PRISM: Prior Rectification and Uncertainty-Aware Structure Modeling for Diffusion-Based Text Image Super-Resolution [51.96078493242164]
  PRISM is a single-step diffusion-based Text-SR framework.<n> PRISM achieves state-of-the-art performance with millisecond-level inference.
  arXiv  Detail & Related papers  (2026-05-13T05:31:06Z)
• CycleCap: Improving VLMs Captioning Performance via Self-Supervised Cycle Consistency Fine-Tuning [65.10059440725041]
  Visual-Language Models (VLMs) have achieved remarkable progress in image captioning, visual question answering, and visual reasoning.<n>They remain prone to vision-language misalignment, often producing overly generic or hallucinated descriptions.<n>Existing approaches address this via instruction tuning-requiring costly, large-scale annotated datasets or via complex test-time frameworks for caption refinement.<n>In this work, we revisit image-text alignment through the lens of cycle consistency.
  arXiv  Detail & Related papers  (2026-03-18T20:57:31Z)
• SynC: Synthetic Image Caption Dataset Refinement with One-to-many Mapping for Zero-shot Image Captioning [5.23086948974839]
  Zero-shot Image Captioning (ZIC) increasingly utilizes synthetic datasets generated by text-to-image (T2I) models.<n>Existing dataset pruning techniques are largely designed for removing noisy text in web-crawled data.<n>We introduce SynC, a novel framework specifically designed to refine synthetic image-caption datasets for ZIC.
  arXiv  Detail & Related papers  (2025-07-24T17:53:26Z)
• Unleashing Text-to-Image Diffusion Prior for Zero-Shot Image Captioning [70.98890307376548]
  We propose a novel Patch-wise Cross-modal feature Mix-up (PCM) mechanism to adaptively mitigate the unfaithful contents during training.<n>Our PCM-Net ranks first in both in-domain and cross-domain zero-shot image captioning.
  arXiv  Detail & Related papers  (2024-12-31T13:39:08Z)
• Text-only Synthesis for Image Captioning [26.774411180980994]
  We propose Text-only Synthesis for Image Captioning (ToCa) We deconstruct caption text into structures and lexical words. Massive captions that contain various patterns of lexical words are generated.
  arXiv  Detail & Related papers  (2024-05-28T15:11:17Z)
• PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis [62.29033292210752]
  High-quality images with consistent semantics and layout remains a challenge. We propose the adaPtive LAyout-semantiC fusion modulE (PLACE) that harnesses pre-trained models to alleviate the aforementioned issues. Our approach performs favorably in terms of visual quality, semantic consistency, and layout alignment.
  arXiv  Detail & Related papers  (2024-03-04T09:03:16Z)
• StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis [52.341186561026724]
  Lacking compositionality could have severe implications for robustness and fairness. We introduce a new framework, StyleT2I, to improve the compositionality of text-to-image synthesis. Results show that StyleT2I outperforms previous approaches in terms of consistency between the input text and synthesized images.
  arXiv  Detail & Related papers  (2022-03-29T17:59:50Z)
• USIS: Unsupervised Semantic Image Synthesis [9.613134538472801]
  We propose a new Unsupervised paradigm for Semantic Image Synthesis (USIS) USIS learns to output images with visually separable semantic classes using a self-supervised segmentation loss. In order to match the color and texture distribution of real images without losing high-frequency information, we propose to use whole image wavelet-based discrimination.
  arXiv  Detail & Related papers  (2021-09-29T20:48:41Z)
• You Only Need Adversarial Supervision for Semantic Image Synthesis [84.83711654797342]
  We propose a novel, simplified GAN model, which needs only adversarial supervision to achieve high quality results. We show that images synthesized by our model are more diverse and follow the color and texture of real images more closely.
  arXiv  Detail & Related papers  (2020-12-08T23:00:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.