Human-like Controllable Image Captioning with Verb-specific Semantic Roles

• URL: http://arxiv.org/abs/2103.12204v1
• Date: Mon, 22 Mar 2021 22:17:42 GMT
• Title: Human-like Controllable Image Captioning with Verb-specific Semantic Roles
• Abstract summary: We propose a new control signal for CIC: Verb-specific Semantic Roles (VSR) VSR consists of a verb and some semantic roles, which represents a targeted activity and the roles of entities involved in this activity. Our framework can achieve better controllability than several strong baselines on two challenging CIC benchmarks.
• Score: 15.710374070780077
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Controllable Image Captioning (CIC) -- generating image descriptions following designated control signals -- has received unprecedented attention over the last few years. To emulate the human ability in controlling caption generation, current CIC studies focus exclusively on control signals concerning objective properties, such as contents of interest or descriptive patterns. However, we argue that almost all existing objective control signals have overlooked two indispensable characteristics of an ideal control signal: 1) Event-compatible: all visual contents referred to in a single sentence should be compatible with the described activity. 2) Sample-suitable: the control signals should be suitable for a specific image sample. To this end, we propose a new control signal for CIC: Verb-specific Semantic Roles (VSR). VSR consists of a verb and some semantic roles, which represents a targeted activity and the roles of entities involved in this activity. Given a designated VSR, we first train a grounded semantic role labeling (GSRL) model to identify and ground all entities for each role. Then, we propose a semantic structure planner (SSP) to learn human-like descriptive semantic structures. Lastly, we use a role-shift captioning model to generate the captions. Extensive experiments and ablations demonstrate that our framework can achieve better controllability than several strong baselines on two challenging CIC benchmarks. Besides, we can generate multi-level diverse captions easily. The code is available at: https://github.com/mad-red/VSR-guided-CIC.

Related papers

• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees [59.16312652369709]
  Contrastive Language-Image Pre-training (CLIP)citepradford2021learning has emerged as a pivotal model in computer vision and multimodal learning.<n>CLIP struggles with potential information misalignment in many image-text datasets and suffers from entangled representation.<n>We introduce ours, a novel approach that identifies and aligns the most relevant visual and textual representations in a modular manner.
  arXiv  Detail & Related papers  (2025-07-29T22:26:20Z)
• Language-Guided Visual Perception Disentanglement for Image Quality Assessment and Conditional Image Generation [48.642826318384294]
  Contrastive vision-language models, such as CLIP, have demonstrated excellent zero-shot capability across semantic recognition tasks. This paper presents a new multimodal disentangled representation learning framework, which leverages disentangled text to guide image disentanglement.
  arXiv  Detail & Related papers  (2025-03-04T02:36:48Z)
• Cross-Modal Bidirectional Interaction Model for Referring Remote Sensing Image Segmentation [50.433911327489554]
  The goal of referring remote sensing image segmentation (RRSIS) is to generate a pixel-level mask of the target object identified by the referring expression.<n>To address the aforementioned challenges, a novel RRSIS framework is proposed, termed the cross-modal bidirectional interaction model (CroBIM)<n>To further forster the research of RRSIS, we also construct RISBench, a new large-scale benchmark dataset comprising 52,472 image-language-label triplets.
  arXiv  Detail & Related papers  (2024-10-11T08:28:04Z)
• Semantic-CC: Boosting Remote Sensing Image Change Captioning via Foundational Knowledge and Semantic Guidance [19.663899648983417]
  We introduce a novel change captioning (CC) method based on the foundational knowledge and semantic guidance. We validate the proposed method on the LEVIR-CC and LEVIR-CD datasets.
  arXiv  Detail & Related papers  (2024-07-19T05:07:41Z)
• Controllable Contextualized Image Captioning: Directing the Visual Narrative through User-Defined Highlights [28.963204452040813]
  Contextualized Image Captioning (CIC) evolves traditional image captioning into a more complex domain. This paper introduces a novel domain of Controllable Contextualized Image Captioning (Ctrl-CIC) We present two approaches, Prompting-based Controller (P-Ctrl) and Recalibration-based Controller (R-Ctrl) to generate focused captions.
  arXiv  Detail & Related papers  (2024-07-16T07:32:48Z)
• CIC-BART-SSA: Controllable Image Captioning with Structured Semantic Augmentation [9.493755431645313]
  We propose a novel, fully automatic method to sample additional focused and visually grounded captions. We leverage Abstract Meaning Representation (AMR) to encode all possible semantic-semantic relations between entities. We then develop a new model, CIC-BART-SSA, that sources its control signals from SSA-diversified datasets.
  arXiv  Detail & Related papers  (2024-07-16T05:26:12Z)
• Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion Models [68.47333676663312]
  We show a simple modification of classifier-free guidance can help disentangle image factors in text-to-image models. The key idea of our method, Contrastive Guidance, is to characterize an intended factor with two prompts that differ in minimal tokens. We illustrate whose benefits in three scenarios: (1) to guide domain-specific diffusion models trained on an object class, (2) to gain continuous, rig-like controls for text-to-image generation, and (3) to improve the performance of zero-shot image editors.
  arXiv  Detail & Related papers  (2024-02-21T03:01:17Z)
• SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment [11.556516260190737]
  Multimodal alignment between language and vision is the fundamental topic in current vision-language model research. This paper proposes Contrastive Captioners (CoCa) to integrate Contrastive Language-Image Pretraining (CLIP) and Image Caption (IC) into a unified framework.
  arXiv  Detail & Related papers  (2024-01-04T08:42:36Z)
• Learning Combinatorial Prompts for Universal Controllable Image Captioning [46.34624079520254]
  Controllable Image Captioning -- generating natural language descriptions about images under the guidance of given control signals -- is one of the most promising directions towards next-generation captioning systems. We propose a novel prompt-based framework for CIC by learning Combinatorial Prompts, dubbed as ComPro. Our ComPro can be further extended to more kinds of combined control signals by concatenating these prompts.
  arXiv  Detail & Related papers  (2023-03-11T07:53:15Z)
• Is a Caption Worth a Thousand Images? A Controlled Study for Representation Learning [88.5382122413913]
  We study whether language supervision can result in vision models with more transferable representations than traditional image-only methods. We find that image-only methods do not match CLIP's transfer performance, even when they are trained with more image data. Motivated by our findings, we devise simple prescriptions to enable CLIP to better leverage the language information present in existing pre-training datasets.
  arXiv  Detail & Related papers  (2022-07-15T17:50:51Z)
• Single-Stream Multi-Level Alignment for Vision-Language Pretraining [103.09776737512078]
  We propose a single stream model that aligns the modalities at multiple levels. We achieve this using two novel tasks: symmetric cross-modality reconstruction and a pseudo-labeled key word prediction. We demonstrate top performance on a set of Vision-Language downstream tasks such as zero-shot/fine-tuned image/text retrieval, referring expression, and VQA.
  arXiv  Detail & Related papers  (2022-03-27T21:16:10Z)
• Learning Self-Supervised Low-Rank Network for Single-Stage Weakly and Semi-Supervised Semantic Segmentation [119.009033745244]
  This paper presents a Self-supervised Low-Rank Network ( SLRNet) for single-stage weakly supervised semantic segmentation (WSSS) and semi-supervised semantic segmentation (SSSS) SLRNet uses cross-view self-supervision, that is, it simultaneously predicts several attentive LR representations from different views of an image to learn precise pseudo-labels. Experiments on the Pascal VOC 2012, COCO, and L2ID datasets demonstrate that our SLRNet outperforms both state-of-the-art WSSS and SSSS methods with a variety of different settings.
  arXiv  Detail & Related papers  (2022-03-19T09:19:55Z)
• CRIS: CLIP-Driven Referring Image Segmentation [71.56466057776086]
  We propose an end-to-end CLIP-Driven Referring Image framework (CRIS) CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. Our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.
  arXiv  Detail & Related papers  (2021-11-30T07:29:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.