Learning Combinatorial Prompts for Universal Controllable Image Captioning

• URL: http://arxiv.org/abs/2303.06338v3
• Date: Wed, 2 Aug 2023 12:56:53 GMT
• Title: Learning Combinatorial Prompts for Universal Controllable Image Captioning
• Abstract summary: Controllable Image Captioning -- generating natural language descriptions about images under the guidance of given control signals -- is one of the most promising directions towards next-generation captioning systems. We propose a novel prompt-based framework for CIC by learning Combinatorial Prompts, dubbed as ComPro. Our ComPro can be further extended to more kinds of combined control signals by concatenating these prompts.
• Score: 46.34624079520254
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Controllable Image Captioning (CIC) -- generating natural language descriptions about images under the guidance of given control signals -- is one of the most promising directions towards next-generation captioning systems. Till now, various kinds of control signals for CIC have been proposed, ranging from content-related control to structure-related control. However, due to the format and target gaps of different control signals, all existing CIC works (or architectures) only focus on one certain control signal, and overlook the human-like combinatorial ability. By ``combinatorial", we mean that our humans can easily meet multiple needs (or constraints) simultaneously when generating descriptions. To this end, we propose a novel prompt-based framework for CIC by learning Combinatorial Prompts, dubbed as ComPro. Specifically, we directly utilize a pretrained language model GPT-2 as our language model, which can help to bridge the gap between different signal-specific CIC architectures. Then, we reformulate the CIC as a prompt-guide sentence generation problem, and propose a new lightweight prompt generation network to generate the combinatorial prompts for different kinds of control signals. For different control signals, we further design a new mask attention mechanism to realize the prompt-based CIC. Due to its simplicity, our ComPro can be further extended to more kinds of combined control signals by concatenating these prompts. Extensive experiments on two prevalent CIC benchmarks have verified the effectiveness and efficiency of our ComPro on both single and combined control signals.

Related papers

• MCP2OSC: Parametric Control by Natural Language [0.8158530638728501]
  This study focuses on exploring parametric OSC (OpenSoundControl) control by natural language prompts.<n>Claude is integrated with the MCP2OSC server effective in generating OSC messages by natural language.<n>Results suggest its potential for a LLM-based universal control mechanism for multimedia devices.
  arXiv  Detail & Related papers  (2025-08-14T07:38:01Z)
• Minimal Impact ControlNet: Advancing Multi-ControlNet Integration [35.40147040893738]
  In current ControlNet training, each control is designed to influence all areas of an image.<n>Silent control signals can suppress the generation of textures in related areas.<n>We propose Minimal Impact ControlNet to address this problem.
  arXiv  Detail & Related papers  (2025-06-02T13:41:43Z)
• CIE: Controlling Language Model Text Generations Using Continuous Signals [19.55268309121786]
  We show how an LM can be finetuned to expect a control vector that is interpolated between a "low" and a "high" token embedding.<n>Our method more reliably exerts response-length control than in-context learning methods or fine-tuning methods that represent the control signal as a discrete signal.
  arXiv  Detail & Related papers  (2025-05-19T17:59:58Z)
• UNIC-Adapter: Unified Image-instruction Adapter with Multi-modal Transformer for Image Generation [64.8341372591993]
  We propose a new approach to unify controllable generation within a single framework. Specifically, we propose the unified image-instruction adapter (UNIC-Adapter) built on the Multi-Modal-Diffusion Transformer architecture. Our UNIC-Adapter effectively extracts multi-modal instruction information by incorporating both conditional images and task instructions.
  arXiv  Detail & Related papers  (2024-12-25T15:19:02Z)
• I2VControl: Disentangled and Unified Video Motion Synthesis Control [11.83645633418189]
  We present a disentangled and unified framework, namely I2VControl, that unifies multiple motion control tasks in image-to-video synthesis. Our approach partitions the video into individual motion units and represents each unit with disentangled control signals. Our methodology seamlessly integrates as a plug-in for pre-trained models and remains agnostic to specific model architectures.
  arXiv  Detail & Related papers  (2024-11-26T04:21:22Z)
• Controllable Contextualized Image Captioning: Directing the Visual Narrative through User-Defined Highlights [28.963204452040813]
  Contextualized Image Captioning (CIC) evolves traditional image captioning into a more complex domain. This paper introduces a novel domain of Controllable Contextualized Image Captioning (Ctrl-CIC) We present two approaches, Prompting-based Controller (P-Ctrl) and Recalibration-based Controller (R-Ctrl) to generate focused captions.
  arXiv  Detail & Related papers  (2024-07-16T07:32:48Z)
• AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generation [24.07613591217345]
  Linguistic control enables effective content creation, but struggles with fine-grained control over image generation. AnyControl develops a novel Multi-Control framework that extracts a unified multi-modal embedding to guide the generation process. This approach enables a holistic understanding of user inputs, and produces high-quality, faithful results under versatile control signals.
  arXiv  Detail & Related papers  (2024-06-27T07:40:59Z)
• Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation [79.8881514424969]
  Text-conditional diffusion models are able to generate high-fidelity images with diverse contents. However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery. We propose Cocktail, a pipeline to mix various modalities into one embedding.
  arXiv  Detail & Related papers  (2023-06-01T17:55:32Z)
• UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild [166.25327094261038]
  We introduce UniControl, a new generative foundation model for controllable condition-to-image (C2I) tasks. UniControl consolidates a wide array of C2I tasks within a singular framework, while still allowing for arbitrary language prompts. trained on nine unique C2I tasks, UniControl demonstrates impressive zero-shot generation abilities.
  arXiv  Detail & Related papers  (2023-05-18T17:41:34Z)
• Latent Prompt Tuning for Text Summarization [95.85520030785139]
  We propose Lotus (shorthand for Latent Prompt Tuning for Summarization), which is a single model that can be applied in both controlled and uncontrolled modes. During training, Lotus learns latent prompt representations from prompts with gold control signals using a contrastive learning objective. Experiments show Lotus in uncontrolled mode consistently improves upon strong (uncontrollable) summarization models across four different summarization datasets.
  arXiv  Detail & Related papers  (2022-11-03T14:18:48Z)
• Learning Visual Representation from Modality-Shared Contrastive Language-Image Pre-training [88.80694147730883]
  We investigate a variety of Modality-Shared Contrastive Language-Image Pre-training (MS-CLIP) frameworks. In studied conditions, we observe that a mostly unified encoder for vision and language signals outperforms all other variations that separate more parameters. Our approach outperforms vanilla CLIP by 1.6 points in linear probing on a collection of 24 downstream vision tasks.
  arXiv  Detail & Related papers  (2022-07-26T05:19:16Z)
• UFC-BERT: Unifying Multi-Modal Controls for Conditional Image Synthesis [65.34414353024599]
  Conditional image synthesis aims to create an image according to some multi-modal guidance. We propose a new two-stage architecture, UFC-BERT, to unify any number of multi-modal controls.
  arXiv  Detail & Related papers  (2021-05-29T04:42:07Z)
• Human-like Controllable Image Captioning with Verb-specific Semantic Roles [15.710374070780077]
  We propose a new control signal for CIC: Verb-specific Semantic Roles (VSR) VSR consists of a verb and some semantic roles, which represents a targeted activity and the roles of entities involved in this activity. Our framework can achieve better controllability than several strong baselines on two challenging CIC benchmarks.
  arXiv  Detail & Related papers  (2021-03-22T22:17:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.