Controllable Image Captioning with Prompt-Conditioned Scene Rewards

• URL: http://arxiv.org/abs/2609.00709v1
• Date: Tue, 01 Sep 2026 04:36:23 GMT
• Title: Controllable Image Captioning with Prompt-Conditioned Scene Rewards
• Abstract summary: Fine-grained Captioning Control Using Scene Rewards (FoCUS) is a controllable image captioning method that lets users steer captions toward specific semantic emphases.<n>FoCUS consistently improves controllability and fine-grained caption quality without degrading general caption performance.
• Score: 13.154350707762179
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large Vision-Language Models produce fluent image descriptions but offer limited semantic control: users cannot reliably specify whether captions should emphasize attributes, relations, or particular image regions. We present Fine-grained Captioning Control Using Scene Rewards (FoCUS), a controllable image captioning method that lets users steer captions toward specific semantic emphases through natural-language control prompts. The core idea is a prompt-conditioned control objective based on scene-graph-aligned component scores. Generated captions are parsed and aligned to scene-graph components such as objects, attributes, and relations. These components are differentially weighted, including negative weights, according to the requested emphasis. We optimize this objective with GRPO and further improve its reliability through a stricter object validity threshold and reasoning-based verification for attribute and relation scoring. To evaluate controllability, we introduce Semantic Control and Precision Evaluation (SCoPE), a benchmark with contrastive Include/Avoid constraints for measuring both target content coverage and out-of-scope suppression. Experiments on two VLM backbones show that FoCUS consistently improves controllability and fine-grained caption quality without degrading general caption performance.

Related papers

• CAPEval: A Decoupled Caption Evaluation across Understanding and Generation [29.355541032573285]
  We design a decoupled caption evaluation benchmark, CAPEval, with human-written ground-truth captions and human-verified atomic checklist items.<n>We find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance.
  arXiv  Detail & Related papers  (2026-08-03T17:57:03Z)
• TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment [87.69757277913577]
  We propose TEVI, a framework that uses captions as a signal for what to retain from image embeddings.<n>In a controlled setup with synthetic captions, we show that TEVI is effective at preserving caption-described attributes while discarding others.
  arXiv  Detail & Related papers  (2026-06-05T16:54:40Z)
• CCCaption: Dual-Reward Reinforcement Learning for Complete and Correct Image Captioning [23.289413412387223]
  We introduce CCCaption: a dual-reward reinforcement learning framework with a dedicated fine-tuning corpus.<n>For completeness, we use diverse LVLMs to disentangle the image into a set of visual queries, and reward captions that answer more of these queries.<n>For correctness, we penalize captions that contain hallucinations by validating the authenticity of sub-caption queries.
  arXiv  Detail & Related papers  (2026-02-25T07:34:26Z)
• ASemConsist: Adaptive Semantic Feature Control for Training-Free Identity-Consistent Generation [14.341691123354195]
  ASemconsist enables explicit semantic control over character identity without sacrificing prompt alignment.<n>Our framework achieves state-of-the-art performance, effectively overcoming prior trade-offs.
  arXiv  Detail & Related papers  (2025-12-29T07:06:57Z)
• SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning [21.739084696595427]
  SC-Captioner is a reinforcement learning framework that enables the self-correcting capability of image caption models.<n>We calculate the set difference between sets of initial and self-corrected captions to identify added and removed elements.<n>Experiments show that applying SC-Captioner on large visual-language models can generate better image captions across various scenarios.
  arXiv  Detail & Related papers  (2025-08-08T08:45:52Z)
• Group-based Distinctive Image Captioning with Memory Difference Encoding and Attention [62.246950834745796]
  Group-based Differential Distinctive Captioning Method.<n>Group-based Differential Memory Attention (GDMA) module.<n>New evaluation metric, the Distinctive Word Rate (DisWordRate)
  arXiv  Detail & Related papers  (2025-04-03T11:19:51Z)
• Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioning [77.2852342808769]
  In this paper, we introduce a detailed caption benchmark, termed as CompreCap, to evaluate the visual context from a directed scene graph view.<n>We first manually segment the image into semantically meaningful regions according to common-object vocabulary, while also distinguishing attributes of objects within all those regions.<n>Then directional relation labels of these objects are annotated to compose a directed scene graph that can well encode rich compositional information of the image.
  arXiv  Detail & Related papers  (2024-12-11T18:37:42Z)
• What Makes for Good Image Captions? [50.48589893443939]
  Our framework posits that good image captions should balance three key aspects: informationally sufficient, minimally redundant, and readily comprehensible by humans.<n>We introduce the Pyramid of Captions (PoCa) method, which generates enriched captions by integrating local and global visual information.
  arXiv  Detail & Related papers  (2024-05-01T12:49:57Z)
• Intrinsic Image Captioning Evaluation [53.51379676690971]
  We propose a learning based metrics for image captioning, which we call Intrinsic Image Captioning Evaluation(I2CE) Experiment results show that our proposed method can keep robust performance and give more flexible scores to candidate captions when encountered with semantic similar expression or less aligned semantics.
  arXiv  Detail & Related papers  (2020-12-14T08:36:05Z)
• More Grounded Image Captioning by Distilling Image-Text Matching Model [56.79895670335411]
  We propose a Part-of-Speech (POS) enhanced image-text matching model (SCAN) as the effective knowledge distillation for more grounded image captioning. The benefits are two-fold: 1) given a sentence and an image, POS-SCAN can ground the objects more accurately than SCAN; 2) POS-SCAN serves as a word-region alignment regularization for the captioner's visual attention module.
  arXiv  Detail & Related papers  (2020-04-01T12:42:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.