Are They the Same Picture? Adapting Concept Bottleneck Models for Human-AI Collaboration in Image Retrieval

• URL: http://arxiv.org/abs/2407.08908v1
• Date: Fri, 12 Jul 2024 00:59:32 GMT
• Title: Are They the Same Picture? Adapting Concept Bottleneck Models for Human-AI Collaboration in Image Retrieval
• Authors: Vaibhav Balloli, Sara Beery, Elizabeth Bondi-Kelly,
• Abstract summary: textttCHAIR enables humans to correct intermediate concepts, which helps textitimprove embeddings generated. We show that our method performs better than similar models on image retrieval metrics without any external intervention.
• Score: 3.2495565849970016
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Image retrieval plays a pivotal role in applications from wildlife conservation to healthcare, for finding individual animals or relevant images to aid diagnosis. Although deep learning techniques for image retrieval have advanced significantly, their imperfect real-world performance often necessitates including human expertise. Human-in-the-loop approaches typically rely on humans completing the task independently and then combining their opinions with an AI model in various ways, as these models offer very little interpretability or \textit{correctability}. To allow humans to intervene in the AI model instead, thereby saving human time and effort, we adapt the Concept Bottleneck Model (CBM) and propose \texttt{CHAIR}. \texttt{CHAIR} (a) enables humans to correct intermediate concepts, which helps \textit{improve} embeddings generated, and (b) allows for flexible levels of intervention that accommodate varying levels of human expertise for better retrieval. To show the efficacy of \texttt{CHAIR}, we demonstrate that our method performs better than similar models on image retrieval metrics without any external intervention. Furthermore, we also showcase how human intervention helps further improve retrieval performance, thereby achieving human-AI complementarity.

Related papers

• Safeguard Text-to-Image Diffusion Models with Human Feedback Inversion [51.931083971448885]
  We propose a framework named Human Feedback Inversion (HFI), where human feedback on model-generated images is condensed into textual tokens guiding the mitigation or removal of problematic images. Our experimental results demonstrate our framework significantly reduces objectionable content generation while preserving image quality, contributing to the ethical deployment of AI in the public sphere.
  arXiv  Detail & Related papers  (2024-07-17T05:21:41Z)
• Boost Your Own Human Image Generation Model via Direct Preference Optimization with AI Feedback [5.9726297901501475]
  We introduce a novel approach tailored specifically for human image generation utilizing Direct Preference Optimization (DPO) Specifically, we introduce an efficient method for constructing a specialized DPO dataset for training human image generation models without the need for costly human feedback. Our method demonstrates its versatility and effectiveness in generating human images, including personalized text-to-image generation.
  arXiv  Detail & Related papers  (2024-05-30T16:18:05Z)
• HINT: Learning Complete Human Neural Representations from Limited Viewpoints [69.76947323932107]
  We propose a NeRF-based algorithm able to learn a detailed and complete human model from limited viewing angles. As a result, our method can reconstruct complete humans even from a few viewing angles, increasing performance by more than 15% PSNR.
  arXiv  Detail & Related papers  (2024-05-30T05:43:09Z)
• Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaption [64.07607726562841]
  Existing multi-person human reconstruction approaches mainly focus on recovering accurate poses or avoiding penetration. In this work, we tackle the task of reconstructing closely interactive humans from a monocular video. We propose to leverage knowledge from proxemic behavior and physics to compensate the lack of visual information.
  arXiv  Detail & Related papers  (2024-04-17T11:55:45Z)
• Self-Explainable Affordance Learning with Embodied Caption [63.88435741872204]
  We introduce Self-Explainable Affordance learning (SEA) with embodied caption. SEA enables robots to articulate their intentions and bridge the gap between explainable vision-language caption and visual affordance learning. We propose a novel model to effectively combine affordance grounding with self-explanation in a simple but efficient manner.
  arXiv  Detail & Related papers  (2024-04-08T15:22:38Z)
• Multi Positive Contrastive Learning with Pose-Consistent Generated Images [0.873811641236639]
  We propose the generation of visually distinct images with identical human poses. We then propose a novel multi-positive contrastive learning, which optimally utilize the previously generated images. Despite using only less than 1% amount of data compared to current state-of-the-art method, GenPoCCL captures structural features of the human body more effectively.
  arXiv  Detail & Related papers  (2024-04-04T07:26:26Z)
• Enhancing Image Caption Generation Using Reinforcement Learning with Human Feedback [0.0]
  We explore a potential method to amplify the performance of the Deep Neural Network Model to generate captions that are preferred by humans. This was achieved by integrating Supervised Learning and Reinforcement Learning with Human Feedback. We provide a sketch of our approach and results, hoping to contribute to the ongoing advances in the field of human-aligned generative AI models.
  arXiv  Detail & Related papers  (2024-03-11T13:57:05Z)
• UniHuman: A Unified Model for Editing Human Images in the Wild [49.896715833075106]
  We propose UniHuman, a unified model that addresses multiple facets of human image editing in real-world settings. To enhance the model's generation quality and generalization capacity, we leverage guidance from human visual encoders. In user studies, UniHuman is preferred by the users in an average of 77% of cases.
  arXiv  Detail & Related papers  (2023-12-22T05:00:30Z)
• Exploring the Robustness of Human Parsers Towards Common Corruptions [99.89886010550836]
  We construct three corruption robustness benchmarks, termed LIP-C, ATR-C, and Pascal-Person-Part-C, to assist us in evaluating the risk tolerance of human parsing models. Inspired by the data augmentation strategy, we propose a novel heterogeneous augmentation-enhanced mechanism to bolster robustness under commonly corrupted conditions.
  arXiv  Detail & Related papers  (2023-09-02T13:32:14Z)
• ArtWhisperer: A Dataset for Characterizing Human-AI Interactions in Artistic Creations [26.4215586218117]
  This work investigates how people use text-to-image models to generate desired target images. We created ArtWhisperer, an online game where users are given a target image and are tasked with iteratively finding a prompt that creates a similar-looking image as the target. We recorded over 50,000 human-AI interactions; each interaction corresponds to one text prompt created by a user and the corresponding generated image.
  arXiv  Detail & Related papers  (2023-06-13T21:10:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.