Reflective Human-Machine Co-adaptation for Enhanced Text-to-Image Generation Dialogue System

• URL: http://arxiv.org/abs/2409.07464v1
• Date: Tue, 27 Aug 2024 18:08:00 GMT
• Title: Reflective Human-Machine Co-adaptation for Enhanced Text-to-Image Generation Dialogue System
• Authors: Yuheng Feng, Yangfan He, Yinghui Xia, Tianyu Shi, Jun Wang, Jinsong Yang,
• Abstract summary: We propose a reflective human-machine co-adaptation strategy, named RHM-CAS. externally, the Agent engages in meaningful language interactions with users to reflect on and refine the generated images. Internally, the Agent tries to optimize the policy based on user preferences, ensuring that the final outcomes closely align with user preferences.
• Score: 7.009995656535664
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Today's image generation systems are capable of producing realistic and high-quality images. However, user prompts often contain ambiguities, making it difficult for these systems to interpret users' potential intentions. Consequently, machines need to interact with users multiple rounds to better understand users' intents. The unpredictable costs of using or learning image generation models through multiple feedback interactions hinder their widespread adoption and full performance potential, especially for non-expert users. In this research, we aim to enhance the user-friendliness of our image generation system. To achieve this, we propose a reflective human-machine co-adaptation strategy, named RHM-CAS. Externally, the Agent engages in meaningful language interactions with users to reflect on and refine the generated images. Internally, the Agent tries to optimize the policy based on user preferences, ensuring that the final outcomes closely align with user preferences. Various experiments on different tasks demonstrate the effectiveness of the proposed method.

Related papers

• What Do You Want? User-centric Prompt Generation for Text-to-image Synthesis via Multi-turn Guidance [23.411806572667707]
  Text-to-image synthesis (TIS) models heavily rely on the quality and specificity of textual prompts. Existing solutions relieve this via automatic model-preferred prompt generation from user queries. We propose DialPrompt, a multi-turn dialogue-based TIS prompt generation model that emphasises user-centricity.
  arXiv  Detail & Related papers  (2024-08-23T08:35:35Z)
• Prompt Refinement with Image Pivot for Text-to-Image Generation [103.63292948223592]
  We introduce Prompt Refinement with Image Pivot (PRIP) for text-to-image generation. PRIP decomposes refinement process into two data-rich tasks: inferring representations of user-preferred images from user languages and translating image representations into system languages. It substantially outperforms a wide range of baselines and effectively transfers to unseen systems in a zero-shot manner.
  arXiv  Detail & Related papers  (2024-06-28T22:19:24Z)
• Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation [87.50120181861362]
  VisionPrefer is a high-quality and fine-grained preference dataset that captures multiple preference aspects. We train a reward model VP-Score over VisionPrefer to guide the training of text-to-image generative models and the preference prediction accuracy of VP-Score is comparable to human annotators.
  arXiv  Detail & Related papers  (2024-04-23T14:53:15Z)
• Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents [110.25679611755962]
  Current language model-driven agents often lack mechanisms for effective user participation, which is crucial given the vagueness commonly found in user instructions. We introduce Intention-in-Interaction (IN3), a novel benchmark designed to inspect users' implicit intentions through explicit queries. We empirically train Mistral-Interact, a powerful model that proactively assesses task vagueness, inquires user intentions, and refines them into actionable goals.
  arXiv  Detail & Related papers  (2024-02-14T14:36:30Z)
• RELIC: Investigating Large Language Model Responses using Self-Consistency [58.63436505595177]
  Large Language Models (LLMs) are notorious for blending fact with fiction and generating non-factual content, known as hallucinations. We propose an interactive system that helps users gain insight into the reliability of the generated text.
  arXiv  Detail & Related papers  (2023-11-28T14:55:52Z)
• Human Learning by Model Feedback: The Dynamics of Iterative Prompting with Midjourney [28.39697076030535]
  This paper analyzes the dynamics of the user prompts along such iterations. We show that prompts predictably converge toward specific traits along these iterations. The possibility that users adapt to the model's preference raises concerns about reusing user data for further training.
  arXiv  Detail & Related papers  (2023-11-20T19:28:52Z)
• AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems [112.76941157194544]
  We propose AgentCF for simulating user-item interactions in recommender systems through agent-based collaborative filtering. We creatively consider not only users but also items as agents, and develop a collaborative learning approach that optimize both kinds of agents together. Overall, the optimized agents exhibit diverse interaction behaviors within our framework, including user-item, user-user, item-item, and collective interactions.
  arXiv  Detail & Related papers  (2023-10-13T16:37:14Z)
• Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models [29.057923932305123]
  We present Promptify, an interactive system that supports prompt exploration and refinement for text-to-image generative models. Our user study shows that Promptify effectively facilitates the text-to-image workflow and outperforms an existing baseline tool widely used for text-to-image generation.
  arXiv  Detail & Related papers  (2023-04-18T22:59:11Z)
• FaIRCoP: Facial Image Retrieval using Contrastive Personalization [43.293482565385055]
  Retrieving facial images from attributes plays a vital role in various systems such as face recognition and suspect identification. Existing methods do so by comparing specific characteristics from the user's mental image against the suggested images. We propose a method that uses the user's feedback to label images as either similar or dissimilar to the target image.
  arXiv  Detail & Related papers  (2022-05-28T09:52:09Z)
• Optimizing Interactive Systems via Data-Driven Objectives [70.3578528542663]
  We propose an approach that infers the objective directly from observed user interactions. These inferences can be made regardless of prior knowledge and across different types of user behavior. We introduce Interactive System (ISO), a novel algorithm that uses these inferred objectives for optimization.
  arXiv  Detail & Related papers  (2020-06-19T20:49:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.