A Deep Reinforcement Learning Approach for Interactive Search with Sentence-level Feedback

• URL: http://arxiv.org/abs/2310.03043v1
• Date: Tue, 3 Oct 2023 18:45:21 GMT
• Title: A Deep Reinforcement Learning Approach for Interactive Search with Sentence-level Feedback
• Authors: Jianghong Zhou, Joyce C. Ho, Chen Lin, Eugene Agichtein
• Abstract summary: Interactive search can provide a better experience by incorporating interaction feedback from the users. Existing state-of-the-art (SOTA) systems use reinforcement learning (RL) models to incorporate the interactions. Yet such feedback requires extensive RL action space exploration and large amounts of annotated data. This work proposes a new deep Q-learning (DQ) approach, DQrank.
• Score: 12.712416630402119
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Interactive search can provide a better experience by incorporating interaction feedback from the users. This can significantly improve search accuracy as it helps avoid irrelevant information and captures the users' search intents. Existing state-of-the-art (SOTA) systems use reinforcement learning (RL) models to incorporate the interactions but focus on item-level feedback, ignoring the fine-grained information found in sentence-level feedback. Yet such feedback requires extensive RL action space exploration and large amounts of annotated data. This work addresses these challenges by proposing a new deep Q-learning (DQ) approach, DQrank. DQrank adapts BERT-based models, the SOTA in natural language processing, to select crucial sentences based on users' engagement and rank the items to obtain more satisfactory responses. We also propose two mechanisms to better explore optimal actions. DQrank further utilizes the experience replay mechanism in DQ to store the feedback sentences to obtain a better initial ranking performance. We validate the effectiveness of DQrank on three search datasets. The results show that DQRank performs at least 12% better than the previous SOTA RL approaches. We also conduct detailed ablation studies. The ablation results demonstrate that each model component can efficiently extract and accumulate long-term engagement effects from the users' sentence-level feedback. This structure offers new technologies with promised performance to construct a search system with sentence-level interaction.

Related papers

• UltraFeedback: Boosting Language Models with Scaled AI Feedback [99.4633351133207]
  We present textscUltraFeedback, a large-scale, high-quality, and diversified AI feedback dataset. Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models.
  arXiv  Detail & Related papers  (2023-10-02T17:40:01Z)
• System-Level Natural Language Feedback [83.24259100437965]
  We show how to use feedback to formalize system-level design decisions in a human-in-the-loop-process. We conduct two case studies of this approach for improving search query and dialog response generation. We show the combination of system-level and instance-level feedback brings further gains.
  arXiv  Detail & Related papers  (2023-06-23T16:21:40Z)
• RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generation [30.245143345565758]
  We propose a new retrieval-enhanced approach for personalized response generation. We design a hierarchical transformer retriever trained on dialogue domain data to perform personalized retrieval and a context-aware prefix encoder that fuses the retrieved information to the decoder more effectively. We quantitatively evaluate our model's performance under a suite of human and automatic metrics and find it to be superior compared to state-of-the-art baselines on English Reddit conversations.
  arXiv  Detail & Related papers  (2023-06-12T16:10:21Z)
• Task Oriented Conversational Modelling With Subjective Knowledge [0.0]
  DSTC-11 proposes a three stage pipeline consisting of knowledge seeking turn detection, knowledge selection and response generation. We propose entity retrieval methods which result in an accurate and faster knowledge search. Preliminary results show a 4 % improvement in exact match score on knowledge selection task.
  arXiv  Detail & Related papers  (2023-03-30T20:23:49Z)
• Incorporating Relevance Feedback for Information-Seeking Retrieval using Few-Shot Document Re-Ranking [56.80065604034095]
  We introduce a kNN approach that re-ranks documents based on their similarity with the query and the documents the user considers relevant. To evaluate our different integration strategies, we transform four existing information retrieval datasets into the relevance feedback scenario.
  arXiv  Detail & Related papers  (2022-10-19T16:19:37Z)
• Keyword Extraction for Improved Document Retrieval in Conversational Search [10.798537120200006]
  Mixed-initiative conversational search provides enormous advantages. incorporating additional information provided by the user from the conversation poses some challenges. We have collected two conversational keyword extraction datasets and propose an end-to-end document retrieval pipeline incorporating them.
  arXiv  Detail & Related papers  (2021-09-13T13:55:37Z)
• Leveraging Historical Interaction Data for Improving Conversational Recommender System [105.90963882850265]
  We propose a novel pre-training approach to integrate item- and attribute-based preference sequence. Experiment results on two real-world datasets have demonstrated the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-08-19T03:43:50Z)
• Mining Implicit Relevance Feedback from User Behavior for Web Question Answering [92.45607094299181]
  We make the first study to explore the correlation between user behavior and passage relevance. Our approach significantly improves the accuracy of passage ranking without extra human labeled data. In practice, this work has proved effective to substantially reduce the human labeling cost for the QA service in a global commercial search engine.
  arXiv  Detail & Related papers  (2020-06-13T07:02:08Z)
• Open-Retrieval Conversational Question Answering [62.11228261293487]
  We introduce an open-retrieval conversational question answering (ORConvQA) setting, where we learn to retrieve evidence from a large collection before extracting answers. We build an end-to-end system for ORConvQA, featuring a retriever, a reranker, and a reader that are all based on Transformers.
  arXiv  Detail & Related papers  (2020-05-22T19:39:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.