Exploring Fluent Query Reformulations with Text-to-Text Transformers and
Reinforcement Learning
- URL: http://arxiv.org/abs/2012.10033v1
- Date: Fri, 18 Dec 2020 03:16:37 GMT
- Title: Exploring Fluent Query Reformulations with Text-to-Text Transformers and
Reinforcement Learning
- Authors: Jerry Zikun Chen, Shi Yu, Haoran Wang
- Abstract summary: We explore methods to generate query reformulations by training reformulators using text-to-text transformers.
We apply policy-based reinforcement learning algorithms to further encourage reward learning.
Our framework is demonstrated to be flexible, allowing reward signals to be sourced from different downstream environments.
- Score: 11.205077315939644
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Query reformulation aims to alter potentially noisy or ambiguous text
sequences into coherent ones closer to natural language questions. In this
process, it is also crucial to maintain and even enhance performance in a
downstream environments like question answering when rephrased queries are
given as input. We explore methods to generate these query reformulations by
training reformulators using text-to-text transformers and apply policy-based
reinforcement learning algorithms to further encourage reward learning. Query
fluency is numerically evaluated by the same class of model fine-tuned on a
human-evaluated well-formedness dataset. The reformulator leverages linguistic
knowledge obtained from transfer learning and generates more well-formed
reformulations than a translation-based model in qualitative and quantitative
analysis. During reinforcement learning, it better retains fluency while
optimizing the RL objective to acquire question answering rewards and can
generalize to out-of-sample textual data in qualitative evaluations. Our RL
framework is demonstrated to be flexible, allowing reward signals to be sourced
from different downstream environments such as intent classification.
Related papers
- Advancing Prompt Recovery in NLP: A Deep Dive into the Integration of Gemma-2b-it and Phi2 Models [18.936945999215038]
The design and effectiveness of prompts represent a challenging and relatively untapped field within NLP research.
This paper delves into an exhaustive investigation of prompt recovery methodologies, employing a spectrum of pre-trained language models and strategies.
Through meticulous experimentation and detailed analysis, we elucidate the outstanding performance of the Gemma-2b-it + Phi2 model + Pretrain.
arXiv Detail & Related papers (2024-07-07T02:15:26Z) - RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder
for Language Modeling [79.56442336234221]
We introduce RegaVAE, a retrieval-augmented language model built upon the variational auto-encoder (VAE)
It encodes the text corpus into a latent space, capturing current and future information from both source and target text.
Experimental results on various datasets demonstrate significant improvements in text generation quality and hallucination removal.
arXiv Detail & Related papers (2023-10-16T16:42:01Z) - Self-Convinced Prompting: Few-Shot Question Answering with Repeated
Introspection [13.608076739368949]
We introduce a novel framework that harnesses the potential of large-scale pre-trained language models.
Our framework processes the output of a typical few-shot chain-of-thought prompt, assesses the correctness of the response, scrutinizes the answer, and ultimately produces a new solution.
arXiv Detail & Related papers (2023-10-08T06:36:26Z) - Policy-Gradient Training of Language Models for Ranking [28.7445517835435]
Text retrieval plays a crucial role in incorporating factual knowledge for decision making into language processing pipelines.
Current state-of-the-art text retrieval models leverage pre-trained large language models (LLMs) to achieve competitive performance.
We introduce Neural PG-RANK, a novel training algorithm that learns to rank by instantiating a LLM as a Plackett-Luce ranking policy.
arXiv Detail & Related papers (2023-10-06T17:55:23Z) - Factually Consistent Summarization via Reinforcement Learning with
Textual Entailment Feedback [57.816210168909286]
We leverage recent progress on textual entailment models to address this problem for abstractive summarization systems.
We use reinforcement learning with reference-free, textual entailment rewards to optimize for factual consistency.
Our results, according to both automatic metrics and human evaluation, show that our method considerably improves the faithfulness, salience, and conciseness of the generated summaries.
arXiv Detail & Related papers (2023-05-31T21:04:04Z) - Enhancing Retrieval-Augmented Large Language Models with Iterative
Retrieval-Generation Synergy [164.83371924650294]
We show that strong performance can be achieved by a method we call Iter-RetGen, which synergizes retrieval and generation in an iterative manner.
A model output shows what might be needed to finish a task, and thus provides an informative context for retrieving more relevant knowledge.
Iter-RetGen processes all retrieved knowledge as a whole and largely preserves the flexibility in generation without structural constraints.
arXiv Detail & Related papers (2023-05-24T16:17:36Z) - Reflexion: Language Agents with Verbal Reinforcement Learning [44.85337947858337]
Reflexion is a novel framework to reinforce language agents not by updating weights, but through linguistic feedback.
It is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals.
For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%.
arXiv Detail & Related papers (2023-03-20T18:08:50Z) - Syntax-informed Question Answering with Heterogeneous Graph Transformer [2.139714421848487]
We present a linguistics-informed question answering approach that extends and fine-tunes a pre-trained neural language model.
We illustrate the approach by the addition of syntactic information in the form of dependency and constituency graphic structures connecting tokens and virtual tokens.
arXiv Detail & Related papers (2022-04-01T07:48:03Z) - Enhancing Dialogue Generation via Multi-Level Contrastive Learning [57.005432249952406]
We propose a multi-level contrastive learning paradigm to model the fine-grained quality of the responses with respect to the query.
A Rank-aware (RC) network is designed to construct the multi-level contrastive optimization objectives.
We build a Knowledge Inference (KI) component to capture the keyword knowledge from the reference during training and exploit such information to encourage the generation of informative words.
arXiv Detail & Related papers (2020-09-19T02:41:04Z) - A Controllable Model of Grounded Response Generation [122.7121624884747]
Current end-to-end neural conversation models inherently lack the flexibility to impose semantic control in the response generation process.
We propose a framework that we call controllable grounded response generation (CGRG)
We show that using this framework, a transformer based model with a novel inductive attention mechanism, trained on a conversation-like Reddit dataset, outperforms strong generation baselines.
arXiv Detail & Related papers (2020-05-01T21:22:08Z) - Exploring the Limits of Transfer Learning with a Unified Text-to-Text
Transformer [64.22926988297685]
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP)
In this paper, we explore the landscape of introducing transfer learning techniques for NLP by a unified framework that converts all text-based language problems into a text-to-text format.
arXiv Detail & Related papers (2019-10-23T17:37:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.