A Simple and Plug-and-play Method for Unsupervised Sentence
Representation Enhancement
- URL: http://arxiv.org/abs/2305.07824v1
- Date: Sat, 13 May 2023 02:43:59 GMT
- Title: A Simple and Plug-and-play Method for Unsupervised Sentence
Representation Enhancement
- Authors: Lingfeng Shen, Haiyun Jiang, Lemao Liu, Shuming Shi
- Abstract summary: RepAL is an extremely simple post-processing method that enhances sentence representations.
We show that RepAL is free of training and is a plug-and-play method that can be combined with most existing unsupervised sentence learning models.
- Score: 35.6803390044542
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Generating proper embedding of sentences through an unsupervised way is
beneficial to semantic matching and retrieval problems in real-world scenarios.
This paper presents Representation ALchemy (RepAL), an extremely simple
post-processing method that enhances sentence representations. The basic idea
in RepAL is to de-emphasize redundant information of sentence embedding
generated by pre-trained models. Through comprehensive experiments, we show
that RepAL is free of training and is a plug-and-play method that can be
combined with most existing unsupervised sentence learning models. We also
conducted in-depth analysis to understand RepAL.
Related papers
- Self-regulating Prompts: Foundational Model Adaptation without
Forgetting [112.66832145320434]
We introduce a self-regularization framework for prompting called PromptSRC.
PromptSRC guides the prompts to optimize for both task-specific and task-agnostic general representations.
arXiv Detail & Related papers (2023-07-13T17:59:35Z) - Alleviating Over-smoothing for Unsupervised Sentence Representation [96.19497378628594]
We present a Simple method named Self-Contrastive Learning (SSCL) to alleviate this issue.
Our proposed method is quite simple and can be easily extended to various state-of-the-art models for performance boosting.
arXiv Detail & Related papers (2023-05-09T11:00:02Z) - Learning with Rejection for Abstractive Text Summarization [42.15551472507393]
We propose a training objective for abstractive summarization based on rejection learning.
We show that our method considerably improves the factuality of generated summaries in automatic and human evaluations.
arXiv Detail & Related papers (2023-02-16T19:07:08Z) - Learning Non-Autoregressive Models from Search for Unsupervised Sentence
Summarization [20.87460375478907]
Text summarization aims to generate a short summary for an input text.
In this work, we propose a Non-Autoregressive Unsupervised Summarization approach.
Experiments show that NAUS achieves state-of-the-art performance for unsupervised summarization.
arXiv Detail & Related papers (2022-05-28T21:09:23Z) - Learning to Ask Conversational Questions by Optimizing Levenshtein
Distance [83.53855889592734]
We introduce a Reinforcement Iterative Sequence Editing (RISE) framework that optimize the minimum Levenshtein distance (MLD) through explicit editing actions.
RISE is able to pay attention to tokens that are related to conversational characteristics.
Experimental results on two benchmark datasets show that RISE significantly outperforms state-of-the-art methods.
arXiv Detail & Related papers (2021-06-30T08:44:19Z) - Unsupervised Extractive Summarization by Pre-training Hierarchical
Transformers [107.12125265675483]
Unsupervised extractive document summarization aims to select important sentences from a document without using labeled summaries during training.
Existing methods are mostly graph-based with sentences as nodes and edge weights measured by sentence similarities.
We find that transformer attentions can be used to rank sentences for unsupervised extractive summarization.
arXiv Detail & Related papers (2020-10-16T08:44:09Z) - Predicting What You Already Know Helps: Provable Self-Supervised
Learning [60.27658820909876]
Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks) without requiring labeled data.
We show a mechanism exploiting the statistical connections between certain em reconstruction-based pretext tasks that guarantee to learn a good representation.
We prove the linear layer yields small approximation error even for complex ground truth function class.
arXiv Detail & Related papers (2020-08-03T17:56:13Z) - Embodied Self-supervised Learning by Coordinated Sampling and Training [14.107020105091662]
We propose a novel self-supervised approach to solve inverse problems by employing the corresponding physical forward process.
The proposed approach works in an analysis-by-synthesis manner to learn an inference network by iteratively sampling and training.
We prove the feasibility of the proposed method by tackling the acoustic-to-articulatory inversion problem to infer articulatory information from speech.
arXiv Detail & Related papers (2020-06-20T14:05:47Z) - Iterative Edit-Based Unsupervised Sentence Simplification [30.128553647491817]
Our model is guided by a scoring function involving fluency, simplicity, and meaning preservation.
We iteratively perform word and phrase-level edits on the complex sentence.
arXiv Detail & Related papers (2020-06-17T03:53:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.