Answer is All You Need: Instruction-following Text Embedding via Answering the Question

• URL: http://arxiv.org/abs/2402.09642v1
• Date: Thu, 15 Feb 2024 01:02:41 GMT
• Title: Answer is All You Need: Instruction-following Text Embedding via Answering the Question
• Authors: Letian Peng, Yuwei Zhang, Zilong Wang, Jayanth Srinivasa, Gaowen Liu, Zihan Wang, Jingbo Shang
• Abstract summary: This paper offers a new viewpoint, which treats the instruction as a question about the input text and encodes the expected answers to obtain the representation accordingly. Specifically, we propose InBedder that instantiates this embed-via-answering idea by only fine-tuning language models on abstractive question answering tasks.
• Score: 41.727700155498546
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work aims to build a text embedder that can capture characteristics of texts specified by user instructions. Despite its tremendous potential to deploy user-oriented embeddings, none of previous approaches provides a concrete solution for it. This paper offers a new viewpoint, which treats the instruction as a question about the input text and encodes the expected answers to obtain the representation accordingly. Intuitively, texts with the same (implicit) semantics would share similar answers following the instruction, thus leading to more similar embeddings. Specifically, we propose InBedder that instantiates this embed-via-answering idea by only fine-tuning language models on abstractive question answering tasks. InBedder demonstrates significantly improved instruction-following capabilities according to our proposed instruction awareness tests and instruction robustness tests, when applied to both large language models (LLMs) (e.g., llama-2-7b) and smaller encoder-based LMs (e.g., roberta-large). Additionally, our qualitative analysis of clustering outcomes, achieved by applying different instructions to the same corpus, demonstrates a high degree of interpretability.

Related papers

• Disentangling Dense Embeddings with Sparse Autoencoders [0.0]
  Sparse autoencoders (SAEs) have shown promise in extracting interpretable features from complex neural networks. We present one of the first applications of SAEs to dense text embeddings from large language models. We show that the resulting sparse representations maintain semantic fidelity while offering interpretability.
  arXiv  Detail & Related papers  (2024-08-01T15:46:22Z)
• Crafting Interpretable Embeddings by Asking LLMs Questions [89.49960984640363]
  Large language models (LLMs) have rapidly improved text embeddings for a growing array of natural-language processing tasks. We introduce question-answering embeddings (QA-Emb), embeddings where each feature represents an answer to a yes/no question asked to an LLM. We use QA-Emb to flexibly generate interpretable models for predicting fMRI voxel responses to language stimuli.
  arXiv  Detail & Related papers  (2024-05-26T22:30:29Z)
• PEACH: Pretrained-embedding Explanation Across Contextual and Hierarchical Structure [3.9677082086241433]
  PEACH can explain how text-based documents are classified by using any pretrained contextual embeddings in a tree-based human-interpretable manner. We perform a comprehensive analysis of several contextual embeddings on nine different NLP text classification benchmarks. We show the utility of explanations by visualising the feature selection and important trend of text classification via human-interpretable word-cloud-based trees.
  arXiv  Detail & Related papers  (2024-04-21T12:41:02Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• Enhancing Large Language Models Against Inductive Instructions with Dual-critique Prompting [55.15697111170836]
  This paper reveals the behaviors of large language models (LLMs) towards textitinductive instructions and enhance their truthfulness and helpfulness accordingly. After extensive human and automatic evaluations, we uncovered a universal vulnerability among LLMs in processing inductive instructions. We identify that different inductive styles affect the models' ability to identify the same underlying errors, and the complexity of the underlying assumptions also influences the model's performance.
  arXiv  Detail & Related papers  (2023-05-23T06:38:20Z)
• Elaborative Simplification as Implicit Questions Under Discussion [51.17933943734872]
  This paper proposes to view elaborative simplification through the lens of the Question Under Discussion (QUD) framework. We show that explicitly modeling QUD provides essential understanding of elaborative simplification and how the elaborations connect with the rest of the discourse.
  arXiv  Detail & Related papers  (2023-05-17T17:26:16Z)
• Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions [5.343406649012618]
  We propose the use of the classic construct validity framework to evaluate the validity of text embeddings. We show how this framework can be adapted to the opaque and high-dimensional nature of text embeddings. We show that embeddings can be used to predict respondent's answers to completely new survey questions.
  arXiv  Detail & Related papers  (2022-02-18T12:35:46Z)
• Explaining Latent Representations with a Corpus of Examples [72.50996504722293]
  We propose SimplEx: a user-centred method that provides example-based explanations with reference to a freely selected set of examples. SimplEx uses the corpus to improve the user's understanding of the latent space with post-hoc explanations. We show that SimplEx empowers the user by highlighting relevant patterns in the corpus that explain model representations.
  arXiv  Detail & Related papers  (2021-10-28T17:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.