Citations as Queries: Source Attribution Using Language Models as
Rerankers
- URL: http://arxiv.org/abs/2306.17322v1
- Date: Thu, 29 Jun 2023 22:13:38 GMT
- Title: Citations as Queries: Source Attribution Using Language Models as
Rerankers
- Authors: Ryan Muther and David Smith
- Abstract summary: We conduct experiments on two datasets, English Wikipedia and medieval Arabic historical writing.
We find that semisupervised methods can be nearly as effective as fully supervised methods.
- Score: 2.3605348648054454
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: This paper explores new methods for locating the sources used to write a
text, by fine-tuning a variety of language models to rerank candidate sources.
After retrieving candidates sources using a baseline BM25 retrieval model, a
variety of reranking methods are tested to see how effective they are at the
task of source attribution. We conduct experiments on two datasets, English
Wikipedia and medieval Arabic historical writing, and employ a variety of
retrieval and generation based reranking models. In particular, we seek to
understand how the degree of supervision required affects the performance of
various reranking models. We find that semisupervised methods can be nearly as
effective as fully supervised methods while avoiding potentially costly
span-level annotation of the target and source documents.
Related papers
- Verifiable Generation with Subsentence-Level Fine-Grained Citations [13.931548733211436]
Verifiable generation requires large language models to cite source documents supporting their outputs.
Previous work mainly targets the generation of sentence-level citations, lacking specificity about which parts of a sentence are backed by the cited sources.
This work studies verifiable generation with subsentence-level fine-grained citations for more precise location of generated content supported by the cited sources.
arXiv Detail & Related papers (2024-06-10T09:32:37Z) - Few-Shot Detection of Machine-Generated Text using Style Representations [4.326503887981912]
Language models that convincingly mimic human writing pose a significant risk of abuse.
We propose to leverage representations of writing style estimated from human-authored text.
We find that features effective at distinguishing among human authors are also effective at distinguishing human from machine authors.
arXiv Detail & Related papers (2024-01-12T17:26:51Z) - Beyond Contrastive Learning: A Variational Generative Model for
Multilingual Retrieval [109.62363167257664]
We propose a generative model for learning multilingual text embeddings.
Our model operates on parallel data in $N$ languages.
We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval.
arXiv Detail & Related papers (2022-12-21T02:41:40Z) - Metadata Might Make Language Models Better [1.7100280218774935]
Using 19th-century newspapers as a case study, we compare different strategies for inserting temporal, political and geographical information into a Masked Language Model.
We find that showing relevant metadata to a language model has a beneficial impact and may even produce more robust and fairer models.
arXiv Detail & Related papers (2022-11-18T08:29:00Z) - Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods
in Natural Language Processing [78.8500633981247]
This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning"
Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on language models that model the probability of text directly.
arXiv Detail & Related papers (2021-07-28T18:09:46Z) - "Don't quote me on that": Finding Mixtures of Sources in News Articles [85.92467549469147]
We construct an ontological labeling system for sources based on each source's textitaffiliation and textitrole
We build a probabilistic model to infer these attributes for named sources and to describe news articles as mixtures of these sources.
arXiv Detail & Related papers (2021-04-19T21:57:11Z) - Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language
Model [58.27176041092891]
Recent research indicates that pretraining cross-lingual language models on large-scale unlabeled texts yields significant performance improvements.
We propose a novel unsupervised feature decomposition method that can automatically extract domain-specific features from the entangled pretrained cross-lingual representations.
Our proposed model leverages mutual information estimation to decompose the representations computed by a cross-lingual model into domain-invariant and domain-specific parts.
arXiv Detail & Related papers (2020-11-23T16:00:42Z) - Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting.
Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking.
We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
arXiv Detail & Related papers (2020-10-24T11:55:28Z) - Pretrained Language Models for Dialogue Generation with Multiple Input
Sources [101.17537614998805]
In this work, we study dialogue models with multiple input sources adapted from the pretrained language model GPT2.
We explore various methods to fuse multiple separate attention information corresponding to different sources.
Our experimental results show that proper fusion methods deliver higher relevance with dialogue history than simple fusion baselines.
arXiv Detail & Related papers (2020-10-15T07:53:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.