Evaluating Deduplication Techniques for Economic Research Paper Titles with a Focus on Semantic Similarity using NLP and LLMs

• URL: http://arxiv.org/abs/2410.01141v2
• Date: Wed, 11 Dec 2024 19:37:05 GMT
• Title: Evaluating Deduplication Techniques for Economic Research Paper Titles with a Focus on Semantic Similarity using NLP and LLMs
• Abstract summary: This study investigates efficient deduplication techniques for a large NLP dataset of economic research paper titles.<n>Our findings suggest a potentially low prevalence of duplicates based on the observed semantic similarity across different methods.
• Score: 0.9576327614980397
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study investigates efficient deduplication techniques for a large NLP dataset of economic research paper titles. We explore various pairing methods alongside established distance measures (Levenshtein distance, cosine similarity) and a sBERT model for semantic evaluation. Our findings suggest a potentially low prevalence of duplicates based on the observed semantic similarity across different methods. Further exploration with a human-annotated ground truth set is completed for a more conclusive assessment. The result supports findings from the NLP, LLM based distance metrics.

Related papers

• Explainable Semantic Textual Similarity via Dissimilar Span Detection [53.32175252285023]
  We introduce the task of Dissimilar Span Detection (DSD), which aims to identify semantically differing spans between pairs of texts.<n>This can help users understand which particular words or tokens negatively affect the similarity score, or be used to improve performance in STS-dependent downstream tasks.
  arXiv  Detail & Related papers  (2026-03-22T11:32:31Z)
• LIDS: LLM Summary Inference Under the Layered Lens [3.55997016196929]
  ChatGPT generates summaries based on prompts.<n> evaluating the quality of these summaries remains challenging due to the complexity of language.<n>We suggest a new method of LLM summary inference with BERT-SVD-based direction metric and SOFARI (LIDS)<n>The LIDS uses a latent SVD-based direction metric to measure the similarity between the summaries and original text.
  arXiv  Detail & Related papers  (2026-02-18T07:39:35Z)
• Harnessing Large Language Models for Scientific Novelty Detection [49.10608128661251]
  We propose to harness large language models (LLMs) for scientific novelty detection (ND)<n>To capture idea conception, we propose to train a lightweight retriever by distilling the idea-level knowledge from LLMs.<n> Experiments show our method consistently outperforms others on the proposed benchmark datasets for idea retrieval and ND tasks.
  arXiv  Detail & Related papers  (2025-05-30T14:08:13Z)
• What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis [81.15503859645149]
  In this paper, we aim to theoretically analyze the impact of in-context demonstrations on large language models' reasoning performance. We propose a straightforward, generalizable, and low-complexity demonstration selection method named LMS3.
  arXiv  Detail & Related papers  (2024-12-11T11:38:11Z)
• Evaluating Consistencies in LLM responses through a Semantic Clustering of Question Answering [1.9214041945441436]
  We present a new approach for evaluating semanticencies of Large Language Model (LLM) Our approach evaluates whether LLM re-sponses are semantically congruent for a given question, recognizing that as syntactically different sentences may convey the same meaning. Using the TruthfulQA dataset to assess LLM responses, the study induces N re-sponses per question and clusters semantically equivalent sentences to measure semantic consistency across 37 categories.
  arXiv  Detail & Related papers  (2024-10-20T16:21:25Z)
• Chain of Stance: Stance Detection with Large Language Models [3.528201746844624]
  Stance detection is an active task in natural language processing (NLP) We propose a new prompting method, called textitChain of Stance (CoS)
  arXiv  Detail & Related papers  (2024-08-03T16:30:51Z)
• A Comparative Study of Quality Evaluation Methods for Text Summarization [0.5512295869673147]
  This paper proposes a novel method based on large language models (LLMs) for evaluating text summarization. Our results show that LLMs evaluation aligns closely with human evaluation, while widely-used automatic metrics such as ROUGE-2, BERTScore, and SummaC do not and also lack consistency.
  arXiv  Detail & Related papers  (2024-06-30T16:12:37Z)
• Decompose and Aggregate: A Step-by-Step Interpretable Evaluation Framework [75.81096662788254]
  Large Language Models (LLMs) are scalable and economical evaluators. The question of how reliable these evaluators are has emerged as a crucial research question. We propose Decompose and Aggregate, which breaks down the evaluation process into different stages based on pedagogical practices.
  arXiv  Detail & Related papers  (2024-05-24T08:12:30Z)
• Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoning [50.84938730450622]
  We propose a trajectory-based method TV score, which uses trajectory volatility for OOD detection in mathematical reasoning. Our method outperforms all traditional algorithms on GLMs under mathematical reasoning scenarios. Our method can be extended to more applications with high-density features in output spaces, such as multiple-choice questions.
  arXiv  Detail & Related papers  (2024-05-22T22:22:25Z)
• Expanding Horizons in HCI Research Through LLM-Driven Qualitative Analysis [3.5253513747455303]
  We introduce a new approach to qualitative analysis in HCI using Large Language Models (LLMs) Our findings indicate that LLMs not only match the efficacy of traditional analysis methods but also offer unique insights.
  arXiv  Detail & Related papers  (2024-01-07T12:39:31Z)
• Aligning Large Language Models with Human: A Survey [53.6014921995006]
  Large Language Models (LLMs) trained on extensive textual corpora have emerged as leading solutions for a broad array of Natural Language Processing (NLP) tasks. Despite their notable performance, these models are prone to certain limitations such as misunderstanding human instructions, generating potentially biased content, or factually incorrect information. This survey presents a comprehensive overview of these alignment technologies, including the following aspects.
  arXiv  Detail & Related papers  (2023-07-24T17:44:58Z)
• "It's a Match!" -- A Benchmark of Task Affinity Scores for Joint Learning [74.14961250042629]
  Multi-Task Learning (MTL) promises attractive, characterizing the conditions of its success is still an open problem in Deep Learning. Estimateing task affinity for joint learning is a key endeavor. Recent work suggests that the training conditions themselves have a significant impact on the outcomes of MTL. Yet, the literature is lacking a benchmark to assess the effectiveness of tasks affinity estimation techniques.
  arXiv  Detail & Related papers  (2023-01-07T15:16:35Z)
• Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold [88.83876819883653]
  We show through a manual classification of recent NLP research papers that this is indeed the case. We observe that NLP research often goes beyond the square one setup, focusing not only on accuracy, but also on fairness or interpretability, but typically only along a single dimension.
  arXiv  Detail & Related papers  (2022-06-20T13:04:23Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)
• A Discussion on Practical Considerations with Sparse Regression Methodologies [0.0]
  Two papers published in Statistical Science study the comparative performance of several sparse regression methodologies. We summarize and compare the two studies and aim to provide clarity and value to users.
  arXiv  Detail & Related papers  (2020-11-18T15:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.