From Precision to Perception: User-Centred Evaluation of Keyword Extraction Algorithms for Internet-Scale Contextual Advertising

• URL: http://arxiv.org/abs/2504.21667v1
• Date: Wed, 30 Apr 2025 14:10:00 GMT
• Title: From Precision to Perception: User-Centred Evaluation of Keyword Extraction Algorithms for Internet-Scale Contextual Advertising
• Authors: Jingwen Cai, Sara Leckner, Johanna Björklund,
• Abstract summary: This study provides a comparative evaluation of TF-IDF, KeyBERT, and Llama 2.<n>Findings indicate a slight user preference for KeyBERT, which offers a favourable balance between performance and computational efficiency.<n>Despite a strong overall preference for gold-standard keywords, differences between the algorithmic outputs are not statistically significant.
• Score: 0.25822445089477464
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Keyword extraction is a foundational task in natural language processing, underpinning countless real-world applications. A salient example is contextual advertising, where keywords help predict the topical congruence between ads and their surrounding media contexts to enhance advertising effectiveness. Recent advances in artificial intelligence, particularly large language models, have improved keyword extraction capabilities but also introduced concerns about computational cost. Moreover, although the end-user experience is of vital importance, human evaluation of keyword extraction performances remains under-explored. This study provides a comparative evaluation of three prevalent keyword extraction algorithms that vary in complexity: TF-IDF, KeyBERT, and Llama 2. To evaluate their effectiveness, a mixed-methods approach is employed, combining quantitative benchmarking with qualitative assessments from 552 participants through three survey-based experiments. Findings indicate a slight user preference for KeyBERT, which offers a favourable balance between performance and computational efficiency compared to the other two algorithms. Despite a strong overall preference for gold-standard keywords, differences between the algorithmic outputs are not statistically significant, highlighting a long-overlooked gap between traditional precision-focused metrics and user-perceived algorithm efficiency. The study highlights the importance of user-centred evaluation methodologies and proposes analytical tools to support their implementation.

Related papers

• ProgRoCC: A Progressive Approach to Rough Crowd Counting [66.09510514180593]
  We label Rough Crowd Counting that delivers better accuracy on the basis of training data that is easier to acquire. We propose an approach to the rough crowd counting problem based on CLIP, termed ProgRoCC. Specifically, we introduce a progressive estimation learning strategy that determines the object count through a coarse-to-fine approach.
  arXiv  Detail & Related papers  (2025-04-18T01:57:42Z)
• Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detection [30.836788377666]
  We propose an adaptive prompting approach that predicts the optimal prompt composition ad-hoc for a given input.<n>We apply our approach to social bias detection, a highly context-dependent task that requires semantic understanding.<n>Our approach robustly ensures high detection performance, and is best in several settings.
  arXiv  Detail & Related papers  (2025-02-10T14:06:19Z)
• Scalable Multi-phase Word Embedding Using Conjunctive Propositional Clauses [14.088007380798635]
  We introduce a novel approach incorporating two-phase training to discover contextual embeddings of input sequences.<n>This technique not only facilitates the design of a scalable model but also preserves interpretability.<n>Our experimental findings revealed that the proposed method yields competitive performance compared to the previous approaches.
  arXiv  Detail & Related papers  (2025-01-31T10:39:04Z)
• A Weighted K-Center Algorithm for Data Subset Selection [70.49696246526199]
  Subset selection is a fundamental problem that can play a key role in identifying smaller portions of the training data. We develop a novel factor 3-approximation algorithm to compute subsets based on the weighted sum of both k-center and uncertainty sampling objective functions.
  arXiv  Detail & Related papers  (2023-12-17T04:41:07Z)
• Open-vocabulary Keyword-spotting with Adaptive Instance Normalization [18.250276540068047]
  We propose AdaKWS, a novel method for keyword spotting in which a text encoder is trained to output keyword-conditioned normalization parameters. We show significant improvements over recent keyword spotting and ASR baselines.
  arXiv  Detail & Related papers  (2023-09-13T13:49:42Z)
• Two Approaches to Supervised Image Segmentation [55.616364225463066]
  The present work develops comparison experiments between deep learning and multiset neurons approaches. The deep learning approach confirmed its potential for performing image segmentation. The alternative multiset methodology allowed for enhanced accuracy while requiring little computational resources.
  arXiv  Detail & Related papers  (2023-07-19T16:42:52Z)
• Efficient Model-Free Exploration in Low-Rank MDPs [76.87340323826945]
  Low-Rank Markov Decision Processes offer a simple, yet expressive framework for RL with function approximation. Existing algorithms are either (1) computationally intractable, or (2) reliant upon restrictive statistical assumptions. We propose the first provably sample-efficient algorithm for exploration in Low-Rank MDPs.
  arXiv  Detail & Related papers  (2023-07-08T15:41:48Z)
• KPEval: Towards Fine-Grained Semantic-Based Keyphrase Evaluation [69.57018875757622]
  We propose KPEval, a comprehensive evaluation framework consisting of four critical aspects: reference agreement, faithfulness, diversity, and utility. Using KPEval, we re-evaluate 23 keyphrase systems and discover that established model comparison results have blind-spots.
  arXiv  Detail & Related papers  (2023-03-27T17:45:38Z)
• Keyword Targeting Optimization in Sponsored Search Advertising: Combining Selection and Matching [0.0]
  An optimal keyword targeting strategy guarantees reaching the right population effectively. This paper aims to address the keyword targeting problem, which is a challenging task because of the incomplete information of historical advertising performance indices. Experimental results show that, (a) BB-KSM outperforms seven baselines in terms of profit; (b) BB-KSM shows its superiority as the budget increases.
  arXiv  Detail & Related papers  (2022-10-19T03:37:32Z)
• Improved Algorithms for Neural Active Learning [74.89097665112621]
  We improve the theoretical and empirical performance of neural-network(NN)-based active learning algorithms for the non-parametric streaming setting. We introduce two regret metrics by minimizing the population loss that are more suitable in active learning than the one used in state-of-the-art (SOTA) related work.
  arXiv  Detail & Related papers  (2022-10-02T05:03:38Z)
• Representation Learning for Resource-Constrained Keyphrase Generation [78.02577815973764]
  We introduce salient span recovery and salient span prediction as guided denoising language modeling objectives. We show the effectiveness of the proposed approach for low-resource and zero-shot keyphrase generation.
  arXiv  Detail & Related papers  (2022-03-15T17:48:04Z)
• Benchmarking Simulation-Based Inference [5.3898004059026325]
  Recent advances in probabilistic modelling have led to a large number of simulation-based inference algorithms which do not require numerical evaluation of likelihoods. We provide a benchmark with inference tasks and suitable performance metrics, with an initial selection of algorithms. We found that the choice of performance metric is critical, that even state-of-the-art algorithms have substantial room for improvement, and that sequential estimation improves sample efficiency.
  arXiv  Detail & Related papers  (2021-01-12T18:31:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.