Continuous Online Evaluation of Recommendation Strategies in Social Science Academic Search

• URL: http://arxiv.org/abs/2607.26380v1
• Date: Wed, 29 Jul 2026 01:33:12 GMT
• Title: Continuous Online Evaluation of Recommendation Strategies in Social Science Academic Search
• Abstract summary: We implement and compare a diverse set of algorithms, including traditional lexical similarity, semantic document similarity.<n>Our results show that users prefer recommendations based on semantic similarity, which outperformed term-similarity and session-based methods.<n>Overall, our study provides insights into how continuous evaluation can be incorporated to develop recommendations that better align with the preferences in academic search portals.
• Score: 1.8047694351309207
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Delivering relevant recommendations in academic search engines is a complex task due to the diversity of subject areas, information types, and user preferences. In this case study, we address these challenges by integrating and evaluating a range of recommendation systems within GESIS Search - a domain-specific search engine for the social sciences that provides researchers with access to research data, publications, variables, and measurement instruments. To support continuous, real-time evaluation of multiple recommendation strategies with actual platform users, we utilize the STELLA evaluation framework. We implement and compare a diverse set of algorithms, including traditional lexical similarity, semantic document similarity by using transformer-based embeddings, and session-based recommendations based on click paths from historical user sessions. Our results show that users prefer recommendations based on semantic similarity, which outperformed term-similarity and session-based methods. However, the performance of recommenders varies across categories within GESIS Search, suggesting that information-seeking behavior differs by information type. Overall, our study provides insights into how continuous evaluation can be incorporated to develop recommendations that better align with the preferences in academic search portals.

Related papers

• UniER: A Unified Benchmark for Item-level and Path-level Exercise Recommendation [53.62147575168325]
  We present a Unified Benchmark for Exercise Recommendation (UniER)<n>UniER is a comprehensive evaluation framework that unifies ILER and PLER.<n>Our results reveal the systematic dominance of PLER and expose the pedagogical failure of ILER's fragmented recommendations.
  arXiv  Detail & Related papers  (2026-05-16T02:07:58Z)
• Towards a Real-World Aligned Benchmark for Unlearning in Recommender Systems [49.766845975588275]
  We propose a set of design desiderata and research questions to guide the development of a more realistic benchmark for unlearning in recommender systems.<n>We argue for an unlearning setup that reflects the sequential, time-sensitive nature of real-world deletion requests.<n>We present a preliminary experiment in a next-basket recommendation setting based on our proposed desiderata and find that unlearning also works for sequential recommendation models.
  arXiv  Detail & Related papers  (2025-08-23T16:05:40Z)
• Does Multimodality Improve Recommender Systems as Expected? A Critical Analysis and Future Directions [52.21847626165085]
  Multimodal recommendation systems are increasingly popular for their potential to improve performance by integrating diverse data types.<n>However, the actual benefits of this integration remain unclear, raising questions about when and how it truly enhances recommendations.<n>We propose a structured evaluation framework to systematically assess multimodal recommendations across four dimensions.
  arXiv  Detail & Related papers  (2025-08-07T13:21:00Z)
• User and Recommender Behavior Over Time: Contextualizing Activity, Effectiveness, Diversity, and Fairness in Book Recommendation [2.1775979310224827]
  We present a temporal explanatory analysis of the UCSD Book Graph dataset scraped from Goodreads.<n>We examine whether the introduction of algorithmic recommendations in 2011 was followed by observable changes in user or recommender system behavior.
  arXiv  Detail & Related papers  (2025-05-07T15:45:04Z)
• Interactive Visualization Recommendation with Hier-SUCB [52.11209329270573]
  We propose an interactive personalized visualization recommendation (PVisRec) system that learns on user feedback from previous interactions.<n>For more interactive and accurate recommendations, we propose Hier-SUCB, a contextual semi-bandit in the PVisRec setting.
  arXiv  Detail & Related papers  (2025-02-05T17:14:45Z)
• Online and Offline Evaluations of Collaborative Filtering and Content Based Recommender Systems [0.0]
  This study provides a comparative analysis of a large-scale recommender system operating in Iran. The system employs user-based and item-based recommendations using content-based, collaborative filtering, trend-based methods, and hybrid approaches. Our methods of evaluation include manual evaluation, offline tests including accuracy and ranking metrics like hit-rate@k and nDCG, and online tests consisting of click-through rate (CTR)
  arXiv  Detail & Related papers  (2024-11-02T20:05:31Z)
• Quantifying User Coherence: A Unified Framework for Cross-Domain Recommendation Analysis [69.37718774071793]
  This paper introduces novel information-theoretic measures for understanding recommender systems. We evaluate 7 recommendation algorithms across 9 datasets, revealing the relationships between our measures and standard performance metrics.
  arXiv  Detail & Related papers  (2024-10-03T13:02:07Z)
• A Systematic Literature Review on Task Recommendation Systems for Crowdsourced Software Engineering [8.210764997771532]
  This SLR was conducted according to the Kitchenham and Charters' guidelines.<n>We selected 65 primary studies for data extraction, analysis, and synthesis.<n>Our results revealed that human factors play a major role in CSE task recommendation.
  arXiv  Detail & Related papers  (2024-07-13T12:46:57Z)
• Impression-Aware Recommender Systems [53.48892326556546]
  We present a systematic literature review on recommender systems using impressions.<n>We define a theoretical framework to delimit recommender systems using impressions and a novel paradigm for personalized recommendations, called impression-aware recommender systems.
  arXiv  Detail & Related papers  (2023-08-15T16:16:02Z)
• Thematic recommendations on knowledge graphs using multilayer networks [0.0]
  We present a framework to generate and evaluate thematic recommendations based on multilayer network representations of knowledge graphs (KGs) In this representation, each layer encodes a different type of relationship in the KG, and directed interlayer couplings connect the same entity in different roles. We apply an adaptation of the personalised PageRank algorithm to multilayer models of KGs to generate item-item recommendations.
  arXiv  Detail & Related papers  (2021-05-12T15:30:21Z)
• Comprehensive Empirical Evaluation of Deep Learning Approaches for Session-based Recommendation in E-Commerce [38.42250061908039]
  In session-based recommendation, a recommendation system counts on the sequence of events made by a user within the same session to predict and endorse other items. We present a comprehensive evaluation of the state-of-the-art deep learning approaches used in the session-based recommendation.
  arXiv  Detail & Related papers  (2020-10-17T17:22:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.