Learning Outcomes, Assessment, and Evaluation in Educational Recommender Systems: A Systematic Review

• URL: http://arxiv.org/abs/2407.09500v1
• Date: Wed, 12 Jun 2024 21:53:46 GMT
• Title: Learning Outcomes, Assessment, and Evaluation in Educational Recommender Systems: A Systematic Review
• Abstract summary: We analyse how learning is measured and optimized in Educational Recommender Systems (ERS) Rating-based relevance is the most popular target metric, while less than a half of papers optimize learning-based metrics. Only a third of the papers used outcome-based assessment to measure the pedagogical effect of recommendations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we analyse how learning is measured and optimized in Educational Recommender Systems (ERS). In particular, we examine the target metrics and evaluation methods used in the existing ERS research, with a particular focus on the pedagogical effect of recommendations. While conducting this systematic literature review (SLR), we identified 1395 potentially relevant papers, then filtered them through the inclusion and exclusion criteria, and finally selected and analyzed 28 relevant papers. Rating-based relevance is the most popular target metric, while less than a half of papers optimize learning-based metrics. Only a third of the papers used outcome-based assessment to measure the pedagogical effect of recommendations, mostly within a formal university course. This indicates a gap in ERS research with respect to assessing the pedagogical effect of recommendations at scale and in informal education settings.

Related papers

• UniER: A Unified Benchmark for Item-level and Path-level Exercise Recommendation [53.62147575168325]
  We present a Unified Benchmark for Exercise Recommendation (UniER)<n>UniER is a comprehensive evaluation framework that unifies ILER and PLER.<n>Our results reveal the systematic dominance of PLER and expose the pedagogical failure of ILER's fragmented recommendations.
  arXiv  Detail & Related papers  (2026-05-16T02:07:58Z)
• Enhancing Academic Paper Recommendations Using Fine-Grained Knowledge Entities and Multifaceted Document Embeddings [6.834689860716384]
  This paper proposes a novel academic paper recommendation method.<n>It embeds multidimensional information by integrating new types of fine-grained knowledge entities.<n>Our method outperforms baseline models, achieving an average precision of 27.3%.
  arXiv  Detail & Related papers  (2026-01-27T11:55:10Z)
• LLM4SCREENLIT: Recommendations on Assessing the Performance of Large Language Models for Screening Literature in Systematic Reviews [2.2175470459999636]
  We identify problems with the use of traditional metrics for assessing the performance of Gen-AI tools for identifying relevant literature in systematic reviews.<n>Major weaknesses included a failure to use metrics that are robust to imbalanced data and do not directly indicate whether results are better than chance.<n>On the positive side, we extract good (evaluation) practices on which our recommendations for researchers and practitioners, as well as policymakers, are built.
  arXiv  Detail & Related papers  (2025-11-16T15:04:50Z)
• Optimizing Peer Grading: A Systematic Literature Review of Reviewer Assignment Strategies and Quantity of Reviewers [0.0]
  This paper investigates how reviewer-assignment strategies and the number of reviews per submission impact the accuracy, fairness, and educational value of peer assessment.<n>We identified four common reviewer-assignment strategies: random assignment, competency-based assignment, social-network-based assignment, and bidding.<n>In terms of review count, assigning three reviews per submission emerges as the most common practice.
  arXiv  Detail & Related papers  (2025-08-08T15:28:39Z)
• Research Paper Recommender System by Considering Users' Information Seeking Behaviors [0.8588898349347149]
  One of the leading approaches to paper recommendation is content-based filtering method. We propose a content-based filtering recommendation method that takes this information seeking behavior into account.
  arXiv  Detail & Related papers  (2025-04-03T08:11:58Z)
• Benchmark for Evaluation and Analysis of Citation Recommendation Models [0.0]
  We develop a benchmark specifically designed to analyze and compare citation recommendation models. This benchmark will evaluate the performance of models on different features of the citation context. This will enable meaningful comparisons and help identify promising approaches for further research and development in the field.
  arXiv  Detail & Related papers  (2024-12-10T18:01:33Z)
• Revisiting Reciprocal Recommender Systems: Metrics, Formulation, and Method [60.364834418531366]
  We propose five new evaluation metrics that comprehensively and accurately assess the performance of RRS. We formulate the RRS from a causal perspective, formulating recommendations as bilateral interventions. We introduce a reranking strategy to maximize matching outcomes, as measured by the proposed metrics.
  arXiv  Detail & Related papers  (2024-08-19T07:21:02Z)
• Is Reference Necessary in the Evaluation of NLG Systems? When and Where? [58.52957222172377]
  We show that reference-free metrics exhibit a higher correlation with human judgment and greater sensitivity to deficiencies in language quality. Our study can provide insight into the appropriate application of automatic metrics and the impact of metric choice on evaluation performance.
  arXiv  Detail & Related papers  (2024-03-21T10:31:11Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [58.6354685593418]
  This paper proposes several article-level, field-normalized, and large language model-empowered bibliometric indicators to evaluate reviews. The newly emerging AI-generated literature reviews are also appraised. This work offers insights into the current challenges of literature reviews and envisions future directions for their development.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• A Comprehensive Survey of Evaluation Techniques for Recommendation Systems [0.0]
  This paper introduces a comprehensive suite of metrics, each tailored to capture a distinct aspect of system performance. We identify the strengths and limitations of current evaluation practices and highlight the nuanced trade-offs that emerge when optimizing recommendation systems across different metrics.
  arXiv  Detail & Related papers  (2023-12-26T11:57:01Z)
• Impression-Aware Recommender Systems [57.38537491535016]
  Novel data sources bring new opportunities to improve the quality of recommender systems. Researchers may use impressions to refine user preferences and overcome the current limitations in recommender systems research. We present a systematic literature review on recommender systems using impressions.
  arXiv  Detail & Related papers  (2023-08-15T16:16:02Z)
• Tag-Aware Document Representation for Research Paper Recommendation [68.8204255655161]
  We propose a hybrid approach that leverages deep semantic representation of research papers based on social tags assigned by users. The proposed model is effective in recommending research papers even when the rating data is very sparse.
  arXiv  Detail & Related papers  (2022-09-08T09:13:07Z)
• Evaluating the Predictive Performance of Positive-Unlabelled Classifiers: a brief critical review and practical recommendations for improvement [77.34726150561087]
  Positive-Unlabelled (PU) learning is a growing area of machine learning. This paper critically reviews the main PU learning evaluation approaches and the choice of predictive accuracy measures in 51 articles proposing PU classifiers.
  arXiv  Detail & Related papers  (2022-06-06T08:31:49Z)
• Measuring "Why" in Recommender Systems: a Comprehensive Survey on the Evaluation of Explainable Recommendation [87.82664566721917]
  This survey is based on more than 100 papers from top-tier conferences like IJCAI, AAAI, TheWebConf, Recsys, UMAP, and IUI.
  arXiv  Detail & Related papers  (2022-02-14T02:58:55Z)
• Ranking Scientific Papers Using Preference Learning [48.78161994501516]
  We cast it as a paper ranking problem based on peer review texts and reviewer scores. We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
  arXiv  Detail & Related papers  (2021-09-02T19:41:47Z)
• Academics evaluating academics: a methodology to inform the review process on top of open citations [1.911678487931003]
  We explore whether citation-based metrics, calculated only considering open citation, provide data that can yield insights on how human peer-review of research assessment exercises is conducted. We propose to use a series of machine learning models to replicate the decisions of the committees of the research assessment exercises.
  arXiv  Detail & Related papers  (2021-06-10T13:09:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.