Diffusion Recommender Models and the Illusion of Progress: A Concerning Study of Reproducibility and a Conceptual Mismatch

• URL: http://arxiv.org/abs/2505.09364v2
• Date: Thu, 15 May 2025 21:26:23 GMT
• Title: Diffusion Recommender Models and the Illusion of Progress: A Concerning Study of Reproducibility and a Conceptual Mismatch
• Authors: Michael Benigni, Maurizio Ferrari Dacrema, Dietmar Jannach,
• Abstract summary: We reproduce the latest advancements reported from applying modern Denoising Diffusion Probabilistic Models to recommender systems.<n>Our findings are concerning, revealing persistent methodological problems.<n>We identify key mismatches between the characteristics of diffusion models and those of the traditional emphtop-n recommendation task.
• Score: 12.339884639594626
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Countless new machine learning models are published every year and are reported to significantly advance the state-of-the-art in \emph{top-n} recommendation. However, earlier reproducibility studies indicate that progress in this area may be quite limited. Specifically, various widespread methodological issues, e.g., comparisons with untuned baseline models, have led to an \emph{illusion of progress}. In this work, our goal is to examine whether these problems persist in today's research. To this end, we aim to reproduce the latest advancements reported from applying modern Denoising Diffusion Probabilistic Models to recommender systems, focusing on four models published at the top-ranked SIGIR conference in 2023 and 2024. Our findings are concerning, revealing persistent methodological problems. Alarmingly, through experiments, we find that the latest recommendation techniques based on diffusion models, despite their computational complexity and substantial carbon footprint, are consistently outperformed by simpler existing models. Furthermore, we identify key mismatches between the characteristics of diffusion models and those of the traditional \emph{top-n} recommendation task, raising doubts about their suitability for recommendation. We also note that, in the papers we analyze, the generative capabilities of these models are constrained to a minimum. Overall, our results and continued methodological issues call for greater scientific rigor and a disruptive change in the research and publication culture in this area.

Related papers

• Provable Maximum Entropy Manifold Exploration via Diffusion Models [58.89696361871563]
  Exploration is critical for solving real-world decision-making problems such as scientific discovery.<n>We introduce a novel framework that casts exploration as entropy over approximate data manifold implicitly defined by a pre-trained diffusion model.<n>We develop an algorithm based on mirror descent that solves the exploration problem as sequential fine-tuning of a pre-trained diffusion model.
  arXiv  Detail & Related papers  (2025-06-18T11:59:15Z)
• Diffusion Model Quantization: A Review [36.22019054372206]
  Recent success of large text-to-image models has underscored the exceptional performance of diffusion models in generative tasks.<n>Diffusion model quantization has emerged as a pivotal technique for both compression and acceleration.
  arXiv  Detail & Related papers  (2025-05-08T13:09:34Z)
• A Worrying Reproducibility Study of Intent-Aware Recommendation Models [12.339884639594626]
  We try to reproduce five contemporary IARS models published in top-level outlets.<n>We benchmarked them against a number of traditional non-neural recommendation models.<n> Worryingly, we find that all examined IARS approaches are consistently outperformed by at least one traditional model.
  arXiv  Detail & Related papers  (2025-01-17T12:11:46Z)
• A Survey on Diffusion Models for Inverse Problems [110.6628926886398]
  We provide an overview of methods that utilize pre-trained diffusion models to solve inverse problems without requiring further training. We discuss specific challenges and potential solutions associated with using latent diffusion models for inverse problems.
  arXiv  Detail & Related papers  (2024-09-30T17:34:01Z)
• Alignment of Diffusion Models: Fundamentals, Challenges, and Future [28.64041196069495]
  Diffusion models have emerged as the leading paradigm in generative modeling, excelling in various applications. Despite their success, these models often misalign with human intentions, generating outputs that may not match text prompts or possess desired properties. Inspired by the success of alignment in tuning large language models, recent studies have investigated aligning diffusion models with human expectations and preferences.
  arXiv  Detail & Related papers  (2024-09-11T13:21:32Z)
• Diffusion Models in Low-Level Vision: A Survey [82.77962165415153]
  diffusion model-based solutions have emerged as widely acclaimed for their ability to produce samples of superior quality and diversity.<n>We present three generic diffusion modeling frameworks and explore their correlations with other deep generative models.<n>We summarize extended diffusion models applied in other tasks, including medical, remote sensing, and video scenarios.
  arXiv  Detail & Related papers  (2024-06-17T01:49:27Z)
• Predictive Churn with the Set of Good Models [61.00058053669447]
  This paper explores connections between two seemingly unrelated concepts of predictive inconsistency.<n>The first, known as predictive multiplicity, occurs when models that perform similarly produce conflicting predictions for individual samples.<n>The second concept, predictive churn, examines the differences in individual predictions before and after model updates.
  arXiv  Detail & Related papers  (2024-02-12T16:15:25Z)
• Demystifying Variational Diffusion Models [23.601173340762074]
  We present a more straightforward introduction to diffusion models using directed graphical modelling and variational Bayesian principles. Our exposition constitutes a comprehensive technical review spanning from foundational concepts like deep latent variable models to recent advances in continuous-time diffusion-based modelling. We provide additional mathematical insights that were omitted in the seminal works whenever possible to aid in understanding, while avoiding the introduction of new notation.
  arXiv  Detail & Related papers  (2024-01-11T22:37:37Z)
• Diffusion Models for Image Restoration and Enhancement -- A Comprehensive Survey [96.99328714941657]
  We present a comprehensive review of recent diffusion model-based methods on image restoration. We classify and emphasize the innovative designs using diffusion models for both IR and blind/real-world IR. We propose five potential and challenging directions for the future research of diffusion model-based IR.
  arXiv  Detail & Related papers  (2023-08-18T08:40:38Z)
• Diffusion Models for Time Series Applications: A Survey [23.003273147019446]
  Diffusion models are used in image, video, and text synthesis nowadays. We focus on diffusion-based methods for time series forecasting, imputation, and generation. We conclude the common limitation of diffusion-based methods and highlight potential future research directions.
  arXiv  Detail & Related papers  (2023-05-01T02:06:46Z)
• A Survey on Generative Diffusion Model [75.93774014861978]
  Diffusion models are an emerging class of deep generative models. They have certain limitations, including a time-consuming iterative generation process and confinement to high-dimensional Euclidean space. This survey presents a plethora of advanced techniques aimed at enhancing diffusion models.
  arXiv  Detail & Related papers  (2022-09-06T16:56:21Z)
• Back2Future: Leveraging Backfill Dynamics for Improving Real-time Predictions in Future [73.03458424369657]
  In real-time forecasting in public health, data collection is a non-trivial and demanding task. 'Backfill' phenomenon and its effect on model performance has been barely studied in the prior literature. We formulate a novel problem and neural framework Back2Future that aims to refine a given model's predictions in real-time.
  arXiv  Detail & Related papers  (2021-06-08T14:48:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.