An Embarrassingly Simple Graph Heuristic Reveals Shortcut-Solvable Benchmarks for Sequential Recommendation

• URL: http://arxiv.org/abs/2605.07125v1
• Date: Fri, 08 May 2026 02:00:11 GMT
• Title: An Embarrassingly Simple Graph Heuristic Reveals Shortcut-Solvable Benchmarks for Sequential Recommendation
• Abstract summary: Sequentialsolv has increasingly shifted toward generative recommenders that combine sequential patterns with semantic item information.<n>These methods are often evaluated on a small set of widely used benchmarks, raising a key question: do these benchmarks actually require the advanced modeling capabilities that modern generative recommenders claim to provide?<n>We conduct a benchmark audit with an intentionally simple graph. Starting from only the last one or two items, it retrieves candidates from a few-hop item-transition graph and ranks them by item-feature similarity.
• Score: 50.09718257952108
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sequential recommendation has increasingly shifted toward generative recommenders that combine sequential patterns with semantic item information. Yet these methods are often evaluated on a small set of widely used benchmarks, raising a key question: do these benchmarks actually require the advanced modeling capabilities that modern generative recommenders claim to provide? We conduct a benchmark audit with an intentionally simple graph heuristic. Starting from only the last one or two interacted items, it retrieves candidates from a few-hop item-transition graph and ranks them by item-feature similarity. Despite using no sequence encoder, generative objective, or training, this heuristic matches or outperforms many modern baselines, with relative NDCG@10 improvements of 38.10% and 44.18% over the best competing baseline on Amazon Review Sports and CDs. We show that this behavior reflects shortcut solvability rather than an artifact of one heuristic. We identify three shortcut structures that can make next-item prediction easier than expected: low-branching local transitions, feature-smooth transitions, and limited dependence on long user histories. These shortcuts need not appear together; even one or two strong signals can make simple local retrieval highly competitive, while weakening them makes the benefits of more sophisticated models clearer. Across 14 datasets, model rankings vary substantially with dataset properties, yet the heuristic remains competitive on 10 of them. Our findings suggest that strong performance on standard benchmarks does not always demonstrate advanced sequential, semantic, or generative modeling ability. We call for more careful dataset selection and dataset-level diagnostic analysis when using benchmarks to support claims about new recommendation models.

Related papers

• Harness-G: A Graph-Structured Harness for Search Agents [17.554676400013424]
  Reinforcement learning (RL) search agents commonly model retrieval as free-form natural-language query generation.<n>We propose Harness-G, a graph-structured retrieval framework that redesigns this interface.<n>It reformulates free-form query generation as finite action selection, while the environment constructs the menu, tracks retrieval state, and validates and executes each choice.<n>It achieves the highest average F1 at both evaluated model scales, outperforming the strongest baseline, Graph-R1, by 10.74 points at 1.5B and 3.98 points at 3B.
  arXiv  Detail & Related papers  (2026-07-30T04:05:05Z)
• Heuristic-inspired Reasoning Priors Facilitate Data-Efficient Referring Object Detection [53.988759250627425]
  HeROD (Heuristic-inspired ROD) is a lightweight, model-agnostic framework that injects explicit, interpretable spatial and semantic reasonings.<n>HeROD consistently outperforms strong grounding baselines in scarcelabel regimes.
  arXiv  Detail & Related papers  (2026-03-25T10:33:22Z)
• Aligning Language Model Benchmarks with Pairwise Preferences [15.427340427081843]
  We introduce benchmark alignment, where we use limited amounts of information about model performance to automatically update offline benchmarks.<n>We then propose BenchAlign, which learns preference-aligned weight-ings for benchmark questions.<n>Our experiments show that our aligned benchmarks can accurately rank unseen models according to models of human preferences, even across different sizes.
  arXiv  Detail & Related papers  (2026-02-02T23:11:09Z)
• Learning More from Less: Unlocking Internal Representations for Benchmark Compression [37.69575776639016]
  We introduce REPCORE, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets.<n>Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy.
  arXiv  Detail & Related papers  (2026-01-31T13:11:39Z)
• Scales++: Compute Efficient Evaluation Subset Selection with Cognitive Scales Embeddings [23.9553588103042]
  We propose a item-centric approach to benchmark subset selection, arguing that selection should be based on the intrinsic properties of the task items themselves.<n>We show Scales++ reduces the upfront selection cost by over 18x while achieving competitive predictive fidelity.<n>We demonstrate that this item-centric approach enables more efficient model evaluation without significant fidelity degradation.
  arXiv  Detail & Related papers  (2025-10-30T11:28:58Z)
• QDER: Query-Specific Document and Entity Representations for Multi-Vector Document Re-Ranking [5.469844680867749]
  We introduce QDER, a neural re-ranking model that unifies approaches by integrating knowledge graph semantics into a multi-vector model.<n>QDER's key innovation lies in its modeling of query-document relationships.<n>We first transform these fine-grained representations through learned attention patterns, then apply carefully chosen mathematical operations for precise matches.
  arXiv  Detail & Related papers  (2025-10-13T16:31:06Z)
• Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking [66.83273589348758]
  Link prediction attempts to predict whether an unseen edge exists based on only a portion of edges of a graph. A flurry of methods have been introduced in recent years that attempt to make use of graph neural networks (GNNs) for this task. New and diverse datasets have also been created to better evaluate the effectiveness of these new models.
  arXiv  Detail & Related papers  (2023-06-18T01:58:59Z)
• Simple Cues Lead to a Strong Multi-Object Tracker [3.7189423451031356]
  We propose a new type of tracking-by-detection (TbD) for Multi-Object Tracking. We show that a combination of our appearance features with a simple motion model leads to strong tracking results. Our tracker generalizes to four public datasets, namely MOT17, MOT20, BDD100k, and DanceTrack, achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-06-09T17:55:51Z)
• Autoregressive Search Engines: Generating Substrings as Document Identifiers [53.0729058170278]
  Autoregressive language models are emerging as the de-facto standard for generating answers. Previous work has explored ways to partition the search space into hierarchical structures. In this work we propose an alternative that doesn't force any structure in the search space: using all ngrams in a passage as its possible identifiers.
  arXiv  Detail & Related papers  (2022-04-22T10:45:01Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• Frustratingly Simple Few-Shot Object Detection [98.42824677627581]
  We find that fine-tuning only the last layer of existing detectors on rare classes is crucial to the few-shot object detection task. Such a simple approach outperforms the meta-learning methods by roughly 220 points on current benchmarks.
  arXiv  Detail & Related papers  (2020-03-16T00:29:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.