HORIZON: A Benchmark for In-the-wild User Behaviour Modeling

• URL: http://arxiv.org/abs/2604.17259v1
• Date: Sun, 19 Apr 2026 04:45:48 GMT
• Title: HORIZON: A Benchmark for In-the-wild User Behaviour Modeling
• Abstract summary: We present HORIZON, a new benchmark that reformulates user modeling along three axes i.e. dataset, task, and evaluation.<n>Built from a large-scale, cross-domain reformulation of Amazon Reviews, HORIZON covers 54M users and 35M items.<n>Unlike prior benchmarks, it challenges models to generalize across domains, users, and time, moving beyond standard missing-positive prediction in the same domain.
• Score: 14.815729073163729
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: User behavior in the real world is diverse, cross-domain, and spans long time horizons. Existing user modeling benchmarks however remain narrow, focusing mainly on short sessions and next-item prediction within a single domain. Such limitations hinder progress toward robust and generalizable user models. We present HORIZON, a new benchmark that reformulates user modeling along three axes i.e. dataset, task, and evaluation. Built from a large-scale, cross-domain reformulation of Amazon Reviews, HORIZON covers 54M users and 35M items, enabling both pretraining and realistic evaluation of models in heterogeneous environments. Unlike prior benchmarks, it challenges models to generalize across domains, users, and time, moving beyond standard missing-positive prediction in the same domain. We propose new tasks and evaluation setups that better reflect real-world deployment scenarios. These include temporal generalization, sequence-length variation, and modeling unseen users, with metrics designed to assess general user behavior understanding rather than isolated next-item prediction. We benchmark popular sequential recommendation architectures alongside LLM-based baselines that leverage long-term interaction histories. Our results highlight the gap between current methods and the demands of real-world user modeling, while establishing HORIZON as a foundation for research on temporally robust, cross-domain, and general-purpose user models.

Related papers

• Action-Aware Generative Sequence Modeling for Short Video Recommendation [50.74467504063892]
  Short videos often contain diverse segments, and users may not hold the same attitude toward all of them.<n>Traditional binary-classification recommendation models, which treat a video as a single holistic entity, face limitations in accurately capturing such nuanced preferences.<n>We propose a novel modeling paradigm: Action-Aware Generative Sequence Network (A2Gen), which refines user actions along the temporal dimension.
  arXiv  Detail & Related papers  (2026-04-28T16:41:04Z)
• Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Traces [81.41397370235102]
  We introduce OmniBehavior, the first user simulation benchmark constructed entirely from real-world data.<n>We show that current models struggle to accurately simulate complex behaviors, with performance plateauing even as context windows expand.<n>This results in the loss of individual differences and long-tail behaviors, highlighting critical directions for future high-fidelity simulation research.
  arXiv  Detail & Related papers  (2026-04-09T15:26:21Z)
• It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks [87.7937890373758]
  Time series foundation models (TSFMs) are revolutionizing the forecasting landscape from specific dataset modeling to generalizable task evaluation.<n>We introduce TIME, a next-generation task-centric benchmark comprising 50 fresh datasets and 98 forecasting tasks.<n>We propose a novel pattern-level evaluation perspective that moves beyond traditional dataset-level evaluations based on static meta labels.
  arXiv  Detail & Related papers  (2026-02-12T16:31:01Z)
• P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling [66.55381105691818]
  We propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling.<n>P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics.<n>It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism.
  arXiv  Detail & Related papers  (2026-02-12T16:07:22Z)
• REVELIO -- Universal Multimodal Task Load Estimation for Cross-Domain Generalization [2.689067085628911]
  This paper introduces a new multimodal dataset that extends established cognitive load detection benchmarks with a real-world gaming application.<n>Task load annotations are derived from objective performance, subjective NASA-TLX ratings, and task-level design.<n>State-of-the-art end-to-end model, including xLSTM, ConvNeXt, and Transformer architectures are systematically trained and evaluated.
  arXiv  Detail & Related papers  (2025-09-01T17:36:09Z)
• A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench [18.149327897427234]
  We present clembench, which has been in continuous development since 2023 and has in its latest release been optimized for ease of general use.<n>We describe how it can be used to benchmark one's own models (using a provided set of benchmark game instances in English)
  arXiv  Detail & Related papers  (2025-07-11T11:16:01Z)
• Time Series Generation Under Data Scarcity: A Unified Generative Modeling Approach [7.631288333466648]
  We conduct the first large-scale study evaluating leading generative models in data-scarce settings.<n>We propose a unified diffusion-based generative framework that can synthesize high-fidelity time series using just a few examples.
  arXiv  Detail & Related papers  (2025-05-26T18:39:04Z)
• EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactions [0.6650227510403052]
  This paper introduces a novel dataset designed to assess and improve small language models deployable on edge devices.<n>At the core of the dataset are structured user profiles, each defined by a set of routines.<n>A large language model (LLM) generates corresponding interaction sessions that simulate realistic, diverse, and context-aware dialogues.
  arXiv  Detail & Related papers  (2025-05-16T16:29:21Z)
• Modeling the Heterogeneous Duration of User Interest in Time-Dependent Recommendation: A Hidden Semi-Markov Approach [11.392605386729699]
  We propose a hidden semi-Markov model to track the change of users' interests.<n>This model allows for capturing the different durations of user stays in a (latent) interest state.<n>We derive an algorithm to estimate the parameters and predict users' actions.
  arXiv  Detail & Related papers  (2024-12-15T09:17:45Z)
• Domain-Expanded ASTE: Rethinking Generalization in Aspect Sentiment Triplet Extraction [67.54420015049732]
  Aspect Sentiment Triplet Extraction (ASTE) is a challenging task in sentiment analysis, aiming to provide fine-grained insights into human sentiments. Existing benchmarks are limited to two domains and do not evaluate model performance on unseen domains. We introduce a domain-expanded benchmark by annotating samples from diverse domains, enabling evaluation of models in both in-domain and out-of-domain settings.
  arXiv  Detail & Related papers  (2023-05-23T18:01:49Z)
• Global-to-Local Modeling for Video-based 3D Human Pose and Shape Estimation [53.04781510348416]
  Video-based 3D human pose and shape estimations are evaluated by intra-frame accuracy and inter-frame smoothness. We propose to structurally decouple the modeling of long-term and short-term correlations in an end-to-end framework, Global-to-Local Transformer (GLoT) Our GLoT surpasses previous state-of-the-art methods with the lowest model parameters on popular benchmarks, i.e., 3DPW, MPI-INF-3DHP, and Human3.6M.
  arXiv  Detail & Related papers  (2023-03-26T14:57:49Z)
• Federated Privacy-preserving Collaborative Filtering for On-Device Next App Prediction [52.16923290335873]
  We propose a novel SeqMF model to solve the problem of predicting the next app launch during mobile device usage. We modify the structure of the classical matrix factorization model and update the training procedure to sequential learning. One more ingredient of the proposed approach is a new privacy mechanism that guarantees the protection of the sent data from the users to the remote server.
  arXiv  Detail & Related papers  (2023-02-05T10:29:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.