Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learning

• URL: http://arxiv.org/abs/2607.14192v2
• Date: Mon, 20 Jul 2026 20:30:04 GMT
• Title: Long-term User Engagement Optimization through Model-agnostic Downstream Rewards Learning
• Abstract summary: We present a unified, model-agnostic downstream reward framework for optimizing long-term user value in large-scale recommendation systems.<n>We propose several model-agnostic downstream rewards signals derived from observed user action patterns across multiple sources.<n>Online A/B experiments demonstrate consistent improvements in engagement and retention-related metrics.
• Score: 2.66348673166416
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As recommender systems mature in the past few years, their optimization objectives have evolved from a primary focusing on short-term behavioral signals to a broader emphasis on long-term user engagement and retention. However, directly optimizing retention is difficult because return signals are sparse, delayed, and only partially attributable to earlier recommendations. Prior work has addressed this challenge with sequential modeling and reinforcement learning, but these approaches typically require task specific reward engineering, substantial computational overhead, and surface specific implementations that are difficult to generalize. In this paper, we present a unified, model-agnostic downstream reward framework for optimizing long-term user value in large-scale recommendation systems. First, we formulate the downstream reward learning problem and develop an offline screening framework to identify session level behaviors that are both observable early and predictive of future retention. We then propose several model-agnostic downstream rewards signals derived from observed user action patterns across multiple sources. We further discuss the engineering effort to productionize the proposed rewards derivations and challenges we faced when adding them to our ranking models. Online A/B experiments demonstrate consistent improvements in engagement and retention-related metrics, and the framework has been deployed across multiple Pinterest surfaces, including Homefeed, Related Pins, Search, and Notifications.

Related papers

• Think2Go: Generative Next POI Recommendation with LLM Reasoning [57.37706163990998]
  Next Point-of-Interest (POI) recommendation task focuses on mining user behavioral preference patterns from historical check-ins to provide personalized suggestions for the next destination.<n>Existing methods rely on shallow contextual information and handcrafted feature interactions to predict the next POI.<n>We propose Think2Go, a novel generative next POI recommendation framework, which enhances the model's comprehension of SID representations and explores diverse spatial-temporal patterns via test-time computational scaling.
  arXiv  Detail & Related papers  (2026-07-31T03:52:20Z)
• GEMs: Breaking the Long-Sequence Barrier in Generative Recommendation with a Multi-Stream Decoder [54.64137490632567]
  We propose a novel and unified framework designed to capture users' sequences from long-term history.<n>Generative Multi-streamers ( GEMs) break user sequences into three streams.<n>Extensive experiments on large-scale industrial datasets demonstrate that GEMs significantly outperforms state-the-art methods in recommendation accuracy.
  arXiv  Detail & Related papers  (2026-02-14T06:42:56Z)
• Climber-Pilot: A Non-Myopic Generative Recommendation Model Towards Better Instruction-Following [19.550149895505683]
  We present Climber-Pilot, a unified generative retrieval framework.<n>We introduce Time-Aware Multi-Item Prediction (TAMIP), a novel training paradigm designed to mitigate inherent myopia in generative retrieval.<n>We also propose Condition-Guided Sparse Attention (CGSA), which incorporates business constraints directly into the generative process via sparse attention.
  arXiv  Detail & Related papers  (2026-02-14T03:46:06Z)
• Towards End-to-End Alignment of User Satisfaction via Questionnaire in Video Recommendation [24.788289121071575]
  Short-video recommender systems typically optimize ranking models using dense user behavioral signals, such as clicks and watch time.<n>Recently, explicit satisfaction feedback collected through questionnaires has emerged as a high-quality direct alignment supervision.<n>We propose a novel framework towards End-to-End Alignment of user Satisfaction via Questionaire, named EASQ, to enable real-time alignment of ranking models with true user satisfaction.
  arXiv  Detail & Related papers  (2026-01-28T03:32:21Z)
• Generative Actor Critic [74.04971271003869]
  Generative Actor Critic (GAC) is a novel framework that decouples sequential decision-making by reframing textitpolicy evaluation as learning a generative model of the joint distribution over trajectories and returns.<n>Experiments on Gym-MuJoCo and Maze2D benchmarks demonstrate GAC's strong offline performance and significantly enhanced offline-to-online improvement compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-12-25T06:31:11Z)
• Don't Waste It: Guiding Generative Recommenders with Structured Human Priors via Multi-head Decoding [36.10174210922267]
  We introduce a backbone-agnostic framework that seamlessly integrates human priors directly into the end-to-end training of generative recommenders.<n>Our method significantly enhances both accuracy and beyond-accuracy objectives.
  arXiv  Detail & Related papers  (2025-11-13T16:59:22Z)
• Rectifying Shortcut Behaviors in Preference-based Reward Learning [46.09046818725698]
  In reinforcement learning, preference-based reward models play a central role in aligning large language models to human-aligned behavior.<n>Recent studies show that these models are prone to reward hacking and often fail to generalize well due to over-optimization.<n>We introduce a principled yet flexible approach to mitigate shortcut behaviors in preference-based reward learning.
  arXiv  Detail & Related papers  (2025-10-21T20:08:32Z)
• Retentive Relevance: Capturing Long-Term User Value in Recommendation Systems [29.596401271139797]
  We introduce Retentive Relevance, a novel content-level survey-based feedback measure.<n>Retentive Relevance directly assesses users' intent to return to the platform for similar content.<n>We show that Retentive Relevance significantly outperforms both engagement signals and other survey measures in predicting next-day retention.
  arXiv  Detail & Related papers  (2025-10-08T23:38:57Z)
• Intra-Trajectory Consistency for Reward Modeling [67.84522106537274]
  We develop an intra-trajectory consistency regularization to enforce that adjacent processes with higher next-token generation probability maintain more consistent rewards.<n>We show that the reward model trained with the proposed regularization induces better DPO-aligned policies and achieves better best-of-N (BON) inference-time verification results.
  arXiv  Detail & Related papers  (2025-06-10T12:59:14Z)
• Slow Thinking for Sequential Recommendation [88.46598279655575]
  We present a novel slow thinking recommendation model, named STREAM-Rec.<n>Our approach is capable of analyzing historical user behavior, generating a multi-step, deliberative reasoning process, and delivering personalized recommendations.<n>In particular, we focus on two key challenges: (1) identifying the suitable reasoning patterns in recommender systems, and (2) exploring how to effectively stimulate the reasoning capabilities of traditional recommenders.
  arXiv  Detail & Related papers  (2025-04-13T15:53:30Z)
• Long-Sequence Recommendation Models Need Decoupled Embeddings [49.410906935283585]
  We identify and characterize a neglected deficiency in existing long-sequence recommendation models.<n>A single set of embeddings struggles with learning both attention and representation, leading to interference between these two processes.<n>We propose the Decoupled Attention and Representation Embeddings (DARE) model, where two distinct embedding tables are learned separately to fully decouple attention and representation.
  arXiv  Detail & Related papers  (2024-10-03T15:45:15Z)
• SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning [168.89470249446023]
  We present SURF, a semi-supervised reward learning framework that utilizes a large amount of unlabeled samples with data augmentation. In order to leverage unlabeled samples for reward learning, we infer pseudo-labels of the unlabeled samples based on the confidence of the preference predictor. Our experiments demonstrate that our approach significantly improves the feedback-efficiency of the preference-based method on a variety of locomotion and robotic manipulation tasks.
  arXiv  Detail & Related papers  (2022-03-18T16:50:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.