SaFRO: Satisfaction-Aware Fusion via Dual-Relative Policy Optimization for Short-Video Search

• URL: http://arxiv.org/abs/2603.19585v1
• Date: Fri, 20 Mar 2026 02:57:50 GMT
• Title: SaFRO: Satisfaction-Aware Fusion via Dual-Relative Policy Optimization for Short-Video Search
• Abstract summary: Multi-Task Fusion plays a pivotal role in industrial short-video search systems by aggregating heterogeneous prediction signals into a unified ranking score.<n>Existing approaches predominantly optimize for immediate engagement metrics, which often fail to align with long-term user satisfaction.<n>We propose SaFRO, a novel framework designed to optimize user satisfaction in short-video search.
• Score: 6.248205896178263
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-Task Fusion plays a pivotal role in industrial short-video search systems by aggregating heterogeneous prediction signals into a unified ranking score. However, existing approaches predominantly optimize for immediate engagement metrics, which often fail to align with long-term user satisfaction. While Reinforcement Learning (RL) offers a promising avenue for user satisfaction optimization, its direct application to search scenarios is non-trivial due to the inherent data sparsity and intent constraints compared to recommendation feeds. To this end, we propose SaFRO, a novel framework designed to optimize user satisfaction in short-video search. We first construct a satisfaction-aware reward model that utilizes query-level behavioral proxies to capture holistic user satisfaction beyond item-level interactions. Then we introduce Dual-Relative Policy Optimization (DRPO), an efficient policy learning method that updates the fusion policy through relative preference comparisons within groups and across batches. Furthermore, we design a Task-Relation-Aware Fusion module to explicitly model the interdependencies among different objectives, enabling context-sensitive weight adaptation. Extensive offline evaluations and large-scale online A/B tests on Kuaishou short-video search platform demonstrate that SaFRO significantly outperforms state-of-the-art baselines, delivering substantial gains in both short-term ranking quality and long-term user retention.

Related papers

• Optimizing Sparse Outcomes Through Dense Behavioral Signals via Value-Guided Preference Distillation [5.277024349608834]
  We formulate long-horizon dialogue optimization as a multi-objective reinforcement learning problem.<n>We show that a scalarized composite of dense auxiliary behavioral signals enables effective credit assignment and optimization of sparse outcomes.<n>We demonstrate that our distilled policy significantly improves long-term user retention, while simultaneously enhancing the positive behaviors and therapeutic-process markers.
  arXiv  Detail & Related papers  (2026-09-13T16:32:54Z)
• Think2Go: Generative Next POI Recommendation with LLM Reasoning [57.37706163990998]
  Next Point-of-Interest (POI) recommendation task focuses on mining user behavioral preference patterns from historical check-ins to provide personalized suggestions for the next destination.<n>Existing methods rely on shallow contextual information and handcrafted feature interactions to predict the next POI.<n>We propose Think2Go, a novel generative next POI recommendation framework, which enhances the model's comprehension of SID representations and explores diverse spatial-temporal patterns via test-time computational scaling.
  arXiv  Detail & Related papers  (2026-07-31T03:52:20Z)
• Aligning Large Language Models with Searcher Preferences [26.974618053554394]
  We introduce SearchLLM, the first large language model (LLM) for open-ended generative search.<n>We design a hierarchical, multi-dimensional reward system that separates bottom-line constraints.<n> offline evaluations and online A/B tests show improved generation quality and user engagement.
  arXiv  Detail & Related papers  (2026-03-11T06:44:30Z)
• RankGR: Rank-Enhanced Generative Retrieval with Listwise Direct Preference Optimization in Recommendation [36.297513746770456]
  We propose RankGR, a Generative Retrieval method that incorporates listwise direct preference optimization for recommendation.<n>In IAP, we incorporate a novel listwise direct preference optimization strategy into GR, thus facilitating a more comprehensive understanding of the hierarchical user preferences.<n>We implement several practical improvements in training and deployment, ultimately achieving a real-time system capable of handling nearly ten thousand requests per second.
  arXiv  Detail & Related papers  (2026-02-09T12:13:43Z)
• Towards End-to-End Alignment of User Satisfaction via Questionnaire in Video Recommendation [24.788289121071575]
  Short-video recommender systems typically optimize ranking models using dense user behavioral signals, such as clicks and watch time.<n>Recently, explicit satisfaction feedback collected through questionnaires has emerged as a high-quality direct alignment supervision.<n>We propose a novel framework towards End-to-End Alignment of user Satisfaction via Questionaire, named EASQ, to enable real-time alignment of ranking models with true user satisfaction.
  arXiv  Detail & Related papers  (2026-01-28T03:32:21Z)
• From Clicks to Preference: A Multi-stage Alignment Framework for Generative Query Suggestion in Conversational System [11.373145953200137]
  We introduce a multi-stage framework designed for progressive alignment between the generation policy and user intent.<n>Our framework significantly outperforms baselines on both automatic and human evaluations.
  arXiv  Detail & Related papers  (2025-08-15T10:17:01Z)
• Multi-agents based User Values Mining for Recommendation [52.26100802380767]
  We propose a zero-shot multi-LLM collaborative framework for effective and accurate user value extraction.<n>We apply text summarization techniques to condense item content while preserving essential meaning.<n>To mitigate hallucinations, we introduce two specialized agent roles: evaluators and supervisors.
  arXiv  Detail & Related papers  (2025-05-02T04:01:31Z)
• In-context Ranking Preference Optimization [65.5489745857577]
  We propose an In-context Ranking Preference Optimization (IRPO) framework to optimize large language models (LLMs) based on ranking lists constructed during inference.<n>We show IRPO outperforms standard DPO approaches in ranking performance, highlighting its effectiveness in aligning LLMs with direct in-context ranking preferences.
  arXiv  Detail & Related papers  (2025-04-21T23:06:12Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment [104.18002641195442]
  We introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation.
  arXiv  Detail & Related papers  (2024-05-31T14:21:04Z)
• Sequential Search with Off-Policy Reinforcement Learning [48.88165680363482]
  We propose a highly scalable hybrid learning model that consists of an RNN learning framework and an attention model. As a novel optimization step, we fit multiple short user sequences in a single RNN pass within a training batch, by solving a greedy knapsack problem on the fly. We also explore the use of off-policy reinforcement learning in multi-session personalized search ranking.
  arXiv  Detail & Related papers  (2022-02-01T06:52:40Z)
• Sample-Rank: Weak Multi-Objective Recommendations Using Rejection Sampling [0.5156484100374059]
  We introduce a method involving multi-goal sampling followed by ranking for user-relevance (Sample-Rank) to nudge recommendations towards multi-objective goals of the marketplace. The proposed method's novelty is that it reduces the MO recommendation problem to sampling from a desired multi-goal distribution then using it to build a production-friendly learning-to-rank model.
  arXiv  Detail & Related papers  (2020-08-24T09:17:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.