Aligning Large Language Models with Searcher Preferences

• URL: http://arxiv.org/abs/2603.10473v1
• Date: Wed, 11 Mar 2026 06:44:30 GMT
• Title: Aligning Large Language Models with Searcher Preferences
• Abstract summary: We introduce SearchLLM, the first large language model (LLM) for open-ended generative search.<n>We design a hierarchical, multi-dimensional reward system that separates bottom-line constraints.<n> offline evaluations and online A/B tests show improved generation quality and user engagement.
• Score: 26.974618053554394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The paradigm shift from item-centric ranking to answer-centric synthesis is redefining the role of search engines. While recent industrial progress has applied generative techniques to closed-set item ranking in e-commerce, research and deployment of open-ended generative search on large content platforms remain limited. This setting introduces challenges, including robustness to noisy retrieval, non-negotiable safety guarantees, and alignment with diverse user needs. In this work, we introduce SearchLLM, the first large language model (LLM) for open-ended generative search. We design a hierarchical, multi-dimensional reward system that separates bottom-line constraints, including factual grounding, basic answer quality and format compliance, from behavior optimization objectives that promote robustness to noisy retrieval and alignment with user needs. Concretely, our reward model evaluates responses conditioned on the user query, session history, and retrieved evidence set, combining rule-based checks with human-calibrated LLM judges to produce an interpretable score vector over these dimensions. We introduce a Gated Aggregation Strategy to derive the training reward for optimizing SearchLLM with Group Relative Policy Optimization (GRPO). We deploy SearchLLM in the AI search entry of RedNote. Offline evaluations and online A/B tests show improved generation quality and user engagement, increasing Valid Consumption Rate by 1.03% and reducing Re-search Rate by 2.81%, while upholding strict safety and reliability standards.

Related papers

• CAMI: Cost-Aware Agent-Guided Multi-Indexing for Semantic Retrieval [7.07006462273528]
  CAMI (Cost-Aware Multi-Indexing) is a framework that formalizes multi-index construction as a budgeted, multi-objective portfolio selection problem.<n>Our findings reveal that the framework systematically isolates high-recall portfolios under strict budget constraints.<n> CAMI is able to systematically identify these high-recall portfolios using up to 5x less budget compared to random search baselines.
  arXiv  Detail & Related papers  (2026-06-14T16:59:18Z)
• F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking [79.49893545611779]
  Large Language Models (LLMs) can generate a subset and order it within a single autoregressive pass.<n>This flexibility introduces a new optimization challenge: the model must search an output space while receiving utility feedback only after the full ranked list is generated.<n>This credit assignment gap makes end-to-end optimization unstable and sample-inefficient.<n>We propose a unified framework that performs both within a single autoregressive rollout and optimize them end-to-end via factorized group-relative policy optimization.
  arXiv  Detail & Related papers  (2026-05-13T04:52:33Z)
• CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search [51.911048955965136]
  CoSearch is a framework that jointly trains a multi-step reasoning agent and a generative document ranking model.<n>Our results show that joint training of the reasoning agent and retrieval system is both feasible and strongly performant.
  arXiv  Detail & Related papers  (2026-04-19T17:48:17Z)
• OneSearch-V2: The Latent Reasoning Enhanced Self-distillation Generative Search Framework [41.83974943844478]
  Generative Retrieval (GR) has emerged as a promising paradigm for modern search systems.<n>OneSearch, as a representative industrial-scale deployed generative search framework, has brought significant commercial and operational benefits.<n>We propose textbfOneSearch-V2, a latent reasoning enhanced self-distillation generative search framework.
  arXiv  Detail & Related papers  (2026-03-25T15:33:34Z)
• SaFRO: Satisfaction-Aware Fusion via Dual-Relative Policy Optimization for Short-Video Search [6.248205896178263]
  Multi-Task Fusion plays a pivotal role in industrial short-video search systems by aggregating heterogeneous prediction signals into a unified ranking score.<n>Existing approaches predominantly optimize for immediate engagement metrics, which often fail to align with long-term user satisfaction.<n>We propose SaFRO, a novel framework designed to optimize user satisfaction in short-video search.
  arXiv  Detail & Related papers  (2026-03-20T02:57:50Z)
• SmartSearch: Process Reward-Guided Query Refinement for Search Agents [63.46067892354375]
  Large language model (LLM)-based search agents have proven promising for addressing knowledge-intensive problems.<n>Existing works largely focus on optimizing the reasoning paradigms of search agents, yet the quality of intermediate search queries during reasoning remains overlooked.<n>We introduce SmartSearch, a framework built upon two key mechanisms to mitigate this issue.
  arXiv  Detail & Related papers  (2026-01-08T12:39:05Z)
• Towards Context-aware Reasoning-enhanced Generative Searching in E-commerce [61.03081096959132]
  We propose a context-aware reasoning-enhanced generative search framework for better textbfunderstanding the complicated context.<n>Our approach achieves superior performance compared with strong baselines, validating its effectiveness for search-based recommendation.
  arXiv  Detail & Related papers  (2025-10-19T16:46:11Z)
• OneSearch: A Preliminary Exploration of the Unified End-to-End Generative Framework for E-commerce Search [43.94443394870866]
  OneSearch is the first industrial-deployed end-to-end generative framework for e-commerce search.<n>OneSearch reduces operational expenditure by 75.40% and improves Model FLOPs Utilization from 3.26% to 27.32%.<n>The system has been successfully deployed across multiple search scenarios in Kuaishou, serving millions of users.
  arXiv  Detail & Related papers  (2025-09-03T11:50:04Z)
• MMSearch-R1: Incentivizing LMMs to Search [49.889749277236376]
  We present MMSearch-R1, the first end-to-end reinforcement learning framework that enables on-demand, multi-turn search in real-world Internet environments.<n>Our framework integrates both image and text search tools, allowing the model to reason about when and how to invoke them guided by an outcome-based reward with a search penalty.
  arXiv  Detail & Related papers  (2025-06-25T17:59:42Z)
• Constrained Auto-Regressive Decoding Constrains Generative Retrieval [71.71161220261655]
  Generative retrieval seeks to replace traditional search index data structures with a single large-scale neural network.<n>In this paper, we examine the inherent limitations of constrained auto-regressive generation from two essential perspectives: constraints and beam search.
  arXiv  Detail & Related papers  (2025-04-14T06:54:49Z)
• CRM: Retrieval Model with Controllable Condition [23.936944737868465]
  Controllable Retrieval Model integrates regression information as conditional features into the two-tower retrieval paradigm.<n>We validate the effectiveness of CRM through real-world A/B testing and demonstrate its successful deployment in Kuaishou short-video recommendation system.
  arXiv  Detail & Related papers  (2024-12-18T13:37:36Z)
• Self-Calibrated Listwise Reranking with Large Language Models [137.6557607279876]
  Large language models (LLMs) have been employed in reranking tasks through a sequence-to-sequence approach. This reranking paradigm requires a sliding window strategy to iteratively handle larger candidate sets. We propose a novel self-calibrated listwise reranking method, which aims to leverage LLMs to produce global relevance scores for ranking.
  arXiv  Detail & Related papers  (2024-11-07T10:31:31Z)
• CART: A Generative Cross-Modal Retrieval Framework with Coarse-To-Fine Semantic Modeling [53.97609687516371]
  Cross-modal retrieval aims to search for instances, which are semantically related to the query through the interaction of different modal data.<n>Traditional solutions utilize a single-tower or dual-tower framework to explicitly compute the score between queries and candidates.<n>We propose a generative cross-modal retrieval framework (CART) based on coarse-to-fine semantic modeling.
  arXiv  Detail & Related papers  (2024-06-25T12:47:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.