Joint Optimization of Relevance and Engagement in Multi-Task Ranking for E-Commerce with Efficient LLM Supervision

• URL: http://arxiv.org/abs/2605.27704v1
• Date: Tue, 26 May 2026 21:26:17 GMT
• Title: Joint Optimization of Relevance and Engagement in Multi-Task Ranking for E-Commerce with Efficient LLM Supervision
• Abstract summary: We present a production-scale multi-task ranking system that integrates semantic relevance as a primary optimization objective.<n>Our architecture employs an ordinal relevance head that predicts cumulative probabilities over relevance thresholds.<n>Our approach significantly improves semantic alignment while preserving core engagement objectives.
• Score: 2.510555578533785
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Optimizing industrial search ranking models solely for user engagement signals often introduces systematic biases, prioritizing popular or price-anchored items that may not satisfy semantic intent. We present a production-scale multi-task ranking system that integrates semantic relevance as a primary optimization objective, enabling explicit and controllable relevance-engagement trade-offs. Our architecture employs an ordinal relevance head that predicts cumulative probabilities over relevance thresholds, preserving the inherent ordering of labels. These outputs are integrated with engagement heads through a unified value model scoring function, enabling systematic balancing of semantic quality and short-term behavioral signals. To provide high-quality supervision for this multi-task framework, we utilize fine-tuned lightweight Large Language Models (LLMs) to generate three-level ordinal relevance labels: irrelevant, moderately relevant, and highly relevant. We address challenges regarding label distribution sensitivity and ensure high alignment with human annotations to enable efficient labeling for over 100 million query-item pairs. Evaluation across offline metrics, including NDCG@10, and online A/B experiments demonstrates that our approach significantly improves semantic alignment while preserving core engagement objectives.

Related papers

• Rethinking Sales Lead Scoring with LLM-based Hierarchical Preference Ranking [10.224056739324702]
  Sales lead conversion differs fundamentally from e-commerce recommendation due to prolonged decision cycles and multi-stage funnels.<n>Traditional lead scoring methods face severe challenges: sparse supervision, a semantic gap in unstructured CRM logs, and inability to capture relative lead priority.<n>We introduce an LLM-based discriminative framework for sales lead scoring, which supports joint modeling of structured CRM features and unstructured customer interactions.
  arXiv  Detail & Related papers  (2026-06-03T03:05:57Z)
• Breaking User-Centric Agency: A Tri-Party Framework for Agent-Based Recommendation [59.362016745945375]
  We propose the first Tri-party LLM-agent Recommendation framework (TriRec)<n>TriRec explicitly coordinates user utility, item exposure, and platform-level fairness.<n>We show consistent gains in accuracy, fairness, and item-level utility.
  arXiv  Detail & Related papers  (2026-03-11T11:40:13Z)
• Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments [0.0]
  Large-scale commercial search systems optimize for relevance to drive successful sessions that help users find what they are looking for.<n>To maximize relevance, we leverage two complementary objectives: behavioral relevance (results users tend to click or download) and textual relevance (a result's semantic fit to the query).<n>A persistent challenge is the scarcity of expert-provided textual relevance labels relative to abundant behavioral relevance labels.
  arXiv  Detail & Related papers  (2026-02-26T17:11:26Z)
• RGAlign-Rec: Ranking-Guided Alignment for Latent Query Reasoning in Recommendation Systems [25.34524038198569]
  We propose RGAlign-Rec, a closed-loop alignment framework for proactive intent prediction.<n>We also introduce Ranking-Guided Alignment (RGA), a multi-stage training paradigm.<n>Our framework achieves a 0.12% gain in GAUC, leading to a significant 3.52% relative reduction in error rate, and a 0.56% improvement in Recall@3.
  arXiv  Detail & Related papers  (2026-02-13T14:38:02Z)
• MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization [56.074760766965085]
  Group-Relative Policy Optimization has emerged as an efficient paradigm for aligning Large Language Models (LLMs)<n>We propose MAESTRO, which treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck.<n>We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal.
  arXiv  Detail & Related papers  (2026-01-12T05:02:48Z)
• TaoSR1: The Thinking Model for E-commerce Relevance Search [15.137901457184839]
  BERT-based models excel at semantic matching but lack complex reasoning capabilities.<n>We propose a framework to directly deploy Large Language Models for this task, addressing key challenges: Chain-of-Thought (CoT) error accumulation, discriminative hallucination, and deployment feasibility.<n>Our framework, TaoSR1, involves three stages: (1) Supervised Fine-Tuning (SFT) with CoT to instill reasoning; (2) Offline sampling with a pass@N strategy and Direct Preference Optimization (DPO) to improve generation quality; and (3) Difficulty-based dynamic sampling with Group Relative Policy Optimization (GRPO)
  arXiv  Detail & Related papers  (2025-08-17T13:48:48Z)
• Balancing Semantic Relevance and Engagement in Related Video Recommendations [21.2575040646784]
  Related video recommendations commonly use collaborative filtering (CF) driven by co-engagement signals.<n>This paper introduces a novel multi-objective retrieval framework to balance semantic relevance and user engagement.
  arXiv  Detail & Related papers  (2025-07-12T21:04:25Z)
• Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks [81.44256822500257]
  RLHF has emerged as a predominant approach for aligning artificial intelligence systems with human preferences.<n> RLHF exhibits insufficient compliance capabilities when confronted with complex multi-instruction tasks.<n>We propose a novel Multi-level Aware Preference Learning (MAPL) framework, capable of enhancing multi-instruction capabilities.
  arXiv  Detail & Related papers  (2025-05-19T08:33:11Z)
• In-context Ranking Preference Optimization [65.5489745857577]
  We propose an In-context Ranking Preference Optimization (IRPO) framework to optimize large language models (LLMs) based on ranking lists constructed during inference.<n>We show IRPO outperforms standard DPO approaches in ranking performance, highlighting its effectiveness in aligning LLMs with direct in-context ranking preferences.
  arXiv  Detail & Related papers  (2025-04-21T23:06:12Z)
• On the Role of Feedback in Test-Time Scaling of Agentic AI Workflows [71.92083784393418]
  Agentic AI (systems that autonomously plan and act) are becoming widespread, yet their task success rate on complex tasks remains low.<n>Inference-time alignment relies on three components: sampling, evaluation, and feedback.<n>We introduce Iterative Agent Decoding (IAD), a procedure that repeatedly inserts feedback extracted from different forms of critiques.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• Unleash LLMs Potential for Recommendation by Coordinating Twin-Tower Dynamic Semantic Token Generator [60.07198935747619]
  We propose Twin-Tower Dynamic Semantic Recommender (T TDS), the first generative RS which adopts dynamic semantic index paradigm. To be more specific, we for the first time contrive a dynamic knowledge fusion framework which integrates a twin-tower semantic token generator into the LLM-based recommender. The proposed T TDS recommender achieves an average improvement of 19.41% in Hit-Rate and 20.84% in NDCG metric, compared with the leading baseline methods.
  arXiv  Detail & Related papers  (2024-09-14T01:45:04Z)
• Learnable Item Tokenization for Generative Recommendation [113.80559032128065]
  We propose LETTER (a LEarnable Tokenizer for generaTivE Recommendation), which integrates hierarchical semantics, collaborative signals, and code assignment diversity.<n> LETTER incorporates Residual Quantized VAE for semantic regularization, a contrastive alignment loss for collaborative regularization, and a diversity loss to mitigate code assignment bias.
  arXiv  Detail & Related papers  (2024-05-12T15:49:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.