Pairwise Ranking Outperforms Single-Action RL for Offline Explanation Selection: A Practical Lesson

• URL: http://arxiv.org/abs/2608.18531v1
• Date: Wed, 19 Aug 2026 04:29:04 GMT
• Title: Pairwise Ranking Outperforms Single-Action RL for Offline Explanation Selection: A Practical Lesson
• Abstract summary: Industrial explainable-recommendation systems built on LLMs incur a substantial serving cost.<n>We separate generation from selection: explanations are produced ahead of time as a frozen candidate pool.<n>Our primary benchmark is a 2,958-pair XRec Google Local subset.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Industrial explainable-recommendation systems built on LLMs incur a substantial serving cost: each request triggers an LLM generation, with latency in the hundreds of milliseconds and cost that scales linearly with traffic. We separate generation from selection: explanations are produced ahead of time as a frozen candidate pool (six prompt styles, two commodity LLMs), and a small CPU-resident selector picks one at request time. The stack needs no GPU and returns in under 100 ms. Our primary benchmark is a 2,958-pair XRec Google Local subset, evaluating six offline-pool selectors (LambdaRank, PPO, GRPO, DPO, teacher-student distillation) and three KG-path selectors (random walks, edge-disjoint enumeration, MMR-reranked paths). A 300-pair MovieLens-1M split with Claude-Sonnet-4.5 references serves as an internal cross-dataset check, since no public benchmark exists for this setting. All variants use the same BERTScore-F1 protocol as XRec and G-Refer, averaged across five seeds. LambdaRank reaches F1 = 0.500 on Google Local, exceeding both G-Refer and XRec, and F1 = 0.329 on the MovieLens-1M check. With seed variance below 0.003 F1, the ordering is reliable: pairwise learning-to-rank outperforms single-action RL (PPO, GRPO, DPO), which use only one labelled candidate per rollout, leaving K-1 labels unused. The KG-path family targets a different objective: all three variants reach USR = 1.000 on Google Local and 0.997-1.000 on MovieLens-1M, since per-request path grounding yields a unique output per query, avoiding template-collapse failures affecting cached-LLM outputs. A generator-pool study comparing Claude 3 Haiku and Claude Haiku 4.5 shows small F1 shifts (0.001-0.006) while preserving selector ranking: selector and generator can be evaluated independently, though absolute F1 depends on the generator. End-to-end build cost is near $15 on commodity hardware.

Related papers

• Your Retriever Already Knows: Distribution-Shape QPP for RAG Retrieval Sufficiency [0.0]
  Motivated by a Czech nuclear-regulator deployment, we compare three Query Performance Prediction (QPP) paradigms for retrieval sufficiency in RAG.<n>Our 24 non-lexical features reach a weighted-average AUROC of 0.856 at 2 ms per query, ahead of a classic-QPP literature pool.<n>Adding the LLM judgment as one feature (hybrid) matches S1 on ViDoRe (0.863) but gains a statistically significant edge on SJB.
  arXiv  Detail & Related papers  (2026-09-10T14:52:16Z)
• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• RAG-Based Auto-Configuration for Industrial Fieldbus Devices [2.623633900557428]
  Industrial device commissioning requires engineers to manually extract hundreds of protocol-specific parameters from PDF manuals.<n>This paper presents a production-oriented pipeline that automates device configuration end-to-end for OPC-UA, Profibus DP, and CANopen.
  arXiv  Detail & Related papers  (2026-08-09T10:07:13Z)
• Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs [11.997694190254974]
  Practice often treats automatic metrics for attribution as interchangeable.<n>We audit eight automatic scorers across three evaluation constructs.<n>A naive "best-on-average" rule for choosing an evaluator fails leave-one-dataset-out.
  arXiv  Detail & Related papers  (2026-06-22T20:25:36Z)
• Which Defense Closes Which Threat? Attributing OWASP-LLM-Top-10 Coverage and Its Brittleness Under Paraphrasing [51.56484100374058]
  Production LLM applications stack several defense families -- refusal-phrase filters, tokenbudget controls, model allowlists, rate limits, tool-registry authentication -- yetBAS benchmarks report a single aggregate coverage number.<n>We add four-LLM-Top-10-aware agents to a 21-agent baseline scanner and target a lattice of four synthetic LLM endpoints.
  arXiv  Detail & Related papers  (2026-06-01T19:39:25Z)
• LogDx-CI: Benchmarking Log Reduction Tools for LLM Root-Cause Diagnosis [7.571129923187892]
  CI failure logs are large (median 5k lines, max 200k in this corpus) and noisy.<n> Coding agents that try to debug them depend on an upstream tool to reduce the log to a manageable context.<n>We introduce LogDx-CI, a benchmark that compares 11 context-reduction tools.
  arXiv  Detail & Related papers  (2026-05-26T06:34:47Z)
• Fidelity Probes for Specification--Code Alignment [7.754687669049819]
  We introduce fidelity probes: natural-dependence questions generated from a reference artifact with code-derived ground-truth answers.<n>The fidelity probes decompose into contradiction and coverage-gap rates that drive targeted spec edits to convergence.<n>On a 15-program, roughly 12k-line benchmark, we raise frozen-test specification fidelity from 0.63 to 0.94 over eight iterations.
  arXiv  Detail & Related papers  (2026-05-17T04:05:54Z)
• OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregation [53.88666485159289]
  We introduce OpenDeepThink, a population-based test-time compute framework that selects via pairwise Bradley-Terry comparison.<n>OpenDeepThink raises Gemini 3.1 Pro's effective Codeforces Elo by +405 points in eight sequential LLM-call rounds.<n>We release CF-73, a curated set of 73 expert-rated Codeforces problems with International Grandmaster annotation and 99% local-evaluation agreement against the official verdict.
  arXiv  Detail & Related papers  (2026-05-14T17:57:40Z)
• Metal-Sci: A Scientific Compute Benchmark for Evolutionary LLM Kernel Search on Apple Silicon [6.599344783327054]
  Metal-Sci is a 10-task benchmark of scientific Apple Silicon Metal compute kernels.<n>Each task ships a CPU reference, a roofline-anchored fitness function, and a held-out generalization size.<n>We report matched single-model sweeps of Claude Opus 4.7, Gemini 3.1 Pro, and GPT 5.5 on M1 Pro.
  arXiv  Detail & Related papers  (2026-05-10T19:11:26Z)
• HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference [68.59839755875252]
  HiRE comprises of two novel components: (i) a compression scheme to cheaply predict top-$k$ rows/columns with high recall, followed by full computation restricted to the predicted subset, and (ii) DA-TOP-$k$: an efficient multi-device approximate top-$k$ operator. We demonstrate that on a one billion parameter model, HiRE applied to both the softmax as well as feedforward layers, achieves almost matching pretraining and downstream accuracy, and speeds up inference latency by $1.47times$ on a single TPUv5e device.
  arXiv  Detail & Related papers  (2024-02-14T18:04:36Z)
• Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting [65.00288634420812]
  Pairwise Ranking Prompting (PRP) is a technique to significantly reduce the burden on Large Language Models (LLMs) Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs.
  arXiv  Detail & Related papers  (2023-06-30T11:32:25Z)
• Generating EDU Extracts for Plan-Guided Summary Re-Ranking [77.7752504102925]
  Two-step approaches, in which summary candidates are generated-then-reranked to return a single summary, can improve ROUGE scores over the standard single-step approach. We design a novel method to generate candidates for re-ranking that addresses these issues. We show large relevance improvements over previously published methods on widely used single document news article corpora.
  arXiv  Detail & Related papers  (2023-05-28T17:22:04Z)
• IRLI: Iterative Re-partitioning for Learning to Index [104.72641345738425]
  Methods have to trade between obtaining high accuracy while maintaining load balance and scalability in distributed settings. We propose a novel approach called IRLI, which iteratively partitions the items by learning the relevant buckets directly from the query-item relevance data. We mathematically show that IRLI retrieves the correct item with high probability under very natural assumptions and provides superior load balancing.
  arXiv  Detail & Related papers  (2021-03-17T23:13:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.