RIMS: Preference Optimization via Smoothed Multi-pair Aggregation for Small-Scale LLM Retrieval-Augmented Generation

• URL: http://arxiv.org/abs/2607.16431v1
• Date: Fri, 17 Jul 2026 18:24:19 GMT
• Title: RIMS: Preference Optimization via Smoothed Multi-pair Aggregation for Small-Scale LLM Retrieval-Augmented Generation
• Abstract summary: Small-scale language models (SLMs) are attractive for retrieval-augmented generation (RAG) in resource-constrained settings.<n>We propose RIMS, a three-stage preference optimization framework.<n>We show that our approach outperforms state-of-the-art baselines across multiple SLM backbones.
• Score: 19.76481955873559
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Small-scale language models (SLMs) are attractive for retrieval-augmented generation (RAG) in resource-constrained settings, but their limited capacity makes them highly sensitive to noisy or spurious retrieved evidence. Existing preference-based methods such as RoseRAG select only the hardest single preference pair via hard argmin/argmax, discarding the remaining signal; others treat multiple pairs as independent binary comparisons, resulting in low data utilization. We propose RIMS, a three-stage preference optimization framework comprising (1) synthetic chain-of-thought preference data generation via rejection sampling using the target SLM itself without relying on proprietary models, (2) a differentiable soft aggregation mechanism that replaces hard selection with a smooth operator, preserving gradient signal from all preference pairs while retaining the discriminative structure of margin-aware selection, and (3) preference optimization with the smoothed objective applied to multiple alignment algorithms. We theoretically show that the smoothed approximation admits a controllable error bound and that smooth aggregation yields provably tighter gradient alignment to the oracle objective than hard selection. Experiments on four multi-hop question answering benchmarks show that our approach outperforms state-of-the-art baselines across multiple SLM backbones, achieving consistent gains in Exact Match and F1 under noisy retrieval conditions. Our implementation is available at https://github.com/tptrix29/RIMS.

Related papers

• Data Efficient Sample Selection for In-Context Learning [7.910984819642885]
  In-context learning (ICL) aids large language models (LLMs) to adapt to new tasks without need for fine-tuning.<n>Existing approaches for selection do not model the complex relationship between ICL samples and downstream LLM performance.<n>We introduce DearICL, a new framework that models demonstration example selection as a subset ranking problem.
  arXiv  Detail & Related papers  (2026-09-06T15:23:37Z)
• Neighbor-aware Instance Refining with Noisy Labels for Cross-Modal Retrieval [12.062625455647265]
  Cross-Modal Retrieval (CMR) has made significant progress in the field of multi-modal analysis.<n> CMR methods often fail to simultaneously satisfy model performance ceilings, calibration reliability, and data utilization rate.<n>We propose a novel robust cross-modal learning framework, namely Neighbor-aware Instance Refining with Noisy Labels (NIRNL)
  arXiv  Detail & Related papers  (2025-12-30T08:19:07Z)
• Reverse Preference Optimization for Complex Instruction Following [61.39734201711077]
  We propose a simple yet effective method called Reverse Preference Optimization (RPO)<n>It mitigates noise in preference pairs by dynamically reversing the constraints within the instruction to ensure the chosen response is perfect.<n>RPO scales effectively across model sizes, with the 70B RPO model surpassing GPT-4o.
  arXiv  Detail & Related papers  (2025-05-28T09:44:27Z)
• ComPO: Preference Alignment via Comparison Oracles [26.629761802730258]
  We propose a new preference alignment method based on zeroth-order, comparison-based optimization via comparison oracles.<n>We conduct experiments to demonstrate the flexibility and compatibility of practical scheme in improving the performance of LLMs using noisy preference pairs.
  arXiv  Detail & Related papers  (2025-05-08T17:56:57Z)
• AMPO: Active Multi-Preference Optimization for Self-play Preference Selection [16.230186347702737]
  Multi-preference optimization enriches language-model alignment beyond pairwise preferences by contrasting entire sets of helpful and undesired responses.<n>We propose $textitActive Multi-Preference Optimization$ (AMPO), a novel approach that combines on-policy generation, a multi-preference group-contrastive loss, and active subset selection.<n>AMPO achieves state-of-the-art results on $textitAlpacaEval$ using Llama 8B and Mistral Mist 7B.
  arXiv  Detail & Related papers  (2025-02-25T15:29:51Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications.<n> Ensuring their alignment with the diverse preferences of individual users has become a critical challenge.<n>We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)
• A Systematic Examination of Preference Learning through the Lens of Instruction-Following [83.71180850955679]
  We use a novel synthetic data generation pipeline to generate 48,000 instruction unique-following prompts.<n>With our synthetic prompts, we use two preference dataset curation methods - rejection sampling (RS) and Monte Carlo Tree Search (MCTS)<n>Experiments reveal that shared prefixes in preference pairs, as generated by MCTS, provide marginal but consistent improvements.<n>High-contrast preference pairs generally outperform low-contrast pairs; however, combining both often yields the best performance.
  arXiv  Detail & Related papers  (2024-12-18T15:38:39Z)
• Geometric-Averaged Preference Optimization for Soft Preference Labels [78.2746007085333]
  Many algorithms for aligning LLMs with human preferences assume that human preferences are binary and deterministic.<n>In this work, we introduce the distributional soft preference labels and improve Direct Preference Optimization (DPO) with a weighted geometric average of the LLM output likelihood in the loss function.
  arXiv  Detail & Related papers  (2024-09-10T17:54:28Z)
• An incremental preference elicitation-based approach to learning potentially non-monotonic preferences in multi-criteria sorting [53.36437745983783]
  We first construct a max-margin optimization-based model to model potentially non-monotonic preferences. We devise information amount measurement methods and question selection strategies to pinpoint the most informative alternative in each iteration. Two incremental preference elicitation-based algorithms are developed to learn potentially non-monotonic preferences.
  arXiv  Detail & Related papers  (2024-09-04T14:36:20Z)
• ROPO: Robust Preference Optimization for Large Language Models [59.10763211091664]
  We propose an iterative alignment approach that integrates noise-tolerance and filtering of noisy samples without the aid of external models. Experiments on three widely-used datasets with Mistral-7B and Llama-2-7B demonstrate that ROPO significantly outperforms existing preference alignment methods.
  arXiv  Detail & Related papers  (2024-04-05T13:58:51Z)
• ANIMC: A Soft Framework for Auto-weighted Noisy and Incomplete Multi-view Clustering [59.77141155608009]
  We propose a novel Auto-weighted Noisy and Incomplete Multi-view Clustering framework (ANIMC) via a soft auto-weighted strategy and a doubly soft regular regression model. ANIMC has three unique advantages: 1) it is a soft algorithm to adjust our framework in different scenarios, thereby improving its generalization ability; 2) it automatically learns a proper weight for each view, thereby reducing the influence of noises; and 3) it aligns the same instances in different views, thereby decreasing the impact of missing instances.
  arXiv  Detail & Related papers  (2020-11-20T10:37:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.