MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models

• URL: http://arxiv.org/abs/2511.20629v1
• Date: Tue, 25 Nov 2025 18:49:21 GMT
• Title: MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models
• Abstract summary: Reinforcement learning from human feedback has advanced alignment of generative models to human aesthetic and perceptual preferences.<n>We introduce two complementary methods: MapReduce LoRA and Reward-aware Token Embedding (RaTE)<n>Our framework sets a new state-of-the-art multi-preference alignment recipe across modalities.
• Score: 52.32146943039743
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning from human feedback (RLHF) with reward models has advanced alignment of generative models to human aesthetic and perceptual preferences. However, jointly optimizing multiple rewards often incurs an alignment tax, improving one dimension while degrading others. To address this, we introduce two complementary methods: MapReduce LoRA and Reward-aware Token Embedding (RaTE). MapReduce LoRA trains preference-specific LoRA experts in parallel and iteratively merges them to refine a shared base model; RaTE learns reward-specific token embeddings that compose at inference for flexible preference control. Experiments on Text-to-Image generation (Stable Diffusion 3.5 Medium and FLUX.1-dev) show improvements of 36.1%, 4.6%, and 55.7%, and 32.7%, 4.3%, and 67.1% on GenEval, PickScore, and OCR, respectively. On Text-to-Video generation (HunyuanVideo), visual and motion quality improve by 48.1% and 90.0%, respectively. On the language task, Helpful Assistant, with Llama-2 7B, helpful and harmless improve by 43.4% and 136.7%, respectively. Our framework sets a new state-of-the-art multi-preference alignment recipe across modalities.

Related papers

• Benchmarking and Evolving Reason-Reflect-Rectify for Reflective Visual Generation [65.7390808636333]
  We formalize the Reason-Reflect-Rectify (R3) loop as a core framework to enable multi-round Reflective Visual Generation (RVG)<n>We introduce R3-Bench, a benchmark of over 600 expert-annotated instances that quantifies iterative reasoning and rectification capabilities.<n>Experiments show that R3-Refiner achieves significant improvements on R3-Bench.
  arXiv  Detail & Related papers  (2026-05-19T10:24:31Z)
• Step-wise Rubric Rewards for LLM Reasoning [72.17879367869503]
  Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models.<n>We find 18.2% of steps in correct responses are wrong yet positively rewarded.<n>Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points.
  arXiv  Detail & Related papers  (2026-05-17T07:08:14Z)
• PersonalAI 2.0: Enhancing knowledge graph traversal/retrieval with planning mechanism for Personalized LLM Agents [53.369140518949656]
  We introduce PersonalAI 2.0 (PAI-2), a novel framework to enhance large language model (LLM) based systems through integration of external knowledge graphs (KG)<n>The proposed approach addresses key limitations of existing Retrieval-Augmented Generation (GraphRAG) methods by incorporating a dynamic, multistage query processing pipeline.<n>We show that use of graph algorithms (e.g. BeamSearch, WaterCircles) gain superior results compared to standard flatten retriever on average 6%, while enabled search plan enhancement mechanism gain 18% boost compared to disabled one by LLM-as-a-Judge across six datasets.
  arXiv  Detail & Related papers  (2026-05-13T13:06:30Z)
• Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration [0.0]
  Training tool-calling agents with reinforcement learning on multi-turn tasks remains challenging.<n>We present the first application of MT-GRPO and GTPO for training a tool-calling agent on realistic customer service tasks.
  arXiv  Detail & Related papers  (2026-04-03T08:36:03Z)
• Reasoning-guided Collaborative Filtering with Language Models for Explainable Recommendation [11.176352617829481]
  We present RGCF-XRec, a hybrid framework that introduces reasoning-guided collaborative filtering (CF) knowledge into a language model to deliver sequential recommendations.<n>It demonstrates consistent improvements across Amazon datasets, Sports, Toys, and Beauty, comprising 642,503 user-item interactions.<n>It reduces the cold warm performance gap, achieving overall gains of 14.5% in cold-start and 11.9% in warm start scenarios.
  arXiv  Detail & Related papers  (2026-02-05T11:05:09Z)
• SARA: Scene-Aware Reconstruction Accelerator [8.403194119298622]
  We present SARA, a geometry-driven pair selection module for Structure-from-Motion (SfM)<n>Unlike conventional pipelines that select pairs based on visual similarity alone, SARA introduces geometry-first pair selection.<n>A lightweight pre-matching stage uses mutual nearest neighbors and RANSAC to estimate reconstruction informativeness.
  arXiv  Detail & Related papers  (2026-01-11T09:45:03Z)
• Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performance [0.0]
  Reward-model training is the cost bottleneck in modern Reinforcement Learning Human Feedback (RLHF) pipelines.<n>In the proposed method, a frozen, instruction-tuned 7B LLM is augmented with only a one line and a rank-16 LoRA adapter.<n>The plug-and-play judge 96.2% achieves accuracy on RewardBench, outperforming specialized reward networks ranging from 27B to 70B parameters.
  arXiv  Detail & Related papers  (2025-06-06T05:18:54Z)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example [117.86853102104256]
  We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the math reasoning capabilities of large language models (LLMs)<n>We identify some interesting phenomena during 1-shot RLVR, including cross-category generalization, increased frequency of self-reflection, and sustained test performance improvement.
  arXiv  Detail & Related papers  (2025-04-29T09:24:30Z)
• Anyprefer: An Agentic Framework for Preference Data Synthesis [62.3856754548222]
  We propose Anyprefer, a framework designed to synthesize high-quality preference data for aligning the target model.<n> external tools are introduced to assist the judge model in accurately rewarding the target model's responses.<n>The synthesized data is compiled into a new preference dataset, Anyprefer-V1, consisting of 58K high-quality preference pairs.
  arXiv  Detail & Related papers  (2025-04-27T15:21:59Z)
• Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoning [15.61345581743979]
  We present Skywork R1V2, a next-generation multimodal reasoning model.<n>At its core, R1V2 introduces a hybrid reinforcement learning paradigm.
  arXiv  Detail & Related papers  (2025-04-23T12:24:10Z)
• Preference Optimization for Reasoning with Pseudo Feedback [100.62603571434167]
  We introduce a novel approach to generate pseudo feedback for reasoning tasks by framing the labeling of solutions as an evaluation against associated test cases.<n>We conduct experiments on both mathematical reasoning and coding tasks using pseudo feedback for preference optimization, and observe improvements across both tasks.
  arXiv  Detail & Related papers  (2024-11-25T12:44:02Z)
• Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuning [15.776175440446414]
  We introduce Dr.SoW (Density Ratio of Strong over Weak) a cost-effective method that eliminates the reliance for human annotation.<n>Dr.SoW uses the log-density ratio between a better-aligned and a less-aligned LLM as a reward signal.<n>We preference-tune Llama-3-8B-Instruct using data annotated by Dr.SoW.
  arXiv  Detail & Related papers  (2024-11-04T18:54:39Z)
• Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play [52.3079697845254]
  eva is the first method that allows language models to adaptively create training prompts in both offline and online RL post-training.<n>We show eva can create effective RL curricula and is robust across ablations.
  arXiv  Detail & Related papers  (2024-10-31T08:15:32Z)
• Building Math Agents with Multi-Turn Iterative Preference Learning [56.71330214021884]
  This paper studies the complementary direct preference learning approach to further improve model performance.<n>Existing direct preference learning algorithms are originally designed for the single-turn chat task.<n>We introduce a multi-turn direct preference learning framework, tailored for this context.
  arXiv  Detail & Related papers  (2024-09-04T02:41:04Z)
• Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences [21.5605000515622]
  This paper studies post-training large language models (LLMs) using preference feedback from an oracle to help a model iteratively improve over itself. We introduce Direct Nash Optimization (DNO), a provable and efficient algorithm that marries the simplicity and stability of contrastive learning with theoretical generality from optimizing general preferences. In our experiments, a resulting 7B parameter Orca-2.5 model achieves the state-of-the-art win-rate against GPT-4-Turbo of 33% on AlpacaE 2.0 (even after controlling for response length), an absolute gain of 26% (7% to 33%) over the initializing model
  arXiv  Detail & Related papers  (2024-04-04T17:56:41Z)
• Advancing LLM Reasoning Generalists with Preference Trees [119.57169648859707]
  We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Eurus models achieve state-of-the-art results among open-source models on a diverse set of benchmarks.
  arXiv  Detail & Related papers  (2024-04-02T16:25:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.