SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models

• URL: http://arxiv.org/abs/2509.24781v1
• Date: Mon, 29 Sep 2025 13:42:41 GMT
• Title: SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models
• Authors: Jun Rao, Yunjie Liao, Xuebo Liu, Zepeng Lin, Lian Lian, Dong Jin, Shengjun Cheng, Jun Yu, Min Zhang,
• Abstract summary: We introduce SeaPO, a Strategic Error Amplification method that leverages three error types commonly occurring in large language models.<n>We show that SeaPO significantly improved overall model performance, particularly in terms of truthfulness.<n>Injecting the most common error types improves performance in related tasks, while a mix of error types leads to a broader performance enhancement.
• Score: 25.689746306171276
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing alignment methods for preference optimization of large language models (LLMs) aim to enhance model performance by utilizing pairs of positive and negative samples. However, due to the limited capacity of models in scoring or generating responses, the quality of positive and negative samples may become similar during training, which complicates optimization for preference learning. To address this issue, we introduce SeaPO, a Strategic Error Amplification method that leverages three error types commonly occurring in LLMs to introduce specific error patterns into the model Preference Optimization. This strategy ensures that negative samples are more erroneous than positive samples and preference-based training is employed to mitigate the occurrence of these errors, thereby enhancing model performance. Evaluations across five capability dimensions and different model scales (1.5B to 14B) demonstrate that the generated data significantly improved overall model performance, particularly in terms of truthfulness, with improvements of 5-10 percentage points observed. Further analysis reveals that task performance varies depending on the error types introduced. Injecting the most common error types improves performance in related tasks, while a mix of error types leads to a broader performance enhancement: most tasks show stable improvements, while a few tasks exhibit significant gains.

Related papers

• Fault-Tolerant Evaluation for Sample-Efficient Model Performance Estimators [13.227055178509524]
  We propose a fault-tolerant evaluation framework that integrates bias and variance considerations within an adjustable tolerance level.<n>We show that proper calibration of $varepsilon$ ensures reliable evaluation across different variance regimes.<n> Experiments on real-world datasets demonstrate that our framework provides comprehensive and actionable insights into estimator behavior.
  arXiv  Detail & Related papers  (2026-02-06T22:14:46Z)
• Learning from Generalization Patterns: An Evaluation-Driven Approach to Enhanced Data Augmentation for Fine-Tuning Small Language Models [16.470481192733676]
  PaDA-Agent is an evaluation-driven approach that streamlines the data augmentation process for SLMs.<n>Our experimental results demonstrate significant improvements over state-of-the-art LLM-based data augmentation approaches for Llama 3.2 1B Instruct model fine-tuning.
  arXiv  Detail & Related papers  (2025-10-20T22:36:46Z)
• Divergence Minimization Preference Optimization for Diffusion Model Alignment [66.31417479052774]
  Divergence Minimization Preference Optimization (DMPO) is a principled method for aligning diffusion models by minimizing reverse KL divergence.<n>DMPO can consistently outperform or match existing techniques across different base models and test sets.
  arXiv  Detail & Related papers  (2025-07-10T07:57:30Z)
• Unveiling and Mitigating Adversarial Vulnerabilities in Iterative Optimizers [23.164320754562645]
  We show that non-learned iterative robustnesss share the sensitivity to adversarial examples of ML models.<n>For a class of proximal gradients, we prove how their learning affects adversarial sensitivity.<n>We numerically back our findings, showing the vulnerability of various sensitivitys, as well as the robustness induced by unfolding and adversarial training.
  arXiv  Detail & Related papers  (2025-04-26T19:03:54Z)
• Teaching Your Models to Understand Code via Focal Preference Alignment [50.5876666815306]
  In existing approaches, a set of n candidate solutions is evaluated based on test case success rates.<n>Because this approach aligns entire failing code blocks rather than pinpointing specific errors, it lacks the granularity necessary to capture meaningful error-correction relationships.<n>We propose Target-DPO, a new preference alignment framework that mimics human iterative debug to refine Code LLMs.
  arXiv  Detail & Related papers  (2025-03-04T16:56:34Z)
• DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization [59.96455188197593]
  Large language models (LLMs) deliver impressive results but face challenges from increasing model sizes and computational costs.<n>We propose DRPruning, a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data.<n> Experiments in monolingual and multilingual settings show that DRPruning surpasses similarly sized models in both pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning.
  arXiv  Detail & Related papers  (2024-11-21T12:02:39Z)
• A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models [63.949883238901414]
  We present a unique angle of gradient analysis of loss functions that simultaneously reward good examples and penalize bad ones in LMs. We find that ExMATE serves as a superior surrogate for MLE, and that combining DPO with ExMATE instead of MLE further enhances both the statistical (5-7%) and generative (+18% win rate) performance.
  arXiv  Detail & Related papers  (2024-08-29T17:46:18Z)
• Learning From Failure: Integrating Negative Examples when Fine-tuning Large Language Models as Agents [41.14201835950814]
  Large language models (LLMs) have achieved success in acting as agents, which interact with environments through tools such as search engines. Previous work has first collected interaction trajectories between LLMs and environments, using only trajectories that successfully finished the task to fine-tune smaller models. We argue that unsuccessful trajectories offer valuable insights, and LLMs can learn from these trajectories through appropriate quality control and fine-tuning strategies.
  arXiv  Detail & Related papers  (2024-02-18T17:10:07Z)
• On Diversified Preferences of Large Language Model Alignment [51.26149027399505]
  This paper presents the first quantitative analysis of the experimental scaling law for reward models with varying sizes. Our analysis reveals that the impact of diversified human preferences depends on both model size and data size. Larger models with sufficient capacity mitigate the negative effects of diverse preferences, while smaller models struggle to accommodate them.
  arXiv  Detail & Related papers  (2023-12-12T16:17:15Z)
• Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning [24.765911297156855]
  FISH-DIP is a sample-aware dynamic sparse finetuning strategy that selectively focuses on a fraction of parameters. We demonstrate that FISH-DIP can smoothly optimize the model in low resource settings offering upto 40% performance improvements.
  arXiv  Detail & Related papers  (2023-11-07T06:19:37Z)
• Generalizable Error Modeling for Human Data Annotation: Evidence From an Industry-Scale Search Data Annotation Program [0.0]
  This paper presents a predictive error model trained to detect potential errors in search relevance annotation tasks. We show that errors can be predicted with moderate model performance (AUC=0.65-0.75) and that model performance generalizes well across applications. We demonstrate the usefulness of the model in the context of auditing, where prioritizing tasks with high predicted error probabilities considerably increases the amount of corrected annotation errors.
  arXiv  Detail & Related papers  (2023-10-08T21:21:19Z)
• Learning to Refit for Convex Learning Problems [11.464758257681197]
  We propose a framework to learn to estimate optimized model parameters for different training sets using neural networks. We rigorously characterize the power of neural networks to approximate convex problems.
  arXiv  Detail & Related papers  (2021-11-24T15:28:50Z)
• MEMO: Test Time Robustness via Adaptation and Augmentation [131.28104376280197]
  We study the problem of test time robustification, i.e., using the test input to improve model robustness. Recent prior works have proposed methods for test time adaptation, however, they each introduce additional assumptions. We propose a simple approach that can be used in any test setting where the model is probabilistic and adaptable.
  arXiv  Detail & Related papers  (2021-10-18T17:55:11Z)
• Automatically Learning Compact Quality-aware Surrogates for Optimization Problems [55.94450542785096]
  Solving optimization problems with unknown parameters requires learning a predictive model to predict the values of the unknown parameters and then solving the problem using these values. Recent work has shown that including the optimization problem as a layer in a complex training model pipeline results in predictions of iteration of unobserved decision making. We show that we can improve solution quality by learning a low-dimensional surrogate model of a large optimization problem.
  arXiv  Detail & Related papers  (2020-06-18T19:11:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.