A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents

• URL: http://arxiv.org/abs/2607.17205v1
• Date: Sun, 19 Jul 2026 11:52:32 GMT
• Title: A Systematic Evaluation of Trajectory Data Curation for LoRA Fine-Tuning of Code Agents
• Abstract summary: Supervised fine-tuning of open-weight LLMs on expert agent trajectories has emerged as a prominent approach to building capable code agents.<n>We present a systematic empirical study of trajectory data filtering for LoRA fine-tuning of Qwen2.5-Coder-7B-Instruct on the SWE-trajectory dataset.<n>We propose a two-axis quality scoring framework -- Efficiency and Style -- and evaluate it through 16 controlled experiments spanning strategy, scale, and ablation analyses.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Supervised fine-tuning (SFT) of open-weight LLMs on expert agent trajectories has emerged as a prominent approach to building capable code agents without reliance on proprietary models. A central yet underexplored question is how trajectory quality and quantity jointly shape model performance. We present a systematic empirical study of trajectory data filtering for LoRA fine-tuning of Qwen2.5-Coder-7B-Instruct on the SWE-trajectory dataset (67,074 trajectories, of which 32,161 are resolved). We propose a two-axis quality scoring framework -- Efficiency and Style -- and evaluate it through 16 controlled experiments spanning strategy, scale, and ablation analyses. Since 7B-scale models attain near-zero SWE-bench resolve rates, we adopt cross-entropy (CE) loss on held-out trajectories as the primary metric, validated via first-action generation: CE loss and ROUGE-L are perfectly rank-correlated (Spearman $ρ$ = -1.00), with limited-sample evidence supporting but not conclusively establishing this proxy. Our results reveal a scale-dependent quality-quantity trade-off: at small scales, doubling the dataset (500 to 1,000) yields ~12.7% CE-loss reduction whereas the TopQ-Random gap stays <1% (Mann-Whitney p > 0.10); at 2,000 trajectories this same gap widens to 3.6% (p = 0.016). Ablation further identifies error-retry rate as the dominant sub-dimension, performing comparably to the full composite ($Δ$ < 0.2%). Together, these findings establish trajectory-level quality scoring as a viable but scale-sensitive lever for code-agent SFT and offer a proxy-validated evaluation protocol for the regime where end-to-end resolve rate is statistically infeasible.

Related papers

• Post-Hoc Trajectory-Risk Certification for Modular LLM-Based Security Agents [1.4915002678801434]
  We show that Bonferroni allocation is distribution-free but conservative under correlated errors.<n> coarse-to-fine label selection can create near-perfect measured correlation without learned dependence.<n>A modular certificate using per-stage certificates and a pairwise overlap bound yields a positive average gain of 0.6%.
  arXiv  Detail & Related papers  (2026-08-04T23:48:36Z)
• How Faithful Is Trajectory-Based Data Attribution? Error Sources, Remedies, and Practical Guidelines [21.57090069950487]
  Trajectory-based data attribution methods estimate the influence of training samples on model predictions by unrolling the training trajectory.<n>There is a lack of comprehensive error analysis of these methods, raising concerns about method faithfulness and hindering reliable deployment.<n>We provide the first systematic analysis of error sources in trajectory-based data attribution, together with concrete remedies to mitigate them and practical guidelines for downstream use.
  arXiv  Detail & Related papers  (2026-05-12T09:50:45Z)
• Open-Set Vein Biometric Recognition with Deep Metric Learning [0.0]
  We rigorously evaluate the computational boundaries of Deep Metric Learning (DML) under strict open-set constraints.<n>Our approach learns discnative L2-normalised embeddings and employs prototype-based matching with a similarity threshold.<n>On the large-scale MMCBNU 6000 benchmark, our best model (ResNet50-CBAM) achieves an OSCR of 0.9945, AUROC of 0.9974, and EER of 1.57%, maintaining high identification accuracy (99.6% Rank-1) while robustly rejecting unknown subjects.
  arXiv  Detail & Related papers  (2026-04-16T11:03:11Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training [53.93696896939915]
  Training tool-use agents typically rely on Supervised Fine-Tuning (SFT) on successful trajectories and Reinforcement Learning (RL) on pass-rate-selected tasks.<n>We propose TopoCurate, an interaction-aware framework that projects multi-trial rollouts from the same task into a unified semantic quotient topology.<n>TopoCurate achieves consistent gains of 4.2% (SFT) and 6.9% (RL) over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-02T10:38:54Z)
• Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning [51.87858735871145]
  We introduce Iprox, a framework that derives influence-preserving proxies directly from the target model.<n>Iprox consistently outperforms off-the-shelf proxies and baseline methods.
  arXiv  Detail & Related papers  (2026-02-19T20:57:30Z)
• Latent Sculpting for Zero-Shot Generalization: A Manifold Learning Approach to Out-of-Distribution Anomaly Detection [2.8547732086436306]
  A fundamental limitation of supervised deep learning is "Generalization Collapse"<n>We propose Latent Sculpting, a hierarchical two-stage representation learning framework.<n>We report an 88.89% detection rate on "Infiltration" scenarios.
  arXiv  Detail & Related papers  (2025-12-19T11:37:02Z)
• Quantifying Return on Security Controls in LLM Systems [0.0]
  This paper introduces a decision-oriented framework to quantify residual risk.<n>It converts adversarial probe outcomes into financial risk estimates and return-on-control metrics.
  arXiv  Detail & Related papers  (2025-12-17T04:58:09Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Graph-Structured Data Analysis of Component Failure in Autonomous Cargo Ships Based on Feature Fusion [20.287188044863925]
  This paper proposes a novel hybrid feature fusion framework for constructing a graph-structured dataset of failure modes.<n>A hierarchical feature fusion framework is constructed, using Word2Vec encoding to encode subsystem/component features, BERT-KPCA to process failure modes/reasons, and Sentence-BERT to quantify the semantic association between failure impact and emergency decision-making.
  arXiv  Detail & Related papers  (2025-07-18T08:02:49Z)
• Truncating Trajectories in Monte Carlo Policy Evaluation: an Adaptive Approach [51.76826149868971]
  Policy evaluation via Monte Carlo simulation is at the core of many MC Reinforcement Learning (RL) algorithms. We propose as a quality index a surrogate of the mean squared error of a return estimator that uses trajectories of different lengths. We present an adaptive algorithm called Robust and Iterative Data collection strategy Optimization (RIDO)
  arXiv  Detail & Related papers  (2024-10-17T11:47:56Z)
• Scale-Equivalent Distillation for Semi-Supervised Object Detection [57.59525453301374]
  Recent Semi-Supervised Object Detection (SS-OD) methods are mainly based on self-training, generating hard pseudo-labels by a teacher model on unlabeled data as supervisory signals. We analyze the challenges these methods meet with the empirical experiment results. We introduce a novel approach, Scale-Equivalent Distillation (SED), which is a simple yet effective end-to-end knowledge distillation framework robust to large object size variance and class imbalance.
  arXiv  Detail & Related papers  (2022-03-23T07:33:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.