Diagnosing Capability Gaps in Fine-Tuning Data

• URL: http://arxiv.org/abs/2604.27547v1
• Date: Thu, 30 Apr 2026 07:52:28 GMT
• Title: Diagnosing Capability Gaps in Fine-Tuning Data
• Abstract summary: We introduce GoalCover, a framework that helps practitioners detect capability gaps in fine-tuning datasets.<n>GoalCover guides a practitioner through structured decomposition of a high-level goal into atomic, independently evaluable subgoals.<n>We show that GoalCover reliably distinguishes targeted from non-targeted capability impacts.
• Score: 24.893074850115394
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning large language models (LLMs) for domain-specific tasks requires training datasets that comprehensively cover the target capabilities a practitioner needs. Yet identifying which capabilities a dataset fails to support, and doing so before an expensive fine-tuning run, remains a largely unsolved problem. We introduce GoalCover, a framework that helps practitioners systematically detect capability gaps in fine-tuning datasets through interactive goal decomposition and automated coverage assessment. GoalCover guides a practitioner through structured decomposition of a high-level goal into atomic, independently evaluable subgoals; assigns each training sample an LLM-based alignment score against every subgoal; and surfaces missing capabilities through automated analysis of low-scoring sample explanations. We validate the framework along two complementary axes. First, through controlled corruption experiments across three domains (medical QA, legal summarization, code generation), we show that GoalCover reliably distinguishes targeted from non-targeted capability impacts: target subgoals degrade by 25.6% on average versus 2.1% for non-target subgoals (Cohen's d=1.24). Second, we demonstrate downstream utility on a financial-summarization Reinforcement Fine-Tuning (RFT) task with Qwen-3-14B: training on GoalCover-filtered data improves the LLM-judge reward from 3.77 to 4.12 (out of 5) over the unfiltered baseline, and combining filtered data with goal-conditioned synthetic samples yields the strongest result (4.20). The two results together show that GoalCover works as a practical pre-fine-tuning diagnostic: it detects capability gaps and produces concrete signal for closing them.

Related papers

• From Two Passes to One: Compact and Efficient Target-Stance Extraction [0.0]
  We present a one-pass, joint architecture that predicts both in a single forward pass.<n>Standard target-scrubbing practices artificially suppress target prediction accuracy.<n> Retaining explicit target mentions improves F1 by at least 6 points across both target classification and target generation settings.
  arXiv  Detail & Related papers  (2026-09-05T14:07:43Z)
• A Step Towards Robust Unsupervised Domain Adaptation via Fine-Tuning and Reinforcement Learning [2.436631469537453]
  We propose textbfSFT+RL, a two-stage robust UDA framework that integratesSupervised Fine Tuning and Reinforcement Learning.<n>textbfSFT+RL achieves average improvements of textbf10.2% in clean accuracy and textbf15.8% in adversarial robustness across three datasets.
  arXiv  Detail & Related papers  (2026-07-03T20:56:12Z)
• Robust Zero-shot Anomaly Detection under Limited Auxiliary Anomaly Priors [10.175293570409684]
  This study introduces DIVE, the first approach to investigate the scenario of limited auxiliary anomaly priors.<n>DIVE learns to abstract generic anomaly concepts shared across the auxiliary training domain and diverse target domains.<n> Experiments demonstrate that, under the setting of limited anomaly patterns in auxiliary data, DIVE outperforms SOTA baselines.
  arXiv  Detail & Related papers  (2026-06-28T14:44:53Z)
• From Insight to Action: A Novel Framework for Interpretability-Guided Data Selection in Large Language Models [73.72877445629383]
  Interpretability-Guided Data Selection (IGDS) is a framework that first identifies causal task features through frequency recall and interventional filtering.<n>We validate IGDS on mathematical reasoning, summarization, and translation tasks within Gemma-2, LLaMA-3.1, and Qwen3 models.
  arXiv  Detail & Related papers  (2026-04-28T03:16:24Z)
• When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift [0.0]
  We present a comprehensive analysis using a benchmark of 18 datasets spanning harmful requests, jailbreaks, indirect prompt injections, and extraction attacks.<n>We propose Leave-One-Dataset-Out (LODO) evaluation to measure true out-of-distribution generalization.
  arXiv  Detail & Related papers  (2026-02-15T14:21:43Z)
• Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuning [71.30276778807068]
  We propose a unified framework that strategically coordinates sample pruning and token pruning.<n>Q-Tuning achieves a +38% average improvement over the full-data SFT baseline using only 12.5% of the original training data.
  arXiv  Detail & Related papers  (2025-09-28T13:27:38Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• A Vision-Language Pre-training Model-Guided Approach for Mitigating Backdoor Attacks in Federated Learning [43.847168319564844]
  We propose an FL backdoor defense framework, named CLIP-Fed, that utilizes the zero-shot learning capabilities of vision-language pre-training models.<n>Our scheme overcomes the limitations of Non-IID imposed on defense effectiveness by integrating pre-aggregation and post-aggregation defense strategies.
  arXiv  Detail & Related papers  (2025-08-14T03:39:54Z)
• Divide and Contrast: Source-free Domain Adaptation via Adaptive Contrastive Learning [122.62311703151215]
  Divide and Contrast (DaC) aims to connect the good ends of both worlds while bypassing their limitations. DaC divides the target data into source-like and target-specific samples, where either group of samples is treated with tailored goals. We further align the source-like domain with the target-specific samples using a memory bank-based Maximum Mean Discrepancy (MMD) loss to reduce the distribution mismatch.
  arXiv  Detail & Related papers  (2022-11-12T09:21:49Z)
• Source-Free Progressive Graph Learning for Open-Set Domain Adaptation [44.63301903324783]
  Open-set domain adaptation (OSDA) has gained considerable attention in many visual recognition tasks. We propose a Progressive Graph Learning (PGL) framework that decomposes the target hypothesis space into the shared and unknown subspaces. We also tackle a more realistic source-free open-set domain adaptation (SF-OSDA) setting that makes no assumption about the coexistence of source and target domains.
  arXiv  Detail & Related papers  (2022-02-13T01:19:41Z)
• Assessing Data Efficiency in Task-Oriented Semantic Parsing [54.87705549021248]
  We introduce a four-stage protocol which gives an approximate measure of how much in-domain "target" data a requires to achieve a certain quality bar. We apply our protocol in two real-world case studies illustrating its flexibility and applicability to practitioners in task-oriented semantic parsing.
  arXiv  Detail & Related papers  (2021-07-10T02:43:16Z)
• Deep F-measure Maximization for End-to-End Speech Understanding [52.36496114728355]
  We propose a differentiable approximation to the F-measure and train the network with this objective using standard backpropagation. We perform experiments on two standard fairness datasets, Adult, Communities and Crime, and also on speech-to-intent detection on the ATIS dataset and speech-to-image concept classification on the Speech-COCO dataset. In all four of these tasks, F-measure results in improved micro-F1 scores, with absolute improvements of up to 8% absolute, as compared to models trained with the cross-entropy loss function.
  arXiv  Detail & Related papers  (2020-08-08T03:02:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.