Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Models

• URL: http://arxiv.org/abs/2510.00040v1
• Date: Sat, 27 Sep 2025 02:57:37 GMT
• Title: Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Models
• Abstract summary: We introduce Capability-Attributed Data Curation (CADC), a framework that shifts from task-specific curations to intrinsic capability analysis.<n>CADC discovers intrinsic capabilities in an unsupervised manner from gradient-based learning trajectories.<n>With as little as 5% of the original data, CADC surpasses full-data training on multimodal benchmarks.
• Score: 26.897329298659717
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large vision-language models (VLMs) achieve strong benchmark performance, but controlling their behavior through instruction tuning remains difficult. Reducing the budget of instruction tuning dataset often causes regressions, as heuristic strategies treat models as black boxes and overlook the latent capabilities that govern learning. We introduce Capability-Attributed Data Curation (CADC), a framework that shifts curation from task-specific heuristics to intrinsic capability analysis. CADC discovers intrinsic capabilities in an unsupervised manner from gradient-based learning trajectories, attributes training data to these capabilities via influence estimation, and curates capability-aware curricula through balanced selection and staged sequencing. This transforms black-box instruction tuning into a controllable, capability-driven process. With as little as 5% of the original data, CADC surpasses full-data training on multimodal benchmarks. These results validate intrinsic capabilities as the fundamental building blocks of model learning and establish CADC as a principle paradigm for instruction data curation.

Related papers

• Mechanistic Circuit Identification for Controllable Data Generation [8.346479730119436]
  We propose a circuit-grounded framework that connects training-dynamics-based data valuation with mechanistic interpretability.<n> Specifically, we conceptualize data quality along three complementary utility axes, learnability, challenge, and alignment.<n>Building on this capability, we introduce SAMS (Stage-Aware Mechanistic Scheduling), which schedules circuit-steered data according to the model's evolving optimization needs.
  arXiv  Detail & Related papers  (2026-08-25T04:53:19Z)
• CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training [48.70704477452434]
  We introduce textbfCapTrack, a capability-centric framework for analyzing forgetting in large language models.<n>We conduct a large-scale empirical study across post-training algorithms, domains, and model families.<n>We find that forgetting extends beyond parametric knowledge, with pronounced drift in robustness and default behaviors.
  arXiv  Detail & Related papers  (2026-02-19T09:46:24Z)
• Learning What Matters: Prioritized Concept Learning via Relative Error-driven Sample Selection [38.35524024887503]
  We propose PRioritized cOncept learninG via Relative Error-driven Sample Selection (PROGRESS)<n>PROGRESS is a data- and compute-efficient framework that enables vision-language models to dynamically select what to learn next.<n>We show that PROGRESS consistently outperforms state-of-the-art baselines with much less data and supervision.
  arXiv  Detail & Related papers  (2025-06-01T17:05:35Z)
• Diffusion Guidance Is a Controllable Policy Improvement Operator [98.11511661904618]
  CFGRL is trained with the simplicity of supervised learning, yet can further improve on the policies in the data.<n>On offline RL tasks, we observe a reliable trend -- increased guidance weighting leads to increased performance.
  arXiv  Detail & Related papers  (2025-05-29T14:06:50Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance [61.06245197347139]
  We propose a novel approach to explain the behavior of a black-box model under feature shifts. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation.
  arXiv  Detail & Related papers  (2024-08-24T18:28:19Z)
• In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Models [37.45103473809928]
  We propose the In2Core algorithm, which selects a coreset by analyzing the correlation between training and evaluation samples with a trained model. By applying our algorithm to instruction fine-tuning data of LLMs, we can achieve similar performance with just 50% of the training data.
  arXiv  Detail & Related papers  (2024-08-07T05:48:05Z)
• Transformer-based Causal Language Models Perform Clustering [20.430255724239448]
  We introduce a simplified instruction-following task and use synthetic datasets to analyze a Transformer-based causal language model. Our findings suggest that the model learns task-specific information by clustering data within its hidden space, with this clustering process evolving dynamically during learning.
  arXiv  Detail & Related papers  (2024-02-19T14:02:31Z)
• CUDC: A Curiosity-Driven Unsupervised Data Collection Method with Adaptive Temporal Distances for Offline Reinforcement Learning [62.58375643251612]
  We propose a Curiosity-driven Unsupervised Data Collection (CUDC) method to expand feature space using adaptive temporal distances for task-agnostic data collection. With this adaptive reachability mechanism in place, the feature representation can be diversified, and the agent can navigate itself to collect higher-quality data with curiosity. Empirically, CUDC surpasses existing unsupervised methods in efficiency and learning performance in various downstream offline RL tasks of the DeepMind control suite.
  arXiv  Detail & Related papers  (2023-12-19T14:26:23Z)
• Feeding What You Need by Understanding What You Learned [54.400455868448695]
  Machine Reading (MRC) reveals the ability to understand a given text passage and answer questions based on it. Existing research works in MRC rely heavily on large-size models and corpus to improve the performance evaluated by metrics such as Exact Match. We argue that a deep understanding of model capabilities and data properties can help us feed a model with appropriate training data.
  arXiv  Detail & Related papers  (2022-03-05T14:15:59Z)
• How Training Data Impacts Performance in Learning-based Control [67.7875109298865]
  This paper derives an analytical relationship between the density of the training data and the control performance. We formulate a quality measure for the data set, which we refer to as $rho$-gap. We show how the $rho$-gap can be applied to a feedback linearizing control law.
  arXiv  Detail & Related papers  (2020-05-25T12:13:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.