Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows

• URL: http://arxiv.org/abs/2607.07504v1
• Date: Wed, 08 Jul 2026 15:00:16 GMT
• Title: Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows
• Abstract summary: Reusable skill files are meant to avoid prompting from scratch by packaging guidance for a task family.<n>Expert-written skills can encode high-quality guidance, but writing and maintaining them across many data-science task families creates a manual bottleneck.<n>We ask whether LLM-generated skills offer a useful low-curation alternative: do they improve performance over the task prompt alone?
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Product data scientists often ask LLM-based agents to help with recurring execution tasks such as cleaning data, writing SQL, choosing statistical tests, and formatting results. Reusable skill files are meant to avoid prompting from scratch by packaging guidance for a task family. Expert-written skills can encode high-quality guidance, but writing and maintaining them across many data-science task families creates a manual bottleneck. We ask whether LLM-generated skills offer a useful low-curation alternative: do they improve performance over the task prompt alone? We test this question across four lifecycle stages: data preparation, data extraction, statistical analysis, and reporting, using one generated skill per stage. We find no reliable improvement from full generated skills over No-Skill prompting. We then ask whether any part of the skill is useful by ablating different skill components. The main ablation covers 56 tasks, nine model configurations, and three providers, yielding 7,560 runs. Compared with prompting using the task alone, neither the full generated skill nor any ablated skill variant significantly improves performance; all p-values are at least 0.396, and the total spread across variants is only 1.2 pp. A supplemental token-matched control adds 1,512 runs and finds that Full skills perform similarly to task-irrelevant skill-formatted content. The results caution against using one LLM-generated skill per data-science workflow as a default single-shot prompting strategy.

Related papers

• Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents [17.291924088593408]
  Large language model (LLM) agents can induce skills from completed tasks and reuse them later to grow more capable with experience.<n>In practice, induced skills may transfer unreliably and can even harm the agent that retrieves them.<n>We conduct a comprehensive and controlled study of how the way skills are induced shapes their transfer across tasks.
  arXiv  Detail & Related papers  (2026-08-20T17:12:08Z)
• Demystifying Agent Skills: Why They Work-Until They Don't [58.92875630998418]
  Existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: emphtextbfWhen do skills help, why do they work, and where do they fail?<n>We isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework of skills.<n>We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes.
  arXiv  Detail & Related papers  (2026-08-14T07:26:38Z)
• SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance [10.175883215226776]
  SkillSentry is a skill-oriented runtime assurance framework for representing runtime guidance for skill execution.<n>We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models.<n>Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.
  arXiv  Detail & Related papers  (2026-08-10T08:13:52Z)
• Not All Skills Help: Measuring and Repairing Agent Knowledge [47.498075849595374]
  ASSAY is a framework that separates generation from curation.<n>It computes a per-skill causal attribution on a small development set.<n>It restructures the library offline, and suppresses skills with negative predicted effect for each test task.
  arXiv  Detail & Related papers  (2026-06-13T16:37:38Z)
• SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization [61.37478254738943]
  Inference-time skill augmentation is fundamentally limited.<n>We introduce SKILL0, an in-context reinforcement learning framework designed for skill internalization.<n>SKILL0 achieves substantial improvements over the standard RL baseline.
  arXiv  Detail & Related papers  (2026-04-02T17:03:05Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning [93.96463520716759]
  Large language model (LLM) agents have demonstrated impressive capabilities in utilizing external tools and knowledge to boost accuracy and hallucinations. Here, we introduce AvaTaR, a novel and automated framework that optimize an LLM agent to effectively leverage provided tools, improving performance on a given task.
  arXiv  Detail & Related papers  (2024-06-17T04:20:02Z)
• Instances Need More Care: Rewriting Prompts for Instances with LLMs in the Loop Yields Better Zero-Shot Performance [11.595274304409937]
  Large language models (LLMs) have revolutionized zero-shot task performance. Current methods using trigger phrases such as "Let's think step by step" remain limited. This study introduces PRomPTed, an approach that optimize the zero-shot prompts for individual task instances.
  arXiv  Detail & Related papers  (2023-10-03T14:51:34Z)
• Large Language Models as Batteries-Included Zero-Shot ESCO Skills Matchers [0.0]
  We propose an end-to-end zero-shot system for skills extraction from job descriptions based on large language models (LLMs) We generate synthetic training data for the entirety of ESCO skills and train a classifier to extract skill mentions from job posts. We also employ a similarity retriever to generate skill candidates which are then re-ranked using a second LLM.
  arXiv  Detail & Related papers  (2023-07-07T12:04:12Z)
• Design of Negative Sampling Strategies for Distantly Supervised Skill Extraction [19.43668931500507]
  We propose an end-to-end system for skill extraction, based on distant supervision through literal matching. We observe that using the ESCO taxonomy to select negative examples from related skills yields the biggest improvements. We release the benchmark dataset for research purposes to stimulate further research on the task.
  arXiv  Detail & Related papers  (2022-09-13T13:37:06Z)
• KnowDA: All-in-One Knowledge Mixture Model for Data Augmentation in Few-Shot NLP [68.43279384561352]
  Existing data augmentation algorithms leverage task-independent rules or fine-tune general-purpose pre-trained language models. These methods have trivial task-specific knowledge and are limited to yielding low-quality synthetic data for weak baselines in simple tasks. We propose the Knowledge Mixture Data Augmentation Model (KnowDA): an encoder-decoder LM pretrained on a mixture of diverse NLP tasks.
  arXiv  Detail & Related papers  (2022-06-21T11:34:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.