The Power of Power Law: Asymmetry Enables Compositional Reasoning

• URL: http://arxiv.org/abs/2604.22951v1
• Date: Fri, 24 Apr 2026 18:49:08 GMT
• Title: The Power of Power Law: Asymmetry Enables Compositional Reasoning
• Abstract summary: We show that learning under a power-law distribution provably requires significantly less training data.<n>Our results offer an alternative perspective on what constitutes an effective data distribution for training models.
• Score: 76.90253944094773
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Natural language data follows a power-law distribution, with most knowledge and skills appearing at very low frequency. While a common intuition suggests that reweighting or curating data towards a uniform distribution may help models better learn these long-tail skills, we find a counterintuitive result: across a wide range of compositional reasoning tasks, such as state tracking and multi-step arithmetic, training under power-law distributions consistently outperforms training under uniform distributions. To understand this advantage, we introduce a minimalist skill-composition task and show that learning under a power-law distribution provably requires significantly less training data. Our theoretical analysis reveals that power law sampling induces a beneficial asymmetry that improves the pathological loss landscape, which enables models to first acquire high-frequency skill compositions with low data complexity, which in turn serves as a stepping stone to efficiently learn rare long-tailed skills. Our results offer an alternative perspective on what constitutes an effective data distribution for training models.

Related papers

• Superposition unifies power-law training dynamics [6.874132000933929]
  We investigate the role of feature superposition in the emergence of power-law training dynamics.<n>We first derive an analytic theory for training without superposition.<n>We discover that a superposition bottleneck induces a transition to a universal power-law exponent of $sim 1$.
  arXiv  Detail & Related papers  (2026-02-01T06:05:18Z)
• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training [25.91869315787235]
  We present a principled curriculum learning framework grounded in the notion of distribution-level learnability.<n>Our framework prioritizes distributions with either high average advantage (exploitation) or low sample count (exploration)<n>Our experiments show that our framework significantly improves convergence speed and final performance.
  arXiv  Detail & Related papers  (2025-04-13T20:10:27Z)
• Probabilistic Contrastive Learning for Long-Tailed Visual Recognition [78.70453964041718]
  Longtailed distributions frequently emerge in real-world data, where a large number of minority categories contain a limited number of samples. Recent investigations have revealed that supervised contrastive learning exhibits promising potential in alleviating the data imbalance. We propose a novel probabilistic contrastive (ProCo) learning algorithm that estimates the data distribution of the samples from each class in the feature space.
  arXiv  Detail & Related papers  (2024-03-11T13:44:49Z)
• Agree to Disagree: Diversity through Disagreement for Better Transferability [54.308327969778155]
  We propose D-BAT (Diversity-By-disAgreement Training), which enforces agreement among the models on the training data. We show how D-BAT naturally emerges from the notion of generalized discrepancy.
  arXiv  Detail & Related papers  (2022-02-09T12:03:02Z)
• Resource-Efficient Federated Learning [3.654036881216688]
  Federated Learning (FL) enables distributed training by learners using local data. It presents numerous challenges relating to the data distribution, device capabilities, and participant availability as scale deployments.
  arXiv  Detail & Related papers  (2021-11-01T17:21:07Z)
• Powerpropagation: A sparsity inducing weight reparameterisation [65.85142037667065]
  We introduce Powerpropagation, a new weight- parameterisation for neural networks that leads to inherently sparse models. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark.
  arXiv  Detail & Related papers  (2021-10-01T10:03:57Z)
• Adversarial Training Helps Transfer Learning via Better Representations [17.497590668804055]
  Transfer learning aims to leverage models pre-trained on source data to efficiently adapt to target setting. Recent works empirically demonstrate that adversarial training in the source data can improve the ability of models to transfer to new domains. We show that adversarial training in the source data generates provably better representations, so fine-tuning on top of this representation leads to a more accurate predictor of the target data.
  arXiv  Detail & Related papers  (2021-06-18T15:41:07Z)
• Fair Densities via Boosting the Sufficient Statistics of Exponential Families [72.34223801798422]
  We introduce a boosting algorithm to pre-process data for fairness. Our approach shifts towards better data fitting while still ensuring a minimal fairness guarantee. Empirical results are present to display the quality of result on real-world data.
  arXiv  Detail & Related papers  (2020-12-01T00:49:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.