Mapping the Edge of Chaos: Fractal-Like Boundaries in The Trainability of Decoder-Only Transformer Models

• URL: http://arxiv.org/abs/2501.04286v2
• Date: Sat, 15 Feb 2025 01:26:37 GMT
• Title: Mapping the Edge of Chaos: Fractal-Like Boundaries in The Trainability of Decoder-Only Transformer Models
• Authors: Bahman Torkamandi,
• Abstract summary: Recent evidence from miniature neural networks suggests that the boundary separating these outcomes displays fractal characteristics. This study extends them to medium-sized, decoder-only transformer architectures by employing a more consistent convergence measure. The results show that the trainability frontier is not a simple threshold; rather, it forms a self-similar yet seemingly random structure at multiple scales.
• Score: 0.0
• License:
• Abstract: In the realm of fractal geometry, intricate structures emerge from simple iterative processes that partition parameter spaces into regions of stability and instability. Likewise, training large language models involves iteratively applying update functions, such as Adam, where even slight hyperparameter adjustments can shift the training process from convergence to divergence. Recent evidence from miniature neural networks suggests that the boundary separating these outcomes displays fractal characteristics. Building on these insights, this study extends them to medium-sized, decoder-only transformer architectures by employing a more consistent convergence measure and examining the learning rate hyperparameter landscape for attention and fully connected layers. The results show that the trainability frontier is not a simple threshold; rather, it forms a self-similar yet seemingly random structure at multiple scales, with statistically consistent and repeating patterns. Within this landscape, a region of stable convergence is surrounded by a complex chaotic border, illustrating the sensitive nature of the underlying training dynamics.

Related papers

• ConsistentFeature: A Plug-and-Play Component for Neural Network Regularization [0.32885740436059047]
  Over- parameterized neural network models often lead to significant performance discrepancies between training and test sets. We introduce a simple perspective on overfitting: models learn different representations in different i.i.d. datasets. We propose an adaptive method, ConsistentFeature, that regularizes the model by constraining feature differences across random subsets of the same training set.
  arXiv  Detail & Related papers  (2024-12-02T13:21:31Z)
• Isomorphic Pruning for Vision Models [56.286064975443026]
  Structured pruning reduces the computational overhead of deep neural networks by removing redundant sub-structures. We present Isomorphic Pruning, a simple approach that demonstrates effectiveness across a range of network architectures.
  arXiv  Detail & Related papers  (2024-07-05T16:14:53Z)
• The boundary of neural network trainability is fractal [23.4886323538853]
  Some fractals are computed by iterating a function. Neural network training can result in convergent or divergent behavior. We find that this boundary is fractal over more than ten decades of scale in all tested configurations.
  arXiv  Detail & Related papers  (2024-02-09T04:46:48Z)
• Bayesian Unsupervised Disentanglement of Anatomy and Geometry for Deep Groupwise Image Registration [50.62725807357586]
  This article presents a general Bayesian learning framework for multi-modal groupwise image registration. We propose a novel hierarchical variational auto-encoding architecture to realise the inference procedure of the latent variables. Experiments were conducted to validate the proposed framework, including four different datasets from cardiac, brain, and abdominal medical images.
  arXiv  Detail & Related papers  (2024-01-04T08:46:39Z)
• Exploiting hidden structures in non-convex games for convergence to Nash equilibrium [62.88214569402201]
  A wide array of modern machine learning applications can be formulated as non-cooperative Nashlibria. We provide explicit convergence guarantees for both deterministic and deterministic environments.
  arXiv  Detail & Related papers  (2023-12-27T15:21:25Z)
• Curve Your Attention: Mixed-Curvature Transformers for Graph Representation Learning [77.1421343649344]
  We propose a generalization of Transformers towards operating entirely on the product of constant curvature spaces. We also provide a kernelized approach to non-Euclidean attention, which enables our model to run in time and memory cost linear to the number of nodes and edges.
  arXiv  Detail & Related papers  (2023-09-08T02:44:37Z)
• Latent Traversals in Generative Models as Potential Flows [113.4232528843775]
  We propose to model latent structures with a learned dynamic potential landscape. Inspired by physics, optimal transport, and neuroscience, these potential landscapes are learned as physically realistic partial differential equations. Our method achieves both more qualitatively and quantitatively disentangled trajectories than state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-04-25T15:53:45Z)
• Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry [3.712728573432119]
  We develop a standardized parameterization in which all symmetries are removed, resulting in a toroidal topology. We derive a meaningful notion of the flatness of minimizers and of the geodesic paths connecting them. We also find that minimizers found by variants of gradient descent can be connected by zero-error paths with a single bend.
  arXiv  Detail & Related papers  (2022-02-07T09:57:54Z)
• CSformer: Bridging Convolution and Transformer for Compressive Sensing [65.22377493627687]
  This paper proposes a hybrid framework that integrates the advantages of leveraging detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning. The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery. The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing.
  arXiv  Detail & Related papers  (2021-12-31T04:37:11Z)
• ProAlignNet : Unsupervised Learning for Progressively Aligning Noisy Contours [12.791313859673187]
  "ProAlignNet" accounts for large scale misalignments and complex transformations between the contour shapes. It learns by training with a novel loss function which is derived an upperbound of a proximity-sensitive and local shape-dependent similarity metric. In two real-world applications, the proposed models consistently perform superior to state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-05-23T14:56:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.