Graph Machine: Towards Better Pretraining via Edges

• URL: http://arxiv.org/abs/2609.02881v1
• Date: Wed, 02 Sep 2026 17:56:41 GMT
• Title: Graph Machine: Towards Better Pretraining via Edges
• Abstract summary: Graph Machine (GM) architecture maintains a $O(n)$-sized state and accesses it through sparse, dynamic routing.<n>We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens.<n>With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce the Graph Machine (GM), an architecture that maintains an $O(n)$-sized state and accesses it through sparse, dynamic routing. Unlike methods with fixed-size states or sparse but static routing, GM preserves $O(n)$ complexity in its sparse layers without restricting the potentially accessible state size to $O(1)$. Instead, GM uses edges - pointer-like objects updated differentiably by a referral mechanism resembling pointer chasing. We replace 75% of the dense Transformer layers in Qwen3-0.6B with GM sparse layers and pretrain from scratch on 15.7B tokens. With only 2 of 4,096 tokens retrieved per KV head in each sparse layer, loss degrades only slightly; with 4, the best model marginally improves loss.

Related papers

• LinearKV: One Cached State Suffices for Position-Independent Caching in Hybrid LLMs [43.666302638883025]
  We present LinearKV, a training-free hybrid-PIC framework.<n>We find that a emphsingle cached state suffices as the linear layer's initializer.<n>We compare the two across three hybrid models and three PIC selectors.
  arXiv  Detail & Related papers  (2026-07-31T02:10:44Z)
• More Structure, Not More Capacity: Object-Centric Representations for Visuomotor Imitation Learning [54.65906330923846]
  We show that object-centric slot representations are a structured alternative to pre-trained vision models.<n>A dense patch grid with 16x the tokens performs no better than the global feature.<n>An explicit 2D spatial goal and native-resolution rendering raises the full system to 68.7$pm$4.2%, just below a privileged 3D-oracle upper bound.
  arXiv  Detail & Related papers  (2026-07-10T10:35:24Z)
• Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation [0.5266471639657487]
  Vision Transformers (ViTs) achieve state-of-the-art performance on challenging vision tasks, but their deployment on edge devices is hindered.<n>Recent methods attempt to bypass this by replacing normalization layers with hardware-friendly scalar approximations.<n>We propose a highly efficient framework that utilizes genetic programming (GP) to evolve heterogeneous, layer-specific scalar functions.
  arXiv  Detail & Related papers  (2026-05-13T19:08:55Z)
• Causal Concept Graphs in LLM Latent Space for Stepwise Reasoning [0.0]
  Causal Concept Graphs (CCG): a directed acyclic graph over sparse, interpretable latent features.<n>We combine task-conditioned sparse autoencoders for concept discovery with DAGMA-style differentiable structure learning for graph recovery.
  arXiv  Detail & Related papers  (2026-03-11T03:46:38Z)
• LoRIF: Low-Rank Influence Functions for Scalable Training Data Attribution [62.830878652285406]
  Training data attribution identifies which training examples most influenced a model's prediction.<n>LoRIF exploits low-rank structures of gradient to address both bottlenecks.<n>On models from 0.1B to 70B parameters trained on datasets with millions of examples, LoRIF achieves up to 20$times$ storage reduction and query-time speedup.
  arXiv  Detail & Related papers  (2026-01-29T16:18:34Z)
• LitePT: Lighter Yet Stronger Point Transformer [50.6430530112838]
  We analyse the role of different computational blocks in 3D point cloud networks.<n>We propose a new, improved 3D point cloud backbone that employs convolutions in early stages and switches to attention for deeper layers.<n>The resulting LitePT model has $3.6times$ fewer parameters, runs $2times$ faster, and uses $2times$ less memory than the state-of-the-art Point Transformer V3.
  arXiv  Detail & Related papers  (2025-12-15T18:59:57Z)
• Hierarchical Federated Graph Attention Networks for Scalable and Resilient UAV Collision Avoidance [0.5505634045241287]
  Real-time performance, adversarial resiliency, and privacy preservation are the most important metrics that need to be balanced to practice collision avoidance.<n>We have proposed an adaptive differential privacy mechanism, wherein the noise level $(in [0.1, 1.0])$ is dynamically reduced based on an evaluation of the measured real-time threat.<n>This architecture provides a scalable scenario of 500 UAVs with a collision rate of $ 2.0%$ and the Byzantine fault tolerance of $f n/3$.
  arXiv  Detail & Related papers  (2025-11-05T12:01:00Z)
• A Simple Linear Patch Revives Layer-Pruned Large Language Models [58.056251480151104]
  Layer pruning has emerged as a widely used technique for compressing large language models (LLMs)<n>textscLinearPatch fuses two operations into one matrix multiply at the pruning interface.<n>The patch can be further refined with 5K unlabeled samples via memory-efficient offline distillation, pushing the retention to 95.16% within only 30 minutes on a single GPU.
  arXiv  Detail & Related papers  (2025-05-30T15:06:08Z)
• Densely Connected $G$-invariant Deep Neural Networks with Signed Permutation Representations [6.200483285433661]
  We introduce and investigate, for finite groups $G$, $G$-invariant deep neural network ($G$-DNN) architectures with ReLU activation. The preactivations of the $G$-DNNs are able to transform by emphsigned permutation representations (signed perm-reps) of $G$. We show that there are far more admissible $G$-DNN architectures than those accessible with the concatenated ReLU'' activation function from the literature.
  arXiv  Detail & Related papers  (2023-03-08T14:35:03Z)
• Improved techniques for deterministic l2 robustness [63.34032156196848]
  Training convolutional neural networks (CNNs) with a strict 1-Lipschitz constraint under the $l_2$ norm is useful for adversarial robustness, interpretable gradients and stable training. We introduce a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last linear layer with a 1-hidden layer. We significantly advance the state-of-the-art for standard and provable robust accuracies on CIFAR-10 and CIFAR-100.
  arXiv  Detail & Related papers  (2022-11-15T19:10:12Z)
• An Information Theory-inspired Strategy for Automatic Network Pruning [97.03772272417599]
  Deep convolution neural networks are well known to be compressed on devices with resource constraints.<n>Most existing network pruning methods require laborious human efforts and prohibitive computation resources.<n>We propose an information theory-inspired strategy for automatic model compression.
  arXiv  Detail & Related papers  (2021-08-19T07:03:22Z)
• Large Scale Private Learning via Low-rank Reparametrization [77.38947817228656]
  We propose a reparametrization scheme to address the challenges of applying differentially private SGD on large neural networks. We are the first able to apply differential privacy on the BERT model and achieve an average accuracy of $83.9%$ on four downstream tasks.
  arXiv  Detail & Related papers  (2021-06-17T10:14:43Z)
• Deep Learning Meets Projective Clustering [66.726500395069]
  A common approach for compressing NLP networks is to encode the embedding layer as a matrix $AinmathbbRntimes d$. Inspired by emphprojective clustering from computational geometry, we suggest replacing this subspace by a set of $k$ subspaces.
  arXiv  Detail & Related papers  (2020-10-08T22:47:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.