Tensor Seeks Layout: Formalizing Layout Selection for ML Compilers

• URL: http://arxiv.org/abs/2608.21555v1
• Date: Fri, 21 Aug 2026 18:44:48 GMT
• Title: Tensor Seeks Layout: Formalizing Layout Selection for ML Compilers
• Abstract summary: This paper presents the first formal study of layout selection in machine learning compilers.<n>We formulate the problem as optimization over dataflow graphs, minimizing the sum of operator execution costs and the per-tensor cost of these conversions.<n>We implement the formalization in a production compiler for an AI accelerator and measure the execution time of the compiled models under greedys.
• Score: 3.2480918384725306
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern machine learning compilers select tensor memory layouts to minimize execution cost under hardware constraints. Layout selection is global: an operator may be fastest under one layout while its consumers prefer another, and aligning these preferences requires explicit layout conversions that can hurt model performance. Despite its practical importance, layout selection lacks a formal basis, so current compilers rely on ad-hoc heuristics. This paper presents the first formal study of layout selection in machine learning compilers. We formulate the problem as combinatorial optimization over dataflow graphs, minimizing the sum of operator execution costs and the per-tensor cost of these conversions. Our theoretical analysis shows that optimal layout selection is computationally hard, even for programs containing only matrix multiplications over two-dimensional tensors. We design an optimal polynomial-time algorithm for dataflow graphs of bounded treewidth. For general instances, we give a weighted MaxSAT encoding that an off-the-shelf solver can optimize. The formulation unifies several existing layout optimization strategies, including XLA's layout assignment, partition dimension selection in systolic array compilers, and layout planning in mobile GPU optimizers. We implement the formalization in a production compiler for an AI accelerator and measure the execution time of the compiled models under greedy heuristics, the compiler's rule-based strategy, and an optimal solver. Simple heuristics degrade execution time by up to $5\times$ on some workloads. Where the compiler's cost model is accurate, the solver matches or beats the rule-based strategy. On workloads with complex data movement it falls behind, and since the solver minimizes the stated objective exactly, that gap isolates cost-model error from search quality, showing where compiler effort actually pays off.

Related papers

• SkyEgg: Joint Implementation Selection and Scheduling for Hardware Synthesis using E-graphs [2.363352915727704]
  We present SkyEgg, a novel hardware synthesis framework that jointly optimize implementation selection and scheduling.<n>SkyEgg achieves an average speedup of 3.01x over Vitis, with improvements up to 5.22x for complex expressions.
  arXiv  Detail & Related papers  (2025-11-19T10:39:45Z)
• Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression [57.54335545892155]
  We introduce a Grouped Lattice Vector Quantization (GLVQ) framework that assigns each group of weights a customized lattice codebook.<n>Our approach achieves a better trade-off between model size and accuracy compared to existing post-training quantization baselines.
  arXiv  Detail & Related papers  (2025-10-23T20:19:48Z)
• FFT-based Dynamic Subspace Selection for Low-Rank Adaptive Optimization of Large Language Models [49.397861654088636]
  We propose a two-step procedure to approximate SVD/QR-based gradient projections into lower-dimensional spaces.<n>We show that our strategy achieves faster runtime and reduced memory usage by up to $25%$ across different model sizes.
  arXiv  Detail & Related papers  (2025-05-23T14:37:00Z)
• Morello: Compiling Fast Neural Networks with Dynamic Programming and Spatial Compression [5.995843028932167]
  We introduce a dynamic-programming-based approach to explore more of the search space by decomposing large program specifications into smaller specifications.<n>To reduce memory requirements, we employ a novel memoization table representation, which indexes specifications by coordinates in $Z_geq 0$ and compresses identical, adjacent solutions.
  arXiv  Detail & Related papers  (2025-05-03T00:14:31Z)
• CompilerDream: Learning a Compiler World Model for General Code Optimization [58.87557583347996]
  We introduce CompilerDream, a model-based reinforcement learning approach to general code optimization.<n>It comprises a compiler world model that accurately simulates the intrinsic properties of optimization passes and an agent trained on this model to produce effective optimization strategies.<n>It excels across diverse datasets, surpassing LLVM's built-in optimizations and other state-of-the-art methods in both settings of value prediction and end-to-end code optimization.
  arXiv  Detail & Related papers  (2024-04-24T09:20:33Z)
• Large Language Models for Compiler Optimization [22.52765975286403]
  We present a transformer model trained from scratch to optimize LLVM assembly for code size. We ask the model to predict the instruction counts before and after optimization, and the optimized code itself. Our approach achieves a 3.0% improvement in reducing instruction counts over the compiler.
  arXiv  Detail & Related papers  (2023-09-11T22:11:46Z)
• ALT: Breaking the Wall between Graph and Operator Level Optimizations for Deep Learning Compilation [38.8918502461244]
  ALT is a compiler that performs joint graph- and operator-level optimizations for deep models. JOG significantly outperforms state-of-the-art compilers (e.g., Ansor) in terms of both single operator performance and end-to-end inference performance.
  arXiv  Detail & Related papers  (2022-10-22T11:09:36Z)
• Towards Optimal VPU Compiler Cost Modeling by using Neural Networks to Infer Hardware Performances [58.720142291102135]
  'VPUNN' is a neural network-based cost model trained on low-level task profiling. It consistently outperforms the state-of-the-art cost modeling in Intel's line of VPU processors.
  arXiv  Detail & Related papers  (2022-05-09T22:48:39Z)
• A Deep Learning Based Cost Model for Automatic Code Optimization [0.24629531282150877]
  We present a novel deep learning based cost model for automatic code optimization. It was integrated in the Tiramisu compiler to select the best code transformations. The proposed model has only 16% of mean absolute percentage error in predicting speedups on full programs.
  arXiv  Detail & Related papers  (2021-04-11T08:32:42Z)
• Kernel methods through the roof: handling billions of points efficiently [94.31450736250918]
  Kernel methods provide an elegant and principled approach to nonparametric learning, but so far could hardly be used in large scale problems. Recent advances have shown the benefits of a number of algorithmic ideas, for example combining optimization, numerical linear algebra and random projections. Here, we push these efforts further to develop and test a solver that takes full advantage of GPU hardware.
  arXiv  Detail & Related papers  (2020-06-18T08:16:25Z)
• Global Optimization of Gaussian processes [52.77024349608834]
  We propose a reduced-space formulation with trained Gaussian processes trained on few data points. The approach also leads to significantly smaller and computationally cheaper sub solver for lower bounding. In total, we reduce time convergence by orders of orders of the proposed method.
  arXiv  Detail & Related papers  (2020-05-21T20:59:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.