Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients

• URL: http://arxiv.org/abs/2010.01709v1
• Date: Sun, 4 Oct 2020 22:32:51 GMT
• Title: Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients
• Authors: William S. Moses and Valentin Churavy
• Abstract summary: This paper presents Enzyme, a high-performance automatic differentiation (AD) compiler plugin for the LLVM compiler framework. Enzyme synthesizes gradients for programs written in any language whose compiler targets LLVM intermediate representation (IR) On a machine-learning focused benchmark suite including Microsoft's ADBench, AD on optimized IR achieves a geometric mean speedup of 4.5x over AD on IR.
• Score: 6.09170287691728
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Applying differentiable programming techniques and machine learning algorithms to foreign programs requires developers to either rewrite their code in a machine learning framework, or otherwise provide derivatives of the foreign code. This paper presents Enzyme, a high-performance automatic differentiation (AD) compiler plugin for the LLVM compiler framework capable of synthesizing gradients of statically analyzable programs expressed in the LLVM intermediate representation (IR). Enzyme synthesizes gradients for programs written in any language whose compiler targets LLVM IR including C, C++, Fortran, Julia, Rust, Swift, MLIR, etc., thereby providing native AD capabilities in these languages. Unlike traditional source-to-source and operator-overloading tools, Enzyme performs AD on optimized IR. On a machine-learning focused benchmark suite including Microsoft's ADBench, AD on optimized IR achieves a geometric mean speedup of 4.5x over AD on IR before optimization allowing Enzyme to achieve state-of-the-art performance. Packaging Enzyme for PyTorch and TensorFlow provides convenient access to gradients of foreign code with state-of-the art performance, enabling foreign code to be directly incorporated into existing machine learning workflows.

Related papers

• Forklift: An Extensible Neural Lifter [11.633770744027682]
  We propose Forklift, the first neural lifter that learns how to translate assembly to LLVM IR using a token-level encoder-decoder Transformer. We collect millions of parallel LLVM IR, x86, ARM, and RISC-V programs across compilers and optimization levels to train Forklift and set up an input/output-based accuracy harness. We evaluate Forklift on two challenging benchmark suites and translate 2.5x more x86 programs than a state-of-the-art hand-written lifter and 4.4x more x86 programs than GPT-4 as well as enabling translation from new ISAs.
  arXiv  Detail & Related papers  (2024-04-01T17:27:58Z)
• LILO: Learning Interpretable Libraries by Compressing and Documenting Code [71.55208585024198]
  We introduce LILO, a neurosymbolic framework that iteratively synthesizes, compresses, and documents code. LILO combines LLM-guided program synthesis with recent algorithmic advances in automated from Stitch. We find that AutoDoc boosts performance by helping LILO's synthesizer to interpret and deploy learned abstractions.
  arXiv  Detail & Related papers  (2023-10-30T17:55:02Z)
• mlirSynth: Automatic, Retargetable Program Raising in Multi-Level IR using Program Synthesis [48.01697184432969]
  mlirSynth translates programs from lower-level MLIR dialects to high-level ones without manually defined rules. We demonstrate its effectiveness reviby raising C programs to two distinct high-level MLIR dialects, which enables us to use existing high-level dialect specific compilation flows.
  arXiv  Detail & Related papers  (2023-10-06T12:21:50Z)
• Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering [52.09178018466104]
  We introduce Context-Aware Automated Feature Engineering (CAAFE) to generate semantically meaningful features for datasets. Despite being methodologically simple, CAAFE improves performance on 11 out of 14 datasets. We highlight the significance of context-aware solutions that can extend the scope of AutoML systems to semantic AutoML.
  arXiv  Detail & Related papers  (2023-05-05T09:58:40Z)
• ML-driven Hardware Cost Model for MLIR [1.2987894327817158]
  We develop a machine learning-based cost model for high-level MLIR. By considering the incoming MLIR as a text input a la NLP models we can apply well-known techniques from modern NLP research. We show that these models can provide reasonably good estimates with low error bounds for various hardware characteristics of interest.
  arXiv  Detail & Related papers  (2023-02-14T11:32:47Z)
• Efficient and Sound Differentiable Programming in a Functional Array-Processing Language [4.1779847272994495]
  Automatic differentiation (AD) is a technique for computing the derivative of a function represented by a program. We present an AD system for a higher-order functional array-processing language. In combination, computation with forward-mode AD can be as efficient as reverse mode.
  arXiv  Detail & Related papers  (2022-12-20T14:54:47Z)
• Enabling Retargetable Optimizing Compilers for Quantum Accelerators via a Multi-Level Intermediate Representation [78.8942067357231]
  We present a multi-level quantum-classical intermediate representation (IR) that enables an optimizing, retargetable, ahead-of-time compiler. We support the entire gate-based OpenQASM 3 language and provide custom extensions for common quantum programming patterns and improved syntax. Our work results in compile times that are 1000x faster than standard Pythonic approaches, and 5-10x faster than comparative standalone quantum language compilers.
  arXiv  Detail & Related papers  (2021-09-01T17:29:47Z)
• Deep Data Flow Analysis [14.583644439728895]
  ProGraML is a portable representation of whole-program semantics for deep learning. We benchmark current and future learning techniques for compiler analyses. We show that, using ProGraML, standard analyses can be learned and improved performance on downstream compiler optimization tasks.
  arXiv  Detail & Related papers  (2020-11-21T03:29:14Z)
• Predictive Coding Approximates Backprop along Arbitrary Computation Graphs [68.8204255655161]
  We develop a strategy to translate core machine learning architectures into their predictive coding equivalents. Our models perform equivalently to backprop on challenging machine learning benchmarks. Our method raises the potential that standard machine learning algorithms could in principle be directly implemented in neural circuitry.
  arXiv  Detail & Related papers  (2020-06-07T15:35:47Z)
• Automatic Differentiation in ROOT [62.997667081978825]
  In mathematics and computer algebra, automatic differentiation (AD) is a set of techniques to evaluate the derivative of a function specified by a computer program. This paper presents AD techniques available in ROOT, supported by Cling, to produce derivatives of arbitrary C/C++ functions.
  arXiv  Detail & Related papers  (2020-04-09T09:18:50Z)
• EVA: An Encrypted Vector Arithmetic Language and Compiler for Efficient Homomorphic Computation [11.046862694768894]
  This paper presents a new FHE language called Encrypted Vector Arithmetic (EVA) EVA includes an optimizing compiler that generates correct and secure FHE programs. programmers can develop efficient general-purpose FHE applications directly in EVA.
  arXiv  Detail & Related papers  (2019-12-27T00:24:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.