Differentiate Everything with a Reversible Embeded Domain-Specific Language

• URL: http://arxiv.org/abs/2003.04617v3
• Date: Sun, 31 Jan 2021 23:03:03 GMT
• Title: Differentiate Everything with a Reversible Embeded Domain-Specific Language
• Abstract summary: Reverse-mode automatic differentiation (AD) suffers from the issue of having too much space overhead to trace back intermediate computational states for back-propagation. We propose to use reverse computing to trace back states by designing and implementing a reversible programming e. The absence of implicit stack operations makes the program compatible with existing compiler features.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reverse-mode automatic differentiation (AD) suffers from the issue of having too much space overhead to trace back intermediate computational states for back-propagation. The traditional method to trace back states is called checkpointing that stores intermediate states into a global stack and restore state through either stack pop or re-computing. The overhead of stack manipulations and re-computing makes the general purposed (not tensor-based) AD engines unable to meet many industrial needs. Instead of checkpointing, we propose to use reverse computing to trace back states by designing and implementing a reversible programming eDSL, where a program can be executed bi-directionally without implicit stack operations. The absence of implicit stack operations makes the program compatible with existing compiler features, including utilizing existing optimization passes and compiling the code as GPU kernels. We implement AD for sparse matrix operations and some machine learning applications to show that our framework has the state-of-the-art performance.

Related papers

• No Bit Left Behind: Using Brute-Force Lifting to Achieve Fully Static Binary Recompilation [1.5034998782245808]
  This article presents a fully static, whole-program binary lifting system requiring no runtime translation support on the target.<n>We have implemented this approach as a prototype binary runtimer from x86-64 binaries to LLVM IR, requiring no code/data runtimes.
  arXiv  Detail & Related papers  (2026-09-14T23:00:09Z)
• A Symbolic Neural CPU for Quantization-Simulated Writeback and Interpretable Program Execution [0.0]
  We introduce a trace-supervised symbolic neural CPU that combines recurrent control, an explicit operation router and trajectory supervision.<n>The model exposes the selected operation, source and destination registers, register trajectory, memory signals and writeback semantics.<n>Together, these results establish a trace-verifiable framework for interpretable, low-precision and controllable neural execution.
  arXiv  Detail & Related papers  (2026-07-10T22:55:23Z)
• PLACEMEM: Toward a Compute-Aware Memory Plane for Lifelong Agents [0.0]
  We present PLACEMEM as a systems position on lifelong-agent memory, instantiated by an executable control-plane prototype.<n>The central claim is that agent memory should be represented as versioned capsules that unify semantics, provenance, validity, and reusable runtime state under one correction-aware.<n>The result is both an executable artifact that demonstrates correction-aware control-plane behavior today and a concrete roadmap for replay-aware serving integration in future lifelong-agent systems.
  arXiv  Detail & Related papers  (2026-07-05T02:55:10Z)
• Compile Once, Differentiate Everywhere: A Differentiable Meta-Circular Interpreter [0.0]
  We present a compiler that translates a self-hosting subset of Scheme into differentiable graphs for autograd backends.<n>Because the subset can compile its own evaluator, this yields differentiable meta-circular interpretation (DMCI)<n>DMCI executes programs supplied as data, while reverse-mode autodiff propagates gradients to continuous constants embedded in those programs.
  arXiv  Detail & Related papers  (2026-06-07T15:13:41Z)
• Efficient Matrix Implementation for Rotary Position Embedding [50.45885099010382]
  Rotary Position Embedding (RoPE) has become a core component of modern Transformer architectures across language, vision, and 3D domains.<n>We propose RoME, a mathematically equivalent yet computationally efficient reformulation of RoPE that replaces vector operations with unified matrix transformations.<n> experiments show that RoME delivers substantial acceleration at both the operator and full-model levels.
  arXiv  Detail & Related papers  (2026-04-10T00:17:47Z)
• Compiling Quantum Regular Language States [0.08030359871216612]
  We develop a quantum state-preparation compiler for regular language states.<n>Users describe the target state via (i) a finite set of bitstrings, (ii) a regular expression, or (iii) a deterministic finite automaton (DFA)<n>The efficient DFA representation and offloads expensive linear algebra minimization in exchange of simpler automata manipulations.
  arXiv  Detail & Related papers  (2026-02-02T19:11:38Z)
• DaCe AD: Unifying High-Performance Automatic Differentiation for Machine Learning and Scientific Computing [54.73410106410609]
  This work presents DaCe AD, a general, efficient automatic differentiation engine that requires no code modifications.<n>DaCe AD uses a novel ILP-based algorithm to optimize the trade-off between storing and recomputing to achieve maximum performance within a given memory constraint.
  arXiv  Detail & Related papers  (2025-09-02T11:09:45Z)
• StackTrans: From Large Language Model to Large Pushdown Automata Model [63.37860663635374]
  The Transformer architecture has emerged as a landmark advancement within the broad field of artificial intelligence.<n>We propose StackTrans to address the issue within large language models (LLMs)<n>Unlike previous approaches that modify the attention computation, StackTrans explicitly incorporates hidden state stacks between Transformer layers.
  arXiv  Detail & Related papers  (2025-07-21T07:58:03Z)
• MIND-Stack: Modular, Interpretable, End-to-End Differentiability for Autonomous Navigation [2.07180164747172]
  We present MIND-Stack, a modular software stack consisting of a localization network and a Stanley Controller.<n>Our approach enables the upstream localization module to reduce the downstream control error, extending its role beyond state estimation.<n>We conduct experiments that demonstrate the ability of the localization module to reduce the downstream control loss through its end-to-end differentiability.
  arXiv  Detail & Related papers  (2025-05-27T20:23:21Z)
• PaTH Attention: Position Encoding via Accumulating Householder Transformations [56.32365080761523]
  PaTH is a flexible data-dependent position encoding scheme based on accumulated products of Householder transformations.<n>We derive an efficient parallel algorithm for training through exploiting a compact representation of products of Householder matrices.
  arXiv  Detail & Related papers  (2025-05-22T08:36:09Z)
• ReF Decompile: Relabeling and Function Call Enhanced Decompile [50.86228893636785]
  The goal of decompilation is to convert compiled low-level code (e.g., assembly code) back into high-level programming languages. This task supports various reverse engineering applications, such as vulnerability identification, malware analysis, and legacy software migration.
  arXiv  Detail & Related papers  (2025-02-17T12:38:57Z)
• Sparser, Better, Faster, Stronger: Efficient Automatic Differentiation for Sparse Jacobians and Hessians [0.0]
  Jacobians and Hessians have many potential use cases in Machine Learning (ML), but conventional wisdom views them as computationally prohibitive. This paper presents advances in Automatic Sparse Differentiation (ASD), starting with a new perspective on sparsity detection. We also describe a novel ASD pipeline in Julia, consisting of independent software packages for sparsity detection, matrix coloring, and differentiation.
  arXiv  Detail & Related papers  (2025-01-29T16:21:54Z)
• SNIP: Speculative Execution and Non-Interference Preservation for Compiler Transformations [0.15800607910450126]
  We address the problem of preserving non-interference across compiler transformations under speculative semantics. We develop a proof method that ensures the preservation uniformly across all source programs.
  arXiv  Detail & Related papers  (2024-07-21T07:30:30Z)
• ReGAL: Refactoring Programs to Discover Generalizable Abstractions [59.05769810380928]
  Generalizable Abstraction Learning (ReGAL) is a method for learning a library of reusable functions via codeization. We find that the shared function libraries discovered by ReGAL make programs easier to predict across diverse domains. For CodeLlama-13B, ReGAL results in absolute accuracy increases of 11.5% on LOGO, 26.1% on date understanding, and 8.1% on TextCraft, outperforming GPT-3.5 in two of three domains.
  arXiv  Detail & Related papers  (2024-01-29T18:45:30Z)
• CRIL: A Concurrent Reversible Intermediate Language [0.0]
  We present a reversible intermediate language with composing for translating a high-level concurrent programming language to another lower-level concurrent programming language, keeping reversibility. We propose CRIL as an extension of RIL used by Mogensen for a functional reversible language, incorporating a multi-thread process invocation and the synchronization primitives based on the P-V operations.
  arXiv  Detail & Related papers  (2023-09-13T20:52:54Z)
• AutoCoreset: An Automatic Practical Coreset Construction Framework [65.37876706107764]
  A coreset is a tiny weighted subset of an input set, that closely resembles the loss function. We propose an automatic framework for constructing coresets, which requires only the input data and the desired cost function from the user. We show that while this set is limited, the coreset is quite general.
  arXiv  Detail & Related papers  (2023-05-19T19:59:52Z)
• DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles [75.43355868143209]
  We present DiffStack, a differentiable and modular stack for prediction, planning, and control. Our results on the nuScenes dataset indicate that end-to-end training with DiffStack yields substantial improvements in open-loop and closed-loop planning metrics.
  arXiv  Detail & Related papers  (2022-12-13T09:05:21Z)
• QParallel: Explicit Parallelism for Programming Quantum Computers [62.10004571940546]
  We present a language extension for parallel quantum programming. QParallel removes ambiguities concerning parallelism in current quantum programming languages. We introduce a tool that guides programmers in the placement of parallel regions by identifying the subroutines that profit most from parallelization.
  arXiv  Detail & Related papers  (2022-10-07T16:35:16Z)
• Exploiting Long-Distance Interactions and Tolerating Atom Loss in Neutral Atom Quantum Architectures [4.979871961444077]
  We evaluate the advantages and disadvantages of a Neutral Atom (NA) architecture. NA systems offer several promising advantages such as long range interactions and native multiqubit gates. We propose hardware and compiler methods to increase system resilience to atom loss dramatically reducing total computation time.
  arXiv  Detail & Related papers  (2021-11-11T21:34:31Z)
• High-performance symbolic-numerics via multiple dispatch [52.77024349608834]
  Symbolics.jl is an extendable symbolic system which uses dynamic multiple dispatch to change behavior depending on the domain needs. We show that by formalizing a generic API on actions independent of implementation, we can retroactively add optimized data structures to our system. We demonstrate the ability to swap between classical term-rewriting simplifiers and e-graph-based term-rewriting simplifiers.
  arXiv  Detail & Related papers  (2021-05-09T14:22:43Z)
• Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients [6.09170287691728]
  This paper presents Enzyme, a high-performance automatic differentiation (AD) compiler plugin for the LLVM compiler framework. Enzyme synthesizes gradients for programs written in any language whose compiler targets LLVM intermediate representation (IR) On a machine-learning focused benchmark suite including Microsoft's ADBench, AD on optimized IR achieves a geometric mean speedup of 4.5x over AD on IR.
  arXiv  Detail & Related papers  (2020-10-04T22:32:51Z)
• Tensor Relational Algebra for Machine Learning System Design [7.764107702934616]
  We present an alternative implementation abstraction called the relational tensor algebra (TRA) TRA is a set-based algebra based on the relational algebra. Our empirical study shows that the optimized TRA-based back-end can significantly outperform alternatives for running ML in distributed clusters.
  arXiv  Detail & Related papers  (2020-09-01T15:51:24Z)
• Automatic Differentiation in ROOT [62.997667081978825]
  In mathematics and computer algebra, automatic differentiation (AD) is a set of techniques to evaluate the derivative of a function specified by a computer program. This paper presents AD techniques available in ROOT, supported by Cling, to produce derivatives of arbitrary C/C++ functions.
  arXiv  Detail & Related papers  (2020-04-09T09:18:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.