TinyTorch: Building Machine Learning Systems from First Principles

• URL: http://arxiv.org/abs/2601.19107v2
• Date: Wed, 28 Jan 2026 19:22:22 GMT
• Title: TinyTorch: Building Machine Learning Systems from First Principles
• Abstract summary: Students learn algorithms without understanding the systems that execute them.<n>This "algorithmsystems divide" produces practitioners who can train models but cannot debug memory failures.<n>We present TinyTorch, a curriculum that closes this gap through "Requiring systems pedagogy"<n>The curriculum is available open-source at mlsysbook.ai/tinytorch.
• Score: 7.557409593010283
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning education faces a fundamental gap: students learn algorithms without understanding the systems that execute them. They study gradient descent without measuring memory, attention mechanisms without analyzing O(N^2) scaling, optimizer theory without knowing why Adam requires 3x the memory of SGD. This "algorithm-systems divide" produces practitioners who can train models but cannot debug memory failures, optimize inference latency, or reason about deployment trade-offs--the very skills industry demands as "ML systems engineering." We present TinyTorch, a 20-module curriculum that closes this gap through "implementation-based systems pedagogy": students construct PyTorch's core components (tensors, autograd, optimizers, CNNs, transformers) in pure Python, building a complete framework where every operation they invoke is code they wrote. The design employs three patterns: "progressive disclosure" of complexity, "systems-first integration" of profiling from the first module, and "build-to-validate milestones" recreating 67 years of ML breakthroughs--from Perceptron (1958) through Transformers (2017) to MLPerf-style benchmarking. Requiring only 4GB RAM and no GPU, TinyTorch demonstrates that deep ML systems understanding is achievable without specialized hardware. The curriculum is available open-source at mlsysbook.ai/tinytorch.

Related papers

• BluTrain: A C++/CUDA Framework for AI Systems [2.79727705929128]
  BluTrain is a robust, lightweight, architecture-general training framework in standard C++ and the core programming model.<n>In formal evaluations BluTrain outperforms industry-standard baselines in both throughput and memory efficiency.<n>With every layer explicitly open to native tuning, the performance ceiling is the framework's own to raise.
  arXiv  Detail & Related papers  (2026-06-23T16:39:25Z)
• MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI [92.82821272460423]
  MLS-Bench is a benchmark for evaluating whether AI systems can invent generalizable and scalable ML methods.<n>We study the effects of test-time scaling, adaptive compute allocation, and context provision on agents' discovery performance.
  arXiv  Detail & Related papers  (2026-05-09T04:29:46Z)
• Teaching Machine Learning Fundamentals with LEGO Robotics [6.474217812459663]
  This paper presents the web-based platform Machine Learning with Bricks and an accompanying two-day course designed to teach machine learning concepts to students aged 12 to 17.<n>Machine Learning with Bricks is an open source platform and combines interactive visualizations with LEGO robotics to teach three core algorithms: KNN, linear regression, and Q-learning.<n>Students learn by collecting data, training models, and interacting with robots via a web-based interface.
  arXiv  Detail & Related papers  (2026-01-27T08:59:57Z)
• AtlasKV: Augmenting LLMs with Billion-Scale Knowledge Graphs in 20GB VRAM [60.583864162256525]
  We propose a scalable, effective, and general way to augment large language models with billion-scale knowledge graphs.<n> KG2KV and HiKVP integrate KG triples into LLMs at scale with sub-linear time and memory complexity.<n>It maintains strong knowledge grounding and generalization performance using the LLMs' inherent attention mechanism.
  arXiv  Detail & Related papers  (2025-10-20T15:40:14Z)
• Quantum Machine Learning: A Hands-on Tutorial for Machine Learning Practitioners and Researchers [51.03113410951073]
  This tutorial introduces readers with a background in AI to quantum machine learning (QML)<n>For self-consistency, this tutorial covers foundational principles, representative QML algorithms, their potential applications, and critical aspects such as trainability, generalization, and computational complexity.
  arXiv  Detail & Related papers  (2025-02-03T08:33:44Z)
• Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment [56.44025052765861]
  Large language models (LLMs) have revolutionized Natural Language Processing (NLP), but their size creates computational bottlenecks. We introduce a novel approach to create accurate, sparse foundational versions of performant LLMs. We show a total speedup on CPUs for sparse-quantized LLaMA models of up to 8.6x.
  arXiv  Detail & Related papers  (2024-05-06T16:03:32Z)
• On-device Self-supervised Learning of Visual Perception Tasks aboard Hardware-limited Nano-quadrotors [53.59319391812798]
  Sub-SI50gram nano-drones are gaining momentum in both academia and industry. Their most compelling applications rely on onboard deep learning models for perception. When deployed in unknown environments, these models often underperform due to domain shift. We propose for the first time, on-device learning aboard nano-drones, where the first part of the in-field mission is dedicated to self-supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-03-06T22:04:14Z)
• RedCoast: A Lightweight Tool to Automate Distributed Training of LLMs on Any GPU/TPUs [32.01139974519813]
  We present RedCoast, a tool crafted to automate distributed training and inference for large language models (LLMs) We also propose a mechanism that allows for the customization of diverse ML pipelines through the definition of merely three functions. As a result, Redco implementations exhibit significantly fewer lines of code compared to their official counterparts.
  arXiv  Detail & Related papers  (2023-10-25T04:32:35Z)
• WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct [130.37945867605302]
  We present WizardMath, which enhances the mathematical CoT reasoning abilities of large language models (LLMs) without using external python tools.<n>Remarkably, WizardMath-Mistral 7B surpasses top-tier open-source LLMs by a substantial margin with higher data efficiency.<n>Our preliminary exploration highlights the pivotal role of instruction evolution and process supervision in achieving exceptional math performance.
  arXiv  Detail & Related papers  (2023-08-18T14:23:21Z)
• TensorKrowch: Smooth integration of tensor networks in machine learning [46.0920431279359]
  We introduceKrowch, an open source Python library built on top of PyTorch. Krowch allows users to construct any tensor network, train it, and integrate it as a layer in more intricate deep learning models.
  arXiv  Detail & Related papers  (2023-06-14T15:55:19Z)
• Classification of integers based on residue classes via modern deep learning algorithms [3.6396223542930772]
  We tested multiple deep learning architectures and feature engineering approaches on classifying integers based on their residues when divided by small prime numbers. We also evaluated Automated Machine Learning platforms from Amazon, Google and Microsoft, and found that they failed on this task without appropriately engineered features. In conclusion, feature engineering remains an important task to improve performance and increase interpretability of machine-learning models.
  arXiv  Detail & Related papers  (2023-04-03T19:53:31Z)
• Deep learning applied to computational mechanics: A comprehensive review, state of the art, and the classics [77.34726150561087]
  Recent developments in artificial neural networks, particularly deep learning (DL), are reviewed in detail. Both hybrid and pure machine learning (ML) methods are discussed. History and limitations of AI are recounted and discussed, with particular attention at pointing out misstatements or misconceptions of the classics.
  arXiv  Detail & Related papers  (2022-12-18T02:03:00Z)
• Preprocessors Matter! Realistic Decision-Based Attacks on Machine Learning Systems [56.64374584117259]
  Decision-based attacks construct adversarial examples against a machine learning (ML) model by making only hard-label queries. We develop techniques to (i) reverse-engineer the preprocessor and then (ii) use this extracted information to attack the end-to-end system. Our preprocessors extraction method requires only a few hundred queries, and our preprocessor-aware attacks recover the same efficacy as when attacking the model alone.
  arXiv  Detail & Related papers  (2022-10-07T03:10:34Z)
• An Experimental Evaluation of Machine Learning Training on a Real Processing-in-Memory System [9.429605859159023]
  Training machine learning (ML) algorithms is a computationally intensive process, which is frequently memory-bound. Memory-centric computing systems, with processing-in-memory capabilities, can alleviate this data movement bottleneck. We implement several representative classic ML algorithms on a real-world general-purpose PIM architecture.
  arXiv  Detail & Related papers  (2022-07-16T09:39:53Z)
• Flashlight: Enabling Innovation in Tools for Machine Learning [50.63188263773778]
  We introduce Flashlight, an open-source library built to spur innovation in machine learning tools and systems. We see Flashlight as a tool enabling research that can benefit widely used libraries downstream and bring machine learning and systems researchers closer together.
  arXiv  Detail & Related papers  (2022-01-29T01:03:29Z)
• PyCIL: A Python Toolbox for Class-Incremental Learning [34.32500654158169]
  We propose a Python toolbox that implements several key algorithms for class-incremental learning. The toolbox contains implementations of a number of founding works of CIL such as EWC and iCaRL. It also provides current state-of-the-art algorithms that can be used for conducting novel fundamental research.
  arXiv  Detail & Related papers  (2021-12-23T13:41:24Z)
• PyKale: Knowledge-Aware Machine Learning from Multiple Sources in Python [6.276936701568444]
  Pykale is a Python library for knowledge-aware machine learning on graphs, images, texts, and videos. We formulate new green machine learning guidelines based on standard software engineering practices. We build PyKale on PyTorch and leverage the rich PyTorch ecosystem.
  arXiv  Detail & Related papers  (2021-06-17T18:35:37Z)
• MLPerf Tiny Benchmark [1.1178096184080788]
  We present Tinyerf Tiny, the first industry-standard benchmark suite for ultra-low-power tiny machine learning systems. Tinyerf Tiny measures the accuracy, latency, and energy of machine learning inference to properly evaluate the tradeoffs between systems.
  arXiv  Detail & Related papers  (2021-06-14T17:05:17Z)
• Ten Quick Tips for Deep Learning in Biology [116.78436313026478]
  Machine learning is concerned with the development and applications of algorithms that can recognize patterns in data and use them for predictive modeling. Deep learning has become its own subfield of machine learning. In the context of biological research, deep learning has been increasingly used to derive novel insights from high-dimensional biological data.
  arXiv  Detail & Related papers  (2021-05-29T21:02:44Z)
• MLGO: a Machine Learning Guided Compiler Optimizations Framework [0.0]
  This work is the first full integration of machine learning in a complex compiler pass in a real-world setting. We use two different ML algorithms to train the inlining-for-size model, and achieve up to 7% size reduction. The same model generalizes well to a diversity of real-world targets, as well as to the same set of targets after months of active development.
  arXiv  Detail & Related papers  (2021-01-13T00:02:49Z)
• PeleNet: A Reservoir Computing Framework for Loihi [0.0]
  PeleNet aims to simplify reservoir computing for the neuromorphic hardware Loihi. It provides an automatic and efficient distribution of networks over several cores and chips.
  arXiv  Detail & Related papers  (2020-11-24T19:33:08Z)
• Fast Few-Shot Classification by Few-Iteration Meta-Learning [173.32497326674775]
  We introduce a fast optimization-based meta-learning method for few-shot classification. Our strategy enables important aspects of the base learner objective to be learned during meta-training. We perform a comprehensive experimental analysis, demonstrating the speed and effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-10-01T15:59:31Z)
• Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence [3.1314898234563295]
  Python continues to be the most preferred language for scientific computing, data science, and machine learning. This survey offers insight into the field of machine learning with Python, taking a tour through important topics to identify some of the core hardware and software paradigms that have enabled it.
  arXiv  Detail & Related papers  (2020-02-12T05:20:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.