Reducing the Barriers to Entry for Foundation Model Training

• URL: http://arxiv.org/abs/2404.08811v1
• Date: Fri, 12 Apr 2024 20:58:25 GMT
• Title: Reducing the Barriers to Entry for Foundation Model Training
• Authors: Paolo Faraboschi, Ellis Giles, Justin Hotard, Konstanty Owczarek, Andrew Wheeler,
• Abstract summary: The world has recently witnessed an unprecedented acceleration in demands for Machine Learning and Artificial Intelligence applications. This spike in demand has imposed tremendous strain on the underlying technology stack in supply chain. We propose a fundamental change in the AI training infrastructure throughout the technology ecosystem.
• Score: 0.28756346738878485
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The world has recently witnessed an unprecedented acceleration in demands for Machine Learning and Artificial Intelligence applications. This spike in demand has imposed tremendous strain on the underlying technology stack in supply chain, GPU-accelerated hardware, software, datacenter power density, and energy consumption. If left on the current technological trajectory, future demands show insurmountable spending trends, further limiting market players, stifling innovation, and widening the technology gap. To address these challenges, we propose a fundamental change in the AI training infrastructure throughout the technology ecosystem. The changes require advancements in supercomputing and novel AI training approaches, from high-end software to low-level hardware, microprocessor, and chip design, while advancing the energy efficiency required by a sustainable infrastructure. This paper presents the analytical framework that quantitatively highlights the challenges and points to the opportunities to reduce the barriers to entry for training large language models.

Related papers

• AI-Driven Innovations in Modern Cloud Computing [2.3931689873603594]
  This paper explores how AI and cloud computing intersect to deliver transformative capabilities for modernizing applications. Harnessing the combined potential of both AI & Cloud technologies, technology providers can now exploit intelligent resource management, predictive analytics, automated deployment & scaling.
  arXiv  Detail & Related papers  (2024-10-21T12:45:10Z)
• From Cloud to Edge: Rethinking Generative AI for Low-Resource Design Challenges [7.1341189275030645]
  We consider the potential, challenges, and promising approaches for generative AI for design on the edge. The objective is to harness the power of generative AI in creating bespoke solutions for design problems.
  arXiv  Detail & Related papers  (2024-02-20T03:59:27Z)
• Causal Reasoning: Charting a Revolutionary Course for Next-Generation AI-Native Wireless Networks [63.246437631458356]
  Next-generation wireless networks (e.g., 6G) will be artificial intelligence (AI)-native. This article introduces a novel framework for building AI-native wireless networks; grounded in the emerging field of causal reasoning. We highlight several wireless networking challenges that can be addressed by causal discovery and representation.
  arXiv  Detail & Related papers  (2023-09-23T00:05:39Z)
• A new solution and concrete implementation steps for Artificial General Intelligence [4.320142895840622]
  In areas that need to interact with the actual environment, such as elderly care, home nanny, agricultural production, vehicle driving, trial and error are expensive. In this paper, we analyze the limitations of the technical route of large models, and by addressing these limitations, we propose solutions.
  arXiv  Detail & Related papers  (2023-08-12T13:31:02Z)
• AI Maintenance: A Robustness Perspective [91.28724422822003]
  We introduce highlighted robustness challenges in the AI lifecycle and motivate AI maintenance by making analogies to car maintenance. We propose an AI model inspection framework to detect and mitigate robustness risks. Our proposal for AI maintenance facilitates robustness assessment, status tracking, risk scanning, model hardening, and regulation throughout the AI lifecycle.
  arXiv  Detail & Related papers  (2023-01-08T15:02:38Z)
• Vision Paper: Causal Inference for Interpretable and Robust Machine Learning in Mobility Analysis [71.2468615993246]
  Building intelligent transportation systems requires an intricate combination of artificial intelligence and mobility analysis. The past few years have seen rapid development in transportation applications using advanced deep neural networks. This vision paper emphasizes research challenges in deep learning-based mobility analysis that require interpretability and robustness.
  arXiv  Detail & Related papers  (2022-10-18T17:28:58Z)
• Future Computer Systems and Networking Research in the Netherlands: A Manifesto [137.47124933818066]
  We draw attention to CompSys as a vital part of ICT. Each of the Top Sectors of the Dutch Economy, each route in the National Research Agenda, and each of the UN Sustainable Development Goals pose challenges that cannot be addressed without CompSys advances.
  arXiv  Detail & Related papers  (2022-05-26T11:02:29Z)
• INTERN: A New Learning Paradigm Towards General Vision [117.3343347061931]
  We develop a new learning paradigm named INTERN. By learning with supervisory signals from multiple sources in multiple stages, the model being trained will develop strong generalizability. In most cases, our models, adapted with only 10% of the training data in the target domain, outperform the counterparts trained with the full set of data.
  arXiv  Detail & Related papers  (2021-11-16T18:42:50Z)
• Artificial Intelligence at the Edge [25.451110446336276]
  5G mobile communication networks increase communication capacity, reduce transmission latency and error, and save energy. The envisioned future 6G technology will integrate many more technologies, including for example visible light communication. Many applications require computations and analytics close to application end-points: that is, at the edge of the network, rather than in a centralized cloud.
  arXiv  Detail & Related papers  (2020-12-10T02:08:47Z)
• Qlib: An AI-oriented Quantitative Investment Platform [86.8580406876954]
  AI technologies have raised new challenges to the quantitative investment system. Qlib aims to realize the potential, empower the research, and create the value of AI technologies in quantitative investment.
  arXiv  Detail & Related papers  (2020-09-22T12:57:10Z)
• Convergence of Artificial Intelligence and High Performance Computing on NSF-supported Cyberinfrastructure [3.4291439418246177]
  Artificial Intelligence (AI) applications have powered transformational solutions for big data challenges in industry and technology. As AI continues to evolve into a computing paradigm endowed with statistical and mathematical rigor, it has become apparent that single- GPU solutions for training, validation, and testing are no longer sufficient. This realization has been driving the confluence of AI and high performance computing to reduce time-to-insight.
  arXiv  Detail & Related papers  (2020-03-18T18:00:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.