Training and Serving System of Foundation Models: A Comprehensive Survey
- URL: http://arxiv.org/abs/2401.02643v1
- Date: Fri, 5 Jan 2024 05:27:15 GMT
- Title: Training and Serving System of Foundation Models: A Comprehensive Survey
- Authors: Jiahang Zhou, Yanyu Chen, Zicong Hong, Wuhui Chen, Yue Yu, Tao Zhang,
Hui Wang, Chuanfu Zhang, Zibin Zheng
- Abstract summary: This paper extensively explores the methods employed in training and serving foundation models from various perspectives.
It provides a detailed categorization of these state-of-the-art methods, including finer aspects such as network, computing, and storage.
- Score: 32.0115390377174
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Foundation models (e.g., ChatGPT, DALL-E, PengCheng Mind, PanGu-$\Sigma$)
have demonstrated extraordinary performance in key technological areas, such as
natural language processing and visual recognition, and have become the
mainstream trend of artificial general intelligence. This has led more and more
major technology giants to dedicate significant human and financial resources
to actively develop their foundation model systems, which drives continuous
growth of these models' parameters. As a result, the training and serving of
these models have posed significant challenges, including substantial computing
power, memory consumption, bandwidth demands, etc. Therefore, employing
efficient training and serving strategies becomes particularly crucial. Many
researchers have actively explored and proposed effective methods. So, a
comprehensive survey of them is essential for system developers and
researchers. This paper extensively explores the methods employed in training
and serving foundation models from various perspectives. It provides a detailed
categorization of these state-of-the-art methods, including finer aspects such
as network, computing, and storage. Additionally, the paper summarizes the
challenges and presents a perspective on the future development direction of
foundation model systems. Through comprehensive discussion and analysis, it
hopes to provide a solid theoretical basis and practical guidance for future
research and applications, promoting continuous innovation and development in
foundation model systems.
Related papers
- AI Foundation Models in Remote Sensing: A Survey [6.036426846159163]
This paper provides a comprehensive survey of foundation models in the remote sensing domain.
We categorize these models based on their applications in computer vision and domain-specific tasks.
We highlight emerging trends and the significant advancements achieved by these foundation models.
arXiv Detail & Related papers (2024-08-06T22:39:34Z) - The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources [100.23208165760114]
Foundation model development attracts a rapidly expanding body of contributors, scientists, and applications.
To help shape responsible development practices, we introduce the Foundation Model Development Cheatsheet.
arXiv Detail & Related papers (2024-06-24T15:55:49Z) - A Survey of Latent Factor Models in Recommender Systems [0.0]
This survey systematically reviews latent factor models in recommender systems.
The literature is examined through a structured framework covering learning data, model architecture, learning strategies, and optimization techniques.
arXiv Detail & Related papers (2024-05-28T11:28:59Z) - A Survey of Resource-efficient LLM and Multimodal Foundation Models [22.23967603206849]
Large foundation models, including large language models (LLMs), vision transformers (ViTs), diffusion, and multimodal models, are revolutionizing the entire machine learning lifecycle.
However, the substantial advancements in versatility and performance these models offer come at a significant cost in terms of hardware resources.
This survey delves into the critical importance of such research, examining both algorithmic and systemic aspects.
arXiv Detail & Related papers (2024-01-16T03:35:26Z) - Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models [33.50873478562128]
Large Language Models (LLMs) bring forth challenges in the high consumption of computational, memory, energy, and financial resources.
This survey aims to systematically address these challenges by reviewing a broad spectrum of techniques designed to enhance the resource efficiency of LLMs.
arXiv Detail & Related papers (2024-01-01T01:12:42Z) - A Survey of Reasoning with Foundation Models [235.7288855108172]
Reasoning plays a pivotal role in various real-world settings such as negotiation, medical diagnosis, and criminal investigation.
We introduce seminal foundation models proposed or adaptable for reasoning.
We then delve into the potential future directions behind the emergence of reasoning abilities within foundation models.
arXiv Detail & Related papers (2023-12-17T15:16:13Z) - On Efficient Training of Large-Scale Deep Learning Models: A Literature
Review [90.87691246153612]
The field of deep learning has witnessed significant progress, particularly in computer vision (CV), natural language processing (NLP), and speech.
The use of large-scale models trained on vast amounts of data holds immense promise for practical applications.
With the increasing demands on computational capacity, a comprehensive summarization on acceleration techniques of training deep learning models is still much anticipated.
arXiv Detail & Related papers (2023-04-07T11:13:23Z) - Foundation Models for Decision Making: Problems, Methods, and
Opportunities [124.79381732197649]
Foundation models pretrained on diverse data at scale have demonstrated extraordinary capabilities in a wide range of vision and language tasks.
New paradigms are emerging for training foundation models to interact with other agents and perform long-term reasoning.
Research at the intersection of foundation models and decision making holds tremendous promise for creating powerful new systems.
arXiv Detail & Related papers (2023-03-07T18:44:07Z) - Causal Reasoning Meets Visual Representation Learning: A Prospective
Study [117.08431221482638]
Lack of interpretability, robustness, and out-of-distribution generalization are becoming the challenges of the existing visual models.
Inspired by the strong inference ability of human-level agents, recent years have witnessed great effort in developing causal reasoning paradigms.
This paper aims to provide a comprehensive overview of this emerging field, attract attention, encourage discussions, bring to the forefront the urgency of developing novel causal reasoning methods.
arXiv Detail & Related papers (2022-04-26T02:22:28Z) - On the Opportunities and Risks of Foundation Models [256.61956234436553]
We call these models foundation models to underscore their critically central yet incomplete character.
This report provides a thorough account of the opportunities and risks of foundation models.
To tackle these questions, we believe much of the critical research on foundation models will require deep interdisciplinary collaboration.
arXiv Detail & Related papers (2021-08-16T17:50:08Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.