IoT-LM: Large Multisensory Language Models for the Internet of Things
- URL: http://arxiv.org/abs/2407.09801v1
- Date: Sat, 13 Jul 2024 08:20:37 GMT
- Title: IoT-LM: Large Multisensory Language Models for the Internet of Things
- Authors: Shentong Mo, Russ Salakhutdinov, Louis-Philippe Morency, Paul Pu Liang,
- Abstract summary: IoT ecosystem provides rich source of real-world modalities such as motion, thermal, geolocation, imaging, depth, sensors, and audio.
Machine learning presents a rich opportunity to automatically process IoT data at scale.
We introduce IoT-LM, an open-source large multisensory language model tailored for the IoT ecosystem.
- Score: 70.74131118309967
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The Internet of Things (IoT) network integrating billions of smart physical devices embedded with sensors, software, and communication technologies is a critical and rapidly expanding component of our modern world. The IoT ecosystem provides a rich source of real-world modalities such as motion, thermal, geolocation, imaging, depth, sensors, and audio to recognize the states of humans and physical objects. Machine learning presents a rich opportunity to automatically process IoT data at scale, enabling efficient inference for understanding human wellbeing, controlling physical devices, and interconnecting smart cities. To realize this potential, we introduce IoT-LM, an open-source large multisensory language model tailored for the IoT ecosystem. IoT-LM is enabled by two technical contributions: the first is MultiIoT, the most expansive unified IoT dataset to date, encompassing over 1.15 million samples from 12 modalities and 8 tasks prepared for multisensory pre-training and instruction-tuning. The second is a new multisensory multitask adapter layer to condition pre-trained large language models on multisensory IoT data. Not only does IoT-LM yield substantial improvements on 8 supervised IoT classification tasks, but it also demonstrates new interactive question-answering, reasoning, and dialog capabilities conditioned on IoT sensors. We release IoT-LM's data sources and new multisensory language modeling framework.
Related papers
- Leveraging Foundation Models for Zero-Shot IoT Sensing [5.319176383069102]
Deep learning models are increasingly deployed on edge Internet of Things (IoT) devices.
ZSL aims to classify data of unseen classes with the help of semantic information.
In this work, we align the IoT data embeddings with the semantic embeddings generated by an FM's text encoder for zero-shot IoT sensing.
arXiv Detail & Related papers (2024-07-29T11:16:48Z) - MultiIoT: Benchmarking Machine Learning for the Internet of Things [70.74131118309967]
The next generation of machine learning systems must be adept at perceiving and interacting with the physical world.
sensory data from motion, thermal, geolocation, depth, wireless signals, video, and audio are increasingly used to model the states of physical environments.
Existing efforts are often specialized to a single sensory modality or prediction task.
This paper proposes MultiIoT, the most expansive and unified IoT benchmark to date, encompassing over 1.15 million samples from 12 modalities and 8 real-world tasks.
arXiv Detail & Related papers (2023-11-10T18:13:08Z) - Towards Artificial General Intelligence (AGI) in the Internet of Things
(IoT): Opportunities and Challenges [55.82853124625841]
Artificial General Intelligence (AGI) possesses the capacity to comprehend, learn, and execute tasks with human cognitive abilities.
This research embarks on an exploration of the opportunities and challenges towards achieving AGI in the context of the Internet of Things.
The application spectrum for AGI-infused IoT is broad, encompassing domains ranging from smart grids, residential environments, manufacturing, and transportation to environmental monitoring, agriculture, healthcare, and education.
arXiv Detail & Related papers (2023-09-14T05:43:36Z) - The Internet of Senses: Building on Semantic Communications and Edge
Intelligence [67.75406096878321]
The Internet of Senses (IoS) holds the promise of flawless telepresence-style communication for all human receptors'
We elaborate on how the emerging semantic communications and Artificial Intelligence (AI)/Machine Learning (ML) paradigms may satisfy the requirements of IoS use cases.
arXiv Detail & Related papers (2022-12-21T03:37:38Z) - Graph Neural Networks in IoT: A Survey [9.257834364029547]
The Internet of Things (IoT) boom has revolutionized almost every corner of people's daily lives.
Deep learning models have been extensively employed in solving IoT tasks.
Graph Neural Networks (GNNs) have been demonstrated to achieve state-of-the-art results in numerous IoT learning tasks.
arXiv Detail & Related papers (2022-03-29T22:27:59Z) - Optimizing Resource-Efficiency for Federated Edge Intelligence in IoT
Networks [96.24723959137218]
We study an edge intelligence-based IoT network in which a set of edge servers learn a shared model using federated learning (FL)
We propose a novel framework, called federated edge intelligence (FEI), that allows edge servers to evaluate the required number of data samples according to the energy cost of the IoT network.
We prove that our proposed algorithm does not cause any data leakage nor disclose any topological information of the IoT network.
arXiv Detail & Related papers (2020-11-25T12:51:59Z) - Empowering Things with Intelligence: A Survey of the Progress,
Challenges, and Opportunities in Artificial Intelligence of Things [98.10037444792444]
We show how AI can empower the IoT to make it faster, smarter, greener, and safer.
First, we present progress in AI research for IoT from four perspectives: perceiving, learning, reasoning, and behaving.
Finally, we summarize some promising applications of AIoT that are likely to profoundly reshape our world.
arXiv Detail & Related papers (2020-11-17T13:14:28Z) - Personalized Federated Learning for Intelligent IoT Applications: A
Cloud-Edge based Framework [12.199870302894439]
Internet of Things (IoT) have widely penetrated in different aspects of modern life.
In this article we advocate a personalized federated learning framework in a cloud-edge architecture for intelligent IoT applications.
arXiv Detail & Related papers (2020-02-25T05:11:06Z) - IoT Behavioral Monitoring via Network Traffic Analysis [0.45687771576879593]
This thesis is the culmination of our efforts to develop techniques to profile the network behavioral pattern of IoTs.
We develop a robust machine learning-based inference engine trained with attributes from traffic patterns.
We demonstrate real-time classification of 28 IoT devices with over 99% accuracy.
arXiv Detail & Related papers (2020-01-28T23:13:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.