A Vision for Multisensory Intelligence: Sensing, Science, and Synergy

• URL: http://arxiv.org/abs/2601.04563v3
• Date: Tue, 13 Jan 2026 18:24:14 GMT
• Title: A Vision for Multisensory Intelligence: Sensing, Science, and Synergy
• Authors: Paul Pu Liang,
• Abstract summary: This paper outlines a research vision for multisensory artificial intelligence over the next decade.<n>This new set of technologies can change how humans and AI experience and interact with one another.<n>Research in sensing should extend how AI captures the world in richer ways beyond the digital medium.<n>We present new technical challenges to learn synergy between modalities and between humans and AI.
• Score: 32.03738031386344
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Our experience of the world is multisensory, spanning a synthesis of language, sight, sound, touch, taste, and smell. Yet, artificial intelligence has primarily advanced in digital modalities like text, vision, and audio. This paper outlines a research vision for multisensory artificial intelligence over the next decade. This new set of technologies can change how humans and AI experience and interact with one another, by connecting AI to the human senses and a rich spectrum of signals from physiological and tactile cues on the body, to physical and social signals in homes, cities, and the environment. We outline how this field must advance through three interrelated themes of sensing, science, and synergy. Firstly, research in sensing should extend how AI captures the world in richer ways beyond the digital medium. Secondly, developing a principled science for quantifying multimodal heterogeneity and interactions, developing unified modeling architectures and representations, and understanding cross-modal transfer. Finally, we present new technical challenges to learn synergy between modalities and between humans and AI, covering multisensory integration, alignment, reasoning, generation, generalization, and experience. Accompanying this vision paper are a series of projects, resources, and demos of latest advances from the Multisensory Intelligence group at the MIT Media Lab, see https://mit-mi.github.io/.

Related papers

• Neural Brain: A Neuroscience-inspired Framework for Embodied Agents [78.61382193420914]
  Current AI systems, such as large language models, remain disembodied, unable to physically engage with the world.<n>At the core of this challenge lies the concept of Neural Brain, a central intelligence system designed to drive embodied agents with human-like adaptability.<n>This paper introduces a unified framework for the Neural Brain of embodied agents, addressing two fundamental challenges.
  arXiv  Detail & Related papers  (2025-05-12T15:05:34Z)
• Digitizing Touch with an Artificial Multimodal Fingertip [51.7029315337739]
  Humans and robots both benefit from using touch to perceive and interact with the surrounding environment. Here, we describe several conceptual and technological innovations to improve the digitization of touch. These advances are embodied in an artificial finger-shaped sensor with advanced sensing capabilities.
  arXiv  Detail & Related papers  (2024-11-04T18:38:50Z)
• Artificial General Intelligence (AGI)-Native Wireless Systems: A Journey Beyond 6G [58.440115433585824]
  Building future wireless systems that support services like digital twins (DTs) is challenging to achieve through advances to conventional technologies like meta-surfaces. While artificial intelligence (AI)-native networks promise to overcome some limitations of wireless technologies, developments still rely on AI tools like neural networks. This paper revisits the concept of AI-native wireless systems, equipping them with the common sense necessary to transform them into artificial general intelligence (AGI)-native systems.
  arXiv  Detail & Related papers  (2024-04-29T04:51:05Z)
• On the Emergence of Symmetrical Reality [51.21203247240322]
  We introduce the symmetrical reality framework, which offers a unified representation encompassing various forms of physical-virtual amalgamations. We propose an instance of an AI-driven active assistance service that illustrates the potential applications of symmetrical reality.
  arXiv  Detail & Related papers  (2024-01-26T16:09:39Z)
• When Brain-inspired AI Meets AGI [40.96159978312796]
  We provide a comprehensive overview of brain-inspired AI from the perspective of Artificial General Intelligence. We begin with the current progress in brain-inspired AI and its extensive connection with AGI. We then cover the important characteristics for both human intelligence and AGI.
  arXiv  Detail & Related papers  (2023-03-28T12:46:38Z)
• See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation [49.925499720323806]
  We study how visual, auditory, and tactile perception can jointly help robots to solve complex manipulation tasks. We build a robot system that can see with a camera, hear with a contact microphone, and feel with a vision-based tactile sensor.
  arXiv  Detail & Related papers  (2022-12-07T18:55:53Z)
• Artificial Intelligence for the Metaverse: A Survey [66.57225253532748]
  We first deliver a preliminary of AI, including machine learning algorithms and deep learning architectures, and its role in the metaverse. We then convey a comprehensive investigation of AI-based methods concerning six technical aspects that have potentials for the metaverse. Several AI-aided applications, such as healthcare, manufacturing, smart cities, and gaming, are studied to be deployed in the virtual worlds.
  arXiv  Detail & Related papers  (2022-02-15T03:34:56Z)
• Building Human-like Communicative Intelligence: A Grounded Perspective [1.0152838128195465]
  After making astounding progress in language learning, AI systems seem to approach the ceiling that does not reflect important aspects of human communicative capacities. This paper suggests that the dominant cognitively-inspired AI directions, based on nativist and symbolic paradigms, lack necessary substantiation and concreteness to guide progress in modern AI. I propose a list of concrete, implementable components for building "grounded" linguistic intelligence.
  arXiv  Detail & Related papers  (2022-01-02T01:43:24Z)
• Social Neuro AI: Social Interaction as the "dark matter" of AI [0.0]
  We argue that empirical results from social psychology and social neuroscience along with the framework of dynamics can be of inspiration to the development of more intelligent artificial agents.
  arXiv  Detail & Related papers  (2021-12-31T13:41:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.