Teaching Physical Awareness to LLMs through Sounds

• URL: http://arxiv.org/abs/2506.08524v2
• Date: Wed, 11 Jun 2025 05:18:01 GMT
• Title: Teaching Physical Awareness to LLMs through Sounds
• Abstract summary: ACORN is a framework that teaches Large Language Models (LLMs) physical awareness through sound.<n>We build AQA-PHY, a comprehensive Audio Question-Answer dataset, and propose an audio encoder that processes both magnitude and phase information.<n>We demonstrate reasonable results in both simulated and real-world tasks, such as line-of-sight detection, Doppler effect estimation, and Direction-of-Arrival estimation.
• Score: 2.5260091444764554
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have shown remarkable capabilities in text and multimodal processing, yet they fundamentally lack physical awareness--understanding of real-world physical phenomena. In this work, we present ACORN, a framework that teaches LLMs physical awareness through sound, focusing on fundamental physical phenomena like the Doppler effect, multipath effect, and spatial relationships. To overcome data scarcity, ACORN introduce a physics-based simulator combining real-world sound sources with controlled physical channels to generate diverse training data. Using this simulator, we build AQA-PHY, a comprehensive Audio Question-Answer dataset, and propose an audio encoder that processes both magnitude and phase information. By connecting our audio encoder to state-of-the-art LLMs, we demonstrate reasonable results in both simulated and real-world tasks, such as line-of-sight detection, Doppler effect estimation, and Direction-of-Arrival estimation, paving the way for enabling LLMs to understand physical world.

Related papers

• LLM-Based Synthetic Ground Truth Generation for Audio-Based Emotion Classification via In-Context Learning [9.928781897312971]
  We propose a large language model (LLM)-driven, agentic inference workflow for emotion-related synthetic ground truth generation from streaming speech data in VR environments.<n>We use In-Context Learning (ICL) with few-shot demonstrations of paired audio-based samples and their corresponding transcriptions to construct informative and robust in-context prompts.
  arXiv  Detail & Related papers  (2026-06-10T11:08:21Z)
• Exploring Physical Intelligence Emergence via Omni-Modal Architecture and Physical Data Engine [50.62040226184694]
  We present OmniFysics, a compact omni-modal model that unifies understanding across images, audio, video, and text.<n>To inject explicit physical knowledge, we build a physical data engine with two components.<n>Experiments show competitive performance on standard multimodal benchmarks and improved results on physics-oriented evaluations.
  arXiv  Detail & Related papers  (2026-02-05T14:04:51Z)
• DOA Estimation with Lightweight Network on LLM-Aided Simulated Acoustic Scenes [46.0445214387366]
  Direction-of-Arrival (DOA) estimation is critical in spatial audio and acoustic signal processing.<n>We propose LightDOA, a lightweight DOA estimation model based on depthwise separable convolutions.<n> Experimental results show that LightDOA achieves satisfactory accuracy and robustness across various acoustic scenes.
  arXiv  Detail & Related papers  (2025-11-11T09:15:06Z)
• SPUR: A Plug-and-Play Framework for Integrating Spatial Audio Understanding and Reasoning into Large Audio-Language Models [62.14165748145729]
  We introduce SPUR, a lightweight, plug-in approach that equips large audio-speaker models with spatial perception.<n>SPUR consists of: (i) a First-Order Ambisonics (FOA) encoder that maps channels to rotation-aware, listener-centric spatial features, integrated into target LALMs via a multimodal adapter; and (ii) SPUR-Set, a spatial QA dataset combining open-source FOA recordings with controlled simulations, emphasizing relative direction, elevation, distance, and overlap for supervised spatial reasoning.
  arXiv  Detail & Related papers  (2025-11-10T01:29:26Z)
• Resounding Acoustic Fields with Reciprocity [13.126858950459557]
  We introduce Versa, a physics-inspired approach to facilitating acoustic field learning.<n>Our method creates physically valid samples with dense virtual emitter positions by exchanging emitter and listener poses.<n>Results show Versa substantially improve the performance of acoustic field learning on both simulated and real-world datasets.
  arXiv  Detail & Related papers  (2025-10-23T14:30:09Z)
• Uncovering Emergent Physics Representations Learned In-Context by Large Language Models [1.8749305679160366]
  Large language models (LLMs) exhibit impressive in-context learning (ICL) abilities, enabling them to solve wide range of tasks via textual prompts alone.<n>Here, we investigate the ICL ability of LLMs, especially focusing on their ability to reason about physics.<n>Using a dynamics forecasting task in physical systems as a proxy, we evaluate whether LLMs can learn physics in context.
  arXiv  Detail & Related papers  (2025-08-17T17:49:17Z)
• MOSPA: Human Motion Generation Driven by Spatial Audio [56.735282455483954]
  We introduce the first comprehensive Spatial Audio-Driven Human Motion dataset, which contains diverse and high-quality spatial audio and motion data.<n>We develop a simple yet effective diffusion-based generative framework for human MOtion generation driven by SPatial Audio, termed MOSPA.<n>Once trained, MOSPA could generate diverse realistic human motions conditioned on varying spatial audio inputs.
  arXiv  Detail & Related papers  (2025-07-16T06:33:11Z)
• PAL: Probing Audio Encoders via LLMs -- A Study of Information Transfer from Audio Encoders to LLMs [16.820927353576774]
  The integration of audio perception capabilities into Large Language Models (LLMs) has enabled significant advances in Audio-LLMs.<n>We conceptualize effective audio-LLM interaction as the LLM's ability to proficiently probe the audio encoder representations to satisfy textual queries.<n>This paper presents a systematic investigation on how architectural design choices can affect that.
  arXiv  Detail & Related papers  (2025-06-12T07:23:07Z)
• Think Before You Diffuse: LLMs-Guided Physics-Aware Video Generation [28.79821758835663]
  We propose DiffPhy, a generic framework that enables physically-correct and photo-realistic video generation.<n>Our method leverages large language models (LLMs) to explicitly reason a comprehensive physical context from the text prompt.<n>We also establish a high-quality physical video dataset containing diverse phyiscal actions and events to facilitate effective finetuning.
  arXiv  Detail & Related papers  (2025-05-27T18:26:43Z)
• From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data [55.2480439325792]
  Audio-aware large language models (ALLMs) have recently made great strides in understanding and processing audio inputs.<n>These models are typically adapted from text-based large language models (LLMs) through additional training on audio-related tasks.<n>We propose a data generation framework that produces contrastive-like training data, designed to enhance ALLMs' ability to differentiate between present and absent sounds.
  arXiv  Detail & Related papers  (2025-05-26T16:08:41Z)
• Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples [55.2480439325792]
  Recent advancements in audio-aware large language models (ALLMs) enable them to process and understand audio inputs.<n>These models often hallucinate non-existent sound events, reducing their reliability in real-world applications.<n>We propose LISTEN, a contrastive-like training method that enhances ALLMs' ability to distinguish between present and absent sounds.
  arXiv  Detail & Related papers  (2025-05-20T15:44:01Z)
• The Sound of Water: Inferring Physical Properties from Pouring Liquids [85.30865788636386]
  We study the connection between audio-visual observations and the underlying physics of pouring liquids.<n>Our objective is to automatically infer physical properties such as the liquid level, the shape and size of the container, the pouring rate and the time to fill.
  arXiv  Detail & Related papers  (2024-11-18T01:19:37Z)
• LLM4Brain: Training a Large Language Model for Brain Video Understanding [9.294352205183726]
  We introduce an LLM-based approach for reconstructing visual-semantic information from fMRI signals elicited by video stimuli. We employ fine-tuning techniques on an fMRI encoder equipped with adaptors to transform brain responses into latent representations aligned with the video stimuli. In particular, we integrate self-supervised domain adaptation methods to enhance the alignment between visual-semantic information and brain responses.
  arXiv  Detail & Related papers  (2024-09-26T15:57:08Z)
• Large Language Models are Strong Audio-Visual Speech Recognition Learners [53.142635674428874]
  Multimodal large language models (MLLMs) have recently become a focal point of research due to their formidable multimodal understanding capabilities.<n>We propose Llama-AVSR, a new MLLM with strong audio-visual speech recognition capabilities.<n>We evaluate our proposed approach on LRS3, the largest public AVSR benchmark, and we achieve new state-of-the-art results for the tasks of ASR and AVSR with a WER of 0.79% and 0.77%, respectively.
  arXiv  Detail & Related papers  (2024-09-18T21:17:27Z)
• ML-ASPA: A Contemplation of Machine Learning-based Acoustic Signal Processing Analysis for Sounds, & Strains Emerging Technology [0.0]
  This inquiry explores recent advancements and transformative potential within the domain of acoustics, specifically focusing on machine learning (ML) and deep learning. ML adopts a data-driven approach, unveiling intricate relationships between features and desired labels or actions, as well as among features themselves. The application of ML to expansive sets of training data facilitates the discovery of models elucidating complex acoustic phenomena such as human speech and reverberation.
  arXiv  Detail & Related papers  (2023-12-18T03:04:42Z)
• Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos [78.49864987061689]
  Traditional methods of impact sound synthesis use physics simulation to obtain a set of physics parameters that could represent and synthesize the sound. Existing video-driven deep learning-based approaches could only capture the weak correspondence between visual content and impact sounds. We propose a physics-driven diffusion model that can synthesize high-fidelity impact sound for a silent video clip.
  arXiv  Detail & Related papers  (2023-03-29T17:59:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.