EigeNet: Geometry-Informed Multi-Modal Learning for Few-shot Novel View RIR Prediction

• URL: http://arxiv.org/abs/2605.28101v1
• Date: Wed, 27 May 2026 07:54:47 GMT
• Title: EigeNet: Geometry-Informed Multi-Modal Learning for Few-shot Novel View RIR Prediction
• Abstract summary: We present EIGENET, a geometry-informed multi-modal framework for few-shot novel view RIR prediction.<n>We empirically demonstrate that this architecture is capable of making full use of the multi-view multi-modal context.<n>EigeNET achieves both state-of-the-art performance in few-shot novel view RIR prediction and sim-to-real generalization.
• Score: 5.156786627043761
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Predicting spatially varying Room Impulse Response (RIR) from sparse observations is a critical but highly challenging inverse problem for immersive spatial audio rendering. In this work, we present EIGENET, a geometry-informed multi-modal framework for few-shot novel view RIR prediction. At its core is a Cross-view Alternate-attention Transformer that iteratively refines local intra-view acoustic structures and global cross-view spatial relationships. We empirically demonstrate that this architecture is capable of making full use of the multi-view multi-modal context while performing spatial-temporal reasoning for RIR prediction. Inspired by acoustic ray tracing, we design a geometry-informed modulation block to formulate the connection between geometric features and RIR power spectrum. In the mean time, an auxiliary loss is introduced to transform the single-target waveform prediction into a multi-task learning framework. Through ablation studies, we demonstrate that this design yields consistent performance gains regardless of the underlying backbone, thereby confirming its foundational utility and architecture-agnostic generalizability for RIR prediction task. Evaluated on both simulated and real-world benchmarks, EIGENET achieves both state-of-the-art performance in few-shot novel view RIR prediction and sim-to-real generalization. Codes and checkpoints are available on https://github.com/FEAfeatherTHER/EigeNet.

Related papers

• A Geometric Algebra-informed NeRF Framework for Generalizable Wireless Channel Prediction [28.17627433640575]
  We propose the geometric algebra-informed neural radiance fields (GAI-NeRF) for wireless channel prediction.<n>Our approach incorporates global token representations, drawing inspiration from transformer architectures in language and vision domains.<n> Experimental results demonstrate that GAI-NeRF achieves superior performance in channel prediction tasks.
  arXiv  Detail & Related papers  (2026-04-13T19:20:29Z)
• StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Models [98.72926158261937]
  We propose a training-free token pruning framework for Visual AutoRegressive models.<n>We employ a lightweight high-pass filter to capture local texture details, while leveraging Principal Component Analysis (PCA) to preserve global structural information.<n>To maintain valid next-scale prediction under sparse tokens, we introduce a nearest neighbor feature propagation strategy.
  arXiv  Detail & Related papers  (2026-03-02T11:35:05Z)
• RIR-Former: Coordinate-Guided Transformer for Continuous Reconstruction of Room Impulse Responses [21.84404827658177]
  RIR-Former is a grid-free, one-step feed-forward model for RIR reconstruction.<n>By introducing a sinusoidal encoding module into a transformer backbone, our method effectively incorporates microphone position information.<n> Experiments on diverse simulated acoustic environments demonstrate that RIR-Former consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-02-02T09:33:54Z)
• Explicit Context-Driven Neural Acoustic Modeling for High-Fidelity RIR Generation [17.013738637228553]
  We present Mesh-infused Neural Acoustic Field (MiNAF), which queries a rough room mesh at given locations and extracts distance distributions as an explicit representation of local context.<n>Our approach demonstrates that incorporating explicit local geometric features can better guide the neural network in generating more accurate RIR predictions.
  arXiv  Detail & Related papers  (2025-09-18T17:57:07Z)
• Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation [63.15257949821558]
  Referring Remote Sensing Image (RRSIS) is a new challenge that combines computer vision and natural language processing. Traditional Referring Image (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery. We introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS.
  arXiv  Detail & Related papers  (2023-12-19T08:14:14Z)
• MuRF: Multi-Baseline Radiance Fields [117.55811938988256]
  We present Multi-Baseline Radiance Fields (MuRF), a feed-forward approach to solving sparse view synthesis. MuRF achieves state-of-the-art performance across multiple different baseline settings. We also show promising zero-shot generalization abilities on the Mip-NeRF 360 dataset.
  arXiv  Detail & Related papers  (2023-12-07T18:59:56Z)
• AV-RIR: Audio-Visual Room Impulse Response Estimation [49.469389715876915]
  Accurate estimation of Room Impulse Response (RIR) is important for speech processing and AR/VR applications. We propose AV-RIR, a novel multi-modal multi-task learning approach to accurately estimate the RIR from a given reverberant speech signal and visual cues of its corresponding environment.
  arXiv  Detail & Related papers  (2023-11-30T22:58:30Z)
• Few-Shot Audio-Visual Learning of Environment Acoustics [89.16560042178523]
  Room impulse response (RIR) functions capture how the surrounding physical environment transforms the sounds heard by a listener. We explore how to infer RIRs based on a sparse set of images and echoes observed in the space. In experiments using a state-of-the-art audio-visual simulator for 3D environments, we demonstrate that our method successfully generates arbitrary RIRs.
  arXiv  Detail & Related papers  (2022-06-08T16:38:24Z)
• PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning [109.84770951839289]
  We present PredRNN, a new recurrent network for learning visual dynamics from historical context. We show that our approach obtains highly competitive results on three standard datasets.
  arXiv  Detail & Related papers  (2021-03-17T08:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.