SIREM: Speech-Informed MRI Reconstruction with Learned Sampling

• URL: http://arxiv.org/abs/2605.18221v1
• Date: Mon, 18 May 2026 11:05:10 GMT
• Title: SIREM: Speech-Informed MRI Reconstruction with Learned Sampling
• Abstract summary: Real-time magnetic resonance imaging (rtMRI) of speech production enables non-invasive visualization of dynamic vocal-tract motion.<n>We propose SIREM, a speech-informed MRI reconstruction framework that uses synchronized speech as a cross-modal prior.<n>We evaluate SIREM on the USC speech rtMRI benchmark against standard baselines, including gridding, wavelet-based compressed sensing, and total variation.
• Score: 5.140545130871631
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-time magnetic resonance imaging (rtMRI) of speech production enables non-invasive visualization of dynamic vocal-tract motion and is valuable for speech science and clinical assessment. However, rtMRI is fundamentally constrained by trade-offs among spatial resolution, temporal resolution, and acquisition speed, often leading to undersampled k-space measurements and degraded reconstructions. We propose SIREM, a speech-informed MRI reconstruction framework that uses synchronized speech as a cross-modal prior. The central idea is that vocal-tract configurations during speech are correlated with the produced acoustics, making part of the image content predictable from audio. SIREM models each frame as a fusion of an audio-driven component and an MRI-driven component through a spatial weighting map. The audio branch predicts articulator-related structure from speech, while the MRI branch reconstructs complementary content from measured k-space data. We further introduce a learnable soft weighting profile over spiral arms, enabling a differentiable study of how k-space arm usage interacts with speech-informed fusion. This yields a unified multimodal formulation that combines audio-driven prediction, MRI reconstruction, and sampling adaptation. We evaluate SIREM on the USC speech rtMRI benchmark against standard baselines, including gridding, wavelet-based compressed sensing, and total variation. SIREM introduces a speech-informed reconstruction paradigm that operates in a substantially higher-throughput regime than iterative methods while preserving anatomically plausible vocal-tract structure. These results establish an initial benchmark for multimodal speech-informed rtMRI reconstruction and highlight the potential of synchronized speech as an auxiliary prior for fast reconstruction. The source code is available at https://github.com/mdhasanai/SIREM

Related papers

• A Speech-to-Video Synthesis Approach Using Spatio-Temporal Diffusion for Vocal Tract MRI [22.92611067883196]
  We introduce an audio-to-video generation framework for creating visuals of the vocal tract from speech signals.<n>Our framework first preprocesses RT-/cine-MRI sequences and speech samples to achieve temporal alignment.<n>We evaluate our framework on healthy controls and tongue cancer patients by analyzing and comparing the vocal tract movements in synthesized videos.
  arXiv  Detail & Related papers  (2025-03-15T12:12:50Z)
• ContextMRI: Enhancing Compressed Sensing MRI through Metadata Conditioning [51.26601171361753]
  We propose ContextMRI, a text-conditioned diffusion model for MRI that integrates granular metadata into the reconstruction process.<n>We show that increasing the fidelity of metadata, ranging from slice location and contrast to patient age, sex, and pathology, systematically boosts reconstruction performance.
  arXiv  Detail & Related papers  (2025-01-08T05:15:43Z)
• MRI2Speech: Speech Synthesis from Articulatory Movements Recorded by Real-time MRI [23.54023878857057]
  We introduce a novel approach that adapts the multi-modal self-supervised AV-HuBERT model for text prediction from rtMRI.<n>The predicted text and durations are then used by a speech decoder to synthesize aligned speech in any novel voice.<n>Our method achieves a $15.18%$ Word Error Rate (WER) on the USC-TIMIT MRI corpus, marking a huge improvement over the current state-of-the-art.
  arXiv  Detail & Related papers  (2024-12-25T08:49:43Z)
• MambaRecon: MRI Reconstruction with Structured State Space Models [30.506544165999564]
  The advent of deep learning has catalyzed the development of cutting-edge methods for the expedited reconstruction of MRI scans. We propose an innovative MRI reconstruction framework that employs structured state space models at its core, aimed at amplifying both long-range contextual sensitivity and reconstruction efficacy.
  arXiv  Detail & Related papers  (2024-09-19T01:50:10Z)
• UNIT-DSR: Dysarthric Speech Reconstruction System Using Speech Unit Normalization [60.43992089087448]
  Dysarthric speech reconstruction systems aim to automatically convert dysarthric speech into normal-sounding speech. We propose a Unit-DSR system, which harnesses the powerful domain-adaptation capacity of HuBERT for training efficiency improvement. Compared with NED approaches, the Unit-DSR system only consists of a speech unit normalizer and a Unit HiFi-GAN vocoder, which is considerably simpler without cascaded sub-modules or auxiliary tasks.
  arXiv  Detail & Related papers  (2024-01-26T06:08:47Z)
• Volumetric Reconstruction Resolves Off-Resonance Artifacts in Static and Dynamic PROPELLER MRI [76.60362295758596]
  Off-resonance artifacts in magnetic resonance imaging (MRI) are visual distortions that occur when the actual resonant frequencies of spins within the imaging volume differ from the expected frequencies used to encode spatial information. We propose to resolve these artifacts by lifting the 2D MRI reconstruction problem to 3D, introducing an additional "spectral" dimension to model this off-resonance.
  arXiv  Detail & Related papers  (2023-11-22T05:44:51Z)
• A Long Short-term Memory Based Recurrent Neural Network for Interventional MRI Reconstruction [50.1787181309337]
  We propose a convolutional long short-term memory (Conv-LSTM) based recurrent neural network (RNN), or ConvLR, to reconstruct interventional images with golden-angle radial sampling. The proposed algorithm has the potential to achieve real-time i-MRI for DBS and can be used for general purpose MR-guided intervention.
  arXiv  Detail & Related papers  (2022-03-28T14:03:45Z)
• ReconFormer: Accelerated MRI Reconstruction Using Recurrent Transformer [60.27951773998535]
  We propose a recurrent transformer model, namely textbfReconFormer, for MRI reconstruction. It can iteratively reconstruct high fertility magnetic resonance images from highly under-sampled k-space data. We show that it achieves significant improvements over the state-of-the-art methods with better parameter efficiency.
  arXiv  Detail & Related papers  (2022-01-23T21:58:19Z)
• Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem [65.25725367771075]
  This study demonstrates, for the first time, that the synthesis-based approach can also perform well on this problem. Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols. By utilizing the synthesis model with the input of discrete symbols, after the prediction of discrete symbol sequence, each target speech could be re-synthesized.
  arXiv  Detail & Related papers  (2021-12-17T08:35:40Z)
• Silent Speech and Emotion Recognition from Vocal Tract Shape Dynamics in Real-Time MRI [9.614694312155798]
  We propose a novel deep neural network-based learning framework that understands acoustic information in the variable-length sequence of vocal tract shaping during speech production. The proposed framework comprises of convolutions, recurrent network, and connectionist temporal classification loss, trained entirely end-to-end. To the best of our knowledge, this is the first study that demonstrates the recognition of entire spoken sentence based on an individual's arttory motions captured by rtMRI video.
  arXiv  Detail & Related papers  (2021-06-16T11:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.