Sense Less, Infer More: Agentic Multimodal Transformers for Edge Medical Intelligence

• URL: http://arxiv.org/abs/2604.10404v1
• Date: Sun, 12 Apr 2026 01:46:38 GMT
• Title: Sense Less, Infer More: Agentic Multimodal Transformers for Edge Medical Intelligence
• Abstract summary: We introduce Adaptive Multimodal Intelligence (AMI), an end-to-end framework that jointly learns when to sense and how to infer.<n>AMI integrates three components: (1) a lightweight Agentic Modality Controller that uses differentiable Gumbel-Sigmoid gating to dynamically select active sensors based on model confidence and task relevance; (2) a Learned Sigma-Delta Sensing module that applies patch-wise Delta-Sigma operations with learnable thresholds to skip temporally redundant samples; and (3) a Foundation-backed Multimodal Prediction Model built on unimodal foundation encoders and a cross-modal transformer with temporal context.
• Score: 11.75125432258758
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Edge-based multimodal medical monitoring requires models that balance diagnostic accuracy with severe energy constraints. Continuous acquisition of ECG, PPG, EMG, and IMU streams rapidly drains wearable batteries, often limiting operation to under 10 hours, while existing systems overlook the high temporal redundancy present in physiological signals. We introduce Adaptive Multimodal Intelligence (AMI), an end-to-end framework that jointly learns when to sense and how to infer. AMI integrates three components: (1) a lightweight Agentic Modality Controller that uses differentiable Gumbel-Sigmoid gating to dynamically select active sensors based on model confidence and task relevance; (2) a Learned Sigma-Delta Sensing module that applies patch-wise Delta-Sigma operations with learnable thresholds to skip temporally redundant samples; and (3) a Foundation-backed Multimodal Prediction Model built on unimodal foundation encoders and a cross-modal transformer with temporal context, enabling robust fusion even under gated or missing inputs. These components are trained jointly via a multi-objective loss combining classification accuracy, sparsity regularization, cross-modal alignment, and predictive coding. AMI is hardware-aware, supporting dynamic computation graphs and masked operations, leading to real energy and latency savings. Across MHEALTH, HMC Sleep, and WESAD datasets, it reduces sensor usage by 48.8% while improving state-of-the-art accuracy by 1.9% on average.

Related papers

• RAMamba-Net: A Reliability-Aware and Mamba-Based Multimodal Fusion Network for Auditory Attention Detection [17.489739821644598]
  Auditory attention decoding (AAD) identifies the attended speaker from physiological signals.<n>EEG is the dominant modality for AAD but provides incomplete evidence in naturalistic audio-visual scenes.<n>We propose RAMamba-Net, a reliability-aware Mamba-based multimodal fusion network for AAD.
  arXiv  Detail & Related papers  (2026-09-10T11:08:15Z)
• S$^3$AM: A Single-Stream SAM with Reliability-Calibrated Frequency Adapter for Multi-modal Salient Object Detection [74.15663758681849]
  Single-stream framework integrates reliability-calibrated frequency adaptation into the adopted SAM backbone for MSOD.<n>Experiments on RGB-D, RGB-T, and RGB-NIR salient object detection benchmarks validate that the proposed framework achieves competitive performance.
  arXiv  Detail & Related papers  (2026-08-18T08:05:00Z)
• A Multi-Scale Temporal Framework with Dynamic Fusion for EEG-Based Emotion Recognition [8.619946906050457]
  Mixed emotions represent a clinically relevant but still underexplored target for automatic emotion recognition.<n>This study introduces a multi-scale temporal framework for EEG-based emotion recognition.
  arXiv  Detail & Related papers  (2026-08-10T03:39:20Z)
• Multimodal Wearable-Based Olfactory-Induced Emotion Recognition in Arousal-Valence Dimensions [15.21606334710936]
  Olfaction acts as a non-intrusive and lightweight pathway that directly engages the brain s affective circuitry and emotional modulation.<n>Current olfactory emotion research has two key limitations.<n>It over-emphasises the valence dimension while neglecting arousal.
  arXiv  Detail & Related papers  (2026-07-24T01:42:27Z)
• AdaSurvMamba: Dynamic Fusion and Semantic Scanning for Multimodal Survival Analysis [57.942670844512016]
  We introduce AdaSurvMamba as a novel adaptive framework for multimodal survival analysis.<n>The framework features a Dual-Scale Importance-Aware Reconstruction (DSIR) module to dynamically modulate cross-modal interaction strength.<n>Experiments across five TCGA cohorts demonstrate consistent gains over existing methods.
  arXiv  Detail & Related papers  (2026-06-28T11:36:31Z)
• Deep Temporal Modeling and Ensemble Fusion for Multimodal Emotion Recognition from Physiological Signals [6.265137019897129]
  We present a comprehensive evaluation of deep learning models such as Long Short-Term Memory (LSTM), Temporal Convolutional Networks (TCN), and Transformer on the WESAD dataset for multimodal affect recognition.<n>We implement a late-fusion ensemble strategy that combines predictions from all three architectures trained on multimodal input.<n>Our results show that Transformer models consistently achieve the highest accuracy in multimodal settings, while TCN models perform best in the wrist-only configuration.
  arXiv  Detail & Related papers  (2026-06-12T23:50:33Z)
• Multi-scale Coarse-to-fine Modeling for Test-time Human Motion Control [51.92884966472683]
  MSCoT is a multi-scale, coarse-to-fine model for test-time human motion synthesis and control.<n> MSCoT discretizes motion into a multi-scale hierarchical representation and predicts the entire token sequence at each temporal scale.
  arXiv  Detail & Related papers  (2026-05-14T15:09:49Z)
• MedMamba: Recasting Mamba for Medical Time Series Classification [6.852021597208217]
  Medical time series, such as electrocardiograms (ECG) and electroencephalograms (EEG), exhibit complex temporal dynamics.<n>We propose MedMamba, a principle-driven multi-scale bidirectional state space architecture tailored for medical time series classification.<n>Our design is guided by three key inductive biases of physiological signals: spatial centralization, multi-timescale temporal composition, and non-causal contextual dependency.
  arXiv  Detail & Related papers  (2026-04-17T01:20:02Z)
• Towards Dexterous Embodied Manipulation via Deep Multi-Sensory Fusion and Sparse Expert Scaling [6.026400295270904]
  DeMUSE is a framework to integrate RGB, depth, and 6-axis force into a unified serialized stream.<n>AdaMN is employed to recalibrate modality-aware features, mitigating representation imbalance.<n>A Joint denoising objective synchronously synthesizes environmental evolution and action sequences to ensure physical consistency.
  arXiv  Detail & Related papers  (2026-02-23T12:12:51Z)
• Time2Vec Transformer for Robust Gesture Recognition from Low-Density sEMG [1.231764991565978]
  This paper presents a novel, data-efficient deep learning framework for myoelectric prosthesis control.<n>Our approach implements a hybrid Transformer optimized for sparse, two-channel surface electromyography (sEMG)<n>The proposed framework offers a robust, cost-effective blueprint for next-generation prosthetic interfaces capable of rapid personalization.
  arXiv  Detail & Related papers  (2026-02-02T09:28:27Z)
• MedSeqFT: Sequential Fine-tuning Foundation Models for 3D Medical Image Segmentation [55.37355146924576]
  MedSeqFT is a sequential fine-tuning framework for medical image analysis.<n>It adapts pre-trained models to new tasks while refining their representational capacity.<n>It consistently outperforms state-of-the-art fine-tuning strategies.
  arXiv  Detail & Related papers  (2025-09-07T15:22:53Z)
• Towards Robust Multimodal Physiological Foundation Models: Handling Arbitrary Missing Modalities [9.785262633953794]
  Physio Omni is a foundation model for multimodal physiological signal analysis.<n>It trains a decoupled multimodal tokenizer, enabling masked signal pre-training.<n>It achieves state-of-the-art performance while maintaining strong robustness to missing modalities.
  arXiv  Detail & Related papers  (2025-04-28T09:00:04Z)
• DA-Flow: Dual Attention Normalizing Flow for Skeleton-based Video Anomaly Detection [52.74152717667157]
  We propose a lightweight module called Dual Attention Module (DAM) for capturing cross-dimension interaction relationships in-temporal skeletal data. It employs the frame attention mechanism to identify the most significant frames and the skeleton attention mechanism to capture broader relationships across fixed partitions with minimal parameters and flops.
  arXiv  Detail & Related papers  (2024-06-05T06:18:03Z)
• Convolutional Monge Mapping Normalization for learning on sleep data [63.22081662149488]
  We propose a new method called Convolutional Monge Mapping Normalization (CMMN) CMMN consists in filtering the signals in order to adapt their power spectrum density (PSD) to a Wasserstein barycenter estimated on training data. Numerical experiments on sleep EEG data show that CMMN leads to significant and consistent performance gains independent from the neural network architecture.
  arXiv  Detail & Related papers  (2023-05-30T08:24:01Z)
• Collaborative Intelligent Reflecting Surface Networks with Multi-Agent Reinforcement Learning [63.83425382922157]
  Intelligent reflecting surface (IRS) is envisioned to be widely applied in future wireless networks. In this paper, we investigate a multi-user communication system assisted by cooperative IRS devices with the capability of energy harvesting.
  arXiv  Detail & Related papers  (2022-03-26T20:37:14Z)
• Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysis [96.46952672172021]
  Bi-Bimodal Fusion Network (BBFN) is a novel end-to-end network that performs fusion on pairwise modality representations. Model takes two bimodal pairs as input due to known information imbalance among modalities.
  arXiv  Detail & Related papers  (2021-07-28T23:33:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.