Exploring Temporal Representation in Neural Processes for Multimodal Action Prediction

• URL: http://arxiv.org/abs/2604.08418v1
• Date: Thu, 09 Apr 2026 16:19:08 GMT
• Title: Exploring Temporal Representation in Neural Processes for Multimodal Action Prediction
• Abstract summary: We study the applicability of Conditional robotics (CNP) to the task of self-supervised action prediction in robotics.<n>We find a good MNS-inspired model in the existing Deep Modality Blending Network (DMBN), able to reconstruct the visuo-motor sensory signal during a partially observed action sequence.<n>After a qualitative quantitative evaluation, we highlight difficulties in generalizing to unseen action sequences, and identify the cause in its inner representation of time.
• Score: 42.72014627111101
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inspired by the human ability to understand and predict others, we study the applicability of Conditional Neural Processes (CNP) to the task of self-supervised multimodal action prediction in robotics. Following recent results regarding the ontogeny of the Mirror Neuron System (MNS), we focus on the preliminary objective of self-actions prediction. We find a good MNS-inspired model in the existing Deep Modality Blending Network (DMBN), able to reconstruct the visuo-motor sensory signal during a partially observed action sequence by leveraging the probabilistic generation of CNP. After a qualitative and quantitative evaluation, we highlight its difficulties in generalizing to unseen action sequences, and identify the cause in its inner representation of time. Therefore, we propose a revised version, termed DMBN-Positional Time Encoding (DMBN-PTE), that facilitates learning a more robust representation of temporal information, and provide preliminary results of its effectiveness in expanding the applicability of the architecture. DMBN-PTE figures as a first step in the development of robotic systems that autonomously learn to forecast actions on longer time scales refining their predictions with incoming observations.

Related papers

• Probabilistic Prediction of Neural Dynamics via Autoregressive Flow Matching [1.8305771342813424]
  We introduce a generative forecasting framework for modeling neural dynamics based on autoregressive flow matching (AFM)<n>We learn the conditional distribution of future neural activity given past neural dynamics and concurrent sensory input, explicitly modeling neural activity as a temporally evolving process.<n>AFM significantly outperforms both a non-autoregressive flow-matching baseline and the official challenge general linear model baseline in predicting short-term parcel-wise blood oxygenation level-dependent (BOLD) activity.
  arXiv  Detail & Related papers  (2026-04-13T08:40:15Z)
• Proportional integral derivative booster for neural networks-based time-series prediction: Case of water demand prediction [4.801953936422696]
  The PID-based method is applied to the predicted value at each time step to bring that value closer to the real value.<n>To prove the applicability of this PID-based booster to other types of periodic time-series prediction problems, it is applied to enhance the accuracy of a neural network model used for multi-step forecasting of hourly energy consumption.
  arXiv  Detail & Related papers  (2025-12-06T09:08:05Z)
• Spiking Neural Networks for Temporal Processing: Status Quo and Future Prospects [41.8742357294068]
  Spiking Neural Networks (SNNs) excel in handling data with high efficiency, owing to their rich neuronal dynamics and sparse activity patterns.<n>Given the recent surge in the development of SNNs, there is an urgent need for a comprehensive evaluation of their temporal processing capabilities.
  arXiv  Detail & Related papers  (2025-02-13T16:17:57Z)
• Efficient and Effective Time-Series Forecasting with Spiking Neural Networks [47.371024581669516]
  Spiking neural networks (SNNs) provide a unique pathway for capturing the intricacies of temporal data. Applying SNNs to time-series forecasting is challenging due to difficulties in effective temporal alignment, complexities in encoding processes, and the absence of standardized guidelines for model selection. We propose a framework for SNNs in time-series forecasting tasks, leveraging the efficiency of spiking neurons in processing temporal information.
  arXiv  Detail & Related papers  (2024-02-02T16:23:50Z)
• Knowledge-Driven Modulation of Neural Networks with Attention Mechanism for Next Activity Prediction [8.552757384215813]
  We present a Symbolic[Neuro] system that leverages background knowledge expressed in terms of a procedural process model to offset the under-sampling in the training data. More specifically, we make predictions using NNs with attention mechanism, an emerging technology in the NN field. The system has been tested on several real-life logs showing an improvement in the performance of the prediction task.
  arXiv  Detail & Related papers  (2023-12-14T12:02:35Z)
• Understanding Self-attention Mechanism via Dynamical System Perspective [58.024376086269015]
  Self-attention mechanism (SAM) is widely used in various fields of artificial intelligence. We show that intrinsic stiffness phenomenon (SP) in the high-precision solution of ordinary differential equations (ODEs) also widely exists in high-performance neural networks (NN) We show that the SAM is also a stiffness-aware step size adaptor that can enhance the model's representational ability to measure intrinsic SP.
  arXiv  Detail & Related papers  (2023-08-19T08:17:41Z)
• A Neural Active Inference Model of Perceptual-Motor Learning [62.39667564455059]
  The active inference framework (AIF) is a promising new computational framework grounded in contemporary neuroscience. In this study, we test the ability for the AIF to capture the role of anticipation in the visual guidance of action in humans. We present a novel formulation of the prior function that maps a multi-dimensional world-state to a uni-dimensional distribution of free-energy.
  arXiv  Detail & Related papers  (2022-11-16T20:00:38Z)
• Multi-head Temporal Attention-Augmented Bilinear Network for Financial time series prediction [77.57991021445959]
  We propose a neural layer based on the ideas of temporal attention and multi-head attention to extend the capability of the underlying neural network. The effectiveness of our approach is validated using large-scale limit-order book market data.
  arXiv  Detail & Related papers  (2022-01-14T14:02:19Z)
• Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks [55.0627904986664]
  Spiking Neural Networks (SNNs) usetemporal spike patterns to represent and transmit information, which is not only biologically realistic but also suitable for ultra-low-power event-driven neuromorphic implementation. This paper investigates the contribution of spike timing dynamics to information encoding, synaptic plasticity and decision making, providing a new perspective to design of future DeepSNNs and neuromorphic hardware systems.
  arXiv  Detail & Related papers  (2020-03-26T11:13:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.