Attention-based Neural Bag-of-Features Learning for Sequence Data

• URL: http://arxiv.org/abs/2005.12250v1
• Date: Mon, 25 May 2020 17:51:54 GMT
• Title: Attention-based Neural Bag-of-Features Learning for Sequence Data
• Authors: Dat Thanh Tran, Nikolaos Passalis, Anastasios Tefas, Moncef Gabbouj, Alexandros Iosifidis
• Abstract summary: 2D-Attention (2DA) is a generic attention formulation for sequence data. The proposed attention module is incorporated into the recently proposed Neural Bag of Feature (NBoF) model to enhance its learning capacity. Our empirical analysis shows that the proposed attention formulations can not only improve performances of NBoF models but also make them resilient to noisy data.
• Score: 143.62294358378128
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose 2D-Attention (2DA), a generic attention formulation for sequence data, which acts as a complementary computation block that can detect and focus on relevant sources of information for the given learning objective. The proposed attention module is incorporated into the recently proposed Neural Bag of Feature (NBoF) model to enhance its learning capacity. Since 2DA acts as a plug-in layer, injecting it into different computation stages of the NBoF model results in different 2DA-NBoF architectures, each of which possesses a unique interpretation. We conducted extensive experiments in financial forecasting, audio analysis as well as medical diagnosis problems to benchmark the proposed formulations in comparison with existing methods, including the widely used Gated Recurrent Units. Our empirical analysis shows that the proposed attention formulations can not only improve performances of NBoF models but also make them resilient to noisy data.

Related papers

• Dimension-free Score Matching and Time Bootstrapping for Diffusion Models [11.743167854433306]
  Diffusion models generate samples by estimating the score function of the target distribution at various noise levels. In this work, we establish the first (nearly) dimension-free sample bounds complexity for learning these score functions. A key aspect of our analysis is the use of a single function approximator to jointly estimate scores across noise levels.
  arXiv  Detail & Related papers  (2025-02-14T18:32:22Z)
• Enhancing and Exploring Mild Cognitive Impairment Detection with W2V-BERT-2.0 [1.3988930016464454]
  This study explores a multi-lingual audio self-supervised learning model for detecting mild cognitive impairment (MCI) using the TAUKADIAL cross-lingual dataset. To address these issues, the study utilizes features directly from speech utterances with W2V-BERT-2.0. The experiment shows competitive results, and the proposed inference logic significantly contributes to the improvements from the baseline.
  arXiv  Detail & Related papers  (2025-01-27T16:55:38Z)
• Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorial [5.217870815854702]
  This article focuses on the score-based diffusion models, with a particular focus on the formulation via differential equations (SDE) After a gentle introduction, we discuss the two pillars in the diffusion modeling -- sampling and score matching. Short proofs are given to illustrate the main idea of the stated results.
  arXiv  Detail & Related papers  (2024-02-12T08:52:35Z)
• Harnessing Diffusion Models for Visual Perception with Meta Prompts [68.78938846041767]
  We propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. We introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes.
  arXiv  Detail & Related papers  (2023-12-22T14:40:55Z)
• Entity-Conditioned Question Generation for Robust Attention Distribution in Neural Information Retrieval [51.53892300802014]
  We show that supervised neural information retrieval models are prone to learning sparse attention patterns over passage tokens. Using a novel targeted synthetic data generation method, we teach neural IR to attend more uniformly and robustly to all entities in a given passage.
  arXiv  Detail & Related papers  (2022-04-24T22:36:48Z)
• Self-Attention Neural Bag-of-Features [103.70855797025689]
  We build on the recently introduced 2D-Attention and reformulate the attention learning methodology. We propose a joint feature-temporal attention mechanism that learns a joint 2D attention mask highlighting relevant information.
  arXiv  Detail & Related papers  (2022-01-26T17:54:14Z)
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach [80.8446673089281]
  We propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
  arXiv  Detail & Related papers  (2021-10-09T09:02:45Z)
• Learning Neural Causal Models with Active Interventions [83.44636110899742]
  We introduce an active intervention-targeting mechanism which enables a quick identification of the underlying causal structure of the data-generating process. Our method significantly reduces the required number of interactions compared with random intervention targeting. We demonstrate superior performance on multiple benchmarks from simulated to real-world data.
  arXiv  Detail & Related papers  (2021-09-06T13:10:37Z)
• Toward Robust Drug-Target Interaction Prediction via Ensemble Modeling and Transfer Learning [0.0]
  We introduce an ensemble of deep learning models (EnsembleDLM) for robust DTI prediction. EnsembleDLM only uses the sequence information of chemical compounds and proteins, and it aggregates the predictions from multiple deep neural networks. It achieves state-of-the-art performance in Davis and KIBA datasets.
  arXiv  Detail & Related papers  (2021-07-02T04:00:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.