Attention-based Neural Bag-of-Features Learning for Sequence Data

• URL: http://arxiv.org/abs/2005.12250v1
• Date: Mon, 25 May 2020 17:51:54 GMT
• Title: Attention-based Neural Bag-of-Features Learning for Sequence Data
• Authors: Dat Thanh Tran, Nikolaos Passalis, Anastasios Tefas, Moncef Gabbouj, Alexandros Iosifidis
• Abstract summary: 2D-Attention (2DA) is a generic attention formulation for sequence data. The proposed attention module is incorporated into the recently proposed Neural Bag of Feature (NBoF) model to enhance its learning capacity. Our empirical analysis shows that the proposed attention formulations can not only improve performances of NBoF models but also make them resilient to noisy data.
• Score: 143.62294358378128
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we propose 2D-Attention (2DA), a generic attention formulation for sequence data, which acts as a complementary computation block that can detect and focus on relevant sources of information for the given learning objective. The proposed attention module is incorporated into the recently proposed Neural Bag of Feature (NBoF) model to enhance its learning capacity. Since 2DA acts as a plug-in layer, injecting it into different computation stages of the NBoF model results in different 2DA-NBoF architectures, each of which possesses a unique interpretation. We conducted extensive experiments in financial forecasting, audio analysis as well as medical diagnosis problems to benchmark the proposed formulations in comparison with existing methods, including the widely used Gated Recurrent Units. Our empirical analysis shows that the proposed attention formulations can not only improve performances of NBoF models but also make them resilient to noisy data.

Related papers

• Feature Attribution with Necessity and Sufficiency via Dual-stage Perturbation Test for Causal Explanation [18.485632810973122]
  We introduce Feature Attribution with Necessity and Sufficiency (FANS) FANS find a neighborhood of the input such that perturbing samples within this neighborhood have a high probability of being Necessity and Sufficiency (PNS) cause for the change in predictions. We demonstrate that FANS outperforms existing attribution methods on six benchmarks.
  arXiv  Detail & Related papers  (2024-02-13T23:25:01Z)
• Score-based Diffusion Models via Stochastic Differential Equations -- a Technical Tutorial [5.217870815854702]
  This article focuses on the score-based diffusion models, with a particular focus on the formulation via differential equations (SDE) After a gentle introduction, we discuss the two pillars in the diffusion modeling -- sampling and score matching. Short proofs are given to illustrate the main idea of the stated results.
  arXiv  Detail & Related papers  (2024-02-12T08:52:35Z)
• Harnessing Diffusion Models for Visual Perception with Meta Prompts [68.78938846041767]
  We propose a simple yet effective scheme to harness a diffusion model for visual perception tasks. We introduce learnable embeddings (meta prompts) to the pre-trained diffusion models to extract proper features for perception. Our approach achieves new performance records in depth estimation tasks on NYU depth V2 and KITTI, and in semantic segmentation task on CityScapes.
  arXiv  Detail & Related papers  (2023-12-22T14:40:55Z)
• Entity-Conditioned Question Generation for Robust Attention Distribution in Neural Information Retrieval [51.53892300802014]
  We show that supervised neural information retrieval models are prone to learning sparse attention patterns over passage tokens. Using a novel targeted synthetic data generation method, we teach neural IR to attend more uniformly and robustly to all entities in a given passage.
  arXiv  Detail & Related papers  (2022-04-24T22:36:48Z)
• Self-Attention Neural Bag-of-Features [103.70855797025689]
  We build on the recently introduced 2D-Attention and reformulate the attention learning methodology. We propose a joint feature-temporal attention mechanism that learns a joint 2D attention mask highlighting relevant information.
  arXiv  Detail & Related papers  (2022-01-26T17:54:14Z)
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach [80.8446673089281]
  We propose a new learning paradigm with graph representation and learning. Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
  arXiv  Detail & Related papers  (2021-10-09T09:02:45Z)
• Learning Neural Causal Models with Active Interventions [83.44636110899742]
  We introduce an active intervention-targeting mechanism which enables a quick identification of the underlying causal structure of the data-generating process. Our method significantly reduces the required number of interactions compared with random intervention targeting. We demonstrate superior performance on multiple benchmarks from simulated to real-world data.
  arXiv  Detail & Related papers  (2021-09-06T13:10:37Z)
• Toward Robust Drug-Target Interaction Prediction via Ensemble Modeling and Transfer Learning [0.0]
  We introduce an ensemble of deep learning models (EnsembleDLM) for robust DTI prediction. EnsembleDLM only uses the sequence information of chemical compounds and proteins, and it aggregates the predictions from multiple deep neural networks. It achieves state-of-the-art performance in Davis and KIBA datasets.
  arXiv  Detail & Related papers  (2021-07-02T04:00:03Z)
• D-LEMA: Deep Learning Ensembles from Multiple Annotations -- Application to Skin Lesion Segmentation [14.266037264648533]
  Leveraging a collection of annotators' opinions for an image is an interesting way of estimating a gold standard. We propose an approach to handle annotators' disagreements when training a deep model.
  arXiv  Detail & Related papers  (2020-12-14T01:51:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.