HAN: Higher-order Attention Network for Spoken Language Understanding

• URL: http://arxiv.org/abs/2108.11916v1
• Date: Thu, 26 Aug 2021 17:13:08 GMT
• Title: HAN: Higher-order Attention Network for Spoken Language Understanding
• Authors: Dongsheng Chen, Zhiqi Huang, Yuexian Zou
• Abstract summary: We propose to replace the conventional attention with our proposed Bilinear attention block. We conduct wide analysis to explore the effectiveness brought from the higher-order attention.
• Score: 31.326152465734747
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spoken Language Understanding (SLU), including intent detection and slot filling, is a core component in human-computer interaction. The natural attributes of the relationship among the two subtasks make higher requirements on fine-grained feature interaction, i.e., the token-level intent features and slot features. Previous works mainly focus on jointly modeling the relationship between the two subtasks with attention-based models, while ignoring the exploration of attention order. In this paper, we propose to replace the conventional attention with our proposed Bilinear attention block and show that the introduced Higher-order Attention Network (HAN) brings improvement for the SLU task. Importantly, we conduct wide analysis to explore the effectiveness brought from the higher-order attention.

Related papers

• Top-Down Visual Attention from Analysis by Synthesis [87.47527557366593]
  We consider top-down attention from a classic Analysis-by-Synthesis (AbS) perspective of vision. We propose Analysis-by-Synthesis Vision Transformer (AbSViT), which is a top-down modulated ViT model that variationally approximates AbS, and controllable achieves top-down attention.
  arXiv  Detail & Related papers  (2023-03-23T05:17:05Z)
• Guiding Visual Question Answering with Attention Priors [76.21671164766073]
  We propose to guide the attention mechanism using explicit linguistic-visual grounding. This grounding is derived by connecting structured linguistic concepts in the query to their referents among the visual objects. The resultant algorithm is capable of probing attention-based reasoning models, injecting relevant associative knowledge, and regulating the core reasoning process.
  arXiv  Detail & Related papers  (2022-05-25T09:53:47Z)
• On the Efficacy of Co-Attention Transformer Layers in Visual Question Answering [5.547800834335381]
  We investigate the efficacy of co-attention transformer layers in helping the network focus on relevant regions while answering the question. We generate visual attention maps using the question-conditioned image attention scores in these co-attention layers. Our work sheds light on the function and interpretation of co-attention transformer layers, highlights gaps in current networks, and can guide the development of future VQA models.
  arXiv  Detail & Related papers  (2022-01-11T14:25:17Z)
• Alignment Attention by Matching Key and Query Distributions [48.93793773929006]
  This paper introduces alignment attention that explicitly encourages self-attention to match the distributions of the key and query within each head. It is simple to convert any models with self-attention, including pre-trained ones, to the proposed alignment attention. On a variety of language understanding tasks, we show the effectiveness of our method in accuracy, uncertainty estimation, generalization across domains, and robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2021-10-25T00:54:57Z)
• Towards Joint Intent Detection and Slot Filling via Higher-order Attention [47.78365472691051]
  Intent detection (ID) and Slot filling (SF) are two major tasks in spoken language understanding (SLU) We propose a Bilinear attention block, which exploits both the contextual and channel-wise bilinear attention distributions. We show that our approach yields improvements compared with the state-of-the-art approach.
  arXiv  Detail & Related papers  (2021-09-18T09:50:23Z)
• DGA-Net Dynamic Gaussian Attention Network for Sentence Semantic Matching [52.661387170698255]
  We propose a novel Dynamic Gaussian Attention Network (DGA-Net) to improve attention mechanism. We first leverage pre-trained language model to encode the input sentences and construct semantic representations from a global perspective. Finally, we develop a Dynamic Gaussian Attention (DGA) to dynamically capture the important parts and corresponding local contexts from a detailed perspective.
  arXiv  Detail & Related papers  (2021-06-09T08:43:04Z)
• Understanding top-down attention using task-oriented ablation design [0.22940141855172028]
  Top-down attention allows neural networks, both artificial and biological, to focus on the information most relevant for a given task. We aim to answer this with a computational experiment based on a general framework called task-oriented ablation design. We compare the performance of two neural networks, one with top-down attention and one without.
  arXiv  Detail & Related papers  (2021-06-08T21:01:47Z)
• More Than Just Attention: Learning Cross-Modal Attentions with Contrastive Constraints [63.08768589044052]
  We propose Contrastive Content Re-sourcing ( CCR) and Contrastive Content Swapping ( CCS) constraints to address such limitation. CCR and CCS constraints supervise the training of attention models in a contrastive learning manner without requiring explicit attention annotations. Experiments on both Flickr30k and MS-COCO datasets demonstrate that integrating these attention constraints into two state-of-the-art attention-based models improves the model performance.
  arXiv  Detail & Related papers  (2021-05-20T08:48:10Z)
• Attention in Attention Network for Image Super-Resolution [18.2279472158217]
  We quantify and visualize the static attention mechanisms and show that not all attention modules are equally beneficial. We propose attention in attention network (A$2$N) for highly accurate image SR. Our model could achieve superior trade-off performances comparing with state-of-the-art lightweight networks.
  arXiv  Detail & Related papers  (2021-04-19T17:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.