Multimodal Continuous Visual Attention Mechanisms
- URL: http://arxiv.org/abs/2104.03046v1
- Date: Wed, 7 Apr 2021 10:47:51 GMT
- Title: Multimodal Continuous Visual Attention Mechanisms
- Authors: Ant\'onio Farinhas, Andr\'e F. T. Martins, Pedro M. Q. Aguiar
- Abstract summary: We introduce a new continuous attention mechanism that produces multimodal densities in the form of mixtures of Gaussians.
Our densities decompose as a linear combination of unimodal attention mechanisms, enabling closed-form Jacobians for the backpropagation step.
- Score: 3.222802562733787
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Visual attention mechanisms are a key component of neural network models for
computer vision. By focusing on a discrete set of objects or image regions,
these mechanisms identify the most relevant features and use them to build more
powerful representations. Recently, continuous-domain alternatives to discrete
attention models have been proposed, which exploit the continuity of images.
These approaches model attention as simple unimodal densities (e.g. a
Gaussian), making them less suitable to deal with images whose region of
interest has a complex shape or is composed of multiple non-contiguous patches.
In this paper, we introduce a new continuous attention mechanism that produces
multimodal densities, in the form of mixtures of Gaussians. We use the EM
algorithm to obtain a clustering of relevant regions in the image, and a
description length penalty to select the number of components in the mixture.
Our densities decompose as a linear combination of unimodal attention
mechanisms, enabling closed-form Jacobians for the backpropagation step.
Experiments on visual question answering in the VQA-v2 dataset show competitive
accuracies and a selection of regions that mimics human attention more closely
in VQA-HAT. We present several examples that suggest how multimodal attention
maps are naturally more interpretable than their unimodal counterparts, showing
the ability of our model to automatically segregate objects from ground in
complex scenes.
Related papers
- Multi-view Aggregation Network for Dichotomous Image Segmentation [76.75904424539543]
Dichotomous Image (DIS) has recently emerged towards high-precision object segmentation from high-resolution natural images.
Existing methods rely on tedious multiple encoder-decoder streams and stages to gradually complete the global localization and local refinement.
Inspired by it, we model DIS as a multi-view object perception problem and provide a parsimonious multi-view aggregation network (MVANet)
Experiments on the popular DIS-5K dataset show that our MVANet significantly outperforms state-of-the-art methods in both accuracy and speed.
arXiv Detail & Related papers (2024-04-11T03:00:00Z) - Fusion of Infrared and Visible Images based on Spatial-Channel
Attentional Mechanism [3.388001684915793]
We present AMFusionNet, an innovative approach to infrared and visible image fusion (IVIF)
By assimilating thermal details from infrared images with texture features from visible sources, our method produces images enriched with comprehensive information.
Our method outperforms state-of-the-art algorithms in terms of quality and quantity.
arXiv Detail & Related papers (2023-08-25T21:05:11Z) - Multi-Spectral Image Stitching via Spatial Graph Reasoning [52.27796682972484]
We propose a spatial graph reasoning based multi-spectral image stitching method.
We embed multi-scale complementary features from the same view position into a set of nodes.
By introducing long-range coherence along spatial and channel dimensions, the complementarity of pixel relations and channel interdependencies aids in the reconstruction of aligned multi-view features.
arXiv Detail & Related papers (2023-07-31T15:04:52Z) - Learning to Fuse Monocular and Multi-view Cues for Multi-frame Depth
Estimation in Dynamic Scenes [51.20150148066458]
We propose a novel method to learn to fuse the multi-view and monocular cues encoded as volumes without needing the generalizationally crafted masks.
Experiments on real-world datasets prove the significant effectiveness and ability of the proposed method.
arXiv Detail & Related papers (2023-04-18T13:55:24Z) - Exploring Global Diversity and Local Context for Video Summarization [4.452227592307381]
Video summarization aims to automatically generate a diverse and concise summary which is useful in large-scale video processing.
Most of methods tend to adopt self attention mechanism across video frames, which fails to model the diversity of video frames.
We propose global diverse attention by using the squared Euclidean distance instead to compute the affinities.
arXiv Detail & Related papers (2022-01-27T06:56:01Z) - An attention-driven hierarchical multi-scale representation for visual
recognition [3.3302293148249125]
Convolutional Neural Networks (CNNs) have revolutionized the understanding of visual content.
We propose a method to capture high-level long-range dependencies by exploring Graph Convolutional Networks (GCNs)
Our approach is simple yet extremely effective in solving both the fine-grained and generic visual classification problems.
arXiv Detail & Related papers (2021-10-23T09:22:22Z) - Bayesian Attention Belief Networks [59.183311769616466]
Attention-based neural networks have achieved state-of-the-art results on a wide range of tasks.
This paper introduces Bayesian attention belief networks, which construct a decoder network by modeling unnormalized attention weights.
We show that our method outperforms deterministic attention and state-of-the-art attention in accuracy, uncertainty estimation, generalization across domains, and adversarial attacks.
arXiv Detail & Related papers (2021-06-09T17:46:22Z) - Improve the Interpretability of Attention: A Fast, Accurate, and
Interpretable High-Resolution Attention Model [6.906621279967867]
We propose a novel Bilinear Representative Non-Parametric Attention (BR-NPA) strategy that captures the task-relevant human-interpretable information.
The proposed model can be easily adapted in a wide variety of modern deep models, where classification is involved.
It is also more accurate, faster, and with a smaller memory footprint than usual neural attention modules.
arXiv Detail & Related papers (2021-06-04T15:57:37Z) - Multimodal Face Synthesis from Visual Attributes [85.87796260802223]
We propose a novel generative adversarial network that simultaneously synthesizes identity preserving multimodal face images.
multimodal stretch-in modules are introduced in the discriminator which discriminates between real and fake images.
arXiv Detail & Related papers (2021-04-09T13:47:23Z) - Spatial-spectral Hyperspectral Image Classification via Multiple Random
Anchor Graphs Ensemble Learning [88.60285937702304]
This paper proposes a novel spatial-spectral HSI classification method via multiple random anchor graphs ensemble learning (RAGE)
Firstly, the local binary pattern is adopted to extract the more descriptive features on each selected band, which preserves local structures and subtle changes of a region.
Secondly, the adaptive neighbors assignment is introduced in the construction of anchor graph, to reduce the computational complexity.
arXiv Detail & Related papers (2021-03-25T09:31:41Z) - Attention-based Image Upsampling [14.676228848773157]
We show how attention mechanisms can be used to replace another canonical operation: strided transposed convolution.
We show that attention-based upsampling consistently outperforms traditional upsampling methods.
arXiv Detail & Related papers (2020-12-17T19:58:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.