Hyper-Transformer for Amodal Completion

• URL: http://arxiv.org/abs/2405.19949v1
• Date: Thu, 30 May 2024 11:11:54 GMT
• Title: Hyper-Transformer for Amodal Completion
• Authors: Jianxiong Gao, Xuelin Qian, Longfei Liang, Junwei Han, Yanwei Fu,
• Abstract summary: Amodal object completion is a complex task that involves predicting the invisible parts of an object based on visible segments and background information. We introduce a novel framework named the Hyper-Transformer Amodal Network (H-TAN) This framework utilizes a hyper transformer equipped with a dynamic convolution head to directly learn shape priors and accurately predict amodal masks.
• Score: 82.4118011026855
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Amodal object completion is a complex task that involves predicting the invisible parts of an object based on visible segments and background information. Learning shape priors is crucial for effective amodal completion, but traditional methods often rely on two-stage processes or additional information, leading to inefficiencies and potential error accumulation. To address these shortcomings, we introduce a novel framework named the Hyper-Transformer Amodal Network (H-TAN). This framework utilizes a hyper transformer equipped with a dynamic convolution head to directly learn shape priors and accurately predict amodal masks. Specifically, H-TAN uses a dual-branch structure to extract multi-scale features from both images and masks. The multi-scale features from the image branch guide the hyper transformer in learning shape priors and in generating the weights for dynamic convolution tailored to each instance. The dynamic convolution head then uses the features from the mask branch to predict precise amodal masks. We extensively evaluate our model on three benchmark datasets: KINS, COCOA-cls, and D2SA, where H-TAN demonstrated superior performance compared to existing methods. Additional experiments validate the effectiveness and stability of the novel hyper transformer in our framework.

Related papers

• Multi-layer Learnable Attention Mask for Multimodal Tasks [2.378535917357144]
  Learnable Attention Mask (LAM) strategically designed to globally regulate attention maps and prioritize critical tokens. LAM adeptly captures associations between tokens in BERT-like transformer network. Comprehensive experimental validation on various datasets, such as MADv2, QVHighlights, ImageNet 1K, and MSRVTT.
  arXiv  Detail & Related papers  (2024-06-04T20:28:02Z)
• GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer [44.44603063754173]
  Cross-modal transformers have demonstrated superiority in various vision tasks by effectively integrating different modalities. We propose GeminiFusion, a pixel-wise fusion approach that capitalizes on aligned cross-modal representations. We employ a layer-adaptive noise to adaptively control their interplay on a per-layer basis, thereby achieving a harmonized fusion process.
  arXiv  Detail & Related papers  (2024-06-03T11:24:15Z)
• Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers [58.802352477207094]
  We explore the great potential of a pre-trained vision Transformer (ViT) to bridge the vast distribution gap between two modalities. We propose a mask modeling strategy that randomly masks a specific modality of some tokens to enforce the interaction between tokens from different modalities interacting proactively. Experiments demonstrate that our plug-and-play training augmentation techniques can significantly boost state-of-the-art one-stream and two trackersstream to a large extent in terms of both tracking precision and success rate.
  arXiv  Detail & Related papers  (2023-07-09T08:58:47Z)
• FM-ViT: Flexible Modal Vision Transformers for Face Anti-Spoofing [88.6654909354382]
  We present a pure transformer-based framework, dubbed the Flexible Modal Vision Transformer (FM-ViT) for face anti-spoofing. FM-ViT can flexibly target any single-modal (i.e., RGB) attack scenarios with the help of available multi-modal data. Experiments demonstrate that the single model trained based on FM-ViT can not only flexibly evaluate different modal samples, but also outperforms existing single-modal frameworks by a large margin.
  arXiv  Detail & Related papers  (2023-05-05T04:28:48Z)
• Multimodal Token Fusion for Vision Transformers [54.81107795090239]
  We propose a multimodal token fusion method (TokenFusion) for transformer-based vision tasks. To effectively fuse multiple modalities, TokenFusion dynamically detects uninformative tokens and substitutes these tokens with projected and aggregated inter-modal features. The design of TokenFusion allows the transformer to learn correlations among multimodal features, while the single-modal transformer architecture remains largely intact.
  arXiv  Detail & Related papers  (2022-04-19T07:47:50Z)
• Mask Attention Networks: Rethinking and Strengthen Transformer [70.95528238937861]
  Transformer is an attention-based neural network, which consists of two sublayers, Self-Attention Network (SAN) and Feed-Forward Network (FFN)
  arXiv  Detail & Related papers  (2021-03-25T04:07:44Z)
• Transformers Solve the Limited Receptive Field for Monocular Depth Prediction [82.90445525977904]
  We propose TransDepth, an architecture which benefits from both convolutional neural networks and transformers. This is the first paper which applies transformers into pixel-wise prediction problems involving continuous labels.
  arXiv  Detail & Related papers  (2021-03-22T18:00:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.