Fine-Grained Scene Image Classification with Modality-Agnostic Adapter

• URL: http://arxiv.org/abs/2407.02769v1
• Date: Wed, 3 Jul 2024 02:57:14 GMT
• Title: Fine-Grained Scene Image Classification with Modality-Agnostic Adapter
• Authors: Yiqun Wang, Zhao Zhou, Xiangcheng Du, Xingjiao Wu, Yingbin Zheng, Cheng Jin,
• Abstract summary: We present a new multi-modal feature fusion approach named MAA (Modality-Agnostic Adapter) We eliminate the modal differences in distribution and then use a modality-agnostic Transformer encoder for a semantic-level feature fusion. Our experiments demonstrate that MAA achieves state-of-the-art results on benchmarks by applying the same modalities with previous methods.
• Score: 8.801601759337006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When dealing with the task of fine-grained scene image classification, most previous works lay much emphasis on global visual features when doing multi-modal feature fusion. In other words, models are deliberately designed based on prior intuitions about the importance of different modalities. In this paper, we present a new multi-modal feature fusion approach named MAA (Modality-Agnostic Adapter), trying to make the model learn the importance of different modalities in different cases adaptively, without giving a prior setting in the model architecture. More specifically, we eliminate the modal differences in distribution and then use a modality-agnostic Transformer encoder for a semantic-level feature fusion. Our experiments demonstrate that MAA achieves state-of-the-art results on benchmarks by applying the same modalities with previous methods. Besides, it is worth mentioning that new modalities can be easily added when using MAA and further boost the performance. Code is available at https://github.com/quniLcs/MAA.

Related papers

• U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation [63.31007867379312]
  We introduce U3M: An Unbiased Multiscale Modal Fusion Model for Multimodal Semantics. We employ feature fusion at multiple scales to ensure the effective extraction and integration of both global and local features. Experimental results demonstrate that our approach achieves superior performance across multiple datasets.
  arXiv  Detail & Related papers  (2024-05-24T08:58:48Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• Multimodal Fusion with Pre-Trained Model Features in Affective Behaviour Analysis In-the-wild [37.32217405723552]
  We present an approach for addressing the task of Expression (Expr) Recognition and Valence-Arousal (VA) Estimation. We evaluate the Aff-Wild2 database using pre-trained models, then extract the final hidden layers of the models as features. Following preprocessing and or convolution to align the extracted features, different models are employed for modal fusion.
  arXiv  Detail & Related papers  (2024-03-22T09:00:24Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion [54.33764537135906]
  VideoQA Transformer models demonstrate competitive performance on standard benchmarks. Do these models capture the rich multimodal structures and dynamics from video and text jointly? Are they achieving high scores by exploiting biases and spurious features?
  arXiv  Detail & Related papers  (2023-06-15T06:45:46Z)
• MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid [40.745848169903105]
  Multi-modal entity alignment (MMEA) aims to discover identical entities across different knowledge graphs. MMEA algorithms rely on KG-level modality fusion strategies for multi-modal entity representation. This paper introduces MEAformer, a multi-modal entity alignment transformer approach for meta modality hybrid.
  arXiv  Detail & Related papers  (2022-12-29T20:49:58Z)
• Multi-Modal Fusion by Meta-Initialization [0.0]
  We propose an extension to the Model-Agnostic Meta-Learning algorithm (MAML) This allows the model to adapt using auxiliary information as well as task experience. FuMI significantly outperforms uni-modal baselines such as MAML in the few-shot regime.
  arXiv  Detail & Related papers  (2022-10-10T17:00:58Z)
• Attention Bottlenecks for Multimodal Fusion [90.75885715478054]
  Machine perception models are typically modality-specific and optimised for unimodal benchmarks. We introduce a novel transformer based architecture that uses fusion' for modality fusion at multiple layers. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks.
  arXiv  Detail & Related papers  (2021-06-30T22:44:12Z)
• Mutual Modality Learning for Video Action Classification [74.83718206963579]
  We show how to embed multi-modality into a single model for video action classification. We achieve state-of-the-art results in the Something-Something-v2 benchmark.
  arXiv  Detail & Related papers  (2020-11-04T21:20:08Z)
• VMLoc: Variational Fusion For Learning-Based Multimodal Camera Localization [46.607930208613574]
  We propose an end-to-end framework, termed VMLoc, to fuse different sensor inputs into a common latent space. Unlike previous multimodal variational works directly adapting the objective function of vanilla variational auto-encoder, we show how camera localization can be accurately estimated.
  arXiv  Detail & Related papers  (2020-03-12T14:52:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.