Exploring Fusion Strategies for Multimodal Vision-Language Systems

• URL: http://arxiv.org/abs/2511.21889v1
• Date: Wed, 26 Nov 2025 20:12:32 GMT
• Title: Exploring Fusion Strategies for Multimodal Vision-Language Systems
• Authors: Regan Willis, Jason Bakos,
• Abstract summary: We investigate different fusion strategies using a hybrid BERT and vision network framework.<n>We propose three models for each vision network, which fuse data at late, intermediate, and early stages in the architecture.<n>Our experimental results demonstrate that while late fusion yields the highest accuracy, early fusion offers the lowest inference latency.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern machine learning models often combine multiple input streams of data to more accurately capture the information that informs their decisions. In multimodal machine learning, choosing the strategy for fusing data together requires careful consideration of the application's accuracy and latency requirements, as fusing the data at earlier or later stages in the model architecture can lead to performance changes in accuracy and latency. To demonstrate this tradeoff, we investigate different fusion strategies using a hybrid BERT and vision network framework that integrates image and text data. We explore two different vision networks: MobileNetV2 and ViT. We propose three models for each vision network, which fuse data at late, intermediate, and early stages in the architecture. We evaluate the proposed models on the CMU MOSI dataset and benchmark their latency on an NVIDIA Jetson Orin AGX. Our experimental results demonstrate that while late fusion yields the highest accuracy, early fusion offers the lowest inference latency. We describe the three proposed model architectures and discuss the accuracy and latency tradeoffs, concluding that data fusion earlier in the model architecture results in faster inference times at the cost of accuracy.

Related papers

• Adaptive Fuzzy Time Series Forecasting via Partially Asymmetric Convolution and Sub-Sliding Window Fusion [0.0]
  We propose a novel convolutional architecture with partially asymmetric design based on the time of sliding window.<n>The proposed method achieves state-of-the-art results on most of popular time series datasets.
  arXiv  Detail & Related papers  (2025-07-28T08:58:25Z)
• Breaking Silos: Adaptive Model Fusion Unlocks Better Time Series Forecasting [64.45587649141842]
  Time-series forecasting plays a critical role in many real-world applications.<n>No single model consistently outperforms others across different test samples, but instead (ii) each model excels in specific cases.<n>We introduce TimeFuse, a framework for collective time-series forecasting with sample-level adaptive fusion of heterogeneous models.
  arXiv  Detail & Related papers  (2025-05-24T00:45:07Z)
• IncepFormerNet: A multi-scale multi-head attention network for SSVEP classification [12.935583315234553]
  This study proposes a new model called IncepFormerNet, which is a hybrid of the Inception and Transformer architectures.<n>IncepFormerNet adeptly extracts multi-scale temporal information from time series data using parallel convolution kernels of varying sizes.<n>It takes advantage of filter bank techniques to extract features based on the spectral characteristics of SSVEP data.
  arXiv  Detail & Related papers  (2025-02-04T13:04:03Z)
• Tackling Data Heterogeneity in Federated Time Series Forecasting [61.021413959988216]
  Time series forecasting plays a critical role in various real-world applications, including energy consumption prediction, disease transmission monitoring, and weather forecasting. Most existing methods rely on a centralized training paradigm, where large amounts of data are collected from distributed devices to a central cloud server. We propose a novel framework, Fed-TREND, to address data heterogeneity by generating informative synthetic data as auxiliary knowledge carriers.
  arXiv  Detail & Related papers  (2024-11-24T04:56:45Z)
• Efficient Partitioning Vision Transformer on Edge Devices for Distributed Inference [13.533267828812455]
  We propose a novel framework, ED-ViT, which is designed to efficiently split and execute complex Vision Transformers across multiple edge devices.<n>Our approach involves partitioning Vision Transformer models into several sub-models, while each dedicated to handling a specific subset of data classes.<n>We demonstrate that our method significantly cuts down inference latency on edge devices and achieves a reduction in model size by up to 28.9 times and 34.1 times, respectively.
  arXiv  Detail & Related papers  (2024-10-15T14:38:14Z)
• Robust Modality-incomplete Anomaly Detection: A Modality-instructive Framework with Benchmark [69.02666229531322]
  We introduce a pioneering study that investigates Modality-Incomplete Industrial Anomaly Detection (MIIAD)<n>We find that most existing MIAD methods perform poorly on the MIIAD Bench, leading to significant performance degradation.<n>We propose a novel two-stage Robust modAlity-aware fusing and Detecting framewoRk, abbreviated as RADAR.
  arXiv  Detail & Related papers  (2024-10-02T16:47:55Z)
• GOOD: General Optimization-based Fusion for 3D Object Detection via LiDAR-Camera Object Candidates [10.534984939225014]
  3D object detection serves as the core basis of the perception tasks in autonomous driving. Good is a general optimization-based fusion framework that can achieve satisfying detection without training additional models. Experiments on both nuScenes and KITTI datasets are carried out and the results show that GOOD outperforms by 9.1% on mAP score compared with PointPillars.
  arXiv  Detail & Related papers  (2023-03-17T07:05:04Z)
• Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection [58.81316192862618]
  Two critical sensors for 3D perception in autonomous driving are the camera and the LiDAR. fusing these two modalities can significantly boost the performance of 3D perception models. We benchmark the state-of-the-art fusion methods for the first time.
  arXiv  Detail & Related papers  (2022-05-30T09:35:37Z)
• PIETS: Parallelised Irregularity Encoders for Forecasting with Heterogeneous Time-Series [5.911865723926626]
  Heterogeneity and irregularity of multi-source data sets present a significant challenge to time-series analysis. In this work, we design a novel architecture, PIETS, to model heterogeneous time-series. We show that PIETS is able to effectively model heterogeneous temporal data and outperforms other state-of-the-art approaches in the prediction task.
  arXiv  Detail & Related papers  (2021-09-30T20:01:19Z)
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers [115.90778814368703]
  Our objective is language-based search of large-scale image and video datasets. For this task, the approach that consists of independently mapping text and vision to a joint embedding space, a.k.a. dual encoders, is attractive as retrieval scales. An alternative approach of using vision-text transformers with cross-attention gives considerable improvements in accuracy over the joint embeddings.
  arXiv  Detail & Related papers  (2021-03-30T17:57:08Z)
• A Generative Learning Approach for Spatio-temporal Modeling in Connected Vehicular Network [55.852401381113786]
  This paper proposes LaMI (Latency Model Inpainting), a novel framework to generate a comprehensive-temporal quality framework for wireless access latency of connected vehicles. LaMI adopts the idea from image inpainting and synthesizing and can reconstruct the missing latency samples by a two-step procedure. In particular, it first discovers the spatial correlation between samples collected in various regions using a patching-based approach and then feeds the original and highly correlated samples into a Varienational Autocoder (VAE)
  arXiv  Detail & Related papers  (2020-03-16T03:43:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.