Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications

• URL: http://arxiv.org/abs/2406.02601v1
• Date: Sun, 2 Jun 2024 01:13:01 GMT
• Title: Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications
• Authors: David Restrepo, Chenwei Wu, Sebastián Andrés Cajas, Luis Filipe Nakayama, Leo Anthony Celi, Diego M López,
• Abstract summary: We advocate for leveraging vector embeddings to enable flexible and efficient computational methodologies. Our paper investigates the efficiency of using vector embeddings from single-modal foundation models and multi-modal Vision-Language Models. We propose a simple yet effective inference-time method to enhance performance by aligning image-text embeddings.
• Score: 3.2549142515720044
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale multi-modal deep learning models have revolutionized domains such as healthcare, highlighting the importance of computational power. However, in resource-constrained regions like Low and Middle-Income Countries (LMICs), limited access to GPUs and data poses significant challenges, often leaving CPUs as the sole resource. To address this, we advocate for leveraging vector embeddings to enable flexible and efficient computational methodologies, democratizing multimodal deep learning across diverse contexts. Our paper investigates the efficiency and effectiveness of using vector embeddings from single-modal foundation models and multi-modal Vision-Language Models (VLMs) for multimodal deep learning in low-resource environments, particularly in healthcare. Additionally, we propose a simple yet effective inference-time method to enhance performance by aligning image-text embeddings. Comparing these approaches with traditional methods, we assess their impact on computational efficiency and model performance using metrics like accuracy, F1-score, inference time, training time, and memory usage across three medical modalities: BRSET (ophthalmology), HAM10000 (dermatology), and SatelliteBench (public health). Our findings show that embeddings reduce computational demands without compromising model performance. Furthermore, our alignment method improves performance in medical tasks. This research promotes sustainable AI practices by optimizing resources in constrained environments, highlighting the potential of embedding-based approaches for efficient multimodal learning. Vector embeddings democratize multimodal deep learning in LMICs, particularly in healthcare, enhancing AI adaptability in varied use cases.

Related papers

• Feedback-aligned Mixed LLMs for Machine Language-Molecule Translation [11.778576032848482]
  We focus on the task of automated language-molecule translation. We are the first to use state-of-the art (SOTA) human-centric optimisation algorithms in the cross-modal setting. We conduct experiments using only 10% of the available data to mitigate memorisation effects.
  arXiv  Detail & Related papers  (2024-05-22T20:40:53Z)
• Dynamic Self-adaptive Multiscale Distillation from Pre-trained Multimodal Large Model for Efficient Cross-modal Representation Learning [12.00246872965739]
  We propose a novel dynamic self-adaptive multiscale distillation from pre-trained multimodal large model. Our strategy employs a multiscale perspective, enabling the extraction structural knowledge across from the pre-trained multimodal large model. Our methodology streamlines pre-trained multimodal large models using only their output features and original image-level information.
  arXiv  Detail & Related papers  (2024-04-16T18:22:49Z)
• Adaptive Affinity-Based Generalization For MRI Imaging Segmentation Across Resource-Limited Settings [1.5703963908242198]
  This paper introduces a novel relation-based knowledge framework by seamlessly combining adaptive affinity-based and kernel-based distillation. To validate our innovative approach, we conducted experiments on publicly available multi-source prostate MRI data.
  arXiv  Detail & Related papers  (2024-04-03T13:35:51Z)
• Machine Learning Insides OptVerse AI Solver: Design Principles and Applications [74.67495900436728]
  We present a comprehensive study on the integration of machine learning (ML) techniques into Huawei Cloud's OptVerse AI solver. We showcase our methods for generating complex SAT and MILP instances utilizing generative models that mirror multifaceted structures of real-world problem. We detail the incorporation of state-of-the-art parameter tuning algorithms which markedly elevate solver performance.
  arXiv  Detail & Related papers  (2024-01-11T15:02:15Z)
• Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning [53.00683059396803]
  Mask image model (MIM) has been widely used due to its simplicity and effectiveness in recovering original information from masked images. We propose a decision-based MIM that utilizes reinforcement learning (RL) to automatically search for optimal image masking ratio and masking strategy. Our approach has a significant advantage over alternative self-supervised methods on the task of neuron segmentation.
  arXiv  Detail & Related papers  (2023-10-06T10:40:46Z)
• LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark [81.42376626294812]
  We present Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark. Our aim is to establish LAMM as a growing ecosystem for training and evaluating MLLMs. We present a comprehensive dataset and benchmark, which cover a wide range of vision tasks for 2D and 3D vision.
  arXiv  Detail & Related papers  (2023-06-11T14:01:17Z)
• Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques [10.715525749057495]
  Deep learning is pervasive in our daily life, including self-driving cars, virtual assistants, social network services, healthcare services, face recognition, etc. Deep neural networks demand substantial compute resources during training and inference. This article provides a survey on resource-efficient deep learning techniques in terms of model-, arithmetic-, and implementation-level techniques.
  arXiv  Detail & Related papers  (2021-12-30T17:00:06Z)
• Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning [89.31889875864599]
  We propose an efficient model-based reinforcement learning algorithm for learning in multi-agent systems. Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC. We provide a practical parametrization of the core optimization problem.
  arXiv  Detail & Related papers  (2021-07-08T18:01:02Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition [61.51188561808917]
  We propose an adaptive multi-modal learning framework, called AdaMML, that selects on-the-fly the optimal modalities for each segment conditioned on the input for efficient video recognition. We show that our proposed approach yields 35%-55% reduction in computation when compared to the traditional baseline.
  arXiv  Detail & Related papers  (2021-05-11T16:19:07Z)
• Resource-Efficient Neural Networks for Embedded Systems [23.532396005466627]
  We provide an overview of the current state of the art of machine learning techniques. We focus on resource-efficient inference based on deep neural networks (DNNs), the predominant machine learning models of the past decade. We substantiate our discussion with experiments on well-known benchmark data sets using compression techniques.
  arXiv  Detail & Related papers  (2020-01-07T14:17:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.