Mind the Gap: A Generalized Approach for Cross-Modal Embedding Alignment

• URL: http://arxiv.org/abs/2410.23437v1
• Date: Wed, 30 Oct 2024 20:28:10 GMT
• Title: Mind the Gap: A Generalized Approach for Cross-Modal Embedding Alignment
• Authors: Arihan Yadav, Alan McMillan,
• Abstract summary: Retrieval-Augmented Generation (RAG) systems retrieve context across different text modalities due to semantic gaps. We introduce a generalized projection-based method, inspired by adapter modules in transfer learning, that efficiently bridges these gaps. Our approach emphasizes speed, accuracy, and data efficiency, requiring minimal resources for training and inference.
• Score: 0.0
• License:
• Abstract: Retrieval-Augmented Generation (RAG) systems enhance text generation by incorporating external knowledge but often struggle when retrieving context across different text modalities due to semantic gaps. We introduce a generalized projection-based method, inspired by adapter modules in transfer learning, that efficiently bridges these gaps between various text types, such as programming code and pseudocode, or English and French sentences. Our approach emphasizes speed, accuracy, and data efficiency, requiring minimal resources for training and inference. By aligning embeddings from heterogeneous text modalities into a unified space through a lightweight projection network, our model significantly outperforms traditional retrieval methods like the Okapi BM25 algorithm and models like Dense Passage Retrieval (DPR), while approaching the accuracy of Sentence Transformers. Extensive evaluations demonstrate the effectiveness and generalizability of our method across different tasks, highlighting its potential for real-time, resource-constrained applications.

Related papers

• Diffusion Features to Bridge Domain Gap for Semantic Segmentation [2.8616666231199424]
  This paper investigates the approach that leverages the sampling and fusion techniques to harness the features of diffusion models efficiently. By leveraging the strength of text-to-image generation capability, we introduce a new training framework designed to implicitly learn posterior knowledge from it.
  arXiv  Detail & Related papers  (2024-06-02T15:33:46Z)
• Text-Video Retrieval with Global-Local Semantic Consistent Learning [122.15339128463715]
  We propose a simple yet effective method, Global-Local Semantic Consistent Learning (GLSCL) GLSCL capitalizes on latent shared semantics across modalities for text-video retrieval. Our method achieves comparable performance with SOTA as well as being nearly 220 times faster in terms of computational cost.
  arXiv  Detail & Related papers  (2024-05-21T11:59:36Z)
• Language Model Decoding as Direct Metrics Optimization [87.68281625776282]
  Current decoding methods struggle to generate texts that align with human texts across different aspects. In this work, we frame decoding from a language model as an optimization problem with the goal of strictly matching the expected performance with human texts. We prove that this induced distribution is guaranteed to improve the perplexity on human texts, which suggests a better approximation to the underlying distribution of human texts.
  arXiv  Detail & Related papers  (2023-10-02T09:35:27Z)
• Decoding Layer Saliency in Language Transformers [0.0]
  In visual networks where saliency is more well-studied, saliency is naturally localized through the convolutional layers of the network. We adapt gradient-based saliency methods for these networks, propose a method for evaluating the degree of semantic coherence of each layer, and demonstrate consistent improvement on multiple benchmark classification datasets.
  arXiv  Detail & Related papers  (2023-08-09T20:53:22Z)
• Text Generation with Efficient (Soft) Q-Learning [91.47743595382758]
  Reinforcement learning (RL) offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. We introduce a new RL formulation for text generation from the soft Q-learning perspective. We apply the approach to a wide range of tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation.
  arXiv  Detail & Related papers  (2021-06-14T18:48:40Z)
• SDA: Improving Text Generation with Self Data Augmentation [88.24594090105899]
  We propose to improve the standard maximum likelihood estimation (MLE) paradigm by incorporating a self-imitation-learning phase for automatic data augmentation. Unlike most existing sentence-level augmentation strategies, our method is more general and could be easily adapted to any MLE-based training procedure.
  arXiv  Detail & Related papers  (2021-01-02T01:15:57Z)
• On Learning Text Style Transfer with Direct Rewards [101.97136885111037]
  Lack of parallel corpora makes it impossible to directly train supervised models for the text style transfer task. We leverage semantic similarity metrics originally used for fine-tuning neural machine translation models. Our model provides significant gains in both automatic and human evaluation over strong baselines.
  arXiv  Detail & Related papers  (2020-10-24T04:30:02Z)
• Collaborative Training of GANs in Continuous and Discrete Spaces for Text Generation [21.435286755934534]
  We propose a novel text GAN architecture that promotes the collaborative training of the continuous-space and discrete-space methods. Our model substantially outperforms state-of-the-art text GANs with respect to quality, diversity, and global consistency.
  arXiv  Detail & Related papers  (2020-10-16T07:51:16Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.