Towards Cross-Modal Text-Molecule Retrieval with Better Modality Alignment

• URL: http://arxiv.org/abs/2410.23715v1
• Date: Thu, 31 Oct 2024 08:03:13 GMT
• Title: Towards Cross-Modal Text-Molecule Retrieval with Better Modality Alignment
• Abstract summary: Cross-modal text-molecule retrieval model aims to learn a shared feature space of the text and molecule modalities for accurate similarity calculation. Previous works are inadequate in capturing modality-shared features considering the significant gap between text sequences and molecule graphs. We propose a novel cross-modal text-molecule retrieval model with two-fold improvements.
• Score: 23.86314167123294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cross-modal text-molecule retrieval model aims to learn a shared feature space of the text and molecule modalities for accurate similarity calculation, which facilitates the rapid screening of molecules with specific properties and activities in drug design. However, previous works have two main defects. First, they are inadequate in capturing modality-shared features considering the significant gap between text sequences and molecule graphs. Second, they mainly rely on contrastive learning and adversarial training for cross-modality alignment, both of which mainly focus on the first-order similarity, ignoring the second-order similarity that can capture more structural information in the embedding space. To address these issues, we propose a novel cross-modal text-molecule retrieval model with two-fold improvements. Specifically, on the top of two modality-specific encoders, we stack a memory bank based feature projector that contain learnable memory vectors to extract modality-shared features better. More importantly, during the model training, we calculate four kinds of similarity distributions (text-to-text, text-to-molecule, molecule-to-molecule, and molecule-to-text similarity distributions) for each instance, and then minimize the distance between these similarity distributions (namely second-order similarity losses) to enhance cross-modal alignment. Experimental results and analysis strongly demonstrate the effectiveness of our model. Particularly, our model achieves SOTA performance, outperforming the previously-reported best result by 6.4%.

Related papers

• Retrieval-guided Twin Fusion with Similarity-aware Contrast for Molecule-Text Alignment [16.818672826278327]
  We propose a novel approach named Retrieval-guided Twin Fusion with Similarity-aware Contrast (RISEN) for molecule-text alignment.<n>The core idea of RISEN is to construct a latent twin molecule for each substructure with cross-modal retrieval for semantic enhancement.<n>In particular, for each substructure query, we retrieve relevant textual descriptions and sample several molecules that share similar descriptions of substructures.
  arXiv  Detail & Related papers  (2026-08-17T01:52:54Z)
• Leveraging Text-to-Image Generation for Handling Spurious Correlation [24.940576844328408]
  Deep neural networks trained with Empirical Risk Minimization (ERM) perform well when both training and test data come from the same domain. ERM models may rely on spurious correlations that often exist between labels and irrelevant features of images, making predictions unreliable when those features do not exist. We propose a technique to generate training samples with text-to-image (T2I) diffusion models for addressing the spurious correlation problem.
  arXiv  Detail & Related papers  (2025-03-21T15:28:22Z)
• Cross-Modal Few-Shot Learning with Second-Order Neural Ordinary Differential Equations [26.46540034821343]
  We introduce SONO, a novel method leveraging Second-Order Neural Ordinary Differential Equations (Second-Order NODEs) to enhance cross-modal few-shot learning.<n>Our second-order approach can approximate a broader class of functions, enhancing the model's expressive power and feature generalization capabilities.<n>We utilize text-based image augmentation, exploiting CLIP's robust image-text correlation to enrich training data significantly.
  arXiv  Detail & Related papers  (2024-12-20T11:42:41Z)
• Exploring Optimal Transport-Based Multi-Grained Alignments for Text-Molecule Retrieval [24.061535843472427]
  We introduce the Optimal TRansport-based Multi-grained Alignments model (ORMA) ORMA is a novel approach that facilitates multi-grained alignments between textual descriptions and molecules. Experimental results on the ChEBI-20 and PCdes datasets demonstrate that ORMA significantly outperforms existing state-of-the-art (SOTA) models.
  arXiv  Detail & Related papers  (2024-11-04T06:30:52Z)
• Bridging the Modality Gap: Dimension Information Alignment and Sparse Spatial Constraint for Image-Text Matching [10.709744162565274]
  We propose a novel method called DIAS to bridge the modality gap from two aspects. The method achieves 4.3%-10.2% rSum improvements on Flickr30k and MSCOCO benchmarks.
  arXiv  Detail & Related papers  (2024-10-22T09:37:29Z)
• GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning [51.677086019209554]
  We propose a Generalized Structural Sparse to capture powerful relationships across modalities for pair-wise similarity learning. The distance metric delicately encapsulates two formats of diagonal and block-diagonal terms. Experiments on cross-modal and two extra uni-modal retrieval tasks have validated its superiority and flexibility.
  arXiv  Detail & Related papers  (2024-10-20T03:45:50Z)
• Optimizing OOD Detection in Molecular Graphs: A Novel Approach with Diffusion Models [71.39421638547164]
  We propose to detect OOD molecules by adopting an auxiliary diffusion model-based framework, which compares similarities between input molecules and reconstructed graphs. Due to the generative bias towards reconstructing ID training samples, the similarity scores of OOD molecules will be much lower to facilitate detection. Our research pioneers an approach of Prototypical Graph Reconstruction for Molecular OOD Detection, dubbed as PGR-MOOD and hinges on three innovations.
  arXiv  Detail & Related papers  (2024-04-24T03:25:53Z)
• On the Multi-modal Vulnerability of Diffusion Models [56.08923332178462]
  We propose MMP-Attack to manipulate the generation results of diffusion models by appending a specific suffix to the original prompt. Our goal is to induce diffusion models to generate a specific object while simultaneously eliminating the original object.
  arXiv  Detail & Related papers  (2024-02-02T12:39:49Z)
• Unified Molecular Modeling via Modality Blending [35.16755562674055]
  We introduce a novel "blend-then-predict" self-supervised learning method (MoleBLEND) MoleBLEND blends atom relations from different modalities into one unified relation for matrix encoding, then recovers modality-specific information for both 2D and 3D structures. Experiments show that MoleBLEND achieves state-of-the-art performance across major 2D/3D benchmarks.
  arXiv  Detail & Related papers  (2023-07-12T15:27:06Z)
• BiCro: Noisy Correspondence Rectification for Multi-modality Data via Bi-directional Cross-modal Similarity Consistency [66.8685113725007]
  BiCro aims to estimate soft labels for noisy data pairs to reflect their true correspondence degree. experiments on three popular cross-modal matching datasets demonstrate that BiCro significantly improves the noise-robustness of various matching models.
  arXiv  Detail & Related papers  (2023-03-22T09:33:50Z)
• Cross-modal Contrastive Learning for Multimodal Fake News Detection [10.760000041969139]
  COOLANT is a cross-modal contrastive learning framework for multimodal fake news detection. A cross-modal fusion module is developed to learn the cross-modality correlations. An attention guidance module is implemented to help effectively and interpretably aggregate the aligned unimodal representations.
  arXiv  Detail & Related papers  (2023-02-25T10:12:34Z)
• Disentangled Representation Learning for Text-Video Retrieval [51.861423831566626]
  Cross-modality interaction is a critical component in Text-Video Retrieval (TVR) We study the interaction paradigm in depth, where we find that its computation can be split into two terms. We propose a disentangled framework to capture a sequential and hierarchical representation.
  arXiv  Detail & Related papers  (2022-03-14T13:55:33Z)
• Molecular Attributes Transfer from Non-Parallel Data [57.010952598634944]
  We formulate molecular optimization as a style transfer problem and present a novel generative model that could automatically learn internal differences between two groups of non-parallel data. Experiments on two molecular optimization tasks, toxicity modification and synthesizability improvement, demonstrate that our model significantly outperforms several state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-11-30T06:10:22Z)
• Understanding Neural Abstractive Summarization Models via Uncertainty [54.37665950633147]
  seq2seq abstractive summarization models generate text in a free-form manner. We study the entropy, or uncertainty, of the model's token-level predictions. We show that uncertainty is a useful perspective for analyzing summarization and text generation models more broadly.
  arXiv  Detail & Related papers  (2020-10-15T16:57:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.