BarcodeBERT: Transformers for Biodiversity Analysis

• URL: http://arxiv.org/abs/2311.02401v2
• Date: Wed, 22 Jan 2025 00:06:31 GMT
• Title: BarcodeBERT: Transformers for Biodiversity Analysis
• Authors: Pablo Millan Arias, Niousha Sadjadi, Monireh Safari, ZeMing Gong, Austin T. Wang, Joakim Bruslund Haurum, Iuliia Zarubiieva, Dirk Steinke, Lila Kari, Angel X. Chang, Scott C. Lowe, Graham W. Taylor,
• Abstract summary: We introduce BarcodeBERT, a family of models tailored to biodiversity analysis. BarcodeBERT is trained exclusively on data from a reference library of 1.5M invertebrate DNA barcodes.
• Score: 18.582770076266737
• License:
• Abstract: In the global challenge of understanding and characterizing biodiversity, short species-specific genomic sequences known as DNA barcodes play a critical role, enabling fine-grained comparisons among organisms within the same kingdom of life. Although machine learning algorithms specifically designed for the analysis of DNA barcodes are becoming more popular, most existing methodologies rely on generic supervised training algorithms. We introduce BarcodeBERT, a family of models tailored to biodiversity analysis and trained exclusively on data from a reference library of 1.5M invertebrate DNA barcodes. We compared the performance of BarcodeBERT on taxonomic identification tasks against a spectrum of machine learning approaches including supervised training of classical neural architectures and fine-tuning of general DNA foundation models. Our self-supervised pretraining strategies on domain-specific data outperform fine-tuned foundation models, especially in identification tasks involving lower taxa such as genera and species. We also compared BarcodeBERT with BLAST, one of the most widely used bioinformatics tools for sequence searching, and found that our method matched BLAST's performance in species-level classification while being 55 times faster. Our analysis of masking and tokenization strategies also provides practical guidance for building customized DNA language models, emphasizing the importance of aligning model training strategies with dataset characteristics and domain knowledge. The code repository is available at https://github.com/bioscan-ml/BarcodeBERT.

Related papers

• Life-Code: Central Dogma Modeling with Multi-Omics Sequence Unification [53.488387420073536]
  Life-Code is a comprehensive framework that spans different biological functions. Life-Code achieves state-of-the-art performance on various tasks across three omics.
  arXiv  Detail & Related papers  (2025-02-11T06:53:59Z)
• BarcodeMamba: State Space Models for Biodiversity Analysis [14.524535359259414]
  BarcodeMamba is a performant and efficient foundation model for DNA barcodes in biodiversity analysis. Our study shows that BarcodeMamba has better performance than BarcodeBERT even when using only 8.3% as many parameters. In our scaling study, BarcodeMamba with 63.6% of BarcodeBERT's parameters achieved 70.2% genus-level accuracy in 1-nearest neighbor (1-NN) probing for unseen species.
  arXiv  Detail & Related papers  (2024-12-15T06:52:18Z)
• BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity [19.003642885871546]
  BIOSCAN-5M is a comprehensive dataset containing multi-modal information for over 5 million insect specimens. We propose three benchmark experiments to demonstrate the impact of the multi-modal data types on the classification and clustering accuracy.
  arXiv  Detail & Related papers  (2024-06-18T15:45:21Z)
• BEACON: Benchmark for Comprehensive RNA Tasks and Language Models [60.02663015002029]
  We introduce the first comprehensive RNA benchmark BEACON (textbfBEnchmtextbfArk for textbfCOmprehensive RtextbfNA Task and Language Models). First, BEACON comprises 13 distinct tasks derived from extensive previous work covering structural analysis, functional studies, and engineering applications. Second, we examine a range of models, including traditional approaches like CNNs, as well as advanced RNA foundation models based on language models, offering valuable insights into the task-specific performances of these models. Third, we investigate the vital RNA language model components
  arXiv  Detail & Related papers  (2024-06-14T19:39:19Z)
• BEND: Benchmarking DNA Language Models on biologically meaningful tasks [7.005668635562045]
  We introduce BEND, a Benchmark for DNA language models, featuring a collection of realistic and biologically meaningful downstream tasks. We find that embeddings from current DNA LMs can approach performance of expert methods on some tasks, but only capture limited information about long-range features.
  arXiv  Detail & Related papers  (2023-11-21T12:34:00Z)
• DNAGPT: A Generalized Pre-trained Tool for Versatile DNA Sequence Analysis Tasks [14.931476374660944]
  DNAGPT is a generalized DNA pre-training model trained on over 200 billion base pairs from all mammals. By enhancing the classic GPT model with a binary classification task, a numerical regression task, and a comprehensive token language, DNAGPT can handle versatile DNA analysis tasks.
  arXiv  Detail & Related papers  (2023-07-11T06:30:43Z)
• DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome [10.051595222470304]
  We argue that the computation and sample inefficiencies introduced by k-mer tokenization are primary obstacles in developing large genome foundational models. We provide conceptual and empirical insights into genome tokenization, building on which we propose to replace k-mer tokenization with Byte Pair$. We introduce DNABERT-2, a refined genome foundation model that adapts an efficient tokenizer and employs multiple strategies to overcome input length constraints.
  arXiv  Detail & Related papers  (2023-06-26T18:43:46Z)
• A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks [65.34977803841007]
  Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. We show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one.
  arXiv  Detail & Related papers  (2022-11-16T00:11:04Z)
• Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types [75.65676405302105]
  We propose a simple yet effective approach for pre-training genome data in a multi-modal and self-supervised manner, which we call GeneBERT. We pre-train our model on the ATAC-seq dataset with 17 million genome sequences.
  arXiv  Detail & Related papers  (2021-10-11T12:48:44Z)
• AutoBERT-Zero: Evolving BERT Backbone from Scratch [94.89102524181986]
  We propose an Operation-Priority Neural Architecture Search (OP-NAS) algorithm to automatically search for promising hybrid backbone architectures. We optimize both the search algorithm and evaluation of candidate models to boost the efficiency of our proposed OP-NAS. Experiments show that the searched architecture (named AutoBERT-Zero) significantly outperforms BERT and its variants of different model capacities in various downstream tasks.
  arXiv  Detail & Related papers  (2021-07-15T16:46:01Z)
• G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classification [49.53651166356737]
  We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data.
  arXiv  Detail & Related papers  (2021-01-27T19:28:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.