To Transformers and Beyond: Large Language Models for the Genome

• URL: http://arxiv.org/abs/2311.07621v1
• Date: Mon, 13 Nov 2023 02:13:58 GMT
• Title: To Transformers and Beyond: Large Language Models for the Genome
• Authors: Micaela E. Consens, Cameron Dufault, Michael Wainberg, Duncan Forster, Mehran Karimzadeh, Hani Goodarzi, Fabian J. Theis, Alan Moses, Bo Wang
• Abstract summary: This review focuses on the transformative role of Large Language Models (LLMs), which are mostly based on the transformer architecture, in genomics. Building on the foundation of traditional convolutional neural networks and recurrent neural networks, we explore both the strengths and limitations of transformers. We contemplate the future of genomic modeling beyond the transformer architecture based on current trends in research.
• Score: 2.799755865110429
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In the rapidly evolving landscape of genomics, deep learning has emerged as a useful tool for tackling complex computational challenges. This review focuses on the transformative role of Large Language Models (LLMs), which are mostly based on the transformer architecture, in genomics. Building on the foundation of traditional convolutional neural networks and recurrent neural networks, we explore both the strengths and limitations of transformers and other LLMs for genomics. Additionally, we contemplate the future of genomic modeling beyond the transformer architecture based on current trends in research. The paper aims to serve as a guide for computational biologists and computer scientists interested in LLMs for genomic data. We hope the paper can also serve as an educational introduction and discussion for biologists to a fundamental shift in how we will be analyzing genomic data in the future.

Related papers

• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters. Trained on an expansive dataset comprising 386B bp of DNA, the GENERator demonstrates state-of-the-art performance across both established and newly proposed benchmarks. It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of enhancer sequences with specific activity profiles.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• A Review on the Applications of Transformer-based language models for Nucleotide Sequence Analysis [0.8049701904919515]
  This paper introduces the major developments of Transformer-based models in the recent past in the context of nucleotide sequences. We believe this review will help the scientific community in understanding the various applications of Transformer-based language models to nucleotide sequences.
  arXiv  Detail & Related papers  (2024-12-10T05:33:09Z)
• A Review of Transformer-Based Models for Computer Vision Tasks: Capturing Global Context and Spatial Relationships [0.5639904484784127]
  Transformer-based models have transformed the landscape of natural language processing (NLP) These models are renowned for their ability to capture long-range dependencies and contextual information. We discuss potential research directions and applications of transformer-based models in computer vision.
  arXiv  Detail & Related papers  (2024-08-27T16:22:18Z)
• Hidden Holes: topological aspects of language models [1.1172147007388977]
  We study the evolution of topological structure in GPT based large language models across depth and time during training. We show that the latter exhibit more topological complexity, with a distinct pattern of changes common to all natural languages but absent from synthetically generated data.
  arXiv  Detail & Related papers  (2024-06-09T14:25:09Z)
• Explaining Text Similarity in Transformer Models [52.571158418102584]
  Recent advances in explainable AI have made it possible to mitigate limitations by leveraging improved explanations for Transformers. We use BiLRP, an extension developed for computing second-order explanations in bilinear similarity models, to investigate which feature interactions drive similarity in NLP models. Our findings contribute to a deeper understanding of different semantic similarity tasks and models, highlighting how novel explainable AI methods enable in-depth analyses and corpus-level insights.
  arXiv  Detail & Related papers  (2024-05-10T17:11:31Z)
• A Survey on Large Language Models from Concept to Implementation [4.219910716090213]
  Recent advancements in Large Language Models (LLMs) have broadened the scope of natural language processing (NLP) applications. This paper investigates the multifaceted applications of these models, with an emphasis on the GPT series. This exploration focuses on the transformative impact of artificial intelligence (AI) driven tools in revolutionizing traditional tasks like coding and problem-solving.
  arXiv  Detail & Related papers  (2024-03-27T19:35:41Z)
• Advancing bioinformatics with large language models: components, applications and perspectives [12.728981464533918]
  Large language models (LLMs) are a class of artificial intelligence models based on deep learning. We will provide a comprehensive overview of the essential components of large language models (LLMs) in bioinformatics. Key aspects covered include tokenization methods for diverse data types, the architecture of transformer models, and the core attention mechanism.
  arXiv  Detail & Related papers  (2024-01-08T17:26:59Z)
• Anatomy of Neural Language Models [0.0]
  Transformer-based Language Models (LMs) have led to new state-of-the-art results in a wide spectrum of applications. Transformers pretrained on language-modeling-like tasks have been widely adopted in computer vision and time series applications.
  arXiv  Detail & Related papers  (2024-01-08T10:27:25Z)
• Towards Graph Foundation Models: A Survey and Beyond [66.37994863159861]
  Foundation models have emerged as critical components in a variety of artificial intelligence applications. The capabilities of foundation models to generalize and adapt motivate graph machine learning researchers to discuss the potential of developing a new graph learning paradigm. This article introduces the concept of Graph Foundation Models (GFMs), and offers an exhaustive explanation of their key characteristics and underlying technologies.
  arXiv  Detail & Related papers  (2023-10-18T09:31:21Z)
• A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks [60.38369406877899]
  Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. Our survey encompasses the identification of the top five application domains for transformer-based models.
  arXiv  Detail & Related papers  (2023-06-11T23:13:51Z)
• Transformers for prompt-level EMA non-response prediction [62.41658786277712]
  Ecological Momentary Assessments (EMAs) are an important psychological data source for measuring cognitive states, affect, behavior, and environmental factors. Non-response, in which participants fail to respond to EMA prompts, is an endemic problem. The ability to accurately predict non-response could be utilized to improve EMA delivery and develop compliance interventions.
  arXiv  Detail & Related papers  (2021-11-01T18:38:47Z)
• A Survey on Visual Transformer [126.56860258176324]
  Transformer is a type of deep neural network mainly based on the self-attention mechanism. In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages.
  arXiv  Detail & Related papers  (2020-12-23T09:37:54Z)
• Towards an Automatic Analysis of CHO-K1 Suspension Growth in Microfluidic Single-cell Cultivation [63.94623495501023]
  We propose a novel Machine Learning architecture, which allows us to infuse a neural deep network with human-powered abstraction on the level of data. Specifically, we train a generative model simultaneously on natural and synthetic data, so that it learns a shared representation, from which a target variable, such as the cell count, can be reliably estimated.
  arXiv  Detail & Related papers  (2020-10-20T08:36:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.