Lyra: An Efficient and Expressive Subquadratic Architecture for Modeling Biological Sequences

• URL: http://arxiv.org/abs/2503.16351v1
• Date: Thu, 20 Mar 2025 17:09:18 GMT
• Title: Lyra: An Efficient and Expressive Subquadratic Architecture for Modeling Biological Sequences
• Authors: Krithik Ramesh, Sameed M. Siddiqui, Albert Gu, Michael D. Mitzenmacher, Pardis C. Sabeti,
• Abstract summary: We introduce Lyra, a subquadratic architecture for sequence modeling, grounded in the biological framework of epistasis.<n>We demonstrate that Lyra is performant across over 100 wide-ranging biological tasks, achieving state-of-the-art (SOTA) performance in many key areas.
• Score: 12.6343176904061
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Deep learning architectures such as convolutional neural networks and Transformers have revolutionized biological sequence modeling, with recent advances driven by scaling up foundation and task-specific models. The computational resources and large datasets required, however, limit their applicability in biological contexts. We introduce Lyra, a subquadratic architecture for sequence modeling, grounded in the biological framework of epistasis for understanding sequence-to-function relationships. Mathematically, we demonstrate that state space models efficiently capture global epistatic interactions and combine them with projected gated convolutions for modeling local relationships. We demonstrate that Lyra is performant across over 100 wide-ranging biological tasks, achieving state-of-the-art (SOTA) performance in many key areas, including protein fitness landscape prediction, biophysical property prediction (e.g. disordered protein region functions) peptide engineering applications (e.g. antibody binding, cell-penetrating peptide prediction), RNA structure analysis, RNA function prediction, and CRISPR guide design. It achieves this with orders-of-magnitude improvements in inference speed and reduction in parameters (up to 120,000-fold in our tests) compared to recent biology foundation models. Using Lyra, we were able to train and run every task in this study on two or fewer GPUs in under two hours, democratizing access to biological sequence modeling at SOTA performance, with potential applications to many fields.

Related papers

• Leveraging State Space Models in Long Range Genomics [1.130790932059036]
  Long-range dependencies are critical for understanding genomic structure and function, yet most conventional methods struggle with them. We explore State Space Models (SSMs) as a promising alternative by benchmarking two SSM-inspired architectures on long-range genomics modeling tasks. SSMs match transformer performance and exhibit impressive zero-shot extrapolation across multiple tasks, handling contexts 10 to 100 times longer than those seen during training.
  arXiv  Detail & Related papers  (2025-04-07T18:34:06Z)
• UniGenX: Unified Generation of Sequence and Structure with Autoregressive Diffusion [61.690978792873196]
  Existing approaches rely on either autoregressive sequence models or diffusion models.<n>We propose UniGenX, a unified framework that combines autoregressive next-token prediction with conditional diffusion models.<n>We validate the effectiveness of UniGenX on material and small molecule generation tasks.
  arXiv  Detail & Related papers  (2025-03-09T16:43:07Z)
• GENERator: A Long-Context Generative Genomic Foundation Model [66.46537421135996]
  We present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs (bp) and 1.2B parameters.<n>Trained on an expansive dataset comprising 386B bp of DNA, the GENERator demonstrates state-of-the-art performance across both established and newly proposed benchmarks.<n>It also shows significant promise in sequence optimization, particularly through the prompt-responsive generation of enhancer sequences with specific activity profiles.
  arXiv  Detail & Related papers  (2025-02-11T05:39:49Z)
• Biology Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Models [51.316001071698224]
  We introduce Biology-Instructions, the first large-scale multi-omics biological sequences-related instruction-tuning dataset.<n>This dataset can bridge the gap between large language models (LLMs) and complex biological sequences-related tasks.<n>We also develop a strong baseline called ChatMultiOmics with a novel three-stage training pipeline.
  arXiv  Detail & Related papers  (2024-12-26T12:12:23Z)
• MAMMAL -- Molecular Aligned Multi-Modal Architecture and Language [0.24434823694833652]
  MAMMAL is a versatile multi-task multi-align foundation model that learns from large-scale biological datasets. We introduce a prompt syntax that supports a wide range of classification, regression, and generation tasks. We evaluate the model on 11 diverse downstream tasks spanning different steps within a typical drug discovery pipeline.
  arXiv  Detail & Related papers  (2024-10-28T20:45:52Z)
• The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks [64.08042492426992]
  We introduce the Expressive Memory (ELM) neuron model, a biologically inspired model of a cortical neuron. Our ELM neuron can accurately match the aforementioned input-output relationship with under ten thousand trainable parameters. We evaluate it on various tasks with demanding temporal structures, including the Long Range Arena (LRA) datasets.
  arXiv  Detail & Related papers  (2023-06-14T13:34:13Z)
• Generative Pretrained Autoregressive Transformer Graph Neural Network applied to the Analysis and Discovery of Novel Proteins [0.0]
  We report a flexible language-model based deep learning strategy, applied here to solve complex forward and inverse problems in protein modeling. The model is applied to predict secondary structure content (per-residue level and overall content), protein solubility, and sequencing tasks. We find that adding additional tasks yields emergent synergies that the model exploits in improving overall performance.
  arXiv  Detail & Related papers  (2023-05-07T12:30:24Z)
• Deep Bayesian Active Learning for Accelerating Stochastic Simulation [74.58219903138301]
  Interactive Neural Process (INP) is a deep active learning framework for simulations and with active learning approaches. For active learning, we propose a novel acquisition function, Latent Information Gain (LIG), calculated in the latent space of NP based models. The results demonstrate STNP outperforms the baselines in the learning setting and LIG achieves the state-of-the-art for active learning.
  arXiv  Detail & Related papers  (2021-06-05T01:31:51Z)
• Align-gram : Rethinking the Skip-gram Model for Protein Sequence Analysis [0.8733639720576208]
  We propose a novel embedding scheme, Align-gram, which is capable of mapping the similar $k$-mers close to each other in a vector space. Our experiments with a simple baseline LSTM model and a much complex CNN model of DeepGoPlus shows the potential of Align-gram in performing different types of deep learning applications for protein sequence analysis.
  arXiv  Detail & Related papers  (2020-12-06T17:04:17Z)
• Towards an Automatic Analysis of CHO-K1 Suspension Growth in Microfluidic Single-cell Cultivation [63.94623495501023]
  We propose a novel Machine Learning architecture, which allows us to infuse a neural deep network with human-powered abstraction on the level of data. Specifically, we train a generative model simultaneously on natural and synthetic data, so that it learns a shared representation, from which a target variable, such as the cell count, can be reliably estimated.
  arXiv  Detail & Related papers  (2020-10-20T08:36:51Z)
• Interpretable Structured Learning with Sparse Gated Sequence Encoder for Protein-Protein Interaction Prediction [2.9488233765621295]
  Predicting protein-protein interactions (PPIs) by learning informative representations from amino acid sequences is a challenging yet important problem in biology. We present a novel deep framework to model and predict PPIs from sequence alone. Our model incorporates a bidirectional gated recurrent unit to learn sequence representations by leveraging contextualized and sequential information from sequences.
  arXiv  Detail & Related papers  (2020-10-16T17:13:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.