Synthesizing Proteins on the Graphics Card. Protein Folding and the Limits of Critical AI Studies

• URL: http://arxiv.org/abs/2405.09788v2
• Date: Sat, 07 Dec 2024 18:24:44 GMT
• Title: Synthesizing Proteins on the Graphics Card. Protein Folding and the Limits of Critical AI Studies
• Authors: Fabian Offert, Paul Kim, Qiaoyu Cai,
• Abstract summary: This paper investigates the application of the transformer architecture in protein folding.<n>We contend that our search for intelligent machines has to begin with the shape, rather than the place, of intelligence.
• Score: 0.8192907805418581
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper investigates the application of the transformer architecture in protein folding, as exemplified by DeepMind's AlphaFold project, and its implications for the understanding of so-called large language models. The prevailing discourse often assumes a ready-made analogy between proteins, encoded as sequences of amino acids, and natural language, which we term the language paradigm of computational (structural) biology. Instead of assuming this analogy as given, we critically evaluate it to assess the kind of knowledge-making afforded by the transformer architecture. We first trace the analogy's emergence and historical development, carving out the influence of structural linguistics on structural biology beginning in the mid-20th century. We then examine three often overlooked preprocessing steps essential to the transformer architecture, including subword tokenization, word embedding, and positional encoding, to demonstrate its regime of representation based on continuous, high-dimensional vector spaces, which departs from the discrete nature of language. The successful deployment of transformers in protein folding, we argue, discloses what we consider a non-linguistic approach to token processing intrinsic to the architecture. We contend that through this non-linguistic processing, the transformer architecture carves out unique epistemological territory and produces a new class of knowledge, distinct from established domains. We contend that our search for intelligent machines has to begin with the shape, rather than the place, of intelligence. Consequently, the emerging field of critical AI studies should take methodological inspiration from the history of science in its quest to conceptualize the contributions of artificial intelligence to knowledge-making, within and beyond the domain-specific sciences.

Related papers

• Epistemology of Generative AI: The Geometry of Knowing [0.7252027234425333]
  Generative AI presents an unprecedented challenge to our understanding of knowledge and its production.<n>This paper argues that the missing account must begin with a paradigmatic break that has not yet received adequate philosophical attention.
  arXiv  Detail & Related papers  (2026-02-19T06:34:34Z)
• The Vibe-Automation of Automation: A Proactive Education Framework for Computer Science in the Age of Generative AI [0.7252027234425333]
  generative artificial intelligence (GenAI) represents a qualitative shift in computer science.<n>GenAI operates by navigating contextual, semantic, and coherence rather than optimizing predefined objective metrics.<n>The paper proposes a conceptual framework structured across three analytical levels and three domains of action.
  arXiv  Detail & Related papers  (2026-02-09T06:02:04Z)
• Evolutionary Developmental Biology Can Serve as the Conceptual Foundation for a New Design Paradigm in Artificial Intelligence [15.376349115976534]
  Artificial intelligence (AI) has made significant strides in solving complex tasks.<n>Current neural network-based paradigm, while effective, is heavily constrained by inherent limitations.<n>Recent paradigm shift in evolutionary understanding has been largely overlooked in AI literature.
  arXiv  Detail & Related papers  (2025-06-15T15:41:44Z)
• Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures [49.19753720526998]
  We derive theoretical scaling laws for neural network performance on synthetic datasets.<n>We validate that convolutional networks, whose structure aligns with that of the generative process through locality and weight sharing, enjoy a faster scaling of performance.<n>This finding clarifies the architectural biases underlying neural scaling laws and highlights how representation learning is shaped by the interaction between model architecture and the statistical properties of data.
  arXiv  Detail & Related papers  (2025-05-11T17:44:14Z)
• Advanced Deep Learning Methods for Protein Structure Prediction and Design [28.575821996185024]
  We comprehensively explore advanced deep learning methods applied to protein structure prediction and design. The text analyses key components including structure generation, evaluation metrics, multiple sequence alignment processing, and network architecture. Strategies for enhancing prediction accuracy and integrating deep learning techniques with experimental validation are thoroughly explored.
  arXiv  Detail & Related papers  (2025-03-14T21:28:29Z)
• FinchGPT: a Transformer based language model for birdsong analysis [24.273645850815207]
  Long-range dependencies among tokens are a defining hallmark of human language. In this study, we employed the Transformer architecture to analyze the songs of Bengalese finch (Lonchura striata domestica) We developed FinchGPT, a Transformer-based model trained on a textualized corpus of birdsongs.
  arXiv  Detail & Related papers  (2025-02-01T07:06:34Z)
• Understanding Token-level Topological Structures in Transformer-based Time Series Forecasting [52.364260925700485]
  Transformer-based methods have achieved state-of-the-art performance in time series forecasting (TSF)<n>It remains unclear whether existing Transformers fully leverage the intrinsic topological structure among tokens throughout intermediate layers.<n>We propose the Topology Enhancement Method (TEM), a novel Transformer-based TSF method that explicitly and adaptively preserves token-level topology.
  arXiv  Detail & Related papers  (2024-04-16T07:21:39Z)
• Language Evolution with Deep Learning [49.879239655532324]
  Computational modeling plays an essential role in the study of language emergence. It aims to simulate the conditions and learning processes that could trigger the emergence of a structured language. This chapter explores another class of computational models that have recently revolutionized the field of machine learning: deep learning models.
  arXiv  Detail & Related papers  (2024-03-18T16:52:54Z)
• FoldToken: Learning Protein Language via Vector Quantization and Beyond [56.19308144551836]
  We introduce textbfFoldTokenizer to represent protein sequence-structure as discrete symbols. We refer to the learned symbols as textbfFoldToken, and the sequence of FoldTokens serves as a new protein language.
  arXiv  Detail & Related papers  (2024-02-04T12:18:51Z)
• Endowing Protein Language Models with Structural Knowledge [5.587293092389789]
  We introduce a novel framework that enhances protein language models by integrating protein structural data. The refined model, termed Protein Structure Transformer (PST), is further pretrained on a small protein structure database. PST consistently outperforms the state-of-the-art foundation model for protein sequences, ESM-2, setting a new benchmark in protein function prediction.
  arXiv  Detail & Related papers  (2024-01-26T12:47:54Z)
• A Neuro-mimetic Realization of the Common Model of Cognition via Hebbian Learning and Free Energy Minimization [55.11642177631929]
  Large neural generative models are capable of synthesizing semantically rich passages of text or producing complex images. We discuss the COGnitive Neural GENerative system, such an architecture that casts the Common Model of Cognition.
  arXiv  Detail & Related papers  (2023-10-14T23:28:48Z)
• Large Language Models for Scientific Synthesis, Inference and Explanation [56.41963802804953]
  We show how large language models can perform scientific synthesis, inference, and explanation. We show that the large language model can augment this "knowledge" by synthesizing from the scientific literature. This approach has the further advantage that the large language model can explain the machine learning system's predictions.
  arXiv  Detail & Related papers  (2023-10-12T02:17:59Z)
• LOGICSEG: Parsing Visual Semantics with Neural Logic Learning and Reasoning [73.98142349171552]
  LOGICSEG is a holistic visual semantic that integrates neural inductive learning and logic reasoning with both rich data and symbolic knowledge. During fuzzy logic-based continuous relaxation, logical formulae are grounded onto data and neural computational graphs, hence enabling logic-induced network training. These designs together make LOGICSEG a general and compact neural-logic machine that is readily integrated into existing segmentation models.
  arXiv  Detail & Related papers  (2023-09-24T05:43:19Z)
• How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding [56.222097640468306]
  We provide mechanistic understanding of how transformers learn "semantic structure" We show, through a combination of mathematical analysis and experiments on Wikipedia data, that the embedding layer and the self-attention layer encode the topical structure.
  arXiv  Detail & Related papers  (2023-03-07T21:42:17Z)
• Strong-AI Autoepistemic Robots Build on Intensional First Order Logic [0.0]
  We consider the intensional First Order Logic (IFOL) as a symbolic architecture of modern robots. We present a particular example of robots autoepistemic deduction capabilities by introduction of the special temporal $Konow$ predicate and deductive axioms.
  arXiv  Detail & Related papers  (2022-12-14T16:23:56Z)
• Learning Geometrically Disentangled Representations of Protein Folding Simulations [72.03095377508856]
  This work focuses on learning a generative neural network on a structural ensemble of a drug-target protein. Model tasks involve characterizing the distinct structural fluctuations of the protein bound to various drug molecules. Results show that our geometric learning-based method enjoys both accuracy and efficiency for generating complex structural variations.
  arXiv  Detail & Related papers  (2022-05-20T19:38:00Z)
• Testing the Quantitative Spacetime Hypothesis using Artificial Narrative Comprehension (II) : Establishing the Geometry of Invariant Concepts, Themes, and Namespaces [0.0]
  This study contributes to an ongoing application of the Semantic Spacetime Hypothesis, and demonstrates the unsupervised analysis of narrative texts. Data streams are parsed and fractionated into small constituents, by multiscale interferometry, in the manner of bioinformatic analysis. Fragments of the input act as symbols in a hierarchy of alphabets that define new effective languages at each scale.
  arXiv  Detail & Related papers  (2020-09-23T11:19:17Z)
• Deep Learning in Protein Structural Modeling and Design [6.282267356230666]
  Deep learning is catalyzing a scientific revolution fueled by big data, accessible toolkits, and powerful computational resources. Protein structural modeling is critical to understand and engineer biological systems at the molecular level. This review is directed to help both computational biologists to gain familiarity with the deep learning methods applied in protein modeling, and computer scientists to gain perspective on the biologically meaningful problems that may benefit from deep learning techniques.
  arXiv  Detail & Related papers  (2020-07-16T14:59:38Z)
• BERTology Meets Biology: Interpreting Attention in Protein Language Models [124.8966298974842]
  We demonstrate methods for analyzing protein Transformer models through the lens of attention. We show that attention captures the folding structure of proteins, connecting amino acids that are far apart in the underlying sequence, but spatially close in the three-dimensional structure. We also present a three-dimensional visualization of the interaction between attention and protein structure.
  arXiv  Detail & Related papers  (2020-06-26T21:50:17Z)
• The Unstoppable Rise of Computational Linguistics in Deep Learning [17.572024590374305]
  We trace the history of neural networks applied to natural language understanding tasks. We argue that Transformer is not a sequence model but an induced-structure model.
  arXiv  Detail & Related papers  (2020-05-13T16:51:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.