Retrieval and competition: how a protein foundation model starts a protein

• URL: http://arxiv.org/abs/2605.16331v1
• Date: Tue, 05 May 2026 17:51:21 GMT
• Title: Retrieval and competition: how a protein foundation model starts a protein
• Abstract summary: A near-universal biological rule, that proteins begin with methionine, is studied by tracing the computational pathway through which ESM2-8M produces this prediction.<n>The model does not detect methionine at the masked position. Instead, it retrieves a methionine-favouring signal from a reference representation at the beginning-of-sequence token via a position-specific query.<n>The model's prediction is mediated by a distributed computational circuit rather than direct recognition, suggesting that increasing task complexity will further obscure the relationship between model confidence and underlying biological evidence.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Protein language models are increasingly used to guide experimental and clinical decisions, yet it is often unclear whether a confident prediction reflects recognition of biological evidence or retrieval of a statistical default. We examine this distinction for a near-universal biological rule, that proteins begin with methionine, by tracing the computational pathway through which ESM2-8M produces this prediction. The model does not detect methionine at the masked position. Instead, it retrieves a methionine-favouring signal from a reference representation at the beginning-of-sequence token via a position-specific query assembled across layers, with the final output emerging through competition with context-dependent circuits. To understand how positional information reaches the readout, we introduce a norm-direction decomposition of attention scores within rotary frequency bands. Positional encoding operates through coupled changes in query norm and angular alignment distributed across these bands. On sequences whose true N-terminus is not methionine, where the biological question matters, the model predicts methionine anyway. This is not a correct prediction produced by an unexpected mechanism, but the output of a positional-prior retrieval circuit that matches the statistical average and fails where biology diverges from it. Distinguishing the two requires resolution at the level of individual circuits, frequency bands, and query composition, suggesting that mechanistic verification will be necessary, and challenging, for predictions where the biological stakes are higher. Even for the simplest biological rule, the model's prediction is mediated by a distributed computational circuit rather than direct recognition, suggesting that increasing task complexity will further obscure the relationship between model confidence and underlying biological evidence.

Related papers

• SeqMaestro: From nucleotide sequences to biological hypotheses through interpretable machine learning [35.97164357682394]
  SeqMaestro is a machine learning framework that proposes hypotheses from nucleotide sequences using interpretable models.<n>Our solution is centered around a two-layer interface that connects nucleotide sequences with the broader ecosystem of interpretable machine learning.
  arXiv  Detail & Related papers  (2026-09-14T01:04:09Z)
• Orthogonal JEPA: Factorized Predictive States for Latent World Models [95.20158869896393]
  We introduce method, a latent world-modeling framework based on predictive factorization.<n>We show that method can allocate redundant capacity to dominant signals while providing weak or conflicting gradients to less dominant predictive structure.<n>Experiments on controlled vision, single-cell transcriptomics, longitudinal health records, continuous control, and molecular dynamics evaluate representation quality, forecasting, planning, and long-horizon stability.
  arXiv  Detail & Related papers  (2026-08-20T13:59:57Z)
• PertReason: A Knowledge-Grounded Benchmark and Framework for Cell-State-Conditioned Mechanistic Reasoning of Perturbation Effects [11.347908338100515]
  We introduce PertReason, a knowledge-grounded benchmark and framework suite for cell-state-conditioned reasoning.<n>PertReasonQA is a benchmark that tests whether models can generate mechanistically faithful explanations.<n>We present PertReasonLM, a large language model trained to align outcome predictions with context-specific mechanistic reasoning.
  arXiv  Detail & Related papers  (2026-07-21T06:59:25Z)
• Structured Gaussian Processes for Uncertainty-Aware Classification of High-Dimensional, Small-Sampled Omics Data [5.649446622508846]
  Classifying heterogeneous omics data remains a fundamental challenge in computational biology.<n>We propose a structured Gaussian process classification framework that integrates graph-encoded biological pathways directly into the kernel construction.<n>We benchmark our proposed methodology on three publicly available gut and fecal microbiome datasets.
  arXiv  Detail & Related papers  (2026-07-02T12:37:40Z)
• Neural Conjugate Aggregation: Identifiable Unsupervised Multi-Sensor Regression under Heterogeneous Sensor Bias [0.0]
  We study regression-based data fusion under uncertainty, where multiple noisy and biased measurement sources are available but ground-truth labels are absent during training.<n>This setting arises in sensor networks, simulation ensembles, and scientific monitoring systems where supervision is costly or infeasible.<n>We propose the Neural Conjugate Aggregation Model (NCAM), a hierarchical Bayesian framework that combines neural networks with conjugate Gaussian inference for unsupervised multi-source fusion.
  arXiv  Detail & Related papers  (2026-06-20T19:30:21Z)
• Protein Thoughts: Interpretable Reasoning with Tree of Thoughts and Embedding-Space Flow Matching for Protein-Protein Interaction Discovery [9.70057752523548]
  We present textbfProtein Thoughts, a framework that reformulates PPI discovery as an interpretable search problem with explicit reasoning.<n>Protein Thoughts achieves mean best-binder rank of 11.2 versus 47.7 for an entropic tree search baseline, a 76% improvement, and for binding prediction the trained value function achieves $91.08 pm 0.19$ Micro-F1.
  arXiv  Detail & Related papers  (2026-05-19T04:14:06Z)
• A neurosymbolic Approach with Epistemic Deep Learning for Hierarchical Image Classification [7.18839452757154]
  We propose a unified neurosymbolic and epistemic modelling framework to augment Swin Transformers.<n>Rather than treating labels as isolated categories, our method induces data-driven focal sets.<n>These focal sets form the basis of a belief-theoretic layer.<n>A learnable loss balances calibration, mass regularisation, and logical consistency, allowing the model to adaptively trade off symbolic structure with data-driven evidence.
  arXiv  Detail & Related papers  (2026-05-11T09:43:43Z)
• Decoupled PFNs: Identifiable Epistemic-Aleatoric Decomposition via Structured Synthetic Priors [25.166488691037078]
  Prior-Fitted Networks (PFNs) amortize Bayesian prediction by meta-learning over a synthetic task prior, but their standard output is a posterior predictive distribution over noisy observations.<n>We show that this epistemic--aleatoric split is not identifiable in general from the posterior predictive distribution alone.<n>We use these labels to train a decoupled PFN with separate latent-signal and aleatoric heads.
  arXiv  Detail & Related papers  (2026-05-07T15:22:35Z)
• Detecting Batch Heterogeneity via Likelihood Clustering [0.9668407688201359]
  Batch effects represent a major confounder in genomic diagnostics.<n>We introduce a method that addresses both limitations by clustering samples according to their Bayesian model evidence.<n>Our method achieves superior clustering accuracy compared to standard correlation-based and dimensionality-reduction approaches.
  arXiv  Detail & Related papers  (2026-01-14T01:49:21Z)
• Chaos, Entanglement and Measurement: Field-Theoretic Perspectives on Quantum Information Dynamics [0.0]
  I study scrambling and pseudorandomness in the Brownian Sachdev-Ye-Kitaev (SYK) model.<n>I construct a field theory for weakly measured SYK clusters.<n>I develop a strong-disorder renormalization group for measurement-only SYK clusters.
  arXiv  Detail & Related papers  (2025-12-11T10:04:30Z)
• Departures: Distributional Transport for Single-Cell Perturbation Prediction with Neural Schrödinger Bridges [51.83259180910313]
  A major bottleneck in gene function analysis is the unpaired nature of single-cell data.<n>We approximate Schrdinger Bridge (SB) to tackle unpaired single-cell perturbation data.<n>Our model effectively captures heterogeneous single-cell responses and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-11-17T08:27:13Z)
• Unlasting: Unpaired Single-Cell Multi-Perturbation Estimation by Dual Conditional Diffusion Implicit Bridges [68.98973318553983]
  We propose a framework based on Dual Diffusion Implicit Bridges (DDIB) to learn the mapping between different data distributions.<n>We integrate gene regulatory network (GRN) information to propagate perturbation signals in a biologically meaningful way.<n>We also incorporate a masking mechanism to predict silent genes, improving the quality of generated profiles.
  arXiv  Detail & Related papers  (2025-06-26T09:05:38Z)
• Detecting and Identifying Selection Structure in Sequential Data [53.24493902162797]
  We argue that the selective inclusion of data points based on latent objectives is common in practical situations, such as music sequences. We show that selection structure is identifiable without any parametric assumptions or interventional experiments. We also propose a provably correct algorithm to detect and identify selection structures as well as other types of dependencies.
  arXiv  Detail & Related papers  (2024-06-29T20:56:34Z)
• Neuro-Symbolic Entropy Regularization [78.16196949641079]
  In structured prediction, the goal is to jointly predict many output variables that together encode a structured object. One approach -- entropy regularization -- posits that decision boundaries should lie in low-probability regions. We propose a loss, neuro-symbolic entropy regularization, that encourages the model to confidently predict a valid object.
  arXiv  Detail & Related papers  (2022-01-25T06:23:10Z)
• The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
  The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data. This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
  arXiv  Detail & Related papers  (2020-12-05T17:30:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.