Beyond Tokens: Probing Higher-Order Epistasis in Learned Protein Representations

• URL: http://arxiv.org/abs/2608.24953v1
• Date: Mon, 24 Aug 2026 23:35:36 GMT
• Title: Beyond Tokens: Probing Higher-Order Epistasis in Learned Protein Representations
• Abstract summary: We introduce ORBIT, an Order-Resolved Benchmarking of Interaction Transformations framework that separates interaction presence, representation accessibility, and functional recovery.<n>We compare ridge regression, a standard, independent tokens, nonlinear independent tokens, and Residual Interaction Tokenization (RIT)<n>RIT significantly increased pairwise accessibility at the token stage relative to both independent-token controls.
• Score: 3.1035948531044912
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Protein fitness landscapes contain nonlinear interactions in which mutation effects depend on other residues. We introduce ORBIT, an Order-Resolved Benchmarking of Interaction Transformations framework that separates interaction presence, representation accessibility, and functional recovery. ORBIT first validates Walsh-based diagnostics on synthetic landscapes with known interaction order, then analyzes the experimentally measured GB1 fitness landscape under the FLIP 2-vs-rest setting. We compare ridge regression, a standard MLP, independent tokens, nonlinear independent tokens, and Residual Interaction Tokenization (RIT). Across 20 paired training seeds, the primary two-hidden-layer comparison found no significant architecture differences in FLIP test R^2, third- or fourth-order functional recovery, or final-layer third- or fourth-order accessibility. However, RIT significantly increased pairwise accessibility at the token stage relative to both independent-token controls (Delta A_tok,2 = 0.2468, d_z = 1.67, Holm-adjusted p = 1.14 x 10^-5), without a detectable downstream higher-order advantage. A pre-specified depth/capacity analysis showed that deeper MLPs improved FLIP prediction, third-order functional recovery, and final-layer third-order accessibility; fourth-order accessibility also improved relative to the shallow MLP but remained below zero in absolute held-out R^2. ORBIT therefore reveals representation-level changes hidden by conventional prediction metrics and distinguishes early interaction-aware encoding from higher-order structure constructed by downstream nonlinear capacity.

Related papers

• One Step, One Lead: Mitigating Higher-Order Interference in Multi-Domain Reinforcement Learning via Cross-Step Control [62.82372407840088]
  We show that same-point domain gradients may remain nearly even when consecutive realized updates partially reverse one another in output space.<n>We propose OSOL, which designates a focus domain at each iteration, to rank token-level rebound risk.<n>OSOL reaches a domain-macro average of 0.4822, improving by 5.7% over the strongest compared baseline, without explicit higher-order differentiation.
  arXiv  Detail & Related papers  (2026-09-06T08:32:59Z)
• ProCA: Progressive Contrastive Alignment for Robust EEG Visual Decoding [47.35612571473169]
  EEG visual decoding aims to recover visual semantics from non-invasive neural time-series signals.<n>Existing methods rely on fixed visual or textual anchors whose semantic relations may become misaligned with EEG representations that vary across trials, subjects, and learning stages.<n>We propose Progressive Contrastive Alignment (ProCA), a unified and model-agnostic framework for adaptive neural-semantic alignment.
  arXiv  Detail & Related papers  (2026-09-04T12:48:25Z)
• TESTNAV: Pareto-Guided Search for Compositional Robustness Testing [42.86570387250456]
  TESTNAV is a bi-objective robustness testing framework for deep learning models.<n>It maximises performance degradation while preserving input fidelity measured by modality-specific metrics.
  arXiv  Detail & Related papers  (2026-08-20T10:43:22Z)
• Feature Priming in Online Linear Regression: Sparse-Regret Lower Bounds and a Tight Univariate Rate [47.36630149538994]
  In high-dimensional online prediction, the best predictor may depend on only a few features, so regret should scale with sparsity rather than the ambient dimension.<n>Warmuth and Amid asked at COLT 2023 whether any of three such rules admits a competitive online regret guarantee.<n>Our analysis identifies a common obstruction: cheap nuisance causes the refit to underweight the truly predictive coordinate.
  arXiv  Detail & Related papers  (2026-08-18T09:32:03Z)
• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention [16.36212521143563]
  We treat the Transformer's attention mechanism as a perturbation of the trained residual-stack fixed point.<n>Testable predictions are made for the fixed-point geometry, effective rank profile, layer specificity, and perturbation decay spectrum.
  arXiv  Detail & Related papers  (2026-07-16T20:42:02Z)
• Attention Degradation, Function Token Anchoring, and the Limits of Attention-Based Intervention in Large Language Models [0.0]
  We present six coordinated experiments across GPT-2, LLaMA-3.2-1B/3B, OPT-1.3B, and distilgpt2.<n>We first characterise short-term (5-100 token) attention degradation, finding a universal exponential-then-plateau pattern.<n>We conclude that mean attention degradation is largely rather than prescriptive.
  arXiv  Detail & Related papers  (2026-07-09T07:04:07Z)
• Graph Diffusion Residuals for Control-Function Instrumental Variables [82.39671985893028]
  High-capacity first stages can interpolate treatment and leave too little residual information for the outcome equation.<n>We study Adaptive Anisotropic Instrumental Heat Flow (A-IHF), a deterministic graph-diffusion residual extractor for flexible control functions.<n>A-IHF treats treatment as a signal on a graph of first-stage features, uses pilot diffusion to detect large treatment jumps, attenuates conductance across those jumps, and computes the generated control with a sparse graph resolvent.
  arXiv  Detail & Related papers  (2026-06-12T17:02:53Z)
• When Eyes Betray AI: Social Gaze Consistency as a Semantic Cue for AI-Generated Image Detection [12.446807294893638]
  We introduce Social Gaze Consistency, a high-level semantic cue defined as the mutual coherence of gaze direction, head-eye alignment, and pupil placement between interacting individuals.<n>We show that it constitutes a previously underutilized detection axis to existing low-level paradigms.<n>A four-step account explains why training on a single inpainter (FLUX.1-Fill) transfers to multi-generator suites.
  arXiv  Detail & Related papers  (2026-05-26T17:50:17Z)
• Auxiliary Finite-Difference Residual-Gradient Regularization for PINNs [0.0]
  We study a hybrid design in which the governing PDE residual remains automatic-differentiation based, while finite differences appear only in a weak auxiliary term.<n>The FD term regularizes the residual field without replacing the PDE residual itself.<n>Across seeds 0-5 and 100k epochs, the most reliable tested configuration is a fixed shell weight of 5e-4 under the Kourkoutas-beta regime.
  arXiv  Detail & Related papers  (2026-04-15T23:09:19Z)
• Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials [34.77694214755808]
  We introduce Visual-Contrast Attention (VCA), a drop-in replacement for Multi-Head Self-Attention (MHSA)<n>VCA injects an explicit notion of discrimination while reducing the theoretical complexity from O(N N C) to O(N n C) with n N.<n>The module adds fewer than 0.3M parameters to a DeiT-Tiny backbone, requires no extra FLOPs, and is wholly architecture-agnostic.
  arXiv  Detail & Related papers  (2025-11-02T07:04:12Z)
• Pearl: A Foundation Model for Placing Every Atom in the Right Location [52.35027831422145]
  We introduce Pearl, a foundation model for protein-ligand cofolding at scale.<n>Pearl establishes a new state-of-the-art performance in protein-ligand cofolding.<n>Pearl surpasses AlphaFold 3 and other open source baselines on the public Runs N' Poses and PoseBusters benchmarks.
  arXiv  Detail & Related papers  (2025-10-28T17:36:51Z)
• Contrastive Conditional Neural Processes [45.70735205041254]
  Conditional Neural Processes(CNPs) bridge neural networks with probabilistic inference to approximate functions of Processes under meta-learning settings. Two auxiliary contrastive branches are set up hierarchically, namely in-instantiation temporal contrastive learning(tt TCL) and cross-instantiation function contrastive learning(tt FCL) We empirically show that tt TCL captures high-level abstraction of observations, whereas tt FCL helps identify underlying functions, which in turn provides more efficient representations.
  arXiv  Detail & Related papers  (2022-03-08T10:08:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.