PairSAE: Mechanistic Interpretability from Pair Representations in Protein Co-Folding

• URL: http://arxiv.org/abs/2606.27440v1
• Date: Thu, 25 Jun 2026 18:01:44 GMT
• Title: PairSAE: Mechanistic Interpretability from Pair Representations in Protein Co-Folding
• Abstract summary: We introduce PairSAE, which summarizes pairwise tensors via an N-mode SVD into token-wise interaction roles.<n>We then use a sparse autoencoder to learn a shared set of token-level features that decode into both sequence and pair representations.<n>These results indicate that PairSAE links the latent space of foundation models for structural biology to interpretable structural concepts.
• Score: 2.360330081206126
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models for structural biology have achieved remarkable performance in predicting biomolecular structure and show promise for the design of proteins and small molecules. Yet understanding which internal features drive their outputs remains challenging. Standard sparse autoencoders (SAEs), effective on transformer-style sequence embeddings, do not transfer cleanly to pairformer-like architectures: naively operating on pairwise representations yields a quadratic blow-up of features and obscures concepts distributed jointly across sequence and pair representations. We introduce PairSAE, which summarizes pairwise tensors via an N-mode SVD into token-wise interaction roles, then uses a sparse autoencoder to learn a shared set of token-level features that decode into both sequence and pair representations. Evaluated on Boltz-2 activations for PLINDER protein-ligand complexes, PairSAE yields interpretable features that align with UniProt annotations and predict Boltz-2 affinity values. These results indicate that PairSAE links the latent space of foundation models for structural biology to interpretable structural concepts, clarifying what the model "knows" while avoiding pairformer-induced pitfalls that limit conventional SAEs.

Related papers

• SimpleDesign: A Joint Model for Protein Sequence and Structure Codesign [53.93453139057735]
  We present SimpleDesign, an effective multi-modal protein design model trained directly in the data space.<n>We train SimpleDesign on over 2M sequence-structure pairs achieving strong performance across co-design and unconditional sequence/structure generation benchmarks.
  arXiv  Detail & Related papers  (2026-09-03T05:25:05Z)
• PathAR: Structure-First Autoregressive Synthesis of Multimodal Pathology Images [51.428093790826814]
  We propose pathology Autorgressive modeling (PathAR) to factorize structure and appearance for modality-label-conditioned pathology generation.<n>PathAR stabilizes morphology under heterogeneous modality-specific appearances and enables spatially aligned image-mask pair generation.
  arXiv  Detail & Related papers  (2026-06-01T01:43:48Z)
• From Token Lists to Graph Motifs: Weisfeiler-Lehman Analysis of Sparse Autoencoder Features [0.0]
  We introduce a graph-structured representation in which each SAE feature is modelled as a token co-occurrence graph.<n>A custom WL-style, frequency-binned graph kernel then provides a similarity measure over this structural space.<n>A token-histogram baseline characterises higher overall purity, so the contribution of the graph view is complementary rather than dominant.
  arXiv  Detail & Related papers  (2026-05-07T16:15:16Z)
• Mechanisms of AI Protein Folding in ESMFold [60.598995350547]
  We trace how ESMFold folds a beta hairpin, a prevalent structural motif.<n>We demonstrate that the mechanisms underlying structural decisions of ESMFold can be localized, traced through interpretable representations, and manipulated with strong causal effects.
  arXiv  Detail & Related papers  (2026-02-05T18:54:54Z)
• PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decoding [16.10987386200554]
  We introduce PolySAE, which extends the SAE decoder with higher-order terms to model feature interactions.<n>Across four language models and three SAE variants, PolySAE achieves an average improvement of approximately 8% in probing F1.
  arXiv  Detail & Related papers  (2026-02-01T16:34:45Z)
• HD-Prot: A Protein Language Model for Joint Sequence-Structure Modeling with Continuous Structure Tokens [33.896084921181476]
  HD-Prot is a hybrid diffusion protein language model that embeds a continuous-valued diffusion head atop a discrete pLM.<n>It achieves competitive performance in unconditional sequence-structure co-generation, motif-scaffolding, protein structure prediction, and inverse folding tasks.
  arXiv  Detail & Related papers  (2025-12-17T06:46:27Z)
• Understanding protein function with a multimodal retrieval-augmented foundation model [4.281723404774888]
  PoET-2 is a retrieval-augmented protein foundation model that incorporates in-context learning of family-specific evolutionary constraints.<n>PoET-2 achieves state-of-the-art performance on zero-shot variant effect prediction.
  arXiv  Detail & Related papers  (2025-08-05T15:11:25Z)
• Loss-Complexity Landscape and Model Structure Functions [53.92822954974537]
  We develop a framework for dualizing the Kolmogorov structure function $h_x(alpha)$.<n>We establish a mathematical analogy between information-theoretic constructs and statistical mechanics.<n>We explicitly prove the Legendre-Fenchel duality between the structure function and free energy.
  arXiv  Detail & Related papers  (2025-07-17T21:31:45Z)
• DPLM-2: A Multimodal Diffusion Protein Language Model [75.98083311705182]
  We introduce DPLM-2, a multimodal protein foundation model that extends discrete diffusion protein language model (DPLM) to accommodate both sequences and structures. DPLM-2 learns the joint distribution of sequence and structure, as well as their marginals and conditionals. Empirical evaluation shows that DPLM-2 can simultaneously generate highly compatible amino acid sequences and their corresponding 3D structures.
  arXiv  Detail & Related papers  (2024-10-17T17:20:24Z)
• FoldToken: Learning Protein Language via Vector Quantization and Beyond [56.19308144551836]
  We introduce textbfFoldTokenizer to represent protein sequence-structure as discrete symbols. We refer to the learned symbols as textbfFoldToken, and the sequence of FoldTokens serves as a new protein language.
  arXiv  Detail & Related papers  (2024-02-04T12:18:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.