Structure-Aware Masking for Protein Representation Learning

• URL: http://arxiv.org/abs/2605.16581v1
• Date: Fri, 15 May 2026 19:36:54 GMT
• Title: Structure-Aware Masking for Protein Representation Learning
• Abstract summary: We introduce Bucket Masking, a structure-aware masking strategy that selects groups of residues based on their proximity in three-dimensional space.<n>By conditioning the masking distribution on residue contacts, Bucket Masking shifts the learning objective toward modeling long-range interactions.<n>Across four downstream protein fitness prediction tasks, Bucket Masking enables up to a 14% improvement over standard random masking.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Masked language modeling (MLM) is the standard objective for training protein language models, typically implemented by randomly masking individual residues at a fixed rate (e.g., 15%). This practice implicitly assumes that all sequence positions contribute equally to representation learning. In downstream fitness prediction tasks, however, protein sequences are governed by three-dimensional structural dependencies and long-range residue contacts that induce strong nonlocal couplings between residues. We introduce Bucket Masking, a structure-aware masking strategy that selects groups of residues based on their proximity in three-dimensional space, preferentially masking structurally coupled regions during training. By conditioning the masking distribution on residue contacts, Bucket Masking shifts the learning objective toward modeling long-range interactions that are critical for protein function. Across four downstream protein fitness prediction tasks, Bucket Masking enables up to a 14% improvement over standard random masking, excelling at predicting higher-order mutational interactions. Through controlled ablations, we show that these improvements arise from mask placement rather than span size, establishing masking as a positional inductive bias.

Related papers

• Learning Task-Specific Antibody Representations via Function-Aware Masking [0.0]
  We introduce function-aware masking, a family of pretraining algorithms that align mask placement with specific functional priors.<n>We show that these specialist masking strategies significantly improve performance on their respective objectives.<n>Our results demonstrate that informed mask placement provides a parameter-free mechanism for imposing functional inductive biases in antibody language model training.
  arXiv  Detail & Related papers  (2026-09-01T00:37:56Z)
• When Do Concepts Become Functionally Sufficient During Language-Model Training? [5.287915290004618]
  We decompose activations, select sparse soft masks, and inject reconstructions into the model.<n>We compare sufficiency for activation reconstruction, linear decodability, true downstream preservation, and checkpoint transfer.<n>At the shared fixed-penalty operating point across seven models, downstream masks retain substantially less soft mass than reconstruction masks.
  arXiv  Detail & Related papers  (2026-08-15T16:59:57Z)
• Adaptive Order Policies for Masked Diffusion [56.27207032553083]
  We propose a scheme for learning the unmasking order using an additional lightweight policy network on top of a diffusion model.<n>Our proposed loss reweights terms in the masked diffusion loss according to policy probabilities, and results in a policy that prefers positions where the denoiser is more likely to be correct.
  arXiv  Detail & Related papers  (2026-05-29T19:26:53Z)
• Observation-Aligned Mask Priors for Learning Physical Dynamics from Authentic Occlusions [5.712248310668237]
  Observation-Aligned Mask Priors is a framework that learns the distribution of authentic observation masks.<n>We show that this intersection-based partitioning gives every valid dimension a strictly positive probability of being queried.
  arXiv  Detail & Related papers  (2026-05-16T05:23:49Z)
• GenMask: Adapting DiT for Segmentation via Direct Mask Generation [81.54526445834294]
  We argue that instead of indirect adaptation, segmentation tasks should be trained directly in a generative manner.<n>We introduce timesteps sampling strategy for binary masks that emphasizes extreme noise levels for segmentation and moderate noise for image generation.<n>We present GenMask, a DiT trains to generate black-and-white segmentation masks as well as colorful images in RGB space under the original generative objective.
  arXiv  Detail & Related papers  (2026-03-25T03:52:05Z)
• A Random Matrix Theory of Masked Self-Supervised Regression [16.836043197411378]
  We show how training aggregates predictions across many masking patterns, giving rise to a joint, matrix-valued predictor.<n>This object encodes how coordinates condition on one another and poses new analytical challenges.<n>We identify structured regimes in which masked self-supervised learning provably outperforms PCA.
  arXiv  Detail & Related papers  (2026-01-30T17:32:33Z)
• S$^2$Drug: Bridging Protein Sequence and 3D Structure in Contrastive Representation Learning for Virtual Screening [72.89086338778098]
  We propose a two-stage framework for protein-ligand contrastive representation learning.<n>In the first stage, we perform protein sequence pretraining on ChemBL using an ESM2-based backbone.<n>In the second stage, we fine-tune on PDBBind by fusing sequence and structure information through a residue-level gating module.<n>This auxiliary task guides the model to accurately localize binding residues within the protein sequence and capture their 3D spatial arrangement.
  arXiv  Detail & Related papers  (2025-11-10T11:57:47Z)
• Soften the Mask: Adaptive Temporal Soft Mask for Efficient Dynamic Facial Expression Recognition [4.151073288078749]
  Dynamic Facial Expression Recognition (DFER) facilitates the understanding of psychological intentions through non-verbal communication.<n>Existing methods struggle to manage irrelevant information, such as background noise and redundant semantics, which impacts both efficiency and effectiveness.<n>We propose a novel supervised temporal soft masked autoencoder network for DFER, namely AdaTosk, which integrates a parallel supervised classification branch with the self-supervised reconstruction branch.
  arXiv  Detail & Related papers  (2025-02-28T12:45:08Z)
• A Theoretical Analysis of Self-Supervised Learning for Vision Transformers [66.08606211686339]
  Masked autoencoders (MAE) and contrastive learning (CL) capture different types of representations.<n>We study the training dynamics of one-layer softmax-based vision transformers (ViTs) on both MAE and CL objectives.
  arXiv  Detail & Related papers  (2024-03-04T17:24:03Z)
• FoldToken: Learning Protein Language via Vector Quantization and Beyond [56.19308144551836]
  We introduce textbfFoldTokenizer to represent protein sequence-structure as discrete symbols. We refer to the learned symbols as textbfFoldToken, and the sequence of FoldTokens serves as a new protein language.
  arXiv  Detail & Related papers  (2024-02-04T12:18:51Z)
• Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model [93.9943278892735]
  Key problem in protein sequence representation learning is to capture the co-evolutionary information reflected by the inter-residue co-variation in the sequences. We propose a novel method to capture this information directly by pre-training via a dedicated language model, i.e., Pairwise Masked Language Model (PMLM) Our result shows that the proposed method can effectively capture the interresidue correlations and improves the performance of contact prediction by up to 9% compared to the baseline.
  arXiv  Detail & Related papers  (2021-10-29T04:01:32Z)
• Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model [57.77981008219654]
  Masked Language Model (MLM) framework has been widely adopted for self-supervised language pre-training. We propose a fully-explored masking strategy, where a text sequence is divided into a certain number of non-overlapping segments.
  arXiv  Detail & Related papers  (2020-10-12T21:28:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.