Learning Task-Specific Antibody Representations via Function-Aware Masking

• URL: http://arxiv.org/abs/2609.00518v1
• Date: Tue, 01 Sep 2026 00:37:56 GMT
• Title: Learning Task-Specific Antibody Representations via Function-Aware Masking
• Abstract summary: We introduce function-aware masking, a family of pretraining algorithms that align mask placement with specific functional priors.<n>We show that these specialist masking strategies significantly improve performance on their respective objectives.<n>Our results demonstrate that informed mask placement provides a parameter-free mechanism for imposing functional inductive biases in antibody language model training.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Antibody-specific language models pretrained via masked language modeling (MLM) learn representations that are critical for downstream sequence design and property prediction tasks. Yet, the corruption process itself is rarely leveraged as a source of inductive bias during pretraining. While preferentially masking complementarity-determining regions (CDRs) improves binding-related predictions, antibodies possess diverse biological priors over a variety of functions. Herein, we introduce function-aware masking, a family of pretraining algorithms that align mask placement with specific functional priors (e.g., from IMGT annotations or structure predictions) to shape the learned representation space. We show that these specialist masking strategies significantly improve performance on their respective objectives, yielding up to a 14% gain on structure-related tasks and up to a 5.9x improvement on CDR-related tasks. To further improve performance across multiple functional axes, we develop hybrid masking strategies that integrate multiple priors, balancing reconstruction over binding, structural, and biophysical objectives. Our results demonstrate that informed mask placement provides a parameter-free mechanism for imposing functional inductive biases in antibody language model training.

Related papers

• Evolution-Aware MSA Reasoning for Subsampling via Factor Graphs [55.715754117956514]
  Multiple Sequence Alignments provide protein language models with explicit evolutionary context.<n>MSA subsampling is unavoidable under limited token budgets.<n>We introduce AP-REASONER, which infers a fixed-budget MSA subset using factor-graph reasoning.
  arXiv  Detail & Related papers  (2026-07-24T13:55:21Z)
• A Unified Framework for In-Context Learning with Causal and Masked Language Models [9.893068784551879]
  In-context learning (ICL) has emerged as a central capability of pretrained language models.<n>Masked language models recover masked tokens from bidirectional context.<n>ICL behavior is not specific to causal language models.
  arXiv  Detail & Related papers  (2026-07-05T02:23:18Z)
• Structure-Aware Masking for Protein Representation Learning [0.0]
  We introduce Bucket Masking, a structure-aware masking strategy that selects groups of residues based on their proximity in three-dimensional space.<n>By conditioning the masking distribution on residue contacts, Bucket Masking shifts the learning objective toward modeling long-range interactions.<n>Across four downstream protein fitness prediction tasks, Bucket Masking enables up to a 14% improvement over standard random masking.
  arXiv  Detail & Related papers  (2026-05-15T19:36:54Z)
• AntigenLM: Structure-Aware DNA Language Modeling for Influenza [5.938702748853349]
  We present AntigenLM, a generative DNA language model pretrained on influenza genomes with intact, aligned functional units.<n>AntigenLM accurately forecasts future antigenic variants across regions and subtypes, including those unseen during training.<n>It also achieves near-perfect subtype classification.
  arXiv  Detail & Related papers  (2026-02-09T08:52:04Z)
• Understanding the Transfer Limits of Vision Foundation Models [38.99867932557529]
  Foundation models leverage large-scale pretraining to capture extensive knowledge, demonstrating generalization in a wide range of language tasks.<n>We postulate that this limitation arises from a mismatch between pretraining objectives and the demands of downstream vision-and-imaging tasks.<n>Pretraining strategies like masked image reconstruction or contrastive learning shape representations for tasks such as recovery of generic visual patterns or global semantic structures.<n>Our findings indicate that better alignment between pretraining and downstream tasks, measured by simple divergence metrics such as maximum-mean-discrepancy (MMD) between the same features before and after fine-tuning, correlates with greater performance improvements and
  arXiv  Detail & Related papers  (2026-01-22T12:07:56Z)
• Bridging the Task Gap: Multi-Task Adversarial Transferability in CLIP and Its Derivatives [61.58574200236532]
  Adversarial examples generated from fine-grained tasks often exhibit stronger transfer potential than those from coarse-grained tasks.<n>We propose a novel framework, Multi-Task Adversarial CLIP (MT-AdvCLIP), which introduces a task-aware feature aggregation loss and generates perturbations with enhanced cross-task generalization capability.
  arXiv  Detail & Related papers  (2025-09-28T14:46:52Z)
• Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction [88.65168366064061]
  We introduce Discrete Denoising Posterior Prediction (DDPP), a novel framework that casts the task of steering pre-trained MDMs as a problem of probabilistic inference. Our framework leads to a family of three novel objectives that are all simulation-free, and thus scalable. We substantiate our designs via wet-lab validation, where we observe transient expression of reward-optimized protein sequences.
  arXiv  Detail & Related papers  (2024-10-10T17:18:30Z)
• Enhancing Few-shot CLIP with Semantic-Aware Fine-Tuning [61.902254546858465]
  Methods based on Contrastive Language-Image Pre-training have exhibited promising performance in few-shot adaptation tasks. We propose fine-tuning the parameters of the attention pooling layer during the training process to encourage the model to focus on task-specific semantics.
  arXiv  Detail & Related papers  (2023-11-08T05:18:57Z)
• Reprogramming Pretrained Language Models for Antibody Sequence Infilling [72.13295049594585]
  Computational design of antibodies involves generating novel and diverse sequences, while maintaining structural consistency. Recent deep learning models have shown impressive results, however the limited number of known antibody sequence/structure pairs frequently leads to degraded performance. In our work we address this challenge by leveraging Model Reprogramming (MR), which repurposes pretrained models on a source language to adapt to the tasks that are in a different language and have scarce data.
  arXiv  Detail & Related papers  (2022-10-05T20:44:55Z)
• Pre-training Text Representations as Meta Learning [113.3361289756749]
  We introduce a learning algorithm which directly optimize model's ability to learn text representations for effective learning of downstream tasks. We show that there is an intrinsic connection between multi-task pre-training and model-agnostic meta-learning with a sequence of meta-train steps.
  arXiv  Detail & Related papers  (2020-04-12T09:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.