Contextualizing Biological Language Models across Modalities via Logit-Space Contrastive Alignment

• URL: http://arxiv.org/abs/2606.18703v1
• Date: Wed, 17 Jun 2026 05:30:47 GMT
• Title: Contextualizing Biological Language Models across Modalities via Logit-Space Contrastive Alignment
• Abstract summary: LOGICA is a framework for context-conditioned prediction that performs contrastive learning directly in output-logit space.<n>It is particularly effective for mutation-local variant ranking, where comparisons reduce to context-conditioned likelihoods of mutant tokens at perturbed sites.
• Score: 2.249367818885166
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretrained biological language models expose per-token probability distributions through masked-token prediction, providing the likelihood interface central to sequence design, variant scoring, and mechanistic interpretation. Yet these distributions are learned from broad unlabeled corpora and are not naturally conditioned on task-specific biological contexts such as interaction partners, cellular environments, or therapeutic interventions. Existing contextual matching methods often distort this interface through pooled embeddings, contrastive latent spaces, or task-specific prediction heads. We introduce LOGICA (Logit-space Contrastive Alignment), a framework for context-conditioned prediction that performs contrastive learning directly in output-logit space. Using gated cross-modal adapters compatible with each model's native token head, LOGICA preserves the pretrained likelihood interface and converts contextualized token log-likelihoods into matching scores. Alignment is defined through context-sensitive token probabilities rather than proximity in a shared embedding space, enabling learning from sparse paired data across models with distinct vocabularies, without a shared tokenizer or decoder. LOGICA is particularly effective for mutation-local variant ranking, where comparisons reduce to context-conditioned likelihoods of mutant tokens at perturbed sites. Across protein--ligand binding, TCR--peptide activity, and drug-conditioned resistance prediction, LOGICA improves over prior state-of-the-art methods, including matched latent-contrastive and conditional MLM baselines, while retaining a token-level interface for interpretation and generation. On held-out-gene single-mutation drug-resistance prediction, LOGICA improves AUC from near-random latent-space baselines of $\sim$0.55 to $\sim$0.65.

Related papers

• ProtLingo: Efficient Protein Language Modeling via Conditional Memory and Expert Routing [39.80134572747517]
  Protein language models (PLMs) provide a scalable approach for modeling sequence-function relationships from unlabeled sequences.<n>We introduce ProtLingo, an efficient PLM framework that augments a pretrained single-sequence backbone with conditional local memory and sparse expert routing.<n>ProtLingo achieves competitive performance with a 150M-scale backbone, including strong parameter efficiency on mutation-effect prediction.
  arXiv  Detail & Related papers  (2026-09-04T06:46:42Z)
• Structuring Semantic Embeddings for Principle Evaluation: A Prototype-Guided Contrastive Learning Approach [58.15427952222424]
  General-purpose text embeddings are widely deployed for such tasks, but broad semantic similarity can place semantically similar yet task-distinct examples.<n>We introduce Prototype-Guided Contrastive Learning (PGCL), a prototype-guided geometric regularization module built on top of frozen text embeddings.
  arXiv  Detail & Related papers  (2026-08-15T13:19:25Z)
• Behaviour-Conditioned Neural Processes for Adaptive Residential Short-Term Load Forecasting [1.1240669509034296]
  This work investigates whether inferred behavioural structure can be embedded within a Neural Process-based probabilistic model.<n>We propose a behaviour-conditioned Attentive Neural Process framework that treats each load profile as a forecasting task.<n>Experiments on the Smart Grid, Smart City dataset use user-disjoint train/validation/test splits, variable context lengths, and multi-step forecast horizons.
  arXiv  Detail & Related papers  (2026-07-17T17:48:22Z)
• COAST: Context-Aware Differential Learning for Gene Expression Prediction in Spatial Transcriptomics [2.073620179775842]
  We propose COAST, a context-aware differential learning framework for spatial gene expression prediction.<n> COAST aggregates the target and context spot tokens using a Transformer encoder to capture both fine-grained local patterns and slide-level structure.<n> Experiments on multiple spatial transcriptomics datasets show consistent improvements in correlation- and distribution-based metrics.
  arXiv  Detail & Related papers  (2026-07-10T07:50:23Z)
• Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attention [0.0]
  Zonkey is a hierarchical diffusion model that addresses limitations through a fully trainable pipeline from raw characters to document-level representations.<n>At its core is a differentiable tokenizer that learns probabilistic beginning-of-sequence (BOS) decisions.<n>Zonkey generates coherent, variable-length text from noise, demonstrating emergent hierarchies.
  arXiv  Detail & Related papers  (2026-01-29T14:17:37Z)
• STEAD: Robust Provably Secure Linguistic Steganography with Diffusion Language Model [71.35577462669856]
  We propose a robust, provably secure linguistic steganography with diffusion language models (DLMs)<n>We introduce error correction strategies, including pseudo-random error correction and neighborhood search correction, during steganographic extraction.
  arXiv  Detail & Related papers  (2026-01-21T08:58:12Z)
• Interpretable and Adaptive Node Classification on Heterophilic Graphs via Combinatorial Scoring and Hybrid Learning [1.2691047660244335]
  Graph neural networks (GNNs) achieve strong performance on homophilic graphs but often struggle underphily, where adjacent nodes frequently belong to different classes.<n>We propose an interpretable and adaptive framework for semi-supervised node classification based on explicit inference rather than deep message passing.<n> Experiments on heterophilic and transitional benchmarks demonstrate competitive performance with modern GNNs while offering advantages in interpretability, tuna, and computational efficiency.
  arXiv  Detail & Related papers  (2025-12-22T20:50:44Z)
• A high-capacity linguistic steganography based on entropy-driven rank-token mapping [81.29800498695899]
  Linguistic steganography enables covert communication through embedding secret messages into innocuous texts.<n>Traditional modification-based methods introduce detectable anomalies, while retrieval-based strategies suffer from low embedding capacity.<n>We propose an entropy-driven framework called RTMStega that integrates rank-based adaptive coding and context-aware decompression with normalized entropy.
  arXiv  Detail & Related papers  (2025-10-27T06:02:47Z)
• Framework for Machine Evaluation of Reasoning Completeness in Large Language Models For Classification Tasks [0.0]
  This paper introduces RACE-Reasoning Alignment for Completeness of Explanations.<n>We analyze four widely used text classification datasets-WIKI ONTOLOGY, AG NEWS, IMDB, and GOEMOTIONS.<n>We show that correct predictions exhibit higher coverage of supporting features, while incorrect predictions are associated with elevated coverage of contradicting features.
  arXiv  Detail & Related papers  (2025-10-23T20:22:22Z)
• GenAR: Next-Scale Autoregressive Generation for Spatial Gene Expression Prediction [15.143858141542532]
  GenAR is a multi-scale autoregressive framework that refines predictions from coarse to fine.<n>We introduce GenAR, a multi-scale autoregressive framework that refines predictions from coarse to fine.<n>GenAR achieves principled state-of-the-art performance, offering potential implications for precision medicine and cost-effective molecular profiling.
  arXiv  Detail & Related papers  (2025-10-05T18:28:21Z)
• Following the Autoregressive Nature of LLM Embeddings via Compression and Alignment [84.74716380180428]
  We propose AutoRegEmbed, a contrastive learning method built on embedding conditional probability distributions.<n>We show that our method significantly outperforms traditional contrastive learning approaches.
  arXiv  Detail & Related papers  (2025-02-17T03:36:25Z)
• KL-geodesics flow matching with a novel sampling scheme [4.347494885647007]
  Non-autoregressive language models generate all tokens simultaneously, offering potential speed advantages over traditional autoregressive models.<n>We investigate a conditional flow matching approach for text generation.
  arXiv  Detail & Related papers  (2024-11-25T17:15:41Z)
• Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model [93.9943278892735]
  Key problem in protein sequence representation learning is to capture the co-evolutionary information reflected by the inter-residue co-variation in the sequences. We propose a novel method to capture this information directly by pre-training via a dedicated language model, i.e., Pairwise Masked Language Model (PMLM) Our result shows that the proposed method can effectively capture the interresidue correlations and improves the performance of contact prediction by up to 9% compared to the baseline.
  arXiv  Detail & Related papers  (2021-10-29T04:01:32Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.