Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning

• URL: http://arxiv.org/abs/2606.05173v1
• Date: Thu, 16 Apr 2026 22:26:47 GMT
• Title: Predict and Reconstruct: Joint Objectives for Self-Supervised Language Representation Learning
• Abstract summary: Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT.<n>Inspired by the success of Joint Embedding Predictive Architectures (JEPA) in vision and audio, we propose a hybrid objective.<n>We pre-train both a hybrid model and a pure-MLM baseline on English Wikipedia using identical architectures and compute budgets.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Masked language modelling (MLM) has been the dominant pre-training objective for text encoders since BERT, yet it encourages representations that are strongly anchored to surface-form token identity rather than deeper semantic structure. Inspired by the success of Joint Embedding Predictive Architectures (JEPA) (LeCun, 2022) in vision and audio, we propose a hybrid pre-training objective that combines a JEPA-style latent-space prediction loss with a standard MLM objective over a single shared encoder. A learnable scalar parameter continuously balances the two objectives during training. We pre-train both a hybrid model and a pure-MLM baseline on English Wikipedia using identical architectures and compute budgets (NVIDIA H100). Extensive representation analysis across five GLUE benchmarks (SST-2, MRPC, MNLI, CoLA, STS-B) using four pooling strategies reveals that the hybrid encoder produces significantly more uniform embeddings (uniformity less than -0.16 vs -0.05 for MLM), exhibits richer spectral geometry under max pooling, encodes less surface-level lexical information, and achieves a better semantic-to-lexical balance. Despite similar linear-probe downstream accuracy, the geometric differences are consistent and significant, suggesting that the JEPA predictive objective reshapes the latent space in ways that standard accuracy metrics alone cannot capture.

Related papers

• Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs [18.457250148137494]
  We present an Adaptive Depth Sparse Framework (AdaDSF) that converts off-the-shelf pre-trained large language models into depth-sparse models without retraining.<n>On GPT-NeoX and Qwen2.5 over language modeling and commonsense reasoning, AdaDSF substantially reduces inference FLOPs while preserving performance close to dense counterparts.
  arXiv  Detail & Related papers  (2026-07-23T13:13:04Z)
• HQ-JEPA: Hybrid Quantum Joint-Embedding Predictive Architecture for Cross-Modal Remote Sensing Representation Learning [17.65267962254814]
  HQ-JEPA is a hybrid quantum-classical joint-embedding predictive architecture for cross-modal remote sensing representation learning.<n>Results show that HQ-JEPA achieves competitive and often superior performance over strong self-supervised and remote sensing foundation-model baselines.
  arXiv  Detail & Related papers  (2026-05-29T09:37:35Z)
• Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation [3.0739122323721766]
  MoA-DepthCLIP adapts pretrained CLIP representations for monocular depth estimation with minimal supervision.<n>Our method integrates a lightweight Mixture-of-Adapters (MoA) module into the pretrained Vision Transformer (ViT-B/32) backbone.<n>On the NYU Depth V2 benchmark, MoA-DepthCLIP achieves competitive results, significantly outperforming the DepthCLIP baseline.
  arXiv  Detail & Related papers  (2026-04-01T16:41:04Z)
• ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model [53.15040805435013]
  Vision-language models (VLMs) provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames.<n>We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance.
  arXiv  Detail & Related papers  (2026-03-23T17:59:42Z)
• SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy [12.070587084660096]
  We present an original, PhD-level multimodal benchmark specifically designed for scanning probe microscopy (SPM)<n>By employing Anchor-Gated Sieve (AGS) technology, we efficiently extract high-value image-text pairs from arXiv and journal papers published between 2023 and 2025.<n>Our pipeline achieves extreme token savings while maintaining high dataset purity.
  arXiv  Detail & Related papers  (2026-02-26T13:08:56Z)
• VL-JEPA: Joint Embedding Predictive Architecture for Vision-language [54.86811250366009]
  We introduce VL-JEPA, a vision-language model built on a Joint Embedding Predictive Architecture (JEPA)<n>By learning in an abstract representation space, the model focuses on task-relevant semantics while abstracting away surface-level linguistic variability.<n>At inference time, a lightweight text decoder is invoked only when needed to translate VL-JEPA predicted embeddings into text.
  arXiv  Detail & Related papers  (2025-12-11T18:59:22Z)
• LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics [53.247652209132376]
  Joint-Embedding Predictive Architectures (JEPAs) offer a promising blueprint, but lack of practical guidance and theory has led to ad-hoc R&D.<n>We present a comprehensive theory of JEPAs and instantiate it in bf LeJEPA, a lean, scalable, and theoretically grounded training objective.
  arXiv  Detail & Related papers  (2025-11-11T18:21:55Z)
• Enhancing DNA Foundation Models to Address Masking Inefficiencies [18.54660252939211]
  We propose a modified encoder-decoder architecture based on the masked autoencoder framework.<n>We evaluate our approach on the BIOSCAN-5M dataset, comprising over 2 million unique DNA barcodes.
  arXiv  Detail & Related papers  (2025-02-25T17:56:25Z)
• Adaptive Pruning for Large Language Models with Structural Importance Awareness [66.2690963378878]
  Large language models (LLMs) have significantly improved language understanding and generation capabilities.<n>LLMs are difficult to deploy on resource-constrained edge devices due to their high computational and storage resource demands.<n>We propose structurally-aware adaptive pruning (SAAP) to significantly reduce the computational and memory costs while maintaining model performance.
  arXiv  Detail & Related papers  (2024-12-19T18:08:04Z)
• Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models [56.00251589760559]
  Large language models (LLMs) can act as gradient priors in a zero-shot setting.<n>We introduce LM-GC, a novel method that integrates LLMs with arithmetic coding.<n>Experiments indicate that LM-GC surpasses existing state-of-the-art lossless compression methods.
  arXiv  Detail & Related papers  (2024-09-26T13:38:33Z)
• Model Inversion Attacks Through Target-Specific Conditional Diffusion Models [54.69008212790426]
  Model inversion attacks (MIAs) aim to reconstruct private images from a target classifier's training set, thereby raising privacy concerns in AI applications. Previous GAN-based MIAs tend to suffer from inferior generative fidelity due to GAN's inherent flaws and biased optimization within latent space. We propose Diffusion-based Model Inversion (Diff-MI) attacks to alleviate these issues.
  arXiv  Detail & Related papers  (2024-07-16T06:38:49Z)
• How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks [14.338754598043968]
  Two competing paradigms exist for self-supervised learning of data representations. Joint Embedding Predictive Architecture (JEPA) is a class of architectures in which semantically similar inputs are encoded into representations that are predictive of each other.
  arXiv  Detail & Related papers  (2024-07-03T19:43:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.