Text-Conditional JEPA for Learning Semantically Rich Visual Representations

• URL: http://arxiv.org/abs/2605.03245v1
• Date: Tue, 05 May 2026 00:26:57 GMT
• Title: Text-Conditional JEPA for Learning Semantically Rich Visual Representations
• Abstract summary: We propose Text-Conditional JEPA that uses image captions to reduce the prediction uncertainty.<n>We show TC-JEPA improves downstream performance and training stability, with promising scaling properties.<n> TC-JEPA also offers a new vision-language pretraining paradigm based on feature prediction only, outperforming contrastive methods on diverse tasks.
• Score: 18.194102334040817
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image-based Joint-Embedding Predictive Architecture (I-JEPA) offers a promising approach to visual self-supervised learning through masked feature prediction. However with the inherent visual uncertainty at masked positions, feature prediction remains challenging and may fail to learn semantic representations. In this work, we propose Text-Conditional JEPA (TC-JEPA) that uses image captions to reduce the prediction uncertainty. Specifically, we modulate the predicted patch features using a fine-grained text conditioner that computes sparse cross-attention over input text tokens. With such conditioning, patch features become predictable as a function of text, thus are more semantically meaningful. We show TC-JEPA improves downstream performance and training stability, with promising scaling properties. TC-JEPA also offers a new vision-language pretraining paradigm based on feature prediction only, outperforming contrastive methods on diverse tasks, especially those requiring fine-grained visual understanding and reasoning.

Related papers

• AlignJEPA: Predictive Vision-Language Alignment for Remote Sensing Foundation Models [17.02495765712523]
  AlignJEPA is a JEPA-inspired predictive vision-language alignment framework for remote sensing foundation models.<n>We train and evaluate AlignJEPA on BigEarthNet.txt for natural-language Sentinel retrieval, evaluate cross-dataset adaptation on RSICD, and use RSVQA only as a closed-set representation probe.
  arXiv  Detail & Related papers  (2026-08-16T00:27:07Z)
• LaCoVL-FER: Landmark-Guided Contrastive Learning Network with Vision-Language Enhancement for Facial Expression Recognition [51.70817823155725]
  We propose a novel landmark-guided contrastive learning network with vision-language enhancement for Facial Expression Recognition (FER)<n>LaCoVL-FER integrates geometric priors from facial landmarks and semantic priors from a vision-language model.<n>Experiments demonstrate that LaCoVL-FER outperforms state-of-the-art methods on three representative real-world FER datasets.
  arXiv  Detail & Related papers  (2026-05-19T13:15:41Z)
• JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generation [10.00677022779314]
  We propose a unified multimodal framework that encodes images and captions into discrete visual and textual tokens.<n>We incorporate cross-attention after the feature predictor for conditional denoising while maintaining a task-agnostic backbone.<n>Our approach shows that late architectural fusion combined with objective-level alignment offers an effective balance between conditioning strength and backbone generality in token-based T2I.
  arXiv  Detail & Related papers  (2025-10-01T14:51:10Z)
• Descriminative-Generative Custom Tokens for Vision-Language Models [101.40245125955306]
  This paper explores the possibility of learning custom tokens for representing new concepts in Vision-Language Models (VLMs)<n>Our aim is to learn tokens that can be effective for both discriminative and generative tasks while composing well with words to form new input queries.
  arXiv  Detail & Related papers  (2025-02-17T18:13:42Z)
• Enhancing JEPAs with Spatial Conditioning: Robust and Efficient Representation Learning [7.083341587100975]
  Image-based Joint-Embedding Predictive Architecture (IJEPA) offers an attractive alternative to Masked Autoencoder (MAE) IJEPA drives representations to capture useful semantic information by predicting in latent rather than input space. Our "conditional" encoders show performance gains on several image classification benchmark datasets.
  arXiv  Detail & Related papers  (2024-10-14T17:46:24Z)
• FILS: Self-Supervised Video Feature Prediction In Semantic Language Space [11.641926922266347]
  This paper demonstrates a self-supervised approach for learning semantic video representations. We present FILS, a novel self-supervised video Feature prediction In semantic Language Space.
  arXiv  Detail & Related papers  (2024-06-05T16:44:06Z)
• SILC: Improving Vision Language Pretraining with Self-Distillation [113.50400246862056]
  We introduce SILC, a novel framework for vision language pretraining. SILC improves image-text contrastive learning with the simple addition of local-to-global correspondence learning by self-distillation. We show that distilling local image features from an exponential moving average (EMA) teacher model significantly improves model performance on dense predictions tasks like detection and segmentation.
  arXiv  Detail & Related papers  (2023-10-20T08:44:47Z)
• CPL: Counterfactual Prompt Learning for Vision and Language Models [76.18024920393245]
  This paper presents a novel underlinetextbfCounterfactual underlinetextbfPrompt underlinetextbfLearning (CPL) method for vision and language models. CPL simultaneously employs counterfactual generation and contrastive learning in a joint optimization framework. Experiments demonstrate that CPL can obtain superior few-shot performance on different vision and language tasks.
  arXiv  Detail & Related papers  (2022-10-19T08:06:39Z)
• CLIP-PAE: Projection-Augmentation Embedding to Extract Relevant Features for a Disentangled, Interpretable, and Controllable Text-Guided Face Manipulation [5.163945166074474]
  Contrastive Language-Image Pre-Training (CLIP) bridges images and text by embedding them into a joint latent space.<n>Due to the discrepancy between image and text embeddings in the joint space, using text embeddings as the optimization target often introduces undesired artifacts in the resulting images.<n>We introduce CLIP Projection-Augmentation Embedding (PAE) as an optimization target to improve the performance of text-guided image manipulation.
  arXiv  Detail & Related papers  (2022-10-08T05:12:25Z)
• Vision-Language Pre-Training for Boosting Scene Text Detectors [57.08046351495244]
  We specifically adapt vision-language joint learning for scene text detection. We propose to learn contextualized, joint representations through vision-language pre-training. The pre-trained model is able to produce more informative representations with richer semantics.
  arXiv  Detail & Related papers  (2022-04-29T03:53:54Z)
• DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting [91.56988987393483]
  We present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP. Specifically, we convert the original image-text matching problem in CLIP to a pixel-text matching problem and use the pixel-text score maps to guide the learning of dense prediction models. Our method is model-agnostic, which can be applied to arbitrary dense prediction systems and various pre-trained visual backbones.
  arXiv  Detail & Related papers  (2021-12-02T18:59:32Z)
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters [84.88106370842883]
  We show that there is an alternative path to achieve better vision-language models other than prompt tuning.<n>CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending.<n> Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2021-10-09T11:39:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.