Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning

• URL: http://arxiv.org/abs/2607.03143v1
• Date: Fri, 03 Jul 2026 09:32:01 GMT
• Title: Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning
• Abstract summary: Vision-language alignment powers open-vocabulary recognition, retrieval, and LVLM grounding, yet natural captions are often underspecified.<n>We propose Text as Partial Constraint, a core-residual alignment framework that treats multi-view captions as incomplete supervision.<n>Across zero-shot recognition and adversarial robustness, TPC achieves 81.42/64.05 Top-1 clean/robust accuracy on ImageNet and 76.19/52.03 on an Avg-14 transfer suite.
• Score: 9.515040235138084
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language alignment powers open-vocabulary recognition, retrieval, and LVLM grounding, yet natural captions are often underspecified, making similarity brittle and overly confident under paraphrase and omitted details. We aim to learn representations whose matching is stable across caption views and whose confidence reflects how strongly text constrains an image. We propose Text as Partial Constraint (TPC), a core-residual alignment framework that treats multi-view captions as incomplete supervision. It distills a consensus semantic core as the alignment target, learns a single-view core predictor for standard inference with one query, and explicitly discourages vision-language similarity from depending on the orthogonal unsaid residual. An uncertainty-aware contrastive objective further softens alignment when caption views disagree, reducing overconfident updates under weak language constraints. Across zero-shot recognition and adversarial robustness, TPC achieves 81.42/64.05 Top-1 clean/robust accuracy on ImageNet and 76.19/52.03 on an Avg-14 transfer suite, while improving LVLM transfer with 85.16 POPE F1 and 59.57 OKVQA accuracy under an LLaVA-1.5-7B stack. These results suggest that modeling text as a partial constraint is a practical and principled route to more reliable vision-language representations under underspecified language supervision.

Related papers

• Controllable Image Captioning with Prompt-Conditioned Scene Rewards [13.154350707762179]
  Fine-grained Captioning Control Using Scene Rewards (FoCUS) is a controllable image captioning method that lets users steer captions toward specific semantic emphases.<n>FoCUS consistently improves controllability and fine-grained caption quality without degrading general caption performance.
  arXiv  Detail & Related papers  (2026-09-01T04:36:23Z)
• Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models [52.78477729846771]
  We introduce COAST (COntrastive Adaptive Semantic Token Pruning), a training-free pruning framework that casts compression as adaptive semantic routing.<n> COAST consistently outperforms strong pruning baselines across token budgets and generalizes across multiple LVLM families.
  arXiv  Detail & Related papers  (2026-05-10T09:07:04Z)
• LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment [11.785675631199455]
  In fine-grained settings, the relevant evidence often lies in localized parts, attributes, or textures rather than in the full image.<n>Recent localized visual-text alignment methods address this by comparing class descriptions with multiple image regions.<n>We propose LAGO, a framework for efficient and robust zero-shot localized visual-text alignment.
  arXiv  Detail & Related papers  (2026-05-04T09:07:00Z)
• InterPartAbility: Text-Guided Part Matching for Interpretable Person Re-Identification [15.419626926769595]
  TI-ReID relies on natural-language text description to retrieve top matching individuals from a large gallery of images.<n>Existing interpretability approaches in TI-ReID rely solely on slot-attention to highlight attended regions.<n>This paper introduces InterPart, an interpretable TI-ReID method that performs explicit part-wise matching.
  arXiv  Detail & Related papers  (2026-04-29T19:18:36Z)
• Enhancing CLIP Robustness via Cross-Modality Alignment [54.01929554563447]
  We propose Cross-modality Alignment, an optimal transport-based framework for vision-language models.<n> COLA restores global image-text alignment and local structural consistency in the feature space.<n> COLA is training-free and compatible with existing fine-tuned models.
  arXiv  Detail & Related papers  (2025-10-28T03:47:44Z)
• Weakly-Supervised Image Forgery Localization via Vision-Language Collaborative Reasoning Framework [16.961220047066792]
  ViLaCo is a vision-language collaborative reasoning framework that introduces auxiliary semantic supervision distilled from pre-trained vision-language models.<n>ViLaCo substantially outperforms existing WSIFL methods, achieving state-of-the-art performance in both detection and localization accuracy.
  arXiv  Detail & Related papers  (2025-08-02T12:14:29Z)
• Collaborative Temporal Consistency Learning for Point-supervised Natural Language Video Localization [129.43937834515688]
  We propose a new COllaborative Temporal consistEncy Learning (COTEL) framework to strengthen the video-language alignment.<n>Specifically, we first design a frame- and a segment-level Temporal Consistency Learning (TCL) module that models semantic alignment across frame saliencies and sentence-moment pairs.
  arXiv  Detail & Related papers  (2025-03-22T05:04:12Z)
• Contrastive Vision-Language Alignment Makes Efficient Instruction Learner [31.281236193979165]
  We study the task of extending the large language model (LLM) into a vision-language instruction-following model. Existing methods typically train a visual adapter to align the representation between a pre-trained vision transformer (ViT) and the LLM by a generative image captioning loss. We propose CG-VLM that applies Contrastive and Generative alignment objectives to effectively align the representation of ViT and LLM.
  arXiv  Detail & Related papers  (2023-11-29T03:29:46Z)
• Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation [100.81837601210597]
  We propose Concept Curation (CoCu) to bridge the gap between visual and textual semantics in pre-training data. CoCu achieves superb zero-shot transfer performance and greatly boosts language-supervised segmentation baseline by a large margin.
  arXiv  Detail & Related papers  (2023-09-24T00:05:39Z)
• Weakly Supervised Vision-and-Language Pre-training with Relative Representations [76.63610760577214]
  Weakly supervised vision-and-language pre-training has been shown to effectively reduce the data cost of pre-training. Current methods use only local descriptions of images, i.e., object tags, as cross-modal anchors to construct weakly-aligned image-text pairs for pre-training.
  arXiv  Detail & Related papers  (2023-05-24T18:10:24Z)
• Simple Token-Level Confidence Improves Caption Correctness [117.33497608933169]
  Token-Level Confidence, or TLC, is a simple yet surprisingly effective method to assess caption correctness. We fine-tune a vision-language model on image captioning, input an image and proposed caption to the model, and aggregate token confidences over words or sequences to estimate image-caption consistency.
  arXiv  Detail & Related papers  (2023-05-11T17:58:17Z)
• CyCLIP: Cyclic Contrastive Language-Image Pretraining [34.588147979731374]
  Recent advances in contrastive representation learning over paired image-text data have led to models such as CLIP that achieve state-of-the-art performance for zero-shot classification and distributional robustness. We demonstrate that the image and text representations learned via a standard contrastive objective are not interchangeable and can lead to inconsistent downstream predictions. We propose CyCLIP, a framework for contrastive representation learning that explicitly optimize for the learned representations to be geometrically consistent in the image and text space.
  arXiv  Detail & Related papers  (2022-05-28T15:31:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.