Language-Instructed Vision Embeddings for Controllable and Generalizable Perception

• URL: http://arxiv.org/abs/2606.19584v1
• Date: Wed, 17 Jun 2026 20:39:04 GMT
• Title: Language-Instructed Vision Embeddings for Controllable and Generalizable Perception
• Abstract summary: Language-Instructed Vision Embeddings (LIVE) uses language as high-level guidance to produce task-centric embeddings.<n>LIVE reduces visual hallucinations, surpasses vision-language models with orders of magnitude more parameters on visual question answering.
• Score: 21.164230220588873
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision foundation models are typically trained as static feature extractors, placing the burden of task adaptation onto large downstream models. We propose an alternative paradigm: instead of solely feeding visual features into language models, we use language itself to dynamically guide the vision encoder. Our method, Language-Instructed Vision Embeddings (LIVE), leverages language as high-level guidance to produce task-centric embeddings at inference time, removing the need for task-specific retraining. This enables the encoder to focus on contextually relevant aspects of the input, yielding more controllable and generalizable representations. Empirically, LIVE reduces visual hallucinations (+34 points on MMVP), surpasses vision-language models with orders of magnitude more parameters on visual question answering, and generalizes to unseen instructions and tasks -- offering a direct path toward adaptive, instruction-driven visual intelligence.

Related papers

• Boosting Visual Instruction Tuning with Self-Supervised Guidance [25.75897924846719]
  Multimodal large language models (MLLMs) perform well on many vision-centric tasks but often struggle with vision-centric problems.<n>Recent evidence suggests that this limitation arises from under-utilization of visual information during instruction tuning.<n>We propose a simple and lightweight approach that augments visual instruction tuning with a small number of visually grounded self-supervised tasks.
  arXiv  Detail & Related papers  (2026-04-14T16:59:53Z)
• Steerable Visual Representations [72.39044430620977]
  We introduce Steerable Visual Representations, whose global and local features can be steered with natural language.<n>Our method also matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination.
  arXiv  Detail & Related papers  (2026-04-02T17:59:49Z)
• Vega: Learning to Drive with Natural Language Instructions [93.49357278917223]
  We propose a unified Vision-Language-World-Action model, Vega, for instruction-based generation and planning.<n>We employ the autoregressive paradigm to process visual inputs (vision) and language instructions (language) and the diffusion paradigm to generate future predictions.<n>Our method achieves superior planning performance and exhibits strong instruction-following abilities.
  arXiv  Detail & Related papers  (2026-03-26T17:59:56Z)
• Visual Jigsaw Post-Training Improves MLLMs [58.29961336087896]
  We introduce Visual Jigsaw, a generic self-supervised post-training framework designed to strengthen visual understanding in large language models (MLLMs)<n>Visual Jigsaw is formulated as a general ordering task: visual inputs are partitioned, shuffled, and the model must reconstruct the visual information by producing the correct permutation in natural language.<n>Extensive experiments demonstrate substantial improvements in fine-grained perception, temporal reasoning, and 3D spatial understanding.
  arXiv  Detail & Related papers  (2025-09-29T17:59:57Z)
• Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models [1.9253106218929117]
  Multimodal Large Language Models (MLLMs) often fail to fully leverage visual input, defaulting to strong language priors.<n>Our approach first provides insights into how MLLMs internally build visual understanding of image regions and then introduces techniques to amplify this capability.<n>We demonstrate the superior multimodal understanding of our resultant model through a detailed upstream analysis quantifying its ability to predict visually-dependent tokens as well as 10 pt boost on visually challenging tasks.
  arXiv  Detail & Related papers  (2025-05-08T20:04:27Z)
• OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction [95.6266030753644]
  Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions.<n>Existing approaches require fine-tuning pre-trained vision-language models (VLMs) as visual and language features are independently fed into downstream policies.<n>We propose OTTER, a novel VLA architecture that leverages existing alignments through explicit, text-aware visual feature extraction.
  arXiv  Detail & Related papers  (2025-03-05T18:44:48Z)
• ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension [71.03445074045092]
  We propose ClawMachine, offering a new methodology that explicitly notates each entity using token collectives groups of visual tokens.<n>Our method unifies the prompt and answer of visual referential tasks without using additional syntax.<n>ClawMachine achieves superior performance on scene-level and referential understanding tasks with higher efficiency.
  arXiv  Detail & Related papers  (2024-06-17T08:39:16Z)
• EC^2: Emergent Communication for Embodied Control [72.99894347257268]
  Embodied control requires agents to leverage multi-modal pre-training to quickly learn how to act in new environments. We propose Emergent Communication for Embodied Control (EC2), a novel scheme to pre-train video-language representations for few-shot embodied control. EC2 is shown to consistently outperform previous contrastive learning methods for both videos and texts as task inputs.
  arXiv  Detail & Related papers  (2023-04-19T06:36:02Z)
• Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models [57.08925810659545]
  We conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models. Our empirical observations suggest that vision-and-language models are better at label prediction tasks. We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models.
  arXiv  Detail & Related papers  (2022-12-01T05:00:18Z)
• Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics [29.393661499333284]
  We propose to "discretize" the visual representation by joint learning a codebook that imbues each visual token a semantic. We then utilize these discretized visual semantics as self-supervised ground-truths for building our Masked Image Modeling objective. Experiments validate the effectiveness of our approach across common vision-language benchmarks.
  arXiv  Detail & Related papers  (2022-07-31T17:36:09Z)
• Learning Visual Representations with Caption Annotations [19.24013129952071]
  We propose a proxy task to learn visual representations over image-caption pairs. ICMLM consists in predicting masked words in captions by relying on visual cues. Our experiments confirm that image captions can be leveraged to inject global and localized semantic information into visual representations.
  arXiv  Detail & Related papers  (2020-08-04T08:04:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.