Vision-Guided Text Prompt Tuning for Multimodal Sentiment Analysis

• URL: http://arxiv.org/abs/2609.06497v1
• Date: Sun, 06 Sep 2026 09:25:32 GMT
• Title: Vision-Guided Text Prompt Tuning for Multimodal Sentiment Analysis
• Abstract summary: Vision-Guided Text Prompt Tuning (VG-TPT) formulates visual-text sentiment modeling as controllable visual calibration of frozen text representations.<n> Experiments on CMU-MOSEI and CMU-MOSI show that VG-TPT consistently improves over text-only baselines.
• Score: 15.965349585231932
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal sentiment analysis requires effective modeling of both verbal semantics and non-verbal affective cues. A central challenge is to calibrate text-centered sentiment understanding with visual facial evidence in a controlled, adaptive, and parameter-efficient manner. Text usually serves as the semantic anchor, whereas visual cues provide complementary evidence for ambiguous or implicit expressions; however, indiscriminate fusion may introduce visual noise and distort textual semantics. Moreover, fully fine-tuning large visual and textual encoders is costly and prone to overfitting on limited and scenario-dependent MSA benchmarks. To address these issues, we propose Vision-Guided Text Prompt Tuning (VG-TPT), which formulates visual-text sentiment modeling as controllable visual calibration of frozen text representations. VG-TPT injects visual affective cues into a frozen BERT encoder through layer-wise adaptive prompts, rather than relying on late-stage feature fusion or full backbone tuning. A co-guided router composes prompts from a trainable prompt bank according to both the evolving text state and the visual guidance feature, enabling sample-specific and layer-specific modulation. Experiments on CMU-MOSEI and CMU-MOSI show that VG-TPT consistently improves over text-only baselines and achieves competitive or superior performance compared with several full-modality methods, while updating only 2.4M trainable parameters. The code is available at https://github.com/ma-tubu/VG-TPT.

Related papers

• OPUS: A Simple yet Effective Unified Framework for Open-Vocabulary Detection [9.87006529425909]
  We present OPUS, a unified detector supporting text, interactive visual, generic visual, and mixed prompting within one framework.<n> OPUS achieves state-of-the-art Visual-I performance, reaching 68.1/69.2/54.7 AP, while maintaining balanced Text and Visual-G accuracy.
  arXiv  Detail & Related papers  (2026-08-31T04:57:12Z)
• VirPro: Visual-referred Probabilistic Prompt Learning for Weakly-Supervised Monocular 3D Detection [12.835071167163607]
  VirPro is an adaptive multi-modal pretraining paradigm that can be seamlessly integrated into diverse weakly supervised monocular 3D detection frameworks.<n>We generate a diverse set of learnable, instance-conditioned prompts across scenes and store them in an Adaptive Prompt Bank (APB)
  arXiv  Detail & Related papers  (2026-03-18T08:23:55Z)
• Mostly Text, Smart Visuals: Asymmetric Text-Visual Pruning for Large Vision-Language Models [51.475129418355465]
  We propose a simple yet effective Asymmetric Text-Visual Weight Pruning method for Large Vision-Language Models.<n>ATV-Pruning integrates two primary innovations: first, a calibration pool is adaptively constructed by drawing on all textual tokens and a subset of visual tokens.
  arXiv  Detail & Related papers  (2026-03-16T23:23:05Z)
• VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text? [51.02924254085878]
  Vision-Language Models (VLMs) have achieved impressive performance in cross-modal understanding across textual and visual inputs.<n>We introduce VISTA-Bench, a benchmark from multimodal perception, reasoning, to unimodal understanding domains.
  arXiv  Detail & Related papers  (2026-02-04T17:48:55Z)
• Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models [57.357091028792325]
  Vision-language models (VLMs) pre-trained on web-scale data exhibit promising zero-shot generalization but often suffer from semantic misalignment.<n>We propose a novel constrained prompt enhancement (CPE) method to improve visual-textual alignment.<n>Our approach consists of two key components: Topology-Guided Synonymous Semantic Generation (TGSSG) and Category-Agnostic Discriminative Region Selection (CADRS)
  arXiv  Detail & Related papers  (2025-08-24T15:45:22Z)
• Prompt Disentanglement via Language Guidance and Representation Alignment for Domain Generalization [75.88719716002014]
  Domain Generalization (DG) seeks to develop a versatile model capable of performing effectively on unseen target domains.<n>Recent advances in pre-trained Visual Foundation Models (VFMs) have demonstrated considerable potential in enhancing the generalization capabilities of deep learning models.<n>We propose addressing this challenge by leveraging the controllable and flexible language prompt of the VFM.
  arXiv  Detail & Related papers  (2025-07-03T03:52:37Z)
• ControlThinker: Unveiling Latent Semantics for Controllable Image Generation through Visual Reasoning [76.2503352325492]
  ControlThinker is a novel framework that employs a "comprehend-then-generate" paradigm.<n>Latent semantics from control images are mined to enrich text prompts.<n>This enriched semantic understanding then seamlessly aids in image generation without the need for additional complex modifications.
  arXiv  Detail & Related papers  (2025-06-04T05:56:19Z)
• Contextualized Diffusion Models for Text-Guided Image and Video Generation [67.69171154637172]
  Conditional diffusion models have exhibited superior performance in high-fidelity text-guided visual generation and editing. We propose a novel and general contextualized diffusion model (ContextDiff) by incorporating the cross-modal context encompassing interactions and alignments between text condition and visual sample. We generalize our model to both DDPMs and DDIMs with theoretical derivations, and demonstrate the effectiveness of our model in evaluations with two challenging tasks: text-to-image generation, and text-to-video editing.
  arXiv  Detail & Related papers  (2024-02-26T15:01:16Z)
• Unified Vision and Language Prompt Learning [86.1530128487077]
  We present a systematic study on two representative prompt tuning methods, namely text prompt tuning and visual prompt tuning. A major finding is that text prompt tuning fails on data with high intra-class visual variances while visual prompt tuning cannot handle low inter-class variances. To combine the best from both worlds, we propose a simple approach called Unified Prompt Tuning (UPT), which essentially learns a tiny neural network to jointly optimize prompts across different modalities.
  arXiv  Detail & Related papers  (2022-10-13T17:50:24Z)
• Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model [39.722927180264584]
  We propose a novel Dual-modality Prompt Tuning (DPT) paradigm through learning text and visual prompts simultaneously. To make the final image feature concentrate more on the target visual concept, a Class-Aware Visual Prompt Tuning scheme is proposed.
  arXiv  Detail & Related papers  (2022-08-17T15:06:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.