TriSPrompt: A Hierarchical Soft Prompt Model for Multimodal Rumor Detection with Incomplete Modalities

• URL: http://arxiv.org/abs/2509.19352v1
• Date: Thu, 18 Sep 2025 02:46:51 GMT
• Title: TriSPrompt: A Hierarchical Soft Prompt Model for Multimodal Rumor Detection with Incomplete Modalities
• Abstract summary: We propose a hierarchical soft prompt model textsfTriSPrompt, which integrates three types of prompts, textiti.e., emphmodality-aware (MA) prompt, emphmodality-missing (MM) prompt, and emphmutual-views (MV) prompt.<n>textsfTriSPrompt achieves an accuracy gain of over 13% compared to state-of-the-art methods.
• Score: 19.93760428537927
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The widespread presence of incomplete modalities in multimodal data poses a significant challenge to achieving accurate rumor detection. Existing multimodal rumor detection methods primarily focus on learning joint modality representations from \emph{complete} multimodal training data, rendering them ineffective in addressing the common occurrence of \emph{missing modalities} in real-world scenarios. In this paper, we propose a hierarchical soft prompt model \textsf{TriSPrompt}, which integrates three types of prompts, \textit{i.e.}, \emph{modality-aware} (MA) prompt, \emph{modality-missing} (MM) prompt, and \emph{mutual-views} (MV) prompt, to effectively detect rumors in incomplete multimodal data. The MA prompt captures both heterogeneous information from specific modalities and homogeneous features from available data, aiding in modality recovery. The MM prompt models missing states in incomplete data, enhancing the model's adaptability to missing information. The MV prompt learns relationships between subjective (\textit{i.e.}, text and image) and objective (\textit{i.e.}, comments) perspectives, effectively detecting rumors. Extensive experiments on three real-world benchmarks demonstrate that \textsf{TriSPrompt} achieves an accuracy gain of over 13\% compared to state-of-the-art methods. The codes and datasets are available at https: //anonymous.4open.science/r/code-3E88.

Related papers

• MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models [70.34265674686516]
  Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space.<n>We introduce MMEB-V3, a benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios.<n>We conduct a systematic analysis of full-modality embeddings and identify three key findings.
  arXiv  Detail & Related papers  (2026-04-25T14:15:05Z)
• Evaluation Before Generation: A Paradigm for Robust Multimodal Sentiment Analysis with Missing Modalities [21.767502810187477]
  The missing modality problem poses a fundamental challenge in multimodal sentiment analysis.<n>Existing approaches primarily improve robustness through prompt learning and pre trained models.<n>A Prompt based Missing Modality Adaptation framework is proposed to address these issues.
  arXiv  Detail & Related papers  (2026-04-07T07:59:06Z)
• OmniRet: Efficient and High-Fidelity Omni Modality Retrieval [51.80205678389465]
  We present OmniRet, the first retrieval model capable of handling complex, composed queries spanning three key modalities: text, vision, and audio.<n>Our model demonstrates significant improvements on composed query, audio and video retrieval tasks, while achieving on-par performance with state-of-the-art models on others.
  arXiv  Detail & Related papers  (2026-03-02T17:19:55Z)
• I$^3$-MRec: Invariant Learning with Information Bottleneck for Incomplete Modality Recommendation [42.37107069331169]
  We introduce textbfI$3$-MRec, which learns with textbfIn bottleneck principle for textbfIncomplete textbfModality textbfRecommendation.<n>I$3$-MRec enforces two pivotal properties: (i) cross-modal preference invariance, and (ii) compact yet effective multimodal representation.<n>I$3$-MRec consistently outperforms existing state-of-the-art MRS methods across various modality-missing scenarios.
  arXiv  Detail & Related papers  (2025-08-06T09:29:50Z)
• MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching [54.740256498985026]
  Keypoint detection and description methods often struggle with multimodal data.<n>We propose a modality-invariant feature learning network (MIFNet) to compute modality-invariant features for keypoint descriptions in multimodal image matching.
  arXiv  Detail & Related papers  (2025-01-20T06:56:30Z)
• Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning [27.867369806400834]
  We propose RAGPT, a novel Retrieval-AuGmented dynamic Prompt Tuning framework.<n>RAGPT comprises three modules: (I) the multi-channel retriever, (II) the missing modality generator, and (III) the context-aware prompter.<n>Experiments conducted on three real-world datasets show that RAGPT consistently outperforms all competitive baselines in handling incomplete modality problems.
  arXiv  Detail & Related papers  (2025-01-02T07:39:48Z)
• Deep Correlated Prompting for Visual Recognition with Missing Modalities [22.40271366031256]
  Large-scale multimodal models have shown excellent performance over a series of tasks powered by the large corpus of paired multimodal training data. However, this simple assumption may not always hold in the real world due to privacy constraints or collection difficulty. We refer to prompt learning to adapt large pretrained multimodal models to handle missing-modality scenarios by regarding different missing cases as different types of input.
  arXiv  Detail & Related papers  (2024-10-09T05:28:43Z)
• Robust Modality-incomplete Anomaly Detection: A Modality-instructive Framework with Benchmark [69.02666229531322]
  We introduce a pioneering study that investigates Modality-Incomplete Industrial Anomaly Detection (MIIAD)<n>We find that most existing MIAD methods perform poorly on the MIIAD Bench, leading to significant performance degradation.<n>We propose a novel two-stage Robust modAlity-aware fusing and Detecting framewoRk, abbreviated as RADAR.
  arXiv  Detail & Related papers  (2024-10-02T16:47:55Z)
• OmniBench: Towards The Future of Universal Omni-Language Models [63.16606414452612]
  We introduce OmniBench, a novel benchmark designed to evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously.<n>Our evaluation reveals that open-source OLMs show significant limitations in instruction-following and reasoning in tri-modal contexts.<n>We advocate for developing more robust tri-modal integration techniques and training strategies to enhance OLM performance.
  arXiv  Detail & Related papers  (2024-09-23T17:59:05Z)
• Adapting Vision-Language Models to Open Classes via Test-Time Prompt Tuning [50.26965628047682]
  Adapting pre-trained models to open classes is a challenging problem in machine learning. In this paper, we consider combining the advantages of both and come up with a test-time prompt tuning approach. Our proposed method outperforms all comparison methods on average considering both base and new classes.
  arXiv  Detail & Related papers  (2024-08-29T12:34:01Z)
• Cross-Modal Augmentation for Few-Shot Multimodal Fake News Detection [0.21990652930491858]
  Few-shot learning is critical for detecting fake news in its early stages. This paper presents a multimodal fake news detection model which augments multimodal features using unimodal features. The proposed CMA achieves SOTA results over three benchmark datasets.
  arXiv  Detail & Related papers  (2024-07-16T09:32:11Z)
• Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognition [52.522244807811894]
  We propose a novel multimodal Transformer framework using prompt learning to address the issue of missing modalities. Our method introduces three types of prompts: generative prompts, missing-signal prompts, and missing-type prompts. Through prompt learning, we achieve a substantial reduction in the number of trainable parameters.
  arXiv  Detail & Related papers  (2024-07-07T13:55:56Z)
• Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation [37.06795681738417]
  Modality-Incomplete Scene (MISS) is a task that encompasses both system-level modality absence and sensor-level modality errors. We introduce a Missing-aware Modal Switch (MMS) strategy to proactively manage missing modalities during training. We show an improvement of 5.84% mIoU over the prior state-of-the-art parameter-efficient methods in modality missing.
  arXiv  Detail & Related papers  (2024-01-30T11:46:27Z)
• Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarks [21.139161163767884]
  Modal information might miss due to factors such as thermal sensor self-calibration and data transmission error.<n>We propose a novel invertible prompt learning approach, which integrates the content-preserving prompts into a well-trained tracking model.<n>Our method achieves significant performance improvements compared with state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-25T11:39:00Z)
• Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection [53.48346699224921]
  A novel Knowledge-guided Dualconsistency Network is proposed to detect rumors with multimedia contents. It uses two consistency detectionworks to capture the inconsistency at the cross-modal level and the content-knowledge level simultaneously. It also enables robust multi-modal representation learning under different missing visual modality conditions.
  arXiv  Detail & Related papers  (2023-06-03T15:32:20Z)
• Diversity-Aware Meta Visual Prompting [111.75306320834629]
  We present Diversity-Aware Meta Visual Prompting(DAM-VP), an efficient prompting method for transferring pre-trained models to downstream tasks with frozen backbone. We cluster the downstream dataset into small subsets in a diversity-strapped way, with each subset has its own prompt separately. All the prompts are optimized with a meta-prompt, which is learned across several datasets.
  arXiv  Detail & Related papers  (2023-03-14T17:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.