KITE: A Tri-Modal Transformer Integrating Text, Images, and Knowledge Graphs for Fake News Detection

• URL: http://arxiv.org/abs/2606.07651v1
• Date: Tue, 02 Jun 2026 18:03:17 GMT
• Title: KITE: A Tri-Modal Transformer Integrating Text, Images, and Knowledge Graphs for Fake News Detection
• Abstract summary: We introduce KITE, a tri-modal fake news detection framework that jointly models textual, visual, and factual knowledge representations.<n>KITE uses cross-modal attention within a multimodal transformer to integrate text, visual, and knowledge features, helping it understand how each modality relates to one another.<n> Evaluations on benchmark datasets demonstrate that KITE significantly outperforms unimodal and bimodal baselines.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional fake news detection methods are falling behind as multimodal misinformation grows more advanced, seamlessly blending deceptive text, manipulated visuals, and factually incorrect claims. Most prior work focuses on text-image fusion or applies external knowledge only as a post-processing step, limiting their ability to detect deeper semantic inconsistencies. In this paper, we introduce KITE (Knowledge-Integrated Text-Image Encoder), a tri-modal fake news detection framework that jointly models textual, visual, and factual knowledge representations. KITE leverages Roberta [23,14] and CLIP [24] for linguistic and visual encoding, while a Graph Attention Network (GAT) processes structured facts retrieved from Wikidata. KITE uses cross-modal attention [9] within a multimodal transformer to integrate text, visual, and knowledge features, helping it understand how each modality relates to one another. Modality-specific confidence scores are generated alongside the final prediction, offering interpretability by indicating which input type most influenced the decision. Evaluations on benchmark datasets demonstrate that KITE significantly outperforms unimodal and bimodal baselines, particularly in scenarios involving image-text mismatches or contradictions with external knowledge.

Related papers

• Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessment [57.41731607488101]
  TIC-Bench is designed to evaluate the capability of models to integrate text-image clues and recover the ground truth facts within deeply interleaved contexts.<n>We evaluated 10 state-of-the-art MLLMs and observed a substantial performance gap compared to human experts.
  arXiv  Detail & Related papers  (2026-09-02T13:19:51Z)
• Token-Region Guided Cross-Attention Fusion for Multimodal Affect Interpretation [0.0]
  This paper addresses the detection of political intent in Bengali memes by introducing Multimodal Cross-Attention Fusion framework.<n>We first leverage a Vision-Language Model to extract high-fidelity OCR text from noisy meme images.<n>We then encode visual and textual features and synthesize them through a cross-modal multi-head attention mechanism.
  arXiv  Detail & Related papers  (2026-07-26T06:36:44Z)
• VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text? [51.02924254085878]
  Vision-Language Models (VLMs) have achieved impressive performance in cross-modal understanding across textual and visual inputs.<n>We introduce VISTA-Bench, a benchmark from multimodal perception, reasoning, to unimodal understanding domains.
  arXiv  Detail & Related papers  (2026-02-04T17:48:55Z)
• A Cross-Modal Rumor Detection Scheme via Contrastive Learning by Exploring Text and Image internal Correlations [15.703292627605304]
  This paper presents a novel cross-modal rumor detection scheme based on contrastive learning.<n>A scale-aware fusion network is designed to integrate the highly pertinent multi-scale image features with global text features.<n>The experimental results demonstrate that it achieves a substantial performance improvement over existing state-of-the-art approaches in rumor detection.
  arXiv  Detail & Related papers  (2025-08-15T01:13:50Z)
• KGAlign: Joint Semantic-Structural Knowledge Encoding for Multimodal Fake News Detection [3.221633413708485]
  We propose a novel multi-modal fake news detection framework that integrates visual, textual, and knowledge-based representations.<n>Our proposal introduces a new paradigm: knowledge-grounded multimodal reasoning.
  arXiv  Detail & Related papers  (2025-05-18T13:08:38Z)
• Dynamic Visual Semantic Sub-Embeddings and Fast Re-Ranking [0.5242869847419834]
  We propose a Dynamic Visual Semantic Sub-Embeddings framework (DVSE) to reduce the information entropy. To encourage the generated candidate embeddings to capture various semantic variations, we construct a mixed distribution. We compare the performance with existing set-based method using four image feature encoders and two text feature encoders on three benchmark datasets.
  arXiv  Detail & Related papers  (2023-09-15T04:39:11Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• PV2TEA: Patching Visual Modality to Textual-Established Information Extraction [59.76117533540496]
  We patch the visual modality to the textual-established attribute information extractor. PV2TEA is an encoder-decoder architecture equipped with three bias reduction schemes. Empirical results on real-world e-Commerce datasets demonstrate up to 11.74% absolute (20.97% relatively) F1 increase over unimodal baselines.
  arXiv  Detail & Related papers  (2023-06-01T05:39:45Z)
• Interpretable Detection of Out-of-Context Misinformation with Neural-Symbolic-Enhanced Large Multimodal Model [16.348950072491697]
  Misinformation creators now more tend to use out-of- multimedia contents to deceive the public and fake news detection systems. This new type of misinformation increases the difficulty of not only detection but also clarification, because every individual modality is close enough to true information. In this paper we explore how to achieve interpretable cross-modal de-contextualization detection that simultaneously identifies the mismatched pairs and the cross-modal contradictions.
  arXiv  Detail & Related papers  (2023-04-15T21:11:55Z)
• Contextual information integration for stance detection via cross-attention [59.662413798388485]
  Stance detection deals with identifying an author's stance towards a target. Most existing stance detection models are limited because they do not consider relevant contextual information. We propose an approach to integrate contextual information as text.
  arXiv  Detail & Related papers  (2022-11-03T15:04:29Z)
• Multimodal Fake News Detection with Adaptive Unimodal Representation Aggregation [28.564442206829625]
  AURA is a multimodal fake news detection network with adaptive unimodal representation aggregation. We perform coarse-level fake news detection and cross-modal cosistency learning according to the unimodal and multimodal representations. Experiments on Weibo and Gossipcop prove that AURA can successfully beat several state-of-the-art FND schemes.
  arXiv  Detail & Related papers  (2022-06-12T14:06:55Z)
• Language Matters: A Weakly Supervised Pre-training Approach for Scene Text Detection and Spotting [69.77701325270047]
  This paper presents a weakly supervised pre-training method that can acquire effective scene text representations. Our network consists of an image encoder and a character-aware text encoder that extract visual and textual features. Experiments show that our pre-trained model improves F-score by +2.5% and +4.8% while transferring its weights to other text detection and spotting networks.
  arXiv  Detail & Related papers  (2022-03-08T08:10:45Z)
• CRIS: CLIP-Driven Referring Image Segmentation [71.56466057776086]
  We propose an end-to-end CLIP-Driven Referring Image framework (CRIS) CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. Our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.
  arXiv  Detail & Related papers  (2021-11-30T07:29:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.