Stance-Driven Multimodal Controlled Statement Generation: New Dataset and Task

• URL: http://arxiv.org/abs/2504.03295v1
• Date: Fri, 04 Apr 2025 09:20:19 GMT
• Title: Stance-Driven Multimodal Controlled Statement Generation: New Dataset and Task
• Abstract summary: We study the new problem of stance-driven controllable content generation for tweets with text and images.<n>We create the Multimodal Stance Generation dataset (StanceGen2024), the first resource explicitly designed for multimodal stance-controllable text generation in political discourse.<n>We propose a Stance-Driven Multimodal Generation framework that integrates weighted fusion of multimodal features and stance guidance to improve semantic consistency and stance control.
• Score: 14.63475566746729
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Formulating statements that support diverse or controversial stances on specific topics is vital for platforms that enable user expression, reshape political discourse, and drive social critique and information dissemination. With the rise of Large Language Models (LLMs), controllable text generation towards specific stances has become a promising research area with applications in shaping public opinion and commercial marketing. However, current datasets often focus solely on pure texts, lacking multimodal content and effective context, particularly in the context of stance detection. In this paper, we formally define and study the new problem of stance-driven controllable content generation for tweets with text and images, where given a multimodal post (text and image/video), a model generates a stance-controlled response. To this end, we create the Multimodal Stance Generation Dataset (StanceGen2024), the first resource explicitly designed for multimodal stance-controllable text generation in political discourse. It includes posts and user comments from the 2024 U.S. presidential election, featuring text, images, videos, and stance annotations to explore how multimodal political content shapes stance expression. Furthermore, we propose a Stance-Driven Multimodal Generation (SDMG) framework that integrates weighted fusion of multimodal features and stance guidance to improve semantic consistency and stance control. We release the dataset and code (https://anonymous.4open.science/r/StanceGen-BE9D) for public use and further research.

Related papers

• Token-Region Guided Cross-Attention Fusion for Multimodal Affect Interpretation [0.0]
  This paper addresses the detection of political intent in Bengali memes by introducing Multimodal Cross-Attention Fusion framework.<n>We first leverage a Vision-Language Model to extract high-fidelity OCR text from noisy meme images.<n>We then encode visual and textual features and synthesize them through a cross-modal multi-head attention mechanism.
  arXiv  Detail & Related papers  (2026-07-26T06:36:44Z)
• Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigms [70.51538670020267]
  We propose a hierarchical taxonomy that organizes the field into two primary dimensions: multimodal datasets and multimodal methods.<n>On the data side, we categorize resources ranging from Tactile-Vision datasets, Tactile-Language datasets, Tactile-Vision-Language datasets, and Tactile-Vision-Other datasets.<n>On the method side, we structure prior work into three core pillars: (1) Multimodal Perception and Recognition, (2) Cross-Modal Generation, focusing on bidirectional translation between tactile, vision, and text, and (3) Multimodal Interaction, emphasizing feedback control and language-guided manipulation.
  arXiv  Detail & Related papers  (2026-05-17T09:09:30Z)
• Extending a Parliamentary Corpus with MPs' Tweets: Automatic Annotation and Evaluation Using MultiParTweet [37.712808276294]
  We present MultiParTweet, a tweet corpus from X that connects politicians' social media discourse with German political corpus GerParCor.<n>MultiParTweet contains 39 546 tweets, including 19 056 media items.<n>We enrich the annotation with nine text-based models and one vision-language model (VLM) to annotate MultiParTweet with emotion, sentiment, and topic annotations.
  arXiv  Detail & Related papers  (2025-12-12T13:55:02Z)
• Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning [81.43257201833154]
  We propose Bi-IRRA: a Bidirectional Implicit Relation Reasoning and Aligning framework to learn alignment across languages and modalities.<n>Within Bi-IRRA, a bidirectional implicit relation reasoning module enables bidirectional prediction of masked image and text.<n>The proposed method achieves new state-of-the-art results on all multilingual TIPR datasets.
  arXiv  Detail & Related papers  (2025-10-20T16:01:11Z)
• Two Stage Context Learning with Large Language Models for Multimodal Stance Detection on Climate Change [3.563409707133756]
  We propose a multimodal stance detection framework that integrates textual and visual information through a hierarchical fusion approach.<n>Our method first employs a Large Language Model to retrieve stance-relevant summaries from source text, while a domain-aware image caption generator interprets visual content in the context of the target topic.<n>We evaluate our approach on the MultiClimate dataset, a benchmark for climate change-related stance detection containing aligned video frames and transcripts.
  arXiv  Detail & Related papers  (2025-09-09T10:22:10Z)
• Exploring Vision Language Models for Multimodal and Multilingual Stance Detection [9.079302402271491]
  Social media's global reach amplifies the spread of information, highlighting the need for robust Natural Language Processing tasks. Prior research predominantly focuses on text-only inputs, leaving multimodal scenarios relatively underexplored. This paper evaluates state-of-the-art Vision-Language Models (VLMs) on multimodal and multilingual stance detection tasks.
  arXiv  Detail & Related papers  (2025-01-29T13:39:53Z)
• Multimodal Multi-turn Conversation Stance Detection: A Challenge Dataset and Effective Model [9.413870182630362]
  We introduce a new multimodal multi-turn conversational stance detection dataset (called MmMtCSD) We propose a novel multimodal large language model stance detection framework (MLLM-SD), that learns joint stance representations from textual and visual modalities. Experiments on MmMtCSD show state-of-the-art performance of our proposed MLLM-SD approach for multimodal stance detection.
  arXiv  Detail & Related papers  (2024-09-01T03:16:30Z)
• Multi-modal Stance Detection: New Datasets and Model [56.97470987479277]
  We study multi-modal stance detection for tweets consisting of texts and images. We propose a simple yet effective Targeted Multi-modal Prompt Tuning framework (TMPT) TMPT achieves state-of-the-art performance in multi-modal stance detection.
  arXiv  Detail & Related papers  (2024-02-22T05:24:19Z)
• Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models [114.69732301904419]
  We present an approach to apply end-to-end open-set (any environment/scene) autonomous driving that is capable of providing driving decisions from representations queryable by image and text. Our approach demonstrates unparalleled results in diverse tests while achieving significantly greater robustness in out-of-distribution situations.
  arXiv  Detail & Related papers  (2023-10-26T17:56:35Z)
• Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation [53.97962603641629]
  We propose a novel mulTi-source sEmantic grAph-based Multimodal sarcasm explanation scheme, named TEAM. TEAM extracts the object-level semantic meta-data instead of the traditional global visual features from the input image. TEAM introduces a multi-source semantic graph that comprehensively characterize the multi-source semantic relations.
  arXiv  Detail & Related papers  (2023-06-29T03:26:10Z)
• A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues [23.743431157431893]
  Conditional inference on joint textual and visual clues is a multi-modal reasoning task. We propose a Multi-modal Context Reasoning approach, named ModCR. We conduct extensive experiments on two corresponding data sets and experimental results show significantly improved performance.
  arXiv  Detail & Related papers  (2023-05-08T08:05:40Z)
• Contextual information integration for stance detection via cross-attention [59.662413798388485]
  Stance detection deals with identifying an author's stance towards a target. Most existing stance detection models are limited because they do not consider relevant contextual information. We propose an approach to integrate contextual information as text.
  arXiv  Detail & Related papers  (2022-11-03T15:04:29Z)
• MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text [58.655375327681774]
  We propose the first Multimodal Retrieval-Augmented Transformer (MuRAG) MuRAG accesses an external non-parametric multimodal memory to augment language generation. Our results show that MuRAG achieves state-of-the-art accuracy, outperforming existing models by 10-20% absolute on both datasets.
  arXiv  Detail & Related papers  (2022-10-06T13:58:03Z)
• Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation [70.81596088969378]
  Cross-lingual Outline-based Dialogue dataset (termed COD) enables natural language understanding. COD enables dialogue state tracking, and end-to-end dialogue modelling and evaluation in 4 diverse languages.
  arXiv  Detail & Related papers  (2022-01-31T18:11:21Z)
• FiLMing Multimodal Sarcasm Detection with Attention [0.7340017786387767]
  Sarcasm detection identifies natural language expressions whose intended meaning is different from what is implied by its surface meaning. We propose a novel architecture that uses the RoBERTa model with a co-attention layer on top to incorporate context incongruity between input text and image attributes. Our results demonstrate that our proposed model outperforms the existing state-of-the-art method by 6.14% F1 score on the public Twitter multimodal detection dataset.
  arXiv  Detail & Related papers  (2021-08-09T06:33:29Z)
• Exploiting BERT For Multimodal Target SentimentClassification Through Input Space Translation [75.82110684355979]
  We introduce a two-stream model that translates images in input space using an object-aware transformer. We then leverage the translation to construct an auxiliary sentence that provides multimodal information to a language model. We achieve state-of-the-art performance on two multimodal Twitter datasets.
  arXiv  Detail & Related papers  (2021-08-03T18:02:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.