RMPL: Relation-aware Multi-task Progressive Learning with Stage-wise Training for Multimedia Event Extraction

• URL: http://arxiv.org/abs/2602.13748v1
• Date: Sat, 14 Feb 2026 12:43:25 GMT
• Title: RMPL: Relation-aware Multi-task Progressive Learning with Stage-wise Training for Multimedia Event Extraction
• Authors: Yongkang Jin, Jianwen Luo, Jingjing Wang, Jianmin Yao, Yu Hong,
• Abstract summary: We propose a Relation-aware Multi-task Progressive Learning framework for Multimedia Event Extraction.<n>RMPL incorporates heterogeneous supervision from unimodal event extraction and multimedia relation extraction with stage-wise training.<n> Experiments on the M2E2 benchmark with multiple VLMs show consistent improvements across different modality settings.
• Score: 8.874574211900562
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimedia Event Extraction (MEE) aims to identify events and their arguments from documents that contain both text and images. It requires grounding event semantics across different modalities. Progress in MEE is limited by the lack of annotated training data. M2E2 is the only established benchmark, but it provides annotations only for evaluation. This makes direct supervised training impractical. Existing methods mainly rely on cross-modal alignment or inference-time prompting with Vision--Language Models (VLMs). These approaches do not explicitly learn structured event representations and often produce weak argument grounding in multimodal settings. To address these limitations, we propose RMPL, a Relation-aware Multi-task Progressive Learning framework for MEE under low-resource conditions. RMPL incorporates heterogeneous supervision from unimodal event extraction and multimedia relation extraction with stage-wise training. The model is first trained with a unified schema to learn shared event-centric representations across modalities. It is then fine-tuned for event mention identification and argument role extraction using mixed textual and visual data. Experiments on the M2E2 benchmark with multiple VLMs show consistent improvements across different modality settings.

Related papers

• MuCo: Multi-turn Contrastive Learning for Multimodal Embedding Model [57.89395815934156]
  Multi-Turn Contrastive Learning (MuCo) is a dialogue-inspired framework that revisits this process.<n>Experiments exhibit MuCo with a newly curated 5M multimodal multi-turn dataset (M3T)
  arXiv  Detail & Related papers  (2026-02-06T05:18:33Z)
• ESG-Net: Event-Aware Semantic Guided Network for Dense Audio-Visual Event Localization [16.343327655965858]
  We introduce multi-stage semantic guidance and multi-event relationship modeling.<n>This enables hierarchical semantic understanding of audio-visual events and adaptive extraction of event dependencies.<n>Our method significantly surpasses the state-of-the-art methods, while greatly reducing parameters and computational load.
  arXiv  Detail & Related papers  (2025-07-14T05:42:00Z)
• Grounding Partially-Defined Events in Multimodal Data [61.0063273919745]
  We introduce a multimodal formulation for partially-defined events and cast the extraction of these events as a three-stage span retrieval task. We propose a benchmark for this task, MultiVENT-G, that consists of 14.5 hours of densely annotated current event videos and 1,168 text documents, containing 22.8K labeled event-centric entities. Results illustrate the challenges that abstract event understanding poses and demonstrates promise in event-centric video-language systems.
  arXiv  Detail & Related papers  (2024-10-07T17:59:48Z)
• VIMI: Grounding Video Generation through Multi-modal Instruction [89.90065445082442]
  Existing text-to-video diffusion models rely solely on text-only encoders for their pretraining. We construct a large-scale multimodal prompt dataset by employing retrieval methods to pair in-context examples with the given text prompts. We finetune the model from the first stage on three video generation tasks, incorporating multi-modal instructions.
  arXiv  Detail & Related papers  (2024-07-08T18:12:49Z)
• NoteLLM-2: Multimodal Large Representation Models for Recommendation [71.87790090964734]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks.<n>Their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored.<n>We propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation.
  arXiv  Detail & Related papers  (2024-05-27T03:24:01Z)
• GenEARL: A Training-Free Generative Framework for Multimodal Event Argument Role Labeling [89.07386210297373]
  GenEARL is a training-free generative framework that harnesses the power of modern generative models to understand event task descriptions. We show that GenEARL outperforms the contrastive pretraining (CLIP) baseline by 9.4% and 14.2% accuracy for zero-shot EARL on the M2E2 and SwiG datasets.
  arXiv  Detail & Related papers  (2024-04-07T00:28:13Z)
• Training Multimedia Event Extraction With Generated Images and Captions [6.291564630983316]
  We propose Cross-modality Augmented Multimedia Event Learning (CAMEL) We start with two labeled unimodal datasets in text and image respectively, and generate the missing modality using off-the-shelf image generators like Stable Diffusion and image captioners like BLIP. In order to learn robust features that are effective across domains, we devise an iterative and gradual training strategy.
  arXiv  Detail & Related papers  (2023-06-15T09:01:33Z)
• Beyond Grounding: Extracting Fine-Grained Event Hierarchies Across Modalities [43.048896440009784]
  We propose the task of extracting event hierarchies from multimodal (video and text) data. This reveals the structure of events and is critical to understanding them. We show the limitations of state-of-the-art unimodal and multimodal baselines on this task.
  arXiv  Detail & Related papers  (2022-06-14T23:24:15Z)
• Multi-Modulation Network for Audio-Visual Event Localization [138.14529518908736]
  We study the problem of localizing audio-visual events that are both audible and visible in a video. Existing works focus on encoding and aligning audio and visual features at the segment level. We propose a novel MultiModulation Network (M2N) to learn the above correlation and leverage it as semantic guidance.
  arXiv  Detail & Related papers  (2021-08-26T13:11:48Z)
• Referring Transformer: A One-step Approach to Multi-task Visual Grounding [45.42959940733406]
  We propose a simple one-stage multi-task framework for visual grounding tasks. Specifically, we leverage a transformer architecture, where two modalities are fused in a visual-lingual encoder. We show that our model benefits greatly from contextualized information and multi-task training.
  arXiv  Detail & Related papers  (2021-06-06T10:53:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.