E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition

• URL: http://arxiv.org/abs/2604.17319v1
• Date: Sun, 19 Apr 2026 08:18:06 GMT
• Title: E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition
• Abstract summary: Grounded Multimodal Named Entity Recognition (GMNER) aims to jointly identify named entity mentions in text.<n>We propose E2E-GMNER, a fully end-to-end generative framework that unifies entity recognition, semantic typing, visual grounding, and implicit knowledge reasoning.
• Score: 33.81090014865745
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Grounded Multimodal Named Entity Recognition (GMNER) aims to jointly identify named entity mentions in text, predict their semantic types, and ground each entity to a corresponding visual region in an associated image. Existing approaches predominantly adopt pipeline-based architectures that decouple textual entity recognition and visual grounding, leading to error accumulation and suboptimal joint optimization. In this paper, we propose E2E-GMNER, a fully end-to-end generative framework that unifies entity recognition, semantic typing, visual grounding, and implicit knowledge reasoning within a single multimodal large language model. We formulate GMNER as an instruction-tuned conditional generation task and incorporate chain-of-thought reasoning to enable the model to adaptively determine when visual evidence or background knowledge is informative, reducing reliance on noisy cues. To further address the instability of generative bounding box prediction, we introduce Gaussian Risk-Aware Box Perturbation (GRBP), which replaces hard box supervision with probabilistically perturbed soft targets to improve robustness against annotation noise and discretization errors. Extensive experiments on the Twitter-GMNER and Twitter-FMNERG benchmarks demonstrate that E2E-GMNER achieves highly competitive performance compared with state of the art methods, validating the effectiveness of unified end-to-end optimization and noise-aware grounding supervision. Code is available at:https://github.com/Finch-coder/E2E-GMNER

Related papers

• Where Does Generative Difficulty Reside? An Empirical Study of Target Representations [62.54876287800644]
  The target representation defines the distribution an image generator must learn, yet it is often treated as an interchangeable interface.<n>We study raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model.<n>Our results show that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior.
  arXiv  Detail & Related papers  (2026-08-01T12:32:31Z)
• OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibration [48.11927189422178]
  We investigate multimodal meta-verification, which leverages verifier-generated rationales rather than decision-only signals.<n>We train OmniVerifier-M1, a generalist visual verifier leveraging symbolic meta-verification and decoupled reinforcement learning.<n>This approach paves the way for more reliable, interpretable, and fine-grained multimodal verification, supporting safer and more controllable foundation model deployment.
  arXiv  Detail & Related papers  (2026-05-27T17:56:04Z)
• AffectAgent: Collaborative Multi-Agent Reasoning for Retrieval-Augmented Multimodal Emotion Recognition [62.16431420189863]
  LLM-based multimodal emotion recognition relies on static parametric memory and often hallucinates when interpreting nuanced affective states.<n>We introduce AffectAgent, an affect-oriented multi-agent retrieval-augmented generation framework.<n>AffectAgent comprises three jointly optimized agents, namely a query planner, an evidence filter, and an emotion generator.
  arXiv  Detail & Related papers  (2026-04-14T13:49:19Z)
• OmniVL-Guard: Towards Unified Vision-Language Forgery Detection and Grounding via Balanced RL [63.388513841293616]
  Existing forgery detection methods fail to handle the interleaved text, images, and videos prevalent in real-world misinformation.<n>To bridge this gap, this paper targets to develop a unified framework for omnibus vision-language forgery detection and grounding.<n>We propose textbf OmniVL-Guard, a balanced reinforcement learning framework for omnibus vision-language forgery detection and grounding.
  arXiv  Detail & Related papers  (2026-02-11T09:41:36Z)
• GenCNER: A Generative Framework for Continual Named Entity Recognition [22.669221793494163]
  Traditional named entity recognition (NER) aims to identify text mentions into pre-defined entity types.<n>Existing continual learning (CL) methods for NER face challenges of catastrophic forgetting and semantic shift of non-entity type.<n>We propose GenCNER, a simple but effective Generative framework for CNER to mitigate the drawbacks.
  arXiv  Detail & Related papers  (2025-10-13T14:15:31Z)
• Advancing Grounded Multimodal Named Entity Recognition via LLM-Based Reformulation and Box-Based Segmentation [48.47565361014847]
  Grounded Multimodal Named Entity Recognition (GMNER) task aims to identify named entities, entity types and their corresponding visual regions.<n>We propose RiVEG, a unified framework that reformulates GMNER into a joint MNER-VE-VG task by leveraging large language models.
  arXiv  Detail & Related papers  (2024-06-11T13:52:29Z)
• SCANNER: Knowledge-Enhanced Approach for Robust Multi-modal Named Entity Recognition of Unseen Entities [10.193908215351497]
  We propose SCANNER, a model capable of effectively handling all three NER variants. SCANNER is a two-stage structure; we extract entity candidates in the first stage and use it as a query to get knowledge. To tackle the challenges arising from noisy annotations in NER datasets, we introduce a novel self-distillation method.
  arXiv  Detail & Related papers  (2024-04-02T13:05:41Z)
• LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition [28.136662420053568]
  Grounded Multimodal Named Entity Recognition (GMNER) is a nascent multimodal task that aims to identify named entities, entity types and their corresponding visual regions. We propose RiVEG, a unified framework that reformulates GMNER into a joint MNER-VE-VG task by leveraging large language models (LLMs) as a connecting bridge.
  arXiv  Detail & Related papers  (2024-02-15T14:54:33Z)
• Multi-task Transformer with Relation-attention and Type-attention for Named Entity Recognition [35.44123819012004]
  Named entity recognition (NER) is an important research problem in natural language processing. This paper proposes a multi-task Transformer, which incorporates an entity boundary detection task into the named entity recognition task.
  arXiv  Detail & Related papers  (2023-03-20T05:11:22Z)
• Be Your Own Neighborhood: Detecting Adversarial Example by the Neighborhood Relations Built on Self-Supervised Learning [64.78972193105443]
  This paper presents a novel AE detection framework, named trustworthy for predictions. performs the detection by distinguishing the AE's abnormal relation with its augmented versions. An off-the-shelf Self-Supervised Learning (SSL) model is used to extract the representation and predict the label.
  arXiv  Detail & Related papers  (2022-08-31T08:18:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.