MIP-GAF: A MLLM-annotated Benchmark for Most Important Person Localization and Group Context Understanding

• URL: http://arxiv.org/abs/2409.06224v1
• Date: Tue, 10 Sep 2024 05:28:38 GMT
• Title: MIP-GAF: A MLLM-annotated Benchmark for Most Important Person Localization and Group Context Understanding
• Abstract summary: Estimating the Most Important Person (MIP) in any social event setup is a challenging problem due to contextual complexity and scarcity of labeled data. We aim to address the problem by annotating a large-scale in-the-wild' dataset for identifying human perceptions about MIP in an image. The proposed dataset will play a vital role in building the next-generation social situation understanding methods.
• Score: 12.572321050617571
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Estimating the Most Important Person (MIP) in any social event setup is a challenging problem mainly due to contextual complexity and scarcity of labeled data. Moreover, the causality aspects of MIP estimation are quite subjective and diverse. To this end, we aim to address the problem by annotating a large-scale `in-the-wild' dataset for identifying human perceptions about the `Most Important Person (MIP)' in an image. The paper provides a thorough description of our proposed Multimodal Large Language Model (MLLM) based data annotation strategy, and a thorough data quality analysis. Further, we perform a comprehensive benchmarking of the proposed dataset utilizing state-of-the-art MIP localization methods, indicating a significant drop in performance compared to existing datasets. The performance drop shows that the existing MIP localization algorithms must be more robust with respect to `in-the-wild' situations. We believe the proposed dataset will play a vital role in building the next-generation social situation understanding methods. The code and data is available at https://github.com/surbhimadan92/MIP-GAF.

Related papers

• From Single- to Cross-Document: Benchmarking Multi-Granularity Event Analysis of Large Language Models [27.87668856281766]
  We introduce MiGUE-Bench, a systematic benchmark for assessing the performance of large language models (LLMs) in event analysis tasks.<n>We first develop an LLM-driven self-correcting annotation framework called MiGUE-Pipeline, enabling scalable acquisition of high-quality source data of events.<n>Then, we design four core tasks in our benchmark, i.e., event detection, relation reasoning, structure induction, and future prediction, to probe model competence at different levels.
  arXiv  Detail & Related papers  (2026-07-30T04:06:55Z)
• Achieving Text-based Person Retrieval with Any Granularity [80.69427598925718]
  This paper introduces a new paradigm, Text-based Person Retrieval with Any Granularity.<n>We formalize a five-level granularity spectrum and construct UFine6926-MG, a high-quality multi-grained annotated dataset.<n>We then propose the Cross-modal Multi-grained Aligning and Matching framework.
  arXiv  Detail & Related papers  (2026-07-23T08:42:25Z)
• MISID: A Multimodal Multi-turn Dataset for Complex Intent Recognition in Strategic Deception Games [7.184720967960135]
  We introduce MISID, a comprehensive benchmark for intent recognition in multi-turn interactions.<n>Sourced from high-stakes social strategy games, MISID features a fine-grained, two-tier multi-dimensional annotation scheme tailored for long-context discourse analysis and evidence-based causal tracking.<n>Our systematic evaluation of state-of-the-art Multimodal Large Language Models (MLLMs) on MISID reveals critical deficiencies in complex scenarios, including text-prior visual hallucination, impaired cross-modal synergy, and limited capacity in chaining causal cues.
  arXiv  Detail & Related papers  (2026-04-14T13:07:54Z)
• Beyond Description: A Multimodal Agent Framework for Insightful Chart Summarization [18.33134893463544]
  We propose a plan-and-execute multi-agent framework to uncover profound insights directly from chart images.<n>To overcome the lack of suitable benchmarks, we introduce ChartSummInsights, a new dataset featuring a diverse collection of real-world charts.
  arXiv  Detail & Related papers  (2026-02-21T06:17:37Z)
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding [61.36285696607487]
  Document understanding is critical for applications from financial analysis to scientific discovery.<n>Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs) face key limitations.<n>Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG.
  arXiv  Detail & Related papers  (2025-10-17T02:33:16Z)
• Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation [25.283739839182147]
  We show that training an MLLM with Chain-of-Thought (CoT) reasoning data can facilitate model adaptation in specialized vision tasks.<n>We propose Grounded Chain-of-Thought (GCoT), a simple bootstrapping-based approach that aims to inject grounding information into CoT data.<n>We evaluate our approach on five specialized vision tasks, which cover a variety of visual formats.
  arXiv  Detail & Related papers  (2025-07-03T17:59:29Z)
• Unsupervised Topic Models are Data Mixers for Pre-training Language Models [6.77198566340415]
  We propose a topic-based data mixing strategy for large language models (LLMs) DataWeave employs a multi-stage clustering process to group semantically similar documents. We confirm that the topics Science and Relationships are particularly effective, yielding the most substantial performance improvements.
  arXiv  Detail & Related papers  (2025-02-24T03:25:56Z)
• Beyond Confusion: A Fine-grained Dialectical Examination of Human Activity Recognition Benchmark Datasets [4.249657064343807]
  The research of machine learning algorithms for human activity recognition (HAR) has made significant progress with publicly available datasets. Recent models like transformers have been applied to HAR datasets with limited success from the benchmark metrics. This paper aims to address these open questions by conducting a fine-grained inspection of six popular HAR benchmark datasets.
  arXiv  Detail & Related papers  (2024-12-12T07:53:17Z)
• Matchmaker: Self-Improving Large Language Model Programs for Schema Matching [60.23571456538149]
  We propose a compositional language model program for schema matching, comprised of candidate generation, refinement and confidence scoring. Matchmaker self-improves in a zero-shot manner without the need for labeled demonstrations. Empirically, we demonstrate on real-world medical schema matching benchmarks that Matchmaker outperforms previous ML-based approaches.
  arXiv  Detail & Related papers  (2024-10-31T16:34:03Z)
• An Information Criterion for Controlled Disentanglement of Multimodal Data [39.601584166020274]
  Multimodal representation learning seeks to relate and decompose information inherent in multiple modalities. Disentangled Self-Supervised Learning (DisentangledSSL) is a novel self-supervised approach for learning disentangled representations.
  arXiv  Detail & Related papers  (2024-10-31T14:57:31Z)
• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• Context is Key: A Benchmark for Forecasting with Essential Textual Information [87.3175915185287]
  "Context is Key" (CiK) is a time series forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context. We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters. Our experiments highlight the importance of incorporating contextual information, demonstrate surprising performance when using LLM-based forecasting models, and also reveal some of their critical shortcomings.
  arXiv  Detail & Related papers  (2024-10-24T17:56:08Z)
• Detecting Training Data of Large Language Models via Expectation Maximization [62.28028046993391]
  Membership inference attacks (MIAs) aim to determine whether a specific instance was part of a target model's training data. Applying MIAs to large language models (LLMs) presents unique challenges due to the massive scale of pre-training data and the ambiguous nature of membership. We introduce EM-MIA, a novel MIA method for LLMs that iteratively refines membership scores and prefix scores via an expectation-maximization algorithm.
  arXiv  Detail & Related papers  (2024-10-10T03:31:16Z)
• FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension [10.482908189805872]
  Referring Expression (REC) is a crucial cross-modal task that objectively evaluates the capabilities of language understanding, image comprehension, and language-to-image grounding. We have established a new REC dataset characterized by two key features. It includes negative text and images created through fine-grained editing and generation based on existing data.
  arXiv  Detail & Related papers  (2024-09-23T06:56:51Z)
• Minimally Supervised Learning using Topological Projections in Self-Organizing Maps [55.31182147885694]
  We introduce a semi-supervised learning approach based on topological projections in self-organizing maps (SOMs) Our proposed method first trains SOMs on unlabeled data and then a minimal number of available labeled data points are assigned to key best matching units (BMU) Our results indicate that the proposed minimally supervised model significantly outperforms traditional regression techniques.
  arXiv  Detail & Related papers  (2024-01-12T22:51:48Z)
• Semantic-Preserving Feature Partitioning for Multi-View Ensemble Learning [11.415864885658435]
  We introduce the Semantic-Preserving Feature Partitioning (SPFP) algorithm, a novel method grounded in information theory. The SPFP algorithm effectively partitions datasets into multiple semantically consistent views, enhancing the multi-view ensemble learning process. It maintains model accuracy while significantly improving uncertainty measures in scenarios where high generalization performance is achievable.
  arXiv  Detail & Related papers  (2024-01-11T20:44:45Z)
• Improving Classifier Training Efficiency for Automatic Cyberbullying Detection with Feature Density [58.64907136562178]
  We study the effectiveness of Feature Density (FD) using different linguistically-backed feature preprocessing methods. We hypothesise that estimating dataset complexity allows for the reduction of the number of required experiments. The difference in linguistic complexity of datasets allows us to additionally discuss the efficacy of linguistically-backed word preprocessing.
  arXiv  Detail & Related papers  (2021-11-02T15:48:28Z)
• The Surprising Performance of Simple Baselines for Misinformation Detection [4.060731229044571]
  We examine the performance of a broad set of modern transformer-based language models. We present our framework as a baseline for creating and evaluating new methods for misinformation detection.
  arXiv  Detail & Related papers  (2021-04-14T16:25:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.