Coreference as an indicator of context scope in multimodal narrative

• URL: http://arxiv.org/abs/2503.05298v2
• Date: Fri, 20 Jun 2025 12:55:12 GMT
• Title: Coreference as an indicator of context scope in multimodal narrative
• Abstract summary: We show that large multimodal language models differ substantially from humans in the distribution of coreferential expressions.<n>We introduce a number of metrics to quantify the characteristics of coreferential patterns in both human- and machine-written texts.
• Score: 3.9248546555042356
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We demonstrate that large multimodal language models differ substantially from humans in the distribution of coreferential expressions in a visual storytelling task. We introduce a number of metrics to quantify the characteristics of coreferential patterns in both human- and machine-written texts. Humans distribute coreferential expressions in a way that maintains consistency across texts and images, interleaving references to different entities in a highly varied way. Machines are less able to track mixed references, despite achieving perceived improvements in generation quality. Materials, metrics, and code for our study are available at https://github.com/GU-CLASP/coreference-context-scope.

Related papers

• Explaining How Visual, Textual and Multimodal Encoders Share Concepts [4.203741981113582]
  Sparse autoencoders (SAEs) have emerged as a powerful technique for extracting human-interpretable features from neural networks activations.<n>Previous works compared different models based on SAE-derived features but those comparisons have been restricted to models within the same modality.<n>We propose a novel indicator allowing quantitative comparison of models across SAE features, and use it to conduct a comparative study of visual, textual and multimodal encoders.
  arXiv  Detail & Related papers  (2025-07-24T15:33:31Z)
• RKadiyala at SemEval-2024 Task 8: Black-Box Word-Level Text Boundary Detection in Partially Machine Generated Texts [0.0]
  This paper introduces few reliable approaches for identifying which part of a given text is machine generated at a word level. We present a comparison with proprietary systems, performance of our model on unseen domains' and generators' texts. The findings reveal significant improvements in detection accuracy along with comparison on other aspects of detection capabilities.
  arXiv  Detail & Related papers  (2024-10-22T03:21:59Z)
• CAST: Corpus-Aware Self-similarity Enhanced Topic modelling [16.562349140796115]
  We introduce CAST: Corpus-Aware Self-similarity Enhanced Topic modelling, a novel topic modelling method.<n>We find self-similarity to be an effective metric to prevent functional words from acting as candidate topic words.<n>Our approach significantly enhances the coherence and diversity of generated topics, as well as the topic model's ability to handle noisy data.
  arXiv  Detail & Related papers  (2024-10-19T15:27:11Z)
• MUGC: Machine Generated versus User Generated Content Detection [1.6602942962521352]
  We show that traditional methods demonstrate a high level of accuracy in identifying machine-generated data. Machine-generated texts tend to be shorter and exhibit less word variety compared to human-generated content. readability, bias, moral, and affect comparisons reveal a discernible contrast between machine-generated and human generated content.
  arXiv  Detail & Related papers  (2024-03-28T07:33:53Z)
• CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models [58.95889895912716]
  We introduce a new benchmark, named as CODIS, designed to assess the ability of models to use context provided in free-form text to enhance visual comprehension. Our findings indicate that MLLMs consistently fall short of human performance on this benchmark. This underscores the pressing need to enhance the ability of MLLMs to comprehend visuals in a context-dependent manner.
  arXiv  Detail & Related papers  (2024-02-21T08:21:12Z)
• Language Model Decoding as Direct Metrics Optimization [87.68281625776282]
  Current decoding methods struggle to generate texts that align with human texts across different aspects. In this work, we frame decoding from a language model as an optimization problem with the goal of strictly matching the expected performance with human texts. We prove that this induced distribution is guaranteed to improve the perplexity on human texts, which suggests a better approximation to the underlying distribution of human texts.
  arXiv  Detail & Related papers  (2023-10-02T09:35:27Z)
• Auditing Gender Presentation Differences in Text-to-Image Models [54.16959473093973]
  We study how gender is presented differently in text-to-image models. By probing gender indicators in the input text, we quantify the frequency differences of presentation-centric attributes. We propose an automatic method to estimate such differences.
  arXiv  Detail & Related papers  (2023-02-07T18:52:22Z)
• Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval [109.62363167257664]
  We propose a generative model for learning multilingual text embeddings. Our model operates on parallel data in $N$ languages. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval.
  arXiv  Detail & Related papers  (2022-12-21T02:41:40Z)
• MGDoc: Pre-training with Multi-granular Hierarchy for Document Image Understanding [53.03978356918377]
  spatial hierarchical relationships between content at different levels of granularity are crucial for document image understanding tasks. Existing methods learn features from either word-level or region-level but fail to consider both simultaneously. We propose MGDoc, a new multi-modal multi-granular pre-training framework that encodes page-level, region-level, and word-level information at the same time.
  arXiv  Detail & Related papers  (2022-11-27T22:47:37Z)
• Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables [17.57873577962635]
  We develop a topic-informed discrete latent variable model for semantic textual similarity. Our model learns a shared latent space for sentence-pair representation via vector quantization. We show that our model is able to surpass several strong neural baselines in semantic textual similarity tasks.
  arXiv  Detail & Related papers  (2022-11-07T15:09:58Z)
• Distribution Aware Metrics for Conditional Natural Language Generation [3.6350564275444173]
  We argue that existing metrics are not appropriate for domains such as visual description or summarization where ground truths are semantically diverse. We propose a novel paradigm for multi-candidate evaluation of conditional language generation models.
  arXiv  Detail & Related papers  (2022-09-15T17:58:13Z)
• Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets [56.018551958004814]
  This paper addresses the task of generating fluent descriptions by training on a non-uniform combination of data sources. Large-scale datasets with noisy image-text pairs provide a sub-optimal source of supervision. We propose to leverage and separate semantics and descriptive style through the incorporation of a style token and keywords extracted through a retrieval component.
  arXiv  Detail & Related papers  (2021-11-24T19:00:05Z)
• Cross-Modal Discrete Representation Learning [73.68393416984618]
  We present a self-supervised learning framework that learns a representation that captures finer levels of granularity across different modalities. Our framework relies on a discretized embedding space created via vector quantization that is shared across different modalities.
  arXiv  Detail & Related papers  (2021-06-10T00:23:33Z)
• Sentiment analysis in tweets: an assessment study from classical to modern text representation models [59.107260266206445]
  Short texts published on Twitter have earned significant attention as a rich source of information. Their inherent characteristics, such as the informal, and noisy linguistic style, remain challenging to many natural language processing (NLP) tasks. This study fulfils an assessment of existing language models in distinguishing the sentiment expressed in tweets by using a rich collection of 22 datasets.
  arXiv  Detail & Related papers  (2021-05-29T21:05:28Z)
• Combining Pre-trained Word Embeddings and Linguistic Features for Sequential Metaphor Identification [12.750941606061877]
  We tackle the problem of identifying metaphors in text, treated as a sequence tagging task. The pre-trained word embeddings GloVe, ELMo and BERT have individually shown good performance on sequential metaphor identification. We show that leveraging GloVe, ELMo and feature-based BERT can significantly outperform any single word embedding method and the combination of the two embeddings.
  arXiv  Detail & Related papers  (2021-04-07T17:43:05Z)
• Accurate Word Representations with Universal Visual Guidance [55.71425503859685]
  This paper proposes a visual representation method to explicitly enhance conventional word embedding with multiple-aspect senses from visual guidance. We build a small-scale word-image dictionary from a multimodal seed dataset where each word corresponds to diverse related images. Experiments on 12 natural language understanding and machine translation tasks further verify the effectiveness and the generalization capability of the proposed approach.
  arXiv  Detail & Related papers  (2020-12-30T09:11:50Z)
• Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations [67.4375210552593]
  We design experiments to understand an important but often ignored problem in visually grounded language generation. Given that humans have different utilities and visual attention, how will the sample variance in multi-reference datasets affect the models' performance? We show that it is of paramount importance to report variance in experiments; that human-generated references could vary drastically in different datasets/tasks, revealing the nature of each task.
  arXiv  Detail & Related papers  (2020-10-07T20:45:14Z)
• Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis [103.69656907534456]
  Recent multimodal learning with strong performances on human-centric tasks are often black-box. We propose Multimodal Routing, which adjusts weights between input modalities and output representations differently for each input sample.
  arXiv  Detail & Related papers  (2020-04-29T13:42:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.