Unifying Scientific Communication: Fine-Grained Correspondence Across Scientific Media

• URL: http://arxiv.org/abs/2605.05831v2
• Date: Mon, 11 May 2026 06:58:11 GMT
• Title: Unifying Scientific Communication: Fine-Grained Correspondence Across Scientific Media
• Abstract summary: Communication of scientific knowledge has become increasingly multimodal.<n>Lack of explicit links across formats makes it difficult to trace how concepts, visuals, and explanations correspond.<n>First benchmark that integrates research papers, presentation videos, explanatory videos, and slides from same works.<n>We evaluate a range of embedding-based and vision-language models to assess their ability to discover fine-grained cross-format correspondences.
• Score: 40.40617019402065
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The communication of scientific knowledge has become increasingly multimodal, spanning text, visuals, and speech through materials such as research papers, slides, and recorded presentations. These different representations collectively convey a study's reasoning, results, and insights, offering complementary perspectives that enrich understanding. However, despite their shared purpose, such materials are rarely connected in a structured way. The absence of explicit links across formats makes it difficult to trace how concepts, visuals, and explanations correspond, limiting unified exploration and analysis of research content. To address this gap, we introduce the Multimodal Conference Dataset (MCD), the first benchmark that integrates research papers, presentation videos, explanatory videos, and slides from the same works. We evaluate a range of embedding-based and vision-language models to assess their ability to discover fine-grained cross-format correspondences, establishing the first systematic benchmark for this task. Our results show that vision-language models are robust but struggle with fine-grained alignment, while embedding-based models capture text-visual correspondences well but equations and symbolic content form distinct clusters in the embedding space. These findings highlight both the strengths and limitations of current approaches and point to key directions for future research in multimodal scientific understanding. To ensure reproducibility, we release the resources for MCD at https://github.com/meghamariamkm2002/MCD

Related papers

• ViSR-KGC: Visual Subgraph Reasoning with Vision-Language Models for Multimodal Knowledge Graph Completion [13.875849657353292]
  We propose ViSR-KGC, a visual subgraph reasoning approach for knowledge graph completion.<n>It integrates three complementary capabilities to capture semantic correlations.<n>Based on learned multimodal embeddings, our framework first extracts a compact and query-aware subgraph.
  arXiv  Detail & Related papers  (2026-08-06T10:01:54Z)
• Token-Region Guided Cross-Attention Fusion for Multimodal Affect Interpretation [0.0]
  This paper addresses the detection of political intent in Bengali memes by introducing Multimodal Cross-Attention Fusion framework.<n>We first leverage a Vision-Language Model to extract high-fidelity OCR text from noisy meme images.<n>We then encode visual and textual features and synthesize them through a cross-modal multi-head attention mechanism.
  arXiv  Detail & Related papers  (2026-07-26T06:36:44Z)
• Large-Scale Multidimensional Knowledge Profiling of Scientific Literature [46.15403461273178]
  We compile a unified corpus of more than 100,000 papers from 22 major conferences between 2020 and 2025.<n>Our analysis highlights several notable shifts, including the growth of safety, multimodal reasoning, and agent-oriented studies.<n>These findings provide an evidence-based view of how AI research is evolving and offer a resource for understanding broader trends and identifying emerging directions.
  arXiv  Detail & Related papers  (2026-01-21T16:47:05Z)
• Multimodal Fact Checking with Unified Visual, Textual, and Contextual Representations [2.139909491081949]
  We propose a unified framework for fine-grained multimodal fact verification called "MultiCheck"<n>Our architecture combines dedicated encoders for text and images with a fusion module that captures cross-modal relationships using element-wise interactions.<n>We evaluate our approach on the Factify 2 dataset, achieving a weighted F1 score of 0.84, substantially outperforming the baseline.
  arXiv  Detail & Related papers  (2025-08-07T07:36:53Z)
• Re-calibrating methodologies in social media research: Challenge the visual, work with Speech [0.0]
  This article reflects on how social media scholars can effectively engage with speech-based data in their analyses.<n>I conclude that the expansion of our methodological repertoire enables richer interpretations of platformised content.
  arXiv  Detail & Related papers  (2024-12-17T18:47:57Z)
• SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval [64.03631654052445]
  Current benchmarks for evaluating MMIR performance in image-text pairing within the scientific domain show a notable gap. We develop a specialised scientific MMIR benchmark by leveraging open-access paper collections. This benchmark comprises 530K meticulously curated image-text pairs, extracted from figures and tables with detailed captions in scientific documents.
  arXiv  Detail & Related papers  (2024-01-24T14:23:12Z)
• Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
  Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world. The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time. The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
  arXiv  Detail & Related papers  (2023-07-25T17:59:18Z)
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models [122.27878464009181]
  We conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks. OCRBench contains 29 datasets, making it the most comprehensive OCR evaluation benchmark available.
  arXiv  Detail & Related papers  (2023-05-13T11:28:37Z)
• A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues [23.743431157431893]
  Conditional inference on joint textual and visual clues is a multi-modal reasoning task. We propose a Multi-modal Context Reasoning approach, named ModCR. We conduct extensive experiments on two corresponding data sets and experimental results show significantly improved performance.
  arXiv  Detail & Related papers  (2023-05-08T08:05:40Z)
• Vision+X: A Survey on Multimodal Learning in the Light of Data [64.03266872103835]
  multimodal machine learning that incorporates data from various sources has become an increasingly popular research area. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels.
  arXiv  Detail & Related papers  (2022-10-05T13:14:57Z)
• Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language [148.0843278195794]
  We propose a new model architecture for learning multi-modal neuro-symbolic representations for video captioning. Our approach uses a dictionary learning-based method of learning relations between videos and their paired text descriptions.
  arXiv  Detail & Related papers  (2020-11-18T20:21:19Z)
• Consensus-Aware Visual-Semantic Embedding for Image-Text Matching [69.34076386926984]
  Image-text matching plays a central role in bridging vision and language. Most existing approaches only rely on the image-text instance pair to learn their representations. We propose a Consensus-aware Visual-Semantic Embedding model to incorporate the consensus information.
  arXiv  Detail & Related papers  (2020-07-17T10:22:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.