Beyond Single Character: Evaluating MLLMs for Sentence-Level Oracle Bone Inscription Understanding

• URL: http://arxiv.org/abs/2606.31169v1
• Date: Tue, 30 Jun 2026 05:59:44 GMT
• Title: Beyond Single Character: Evaluating MLLMs for Sentence-Level Oracle Bone Inscription Understanding
• Abstract summary: We introduce S-OBI, a novel benchmark for evaluating MLLMs in Sentence-level OBI understanding.<n>S-OBI synthesizes clear and standardized sentence-level OBI instances through glyph substitution and composition.<n>Experiments reveal the inferiority of contemporary MLLMs on sentence-level OBI understanding.
• Score: 52.79561716745463
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing AI-assisted oracle bone inscription (OBI) visual recognition and understanding studies mainly focus on character-level, ignoring the long-form textual coherence and contextual dependencies embedded in complete divination charges. Recently, the powerful visual perception capabilities of multimodal large language models (MLLMs) have opened new possibilities for OBI information processing. In this work, we introduce S-OBI, a novel benchmark for evaluating MLLMs in Sentence-level OBI understanding. Instead of using noisy and incomplete rubbings as the visual input, S-OBI synthesizes clear and standardized sentence-level OBI instances through glyph substitution and composition. According to 95 original rubbings with translations that have been identified, corrected, and verified by experts, we replace characters in the original rubbings with corresponding clean glyph samples sourced from existing OBI datasets while preserving the overall inscriptional structure and semantic organization. This mitigates the influence of low-level distortions and enables a more focused evaluation of sentence-level OBI understanding. Based on this, we design semantic matching, semantic slot extraction, and contextual reasoning tasks and obtain 695 question-answer pairs. Experiments reveal the inferiority of contemporary MLLMs on sentence-level OBI understanding. In particular, visual perception errors in unmasked regions propagate through the reasoning chain, leading to erroneous predictions for masked characters, which indicates that sentence-level OBI understanding in current models remains strongly dependent on character-level recognition. Overall, S-OBI provides a diagnostic benchmark for evaluating whether MLLMs can move beyond isolated character recognition toward structured inscription-level understanding.

Related papers

• AlphaOracle: Oracle bone script decipherment via human-workflow-inspired deep learning [74.08626949007903]
  Approximately 3,000 of the 4,500 bone script (OBS) characters remain undeciphered due to fragmentary inscriptions and sparse evidence.<n>We introduce AlphaOracle, a human-workflow-inspired framework that systematizes OBS decipherment using the largest digitized corpus to date.
  arXiv  Detail & Related papers  (2026-07-20T11:46:00Z)
• Enhancing Oracle Bone Inscription Recognition via Multi-Scale Layer Attention [15.00246240540568]
  Oracle Bone Inscriptions (OBIs) recognition plays a crucial role in understanding ancient Chinese culture.<n>Traditional methods rely on expert knowledge and manual analysis, which are time-consuming and error-prone.<n>We propose Multi-Scale Layer Attention (MSLA), a novel paradigm that explicitly models both multi-scale and cross-layer feature interactions.
  arXiv  Detail & Related papers  (2026-06-30T07:48:05Z)
• Think in Sentences: Explicit Sentence Boundaries Enhance Language Model's Capabilities [3.0508522904185438]
  Large language models (LLMs) acquire linguistic capabilities through exposure to human-generated texts.<n>Motivated by this gap, we propose an approach that insertss at sentence boundaries in LLM inputs.<n>Our results demonstrate consistent improvements across various tasks, with notable gains of up to 7.7% on GSM8k and 12.5% on DROP.
  arXiv  Detail & Related papers  (2026-04-11T10:00:52Z)
• Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs [100.02824137397464]
  We investigate how Large Language Models adapt their internal representations when encountering inputs of increasing difficulty.<n>We reveal a consistent and quantifiable phenomenon: as task difficulty increases, the last hidden states of LLMs become substantially sparser.<n>This sparsity--difficulty relation is observable across diverse models and domains.
  arXiv  Detail & Related papers  (2026-03-03T18:48:15Z)
• Framework for Machine Evaluation of Reasoning Completeness in Large Language Models For Classification Tasks [0.0]
  This paper introduces RACE-Reasoning Alignment for Completeness of Explanations.<n>We analyze four widely used text classification datasets-WIKI ONTOLOGY, AG NEWS, IMDB, and GOEMOTIONS.<n>We show that correct predictions exhibit higher coverage of supporting features, while incorrect predictions are associated with elevated coverage of contradicting features.
  arXiv  Detail & Related papers  (2025-10-23T20:22:22Z)
• Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction [80.88654868264645]
  Arranged and Organized Extraction Benchmark designed to evaluate ability of large language models to comprehend fragmented documents.<n>AOE includes 11 carefully crafted tasks across three diverse domains, requiring models to generate context-specific schema tailored to varied input queries.<n>Results show that even the most advanced models struggled significantly.
  arXiv  Detail & Related papers  (2025-07-22T06:37:51Z)
• Rethinking the Understanding Ability across LLMs through Mutual Information [22.16559695572131]
  We formalize the understanding as MI between an input sentence and its latent representation (sentence-level MI)<n>We decompose sentence-level MI into token-level MI between tokens and sentence embeddings, establishing theoretical bounds connecting these measures.<n>We implement this recoverability task to comparatively measure MI across different language models.
  arXiv  Detail & Related papers  (2025-05-25T22:31:24Z)
• How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective [64.00022624183781]
  Large language models (LLMs) can assess relevance and support information retrieval (IR) tasks.<n>We investigate how different LLM modules contribute to relevance judgment through the lens of mechanistic interpretability.
  arXiv  Detail & Related papers  (2025-04-10T16:14:55Z)
• Do LLMs Understand Your Translations? Evaluating Paragraph-level MT with Question Answering [68.3400058037817]
  We introduce TREQA (Translation Evaluation via Question-Answering), a framework that extrinsically evaluates translation quality.<n>We show that TREQA is competitive with and, in some cases, outperforms state-of-the-art neural and LLM-based metrics in ranking alternative paragraph-level translations.
  arXiv  Detail & Related papers  (2025-04-10T09:24:54Z)
• Oracle Bone Inscriptions Multi-modal Dataset [58.20314888996118]
  Oracle bone inscriptions(OBI) is the earliest developed writing system in China, bearing invaluable written exemplifications of early Shang history and paleography. This paper proposes an Oracle Bone Inscriptions Multi-modal dataset, which includes annotation information for 10,077 pieces of oracle bones. This dataset can be used for a variety of AI-related research tasks relevant to the field of OBI, such as OBI Character Detection and Recognition, Rubbing Denoising, Character Matching, Character Generation, Reading Sequence Prediction, Missing Characters Completion task and so on.
  arXiv  Detail & Related papers  (2024-07-04T12:47:32Z)
• IERL: Interpretable Ensemble Representation Learning -- Combining CrowdSourced Knowledge and Distributed Semantic Representations [11.008412414253662]
  Large Language Models (LLMs) encode meanings of words in the form of distributed semantics. Recent studies have shown that LLMs tend to generate unintended, inconsistent, or wrong texts as outputs. We propose a novel ensemble learning method, Interpretable Ensemble Representation Learning (IERL), that systematically combines LLM and crowdsourced knowledge representations.
  arXiv  Detail & Related papers  (2023-06-24T05:02:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.