ChartREG++: Towards Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring

• URL: http://arxiv.org/abs/2605.07415v1
• Date: Fri, 08 May 2026 08:07:11 GMT
• Title: ChartREG++: Towards Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring
• Abstract summary: We introduce a chart referring expression grounding benchmark that supports multiple localization forms, multiple referred targets, diverse grounding cues and diverse chart types.<n>Results across representative multimodal large models reveal a significant performance gap.<n>We train an instance segmentation model with the synthesized masks and integrate it into a general-purpose multimodal grounding framework.
• Score: 46.60256399800486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Referring expression grounding is a core problem in visual grounding and is widely used as a diagnostic of spatial grounding and reasoning in vision and language models, yet most prior work focuses on natural images. In contrast, existing chart referring expression grounding-related benchmarks remain limited: (1) they largely adopt bounding boxes, constraining localization precision for fine chart elements (2) they mostly assume a single and two referred target instances, failing to handle multi-instance target references; (3) the language expressions over-rely on textual cues or data-rank clues (4) they cover only a narrow range of chart types. To address these issues, we introduce a chart referring expression grounding benchmark that systematically supports multiple localization forms, multiple referred targets, diverse grounding cues and diverse chart types. Results across representative multimodal large models reveal a significant performance gap. We further introduce a code-driven synthesis pipeline that exploits the inherent alignment between plotting programs and rendered chart primitives to derive pixel accurate instance masks across chart element types and granularities. We train an instance segmentation model with the synthesized masks and integrate it into a general-purpose multimodal grounding framework. The resulting system consistently outperforms baselines on our benchmark and generalizes well to a ChartQA-derived real-chart grounding benchmark.

Related papers

• ChartAnno: Evaluating MLLMs for Chart Annotation Generation [32.95031991526117]
  We introduce ChartAnno, a benchmark for evaluating MLLMs on chart annotation generation.<n>It contains 1,200 real-world charts with paired code and annotation instructions across three levels of instruction specificity.<n>Results show that proprietary models remain stronger overall, although large-scale open-source models narrow the gap.
  arXiv  Detail & Related papers  (2026-08-04T11:01:38Z)
• Semantic-Structural Alignment for Generative Pictorial Charts [59.45629259524998]
  We present a generative framework for the automated synthesis of pictorial charts.<n>We frame the problem as a dual-conditioned generation task guided by two parallel external control signals.<n>Our method produces pictorial charts that are both artistically compelling and structurally consistent.
  arXiv  Detail & Related papers  (2026-05-05T05:20:46Z)
• ChartAnchor: Chart Grounding with Structural-Semantic Fidelity [19.798612765001746]
  Chart grounding refers to the bidirectional alignment between a chart's visual appearance and the structured semantics.<n>ChartAnchor is a benchmark of 8k+ chart-table-code triples spanning 30 chart types drawn from diverse real-world and augmented sources.<n>A multi-level evaluation framework integrates semantic validation, stylistic analysis, and perceptual metrics to assess both structural and content-level correctness.
  arXiv  Detail & Related papers  (2025-11-30T18:28:09Z)
• ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoning [54.86473583610112]
  We propose PointCoT, which integrates reflective interaction into chain-of-thought reasoning in charts.<n>By prompting MLLMs to generate bounding boxes and re-render charts based on location annotations, we establish connections between textual reasoning steps and visual grounding regions.<n>We develop two instruction-tuned models, ChartPointQ2 and ChartPointQ2.5, which outperform state-of-the-art across several chart benchmarks.
  arXiv  Detail & Related papers  (2025-11-29T04:01:55Z)
• ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering [23.455587605758396]
  We introduce ChartAgent, a novel agentic framework that explicitly performs visual reasoning directly within the chart's spatial domain.<n>Our work is among the first to demonstrate visually grounded reasoning for chart understanding using tool-augmented multimodal agents.
  arXiv  Detail & Related papers  (2025-10-06T06:05:36Z)
• InterChart: Benchmarking Visual Reasoning Across Decomposed and Distributed Chart Information [44.79888692172093]
  We introduce InterChart, a diagnostic benchmark that evaluates how well vision-language models (VLMs) reason across multiple related charts.<n>We organize the benchmark into three tiers of increasing difficulty: factual reasoning over individual charts, integrative analysis across synthetically aligned chart sets, and semantic inference over visually complex, real-world chart pairs.
  arXiv  Detail & Related papers  (2025-08-11T05:19:23Z)
• POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answering [69.52231076699756]
  PolyChartQA is the first large-scale multilingual chart question answering benchmark covering 22,606 charts and 26,151 question-answering pairs across 10 diverse languages.<n>We leverage state-of-the-art LLM-based translation and enforce rigorous quality control in the pipeline to ensure the linguistic and semantic consistency of the generated multilingual charts.
  arXiv  Detail & Related papers  (2025-07-16T06:09:02Z)
• Socratic Chart: Cooperating Multiple Agents for Robust SVG Chart Understanding [14.75820681491341]
  Existing benchmarks reveal reliance on text-based shortcuts and probabilistic pattern-matching rather than genuine visual reasoning.<n>We propose Socratic Chart, a new framework that transforms chart images into Scalable Vector Graphics representations.<n>Our framework surpasses state-of-the-art models in accurately capturing chart primitives and improving reasoning performance.
  arXiv  Detail & Related papers  (2025-04-14T00:07:39Z)
• RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuning [63.599057862999]
  RefChartQA is a novel benchmark that integrates Chart Question Answering (ChartQA) with visual grounding.<n>Our experiments demonstrate that incorporating spatial awareness via grounding improves response accuracy by over 15%.
  arXiv  Detail & Related papers  (2025-03-29T15:50:08Z)
• Graph-Based Multimodal Contrastive Learning for Chart Question Answering [11.828192162922436]
  This work introduces a novel joint multimodal scene graph framework that explicitly models the relationships among chart components and their underlying structures.<n>The framework integrates both visual and textual graphs to capture structural and semantic characteristics.<n>A graph contrastive learning strategy aligns node representations across modalities enabling their seamless incorporation into a transformer decoder as soft prompts.
  arXiv  Detail & Related papers  (2025-01-08T06:27:07Z)
• Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation [55.429541407920304]
  Recognizing the predicate between subject and object pairs is imbalanced and multi-label in nature. Recent state-of-the-art methods predominantly focus on the most frequently occurring predicate classes. We introduce a multi-label meta-learning framework to deal with the biased predicate distribution.
  arXiv  Detail & Related papers  (2023-06-16T18:14:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.