BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension

• URL: http://arxiv.org/abs/2607.05614v1
• Date: Mon, 06 Jul 2026 20:17:38 GMT
• Title: BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension
• Abstract summary: BaFCo is a benchmark dataset for Bangla form comprehension.<n>BaFCo curates 200 multi-page complex Bangladeshi government forms.<n>We evaluate the latest MLLMs from the ChatGPT, Gemini, Claude, Qwen, and Kimi series.
• Score: 12.574173764520728
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Document comprehension is a challenging yet impactful task for Multimodal Large Language Models, especially as these systems see growing adoption in real-world, human-centric applications. However, this adoption is limited for low-resource languages such as Bangla due to the scarcity of high-quality annotated data. To address this gap, we introduce BaFCo, a benchmark dataset for Bangla form comprehension with a focus on Document Layout Analysis (DLA) and Key Information Extraction (KIE). BaFCo curates 200 multi-page complex Bangladeshi government forms, sourced from across diverse sectors including agriculture, education, banking, and land management. To accurately capture the structural and contextual complexity of these forms, we define a fine-grained annotation schema comprising 26 types of form entities, along with a separate coarse form entity set consisting of 5 types. We evaluate the latest MLLMs from the ChatGPT, Gemini, Claude, Qwen, and Kimi series using zero-shot and chain-of-thought prompts under both low and high reasoning setups. Our results reveal limitations in current MLLMs' ability in comprehending Bangla forms, particularly in accurately localizing highly granular form entities. Our dataset and code is available at: https://huggingface.co/datasets/Mausul/bafco

Related papers

• Khondo: A Multimodal Benchmark for Document Packet Splitting of Bangla Forms [13.071620971838316]
  Khondo is the first benchmark for document packet splitting on Bangladeshi government forms.<n>Unlike prior English and OCR-text-based datasets, Khondo is bilingual (Bangla--English) and vision-native.<n>It spans five concatenation schemes, from sequential to fully shuffled, across 14 administrative domains.
  arXiv  Detail & Related papers  (2026-07-23T19:52:58Z)
• Polite on the Surface, Wrong in Practice: A Curated Dataset for Fixing Honorific Failures in Multilingual Bangla Generation [0.0815557531820863]
  We introduce a novel, culturally aligned instruction-tuning dataset for textbfBangLa Application and DialoguE generation.<n>We leverage this resource to systematically fine-tune and evaluate leading open-weight architectures, including DeepSeek-8B and LLaMA-3.2-3B.<n>Our empirical evaluations demonstrate that models fine-tuned on our dataset yield substantial improvements in structural fidelity and honorific alignment.
  arXiv  Detail & Related papers  (2026-05-21T13:43:07Z)
• From Recognition to Reasoning: Benchmarking and Enhancing MLLMs on Real-World Receipt Document Understanding [15.376371030121094]
  ReceiptBench is a large-scale, human-annotated benchmark consisting of 10k diverse receipts.<n>We introduce ReceiptBench, a large-scale, human-annotated benchmark consisting of 10k diverse receipts.
  arXiv  Detail & Related papers  (2026-05-21T12:37:03Z)
• UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameters [55.34921520578968]
  Vision-language models (VLMs) have achieved impressive unified recognition of text and formulas.<n>We propose UniRec-0.1B, a unified recognition model with only 0.1B parameters.<n>It is capable of performing text and formula recognition at multiple levels, including characters, words, lines, paragraphs, and documents.
  arXiv  Detail & Related papers  (2025-12-24T10:35:21Z)
• IndicDLP: A Foundational Dataset for Multi-Lingual and Multi-Domain Document Layout Parsing [8.447520506307567]
  IndicDLP is a large-scale foundational document layout dataset spanning 11 representative Indic languages alongside English and 12 common document domains.<n>Our experiments demonstrate that fine-tuning existing English models on IndicDLP significantly boosts performance.<n>Models trained on IndicDLP generalize well beyond Indic layouts, making it a valuable resource for document digitization.
  arXiv  Detail & Related papers  (2025-12-23T10:49:37Z)
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding [61.36285696607487]
  Document understanding is critical for applications from financial analysis to scientific discovery.<n>Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs) face key limitations.<n>Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG.
  arXiv  Detail & Related papers  (2025-10-17T02:33:16Z)
• Instruction Tuning on Public Government and Cultural Data for Low-Resource Language: a Case Study in Kazakh [57.002807772016524]
  We introduce and open-source a large-scale (10,600 samples) instruction-following dataset, covering key institutional and cultural knowledge relevant to Kazakhstan.<n>We employ LLM-assisted data generation, comparing open-weight and closed-weight models for dataset construction, and select GPT-4o as the backbone.<n>We show that fine-tuning Qwen, Falcon, and Gemma on our dataset leads to consistent performance improvements in both multiple-choice and generative tasks.
  arXiv  Detail & Related papers  (2025-02-19T11:44:27Z)
• BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data [61.936320820180875]
  Large language models (LLMs) have become increasingly pivotal across various domains. BabelBench is an innovative benchmark framework that evaluates the proficiency of LLMs in managing multimodal multistructured data with code execution. Our experimental findings on BabelBench indicate that even cutting-edge models like ChatGPT 4 exhibit substantial room for improvement.
  arXiv  Detail & Related papers  (2024-10-01T15:11:24Z)
• SRFUND: A Multi-Granularity Hierarchical Structure Reconstruction Benchmark in Form Understanding [55.48936731641802]
  We present the SRFUND, a hierarchically structured multi-task form understanding benchmark. SRFUND provides refined annotations on top of the original FUNSD and XFUND datasets. The dataset includes eight languages including English, Chinese, Japanese, German, French, Spanish, Italian, and Portuguese.
  arXiv  Detail & Related papers  (2024-06-13T02:35:55Z)
• XFormParser: A Simple and Effective Multimodal Multilingual Semi-structured Form Parser [32.62155069664013]
  We introduce textbfMultilingual semi-structured textbfXForm textbfPARSER (textbfXForm), which anchored on a comprehensive pre-trained language model.<n>We also develop InDFormSFT, a dataset that specifically addresses the parsing needs of forms in various industrial contexts.
  arXiv  Detail & Related papers  (2024-05-27T16:37:17Z)
• Can Large Language Models Understand Real-World Complex Instructions? [54.86632921036983]
  Large language models (LLMs) can understand human instructions, but struggle with complex instructions. Existing benchmarks are insufficient to assess LLMs' ability to understand complex instructions. We propose CELLO, a benchmark for evaluating LLMs' ability to follow complex instructions systematically.
  arXiv  Detail & Related papers  (2023-09-17T04:18:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.