Overview of FinMMEval 2026 Task 2: Multilingual Financial Short-Answer Question Answering

• URL: http://arxiv.org/abs/2607.19867v1
• Date: Wed, 22 Jul 2026 07:54:17 GMT
• Title: Overview of FinMMEval 2026 Task 2: Multilingual Financial Short-Answer Question Answering
• Abstract summary: FinMMEval 2026 Task 2 evaluates short-answer financial question answering over multilingual evidence.<n>The final-test set contains 256 items, split evenly between easy and expert tiers.<n>The strongest systems are closely clustered, with the top four separated by less than one percentage point in ROUGE-1 F1.
• Score: 59.6753725406784
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: FinMMEval 2026 Task 2 evaluates short-answer financial question answering over multilingual evidence. Each final-test item pairs an English question with financial statements and news in English, Chinese, Japanese, Spanish, and Greek. Participating systems submit one concise answer per item in JSONL format. The final-test set contains 256 items, split evenly between easy and expert tiers; each tier contains four question templates instantiated over 32 company-report groups. Gold answers were withheld during submission, and systems were ranked by macro-averaged item-level ROUGE-1 F1 against organizer-held reference answers. The final leaderboard includes 12 ranked submissions. The strongest systems are closely clustered, with the top four separated by less than one percentage point in ROUGE-1 F1. The submitted system papers document retrieval-augmented generation, cross-lingual evidence handling, structured prompting, answer compression, and validation strategies.

Related papers

• IGT @ FinMMEval 2026 Task 2: Question-Type Prompting with Targeted Extraction for Multilingual Financial QA [0.0]
  PolyFiQA Task 2 was a multilingual financial question answering task over English SEC filings and multilingual news articles.<n>Our central observation is that the 344 development questions divide into two families requiring fundamentally different approaches.<n>A dataset analysis reveals that 17-18 of 19 ground-truth reference answers per synthesis type share an exact evidence label prefix.<n>The final system achieves development ROUGE-1 approximately 0.395, a 60% relative improvement over a generic RAG baseline.
  arXiv  Detail & Related papers  (2026-09-08T02:23:31Z)
• Overview of FinMMEval 2026 Task 1: Multilingual Financial Multiple-Choice Question Answering [59.67537254067838]
  FinMMEval 2026 Task 1 evaluates multilingual financial multiple-choice question answering in English, Chinese, Arabic, and Hindi.<n>The final-test set contains 800 questions, with 200 questions per language; gold answers were withheld during submission, and each language was ranked independently by accuracy.<n>Top accuracies range from 92.0% in Hindi to 97.5% in English and Arabic, with the same leading teams appearing near the top across all four languages.
  arXiv  Detail & Related papers  (2026-07-22T07:43:44Z)
• Overview of the EReL@MIR 2025 Multimodal Document Retrieval Challenge (Track 1) [4.198923768732269]
  The MIR Challenge was held at the first EReL@MIR workshop, co-located with The Web Conference 2025.<n>It asked participants to build a emphsingle retrieval system that handles two complementary regimes.<n>This report describes the challenge design, datasets, and evaluation protocol; reports the final standings; and analyses the three winning teams' systems.
  arXiv  Detail & Related papers  (2026-06-02T21:39:32Z)
• SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures [93.36078244548077]
  The task data consist of an extended version of our manually constructed BLEnD benchmark (Myung et al. 2024), covering more than 30 language-culture pairs.<n>As the task is designed strictly for evaluation, participants were not permitted to use the data for training, fine-tuning, few-shot learning, or any other form of model modification.<n>We report the results and present an analysis of the best-performing systems and the most commonly adopted approaches.
  arXiv  Detail & Related papers  (2026-05-04T13:49:44Z)
• The CLEF-2026 FinMMEval Lab: Multilingual and Multimodal Evaluation of Financial AI Systems [54.12165004393043]
  FinMMEval 2026 offers three interconnected tasks that span financial understanding, reasoning, and decision-making.<n>The lab aims to promote the development of robust, transparent, and globally inclusive financial AI systems.
  arXiv  Detail & Related papers  (2026-02-11T14:14:06Z)
• MultiFinBen: Benchmarking Large Language Models for Multilingual and Multimodal Financial Application [118.63802040274999]
  MultiFinBen is the first expert-annotated multilingual (five languages) and multimodal benchmark for evaluating LLMs in realistic financial contexts.<n>Financial reasoning tests cross-lingual evidence integration from filings and news, and financial OCR, which extracts structured text from scanned documents.<n> evaluating 21 leading LLMs shows that even frontier multimodal models like GPT-4o achieve only 46.01% overall, stronger on vision and audio but dropping sharply in multilingual settings.
  arXiv  Detail & Related papers  (2025-06-16T22:01:49Z)
• FAMMA: A Benchmark for Financial Domain Multilingual Multimodal Question Answering [18.821122274064116]
  We introduce FAMMA, an open-source benchmark for underlinefinunderlineancial underlinemultilingual underlinemultimodal question underlineanswering (QA)<n>Our benchmark aims to evaluate the abilities of large language models (LLMs) in answering complex reasoning questions that require advanced financial knowledge.
  arXiv  Detail & Related papers  (2024-10-06T15:41:26Z)
• MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languages [54.002969723086075]
  We evaluate cross-lingual open-retrieval question answering systems in 16 typologically diverse languages. The best system leveraging iteratively mined diverse negative examples achieves 32.2 F1, outperforming our baseline by 4.5 points. The second best system uses entity-aware contextualized representations for document retrieval, and achieves significant improvements in Tamil (20.8 F1), whereas most of the other systems yield nearly zero scores.
  arXiv  Detail & Related papers  (2022-07-02T06:54:10Z)
• The SIGMORPHON 2022 Shared Task on Morpheme Segmentation [39.44280269663147]
  The SIGMORPHON 2022 shared task on morpheme segmentation challenged systems to decompose a word into a sequence of morphemes. The best systems outperformed all three state-of-the-art subword tokenization methods (BPE, ULM, Morfessor2) by 30.71% absolute. To facilitate error analysis and support any type of future studies, we released all system predictions, the evaluation script, and all gold standard datasets.
  arXiv  Detail & Related papers  (2022-06-15T15:57:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.