BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams

• URL: http://arxiv.org/abs/2606.22723v2
• Date: Tue, 30 Jun 2026 12:58:36 GMT
• Title: BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams
• Abstract summary: We introduce BLUEX v2, a benchmark derived from the second-phase entrance exams of Brazil's two leading universities.<n>Our dataset comprises 395 questions unfolding into 919 graded subquestions, with 55.7% of questions containing associated images.<n>Results reveal a 4.92-point performance spread across models, with Mathematical Reasoning and Image Understanding emerging as the hardest capability dimensions.
• Score: 5.232617124162657
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although Large Language Models (LLMs) excel in many tasks, their assessment in Portuguese has received less attention, particularly for open-ended, discursive tasks that demand deeper reasoning and generation capabilities. While the original BLUEX benchmark addressed the scarcity of Portuguese evaluation datasets through multiple-choice questions from Brazilian university entrance exams, it did not cover the more challenging second-phase examinations, which require free-form written responses. In this work, we introduce BLUEX v2, a benchmark derived from the second-phase entrance exams of Brazil's two leading universities: UNICAMP (Comvest) and USP (Fuvest), spanning exam years 2022--2025. Our dataset comprises 395 questions unfolding into 919 graded subquestions, with 55.7% of questions containing associated images (represented as context-aware captions during inference to enable evaluation across both vision-capable and text-only models). Each question is annotated with subject area, official reference answers, LLM-generated rubric criteria, and six cognitive capability tags. We evaluate 21 state-of-the-art LLMs using an LLM-as-a-judge protocol. Results reveal a 4.92-point performance spread across models (4.18-9.10 on a 0-10 scale), with Mathematical Reasoning and Image Understanding emerging as the hardest capability dimensions. The evaluation code, model outputs, and dataset are publicly available at https://github.com/TropicAI-Research/BLUEXv2 and on Hugging Face at https://huggingface.co/datasets/Tropic-AI/BLUEX-v2.

Related papers

• BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning [3.364554138758565]
  We present an updated version of the BLUEX dataset, now including 2024-2025 exams and automatically generated image captions using state-of-the-art models.<n> Captioning strategies increase accessibility to text-only models by more than 40%, producing 1,422 usable questions.
  arXiv  Detail & Related papers  (2025-08-29T01:23:28Z)
• Agri-Query: A Case Study on RAG vs. Long-Context LLMs for Cross-Lingual Technical Question Answering [1.1458853556386799]
  We present a case study evaluating large language models (LLMs) with 128K-token context windows on a technical question answering (QA) task.<n>Our benchmark is built on a user manual for an agricultural machine, available in English, French, and German.<n>It simulates a cross-lingual information retrieval scenario where questions are posed in English against all three language versions of the manual.<n>The evaluation focuses on realistic "needle-in-a-haystack" challenges and includes unanswerable questions to test for hallucinations.
  arXiv  Detail & Related papers  (2025-08-25T14:54:46Z)
• Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering [73.73820209993515]
  We introduce KoLasSimpleQA, the first benchmark evaluating the multilingual factual ability of Large Language Models (LLMs)<n>Inspired by existing research, we created the question set with features such as single knowledge point coverage, absolute objectivity, unique answers, and temporal stability.<n>Results show significant performance differences between the two domains.
  arXiv  Detail & Related papers  (2025-05-22T12:27:02Z)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models [59.0256377330646]
  Lens is a benchmark with 3.4K contemporary images and 60K+ human-authored questions covering eight tasks and 12 daily scenarios.<n>This dataset intrinsically supports to evaluate MLLMs to handle image-invariable prompts, from basic perception to compositional reasoning.<n>We evaluate 15+ frontier MLLMs such as Qwen2.5-VL-72B, InternVL3-78B, GPT-4o and two reasoning models QVQ-72B-preview and Kimi-VL.
  arXiv  Detail & Related papers  (2025-05-21T15:06:59Z)
• LEXam: Benchmarking Legal Reasoning on 340 Law Exams [76.3521146499006]
  We introduce textscLEXam, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels.<n>The dataset comprises 4,886 law exam questions in English and German, including 2,841 long-form, open-ended questions and 2,045 multiple-choice questions.<n>Our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities.
  arXiv  Detail & Related papers  (2025-05-19T08:48:12Z)
• HellaSwag-Pro: A Large-Scale Bilingual Benchmark for Evaluating the Robustness of LLMs in Commonsense Reasoning [56.221060995324436]
  Large language models (LLMs) have shown remarkable capabilities in commonsense reasoning.<n>Do these models truly understand commonsense knowledge, or just memorize expression patterns?<n>We introduce HellaSwag-Pro, a large-scale bilingual benchmark consisting of 11,200 cases.
  arXiv  Detail & Related papers  (2025-02-17T03:24:02Z)
• U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs [2.2330469342127577]
  We introduce U-MATH, a novel benchmark of 1,100 unpublished open-ended university-level problems sourced from teaching materials.<n>It is balanced across six core subjects, with 20% of multimodal problems.<n>Given the open-ended nature of U-MATH problems, we employ an LLM to judge the correctness of generated solutions.<n>Our findings reveal that LLMs achieve a maximum accuracy of only 63% on text-based tasks, with even lower 45% on visual problems.
  arXiv  Detail & Related papers  (2024-12-04T10:44:50Z)
• Revisiting Multi-Modal LLM Evaluation [29.094387692681337]
  We pioneer evaluating recent MLLMs (LLaVA 1.5, LLaVA-NeXT, BLIP2, InstructBLIP, GPT-4V, and GPT-4o) on datasets designed to address weaknesses in earlier ones. Our code is integrated into the widely used LAVIS framework for MLLM evaluation, enabling the rapid assessment of future MLLMs.
  arXiv  Detail & Related papers  (2024-08-09T20:55:46Z)
• Benchmarking Large Language Models on CFLUE -- A Chinese Financial Language Understanding Evaluation Dataset [7.954348293179786]
  We propose CFLUE, a benchmark to assess the capability of large language models (LLMs) across various dimensions. In knowledge assessment, it consists of 38K+ multiple-choice questions with associated solution explanations. In application assessment, it features 16K+ test instances across distinct groups of NLP tasks such as text classification, machine translation, relation extraction, reading comprehension, and text generation.
  arXiv  Detail & Related papers  (2024-05-17T05:03:40Z)
• Are We on the Right Way for Evaluating Large Vision-Language Models? [92.5761176224556]
  Large vision-language models (LVLMs) have recently achieved rapid progress, sparking numerous studies to evaluate their multi-modal capabilities. We identify two primary issues: Visual content is unnecessary for many samples and intentional data leakage exists. We present MMStar, an elite vision-indispensable multi-modal benchmark comprising 1,500 samples meticulously selected by humans.
  arXiv  Detail & Related papers  (2024-03-29T17:59:34Z)
• Hint-enhanced In-Context Learning wakes Large Language Models up for knowledge-intensive tasks [54.153914606302486]
  In-context learning (ICL) ability has emerged with the increasing scale of large language models (LLMs) We propose a new paradigm called Hint-enhanced In-Context Learning (HICL) to explore the power of ICL in open-domain question answering.
  arXiv  Detail & Related papers  (2023-11-03T14:39:20Z)
• L-Eval: Instituting Standardized Evaluation for Long Context Language Models [91.05820785008527]
  We propose L-Eval to institute a more standardized evaluation for long context language models (LCLMs) We build a new evaluation suite containing 20 sub-tasks, 508 long documents, and over 2,000 human-labeled query-response pairs. Results show that popular n-gram matching metrics generally can not correlate well with human judgment.
  arXiv  Detail & Related papers  (2023-07-20T17:59:41Z)
• BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams [4.9069311006119865]
  We introduce BLUEX, a dataset of entrance exams from the two leading universities in Brazil: UNI CAMP and USP. The dataset includes annotated metadata for evaluating the performance of NLP models on a variety of subjects. We establish a benchmark through experiments with state-of-the-art LMs, demonstrating its potential for advancing the state-of-the-art in natural language understanding and reasoning in Portuguese.
  arXiv  Detail & Related papers  (2023-07-11T16:25:09Z)
• Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis [103.89753784762445]
  Large language models (LLMs) have demonstrated remarkable potential in handling multilingual machine translation (MMT) This paper systematically investigates the advantages and challenges of LLMs for MMT. We thoroughly evaluate eight popular LLMs, including ChatGPT and GPT-4.
  arXiv  Detail & Related papers  (2023-04-10T15:51:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.