Alvorada-Bench: Can Language Models Solve Brazilian University Entrance Exams?

• URL: http://arxiv.org/abs/2508.15835v1
• Date: Tue, 19 Aug 2025 03:59:04 GMT
• Title: Alvorada-Bench: Can Language Models Solve Brazilian University Entrance Exams?
• Abstract summary: This paper presents Alvorada-Bench, a 4,515-question, text-only benchmark drawn from five Brazilian university entrance examinations.<n>Top models exceed 94% accuracy overall, but accuracy declines on Mathematics and on the engineering oriented IME and ITA exams.<n>A cost accuracy analysis shows that high accuracy is achievable at under $2 per 1K tokens.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language models are increasingly used in Brazil, but most evaluation remains English-centric. This paper presents Alvorada-Bench, a 4,515-question, text-only benchmark drawn from five Brazilian university entrance examinations. Evaluating twenty models under zero-shot, role-playing, and chain-of-thought prompting, producing 270,900 responses with structured self-reports of confidence, perceived difficulty, and Bloom level. The top models exceed 94% accuracy overall, but accuracy declines on Mathematics and on the engineering oriented IME and ITA exams, indicating persistent weaknesses in multi-step reasoning. Confidence is well calibrated and correlates with perceived difficulty, revealing that models can accurately assess their own certainty capabilities. A cost accuracy analysis shows that high accuracy is achievable at under $2 per 1K tokens. On ENEM 2024 the top model (O3) achieved perfect scores in Languages subject questions while even the weakest system (GPT-4.1 Nano) only underperforms humans in Mathematics. Through exams that distill decades of Brazilian educational priorities and assess millions of students yearly, Alvorada-Bench establishes whether language models can navigate the intersection of language, culture, and reasoning that defines academic readiness in Brazil.

Related papers

• CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context [8.678622777553267]
  CAPITU is a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese.<n>The benchmark comprises 59 instruction types organized into seven categories, all designed to be automatically verifiable.<n>We evaluate 18 state-of-the-art models across single-turn and multi-turn settings.
  arXiv  Detail & Related papers  (2026-03-23T21:16:54Z)
• Evaluating Large Language Models on the 2026 Korean CSAT Mathematics Exam: Measuring Mathematical Ability in a Zero-Data-Leakage Setting [5.313647446600863]
  This study systematically evaluated the mathematical reasoning capabilities of Large Language Models (LLMs) using the 2026 Korean College Scholastic Ability Test (CSAT) Mathematics section.<n>To address data leakage issues in existing benchmarks, we digitized all 46 questions (22 common and 24 elective) within two hours of the exam's public release.
  arXiv  Detail & Related papers  (2025-11-23T23:09:33Z)
• mmJEE-Eval: A Bilingual Multimodal Benchmark for Evaluating Scientific Reasoning in Vision-Language Models [2.0467354053171243]
  We introduce textbfmmJEE-Eval, a multimodal bilingual (English and Hindi) benchmark comprising 1,460 questions from India's Chemistry Advanced examination ( 2019-2025)<n>Our evaluation of 17 state-of-the-art models reveals that while frontier VLMs (GPT-5, Gemini 2.5 Pro/Flash) achieve 77-84% accuracy on held-out 2025 questions, open-source models plateau at 37-45% despite scaling to 400B parameters.
  arXiv  Detail & Related papers  (2025-11-12T13:52:37Z)
• The ML-SUPERB 2.0 Challenge: Towards Inclusive ASR Benchmarking for All Language Varieties [107.57160730151975]
  We construct a new test suite that consists of data from 200+ languages, accents, and dialects to evaluate SOTA multilingual speech models.<n>The best-performing submission achieved an absolute improvement in LID accuracy of 23% and a reduction in CER of 18%.<n>On accented and dialectal data, the best submission obtained 30.2% lower CER and 15.7% higher LID accuracy.
  arXiv  Detail & Related papers  (2025-09-08T18:42:36Z)
• GRILE: A Benchmark for Grammar Reasoning and Explanation in Romanian LLMs [44.99833362998488]
  We present GRILE, the first open benchmark of 1,151 multiple-choice questions harvested from Romanian-stakes exams.<n> GRILE enables us to probe two complementary abilities of seven state-the-art multilingual and Romanian-specific LLMs.
  arXiv  Detail & Related papers  (2025-08-19T21:27:06Z)
• R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation [75.33671166231096]
  We introduce a graduate-level, multi-disciplinary, EnglishChinese benchmark, dubbed as Reasoning Bench (R-Bench)<n>RBench spans 1,094 questions across 108 subjects for language model evaluation and 665 questions across 83 subjects for multimodal model testing.<n>We evaluate widely used models, including OpenAI o1, GPT-4o, DeepSeek-R1, etc.
  arXiv  Detail & Related papers  (2025-05-04T07:48:36Z)
• MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation [86.7047714187813]
  MMLU-ProX is a benchmark covering 29 languages, built on an English benchmark.<n>Each language version consists of 11,829 identical questions, enabling direct cross-linguistic comparisons.<n>To meet efficient evaluation needs, we provide a lite version containing 658 questions per language.
  arXiv  Detail & Related papers  (2025-03-13T15:59:20Z)
• ZNO-Eval: Benchmarking reasoning capabilities of large language models in Ukrainian [0.0]
  This paper presents the ZNO-Eval benchmark based on real exam tasks from Ukraine's standardized educational testing system.<n>It paves the way toward a thorough analysis of reasoning capabilities across different domains and complexities.<n> evaluation of several well-known language models, such as GPT-3.5-Turbo, GPT-4o, GPT-4-Turbo, Mistral Large, Claude 3 Opus, and Gemini-1.5 Pro.
  arXiv  Detail & Related papers  (2025-01-12T04:49:06Z)
• Bilingual Evaluation of Language Models on General Knowledge in University Entrance Exams with Minimal Contamination [35.88131356701857]
  This dataset consists of 1003 multiple-choice questions of university entrance level exams in Spanish and English.<n>A selection of current open-source and proprietary models are evaluated in a uniform zero-shot experimental setting.
  arXiv  Detail & Related papers  (2024-09-19T13:13:07Z)
• Linguini: A benchmark for language-agnostic linguistic reasoning [37.73108306825993]
  We propose a new benchmark to measure a language model's linguistic reasoning skills without relying on pre-existing language-specific knowledge. The test covers 894 questions grouped in 160 problems across 75 (mostly) extremely low-resource languages. We find that while all analyzed models rank below 25% accuracy, there is a significant gap between open and closed models.
  arXiv  Detail & Related papers  (2024-09-18T16:51:02Z)
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems [62.06169250463104]
  We present OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark, featuring 8,476 problems from Olympiad-level mathematics and physics competitions. The best-performing model, GPT-4V, attains an average score of 17.97% on OlympiadBench, with a mere 10.74% in physics. Our analysis orienting GPT-4V points out prevalent issues with hallucinations, knowledge omissions, and logical fallacies.
  arXiv  Detail & Related papers  (2024-02-21T18:49:26Z)
• AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models [122.63704560157909]
  We introduce AGIEval, a novel benchmark designed to assess foundation model in the context of human-centric standardized exams. We evaluate several state-of-the-art foundation models, including GPT-4, ChatGPT, and Text-Davinci-003. GPT-4 surpasses average human performance on SAT, LSAT, and math competitions, attaining a 95% accuracy rate on the SAT Math test and a 92.5% accuracy on the English test of the Chinese national college entrance exam.
  arXiv  Detail & Related papers  (2023-04-13T09:39:30Z)
• No Language Left Behind: Scaling Human-Centered Machine Translation [69.28110770760506]
  We create datasets and models aimed at narrowing the performance gap between low and high-resource languages. We propose multiple architectural and training improvements to counteract overfitting while training on thousands of tasks. Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art.
  arXiv  Detail & Related papers  (2022-07-11T07:33:36Z)
• Few-shot Learning with Multilingual Language Models [66.49496434282564]
  We train multilingual autoregressive language models on a balanced corpus covering a diverse set of languages. Our largest model sets new state of the art in few-shot learning in more than 20 representative languages. We present a detailed analysis of where the model succeeds and fails, showing in particular that it enables cross-lingual in-context learning.
  arXiv  Detail & Related papers  (2021-12-20T16:52:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.