K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts

• URL: http://arxiv.org/abs/2606.02404v1
• Date: Mon, 01 Jun 2026 15:50:03 GMT
• Title: K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts
• Abstract summary: We introduce K-BrowseComp, a web-browsing agent benchmark grounded in Korean contexts, consisting of 400 problems.<n>The 300-problem K-BrowseComp-Verified subset is manually constructed and validated by native Korean speakers.<n>On this subset, frontier LLMs, including GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1, reach only 30.00--45.67%, a substantial drop from BrowseComp.
• Score: 43.3460021951349
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Frontier model evaluations are shifting from foundational capabilities (e.g., instruction following and reasoning) toward compositional, agentic ones, but Korean agentic benchmarks remain scarce. We introduce K-BrowseComp, a web-browsing agent benchmark grounded in Korean contexts, consisting of 400 problems. The 300-problem K-BrowseComp-Verified subset is manually constructed and validated by native Korean speakers. On this subset, frontier LLMs, including GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1, reach only 30.00--45.67\%, a substantial drop from BrowseComp, while Korean LLMs released through Korea's Proprietary AI Foundation Model program obtain only 0.00--10.33\%. We further construct a 100-problem synthetic split using hard few-shot exemplars and failure-mode-targeted generation to exploit the asymmetry between solving and creating web browsing problems. On the adversarially filtered synthetic diagnostic split, the strongest model reaches only 26.00\%, and we report this split separately as a targeted stress test. We publicly release our data and code.

Related papers

• Post-Training Language Models for Gold-Medal Performance in Coding Competitions [70.8192632995446]
  We present an end-to-end specialization pipeline combining large-scale problem curation, synthetic reasoning traces, supervised fine-tuning, and reinforcement learning.<n>Using 22,000 curated problems, we train Nemotron-3-Nano-CC with SFT and RL and Nemotron-3-Ultra-CC (550B-A55B) with SFT alone.<n>On IOI 2025, Nano-CC improves from 130 points to 291 after post-training and to 468 with GenCorrect, exceeding the gold threshold of 438.3 while Ultra-CC reaches 502.
  arXiv  Detail & Related papers  (2026-09-02T17:33:37Z)
• NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap [6.166565297478229]
  NOLLI is a benchmark designed to diagnose where Korean performance gaps arise.<n>It comprises 15 puzzle types (25 tasks; 7,500 items), with every instance seed-regenerable, verified to have a unique solution.<n>It combines matched direct translations, script adaptations over Hangul jamo (sub-syllabic letters), and Korean-only tasks grounded in Korean culture or orthography.
  arXiv  Detail & Related papers  (2026-08-05T02:58:53Z)
• CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment [55.74660714572696]
  Malicious content generated from large language models (LLMs) could pose severe safety risks and ethical concerns.<n>Existing LLM safety guardrails excel in English or multilingual settings, but lack adaptation to Chinese-specific regulatory policies, cultural context and linguistic nuances.<n>We introduce a 5-macro, 31-micro category fine-grained risk taxonomy for Chinese scenarios, and build CHILLGuard: a dedicated Chinese LLM content safety guardrail.
  arXiv  Detail & Related papers  (2026-06-13T16:57:51Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• GLAN-QnA-KR: A Seedless Taxonomy-Driven Korean Instruction Corpus [0.0]
  We release GLAN-QnA-KR, a 303,581-row openly redistributable Korean instruction-QA corpus.<n>The corpus spans a flat taxonomy of 1,084 English-labelled disciplines paired with Korean question/answer text.
  arXiv  Detail & Related papers  (2026-05-12T15:23:59Z)
• KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context [8.518732134730355]
  We introduce KMMMU, a native Korean benchmark for evaluating multimodal understanding in Korean cultural and institutional settings.<n>KMMMU contains 3,466 questions from exams written in Korean, covering nine disciplines and nine visual modality categories, along with a 300-item Korean-specific subset and a hard subset of 627 questions.<n>Experiments show that the strongest open-source model reaches only 42.05% accuracy on the full set, while the best proprietary model achieves 52.42% on the hard subset.
  arXiv  Detail & Related papers  (2026-03-18T01:58:14Z)
• K-EXAONE Technical Report [76.23621600385238]
  K-EXAONE is a large-scale multilingual language model developed by LG AI Research.<n>It supports a 256K-token context window and covers six languages: Korean, English, Spanish, German, Japanese, and Vietnamese.<n>We evaluate K-EXAONE on a comprehensive benchmark suite spanning reasoning, agentic, general, Korean, and multilingual abilities.
  arXiv  Detail & Related papers  (2026-01-05T02:30:59Z)
• AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Security [74.22452069013289]
  AegisLLM is a cooperative multi-agent defense against adversarial attacks and information leakage.<n>We show that scaling agentic reasoning system at test-time substantially enhances robustness without compromising model utility.<n> Comprehensive evaluations across key threat scenarios, including unlearning and jailbreaking, demonstrate the effectiveness of AegisLLM.
  arXiv  Detail & Related papers  (2025-04-29T17:36:05Z)
• Multi-Step Reasoning in Korean and the Emergent Mirage [0.0]
  We introduce HRMCR (HAE-RAE Multi-Step Commonsense Reasoning), a benchmark designed to evaluate large language models' ability to perform multi-step reasoning in culturally specific contexts.<n>The questions are automatically generated via templates and algorithms, requiring LLMs to integrate Korean cultural knowledge into sequential reasoning steps.<n>Our experiments reveal that models trained on fewer than (2 cdot 1025) training FLOPs struggle to solve any questions, showing near-zero performance.
  arXiv  Detail & Related papers  (2025-01-10T05:07:27Z)
• CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models [53.9835961434552]
  We introduce the Chinese Instruction-Following Benchmark (CIF-Bench) to evaluate the generalizability of large language models (LLMs) to the Chinese language. CIF-Bench comprises 150 tasks and 15,000 input-output pairs, developed by native speakers to test complex reasoning and Chinese cultural nuances. To mitigate data contamination, we release only half of the dataset publicly, with the remainder kept private, and introduce diversified instructions to minimize score variance.
  arXiv  Detail & Related papers  (2024-02-20T16:02:12Z)
• KMMLU: Measuring Massive Multitask Language Understanding in Korean [32.06346608507584]
  We propose KMMLU, a new Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects ranging from humanities to STEM. While prior Korean benchmarks are translated from existing English benchmarks, KMMLU is collected from original Korean exams.
  arXiv  Detail & Related papers  (2024-02-18T11:41:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.