Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks

• URL: http://arxiv.org/abs/2603.23646v1
• Date: Tue, 24 Mar 2026 18:41:07 GMT
• Title: Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks
• Abstract summary: No existing benchmark evaluates frontier model performance on applied Swiss regulatory tasks.<n>I introduce Swiss-Bench SBP, a trilingual benchmark of 395 expert-crafted items spanning three Swiss regulatory domains.<n>I evaluate ten frontier models from March 2026 using a structured three-dimension scoring framework.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: While recent work has benchmarked large language models on Swiss legal translation (Niklaus et al., 2025) and academic legal reasoning from university exams (Fan et al., 2025), no existing benchmark evaluates frontier model performance on applied Swiss regulatory compliance tasks. I introduce Swiss-Bench SBP-002, a trilingual benchmark of 395 expert-crafted items spanning three Swiss regulatory domains (FINMA, Legal-CH, EFK), seven task types, and three languages (German, French, Italian), and evaluate ten frontier models from March 2026 using a structured three-dimension scoring framework assessed via a blind three-judge LLM panel (GPT-4o, Claude Sonnet 4, Qwen3-235B) with majority-vote aggregation and weighted kappa = 0.605, with reference answers validated by an independent human legal expert on a 100-item subset (73% rated Correct, 0% Incorrect, perfect Legal Accuracy). Results reveal three descriptive performance clusters: Tier A (35-38% correct), Tier B (26-29%), and Tier C (13-21%). The benchmark proves difficult: even the top-ranked model (Qwen 3.5 Plus) achieves only 38.2% correct, with 47.3% incorrect and 14.4% partially correct. Task type difficulty varies widely: legal translation and case analysis yield 69-72% correct rates, while regulatory Q&A, hallucination detection, and gap analysis remain below 9%. Within this roster (seven open-weight, three closed-source), an open-weight model leads the ranking, and several open-weight models match or outperform their closed-source counterparts. These findings provide an initial empirical reference point for assessing frontier model capability on Swiss regulatory tasks under zero-retrieval conditions.

Related papers

• Annotated Surrogate Retrieval for Polish Statutory Law [0.0]
  We present a family of retrieval methods for Polish statutory law built on document surrogates.<n>ASCR is a surrogate cascade with reranking; ASCR-H fuses a dense list into that cascade; and DTF replaces both language-model stages with three lexical and dense retrievers.<n>We evaluate all three against fourteen lexical, dense, fused and ablated baselines plus four controls, on 300 questions from the 2024 and 2025 Polish bar and legal counsel entrance examinations.
  arXiv  Detail & Related papers  (2026-08-31T15:05:00Z)
• Small Language Models as Judges for Rubric-Based Reinforcement Learning [57.707881387886516]
  We study whether smaller language models can serve as efficient and reliable rubric-based judges.<n>We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges.<n>Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods.
  arXiv  Detail & Related papers  (2026-08-30T20:00:17Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• Subtitle-Aligned Fine-Tuning of Whisper for Swiss German ASR: Benchmark Contamination, Convention Mismatch, and an Honest Baseline at 25.6% WER (13.8% cWER) [0.0]
  We fine-tune OpenAI's Whisper large-v3 for Swiss German ASR using 1,367 hours of broadcast speech paired with Standard German subtitles as weak supervision.<n>Our best model achieves 25.6% measured WER on the All Swiss German Dialects Test Set (ASGDTS) in an honest evaluation on strictly disjoint data.<n>A vanilla Whisper model self-trained on the ASGDTS test set with zero Swiss German data achieves 13.88% WER, surpassing all published systems.
  arXiv  Detail & Related papers  (2026-05-29T13:43:57Z)
• Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions [0.0]
  Multi-Legal-Bench is the first cross-jurisdictional legal benchmark that evaluates identical tasks across six countries.<n> benchmark defines five tasks court-type classification, judgment form classification, case-outcome prediction, legal norm extraction, and cause category prediction.<n>We evaluate 7 frontier LLMs under zero-shot and 3-shot prompting via AWS Bedrock, with 4 additional small/medium models (3-12B) for scaling analysis.
  arXiv  Detail & Related papers  (2026-05-28T10:31:37Z)
• UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning [0.0]
  We introduce UA-Legal-Bench, a five-task benchmark for evaluating large language models on Ukrainian legal reasoning.<n>We evaluate 11 LLMs from five families under zero-shot and 3-shot prompting via AWS Bedrock with 158K API calls.<n>Our results reveal sharply task-dependent few-shot effects: few-shot prompting improves judgment form classification by up to +38.6 pp but has mixed effects on outcome prediction.
  arXiv  Detail & Related papers  (2026-05-27T23:12:20Z)
• Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study [0.0]
  Tokenizer fertility varies 1.6x across foundation models on Ukrainian legal text.<n>Qwen 3 models consume 60% more tokens than Llama-family models on identical input.<n>Few-shot degrades performance by up to 26 percentage points.
  arXiv  Detail & Related papers  (2026-05-14T14:35:05Z)
• Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs [86.49905745865038]
  Soohak is a 439-problem benchmark newly authored from scratch by 64 mathematicians.<n>The dataset will be publicly released in late 2026, with model evaluations available upon request in the interim.
  arXiv  Detail & Related papers  (2026-05-09T17:14:22Z)
• CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context [8.678622777553267]
  CAPITU is a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese.<n>The benchmark comprises 59 instruction types organized into seven categories, all designed to be automatically verifiable.<n>We evaluate 18 state-of-the-art models across single-turn and multi-turn settings.
  arXiv  Detail & Related papers  (2026-03-23T21:16:54Z)
• Bielik-Q2-Sharp: A Comparative Study of Extreme 2-bit Quantization Methods for a Polish 11B Language Model [0.0]
  We present Bielik-Q2-Sharp, the first systematic academic evaluation of extreme 2-bit quantization applied to a Polish large language model.<n>We compare six state-of-the-art post-training quantization methods -- QuIP#, SpinQuant+GPTQ, ButterflyQuant, QTIP, VPTQ, and AQLM.<n>Our best variant (QuIP# E8P12) achieves 71.92% across 22 Polish benchmarks versus 72.07% for the IQ2_XXS baseline.
  arXiv  Detail & Related papers  (2026-03-04T15:19:35Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
  Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
  arXiv  Detail & Related papers  (2026-02-15T02:50:15Z)
• LEXam: Benchmarking Legal Reasoning on 340 Law Exams [76.3521146499006]
  We introduce textscLEXam, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels.<n>The dataset comprises 4,886 law exam questions in English and German, including 2,841 long-form, open-ended questions and 2,045 multiple-choice questions.<n>Our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities.
  arXiv  Detail & Related papers  (2025-05-19T08:48:12Z)
• Classification or Prompting: A Case Study on Legal Requirements Traceability [4.629156733452248]
  Legal requirements traceability is a key task where engineers must analyze technical requirements against target artifacts.<n>In this paper, we investigate two automated solutions based on language models, including large ones (LLMs)<n>The first solution, Kashif, is a classifier that leverages sentence transformers and semantic similarity.<n>The second solution, RICE_LRT, prompts a recent generative LLM based on RICE, a prompt engineering framework.
  arXiv  Detail & Related papers  (2025-02-07T13:33:40Z)
• Few-shot Learning with Multilingual Language Models [66.49496434282564]
  We train multilingual autoregressive language models on a balanced corpus covering a diverse set of languages. Our largest model sets new state of the art in few-shot learning in more than 20 representative languages. We present a detailed analysis of where the model succeeds and fails, showing in particular that it enables cross-lingual in-context learning.
  arXiv  Detail & Related papers  (2021-12-20T16:52:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.