Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study

• URL: http://arxiv.org/abs/2605.14890v2
• Date: Mon, 18 May 2026 08:33:19 GMT
• Title: Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study
• Abstract summary: Tokenizer fertility varies 1.6x across foundation models on Ukrainian legal text.<n>Qwen 3 models consume 60% more tokens than Llama-family models on identical input.<n>Few-shot degrades performance by up to 26 percentage points.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Tokenizer fertility varies 1.6x across foundation models on Ukrainian legal text, yet this cost-critical dimension is absent from model selection practice. We benchmark seven models from five providers on 273 validated court decisions from Ukraine's state registry (EDRSR), measuring tokenizer fertility and zero-shot performance on three tasks. Four findings emerge. (1) Qwen 3 models consume 60% more tokens than Llama-family models on identical input, making tokenizer analysis a prerequisite for cost-efficient deployment. (2) NVIDIA Nemotron Super 3 (120B) achieves the highest composite score (83.1), outperforming Mistral Large 3 (5.6x more total parameters) at one-third the API cost model scale is a poor proxy for domain performance. (3) Few-shot prompting degrades performance by up to 26 percentage points; stratified and prompt-sensitivity ablations confirm this is intrinsic to Ukrainian-language demonstrations, not an artifact of example selection. (4) A cross-temporal generalization experiment reveals that classifiers trained on pre-war court ecisions (2008-2013) lose 27.9 percentage points when applied to full-scale invasion era decisions (2022-2026), with a pronounced forward-backward asymmetry: newer models transfer backward (+14.6 pp above forward transfer), but older models fail catastrophically on wartime legal language. For practitioners: tokenizer analysis should precede model selection, and zero-shot is a more reliable default than few-shot for morphologically rich languages. To support reproducibility and address the absence of Ukrainian from legal NLP benchmarks, we release a public dataset of 14,452 court decisions spanning 2008-2026, annotated with seven outcome labels across three temporal epochs that capture the impact of armed conflict on judicial proceedings.

Related papers

• State of Thought Enables Endogenous Reasoning [48.902117018115256]
  Test-time reasoning has emerged as a major approach to improving the capabilities of Large Language Models (LLMs)<n>We propose State of Thought, a new reasoning paradigm that enables endogenous reasoning in LLMs.<n>We show that SoT consistently improves mean-baseline accuracy while reducing generated tokens by 62.6% and end-to-end latency by 44.6%.
  arXiv  Detail & Related papers  (2026-09-13T05:55:03Z)
• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• Annotated Surrogate Retrieval for Polish Statutory Law [0.0]
  We present a family of retrieval methods for Polish statutory law built on document surrogates.<n>ASCR is a surrogate cascade with reranking; ASCR-H fuses a dense list into that cascade; and DTF replaces both language-model stages with three lexical and dense retrievers.<n>We evaluate all three against fourteen lexical, dense, fused and ablated baselines plus four controls, on 300 questions from the 2024 and 2025 Polish bar and legal counsel entrance examinations.
  arXiv  Detail & Related papers  (2026-08-31T15:05:00Z)
• Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions [0.0]
  Multi-Legal-Bench is the first cross-jurisdictional legal benchmark that evaluates identical tasks across six countries.<n> benchmark defines five tasks court-type classification, judgment form classification, case-outcome prediction, legal norm extraction, and cause category prediction.<n>We evaluate 7 frontier LLMs under zero-shot and 3-shot prompting via AWS Bedrock, with 4 additional small/medium models (3-12B) for scaling analysis.
  arXiv  Detail & Related papers  (2026-05-28T10:31:37Z)
• UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning [0.0]
  We introduce UA-Legal-Bench, a five-task benchmark for evaluating large language models on Ukrainian legal reasoning.<n>We evaluate 11 LLMs from five families under zero-shot and 3-shot prompting via AWS Bedrock with 158K API calls.<n>Our results reveal sharply task-dependent few-shot effects: few-shot prompting improves judgment form classification by up to +38.6 pp but has mixed effects on outcome prediction.
  arXiv  Detail & Related papers  (2026-05-27T23:12:20Z)
• Temporal Concept Drift in Legal Judgment Prediction: Neural Baselines Across Three Epochs of Ukrainian Court Decisions [0.0]
  Legal NLP benchmarks evaluate models on randomly split data, implicitly assuming that legal language is stationary.<n>We test this assumption by fine-tuning four transformer encoders on Ukrainian court decisions from three temporal epochs.<n>Each model is trained on one epoch and evaluated on all three, producing a 3x3 cross-temporal generalization matrix.
  arXiv  Detail & Related papers  (2026-05-23T08:01:14Z)
• Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty? [59.923111838399144]
  We introduce the Visual Aesthetic Benchmark (VAB), which casts aesthetic evaluation as comparative selection over candidate sets with matched subject matter.<n>VAB contains 400 tasks and 1,195 images across fine art, photography, and illustration, with labels derived from the consensus of 10 independent expert judges per task.<n>We find that the strongest system identifies both the best and the worst image correctly across three random permutations of the candidate order in only 26.5% of tasks, far below the 68.9% achieved by human experts.
  arXiv  Detail & Related papers  (2026-05-12T19:33:28Z)
• Can Small Models Reason About Legal Documents? A Comparative Study [0.0]
  Large language models show promise for legal applications, but deploying frontier models raises concerns about cost, latency, and data privacy.<n>We evaluate whether sub-10B parameter models can serve as practical alternatives by testing nine models across three legal benchmarks.
  arXiv  Detail & Related papers  (2026-03-26T22:28:20Z)
• Evaluating Small Language Models for Front-Door Routing: A Harmonized Benchmark and Synthetic-Traffic Experiment [0.05586191108738562]
  Small language models (SLMs) have sufficient reasoning capability for sub-second, zero-marginal-cost, self-hosted task classification.<n>Study 1 is a harmonized offline benchmark of Phi-3.5-mini, Qwen2.5-1.5B, and Qwen-2.5-3B on identical Azure T4 hardware, serving stack, quantization, and a fixed 60-case corpus.<n>Study 2 is a pre-registered four-arm randomized experiment under synthetic traffic with an effective sample size of 60 unique cases per arm.
  arXiv  Detail & Related papers  (2026-03-26T15:57:46Z)
• Benchmarking Linguistic Adaptation in Comparable-Sized LLMs: A Study of Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B on Romanized Nepali [0.0]
  Romanized Nepali, the Nepali language written in the Latin alphabet, is the dominant medium for informal digital communication in Nepal.<n>This study presents a systematic benchmarking of linguistic adaptation across three comparable-sized open-weight models.
  arXiv  Detail & Related papers  (2026-03-25T07:02:51Z)
• Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks [0.0]
  No existing benchmark evaluates frontier model performance on applied Swiss regulatory tasks.<n>I introduce Swiss-Bench SBP, a trilingual benchmark of 395 expert-crafted items spanning three Swiss regulatory domains.<n>I evaluate ten frontier models from March 2026 using a structured three-dimension scoring framework.
  arXiv  Detail & Related papers  (2026-03-24T18:41:07Z)
• Qwen3-ASR Technical Report [71.87071808763484]
  We introduce Qwen3-ASR family, which includes two powerful all-in-one speech recognition models and a novel non-autoregressive speech forced alignment model.<n>Qwen3-ASR-1.7B and Qwen3-ASR-0.6B are ASR models that support language identification and ASR for 52 languages and dialects.
  arXiv  Detail & Related papers  (2026-01-29T06:58:13Z)
• Large-Scale Aspect-Based Sentiment Analysis with Reasoning-Infused LLMs [1.4732811715354455]
  Arctic-ABSA is a collection of powerful models for real-life aspect-based sentiment analysis (ABSA)<n>Our models are tailored to commercial needs, trained on a large corpus of public data alongside carefully generated synthetic data, resulting in a dataset 20 times larger than SemEval14.<n>A single multilingual model maintains 87-91% accuracy across six languages without degrading English performance.
  arXiv  Detail & Related papers  (2026-01-07T13:58:29Z)
• Evaluating Prompt-Based and Fine-Tuned Approaches to Czech Anaphora Resolution [0.0]
  Anaphora resolution plays a critical role in natural language understanding in morphologically rich languages like Czech.<n>This paper presents a comparative evaluation of two modern approaches to anaphora resolution on Czech text.<n>We compare prompt engineering with large language models (LLMs) and fine-tuning compact generative models.
  arXiv  Detail & Related papers  (2025-06-22T16:32:57Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• Advancing LLM Reasoning Generalists with Preference Trees [119.57169648859707]
  We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Eurus models achieve state-of-the-art results among open-source models on a diverse set of benchmarks.
  arXiv  Detail & Related papers  (2024-04-02T16:25:30Z)
• Text Classification via Large Language Models [63.1874290788797]
  We introduce Clue And Reasoning Prompting (CARP) to address complex linguistic phenomena involved in text classification. Remarkably, CARP yields new SOTA performances on 4 out of 5 widely-used text-classification benchmarks. More importantly, we find that CARP delivers impressive abilities on low-resource and domain-adaptation setups.
  arXiv  Detail & Related papers  (2023-05-15T06:24:45Z)
• Making Large Language Models Better Reasoners with Step-Aware Verifier [49.16750018427259]
  DIVERSE (Diverse Verifier on Reasoning Step) is a novel approach that further enhances the reasoning capability of language models. We evaluate DIVERSE on the latest language model code-davinci and show that it achieves new state-of-the-art results on six of eight reasoning benchmarks.
  arXiv  Detail & Related papers  (2022-06-06T03:38:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.