A Few Good Clauses: Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract Extraction

• URL: http://arxiv.org/abs/2605.05532v1
• Date: Thu, 07 May 2026 00:20:13 GMT
• Title: A Few Good Clauses: Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract Extraction
• Abstract summary: We test Olava Extract, a self hosted legal domain Mixture of Experts model, against five frontier models.<n>Olava Extract achieved the strongest aggregate performance in the study, with a macro F1 of 0.812 and a micro F1 of 0.842.
• Score: 0.8399688944263843
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: This paper evaluates whether a domain trained Small Language Model (SLM) can outperform frontier Large Language Models on structured contract extraction at radically lower cost. We test Olava Extract, a self hosted legal domain Mixture of Experts model, against five frontier models. Olava Extract achieved the strongest aggregate performance in the study, with a macro F1 of 0.812 and a micro F1 of 0.842, while reducing inference cost by 78% to 97% compared with the frontier models tested. It also achieved the highest precision scores, producing fewer hallucinated and unsupported extractions, an important distinction in legal workflows where hallucinations create operational risk and downstream review burden. The findings shows that high performing, human comparable legal AI no longer requires the largest externally hosted models. More broadly, they challenge the assumption that commercially valuable enterprise AI capability must remain tied to ever larger models, massive infrastructure expenditure, and centrally hosted providers.

Related papers

• Which Models Perform Better in Inheritance Reasoning? [0.0]
  This paper presents the participation of team PSL in the QIAS 2026 Shared Task on Arabic Islamic inheritance reasoning.<n>The task evaluates the ability of large language models to solve inheritance cases that require legal interpretation, multi-step reasoning, and precise numerical computation.<n>We compare textitcommercial and textitopen-source models under a unified prompting strategy to assess their effectiveness in structured legal reasoning with minimal task-specific adaptation.
  arXiv  Detail & Related papers  (2026-06-11T15:23:32Z)
• RouteLMT: Learned Sample Routing for Hybrid LLM Translation Deployment [57.588738943463646]
  Large Language Models (LLMs) have achieved remarkable performance in Machine Translation (MT)<n> deploying them at scale remains prohibitively expensive.<n>We propose textbfRouteLMT, an in-model router that predicts the expected gain by probing the small translators prompt-token representation.
  arXiv  Detail & Related papers  (2026-04-24T13:02:45Z)
• Can Small Models Reason About Legal Documents? A Comparative Study [0.0]
  Large language models show promise for legal applications, but deploying frontier models raises concerns about cost, latency, and data privacy.<n>We evaluate whether sub-10B parameter models can serve as practical alternatives by testing nine models across three legal benchmarks.
  arXiv  Detail & Related papers  (2026-03-26T22:28:20Z)
• Affordances Enable Partial World Modeling with LLMs [68.52975612311575]
  We show that agents achieving task-agnostic, language-conditioned intents possess predictive partial-world models informed by affordances.<n>In the multi-task setting, we introduce distribution-robust affordances and show that partial models can be extracted to significantly improve search efficiency.
  arXiv  Detail & Related papers  (2026-02-11T00:25:25Z)
• Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning [20.41220110321494]
  We propose Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning.<n>STEER is a domain-agnostic framework that performs fine-grained, step-level routing between smaller and larger language models.<n>Our results establish model-internal confidence as a robust, domain-agnostic signal for model routing.
  arXiv  Detail & Related papers  (2025-11-09T02:33:08Z)
• Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMs [10.044336754070203]
  We investigated agentic RL with large language models on the textscTravelPlanner benchmark.<n>Our approach, textscNatural-R1, achieved a textbf56.9% final-pass rate with only 180 training queries.
  arXiv  Detail & Related papers  (2025-09-30T04:49:36Z)
• Low-Resource Fine-Tuning for Multi-Task Structured Information Extraction with a Billion-Parameter Instruction-Tuned Model [0.0]
  Deploying large language models (LLMs) for structured data extraction in domains such as financial compliance reporting is often impractical for smaller teams due to the high cost of running large architectures and the difficulty of preparing large, high-quality datasets.<n>This work presents a billion- parameter LLaMA-based model fine-tuned with low-rank adaptation on only a few hundred samples per task for extraction, knowledge graph extraction, and named entity recognition.<n>These findings demonstrate that well-tuned small models can deliver stable and accurate structured outputs at a fraction of the computational cost, enabling cost-effective and reliable information extraction pipelines in resource-
  arXiv  Detail & Related papers  (2025-09-10T08:19:07Z)
• Explicit Reasoning Makes Better Judges: A Systematic Study on Accuracy, Efficiency, and Robustness [12.513874407270142]
  We present a systematic comparison of "thinking" and "non-thinking" Large Language Models (LLMs)<n>We evaluate both accuracy and computational efficiency (FLOPs) on RewardBench tasks.<n>Our results show that thinking models achieve approximately 10% points higher accuracy with little overhead.
  arXiv  Detail & Related papers  (2025-09-09T18:36:02Z)
• Predicting Business Angel Early-Stage Decision Making Using AI [0.0]
  Business angels offer a critical source of funding, but their decision-making is often subjective and resource-intensive.<n>One such tool, the Critical Factor Assessment (CFA), has been evaluated post-decision and found to be significantly more accurate than investors' own decisions.<n>This study builds on previous work to investigate whether the constraints inhibiting its adoption can be overcome using a trained AI model.
  arXiv  Detail & Related papers  (2025-07-04T17:17:34Z)
• Model Extrapolation Expedites Alignment [135.12769233630362]
  We propose a method called ExPO to expedite alignment training with human preferences.<n>We demonstrate that ExPO boosts a DPO model trained with only 20% steps to outperform the fully-trained one.<n>We show that ExPO notably improves existing open-source LLMs on the leading AlpacaEval 2.0 and MT-Bench benchmarks.
  arXiv  Detail & Related papers  (2024-04-25T17:39:50Z)
• Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models [53.859446823312126]
  SoTA open source models of varying sizes from 7B - 65B, on average, improve 8.2% from their baseline performance. Strikingly, even models with extremely small memory footprints, such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39% improvement in high-creativity, open ended tasks.
  arXiv  Detail & Related papers  (2023-10-11T15:56:00Z)
• MiniSUPERB: Lightweight Benchmark for Self-supervised Speech Models [90.99663022952498]
  SuperB was proposed to evaluate the generalizability of self-supervised learning (SSL) speech models across various tasks. SuperB incurs high computational costs due to the large datasets and diverse tasks. We introduce MiniSUPERB, a lightweight benchmark that efficiently evaluates SSL speech models with comparable results to SUPERB but lower computational costs significantly.
  arXiv  Detail & Related papers  (2023-05-30T13:07:33Z)
• Efficient Large Scale Language Modeling with Mixtures of Experts [61.45159383372181]
  Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings.
  arXiv  Detail & Related papers  (2021-12-20T17:05:11Z)
• Exploring Sparse Expert Models and Beyond [51.90860155810848]
  Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost. We propose a simple method called expert prototyping that splits experts into different prototypes and applies $k$ top-$1$ routing. This strategy improves the model quality but maintains constant computational costs, and our further exploration on extremely large-scale models reflects that it is more effective in training larger models.
  arXiv  Detail & Related papers  (2021-05-31T16:12:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.