ActuBench: A Multi-Agent LLM Pipeline for Generation and Evaluation of Actuarial Reasoning Tasks

• URL: http://arxiv.org/abs/2604.20273v1
• Date: Wed, 22 Apr 2026 07:20:03 GMT
• Title: ActuBench: A Multi-Agent LLM Pipeline for Generation and Evaluation of Actuarial Reasoning Tasks
• Abstract summary: ActuBench is a pipeline for the automated generation and evaluation of actuarial assessment items.<n>One agent drafts items, one constructs distractors, a third independently verifies both stages and drives bounded one-shot repair loops.<n>The items, per-model responses and complete leaderboard are published as a browsable web interface.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present ActuBench, a multi-agent LLM pipeline for the automated generation and evaluation of advanced actuarial assessment items aligned with the International Actuarial Association (IAA) Education Syllabus. The pipeline separates four LLM roles by adapter: one agent drafts items, one constructs distractors, a third independently verifies both stages and drives bounded one-shot repair loops, and a cost-optimized auxiliary agent handles Wikipedia-note summarization and topic labelling. The items, per-model responses and complete leaderboard are published as a browsable web interface at https://actubench.de/en/, allowing readers and practitioners to inspect individual items without a repository checkout. We evaluate 50 language models from eight providers on two complementary benchmarks -- 100 empirically hardest multiple-choice items and 100 open-ended items scored by an LLM judge -- and report three headline findings. First, multi-agent verification is load-bearing: the independent verifier flags a majority of drafted items on first pass, most of which the one-shot repair loop resolves. Second, locally-hosted open-weights inference sits on the cost-performance Pareto front: a Gemma~4 model running on consumer hardware and a Cerebras-hosted 120B open-weights model dominate the near-zero-cost region, with the latter within one item of the top of the leaderboard. Third, MCQ and LLM-as-Judge rankings differ meaningfully: the MCQ scaffold inflates the performance ceiling, and Judge-mode evaluation is needed to discriminate at the frontier.

Related papers

• Iris: Climbing to the Search Frontier [21.066541702697112]
  We present Iris-mini and Iris-pro, two search agents trained at the 35B-A3B and 397B-A17B scales, together with the data pipeline and training recipe behind them.
  arXiv  Detail & Related papers  (2026-09-03T17:51:01Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents [0.9924185669370708]
  We introduce CLQT, which reframes closed-loop trading evaluation as diagnosis rather than ranking.<n> CLQT is a fully closed-loop, cost-aware, strategy-consistent, temporally-gated environment.
  arXiv  Detail & Related papers  (2026-06-29T04:26:30Z)
• Divide, Deliberate, Decide: A Multi-Agent Framework for Fine-Grained Egocentric Action Recognition [86.4487145812318]
  Fine-grained action recognition in egocentric video is challenging for Vision-Language Models.<n>We propose Divide, Deliberate, Decide, a fully-local, zero-shot multi-agent framework.<n> Experiments show that our method positively improves zero-shot action recognition performance over the baseline.
  arXiv  Detail & Related papers  (2026-06-16T07:31:27Z)
• UXBench: Measuring the Actionability of LLM-Generated UX Critiques [58.14408744110387]
  We introduce UXBench, a benchmark for evaluating large language models (LLMs) as interaction-grounded UX judges.<n> UXBench comprises local-first runnable web fixtures spanning ten product-surface families, paired with coverage-gated rubric exploration.<n>We evaluate eight frontier models under both an automated repair-lift protocol and a blind human validation study.
  arXiv  Detail & Related papers  (2026-06-15T06:08:39Z)
• Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts [0.07812854697536452]
  We present a large-scale study of multi-tier LLM routing with 206,000 query-model pairs across six benchmarks.<n>We show that a substantial portion of reported unsolvability stems from evaluation artifacts.
  arXiv  Detail & Related papers  (2026-05-08T07:49:24Z)
• Prism-Reranker: Beyond Relevance Scoring -- Jointly Producing Contributions and Evidence for Agentic Retrieval [0.0]
  We introduce Prism-Reranker, a family of reranker models built on Qwen3.5 at four sizes (0.8B, 2B, 4B, 9B)<n>In addition to the standard yes/no relevance judgement, whenever the verdict is yes the model emits a contribution statement summarizing how the document helps the query.<n>We show that the same recipe extends existing LLM-based rerankers, augmenting Qwen3-Reranker-4B with contribution and evidence capabilities.
  arXiv  Detail & Related papers  (2026-04-26T14:28:48Z)
• BenchBench: Benchmarking Automated Benchmark Generation [10.44497524694021]
  BenchBench is a pipeline and dataset for benchmarking automated benchmark generation.<n>We generate 16.7K items, retain 15K core items post-filtering, and produce 152K graded model--item responses.
  arXiv  Detail & Related papers  (2026-03-21T13:05:32Z)
• Agentic AI for Human Resources: LLM-Driven Candidate Assessment [4.271562161453985]
  We present a modular and interpretable framework that uses Large Language Models (LLMs) to automate candidate assessment in recruitment.<n>The system integrates diverse sources, including job descriptions, CVs, interview transcripts, and HR feedback, to generate structured evaluation reports that mirror expert judgment.<n>The framework outputs detailed assessment reports, candidate comparisons, and ranked recommendations that are transparent, auditable, and suitable for real-world hiring.
  arXiv  Detail & Related papers  (2026-03-17T21:32:08Z)
• The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality [70.45240108873001]
  The FACTS Leaderboard is an online leaderboard suite that comprehensively evaluates the ability of language models to generate factually accurate text.<n>The suite provides a holistic measure of factuality by aggregating the performance of models on four distinct sub-leaderboards.
  arXiv  Detail & Related papers  (2025-12-11T16:35:14Z)
• Practical RAG Evaluation: A Rarity-Aware Set-Based Metric and Cost-Latency-Quality Trade-offs [0.0]
  This paper addresses the guessing game in building production RAG.<n>There is no standardized, reproducible way to build and audit golden sets.<n>Rath-gs (MIT) is a lean golden-set pipeline with Plackett-Luce listwise refinement.
  arXiv  Detail & Related papers  (2025-11-12T18:49:21Z)
• LLM Driven Processes to Foster Explainable AI [0.0]
  We present a modular, explainable LLM-agent pipeline for decision support that externalizes reasoning into auditable artifacts.<n>The system instantiates three frameworks: Vester's Sensitivity Model (factor set, signed impact matrix, systemic roles, feedback loops); normal-form games (strategies, payoff matrix, equilibria); and sequential games (role-conditioned agents, tree construction, backward induction), with swappable modules at every step.
  arXiv  Detail & Related papers  (2025-11-10T13:20:00Z)
• Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs [69.2486294522259]
  BaRP is a Bandit Routing-feedback with Preferences approach that trains under the same partial-feedback restriction as deployment.<n> Framed as a contextual bandit over prompt features and a user preference vector, our method simulates an online feedback setting during training and adapts its routing decisions to each new prompt.
  arXiv  Detail & Related papers  (2025-10-08T18:24:59Z)
• AutoSCORE: Enhancing Automated Scoring with Multi-Agent Large Language Models via Structured Component Recognition [27.312190686305588]
  Large language models (LLMs) have shown strong potential in automated scoring.<n>Their use as end-to-end raters faces challenges such as low accuracy, prompt sensitivity, limited interpretability, and rubric misalignment.<n>We propose AutoSCORE, a multi-agent LLM framework enhancing automated scoring via rubric-aligned Structured COmponent REcognition.
  arXiv  Detail & Related papers  (2025-09-26T05:45:14Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• SPARE: Single-Pass Annotation with Reference-Guided Evaluation for Automatic Process Supervision and Reward Modelling [58.05959902776133]
  We introduce Single-Pass.<n>with Reference-Guided Evaluation (SPARE), a novel structured framework that enables efficient per-step annotation.<n>We demonstrate SPARE's effectiveness across four diverse datasets spanning mathematical reasoning (GSM8K, MATH), multi-hop question answering (MuSiQue-Ans), and spatial reasoning (SpaRP)<n>On ProcessBench, SPARE demonstrates data-efficient out-of-distribution generalization, using only $sim$16% of training samples compared to human-labeled and other synthetically trained baselines.
  arXiv  Detail & Related papers  (2025-06-18T14:37:59Z)
• SEED-Bench-2: Benchmarking Multimodal Large Language Models [67.28089415198338]
  Multimodal large language models (MLLMs) have recently demonstrated exceptional capabilities in generating not only texts but also images given interleaved multimodal inputs. SEED-Bench-2 comprises 24K multiple-choice questions with accurate human annotations, which spans 27 dimensions. We evaluate the performance of 23 prominent open-source MLLMs and summarize valuable observations.
  arXiv  Detail & Related papers  (2023-11-28T05:53:55Z)
• AgentBench: Evaluating LLMs as Agents [99.12825098528212]
  Large Language Model (LLM) as agents has been widely acknowledged recently.<n>We present AgentBench, a benchmark that consists of 8 distinct environments to assess LLM-as-Agent's reasoning and decision-making abilities.
  arXiv  Detail & Related papers  (2023-08-07T16:08:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.