Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems

• URL: http://arxiv.org/abs/2607.10768v1
• Date: Sun, 12 Jul 2026 13:47:57 GMT
• Title: Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems
• Abstract summary: LLM-based agents are increasingly deployed to solve optimization problems, yet existing benchmarks evaluate them on pre-structured benchmarks.<n>We introduce an end-to-end benchmark that evaluates Large Models (LLMs) from implementation through integer code, descriptions and report generation.
• Score: 1.7689718394104148
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-based agents are increasingly deployed to solve optimization problems, yet existing benchmarks evaluate them on pre-structured mathematical formulations that bypass the most critical challenge: translating complex business requirements into correct models and solve efficiently. We introduce Opti-Agent-Bench, an end-to-end benchmark that evaluates Large Language Models (LLMs) across the complete optimization R&D pipeline, from understanding business-language descriptions through mathematical modeling, algorithm selection, and code implementation, to solution report generation. Our design rests on three pillars: (1) businesssemantic authenticity with anti-template traps that defeat pattern matching; (2) modular evaluation with cross-module consistency checking across Problem Understanding, Formal Modeling, Implementation, and Reporting; and (3) the ORAC bi-level validity framework that simultaneously ensures task quality and scoring integrity. Across several industrialscale tasks spanning integer programming, robust optimization, stochastic programming, and non-convex optimization, we expose critical failure modes of current models, including constraint omission, model-code inconsistency, and report-implementation divergence, that remain invisible under conventional single-metric evaluation.

Related papers

• little m: An AI Agent for Industrial Process Optimization [6.819597984555337]
  We introduce little m, an AI agent designed to assist the formulation of industrial process control models.<n>For systematic evaluation, we introduce the Industrial Process Control Benchmark (IPC-Bench), a novel dataset of 50 canonical scenarios.<n>Little m substantially outperforms state-of-the-art LLMs, generating semantically correct models.
  arXiv  Detail & Related papers  (2026-09-15T06:01:18Z)
• OptiAgent: End-to-End Optimization Modeling via Multi-Agent Iterative Refinement [0.0]
  OptiAgent is a multi-agent framework that is able to output a solver-ready mathematical formulation as well as executable code.<n>Our architecture prioritizes the mathematical modeling step, where dedicated agents extract structures.<n>We introduce a novel multi-loop validation architecture with four specialized feedback mechanisms, each targeting a distinct failure mode.
  arXiv  Detail & Related papers  (2026-07-06T17:27:16Z)
• Opt-Verifier: Unleashing the Power of LLMs for Optimization Modeling via Dual-Side Verification [53.763212981479455]
  We propose a novel framework with Dual-side Verification (Opt-Verifier) from both structure and solution perspectives.<n> Experiments on popular benchmarks demonstrate that our approach achieves over 20% improvement in accuracy.
  arXiv  Detail & Related papers  (2026-05-28T08:09:52Z)
• FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization [61.43300970020897]
  Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation.<n>Existing benchmarks are limited to small or simplified examples far below real-world scale and complexity.<n>We introduce FrontierOR, among the first benchmarks to evaluate LLM-based efficient algorithm design for realistic large-scale optimization problems.
  arXiv  Detail & Related papers  (2026-05-24T20:10:42Z)
• OpenCompass: A Universal Evaluation Platform for Large Language Models [62.59671563145442]
  General-purpose large language models (LLMs) have become a critical link in advancing technological development.<n>Mainstream benchmark datasets face challenges such as the diversity of task types, inconsistent evaluation criteria, and fragmentation of data and processing.<n>This paper proposes and open-source Open LLMs, a one-stop, scalable, and high-concurrency general-purpose evaluation platform.
  arXiv  Detail & Related papers  (2026-05-19T02:50:11Z)
• Peering Inside the Black Box: Uncovering LLM Errors in Optimization Modelling through Component-Level Evaluation [0.0]
  We present a component-level evaluation framework for large language models (LLMs)<n>We evaluate GPT-5, LLaMA 3.1 Instruct, and DeepSeek Math across optimization problems of varying complexity.<n>Results show GPT-5 consistently outperforms other models, with chain-of-thought, self-consistency, and modular prompting proving most effective.
  arXiv  Detail & Related papers  (2025-10-19T17:47:59Z)
• RefactorCoderQA: Benchmarking LLMs for Multi-Domain Coding Question Solutions in Cloud and Edge Deployment [20.416910591388618]
  We introduce RefactorCoderQA, a benchmark designed to evaluate and enhance the performance of Large Language Models (LLMs) across coding tasks.<n>Our fine-tuned model, RefactorCoder-MoE, achieves state-of-the-art performance, significantly outperforming leading open-source and commercial baselines with an overall accuracy of 76.84%.
  arXiv  Detail & Related papers  (2025-09-12T17:44:22Z)
• Automated Optimization Modeling through Expert-Guided Large Language Model Reasoning [43.63419208391747]
  We present a novel framework that leverages expert-level optimization modeling principles through chain-of-thought reasoning to automate the optimization process.<n>We also introduce LogiOR, a new optimization modeling benchmark from the logistics domain, containing more complex problems with standardized annotations.
  arXiv  Detail & Related papers  (2025-08-20T04:14:54Z)
• ORMind: A Cognitive-Inspired End-to-End Reasoning Framework for Operations Research [56.961539386979354]
  We introduce ORMind, a cognitive-inspired framework that enhances optimization through counterfactual reasoning.<n>Our approach emulates human cognition, implementing an end-to-end workflow that transforms requirements into mathematical models and executable code.<n>It is currently being tested internally in Lenovo's AI Assistant, with plans to enhance optimization capabilities for both business and consumer customers.
  arXiv  Detail & Related papers  (2025-06-02T05:11:21Z)
• ModelingAgent: Bridging LLMs and Mathematical Modeling for Real-World Challenges [72.19809898215857]
  We introduce ModelingBench, a novel benchmark featuring real-world-inspired, open-ended problems from math modeling competitions across diverse domains.<n>These tasks require translating natural language into formal mathematical formulations, applying appropriate tools, and producing structured, defensible reports.<n>We also present ModelingAgent, a multi-agent framework that coordinates tool use, supports structured, creative solutions, and generates well-grounded, creative solutions.
  arXiv  Detail & Related papers  (2025-05-21T03:33:23Z)
• Autoformulation of Mathematical Optimization Models Using LLMs [50.030647274271516]
  This paper approaches the problem of $textitautoformulation$: the automated creation of solver-ready optimization models from natural language problem descriptions.<n>We identify three core challenges of autoformulation: $textit(1)$ the vast, problem-dependent hypothesis space, and $textit(2)$ efficient and diverse exploration of this space under uncertainty.<n>We present a novel method leveraging $textitLarge Language Models$ with $textitMonte-Carlo Tree Search$, exploiting the hierarchical nature of optimization modeling to generate and systematically explore possible formulations
  arXiv  Detail & Related papers  (2024-11-03T20:41:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.