CSCBench: A PVC Diagnostic Benchmark for Commodity Supply Chain Reasoning

• URL: http://arxiv.org/abs/2601.01825v1
• Date: Mon, 05 Jan 2026 06:44:29 GMT
• Title: CSCBench: A PVC Diagnostic Benchmark for Commodity Supply Chain Reasoning
• Abstract summary: Large Language Models (LLMs) have achieved remarkable success in general benchmarks, yet their competence in commodity supply chains remains under-explored.<n>We introduce CSCBench, a 2.3K+ single-choice benchmark for CSC reasoning, instantiated through our PVC 3D Evaluation Framework.
• Score: 9.840374380176135
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have achieved remarkable success in general benchmarks, yet their competence in commodity supply chains (CSCs) -- a domain governed by institutional rule systems and feasibility constraints -- remains under-explored. CSC decisions are shaped jointly by process stages (e.g., planning, procurement, delivery), variety-specific rules (e.g., contract specifications and delivery grades), and reasoning depth (from retrieval to multi-step analysis and decision selection). We introduce CSCBench, a 2.3K+ single-choice benchmark for CSC reasoning, instantiated through our PVC 3D Evaluation Framework (Process, Variety, and Cognition). The Process axis aligns tasks with SCOR+Enable; the Variety axis operationalizes commodity-specific rule systems under coupled material-information-financial constraints, grounded in authoritative exchange guidebooks/rulebooks and industry reports; and the Cognition axis follows Bloom's revised taxonomy. Evaluating representative LLMs under a direct prompting setting, we observe strong performance on the Process and Cognition axes but substantial degradation on the Variety axis, especially on Freight Agreements. CSCBench provides a diagnostic yardstick for measuring and improving LLM capabilities in this high-stakes domain.

Related papers

• Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulation [5.488097487012986]
  evaluating the decision-making capabilities of large language models (LLMs) is a growing research priority.<n>We introduce textscCEO-Bench, a benchmark that evaluates LLMs on CEO-level strategic resource reallocation.<n> Experiments across five frontier models on 13 scenarios reveal that all models achieve high structural validity but diverge sharply on strategic calibration.
  arXiv  Detail & Related papers  (2026-06-16T03:20:04Z)
• Robust Checkpoint Selection for Multimodal LLMs via Agentic Evaluation and Stability-Aware Ranking [0.4562811934651716]
  We formulate checkpoint selection as a robust decision problem under evaluation uncertainty.<n>We propose a multi-stage framework that integrates curated real-world data, structured LLM-based judgment, and multi-stage ranking protocols.<n>We demonstrate that data quality, specifically OCR readability, is a critical determinant of evaluation validity.
  arXiv  Detail & Related papers  (2026-05-13T12:18:32Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning [79.88942231770629]
  Reinforcement learning (RL) has become a central post-training tool for improving the reasoning abilities of large language models (LLMs)<n>We formalize rollout pipelines with unified notation and introduce Generate-Filter-Control-Replay (GFCR)<n>We synthesize methods spanning RL with verifiable rewards, process supervision, judge-based gating, guided and tree/segment rollouts, adaptive compute allocation, early-exit and partial rollouts, throughput optimization, and replay/recomposition for self-improvement.
  arXiv  Detail & Related papers  (2026-04-08T00:53:29Z)
• AI Agent Systems for Supply Chains: Structured Decision Prompts and Memory Retrieval [3.3703751888858675]
  This study investigates large language model (LLM) -based multi-agent systems (MASs) as a promising approach to inventory management.<n>It is unclear whether LLM-based MASs can consistently derive optimal ordering policies and adapt to diverse supply chain scenarios.
  arXiv  Detail & Related papers  (2026-02-05T10:35:00Z)
• State-Augmented Graphs for Circular Economy Triage [2.7459577246953963]
  Circular economy (CE) triage is the assessment of products to determine which sustainable pathway they can follow once they reach the end of their usefulness.<n>This paper presents a novel decision-making framework as a simple deterministic solver over a state-augmented Disassembly Sequencing Planning graph.
  arXiv  Detail & Related papers  (2025-12-17T16:23:47Z)
• Practical RAG Evaluation: A Rarity-Aware Set-Based Metric and Cost-Latency-Quality Trade-offs [0.0]
  This paper addresses the guessing game in building production RAG.<n>There is no standardized, reproducible way to build and audit golden sets.<n>Rath-gs (MIT) is a lean golden-set pipeline with Plackett-Luce listwise refinement.
  arXiv  Detail & Related papers  (2025-11-12T18:49:21Z)
• When Continuous Delivery Is Not an Option: Practical Paths to Continuous Engineering in Complex Organizations [0.41998444721319217]
  Continuous Software Engineering promises improved efficiency, quality, and responsiveness in software-intensive organizations.<n> fully adopting CSE is often constrained by complex products, legacy systems, organizational inertia, and regulatory requirements.<n>This paper examines four industrial cases from the automation, automotive, retail, and chemical sectors to explore how such constraints shape CSE adoption in practice.
  arXiv  Detail & Related papers  (2025-11-04T10:18:35Z)
• Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization [53.82400605816587]
  Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation.<n>A major challenge lies in the non-stationary nature of quality distributions in real-world scenarios.<n>We introduce Continual AQA (CAQA), which equips with Continual Learning capabilities to handle evolving distributions.
  arXiv  Detail & Related papers  (2025-10-08T10:09:47Z)
• Personality as a Probe for LLM Evaluation: Method Trade-offs and Downstream Effects [0.6087817758152709]
  We present a systematic study of personality control using the Big Five traits.<n>Trait-level analysis shows openness as uniquely challenging, agreeableness as most resistant to ICL.<n>Experiments on Gemma-2-2B-IT and LLaMA-3-8B-Instruct reveal clear trade-offs.
  arXiv  Detail & Related papers  (2025-09-05T04:19:15Z)
• CCRS: A Zero-Shot LLM-as-a-Judge Framework for Comprehensive RAG Evaluation [1.7925692773093167]
  CCRS is a novel suite of five metrics that utilize a single, powerful, pretrained LLM as a zero-shot, end-to-end judge.<n>We apply CCRS to evaluate six diverse RAG system configurations on the challenging BioASQ dataset.
  arXiv  Detail & Related papers  (2025-06-25T04:49:03Z)
• RBF++: Quantifying and Optimizing Reasoning Boundaries across Measurable and Unmeasurable Capabilities for Chain-of-Thought Reasoning [60.84707424369494]
  Chain-of-Thought (CoT) reasoning has proven effective in enhancing large language models (LLMs) on complex tasks.<n>We introduce the Reasoning Boundary Framework++ (RBF++), a framework for evaluating and optimizing measurable boundaries of CoT capability.
  arXiv  Detail & Related papers  (2025-05-19T16:25:55Z)
• FinTSB: A Comprehensive and Practical Benchmark for Financial Time Series Forecasting [58.70072722290475]
  Financial time series (FinTS) record the behavior of human-brain-augmented decision-making.<n>FinTSB is a comprehensive and practical benchmark for financial time series forecasting.
  arXiv  Detail & Related papers  (2025-02-26T05:19:16Z)
• Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs [63.36637269634553]
  We introduce a novel approach where LLMs are fine-tuned to generate a sequence of Diverse Chains of Thought (DCoT) within a single inference step.<n>We show that fine-tuning on DCoT improves performance over the CoT baseline across model families and scales.<n>Our work is also significant because both quantitative analyses and manual evaluations reveal the observed gains stem from the models' ability to refine an initial reasoning chain.
  arXiv  Detail & Related papers  (2024-07-03T15:01:18Z)
• CSL: Class-Agnostic Structure-Constrained Learning for Segmentation Including the Unseen [62.72636247006293]
  Class-Agnostic Structure-Constrained Learning is a plug-in framework that can integrate with existing methods. We propose soft assignment and mask split methodologies that enhance OOD object segmentation. Empirical evaluations demonstrate CSL's prowess in boosting the performance of existing algorithms spanning OOD segmentation, ZS3, and DA segmentation.
  arXiv  Detail & Related papers  (2023-12-09T11:06:18Z)
• Continuous Monte Carlo Graph Search [61.11769232283621]
  Continuous Monte Carlo Graph Search ( CMCGS) is an extension of Monte Carlo Tree Search (MCTS) to online planning. CMCGS takes advantage of the insight that, during planning, sharing the same action policy between several states can yield high performance. It can be scaled up through parallelization, and it outperforms the Cross-Entropy Method (CEM) in continuous control with learned dynamics models.
  arXiv  Detail & Related papers  (2022-10-04T07:34:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.