Code-Guided Reasoning for Small Language Models: Evaluating Executable MCQA Scaffolds

• URL: http://arxiv.org/abs/2605.18827v1
• Date: Tue, 12 May 2026 20:20:06 GMT
• Title: Code-Guided Reasoning for Small Language Models: Evaluating Executable MCQA Scaffolds
• Abstract summary: We introduce Code-Guided Reasoning (CGR), an evaluation protocol and generated-program resource.<n>CGR standardizes six components: a normalized item interface, a direct solver prompt, a generator prompt, a Python scaffold, solver-call and extraction helpers, and a three-channel result record.<n>On 20,498 retained result rows from a locally prepared MCQA bundle and six metadata-registered solver models, the observed non-zero-baseline partition shows 66.21% macro assisted accuracy versus 38.11% direct accuracy.
• Score: 14.64394876057282
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multiple-choice QA benchmarks usually evaluate small language models (SLMs) as direct answerers, but deployed language-model systems increasingly rely on external scaffolds such as tools, code, and repeated model calls. We introduce Code-Guided Reasoning (CGR), an evaluation protocol and generated-program resource for measuring when executable reasoning scaffolds improve SLM performance on MCQA tasks. CGR standardizes six components: a normalized item interface, a direct solver prompt, a generator prompt, a Python scaffold, solver-call and extraction helpers, and a three-channel result record. On 20,498 retained result rows from a locally prepared MCQA bundle and six metadata-registered solver models, the observed non-zero-baseline partition shows 66.21% macro assisted accuracy versus 38.11% direct accuracy, a +28.10 percentage-point difference with a pair-bootstrap interval of [20.32, 36.43]. Under a stricter Ab > 30% direct-signal gate, the macro difference is +14.11 points. These estimates are descriptive. Assisted inference uses a larger solver-call budget, answer extraction is brittle, Time-MQA contains the observed regressions, and some generated programs violate the no-hard-coding instruction. CGR provides the trace package needed to interpret these results, including direct, assisted, and generator-side answers, partition definitions, generated programs, response metadata, and audits.

Related papers

• Joint Optimization of Tool Creation and Use for Large Language Model Agents [69.07066297088187]
  SMITH (-grounded Multi-task Iterative Tool Honing) trains tool creation and tool use inside a single policy.<n>A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.8 macro-average accuracy on held-out tasks.<n>It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline)
  arXiv  Detail & Related papers  (2026-08-25T13:59:34Z)
• A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordination [51.06539604709775]
  We propose A-SR, a self-evolving agentic framework that shifts the control unit from expression edits to role-conditioned evidence views.<n>A-SR coordinates formula discovery through routing among coordination protocols, an online evaluator-reward role policy, and state-routed process memory.
  arXiv  Detail & Related papers  (2026-08-05T14:01:10Z)
• SPL: Orchestrating Workflows with Declarative Deterministic-Probabilistic Composition [0.0]
  SPL is a declarative language that composes deterministic computation and probabilistic computation modes in a single specification.<n>A.spl specification runs unchanged across local nodes, cloud APIs, and distributed grids.<n>We validate SPL through an extensive 78-recipe cookbook and a controlled 1,200-run experiment.
  arXiv  Detail & Related papers  (2026-07-06T03:53:10Z)
• Trace2Policy: From Expert Behavior Traces to Self-Evolving Decision Agents [5.689042186242701]
  Decision rules that enterprise experts apply tacitly can be systematically recovered and improved through iterative error analysis.<n>We present textbfTrace2Policy, whose core mechanism is textbfEISR.<n>Each round executes the rules on a validation set, clusters errors by root cause into MISSING, WRONG, or CONFLICT types, applies targeted patches, and commits only those that pass a regression gate.
  arXiv  Detail & Related papers  (2026-06-09T06:05:29Z)
• Prompt Codebooks: Discrete Compositional Optimization for Language Model Instruction Refinement [10.441136483320205]
  We propose Prompt Codebooks, a novel compositional prompt optimization framework.<n>It recasts APO as discrete learning over a finite vocabulary of natural-language instincts.<n>It improves over zero-shot by up to +30.36 points, surpasses the strongest prior baseline (GEPA) by +3.34 on HotpotQA and +1.11 in aggregate.
  arXiv  Detail & Related papers  (2026-05-27T11:57:12Z)
• Spatial Atlas: Compute-Grounded Reasoning for Spatial-Aware Research Agent Benchmarks [1.5229257192293202]
  We introduce compute-grounded reasoning (CGR) for spatial-aware research agents.<n>CGR is a design paradigm for spatial-aware research agents in which every answerable sub-problem is resolved by deterministic computation.<n>We show that CGR yields competitive accuracy while maintaining interpretability.
  arXiv  Detail & Related papers  (2026-04-13T22:22:07Z)
• DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensembles [5.647839536820347]
  We introduce DiscoUQ, a framework that extracts and leverages the structure of inter-agent disagreement to produce well-calibrated confidence estimates.<n>DiscoUQ-LLM achieves an average AUROC of 0.802, outperforming the best baseline.<n>The learned features generalize across benchmarks with near-zero performance degradation.
  arXiv  Detail & Related papers  (2026-03-21T23:24:12Z)
• SPARC: Scenario Planning and Reasoning for Automated C Unit Test Generation [1.0010193170880752]
  We introduce a neuro-symbolic, scenario-based framework that bridges the gap between high-level program intent and the rigid syntactic constraints of pointer arithmetic and manual memory management.<n>We evaluate it on 59 real-world and algorithmic subjects, where it outperforms the vanilla prompt generation baseline by 31.36% in line coverage, 26.01% in branch coverage, and 20.78% in mutation score, matching or exceeding the symbolic execution tool KLEE.
  arXiv  Detail & Related papers  (2026-02-18T18:09:03Z)
• ZeroGR: A Generalizable and Scalable Framework for Zero-Shot Generative Retrieval [125.19156877994612]
  Generative retrieval (GR) reformulates information retrieval (IR) by framing it as the generation of document identifiers (docids)<n>We propose textscZeroGR, a zero-shot generative retrieval framework that leverages natural language instructions to extend GR across a wide range of IR tasks.<n>Specifically, textscZeroGR is composed of three key components: (i) an LM-based docid generator that unifies heterogeneous documents into semantically meaningful docids; (ii) an instruction-tuned query generator that generates diverse types of queries from natural language task descriptions to enhance
  arXiv  Detail & Related papers  (2025-10-12T03:04:24Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• Divide by Question, Conquer by Agent: SPLIT-RAG with Question-Driven Graph Partitioning [62.640169289390535]
  SPLIT-RAG is a multi-agent RAG framework that addresses the limitations with question-driven semantic graph partitioning and collaborative subgraph retrieval.<n>The innovative framework first create Semantic Partitioning of Linked Information, then use the Type-Specialized knowledge base to achieve Multi-Agent RAG.<n>The attribute-aware graph segmentation manages to divide knowledge graphs into semantically coherent subgraphs, ensuring subgraphs align with different query types.<n>A hierarchical merging module resolves inconsistencies across subgraph-derived answers through logical verifications.
  arXiv  Detail & Related papers  (2025-05-20T06:44:34Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• Enhancing IR-based Fault Localization using Large Language Models [5.032687557488094]
  This paper enhances Fault Localization (IRFL) by categorizing bug reports based on programming entities, stack traces, and natural language text.<n>To address inaccuracies in queries, we introduce a user and conversational-based query reformulation approach, termed LLmiRQ+.<n> Evaluation on 46 projects with 6,340 bug reports yields an MRR of 0.6770 and MAP of 0.5118, surpassing seven state-of-the-art IRFL techniques.
  arXiv  Detail & Related papers  (2024-12-04T22:47:51Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.