Semantic Voting: Execution-Grounded Consensus for LLM Code Generation

• URL: http://arxiv.org/abs/2605.08680v1
• Date: Sat, 09 May 2026 04:33:39 GMT
• Title: Semantic Voting: Execution-Grounded Consensus for LLM Code Generation
• Abstract summary: Best execution-based selector exceeds output-pattern majority voting by 19-52 percentage points on every configuration.<n>Deeper thinking improves majority voting by 12 pp but execution-based methods stay flat or degrade as candidate diversity falls.<n>These results frame inference-time code selection as a signal-quality problem rather than an aggregation-rule problem.
• Score: 11.731523303184472
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM code-generation pipelines often sample multiple candidates and select one final answer without access to a complete oracle. Existing pipelines mix textual voting, ranking, and execution-based agreement, but the relative contribution of each component remains unclear. We study 18 configurations across different models, thinking levels, and benchmarks, comparing output-pattern majority voting, weighted voting, MBR-Exec, and SemanticVote - a method that clusters candidates by execution fingerprints on LLM-generated inputs. Three findings emerge. (1) The best execution-based selector exceeds output-pattern majority voting by 19-52 percentage points on every configuration, with every execution-based selector exceeding it by at least 18 points. (2) Once candidates are executed on diverse inputs, aggregation rule has limited effect: SemanticVote, weighted voting, and MBR-Exec are statistically indistinguishable across all 18 configurations. The largest factor is input quality: sketch-based input generation consistently outperforms direct LLM generation by 0.6-2.1 pp and random fuzzing by up to 11.3 pp. (3) Thinking level interacts differently with selection families: deeper thinking improves majority voting by 12 pp but execution-based methods stay flat or degrade as candidate diversity falls. These results frame inference-time code selection as a signal-quality problem rather than an aggregation-rule problem: when oracles are unavailable, the behavioral evidence matters more than the aggregation rule.

Related papers

• Partition Scores Are Not System Scores: Deployment-Fidelity Gaps in Decomposed Algorithm Selection [7.302544154053966]
  In decomposed algorithm selection, an analogous partition-level score grants an oracle choice of the best algorithm within the selected family.<n>We define the deployment-fidelity gap G(R) as the difference between partition-level and deployable end-to-end utility.
  arXiv  Detail & Related papers  (2026-09-12T07:55:42Z)
• When Many Answers Are Valid, Voting Fails: Symbolic Verification for Best-of-K Causal Reasoning in LLMs [42.32694162421078]
  We introduce CALVER, a training-free symbolic verifier that scores structured traces against Pearl's causal criteria.<n>On CLEAR find-one-valid queries that admit multiple graph-valid answers, CALVER reaches 42.1% where plurality, a reward model, an LLM judge, and model confidence remain near 30%.
  arXiv  Detail & Related papers  (2026-08-04T11:45:46Z)
• OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregation [53.88666485159289]
  We introduce OpenDeepThink, a population-based test-time compute framework that selects via pairwise Bradley-Terry comparison.<n>OpenDeepThink raises Gemini 3.1 Pro's effective Codeforces Elo by +405 points in eight sequential LLM-call rounds.<n>We release CF-73, a curated set of 73 expert-rated Codeforces problems with International Grandmaster annotation and 99% local-evaluation agreement against the official verdict.
  arXiv  Detail & Related papers  (2026-05-14T17:57:40Z)
• F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking [79.49893545611779]
  Large Language Models (LLMs) can generate a subset and order it within a single autoregressive pass.<n>This flexibility introduces a new optimization challenge: the model must search an output space while receiving utility feedback only after the full ranked list is generated.<n>This credit assignment gap makes end-to-end optimization unstable and sample-inefficient.<n>We propose a unified framework that performs both within a single autoregressive rollout and optimize them end-to-end via factorized group-relative policy optimization.
  arXiv  Detail & Related papers  (2026-05-13T04:52:33Z)
• Parallel Test-Time Scaling with Multi-Sequence Verifiers [23.363130292302483]
  Parallel test-time scaling is a powerful technique for improving large language model performance.<n>It is hindered by two key bottlenecks: accurately selecting the correct solution from the candidate pool, and the high inference latency from generating many full solutions.<n>We introduce the Multi-Sequence Verifier (MSV), the first verifier designed to jointly process all candidate solutions and model their interactions.
  arXiv  Detail & Related papers  (2026-03-03T18:53:56Z)
• dVoting: Fast Voting for dLLMs [71.572316901001]
  Diffusion Large Language Models (dLLMs) represent a new paradigm beyond autoregressive modeling.<n>dLLMs can generate tokens at arbitrary positions in parallel, endowing them with significant potential for parallel test-time scaling.<n>We introduce dVoting, a fast voting technique that boosts reasoning capability without training, with only an acceptable extra computational overhead.
  arXiv  Detail & Related papers  (2026-02-12T16:35:05Z)
• A Single Revision Step Improves Token-Efficient LLM Reasoning [3.344806691289323]
  We introduce Packet-Conditioned Revision (PACER), a training-free, inference-only framework for large language models.<n>PACER enables reasoning traces to revise their conclusions through a structured coordination step.<n>On challenging competitive math benchmarks, PACER matches or exceeds the accuracy of 256-sample majority voting.
  arXiv  Detail & Related papers  (2026-02-02T21:28:42Z)
• Quantifying and Mitigating Selection Bias in LLMs: A Transferable LoRA Fine-Tuning and Efficient Majority Voting Approach [13.829059542429876]
  Multiple Choice Question (MCQ) answering is a widely used method for evaluating the performance of Large Language Models (LLMs)<n>LLMs often exhibit selection bias in MCQ tasks, where their choices are influenced by factors like answer position or option symbols rather than the content.
  arXiv  Detail & Related papers  (2025-11-17T21:31:37Z)
• Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information [57.397381631496906]
  We develop two new aggregation algorithms called Optimal Weight (OW) and Inverse Surprising Popularity (ISP)<n>Our theoretical analysis shows these methods provably mitigate inherent limitations of majority voting under mild assumptions.<n>We empirically validate our algorithms on synthetic datasets, popular LLM fine-tuning benchmarks such as UltraFeedback and MMLU, and a real-world healthcare setting ARMMAN.
  arXiv  Detail & Related papers  (2025-10-01T22:21:50Z)
• From Ranking to Selection: A Simple but Efficient Dynamic Passage Selector for Retrieval Augmented Generation [10.904177110367122]
  We introduce Dynamic Passage Selector (DPS), a novel reranking framework that treats passage selection as a supervised learning problem.<n>As a seamless plug-and-play module, DPS requires no modifications to the standard RAG pipeline.<n>Our results demonstrate that by enabling adaptive evidence selection, DPS substantially enhances reasoning capabilities in complex RAG scenarios.
  arXiv  Detail & Related papers  (2025-08-13T05:05:34Z)
• Efficient Weighting Schemes for Auditing Instant-Runoff Voting Elections [57.67176250198289]
  AWAIRE involves adaptively weighted averages of test statistics, essentially "learning" an effective set of hypotheses to test. We explore schemes and settings more extensively, to identify and recommend efficient choices for practice. A limitation of the current AWAIRE implementation is its restriction to a small number of candidates.
  arXiv  Detail & Related papers  (2024-02-18T10:13:01Z)
• Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency [127.97467912117652]
  Large language models (LLMs) have exhibited remarkable ability in code generation. However, generating the correct solution in a single attempt still remains a challenge. We propose the Multi-Perspective Self-Consistency (MPSC) framework incorporating both inter- and intra-consistency.
  arXiv  Detail & Related papers  (2023-09-29T14:23:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.