Process Supervision via Verbal Critique Improves Reasoning in Large Language Models

• URL: http://arxiv.org/abs/2604.21611v1
• Date: Thu, 23 Apr 2026 12:36:12 GMT
• Title: Process Supervision via Verbal Critique Improves Reasoning in Large Language Models
• Abstract summary: Verbal Process Supervision (VPS) is a training-free framework that uses structured natural-language critique from a stronger supervisor to guide an iterative generate-critique-refine loop up to a round budget R.<n>VPS yields three key results across GPQA Diamond, AIME 2025, and LiveCodeBench V6.<n>These results establish critique granularity as a new axis of inference-time scaling.
• Score: 1.472161528588343
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Inference-time scaling for LLM reasoning has focused on three axes: chain depth, sample breadth, and learned step-scorers (PRMs). We introduce a fourth axis, granularity of external verbal supervision, via Verbal Process Supervision (VPS), a training-free framework that uses structured natural-language critique from a stronger supervisor to guide an iterative generate-critique-refine loop up to a round budget R. Across GPQA Diamond, AIME 2025, and LiveCodeBench V6 (covering both closed and open models), VPS yields three key results. First, on GPQA Diamond, GPT-5.4 (High) | GPT-5.4 (Low) reaches 94.9% at R=4, surpassing the 94.1% state of the art without gradient updates. Second, on AIME 2025, VPS enables strong weak-actor rescue, boosting scores from 11.7-26.7% to 63.3-90.0% (up to +63.3 points). Third, at matched compute, VPS outperforms Reflexion by +8.5 to +12.1 points and Self-Consistency@5 by +5.0 pp (GPQA) and +8.3 pp (LiveCodeBench), isolating critique granularity as the key driver. Performance scales with the supervisor-actor capability gap (Pearson r=0.90) and degrades when errors are not linguistically expressible (e.g., code synthesis), motivating hybrid verbal-executable methods. These results establish critique granularity as a new axis of inference-time scaling.

Related papers

• Evaluating Fine-Tuning and Metrics for Neural Decompilation of Dart AOT Binaries [0.0]
  We present a systematic empirical study of fine-tuning effectiveness and validity for neural decompilation.<n>We evaluate six finetuned model variants across three base architectures using three metrics: CodeBLEU, compile@k, and pass@k.<n>Crosslingual interference from Swift training is highly significant at 4B (-2.66 pp, p but statistically indistinguishable from zero at 8B, consistent with the scaling hypothesis)
  arXiv  Detail & Related papers  (2026-07-07T10:36:12Z)
• PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation [17.919178151901573]
  We show that standard on-policy distillation (OPD) for large language models suffers from severe training pathologies in practice.<n>PowerOPD is a sign-consistent reward from the Box-Cox power transformation, parameterized by alpha > 0, of which the log-ratio is the degenerate alpha -> 0 limit.<n>It achieves benchmark-averaged Avg@8/Pass@8 gains of up to +6.37/+5.71 over vanilla OPD, +3.01/+3.54 over post-hoc stabilization, and +2.59/+8.90 over full-vocabulary OPD.
  arXiv  Detail & Related papers  (2026-06-15T18:37:51Z)
• A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedback [4.062253229926774]
  A-ProS is an autonomous AI agent that solves competitive programming problems through a hybrid multi-model feedback framework.<n>A-ProS achieves over 2x greater gains compared to baseline agent loops.
  arXiv  Detail & Related papers  (2026-05-18T08:55:30Z)
• In-Context Environments Induce Evaluation-Awareness in Language Models [0.12691047660244334]
  Humans often become more self-aware under threat, yet can lose self-awareness when absorbed in a task.<n>We introduce a black-box adversarial optimization framework treating the in-context prompt as an optimizable environment.<n>We show that adversarially optimized prompts pose a substantially greater threat to evaluation reliability than previously understood.
  arXiv  Detail & Related papers  (2026-03-04T08:22:02Z)
• UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment [17.430091762374744]
  We show that for subjective perception tasks, this alignment can be achieved without any model training.<n>We propose a training-free post-hoc concept-bottleneck pipeline consisting of three tightly coupled stages.
  arXiv  Detail & Related papers  (2026-02-23T02:24:55Z)
• Once Upon an Input: Reasoning via Per-Instance Program Synthesis [19.86168542588911]
  We introduce Per-Instance Program Synthesis (PIPS), a method that generates and refines programs at the instance-level using structural feedback.<n>To further improve performance, PIPS incorporates a confidence metric that dynamically chooses between direct inference and program synthesis on a per-instance basis.
  arXiv  Detail & Related papers  (2025-10-26T21:58:33Z)
• Think Twice, Generate Once: Safeguarding by Progressive Self-Reflection [18.467741067831877]
  We introduce Progressive Self-Reflection, a novel inference-time technique that empowers large language models to self-monitor and correct their outputs dynamically.<n> Experimental results demonstrate that applying our proposed method to Llama-3.1-8B-Instruct reduces the attack success rate from 77.5% to 5.9%.<n>Our approach acts as a test-time scaling method, where additional self-reflection rounds enhance safety at the cost of inference overhead.
  arXiv  Detail & Related papers  (2025-09-29T12:54:28Z)
• Weights-Rotated Preference Optimization for Large Language Models [30.25242193651982]
  We propose a novel Weights-Rotated Preference Optimization (RoPO) algorithm, which implicitly constrains the output layer logits with the KL divergence inherited from DPO.<n>Our RoPO achieves up to a 3.27-point improvement on AlpacaEval 2, and surpasses the best baseline by 6.2 to 7.5 points on MT-Bench with merely 0.015% of the trainable parameters.
  arXiv  Detail & Related papers  (2025-08-25T03:57:17Z)
• Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback [59.078756231841574]
  Critique-GRPO is an online RL framework that integrates both natural language and numerical feedback for effective policy optimization.<n>We show Critique-GRPO consistently outperforms supervised learning and RL-based fine-tuning methods across eight challenging mathematical, STEM, and general reasoning tasks.
  arXiv  Detail & Related papers  (2025-06-03T17:39:02Z)
• Skywork Open Reasoner 1 Technical Report [51.403686909760914]
  We present Skywork-OR1, an effective and scalable reinforcement learning (RL) implementation for long Chain-of-Thought (CoT) models.<n>Building on the DeepSeek-R1-Distill model series, our RL approach achieves notable performance gains.<n>Our Skywork-OR1-32B model surpasses both DeepSeek-R1 and Qwen3-32B on the AIME24 and AIME25 benchmarks.
  arXiv  Detail & Related papers  (2025-05-28T12:56:04Z)
• Learning Adaptive Parallel Reasoning with Language Models [70.1745752819628]
  We propose Adaptive Parallel Reasoning (APR), a novel reasoning framework that enables language models to orchestrate both serialized and parallel computations end-to-end.<n> APR generalizes existing reasoning methods by enabling adaptive multi-threaded inference using spawn() and join() operations.<n>A key innovation is our end-to-end reinforcement learning strategy, optimizing both parent and child inference threads to enhance task success rate without requiring predefined reasoning structures.
  arXiv  Detail & Related papers  (2025-04-21T22:29:02Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• Preference Optimization for Reasoning with Pseudo Feedback [100.62603571434167]
  We introduce a novel approach to generate pseudo feedback for reasoning tasks by framing the labeling of solutions as an evaluation against associated test cases.<n>We conduct experiments on both mathematical reasoning and coding tasks using pseudo feedback for preference optimization, and observe improvements across both tasks.
  arXiv  Detail & Related papers  (2024-11-25T12:44:02Z)
• Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding [74.31981011985681]
  Large language models (LLMs) have shown impressive capabilities, but still struggle with complex reasoning tasks requiring multiple steps. We introduce LaTent Reasoning Optimization (LaTRO), a principled framework that formulates reasoning as sampling from a latent distribution. We validate LaTRO through experiments on GSM8K and ARC-Challenge datasets using multiple model architectures.
  arXiv  Detail & Related papers  (2024-11-06T22:02:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.