Polaris: A Gödel Agent Framework for Small Language Models through Experience-Abstracted Policy Repair

• URL: http://arxiv.org/abs/2603.23129v1
• Date: Tue, 24 Mar 2026 12:25:32 GMT
• Title: Polaris: A Gödel Agent Framework for Small Language Models through Experience-Abstracted Policy Repair
• Abstract summary: We introduce Polaris, a Gdel agent for compact models that performs policy repair via experience abstraction.<n>Unlike response level self correction or parameter tuning, Polaris makes policy level changes with small, auditable patches that persist in the policy.<n>As part of the loop, the agent engages in meta reasoning: it explains its errors, proposes concrete revisions to its own policy, and then updates the policy.
• Score: 0.9780193377233132
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Gödel agent realize recursive self-improvement: an agent inspects its own policy and traces and then modifies that policy in a tested loop. We introduce Polaris, a Gödel agent for compact models that performs policy repair via experience abstraction, turning failures into policy updates through a structured cycle of analysis, strategy formation, abstraction, and minimal code pat ch repair with conservative checks. Unlike response level self correction or parameter tuning, Polaris makes policy level changes with small, auditable patches that persist in the policy and are reused on unseen instances within each benchmark. As part of the loop, the agent engages in meta reasoning: it explains its errors, proposes concrete revisions to its own policy, and then updates the policy. To enable cumulative policy refinement, we introduce experience abstraction, which distills failures into compact, reusable strategies that transfer to unseen instances. On MGSM, DROP, GPQA, and LitBench (covering arithmetic reasoning, compositional inference, graduate-level problem solving, and creative writing evaluation), a 7-billion-parameter model equipped with Polaris achieves consistent gains over the base policy and competitive baselines.

Related papers

• Policy Loopholes in Agent Evaluation: When Policy Ambiguity Masquerades as Agent Error [2.0305676256390934]
  Benchmark developers should audit policies before collecting gold annotations.<n>A cross-domain comparison reveals that exploitability requires both policy ambiguity and tool permissiveness.
  arXiv  Detail & Related papers  (2026-09-13T10:00:13Z)
• PolicyMem: Geometric Policy Memory for LLM Governance [72.04962480384322]
  We introduce PolicyMem, a geometric policy memory that externalizes natural-language policies as reusable geometric memory objects.<n>A memory writer compiles natural-language policies into policy memory slots, and query-response pairs read the policy memory through projection energy.<n>The resulting policy-evidence profile directly mediates the safety verdict and is reused for policy attribution and post-intervention verification.
  arXiv  Detail & Related papers  (2026-09-12T06:07:43Z)
• ORPA: Online Residual Policy Adaptation for Robot Manipulation Control with Human Feedback [1.711436622393758]
  Online Residual Policy Adaptation (ORPA) is a framework that enables feedback-driven correction of robot actions without modifying the underlying policy parameters.<n>ORPA augments a pretrained control policy with a lightweight, feedback-conditioned module that predicts residual adjustments directly in joint space.<n>We evaluate ORPA on a set of precision-sensitive manipulation tasks using the ALOHA platform, demonstrating improvements in success rate and recovery from small perturbations compared to baseline control policies and rule-based inverse kinematics corrections.
  arXiv  Detail & Related papers  (2026-08-18T03:30:12Z)
• Structural Distinguishability of Static and Adaptive Policy Regimes in Agent-Based Regulatory Simulation [0.0]
  This paper contributes a controlled simulation benchmark rather than a new general framework.<n>We evaluate naive fixed policies, tracking-aware fixed policies, and three adaptive controllers: setpoint, safety-margin, and one-sided control.<n>The contribution is methodological: scalar indicators, cap-relative symbolic diagnostics, trajectory motifs, and visual inspection jointly reveal how regulatory conclusions can differ even when average outcomes appear similar.
  arXiv  Detail & Related papers  (2026-06-15T10:06:32Z)
• One-Way Policy Optimization for Self-Evolving LLMs [63.8638342097375]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a promising paradigm for scaling reasoning capabilities of Large Language Models (LLMs)<n>We propose One-Way Policy Optimization (OWPO), a method based on the principle of decoupling optimization direction from update magnitude.<n> Experimental results demonstrate that OWPO outperforms strong baselines, including DAPO, OPD, and MOPD.
  arXiv  Detail & Related papers  (2026-05-21T08:25:27Z)
• Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent [53.828537014796574]
  Online off-policy reinforcement learning (RL) is shaped by two choices: the policy class and the update rule.<n>We propose MeanFlow Policies, a one-step generative policy class that maps noise to actions through a MeanFlow transformation.<n>Across seven MuJoCo benchmarks, S improves over Gaussian and generative baselines while retaining single-step inference efficiency.
  arXiv  Detail & Related papers  (2026-05-20T15:14:14Z)
• Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models [5.417332705560665]
  We introduce a new model-agnostic method called Critic-Driven Voronoi State Partitioning.<n>By exploiting the critic value network of the original policy, we introduce new subpolicies in regions with insufficient value.<n>We validate our approach on several well known benchmarks and proof that this distillation approaches the original policy using a reasonable sized set of linear functions.
  arXiv  Detail & Related papers  (2026-05-14T14:38:56Z)
• OGPO: Sample Efficient Full-Finetuning of Generative Control Policies [53.42266064673132]
  Generative control policies (GCPs) have emerged as effective parameterizations for robot learning.<n>This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient for finetuning GCPs.<n>OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control.
  arXiv  Detail & Related papers  (2026-05-04T18:36:40Z)
• Bayesian Conservative Policy Optimization (BCPO): A Novel Uncertainty-Calibrated Offline Reinforcement Learning with Credible Lower Bounds [1.2183405753834562]
  offline reinforcement learning (RL) aims to learn decision policies from a fixed batch of logged transitions.<n>We propose emphBayesian Conservative Policy Optimization (BCPO), a unified framework that converts uncertainty into emphprovably conservative policy improvement.<n>BCPO maintains a hierarchical Bayesian posterior over environment/value models, constructs a emphcredible lower bound (LCB) on action values, and performs policy updates under explicit KL regularization.
  arXiv  Detail & Related papers  (2026-03-06T01:46:02Z)
• Coverage Improvement and Fast Convergence of On-policy Preference Learning [67.36750525893514]
  Online on-policy preference learning algorithms for language model alignment can significantly outperform their offline counterparts.<n>We analyze how the sampling policy's coverage evolves throughout on-policy training.<n>We develop principled on-policy schemes for reward distillation in the general function class setting.
  arXiv  Detail & Related papers  (2026-01-13T10:46:06Z)
• Policy-Conditioned Policies for Multi-Agent Task Solving [53.67744322553693]
  In this work, we propose a paradigm shift that bridges the gap by representing policies as human-interpretable source code.<n>We reformulate the learning problem by utilizing Large Language Models (LLMs) as approximate interpreters.<n>We formalize this process as textitProgrammatic Iterated Best Response (PIBR), an algorithm where the policy code is optimized by textual gradients.
  arXiv  Detail & Related papers  (2025-12-24T07:42:10Z)
• Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning [52.97053840476386]
  We show that well-designed behaviour policies can be used to collect off-policy data for provably lower variance return estimates.<n>We extend this key insight to the online reinforcement learning setting, where both policy evaluation and improvement are interleaved.
  arXiv  Detail & Related papers  (2025-11-13T23:06:40Z)
• Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards [47.557539197058496]
  We introduce Random Policy Valuation for Diverse Reasoning (ROVER)<n>ROVER is a minimalist yet highly effective RL method that samples actions from a softmax over uniform-policy Q-values.<n>It demonstrates superior performance in both textbfquality (textbf+8.2 on pass@1, textbf+16.8 on pass@256) and textbfdiversity (textbf+17.6%)
  arXiv  Detail & Related papers  (2025-09-29T16:09:07Z)
• Statistical Analysis of Policy Space Compression Problem [54.1754937830779]
  Policy search methods are crucial in reinforcement learning, offering a framework to address continuous state-action and partially observable problems. Reducing the policy space through policy compression emerges as a powerful, reward-free approach to accelerate the learning process. This technique condenses the policy space into a smaller, representative set while maintaining most of the original effectiveness.
  arXiv  Detail & Related papers  (2024-11-15T02:46:55Z)
• Periodic agent-state based Q-learning for POMDPs [23.296159073116264]
  A widely used alternative is to use an agent state, which is a model-free,periodicly updateable function of the observation history. We propose PA ( agent-state based Q-learning), which is a variant of agent-state-based Q-learning that learns periodic policies. By combining ideas from periodic Markov chains and approximation, we rigorously establish that PA converges to a cyclic limit and characterize the approximation error of the periodic policy.
  arXiv  Detail & Related papers  (2024-07-08T16:58:57Z)
• Off-Policy Evaluation for Large Action Spaces via Policy Convolution [60.6953713877886]
  Policy Convolution family of estimators uses latent structure within actions to strategically convolve the logging and target policies. Experiments on synthetic and benchmark datasets demonstrate remarkable mean squared error (MSE) improvements when using PC.
  arXiv  Detail & Related papers  (2023-10-24T01:00:01Z)
• Mutual Information Regularized Offline Reinforcement Learning [76.05299071490913]
  We propose a novel MISA framework to approach offline RL from the perspective of Mutual Information between States and Actions in the dataset. We show that optimizing this lower bound is equivalent to maximizing the likelihood of a one-step improved policy on the offline dataset. We introduce 3 different variants of MISA, and empirically demonstrate that tighter mutual information lower bound gives better offline RL performance.
  arXiv  Detail & Related papers  (2022-10-14T03:22:43Z)
• PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning [35.044047991893365]
  This work introduces the the Policy Cover-Policy Gradient (PC-PG) algorithm, which balances the exploration vs. exploitation tradeoff using an ensemble of policies (the policy cover) We show that PC-PG has strong guarantees under model misspecification that go beyond the standard worst case $ell_infty$ assumptions. We also complement the theory with empirical evaluation across a variety of domains in both reward-free and reward-driven settings.
  arXiv  Detail & Related papers  (2020-07-16T16:57:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.