CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2608.07719v1
• Date: Fri, 07 Aug 2026 19:11:54 GMT
• Title: CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning
• Abstract summary: We introduce CODS, a critic-guided selector that alternates between fitting an algorithm-matched critic and acquiring high-residual transitions.<n>At a 10% budget, CODS retains 96.6% of eligible-pool performance across 20 valid D4RL task--algorithm cells.
• Score: 5.894992157122175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Offline reinforcement learning repeatedly trains policies from a fixed transition pool, making redundant data costly across seeds and hyperparameters, while naive subsampling can remove rare transitions needed for long-horizon credit assignment. We introduce CODS, a critic-guided selector that alternates between fitting an algorithm-matched critic and acquiring high-residual transitions before freezing a reusable subset. Unlike prioritized replay, CODS produces a static artifact; unlike one-shot residual selection, it refreshes scores as the critic changes. At a 10\% budget, CODS retains 96.6\% of eligible-pool performance across 20 valid D4RL task--algorithm cells. It exceeds ReDOR and OPER on 19/20 cells and every other subset baseline on 20/20; all six subset advantages remain significant under predeclared hierarchical inference with Holm correction. Holding total selector updates fixed, five acquisition rounds improve four representative cells by 11.23 points over one round and saturate thereafter. Equal-pass and equal-hour evaluations clarify that reuse, rather than a single-run speedup, creates the compute advantage. Mechanism and corruption interventions expose both useful sparse-reward enrichment and sensitivity to outliers. Finally, a whole-trace extension retains 95.4\% of pooled ALFWorld success and 96.5\% of pooled GSM8K exact match. CODS is therefore a reusable selection procedure, not a formal coreset guarantee.

Related papers

• ReNFT: Repairing Mode Collapse in Reward Post-Training via Internal Probability-Mass Recalibration [84.10540890311843]
  Reward post-training inevitably concentrates probability mass on a few reward-favored modes, a mode collapse that erases within-prompt diversity.<n>We propose ReNFT, which repairs a high-reward, low-diversity adapter through internal probability-mass recalibration.<n>On PickScore and GenEval, ReNFT retains 98.9% and 99.0% of NFT's reward while improving DreamSim-Div by 58.8% and 55.0%, respectively.
  arXiv  Detail & Related papers  (2026-08-30T14:13:50Z)
• Efficient GPU Retrieval for Semantic Search [11.631441668563802]
  We present a policy-aligned retrieval framework for LinkedIn.<n>A lightweight single-slot Stage-1 scorer generates high-recall candidates.<n>On 21K held-out queries, this representation improves offline relevance over a matched-capacity baseline.
  arXiv  Detail & Related papers  (2026-08-29T00:38:27Z)
• RecGPT-Mobile-V2 Technical Report [22.10290138930905]
  RecGPT-Mobile-V2 is an end-to-end framework that treats intent quality and execution efficiency as coupled objectives within a staged design.<n>The framework transforms heterogeneous interactions into an evidence-preserving trajectory, establishes a recommendation-native foundation through domain adaptation and supervised alignment.<n>Online retrieval analysis indicates that the Query recall channel retrieves inventory complementary to that surfaced by established recall channels.
  arXiv  Detail & Related papers  (2026-08-25T09:23:17Z)
• DG-FedReuse: Proxy-Gradient-Gated Cached-Update Reuse with Matched Sparse Uplink Accounting [0.0]
  DG-FedReuse is a simulator-level mechanism that allows selected clients to contribute cached updates.<n>Experiments cover six image-classification datasets, 50 virtual clients, Dirichlet label clients, and three seeds.<n> DG-FedReuse yields 83.36-85.42% modeled update-data-field uplink saving, compared with 76.88% for matched Top-K FedAvg.
  arXiv  Detail & Related papers  (2026-08-05T19:30:04Z)
• ISO: An RLVR-Native Optimization Stack [50.40395312897848]
  We operationalize spectral inheritance as Isospectral Optimization (ISO)<n>Offline, ISO-Merger combines the frame changes of shared-base specialists into a single fixed-spectrum model.<n>Online, ISO-Merger applies a chosen base, including AdamW and Muon, to the frame variables while keeping the base spectra fixed.
  arXiv  Detail & Related papers  (2026-07-21T17:51:36Z)
• Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0 [53.71882250666667]
  Most reported gains from agent-optimization methods are one-shot.<n>This does not test the setting that matters for deployed agents.<n>We compare three approaches to agent-harness optimization.
  arXiv  Detail & Related papers  (2026-07-15T16:36:04Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking [22.825786049667602]
  We propose a replication-first paradigm to anchor validity to a single human-rater consensus.<n>We certify the instrument via four properties -- reliability across K runs, cross-instrument replication across architecturally distinct judges, historical-footprint calibration via judges from earlier training cohorts, and pre-registered prediction.<n>We test it on emotional accompaniment by letting the self-evolve data-driven across: the dimensions are not pre-stipulated and the procedure stabilizes to a 9-dimension set.
  arXiv  Detail & Related papers  (2026-05-27T03:41:11Z)
• ReacTOD: Bounded Neuro-Symbolic Agentic NLU for Zero-Shot Dialogue State Tracking [2.2573512203799626]
  We propose ReacTOD, a bounded neuro-symbolic architecture that reformulates NLU as discrete tool calls within a self-correcting ReAct loop governed by deterministic validation.<n>A bounded ReAct loop enables iterative self-correction, improving accuracy by up to 9.3 percentage points over single-pass inference on MultiWOZ.
  arXiv  Detail & Related papers  (2026-05-18T20:06:04Z)
• Step-wise Rubric Rewards for LLM Reasoning [72.17879367869503]
  Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models.<n>We find 18.2% of steps in correct responses are wrong yet positively rewarded.<n>Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points.
  arXiv  Detail & Related papers  (2026-05-17T07:08:14Z)
• Multi-Dimensional Behavioral Evaluation of Agentic Stock Prediction Systems Using Large Language Model Judges with Closed-Loop Reinforcement Learning Feedback [1.2362187555287152]
  Forecast evaluation in finance has relied on aggregate accuracy metrics and predictive-accuracy tests built on point-forecast errors.<n>We propose a behavioral forecast-evaluation methodology that complements accuracy tests by assessing the intermediate decision process itself.
  arXiv  Detail & Related papers  (2026-05-07T06:31:34Z)
• Perturbation Dose Responses in Recursive LLM Loops: Raw Switching, Stochastic Floors, and Persistent Escape under Append, Replace, and Dialog Updates [0.0]
  We study how much injected text is needed to move a settled loop somewhere else, and whether that move lasts.<n>Under a 12,000-character tail clip, destination-coherent persistence plateaus near 16 percent and retained source-basin escape near 36 percent at dose 400.<n>A four-step falsification battery recasts the high-dose destination-coherent dip as a finite-horizon, endpoint-definition-sensitive feature.
  arXiv  Detail & Related papers  (2026-05-04T05:16:43Z)
• Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration [76.08899010904652]
  CapCal is a training-free framework that mechanically decouples positional bias from ranking decisions.<n>It achieves superior performance among training-free methods while preserving single-pass efficiency.
  arXiv  Detail & Related papers  (2026-04-11T10:47:22Z)
• Non-Parametric Adaptive Network Pruning [125.4414216272874]
  We introduce non-parametric modeling to simplify the algorithm design. Inspired by the face recognition community, we use a message passing algorithm to obtain an adaptive number of exemplars. EPruner breaks the dependency on the training data in determining the "important" filters.
  arXiv  Detail & Related papers  (2021-01-20T06:18:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.