GEM: Guided Expectation-Maximization for Behavior-Normalized Candidate Action Selection in Offline RL

• URL: http://arxiv.org/abs/2603.23232v1
• Date: Tue, 24 Mar 2026 14:04:43 GMT
• Title: GEM: Guided Expectation-Maximization for Behavior-Normalized Candidate Action Selection in Offline RL
• Abstract summary: We introduce GEM (Guided Expectation-Maximization), an analytical framework that makes action selection both multimodal and controllable.<n>During inference, GEM performs candidate-based selection using a conservative ensemble lower-confidence bound together with behavior-normalized support.<n> Empirically, GEM is competitive across D4RL benchmarks, and offers a simple inference-time budget knob (candidate count) that trades compute for decision quality without retraining.
• Score: 7.213487945222728
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Offline reinforcement learning (RL) can fit strong value functions from fixed datasets, yet reliable deployment still hinges on the action selection interface used to query them. When the dataset induces a branched or multimodal action landscape, unimodal policy extraction can blur competing hypotheses and yield "in-between" actions that are weakly supported by data, making decisions brittle even with a strong critic. We introduce GEM (Guided Expectation-Maximization), an analytical framework that makes action selection both multimodal and explicitly controllable. GEM trains a Gaussian Mixture Model (GMM) actor via critic-guided, advantage-weighted EM-style updates that preserve distinct components while shifting probability mass toward high-value regions, and learns a tractable GMM behavior model to quantify support. During inference, GEM performs candidate-based selection: it generates a parallel candidate set and reranks actions using a conservative ensemble lower-confidence bound together with behavior-normalized support, where the behavior log-likelihood is standardized within each state's candidate set to yield stable, comparable control across states and candidate budgets. Empirically, GEM is competitive across D4RL benchmarks, and offers a simple inference-time budget knob (candidate count) that trades compute for decision quality without retraining.

Related papers

• Selective Ensemble Based on Preference-Directed Multi-Objective Bandits [90.75513823660775]
  We formalize a sequential decision problem under partially specified linear preferences.<n>We then propose a preference-directed upper confidence bound (PrefUCB) algorithm.<n> Experiments on large pre-trained model selective ensemble tasks and online asset allocation under institutional mandates validate our method.
  arXiv  Detail & Related papers  (2026-06-20T07:52:46Z)
• How Useful is Causal Invariance for Domain Adaptation in Finite-Sample Settings? [58.740078141879984]
  Machine learning models often degrade when they are deployed on a target distribution that differs from the source distributions they were trained on.<n>Recent work in causality-based domain generalization has shown how shared causal structure can induce invariant predictors.<n>In this paper, we study when full or partial causal knowledge can provably improve supervised domain adaptation.
  arXiv  Detail & Related papers  (2026-06-10T21:07:49Z)
• gym-invmgmt: An Open Benchmarking Framework for Inventory Management Methods [1.0799969476894555]
  Gym-invmgmt is an extension of the OR-Gym inventory-management lineage for auditable cross-paradigm evaluation.<n>We present gym-invmgmt, a Gymnasium-compatible extension of the OR-Gym inventory-management lineage for auditable cross-paradigm evaluation.
  arXiv  Detail & Related papers  (2026-05-12T00:22:15Z)
• Inference Time Causal Probing in LLMs [26.69306706280534]
  In causal probing, an intervention modifies hidden states so that a property takes on a different value.<n>We propose Hidden-state Driven Margin Intervention (HDMI), a probe-free, gradient-based technique that directly steers hidden states.<n> HDMI consistently achieves higher reliability than prior methods on the LGD agreement corpus and the CausalGym benchmark.
  arXiv  Detail & Related papers  (2026-05-08T11:59:13Z)
• Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration [76.08899010904652]
  CapCal is a training-free framework that mechanically decouples positional bias from ranking decisions.<n>It achieves superior performance among training-free methods while preserving single-pass efficiency.
  arXiv  Detail & Related papers  (2026-04-11T10:47:22Z)
• Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuning [37.39836347901149]
  We assess whether Large Language Models (LLMs) can generate outputs that adhere to a desired target distribution.<n>We formulate distribution alignment using the attributes of gender, race, and sentiment within occupational contexts.<n>We propose a novel fine-tuning framework that couples Steering Token with Semantic Alignment.
  arXiv  Detail & Related papers  (2026-04-07T11:58:16Z)
• ORCH: many analyses, one merge-a deterministic multi-agent orchestrator for discrete-choice reasoning with EMA-guided routing [0.6445605125467574]
  ORCH is a framework for discrete-choice reasoning that orchestrates heterogeneous language models.<n>It uses fixed rules for task decomposition and answer aggregation, keeping the pipeline predictable, reproducible, and training-free.<n>Experiments on MMLU, MMLU-Pro, and GSM8K show that ORCH consistently outperforms single-model baselines and a majority-vote ensemble.
  arXiv  Detail & Related papers  (2026-02-02T08:27:58Z)
• CreditAudit: 2$^\text{nd}$ Dimension for LLM Evaluation and Selection [44.251742023911135]
  CreditAudit is a deployment oriented credit audit framework that evaluates models under a family of semantically aligned and non adversarial system prompt templates.<n>We show that models with similar mean ability can exhibit substantially different fluctuation, and stability risk can overturn prioritization decisions in agentic or high failure cost regimes.
  arXiv  Detail & Related papers  (2026-01-23T07:53:25Z)
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization [133.27496265096445]
  We show how to apply Group Relative Policy Optimization under multi-reward setting without examining its suitability.<n>We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues.<n>GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
  arXiv  Detail & Related papers  (2026-01-08T18:59:24Z)
• Sparsity-Controllable Dynamic Top-p MoE for Large Foundation Model Pre-training [30.589225478300023]
  DTop-p is a sparsity-controllable dynamic Top-p routing mechanism.<n>We show that DTop-p consistently outperforms both Top-k and fixed-threshold Top-p baselines.<n>DTop-p exhibits strong scaling properties with respect to expert granularity, expert capacity, model size, and dataset size.
  arXiv  Detail & Related papers  (2025-12-16T01:28:57Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Multivariate Conformal Selection [9.431551477608528]
  We propose a generalization of Conformal Selection (CS) to provide rigorous uncertainty quantification.<n>We present two variants: mCS-dist, using distance-based scores, and mCS-learn, which learns optimal scores via differentiable optimization.<n>Experiments on simulated and real-world datasets demonstrate that mCS significantly improves selection power while maintaining False Discovery Rate (FDR) control.
  arXiv  Detail & Related papers  (2025-05-01T23:33:57Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Policy Gradient With Serial Markov Chain Reasoning [10.152838128195468]
  We introduce a new framework that performs decision-making in reinforcement learning as an iterative reasoning process. We show our framework has several useful properties that are inherently missing from traditional RL. Our resulting algorithm achieves state-of-the-art performance in popular Mujoco and DeepMind Control benchmarks.
  arXiv  Detail & Related papers  (2022-10-13T06:15:29Z)
• Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic [59.94347858883343]
  This paper investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states.
  arXiv  Detail & Related papers  (2021-02-24T01:11:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.