Diffusion-MPC in Discrete Domains: Feasibility Constraints, Horizon Effects, and Critic Alignment: Case study with Tetris

• URL: http://arxiv.org/abs/2603.02348v1
• Date: Mon, 02 Mar 2026 19:35:38 GMT
• Title: Diffusion-MPC in Discrete Domains: Feasibility Constraints, Horizon Effects, and Critic Alignment: Case study with Tetris
• Abstract summary: We study diffusion-based model predictive control (Diffusion-MPC) in discrete domains using Tetris as a case study.<n>Our planner samples candidate placement sequences with a MaskGIT-style discrete denoiser and selects actions via reranking.<n>We analyze three key factors: (1) feasibility-constrained sampling via logit masking over valid placements, (2) reranking strategies using a score, a pretrained DQN critic, and a hybrid combination, and (3) compute scaling in candidate count and planning horizon.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We study diffusion-based model predictive control (Diffusion-MPC) in discrete combinatorial domains using Tetris as a case study. Our planner samples candidate placement sequences with a MaskGIT-style discrete denoiser and selects actions via reranking. We analyze three key factors: (1) feasibility-constrained sampling via logit masking over valid placements, (2) reranking strategies using a heuristic score, a pretrained DQN critic, and a hybrid combination, and (3) compute scaling in candidate count and planning horizon. We find that feasibility masking is necessary in discrete domains, removing invalid action mass (46%) and yielding a 6.8% improvement in score and 5.6% improvement in survival over unconstrained sampling. Naive DQN reranking is systematically misaligned with rollout quality, producing high decision regret (mean 17.6, p90 36.6). Shorter planning horizons outperform longer ones under sparse and delayed rewards, suggesting uncertainty compounding in long imagined rollouts. Overall, compute choices (K, H) determine dominant failure modes: small K limits candidate quality, while larger H amplifies misranking and model mismatch. Our findings highlight structural challenges of diffusion planners in discrete environments and provide practical diagnostics for critic integration.

Related papers

• Cost-Sensitive Conformal Prediction and Human-in-the-Loop Abstention for Imbalanced High-Stakes Decision Support: A Multi-Domain Benchmark [2.5130933318193915]
  Mondrian CP restores valid minority-class coverage, achieving an average minority-coverage improvement of 61.7 percentage points over marginal CP.<n>We quantify dataset-specific break-even thresholds at which deferring ambiguous instances to human experts becomes cost-effective.
  arXiv  Detail & Related papers  (2026-07-29T17:15:33Z)
• How Useful is Causal Invariance for Domain Adaptation in Finite-Sample Settings? [58.740078141879984]
  Machine learning models often degrade when they are deployed on a target distribution that differs from the source distributions they were trained on.<n>Recent work in causality-based domain generalization has shown how shared causal structure can induce invariant predictors.<n>In this paper, we study when full or partial causal knowledge can provably improve supervised domain adaptation.
  arXiv  Detail & Related papers  (2026-06-10T21:07:49Z)
• Beyond Mode Collapse: Distribution Matching for Diverse Reasoning [69.88237286885065]
  On-policy reinforcement learning methods like GRPO suffer from mode collapse.<n>We show this stems from reverse KL minimization's mode-seeking behavior.<n>We propose DMPO, which prevents mode collapse through principled approximation of forward KL minimization.
  arXiv  Detail & Related papers  (2026-05-19T07:13:00Z)
• DARTS: Targeting Prognostic Covariates in Budget-Constrained Sequential Experiments [47.61857875238484]
  We introduce Dynamic Adaptive Rerandomization via Thompson Sampling (DARTS)<n>DARTS treats covariate acquisition as a sequential optimization problem embedded within a design-based causal inference task.<n>We derive a Bayes risk bound for the acquisition layer that matches the minimax lower bound up to logarithmic factors.
  arXiv  Detail & Related papers  (2026-05-07T17:27:51Z)
• Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR [7.971447120817466]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become a key approach for improving the reasoning abilities of large language models.<n>We propose Selective Eligibility Traces (S-trace) as a sample-efficient, critic-free eligibility traces method.<n> Experiments demonstrate that S-trace not only outperforms GRPO, showing gains of 0.49% on Qwen3-1.7B and 3.16% on Qwen3-4B, and maintaining a robust 2.98% improvement when scaled further to Qwen3-8B.
  arXiv  Detail & Related papers  (2026-05-07T10:11:51Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference [60.958331943869126]
  ODAR-Expert is an adaptive routing framework that optimize the accuracy-efficiency trade-off via principled resource allocation.<n>We show strong and consistent gains, including 98.2% accuracy on MATH and 54.8% on Humanity's Last Exam.
  arXiv  Detail & Related papers  (2026-02-27T05:22:01Z)
• Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE [27.8012190589404]
  Fine-grained MoE features hundreds of well-trained experts per layer and multi-expert activation per token.<n>We empirically characterize fine-grained MoE routing and uncover an informative pattern: router scores exhibit a certain head of high-confidence experts followed by an uncertain tail of low-confidence candidates.<n>We propose Expert-Sample, a training-free method that preserves high-confidence selections while injecting controlledity into the uncertain tail, enabling diverse generation without destabilizing outputs.
  arXiv  Detail & Related papers  (2026-02-02T18:39:33Z)
• Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answering [52.69447404069251]
  Large vision-language models (VLMs) have improved embodied question answering (EQA) agents by providing strong semantic priors for open-vocabulary reasoning.<n>We propose Prune-Then-Plan, a framework that stabilizes exploration through step-level calibration.
  arXiv  Detail & Related papers  (2025-11-24T22:50:50Z)
• Agentic World Modeling for 6G: Near-Real-Time Generative State-Space Reasoning [70.56067503630486]
  We argue that sixth-generation (6G) intelligence is not fluent token prediction but calibrated the capacity to imagine and choose.<n>We show that WM-MS3M cuts mean absolute error (MAE) by 1.69% versus MS3M with 32% fewer parameters and similar latency, and achieves 35-80% lower root mean squared error (RMSE) than attention/hybrid baselines with 2.3-4.1x faster inference.
  arXiv  Detail & Related papers  (2025-11-04T17:22:22Z)
• Test-time Verification via Optimal Transport: Coverage, ROC, & Sub-optimality [53.03186946689658]
  Test-time scaling with verification has shown promise in improving the performance of large language models.<n>The effect of verification manifests through interactions of three quantities: (i) the generator's coverage, (ii) the verifier's region of convergence (ROC), and (iii) the sampling algorithm's sub-optimality.<n>We frame verifiable test-time scaling as a transport problem. This characterizes the interaction of coverage, ROC, and sub-optimality.
  arXiv  Detail & Related papers  (2025-10-21T18:05:42Z)
• Sample-Centric Multi-Task Learning for Detection and Segmentation of Industrial Surface Defects [3.0497940633809395]
  Industrial surface defect inspection for sample-wise quality control (QC) must simultaneously decide whether a given sample contains defects and localize those defects.<n>Existing models achieve strong pixel-overlap metrics but exhibit insufficient stability at the sample level.<n>We propose a sample-centric multi-task learning framework and evaluation suite.
  arXiv  Detail & Related papers  (2025-10-15T07:24:26Z)
• Multi-Agent Regime-Conditioned Diffusion (MARCD) for CVaR-Constrained Portfolio Decisions [0.0]
  We present MARCD, a generative-to-decision framework with: (i) a Gaussian HMM to infer latent regimes; (ii) a diffusion generator that produces regime-conditioned scenarios; (iii) signal extraction via blended, shrunk moments; and (iv) a governed CVaR epigraph program.
  arXiv  Detail & Related papers  (2025-10-12T20:56:10Z)
• Hierarchical Budget Policy Optimization for Adaptive Reasoning [49.621779447691665]
  We present Hierarchical Budget Policy Optimization (HBPO), a reinforcement learning framework that enables models to learn problem-specific reasoning depths without sacrificing capability.<n>HBPO partitions the exploration space into budget-constrained hierarchies (512-2560 tokens), each with differentiated reward structures that preserve both efficiency incentives and reasoning capabilities.<n>Extensive experiments demonstrate that HBPO reduces average token usage by up to 60.6% while improving accuracy by 3.14% across four reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-07-21T17:52:34Z)
• A Deep Reinforcement Learning Approach to Rare Event Estimation [30.670114229970526]
  An important step in the design of autonomous systems is to evaluate the probability that a failure will occur. In safety-critical domains, the failure probability is extremely small so that the evaluation of a policy through Monte Carlo sampling is inefficient. We develop two adaptive importance sampling algorithms that can efficiently estimate the probability of rare events for sequential decision making systems.
  arXiv  Detail & Related papers  (2022-11-22T18:29:14Z)
• Uncertainty-Driven Action Quality Assessment [11.958132175629368]
  We propose a novel probabilistic model, named Uncertainty-Driven AQA (UD-AQA), to capture the diversity among multiple judge scores.<n>We generate the estimation of uncertainty for each prediction, which is employed to re-weight AQA regression loss.<n>Our proposed method achieves competitive results on three benchmarks including the Olympic events MTL-AQA and FineDiving, and the surgical skill JIGSAWS datasets.
  arXiv  Detail & Related papers  (2022-07-29T07:21:15Z)
• Learning Calibrated Uncertainties for Domain Shift: A Distributionally Robust Learning Approach [150.8920602230832]
  We propose a framework for learning calibrated uncertainties under domain shifts. In particular, the density ratio estimation reflects the closeness of a target (test) sample to the source (training) distribution. We show that our proposed method generates calibrated uncertainties that benefit downstream tasks.
  arXiv  Detail & Related papers  (2020-10-08T02:10:54Z)
• Privacy Preserving Recalibration under Domain Shift [119.21243107946555]
  We introduce a framework that abstracts out the properties of recalibration problems under differential privacy constraints. We also design a novel recalibration algorithm, accuracy temperature scaling, that outperforms prior work on private datasets.
  arXiv  Detail & Related papers  (2020-08-21T18:43:37Z)
• REAK: Reliability analysis through Error rate-based Adaptive Kriging [2.066555810789929]
  This paper introduces Reliability analysis through Error rate-based Adaptive Kriging (REAK) An extension of the Central Limit Theorem based on Lindeberg condition is adopted here to derive the distribution of the number of design samples with wrong sign estimate. Results indicate that REAK is able to reduce the computational demand by as high as 50% compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-02-04T03:47:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.