Adaptive Compute in Latent World Models: When Depth Helps, Hurts, or Doesn't Matter

• URL: http://arxiv.org/abs/2607.10203v3
• Date: Sat, 18 Jul 2026 16:02:20 GMT
• Title: Adaptive Compute in Latent World Models: When Depth Helps, Hurts, or Doesn't Matter
• Abstract summary: We show that depth helps (intrinsic, 6/9 tasks, rho up to 8x), depth actively hurts (inversion, 2/9, rho down to 0.87x), or depth barely matters (flat)<n>Inversion reproduces under a transformer predictor, yet its manifestation is configuration-dependent, shifting with metric space, horizon, encoder, backbone, and -- most strongly -- training data.<n>Whether more compute helps a world model is not a task property; it is a property of the operating configuration, with a stable, predictable, mechanism-backed core.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adaptive compute for world models -- early-exit or mixture-of-depths predictors that spend variable depth per rollout step -- presumes that extra depth buys better predictions. In autoregressive rollouts, where planning actually happens, that premise requires depth's per-step precision to survive composition. We test it directly with one pre-registered instrument, the shallow penalty rho = err(shallowest-exit rollout)/err(full-depth rollout), on nine DeepMind Control tasks under matched single-step (K=1) and multi-step (K=4) training, eight seeds each. Three regimes emerge: depth helps (intrinsic, 6/9 tasks, rho up to 8x), depth actively hurts (inversion, 2/9, rho down to 0.87x), or depth barely matters (flat). The inversion is created by training, not the dynamics: supervising early exits only at the first rollout step erases it (Delta=+0.28, n=8, non-overlapping distributions) -- a routability catch-22: the per-step deep supervision that makes exits routable also trains them to out-roll the full stack. The regime is predictable: a frozen dimensionality-only classifier, committed before training, labels held-out tasks correctly out-of-sample, including an extreme extrapolation. The inversion reproduces under a transformer predictor, yet its manifestation is configuration-dependent, shifting with metric space, horizon, encoder, backbone, and -- most strongly -- training data: on the two tasks we retrained, competent-policy data removes both the inversion and the intrinsic tradeoff, loss unchanged. In a CEM planner, rho predicts whether planning benefits from depth. Every threshold and gate was committed before the corresponding compute, including a pre-registered negative for the motivating hypothesis. Whether more compute helps a world model is not a task property; it is a property of the operating configuration, with a stable, predictable, mechanism-backed core.

Related papers

• Online Distributional Prediction via Latent Cluster Geometry Under Drift and Corruption [14.35871574542677]
  We study online distributional prediction under drift and adversarial corruption.<n>A Gibbs quasi-posterior over these configurations yields an online predictor by posterior averaging.<n>We evaluate performance by cumulative Wasserstein-1 regret against the time-varying true law.
  arXiv  Detail & Related papers  (2026-06-17T07:41:41Z)
• Exact equivariance, kept through training, buys zero-shot generalisation across the symmetry group [6.230579198456525]
  A latent world model built from an equivariant encoder $E$ and an equivariant predictor $f$ inherits a provable symmetry of its training loss.<n>We verify this end-to-end at laptop scale (CPU/MPS, fully seeded)
  arXiv  Detail & Related papers  (2026-06-02T01:20:24Z)
• Nonstationary Generalized Linear Bandits with Discounted Online Mirror Descent [39.805192541498634]
  We study nonstationary linear computation (GLBs), where the expected reward is modeled through a nonlinear link function with an unknown time-varying parameter.<n>We propose a new algorithm for nonstationary GLBs that utilize discounted online mirror descent (DOMD) for parameter estimation, thereby incurring only $O(1)$ for memory costs per round.
  arXiv  Detail & Related papers  (2026-05-25T08:40:32Z)
• Dynamic Mode Decomposition along Depth in Vision Transformers [2.899294572150795]
  We ask whether ViT depth implements approximately textitautonomous linear dynamics.<n>We test this using Dynamic Mode Decomposition (DMD), which fits $K$ from selected, consecutive hidden-state pairs.<n>On four pretrained DINO ViTs, we study the regularization, rank, and calibration budget required for stable fitting.
  arXiv  Detail & Related papers  (2026-05-08T10:33:03Z)
• Recover to Predict: Progressive Retrospective Learning for Variable-Length Trajectory Prediction [34.346912198889505]
  Trajectory prediction is critical for autonomous driving, enabling safe and efficient planning in dense, dynamic traffic.<n>Most existing methods optimize prediction accuracy under fixed-length observations.<n>A common strategy is to directly map features from incomplete observations to those from complete ones.<n>This one-shot mapping, however, struggles to learn accurate representations for short trajectories due to significant information gaps.<n>We propose a Progressive Retrospective Framework (PRF), which gradually aligns features from incomplete observations with those from complete ones.
  arXiv  Detail & Related papers  (2026-03-11T09:51:22Z)
• BASIS: Balanced Activation Sketching with Invariant Scalars for "Ghost Backpropagation" [0.0]
  Activation memory required for exact backpropagation scales linearly with network depth, context length, and feature dimensionality.<n>In this paper, we introduce an efficient backpropagation algorithm that fully decouples activation memory from the batch and sequence dimensions.
  arXiv  Detail & Related papers  (2026-03-05T20:38:25Z)
• Arithmetic-Mean $μ$P for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets [9.94514344279733]
  Arithmetic-Mean $mu$P constrains not each individual layer but the network-wide average one-step pre-activation second moment to a constant scale.<n>We prove that, for one- and two-dimensional convolutional networks, the maximal-update learning rate satisfies $etastar(L)propto L-3/2$; with zero padding, boundary effects are constant-level as $Ngg k$.
  arXiv  Detail & Related papers  (2025-10-05T19:22:50Z)
• Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration [61.350777880329815]
  Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful paradigm for unlocking reasoning capabilities in large language models.<n>We show that RLVR's full potential is hindered by two under-explored dimensions: depth-the hardest problem a model can sample; Breadth-the number of instances consumed in a single iteration.<n>We introduce Difficulty Adaptive Rollout Sampling (DARS), which re-weights hard problems through targeted multi-stage rollouts.
  arXiv  Detail & Related papers  (2025-08-19T11:51:40Z)
• Reinforcement Learning from Adversarial Preferences in Tabular MDPs [62.73758165845971]
  We introduce a new framework of episodic Markov decision processes (MDPs) with adversarial preferences.<n>Unlike standard episodic MDPs with adversarial losses, in PbMDPs the learner instead observes preferences between two candidate arms.<n>We develop algorithms that achieve a regret bound of order $T2/3$ under known transitions.
  arXiv  Detail & Related papers  (2025-07-15T20:19:32Z)
• From Continual Learning to SGD and Back: Better Rates for Continual Linear Models [50.11453013647086]
  We analyze the forgetting, i.e., loss on previously seen tasks, after $k$ iterations.<n>We develop novel last-iterate upper bounds in the realizable least squares setup.<n>We prove for the first time that randomization alone, with no task repetition, can prevent catastrophic in sufficiently long task sequences.
  arXiv  Detail & Related papers  (2025-04-06T18:39:45Z)
• Position-Aware Depth Decay Decoding ($D^3$): Boosting Large Language Model Inference Efficiency [26.173523821684306]
  A token-position aware layer skipping framework is proposed to save 1.5x times operations efficiently while maintaining performance.<n>Experiments on large language models with $7 sim 70$ billion parameters show that $D3$ can achieve an average 1.5x speedup compared with the full-inference pipeline.
  arXiv  Detail & Related papers  (2025-03-11T15:15:54Z)
• Scalable 3D Registration via Truncated Entry-wise Absolute Residuals [65.04922801371363]
  A $3$D registration approach can process more than ten million ($107$) point pairs with over $99%$ random outliers. We call our method TEAR, as it involves minimizing an outlier-robust loss that computes Truncated Entry-wise Absolute Residuals.
  arXiv  Detail & Related papers  (2024-04-01T04:43:39Z)
• Pre-training on Synthetic Driving Data for Trajectory Prediction [61.520225216107306]
  We propose a pipeline-level solution to mitigate the issue of data scarcity in trajectory forecasting. We adopt HD map augmentation and trajectory synthesis for generating driving data, and then we learn representations by pre-training on them. We conduct extensive experiments to demonstrate the effectiveness of our data expansion and pre-training strategies.
  arXiv  Detail & Related papers  (2023-09-18T19:49:22Z)
• Attention Map Guided Transformer Pruning for Edge Device [98.42178656762114]
  Vision transformer (ViT) has achieved promising success in both holistic and occluded person re-identification (Re-ID) tasks. We propose a novel attention map guided (AMG) transformer pruning method, which removes both redundant tokens and heads. Comprehensive experiments on Occluded DukeMTMC and Market-1501 demonstrate the effectiveness of our proposals.
  arXiv  Detail & Related papers  (2023-04-04T01:51:53Z)
• Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction [12.86325214182021]
  Dynamic model pruning is a recent direction that allows for the inference of a different sub-network for each input sample during deployment. Current dynamic methods rely on learning a continuous channel gating through regularization by inducing sparsity loss. We show experiments on several neural architectures, such as VGG, ResNet, and MobileNet on CIFAR and ImageNet.
  arXiv  Detail & Related papers  (2021-10-15T17:39:53Z)
• Regressive Domain Adaptation for Unsupervised Keypoint Detection [67.2950306888855]
  Domain adaptation (DA) aims at transferring knowledge from a labeled source domain to an unlabeled target domain. We present a method of regressive domain adaptation (RegDA) for unsupervised keypoint detection. Our method brings large improvement by 8% to 11% in terms of PCK on different datasets.
  arXiv  Detail & Related papers  (2021-03-10T16:45:22Z)
• Variance-Aware Confidence Set: Variance-Dependent Bound for Linear Bandits and Horizon-Free Bound for Linear Mixture MDP [76.94328400919836]
  We show how to construct variance-aware confidence sets for linear bandits and linear mixture Decision Process (MDP) For linear bandits, we obtain an $widetildeO(mathrmpoly(d)sqrt1 + sum_i=1Ksigma_i2) regret bound, where $d is the feature dimension. For linear mixture MDP, we obtain an $widetildeO(mathrmpoly(d)sqrtK)$ regret bound, where
  arXiv  Detail & Related papers  (2021-01-29T18:57:52Z)
• Faster Depth-Adaptive Transformers [71.20237659479703]
  Depth-adaptive neural networks can dynamically adjust depths according to the hardness of input words. Previous works generally build a halting unit to decide whether the computation should continue or stop at each layer. In this paper, we get rid of the halting unit and estimate the required depths in advance, which yields a faster depth-adaptive model.
  arXiv  Detail & Related papers  (2020-04-27T15:08:10Z)
• Naive Exploration is Optimal for Online LQR [49.681825576239355]
  We show that the optimal regret scales as $widetildeTheta(sqrtd_mathbfu2 d_mathbfx T)$, where $T$ is the number of time steps, $d_mathbfu$ is the dimension of the input space, and $d_mathbfx$ is the dimension of the system state. Our lower bounds rule out the possibility of a $mathrmpoly(logT)$-regret algorithm, which had been
  arXiv  Detail & Related papers  (2020-01-27T03:44:54Z)
• Simple and Effective Prevention of Mode Collapse in Deep One-Class Classification [93.2334223970488]
  We propose two regularizers to prevent hypersphere collapse in deep SVDD. The first regularizer is based on injecting random noise via the standard cross-entropy loss. The second regularizer penalizes the minibatch variance when it becomes too small.
  arXiv  Detail & Related papers  (2020-01-24T03:44:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.