DynamicGate MLP Conditional Computation via Learned Structural Dropout and Input Dependent Gating for Functional Plasticity

• URL: http://arxiv.org/abs/2603.16367v1
• Date: Tue, 17 Mar 2026 10:54:16 GMT
• Title: DynamicGate MLP Conditional Computation via Learned Structural Dropout and Input Dependent Gating for Functional Plasticity
• Abstract summary: Dropout is a representative regularization technique that deactivates hidden units during training to mitigate overfitting.<n>Standard inference executes the full network with dense computation, so its goal and mechanism differ from conditional computation.<n>This paper organizes DynamicGate-MLP into a single framework that simultaneously satisfies both the regularization view and the conditional-computation view.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dropout is a representative regularization technique that stochastically deactivates hidden units during training to mitigate overfitting. In contrast, standard inference executes the full network with dense computation, so its goal and mechanism differ from conditional computation, where the executed operations depend on the input. This paper organizes DynamicGate-MLP into a single framework that simultaneously satisfies both the regularization view and the conditional-computation view. Instead of a random mask, the proposed model learns gates that decide whether to use each unit (or block), suppressing unnecessary computation while implementing sample-dependent execution that concentrates computation on the parts needed for each input. To this end, we define continuous gate probabilities and, at inference time, generate a discrete execution mask from them to select an execution path. Training controls the compute budget via a penalty on expected gate usage and uses a Straight-Through Estimator (STE) to optimize the discrete mask. We evaluate DynamicGate-MLP on MNIST, CIFAR-10, Tiny-ImageNet, Speech Commands, and PBMC3k, and compare it with various MLP baselines and MoE-style variants. Compute efficiency is compared under a consistent criterion using gate activation ratios and a layerweighted relative MAC metric, rather than wall-clock latency that depends on hardware and backend kernels.

Related papers

• ELASTIC: Efficiently Learning to Adaptively Scale Test-Time Compute for Generative Control Policies [36.28264388380867]
  Generative control policies (GCPs) enable test-time scaling in robot control.<n>Test-time compute can be allocated along two axes: sequential scaling and parallel scaling.<n>We present ELASTIC, an algorithm that learns state-dependent test-time compute schedules for GCPs.
  arXiv  Detail & Related papers  (2026-06-30T05:00:32Z)
• Manifold-Constrained MPPI: Real-Time Sampling-Based Control Under Hard Constraints [8.747309323431802]
  Manifold-Constrained MPPI (MC-MPPI) is a real-time sampling-based control framework.<n>It enforces manifold-based equality constraints while preserving the computational advantages of MPPI.<n>Experiments on a 14-DoF closed-chain dual-arm system in both simulation and real-world settings demonstrate that MC-MPPI operates stably at 100 Hz.
  arXiv  Detail & Related papers  (2026-05-24T01:57:18Z)
• DiscreteRTC: Discrete Diffusion Policies are Natural Asynchronous Executors [57.944744187489185]
  We propose DiscreteRTC, which replaces external corrections with native unmasking.<n>DiscreteRTC is simpler to implement with 0 lines of code for async inpainting, faster at inference with only 0.7x computation compared with generating actions from scratch, and better at execution with 50% higher success rate in real-world dynamic pick task compared with flow-based RTC.
  arXiv  Detail & Related papers  (2026-04-27T23:04:03Z)
• Deterministic Differentiable Structured Pruning for Large Language Models [37.33389749907146]
  Structured pruning reduces LLM inference cost by removing low-importance architectural components.<n>We propose Deterministic Differentiable Pruning (DDP), a mask-only optimization method.<n>Compared with prior approaches, DDP offers greater expressiveness, reduced train--test mismatch, and faster convergence.
  arXiv  Detail & Related papers  (2026-03-09T07:59:17Z)
• AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth [23.442686851761298]
  AdaPonderLM is a self-supervised recurrent language model that learns token-wise early exiting during pretraining.<n>AdaPonderLM reduces inference compute at about 10% while maintaining comparable language modeling perplexity and competitive downstream accuracy.
  arXiv  Detail & Related papers  (2026-03-02T14:28:16Z)
• Para-B&B: Load-Balanced Deterministic Parallelization of Solving MIP [50.917107318582715]
  Mixed-integer programming (MIP) extends linear programming by incorporating both continuous and integer decision variables.<n>This paper presents the first fully open-source implementation of deterministic parallel branch-and-bound for HiGHS, a high-performance MIP solver.<n>Our approach introduces a novel data-parallel architecture ensuring strict determinism by replicating complete solver state across worker threads.
  arXiv  Detail & Related papers  (2026-02-10T14:17:53Z)
• What If We Allocate Test-Time Compute Adaptively? [2.1713977971908944]
  Test-time scaling allocates inference computation uniformly, uses fixed sampling strategies, and applies verification only for reranking.<n>We propose a verifier-guided adaptive framework treating reasoning as iterative trajectory generation and selection.<n>Across datasets, our dynamic, PRM-guided approach consistently outperforms direct test-time scaling.
  arXiv  Detail & Related papers  (2026-02-01T07:30:22Z)
• Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling [90.86991492288487]
  evaluating constraint on every token can be prohibitively expensive.<n> LCD can distort the global distribution over strings, sampling tokens based only on local information.<n>We show that our approach is superior to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-04-07T18:30:18Z)
• Progressive Mixed-Precision Decoding for Efficient LLM Inference [49.05448842542558]
  We introduce Progressive Mixed-Precision Decoding (PMPD) to address the memory-boundedness of decoding.<n>PMPD achieves 1.4$-$12.2$times$ speedup in matrix-vector multiplications over fp16 models.<n>Our approach delivers a throughput gain of 3.8$-$8.0$times$ over fp16 models and up to 1.54$times$ over uniform quantization approaches.
  arXiv  Detail & Related papers  (2024-10-17T11:46:33Z)
• Mixture-of-Depths: Dynamically allocating compute in transformer-based language models [8.774705201394916]
  Transformer-based language models spread FLOPs uniformly across input sequences. We show that transformers can learn to dynamically allocate FLOPs to specific positions in a sequence.
  arXiv  Detail & Related papers  (2024-04-02T19:28:11Z)
• Stochastic Optimal Control Matching [53.156277491861985]
  Our work introduces Optimal Control Matching (SOCM), a novel Iterative Diffusion Optimization (IDO) technique for optimal control. The control is learned via a least squares problem by trying to fit a matching vector field. Experimentally, our algorithm achieves lower error than all the existing IDO techniques for optimal control.
  arXiv  Detail & Related papers  (2023-12-04T16:49:43Z)
• AQD: Towards Accurate Fully-Quantized Object Detection [94.06347866374927]
  We propose an Accurate Quantized object Detection solution, termed AQD, to get rid of floating-point computation. Our AQD achieves comparable or even better performance compared with the full-precision counterpart under extremely low-bit schemes.
  arXiv  Detail & Related papers  (2020-07-14T09:07:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.