Model Compression with Exact Budget Constraints via Riemannian Manifolds

• URL: http://arxiv.org/abs/2605.00649v2
• Date: Thu, 07 May 2026 11:36:50 GMT
• Title: Model Compression with Exact Budget Constraints via Riemannian Manifolds
• Abstract summary: Assigning one of K options to each of N groups under a total cost budget is a recurring problem in efficient AI.<n>We present a new approach by showing that, under softmax relaxation, the budget constraint defines a smooth Riemannian manifold in logit space with unusually simple geometry.<n>Building on these properties, we propose Riemannian Constrained Optimization (RCO), which augments a standard Adam step with tangent projection, binary-search retraction, and momentum transport.
• Score: 39.54576236079211
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Assigning one of K options to each of N groups under a total cost budget is a recurring problem in efficient AI, including mixed-precision quantization, non-uniform pruning, and expert selection. The objective, typically model loss, depends jointly on all assignments and does not decompose across groups, preventing combinatorial solvers from directly optimizing the true objective and forcing reliance on proxy formulations. Methods such as evolutionary search evaluate the actual loss but lack gradient information, while penalty-based approaches enforce the budget only approximately and often require extensive hyperparameter tuning. We present a new approach by showing that, under softmax relaxation, the budget constraint defines a smooth Riemannian manifold in logit space with unusually simple geometry. The normal vector admits a closed-form expression, shifting logits along the cost vector changes expected cost monotonically, and vector transport reduces to a single inner product. Building on these properties, we propose Riemannian Constrained Optimization (RCO), which augments a standard Adam step with tangent projection, binary-search retraction, and momentum transport. Combined with Gumbel straight-through estimation and budget-constrained dynamic programming for discrete feasibility, RCO enables first-order optimization of the actual loss under exact budget enforcement without introducing constraint-specific hyperparameters. Across both synthetic benchmarks and realistic LLM compression settings, RCO matches or exceeds state-of-the-art methods while often requiring substantially less wall-clock time. Source code is available at https://github.com/IST-DASLab/RCO.

Related papers

• Eigenvalue-Decomposition Cost Denoising as an Alternative to Predict-then-Optimize for Shortest-Path Problems [0.0]
  We propose and evaluate a structurally different remedy for a specific but common setting.<n>When the decision-maker observes many noisy realizations of the same underlying cost process, the realized cost themselves can be treated as a noisy signal.<n>We find that the choice of $k$ is decisive: keeping only $k=2$ eigenvectors discards real signal and underperforms even the noisy-cost baseline.
  arXiv  Detail & Related papers  (2026-09-14T06:41:25Z)
• Wall-Clock Complexity for Zeroth-Order Optimization with Tunable Oracle Fidelity [68.54852541944625]
  Black-box optimization is applied when gradients are unavailable and objective evaluations rely on expensive simulations.<n>We show how the choice of oracle type, noise model, and optimization scheme induces explicit wall-clock-optimal choices for the algorithmic parameters.
  arXiv  Detail & Related papers  (2026-05-29T14:24:54Z)
• Dimensionality Reduction for Robust Federated Learning: A Theoretical Analysis and Convergence Guarantee [14.954293251332894]
  Federated Learning (FL) enables clients to collaboratively train models without sharing raw data, but it is highly vulnerable to Byzantine attacks.<n>Existing robust approaches can neutralize these threats but incur substantial computational overhead.<n>We propose a universal acceleration framework for vector-level distance-based robust aggregators.
  arXiv  Detail & Related papers  (2026-05-27T11:39:47Z)
• Cost-Aware Learning [72.31444819326795]
  We consider the problem of Cost-Aware Learning, where sampling different component functions incur different costs.<n>We propose the Cost-Aware Descent algorithm for convex functions, and derive its cost complexity to attain an error of $$.<n>We introduce Cost-Aware GRPO, an algorithm designed to reduce the cost of policy optimization while preserving performance.
  arXiv  Detail & Related papers  (2026-04-30T15:39:09Z)
• DRAFTO: Decoupled Reduced-space and Adaptive Feasibility-repair Trajectory Optimization for Robotic Manipulators [4.0407133618465005]
  This paper introduces a new algorithm for trajectory optimization, Decoupled Reduced-space and Adaptive Feasibility-repair Trajectory Optimization (DRAFTO)<n>To reduce the number of repeated constrained optimizations while handling joint-limit feasibility, the optimization is decoupled into a reduced-space Gauss-Newton (GN) descent.<n>The results of our benchmark tests against optimization-based planners, such as CHOMP, TrajOpt, GPMP2, and FACTO, validate the high efficiency and reliability across diverse scenarios and tasks.
  arXiv  Detail & Related papers  (2026-03-10T20:24:42Z)
• First-Order Softmax Weighted Switching Gradient Method for Distributed Stochastic Minimax Optimization with Stochastic Constraints [9.141425189503794]
  We propose a novel first-order Softmax-Weighted Switching Gradient method tailored for federated learning.<n>Under full client participation, our algorithm achieves the standard $mathcalO(-4)$ oracle complexity.<n>We provide a unified error decomposition and establish a sharp $mathcalO(logfrac1)$ high-probability convergence guarantee.
  arXiv  Detail & Related papers  (2026-03-06T00:14:46Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentum [62.691095807959215]
  We establish an optimal sample complexity of $O(-2)$ for obtaining an $$-optimal global policy using a single-timescale actor-critic (AC) algorithm.<n>These mechanisms are compatible with existing deep learning architectures and require only minor modifications, without compromising practical applicability.
  arXiv  Detail & Related papers  (2026-02-02T00:35:42Z)
• Residual subspace evolution strategies for nonlinear inverse problems [1.14219428942199]
  inverse problems pervade engineering and science, yet noisy, non-differentiable, or expensive residual evaluations routinely defeat Jacobian-based solvers.<n>This paper introduces residual subspace evolution strategies (RSES), a derivative-free solver that draws Gaussian probes around the current gradient, records how residuals change along those directions, and recombines the probes through a least-squares solve to produce an optimal update.
  arXiv  Detail & Related papers  (2025-12-11T06:20:13Z)
• Differentiable Sparsity via $D$-Gating: Simple and Versatile Structured Penalization [22.883367233817836]
  We show that $D$-Gating is theoretically equivalent to solving the original group sparsity problem.<n>We validate our theory across vision, language, and tasks, where $D$-Gating consistently delivers strong performance-sparsity tradeoffs.
  arXiv  Detail & Related papers  (2025-09-28T14:08:29Z)
• Two-Timescale Optimization Framework for Sparse-Feedback Linear-Quadratic Optimal Control [3.746304628644379]
  A $mathcalHfeedback$-guaranteed sparse-feedback linear-quadratic (LQ) optimal control with convex parameterization and convex-bounded uncertainty is studied.
  arXiv  Detail & Related papers  (2024-06-17T03:17:33Z)
• Sample-Efficient Constrained Reinforcement Learning with General Parameterization [35.22742439337603]
  We consider a constrained Markov Decision Problem (CMDP) where the goal of an agent is to maximize the expected discounted sum of rewards over an infinite horizon. We develop the Primal-Dual Accelerated Natural Policy Gradient (PD-ANPG) algorithm that ensures an $epsilon$ global optimality gap and $epsilon$ constraint violation.
  arXiv  Detail & Related papers  (2024-05-17T08:39:05Z)
• Autonomous Sparse Mean-CVaR Portfolio Optimization [6.358973724565783]
  We propose an innovative autonomous sparse mean-CVaR portfolio model, capable of approximating the original model with arbitrary accuracy. We then propose a proximal alternating linearized minimization algorithm, coupled with a nested fixed-point proximity algorithm (both convergent) to iteratively solve the model.
  arXiv  Detail & Related papers  (2024-05-13T15:16:22Z)
• Quantization for decentralized learning under subspace constraints [61.59416703323886]
  We consider decentralized optimization problems where agents have individual cost functions to minimize subject to subspace constraints. We propose and study an adaptive decentralized strategy where the agents employ differential randomized quantizers to compress their estimates. The analysis shows that, under some general conditions on the quantization noise, the strategy is stable both in terms of mean-square error and average bit rate.
  arXiv  Detail & Related papers  (2022-09-16T09:38:38Z)
• Faster Algorithm and Sharper Analysis for Constrained Markov Decision Process [56.55075925645864]
  The problem of constrained decision process (CMDP) is investigated, where an agent aims to maximize the expected accumulated discounted reward subject to multiple constraints. A new utilities-dual convex approach is proposed with novel integration of three ingredients: regularized policy, dual regularizer, and Nesterov's gradient descent dual. This is the first demonstration that nonconcave CMDP problems can attain the lower bound of $mathcal O (1/epsilon)$ for all complexity optimization subject to convex constraints.
  arXiv  Detail & Related papers  (2021-10-20T02:57:21Z)
• Momentum Accelerates the Convergence of Stochastic AUPRC Maximization [80.8226518642952]
  We study optimization of areas under precision-recall curves (AUPRC), which is widely used for imbalanced tasks. We develop novel momentum methods with a better iteration of $O (1/epsilon4)$ for finding an $epsilon$stationary solution. We also design a novel family of adaptive methods with the same complexity of $O (1/epsilon4)$, which enjoy faster convergence in practice.
  arXiv  Detail & Related papers  (2021-07-02T16:21:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.