Explicitly Encouraging Low Fractional Dimensional Trajectories Via Reinforcement Learning

• URL: http://arxiv.org/abs/2012.11662v1
• Date: Mon, 21 Dec 2020 20:09:17 GMT
• Title: Explicitly Encouraging Low Fractional Dimensional Trajectories Via Reinforcement Learning
• Authors: Sean Gillen and Katie Byl
• Abstract summary: We show that the dimensionality of trajectories induced by model free reinforcement learning agents can be influenced adding a post processing function to the agents reward signal. We verify that the dimensionality reduction is robust to noise being added to the system and show that that the modified agents are more actually more robust to noise and push disturbances in general for the systems we examined.
• Score: 6.548580592686076
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A key limitation in using various modern methods of machine learning in developing feedback control policies is the lack of appropriate methodologies to analyze their long-term dynamics, in terms of making any sort of guarantees (even statistically) about robustness. The central reasons for this are largely due to the so-called curse of dimensionality, combined with the black-box nature of the resulting control policies themselves. This paper aims at the first of these issues. Although the full state space of a system may be quite large in dimensionality, it is a common feature of most model-based control methods that the resulting closed-loop systems demonstrate dominant dynamics that are rapidly driven to some lower-dimensional sub-space within. In this work we argue that the dimensionality of this subspace is captured by tools from fractal geometry, namely various notions of a fractional dimension. We then show that the dimensionality of trajectories induced by model free reinforcement learning agents can be influenced adding a post processing function to the agents reward signal. We verify that the dimensionality reduction is robust to noise being added to the system and show that that the modified agents are more actually more robust to noise and push disturbances in general for the systems we examined.

Related papers

• Quantum Engineering of Qudits with Interpretable Machine Learning [0.0]
  We present a machine-learning-based graybox framework for the control and noise characterization of qudits with arbitrary dimension.<n>We also introduce a local analytic expansion that enables interpretable modelling of the noise dynamics.
  arXiv  Detail & Related papers  (2025-06-16T03:46:23Z)
• Weight Spectra Induced Efficient Model Adaptation [54.8615621415845]
  Fine-tuning large-scale foundation models incurs prohibitive computational costs.<n>We show that fine-tuning predominantly amplifies the top singular values while leaving the remainder largely intact.<n>We propose a novel method that leverages learnable rescaling of top singular directions.
  arXiv  Detail & Related papers  (2025-05-29T05:03:29Z)
• Mitigating Overthinking in Large Reasoning Models via Manifold Steering [32.666911833023526]
  Large Reasoning Models (LRMs) exhibit a phenomenon known as overthinking during inference.<n>We propose Manifold Steering, a novel approach that elegantly projects the steering direction onto the low-dimensional activation manifold.<n>Our method reduces output tokens by up to 71% while maintaining and even improving the accuracy on several mathematical benchmarks.
  arXiv  Detail & Related papers  (2025-05-28T14:39:26Z)
• FreSca: Scaling in Frequency Space Enhances Diffusion Models [55.75504192166779]
  This paper explores frequency-based control within latent diffusion models.<n>We introduce FreSca, a novel framework that decomposes noise difference into low- and high-frequency components.<n>FreSca operates without any model retraining or architectural change, offering model- and task-agnostic control.
  arXiv  Detail & Related papers  (2025-04-02T22:03:11Z)
• Latent feedback control of distributed systems in multiple scenarios through deep learning-based reduced order models [3.5161229331588095]
  Continuous monitoring and real-time control of high-dimensional distributed systems are crucial in applications to ensure a desired physical behavior. Traditional feedback control design that relies on full-order models fails to meet these requirements due to the delay in the control computation. We propose a real-time closed-loop control strategy enhanced by nonlinear non-intrusive Deep Learning-based Reduced Order Models (DL-ROMs)
  arXiv  Detail & Related papers  (2024-12-13T08:04:21Z)
• Interpretable and Efficient Data-driven Discovery and Control of Distributed Systems [1.5195865840919498]
  Reinforcement Learning (RL) has emerged as a promising control paradigm for systems with high-dimensional, nonlinear dynamics. We propose a data-efficient, interpretable, and scalable framework for PDE control.
  arXiv  Detail & Related papers  (2024-11-06T18:26:19Z)
• Deep Autoencoder with SVD-Like Convergence and Flat Minima [1.0742675209112622]
  We propose a learnable weighted hybrid autoencoder to overcome the Kolmogorov barrier. We empirically find that our trained model has a sharpness thousands of times smaller compared to other models.
  arXiv  Detail & Related papers  (2024-10-23T00:04:26Z)
• Input-to-State Stable Coupled Oscillator Networks for Closed-form Model-based Control in Latent Space [2.527926867319859]
  We argue that a promising avenue is to leverage powerful and well-understood closed-form strategies from control theory literature. We identify three fundamental shortcomings in existing latent-space models that have so far prevented this powerful combination. We propose a novel Coupled Network (CON) model that simultaneously tackles all these issues.
  arXiv  Detail & Related papers  (2024-09-13T00:11:09Z)
• Dimensionless Policies based on the Buckingham $\pi$ Theorem: Is This a Good Way to Generalize Numerical Results? [66.52698983694613]
  This article explores the use of the Buckingham $pi$ theorem as a tool to encode the control policies of physical systems into a generic form of knowledge. We show, by restating the solution to a motion control problem using dimensionless variables, that (1) the policy mapping involves a reduced number of parameters and (2) control policies generated numerically for a specific system can be transferred exactly to a subset of dimensionally similar systems by scaling the input and output variables appropriately. It remains to be seen how practical this approach can be to generalize policies for more complex high-dimensional problems, but the early results show that it is a
  arXiv  Detail & Related papers  (2023-07-29T00:51:26Z)
• In-Distribution Barrier Functions: Self-Supervised Policy Filters that Avoid Out-of-Distribution States [84.24300005271185]
  We propose a control filter that wraps any reference policy and effectively encourages the system to stay in-distribution with respect to offline-collected safe demonstrations. Our method is effective for two different visuomotor control tasks in simulation environments, including both top-down and egocentric view settings.
  arXiv  Detail & Related papers  (2023-01-27T22:28:19Z)
• An Intrusion Response System utilizing Deep Q-Networks and System Partitions [0.415623340386296]
  We introduce and develop an IRS software prototype, named irs-partition. It exploits transfer learning to follow the evolution of non-stationary systems.
  arXiv  Detail & Related papers  (2022-02-16T16:38:20Z)
• Deep Learning Approximation of Diffeomorphisms via Linear-Control Systems [91.3755431537592]
  We consider a control system of the form $dot x = sum_i=1lF_i(x)u_i$, with linear dependence in the controls. We use the corresponding flow to approximate the action of a diffeomorphism on a compact ensemble of points.
  arXiv  Detail & Related papers  (2021-10-24T08:57:46Z)
• Sparsity in Partially Controllable Linear Systems [56.142264865866636]
  We study partially controllable linear dynamical systems specified by an underlying sparsity pattern. Our results characterize those state variables which are irrelevant for optimal control.
  arXiv  Detail & Related papers  (2021-10-12T16:41:47Z)
• Pure Exploration in Kernel and Neural Bandits [90.23165420559664]
  We study pure exploration in bandits, where the dimension of the feature representation can be much larger than the number of arms. To overcome the curse of dimensionality, we propose to adaptively embed the feature representation of each arm into a lower-dimensional space.
  arXiv  Detail & Related papers  (2021-06-22T19:51:59Z)
• Limited-angle tomographic reconstruction of dense layered objects by dynamical machine learning [68.9515120904028]
  Limited-angle tomography of strongly scattering quasi-transparent objects is a challenging, highly ill-posed problem. Regularizing priors are necessary to reduce artifacts by improving the condition of such problems. We devised a recurrent neural network (RNN) architecture with a novel split-convolutional gated recurrent unit (SC-GRU) as the building block.
  arXiv  Detail & Related papers  (2020-07-21T11:48:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.