Generalizable Image Repair for Robust Visual Control
- URL: http://arxiv.org/abs/2503.05911v2
- Date: Thu, 31 Jul 2025 16:39:27 GMT
- Title: Generalizable Image Repair for Robust Visual Control
- Authors: Carson Sobolewski, Zhenjiang Mao, Kshitij Maruti Vejre, Ivan Ruchkin,
- Abstract summary: Vision-based control relies on accurate perception to achieve robustness.<n>Image distribution changes caused by sensor noise, adverse weather, and dynamic lighting can degrade perception, leading to suboptimal control decisions.<n>We propose a real-time image repair module that restores corrupted images before they are used by the controller.
- Score: 0.12499537119440243
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Vision-based control relies on accurate perception to achieve robustness. However, image distribution changes caused by sensor noise, adverse weather, and dynamic lighting can degrade perception, leading to suboptimal control decisions. Existing approaches, including domain adaptation and adversarial training, improve robustness but struggle to generalize to unseen corruptions while introducing computational overhead. To address this challenge, we propose a real-time image repair module that restores corrupted images before they are used by the controller. Our method leverages generative adversarial models, specifically CycleGAN and pix2pix, for image repair. CycleGAN enables unpaired image-to-image translation to adapt to novel corruptions, while pix2pix exploits paired image data when available to improve the quality. To ensure alignment with control performance, we introduce a control-focused loss function that prioritizes perceptual consistency in repaired images. We evaluated our method in a simulated autonomous racing environment with various visual corruptions. The results show that our approach significantly improves performance compared to baselines, mitigating distribution shift and enhancing controller reliability.
Related papers
- Robust and Efficient 3D Gaussian Splatting for Urban Scene Reconstruction [10.948843569839305]
We present a framework that enables fast reconstruction and real-time rendering of urban-scale scenes.<n>Our method effectively reconstructs urban-scale scenes and outperforms previous approaches in both efficiency and quality.
arXiv Detail & Related papers (2025-07-30T18:11:55Z) - ControlMambaIR: Conditional Controls with State-Space Model for Image Restoration [13.577709018178364]
This paper proposes ControlMambaIR, a novel image restoration method designed to address perceptual challenges in image deraining, deblurring, and denoising tasks.<n>By integrating the Mamba network architecture with the diffusion model, the condition network achieves refined conditional control, thereby enhancing the control and optimization of the image generation process.
arXiv Detail & Related papers (2025-06-03T08:50:00Z) - Fine-Tuning Adversarially-Robust Transformers for Single-Image Dehazing [2.0209172586699173]
We show that state-of-the-art image-to-image dehazing transformers are susceptible to adversarial noise.
We propose two lightweight fine-tuning strategies aimed at increasing the robustness of pre-trained transformers.
arXiv Detail & Related papers (2025-04-24T08:52:14Z) - Efficient Camera Exposure Control for Visual Odometry via Deep Reinforcement Learning [10.886819238167286]
This study employs a deep reinforcement learning framework to train agents for exposure control.<n>A lightweight image simulator is developed to facilitate the training process.<n>Different levels of reward functions are crafted to enhance the VO systems.
arXiv Detail & Related papers (2024-08-30T04:37:52Z) - CODE: Confident Ordinary Differential Editing [62.83365660727034]
Confident Ordinary Differential Editing (CODE) is a novel approach for image synthesis that effectively handles Out-of-Distribution (OoD) guidance images.
CODE enhances images through score-based updates along the probability-flow Ordinary Differential Equation (ODE) trajectory.
Our method operates in a fully blind manner, relying solely on a pre-trained generative model.
arXiv Detail & Related papers (2024-08-22T14:12:20Z) - Diffusion Model Driven Test-Time Image Adaptation for Robust Skin Lesion Classification [24.08402880603475]
We propose a test-time image adaptation method to enhance the accuracy of the model on test data.
We modify the target test images by projecting them back to the source domain using a diffusion model.
Our method makes the robustness more robust across various corruptions, architectures, and data regimes.
arXiv Detail & Related papers (2024-05-18T13:28:51Z) - Unexplored Faces of Robustness and Out-of-Distribution: Covariate Shifts in Environment and Sensor Domains [2.4572304328659595]
We introduce a new distribution shift dataset, ImageNet-ES.
We evaluate out-of-distribution (OOD) detection and model robustness.
Our results suggest that effective shift mitigation via camera sensor control can significantly improve performance without increasing model size.
arXiv Detail & Related papers (2024-04-24T13:59:19Z) - Towards Robust Image Stitching: An Adaptive Resistance Learning against
Compatible Attacks [66.98297584796391]
Image stitching seamlessly integrates images captured from varying perspectives into a single wide field-of-view image.
Given a pair of captured images, subtle perturbations and distortions which go unnoticed by the human visual system tend to attack the correspondence matching.
This paper presents the first attempt to improve the robustness of image stitching against adversarial attacks.
arXiv Detail & Related papers (2024-02-25T02:36:33Z) - DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior [70.46245698746874]
We present DiffBIR, a general restoration pipeline that could handle different blind image restoration tasks.
DiffBIR decouples blind image restoration problem into two stages: 1) degradation removal: removing image-independent content; 2) information regeneration: generating the lost image content.
In the first stage, we use restoration modules to remove degradations and obtain high-fidelity restored results.
For the second stage, we propose IRControlNet that leverages the generative ability of latent diffusion models to generate realistic details.
arXiv Detail & Related papers (2023-08-29T07:11:52Z) - DeltaNN: Assessing the Impact of Computational Environment Parameters on the Performance of Image Recognition Models [2.379078565066793]
Failure in real-time image recognition tasks can occur due to sub-optimal mapping on hardware accelerators.
We present a differential testing framework, DeltaNN, that allows us to assess the impact of different computational environment parameters on the performance of image recognition models.
arXiv Detail & Related papers (2023-06-05T23:07:01Z) - Robust Single Image Dehazing Based on Consistent and Contrast-Assisted
Reconstruction [95.5735805072852]
We propose a novel density-variational learning framework to improve the robustness of the image dehzing model.
Specifically, the dehazing network is optimized under the consistency-regularized framework.
Our method significantly surpasses the state-of-the-art approaches.
arXiv Detail & Related papers (2022-03-29T08:11:04Z) - AdaViT: Adaptive Vision Transformers for Efficient Image Recognition [78.07924262215181]
We introduce AdaViT, an adaptive framework that learns to derive usage policies on which patches, self-attention heads and transformer blocks to use.
Our method obtains more than 2x improvement on efficiency compared to state-of-the-art vision transformers with only 0.8% drop of accuracy.
arXiv Detail & Related papers (2021-11-30T18:57:02Z) - Improving robustness against common corruptions with frequency biased
models [112.65717928060195]
unseen image corruptions can cause a surprisingly large drop in performance.
Image corruption types have different characteristics in the frequency spectrum and would benefit from a targeted type of data augmentation.
We propose a new regularization scheme that minimizes the total variation (TV) of convolution feature-maps to increase high-frequency robustness.
arXiv Detail & Related papers (2021-03-30T10:44:50Z) - On Interaction Between Augmentations and Corruptions in Natural
Corruption Robustness [78.6626755563546]
Several new data augmentations have been proposed that significantly improve performance on ImageNet-C.
We develop a new measure in this space between augmentations and corruptions called the Minimal Sample Distance to demonstrate there is a strong correlation between similarity and performance.
We observe a significant degradation in corruption robustness when the test-time corruptions are sampled to be perceptually dissimilar from ImageNet-C.
Our results suggest that test error can be improved by training on perceptually similar augmentations, and data augmentations may not generalize well beyond the existing benchmark.
arXiv Detail & Related papers (2021-02-22T18:58:39Z) - Encoding Robustness to Image Style via Adversarial Feature Perturbations [72.81911076841408]
We adapt adversarial training by directly perturbing feature statistics, rather than image pixels, to produce robust models.
Our proposed method, Adversarial Batch Normalization (AdvBN), is a single network layer that generates worst-case feature perturbations during training.
arXiv Detail & Related papers (2020-09-18T17:52:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.