Test-Time Canonicalization by Foundation Models for Robust Perception

• URL: http://arxiv.org/abs/2507.10375v1
• Date: Mon, 14 Jul 2025 15:14:38 GMT
• Title: Test-Time Canonicalization by Foundation Models for Robust Perception
• Authors: Utkarsh Singhal, Ryan Feng, Stella X. Yu, Atul Prakash,
• Abstract summary: FOCAL is a test-time, data-driven framework for robust perception.<n>It enhances robustness without re-training or architectural changes.<n>Our experiments demonstrate improved robustness of CLIP and SAM across challenging transformations.
• Score: 33.00574202314593
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-world visual perception requires invariance to diverse transformations, yet current methods rely heavily on specialized architectures or training on predefined augmentations, limiting generalization. We propose FOCAL, a test-time, data-driven framework that achieves robust perception by leveraging internet-scale visual priors from foundation models. By generating and optimizing candidate transformations toward visually typical, "canonical" views, FOCAL enhances robustness without re-training or architectural changes. Our experiments demonstrate improved robustness of CLIP and SAM across challenging transformations, including 2D/3D rotations, illumination shifts (contrast and color), and day-night variations. We also highlight potential applications in active vision. Our approach challenges the assumption that transform-specific training is necessary, instead offering a scalable path to invariance. Our code is available at: https://github.com/sutkarsh/focal.

Related papers

• Zero-Shot Visual Generalization in Robot Manipulation [0.13280779791485384]
  Current approaches often sidestep the problem by relying on invariant representations such as point clouds and depth.<n>Disentangled representation learning has recently shown promise in enabling vision-based reinforcement learning policies to be robust to visual distribution shifts.<n>We demonstrate zero-shot adaptability to visual perturbations in both simulation and on real hardware.
  arXiv  Detail & Related papers  (2025-05-16T22:01:46Z)
• ViT-VS: On the Applicability of Pretrained Vision Transformer Features for Generalizable Visual Servoing [40.67171259494469]
  We present a visual servoing approach that leverages pretrained vision transformers for semantic feature extraction.<n>Our approach surpasses classical image-based visual servoing by up to 31.2% relative improvement in perturbed scenarios.<n>Real-world evaluations confirm robust performance in end-effector positioning, industrial box manipulation, and grasping of unseen objects.
  arXiv  Detail & Related papers  (2025-03-06T15:33:19Z)
• Robust Scene Change Detection Using Visual Foundation Models and Cross-Attention Mechanisms [27.882122236282054]
  We present a novel method for scene change detection that leverages the robust feature extraction capabilities of a visual foundational model, DINOv2.<n>We evaluate our approach on two benchmark datasets, VL-CMU-CD and PSCD, along with their viewpoint-varied versions.<n>Our experiments demonstrate significant improvements in F1-score, particularly in scenarios involving geometric changes between image pairs.
  arXiv  Detail & Related papers  (2024-09-25T11:55:27Z)
• RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictability [25.943330238941602]
  We propose a visual model-based RL method that learns a latent representation resilient to spurious variations. Our training objective encourages the representation to be maximally predictive of dynamics and reward. Our effort is a step towards making model-based RL a practical and useful tool for dynamic, diverse domains.
  arXiv  Detail & Related papers  (2023-08-31T18:43:04Z)
• Bilevel Fast Scene Adaptation for Low-Light Image Enhancement [50.639332885989255]
  Enhancing images in low-light scenes is a challenging but widely concerned task in the computer vision. Main obstacle lies in the modeling conundrum from distribution discrepancy across different scenes. We introduce the bilevel paradigm to model the above latent correspondence. A bilevel learning framework is constructed to endow the scene-irrelevant generality of the encoder towards diverse scenes.
  arXiv  Detail & Related papers  (2023-06-02T08:16:21Z)
• Geometric-aware Pretraining for Vision-centric 3D Object Detection [77.7979088689944]
  We propose a novel geometric-aware pretraining framework called GAPretrain. GAPretrain serves as a plug-and-play solution that can be flexibly applied to multiple state-of-the-art detectors. We achieve 46.2 mAP and 55.5 NDS on the nuScenes val set using the BEVFormer method, with a gain of 2.7 and 2.1 points, respectively.
  arXiv  Detail & Related papers  (2023-04-06T14:33:05Z)
• Vision Transformer with Quadrangle Attention [76.35955924137986]
  We propose a novel quadrangle attention (QA) method that extends the window-based attention to a general quadrangle formulation. Our method employs an end-to-end learnable quadrangle regression module that predicts a transformation matrix to transform default windows into target quadrangles. We integrate QA into plain and hierarchical vision transformers to create a new architecture named QFormer, which offers minor code modifications and negligible extra computational cost.
  arXiv  Detail & Related papers  (2023-03-27T11:13:50Z)
• A Simple Strategy to Provable Invariance via Orbit Mapping [14.127786615513978]
  We propose a method to make network architectures provably invariant with respect to group actions. In a nutshell, we intend to 'undo' any possible transformation before feeding the data into the actual network.
  arXiv  Detail & Related papers  (2022-09-24T03:40:42Z)
• Vision Transformers: From Semantic Segmentation to Dense Prediction [139.15562023284187]
  We explore the global context learning potentials of vision transformers (ViTs) for dense visual prediction. Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information. We formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture.
  arXiv  Detail & Related papers  (2022-07-19T15:49:35Z)
• B-cos Networks: Alignment is All We Need for Interpretability [136.27303006772294]
  We present a new direction for increasing the interpretability of deep neural networks (DNNs) by promoting weight-input alignment during training. A B-cos transform induces a single linear transform that faithfully summarises the full model computations. We show that it can easily be integrated into common models such as VGGs, ResNets, InceptionNets, and DenseNets.
  arXiv  Detail & Related papers  (2022-05-20T16:03:29Z)
• Visformer: The Vision-friendly Transformer [105.52122194322592]
  We propose a new architecture named Visformer, which is abbreviated from the Vision-friendly Transformer' With the same computational complexity, Visformer outperforms both the Transformer-based and convolution-based models in terms of ImageNet classification accuracy.
  arXiv  Detail & Related papers  (2021-04-26T13:13:03Z)
• A Flexible Framework for Designing Trainable Priors with Adaptive Smoothing and Game Encoding [57.1077544780653]
  We introduce a general framework for designing and training neural network layers whose forward passes can be interpreted as solving non-smooth convex optimization problems. We focus on convex games, solved by local agents represented by the nodes of a graph and interacting through regularization functions. This approach is appealing for solving imaging problems, as it allows the use of classical image priors within deep models that are trainable end to end.
  arXiv  Detail & Related papers  (2020-06-26T08:34:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.