OmniGlue: Generalizable Feature Matching with Foundation Model Guidance

• URL: http://arxiv.org/abs/2405.12979v1
• Date: Tue, 21 May 2024 17:59:22 GMT
• Title: OmniGlue: Generalizable Feature Matching with Foundation Model Guidance
• Abstract summary: We introduce OmniGlue, the first learnable image matcher that is designed with generalization as a core principle. We propose a novel keypoint position-guided attention mechanism which disentangles spatial and appearance information, leading to enhanced matching descriptors.
• Score: 21.692165943308797
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The image matching field has been witnessing a continuous emergence of novel learnable feature matching techniques, with ever-improving performance on conventional benchmarks. However, our investigation shows that despite these gains, their potential for real-world applications is restricted by their limited generalization capabilities to novel image domains. In this paper, we introduce OmniGlue, the first learnable image matcher that is designed with generalization as a core principle. OmniGlue leverages broad knowledge from a vision foundation model to guide the feature matching process, boosting generalization to domains not seen at training time. Additionally, we propose a novel keypoint position-guided attention mechanism which disentangles spatial and appearance information, leading to enhanced matching descriptors. We perform comprehensive experiments on a suite of $7$ datasets with varied image domains, including scene-level, object-centric and aerial images. OmniGlue's novel components lead to relative gains on unseen domains of $20.9\%$ with respect to a directly comparable reference model, while also outperforming the recent LightGlue method by $9.5\%$ relatively.Code and model can be found at https://hwjiang1510.github.io/OmniGlue

Related papers

• OmniGen-AR: AutoRegressive Any-to-Image Generation [89.27614937810836]
  We present OmniGen-AR, a unified autoregressive framework for Any-to-Image generation.<n>By discretizing various visual conditions through a shared visual tokenizer and text prompts with a text tokenizer, OmniGen-AR supports a broad spectrum of conditional inputs within a single model.
  arXiv  Detail & Related papers  (2026-06-08T07:47:53Z)
• Redefining Generalization in Visual Domains: A Two-Axis Framework for Fake Image Detection with FusionDetect [19.97092710696699]
  We present the OmniGen Benchmark to evaluate detector performance under realistic conditions.<n>We also introduce a new method, FusionDetect, aimed at addressing both vectors of generalization.<n>Our experiments demonstrate that FusionDetect delivers not only a new state-of-the-art, but also a 4.48% increase in accuracy on OmniGen.
  arXiv  Detail & Related papers  (2025-10-07T10:01:32Z)
• OmniDFA: A Unified Framework for Open Set Synthesis Image Detection and Few-Shot Attribution [21.62979058692505]
  OmniDFA is a novel framework for AIGI that assesses the authenticity of images, and determines the origins in a few-shot manner.<n>We construct OmniFake, a large class-aware synthetic image dataset that curates $1.17$ M images from $45$ distinct generative models.<n>Experiments demonstrate that OmniDFA exhibits excellent capability in open-set attribution and achieves state-of-the-art generalization performance on AIGI detection.
  arXiv  Detail & Related papers  (2025-09-30T02:36:40Z)
• PractiLight: Practical Light Control Using Foundational Diffusion Models [78.75949075070595]
  PractiLight is a practical approach to light control in generated images.<n>Our key insight is that lighting relationships in an image are similar in nature to token interaction in self-attention layers.<n>We demonstrate state-of-the-art performance in terms of quality and control with proven parameter and data efficiency.
  arXiv  Detail & Related papers  (2025-09-01T23:38:40Z)
• OmniGenBench: A Benchmark for Omnipotent Multimodal Generation across 50+ Tasks [77.19223035769248]
  Recent breakthroughs in large multimodal models (LMMs) have demonstrated remarkable proficiency in following general-purpose instructions for image generation.<n>We introduce OmniGenBench, a novel benchmark meticulously designed to assess the instruction-following abilities of state-of-the-art LMMs.<n>Our OmniGenBench includes 57 diverse sub-tasks grounded in real-world scenarios, systematically categorized according to the specific model capabilities they demand.
  arXiv  Detail & Related papers  (2025-05-24T16:29:34Z)
• Zooming In on Fakes: A Novel Dataset for Localized AI-Generated Image Detection with Forgery Amplification Approach [69.01456182499486]
  textbfBR-Gen is a large-scale dataset of 150,000 locally forged images with diverse scene-aware annotations. textbfNFA-ViT is a Noise-guided Forgery Amplification Vision Transformer that enhances the detection of localized forgeries.
  arXiv  Detail & Related papers  (2025-04-16T09:57:23Z)
• Grounding Descriptions in Images informs Zero-Shot Visual Recognition [47.66166611138081]
  We propose GRAIN, a new pretraining strategy aimed at aligning representations at both fine and coarse levels simultaneously. We demonstrate the enhanced zero-shot performance of our model compared to current state-of-the art methods.
  arXiv  Detail & Related papers  (2024-12-05T18:52:00Z)
• Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detection [58.87142367781417]
  A naively trained detector tends to favor overfitting to the limited and monotonous fake patterns, causing the feature space to become highly constrained and low-ranked. One potential remedy is incorporating the pre-trained knowledge within the vision foundation models to expand the feature space. By freezing the principal components and adapting only the remained components, we preserve the pre-trained knowledge while learning forgery-related patterns.
  arXiv  Detail & Related papers  (2024-11-23T19:10:32Z)
• Novel View Synthesis with Pixel-Space Diffusion Models [4.844800099745365]
  generative models are being increasingly employed in novel view synthesis (NVS) We adapt a modern diffusion model architecture for end-to-end NVS in the pixel space. We introduce a novel NVS training scheme that utilizes single-view datasets, capitalizing on their relative abundance.
  arXiv  Detail & Related papers  (2024-11-12T12:58:33Z)
• Dual-Image Enhanced CLIP for Zero-Shot Anomaly Detection [58.228940066769596]
  We introduce a Dual-Image Enhanced CLIP approach, leveraging a joint vision-language scoring system. Our methods process pairs of images, utilizing each as a visual reference for the other, thereby enriching the inference process with visual context. Our approach significantly exploits the potential of vision-language joint anomaly detection and demonstrates comparable performance with current SOTA methods across various datasets.
  arXiv  Detail & Related papers  (2024-05-08T03:13:20Z)
• Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Experts [88.23732496104667]
  Cross-scene generalizable NeRF models have become a new spotlight of the NeRF field. We bridge "neuralized" architectures with the powerful Mixture-of-Experts (MoE) idea from large language models. Our proposed model, dubbed GNT with Mixture-of-View-Experts (GNT-MOVE), has experimentally shown state-of-the-art results when transferring to unseen scenes.
  arXiv  Detail & Related papers  (2023-08-22T21:18:54Z)
• Similarity Min-Max: Zero-Shot Day-Night Domain Adaptation [52.923298434948606]
  Low-light conditions not only hamper human visual experience but also degrade the model's performance on downstream vision tasks. This paper challenges a more complicated scenario with border applicability, i.e., zero-shot day-night domain adaptation. We propose a similarity min-max paradigm that considers them under a unified framework.
  arXiv  Detail & Related papers  (2023-07-17T18:50:15Z)
• Intra- & Extra-Source Exemplar-Based Style Synthesis for Improved Domain Generalization [21.591831983223997]
  We propose an exemplar-based style synthesis pipeline to improve domain generalization in semantic segmentation. Our method is based on a novel masked noise encoder for StyleGAN2 inversion. We achieve up to $12.4%$ mIoU improvements on driving-scene semantic segmentation under different types of data shifts.
  arXiv  Detail & Related papers  (2023-07-02T19:56:43Z)
• GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images [79.39247661907397]
  We introduce an effective framework Generalizable Model-based Neural Radiance Fields to synthesize free-viewpoint images. Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy.
  arXiv  Detail & Related papers  (2023-03-24T03:32:02Z)
• When Neural Networks Fail to Generalize? A Model Sensitivity Perspective [82.36758565781153]
  Domain generalization (DG) aims to train a model to perform well in unseen domains under different distributions. This paper considers a more realistic yet more challenging scenario, namely Single Domain Generalization (Single-DG) We empirically ascertain a property of a model that correlates strongly with its generalization that we coin as "model sensitivity" We propose a novel strategy of Spectral Adversarial Data Augmentation (SADA) to generate augmented images targeted at the highly sensitive frequencies.
  arXiv  Detail & Related papers  (2022-12-01T20:15:15Z)
• Generalizable Patch-Based Neural Rendering [46.41746536545268]
  We propose a new paradigm for learning models that can synthesize novel views of unseen scenes. Our method is capable of predicting the color of a target ray in a novel scene directly, just from a collection of patches sampled from the scene. We show that our approach outperforms the state-of-the-art on novel view synthesis of unseen scenes even when being trained with considerably less data than prior work.
  arXiv  Detail & Related papers  (2022-07-21T17:57:04Z)
• DGSS : Domain Generalized Semantic Segmentation using Iterative Style Mining and Latent Representation Alignment [38.05196030226661]
  Current state-of-the-art (SoTA) have proposed different mechanisms to bridge the domain gap, but they still perform poorly in low illumination conditions. We propose a two-step framework wherein we first identify an adversarial style that maximizes the domain gap between stylized and source images. We then propose a style mixing mechanism wherein the same objects from different styles are mixed to construct a new training image.
  arXiv  Detail & Related papers  (2022-02-26T13:54:57Z)
• Bridging Composite and Real: Towards End-to-end Deep Image Matting [88.79857806542006]
  We study the roles of semantics and details for image matting. We propose a novel Glance and Focus Matting network (GFM), which employs a shared encoder and two separate decoders. Comprehensive empirical studies have demonstrated that GFM outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-10-30T10:57:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.