HA-FGOVD: Highlighting Fine-grained Attributes via Explicit Linear Composition for Open-Vocabulary Object Detection

• URL: http://arxiv.org/abs/2409.16136v1
• Date: Tue, 24 Sep 2024 14:43:14 GMT
• Title: HA-FGOVD: Highlighting Fine-grained Attributes via Explicit Linear Composition for Open-Vocabulary Object Detection
• Abstract summary: We propose a universal and explicit approach for frozen mainstream OVD models that boosts their attribute-level detection capabilities. Our proposed method uniformly improves fine-grained attribute-level OVD of various mainstream models and achieves new state-of-the-art performance.
• Score: 11.792370846774133
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-vocabulary object detection (OVD) models are considered to be Large Multi-modal Models (LMM), due to their extensive training data and a large number of parameters. Mainstream OVD models prioritize object coarse-grained category rather than focus on their fine-grained attributes, e.g., colors or materials, thus failed to identify objects specified with certain attributes. However, OVD models are pretrained on large-scale image-text pairs with rich attribute words, whose latent feature space can represent the global text feature as a linear composition of fine-grained attribute tokens without highlighting them. Therefore, we propose in this paper a universal and explicit approach for frozen mainstream OVD models that boosts their attribute-level detection capabilities by highlighting fine-grained attributes in explicit linear space. Firstly, a LLM is leveraged to highlight attribute words within the input text as a zero-shot prompted task. Secondly, by strategically adjusting the token masks, the text encoders of OVD models extract both global text and attribute-specific features, which are then explicitly composited as two vectors in linear space to form the new attribute-highlighted feature for detection tasks, where corresponding scalars are hand-crafted or learned to reweight both two vectors. Notably, these scalars can be seamlessly transferred among different OVD models, which proves that such an explicit linear composition is universal. Empirical evaluation on the FG-OVD dataset demonstrates that our proposed method uniformly improves fine-grained attribute-level OVD of various mainstream models and achieves new state-of-the-art performance.

Related papers

• DSAA: Dual-Stage Attribute Activation for Fine-grained Open Vocabulary Detection [10.512210722250236]
  Open-Vocabulary Object Detection (OVD) models break the limitations of closed-set detection.<n>They exhibit notable limitations in fine-grained detection tasks involving attributes like color, ma- terial, and texture.<n>We propose the Dual-Stage Attribute Activation framework, which enhances fine-grained detection capa- bilities.
  arXiv  Detail & Related papers  (2026-05-18T08:13:13Z)
• GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection [54.19989440021701]
  Fine-grained open-vocabulary object detection (FG-OVD) aims to detect novel object categories described by attribute-rich texts.<n>FG-OVD achieves new state-of-the-art results, demonstrating the benefits of disentangled modeling and modular optimization.
  arXiv  Detail & Related papers  (2026-03-27T22:08:11Z)
• Attribute Distribution Modeling and Semantic-Visual Alignment for Generative Zero-shot Learning [32.153364858061614]
  Generative zero-shot learning (ZSL) synthesizes features for unseen classes, leveraging semantic conditions to transfer knowledge from seen classes.<n>It also introduces two intrinsic challenges: (1) class-level attributes fails to capture instance-specific visual appearances due to substantial intra-class variability, thus causing the class-instance gap.<n>We propose an Attribute Distribution Modeling and Semantic-Visual Alignment (ADiVA) approach, jointly modeling attribute distributions and performing explicit semantic-visual alignment.
  arXiv  Detail & Related papers  (2026-03-06T13:41:57Z)
• IAD-GPT: Advancing Visual Knowledge in Multimodal Large Language Model for Industrial Anomaly Detection [70.02774285130238]
  This paper explores the combination of rich text semantics with both image-level and pixel-level information from images.<n>We propose IAD-GPT, a novel paradigm based on MLLMs for Industrial Anomaly Detection.<n>Experiments on MVTec-AD and VisA datasets demonstrate our state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-10-16T02:48:05Z)
• Towards Open-Vocabulary Multimodal 3D Object Detection with Attributes [22.856150330288184]
  OVODA is a novel framework enabling open-vocabulary 3D object and attribute detection with no need to know the novel class anchor size.<n>OVODA incorporates several key innovations, including foundation model concatenation, prompt tuning strategies, and specialized techniques for attribute detection.<n>Our results on both the nuScenes and Argoverse 2 datasets show that under the condition of no given anchor sizes of novel classes, OVODA outperforms the state-of-the-art methods in open-vocabulary 3D object detection.
  arXiv  Detail & Related papers  (2025-08-22T22:02:49Z)
• ROVI: A VLM-LLM Re-Captioned Dataset for Open-Vocabulary Instance-Grounded Text-to-Image Generation [23.118080583803266]
  We present ROVI, a high-quality synthetic dataset for instance-grounded text-to-image generation.<n>Our key innovation is a strategy called recaptioning, focusing on the pre-detection stage.<n>For demonstrative purposes, a text-to-image model GLIGEN trained on ROVI significantly outperforms state-of-the-art alternatives in instance grounding accuracy, prompt fidelity, and aesthetic quality.
  arXiv  Detail & Related papers  (2025-08-01T18:19:51Z)
• Beyond Accuracy: Metrics that Uncover What Makes a 'Good' Visual Descriptor [4.76296755805531]
  Descriptors are used in visual concept discovery and image classification with vision-based models (VLMs)<n>We systematically analyze descriptor quality along two key dimensions: (1) representational capacity, and (2) relationship with VLM pre-training data.<n>Motivated by ideas from representation alignment and language understanding, we introduce two alignment-based metrics--Global Alignment and CLIP Similarity--that move beyond accuracy.
  arXiv  Detail & Related papers  (2025-07-04T12:50:04Z)
• Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddings [0.0]
  This paper proposes a framework for uncovering the specific dimensions of vector embeddings that encode distinct linguistic properties (LPs) We introduce the Linguistically Distinct Sentence Pairs dataset, which isolates ten key linguistic features such as synonymy, negation, tense, and quantity. Using this dataset, we analyze BERT embeddings with various methods to identify the most influential dimensions for each LP. Our findings show that certain properties, such as negation and polarity, are robustly encoded in specific dimensions, while others, like synonymy, exhibit more complex patterns.
  arXiv  Detail & Related papers  (2025-04-20T23:38:16Z)
• LATex: Leveraging Attribute-based Text Knowledge for Aerial-Ground Person Re-Identification [63.07563443280147]
  We propose a novel framework named LATex for AG-ReID. It adopts prompt-tuning strategies to leverage attribute-based text knowledge. Our framework can fully leverage attribute-based text knowledge to improve the AG-ReID.
  arXiv  Detail & Related papers  (2025-03-31T04:47:05Z)
• The Representation and Recall of Interwoven Structured Knowledge in LLMs: A Geometric and Layered Analysis [0.0]
  Large language models (LLMs) represent and recall multi-associated attributes across transformer layers. intermediate layers encode factual knowledge by superimposing related attributes in overlapping spaces. later layers refine linguistic patterns and progressively separate attribute representations.
  arXiv  Detail & Related papers  (2025-02-15T18:08:51Z)
• Hybrid Discriminative Attribute-Object Embedding Network for Compositional Zero-Shot Learning [83.10178754323955]
  Hybrid Discriminative Attribute-Object Embedding (HDA-OE) network is proposed to solve the problem of complex interactions between attributes and object visual representations. To increase the variability of training data, HDA-OE introduces an attribute-driven data synthesis (ADDS) module. To further improve the discriminative ability of the model, HDA-OE introduces the subclass-driven discriminative embedding (SDDE) module. The proposed model has been evaluated on three benchmark datasets, and the results verify its effectiveness and reliability.
  arXiv  Detail & Related papers  (2024-11-28T09:50:25Z)
• Verbalized Representation Learning for Interpretable Few-Shot Generalization [130.8173035901391]
  Verbalized Representation Learning (VRL) is a novel approach for automatically extracting human-interpretable features for object recognition. Our method captures inter-class differences and intra-class commonalities in the form of natural language. VRL achieves a 24% absolute improvement over prior state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-11-27T01:55:08Z)
• CASA: Class-Agnostic Shared Attributes in Vision-Language Models for Efficient Incremental Object Detection [30.46562066023117]
  We propose a novel method utilizing attributes in vision-language foundation models for incremental object detection. Our method constructs a Class-Agnostic Shared Attribute base (CASA) to capture common semantic information among incremental classes. Our method adds only 0.7% to parameter storage through parameter-efficient fine-tuning to significantly enhance the scalability and adaptability of our proposed method.
  arXiv  Detail & Related papers  (2024-10-08T08:36:12Z)
• LLM-based feature generation from text for interpretable machine learning [0.0]
  Existing text representations such as embeddings and bag-of-words are not suitable for rule learning due to their high dimensionality and absent or questionable feature-level interpretability. This article explores whether large language models (LLMs) could address this by extracting a small number of interpretable features from text.
  arXiv  Detail & Related papers  (2024-09-11T09:29:28Z)
• Grouped Discrete Representation Guides Object-Centric Learning [18.44580501357929]
  Transformer-based Object-Centric Discrete Learning can abstract dense images or textures into sparse object-level features. We propose textitGrouped Representation (GDR) to address these issues by grouping features into attributes and indexing them with numbers.
  arXiv  Detail & Related papers  (2024-07-01T19:00:40Z)
• CFPL-FAS: Class Free Prompt Learning for Generalizable Face Anti-spoofing [66.6712018832575]
  Domain generalization (DG) based Face Anti-Spoofing (FAS) aims to improve the model's performance on unseen domains. We make use of large-scale VLMs like CLIP and leverage the textual feature to dynamically adjust the classifier's weights for exploring generalizable visual features.
  arXiv  Detail & Related papers  (2024-03-21T11:58:50Z)
• Domain Adaptation for Large-Vocabulary Object Detectors [103.16365373806829]
  This paper presents KGD, a Knowledge Graph Distillation technique that exploits the implicit knowledge graphs (KG) in CLIP for effectively adapting LVDs to various downstream domains. Experiments over multiple widely adopted detection benchmarks show that KGD outperforms the state-of-the-art consistently by large margins.
  arXiv  Detail & Related papers  (2024-01-13T03:51:18Z)
• Exploiting Contextual Target Attributes for Target Sentiment Classification [53.30511968323911]
  Existing PTLM-based models for TSC can be categorized into two groups: 1) fine-tuning-based models that adopt PTLM as the context encoder; 2) prompting-based models that transfer the classification task to the text/word generation task. We present a new perspective of leveraging PTLM for TSC: simultaneously leveraging the merits of both language modeling and explicit target-context interactions via contextual target attributes.
  arXiv  Detail & Related papers  (2023-12-21T11:45:28Z)
• DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detection [56.513637720967566]
  Large language models (LLMs) can generate texts that pose risks of misuse, such as plagiarism, planting fake reviews on e-commerce platforms, or creating inflammatory false tweets. Existing high-quality detection methods usually require access to the interior of the model to extract the intrinsic characteristics. We propose to extract deep intrinsic characteristics of the black-box model generated texts.
  arXiv  Detail & Related papers  (2023-05-21T17:26:16Z)
• OvarNet: Towards Open-vocabulary Object Attribute Recognition [42.90477523238336]
  We propose a naive two-stage approach for open-vocabulary object detection and attribute classification, termed CLIP-Attr. The candidate objects are first proposed with an offline RPN and later classified for semantic category and attributes. We show that recognition of semantic category and attributes is complementary for visual scene understanding.
  arXiv  Detail & Related papers  (2023-01-23T15:59:29Z)
• Attribute-based Explanations of Non-Linear Embeddings of High-Dimensional Data [2.397739143553337]
  Non-linear Embeddings Surveyor (NoLiES) combines a novel augmentation strategy for projected data (rangesets) with interactive analysis in a small multiples setting. Rangesets use a set-based visualization approach for binned attribute values that enable the user to quickly observe structure and detect outliers.
  arXiv  Detail & Related papers  (2021-07-28T12:09:29Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)
• Attribute-based Regularization of Latent Spaces for Variational Auto-Encoders [79.68916470119743]
  We present a novel method to structure the latent space of a Variational Auto-Encoder (VAE) to encode different continuous-valued attributes explicitly. This is accomplished by using an attribute regularization loss which enforces a monotonic relationship between the attribute values and the latent code of the dimension along which the attribute is to be encoded.
  arXiv  Detail & Related papers  (2020-04-11T20:53:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.