Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models

• URL: http://arxiv.org/abs/2607.02909v1
• Date: Fri, 03 Jul 2026 03:16:41 GMT
• Title: Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models
• Abstract summary: Large multimodal models (LMMs) often lack taxonomic knowledge, leading to low hierarchical visual recognition (HVR) consistency.<n>We propose Hierarchical Representation Regularization ($HiR2$), a simple plug-and-play regularizer that improves hierarchical consistency in LMMs.<n>$HiR2$ effectively captures taxonomic structures across diverse LMMs and fine-tuning methods.
• Score: 47.868429337792314
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Taxonomies provide key information about the semantic relationships between concepts and the inherent organization of vision and language. Despite their impressive capabilities, large multimodal models (LMMs) often lack taxonomic knowledge, leading to low hierarchical visual recognition (HVR) consistency. These models typically only rely on language modeling objectives during fine-tuning and lack explicit taxonomy-aware regularization. To address this, we propose Hierarchical Representation Regularization ($HiR^2$), a simple plug-and-play regularizer that improves hierarchical consistency in LMMs. Specifically, we introduce a semantic-aware visual tree construction framework that extracts coarse-to-fine visual features from intermediate LLM layers guided by textual cues. The regularizer combines two complementary objectives: a taxonomic entailment loss that enforces hierarchy via hyperbolic entailment cones in the Lorentz model, and a discriminative dispersive loss that promotes angular separation of semantically similar embeddings on the unit sphere without disturbing the radial hierarchical structure. Extensive experiments demonstrate that $HiR^2$ effectively captures taxonomic structures across diverse LMMs and fine-tuning methods. Code is available at https://github.com/PKU-ICST-MIPL/HiR2_ICML2026.

Related papers

• Structured-Condensed Prompt Tuning in Vision-Language Models for Fine-grained Image Recognition [38.294322671641545]
  Fine-grained image recognition is a significant challenge due to the substantial expertise and effort required for manual annotation.<n>Current methods treat class labels as isolated, discrete entities, overlooking the rich semantic relationships between them.<n>We propose Structured-Condensed Prompt Tuning ( SCPT), which enhances semantic structure modeling in prompt learning.
  arXiv  Detail & Related papers  (2026-07-07T12:09:49Z)
• Semantic Generative Tuning for Unified Multimodal Models [62.18894352635965]
  Unified multimodal models (UMMs) strive to consolidate visual understanding and visual generation within a single architecture.<n> prevailing training paradigms independently optimize understanding via sparse text signals and generation through dense pixel objectives.<n>This work presents the first systematic investigation into generative post-training, where we formulate hierarchical visual tasks as generative proxies to bridge the isolation in UMMs.
  arXiv  Detail & Related papers  (2026-05-18T17:46:46Z)
• PCA-Seg: Revisiting Cost Aggregation for Open-Vocabulary Semantic and Part Segmentation [58.1914505657064]
  This paper proposes a simple yet effective parallel cost aggregation (PCA-Seg) paradigm to alleviate the challenge of knowledge interference between class-level semantics and spatial context.<n>Experiments on eight benchmarks show that each parallel block in PCA-Seg adds merely 0.35M parameters while achieving state-of-the-art OSPS performance.
  arXiv  Detail & Related papers  (2026-03-18T09:26:43Z)
• Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models [47.868429337792314]
  A high-performing, general-purpose visual understanding model should map visual inputs to a taxonomic tree of labels.<n>We propose taxonomy-Aware Representation Alignment (TARA), a simple yet effective strategy to inject taxonomic knowledge into LMMs.<n>TARA consistently enhances LMMs' hierarchical consistency and leaf node accuracy, enabling reliable recognition of both known and novel categories.
  arXiv  Detail & Related papers  (2026-02-28T03:13:15Z)
• Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning [47.868429337792314]
  Fine-R1 is an R1-style MLLM tailored for Fine-Grained Visual Recognition.<n>With only 4-shot training, Fine-R1 outperforms existing general MLLMs, reasoning MLLMs, and even contrastive CLIP models.
  arXiv  Detail & Related papers  (2026-02-07T16:16:51Z)
• Hierarchy-Aware Fine-Tuning of Vision-Language Models [18.244518940229202]
  Vision-Language Models learn powerful multimodal representations through large-scale image-text pretraining.<n>Standard approaches treat labels as flat categories and require full fine-tuning, which is expensive and produces inconsistent predictions.<n>We propose an efficient hierarchy-aware fine-tuning framework that updates a few parameters while enforcing structural consistency.
  arXiv  Detail & Related papers  (2025-12-25T06:44:33Z)
• ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interaction [55.21514454560188]
  Unified multimodal models significantly improve visual generation by combining vision-grained models (VLMs) with diffusion models.<n>Existing methods struggle to fully balance sufficient interaction and flexible implementation due to vast representation difference.<n>We propose textbfParaUni, which extracts features from variants VLM's layers in a textbfParallel way for comprehensive information interaction.
  arXiv  Detail & Related papers  (2025-12-05T04:41:57Z)
• Leveraging Taxonomy and LLMs for Improved Multimodal Hierarchical Classification [19.80724112056431]
  We propose a taxonomy-embedded transitional LLM-agnostic framework for multimodality classification.<n>Our evaluations on the MEP-3M dataset demonstrated a significant performance improvement compared to conventional LLM structures.
  arXiv  Detail & Related papers  (2025-01-12T14:43:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.