Leveraging Taxonomy and LLMs for Improved Multimodal Hierarchical Classification

• URL: http://arxiv.org/abs/2501.06827v1
• Date: Sun, 12 Jan 2025 14:43:06 GMT
• Title: Leveraging Taxonomy and LLMs for Improved Multimodal Hierarchical Classification
• Authors: Shijing Chen, Mohamed Reda Bouadjenek, Shoaib Jameel, Usman Naseem, Basem Suleiman, Flora D. Salim, Hakim Hacid, Imran Razzak,
• Abstract summary: We propose a taxonomy-embedded transitional LLM-agnostic framework for multimodality classification.<n>Our evaluations on the MEP-3M dataset demonstrated a significant performance improvement compared to conventional LLM structures.
• Score: 19.80724112056431
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-level Hierarchical Classification (MLHC) tackles the challenge of categorizing items within a complex, multi-layered class structure. However, traditional MLHC classifiers often rely on a backbone model with independent output layers, which tend to ignore the hierarchical relationships between classes. This oversight can lead to inconsistent predictions that violate the underlying taxonomy. Leveraging Large Language Models (LLMs), we propose a novel taxonomy-embedded transitional LLM-agnostic framework for multimodality classification. The cornerstone of this advancement is the ability of models to enforce consistency across hierarchical levels. Our evaluations on the MEP-3M dataset - a multi-modal e-commerce product dataset with various hierarchical levels - demonstrated a significant performance improvement compared to conventional LLM structures.

Related papers

• Distilling Transitional Pattern to Large Language Models for Multimodal Session-based Recommendation [67.84581846180458]
  Session-based recommendation (SBR) predicts the next item based on anonymous sessions. Recent Multimodal SBR methods utilize simplistic pre-trained models for modality learning but have limitations in semantic richness. We propose a multimodal LLM-enhanced framework TPAD, which extends a distillation paradigm to decouple and align transitional patterns for promoting MSBR.
  arXiv  Detail & Related papers  (2025-04-13T07:49:08Z)
• Enforcing Consistency and Fairness in Multi-level Hierarchical Classification with a Mask-based Output Layer [25.819440955594736]
  We introduce a fair, model-agnostic layer designed to enforce taxonomy and optimize objectives, including consistency, fairness, and exact match. Our evaluations demonstrate that the proposed layer not only improves the fairness of predictions but also enforces the taxonomy, resulting in consistent predictions and superior performance.
  arXiv  Detail & Related papers  (2025-03-19T06:30:04Z)
• Category-Adaptive Cross-Modal Semantic Refinement and Transfer for Open-Vocabulary Multi-Label Recognition [59.203152078315235]
  We propose a novel category-adaptive cross-modal semantic refinement and transfer (C$2$SRT) framework to explore the semantic correlation.<n>The proposed framework consists of two complementary modules, i.e., intra-category semantic refinement (ISR) module and inter-category semantic transfer (IST) module.<n>Experiments on OV-MLR benchmarks clearly demonstrate that the proposed C$2$SRT framework outperforms current state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2024-12-09T04:00:18Z)
• Can Large Language Models Serve as Effective Classifiers for Hierarchical Multi-Label Classification of Scientific Documents at Industrial Scale? [1.0562108865927007]
  Large Language Models (LLMs) have demonstrated great potential in complex tasks such as multi-label classification.<n>We present methods that combine the strengths of LLMs with dense retrieval techniques to overcome these challenges.<n>We evaluate the effectiveness of our methods on SSRN, a large repository of preprints spanning multiple disciplines.
  arXiv  Detail & Related papers  (2024-12-06T15:51:22Z)
• FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data [64.50893177169996]
  Fine-tuning Multimodal Large Language Models (MLLMs) with Federated Learning (FL) allows for expanding the training data scope by including private data sources. We introduce a benchmark for evaluating various downstream tasks in the federated fine-tuning of MLLMs within multimodal heterogeneous scenarios. We develop a general FedMLLM framework that integrates four representative FL methods alongside two modality-agnostic strategies.
  arXiv  Detail & Related papers  (2024-11-22T04:09:23Z)
• Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts [54.529880848937104]
  We develop a unified MLLM with the MoE architecture, named Uni-MoE, that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets.
  arXiv  Detail & Related papers  (2024-05-18T12:16:01Z)
• Model Composition for Multimodal Large Language Models [71.5729418523411]
  We propose a new paradigm through the model composition of existing MLLMs to create a new model that retains the modal understanding capabilities of each original model. Our basic implementation, NaiveMC, demonstrates the effectiveness of this paradigm by reusing modality encoders and merging LLM parameters.
  arXiv  Detail & Related papers  (2024-02-20T06:38:10Z)
• Multi-Objective Reinforcement Learning Based on Decomposition: A Taxonomy and Framework [0.3069335774032178]
  Multi-objective reinforcement learning (MORL) extends traditional RL by seeking policies making different compromises among conflicting objectives. A clear categorization based on both RL and MOO/D is lacking in the existing literature. A comprehensive taxonomy for MORL/D is presented, providing a structured foundation for categorizing existing and potential MORL works. The introduced taxonomy is then used to scrutinize MORL research, enhancing clarity and conciseness through well-defined categorization.
  arXiv  Detail & Related papers  (2023-11-21T10:11:19Z)
• Learning Hierarchical Features with Joint Latent Space Energy-Based Prior [44.4434704520236]
  We study the fundamental problem of multi-layer generator models in learning hierarchical representations. We propose a joint latent space EBM prior model with multi-layer latent variables for effective hierarchical representation learning.
  arXiv  Detail & Related papers  (2023-10-14T15:44:14Z)
• HFT-ONLSTM: Hierarchical and Fine-Tuning Multi-label Text Classification [7.176984223240199]
  Hierarchical multi-label text classification (HMTC) with higher accuracy over large sets of closely related categories has become a challenging problem. We present a hierarchical and fine-tuning approach based on the Ordered Neural LSTM neural network, abbreviated as HFT-ONLSTM, for more accurate level-by-level HMTC.
  arXiv  Detail & Related papers  (2022-04-18T00:57:46Z)
• Coherent Hierarchical Multi-Label Classification Networks [56.41950277906307]
  C-HMCNN(h) is a novel approach for HMC problems, which exploits hierarchy information in order to produce predictions coherent with the constraint and improve performance. We conduct an extensive experimental analysis showing the superior performance of C-HMCNN(h) when compared to state-of-the-art models.
  arXiv  Detail & Related papers  (2020-10-20T09:37:02Z)
• A Multilayer Framework for Online Metric Learning [71.31889711244739]
  This paper proposes a multilayer framework for online metric learning to capture the nonlinear similarities among instances. A new Mahalanobis-based Online Metric Learning (MOML) algorithm is presented based on the passive-aggressive strategy and one-pass triplet construction strategy. The proposed MLOML enjoys several nice properties, indeed learns a metric progressively, and performs better on the benchmark datasets.
  arXiv  Detail & Related papers  (2018-05-15T01:10:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.