A Hybrid Multimodal Deep Learning Framework for Intelligent Fashion Recommendation

• URL: http://arxiv.org/abs/2511.07573v2
• Date: Tue, 18 Nov 2025 20:29:32 GMT
• Title: A Hybrid Multimodal Deep Learning Framework for Intelligent Fashion Recommendation
• Authors: Kamand Kalashi, Babak Teimourpour,
• Abstract summary: This paper proposes a hybrid multimodal deep learning framework for fashion recommendation.<n>It addresses two key tasks: outfit compatibility prediction and complementary item retrieval.<n>The proposed approach demonstrates strong performance across both tasks, highlighting the effectiveness of multimodal learning for fashion recommendation.
• Score: 0.8602553195689512
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid expansion of online fashion platforms has created an increasing demand for intelligent recommender systems capable of understanding both visual and textual cues. This paper proposes a hybrid multimodal deep learning framework for fashion recommendation that jointly addresses two key tasks: outfit compatibility prediction and complementary item retrieval. The model leverages the visual and textual encoders of the CLIP architecture to obtain joint latent representations of fashion items, which are then integrated into a unified feature vector and processed by a transformer encoder. For compatibility prediction, an "outfit token" is introduced to model the holistic relationships among items, achieving an AUC of 0.95 on the Polyvore dataset. For complementary item retrieval, a "target item token" representing the desired item description is used to retrieve compatible items, reaching an accuracy of 69.24% under the Fill-in-the-Blank (FITB) metric. The proposed approach demonstrates strong performance across both tasks, highlighting the effectiveness of multimodal learning for fashion recommendation.

Related papers

• Recommending Composite Items Using Multi-Level Preference Information: A Joint Interaction Modeling Approach [2.204009290624209]
  We propose JIMA, a joint interaction modeling approach that uses a single model to take advantage of all data from different levels of granularity.<n>We evaluate the proposed method and compare it with advanced baselines through multiple simulation studies as well as with real data in both offline and online settings.
  arXiv  Detail & Related papers  (2026-01-26T22:35:17Z)
• Learning Item Representations Directly from Multimodal Features for Effective Recommendation [51.49251689107541]
  multimodal recommender systems predominantly leverage Bayesian Personalized Ranking (BPR) optimization to learn item representations.<n>We propose a novel model (i.e., LIRDRec) that learns item representations directly from multimodal features to augment recommendation performance.
  arXiv  Detail & Related papers  (2025-05-08T05:42:22Z)
• Learning to Synthesize Compatible Fashion Items Using Semantic Alignment and Collocation Classification: An Outfit Generation Framework [59.09707044733695]
  We propose a novel outfit generation framework, i.e., OutfitGAN, with the aim of synthesizing an entire outfit.<n> OutfitGAN includes a semantic alignment module, which is responsible for characterizing the mapping correspondence between the existing fashion items and the synthesized ones.<n>In order to evaluate the performance of our proposed models, we built a large-scale dataset consisting of 20,000 fashion outfits.
  arXiv  Detail & Related papers  (2025-02-05T12:13:53Z)
• Leveraging Weak Cross-Modal Guidance for Coherence Modelling via Iterative Learning [66.28872204574648]
  Cross-modal coherence modeling is essential for intelligent systems to help them organize and structure information. Previous work on cross-modal coherence modeling attempted to leverage the order information from another modality to assist the coherence recovering of the target modality. This paper explores a new way to take advantage of cross-modal guidance without gold labels on coherency.
  arXiv  Detail & Related papers  (2024-08-01T06:04:44Z)
• OutfitTransformer: Learning Outfit Representations for Fashion Recommendation [6.890771095769622]
  We present a framework, OutfitTransformer, that learns effective outfit-level representations encoding the compatibility relationships between all items in the entire outfit. For compatibility prediction, we design an outfit token to capture a global outfit representation and train the framework using a classification loss. For complementary item retrieval, we design a target item token that additionally takes the target item specification into consideration. The generated target item embedding is then used to retrieve compatible items that match the rest of the outfit.
  arXiv  Detail & Related papers  (2022-04-11T00:55:40Z)
• Fashionformer: A simple, Effective and Unified Baseline for Human Fashion Segmentation and Recognition [80.74495836502919]
  In this work, we focus on joint human fashion segmentation and attribute recognition. We introduce the object query for segmentation and the attribute query for attribute prediction. For attribute stream, we design a novel Multi-Layer Rendering module to explore more fine-grained features.
  arXiv  Detail & Related papers  (2022-04-10T11:11:10Z)
• Latent Structures Mining with Contrastive Modality Fusion for Multimedia Recommendation [22.701371886522494]
  We argue that the latent semantic item-item structures underlying multimodal contents could be beneficial for learning better item representations. We devise a novel modality-aware structure learning module, which learns item-item relationships for each modality.
  arXiv  Detail & Related papers  (2021-11-01T03:37:02Z)
• Mining Latent Structures for Multimedia Recommendation [46.70109406399858]
  We propose a LATent sTructure mining method for multImodal reCommEndation, which we term LATTICE for brevity. We learn item-item structures for each modality and aggregates multiple modalities to obtain latent item graphs. Based on the learned latent graphs, we perform graph convolutions to explicitly inject high-order item affinities into item representations.
  arXiv  Detail & Related papers  (2021-04-19T03:50:24Z)
• Pre-training Graph Transformer with Multimodal Side Information for Recommendation [82.4194024706817]
  We propose a pre-training strategy to learn item representations by considering both item side information and their relationships. We develop a novel sampling algorithm named MCNSampling to select contextual neighbors for each item. The proposed Pre-trained Multimodal Graph Transformer (PMGT) learns item representations with two objectives: 1) graph structure reconstruction, and 2) masked node feature reconstruction.
  arXiv  Detail & Related papers  (2020-10-23T10:30:24Z)
• Learning Diverse Fashion Collocation by Neural Graph Filtering [78.9188246136867]
  We propose a novel fashion collocation framework, Neural Graph Filtering, that models a flexible set of fashion items via a graph neural network. By applying symmetric operations on the edge vectors, this framework allows varying numbers of inputs/outputs and is invariant to their ordering. We evaluate the proposed approach on three popular benchmarks, the Polyvore dataset, the Polyvore-D dataset, and our reorganized Amazon Fashion dataset.
  arXiv  Detail & Related papers  (2020-03-11T16:17:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.