Enhancing Multimodal Recommendations with Vision-Language Models and Information-Aware Fusion

• URL: http://arxiv.org/abs/2511.02113v2
• Date: Mon, 10 Nov 2025 06:11:16 GMT
• Title: Enhancing Multimodal Recommendations with Vision-Language Models and Information-Aware Fusion
• Authors: Hai-Dang Kieu, Min Xu, Thanh Trung Huynh, Dung D. Le,
• Abstract summary: VIRAL is a novel Vision-Language and Information-aware Recommendation framework.<n>It generates fine-grained, title-guided descriptions for semantically aligned image representations.<n>Experiments on three Amazon datasets show that VIRAL consistently outperforms strong multimodal baselines.
• Score: 11.914081442317494
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in multimodal recommendation (MMR) highlight the potential of integrating visual and textual content to enrich item representations. However, existing methods often rely on coarse visual features and naive fusion strategies, resulting in redundant or misaligned representations. From an information-theoretic perspective, effective fusion should balance unique, shared, and redundant modality information to preserve complementary cues. To this end, we propose VIRAL, a novel Vision-Language and Information-aware Recommendation framework that enhances multimodal fusion through two components: (i) a VLM-based visual enrichment module that generates fine-grained, title-guided descriptions for semantically aligned image representations, and (ii) an information-aware fusion module inspired by Partial Information Decomposition (PID) to disentangle and integrate complementary signals. Experiments on three Amazon datasets show that VIRAL consistently outperforms strong multimodal baselines and substantially improves the contribution of visual features.

Related papers

• Query-Kontext: An Unified Multimodal Model for Image Generation and Editing [53.765351127477224]
  Unified Multimodal Models (UMMs) have demonstrated remarkable performance in text-to-image generation (T2I) and editing (TI2I)<n>We introduce Query-Kontext, a novel approach that bridges the VLM and diffusion model via a multimodal kontext'' composed of semantic cues and coarse-grained image conditions encoded from multimodal inputs.<n> Experiments show that our approach matches strong unified baselines and even outperforms task-specific state-of-the-art methods in several cases.
  arXiv  Detail & Related papers  (2025-09-30T17:59:46Z)
• Do Recommender Systems Really Leverage Multimodal Content? A Comprehensive Analysis on Multimodal Representations for Recommendation [9.37169920239321]
  Multimodal Recommender Systems aim to improve recommendation accuracy by integrating heterogeneous content.<n>While effective, it remains unclear whether their gains stem from true multimodal understanding or increased model complexity.<n>This work investigates the role of multimodal item embeddings, emphasizing the semantic informativeness of the representations.
  arXiv  Detail & Related papers  (2025-08-06T15:53:58Z)
• Gated Multimodal Graph Learning for Personalized Recommendation [9.466822984141086]
  Multimodal recommendation has emerged as a promising solution to alleviate the cold-start and sparsity problems in collaborative filtering.<n>We propose RLMultimodalRec, a lightweight and modular recommendation framework that combines graph-based user modeling with adaptive multimodal item encoding.
  arXiv  Detail & Related papers  (2025-05-30T16:57:17Z)
• Learning Item Representations Directly from Multimodal Features for Effective Recommendation [51.49251689107541]
  multimodal recommender systems predominantly leverage Bayesian Personalized Ranking (BPR) optimization to learn item representations.<n>We propose a novel model (i.e., LIRDRec) that learns item representations directly from multimodal features to augment recommendation performance.
  arXiv  Detail & Related papers  (2025-05-08T05:42:22Z)
• Multimodal-Aware Fusion Network for Referring Remote Sensing Image Segmentation [7.992331117310217]
  Referring remote sensing image segmentation (RRSIS) is a novel visual task in remote sensing images segmentation.<n>We design a multimodal-aware fusion network (MAFN) to achieve fine-grained alignment and fusion between the two modalities.
  arXiv  Detail & Related papers  (2025-03-14T08:31:21Z)
• Multimodality Helps Few-shot 3D Point Cloud Semantic Segmentation [61.91492500828508]
  Few-shot 3D point cloud segmentation (FS-PCS) aims at generalizing models to segment novel categories with minimal support samples.<n>We introduce a multimodal FS-PCS setup, utilizing textual labels and the potentially available 2D image modality.<n>We propose a simple yet effective Test-time Adaptive Cross-modal (TACC) technique to mitigate training bias.
  arXiv  Detail & Related papers  (2024-10-29T19:28:41Z)
• X-Reflect: Cross-Reflection Prompting for Multimodal Recommendation [46.76427517818944]
  Cross-Reflection Prompting is designed to explicitly identify and reconcile supportive and conflicting information between text and images.<n>Experiments conducted on two widely used benchmarks demonstrate that our method outperforms existing prompting baselines in downstream recommendation accuracy.<n>We also introduce X-Reflect-keyword, a lightweight variant that summarizes image content using keywords and replaces the base model with a smaller backbone.
  arXiv  Detail & Related papers  (2024-08-27T16:10:21Z)
• MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model [49.931663904599205]
  MaVEn is an innovative framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. We show that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.
  arXiv  Detail & Related papers  (2024-08-22T11:57:16Z)
• Leveraging Entity Information for Cross-Modality Correlation Learning: The Entity-Guided Multimodal Summarization [49.08348604716746]
  Multimodal Summarization with Multimodal Output (MSMO) aims to produce a multimodal summary that integrates both text and relevant images. In this paper, we propose an Entity-Guided Multimodal Summarization model (EGMS) Our model, building on BART, utilizes dual multimodal encoders with shared weights to process text-image and entity-image information concurrently.
  arXiv  Detail & Related papers  (2024-08-06T12:45:56Z)
• NoteLLM-2: Multimodal Large Representation Models for Recommendation [71.87790090964734]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks.<n>Their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored.<n>We propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation.
  arXiv  Detail & Related papers  (2024-05-27T03:24:01Z)
• Multi-Grained Multimodal Interaction Network for Entity Linking [65.30260033700338]
  Multimodal entity linking task aims at resolving ambiguous mentions to a multimodal knowledge graph. We propose a novel Multi-GraIned Multimodal InteraCtion Network $textbf(MIMIC)$ framework for solving the MEL task.
  arXiv  Detail & Related papers  (2023-07-19T02:11:19Z)
• A Clustering-guided Contrastive Fusion for Multi-view Representation Learning [7.630965478083513]
  We propose a deep fusion network to fuse view-specific representations into the view-common representation. We also design an asymmetrical contrastive strategy that aligns the view-common representation and each view-specific representation. In the incomplete view scenario, our proposed method resists noise interference better than those of our competitors.
  arXiv  Detail & Related papers  (2022-12-28T07:21:05Z)
• Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation [87.01669173673288]
  We propose an encoder fusion network (EFN), which transforms the visual encoder into a multi-modal feature learning network. A co-attention mechanism is embedded in the EFN to realize the parallel update of multi-modal features. The experiment results on four benchmark datasets demonstrate that the proposed approach achieves the state-of-the-art performance without any post-processing.
  arXiv  Detail & Related papers  (2021-05-05T02:27:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.