Text-Guided Visual Representation Learning for Robust Multimodal E-Commerce Recommendation

• URL: http://arxiv.org/abs/2605.17366v1
• Date: Sun, 17 May 2026 10:20:23 GMT
• Title: Text-Guided Visual Representation Learning for Robust Multimodal E-Commerce Recommendation
• Abstract summary: Text-Guided Q-Former is a text-guided visual representation learning framework.<n>It disentangles metadata-anchored and exploratory visual streams, and introduces a lightweight reliability-aware dual-gated vector modulation module.<n>Experiments on large-scale, real-world e-commerce datasets with full-pool retrieval show that TGQ-Former consistently outperforms strong connector baselines and end-to-end MLLMs.
• Score: 51.52908699287466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal item embeddings are crucial for e-commerce item-to-item (I2I) retrieval, yet real-world product images often contain promotional overlays and background clutter that inject spurious visual cues and degrade retrieval robustness. This issue is particularly pronounced in MLRM-style pipelines, where a frozen vision encoder is connected to an LLM through a lightweight connector that must selectively aggregate visual tokens. We propose Text-Guided Q-Former (TGQ-Former), a text-guided visual representation learning framework that leverages structured metadata as semantic guidance for visual token extraction while preserving complementary visual evidence. Concretely, TGQ-Former employs a hybrid-query connector to disentangle metadata-anchored and exploratory visual streams, and introduces a lightweight reliability-aware dual-gated vector modulation module to adaptively calibrate their contributions under noisy inputs. Experiments on large-scale, real-world e-commerce datasets with full-pool retrieval show that TGQ-Former consistently outperforms strong connector baselines and end-to-end MLLMs. On average, it improves Hit Rate@100 (H@100) by 6.04%, demonstrating the effectiveness of text-guided visual encoding for robust multimodal retrieval.

Related papers

• Visual Search Augmented Chain-of-Thought Reasoning for Attribute Value Extraction from Product Videos [14.43540077402288]
  We propose visual search augmented chain-of-thought reasoning (ViS-CoT) for visual attribute value extraction.<n>ViS-CoT can be easily applied to any open-source video VLM for video-to-text AVE in e-Commerce.
  arXiv  Detail & Related papers  (2026-09-06T06:01:57Z)
• AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commerce [16.733243065736023]
  We propose Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning (AFMRL)<n>It defines product fine-grained understanding as an attribute generation task.<n>It leverages the generative power of Multimodal Large Language Models (MLLMs) to extract key attributes from product images and text.
  arXiv  Detail & Related papers  (2026-04-22T03:02:21Z)
• Exploring a Unified Vision-Centric Contrastive Alternatives on Multi-Modal Web Documents [99.62178668680578]
  We propose Vision-Centric Contrastive Learning (VC2L), a unified framework that models text, images, and their combinations using a single vision transformer.<n> VC2L operates entirely in pixel space by rendering all inputs, whether textual, visual, or combined, as images.<n>To capture complex cross-modal relationships in web documents, VC2L employs a snippet-level contrastive learning objective that aligns consecutive multimodal segments.
  arXiv  Detail & Related papers  (2025-10-21T14:59:29Z)
• Learning Item Representations Directly from Multimodal Features for Effective Recommendation [51.49251689107541]
  multimodal recommender systems predominantly leverage Bayesian Personalized Ranking (BPR) optimization to learn item representations.<n>We propose a novel model (i.e., LIRDRec) that learns item representations directly from multimodal features to augment recommendation performance.
  arXiv  Detail & Related papers  (2025-05-08T05:42:22Z)
• QID: Efficient Query-Informed ViTs in Data-Scarce Regimes for OCR-free Visual Document Understanding [53.69841526266547]
  Fine-tuning a pre-trained Vision-Language Model with new datasets often falls short in optimizing the vision encoder.<n>We introduce QID, a novel, streamlined, architecture-preserving approach that integrates query embeddings into the vision encoder.
  arXiv  Detail & Related papers  (2025-04-03T18:47:16Z)
• Towards Text-Image Interleaved Retrieval [49.96332254241075]
  We introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences.<n>We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries.<n>We propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity.
  arXiv  Detail & Related papers  (2025-02-18T12:00:47Z)
• Instruction Tuning-free Visual Token Complement for Multimodal LLMs [51.138806401996696]
  multimodal large language models (MLLMs) have promised an elegant bridge between vision and language. We propose a Visual Token Complement framework (VTC) that helps MLLMs regain the missing visual features. Our VTC integrates text-to-image generation as a guide to identifying the text-irrelevant features, and a visual selector is then developed to generate complementary visual tokens.
  arXiv  Detail & Related papers  (2024-08-09T12:13:01Z)
• Mobile User Interface Element Detection Via Adaptively Prompt Tuning [15.438504077368936]
  We develop a new MUI element detection dataset named MUI-zh. We propose an Adaptively Prompt Tuning (APT) module to take advantage of discriminating OCR information.
  arXiv  Detail & Related papers  (2023-05-16T07:16:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.