ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant

• URL: http://arxiv.org/abs/2505.03654v2
• Date: Mon, 19 May 2025 08:25:06 GMT
• Title: ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant
• Authors: Yifan Xiang, Zhenxi Zhang, Bin Li, Yixuan Weng, Shoujun Zhou, Yangfan He, Keqin Li,
• Abstract summary: We present a new dataset named ReGraP, consisting of 120 sets of personalized knowledge.<n>We propose ReGraP-LLaVA, an MLLM trained with the corresponding KGs and CoT QA pairs.<n>Results show that the proposed model not only learns personalized knowledge but also performs relational reasoning in responses.
• Score: 16.253265097323432
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in personalized MLLMs enable effective capture of user-specific concepts, supporting both recognition of personalized concepts and contextual captioning. However, humans typically explore and reason over relations among objects and individuals, transcending surface-level information to achieve more personalized and contextual understanding. To this end, existing methods may face three main limitations: Their training data lacks multi-object sets in which relations among objects are learnable. Building on the limited training data, their models overlook the relations between different personalized concepts and fail to reason over them. Their experiments mainly focus on a single personalized concept, where evaluations are limited to recognition and captioning tasks. To address the limitations, we present a new dataset named ReGraP, consisting of 120 sets of personalized knowledge. Each set includes images, KGs, and CoT QA pairs derived from the KGs, enabling more structured and sophisticated reasoning pathways. We propose ReGraP-LLaVA, an MLLM trained with the corresponding KGs and CoT QA pairs, where soft and hard graph prompting methods are designed to align KGs within the model's semantic space. We establish the ReGraP Benchmark, which contains diverse task types: multiple-choice, fill-in-the-blank, True/False, and descriptive questions in both open- and closed-ended settings. The proposed benchmark is designed to evaluate the relational reasoning and knowledge-connection capability of personalized MLLMs. We conduct experiments on the proposed ReGraP-LLaVA and other competitive MLLMs. Results show that the proposed model not only learns personalized knowledge but also performs relational reasoning in responses, achieving the SoTA performance compared with the competitive methods. All the codes and datasets are released at: https://github.com/xyfyyds/ReGraP.

Related papers

• MC-LLaVA: Multi-Concept Personalized Vision-Language Model [51.645660375766575]
  This paper proposes the first multi-concept personalization paradigm, MC-LLaVA.<n>MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step.<n> Comprehensive qualitative and quantitative experiments demonstrate that MC-LLaVA can achieve impressive multi-concept personalized responses.
  arXiv  Detail & Related papers  (2025-03-24T16:32:17Z)
• Training-Free Personalization via Retrieval and Reasoning on Fingerprints [31.025439143093585]
  Vision Language Models (VLMs) have lead to major improvements in multimodal reasoning, yet they still struggle to understand user-specific concepts.<n>We propose a novel method, Retrieval and Reasoning for Personalization (R2P), leveraging internal knowledge of VLMs.<n>R2P consistently outperforms state-of-the-art approaches on various downstream tasks.
  arXiv  Detail & Related papers  (2025-03-24T12:36:24Z)
• Efficient Relational Context Perception for Knowledge Graph Completion [25.903926643251076]
  Knowledge Graphs (KGs) provide a structured representation of knowledge but often suffer from challenges of incompleteness.<n>Previous knowledge graph embedding models are limited in their ability to capture expressive features.<n>We propose Triple Receptance Perception architecture to model sequential information, enabling the learning of dynamic context.
  arXiv  Detail & Related papers  (2024-12-31T11:25:58Z)
• MC-LLaVA: Multi-Concept Personalized Vision-Language Model [51.645660375766575]
  This paper proposes the first multi-concept personalization paradigm, MC-LLaVA.<n>MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step.<n> Comprehensive qualitative and quantitative experiments demonstrate that MC-LLaVA can achieve impressive multi-concept personalized responses.
  arXiv  Detail & Related papers  (2024-11-18T16:33:52Z)
• RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models [53.304699445700926]
  We introduce the Retrieval Augmented Personalization framework for MLLMs' personalization.<n>Starting from a general MLLM, we turn it into a personalized assistant in three steps.<n>By pretraining on large-scale dataset, RAP-MLLMs can generalize to infinite visual concepts without additional finetuning.
  arXiv  Detail & Related papers  (2024-10-17T09:10:26Z)
• SrTR: Self-reasoning Transformer with Visual-linguistic Knowledge for Scene Graph Generation [12.977857322594206]
  One-stage scene graph generation approaches infer the effective relation between entity pairs using sparse proposal sets and a few queries. A Self-reasoning Transformer with Visual-linguistic Knowledge (SrTR) is proposed to add flexible self-reasoning ability to the model. Inspired by the large-scale pre-training image-text foundation models, visual-linguistic prior knowledge is introduced.
  arXiv  Detail & Related papers  (2022-12-19T09:47:27Z)
• MGA-VQA: Multi-Granularity Alignment for Visual Question Answering [75.55108621064726]
  Learning to answer visual questions is a challenging task since the multi-modal inputs are within two feature spaces. We propose Multi-Granularity Alignment architecture for Visual Question Answering task (MGA-VQA) Our model splits alignment into different levels to achieve learning better correlations without needing additional data and annotations.
  arXiv  Detail & Related papers  (2022-01-25T22:30:54Z)
• ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection [101.56529337489417]
  We consider the problem of Human-Object Interaction (HOI) Detection, which aims to locate and recognize HOI instances in the form of human, action, object> in images. We argue that multi-level consistencies among objects, actions and interactions are strong cues for generating semantic representations of rare or previously unseen HOIs. Our model takes visual features of candidate human-object pairs and word embeddings of HOI labels as inputs, maps them into visual-semantic joint embedding space and obtains detection results by measuring their similarities.
  arXiv  Detail & Related papers  (2020-08-14T09:11:18Z)
• KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion [99.47414073164656]
  A comprehensive knowledge graph (KG) contains an instance-level entity graph and an ontology-level concept graph. The two-view KG provides a testbed for models to "simulate" human's abilities on knowledge abstraction, concretization, and completion. We propose a unified KG benchmark by improving existing benchmarks in terms of dataset scale, task coverage, and difficulty.
  arXiv  Detail & Related papers  (2020-04-28T16:21:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.