The Role of Visualization in LLM-Assisted Knowledge Graph Systems: Effects on User Trust, Exploration, and Workflows

• URL: http://arxiv.org/abs/2505.21512v1
• Date: Tue, 20 May 2025 18:54:59 GMT
• Title: The Role of Visualization in LLM-Assisted Knowledge Graph Systems: Effects on User Trust, Exploration, and Workflows
• Authors: Harry Li, Gabriel Appleby, Kenneth Alperin, Steven R Gomez, Ashley Suh,
• Abstract summary: LinkQ is an exploration system that converts natural language questions into structured queries with a large language model (LLMs)<n>From a qualitative evaluation with 14 practitioners, we found that users - even KG experts - tended to overtrust LinkQ's outputs due to its "helpful" visualizations.
• Score: 2.40997250653065
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge graphs (KGs) are powerful data structures, but exploring them effectively remains difficult for even expert users. Large language models (LLMs) are increasingly used to address this gap, yet little is known empirically about how their usage with KGs shapes user trust, exploration strategies, or downstream decision-making - raising key design challenges for LLM-based KG visual analysis systems. To study these effects, we developed LinkQ, a KG exploration system that converts natural language questions into structured queries with an LLM. We collaborated with KG experts to design five visual mechanisms that help users assess the accuracy of both KG queries and LLM responses: an LLM-KG state diagram that illustrates which stage of the exploration pipeline LinkQ is in, a query editor displaying the generated query paired with an LLM explanation, an entity-relation ID table showing extracted KG entities and relations with semantic descriptions, a query structure graph that depicts the path traversed in the KG, and an interactive graph visualization of query results. From a qualitative evaluation with 14 practitioners, we found that users - even KG experts - tended to overtrust LinkQ's outputs due to its "helpful" visualizations, even when the LLM was incorrect. Users exhibited distinct workflows depending on their prior familiarity with KGs and LLMs, challenging the assumption that these systems are one-size-fits-all - despite often being designed as if they are. Our findings highlight the risks of false trust in LLM-assisted data analysis tools and the need for further investigation into the role of visualization as a mitigation technique.

Related papers

• Enhancing Manufacturing Knowledge Access with LLMs and Context-aware Prompting [9.520082987178851]
  Large Language Models (LLMs) can automatically translate natural language queries into the SPARQL format.<n>We evaluate strategies that use LLMs as mediators to facilitate information retrieval from Knowledge graphs (KGs)<n>Our findings show that LLMs can significantly improve their performance on generating correct and complete queries when provided only the adequate context of the KG schema.
  arXiv  Detail & Related papers  (2025-07-30T12:39:01Z)
• ClaimPKG: Enhancing Claim Verification via Pseudo-Subgraph Generation with Lightweight Specialized LLM [3.864321514889099]
  ClaimPKG is an end-to-end framework that seamlessly integrates LLM reasoning with structured knowledge from knowledge graphs (KGs)<n>ClaimPKG achieves state-of-the-art performance, outperforming strong baselines in this research field by 9%-12% accuracy points across multiple categories.
  arXiv  Detail & Related papers  (2025-05-28T16:34:14Z)
• LightPROF: A Lightweight Reasoning Framework for Large Language Model on Knowledge Graph [57.382255728234064]
  Large Language Models (LLMs) have impressive capabilities in text understanding and zero-shot reasoning.<n> Knowledge Graphs (KGs) provide rich and reliable contextual information for the reasoning process of LLMs.<n>We propose a novel Lightweight and efficient Prompt learning-ReasOning Framework for KGQA (LightPROF)
  arXiv  Detail & Related papers  (2025-04-04T03:03:47Z)
• Augmented Knowledge Graph Querying leveraging LLMs [2.5311562666866494]
  We introduce SparqLLM, a framework that enhances the querying of Knowledge Graphs (KGs)<n>SparqLLM executes the Extract, Transform, and Load (ETL) pipeline to construct KGs from raw data.<n>It also features a natural language interface powered by Large Language Models (LLMs) to enable automatic SPARQL query generation.
  arXiv  Detail & Related papers  (2025-02-03T12:18:39Z)
• Decoding on Graphs: Faithful and Sound Reasoning on Knowledge Graphs through Generation of Well-Formed Chains [66.55612528039894]
  Knowledge Graphs (KGs) can serve as reliable knowledge sources for question answering (QA) We present DoG (Decoding on Graphs), a novel framework that facilitates a deep synergy between LLMs and KGs. Experiments across various KGQA tasks with different background KGs demonstrate that DoG achieves superior and robust performance.
  arXiv  Detail & Related papers  (2024-10-24T04:01:40Z)
• Graph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models [92.71304585906624]
  Large language models (LLMs) struggle with faithful reasoning due to knowledge gaps and hallucinations.<n>We introduce graph-constrained reasoning (GCR), a novel framework that bridges structured knowledge in KGs with unstructured reasoning in LLMs.<n>GCR achieves state-of-the-art performance and exhibits strong zero-shot generalizability to unseen KGs without additional training.
  arXiv  Detail & Related papers  (2024-10-16T22:55:17Z)
• LinkQ: An LLM-Assisted Visual Interface for Knowledge Graph Question-Answering [1.5238808518078564]
  LinkQ is a system that leverages a large language model (LLM) to facilitate knowledge graph (KG) query construction through natural language question-answering.<n>LLM interprets a user's question, then systematically converts it into a well-formed query.<n>LinkQ guards against the LLM hallucinating outputs by ensuring users' questions are only ever answered from ground truth KG data.
  arXiv  Detail & Related papers  (2024-06-07T15:28:31Z)
• KG-RAG: Bridging the Gap Between Knowledge and Creativity [0.0]
  Large Language Model Agents (LMAs) face issues such as information hallucinations, catastrophic forgetting, and limitations in processing long contexts. This paper introduces a KG-RAG (Knowledge Graph-Retrieval Augmented Generation) pipeline to enhance the knowledge capabilities of LMAs. Preliminary experiments on the ComplexWebQuestions dataset demonstrate notable improvements in the reduction of hallucinated content.
  arXiv  Detail & Related papers  (2024-05-20T14:03:05Z)
• Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering [87.67177556994525]
  We propose a training-free method called Generate-on-Graph (GoG) to generate new factual triples while exploring Knowledge Graphs (KGs) GoG performs reasoning through a Thinking-Searching-Generating framework, which treats LLM as both Agent and KG in IKGQA.
  arXiv  Detail & Related papers  (2024-04-23T04:47:22Z)
• KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge Graph [134.8631016845467]
  We propose an autonomous LLM-based agent framework, called KG-Agent. In KG-Agent, we integrate the LLM, multifunctional toolbox, KG-based executor, and knowledge memory. To guarantee the effectiveness, we leverage program language to formulate the multi-hop reasoning process over the KG.
  arXiv  Detail & Related papers  (2024-02-17T02:07:49Z)
• Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting [51.7049140329611]
  This paper proposes Knowledge Graph-based Retrofitting (KGR) to mitigate factual hallucination during the reasoning process. Experiments show that KGR can significantly improve the performance of LLMs on factual QA benchmarks.
  arXiv  Detail & Related papers  (2023-11-22T11:08:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.