LLMography: Transforming Human-AI Conversations into Traceability, Oversight, and Auditability Indicators

• URL: http://arxiv.org/abs/2606.29437v1
• Date: Sun, 28 Jun 2026 14:54:03 GMT
• Title: LLMography: Transforming Human-AI Conversations into Traceability, Oversight, and Auditability Indicators
• Abstract summary: This paper introduces a framework for transforming Human-AI conversations into measurable indicators of provenance, human contribution, AI dependency, and auditability.<n>We present a prototype that analyzes Human-AI conversation traces and generates reports including Human-AI co-production reports and a recommended LLMography label.<n>Most interactions were classified as Human-AI co-produced, with average scores of 86.8/100 for Human Direction, 81.9/100 for Prompt Quality, 72.8/100 for Auditability, and 77.1/100 for Final Output Traceability.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The growing use of Large Language Models (LLMs) in education, software engineering, academic writing, and technical documentation raises a key question: how can we evaluate not only AI-assisted outputs, but also the interaction process that produced them? Current debates often focus on detecting whether a final artifact was generated by AI, while overlooking the conversation history that reveals human direction, AI contribution, corrections, validation, and traceability. This paper introduces LLMography, a framework for transforming Human-AI conversations into measurable indicators of provenance, human contribution, AI dependency, reproducibility, and auditability. By analogy with bibliography and webography, LLMography documents the dynamic trajectory of interaction between a human and a Large Language Model as a structured trace of Human-AI co-production. We present a prototype that analyzes Human-AI conversation traces and generates KPI reports including Prompt Quality Score, Human Direction Score, AI Dependency Level, Auditability Score, Final Output Traceability, Privacy Risk Level, and a recommended LLMography label. A preliminary exploratory evaluation was conducted on 19 anonymized audit reports from engineering students. Most interactions were classified as Human-AI co-produced, with average scores of 86.8/100 for Human Direction, 81.9/100 for Prompt Quality, 72.8/100 for Auditability, and 77.1/100 for Final Output Traceability. The paper also applies LLMography to its own writing process, classified as human-originated, human-directed, AI-assisted co-production. The findings suggest that AI transparency should move beyond output detection toward documenting the history of interaction.

Related papers

• Principal Trait Analysis: Towards Deriving "Skills" in Human-AI Collaboration [3.3894363032046506]
  Large Language Model-powered agents are increasingly used in the workplace via human-artificial intelligence (AI) collaboration.<n>We propose Principal Trait Analysis, a Principal Component Analysis-inspired algorithm for deriving common traits from patterns in conversations.<n>We find that PTA-derived traits are significant in explaining collaborator behavior across both settings and can help predict task outcomes.
  arXiv  Detail & Related papers  (2026-08-11T21:49:35Z)
• AIssistant: An Agentic Approach for Human--AI Collaborative Scientific Work on Reviews and Perspectives in Machine Learning [2.464267718050055]
  We present here the first experiments with AIssistant for perspective and review research papers in machine learning.<n>Our system integrates modular tools and agents for literature, section-wise experimentation, citation management, and automatic paper text generation.<n>Despite its effectiveness, we identify key limitations, including hallucinated citations, difficulty adapting to dynamic paper structures, and incomplete integration of multimodal content.
  arXiv  Detail & Related papers  (2025-09-14T15:50:31Z)
• The next question after Turing's question: Introducing the Grow-AI test [51.56484100374058]
  This study aims to extend the framework for assessing artificial intelligence, called GROW-AI.<n>GROW-AI is designed to answer the question "Can machines grow up?" -- a natural successor to the Turing Test.<n>The originality of the work lies in the conceptual transposition of the process of "growing" from the human world to that of artificial intelligence.
  arXiv  Detail & Related papers  (2025-08-22T10:19:42Z)
• Reflection on Data Storytelling Tools in the Generative AI Era from the Human-AI Collaboration Perspective [54.152639172274]
  Large-scale generative AI techniques have the potential to enhance data storytelling with their power in visual and narration generation.<n>We compare the collaboration patterns of the latest tools with those of earlier ones using a dedicated framework for understanding human-AI collaboration in data storytelling.<n>The benefits of these AI techniques and implications to human-AI collaboration are also revealed.
  arXiv  Detail & Related papers  (2025-03-04T13:56:18Z)
• Decoding AI and Human Authorship: Nuances Revealed Through NLP and Statistical Analysis [0.0]
  This research explores the nuanced differences in texts produced by AI and those written by humans. The study investigates various linguistic traits, patterns of creativity, and potential biases inherent in human-written and AI- generated texts.
  arXiv  Detail & Related papers  (2024-07-15T18:09:03Z)
• Detecting AI-Generated Text: Factors Influencing Detectability with Current Methods [13.14749943120523]
  Knowing whether a text was produced by human or artificial intelligence (AI) is important to determining its trustworthiness.<n>State-of-the art approaches to AIGT detection include watermarking, statistical and stylistic analysis, and machine learning classification.<n>We aim to provide insight into the salient factors that combine to determine how "detectable" AIGT text is under different scenarios.
  arXiv  Detail & Related papers  (2024-06-21T18:31:49Z)
• Detecting AI Generated Text Based on NLP and Machine Learning Approaches [0.0]
  Recent advances in natural language processing may enable AI models to generate writing that is identical to human written form in the future. This might have profound ethical, legal, and social repercussions. Our approach includes a machine learning methods that can differentiate between electronically produced text and human-written text.
  arXiv  Detail & Related papers  (2024-04-15T16:37:44Z)
• Beyond Static Evaluation: A Dynamic Approach to Assessing AI Assistants' API Invocation Capabilities [48.922660354417204]
  We propose Automated Dynamic Evaluation (AutoDE) to assess an assistant's API call capability without human involvement. In our framework, we endeavor to closely mirror genuine human conversation patterns in human-machine interactions.
  arXiv  Detail & Related papers  (2024-03-17T07:34:12Z)
• Can AI Serve as a Substitute for Human Subjects in Software Engineering Research? [24.39463126056733]
  This vision paper proposes a novel approach to qualitative data collection in software engineering research by harnessing the capabilities of artificial intelligence (AI) We explore the potential of AI-generated synthetic text as an alternative source of qualitative data. We discuss the prospective development of new foundation models aimed at emulating human behavior in observational studies and user evaluations.
  arXiv  Detail & Related papers  (2023-11-18T14:05:52Z)
• Unsupervised Sentiment Analysis of Plastic Surgery Social Media Posts [91.3755431537592]
  The massive collection of user posts across social media platforms is primarily untapped for artificial intelligence (AI) use cases. Natural language processing (NLP) is a subfield of AI that leverages bodies of documents, known as corpora, to train computers in human-like language understanding. This study demonstrates that the applied results of unsupervised analysis allow a computer to predict either negative, positive, or neutral user sentiment towards plastic surgery.
  arXiv  Detail & Related papers  (2023-07-05T20:16:20Z)
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
  The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
  arXiv  Detail & Related papers  (2022-12-08T23:23:39Z)
• Can Machines Imitate Humans? Integrative Turing-like tests for Language and Vision Demonstrate a Narrowing Gap [56.611702960809644]
  We benchmark AI's ability to imitate humans in three language tasks and three vision tasks.<n>Next, we conducted 72,191 Turing-like tests with 1,916 human judges and 10 AI judges.<n>Imitation ability showed minimal correlation with conventional AI performance metrics.
  arXiv  Detail & Related papers  (2022-11-23T16:16:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.