Context and Symmetry in Auditing: A Case Study of Skeleton Inference in Motion Capture

• URL: http://arxiv.org/abs/2608.10194v1
• Date: Mon, 10 Aug 2026 20:12:40 GMT
• Title: Context and Symmetry in Auditing: A Case Study of Skeleton Inference in Motion Capture
• Abstract summary: Conducting an AI audit requires identifying how a system behaves and then observing and documenting actual system behavior.<n>We argue that this is best done through a contextual audit, which we introduce as a method for auditing measurements within the context of the practices that produce them.<n>We show how contextual auditing enables the interrogation of assumptions implicit in the audit process and allows auditors to be explicit about what serves as ground truth.
• Score: 6.279749738323978
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans are increasingly expected to interact with AI systems that observe and make inferences about them - but do these systems actually work? A standard approach to answering this question is AI auditing. Conducting an AI audit requires identifying how a system behaves (i.e., determining what types of inputs to audit it with and then observing and documenting actual system behavior) and contrasting that with how a system should behave (i.e., determining what the nominal outputs of a system should look like). We argue that this is best done through a contextual audit, which we introduce as a method for auditing measurements within the context of the practices that produce them. We show how contextual auditing enables the interrogation of assumptions implicit in the audit process and allows auditors to be explicit about what serves as ground truth, which we define as verifiable measurements about the real world against which systems are evaluated. We outline how the concept of symmetry from science and technology studies can enable audits when ground truth is unknown, unknowable, or contested. Finally, to demonstrate how contextual and symmetric audits can be conducted in practice, we present a case study of skeleton inference in motion capture and propose areas suggested by our case study as particularly fruitful for future audits of motion capture systems.

Related papers

• Judgment-Grounded Expansion for Peer Review Generation [68.04679314462481]
  We formalize judgment-grounded expansion, a human-AI collaboration mode where a reviewer provides an evaluative claim and the system expands it into review comment candidate(s)<n>We study two practical challenges for judgment-grounded expansion: scalable evaluation and candidate set curation.<n>Our work establishes judgment-grounded expansion as a concrete task and provides empirical and methodological foundations for the design of future collaborative review generation systems.
  arXiv  Detail & Related papers  (2026-06-22T12:20:11Z)
• Monitoring Data-aware Temporal Properties (Extended Version) [56.386411908764494]
  We consider anticipatory monitoring of linear-time properties enriched with an arbitrary SMT theory over finite traces.<n>Anticipatory monitoring in this setting is highly challenging, as the monitoring state depends on both the trace prefix seen so far and all its possible finite continuations.<n>We present and formally prove the correctness of a novel foundational framework for monitoring properties in an expressive fragment offMT.
  arXiv  Detail & Related papers  (2026-05-14T10:23:11Z)
• Robust Verification of Controllers under State Uncertainty via Hamilton-Jacobi Reachability Analysis [49.31947916567367]
  Hamilton-Jacobi (J) reachability analysis is a popular formal verification tool for general nonlinear systems that can compute optimal reachable under worst-case uncertainties.<n>This work is the first HJ-based reachability-based system verification framework for the Robust Verification Controllers via HJ rover.<n>Within Ro-CoRe, we propose novel methods for safety verification and controller design.
  arXiv  Detail & Related papers  (2025-11-18T18:55:20Z)
• Safe and Certifiable AI Systems: Concepts, Challenges, and Lessons Learned [45.44933002008943]
  This white paper presents the T"UV AUSTRIA Trusted AI framework.<n>It is an end-to-end audit catalog and methodology for assessing and certifying machine learning systems.<n>Building on three pillars - Secure Software Development, Functional Requirements, and Ethics & Data Privacy - it translates the high-level obligations of the EU AI Act into specific, testable criteria.
  arXiv  Detail & Related papers  (2025-09-08T17:52:08Z)
• Evaluating Generative AI Systems is a Social Science Measurement Challenge [78.35388859345056]
  We present a framework for measuring concepts related to the capabilities, impacts, opportunities, and risks of GenAI systems. The framework distinguishes between four levels: the background concept, the systematized concept, the measurement instrument(s), and the instance-level measurements themselves.
  arXiv  Detail & Related papers  (2024-11-17T02:35:30Z)
• Assessing the Auditability of AI-integrating Systems: A Framework and Learning Analytics Case Study [0.0]
  We argue that the efficacy of an audit depends on the auditability of the audited system. We present a framework for assessing the auditability of AI-integrating systems.
  arXiv  Detail & Related papers  (2024-10-29T13:43:21Z)
• From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing [1.196505602609637]
  Audits can take many forms, including pre-deployment risk assessments, ongoing monitoring, and compliance testing. There are many operational challenges to AI auditing that complicate its implementation. We argue that auditing can be cast as a natural hypothesis test, draw parallels hypothesis testing and legal procedure, and argue that this framing provides clear and interpretable guidance on audit implementation.
  arXiv  Detail & Related papers  (2024-10-07T06:15:46Z)
• A Unified Causal Framework for Auditing Recommender Systems for Ethical Concerns [40.793466500324904]
  We view recommender system auditing from a causal lens and provide a general recipe for defining auditing metrics. Under this general causal auditing framework, we categorize existing auditing metrics and identify gaps in them. We propose two classes of such metrics:future- and past-reacheability and stability, that measure the ability of a user to influence their own and other users' recommendations.
  arXiv  Detail & Related papers  (2024-09-20T04:37:36Z)
• A Framework for Assurance Audits of Algorithmic Systems [2.2342503377379725]
  We propose the criterion audit as an operationalizable compliance and assurance external audit framework. We argue that AI audits should similarly provide assurance to their stakeholders about AI organizations' ability to govern their algorithms in ways that harms and uphold human values. We conclude by offering a critical discussion on the benefits, inherent limitations, and implementation challenges of applying practices of the more mature financial auditing industry to AI auditing.
  arXiv  Detail & Related papers  (2024-01-26T14:38:54Z)
• AI auditing: The Broken Bus on the Road to AI Accountability [1.9758196889515185]
  "AI audit" ecosystem is muddled and imprecise, making it difficult to work through various concepts and map out the stakeholders involved in the practice. First, we taxonomize current AI audit practices as completed by regulators, law firms, civil society, journalism, academia, consulting agencies. We find that only a subset of AI audit studies translate to desired accountability outcomes.
  arXiv  Detail & Related papers  (2024-01-25T19:00:29Z)
• Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystem [0.971392598996499]
  We provide the first comprehensive field scan of the AI audit ecosystem. We identify emerging best practices as well as methods and tools that are becoming commonplace. We outline policy recommendations to improve the quality and impact of these audits.
  arXiv  Detail & Related papers  (2023-10-04T01:40:03Z)
• Perspectives on Large Language Models for Relevance Judgment [56.935731584323996]
  Large language models (LLMs) claim that they can assist with relevance judgments. It is not clear whether automated judgments can reliably be used in evaluations of retrieval systems.
  arXiv  Detail & Related papers  (2023-04-13T13:08:38Z)
• Towards a multi-stakeholder value-based assessment framework for algorithmic systems [76.79703106646967]
  We develop a value-based assessment framework that visualizes closeness and tensions between values. We give guidelines on how to operationalize them, while opening up the evaluation and deliberation process to a wide range of stakeholders.
  arXiv  Detail & Related papers  (2022-05-09T19:28:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.