Who Gets Credit or Blame? Attributing Accountability in Modern AI Systems

• URL: http://arxiv.org/abs/2506.00175v3
• Date: Fri, 05 Sep 2025 21:05:27 GMT
• Title: Who Gets Credit or Blame? Attributing Accountability in Modern AI Systems
• Authors: Shichang Zhang, Hongzhe Du, Jiaqi W. Ma, Himabindu Lakkaraju,
• Abstract summary: AI systems are typically developed through multiple stages-pretraining, fine-tuning rounds, and subsequent adaptation or alignment.<n>This raises a critical question of accountability: when a deployed model succeeds or fails, which stage is responsible, and to what extent?<n>We pose the accountability attribution problem for tracing model behavior back to specific stages of the model development process.
• Score: 31.897288482449003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern AI systems are typically developed through multiple stages-pretraining, fine-tuning rounds, and subsequent adaptation or alignment, where each stage builds on the previous ones and updates the model in distinct ways. This raises a critical question of accountability: when a deployed model succeeds or fails, which stage is responsible, and to what extent? We pose the accountability attribution problem for tracing model behavior back to specific stages of the model development process. To address this challenge, we propose a general framework that answers counterfactual questions about stage effects: how would the model's behavior have changed if the updates from a particular stage had not occurred? Within this framework, we introduce estimators that efficiently quantify stage effects without retraining the model, accounting for both the data and key aspects of model optimization dynamics, including learning rate schedules, momentum, and weight decay. We demonstrate that our approach successfully quantifies the accountability of each stage to the model's behavior. Based on the attribution results, our method can identify and remove spurious correlations learned during image classification and text toxicity detection tasks that were developed across multiple stages. Our approach provides a practical tool for model analysis and represents a significant step toward more accountable AI development.

Related papers

• ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models [102.4511331368587]
  ARISE (Adaptive Resolution-aware Scaling Evaluation) is a novel metric designed to assess the test-time scaling effectiveness of large reasoning models.<n>We conduct comprehensive experiments evaluating state-of-the-art reasoning models across diverse domains.
  arXiv  Detail & Related papers  (2025-10-07T15:10:51Z)
• StepWiser: Stepwise Generative Judges for Wiser Reasoning [52.32416311990343]
  Process reward models address this by providing step-by-step feedback.<n>Inspired by recent advances, we reframe stepwise reward modeling from a classification task to a reasoning task itself.<n>We show it provides (i) better judgment accuracy on intermediate steps than existing methods; (ii) can be used to improve the policy model at training time; and (iii) improves inference-time search.
  arXiv  Detail & Related papers  (2025-08-26T17:45:05Z)
• EvoLM: In Search of Lost Language Model Training Dynamics [97.69616550374579]
  EvoLM is a model suite that enables systematic and transparent analysis of LMs' training dynamics across pre-training, continued pre-training, supervised fine-tuning, and reinforcement learning.<n>By training over 100 LMs with 1B and 4B parameters from scratch, we rigorously evaluate both upstream (language modeling) and downstream (problem-solving) reasoning capabilities.
  arXiv  Detail & Related papers  (2025-06-19T04:58:47Z)
• Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instruction [57.19302613163439]
  We introduce neural network reprogrammability as a unifying framework for model adaptation.<n>We present a taxonomy that categorizes such information manipulation approaches across four key dimensions.<n>We also analyze remaining technical challenges and ethical considerations.
  arXiv  Detail & Related papers  (2025-06-05T05:42:27Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• Transferable Post-training via Inverse Value Learning [83.75002867411263]
  We propose modeling changes at the logits level during post-training using a separate neural network (i.e., the value network)<n>After training this network on a small base model using demonstrations, this network can be seamlessly integrated with other pre-trained models during inference.<n>We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes.
  arXiv  Detail & Related papers  (2024-10-28T13:48:43Z)
• Demystifying the Token Dynamics of Deep Selective State Space Models [3.829322478948515]
  Selective state space models (SSM) have gained prominence for their effectiveness in modeling sequential data.<n>Despite their outstanding empirical performance, a comprehensive theoretical understanding of deep selective SSM remains elusive.<n>In this paper, we investigate the dynamical properties of tokens in a pre-trained Mamba model.
  arXiv  Detail & Related papers  (2024-10-04T10:06:17Z)
• Model-Based Transfer Learning for Contextual Reinforcement Learning [5.5597941107270215]
  We introduce Model-Based Transfer Learning to solve contextual RL problems.<n>We show theoretically that the method exhibits sublinear regret in the number of training tasks.<n>We experimentally validate our methods using urban traffic and standard continuous control benchmarks.
  arXiv  Detail & Related papers  (2024-08-08T14:46:01Z)
• Distilled Datamodel with Reverse Gradient Matching [74.75248610868685]
  We introduce an efficient framework for assessing data impact, comprising offline training and online evaluation stages. Our proposed method achieves comparable model behavior evaluation while significantly speeding up the process compared to the direct retraining method.
  arXiv  Detail & Related papers  (2024-04-22T09:16:14Z)
• The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis [27.310894780313618]
  This paper undertakes a comprehensive comparison of model capabilities at various pretraining intermediate checkpoints. We confirm that specific downstream metrics exhibit similar training dynamics across models of different sizes. In addition to our core findings, we've reproduced Amber and OpenLLaMA, releasing their intermediate checkpoints.
  arXiv  Detail & Related papers  (2024-04-01T16:00:01Z)
• Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks [37.278707106871295]
  We study how fine-tuning alters the underlying capabilities learned by a model during pretraining. We show that fine-tuning rarely alters the underlying model capabilities. We also show that fine-tuning can unintentionally remove a model's safety wrapper.
  arXiv  Detail & Related papers  (2023-11-21T18:51:04Z)
• A Framework for Monitoring and Retraining Language Models in Real-World Applications [3.566775910781198]
  continuous model monitoring and model retraining is required in many real-world applications. There are multiple reasons for retraining, including data or concept drift, which may be reflected on the model performance as monitored by an appropriate metric. We examine the impact of various retraining decision points on crucial factors, such as model performance and resource utilization, in the context of Multilabel Classification models.
  arXiv  Detail & Related papers  (2023-11-16T14:32:18Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• Learning to Modulate pre-trained Models in RL [22.812215561012874]
  Fine-tuning a pre-trained model often suffers from catastrophic forgetting. Our study shows that with most fine-tuning approaches, the performance on pre-training tasks deteriorates significantly. We propose a novel method, Learning-to-Modulate (L2M), that avoids the degradation of learned skills by modulating the information flow of the frozen pre-trained model.
  arXiv  Detail & Related papers  (2023-06-26T17:53:05Z)
• Architecture, Dataset and Model-Scale Agnostic Data-free Meta-Learning [117.48444197402858]
  We propose ePisode cUrriculum inveRsion (ECI) during data-free meta training and invErsion calibRation following inner loop (ICFIL) during meta testing.<n>ECI adaptively increases the difficulty level of pseudo episodes according to the real-time feedback of the meta model.<n>We formulate the optimization process of meta training with ECI as an adversarial form in an end-to-end manner.
  arXiv  Detail & Related papers  (2023-03-20T15:10:41Z)
• SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models [58.78043959556283]
  We study the behaviors of models under different fine-tuning methodologies, including full fine-tuning and Low-Rank Adaptation (LoRA) fine-tuning methods. Our analysis uncovers that LoRA fine-tuning consistently exhibits significantly better OOD generalization performance than full fine-tuning across various scenarios.
  arXiv  Detail & Related papers  (2022-10-10T16:07:24Z)
• Training Dynamics for Text Summarization Models [45.62439188988816]
  We analyze the training dynamics for generation models, focusing on news summarization. Across different datasets (CNN/DM, XSum, MediaSum) and summary properties, we study what the model learns at different stages of its fine-tuning process. We find that properties such as copy behavior are learnt earlier in the training process and these observations are robust across domains. On the other hand, factual errors, such as hallucination of unsupported facts, are learnt in the later stages, and this behavior is more varied across domains.
  arXiv  Detail & Related papers  (2021-10-15T21:13:41Z)
• Learning Dynamics Models for Model Predictive Agents [28.063080817465934]
  Model-Based Reinforcement Learning involves learning a textitdynamics model from data, and then using this model to optimise behaviour. This paper sets out to disambiguate the role of different design choices for learning dynamics models, by comparing their performance to planning with a ground-truth model.
  arXiv  Detail & Related papers  (2021-09-29T09:50:25Z)
• Multi-Stage Influence Function [97.19210942277354]
  We develop a multi-stage influence function score to track predictions from a finetuned model all the way back to the pretraining data. We study two different scenarios with the pretrained embeddings fixed or updated in the finetuning tasks.
  arXiv  Detail & Related papers  (2020-07-17T16:03:11Z)
• Goal-Aware Prediction: Learning to Model What Matters [105.43098326577434]
  One of the fundamental challenges in using a learned forward dynamics model is the mismatch between the objective of the learned model and that of the downstream planner or policy. We propose to direct prediction towards task relevant information, enabling the model to be aware of the current task and encouraging it to only model relevant quantities of the state space. We find that our method more effectively models the relevant parts of the scene conditioned on the goal, and as a result outperforms standard task-agnostic dynamics models and model-free reinforcement learning.
  arXiv  Detail & Related papers  (2020-07-14T16:42:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.