The Foundation Model Transparency Index

• URL: http://arxiv.org/abs/2310.12941v1
• Date: Thu, 19 Oct 2023 17:39:02 GMT
• Title: The Foundation Model Transparency Index
• Authors: Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, Percy Liang
• Abstract summary: The Foundation Model Transparency Index specifies 100 indicators that codify transparency for foundation models. We score developers in relation to their practices for their flagship foundation model. Overall, the Index establishes the level of transparency today to drive progress on foundation model governance.
• Score: 55.862805799199194
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models have rapidly permeated society, catalyzing a wave of generative AI applications spanning enterprise and consumer-facing contexts. While the societal impact of foundation models is growing, transparency is on the decline, mirroring the opacity that has plagued past digital technologies (e.g. social media). Reversing this trend is essential: transparency is a vital precondition for public accountability, scientific innovation, and effective governance. To assess the transparency of the foundation model ecosystem and help improve transparency over time, we introduce the Foundation Model Transparency Index. The Foundation Model Transparency Index specifies 100 fine-grained indicators that comprehensively codify transparency for foundation models, spanning the upstream resources used to build a foundation model (e.g data, labor, compute), details about the model itself (e.g. size, capabilities, risks), and the downstream use (e.g. distribution channels, usage policies, affected geographies). We score 10 major foundation model developers (e.g. OpenAI, Google, Meta) against the 100 indicators to assess their transparency. To facilitate and standardize assessment, we score developers in relation to their practices for their flagship foundation model (e.g. GPT-4 for OpenAI, PaLM 2 for Google, Llama 2 for Meta). We present 10 top-level findings about the foundation model ecosystem: for example, no developer currently discloses significant information about the downstream impact of its flagship model, such as the number of users, affected market sectors, or how users can seek redress for harm. Overall, the Foundation Model Transparency Index establishes the level of transparency today to drive progress on foundation model governance via industry standards and regulatory intervention.

Related papers

• The Foundation Model Transparency Index v1.1: May 2024 [54.78174872757794]
  The October 2023 Index assessed 10 major foundation model developers on 100 transparency indicators. At the time, developers publicly disclosed very limited information with the average score being 37 out of 100. We find that developers now score 58 out of 100 on average, a 21 point improvement over v1.0.
  arXiv  Detail & Related papers  (2024-07-17T18:03:37Z)
• Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them? [11.040101172803727]
  New capabilities in foundation models are owed in large part to massive, widely-sourced, and under-documented training data collections. Existing practices in data collection have led to challenges in tracing authenticity, verifying consent, preserving privacy, addressing representation and bias, respecting copyright, and overall developing ethical and trustworthy foundation models.
  arXiv  Detail & Related papers  (2024-04-19T07:42:35Z)
• Foundation Model Transparency Reports [61.313836337206894]
  We propose Foundation Model Transparency Reports, drawing upon the transparency reporting practices in social media. We identify 6 design principles given the successes and shortcomings of social media transparency reporting. Well-designed transparency reports could reduce compliance costs, in part due to overlapping regulatory requirements across different jurisdictions.
  arXiv  Detail & Related papers  (2024-02-26T03:09:06Z)
• Window-based Model Averaging Improves Generalization in Heterogeneous Federated Learning [29.140054600391917]
  Federated Learning (FL) aims to learn a global model from distributed users while protecting their privacy. We propose WIMA (Window-based Model Averaging), which aggregates global models from different rounds using a window-based approach. Our experiments demonstrate the robustness of WIMA against distribution shifts and bad client sampling, resulting in smoother and more stable learning trends.
  arXiv  Detail & Related papers  (2023-10-02T17:30:14Z)
• Rethinking People Analytics With Inverse Transparency by Design [57.67333075002697]
  We propose a new design approach for workforce analytics we refer to as inverse transparency by design. We find that architectural changes are made without inhibiting core functionality. We conclude that inverse transparency by design is a promising approach to realize accepted and responsible people analytics.
  arXiv  Detail & Related papers  (2023-05-16T21:37:35Z)
• Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning [86.59588262014456]
  Federated Learning (FL) is an emerging distributed learning paradigm under privacy constraint. We propose a data-free knowledge distillation method to fine-tune the global model in the server (FedFTG) Our FedFTG significantly outperforms the state-of-the-art (SOTA) FL algorithms and can serve as a strong plugin for enhancing FedAvg, FedProx, FedDyn, and SCAFFOLD.
  arXiv  Detail & Related papers  (2022-03-17T11:18:17Z)
• Trustworthy Transparency by Design [57.67333075002697]
  We propose a transparency framework for software design, incorporating research on user trust and experience. Our framework enables developing software that incorporates transparency in its design.
  arXiv  Detail & Related papers  (2021-03-19T12:34:01Z)
• Interpretabilit\'e des mod\`eles : \'etat des lieux des m\'ethodes et application \`a l'assurance [1.6058099298620423]
  Data is the raw material of many models today make it possible to increase the quality and performance of digital services. Models users must ensure that models do not discriminate against and that it is also possible to explain its result. The widening of the panel of predictive algorithms leads scientists to be vigilant about the use of models.
  arXiv  Detail & Related papers  (2020-07-25T12:18:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.