FactSheets: Increasing trust in AI services through supplier's declarations of conformity
Matthew Arnold,Rachel K. E. Bellamy,Michael Hind,Stephanie Houde,Sameep Mehta,Aleksandra Mojsilovic,Ravi Nair,K. Natesan Ramamurthy,Alexandra Olteanu,David Piorkowski,Darrell C. Reimer,John T. Richards,Jason Tsay,Kush R. Varshney +13 more
443
TL;DR: This paper envisiones an SDoC for AI services to contain purpose, performance, safety, security, and provenance information to be completed and voluntarily released by AI service providers for examination by consumers.
read more
Abstract: Accuracy is an important concern for suppliers of artificial intelligence (AI) services, but considerations beyond accuracy, such as safety (which includes fairness and explainability), security, and provenance, are also critical elements to engender consumers’ trust in a service. Many industries use transparent, standardized, but often not legally required documents called supplier's declarations of conformity (SDoCs) to describe the lineage of a product along with the safety and performance testing it has undergone. SDoCs may be considered multidimensional fact sheets that capture and quantify various aspects of the product and its development to make it worthy of consumers’ trust. In this article, inspired by this practice, we propose FactSheets to help increase trust in AI services. We envision such documents to contain purpose, performance, safety, security, and provenance information to be completed by AI service providers for examination by consumers. We suggest a comprehensive set of declaration items tailored to AI in the Appendix of this article.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Experiences with Improving the Transparency of AI Models and Services
Michael Hind,Stephanie Houde,Jacquelyn A. Martino,Aleksandra Mojsilovic,David Piorkowski,John T. Richards,Kush R. Varshney +6 more
- 25 Apr 2020
TL;DR: In this article, the authors make recommendations for easing the collection and flexible presentation of AI facts to promote transparency, based on the observations from this work, supplemented by feedback received during multiple design explorations and stakeholder conversations.
30
Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards
Angelina McMillan-Major,Salomey Osei,Juan Diego Rodriguez,Pawan Sasanka Ammanamanchi,Sebastian Gehrmann,Yacine Jernite +5 more
TL;DR: In this paper, the authors present two case studies of efforts that aim to develop reusable documentation templates for datasets and models, including the HuggingFace data card, a general purpose card for datasets in NLP, and the GEM benchmark data and model cards with a focus on natural language generation.
Behavioral Use Licensing for Responsible AI
20 Jun 2022
TL;DR: In this paper , the authors advocate the use of licensing to enable legally enforceable behavioral use conditions on software and code and provide several case studies that demonstrate the feasibility of behavioral use licensing.
29
Fairness Score and process standardization: framework for fairness certification in artificial intelligence systems
TL;DR: In this paper , the authors proposed a novel Fairness Score to measure the fairness of a data-driven AI system and a Standard Operating Procedure (SOP) for issuing Fairness Certification for such systems.
29
•Journal Article
Towards traceability in data ecosystems using a Bill of Materials model
TL;DR: This paper introduces the conceptual design of a model for data traceability based on a Bill of Materials scheme, widely used for supply chain traceability in manufacturing industries, and presents details of the architecture and implementation of a gateway built upon the model.
References
The Market for “Lemons”: Quality Uncertainty and the Market Mechanism
TL;DR: In this paper, the authors present a struggling attempt to give structure to the statement: "Business in under-developed countries is difficult"; in particular, a structure is given for determining the economic costs of dishonesty.
•Book
A Theory of Human Motivation
Abraham H. Maslow
- 01 Jan 2013
Abstract: 1. The integrated wholeness of the organism must be one of the foundation stones of motivation theory. 2. The hunger drive (or any other physiological drive) was rejected as a centering point or model for a definitive theory of motivation. Any drive that is somatically based and localizable was shown to be atypical rather than typical in human motivation. 3. Such a theory should stress and center itself upon ultimate or basic goals rather than partial or superficial ones, upon ends rather than means to these ends. Such a stress would imply a more central place for unconscious than for conscious motivations. 4. There are usually available various cultural paths to the same goal. Therefore conscious, specific, local-cultural desires are not as fundamental in motivation theory as the more basic, unconscious goals. 5. Any motivated behavior, either preparatory or consummatory, must be understood to be a channel through which many basic needs may be simultaneously expressed or satisfied. Typically an act has more than one motivation. 6. Practically all organismic states are to be understood as motivated and as motivating. 7. Human needs arrange themselves in hierarchies of prepotency. That is to say, the appearance of one need usually rests on the prior satisfaction of another, more pre-potent need. Man is a perpetually wanting animal. Also no need or drive can be treated as if it were isolated or discrete; every drive is related to the state of satisfaction or dissatisfaction of other drives. 8. Lists of drives will get us nowhere for various theoretical and practical reasons. Furthermore any classification of motivations
19.7K
Another look at measures of forecast accuracy
Rob J. Hyndman,Anne B. Koehler +1 more
TL;DR: In this paper, the mean absolute scaled error (MESEME) was proposed as the standard measure for comparing forecast accuracy across multiple time series across different time series types, and was used in the M-competition as well as the M3competition.
5K
The Limitations of Deep Learning in Adversarial Settings
Nicolas Papernot,Patrick McDaniel,Somesh Jha,Matt Fredrikson,Z. Berkay Celik,Ananthram Swami +5 more
- 21 Mar 2016
TL;DR: This work formalizes the space of adversaries against deep neural networks (DNNs) and introduces a novel class of algorithms to craft adversarial samples based on a precise understanding of the mapping between inputs and outputs of DNNs.
5K
•Posted Content
Towards A Rigorous Science of Interpretable Machine Learning
Finale Doshi-Velez,Been Kim +1 more
TL;DR: This position paper defines interpretability and describes when interpretability is needed (and when it is not), and suggests a taxonomy for rigorous evaluation and exposes open questions towards a more rigorous science of interpretable machine learning.
4K