A Survey of Methods for Explaining Black Box Models
Riccardo Guidotti,Anna Monreale,Salvatore Ruggieri,Franco Turini,Fosca Giannotti,Dino Pedreschi +5 more
TL;DR: In this paper, the authors provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box decision support systems, given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work.
read more
Abstract: In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Figures
Table 3. Summary of methods for opening and explaining black boxes with respect to the problem faced. 
Fig. 6. Black Box Inspection Problem. 
Table 2. Legend of Table 1. In the following are described the features reported and the abbreviations adopted. ![Fig. 11. Saliency Masks for explanation of deep neural network. (Left) From [108] the elements of the image highlighted. (Right) From [25] the mask and the level of accuracy on the image considering and not considering the learned mask.](/figures/figure11-1-5ao3relmau4d.png)
Fig. 11. Saliency Masks for explanation of deep neural network. (Left) From [108] the elements of the image highlighted. (Right) From [25] the mask and the level of accuracy on the image considering and not considering the learned mask. 
Table 4. Summary of methods for opening and explaining black boxes with respect to the explanator adopted. 
Fig. 9. (Left) Generalizable reverse engineering approach: internal peculiarities of the black box are not exploited to build the comprehensible predictor. (Right) Not Generalizable reverse engineering approach: the comprehensible predictor is the result of a procedure involving internal characteristics of the black box.
![Fig. 11. Saliency Masks for explanation of deep neural network. (Left) From [108] the elements of the image highlighted. (Right) From [25] the mask and the level of accuracy on the image considering and not considering the learned mask.](/figures/figure11-1-5ao3relmau4d.webp)
Citations
Human-Centered Explainable AI (HCXAI): Beyond Opening the Black-Box of AI
Upol Ehsan,Philipp Wintersberger,Q. Vera Liao,Elizabeth Anne Watkins,Carina Manger,Hal Daumé III,Andreas Riener,Mark O. Riedl +7 more
- 27 Apr 2022
TL;DR: This second CHI workshop on Human-centered XAI (HCXAI), which builds on the success of the first installment from CHI 2021 to expand the conversation around XAI, examines how human-centered perspectives in XAI can be operationalized at the conceptual, methodological, and technical levels.
80
Property inference for deep neural networks
Divya Gopinath,Hayes Converse,Corina S. Păsăreanu,Ankur Taly +3 more
- 10 Nov 2019
TL;DR: In this article, the authors propose to extract patterns based on neuron decisions as preconditions that imply certain desirable output properties, e.g., the prediction being a certain class.
80
An artificial intelligence life cycle: From conception to production
TL;DR: The CDAC AI life cycle as discussed by the authors is a comprehensive life cycle for the design, development, and deployment of artificial intelligence (AI) systems and solutions, which addresses the void of a practical and inclusive approach that spans beyond the technical constructs to also focus on the challenges of risk analysis of AI adoption, transferability of prebuilt models, increasing importance of ethics and governance, and the composition, skills, and knowledge of an AI team required for successful completion.
80
OpenXAI: Towards a Transparent Evaluation of Model Explanations
Chirag Agarwal,Eshika Saxena,Satya Sai Krishna,Martin Pawelczyk,Nari Johnson,Isha Puri,Marinka Zitnik,Himabindu Lakkaraju +7 more
- 22 Jun 2022
TL;DR: Overall, OpenXAI provides an automated end-to-end pipeline that not only simplifies and standardizes the evaluation of post hoc explanation methods, but also promotes transparency and reproducibility in benchmarking these methods.
79
It’s Complicated: The Relationship between User Trust, Model Accuracy and Explanations in AI
TL;DR: In this article , the authors examined the practical consequences of adding explanations for user trust and found that the influence of their explanations on trust differs depending on the classifier's accuracy, revealing discrepancies between self-reported and behavioural trust.
79
References
I and i
TL;DR: There is, I think, something ethereal about i —the square root of minus one, which seems an odd beast at that time—an intruder hovering on the edge of reality.
38.1K
•Book
C4.5: Programs for Machine Learning
J. Ross Quinlan
- 15 Oct 1992
TL;DR: A complete guide to the C4.5 system as implemented in C for the UNIX environment, which starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting.
27.2K
Classification and regression trees
TL;DR: This article gives an introduction to the subject of classification and regression trees by reviewing some widely available algorithms and comparing their capabilities, strengths, and weakness in two examples.
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Marco Tulio Ribeiro,Sameer Singh,Carlos Guestrin +2 more
- 13 Aug 2016
TL;DR: In this article, the authors propose LIME, a method to explain models by presenting representative individual predictions and their explanations in a non-redundant way, framing the task as a submodular optimization problem.