A Survey of Methods for Explaining Black Box Models
Riccardo Guidotti,Anna Monreale,Salvatore Ruggieri,Franco Turini,Fosca Giannotti,Dino Pedreschi +5 more
TL;DR: In this paper, the authors provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box decision support systems, given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work.
read more
Abstract: In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Figures
Table 3. Summary of methods for opening and explaining black boxes with respect to the problem faced. 
Fig. 6. Black Box Inspection Problem. 
Table 2. Legend of Table 1. In the following are described the features reported and the abbreviations adopted. ![Fig. 11. Saliency Masks for explanation of deep neural network. (Left) From [108] the elements of the image highlighted. (Right) From [25] the mask and the level of accuracy on the image considering and not considering the learned mask.](/figures/figure11-1-5ao3relmau4d.png)
Fig. 11. Saliency Masks for explanation of deep neural network. (Left) From [108] the elements of the image highlighted. (Right) From [25] the mask and the level of accuracy on the image considering and not considering the learned mask. 
Table 4. Summary of methods for opening and explaining black boxes with respect to the explanator adopted. 
Fig. 9. (Left) Generalizable reverse engineering approach: internal peculiarities of the black box are not exploited to build the comprehensible predictor. (Right) Not Generalizable reverse engineering approach: the comprehensible predictor is the result of a procedure involving internal characteristics of the black box.
![Fig. 11. Saliency Masks for explanation of deep neural network. (Left) From [108] the elements of the image highlighted. (Right) From [25] the mask and the level of accuracy on the image considering and not considering the learned mask.](/figures/figure11-1-5ao3relmau4d.webp)
Citations
What to account for when accounting for algorithms: a systematic literature review on algorithmic accountability
Maranke Wieringa
- 27 Jan 2020
TL;DR: A definition ofgorithmic accountability based on accountability theory and algorithmic accountability literature is provided, which pays extra attention to accountability risks in algorithmic systems.
233
Explaining anomalies detected by autoencoders using Shapley Additive Explanations
TL;DR: This paper proposes a black-box explanation method, which uses Kernel SHAP to explain anomalies detected by an autoencoder, which is an unsupervised model, and demonstrates that the explanations it provides are more effective at reducing the anomaly score than other methods.
232
Making deep neural networks right for the right scientific reasons by interacting with their explanations
Patrick Schramowski,Wolfgang Stammer,Stefano Teso,Anna Brugger,Franziska Herbert,Xiaoting Shao,Hans-Georg Luigs,Anne-Katrin Mahlein,Kristian Kersting +8 more
TL;DR: The novel learning setting of explanatory interactive learning is introduced and its benefits on a plant phenotyping research task are illustrated and it is demonstrated that explanatory interactiveLearning can help to avoid Clever Hans moments in machine learning.
A Survey of Data-driven and Knowledge-aware eXplainable AI
Xiao-Hui Li,Caleb Chen Cao,Yuhan Shi,Wei Bai,Han Gao,Luyu Qiu,Cong Wang,Yuanyuan Gao,Shenjia Zhang,Xun Xue,Lei Chen +10 more
TL;DR: A survey, reviewing and taxonomizing existing efforts from the view-point of DKE, summarizing their contribution, technical essence and comparative characteristics, and categorizing methods into data-driven methods where explanation comes from the task-related data, and knowledge-aware methods where extraneous knowledge is incorporated.
229
Re-Thinking Data Strategy and Integration for Artificial Intelligence: Concepts, Opportunities, and Challenges
TL;DR: In this article , the authors comprehensively review and critically examine the challenges of using data for AI, including data quality, data volume, privacy and security, bias and fairness, interpretability and explainability, ethical concerns, and technical expertise and skills.
References
I and i
TL;DR: There is, I think, something ethereal about i —the square root of minus one, which seems an odd beast at that time—an intruder hovering on the edge of reality.
38.1K
•Book
C4.5: Programs for Machine Learning
J. Ross Quinlan
- 15 Oct 1992
TL;DR: A complete guide to the C4.5 system as implemented in C for the UNIX environment, which starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting.
27.2K
Classification and regression trees
TL;DR: This article gives an introduction to the subject of classification and regression trees by reviewing some widely available algorithms and comparing their capabilities, strengths, and weakness in two examples.
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Marco Tulio Ribeiro,Sameer Singh,Carlos Guestrin +2 more
- 13 Aug 2016
TL;DR: In this article, the authors propose LIME, a method to explain models by presenting representative individual predictions and their explanations in a non-redundant way, framing the task as a submodular optimization problem.