Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing
Anurag Dwarakanath,Manish Ahuja,Samarth Sikand,Raghotham M. Rao,R. P. Jagadeesh Chandra Bose,Neville Dubash,Sanjay Podder +6 more
TL;DR: This work presents the solution approach, based on the concept of Metamorphic Testing, which aims to identify implementation bugs in ML based image classifiers and developed metamorphic relations for an application based on Support Vector Machine and a Deep Learning based application.
read more
Abstract: We have recently witnessed tremendous success of Machine Learning (ML) in practical applications. Computer vision, speech recognition and language translation have all seen a near human level performance. We expect, in the near future, most business applications will have some form of ML. However, testing such applications is extremely challenging and would be very expensive if we follow today's methodologies. In this work, we present an articulation of the challenges in testing ML based applications. We then present our solution approach, based on the concept of Metamorphic Testing, which aims to identify implementation bugs in ML based image classifiers. We have developed metamorphic relations for an application based on Support Vector Machine and a Deep Learning based application. Empirical validation showed that our approach was able to catch 71% of the implementation bugs in the ML applications.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Machine Learning Testing: Survey, Landscapes and Horizons
TL;DR: A comprehensive survey of machine learning testing can be found in this article, which covers 138 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (i.e., the data, learning program, and framework), testing workflow, and application scenarios.
•Posted Content
Testing Deep Neural Networks
TL;DR: This paper proposes a family of four novel test criteria that are tailored to structural features of DNNs and their semantics, and validated by demonstrating that the generated test inputs guided via the proposed coverage criteria are able to capture undesired behaviours in a DNN.
Testing machine learning based systems: a systematic mapping
Vincenzo Riccio,Gunel Jahangirova,Andrea Stocco,Nargiz Humbatova,Michael Weiss,Paolo Tonella +5 more
TL;DR: A systematic mapping study about testing techniques for MLSs driven by 33 research questions and investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles.
Identifying implementation bugs in machine learning based image classifiers using metamorphic testing
Anurag Dwarakanath,Manish Ahuja,Samarth Sikand,Raghotham M. Rao,R. P. Jagadeesh Chandra Bose,Neville Dubash,Sanjay Podder +6 more
- 12 Jul 2018
TL;DR: In this article, the authors present an articulation of the challenges in testing ML based applications and present a solution approach based on the concept of metamorphic testing, which aims to identify implementation bugs in ML based image classifiers.
201
Deep learning library testing via effective model generation
Zan Wang,Ming Yan,Junjie Chen,Shuang Liu,Dongdi Zhang +4 more
- 08 Nov 2020
TL;DR: This work designs a series of mutation rules for DL models, with the purpose of exploring different invoking sequences of library code and hard-to-trigger behaviors, and proposes a heuristic strategy to guide the model generation process towards the direction of amplifying the inconsistent degrees of the inconsistencies between different DL libraries caused by bugs.
154
References
Deep Residual Learning for Image Recognition
Kaiming He,Xiangyu Zhang,Shaoqing Ren,Jian Sun +3 more
- 27 Jun 2016
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
•Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
- 04 Sep 2014
TL;DR: This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
102.6K
•Proceedings Article
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky,Ilya Sutskever,Geoffrey E. Hinton +2 more
- 03 Dec 2012
TL;DR: The state-of-the-art performance of CNNs was achieved by Deep Convolutional Neural Networks (DCNNs) as discussed by the authors, which consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.
•Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
- 01 Jan 2015
TL;DR: In this paper, the authors investigated the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting and showed that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 layers.
51.9K
Microsoft COCO: Common Objects in Context
Tsung-Yi Lin,Michael Maire,Serge Belongie,James Hays,Pietro Perona,Deva Ramanan,Piotr Dollár,C. Lawrence Zitnick +7 more
- 06 Sep 2014
TL;DR: A new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding by gathering images of complex everyday scenes containing common objects in their natural context.