Identifying implementation bugs in machine learning based image classifiers using metamorphic testing
Anurag Dwarakanath,Manish Ahuja,Samarth Sikand,Raghotham M. Rao,R. P. Jagadeesh Chandra Bose,Neville Dubash,Sanjay Podder +6 more
- 12 Jul 2018
- pp 118-128
TL;DR: In this article, the authors present an articulation of the challenges in testing ML based applications and present a solution approach based on the concept of metamorphic testing, which aims to identify implementation bugs in ML based image classifiers.
read more
Abstract: We have recently witnessed tremendous success of Machine Learning (ML) in practical applications. Computer vision, speech recognition and language translation have all seen a near human level performance. We expect, in the near future, most business applications will have some form of ML. However, testing such applications is extremely challenging and would be very expensive if we follow today's methodologies. In this work, we present an articulation of the challenges in testing ML based applications. We then present our solution approach, based on the concept of Metamorphic Testing, which aims to identify implementation bugs in ML based image classifiers. We have developed metamorphic relations for an application based on Support Vector Machine and a Deep Learning based application. Empirical validation showed that our approach was able to catch 71% of the implementation bugs in the ML applications.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
Machine Learning Testing: Survey, Landscapes and Horizons
TL;DR: A comprehensive survey of machine learning testing can be found in this article, which covers 138 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (i.e., the data, learning program, and framework), testing workflow, and application scenarios.
•Posted Content
Machine Learning Testing: Survey, Landscapes and Horizons
TL;DR: This paper provides a comprehensive survey of techniques for testing machine learning systems; Machine Learning Testing (ML testing) research, covering 144 papers on testing properties, testing components, and application scenarios.
503
•Posted Content
Testing Deep Neural Networks
TL;DR: This paper proposes a family of four novel test criteria that are tailored to structural features of DNNs and their semantics, and validated by demonstrating that the generated test inputs guided via the proposed coverage criteria are able to capture undesired behaviours in a DNN.
Testing machine learning based systems: a systematic mapping
Vincenzo Riccio,Gunel Jahangirova,Andrea Stocco,Nargiz Humbatova,Michael Weiss,Paolo Tonella +5 more
TL;DR: A systematic mapping study about testing techniques for MLSs driven by 33 research questions and investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles.
On testing machine learning programs
Houssem Ben Braiek,Foutse Khomh +1 more
TL;DR: A comprehensive review of software testing practices for ML programs can be found in this paper, where the authors identify and explain challenges that should be addressed when testing ML programs and report existing solutions found in the literature.
209
References
Deep Residual Learning for Image Recognition
Kaiming He,Xiangyu Zhang,Shaoqing Ren,Jian Sun +3 more
- 27 Jun 2016
TL;DR: In this article, the authors proposed a residual learning framework to ease the training of networks that are substantially deeper than those used previously, which won the 1st place on the ILSVRC 2015 classification task.
•Proceedings Article
Very Deep Convolutional Networks for Large-Scale Image Recognition
Karen Simonyan,Andrew Zisserman +1 more
- 04 Sep 2014
TL;DR: This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.
102.6K
ImageNet classification with deep convolutional neural networks
TL;DR: A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
•Proceedings Article
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky,Ilya Sutskever,Geoffrey E. Hinton +2 more
- 03 Dec 2012
TL;DR: The state-of-the-art performance of CNNs was achieved by Deep Convolutional Neural Networks (DCNNs) as discussed by the authors, which consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax.
Microsoft COCO: Common Objects in Context
Tsung-Yi Lin,Michael Maire,Serge Belongie,James Hays,Pietro Perona,Deva Ramanan,Piotr Dollár,C. Lawrence Zitnick +7 more
- 06 Sep 2014
TL;DR: A new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding by gathering images of complex everyday scenes containing common objects in their natural context.