Software testing with performance profiles clustering using unsupervised learning

TL;DR: Tricorder as mentioned in this paper is a non-functional testing methodology for automated workload characterization and detecting potential performance anomalies, caused by code-faults in autonomous data processing systems, which can be used as a complementary testing technique.

Abstract: Abstract The functional testing technique has been widely applied to reveal unknown faults in the software caused by programmer mistakes. Nevertheless, in autonomous data processing systems with highly variable inputs and outputs, such as embedded applications, data streams, and machine learning algorithms, the non-functional testing helps to reveal unknown-nontrivial faults in the deployment of software products. This paper addresses the detection of unknown code-faults by employing performance analysis as a nonfunctional requirement without predefined test cases, test oracles, or source-code analyses. The premise is that codefaults change demands for hardware resources during software execution, and a novel testing methodology can automatically detect them. This paper proposes the Tricorder testing methodology for automating workload characterization and detecting potential performance anomalies, caused by code-faults in autonomous data processing systems. Tricorder evaluates the performance profiles of hardware regarding the detection of the source code-faults. DAMICORE, a non-parametric multipurpose clustering methodology, enables Tricorder to group performance profiles of the software under testing and identify performance anomalies using non-parametric data in unsupervised learning based on Normalized Compression Distance (NCD). Tricorder reveals unknown source code faults, with no specialist to determine standards for input and output data, previous models inherent to architecture, test case creation, or workload characterization. We evaluate the capability of Tricorder in revealing faults through experiments based on three benchmarks: cryptography system, machine learning algorithm, and data stream processing server. Tricorder detects faults even under various workloads for different applications in our experiments. Tricorder helps the maintenance phase of the software development life cycle, providing additional information regarding the proper functioning of the application release before and after the updating process. This work contributes to the cost reduction of regression testing during the maintenance phase of autonomous data processing applications and can be used as a complementary testing technique.