Showing papers on "Orthogonal array testing published in 2017"
1 Jul 2017
TL;DR: Experimental results showed that CIT is an effective testing technique for this kind of application and the results also showed the usefulness of model-based mutation testing to assess CIT applications.
Abstract: Combinatorial interaction testing (CIT) stands as one of the efficient testing techniques that have been used in different applications recently. The technique is useful when there is a need to take the interaction of input parameters into consideration for testing a system. The key insight the technique is that not every single parameter may contribute to the failure of the system and there could be interactions among these parameters. Hence, there must be combinations of these input parameters based on the interaction strength. This technique has been used in many applications to assess its effectiveness. In this paper, we are addressing the effectiveness of CIT for a real-world case study using model-based mutation testing experiments. The contribution of the paper is threefold: First we introduce an effective testing application for CIT; Second, we address the effectiveness of increasing the interaction strength beyond the pairwise (i.e., interaction of more than two parameters); Third, model-based mutation testing is used to mutate the input model of the program in contrast to the traditional code-based mutation testing process. Experimental results showed that CIT is an effective testing technique for this kind of application. In addition, the results also showed the usefulness of model-based mutation testing to assess CIT applications. For the subject of this case study, the results also indicate that 3-way test suite (i.e., interaction of three parameters) could detect new faults that can not be detected by pairwise.
22 citations
TL;DR: Fractional factorial design, specifically orthogonal array testing, was employed to screen and optimize multiple extraction parameters in concise but representative experiments to determine organophosphorus and organochlorine pesticide residues in tobacco, via microwave-assisted extraction and gas chromatography coupled with mass spectrometry detection.
Abstract: Sample preparation is often the main bottleneck in analyzing biological samples. Particularly, effective evaluation of sample preparation conditions usually involves multiple factors and tedious and time-consuming experiments. In this study, fractional factorial design, specifically orthogonal array testing, was employed to screen and optimize multiple extraction parameters in concise but representative experiments. An efficient and sensitive method was developed to determine organophosphorus and organochlorine pesticide residues in tobacco, via microwave-assisted extraction and gas chromatography coupled with mass spectrometry detection. With orthogonal array design, screening, and optimization tests were subsequently conducted to determine the range, impact rank, and possible interactions of extraction temperature, time, microwave power, additive salt, and additive water. Orthogonal array testing selectively reduces the size and cost of experiments and meanwhile provides more information compared to the traditional experimental design that optimizes one factor at a time. A good linear range (0.02–2.00 μg/mL), limits of detection (0.001–0.098 μg/mL), and recovery rates (70.4–107.1%) were demonstrated by spiking known concentrations of multiple pesticide standards in tobacco samples. The established method was then successfully applied to the determination of multipesticide residues in raw tobacco leaves and commercial cigarettes.
17 citations
TL;DR: A combinatorial strategy to achieve a full form test and build an automated test model called pairwise testing with constraints (PTC), which uses pairwise coverage and handles the issues of semantic constraints and illegal values.
Abstract: Recently, testing techniques based on dynamic exploration, which try to automatically exercise every possible user interface element, have been extensively used to facilitate fully testing web applications. Most of such testing tools are however not effective in reaching dynamic pages induced by form interactions due to their emphasis on handling client-side scripting. In this paper, we present a combinatorial strategy to achieve a full form test and build an automated test model. We propose an algorithm called pairwise testing with constraints (PTC) to implement the strategy. Our PTC algorithm uses pairwise coverage and handles the issues of semantic constraints and illegal values. We have implemented a prototype tool ComjaxTest and conducted an empirical study on five web applications. Experimental results indicate that our PTC algorithm generates less form test cases while achieving a higher coverage of dynamic pages than the general pairwise testing algorithm. Additionally, our ComjaxTest generates a relatively complete test model and then detects more faults in a reasonable amount of time, as compared with other existing tools based on dynamic exploration.
16 citations
TL;DR: Experimental results indicate that the approach to identify the interactions that exist in the source code, thereby reducing the count of interactions to be tested without significant loss of fault detection capability.
12 citations
TL;DR: Genetic Algorithms can be used with the neural networks and fuzzy systems for different types of testing to improve the performance and provides a means of an automatic test case generator.
Abstract: Background/Objectives: This research paper presents how Genetic algorithm is efficiently used in random test case generation during functional software testing. Methods/Statistical Analysis: Different hybridized Genetic Algorithms are used to generate test data automatically and optimized those test cases to solve many complex problem related to software testing. Findings: Genetic Algorithms are successfully used in software testing with increasing number of test case generation and provides a means of an automatic test case generator. Applications/Improvements: This study gives us a brief idea to implement Genetic Algorithms in software testing for optimum results and also it can be used with the neural networks and fuzzy systems for different types of testing to improve the performance.
12 citations
20 May 2017
TL;DR: This paper proposes a cluster-based prioritization technique to sample similar products with respect to the feature selection and evaluates the approach using feature models of different sizes and shows that it can enhance the effectiveness of product-line testing.
Abstract: A software product-line comprises a set of products that share a common set of features. These features can be reused to customize a product to satisfy specific needs of certain customers or markets. As the number of possible products increases exponentially for new features, testing all products is infeasible. Existing testing approaches reduce their effort by restricting the number of products (sampling) and improve their effectiveness by considering the order of tests (prioritization). In this paper, we propose a cluster-based prioritization technique to sample similar products with respect to the feature selection. We evaluate our approach using feature models of different sizes and show that cluster-based prioritization can enhance the effectiveness of product-line testing.
12 citations
1 Jun 2017
TL;DR: This paper proposes an improved grouping strategy of target paths to balance the load of each calculation resource and can accelerate the convergence of search process and improve the efficiency of search-based software testing.
Abstract: Search-based software testing has achieved great attention recently, but the efficiency is still the bottleneck of it. This paper focuses on improving the efficiency of generating test data for multiple paths. Genetic algorithms are chosen as the heuristic algorithms in search-based software testing in this paper. First, we propose an improved grouping strategy of target paths to balance the load of each calculation resource. This work makes a contribution to the parallel execution in search-based software testing. Then, common constraints of the target paths in the same group are collected to reduce the search space of test data. Symbolic execution technique is used in this phase. Based on the reduced search space, we can accelerate the convergence of search process and improve the efficiency of search-based software testing. Finally, our method is applied to some study cases to compare with other methods.
11 citations
1 Jul 2017
TL;DR: Experimental results suggest that, the fault detecting ability of adaptive random testing is influenced by the distance calculation error and the modeling mistake with the block failure pattern.
Abstract: Adaptive random testing, which is an enhanced random testing technique, has been studied for over 10 years. This paper aims to research the influence of the distance problems on the performance of adaptive random testing, where we focus on its efficiency of fault detecting. Experimental results suggest that, the fault detecting ability of adaptive random testing is influenced by the distance calculation error and the modeling mistake with the block failure pattern. It is interesting that, the former does the negative influence to the performance, and the latter does the positive one.
6 citations
1 Mar 2017
TL;DR: A method to automatically determine the weights of parameter-values by Bayesian inference using previous testing results is proposed, and fault detection effectiveness of the proposed weighting based prioritized combinatorial testing is evaluated.
Abstract: Combinatorial testing (CT) is a widely-used technique to detect system interaction failures. To improve the test effectiveness of CT, prioritized combinatorial testing inputs priority weights of parameter-values, and generates combinatorial test suites based on the weights. This paper proposes a method to automatically determine the weights of parameter-values by Bayesian inference using previous testing results. Using two open source projects, we evaluate the fault detection effectiveness of the proposed weighting based prioritized combinatorial testing.
5 citations
TL;DR: This paper proposed automatic testing of embedded board, which reduces manual errors, increases test coverage and reduces the test time in the production line.
Abstract: Objective: An electronic system developed for specific application with the integration of hardware and software is known as Embedded System. Due to complexity of hardware and software in a single system, it requires specific technique for testing before deployment of the device. This paper proposes the comparative study of testing techniques of embedded system. Methods/Analysis: In this paper, a test methodology of embedded system using PCBA has been proposed, that covers all the testing aspects of the embedded system from electrical board level hardware to the embedded system software. Three tests namely power rail test, interconnect test and the infrastructure tests are used for hardware functionality for stuck at and bridging faults. The functionality test has been used for programming validation using port interface. Findings: This paper proposed automatic testing of embedded board, which reduces manual errors, increases test coverage and reduces the test time in the production line. The proposed method showed that it can be applied to any board in any form with the same test hardware barring the input-output cables. The software in host machine has been used for testing purpose of different boards. This testing technique is useful for embedded systems implemented in control systems.
4 citations
Journal Article•
TL;DR: The comparison for modeling of SUT in pairwise testing is performed, and the enhancement of Classification Tree Method is proposed, and an example based on steps of proposed model method is provided.
Abstract: Software testing is one of the most important activities to produce a high-quality system, which can increase the trust level of users. There are many types of software testing. One of those testing is called exhaustive testing. Exhaustive testing is used to produce a test suite that will be used in other testing types such as unit testing, system testing, integration testing and also acceptance testing. However, exhaustive testing is infeasible and will be time consuming. Therefore, the combinatorial testing is proposed to solve the exhaustive testing problem. There are many techniques of combinatorial testing. The popular one is called pairwise testing. It also is known as Allpairs or 2-way testing. It involves the interaction of 2 parameters. In order to perform the pairwise testing, there are procedures that need to be fulfilled. The first procedure is modeling of System Under Test (SUT). There are many models that can be used to design the test suite for pairwise testing. In this paper, the comparison for modeling of SUT in pairwise testing is performed, and the enhancement of Classification Tree Method is proposed. An example based on steps of proposed model method is also provided.
1 Jul 2017
TL;DR: Experimental results show that GUI tests do not necessarily entirely exercise application logic functionality, at least not as much as system tests directly interacting with application logic code.
Abstract: The practitioner interested in reducing software verification effort may found herself lost in the many alternative definitions of Graphical User Interface (GUI) testing that exist and their relation to the notion of system testing. One result of these many definitions is that one may end up testing the same parts of the Software Under Test (SUT), specifically the application logic, twice. To clarify two important testing activities and avoid duplicate testing effort, this paper empirically evaluates to what extent GUI tests exercise the application logic of the software under test (and not only the GUI code). Experimental results show that GUI tests do not necessarily entirely exercise application logic functionality, at least not as much as system tests directly interacting with application logic code.
TL;DR: In this article, a divide-and-conquer approach based on adaptive random testing strategy is proposed to generate test data for path coverage based testing, which takes as input the constraints of an executable path and computes a tight over-approximation of their associated sub-domain.
Abstract: In this paper, we have developed an approach to generate test data for path coverage based testing. The main challenge of this kind testing lies in its ability to build efficiently such a test suite in order to minimize the number of rejects. We address this problem with a novel divide-and-conquer approach based on adaptive random testing strategy. Our approach takes as input the constraints of an executable path and computes a tight over-approximation of their associated sub-domain by using a dynamic domain partitioning approach. We implemented this approach and got experimental results that show the practical benefits compared to existing approaches. Our method generates less invalid inputs and is capable of obtaining the sub-domain of many complex constraints.
TL;DR: The sequential testing approach enables to control the probability of a type 1 and type 2 error at arbitrary small pre-specified levels and approaches the theoretical minimum of expected number of required permutations as these levels go to zero.
Abstract: The following technical report describes the technical details for the implementation of a sequential testing approach to permutation-based association testing in whole-genome sequencing studies. The sequential testing approach enables to control the probability of a type 1 and type 2 error at arbitrary small pre-specified levels and approaches the theoretical minimum of expected number of required permutations as these levels go to zero.
7 Aug 2017
TL;DR: This presents a new model to support empirical failure probability estimation for a software-intensive system that combines the results of testing using a simulated hardware platform with results from testing on the real platform.
Abstract: This presents a new model to support empirical failure probability estimation for a software-intensive system. The new element of the approach is that it combines the results of testing using a simulated hardware platform with results from testing on the real platform. This approach addresses a serious practical limitation of a technique known as statistical testing. This limitation will be called the test time expansion problem (or simply the 'time problem'), which is that the amount of testing required to demonstrate useful levels of reliability over a time period T is many orders of magnitude greater than T. The time problem arises whether the aim is to demonstrate ultra-high reliability levels for protection system, or to demonstrate any (desirable) reliability levels for continuous operation ('high demand') systems. Specifically, the theoretical feasibility of a platform simulation approach is considered since, if this is not proven, questions of practical implementation are moot. Subject to the assumptions made in the paper, theoretical feasibility is demonstrated.
1 Jul 2017
TL;DR: An automated, multi-objective test case selection technique in black-box systems using genetic algorithms is introduced and results indicate that this approach is applicable based on different data available and is able to outperform random test caseselection and retest-all.
Abstract: Testing is a fundamental task to ensure software quality. Regression testing aims to ensure that changes to software do not introduce new failures. As resources are often limited and testing comprises a vast amount of test cases, different regression strategies have been proposed to reduce testing effort by selecting or prioritizing important test cases, e.g., code coverage (to ensure a sufficient testing depth). However, in system testing, source code is often not available creating a black-box system. In this paper, we introduce an automated, multi-objective test case selection technique in black-box systems using genetic algorithms. We define seven different objectives, based on meta-data, allowing a flexible test case selection for a variety of systems. For evaluation, we apply our technique on two different subject systems assessing the feasibility and suitability of our test case selection approach. Results indicate that our approach is applicable based on different data available and is able to outperform random test case selection and retest-all.
10 Jul 2017
TL;DR: Target property-based testing is introduced, an enhanced form of property- based testing that aims to make the input generation component of a property-Based testing tool guided by a search strategy rather than being completely random.
Abstract: We introduce targeted property-based testing, an enhanced form of property-based testing that aims to make the input generation component of a property-based testing tool guided by a search strategy rather than being completely random. Thus, this testing technique combines the advantages of both search-based and property-based testing. We demonstrate the technique with the framework we have built, called Target, and show its effectiveness on three case studies. The first of them demonstrates how Target can employ simulated annealing to generate sensor network topologies that form configurations with high energy consumption. The second case study shows how the generation of routing trees for a wireless network equipped with directional antennas can be guided to fulfill different energy metrics. The third case study employs Target to test the noninterference property of information-flow control abstract machine designs, and compares it with a sophisticated hand-written generator for programs of these abstract machines.
1 Mar 2017
TL;DR: Stochastic Performance Logic, a formalism for expressing performance requirements, together with interpretations that facilitate performance evaluation in the unit test context are implemented in a performance testing framework and evaluated in multiple experiments, demonstrating the ability to identify performance differences in realistic unit test scenarios.
Abstract: Unit testing is an attractive quality management tool in the software development process, however, practical obstacles make it difficult to use unit tests for performance testing. We present Stochastic Performance Logic, a formalism for expressing performance requirements, together with interpretations that facilitate performance evaluation in the unit test context. The formalism and the interpretations are implemented in a performance testing framework and evaluated in multiple experiments, demonstrating the ability to identify performance differences in realistic unit test scenarios.
25 Jul 2017
TL;DR: In this paper, the authors performed an experimental evaluation of the coverage levels of random testing for two criteria: MC/DC and combinatorial t-way testing, and found that, when the number of random test cases increased, a high level of coverage was reached rapidly, both for MC and T-way, but many more random tests are required to reach 100% coverage.
Abstract: Software testing criteria differ in their effectiveness, the numbers of test cases required, and the processes of test generation. Specific criteria often are compared to random testing, and in some cases, random testing shows a surprisingly high level of effectiveness. One reason that this is the case is that any random test set has a specific level of coverage according to any coverage criterion. Numerical evaluation of coverage levels of random testing according to various coverage criteria is an interesting research task and is important in understanding the relationship between different testing approaches. In this paper, we performed an experimental evaluation of the coverage levels of random testing for two criteria: MC/DC and combinatorial t-way testing. Our experiments showed that, when the number of random test cases increased, a high level of coverage was reached rapidly, both for MC/DC and t-way. However, many more random tests are required to reach 100% coverage. An unexpected result was that there were significant differences in the measurement of partial MC/DC coverage by various tools. The results may be used to select optimal methods for practical testing and develop new testing methods based on the integration of existing approaches.