Abstract: A software product line comprises a family of software products that share a common set of features. It enables customers to compose software systems from a managed set of features. Testing every product of a product line individually is often infeasible due to the exponential number of possible products in the number of features. Several approaches have been proposed to restrict the number of products to be tested by sampling a subset of products achieving sufficient combinatorial interaction coverage. However, existing sampling algorithms do not scale well to large product lines, as they require a considerable amount of time to generate the samples. Moreover, samples are not available until a sampling algorithm completely terminates. As testing time is usually limited, we propose an incremental approach of product sampling for pairwise interaction testing (called IncLing), which enables developers to generate samples on demand in a step-wise manner. Furthermore, IncLing uses heuristics to efficiently achieve pairwise interaction coverage with a reasonable number of products. We evaluated IncLing by comparing it against existing sampling algorithms using feature models of different sizes. The results of our approach indicate efficiency improvements for product-line testing.

Abstract: Random testing can be a powerful and scalable method for finding faults in software. However, sophisticated random testers usually test a whole program, not individual components. Writing random testers for individual components of complex programs may require unreasonable effort. In this paper we present a novel method, directed swarm testing, that uses statistics and a variation of random testing to produce random tests that focus on only part of a program, increasing the frequency with which tests cover the targeted code. We demonstrate the effectiveness of this technique using real-world programs and test systems (the YAFFS2 file system, GCC, and Mozilla's SpiderMonkey JavaScript engine), and discuss various strategies for directed swarm testing. The best strategies can improve coverage frequency for targeted code by a factor ranging from 1.1-4.5x on average, and from nearly 3x to nearly 9x in the best case. For YAFFS2, directed swarm testing never decreased coverage, and for GCC and SpiderMonkey coverage increased for over 99% and 73% of targets, respectively, using the best strategies. Directed swarm testing improves detection rates for real SpiderMonkey faults, when the code in the introducing commit is targeted. This lightweight technique is applicable to existing industrial-strength random testers.

Abstract: Software testing is a process for insuring the delivery of an error free software to customers. The success or failure of the software to achieve its desirability is depend on the software testing components. All the software industries spend a lot of their money and time for software testing process. Automatic testing is a key area where cost and time of software testing process has been considerably reduced as compared to manual testing. Path testing is a process of finding all du-path coverage for a given program. In this paper genetic algorithm is used for automatic test cases generation and optimization. The proposed method with genetic algorithm accepts a set of inputs and check of for path coverage.

Abstract: In this work, we give a novel general approach for distribution testing. We describe two techniques: our first technique gives sample-optimal testers, while our second technique gives matching sample lower bounds. As a consequence, we resolve the sample complexity of a wide variety of testing problems. Our upper bounds are obtained via a modular reduction-based approach. Our approach yields optimal testers for numerous problems by using a standard $\ell_2$-identity tester as a black-box. Using this recipe, we obtain simple estimators for a wide range of problems, encompassing most problems previously studied in the TCS literature, namely: (1) identity testing to a fixed distribution, (2) closeness testing between two unknown distributions (with equal/unequal sample sizes), (3) independence testing (in any number of dimensions), (4) closeness testing for collections of distributions, and (5) testing histograms. For all of these problems, our testers are sample-optimal, up to constant factors. With the exception of (1), ours are the {\em first sample-optimal testers for the corresponding problems.} Moreover, our estimators are significantly simpler to state and analyze compared to previous results. As an application of our reduction-based technique, we obtain the first {\em nearly instance-optimal} algorithm for testing equivalence between two {\em unknown} distributions. Moreover, our technique naturally generalizes to other metrics beyond the $\ell_1$-distance. Our lower bounds are obtained via a direct information-theoretic approach: Given a candidate hard instance, our proof proceeds by bounding the mutual information between appropriate random variables. While this is a classical method in information theory, prior to our work, it had not been used in distribution property testing.

Abstract: A goal of performance testing is to find situations when applications unexpectedly exhibit worsened characteristics for certain combinations of input values A fundamental question of performance testing is how to select a manageable subset of the input data faster to find performance problems in applications automatically We present a novel tool, FOREPOST, for finding performance problems in applications automatically using black-box software testing In this paper, we demonstrate how FOREPOST extracts rules from execution traces of applications by using machine learning algorithms, and then uses these rules to select test input data automatically to steer applications towards computationally intensive paths and to find performance problems FOREPOST is available in our online appendix (http://wwwcswmedu/semeru/data/ICSE16-FOREPOST), which contains the tool, source code and demo video

Abstract: Evaluation of industrial embedded control system designs is a time-consuming and imperfect process. While an ideal process would apply a formal verification technique such as model checking or theorem proving, these techniques do not scale to industrial design problems, and it is often difficult to use these techniques to verify performance aspects of control system designs, such as stability or convergence. For industrial designs, engineers rely on testing processes to identify critical or unexpected behaviors. We propose a novel framework called Underminer to improve the testing process; this is an automated technique to identify non-converging behaviors in embedded control system designs. Underminer treats the system as a black box, and lets the designer indicate the model parameters, inputs and outputs that are of interest. It supports a multiplicity of convergence-like notions, such as those based on Lyapunov analysis and those based on temporal logic formulae. Underminer can be applied in the context of testing models created in the controller-design phase, and can also be applied in a scenario such as hardware-in-the-loop testing. We demonstrate the efficacy of Underminer by evaluating its performance on several examples.

Abstract: Parameterized unit testing, recent advances in unit testing, is a new methodology extending the previous industry practice based on traditional unit tests without parameters. A parameterized unit test (PUT) is simply a test method that takes parameters, calls the code under test, and states assertions. Parameterized unit testing allows the separation of two testing concerns or tasks: the specification of external, black-box behavior (i.e., assertions or specifications) by developers and the generation and selection of internal, white-box test inputs (i.e., high-code-covering test inputs) by tools. PUTs have been supported by various testing frameworks. Various open source and industrial testing tools also exist to generate test inputs for PUTs. This technical briefing presents latest research on principles and techniques, as well as practical considerations to apply parameterized unit testing on real-world programs, highlighting success stories, research and education achievements, and future research directions in developer testing.

Abstract: Combinatorial testing (CT) technique could significantly reduce testing cost and increase software system quality By using the test suite generated by CT as input to conduct black-box testing towards a system, we are able to detect interactions that trigger the system’s faults Given a test case, there may be only part of all its parameters relevant to the defects in system and the interaction constructed by those partial parameters is key factor of triggering fault If we can locate those parameters accurately, this will facilitate the software diagnosing and testing process This paper proposes a novel algorithm named complete Fault Interaction Location (comFIL) to locate those interactions that cause system’s failures and meanwhile obtains the minimal set of target interactions in test suite produced by CT By applying this method, testers can analyze and locate the factors relevant to defects of system more precisely, thus making the process of software testing and debugging easier and more efficient The results of our empirical study indicate that comFIL performs better compared with known fault location techniques in combinatorial testing because of its improved effectiveness and precision

Abstract: Pairwise testing is an effective combinatorial test case generation approach in which test cases are developed to execute all possible pairwise combinations of system inputs. It can help reduce the number of test cases and save testing time yet still effective in finding defects. However, it is very difficult for practitioners to effectively apply pairwise testing in the real world because of the lack of suitable techniques and guidelines. To redress this situation, this paper conducts a case study of applying pairwise testing to system data derived from real-valued variable inputs. In order to apply pairwise testing to this case study, this paper develops a test procedure and a novel partitioning method to test derived data as a naive application of the conventional pairwise testing that would produce a huge number of test cases. A comparative evaluation shows that the pairwise testing of the proposed approach is more effective than the random testing with a 12-20% higher fault detection ratio. Based on our experience, guidelines for applying pairwise testing in practice are also presented. Copyright © 2014 John Wiley & Sons, Ltd.

Abstract: As part of the process to test a new release of an application, the performance testing team need to confirm that the existing functionalities do not perform worse than those in the previous release, a problem known as performance regression anomaly. Most existing approaches to analyse performance regression testing data vary according to the applied workload, which usually leads to the need for an extra performance testing run. To ease such lengthy tasks, we propose a new workload-independent, automated technique to detect anomalies in performance regression testing data using the concept known as transaction profile TP. The TP is inferred from the performance regression testing data along with the queueing network model of the testing system. Based on a case study conducted against two web applications, one open source and one industrial, we have been able to automatically generate the 'TP run report' and verify that it can be used to uncover performance regression anomalies caused by software updates. In particular, the report helped us to isolate the real anomaly issues from those caused by workload changes with an average F1 measure of 85% for the open source application and 90% for the industrial application. Such results support our proposal to use the TP as a more efficient technique in identifying performance regression anomalies than the state of the art industry and research techniques. Copyright © 2015 John Wiley & Sons, Ltd.

Abstract: Software Product Line (SPL) is a software engineering paradigm that is inspired by the concept of reusability of common features, formulated for different software product. Complete testing on entire SPL is known to be unfeasible, due to the very large number of possible products to be produced, configured using a subset or all possible features in the SPL. This paper reports a systematic mapping study (SMS) of relevant primary studies as the evidence on the application of Combinatorial Interaction Testing (CIT) for SPL. In CIT, one has to construct a covering array, which is a set of configurations having valid feature combinations and every combination of t features appears at least once in the array. This is also known as t-wise testing. By following the systematic mapping study guidelines, we have selected and filtered 44 primary studies for review. The most prominent CIT techniques in aiding SPL testing are those based on greedy algorithms followed by meta-heuristics algorithms. The motivation of SPL testing is to anticipate the feature interaction problem, in which the majority of the works were reported to leverage test configuration selection approach, while some employed test configuration prioritization approach. Numerous works have been reported, but only few works managed to demonstrate their scalability, as most primary studies only deal with low strength (t is less than 4) of t-wise testing.

Abstract: In a system with large number of input parameters, it is necessary to check for errors that can occur as a result of interactions between various input parameters. However, checking of all possible combinations of input parameters is often restricted due to time and budget constraints. In order to overcome the constraints of exhaustive testing, combinatorial testing has been employed to generate optimal and efficient test set that covers all t-way combinations of input parameters. Pair-wise testing, a combinatorial testing technique, tests all possible combinations of each pair of input parameter values. In this paper, we present an efficient algorithm pair-wise test set generator using genetic algorithm (PTSG-GA) for generating test set for pair-wise testing. PTSG-GA is an extension of our previous work that applies genetic algorithm (GA) to generate optimal test set for pair-wise testing. In this paper, combinatorial objects, namely covering array (CA) and mixed covering arrays (MCA), are used to represent test set. The major contribution of algorithm PTSG-GA is that it uses a probabilistic approach to generate initial population of CAs/MCAs to improve the performance of GA. The algorithm PTSG-GA is implemented using an open-source tool PWiseGen. We have reported experimental results that illustrate the effectiveness of PTSG-GA as compared to existing state-of-the-art algorithms.

Abstract: Modified Condition / Decision Coverage (MC/DC) is the second strongest coverage criterion in white-box testing. According to DO178C/RTCA criterion it is mandatory to achieve Level A certification for MC/DC. Concolic testing is the combination of Concrete and Symbolic execution. It is a systematic technique that performs symbolic execution but uses randomly-generated test inputs to initialize the search and to allow the tool to execute programs when symbolic execution fails. In this paper, we extend concolic testing by computing MC/DC using the automatically generated test cases. On the other hand Feedback-Directed Random Test Generation builds inputs incrementally by randomly selecting a method call to apply and find arguments from among previously-constructed inputs. As soon as the input is built, it is executed and checked against a set of contracts and filters. In our proposed work, we combine feedback-directed test cases generation with concolic testing to form Java-Hybrid Concolic Testing (Java-HCT). Java-HCT generates more number of test cases since it combines the features of both Feedback- Directed Random Test and Concolic Testing. Hence, through Java-HCT, we achieve high MC/DC. Combinations of approaches represent different tradeoffs of completeness and scalability. We develop Java-HCT using RANDOOP, jCUTE, and COPECA. Combination of RANDOOP and jCUTE creates more test cases. COPECA is used to measure MC/DC% using the generated test cases. Experimental study shows that Java-HCT produces better MC/DC% than individual testing techniques(feedback-directed random testing and concolic testing). We have improved MC/DC by ×1.62 and by ×1.26 for feedback-directed random testing and concolic testing respectively.

Abstract: Reachability testing is an important approach to testing concurrent programs. It generates and exercises synchronization sequences automatically and on-the-fly without saving any test history. Existing reachability testing can be classified into exhaustive and t-way testing. Exhaustive testing is impractical in many cases while t-way testing may decrease the capability of fault detection in some cases. In this paper, we present a variable strength reachability testing strategy, which adopts the dynamic framework of reachability testing and uses a variable strength combinatorial strategy. Different parameter groups are provided with different covering strength. Variable strength testing covers no t-way combinations but the necessary combinations of parameters having mutual interactions in a concurrent program. It is more reasonable than t-way testing because uniform interactions between parameters do not often exist in concurrent systems. We propose a merging algorithmthat implements the variable strength combinatorial testing strategy and conduct our experiment on several concurrent programs. The experimental results indicate that our variable strength reachability testing reaches a good tradeoff between the effectiveness and efficiency. It can keep the same capability of fault detection as exhaustive reachability testing while substantially reducing the number of synchronization sequences and decreasing the execution time in most cases.

Abstract: Testing is an indispensable part of the software development life cycle. It is performed to improve the performance, quality and reliability of the software. Various types of testing such as functional testing and structural testing are performed on software to uncover the faults caused by an incorrect code, interaction of input parameters, etc. One of the major factors in deciding the quality of testing is the design of relevant test cases which is crucial for the success of testing. In this paper we concentrate on generating test cases to uncover faults caused by the interaction of input parameters. It is advisable to perform thorough testing but the number of test cases grows exponentially with the increase in the number of input parameters, which makes exhaustive testing of interaction of input parameters imprudent. An alternative to exhaustive testing is combinatorial interaction testing (CIT) which requires that every t-way interaction of input parameters be covered by at least one test case. Here, we present a novel strategy ABC-CAG (Artificial Bee Colony-Covering Array Generator) based on the Artificial Bee Colony (ABC) algorithm to generate covering an array and a mixed covering array for pair-wise testing. The proposed ABC-CAG strategy is implemented in a tool and experiments are conducted on various benchmark problems to evaluate the ecacy of the proposed approach. Experimental results show that ABC-CAG generates better/comparable results as compared to the existing state-of-the-art algorithms.

Abstract: Under the concept of using software testing, the problems are preparing the smallest size of the selected test cases, fixing bugs, predicting the estimated testing time and numbers of the tester. The traditional methods are developed to solve these issues. Unfortunately, they cannot be applied for all reasons. Therefore, the filtering test case selection model is proposed to solve these problems and to increase the efficiency of the regression testing. It prepares the methods of filtering, classifying, and selecting the appropriate test cases. Accordingly, it gives the smaller size of the selected test cases than the traditional selections as 0.23–3.96%. When consider the fault rate measured by the developed technique is also less than those methods equivalent to 13–69%. Furthermore, the testing time and the amount of testers are also less than the comparative studies as about 2–71%, and 13–69% respectively.

Abstract: This paper presents Mathematical Execution (ME), a new, unified approach for testing numerical code. The key idea is to (1) capture the desired testing objective via a representing function and (2) transform the automated testing problem to the minimization problem of the representing function. The minimization problem is to be solved via mathematical optimization. The main feature of ME is that it directs input space exploration by only executing the representing function, thus avoiding static or symbolic reasoning about the program semantics, which is particularly challenging for numerical code. To illustrate this feature, we develop an ME-based algorithm for coverage-based testing of numerical code. We also show the potential of applying and adapting ME to other related problems, including path reachability testing, boundary value analysis, and satisfiability checking. To demonstrate ME's practical benefits, we have implemented CoverMe, a proof-of-concept realization for branch coverage based testing, and evaluated it on Sun's C math library (used in, for example, Android, Matlab, Java and JavaScript). We have compared CoverMe with random testing and Austin, a publicly available branch coverage based testing tool that supports numerical code (Austin combines symbolic execution and search-based heuristics). Our experimental results show that CoverMe achieves near-optimal and substantially higher coverage ratios than random testing on all tested programs, across all evaluated coverage metrics. Compared with Austin, CoverMe improves branch coverage from 43% to 91%, with significantly less time (6.9 vs. 6058.4 seconds on average).

Abstract: In order to verify whether the functions have been rightly implemented into the software, the developer is involved with the task of developing a set of test cases thereby influencing the set of functions and modules present in the software so that the developer can judge upon the defects laid into the implemented functions. The set of functions which these test cases influence comprises what in common terms is known as testing domain and that the rate of increase of the testing domain is vulnerable to the progress in testing process. The growth rate of the testing domain goes hand in hand with the fault content of the software, that is to say, as the testing domain spreads more number of existing faults are detected and subjected to removal, thereby causing the software fault content to decrease. Further the software developer is endowed with the task of determining the appropriate releasing time of the software into the market such that the cost is minimized and the reliability is maximized in the bargain. In order to counter effect the growing software fault content, this paper includes a testing domain dependent software reliability growth model (SRGM) incorporating the ideas of testing domain coverage efficiency as also the fault removal efficiency. The model so framed is based on Non-Homogenous Poisson Process (NHPP) and its validity has been verified by testing on fault data observed in earlier actual software development process. Along with illustration to the optimal time of software release the model has been subjected to goodness of fit comparison justifying that the developed model outperforms some of the existing ones in its capability of measuring the reliability.

Abstract: Combinatorial methods are useful and effective approaches for software testing. In this paper, we provide a brief review of the research results in one area of combinatorial methods applications: software with binary or Boolean inputs. This includes the testing of logical expressions that is crucial for many safety-critical applications such as avionics software and computer systems at nuclear power plants. We consider three different aspects of this research direction: (1) effectiveness of combinatorial testing with binary inputs, (2) combinatorial test generation, and (3) comparison combinatorial testing with other approaches for testing logical expressions.

Abstract: Pairwise testing is a wildly used approach in order to reduce the size of test suite and take steps to combinatorial testing problems because of an extensively large number of possible combinations between input parameters and values. In some cases, there will be invalid combinations between input parameters and values if constraints have not been handled. In this paper, we present a pairwise test generation tool called CPTG, a tool to generate test cases for pairwise testing by applying user profile for guiding and prioritizing in order to select optimal input parameters and values which do not depend on individual tester skills and also providing constraint handling solution between input parameters and values. We performed experiments and comparison with other tools. The experimental results of our tool demonstrated that our tool becomes particularly valuable in guiding testing with a maximized reliability by testing the most frequently used of the system and can generate comparable results of the size of the test case set.

Abstract: Combinatorial testing (CT) is a proven method for more effective software testing at lower cost. The key insight underlying combinatorial testing’s effectiveness resulted from a series of studies by NIST from 1999 to 2004. NIST research showed that most software bugs and failures are caused by one or two parameters, with progressively fewer by three or more. This finding, referred to as the interaction rule, has important implications for software testing because it means that testing parameter combinations can provide more efficient fault detection than conventional methods. New algorithms compressing combinations into a small number of tests have made CT practical for industrial use, making it possible to do better testing at lower cost. Background Software developers often notice an interesting – though not surprising – phenomenon: when the number of system users increases significantly, components that had previously operated correctly will suddenly fail or display errors. For example, if there are many new transactions or accounts, some are likely to contain combinations of values that have not been seen before. Some of these rare combinations trigger faults that escaped previous testing and extensive use. Combinatorial testing can help detect problems early in the testing life cycle. [1] Because system failures often result from the interaction of conditions that might be innocuous individually, testing combinations of parameter values can be nearly as effective as testing all possible combinations for many types of applications. Implementing this form of testing requires a covering array, a matrix that includes all t-way combinations of values for some specified interaction level, t. As noted, most failures are caused by one or two parameters; empirical data show that the number of failures triggered by three or more parameters is small, and no failures discovered by NIST or other researchers have involved more than six parameters. As a result of this phenomenon, covering arrays that include all 3-way to 6-way combinations can provide strong testing.

Abstract: Interface testing for opto-electric tracking system is one important work to assure system running performance, aiming to verify the design result of every electronic interface matching the communication protocols or not, by different levels. Opto-electric tracking system nowadays is more complicated, composed of many functional units. Usually, interface testing is executed between units manufactured completely, highly depending on unit design and manufacture progress as well as relative people. As a result, it always takes days or weeks, inefficiently. To solve the problem, this paper promotes an efficient and simple interface testing equipment for opto-electric tracking system, consisting of optional interface circuit card, processor and test program. The hardware cards provide matched hardware interface(s), easily offered from hardware engineer. Automatic code generation technique is imported, providing adaption to new communication protocols. Automatic acquiring items, automatic constructing code architecture and automatic encoding are used to form a new program quickly with adaption. After simple steps, a standard customized new interface testing equipment with matching test program and interface(s) is ready for a waiting-test system in minutes. The efficient and simple interface testing equipment for opto-electric tracking system has worked for many opto-electric tracking system to test entire or part interfaces, reducing test time from days to hours, greatly improving test efficiency, with high software quality and stability, without manual coding. Used as a common tool, the efficient and simple interface testing equipment for opto-electric tracking system promoted by this paper has changed traditional interface testing method and created much higher efficiency.

Abstract: Software testing is an embedded activity in all software development life cycle phases. Due to the difficulties and high costs of software testing, many testing techniques have been developed with the common goal of testing software in the most optimal and cost-effective manner. Model-based testing (MBT) is used to direct testing activities such as test verification and selection. MBT is employed to encapsulate and understand the behavior of the system under test, which supports and helps software engineers to validate the system with various likely actions. The widespread usage of models has influenced the usage of MBT in the testing process, especially with UML. In this research, we proposed an improved model based testing strategy, which involves and uses four different diagrams in the testing process. This paper also discusses and explains the activities in the proposed model with the finite state model (FSM). The comparisons have been done with traditional model based testings in terms of test case generation and result.

Abstract: Testing the product just before its release to the customer is very important as it marks the success of that product. In software development, testing of the software is conducted at the last stage of the development cycle. During software testing, there will a large number of variables and their possible values. For this if conventional testing process becomes cumbersome and hence a more efficient testing strategy has to be applied. This paper discusses the application of Orthogonal Array based Software Testing (OATS) for minimizing the number of bugs in the software. In this paper, Taguchi’s Orthogonal Array (OA) is applied for compatibility testing of a website, by considering five control factors and one interaction between the factors. Experiments have been conducted using L18 orthogonal array; experimental results are analyzed using Smaller-The-Better (STB) type of Signal-toNoise (S/N) ratio and optimal levels for the control factors are obtained. With these optimal levels it is found that the number of bugs has been considerably reduced.

Abstract: In software development of configurable systems, effective software testing is essential. As high configurability entails high testing effort, effective testing is required to meet cost constraints while ensuring satisfactory end product quality. In this paper, we present an initial study on the potential of using statistical testing techniques for improving the efficiency of test selection for configurable software. The study aims to answer whether statistical testing can reduce the effort of localizing the most critical software faults, seen from user perspective. We use probabilistic transition system as the formalism for modeling a probabilistic behavior of the system, and usage models to detect faults that would impact users the most. We present an exemplary case study used to assess the performance of statistical test selection compared to manual practice.

Abstract: Software testing is the process of executing a program or system with the intent of finding errors. Budgets assigned for software testing are generally limited. Performing exhaustive testing which tests all possible input combinations (test cases) is practically impossible. A major challenge in testing is how to achieve maximum test coverage using limited number of test cases. In this article, we propose an algebraic method of creating test suites with high 3-way configuration coverage within a fixed number of test cases.

Abstract: A testing method and device for a deep network analysis system. The testing method comprises: generating a property document according to a configured network topology and service parameters; generating simulation performance data and comparison data according to the property document; testing a deep network analysis (DNA) system by utilizing the simulation performance data, and acquiring data obtained after performing analysis processing on the simulation performance data by the DNA system; and when the data obtained after the analysis processing are verified to be correct data according to the comparison data, determining that the DNA system can correctly analyze a network. By means of the solution of the embodiment of the present invention, testing of a DNA system can be realized without being limited by a testing environment, and the testing efficiency can be increased.