Abstract: The fundamental task of group testing is to recover a small distinguished subset of items from a large population while efficiently reducing the total number of tests (measurements). The key contribution of this paper is in adopting a new information-theoretic perspective on group testing problems. We formulate the group testing problem as a channel coding/decoding problem and derive a single-letter characterization for the total number of tests used to identify the defective set. Although the focus of this paper is primarily on group testing, our main result is generally applicable to other compressive sensing models.

Abstract: A goal of performance testing is to find situations when applications unexpectedly exhibit worsened characteristics for certain combinations of input values. A fundamental question of performance testing is how to select a manageable subset of the input data faster to find performance problems in applications automatically. We offer a novel solution for finding performance problems in applications automatically using black-box software testing. Our solution is an adaptive, feedback-directed learning testing system that learns rules from execution traces of applications and then uses these rules to select test input data automatically for these applications to find more performance problems when compared with exploratory random testing. We have implemented our solution and applied it to a medium-size application at a major insurance company and to an open-source application. Performance problems were found automatically and confirmed by experienced testers and developers.

Abstract: One motivation for property testing of boolean functions is the idea that testing can provide a fast preprocessing step before learning. However, in most machine learning applications, it is not possible to request for labels of arbitrary examples constructed by an algorithm. Instead, the dominant query paradigm in applied machine learning, called *active learning*, is one where the algorithm may query for labels, but only on points in a given (polynomial-sized) unlabeled sample, drawn from some underlying distribution D. In this work, we bring this well-studied model to the domain of testing. We develop both general results for this *active testing* model as well as efficient testing algorithms for several important properties for learning, demonstrating that testing can still yield substantial benefits in this restricted setting. For example, we show that testing unions of d intervals can be done with O(1) label requests in our setting, whereas it is known to require Omega(d) labeled examples for learning (and Omega(sqrt{d}) for passive testing [KR00] where the algorithm must pay for every example drawn from D). In fact, our results for testing unions of intervals also yield improvements on prior work in both the classic query model (where any point in the domain can be queried) and the passive testing model as well. For the problem of testing linear separators in R^n over the Gaussian distribution, we show that both active and passive testing can be done with O(sqrt{n}) queries, substantially less than the Omega(n) needed for learning, with near-matching lower bounds. We also present a general combination result in this model for building testable properties out of others, which we then use to provide testers for a number of assumptions used in semi-supervised learning. In addition to the above results, we also develop a general notion of the *testing dimension* of a given property with respect to a given distribution, that we show characterizes (up to constant factors) the intrinsic number of label requests needed to test that property. We develop such notions for both the active and passive testing models. We then use these dimensions to prove a number of lower bounds, including for linear separators and the class of dictator functions.

Abstract: Bucket testing, also known as split testing, A/B testing, or 0/1 testing, is a widely used method for evaluating users' satisfaction with new features, products, or services. In order not to expose the whole user base to the new service, the mean user satisfaction rate is estimated by exposing the service only to a few uniformly chosen random users. In a recent work, Backstrom and Kleinberg, defined the notion of network bucket testing for social services. In this context, users' interactions are only valid for measurement if some minimal number of their friends are also given the service. The goal is to estimate the mean user satisfaction rate while providing the service to the least number of users. This constraint makes uniform sampling, which is optimal for the traditional case, grossly inefficient. In this paper we introduce a simple general framework for designing and evaluating sampling techniques for network bucket testing. The framework is constructed in a way that sampling algorithms are only required to generate sets of users to which the service should be provided. Given an algorithm, the framework produces an unbiased user satisfaction rate estimator and a corresponding variance bound for any network and any user satisfaction function. Furthermore, we present several simple sampling algorithms that are evaluated using both synthetic and real social networks. Our experiments corroborate the theoretical results and demonstrate the effectiveness of the proposed framework and algorithms.

Abstract: System testing of real-time embedded systems (RTES) is a challenging task and only a fully automated testing approach can scale up to the testing requirements of industrial RTES. One such approach, which offers the advantage for testing teams to be black-box, is to use environment models to automatically generate test cases and oracles and an environment simulator to enable earlier and more practical testing. In this paper, we propose novel heuristics for search-based, RTES system testing which are based on these environment models. We evaluate the fault detection effectiveness of two search-based algorithms, i.e., Genetic Algorithms and (1+1) Evolutionary Algorithm, when using these novel heuristics and their combinations. Preliminary experiments on 13 carefully selected, non-trivial artificial problems, show that, under certain conditions, these novel heuristics are effective at bringing the environment into a state exhibiting a system fault. The heuristic combination that showed the best overall performance on the artificial problems was applied on an industrial case study where it showed consistent results.

Abstract: Structural testing is most important and high demand testing technique for code-based criteria in software testing. In structural testing, path testing technique is the most useful technique. In path testing, generation of all independent paths (non redundant) is a complex one. The aim of this paper is to present a simple and efficient algorithm that can automatically generate all possible paths in a Control Flow Graph for Path testing. Cuckoo behaviour is used in this algorithm for extracting optimal paths. This cuckoo search algorithm generates paths equal to the cyclomatic complexity. It can be shown that the proposed approach guarantees full path coverage.

Abstract: Effective system testing of real-time embedded systems (RTES) requires a fully automated approach. One such black-box system testing approach is to use environment models to automatically generate test cases and test oracles along with an environment simulator to enable early testing of RTES. In this paper, we propose a hybrid strategy, which combines (1+1) Evolutionary Algorithm (EA) and Adaptive Random Testing (ART), to improve the overall performance of system testing that is obtained when using each single strategy in isolation. An empirical study is carried out on a number of artificial problems and one industrial case study. The novel strategy shows significant overall improvement in terms of fault detection compared to individual performances of both (1+1) EA and ART.

Abstract: Testing Software Product Lines is a very challenging task due to variability. Frequently, approaches such as combinatorial testing are used to generate representative sets of products for testing purposes instead of testing each individual product of the SPL under test. In this contribution we present the results of applying the MoSo-PoLiTe framework at Danfoss Power Electronics A/S to calculate a representative set of product configurations for black box testing purposes. Within this evaluation we use MoSo-PoLiTe's pairwise configuration selection component on the basis of a feature model. This component implements a heuristics finding a minimal subset of configurations covering 100% T-wise feature interaction. According to the best of our knowledge, this is the first publication providing industrial results about pairwise SPL testing.

Abstract: Configurable systems that let users customize system behaviors are becoming increasingly prevalent. Testing a configurable system with all possible configurations is very expensive and often impractical. For a single version of a configurable system, sampling approaches exist that select a subset of configurations from the full configuration space for testing. However, when a configurable system changes and evolves, existing approaches for regression testing select all configurations that are used to test the old versions for testing the new version. As demonstrated in our experiments, this retest-all approach for regression testing configurable systems turns out to be highly redundant. To address this redundancy, we propose a configuration selection approach for regression testing. Formally, given two versions of a configurable system, S (old) and S' (new), and given a set of configurations C S for testing S, our approach selects a subset C S' of C S for regression testing S'. Our study results on two open source systems and a large industrial system show that, compared to the retest-all approach, our approach discards 15% to 60% of configurations as redundant. Our approach also saves 20% to 55% of the regression testing time, while retaining the same fault detection capability and code coverage of the retest-all approach.

Abstract: Adaptive random testing is an enhancement of random testing. Previous studies on adaptive random testing assumed that once a failure is detected, testing is terminated and debugging is conducted immediately. It has been shown that adaptive random testing normally uses fewer test cases than random testing for detecting the first software failure. However, under many practical situations, testing should not be withheld after the detection of a failure. Thus, it is important to investigate the effectiveness with respect to the detection of multiple failures. In this paper, we compare adaptive random testing and random testing under various scenarios and examine whether adaptive random testing is still able to use fewer test cases than random testing to detect multiple software failures. Our study delivers some interesting results and highlights a number of promising research projects. Copyright © 2011 John Wiley & Sons, Ltd.

Abstract: According to one aspect of the invention, a system and method for unit and regression testing has been developed. The system is capable to automate unit/regression test case preparation, remote execution, provide XML based assert mechanism to evaluate test results, maintenance and versioning. The present system enables model driven testing of domain specific languages and provides a user friendly mechanism to perform Unit and Regression Testing.

Abstract: Given oracle access to some boolean function f, how many queries do we need to test whether f is linear? Or monotone? Or whether its output is completely determined by a small number of the input variables? This thesis studies these and related questions in the framework of property testing introduced by Rubinfeld and Sudan ('96). The results of this thesis are grouped into three main lines of research. I. We determine nearly optimal bounds on the number of queries required to test k-juntas (functions that depend on at most k variables) and k-linearity (functions that return the parity of exactly k of the input bits). These two problems are fundamental in the study of boolean functions and the bounds obtained for these two properties lead to tight or improved bounds on the query complexity for testing many other properties including, for example, testing sparse polynomials, testing low Fourier degree, and testing computability by small-size decision trees. We give a partial characterization of the set of functions for which we can test isomorphism—that is, identity up to permutation of the labels of the variables—with a constant number of queries. This result provides some progress on the question of characterizing the set of properties of boolean functions that can be tested with a constant number of queries. II. We establish new connections between property testing and other areas of computer science. First, we present a new reduction between testing problems and communication problems. We use this reduction to obtain many new lower bounds in property testing from known results in communication complexity. Second, we introduce a new model of property testing that closely minors the active learning model. We show how testing results in this new model may be used to improve the efficiency of model selection algorithms in learning theory. The results presented in this thesis are obtained by applying tools from various mathematical areas, including probability theory, the analysis of boolean functions, orthogonal polynomials, and extremal combinatorics.

Abstract: A testing system can perform scalability testing on a target system, including automatically identifying a performance bottleneck component of the target system when the target system includes multiple components. The testing system can specify a target load of a specified component of the target system. The testing system can provide a simulated request to the target system and measure performance of the target system. Based on the measurement, the testing system can determine a scaling factor. The testing system can scale up the simulated request by the scaling factor, and determine whether one or more components of the target system have reached full capacity. The testing system can then adjust the scaling factor and the simulated request, until the testing system identifies a component of the target system that is a performance bottleneck of the target system when a specific number of requests are provided.

Abstract: A good strategy to test a software component involves the generation of the whole set of cases that participate in its operation. While testing only individual values may not be enough, exhaustive testing of all possible combinations is not always feasible. An alternative technique to accomplish this goal is called combinato- rial testing. Combinatorial testing is a method that can reduce cost and increase the effectiveness of software testing for many applications. It is based on con- structing functional test-suites of economical size, which provide coverage of the most prevalent configurations. Covering arrays are combinatorial objects, that have been applied to do functional tests of software components. The use of cov- ering arrays allows to test all the interactions, of a given size, among the input parameters using the minimum number of test cases. For software testing, the fundamental problem is finding a covering array with the minimum possible number of rows, thus reducing the number of tests, the cost, and the time expended on the software testing process. Because of the importance of the construction of (near) optimal covering arrays, much research has been carried out in developing effective methods for constructing them. There are several reported methods for constructing these combinatorial models, among them are: (1) algebraic methods, recursive methods, (3) greedy methods, and (4) metaheuristics methods. Metaheuristic methods, particularly through the application of simulated anneal- ing has provided the most accurate results in several instances to date. Simulated annealing algorithm is a general-purpose stochastic optimization method that has proved to be an effective tool for approximating globally optimal solutions to many optimization problems. However, one of the major drawbacks of the simulated an- nealing is the time it requires to obtain good solutions. In this thesis, we propose the development of an improved simulated annealing algorithm

Abstract: Software testing is an inevitable activity of software development which is crucial to the software quality and consumes approximately 50% of the software development cost. Test case design is the most important activity in testing which determines software quality. The program with the moderate complexity cannot be tested completely but verified only for input situations selected as test data. Innovative methods are emerging to perform testing as a whole and unit testing in particular with minimum effort and time. Unit testing is mostly done by developers under a lot of schedule pressure since the software companies find a compromise among functionality, time to market and quality. Thus there is a need for reducing unit testing time by optimizing and automating the process. Test suite generation is an errorprone, tedious and time consuming part of unit testing. A novel technique is proposed to automatically generate test cases from the input domain using meta heuristic search technique scatter search for branch coverage criteria with respect to cyclomatic complexity measure.

Abstract: The present invention proposes an information interaction testing device and method based on the associated testing case automatic generation. The associated testing case generation module in said device may automatically generate the associated testing case files corresponding to all associated information interactions which can be triggered by said reference information interaction based on the reference information interaction and the predefined rules determined by the application type provided by the system under test. The information interaction testing device and method based on the associated testing case automatic generation disclosed in the present invention have the higher testing speed and the higher testing usability as well as are low-cost.

Abstract: Dimensional software GT-Power was applied to simulate the engine performance. The simulation results were compared to the test results provided by the JH600 engine manufacturer with good agreement. The parameters including the sphere style plenum diameter, the intake runner diameter, exhaust runner lengths and the position of retractor were optimized via a combination of the 1-Dimensional simulation and an orthogonal L9 (3 4 ) testing design. The optimal results can limit the effect of restrictor, and then improve the restricted engine's ability effectively in the high speed range (4000 to 7000rpm).

Abstract: Set-Top Box device functionalities have been advancing fast during the last decade. This has been mainly triggered by appearance of new multimedia services and applications. Given the high complexity of the STB and large amount of services to be supported, the STB functional verification has become highly challenging task, from both the testing time and testing reliability perspective. High competitiveness and the need to reduce time-to-market make efficient automatic STB testing absolutely necessary. Although automatic testing is considered superior to the manual testing, in general it is difficult to design it for different real case scenarios, and especially in case the input multimedia signal is unknown (live video stream). Another problem is that optimally multiple STB should be tested simultaneously in order to reduce the testing time, in cases when detailed stress testing is required. In order to address these issues, in this paper we propose a new solution and system (with hardware and software modules) for automatic simultaneous functional verification system of multiple Set-Top Boxes based on Black Box Testing (BBT). In the proposed testing system one STB serves as the referent one (for the rest of the STBs under test) with a priory known "faultless" functionality. This enables the STB functional testing with unknown input signal (during live video stream), because the output of the referent STB will serve as the expected output given the same unknown input signal. This is considered as the main contribution of the proposed testing approach along with the acceleration of the STB functional verification process which ensures reduction of time and costs that is considered to be crucial in STB production. The experimental results show high testing accuracy, robustness and efficiency in case of live broadcast video (DVB-S).

Abstract: Most BIST architectures use pseudo-random test pattern generators. However, whenever this technique has been applied to on-chip interconnection networks, overly large testing latencies have been reported. On the other hand, alternative approaches either suffer from large area penalties (like scan-based testing or the use of deterministic test patterns) or poor coverage of faults in the control path (functional testing). This paper presents the optimization of a built-in self-testing framework based on pseudo-random test patterns to the microarchitecture of network-on-chip switches. As a result, fault coverage and testing latency approach those achievable with deterministic test patterns while materializing relevant area savings and enhanced flexibility.

Abstract: Software Product Lines (SPLs) are increasing in relevance and importance as various domains strive to cope with the challenges of supporting a high degree of variability in modern software systems. Especially the systematic testing of SPLs is non-trivial as a high degree of variability implies a vast number of possible products.As testing every valid product individually quickly becomes infeasible, heuristics are often used to choose a representative subset of products to be tested. MoSo-PoLiTe (Model-Based Software Product Line Testing) is a framework for SPL testing that combines and applies combinatorial (in particular pairwise) and model-based testing to SPL feature models. In this paper, we (1) present MoSo-PoLiTe as a novel case study for graph transformations in general and Story Driven Modelling (SDM) in particular, (2) show why we consider SDMs to be ideal for rapid prototyping optimization strategies in this context, and (3) evaluate our implemented optimizations and quantify the realized improvements for MoSo-PoLiTe.

Abstract: Embedded systems are very complex and integrate heterogenous components on a single chip. Testing methodologies differ in hardware and software domains. Hardware testing concerns with the functional verification whereas the software testing is concerned with the series of processes of dynamically executing a program with the given inputs to make sure that code does that what it was designed for. A multilevel testing method is concerned with both the hardware and software domain. In comparison to conventional approach it involves all levels of testing. This paper explores the multilevel test method for testing microcontroller based ECG System.

Abstract: In this paper, we systematically studied the complexity measurement, SIMD (Single Input Multiple Data) fault types and testing methodologies for H.264 codec. To the best of our knowledge, it is the first attempt to address these problems. Firstly, two complexity metrics are calculated for various optimized functions in H.264 reference software. These measures have been found to be strongly correlated to the number of faults in software testing. Secondly, we introduced a new category of SIMD faults. Conformance testing, random testing and manual testing are proposed to deal with these SIMD faults as well as conventional faults. Results have shown that, conformance testing, often used as a mechanism to verify the conformity of a decoder under test (DUT), can also be used to discover faults in the studied optimized functions. Random testing is able to detect simple faults at both encoder and decoder functions. Manual testing is especially effective for difficult faults. In practice, one or more of the three techniques can be chosen as needed to increase fault detection rate and speed for H.264 video codec testing.

Abstract: The effects of stamping process parameters (blank thickness, blank holder force, friction coefficient, die clearance) on the formability of an automobile reinforced plate were investigated. The process parameters are optimized based on the results of orthogonal testing. The optimal process parameters were obtained by simulation result analysis of variance. The accuracy of numerical simulation is verified by the experiment, which provides guidance for the actual production.

Abstract: Test data generation is a key problem of combination software testing. However, the complexity of test case generation problem for combinatorial testing is NP-complete. In this study, we propose a global optimization and generation method to construct combinatorial testing data. Firstly, an encoding mechanism is used to map the combinatorial testing problem domain to a binary coding space. Then, an improving ethnic group evolution algorithm (EGEA/H) is used to search the binary coding space in order to find the optimal code schema. After that, a decoding mechanism is used to read out the composition information of combinatorial testing data from the optimal code schema and construct optimal combinatorial testing suite according to it. The simulation results show this method is simple and effective, and it has the characteristics of less producing test data and time consumption.

Abstract: In this paper, we pursue our works on generic modeling and conformance testing of componentbased systems. Hence, we defined a unified framework for modeling abstract components, and proposed a conformance testing theory for components. This was based on coalgebras, which are categorical representations allowing the unification of a large family of formalisms for specifying state-based systems. Here, we extend our theory of conformance testing to the testing of component-based systems. We first show that testing a global system can be done by testing its components thanks to the projection of global behaviors onto local ones. As a consequence of this result, there is no need to re-test the global system for conformance. Secondly, based on our projection techniques, we define a framework to build adequate test purposes automatically for testing components in the context of the global system where they are plugged in. The underlying idea is to identify from any trace tr of the global system, the trace of any component involved in tr. Those projected traces can be then seen as test cases that should be tested on individual components. Hence, those test purposes will capture behaviors of sub-systems that typically occur in the whole system. This will be done by combining projection mechanisms and execution mechanisms to generate system computation trees.