TL;DR: This paper presents and evaluates a stopping rule that can be used to determine when to stop the current testing phase using a given testing technique, and compares savings and quality of testing both with and without using the stopping rule.
Abstract: Testing behavioral models before they are released to the synthesis and logic design phase is a tedious process, to say the least. A common practice is the test-it-to-death approach in which millions or even billions of vectors are applied and the results are checked for possible bugs. The vectors applied to behavioral models include functional vectors, but the significant amount of the vectors are random in nature, including random combinations of instructions. In this paper, we present and evaluate a stopping rule that can be used to determine when to stop the current testing phase using a given testing technique. We demonstrate the use of the stopping rule on two complex VHDL models that were tested for branch coverage with 4 different testing phases. We compare savings and quality of testing both with and without using the stopping rule.
TL;DR: A stopping rule is presented and evaluated that can be used to determine when it is time to switch to a different testing technique, because the current one is not likely to increase criteria fulfilment.
Abstract: When testing software, testers rarely use only one technique to generate tests that, they hope, will fulfill their testing criteria. Malaiya showed that testers switch strategies when testing yield saturates. We present and evaluate a stopping rule that can be used to determine when it is time to switch to a different testing technique, because the current one is not likely to increase criteria fulfilment. We demonstrate use of the stopping rule on a program that is being tested for branch coverage with five different testing techniques. We compare savings and accuracy of stopping both with and without using the stopping rule.
TL;DR: The proposed method for reproducible and deterministic testing of distributed real-time systems is extended to also handle interrupts, essential since critical parts of most real world systems are managed by interrupts.
Abstract: Testing techniques for sequential software are well established. Unfortunately, these testing techniques are not directly applicable to distributed real-time systems. In previous work we have proposed a method for reproducible and deterministic testing of distributed real-time systems. This method identifies all possible orderings of task starts, preemptions and completions. Each ordering is regarded as a sequential program, and can as such be tested with standard sequential testing techniques. In this paper we extend our method to also handle interrupts. This is essential since critical parts of most real world systems are managed by interrupts.
TL;DR: In this paper, a random number generator is used in the selection of the test parameters, when errors occur, the array of test parameters is dynamically adapted during testing to change the probability of selecting certain test parameters in response to the error.
Abstract: An apparatus for testing electro-mechanical storage devices, such as disk drives, is provided. A testing device in a digital computer selects test parameters from an array of test parameters stored in the digital computer. A random number generator is used in the selection of the test parameters. When errors occur, the array of test parameters is dynamically adapted during testing to change the probability of selecting certain test parameters in response to the error.
TL;DR: This paper presents an integrated approach that keeps the strengths of the two testing strategies while avoiding their deficiencies, and lets a variable definition be “domain tested”.
TL;DR: This presentation proposes a diagram capturing all possible dependencies/interactions in an object-oriented program and gives algorithms and coverage criteria to identify integration resp.
Abstract: Systematic testing of object-oriented software turned out to be much more complex than testing conventional
software Especially the highly incremental and iterative development cycle demands both many more
changes and partially implemented resp re-implemented classes Much more integration and regression testing
has to be done to reach stable stages during the development In this presentation we propose a diagram capturing all possible dependencies/interactions in an object-oriented program Then we give algorithms and coverage criteria to identify integration resp regression test strategys and all test cases to be executed after some implementation resp modification activities Finally, we summarize some practical experiences and heuristics
TL;DR: This paper investigates the effectiveness of the optimally refined proportional sampling (ORPS) strategy, which is a special case of the proportional sampling strategy, and the performance was found to be better than random testing.
Abstract: Recently, the effectiveness of subdomain testing and random testing has been studied analytically. T.Y. Chen and Y.T. Yu (1994) found that, for the case of disjoint subdomains, as long as the number of test cases selected from each subdomain is proportional to its size (the proportional sampling strategy), the probability of revealing at least one failure using subdomain testing is not less than that using random testing. This paper investigates the effectiveness of the optimally refined proportional sampling (ORPS) strategy, which is a special case of the proportional sampling strategy. The ORPS strategy is simple in concept, and the implementation cost is usually low. An empirical study has been conducted for a sample of published programs with seeded errors. The performance of this strategy was found to be better than random testing.
TL;DR: The aim of this paper is to find an allocation minimizing the variance of this dispersion of discrete testing resources in module testing where software modules are tested independently.
Abstract: This paper treats an allocation problem of discrete testing resources in module testing where software modules are tested independently. The total amount of discrete testing resources is given beforehand. When an allocation is determined in module testing, each software module defines the share per size. Since discrete resources do not allow each software module to get an equal share per size, there exists a dispersion. The aim of this paper is to find an allocation minimizing the variance of this dispersion.
TL;DR: The use of approximation methods to generate test distributions satisfying the Generalised Proportional Sampling strategy are studied, showing that on average about 98.72% to almost 100% of the test distributions obtained in this way are better than random testing in terms of the P-measure.
Abstract: Previous studies have shown that partition testing strategies can be very effective in detecting faults, but they can also be less effective than random testing under unfavourable circumstances. When test cases are allocated in proportion to the size of subdomains, partition testing strategies are provably better than random testing, in the sense of having a higher or equal probability of detecting at least one failure (the P-measure). Recently, the Generalised Proportional Sampling (GPS) strategy, which is always satisfiable, was proposed to relax the proportionality condition. The paper studies the use of approximation methods to generate test distributions satisfying the GPS strategy, and evaluates this proposal empirically. Our results are very encouraging, showing that on average about 98.72% to almost 100% of the test distributions obtained in this way are better than random testing in terms of the P-measure.
TL;DR: A software testing package using structural testing that can analyze any program written in C language and is applied to a wide spectrum of C-like languages is proposed and implemented.
Abstract: In this paper, a software testing package using structural testing had been proposed and implemented. It can analyze any program written in C language. Moreover it can be applied to a wide spectrum of C-like languages. The C language had been chosen as an example for it. It makes two useful functions; one is a static analysis for the program under test and the other is a dynamic analysis for that program.
TL;DR: It is shown that under uncertainty, partition testing compares more favorably to random testing than suggested by prior investigations concerning the deterministic case: the restriction to failure rates that are known with certainty systematically favors random testing.
Abstract: This paper compares partition testing and random testing on the assumption that program failure rates are not known with certainty before testing and are, therefore, modeled by random variables. It is shown that under uncertainty, partition testing compares more favorably to random testing than suggested by prior investigations concerning the deterministic case: the restriction to failure rates that are known with certainty systematically favors random testing. In particular, we generalize a result by Weyuker and Jeng (1991) stating equal fault detection probabilities for partition testing and random testing in the case where the failure rates in the subdomains defined by the partition are equal. It turns out that for independent random failure rates with equal expectation, the case above is a boundary case (the worst case for partition testing), and the fault detection probability of partition testing can be up to k times higher than that of random testing, where k is the number of subdomains. Also in a related model for dependent failure rates, partition testing turns out to be consistently better than random testing. The dominance can also be verified for the expected (weighted) number of detected faults as an alternative comparison criterion.