Abstract: What would it take to reduce speed binning's dependency on functional testing? One answer is a structural at-speed test approach that can achieve the same effectiveness as functional testing. The authors of this article offer a formula to relate structural critical-path testing frequency to system operation frequency. They demonstrate that there can be a high correlation between frequencies resulting from structural testing and those resulting from functional testing.

Abstract: Evolutionary Testing (ET) has been shown to be very successful for testing real world applications [10]. The original ET approach focuses on searching for a high coverage of the test object by generating separate inputs for single function calls. We have identified a large set of real world application for which this approach does not perform well because only sequential calls of the tested function can reach a high structural coverage (white box test) or can check functional behavior (black box tests). Especially, control software which is responsible for controlling and constraining a system cannot be tested successfully with ET. Such software is characterized by storing internal data during a sequence of calls. In this paper we present the Evolutionary Sequence Testing approach for white box and black box tests. For automatic sequence testing, a fitness function for the application of ET will be introduced, which allows the optimization of input sequences that reach a high coverage of the software under test. The authors also present a new compact description for the generation of real-world input sequences for functional testing. A set of objective functions to evaluate the test output of systems under test have been developed. These approaches are currently used for the structural and safety testing of car control systems.

Abstract: Symmetries often appear as properties of many artifical settings. In program testing, they can be viewed as properties of programs and can be given by the tester to check the correctness of the computed outcome. In this paper, we consider symmetries to be permutation relations between program executions and use them to automate the testing process. We introduce a software testing paradigm called symmetric testing, where automatic test data generation is coupled with symmetries checking to uncover faults inside the programs. A practical procedure for checking that a program satisfies a given symmetry relation is described. The paradigm makes use of group theoretic results as a formal basis to minimize the number of outcome comparisons required by the method. This approach appears to be of particular interest for programs for which neither an oracle, nor any formal specification is available. We implemented symmetric testing by using the primitive operations of the Java unit testing tool Roast by N. Daley, D. Hoffman, and P. Strooper (2002). The experimental results we got on faulty versions of classical programs of the software testing community tend to show the effectiveness of the approach.

Abstract: The two most common strategies for verification and validation, inspection and testing, are in a controlled experiment evaluated in terms of their fault detection capabilities. These two techniques are in the previous work compared applied to code. In order to compare the efficiency and effectiveness of these techniques on a higher abstraction level than code, this experiment investigates inspection of design documents and testing of the corresponding program, to detect faults originating from the design document. Usage-based reading (UBR) and usage-based testing (UBT) were chosen for inspections and testing, respectively. These techniques provide similar aid to the reviewers as to the testers. The purpose of both fault detection techniques is to focus the inspection and testing from a user's viewpoint. The experiment was conducted with 51 Master's students in a two-factor blocked design; each student applied each technique once, each application on different versions of the same program. The two versions contained different sets of faults, including 13 and 14 faults, respectively. The general results from this study show that when the two groups of subjects are combined, the efficiency and effectiveness are significantly higher for usage-based reading and that testing tends to require more learning. Rework is not taken into account, thus the experiment indicates strong support for design inspection over testing.

Abstract: Model Based Testing (MBT) is a functional testing technique that makes use of information from behavioral models of the software to carry out the testing task. This technique has been commonly used in testing of interactive systems, where the used model represents the system behavior reacting to user’s actions. Finite State Machines (FSMs) are one of the most used modeling techniques for MBT. However, traditional FSMs do not provide mechanisms to model important behavioral aspects of the software such as its data flow. This paper proposes an extension to the traditional FSMs, which provides data flow modeling mechanisms and is used as a basis to define a set of functional testing criteria, extending known structural testing criteria. Moreover, the application of the defined functional testing criteria is compared, through a practical experiment, to the application of their corresponding structural testing criteria – both applied as adequacy criteria.

Abstract: TFP 2003: Fourth Symposium on Trends in Functional Programming, TFP 2003, Edinburgh, United Kingdom, 11-12 September 2003

Abstract: Software stopping rules are tools to effectively minimize the time and cost involved in software testing. The algorithms serve to guide the testing process such that if a certain level of branch or fault (or failure) coverage is obtained without the expectation of further significant coverage, then the testing strategy can be stopped or changed to accommodate further, more advanced testing strategies. By combining cost analysis with a variety of stopping-rule algorithms, a comparison can be made to determine an optimally cost-effective stopping point. A novel cost-effective stopping rule using empirical Bayesian principles for a nonhomogeneous Poisson counting process compounded with logarithmic-series distribution (LSD) is derived and satisfactorily applied to digital software testing and verification. It is assumed that the software failures or branches covered, whichever the case may be, clustered at the application of a given test-case are positively correlated, i.e., contagious, implying that the occurrence of one software failure (or coverage of a branch) positively influences the occurrence of the next. This phenomenon of clustering of software failures or branch coverage is often observed in software testing practice. The r.v. w/sub i/ of the failure-clump size of the interval is assumed to have LSD(/spl theta/) and justified on the data sets by employing a chi-square goodness of fit testing while the distribution of the number of test cases is Poisson(/spl lambda/). Then, the distribution of the total number of observed failures, or similarly covered branches, X is a compound Poisson /sup /spl and// LSD, i.e., negative binomial distribution, given that a certain mathematical identity holds. For each checkpoint in time, either the software satisfies a desired reliability attached to an economic criterion, or else the software testing is allowed to continue. By using a one-step-look-ahead formula derived for the model, the proposed stopping rule is applied to five test case-based data sets acquired by testing embedded chips through the complex VHDL models. Further, multistrategy testing is conducted to show its superiority to single-stage testing. Results are satisfactorily interpreted from a practitioner's viewpoint as an innovative alternative to the ubiquitous test-it-to-death approach, which is known to waste billions of test cases in a tedious process of finding more bugs. Moreover, the proposed dynamic stopping-rule algorithm can validly be employed as an alternative paradigm to the existing on-line statistical process control methods static in nature for the manufacturing industry, provided that underlying statistical assumptions hold. A detailed comparative literature survey of stopping-rule methods is also included in terms of pros and cons, and cost effectiveness.

Abstract: The aim of passive testing is to detect faults in a system while observing the system during normal operation, that is, without forcing the system to specialized inputs explicitly for the purposes of testing. We formulate some general correctness requirements on any passive-testing algorithm which we term soundness and completeness. With respect to these definitions, we show that the homing algorithm, first proposed in [4], and subsequently used in [6–8], is sound and complete for passively testing the conformance of an implementation for several distinct conformance notions ranging from trace-containment to observational equivalence to even exact identity. This implies that, for some notions of conformance, there are faulty implementations that would not be detectable by any sound passive testing algorithm. We define a property to be passively testable as one admitting complete fault coverage under passive testing, i.e., one for which any faulty execution can be detected through passive testing. We provide an exact characterization of passively testable properties as being a natural subclass of safety properties, namely, those that are trace-contained in sets that are prefix- and suffix-closed. For such properties, we derive efficient complete passive testing algorithms that take constant time. We demonstrate the applicability of these results to networks and network devices by considering the problem of passively testing an implementation for conformance to the TCP protocol.

Abstract: Extended Finite State Machine (EFSM) is an extension of Finite State Machine (FSM). It add variables, operation and the precondition of transitions on the basis of FSM, and we can characterize the dynamic behavior of software system more precisely using EFSM. Testing based on EFSM, which can be used in many fields, have great research value and practical significance. Many researchers have proposed many testing sequence generation methods based on FSM, but this problem is more complicated in EFSM context. We consider that the testing input of the testing based on EFSM should include two parts: testing sequences and the determinate input variables’ value in the sequences, which is called testing data. The work of selection testing data by hand is very tedious and may increase the cost of testing. If we can generate those testing data automatically, the efficiency of testing can be improved and the cost can be reduced also. In this paper, we propose an automatic testing data generation method in the testing based on EFSM. In this method, we use two key phases, interval narrowing and subsection gradient optimal descent algorithms to generate testing data automatically. In the interval-narrowing phase, we design a group of interval narrowing operators to reduce the variables’ value scope. If the precondition in the state transitions is simple, we can get the testing data in this phase. In the second phase, we use subsection gradient optimal descent algorithms to generate testing data from reduced variables’ value scope. The experiment shows that we can generate most of testing data automatically using this method, and has speedy convergence. In the situation of having no solution of input variables, interval narrowing can provide small intervals of the value of input variables for tester, which would be convenient for tester to select testing data from those small intervals.

Abstract: In this paper we address the problem of testing real-time software in the functional domain In order to achieve reproducible and deterministic test results of an entire multitasking real-time system it is essential not to only consider inputs and outputs, but also the order in which tasks communicate and synchronize with each other We present a deterministic white-box system-level control-flow testing method for deterministic integration testing of real-time system software We specifically address fixed priority scheduled real-time systems where synchronization is resolved using the Priority Ceiling Emulation Protocol or offsets in time The method includes a testing strategy where the coverage criterion is defined by the number of paths in the system control flow The method also includes a reachability algorithm for deriving all possible paths in terms of orderings of task starts, preemptions and completions of tasks executing in a real-time system The deterministic testing strategy allows test methods for sequential programs to be applied, since each identified ordering can be regarded as a sequential program

Abstract: A method for performing testing of a simulated direct access storage device in a testing simulation environment is disclosed. The method provides a software representation of a plurality of hardware components within the simulated direct access storage device. The method also uses a control program module within the testing simulation environment, wherein the control program module interacts with the software representation of the plurality of hardware components, and a testing program for interacting with the control program module and the software representation of the plurality of hardware components. In response to detection of an occurrence of a pre-selected event within the simulated direct access storage device, one or more codes are sent from the testing program to the software representation of the plurality of hardware components and whether or not a response by the control program module to the one or more codes is correct is determined.

Abstract: Testing and debugging are activities that consume a significant amount of the software quality and maintenance budget. To reduce such cost, the use of testing information for debugging purposes has been advocated In general, heuristics are used to select structural testing requirements (nodes, branches, and definition use associations) more closely related to the manifestation of failures which are then mapped into pieces of code. The problem with this approach is that it relies only on a static information-a fragment of code. We introduce a strategy for fault localization based on the investigation of indications (or hints) provided at run-time by data-flow testing requirements. The intuition is that the selected data-flow testing requirements may fail to hit the fault site but they still provide useful information for fault localization. The strategy-called Debugging strategy based on Requirements of Testing (DRT)-is targeted to such situations. The strategy's novelty and attractiveness are threefold: (i) it focuses on dynamic information related to testing data; (ii) it can be implemented in state-of-the-practice symbolic debuggers with limited overhead; and (iii) it utilizes algorithms that consume constant memory and are linear with respect to the number of branches in the program. The paper presents a case study which shows that our intuition is valid (for the subject program) and a proof-of-concept tool that implements the strategy.

Abstract: This paper presents a testing method for electronic devices with no a-priori yield information. This problem is referred to as the unknown-good-yield (UKGY) problem. The UKGY problem of systems-on-chip (SoC) is discussed in this paper as SoCs are in general built with embedded intellectual property (IP) cores, each of which is procured from IP providers with no information on known-good-yield (KGY). In general, partial testing is a practical choice for assuring the yield of the product under the stringent time-to-market requirement in today's high density/complexity electronic devices such as SoCs built with deep submicron or nano technology. Therefore, an efficient and effective sampling technique is a key to the success of high confidence testing. An experimental characterization-based testing (referred to as ET) method for SoC has been proposed prior to this work, in which a stratified sampling method was employed based on environmental-based characterization and an experimental design technique to enhance the confidence level of the estimation of yield. The proposed testing method, referred to as regressive testing (RegT), in this paper exploits another method by using parameters (referred to as assistant variables (AV)) free from UKGY that determines the criteria to sample and test SoCs, and employs the regression analysis method to evaluate the yield with regard to confidence interval. A numerical simulation is conducted to demonstrate the efficiency and effectiveness of the proposed RegT in comparison with generic random testing method.

Abstract: The actual and appealing objective of XP's approach to unit testing is to improve quality by avoiding errors beforehand rather than to find and fix bugs afterwards Conventional testing, on the contrary, focuses on a posteriori analysis to find errors and issues that should be corrected Both approaches have their advantages and drawbacks, and both are valuable and necessary This paper describes how we combined both approaches by extending our test management environment TEMPPO for unit testing with JUnit to include testers in early unit testing activities

Abstract: Performance testing is an important way to ensure the success of a Web application However, if the test is not properly performed or the dependency on the testing environment is not fully understood, the results may be misleading Response time is a key measure of performance, especially from the user's perspective A series of studies to characterize response time of Web applications are presented The experimental results are analyzed using statistical methods The study shows that the response time increases linearly with the number of users, and it is also affected by the time of testing and test configuration

Abstract: We describe solutions, how to improve a quality of basic testing methods to be possible to use them for dynamic testing high-resolution/middle speed AD converters and AD modules. The following methods are described: an application of low-distortion band pass filters, a digital correction of a known distortion of a testing signal, an application of a two-tone (or multitone) signal, an application of a nondeterministic signal (noise). The method, which combines a filtration of testing signal and a following digital correction, is described and proposed as the perspective one. The AD transfer standard, which is applied for an evaluation of testing methods and systems is also mentioned

Abstract: This paper proposes a scheme of at-speed current testing by applying two alternative vectors to circuits under test for a fault, to enable a slow measurement under a high frequency operation. The paper presents test generation of the two alternative vectors at gate level by means of counting only logical up-transitions based on a Boolean process, and employing the Bayesian optimization algorithm. SPICE simulation shows that the responses of tests generated by the algorithm can be observed either by a low-cost ATE or a waveform sensor.

Abstract: (337) 482-6206 Abstract - Re-implementation testing is performed to capture software errors, reduce unnecessary post- implementation testing, and prevent error recurrence. Rigorous pre-implementation testing is never sufficient to eliminate all probability of error occurrence and generally is cost-prohibitive; however, a minimum level of testing is necessary to reduce the probability of serious errors. Regardless of the importance that this minimum level of testing be performed prior to implementation, no defined framework designed to ensure sufficient testing of objects currently exists. The purpose of this paper is to describe such a framework. Testing all methods in an object generally is sufficiently similar to the testing of an object for pre-implementation testing purposes. In this paper, we present a remodeled testing assistant, the Object-Oriented Testing Assistant 11 (OOTA III) (1,14,15,16) that facilitates the testing of object-oriented code by enforcing .method coverage for each class. Method coverage involves triggering each method in every object in order to test all methods in that object. Consequently, testing at the object level is completed before proceeding to the class-level of testing. OOTA' was updated to generate test drivers for C" , C#, and Java objects. Testing is a critical phase of Object Oriented (00) software development, and

Abstract: The comparison of partition and random sampling methods for software testing has received considerable attention in the literature. A standard criterion for comparisons between random and partition testing based on their expected efficacy in program debugging is the probability of detecting at least one failure causing input in the program's domain. We investigate the relative effectiveness of partition testing versus random testing through the powerful mathematical technique of majorization, which was introduced by Hardy et al. (1952). The tools of majorization and the concepts of Schur (convex and concave) functions (1923) enable us to derive general conditions under which partition testing is superior to random testing and, consequently, to give further insights into the value of partition testing strategies.

Abstract: Group Testing refers to the situation in which one is given a set of objects O, an unknown subset P ⊆ O, and the task is to determine P by asking queries of the type "does P intersect Q?", where Q is a subset of O. Group testing is a basic search paradigm that occurs in a variety of situations such as quality control in product testing, searching in storage systems, multiple access communications, and software testing, among the others. Group testing procedures have been recently applied in Computational Molecular Biology, where they are used for screening library of clones with hybridization probes and sequencing by hybridization. Motivated by particular features of group testing algorithms used in biological screening, we study the efficiency of two-stage group testing procedures. Our main result is the first optimal two-stage algorithm that uses a number of tests of the same order as the information theoretic lower bound on the problem. We also provide efficient algorithms for the case in which there is a Bernoulli probability distribution on the possible sets P, and an optimal algorithm for the case in which the outcome of tests may be unreliable because of the presence of "inhibitory" items in O. Our results depend on a combinatorial structure introduced in this paper. We believe that it will prove useful in other contexts too.

Abstract: Complete interaction testing of components is too costly in all but the smallest systems. Yet component interactions are likely to cause unexpected faults. Recently, design of experiment techniques have been applied to software testing to guarantee a minimum coverage of all t-way interactions across components. However, t is always fixed. This paper examines the need to vary the size of t in an individual test suite and defines a new object, the variable strength covering array that has this property. We present some computational methods to find variable strength arrays and provide initial bounds for a group of these objects.