Abstract: This paper considers test case selection for programs whose specifications are expressed by Boolean algebra. The approach is to select test cases based on Boolean specifications. Three test case selection strategies are proposed that aim at the detection of the literal insertion fault and the literal reference fault. Although the MAX-B strategy proposed by Weyuker et al. guarantees detection of these types of faults, the proposed strategies are more effective in the sense that the derived test cases form a subset of those selected by the MAX-B strategy. Copyright © 2001 John Wiley & Sons, Ltd.

Abstract: A number of current control systems for aircraft have been specified with statecharts. The risk of failures requires the use of a formal testing approach to ensure that all possible faults are considered. However, testing the compliance of an implementation of a system to its specification is dependent on the specification method and little work has been reported relating to the use of statechart-specific methods. This paper describes a modification of a formal testing method for extended finite-state machines to handle the above problem. The method allows one to demonstrate correct behaviour of an implementation of some system, with respect to its specification, provided certain specific requirements for both of them are satisfied. The case study illustrates these and shows the applicability of the method. By considering the process used to develop the system it is possible to reduce the size of the test set dramatically; the method to be described is easy to automate. Copyright © 2001 John Wiley & Sons, Ltd.

Abstract: Mutation testing, originally proposed for unit testing, has been extended to integration testing with the proposition of the Interface Mutation criterion. This paper presents the results of an experiment using two mutation-based testing criteria for unit and integration testing phases: the Mutation Analysis and the Interface Mutation adequacy criteria, respectively. The aim is to investigate how they can be used in a complementary way during the testing activity, establishing an incremental testing strategy comprising the unit and integration testing phases and guidelines on how to obtain a high mutation score with respect to mutation testing with a low cost, in terms of the number of mutants generated. Copyright © 2001 John Wiley & Sons, Ltd.

Abstract: The importance of effective requirements analysis techniques cannot be overemphasized when developing software requiring high levels of assurance. Requirements analysis can be largely classified as either structural or functional. The former investigates whether definitions and uses of variables and functions are consistent, while the latter addresses whether requirements accurately reflect users' needs. Verification of structural properties for large and complex software requirements is often repetitive, especially if requirements are subject to frequent changes. While inspection has been successfully applied to many industrial applications, the authors found inspection to be ineffective when reviewing requirements to find errors violating structural properties. Moreover, current tools used in requirements engineering provide only limited support in automatically enforcing structural correctness of the requirements. Such experience has motivated research to automate straightforward but tedious activities. This paper demonstrates that a theorem prover, PVS (Prototype Verification System), is useful in automatically verifying structural correctness of software requirements specifications written in SCR (Software Cost Reduction)-style. Requirements are automatically translated into a semantically equivalent PVS specification. Users need not be experts in formal methods or power users of PVS. Structural properties to be proved are expressed in PVS theorems, and the PVS proof commands are used to carry out the proof automatically. Since these properties are application independent, the same verification procedure can be applied to requirements of various software systems. Copyright © 2001 John Wiley & Sons, Ltd.

Abstract: A new style of formal methods course is described, based on a pragmatic approach that emphasizes testing. The course introduces students to formal specification using Z, and shows how formal specification and testing can benefit each other, in both the validation and verification phases. It uses a tools-based approach, with practical work that reinforces formal specification techniques as well as traditional software engineering skills, such as unit and system testing, inspection and defensive programming with assertions. The two main results are to identify several practical uses of formal specifications that are not widely practised or taught, and to demonstrate that teaching them results in a more interesting and relevant formal methods course. Copyright © 2001 John Wiley & Sons, Ltd.

Abstract: This paper describes a rigorous method that investigates the suitability of formal specifications written in Object-Z specification language for testing object-oriented software implementation in a black-box fashion. The insight gained in the formalization of a model, the inherent abstractions, and formally specified intended behaviours and exceptions lead to the generation of test templates that are free from any implementation bias. The method described in this paper is an extension of the one proposed by Stocks and Carrington. In particular, the focus of the paper is on generating test templates for composite operations in an Object-Z specification. The method is illustrated using the specification for an electronic mail system. The specification and the test templates generated for the electronic mail system show several interesting properties of the application that require considerable attention during testing. Copyright © 2001 John Wiley & Sons, Ltd.

Abstract: The last issue of STVR, 10(4), was a special issue dedicated to specification-based testing and edited by Rob Hierons and John Derrick. Three papers were included in that issue, one by Edwards providing component testing based on flowgraph sequencing information, a second paper by Wimmel et al., indicating how testing can be determined from a state-based specification, and a third by Hong et al., describing how tests can be systematically generated from a statechart, in effect a special form of state-based specification. The interesting observation is that both of the papers in this issue of STVR, 11(1), also involve specification-based testing. For example, the first paper in this issue, by Periyasamy and Alagar, ‘A rigorous method for test templates generation from object-oriented specifications’, investigates specifications written in Object-Z to generate specification-based tests. Object-Z is an extension of the Z notation that preserves the basic syntax and semantics of Z, but provides additional syntax for features such as inheritance, encapsulation and polymorphism. The primary emphasis of the paper is to generate test templates and oracles for composite operations in an Object-Z specification. A composite operationin Object-Z is specified using the names of operations that are composed using unnamed schemas. These unnamed schemas may introduce additional variables that are subject to scope rules, thus imposing additional constraints on the composed operations. Note that as a result, the input and output spaces are not explicit from the syntax of the composite operation. The correctness of the proposed method is informally justified based upon the semantics of the composite operators in Object-Z. The authors have applied the method to an electronic mail system example. Challenges for future research are to establish scalability and to automate the method with one or more tools. The second paper is written by Bogdanov and Holcombe, ‘Statechart testing method for aircraft control systems’, and could be considered a companion paper to the one on statecharts by Hong et al. that was included in the special STVRissue10(4). In the Bogdanov and Holcombe paper, the authors describe a modification of a formal testing method for extended finite state machines (FSMs) to allow the testing of a system based upon statechart specifications. This research was motivated by the fact that a number of concurrent control systems for aircraft have been specified using statecharts. It is interesting that, as a starting point to test simple statecharts, the authors reference and utilize Chow’s W method [ 1]. Recently, this Editor used an FSM to partially model a GUI for testing, and Chow’s W method was rediscovered for use in testing this FSM model. Readers unfamiliar with the Chow paper might want to check on it for the solution to a commonly occurring problem in testing. Bogdanov and Holcombe have applied their testing method to statechart specifications of part of an autopilot as a case

Abstract: For September 2001, STVR reverts to being a regular issue, after the previous one which was devoted to a collection of papers from the WAPATV 2000 workshop. In fact, two out of the past three issues have been special. Readers will recall that STVR 10(4) in December 2000 was a themed issue on specification-based testing. As it happens, STVR 11(1) in March 2001 also contained papers on that topic. The three papers in this issue seem to be linked by the theme of formal specifications also. In passing, another link is noted and that is a geographic one. The three sets of authors all come from Pacific Rim countries: Korea, Australia and New Zealand. It is good to see that STVR is achieving a truly global dimension! Kim and Cha, in their paper, describe how they built a tool to take requirements specifications in the SCR (Software Cost Reduction) style and translate them into equivalent PVS (Prototype Verification System) specifications. Structural properties concerning the inputs, the outputs and the consistency of the data flows were expressed as PVS theorems which were checked automatically by the PVS theorem prover. The method has been used with success on part of the emergency shutdown system of the Wolsung nuclear power plant in Korea. The structural properties considered were generic, so the same procedure could be applied without alteration to the requirements of other systems. In the future the authors hope to handle application-specific safety properties, provided they can be expressed as PVS theorems. The paper by Chen and Lau considers specifications in the form of Boolean expressions. Although the format might sound straightforward, especially by comparison with more specialist notations, it is not uncommon for a large number of variables to be involved, making exhaustive testing impossible. Certainly the motivation for considering such specifications is firmly rooted in the real world of avionics or avionics-related systems. For example, an earlier study by Weyuker et al. [1] represented the conditions for state transitions in the statechart specification of the Traffic alert and Collision Avoidance System II (TCAS II) as Boolean formulae. In this issue, Chen and Lau adopt an approach where test cases are generated or selected with the aim of detecting particular types of faults. Three strategies are proposed which, when taken in combination, are more cost-effective than the most powerful strategy in the family of ‘meaningful impact’ strategies previously proposed by Weyuker et al. for such specifications. The paper by Utting and Reeves will be of particular interest to university educators. It describes their experiences in presenting a formal methods course that was modified to emphasize, amongst

Abstract: The verification and validation of software through dynamic testing is an area of software engineering where progress towards automation has been slow. In particular the automatic design and generation of test data remains, by and large, a manual activity. This is despite the high promises that the symbolic execution technique engendered when it was first proposed as a method for automatic test data generation. This paper presents an automatic test data generator based on constraint logic programming and symbolic execution. After reviewing the symbolic execution technique, approaches for the resolution of the technical difficulties that have so far prevented symbolic execution from reaching its full potential in the area of automatic test data generation are presented. ATGen, an automatic test data generator, based on symbolic execution and that uses constraint logic programming, is then discussed. Copyright © 2001 John Wiley & Sons, Ltd.

Abstract: The mutation method assesses test quality by examining the ability of a test set to distinguish syntactic deviations representing specific types of faults from the program under test. This paper describes an empirical study performed to evaluate the effectiveness of object-oriented (OO) test strategies using the mutation method. The test sets for the experimental system are generated according to three selected OO test methods and their effectiveness is compared by determining how well the developed test sets kill injected mutants derived from an established mutation system Mothra, and our own OO-specific mutation technique which is termed Class Mutation.

Abstract: This paper outlines a general strategy for automated black-box testing of software components that includes: automatic generation of component test drivers, automatic generation of black-box test data, and automatic or semi-automatic generation of component wrappers that serve as test oracles. This research in progress unifies several threads of testing research, and preliminary work indicates that practical levels of testing automation are possible.

Abstract: Mutation testing (MT) has been found to be effective at revealing faults. However, its high cost of application, due to the high number of mutants created and the effort to determine the equivalent ones, has motivated the proposition of alternative approaches for its application. One of them, named ‘selective mutation’, aims to reduce the number of generated mutants through a reduction in the number of mutant operators. A previous relevant study resulted in the proposition of a sufficient mutant operators set for FORTRAN, indicating that it is possible to have a large cost reduction in MT application, whilst preserving a high MT score. This work investigates procedures for the determination of a sufficient mutant operators set for C programs with the perspective of contributing to the establishment of low-cost, effective mutation-based testing strategies. Copyright © 2001 John Wiley & Sons, Ltd.