Robustness testing

Abstract: JCrasher is an automatic robustness testing tool for Java code. JCrasher examines the type information of a set of Java classes and constructs code fragments that will create instances of different types to test the behavior of public methods under random data. JCrasher attempts to detect bugs by causing the program under test to 'crash', that is, to throw an undeclared runtime exception. Although in general the random testing approach has many limitations, it also has the advantage of being completely automatic: no supervision is required except for off-line inspection of the test cases that have caused a crash. Compared to other similar commercial and research tools, JCrasher offers several novelties: it transitively analyzes methods, determines the size of each tested method's parameter-space and selects parameter combinations and therefore test cases at random, taking into account the time allocated for testing; it defines heuristics for determining whether a Java exception should be considered as a program bug or whether the JCrasher supplied inputs have violated the code's preconditions; it includes support for efficiently undoing all the state changes introduced by previous tests; it produces test files for JUnit, a popular Java testing tool; and it can be integrated in the Eclipse IDE.

Abstract: INTRODUCTION TO CODE VERIFICATION THE MATHEMATICAL MODEL AND NUMERICAL ALGORITHM The Mathematical Model Numerical Methods for Solving Differential Equations THE CODE VERIFICATION PROCEDURE Static Testing Dynamic Testing Overview of the Order-Verification Procedure Details of the Procedure Closing Remarks DESIGN OF COVERAGE TEST SUITE Basic Design Issues Coverage Issues Related to Boundary Conditions Coverage Issues Related to Grids and Grid Refinement FINDING EXACT SOLUTIONS Obtaining Exact Solutions from the Forward Problem The Method of Manufactured Exact Solutions BENEFITS OF THE ORDER-VERIFICATION PROCEDURE A Taxonomy of Coding Mistakes A Simple PDE Code Blind Tests RELATED CODE DEVELOPMENT ACTIVITIES Numerical Algorithm Development Testing for Code Robustness Testing for Code Efficiency Code Confirmation Exercises Solution Verification Code Validation Software Quality Engineering SAMPLE CODE VERIFICATION EXERCISES Burgers' Equation in Cartesian Coordinates (Code 1) Burgers' Equation in Curvilinear Coordinates (Code 2) Incompressible Navier-Stokes (Code 3) Compressible Navier-Stokes (Code 4) ADVANCED TOPICS Computer Platforms Look-up Tables Automatic Time-Stepping Options Hard-Wired Boundary Conditions Codes with Artificial Dissipation Terms Eigenvalue Problems Solution Uniqueness Solution Smoothness Codes with Shock Capturing Schemes Dealing with Codes that Make Non-Ordered Approximations SUMMARY AND CONCLUSIONS REFERENCES APPENDICES Other Methods for PDE Code Testing Implementation Issues in the Forward Approach Results of Blind Tests A Manufactured Solution to the Free-Surface Porous Media Equations INDEX

Abstract: Mission-critical system designers may have to use a commercial off-the-shelf (COTS) approach to reduce costs and shorten development time, even though COTS software components may not specifically be designed for robust operation. Automated testing can assess component robustness without sacrificing the advantages of a COTS approach. This paper describes the Ballista methodology for scalable, portable, automated robustness testing of component interfaces. An object-oriented approach based on parameter data types rather than component functionality essentially eliminates the need for function-specific test scaffolding. A full-scale implementation that automatically tests the robustness of 233 operating system software components has been ported to ten POSIX systems. Between 42% and 63% of components tested had robustness problems, with a normalized failure rate ranging from 10% to 23% of tests conducted. Robustness testing could be used by developers to measure and improve robustness, or by consumers to compare the robustness of competing COTS component libraries.