Conformance testing

Abstract: With advanced computer technology, systems are getting larger to fulfill more complicated tasks: however, they are also becoming less reliable. Consequently, testing is an indispensable part of system design and implementation; yet it has proved to be a formidable task for complex systems. This motivates the study of testing finite stare machines to ensure the correct functioning of systems and to discover aspects of their behavior. A finite state machine contains a finite number of states and produces outputs on state transitions after receiving inputs. Finite state machines are widely used to model systems in diverse areas, including sequential circuits, certain types of programs, and, more recently, communication protocols. In a testing problem we have a machine about which we lack some information; we would like to deduce this information by providing a sequence of inputs to the machine and observing the outputs produced. Because of its practical importance and theoretical interest, the problem of testing finite state machines has been studied in different areas and at various times. The earliest published literature on this topic dates back to the 1950's. Activities in the 1960's mid early 1970's were motivated mainly by automata theory and sequential circuit testing. The area seemed to have mostly died down until a few years ago when the testing problem was resurrected and is now being studied anew due to its applications to conformance testing of communication protocols. While some old problems which had been open for decades were resolved recently, new concepts and more intriguing problems from new applications emerge. We review the fundamental problems in testing finite state machines and techniques for solving these problems, tracing progress in the area from its inception to the present and the stare of the art. In addition, we discuss extensions of finite state machines and some other topics related to testing.

Abstract: Testing of Finite State Machines.- I. Testing of Finite State Machines.- 1 Homing and Synchronizing Sequences.- 2 State Identification.- 3 State Verification.- 4 Conformance Testing.- II. Testing of Labeled Transition Systems.- Testing of Labeled Transition Systems.- 5 Preorder Relations.- 6 Test Generation Algorithms Based on Preorder Relations.- 7 I/O-automata Based Testing.- 8 Test Derivation from Timed Automata.- 9 Testing Theory for Probabilistic Systems.- III. Model-Based Test Case Generation.- Model-Based Test Case Generation.- 10 Methodological Issues in Model-Based Testing.- 11 Evaluating Coverage Based Testing.- 12 Technology of Test-Case Generation.- 13 Real-Time and Hybrid Systems Testing.- IV. Tools and Case Studies.- Tools and Case Studies.- 14 Tools for Test Case Generation.- 15 Case Studies.- V. Standardized Test Notation and Execution Architecture.- Standardized Test Notation and Execution Architecture.- 16 TTCN-3.- 17 UML 2.0 Testing Profile.- VI. Beyond Testing.- Beyond Testing.- 18 Run-Time Verification.- 19 Model Checking.- VII. Appendices.- Appendices.- 20 Model-Based Testing - A Glossary.- 21 Finite State Machines.- 22 Labelled Transition Systems.

Abstract: A method for generating test sequences for checking the conformance of a protocol implementation to its specification is described. A rural Chinese postman tour problem algorithm is used to determine a minimum-cost tour of the transition graph of a finite-state machine. It is shown that, when the unique input/output sequence (UIO) is used in place of the more cumbersome distinguishing sequence, both the controllability and observability problems of the protocol testing problem are addressed, providing an efficient method for computing a test sequence for protocol conformance testing. >

Abstract: The authors present a detailed study of four formal methods (T-, U-, D-, and W-methods) for generating test sequences for protocols. Applications of these methods to the NBS Class 4 Transport Protocol are discussed. An estimation of fault coverage of four protocol-test-sequence generation techniques using Monte Carlo simulation is also presented. The ability of a test sequence to decide whether a protocol implementation conforms to its specification heavily relies on the range of faults that it can capture. Conformance is defined at two levels, namely, weak and strong conformance. This study shows that a test sequence produced by T-method has a poor fault detection capability, whereas test sequences produced by U-, D-, and W-methods have comparable (superior to that for T-method) fault coverage on several classes of randomly generated machines used in this study. Also, some problems with a straightforward application of the four protocol-test-sequence generation methods to real-world communication protocols are pointed out. >