Abstract: We know software evolution to be inevitable if the system is to survive in the long-term. Equally well-understood is the necessity of having a good test suite available in order to (1) ensure the quality of the current state of the software system and (2) to ease future change. In that light, this chapter explores the interplay that exists between software testing and software evolution, because as tests ease software evolution by offering a safety net against unwanted change, they can equally be experienced as a burden because they are subject to the very same forces of software evolution themselves.In particular, in this chapter, we describe how typical refactorings of production code can invalidate tests, how test code can (structurally) be improved by applying specialized test refactorings. Building upon these concepts, we introduce “test-driven refactoring”, or refactorings of production code that are induced by the (re)structuring of the tests. We also report on typical source code design metrics that can serve as indicators for testability. To conclude, we present a research agenda that contains pointers to—as yet—unexplored research topics in the domain of testing.

Abstract: There has been little attention to search based test data generation in the presence of pointer inputs and dynamic data structures, an area in which recent concolic methods have excelled. This paper introduces a search based testing approach which is able to handle pointers and dynamic data structures. It combines an alternating variable hill climb with a set of constraint solving rules for pointer inputs. The result is a lightweight and efficient method, as shown in the results from a case study, which compares the method to CUTE, a concolic unit testing tool.

Abstract: Since annotations were added to the Java language, many frameworks have moved to using annotated Plain Old Java Objects (POJOs) in their newest releases Legacy applications are thus forced to undergo extensive restructuring in order to migrate from old framework versions to new versions based on annotations (Version Lock-in) Additionally, because annotations are embedded in the application code, changing between framework vendors may also entail largescale manual changes (Vendor Lock-in) This paper presents a novel refactoring approach that effectively solves these two problems Our approach infers a concise set of semantics-preserving transformation rules from two versions of a single class Unlike prior approaches that detect only simple structural refactorings, our algorithm can infer general composite refactorings and is more than 97% accurate on average We demonstrate the effectiveness of our approach by automatically upgrading more than 80K lines of the unit testing code of four open-source Java applications to use the latest version of the popular JUnit testing framework

Abstract: A computer-based method for qualifying a software application such as a monitoring and/or management network-delivered product. The method includes operating a computer to run a software application with network-delivered services. A qualification computer system is provided with a code generator adapted for processing device qualification data and with a comparator module. The method includes generating a set of event messages with the code generator by processing a set of received or retrieved device qualification data. The event messages are processed with the services or serviceability features of the software application to generate a set of output. With the comparator module, the actual output is compared with expected output for the event messages. Based on the comparison, a qualification report is generated, and the expected output may be output from unit test cases. A code generator may be included that generates one or more native protocol event generators.

Abstract: This hands-on tutorial will teach the principles of Parameterized Unit Testing [5,4] with Pex [2], an automatic test input generator for .NET which performs a systematic program analysis, similar to path bounded model-checking. A parameterized unit test is simply a method that takes parameters, calls the code under test, and states assertions.

Abstract: Software libraries organize useful functionalities in order to promote modularity and code reuse. A typical library is used by client programs through an application programming interface (API) that hides its internals from the client. Typically, the rules governing the correct usage of the API are documented informally. In many cases, libraries may have complex API usage rules and unclear documentation. As a result, the behaviour of the library under some corner cases may not be well understood by the programmer. Formal specifications provide a precise understanding of the API behaviour. We propose a methodology for learning interface specifications using Inductive Logic Programming (ILP). Our technique runs several unit tests on the library in order to generate relations describing the operation of the library. The data collected from these tests are used by an inductive learner to obtain rich Datalog/Prolog specifications. Such specifications capture essential properties of interest to the user. They may be used for applications such as reverse engineering the library internals or constructing checks on the application code to enforce proper API usage along with other properties of interest.

Abstract: A test data creation and execution system provides an efficient way to dramatically improve testing of web service based applications. The test data creation and execution system assists testers in identifying and executing various combinations of unit tests, as well as integration and regression testing of web services and methods of web services. The test data creation and execution system provides testers the functionality to rapidly produce high quality web services based applications.

Abstract: The present invention discloses a solution for adding a profiling agent to a virtual machine (VM), which permits unit test programs to access information relating to memory and performance from runtime data areas of the VM. Operations permitted by the agent can include starting/stopping VM monitor, getting objects from the heap, retrieving an invocation count, determining a size of an instantiated object, getting a current thread time, and the like. Memory and performance information gathered during a test can be stored for later analysis under a test version identifier. A comparison engine can create reports that use the stored performance and memory data, which can help developers compare memory/performance deltas among different unit versions. In one embodiment, the VM can be a JAVA VIRTUAL MACHINE (JVM) and the unit test programs can be JUNIT programs.

Abstract: Tools like JUnit and its relatives are making software testing reachable even for introductory students. At the same time, however, many introductory computer sciences courses use graphical interfaces as an "attention grabber" for students and as a metaphor for teaching object-oriented programming. Unfortunately, developing software tests for programs that have significant graphical user interfaces is beyond the abilities of typical students (and, for that matter, many educators). This paper describes a framework for combining readily available tools to create an infrastructure for writing tests for Java programs that have graphical user interfaces. These tests are level-appropriate for introductory students and fit in with current approaches in computer science education that incorporate testing in programming assignments. An analysis of data collected during actual student use of the framework in a CS1 course is presented.

Abstract: Marking programming assignments in programming courses involves a lot of work: each program must be tested, the source code must be read and evaluated, etc. With the large classes encountered nowadays, the feedback provided to students through marking is thus rather limited, and often late. Tools providing support for marking programming assignments do exist, ranging from support for administrative aspects through automation of program testing or support for source code evaluation based on metrics. In this paper, we introduce a tool, called Oto, that provides support for submission and marking of assignments. Oto aims at reducing the workload associated with the marking task. Oto also aims at providing timely feedback to the students, including feedback before the final submission. Furthermore, the tool has been designed to be generic and extensible, so that the marking process for a specific assignment can easily be customized and the tool can be extended with various marking components (modules) that allows it to deal with various aspects of marking (testing, style, structure, etc.) and with programs written in various programming languages. Copyright © 2007 John Wiley & Sons, Ltd.

Abstract: Functional testing of web applications has become increasingly complex. Browser-based interfaces incorporate rich, client-side scripting that is increasingly independent of server-side application logic. At the same time, the server side application logic interacts with reusable components for key elements (security, shopping cart, product catalog, order processing) within the framework of a component-based architecture, using beans and web services. In this paper, we illustrate how a test specification approach using a language like TTCN-3 can be used to define test cases at different levels of abstraction that are more robust in the face of volatile presentation and implementation details. A case study of a shopping cart scenario with order processing is used as an illustration. Features of TTCN-3 are demonstrated, including a powerful matching mechanism that allows a separation between behavior and the conditions governing behavior. As well, TTCN-3’s data types and set-based operations allow one to track and verify the information management done by a web application, independent of implementation details. These features allow a tester to take a systematic approach to testing web applications but requires more sophistication and skills. The advantages and challenges of a test specification approach are characterized in comparison to approaches based on unit testing and test automation tools.

Abstract: A test system (and corresponding method and computer program product) for generating unit tests for a heterogeneous network system and validating test results to ensure that the network system functions properly is described. In one embodiment, the test system is an appliance that is capable of normalizing communication protocols supported by component systems of the network system. The test system creates objects and methods corresponding to component systems and their supported protocol commands in the network system, and generates unit test cases based on the objects, the methods, and the normalized protocols. The test system transmits the unit test cases to the component systems, receives test results, and validates the test results to ensure that the network system functions properly.

Abstract: This paper presents a novel and innovative pedagogical approach for teaching software quality assurance in the undergraduate computer science curriculum. The approach is based on students contributing programming problems to an open source web-based system that is used for student practice and instructor assessment of assignments. WeBWorK, and some of the latest web-based systems, use a mechanism based on unit testing to account for variation in the way in which the same problem can be answered in an accurate manner, making such systems highly appealing for education. Tackling open-ended programming problems within WeBWorK therefore requires students to write a code fragment that is then checked for semantic correctness. Given that WeBWorK is open source, the teaching approach that we have evolved revolves around students creating their own problems for other students to practice with. This requires students to construct comprehensive unit tests that can assure both the usability and accuracy of their work prior to deployment. The paper describes this approach, gives examples of student work, presents findings from the experience of using the approach in the classroom, and discusses broader lessons and reasons for integrating software quality assurance practices into the computer science curriculum.

Abstract: Problem statement: Program testing is expensive and labor intensive, often consuming more than half of the total development costs, and yet it is frequently not done well and the results are not always satisfactory. The objective of this paper is to present an automatic test data generation tool that aims to completely automate unit testing of JavaScript functions. The methodology: In order to use the proposed tool, the tester annotates the files that contain the class to be tested. Moreover, the tester must specify the test data coverage criterion to be used, either branch coverage or mutation analysis. However, the tool is then integrated into the JavaScript compiler and test generation is invoked by a command line option. Also, the code to be tested is parsed into an abstract syntax tree from which the test tool generates a program dependency graph for the function under test. However, if mutation analysis coverage is required, the abstract syntax tree for a meta-mutant program is also generated. To provide guidance for the test data search, the function under test instrumented in accordance with the coverage criterion. Branch predicate expressions are always instrumented, in the case of mutation coverage, mutated statements are also instrumented. Compilation then continues from the modified abstract syntax tree to generate instrumented executables that were loaded into the test data search module. Results: The experiment done in our study by using the proposed tool for branch coverage shows that the most effective result for string equality was obtained using the edit distance fitness function, while no significant difference was found in the fitness function for string ordering. Through exhaustive mulation coverage 8% are found to be equivalent. Conclusion: By having a complete automation it reduces the cost of software testing dramatically and also facilitates continuous testing. It is reported that at least 50% of the total software development costs is due to testing, and 10-15% of development time is wasted due to frequent stops for regression testing. Automation will also help get rid of cognitive biases that have been found in human testers. Acknowledgment: The researcher would like to express their gratitude to the anonymous referees for their valuable and helpful comments and suggestions in improving the study.

Abstract: The evidence regarding code coverage as a predictor of software quality is conflicting and inconclusive. However, an estimate of the software testing practices of the majority of professionals can help researchers know how code coverage is being used—or whether it is being used at all. The purpose of this report is to present the results of an online survey we conducted to estimate the percentage of software developers who use code coverage as a stopping criterion for unit testing. We find that a majority of participants 1) perform automated unit testing; and 2) use code coverage, though not always as a stopping criterion. Those people who do not use code coverage, do not find it useful or provide some other reason. Finally, in place of code coverage, we find that most participants stop testing when they have “tested the most important parts of the code”.

Abstract: This paper presents an automatic trace-based unit testing approach to test object-oriented programs. Most automated testing tools test a class C by testing each of its methods in isolation. Such an approach works poorly if specifications are only partial, which is usually the case in practice. In contrast, our approach generates sequences of calls to the methods of C that are checked on-the-fly. This is more interactive, and has the side effect that methods are checking each other. Although simple, it seems to work quite well, even when specifications are only partially provided. We implement the approach in a tool called T2. It targets Java. It can test internal errors, Hoare triple specifications, class invariant, and even temporal properties. Furthermore, T2 accepts 'in-code' specifications, these are specifications written in the specified class itself, and are written in plain Java; hence reducing the cost usually needed to maintain specifications to minimum.

Abstract: A genetic based technique is presented for solving the unit commitment optimization problem. The proposed technique consists mainly of two stages. In the first stage, economic dispatch for each interval (hour) of study is executed. Several solutions (individuals) are generated around the previous economic dispatch solution. These individuals are introduced as a part of the initial population of the genetic algorithm which is applied as a second stage to optimally identify the solution of the unit commitment optimization problem. The proposed technique is applied to the 10 unit, and the 26 unit test systems.

Abstract: Test Driven Development (TDD) is an agile method that emphasizes writing tests before writing code as a means of 1) assuring the satisfaction of customer requirements, and 2) reinforcing good design habits. While the first objective is usually accomplished by acceptance tests, the second objective is achieved by unit tests. The tests also serve as a multilevel cohesive reference of the system specifications. We propose the use of this referencing mechanism – test artifacts – to establish and manage agile product lines. In this paper, we delve into some of the issues that need to be tackled before test artifacts are relied on as a driving force for reuse in product lines. These issues include establishing a framework for reuse, tests comparability, test traceability, test refactoring and test versioning. We also discuss the suitability of acceptance tests and unit tests as reusable artifacts, and we present a preliminary study to analyze their utilization.

Abstract: The method comprises: modifying the computer program according to running problem capable of happening in the one or more target program units concerned by the computer program; testing and executing the modified program; according to the running problem occurring in the execution of the target program unit, automatically generating a unit test case. Said modification step comprises: adding a capture code and problem code into the program; the capture code is used for recording the execution path and the execution context of the target problem unit; said problem code is used for detecting the running problem in the process of executing said target program unit. The invention also provides a method and a system which uses said method and system to make debugging and regression testing.

Abstract: In agile software development, test code can considerably contribute to the overall source code size. Being a valuable asset both in terms of verification and documentation, the composition of a test suite needs to be well understood in order to identify opportunities as well as weaknesses for further evolution. In this paper, we argue that the visualization of structural characteristics is a viable means to support the exploration of test suites. Thanks to general agreement on a limited set of key test design principles, such visualizations are relatively easy to interpret. In particular, we present visualizations that support testers in (i) locating test cases; (ii) examining the relation between test code and production code; and (iii) studying the composition of and dependencies within test cases. By means of two case studies, we demonstrate how visual patterns help to identify key test suite characteristics.

Abstract: Flash memory has become virtually indispensable in most mobile devices. In order for mobile devices to operate successfully, it is essential that the flash memory be controlled correctly through the device driver software. However, as is typical for embedded software, conventional testing methods often fail to detect hidden flaws in the complex device driver software. This deficiency incurs significant development and operation overheads to the manufacturers. Model checking techniques have been proposed to compensate for the weaknesses of conventional testing methods through exhaustive analyses. These techniques, however, require significant manual efforts to create an abstract target model and, thus, are not widely applied in industry. In this project, we applied a model checking technique based on a Boolean satisfiability (SAT) solver. One advantage of SAT-based model checking is that a target C code can be analyzed directly without an abstract model, thereby enabling automated and bit-level accurate verification. In this project, we have applied CBMC, a SAT-based software model checker, to the unit testing of the Samsung OneNANDtrade device driver. Through this project, we detected several bugs that had not been discovered previously.

Abstract: Test coverage is traditionally considered as how much of the code is covered by the test suite in whole. However, test suites typically contain different types of tests with different roles, such as unit tests, integration tests and functional tests. As traditional measures of test coverage make no distinction between the different types of tests, the overall view of test coverage is limited to what is covered by the tests in general. This paper proposes a quantitative way to measure the test coverage of the different parts of the software at different testing levels. It is also shown how this information can be used in software maintenance and development to further evolve the test suite and the system under test. The technique is applied to an open-source project to show its application in practice. Copyright © 2007 John Wiley & Sons, Ltd.

Abstract: A model management workflow typically consists of a number of interdependent model-management tasks such as model validation, transformation, merging, comparison and text generation. As such workflows are becoming more and more complex, errors in individual tasks are likely to be propagated to subsequent tasks and eventually compromise the operation of the entire workflow. Therefore, building confidence about the correctness of each individual task before integrating them is becoming increasingly important. In this paper we present an approach to uniform unit-testing of individual model management operations and a prototype for specifying and executing test-cases, implemented atop the infrastructure provided by the Epsilon model management platform.

Abstract: This paper tackles the problem of structural integration testing of stateful classes. Previous work on structural testing of objectoriented software exploits data flow analysis to derive test requirements for class testing and defines contextual def-use associations to characterize inter-method relations. Non-contextual data flow testing of classes works well for unit testing, but not for integration testing, since it misses definitions and uses when properly encapsulated. Contextual data flow analysis approaches investigated so far either do not focus on state dependent behavior, or have limited applicability due to high complexity. This paper proposes an efficient structural technique based on contextual data flow analysis to test state-dependent behavior of classes that aggregate other classes as part of their state.

Abstract: This paper describes the design of the unit testing framework JUnit v3.8. The documentation technique employed is an enhanced version of collaboration-based design, also known as role modeling. In collaboration-based design, objects are viewed as playing multiple roles in different contexts, and different contexts are viewed as task specific collaborations. The documentation accounts for every method in the JUnit 3.8 framework by assigning it to a role. It thereby investigates whether roles and collaborations can serve as basic units of functionality provided by a design like a framework. Such a measure of functionality can serve multiple purposes, for example estimating implementation efforts or measuring complexity.

Abstract: Writing unit tests for legacy systems is a key maintenance task. When writing tests for object-oriented programs, objects need to be set up and the expected effects of executing the unit under test need to be verified. If developers lack internal knowledge of a system, the task of writing tests is non-trivial. To address this problem, we propose an approach that exposes side effects detected in example runs of the system and uses these side effects to guide the developer when writing tests. We introduce a visualization called Test Blueprint, through which we identify what the required fixture is and what assertions are needed to verify the correct behavior of a unit under test. The dynamic analysis technique that underlies our approach is based on both tracing method executions and on tracking the flow of objects at runtime. To demonstrate the usefulness of our approach we present results from two case studies.

Abstract: Pursuant to some embodiments, a platform matching system, method, apparatus and means are provided which include identifying at least a first benchmark test procedure to evaluate at least a first and a second target platform for use with an application, the at least first benchmark test procedure including at least a first unit test; receiving test result data from tests performed on the at least first and second target platform using the at least first benchmark test procedure; and performing a matching analysis to determine which of the at least first and second target platform is a best fit for the application.