Abstraction in Assertion-Based Test Oracles

TL;DR: This work advocates the use of model variables - specification-only variables of which abstract values are given as mappings from concrete program states - to write abstract assertions for test oracles and performs a mutation testing experiment to evaluate the effectiveness of the useOf model variables in assertion-based test oracle assertions.

Abstract: Assertions can be used as test oracles. However, writing effective assertions of right abstraction levels is difficult because on the one hand, detailed assertions are preferred for thorough testing (i.e., to detect as many errors as possible), but on the other hand abstract assertions are preferred for readability, maintainability, and reusability. As assertions become a practical tool for testing and debugging programs, this is an important and practical problem to solve for the effective use of assertions. We advocate the use of model variables - specification-only variables of which abstract values are given as mappings from concrete program states - to write abstract assertions for test oracles. We performed a mutation testing experiment to evaluate the effectiveness of the use of model variables in assertion-based test oracles. According to our experiment, assertions written in terms of model variables are as effective as assertions written without using model variables in detecting (injected) faults, and the execution time overhead of model variables are negligible. Our findings are applicable to other use of runtime checkable assertions.