This paper presents an empirical study of the mutation techniques used by the tool PASTE. This tool allows the automation of our passive testing methodology for systems that present stochastic-time information. In our proposal, invariants express the fact that each time the implementation under test performs a given sequence of actions, then it must exhibit a behavior according to the probability distribution functions reflected in the invariant. We briefly review the theoretical framework of our methodology and the main features of our tool. Next, we present in detail the Mutants module that provides us with a functionality to test the effectiveness of invariants for detecting errors. Finally, we present a study of the results obtained from the performed experiments.

Available soon...

Available soon...

If a mutation is not killed by a test suite, this usuallymeans that the test suite is not adequate. However, itmay also be that the mutant keeps the program’s seman-tics unchanged—and thus cannot be detected by any test.We found such equivalent mutants to be surprisingly com-mon: In an experiment on the JAXEN XPATH query engine,8/20 = 40% of all mutations turned out to be equivalent.Worse, checking the equivalency took us 15 minutes for asingle mutation. Equivalent mutants thus make it impossi-ble to automatically assess test suites by means of mutationtesting. To identify equivalent mutants, we are currently investi-gating the impact of a mutation on the execution: the morea mutation alters the execution, the higher the chance of itbeing non-equivalent. First experiments assessing the im-pact on code coverage are promising.

Available soon...

Aspect-oriented programming introduces new challenges for software testing. Inparticular the pointcut descriptor (PCD) requires particular attention fromtesters. The PCD describes the set of joinpoints where the advices are woven.In this paper we present a tool, AjMutator, for the mutation analysis of PCDs.AjMutator implements several mutation operators that introduce faults in thePCDs to generate a set of mutants. AjMutator classifies the mutants accordingto the set of joinpoints they match compared to the set of joinpoints matchedby the initial PCD. An interesting result is that this automaticclassification can identify equivalent mutants for a particular class of PCDs.AjMutator can also run a set of test cases on the mutants to give a mutationscore. We have applied AjMutator on two systems to show that this tool issuitable for the mutation analysis of PCDs on large AspectJ systems.

The rise of Web Services and their WS-BPEL compositions in recent years makes necessary to pay special attention to testing in this context. Mutation testing is a white box testing technique that has been applied successfully to programs written in different languages. In order to apply it we need a set of mutation operators and a system for mutant generation. This paper introduces a set of mutation operators for the WS-BPEL 2.0 language and a framework, based in genetic algorithms, for automatic mutant generation without rendering all possible mutants. This framework can also detect potentially equivalent mutants.

Available soon...

This paper introduces a new paradigm for Mutation Testing, which we call Higher Order Mutation Testing (HOM Testing). Traditional Mutation Testing considers only first order mutants, created by the injection of a single fault. Often these first order mutants denote trivial faults that are easily killed. Higher order mutants are created by the insertion of two or more faults. The paper introduces the concept of a subsuming HOM; one that is harder to kill than the first order mutants from which it is constructed. By definition, subsuming HOMs denote subtle fault combinations. The paper reports the results of an empirical study of HOM Testing using ten programs, including several non trivial real–world subjects for which test suites are available.

This paper presents a theory of testing that integrates into Hoare and He’s Unifying Theory of Programming (UTP). We give test cases a denotational semantics by viewing them as specification predicates. This reformulation of test cases allows for relating test cases via refinement to specifications and programs. Having such a refinement order that integrates test cases, we develop a testing theory for fault-based testing. Fault-based testing uses test data designed to demonstrate the absence of a set of pre-specified faults. A well-known fault-based technique is mutation testing. In mutation testing, first, faults are injected into a program by altering (mutating) its source code. Then, test cases that can detect these errors are designed. The assumption is that other faults will be caught, too. In this paper, we apply the mutation technique to both, specifications and programs. Using our theory of testing, two new test case generation laws for detecting injected (anticipated) faults are presented: one is based on the semantic level of UTP design predicates, the other on the algebraic properties of a small programming language.

The complex interactions appearing in service-oriented computing make coordination a key concern in service-oriented systems. In this paper, we present a fault-based method to generate test cases for component connectors from specifications. For connectors, faults are caused by possible errors during the development process, such as wrongly used channels, missing or redundant subcircuits, or circuits with wrongly constructed topology. We give test cases and connectors a unifying formal semantics by using the notion of design, and generate test cases by solving constraints obtained from the specification and faulty connectors. A prototype symbolic test case generator serves to demonstrate the automatizing of the approach.

Available soon...

Available soon...

Specification mutation involves mutating a specification, and for each mutation a test is derived that distinguishes the behaviours of the mutated and original specifications. This approach has been applied with finite state machine based models. This paper extends mutation testing to finite state machine models that contain non-functional properties. The paper describes several ways of mutating a finite state machine with probabilities (PFSM) or stochastic time (PSFSM) attached to its transitions and shows how we can generate test sequences that distinguish between such a model and its mutants. Testing then involves applying each test sequence multiple times, observing the resultant behaviours and using results from statistical sampling theory in order to compare the observed frequency and execution time of each output sequence with that expected.

Available soon...

While traditional testing can give evidence that an implementation conforms to a specification there is no guarantee that the implementation therefore also satisfies user requirements. Consequently, it is useful to derive test cases also from requirement properties. In this paper we introduce two new test criteria to measure adequacy of existing test suites and to derive test cases with regard to requirements formalized in linear temporal logic (LTL). The first criterion is complementary to existing criteria for requirement properties in that it focuses on cases where literals in a property should not affect the outcome of the property. The second criterion uses mutation to measure the sensitivity of a test suite with regard to the implemented requirements. In addition to coverage measurement these criteria can also be used to automatically create coverage adequate test suites with a model checker. The results of a case study illustrate the feasibility of the new criteria and demonstrate that they complement existing criteria.

Available soon...

Java exception mechanism can effectively free a program from abnormal exits and help developers locate faults with the exception tracing stacks. It is necessary to verify whether the exception handling constructs are arranged appropriately. Some approaches have been developed to evaluate the test sets and improve the quality of them, so that they can raise more number of exceptions in programs. Mutation analysis is a practical method to evaluate the quality of test sets. This paper presents some new mutation operators for Java exception handling constructs. Moreover, equivalent mutants can be identified by our approach. A case study illustrates the effectiveness and characteristic features of these mutation operators.

Based on 1) research into mutation testing for general-purpose programming languages and 2) spreadsheet errors that have been reported in the literature, we have developed a suite of mutation operators for spreadsheets. We present an evaluation of the mutation adequacy of definition-use adequate test suites generated by a constraint-based automatic test-case generation system we have developed in previous work. The results of the evaluation suggest additional constraints that can be incorporated into the system to target mutation adequacy. In addition to being useful in mutation testing of spreadsheets, the operators can be used in the evaluation of error-detection tools and also for seeding spreadsheets with errors for empirical studies. We describe two case studies where the suite of mutation operators helped us carry out such empirical evaluations. The main contribution of this paper is a suite of mutation operators for spreadsheets that can be used for performing empirical evaluations of spreadsheet tools to indicate ways in which the tools can be improved.

Available soon...