This paper reports the results of a study comparing the effectiveness of automatically generated tests constructed using random and t-way combinatorial techniques on safety related industrial code using mutation adequacy criteria. A reference point is provided by hand generated test vectors constructed during development to establish minimum acceptance criteria. The study shows that 2-way testing is not adequate measured by mutants kill rate compared with hand generated test set of similar size, but that higher factor t-way test sets can perform at least as well. To reduce the computation overhead of testing large numbers of vectors over large numbers of mutants a staged optimising approach to applying t-way tests is proposed and evaluated which shows improvements in execution time and final test set size.

Web services present a new promising software technology. However, some new issues and challenges in testing of them come out due to their characteristics of distribution, source code invisibility etc. This paper discusses the traditional mutation testing and then a new methodology of OWL-S requirement model-based web service mutation testing is brought forward. The traits of this methodology are as follows. Firstly, requirements are used effectively to reduce the magnitude of mutants. Secondly, mutants are generated by AOP technology conveniently and promptly. Thirdly, to reducing testing cost, using business logic implied in OWL-S requirement model as assistant of the process of killing the mutants. Fourthly, two sufficient measurement criteria are employed to evaluate the testing process. Finally, our empirical results have shown the usefulness of this testing method.

Available soon...

Mutation testing is a fault-based testing technique used to inject faults into an existing program and see if its test suite is sensitive enough to detect common faults. We are interested in using the mutation analysis to evaluate, compare and improve quality assurance techniques for testing object-oriented mechanisms and other advanced features of C# programs. This paper provides an overview of a current version of the CREAM system (CREAtor of Mutants), and reports on its use in experimental research. We apply advanced, object-oriented mutation operators to testing of open-source C# programs and discuss the results.

Available soon...

The critical requirement on reliability, fault-tolerance and security of network devices highlights the necessity of protocol robustness testing. Mutation testing of protocol messages is an important part of robustness testing, but related theory and practices are not well developed. This paper builds a NFSM model for mutation testing of protocol messages and proposes two types of Normal-Verification Sequence to enhance verdict mechanism. For single-field mutation testing of protocol messages, we propose the concept of compound anomalous test case to further simplify test sequences. As a standard test specification language, TTCN-3 reveals strong excellence in conformance testing, so we apply TTCN-3 to mutation testing and extend it according to test requirements. Using our method we test OSPFv2 sufficiently with a test system based on extended TTCN-3. The results indicate that our method has good capability of error-finding.

Most commercial software testing tools used in industry do not automatically generate test data even though many testing strategies have been proposed in academia. In this paper, we introduce a prototype software testing platform that we developed to automatically generate test data from tabular specifications. This method of documenting software specifications has been used successfully in industry to help produce reliable software. This platform can also be used to measure the effectiveness of new testing strategies.

We present a model-based approach to testing access control requirements. By using combinatorial testing, we first automatically generate test cases from and without access control policies—i.e., the model—and assess the effectiveness of the test suites by means of mutation testing. We also compare them to purely random tests. For some of the investigated strategies, non-random tests kill considerably more mutants thanthe same number of random tests. Since we rely on policies only, no information on the application is required at this stage. As a consequence, our methodology applies to arbitrary implementations of the policy decision points.

Traditional mutation testing considers only first order mutants, created by the injection of a single fault. Often these first order mutants denote trivial faults that are easily killed. This paper investigates higher order mutants (HOMs). It introduces the concept of a subsuming HOM; one that is harder to kill than the first order mutants from which it is constructed. By definition, subsuming HOMs denote subtle fault combinations. The paper reports the results of an empirical study into subsuming HOMs, using six benchmark programs. This is the largest study of mutation testing to date. To overcome the exponential explosion in the number of mutants considered, the paper introduces a search based approach to the identification of subsuming HOMs. Results are presented for a greedy algorithm, a genetic algorithm and a hill climbing algorithm.

This paper introduces MILU, a C mutation testing tool designed for both first order and higher order mutation testing. All previous mutation testing tools apply all possible mutation operators to the program under test. By contrast, MILU allows customization of the set of mutation operators to be applied. To reduce runtime cost, MILU uses a novel 'test harness' technique to embed mutants and their associated test sets into a single-invocation procedure.