This paper reports the results of a study comparing the effectiveness of automatically generated tests constructed using random and t-way combinatorial techniques on safety related industrial code using mutation adequacy criteria. A reference point is provided by hand generated test vectors constructed during development to establish minimum acceptance criteria. The study shows that 2-way testing is not adequate measured by mutants kill rate compared with hand generated test set of similar size, but that higher factor t-way test sets can perform at least as well. To reduce the computation overhead of testing large numbers of vectors over large numbers of mutants a staged optimising approach to applying t-way tests is proposed and evaluated which shows improvements in execution time and final test set size.

There is a perceived lack of mutation tools for the C language and information about the tools that are available was not widely disseminated until recently. This has lead to the development of a small set of programs to support the use of mutation on C language programs as part of a research effort into the use of automatically generated test sets. This paper describes a lightweight C language mutation tool and some initial results from its application to sorting algorithms. The paper describes how kill rates could be improved using a different encoding for Boolean values and details how the use of CPU time can be used as an additional means of detecting mutants.