Due to increasing importance of test cases in software development, there is a need to verify and assure their quality. Mutation testing is an effective technique of checking if tests react properly to changes by introducing alterations to the original source code. A mutant which survives all test cases indicates insufficient or inappropriate testing assertions. The most onerous disadvantage of this technique is considerable time required to generate, compile mutants and then execute test cases against each of them. In the paper we propose an aspect-oriented approach to generation and execution of mutants, called response injection, which excludes the need for separate compilation of every mutant.

Fault-based testing is often advocated to overcome limitations ofother testing approaches; however it is also recognized as beingexpensive. On the other hand, evolutionary algorithms have beenproved suitable for reducing the cost of data generation in the contextof coverage based testing. In this paper, we propose a newevolutionary approach based on ant colony optimization for automatictest input data generation in the context of mutation testingto reduce the cost of such a test strategy. In our approach the antcolony optimization algorithm is enhanced by a probability densityestimation technique. We compare our proposal with otherevolutionary algorithms, e.g., Genetic Algorithm. Our preliminaryresults on JAVA testbeds show that our approach performed significantlybetter than other alternatives.

Modern software is increasingly concurrent, timed, distributed, and therefore, non-deterministic. While it is well known that tests can be generated as LTL or CTL model checker counterexamples, we argue that non-determinism creates difficulties that need to be resolved and propose test generation methods to overcome them. The proposed methods rely on fault modeling by mutation and use conventional (closed) and modular (open) model checkers.

Generation of adequate test cases is difficult and expensive, especially for testing software systems whose input is structurally complex. This paper presents an approach called data mutation to generating a large number of test data from a few seed test cases. It is inspired by mutation testing methods, but differs from them in the aim and the way that mutation operators are defined and used. While mutation testing is a method for measuring test adequacy, data mutation is a method of test case generation. In traditional mutation testing, mutation operators are used to transform the program under test. In contrast, mutation operators in our approach are applied on input data to generate test cases, hence called data mutation operators. The paper reports a case study with the method on testing an automated modelling tool to illustrate the applicability of the proposed method. Experiment data clearly demonstrate that the method is adequate and cost effective, and able to detect a large proportion of faults.

LUTESS is a test data generator dedicated to synchronous software validation. The tool produces test with respect to an environment description. To check if this description is really as expected, we use mutation analysis. The key point of the approach is to select a subset of mutants which characterizes some kind of "interesting situations" that are supposed to be often produced thanks to environment description. Intuitively, if preselected mutants are killed "very often" during tests, environment description is as expected (with respect to these "interesting situations").

Jumble is a byte code level mutation testing tool for Java which inter-operates with JUnit. It has been designed to operate in an industrial setting with large projects. Heuristics have been included to speed the checking of mutations, for example, noting which test fails for each mutation and running this first in subsequent mutation checks. Significant effort has been put into ensuring that it can test code which uses custom class loading and reflection. This requires careful attention to class path handling and coexistence with foreign class-loaders. Jumble is currently used on a continuous basis within an agile programming environment with approximately 370,000 lines of Java code under source control. This checks out project code every fifteen minutes and runs an incremental set of unit tests and mutation tests for modified classes. Jumble is being made available as open source.

Mutation analysis is usually used to provide an indication of the fault detection ability of a test set. It is mainly used for unit testing evaluation, but has also been extended for integration testing evaluation. This paper describes adaptation of mutation analysis to the Lustre programming language, including both unit and integration testing. This paper focuses on the fault model, which has been extended since our previous works. Validation of the fault model is presented.

Available soon...

Regression testing, which plays an important role in software maintenance, usually relies on test adequacy criteria to select and prioritize test cases. However, with the wide use and reuse of black-box components, such as reusable class libraries and COTS components, it is challenging to establish test adequacy criteria for testing software systems built on components whose source code is not available. Without source code or detailed documents, the misunderstanding between the system integrators and component providers has become a main factor of causing faults in component-based software. In this paper, we apply mutation on interface contracts, which can describe the rights and obligations between component users and providers, to simulate the faults that may occur in this way of software development. The mutation adequacy score for killing the mutants of interface contracts can serve as a test adequacy criterion. We performed an experimental study on three subject systems to evaluate the proposed approach together with four other existing criteria. The experimental results show that our adequacy criterion is helpful for both selecting good-quality test cases and scheduling test cases in an order of exposing faults quickly in regression testing of component-based software.

Developing correct concurrent code is more difficult than developing correct sequential code. This difficulty is due in part to the many different, possibly unexpected, executions of the program, and leads to the need for special quality assurance techniques for concurrent programs such as randomized testing and state space exploration. In this paper an approach is used that assesses testing and formal analysis tools using metrics to measure the effectiveness and efficiency of each technique at finding concurrency bugs. Using program mutation, the assessment method creates a range of faulty versions of a program and then evaluates the ability of various testing and formal analysis tools to detect these faults. The approach is implemented and automated in an experimental mutation analysis framework (ExMAn) which allows results to be more easily reproducible. To demonstrate the approach, we present the results of a comparison of testing using the IBM tool ConTest and model checking using the NASA tool Java PathFinder (JPF).

Mutation of model-checker specifications is used to generate test-cases automatically. In general, model-checker based techniques are efficient when using small, abstract models, but are limited in their applicability by the performance of the model-checker. Issues that contribute to potentially bad performance are equivalent mutants which result in no test-cases, and the fact that many different mutants lead to identical or subsumed test-cases. Another drawback of such approaches is that the size of resulting test-suites can be rather large, which might require test-suite reduction as an additional post-processing step. In this paper, we describe how the number of mutants can be minimized with special properties, which help to detect equivalent mutants more quickly and to avoid the creation of redundant test-cases. Consequently, the size of created test-suites is reduced.

A presentation of Certitude, which is a commercial software tool performing mutation analysis (MA). The product has been deployed within the microelectronics industry. Certitude is presented with specific emphasis on the product's standard processing model and feature extension mechanisms to support mutation analysis research an extension of Certitude to news languages.

In this paper, we apply our timing fault modeling strategy to writing formal specifications for communication protocols. Using the formal language of Specification and Description Language (SDL), we specify the Controller process of rail-road crossing system, a popular benchmark for real-time systems. Our extended finite state machine (EFSM) model has the capability of representing a class of timing faults, which otherwise may not be detected in an IUT. Hit-or-Jump algorithm is applied to the SDL specification based on our EFSM model to generate a test sequence that can detect these timing faults. This application of fault modeling into SDL specification ensures the synchronization among the timing constraints of different processes, and enables generation of portable test sequences since they can be easily represented in other formal notations such as TTCN or MSC.

Mutation testing is used to assess the fault-finding effectiveness of a test suite. Information provided by mutation testing can also be used to guide the creation of additional valuable tests and/or to reveal faults in the implementation code. However, concerns about the time efficiency of mutation testing may prohibit its widespread, practical use. We conducted an empirical study using the MuClipse automated mutation testing plug-in for Eclipse on the back end of a small web-based application. The first objective of our study was to categorize the behavior of the mutants generated by selected mutation operators during successive attempts to kill the mutants. The results of this categorization can be used to inform developers in their mutant operator selection to improve the efficiency and effectiveness of their mutation testing. The second outcome of our study identified patterns in the implementation code that remained untested after attempting to kill all mutants.

Along with the evolution of the technique and use of aspect-oriented programming (AOP), the difficulty of testing the aspect-oriented programs is now receiving much attention. In this position paper, we describe an AspectJ program testing method based on fault model with the help of dependency model and interaction model.

State based protocols are protocols in which the handling of one message depends on the contents of previous messages. Testing such protocols, for security or for other purposes usually means specifying the state space of the protocol in some manner. This paper introduces a novel method of using an existing client to explore the state space. The messages exchanged between the client and test system are captured and mutated. To send the mutated test messages, the previous messages must be resent. Constraints expressed in an extended version of the Semantic Constraint Language are used to automatically derive the data dependencies between the messages.

Available soon...

The level of confidence in a VHDL description directly depends on the quality of its verification. This quality can be evaluated by mutation-based test, but the improvement of this quality requires tremendous efforts. In this paper, we propose a new approach that both qualifies and improves the functional verification process. First, we qualify test cases thanks to the mutation testing metrics: faults are injected in the design under verification (DUV) (making DUV's mutants) to check the capacity of test cases to detect theses mutants. Then, a heuristic is used to automatically improve IPs validation data. Experimental results obtained on RTL descriptions from ITC'99 benchmark show how efficient is our approach.

Available soon...

We present a tool to automatically generate mutants of SQL database queries. The SQLMutation tool is available on the Web and it can be accessed using two different interfaces: A Web application to interactively generate the mutants and a Web service that allows it to be integrated with other applications developed using different platforms.