Available soon...

Available soon...

Data from projects worldwide show that many software projects fail and most are completed late or over budget. Unit testing is a simple but effective technique to improve software in terms of quality, flexibility, and time-to-market. A key idea of unit testing is that each piece of code needs its own tests and the best person to design those tests is the developer who wrote the software. However, generating tests for each unit by hand is very expensive, possibly prohibitively so. Automatic test data generation is essential to support unit testing and as unit testing is achieving more attention, developers have a greater need for automated unit test data generation tools. However, developers have very little information about which tools are effective. This experiment compared three well-known public-accessible unit test data generation tools, JCrasher, TestGen4j, and JUB. We applied them to Java classes and evaluated them based on their mutation scores. As a comparison, we created two additional sets of tests for each class. One test set contained random values and the other contained values to satisfy edge coverage. Results showed that the automatic test data generation tools generated tests with almost the same mutation scores as the random tests.

With recent increased expectations for quality, and the growth of agile processes and test driven development, developers are expected to do more and more effective unittesting. Yet, our knowledge of when to use the variousunit level test criteria is incomplete. The paper presentsresults from a comparison of four unit level software testing criteria. Mutation testing, prime path coverage, edgepair coverage, and all-uses testing were compared on twobases: the number of seeded faults found and the numberof tests needed to satisfy the criteria. The comparisonused a collection of Java classes taken from varioussources and hand-seeded faults. Tests were designed andgenerated mostly by hand with help from tools that computetest requirements and muJava. The findings are that mutationtests detected more faults and the other three criteriawere very similar. The paper also presents a secondarymeasure, a cost benefit ratio, computed as the number oftests needed to detect each fault. Surprisingly, mutation required the fewest number of tests. The paper also discusses some specific faults that were not found and presents analysis for why not.

Testability is a quality factor used to predict the amount of effort required for software testing and to indicate the difficulty of revealing faults. This paper presents a quantitative testability analysis method for a software component that can be used when the source program is not available, but the bytecode is (as in Java .class files). This process analyzes the testability of each location to evaluate the component testability. The testability of a location is analyzed by computing the probability that the location will be executed and, if the location contains a fault, the execution will cause the fault to be revealed as a failure. This analysis process helps developers measure component testability and determine whether the component testability should be increased before the component is reused. In addition, low testability locations are identified.

Temporal correctness is crucial for real-time systems. Few methods exist to test temporal correctness and most methods used in practice are ad-hoc. A problem with testing real-time applications is the response-time dependency on the execution order of concurrent tasks. Execution order in turn depends on execution environment properties such as scheduling protocols, use of mutual exclusive resources as well as the point in time when stimuli is injected. Model based mutation testing has previously been proposed to determine the execution orders that need to be verified to increase confidence in timeliness. An effective way to automatically generate such test cases for dynamic real-time systems is still needed. This paper presents a method using heuristic-driven simulation to generate test cases.

This paper presents results from empirical studies of object-oriented, class level mutation operators, using the auto- mated analysis and testing tool MuJava. Class mutation operators modify OO programming language features such as inheritance, polymorphism, dynamic binding and encapsulation. This paper presents data from 866 classes in six open-source programs. Several new class-level mutation operators are defined in this paper and an analysis of the number of mutants generated is provided. Techniques for eliminating some equivalent mutants are described and data from an automated tool are provided. One important result is that class-level mutation operators yield far more equivalent mutants than traditional, statement-level, operators. Another is that there are far fewer class-level mutants than statement-level mutants. Together, these data suggest that mutation for inter-class testing can be practically affordable.

Mutation testing is a valuable experimental research technique that has been used in many studies. It has been experimentally compared with other test criteria, and also used to support experimental comparisons of other test criteria, by using mutants as a method to create faults. In effect, mutation is often used as a ``gold standard'' for experimental evaluations of test methods. Although mutation testing is powerful, it is a complicated and computationally expensive testing method. Therefore, automated tool support is indispensable for conducting mutation testing. This demo presents a publicly available mutation system for Java that supports both method-level mutants and class-level mutants. MuJava can be freely downloaded and installed with relative ease under both Unix and Windows. MuJava is offered as a free service to the community and we hope that it will promote the use of mutation analysis for experimental research in software testing.

This paper presents an abstract view of mutation analysis. Mutation was originally thought of as making changes to program source, but similar kinds of changes have been applied to other artifacts, including program specifications, XML, and input languages. This paper argues that mutation analysis is actually a way to modify any software artifact based on its syntactic description, and is in the same family of test generation methods that create inputs from syntactic descriptions. The essential characteristic of mutation is that a syntactic description such as a grammar is used to create tests. We call this abstract view grammar-based testing, and view it as an interface, which mutation analysis implements. This shift in view allows mutation to be defined in a general way, yielding three benefits. First, it provides a simpler way to understand mutation. Second, it makes it easier to develop future applications of mutation analysis, such as finite state machines and use case collaboration diagrams. The third benefit, which due to space limitations is not explored in this paper, is ensuring that existing techniques are complete according to the criteria defined here.

Several module and class testing techniques have been applied to object-oriented (OO) programs, but researchers have only recently begun developing test criteria that evaluate the use of key OO features such as inheritance, polymorphism, and encapsulation. Mutation testing is a powerful testing technique for generating software tests and evaluating the quality of software. However, the cost of mutation testing has traditionally been so high that it cannot be applied without full automated tool support. This paper presents a method to reduce the execution cost of mutation testing for OO programs by using two key technologies, mutant schemata generation (MSG) and bytecode translation. This method adapts the existing MSG method for mutants that change the program behaviour and uses bytecode translation for mutants that change the program structure. A key advantage is in performance: only two compilations are required and both the compilation and execution time for each is greatly reduced. A mutation tool based on the MSG/bytecode translation method has been built and used to measure the speedup over the separate compilation approach. Experimental results show that the MSG/bytecode translation method is about five times faster than separate compilation.

Available soon...

Mutation is a powerful but complicated and computationally expensive testing method. Mutation is also a valuable experimental research technique that has been used in many studies. Mutation has been experimentally compared with other test criteria, and also used to support experimental comparisons of other test criteria, by using mutants as a method to create faults. In effect, mutation is often used as a "gold standard" for experimental evaluations of test methods. This paper presents a publicly available mutation system for Java that supports both traditional statement-level mutants and newer inter-class mutants. MUJAVA can be freely downloaded and installed with relative ease under both Unix and Windows. MUJAVA is offered as a free service to the community and we hope that it will promote the use of mutation analysis for experimental research in software testing.

Available soon...

Available soon...

Available soon...

Although program faults are widely studied, there are many aspects of faults that we still do not understand, particularly about OO software. In addition to the simple fact that one important goal during testing is to cause failures and thereby detect faults, a full understanding of the characteristics of faults is crucial to several research areas. The power that inheritance and polymorphism brings to the expressiveness of programming languages also brings a number of new anomalies and fault types. In prior work we presented a fault model for the appearance and realization of OO faults that are specific to the use of inheritance and polymorphism. Many of these faults cannot appear unless certain syntactic patterns are used. The patterns are based on language constructs, such as overriding methods that directly define inherited state variables and non-inherited methods that call inherited methods. If one of these syntactic patterns is used, then we say the software contains an anomaly and possibly a fault. We describe the syntactic patterns for each OO fault type. These syntactic patterns can potentially be found with an automatic tool. Thus, faults can be uncovered and removed early in development.

Available soon...

Available soon...

Available soon...

Test data generation is one of the most technically challenging steps of testing software, but most commercial systems currently incorporate very little automation for this step. This paper presents results from a project that is trying to find ways to incorporate test data generation into practical test processes. The results include a new procedure for automatically generating test data that incorporates ideas from symbolic evaluation, constraint-based testing, and dynamic test data generation. It takes an initial set of values for each input, and dynamically ‘pushes’ the values through the control-flow graph of the program, modifying the sets of values as branches in the program are taken. The result is usually a set of values for each input parameter that has the property that any choice from the sets will cause the path to be traversed. This procedure uses new analysis techniques, offers improvements over previous research results in constraint-based testing, and combines several steps into one coherent process. The dynamic nature of this procedure yields several benefits. Moving through the control flow graph dynamically allows path constraints to be resolved immediately, which is more efficient both in space and time, and more often successful than constraint-based testing. This new procedure also incorporates an intelligent search technique based on bisection. The dynamic nature of this procedure also allows certain improvements to be made in the handling of arrays, loops, and expressions; language features that are traditionally difficult to handle in test data generation systems. The paper presents the test data generation procedure, examples to explain the working of the procedure, and results from a proof-of-concept implementation.

We present a fault-classification scheme and a fault-seeding method that are based on the manifestation of faults in the program dependence graph (PDG). We enhance the domain/computation fault classification scheme developed by Howden to further characterize faults as structural and statement-level depending on the differences between the PDG for the original program and the PDG for the faulty program. We perform transformations on the PDG to produce the different types of faults described in our PDG-based fault-classification scheme. To demonstrate the usefulness of our technique, we implemented a fault seeder to embed faults in C programs. Our fault seeder makes controlled fault transformations to the PDG for a C program, and generates C code from the transformed PDG. The current version of the fault seeder creates multiple fault-seeded versions of the original program, each with one known fault. To demonstrate the operation of the fault seeder, we used it to perform a study of the effectiveness of dataflow testing and mutation testing using a set of faulty programs generated by our fault seeder. We also used the faulty programs to determine the mutation adequacy and detaflow adequacy of the fault-detecting test sets.

Mutation testing is a technique for testing software units that has great potential for improving the quality of testing, and thereby increasing our ability to assure the high reliability of critical software. It will be shown that recent advances in mutation research have brought a practical mutation testing system closer to reality. One recent advance is a partial solution to the problem of automatically detecting equivalent mutant programs. Equivalent mutants are currently detected by hand, which makes it very expensive and time-consuming. The problem of detecting equivalent mutants is a specific instance of a more general problem, commonly called the feasible path problem, which says that for certain structural testing criteria some of the test requirements are infeasible in the sense that the semantics of the program imply that no test case satisfies the test requirements. Equivalent mutants, unreachable statements in path testing techniques, and infeasible DU-pairs in data flow testing are all instances of the feasible path problem. This paper presents a technique that uses mathematical constraints, originally developed for test data generation, to automatically detect some equivalent mutants and infeasible paths.

Available soon...

Available soon...

Mutation testing is a technique for unit testing software that, although powerful, is computationally expensive. The principal expense of mutation is that many variants of the test program, called mutants, must be repeatedly executed. This paper quanti es the expense of mutation in terms of the number of mutants that are created, then proposes and evaluates a technique that reduces the number of mutants by an order of magnitude. Selective mutation reduces the cost of mutation testing by reducing the number of mutants. This paper reports experimental results that compare selective mutation testing with standard, or non-selective, mutation testing, and results that quantify the savings achieved by selective mutation testing. The results support the hypothesis that selective mutation is almost as strong as non-selective mutation; in experimental trials selective mutation provides almost the same coverage as non-selective mutation, with a four-fold or more reduction in the number of mutants.

Available soon...

Available soon...

Available soon...

Two experimental comparisons of data flow and mutation testing are presented. These techniques are widely considered to be effective for unit-level software testing, but can only be analytically compared to a limited extent. We compare the techniques by evaluating the effectiveness of test data developed for each. We develop ten independent sets of test data for a number of programs: five to satisfy the mutation criterion and five to satisfy the all-uses data-flow criterion. These test sets are developed using automated tools, in a manner consistent with the way a test engineer might be expected to generate test data in practice. We use these test sets in two separate experiments. First we measure the effectiveness of the test data that was developed for one technique in terms of the other. Second, we investigate the ability of the test sets to find faults. We place a number of faults into each of our subject programs, and measure the number of faults that are detected by the test sets. Our results indicate that while both techniques are effective, mutation-adequate test sets are closer to satisfying the data flow criterion, and detect more faults.

Mutation testing is a fault-based technique for unit-level software testing. Weak mutation was proposed as a way to reduce the expense of mutation testing. Unfortunately, weak mutation is also expected to provide a weaker test of the software than mutation testing does. This paper presents results from an implementation of weak mutation, which we used to evaluate the effectiveness versus the efficiency of weak mutation. Additionally, we examined several options in an attempt to find the most appropriate way to implement weak mutation. Our results indicate that weak mutation can be applied in a manner that is almost as effective as mutation testing, and with significant computational savings.

Mutation analysis is a software testing technique that requires the tester to generate test data that will find specific, well-defined errors. Mutation testing executes many slightly differing versions, called mutants, of the same program to evaluate the quality of the data used to test the program. Although these mutants are generated and executed efficiently by automated methods, many of the mutants are functionally equivalent to the original program and are not useful for testing. Recognizing and eliminating equivalent mutants is currently done by hand, a time-consuming and arduous task. This problem is currently a major obstacle to the practical application of mutation testing. This paper presents extensions to previous work in detecting equivalent mutants; specifically, algorithms for determining several classes of equivalent mutants are presented, an implementation of these algorithms is discussed, and results from using this implementation are presented. These algorithms are based on data flow analysis and six compiler optimization techniques. Each of these techniques is described together with how they are used to detect equivalent mutants. The design of the tool and some experimental results using it are also presented.

Available soon...

Available soon...

Available soon...

Mutation analysis is a powerful technique for assessing and improving the quality of test data used to unit test software. Unfortunately, current automated mutation analysis systems suffer from severe performance problems. This paper presents a new method for performing mutation analysis that uses program schemata to encode all mutants for a program into one metaprogram, which is subsequently compiled and run at speeds substantially higher than achieved by previous interpretive systems. Preliminary performance improvements of over 300% are reported. This method has the additional advantages of being easier to implement than interpretive systems, being simpler to port across a wide range of hardware and software platforms, and using the same compiler and run-time support system that is used during development and/or deployment.

Available soon...

Available soon...

Constraint-based testing is a novel way of generating test data to detect specific types of common programming faults. The conditions under which faults will be detected are encoded as mathematical systems of constraints in terms of program symbols. A set of tools, collectively called Godzilla, has been implemented that automatically generates constraint systems and solves them to create test cases for use by the Mothra testing system. Experimental results from using Godzilla show that the technique can produce test data that is very close in terms of mutation adequacy to test data that is produced manually, and at substantially reduced cost. Additionally, these experiments have suggested a new procedure for unit testing, where test cases are viewed as throw-away items rather than scarce resources.

Available soon...

Available soon...

Available soon...

Fault-based testing strategies test software by focusing on specific, common types of faults. The coupling effect hypothesizes that test data sets that detect simple types of faults are sensitive enough to detect more complex types of faults. This paper describes empirical investigations into the coupling effect over a specific class of software faults. All of the results from this investigation support the validity of the coupling effect. The major conclusion from this investigation is the fact that by explicitly testing for simple faults, we are also implicitly testing for more complicated faults, giving us confidence that fault-based testing is an effective way to test software.

A simulation-based method for obtaining numerical estimates of the reliability of N-version, real-time software is proposed. An extended stochastic Petri net is used to represent the synchronization structure of N versions of the software, where dependencies among versions are modeled through correlated sampling of module execution times. The distributions of execution times are derived from automatically generated test cases that are based on mutation testing. Since these test cases are designed to reveal software faults, the associated execution times and reliability estimates are likely to be conservative. Experimental results using specifications for NASA's planetary lander control software suggest that mutation-based testing could hold greater potential for enhancing reliability than the desirable but perhaps unachievable goal of independence among N versions. Nevertheless, some support for N-version enhancement of high-quality, mutation-tested code is also offered. Mutation analysis could also be valuable in the design of fault-tolerant software systems.

A novel technique for automatically generating test data is presented. The technique is based on mutation analysis and creates test data that approximate relative adequacy. It is a fault-based technique that uses algebraic constraints to describe test cases designed to find particular types of faults. A set of tools (collectively called Godzilla) that automatically generates constraints and solves them to create test cases for unit and module testing has been implemented. Godzilla has been integrated with the Mothratesting system and has been used as an effective way to generate test data that kill program mutants. The authors present an initial list of constraints and discuss some of the problems that have been solved to develop the complete implementation of the technique.

Available soon...

Available soon...

Mutation analysis is a powerful technique for testing software systems. The Mothra software testing project uses mutation analysis as the basis for an integrated software testing environment. Mutation analysis requires executing many slightly differing versions of the same program to evaluate the quality of the data used to test the program. The current version of Mothra includes a complete language system that translates a program to be tested into intermediate code so that it and its mutated versions can be executed by an interpreter. In this paper, we discuss some of the unique requirements of a language system used in a mutation-based testing environment. We then describe how these requirements affected the design and implementation of the Fortran 77 version of the Mothra system. We also describe the intermediate language used by Mothra and the features of the language system that are needed for software testing. The appendices contain a full description of the intermediate language and the mutation operators used by Mothra. The design and implementation techniques that were developed for Mothra are applicable for constructing not just software testing systems, but any type of program analysis system or language system for a special-purpose application. In particular, we discuss decisions made and techniques developed by the Mothra team that can be useful in such applications as debuggers, program measurement tools, software development environments and other types of program analysis systems.

Available soon...

Fault-based testing strategies test software by focusing on specific, common types of errors. The coupling effect states that test data sets that detect simple types of faults are sensitive enough to detect more complex types of faults. This paper describes empirical investigations into the coupling effect over a specific domain of software faults. All the results from this investigation support the validity of the coupling effect. The major conclusion from this investigation is that by explicitly testing for simple faults, we are also implicitly testing for more complicated faults. This gives us confidence that fault-based testing is an effective means of testing software.

Mothra is a software test environment that supports mutation-based testing of software systems. Mutation analysis is a powerful software testing technique that evaluates the adequacy of test data based on its ability to differentiate between the program under test and its mutants, where mutants are constructed by inserting single, simple errors into the program under test. This evaluation process also provides guidance in the creation of new test cases to provide more adequate testing. Mothra consists of a collection of individual tools, each of which implements a separate, independent function for the testing system. The initial Mothra tool set, for the most part, duplicates functionality existing in previous mutation analysis systems. Current efforts are concentrated on extending this basic tool set to include capabilities previously unavailable to the software testing community. The authors describe Mothra tool set and extensions planned for the future.

Available soon...

Mutation analysis is a powerful technique for testing software systems. In the Mothra project, conducted at Georgia Tech's Software Engineering Research Center, mutation analysis is used as a basis for building an integrated software testing environment. Mutation analysis requires the execution of many slightly differing versions of the same program to evaluate the quality of the data used to test the program. In the current version of the Mothra system, a program to be tested is translated to intermediate code, where it and its mutated versions are executed by an interpreter.In this paper, we discuss some of the unique requirements of an interpreter used in a mutation-based testing environment. We then describe how these requirements affected the design and implementation of the Fortran 77 version of the Mothra interpreter. Other topics covered include the architecture of the interpreter and many of the design elements that it incorporates. We also describe the intermediate language used by Mothra and the features of the interpreter that are needed for software testing.