Available soon...

Available soon...

Regression testing is an important activity in the software life cycle, but it can also be very expensive. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of test case prioritization techniques is to increase a test suite's rate of fault detection (how quickly, in a run of its test cases, that test suite can detect faults). Previous work has shown that prioritization can improve a test suite's rate of fault detection, but the assessment of prioritization techniques has been limited primarily to hand-seeded faults, largely due to the belief that such faults are more realistic than automatically generated (mutation) faults. A recent empirical study, however, suggests that mutation faults can be representative of real faults and that the use of hand-seeded faults can be problematic for the validity of empirical results focusing on fault detection. We have therefore designed and performed two controlled experiments assessing the ability of prioritization techniques to improve the rate of fault detection of test case prioritization techniques, measured relative to mutation faults. Our results show that prioritization can be effective relative to the faults considered, and they expose ways in which that effectiveness can vary with characteristics of faults and test suites. More importantly, a comparison of our results with those collected using hand-seeded faults reveals several implications for researchers performing empirical studies of test case prioritization techniques in particular and testing techniques in general.

The mutation technique inserts faults in a program under test in order to assess or generate test cases, or evaluate the reliability of the program. Faults introduced into the source code are defined using mutation operators. They should be related to different, also object-oriented features of a program. The most research on OO mutations was devoted to Java programs. This paper describes analytical and empirical study performed to evaluate the quality of advanced mutation operators for C# programs. Experimental results demonstrate effectiveness of different mutation operators. Unit tests suites and functional tests were used in experiments. A detailed analysis was conducted on mutation operators dealing with delegates and exception handling.

Testability is a quality factor used to predict the amount of effort required for software testing and to indicate the difficulty of revealing faults. This paper presents a quantitative testability analysis method for a software component that can be used when the source program is not available, but the bytecode is (as in Java .class files). This process analyzes the testability of each location to evaluate the component testability. The testability of a location is analyzed by computing the probability that the location will be executed and, if the location contains a fault, the execution will cause the fault to be revealed as a failure. This analysis process helps developers measure component testability and determine whether the component testability should be increased before the component is reused. In addition, low testability locations are identified.

Fault-based testing focuses on detecting faults in a software. Test data is typically generated considering the presence of a single fault at a time, under the assumption of coupling hypothesis. Fault-based testing approach, in particular mutation testing, assumes that the coupling hypothesis holds. According to the hypothesis, a test set that can detect presence of single faults in an implementation, is also likely to detect presence of multiple faults. In this paper it is formally shown that the hypothesis is guaranteed to hold for a large number of logical fault classes.

Available soon...

Reactive systems used in safety-critical domains demand high level of confidence. The development of these systems, which are submitted to several normative recommendations, is complex and expensive. Reactive systems can be developed by using the data-flow approach: many languages support this approach such as MATLAB/SlMULINK, LUSTRE/SCADE. This paper concentrates on the LUSTRE/SCADE language, especially the description of reactive system environment properties in this language. The description of environment properties, which is important for the validation (the proof and the test) of reactive systems, is not easy. Hence, we would like to use the mutation technique to consolidate this difficult task: we use the LESAR model-checking tool to detect equivalent mutants and some test case generators such as GATEL or LUTESS tools to kill non-equivalent mutants.

Mutation testing is a valuable experimental research technique that has been used in many studies. It has been experimentally compared with other test criteria, and also used to support experimental comparisons of other test criteria, by using mutants as a method to create faults. In effect, mutation is often used as a ``gold standard'' for experimental evaluations of test methods. Although mutation testing is powerful, it is a complicated and computationally expensive testing method. Therefore, automated tool support is indispensable for conducting mutation testing. This demo presents a publicly available mutation system for Java that supports both method-level mutants and class-level mutants. MuJava can be freely downloaded and installed with relative ease under both Unix and Windows. MuJava is offered as a free service to the community and we hope that it will promote the use of mutation analysis for experimental research in software testing.

The effectiveness of mutation testing depends heavily on the types of faults that the mutation operators are designed to represent. Thus, the quality of the mutation operators is key to mutation testing. Although, mutation operators for object-oriented languages have previously been presented, little research has been done to show the usefulness of the class mutation operators. To assess the usefulness of class mutation operators, we conducted two empirical studies. In the first study, we examine the number and kinds of mutants that are generated for object-oriented programs. In the second study, we investigate the way in which class mutation operators model faults that are not detected by traditional mutation testing. We conducted our studies using a well-known object-oriented system, BCEL.

One of the main concerns in safety-critical software is to ensure sufficient reliability because proof of the absence of systematic failures has proved to be an unrealistic goal. fault-tolerance (FT) is one method for improving reliability claims. It is reasonable to assume that some software FT techniques offer more protection than others, but the relative effectiveness of different software FT schemes remains unclear. We present the principles of a method to assess the effectiveness of FT using mutation analysis. The aim of this approach is to observe the power of FT directly and use this empirical process to evolve more powerful forms of FT. We also investigate an approach to FT that integrates data diversity (DD) assertions and TA. This work is part of a longer term goal to use FT in quantitative safety arguments for safety critical systems.

The injection of faults has been widely used to evaluate fault tolerance mechanisms and to assess the impact of faults in computer systems. However, the injection of software faults is not as well understood as other classes of faults (e.g., hardware faults). In this paper, we analyze how software faults can be injected (emulated) in a source-code independent manner. We specifically address important emulation requirements such as fault representativeness and emulation accuracy. We start with the analysis of an extensive collection of real software faults. We observed that a large percentage of faults falls into well-defined classes and can be characterized in a very precise way, allowing accurate emulation of software faults through a small set of emulation operators. A new software fault injection technique (G-SWFIT) based on emulation operators derived from the field study is proposed. This technique consists of finding key programming structures at the machine code-level where high-level software faults can be emulated. The fault-emulation accuracy of this technique is shown. This work also includes a study on the key aspects that may impact the technique accuracy. The portability of the technique is also discussed and it is shown that a high degree of portability can be achieved.

Fault-based testing is a technique where testers anticipate errors in a system under test in order to assess or generate test cases. The idea is to have enough test cases capable of detecting these anticipated errors. This paper presents a theory and technique for generating fault-based test cases for concurrent systems. The novel idea is to generate test purposes from faults that have been injected into a model of the system under test. Such test purposes form a specification of a more detailed test case that can detect the injected fault. The theory is based on the notion of refinement. The technique is automated using the TGV test case generator and an equivalence checker of the CADP tools. A case study of testing web servers demonstrates the practicability of the approach.

This paper presents a novel approach to unit testing that lets users of deployed software assist in performing mutation testing of the software. Our technique, MUGAMMA, provisions a software system so that when it executes in the field, it will determine whether users' executions would have killed mutants (without actually executing the mutants), and if so, captures the state information about those executions. In the absence of bug reports, knowledge of executions that would have killed mutants provides additional confidence in the system over that gained by the testing performed before deployment. Captured information about the state before and after execution of units (e.g., methods) can be used to construct test cases for use in unit testing when changes are made to the software. The paper also describes our prototype MUGAMMA implementation along with a case study that demonstrates its potential efficacy.

Software testing is an important technique for assurance of software quality. Mutation testing has been identified as a powerful fault-based technique for unit testing, and there has been some research on automatic generation of test data for mutation testing. However, existing approaches to this kind of test data generation usually generate test data according to one mutant at one time. Thus, more test data that are needed for achieving a given mutation score. In this paper, we propose a new approach to generating one test data according to multiple mutants that are mutated at the same location at one time. Thus, our approach can generate smaller test suite that can achieve the same mutation testing score. To evaluate our approach, we implemented a prototype tool based on our approach and carried out some preliminary experiments. The experimental results show that our approach is more cost-effective.

Software testing is one of the most relevant activities regarding system development but, at the same time, it is a difficult topic to learn or teach without the appropriate supporting mechanisms. A learning process of software testing should involve the cooperation of theoretical and experimental knowledge with related tools in order to facilitate the apprenticeship of specific testing theories and skills. In previous works we investigated the establishment of supporting mechanisms for developing educational modules. The idea was contribute to the improvement of learning processes in general by producing quality educational products, capable of motivating the learners and effectively contribute to their knowledge construction process. In this paper we illustrate the application of our ideas in the context of software testing, more specifically by the development of an educational module for mutation testing. We consider mutation testing since it is a sound and powerful approach, but its use is still limited, mainly due to its high cost of application and, also, due to the need of an adequate learning process.

Available soon...

In MDE, model transformations should be efficiently tested so that it may be used and reused safely. Mutation analysis is an efficient technique to evaluate the quality of test data, and has been extensively studied both for procedural and object-oriented languages. In this paper, we study how it can be adapted to model oriented programming. Since no model transformation language has been widely accepted today, we propose generic fault models that are related to the model transformation process. First, we identify abstract operations that constitute this process: model navigation, model’s elements filtering, output model creation and input model modification. Then, we propose a set of specific mutation operators which are directly inspired from these operations. We believe that these operators are meaningful since a large part of the errors in a transformation are due to the manipulation of complex models regardless of the concrete implementation language.

Along with growing popularity of agile methodologies and open source movement, unit testing has become one of the core practices in modern software engineering. It is particularly important in eXtreme Programming [1], which explicitly diminish the importance of other artifacts than source code and tests cases. In XP unit test cases not only verify if software meets functional requirements, but also enable refactoring, alleviate comprehension and provide guidance on how the production code should be used. Therefore, they contribute to many other important practices of XP, which explicitly or implicitly rely on their ability to effectively discover bugs.

Available soon...