This paper reports the results of a study comparing the effectiveness of automatically generated tests constructed using random and t-way combinatorial techniques on safety related industrial code using mutation adequacy criteria. A reference point is provided by hand generated test vectors constructed during development to establish minimum acceptance criteria. The study shows that 2-way testing is not adequate measured by mutants kill rate compared with hand generated test set of similar size, but that higher factor t-way test sets can perform at least as well. To reduce the computation overhead of testing large numbers of vectors over large numbers of mutants a staged optimising approach to applying t-way tests is proposed and evaluated which shows improvements in execution time and final test set size.

Web services present a new promising software technology. However, some new issues and challenges in testing of them come out due to their characteristics of distribution, source code invisibility etc. This paper discusses the traditional mutation testing and then a new methodology of OWL-S requirement model-based web service mutation testing is brought forward. The traits of this methodology are as follows. Firstly, requirements are used effectively to reduce the magnitude of mutants. Secondly, mutants are generated by AOP technology conveniently and promptly. Thirdly, to reducing testing cost, using business logic implied in OWL-S requirement model as assistant of the process of killing the mutants. Fourthly, two sufficient measurement criteria are employed to evaluate the testing process. Finally, our empirical results have shown the usefulness of this testing method.

Available soon...

Mutation testing is a fault-based testing technique used to inject faults into an existing program and see if its test suite is sensitive enough to detect common faults. We are interested in using the mutation analysis to evaluate, compare and improve quality assurance techniques for testing object-oriented mechanisms and other advanced features of C# programs. This paper provides an overview of a current version of the CREAM system (CREAtor of Mutants), and reports on its use in experimental research. We apply advanced, object-oriented mutation operators to testing of open-source C# programs and discuss the results.

Available soon...

The critical requirement on reliability, fault-tolerance and security of network devices highlights the necessity of protocol robustness testing. Mutation testing of protocol messages is an important part of robustness testing, but related theory and practices are not well developed. This paper builds a NFSM model for mutation testing of protocol messages and proposes two types of Normal-Verification Sequence to enhance verdict mechanism. For single-field mutation testing of protocol messages, we propose the concept of compound anomalous test case to further simplify test sequences. As a standard test specification language, TTCN-3 reveals strong excellence in conformance testing, so we apply TTCN-3 to mutation testing and extend it according to test requirements. Using our method we test OSPFv2 sufficiently with a test system based on extended TTCN-3. The results indicate that our method has good capability of error-finding.

Most commercial software testing tools used in industry do not automatically generate test data even though many testing strategies have been proposed in academia. In this paper, we introduce a prototype software testing platform that we developed to automatically generate test data from tabular specifications. This method of documenting software specifications has been used successfully in industry to help produce reliable software. This platform can also be used to measure the effectiveness of new testing strategies.

We present a model-based approach to testing access control requirements. By using combinatorial testing, we first automatically generate test cases from and without access control policies—i.e., the model—and assess the effectiveness of the test suites by means of mutation testing. We also compare them to purely random tests. For some of the investigated strategies, non-random tests kill considerably more mutants thanthe same number of random tests. Since we rely on policies only, no information on the application is required at this stage. As a consequence, our methodology applies to arbitrary implementations of the policy decision points.

Traditional mutation testing considers only first order mutants, created by the injection of a single fault. Often these first order mutants denote trivial faults that are easily killed. This paper investigates higher order mutants (HOMs). It introduces the concept of a subsuming HOM; one that is harder to kill than the first order mutants from which it is constructed. By definition, subsuming HOMs denote subtle fault combinations. The paper reports the results of an empirical study into subsuming HOMs, using six benchmark programs. This is the largest study of mutation testing to date. To overcome the exponential explosion in the number of mutants considered, the paper introduces a search based approach to the identification of subsuming HOMs. Results are presented for a greedy algorithm, a genetic algorithm and a hill climbing algorithm.

This paper introduces MILU, a C mutation testing tool designed for both first order and higher order mutation testing. All previous mutation testing tools apply all possible mutation operators to the program under test. By contrast, MILU allows customization of the set of mutation operators to be applied. To reduce runtime cost, MILU uses a novel 'test harness' technique to embed mutants and their associated test sets into a single-invocation procedure.

Web Ontology Language for Services (OWL-S) is a standard XML-based language for specifying workflows and integration semantics among Web services (WS), which form composite WS. This paper analyzes the fault patterns of OWL-S specified composite WS and their workflows, proposes an ontology-based mutation analysis method, and applies specification-based mutation techniques for composite WS simulation and testing. Four categories of OWL-S mutant operators are specified, including data mutation, condition mutation, control flow mutation, and data flow mutation. Finally, the paper studies the ontology-based input mutation technique using a BookFinder service as a case study, which shows that ontology-based mutation provides viable test adequacy criteria for testing OWL-S specified composite WS.

Firewalls are the mainstay of enterprise security and the most widely adopted technology for protecting private networks. As the quality of protection provided by a firewall directly depends on the quality of its policy (i.e., configuration), ensuring the correctness of security policies is important and yet difficult.To help ensure the correctness of a firewall policy, we propose a systematic structural testing approach for firewall policies. We define structural coverage (based on coverage criteria of rules, predicates, and clauses) on the policy under test. Considering achieving higher structural coverage effectively, we develop three automated packet generation techniques: the random packet generation, the one based on local constraint solving (considering individual rules locally in a policy), and the most sophisticated one based on global constraint solving (considering multiple rules globally in a policy).We have conducted an experiment on a set of real policies and a set of faulty policies to detect faults with generated packet sets. Generally, our experimental results show that a packet set with higher structural coverage has higher fault detection capability (i.e., detecting more injected faults). Our experimental results show that a reduced packet set (maintaining the same level of structural coverage with the corresponding original packet set) maintains similar fault detection capability with the original set.

Available soon...

One of the most well known and used approach to dynamically analyze a Web application requires to the user to analyze code and requirements of the application to extract its scenarios and the needed inputs. This information is used to exercise the application behavior and so build its model. Hence, high knowledge and effort are required to apply that kind of analysis. Moreover, it is well recognized that a dynamic approach builds partial models since it is strictly related to the application execution. In this paper, we talk about an approach that uses code mutation to reverse engineer a Web application and build its model then used for testing. We document an experiment done to evaluate feasibility and effectiveness of that approach comparing it with other traditional ones. The results show that the approach automatically builds models containing a limited degree of inaccuracy that can be pruned during the application testing.

We present a new approach for mutation analysis of security policies test cases. We propose a metamodel that provides a generic representation of security policies access control models and define a set of mutation operators at this generic level. We use Kermeta to build the metamodel and implement the mutation operators. We also illustrate our approach with two successful instantiation of this metamodel: we defined policies with RBAC and OrBAC and mutated these policies.

Web services are considered a new paradigm for building software applications that has many advantages over the previous paradigms; however, Web services are still not widely used because Service Requesters do not trust Web Services that were built by others. Testing can participate in solving this problem because it can be used to assess the quality attributes of Web Services and hence increase the requesters’ trustworthiness. This paper proposes an approach that can be used to test the robustness and other related attribute of Web Services, and that can be easily enhanced to assess other quality attributes. The framework is based on rules for test case generation that are designed by, firstly, analyzing WSDL document to know what faults could affect the robustness quality attribute of Web Services, and secondly, using the fault-based testing techniques to detect such faults. A proof of concept tool that depends on these rules has been implemented in order to assess the usefulness of the rules in detecting robustness faults in different Web Services platforms.

SQL injection is one of the most prominent vulnerabilities for web-based applications. Exploitation of SQL injection vulnerabilities (SQLIV) through successful attacks might result in severe consequences such as authentication bypassing, leaking of private information etc. Therefore, testing an application for SQLIV is an important step for ensuring its quality. However, it is challenging as the sources of SQLIV vary widely, which include the lack of effective input filters in applications, insecure coding by programmers, inappropriate usage of APIs for manipulating databases etc. Moreover, existing testing approaches do not address the issue of generating adequate test data sets that can detect SQLIV. In this work, we present a mutation-based testing approach for SQLIV testing. We propose nine mutation operators that inject SQLIV in application source code. The operators result in mutants, which can be killed only with test data containing SQL injection attacks. By this approach, we force the generation of an adequate test data set containing effective test cases capable of revealing SQLIV. We implement a MUtation-based SQL Injection vulnerabilities Checking (testing) tool (MUSIC) that automatically generates mutants for the applications written in Java Server Pages (JSP) and performs mutation analysis. We validate the proposed operators with five open source web-based applications written in JSP. We show that the proposed operators are effective for testing SQLIV.

The assessment of a testing strategy and the comparison of different testing strategies is a crucial part in current research on software testing. Often, manual error seeding is used to generate faulty programs. As a consequence, the results obtained from the examination of these programs are often not reproducible and likely to be biased. In this paper, a flexible approach to the benchmarking of testing strategies is presented. The approach utilizes well- known results from mutation analysis to construct an objective effectiveness measure for test oracles. This measure allows to draw conclusions not only on the effectiveness of a single testing strategy but also to compare different testing strategies by their effectiveness measure.

We present a model-based approach to testing access control requirements. By using combinatorial testing, we first automatically generate test cases from and without access control policies—i.e., the model—and assess the effectiveness of the test suites by means of mutation testing. We also compare them to purely random tests. For some of the investigated strategies, non-random tests kill considerably more mutants thanthe same number of random tests. Since we rely on policies only, no information on the application is required at this stage. As a consequence, our methodology applies to arbitrary implementations of the policy decision points.

Research has shown that modern micro-architectures are vulnerable to soft errors, i.e., temporary errors caused by voltage spikes produced by cosmic radiation. Soft-error impact is usually evaluated using fault injection, a black-box testing approach similar to mutation testing. In this paper, we complement an existing evaluation of a prototype brake-by-wire controller, developed by Volvo Technology, with static-analysis techniques to improve test effectiveness. The fault-injection tests are both time- and data-intensive, which renders their qualitative and quantitative assessment difficult. We devise a prototype visualization tool, which groups experiments by injection point and provides an overview of both instruction and fault coverage, and the ability to detect patterns and anomalies. We use the program-dependence graph to identify experiments with a priori known outcome, and implement a static analysis to reduce the test volume. The existing pre-injection heuristic is extended with liveness analysis to enable an unbiased fault-to-failure probability.