Mutation testing is a powerful software engineering technique for fault finding. It works by injecting known faults (mutations) into software and seeing if the test suite finds them. It remains very expensive and the few valuable traditional mutants that resemble real faults are mixed in with many others that denote unrealistic faults. The expense and lack of realism inhibit industrial uptake of mutation testing. Genetic programming searches the space of complex faults to find realistic higher order mutants. Despite the much larger search space, we have found mutants composed of multiple changes to the C source code that challenge the tester and which cannot be represented in the first order space.

Mutation testing is a powerful software engineering technique for fault finding. It works by injecting known faults (mutations) into software and seeing if the test suite finds them. It remains very expensive and the few valuable traditional mutants that resemble real faults are mixed in with many others that denote unrealistic faults. The expense and lack of realism inhibit industrial uptake of mutation testing. Genetic programming searches the space of complex faults to find realistic higher order mutants. Despite the much larger search space, we have found mutants composed of multiple changes to the C source code that challenge the tester and which cannot be represented in the first order space.

Available soon...

Available soon...

utation analysis is a program testing method which seeds a fault in a program and tries to identify it with test data, thus promoting the test efficiency. The paper investigates the application of mutation analysis in model-based testing for the modeling language of finite state machines (FSM). We describe a set of mutation operators for FSM based on the fault category; present an algorithm of selecting a test suite for the mutation testing of system models in FSM. In an experiment, other five methods of test suites generating and selecting for FSM are chosen to compare with the mutation testing method. The experiment shows that in respect of faults detecting in FSM, the mutation testing is more effective and efficient than the other FSM testing methods including D-method, W-method and T-method.

Buffer overflow (BOF) is one of the major vulnerabilities that lead to non-secure software. Testing an implementation for BOF vulnerabilities is challenging as the underlying reasons of buffer overflow vary widely. This paper presents a novel method for BOF test for ANSI C language, which uses program instrumentation and mutation test technology to test the BOF vulnerabilities, on the basis of analyzing the invariants for BOF vulnerabilities. The implementation shows that it can check the attack of BOF vulnerabilities adequately and accurately, in the circumstances of no large losses in performance.

With recent increased expectations for quality, and the growth of agile processes and test driven development, developers are expected to do more and more effective unittesting. Yet, our knowledge of when to use the variousunit level test criteria is incomplete. The paper presentsresults from a comparison of four unit level software testing criteria. Mutation testing, prime path coverage, edgepair coverage, and all-uses testing were compared on twobases: the number of seeded faults found and the numberof tests needed to satisfy the criteria. The comparisonused a collection of Java classes taken from varioussources and hand-seeded faults. Tests were designed andgenerated mostly by hand with help from tools that computetest requirements and muJava. The findings are that mutationtests detected more faults and the other three criteriawere very similar. The paper also presents a secondarymeasure, a cost benefit ratio, computed as the number oftests needed to detect each fault. Surprisingly, mutation required the fewest number of tests. The paper also discusses some specific faults that were not found and presents analysis for why not.

Available soon...

Available soon...

Available soon...

Available soon...

Available soon...

Recently parameterized unit testing has emerged as a promising and effective methodology to allow the separation of (1) specifying external, black-box behavior (e.g., assumptions and assertions) by developers and (2) generating and selecting internal, white-box test inputs (i.e., high-code-covering test inputs) by tools. A parameterized unit test (PUT) is simply a test method that takes parameters, specifies assumptions on the parameters, calls the code under test, and specifies assertions. The test effectiveness of PUTs highly depends on the way that they are written by developers. For example, if stronger assumptions are specified, only a smaller scope of test inputs than intended are generated by tools, leading to false negatives in terms of fault detection. If weaker assertions are specified, erroneous states induced by the test execution do not necessarily cause assertion violations, leading to false negatives. Detecting these false negatives is challenging since the insufficiently written PUTs would just pass. In this paper, we propose a novel mutation analysis approach for analyzing PUTs written by developers and identifying likely locations in PUTs for improvement. The proposed approach is a first step towards helping developers write better PUTs in practice.

Conformance testing procedures for generating tests from the finite state model representation of Role-Based Access Control (RBAC) policies are proposed and evaluated. A test suite generated using one of these procedures has excellent fault detection ability but is astronomically large. Two approaches to reduce the size of the generated test suite were investigated. One is based on a set of six heuristics and the other directly generates a test suite from the finite state model using random selection of paths in the policy model. Empirical studies revealed that the second approach to test suite generation, combined with one or more heuristics, is most effective in the detection of both first-order mutation and malicious faults and generates a significantly smaller test suite than the one generated directly from the finite state models.

We present a model-based approach to testing access control requirements. By using combinatorial testing, we first automatically generate test cases from and without access control policies—i.e., the model—and assess the effectiveness of the test suites by means of mutation testing. We also compare them to purely random tests. For some of the investigated strategies, non-random tests kill considerably more mutants thanthe same number of random tests. Since we rely on policies only, no information on the application is required at this stage. As a consequence, our methodology applies to arbitrary implementations of the policy decision points.

Data from projects worldwide show that many software projects fail and most are completed late or over budget. Unit testing is a simple but effective technique to improve software in terms of quality, flexibility, and time-to-market. A key idea of unit testing is that each piece of code needs its own tests and the best person to design those tests is the developer who wrote the software. However, generating tests for each unit by hand is very expensive, possibly prohibitively so. Automatic test data generation is essential to support unit testing and as unit testing is achieving more attention, developers have a greater need for automated unit test data generation tools. However, developers have very little information about which tools are effective. This experiment compared three well-known public-accessible unit test data generation tools, JCrasher, TestGen4j, and JUB. We applied them to Java classes and evaluated them based on their mutation scores. As a comparison, we created two additional sets of tests for each class. One test set contained random values and the other contained values to satisfy edge coverage. Results showed that the automatic test data generation tools generated tests with almost the same mutation scores as the random tests.

Available soon...

Format string bugs (FSBs) make an implementation vulnerable to numerous types of malicious attacks. Testing an implementation against FSBs can avoid consequences due to exploits of FSBs such as denial of services, corruption of application states, etc. Obtaining an adequate test data set is essential for testing of FSBs. An adequate test data set contains effective test cases that can reveal FSBs. Unfortunately, traditional techniques do not address the issue of adequate testing of an application for FSB. Moreover, the application of source code mutation has not been applied for testing FSB. In this work, we apply the idea of mutation-based testing technique to generate an adequate test data set for testing FSBs. Our work addresses FSBs related to ANSI C libraries. We propose eight mutation operators to force the generation of adequate test dataset. A prototype mutation-based testing tool named MUFORMAT is developed to generate mutants automatically and perform mutation analysis. The proposed operators are validated by using four open source programs having FSBs. The results indicate that the proposed operators are effective for testing FSBs.

Buffer overflow (BOF) is one of the major vulnerabilities that leads to non-secure software.Testing an implementation for BOF vulnerabilities is challenging as the underlying reasons of buffer overflow vary widely. Moreover, the existing vulnerability testing approaches do not address the issue of generating adequate test data sets for testing BOF vulnerabilities. In this work, we apply the idea of mutation-based testing technique to generate adequate test data set for BOF vulnerabilities. Our work addresses those BOF vulnerabilities, which are related to an implementation language and its associated libraries. We apply the concept for ANSI Clanguage and its associated libraries. We propose 12 mutation operators to force the generation of adequate test data set for BOF vulnerabilities. The proposed operators are validated by using four open source programs. The results indicate that the proposed operators are effective for testing BOF vulnerabilities.

Mutation analysis is a widely-adopted strategy in software testing with two main purposes: measuring the quality of test suites, and identifying redundant code in programs. Similar approaches are applied in hardware verification and testing too, especially at RTL or gate level, where mutants are generally referred as faults, and mutation analysis is performed by means of fault modeling and fault simulation. However, in modern embedded systems there is a close integration between HW and SW parts, and verification strategies should be applied early in the design flow. This requires the definition of new mutation analysis-based strategies that work at system level, where HW and SW functionalities are not partitioned yet. In this context, the paper proposes a mutation model for perturbing transaction level modeling (TLM) SystemC descriptions. In particular, the main constructs provided by the SystemC TLM 2.0 library have been analyzed, and a set of mutants is proposed to perturb the primitives related to the TLM communication interfaces.