While important efforts are dedicated to system functional testing, very few works study how to test specifically security mechanisms, implementing a security policy. This paper introduces security policy testing as a specific target for testing. We propose two strategies for producing security policy test cases, depending if they are built in complement of existing functional test cases or independently from them. Indeed, any security policy is strongly connected to system functionality: testing functions includes exercising many security mechanisms. However, testing functionality does not intend at putting to the test security aspects. We thus propose test selection criteria to produce tests from a security policy. To quantify the effectiveness of a set of test cases to detect security policy flaws, we adapt mutation analysis and define security policy mutation operators. A library case study, a 3-tiers architecture, is used to obtain experimental trends. Results confirm that security must become a specific target of testing to reach a satisfying level of confidence in security mechanisms.

In model-based development of embedded real-time systems, statecharts are widely used for formal specification of their behavior, providing a sound basis for test generation and for detection of faults early in the development process. The paper presents a variety of new mutations for adequacy evaluation of tests used in validation of statecharts-based models. These mutations focus on key features of statecharts used in modeling of embedded systems: hierarchy, orthogonality and time expressions. We distinguish between two levels of tests adequacy. In the first, test results are expected to strictly follow the statecharts semantics. The second one takes into account possible deviations from this semantics based on typical implementation oriented decisions, e.g. mapping of orthogonal components into separate tasks executed concurrently. The considered mutations address both types of adequacy. In particular, we consider interleaving enforcing mutations, for testing of alternatives to the canonical "maximum parallelism" execution of statecharts.

This paper is devoted to advanced mutation operators for C# source code. They deal with object-oriented (OO mutations) and other complex features of the code. They require structural information about a code, unlike the standard mutations. Applicability of OO operators in C# is compared with those for other OO languages. Operators for specific features of C# language are also proposed. The detailed specification of operators can be provided in terms of pre- and post-conditions of a program transformation. Based on the operators’ specification, the generation of mutated C# programs can be automated.

Available soon...

Mutation is the practice of automatically generating possibly faulty variants of a program, for the purpose of assessing the adequacy of a test suite or comparing testing techniques. The cost of mutation often makes its application infeasible. The cost of mutation is usually assessed in terms of the number of mutants, and consequently the number of "mutation operators" that produce them. We address this problem by finding a smaller subset of mutation operators, called "sufficient", that can model the behaviour of the full set. To do this, we provide an experimental procedure and adapt statistical techniques proposed for variable reduction, model selection and nonlinear regression. Our preliminary results reveal interesting information about mutation operators.

Available soon...

Mutation testing traditionally involves mutating a program in order to produce a set of mutants and using these mutants in order to either estimate the effectiveness of a test suite or to drive test generation. Recently, however, this approach has been applied to specifications such as those written as finite state machines. This paper extends mutation testing to finite state machine models in which transitions have associated probabilities. The paper describes several ways of mutating a probabilistic finite state machine (PFSM) and shows how test sequences that distinguish between a PFSM and its mutants can be generated. Testing then involves applying each test sequence multiple times, observing the resultant output sequences and using results from statistical sampling theory in order to compare the observed frequency of each output sequence with that expected.

A set of mutation operators for SQL queries that retrieve information from a database is developed and tested against a set of queries drawn from the NIST SQL Conformance Test Suite. The mutation operators cover a wide spectrum of SQL features, including the handling of null values. Additional experiments are performed to explore whether the cost of executing mutants can be reduced using selective mutation or the test suite size can be reduced by using an appropriate ordering of the mutants. The SQL mutation approach can be helpful in assessing the adequacy of database test cases and their development, and as a tool for systematically injecting faults in order to compare different database testing techniques.

Available soon...

Software simulation remains the most used method for VHDL RTL functional verification. The functional verification process essentially consists of two parts. The first one is the functional qualification; the second one is the qualification- driven stimuli generation. Currently, the qualification and the generation tasks are iterative processes based on VHDL simulation which is dramatically time consuming. The simulation time increases with the circuits’ size and the required level of quality. In our previous works, we have proposed some approaches based on the mutation testing technique to evaluate and to improve functional validation quality. Now, to reduce this simulation time, we propose in this paper a new approach based on FPGA emulation. So, an hardware-software platform called “Meta-Mutant Testbench” is used to emulate mutants. Experimental results for some ITC’99 benchmark circuits show that our mutation emulator is about 20 times faster than classical software simulators; this speedup increases with the circuits’ size.