Mutation testing tools

One important factor for the successful application of mutation is the availability of automated frameworks that support its application steps. In the following, we present the tools that were introduced or were used in the studies we surveyed [PapadakisKZJTH18]. The table outlines the corresponding tools along with the year of their creation, their application artefact and a concise description of key characteristics.

As it can be seen from the table, our analysis concluded in 76 tools, most of which where introduced between 2008 and 2017, that apply mutation to different software artefacts. By closely examining the table, it becomes obvious that there is an increasing growth in mutation testing tools with the creation of approximately 10 tools per year. Most of these tools target the implementation level languages but there are also tools that target specification languages and models.

At the implementation level, the mutation testing tools target mostly the C and Java programming languages. Most of the tools focus on the support of traditional, method-level mutant operators and strong mutation, with few tools supporting Object-Oriented operators and weak or higher order mutation. Additionally, there have been various tools proposed that apply mutation to dynamically-typed programming languages and concurrency-related aspects.

For the non-code-based tools, there have been proposed various tools for many model notations, including Extended and Timed Finite State Machines, Simulink models, Feature Models, etc, that automate the application of mutation. Furthermore, automated frameworks for mutating security policies and protocols have also been introduced.

Name Year Application Description
mutate [mutatepy] N/A C supports method-level mutant operators
Jester [jester] 2001 Java supports source-code-level (src-level) mutant generation
Proteum [Delamaro2001_proteum] 2001 C supports an extensive set of method-level mutant operators and interface mutation (inter-method level operators)
mutgen [andrews_TSE_03], [andrews_ICSE_05_mutgen] 2003 C supports method-level mutant operators
muJava [ma_ICSE_06_mujava], [MaOK05], [offutt04_mujava] 2004 Java implements src-level mutant generation and supports method-level and Object-Oriented (OO) mutant operators
ByteME [do_TSE_06] 2006 Java implements bytecode-level mutant generation and supports method-level & Object-Oriented (OO) mutant operators
SQLMutation [tuya_MUTATION_06_sqlmutation] 2006 SQL supports traditional & SQL-specific mutant operators for SQL queries
Jumble [jumble] 2007 Java implements bytecode-level mutant generation and supports method-level mutant operators
ESTP [feng_taic_part_08] 2008 C supports 20 traditional C mutant operators
Not Named [tan_ICST_08] 2008 Sulu supports method-level mutant operators (drawn from the study of Andrews et al. [andrews_ICSE_05_mutgen])
Milu [JiaH08a] 2008 C supports method-level mutant operators and higher order mutation
Not Named [fraser_SQJ_08] 2008 NuSMV models supports specification-based mutation (drawn from the study of Black et al. [BlackOY00])
Not Named [rajan_other_08] 2008 Code generated from Simulink models seeds faults into the implementations generated from Simulink models
Not Named [mouelhi_icst_08] 2008 Security Policies supports security-policy-access-control meta-model mutation (applied on policies defined in various notations (e.g. RBAC and OrBAC)
Not Named [weiglhofer_other_08] 2008 LOTOS specifications supports mutation testing for LOTOS specifications
Not Named [AnbalaganX08] 2008 AspectJ supports mutant operators for the creation of pointcut mutants that vary the strength of the corresponding point-cut in terms of the number of joint points it matches
Javalanche [schuler_FSE_09] 2009 Java implements bytecode-level mutant generation and supports method-level mutant operators and mutant classification based on mutants' impact
JDama [zhou_ICST_09], [zhou_STVR_11] 2009 SQL/JDBC implements bytecode-level mutant generation and supports SQL-related operators and weak mutation
AjMutator [delamare_ICST_09], [delamare_ICSTW_09] 2009 AspectJ supports mutant operators for AspectJ Point-cut Descriptors (PCDs) [FerrariMR08] and automated equivalent mutant detection
GAmera [dominguez_jimenez09_gamera] 2009 WS-BPEL supports mutation testing for WS-BPEL composition
Not Named [KaminskiA09a] 2009 boolean logic supports mutant operators for possible DNF logic faults
PASTE [andres_ICST_09], [andres_ICSTW_09], [andres_STVR_12] 2009 TFSM supports passive testing of systems presenting stochastic-time information using mutant operators specific to Timed Finite State Machices (TFSM)
Not Named [aydal_ICST_09] 2009 Z supports mutant operators for Z specifications
Not Named [dobolyi_ISSRE_09] 2009 GCC-XML supports a subset of mutant operators proposed by Ellims et al. [ellims_taic_part_07]
Not Named [lakehal_STVR_09] 2009 LUSTRE / SCADE supports mutant operators for LUSTRE/SCADE programs
Not Named [KnauthFF09] 2009 Java supports mutant operators that follow the fault classification of Duraes and Madeira [duraes_other_03]
PIT [coles_ISSTA_16] 2010 Java implements bytecode-level mutant generation and supports method-level mutant operators
MutMut [GligoricJM10] 2010 Java supports concurrency-related mutant operators
GenMutants [ZhangXZTHM10] 2010 .Net supports method-level mutant operators
Judy [madeyski_IET_10_judy] 2010 Java implements src-level mutant generation and supports method-level & OO mutant operators
webMuJava [PraphamontripongO10] 2010 HTML/JSP supports specific mutant operators for web components written in HTML and JSP languages
Bacterio [MateoUO10] 2010 Java supports method-level mutant operators for system-level testing using flexible weak mutation
Not Named [PapadakisM10a], [PapadakisMK10], [PapadakisM11b], [PapadakisM11a], [ Papadakis2013] 2010   supports method-level mutant operators
Major [just_ISSTA_14] 2011 Java supports method-level mutant operators
Paraμ [madiraju_ICSTW_11] 2011 Java supports OO & concurrency-related mutant operators and higher order mutation
ILMutator [DerezinskaK11] 2011 C# implements CLI-level mutant generation and supports method-level & OO mutant operators
SMutant [GligoricBJ11] 2011 Smalltalk supports traditional, method-level mutant operators in a dynamically typed language
MuBPEL [mubpel] 2011 WS-BPEL N/A
jMuHLPSL [dadeau_ICST_11] 2011 HLPSL supports mutant operators that introduce leaks in security protocols
Not Named [winbladh_ASE_11] 2011 SPADE mutates the flow pattern description of input and output streams and the SPADE code of components
Not Named [saifan_ICST_11] 2011 Aglets supports mutant operators specific to Mobile Agent Systems that affect the movement, communication, run method, creation, event listeners and agent proxy of an agent
Not Named [JustKS11] 2011 Java supports method-level mutant operators based on the selective mutation approach and higher order mutation
SMT-C [dan_ICSTW_12] 2012 C supports the semantic-related and method-level mutant operators
mutant ( muRuby) [mutant], [li_ICSTW_15] 2012 Ruby supports Ruby-specific mutant operators
Not Named [elrakaiby_icstw_12] 2012 Obligation Policies supports mutant operators specific to obligation policy enforcement
Not Named [PapadakisM12a] 2012   supports traditional, method-level mutant operators
CCMUTATOR [kusano_ASE_13] 2013 C/C++ supports concurrency-related mutant operators, higher order mutation and targets applications written using the PThreads and C++11 concurrency constructs
Comutation [GligoricZPP13] 2013 Java supports concurrency-related mutant operators [bradbury_Mutation_06]
SchemaAnalyst [kapfhammer_ICST_13] 2013 SQL supports mutant operators related to relational schema integrity constraints, applied to multiple database management systems
XACMUT [bertolino_mutation_13_xacmut] 2013 XACML supports mutant operators targeting XACML 2.0 security policies
Mutandis [MirshokraieMP13b], [Mirshokraie0P15a] 2013 JavaScript supports JavaScript-specific mutant operators
Not Named [ye_TSE_13] 2013 Web service compositions supports 2 types of mutant operators for web service compositions: one that is internal to the service and one that models inconsistencies across different services of the composition
Not Named [nguyen_ICST_13] 2013 Security Policies supports mutant operators specific to delegation policies based on a formal analysis of key delegation features
Not Named [henard_ICST_13] 2013 Feature Models supports mutant operators for mutating Feature Models
MutPy[mutpy] 2014 Python implements traditional and python-specific mutation operators
MuCheck [le_ISSTA_14] 2014 Haskell supports mutant operators targeting functional constructs and higher order mutation
HOMAJ [omar_ICSTW_14] 2014 AspectJ/Java supports higher order mutation
Not Named [mahajan_ICST_14] 2014 HTML/CSS supports mutant operators that seed presentation defects to web pages
Not Named [el_fakih_ICSTW_14] 2014 EFSM supports mutants that introduce Single Transfer Faults (STFs) and Double Transfer Faults (DTFs) to Extended Finite State Machices (EFSM) models
Not Named [maruchi_ICSTW_14] 2014 Data flow languages supports 2 mutant operators that model common mistakes when creating power plant control programs
MutaLog [HenardPT14] 2014 Logic Mutation supports mutant operators for mutating logic expressions
REDECHECK [walsh_ASE_15] 2015 HTML/CSS supports mutant operators for layout defects in responsive web sites
Not Named [abreu_SQJ_15] 2015 spreadsheets supports mutant operators for spreadsheets (spreadsheet mutation)
Not Named [belli_SQJ_15] 2015 FSM supports mutant operators for FSM specifications (based on the studies of Fabbri et al. [fabbri_ISSRE_94] and Petrenko et. al [petrenko_IET_09] )
Not Named [guan_ICSTW_15] 2015 Component-level sequence and state diagrams supports architecture- and design-level mutant operators
Not Named [ocariza_ICSE_15] 2015 HTML / JavaScript supports mutant operators for the Model-View-Controller frameworks of web application development
Not Named [NanavatiWHJK15], [WuNHJK17] 2015 C supports memory-related mutant operators that model memory faults and control flow deviation as a mutant-killing condition
Not Named [DengMAO15] 2015 Android apps supports android-specific mutant operators, affecting intents, events, activity lifecycle and XML files; and the method-level mutant operators of muJava
MoMut [krenn15_momut] 2015 UML models supports model-based mutation testing for UML state charts, class diagrams and instance diagrams
MuVM [TokumotoYSH16] 2016 C implements bitcode-level mutant generation and supports higher order mutation
Not Named [enoiu_ICST_16] 2016 FBD supports mutant operators for FBD language
Not Named [matinnejad_ICSE_16] 2016 Simulink supports mutant operators that model common Simulink fault patterns
Not Named [YoshidaTPGU16a] 2016 C++ supports mutant operators similar to the ones of PIT for the Java language
Not Named [HaririSCKM16] 2016 C implements LLVM-level mutant generation and supports method-level mutant operators typically used in other tools, e.g. Milu
Not Named [henard_ICSE_16], [PapadakisHHJT16] 2016 C supports traditional, method-level mutant operators
Not Named [ChekamPT16] 2016   supports traditional, method-level mutant operators
Vibes [devroey_ICSE_15], [devroey_ICSE_16] 2016 Transition Systems, Statechart Models Implements featured model-based mutation analysis and supports the Fabbri et al. [fabbri_ISSRE_94] operator set (both first order and higher order)
$μ$Droid [jabbarvand_FSE_17] 2017 Android apps Implements energy-aware mutation operators derived from a specific energy defect model
MDroid+[linares-vasquez_FSE_17] 2017 Android apps Implements mutation operators to test Android applications based on a specifically-designed Android fault model derived from manual analysis of various software artefacts
Not named[brown_FSE_17] 2017 Source Code Extracts mutation operators from changes made in the development history of projects in an attempt to produce more ``realistic'' mutants
LittleDarwin[Parsai2017] 2017 Java Supports method-level mutation operators, higher order mutation, mutant sampling and disjoint/subsuming mutant analysis
MuCPP[Delgado-PerezMP17] 2017 C++ Implements class-level, object-oriented mutation operators for C++ programs
MutRex[arcaini_ICSTW_17] 2017 Regular Expressions Implements mutation operators based on a specific fault model for regular expressions
BacterioWeb[usaola_ICSTW_17] 2017 Android apps Implements mutation operators for Android applications
Not Named[ChekamPTH17] 2017 C Supports method-level mutant operators
Not Named [WangXSZH17] 2017 C Implements method-level mutation operators and the AccMut approach [WangXSZH17] to reduce the cost of mutant execution
Not Named [LoiseDPPH17] 2017 Java Implements security-aware mutation operators