Applicability of Mutation Testing method for Flight Software
Programme
GSTP
Programme Reference
G617-206SW
Prime Contractor
UNIVERSITE DU LUXEMBOURG
Start Date
End Date
Status
Closed
Country
Luxembourg
Objectives
The main objectives of the activity are:
- Searching for an alternative or complementary method of measuring the effectiveness of a test suite (against the typical verification done by hand).
- ;Searching for an alternative or complementary method to build a test suite (i.e. test data generation) against the typical approach of building them by hand.
- ;The approach could be used within the nominal SW development life cycle, or within an ISVV context.
;
Description
Test suites are typically designed and verified by hand. Having a systematic way of designing / verifying a test suite would be beneficial to improve the test quality of critical software. Mutation testing could be an alternative or complementary testing approach, that would help to improve the design amp; verification of a test suite. Mutation would provide a systematic and repeatable process for measuring the effectiveness of a test suite and it would provide a level of assurance beyond that offered by manual review.
;
It is suggested to have a 2-phased approach; a first phase to explore the technique and a second phase to consolidate the approach.
;
Phase 1: The objective is to study the applicability of mutation testing, and to prototype the approach, in the context of spacecraft flight software. In particular, the following tasks shall be done:
;
- Perform a comprehensive analysis and survey of mutation testing (e.g. state of the art of the approach, theories, trends, results, applications, empirical studies, evaluation of the tool chain).
- ;Evaluate the applicability of mutation testing to spacecraft flight software, by:
- ;Empirically evaluating mutation testing by applying it into a realistic flight software use case (e.g. a Boot SW, an Application SW, a set of low-level drivers), on SW that has gone formal testing/validation, and coded in a typical language used in space applications (e.g. Ansi C following Misra rules).
- ;Prototype the mutation testing process to be applied on spacecraft flight SW: identify most effective mutants types (first order vs high order, mutation set,..) , identify tools (e.g. academic tools, industry and open source tools), identify reduction techniques to mitigate the main drawbacks of the approach e.g. labour intensive amp; high level of computing.
- ;Evaluating the applicability, scalability, efficiency and effectiveness of the approach in the space domain (e.g. effort to put it in practice, potential to uncover errors).
- ;Evaluating ;how mutation testing can be integrated into a typical verification amp; validation life cycle of embedded flight software in the space domain (e.g. analysing the applicability to SW developed according to ECSS standards, identifying the testing campaign UT/IT/TS-RB Validations it would best fit, applicability in an ISVV context).
- ;Identifying limitations of the approach (e.g. whether it can be used as a standalone validation technique or only to profile an existing test-case suite, manual effort, tool chain limitations).
;
Phase 2: The objective is to define in detail the mutation testing approach as to be applied to embedded flight software in the space domain. In particular, the following shall be implemented:
;
- Define a systematic approach of applying mutation testing to spacecraft software.
- ;Establish the mutation testing process: definition of a base set of mutants, define reduction techniques to reduce human effort and machine time, address the languages most commonly used (i.e. C and ADA)
- ;Establish a simplified mutation test tool chain, based on open source tools.
- ;State on the applicability to ECSS developed SW: Usage on nominal SW life cycle, and usage on ISVV context.
- ;Identify trends in mutation testing (e.g. model-based mutation testing, generation of unit tests amp; oracles based on mutation), and identify improvements of the approach.
- ;Spread the technique among the different actors (i.e. space SW developers and ISVV practitioners).
;
;
(change ESA/IPC(2019)61)
It is proposed to change the budget and scope and duration of this activity.
;
It is proposed to assess the applicability of mutation testing, and to prototype the approach, in the context of spacecraft flight software. The Purpose is to perform a comprehensive analysis and survey of mutation testing (e.g. state of the art of the approach, theories, trends, results, applications, empirical studies, evaluation of the tool chain) and to assess how (or if) mutation test methodology can be applied to Flight Software Development.
;
Objectives and tasks of the first phase of the activity remain as described in the original description.
;
The objectives of the second phase of the activity are modified. The aim of this phase will be to define in detail the mutation testing approach as to be applied to embedded flight software in the space domain.
;
;
Application Domain
Generic Technologies
Technology Domain
2 - Space System Software
Competence Domain
3-Avionic Systems
Initial TRL
TRL 3
Target TRL
TRL 4
Achieved TRL
TRL N/A
Public Document
Executive Summary