Improve mutation testing in space software systems
Programme
GSTP
Programme Reference
GT17-463SW
Prime Contractor
UNIVERSITE DU LUXEMBOURG
Start Date
End Date
Status
Closed
Country
Luxembourg
Objectives
The objective of the activity is to improve mutation testing methodology and the corresponding toolset in the context of verification and validation of space software systems.
Description
The proposed activity aims at consolidating and extending the work performed within the previous GSTP activity G617-206SW - Applicability of Mutation Testing method for Flight Software. This activity concerned the investigation of mutation analysis as a method to evaluate the quality of software test suites and mutation testing as a method to derive new software test cases.
;
The proposed activity concerns the definition of supporting methods for the adoption of mutation analysis/testing within ECSS practices and the development of complementary approaches to overcome some limitations of the mutation testing approaches delivered by the previous activity.
;
By relying on the background delivered in the previous activity, the following tasks are foreseen for this activity:
- To identify a set of case study systems consisting of flight software and historical data about the development and maintenance of the software. These case studies should enable the evaluation of mutation analysis/testing with respect to fault finding capabilities, in the context of space software.
- To ensure generalisability of previous results, apply the code-driven and data-driven mutation analysis/testing approaches developed in the previous activity to the newly identified case study systems. The experiments should include an evaluation of the accuracy and effectiveness of the strategies addressing scalability problems (i.e. proportional uniform sampling, uniform fixed-size sampling, and FSCI).
- To provide guidelines for the definition of acceptance criteria for code/data-driven mutation analysis/testing. Identify a threshold for the mutation score for code/data-driven mutation analysis/testing that enables achieving certain quality objectives. Such study shall be conducted by analysing projects history; for example, to empirically determine a threshold above which the test suites of the case study systems would have detected all the field failures discovered in later stages.
- To perform the complementarity of data-driven and code-driven mutation testing based on quality objectives.
- To identify strategies to prioritise the inspection of code-driven/data-driven mutation analysis results based on quality objectives. It is reasonable to believe that certain mutants spot test suite weaknesses that are more severe than others (e.g., they prevent the detection of faults affecting the system).
- To investigate alternative methods for data-driven mutation testing. Ensure applicability to a large set of systems, the data-driven mutation testing technique developed by the previous activity relies on fault models captured through block models and mutation operators capable of altering array buffers. ;
- To survey test case generation techniques that may complement approaches based on symbolic execution with the objective to analyse and catalog test generation approaches that proved effective in generating test cases in scenarios other than unit testing.;
- To evaluate the applicability of test generation techniques complementing symbolic execution by:
- Prototyping a process for test case generation to support code-driven and data-driven mutation testing: identify the testing scenarios to be addressed, identify a set of applicable tools, define methods to configure the test generation techniques for the task, define methods to derive executable test cases based on tool results.
- Perform an assessment of a subset of the available techniques in terms of applicability, scalability, efficiency, and effectiveness based on project case studies.
- To develop a mutation testing toolset that complements the mutation testing toolset previously developed.;
;
Software shall be delivered under an ESA Software Community Licence, so that any individuals or entities within ESA Member States can access to it and can provide update to the community of users.
Application Domain
GEN-Generic Technologies
Technology Domain
2 - Space System Software
Competence Domain
3-Avionic Systems
Initial TRL
TRL 3
Target TRL
TRL 5
Achieved TRL
TRL N/A
Public Document
Final Presentation
Executive Summary