Proactive performance monitoring engine using predictive machine learning techniques for systems evaluated on space and non-space use cases

The objective of this activity is to augment the Syer correlation engine, developed and extended in previous GSTP activities, with predictive machine learning algorithms to create a proactive error detection system. 

In a previous GSTP activity G517-156GI "Leveraging System Performance Metrics and Execution Logs" a prototype correlation engine was developed to reduce misuse of resources (e.g. memory) in software systems limiting time consuming actions for the analyst to detect problems. It uses a technique to non-intrusively analyse system log files and metrics in combination to help locate issues. In a follow-on activity G517-161GI "System Performance Assessment with Data Mining and Process Mining Technique", the use of the engine (based on the Syer et al. algorithm) was extended to a real system. The use of data not only focused on technical performances but also on larger scope of performances like operational performances. Building on these results, the objective of this activity is to augment the correlation engine. The aim is to implement an enterprise application performance monitoring engine, based on machine learning algorithms, capable of detecting problematic application behavior and predicting short-term future performance. The application of predictive techniques in an engine that automatically combines application logfiles and performance counters is novel and will enable the use of predictive machine learning models in production systems. To accomplish the above, the following are required:Collection of data for representative use cases, including important failure modes, with data coming from operational ESA systems and own server systems/systems from another market segmentAssembly of the data into a standardised, annotated dataset that can be used to train machine learning algorithms on.Selection, implementation and testing of supervised machine learning algorithms, appropriate for the assembled dataset.Use the dataset to train the algorithm and generate predictive models for ESA systems.Use of the dataset to train the algorithm and generate predictive models for own server/other market segment systems.