To optimise the performance, fidelity and personalisation level of AR/VR applications using AI methods and to optimise AI methods for image processing by Generating datasets from AR/VR environments to support improved training of Machine Learning algorithms

A number of AR/VR projects have been implemented so far within ESA. The main focus has been on the application of AR/VR/MR to human and robotic operations as well as teleoperations/ maintenance tasks and lately on Space Safety use cases. With an increasing volume of information being represented on AR/VR/MR applications, especially for use cases involving visualisation of large datasets (e.g. scientific data) and Big Data, choosing the right approach is critical to meet performance limitations and offer an intriguing experience with greater fidelity without overwhelming the user. AI techniques can significantly optimise the feed of data into the AR/VR/MR environment: from using AI for selective rendering where only the areas where the user is looking are dynamically generated in full visual precision to performing smart compression for wireless transmission. AI will thus enable the shift to more engaging and realistic experiences, and more powerful personalisation based on user interaction input (movement, natural language, decisions). At the same time, AI can benefit from AR/VR/MR applications by means of data generation. Machine Learning algorithms work best when trained on large datasets. However, in some application domains such datasets are not available or only in limited numbers. Examples are Visual Navigation, Localisation of Rovers, Image based telemetry extractions, etc. AR/VR/XR can serve as a good source for generation of representative datasets for such use cases, in order to train the algorithms before putting them in operational use. This is not limited to visual aspects but can be extended to data related to the interaction of the use/operator in the VR/AR/MR. Use cases can be:- Vision based in-situ planning using AI for real-time object tracking, environment mapping, highlighting and presenting danger conditions. - Recreation of accurate virtual planet environments based on images, using generative models, for more realistic training scenarios.- Collected datasets from multiple users? interactions in an AR/VR simulation can be used for automating a process and training e.g. a robotic asset to perform it. The activity encompasses the following tasks:- Consolidate the use cases in both directions (AI applied to AR/VR and AR/VR at the service of AI)- Assess the technology - Assess integration with and use of existing systems- Implement a PoC for both directions