Vision-based Geo-Localization of Future Mars Rotocraft in Challenging Illumination Conditions using Deep Learning

About this technology

Overview

This work investigates deep learning-based methods to perform robust Map-based Localization (MbL) in challenging lighting. We proposed a novel multi-modal deep-learning framework that utilizes cross-attention mechanisms to fuse visual and depth data from ortho-projected maps and Digital Terrain Models (DTMs) obtained from orbital assets, effectively leveraging geometric context to learn illumination and scale invariance. To support training and validation, we developed a custom rendering framework to generate a synthetic Mars dataset with HiRISE ortho-images and DTMs, simulating aerial observations under varying lighting and altitude. Comprehensive evaluations show that our multi-modal method improves localization accuracy and robustness to a wide range of lighting offsets between maps and observations compared to single-modality models, including both deep-learning-based and traditional methods. Additionally, the integration of depth information has been shown to provide a degree of scale invariance and enhanced robustness to varying terrain morphology.