Self-Supervised Foundation Models for Remote Sensing: Bridging Simulated and Real Imagery for Improved Generalization

Deep learning models for Earth Observation and remote sensing are constrained by the sensor-specific nature of real-world archives. These archives sparsely sample the wide range of different conditions of the atmosphere and sensor response functions. While the self-supervised learning (SSL) paradigm is a powerful tool to learn transferable representations from large, unlabeled datasets, representations are limited in that they are learned exclusively through real sensor archives that inherit physical biases. This in turn limits the generalization that can be done across different sensors and environment conditions. This study explores the physics-based self-supervised pre training to learn sensor-agnostic representations. This is achievable through the use of the Digital Imaging and Remote Sensing Generation (DIRSIG^TM) tool that parametrizes the imaging chain, generating synthetic images that span the underrepresented physical acquisitions conditions.

To date, we have a robust SSL baseline of pretraining a ResNet-50 encoder using Momentum Contrast (MoCo) v2 on a readily available Sentinel-2 Dataset called SSL4EO-S12. Then, a downstream evaluation pipeline was created utilizing the EuroSAT land cover classification benchmark, assessing the model using both linear probing and fine-tuning. Furthermore, we conducted initial experiments where the deep learning model was exposed to DIRSIG imagery that varied the visibility during the fine-tuning phase . Preliminary results show that there is a positive trend when exposed to controlled variability. The encoder is successfully learning the representations of intrinsic surface material properties rather than relying on instrument-specific artifacts.

Future work now focuses on directly building on these findings, expanding the DIRSIG^TM parameterization to encompass broader atmospheric conditions and multi-sensor characteristics. We aim to establish a robust, sensor-agnostic foundation model that is able to generalize seamlessly across diverse real world Earth observation data to conduct downstream tasks.