# MetaEarth: A Generative Foundation Model for Global-scale Remote Sensing
Image Generation

Zhiping Yu, Chenyang Liu, Liqin Liu, Zhenwei Shi, ,
and Zhengxia Zou∗,  Zhiping Yu, Chenyang Liu, Liqin Liu, and Zhenwei Shi are
with the Image Processing Center, School of Astronautics, Beihang University,
Beijing 100191, China, and with the State Key Laboratory of Virtual Reality
Technology and Systems, Beihang University, Beijing 100191, China, and also
with the Shanghai Artificial Intelligence Laboratory, Shanghai 200232, China.
Chenyang Liu is also with Shen Yuan Honors College of Beihang University,
Beijing 100191, China. Zhengxia Zou is with the Department of Guidance,
Navigation and Control, School of Astronautics, Beihang University, Beijing
100191, China. Corresponding author: Zhengxia Zou<EMAIL_ADDRESS>The work was supported by the National Natural Science Foundation of China
under Grant 62125102, the National Key Research and Development Program of
China (Grant No. 2022ZD0160401), the Beijing Natural Science Foundation under
Grant JL23005, and the Fundamental Research Funds for the Central
Universities.

###### Abstract

The recent advancement of generative foundational models has ushered in a new
era of image generation in the realm of natural images, revolutionizing art
design, entertainment, environment simulation, and beyond. Despite producing
high-quality samples, existing methods are constrained to generating images of
scenes at a limited scale. In this paper, we present MetaEarth - a generative
foundation model that breaks the barrier by scaling image generation to a
global level, exploring the creation of worldwide, multi-resolution,
unbounded, and virtually limitless remote sensing images. In MetaEarth, we
propose a resolution-guided self-cascading generative framework, which enables
the generating of images at any region with a wide range of geographical
resolutions. To achieve unbounded and arbitrary-sized image generation, we
design a novel noise sampling strategy for denoising diffusion models by
analyzing the generation conditions and initial noise. To train MetaEarth, we
construct a large dataset comprising multi-resolution optical remote sensing
images with geographical information. Experiments have demonstrated the
powerful capabilities of our method in generating global-scale images.
Additionally, the MetaEarth serves as a data engine that can provide high-
quality and rich training data for downstream tasks. Our model opens up new
possibilities for constructing generative world models by simulating Earth’s
visuals from an innovative overhead perspective.

###### Index Terms:

Generative foundation model, diffusion model, remote sensing, self-cascading
generation, unbounded generation.

##  1 Introduction

The ongoing advancement of generative foundation models has propelled natural
image generation techniques to a new level. These encompass text-to-image
generation [1, 2, 3], video generation [4, 5], image translation [6, 7, 8],
image editing [9, 10], etc, contributing to many practical applications,
including art design [11], training image augmentation [12, 13, 14], and
building generative world models [15, 16, 17].

Figure 1:  We propose MetaEarth, a generative foundation model that simulates
Earth’s visuals from an overhead perspective. MetaEarth shows powerful
capabilities on generating worldwide, multi-resolution, unbounded, and
virtually limitless remote sensing images. For more visualization and animated
results, please refer to our project page.

Despite the recent progress in image generation quality and diversity, the
scale of the scene generated is still constrained to daily human activity
scenarios, resulting in a single generated image with limited resolution and
limited information capacity. In this paper, we aim to push the boundaries of
generative models by extending image generation capabilities from localized
human daily scenes to a global scale. We employ an overhead remote sensing
perspective as a novel viewpoint to study this problem and present MetaEarth -
a foundational generative model capable of generating worldwide, multi-
resolution, unbounded, and virtually limitless remote sensing images.

In our research, we primarily face three challenges. The first challenge is
about model capacity. Generating world-scale images involves a wide range of
geographical features, including cities, forests, deserts, oceans, glaciers,
and snowfields. Even only within the category of cities, remote sensing images
vary significantly across different latitudes, climates, and cultural
landscapes. This diversity imposes substantial demands on the capacity of the
generative model. The second challenge involves the generation of images with
controllable resolution. In the overhead imaging process, different imaging
altitudes correspond to images with different resolutions, resulting in
significant differences in the details of geographical features. However,
there are currently few studies on image generation with controllable
resolution. Therefore, generating remote sensing images at specified
resolutions (meters per pixel) from any geographic location is a highly
challenging task. The third challenge involves the generation of unbounded
images. Unlike everyday natural images, remote sensing images are uniquely
characterized by their vast size, often measuring tens of thousands of pixels
in length and width. Current natural image generation algorithms are generally
limited to producing images of sizes such as 512$\times$512 pixels or
1024$\times$1024 pixels. Therefore, in addition to addressing the first two
challenges, generating continuous, unbounded images of arbitrary size remains
a significant unsolved problem in existing methods.

To address the challenge of model capacity, we constructed a world-scale
generative foundation model with over 600 million parameters based on the
denoising diffusion paradigm. To train the model across diverse spatial
resolutions globally, we collected a large-scale remote sensing image dataset
comprising images and their geographical information (latitude, longitude, and
resolution) from multiple spatial resolutions that cover most regions
globally. To generate cross-resolution and controllable remote sensing images,
we propose a novel self-cascading generative framework. Different from
previous single-stage generation methods [1, 2], the proposed framework
generates multi-resolution images from a given geographical location in a
recursive manner. In different cascading stages, images are generated
sequentially from low to high resolutions. Specifically, we constructed a
unified model across different cascading stages, where in each stage, the
previously generated low-resolution image, as well as the geographical
resolution, are embedded as controllable generative variables to guide the
generation of images at higher resolutions. As the cascading stages
accumulate, the generated images manifest diversity in both resolution and
content, enabling the generation of varied parallel scenarios.

To generate unbounded and arbitrary-sized images, a straightforward idea is to
stitch multiple generated image blocks into a larger image. However, due to
the inherent stochasticity of generative models, directly stitching
independently generated image blocks can result in visually discontinuous
seams between the blocks. To solve this problem, we design a novel noise-
sampling strategy for denoising diffusion models by analyzing the generation
conditions and initial noise. The proposed strategy ensures consistency in
style and semantics between generated image tiles, enabling the generation of
arbitrary-sized images from smaller pieces.

We conducted comprehensive experimental analyses on MetaEarth, including both
quantitative and qualitative evaluations. Experiment results have demonstrated
that MetaEarth achieves the generation of high-quality images with broad
coverage, multiple resolutions, and diverse image contents. The ablation
studies also suggest the effectiveness of the above-mentioned methodology
design. Additionally, MetaEarth can serve as a data engine, providing high-
quality and diverse training data for downstream tasks. In our experiments,
choosing image classification as an example, we demonstrated that the high-
quality generated samples provided by MetaEarth can significantly improve
classification accuracy.

The contributions of this paper are summarized as follows:

  * •

We propose MetaEarth, a novel generative foundation model for remote sensing,
exploring the creation of worldwide, multi-resolution, unbounded, and
virtually limitless remote sensing images. To the best of our knowledge,
MetaEarth is the first foundational model to extend image generation to a
global scale.

  * •

We proposed a resolution-guided self-cascading generative framework and a
noise sampling strategy to address the challenges faced by previous
foundational models in generating cross-resolution and continuous unbounded
images.

  * •

As a generative data engine, MetaEarth has the potential to provide virtual
environments and realistic training data support for various downstream tasks
in remote sensing and Earth observation. Our model also opens up new
possibilities for constructing generative world models [15, 16, 17] by
simulating Earth’s visuals from an innovative overhead (bird’s-eye view)
perspective.

##  2 Related work

In this section, we briefly review the recent advancements in visual
generative models, probabilistic diffusion models for image generation, and
image generation models in the field of remote sensing.

###  2.1 Visual Generative Models

In recent years, image generation technology has attracted considerable
attention in computer vision and machine learning. Image generation models aim
to capture underlying patterns from real image data distributions and
synthesize realistic images. With the development of deep learning-based
generative models, significant progress has been made in recent years.
Existing image generation models fall into four main categories: Generative
Adversarial Networks (GANs) [18, 19, 20], Variational Autoencoders (VAEs) [21,
22], Flow-based models [23], and Diffusion models [24, 25, 1].

GANs, consisting of a generator and a discriminator trained simultaneously,
produce high-quality synthetic data resembling real data but struggle with
diversity and stable training due to mode collapse. VAEs, inspired by
variational Bayes inference, map input data to latent space, generating
diverse samples and providing uncertainty estimates, though often producing
blurry reconstructions. Flow-based Models, which learn invertible
transformations for exact likelihood computation and efficient sampling, face
challenges in modeling complex data distributions, making them unsuitable for
global image generation. Diffusion Models, using a noise-corruption and
reverse process, have shown remarkable performance in generating high-quality
images despite slow generation speeds. In this paper, we leverage diffusion
models to generate global remote sensing images.

###  2.2 Diffusion-Based Image Generation

Diffusion-based image generation can be divided into two categories [26]:
unconditional generation and conditional generation.

An unconditional generation model refers to a model that generates images
without depending on any form of control, operating entirely in a random
sampling manner. The most representative work is the Denoising Diffusion
Probability Model (DDPM) proposed by Ho et al. [24]. The DDPM builds upon the
theoretical foundation established by Sohl-Dickstein et al. [27], operating by
reversing a diffusion process. This method disrupts real data during the
forward process, adding noise to the data in small steps until it becomes
Gaussian noise, then, learns to reverse these steps to generate data from
noises. Based on DDPM, Nichol et al. [25] propose learning variance to improve
model performance. This design not only enhances the quality of generated
images but also accelerates sampling speed. To further accelerate sampling,
Song et al. propose the Denoising Diffusion Implicit Model (DDIM) [28],
modeling the diffusion process as a non-Markovian process. Without requiring
adjustments to the training process of DDPM, the DDIM generation framework
significantly speeds up the sampling procedure with only a small impact on
quality. Additionally, the DDIM framework enables a deterministic generation
process, ensuring that generated images depend solely on the initial noise of
the reverse process.

The conditional diffusion model refers to a class of models that generate
images based on additional signals, building upon the unconditional diffusion
model. According to the type of input conditions, they can be divided into
class-to-image generation [29], image-to-image synthesis [6, 7, 30, 31], text-
to-image generation [1, 3], image editing [9, 10] etc. Since DDPM’s reverse
process directly operates in pixel space, more computation and memory are
required to predict high-dimension noise, making it challenging for the model
to generate high-resolution images. There are two main approaches to
addressing this issue. The first is cascaded generation [29, 3], where low-
resolution images are generated first and then used as conditional inputs to
generate higher-resolution images sequentially. The second approach is latent
space generation [1], using an encoder-decoder network structure to compress
the image into a lower-resolution latent space, followed by a denoising
diffusion process within the latent space.

In this paper, we adopt a cascaded generation strategy, but with a novel
approach that differs from previous methods. Firstly, previous models require
designing and training multiple models separately for each resolution stage,
which inevitably increases training costs. In contrast, our model shares the
same network weights across different stages, using resolution embedding
encoding to guide the learning of variable-resolution information.
Additionally, while previous methods typically impose size constraints on
input and output images at each stage, our approach does not have such
restrictions, offering greater flexibility.

###  2.3 Remote Sensing Image Generation

In the field of remote sensing, image generation models have recently garnered
increasing attention. Previous research on this topic mainly focuses on text-
to-image and image-to-image tasks. For the text-to-image generation task, one
common approach is to fine-tune latent diffusion models [1] directly based on
remote sensing text-image datasets [32, 33, 34]. The image-text data pairs are
usually obtained by a pre-trained image captioning model and then are refined
manually assisted by GPT-4. In addition to the direct fintuning, Sebaq et al.
[35] propose a generation framework containing two individual diffusion
models, where the first model generates images from prompts and the second one
upscales the generated images. For the image-to-image generation task, the
input conditional images usually include maps [36] and semantic layouts [37,
38, 39].

Despite some attempts at image generation in the field of remote sensing,
previous methods have been typically trained on small-scale or limited
datasets, lacking diversity in land cover types and resolutions. Additionally,
these methods have only been capable of generating remote sensing images up to
512$\times$512 pixels in size, with few attempts made at generating images for
larger-scale scenes. In this paper, we thoroughly explore these issues by
designing and training a foundational generative model for remote sensing,
achieving promising results.

Figure 2:  The overall structure of MetaEarth. We propose a resolution-guided,
self-cascading framework which is capable of generating scenes and resolutions
for any global region. The generation process unfolds in multiple stages,
starting with low-resolution images and advancing to high-resolution images.
In each stage, the generation is conditioned on the low-resolution images
generated in the preceding stage and their spatial resolution.

###  2.4 Remote Sensing Foundation Model

The foundation models have recently been preliminarily studied in the field of
remote sensing, emerging as a new research direction. These models are per-
trained on large amounts of data and can be adapted to various downstream
tasks through fine-tuning, few-shot learning, and zero-shot learning. Current
research on foundation models in remote sensing mainly focuses on two aspects:
vision-centric foundation models and vision-language foundation models.

In earlier years, vision-centric foundation models in remote sensing are
typically trained using supervised learning. However, these models are
constrained by the limited availability of supervised training data. Recently,
self-supervised techniques have become mainstream, as they can leverage large
amounts of unlabeled data. The self-supervised pre-trained foundation models
are designed to learn robust feature representations from unlabeled remote
sensing images. They can then be transferred to various downstream tasks, such
as image classification, object detection, and segmentation, resulting in
improved performance. These models predominantly rely on the Masked
Autoencoders (MAE) architecture[40] [41, 42, 43] and are tailored to exploit
the distinctive features inherent in remote sensing imagery, including image
content [44], multi-scale properties [45], temporal dynamics [46], and
multispectral attributes [47, 42]. Besides, the improvement of the model’s
structure and training paradigm [48] is used to enhance the model’s capacity
for the representation of remote sensing data.

Remote sensing vision-language foundation models are trained on a large amount
of image-text paired data. Some methods [49, 50, 51] adopt contrastive
learning, which aims to train an image encoder and a text encoder to align the
visual and linguistic domains. The learned aligned vision-language
representations can be used for many downstream tasks, such as zero-shot
classification [49], image-text retrieval [52, 53], and image captioning [54,
55]. For example, Zhang et al. [50] filtered remote sensing image-text pairs
from publicly available large datasets to build the RS5M dataset and fine-
tuned the CLIP [56] model to obtain a GeoRSCLIP model. Recently,
autoregressive pre-training has been widely used for training vision-language
models, such as [57, 58, 59, 60]. Most methods utilize CLIP to extract visual
features and design connectors to map these visual features into large
language models for predicting corresponding text. They typically start with
image-text pre-training and then further refine the model through instruction
fine-tuning.

Although the aforementioned work has made preliminary explorations in
foundational models for remote sensing, research on generative foundational
models in this field is still relatively scarce. In this paper, we propose
MetaEarth, a novel generative foundation model for global-scale remote sensing
image generation, which expands the research boundaries of current remote
sensing foundation models.

##  3 METHODOLOGY

In this section, we will first revisit the Denoising Diffusion Probabilistic
Model. Then, we introduce the proposed self-cascading generation framework and
unbounded image generation method.

###  3.1 Revisit of Denoising Diffusion Probabilistic Model

The Denoising Diffusion Probabilistic Model (DDPM) [24] is a type of
generative model that leverages a diffusion process for image generation. It
involves a forward process, where the image data is gradually corrupted with
noise, and a reverse process, where a neural network learns to denoise the
noisy image step-by-step to recover its original content. Through the reverse
process, high-quality images can be generated by iteratively refining noisy
samples, leading to impressive results in various image synthesis tasks.

Suppose $x_{0}$ denotes a clean image. During the forward process, a set of
Gaussian noises are added to $x_{0}$ gradually according to a variance
schedule ${\beta_{1},\beta_{2},...,\beta_{T}}$ over $T$ time-steps:

$\displaystyle q({x_{t}}|{x_{t-1}})$
$\displaystyle:=\mathcal{N}({x_{t}};\sqrt{1-{\beta_{t}}}{x_{t-1}},{\beta_{t}}I)$
(1a) $\displaystyle q({x_{1},x_{2},...,x_{T}}|{x_{0}})$
$\displaystyle:=\prod_{t=1}^{T}{q({x_{t}}|{x_{t-1}})}$ (1b)

where $q({x_{1},x_{2},...,x_{T}}|{x_{0}})$ represents the joint probability
distribution of $x_{1},x_{2},...,x_{T}$ conditioned by $x_{0}$.
$q({x_{t}}|{x_{t-1}})$ represents the conditional distribution from $x_{t-1}$
to $x_{t}$ in a one-step diffusion processing. Since the forward process of
DDPM operates as a Markov chain, according to Eq. (1), by setting
${{\alpha_{t}}=\prod_{i=1}^{t}(1-\beta_{i})}$, the conditional distribution of
${x_{t}}$ can be directly represented as a multivariate Gaussian distribution:

$q({x_{t}}|{x_{0}})=\mathcal{N}(x_{t};\sqrt{\alpha_{t}}{x_{0}},(1-{\alpha_{t}})I)$
(2)

By utilizing the reparameterization trick and introducing a noisy image layer
$\epsilon$ sampled from a Gaussian distribution,
${{\epsilon}\sim{\mathcal{N}(\bm{0},I)}}$, the relationship between ${x_{t}}$
and ${x_{0}}$ can be written as a one-step blending operation:

$x_{t}=\sqrt{\alpha_{t}}{x_{0}}+\sqrt{1-\alpha_{t}}\epsilon$ (3)

Given the forward process described by Eq. (3), one can easily prove that if
the scheduling parameters ${\beta_{t}}$ are properly set, when $T$ is
sufficiently large, the distribution of ${x_{T}}$ can be approximated to a
standard multivariate Gaussian distribution, i.e.,
${{q(x_{T}|x_{0})}\sim\mathcal{N}(0,I)}$.

In the reverse process, starting with
${p(x_{T})\sim{\mathcal{N}({x_{T}};\bm{0},I)}}$, we aim at recovering the
$x_{0}$ step-by-step through a denoising processing. In each denoising step, a
trainable network with parameters ${\theta}$ is trained to estimate the
distribution ${q(x_{t-1}|x_{t})}$:

$\displaystyle p_{\theta}({x_{t-1}}|{x_{t}})$
$\displaystyle:=\mathcal{N}({x_{t-1}};{\mu_{\theta}(x_{t},t)},{\Sigma_{\theta}(x_{t},t)})$
(4a) $\displaystyle p_{\theta}({x_{0},x_{1},x_{2},...,x_{T}})$
$\displaystyle=p(x_{T})\prod_{t=1}^{T}{p_{\theta}({x_{t-1}}|{x_{t}})}$ (4b)

According to [24], the model can be trained by optimizing the Variational
Lower Bound(VLB):

$\displaystyle L_{vlb}$
$\displaystyle=\mathbb{E}_{q}\\{{L_{T}}+L_{t-1}+\cdots+L_{0}\\}$ (5a)
$\displaystyle L_{T}:$
$\displaystyle=\text{D}_{\rm{KL}}({q(x_{T}|x_{0})}\parallel{p(x_{T})})$ (5b)
$\displaystyle L_{t-1}:$
$\displaystyle=\text{D}_{\rm{KL}}({q(x_{t-1}|x_{t},x_{0})}\parallel{p_{\theta}(x_{t-1}|x_{t}})),\
t>1$ (5c) $\displaystyle L_{0}:$
$\displaystyle=-{\rm{log}}{p_{\theta}({x_{0}}|{x_{1}})}$ (5d)

where $\text{D}_{\rm{KL}}(p\parallel q)$ represent the Kullback–Leibler
divergence between two distributions $p$ and $q$. In Eq. (5),
${q(x_{t-1}|x_{t},x_{0})}$ is tractable:

$\displaystyle q(x_{t-1}|x_{t},x_{0})$
$\displaystyle=\mathcal{N}(x_{t-1};\tilde{\mu}_{t}(x_{t},x_{0}),\tilde{\beta}_{t}{I})$
(6a) $\displaystyle\tilde{\mu}_{t}(x_{t},x_{0})$
$\displaystyle={\frac{\sqrt{\alpha_{t-1}}\beta_{t}}{1-\alpha_{t}}x_{0}}+{\frac{\sqrt{1-\beta_{t}}(1-\alpha_{t-1})}{1-\alpha_{t}}x_{t}}$
(6b) $\displaystyle\tilde{\beta}_{t}$
$\displaystyle=\frac{1-\alpha_{t-1}}{1-\alpha_{t}}\beta_{t}$ (6c)

Assume that the variance ${\Sigma_{\theta}(x_{t},t)}$ in the above denoising
diffusion process is a constant-controlled diagonal matrix
${{\Sigma_{\theta}(x_{t},t)}={\sigma_{t}^{2}}I}$, in this way, the training
can be re-formulated solely to estimate ${\mu_{\theta}}$:

${p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(x_{t-1};{\mu_{\theta}(x_{t},t)},{\sigma_{t}^{2}}I})$
(7)

Then, ${L_{t-1}}$ can be re-written as:

$L_{t-1}=\mathbb{E}_{q}\\{\frac{1}{2{\sigma_{t}^{2}}}\parallel{\tilde{\mu}_{t}(x_{t},x_{0})-\mu_{\theta}(x_{t},t)}\parallel_{2}^{2}\\}+C$
(8)

where $C$ is a constant that does not participate in the optimization process
and can be omitted.

By introducing a noise prediction network ${\epsilon_{\theta}}$, combined with
Eq. (3) and Eq. (6), ${\mu_{\theta}}$ can be expressed as:

$\mu_{\theta}(x_{t},t)=\frac{1}{1-\beta_{t}}(x_{t}-\frac{\beta_{t}}{\sqrt{1-\alpha_{t}}}\epsilon_{\theta}(x_{t},t))$
(9)

Therefore, by setting ${{\sigma_{t}^{2}}=\beta_{t}}$, based on Eq. (8) and Eq.
(9), we can finally obtain the training objective function as:

$L_{t-1}=\mathbb{E}_{x_{0},\epsilon}\\{\frac{1}{\lambda_{t}}\parallel{\epsilon-\epsilon_{\theta}({\sqrt{\alpha_{t}}x_{0}}+{\sqrt{1-\alpha_{t}}\epsilon},t)}\parallel^{2}\\}$
(10)

where $\lambda_{t}=(1-\beta_{t})(1-\alpha_{t})/\beta_{t}$ is a time-variant
scaling factor on different loss terms through different denoising steps.

###  3.2 The MetaEarth Self-Cascaded Generation Framework

We propose a resolution-guided self-cascading framework for generating various
scenes and resolutions. Fig. 2 shows the overall structure of the proposed
MetaEarth. This framework enables recursive enhancement of image resolution
using a unified generation model.

The entire generation process consists of a series of stages, progressing from
low-resolution images to high-resolution images. In each generation stage, the
model is conditioned on the previously generated low-resolution images and
their corresponding spatial resolutions. The low-resolution images provide
scene categories and semantic information, while spatial resolution
information enables the model to perceive and represent features of images at
different scales.

Suppose $x_{0}^{(k)}$ and $s^{(k)}$ represent the clean image and the spatial
resolution (m/pixel) at the $k$-th generation stage. $x_{0}^{(k)}$ and
$s^{(k)}$ are embedded as conditional variables when generating images at
$(k+1)$-th stage. Assuming that at each generation stage, the model increases
the resolution of the input image by a factor of $N$, e.g. $N=4$, if the image
generated at stage $k$ has a size of $(H\times W)$ pixels, then at stage
$k+1$, we will obtain a high-resolution image with a size of $(NH\times NW)$.

Assuming there are no memory constraints, by repeating the aforementioned
operation $m$ times based on stage $k$, we will ultimately obtain a series of
images with varying resolutions and pixel sizes:

$\mathcal{X}=\\{x_{0}^{(k)},x_{0}^{(k+1)},\cdots,x_{0}^{(k+m)}\\}$ (11)

where the image $x_{0}^{(k+m)}$ generated from $(k+m)$-th stage will
eventually has a size of $(N^{m}H\times N^{m}W)$ pixels. To achieve the above
process with reasonable memory and computational costs, we designed a sliding
window generation process and a noise sampling strategy that enables the
generation of continuous, unbounded scenes in a memory-efficient manner. These
approaches will be detailed in Subsection 3.3.

In each generation stage, both the previously generated image and the time-
resolution embeddings are utilized as conditional input variables. For the
conditional input $x_{0}^{(k)}$, whose dimensions mismatched with
$x_{t}^{(k+1)}$, inspired by [30] and [61], we redesigned their network
architecture. We begin by encoding the low-resolution image $x_{0}^{(k)}$ with
an encoder $E_{lr}$. Subsequently, we use a set of upsampling and convolution
layers $F_{up}$ to align the dimensions of the feature maps with
$x_{t}^{(k+1)}$. Lastly, we merge the features by concatenating
$x_{t}^{(k+1)}$ with the feature maps along the channel dimension.

$\tilde{x}_{t}^{(k+1)}=\text{cat}[x_{t}^{(k+1)},F_{up}(E_{lr}(x_{0}^{(k)}))]$
(12)

where $\text{cat}(\cdot)$ represents the concatenation operation between two
tensors along their channel dimension.

For the spatial resolution $s$, we firstly encode it through a frequency
encoding transformation:

$\begin{split}f_{\omega}(s^{(k)})=[&\cos(s^{(k)}f_{1}),\sin(s^{(k)}f_{1}),\cos(s^{(k)}f_{2}),\sin(s^{(k)}f_{2}),\\\
&\cdots,\cos(s^{(k)}f_{n}),\sin(s^{(k)}f_{n})]\end{split}$ (13)

where the frequency factors are defined as:

$f_{i}=\exp(-\frac{i}{n}\log(\omega))=\omega^{-\frac{i}{n}},i=1,2,\cdots,n$
(14)

To let the model perceive more subtle resolution changes, we set ${\omega}$ as
a large number, $\omega=10^{4}$. Then, the encoded resolution is further
projected by a trainable Multi-Layer Perception network $F_{mlp}$.The complete
encoding process for ${s}$ can be represented as follows:

$e_{s}^{(k)}=F_{mlp}(f_{\omega}(s^{(k)}),\theta_{s})$ (15)

where $\theta_{s}$ are the learnable parameters of the $F_{mlp}$.

Through the above process, we obtain the spatial resolution embedding at
$k$-th generation stage $e_{s}^{(k)}\in{\mathbb{R}^{D}}$. Similarly, the time-
step variables $t\in\\{1,\dots,T\\}$ during the denoising process can be also
encoded through the frequency encoding transformation $f_{\omega}(t)$ and then
projected into a $D$-dimensional embedding vector with an additional MLP:

$e_{t}=F_{mlp}(f_{\omega}(t),\theta_{t})$ (16)

where $\theta_{t}$ are the learnable parameters. We finally add $e_{s}^{(k)}$
and $e_{t}$ together to produce the final conditional embedding vector at the
$t$-th denoising step from the $k$-th generation stage:

$e_{t}^{(k)}=e_{s}^{(k)}+e_{t}$ (17)

Given the conditional embedding vector $e_{t}^{(k)}$ all together with the
conditional image features $\tilde{x}_{t}^{(k)}$, we uniformly express them as
the condition variables of the $k$-th generation stage and $t$-th denoising
step:

$c_{t}^{(k)}=\\{e_{t}^{(k)},\tilde{x}_{t}^{(k)}\\}$ (18)

Based on Eq. (4) while maintaining the forward process unchanged, we
incorporate the condition $c_{t}^{(k)}$ as input to build a conditional
denoising model:

$\displaystyle p_{\theta}(x_{0:T}|c_{t}^{(k)})$
$\displaystyle=p(x_{T})\prod_{t=1}^{T}{p_{\theta}(x_{t-1}|x_{t},c_{t}^{(k)}})$
(19a) $\displaystyle p_{\theta}(x_{t-1}|{x_{t}},c_{t}^{(k)})$
$\displaystyle=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t,c_{t}^{(k)}),\sigma_{t}^{2}{I})$
(19b)

Through the aforementioned design, we can finally generate images of different
resolutions in a self-cascading manner driven by resolution and time-step
encodings.

###  3.3 Unbounded Image Generation

To achieve the generation of large-scale remote sensing images of arbitrary
sizes, we propose an unbounded image generation method including a memory-
efficient sliding window generation pipeline and a noise sampling strategy.
Figure 3 illustrates the proposed method.

Figure 3: To generate unbounded images, firstly, the generated image from the
previous stage is cropped into overlapping image blocks as conditions. Then,
with the proposed noise sampling strategy, the shared regions between adjacent
image blocks generate similar content. Lastly, the generated images are tiled
and re-organized.

In the self-cascading generation framework, we crop the conditional input
images into a set of image blocks to control the memory overhead within an
acceptable range. However, simple cropping and stitching will result in
noticeable seams at junctions. We analyze the occurrence of seams from two
factors in generative diffusion models: the conditions and the noise sampling.

From the perspective of generation conditions, when the receptive field of the
model extends beyond the image boundaries, the generated content may differ
semantically from adjacent images. To address this issue, we consider blocks
as sliding windows with overlapping of 1/2 window area so that pixels
originally located at the boundaries are repositioned as central areas. The
overlapped regions serve as semantic transition blocks between adjacent images
so that the semantic discontinuity between image blocks during stitching can
be solved.

Additionally, the initial noise sampling is another crucial factor affecting
the continuity between tiles. Due to the randomness of the noise sampling, the
generated content within the overlapping regions may vary, resulting in miss-
alignment of between adjacent blocks. To mitigate pixel-level discrepancies,
we propose a noise sampling strategy that, without modifying the model
training process, generates identical or near-identical content in the
overlapping regions, ensuring pixel-level continuity.

During the reverse process of conditional diffusion, we adopt the sampling
method of DDIM [28]. The denoising sampling process from $x_{t}^{(k)}$ to
$x_{t-1}^{(k)}$ is represented as follows:

$\begin{split}x_{t-1}^{(k)}=\sqrt{\alpha_{t-1}}(\frac{{x_{t}}^{(k)}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}(x_{t}^{(k)},c_{t}^{(k)})}{\sqrt{\alpha_{t}}})\\\
+\sqrt{1-\alpha_{t-1}-{\sigma_{t}^{2}}}\cdot{\epsilon_{\theta}(x_{t}^{(k)},c_{t}^{(k)})}+{\sigma_{t}}{\epsilon_{t}}\end{split}$
(20)

where ${{\epsilon_{t}}\sim{\mathcal{N}(0,I)}}$, and
${{\sigma_{t}}={\eta}\sqrt{(1-\alpha_{t-1})/(1-\alpha_{t})}\sqrt{1-{\alpha_{t}}/\alpha_{t-1}}}$.
${\eta}$ is a predetermined factor ranging from 0 to 1. When ${\eta=1}$ the
DDIM sampling process is identical to DDPM. When ${\eta=0}$, the denoising
process is fully determined, and the generated image is solely determined by
the initial noise and the condition $c_{T}^{(k)}$. In our method, we set
${\eta=0}$, that is, model it as a completely deterministic sampling process.
In this setting, since $x_{0}^{(k)}$ is solely determined by the initial noise
${x_{T}}$ and the condition $c_{T}^{(k)}$, we represent the generation process
using an injective function:

$\displaystyle x_{0}^{(k)}=f_{\theta}(x_{T},c_{T}^{(k)})$ (21)

Therefore, if the receptive field of the denoising network is smaller than the
overlapped region, with the same condition $c_{T}^{(k)}$ and the same initial
noise at the overlapping regions, we can easily prove that the generated
images at the center of the overlapped region will be strictly same.

Figure 4: Necessary conditions on initial noise sampling for generating
continuous unbounded scenes.

Based on the above analysis, as shown in Fig. 4, to ensure that vertically
adjacent image blocks generate similar content in their overlapping regions,
we need their initial Gaussian noise to satisfy the conditions ${A_{3}=C_{1}}$
and ${A_{4}=C_{2}}$. Similarly, to ensure that horizontally adjacent image
blocks generate similar content in their overlapping regions, we need
${A_{2}=B_{1}}$ and ${A_{4}=B_{3}}$. To make sure that all the above
conditions are always true during the generation process, we can first sample
for one block, and then make copies for the corresponding neighboring blocks.
To further simplify the sampling process, in our implementation, we set all
blocks with the same initial noise, i.e., ${N_{1}=N_{2}=N_{3}=N_{4}}$, where
$N=\\{A,B,C\\}$.

###  3.4 Implimentation Details

####  3.4.1 Denoising Network Design

For the dataset constructed above, we set the model to increase the input
image resolution by a factor of $N=4$. For the noise schedule design, we adopt
a linear schedule ranging from a minimum of 0.0015 to a maximum of 0.0155.
During training, the number of sampling steps is set to 1000, while during
inference, we utilize the DDIM acceleration strategy with the number of
sampling steps set to 50.

We design a U-Net-like architecture to predict noise, with a total number of
about 600 million parameters. The encoder and decoder of the network are
composed of 5 blocks each. $\tilde{x}_{t}^{(k+1)}$ undergoes 4 rounds of
2$\times$ downsampling/upsampling. The number of channels in each block is
multiplied by a factor of [1,2,4,8,8] based on the base number of channels.
Each block contains 3 ResBlocks, with AttentionBlocks included specifically
for blocks with an 8-fold channel multiplier. We utilized RRDBNet [62] as the
encoder for conditional images.

For each feature map $h$ within every ResBlock, we divide the embedding
$e_{t}^{(k)}$ into two parts using the chunk function in PyTorch and then
scale the feature map $h$ with the chunked embeddings:

$\begin{split}&e_{\text{scale}},e_{\text{shift}}={\texttt{chunk}}(e_{t}^{(k)},2)\\\
&h\leftarrow{(1+e_{\text{scale}})}\cdot h+e_{\text{shift}}\end{split}$ (22)

With the above design, we can integrate both low-resolution input and spatial
resolution to predict the noise ${\epsilon_{\theta}}$ for each timestep.

Figure 5: Images of various land features across the globe generated by
MetaEarth, including water bodies, mountains, deserts, farmland, cities, and
countryside areas. Figure 6: MetaEarth-generated city images with different
continental styles. We use large-scale low-resolution images of 256m/pixel in
the Americas, Europe, and Asia as condition guidance, and the model can
generate urban images consistent with the regional style.

####  3.4.2 Training Details

When training a single model for cascaded generation, two main challenges
arise: Firstly, high-resolution and low-resolution images are from different
sensors and are captured under varying spatiotemporal conditions. This results
in significant differences in style, content, and details between pairs of
high and low-resolution images, leading to potential mismatches between image
pairs. Secondly, during the iterative inference process of self-cascading
generation, each input image to the generation model in the cascading steps
originates from the model’s previous output. However, the input data utilized
during model training may differ in distribution from the input data during
inference. This distributional shift may result in distorted or unreasonable
image generation results. To address the aforementioned issues, inspired by
[63], we introduce high-order degradation to simulate the above factors. We
artificially degrade high-resolution images into low-resolution ones, thus
constructing training image pairs. The high-order degradation process can be
represented as follows:

$x_{lr}=\mathcal{D}^{n}(x_{hr})=({\mathcal{D}_{1}}\circ{\mathcal{D}_{2}}\circ{\cdots}\circ{\mathcal{D}_{n}})(x_{hr})$
(23)

where ${\mathcal{D}_{i}}$ represents a simple degradation model typically
comprised of blur, scaling, noise addition, and JPEG compression, which can be
expressed as:

$x_{lr}={D_{i}}(x_{hr})=[(x_{hr}{\ast}{k}){\downarrow}_{r}+n]_{\rm{JPEG}}$
(24)

where ${k}$, ${r}$, ${n}$, ${[\cdot]_{JPEG}}$ represent the blur kernel, the
scaling factor, noise, JPEG compression, respectively. During training, we
utilize high-order degradation to downsample real images into corresponding
low-resolution ones, following the parameter settings as described in [63].
During inference, we do not apply any degradation - the low-resolution
conditional input can be either real images or images generated from the
previous cascading steps.

When constructing the final loss function for denoising model training, we
introduced the Perception Prioritized (P2) weighting and imposed a weighting
factor on different denoising steps [64]. This design forces the model to
focus more on restoring perceptually rich content during training, resulting
in higher-quality image generation. Based on Eq. (10), the final loss function
for training the denoising network is defined as follows:

$L(\theta)={\sum_{t=1}^{T}}{\lambda_{t}^{\prime}}{L_{t-1}}$ (25)

where the weighting factors $\lambda_{t}^{\prime}$ are defined as:

${\lambda_{t}^{\prime}}=\frac{\lambda_{t}}{(k+\text{SNR}(x_{t}))^{\gamma}}$
(26)

where ${k}$ and ${\gamma}$ are hyperparameters, $\text{SNR}(x_{t})$ is the
signal-to-noise ratio of the noisy image $x_{t}$.

Our model is implemented using PyTorch. We utilize the AdamW optimizer with an
initial learning rate of 2e-6 to optimize parameters and set the total number
of epochs to 30. To increase the training batch size, we employ gradient
accumulation and multi-GPU parallelism. The batch size per GPU is set to 1,
and gradient accumulation is set to 8. The training process takes in a total
of 2,000 GPU hours on a set of NVIDIA RTX4090 GPUs. The training is
initialized from scratch without loading pre-trained parameters.

##  4 EXPERIMENTS

###  4.1 Experimental Setup

####  4.1.1 Dataset

To generate multi-resolution remote sensing images for any region globally, we
collected a large dataset, consisting of three levels of remote sensing images
from Google Earth, corresponding to resolutions of 64m/pixel, 16m/pixel, and
4m/pixel, covering the vast majority of the Earth’s landmass. A total of about
731,000 non-overlapping images, each sized 256×256 pixels, were used to train
and evaluate the model. For each image in the dataset, the spatial resolution
and the latitude and longitude are recorded. We partition the dataset into
training, validation, and testing sets in an 8:1:1 ratio.

####  4.1.2 Evaluation Metrics

We use Fréchet Inception Distance (FID) [65] to evaluate the quality and
diversity of the generated images. A lower FID indicates higher fidelity and
diversity in the generated images. FID quantifies the similarity between the
feature representations of generated and real images using a pre-trained deep
convolutional neural network. The FID score is defined as follows:

$\rm{FID}=\parallel{\mu_{r}-\mu_{g}}\parallel_{2}^{2}+{\mathrm{Tr}}(\Sigma_{r}+\Sigma_{g}-2({\Sigma_{r}}{\Sigma_{g}})^{\frac{1}{2}})$
(27)

where ${\mu_{r}}$ and ${\mu_{g}}$ are the mean feature vector of the real and
generated images, respectively. ${\Sigma_{r}}$ and ${\Sigma_{g}}$ are the
covariance matrices of the feature representations of the real and generated
images, respectively. The features used for computing the FID are extracted
from the final average pooling layer of a pre-trained Inception v3 network
[66].

###  4.2 Qualitative Analysis

####  4.2.1 Global-Scale Image Generation

Our experiments show that the proposed MetaEarth can generate a variety of
remote sensing scenes worldwide, including glaciers, snowfields, deserts,
forests, beaches, farmlands, industrial areas, residential areas, etc. Fig. 1
and Fig. 5 showcase some results of our model. Owing to the guidance from the
conditional input, the model is capable of generating vivid and diverse images
with distinct regional characteristics at different resolutions. We also
evaluated the semantical guidance provided by the image conditional input. In
Fig. 6, we use large-scale low-resolution images of 256m/pixel in the
Americas, Europe, and Asia as image condition guidance, and we can see the
model can generate urban images consistent with the regional style.

Figure 7: Visualization of the generated diverse scenes. By combining the
proposed self-cascading generation framework with different noise sampling
conditions, we can generate diverse and “parallel” scenes under the same
initial conditions.

####  4.2.2 Multi-Resolution Image Generation

The self-cascading generation framework in our method enables the model to
generate images with both spatial resolution diversity and content diversity.
Our experiments show that using high-order degradation models to downsample
high-resolution images for training pairs improved the model’s generalization
ability. Consequently, the model can take generated images as input to produce
higher-resolution images with richer details. Fig. 7 illustrates the generated
results of multiple resolutions. At the very beginning stage, the model is
conditioned on images with a resolution of 256 meters, finally resulting in
generated images with a spatial resolution of 4 meters. The results also
suggest the model’s strong diversity-generating capabilities. We can see that
as the number of generation stages increases, the content diversity between
images gradually increases due to the variability introduced by the generation
model.

####  4.2.3 Arbitrary-sized Image Generation

Figure 8: Arbitrary-sized images generated by our method. Figure 9: Some examples of high-resolution large-sized images generated by our model. TABLE I: Ablation studies on the spatial resolution guidance. We divided the Earth into different regions and assessed the quality (FID, a lower score indicates better) of images generated in each region. ${lng}$ denotes the longitude. | Settings | ${-180\leq lng<-90}$ | ${-90\leq lng<0}$ | ${0\leq lng<90}$ | ${90\leq lng<180}$ | Northern Hemisphere | Southern Hemisphere | Total
---|---|---|---|---|---|---|---|---
64m | w/o sr | 33.23 | 40.81 | 42.29 | 45.10 | 29.50 | ${\bm{42.11}}$ | 27.79
w/ sr | ${\bm{30.32}}$ | ${\bm{37.90}}$ | ${\bm{39.04}}$ | ${\bm{42.16}}$ | ${\bm{25.01}}$ | 44.16 | ${\bm{23.97}}$
16m | w/o sr | 19.72 | 21.12 | 26.03 | 26.58 | 16.84 | ${\bm{25.50}}$ | 16.00
w/ sr | ${\bm{19.36}}$ | ${\bm{21.91}}$ | ${\bm{24.65}}$ | ${\bm{25.47}}$ | ${\bm{15.76}}$ | 27.76 | ${\bm{15.12}}$
4m | w/o sr | 19.47 | 13.91 | 18.65 | 17.47 | 13.93 | ${\bm{22.02}}$ | 13.43
w/ sr | ${\bm{18.79}}$ | ${\bm{15.67}}$ | ${\bm{18.13}}$ | ${\bm{17.26}}$ | ${\bm{13.68}}$ | 25.63 | ${\bm{13.32}}$
4m* | w/o sr | 141.15 | 131.65 | 119.67 | 111.16 | 116.63 | 147.16 | 92.61
w/ sr | ${\bm{134.92}}$ | ${\bm{129.60}}$ | ${\bm{113.83}}$ | ${\bm{107.67}}$ | ${\bm{112.42}}$ | ${\bm{139.63}}$ | ${\bm{84.75}}$

Fig. 8 shows the unbounded generation results with the proposed sliding window
generation. Although we applied block-by-block generation, it can be observed
that the proposed method generates continuous land features across image
blocks. This indicates that our window overlapping and noise sampling strategy
greatly alleviate visual discontinuities caused by image block stitching,
thereby achieving boundless and arbitrary-sized image generation. Fig. 9 shows
an example of a high-resolution large-scale image generated by our model.

####  4.2.4 Generalization to Unseen Conditions

Thanks to being trained on large-scale data, our model possesses strong
generalization capabilities and performs well even on unseen scenes. To verify
this, we designed an interesting experiment. In the experiment, we aimed to
create a “parallel world” and used a low-resolution map of “Pandora Planet”
(generated by GPT4-V) as the initial condition for our model and then
generated higher-resolution images sequentially. Fig. 10 shows the generated
results. It can be observed that despite our training data not covering such
scenes, MetaEarth is still able to generate images with reasonable land cover
distribution and realistic details.

Figure 10: The self-cascaded generation results from MetaEarth with unseen
conditions. Figure 11: Comparision between MetaEarth to other text-to-image
models including GPT-4 V, the latest version of Stable Diffusion, and Ernie.
For the three compared methods, the text prompts are given as: “Please
generate a 4m/pixel-resolution satellite remote sensing image of an urban
scene. The image includes detailed city infrastructure such as roads,
buildings, parks, and waterways with clear visibility of the urban layout and
structure.”

####  4.2.5 Comparison with other text-to-image models

Despite the rapid advancements in generative models represented by Stable
Diffusion and DALLE, their capabilities in generating overhead remote sensing
scenes are limited. In Fig. 11, we compare MetaEarth with other state-of-the-
art text-to-image models, including GPT-4 V (DALLE), the latest version of
Stable Diffusion, and Ernie. We can see that MetaEarth significantly
outperforms these models in terms of image realism and the rationality of
scene layouts. The primary reason behind this difference is the lack of
relevant remote sensing training data during the training phase of previous
models. In addition, the three contrasting models lack sensitivity to
resolution control and are difficult to accurately capture the correspondence
between resolution and image content, making it difficult to generate multi-
resolution images or unbounded scenes. MetaEarth, on the other hand, excels in
these aspects, demonstrating its superior capability in generating high-
quality, realistic remote sensing images across various resolutions and scene
layouts.

###  4.3 Ablation Study

In this section, we conduct experiments to validate the effectiveness of the
proposed methods, including the resolution-guided generation and unbounded
generation.

####  4.3.1 Effectiveness of Resolution Guidance

In our method, we introduce spatial resolution as a condition to guide the
model in perceiving features and details of images at different scales,
achieving the generation of images at multiple resolutions using a single
model. To evaluate the impact of spatial resolution guidance on the quality of
generated images, we design a set of ablation experiments where we solely use
the images generated from the previous stage as conditional inputs and compare
their image generation quality with the full-implemented model under various
scenarios. Specifically, we remove the branches that take resolution as input
(frequency encoding $f_{\omega}$ and MLP network $F_{mlp}$) while keeping the
other structures and parameter settings unchanged. During the inference, both
models adopt the DDIM acceleration strategy, setting ${\eta=0}$.

We conducted quantitative analysis in different regions around the world. The
regions are divided by using different criteria, including the division of
regions at different longitudes, or the division of northern and southern
hemispheres. The experimental results are shown in Table I. The “w/o sr” rows
represent the network without using the resolution guidance. The first column
of the table gives the resolution of the generated images. Except for the 4m*,
the model inputs are low-resolution images obtained by downsampling the
corresponding resolution real images through bicubic interpolation. The inputs
for 4m* are randomly cropped from the real 16m resolution images. Due to the
different characteristics of sensors of different resolution levels, the FID
value of 4m* is significantly higher than that of other rows, but our
resolution guidance strategy is still effective.

We can see that compared to the model w/o using resolution guidance, our model
achieves better FID scores in all testing scenarios except for generating
images of the southern hemisphere. In our design, the resolution embedding
scales the features of the denoising network, allowing the model to generate
images with unique features specific to different spatial resolutions and
produce higher-quality outputs. Our experimental results also confirmed this.
Such performance improvement becomes more pronounced when the input consists
of real images with more complex distributions. The decline in image
generation quality for the southern hemisphere is primarily due to the limited
variability of land features at different resolutions in this region. The
southern hemisphere encompasses vast areas of deserts (in Africa and Oceania)
and forests (in South America), where the differences in features across
different resolutions are relatively minor compared to the northern
hemisphere. Therefore, even without resolution guidance, the model may also
generate images of better consistency with the true distribution.

Figure 12: A comparison between the generated images w/ and w/o using the
proposed initial noise sampling strategy. It can be seen that without using
the noise sampling method proposed in this paper, there are noticeable seams
at the junctions of the generated windows, and the semantic content appears
discontinuous. In contrast, the proposed method effectively resolves these
problems. The images in the second row are local zoomed-in views of the inner
regions.

####  4.3.2 Evaluation on Noise Sampling

To measure the continuity of the different generated blocks in the seam area,
in Fig. 3, we compare the generated results w/ and w/o using the proposed
noise sampling strategy. We can see that with the proposed noise sampling
strategy, we obtained smoother results, and the traces of patchwork are
difficult to detect with the naked eye. We also designed a gradient-based
method to quantitatively evaluation the transition smoothness of different
approaches. We randomly sample 750 generated images and calculate the absolute
directional gradient values of adjacent pixels at the seams and then take the
average. Experimental results are shown in Table II. We evaluate the proposed
two sub-strategies respectively, one is overlapped sliding window (marked as
“Overlap” in Table II) and another is the constraints on initial noise
sampling (marked as “Noise Constraint” in Table II). Smaller gradient values
indicate better results. The experimental results demonstrate that the
combined use of the overlapping cropping strategy and our proposed noise
sampling method can effectively alleviate the seams appearing during image
block stitching, thereby enhancing the visual quality of the generated
arbitrary-sized images.

TABLE II: Ablation experiment on our proposed noise sampling method. We compare the averaged directional gradient value of pixels in the window boundary area (a lower value indicates better). Methods | ${{\rm{DirectionalGradient}}\downarrow}$
---|---
Overlap | Noise Constraints | Horizontal | Vertical | Average
${\times}$ | ${\times}$ | 24.45 | 24.87 | 24.67
✓ | ${\times}$ | 17.66 | 18.09 | 17.88
✓ | ✓ | ${\bm{{12.01}}}$ | ${\bm{{13.11}}}$ | ${\bm{{12.58}}}$

###  4.4 Evaluation on Downstream Tasks

The proposed MetaEarth can be used as a powerful data engine to generate high-
quality training data that can benefit downstream tasks. In this experiment,
we choose remote sensing image classification, a fundamental task in remote
sensing image interpretation, as our test bed for downstream evaluation.

To quantitatively demonstrate the application of our model in downstream tasks
and further evaluate the fidelity and diversity of the generated images, we
design a remote sensing image classification experiment, using our model as a
data engine to provide training data support. Our classification dataset
contains seven categories of remote sensing scenes: beach, desert, farmland,
forest, industrial area, mountain, and residential area. Each class contains
approximately 150 images, with image sizes of 256$\times$256 pixels. The
dataset is re-collected from our global 4m resolution images. The training and
testing sets are divided by a 3:1 ratio.

We choose four different models as classifiers: VGG19 [67], ResNet34 [68],
ViT-B/32, and ViT-B/16 [69]. We first train four baseline models using the
original dataset. Then, we use the generation model to augment the training
dataset to five times its original size. After mixing the augmented data with
the original one, we retrain the four models from scratch using the combined
dataset. For the augmentation process, we downsample the images in the
training dataset and utilize the model to generate diverse results.

TABLE III: Accuracy of the downstream remote sensing image classification task w/ and w/o image augmentation. Methods | Traning Data | Accuracy
---|---|---
VGG19 | Real Data | 95.72
Augment Data | ${\bm{97.28}}$
ResNet34 | Real Data | 96.11
Augment Data | ${\bm{99.22}}$
ViT-B/32 | Real Data | 93.77
Augment Data | ${\bm{94.55}}$
ViT-B/16 | Real Data | 93.77
Augment Data | ${\bm{95.33}}$

Table III shows the classification accuracy on the test set for different
models using different training data. Compared with the original models, the
models trained with data augmentation show consistently improved
classification accuracy. This indicates that MetaEarth provides informative
data for training, which also reflects the high quality and diversity of the
images generated. This indicates that when faced with tasks with insufficient
or sparse data, MetaEarth has the potential to serve as a data engine to
provide usable data at lower costs.

###  4.5 Disccusion

In addition to serving as a data engine for data augmentation, MetaEarth also
holds immense potential in the construction of generative world models. World
models are a crucial pathway to achieving general artificial intelligence,
providing essential training and testing environments for an agent’s
perception, decision-making, and evolution. Recently, the emergence of
OpenAI’s Sora model has brought significant attention to research in the field
of world models. Generative world models seek to understand the world through
generative processes, typically integrating closely with technologies such as
large language models, visual foundation models, and video/image generation
models.

As a generative foundation model, MetaEarth opens up new possibilities for
constructing generative world models by simulating Earth’s visuals from an
innovative overhead perspective. This unique approach allows for the creation
of high-resolution, multi-resolution, and unbounded remote sensing images that
cover diverse geographical features and ecological systems. Given a virtual
viewpoint’s altitude and movement path, MetaEarth can generate and predict
content beyond the immediate scene, thus creating a dynamic environment with
intelligent interactive behaviors. By leveraging MetaEarth, researchers may
generate a realistic and dynamic visual environment that can interact with
intelligent agents such as drones and remote sensing satellites in various
scenarios, ranging from urban planning and environmental monitoring to
disaster management and agricultural optimization, facilitating their
training, testing, and validation.

Constructing generative world models in the aerospace intelligence domain is
an important research trend for the future. In this regard, MetaEarth offers
valuable insights and methodologies. We believe that as a starting point for
this line of research, MetaEarth will significantly contribute to the
development of generative world models for aerospace remote sensing, providing
new possibilities for its future advancements.

##  5 Conclusion

In this paper, we introduced MetaEarth, a generative foundation model
specifically designed for global-scale remote sensing image generation.
MetaEarth extends the capabilities of existing generative models by enabling
the production of worldwide, multi-resolution, unbounded, and virtually
limitless remote sensing images. The proposed resolution-guided self-cascading
generative framework and innovative noise sampling strategy allow MetaEarth to
generate high-quality images with diverse geographical features and
resolutions, overcoming significant challenges in model capacity, resolution
control, and unbounded image generation. Extensive experiments demonstrate
MetaEarth’s powerful capabilities in generating diverse and high-fidelity
remote sensing images across different resolutions and regions. It also shows
great potential as a data engine, capable of generating high-quality training
data for downstream tasks in remote sensing. The advancements presented in
this paper pave the way for future research in building generative world
models from an overhead perspective of the earth.

## References

  * [1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 684–10 695.
  * [2] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _arXiv preprint arXiv:2204.06125_ , vol. 1, no. 2, p. 3, 2022.
  * [3] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans _et al._ , “Photorealistic text-to-image diffusion models with deep language understanding,” _Advances in neural information processing systems_ , vol. 35, pp. 36 479–36 494, 2022.
  * [4] W. Harvey, S. Naderiparizi, V. Masrani, C. Weilbach, and F. Wood, “Flexible diffusion modeling of long videos,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 27 953–27 965, 2022.
  * [5] T. Höppe, A. Mehrjou, S. Bauer, D. Nielsen, and A. Dittadi, “Diffusion models for video prediction and infilling,” _arXiv preprint arXiv:2206.07696_ , 2022.
  * [6] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi, “Palette: Image-to-image diffusion models,” in _ACM SIGGRAPH 2022 conference proceedings_ , 2022, pp. 1–10.
  * [7] M. Zhao, F. Bao, C. Li, and J. Zhu, “Egsde: Unpaired image-to-image translation via energy-guided stochastic differential equations,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 3609–3623, 2022.
  * [8] Z. Wan, B. Zhang, D. Chen, P. Zhang, F. Wen, and J. Liao, “Old photo restoration via deep latent space translation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 45, no. 2, pp. 2071–2087, 2022.
  * [9] O. Avrahami, D. Lischinski, and O. Fried, “Blended diffusion for text-driven editing of natural images,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 208–18 218.
  * [10] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon, “Sdedit: Guided image synthesis and editing with stochastic differential equations,” _arXiv preprint arXiv:2108.01073_ , 2021.
  * [11] R. Yi, Y.-J. Liu, Y.-K. Lai, and P. L. Rosin, “Quality metric guided portrait line drawing generation from unpaired training data,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 45, no. 1, pp. 905–918, 2023.
  * [12] M. Xu, S. Yoon, A. Fuentes, and D. S. Park, “A comprehensive survey of image augmentation techniques for deep learning,” _Pattern Recognition_ , vol. 137, p. 109347, 2023.
  * [13] S. Azizi, S. Kornblith, C. Saharia, M. Norouzi, and D. J. Fleet, “Synthetic data from diffusion models improves imagenet classification,” _arXiv preprint arXiv:2304.08466_ , 2023.
  * [14] W. Wu, Y. Zhao, H. Chen, Y. Gu, R. Zhao, Y. He, H. Zhou, M. Z. Shou, and C. Shen, “Datasetdm: Synthesizing data with perception annotations using diffusion models,” _Advances in Neural Information Processing Systems_ , vol. 36, pp. 54 683–54 695, 2023.
  * [15] Z. Zhu, X. Wang, W. Zhao, C. Min, N. Deng, M. Dou, Y. Wang, B. Shi, K. Wang, C. Zhang _et al._ , “Is sora a world simulator? a comprehensive survey on general world models and beyond,” _arXiv preprint arXiv:2405.03520_ , 2024.
  * [16] A. Hu, L. Russell, H. Yeo, Z. Murez, G. Fedoseev, A. Kendall, J. Shotton, and G. Corrado, “Gaia-1: A generative world model for autonomous driving,” _arXiv preprint arXiv:2309.17080_ , 2023.
  * [17] J. Bruce, M. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps _et al._ , “Genie: Generative interactive environments,” _arXiv preprint arXiv:2402.15391_ , 2024.
  * [18] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” _Communications of the ACM_ , vol. 63, no. 11, pp. 139–144, 2020.
  * [19] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 4401–4410.
  * [20] P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high-resolution image synthesis,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 12 873–12 883.
  * [21] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [22] A. Van Den Oord, O. Vinyals _et al._ , “Neural discrete representation learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [23] D. Rezende and S. Mohamed, “Variational inference with normalizing flows,” in _International conference on machine learning_. PMLR, 2015, pp. 1530–1538.
  * [24] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in neural information processing systems_ , vol. 33, pp. 6840–6851, 2020.
  * [25] A. Q. Nichol and P. Dhariwal, “Improved denoising diffusion probabilistic models,” in _International conference on machine learning_. PMLR, 2021, pp. 8162–8171.
  * [26] F.-A. Croitoru, V. Hondru, R. T. Ionescu, and M. Shah, “Diffusion models in vision: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [27] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in _International conference on machine learning_. PMLR, 2015, pp. 2256–2265.
  * [28] J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” _arXiv preprint arXiv:2010.02502_ , 2020.
  * [29] J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans, “Cascaded diffusion models for high fidelity image generation,” _Journal of Machine Learning Research_ , vol. 23, no. 47, pp. 1–33, 2022.
  * [30] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, “Image super-resolution via iterative refinement,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 45, no. 4, pp. 4713–4726, 2022.
  * [31] L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 3836–3847.
  * [32] R. Ou, H. Yan, M. Wu, and C. Zhang, “A method of efficient synthesizing post-disaster remote sensing image with diffusion model and llm,” in _2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 2023, pp. 1549–1555.
  * [33] S. Khanna, P. Liu, L. Zhou, C. Meng, R. Rombach, M. Burke, D. Lobell, and S. Ermon, “Diffusionsat: A generative foundation model for satellite imagery,” _arXiv preprint arXiv:2312.03606_ , 2023.
  * [34] D. Tang, X. Cao, X. Hou, Z. Jiang, and D. Meng, “Crs-diff: Controllable generative remote sensing foundation model,” _arXiv preprint arXiv:2403.11614_ , 2024.
  * [35] A. Sebaq and M. ElHelw, “Rsdiff: Remote sensing image generation from text using diffusion model,” _arXiv preprint arXiv:2309.02455_ , 2023.
  * [36] M. Espinosa and E. J. Crowley, “Generate your own scotland: Satellite image generation conditioned on maps,” _arXiv preprint arXiv:2308.16648_ , 2023.
  * [37] Z. Yuan, C. Hao, R. Zhou, J. Chen, M. Yu, W. Zhang, H. Wang, and X. Sun, “Efficient and controllable remote sensing fake sample generation based on diffusion model,” _IEEE Transactions on Geoscience and Remote Sensing_ , 2023.
  * [38] O. Baghirli, H. Askarov, I. Ibrahimli, I. Bakhishov, and N. Nabiyev, “Satdm: Synthesizing realistic satellite image with semantic layout conditioning using diffusion models,” _arXiv preprint arXiv:2309.16812_ , 2023.
  * [39] C. Zhao, Y. Ogawa, S. Chen, Z. Yang, and Y. Sekimoto, “Label freedom: Stable diffusion for remote sensing image semantic segmentation data generation,” in _2023 IEEE International Conference on Big Data (BigData)_. IEEE, 2023, pp. 1022–1030.
  * [40] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 16 000–16 009.
  * [41] X. Sun, P. Wang, W. Lu, Z. Zhu, X. Lu, Q. He, J. Li, X. Rong, Z. Yang, H. Chang _et al._ , “Ringmo: A remote sensing foundation model with masked image modeling,” _IEEE Transactions on Geoscience and Remote Sensing_ , 2022.
  * [42] D. Hong, B. Zhang, X. Li, Y. Li, C. Li, J. Yao, N. Yokoya, H. Li, P. Ghamisi, X. Jia _et al._ , “Spectralgpt: Spectral remote sensing foundation model,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2024.
  * [43] F. Bastani, P. Wolters, R. Gupta, J. Ferdinando, and A. Kembhavi, “Satlaspretrain: A large-scale dataset for remote sensing image understanding,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 16 772–16 782.
  * [44] D. Wang, Q. Zhang, Y. Xu, J. Zhang, B. Du, D. Tao, and L. Zhang, “Advancing plain vision transformer toward remote sensing foundation model,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 61, pp. 1–15, 2023.
  * [45] C. J. Reed, R. Gupta, S. Li, S. Brockman, C. Funk, B. Clipp, K. Keutzer, S. Candido, M. Uyttendaele, and T. Darrell, “Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2023, pp. 4088–4099.
  * [46] Y. Cong, S. Khanna, C. Meng, P. Liu, E. Rozi, Y. He, M. Burke, D. Lobell, and S. Ermon, “Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 197–211, 2022.
  * [47] M. Noman, M. Naseer, H. Cholakkal, R. M. Anwar, S. Khan, and F. S. Khan, “Rethinking transformers pre-training for multi-spectral satellite imagery,” _arXiv preprint arXiv:2403.05419_ , 2024.
  * [48] M. Mendieta, B. Han, X. Shi, Y. Zhu, and C. Chen, “Towards geospatial foundation models via continual pretraining,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 16 806–16 816.
  * [49] F. Liu, D. Chen, Z. Guan, X. Zhou, J. Zhu, Q. Ye, L. Fu, and J. Zhou, “Remoteclip: A vision language foundation model for remote sensing,” _IEEE Transactions on Geoscience and Remote Sensing_ , 2024.
  * [50] Z. Zhang, T. Zhao, Y. Guo, and J. Yin, “Rs5m: A large scale vision-language dataset for remote sensing vision-language foundation model,” _arXiv preprint arXiv:2306.11300_ , 2023.
  * [51] U. Mall, C. P. Phoo, M. K. Liu, C. Vondrick, B. Hariharan, and K. Bala, “Remote sensing vision-language foundation models without annotations via ground remote alignment,” _arXiv preprint arXiv:2312.06960_ , 2023.
  * [52] Y. Yuan, Y. Zhan, and Z. Xiong, “Parameter-efficient transfer learning for remote sensing image-text retrieval,” _IEEE Transactions on Geoscience and Remote Sensing_ , 2023.
  * [53] J. D. Silva, J. Magalhães, D. Tuia, and B. Martins, “Large language models for captioning and retrieving remote sensing images,” _arXiv preprint arXiv:2402.06475_ , 2024.
  * [54] C. Liu, R. Zhao, and Z. Shi, “Remote sensing image captioning based on multi-layer aggregated transformer,” _IEEE Geoscience and Remote Sensing Letters_ , pp. 1–1, 2022.
  * [55] C. Liu, R. Zhao, J. Chen, Z. Qi, Z. Zou, and Z. Shi, “A decoupling paradigm with prompt learning for remote sensing image change captioning,” _IEEE Transactions on Geoscience and Remote Sensing_ , 2023.
  * [56] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [57] K. Kuckreja, M. S. Danish, M. Naseer, A. Das, S. Khan, and F. S. Khan, “Geochat: Grounded large vision-language model for remote sensing,” _arXiv preprint arXiv:2311.15826_ , 2023.
  * [58] Y. Zhan, Z. Xiong, and Y. Yuan, “Skyeyegpt: Unifying remote sensing vision-language tasks via instruction tuning with large language model,” _arXiv preprint arXiv:2401.09712_ , 2024.
  * [59] W. Zhang, M. Cai, T. Zhang, Y. Zhuang, and X. Mao, “Earthgpt: A universal multi-modal large language model for multi-sensor image comprehension in remote sensing domain,” _arXiv preprint arXiv:2401.16822_ , 2024.
  * [60] Z. Wang, R. Prabha, T. Huang, J. Wu, and R. Rajagopal, “Skyscript: A large and semantically diverse vision-language dataset for remote sensing,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 38, no. 6, 2024, pp. 5805–5813.
  * [61] H. Li, Y. Yang, M. Chang, S. Chen, H. Feng, Z. Xu, Q. Li, and Y. Chen, “Srdiff: Single image super-resolution with diffusion probabilistic models,” _Neurocomputing_ , vol. 479, pp. 47–59, 2022.
  * [62] X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, and C. Change Loy, “Esrgan: Enhanced super-resolution generative adversarial networks,” in _Proceedings of the European conference on computer vision (ECCV) workshops_ , 2018, pp. 0–0.
  * [63] X. Wang, L. Xie, C. Dong, and Y. Shan, “Real-esrgan: Training real-world blind super-resolution with pure synthetic data,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 1905–1914.
  * [64] J. Choi, J. Lee, C. Shin, S. Kim, H. Kim, and S. Yoon, “Perception prioritized training of diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 472–11 481.
  * [65] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [66] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [67] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [68] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [69] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.