# Transformers in Remote Sensing: A Survey

Abdulaziz Amer Aleissaee* Amandeep Kumar* Rao Muhammad Anwer  Salman Khan
Hisham Cholakkal  Gui-Song Xia and Fahad Shahbaz khan A.A. Aleissaee, A.
Kumar, R. Anwer, S. Khan, H. Cholakkal and F. Khan are with MBZ University of
Artificial Intelligence, UAE.
Gui-Song Xia is with Wuhan University, China.
E-mail<EMAIL_ADDRESS>
###### Abstract

Deep learning-based algorithms have seen a massive popularity in different
areas of remote sensing image analysis over the past decade. Recently,
transformers-based architectures, originally introduced in natural language
processing, have pervaded computer vision field where the self-attention
mechanism has been utilized as a replacement to the popular convolution
operator for capturing long-range dependencies. Inspired by recent advances in
computer vision, remote sensing community has also witnessed an increased
exploration of vision transformers for a diverse set of tasks. Although a
number of surveys have focused on transformers in computer vision in general,
to the best of our knowledge we are the first to present a systematic review
of recent advances based on transformers in remote sensing. Our survey covers
more than 60 recent transformers-based methods for different remote sensing
problems in sub-areas of remote sensing: very high-resolution (VHR),
hyperspectral (HSI) and synthetic aperture radar (SAR) imagery. We conclude
the survey by discussing different challenges and open issues of transformers
in remote sensing. Additionally, we intend to frequently update and maintain
the latest transformers in remote sensing papers with their respective code
at: https://github.com/VIROBO-15/Transformer-in-Remote-Sensing ††* Authors
contributed equally

###### Index Terms:

remote sensing, transformers, survey.

##  I Introduction

Remote sensing imaging technology has significantly advanced in the last
decades. Modern airborne sensors provide a large coverage of the Earth surface
with improved spatial, spectral and temporal resolutions, thereby playing a
crucial role in numerous research areas, including ecology, environmental
science, soil science, water contamination, glaciology, land surveying and
analysis of the crust of the Earth. Automatic analysis of remote sensing
imaging brings unique challenges such as, data are generally multi-modal (eg,
optical or synthetic aperture radar sensors), located in the geographical
space (geo-located) and typically on a global-scale with ever growing data
volumes.

Deep learning, especially convolutional neural networks (CNNs) has dominated
many areas of computer vision, including object recognition, detection and
segmentation. These networks typically take an RGB image as an input and
perform a series of convolution, local normalization and pooling operations.
CNNs typically rely on a large amount of training data and the resulting pre-
trained models are then utilized as generic feature extractors for a variety
of downstream applications. The success of deep learning-based techniques in
computer vision has also inspired the remote sensing community with
significant advances being made in many remote sensing tasks, including
hyperspectral image classification, change detection and very high-resolution
satellite instance segmentation.

One of the main building blocks in CNNs is the convolution operation which
captures local interactions between elements (eg, contour and edge
information) in the input image. CNNs encode biases such as, spatial
connectivity and translation equivariance. These charactertistics aid in
constructing generalizable and efficient architectures. However, the local
receptive field in CNNs limits modeling long-range dependencies in an image
(eg, distant part relationships). Moreover, convolutions are content-
independent as the convolutional filter weights are stationary with same
weights applied to all inputs regardless of their nature. Recently, vision
transformers (ViTs) [1] have demonstrated impressive performance across a
variety of tasks in computer vision. ViTs are based on the self-attention
mechanism that effectively captures global interactions by learning the
relationships between the elements of a sequence. Recent works [2, 3] have
shown that ViTs possess content-dependent long-range interaction modeling
capabilities and can flexibly adjust their receptive fields to counter
nuisances in data and learn effective feature representations. As a result,
ViTs and their variants have been successfully utilized for many computer
vision tasks, including classification, detection and segmentation.

Figure 1: Recent transformers-based techniques in the remote sensing imaging.
On the left and middle: Pie-charts are representing statistics of the articles
covered in this survey in terms of different remote sensing imaging problems
and data type representations. On the right: we show a plot illustrating the
consistent increase in number of papers recently.

Following the success of ViTs in computer vision, remote sensing community has
also witnessed a significant growth (see Fig.1) in the employment of
transformers-based frameworks in many tasks such as, very high-resolution
image classification, change detection, pan sharpening, building detection and
image captioning. This has started a new wave of promising research in remote
sensing with different approaches utilizing either ImageNet pre-training [4,
5, 6] or performing remote sensing pre-training [7] with vision transformers.
Similarly, there exist approaches in literature that are based on pure
transformers design [8, 9] or utilize a hybrid approach [10, 11, 12] based on
both transformers and CNNs. It is therefore becoming increasingly challenging
to keep pace with the recent progress due to the rapid influx of transformers-
based methods for different remote sensing problems. In this work, we review
these advances and present an account of recent transformers-based approaches
in the popular field of remote sensing. To summarize, our main contributions
are the following:

  * •

We present a holistic overview of applications of transformers-based models in
remote sensing imaging. To the best of our knowledge, we are the first to
present a survey on transformers in remote sensing, thereby bridging the gap
between recent advances in computer vision and remote sensing in this rapidly
growing and popular area.

  * •

We present an overview of both CNNs and transformers, discussing their
respective strengths and weaknesses.

  * •

We present a review of more than 60 transformers-based research works in the
literature to discuss the recent progress in the field of remote sensing.

  * •

Based on the presented review, we discuss different challenges and research
directions on transformers in remote sensing.

The rest of the paper is organized as follows: Section II discusses other
related surveys on remote sensing imaging. In Section III, we present an
overview of different imaging modalities in remote sensing, whereas Section IV
provides a brief overview of CNNs and vision transformers. Afterwards, we
review advances with respect to transformers-based approaches in very high-
resolution (VHR) imaging (Section V), hyperspectral image analysis (Section
VI) and synthetic aperture radar (SAR) in Section VII. In Section VIII, we
conclude our survey and discuss potential future research directions.

##  II Related Work

In the literature, several works have performed a review of machine learning
techniques for remote sensing imaging in the past decade. Tuia et al. [13]
compare and evaluate different active learning algorithms for supervised
remote sensing image classification task. The work of [14] focuses on the
problem of hyperspectral image classification and reviews recent advances in
relation to machine learning and vision techniques. Zhu et al. [15] present a
comprehensive review of utilizing deep learning techniques for remote sensing
image analysis. Their work provides a comprehensive review of the existing
approaches along with describing a list of resources about deep learning in
remote sensing. Ma et al. [16] review major deep learning concepts in remote
sensing with respect to image resolution and study area. To this end, their
work studies different remote sensing tasks such as, image registration,
fusion, scene classification and object segmentation.

Recently, transformers-based approaches have witnessed a significant surge
within the computer vision community, following the breakthrough from
transformers-based models [17] in natural language processing (NLP). Khan et
al. [18] present an overview of the transformers models in vision with
emphasis on recognition, generative modeling, multi-modal, video processing
and low-level vision tasks. Shamshad et al. [19] survey the use of
transformers models in medical imaging, focusing on different medical imaging
tasks such as, segmentation, detection, reconstruction, registration and
clinical medical report generation. The work of [20] presents an overview of
the growing trend of using transformers to model video data. Their work also
compares the performance of vision transformers on different video tasks such
as, action recognition.

Different from the aforementioned surveys, our work presents a review of
recent advances of transformers-based approaches in the popular area of remote
sensing. To the best of our knowledge, this is the first survey presenting a
comprehensive account of transformers in remote sensing, particularly
dedicated to progress in very high-resolution, hyperspectral and synthetic
aperture radar image analysis.

##  III Remote Sensing Imaging Data

Remote sensing imagery is generally acquired from a range of sources as well
as data collection techniques. Remote sensing image data can be typically
characterised by their spatial, spectral, radiometric, and temporal
resolutions. Spatial resolution refers to each pixel size within an image
along with the area of the surface of the Earth represented by that
corresponding pixel. Spatial resolution characterizes the small and fine-
detailed features in an imaging scene that can be separated. Spectral
resolution refers to the capability of the sensor to collect information about
the scene by discerning finer wavelengths, with having more narrower bands
(e.g., 10 nm). On the other hand, radiometric resolution characterizes the
extent of the information in each pixel, where a larger dynamic range for a
sensor implies more details are to be discerned in the image. The temporal
resolution refers to the time it takes between consecutive images of the same
location on ground acquired by the sensor. Here, we briefly discuss commonly
utilized remote sensing imaging types with few examples shown in Fig.2.

Very High-resolution Imagery: In recent years, the emergence of very high-
resolution (VHR) satellite sensors has paved the way towards yielding the
higher spatial resolution imagery beneficial for land use change detection,
object-based image analysis (object detection and instance segmentation),
precision agriculture farming (e.g., management of crops, soil and pests) and
emergency responses. Furthermore, these recent advances in sensor technology
along with new deep learning-based techniques allow the usage of VHR remote
sensing imagery to analyze the biophysical as well as biogeochemical processes
both in coastal and inland waters. Nowadays, optical sensors produce
panchromatic and multispectral imagery of the Earth’s surface at a much finer
spatial resolutions (e.g., 10 to 100 cm/pixel).

Hyperspectral Imagery: Here, each pixel in the scene is captured using
continuous spectrum of light with fine wavelength resolutions. The continuous
spectrum extends wavelengths beyond the visible spectrum and include
wavelengths from ultraviolet (UV) to infrared (IR). Generally, spectral
resolution of hyperspectral images are expressed using the wave number along
with the nanometers (nm). The most popular continuous spectrum used for
measuring the pixels are mid-infrared, that is near infrared and visible
wavelength bands. In order to acquire hyperspectral imagery, there are
different electromagnetic measurements such as, Raman spectroscopy, X-ray
spectroscopy, Terahertz spectroscopy, 3D ultrasonic imaging, magnetic
resonance and confocal laser microscopy scanners that can measure the entire
emission spectrum for each pixel at a specific excitation wavelength. The
hyperspectral images have high dimensionality and strong resolving power for
fine spectra. The imagery offers a wide range of applications, including in
environmental science [21] and mining [22]. Different from regular images that
contain only the primary colors (red, green, blue) within the visible
spectrum, hyperspectral images are rich in spectral information that can
reflect the physical structure and chemical composition of the item of
interest. In remote sensing, automatically analyzing hyperspectral imagery is
an active research topic.

Synthetic Aperture Radar Imagery: A large amount of synthetic aperture radar
(SAR) images are produced by Earth observation satellites every day through
emission and reception of electromagnetic signals. In the past decades, SAR
images have gained popularity due to their higher spatial resolution, all-
weather capability, de-speckling tools such as, CAESAR along with recent
advances in the SAR specific image processing. SAR imagery can be used for
numerous applications, including geographical localization, object detection,
functionalities of basic radars, and geophysical features estimation of
complex setting such as, roughness, moisture content, and density. Further,
SAR imagery can be used for disaster management (oil slick detection, ice
tracking), forestry and hydrology.

Figure 2: Example hyperspectral (a), VHR (b) and SAR (c) images from different
datasets.

##  IV From CNNs to Vision Transformers

In this section, we first present a brief overview of CNNs and then provide a
brief description of vision transformers recently utilized for different
vision tasks.

###  IV-A Convolutional Neural Networks

Convolutional neural networks (CNNs) have dominated a variety of computer
vision tasks, including image classification [23] and object detection [24].
CNNs are typically made up of series of two main parts: convolutional and
pooling layers. The convolutional layer produces feature maps by convolving
the local region in the input with a set of kernels. These features are
subjected to a non-linear function with the same process repeated for each
convolutional layer. In CNNs, the pooling layer carries out a downsampling
operation (typically utilizing the max or mean operation) to feature maps. In
different existing CNN architectures, the convolutional and pooling layers are
followed by a set of fully connected layers, where the last fully connected
layer is the softmax computing each object category score.
Popular CNN Backbones: Here, we briefly discuss different popular CNN backbone
architectures in literature.
AlexNet: Krizhevsky et al. [23] propose a CNN architecture, named AlexNet, for
image classification task. AlexNet comprises five convolutional layers
followed by three fully-connected layers. The proposed network architecture
utilizes Rectified Linear Units (ReLU) for training efficiency. The network
contains 60 million parameters and 500,000 neurons with network training
performed on the large-scale ImageNet dataset [25]. Different data
augmentation techniques are employed to increase the training set. In the
ImageNet 2012 competition, AlexNet achieved competitive performance with top-1
and top-5 error rates of 39.7% and 18.9%, respectively.
VGGNet: Different from AlexNet, Simonyan and Zisserman [26] introduce an
architecture, named VGGNet that comprises 16 layers in total. The network
takes an input image of 224 $\times$ 224 size and has around 138 million
parameters. It uses different data augmentation techniques, including scale
jittering during network training. The VGGNet architecture comprises
convolution layers of 3 $\times$ 3 filter, where the receptive fields are
convolved at each pixel with a stride of one pixel. The VGGNet contains
multiple pooling layers, performing spatial pooling over 2 $\times$ 2 windows
with a stride of two pixels. Further, VGGNet contains two fully connected
layers followed by a softmax for yielding output predictions. The VGG
architecture achieved top classification accuracy on the 2014 ImageNet
classification challenge.
ResNet: Different from AlexNet and VGGNet, He et al. [27] introduce residual
neural networks (ResNet) that stacks residual blocks to build a network.
ResNet provides a residual learning approach for training networks that are
much deeper than their previously utilised counterparts. Instead of learning
un-referenced functions, it explicitly reformulates the layers as learning
residual functions with reference to the layer inputs. Extensive empirical
evidence demonstrates that residual networks are easier to optimize with
improved accuracy from higher depth.

The development of CNN-based architectures has led to rise of novel
techniques, improved hardware (e.g. GPUs and TPUs), better optimization
methods and many open-source libraries. Interested readers can go through the
survey papers related to CNN methods for remote sensing [15, 16]. Previous
works have analyzed that CNNs are able to capture image-specific inductive
bias which increases their effectiveness in learning better feature
representations. However, CNNs do not capture long-range dependencies that
aids to enhance expressivity of the representations. Next, we briefly present
vision transformers that are capable of modelling long-range dependencies in
the images.

Figure 3:  The Vision Transformer’s architecture is shown on the left and the
encoder block’s specifications are shown on the right. The input image is
first divided into patches. These are then projected (after flattening) into a
feature space, where a transformer encoder analyses them to create the
classification output. Adapted from [1] and [19]. Figure 4: The taxonomy of
transformers in VHR, hyperspectral and SAR imagery with a variety of tasks
such as, classification, detection, segmentation, pan sharpening and change
detection.

###  IV-B Vision Transformers

Recently, transformers-based models have achieved promising results across
many computer vision and natural language processing (NLP) tasks. Vaswani et
al.[17] first introduce transformers as an attention-driven model for machine
translation application. To capture the long-range dependencies, transformers
use self-attention layers instead of traditional recurrent neural network that
struggles to encode such dependencies between the elements of a sequence.

To effectively capture the long-range dependencies within an input image, the
work of [1] introduces vision transformers (ViTs) for image recognition task,
as shown in Fig.3. ViTs [1] interpret an image as a sequence of patches and
process it via a conventional transformers encoder similar to those used in
NLP tasks. The success of ViTs in generic visual data have sparked the
interest not only in different areas of computer vision but also in the remote
sensing community, where a number of ViT-based techniques have been explored
in recent years for various tasks.

Next, we briefly describe the key component of self-attention within
transformers.
Self-Attention: The self-attention mechanism has been an integral component of
transformers as it captures the long-range dependencies and encodes the
interaction between all of the sequences tokens (patch embedding). The key
idea of self-attention is to learn self-alignment, that is to update the token
by aggregating global knowledge from all the other tokens in the sequence[28].
Given a 2D image $x\in\mathbb{R}^{H\times W\times C}$, the process starts with
flattening the image into a series of 2D patches
$x_{p}at\in\mathbb{R}^{M\times(P^{2}C)}$, where $C$ represents number of
channels, $H$ and $W$ represents height and width of the image, $P\times P$ is
the dimension of each individual patch, and $M=HW/P^{2}$ represents the total
number of patches. A learnable linear projection layer of $E$ dimension is
used to project these flattened patches and can be showed as a matrix
$X\in\mathbb{R}^{N\times E}$. The aim of the self-attention is to apprehend
interaction among all the $M$ embeddings, that is achieved by introducing the
three learnable weight matrices to modify input $X$ into queries (as
$W^{Q}\in\mathbb{R}^{E\times E_{q}}$), keys (as $W^{K}\in\mathbb{R}^{E\times
E_{k}}$) and values (as $W^{V}\in\mathbb{R}^{E\times E_{v}}$), where
$E_{q}=E_{k}$. The sequence $X$ is first projected onto these weight matrices
to obtain $K=XW^{K}$, $V=XW^{V}$ and $Q=XW^{Q}$. The relative attention matrix
$A\in\mathbb{R}^{M\times M}$ is

$\bm{Z}=softmax(\frac{QK^{T}}{\sqrt{E_{q}}})V$ (1)

Masked Self-Attention: All entities are attended to the usual self-attention
layer. These self-attention blocks used in the decoder for the transformers
model [17], which is trained to anticipate the next entity in the sequence,
are masked to prevent attending to the subsequent future entities. This task
is performed by an element-wise multiplication operation with a mask
$\mathbf{M}\in\mathbb{R}^{n\times n}$, where $\mathbf{M}$ is an upper-
triangular matrix. Here, masked self-attention is represent by

$softmax(\frac{QK^{T}}{\sqrt{d_{q}}}\circ\mathbb{M})$ (2)

where $\circ$ represents the Hadamard product. In masked self-attention, the
attention ratings of future entities are set to zero when predicting an entity
in the sequence.
Multi-Head Attention: Multi-head attention (MHA) comprises multiple self-
attention blocks concatenated simultaneously channel-wise, in order to capture
different complex interactions between different sequence of embeddings. Each
of the head of the multi-head self-attention has its own learnable weight
matrices represented as $W^{Q_{i}}$, $W^{K_{i}}$ and $W^{V_{i}}$, where
$i=0\cdot\cdot\cdot\cdot\cdot\cdot(h-1)$ were $h$ denotes the number head in
multi-head self-attention. Hence, we can express,

$MHA(Q,K,V)=[Z_{0},...,Z_{h-1}]W^{O}$ (3)

where output of each head is concatenated to form single matrix
$B\in\mathbb{R}^{M\times h\cdot E_{v}}$, whereas
$W^{O}\times\mathbb{R}^{h.E_{v}\times M}$ computes the linear transformation
of the heads.
Popular Transformers Backbones: Here, we briefly discuss some recent
transformers-based backbones.
ViT: The work of [1] introduces an architecture, where a pure transformer is
utilized directly to a sequence of image patches for the task of image
classification. The ViT architecture design does not employ image-specific
inductive biases (e.g., translation equivariance and locality) and the pre-
training is performed on large-scale ImageNet-21k or JFT-300M dataset.
Swin: Liu et al. [29] improve over ViT design by introducing an architecture
that produces hierarchical feature representation. Swin transformer has linear
computational complexity with respect to input image size, where the
efficiency is achieved by restricting the self-attention computation to non-
overlapping local windows while enabling cross-window connection.
PVT: The work of [30] introduces a pyramid vision transformer (PVT)
architecture, to perform pixel-level dense prediction tasks. The PVT
architecture utilizes a progressively shrinking pyramid and a spatial-
reduction attention layer for producing high-resolution multi-scale feature
maps. The PVT backbone has shown to achieve impressive performance on object
detection and segmentation tasks, compared to its CNNs counterpart with
similar number of parameters.

Transformers offer unique characteristics that are useful for different vision
tasks. Compared to the convolution operation in CNNs where static filters are
computed, filters in self-attention are dynamically calculated. Furthermore,
permutations and changes in the number of input points have little effect on
self-attention. Recent studies [2, 3] have explored different interesting
properties of vision transformers and compare them with CNNs. For instance,
the recent work of [2] shows that vision transformers are more robust to
severe occlusions, domain shifts and pertubations. Next, we present a review
of transformers in remote sensing based on the taxonomy shown in Fig.4.

##  V Transformers in VHR Imagery

Here, we review transformers-based approaches utilized to address different
problems in very-high resolution (VHR) imagery.

###  V-A Scene Classification

Remote sensing scene classification is a challenging problem, where the task
is to automatically associate a semantic category label to a given high-
resolution image comprising ground objects and different land cover types.
Among the existing vision transformers-based VHR scene classification
approaches, Bazi et al.[4] explore the impact of standard vision transformers
architecture of [1] (ViT) and investigate different data augmentation
strategies for generating addition data. In addition, their work also evaluate
the impact of compressing the network by pruning the layers while maintaining
the classification accuracy. The work of [31] introduces a joint CNN-
transformers framework, where there is one stream of CNNs and another stream
of ViT, as shown in Fig.5. The features from the two streams are concatenated
and the entire framework is trained using a joint loss function, comprising
cross-entropy and center losses, to optimize the two-stream architecture.
Zhang et al.[32] introduce a framework, called Remote Sensing Transformer
(TRS), that strives to combine the merits of CNNs and transformers by
replacing the spatial convolutions with multi-head self-attention. The
resulting multi-head self attention bottleneck has fewer parameters and is
shown to be effective compared to other bottlenecks. The work of [5]
introduces a two-stream Swin transformers network (TSTNet), that comprises two
streams: original and edge. The original stream extracts standard image
features whereas the edge stream contains a differentiable edge Sobel operator
module and provides edge information. Further, a weighted feature fusion
module is introduced to effectively fuse the features from the two streams for
boosting the classification performance. The work of [6] introduces a
transformers-based framework with a patch generation module designed to
generate homogeneous and heterogeneous patches. The patch generation module
generates the heterogeneous patches directly, whereas the homogeneous patches
are obtained using a superpixel segmentation method.

Figure 5: The CTNet architecture comprising two modules: the ViT stream
(T-stream) and the CNNs stream (C-stream). The T-stream and C-stream are
designed to capture semantic features and the local structural information.
Figure is from [31]. Best viewed zoomed in.  Figure 6:  Comparison in terms of
response maps obtained using different models on example VHR images. The
original images are shown in (a), whereas the evaluated models are: (b) IMP-
ResNet-50, (c) SeCo-ResNet-50, (d) RSP-ResNet-50, (e) IMP-Swin-T, (f) RSP-
Swin-T, (g) IMP-ViTAEv2-S and (h) RSP-ViTAEv2-S. Here, IMP denotes ImageNet
pre-training and RSP refers to remote sensing pre-training. In the response
map, the warmer color indicates higher response. Figure is from [7].

Remote Sensing Pre-training: Different from the aforementioned approaches that
either use only transformers or hybrid CNN-transformers designs with backbone
networks pretrained on ImageNet datasets, the recent work of [7] investigates
training vision transformers backbones, such as Swin, from scratch on the
large-scale MillionAID remote sensing dataset [33]. The resulting trained
backbone models are then fine-tuned for different tasks, including scene
classification. Fig.6 shows the response maps, obtained using Grad-CAM++ [34],
of different ImageNet (IMP) and remote sensing pre-trained (RSP) models. It
can be observed that RSP models learn better semantic representations by
paying more attention to the important targets, compared to IMP counterparts.
Further, the transformers-based backbones, such as Swin-T better captures the
contextual information due to the self-attention mechanism. Moreover,
backbones such as, ViTAEv2-S that combines the merits of CNNs and transformers
along with RSP can achieve better recognition performance.

Tab. I shows a comparison of aforementioned classification approaches on one
of the most commonly used VHR classification benchmarks: AID [35]. The AID
dataset contains images acquired from multi-source sensors. The dataset
possesses a high degree of intra-class variation, since the images are
collected from different countries, under different time and seasons with
variable imaging conditions. There are in total 10,000 images in the dataset
and 30 categories. The performance is measured in terms of mean classification
accuracy over all the categories. For more details on AID, we refer to [35].
Other than RSP that performs an initial pre-training on Million-AID dataset,
all approaches here utilize models pre-trained on ImageNet benchmark.

###  V-B Object Detection

Localizing objects in VHR imaging is a challenging problem due to extreme
scale variations and diversity of different object classes. Here, the task is
to simultaneously recognize and localize (either rectangle or oriented
bounding-boxes) all instances belonging to different object categories in an
image. Most existing approaches employ a hybrid strategy by combining the
merits of CNNs and transformers within existing two-stage and single-stage
detectors. Other than the hybrid strategy, few recent works also explore DETR-
based transformers object detection paradigm [36].

Table I: Performance in terms of classification accuracy of different transformers-based methods on the popular AID dataset with 20:80 train-test ratio.  Method | Venue | Backbone |  AID (20$\%$)
---|---|---|---
V16-21K [4] | Remote Sensing | ViT | 94.97
CTNet [31] | GRSL | ResNet34 + ViT | 96.35
TRS [32] | Remote Sensing | TRS | 95.54
TSTNet [5] | Remote Sensing | Swin-T | 97.20
RSP [7] | TGRS | RSP-Swin-T-E300 | 96.83

Hybrid CNN-Transformers based Methods: The work of [37] introduces a local
perception Swin transformer (LPSW) backbone to improve the standard
transformers for detecting small-sized objects in VHR imagery. The proposed
LPSW strives to combine the merits of transformers and CNNs to improve the
local perception capabilities for better detection performance. The proposed
approach is evaluated with different detectors such as, Mask RCNN [38]. The
work of [39] introduces a transformers-based detection architecture, where a
pre-trained CNN is used to extract features and a transformer is adapted to
process feature pyramid of a remote sensing image. Zhang et al.[40] introduce
a detection framework, where an efficient transformer is utilized as a branch
network to improve CNN’s ability to encode global features. Additionally, a
generative model is employed to expand the input remote sensing aerial images
ahead of the backbone network. The work of [41] proposes a detection framework
based on RetinaNet, where a feature pyramid transformer (FPT) is utilized
between the backbone network and the post-processing network to generate
semantically meaningful features. The FPT enables the interaction among
features at different levels across scale. The work of [42] introduces a
framework, where transformers are adopted to model the relationship of sampled
features in order to group them appropriately. Consequently, better grouping
and bounding box predictions are obtained without any post-processing
operation. The proposed approach effectively eliminates the background
information which helps in achieving improved detection performance.

Zhang et al.[43] introduce a hybrid architecture that combines the local
characteristics of depth separable convolutions with the global (channel)
characteristics of MLP. The work of [44] introduces a two-stage angle-free
detector, where both the RPN and regression are angle-free. Their work also
evaluates the proposed detector with transformers-based backbone (Swin-Tiny).
Liu et al.[45] propose a hybrid network architecture, called TransConvNet,
that aims at combining the advantages of CNNs and transformers by aggregating
both global and local information to address the rotation invariability of
CNNs with a better contextual attention. Furthermore, an adaptive feature
fusion network is designed to capture information from multiple resolutions.
The work of [46] introduces a detection framework, called Oriented Rep-Points,
that utilizes flexible adaptive points as a representation. The proposed
anchor-free approach learns to select the point samples from classification,
localization and orientation. Specifically, to learn geometric features for
arbitrarily-oriented aerial objects, a quality assessment and sample
assignment scheme is introduced that measures and identify high-quality sample
points for training, as shown in Fig. 7. Further, their approach utilizes a
spatial constraint for penalizing the sample points that are outside the
oriented box for robust learning of the points.

Figure 7: Overview of the anchor-free Oriented RepPoints detection
architecture [46] that strives to learn selecting points samples for
classification, regression and orientation. RepPoints utilizes same structure
of the shared head as in [46], except a quality assessment and sample
assignment strategy (APAA) is employed for selecting high-quality sample
points for training. Figure is from [46]. Best viewed zoomed in.

DETR-based Detection Methods: Few recent approaches have investigated adapting
the transformers-based DETR detection framework [36] for oriented object
detection in VHR imaging. The work of [47] adapts the standard DETR for
oriented object detection. In their approach, efficient encoder is designed
for transformers by replacing the standard attention mechanism with a
depthwise separable convolution. Dai et al.[48] propose a transformers-based
detector, called AO2-DETR, where an oriented proposal generation scheme is
employed to explicitly produce oriented object proposals. Further, their
approach comprises an adaptive oriented proposal refinement module that is
designed to compute rotation-invariant features by eliminating the
misalignment between region features and objects. Furthermore, a rotation-
aware matching loss is utilized to perform a matching process for direct set
prediction without the duplicated predictions.

Tab.II shows a comparison of the aforementioned detection approaches on the
most commonly used VHR detection benchmark, DOTA [49]. The dataset comprises
2,806 large aerial images of 15 different object categories plane, baseball
diamond, basketball court, soccer-ball field, bridge, ground track field,
small vehicle, Ship, large vehicle, tennis court, roundabout, swimming pool,
harbor, storage tank and helicopter. The detection performance accuracy is
measured in terms of mean average precision (mAP). For more details on DOTA,
we refer to [49]. The results show that most of these recent methods obtain
similar detection accuracy, with a slight improvement in performance is
obtained when using the Swin-T backbone.

Table II: Comparison in terms of detection accuracy (mAP) of different detectors utilizing a hybrid CNN-transformers design, transformers pre-trained backbone or a DETR-based transformers architecture on DOTA benchmark. The results are presented on the orientated bounding-boxes task of DOTA benchmark.  Method | Venue | Backbone | DOTA
---|---|---|---
ADT-Det [41] | Remote Sensing | ResNet50 | 76.89
RBox [42] | CVPR | ResNet50 | 79.59
Rodformer [43] | Sensors | ResNet50 | 63.89
Rodformer [43] | Sensors | ViT-B4 | 75.60
PointRCNN [44] | Remote Sensing | Swin-T | 80.14
Hybrid Network [45] | Remote Sensing | TransC-T | 78.41
Oriented RepPoints [46] | Arxiv | ResNet50 | 75.97
Oriented RepPoints [46] | Arxiv | Swin-T | 77.63
O2DETR [47] | Arxiv | ResNet50 | 79.66
AO2-DETR [48] | Arxiv | ResNet50 | 79.22

###  V-C Image Change Detection

In remote sensing, image change detection is an important task for detecting
changes on the surface of the Earth with numerous applications in agriculture
[50, 51], urban planning [52], and map revision [53]. Here, the task is to
generate change maps obtained by comparing the multi-temporal or bi-temporal
images, with each pixel in the resulting binary change map having either zero
or one value depending on whether the corresponding position has changed or
not. Among the recent transformers-based change detection approaches, Chen et
al.[54] propose a bi-temporal image transformer, encapsulated in a deep
feature differencing-based framework that is designed to model the spatio-
temporal contextual information. Within the proposed framework, the encoder is
employed to capture context in token-based space-time. The resulting
contextualized tokens are then fed to the decoder where the features are
refined in the pixel-space. Guo et al.[55] propose a deep multi-scale Siamese
architecture, called MSPSNet, that utilizes a parallel convolutional structure
(PCS) and self-attention. The proposed MSPSNet performs feature integration of
different temporal images via PCS and then feature refinement based on self-
attention to further enhance the multi-scale features. The work of [56]
introduces a Swin transformer-based network with a Siamese U-shaped structure,
called SwinSUNet, for change detection. The proposed SwinSUNet comprises three
modules: encoder, fusion, and decoder. The encoder transforms the input image
into tokens and produces multi-scale features by employing a hierarchical Swin
transformer. The resulting features are concatenated in the fusion having
linear projection and Swin transformer blocks. The decoder contains upsampling
and merging within Swin transformer blocks to progressively generate change
predictions.

Wang et al.[57] introduce an architecture, called UVACD, that combines CNNs
and transformers for change detection. Within UVACD, the high-level semantic
features are extracted via a CNN backbone, whereas transformers are utilized
to generate better change features by capturing the temporal information
interaction. The work of [58] introduces a hybrid architecture, TransUNetCD,
that strives to combine the merits of transformers and UNet. Here, the encoder
takes features extracted from CNNs and enrich them with global contextual
information. The corresponding features are then unsampled and combined with
multi-scale features to obtain global-local features for localization. The
work of [59] introduces a hybrid multi-scale transformer, called Hybrid-
TransCD, that captures both fine-grained and large object features by
utilizing heterogeneous tokens via multiple receptive fields.

Table III: Comparison in terms of F1 score of different transformers-based change detection methods on the two popular benchmarks: WHU and LEVIR.  Method | Venue | WHU | LEVIR
---|---|---|---
CD-Trans [54] | TGRS | 83.98 | 89.31
MSPSNet [55] | TGRS | - | 89.18
UVACD [57] | Remote Sensing | 92.84 | 91.30
SwinSUNet [56] | TGRS | 93.8 | -
TransUNetCD [58] | TGRS | 93.59 | 91.1
HybridTransCD [59] | IJGI | - | 90.06

Tab. III shows a comparison of aforementioned change detection approaches on
the most commonly used benchmarks: WHU [60] and LEVIR [61]. The WHU dataset
comprises a single pair of high-resolution (0.075m) images. Here, the images
are of size 32507 $\times$ 15354\. The LEVIR dataset comprises 637 pairs of
high-resolution (0.5m) images. The images are of size 1024 $\times$ 1024\. The
performance is measured in terms of F1 score with respect to the change
category. Fig. 8 presents a qualitative comparison of different methods with
SwinSUNet on example images from WHU-CD dataset.

Figure 8: Results of different CD methods visualized, such as FC-EF[62], FC-
Siam-Conc[62], FC-Siam-Diff[62], CDNet[63], DASNet[64], STANet[61], and
SwinSUNet[56] compared to (a)–(d) sample imagery sets the WHU-CD[60] test set.
Such as Various colors were utilised to convey different denotations; white
represents true positive, black represents true negative, red represents false
positive, and green represents false negative. Figure is from [56].

###  V-D Image Segmentation

In remote sensing, automatically segmenting an image into semantic categories
by performing pixel-level classification is a challenging problem with a wide
range of applications, including geological surveys, urban resources
management, disaster management and monitoring. Most existing transformers-
based remote sensing image segmentation approaches typically employ a hybrid
design with an aim to combine the merits of CNNs and transformers. The work of
[65] introduces a light-weight transformers-based framework, Efficient-T, that
comprises an implicit edge enhancement technique. The proposed Efficient-T
employs hierarchical Swin transformers along with MLP head. A coupled CNN-
transformers framework, called CCTNet, is introduced in [66] that aims at
combining the local details such as, edges and texture captured by the CNNs
along with the global contextual information obtained via transformers for
crop segmentation in remote sensing images. Furthermore, different modules
such as, test time augmentation and post-processing steps are introduced in
order to remove holes and small objects at inference for restoring the
complete segmented images. A CNN-transformers framework, named STransFuse, is
introduced in [67] where both coarse-grained and fine-grained feature
representations at multiple scales are extracted and later combined adaptively
by utilizing self-attentive mechanism. The work of [68] proposes a hybrid
architecture, where Swin transformer backbone that captures long-range
dependencies is combined with a U-shaped decoder which employs an atrous
spatial pyramid pooling block based on depth-wise separable convolution along
with SE block to better preserve local details in an image. The work of [69]
utilizes a pre-trained Swin Transformer backbone along with three decoder
designs namely, U-Net, feature pyramid network and pyramid scene parsing
network for semantic segmentation in aerial images.

Table IV: Performance comparison in terms of overall accuracy (OA) of different transformers-based semantic segmentation methods on two popular benchmarks: Potsdam and Vaihingen.  Method | Venue | Potsdam | Vaihingen
---|---|---|---
Efficient-T [65] | Remote Sensing | 90.08 | 88.41
STransFuse [67] | JSTAR | 86.71 | 86.07
Trans-CNN [68] | TGRS | 91.0 | 90.40
SwinTF [69] | Remote Sensing | - | 90.97
Figure 9: A qualitative comparison between the hybrid Trans-CNN with other
existing segmentation approaches. The examples are from the Potsdam dataset.
Every two rows present the results as a group. here, from left to right and
top to bottom are: (a) the corresponding ground-truth, (b) results obtained
from AFNet + TTA, (c) results of ResUNet, (d) results of CASIA2, (e) results
achieved using Trans-CNN, and (f) the RGB image. The inccorect classification
results from AFNet + TTA, ResUNet, CASIA2 and Trans-CNN are presented in (g),
(h), (i) and (j), respectively. Figure is from [68].

We present in Tab. IV a quantitative comparison of aforementioned approaches
on the two most commonly used semantic segmentation datasets: Potsdam[70] and
Vaihingen[71]. The Potsdam dataset comprises 38 patches, where each patch has
a resolution of 6000 $\times$ 6000 pixels collected over the Potsdam City with
a ground sampling distance of 5 cm. The dataset has six categories. The
Vaihingen dataset comprises 33 samples, where each sample has a resolution
from 1996 $\times$ 1995 to 3816 $\times$ 2550 pixels. Here, the ground
sampling distance is 9 cm. This dataset contains same categories as Potsdam.
The performance is measured in terms of overall accuracy (OA) computed using
true positive, false positive, false negative and true negative. Fig.9
presents a qualitative comparison between Trans-CNN and other approaches on
the Potsdam dataset.

Building Extraction: Transformers-based techniques have also been recently
explored for the problem of building extraction, where the task is to
automatically identify building and non-building pixels in a remote sensing
image. A dual-pathway transformers framework is introduced in [72] that
strives to learn long-range dependencies both in spatial and channel
directions. The work of [73] proposes a transformers framework, STEB-UNet,
comprising Swin transformer-based encoding booster that captures semantic
information from multi-level features generated from different scales. The
encoder booster is further integrated in a U-shaped network design that fuses
local and large-scale semantic features. A transformers-based architectures,
called BuildFormer, comprising a window-based linear attention, a
convolutional MLP along with batch normalization is introduced in [74]. The
work of [75] explores the problem of generalizability of building extraction
models to different areas and propose a transfer learning approach to fine-
tune models from one area to a subset of another unseen area.

Other than semantic image segmentation and building extraction with
transformers, a recent work by [37] explores the problem of instance
segmentation where the task is to automatically classify each pixel into an
object class within an image while also differentiating multiple object
instances. Their approach aims at combining the advantages of CNNs and
transformers by designing a local perception Swin transformer backbone to
enhance both local and global feature information.

###  V-E Others

Apart from the problems discussed above, transformers-based techniques are
also explored for other VHR remote sensing tasks such as, image captioning and
super-resolution.

Table V: Overview of transformers-based approaches in VHR remote sensing
imaging. Here, we highlight transformers-based methods for different VHR
remote sensing tasks.

3 Transformers in Very-High Resolution (VHR) Satellite Imagery
---
2  Method | Task | Datasets | Metrics |  Highlights
2 V16-21K [4] | Classification |  Merced[76], AID[35], Optimal31[77], NWPU[78] | Overall classification accuracy |  Explores vision transformers along with combination of data augmentation techniques for boosting accuracy.
TRS [32] | Classification |  Merced[76], AID[35], Optimal31[77], NWPU[78] | Overall classification accuracy |  Integrates transformers into CNNs by replacing the last three ResNet bottlenecks with encoders having multi-head self-attention bottleneck.
TSTNet[5] | Classification |  Merced[76], AID[35], NWPU[78] |  Overall classification accuracy |  A Swin transformer based two-stream architecture that uses both deep features from the image and edge features from edge stream.
CTNet[31] | Classification |  AID[35], NWPU[78] | Overall classification accuracy |  Comprises a ViT stream that mines semantic features and the CNN stream which captures local structural features.
HHTL[6] | Classification |  Merced[76], AID[35], RSSDIVCS[79], NWPU[78] |  Overall classification accuracy |  Explores integrating heterogenous non-overlapping patches and homogenous patches obtained using superpixel segmentation.
RSP [7] | Classification, Segmentation, Detection |  MillionAID[33], Potsdam [70], iSAID[80], HRSC2016 [81], DOTA[49], CCD[82], LEVIR[61] |  Overall classification accuracy, mAP, F1 score |  Investigates pre-training transformers on a large-scale remote sensing dataset.
SAIEC [37] | Detection, Segmentation |  DIOR[83], HRRSD[84], NWPU VHR-10[85] |  mAP |  Introduces a local perception Swin transformer backbone that aims to combine the merits of transformers and CNNs for improving the local perception capabilities.
T-TRD-DA [39] | Detection |  DIOR[83], NWPU VHR-10[85] |  mAP |  Proposes a transformers-based detector utilizing a pre-trained CNN for feature extraction and multiple-layer transformers for multi-scale feature aggregation at global spatial positions.
GANsformer [40] | Detection |  DIOR[83], NWPU VHR-10[85] |  mAP |  Introduces an efficient transformer, with reduced parameters, as a branch network to capture global features along with a generative model to expand the input image ahead of backbone.
ADT-Det [41] | Detection |  DIOR[83], HRSC2016 [81] |  mAP |  Introduces a RetineNet-based framework with a feature pyramid transformer integrated between the backbone and post-processing network for generating multi-scale semantic features.
PointRCNN [44] | Detection |  DOTA[49], HRSC2016 [81] |  mAP |  Introduces a two-stage angle-free dectection framework which is also evaluated using the transformers-based Swin backbone.
HybridNetwork22 [45] | Detection |  DOTA[49], UCAS-AOD[86], VEDAI[87] |  mAP |  Integrates multi-scale global and local information from transformers and CNNs through an adaptive feature fusion network.
Oriented RepPoints [46] | Detection |  DOTA[49], UCAS-AOD[86], HRSC2016 [81] |  mAP |  Proposes an anchor-free detector learns flexible adaptive points as representations through a quality assessment and sample assignment scheme.
O2DETR [47] | Detection |  DOTA[49], SKU110K-R[88], HRSC2016 [81] |  mAP |  Extends the standard DETR for oriented detection by introducing an encoder employing depthwise separable convolution.
AO2DETR [48] | Detection |  DOTA[49] |  mAP |  Introduces a DETR-based detector with oriented proposal generation scheme, a refine module to compute rotation-invariant features and a rotation-aware matching loss for performing the matching process for direct set predictions.
RBox [42] | Detection |  SynthText[89], ICDAR 2015 (IC15)[90], MLT-2017 (MLT17)[91], MSRA-TD500[92], MTWI[93], Total-Text[94], CTW1500[95] |  mAP |  Proposes a framework employing transformers to model the relationship of sampled features for better grouping and box prediction without requiring post-processing operation.
Rodformer [43] | Detection |  DOTA[49] |  mAP |  A hybrid detection architecture integrating the local characteristics of depth-separable convolutions with the global characteristics of MLP.
CD-Trans [54] | Change Detection |  WHU[60], LEVIR[61], DSIFN[96] |  F1 score |  Introduces a bi-temporal image transformer designed to model the spatio-temporal contextual information. The encoder captures context in token-based space-time, which are then fed to decoder where feature refinement is performed in the pixel-space.

3  Method | Task | Datasets | Metrics |  Highlights
---|---|---|---|---
2 MSPSNet [55] | Change Detection |  SYSU-CD[97], LEVIR[61] |  F1 score |  Introduces a multi-scale Siamese framework employing a parallel convolutional structure for feature integration of different temporal images and self-attention for feature refinement.
SwinSUNet [56] | Change Detection |  CCD[82], WHU[60], OSCD[98], HRSCD[99] |  F1 score |  Introduces a Swin transformer-based network with a Siamese U-shaped structure having encoder, fusion and decoder modules.
TransUNetCD [58] | Change Detection |  WHU[60], LEVIR[61], CCD[82], DSIFN[96], OSCD[98], S2Looking[100] |  F1 score |  Introduces a framework integrating merits of transformers and UNet through capturing enriched contextualized features which are upsampled and fused with multi-scale features to generate global-local features.
Hybrid-TransCD [59] | Change Detection |  LEVIR[61], SYSU-CD[97] |  F1 score |  Introduces a multi-scale transformer that encodes both fine-grained and large object features through heterogeneous tokens via multiple receptive fields.
CCTNet [66] | Segmentation |  Barley Remote Sensing Dataset [101] |  F1 score, Overall accuracy |  Proposes a hybrid CNN-transformers framework to combine local details and global conextual information for crop segmentation.
STransFuse [67] | Segmentation |  Potsdam[70], Vaihingen[71] |  F1 score, Overall accuracy |  Introduces a framework that encodes both coarse-grained as well as fine-grained features at multiple scales which are fused using self-attentive mechanism.
Trans-CNN [68] | Segmentation |  Potsdam[70], Vaihingen[71] |  F1 score, Overall accuracy |  Introduces a framework with a Swin transformer backbone to capture long-range dependencies and a U-shaped decoder with depth-wise separable convolution to encode local details.
SwinTF [69] | Segmentation |  Vaihingen[71], Thailand North Landsat-8 corpus (private), Thailand Isan Landsat-8 corpus (private) |  F1 score, Overall accuracy |  Introduces a framework with pre-trained Swin backbone along with a U-Net, feature pyramid network and a pyramid scene parsing network for segmentation.
Efficient-T [65] | Segmentation |  Potsdam[70], Vaihingen[71] |  F1 score, Overall accuracy |  Proposes a light-weight framework consisting of an implicit edge enhancement scheme along with a Swin transformers.
STT [72] | Building Extraction |  WHU[60], INRIA[102] |  IoU, Overall accuracy, F1 score |  Introduces a transformers framework to learn long-range dependencies both in the spatial and channel direction.
STEB-UNet [73] | Building Extraction |  WHU[60], Massachusetts[102] |  IoU, F1 score |  Introduces a transformers framework capturing semantic information from multi-scale features which are further fused to local features.
BuildFormer [74] | Building Extraction |  WHU[60], Massachusetts[102], INRIA[102] |  IoU, F1 score |  Introduces an architecture consisitng of a window-based linear attention and a convolutional MLP.
T-Trans [75] | Building Extraction |  Massachusetts[102] ,INRIA[102] |  IoU, F1 score |  Explores the task of generalizability of building extraction models to different areas and introduces a transfer learning method to fine-tune models from one area to a subset of another unseen area.
TRL [103] |  Image Captioning |  RSICD[104], UCM-captions [105], Sydney-Caption[106] |  BLEU, ROUGE, METEOR, CIDEr |  Proposes an approach adapting transformers by integrating residual connections, dropout and adatpive feature fusion for remote sensing image caption generation.
MLAT[107] |  Image Captioning |  RSICD[104], UCM-captions[105], Sydney-Caption[106]] |  BLEU, ROUGE, METEOR, CIDEr |  Introduces an architecture where multi-scale features from CNN layers are extracted in encoder and a multi-layer aggregated transformer in the decoder uses those features for sentence generation.
Ren et al.[108] |  Image Captioning |  RSICD[104], UCM-captions[105], Sydney-Caption[106] |  BLEU, ROUGE, METEOR, CIDEr |  Proposes a topic token-based mask transformers with the topic token being integrated into encoder while serving as prior in decoder for capturing global semantic relationships.
TR-MISR [109] |  Image Super Resolution |  RSICD[104], UCM-captions[105], PROBA-V[110] |  cPSNR, cSSIM |  Introduces a transformers-based architecture with an encoder having residual blocks, a fusion module along with a super-pixel convolution-based decoder for multi-image super-resolution.
MSE-Net [111] |  Image Super Resolution |  UCMerced[112], AID[35] |  cPSNR, cSSIM |  Proposes a multi-stage enchancement framework to utilize features from different stages and further integrating them with standard super-resolution technique for combining multi-resolution low as well as high-dimension feature representations.
SRT [113] |  Image Super Resolution |  UCMerced[112] |  cPSNR, cSSIM |  Introduces a hybrid framework that integrates local features from CNNs and global features from transformers.

Image Captioning: Image captioning in remote sensing images is a challenging
problem, where the task is to generate semantically natural description of a
given image. Few recent works have explored using transformers for image
captioning. The work of [103] introduces a framework, where standard
transformers are adapted for remote sensing image caption generation by
integrating residual connections, dropout layers and fusing features
adaptively. Moreover, a reinforcement learning technique is utilized to
further improve the caption generation process. An encoder-decoder
architecture is introduced in [107], where the multi-scale features are first
extracted from different layers of CNNs in the encoder and then a multi-layer
aggregated transformer is utilized in the decoder to effectively exploit the
multi-scale features for generating sentences. The work of [108] introduces a
topic token-based mask transformers framework, where a topic token is
integrated into the encoder and serves as a prior in the decoder for capturing
improved global semantic relationships. Image Super Resolution Remote sensing
image super-resolution is the task of recovering high-resolution images from
their low-resolution counterparts. A few recent works have explored
transformers for this task. A transformer-based multi-stage enhancement
structure is introduced in [111] that leverages features from different
stages. The proposed multi-stage structure can be combined with conventional
super-resolution techniques in order to fuse multi-resolution low as well as
high-dimension features. [113] proposes a CNN-transformer hybrid architecture
to integrate both local and global feature information for super-resolution.
The work of [109] explores the problem of multi-image super-resolution, where
the task is to merge multiple low-resolution remote sensing images of the same
scene into a high-resolution one. Here, a transformers-based approach is
introduced comprising an encoder having residual blocks, a fusion module and a
super-pixel convolution-based decoder.

To summarize the review of transformers in VHR imagery, we present a holistic
overview of different techniques in literature in Tab.V-E.

##  VI Transformers in Hyperspectral Imaging

As discussed earlier, hyperspectral images are represented by several spectral
brands and analyzing hyperspectral data is crucial in a wide range of
problems. Here, we present a review of recent transformers-based approaches
for different hyperspectral imaging (HSI) tasks.

###  VI-A Image Classification

Here, the task is to automatically classify and assign a category label to
each pixel in an image acquired through hyperspectral sensors. Next, we first
review recent works that are either based on the pure transformers design or
utilize a hybrid CNN-transformers approach. Afterwards, we discuss few recent
transformers-based approaches fusing different modalities for hyperspectral
image classification.

Pure Transformers-based Methods: Among existing works, the approach of [114]
introduces a bi-directional encoder representation from transformers, called
HSI-BERT, that strives to capture global dependencies. The proposed
architecture is flexible and can be generalized from different regions with
the need to perform pre-training. A transformers-based backbone, called
SpectralFormer, is introduced in [8] that can take pixel-wise or patch-wise
inputs and is designed to capture spectrally local sequence knowledge from
nearby hyperspectral bands. SpectralFormer utilizes cross-layer skip
connection to circulate information from shallow to deep layers by learning
soft residuals across layers, thereby producing group-wise spectral
embeddings. To circumvent the problem of fixed geometric structure of
convolution kernels, a spectral–spatial transformer network is proposed in
[115] comprising a spatial attention and a spectral association module. While
the spatial attention aims at connecting the local regions through aggregation
of all input feature channels with spatial kernel weights, the spectral
association is achieved through the integration of all spatial locations of
the corresponding masked feature maps. Transformers are also explored in the
spatial and spectral dimensions in [9]. Here, a framework is introduced
comprising spectral self-attention that learns to capture interactions along
the spectral dimension and a spatial self-attention designed to pay attention
to features along the spatial dimension. The resulting features from both
spectral and spatial self-attention are then combined and input to the
classifier.

Figure 10: Overview of the CTN framework [116] for hyperspectral image
classification. Given the HSI data patches, CTN processes them to center
position encoding (CPE), convolutional transformer and classification modules.
Here, the output represents the category label. Figure is from [116]. Best
viewed zoomed in.

Hybrid CNN-Transformers based Methods: Several works recently have explored
combining the merits of CNNs and transformers to better capture both the local
information as well as long-range dependencies for hyperspectral image
classification. To this end, a convolutional transformer network, named CTN,
is introduced in [116] that utilizes center position encoding to generate
spatial position features by combining pixel positions with spectral features
as well as convolutional transformer to further obtain local-global features,
as shown in Fig.10. A hyperspectral image transformer (HiT) classification
approach is proposed in [117], where convolutions are embedded into
transformers architecture to further integrate local spatial contextual
information. The proposed approach comprises two main modules, where one
module, called spectral-adaptive 3-D convolution projection, is designed to
generate spatial-spectral local information via spectral adaptive 3D
convolution layers from hyperspectral images. The other module, named Conv-
Permutator, employs depthwise convolutions to capture spatial–spectral
representations separately along the spectral, height and width dimensions.
The work of [12] introduces a multi-scale convolutional transformer that
effectively captures spatial-spectral information which can be integrated with
transformers network. Further, a self-supervised pre-task is defined that
masks the token of the central pixel in the encoder, whereas remaining tokens
are input to the decoder in order to reconstruct the spectral information
corresponding to the central pixel. In [118], a spectral–spatial feature
tokenization transformer, called SSFTT, is proposed that generates spectral-
spatial and semantic features. The SSFTT comprises a feature extraction module
that produces low-level spectral and spatial features by employing a 3D and a
2D convolution layer. Furthermore, a Gaussian weighted feature tokenizer is
utilized in SSFTT for feature transformation which are then input to a
transformer encoder for feature representation. Consequently, a linear layer
is employed to generate the sample label. Zhao et al. [116] proposes a
convolutional transformer network (CTN) that employs center position encoding
to combine spectral features with pixel positions. The proposed architecture
introduces convolutional transformer blocks that effectively integrates local
and global features from hyperspectral image patches. Yang et al. [117]
introduces a hyperspectral image transformer (HiT) framework, where
convolution operations are embedded within the transformers design for also
integrating local spatial contextual information. The HiT framework comprises
of a spectral-adaptive 3D convolution projection to capture local spatial-
spectral information. Additionally, the HiT framework employs a conv-
permutator module that uses the depthwise convolution for explicitly capturing
the spatial-spectral information along different dimensions: height, width and
spectral. The work of [118] introduces a spectral–spatial feature tokenization
transformer, named SSFTT, that consists of a spectral-spatial feature
extraction scheme for encoding shallow spectral-spatial features, a feature
transformation module which produces transformed features used as input in the
encoder.

Multi-modal Fusion Transformers based Methods: Few recent transformers-based
works also explore fusing different modalities, such as hyperspectral, SAR,
LiDAR for hyperspectral image classification. A Multi-modal fusion
transformer, MFT, is introduced in [119], that comprises a data fusion scheme
to derive class tokens in the transformers from multi-modal data (e.g., LiDAR,
SAR) along with the standard hyperspectral patch tokens. Further, the
attention mechanism within MFT fuses information from tokens of hyperspectral
and other modalities into a new token of integrated features. The work of
[120] introduces an approach, where a spectral sequence transformer is
utilized to extract features from hyperspectral images along the spectral
dimension and a spatial hierarchical transformer to generate spatial features
in a hierarchical manner from both hyperspectral and LiDAR data.

Figure 11: A qualitative comparison, in terms of visualization of classification maps, between HSI-BERT and several CNN-based methods on the Pavia dataset. Here, (a) CNN, (b) CNN-PPF, (c) CDCNN, (d) DRCNN, and (e) HSI-BERT. Figure is from [114]. Table VI: Comparison in terms of overall accuracy (OA) of some representative CNN-based methods with pure transformers and hybrid CNN-transformers based hyperspectral image classification methods on two popular benchmarks: Indian Pines and Pavia. Here, the results are reported using 200 samples for training for each category.  Method | Venue | Type | Indian Pines | Pavia
---|---|---|---|---
CNN [121] | Sensors | CNNs | 87.01 | 92.27
CNN-PPF [122] | TGRS | CNNs | 93.90 | 96.48
HSI-BERT [114] | TGRS | Pure | 99.56 | 99.75
DSS-TRM [9] | EJRS | Pure | 99.43 | 98.50
CTN [116] | GRSL | Hybrid | 99.11 | 97.48

Tab.VI shows a comparison of some representative CNN-based approaches with
both pure transformers and hybrid CNN-transformers based methods on two
popular hyperspectral image classification benchmarks: Indian Pines and Pavia.
The Indian Pines dataset is acquired through airborne visible/infrared imaging
spectrometer (AVIRIS) sensor in Northwestern Indiana, USA. Here, the images
comprise 145 $\times$ 145 pixels in the spatial dimension, at a ground
sampling distance (GSD) of 20m with 220 spectral bands that cover the
wavelength range of 400–2500 nm. After the removal of noisy bands, 200
spectral brands are retained. The original dataset contains 16 class, where
several methods discard the small classes. For the remaining categories, the
number of training samples are 200 per class. The Pavia dataset comprises
images acquired through the reflective optics system imaging spectrometer
(ROSIS) sensor over Pavia, Italy. Here, the images consist of 610 $\times$ 340
pixels in the spatial dimension, at a GSD of 1.3m with 103 spectral bands
covering from 430 to 860 nm. The dataset contains nine categories, where the
number of training samples are 200 per class. Generally, three metrics are
used to evaluate the performance of methods quantitatively: overall accuracy,
average accuracy and kappa coefficient. The overall accuracy (OA) denotes to
the proportion of correctly classified test samples, whereas average accuracy
(AA) reflects the average recognition accuracy for each category. The kappa
coefficient refers to the consistency between the generated classification
maps from the model and the available ground-truth. Fig.11 presents a
qualitative comparison between HSI-Bert [114] and other existing CNN-based
methods on the Pavia dataset.

Table VII: Overview of transformers-based approaches in Hyperspectral imaging.
Here, we highlight methods for different Hyperspectral remote sensing tasks.

3 Transformers in Hyperspectral Imagery
---
2  Method | Task | Datasets | Metrics |  Highlights
2 SpectralFormer [8] | Classification |  Indian Pines[123], Pavia University[124], Houston2013[125] |  Overall classification accuracy, Kappa |  Introduces a transformers-based backbone to capture spectrally local information from nearby hyperspectral bands by generating group-wise spectral embeddings.
MCT [12] | Classification |  Salinas[126], Yellow River Estuary |  Overall classification accuracy, Kappa |  Proposes a multi-scale convolutional transformer to encode spatial-spectral information that is integrated with transformers network.
MFT[119] | Classification |  University of Houston[125], Trento, MUUFL Gulfport[127], Augsburg scenes |  Overall classification accuracy, Kappa |  Proposes a multi-modal transfomers that derives class tokens from multi-modal data along with the standard hyperspectral patch tokens.
CTN [116] | Classification |  Indian Pines[123], Pavia University[124] |  Overall classification accuracy, Kappa |  Introduces a convolutional transformer network with dedicated blocks that integrates local and global features from hyspectral image patches.
DHViT [120] | Classification |  Trento, Houston 2013[125], Houston 2018[128] |  Overall classification accuracy, Kappa |  Introduces an approach comprising a spectral sequence transformer to encode features along the spectral dimension and a spatial hierarchical transformer to produce hierarchical spatial features for hyperspectral and LiDAR data.
DSS-TRM [9] | Classification |  Pavia University[124], Salinas[126], Indian Pines[123] |  Overall classification accuracy, Kappa |  Introduces a transformers-based approach consisting of spectral self-attention and spatial self-attention to capture interactions along spectral and spatial dimension, respectively.
HiT [117] | Classification |  Indian Pines[123], Pavia University[124], Houston2013[125], Xiongan |  Overall classification accuracy, Kappa |  Proposes a hyperspectral image transformer consisting of a 3D convolution projection module to encode local spatial-spectral details and a conv-permutator modue to capture the information along height, width and spectral dimensions.
HSI-BERT [114] | Classification |  Indian Pines[123], Pavia University[124], Salinas[126] |  Overall classification accuracy |  Proposes a transformers-based method that captures capture global dependencies using a bi-direction encoder representation.
SSFTT [118] | Classification |  Indian Pines[123], Pavia University[124], Houston 2013[125] |  Overall classification accuracy, Kappa |  Proposes a spectral–spatial feature tokenization transformer that utilizes both spectral-spatial shallow and semantic features for representation and learning.
SSTN [115] | Classification |  Pavia University[124], Kennedy Space Center, Indian Pines[123], University of Houston[125], Pavia Center[129] |  Overall classification accuracy, Kappa |  Introduces a spectral–spatial transformer with a spatial attention and a spectral association module. The two modules perform spectral and spatial association through the integration of spectral and spatial locations, respectively.
CTIN [130] | Pan-Sharpening |  worldview II[131], worldview III[132], GaoFen-2 |  IQA, ERGAS, PSNR, SAM |  A transformers-based approach is introduced, where multi-spectral and panchromatic features are captured for joint feature learning across modalities. Further, an invertible neural module performs feature fusion to generate pansharpened images.
HyperTransformer [133] | Pan-Sharpening |  Pavia Center[129], Botswana[134], Chikusei[135] |  Cross-correlation(CC), Spectral Angle Mapping (SAM), RSNR, ERGAS, PSNR |  Introduces a transformers-based framework with separate feature extractors for panchromatic and hyperspectral images and a spectral-spatial fusion module to learn cross-feature space dependencies of features.
PMACNet [136] | Pan-Sharpening |  worldview II[131], worldview III[132] |  spatial correlation coefficient(SCC), spectral angle mapper (SAM) |  Introduces a framework with a parallel CNN structure to learn ROIs from low-resolution image and residuals from high-resolution image. It also contains a a pixel-wise attention module to adapt residuals on the learned ROIs.
CPT-noRef [137] | Pan-Sharpening |  Gaofen-1, worldview II[131], Pleiades[138] |  IQA, ERGAS, SAM, correlation coefficient(CC) |  A CNN-transformers framework where global features are generated using transformers and local features are constructed using a shallow CNNs. The features are combined and a loss formulation having spatial and spectral losses are utilized for training.
MSIT [139] | Pan-Sharpening |  GeoEye-1, QuickBird[140] |  ERGAS, SAM, Q4 |  Introduces a multi-scale spatial–spectral interaction transformer with a convolution-transformer encoder for generating multi-scale global and local features from both low-resolution and panchromatic images.
Su et al.[141] | Pan-Sharpening |  worldview II[131], QuickBird[140], GaoFen-2 |  spatial correlation coefficient(SCC), ESGAS, RMSE, SAM, Q4 |  A transformers-based approach with spatial and spectral feature extraction performed using a Swin model.

###  VI-B Hyperspectral Pansharpening

In the hyperspectral pansharpening problem, the task is to enhance low-
resolution hyperspectral image spatially using the spatial information from
registered panchromatic image, while preserving the spectral information of
the low-resolution image. Pansharpening plays an important role in a variety
of tasks in remote sensing, including classification and change detection.
Previously, CNN-based approaches have shown promising results for this task.
Recently, transformers-based methods have performed favorably for this problem
by also utilizing the useful global contextual information. A multi-scale
spatial–spectral interaction transformer, MSIT, is proposed by [139] that
comprises a convolution–transformer encoder to extract multi-scale local and
global features from low-resolution and panchromatic images. The work of [137]
introduces an architecture, where global features are constructed using
transformers and local features are computed using a shallow CNN. These multi-
scale features extracted in a pyramidal fashion are learned simultaneously.
The proposed approach further introduces a loss formulation with spatial and
spectral loss simultaneously used for training using the real data. Liang et
al. [136] propose a framework, named PMACNet, where both the region-of-
interest from the low-resolution image and the residuals for regression to
high-resolution image are learned in a parallel CNN structure. Afterwards, a
pixel-wise attention module is utilized to adapt the residuals based on the
learned region-of-interest.

A transformers-based regression network is introduced by [141], where the
feature extraction of spatial and spectral information is performed by
utilizing a Swin transformer model. The work of [142] introduces a
transformers-based approach, where multi-spectral and panchromatic features
are formulated as keys and queries for enabling joint learning of features
across the modalities. Further, this work employs an invertible neural module
to perform effective fusion of the features for generating the pansharpened
images. Bandara et al. [133] propose a framework comprising separate feature
extractors for panchromatic and hyperspectral images, a soft attention
mechanism and a spectral-spatial fusion module. The pansharpened image quality
is improved by learning cross-feature space dependencies of the different
features.

To summarize the review of transformers in hyperspectral imaging, we provide a
holistic overview of the existing techniques in literature in Tab.VI-A.

##  VII Transformers in SAR Imagery

As discussed earlier, SAR images are constructed from the signals of the
electromagnetic waves, through a a sensor platform, transmitted to the surface
of Earth. SAR possesses unique characteristics due to being unaffected with
different environmental conditions such as, day, night and fog. Here, we
review recent transformers-based approaches for SAR imaging tasks.

###  VII-A SAR Image Interpretation

Classification: Accurately classifying the target categories within SAR images
is a challenging problem with numerous real-world applications. Recently,
transformers have been explored for automatic interpretation and target
recognition in SAR imagery. The work of [143] explores vision transformers for
polarimetric SAR (PolSAR) image classification. In this framework, the pixel
values of the image patches are considered as tokens and the self-attention
mechanism is employed to capture long-range dependencies followed by multi-
layer perceptron (MLP) and learnable class tokens to integrate features. A
contrastive learning technique is utilized within the framework to reduce the
redundancies and perform the classification task. Fig.12 shows the overview of
the framework and a qualitative comparison in terms of supervised
classification is presented in Fig.13. s

Other than the aforementioned pure transformers-based approach, hybrid methods
utilizing both CNNs and transformers also exist in literature. The work of
[144] introduces a global–local network structure (GLNS) framework that
combines the merits of CNNs and transformers for SAR image classification. The
proposed GLNS employs a lightweight CNN along with an efficient vision
transformer to capture both local and global features which are later fused to
perform the classification task. Other than standard fully-supervised
learning, transformers are also explored in the limited supervision regime
such as, few-shot SAR image classification. Cai et al. [145] introduce a few-
shot SAR classification approach, named ST-PN, where a spatial transformer
network is utilized for performing spatial alignment on CNN-based features.

Figure 12: Overview of the ViT-PolSAR framework [143] for supervised
polarimetric SAR image classification. Here, the pixel values of the SAR image
patches are considered as tokens and then the self-attention mechanism is
utilized to encode longe-range dependencies followed by MLP. Figure is from
[143]. Best viewed zoomed in.  Figure 13: A visual comparison, in terms of
supervised classification of the entire map on the ALOS2 San Francisco
dataset. Here, (a), (b), (c), (d), (e), (f), (g), and (h) shows the results
obtained from Wishart, RBF-SVM, CV-CNN, 3D-CNN, PSENet, SF-CNN and ViT-PolSAR,
respectively. Figure is from [143].

Segmentation and Detection: Detection and segmentation in SAR imagery is vital
for different applications such as, crop identification, target detection, and
terrain mapping. In SAR imagery, segmentation can be challenging due to the
appearance of speckles which is a type of multiplicative noise that increases
with the back-scattering radar magnitude. Among recent transformers-based
approaches, the work of [146] introduces a framework, named GCBANet, for SAR
ship instance segmentation. Within the GCBANet framework, a global contextual
block is employed to encode spatial holistic long-range dependencies.
Furthermore, a boundary-aware box prediction technique is introduced to
predict the boundaries of the ship. Xia et al. [147] introduce an approach,
named CRTransSar, that combines the benefits of CNNs and transformers to
capture both local and global information for SAR object detection. The
proposed CRTransSar works by constructing a backbone with attention and
convolutional blocks. A geospatial transformer framework is introduced in
[148], comprising the steps of image decomposition, multi-scale geo-spatial
contextual attention and recomposition for detecting aircrafts in SAR imagery.
A feature relation enhancement framework is proposed in [149] for aircraft
detection in SAR imagery. The proposed framework adopts a fusion pyramid
structure to combine features of different levels and scales. Further, a
context attention enhancement technique is employed to improve the positioning
accuracy in complex backgrounds.

Other than ship and aircraft detection, the recent work of [150] introduces a
transformers-based framework for 3D detection of oil tank targets in SAR
imagery. In this framework, the incidence angle is input to the transformer as
a prior token followed by a feature description operator that utilizes
scattering centers for refining the predictions.

Table VIII: Overview of transformers-based approaches in SAR imaging. Here, we
highlight methods for different SAR remote sensing tasks.

3 Transformers in SAR Satellite Imagery
---
2  Method | Task | Datasets | Metrics |  Highlights
2 ViT-PolSAR [143] | Classification |  AIRSAR Flevoland [151], ESAR Oberpfaffenhofen[152], AIRSAR San Francisco[153], ALOS2 San Francisco[154] |  AA, OA, Kappa |  Explores transformers, where self-attention is used to capture long-range dependencies followed by MLP for polarimetric SAR image classification.
GLNS [144] | Classification |  Gaofen-3 SAR[155], F-SAR[156] |  AA, OA, Kappa |  Introduces a global–local network structure to exploit the merits of CNNs and transformers with local and global features that are fused to perform classification.
ST-PN [145] | Classification |  MSTAR[157] |  Accuracy |  Proposes a spatial transformer network for spatial alignment of features extracted from CNNs for few-shot SAR classification.
GCBANet [146] | Segmentation |  SSDD[158], HRSID[159] |  AP |  Introduces a transformers-based approach with a global contextual block for capturing spatial holistic long-range dependencies and a boundary-aware prediction scheme for estimating the boundaries of ship.
CRTransSar [147] | Detection |  SMCDD[147], SSDD[158] |  Accuracy, Recall, mAP, F1 |  Proposes a backbone based on convolutional and attention blocks for capturing both local and global features.
Geospatial Transformers [148] | Detection |  Gaofen-3[155] |  DR, FAR |  Introduces a framework with multi-scale geo-spatial attention for aircraft detection in SAR imaging.
SFRE-Net [149] | Detection |  Gaofen-3[155] |  Precision, Recall, F1 |  Introduces a feature relation enhancement architecture consisting of a fusion pyramid structure and a context attention enhancement technique.
3DET-ViT [150] | Detection |  L1B SAR[160] |  AP, AR, Mean Offset |  Proposes a transformers-based framework that takes incidence angle as a prior token with a feature description operator employing scattering centers for prediction refinement.
ID-ViT [161] | Despeckling |  Berkeley Segmentation Dataset[162] |  PSNR, SSIM |  Proposes a framework comprising an encoder to learn global dependencies among SAR image regions, where the network is trained using synthetic speckled data.
CLT [163] | Change Detection |  Brazil and Namibia datasets[164], simulation data[163] |  KC |  Introduces a self-supervised contrastive representation learning method with a convolution-enhanced transformer to generate hierarchical representations for distinguishing changes from HR SAR images.
CF-ViT [165] | Image Registration |  MegaDepth[166] |  KC |  A CNN-transformers framework that first performs coarse registration on the down-sampled image, followed by registration of image pairs via a CNN-transformer module with the resulting point pair subsets integrated to obtain final global registration.

###  VII-B Others

Apart from SAR image classification, detection and segmentation, few works
exist exploring transformers for other SAR imaging problems such as, image
despeckling.

SAR Image Despeckling: The aforementioned interpretation of SAR imaging is
made challenging due to the degradation of images caused by a multiplicative
noise known as speckle. Recently, transformers have been explored for SAR
image despeckling. The work of [161] introduces a transformers-based framework
comprising an encoder that learns global dependencies among various SAR image
regions. The transformers-based network is trained in an end-to-end fashion
with synthetic speckled data by utilizing a composite loss function.

Change Detection in SAR Images: SAR images can be affected by imaging noise
which presents challenges when detecting changes in high-resolution (HR) SAR
data. Recently, a self-supervised contrastive representation learning
technique has been proposed by [163], where hierarchical representations are
constructed using a convolution-enhanced transformer to distinguish the
changes from HR SAR images. A convolution-based module is introduced to enable
interactions across windows when performing self-attention computations within
local windows.

SAR Image Registration: Several applications such as, change detection
involves joint analysis and processing of multiple SAR images that are likely
acquired in different imaging conditions. Thus, accurate SAR image
registration is desired where the reference and the sensed images are
registered. The recent work of [165] explores transformers for large-size SAR
dense-matching registration. Here, a hybrid CNN-transformer is employed to
register images under weak texture condition. First, coarse registration is
performed via the down-sampled original SAR image. Then, cluster centers of
registration points are selected from the previous coarse registration step.
Afterwards, the registration of image pairs are performed using a CNN-
transformer module. Lastly, the resulting point pair subsets are integrated to
achieve the final global transformation through RANSAC.

In summary, we present a holistic overview of the existing transformers
techniques in SAR imagery in Tab.VI-A.

##  VIII Discussion and Conclusion

In this work, we presented a broad overview of transformers in remote sensing
imaging: very-high resolution (VHR), hyperspectral and synthetic aperture
radar (SAR). Within these different remote sensory imagery, we further discuss
transformers-based approaches on a variety of tasks, such as classification,
detection and segmentation. Our survey covers more than 60 transformers-based
remote sensing research works in literature. We observed transformers to
obtain favorable performance on different remote sensing tasks likely due to
their capabilities to capture long-range dependencies along with their
representation flexibility. Further, the public availability of several
standard transformers architectures and backbones make it easier to explore
their applicability in remote sensing imaging problems.

Open Research Directions: As discussed earlier, most existing transformer-
based recognition approaches employ backbones pre-trained on the ImageNet
dataset. One exception is the work of [7] that explore pre-training vision
transformers on a large-scale remote sensing dataset. However, in both cases
the pre-training is performed in a supervised fashion. An open direction is to
explore large-scale pre-training in a self-supervised fashion by taking into
account an abundant amount of unlabeled remote sensing imaging data.

Our survey also shows that most existing approaches typically utilize a hybrid
architecture where the aim is to combine the merits of convolutions and self-
attention. However, transformers are typically known to have a higher
computational cost to compute global self-attention. Several recent works have
explored different improvements in the transformers design such as, reduced
computational overhead [167], efficient hybrid CNN-transformers backbones
[168] and unified architectures for image and video classification [169].
Moreover, due to the utilization of more training data by transformers, there
is a need to construct larger-scale datasets in remote sensing imaging. For
most problems discussed in this work and especially in case of object
detection, heavy backbones are typically utilized to achieve better detection
accuracy. However, this significantly slows down the speed of the aerial
detector. An interesting open direction is to design light-weight
transformers-based backbones to classify detect oriented targets in remote
sensing imagery. Another open research direction is to explore the
adaptability of the transformers-based models to heterogeneous source of
images such as, SAR and UAV (e.g., change detection).

In this survey, we also observe several existing approaches to utilize
transformers in a plug-and-play fashion for remote sensing. This leads to the
need of designing effective domain-specific architectural components and loss
formulations to further boost the performance. Moreover, it is intriguing to
study the adversarial feature space of vision transformers models that are
pre-trained on remote sensing benchmarks and their transferability.

## References

  * [1] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in ICLR, 2021.
  * [2] M. Naseer, K. Ranasinghe, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang, “Intriguing properties of vision transformers,” in NeurIPS, 2021.
  * [3] N. Park and S. Kim, “How do vision transformers work?,” in ICLR, 2022.
  * [4] Y. Bazi, L. Bashmal, M. M. A. Rahhal, R. A. Dayil, and N. A. Ajlan, “Vision transformers for remote sensing image classification,” Remote Sensing, 2021\.
  * [5] S. Hao, B. Wu, K. Zhao, Y. Ye, and W. Wang, “Two-stream swin transformer with differentiable sobel operator for remote sensing image classification,” Remote Sensing, 2022.
  * [6] J. Ma, M. Li, X. Tang, X. Zhang, F. Liu, and L. Jiao, “Homo–heterogenous transformer learning framework for rs scene classification,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022\.
  * [7] D. Wang, J. Zhang, B. Du, G.-S. Xia, and D. Tao, “An empirical study of remote sensing pretraining,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
  * [8] D. Hong, Z. Han, J. Yao, L. Gao, B. Zhang, A. Plaza, and J. Chanussot, “Spectralformer: Rethinking hyperspectral image classification with transformers,” IEEE Transactions on Geoscience and Remote Sensing, 2021\.
  * [9] B. Liu, A. Yu, K. Gao, X. Tan, Y. Sun, and X. Yu, “Dss-trm: deep spatial–spectral transformer for hyperspectral image classification,” European Journal of Remote Sensing, 2022.
  * [10] Z. Zhao, D. Hu, H. Wang, and X. Yu, “Convolutional transformer network for hyperspectral image classification,” IEEE Geoscience and Remote Sensing Letters, 2022.
  * [11] X. Yang, W. Cao, Y. Lu, and Y. Zhou, “Hyperspectral image transformer classification networks,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
  * [12] S. Jia and Y. Wang, “Multiscale convolutional transformer with center mask pretraining for hyperspectral image classification,” arXiv preprint arXiv:2203.04771, 2022.
  * [13] D. Tuia, M. Volpi, L. Copa, M. Kanevski, and J. Munoz-Mari, “A survey of active learning algorithms for supervised remote sensing image classification,” IEEE Journal of Selected Topics in Signal Processing, 2011\.
  * [14] G. Camps-Valls, D. Tuia, L. Bruzzone, and J. A. Benediktsson, “Advances in hyperspectral image classification: Earth monitoring with statistical learning methods,” IEEE signal processing magazine, 2013.
  * [15] X. X. Zhu, D. TUIA, L. MOU, G.-S. XIA, L. ZHANG, F. XU, and F. FRAUNDORFER, “Deep learning in remote sensing: A comprehensive review and list of resources,” IEEE Geoscience and Remote Sensing Magazine, 2017.
  * [16] L. Ma, Y. Liu, X. Zhang, Y. Ye, G. Yin, and B. A. Johnson, “Deep learning in remote sensing applications: A meta-analysis and review,” ISPRS journal of photogrammetry and remote sensing, 2019.
  * [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017.
  * [18] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah:, “Transformers in vision: A survey,” ACM Computing Surveys, 2021.
  * [19] F. Shamshad, S. Khan, S. W. Zamir, M. H. Khan, M. Hayat, F. S. Khan, and H. Fu, “Transformers in medical imaging: A survey,” arXiv preprint arXiv:2201.09873, 2022.
  * [20] J. Selva, A. Johansen, S. Escalera, K. Nasrollahi, T. Moeslund, and A. Clapes, “Video transformers: A survey,” arXiv preprint arXiv:2201.05991, 2022\.
  * [21] M. Y. Teng, R. Mehrubeoglu, S. A. King, K. Cammarata, and J. Simons, “Investigation of epifauna coverage on seagrass blades using spatial and spectral analysis of hyperspectral images,” in WHISPERS, 2013.
  * [22] G. Notesco, E. B. Dor, and A. Brook, “Mineral mapping of makhtesh ramon in israel using hyperspectral remote sensing day and night lwir images,” in WHISPERS, 2014.
  * [23] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” NeurIPS, 2012.
  * [24] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” NeurIPS, 2015.
  * [25] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in CVPR, 2009.
  * [26] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  * [27] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in CVPR, 2016.
  * [28] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473, 2014\.
  * [29] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in CVPR, 2021.
  * [30] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in ICCV, 2021.
  * [31] P. Deng, K. Xu, and H. Huang, “When cnns meet vision transformer: A joint framework for remote sensing scene classification,” IEEE Geoscience and Remote Sensing Letters, 2021.
  * [32] J. Zhang, H. Zhao, and J. Li, “Trs: Transformers for remote sensing scene classification,” Remote Sensing, 2021.
  * [33] Y. Long, G.-S. Xia, S. Li, W. Yang, M. Y. Yang, X. X. Zhu, L. Zhang, and D. Li, “On creating benchmark dataset for aerial image interpretation: Reviews, guidances and million-aid,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021.
  * [34] A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, “Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks,” in WACV, 2018.
  * [35] G.-S. Xia, J. Hu, F. Hu, B. Shi, X. Bai, Y. Zhong, L. Zhang, and X. Lu, “Aid: A benchmark data set for performance evaluation of aerial scene classification,” IEEE Transactions on Geoscience and Remote Sensing, 2017\.
  * [36] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in ECCV, 2020.
  * [37] X. Xu, Z. Feng, C. Cao, M. Li, J. Wu, Z. Wu, Y. Shang, and S. Ye, “An improved swin transformer-based model for remote sensing object detection and instance segmentation,” Remote Sensing, 2021.
  * [38] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in ICCV, 2017.
  * [39] Q. Li, Y. Chen, and Y. Zeng, “Transformer with transfer cnn for remote-sensing-image object detection,” Remote Sensing, 2022.
  * [40] Y. Zhang, X. Liu, S. Wa, S. Chen, and Q. Ma, “Gansformer: A detection network for aerial images with high performance combining convolutional network and transformer,” Remote Sensing, 2022.
  * [41] Y. Zheng, P. Sun, Z. Zhou, W. Xu, and Q. Ren, “Adt-det: Adaptive dynamic refined single-stage transformer detector for arbitrary-oriented object detection in satellite optical imagery,” Remote Sensing, 2021.
  * [42] J. Tang, W. Zhang, H. Liu, M. Yang, B. Jiang, G. Hu, and X. Bai, “Few could be better than all: Feature sampling and grouping for scene text detection,” in CVPR, 2022.
  * [43] Y. Dai, J. Yu, D. Zhang, T. Hu, and X. Zheng, “Rodformer: High-precision design for rotating object detection with transformers,” Sensors, 2022\.
  * [44] Q. Zhou and C. Yu, “Point rcnn: An angle-free framework for rotated object detection,” Remote Sensing, 2022.
  * [45] X. Liu, S. Ma, L. He, C. Wang, and Z. Chen, “Hybrid network model: Transconvnet for oriented object detection in remote sensing images,” Remote Sensing, 2022.
  * [46] W. Li, Y. Chen, K. Hu, and J. Zhu, “Oriented reppoints for aerial object detection,” arXiv e-prints, pp. arXiv–2105, 2021.
  * [47] T. Ma, M. Mao, H. Zheng, P. Gao, X. Wang, S. Han, E. Ding, B. Zhang, and D. Doermann, “Oriented object detection with transformer,” arXiv preprint arXiv:2106.03146, 2021.
  * [48] L. Dai, H. Liu, H. Tang, Z. Wu, and P. Song, “Ao2-detr: Arbitrary-oriented object detection transformer,” arXiv preprint arXiv:2205.12785, 2022.
  * [49] G.-S. Xia, X. Bai, J. Ding, Z. Zhu, S. Belongie, J. Luo, M. Datcu, M. Pelillo, and L. Zhang, “Dota: A large-scale dataset for object detection in aerial images,” in CVPR, 2018.
  * [50] B. S. Muzein, Remote sensing & GIS for land cover, land use change detection and analysis in the semi-natural ecosystems and agriculture landscapes of the central Ethiopian Rift Valley.  PhD thesis, 2006.
  * [51] B. Haack, J. Wolf, and R. English, “Remote sensing change detection of irrigated agriculture in afghanistan,” Geocarto international, 1998.
  * [52] J. Bolorinos, N. K. Ajami, and R. Rajagopal, “Consumption change detection for urban planning: monitoring and segmenting water customers during drought,” Water Resources Research, 2020.
  * [53] G. Metternicht, “Change detection assessment using fuzzy sets and remotely sensed data: an application of topographic map revision,” ISPRS Journal of Photogrammetry and Remote Sensing, 1999.
  * [54] H. Chen, Z. Qi, and Z. Shi, “Remote sensing image change detection with transformers,” IEEE Transactions on Geoscience and Remote Sensing, 2021\.
  * [55] Q. Guo, J. Zhang, S. Zhu, C. Zhong, and Y. Zhang, “Deep multiscale siamese network with parallel convolutional structure and self-attention for change detection,” IEEE Transactions on Geoscience and Remote Sensing, 2021.
  * [56] C. Zhang, L. Wang, S. Cheng, and Y. Li, “Swinsunet: Pure transformer network for remote sensing image change detection,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
  * [57] G. Wang, B. Li, T. Zhang, and S. Zhang, “A network combining a transformer and a convolutional neural network for remote sensing image change detection,” Remote Sensing, 2022.
  * [58] Q. Li, R. Zhong, X. Du, and Y. Du, “Transunetcd: A hybrid transformer network for change detection in optical remote-sensing images,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
  * [59] Q. Ke and P. Zhang, “Hybrid-transcd: A hybrid transformer remote sensing image change detection network via token aggregation,” International Journal of Geo-Information, 2022.
  * [60] S. Ji, S. Wei, and M. Lu, “Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set,” IEEE Transactions on Geoscience and Remote Sensing, 2018.
  * [61] H. Chen and Z. Shi, “A spatial-temporal attention-based method and a new dataset for remote sensing image change detection,” Remote Sensing, 2020\.
  * [62] R. C. Daudt, B. Le Saux, and A. Boulch, “Fully convolutional siamese networks for change detection,” in ICIP, pp. 4063–4067, IEEE, 2018.
  * [63] P. F. Alcantarilla, S. Stent, G. Ros, R. Arroyo, and R. Gherardi, “Street-view change detection with deconvolutional networks,” Autonomous Robots, 2018\.
  * [64] J. Chen, Z. Yuan, J. Peng, L. Chen, H. Huang, J. Zhu, Y. Liu, and H. Li, “Dasnet: Dual attentive fully convolutional siamese networks for change detection in high-resolution satellite images,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020.
  * [65] Z. Xu, W. Zhang, T. Zhang, Z. Yang, and J. Li, “Efficient transformer for remote sensing image segmentation,” Remote Sensing, 2021.
  * [66] H. Wang, X. Chen, T. Zhang, Z. Xu, and J. Li, “Cctnet: Coupled cnn and transformer network for crop segmentation of remote sensing images,” Remote Sensing, 2022.
  * [67] L. Gao, H. Liu, M. Yang, L. Chen, Y. Wan, Z. Xiao, and Y. Qian, “Stransfuse: Fusing swin transformer and convolutional neural network for remote sensing image semantic segmentation,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021.
  * [68] C. Zhang, W. Jiang, Y. Zhang, W. Wang, Q. Zhao, and C. Wang, “Transformer and cnn hybrid deep neural network for semantic segmentation of very-high-resolution remote sensing imagery,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
  * [69] T. Panboonyuen, K. Jitkajornwanich, S. Lawawirojwong, P. Srestasathiern, and P. Vateekul, “Transformer-based decoder designs for semantic segmentation on remotely sensed images,” Remote Sensing, 2021.
  * [70] http://www2.isprs.org/commissions/comm3/wg4/2d-sem-label-potsdam.html, “Accessed: 2022-08-27,”
  * [71] https://www2.isprs.org/commissions/comm2/wg4/benchmark/2d-sem-label-vaihingen/, “Accessed: 2022-08-27,”
  * [72] K. Chen, Z. Zou, and Z. Shi, “Building extraction from remote sensing images with sparse token transformers,” Remote Sensing, 2021.
  * [73] X. Xiao, W. Guo, R. Chen, Y. Hui, J. Wang, and H. Zhao, “A swin transformer-based encoding booster integrated in u-shaped network for building extraction,” Remote Sensing, 2022.
  * [74] L. Wang, S. Fang, X. Meng, and R. Li, “Building extraction with vision transformer,” IEEE Transactions on Geoscience and Remote Sensing, 2022\.
  * [75] C. Qiu, H. Li, W. Guo, X. Chen, A. Yu, X. Tong, and M. Schmitt, “Transferring transformer-based models for cross-area building extraction from remote sensing images,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022.
  * [76] Y. Yang and S. Newsam, “Bag-of-visual-words and spatial extensions for land-use classification,” in SIGSPATIAL, 2010.
  * [77] Q. Wang, S. Liu, J. Chanussot, and X. Li, “Scene classification with recurrent attention of vhr remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, 2019.
  * [78] G. Cheng, Z. Li, X. Yao, L. Guo, and Z. Wei, “Remote sensing image scene classification using bag of convolutional features,” IEEE Geoscience and Remote Sensing Letters, 2017.
  * [79] Y. Li, Z. Zhu, J.-G. Yu, and Y. Zhang, “Learning deep cross-modal embedding networks for zero-shot remote sensing image scene classification,” IEEE Transactions on Geoscience and Remote Sensing, 2021.
  * [80] S. Waqas Zamir, A. Arora, A. Gupta, S. Khan, G. Sun, F. Shahbaz Khan, F. Zhu, L. Shao, G.-S. Xia, and X. Bai, “isaid: A large-scale dataset for instance segmentation in aerial images,” in CVPR Workshops, 2019.
  * [81] Z. Liu, L. Yuan, L. Weng, and Y. Yang, “A high resolution optical satellite image dataset for ship recognition and some new baselines,” in ICPRAM, 2017\.
  * [82] M. Lebedev, Y. V. Vizilter, O. Vygolov, V. Knyaz, and A. Y. Rubis, “Change detection in remote sensing images using conditional adversarial networks.,” International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, 2018.
  * [83] K. Li, G. Wan, G. Cheng, L. Meng, and J. Han, “Object detection in optical remote sensing images: A survey and a new benchmark,” ISPRS Journal of Photogrammetry and Remote Sensing, 2020.
  * [84] Y. Zhang, Y. Yuan, Y. Feng, and X. Lu, “Hierarchical and robust convolutional neural network for very high-resolution remote sensing object detection,” IEEE Transactions on Geoscience and Remote Sensing, 2019.
  * [85] G. Cheng, P. Zhou, and J. Han, “Learning rotation-invariant convolutional neural networks for object detection in vhr optical remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, 2016.
  * [86] H. Zhu, X. Chen, W. Dai, K. Fu, Q. Ye, and J. Jiao, “Orientation robust object detection in aerial images using deep convolutional neural network,” in ICIP, 2015.
  * [87] S. Razakarivony and F. Jurie, “Vehicle detection in aerial imagery: A small target detection benchmark,” Journal of Visual Communication and Image Representation, 2016.
  * [88] X. Pan, Y. Ren, K. Sheng, W. Dong, H. Yuan, X. Guo, C. Ma, and C. Xu, “Dynamic refinement network for oriented and densely packed object detection,” in CVPR, 2020.
  * [89] A. Gupta, A. Vedaldi, and A. Zisserman, “Synthetic data for text localisation in natural images,” in CVPR, 2016.
  * [90] D. Karatzas, L. Gomez-Bigorda, A. Nicolaou, S. Ghosh, A. Bagdanov, M. Iwamura, J. Matas, L. Neumann, V. R. Chandrasekhar, S. Lu, et al., “Icdar 2015 competition on robust reading,” in ICDAR, 2015.
  * [91] N. Nayef, F. Yin, I. Bizid, H. Choi, Y. Feng, D. Karatzas, Z. Luo, U. Pal, C. Rigaud, J. Chazalon, et al., “Icdar2017 robust reading challenge on multi-lingual scene text detection and script identification-rrc-mlt,” in ICDAR, 2017.
  * [92] C. Yao, X. Bai, W. Liu, Y. Ma, and Z. Tu, “Detecting texts of arbitrary orientations in natural images,” in CVPR, 2012.
  * [93] M. He, Y. Liu, Z. Yang, S. Zhang, C. Luo, F. Gao, Q. Zheng, Y. Wang, X. Zhang, and L. Jin, “Icpr2018 contest on robust reading for multi-type web images,” in ICPR, 2018.
  * [94] C. K. Ch’ng and C. S. Chan, “Total-text: A comprehensive dataset for scene text detection and recognition,” in ICDAR, 2017.
  * [95] L. Yuliang, J. Lianwen, Z. Shuaitao, and Z. Sheng, “Detecting curve text in the wild: New dataset and new solution,” arXiv preprint arXiv:1712.02170, 2017.
  * [96] C. Zhang, P. Yue, D. Tapete, L. Jiang, B. Shangguan, L. Huang, and G. Liu, “A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sensing images,” ISPRS Journal of Photogrammetry and Remote Sensing, 2020.
  * [97] Q. Shi, M. Liu, S. Li, X. Liu, F. Wang, and L. Zhang, “A deeply supervised attention metric-based network and an open aerial image dataset for remote sensing change detection,” IEEE transactions on geoscience and remote sensing, 2021.
  * [98] R. C. Daudt, B. Le Saux, A. Boulch, and Y. Gousseau, “Urban change detection for multispectral earth observation using convolutional neural networks,” in IEEE International Geoscience and Remote Sensing Symposium, 2018.
  * [99] R. C. Daudt, B. Le Saux, A. Boulch, and Y. Gousseau, “Multitask learning for large-scale semantic change detection,” Computer Vision and Image Understanding, 2019.
  * [100] L. Shen, Y. Lu, H. Chen, H. Wei, D. Xie, J. Yue, R. Chen, S. Lv, and B. Jiang, “S2looking: A satellite side-looking dataset for building change detection,” Remote Sensing, 2021.
  * [101] https://tianchi.aliyun.com/dataset/dataDetail?dataId=74952, “Accessed: 2022-08-27,”
  * [102] E. Maggiori, Y. Tarabalka, G. Charpiat, and P. Alliez, “Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark,” in IGARSS, 2017.
  * [103] X. Shen, B. Liu, Y. Zhou, and J. Zhao, “Remote sensing image caption generation via transformer and reinforcement learning,” Multimedia Tools and Applications, 2020.
  * [104] X. Lu, B. Wang, X. Zheng, and X. Li, “Exploring models and data for remote sensing image caption generation,” IEEE Transactions on Geoscience and Remote Sensing, 2017.
  * [105] https://mega.nz/folder/wCpSzSoS#RXzIlrv--TDt3ENZdKN8JA, “Accessed: 2022-08-27,”
  * [106] https://mega.nz/folder/pG4yTYYA#4c4buNFLibryZnlujsrwEQ, “Accessed: 2022-08-27,”
  * [107] C. Liu, R. Zhao, and Z. Shi, “Remote-sensing image captioning based on multilayer aggregated transformer,” IEEE Geoscience and Remote Sensing Letters, 2022.
  * [108] Z. Ren, S. Gou, Z. Guo, S. Mao, and R. Li, “A mask-guided transformer network with topic token for remote sensing image captioning,” Remote Sensing, 2022\.
  * [109] T. An, X. Zhang, C. Huo, B. Xue, L. Wang, and C. Pan, “Tr-misr: Multiimage super-resolution based on feature fusion with transformers,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022\.
  * [110] M. Märtens, D. Izzo, A. Krzic, and D. Cox, “Super-resolution of proba-v images using convolutional neural networks,” Astrodynamics, 2019.
  * [111] S. Lei, Z. Shi, and W. Mo, “Transformer-based multistage enhancement for remote sensing image super-resolution,” IEEE Transactions on Geoscience and Remote Sensing, 2021.
  * [112] http://vision.ucmerced.edu/datasets/landuse.html, “Accessed: 2022-08-27,”
  * [113] C. Ye, L. Yan, Y. Zhang, J. Zhan, J. Yang, and J. Wang, “A super-resolution method of remote sensing image using transformers,” in IDAACS, 2021.
  * [114] J. He, L. Zhao, H. Yang, M. Zhang, and W. Li, “Hsi-bert: Hyperspectral image classification using the bidirectional encoder representation from transformers,” IEEE Transactions on Geoscience and Remote Sensing, 2019\.
  * [115] Z. Zhong, Y. Li, L. Ma, J. Li, and W.-S. Zheng, “Spectral-spatial transformer network for hyperspectral image classification: A factorized architecture search framework,” IEEE Transactions on Geoscience and Remote Sensing, 2021\.
  * [116] Z. Zhao, D. Hu, H. Wang, and X. Yu, “Convolutional transformer network for hyperspectral image classification,” IEEE Geoscience and Remote Sensing Letters, 2022.
  * [117] X. Yang, W. Cao, Y. Lu, and Y. Zhou, “Hyperspectral image transformer classification networks,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
  * [118] L. Sun, G. Zhao, Y. Zheng, and Z. Wu, “Spectral–spatial feature tokenization transformer for hyperspectral image classification,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
  * [119] S. K. Roy, A. Deria, D. Hong, B. Rasti, A. Plaza, and J. Chanussot, “Multimodal fusion transformer for remote sensing image classification,” arXiv preprint arXiv:2203.16952, 2022.
  * [120] Z. Xue, X. Tan, X. Yu, B. Liu, A. Yu, and P. Zhang, “Deep hierarchical vision transformer for hyperspectral and lidar data classification,” IEEE Transactions on Image Processing, 2022.
  * [121] W. Hu, Y. Huang, L. Wei, F. Zhang, and H. Li, “Deep convolutional neural networks for hyperspectral image classification,” Sensors, 2015.
  * [122] W. Li, G. Wu, F. Zhang, and Q. Du, “Hyperspectral image classification using deep pixel-pair features,” IEEE Transactions on Geoscience and Remote Sensing, 2017.
  * [123] https://purr.purdue.edu/publications/1947/1, “Accessed: 2022-08-27,”
  * [124] https://www.ehu.eus/ccwintco/index.php/Hyperspectral_Remote_Sensing_Scenes#Pavia_Centre_and_University, “Accessed: 2022-08-27,”
  * [125] https://hyperspectral.ee.uh.edu/?page_id=459, “Accessed: 2022-08-27,”
  * [126] https://www.ehu.eus/ccwintco/index.php/Hyperspectral_Remote_Sensing_Scenes#Salinas, “Accessed: 2022-08-27,”
  * [127] P. Gader, A. Zare, R. Close, J. Aitken, and G. Tuell, “Muufl gulfport hyperspectral and lidar airborne data set,” Univ. Florida, Gainesville, FL, USA, Tech. Rep. REP-2013-570, 2013.
  * [128] https://hyperspectral.ee.uh.edu/?page_id=1075, “Accessed: 2022-08-27,”
  * [129] https://www.ehu.eus/ccwintco/index.php/Hyperspectral_Remote_Sensing_Scenes#Pavia_Centre_scene, “Accessed: 2022-08-27,”
  * [130] H. Zhou, Q. Liu, and Y. Wang, “Panformer: a transformer based model for pan-sharpening,” arXiv preprint arXiv:2203.02916, 2022.
  * [131] https://earth.esa.int/eogateway/catalog/worldview-2-full-archive-and-tasking, “Accessed: 2022-08-27,”
  * [132] https://earth.esa.int/eogateway/catalog/worldview-3-full-archive-and-tasking, “Accessed: 2022-08-27,”
  * [133] W. Bandara and V. Patel, “Hypertransformer: A textural and spectral feature fusion transformer for pansharpening,” in CVPR, 2022.
  * [134] https://www.ehu.eus/ccwintco/index.php/Hyperspectral_Remote_Sensing_Scenes#Botswana, “Accessed: 2022-08-27,”
  * [135] N. Yokoya and A. Iwasaki, “Airborne hyperspectral data over chikusei,” tech. rep., 2016.
  * [136] Y. Liang, P. Zhang, Y. Mei, and T. Wang, “Pmacnet: Parallel multiscale attention constraint network for pan-sharpening,” IEEE Geoscience and Remote Sensing Letters, 2022.
  * [137] S. Li, Q. Guo, and A. Li, “Pan-sharpening based on cnn+ pyramid transformer by using no-reference loss,” Remote Sensing, 2022.
  * [138] https://pleiades.stoa.org/downloads, “Accessed: 2022-08-27,”
  * [139] F. Zhang, K. Zhang, and J. Sun, “Multiscale spatial–spectral interaction transformer for pan-sharpening,” Remote Sensing, 2022.
  * [140] https://earth.esa.int/eogateway/catalog/quickbird-full-archive, “Accessed: 2022-08-27,”
  * [141] X. Su, J. Li, and Z. Hua, “Transformer-based regression network for pansharpening remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
  * [142] M. Zhou, J. Huang, Y. Fang, X. Fu, and A. Liu, “Pan-sharpening with customized transformer and invertible neural network,” in AAAI, 2022.
  * [143] H. Dong, L. Zhang, and B. Zou, “Exploring vision transformers for polarimetric sar image classification,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
  * [144] X. Liu, Y. Wu, W. Liang, Y. Cao, and M. Li, “High resolution sar image classification using global-local network structure based on vision transformer and cnn,” IEEE Geoscience and Remote Sensing Letters, 2022\.
  * [145] J. Cai, Y. Zhang, J. Guo, X. Zhao, J. Lv, and Y. Hu, “St-pn: A spatial transformed prototypical network for few-shot sar image classification,” Remote Sensing, 2022.
  * [146] X. Ke, X. Zhang, and T. Zhang, “Gcbanet: A global context boundary-aware network for sar ship instance segmentation,” Remote Sensing, 2022.
  * [147] R. Xia, J. Chen, Z. Huang, H. Wan, B. Wu, L. Sun, B. Yao, H. Xiang, and M. Xing, “Crtranssar: A visual transformer based on contextual joint representation learning for sar ship detection,” Remote Sensing, 2022.
  * [148] L. Chen, R. Luo, J. Xing, Z. Li, Z. Yuan, and X. Cai, “Geospatial transformer is what you need for aircraft detection in sar imagery,” IEEE Transactions on Geoscience and Remote Sensing, 2022.
  * [149] P. Zhang, H. Xu, T. Tian, P. Gao, and J. Tian, “Sfre-net: Scattering feature relation enhancement network for aircraft detection in sar images,” Remote Sensing, 2022.
  * [150] C. Ma, Y. Zhang, J. Guo, Y. Hu, X. Geng, F. Li, B. Lei, and C. Ding, “End-to-end method with transformer for 3d detection of oil tank from single sar image,” IEEE Transactions on Geoscience and Remote Sensing, 2021.
  * [151] L. Norikane, B. Broek, and A. Freeman, “Application of modified vicar/ibis gis to analysis of july 1991 flevoland airsar data,” in AIRSAR Workshop, 1992\.
  * [152] https://www.dlr.de/hr/en/desktopdefault.aspx/tabid-2326/3776_read-5679/, “Accessed: 2022-08-27,”
  * [153] https://ietr-lab.univ-rennes1.fr/polsarpro-bio/san-francisco/dataset/SAN_FRANCISCO_AIRSAR.zip, “Accessed: 2022-08-27,”
  * [154] https://www.eorc.jaxa.jp/ALOS/en/alos-2/a2_data_e.htm, “Accessed: 2022-08-27,”
  * [155] https://directory.eoportal.org/web/eoportal/satellite-missions/g/gaofen-3, “Accessed: 2022-08-27,”
  * [156] https://www.dlr.de/hr/en/desktopdefault.aspx/tabid-2326/3776_read-5691/, “Accessed: 2022-08-27,”
  * [157] https://www.sdms.afrl.af.mil/index.php?collection=mstar, “Accessed: 2022-08-27,”
  * [158] J. Li, C. Qu, and J. Shao, “Ship detection in sar images based on an improved faster r-cnn,” in BIGSARDATA, 2017.
  * [159] S. Wei, X. Zeng, Q. Qu, M. Wang, H. Su, and J. Shi, “Hrsid: A high-resolution sar images dataset for ship detection and instance segmentation,” IEEE Access, 2020.
  * [160] https://earth.esa.int/eogateway/catalog/cryosat-products, “Accessed: 2022-08-27,”
  * [161] M. Perera, W. Bandara, J. Valanarasu, and V. Patel, “Transformer-based sar image despeckling,” arXiv preprint arXiv:2201.09355, 2022.
  * [162] D. Martin, C. Fowlkes, D. Tal, and J. Malik, “A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics,” in ICCV 2001, 2001.
  * [163] H. Dong, W. Ma, L. Jiao, F. Liu, R. Shang, Y. Li, and J. Bai, “A contrastive learning transformer for change detection in high-resolution sar images,” SSRN:ssrn.4169439, 2022.
  * [164] https://earth.esa.int/eogateway/catalog/terrasar-x-esa-archive, “Accessed: 2022-08-27,”
  * [165] Y. Fan, F. Wang, and H. Wang, “A transformer-based coarse-to-fine wide-swath sar image registration method under weak texture conditions,” Remote Sensing, 2022.
  * [166] Z. Li and N. Snavely, “Megadepth: Learning single-view depth prediction from internet photos,” in CVPR, 2018.
  * [167] X. Dong, J. Bao, D. Chen, W. Zhang, N. Yu, L. Yuan, D. Chen, and B. Guo, “Cswin transformer: A general vision transformer backbone with cross-shaped windows,” in CVPR, 2022.
  * [168] S. Mehta and M. Rastegari, “Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer,” in ICLR, 2022.
  * [169] Yanghao, C.-Y. Wu, H. Fan, K. Mangalam, B. Xiong, J. Malik, and C. Feichtenhofer, “Mvitv2: Improved multiscale vision transformers for classification and detection,” in CVPR, 2022.