# Revisiting Crowd Counting: State-of-the-art, Trends, and Future Perspectives

Muhammad Asif Khan, Hamid Menouar, and Ridha Hamila M. A. Khan and H. Menouar
are with Qatar Mobility Innovations Center (QMIC), Doha, Qatar.R. Hamila is
with Qatar University.

###### Abstract

Crowd counting is an effective tool for situational awareness in public
places. Automated crowd counting using images and videos is an interesting yet
challenging problem that has gained significant attention in computer vision.
Over the past few years, various deep learning methods have been developed to
achieve state-of-the-art performance. The methods evolved over time vary in
many aspects such as model architecture, input pipeline, learning paradigm,
computational complexity, and accuracy gains etc. In this paper, we present a
systematic and comprehensive review of the most significant contributions in
the area of crowd counting. Although few surveys exist on the topic, our
survey is most up-to date and different in several aspects. First, it provides
a more meaningful categorization of the most significant contributions by
model architectures, learning methods (i.e., loss functions), and evaluation
methods (i.e., evaluation metrics). We chose prominent and distinct works and
excluded similar works. We also sort the well-known crowd counting models by
their performance over benchmark datasets. We believe that this survey can be
a good resource for novice researchers to understand the progressive
developments and contributions over time and the current state-of-the-art.

###### Index Terms:

Crowd counting, CNN, density estimation, evaluation metrics, loss functions,
transformers

##  I Introduction

Crowd counting is an interesting research area with many real-world
applications. The increasing population and growing urbanization trends often
result in rapid crowds creation in urban areas such as metro stations, sports
venues, musical concerts, exhibition centers, and parade grounds etc. The
prediction of the crowds creation, and the density estimation in crowded
regions is of paramount significance in crowd monitoring, management, and
effective events planning. The social distancing measures to reduce the spread
of virus observed during the recent COVID-19 pandemic further highlight its
significance [1, 2]. Owing to the importance of the problem, a huge amount of
research exists on automated crowd counting using image and video analysis
methods. Although traditional image processing methods have shown limited
performance, the last decade has witnessed major improvements using state-of-
the-art methods in computer vision and deep learning.

Crowd counting is generally implemented in two ways: (i) counting objects
(input is an image and the output is a number i.e., total head count in the
image.), and (ii) density map estimation (input is an image and the output is
the density map of the crowd which is then integrated to get the total head
count.).

Traditional methods for crowd counting were all based on total count approach.
These methods employ image processing techniques to detect hand-crafted
features e.g., body appearance [3, 4, 5], or body parts [6, 7, 8, 9, 10, 11]
and then use machine learning models such as linear regression, ridge
regression, Gaussian process, support vector machines (SVMs), random forest,
gradient boost, and neural networks to find the total head count in an image.
However, all the accuracy of these methods significantly degrades on images
with dense crowds due to challenges such as occlusions, low resolution,
foreshortening and perspectives.

Due to the aforementioned limitations of detection-based approach, regression-
based methods were adopted which regress the total count from an image or
image patch. Instead of detecting body parts or shapes, regression-based
methods use global features such as texture [12], foreground [13], gradients
[14, 15] etc. to estimate image-wise or patch-wise total count [16, 17].
Regression-based method solve the problems raised from occlusions, low
resolution, and perspective. However, regression-based method show poor
performance on high density crowd images.

Recent research on crowd counting shows the efficacy of convolution neural
networks (CNNs) [18, 19, 20, 21, 22, 23] due to their strong capability of
automatic feature extraction. Like many other computer vision problems, such
as image classification [24, 25], object detection [26], and image
segmentation [27, 28], CNN is used as a widely applied method for crowd
counting and it outperforms all the traditional methods. While traditional
methods for crowd counting predict the total head count in an image, CNN-based
methods often employ crowd density estimation. Density estimation is when the
CNN model is used to predict the density map of the crowd scene rather than
just the head count. The density map contains additional location information
along with the total head count of the crowd scene.

First introduced in [29], density estimation using CNNs has been adopted in
all subsequent research contributions. However, there have been major
architectural improvements to achieve optimum performance. As the performance
of any deep learning method typically is evaluated using benchmark datasets,
many crowd datasets were also published over time. These datasets published
over time introduced more challenges in terms of high density, scale
variation, scene variation, and illumination changes, crowd distribution,
extreme occlusions, perspective distortions etc. To overcome these challenges
and to achieve high accuracy over these datasets, complex CNN architectures,
novel learning methods, and sophisticated evaluation criteria were developed
over time. This survey covers all these developments so that a novice
researcher in the area of crowd counting can understand these advancements in
an incremental manner. Although there exists few relevant surveys [30, 31, 32,
33, 34, 35, 36, 37, 38, 39, 40, 41, 42], we believe our work is different in
several aspects:

  * •

There has been several significant contributions over the last 1-3 years which
are not covered in the old surveys [30, 31, 32, 33, 34, 35, 36, 37, 38, 39,
40]. These new contributions span multiple directions including novel and
improved loss functions, more accurate evaluation metrics, and novel network
architectures (e.g., transformer-based models). Thus, this survey is a more up
to date reference on the topic.

  * •

We provide a highly intuitive review of the state-of-the-art by more
meaningful segregation of the literature so that each contribution is
discussed separately. For instance, the segregation of works by network
architecture, loss functions, training methods etc. and then sorting by
performance is of more interest to the novice researchers as well as general
readers.

  * •

Unlike other surveys, we intend to focus our discussion on the most
significant contributions based on their impact and the relative performance
gains over previous works. By excluding redundant and least significant
contributions (in terms of novelty or performance), our survey is concise,
more focused and more readable to follow the actual advancement of the state-
of-the-art rather than repeatedly discussing similar ideas of least impact.

The rest of the paper is organized as follows: We first discuss (in Section
II) the various challenges of crowd counting with a visual illustration. Then
a brief overview of crowd datasets used in benchmarking of previous research
works is provided in Section III. Section IV explains the data annotation
techniques applied in crowd counting and density estimation. Section V
presents a comprehensive review, critical analysis, and performance comparison
of crowd counting models. Section VI presents an indepth analysis of training
and evaluation methods for crowd counting covering all the loss functions and
evaluation metrics used in the literature. Lastly, we concluded the paper in
Section VII with some interesting insights for prospective researchers working
in this area.

##  II Crowd Counting Challenges

A robust crowd counting model aims to accurately predict crowd count and
estimate local as well as global density in crowd scenes under diverse
conditions. The varying conditions heavily impact the performance of the
counting model. Hence, it is therefore important to first understand these
challenges and their possible impact on the model performance. A good
understanding of these challenges helps in developing more robust models. Some
of the more common challenges in crowd counting are listed here with a visual
illustration in Fig. 1.

(a) Single Scene dataset.

(b) Multiple scenes dataset.

(c) Scale variation in same image.

(d) Scale variation in different images of the same scene.

(e) Variations in crowd density (low vs high density).

(f) Uniform vs non-uniform crowd distribution.

(g) Occlusions in crowd images.

(h) Complex backgrounds.

Figure 1: Challenges in crowd counting using computer vision.

  1. 1.

Scene variation: Crowd monitoring across different scenes is more challenging.
For instance, counting people in images taken by a single CCTV camera (e.g.,
in Mall dataset [43]) is much easier than counting people in images taken by
several surveillance cameras (e.g., in WorldExpo’10 dataset [29]). Scene
variations is common in drone surveillance and becomes more challenging when
combined with other variations listed afterwards.

  2. 2.

Scale Variation: Scale variation refers to the situation when objects related
to the same class (i.e., humans) appear in different sizes in a single image
as well as in different images. Scale variation is caused by distance (between
camera and objects) and perspective effect in the same image. Moreover, scale
variations are also observed in images of different resolutions. Scale
variation is one of the most common challenges that hugely impact the model
performance and is frequently addressed in crowd counting research.

  3. 3.

Crowd Density: The number of people (or other objects of interests) vary from
one image to another. Usually, images with low density are easy than highly
dense images. Another challenge is when the same image contains different
crowd densities in different regions.

  4. 4.

Non-uniform People Distribution: Crowd images may differ in terms of
distribution of objects (e.g., people). For instance, people sitting in a
sports stadium are uniformly distributed with constant distance among objects.
In contrast, objects may be distributed randomly in street crowds. Uniformly
distributed crowds can be estimated more accurately than non-uniform crowds in
the absence of other attributes affecting the accuracy.

  5. 5.

Occlusions: Occlusion refers to the overlap in objects. When similar objects
(e.g., people) overlap, it is called intra-class occlusion, whereas when
different objects (e.g., people, cars, walls) overlap, it is called inter-
class occlusion. Dealing with occlusion is often challenging. It is not only
difficult for annotators to annotate the objects in the presence of occlusion,
but also challenging for the object detectors to accurately predict objects.
Occlusion interweave semantic features in images making it difficult to
discriminate object boundaries.

  6. 6.

Complex background: The background region which do not contain objects of
interest vary in the same images as well as across different images. When the
pixel values in the background have similar values as that of the object, it
can make learning hard.

  7. 7.

Illumination variations: The illumination in an image vary at different times
of a day and may vary in different regions in the same images due to lighting
conditions. Thus the same object (e.g., people) in the same image will have
different pixel values, which makes learning challenging.

  8. 8.

Other variations: The aforementioned attributes in crowd images is not an
exhaustive list. There are several other attributes that make counting a more
challenging problem. Few examples are weather changes, image distortion and
noise, object rotation etc.

##  III Crowd Counting Datasets

A number of benchmarking datasets have been introduced over time that are
being used for evaluating crowd counting models. These datasets vary in size
(number of samples), annotations, and image attributes. Table I provides a
brief summary of these datasets.

TABLE I: Crowd counting datasets. Dataset | Attribute | Scene | Images | Resolution
---|---|---|---|---
UCSD [44] | CCTV | Single | 2000 | $238\times 158$
Mall [43] | CCTV | Single | 2000 | $640\times 480$
UCF_CC_50 [45] | Free | Single | 50 | Vary
WorldExpo’10 [29] | Video frames | Cross-scene | 3920 | $576\times 720$
ShanghaiTech-A [18] | Internet | Single | 482 | Vary
ShanghaiTech-B [18] | CCTV | Single | 716 | $768\times 1024$
UCF-QNRF [46] | Free-view | Single | 1535 | Vary
JHU-Crowd [47] | Internet | Cross-scene | 4372 | $1450\times 900$
DroneRGBT [48] | Drone | Cross-scene | 3600 | $512\times 640$
NWPU Crowd [49] | Internet | Cross-scene | 5109 | Vary
TRANCOS [50] | CCTV | Single | 1244 | $640\times 480$
CARPK [51] | Drone | Cross-scene | 1448 | $1280\times 720$
VisDrone-People [52] | Drone | Cross-scene | 3347 | Vary
VisDrone-Vehicles [52] | Drone | Cross-scene | 5303 | Vary
RGBT-CC [53] | Drone | Cross-scene | - | -

##  IV Data Labelling

Crowd monitoring applications mostly use supervised learning or semi-
supervised learning and thus require labelled data (images or videos). There
are mainly three types of data annotations used for crowd images.

###  IV-A Types of Annotation

  1. 1.

Total count annotation: The total number of objects in the image is counted as
a single number. This kind of annotation is used in detection-based and
regression-based counting approaches.

  2. 2.

Dot annotation: The objects inside images are annotated as a single dot over
each object, typically on the person’s head. The dot annotation creates a dot-
map (also called as localization map) i.e., a matrix consisting of all zeros
except those positions containing heads in the image. All pixels corresponding
to head positions are activated i.e., set to 1 per head. Dot annotation is
used in density estimation methods for crowd counting. However, as the sum of
dot map is equal to the total count in an image, it can also be used in count
regression models.

  3. 3.

Bounding box: A rectangular bounding box is drawn around each object (e.g.,
person) in the image. Total count annotation is cheaper; however, it does not
provide a good accuracy due to issues such as object occlusions. Dot
annotation can outperform the total count, but it does not capture information
such as object scaling, and it provide limited accuracy when the objects scale
largely vary in images.

###  IV-B Density-map generation

Dot annotations is the most common and a relatively cheaper (than bounding
box) annotation technique. However, the dot map or localization map which
contains all zeros except head positions, are extremely sparse and it is
generally very hard to train neural network over them [46]. Thus, in crowd
counting research, such localization maps are converted to density maps. If
$x_{i}$ is a pixel containing the the head position, it can be represented by
a delta function $\delta(x-x_{i})$. The density map is generated by convolving
the delta function with a Gaussian kernel $G_{\sigma}$.

$Y=\sum_{i=1}^{N}{\delta(x-x_{i})*G_{\sigma}}$ (1)

where, N is the total number of annotated points (i.e., total count of heads)
in the image. The integral of density map $Y$ is equal to the total head count
in the image.

Figure 2: Ground Truth generation for crowd counting models.

Visually, this operation creates a blurring of each head annotation using the
scale parameter $\sigma$. There are various kernel settings to generate a
variety of density maps.

The most basic approach is to keep $\sigma$ fixed value, which means that the
density map will apply same kernel to all head positions irrespective of their
scale in the image. Typical values of $\sigma$ used in earlier works are 15
[20], X [29].

TABLE II: Gaussian Kernels used in various studies. Model |  Dataset | Kernel |  Scale ($\sigma$)
---|---|---|---
CrowdCNN [29] |  - | Adaptive | -
MCNN [18] |  - | Adaptive | -
MSCNN [54] |  - | Adaptive | -
CSRNet [19] |  ShanghaiTech Part B | Fixed | 15
TRANCOS | Fixed | 10
UCSD, WorldExpo’10 | Fixed | 3
SANet [20] |  - | Adaptive | -
TEDnet [55] |  - | Fixed | -
MobileCount [56] |  ShanghaiTech Part B, UCF-QNRF, UCF-CC-50, WorldExpo’10 | Fixed | 4
Deepcount [57] |  - | Fixed | 5
BL [58] |  ShanghaiTechB | Fixed | 15
C-CNN [59] |  ShanghaiTech Part A & B | Fixed | 15
ASNet [22] |  - | Fixed | -

Fixed Gaussian kernels are not helpful to learn the scale variations in
images. Hence, adaptive kernels are proposed. In adaptive kernels, the value
of $\sigma$ can be calculated as the average distance to k-nearest neighboring
head annotations. Visually, it generates lower degree of Gaussian blur for
dense crowds and higher degree for region of sparse density in crowd scene.
Typical settings includes $k=10$ [20], $k=10$ [54].

Figure 3: Density maps with different scale ($\sigma$) values.

### Summary and Lessons Learned

  * •

Fixed Gaussian kernels are adopted in earlier models because these are
computationally cheaper than adaptive kernels. The value of $\sigma$ is often
pre-determined empirically and can vary across different datasets.
Interestingly, different values of $\sigma$ have been used for the same
dataset in different works which potentially indicate the value may be fine-
tuned for the CNN model to get better results.

  * •

Adaptive Gaussian kernels are increasingly used to cope with the scale
variations in images. Earlier models used pre-determined values calculated as
the weighted inter-head distances using K-nearest neighbors. The value of $k$
depends upon the crowd density i.e., higher value of $k$ for dense images and
vice versa.

  * •

Some recent works propose to learn adaptive kernels alongside model training
(explained in Section VI).

##  V Crowd Counting Models

A wide numbers of DNN models are proposed for crowd counting over the period.
These generally vary in size as well as design structure. We studied these
models and categorized them into three distinct design categories i.e., (i)
single-column, (ii) multi-column, and (iii) hybrid. While one may argue that
such design patterns are common in other CNN models, it is important to
understand the structure of these models in the context of crowd counting.

###  V-A Single-column Models

One of the early models for crowd counting using CNN is reported in CrowdCNN
[29]. The CrowdCNN model consists of three convolution (Conv) layers (with
kernels of $7\times 7$, $7\times 7$, and $5\times 5$ respectively) followed by
three fully connected (FC) layers. The input to the model is $72$ patches
cropped from image, whereas the output is a density map of $18\times 18$ size.
The model is evaluated over UCSD and UCF_CC_50 datasets using mean absolute
error (MAE) as evaluation metric. Early CNN models for crowd counting require
perspective information during training as well as for inference, which may
not be available in practical scenarios. In the absence of perspective maps,
there may be significant performance degradation. Hence, very deep single-
column models (e.g., FusionCount [60]), multi-column models, or single-column
models with transfer learning are proposed over time. In [61], authors propose
a single-column CNN (SaCNN) architecture with seven (7) Conv layers all using
a single filter size ($3\times 3$). The model uses explicitly $1\times 1$
convolutions before $1\times 1$ for dimension reduction and does not employ
any pre-trained front-end CNN. The SaCNN performs multi-task learning i.e.,
simultaneously predict density map and head count. In [58], authors used a
two-layer with kernel size of $3x3$ using VGG-19 as backbone. In compact-CNN
(C-CNN) [59], authors used three parallel filters of different size producing
different outputs, which are merged and fed to five-layer single column CNN
network. Other single-column architectures include LibraNet [62].

###  V-B Multi-column Models

Single-column models usually are not good at detecting scale variations in
object sizes. Multi-column approaches are favoured to learn multi-scale and
perspective-free feature detection capability. The first multi-column CNN
(MCNN) is proposed in [18]. MCNN is a three-layers CNN architecture with
difference receptive fields in each column. Each column in MCNN produces a
density map of the same shape as ground truth map. The individual column’s
outputs are then concatenated to produce the final density map. MCNN takes an
image of arbitrary size. Several works followed similar design pattern. For
instance, the CrowdNet [63] model proposes a two-column CNN architecture. The
first column is five-layer deep network whereas the second column is a three-
layer shallow network. Similar to MCNN, the outputs of both columns are
concatenated to predict the final density map. One of the drawback of multi-
column models is the computation overhead. To dead with this, the Switch CNN
(SCNN) [64] model propose a 3-column network with different sized kernels in
each column. However, an additional CNN network called classifier is used to
automatically select only one column at inference time based on the crowd
density.

A relatively simpler two-column network called cascaded multi-task learning
(CMTL) is proposed in [65] which jointly learns the density map as well as
count density in images. In the two-column architecture, the first column
predicts a high-level prior (i.e., the total head count) whereas the second
column estimates the density map. Authors in [66] propose a detection and
density estimation network (DecideNet) to jointly estimate count and density
map. It consists of three columns however, unlike MCNN [18], each column in
DecideNet performs a different task. The first column RegNet is a 5-layer CNN
which predicts the density map in the absence of target density map. The
second column DetNet uses Faster R-CNN network [67] which first predicts the
bounding box and then generates a detection-based density map. The third
column QualityNet uses the two density maps generated by RegNet and DetNet,
and predicts the final density map. In [68], authors propose a four-column CNN
network with only the first four layers of VGG-16 as backbone. The first three
columns generates density maps. Due to different size of filters in the three
columns, when their outputs (density maps) are combined, it can adapt to the
scale variation in person size. The fourth layer generates two perspective map
which are used by the first two layers as intermediate inputs. The DeepCount
[57] architecture uses a large number of Conv layers and additionally
introduces the intermediate fusion of layers at different stages. A different
architecture is proposed in ASNet [22] which starts as a single-column (first
seven layers), and then branch out into two columns. The first column produces
density map, whereas the second column generates the scaling factor to be used
in density estimation.

Recently, multi-column models are increasingly used for multi-task learning.
In PCCNet [69], authors use a three column network to jointly learn the
density map, density class, and segmentation map in crowd images. The three
columns share encoded features at various stages. Other multi-column
architectures include MATT [70], [71], TAFNet [72].

###  V-C Encoder-Decoder Models

Dense CNN networks (e.g., VGG [24], Inception [25], ResNets [73]) designed for
classification encode low-level spatial features into high-level semantic
information. The down sampling operations used in these models causes loss of
pixel-level localization information while encoding features. To generate the
density map, the decoder layers aggregates the encoded feature maps to produce
the final density map. TEDnet uses the encoder-decoder architecture with
hierarchical feature aggregation at different encoding stages. Both the
encoder and decoder blocks uses multi-path encoding and decoding modules. The
encoder block in TEDnet [55] uses the same multi-scale module as in Fig 5
which are originally proposed in [20]. Nine (9) multi-scale modules are
cascaded with $2\times 2$ pooling after the first two modules only. In the
decoder block, multi-path decoding modules are used to aggregate the features
and restore the spatial resolution in the predicted density maps. The network
loss in TEDnet is computed at three intermediate levels in the decoder block,
and one at the final output of the model. A lightweight encoder-decoder
architecture is MobileCount [56] designed for computational efficiency. The
encoder block consists of first four layers of MobileNet_V2 [74] and decoder
block based on RefineNet [75]. In [76], authors propose a compact encoder-
decoder model called attentional neural fields (ANF). ANF uses conditional
random fields (CRFs) to aggregate multi-scale features inside encoder-decoder
model. Both the encoder and decoder blocks have six (6) Conv layers with both
inter-layer and intra-layer attention mechanism to capture dependencies among
features of same scale and across different scales. Other examples include
SASNet [77], MFCC [78].

###  V-D Single-column Models with Multi-column modules

This type of network architecture has been used in many works mainly to cope
with the shortcomings of both single-column and multi-column modules. Single
column models are less complex but perform poorly in detecting multi-scale
features wheres multi-column can detect multi-scale features but their
capability is limited by the number of columns. The fact that adding more
columns in multi-column models will significantly increase the size of the
model, several works introduce special multi-path or multi-column modules
cascaded into a single-column network. The first crowd counting model using
this type of architecture was multi-scale CNN (MSCNN) [54]. MSCNN is a single-
column CNN network that contains three multi-scale blobs (MSBs). Inspired from
the Inception model [25], the MSB is a naive Inception module (Fig. 4) with
four different sizes of Conv filters (i.e., $3\times 3$, $5\times 5$, $7\times
7$, $9\times 9$).

Figure 4: Inception [25] like multi-scale blob (MSB) used in MSCNN [54].

A slightly different multi-scale module called as scale-aggregation module
(Fig. 5) is proposed in [20] using $1\times 1$ convolution layers. Although
not explicitly stated by authors, the scale-aggregate module is also an
Inception-like module using $1\times 1$ convolution. The $1\times 1$
convolution layer when used before Conv layers of larger receptive field such
as $5\times 5$ and $7\times 7$ apply dimension reduction and thus reduce
computations.

Figure 5: Inception [25] like scale aggregation modules proposed in SANet [20]

Other examples include M-SegNet [79], SGANet [23].

###  V-E Models with Transfer Learning

Small crowd counting models typically suffer from accuracy degradation when
used on high density scenes. To improve the accuracy of crowd models on highly
dense scenes, transfer learning is becoming a promising approach. Although
earlier models such as CrowdNet [63] has been using this approach, its
performance was limited. The first model that showed significant boost in
performance using transfer-learning is congested scene recognition (CSRNet)
[19]. CSRNet uses the first 10 layers of VGG-16 model [24] pre-trained on
ImageNet dataset [80] as front-end CNN. The back-end CNN used dilated-
convolution instead of pooling operation making CSRNet an all-convolution
network with larger receptive fields which is easier to train. Transfer-
learning is typically used in any type of architecture i.e., single-column,
multi-column, encoder-decoder etc.

###  V-F Other CNN Models

Although, most crowd counting models inspires from other works and follow
similar design patterns, there are few works that do not fit into either of
the aforementioned categories. In other word, there is no particular design
pattern followed and the network is formed by connecting various layers in
different ways (i.e., cascaded, parallel, feedback paths, skip connections
etc.). One such model is context-aware network (CANNet) [81] which comprised
of a VGG16 front-end (10 layers), proceeded by a multi-path network
(consisting of convolution and up-sampling layers), which concatenates to
connect to a single-column decoder network. Similarly, the perspective-guided
convolution network (PGCNet) in [21] uses a two column architecture. The first
column is a density map prediction network (DMPNet), whereas the second column
is the perspective estimation network (PENet). The DMPNet uses CSRNet [19] as
a backbone, and then use multiple perspective-guided convolution (PGC) modules
cascaded in a single column. The PENet on the other hand, uses an encoder-
decoder CNN network to estimate the perspective map of the input image.

In [82], authors propose multi-modal counting network that accepts both RGB
images and infrared images of the same scene as inputs to predict the density
maps. The MMCNN architecture is also a complex network with inter-connections
among various layers at different stages. MMCNN also uses ResNet [73] based
multi-scale modules.

###  V-G Transformer-based Models

Transfomer models proposed in [83] have recently emerged as a new paradigm of
neural networks with self-attention mechanism. Due to their remarkable
performance in several applications such as natural language processing and
speech processing . These profound performance of transfomers in other domains
has sparked the computer vision community to apply transfomers in various
vision applications. Vision transformer (ViT) [84] is the first transformer
model applied in computer vision for image classification. Since then,
transformers have been used in various vision applications including image
recognition [85], object detection [86, 87], and image segmentation [88].

The original transformer [83] uses an encoder-decoder structure with self-
attention mechanism. The encoder transform an input sequence (words in NLP)
into a high dimensional feature vector whereas the decoder then translate it
into the output sequence. When applied in vision tasks [84], the input image
is split into patches and then the patches are provided to the transformer’s
encoder as linear embeddings (to mimic sequences). Recently, transformers have
been applied in crowd counting tasks [89, 90, 91, 92, 93, 94, 95, 96]. For
example, the TransCrowd model [96] uses pure transformer model in crowd
counting task. It convert each image of size (1152x768) into fixed size six
(6) patches of size (384x384) and convert patches to sequences of pixel values
in the patch (flattening). The sequences are fed to the encoder of the ViT
transformer (pretrained on ImageNet) [84]. The output is then fed to a
regression head which predict the total count in the image. Unlink TransCrowd
[96] which predicts the total count, authors in [91] propose CCTrans i.e., a
transformer model to predict density maps. CCTrans uses similar input pipeline
as TransCrowd but uses different model architecture. It uses Twins [97] as
backbone as feature encoder. The 1D output is transformed into feature maps
(2D vectors), upsampled to $\frac{1}{8}$ of the input patch size. The feature
maps are then fed to regression heads to predict density maps.

Some recent works also propose alternate techniques for crowd counting such as
deep q-learning (DQN) [98, 99]. However, such contributions have limited
impact in terms of performance improvement.

(a) Single-column models depict convolution layers cascaded in a single
column. (b) Multi-column models consist of two or more CNN layers where the
output of all columns are fused together to produce a single output.

(c) Encoder-decoder models comprised of two parts, where the first part
(encoder module) of the model works as a feature extractor whereas the second
part (decoder module) of the network decodes the features into the required
output shape. (d) Models which consist of a relatively smaller CNN network
with a pre-trained backbone or front-end network (large sized).

(e) A single column model with multiple modules which are usually multi-
column, specifically designed to encode scale-aware feature in crowd images.
(f) Transfomer Models.

Figure 6: Crowd counting models: Common CNN design patterns followed in crowd
counting and density estimation models.

### Summary and Lessons Learned

  * •

Single-column models are compact but achieves lower accuracy over dense
images. They also suffer from scale variations in images.

  * •

Scale variations can be addressed using multi-column models. However, the
capability of a multi-column model to adopt to various object scales is
limited by the number of columns. Multi-scale models are also computationally
expensive due to several columns training in parallel.

  * •

To address scale variations in a computationally efficient way and to adopt to
large scale variations, single column models with multi-scale modules are
proposed. These works are originally inspired from Inception [25] model with
small differences in design.

  * •

Encoder-decoder models preserve spatial resolution when high quality density
maps are desired. They also offer easy implementation of supervision at
different stages of the network (e.g., TEDnet [55]).

  * •

Both single-column and multi-column models trained from scratch have limited
accuracy when tested on large datasets with highly dense images. Hence, pre-
trained deep models such as VGG [24], Inception [25], and ResNets [73] have
been used to improve counting accuracy. Models with pre-trained backbone CNNs
(front-ends) also train faster. However, the model size and execution time
increases which make such models not a good choice for real-time performance.

TABLE III: A list of crowd counting models arranged in chronological order.

Datasets Model Year Architecture Backbone Mall [43] UCSD [44] WorldExpo’10
[29] UCF-CC-50 [45] ShanghaiTech [18] UCF-QNRF [46] SmartCity [61] DroneRGBT
[61] CARPK [61] TRANCOS [50] VisDrone [61] RGBT-CC [53] JHU-Crowd [47] NWPU-
Crowd [49] CrowdCNN [29] 2015 SC No ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ MCNN [18] 2016
MC No ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ CrowdNet [63] 2016 MC VGG16 ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗ MSCNN [54] 2017 SC-MSM No ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ SCNN [64]
2017 MC No ✗ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ CMTL [65] 2017 MC No ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ CSRNet [19] 2018 SC VGG16 ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ SANet [20]
2018 ED/SC-MSM No ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ SaCNN [61] 2018 SC No ✗ ✗ ✓ ✓ ✓
✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ DecideNet [66] 2018 MC No ✗ ✗ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ TEDnet
[55] 2019 ED No ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ CANNet [81] 2019 Complex VGG16 ✗ ✗
✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ GSP [100] 2019 SC VGG16 ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
MobileCount [56] 2019 ED MobileNetV2 ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ANF [76] 2019
ED No ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ BL [58] 2019 SC VGG19 ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ PACNN [68] 2019 MC VGG16 ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ PGCNet [21] 2019
Complex CSRNet ✗ ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ C-CNN [59] 2020 SC VGG ✗ ✗ ✓ ✗ ✓ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ DeepCount [57] 2020 MC CSRNet ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
ASNet [22] 2020 MC VGG16 ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ MMCNN [82] 2020 Complex
ResNet-50 ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ LibraNet [62] 2020 SC VGG16 ✗ ✗ ✗ ✓ ✓ ✓
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ MATT [70] 2020 MC CSRNet ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ Yang et
al. [71] 2020 MC VGG16 ✗ ✓ ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ PCCNet [69] 2020 MC No ✗ ✗
✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ SASNet [77] 2021 ED VGG16 ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
M-SFANet [79] 2021 SC-MSM VGG16 ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ M-SegNet [79] 2021
SC-MSM VGG16 ✗ ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ SGANet [23] 2022 SC-MSM Inception-v3
✗ ✗ ✗ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ FusionCount [60] 2022 SC VGG16 ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✗ ✗ MAN [101] 2022 MC VGG19 ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ MFCC [78] 2022 ED
ResNet ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ TAFNet [72] 2022 MC VGG16 ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
✗ ✗ ✓ ✗ ✗ TransCrowd [96] 2022 MC Non-CNN ✗ ✓ ✓ ✓ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓

##  VI Network Training and Evaluations

The standard process for supervised learning using neural network comprised of
two stages: network training, and evaluation. In training stage, the network
takes images and ground truths (total head count or density maps), compute
gradients and back propagate the loss. The loss function is the objective
function that the network is minimizing. After each one or every few epochs,
the model is evaluated using a metric function. In the case of crowd counting,
both the loss and metric functions compare the predicted density map with the
ground truth (i.e., target) density map and compute the difference. The target
and predicted density maps can be generally compared in three possible ways;
(i) image-wise i.e., compare density maps as a whole, (ii) patch-wise i.e.,
divide the target and predicted maps into non-overlapping patches and compare
all patches that belong to the same location, and (iii) pixel-wise i.e.,
compare each individual pixels (at the same location) of the target and
predicted maps and integrate over the image.

Figure 7: Different levels of comparing predicted density maps with the target
density maps.

###  VI-A Supervised and Weakly-supervised Learning

The state-of-the-art methods for crowd density estimation use point-level
supervision (fully supervised). Although, fully supervised learning produces
more reliable and accurate results, they require expensive annotations.

A small number of research works [70, 102, 96, 103] also propose the use of
weakly supervised learning which use more economical total head count
annotations. The Multiple Auxiliary Tasks Training (MATT) model [70] uses dot
annotations (full supervision) for a small number of samples and count
annotations (weak supervision) for the remaining data. According to the [70],
the count-level supervision although may achieve low counting errors, it does
not encode precise object location. To overcome this, multiple auxiliary
branches were used to produce density maps. The losses for these auxiliary
branches are computed and integrated. The final loss function is the sum of
pixel-wise $l_{2}$ loss (primary branch), count loss (primary branch) and the
integrated auxiliary loss (auxiliary branches).

Recently transformer models are proposed to implement weakly supervised
learning due to their self-attention mechanism to automatically learn the
semantic information in crowd images. The TransCrowd model [96] is based on
the ViT [84] which uses only count level supervision to predict the count.

###  VI-B Data Augmentation

Data augmentation refers to the technique used to increase the amount of data
and introduce more diversity in the data by adding slightly modified copies of
original data. When the input data is images, different augmentation
techniques are applied to transform original images such as random flipping,
croping, resizing, random perspective, rotation, random noise, brightness,
contrast etc. However, the augmentation techniques should be applied carefully
depending on the problem. In crowd counting research, the most commonly
applied augmentation techniques include random cropping [29, 63, 18, 19, 66,
79, 69] and horizontal flipping [19, 66, 79, 69]. In general, cropping helps
to improve learning by addressing the scale variations whereas horizontal
flipping address the perspective distortion in images [63]. Some less-
frequently used transformations include random noise [66], vertical flipping
[66], and image rotation [82]. Image augmentation also acts as a regularize
for the network i.e., helps to avoid overfit the model to the training data
[82].

###  VI-C Loss functions

A loss function is a function that measure the difference between the ground
truth (GT) and the prediction. In density estimation, both the GT and
prediction are density maps, whereas in count regression, both are numbers
(integers).

####  VI-C1 Euclidean ($l_{2}$) loss:

Euclidean loss or $l_{2}$ loss is the most common loss function used in crowd
counting research. The pixel-wise $l_{2}$ loss or mean squared
error($L_{D}(\Theta)$) is calculated as follows:

$L_{D}(\Theta)=\frac{1}{N}\sum_{i=1}^{N}{||F(X_{i};\Theta)-D_{i}||^{2}}$ (2)

where $\Theta$ represents the CNN parameters (i.e., wights), $N$ is the total
number of samples, $F(X_{i};\Theta)$ is the predicted density map, and $D_{i}$
is the ground truth density map.

The $l_{2}$ loss can also be calculated over total counts instead of pixel-
wise values. If

$L_{C}(\Theta)=\frac{1}{N}{\sum_{i}^{N}{||\hat{C}_{i}-C_{i}||^{2}}}$ (3)

where $\hat{C}_{i}$ and $C_{i}$ are the predicted and true counts, which are
obtained by summing the pixel values over entire GT density map and predicted
density map respectively.

####  VI-C2 $l_{1}$ loss

This is the Manhattan distance ($l_{1}$ norm) among corresponding pixels in
the target and predicted density maps.

$L_{D}(\Theta)=\frac{1}{N}\sum_{i=1}^{N}{||F(X_{i};\Theta)-D_{i}||}$ (4)

Similar to the case of $l_{2}$ loss, $l_{1}$ loss can also be computed over
total count instead of individual pixels as follows:

$L_{C}(\Theta)=\frac{1}{N}{\sum_{i}^{N}{||\hat{C}_{i}-C_{i}||}}$ (5)

####  VI-C3 Composition Loss

Composition loss is proposed in [46]. Two density maps and one localization
map is generated at different depths in the CNN network, whereas the final
output is the head count in the image. For the density and localization maps,
the pixel-wise $l_{2}$ loss is computed, whereas the regression $l_{2}$ loss
between the predicted and actual counts is computed.

####  VI-C4 Spatial Correlation Loss

Spatial correlation loss (SCL) is the cross-correlation among the normalized
target and predicted density maps and is calculated as:

$L_{SC}=1-\frac{\sum_{i=1}^{N}\sum_{j=1}^{M}\left(\hat{D_{ij}}.D_{ij}\right)}{\sqrt{\sum_{i=1}^{N}{\hat{D}_{ij}^{2}}.\sum_{i=1}^{N}{D_{ij}^{2}}}}$
(6)

where $D_{ij}$ and $\hat{D_{ij}}$ are the target and predicted density maps,
respectively. $i$ and $j$ are the row and column indices, and $N$, and $M$
represents the total number of rows and columns in the target and predicted
density maps, respectively. SCL has been used in TEDnet [55] in combination
with $l_{2}$ loss. SCL is less-sensitive to changes in intensity levels in the
density map.

####  VI-C5 APLoss

The commonly used $l_{2}$ loss takes the target and predicted density maps as
a whole to compute the pixel-wise euclidean $l_{2}$ distance among them. The
standard $l_{2}$ loss does not consider the various density levels in the
image and lead to generalization errors. To cope with this shortcoming of
$l_{2}$ loss, adaptive pyramid loss (APLoss) is proposed in [22]. To compute
APLoss, the target density map is first divided into four non-overlapping
patches. If the count in each patch is greater than a threshold $T$, the patch
is further divided into four patches. This process is repeated until no patch
is left with count greater than the threshold $T$. When this patching process
is completed, the predicted density map is also divided into patches in the
same pattern (the no. of patches should equal for both target and predicted
density maps). Then the APLoss is calculated as follows:

$APLoss=\frac{1}{N}\sum_{i=1}^{N}\sum_{j_{1}=1}^{4}{l_{R_{j}1}^{i}}$ (7)

where $N$ is the number of density maps, and $l_{R_{j}1}^{i}$ represents the
patch-wise loss.

####  VI-C6 PRA Loss

The APLoss function introduced in [22] aim at reducing estimation errors in
the more dense regions of the image. The Pyramid Region Loss (PRA Loss) [77]
is a natural extension of the APLoss with two improvements. The intuition
behind PRA Loss is that in each crowd image, some regions and hence some
pixels in those regions are harder for the model to learn from. These regions
and pixels are typically overestimated which contribute to more counting
errors. If these harder regions and more specifically the harder pixels in
these regions are determined and assigned higher weights in the loss function,
the network would be able to learn these pixels more consistently. PRA Loss
solves this problem by extending the APLoss function with few improvements.
Similar to APLoss, the predicted density map is divided into four patches and
the patch with highest counting error (over-estimated patch) is selected. The
patching process is repeated until the pixels with the highest errors are
determined. PRA Loss is calculated as:

$l_{PRA}=||D^{est}_{p\in G}-D^{GT}_{p\in G}||^{2}+\lambda||D^{est}_{p\in
H}-D^{GT}_{p\in G}||^{2}$ (8)

where $p$ is the pixel in the predicted density map $D^{est}$, $G$ represents
set of all pixels in $D^{est}$, and $H$ is the set of all hard pixels in
$D^{est}$, and $\lambda$ is the weight term for the hard pixels.

####  VI-C7 Curriculum Loss

In machine learning, curriculum learning is a trained strategy for neural
networks proposed by [104], in which the model is fed with training samples in
the order of increasing complexity. Curriculum learning was used in crowd
counting by such that each image is assigned a difficulty score. Inspired by
the work in [105], a curriculum loss function is proposed in [23]. To design
the curriculum loss, the difficulty level of all pixels in the density map are
calculated. Following the intuition that dense crowd regions are harder to
estimate, the pixels with higher values than a dynamic threshold are
considered as difficult. To compute the difficulty level and the final loss
function, a weight matrix $W$ of the same size as the density map is defined.

Figure 8: An illustration of pixel values in density maps in different crowd
density regions. The localization map shows location of heads (i.e., one head
at the center, three heads at large distance in the top-left corner, and three
heads closer to each other at top-right corner). The density map shows the
corresponding pixel values i.e., higher values in the dense region (top-
right).

####  VI-C8 Combination Loss

Several works propose a combination of two or more loss functions to improve
learning. For instance, [29] proposes a combination of pixel-wise and count
$l_{2}$ loss as follow:

$L(\Theta)=L_{D}(\Theta)+\lambda L_{Y}(\Theta)$ (9)

where, $L_{D}(\Theta)$ is the pixel wise loss as calculated using Eq. 2, and
$\lambda L_{Y}(\Theta)$ is the $l_{2}$ loss over total count. The term
$\lambda$ is the weight term.

####  VI-C9 Optimal Transport (OT) Loss

The optimal transport (OT) loss is first introduced in [106]. To compute the
OT loss, both the target and predicted density maps ($D$ and $\hat{(}D)$) are
first converted into their respective probability density functions (pdfs) by
dividing each pdf by its total mass. The OT loss is computed as follow:

$l_{OT}=\mathcal{W}\left(\frac{D}{\|D\|_{1}},\frac{\hat{D}}{\|\hat{D}\|_{1}}\right)$
(10)

####  VI-C10 Total Variation (TV) Loss

Authors in [106] derived total variation (TV) loss due to the inherited time
complexity for computing OT loss. It is defined as:

$l_{TV}=\frac{1}{2}\left\|\frac{D}{\|D\|_{1}},\frac{\hat{D}}{\|\hat{D}\|_{1}}\right\|_{1}$
(11)

TABLE IV: Comparison of crowd counting models: model architecture, inputs and
outputs

Model Input Input size Output size loss Metric CrowdCNN [29] patch $72\times
72$ $18\times 18$ combination $l_{2}$ loss MSE, MAE MCNN [18] image arbitrary
input size/4 $l_{2}$ loss MSE, MAE CrowdNet [63] patch $225\times 225$ input
size/8 $l_{2}$ loss MSE, MAE MSCNN [54] patch $225\times 225$ $6\times 6$
$l_{2}$ loss MSE, MAE SCNN [64] patch image size/3 - $l_{2}$ loss MSE, MAE
CMTL [65] patch image size/4 input size $l_{2}$ loss, cross-entropy MSE, MAE
CSRNet [19] patch image size/8 input size/8 $l_{2}$ loss MSE, MAE, PSNR, SSIM
SANet [20] patch image size/4 input size composite ($l_{2}+l^{SSIM}$) loss
MSE, MAE SaCNN [61] patch image size/8 $l_{2}$ loss MSE, MAE DecideNet [66]
patch $4x3$ - composite ($l_{1}+l_{2}$) loss MSE, MAE TEDnet [55] image
arbitrary - SAL, SCL MSE, MAE, PSIM, SSIM CANNet [81] patch $224\times 224$ -
$l_{2}$ loss RMSE, MAE GSP [100] patch multiple - $l_{2}$ loss RMSE, MAE
MobileCount [56] patch (image size x 0.8) - $l_{2}$ loss MSE, MAE ANF [76]
image arbitrary - $l_{2}$ loss MSE, MAE BL [58] patches $256\times 256$ - BL
($l^{Bayes}$) MSE, MAE PACNN [68] patch image size/4 - composite
($l_{2}+l^{DSSIM}$) loss, MSE, MAE PGCNet [21] image arbitrary - $l_{2}$ loss
MSE, MAE CCNN [59] image arbitrary $l_{2}$ loss MSE, MAE DeepCount [57] patch
$384\times 512$ multiple weighted $l_{1}$ loss RMSE, MAE ASNet [22] patch
$128\times 128$ - APA loss MSE, MAE MMCNN [82] image - - composite loss RMSE,
MAE LibraNet [62] patch image size/2 - $l_{1}$ loss MSE, MAE MATT [70] image
arbitrary - composite ($l_{1}$ \+ $l_{2}$) loss MSE, MAE, RER Yang et al. [71]
- - - composite ($l_{2}$ \+ cross-entropy) loss MSE, MAE PCCNet [69] patches
$512\times 680$ - $l_{2}$ loss MSE, MAE SASNet [77] patch $128\times 128$ -
composite ($l_{2}+l^{SSIM}$) loss MSE, MAE M-SFANet [79] image arbitrary image
size/2 BL ($l^{Bayes}$) RMSE, MAE, GAME M-SegNet [79] image arbitrary image
size/2 BL ($l^{Bayes}$) RMSE, MAE, GAME SGANet [23] patch $128\times 128$
input size/4 curriculum loss RMSE, MAE FusionCount [60] patch $384\times 512$
- $l_{2}$ loss RMSE, MAE MAN [101] patch $512\times 512$ - composite loss MSE,
MAE MFCC [78] image $512\times 640$ input size $l_{2}$ loss RMSE, MAE TAFNet
[72] image arbitrary - $l_{2}$ loss RMSE, GAME TransCrowd [96] image
$1152\times 768$ - $l_{1}$ loss MSE, MAE

(a) ShanghaiTech Part B [18] and WorldExpo10 [29] datasets.

(b) UCF-CC-50 [45], ShanghaiTech Part A [18] and UCF-QNRF [46] datasets.

Figure 9: Performance analysis of crowd counting models over benchmark
datasets.

###  VI-D Evaluation Metrics

####  VI-D1 Mean Absolute Error (MAE)

It is the $l_{1}$ or Manhattan distance between the actual and predicted count
in an image and is calculated as follows:

$MAE=\frac{1}{N}\sum_{i=1}^{N}{(C^{pred}_{i}-\hat{C^{gt}_{i}})}$ (12)

MAE is the most commonly used metric in all crowd counting works. A limitation
of MAE is that it is more robust to outliers (i.e., large counting errors).
Thus mean squared error is used as an additional metric which is more
sensitive to outliers.

####  VI-D2 Mean Squared Error (MSE)

$MSE=\frac{1}{N}\sum_{i=1}^{N}{(C^{pred}_{i}-\hat{C^{gt}_{i}})}^{2}$ (13)

where, $N$ is the total number of images in the dataset, $C^{gt}_{i}$ is the
ground truth (actual count) and $C^{pred}_{i}$ is the prediction (estimated
count) in the $i^{th}$ image. MSE is the second most used metric after MAE in
crowd counting research.

####  VI-D3 Grid Average Mean Error (GAME)

While MAE is the most widely used metric in crowd counting research and is
often used to compare various models, MAE provide image-wide counting and does
not provide where the estimations have been done in the image. Owing to the
possible estimation errors in MAE, authors in [107] proposed Grid Average Mean
absolute Error (GAME). In GAME, an image is divided into $4^{L}$ non-
overlapping patches and compute MAE separately for each patch. Thus, GAME
poses a more robust and accurate estimation for crowd counting applications.

$GAME=\frac{1}{N}\sum_{i=1}^{N}{(\sum_{l=1}^{4^{L}}{|C^{pred}_{(n,l)}-C^{gt}_{(n,l)}|)}}$
(14)

Figure 10: An illustration of robustness of GAME metric to localization
errors. For a single prediction of a given ground truth, MAE is calculated as
zero (ignoring localization error in prediction), whereas GAME=1 (capturing
localization error).

The GAME metric is more robust to localisation errors in density estimation.
It can be easily observed in Fig. 10.

####  VI-D4 Patch Mean Absolute Error (PMAE)

$PMAE=\frac{1}{m\times N}\sum_{i=1}^{m\times
N}{|C^{pred}_{I_{i}}-C^{gt}_{I_{i}}|}$ (15)

where $m$ is the number of non-overlapping patches in each image. When $m=1$,
PMAE become equivalent to MAE. PMAE is effectively equivalent to GAME for
$m=4^{L}$.

####  VI-D5 Patch Mean Squared Error (PMSE)

$PMSE=\frac{1}{m\times N}\sum_{i=1}^{m\times
N}{(C^{pred}_{I_{i}}-C^{gt}_{I_{i}})^{2}}$ (16)

where $m$ is the number of non-overlapping patches in each image. When $m=1$,
PMSE become equivalent to MSE.

####  VI-D6 Mean Pixel Level Absolute Error (MPAE)

This is equivalent to pixel-wise $l_{1}$ error and is calculated using Eq. 4.

In addition to metrics for counting accuracy, some works [19, 55] used
additional metrics to measure the quality of predicted density maps. These are
listed as follows:

####  VI-D7 Peak Signal to Noise Ratio (PSNR)

PSNR measures the error between the corresponding pixels in the target and
predicted density maps. The value of PSNR is always in the range of [0-1].
Higher value of PSNR means lower error and vice versa. PSNR metric is used in
[19, 55].

####  VI-D8 Structural Similarity Index (SSIM)

SSIM measures the difference between the target and predicted density maps
over three attributes i.e., brightness, contrast, and structure. Similar to
PSNR, the value of SSIM is also in the range of [0-1] and the higher value is
preferred over lower values. SSIM metric is used in [19, 55].

A summary of different learning methods and evaluation criteria is presented
in Table IV. We further compared the performance of these models over the
commonly used metric i.e., MAE in Fig. 9 over benchmark datasets.

### Summary and Lessons Learned

  * •

The pixel-wise euclidean or $l_{2}$ loss (in Eq. 2) is the most commonly used
loss function used almost in all crowd counting models. It is typically used
alone or in combination with other loss functions to improve learning
performance and convergence.

  * •

The euclidean loss can not detect the localization errors. Hence, more complex
loss functions are proposed such as APLoss which attentively divide target
density maps into patches or sub-regions and compute loss loss for each patch
and finally sum these losses to calculate loss for the whole density map.

  * •

Almost all crowd counting models use $l_{2}$ distance among respective pixels
in the target and predicted density maps. There is an underlying assumption
that the individual pixel values are independent of their neighbors (non-
correlated). Recently, authors in [108] show that how human annotation errors
can cause noise in density maps generation and that such noise can be modeled
as a multi-variate Gaussian approximation.

  * •

Despite the various loss functions proposed over time, the pixel-wise $l_{2}$
loss is the widely used function to compute loss in crowd density estimation
either directly or integrated in other complex loss functions such as APLoss,
PRA Loss etc.

##  VII Conclusion and Future Research Directions

The paper presents a comprehensive review of the most significant
contributions in the area of crowd counting using computer vision and deep
learning. Based on our review of the literature, we mark few interesting
observations listed as follows. In addition, we present insights on each
observation for prospective readers.

  * •

State-of-the-art: in brief? Despite very deep models and complex
architectures, the accuracy gains over dense and large datasets is reasonably
low (e.g., ShanghaiTech Part A [18], UCF-QNRF [46], JHU-Crowd [47] etc.). A
large amount of research works contribute incrementally very low accuracy
gains over these datasets. In very dense crowd counting, there is still
sufficient room for accuracy gains which incites further investigation. Two
recent research directions in this area are improved loss functions and novel
architecture such as vision transformer-based models.

  * •

Deep or shallow models: how to choose? The attempts to gain higher accuracy
over large datasets generally lead to deeper and more complex architectures,
but on sparse datasets with low crowd density images (e.g., UCSD [44], Mall
[43], and ShanghaiTech Part B [18]), shallow models provide reasonably
sufficient accuracy and deeper models may not be required in scenarios
captured in these datasets. Noteably, single-scene crowd counting is the
simplest task, followed by sparse multi-scene crowd detection.

  * •

Need for benchmarking: Research efforts are mostly contributed towards
achieving accuracy gains (even if very small gains), and the resulting model
complexity is often ignored. This results in increasing model complexity for
small and often negligible improvement in accuracy. We believe that
benchmarking over model training and inference time of crowd models need to be
investigated.

  * •

Design patterns in crowd counting models: While the initial crowd models
followed a single column simple architecture, followed by other types such as
multi-column, encoder-decoder etc, the most recent architectures are
increasingly complex with no clear design patterns. Several works proposing
novel architecture also adopted improved training method (e.g., new loss
functions). The effective gain by the architecture and training method alone
were not thoroughly investigated. Furthermore, the increasing complexity of
crowd counting models makes it hard to understand the pattern which
contributed in the accuracy and can preclude innovation and improvements in
such models.

Based on our review and the aforementioned observations, we provide few useful
insights for prospective researchers.

  * •

Cross-scene counting and domain adaptation: Most of the crowd counting models
are trained and tested on the same benchmark datasets. In general the research
on model generalization is still very limited. Few studies investigate models
model generalization i.e., model pretrained on one dataset and then fine-tuned
on other dataset. However, the results typically reported belongs to the fine-
tuned model. This creates a huge gap on cross-scene model generalization of
crowd counting models which needs further investigation. It would be
interesting to see model generalization in a range of diverse scenarios e.g.,
CCTV images, drone images, indoor and outdoor scenarios. Furthermore, domain
adaptations is another exciting direction in crowd counting research.

  * •

Can we have a universal crowd counting model? The potential applications of
crowd counting will require the model to operate at various hardware platforms
(e.g., servers, drones, cameras, mobile phones etc.) with varying computing
capabilities. Also, the performance requirement vary by applications (e.g.,
real-time, non real-time), surveillance type (e.g., CCTV-based surveillance,
drone-based surveillance) and scenarios (e.g., shopping malls, metro stations,
stadiums etc.). Thus, developing a single best model with top accuracy in all
applications, using any surveillance method, and in scenarios is not an
effective solution. In fact, considering the existing trends in crowd counting
model design, such a model will have large size, require huge compute resource
for fine tuning, and incur longer delays during inference. Such a model will
not be suitable for applications with limited on-chip memory (in edge
devices), battery-powered devices, and real-time inference requirement. Thus
we envision and also witnessed some recent efforts to have application
specific model designing i.e., lightweight models for real-time applications
on resource-constrained devices, whereas dense models for optimum accuracy
over dense crowds in server-based platforms.

  * •

CNNs versus transformers? Although transformers have shown breakthrough
performance in applications like NLP and other sequence-to-sequence modelling,
their performance in computer vision is not comparably the same as in the
other domains. Specifically, in crowd counting tasks, transformers so far have
limited accuracy improvements over large benchmark datasets. Moreover,
transformer models for crowd counting available to date are relatively deep
models. This is because of the use of pretrained ViT which has 86M parameters
(base model with 12 layers) as compared to CNN-based models (e.g., CSRNet has
16M parameters). We believe transformer models yet have to bring substantial
accuracy gains and efficiency improvement to complement CNN models.

## Acknowledgement

This publication was made possible by the PDRA award PDRA7-0606-21012 from the
Qatar National Research Fund (a member of The Qatar Foundation). The
statements made herein are solely the responsibility of the authors.

## References

  * [1] M. S. A. Salma Kammoun Jarraya, Maha Hamdan Alotibi, “A deep-cnn crowd counting model for enforcing social distancing during covid19 pandemic: Application to saudi arabia’s public places,” Computers, Materials & Continua, vol. 66, no. 2, pp. 1315–1328, 2021.
  * [2] M. Al-Sa’d, S. Kiranyaz, I. Ahmad, C. Sundell, M. Vakkuri, and M. Gabbouj, “A social distance estimation and crowd monitoring system for surveillance cameras,” Sensors, vol. 22, no. 2, 2022.
  * [3] Z. Lin and L. S. Davis, “Shape-based human detection and segmentation via hierarchical part-template matching,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 4, pp. 604–618, 2010.
  * [4] O. Tuzel, F. Porikli, and P. Meer, “Pedestrian detection via classification on riemannian manifolds,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 10, pp. 1713–1727, 2008.
  * [5] B. Leibe, E. Seemann, and B. Schiele, “Pedestrian detection in crowded scenes,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 878–885 vol. 1, 2005.
  * [6] P. Viola and M. Jones, “Robust real-time face detection,” in Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 2, pp. 747–747, 2001.
  * [7] S.-F. Lin, J.-Y. Chen, and H.-X. Chao, “Estimation of number of people in crowded scenes using perspective transformation,” IEEE Trans. Syst. Man Cybern. Part A, vol. 31, pp. 645–654, 2001.
  * [8] M. Li, Z. Zhang, K. Huang, and T. Tan, “Estimating the number of people in crowded scenes by mid based foreground segmentation and head-shoulder detection,” in 2008 19th International Conference on Pattern Recognition, pp. 1–4, 2008.
  * [9] P. F. Felzenszwalb, R. B. Girshick, D. McAllester, and D. Ramanan, “Object detection with discriminatively trained part-based models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 9, pp. 1627–1645, 2010.
  * [10] I. S. Topkaya, H. Erdogan, and F. Porikli, “Counting people by clustering person detector outputs,” in 2014 11th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 313–318, 2014.
  * [11] B. Wu and R. Nevatia, “Detection and tracking of multiple, partially occluded humans by bayesian combination of edgelet based part detectors,” International Journal of Computer Vision, vol. 75, pp. 247–266, 2006.
  * [12] K. Chen, C. C. Loy, S. Gong, and T. Xiang, “Feature mining for localised crowd counting,” in BMVC, 2012.
  * [13] A. C. Davies, J. H. Yin, and S. A. Velastín, “Crowd monitoring using image processing,” Electronics & Communication Engineering Journal, vol. 7, pp. 37–47, 1995.
  * [14] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 1, pp. 886–893 vol. 1, 2005.
  * [15] A. B. Chan and N. Vasconcelos, “Bayesian poisson regression for crowd counting,” in 2009 IEEE 12th International Conference on Computer Vision, pp. 545–551, 2009.
  * [16] N. Paragios and V. Ramesh, “A mrf-based approach for real-time subway monitoring,” in Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, vol. 1, pp. I–I, 2001.
  * [17] Y. Tian, L. Sigal, H. Badino, F. D. la Torre, and Y. Liu, “Latent gaussian mixture regression for human pose estimation,” in ACCV, 2010.
  * [18] Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma, “Single-image crowd counting via multi-column convolutional neural network,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 589–597, 2016.
  * [19] Y. Li, X. Zhang, and D. Chen, “Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1091–1100, 2018.
  * [20] X. Cao, Z. Wang, Y. Zhao, and F. Su, “Scale aggregation network for accurate and efficient crowd counting,” in ECCV, 2018.
  * [21] Z. Yan, Y. Yuan, W. Zuo, X. Tan, Y. Wang, S. Wen, and E. Ding, “Perspective-guided convolution networks for crowd counting,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 952–961, 2019\.
  * [22] X. Jiang, L. Zhang, M. Xu, T. Zhang, P. Lv, B. Zhou, X. Yang, and Y. Pang, “Attention scaling for crowd counting,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4705–4714, 2020.
  * [23] Q. Wang and T. Breckon, “Crowd counting via segmentation guided attention networks and curriculum loss,” IEEE Transactions on Intelligent Transportation Systems, 2022.
  * [24] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [25] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9, 2015.
  * [26] J. Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788, 2016.
  * [27] V. Badrinarayanan, A. Handa, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for robust semantic pixel-wise labelling,” ArXiv, vol. abs/1505.07293, 2015.
  * [28] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” ArXiv, vol. abs/1505.04597, 2015.
  * [29] C. Zhang, H. Li, X. Wang, and X. Yang, “Cross-scene crowd counting via deep convolutional neural networks,” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 833–841, 2015.
  * [30] S. A. M. Saleh, S. A. Suandi, and H. Ibrahim, “Recent survey on crowd density estimation and counting for visual surveillance,” Eng. Appl. Artif. Intell., vol. 41, pp. 103–114, 2015.
  * [31] Jeevitha, “A review of crowd counting techniques,” 2018.
  * [32] R. S. P. K, S. Nithya, and S. Borra, “Crowd density estimation using image processing : A survey,” 2018.
  * [33] V. A. Sindagi and V. M. Patel, “A survey of recent advances in cnn-based single image crowd counting and density estimation,” Pattern Recognit. Lett., vol. 107, pp. 3–16, 2018.
  * [34] T. W. Cenggoro, “Deep learning for crowd counting: A survey,” Engineering, MAthematics and Computer Science (EMACS) Journal, 2019.
  * [35] N. Ilyas, A. Shahzad, and K. Kim, “Convolutional-neural network-based image crowd counting: Review, categorization, analysis, and performance evaluation,” Sensors (Basel, Switzerland), vol. 20, 2020.
  * [36] G. Gao, J. Gao, Q. Liu, Q. Wang, and Y. Wang, “Cnn-based density estimation and crowd counting: A survey,” ArXiv, vol. abs/2003.12783, 2020.
  * [37] M. S. Abdou and A. Erradi, “Crowd counting: A survey of machine learning approaches,” 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT), pp. 48–54, 2020.
  * [38] Y. Luo, J. Lu, and B. Zhang, “Crowd counting for static images: A survey of methodology,” 2020 39th Chinese Control Conference (CCC), pp. 6602–6607, 2020.
  * [39] H. Bai, J. Mao, and S.-H. G. Chan, “A survey on deep learning-based single image crowd counting: Network design, loss function and supervisory signal,” 2020\.
  * [40] W. Jingying, “A survey on crowd counting methods and datasets,” 2020.
  * [41] R. Gouiaa, M. A. Akhloufi, and M. Shahbazi, “Advances in convolution neural networks based crowd counting and density estimation,” Big Data Cogn. Comput., vol. 5, p. 50, 2021.
  * [42] Z. Fan, H. Zhang, Z. Zhang, G. Lu, Y. Zhang, and Y. Wang, “A survey of crowd counting and density estimation based on convolutional neural network,” Neurocomputing, vol. 472, pp. 224–251, 2022.
  * [43] K. Chen, C. C. Loy, S. Gong, and T. Xiang, “Feature mining for localised crowd counting,” in Proceedings of the British Machine Vision Conference, pp. 21.1–21.11, BMVA Press, 2012.
  * [44] A. B. Chan, Z.-S. J. Liang, and N. Vasconcelos, “Privacy preserving crowd monitoring: Counting people without people models or tracking,” 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–7, 2008.
  * [45] H. Idrees, I. Saleemi, C. Seibert, and M. Shah, “Multi-source multi-scale counting in extremely dense crowd images,” 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2547–2554, 2013.
  * [46] H. Idrees, M. Tayyab, K. Athrey, D. Zhang, S. A. Al-Maadeed, N. M. Rajpoot, and M. Shah, “Composition loss for counting, density map estimation and localization in dense crowds,” ArXiv, vol. abs/1808.01050, 2018.
  * [47] V. A. Sindagi, R. Yasarla, and V. M. Patel, “Jhu-crowd++: Large-scale crowd counting dataset and a benchmark method,” Technical Report, 2020.
  * [48] T. Peng, Q. Li, and P. Zhu, “Rgb-t crowd counting from drone: A benchmark and mmccn network,” in Computer Vision – ACCV 2020: 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part VI, (Berlin, Heidelberg), p. 497–513, Springer-Verlag, 2020.
  * [49] Q. Wang, J. Gao, W. Lin, and X. Li, “Nwpu-crowd: A large-scale benchmark for crowd counting and localization,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, pp. 2141–2149, 2021.
  * [50] R. Guerrero-Gómez-Olmedo, B. Torre-Jiménez, R. López-Sastre, S. Maldonado-Bascón, and D. O. noro Rubio, “Extremely overlapping vehicle counting,” in Iberian Conference on Pattern Recognition and Image Analysis (IbPRIA), 2015.
  * [51] M.-R. Hsieh, Y.-L. Lin, and W. H. Hsu, “Drone-based object counting by spatially regularized regional proposal network,” 2017 IEEE International Conference on Computer Vision (ICCV), pp. 4165–4173, 2017.
  * [52] P. Zhu, L. Wen, D. Du, X. Bian, H. Fan, Q. Hu, and H. Ling, “Detection and tracking meet drones challenge,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1, 2021.
  * [53] L. Liu, J. Chen, H. Wu, G. Li, C. Li, and L. Lin, “Cross-modal collaborative representation learning and a large-scale rgbt benchmark for crowd counting,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4821–4831, 2021.
  * [54] L. Zeng, X. Xu, B. Cai, S. Qiu, and T. Zhang, “Multi-scale convolutional neural networks for crowd counting,” 2017 IEEE International Conference on Image Processing (ICIP), pp. 465–469, 2017.
  * [55] X. Jiang, Z. Xiao, B. Zhang, X. Zhen, X. Cao, D. S. Doermann, and L. Shao, “Crowd counting and density estimation by trellis encoder-decoder networks,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6126–6135, 2019.
  * [56] C. Gao, P. Wang, and Y. Gao, “Mobilecount: An efficient encoder-decoder framework for real-time crowd counting,” in Pattern Recognition and Computer Vision: Second Chinese Conference, PRCV 2019, Xi’an, China, November 8–11, 2019, Proceedings, Part II, p. 582–595, Springer-Verlag, 2019\.
  * [57] Z. Chen, J. Cheng, Y. Yuan, D. Liao, Y. Li, and J. Lv, “Deep density-aware count regressor,” in ECAI, 2020.
  * [58] Z. Ma, X. Wei, X. Hong, and Y. Gong, “Bayesian loss for crowd count estimation with point supervision,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), (Los Alamitos, CA, USA), pp. 6141–6150, nov 2019.
  * [59] X. Shi, X. Li, C. Wu, S. Kong, J. Yang, and L. He, “A real-time deep network for crowd counting,” ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2328–2332, 2020.
  * [60] Y. Ma, V. Sanchez, and T. Guha, “Fusioncount: Efficient crowd counting via multiscale feature fusion,” ArXiv, vol. abs/2202.13660, 2022.
  * [61] L. Zhang, M. Shi, and Q. Chen, “Crowd counting via scale-adaptive convolutional neural network,” 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1113–1121, 2018.
  * [62] L. Liu, H. Lu, H. Zou, H. Xiong, Z. Cao, and C. Shen, “Weighing counts: Sequential crowd counting by reinforcement learning,” ArXiv, vol. abs/2007.08260, 2020.
  * [63] L. Boominathan, S. S. S. Kruthiventi, and R. V. Babu, “Crowdnet: A deep convolutional network for dense crowd counting,” Proceedings of the 24th ACM international conference on Multimedia, 2016.
  * [64] D. Sam, S. Surya, and R. Babu, “Switching convolutional neural network for crowd counting,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (Los Alamitos, CA, USA), pp. 4031–4039, IEEE Computer Society, jul 2017.
  * [65] V. A. Sindagi and V. M. Patel, “Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting,” 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6, 2017.
  * [66] J. Liu, C. Gao, D. Meng, and A. G. Hauptmann, “Decidenet: Counting varying density crowds through attention guided detection and density estimation,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5197–5206, 2018.
  * [67] R. Girshick, “Fast r-cnn,” in 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1440–1448, 2015.
  * [68] M. Shi, Z. Yang, C. Xu, and Q. Chen, “Revisiting perspective information for efficient crowd counting,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7271–7280, 2019.
  * [69] J. Gao, Q. Wang, and X. Li, “Pcc net: Perspective crowd counting via spatial convolutional network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, pp. 3486–3498, 2020.
  * [70] Y. Lei, Y. Liu, P. Zhang, and L. Liu, “Towards using count-level weak supervision for crowd counting,” Pattern Recognit., vol. 109, p. 107616, 2021.
  * [71] Y. Yang, G. Li, Z. Wu, L. Su, Q. Huang, and N. Sebe, “Weakly-supervised crowd counting learns from sorting rather than locations,” in Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII, (Berlin, Heidelberg), p. 1–17, Springer-Verlag, 2020\.
  * [72] H. Tang, Y. Wang, and L.-P. Chau, “Tafnet: A three-stream adaptive fusion network for rgb-t crowd counting,” ArXiv, vol. abs/2202.08517, 2022.
  * [73] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2016.
  * [74] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510–4520, 2018.
  * [75] V. Nekrasov, C. Shen, and I. D. Reid, “Light-weight refinenet for real-time semantic segmentation,” ArXiv, vol. abs/1810.03272, 2018.
  * [76] A. Zhang, L. Yue, J. Shen, F. Zhu, X. Zhen, X. Cao, and L. Shao, “Attentional neural fields for crowd counting,” in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5713–5722, 2019.
  * [77] Q. Song, C. Wang, Y. Wang, Y. Tai, C. Wang, J. Li, J. Wu, and J. Ma, “To choose or to fuse? scale selection for crowd counting,” in AAAI, 2021.
  * [78] S. Gu and Z. Lian, “A unified multi-task learning framework of real-time drone supervision for crowd counting,” ArXiv, vol. abs/2202.03843, 2022.
  * [79] P. Thanasutives, K. ichi Fukui, M. Numao, and B. Kijsirikul, “Encoder-decoder based convolutional neural networks with multi-scale-aware modules for crowd counting,” 2020 25th International Conference on Pattern Recognition (ICPR), pp. 2382–2389, 2021.
  * [80] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
  * [81] W. Liu, M. Salzmann, and P. V. Fua, “Context-aware crowd counting,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5094–5103, 2019.
  * [82] T. Peng, Q. Li, and P. F. Zhu, “Rgb-t crowd counting from drone: A benchmark and mmccn network,” in ACCV, 2020.
  * [83] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, (Red Hook, NY, USA), p. 6000–6010, Curran Associates Inc., 2017.
  * [84] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” ArXiv, vol. abs/2010.11929, 2021.
  * [85] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. J’egou, “Training data-efficient image transformers & distillation through attention,” in ICML, 2021.
  * [86] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” ArXiv, vol. abs/2005.12872, 2020.
  * [87] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” ArXiv, vol. abs/2010.04159, 2021.
  * [88] L. Ye, M. Rochan, Z. Liu, and Y. Wang, “Cross-modal self-attention network for referring image segmentation,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10494–10503, 2019.
  * [89] V. Ranjan, M. Shah, and M. H. Nguyen, “Crowd transformer network,” ArXiv, vol. abs/1904.02774, 2019.
  * [90] J. Gao, M. Gong, and X. Li, “Congested crowd instance localization with dilated convolutional swin transformer,” ArXiv, vol. abs/2108.00584, 2021\.
  * [91] Y. Tian, X. Chu, and H. Wang, “Cctrans: Simplifying and improving crowd counting with transformer,” ArXiv, vol. abs/2109.14483, 2021.
  * [92] G. Sun, Y. Liu, T. Probst, D. P. Paudel, N. Popovic, and L. V. Gool, “Boosting crowd counting with transformers,” ArXiv, vol. abs/2105.10926, 2021.
  * [93] P. T. Do, “Attention in crowd counting using the transformer and density map to improve counting result,” 2021 8th NAFOSTED Conference on Information and Computer Science (NICS), pp. 65–70, 2021.
  * [94] D. Liang, W. Xu, and X. Bai, “An end-to-end transformer model for crowd localization,” ArXiv, vol. abs/2202.13065, 2022.
  * [95] X. Wei, Y. Kang, J. Yang, Y. Qiu, D. Shi, W. Tan, and Y. Gong, “Scene-adaptive attention network for crowd counting,” ArXiv, vol. abs/2112.15509, 2021\.
  * [96] D. Liang, X. Chen, W. Xu, Y. Zhou, and X. Bai, “Transcrowd: weakly-supervised crowd counting with transformers,” Science China Information Sciences, 2022\.
  * [97] X. Chu, Z. Tian, Y. Wang, B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen, “Twins: Revisiting the design of spatial attention in vision transformers,” in NeurIPS, 2021.
  * [98] L. Liu, H. Lu, H. Zou, H. Xiong, Z. Cao, and C. Shen, “Weighing counts: Sequential crowd counting by reinforcement learning,” ArXiv, vol. abs/2007.08260, 2020.
  * [99] J. Xu, Z. Sun, and C. Ma, “Crowd aware summarization of surveillance videos by deep reinforcement learning,” Multim. Tools Appl., vol. 80, pp. 6121–6141, 2021.
  * [100] S. Aich and I. Stavness, “Global sum pooling: A generalization trick for object counting with small datasets of large images,” arXiv preprint arXiv:1805.11123, 2018.
  * [101] H. Lin, Z. Ma, R. Ji, Y. Wang, and X. Hong, “Boosting crowd counting via multifaceted attention,” ArXiv, vol. abs/2203.02636, 2022.
  * [102] Q. Wang, J. Gao, W. Lin, and Y. Yuan, “Learning from synthetic data for crowd counting in the wild,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8190–8199, 2019.
  * [103] X. Liu, J. van de Weijer, and A. D. Bagdanov, “Exploiting unlabeled data in cnns by self-supervised learning to rank,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, pp. 1862–1878, 2019.
  * [104] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in Proceedings of the 26th Annual International Conference on Machine Learning, ICML ’09, (New York, NY, USA), p. 41–48, Association for Computing Machinery, 2009.
  * [105] Y. Liu, M. Shi, Q. Zhao, and X. Wang, “Point in, box out: Beyond counting persons in crowds,” in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6462–6471, 2019.
  * [106] B. Wang, H. Liu, D. Samaras, and M. Hoai, “Distribution matching for crowd counting,” ArXiv, vol. abs/2009.13077, 2020.
  * [107] R. Guerrero-Gómez-Olmedo, B. Torre-Jiménez, R. J. López-Sastre, S. Maldonado-Bascón, and D. Oñoro-Rubio, “Extremely overlapping vehicle counting,” in IbPRIA, 2015.
  * [108] J. Wan and A. B. Chan, “Modeling noisy annotations for crowd counting,” in NeurIPS, 2020.