# A Study of Deep Perceptual Metrics for Image quality Assessment

###### Abstract

Several metrics exist to quantify the similarity between images, but they are
inefficient when it comes to measure the similarity of highly distorted
images. In this work, we propose to empirically investigate perceptual metrics
based on deep neural networks for tackling the Image Quality Assessment (IQA)
task. We study deep perceptual metrics according to different hyperparameters
like the network’s architecture or training procedure. Finally, we propose our
multi-resolution perceptual metric (MR-Perceptual), that allows us to
aggregate perceptual information at different resolutions and outperforms
standard perceptual metrics on IQA tasks with varying image deformations. Our
code is available at https://github.com/ENSTA-U2IS/MR_perceptual

Index Terms—  Perceptual metric, Image quality assessment, deep neural
networks

##  1 Introduction

Image Quality Assessment (IQA) plays an essential role in image-based
applications [1, 2] where the acquisition systems or algorithms can introduce
image quality variations. Although IQA is a well-known problem [3], it is
difficult to define a metric directly linked to human perception. Indeed, for
humans, IQA is intuitive and effortless [4]. Still, it remains a subjective
measurement that is insufficient for validating algorithms or acquisition
systems.

Several perceptual metrics have been investigated [2, 1] for IQA, such as the
$L_{2}$ Euclidean distance, or SSIM [5]. Yet, the human perception of image
similarity relies on psychological vision mechanisms that are in a large
extent unknown, then hard to implement. On the other hand, existing metrics
only rely on estimating global or local variations between images.

Deep Learning-based perceptual metrics have been first proposed in [6], and
[7] for the style transfer problem. It was followed by several applications,
for the quality of super-resolution algorithms [6], semantic segmentation [8]
task, or Generative Adversarial Network (GAN) [9] outputs quality. A
perceptual metric is typically a $L_{2}$ distance between features extracted
from Deep Neural Networks (DNNs) after a forward pass of the input images.
While some perceptual metrics like the Fréchet Inception Distance (FID) [10]
are widely used to evaluate the quality of images generated by GANs, they are
limited to the comparison of the estimated distribution of two set of images.

In this paper, we propose investigating IQA using Deep Learning-based
perceptual metrics to compute the similarity between two images. Unlike
previous studies [11, 12] that learn an image quality metric using DNN, we
evaluate different DNNs and their associated hyperparameters (loss function,
normalisation, resolution of input images, and the features extraction
strategy), with the main goal of identifying a deep perceptual distance as
general-purpose metric closer to human perception.

Our contribution is threefold: first, we empirically investigate different DNN
perceptual metrics related to the network architecture. Next, we perform an
ablation study highlighting the relationship between the training procedure of
DNN parameters and the performances of deep perceptual metrics. Finally, we
propose a perceptual metric that achieves the state-of-the-art results in
unsupervised IQA by studying various hyperparameters impacting the computation
of perceptual losses.

##  2 Multi Resolution Perceptual metric

###  2.1 Notations and formalism

We denote $\mathcal{D}=\\{x_{i}\\}_{i=1}^{n}$ a dataset composed of $n$
images. Let $f_{\boldsymbol{\mathbf{\omega}}}(x_{i})$ denote the output of the
DNN $f$ with trainable parameters
$\boldsymbol{\mathbf{\omega}}=\\{\omega_{k}\\}$ applied on image $x_{i}$.

DNNs can be decomposed into $B$ blocks applied sequentially. For example, we
can decompose AlexNet [13], which comprises five convolutional layers, into
$B=5$ blocks, where each block is a convolutional layer. Let us denote
$\\{f^{b}_{\boldsymbol{\mathbf{\omega}}}(x_{i})\\}_{b=1}^{B}$ the set of the
feature maps of the $B$ blocks output from image $x_{i}$. For $b\in\llbracket
1,B\rrbracket$, the feature map
$f^{b}_{\boldsymbol{\mathbf{\omega}}}(x_{i})\in\mathbb{R}^{H_{b}\times
W_{b}\times C_{b}}$ is a three dimension tensor where $H_{b}$ and $W_{b}$
represent the height and width of the feature map and $C_{b}$ is the number of
channels. We denote $f^{b}_{\boldsymbol{\mathbf{\omega}}}(x_{i})[h,w,c]$ with
$(h,w,c)\in\llbracket 1,H_{b}\rrbracket\times\llbracket
1,W_{b}\rrbracket\times\llbracket 1,C_{b}\rrbracket$ a

particular value of the feature map.

###  2.2 Perceptual metric

The process of computing a perceptual metric can be divided into three stages:
the deep feature extraction strategy from an image given a DNN architecture,
the normalization strategy of the feature space, followed by the dissimilarity
measure to compare the features. We now present these different stages.

####  2.2.1 Deep Feature Extraction

The deep feature extraction is the initial step that allows representing the
data into a new feature space. Contrary to handcrafted features like GLCM
[14], we can use a trained DNN to extract features at different levels of the
network.

If in some cases DNNs parameters are obtained from pretrained general purpose
networks like ImageNet [15], some DNNs are finetuned to achieve better
performances in a dataset tailored to the evaluation of perceptual metrics
[11]

Also, in previous works, features were extracted from the image at the
original dimension ($\times 1$). However, we also explore the feature results
after upscaling by two the image, thanks to a bilinear interpolation ($\times
2$). Let us denote $\phi(x_{i})$ the latent representation of the image
$x_{i}$.

Given extracted features at different levels of a DNN, a straightforward
strategy, termed as linear features, takes all the feature maps containing the
perceptual and contextual information at different resolutions, and
concatenate them. It is defined as follows:

$\phi_{1}(x_{i})=\left[f^{1}_{\boldsymbol{\mathbf{\omega}}}(x_{i}),\ldots,f^{B}_{\boldsymbol{\mathbf{\omega}}}(x_{i})\right]$
(1)

An alternative strategy consists in combining features, using the Gram matrix
as proposed in [7], allowing to extract new features termed as quadratic
features. The Gram matrix $G_{\boldsymbol{\mathbf{\omega}}}^{b}$ of the layer
$b$ is a square matrix of size $C_{b}\times C_{b}$. Let
$(c_{1},c_{2})\in\llbracket 1,C_{b}\rrbracket^{2}$ . The Gram matrix’s
coefficient at position $(c_{1},c_{2})$ defined for a feature map
$f^{b}_{\boldsymbol{\mathbf{\omega}}}(x_{i})$ is given by:
$G_{\boldsymbol{\mathbf{\omega}}}^{b}[c_{1},c_{2}](x_{i})=\sum_{h,w}f^{b}_{\boldsymbol{\mathbf{\omega}}}(x_{i})[h,w,c_{1}]f^{b}_{\boldsymbol{\mathbf{\omega}}}(x_{i})[h,w,c_{2}].$
We can now define the quadratic features of an image $x_{i}$ as:

$\phi_{2}(x_{i})=\left[G_{\boldsymbol{\mathbf{\omega}}}^{1}(x_{i}),\ldots,G_{\boldsymbol{\mathbf{\omega}}}^{B}(x_{i})\right]$
(2)

On the one hand, the linear features are directly linked to the content
(layout) of an image and to the first moments of the feature maps. On the
other hand, quadratic features are linked to the style of an image [7], and
capture stationary information related to the second moment, i.e. the
covariance.

####  2.2.2 Features Normalization

Because the values vary in magnitudes between feature maps, it is essential to
normalize them to homogenize all the layers and their importance. This work
compares two normalization strategies. Current solutions consider an $L_{2}$
normalization, that divides each value by the $L_{2}$ norm of the feature map.
Yet it is also possible to normalize with $L_{1}$ or with a sigmoid function.
We propose to normalize using a sigmoid function, bounding all values of the
feature map within $[0,1]$. Another operation which can be done is to use the
ReLu function before normalising.

####  2.2.3 Dissimilarity measure

In order to quantify the difference between the latent representation of two
images, we need to define a dissimilarity measure $D$ in the feature space,
not necessarily limited to distance metrics. We expect that $D$ associated
with the extracted features is linked to human perceptual dissimilarity
metric.

Typically perceptual loss uses the $L_{2}$ norm (MSE) between features. In
addition, we propose to use different dissimilarities such as the $L_{1}$ norm
(MAE), and the binary cross-entropy (CE).

###  2.3 MR-Perceptual loss

Classically the perceptual loss [11] is composed of VGG [16] linear features,
followed by a $L_{2}$ normalization and the dissimilarity metrics is the MSE.
In the rest of the paper we will refer to this loss as the classical
perceptual loss.

Based on these three main stages, we propose to change this classical
perceptual loss [11] by first proposing a multi-scale and multi-statistic
feature space. Our feature space is multi-scale because instead of extracting
the feature at just one resolution, we extract the descriptor at two
resolutions ($\times 1$ and $\times 2$). Our descriptor is also multi-
statistic since we concatenate quadratic and linear features for the standard
resolution. We use the sigmoid function as a normalization function and then
use a Binary cross-entropy as a dissimilarity measure. The full process is
illustrated in Fig. 1.

Fig. 1:  MR-Perceptual loss pipeline: from two images of size $3\times
64\times 64$, we create 6 outputs (3 per image). This permits to extract more
information than classical perceptual loss.

##  3 Experiments

In this section, we first introduce the used dataset (Sec. 3.1), then compare
different feature spaces from different architectures (Sec. 3.2). Next, we
discuss the importance of how to learn the representation (Sec. 3.3), and
finally we perform an ablation study with our technique (Sec. 3.4).

###  3.1 Dataset

To evaluate the performances of perceptual metrics, we use the Two-Alternative
Forced-Choices (2AFC) dataset [11]. The test sets include 36.3$K$ triplets
composed of one reference image and two distorted images associated with their
scores in $[0,1]$ defining the ground truth perceptual dissimilarity.
According to a human panel, the score reflects the proportion of votes for the
chosen image for each tuple. Specifically, a ground truth will have a score of
0 if all the testers wave chosen the first image and 1 if all the testers wave
chosen the second image.

The dataset is organized into six groups according to the transformations
applied to the distorted images as follows: Trad uses photometric and
geometric transformations.
CNN uses transformations coming from DNN, like denoising autoencoders.
SuperRes uses super resolution algorithms on images coming from the NITRE 2017
challenge [17].
Deblur uses image extracted from video clips [18], with video deblurring
algorithms.
Color uses the output of image translation algorithms for image colorization
applied on ImageNet [19].
FrameInterp uses different frame interpolation algorithms applied on the Davis
Middlebury dataset [20].

###  3.2 Link between the IQA and the architectures

To evaluate the deep feature from different networks, we extracted linear
feature from AlexNet [13], SqueezeNet [21], VGG [16], Resnet [22] and VIT
[23]. All theses networks are pretrained on imagenet [19].

These architectures are organized in convolutional blocks followed by dense
layers. Similarly to [11],we use the output of the five convolution blocks to
extract features, except for SqueezeNet of which we use seven blocks.

Table 2 presents the results of a handcrafted technique (SSIM) and classical
perceptual losses on different DNN architectures. The first result is that
perceptual losses highly outperform SSIM, which supports the hypothesis of a
better representation of human similarity perception. Moreover, the AlexNet
feature space seems to perform better than other architectures. Our
interpretation is twofold: first, ImageNet accuracy is not necessarily linked
to the quality of the feature space since the tasks are different; secondly,
the deeper an architecture is, the worse it performs. This might be linked to
the propagation of the perceptual information throughout all the layers, such
that all layers are initially trained for the ImageNet task.

Loss | MSE | CE | MSE | MSE | MSE | MSE | MSE | CE | CE
---|---|---|---|---|---|---|---|---|---
Normalization | L2 | sigmoid | L2 | L2 | sigmoid | sigmoid | L2 | ReLu + L1 | sigmoid
Feature | Linear | Linear | Linear | Linear | Linear | Linear | Quadratic | Linear +Quadratic | Linear +Quadratic
resolution | $\times 1$ | $\times 1$ | $\times 2$ |  $\times 1$ \+ $\times 2$ | $\times 1$ | $\times 1$ | $\times 2$ |  $\times 1$ \+ $\times 2$ |  $\times 1$ \+ $\times 2$
traditional | 70,56 | 71,3 | 72,93 | 72,81 | 71,74 | 72,31 | 73,02 | 72,66 | 73,78
cnn | 83,17 | 83,4 | 83,07 | 83,26 | 83,03 | 82,9 | 82,85 | 83,8 | 83,76
super resolution | 71,65 | 71,42 | 71,56 | 71,7 | 71,55 | 71,18 | 71,59 | 71,72 | 71,73
debluring | 60,68 | 61,27 | 60,64 | 60,8 | 60,67 | 60,48 | 60,99 | 61,48 | 61,37
colorization | 65,01 | 64,76 | 63,59 | 64,81 | 64,78 | 63,06 | 64,73 | 64,93 | 64,81
frameinterp | 62,65 | 63,55 | 62,02 | 62,49 | 62,56 | 62 | 62,08 | 63,82 | 63,46
AVERAGE | 68,95 | 69,28 | 68,97 | 69,31 | 69,06 | 68,66 | 69,21 | 69,74 | 69,82

Table 1: Ablation study on AlexNet [13] pretrained on ImageNet with a
supervised strategy.

Datasets | SSIM | Alexnet | VGG | SqueezeNet | Resnet18 | Resnet50 | Resnet101 | VIT
---|---|---|---|---|---|---|---|---
Trad | 62,73 | 70,56 | 70,05 | 73,3 | 69,66 | 70,73 | 70,71 | 57,68
CNN | 77,6 | 83,17 | 81,28 | 82,64 | 81,59 | 81,43 | 80,88 | 80,38
SuperRes | 63,13 | 71,65 | 69,02 | 70,15 | 69,69 | 68,99 | 68,65 | 64,94
Deblur | 54,23 | 60,68 | 59,05 | 60,13 | 59,8 | 58,9 | 58,9 | 58,93
Color | 60,89 | 65,01 | 60,19 | 63,57 | 60,49 | 60,12 | 59,46 | 63,23
FrameInterp | 57,11 | 62,65 | 62,11 | 61,98 | 62,54 | 61,33 | 61,93 | 56,09
AVERAGE | 62,61 | 68,95 | 66,95 | 68,63 | 67,30 | 66,92 | 66,76 | 63,54
| ImageNet Top1 acc
---
NA | 63,30 | 74,50 | 57.50 | 73.19 | 77.15 | 80.9 | 77.91

Table 2: Comparative results of different DNN architectures linear
features.The firsts rows denotes the 2AFC score. The last row shows the Top1
accuracy on ImageNet [19].

###  3.3 Link between the IQA and the training procedure

In Section 3.2, first experiments focused on the impact of the architecture
against a classical perceptual metric. Now, we focus our experiments on the
strategy to train a DNN for an optimal representation for perceptual queries.
For this purpose, we consider a ResNet50 architecture trained on ImageNet and
the following training strategies: a supervised training, DeepCluster [24],
Dino [25], MoCo v2 [26], OBoW [27], SimCLR [28], SwAV [29], and finally a
random initialization of the parameters. In Table 3, we compare the
performance of ResNet50 with the different pretrained parameters, and we
observe that our supervised training outperforms the others in most of the
tasks. This shows that a supervised procedure helps to inject in the network
useful information for the perceptual task.

Dataset | Random | Supervised | Deepcluster | Dino | MoCo | Obow | SimCLR | SwAV
---|---|---|---|---|---|---|---|---
Trad | 58,54 | 70,73 | 68,73 | 69,78 | 67,91 | 69,74 | 68,97 | 68,77
CNN | 80,07 | 81,43 | 80,21 | 80,03 | 78,3 | 79,04 | 79,12 | 79,74
SuperRes | 65,97 | 68,99 | 66,7 | 66,25 | 67,17 | 65,88 | 67,43 | 66,48
Deblur | 59,32 | 58,9 | 58,26 | 58,13 | 58,45 | 57,83 | 57,92 | 58,09
Color | 63,03 | 60,12 | 55,92 | 56,12 | 56,17 | 55,81 | 56,37 | 56,26
FrameInterp | 56,99 | 61,33 | 61,94 | 62,27 | 62,45 | 61,54 | 61,66 | 62,48
AVERAGE | 63,99 | 66,92 | 65,29 | 65,43 | 65,08 | 64,97 | 65,25 | 65,3

Table 3: Comparative results showing the impact of supervised training on 2AFC
[11] with Resnet 50 [22] architecture. We run a linear feature extraction with
different pretraining conditions.

###  3.4 Importance of the different components for IQA

Based on previous results in Sections 3.2 and 3.3, AlexNet trained on ImageNet
in a supervised manner is the best to quantify the perceptual dissimilarity.
We studied in the table 4 the performance according to the extracted features
and observed that features extracted from 4th and 5th layers are the best for
the Trad set. But features extracted from the 2nd and 3rd outperform on CNN,
SuperRes, Deblur and Color distortions.

This suggests that some layers might focus on particular details in the
distorted images which provides clues for being invariant to some distortions.

Dataset | Block 1 | Block 2 | Block 3 | Block 4 | Block 5 | All
---|---|---|---|---|---|---
Trad | 59,15 | 69,35 | 71,97 | 72,92 | 73,29 | 70,56
CNN | 81,65 | 82,88 | 82,94 | 82,84 | 82,03 | 83,17
SuperRes | 65,94 | 71,6 | 71,63 | 71,19 | 70,6 | 71,65
Deblur | 59,08 | 60,87 | 60,67 | 60,43 | 60,1 | 60,68
Color | 62,82 | 64,73 | 64,42 | 63,88 | 63,9 | 65,01
FrameInterp | 57,23 | 61,95 | 62,69 | 62,67 | 62,71 | 62,65
AVERAGE | 64,31 | 68,56 | 69,05 | 68,99 | 68,77 | 68,95

Table 4: Comparative results showing the impact the chosen layer on 2AFC [11]
with AlexNet [13] architecture. The bolded results shows the best results
among blocks

Table 1 shows an ablation study to evaluate the relevant hyperparameters for
designing a novel perceptual metric as detailed in Section 2.2. We consider
the features extraction strategies, the type of dissimilarity metric in the
feature space, the normalization strategy, and finally, the resolution. Multi-
resolution seems to be the key to improving performances. In addition, Multi
statistic seems to improve also the performances for certain distortions.

As shown in Tab. 5, the classic perceptual metric setup is outperformed in all
the distortions; this remains true for all the networks, including Watching
[30], Split-brain [31], Puzzle [32] and BiGAN [33] .

Dataset | Ours | Watching | Split-Brain | Puzzle | BiGAN
---|---|---|---|---|---
Trad | 73,8 | 66,5 | 69,5 | 71,5 | 69,8
CNN | 83,8 | 80,7 | 81,4 | 82,0 | 83,0
SuperRes | 71.7 | 69,6 | 69,6 | 70,2 | 70,7
Deblur | 61,4 | 60,6 | 59.3 | 60,2 | 60,5
Color | 64,9 | 64,4 | 64,3 | 62,8 | 63,7
FrameInterp | 63.5 | 61,6 | 61,1 | 61,8 | 62,5
AVERAGE | 69,8 | 67,2 | 67,5 | 68,1 | 68,4

Table 5: Comparison of our Alexnet with MR-perceptual loss with the setup
presented in [11].

##  4 Conclusions

We empirically investigated general-purpose deep perceptual metrics w.r.t.
different experimental settings on an IQA task. First, we show that it is
unnecessary to use deeper DNN with complex architecture; a simple AlexNet is
sufficient for perceptual metrics. Despite convincing results of self-
supervised, we show that a supervised strategy remains the best choice.
Finally, we confirm that combining features at different resolutions is
relevant as it forces the DNN to be more robust against various types of
distortion. Future work would involve combining this new perceptual metric and
image-to-image translation DNNs to improve the quality of generated images.

## References

  * [1] Guangtao Zhai and Xiongkuo Min,  “Perceptual image quality assessment: a survey,”  Science China Information Sciences, vol. 63, no. 11, pp. 211301, 2020.
  * [2] Vipin Kamble and KM Bhurchandi,  “No-reference image quality assessment algorithms: A survey,”  Optik, vol. 126, no. 11-12, pp. 1090–1097, 2015.
  * [3] Zhou Wang,  “Applications of objective image quality assessment methods [applications corner],”  IEEE signal processing magazine, vol. 28, no. 6, pp. 137–142, 2011\.
  * [4] Oleg S Pianykh, Ksenia Pospelova, and Nick H Kamboj,  “Modeling human perception of image quality,”  Journal of digital imaging, vol. 31, no. 6, pp. 768–775, 2018.
  * [5] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli,  “Image quality assessment: from error visibility to structural similarity,”  IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
  * [6] Justin Johnson, Alexandre Alahi, and Li Fei-Fei,  “Perceptual losses for real-time style transfer and super-resolution,”  in ECCV. Springer, 2016, pp. 694–711.
  * [7] LA Gatys, AS Ecker, and M Bethge,  “A neural algorithm of artistic style,”  in 16th Annual Meeting of the Vision Sciences Society (VSS 2016). Scholar One, Inc., 2016, p. 326.
  * [8] Pauline Luc, Camille Couprie, Soumith Chintala, and Jakob Verbeek,  “Semantic segmentation using adversarial networks,”  in NIPS Workshop on Adversarial Training, 2016.
  * [9] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros,  “Image-to-image translation with conditional adversarial networks,”  in CVPR, 2017, pp. 1125–1134.
  * [10] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter,  “Gans trained by a two time-scale update rule converge to a local nash equilibrium,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [11] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang,  “The unreasonable effectiveness of deep features as a perceptual metric,”  in CVPR, 2018, pp. 586–595.
  * [12] Hossein Talebi and Peyman Milanfar,  “Learned perceptual image enhancement,”  in 2018 IEEE international conference on computational photography (ICCP). IEEE, 2018, pp. 1–13.
  * [13] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton,  “Imagenet classification with deep convolutional neural networks,”  in Advances in Neural Information Processing Systems, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds. 2012, vol. 25, Curran Associates, Inc.
  * [14] Robert M Haralick, Karthikeyan Shanmugam, and Its’ Hak Dinstein,  “Textural features for image classification,”  IEEE Transactions on systems, man, and cybernetics, , no. 6, pp. 610–621, 1973.
  * [15] Marcel Simon, Erik Rodner, and Joachim Denzler,  “Imagenet pre-trained models with batch normalization,”  arXiv preprint arXiv:1612.01452, 2016.
  * [16] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  in ICLR, 2015.
  * [17] Eirikur Agustsson and Radu Timofte,  “Ntire 2017 challenge on single image super-resolution: Dataset and study,”  in CVPR workshops, 2017, pp. 126–135.
  * [18] Shuochen Su, Mauricio Delbracio, Jue Wang, Guillermo Sapiro, Wolfgang Heidrich, and Oliver Wang,  “Deep video deblurring for hand-held cameras,”  in CVPR, 2017, pp. 1279–1288.
  * [19] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.,  “Imagenet large scale visual recognition challenge,”  International journal of computer vision, vol. 115, no. 3, pp. 211–252, 2015.
  * [20] Daniel Scharstein and Richard Szeliski,  “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,”  International journal of computer vision, vol. 47, no. 1, pp. 7–42, 2002.
  * [21] Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer,  “Squeezenet: Alexnet-level accuracy with 50x fewer parameters and ¡0.5mb model size,” 2016.
  * [22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in CVPR, 2016, pp. 770–778.
  * [23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby,  “An image is worth 16x16 words: Transformers for image recognition at scale,” 2021.
  * [24] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze,  “Deep clustering for unsupervised learning of visual features,” 2019\.
  * [25] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin,  “Emerging properties in self-supervised vision transformers,” 2021.
  * [26] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He,  “Improved baselines with momentum contrastive learning,” 2020.
  * [27] Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Pérez,  “Obow: Online bag-of-visual-words generation for self-supervised learning,”  in 2021 CVPR. IEEE, 2021, pp. 6826–6836.
  * [28] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton,  “A simple framework for contrastive learning of visual representations,” 2020.
  * [29] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin,  “Unsupervised learning of visual features by contrasting cluster assignments,” 2021.
  * [30] Deepak Pathak, Ross B. Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan,  “Learning features by watching objects move,”  CoRR, vol. abs/1612.06370, 2016.
  * [31] Richard Zhang, Phillip Isola, and Alexei A. Efros,  “Split-brain autoencoders: Unsupervised learning by cross-channel prediction,” 2017.
  * [32] Mehdi Noroozi and Paolo Favaro,  “Unsupervised learning of visual representations by solving jigsaw puzzles,” 2017.
  * [33] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell,  “Adversarial feature learning,”  CoRR, vol. abs/1605.09782, 2016.


# Resource allocation method using tug-of-war-based synchronization

Song-Ju Kim†‡∗, Hiroyuki Yasuda§, Ryoma Kitagawa¶, and Mikio Hasegawa¶

† SOBIN Institute LLC, Kawanishi, Japan
https://sobin.org
‡ Graduate School of Media and Governance, Keio University, Fujisawa, Japan
§ Graduate Schools for Law and Politics, The University of Tokyo, Tokyo, Japan
¶ Department of Electrical Engineering, Tokyo University of Science, Tokyo,
Japan
∗Email<EMAIL_ADDRESS>
###### Abstract

We propose a simple channel-allocation method based on tug-of-war (TOW)
dynamics, combined with the time scheduling based on nonlinear oscillator
synchronization to efficiently use of the space (channel) and time resources
in wireless communications. This study demonstrates that synchronization
groups, where each node selects a different channel, are non-uniformly
distributed in phase space such that every distance between groups is larger
than the area of influence. New type of self-organized spatiotemporal patterns
can be formed for resource allocation according to channel rewards.

Keyword:
Resource allocation, channel allocation, collision avoidance, reinforcement
learning, multi-armed bandit problem, synchronization

##  1 Introduction

Following the COVID-19 pandemic, humanity is increasingly demanding a more
efficient and equitable use of resources [1]. One of the key issues is the
allocation of communication resources. The demand for the Internet of Things
(IoT) applications has recently been rapidly increased remarkably. The use of
long-distance and power-saving communications, particularly low-power wide-
area networks (LPWA), is expected in IoT communication. However, due to its
characteristics, frequency congestion has become a serious problem,
necessitating the use of a dynamic multi-channel selection method.

Using lightweight reinforcement-learning, called “tug-of-war (TOW) dynamics
[2, 3],” Ma et al. proposed an autonomous channel selection method in our
previous study [4]. All nodes learned the channel status by acknowledgement
(ACK) in a repeated cycle of sleep, wake, data transmission, and sleep. We
also implemented this algorithm in $IEEE802.15.4e/4g$-compliant IoT devices
and experimented with $50$ IoT devices dynamically selecting five channels.
However, this method did not address time-scheduling optimization for overall
node transmission timings.

For efficient use of space (channel) and time resources, we present the first
model toward achieving “an economic allocation of resources.” In this study,
we propose a simple channel-allocation method based on TOW dynamics, combined
with the time scheduling based on the synchronization of nonlinear
oscillators. The synchronization is named “TOW-based synchronization” because
of the competing pushes and pulls that occur in phase space. In other words,
the TOW behaviors are active in both space and time in our new model.

##  2 Model

We have made the following assumptions in constructing our new model.

  1. 1.

There are $M$ nodes and $N$ communication channels.

  2. 2.

Regardless of whether there is any interaction between the nodes, each node’s
phase $\theta_{i}(t)$ ($i$$=$$1$, $\cdots$, $M$) is updated by adding $\Omega$
at every discrete time step.

  3. 3.

Each node can transmit a packet at every discrete time step through the
channel selected via TOW dynamics [2].

  4. 4.

Each packet transmission does not affect any other transmissions only if those
phase distances are larger than the area of influence $\phi_{th}$. In this
study, we fix $\Omega=$$\phi_{th}$ for all nodes.

  5. 5.

At some time steps (e.g. $\theta_{i}$$=$$0$), each node not only advances by
angle $\Omega$ but also interacts with other nodes by the interaction term
$\pm K\sin(\phi)$ which represents “push $(-)$” or “pull $(+)$,” where $K$
represents the coupling strength (parameter), and $\phi$ represents the phase
difference.

  6. 6.

Each node receives a reward ($+1$ or $-\omega$) according to the success
(winning at the slot machine of selected channel (say $\\#k$), whose reward
probability $P_{k}$ under the condition that there is no node using the same
channel within $\phi_{th}$ ) or failure (otherwise).

In this study, we propose the following channel-allocation method using the
TOW-based synchronization which is based on the Kuramoto’s de-synchronization
model [5, 6]; For node $i$ $=$$1,\cdots,M$, each phase $\theta_{i}(t)$ is
updated by the following equation:

$\displaystyle\theta_{i}(t+1)$ $\displaystyle=$
$\displaystyle\theta_{i}(t)+\Omega-K\sum_{|\theta_{j}-\theta_{i}|<\phi_{th}}(2\delta_{s_{j}(t),s_{i}(t)}-1)\sin(\theta_{j}(t)-\theta_{i}(t)),$
(1) $\displaystyle s_{i}(t)$ $\displaystyle=$ $\displaystyle
argmax_{k}(X_{k}(t)),\hskip 2.84526ptfor\hskip 2.84526ptk=1,\cdots,N.$ (2)

Here, $\delta_{i,j}$ is the Kronecker-Delta.

The selected channel $s_{i}(t)$ for node $i$ was determined by the following
TOW dynamics of $X_{k}(t)$ ($k=1,\cdots,N$) [2];

$\displaystyle X_{k}(t+1)$ $\displaystyle=$ $\displaystyle
Q_{k}(t+1)-\frac{1}{N-1}\sum_{l=1,l\neq k}^{N}Q_{l}(t+1)+\xi(t+1),$ (3)
$\displaystyle Q_{k}(t+1)$ $\displaystyle=$ $\displaystyle R_{k}(t)+\alpha
Q_{k}(t),$ (4) $\displaystyle R_{k}(t)$ $\displaystyle=$
$\displaystyle\left\\{\begin{array}[]{rr}+1&(success),\\\
-\omega&(failure).\end{array}\right.$ (7)

Here, $\xi(t)$ is an arbitrary random noise, $\alpha$ is the memory parameter,
and $R_{k}(t)$ is the reward from each slot machine whose reward probability
is $P_{k}$.

Figure 1 shows the interaction term between nodes for each phase difference.
The phase difference between two nodes increases if the same channel is
selected by both nodes (de-synchronization), whereas the phase difference
decreases if different channels are selected (synchronization). This TOW-like
“push and pull dynamics” between nodes in phase space can organize grouping
(synchronization group) of various selection nodes. The groups can be
distributed in the phase space such that every distance between groups should
be larger than $\phi_{th}$.

Figure 1: The interaction term between nodes for each phase difference.

###  2.1 Results

In this study, we fix $\Omega=$$\phi_{th}$$=\pi/4$ for all nodes, and only
once when crossing the $\theta_{i}$ $=0$ (once in eight time steps), the
interaction term $\pm K\sin(\phi)$ works. We confirmed that self-organized
spatiotemporal patterns can be formed. Synchronization groups, particularly
those where each node selects a different channel, are non-uniformly
distributed in phase space such that every distance between groups is larger
than $\phi_{th}$ (area of influence), as expected.

Figure 2(a) shows an example of our computer simulation results. We used the
following parameters: $M=10$, $N=5$, $\phi_{th}$$=$$\pi/4$, $\alpha$$=$$0.95$,
$Amp$$=$$0.1$ (noise amplitude), $K$$=$$0.5$, $P_{1}$$=$$0.1$,
$P_{2}$$=$$0.2$, $P_{3}$$=$$0.3$, $P_{4}$$=$$0.4$, and $P_{5}$$=$$0.5$
($\omega$ is calculated by the system itself.). Four synchronization groups
can be formed in phase space. Each red number denotes the node number which
selected the channel $\\#5$ (highest reward probability).

Figure 2(b) shows a typical example for two time-series of the phase of node
$\\#1$ and $\\#7$. The phase distance is fixed ($203$∘) after the emergence of
the self-organizing pattern. In Fig. 2(b), each distance between diagonal
stripes is $45$∘ because we set $\phi_{th}=\pi/4$. The difference $203$∘ can
be seen as $23$∘ because $203-(45$ $\times$ $4)$$=$$23$ in the figure.

Figure 2: An example of computer simulation results.

##  3 Conclusion

We can draw the following conclusions:

  1. 1.

More than $\lceil M/N\rceil$ synchronization groups can be non-uniformly
distributed in the phase space. Here, $\lceil x\rceil$ represents the ceiling
function of $x$.

  2. 2.

Each group is separated from the other by a distance of larger than
$\phi_{th}$.

There should be no collisions of selected channels in each group; however,
there are some collisions in our results. The groups are reorganized if the
collisions occur frequently, as shown in Fig. 2(b). In this first study, we
used TOW dynamics with parameter $\gamma=P_{[1]}+P_{[2]}$ for the channel
selection, where $P_{[n]}$ is the top $n$-th reward probability. This means
that all nodes are selfishly searching for the highest channel only. Here, it
is known that TOW dynamics can be optimized when $\omega$ $=$ $\gamma$ $/$
($2$ $-$ $\gamma$). The optimization in each group will be improved if we use
the model setting with $\gamma=P_{[N^{\prime}]}+P_{[N^{\prime}+1]}$ (see ref.
[3]), where $N^{\prime}$ represents the number of nodes in the group.

The maximum resource can be estimated as
$\frac{2\pi}{\phi_{th}}$$\times$$\sum_{k=1}^{N}$$P_{k}$. In the parameter
settings of Fig.2 case, the maximum $12$ ($8$ $\times$ $1.5$) nodes can
successfully transmit a packet at every time step. Our simulation results
showed the average $2.86$ nodes, which is about $1/4$ of the maximum resource.
This is because we used only $M=10$. We can use more nodes until $M=40$
(independent $5$ channels $\times$ $8$ phases $=$ $40$ nodes). Details of
performance evaluation for the throughputs of this method will be presented in
our future works.

## Acknowledgments

This work was supported by the research grant SP002 from SOBIN Institute, LLC.
We would like to thank Dr. Takahito Mitsui (Free University of Berlin) for
fruitful discussions in this study.

## References

  * [1] S. -J. Kim, T. Takahashi, and K. Sano, “A balance for fairness: fair distribution utilising physics,” Humanities & Social Sciences Communications, vol. 8, 131, May 2021.  DOI:10.1057/s41599-021-00806-w
  * [2] S. -J. Kim, M. Aono, and E. Nameda, “Efficient decision-making by volume-conserving physical object,” New Journal of Physics, vol. 17, 083023, Aug. 2015.  DOI:10.1088/1367-2630/17/8/083023
  * [3] S. -J. Kim, M. Naruse, and M. Aono, “Harnessing the computational power of fluids for optimization of collective decision making,” Philosophies, vol. 1, no. 3, pp. 245–260, Dec. 2016.  DOI:10.3390/philosophies1030245
  * [4] J. Ma, S. Hasegawa, S. -J. Kim, and M. Hasegawa, “A reinforcement-learning-based distributed resource selection algorithm for massive IoT,” Applied Science, vol. 9, no. 18, 3730, Sep. 2019.  DOI:10.3390/app9183730
  * [5] Y. Kuramoto, Chemical Oscillations, Waves, and Turbulence, Springer-Verlag, Berlin, Heidelberg, 1984.  DOI:10.1007/978-3-642-69689-3
  * [6] J. Degesys, I. Rose, A. Patel, and R. Nagpal, “DESYNC: self-organizing desynchronization and TDMA on wireless sensor networks,” Proc. 6th International Symposium on Information Processing in Sensor Networks, Cambridge, USA, 9691013, Apr. 2007. DOI:10.1109/IPSN.2007.4379660


# Short-Term Load Forecasting using Bi-directional Sequential Models and
Feature Engineering for Small Datasets

Abdul Wahab, Muhammad Anas Tahir, Naveed Iqbal, Faisal Shafait , and Syed
Muhammad Raza Kazmi

###### Abstract

Electricity load forecasting enables the grid operators to optimally implement
the smart grid’s most essential features such as demand response and energy
efficiency. Electricity demand profiles can vary drastically from one region
to another on diurnal, seasonal and yearly scale. Hence to devise a load
forecasting technique that can yield the best estimates on diverse datasets,
specially when the training data is limited, is a big challenge. This paper
presents a deep learning architecture for short-term load forecasting based on
bidirectional sequential models in conjunction with feature engineering that
extracts the hand-crafted derived features in order to aid the model for
better learning and predictions. In the proposed architecture, named as Deep
Derived Feature Fusion (DeepDeFF), the raw input and hand-crafted features are
trained at separate levels and then their respective outputs are combined to
make the final prediction. The efficacy of the proposed methodology is
evaluated on datasets from five countries with completely different patterns.
The results demonstrate that the proposed technique is superior to the
existing state of the art.

###### Index Terms:

Short Term Load Forecasting, Small Datasets, Deep Learning, Feature
Engineering, Bi-directional Sequential Models, Deep Derived Feature Fusion
(DeepDeFF).

##  I Introduction

Smart grid, in simple terms, implies monitoring and control of the power
system’s assets in the generation, transmission, distribution, and
utilization, to achieve high efficiency and reliability at low operational
costs. Its cardinal feature – demand response, can only be fully realized
through accurate forecasting of various variables, the most important of which
is the forecasting of electrical load [1]. Artificial intelligence is fast
becoming cardinal for data analytics and enhanced control of modern power
systems. One of its most desirable application in recent times is the load
forecasting through machine learning for predicting the trends in energy
demand, so that the control decisions can be proactively optimized.

Long-term [2], mid-term [3], and short-term [4] are the different types of
load forecasting found in the literature based on their duration of prediction
from years to minutes. Short-term load forecasting (SLF) is more difficult
than the mid and long term forecasting because of the greater variance in the
respective energy consumption patterns [5]. The advantage of SLF is that it
provides better insight into the electricity consumption patterns and a
greater degree of freedom for demand-side management. Also, SLF can be
aggregated to get mid-term and long-term forecasts. Therefore this paper
focuses on SLF.

One of the most challenging problems in SLF is posed by small datasets;
particularly in the case of individual households which usually exhibit wide
variations in energy consumption in short intervals, thus making it harder for
the deep learning models to learn the underlying patterns [5]. Before the deep
learning era, lot of research went into the engineering of hand-crafted
features which were required as the inputs to machine learning algorithms. The
capability of deep learning to extract implicit features, removed the need of
such complicated pre-processing of raw data. However deep learning models
require large training data to extract useful features, more so for the
datasets with high variances.

This paper presents a novel deep learning architecture that combines the use
of hand-crafted features with raw data, such that the deep learning model can
work well for SLF of small datasets. The results demonstrate significant
improvements in the performance, specially for small datasets by using the
proposed architecture.

##  II Literature Review

A significant amount of research has been carried out in recent times to
develop SLF as the enabling tool for efficient monitoring and control of power
system. Before the deep learning era, hand-crafted features were used to be
fed to a machine learning model for making predictions. In [6] feature
engineering was done to design a feature vector by performing entropy analysis
with a specific tolerance band and auto-correlation function. The designed
feature vector was then passed through an artificial neural network (ANN) for
prediction. Ferreira and da Silva used a Bayesian based approach to solve the
complexity of neural network and variable selection [7]. The approach has
theoretical ground but relies on various assumptions regarding the network
parameters distribution, requires three relevance thresholds and is
computationally expensive. Phase-space embedding method was used for the
selection of input variable which allowed to include the preference of the
past values of prediction quantity in the input vector [8]. A neural networks
based approach to forecast next 24 hour load on medium and low voltage
substations was presented in [9]. The use of separate models each for daily
average power and for intraday variation in power, improved the accuracy of
prediction compared to the model based on time series.

Learning the daily routine of the usage of various appliances can help in
better forecasting of an household’s load profile. It was shown in [10, 11]
that using the consumption data of the appliances together with the aggregated
data of the whole house as the input to the long short term memory (LSTM)
models gave better results than using the whole house readings alone.

Recently recurrent neural networks (RNN) have become the popular choice for
load forecasting. In [12] machine learning models were used for predicting the
energy demand on publicly available RTE dataset [13]. The performances of RNN
and support vector machine (SVM) models were compared using different input
features. The models were evaluated on a test set of 10 days of year 2017. The
results demonstrated that RNN performed better, with a MAPE of 3.52%, compared
to SVM with a MAPE of 14.00%.

A recent study [5] demonstrated how the individual household level load
forecasting can be challenging because of different patterns of energy
consumption of individual consumers [14]. A two layers based LSTM model was
proposed and compared with other models based on back-propagation neural
network (BPNN), k-nearest neighbour (KNN), extreme learning machine (ELM) and
input scheme combined with a hybrid forecasting framework (IS-HF). Individual
models for each household were trained and the best average MAPE of 44.06% was
achieved through LSTM. Also it was demonstrated that aggregating these
individual forecasts resulted into quite the same net MAPE of $\sim$8% that
was yielded when the aggregated data of the consumers was trained and tested
on a single model. This difference in the MAPE of individual versus aggregated
forecast established that the individual SLF is harder compared to aggregated
one; however the advantage of individual SLF is that it provides better
insight into the trend of each constituent customer and can easily be
aggregated together to provide the net trend.

Electricity demand is influenced by weather, holiday, time of day etc. Time
dependant convolution neural network (TD-CNN) and cycle based long short term
memory (C-LSTM) for short- and medium-term load forecasting was presented in
[15]. Electric load on weekly basis was arranged in image format on which TD-
CNN was run. C-LSTM helped to extract time dependencies between sequences. The
models performed better than the traditional LSTM model while reducing the
training time.

Another important application of SLF is in energy trading which is a complex
process due to non-periodic variations in energy consumption. Accurate
forecasting for hourly spot price is the key to achieve the best trading
decision which is vital for investors and retailers in electricity market. A
model based on a hybrid approach comprising of ARIMA, multiple linear
regression (MLR), and Holt-Winter model was proposed in [16]. The hybrid model
was tested for Iberian electricity market dataset to forecast hourly spot
prices for various numbers of days. A hybrid model based on non-linear
regression and SVM was proposed in [17], that was tested on ERCOT data [18].
This hybrid model achieved MAPE of 7.30% compared to the individual models
with 8.99% and 8.63% MAPE respectively.

Improvement of forecasting accuracy using standard LSTM model by feeding it
processed features rather than raw data was proposed in [19]. The power load
sequence was decomposed by complementary ensemble empirical mode decomposition
(CEEMD), then the approximate entropy (AE) values of the obtained subsequences
were calculated. The subsequences with similar AE values were merged into new
sequence to form the inputs of the load forecasting model. This reduced the
complexity of the power load sequence and improved the accuracy of load
forecasting. The vanilla LSTM network was improved in [20] by cleaning and
processing the raw load data using isolated forest algorithm.

Electric load forecasting requires training of large number of neurons in
hidden layer, which increases the size of the network and slows overall
training process. To reduce this overhead, a multi-column radial bias function
(MCRF) with error correction algorithm designed to reduce the number of hidden
neurons in a network, was proposed in [21]. It was shown that MCRF with only
50 neurons in hidden layer took only 10 minutes to train and achieved the MAPE
of 4.59% compared to other models with more than 150 neurons that achieved
better MAPE of 1.77% but took hours to train.

Accuracy of SLF can be improved through careful analysis of the load data to
find the effectiveness of selected features. A technique was proposed in [22]
for features selection where the bisecting K-means algorithm was used to
cluster the load data with high similarity for a forecast date. The ensemble
empirical mode decomposition (EEMD) helped to combine components with similar
entropy. A bidirectional recurrent neural network (BRNN) model was proposed to
forecast the load of the network. The model was verified on two datasets
including a dataset from load forecasting competition. The results showed that
BRNN model performed better even than the winner of the competition.

The literature survey therefore implies that a better load forecasting
technique with reduced statistical error is a hot topic of research for modern
power systems.

##  III Proposed Methodology

Recently [15, 5, 10] deep learning solutions, particularly sequential models
such as RNN and LSTM models are becoming popular choices for load forecasting.
LSTM [23] has become a state of the art tool for time series problems owing to
its ability to learn temporal patterns in sequential data. This paper presents
a novel architecture named as deep derived feature fusion (DeepDeFF),
comprising of a bidirectional sequential model with feature engineering for
realizing a more accurate SLF technique.

###  III-A Bidirectional Sequential Models

Bidirectional model trains forward and reverse nodes using respectively:  1.
input in positive time, i.e. the given input as it is,  2. input in negative
time, i.e. a time-reversed copy of the original input.  The advantage of
bidirectional model compared to conventional ANN models is that it observes
the input in both forward and reverse directions to extract more information
from the input sequence. This technique of negative time and bidirectional
layer was first discussed in [24].

This paper implements LSTM, RNN, gated recurrent unit (GRU) as well as their
bidirectional counterparts (BLSTM) [25], BRNN and (BGRU) on several datasets
for a comprehensive comparison presented in the results.

###  III-B Derived Features

The aim of derived features is to enrich the training data with useful
features for more accurate predictions. A deep learning model with enough
computation time and data may extract such features on its own, but this
cannot be guaranteed within the constraints of time and resource. Thus
providing these derived features explicitly as inputs can enable the model to
learn more from the data and converge quickly, specially for small datasets.
Generally the performance of deep learning models improve by increasing the
number of relevant input features unless it starts to over fit.

The basic features used for generation of derived features and as input for
the DeepDeFF model are:

  * •

Energy load consumption $E$.

  * •

Time-stamp of the day $T$, divided into 30 minutes interval each. The feature
is converted into One-hot encoding.

  * •

Current day of the week $W$, converted into one-hot encoding.

  * •

Holidays represented by a binary label $H$. At the moment only weekends are
marked as holidays, but in future work this can be expanded and synced with
other public and national holidays.

Derived features are calculated for each record in the input sequence (1, $K$,
$f$), where $K$ represents the number of past records used for creating the
input sequence and $f$ represents the basic features. Following are the
derived features that are calculated and used as input to the DeepDeFF model.

  * •

Average load consumption of $K$ time-steps.

  * •

Standard deviation of load consumption of $K$ time-steps.

  * •

Average load consumption of the time-stamp $t$ that is to be predicted, for
past $K$ days.

  * •

Standard deviation of load consumption of the time-stamp $t$ that is to be
predicted, for past $K$ days.

###  III-C Proposed Architecture

This paper proposes a two-layer bidirectional sequential model architecture
DeepDeFF, which inputs the raw and derived input features into separate layers
to extract learned features. The idea behind using separate input layers for
basic and derived sequences is to allow the sequential layers to learn from
the two input sequences independently. The goal is to exploit the relevance of
basic and derived sequences with the predictions individually. The learned
representation from the individual sequential layers is then merged and fed to
a dense layer followed by a linear activation output layer to make the final
prediction of the load at the next time interval.

Fig. 1 shows the schematics of the DeepDeFF architecture. The hyper parameter
settings consists of:

  1. (a)

20 nodes sequential layer

  2. (b)

a dropout of 0.2

  3. (c)

Adam optimizer

  4. (d)

MAPE as loss function

  5. (e)

Learning rate: 0.01

Figure 1: System Architecture: The input data is first pre-processed to
achieve the derived input features. The raw input features $T$, $W$, $H$ are
converted to one-hot encoding. The raw and derived input features are then fed
to two individual bi-directional sequential layers. The information extracted
from these separate layers is then merged and used as input to a dense layer
followed by a final feed forward layer with $Linear$ activation function.

##  IV The Datasets

The proposed methodology for SLF has been evaluated on five energy load
datasets from different sources. This section provides the salient parameters
of the dataset and presents the pre-processing technique adopted for each.

###  IV-A Smart Grid Smart City (SGSC) dataset

SGSC project was initiated by the Australian Government in year 2010 [14]. It
gathered smart-meter data from around 78,000 customers for a period of
4-years. In [5], individual models for each customer was proposed. However
since it is not feasible to train individual models for $\sim$78,000
customers, therefore 69 customers having “hot water system” were selected. The
same subset is extracted here to evaluate the DeepDeFF architecture.

###  IV-B The Almanac of Minutely Power dataset (AMPds)

AMPds [26] contains electricity, water and natural gas measurements of a
single Canadian household with 19 appliances, recorded for 1 year with 1
minute resolution, which is down-sampled to 30 minutes resolution [10]. The
variables for raw features used here are the same as for SGSC except that $E$
here is assigned to the Ampere reading.

###  IV-C Réseau de Transport d’Électricité (RTE) France dataset

RTE dataset [13] is also used here to evaluate the proposed technique. The
dataset used spans from year 2013 to 2016 with the sampling interval of 30
minutes. The raw inputs are programmed with same variables as for SGSC above.

###  IV-D The Electric Reliability Council of Texas (ERCOT) dataset

ERCOT dataset [18] provides real time and historical statistics surrounding
independent system operator (ISO) operations of the Texas region for a period
of $\sim$5 years recorded every 1 hour. The raw features variables used here
are the same as for SGSC except that the time $T$ here ranges 1-24 since the
resolution is 1 hour.

###  IV-E Pakistan Residential Electricity Consumption (PRECON)

PRECON dataset [27] records the electricity consumption patterns in a
developing country for 42 households of varying financial status, daily
routine and load profile. The data is collected with 1 minute interval from
01-06-2018 to 31-09-2019. The amount of data varies for each household due to
different number and types of appliances that are selected for monitoring.
This dataset also captures the problem of power outages rampant in developing
countries. This is evident from several long 0KW data intervals. For raw
features, same variables as in SGSC are used here except that $E$ here refers
to the KW usage.

##  V Experiments & Results

The proposed framework for SLF is achieved through an evolutionary process
after numerous rigorous experiments on all five datasets. This section
discusses these experiments in sufficient detail and infers the results
obtained. The results from the DeepDeFF architecture are compared with the
results of simple two layer sequential models trained on basic features and
MAE as loss as proposed in [5].

###  V-A SGSC Dataset

####  V-A1 Train & Test Setting

The same settings provided in [5] are used to extract the subset of SGSC data
for fair comparison on the same test set. The data spanning the whole winter
season of New South Wales Australia is subdivided into a split ratio of
0.7/0.2/0.1 as:

  1. (a)

Training set (01-Jun-2013 to 05-Aug-2013)

  2. (b)

Validation set (06-Aug-2013 to 22-Aug-2013)

  3. (c)

Test set (23-Aug-2013 to 31-Aug-2013)

The first set is to train the DeepDeFF model, validation set is used to select
the best model weights based on performance on validation set, while the test
set is for the evaluation of the DeepDeFF model. The data is spaced between 30
minutes interval; so for 69 customers the 9 days of evaluation implies the
forecasting of 29,808 time points.

(a) Similar train and test pattern for customer 8804804

(b) Dissimilar train and test pattern for customer 8655993

Figure 2: A comparison of train and test load data for two types of customers
from SGSC dataset is shown, (a) shows a customer whose train and test data
patterns have high correlation, (b) shows a customer who has very different
load patterns for train and test data.

Fig. 2 provides some insight into the diversity of customers by showing the
similarity between their train and test data. Test data is plotted over
training data with matching numeric dates. Fig. 2(a) shows the data of a
customer with similarity between train and test data patterns, whereas Fig.
2(b) shows no similarity for another customer. This indeed effects the results
of DeepDeFF architecture which is reflected in their respective MAPE of 26.04%
and 50.78% using BLSTM layer; thus the DeepDeFF model has been able to learn
the underlying patterns and temporal relations for Fig. 2(a) but not as good
for Fig. 2(b).

####  V-A2 Results

Table-I shows the comparison of results from rigorous experiments that are
performed on SGSC dataset using the proposed DeepDeFF method in contrast with
the implementation of the model proposed in [5]. The addition of derived
features in the proposed architecture along with MAPE as loss function,
outperforms the state of the art on the SGSC dataset as evident from the
average MAPE computed in Table-I.

TABLE I: Results achieved on SGSC dataset Method |  Time-steps |  Avg.MAPE %
---|---|---
|  |  DeepDeFF |  Basic
BGRU | 2 | 34.87 | 42.83
GRU | 2 | 35.01 | 42.85
BLSTM | 2 | 35.40 | 43.04
LSTM | 2 | 36.88 | 43.21
RNN | 2 | 35.94 | 42.04
BRNN | 2 | 35.79 | 42.74
BGRU | 6 | 36.02 | 42.08
GRU | 6 | 35.72 | 41.78
BLSTM | 6 | 36.41 | 42.96
LSTM | 6 | 37.28 | 42.41
RNN | 6 | 38.97 | 42.28
BRNN | 6 | 38.60 | 42.61
BGRU | 12 | 36.48 | 42.40
GRU | 12 | 36.34 | 41.54
BLSTM | 12 | 37.74 | 43.88
LSTM | 12 | 38.09 | 42.46
RNN | 12 | 40.57 | 43.29
BRNN | 12 | 39.64 | 43.20

(a) Predicted vs. actual data for customer 8804804

(b) Predicted vs. actual data for customer 8655993

Figure 3: Actual load versus load predicted by DeepDeFF architecture utilizing
different sequential layers for the two customers shown in Fig.2

Fig. 3(a) confirms that the DeepDeFF model indeed predicts the actual load
very well for customer 8804804. However, Fig. 3(b) shows that the model under
performs for customer 8655993 due to uncorrelated train and test data, owing
to disjoint customer behavior during training and testing days.

###  V-B AMPds

####  V-B1 Train & Test Setting

The AMPds data is converted from 1 minute resolution to 30 minutes, yielding
17,483 data points [10]. The data is subdivided with a split ratio of
0.7/0.2/0.1 into:

  1. (a)

Training set (01-Apr-2012 to 17-Dec-2012)

  2. (b)

Validation set (18-Dec-2012 to 23-Feb-2013)

  3. (c)

Test set (24-Feb-2013 to 01-Apr-2013)

(a) Same dates of a month

(b) Same week days of a month, nearest to the test dates

Figure 4: A comparison of train and test data, AMPds

Fig. 4 shows that the train and test data have some common pattern and that
there are no abrupt changes like in Fig.2(b). Even though, the training and
testing data is not available for same dates of different years, there is a
general trend that is being followed in the test data.

####  V-B2 Results

Table-II shows a comparison of the results produced by the simple two layer
sequential model and the DeepDeFF architecture with derived features. The
proposed architecture beats the benchmark of 26.23% achieved in [10] for 6
time-steps.

TABLE II: Results achieved on AMP dataset Method |  Time-steps |  Avg.MAPE %
---|---|---
|  |  DeepDeFF |  Basic
BGRU | 2 | 25.08 | 28.17
GRU | 2 | 25.48 | 27.65
BLSTM | 2 | 25.44 | 28.14
LSTM | 2 | 25.57 | 28.57
BRNN | 2 | 24.77 | 26.88
RNN | 2 | 25.62 | 26.26
BGRU | 6 | 24.64 | 31.58
GRU | 6 | 25.33 | 30.49
BLSTM | 6 | 25.98 | 32.32
LSTM | 6 | 26.00 | 33.47
BRNN | 6 | 25.47 | 30.10
RNN | 6 | 25.32 | 27.97
BGRU | 12 | 25.19 | 32.85
GRU | 12 | 25.17 | 30.00
BLSTM | 12 | 26.62 | 36.08
LSTM | 12 | 25.81 | 30.65
BRNN | 12 | 25.35 | 31.62
RNN | 12 | 25.87 | 29.54
Figure 5: Actual load versus load predicted by DeepDeFF architecture utilizing
different sequential layers.

Fig. 5 shows that the DeepDeFF architecture performs well in predicting the
general load and suffers in case of outliers. This is because the model was
able to learn the underlying general pattern from the training data, and gave
it more importance than to outliers. This problem occurred because the
training data was not enough and does not cover all the months; so the test
data is of a month that was never seen during training.

###  V-C RTE Dataset

####  V-C1 Train & Test Setting

RTE data is subdivided into three subsets with a split ratio of 0.7/0.2/0.1
as:

  1. (a)

Training set (01-Jan-2013 to 18-Nov-2015)

  2. (b)

Validation set (19-Nov-2015 to 07-Aug-2016)

  3. (c)

Test set (08-Aug-2016 to 31-Dec-2016)

Figure 6: Training and testing data comparison, RTE dataset

Fig. 6 show the subsets of training and testing data for the dates mentioned
in the figures’ legends. Such close resemblance in the test and train data
helps the model to make accurate predictions as evident from the results.

####  V-C2 Results

It is observed from the results for SGSC and AMPds datasets that the
experiments with 2 time-steps mostly yield the best results. Henceforth 2 time
steps is used for the experiments on other datasets. Table-III shows the
results for RTE dataset. The proposed model with GRU and derived features
performed best with average MAPE of 0.81%. Fig. 7 shows the prediction results
against the actual system load which further confirms the excellent
performance of the DeepDeFF architecture.

TABLE III: Results achieved on RTE dataset Method |  Time-steps |  Avg.MAPE %
---|---|---
|  |  DeepDeFF |  Basic
BGRU | 2 | 0.84 | 1.06
GRU | 2 | 0.81 | 1.09
BLSTM | 2 | 1.63 | 1.04
LSTM | 2 | 1.17 | 1.19
BRNN | 2 | 1.26 | 1.13
RNN | 2 | 1.39 | 1.14
Figure 7: Actual load versus load predicted by DeepDeFF architecture utilizing
different sequential layers.

###  V-D ERCOT Dataset

####  V-D1 Train & Test Setting

ERCOT data is subdivided into three subsets with a split ratio of 0.5/0.1/0.4
as:

  1. (a)

Training set (01-Jan-2011 to 26-May-2013)

  2. (b)

Validation set (27-May-2013 to 31-Dec-2013)

  3. (c)

Test set (01-Jan-2014 to 31-Dec-2015)

Similar to RTE, ERCOT is also the accumulated load consumption data of Texas.
The train and test data for ERCOT also has close resemblance similar to Fig. 6

####  V-D2 Results

Table-IV shows the results for ERCOT dataset where the DeepDeFF model with
BGRU performed best with average MAPE of 0.91%. Figure-8 shows the results
that establishes the effectiveness of the DeepDeFF architecture.

TABLE IV: Results achieved on ERCOT dataset Method |  Time-steps |  Avg.MAPE %
---|---|---
|  |  DeepDeFF |  Basic
BGRU | 2 | 0.91 | 1.40
GRU | 2 | 1.38 | 1.76
BLSTM | 2 | 0.98 | 2.58
LSTM | 2 | 1.17 | 1.96
BRNN | 2 | 0.92 | 1.37
RNN | 2 | 1.38 | 2.88
Figure 8: Actual load versus load predicted by DeepDeFF architecture utilizing
different sequential layers.

###  V-E PRECON

####  V-E1 Train & Test Setting

Owing to the peculiar nature of the PRECON dataset, it is pre-processed in two
steps in this research. First, the data is converted from 1-minute interval to
30-minute intervals by taking the average over the 30 consecutive load
readings. The second step is to take care of close to zero values in the data
that are mostly due to power outages. Otherwise these values cause divide-by-
zero problem when using MAPE function for evaluation, resulting into
unrealistically high MAPE and adversely effecting the performance of the
machine learning algorithm. This is countered simply by adding a small offset
of 0.1 KW to all the readings. The offset is small enough to makes no
significant change in the nominal values of the load and takes effect only for
the near zero data. This simplest pre-processing has shown remarkable impact
on the performance of the DeepDeFF algorithm as evident from the results.

The data splitting is done in a unique way here due to the reason that it
spanned over a period of only one year with no repeated data for any month. So
instead of using an overall split of data, as done in previous datasets, a
month-wise split is proposed. Here the training, validation, and testing data
is taken from days 1 – 21, 22 – 26, 27 – 30/31 respectively for each month.
This corresponds roughly to an overall split of 0.7/0.2/0.1.

####  V-E2 Results

Table V shows the comparison of results obtained for PRECON dataset. DeepDeFF
models have consistently outperformed the basic models on all the houses,
achieving on average 8.9% lower MAPE than basic models.

The value of the MAPE achieved by DeepDeFF models ranged from 7.67% on House 3
to 37.61% on House 29. The graphs of predicted versus actual load of these two
houses are shown in Fig. 9(a) and Fig. 9(b) respectively.

TABLE V: Results achieved on PRECON dataset Method |  Time-steps |  Avg.MAPE %
---|---|---
|  |  DeepDeFF |  Basic
BGRU | 2 | 21.87 | 24.01
GRU | 2 | 22.1 | 24.04
BLSTM | 2 | 22 | 24.18
LSTM | 2 | 22.18 | 24.31
BRNN | 2 | 21.67 | 23.9
RNN | 2 | 21.89 | 24.11

(a) Prediction of DeepDeFF models vs actual load for House 3

(b) Prediction of DeepDeFF models vs actual load for House 29

Figure 9: Actual load versus load predicted by DeepDeFF architecture utilizing
different sequential layers.

The above mentioned results on five public datasets infer that SLF for
individual households: SGSC, AMPDs and PRECON is more difficult than
aggregated load forecasting of a country or state wide dataset because of high
variances in load consumption patterns of the former ones. However the
proposed DeepDeFF architecture has been able to forecast better than the
previously published techniques.

##  VI Conclusion

Load forecasting is of critical importance to optimally schedule and reliably
manage the operations of power systems. This manuscript presented a deep
learning architecture based on sequential layers, and a pre-processing method
for introducing hand-crafted features into the end-to-end learning pipeline of
the deep learning model, for short-term load forecasting. It is demonstrated
with rigorous experimentation that the inclusion of hand-crafted features has
improved the learning and predictions of the model, specially for smaller
datasets. The proposed DeepDeFF architecture has been comprehensively tested
on five different datasets – two country/state wide datasets and three
household datasets. The results achieved from the proposed methodology beat
the current benchmark of these datasets for SLF.

## References

  * [1] P. B. Luh, L. D. Michel, P. Friedland, C. Guan, and Yuting Wang, “Load forecasting and demand response,” in _IEEE PES General Meeting_ , 2010, pp. 1–3.
  * [2] T. Hong, J. Wilson, and J. Xie, “Long term probabilistic load forecasting and normalization with hourly information,” _IEEE Transactions on Smart Grid_ , vol. 5, no. 1, pp. 456–462, Jan 2014.
  * [3] H. Al-Hamadi and S. Soliman, “Long-term/mid-term electric load forecasting based on short-term correlation and annual growth,” _Electric Power Systems Research_ , vol. 74, no. 3, pp. 353 – 361, 2005. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0378779605000817
  * [4] B. Hodge, D. Lew, and M. Milligan, “Short-term load forecast error distributions and implications for renewable integration studies,” in _2013 IEEE Green Technologies Conference (GreenTech)_ , April 2013, pp. 435–442.
  * [5] W. Kong, Z. Y. Dong, Y. Jia, D. J. Hill, Y. Xu, and Y. Zhang, “Short-term residential load forecasting based on LSTM recurrent neural network,” _IEEE Transactions on Smart Grid_ , vol. 10, no. 1, pp. 841–851, Jan 2019.
  * [6] P. Santos, A. Martins, and A. Pires, “Designing the input vector to ANN-based models for short-term load forecast in electricity distribution systems,” _International Journal of Electrical Power & Energy Systems_, vol. 29, no. 4, pp. 338 – 347, 2007. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0142061506001670
  * [7] V. H. Ferreira and A. P. A. da Silva, “Toward estimating autonomous neural network-based electric load forecasters,” _IEEE Transactions on Power Systems_ , vol. 22, no. 4, pp. 1554–1562, Nov 2007.
  * [8] I. Drezga and S. Rahman, “Input variable selection for ANN-based short-term load forecasting,” _IEEE Transactions on Power Systems_ , vol. 13, no. 4, pp. 1238–1244, Nov 1998.
  * [9] N. Ding, C. Benoit, G. Foggia, Y. Bésanger, and F. Wurtz, “Neural network-based model design for short-term load forecast in distribution systems,” _IEEE Transactions on Power Systems_ , vol. 31, no. 1, pp. 72–81, Jan 2016.
  * [10] W. Kong, Z. Y. Dong, D. J. Hill, F. Luo, and Y. Xu, “Short-term residential load forecasting based on resident behaviour learning,” _IEEE Transactions on Power Systems_ , vol. 33, no. 1, pp. 1087–1088, Jan 2018.
  * [11] W. Kong, Z. Y. Dong, F. Luo, K. Meng, W. Zhang, F. Wang, and X. Zhao, “Effect of automatic hyperparameter tuning for residential load forecasting via deep learning,” in _2017 Australasian Universities Power Engineering Conference (AUPEC)_ , 2017, pp. 1–6.
  * [12] P. Theile, A. Towle, K. Karnataki, A. Crosara, K. Paridari, G. Turk, and L. Nordstrom, “Day-ahead electricity consumption prediction of a population of households: analyzing different machine learning techniques based on real data from RTE in France,” in _2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)_ , Oct 2018, pp. 1–6.
  * [13] _RTE, Grid data._ , 2019 (Accessed online:27.08.2019), https://data.rte-france.com/.
  * [14] _Smart-Grid Smart-City Customer Trial Data_ , 2019 (Accessed online:10.09.2019), https://data.gov.au/data/dataset/4e21dea3-9b87-4610-94c7-15a8a77907ef.
  * [15] L. Han, Y. Peng, Y. Li, B. Yong, Q. Zhou, and L. Shu, “Enhanced deep networks for short-term and medium-term load forecasting,” _IEEE Access_ , vol. 7, pp. 4045–4055, Jan 2019.
  * [16] D. Bissing, M. T. Klein, R. A. Chinnathambi, D. F. Selvaraj, and P. Ranganathan, “A hybrid regression model for day-ahead energy price forecasting,” _IEEE Access_ , vol. 7, pp. 36 833–36 842, Mar 2019.
  * [17] M. Ziming, H. Zhong, X. Le, X. Qing, and K. Chongqing, “Month ahead average daily electricity price profile forecasting based on a hybrid nonlinear regression and SVM model: an ERCOT case study,” _Journal of Modern Power Systems and Clean Energy_ , vol. 6, no. 2, pp. 281–291, 2018.
  * [18] _ERCOT, Grid data._ , 2019 (Accessed online:27.08.2019), https://ercot.com/.
  * [19] J. Cui, Q. Gao, and D. Li, “Improved long short-term memory network based short term load forecasting,” in _2019 Chinese Automation Congress (CAC)_ , 2019, pp. 4428–4433.
  * [20] Y. Ma, Q. Zhang, J. Ding, Q. Wang, and J. Ma, “Short term load forecasting based on iforest-lstm,” in _2019 14th IEEE Conference on Industrial Electronics and Applications (ICIEA)_ , 2019, pp. 2278–2282.
  * [21] A. O. Hoori, A. A. Kazzaz, R. Khimani, Y. Motai, and A. J. Aved, “Electric load forecasting model using a multicolumn deep neural networks,” _IEEE Transactions on Industrial Electronics_ , vol. 67, no. 8, pp. 6473–6482, 2020.
  * [22] X. Tang, Y. Dai, Q. Liu, X. Dang, and J. Xu, “Application of bidirectional recurrent neural network combined with deep belief network in short-term load forecasting,” _IEEE Access_ , vol. 7, pp. 160 660–160 670, 2019.
  * [23] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Comput._ , vol. 9, no. 8, pp. 1735–1780, Nov. 1997. [Online]. Available: http://dx.doi.org/10.1162/neco.1997.9.8.1735
  * [24] M. Schuster and K. K. Paliwal, “Bidirectional recurrent neural networks,” _IEEE Transactions on Signal Processing_ , vol. 45, no. 11, pp. 2673–2681, Nov 1997.
  * [25] A. Graves and J. Schmidhuber, “Framewise phoneme classification with bidirectional LSTM and other neural network architectures,” _Neural Networks_ , vol. 18, no. 5, pp. 602 – 610, 2005, iJCNN 2005. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0893608005001206
  * [26] S. Makonin, F. Popowich, L. Bartram, B. Gill, and I. V. Bajić, “AMPds: A public dataset for load disaggregation and eco-feedback research,” in _2013 IEEE Electrical Power Energy Conference_ , Aug 2013, pp. 1–6.
  * [27] A. Nadeem and N. Arshad, “PRECON: Pakistan residential electricity consumption dataset,” in _Proceedings of the Tenth ACM International Conference on Future Energy Systems_ , ser. e-Energy ’19. New York, NY, USA: ACM, 2019, pp. 52–57. [Online]. Available: http://doi.acm.org/10.1145/3307772.3328317