# PatchUp: A Regularization Technique for Convolutional Neural Networks

Mojtaba Faramarzi1,2, Mohammad Amini1,3, Akilesh Badrinaaraayanan1,2, Vikas
Verma1,2,4

Sarath Chandar1,5,6

1Mila - Quebec AI Institute, 2University of Montreal, 3McGill University,
4Aalto Univeristy, Finland,
5École Polytechnique de Montréal, 6Canada CIFAR AI Chair

###### Abstract

Large capacity deep learning models are often prone to a high generalization
gap when trained with a limited amount of labeled training data. A recent
class of methods to address this problem uses various ways to construct a new
training sample by mixing a pair (or more) of training samples. We propose
PatchUp, a hidden state block-level regularization technique for Convolutional
Neural Networks (CNNs), that is applied on selected contiguous blocks of
feature maps from a random pair of samples. Our approach improves the
robustness of CNN models against the manifold intrusion problem that may occur
in other state-of-the-art mixing approaches like Mixup and CutMix. Moreover,
since we are mixing the contiguous block of features in the hidden space,
which has more dimensions than the input space, we obtain more diverse samples
for training towards different dimensions. Our experiments on CIFAR-10,
CIFAR-100, and SVHN datasets with PreactResnet18, PreactResnet34, and
WideResnet-28-10 models show that PatchUp improves upon, or equals, the
performance of current state-of-the-art regularizers for CNNs. We also show
that PatchUp can provide better generalization to affine transformations of
samples and is more robust against adversarial attacks.

##  1 Introduction

Deep Learning (DL), particularly deep Convolutional Neural Networks (CNNs)
have achieved exceptional performance in many machine learning tasks,
including object recognition [1], image classification [1, 2, 3], speech
recognition [4] and natural language understanding [5, 6]. However, in a very
deep and wide network, the network has a tendency to memorize the samples,
which yields poor generalization for data outside of the training data
distribution [7, 8]. To address this issue, noisy computation is often
employed during the training, making the model more robust against invariant
samples and thus improving the generalization of the model [9]. This idea is
exploited in several state-of-the-art regularization techniques.

Such noisy computation based regularization techniques can be categorized into
data-dependent and data-independent techniques [10]. Earlier work in this area
has been more focused on the data-independent techniques such as Dropout [11],
SpatialDropout [12], and DropBlock [13]. Dropout performs well on fully
connected layers [14]. However, it is less effective on convolutional layers
[15]. One of the reasons for the lack of success of dropout on CNN layers is
perhaps that the activation units in the convolutional layers are correlated,
thus despite dropping some of the activation units, information can still flow
through these layers. SpatialDropout [15] addresses this issue by dropping the
entire feature map from a convolutional layer. DropBlock [13] further improves
SpatialDropout by dropping random continuous feature blocks from feature maps
instead of dropping the entire feature map in the convolutional layers.

Recent works show that data-dependent regularizers can achieve better
generalization for CNN models. Mixup [16], one such data-dependent
regularizer, synthesizes additional training examples by interpolating random
pairs of inputs $x_{i}$, $x_{j}$ and their corresponding labels $y_{i}$,
$y_{j}$ as:

$\begin{array}[]{ll}{\tilde{x}=\lambda x_{i}+(1-\lambda)x_{j}}\text{\quad and
\quad}{\tilde{y}=\lambda y_{i}+(1-\lambda)y_{j}}\end{array}$ (1)

where $\lambda\in[0,1]$ is sampled from a Beta distribution such that
$\lambda\sim Beta(\alpha,\alpha)$ and ($\tilde{x},\;\tilde{y}$) is the new
example. By using these types of synthetic samples, Mixup encourages the model
to behave linearly in-between the training samples.

The mixing coefficient $\lambda$ in Mixup is sampled from a prior
distribution. This may lead to the manifold intrusion problem [10]: the mixed
synthetic example may collide (i.e. have the same value in the input space)
with other examples in the training data, essentially leading to two training
samples which have the same inputs but different targets. To overcome the
manifold intrusion problem, Mai et al. [17] used a meta-learning approach to
learn $\lambda$ with a lower possibility of causing such collisions. However,
this meta-learning approach adds significant computation complexity.
ManifoldMixup [18] attempts to avoid the manifold intrusion problem by
interpolating the hidden states (instead of input states) of a randomly chosen
layer at every training update.

Different from the interpolation based regularizers discussed above, cutout
[19] drops the contiguous regions from the image in the input space. This kind
of noise encourages the network to learn the full context of the images
instead of overfitting to the small set of visual features. CutMix [20] is
another data-dependent regularization technique that cuts and fills
rectangular shape parts from two randomly selected pairs in a mini-batch
instead of interpolating two selected pairs completely. Applying CutMix at the
input space improves the generalization of the CNN model by spreading the
focus of the model across all places in the input instead of just a small
region or a small set of intermediate activations. CutMix also improves the
generalization performance of a very deep and wide CNN model such as
PyramidNet. According to the CutMix paper, applying CutMix at the latent
space, Feature CutMix, is not as effective as applying CutMix in the input
space [20].

In this work, we propose PatchUp which is a regularization technique that
operates in the hidden space by masking out contiguous blocks of the feature
map of a random pair of samples, and then either mixes (Soft PatchUp) or swaps
(Hard PatchUp) these selected contiguous blocks. Our experiments verify that
Hard PatchUp achieves a better generalization performance compared to other
state-of-the-art regularization techniques for CNNs such as Mixup, cutout,
CutMix and ManifoldMixup on CIFAR-10, CIFAR-100, and SVHN datasets. Soft
PatchUp achieves the second-best performance on CIFAR-10, CIFAR-100 with
PreactResnet18, PreactResnet34, and WideResnet-28-10 models while achieving
comparable results to ManifoldMixup on SVHN with PreactResnet18 and
PreactResnet34. Furthermore, PatchUp provides significant improvements in the
generalization on deformed images and better robustness against Fast Gradient
Sign Method (FGSM) adversarial attack.

##  2 PatchUp

PatchUp is a hidden state block-level regularization technique that can be
used after any convolutional layer in CNN models. Given a deep neural network
$f(x)$ where $x$ is the input, let $g_{k}$ be the $k$-th convolutional layer.
The network $f(x)$ can be represented as $f(x)=f_{k}(g_{k}(x))$ where $g_{k}$
is the mapping from the input data to the hidden representation at layer $k$
and $f_{k}$ is the mapping from the hidden representation at layer $k$ to the
output [18]. In every training step, PatchUp applies block-level
regularization at a randomly selected convolutional layer $k$ from a set of
intermediate convolutional layers. Appendix-B gives a formal intuition for
selecting $k$ randomly.

###  2.1 Binary Mask Creation

Figure 1: PatchUp process for two hidden representations associated with two
samples randomly selected in the mini-batch ($a,\;b)$. $X_{1}=g_{k}^{(i)}(a)$
and $X_{2}=g_{k}^{(i)}(b)$ where $i$ is the feature map index. Right top shows
Hard PatchUp output and the right bottom shows the interpolated samples with
Soft PatchUp. The yellow continuous blocks represent the interpolated selected
blocks.

Once a convolutional layer $k$ is chosen, the next step is to create a binary
mask $M$ (of the same size as the feature map in layer $k$) that will be used
to PatchUp a pair of examples in the space of $g_{k}(x)$. The mask creation
process is similar to that of DropBlock [13]. The idea is to select contiguous
blocks of features from the feature map that will be either mixed or swapped
with the same features in another example. To do so, we first select a set of
features that can be altered (mixed or swapped). This is done by using the
hyper-parameter $\gamma$ which decides the probability of altering a feature.
When we alter a feature, we also alter a square block of features centered
around that feature which is controlled by the side length of this square
block, $block\\_size$. Hence, the altering probabilities are readjusted using
the following formula [13]:

$\gamma_{adj}=\frac{\gamma\times\textit{(feature map's
area)}}{\textit{(block's area)}\times\textit{(valid region to build block)}}.$
(2)

where the area of the feature map and block are the $feat\\_size^{2}$ and
$block\\_size^{2}$, respectively, and the valid region to build the block is
$(feat\\_size-block\\_size+1)^{2}$.

For each feature in the feature map, we sample from $Bernoulli(\gamma_{adj})$.
If the result of this sampling for feature $f_{ij}$ is 0, then $M_{ij}=1$. If
the result of this sampling for $f_{ij}$ is 1, then the entire square region
in the mask with the center $M_{ij}$ and the width and height of the square of
$block\\_size$ is set to 0. Note that these feature blocks to be altered can
overlap which will result in more complex block structures than just squares.
The block structures created are called patches. Figure-1 illustrates an
example mask used by PatchUp. The mask $M$ has 1 for features outside the
patches (which are not altered) and 0 for features inside the patches (which
are altered).

###  2.2 PatchUp Operation

Once the mask is created, we can use the mask to select patches from the
feature maps and either swap these patches (Hard PatchUp) or mix them (Soft
PatchUp).

Consider two samples $x_{i}$ and $x_{j}$. The Hard PatchUp operation at layer
$k$ is defined as follows:

$\mathbf{\phi_{hard}}(g_{k}(x_{i}),g_{k}(x_{j}))=\mathbf{M}\odot
g_{k}(x_{i})+(\mathbf{1}-\mathbf{M})\odot g_{k}(x_{j}),$ (3)

where $\odot$ is known as the element-wise multiplication operation and
$\mathbf{M}$ is the binary mask described in section 2.1.

To define Soft PatchUp operation, we first define the mixing operation for any
two vectors $a$ and $b$ as follows:

$\text{Mix}_{\lambda}(a,\;b)=\lambda\cdot a+(1-\lambda)\cdot b,$ (4)

where $\lambda\in[0,1]$ is the mixing coefficient. Thus, the Soft PatchUp
operation at layer $k$ is defined as follows:

$\mathbf{\phi_{soft}}(g_{k}(x_{i}),g_{k}(x_{j}))=\mathbf{M}\odot
g_{k}(x_{i})+\text{Mix}_{\lambda}[((\mathbf{1}-\mathbf{M})\odot
g_{k}(x_{i})),\;((\mathbf{1}-\mathbf{M})\odot g_{k}(x_{j}))].$ (5)

where $\lambda$ in the range of $[0,\;1]$ is sampled from a Beta distribution
such that $\lambda\sim Beta(\alpha,\alpha)$. $\alpha$ controls the shape of
the Beta distribution. Consequently, it controls the strength of interpolation
[16]. Both PatchUp operations are illustrated in Figure 1.

###  2.3 Learning Objective

After applying the PatchUp operation, the CNN model continues the forward pass
from layer $k$ to the last layer in the model. The output of the model is used
for the learning objective, including the loss minimization process and
updating the model parameters accordingly.

Again, consider the example pairs $(x_{i},y_{i})$ and $(x_{j},y_{j})$. Let
$\phi_{k}=\phi(g_{k}(x_{i}),g_{k}(x_{j}))$ be the output of PatchUp after the
$k$-th layer. Mathematically, the CNN with PatchUp minimizes the following
loss function:

$\displaystyle L(f)=$
$\displaystyle\operatorname{\mathbb{E}}_{(x_{i},\;y_{i})\sim
P}\,\operatorname{\mathbb{E}}_{(x_{j},\;y_{j})\sim
P}\,\operatorname{\mathbb{E}}_{\lambda\sim\text{Beta}(\alpha,\;\alpha)}\operatorname{\mathbb{E}}_{k\sim\mathcal{S}}$
(6)
$\displaystyle~{}~{}~{}~{}\text{Mix}_{p_{u}}[\ell(f_{k}(\phi_{k}),\;y_{i})\;,\ell(f_{k}(\phi_{k}),\;Y)]\;+\ell(f_{k}(\phi_{k}),\;W(y_{i},\;y_{j})).$

where $p_{u}$ is the fraction of the unchanged features from feature maps in
$g_{k}(x_{i})$ and $\mathcal{S}$ is the set of layers where PatchUp is applied
randomly. $\phi$ is $\phi_{hard}$ for Hard PatchUp and $\phi_{soft}$ for Soft
PatchUp.

$Y$ is the target corresponding to the changed features. In the case of Hard
PatchUp, $Y=y_{j}$ and in the case of Soft PatchUp,
$Y=\textit{Mix}_{\lambda}(y_{i},y_{j})$. $W(y_{i},y_{j})$ calculates the re-
weighted target according to the interpolation policy for $y_{i}$ and $y_{j}$.
$W$ for Hard PatchUp and Soft PatchUp is defined as follows:

$\displaystyle W_{hard}(y_{i},y_{j})$
$\displaystyle=\text{Mix}_{p_{u}}(y_{i},y_{j}).$ (7) $\displaystyle
W_{soft}(y_{i},y_{j})$
$\displaystyle=\text{Mix}_{p_{u}}(y_{i},\;\text{Mix}_{\lambda}(y_{i},y_{j})).$
(8)

The PatchUp loss function has two terms where the first term is inspired from
the CutMix loss function and the second term is inspired from the MixUp loss
function.

###  2.4 PatchUp in Input Space

When $k=0$, PatchUp only gets applied to the input space. To apply PatchUp to
the input space, only the Hard PatchUp operation is used since swapping in the
input space provides better generalization compared to mixing [20]. Only one
random rectangular patch is selected in the input space (similar to CutMix)
because the PatchUp binary mask is too strong for the input space, which has
only three channels, compared to hidden layers in which each layer has
numerous channels.

##  3 Relation to Other Methods

Figure 2: Left: ManifoldMixup interpolated samples for any combination of the
three blue hidden states selected only from along orange line. Right: PatchUp
can produce interpolated hidden representations for these three hidden states
in almost all possible places in all dimensions except the samples which lie
directly on the orange lines.

Figure 3: The two possible block selections from CutMix for two samples (cat
and dog) with a large background. Swapping a similar part of the background or
an essential element correlated to the label in the selected images can have a
negative effect on the CutMix learning objective.

####  PatchUp Vs. ManifoldMixup:

Both PatchUp and ManifoldMixup try to improve the generalization of a model by
combining the latent representations of a pair of examples. ManifoldMixup
combines two hidden representations by using the mixing operation defined in
Equation 4 which produces a new latent representation in a linear way for a
pair of two hidden representations. PatchUp uses a more complex approach to
find a combination of two hidden representations, ensuring that a more diverse
subspace of the hidden space gets explored. To understand the behaviour and
the limitation that exist in the ManifoldMixup, assume that we have a 3D
hidden space representation as illustrated in figure 3. Figure 3 presents the
possible combinations of hidden representations explored via ManifoldMixup and
PatchUp. Blue dots represent real hidden representation samples. ManifoldMixup
can produce new samples that lie directly on the orange lines which connect
the blue point pairs due to its linear interpolation strategy. On the other
hand, PatchUp can select various points in all dimensions, and can also select
points extremely close to the orange lines. The proximity to the orange lines
depends on the selected pairs and $\lambda$ sampled from the beta
distribution. Appendix-C provides the mathematical and experimental
justifications to show the comparative advantage of PatchUp over
ManifoldMixup.

####  PatchUp Vs. CutMix:

The CutMix strategy is to cut and fill some parts of the selected pairs
instead of using interpolation for creating a new sample in the input space.
Therefore, the CutMix method has less potential for a manifold intrusion
problem, however, CutMix may still suffer from a manifold intrusion problem.
Figure 3 shows two samples with small portions that correspond to their
labels. CutMix cuts and fills the rectangular parts of the selected image
randomly. In this example, if only the parts within the yellow bounding boxes
are swapped, then the label does not change. However, if the parts within the
white bounding boxes are swapped, then the entire label is swapped. In both
scenarios, CutMix only learns the interpolated target based on the fraction of
the images that is swapped. In contrast, these scenarios cannot occur in
PatchUp since it works in the hidden representation space. Another difference
between CutMix and PatchUp is how the masks are created. PatchUp can create
arbitrarily shaped masks while CutMix masks can only be rectangular. Figure
A.6 shows an example of CutMix Mask and PatchUp mask in input space and hidden
representation space, respectively.

Feature-CutMix applies CutMix in the latent space. According to Yun et al.
[20], Feature-CutMix is not as effective as CutMix. Both the learning
objective of PatchUp, as well as the binary mask selection are different from
Feature-CutMix.

##  4 Experiments

To evaluate the generalization improvements that PatchUp can provide with
either Hard or Soft PatchUp, we applied PatchUp to image classification tasks
on CIFAR-10, CIFAR-100 [21], and SVHN [22] datasets with PreActResNet18,
PreActResNet34, and WideResNet-28-10 models111The code to reproduce all the
results is available at https://github.com/chandar-lab/PatchUp.. We used the
same set of base hyper-parameters for all the models to be able to compare and
evaluate the generalization improvements due to different regularization
methods in a fair way. Appendix-D explains the experiment setup and the hyper-
parameter tuning processes in detail. PatchUp adds $patchup\\_prob$, $\gamma$
and $block\\_size$ to the set of hyper-parameters. $patchup\\_prob$ is the
probability that PatchUp is performed for a given mini-batch. We set $\alpha$
to $2$ in PatchUp. And, based on the hyper-parameter tuning described in
Appendix-D, Hard PatchUp yields the best performance with $patchup\\_prob$,
$\gamma$, and $block\\_size$ as $0.7$, $0.5$, and $7$, respectively. Soft
PatchUp achieves the best performance with $patchup\\_prob$, $\gamma$, and
$block\\_size$ as $1.0$, $0.75$, and $7$, respectively.

Table 1: Image classification task error rates on CIFAR-10 and CIFAR-100. We
run experiments five times to report the mean and the standard deviation of
errors and neg-log-likelihoods. Best performance result is shown in bold,
second best is underlined. The lower number is better.

PreActResNet18 |  Test Error (%) |  Test NLL |  | PreActResNet18 |  Test Error (%) |  Test NLL
---|---|---|---|---|---|---
No Mixup | $4.800\pm 0.135$ | $0.184\pm 0.004$ |  |  No Mixup | $24.622\pm 0.358$ | $1.062\pm 0.017$
Input Mixup ($\alpha$ = 1) | $3.628\pm 0.201$ | $0.192\pm 0.012$ |  |  Input Mixup ($\alpha$ = 1) | $22.326\pm 0.323$ | $1.011\pm 0.012$
ManifoldMixup ($\alpha$ = 1.5) | $3.388\pm 0.048$ | $0.147\pm 0.016$ |  |  ManifoldMixup ($\alpha$ = 1.5) | $21.396\pm 0.384$ | $0.931\pm 0.008$
Cutout | $4.218\pm 0.046$ | $0.158\pm 0.005$ |  |  Cutout | $23.386\pm 0.185$ | $1.004\pm 0.004$
DropBlock | $5.038\pm 0.147$ | $0.185\pm 0.005$ |  |  DropBlock | $25.022\pm 0.259$ | $1.067\pm 0.016$
CutMix | $3.518\pm 0.898$ | $0.131\pm 0.002$ |  |  CutMix | $22.184\pm 0.176$ | $0.949\pm 0.012$
Soft PatchUp | $2.956\pm 0.119$ | $0.169\pm 0.031$ |  |  Soft PatchUp | $19.950\pm 0.180$ | $0.833\pm 0.005$
Hard PatchUp | $\mathbf{2.918\pm 0.131}$ | $0.146\pm 0.718$ |  |  Hard PatchUp | $\mathbf{19.120\pm 0.172}$ | $0.748\pm 0.013$
PreActResNet34 |  |  |  |  PreActResNet34 |  |
No Mixup | $4.640\pm 0.099$ | $0.204\pm 0.004$ |  |  No Mixup | $23.342\pm 0.269$ | $1.103\pm 0.006$
Input Mixup ($\alpha$ = 1) | $3.260\pm 0.075$ | $0.175\pm 0.004$ |  |  Input Mixup ($\alpha$ = 1) | $21.000\pm 0.440$ | $0.950\pm 0.019$
ManifoldMixup ($\alpha$ = 1.5) | $2.926\pm 0.062$ | $0.124\pm 0.004$ |  |  ManifoldMixup ($\alpha$ = 1.5) | $18.724\pm 0.305$ | $0.810\pm 0.008$
Cutout | $3.690\pm 0.141$ | $0.150\pm 0.012$ |  |  Cutout | $22.420\pm 0.075$ | $1.043\pm 0.001$
DropBlock | $4.950\pm 0.188$ | $0.221\pm 0.010$ |  |  DropBlock | $23.744\pm 0.125$ | $1.113\pm 0.007$
CutMix | $3.332\pm 0.071$ | $0.142\pm 0.004$ |  |  CutMix | $19.944\pm 0.141$ | $0.907\pm 0.008$
Soft PatchUp | $2.570\pm 0.062$ | $0.108\pm 0.005$ |  |  Soft PatchUp | $18.630\pm 0.153$ | $0.816\pm 0.016$
Hard PatchUp | $\mathbf{2.534\pm 0.048}$ | $0.108\pm 0.005$ |  |  Hard PatchUp | $\mathbf{17.692\pm 0.125}$ | $0.758\pm 0.016$
WideResNet-28-10 |  |  |  |  WideResNet-28-10 |  |
No Mixup | $4.244\pm 0.142$ | $0.162\pm 0.011$ |  |  No Mixup | $22.442\pm 0.226$ | $1.065\pm 0.010$
Input Mixup ($\alpha$ = 1) | $3.272\pm 0.353$ | $0.191\pm 0.018$ |  |  Input Mixup ($\alpha$ = 1) | $18.726\pm 0.149$ | $0.854\pm 0.013$
ManifoldMixup ($\alpha$ = 1.5) | $3.252\pm 0.183$ | $0.155\pm 0.034$ |  |  ManifoldMixup ($\alpha$ = 1.5) | $18.352\pm 0.378$ | $0.833\pm 0.023$
Cutout | $3.134\pm 0.119$ | $0.122\pm 0.005$ |  |  Cutout | $20.164\pm 0.351$ | $0.931\pm 0.016$
DropBlock | $4.182\pm 0.069$ | $0.157\pm 0.003$ |  |  DropBlock | $22.364\pm 0.149$ | $1.049\pm 0.013$
CutMix | $3.148\pm 0.118$ | $0.126\pm 0.004$ |  |  CutMix | $18.316\pm 0.185$ | $0.839\pm 0.020$
Soft PatchUp | $2.606\pm 0.0.052$ | $0.132\pm 0.029$ |  |  Soft PatchUp | $16.726\pm 0.110$ | $0.722\pm 0.017$
Hard PatchUp | $\mathbf{2.528\pm 0.095}$ | $0.114\pm 0.014$ |  |  Hard PatchUp | $\mathbf{16.134\pm 0.197}$ | $0.660\pm 0.017$
Comparison on CIFAR-10 |  |  Comparison on CIFAR-100

Table 2: Error rates comparison on SVHN. We run experiments five times to
report the mean and the standard deviation of errors and neg-log-likelihoods.
Best performance result is shown in bold, second best is underlined. The lower
number is better.

|  PreActResNet18 |  |  PreActResNet34 |  |  WideResnet-28-10
---|---|---|---|---|---
|  Error |  Loss |  |  Error |  Loss |  |  Eror |  Loss
No Mixup | $3.035\pm 0.092$ | $0.138\pm 0.004$ |  | $3.087\pm 0.659$ | $0.164\pm 0.008$ |  | $2.833\pm 0.081$ | $0.137\pm 0.008$
Input Mixup ($\alpha$ = 1) | $2.930\pm 0.221$ | $0.233\pm 0.016$ |  | $2.855\pm 0.096$ | $0.223\pm 0.024$ |  | $2.643\pm 0.161$ | $0.207\pm 0.041$
ManifoldMixup ($\alpha$= 2) | $2.436\pm 0.056$ | $0.157\pm 0.062$ |  | $2.423\pm 0.428$ | $0.146\pm 0.064$ |  | $2.425\pm 0.101$ | $0.157\pm 0.029$
Cutout | $2.794\pm 0.121$ | $0.122\pm 0.010$ |  | $2.654\pm 0.152$ | $0.114\pm 0.007$ |  | $2.475\pm 0.148$ | $0.109\pm 0.009$
DropBlock | $2.961\pm 0.111$ | $0.134\pm 0.005$ |  | $3.101\pm 0.083$ | $0.158\pm 0.006$ |  | $2.732\pm 0.055$ | $0.132\pm 0.002$
CutMix | $3.040\pm 0.054$ | $0.135\pm 0.031$ |  | $2.658\pm 0.049$ | $0.121\pm 0.005$ |  | $2.433\pm 0.045$ | $0.110\pm 0.003$
Soft PatchUp | $2.551\pm 0.056$ | $0.129\pm 0.023$ |  | $2.467\pm 0.081$ | $0.111\pm 0.005$ |  | $\mathbf{2.081\pm 0.066}$ | $0.111\pm 0.010$
Hard PatchUp | $\mathbf{2.286\pm 0.084}$ | $0.107\pm 0.004$ |  | $\mathbf{2.123\pm 0.024}$ | $0.101\pm 0.007$ |  | $\uline{2.088\pm 0.061}$ | $0.105\pm 0.012$

###  4.1 Generalization on Image Classification

Table 1 shows the comparison of the generalization performance of PatchUp with
five other state-of-the-art regularization techniques on the CIFAR-10 and
CIFAR-100 datasets. Our experiments show that Hard PatchUp leads to a lower
test error for all the models on both CIFAR-10 and CIFAR-100. Soft PatchUp has
the second-best result for this task. From Table 1, we see that PatchUp
provides a significant improvement over previous state-of-the-art
regularization techniques across different architectures. We observed that
Soft PatchUp has the second-best performance across all the architectures
indicating the potential that Soft PatchUp is beneficial in fine-grained
classification. Table 2 shows that Hard PatchUp achieves the best top-1 error
across the different models. Our experiments show that ManifoldMixup has the
second-best performance for PreActResNet18 on SVHN. Soft PatchUp also achieves
the second-best performance for WideResNet-28-10 on SVHN. It is also worth
noting that Soft PatchUp performs reasonably well and is comparable to
ManifoldMixup for PreActResNet34 on SVHN.

Table 3: Error rates in the test set on samples subject to affine
transformations for PreActResNet34 trained on CIFAR-100 with indicated
regularization method. We repeated each test for five trained models to report
the mean and the standard deviation of errors. Best performance result is
shown in bold, second best is underlined. The lower number is better.

Transformation |  cutout |  CutMix |  ManifoldMixup |  Soft PatchUp |  Hard PatchUp
---|---|---|---|---|---
Rotate (-20, 20) | $37.448\pm 0.526$ | $35.418\pm 0.328$ | $35.444\pm 0.572$ | $\uline{31.136\pm 0.524}$ | $\mathbf{30.406\pm 0.520}$
Rotate (-40, 40) | $58.752\pm 0.995$ | $57.830\pm 0.586$ | $54.424\pm 0.946$ | $\uline{53.422\pm 0.420}$ | $\mathbf{49.956\pm 0.798}$
Shear (-28.6, 28.6) | $36.552\pm 0.487$ | $34.148\pm 0.473$ | $34.150\pm 0.416$ | $\mathbf{28.984\pm 0.497}$ | $\uline{29.574\pm 0.410}$
Shear (-57.3, 57.3) | $57.736\pm 0.574$ | $53.640\pm 0.587$ | $55.444\pm 0.683$ | $\mathbf{49.102\pm 0.532}$ | $\uline{50.318\pm 0.616}$
Scale (0.6) | $72.994\pm 1.231$ | $54.304\pm 1.268$ | $78.998\pm 1.126$ | $\mathbf{46.246\pm 1.204}$ | $\uline{50.062\pm 2.692}$
Scale (0.8) | $35.092\pm 0.857$ | $29.380\pm 0.577$ | $34.624\pm 0.370$ | $\mathbf{23.942\pm 0.212}$ | $\uline{25.338\pm 0.328}$
Scale (1.2) | $42.310\pm 0.706$ | $49.522\pm 2.035$ | $\uline{41.322\pm 0.638}$ | $43.414\pm 0.652$ | $\mathbf{38.002\pm 0.703}$
Scale (1.4) | $69.404\pm 0.901$ | $78.664\pm 1.854$ | $\mathbf{65.938\pm 0.751}$ | $77.068\pm 1.189$ | $\uline{66.338\pm 1.219}$

###  4.2 Generalization on Deformed Images

Regularization methods aim to improve the generalization of a model to unseen
data. Applying affine transformations on the test set can provide novel
deformed data that can be used to evaluate and compare the minimality and
sufficiency of the representations learned by models with state-of-the-art
regularization techniques [18]. We trained PreActResNet34 and WideResNet-28-10
on the CIFAR100 dataset. And then, we created deformed test sets from CIFAR100
by applying random rotations, random shearings, and different rescalings.
Table 3 shows that PatchUp provides the best performance on affine transformed
test sets and better generalization in PreActResNet34. Table 6 in Appendix-E
illustrates that the quality of representations is improved by PatchUp and it
also shows better generalization in deformed test sets on WideResNet-28-10.
Generalization is significantly improved by PatchUp, as are the quality of
representations learned by PatchUp as demonstrated by this experiment.

(a) Comparison on PreActResNet18 for CIFAR-100.

(b) Comparison on WideResNet28-10 for SVHN.

Figure 4: Robustness to FGSM attack. Plots are based on the mean and standard
deviation of the accuracy of five trained models for each method against FGSM
attack. The x-axis represents $\epsilon$ which is the magnitude that controls
the perturbation.

(a) Comparison on 1st Residual Block.

(b) Comparison on 2nd Residual Block.

Figure 5: The effect of the state-of-the-art regularization techniques on
activations in WideResNet28-10 for CIFAR100 test set on the first and second
Residual Blocks. Each curve is the magnitude of the feature activations,
sorted in descending value, and averaged over all test samples for each
method. The higher magnitude indicates a wider variety of the produced
features by the model at each block.

###  4.3 Robustness to Adversarial Examples

Since neural networks are trained based on Empirical Risk Minimization (ERM),
slight changes in the data distribution have a significant effect on the model
performance [23, 16]. Such unseen data used to confuse the models are known as
adversarial examples. Certain data-dependent regularization techniques can
alleviate such fragility to adversarial examples by training the models with
interpolated data. Therefore, the robustness of a regularized model to
adversarial examples can be considered as a criterion for comparison [16, 18,
20]. To evaluate the robustness of PatchUp against adversarial attacks, we
compared the performance of PreActResNet18, PreActResNet34, and
WideResNet-28-10 on CIFAR-10 and CIFAR-100 with adversarial examples created
by the FGSM attack described in [24]. We compared the performance of
WideResNet28-10 for SVHN against the FGSM attack. Figure 4 shows the
robustness of PatchUp against FGSM attack. Figure A.11 in the appendix shows
further comparisons. Based on the results, we can see that Soft PatchUp is
more robust to adversarial attacks when compared to other regularization
methods. Hard PatchUp and ManifoldMixup achieve the second-best performance in
most experiments. While Hard PatchUp achieves better performance in terms of
classification accuracy, Soft PatchUp seems to trade-off a slight loss of
accuracy in order to achieve more robustness.

###  4.4 Effect on Activations

To study the effect of the state-of-the-art regularization techniques on the
activations in the residual blocks, we compared the mean magnitude of feature
activations in the residual blocks following [19] in WideResNet28-10 for the
test set in CIFAR-100. We first train the models with regularization
techniques and then calculate the magnitudes of activations in the validation
set. The higher mean magnitude of features shows that the models tried to
produce a wider variety of features in the residual blocks [19]. Our
WideResNet28-10 has a conv2d module followed by three residual blocks. For
this ablation study, we selected $k$ randomly such that $k\in\\{0,1,2,3\\}$.
Therefore, we apply the ManifoldMixup and PatchUp in either input space, first
conv2d, first residual block, or second residual block. Figure 5 illustrates
the comparison of ManifoldMixup, cutout, CutMix, Soft PatchUp, and Hard
PatchUp. Figures 5(a) and 5(b) illustrate that PatchUp produces more diverse
features in the layers where we apply PatchUp. In Appendix-G, figure A.12
shows this ablation study results in first conv2d, first residual block,
second residual block and third residual block. Since we are not applying the
PatchUp in the third residual block, the mean magnitude of the feature
activations are below, but very close to, cutout and CutMix. This experiment
also shows that producing a wide variety of features can be an advantage for a
model. However, according to our experiments, a larger magnitude of
activations does not always mean better performance. Figure 5 shows that for
ManifoldMixup, the mean magnitude of the feature activations is less than
other approaches. But, it performs better than cutout and CutMix in image
classification, affine transformations, and FGSM attacks.

###  4.5 Significance of loss terms

PatchUp uses the loss that is introduced in Equation 6. We can paraphrase the
PatchUp learning objective for this ablation study as follow:

$\displaystyle L(f)=$
$\displaystyle\operatorname{\mathbb{E}}_{(x_{i},\;y_{i})\sim
P}\,\operatorname{\mathbb{E}}_{(x_{j},\;y_{j})\sim
P}\,\operatorname{\mathbb{E}}_{\lambda\sim\text{Beta}(\alpha,\;\alpha)}\operatorname{\mathbb{E}}_{k\sim\mathcal{S}}~{}~{}(L_{1}+L_{2}),$
(9)

where

$\displaystyle L_{1}$
$\displaystyle=\text{Mix}_{p_{u}}[\ell(f_{k}(\phi_{k}),y_{i}),\ell(f_{k}(\phi_{k}),Y)],$
(10) $\displaystyle L_{2}$
$\displaystyle=\ell(f_{k}(\phi_{k}),W(y_{i},y_{j})),$ (11)

This section is an ablation study to show the effect of $L_{1}$ and $L_{2}$ in
PatchUp loss. Table 4 shows the error rate on the validation set for
WideResNet-28-10 on CIFAR-100. This study shows that the summation of the
$L_{1}$ and $L_{2}$ reduces error rate by $.1\%$ in both Soft PatchUp and Hard
PatchUp.

Table 4: The error rate on the validation set on CIFAR-100 for
WideResNet-28-10 with Hard PatchUp and Soft PatchUp. The result is the mean
and standard deviation of the experiment for five runs. A smaller number
indicates better performance.

Simple WideResNet-28-10 |  |  | Error Rate: $23.256\pm 0.586$
---|---|---|---
| Error rates with $L1$ | Error rates with $L2$ | Error rates with $L(f)$
Soft PatchUp | $16.856\pm 0.666$ | $16.865\pm 0.339$ | $\mathbf{16.75\pm 0.291}$
Hard PatchUp | $16.135\pm 0.229$ | $16.79\pm 0.457$ | $\mathbf{16.02\pm 0.358}$

##  5 Conclusion

We presented PatchUp, a simple and efficient regularizer scheme for CNNs that
alleviates some of the drawbacks of the previous mixing-based regularizers.
Our experimental results show that with the proposed approach, PatchUp, we can
achieve state-of-the-art results on image classification tasks across
different architectures and datasets. Similar to previous mixing based
approaches, our approach also has the advantage of avoiding any added
computational overhead. The strong test accuracy achieved by PatchUp, with no
additional computational overhead, makes it particularly appealing for
practical applications.

## Acknowledgments

We would like to acknowledge Compute Canada and Calcul Quebec for providing
computing resources used in this work. The authors would also like to thank
Damien Scieur, Hannah Alsdurf, Alexia Jolicoeur-Martineau, and Yassine
Yaakoubi for reviewing the manuscript. SC is supported by a Canada CIFAR AI
Chair and an NSERC Discovery Grant.

## References

  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 25_ , pages 1097–1105. Curran Associates, Inc., 2012.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_ , pages 91–99. Curran Associates, Inc., 2015.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  _CoRR_ , abs/1512.03385, 2015.  URL http://arxiv.org/abs/1512.03385.
  * Hinton et al. [2012] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath, and B. Kingsbury.  Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups.  _IEEE Signal Processing Magazine_ , 29(6):82–97, Nov 2012.  ISSN 1558-0792.  doi: 10.1109/MSP.2012.2205597.
  * Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.  Sequence to sequence learning with neural networks.  In _Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’14, page 3104–3112, Cambridge, MA, USA, 2014. MIT Press.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _CoRR_ , abs/1706.03762, 2017.  URL http://arxiv.org/abs/1706.03762.
  * Arpit et al. [2017] Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al.  A closer look at memorization in deep networks.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 233–242. JMLR. org, 2017.
  * Goodfellow et al. [2016] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  _Deep learning_.  MIT press, 2016.
  * Achille and Soatto [2018] A. Achille and S. Soatto.  Information dropout: Learning optimal representations through noisy computation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(12):2897–2905, Dec 2018.  ISSN 1939-3539.  doi: 10.1109/TPAMI.2017.2784440.
  * Guo et al. [2018] Hongyu Guo, Yongyi Mao, and Richong Zhang.  Mixup as locally linear out-of-manifold regularization.  _CoRR_ , abs/1809.02499, 2018.  URL http://arxiv.org/abs/1809.02499.
  * Srivastava et al. [2014a] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15:1929–1958, 06 2014a.
  * Tompson et al. [2014a] Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler.  Efficient object localization using convolutional networks.  _CoRR_ , abs/1411.4280, 2014a.  URL http://arxiv.org/abs/1411.4280.
  * Ghiasi et al. [2018] Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le.  Dropblock: A regularization method for convolutional networks.  _CoRR_ , abs/1810.12890, 2018.  URL http://arxiv.org/abs/1810.12890.
  * Srivastava et al. [2014b] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15:1929–1958, 2014b.  URL http://jmlr.org/papers/v15/srivastava14a.html.
  * Tompson et al. [2014b] Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler.  Efficient object localization using convolutional networks.  _CoRR_ , abs/1411.4280, 2014b.  URL http://arxiv.org/abs/1411.4280.
  * Zhang et al. [2017] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization, 2017.
  * Mai et al. [2019] Zhijun Mai, Guosheng Hu, Dexiong Chen, Fumin Shen, and Heng Tao Shen.  Metamixup: Learning adaptive interpolation policy of mixup with meta-learning, 2019.
  * Verma et al. [2019] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio.  Manifold mixup: Better representations by interpolating hidden states.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 6438–6447, Long Beach, California, USA, 09–15 Jun 2019. PMLR.  URL http://proceedings.mlr.press/v97/verma19a.html.
  * DeVries and Taylor [2017] Terrance DeVries and Graham W. Taylor.  Improved regularization of convolutional neural networks with cutout, 2017\.
  * Yun et al. [2019] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features, 2019.
  * Krizhevsky [2009] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  2009\.
  * Netzer et al. [2011] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng.  Reading digits in natural images with unsupervised feature learning.  In _NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011_ , 2011.
  * Szegedy et al. [2013] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ , 2013.
  * Goodfellow et al. [2014] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples, 2014.
  * Tishby and Zaslavsky [2015] Naftali Tishby and Noga Zaslavsky.  Deep learning and the information bottleneck principle.  _CoRR_ , abs/1503.02406, 2015.  URL http://arxiv.org/abs/1503.02406.

## Appendices

##  Appendix A Algorithm

In this appendix, we provide a detailed algorithm for implementing PatchUp. As
with most regularization techniques, PatchUp also has two modes (either
inference or training). It also needs the combining type (either Soft PatchUp
or Hard PatchUp), $\gamma$, and $block\\_size$. This algorithm shows how
PatchUp generates a new hidden representation from $(g_{k}(x_{i}),y_{i})$ and
$(g_{k}(x_{j}),y_{j})$. Lines 4 to 9 in the algorithm1 are the binary mask
creation process used in both Soft PatchUp and Hard PatchUp.

Algorithm 1 PatchUp

Input:
$(g_{k}(x_{i}),y_{i})$: the hidden representation for the sample
$(x_{i},\,y_{i})$ at layer $k$.
$(g_{k}(x_{j}),y_{j})$: the hidden representation for the sample
$(x_{j},\,y_{j})$ at layer $k$.
$mode:$ either inference or training.
${mixing\\_type}$: soft or hard.
${\gamma}$: the probability of altering a feature.
${block\\_size}$: the size of each block in the binary mask.
Output
$y_{i}$, $y_{j}$: original labels for samples $i$ and $j$.
$H^{\prime}$: the new hidden representation computed by PatchUp.
$p_{u}$: The portion of the feature maps that remained unchanged.
$Y$: the target corresponding to the changed features.
$W$: re-weighted target according to the interpolation policy.

1: if ${mode}$ == Inference then

2: return $(g_{k}(x_{i}),y_{i})$, $(g_{k}(x_{j}),y_{j})$

3: end if

4: $kernel\\_size\leftarrow(block\\_size,\,block\\_size$)

5: $stride\leftarrow(1,\,1)$

6: $padding\leftarrow(\frac{block\\_size}{2},\frac{block\\_size}{2})$

7: $\gamma_{adj}\leftarrow$ adjust $\gamma$ using (2)

8: $holes\leftarrow
max\\_pool2d(Bernoulli(\gamma_{adj}),\;kernel\\_size,\;stride,\;$ $padding)$

9: $Mask\leftarrow 1-holes$

10: $unchanged\leftarrow Mask\odot g_{k}(x_{i})$

11: $p_{u}\leftarrow\text{calculate the portion of changed features map.}$

12: $Patch_{i}\leftarrow holes\odot g_{k}(x_{i})$

13: $Patch_{j}\leftarrow holes\odot g_{k}(x_{j})$

14: if ${mixing\\_type}$ == hard then

15: $Patch_{i}\leftarrow Patch_{j}$

16: $Y\leftarrow y_{j}$

17: $W$ $\leftarrow$ $W_{hard}(y_{i},\;y_{j})$ using (8)

18: else if ${mixing\\_type}$ == soft then

19: $\lambda\sim Beta(\alpha,\;\alpha)$

20: $Y\leftarrow\textit{Mix}_{\lambda}(y_{i},\;y_{j})$

21: $W$ $\leftarrow$ $W_{soft}(y_{i},\;y_{j})$ using (8)

22: $Patch_{i}\leftarrow\textit{Mix}_{\lambda}(Patch_{i},\;Patch_{j})$

23: end if

24: $H^{\prime}\leftarrow unchanged+Patch_{i}$

25: return $y_{i},\,y_{j},\,H^{\prime},\,p_{u},\,Y,\,W$

Figure A.8 briefly illustrates and summarizes the binary mask creation process
in PatchUp. Lines 11 to 25 correspond to the interpolation and combination of
hidden representations in the mini-batch in PatchUp. Figure A.6 compares the
masks generated by PatchUp and CutMix.

(a) Mask sampling in PatchUp.

(b) Mask sampling in CutMix.

Figure A.6: Mask sampling in PatchUp is applied in the hidden state, compared
to CutMix which is applied in the input space. Red areas show the blocks that
should be altered.

##  Appendix B Why random $k$?

PatchUp applies block-level regularization at a randomly selected hidden
representation layer $k$. The Information Bottleneck (IB) principle,
introduced by Tishby and Zaslavsky [25], gives a formal intuition for
selecting $k$ randomly. First, let us encapsulate the layers of the network
into blocks where each block could contain more than one layer. Let $g_{k}$ be
the $k$-th block of layers. In this case, sequential blocks share the
information as a hidden representation to the next block of layers,
sequentially. We can consider this case as a Markov chain of the block of
layers as follows:

$x\to g_{1}(x)\to g_{2}(x)\to g_{3}(x).$ (12)

In this scenario, the sequential communication between the intermediate hidden
representations are considered to be an information bottleneck. Therefore,

$I(g_{3}(x);\;g_{2}(x))<I(g_{2}(x);\;g_{1}(x))<I(g_{1}(x);\;x),$ (13)

where $I(g_{k}(x);\;g_{k-1}(x))$ is the mutual information between the $k$-th
and $(k-1)$-th layer.

If $g_{k=3}(x)$ has enough information to represent $x$, then applying
regularization techniques in $g_{k=3}(x)$ will provide a better generalization
to unseen data. However, most of the current state-of-the-art CNN models
contain residual connections which break the Markov chain described above
(since information can skip the $g_{k=3}$ layer). One solution to this
challenge is to randomly select a residual block and apply regularization
techniques like ManifoldMixup or PatchUp.

Figure A.7: PatchUp mask creation process ($block\\_size=5$). The left matrix
shows the process of feature selection from feature maps. By using a
$max\\_pool2d$ function, we can create blocks around selected features. The
$max\\_pool2d$ function uses $stride=(1,\;1)$,
$kernel\\_size=(block\\_size,\,block\\_size)$, and
$padding=(\frac{block\\_size}{2},\frac{block\\_size}{2})$. Red and blue points
are $1$ and $0$ in the generated binary mask, respectively.

Figure A.8: The comparison of $\rho$ for flattened hidden representations of a
mini-batch of samples at the second residual block (layer $k=3$) of
WideResNet-28-10 with corresponding regularization method.

##  Appendix C PatchUp Interpolation Policy Effect

Assume that $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ are flattened hidden
representations of two examples produced at layer $k$. And, $\mathcal{H}$ is
the flattened interpolated hidden representation of these two paired samples
at layer $k$. First, we calculate the cosine distance of the pairs
($\mathcal{H}_{2}$, $\mathcal{H}_{1}$), ($\mathcal{H}_{1}$, $\mathcal{H}$),
and ($\mathcal{H}_{2}$, $\mathcal{H}$). Reversing the cosine of these cosine
similarities give the angular distance between each pair of vectors denoted as
$\angle\theta$, $\angle\alpha_{1}$, and $\angle\alpha_{2}$, respectively.
There is always a surface that contains $\mathcal{H}_{2}$ and
$\mathcal{H}_{1}$ denoted as $\mathcal{S}$. Mathematically, we have:

$\begin{array}[]{l}\angle\alpha_{1}=\cos^{-1}(\frac{\mathbf{\mathcal{H}_{1}}\cdot\mathbf{\mathcal{H}}}{\|\mathbf{\mathcal{H}_{1}}\|\|\mathbf{\mathcal{H}}\|})\And\angle\alpha_{2}=\cos^{-1}(\frac{\mathbf{\mathcal{H}_{2}}\cdot\mathbf{\mathcal{H}}}{\|\mathbf{\mathcal{H}_{2}}\|\|\mathbf{\mathcal{H}}\|})\And\angle\theta=\cos^{-1}(\frac{\mathbf{\mathcal{H}_{2}}\cdot\mathbf{\mathcal{H}_{1}}}{\|\mathbf{\mathcal{H}_{2}}\|\|\mathbf{\mathcal{H}_{1}}\|}),\end{array}$
(14)

Let us define $\angle\rho=(\angle\alpha_{1}+\angle\alpha_{2})-\angle\theta$
and $\angle\rho^{\prime}=|\angle\alpha_{1}-\angle\alpha_{2}|-\angle\theta$.
According to the triangle inequality principle, either $\angle\rho$ or
$\angle\rho^{\prime}$ will be zero if, and only if,
$\mathcal{H}\in\mathcal{S}$. Figure A.9 illustrates three possible scenarios
for two paired flattened hidden representations and their flattened
interpolated hidden representations. $\angle\rho$ and $\angle\rho^{\prime}$
are zero for the left and right figures, respectively. We try to empirically
show that $\mathcal{H}_{2}$, $\mathcal{H}_{1}$, and $\mathcal{H}$ always lie
in the same surface $\mathcal{S}$ and $\mathcal{H}$ lies between
$\mathcal{H}_{2}$ and $\mathcal{H}_{1}$ in ManifoldMixup. This means that
$\angle\rho=0$ for ManifoldMixup because of its linear interpolation policy.
The middle figure in A.9 is the case that both $\angle\rho$ and
$\angle\rho^{\prime}$ are not equal to zero. This figure shows that one
possible situation is that flattened interpolated hidden representation does
not lie in the surface $\mathcal{S}$. Our goal is to produce the interpolated
hidden representation that lies in all possible places towards all dimensions
in the hidden space.

Figure A.9: $\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ are the flattened hidden
representations. $\mathcal{H}$ is the flattened interpolated hidden
representation that can be produced by either ManifoldMixup, Soft PatchUp or
Hard PatchUp.

As discussed in section 3, ManifoldMixup can provide interpolated hidden
representation only in a limited space. However, Soft PatchUp and Hard PatchUp
can produce a wide variety of interpolated hidden representations towards
different dimensions. To support that, in WideResNet-28-10, for a mini-batch
of 100 samples, we calculated the $\angle\rho$ for the flattened interpolated
hidden representation produced by ManifoldMixup, Hard PatchUp, and Soft
PatchUp at the second residual block (layer $k=3$) with the same interpolation
policy ($\lambda=.4$, $\gamma=.5$, and $block\\_size=7$) for both Soft PatchUp
and Hard PatchUp for all samples in the mini-batch. The swarmplot A.8 shows
all $\angle\rho$ for the mini-batch are equal to zero in ManifoldMixup, which
empirically supports our hypothesis. However, PatchUp produces more diverse
interpolated hidden representations towards all dimensions in the hidden
space. It is worth mentioning that few $\angle\rho$ that are equal to zero in
Soft PatchUp and Hard PatchUp belong to the interpolated hidden representation
that was constructed from the pairs with the same labels.

##  Appendix D PatchUp Experiment Setup and Hyper-parameter Tuning

This section describes the hyper-parameters of each model in table 5 following
the hyper-parameter setup from ManifoldMixup [18] experiments in order to
create a fair comparison. First, we performed hyper-parameter tuning for the
PatchUp to achieve the best validation performance. Then we ran all the
experiments five times, reporting the mean and standard deviation of errors
and negative log likelihoods for the selected models. We let models train for
defined epochs and checkpoint the best model in terms of validation
performance during the training. In our study, we used PreActResNet18,
PreActResNet34, and WideResNet-28-10 models. Table 5 shows the hyper-
parameters used for training the models.

(a) Impact of $\alpha$ in ManifoldMixup approach.

(b) Impact of $cutmix\\_prob$ in CutMix approach.

(c) Impact of $\gamma$, $block\\_size$ with $patchup\\_prob$ as 1.0 for Soft
PatchUp.

(d) Impact of $\gamma$, $block\\_size$ with $patchup\\_prob$ as 0.7 for Hard
PatchUp.

Figure A.10: Impact of hyper-parameters $\gamma$, $block\\_size$ and
$patchup\\_prob$ on error rates in the CIFAR-10 validation set for
PreActResNet18. We repeated each job three times to collect the mean and the
standard deviation of errors. Marked points are the mean of the error rate in
the validation set. And, the shadow shows the bootstrapping of results for
each hyper-parameter setting. The lower numbers on the y-axes correspond to
better performance. Table 5: The hyper-parameters used for each model to
compare the effect of each regularization technique. The learning rate is
denoted as lr. And, lr is multiplied at each learning rate schedule step by
the step factor.

Model | lr |  lr steps | step factor | Epochs
---|---|---|---|---
PreactResnet18 | $0.1$ |  $500$-$1000$-$1500$ | $0.1$ | $2000$
PreactResnet34 | $0.1$ |  $500$-$1000$-$1500$ | $0.1$ | $2000$
WideResnet-28-10 | $0.1$ |  $200$-$300$ | $0.1$ | $400$

PatchUp adds $patchup\\_prob$, $\gamma$ and $block\\_size$ as hyper-
parameters. $patchup\\_prob$ is the probability that the PatchUp operation is
performed for a given mini-batch, i.e if there are $N$ mini-batches and
$patchup\\_prob$ is $p$, PatchUp is performed in $p$ fraction of $N$ mini-
batches. $\gamma$ and $block\\_size$ are described in section 2. We tuned the
PatchUp hyper-parameter on CIFAR-10 with the PreActResNet18. To create a
validation set, we split $10\%$ of training samples into a validation set. We
set $\alpha$ to $2$ in PatchUp. For Soft PatchUp, we set $patchup\\_prob$ to
$1.0$ and applied PatchUp to all mini-batches in training. Then, we did a grid
search by varying $\gamma$ from $0.45$ to $0.9$ and $block\\_size$ from $3$ to
$9$. We found that $\gamma$ of $0.75$ and $block\\_size$ of $7$ work best for
Soft PatchUp as shown in figure 10(c). Similarly, for Hard PatchUp, we set
$patchup\\_prob$ to $0.7$ and performed a grid search by varying $\gamma$ from
$0.2$ to $0.6$ and $block\\_size$ from $3$ to $9$. We found that
$block\\_size$ of $7$ and $\gamma$ of $0.5$ yield the best results for Hard
PatchUp as shown in figure 10(d). Figure 10(a) shows that ManifoldMixup with
($\alpha=1.5$) achieves the best validation performance. For cutout, we used
the same hyper-parameters proposed in [19], setting cutout to $16$ for
CIFAR10, $8$ for CIFAR100, and $20$ for SVHN following [19]. Figure 10(b)
shows that CutMix achieves its best validation performance in PreActResNet18
in CIFAR-10 with $cutmix\\_prob=0.4$. Furthermore, DropBlock achieves its best
validation performance on this task by setting the block size and $\gamma$ to
$7$ and $0.9$, respectively [13].

##  Appendix E  Generalization on Deformed Images

Table 6: Error rates in the test set on samples subject to affine
transformations for WideResNet-28-10 trained on CIFAR-100 with indicated
regularization method. We repeated each test for five trained models to report
the mean and the standard deviation of errors. Best performance result is
shown in bold, second best is underlined. The lower number is better.

Transformation |  cutout |  CutMix |  ManifoldMixup |  Soft PatchUp |  Hard PatchUp
---|---|---|---|---|---
Rotate (-20, 20) | $36.162\pm 0.633$ | $34.236\pm 0.785$ | $35.774\pm 0.621$ | $\mathbf{31.282\pm 0.622}$ | $\uline{31.340\pm 0.318}$
Rotate (-40, 40) | $57.220\pm 0.549$ | $56.512\pm 0.752$ | $56.610\pm 0.877$ | $\mathbf{52.014\pm 0.916}$ | $\uline{53.804\pm 0.576}$
Shear (-28.6, 28.6) | $33.482\pm 0.463$ | $31.770\pm 0.312$ | $32.300\pm 0.317$ | $\uline{30.898\pm 0.836}$ | $\mathbf{28.426\pm 0.430}$
Shear (-57.3, 57.3) | $53.328\pm 0.587$ | $\uline{50.618\pm 0.552}$ | $52.366\pm 0.170$ | $51.908\pm 0.632$ | $\mathbf{48.334\pm 0.631}$
Scale (0.6) | $56.770\pm 0.376$ | $\mathbf{45.980\pm 0.404}$ | $63.924\pm 2.160$ | $52.648\pm 0.616$ | $\uline{46.924\pm 1.035}$
Scale (0.8) | $30.550\pm 0.611$ | $\uline{26.818\pm 0.328}$ | $29.012\pm 0.372$ | $27.188\pm 0.597$ | $\mathbf{23.840\pm 0.535}$
Scale (1.2) | $47.268\pm 0.639$ | $51.258\pm 0.817$ | $\mathbf{41.644\pm 0.846}$ | $\uline{42.108\pm 0.985}$ | $43.370\pm 1.223$
Scale (1.4) | $79.000\pm 0.933$ | $82.562\pm 0.575$ | $\uline{72.752\pm 0.846}$ | $\mathbf{70.970\pm 1.433}$ | $77.370\pm 1.457$

We created the deformed test sets from CIFAR100, as described in Section 4.2.
Table 6 shows improved quality of representations learned by a
WideResNet-28-10 model regularized by PatchUp on CIFAR-100 deformed test sets.
The significant improvements in generalization provided by PatchUp in this
experiment shows the high quality of representations learned with PatchUp.

##  Appendix F Robustness to Adversarial Examples

The adversarial attacks refer to small and unrecognizable perturbations on the
input images that can mislead deep learning models [24, 8]. One approach to
creating adversarial examples is using the Fast Gradient Sign Method (FGSM),
also known as a white-box attack [24]. FGSM creates examples by adding small
perturbations to the original examples. Once a regularized model is trained
then FSGM creates adversarial examples as follows [24]:

$\begin{array}[]{l}x^{\prime}=x+\epsilon\times\operatorname{sign}\left(\nabla_{\boldsymbol{x}}J(\boldsymbol{\theta},\boldsymbol{x},y)\right).\end{array}$
(15)

where $x^{\prime}$ is an adversarial example, $x$ is the original example, $y$
is the ground truth label for $x$, and
$J(\boldsymbol{\theta},\boldsymbol{x},y)$ is the loss of the model with
parameters of $\theta$. $\epsilon$ controls the perturbation.

Our experiments show the effectiveness of Soft PatchUp against the attacks in
most cases. However, Hard PatchUp performed well against the FGSM attack only
on PreActResNet34 for CIFAR-100. Figure A.11 shows the comparison of the
state-of-the-art regularization techniques’ effect on model robustness against
the FGSM attack.

(a) Comparison on PreActResNet18 for CIFAR-10.

(b) Comparison on PreActResNet18 for CIFAR-100.

(c) Comparison on PreActResNet34 for CIFAR-10.

(d) Comparison on PreActResNet34 for CIFAR-100.

(e) Comparison on WideResNet28-10 for CIFAR-10.

(f) Comparison on WideResNet28-10 for CIFAR-100.

(g) Comparison on WideResNet28-10 for SVHN.

Figure A.11: Robustness to the FGSM attack, known as a white-box attack. We
repeated each test for five trained models to report the mean and the standard
deviation of the accuracy of each method against the FGSM attack. The higher
values on the y-axes show the robustness of the model against the attack. And,
$\epsilon$ is the magnitude that controls the perturbation.

##  Appendix G Analysis of PatchUp’s Effect on Activations

In our implementation WideResNet28-10 has a conv2d module followed by three
residual blocks. Figure A.12 illustrates the comparison of ManifoldMixup,
cutout, CutMix, Soft PatchUp, and Hard PatchUp. Figure 12(a), 12(b), and 12(c)
show that PatchUp produces more variety of features in layers that we apply
PatchUp on.

(a) Comparison on the first convolution module.

(b) Comparison on 1st Residual Block.

(c) Comparison on 2nd Residual Block.

(d) Comparison on 3rd Residual Block.

Figure A.12: The effect of the state-of-the-art regularization techniques on
activations in WideResNet28-10 for CIFAR100 test set. Each curve is the
magnitude of feature activations, sorted by descending value, and averaged
over all test samples for each method. The higher magnitude shows a wider
variety of the produced features by the model at each block.