# Ortho-Shot: Low Displacement Rank Regularization with Data Augmentation for
Few-Shot Learning

Uche Osahor
West Virginia University
<EMAIL_ADDRESS>Nasser M. Nasrabadi
West Virginia University
<EMAIL_ADDRESS>
###### Abstract

In few-shot classification, the primary goal is to learn representations from
a few samples that generalize well for novel classes. In this paper, we
propose an efficient low displacement rank (LDR) regularization strategy
termed Ortho-Shot; a technique that imposes orthogonal regularization on the
convolutional layers of a few-shot classifier, which is based on the doubly-
block toeplitz (DBT) matrix structure. The regularized convolutional layers of
the few-shot classifier enhances model generalization and intra-class feature
embeddings that are crucial for few-shot learning. Overfitting is a typical
issue for few-shot models, the lack of data diversity inhibits proper model
inference which weakens the classification accuracy of few-shot learners to
novel classes. In this regard, we broke down the pipeline of the few-shot
classifier and established that the support, query and task data augmentation
collectively alleviates overfitting in networks. With compelling results, we
demonstrated that combining a DBT-based low-rank orthogonal regularizer with
data augmentation strategies, significantly boosts the performance of a few-
shot classifier. We perform our experiments on the miniImagenet, CIFAR-FS and
Stanford datasets with performance values of about 5% when compared to state-
of-the-art.

##  1 Introduction

The performance of convolutional neural network (CNN) models largely depend on
training a network with a lot of labelled instances and a spectrum of visual
variations which are mostly in thousands per class [29]. The cost of labelling
these data manually by human annotation as well as the scarcity of data that
captures the complete diversity in a specific class significantly limits the
potential of current vision models. However, the human visual system (HVS) can
identify new classes with fewer labelled examples [27, 40], this unique trait
of the HVS reveals the need to dive into new paradigms that would learn to
generalize new classes with a limited amount of labelled data for each novel
class. Recently, significant progress has been made towards better solutions
using ideas of meta-learning [42, 48, 69, 32, 33, 38].

Figure 1: The convolution expression; Conv(K, X) is converted into a faster
DBT vector representation; $y=Mx$.

Empirically, it has been observed that the convolutional filters learned in
deeper layers are highly correlated and redundant [64], thereby resulting in
unstable training performance and vanishing gradients. These shortcomings of
convolutional neural networks are also more damaging in few-shot
classification due to the small data size. The potential pitfalls of such
convolutional layers could result in under-utilization of model capacity,
overfitting, vanishing and exploding gradients [16, 7], growth in saddle
points [13] and shifts in feature statistics [24], which collectively affect
model generalization.

The doubly block-toeplitz (DBT) matrix [18] is part of a class of low
displacement rank (LDR) matrix constructions [72] that guarantee model
reduction and computational complexity reduction in neural networks which is
achieved by regularizing the weight matrices of network layers. The storage
requirement of such a DBT-regularized network is reduced from $O(n^{2})$ to
$O(n)$ and the computational complexity can be reduced from $O(n^{2})$ to
$O(nrlogn)$, due to the fast matrix-vector multiplication property of LDR
structured matrices as shown in Figure 2. It is also well established [34, 58]
that when filters are learned to be as orthogonal as possible, model capacity
is better utilized which in-turn improves feature expressiveness and intra-
class feature representation [1, 62, 59, Aghdaie2021AttentionAW].

Our goal is to present an effective baseline model that harnesses good learned
representations for few-shot classification kinds of tasks which perform
better or at par with current few-shot algorithms [66, 63, 61, 53, 55, 42, 48,
38]. In a nutshell, we tackled the few-shot learning limitations by imposing
orthogonal regularization on the model baseline which is a simpler yet
effective approach compared to techniques used previously in [65, 15, 44]. We
also incorporated data augmentation strategies that significantly improved
data diversity and overall model performance.

Figure 2: Toeplitz covariance matrices from features samples. This requires
learning $O(n)$ parameters, in contrast to $O(n)^{2}$ for generic covariance
matrices.

###  1.1 Contributions:

  * •

We adopted an efficient orthogonal regularization technique on convolutional
layers of the few-shot classifier that enhances model generalization and
intra-class feature embedding, using the doubly block toeplitz (DBT) matrix
structure.

  * •

We broke down the pipeline of a few-shot learner, and based on our findings,
we established three augmentations strategies namely: support augmentation,
query augmentation and task augmentation that aid in minimizing overfitting.

  * •

We show with compelling results that combining a DBT-based regularizer with a
robust augmentation strategy improves few-shot learning performance at an
average of 5%.

##  2 Related works

Orthogonal regularization. In convolutional networks, orthogonal weights have
being used to stabilize layer-wise distributions and to make optimization as
efficient as possible. In [4, 37] the authors introduced orthogonal weight
initialization driven by the norm preserving property of an orthogonal matrix.
However, it was shown that the orthogonality and isometry property does not
necessarily sustain throughout training [4] if the convolutional layers are
not properly regularized. In other works, [25, 43, 23] considered Stiefel
manifold-based hard constraints of weights [tagare2011notes], but their
performance reported on VGG networks [52] were not as promising. These
aforementioned methods [25, 43, 23] are associated with hard orthogonality
constraints and in most cases, they have to repeat singular value
decomposition (SVD) during training which is computationally expensive on the
GPUs. A recent work adopted soft orthogonality [2, 3, 4, 67], where the Gram
matrix of the weight matrix $K$ is required to be close to identity, given as
$\lambda\left\|K^{T}K-I\right\|_{F}^{2}$, where $\lambda$ is the Frobenius
norm-based regularization coefficient. It’s a more efficient approach than the
hard orthogonality assumption [25, 43, 23, 20, 68] and can be viewed as a
different weight decay term limiting the set of parameters close to a Stiefel
manifold [tagare2011notes]. Their approach constrained orthogonality among
filters in one layer, leading to smaller correlations among learned features
and implicitly reducing the filter redundancy. However, there are special
cases where the Gram matrix cannot be close to identity which implies that
matrix $K$ is overcomplete [58]. Similarly, other works explored orthogonal
weight initialization [55], mutual coherence with the isometry property [4],
penalizing off-diagonal elements [9] towards improving kernel orthogonality.

In general, the orthogonality of $K$ alone is not sufficient to make the
linear convolutional layer orthogonal among its filters. Due to these
shortcomings, we apply the improved regularization technique used in [64, 31].
We adopt the DBT matrix denoted as $M$ with a filter $K$, while we keep the
reshaped input $x$ and output $y$ intact. The matrix multiplication; $y=Mx$
enforces the orthogonality of $M$ as shown in Figure 1 and Figure 3.

Augmentation. Data augmentation has become a well established technique for
most image classifiers and deep networks, as it provides an efficient strategy
that significantly mitigates the models’ vulnerability to overfitting. In
contrast, data augmentation still has room for expansion and adaptation in
few-shot classification or other derivatives of meta-learning in general.
Existing works [57, 26, 56], apply basic data augmentation strategies like
random crops, horizontal flips and color jitter as the staple method for most
meta-learning applications. However, these aforementioned techniques have
plateaued in performance with little room for significant improvement [47,
41]. Other works have added random noise to labels to alleviate overfitting
[46], some techniques rotate all the images in a class and consider the newly
rotated class as distinct from its parent class. Recent works [12, 51, 41, 15,
45] are recording better performance values when augmentation strategies are
injected within the meta-learning pipeline.

In our work, we explored the benefits of including augmentation strategies
along the pipeline of a DBT regularized few-shot classifier. We identified how
different augmentation approaches could affect a few-shot classifier when
placed strategically along the classifier pipeline. At the core of our
findings, we observed that the classifier is more sensitive to query data than
support data.

Toeplitz matrix applications. Kimitei et al. [28] used toeplitz matrices with
Tikhonov regularization [39] as a mathematical approach to restoring blurred
images. They explored their techniques on image restoration, enhancement,
compression and recognition. In [19], the authors presented modern
computational methods for treating linear deconvolution problems, they showed
how to exploit the toeplitz structure to derive efficient numerical
deconvolution algorithms. In compressive sensing applications [54], toeplitz-
like matrices allow the entire signal to be efficiently acquired and
reconstructed from relatively few measurements, compared to previous
compressive sensing frameworks where a random measurement matrix is employed.

Figure 3: A doubly block-Toeplitz (DBT) matrix
$M\in\mathbb{R}^{(NH^{\prime}W^{\prime})\times(CHW)}$ derived from the kernel
tensor $K\in\mathbb{R}^{N\times C\times k\times k}$.

##  3 Background

We consider a meta learning scenario for an N-shot, K-way classification
problem where the training and testing task datasets can be represented as
$\mathcal{T}=\\{\mathcal{D}_{i}^{train},\mathcal{D}_{i}^{test}\\}_{i=1}^{I}$.
Such a meta-training task is divided into
$\mathcal{D}_{i}^{train}=\\{(x_{t},y_{t})\\}_{t=1}^{T}$ and
$\mathcal{D}_{i}^{test}=\\{(x_{q},y_{q})\\}_{q=1}^{Q}$, called a meta-training
set [42, 48, 69, 32, 33, 38]. The set of $D^{train}$ and $D^{test}$ represent
a small number of samples from the same distribution. We implement a DBT-based
learner $\mathcal{B}_{dbt}(\cdot)$ to train the model for a given input
feature denoted as $y=f_{\theta}(\mathbf{x}_{*})$, where (*) denotes
implementations for train and test sets. We then map train and test examples
into a DBT structured embedding space $\Psi_{*}=f_{\theta}(\mathbf{x}_{*})$.

The objective of our model becomes:

$\displaystyle\theta=\mathcal{B}_{dbt}(\mathcal{D}_{i}^{train};\phi)$ (1)
$\displaystyle\;\,=\arg\min_{\theta}\mathcal{L}^{base}(\mathcal{D}_{i}^{train};\theta,\phi)+\mathcal{R}(\theta),$

where $\phi$ represents the parameters of the embedding model,
$\mathcal{L}^{base}$ is the loss function and $\mathcal{R}$ is the
regularization as described in Section 4.2. At the end of meta-training, the
performance of the model is evaluated on a set of tasks
$\mathcal{S}=\\{\mathcal{(}{D}_{i}^{train},D^{test})\\}_{i=1}^{I}$ called the
meta-testing set. The final evaluation representation over the test set is:

$\displaystyle E_{S}[L^{meta}(D^{test};\theta,\phi)].$ (2)

The goal of meta learning is to learn a transferable efficient embedding model
$f_{\theta}$ that generalizes to new tasks. As described in section 4, we
deviated from popular techniques [63, 53, 55, 14] that train classifiers with
convolutional blocks with some form of hard orthogonality constraint [41]. Our
strategy, imposes a better low displacement rank DBT-based soft orthogonality
constraint on the classifier network to produce more efficient embeddings for
the base learner. The final embedding model is given as:

$\displaystyle\phi=\arg\min_{\phi}\mathcal{L}^{ce}(\mathcal{D}^{new};\phi),$
(3)

where $\mathcal{D}_{i}^{new}$ is the task from $\mathcal{T}$ and
$\mathcal{L}^{ce}$ denotes the cross-entropy loss between predictions and
ground truth labels.

Figure 4: The network depicts a DBT-regularized few shot learner. The network
embeddings $B_{dbt}(\cdot)$ are regularized based on the DBT structured
matrix. The dotted box of the CNN block illustrates the inner translation
between convolution layer embeddings $f{(\cdot)}$ and the more efficient DBT-
based embeddings denoted as $f^{\prime}{(\cdot)}$, see above. The Algorithm 1
gives a logical representation of the training process.

###  3.1 Doubly-block toeplitz (DBT) regularization

The feature interaction between two weights vectors $v$ and $w$, within the
layers of a few-shot classifier involves a convolution operation which can
simply be represented as $v*w=\sum\nolimits_{i=0}^{k}v(i)w(k-i)$, such that if
$v$ has length $m$ and $w$ has length $n$ then $v*w$ has length $m+(n-1)$.
Unfortunately, this computation involves $O(nm)$ operations which is not
suitable for fast linear algebraic computations and intra-class parameter
sharing which is critical for few shot learning. For such computations, if we
consider a single convolution layer with input tensor $X\in\mathbb{R}^{C\times
H\times W}$ and kernel $K\in\mathbb{R}^{N\times C\times k\times k}$, the
convolution’s output tensor is expressed as $Y=Conv(K,X)$, where
$Y\in\mathbb{R}^{N\times H^{\prime}\times W^{\prime}}$, we replaced the
convolution operator $(*)$ with $Conv(.)$ for simplicity. $N$, $H$, $W$ and
$C$ are the number of kernels, height, width and channel of the input tensor,
respectively. While $k$ represents the kernel size and $H^{\prime}$,
$W^{\prime}$ are the height and width for the output tensor, respectively.

Inline with our goal to improve the computational complexity and enhance
better feature representation, we adapted a DBT matrix construction by
utilizing the linear property of the convolution operation. The convolution
expression; Conv(K, X) is converted into a faster DBT matrix-vector
representation given as:

$\displaystyle Y=Conv(K,X)\Leftrightarrow y=Mx.$ (4)

This simple rearrangement establishes the foundation for adapting the DBT
regularizer in our few-shot classifier network. Where $M$ is the DBT matrix,
$x$ and $y$ represent flattened input and output tensors, respectively. $M$ is
structured and is of rank $r<<min(m,n)$ [58], this representation minimizes
the storage requirements to $(mr+nr)$ parameters and accelerates the matrix-
vector multiplication time to $O(mr+nr)$. Section 1: Figure 1 in the
supplementary material shows the hierarchy for storage cost and operation
count for matrix-vector multiplications. This DBT formulation stabilizes the
spectrum of the newly derived DBT-based matrix $M$. In section 1.1 and 1.4 of
the supplementary material, we reflect the overall benefit of the DBT model.

Algorithm 1 Ortho-shot algorithm

1:$\mathcal{D}\leftarrow\\{\mathcal{X},\mathcal{Y}\\}_{i=1}^{N}$

2:procedure Orthogonal-regularizer

3: $I_{ro}\leftarrow t(I)$ $\triangleright$ toeplitz matrix

4: $M\leftarrow Conv(K,K)$

5: $y\leftarrow\lambda(\left\|M-I_{ro}\right\|)$

6: ${\psi}(\cdot)\leftarrow Mx$ $\triangleright$ DBT based output

7:
$\mathcal{D}^{train},\mathcal{D}^{test}\leftarrow\\{\mathcal{X}^{t},\mathcal{Y}^{t}\,;\,\mathcal{X}^{q},\mathcal{Y}^{q}\\}_{i=1}^{K}$

8:procedure Few Shot Learning

9: if Train then

10: for $i=1$ do, T

11: $B_{dbt}\leftarrow S_{\psi}$ $\triangleright$ DBT model

12: $\mathcal{L}^{dce}+\mathcal{L}^{orth}\leftarrow
loss(B_{dbt}(\mathcal{X}^{t}),\mathcal{Y}^{t})$

13:
$\mathcal{L}^{total}_{dbt}\leftarrow\mathcal{L}^{dce}+\lambda\mathcal{L}^{orth}$

14: $\triangleright$ (orthogonal regularization)

15: if Test then

16: for $i=1$ do, Q

17: $\mathcal{L}^{test}\leftarrow loss(B_{dbt}(\mathcal{X}^{q}),Y^{q})$

##  4 The proposed method

We present an efficient low displacement rank (LDR) regularization strategy
termed Ortho-Shot that imposes orthogonal regularization on the convolutional
layers of a few-shot classifier which is based on the doubly-block toeplitz
(DBT) matrix structure [64, 23]. Our technique, as reflected in section 4.1
deviates from popular methods that train classifiers with convolutional blocks
with some form of hard orthogonality constraint. We also adapted a set of
augmentation strategies based on the support, query and task datasets to boost
overall model performance. In general, our approach enhances model
generalization, intra-class feature embeddings and also minimizes overfitting
for a few-shot classifier. To further describe our approach, we consider a
single convolutional layer case. We extract feature embeddings
$X\in\mathbb{R}^{C\times H\times W}$ from the intermediate convolutional
layers of the few-shot classifier and then flatten it to a vector
$x\in\mathbb{R}^{1\times(H\times W)}$. The weight tensor; $K$ of our model is
also converted to a doubly block-Toeplitz (DBT) matrix
$M\in\mathbb{R}^{(NH^{\prime}W^{\prime})\times(CHW)}$ derived from kernel
tensor $K\in\mathbb{R}^{N\times C\times k\times k}$ as shown in Figure 3. With
the aforementioned matrix structure, we are able to apply a better
orthogonality constraint as described by the Lemma in 4.2. In Figure 4, we
show a fully regularized setup for a single CNN block. The network embeddings
$B_{dbt}(\cdot)$ are regularized based on the DBT structure and the entire
losses from each respective layer is summed up to $L_{orth}$. We show
promising results for our technique as described by the CAM plots in Figure 5.

###  4.1 Convolutional orthogonality

A DBT kernel matrix $M$ can be applied on both a rectangular or square case,
where kernel $M\in\mathbb{R}^{(NH^{\prime}W^{\prime})\times(CHW)}$ dimensions
can be rectangular $(NH^{\prime}W^{\prime})\leq(CHW)$ or square,
$(NH^{\prime}W^{\prime})>(CHW)$. In the rectangular case, the uniform spectrum
applies row orthogonal convolution while the square case requires column
orthogonal convolution. In theory, the DBT kernel $M$ is highly structured and
sparse [31] as a result, an equivalent representation is required to
regularize the spectrum of $M$ to be uniform [64, 23]. We give the cases for
both row and column orthogonality and we also propose an equivalent
representation in this section.

Row orthogonality case. The row of matrix $M$ corresponds to a filter at a
particular spatial location flattened to a vector, denoted as
$M_{i,h^{\prime}w^{\prime}}\in\mathbb{R}^{CHW}$. The row orthogonality
condition is given as:

$\displaystyle\langle{M_{i,h^{\prime}_{1},w^{\prime}_{1}}\cdot
M_{jh^{\prime}_{2},w^{\prime}_{2}}}\rangle=\begin{cases}1,&i_{h^{\prime}_{1},w^{\prime}{{}_{1}}}=j_{h^{\prime}_{2},w^{\prime}{{}_{2}}}\\\
0,&otherwise.\end{cases}$ (5)

This results to an equivalent of Equation 4 as the following self-convolution:

$\displaystyle Y=Conv(K,K,padding=P,stride=S)=I_{r0},$ (6)

where $I_{r0}\in\mathbb{R}^{N\times N\times(2P/S+1)\times(2P/S+1)}$ is a
tensor with an identity matrix at the centre and zeros entries elsewhere.

Figure 5: Illustration of CAM plots. (a) The second row represents CAM plots
for single classes. The red squares highlight regions of interest clearly
highlighted by the model.(b) Shows more complex scenarios where multi classes
are involved. Classes are clearly separated from non-classes of interest. In
(c), all the objects separated by bounding boxes are clearly localised as
indicated by the CAM plot.

Column orthogonality case. If $X_{i,hw}\in\mathbb{R}^{C\times H\times W}$
denotes an input tensor, which has all zero except an entry at the $i^{th}$
input channel at spatial location $(h,w)$. Then we can denote the flattened
vector as $x_{ihw}\in\mathbb{R}^{C\times H\times W}$ derived from $X_{i,hw}$.
A column vector $M_{i,hw}$ of $M$ is obtained by multiplying $M$ and column
vector $x_{i,hw}$. Similar to the row orthogonality,

$\displaystyle Y=Conv(K^{T},K^{T},padding=k-1,stride=1)=I_{c0},$ (7)

where $K^{T}$ is the input-output transposed $K$, i.e.,
$K^{T}\in\mathbb{R}^{C\times N\times k\times k},I_{c0}\in\mathbb{R}^{C\times
C\times(2k-1)\times(2k-1)}$ has all zeros except for the center $C\times C$
entries as an identity matrix. Figure 2 illustrates the DBT matrix $M$
structure of our model.

###  4.2 Row-column orthogonality equivalence

To develop an equivalent representation for row and column orthogonality, we
build on the equation described by lemma 1, which states that the minimizing
of the column orthogonality and row orthogonality costs are equivalent [31]
due to the property of the Frobenius norm.

Lemma 1: The row orthogonality cost
$\lambda\left\|KK^{T}-I_{r0}\right\|_{F}^{2}$ is equivalent to the column
orthogonality cost $\lambda\left\|K^{T}K-I_{c0}\right\|_{F}^{2}+U$ where $U$
is a constant. This implies that convolution orthogonality independent of the
shape of $M$ (square or rectangular) can be regularized, given as:

$\displaystyle L_{orth}=\lambda\left\|K^{T}K-I_{r0}\right\|_{F}^{2},$ (8)

where $L_{orth}$ is the DBT-based orthogonal regularization term that depends
only on Equation 6 and replaces the $\mathcal{R(\cdot)}$ term in Equation 1.

##  5 Experimental setup and analysis

Our experiments were conducted on the miniImagNet, CIFAR-FS, Stanford Dogs and
Stanford Cars datasets, respectively. We used the R2-D2 base leaner [8], the
”ResNet-12” and ”64-64-64-64” backbone for different few-shot learning modes
used in our work. Data augmentation strategies were also analysed to determine
the best combination for a DBT-regularized model. The complete details for of
the entire setup is expressed in section 1.2 of the supplementary material.

###  5.1 Data augmentation strategy

Motivated by the impact of applying a diverse augmentation strategy on meta-
learners, we established three unique augmentation approaches; support, query
and task augmentation that contribute to the overall classifier performance,
aimed at minimising overfitting. Our empirical analysis confirm that support
augmentation increases the number of fine tuning data while the query data
improves evaluation performance while training the classifier. Similarly, task
augmentation is used to increase the number of classes per task while
training. We adapted a couple of augmentation techniques such as CutMix [70],
where image patches are cut and pasted among training images and the ground
truth labels are also mixed proportionally within the area of the patches.
Mixup [50], a technique that generates convex combinations of pairs of
examples and their labels, which proved to be effective for support and query
augmentation strategies. As well as Self-Mix [71] in which an image is
substituted into other regions in the same image. This dropout effect improves
few-shot learning generalization overall. In addition, we implemented standard
data augmentation techniques by randomly erasing patches from the images
(Random Erase), horizontally flipping the images (Horizontal Flip), rotating
the images at different specified angles (Rotation) and Color Jitter, where we
randomly change the brightness, contrast and saturation of the images. To
boost the performance of our augmentation strategy, we combine different
augmentation techniques using the MaxUp augmentation approach proposed in
[17]. The rationale behind MaxUp augmentation is to minimize training loss by
performing parameter updates on the task that maximizes loss in a min-max
optimization manner, the MaxUp expression is given as:

$\displaystyle\min_{\theta}E_{\mathcal{T}}[\max_{M\in\mathcal{S}}\mathcal{L}(\mathcal{B}_{\theta^{\prime}},M(\mathcal{T}^{q}))],$
(9)

where $\theta^{\prime}$ represents the model parameters, $\mathcal{B}$ is the
base model, $\mathcal{L}$ is the loss function and $\mathcal{T}$ is a task for
both support and query data; $\mathcal{T}^{s}$ and $\mathcal{T}^{q}$,
respectively.

###  5.2 Augmentation performance

In this section, we investigate the performance of a few-shot classifier for
different augmentation strategies. We investigated three test cases that check
the training performance when data is sampled from the support, query and task
data, respectively. Our approach is similar to techniques adapted by [41, 30,
50] that examine the impact of augmentation on a diverse set of data
combinations.

Case 1: We trained the model at an equal number of support and query data as
indicated in Table 1, so as to establish a baseline performance of the model.
We use this strategy to compare the impact of any of the data pools (support
or query) when any of the augmented pairs is reduced.

Case 2: We initiated training of the classifier by randomly sampling from 5
and 10 unique samples per class of the support data while using the entire
query data pool. Using this approach, we reduced the influence of support data
in order to examine the impact of the diverse pool of query data on the
classifier. Our findings reflected in Table 1 show accuracy values at $\pm$
2%. This is a clear indication that augmentation of query data plays a more
significant role in the overall model performance. In contrast, we reduced the
number of query data while maintaining the initially set cap for support data
and recorded a decline in accuracy.

Case 3: To evaluate the impact of task augmentation, we used the CIFAR-FS data
to initially allocate 10 distinct 5-way classification tasks (252
combinations) before training, while the support and query datasets are
maintained equally at 500, respectively. We observed a decline in performance.
However, as we increased the amount of task data, significant improvement is
recorded, which confirms that task augmentation is crucial in few-shot
learning.

In summary, we broke down the few-shot learning process to determine the
influence of support, query and task augmentation, respectively. Our findings
confirm that our baseline learner is most sensitive to query data [41]. In
addition, task augmentation provided significant value (about 2%) that cannot
be overlooked by the classifier.

Table 1: Few-shot classification accuracy (%) using R2-D2 base leaner with a
ResNet-12 backbone on CIFAR-FS dataset. Support, Query and Task columns
represent the number of samples per class for support, query data and the
total number of tasks available.

Support Query Task 1-shot 5-shot 500 500 full 71.41$\pm$ 0.21 86.01$\pm$ 0.08
100 500 full 70.11$\pm$ 0.01 83.00$\pm$ 0.03 10 500 full 70.72$\pm$ 0.01
81.41$\pm$ 0.32 500 300 full 69.41$\pm$ 0.11 72.41$\pm$ 0.08 500 100 full
59.00$\pm$ 0.21 70.41$\pm$ 0.08 5 (random) 500 full 61.01$\pm$ 0.11 80.41$\pm$
0.08 10 (random) 500 full 63.01$\pm$ 0.30 81.24$\pm$ 0.02

Figure 6: Accuracy results for training and validation on R2-D2 base-learner
[8] with a DBT-regularized ResNet-12 backbone on the CIFAR-FS dataset. (Top
Left) Baseline model and (Top right) Augmentation ”Aug” and MaxUp. The MaxUp
augmentation strategy narrows down the generalization gap and reduces
overfitting. (Bottom left) 1-shot classification and (Bottom Right) 5-shot
classification for Query data augmentation.

Figure 7:  Accuracy plots for different datasets compared to the baseline
model. Augmentations techniques were applied on Task ”T” and Support ”S”
datasets. Overall, accuracy for 5-shot is maintained at 85-88% while for
1-shot, a range of 65-68% is recorded.

Table 2: Few-shot classification accuracy (%) using R2-D2 base leaner with a
ResNet-12 backbone on the CIFAR-FS dataset. Support(S), Query(Q) and Task(T)
data are used on different augmentation strategies.

Augmentation 1-shot 5-shot CutMix(Q) 76.01$\pm$ 0.21 87.14$\pm$ 0.08 \+
CutMix(S) 75.11$\pm$ 0.31 85.30$\pm$ 0.14 \+ Horizontal Flip(S) 76.32$\pm$
0.11 87.01$\pm$ 0.23 \+ Rotation(T) 75.33$\pm$ 0.25 87.68$\pm$ 0.03 SelfMix(Q)
76.04$\pm$ 0.21 86.81$\pm$ 0.08 \+ CutMix(S) 76.19$\pm$ 0.29 86.35$\pm$ 0.16
\+ Horizontal Flip(S) 75.27$\pm$ 0.32 86.88$\pm$ 0.03 \+ Rotation(T)
75.61$\pm$ 0.22 87.40$\pm$ 0.18 MixUp(Q) 72.14$\pm$ 0.01 82.81$\pm$ 0.08 \+
CutMix(S) 71.03$\pm$ 0.29 85.15$\pm$ 0.11 \+ Horizontal Flip(S) 72.27$\pm$
0.10 83.08$\pm$ 0.01 \+ Rotation(T) 74.10$\pm$ 0.11 85.10$\pm$ 0.22

Table 3:  Comparison to prior work on miniImageNet and CIFAR-FS. Few-shot
classification accuracy (%) using R2-D2 base leaner a ”ResNet-12” and
”64-64-64-64” backbone on CIFAR-FS and miniImageNet datasets, respectively. We
applied Rotation (R) to the CutMix and Horizontal Flip (HF) to the SelfMix
augmentation modes. ”Q” denotes query data, ”S” represents support data and
”M” dentotes MaxUp.

CIFAR-FS 5-way miniImageNet 5-way DBT-model Backbone 1-shot 5-shot 1-shot
5-shot Basline(No Aug) ResNet-12 70.26 $\pm$ 0.61 83.12 $\pm$ 0.53 55.03 $\pm$
0.40 74.06 $\pm$ 0.24 CutMix(Q) ResNet-12 71.46 $\pm$ 0.24 84.32 $\pm$ 0.73
57.36 $\pm$ 0.24 74.46 $\pm$ 0.11 CutMix(Q) + M ResNet-12 72.00 $\pm$ 0.01
86.20 $\pm$ 0.61 58.13 $\pm$ 0.25 75.69 $\pm$ 0.74 SelfMix(S) + R ResNet-12
62.56 $\pm$ 0.54 79.82 $\pm$ 0.33 50.38 $\pm$ 0.63 71.44 $\pm$ 0.08 SelfMix(S)
+ M ResNet-12 63.51 $\pm$ 0.78 80.20 $\pm$ 0.66 57.31 $\pm$ 0.89 72.69 $\pm$
0.70 CutMix(S) + HF 64-64-64-64 60.56 $\pm$ 0.29 85.32 $\pm$ 0.73 62.26 $\pm$
0.63 79.28 $\pm$ 0.63 CutMix(S) + M 64-64-64-64 63.42 $\pm$ 0.17 86.33 $\pm$
0.66 63.31 $\pm$ 0.89 80.69 $\pm$ 0.54 SelfMix(Q) + HF 64-64-64-64 75.56 $\pm$
0.84 84.32 $\pm$ 0.73 66.31 $\pm$ 0.89 82.69 $\pm$ 0.74 SelfMix(Q) + M
64-64-64-64 76.42 $\pm$ 0.38 86.10 $\pm$ 0.36 67.39 $\pm$ 0.34 83.44 $\pm$
0.24

Table 4: Experimental results that compare prior work on the Stanford Dogs,
Stanford Cars and CIFAR-FS dataset. Average few-shot classification accuracy
with 95 % confidence intervals. The second column shows which kind of
embedding is employed, we used a 4-layer convolutional network with their
respective filters in each layer.

Stanford Dogs 5-way Stanford Cars 5-way CIFAR-FS 5-way Model 1-shot 5-shot
1-shot 5-shot 1-shot 5-shot Matching Networks [63] 35.80 $\pm$ 0.99 47.50
$\pm$ 1.03 34.80 $\pm$ 0.98 44.70 $\pm$ 1.03 61.16 $\pm$ 0.89 72.86 $\pm$ 0.70
MAML [14] 44.81 $\pm$ 0.34 58.68 $\pm$ 0.31 47.22 $\pm$ 0.39 61.21 $\pm$ 0.28
55.92 $\pm$ 0.95 72.09 $\pm$ 0.76 Relation Nets [55] 43.33 $\pm$ 0.42 55.23
$\pm$ 0.41 47.67 $\pm$ 0.47 60.59 $\pm$ 0.40 62.45 $\pm$ 0.98 76.11 $\pm$ 0.69
Prototypical Networks [53] 37.59 $\pm$ 1.00 48.19 $\pm$ 1.03 40.90 $\pm$ 1.01
52.93 $\pm$ 1.03 51.31 $\pm$ 0.91 70.77 $\pm$ 0.69 DN4 [35] 45.41 $\pm$ 0.76
63.51 $\pm$ 0.62 59.84 $\pm$ 0.80 88.65 $\pm$ 0.44 52.79 $\pm$ 0.86 81.45
$\pm$ 0.70 PABN [22] 45.65 $\pm$ 0.71 61.24 $\pm$ 0.62 54.44 $\pm$ 0.71 67.36
$\pm$ 0.61 63.56 $\pm$ 0.79 75.35 $\pm$ 0.58 MATANet [10] 55.63 $\pm$ 0.88
70.29 $\pm$ 0.62 73.15 $\pm$ 0.88 91.89 $\pm$ 0.45 67.33 $\pm$ 0.84 83.92
$\pm$ 0.63 GNN [49] 46.38 $\pm$ 0.78 62.27 $\pm$ 0.95 55.85 $\pm$ 0.97 71.25
$\pm$ 0.89 51.83 $\pm$ 0.48 63.69 $\pm$ 0.94 Rfs [60] 55.64 $\pm$ 0.28 62.02
$\pm$ 0.63 79.64 $\pm$ 0.44 69.74 $\pm$ 0.72 83.41 $\pm$ 0.55 83.50 $\pm$ 0.11
Rfs-distill [60] 56.01 $\pm$ 0.48 64.82 $\pm$ 0.60 82.14 $\pm$ 0.43 71.52
$\pm$ 0.69 86.03 $\pm$ 0.49 84.10 $\pm$ 0.28 DBT-baseline 56.06 $\pm$ 0.03
71.00 $\pm$ 0.25 73.49 $\pm$ 0.01 92.02 $\pm$ 0.33 74.41 $\pm$ 0.50 84.21
$\pm$ 0.65 \+ CutMix(Q) + R 56.36 $\pm$ 0.64 71.39 $\pm$ 0.04 73.69 $\pm$ 0.51
93.00 $\pm$ 0.15 74.81 $\pm$ 0.37 86.01 $\pm$ 0.67 \+ SelfMix(Q) + HF 56.86
$\pm$ 0.64 72.19 $\pm$ 0.78 74.21 $\pm$ 0.01 93.30 $\pm$ 0.35 75.01 $\pm$ 0.15
87.01 $\pm$ 0.74 \+ MaxUp 57.06 $\pm$ 0.63 73.15 $\pm$ 0.22 75.34 $\pm$ 0.41
94.38 $\pm$ 0.25 76.41 $\pm$ 0.25 87.68 $\pm$ 0.24

###  5.3 Augmentation modes

This section builds on the findings of section 5.1, where we established three
core data augmentation cases; support, query and task data augmentation.
Similar to [41, 17, 12], we used the CutMix, SelfMix, MixUp, Random Crop and
Horizontal Flip augmentation methods on the support, query and task datasets,
respectively. We identified the best augmentation combinations that suit a
few-shot learner and with our findings, we picked the best strategy to
determine which mode of augmentation suits a DBT-regularized few-shot learner.
To start with, we used the R2-D2 base learner [8] and the CIFAR-FS database to
evaluate the augmentation performance on support, query and task augmentations
as shown in Table 1. Our findings show that the pair of CutMix and SelfMix
augmentation produces the best results with over 2.5% in accuracy improvement
[41]. Other approaches lag behind in performance at about $\pm$ 3% for both
1-shot and 5-shot cases. Secondly, since the CutMix and SelfMix methods stand
out as the best augmentation approach for our setup, we used them as bases to
combine augmentations on the three data cases; support, query and task,
respectively as shown in Table 2. Model performance significantly improved
with the best case occurring when CutMix (query) is combined with SelfMix
(support).

Figure 8:  Model accuracy plots for CIFAR-FS and miniImageNet datasets on CNN
and DBT model baselines with augmentation ”Aug” and without Augmentation for
5-shot and 1-shot cases.

###  5.4 DBT-regularization with data augmentation

As discussed in section 1, DBT-based regularization improves model
generalization and intra-class feature expressiveness. Data augmentation on
the other hand creates sufficient data diversity which helps to mitigate
overfitting. In this section, we highlight the collective benefits of combing
a DBT-based regularizer with augmentation strategies for few-shot learning,
using different datasets.

Accuracy results with different datasets: In this section, we setup a testing
scheme where we evaluate our method over four runs which is quite similar to
techniques applied in [60]. We computed the mean accuracy as the accuracy for
every run, the experiments are conducted on the Stanford Dogs, Stanford Cars,
miniImageNet and CIFAR-FS datasets, respectively as shown in Table 4 and Table
3. Our accuracy results for 5-shot were maintained at 80-88% while for 1-shot
at a range of 65-68% as shown in Figure 7 and Figure 8. Our baseline
integrated with the DBT-based regularizer ”DBT-baseline” model performs at
about 2% better than state-of-the art without data augmentation. Applying the
CutMix and SelfMix augmentation on the query ”Q” and support ”S” datasets,
show significant improvement. Rotation ”R” and Horizontal Flip ”HF” are
integrated into the CutMix and SelfMix data augmentation modes, respectively
as indicated in Table 3.

Improvement with MaxUp augmentation: In this section, we evaluate the
performance of our model with the Max-Up approach for both 1-shot and 5-shot
classification. We use a similar experimentation setting described in [41] at
different augmentation pool sizes. Figure 6 and Table 4 depict the impact of
Max-Up with the augmentation strategies; denote generally as ”Aug” for both
train and validation data. We also show results for the baseline model without
augmentation (DBT-baseline), with CutMix and MaxUp augmentation for different
Query data schemes. We observe from Figure 6 (Top right) that the
generalization gap shrinks considerably and by implication, overfitting is
minimized when the MaxUp strategy is adapted. MaxUp also adds the extra boost
with an average of about 2.3% in performance.

Comparison with different methods: We compared our results against different
methods [49, 10, 22, 35] as shown in Table 4. We observed that [60] is closest
to ours but we outperform their approach significantly for the 5-shot cases by
over 6% on the average. We recorded a better performance than GNN [49] and
MATANet [11] using both the 5-way 1-shot and 5-way 5-shot few-shot learning
settings, we saw an improvement of about 3.3% 4.2% and 3.16% on Stanford Dogs,
Stan-ford Cars, and CIFAR-FS, respectively for 5-way 1-shot task while for the
5-way5-shot task, our method achieved about 4.7%, 2.1%, and 4.9% overall.
Clearly, the MaxUp boost is significant in almost all cases.

##  6 Conclusion

We proposed a structured doubly block-toeplitz (DBT) matrix based model that
imposes orthogonal regularization on the filters of the convolutional layers
termed Ortho-Shot. Our approach was aimed at maintaining the stability of
activations, preserving gradient norms, and enhancing feature transferability
of deep networks. We also broke down the pipeline of a few-shot learner and
based on our findings, we established three augmentations strategies that aid
in minimizing overfitting and increasing data diversity. Our findings and
empirical results confirm that a DBT regularized model is beneficial to few-
shot classification and meta-learning in general.

## References

  * [1] A. Araujo, B. Négrevergne, Y. Chevaleyre, and J. Atif.  On lipschitz regularization of convolutional layers using Toeplitz matrix theory.  arXiv: Learning, 2020.
  * [2] R. Balestriero and R. Baraniuk.  Mad Max: Affine spline insights into deep learning.  arXiv: Machine Learning, 2018.
  * [3] R. Balestriero and R. Baraniuk.  A spline theory of deep networks.  In ICML, 2018.
  * [4] N. Bansal, X. Chen, and Z. Wang.  Can we gain more from orthogonality regularizations in training deep cnns?  ArXiv, abs/1810.09102, 2018.
  * [5] P. Bartlett and W. Maass.  Vapnik-Chervonenkis dimension of neural nets.  2003\.
  * [6] P. Bartlett, V. Maiorov, and R. Meir.  Almost linear VC-dimension bounds for piecewise polynomial networks.  Neural Computation, 10:2159–2173, 1998.
  * [7] Y. Bengio, P. Simard, and P. Frasconi.  Learning long-term dependencies with gradient descent is difficult.  IEEE transactions on neural networks, 5 2:157–66, 1994.
  * [8] L. Bertinetto, J. F. Henriques, P. H. S. Torr, and A. Vedaldi.  Meta-learning with differentiable closed-form solvers.  ArXiv, abs/1805.08136, 2019.
  * [9] A. Brock, J. Donahue, and K. Simonyan.  Large scale gan training for high fidelity natural image synthesis.  ArXiv, abs/1809.11096, 2019.
  * [10] H. Chen, H. Li, Y. Li, and C. Chen.  Multi-scale adaptive task attention network for few-shot learning.  ArXiv, abs/2011.14479, 2020.
  * [11] Y. Chen, X. Wang, Z. Liu, H. Xu, and T. Darrell.  A new meta-baseline for few-shot learning.  ArXiv, abs/2003.04390, 2020.
  * [12] A. Dabouei, S. Soleymani, F. Taherkhani, and N. Nasrabadi.  Supermix: Supervising the mixing data augmentation.  ArXiv, abs/2003.05034, 2020.
  * [13] Y. Dauphin, R. Pascanu, Çaglar Gülçehre, K. Cho, S. Ganguli, and Y. Bengio.  Identifying and attacking the saddle point problem in high-dimensional non-convex optimization.  In NIPS, 2014.
  * [14] C. Finn, P. Abbeel, and S. Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In ICML, 2017.
  * [15] S. Gidaris and N. Komodakis.  Dynamic few-shot visual learning without forgetting.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4367–4375, 2018.
  * [16] X. Glorot and Y. Bengio.  Understanding the difficulty of training deep feed forward neural networks.  In AISTATS, 2010.
  * [17] C. Gong, T. Ren, M. Ye, and Q. Liu.  Maxup: A simple way to improve generalization of neural network training.  ArXiv, abs/2002.09024, 2020.
  * [18] R. Gray.  Toeplitz and circulant matrices: A review.  Found. Trends Commun. Inf. Theory, 2, 2005.
  * [19] P. Hansen.  Deconvolution and regularization with Toeplitz matrices.  Numerical Algorithms, 29:323–378, 2004.
  * [20] M. Harandi and B. Fernando.  Generalized backpropagation, étude de cas: Orthogonality.  ArXiv, abs/1611.05927, 2016.
  * [21] N. Harvey, C. Liaw, and A. Mehrabian.  Nearly-tight VC-dimension bounds for piecewise linear neural networks.  ArXiv, abs/1703.02930, 2017.
  * [22] H. Huang, J. Zhang, J. yu Zhang, J. Xu, and Q. Wu.  Low-rank pairwise alignment bilinear network for few-shot fine-grained image classification.  ArXiv, abs/1908.01313, 2019.
  * [23] L. Huang, X. Liu, B. Lang, A. W. Yu, and B. Li.  Orthogonal weight normalization: Solution to optimization over multiple dependent Stiefel manifolds in deep neural networks.  ArXiv, abs/1709.06079, 2018.
  * [24] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  ArXiv, abs/1502.03167, 2015.
  * [25] K. Jia, D. Tao, S. Gao, and X. Xu.  Improving training of deep neural networks via singular value bounding.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3994–4002, 2017.
  * [26] G. Kang, X. Dong, L. Zheng, and Y. Yang.  Patchshuffle regularization.  ArXiv, abs/1707.07103, 2017.
  * [27] T. Kietzmann, C. J. Spoerer, L. K. A. Sörensen, R. M. Cichy, O. Hauk, and N. Kriegeskorte.  Recurrence is required to capture the representational dynamics of the human visual system.  Proceedings of the National Academy of Sciences of the United States of America, 116:21854 – 21863, 2019.
  * [28] S. K. Kimitei.  Algorithms for Toeplitz matrices with applications to image deblurring.  Master’s thesis, Georgia State University Georgia State University, Department of Mathematics., USA, 2011.
  * [29] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60:84 – 90, 2012.
  * [30] S. M. Kye, H. Lee, H. Kim, and S. J. Hwang.  Transductive few-shot learning with meta-learned confidence.  ArXiv, abs/2002.12017, 2020.
  * [31] Q. V. Le, A. Karpenko, J. Ngiam, and A. Ng.  ICA with reconstruction cost for efficient overcomplete feature learning.  In NIPS, 2011.
  * [32] K. Lee, S. Maji, A. Ravichandran, and S. Soatto.  Meta-learning with differentiable convex optimization.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10649–10657, 2019.
  * [33] H. Li, D. Eigen, S. F. Dodge, M. D. Zeiler, and X. Wang.  Finding task-relevant features for few-shot learning by category traversal.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–10, 2019.
  * [34] J. Li, Y. Wu, J. Zhao, and K. Lu.  Low-rank discriminant embedding for multiview learning.  IEEE Transactions on Cybernetics, 47:3516–3529, 2017.
  * [35] W. Li, L. Wang, J. Xu, J. Huo, Y. Gao, and J. Luo.  Revisiting local descriptor based image-to-class measure for few-shot learning.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7253–7260, 2019.
  * [36] J. Liu, F. Chao, and C.-M. Lin.  Task augmentation by rotating for meta-learning.  ArXiv, abs/2003.00804, 2020.
  * [37] D. Mishkin and J. Matas.  All you need is a good init.  CoRR, abs/1511.06422, 2016.
  * [38] S. Motiian, Q. Jones, S. M. Iranmanesh, and G. Doretto.  Few-shot adversarial domain adaptation.  In NIPS, 2017.
  * [39] F. Natterer.  Error bounds for Tikhonov regularization in Hilbert scales.  Applicable Analysis, 18:29–37, 1984.
  * [40] A. Nayebi, D. Bear, J. Kubilius, K. Kar, S. Ganguli, D. Sussillo, J. DiCarlo, and D. Yamins.  Task-driven convolutional recurrent models of the visual system.  In NeurIPS, 2018.
  * [41] R. Ni, M. Goldblum, A. Sharaf, K. Kong, and T. Goldstein.  Data augmentation for meta-learning.  In ICML, 2021.
  * [42] B. N. Oreshkin, P. R. López, and A. Lacoste.  Tadam: Task dependent adaptive metric for improved few-shot learning.  In NeurIPS, 2018.
  * [43] M. Ozay and T. Okatani.  Optimization on submanifolds of convolution kernels in CNNs.  ArXiv, abs/1610.07008, 2016.
  * [44] H. Qi, M. Brown, and D. Lowe.  Low-shot learning with imprinted weights.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5822–5830, 2018.
  * [45] S. Qiao, C. Liu, W. Shen, and A. Yuille.  Few-shot image recognition by predicting parameters from activations.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7229–7238, 2018.
  * [46] J. Rajendran, A. Irpan, and E. Jang.  Meta-learning requires meta-augmentation.  ArXiv, abs/2007.05549, 2020.
  * [47] M. Ren, E. Triantafillou, S. Ravi, J. Snell, K. Swersky, J. Tenenbaum, H. Larochelle, and R. Zemel.  Meta-learning for semi-supervised few-shot classification.  ArXiv, abs/1803.00676, 2018.
  * [48] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell.  Meta-learning with latent embedding optimization.  ArXiv, abs/1807.05960, 2019.
  * [49] V. G. Satorras and J. Bruna.  Few-shot learning with graph neural networks.  ArXiv, abs/1711.04043, 2018.
  * [50] J. Seo, H. G. Jung, and S.-W. Lee.  Self-augmentation: Generalizing deep networks to unseen classes for few-shot learning.  Neural networks : the official journal of the International Neural Network Society, 138:140–149, 2021.
  * [51] C. Shorten and T. Khoshgoftaar.  A survey on image data augmentation for deep learning.  Journal of Big Data, 6:1–48, 2019.
  * [52] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  CoRR, abs/1409.1556, 2015.
  * [53] J. Snell, K. Swersky, and R. Zemel.  Prototypical networks for few-shot learning.  In NIPS, 2017.
  * [54] X. Su, H. Yin, Y. Chai, Y. Xiong, and X. Tan.  An improved Toeplitz measurement matrix for compressive sensing.  International Journal of Distributed Sensor Networks, 10, 2014.
  * [55] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. Torr, and T. M. Hospedales.  Learning to compare: Relation network for few-shot learning.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1199–1208, 2018.
  * [56] R. Takahashi, T. Matsubara, and K. Uehara.  Data augmentation using random image cropping and patching for deep cnns.  IEEE Transactions on Circuits and Systems for Video Technology, 30:2917–2931, 2020.
  * [57] L. Taylor and G. Nitschke.  Improving deep learning with generic data augmentation.  2018 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1542–1547, 2018.
  * [58] A. Thomas, A. Gu, T. Dao, A. Rudra, and C. Ré.  Learning compressed transforms with low displacement rank.  Advances in neural information processing systems, 2018:9052–9060, 2018.
  * [59] A. T. Thomas, A. Gu, T. Dao, A. Rudra, and C. Ré.  Learning compressed transforms with low displacement rank, 2019.
  * [60] Y. Tian, Y. Wang, D. Krishnan, J. Tenenbaum, and P. Isola.  Rethinking few-shot image classification: a good embedding is all you need?  ArXiv, abs/2003.11539, 2020.
  * [61] E. Triantafillou, R. Zemel, and R. Urtasun.  Few-shot learning through an information retrieval lens.  In NIPS, 2017.
  * [62] V. N. Vapnik.  The nature of statistical learning theory.  In Statistics for Engineering and Information Science, 2000.
  * [63] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra.  Matching networks for one-shot learning.  In NIPS, 2016.
  * [64] J. Wang, Y. Chen, R. Chakraborty, and S. X. Yu.  Orthogonal convolutional neural networks.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11502–11512, 2020.
  * [65] Y.-X. Wang, R. B. Girshick, M. Hebert, and B. Hariharan.  Low-shot learning from imaginary data.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7278–7286, 2018.
  * [66] Y.-X. Wang and M. Hebert.  Learning to learn: Model regression networks for easy small sample learning.  In ECCV, 2016.
  * [67] D. Xie, J. Xiong, and S. Pu.  All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5075–5084, 2017.
  * [68] H. Xu, Z. Wang, H. Yang, D. Liu, and J. Liu.  Learning simple thresholded features with sparse support recovery.  IEEE Transactions on Circuits and Systems for Video Technology, 30:970–982, 2020.
  * [69] H.-J. Ye, H. Hu, D. Zhan, and F. Sha.  Learning embedding adaptation for few-shot learning.  ArXiv, abs/1812.03664, 2018.
  * [70] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6022–6031, 2019.
  * [71] H. Zhang, M. Cissé, Y. Dauphin, and D. Lopez-Paz.  mixup: Beyond empirical risk minimization.  ArXiv, abs/1710.09412, 2018.
  * [72] L. Zhao, S. Liao, Y. Wang, J. Tang, and B. Yuan.  Theoretical properties for neural networks with weight matrices of low displacement rank.  ArXiv, abs/1703.00144, 2017.

Supplementary Materials

##  7 VC dimension and sample complexity

The Vapnik–Chervonenkis (VC) dimension is a measure of the capacity
(complexity, expressive power, richness, or flexibility) of a set of
functions. In our setting we focus on neural networks where all the weights
are of low discriminant rank (LDR) such as the Toeplitz-like, Hankel-like,
Vandermonde-like, and Cauchy-like matrices.

###  7.1 Bounding VC dimension

Theorem 1 For input $x\in\mathcal{X}$ and parameter $\theta\in\mathbb{R}^{W}$,
let $f(x,\theta)$ denote the output of the network. let $\mathcal{F}$ be the
class of functions $\\{x\xrightarrow{}f(x,\theta):\theta\in\mathbb{R}^{W}\\}$.
Let $\mathbf{W}_{l}$ be the number of parameters up to layer $l$ i.e the total
number of parameters in layer (1,2,…,$l$). we define the depth effective path
as:

$\displaystyle\bar{L}:=\frac{1}{W}\sum_{l=1}^{L}W_{l},$ (10)

Then the total number of computations units is given as :

$\displaystyle\mathbf{U}:=\sum_{l=0}^{L}n_{l}$ (11)

Inline with works of [6, 21, Warren1968LowerBF, Anthony1999NeuralNL].If k=1,
corresponding to piece-wise linear networks, it can be shown that:

$\displaystyle\mathbf{V}Cdim(sign\,\,\mathcal{F})=O(\bar{L}Wlog(pU))=O(LWlogW).$
(12)

Lemma 1.  Let $p_{q},...,p_{m}$ be polynomials of degree at most d in $n\leq
m$ variables, then we define:

$\displaystyle
K:=|\\{(sign(p_{1}(x)),...,(sign(p_{m}(x))):x\in\mathbb{R}^{n}\\}|,$ (13)

i.e., if $K$ is the number of possible sign vectors given by the polynomials,
then $K\leq 2(2emd/n)^{n}$. To partition the parameter space $\mathbb{R}^{W}$
for a fixed input $x_{j}$, the output $f(x,\theta)$ on each region in the
partition implies $\mathcal{S}={P_{1},...,P_{N}}$ of the parameter
$\mathbb{R}^{\mathbb{W}}$.

Hence, we have:

$\displaystyle
K\leq\sum_{j=1}^{N}|\\{(signf(x_{1},\theta)),...,signf(x_{m},\theta)):\theta\in
P_{j}\\}|,$ (14)

Hence from Lemma 1, we can show that by recursive construction,
$\mathcal{S}_{L-1}$ is a partition of $\mathbb{R}^{W}$ such that for
$\mathcal{S}\in\mathcal{S}_{L-1}$ [59]. The network output for input $x_{j}$
is a fixed polynomial of $\theta\in\mathcal{S}$ which collectively gives:

$\displaystyle\begin{split}K&\leq\sum_{j=1}^{N}|\\{(signf(x_{1},\theta)),...,signf(x_{m},\theta)):\theta\in
P_{j}\\}|\\\ &\leq 2(\frac{2emkd_{L}}{W_{L}})^{W_{L}},\end{split}$ (15)

with the size of $\mathcal{S}_{L-1}$ and equation (6) we get:

$\displaystyle{K}\leq\prod_{l=1}^{L}2(\frac{2emkd_{L}}{W_{L}})^{W_{L}}$ (16)

We can take logarithm and apply Jensen’s inequality, with $\bar{W}$ :=
$\sum_{l=1}^{L}W_{l}:$

$\displaystyle{K}$
$\displaystyle\leq\prod_{l=1}^{L}2(\frac{2emn_{l}pd_{l}}{W_{L}})^{W_{L}}$ (17)
$\displaystyle log_{2}{K}\leq
L+\sum_{l=1}^{L}W_{l}log_{2}2(\frac{2emn_{l}pd_{l}}{W_{L}})$
$\displaystyle=L+\bar{W}\sum_{l=1}^{L}\frac{W_{l}}{\bar{W}}log_{2}2(\frac{2emn_{l}pd_{l}}{W_{L}})$
$\displaystyle\leq
L+\bar{W}log_{2}\bigg{(}\sum_{l=1}^{L}\frac{W_{l}}{\bar{W}}\frac{2emn_{l}pd_{l}}{W_{L}}\bigg{)}$
$\displaystyle=L+\bar{W}log_{2}\frac{2emnp\sum_{l=1}^{L}n_{l}d_{l}}{\bar{W}}$

We bound $\sum_{l=1}^{L}n_{l}d_{l}$ using the bound on $d_{l}$; since the
degree of an LDR matrix $d_{l}$ is at most:

$\displaystyle\sum_{j=0}^{l-1}n_{l}d_{l}\leq
c_{1}k^{l-1}\sum_{j=0}^{l-1}n_{j}^{c_{2}}$ (18) $\displaystyle\leq
LUc_{1}k^{L-1}U^{c_{2}}\leq U^{c_{2}+2_{k}^{L}}$

where $c$ is a constant $L\leq U$ thus

$\displaystyle log_{2}{K}\leq L+\bar{W}log_{2}$
$\displaystyle\bigg{(}\frac{2c_{1}empU^{2+c_{2}k^{L}}}{\bar{W}}\bigg{)}$ (19)

To bound the VC-dimension, if VCdim(sign$\,\mathcal{F}$) = $m$ there exists
$m$ data points $x_{1},...,x_{m}$ such that the output of the model can have
$2^{n}$ sign patterns[62]. The bound on $log_{2}$K then implies:

$\displaystyle VCdim(sign\,\mathcal{F})\leq L$ (20)
$\displaystyle+\bar{W}log_{2}\bigg{(}\frac{2c_{1}epU^{2+c_{2}k^{L}}VCdim(sign\,\mathcal{F}}{\bar{W}}\bigg{)}$
$\displaystyle=O(\bar{L}Wlog(pU)+\bar{L}LWlogk)$

Hence completing the proof. Since the number of parameters $W$ is around the
square root of the number of parameters of a network (e.g doubly block
toeplitz based network) with unstructured layers, the sample complexity of an
LDR network is much smaller than that of unstructured networks (e.g CNN) which
is beneficial for deep networks.

Figure 9: Captions for each class shows the storage cost and operation count
for matrix-vector multiplication. Our proposed Toeplitz-like is of lowest
rank. compared to circulant, standard convolutional filters, and orthogonal
polynomial transforms.

###  7.2 Space and time complexity.

The proposed DBT-model has a time complexity of $O(nrlogn)$ and the small
number of parameters also makes the network perform better with limited amount
of training data which is crucial for few-shot learning [34]. We also ran
tests on the model backbone with two NVIDIA GeForce GTX 1080 Ti GPU and a
batch size of 64. Table 1 reflects the accuracy performance and we see an
overall model performance of about 4%. Similar to [Christiani2019FastLH], the
network used in our test consists of 4 convolutional layers, 1 fully-connected
layer and one softmax layer. Rectified linear units (ReLU) are used as the
activation units. Images were cropped to 24x24 and augmented with horizontal
flips, rotation, and scaling trans-formations. We use an initial learning rate
of 0.0001 and train for 800-400-100 epochs with their respective default
weight decay. Our efficient DBT-based approach obtains a test error of 6.61%,
compared to 5.26% obtained by the conventional CNN model. At the same time,
the DBT-based network is 4x more space efficient and 1.2x more time efficient
than the conventional CNN-based model.

###  7.3 Optimization setup.

An SGD optimizer with a momentum of 0.9 and a weight decay of $2\epsilon^{-5}$
was used for our setup. We used a learning rate initialized at 0.0001 with a
decay factor of 0.1 applied for all datasets. We trained over 100 epochs for
miniImageNet and 200 epochs for both CIFAR-FS and 150 epochs for Stanford Dogs
and Stanford Cats, respectively.

###  7.4 Architecture

The works of [42, 32, 60] used a ResNet12 as backbone for their model, we used
a similar structure but replace the convolutional layer with a doubly-block
toeplitz matrix the network consists of 4 residual blocks and 3 x 3 kernels. A
2x2 max-pooling layer is applied after each of the first 3 blocks; and a
global average-pooling layer is on top of the fourth block to generate feature
embeddings. Similar to [58], we used spectral regularization and changed the
number of filters from (64, 128, 256, 512) to (64, 160, 320, 640).

Usefulness of DBT regularization The DBT matrix represents a class of
structured matrices whose layers interact multiplicatively $(A^{i},B^{i})$ at
$O(nrlogn)$ time as compared to convolutional layers that are linear and
unstructured and are implemented in about $O(n^{2})$ time [34]. The generic
term structured matrix refers to an $n$x$m$ matrix that can be described in
fewer than $nm$ parameters and is capable of fast operation with at most
double the displacement rank, which is far simpler for computations [34].
Hence, if $\mathcal{F}$ denotes a class of neural networks comprising of $L$
DBT layers, $W$ total parameters and piece-wise linear activations, we can
measure the complexity, expressive power, richness, or flexibility of
$\mathcal{F}$ via a measure referred to as the Vapnik–Chervonenkis (VC)
dimension [62, 5].

For a simple classification problem of the form:
$\\{x\rightarrow{sign}\;f(x):f\in\mathcal{F}\\}$, the VC dimension
$(VC_{dim})$ of the class is expressed as:

$\displaystyle VC_{dim}(sign\;\mathcal{F})=O(LWlogW).$ (21)

$\mathcal{V}C_{dim}(\cdot)$ matches the standard bound for unconstrained
weight matrices [6, 21, 59].


††footnotetext: Copyright © 2021 for this paper by its authors. Use permitted
under Creative Commons License Attribution 4.0 International (CC BY
4.0).11institutetext: Faculty of Humanities and Social Sciences, University of
Zagreb, Zagreb 10000, Croatia
11email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# Multi-task Learning for Cross-Lingual Sentiment Analysis

Gaurish Thakkar  0000-0002-8119-5078 Nives Mikelic Preradović
0000-0001-9087-0074 Marko Tadić  0000-0001-6325-820X

###### Abstract

This paper presents a cross-lingual sentiment analysis of news articles using
zero-shot and few-shot learning. The study aims to classify the Croatian news
articles with positive, negative, and neutral sentiments using the Slovene
dataset. The system is based on a trilingual BERT-based model trained in three
languages: English, Slovene, Croatian. The paper analyses different setups
using datasets in two languages and proposes a simple multi-task model to
perform sentiment classification. The evaluation is performed using the few-
shot and zero-shot scenarios in single-task and multi-task experiments for
Croatian and Slovene.

###### Keywords:

sentiment analysis cross-lingual transfer learning multi-task learning news
sentiment under-resourced languages.

##  1 Introduction

Sentiment analysis is one of the most exciting applications of Natural
Language Processing. This field encompasses text analysis ranging from the
average customer reviews of online products and movies to user-generated text
from social media platforms. Other applications include the analysis of
financial news [2, 7] for stock market movement. While the field has a vast
span across multiple subareas, here we are interested in sentiment analysis of
news articles. This paper focuses on improving sentiment analysis of Croatian
news articles tagged with coarse-grained sentiment tags. Since Slovene and
Croatian belong to the same (sub)family of Slavic languages, Slovene was
chosen as the hub language and in the context of rather a small dataset
available for Croatian.

We seen the following items as the main contributions of our work:a) the
mutual dependence of the sentiment analysis task at three levels (i.e.,
document-level, paragraph-level, and sentence-level) is leveraged to improve
the performance of cross-lingual sentiment analysis; b) shared language
encoder representations across both languages are proposed and c) a language
from the same language family is used for cross-lingual sentiment
transfer111Source code: https://github.com/cleopatra-
itn/SentimentAnalyserSLHRNews.

##  2 Related Work

The previous state-of-the-art computational processes for sentiment analysis
relied on sentiment lexicons [17] as well as other classical methods like TF-
IDF [11] and dealing with various features along with SVM [13]. As automatic
feature extraction became a common trend with the usage of deep learning
techniques [15, 18], the Convolutional Neural Nets [21] took the lead, and
then gradually have been replaced by various Recurrent Neural Networks
approaches [9, 16, 23, 12].

The machine-translation, as one of the well-studied cross-lingual techniques,
also aids to sentiment labelling. Its application ranges from lexicon
translation [1, 3] up to instance translations [20]. Several recent works have
explored the use of transformers in multi-task learning setup for mono-lingual
[5] and multi-lingual setups [6, 22] for sentiment classification.

The closest work upon which our research is built [14] performs zero-shot
sentiment learning on Croatian news articles by enriching a masked language
model (mBERT) [8] using sentiment tags from the SentiNews and Croatian news
sentiment dataset. Our work is novel because of the way we use the dataset.
Previous work does not utilise paragraph-level and sentence-level annotations
for document-level classification but uses them for pre-training a masked-
language model. We utilise these annotations in a multi-task setup for aiding
the sentiment classification task for Croatian.

##  3 Datasets

We use two datasets in our experiments.

SentiNews Dataset in Slovene This is a manually annotated dataset
[4]222https://www.clarin.si/repository/xmlui/handle/11356/1110 in the domain
of news. It contains 10,427 documents. All annotations have three levels of
granularity, i.e., document, paragraph, and sentence-level. The dataset covers
news from economics, finance, and politics published between 1 September 2007
and 31 December 2013. It contains all instance annotations of each annotator,
along with the news content and the final sentiment label. The dataset has
been annotated at a five-point Likert scale and has mapping onto three labels
(Positive, Negative, and Neutral) using the scale’s average score. The overall
distribution of this dataset is given in Table 1.

Table 1: Slovene and Croatian dataset statistics. Lang | Level | Examples | Positive | Negative | Neutral
---|---|---|---|---|---
Slovene | Documents | 10,427 | 1,665 | 3,337 | 5,425
Paragraphs | 89,999 | 14,636 | 23,721 | 51,642
Sentences | 165,071 | 27,091 | 44,629 | 93,351
Croatian | Documents | 2,025 | 325 | 456 | 1,244

Sentiment Dataset in Croatian The Croatian dataset
333https://www.clarin.si/repository/xmlui/handle/11356/1342 was created using
guidelines similar to those of the SentiNews dataset. The text content comes
from the Croatian 24sata daily news portal. It covers topics such as health,
lifestyle, and automotive news. Table 1 shows the statistics for this dataset.
Like the Slovenian corpus, this dataset is also annotated with 3-class
sentiment labels and covers the same domain. However, it does not contain
paragraph and sentence-level annotations. Both datasets present an imbalanced-
class distribution phenomenon.

##  4 Methodology

We strive to leverage contextual information from the Slovene dataset
accessible at three distinct levels and dataset from Croatian in our proposed
system to promote knowledge transfer between two languages. As Croatian and
Slovene proved to have the highest level of mutual intelligibility among three
South Slavic languages (Croatian, Slovene and Bulgarian) represented in the
mutual intelligibility study [10], we hypothesise Slovene could be the right
candidate as a hub language for cross-lingual knowledge transfer to Croatian.
Also, both datasets belong to the same text type – news. A simple multi-task
learning setup with three different task heads is employed. Here, each task
head represents the classification layer responsible for classifying the given
instance of a particular type. The instances are of three types, namely
document-level, paragraph-level, and sentence-level. The document-
classification layer is trained by concatenating the Croatian and Slovene
dataset. The paragraph-classification and the sentence-classification layer
are only fed with Slovene instances since the Croatian dataset does not
provide this information.

A shared encoder is used for feature extraction, enabling feature sharing
across all three tasks and both languages. Our model is trained in five
different scenarios in total, while the overall architecture is presented in
Fig. 1. In the zero-shot learning setting, we do not use the Croatian data in
training. However, use the Croatian test for reporting the performance.

  1. 1.

Single-task-SL-Zero-shot-HR \- Train only Slovene document-level data. In this
setting, the Slovene data is used in the zero-shot setting for the Croatian
sentiment analysis.

  2. 2.

Multi-task-SL-Zero-shot-HR \- Train using all three levels on Slovene data.
This setting is the same as the previous one, except that all three
classification heads were learning respective tasks using Slovene data. We did
not use any Croatian data in this or previous setting.

  3. 3.

Single-task-HR \- Train only using Croatian document-level data. This setting
is a classic fine-tuning setup involving a single head learning classification
on a single dataset.

  4. 4.

Multi-task-HR+SL \- Train with Croatian (document-level) and Slovene data (all
three levels). We concatenated the respective document-level instances from
respective languages at the same time. The other heads were trained using
compatible instances from Slovene datasets.

  5. 5.

Single-task-HR+SL \- Train with Slovene and Croatian data (document-level).
This approach is similar to the previous one, but the single document-level
classification head is trained.

Figure 1: The overall architecture of the proposed system. A shared encoder
and three specific classification heads.

##  5 Experimental Setup

This section presents a brief description of the pre-processing steps,
followed by the details of the experiments.

###  5.1 Preprocessing

A few pre-processing steps were performed on the dataset.

  1. 1.

The empty string values, which have neutral tags, were dropped. As null
strings have no content, we decided to drop these instances.

  2. 2.

The strings based on content were de-duplicated. Many strings in the dataset
were duplicates. We performed this step in order to prevent leakage of the
instances into validation or test set split.

Table 2 depicts the overall distribution of the dataset. There is a drop of 3k
instances in the sentence-level Slovene dataset.

Table 2: Dataset statistics after preprocessing. Language | Level | Examples | Positive | Negative | Neutral
---|---|---|---|---|---
Slovene | Documents | 10,417 | 1,665 | 3,337 | 5,418
Paragraphs | 86,803 | 14,270 | 23,265 | 49,268
Sentences | 161,291 | 26,679 | 44,014 | 90,598
Croatian | Documents | 1,988 | 321 | 450 | 1,217

###  5.2 Language Model - Shared encoder

Our pipeline used a shared encoder based on a BERT-based masked language
model. The tri-lingual model named CroSloEngual [19] trained with three
languages (Slovene, Croatian, and English) on 5.9 billion tokens altogether.
This model was chosen as it considers two languages that we are interested in
transferring knowledge. The model outperforms the mBERT [8] for the task of
Part of Speech (POS) tagging, Named Entity Recognition (NER) and dependency
parsing for these three languages.

###  5.3 Experiments

In the datasets defined in Table 2, the data is split into 80:20 train-test
split in a stratified fashion. The 10% or a proportionate train set is kept
aside as a development set, since all the datasets differ in size. All
datasets are combined in a single collection. The combined datasets behave as
our size-proportional population. Our model is trained sequentially in a
single-task setting by randomly sampling tasks from this collection. Using the
test set from Croatian and Slovene, we evaluated our proposed approach. Table
3 shows the model configuration. All our models were trained using Nvidia RTX
3090 (24GB) with a batch size of 32. All hyperparameters were constant during
the whole experiments except epoch which varied from 5 for a single task to 3
for a multi-task setup. These values were chosen to prevent overfitting on the
train set. The overall training time for the MTL setup was 3 hours. We
evaluated the performance on the development set and chose the best model for
reporting test performance.

Table 3 reports all the results of our experiment.

Table 3: Model Configurations. Parameters | Values
---|---
Optimizer | Adam (lr =2e-5)
Loss | Categorical cross-entropy
Output | Softmax
Batch | 32
Epochs | 3-STL & 5-MTL
Dropout | 0.3
Labels | 3

###  5.4 Results

The single-task (STL) results and the multi-task (MTL) setup results are
reported in Table 4. Precision, recall, and F1 are macro averaged for all the
experiments. We used a simple majority-class classifier as our baseline. Our
MTL setup with Croatian and Slovene dataset outperforms the other settings for
Croatian sentiment classification. However, it does not perform best when
tested on document-level classification for Slovene. There is a small drop in
the performance similar to what was previously reported by [14]. The second-
best performing model for Croatian is the single-task variant with Slovene and
Croatian data. The worst performing model on Croatian was the MTL with only
Slovene data and the STL model, which was performing zero-shot learning.
Nevertheless, the SL MTL performs better on Slovene test-sets. The HR STL,
which used the least amount of data compared to other settings, seemed to
perform at par with SL STL in F1(55.61 vs 56.95), but both had contrasting
precision and recall.

For paragraph-level, SL MTL has similar performance to SL+HR MTL, but we see a
slight drop in performance in the latter case. A similar observation can be
made for sentence-level classification.

Table 4: Results of the experiments. Since the datasets are imbalanced, we
present macro-averaged precision, recall, and F1 measure.

Train set Test set Slovene Croatian Document Paragraph Sentence Document P R
F1 P R F1 P R F1 P R F1 Majority class 17.33 33.33 22.80 18.91 33.33 24.13
18.72 33.33 23.97 20.43 33.33 25.33 SL STL 70.56 71.65 71.07 - - - - - - 57.88
63.86 55.61 SL MTL 73.15 77.66 74.86 70.58 73.50 71.83 68.77 70.13 69.40 56.48
62.83 50.07 HR STL 56.84 44.92 43.86 - - - - - - 61.41 56.34 56.95 SL+HR STL
68.99 72.14 70.16 - - - - - - 62.80 64.66 63.53 SL+HR MTL 72.32 78.00 74.21
70.36 74.03 71.84 68.32 70.34 69.21 63.01 67.54 63.86

##  6 Discussion

The research presented in this paper has revealed that having a small amount
of target language (Croatian) data helps in overall cross-lingual transfer
learning. Adding data from another language hinders the source language task’s
performance but improves task performance in the target language. Comparing
our work with [14], which proposes joint optimisation of language modelling
task and paragraph-level sentiment analysis for the document-level sentiment
classification, we did not perform any intermediate training but utilised the
available annotation directly at the fine-tuning stage. Another significant
difference is the use of a trilingual shared encoder which performs better
than mBERT. We have qualitatively analysed the MTL model’s errors on the test
set and discovered that the model does pick up cues from the sentiment bearing
words from the input. However, the analysis of these errors would be beyond
the scope of this paper. The current model tags the news document as positive
or negative when it finds positive or negative words in neutral news. The
articles, which are advertisements or recipes, contain words with positive
sentiment. Most of the errors belong to this class. Since our encoder has
fixed-length input, the truncated text prevents correct classification. We can
solve this problem by performing a sliding window sampling over the text or
using the beginning and the end of each article.

##  7 Conclusion

We presented an overall setup for cross-lingual sentiment analysis (SA) of
Croatian news documents using a Multi-task learning approach. The goal was to
perform knowledge transfer using existing datasets and models to aid SA in
Croatian. For this purpose, we used a large Slovene SentiNews corpus created
similarly to the Croatian corpus.

The publicly available trained trilingual BERT-based language model for
feature representation was utilised and used as a shared encoder for the
downstream tasks. We combined the Croatian and Slovene datasets for document-
level classification and trained three different classification heads. The
results show that the MTL setup outperformed the STL setup for Croatian.

In the future, we would like to experiment more in an under-resourced setting
for Slovene and balance the dataset among distinct levels. Another interesting
approach would be to use each level of the datasets individually in a
hierarchical fashion to help with the next level’s classification task. For
example, using sentence-level features to aid the paragraph classification
could be fused into the document-level prediction process. We believe that
processing documents from different text types and topics separately would be
an easier task. Therefore, in our future work, we plan to cluster documents
into text-types and topics like recipes, advertisements or obituaries and
process them separately. Also, our experiment setting could be further checked
by running it for other language pairs supported by this CroSloEngual BERT-
based model: English-Slovene and English-Croatian. In this way, we could
verify whether the genetically and typologically distant language, as English
here is, would contribute to the performance.

##  8 Acknowledgements

The work presented in this paper has received funding from the European
Union’s Horizon 2020 research and innovation program under the Marie
Skłodowska-Curie grant agreement no. 812997 and under the name CLEOPATRA
(Cross-lingual Event-centric Open Analytics Research Academy).

## References

  * [1] Abdalla, M., Hirst, G.: Cross-lingual sentiment analysis without (good) translation. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 506–515. Asian Federation of Natural Language Processing, Taipei, Taiwan (Nov 2017), https://www.aclweb.org/anthology/I17-1051
  * [2] Agić, Ž., Ljubešić, N., Tadić, M.: Towards sentiment analysis of financial texts in croatian. In: Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10) (2010)
  * [3] Balahur, A., Steinberger, R., Kabadjov, M., Zavarella, V., van der Goot, E., Halkia, M., Pouliquen, B., Belyaeva, J.: Sentiment analysis in the news. In: Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10) (2010)
  * [4] Bučar, J., Žnidaršič, M., Povh, J.: Annotated news corpora and a lexicon for sentiment analysis in slovene. Language Resources and Evaluation 52(3), 895–919 (2018)
  * [5] Cer, D., Yang, Y., Kong, S.y., Hua, N., Limtiaco, N., St. John, R., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., Strope, B., Kurzweil, R.: Universal sentence encoder for English. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 169–174. Association for Computational Linguistics, Brussels, Belgium (Nov 2018). https://doi.org/10.18653/v1/D18-2029, https://www.aclweb.org/anthology/D18-2029
  * [6] Chidambaram, M., Yang, Y., Cer, D., Yuan, S., Sung, Y.H., Strope, B., Kurzweil, R.: Learning cross-lingual sentence representations via a multi-task dual-encoder model. arXiv preprint arXiv:1810.12836 (2018)
  * [7] Day, M.Y., Lee, C.C.: Deep learning for financial sentiment analysis on finance news providers. In: 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM). pp. 1127–1134. IEEE (2016)
  * [8] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [9] Dong, L., Wei, F., Tan, C., Tang, D., Zhou, M., Xu, K.: Adaptive recursive neural network for target-dependent twitter sentiment classification. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: Short papers). pp. 49–54 (2014)
  * [10] Golubović, J., Gooskens, C.: Mutual intelligibility between west and south slavic languages. Russian linguistics 39(3), 351–373 (2015)
  * [11] Lin, K.H.Y., Yang, C., Chen, H.H.: Emotion classification of online news articles from the reader’s perspective. In: 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology. vol. 1, pp. 220–226. IEEE (2008)
  * [12] Majumder, N., Poria, S., Hazarika, D., Mihalcea, R., Gelbukh, A., Cambria, E.: Dialoguernn: An attentive rnn for emotion detection in conversations. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 33, pp. 6818–6825 (2019)
  * [13] Pang, B., Lee, L., Vaithyanathan, S.: Thumbs up? sentiment classification using machine learning techniques. In: Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002). pp. 79–86. Association for Computational Linguistics (Jul 2002). https://doi.org/10.3115/1118693.1118704, https://www.aclweb.org/anthology/W02-1011
  * [14] Pelicon, A., Pranjić, M., Miljković, D., Škrlj, B., Pollak, S.: Zero-shot learning for cross-lingual news sentiment classification. Applied Sciences 10(17), 5993 (2020)
  * [15] Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 conference on empirical methods in natural language processing. pp. 1631–1642 (2013)
  * [16] Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215 (2014)
  * [17] Taboada, M., Brooke, J., Tofiloski, M., Voll, K., Stede, M.: Lexicon-based methods for sentiment analysis. Computational linguistics 37(2), 267–307 (2011)
  * [18] Tai, K.S., Socher, R., Manning, C.D.: Improved semantic representations from tree-structured long short-term memory networks. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 1556–1566 (2015)
  * [19] Ulčar, M., Robnik-Šikonja, M.: Finest bert and crosloengual bert. In: Sojka, P., Kopeček, I., Pala, K., Horák, A. (eds.) Text, Speech, and Dialogue. pp. 104–111. Springer International Publishing, Cham (2020)
  * [20] Wan, X.: Co-training for cross-lingual sentiment classification. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. pp. 235–243 (2009)
  * [21] Wang, W., Feng, S., Gao, W., Wang, D., Zhang, Y.: Personalized microblog sentiment classification via adversarial cross-lingual multi-task learning. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 338–348. Association for Computational Linguistics, Brussels, Belgium (Oct-Nov 2018). https://doi.org/10.18653/v1/D18-1031, https://www.aclweb.org/anthology/D18-1031
  * [22] Yang, Y., Cer, D., Ahmad, A., Guo, M., Law, J., Constant, N., Ábrego, G.H., Yuan, S., Tar, C., Sung, Y., Strope, B., Kurzweil, R.: Multilingual universal sentence encoder for semantic retrieval. CoRR abs/1907.04307 (2019), http://arxiv.org/abs/1907.04307
  * [23] Zadeh, A., Liang, P.P., Poria, S., Vij, P., Cambria, E., Morency, L.P.: Multi-attention recurrent network for human communication comprehension. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018)