# Towards Imbalanced Large Scale Multi-label Classification with Partially
Annotated Labels

1st Xin Zhang  University of South Carolina
Columbia, United States
<EMAIL_ADDRESS>2nd Yuqi Song  University of South Carolina
Columbia, United States
<EMAIL_ADDRESS>3rd Fei Zuo  University of Central Oklahoma
Oklahoma, United States
<EMAIL_ADDRESS>4th Xiaofeng Wang  University of South Carolina
Columbia, United States
<EMAIL_ADDRESS>
###### Abstract

Multi-label classification is a widely encountered problem in daily life,
where an instance can be associated with multiple classes. In theory, this is
a supervised learning method that requires a large amount of labeling.
However, annotating data is time-consuming and may be infeasible for huge
labeling spaces. In addition, label imbalance can limit the performance of
multi-label classifiers, especially when some labels are missing. Therefore,
it is meaningful to study how to train neural networks using partial labels.
In this work, we address the issue of label imbalance and investigate how to
train classifiers using partial labels in large labeling spaces. First, we
introduce the pseudo-labeling technique, which allows commonly adopted
networks to be applied in partially labeled settings without the need for
additional complex structures. Then, we propose a novel loss function that
leverages statistical information from existing datasets to effectively
alleviate the label imbalance problem. In addition, we design a dynamic
training scheme to reduce the dimension of the labeling space and further
mitigate the imbalance. Finally, we conduct extensive experiments on some
publicly available multi-label datasets such as COCO, NUS-WIDE, CUB, and Open
Images to demonstrate the effectiveness of the proposed approach. The results
show that our approach outperforms several state-of-the-art methods, and
surprisingly, in some partial labeling settings, our approach even exceeds the
methods trained with full labels.

###### Index Terms:

partial labels, multi-label classification, label imbalance

##  I Introduction

The task of classifying a set of instances, each associated with a unique
class label from a set of disjoint class labels, is known as multi-class (MC)
classification. MC classification is the most common task in the early stage
of the development of deep learning. However, as the importance of the role
assumed by deep learning increases, multi-label (ML) classification, a
generalized version of the MC problem, attracts more attention: due to the
intrinsic plurality of the physical world [1], the multi-label classification
is a more practical problem in our real lives, which allows the instances to
be associated with more than one class [2]. For example, a CT scan can detect
a variety of possible lesions to help people detect potential diseases early;
materials have multiple properties such as pyroelectricity and hardness [3];
and for vision-based autonomous driving, the ego-vehicle needs to examine
surroundings to find which objects are contained in the current scene and take
corresponding actions [4].

In theory, ML classification is a form of supervised learning that
necessitates a large number of accurate labels [5]. However, in practice,
annotating all labels for each training instance raises a great challenge in
multi-label classification, which is time-consuming and even impractical,
especially in the presence of a large number of categories [6, 7]. Therefore,
training a classifier with the partially annotated labels (PAL) setting
arouses extensive research interest. In this setting, for each instance, only
a subset of all the labels are annotated and the rest remains unknown. The
acquisition of datasets in the PAL setting is much simpler and saves more
annotation efforts compared with the scenario of the fully annotated labels
(FAL) setting. There are two variations of the PAL setting: the partially
annotated positive labels (PPL) and the single annotated positive label (SPL).
In the former case, all negative labels are discarded and only a portion of
positive labels are retained. However, in the latter case, there is only one
positive label and the others are missing, which is more extreme. Table I
shows the difference between these settings.

TABLE I: Different Missing-label Settings. ✓, $\times$, $?$ represent that the current instance belongs to this class, does not belong to this class, and lacks related labels, respectively. Settings | Class 1 | Class 2 | Class 3 | Class 4 | Class 5
---|---|---|---|---|---
FAL | ✓ | $\times$ | ✓ | $\times$ | ✓
PAL | ✓ | $?$ | $?$ | $\times$ | ✓
PPL | ✓ | $?$ | ✓ | $?$ | $?$
SPL | ✓ | $?$ | $?$ | $?$ | $?$

Label imbalance is an issue that requires extra attention for multi-label
classification problems [8, 4]. Even in the FAL setting, there is a
significant imbalance for many commonly used datasets. As an example, the COCO
dataset [9] has only 2.9 positive labels out of 80 labels per image, and in
the NUS-WIDE dataset [10], each image has an average of only 1.9 positive
labels and 78.1 negative labels. This issue is even severer for partially
labeled settings, where the few existing positive labels may also be discarded
(an image may correspond to several positive labels and zero negative label),
leading the classifier to generate trivial solutions where it simply predicts
all categories as negative. This problem significantly limits the performance
of classifiers and the research on the partially labeled classification. Most
existing approaches require that there is at least one positive label per
image, i.e., PPL or SPL setting instead of confronting these challenges
directly [6, 7]. In addition, scaling up to the problems with large labeling
space has not been sufficiently addressed[11].

In this paper, we focus on the issue of label imbalance and investigate how to
address it in partially labeled settings. Also, how to train classifiers using
partial labels with large labeling space will be taken into consideration.
Specifically, We divide the labels of each instance in the training set into
two parts, i.e., label existing and label non-existing. For the non-existing
part, we introduce pseudo-labels as the target when calculating the loss. At
the same time, we directly take the labels as the target for the existing
part. In this way, the conventional neural-networks-based classifier can be
directly applied to the partially labeled settings without adding other
auxiliary structures which usually introduce extra burdens for the computation
and training. Then, we take full advantage of the existing statistical
information of the dataset and design a novel dynamic loss function that
balances the model’s attention to both existing and non-existing parts during
the training. In addition, we propose a new training scheme to further
alleviate the label imbalance problem. During the training, we dynamically
adjust the size of the non-existing part by random sampling and continuously
modify the learning rate as training progresses to ensure that the model does
not produce trivial solutions due to a sudden influx of a large number of
negative labels.

The main contributions are summarized as follows:

  * •

We propose a method for training multi-label classification models using
partially labeled data based on the pseudo-labeling technique. This method can
effectively reduce the manual labeling effort while ensuring the accuracy of
the model.

  * •

We address the issue of label imbalance in multi-label problems, even in
large-scale labeling spaces, by designing sophisticated loss functions that
utilize the statistical information of existing datasets and dynamically
adjusting the training scheme.

  * •

The extensive experiments on four large-scale public image datasets (COCO,
NUS-WIDE, CUB, Open Images) demonstrate that our method outperforms the state-
of-the-art methods.

The rest of the paper is organized as follows. Section II discusses the
related work. The problem is formulated in Section III and our proposed method
is presented in Section IV. Section V shows the experimental results. Finally,
conclusions are drawn in Section VI.

##  II Related Work

From the early stages of computer vision, multi-label problems have been a hot
topic in the research community. Therefore, in this section, we will only
introduce the work that is most relevant to our research due to space
limitations.

###  II-A Partial Labels

Due to the extremely expensive manual labeling costs, ML tasks often involve
incomplete training data [1]. By treating the missing labels as negative
labels, the partially labeled setting then becomes a fully labeled learning
problem [12, 13, 14]. However, performance drops because a lot of ground-truth
positive labels are initialized as negative labels [15], i.e., labels
imbalance. Another solution is to consider each label prediction as an
independent binary classification problem [2]. But this approach is not
scalable when the number of categories grows and it ignores the correlations
amongst labels and the relations amongst instances [1]. In addition, there are
several works on propagating the information of existing labels to the missing
parts by utilizing label correlation techniques such as matrix completion
algorithm [16, 17], low-rank empirical risk minimization [11], and mixed graph
[18]. However, most of these works require solving an optimization problem
with the training set in memory, so it is impossible to use a mini-batch
strategy to fine-tune the model [1].

###  II-B Label Imbalance

Label imbalance is a huge challenge for lots of computer vision-related tasks,
such as multi-label classification and object detection [19]. The issue is
stated as there is only a small fraction of possible labels for most images,
which means that the number of positive labels is much lower than the number
of negatives. Several methods are presented to address it. [20] proposes a
loss function to alleviate the imbalance, but this loss function is only aimed
at long-tail distribution scenarios. Focal loss function [8], which is
presented to solve the imbalance in object detection first, can also be used
for ML classification problems. However, Focal treats the positive and
negative samples equally and it will result in the accumulation of more loss
gradients from negative samples and down-weighting of important contributions
from the rate positive samples [19]. Based on resampling methods, [21] selects
only a subset of the possible examples to solve the problem of imbalance in
object detection. But resampling methods are not suitable for multi-label
classification, since they cannot change the distribution of only a specific
label [19]. In the setting of PAL, this problem is more severe because the
number of positive labels may further decrease, and the phenomenon of
imbalance will become more obvious [4]. Therefore, studying how to solve
imbalance is of great significance.

###  II-C Large Labeling Space

The focus of recent research on ML classification has largely been shifted to
the issue that the number of labels is assumed to be extremely large, with the
key challenge being the design of scalable algorithms that offer real-time
predictions and have a small memory consumption [11]. Common solutions are
proposed that either reduce the dimension of the labels, such as the
Compressed Sensing Approach [22], CPLST [23], and Bayesian CS [24], or reduce
the feature dimension [25], or both, such as WSABIE [26]. However, these
previous work cannot handle partial labeling settings and is tied to a
specific loss function [11].

##  III Preliminary

###  III-A Multi-label Classification

Given a multi-label classifier with full labels, let
$\mathcal{X}=\mathbb{R}^{M}$ be the input attribute space of $M$-dimensional
feature vectors and $\mathcal{Y}=\\{1,2,\ldots,L\\}$ denotes the set of $L$
possible labels. An instance $\textbf{x}\in\mathcal{X}$ is associated with a
subset of labels $\textbf{y}\in 2^{\mathcal{Y}}$, which can be represented as
an $L$-vector $\textbf{y}=[y_{1},y_{2},\ldots,y_{L}]=\\{0,1\\}^{L}$ where
$y_{j}=1$ if and only if the $j$th label is relevant (otherwise $y_{j}=0$).
Let
$\mathcal{D}=\\{(\textbf{x}_{1},\textbf{y}_{1}),\ldots,(\textbf{x}_{N},\textbf{y}_{N})\\}$
is the training set of $N$ samples. Given $\mathcal{D}$, a multi-label
classifier $h:\mathcal{X}\rightarrow\mathcal{Y}$ learns to map the attribute
input space to the label output space. We use $\hat{\textbf{y}}$ to present
the prediction of classifier $h$, that is,
$\hat{\textbf{y}}=f(h(\textbf{x}))$, where $f(\cdot)$ stands for a function
(commonly the sigmoid function as $\sigma(s)=\frac{1}{1+e^{-s}}$) that turns
confidence outputs into a prediction. In this case, most of the existing work
[27, 2, 28, 29] adopts the binary cross entropy (BCE) function as the loss
function, which is formulated by

$\mathcal{L}(\hat{y},y)=-\frac{1}{L}\sum_{i=1}^{L}[(y_{i}\log(\hat{y}_{i})+(1-y_{i})\log(1-\hat{y}_{i})].$
(1)

For multi-label classification with missing-label, we use
$\textbf{z}=[z_{1},z_{2},\ldots,z_{L}]=\\{0,1,\varnothing\\}^{L}$ to represent
the observed labels, where $z_{j}=\varnothing$ means the corresponding label
is missing. It is worth mentioning that the training set is
$\mathcal{D}=\\{(\textbf{x}_{1},\textbf{z}_{1}),\ldots,(\textbf{x}_{N},\textbf{z}_{N})\\}$,
while validation and test set is
$\mathcal{D}_{v}=\mathcal{D}_{t}=\\{(\textbf{x}_{1},\textbf{y}_{1}),\ldots,(\textbf{x}_{N},\textbf{y}_{N})\\}$
in this case. In other words, we still use full labels for validation and
testing.

###  III-B Different Missing-label Settings

According to the number and positive/negative properties of the observable
labels, we divide the multi-label classification problem into several
settings: partially annotated labels (PAL), partially annotated positive
labels (PPL), and an extreme case, i.e., single annotated positive label (SPL)
[6]. Specially, we formulate these settings as the following:

$\displaystyle\textbf{z}_{PAL}=\\{0,1,\varnothing\\}^{L}\text{ and }$ (2)
$\displaystyle\sum_{j=1}^{L}(\mathbf{1}_{[\textbf{z}_{PAL_{j}}=1]}+\mathbf{1}_{[\textbf{z}_{PAL_{j}}=0]})<L,$
$\textbf{z}_{PPL}=\\{1,\varnothing\\}^{L}\text{ and
}\sum_{j=1}^{L}\mathbf{1}_{[\textbf{z}_{PPL_{j}}=1]}<L,$ (3)
$\textbf{z}_{SPL}=\\{1,\varnothing\\}^{L}\text{ and
}\sum_{j=1}^{L}\mathbf{1}_{[\textbf{z}_{PPL_{j}}=1]}=1,$ (4)

where $\mathbf{1}_{[\cdot]}$ stands for the indicator function, that is,
$\mathbf{1}_{[True]}=1$ and $\mathbf{1}_{[False]}=0$. Considering a large
number of unknown labels and the imbalance problem in these settings, we
design a new loss function to handle them, instead of directly using BCE (1).

##  IV Proposed Method

Overall, the pipeline of our proposed method is shown as Fig. 1. Next, we will
introduce three important components: the pseudo labels technique, the novel
loss function, and the dynamic training scheme.

Figure 1: The pipeline of our proposed method. The direction of training
progress is indicated from left to right. Our method does not rely on any
additional structures, so the model structure is composed only of a backbone
and FC layers. We input the statistical information of the current batch, the
predicted results obtained from training, and the ground-truth labels that the
current image has (existing parts, in light blue) and the pseudo labels (non-
existing parts), which are further divided into the picked parts (in red) and
unpicked parts (in dark blue), into the loss function we proposed for
calculation. The dynamic parameters in the loss function are continuously
adjusted as the training progresses. Then, the loss value is backpropagated to
update the parameters in the model.

###  IV-A Pseudo Labeling

For multi-label classification problems, the most commonly used solution is to
employ Binary Cross-Entropy (BCE) 1 as the loss function and train the model
using supervised learning to minimize the BCE loss value. However, in the case
of partial labeling settings, as some labels are missing for each image, we
cannot directly use BCE as the loss function. To avoid introducing the extra
overhead and thus reducing the efficiency of training and prediction (such as
the need to add an additional GNN network [1] or use another network with the
same structure as the label estimator [6]), we choose to introduce the pseudo-
labeling technique to transform the partially labeled setting into a fully
supervised learning method without changing the network structure.
Specifically, we divide the labels corresponding to the image into two
categories: one category corresponds to the existing labels, and the other
corresponds to the missing labels. In this way, by introducing pseudo labels,
we can apply commonly used supervised learning methods to partially labeled
settings without the need for major changes to the model structure.

The two key points of the pseudo-labeling technique are the initialization and
updating of pseudo labels [4]. In our method, we simply initialize all pseudo
labels to 0.5 and use the model’s prediction results as the updated value for
these pseudo labels. We formulate these processes as (5),

$\displaystyle\widetilde{y}_{i_{j}}=\begin{cases}\hat{y}_{i_{j}},&\textit{update}\\\
0.5,&\textit{initialization}\end{cases}$ (5)

where $\widetilde{y}_{i_{j}}$ and $\hat{y}_{i_{j}}$ stands for the pseudo
label and the prediction of model for $i$th image’s $j$th label, respectively.

###  IV-B Loss Function Designing

Our loss function is also divided into two parts, just like we divided the
labels corresponding to an image instance into the existing and the non-
existing parts. The overall loss function in our approach is shown as (6),

$\mathcal{L}=\mathcal{L}_{\text{Exist}}+\mathcal{L}_{\text{Non-exist}}.$ (6)

The simplest way to implement $\mathcal{L}_{\text{Exist}}$ and
$\mathcal{L}_{\text{Non-exist}}$ is to use BCE function. That is,

$\displaystyle\mathcal{L}_{\text{Exist}}$
$\displaystyle=\mathcal{L}_{BCE}(\hat{y},y),$ (7)
$\displaystyle\mathcal{L}_{\text{Non-exist}}$
$\displaystyle=\mathcal{L}_{BCE}(\hat{y},\widetilde{y}).$

However, this approach suffers from label imbalance, particularly when
considering the case where positive labels for the current image may not
exist, which may lead to trivial solutions. Therefore, our loss function does
not directly use BCE to implement it but adopts an imbalance-resistant design.
Specifically, we add a penalty coefficient to (7), which drives the model to
be more inclined to predict a positive label. After adding the penalty
coefficient and merging these two parts together, we obtained the loss
function (8),

$\displaystyle\mathcal{L}=$
$\displaystyle-\alpha[\frac{1}{N_{\text{e}}}\sum_{i=1}^{N_{\text{e}}}(y_{i}\log(\hat{y}_{i}))+\frac{1}{N_{\text{ne}}}\sum_{i=1}^{N_{\text{ne}}}(\widetilde{y}_{i}\log(\hat{y}_{i}))]$
(8)
$\displaystyle-\beta[\frac{1}{N_{\text{e}}}\sum_{i=1}^{N_{\text{e}}}((1-y_{i})\log(1-\hat{y}_{i}))$
$\displaystyle+\frac{1}{N_{\text{ne}}}\sum_{i=1}^{N_{\text{ne}}}((1-\widetilde{y}_{i})\log(1-\hat{y}_{i}))]$

where $\alpha$ and $\beta$ are penalty coefficients, $N_{\text{e}}$ and
$N_{\text{ne}}$ stand for the number of existing labels and pseudo labels for
the current image. During training, we work on a batch level rather than the
entire dataset, so the statistical information of the batch is more refined
and more instructive than that of the entire dataset. Therefore, in our
approach, the value of the penalty coefficient is obtained from the label
proportion information of the current batch, and the values of coefficients
are different for different batches. Specifically, we set
$\beta=\max(\frac{P}{T},\gamma)$ and $\alpha=1-\beta$, where $\gamma$ is a
hypermeter to avoid the value of $\alpha$ is so large that it dominates the
loss calculation. We set $\gamma=0.05$, which is determined by extensive
experiments.

In addition, considering that the initial values of the pseudo labels are 0.5,
we can not treat the loss of the existing and non-exist parts equally in the
early stages of training. Our goal is for the model to prioritize the existing
part in the early stages of training and gradually shift its focus towards the
non-existing part. Therefore, we introduce a dynamic coefficient $d$, which is
determined by the current epoch index, as (9),

$d=e^{\frac{e_{c}+m}{e_{T}}},$ (9)

where $e_{c}$ and $e_{T}$ stand for the current epoch index and the total
epoch number respectively, and $m$ is a hyperparameter for the non-exist part,
which gradually increases with the progress of training. Obviously, when it is
the $e_{T}-m$-th epoch, the model treats the existing and non-existing parts
equally. If the current epoch is less than $e_{T}-m$, the model is more
inclined towards the existing part, while if it is greater than $e_{T}-m$, the
opposite is true. Therefore, the larger the value of $m$, the faster the model
will focus on the non-existent part. In our experiments, we set $m=3$ in the
following.

In addition, to further alleviate the label imbalance, we also designed a
penalty coefficient $p$ to current loss, as shown in (10),

$p=1+|sign(\sum_{i=1}^{N}max(\hat{y}_{i}-0.1,0))|,$ (10)

where $sign()$ is the sign function. When the prediction values for all
classes are less than 0.1, $p=2$, otherwise, $p=1$. Obviously, this penalty
term will attempt to prevent the model from producing all negative predictions
by doubling the current loss value for any input image. Finally, we obtain the
loss function as (11),

$\displaystyle\mathcal{L}=$
$\displaystyle-p\alpha[\frac{1}{N_{\text{e}}}\sum_{i=1}^{N_{\text{e}}}(y_{i}\log(\hat{y}_{i}))+\frac{d}{N_{\text{ne}}}\sum_{i=1}^{N_{\text{ne}}}(\widetilde{y}_{i}\log(\hat{y}_{i}))]$
(11)
$\displaystyle-p\beta[\frac{1}{N_{\text{e}}}\sum_{i=1}^{N_{\text{e}}}((1-y_{i})\log(1-\hat{y}_{i}))$
$\displaystyle+\frac{d}{N_{\text{ne}}}\sum_{i=1}^{N_{\text{ne}}}((1-\widetilde{y}_{i})\log(1-\hat{y}_{i}))]$

###  IV-C Dynamic Training Scheme

In our approach, we further suppress class imbalance and address the large
labeling space problem by modifying the training scheme. After dividing the
corresponding labels of each image into existing and non-existing parts, we
further split the non-existing part. Specifically, we randomly select $q$% of
the labels in the non-existing part to participate in training, ignoring the
rest. In the training process, our loss function is actually as shown in (12),

$\displaystyle\mathcal{L}=$
$\displaystyle-p\alpha[\frac{1}{N_{\text{e}}}\sum_{i=1}^{N_{\text{e}}}(y_{i}\log(\hat{y}_{i}))+\frac{d}{qN_{\text{ne}}}\sum_{i=1}^{qN_{\text{ne}}}(\widetilde{y}_{i}\log(\hat{y}_{i}))]$
(12)
$\displaystyle-p\beta[\frac{1}{N_{\text{e}}}\sum_{i=1}^{N_{\text{e}}}((1-y_{i})\log(1-\hat{y}_{i}))$
$\displaystyle+\frac{d}{qN_{\text{ne}}}\sum_{i=1}^{qN_{\text{ne}}}((1-\widetilde{y}_{i})\log(1-\hat{y}_{i}))]$
$\displaystyle+\mathcal{L}_{\text{ned}},$

where $\mathcal{L}_{\text{ned}}$ stands for the discarded part of non-
existing, and it always equals 0. It should be pointed out that our model
still predicts all categories, but only q% of the non-existing part is
involved in the loss calculation. This means that we control the proportion of
partially labeled data during training, and increase the proportion of
existing labels in the total training labels. This makes the model’s
prediction results more reliable.

BR is a related approach that breaks down the multi-label problem into several
binary classification problems, but its performance is inadequate as it fails
to account for the interdependence between labels, particularly in the context
of large labeling spaces. However, although our method also involves dividing
the labels, for each image we randomly partition the labels, and
theoretically, the selected labels for each training epoch are different.
Therefore, the randomness ensures that our method can retain the dependence
between labels while reducing the label space.

##  V Experiments

In this section, we first provide an introduction to the selected datasets and
the data pre-processing method to obtain their partially labeled versions.
Then, we describe the metrics and baseline methods used to evaluate our
approach. Finally, we analyze the experimental results.

###  V-A Datasets

We perform experiments on several standard multi-label datasets: COCO [9],
NUS-WIDE [10], CUB [30], and Open Images [31]. For these four datasets, we
split the training and testing sets as the official splitting method. The
statistical information for these datasets is summarized in Table II.
Specifically, the COCO, NUS, and CUB datasets are fully labeled. For COCO,
each image has an average of 2.9 positive labels and 77.1 negative labels. For
NUS and CUB, each image has an average of 1.9 positive labels and 78.1
negative labels, and 31.3 positive labels and 280.7 negative labels,
respectively. In contrast, Open Images only has partially labeled data, with
each image corresponding to 600 labels. However, on average, only 5 labels per
image are present.

TABLE II: The statistical information of the four datasets. | COCO | NUS | CUB | Open Images
---|---|---|---|---
The size of training set | 82,081 | 126,034 | 5,994 | 5,655,108
The size of test set | 40,137 | 84,226 | 5,794 | 125,436
Positive per image | 2.9 | 1.9 | 31.3 | 2.4
Negative per image | 77.1 | 78.1 | 280.7 | 2.6
Total categories | 80 | 80 | 312 | 600

In order to obtain the partially labeled variants of COCO, NUS-WIDE and CUB,
inspired by [6], We randomly select a certain proportion of labels to be
retained and discard the rest. Specifically, for SPL, we keep only one
positive label and discard all other labels. For PPL, we retain a certain
proportion of positive labels and discard the remaining negative labels. And
for PAL, we randomly keep a certain proportion of labels, regardless of
whether they are positive or negative. Therefore, in PAL setting, an image may
correspond to several negative labels but zero positive labels. It should be
noted that for these datasets, we only perform the above pre-processing
methods once before training, and will not repeat them in subsequent training
to ensure that the randomness of label selection will not affect the
reliability of the experimental results.

As for the data augmentation, we first resize the original input image of all
these three datasets to the shape of 448 × 448. And then, the horizontal flip
is applied with a probability of 0.5. In the end, by using the standard
ImageNet statistics, we normalize the input images.

###  V-B Baselines and Metrics

In order to prove the effectiveness of our method, we first choose BCE and
BCE-LS in full labeling settings as baselines. These two methods are the most
commonly used methods in multi-label problems. The former takes BCE (1) as the
loss function and the latter uses label smoothing BCE [32], which is proposed
to reduce overfitting and has been shown to be effective in mitigating the
negative impacts of label noises.

In addition, we also compare our method with AN [33], WAN [34], ROLE [6],
Focal [8], and ASL [19]. These methods are implemented slightly differently
for different settings. For AN, WAN, and ROLE, we directly use the authors’
source code and training methods without any modification in the PPL setting.
But as mentioned in [4], these algorithms require that each image has at least
one available positive label. Therefore, before applying these methods to the
PAL setting, we removed the restriction on the number of positive labels in
the source code while keeping the rest unchanged. As for Focal and ASL, these
two methods are originally designed to solve the label imbalance problem and
cannot handle partially labeled problems. Therefore, to apply them to these
partial settings and compare them with our method, we follow [12, 34, 35] to
treat missing labels as negative labels.

As for the metrics, following [1, 6, 4], we also utilize the mean average
precision (mAP) to evaluate our approach.

###  V-C Network Structure

We use an end-to-end network for all experiments: a ResNet-50 [36], pre-
trained on ImageNet [37], as the backbone and fully connected layers, which is
the same as the multi-label classifier under FOL setting. Our approach does
not add any extra structure to the network.

###  V-D Experimental Results

We first compare our approach with other baseline methods in four different
settings on COCO, NUS-WIDE and CUB.

FAL. In FAL, all the labels are available. The results are summarized in Table
III. It can be seen that our method outperforms other methods on all three
datasets. Compared to the commonly used BCE loss function, our method improves
performance by 4.3%, 3.6%, and 3.8% on COCO, NUS-WIDE , and CUB, respectively.
In comparison to Focal and ASL loss functions designed for dealing with labels
imbalance, our method also shows significant improvements on all three
datasets.

TABLE III: The mAP results of our method and other baseline methods in FAL setting. We bold the best performance and underline the second best. |  COCO | NUS-WIDE |  CUB
---|---|---|---
BCE | 75.8 | 52.6 | 32.1
BCE-LS | 76.8 | 53.5 | 32.6
Focal | 79.6 | 55.3 | 34.8
ASL | 80.1 | 55.9 | 34.7
Ours | 81.1 | 56.2 | 35.9

PAL. For the three datasets, we keep 30%, 60%, and 90% of the labels, and we
refer to these settings as PAL_0.3, PAL_0.6, and PAL_0.9, respectively. The
experimental results are shown in Table IV. Our method achieves the best mAP
scores in all settings. On the COCO dataset, our method achieves mAP scores of
65.6, 76.3, and 81.0 when retaining 30%, 60%, and 90% of the labels,
respectively. On the NUS dataset, the mAP scores are 42.3, 49.7, and 56.0,
respectively. On the CUB dataset, the scores are 24.7, 32.2, and 36.0.
Compared with the second-best algorithm, our method shows a significant
improvement. Surprisingly, on the COCO and CUB datasets, we exceed the
performance of the BCE algorithm using only 60% of the labels. This indicates
that our method can save at least 40% of manual annotation costs. As AN, WAN,
and ROLE algorithms require at least one positive label per image, their
performance is significantly degraded in the PAL settings. This well
demonstrates the effectiveness of our algorithm.

TABLE IV: The mAP results in PAL settings. PAL_0.3, PAL_0.6, and PAL_0.9 stands for the missing ratio of labels is 70%, 40% and 10% respectively. We bold the best performance and underline the second best except BCE and BCE-LS there two strong baselines in these 9 different settings. | COCO | NUS-WIDE | CUB
---|---|---|---
BCE | 75.8 | 52.6 | 32.1
BCE-LS | 76.8 | 53.5 | 32.6
| PAL_0.3 | PAL_0.6 | PAL_0.9 | PAL_0.3 | PAL_0.6 | PAL_0.9 | PAL_0.3 | PAL_0.6 | PAL_0.9
AN | 50.1 | 65.6 | 72.7 | 29.4 | 40.5 | 50.1 | 13.2 | 19.2 | 27.4
WAN | 54.7 | 69.9 | 77.0 | 31.2 | 44.8 | 52.6 | 18.1 | 25.9 | 30.3
ROLE | 53.6 | 68.1 | 77.2 | 30.9 | 45.1 | 52.0 | 17.6 | 25.4 | 31.9
Focal | 62.1 | 71.6 | 78.8 | 37.6 | 45.9 | 55.0 | 19.0 | 26.7 | 34.2
ASL | 60.9 | 72.8 | 79.9 | 35.9 | 47.0 | 54.3 | 20.1 | 29.6 | 34.0
Ours | 65.6 | 76.3 | 81.0 | 42.3 | 49.7 | 56.0 | 24.7 | 32.2 | 36.0

PPL. For the PPL setting, we only keep a certain percentage of positive labels
and discard all negative labels. Specifically, we keep 30%, 60%, and 90% of
the positive labels for each dataset. The experimental results are summarized
in Table V. Our method achieves the best mAP scores in all settings of PPL.
Compared to the PAL setting, all methods show significant improvement, as the
proportion of positive labels increases. In this setting, our method can
outperform strong baseline algorithms such as BCE and BCE-LS with only 60% of
positive labels used.

TABLE V: The mAP results in PPL settings. PPL_0.3, PPL_0.6, and PPL_0.9 stands for the missing ratio of positive labels is 70%, 40%, and 10% respectively. We bold the best performance and underline the second best except BCE and BCE-LS there are two strong baselines in these 9 different settings. | COCO | NUS-WIDE | CUB
---|---|---|---
BCE | 75.8 | 52.6 | 32.1
BCE-LS | 76.8 | 53.5 | 32.6
| PPL_0.3 | PPL_0.6 | PPL_0.9 | PPL_0.3 | PPL_0.6 | PPL_0.9 | PPL_0.3 | PPL_0.6 | PPL_0.9
AN | 63.2 | 67.0 | 69.8 | 44.2 | 48.3 | 50.6 | 18.6 | 26.3 | 31.7
WAN | 68.1 | 70.8 | 71.6 | 46.3 | 48.5 | 51.0 | 21.6 | 28.9 | 32.9
ROLE | 71.8 | 75.4 | 77.4 | 47.3 | 50.4 | 52.5 | 21.5 | 29.6 | 33.1
Focal | 72.3 | 75.0 | 76.2 | 48.1 | 51.2 | 52.1 | 22.7 | 30.2 | 34.6
ASL | 71.9 | 75.4 | 76.8 | 47.9 | 52.0 | 52.9 | 22.4 | 29.8 | 34.5
Ours | 73.1 | 80.1 | 83.6 | 50.2 | 54.7 | 59.9 | 26.4 | 33.8 | 37.9

SPL. For the SPL setting, we only keep one positive label for each image and
discard all other labels. Table. VI summarizes the results. It can be seen
that our method has higher mAP scores than all other methods, especially for
the COCO dataset, where our method is 6 points higher than the second-best
method. However, because we only use one positive label, our method still lags
behind the strong baseline even though it outperforms all the comparison
methods.

TABLE VI: The mAP results in SPL setting. We bold the best performance and underline the second best except BCE and BCE-LS. |  COCO | NUS-WIDE |  CUB
---|---|---|---
BCE | 75.8 | 52.6 | 32.1
BCE-LS | 76.8 | 53.5 | 32.6
AN | 64.1 | 42.0 | 19.1
WAN | 64.8 | 46.3 | 20.3
ROLE | 66.3 | 43.1 | 15.0
Ours | 72.0 | 48.3 | 24.7

Open Images. Because the Open Images dataset has only partial labels, we do
not perform pre-processing but run our method directly. The results are shown
in Fig. 2. Our method achieves the highest mAP score, and compared to BCE, our
method improves by 4 approximately.

Figure 2: The mAP scores on Open Images dataset. We select BCE, BCE-LS, FOCAL,
and ASL for comparison to prove the effectiveness of our approach.

###  V-E Ablation study

Finally, to analyze the importance of each contribution, we perform an
ablation study on COCO and NUS-WIDE datasets in the PAL_06 setting and PAL_06
setting. The experimental results are shown in Table VII. We remove $\alpha$
and $\beta$, which are described in (8), $d$ in (9), $p$ in (10), and $q$ in
(12), respectively. The results show that every component in our method plays
a positive role.

TABLE VII: The ablation study of our method. The results are shown in mAP metrics. | COCO | NUS-WIDE
---|---|---
PAL_06 | PPL_06 | PAL_06 | PPL_06
Our method | 76.3 | 80.1 | 49.7 | 54.7
remove $\alpha$ and $\beta$ Equ. 8 | 75.8 | 79.3 | 47.6 | 50.0
remove $d$ Equ. 9 | 75.3 | 78.6 | 48.0 | 50.6
remove $p$ Equ. 10 | 74.9 | 79.0 | 45.4 | 49.0
remove $q$ Equ.12 | 74.3 | 78.9 | 45.6 | 48.9

##  VI Conclusion

In this work, we investigate the problem of training multi-label classifiers
using partially labeled data to significantly reduce manual labeling costs
while maintaining model accuracy. We specifically address the issue of label
imbalance, which has long been a challenge in multi-classification problems
and is particularly pronounced in the context of partially labeled data. To
address this issue, we first introduce the pseudo-labeling technique, which
enables the conventional multi-label model to be directly applied in partial
labeling settings without requiring additional structures. Then, by designing
a novel loss function and presenting a dynamically changing training scheme,
we effectively alleviate the label imbalance, even in large-scale label
spaces. Lastly, extensive experiments on four commonly used image datasets
including COCO, NUS-WIDE, CUB, and Open Images, demonstrate the effectiveness
of our proposed approach.

## References

  * [1] T. Durand, N. Mehrasa, and G. Mori, “Learning a deep convnet for multi-label classification with partial labels,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 647–657.
  * [2] G. Tsoumakas and I. Katakis, “Multi-label classification: An overview,” _International Journal of Data Warehousing and Mining (IJDWM)_ , vol. 3, no. 3, pp. 1–13, 2007.
  * [3] Y. Song, E. M. D. Siriwardane, Y. Zhao, and J. Hu, “Computational discovery of new 2d materials using deep learning generative models,” _ACS Applied Materials & Interfaces_, vol. 13, no. 45, pp. 53 303–53 313, 2021.
  * [4] X. Zhang, R. Abdelfattah, Y. Song, and X. Wang, “An effective approach for multi-label classification with missing labels,” _arXiv preprint arXiv:2210.13651_ , 2022.
  * [5] T. Hastie, R. Tibshirani, and J. Friedman, “Overview of supervised learning,” in _The elements of statistical learning_. Springer, 2009, pp. 9–41.
  * [6] E. Cole, O. Mac Aodha, T. Lorieul, P. Perona, D. Morris, and N. Jojic, “Multi-label learning from single positive labels,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 933–942.
  * [7] Y. Zhang, Y. Cheng, X. Huang, F. Wen, R. Feng, Y. Li, and Y. Guo, “Simple and robust loss design for multi-label learning with missing labels,” _arXiv preprint arXiv:2112.07368_ , 2021.
  * [8] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [9] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [10] T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng, “Nus-wide: a real-world web image database from national university of singapore,” in _Proceedings of the ACM international conference on image and video retrieval_ , 2009, pp. 1–9.
  * [11] H.-F. Yu, P. Jain, P. Kar, and I. Dhillon, “Large-scale multi-label learning with missing labels,” in _International conference on machine learning_. PMLR, 2014, pp. 593–601.
  * [12] S. S. Bucak, R. Jin, and A. K. Jain, “Multi-label learning with incomplete class assignments,” in _CVPR 2011_. IEEE, 2011, pp. 2801–2808.
  * [13] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. Van Der Maaten, “Exploring the limits of weakly supervised pretraining,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 181–196.
  * [14] M. Chen, A. Zheng, and K. Weinberger, “Fast image tagging,” in _International conference on machine learning_. PMLR, 2013, pp. 1274–1282.
  * [15] A. Joulin, L. v. d. Maaten, A. Jabri, and N. Vasilache, “Learning visual features from large weakly supervised data,” in _European Conference on Computer Vision_. Springer, 2016, pp. 67–84.
  * [16] R. Cabral, F. Torre, J. P. Costeira, and A. Bernardino, “Matrix completion for multi-label image classification,” _Advances in neural information processing systems_ , vol. 24, 2011.
  * [17] M. Xu, R. Jin, and Z.-H. Zhou, “Speedup matrix completion with side information: Application to multi-label learning,” _Advances in neural information processing systems_ , vol. 26, 2013.
  * [18] B. Wu, S. Lyu, and B. Ghanem, “Ml-mg: Multi-label learning with missing labels using a mixed graph,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 4157–4165.
  * [19] T. Ridnik, E. Ben-Baruch, N. Zamir, A. Noy, I. Friedman, M. Protter, and L. Zelnik-Manor, “Asymmetric loss for multi-label classification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 82–91.
  * [20] T. Wu, Q. Huang, Z. Liu, Y. Wang, and D. Lin, “Distribution-balanced loss for multi-label classification in long-tailed datasets,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16_. Springer, 2020, pp. 162–178.
  * [21] K. Oksuz, B. C. Cam, S. Kalkan, and E. Akbas, “Imbalance problems in object detection: A review,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 43, no. 10, pp. 3388–3415, 2020.
  * [22] D. J. Hsu, S. M. Kakade, J. Langford, and T. Zhang, “Multi-label prediction via compressed sensing,” _Advances in neural information processing systems_ , vol. 22, 2009.
  * [23] Y.-N. Chen and H.-T. Lin, “Feature-aware label space dimension reduction for multi-label classification,” _Advances in neural information processing systems_ , vol. 25, 2012.
  * [24] A. Kapoor, R. Viswanathan, and P. Jain, “Multilabel classification using bayesian compressed sensing,” _Advances in neural information processing systems_ , vol. 25, 2012.
  * [25] L. Sun, S. Ji, and J. Ye, “Canonical correlation analysis for multilabel classification: A least-squares formulation, extensions, and analysis,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 33, no. 1, pp. 194–200, 2010.
  * [26] J. Weston, S. Bengio, and N. Usunier, “Large scale image annotation: learning to rank with joint word-image embeddings,” _Machine learning_ , vol. 81, pp. 21–35, 2010.
  * [27] J. Nam, J. Kim, E. Loza Mencía, I. Gurevych, and J. Fürnkranz, “Large-scale multi-label text classification—revisiting neural networks,” in _Joint european conference on machine learning and knowledge discovery in databases_. Springer, 2014, pp. 437–452.
  * [28] J. Wang, Y. Yang, J. Mao, Z. Huang, C. Huang, and W. Xu, “Cnn-rnn: A unified framework for multi-label image classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2285–2294.
  * [29] M. S. Sorower, “A literature survey on algorithms for multi-label learning,” _Oregon State University, Corvallis_ , vol. 18, pp. 1–25, 2010.
  * [30] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “The caltech-ucsd birds-200-2011 dataset,” 2011.
  * [31] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov _et al._ , “The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale,” _International Journal of Computer Vision_ , vol. 128, no. 7, pp. 1956–1981, 2020.
  * [32] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826.
  * [33] K. Kundu and J. Tighe, “Exploiting weakly supervised visual patterns to learn from partial annotations,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 561–572, 2020.
  * [34] O. Mac Aodha, E. Cole, and P. Perona, “Presence-only geographical priors for fine-grained image classification,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 9596–9606.
  * [35] Y.-Y. Sun, Y. Zhang, and Z.-H. Zhou, “Multi-label learning with weak label,” in _Twenty-fourth AAAI conference on artificial intelligence_ , 2010.
  * [36] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1492–1500.
  * [37] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F. Li, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.