# Differences Between Hard and Noisy-labeled Samples:
An Empirical Study

Mahsa Forouzesh  Information and Network Dynamics Group (INDY)
School of Computer and Communication Sciences (IC)
École Polytechnique Fédérale de Lausanne  Patrick Thiran  Information and
Network Dynamics Group (INDY)
School of Computer and Communication Sciences (IC)
École Polytechnique Fédérale de Lausanne

###### Abstract

Extracting noisy or incorrectly labeled samples from a labeled dataset with
hard/difficult samples is an important yet under-explored topic. Two general
and often independent lines of work exist, one focuses on addressing noisy
labels, and another deals with hard samples. However, when both types of data
are present, most existing methods treat them equally, which results in a
decline in the overall performance of the model. In this paper, we first
design various synthetic datasets with custom hardness and noisiness levels
for different samples. Our proposed systematic empirical study enables us to
better understand the similarities and more importantly the differences
between hard-to-learn samples and incorrectly-labeled samples. These
controlled experiments pave the way for the development of methods that
distinguish between hard and noisy samples. Through our study, we introduce a
simple yet effective metric that filters out noisy-labeled samples while
keeping the hard samples. We study various data partitioning methods in the
presence of label noise and observe that filtering out noisy samples from hard
samples with this proposed metric results in the best datasets as evidenced by
the high test accuracy achieved after models are trained on the filtered
datasets. We demonstrate this for both our created synthetic datasets and for
datasets with real-world label noise. Furthermore, our proposed data
partitioning method significantly outperforms other methods when employed
within a semi-supervised learning framework111Code is available at
https://github.com/mahf93/Hard-vs-Noisy..

Index terms— hard samples, label noise, datasets, neural networks, image
classification

##  1 Introduction

Deep neural networks have revolutionized many applications, especially in the
field of image classification, mostly due to the availability of large, high-
quality labeled datasets (Rawat and Wang, 2017). In practice, obtaining such
datasets is often challenging, time-consuming, and expensive, thus leading to
the inclusion of label noise in the obtained datasets (Roh et al., 2019).
Label noise can arise for various reasons such as the use of cheap label
collection alternatives, for instance crowdsourcing, or the obtainment of the
label of an image from the accompanying text on the Web (Cordeiro and
Carneiro, 2020; Frenay and Verleysen, 2013; Algan and Ulusoy, 2020; Karimi et
al., 2020). The problem with label noise is that deep neural networks tend to
easily memorize these noisy labels, which can negatively impact their
generalization performance (Zhang et al., 2021b). Consequently, an important
line of research is to address this issue. Many methods propose to mitigate
the effects of label noise, including the use of robust loss functions (Ma et
al., 2020; Thulasidasan et al., 2019; Wang et al., 2019; Patrini et al., 2017)
and modifications to the training procedures (Yu et al., 2019; Zhang et al.,
2020; Jiang et al., 2018; Malach and Shalev-Shwartz, 2017). Another popular
approach to deal with label noise is to use a noisy-label detection method
(Nguyen et al., 2019; Li et al., 2020; Huang et al., 2019; Pleiss et al.,
2020b, a), where some of these methods may require an additional clean
validation set for hyper-parameter selection (Paul et al., 2021). Noisy-label
detection methods can involve data cleansing, where the noisy data is entirely
removed from the data set altogether, data re-weighting, where noisy data are
given lower weights during training, or re-labeling, where the noisy data is
re-annotated by experts. Alternatively, the detected noisy-labeled data could
be used as unlabeled data in a semi-supervised learning fashion.

When using noisy-label detection methods, an issue that often arises is their
inability to differentiate between noisy-labeled samples and hard-to-learn
samples. Hard-to-learn samples, also simply known as hard samples, refer to
samples in a dataset that are particularly challenging for the classifier to
learn (Arpit et al., 2017; Kishida and Nakayama, 2019; Wu et al., 2018).
Empirical observations have revealed that noisy samples and hard samples share
certain characteristics, such as high loss or low confidence or being learned
later in the training process. Consequently, when noisy label detection
methods are employed, which are based on these characteristics, hard samples
are also treated as noisy samples and are either filtered out or given lower
emphasis during training. Unfortunately, discarding hard samples may result in
gaps in the classifier’s knowledge of the true decision boundary, making it
crucial to preserve as many hard samples as possible and prioritize their
leaning (Chang et al., 2017; Bengio et al., 2009; Wang et al., 2020). It is
thus vital to propose noisy-label detection methods capable of distinguishing
between noisy-labeled samples and hard samples, keeping as many hard samples
as possible while removing noisy ones.

Recent studies that propose noisy-label detection methods which are claimed to
retain hard samples lack a quantitative evaluation of such claim, as there is
no precise measure to quantify sample hardness (Liang et al., 2022; Bai and
Liu, 2021; Zhu et al., 2021; Zhang et al., 2022). To overcome this limitation,
in this work, we propose synthetic dataset transformations to simulate varying
levels of hardness and noisiness. Although synthetic label noise has been
previously studied (Zhang et al., 2021b; Rolnick et al., 2017), our work, to
the best of our knowledge, is the first to introduce synthetic hardness levels
and to associate each sample of a dataset with a custom hardness level.

(a)

(b)

(c)

(d)

Figure 1: A schematic visualization of the hard sample creation approaches. We
transform an original dataset (a) into datasets with varying levels of
hardness for different classes (b)-(d). We consider three hardness types:
imbalance (b), diversification (c), and closeness to the decision boundary
(d). In all three transformations, we keep the samples of the red class
(circles) unchanged, and transform samples of the blue class (crosses) to
become harder (in (b) and (d)) or easier (in (c)), compared to the samples of
the red class. In each hardness type, the created dataset has samples with
custom hardness levels depending on the degree to which we apply the above
transformations.

There can be various reasons that a sample is hard to classify: it can be
under-represented in the data set, it can have distinct characteristics from
other samples in its class, or it can be close to the decision boundary. To
introduce synthetic hardness levels into an original dataset, we propose three
main approaches, which involve applying transformations to the original
dataset input samples to artificially make them more/less hard to classify. We
refer to these approaches as _hardness types_. We have: (i) imbalance, (ii)
diversification, and (iii) closeness to the decision boundary. In the first
hardness type (i), sample hardness is introduced by creating an imbalance
dataset, and subsampling different classes with varying cardinality. This
results in under-represented classes being harder to learn. In the second
hardness type (ii), hardness is introduced by making different classes more or
less distinct in their samples. This is achieved by applying a varying number
of data augmentations to different classes, thus resulting in classes with
fewer distinct samples and with more augmented samples per distinct sample
being easier to learn. In the third hardness type (iii), hardness is
introduced by modifying the input samples to be closer to the decision
boundary. The decision boundary is estimated using a pre-trained model with a
high test accuracy, and the samples are modified to be closer to this
estimated decision boundary. A schematic visualization of these different
hardness types and transformations from the original dataset is given in
Figure 1. We do not claim these approaches cover all contexts in which hard
samples arise, and in practice, a sample might be hard because of any
combination of the aforementioned reasons, or some other reason. Yet, these
common-sense approaches of introducing sample hardness enable us to perform
controlled experiments to assess different metrics/methods in terms of their
ability to distinguish between hard and noisy samples.

Our key observation is that the feature embeddings of hard samples become
closer to each other during training, whereas noisy-labeled samples do not
necessarily exhibit this behavior because of the visual dissimilarity to other
samples in the same class. This observation leads us to propose the distance
between the feature layer vector of each sample and the centroid feature
vector of its assigned class, which we call _static centroid distance_ (SCD),
as a metric to distinguish between hard and noisy-labeled samples. Next, we
propose a label noise detection method based on SCD. While other methods
perform well in only one of the two tasks among filtering out noisy samples
and retaining hard samples, our method is the only one that performs well in
_both_ tasks, and consistently so for all hardness types as well as datasets
with real-world label noise. We demonstrate the superior performance of this
method for noisy label detection when used for data cleansing as well as semi-
supervised learning. Our main contributions are as follows:

  * •

We propose a novel approach for synthetically transforming samples to attain
custom hardness levels across three hardness types: imbalance,
diversification, and closeness to the decision boundary. To the best of our
knowledge, we are the first to use controlled experiments to quantitatively
assess how different label noise detection methods perform in terms of
retaining hard samples.

  * •

We study various metrics for detecting noisy labels and show that static
centroid distance (SCD) is the most effective metric in distinguishing between
hard and noisy-labeled samples. While other metrics remain monotonic by
increasing either as a function of hardness or noisiness, we show that SCD is
the only metric that is increasing with noisiness but is _not_ increasing with
hardness.

  * •

We propose and evaluate different methods for data cleansing and sample
selection and we show that a two-dimensional Gaussian mixture model, which
uses the accuracy over the training and SCD as features, performs the best in
terms of filtering out noisy samples while retaining hard ones.

  * •

We empirically show that our method produces the best generalization
performance when models are trained on the filtered datasets. This holds both
in the synthetic datasets and even better in datasets with real-world label
noise. Moreover, if after the data filtration, semi-supervised learning is
applied, our method significantly outperforms other label noise detection
methods.

##  2 Background and Related Work

In this section, we first introduce the problem setup developed in this work
to produce datasets with custom noisiness and hardness levels. Next, we recall
some metrics from previous works that are relevant to our study, and, we
introduce static centroid distance (SCD). Finally, we present various
partitioning approaches that can be used in conjunction with each metric for
sample selection.

We assume that the training set $S$ can be partitioned as $S=S_{n}\cup
S_{e}\cup S_{h}$, where $S_{n}$ are the incorrectly-labeled samples (or noisy
samples), $S_{e}$ are the correctly-labeled and easy-to-learn samples, and
$S_{h}$ are the correctly-labeled and hard-to-learn samples. In practical
applications, it is often challenging to clearly distinguish between easy and
hard samples in a given dataset, and hence correctly place samples in $S_{e}$
and $S_{h}$. This can be a limitation when studying the performance of
different methods in terms of their ability to remove incorrectly labeled
samples while retaining hard samples. To address this issue, we synthetically
apply the transformation $\mathcal{T}$ to some original dataset
$S_{\text{org}}$, such that a spectrum of samples with hardness levels
$h\in\\{0,1,2,3,4\\}$ and noisiness levels $n\in\\{0,1,2,3,4\\}$ emerge:
samples are harder to learn as $h$ increases, and have noisier labels as $n$
increases. We will now discuss how $\mathcal{T}$ transforms $S_{\text{org}}$,
with no label noise and with uniform hardness levels among its samples, to a
dataset $S=\mathcal{T}(S_{\text{org}})$. This transformation provides us with
a knob that we can tune to make alterations, allowing for a systematic study
of easy, hard, and noisy samples while offering a good comparison base.

Consider a classification task with input $x\in\mathcal{X}$ and ground truth
one-hot label vector $y\in\\{0,1\\}^{K}$, where $K$ is the number of classes.
The original set with $N_{\text{org}}$ input-output pairs
$S_{\text{org}}=\\{(x_{i},{y}_{i})\\}_{i=1,\cdots,N_{\text{org}}}$ is
transformed into the given training set with $N$ training samples,
$S=\mathcal{T}(S_{\text{org}})=\\{(\tilde{x}_{i},\tilde{y}_{i})\\}_{i=1,\cdots,N}$.

Noisiness transformation Let
$S^{\prime}=\\{(x_{i},{y}_{i})\\}_{i=1,\cdots,N^{\prime}}\subset
S_{\text{org}}$ be a subset of size $N^{\prime}$ of the original dataset
$S_{\text{org}}$. Transformation $\mathcal{T}_{n}$ maps $S^{\prime}$ to
$S^{\prime}_{t}=\mathcal{T}_{n}(S^{\prime})=\\{(x_{i},\tilde{y}_{i})\\}_{i=1,\cdots,N^{\prime}}$,
where with probability $1-q(n)$, $\tilde{y}_{i}=y_{i}$, and with probability
$q(n)$, the non-zero element of $\tilde{y}_{i}$ is set uniformly at random at
index $j\sim U(1,2,\cdots,K)$. The label noise level $q(n)\in[0,1]$ is
adjusted as a function of the desired noisiness level $n\in\\{0,1,2,3,4\\}$.
Such a dataset transformation is common practice to study label noise in a
controlled setting, which is done by fixing some label noise level $q$ for the
entire dataset (Chatterjee, 2020). In our work, the label noise level $q(n)$
is not fixed for the entire dataset, and depending on the noisiness level $n$,
it varies between different subsets of samples.

Hardness transformation Let
$S^{\prime}=\\{(x_{i},{y}_{i})\\}_{i=1,\cdots,N^{\prime}}\subset
S_{\text{org}}$, be a subset of size $N^{\prime}$ from the original dataset
$S_{\text{org}}$. We want to transform $S^{\prime}$ such that the samples of
$S^{\prime}_{t}=\mathcal{T}_{h}(S^{\prime})$ have hardness level
$h\in\\{0,1,2,3,4\\}$. To the best of our knowledge, such a transformation has
not been done in prior work. This transformation is a composition of two
transformations, i.e., $\mathcal{T}_{h}=\mathcal{F}_{h}\circ\mathcal{I}_{h}$.
The first transformation $\mathcal{I}_{h}$ is one-to-one, and maps
$S^{\prime}$ into
$S^{\prime\prime}=\mathcal{I}_{h}(S^{\prime})=\\{(x_{j},{y}_{j})\\}_{j\in
J_{h}}$, where $J_{h}\subseteq\\{1,\cdots,N^{\prime}\\}$. The second
transformation $\mathcal{F}_{h}$ is one-to-many, and maps $S^{\prime\prime}$
into
$S^{\prime}_{t}=\mathcal{F}_{h}(S^{\prime\prime})=\\{(\tilde{x}_{i},{y}_{j})|i\in
I(j)\\}_{j\in J_{h}}$, where $\tilde{x}_{i\in I(j)}$ is a transformed version
of input sample $x_{j}$, and $I(j)$ can be a one-to-many mapping. In Section
3, we elaborate on the transformation $\mathcal{T}_{h}$ for each hardness
type.

The final transformed dataset with $N$ training samples is
$S=\cup_{S^{\prime}_{h,n}}\mathcal{T}_{n}\circ\mathcal{T}_{h}(S^{\prime}_{h,n})$,
where the subsets $S^{\prime}_{h,n}\subset S_{\text{org}}\forall
h,n\in\\{0,1,2,3,4\\}$ are determined according to a pre-defined policy. With
$S^{\prime}_{t,h,n}=\mathcal{T}_{n}\circ\mathcal{T}_{h}(S^{\prime}_{h,n})$,
and for some hardness threshold $h_{\text{threshold}}\in\\{0,1,2,3,4\\}$, the
three sets partitioning $S$ are

$\displaystyle S_{n}$ $\displaystyle=\\{(\tilde{x}_{i},\tilde{y}_{i})\in
S|\tilde{y}_{i}\neq y_{i}\\},$ $\displaystyle S_{h}$
$\displaystyle=\\{(\tilde{x}_{i},\tilde{y}_{i})\in\cup_{h\geq
h_{\text{threshold}}}S^{\prime}_{t,h,n}|\tilde{y}_{i}=y_{i}\\},$
$\displaystyle S_{e}$ $\displaystyle=S\setminus(S_{h}\cup S_{n}).$ (1)

In the dataset partition, we set $h_{\text{threshold}}=4$, hence the samples
in $S_{h}$ consist of the hardest samples in the dataset.

In this work, the classifier trained on the dataset $S$ is a neural network.
For each input sample $\tilde{x}_{i}$, let
$\textbf{p}(\tilde{x}_{i})=(p_{i}^{1},p_{i}^{2},\cdots,p_{i}^{K})$ be the
prediction probability output vector of the neural network. Furthermore, the
last layer of the neural network is a fully-connected layer with feature
vector $\textbf{h}(\tilde{x}_{i})\in\mathbb{R}^{m}$, where $m$ is the number
of units in the feature layer of the neural network. Note that because the
parameters of the neural network depend on the epoch during training, both
vectors $\textbf{p}(\tilde{x}_{i})$ and $\textbf{h}(\tilde{x}_{i})$ depend on
the epoch $t\in\\{1,2,\cdots,T\\}$, where $T$ is the maximum number of
training epochs. We often remove the explicit dependence on $t$ for
simplicity. Training is done by performing stochastic gradient descent (SGD)
on the cross-entropy training loss function at each epoch $t$:

$L_{S}(t)=\frac{1}{N}\sum_{i=1}^{N}l_{i}(t)=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{K}\tilde{y}_{i}^{j}\log
p_{i}^{j}(t)=-\frac{1}{N}\sum_{i=1}^{N}\log p_{i}^{c_{i}}(t)$

where $l_{i}(t)$ is the training cross-entropy loss of sample $\tilde{x}_{i}$
at epoch $t\in\\{1,2,\cdots,T\\}$. The prediction of the neural network
classifier at epoch $t$ for sample $\tilde{x}_{i}$ is class
$\tilde{c}_{i}=\operatorname*{arg\,max}_{j}p_{i}^{j}(t)$, and the confidence
of the classifier is the prediction probability $p_{i}^{\tilde{c}_{i}}$ of the
classifier for the predicted class label.

Metrics Here, we recall a few metrics that are introduced in prior work and
introduce a new metric, static centroid distance (SCD). In the following
sections, we perform a comprehensive study on all these metrics in order to
detect which metric, or combination of which metrics, is the best at
distinguishing between samples in $S_{n}$ and $S_{h}$. In order to remain on a
computationally limited budget, and on a practical setting, we study metrics
that require only a single model and not an ensemble of models, and that do
not require hyper-parameter tuning using a clean validation set. For each
sample $\tilde{x}_{i}$, we compute the following metrics defined below.
Throughout our study, we primarily focus on loss, confidence, and SCD metrics.
The former two are widely used in the literature due to the valuable
information they provide about each sample, while the latter is a metric
proposed in our work.

  * •

_Loss_ : The training loss $l_{i}(T)$ at the end of the training, i.e., at
epoch $T$.

  * •

_Confidence_ : The prediction probability $p_{i}^{\tilde{c}_{i}}(T)$ at the
end of the training, i.e., at epoch $T$.

  * •

First Prediction Epoch: The epoch $t^{*}$ such that
$\tilde{c}_{i}(t^{*})=c_{i}(t^{*})$ and $\tilde{c}_{i}(s)\neq c_{i}(s)$ for
$s<t^{*}$.

  * •

Accuracy of Predictions over Training: The accuracy of the classifier
predictions for $\tilde{x}_{i}$ against the assigned class label $c_{i}$ over
the training process, i.e.,
$\frac{1}{T}\sum_{t=1}^{T}\mathbbm{1}(c_{i}(t)=\tilde{c}_{i}(t))$, where
$\mathbbm{1}$ is the indicator function. This is used by Sun et al. (2020) to
detect noisy samples and to remove them from the dataset.

  * •

AUL: Area under Loss: $\sum_{t=1}^{T}l_{i}(t)$, which is used by Pleiss et al.
(2020b) to detect noisy samples.

  * •

AUM: Area under Margin:
$\frac{1}{T}\sum_{t=1}^{T}p_{i}^{\tilde{c}_{i}}(t)-p_{i}^{{c}_{i}^{\prime}}(t)$,
where $p_{i}^{{c}_{i}^{\prime}}(t)$ is the largest second logit at time $t$.
This is used by Pleiss et al. (2020a) to detect noisy samples.

  * •

JSD: Jensen-Shannon divergence between $\textbf{p}_{i}$ and $\tilde{y}_{i}$ at
the end of the training, i.e., at epoch $T$. Its weighted version, WJSD, is
also proposed by Zhang et al. (2022) to detect noisy samples.

  * •

ACD: Adaptive Centroid Distance, which is the cosine distance between the
feature vector $\textbf{h}(\tilde{x}_{i})$ and the _adaptive_ centroid of
feature vectors $\textbf{o}_{c_{i}}$ at the end of the training, i.e., at
epoch $T$. The term _adaptive_ refers to the computation of
$\textbf{o}_{c_{i}}$ as it is the centroid for samples that the classifier
suspects to be in class $c_{i}$: either these samples have been assigned to
another class but the classifier predicts them to be in class $c_{i}$, or they
have been assigned to class $c_{i}$ and the classifier has high
$p_{i}^{c_{i}}$. This metric is proposed by Zhang et al. (2022) to detect
noisy samples.

  * •

SCD (in our work): Static Centroid Distance, which is the Euclidean distance
between $\textbf{h}(\tilde{x}_{i})$ and the _static_ centroid of feature
vectors $\textbf{o}_{c_{i}}^{s}$ at the middle of training, $t^{*}$, where
$\text{training accuracy}(t^{*})\geq 50\%$ and $\text{training
accuracy}(s)<50\%$ for $s<t^{*}$. This is developed in our work as opposed to
ACD, as a result of our empirical observations in studying hard samples. The
adaptations from ACD to compute SCD are the following: (1) the distance
function in SCD is Euclidean distance instead of cosine distance as we are
interested in different regions of the feature space and not in different
angles/directions of the feature vectors; (2) SCD is computed in the middle of
training, where the training accuracy is $50\%$. We observe that in this
intermediate stage of training the differences between hard and noisy samples
are at their highest; (3) the centroid vector $\textbf{o}_{c_{i}}^{s}$ is
computed for _all_ samples that are assigned to class $c_{i}$, and not only
for those samples which the classifier predicts to be in class $c_{i}$.
Because for hard classes, the classifier, which might not have learned them
correctly, does not predict samples that actually belong to class $c_{i}$, the
distance to $\textbf{o}_{c_{i}}$ becomes rather meaningless. We observe that
relying too much on the classifier to compute this centroid, which is done in
the computation of ACD, results in inferior performance, especially when the
quality of the data is low. In Table 5, we provide an ablation study that
transitions from using different variations of ACD to using SCD, and we can
observe that using SCD results in both removing incorrectly labeled samples
and retaining correctly labeled hard samples.

Partitioning Methods Using the above metrics, one can then cluster the
available samples in $S$ in order to find estimates for $S_{e}$, $S_{h}$, and
$S_{n}$, which we refer to by $\tilde{S}_{e}$, ${\tilde{S}}_{h}$ and
${\tilde{S}}_{n}$, respectively. Note that our overall goal is to find the
estimated noisy subset ${\tilde{S}}_{n}$ and the estimated clean subset
${\tilde{S}}_{c}={\tilde{S}}_{e}\cup{\tilde{S}}_{h}$, but not precisely the
two subsets ${\tilde{S}}_{e}$ and ${\tilde{S}}_{h}$.

  * •

Thresholding (_Thres_ in short) - In this method, a threshold value is chosen
for the specific metric, which can be the average value over the samples or
any other predetermined value. The samples are then partitioned into two
regions, one containing samples whose values on the metric are greater than
the threshold, and the other one containing the samples whose values on the
metric are smaller than the threshold. Because we work with values in the
dataset that might have outliers (in particular for hard samples and noisy
samples), we choose the median value as the default threshold when using this
method.

  * •

1d-GMM - A one-dimensional Gaussian mixture model is used for partitioning the
dataset into multiple subsets or partitions by modeling the values of the
metric for each sample as a mixture of several Gaussian distributions, and
each Gaussian distribution represents one of the partitions.

  * •

2d-GMM - Similarly to 1d-GMM, a two-dimensional GMM can partition the dataset
by using two metrics instead of only one.

##  3 Dataset Design with Different Hardness Levels

In this section, we discuss about the hardness and noisiness transformations
$\mathcal{T}_{h}$ and $\mathcal{T}_{n}$. Sections 3.1, 3.2, and 3.3 provide
details of the transformation $\mathcal{T}_{h}$ in each of the three hardness
types: imbalance, diversification, and closeness to the decision boundary,
respectively. Finally, Section 3.4 provides details of $\mathcal{T}_{n}$.
The original dataset $S_{\text{org}}$ is taken here as the TinyImagenet
dataset (Le and Yang, 2015) with $200$ classes. We have chosen this dataset
because it possesses sufficient complexity to capture the underlying input-
output relations involved in intricate image classification tasks. Later in
this paper, we assess the generalizability of our findings by testing them on
real-world datasets that contain noisy labels.
To ensure consistency within each class, we assign the same levels $h$ and $n$
to all samples in the same class. To provide a comprehensive spectrum of
samples, we vary $h$ and $n$ along two orthogonal axes and across five levels,
resulting in a $2$D space with four quadrants (and a total of $25$ different
sample categories): easy-clean ($h=0$ and $n=0$), easy-noisy ($h=0$ and
$n=4$), hard-clean ($h=4$ and $n=0$), and hard-noisy ($h=4$ and $n=4$). Each
sample in $S_{\text{org}}$ is defined by a pair $(h,n)$ of levels according to
Figure 2, and the set of samples with levels $(h,n)$ is denoted by
$S^{\prime}_{h,n}$, its size by $N^{\prime}_{h,n}$. We first apply a hardness
transformation (one of the three hardness types) to $S^{\prime}_{h,n}$ to
produce $S^{\prime\prime}_{h,n}=\mathcal{T}_{h}(S^{\prime}_{h,n})$. We then
apply the noisiness transformation to $S^{\prime\prime}_{h,n}$ and finally get
the transformed subset
$S^{\prime}_{t,h,n}=\mathcal{T}_{n}\circ\mathcal{T}_{h}(S^{\prime}_{h,n})$. In
order to demonstrate the effectiveness of the hardness transformation in terms
of actually making samples more or less hard, we use metrics that are widely
accepted from the literature, such as loss (Kishida and Nakayama, 2019;
Loshchilov and Hutter, 2015), and confidence (Swayamdipta et al., 2020; Chang
et al., 2017; Wang et al., 2020). Prior studies show that hard samples have
larger losses and lower confidence. By examining the relation between
loss/confidence and the hardness levels assigned by our approach, we find that
our hardness level assignments are well-founded.

Figure 2: For a dataset with $200$ classes, the hardness level $h$ and
noisiness level $n$ of samples in the dataset are determined by their class,
as shown on this $2$-dimensional figure. For each pair $(h,n)$, we allocate
class number $c=40*(4-n)+8*h+\beta$ where $\beta\in\\{0,1,\cdots,7\\}$, so
that each pair $(h,n)$ is represented by $8$ different classes. For example,
samples with $h=0$ and $n=4$ are in classes $0$ to $7$. The pre-defined policy
to determine subsets $S^{\prime}_{h,n}\subset S_{\text{org}}$ is according to
this figure.

###  3.1 Hardness via Imbalance

In this dataset, we make samples with different levels $h$, by making the
dataset imbalanced. To achieve this, we subsample the dataset so that the
number of samples for each class with hardness $h$ is $X/2^{h}$, where $X$ is
the maximum number of samples per class in the dataset. With this approach,
the number of samples per class decreases exponentially as $h$ increases,
which makes these samples under-represented hence more difficult to learn. We
have

$\displaystyle
S^{\prime\prime}_{h,n}=\mathcal{T}_{h}(S^{\prime}_{h,n})=\mathcal{I}_{h}(S^{\prime}_{h,n})=\\{(x_{j},y_{j})\\}_{j\in
J_{h}}$

where $J_{h}\subseteq\\{1,\cdots,N^{\prime}_{h,n}\\}$ uses stratified sampling
so that there are $X/2^{h}$ samples per class. In this hardness type, the
transformation $\mathcal{F}_{h}$ is the identity map and
$\mathcal{T}_{h}=\mathcal{I}_{h}$.
Are samples in $S_{h}$ with this hardness type actually harder to learn? As we
can see in Figure LABEL:fig:ch5_proof_im, samples that are assigned with a
larger hardness level $h$ have indeed a higher loss and lower confidence
values. We conclude that samples with $h=4$, which are those in $S_{h}$, are
the hardest for the classifier to learn compared to the rest of the samples.

(a)

(b)

(c)

(d)

Figure 3: In each of the three created datasets, we observe that samples in
classes with higher $h$ have generally traits that correspond to their
difficulty. These traits include having a higher loss value and having a lower
confidence (or prediction probability). To further show the significance of
loss and confidence values as $h$ increases, we perform a one-way ANOVA test,
which results in the following F-statistics and p-values: (a) Hardness via
Imbalance: for the loss we have F=$672$ and p$<1e-100$, for the confidence we
have F=$1519$ and p$=0$; (b) Hardness via Diversification: for loss we have
F=$234$ and p$<1e-190$, for confidence we have F=$237$ and p$<1e-200$; (c)
Hardness via Closeness to the Decision Boundary: For loss we have F=$5.7$ and
p$<1e-3$; for confidence we have F=$5.8$ and p$<1e-3$. We can observe that in
all three settings, the F-statistics are relatively large and the p-values are
small. We conclude that our assigned $h$ values do indeed indicate sample
hardness and in all three settings, our approaches for creating hard samples
are justified. (d) Noisiness: We observe that, similarly to the effect of
hardness increase discussed in (a)-(c), an increase in the level of noisiness
increases loss and decreases confidence. Therefore, if we rely solely on these
metrics to identify and remove noisy-labeled samples, hard samples can also be
mistakenly removed from the dataset. This is problematic because hard samples
contain valuable information about the underlying data distribution and should
not be ignored.

###  3.2 Hardness via Diversification

Compare the following two sets of samples: (i) $16$ images of different cats
possibly with different breads, (ii) $16$ images of the same cat with
different image augmentations, (augmentations can be rotation, flipping,
scaling and blurring (Shorten and Khoshgoftaar, 2019)). Which of the two sets
of samples is harder for a classifier to learn? We can argue that the first
set is more difficult, because the classifier requires to learn some common
features that belong to $16$ different images instead of $16$ variations of
the same image. The first set of samples are more _diverse_ and such
diversification is the second hardness type we consider. It is also observed
by Kishida and Nakayama (2019), that easy examples in the datasets are
visually similar to each other whereas hard samples are visually diverse.
In this dataset, we keep the number of samples balanced between classes, but
we vary the diversity of samples per class. To do so, we maintain the original
samples in hard classes in order to preserve their level of
diversity/difficulty. For samples in the easy classes, we perform data
augmentation techniques, such as rotation or flipping, to make samples less
diverse and hence easier to learn. By changing the number of data
augmentations used in each level of hardness $h$, this approach enables us to
generate a range of hardness levels within a single dataset. In particular,
for hardness level $h$, we first subsample the number of samples per class by
$X/2^{4-h}$, where $X$ is the maximum number of unique samples per class.
Then, we create $2^{4-h}-1$ new augmented samples per sample. As a result, all
classes have the same overall number of samples and the dataset is balanced.
We have

$\displaystyle S^{\prime\prime}_{h,n}=\mathcal{T}_{h}(S^{\prime}_{h,n})$
$\displaystyle=\mathcal{F}_{h}\circ\mathcal{I}_{h}(S^{\prime}_{h,n})=\mathcal{F}_{h}\left(\\{(x_{j},y_{j})\\}_{j\in
J_{h}}\right)$ $\displaystyle=\\{(\tilde{x}_{i},y_{j})|i\in I_{h}(j)\\}_{j\in
J_{h}},$

where $J_{h}\subseteq\\{1,\cdots,N^{\prime}_{h,n}\\}$ uses stratified sampling
so that there are $X/2^{4-h}$ samples per class,
$I_{h}(j)=\\{j+a\cdot\mathinner{\\!\left\lvert
J_{h}\right\rvert}\\}_{a\in\\{0,\cdots,2^{4-h}-1\\}}$, and for $i\in
I_{h}(j),i\neq j$, $\tilde{x}_{i}$ is an augmented version of $x_{j}$.
Are samples in $S_{h}$ with this hardness type actually harder to learn? As we
can see in Figure LABEL:fig:ch5_proof_di, samples that are assigned with a
larger hardness level $h$, have higher loss and lower confidence values.
Hence, we conclude that samples with $h=4$, which are in $S_{h}$, are the
hardest for the classifier to learn.

###  3.3 Hardness via Closeness to the Decision Boundary

In this dataset, we produce hard samples by modifying them to be closer to the
decision boundary. To achieve this, we must first identify the true decision
boundary for the classification task at hand. As we are working with the
TinyImagenet dataset, the true decision boundary is unknown. The closest
estimate to the true decision boundary that we found is a DeiT model (Touvron
et al., 2021) that was pre-trained on ImageNet and fine-tuned on TinyImagenet,
with a training accuracy of $98.8\%$ and a test accuracy of $90.88\%$ on the
original train and test sets of TinyImagenet. Then, we create samples that are
closer to the decision boundary of this model, hoping that they will also be
closer to the decision boundary of the actual ground truth model.
We have

$\displaystyle
S^{\prime\prime}_{h,n}=\mathcal{T}_{h}(S^{\prime}_{h,n})=\mathcal{F}_{h}(S^{\prime}_{h,n})=\\{(\tilde{x}_{j},y_{j})\\}_{j\in
J_{h}},$

where $J_{h}=\\{1,\cdots,N^{\prime}_{h,n}\\}$, and $\tilde{x}_{j}$ is a
transformation of the sample $x_{j}$, such that $\tilde{x}_{j}$ is closer to
the decision boundary compared to $x_{j}$. In this hardness type the
transformation $\mathcal{I}_{h}$ is an identical map and
$\mathcal{T}_{h}=\mathcal{F}_{h}$. Our hard sample creation is done similarly
to the fast gradient sign method for creating adversarial samples, which is
introduced by Goodfellow et al. (2014). For an input sample $x_{i}$, this
method creates adversarial input

$x_{i}^{\text{adv}}=x_{i}+\epsilon\quad\text{sign}\left(\nabla_{x}l_{i}\right)$

where $\epsilon$ is a hyper-parameter used for scaling the added noise. For
creating adversarial samples, $\epsilon$ should be large enough to change the
label of the sample. For our dataset creation however, and unlike with
adversarial sample creation, we make sure that the resulting sample
$\tilde{x}_{i}$ does _not_ change its label, by adjusting $\epsilon$.
Different levels of $h$ are obtained by adjusting the value of $\epsilon$, and
hence $\epsilon(h)$ is a function of the hardness level $h$. Higher values of
$\epsilon$ move the sample closer to the decision boundary, making it harder
to learn with a new model. After generating new samples, we keep only those
that maintain their original class label, as our goal is to increase the
difficulty of samples in their original class. It is critical to choose the
appropriate range of $\epsilon(h)$, as a value that is too high can cause most
samples to change their predicted class and be excluded from the dataset. This
would result in the remaining samples being easier to learn instead of harder,
because they were originally very easy and far from the decision boundary.
Therefore, we carefully choose the range of $\epsilon(h)$ during our dataset
creation, which results in a smaller variation of the loss/confidence as a
function of the hardness level $h$, as shown in Figure LABEL:fig:ch5_proof_db.
These samples become hard, particularly for models that make decisions near
the decision boundary, which is why in this setting, unlike the other
settings, we use models which are pre-trained on the ImageNet dataset instead
of using randomly initialized models.
Are samples in $S_{h}$ with this hardness type actually harder to learn? As
shown in Figure LABEL:fig:ch5_proof_db, samples that are assigned with a
larger hardness $h$ value, have higher loss and lower confidence values. The
loss values vary however less with the hardness level $h$ than in the two
previous hardness types, because the sensitivity of the range of $\epsilon(h)$
must be large enough to bring the sample close to the boundary, but small
enough to keep it from changing its label. Figure LABEL:fig:ch5_proof_db shows
that samples with $h=4$, which are in $S_{h}$, are still the hardest for the
classifier to learn.

###  3.4 Addition of Label Noise; Its Similarities to Hardness

After creating each of the hard datasets, we add label noise to samples of the
dataset according to Figure 2. For each sample with noisiness $n$, we assign
the label noise level of $q(n)=\delta*n/4$, where $\delta\in\left[0,1\right]$
is the maximum label noise level of the created dataset (by default we use
$\delta=0.4$ in our experiments). Label-noise level refers to the probability
of the label being replaced uniformly at random with one of the class labels.
For example, samples with $n=1$, have $0.25\delta$ probability of having a
noisy label. To prevent confusion between samples in classes with different
levels of noisiness $n$, we limit the choice of class labels to only those
with the same $n$ when assigning noisy labels. This ensures that samples are
assigned to a class with $n=0$ if and only if they actually belong to that
class. Overall, for the set
$S^{\prime\prime}_{h,n}=\\{(\tilde{x}_{i},y_{i})\\}_{i\in\\{1,\cdots,N^{\prime\prime}_{h,n}\\}}$,
we have

$\displaystyle
S^{\prime}_{t,h,n}=\mathcal{T}_{n}(S^{\prime\prime}_{h,n})=\\{(\tilde{x}_{i},\tilde{y}_{i})\\}_{i\in\\{1,\cdots,N^{\prime\prime}_{h,n}\\}},$

where with probability $1-q(n)$, $\tilde{y}_{i}=y_{i}$, and with probability
$q(n)$, the non-zero element of $\tilde{y}_{i}$ is at index $j\sim
U(1,2,\cdots,K)$, where $U$ is the uniform distribution.
In Figure LABEL:fig:ch5_hardness_naive, we observe that increasing $n$,
similarly to increasing $h$ as discussed in previous sections, results in a
higher loss, and a lower confidence. We observe that the same correlation sign
between loss/confidence and hardness $h$ also exists between loss/confidence
and noisiness $n$. This indicates that hardness and noisiness exhibit similar
characteristics, and that removing noisy samples from the training dataset by
using loss or confidence could also lead to the inadvertent removal of
correctly-labeled hard-to-learn samples. This highlights the need for more
advanced metrics/methods to identify and preserve hard samples while removing
noisy ones.

##  4 Easy-Hard-Noisy Data Partitioning and Training

In this section, we use the metrics and methods discussed in Section 2 to
partition the synthetic datasets constructed in Section 3 into two subsets:
the estimated clean subset $\tilde{S}_{c}$ and the estimated noisy subset
$\tilde{S}_{n}$. We evaluate the effectiveness of these methods and metrics in
this partitioning task, with a focus on the quality of the filtering between
hard and noisy samples. We then compare the performance of different
partitioning methods based on the test accuracy of models trained on the
estimated clean subset produced from each method. It is important to note that
although the training datasets can contain samples with issues, such as
hardness or noisiness in certain classes, the test datasets do not contain any
such noisy-labeled or imbalanced samples. Therefore, traditional evaluation
metrics, such as test accuracy, provide a fair representation of the
generalization performance of models in each setting.

###  4.1 Partitioning

We compare different methods that partition the available dataset into two
subsets: $\tilde{S}_{c}$ and $\tilde{S}_{n}$. Our primary objective is to
include the incorrectly labeled samples within $\tilde{S}_{n}$, which is the
focus of previous studies as well, but in addition, we are also interested in
including all hard samples within $\tilde{S}_{c}$. Therefore, we aim to
achieve high recall for hard samples, i.e.,
$\text{Recall}_{\text{h}}=\frac{|{\tilde{S}}_{c}\cap{{S}}_{h}|}{|{S}_{h}|}$,
while maintaining a high recall for noisy samples, i.e.,
$\text{Recall}_{\text{n}}=\frac{|{\tilde{S}}_{n}\cap{{S}}_{n}|}{|{S}_{n}|}$.
Although having a high precision for noisy samples, i.e.,
$\text{Precision}_{\text{n}}=\frac{|{\tilde{S}}_{n}\cap{{S}}_{n}|}{|{\tilde{S}}_{n}|}$,
is desirable, it is less critical. Even if some easy samples are mistakenly
included in $\tilde{S}_{n}$, this is not too harmful because they can be
either relabeled or not used in training. This is because such easy and
correctly labeled samples are often redundant in the training set, and are
likely to be inexpensive to label or replace (Paul et al., 2021).

(a)

(b)

(c)

(d)

(e)

(f)

Figure 4: ACD (top) and SCD (bottom) applied to the transformed datasets with
noisiness level $n$ and hardness level $h$. We observe that SCD increases with
$n$, but not with $h$, and is thus non-monotonic in $n$ and $h$. In contrast,
ACD is monotonic in $n$ and $h$. This is also observed for other metrics such
as loss, confidence, AUL, AUM, and JSD in Figure 7. We, therefore, propose SCD
as a promising metric to be used in data partitioning for removing noisy
samples while retaining hard samples. We quantitatively compare these two
metrics with also other metrics in terms of their ability to partition
datasets into clean and noisy subsets later in this section in Table 1.

Various label-noise detection and data partitioning methods have been proposed
in the literature, including $\text{Thres}_{\text{Loss}}$ (Huang et al.,
2019), $\text{Thres}_{\text{acc over training}}$ (Sun et al., 2020),
$\text{Thres}_{\text{AUM}}$ (Pleiss et al., 2020a),
$\text{1d-GMM}_{\text{AUL}}$ (Pleiss et al., 2020b), and
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$ (Zhang et al., 2022). However, as
discussed in Section 3, metrics such as loss and confidence fail to
differentiate between hardness $h$ and noisiness $n$, making it challenging to
remove only noisy samples without removing hard ones. The same issue applies
to other metrics such as accuracy over training, AUL, AUM, JSD, and ACD, as
depicted in Figure 7 provided in the appendix; These metrics are monotonic in
both $h$ and $n$. Consequently, if these metrics are used for data
partitioning, the identified noisy subset may mistakenly contain hard samples,
reducing the reliability of these methods when dealing simultaneously with
noisy and hard samples. In contrast, SCD increases with $n$ but does _not_
increase with $h$. We can observe this behavior in Figure 4 and compare it
with for example the behavior of ACD when hardness and noisiness increase.
This behavior makes SCD a particularly promising metric for removing noisy
samples while preserving hard samples.
While SCD is a promising metric for distinguishing hard samples from noisy
ones, it is beneficial to pair it with another metric that can detect easy
samples from the hard and noisy ones. In our experiments, we have observed
that accuracy over training is an effective metric for this purpose. It can
accurately identify the easy samples in the dataset while mixing the hard and
noisy samples together. Since these two metrics - SCD and accuracy over
training - are complementary, we use a 2d-GMM with them as its dimensions to
estimate all three subsets - $S_{e}$, $S_{h}$, and $S_{n}$.
Table 1 presents the results of different partitioning methods on all three
synthetic hard datasets, including the filtered dataset size
$|{\tilde{S}}_{c}|$, correct label percentage of ${\tilde{S}}_{c}$,
$\text{Precision}_{\text{n}}$, $\text{Recall}_{\text{n}}$, and
$\text{Recall}_{\text{h}}$. Several interesting observations emerge from the
results. Firstly, some methods, such as $\text{Thres}_{\text{AUM}}$, have high
$\text{Recall}_{\text{n}}$ but low $\text{Recall}_{\text{h}}$. These methods
struggle to differentiate hard samples from noisy ones, and thus discard most
of the hard samples. Secondly, some methods, such as
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$, have high $\text{Recall}_{\text{h}}$
but low $\text{Recall}_{\text{n}}$. Although they preserve hard samples, they
also retain noisy ones. Thirdly, these observations vary depending on the type
of hardness, making the conclusions less robust. However, we consistently
observe that our proposed method, $\text{2d-GMM}_{\text{acc}-\text{SCD}}$,
performs the best overall and robustly across all three datasets, with high
$\text{Recall}_{\text{n}}$ and $\text{Recall}_{\text{h}}$. Such robustness is
particularly crucial in practice since samples can be hard to learn due to any
combination of the hardness types, making a reliable metric even more
necessary.

Method | $|{\tilde{S}}_{c}|$ | Correct Label | $\text{Precision}_{\text{n}}$ | $\text{Recall}_{\text{n}}$ | $\text{Recall}_{\text{h}}$
---|---|---|---|---|---
$\%$ of ${\tilde{S}}_{c}$ | $\frac{|{\tilde{S}}_{n}\cap{{S}}_{n}|}{|{\tilde{S}}_{n}|}$ | $\frac{|{\tilde{S}}_{n}\cap{{S}}_{n}|}{|{S}_{n}|}$ | $\frac{|{\tilde{S}}_{c}\cap{{S}}_{h}|}{|{S}_{h}|}$
Original dataset | $31k$ | $0.80$ | NA | NA | NA
$\text{Thres}_{\text{Loss}}$ (Huang et al., 2019; Lu and He, 2022) | $15.5k$ | $0.88$ | $0.27$ | $0.69$ | $0.03$
$\text{Thres}_{\text{acc over training}}$ (Sun et al., 2020) | $15.7k$ | $\mathbf{0.95}$ | $0.35$ | $\mathbf{0.88}$ | $0.003$
$\text{Thres}_{\text{AUM}}$ (Pleiss et al., 2020a) | $15.5k$ | $\mathbf{0.95}$ | $0.34$ | $0.86$ | $0.001$
$\text{1d-GMM}_{\text{Loss}}$ (Li et al., 2020) | $29.1k$ | $0.81$ | $0.31$ | $0.10$ | $0.57$
$\text{1d-GMM}_{\text{AUL}}$ (Pleiss et al., 2020b) | $25.6k$ | $0.88$ | $\mathbf{0.59}$ | $0.52$ | $0.12$
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$ (Zhang et al., 2022) | $18.4k$ | $0.77$ | $0.14$ | $0.30$ | $\mathbf{0.73}$
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$ (Ours) | $25.7k$ | $0.83$ | $0.44$ | $0.46$ | $0.61$
(a)

Original dataset | $80k$ |  $0.80$ |  NA |  NA |  NA
---|---|---|---|---|---
$\text{Thres}_{\text{Loss}}$ (Huang et al., 2019; Lu and He, 2022) | $40k$ | $0.89$ | $0.29$ | $0.73$ | $0.38$
$\text{Thres}_{\text{acc over training}}$ (Sun et al., 2020) | $41k$ | $\mathbf{0.96}$ | $0.36$ | $\mathbf{0.90}$ | $0.41$
$\text{Thres}_{\text{AUM}}$ (Pleiss et al., 2020a) | $40k$ | $\mathbf{0.96}$ | $0.35$ | $\mathbf{0.90}$ | $0.37$
$\text{1d-GMM}_{\text{Loss}}$ (Li et al., 2020) | $65.8k$ | $0.84$ | $0.38$ | $0.34$ | $0.66$
$\text{1d-GMM}_{\text{AUL}}$ (Pleiss et al., 2020b) | $63k$ | $0.91$ | $\mathbf{0.58}$ | $0.63$ | $\mathbf{0.69}$
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$ (Zhang et al., 2022) | $46.1k$ | $0.80$ | $0.19$ | $0.41$ | ${0.50}$
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$ (Ours) | $62.2k$ | $0.86$ | $0.38$ | $0.43$ | $0.68$
(b)

Original dataset | $27.6k$ |  $0.80$ |  NA |  NA |  NA
---|---|---|---|---|---
$\text{Thres}_{\text{Loss}}$ (Huang et al., 2019; Lu and He, 2022) | $13.8k$ | $0.93$ | $0.32$ | $0.82$ | $0.46$
$\text{Thres}_{\text{acc over training}}$ (Sun et al., 2020) | $15.9k$ | $\mathbf{0.99}$ | $0.46$ | $\mathbf{0.99}$ | ${0.59}$
$\text{Thres}_{\text{AUM}}$ (Pleiss et al., 2020a) | $13.8k$ | $\mathbf{0.99}$ | $0.39$ | $\mathbf{0.99}$ | $0.51$
$\text{1d-GMM}_{\text{Loss}}$ (Li et al., 2020) | $26.2k$ | $0.84$ | $\mathbf{0.84}$ | $0.22$ | $\mathbf{0.79}$
$\text{1d-GMM}_{\text{AUL}}$ (Pleiss et al., 2020b) | $19.4k$ | $\mathbf{0.99}$ | $0.65$ | $0.97$ | $0.70$
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$ (Zhang et al., 2022) | $22k$ | $0.85$ | $0.37$ | $0.38$ | ${0.67}$
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$ (Ours) | $23.1k$ | $0.92$ | $0.81$ | $0.68$ | $0.77$
(c)

Table 1: Partitioning results obtained by applying various label noise
detection methods on our three synthetic datasets with hard samples. We can
observe that, on the one hand, some methods have a good performance on recall
only on noisy samples $\text{Recall}_{\text{n}}$ (for example
$\text{Thres}_{\text{acc over training}}$), whereas they have very bad
performance in terms of recalling hard samples (low
$\text{Recall}_{\text{h}}$). This means that, in an attempt to remove noisy
samples from the dataset, they remove almost all hard samples as well. On the
other hand, some methods have a good performance in terms of keeping hard
samples (high $\text{Recall}_{\text{h}}$) (for example
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$), but fail to remove the noisy
samples from the training dataset, as evidenced by the very low value of
$\text{Recall}_{\text{n}}$. Our proposed metric,
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$, shows the best overall performance in
terms of removing noisy samples while keeping hard samples, as evidenced by
the relatively high values of $\text{Recall}_{\text{h}}$ and
$\text{Recall}_{\text{n}}$. The result is consistent in all three settings,
unlike some methods that only perform well in one of the hardness types.

###  4.2 Training on the Filtered Subset

In this section, we present the results of training models on the estimated
clean datasets $\tilde{S}_{c}$ obtained in the previous section. Table 2
displays the generalization performance of models trained on the estimated
clean datasets using each method. The results demonstrate that our proposed
method $\text{2d-GMM}_{\text{acc}-\text{SCD}}$ consistently outperforms the
other methods in all three hardness types. We observe that the
$\text{1d-GMM}_{\text{AUL}}$ method performs slightly better than
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$ in the second and third hardness
types. However, in the first hardness type, $\text{1d-GMM}_{\text{AUL}}$
performs significantly worse than $\text{2d-GMM}_{\text{acc}-\text{SCD}}$,
which suggests that $\text{1d-GMM}_{\text{AUL}}$ is not robust to the hardness
type and thus unreliable for practical use.
Overall, the table emphasizes the importance of selecting an appropriate
metric/approach for data filtration in presence of both noisy and hard
samples. Moreover, our experiments highlight the advantage of using the label
noise detection method $\text{2d-GMM}_{\text{acc}-\text{SCD}}$ for different
hardness types.

Method Hardness via Imbalance Hardness via Diversification Hardness via
Closeness to the Decision Boundary $\text{Thres}_{\text{Loss}}$ (Huang et al.,
2019; Lu and He, 2022) $25.83_{\pm 0.06}\%$ $42.65_{\pm 0.07}\%$ $43.30_{\pm
0.33}\%$ $\text{Thres}_{\text{acc over training}}$ (Sun et al., 2020)
$25.34_{\pm 0.42}\%$ $44.39_{\pm 0.16}\%$ $49.13_{\pm 0.43}\%$
$\text{Thres}_{\text{AUM}}$ (Pleiss et al., 2020a) $23.85_{\pm 0.21}\%$
$44.35_{\pm 0.41}\%$ $47.67_{\pm 0.09}\%$ $\text{1d-GMM}_{\text{Loss}}$ (Li et
al., 2020) $31.08_{\pm 0.13}\%$ $42.93_{\pm 0.37}\%$ $44.93_{\pm 0.17}\%$
$\text{1d-GMM}_{\text{AUL}}$ (Pleiss et al., 2020b) $30.27_{\pm 0.34}\%$
$\mathbf{45.56}_{\pm 0.39}\%$ $\mathbf{50.67}_{\pm 0.26}\%$
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$ (Zhang et al., 2022) $32.11_{\pm
0.15}\%$ ${38.83}_{\pm 0.34}\%$ $43.23_{\pm 0.21}\%$
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$ (Ours) $\mathbf{34.88_{\pm 0.13}}\%$
$\mathbf{45.36}_{\pm 0.33}\%$ $\mathbf{49.80}_{\pm 0.24}\%$

Table 2: DenseNet test accuracy comparison on the estimated clean datasets
using different data filtration methods for datasets with different hardness
types. Our method $\text{2d-GMM}_{\text{acc}-\text{SCD}}$ consistently
performs well in all hardness types, unlike the $\text{1d-GMM}_{\text{AUL}}$
method that performs well only in the second and third hardness types. It is
important to choose a method that works well with all hardness types because,
in practice, hard samples can arise from any of the three types tested here.

Dataset Animal-10N CIFAR-10N Method Performance Metric Test Accuracy $\%$ Test
Loss Estimated LNL Test Accuracy $\%$ Test Loss Estimated LNL
$\text{Thres}_{\text{Loss}}$ (Huang et al., 2019; Lu and He, 2022) $80.32_{\pm
0.38}$ $0.62_{\pm 0.01}$ $50\%$ $85.45_{\pm 0.22}$ $0.64_{\pm 0.01}$ $50\%$
$\text{Thres}_{\text{acc over training}}$ (Sun et al., 2020) $83.03_{\pm
0.11}$ $0.52_{\pm 0.01}$ $2.48\%$ $83.61_{\pm 0.17}$ $0.78_{\pm 0.01}$ $46\%$
$\text{Thres}_{\text{AUM}}$ (Pleiss et al., 2020a) $76.30_{\pm 0.16}$
$0.90_{\pm 0.04}$ $50\%$ $81.89_{\pm 0.18}$ $0.83_{\pm 0.01}$ $50\%$
$\text{1d-GMM}_{\text{Loss}}$ (Li et al., 2020) $83.05_{\pm 0.25}$ $0.53_{\pm
0.01}$ $2.82\%$ $85.87_{\pm 0.30}$ $0.62_{\pm 0.00}$ $46\%$
$\text{1d-GMM}_{\text{AUL}}$ (Pleiss et al., 2020b) ${82.80}_{\pm 0.28}$
${0.53}_{\pm 0.01}$ $2.11\%$ $86.21_{\pm 0.06}$ $0.66_{\pm 0.01}$ $30\%$
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$ (Zhang et al., 2022) $83.06_{\pm
0.26}$ ${0.54_{\pm 0.01}}$ $1.03\%$ $89.44_{\pm 0.22}$ $0.35_{\pm 0.01}$
$17\%$ $\text{2d-GMM}_{\text{acc}-\text{SCD}}$ (Ours) $\mathbf{83.11}_{\pm
0.10}$ $\mathbf{0.51}_{\pm 0.01}$ $\mathbf{3.3}\%$ $\mathbf{89.90}_{\pm 0.21}$
$\mathbf{0.31}_{\pm 0.00}$ $\mathbf{10}\%$

Table 3: Performance comparison of models trained on the estimated clean
datasets using different data filtration methods for the Animal-10N and
CIFAR-10N datasets. The estimated label noise level (LNL) of Animal-10N and
CIFAR-10N are around $8\%$ and $9.01\%$, respectively. We observe that our
method provides the best LNL estimate. The LNL estimate of each method is
computed using
$1-\mathinner{\\!\left\lvert\tilde{S}_{c}\right\rvert}/\mathinner{\\!\left\lvert
S\right\rvert}$. Moreover, our method $\text{2d-GMM}_{\text{acc}-\text{SCD}}$
provides the cleanest dataset as evidenced by the large margin in the
generalization performance improvement; both test accuracy and test loss are
the best for $\text{2d-GMM}_{\text{acc}-\text{SCD}}$.

Experiments on datasets with real-world label noise To further evaluate and
compare the performance of each data filtration method, we apply them to
datasets with real-world label noise. Table 3 displays the results of each
method for partitioning and data filtration, for models trained on the
Animal-10N222https://dm.kaist.ac.kr/datasets/animal-10n/ and CIFAR-10N (Wei et
al., 2022) datasets. Both datasets have real-world label noise and unknown
easy-hard-noisy subsets. Since we lack knowledge of which samples are hard or
incorrectly labeled, we cannot compute $\text{Recall}_{\text{h}}$ or
$\text{Recall}_{\text{n}}$, as we did for our created synthetic datasets in
Table 1. Nonetheless, we can apply each partitioning/label-noise detection
method to these datasets, partition them into an estimated clean subset
$\tilde{S}_{c}$ and an estimated noisy subset $\tilde{S}_{n}$, and train
models on $\tilde{S}_{c}$. The generalization performance of models trained on
$\tilde{S}_{c}$ obtained from each method is an indication of the quality of
the estimated clean subsets $\tilde{S}_{c}$. Our results in Table 3
demonstrate that our proposed method, $\text{2d-GMM}_{\text{acc}-\text{SCD}}$,
outperforms all other methods by a significant margin in terms of test
performance and in terms of estimating the label noise level of the given
dataset.

###  4.3 Semi-supervised Learning on the Filtered Subsets

After the data partitioning step to partition the dataset into an estimated
clean subset $\tilde{S}_{c}$ and an estimated noisy subset $\tilde{S}_{n}$, we
can then apply a semi-supervised learning algorithm and use $\tilde{S}_{c}$ as
the labeled set and $\tilde{S}_{n}$ as the unlabeled set (by discarding the
noisy labels in $\tilde{S}_{n}$). We use the Flex-Match semi-supervised
learning algorithm (Zhang et al., 2021a), which is shown to have good
performance. The results are shown in Table 4, for the two real-world datasets
Animal-10N and CIFAR-10N. We can observe the significant performance
improvement brought by using $\text{2d-GMM}_{\text{acc}-\text{SCD}}$ as a data
partitioning method, compared to other partitioning methods. This once again
indicates that our data partitioning method is able to remove incorrectly
labeled samples that do not help generalization, and to further retain hard
samples which help generalization.
Note that semi-supervised learning algorithms are well-suited for settings
where there is a large set of unlabeled data and a small set of labeled data.
However, in our settings, where we use different data partitioning methods on
the Animal-10N and CIFAR-10N datasets which are datasets with low label noise
levels, the estimated noisy subset is not large. Hence, algorithms such as
Flex-Match require a large number of epochs in order to get to convergence.
This was computationally expensive and hence we present results of networks
that are stopped at a training loss value of $0.45$. This is the reason we are
observing a relatively lower performance when using a semi-supervised learning
approach compared to training only on the filtered labeled dataset (reported
in Table 3).

Method Dataset | Animal-10N | CIFAR-10N
---|---|---
$\text{Thres}_{\text{Loss}}$ (Huang et al., 2019; Lu and He, 2022) | $\mathbf{77.84}_{\pm 0.50}$ | $63.85_{\pm 2.25}$
$\text{Thres}_{\text{acc over training}}$ (Sun et al., 2020) | $71.60_{\pm 0.54}$ | $69.44_{\pm 0.18}$
$\text{Thres}_{\text{AUM}}$ (Pleiss et al., 2020a) | $72.40_{\pm 0.39}$ | $61.32_{\pm 0.57}$
$\text{1d-GMM}_{\text{Loss}}$ (Li et al., 2020) | $72.15_{\pm 1.81}$ | $65.67_{\pm 0.84}$
$\text{1d-GMM}_{\text{AUL}}$ (Pleiss et al., 2020b) | $70.11_{\pm 1.55}$ | $66.65_{\pm 0.94}$
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$ (Zhang et al., 2022) | $69.88_{\pm 0.67}$ | $76.43_{\pm 0.80}$
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$ (Ours) | $\mathbf{77.43}_{\pm 0.46}$ | $\mathbf{80.45}_{\pm 0.20}$
Table 4: Test accuracy percentage comparison for models trained using Flex-
Match algorithm on the partitioned datasets found from each method. The
estimated clean subset $\tilde{S}_{c}$ and noisy subset $\tilde{S}_{n}$ are
used as the labeled and unlabeled sets, respectively. We can observe that
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$ achieves the best dataset quality as
evidenced by the high test accuracy of the trained models in both Animal-10N
and CIFAR-10N datasets.

##  5 Discussion

Robustness to Hardness Type In this study, we aim to explore different
approaches for manipulating the hardness of samples and investigate their
impact on label noise detection methods. We simulate three hardness types and
compared the performance of various methods in identifying and removing noisy
samples while retaining hard ones. It is important to highlight that sample
hardness level $h$ is a relative concept and can only be evaluated when
comparing samples within a given dataset. This is different from the noisiness
level $n$, which is an absolute measure. Our simulations showed that the three
hardness types that we test are sufficiently distinct. Interestingly, we find
that some methods perform better in distinguishing certain types of hard
samples from noisy ones, while not performing well in distinguishing other
hard samples. However, the 2D-GMM on top of accuracy over training and SCD,
demonstrated the most robust performance across all hardness types. The reason
for the rather significant superior performance of this method in real-world
datasets stems from its ability to effectively detect label noise and hard
samples caused by various underlying reasons, which are often unpredictable in
real-world datasets. When proposing a label noise detection method, it is
essential to consider the unpredictability of hard samples in real-world
datasets.

(a)

(b)

(c)

Figure 6: (a) Easy-hard-noisy clusters for samples of the hardness via
diversification dataset with their associated static centroid distance (SCD)
and accuracy over training values. We can observe that noisy samples are
located on the top left of the figure, followed by the hard samples in the
middle parts and the easy samples on the other end, i.e., the bottom right.
Easy samples have a high accuracy over training and a low SCD. Depicting such
figures requires full ground-truth access to the available dataset and
knowledge about label noise and hardness levels of samples. However, this
information is not possible in practice, hence in the sub-figures (b) and (c)
we evaluate the results of Gaussian mixture models to recover these clusters
without any knowledge about samples. (b) Clustering results of
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$ with $2$ number of clusters. This
clustering does not require the ground-truth access of samples and only
requires the computation of SCD and accuracy over training. If we compare the
clusters with actual easy-hard-noisy partitions in sub-figure (a) we can
observe that most of the hard and noisy samples are mixed in the second
cluster. (c) Clustering results of $\text{2d-GMM}_{\text{acc}-\text{SCD}}$
with $3$ number of clusters. In contrast to clustering with $2$ clusters, we
can observe that clustering with $3$ cluster is much better at including as
many noisy samples as possible in the second cluster while not including hard
samples. Hence, throughout our study, when referring to our label noise
detection method $\text{2d-GMM}_{\text{acc}-\text{SCD}}$, we apply $3$
clusters and use the top left cluster as the detected noisy subset.

Number of Clusters in GMM We observe that in our synthetic datasets, the
location of easy $S_{e}$, hard $S_{h}$, and noisy $S_{n}$ samples in the 2D
spectrum with accuracy over training and SCD followed a certain pattern. We
illustrate this pattern in Figure 6 for the hardness via diversification
dataset. Furthermore, we find that this pattern is consistent across different
hardness types. To further investigate this observation, we compare GMM models
with two and three clusters and analyze the resulting clusters. Our
observation is that when we use two clusters, many hard and noisy samples are
clustered together, resulting in poor detection performance. However, when we
use three clusters, the detection performance improves significantly, and the
resulting clusters are much more coherent with the actual clusters. This
observation led us to use a 3-cluster GMM in our proposed method,
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$, which produced more accurate results.
It is important to note that such investigation is only possible through our
controlled experiments on synthetic datasets because we know the exact
partitions of the datasets into easy, hard, and noisy samples. Nevertheless,
our findings provide valuable insights into the design of label noise
detection methods even when used with real-world datasets.

Conclusion We propose an empirical approach to investigate hard samples in an
image classification setting. To this end, we create synthetic datasets that
enable us to study hard and noisy samples in a controlled environment. Through
our investigation, we make several interesting observations, including the
importance of analyzing feature layer vectors of neural networks to
distinguish between hard and noisy samples. Furthermore, we propose a label
noise detection method that outperforms other existing methods in terms of
both removing noisy samples and retaining hard ones. Our method can be applied
to filter datasets with label noise, leading to better generalization
performance when training models on the filtered set. Importantly, our label
noise detection method is of quite general use and can be used in combination
with any other method designed to deal with label noise and/or hard samples.
Although our synthetic datasets were tailored toward image classification
tasks, our conclusions could be applied to other applications. Moreover, our
data filtration method could be combined with semi-supervised learning
methods, such as FixMatch and FlexMatch, to further improve the model’s
generalization performance by making use of the discarded subset of the data.
Overall, our study provides valuable insights into the design and optimization
of label noise detection methods and their applications in improving the
performance of machine learning models in real-world settings.

## References

  * Algan and Ulusoy [2020] G. Algan and I. Ulusoy.  Label noise types and their effects on deep learning.  _arXiv preprint arXiv:2003.10471_ , 2020.
  * Arpit et al. [2017] D. Arpit, S. Jastrzębski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, et al.  A closer look at memorization in deep networks.  In _International conference on machine learning_ , pages 233–242. PMLR, 2017.
  * Bai and Liu [2021] Y. Bai and T. Liu.  Me-momentum: Extracting hard confident examples from noisily labeled data.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9312–9321, 2021.
  * Bengio et al. [2009] Y. Bengio, J. Louradour, R. Collobert, and J. Weston.  Curriculum learning.  In _Proceedings of the 26th annual international conference on machine learning_ , pages 41–48, 2009.
  * Chang et al. [2017] H.-S. Chang, E. Learned-Miller, and A. McCallum.  Active bias: Training more accurate neural networks by emphasizing high variance samples.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Chatterjee [2020] S. Chatterjee.  Coherent gradients: An approach to understanding generalization in gradient descent-based optimization.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=ryeFY0EFwS.
  * Cordeiro and Carneiro [2020] F. R. Cordeiro and G. Carneiro.  A survey on deep learning with noisy labels: How to train your model when you cannot trust on the annotations?  In _2020 33rd SIBGRAPI conference on graphics, patterns and images (SIBGRAPI)_ , pages 9–16. IEEE, 2020.
  * Frenay and Verleysen [2013] B. Frenay and M. Verleysen.  Classification in the presence of label noise: a survey.  _IEEE transactions on neural networks and learning systems_ , 25(5):845–869, 2013.
  * Goodfellow et al. [2014] I. J. Goodfellow, J. Shlens, and C. Szegedy.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ , 2014.
  * Huang et al. [2019] J. Huang, L. Qu, R. Jia, and B. Zhao.  O2u-net: A simple noisy label detection approach for deep neural networks.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 3326–3334, 2019.
  * Jiang et al. [2018] L. Jiang, Z. Zhou, T. Leung, L.-J. Li, and L. Fei-Fei.  Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In _International conference on machine learning_ , pages 2304–2313. PMLR, 2018.
  * Karimi et al. [2020] D. Karimi, H. Dou, S. K. Warfield, and A. Gholipour.  Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis.  _Medical image analysis_ , 65:101759, 2020.
  * Kishida and Nakayama [2019] I. Kishida and H. Nakayama.  Empirical study of easy and hard examples in cnn training.  In _Neural Information Processing: 26th International Conference, ICONIP 2019, Sydney, NSW, Australia, December 12–15, 2019, Proceedings, Part IV 26_ , pages 179–188. Springer, 2019.
  * Le and Yang [2015] Y. Le and X. Yang.  Tiny imagenet visual recognition challenge.  _CS 231N_ , 7(7):3, 2015.
  * Li et al. [2020] J. Li, R. Socher, and S. C. Hoi.  Dividemix: Learning with noisy labels as semi-supervised learning.  _arXiv preprint arXiv:2002.07394_ , 2020.
  * Liang et al. [2022] X. Liang, L. Yao, X. Liu, and Y. Zhou.  Tripartite: Tackle noisy labels by a more precise partition.  _arXiv preprint arXiv:2202.09579_ , 2022.
  * Loshchilov and Hutter [2015] I. Loshchilov and F. Hutter.  Online batch selection for faster training of neural networks.  _arXiv preprint arXiv:1511.06343_ , 2015.
  * Lu and He [2022] Y. Lu and W. He.  Selc: Self-ensemble label correction improves learning with noisy labels.  _arXiv preprint arXiv:2205.01156_ , 2022.
  * Ma et al. [2020] X. Ma, H. Huang, Y. Wang, S. Romano, S. Erfani, and J. Bailey.  Normalized loss functions for deep learning with noisy labels.  In _International conference on machine learning_ , pages 6543–6553. PMLR, 2020.
  * Malach and Shalev-Shwartz [2017] E. Malach and S. Shalev-Shwartz.  Decoupling" when to update" from" how to update".  _Advances in neural information processing systems_ , 30, 2017.
  * Nguyen et al. [2019] D. T. Nguyen, C. K. Mummadi, T. P. N. Ngo, T. H. P. Nguyen, L. Beggel, and T. Brox.  Self: Learning to filter noisy labels with self-ensembling.  _arXiv preprint arXiv:1910.01842_ , 2019.
  * Patrini et al. [2017] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu.  Making deep neural networks robust to label noise: A loss correction approach.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1944–1952, 2017.
  * Paul et al. [2021] M. Paul, S. Ganguli, and G. K. Dziugaite.  Deep learning on a data diet: Finding important examples early in training.  _Advances in Neural Information Processing Systems_ , 34:20596–20607, 2021.
  * Pleiss et al. [2020a] G. Pleiss, T. Zhang, E. Elenberg, and K. Q. Weinberger.  Identifying mislabeled data using the area under the margin ranking.  _Advances in Neural Information Processing Systems_ , 33:17044–17056, 2020a.
  * Pleiss et al. [2020b] G. Pleiss, T. Zhang, E. R. Elenberg, and K. Q. Weinberger.  Detecting noisy training data with loss curves, 2020b.  URL https://openreview.net/forum?id=HyenUkrtDB.
  * Rawat and Wang [2017] W. Rawat and Z. Wang.  Deep convolutional neural networks for image classification: A comprehensive review.  _Neural computation_ , 29(9):2352–2449, 2017\.
  * Roh et al. [2019] Y. Roh, G. Heo, and S. E. Whang.  A survey on data collection for machine learning: a big data-ai integration perspective.  _IEEE Transactions on Knowledge and Data Engineering_ , 33(4):1328–1347, 2019.
  * Rolnick et al. [2017] D. Rolnick, A. Veit, S. Belongie, and N. Shavit.  Deep learning is robust to massive label noise.  _arXiv preprint arXiv:1705.10694_ , 2017.
  * Shorten and Khoshgoftaar [2019] C. Shorten and T. M. Khoshgoftaar.  A survey on image data augmentation for deep learning.  _Journal of big data_ , 6(1):1–48, 2019.
  * Sun et al. [2020] Z. Sun, X.-S. Hua, Y. Yao, X.-S. Wei, G. Hu, and J. Zhang.  Crssc: salvage reusable samples from noisy data for robust learning.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pages 92–101, 2020.
  * Swayamdipta et al. [2020] S. Swayamdipta, R. Schwartz, N. Lourie, Y. Wang, H. Hajishirzi, N. A. Smith, and Y. Choi.  Dataset cartography: Mapping and diagnosing datasets with training dynamics.  _arXiv preprint arXiv:2009.10795_ , 2020.
  * Thulasidasan et al. [2019] S. Thulasidasan, T. Bhattacharya, J. Bilmes, G. Chennupati, and J. Mohd-Yusof.  Combating label noise in deep learning using abstention.  _arXiv preprint arXiv:1905.10964_ , 2019.
  * Touvron et al. [2021] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou.  Training data-efficient image transformers & distillation through attention.  In _International conference on machine learning_ , pages 10347–10357. PMLR, 2021.
  * Wang et al. [2019] Y. Wang, X. Ma, Z. Chen, Y. Luo, J. Yi, and J. Bailey.  Symmetric cross entropy for robust learning with noisy labels.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 322–330, 2019.
  * Wang et al. [2020] Y. Wang, T. Peng, J. Duan, C. Zhu, J. Liu, J. Ye, and M. Jin.  Pathological image classification based on hard example guided cnn.  _IEEE Access_ , 8:114249–114258, 2020.
  * Wei et al. [2022] J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y. Liu.  Learning with noisy labels revisited: A study using real-world human annotations.  In _International Conference on Learning Representations_ , 2022.
  * Wu et al. [2018] Z. Wu, T. Nagarajan, A. Kumar, S. Rennie, L. S. Davis, K. Grauman, and R. Feris.  Blockdrop: Dynamic inference paths in residual networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8817–8826, 2018.
  * Yu et al. [2019] X. Yu, B. Han, J. Yao, G. Niu, I. Tsang, and M. Sugiyama.  How does disagreement help generalization against label corruption?  In _International Conference on Machine Learning_ , pages 7164–7173. PMLR, 2019.
  * Zhang et al. [2021a] B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  _Advances in Neural Information Processing Systems_ , 34:18408–18419, 2021a.
  * Zhang et al. [2021b] C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals.  Understanding deep learning (still) requires rethinking generalization.  _Communications of the ACM_ , 64(3):107–115, 2021b.
  * Zhang et al. [2022] Y. Zhang, Y. Lu, B. Han, Y.-m. Cheung, and H. Wang.  Combating noisy-labeled and imbalanced data by two stage bi-dimensional sample selection.  _arXiv preprint arXiv:2208.09833_ , 2022.
  * Zhang et al. [2020] Z. Zhang, H. Zhang, S. O. Arik, H. Lee, and T. Pfister.  Distilling effective supervision from severe label noise.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9294–9303, 2020.
  * Zhu et al. [2021] C. Zhu, W. Chen, T. Peng, Y. Wang, and M. Jin.  Hard sample aware noise robust learning for histopathology image classification.  _IEEE Transactions on Medical Imaging_ , 41(4):881–894, 2021.

##  Appendix A Experimental Setup

Generating Noisy-labeled Datasets We modify original datasets similarly to
Chatterjee [2020]; for a fraction of samples denoted by the label noise level
(LNL), we replace the labels with independent random variables drawn uniformly
from other classes with the same noisiness level $n$.

For each hardness type, there are two sets of experiments: (i) experiments
done to detect clean-noisy subsets, (ii) experiments done to train on filtered
sets. We provide details for each of these experiments below.

Hardness via Imbalance Experiments (i) The models are trained for $200$ epochs
on the cross-entropy objective function with batch size $100$, using SGD with
learning rate $0.001$, weight decay $5\cdot 10^{-4}$, and momentum $0.9$. The
maximum label noise is set as $40\%$. The neural network architecture is
MobileNetV2. (ii) The models are trained for $50$ epochs on the cross-entropy
objective function with batch size $100$, using SGD with learning rate
$0.001$, weight decay $5\cdot 10^{-4}$, and momentum $0.9$. The neural network
architecture is DenseNet121, pre-trained on ImageNet dataset.

Hardness via Diversification Experiments (i) The models are trained for $200$
epochs on the cross-entropy objective function with batch size $100$, using
SGD with learning rate $0.001$, weight decay $5\cdot 10^{-4}$, and momentum
$0.9$. The maximum label noise is set as $40\%$. The neural network
architecture is MobileNetV2. (ii) The models are trained for $50$ epochs on
the cross-entropy objective function with batch size $100$, using SGD with
learning rate $0.001$, weight decay $5\cdot 10^{-4}$, and momentum $0.9$. The
neural network architecture is DenseNet121, pre-trained on ImageNet dataset.

Hardness via Closeness to the Decision Boundary Experiments (i) The models are
trained for $50$ epochs on the cross-entropy objective function with batch
size $100$, using SGD with learning rate $0.001$, weight decay $5\cdot
10^{-4}$, and momentum $0.9$. The maximum label noise is set as $40\%$. The
neural network architecture is DesneNet121, pre-trained on ImageNet. (ii) The
models are trained for $50$ epochs on the cross-entropy objective function
with batch size $100$, using SGD with learning rate $0.001$, weight decay
$5\cdot 10^{-4}$, and momentum $0.9$. The neural network architecture is
DenseNet121, pre-trained on ImageNet dataset.

CIFAR-10N Experiments The models are trained for $50$ epochs on the cross-
entropy objective function with batch size $64$, using SGD with learning rate
$0.05$, weight decay $5\cdot 10^{-4}$, and momentum $0.9$. The neural network
architecture is MobileNetV2.

Animal-10N Experiments The models are trained for $40$ epochs on the cross-
entropy objective function with batch size $64$, using SGD with learning rate
$0.001$, weight decay $5\cdot 10^{-4}$, and momentum $0.9$. The neural network
architecture is AlexNet.

##  Appendix B Comparison to Other Metrics

Figure 7 shows a comparison of different metrics and how they behave
concerning changes in the hardness level $h$ and noisiness level $n$. From the
graph, we can see that all metrics either increase or decrease as the value of
$n$ increases. However, we also observe that the same increase or decrease
happens when at least one type of hardness $h$ increases. In contrast, the SCD
metric stands out as it does not show the same trend towards an increase in
any type of hardness level $h$ as the value of $n$ increases.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

Figure 7: Different metrics applied to the transformed datasets with noisiness
level $n$ and hardness level $h$. Left, middle and right figures correspond to
hardness types imbalanced, diversification, and closeness to the decision
boundary, respectively. We observe that all the metrics have at least one type
of hardness $h$ that with its increase, the metric shows the same behavior as
with increase of noisiness $n$. In contrast, SCD is the only metric that shows
different behaviors with increases in $h$ and $n$.

Detailed comparison between $\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$ and
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$ Table 5 shows a transition from the
partitioning method introduced in Zhang et al. [2022],
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$, to our proposed method,
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$. A few findings can be concluded from
this table. Computing the ACD metric in the middle of the training improves
$\text{Recall}_{\text{n}}$. In addition, using a Euclidean norm instead of
cosine, and using the static centroid instead of adaptive, improves
$\text{Recall}_{\text{h}}$. On one hand, the most improvement on
$\text{Recall}_{\text{h}}$ is achieved by transitioning from two-cluster GMM
to three-cluster GMM. On the other hand, the most improvement on
$\text{Recall}_{\text{n}}$ is achieved by transitioning from using WJSD to
using accuracy over training. Such transition may degrade however
$\text{Recall}_{\text{h}}$. Hence, the best overall performance is achieved if
we use the combination of all these transitions, i.e., computing feature
distance in the middle of the training, using Euclidean norm, taking the
static centroid distance, using three clusters in the GMM, and using the
accuracy over training. Such transitions result in the
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$ method, which is what we propose as
our data partitioning method.

Method | $|{\tilde{S}}_{c}|$ | Correct Label | $\text{Precision}_{\text{n}}$ | $\text{Recall}_{\text{n}}$ | $\text{Recall}_{\text{h}}$
---|---|---|---|---|---
$\%$ of ${\tilde{S}}_{c}$ | $\frac{|{\tilde{S}}_{n}\cap{{S}}_{n}|}{|{\tilde{S}}_{n}|}$ | $\frac{|{\tilde{S}}_{n}\cap{{S}}_{n}|}{|{S}_{n}|}$ | $\frac{|{\tilde{S}}_{c}\cap{{S}}_{h}|}{|{S}_{h}|}$
Original dataset | $31k$ | $0.80$ | NA | NA | NA
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$ [Zhang et al., 2022] | $23.5k$ | $0.842$ | $0.325$ | $0.394$ | ${0.191}$
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}_{\text{mid}}}$ | $23.5k$ | $0.845$ | $0.332$ | $0.408$ | ${0.196}$
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}_{\text{mid}-\text{norm}}}$ | $23.6k$ | $0.842$ | $0.325$ | $0.391$ | ${0.198}$
$\text{2d-GMM}_{\text{WJSD}-\text{ACD}_{\text{mid}-\text{static}}}$ | $23.6k$ | $0.842$ | $0.325$ | $0.392$ | ${0.201}$
$\text{2d-GMM-3clusters}_{\text{WJSD}-\text{ACD}_{\text{mid}}}$ | $28.6k$ | $0.814$ | $0.344$ | $0.132$ | ${0.541}$
$\text{2d-GMM-3clusters}_{\text{WJSD}-\text{ACD}_{\text{mid}-\text{norm}}}$ | $28k$ | $0.819$ | $0.351$ | $0.172$ | ${0.485}$
$\text{2d-GMM-3clusters}_{\text{WJSD}-\text{ACD}_{\text{mid}-\text{static}}}$ | $28.7k$ | $0.813$ | $0.337$ | $0.127$ | ${0.542}$
$\text{2d-GMM-3clusters}_{\text{acc}-\text{ACD}}$ | $21.7k$ | $0.912$ | $0.452$ | $0.688$ | ${0.009}$
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$ (Ours) | $24.1k$ | $0.892$ | $0.513$ | $0.576$ | $0.473$
Table 5: Data partitioning results on the hardness via imbalance dataset
comparing different methods that range from using the proposed method in
[Zhang et al., 2022] to using our proposed method,
$\text{2d-GMM}_{\text{acc}-\text{SCD}}$. We can observe that each modification
that is made from the method $\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$ either
improves $\text{Recall}_{\text{n}}$ or $\text{Recall}_{\text{h}}$. Overall, we
can observe that method $\text{2d-GMM}_{\text{acc}-\text{SCD}}$ outperforms
method $\text{2d-GMM}_{\text{WJSD}-\text{ACD}}$ in terms of removing
incorrectly labeled samples while retaining hard samples.