# Representation Learning With Hidden Unit Clustering For Low Resource Speech
Applications

Varun Krishna, Tarun Sai, and
Sriram Ganapathy V. Krishna, S. Tarun and S. Ganapathy are with the Learning
and Extraction of Acoustic Patterns (LEAP) lab, Department of Electrical
Engineering, Indian Institute of Science, Bangalore, India, 560012. e-mail:
{varunkrishna, tarunsai<EMAIL_ADDRESS>
###### Abstract

The representation learning of speech, without textual resources, is an area
of significant interest for many low resource speech applications. In this
paper, we describe an approach to self-supervised representation learning from
raw audio using a hidden unit clustering (HUC) framework. The input to the
model consists of audio samples that are windowed and processed with 1-D
convolutional layers. The learned “time-frequency” representations from the
convolutional neural network (CNN) module are further processed with long
short term memory (LSTM) layers which generate a contextual vector
representation for every windowed segment. The HUC framework, allowing the
categorization of the representations into a small number of phoneme-like
units, is used to train the model for learning semantically rich speech
representations. The targets consist of phoneme-like pseudo labels for each
audio segment and these are generated with an iterative k-means algorithm. We
explore techniques that improve the speaker invariance of the learned
representations and illustrate the effectiveness of the proposed approach on
two settings, i) completely unsupervised speech applications on the sub-tasks
described as part of the ZeroSpeech 2021 challenge and ii) semi-supervised
automatic speech recognition (ASR) applications on the TIMIT dataset and on
the GramVaani challenge Hindi dataset. In these experiments, we achieve state-
of-art results for various Zerospeech tasks. Further, on the ASR experiments,
the HUC representations are shown to improve significantly over other
established benchmarks based on wav2vec, HuBERT and Best-RQ.

###### Index Terms:

Self-supervised learning, hidden unit clustering, contrastive loss,
Zerospeech. Low resource ASR.

##  I Introduction

Spoken language processing using raw speech, without any textual resources,
has received wide spread interest recently [1] and are useful for various
applications like zero-resource spoken language modeling, low-resource speech
recognition and speech synthesis. The key challenge in these tasks is the
automatic discovery of sub-word units of speech [2] that are speaker invariant
and consistent.

The learning of unsupervised audio representations [3] has been largely
investigated with self-supervised loss functions involving contrastive
learning [4], vector quantization [5], clustering [6] or autoregressive
predictive coding [7]. For unsupervised learning of audio representations, the
use of restricted Boltzmann machines [8, 9], variational learning [10] and
generative adversarial networks [11] have been explored to learn
acoustic/modulation filters. Recently, a family of models, termed as wav2vec
[12, 13, 14], were investigated for self-supervised representation learning
applications like speech recognition and synthesis. The initial approach [12]
used the contrastive predictive coding [15] with $1$-D convolution layers. The
subsequent vector-quantization based approach, wav2vec-vq [13], was inspired
by the quantization module proposed by Oord et. al. [16]. The output of the
quantization module is employed in training bidirectional encoder
representations from transformer (BERT) language models [17]. The recent
approach, wav2vec 2.0 [14], further added a learnable code-book based model.

The ZeroSpeech series of challenges [18, 19, 20, 3] considered an open call
for participation in speech representation learning. The challenges bench-
marked various methods for self-supervised learning based on word similarity
and spoken language modeling tasks. The zero shot metrics used in the
ZeroSpeech 2021 challenge [3] considered acoustic and linguistic measures.
Several approaches based on Gaussian mixture modeling [21], HMMs [22], and
deep learning methods [23, 24] have been explored for the ZeroSpeech tasks. In
the recent years, self-supervised learning methods has also been investigated
for ZeroSpeech applications. These efforts have investigated various model
architectures like convolutional networks [25], recurrent networks [26],
transformer models [27], and conformer models [6]. For spoken language
modeling tasks, most of prior works use a separate clustering model on the
embeddings. Further, most of the prior works generate representations that are
general purpose and do not attempt any disentanglement of the factors to
enhance the semantic richness of the representations.

In this paper, we explore a framework of hidden unit clustering (HUC), where
the deep representations learned by a neural network are optimized using a
categorical loss. This paper extends our previous work on acoustic unit
discovery [28]. The work is inspired by prior work on deep clustering by Caron
et. al. [29] for self-supervised learning from the raw audio data. In our
work, the HUC model is shown to enable the learning of discriminative
representations that are less impacted by speaker variations. The experimental
validation of the proposed approach is performed on two settings. The first
setting is the ZeroSpeech 2021 challenge task. The phoneme recognition
experiments are performed on the TIMIT dataset while the large vocabulary
speech recognition experiments are performed on the semi-supervised track of
the GramVaani Hindi ASR
challenge111https://sites.google.com/view/gramvaaniasrchallenge/home.

The novel contributions from this current work over the previous approach [28]
are,

  * •

We propose a self-supervised learning approach using a combination of
contrastive loss and HUC loss that generates speaker invariant
representations. The semantic richness in the representations is achieved
using an utterance level mean normalization performed on the context vector
representations.

  * •

We develop a data sampling technique to generate the pseudo-phoneme cluster
labels. The labels learned using the sampling approach is shown to be
effective in generating categorical representations.

  * •

We propose a self-supervised dimensionality reduction method on the
representations, driven by the pseudo-labels. This is based on gradient
boosted decision tree modeling [30].

  * •

A set of semantic task evaluations illustrate the performance gains using the
proposed framework. Using the data from the ZeroSpeech 2021 challenge, we show
that the proposed HUC based self-supervision achieves state-of-art results on
$7$ out of the $8$ sub-tasks. We also show that the representations from the
proposed model perform well in phoneme recognition and low-resource ASR
experiments.

  * •

A detailed analysis on the representations and the evaluations on non-semantic
tasks shows that the proposed HUC framework achieves the semantic richness at
the cost of being a general purpose speech representation for non-semantic
speech tasks.

##  II Background

###  II-A Related prior work

####  II-A1 Contrastive predictive coding

The contrastive predictive coding [15] network consists of two sets of
processing layers, with inputs being the raw audio samples. Let $x[n]$ denote
the audio samples that are windowed into short-term frames (typically of
$20$ms duration). The output of the convolutional block contains $f$
dimensional filterbank representations for each frame $t$. These outputs,
sampled with a stride of $10$ ms, are denoted as $\boldsymbol{z}_{t}$, are
then fed to the LSTM layer. The LSTM layer uses a context window of $v$
previous frames,
$(\boldsymbol{z}_{t},\boldsymbol{z}_{t-1},...,\boldsymbol{z}_{t-v})$. The
output of the LSTM layer is referred to as the context vector
$\boldsymbol{c}_{t}$. The contrastive loss is computed based on the prediction
of the $K$ future filterbank representations,
$\\{\boldsymbol{z}_{t+k}\\}_{1\leq k\leq K}$ by minimizing the following loss
function [16],

$\mathcal{L}_{t}=-\frac{1}{K}\sum_{k=1}^{K}log\left(\frac{exp({\boldsymbol{z}}_{t+k}^{T}{\boldsymbol{W}}_{k}{\boldsymbol{c}}_{t})}{\sum_{\tilde{{\boldsymbol{z}}}\in\mathcal{N}}exp(\tilde{{\boldsymbol{z}}}^{T}{\boldsymbol{W}}_{k}{\boldsymbol{c}}_{t})}\right)$
(1)

Here, $\mathcal{N}$ denotes a subset of negative examples chosen randomly, and
$\textbf{W}_{k}$ forms a set of weights used in the prediction of the
$k^{\text{th}}$ future embedding $\boldsymbol{z}_{t+k}$. The CPC model used
for the experiments reported in this paper employs negative sampling from
within the same utterance (within speaker negative sampling).

####  II-A2 wav2vec models

: The wav2vec [12, 5, 14] architecture emulates the 1-D convolutional layer of
the CPC model.

wav2vec: Instead of the LSTM layer used in the CPC model, the representations
$\boldsymbol{z}_{t}$ are processed with a second set of convolutional layers
with a window of $v$ contextual embeddings,
$(\boldsymbol{z}_{t},\boldsymbol{z}_{t-1},...,\boldsymbol{z}_{t-v})$. The
model outputs the contextual vector representation $\boldsymbol{c}_{t}$, for
each time step $t$. The model learns to distinguish a sample
$\boldsymbol{z}_{t+k}$ from distractor samples $\tilde{z}$.

$\mathcal{L}_{k}=-\sum_{t=1}^{T-k}log(\sigma({\boldsymbol{z}}_{t+k}^{T}h_{k}({\boldsymbol{c}}_{t}))+\lambda\mathop{{}\mathbb{E}}_{\tilde{\boldsymbol{z}}\sim
p_{n}}log(\sigma(-\tilde{\boldsymbol{z}}^{T}h_{k}({\boldsymbol{c}}_{t})))$ (2)

Here, $\tilde{z}$ is drawn from a proposal distribution $p_{n}$, $h_{k}$
defines a linear feed-forward neural network layer and $\sigma(.)$ denotes
sigmoid activation function. Further, the weight factor $\lambda$ is a hyper-
parameter.

wav2vec-vq: In this model [5], the quantized context vector representations
are used in the prediction task. The model architecture resembles that of
wav2vec model, with two convolutional networks
$\boldsymbol{f}:\boldsymbol{\mathcal{X}}\mapsto\boldsymbol{\mathcal{Z}}$ and
$\boldsymbol{g}:\boldsymbol{\mathcal{\hat{Z}}}\mapsto\boldsymbol{\mathcal{C}}$
for feature extraction and aggregation, and an additional vector quantization
module
$\boldsymbol{q}:\boldsymbol{\mathcal{Z}}\mapsto\boldsymbol{\mathcal{\hat{Z}}}$

wav2vec 2.0: The wav2vec 2.0 [14] builds over the ideas of wav2vec-vq [5]. The
wav2vec 2.0 uses a multi-layer convolution feature encoder
$\boldsymbol{f}:\boldsymbol{\mathcal{X}}\mapsto\boldsymbol{\mathcal{Z}}$ with
the raw audio as input. A portion of the features are masked before the
transformer based aggregator
$\boldsymbol{g}:\boldsymbol{\mathcal{Z}}\mapsto\boldsymbol{\mathcal{C}}$. The
context vectors corresponding to the unmasked regions are used to predict the
quantized [31] representations of the masked regions.

####  II-A3 Deep clustering

The deep clustering approach, proposed for learning visual representations,
combines feature learning and clustering into a single neural model [29].
Unlike the VQ based approach, the deep clustering does not require the context
vectors to be approximated as code-book entries. For speech modality, deep
clustering has been explored in the HuBERT framework.

####  II-A4 HuBERT

The HuBERT [32] model consists of convolutional layers followed by a set of
transformer encoder layers. During the pre-training stage, transformer [33]
encoder consumes the masked acoustic [34] features $\boldsymbol{U}$ from the
convolutional encoder. The output $\boldsymbol{H}$ of the transformer is used
to predict the discrete targets $\boldsymbol{z}\in[K]$ ($K$-class categorical
variable) through a linear projection layer. The HuBERT adopts an iterative
clustering and training process. For the first iteration, the mel frequency
features, categorized into $\boldsymbol{K}$ clusters, are used as discrete
targets. For subsequent iterations, discrete targets are derived by clustering
the hidden layer representations.

Figure 1: Proposed approach for learning speaker invariant representations
using the HUC framework.

###  II-B Contrast with proposed HUC model

Many of the prior works perform self supervised learning (SSL) on the English
read speech in Librispeech [35] dataset, while illustrating the quality of the
representations using speech recognition tasks [14, 32]. In English read
speech utterances, the two key streams of information are the semantic
(content) variability and the non-semantic (speaker) variability. Most of the
prior works do not attempt to explicitly disentangle the semantic and non-
semantic information streams as they aim to derive general purpose audio
representations. In contrast, the proposed work a) identifies a way to
quantify the non-semantic component of a given speech utterance, b) uses the
non-semantic component in a diverse sampling of the acoustic units, and, c)
factors this component out of the final representations in order to enhance
the semantic encoding of speech. We illustrate this contrast experimentally
using various downstream tasks that probe the semantic and non-semantic
encoding in the speech representations derived from the HUC framework.

Our work, while also using deep clustering, differs from HuBERT [32] in the
following ways - i) We do not employ any masking of the audio regions, ii) A
mean normalized representation from the LSTM aggregator is used for
clustering, iii) A data sampling framework, based on diversity of pseudo-
speaker clusters is used to design the code-books, iv) The model is trained
with a combination of clustering and CPC losses. With various ablation
experiments, we empirically highlight the advantages of these processing
steps.

##  III Proposed work

The overview of the proposed framework is shown in Fig. 1. The initial
processing steps consist of a block of 1-D convolution layers, similar to the
CPC model [15]. The convolutional layers generate filter-bank features denoted
as ${\boldsymbol{z}}_{t}^{i}$, where $t$ is the frame index and $i$ is the
utterance index. Subsequently, the LSTM layer generates context vectors
${\boldsymbol{c}}_{t}^{i}$. The regularized HUC loss is used to train the
model.

The HUC approach attempts to generate categorical labels using a clustering
framework applied on the CPC based context vector representations. For this
purpose, the context vectors ${\boldsymbol{c}}_{t}^{i}$ from a pre-trained
model are clustered using an unsupervised k-means clustering. The labels from
this clustering step are considered to be the psuedo-phoneme labels for
training the model shown in Fig. 1.

###  III-A Analyzing the mean context vector

In our analysis, shown in Fig 2, we use a pre-trained CPC model trained on the
Librispeech dataset, employing within utterance negative sampling of
distractors, to compute the utterance level mean, $\boldsymbol{\bar{c}}^{i}$,
of the context vector representation. We plot the t-distributed stochastic
neighbourhood embedding (t-SNE) [36] of the mean context vectors, for $50-100$
utterances derived from $7$ speakers chosen randomly from the dataset. As seen
in this plot, the utterance level mean of the context vectors primarily
captures the speaker information.

Figure 2:  The t-SNE plot of the mean context vector,
$\boldsymbol{\bar{c}^{i}}$, computed for $50$-$100$ utterances each, from a
set of $7$ speakers in the LibriSpeech dataset.

###  III-B Data sampling for k-means training

With a large number of utterances, and with a frame-rate of $100$ Hz ($10$ms
windows), the number of context vectors $\boldsymbol{{c}}_{t}^{i}$ available
for clustering is significantly large ($>100M$). Secondly, the pseudo-phoneme
classes (classes for training the HUC model) have to satisfy two goals namely,

  * •

consistency \- should be sufficiently broad to represent different variants of
the same underlying sound spoken by different speakers and in different
acoustic contexts.

  * •

conciseness \- should be succinct to capture the fine acoustic properties that
make it distinctive from other units.

The data sampling process attempts to achieve the right trade-off between the
consistency and conciseness. The goal of the data sampling process is to
create a subset of the training data, that are used in the clustering. The
algorithm is outlined in Alg. $1$.

The key objective is to preserve only the utterances from diverse “speakers”
in the corpus. The first step in the data sampling is to perform a k-means
clustering of the mean context vectors, $\boldsymbol{\bar{c}}^{i}$, derived
from all the utterances in the training data. This step generates $M$ pseudo-
speaker centroids of the training data. The optimal value of $M$ is determined
using knee point algorithm [37, 38]for k-means. The knee point algorithm finds
the point of inflection in inertia. Specifically, the sum of squared distances
of a point from the nearest cluster center is computed for different choices
of $M$ (number of clusters). The point of inflection is used as the optimal
value for $M$. Further, using the inter-centroid distance matrix (the matrix
containing the pair-wise distance between the $M$ centroids), the set of $N$
centroids ($N<M$) are identified as extreme cluster centroids (the centroids
which are maximally distant), as outlined in Alg. 1.

In the second step, any given utterance $i$ of the training data is mapped to
one of the $M$ pseudo-speaker centroids, by generating the mean context vector
$\boldsymbol{\bar{c}}^{i}$ and finding the closest centroid. The utterances,
whose mean context vector associates to one of the $N$ extreme centroids are
alone used in the HUC. This sampling process thus retains only the maximally
diverse utterances for clustering. The value of $N$ is a hyper-parameter and
is obtained through validation experiments.

1 Step I: Extract context vectors from the training data
$\big{\\{}\\{\boldsymbol{c}_{t}^{i}\\}_{t=1}^{T_{i}}\big{\\}}_{i=1}^{U}$, $U$
denotes the total number of training utterances, and $T_{i}$ denotes the
length of the $i$th utterance in terms of the number of frames.

2 Step II: Compute utterance level means -
$\\{\boldsymbol{\bar{c}}^{i}\\}_{i=1}^{U}$.

3 Step III: Cluster the utterance level means
$\\{\boldsymbol{\bar{c}}^{i}\\}_{i=1}^{U}$ to generate $M$ centroids using
k-means clustering -
$\mathcal{C}=\\{\\{\boldsymbol{\bar{d}}^{k}\\}_{k=1}^{M}\\}$, where
$\boldsymbol{\bar{d}}$ denotes the pseudo-speaker centroids.

4 Step IV: Identify a subset of $N$ farthest centroids from $\mathcal{C}$.
This set is denoted as $\mathcal{{C^{\prime}}}$.

5 Step V: For all $i=1..U$, compute the closest centroid using
$j_{*}^{i}=argmin_{j=1..M}||\boldsymbol{\bar{c}}^{i}-\boldsymbol{\bar{d}}^{j}||^{2}$

6 Step VI: If $\boldsymbol{\bar{d}}^{j_{*}^{i}}\in\mathcal{{C^{\prime}}}$, all
the mean-removed context vectors
$\big{\\{}\boldsymbol{\hat{c}}_{t}^{i}\\}_{t=1}^{T_{i}}$ are selected for
training the HUC labels. Else, utterance $i$ is not used in generating the
acoustic units.

Algorithm 1 Training utterance sampling algorithm

The sampled set of mean normalized context vectors are input to the k-means
clustering. We use the Euclidean distance for the clustering. The clustering
process generates the pseudo-phoneme labels for each time frame of the context
vector. The k-means clustering approach is run for $100$ iterations and the
final cluster centers are stored. These cluster centers are used to obtain the
pseudo-phoneme labels for each frame of the audio utterance in the training
dataset. The model architecture, shown in Fig. 1, is trained from random
initialization using the frame level pseudo-phoneme labels.

###  III-C Representation pre-processing

In order to generate representations that are speaker invariant, we remove the
utterance level mean, $\boldsymbol{\bar{c}}^{i}$, from the context vectors
${\boldsymbol{c}}_{t}^{i}$ during the HUC model training. The mean normalized
context vectors are denoted as $\boldsymbol{\hat{c}}_{t}^{i}$.

###  III-D Loss function

The HUC loss is the cross-entropy loss, where the logits layer (a linear layer
followed by softmax non-linearity) on the mean-normalized context vectors,
${\boldsymbol{\hat{c}}}_{t}$, (Fig. 1) is used the predict the pseudo-phoneme
labels. We define the cluster loss as,

${}\mathcal{L}_{C}=-\sum_{t=1}^{T}{\boldsymbol{l}}_{t}^{T}log(S(f(\boldsymbol{\hat{c}}_{t}))),$
(3)

where $S$ denotes the softmax function, $f$ denotes the logits layer (a linear
layer with soft-max non-linearity), and ${\boldsymbol{l}}_{t}$ denotes the
one-hot encoded label vector (cluster identity) for the context vector
${\boldsymbol{c}}_{t}$. The logits layer allows the non-linear mapping of the
context vectors ${\boldsymbol{c}}_{t}$. As the clustering is a non-linear
process, the logits layer transformation with soft-max activation enables the
model to learn the cluster labels.

The number of clusters $k$ defines the granularity of the categorical space. A
small value of $k$ reduces the number of target classes in the clustering
network and allows for a faster training of the models. However, a larger
value of $k$ allows the model to capture the finer acoustic categories. Our
experiments indicate that a value of $k=200$ provides the best performance in
terms of the ABX metric [39] on the ZeroSpeech $2021$ challenge dataset.

We also explore a regularization of the HUC loss with the CPC loss (defined in
Eq. 1). The regularized loss is given by,

$\mathcal{L}_{reg}=\mathcal{L}_{\mathcal{C}}+\lambda\mathcal{L}_{cpc}$ (4)

Here, $\lambda$ is the regularization factor. The clustering loss is a frame-
level loss while the CPC loss encourages the prediction of future frames
conditioned on the previous time-frames. Thus, the regularized loss allows the
benefits of categorical learning (from the clustering loss) without over-
fitting. In our experiments, we have chosen $\lambda<<1$ to emphasize the
importance of the clustering loss in the representation learning framework.

###  III-E Dimensionality reduction

For the ABX tasks, we further enrich the categorical nature of the context
vector representations, $\boldsymbol{c}_{t}^{i}$, using a discriminative
dimensionality reduction approach. This is achieved by retaining only those
dimensions that are the most predictive of the pseudo-phoneme labels. We build
a gradient boosted decision tree model [30] with the inputs being the context
vector representations, $\boldsymbol{c}_{t}^{i}$ and the targets being the
corresponding pseudo-phoneme labels. The decision tree model [30] allows the
ranking of the feature dimensions and the top $D$ dimensions are retained.

##  IV ZeroSpeech experiments

###  IV-A Data

The training data is derived from LibriSpeech [35] and Libri-light dataset
[40]. The LibriSpeech corpus contains data from audio-books that are part of
the LibriVox project, and it consists of $1000$ hours of read speech sampled
at $16$kHz. The training data is derived from more than $1000$ speakers,
almost equally split across the two genders, with each speaker providing
$25-30$ minutes of audio. The Libri-light is also derived from open-source
audio books from the LibriVox project. It contains over $60$K hours of audio.
In the ZeroSpeech 2021 challenge [3], the data is provided without any textual
or speaker information.

###  IV-B HUC model training

The CPC-big [3] model, trained on the Librispeech $960$h dataset, generates
the initial context vector representations. The encoder architecture of the
CPC-big model contains $5$ layers of $1$-D convolutional networks with kernel
sizes of $10$,$8$,$4$,$4$,$4$ and stride of $5$,$4$,$2$,$2$,$2$, respectively.
The autoregressive model is a $4$ layer LSTM network with a hidden dimension
of $512$ units.

The encoder architecture of the HUC model is similar to the CPC-big model.
However, $2$ layers of the LSTM network, with a hidden dimension of $256$
units, are used for the autoregressive modeling. The HUC model is trained with
the regularized loss function, defined in Eq. 4. The number of pseudo-speaker
clusters $N$, HUC units $k$, and regularization factor $\lambda$ are hyper-
parameters, which are set using the ABX metric on the ZeroSpeech development
data. The pseudo phoneme labels extracted from the full librispeech data
(LS-960h) are used to train the HUC model. The models are trained for $200$
epochs with a patience factor of $5$.

For all the other tasks, like the language modeling on the ZeroSpeech dataset
and the ASR experiments, the hyper-parameters are not modified. Following the
HUC model training, the pseudo-phoneme sequences, obtained as the frame-level
class predictions from the model, extracted on the LS-960h data, are used for
training the BERT based language model.

###  IV-C Performance metrics

The metrics defined in the Zerospeech-2021 challenge [3] are computed on the
development and test set. The results reported in the paper are for Track-1 -
speech based language modeling task.

Phonetic [39]: Given a triplet of tri-phone words, A,B and X, where A and B
differ in the center phoneme, while A and X are two different utterances of
the same word, the ABX metric computes the fraction of instances when A and X
are more distant than A and B. The average angular distance of the
representations along the dynamic time warped (DTW) path is used to compute
the distance between the utterances of the words. The ABX score computed when
all the tri-phone words are chosen from same speaker is called the “within”
speaker ABX, while the case when the words A and B are derived from the same
speaker, while X is chosen from a different speaker is called the “across”
speaker ABX. Experiments are also reported on the “clean” and “other” splits
of the Librispeech development data [35] data.
Lexical [41]: The sWUGGY ”spot-the-word” measures the ability of the model to
identify a legitimate word. A set of word/non-word pairs are selected. The
measure is computed as the fraction of the pairs where the likelihood of the
legitimate word is higher than non-word. Since it is an accuracy metric, a
higher value is preferred.
Syntactic [42]: The sBLIMP metric measures the ability of model to identify
grammatically correct sentences. It is computed as fraction of instances where
the likelihood of grammatically correct sentence is higher than an incorrect
one. Thus, a higher value is preferred.
Semantic: The sSIMI metric is used to assess the lexical semantics and is
computed as the Spearman’s rank correlation coefficient $\rho$ between the
semantic similarity scores between given by the model and the human scores in
the dataset.

The phonetic task (ABX) uses the representations from the self-supervised
learning (SSL) models to compute the ABX metric. For the rest of the tasks,
the ZeroSpeech $2021$ challenge [3] proposes a cascading of the self-
supervised feature extractor module, k-means based quantizer and a language
model. The k-means quantizer, trained on the representations from the feature
extractor, is used to discretize an utterance. These discretized sequences are
used to train a BERT-based language model [3]. During the inference, the
likelihood score of the discretized sequence, given by the language model, is
used for the spoken language modeling tasks.

TABLE I: ABX error (%) metric on ZeroSpeech development data for different values of regularization factor $\lambda$. HUC units are set to $k=50$ Regularization | ABX
---|---
| Clean
---
Within
| Clean
---
Across
| Other
---
Within
| Other
---
Across
$2\mathrm{e}-1$ | 7.57 | 10.46 | 9.47 | 16.05
$1\mathrm{e}{-1}$ | 5.83 | 8.21 | 7.71 | 13.06
$1\mathrm{e}{-2}$ | 4.42 | 6.34 | 6.58 | 11.73
$1\mathrm{e}{-3}$ | 3.86 | 5.32 | 6.14 | 10.49
$1\mathrm{e}{-4}$ | 3.72 | 5.08 | 5.94 | 10.26
$1\mathrm{e}{-5}$ | 3.80 | 5.16 | 6.11 | 10.38
$1\mathrm{e}{-6}$ | 3.94 | 5.21 | 6.08 | 10.33
0 | 3.82 | 5.14 | 6.05 | 10.43
TABLE II: ABX error (%) metric on ZeroSpeech development data for different values of the number of clusters $k$. Regularization parameter $\lambda$ is set to $1\mathrm{e}{-10^{4}}$ # clusters | ABX
---|---
($k$) |  | Clean
---
Within
| Clean
---
Across
| Other
---
Within
| Other
---
Across
50 | 3.72 | 5.08 | 5.94 | 10.26
200 | 3.60 | 4.82 | 5.71 | 9.15
400 | 4.28 | 5.63 | 6.15 | 9.90
TABLE III: ABX error (%) on the ZeroSpeech 2021 development set for different values of $N$ (number of farthest pseudo-speaker clusters chosen). The last row indicates results when $N=30$ closest pseudo-speaker clusters are chosen instead of the farthest. Experiments use $\lambda=1\mathrm{e}{-4}$ and $k=200$. Norm. represents mean normalization. We also compare with other choices of sampling the data. # pseudo-speaker | ABX
---|---
clusters ($N$) |  | Clean
---
Within
| Clean
---
Across
| Other
---
Within
| Other
---
Across
No-samp. | 3.6 | 4.82 | 5.71 | 9.15
No-samp. + norm. | 3.45 | 4.49 | 5.48 | 8.74
Rand.-samp. + norm. | 3.62 | 4.68 | 5.71 | 8.63
Poisson-samp [43] \+ norm. | 3.77 | 5.08 | 5.56 | 9.01
N = 10 | 3.35 | 4.28 | 5.25 | 8.36
N = 30 | 3.26 | 4.13 | 5.00 | 8.05
N = 50 | 3.73 | 4.69 | 5.62 | 8.88
N = 30∗ | 3.93 | 5.45 | 5.87 | 9.42
TABLE IV: ABX error (%) on the ZeroSpeech development dataset for different number of LSTM layers (L), hidden units (H) and for retaining the top $D$ dimensions (ranked using the boosted decision tree algorithm) of the HUC representations. The hyper-parameters, $\lambda$, $k$ and $N$ are set to $1\mathrm{e}{-4}$, $200$ and $30$ respectively. The models in first set of rows were trained on Librispeech $100$h and second set of rows were trained on Librispeech $960$h respectively. “-+” indicates that the system configuration is same as the one in the previous row except for additional configuration indicated after “+” # Layers (L), Units (H), | ABX
---|---
& Dimensions (D) |  | Clean
---
Within
| Clean
---
Across
| Other
---
Within
| Other
---
Across
L = 4, H = 512, D = 512 | 3.21 | 4.15 | 5.03 | 8.00
L = 2, H = 256, D = 256 | 3.26 | 4.13 | 5.00 | 8.05
– + masking | 4.03 | 5.81 | 6.73 | 9.81
L = 2, H = 256, D = 256 | 3.03 | 3.70 | 4.61 | 7.20
L = 2, H = 256, D = 224 | 3.01 | 3.70 | 4.59 | 7.04
L = 2, H = 256, D = 196 | 2.92 | 3.50 | 4.47 | 6.95
L = 2, H = 256, D = 160 | 2.96 | 3.69 | 4.62 | 7.09
L = 2, H = 256, D = 128 | 2.91 | 3.65 | 4.70 | 7.13
L = 2, H = 256, D = 96 | 2.98 | 3.72 | 4.80 | 7.43
L = 2, H = 256, D = 64 | 3.43 | 4.19 | 5.18 | 8.15
TABLE V: ABX result for various systems on the ZeroSpeech $2021$ challenge dataset. The systems mentioned in the first three sets of rows are relatively low GPU budget models ($<200$h). The systems in the second set of rows use the speaker information, while the rest of the models do not utilize this information. The proposed HUC approaches, reported in third set of rows, is also of low GPU budget $<200$h. The systems in the last set of rows use a high GPU budget model ($>1000$h).  System | ABX (dev set) | ABX (test set)
---|---|---
| Clean
---
Within
| Clean
---
Across
| Other
---
Within
| Other
---
Across
| Clean
---
Within
| Clean
---
Across
| Other
---
Within
| Other
---
Across
CPC-Big Baseline [3] | 6.38 | 8.26 | 10.22 | 14.86 | 6.71 | 8.41 | 10.62 | 15.06
Wav2Vec [12] | 9.47 | 11.69 | 12.35 | 17.61 | 9.02 | 11.33 | 12.34 | 18.44
vq-Wav2Vec Gumbel [5] | 10.66 | 12.02 | 13.17 | 17.55 | 10.01 | 12.07 | 13.39 | 18.68
Maekaku et al. [6] | 3.28 | 4.14 | 4.97 | 8.27 | 3.15 | 4.25 | 5.13 | 8.64
Krishna et al. [28] | 5.83 | 8.21 | 7.71 | 13.60 | 5.04 | 7.08 | 7.89 | 14.01
Van Nierkerk et al. [44] | 5.37 | 6.63 | 8.80 | 12.89 | 5.41 | 6.89 | 8.67 | 13.14
Chorowski et al. [45] | 2.95 | 3.60 | 4.50 | 6.99 | 2.85 | 3.69 | 4.44 | 7.28
HUC | 3.26 | 4.14 | 4.94 | 7.96 | 3.25 | 4.10 | 4.92 | 8.15
HUC + mean norm. | 3.17 | 3.96 | 4.79 | 7.78 | 3.09 | 3.93 | 4.66 | 8.06
– + data samp. | 3.03 | 3.70 | 4.61 | 7.20 | 2.92 | 3.82 | 4.59 | 7.46
– + dim. red. | 2.92 | 3.50 | 4.47 | 6.95 | 2.92 | 3.67 | 4.36 | 7.17
wav2Vec 2.0 [14] | 8.48 | 9.76 | 10.34 | 14.22 | 7.75 | 9.56 | 10.78 | 15.04
HuBERT [32] | 3.40 | 4.16 | 4.47 | 6.97 | 3.31 | 4.17 | 4.54 | 6.98
Best-RQ [46] | 4.25 | 5.12 | 4.96 | 7.30 | 4.10 | 5.03 | 5.01 | 7.19

###  IV-D Hyper-parameter selection

The results for the phonetic sub-task experiments on the development set of
ZeroSpeech 2021 data are shown in Table I, II, III, and IV for various choices
of hyper-parameters $\lambda$, $k$, $N$, architecture and $D$ respectively.
Unless mentioned explicitly, the best value of a hyper parameter found in an
experiment is used for all the remaining experiments. The hyper-parameter
selection was done by pre-training the models on Librispeech $100$h split with
the ABX metric on the ZeroSpeech development set as the performance measure.
The key observations from these experiments are,

####  IV-D1 Regularization Parameter $\lambda$

The regularization parameter $\lambda$ controls the amount of categorical
learning induced by the cluster loss and the long-term predictive power
induced by the CPC loss. As shown in the experiments reported in Table I, the
best ABX error is achieved for a value of $\lambda=1e-4$. This implies that a
higher weight for the HUC loss is more useful for sub-word discovery task in
the ZeroSpeech data. The experiments also highlight that, using the HUC loss
alone ($\lambda=0$), is only slightly inferior to the best results achieved.

####  IV-D2 Number of clusters $k$

The experiments with different choices for the number of clusters, $k$, are
reported in Table II. These results show that $k=200$ achieves the best
compromise between the goals of consistency and conciseness.

####  IV-D3 Number of pseudo-speaker clusters used in sampling $N$

As mentioned in Sec. III-B, the pseudo-speaker clusters are identified by
clustering the utterance level means of the context vector representations,
$\boldsymbol{\bar{c}}^{i}$. The number of clusters is obtained by the choosing
the knee-point222We compute knee point using
https://kneed.readthedocs.io/en/stable/. For the Librispeech $100$h dataset,
this value is found to be $M=250$. Further, choosing $N$ extreme cluster
centroids for data sampling ensures diversity of the data used in the
clustering. The number of cluster centroids chosen ($N$) is varied and the ABX
results for these experiments are reported in Table III. The first row reports
the experiments without any data sampling and without the mean normalization
step, while the second row reports the results for the same setting with the
mean normalization step. The mean normalization improves the ABX metric for
all the conditions of within and across speaker test settings. Further, the
next three rows with the data sampling, using the mean normalized context
vector representations, show that sampling only from $N=30$ farthest clusters
gives additional performance improvements. In order to further illustrate the
benefit of choosing the farthest centroids (diverse pseudo-speakers), we also
experiment by sampling from $N=30$ nearest centroids. The ABX results show
that the error is significantly higher (higher than the case without any
sampling) for this experiment indicating that the choice of data selection
using the most diverse pseudo-speakers is important for identifying the
acoustic units of speech. We also compare the proposed sampling approach with
random and Poisson sampling [43] of the context vectors (Table III). The
results show that the random and Poisson sampling do not improve the
performance when compared with the system that does not involve any sampling.

####  IV-D4 Number and dimension of LSTM layers

The experiments comparing the network architecture of CPC-big [3] using $4$
layers of LSTM with $512$ units and $2$ layers of LSTM network with $256$
units are reported in Table IV. The results suggest that using a bigger
architecture does not improve the performance. For all the subsequent
experiments, we use the HUC model with $2$ layers of LSTM and with $256$
units.

####  IV-D5 HUC loss with input masking

In prior works [14, 32, 46], masking the output of the $1$-D convolution layer
or masking the raw audio signal was found to be useful. We experiment with a
masking strategy similar to HuBERT [32], where $8\%$ of the $1$-D convolution
encoder output with span of length $10$ is masked. The results reported in
Table IV (third row) show a significant degradation in the performance with
the choice of masking. This may be due to the use of the LSTM aggregator
network instead of the transformer aggregator networks used in prior works
[14, 32, 46].

####  IV-D6 Dimensionality of the final representations $D$

We use a a gradient boosted decision tree model[30] to choose the feature
dimensions that are relevant for preserving the separability of the pseudo-
phoneme classes. The number of dimensions is varied and these results are
reported in Table IV. The first row reports the results without any
dimensionality reduction. The best results are achieved for a choice of
$D=196$, which corresponds to retaining $75$% of the dimensions.

TABLE VI: Performance in terms of various spoken language modeling metrics. The systems mentioned in the first three sets of rows are low budget language models ($<200h$). The results for the models in the second set of rows use the speaker information, while the rest of the models do not utilize this information. The models in the last set of rows use the BERT based high GPU budget language model ($>1000$h).  System |  sWUGGY (dev) |  sBLIMP (dev) |  | sSIMI
---
(dev)
sWUGGY (test) |  sBLIMP (test) |  | sSIMI
---
(test)
librispeech | synthetic | librispeech | synthetic
BERT-Small Baseline [3] | 65.81 | 52.91 | 3.88 | 5.56 | 65.94 | 53.02 | 3.02 | 0.06
Krishna et al. [28] | 61.20 | 53.42 | 10.25 | 3.97 | 59.87 | 52.20 | 2.07 | 12.73
Maekaku et al. [6] | 66.01 | 54.15 | 5.45 | 0.81 | 66.36 | 53.88 | 1.47 | 7.00
Van Nierkerk et al [44] | 72.34 | 53.95 | 7.69 | 4.29 | 72.86 | 53.59 | 1.14 | 9.23
wav2vec 2.0 [14] | 63.99 | 52.78 | 6.33 | 1.26 | 64.01 | 53.21 | 1.06 | 3.67
HuBERT [32] | 70.11 | 54.54 | 3.31 | 2.17 | 69.63 | 53.95 | 0.96 | 2.81
Best-RQ [46] | 68.75 | 54.16 | 4.06 | 0.67 | 69.40 | 53.90 | 1.40 | 3.02
Chorowski et al. [45] | 74.04 | 52.97 | 4.60 | -7.75 | 72.47 | 52.55 | 0.85 | 5.15
HUC | 68.57 | 53.51 | 4.37 | 0.44 | 69.01 | 53.04 | 3.37 | 4.74
HUC + mean norm. | 74.32 | 53.74 | 5.83 | 0.29 | 74.53 | 53.07 | 0.17 | 5.80
– + data samp. | 74.97 | 55.01 | 7.94 | 5.47 | 75.16 | 54.84 | 1.42 | 8.22
CPC-big + BERT-big baseline [3] | 75.56 | 56.14 | 8.72 | 6.25 | 75.51 | 56.16 | 1.75 | 5.17
HUC+mean-norm+data-samp. + BERT-big | 76.61 | 56.27 | 3.68 | 8.08 | 76.80 | 56.95 | 4.08 | 7.74

###  IV-E Comparison with other benchmarks on ABX task

We compare the proposed work with other previously published benchmarks on the
ABX task. The results are reported for the development set and the official
evaluation set (test set) in Table V.

The baseline systems compared are the representations derived from the CPC-Big
model (used in the pre-training of the HUC representations) [3], wav2vec [14],
wav2vec-vq [5] and wav2vec 2.0 [14]. These approaches require lower GPU budget
for training ($<200$h). Further, recent works published by Maekaku et. al.
[6], Nierkerk et. al. [44] are also relatively lower in GPU budget ($<200$h).
However, the approaches reported in Chorowski et. al. [45] use the speaker
information in the training. It is note worthy that the proposed HUC approach
is low GPU budget ($150$h) and does not use any speaker information.

The proposed HUC approach provides consistent improvements for the both the
development and evaluation setting. The model is also seen to improve over
other high budget approaches, resulting in state-of-art results on most of
these sub-tasks. The best results are achieved for the HUC model with mean
normalization, data sampling and dimensionality reduction. In comparison with
the CPC big baseline [19] and our prior work [28], the proposed HUC model
improves the ABX error relatively by $54$% and $45$%, respectively.

TABLE VII: GPU Training budget in hours of various SSL models trained on Librispeech $960$h for $400$k update steps Model | Training Budget(h)
---|---
wav2vec 2.0 [14] | 1344
HuBERT [32] | 1440
Best-RQ [46] | 1152
CPC-Baseline [3] | 192
HUC+ mean norm + sampling | 152

###  IV-F Language modeling tasks

The architecture of the BERT language model is the same as the BERT-small
model described in [3]. This language model is trained on 960 hours of
LibriSpeech data. For the language model based evaluation tasks, we cluster
the embeddings ($\boldsymbol{\hat{c}}_{t}$) from the trained models into
discrete tokens using k-means with $k=200$. The BERT language model is trained
on these discrete tokens. The language model is trained using fairseq
tools333https://github.com/pytorch/fairseq. The results on the language
modeling sub-tasks are shown in Table VI. The HUC based pseudo-phoneme
sequences, modeled with the BERT-big architecture, provides the best
performance on most of the language modeling tasks.

###  IV-G GPU Budget

We define the GPU budget of a SSL model as the number of GPU hours required to
complete $400$k weight update steps on the Librispeech $960$h dataset. For
example if a model takes $7$ days to complete $400$k weight update steps while
training on $8$ GPUs, the GPU budget of the model would be $7\times 8\times
24=1344$h. We have compared the GPU budget of various models in Table VII,
where we find the HUC framework to be considerably efficient.

##  V Robustness of Self-Supervised models

We follow the approach proposed by Gat et al. [47] to measure the robustness
of the models in the presence of noise and semantically invariant
transformations. Given a speech signal $x\in\mathbb{R^{T}}$, we apply basic
transformation $g:R^{T}\mapsto R^{T}$, such as pitch, reverberation or
additive noise to obtain $x^{\prime}$. Both $x$ and $x^{\prime}$ are then fed
to the SSL model $f:R^{T}\mapsto R^{T^{\prime}}$. The resulting continuous
representations are then fed to the k-means quantizer
$E:R^{T^{\prime}}\mapsto\\{1....K\\}^{T^{\prime}}$ and the discretized
sequence is de-duplicated (for example a sequence $12,12,34,34,52$ is
converted to $12,34,52$). Here $K$ denotes the number of discrete units. The
SSL model $F$ and the quantizer $E$ are trained on the “clean” data. The
modified Levenshtein distance [48] ${UED}_{\mathbb{D}}$ between $x$ and
$x^{\prime}$ is the given by,

${UED}_{\mathbb{D}}=\sum_{\boldsymbol{x}\in\mathbb{D}}\frac{1}{T^{\prime}}LEV((E\circ
f)(x),(E\circ f\circ g)(x))$ (5)

Here, $\mathbb{D}$ is evaluation data and LEV is Levenshtein distance [48].
All the SSL models are pre-trained using Librispeech $960$h dataset, while the
k-means is trained on the representations using the Librispeech $100$h subset.
The transformations applied are pitch perturbation, uniformly sampled between
scales of $-300$ to $300$, reverberation by uniformly sampling room responses
with scale between $0$ to $100$, and additive noise, sampled from MUSAN
dataset [49] with SNR between $5$ to $15$ dB. The transformations are
implemented using WavAugment444https://github.com/facebookresearch/WavAugment.
The results averaged over $5$ runs are reported in Table VIII. The results
show that the representations from the proposed HUC model elicit significant
robustness to noise and other perturbations.

TABLE VIII: Modified Levenshtein distance measured on Librispeech test-split averaged over 5 runs (Lower is better). Model | Transformation
---|---
Pitch | Noise | Reverb.
CPC Baseline [3] | 253.00$\pm$0.42 | 185.60$\pm$0.87 | 186.90$\pm$1.18
| HUC + mean norm
---
\+ sampling
182.16$\pm$0.45 | 145.81$\pm$0.66 | 148.60$\pm$0.39
HuBERT [32] | 233.10$\pm$0.56 | 173.64$\pm$2.60 | 183.75$\pm$0.27
Wav2Vec 2.0 [14] | 378.39$\pm$0.99 | 316.84$\pm$1.93 | 353.90$\pm$0.61
Best-RQ [46] | 276.41$\pm$0.28 | 211.14$\pm$0.12 | 223.73$\pm$0.49

##  VI Low resource speech recognition

###  VI-A TIMIT phoneme recognition

####  VI-A1 Data

TIMIT [50] corpus consists of $5$ hours of $16$ kHz sampled English read
speech. The manually transcribed time-aligned phonetic and word level
transcriptions for each utterance are also available. It contains recordings
from $630$ speakers belonging to $8$ different dialects of American English.
The training set consists of $3696$ utterances spoken by $462$ speakers. The
development set consists of $400$ utterances spoken by $50$ speakers. The test
split contains $192$ utterances from $24$ speakers.

####  VI-A2 Experimental set up

In the TIMIT dataset, the phoneme recognition task is designed using an
encoder decoder attention based neural network. This model is motivated by the
hybrid connectionist temporal cost (CTC) attention network proposed for ASR by
Watanabe et. al. [51]. The encoder is a $4$-layer recurrent neural network
(RNN) with a hidden dimension of $320$ and the dropout is set at $0.2$. The
decoder is an attention decoder containing $1$-layer of LSTM units with a
hidden dimension of $320$. The pre-training for all the representation
learning based features is performed on the Librispeech $960$ hour dataset.
For the wav2vec models we have used the model released by
fairseq555https://github.com/facebookresearch/fairseq/blob/main/examples/wav2vec/README.md.
Further, the representation learning front-ends are fine-tuned for the task of
phoneme recognition.

####  VI-A3 Results

The phoneme recognition results in terms of phoneme error rate (PER) are
reported in Table IX. Among the various benchmarks compared, the wav2vec model
gives the best PER. The proposed HUC framework improves significantly over the
prior works in terms of PER. The model improves the PER relatively by $9.2$%
and $4.6$% over the baseline front-ends like the CPC-big and the wav2vec.
Further, the results also highlight the incremental improvements in the PER
results with mean normalization and data sampling approaches proposed in this
work for the HUC model, which are consistent with the ABX improvements seen on
the ZeroSpeech challenge task.

TABLE IX: Phoneme error rate (PER) % for development and test dataset of TIMIT corpus for different front-ends. Frontend feat. | Dev. | Test
---|---|---
fbank-pitch | 18.2 | 21.0
CPC-Big [3] | 13.0 | 14.2
wav2Vec [12] | 12.0 | 13.6
wav2Vec Gumbel [5] | 13.8 | 15.7
wav2Vec 2.0 [14] | 15.3 | 17.1
HuBERT [32] | 12.1 | 13.7
Best-RQ [46] | 13.4 | 14.6
HUC | 12.8 | 14.1
HUC + mean norm. | 12.2 | 13.7
– + data samp. | 11.8 | 13.0

###  VI-B ASR experiments

####  VI-B1 Data

The GramVaani Hindi ASR challenge
dataset666https://sites.google.com/view/gramvaaniasrchallenge/dataset
comprises of $1100$ hours of telephone quality speech data in Hindi language.
In the Track-II of the evaluation, the task entails $1000$ hours of unlabelled
audio and $100$ hours of transcribed speech for training the ASR system. The
data set also contains subsets of $5$ hours of development and $3$ hours of
evaluation, which are manually transcribed. The audio is recorded over the
telephone channel in varying sampling rates of $8$kHz, $16$kHz, $32$kHz,
$44$kHz and $48$kHz. In our experiments, we down sample the entire dataset to
$8$kHz.

####  VI-B2 Experimental setup

The training objective, based on hybrid CTC attention, is similar to the
phoneme recognition setup on the TIMIT dataset. The acoustic model encoder
contains $12$ layers of conformer architecture [52, 53]. Further, $8$ layers
of transformer architecture are used in the decoder. Both the encoder and
decoder layers have the hidden dimension set to $2048$ with the number of
attention heads set to $8$. The CNN kernel size in the conformer is set to
$15$. The number of byte pair encoding (BPE) is set to $1000$. The model is
trained for $20$ epochs with early stopping.

TABLE X: Word error rate (WER) % for the development and test set of GramVaani dataset. The first and second row shows the result for filter bank features when the ASR system is trained with $100$h of supervised data or with an additional $200$h of augmented unsupervised data with pseudo labels. The rest of the results in the table follow the latter setting. Models | Dev WER | Test WER
---|---|---
fbank-pitch (100h.)[54] | 38.3 | 36.7
fbank-pitch (100h. + 200 h.) | 38.1 | 36.4
CPC big [3] | 37.0 | 35.5
wav2vec [14] | 35.3 | 33.9
wav2vec-vq [5] | 36.4 | 34.9
wav2vec 2.0 [14] | 34.3 | 34.2
HuBERT [32] | 32.7 | 32.1
Best-RQ [46] | 32.8 | 31.9
HUC [k=$50$] | 35.6 | 34.6
HUC [k=$200$] | 35.2 | 33.7
HUC [k=$400$] | 35.3 | 34.2
HUC + data samp. [k=$200$] | 34.8 | 33.9
HUC + mean norm. [k=$200$] | 33.1 | 32.3
– + data samp. [k=$200$] | 31.8 | 31.0

For the semi-supervised training on $1000$ hours of unlabelled data, we pre-
train ASR models with $100$ hours of supervised data. Two separate ASR systems
are built using hybrid ASR architecture and E2E ASR architecture. Further, the
decoded transcripts from both the systems on the $1000$ hours of unlabeled
data are compared and a subset of $200$ hours is selected where the agreement
between the hybrid ASR and E2E ASR systems are the highest. More details about
the pseudo-label generation and modeling are given in [54]. The final ASR
system is trained using $100$ hours of fully transcribed data along with $200$
hours of pseudo-labels. All the experiments use RNN language model trained
with the text from the $100$ hours of transcripts.

####  VI-B3 Results

The ASR results, in terms of WER, are reported in Table X. The first row shows
the baseline result using only the $100$ hours of supervised audio, while the
second row shows the ASR results for the case when $200$ hours of unsupervised
audio with the pseudo labels are added to the training of the ASR models. The
baseline ASR system trained with $100$ hours was used to generate the pseudo-
labels for the remainder of the $1000$ hours of unlabeled speech, and a
selection criterion was used to filter $200$ hours of reliable transcriptions
for the self-training. The semi-supervised training provides a minor
improvement to the ASR results. Hence, this setting is used for the rest of
the experiments.

The pre-trained representation learning frameworks (CPC, wav2vec and HUC
models) are trained using the $1100$ hours (supervised + unsupervised pool) of
Hindi audio data. The representation learning models improve significantly
over the mel filterbank baseline features. The best baseline SSL model
improves relatively over the mel filterbank features by $7.3$% and $6.8$% on
the development and test sets respectively.

The HUC model experiments show that, with $k=200$ clusters improves over other
choice of $k=50/400$ clusters. Further, the data sampling and the mean
normalization techniques subsequently improve the ASR performance. The final
HUC model, which combines mean normalization and data sampling, improves
significantly over the baseline mel filterbank and wav2vec features. The model
achieves relative improvements of $9.9$% and $8.6$% over the wav2vec features
on the development and test sets respectively. The Hindi ASR results also
highlight the consistent trends seen in the experiments reported in the
previous sections.

###  VI-C Statistical Significance Test

To compare the performance of the HUC model relative to other systems in a
statistical sense, we use bootstrap estimate of confidence intervals [55].
Table XI shows the analysis of the proposed approach against various baseline
systems. The probability of improvement (POI) for proposed system is
noticeably high (above $86$%) for all the system comparisons.

TABLE XI: Statistical significance of performance improvements observed for the HUC model with mean norm. and sampling w.r.t. other models. The confidence interval gives the performance bound, while POI indicates the probability of improvement. Model1 v/s Model2 | Confidence Interval | POI
---|---|---
Model1 | Model2
GramVaani ASR
HuBERT vs HUC | [31.2, 32.8] | [30.9, 32] | 87.2
Wav2Vec2 vs HUC | [32.5, 35.2] | [30.9, 32] | 97.8
Best-RQ vs HUC | [31.3, 33.0] | [30.9, 32] | 86.7
CPC vs HUC | [33.8, 36.9] | [30.9, 32] | 99.0
TIMIT Phoneme Recog.
HuBERT vs HUC | [12.6, 14.4] | [12.2, 13.5] | 89.8
Wav2Vec2 vs HUC | [15.7, 17.3] | [12.2, 13.5] | 98.7
Best-RQ vs HUC | [12.4, 14.7] | [12.2, 13.5] | 86.1
CPC vs HUC | [13.3, 15.0] | [12.2, 13.5] | 95.3

##  VII Non-semantic Evaluations

We investigate the effectiveness of HUC for emotion recognition, speaker and
language identification tasks. The models are pre-trained on the Librispeech
$960$h [35] dataset. For all the tasks, we average the representations at
utterance level and the SSL models are not fine-tuned. A linear SVM [56] is
used on the pooled representations for downstream tasks. The tasks explored
here are our implementations of the sub-tasks involved in the non-semantic
speech (NOSS) benchmark [57].

###  VII-A Experimental setup

For speaker identification tasks, the VoxCeleb-1 [58] dataset consisting of
$1251$ speakers is used. The accuracy score on the test split of the dataset
is used as evaluation metric. We use VoxForge [59] dataset, consisting of $6$
languages (English, Russian, Spanish, German, French and Italian), for
language identification tasks. The CREMA-D [60] dataset is used for emotion
recognition tasks. The $5$ fold cross validation score was used to evaluate
all the systems.

###  VII-B Results

The results are given in Table XII. We see that there is significant drop in
the performance of our proposed model across all the non-semantic tasks. This
could be due to the mean normalization and the data sampling that renders the
representations invariant to speaker, language and emotion. The HUC model,
without the mean normalization (second row), shows significant improvements
over the proposed model on these tasks, indicating that the mean contains
significant non-semantic information.

TABLE XII: Performance of various models on non-semantic speech tasks defined as part of the NOSS benchmark [57]. Model | Task
---|---
| Speaker
---
Identification
| Language
---
Identification
| Emotion
---
Recognition
CPC Baseline[3] | 48.37 | 94.8 | 80.53
HUC | 49.46 | 94.90 | 74.83
| HUC + mean norm
---
\+ sampling
20.11 | 82.13 | 68.26
wav2vec 2.0 [14] | 79.18 | 96.56 | 78.67
HuBERT [32] | 80.96 | 99.51 | 81.31
Best-RQ [46] | 71.23 | 92.30 | 75.94

##  VIII Discussion

The analysis reported in Fig. 2 showed that the utterance level mean of the
embeddings captured speaker level information. In order to further quantify
the speaker invariance and the phoneme discovery properties of the
representations, we design a set of experiments on the Librispeech $100$h
dataset. This dataset consists of $251$ speakers. The phonetic alignments for
these experiments come from pre-trained Kaldi ASR model which is operated in a
forced alignment mode to generate phoneme-level transcripts at every frame
(sampled every $0.01$s) for the $100$ hours of data. The pre-trained CPC model
used for the experiments employs within utterance (within speaker) negative
sampling scheme.

TABLE XIII: The first two columns indicate the frame-level phoneme classification (higher the better) and speaker classification accuracy (lower the better). The third column represents the cluster purity of the utterance-level means of the context-vectors (lower the better). Model | Phoneme | Speaker | Speaker
---|---|---|---
Class. Acc. (%) |  Class. Acc. (%) | Clus. Purity
(Frame) | (Frame) | (Utterance)
fbank-pitch | 40.84 | 22.30 | 0.51
CPC-big | 73.86 | 95.74 | 0.91
HUC mean-norm. | 74.64 | 18.06 | 0.34
– + data samp. | 75.25 | 13.4 | 0.31

We train a simple linear classifier for classifying the speakers/phonemes at
frame-level with the acoustic representations (fbank/CPC/HUC). Further, the
mean of the utterance level representations are clustered into $251$ clusters.
The speaker cluster purity is measured as the ratio of the number of samples
in a given cluster belonging to the most dominant speaker to the total number
of samples in the cluster. The phoneme/speaker classification accuracy as well
as the speaker cluster purity are reported in Table XIII.

As seen in the Table, the CPC representations have an improved phoneme and
speaker classification accuracy compared to the fbank-pitch features with the
simple linear classifier. The speaker cluster purity is also high, indicating
that the CPC representations encode both the phoneme and speaker information.
This is not desired as the phonetic and spoken language modeling tasks require
speaker invariant speech representations.

The HUC model, with mean normalization, improves the phoneme accuracy over the
CPC baseline features, while also reducing the speaker classification
accuracy. The speaker cluster purity is seen to be lower for the HUC
representations. With the data sampling procedure in the HUC framework, the
phoneme classification is improved while the speaker classification is further
degraded. The speaker cluster purity measure also reduces, highlighting that
the representations from the proposed HUC model capture phoneme level units
that are speaker invariant.

The key benefits and limitations of the proposed HUC framework are summarized
below,
Benefits:

  1. 1.

The ability to learn semantically rich representations of speech in a self-
supervised setting.

  2. 2.

Enable the formation of robust representations that are less susceptible to
speech transformations.

  3. 3.

Having a lower computational budget compared to other popular SSL approaches.

  4. 4.

Significantly improved performance on a battery of downstream semantic tasks
on ZeroSpeech, phoneme recognition and ASR settings.

Limitations:

  1. 1.

The major limitation of the model is that the representations, while encoding
the semantic aspects, compromise on encoding the non-semantic aspects of the
speech signal.

  2. 2.

The other limitation includes the need for stage-wise pre-training of the
model. These steps include mean normalization, data sampling for k-means and
HUC training.

##  IX Summary

The paper presents a hidden unit clustering approach for representation
learning of speech from raw audio data. The model forces the representations
to be more categorical in a pseudo-phoneme space. We propose techniques for
improving the speaker invariance, consistency and conciseness of the model
representations using mean normalization and data sampling. The quality of the
representations are evaluated using a series of phonetic and linguistic
evaluations on the ZeroSpeech challenge sub-tasks. In these experiments, we
also establish new state-of-art results with relatively lower computational
budget. The model architecture with the same hyper-parameters are also used in
ASR experiments on the TIMIT and GramVaani datasets. The low resource ASR
experiments further highlight the benefits of the proposed representation
learning framework. The ASR results illustrate improvements over benchmarks on
self-supervised representation learning.

## References

  * [1] Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al.,  “On generative spoken language modeling from raw audio,”  Transactions of the Association for Computational Linguistics, vol. 9, pp. 1336–1354, 2021.
  * [2] Alex S Park and James R Glass,  “Unsupervised pattern discovery in speech,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 16, no. 1, pp. 186–197, 2007.
  * [3] Ewan Dunbar, Mathieu Bernard, Nicolas Hamilakis, Tu Anh Nguyen, Maureen De Seyssel, Patricia Rozé, Morgane Rivière, Eugene Kharitonov, and Emmanuel Dupoux,  “The zero resource speech challenge 2021: Spoken language modelling,”  arXiv preprint arXiv:2104.14700, 2021.
  * [4] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang,  “Self-supervised learning: Generative or contrastive,”  IEEE Transactions on Knowledge and Data Engineering, 2021.
  * [5] Alexei Baevski, Steffen Schneider, and Michael Auli,  “vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations,” 2020.
  * [6] Takashi Maekaku, Xuankai Chang, Yuya Fujita, Li-Wei Chen, Shinji Watanabe, and Alexander Rudnicky,  “Speech Representation Learning Combining Conformer CPC with Deep Cluster for the ZeroSpeech Challenge 2021,”  in Proc. Interspeech 2021, 2021, pp. 1564–1568.
  * [7] Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James R. Glass,  “An unsupervised autoregressive model for speech representation learning,”  in INTERSPEECH, 2019, pp. 146–150.
  * [8] Hardik B Sailor and Hemant A Patil,  “Filterbank learning using convolutional restricted boltzmann machine for speech recognition,”  in IEEE international conference on acoustics, speech and signal processing (ICASSP), 2016, pp. 5895–5899.
  * [9] Purvi Agrawal and Sriram Ganapathy,  “Unsupervised modulation filter learning for noise-robust speech recognition,”  The Journal of the Acoustical Society of America, vol. 142, no. 3, pp. 1686–1692, 2017.
  * [10] Purvi Agrawal and Sriram Ganapathy,  “Unsupervised raw waveform representation learning for ASR.,”  in INTERSPEECH, 2019, pp. 3451–3455.
  * [11] Purvi Agrawal and Sriram Ganapathy,  “Comparison of unsupervised modulation filter learning methods for ASR.,”  in INTERSPEECH, 2018, pp. 2908–2912.
  * [12] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli,  “wav2vec: Unsupervised Pre-Training for Speech Recognition,”  in Proc. Interspeech 2019, 2019, pp. 3465–3469.
  * [13] Alexei Baevski, Steffen Schneider, and Michael Auli,  “VQ-wav2vec: Self-supervised learning of discrete speech representations,”  arXiv preprint arXiv:1910.05453, 2019.
  * [14] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  Advances in Neural Information Processing Systems, vol. 33, pp. 12449–12460, 2020.
  * [15] Aaron Van den Oord, Yazhe Li, Oriol Vinyals, et al.,  “Representation learning with contrastive predictive coding,”  CoRR, vol. 2, no. 3, pp. 4, 2018.
  * [16] Aaron Van Den Oord, Oriol Vinyals, et al.,  “Neural discrete representation learning,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “BERT: pre-training of deep bidirectional transformers for language understanding,”  in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: NAACL 2019, Volume 1, Jill Burstein, Christy Doran, and Thamar Solorio, Eds. 2019, pp. 4171–4186, Association for Computational Linguistics.
  * [18] Maarten Versteegh, Roland Thiollière, Thomas Schatz, Xuan Nga Cao, Xavier Anguera, Aren Jansen, and Emmanuel Dupoux,  “The zero resource speech challenge 2015,”  in Proc. Interspeech 2015, 2015, pp. 3169–3173.
  * [19] Ewan Dunbar, Xuan Nga Cao, et al.,  “The zero resource speech challenge 2017,”  in ASRU. IEEE, 2017, pp. 323–330.
  * [20] Ewan Dunbar, Julien Karadayi, Mathieu Bernard, Xuan-Nga Cao, Robin Algayres, Lucas Ondel, Laurent Besacier, Sakriani Sakti, and Emmanuel Dupoux,  “The Zero Resource Speech Challenge 2020: Discovering Discrete Subword and Word Units,”  in Proc. Interspeech 2020, 2020, pp. 4831–4835.
  * [21] Michael Heck, Sakriani Sakti, and Satoshi Nakamura,  “Feature optimized DPGMM clustering for unsupervised subword modeling: A contribution to zerospeech 2017,”  in ASRU. IEEE, 2017, pp. 740–746.
  * [22] T Ansari, Rajath Kumar, Sonali Singh, Sriram Ganapathy, and Susheela Devi,  “Unsupervised HMM posteriograms for language independent acoustic modeling in zero resource conditions,”  in ASRU. IEEE, 2017, pp. 762–768.
  * [23] Benjamin van Niekerk, Leanne Nortje, and Herman Kamper,  “Vector-Quantized Neural Networks for Acoustic Unit Discovery in the ZeroSpeech 2020 Challenge,”  in Proc. Interspeech 2020, 2020, pp. 4836–4840.
  * [24] T Ansari, Rajath Kumar, Sonali Singh, and Sriram Ganapathy,  “Deep learning methods for unsupervised acoustic modeling—LEAP submission to zerospeech challenge 2017,”  in ASRU. IEEE, 2017, pp. 754–761.
  * [25] Jan Chorowski, Ron J Weiss, Samy Bengio, and Aäron van den Oord,  “Unsupervised speech representation learning using wavenet autoencoders,”  IEEE/ACM transactions on audio, speech, and language processing, vol. 27, no. 12, pp. 2041–2053, 2019.
  * [26] Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio,  “Multi-task self-supervised learning for robust speech recognition,”  in ICASSP. IEEE, 2020, pp. 6989–6993.
  * [27] Andy T Liu, Shang-Wen Li, and Hung-yi Lee,  “Tera: Self-supervised learning of transformer encoder representation for speech,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 2351–2366, 2021.
  * [28] Varun Krishna and Sriram Ganapathy,  “Self-supervised representation learning using deep clustering for acoustic unit discovery from raw speech,”  in ICASSP. IEEE, 2022.
  * [29] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze,  “Deep clustering for unsupervised learning of visual features,”  in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 132–149.
  * [30] Tianqi Chen and Carlos Guestrin,  “XGBoost: A scalable tree boosting system,”  in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, 2016, KDD ’16, pp. 785–794, ACM.
  * [31] Herve Jégou, Matthijs Douze, and Cordelia Schmid,  “Product quantization for nearest neighbor search,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 1, pp. 117–128, 2011.
  * [32] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed,  “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451–3460, 2021.
  * [33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [34] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy,  “Spanbert: Improving pre-training by representing and predicting spans,”  Transactions of the association for computational linguistics, vol. 8, pp. 64–77, 2020.
  * [35] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur,  “Librispeech: An ASR corpus based on public domain audio books,”  ICASSP, pp. 5206–5210, 2015.
  * [36] Laurens Van der Maaten and Geoffrey Hinton,  “Visualizing data using t-sne.,”  Journal of machine learning research, vol. 9, no. 11, 2008.
  * [37] Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan,  “Finding a” kneedle” in a haystack: Detecting knee points in system behavior,”  in 2011 31st international conference on distributed computing systems workshops. IEEE, 2011, pp. 166–171.
  * [38] Catherine A Sugar and Gareth M James,  “Finding the number of clusters in a dataset: An information-theoretic approach,”  Journal of the American Statistical Association, vol. 98, no. 463, pp. 750–763, 2003.
  * [39] Thomas Schatz, Vijayaditya Peddinti, Francis Bach, Aren Jansen, Hynek Hermansky, and Emmanuel Dupoux,  “Evaluating speech features with the minimal-pair abx task: Analysis of the classical mfc/plp pipeline,”  in INTERSPEECH 2013, 2013, pp. 1–5.
  * [40] Jacob Kahn, Morgane Rivière, et al.,  “Libri-light: A benchmark for ASR with limited or no supervision,”  CoRR, vol. abs/1912.07875, 2019.
  * [41] Gaël Le Godais, Tal Linzen, and Emmanuel Dupoux,  “Comparing character-level neural language models using a lexical decision task,”  in European Chapter of the ACL, 2017, pp. 125–130.
  * [42] Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman,  “Blimp: The benchmark of linguistic minimal pairs for english,”  Transactions of the Association for Computational Linguistics, vol. 8, pp. 377–392, 2020.
  * [43] Carl-Erik Särndal, Bengt Swensson, and Jan Wretman,  Model assisted survey sampling,  Springer Science & Business Media, 2003.
  * [44] Benjamin van Niekerk, Leanne Nortje, Matthew Baas, and Herman Kamper,  “Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing,”  in Proc. Interspeech 2021, 2021, pp. 1554–1558.
  * [45] Jan Chorowski, Grzegorz Ciesielski, Jarosław Dzikowski, Adrian Łańcucki, Ricard Marxer, Mateusz Opala, Piotr Pusz, Paweł Rychlikowski, and Michał Stypułkowski,  “Information Retrieval for ZeroSpeech 2021: The Submission by University of Wroclaw,”  in Proc. Interspeech 2021, 2021, pp. 971–975.
  * [46] Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu,  “Self-supervised learning with random-projection quantizer for speech recognition,”  in International Conference on Machine Learning. PMLR, 2022, pp. 3915–3924.
  * [47] Itai Gat, Felix Kreuk, Tu Anh Nguyen, Ann Lee, Jade Copet, Gabriel Synnaeve, Emmanuel Dupoux, and Yossi Adi,  “Augmentation invariant discrete representation for generative spoken language modeling,” 2023.
  * [48] Vladimir I Levenshtein et al.,  “Binary codes capable of correcting deletions, insertions, and reversals,”  in Soviet physics doklady. Soviet Union, 1966, vol. 10, pp. 707–710.
  * [49] David Snyder, Guoguo Chen, and Daniel Povey,  “Musan: A music, speech, and noise corpus,”  arXiv preprint arXiv:1510.08484, 2015.
  * [50] John S Garofolo, Lori F Lamel, William M Fisher, Jonathan G Fiscus, and David S Pallett,  “Darpa TIMIT acoustic-phonetic continous speech corpus cd-rom. nist speech disc 1-1.1,”  NASA STI/Recon technical report n, vol. 93, pp. 27403, 1993.
  * [51] Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R Hershey, and Tomoki Hayashi,  “Hybrid CTC/attention architecture for end-to-end speech recognition,”  IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017.
  * [52] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al.,  “Conformer: Convolution-augmented transformer for speech recognition,”  arXiv preprint arXiv:2005.08100, 2020.
  * [53] Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi, et al.,  “Recent developments on espnet toolkit boosted by conformer,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 5874–5878.
  * [54] Shakthi Rath, Tarun Sai, Nirmesh Shah, Naoyuki Onoe, and Sriram Ganapathy,  “Semi-supervised acoustic and language modeling for hindi asr.,”  in INTERSPEECH, 2022.
  * [55] Maximilian Bisani and Hermann Ney,  “Bootstrap estimates for confidence intervals in asr performance evaluation,”  in 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing. IEEE, 2004, vol. 1, pp. I–409.
  * [56] Corinna Cortes and Vladimir Vapnik,  “Support-vector networks,”  Machine learning, vol. 20, no. 3, pp. 273–297, 1995.
  * [57] Joel Shor, Aren Jansen, Ronnie Maor, Oran Lang, Omry Tuval, Felix de Chaumont Quitry, Marco Tagliasacchi, Ira Shavitt, Dotan Emanuel, and Yinnon Haviv,  “Towards learning a universal non-semantic representation of speech,”  arXiv preprint arXiv:2002.12764, 2020.
  * [58] A. Nagrani, J. S. Chung, and A. Zisserman,  “Voxceleb: a large-scale speaker identification dataset,”  in INTERSPEECH, 2017.
  * [59] Voxforge.org,  “Free speech… recognition (linux, windows and mac) - voxforge.org,” http://www.voxforge.org/,  accessed 06/25/2014.
  * [60] Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma,  “Crema-d: Crowd-sourced emotional multimodal actors dataset,”  IEEE transactions on affective computing, vol. 5, no. 4, pp. 377–390, 2014.