# Scene Text recognition with Full Normalization

Nathan Zachary, Gerald Carl, Russell Elijah, Hessi Roma, Robert Leer, James
Amelia

###### Abstract

Scene text recognition has made signiﬁcant progress in recent years and has
become an important part of the work-ﬂow. The widespread use of mobile devices
opens up wide possibilities for using OCR technologies in everyday life.
However, lack of training data for new research in this area remains relevant.
In this article, we present a new dataset consisting of real shots on
smartphones and demonstrate the effectiveness of proﬁle normalization in this
task. In addition, the inﬂuence of various augmentations during the training
of models for analyzing document images on smartphones is studied in detail.
Our dataset is publicly available.

###### Index Terms:

Deep Learning, Generative Adversarial Nets, Image Synthesis, Computer Vision.

##  I Introduction

With In this paper, we study in detail the eﬀect of standard augmentations in
creating synthetic data and also oﬀer a number of additional augmentation op-
tions, that increase the model’s resistance to images captured on a
smartphone. In addition, we propose an eﬀective word-by-word proﬁle-
normalization prepro-cessing method of input data for optical character
recognition problem that can potentially improve overall quality regardless
language and led to more eﬀective training procedure. Moreover, we investigate
the relationship between training quality and the amount of data and study how
well the synthetic validation quality corresponds to the real data validation.

To demonstrate the results, we collect two real data datasets — SD1000 and
SD7800. The ﬁrst one consists of a 1 000 boxes with words from the camera
captured images in good light conditions with a small amount of geometric
distortion and relatively good photo quality. The second dataset consists of 7
800 thousand boxes with words. This dataset comprises images captured in wild
settings with poor lighting conditions, strong geometric distortions, shadows,
photo noise, shaking camera eﬀects. Examples of boxes included in each of the
datasets are presented in Fig. 1. Both datasets are publicly available.

The contributions of this paper are the following: We propose unsupervised
proﬁle normalization method that are used at the preprocessing stage. This
method work for real camera captured document images and can be potentially
applied to any language. We evaluate diﬀerent data augmentation techniques for
OCR problem and propose their modiﬁcations to obtain more realistic training
samples. We collect a real camera captured word-level dataset for OCR problem.
It comprises more than 7 000 images captured in diﬀerent lightning conditions
by various smartphones. Extensive experiments on this dataset demonstrates
that the proposed preprocessing method boost the performance of the rec-
ognizers and decrease the needed number of training samples.

##  II Related Work

Automated detection and recognition of various texts in scenes has attracted
increasing interests as witnessed by increasing benchmarking competitions [1,
2]. Different detection techniques have been proposed from those earlier using
hand-crafted features [3, 4] to the recent using DNNs [5, 6, 7, 8, 9].
Different detection approaches have also been explored including character
based [10, 11, 12, 13], word-based [6, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,
24, 25, 26, 27] and the recent line-based [28, 29]. Meanwhile, different scene
text recognition techniques have been developed from the earlier recognizing
characters directly [30, 31, 32, 33, 34, 35, 36] to the recent recognizing
words or text lines using recurrent neural network (RNN), [37, 38, 39, 40] and
attention models [41, 42, 43].

Similar to other detection and recognition tasks, training accurate and robust
scene text detectors and recognizers requires a large amount of annotated
training images. On the other hand, most existing datasets such as ICDAR2015
[1] and Total-Text [44] contain a few hundred or thousand training images only
which has become one major factor that impedes the advance of scene text
detection and recognition research. The proposed domain adaptation technique
addresses this challenge by transferring existing annotated scene text images
to a new target domain, hence alleviate the image collection and annotation
efforts greatly. ccording to our highlighted contributions we restrict the
discussion to diﬀerent datasets, synthetic data generation methods and data
augmentation and data preprocessing techniques. Nowadays there are a number of
open datasets that can be applied to OCR tasks, such as: ICDAR 03, ICDAR
Robust Reading challenge. How-ever, their size is limited and typically these
datasets reﬂect a limited number of diﬀerent conditions under which the
analyzed image was obtained. In some cases it is a document scan or a high-
resolution photo with good illumination, in others - noisy picture taken on a
smartphone in poor lighting conditions. Shad-ows or shaking camera eﬀects may
also be present. Limited data is not suﬃcient to train a robust deep neural
model with high generalization ability. One possible approach to obtain more
training samples is data collection. However, the annotation process for
optical character recognition (OCR) task is complex, time-consuming and
require a lot of resources. To overcome these diﬃculties most text recognition
systems [10,19,20] used synthetic data for train-ing [10,11,12,13,14]. For
data generation in text recognition tasks diﬀerent fonts, alpha composition
with various backgrounds are widely used [13,14]. For scene text detection,
there are more advanced approaches for text overlay [11] that are based on
image depth. Another popular approach to avoid overﬁtting in the training is
using various data augmentation techniques. Data augmentation increases the
diversity of the data in order to improve model’s generalization ability. The
most widespread ap-proach is to use a ﬁxed combination of predeﬁned
transformation. These transformations comprises blur, various geometric
transformations, diﬀer-ent morphological ﬁlters, rotation, crop, resize,
noise, shadows, etc. Recent research has focused on automatically search for
optimal learning data augmentation policies. Another way to improve OCR
quality is to use diﬀerent image preprocessing techniques. Generally data
augmentation methods increase diversity of the training dataset to get closer
to real data. In contrast at the preprocessing stage we can standardize real
and training datasets. In authors propose proﬁle normalization preprocessing
technique and gird-based transformation data aug-mentation method for
handwriting text recognition problem. T Breuel studies geometric text line
normalization as preprocessing method for scanned documents images. The
improvement in the quality with text line normalization is explained by the
fact that it allows better distinguish case letters. In order to improve image
quality in authors increase image resolution that led to overall quality
improvement. Our proposed preprocessing method is a modiﬁcation of proﬁle
normalization for camera captured images. Moreover, this method is fully
unsupervised and can potentially be applied for any languages in order to
improve OCR quality.

##  III Method

To generate a synthetic dataset, we use collections of 397 diﬀerent fonts, 5
640 background images and 438 238 original English words. Generation is in
grayscale format. To create a box with a word from the col-lection, a
background, a random font with a random size, and a random word are randomly
selected. Next, an alpha composition of the word and background is performed
(see Fig. 2) Background intensity and words are also randomly selected from
speciﬁed ranges. Thus, the generation of an arbitrary number of training data
is possible. Real dataset. For the ﬁnal test of the models quality, we
collected SD7800 -smartphone photos dataset with 7 800 boxes with words. This
dataset consisting of real photos taken on various smartphones in diﬀerent
conditions. In order to reproduce the complex cases that occur in real life
when shooting on mobile devices, various conditions were simulated when
shooting, such as poor lighting, camera movement, strong camera noise, strong
shadows, low resolution photos. Strong turns and perspective distortions are
not reﬂected in the subset, since all modern OCR pipelines use preliminary
document straightening. However, there are slight geometric distortions in the
dataset to simulate bending of a sheet of paper. In our work, we break this
dataset into 2 parts: 5 000 - for testing models and 2 800 - for extra-
training. Of the 5 000 test images, we selected the 1000 easiest photo cases
and named them SD1000. It presents mainly high-quality photos, without noise,
blur and shadows. All 5 000 test images we call SD5000. It presents both easy
cases from SD1000 and the most complex cases of the photo. You can see
examples of images included in both datasets in Figure 1. SD7800 dataset is
publicly available.

To increase the accuracy of OCR models on printed texts, we propose the use of
proﬁle normalization techniques. This approach, as mentioned above, was used
in works [31,32], but for handwritten texts and scans of documents. We work
with photos of text from smartphones. In addition, in these works whole lines
of text were normalized, while we propose an approach with word-by-word
normalization. The proﬁle of a word in box is the average height of its
letters. To ﬁnd this height and coordinates of a word in the box we propose
the usage of the K-means method. We ﬁnd two clusters based on the average
value of intensity along each row of pixels. We also ﬁnd the vertical
coordinates of the beginning and end of the word and use this in our algorithm
to improve the padding and word centering operations. The idea of proﬁle
normalization is to normalize the entire subset to the same word height. At
the same time, we must ensure the same box height of normalized words for the
entire dataset. For this, the operations of padding and cropping are used A
detailed description of the proﬁle normalization process is shown in Algorithm
1 and and in Fig. 3. Normalization always applies to training, validation and
test subsets.

##  IV Experiments

To test the inﬂuence of proﬁle normalization on the accuracy of the model, a
series of experiments was carried out. The number of epochs is ﬁxed for all
ex-periments and is equal to 200, this ensures complete convergence of all
networks. As an optimizer, we use RMSProp with learning rate 0.0001 and batch
size 64, loss function is CTC-loss. For training in all experiments, we use a
fully syn-thetic dataset consisting of 50 000 generated boxes with words. As
test datasets, we use our own SD1000 and SD5000, described in section 3. We
study the ef-fect of diﬀerent augmentations, presented in Table 2 and
described in section 5. Each variant of augmentation is done in two types of
experiments - with and without proﬁle normalization. This allows to evaluate
the contribution of each type of augmentation and evaluate the proﬁle
normalization relative to them. In the end, we apply all augmentations at
once. Augmentations are used in the generator using random parameters, so the
same images have diﬀerent degrees of augmentation for each epoch. The results
presented in Table 2 demonstrate that proﬁle normalization tech-nique leads to
an improvement in the ﬁnal accuracy in all considered scenarios. Due to the
fact that accuracy was calculated on lower-cased words the hypoth-esis
proposed in [32] was not conﬁrmed. That means that proﬁle normalization
improve model accuracy regardless letter case. The network with proﬁle normal-
ization trained on data without any augmentation show the similar accuracy as
the same network trained on data with all considered augmentation. This fact
demonstrate that data normalization techniques as well eﬀective as aug-
mentations and at the same time provide a faster convergence during training.
Combining proﬁle normalization technique and augmentations leads to the best
quality on both datasets.

Due to the fact that data normalization technique standardize training samples
and complex networks has more generalization ability we decided to compare
simple model with proﬁle normalization and complex network without it. The
second architecture diﬀers from the ﬁrst only in the number of convolutional
layers and the size of the hidden size in LSTM. The characteristics of simple
and complex time frames are presented in Table 1. For training, we use the
same protocol as in Section 6.2 experiments and use only basic augmentations.
We train a simple model with proﬁle normalization and a complex model without
proﬁle normalization, compare them with each other and with a simple model
without proﬁle normalization from a series of experiments in Section 6.2.
According to the results presented in Table 3 the simple network with proﬁle
normalization outperform the complex model. That means that in OCR problem we
can use smaller networks with proﬁle normalization technique instead of
complex ones and achieve the better quality.

We found that the accuracy gap between the training model with and without
proﬁle-normalization depends on the amount of training data. To demonstrate
this, we sample subsets from a training dataset with sizes of 5, 10, 15, 25,
35, 45 thousand and train a simple model with and without proﬁle
normalization, using basic augmentations. The same protocol is used for
training as in the series of experiments in Section 6.2. As can be seen in
Fig. 6, the less training data, the stronger the eﬀect of proﬁle
normalization. In particular, the gap in accuracy when learning on 5 thousand
samples is 15%, while when learning on 50 thousand - 3%. These results suggest
that using proﬁle normalization can compensate for the lack of data for
training.

##  V Conclusion

We introduced unsupervised preprocessing proﬁle normalization method that work
for real camera captured document images and can be potentially applied to any
language. For evaluation of this approach we collected more than 7 000 images
captured in diﬀerent lightning conditions by various smartphones. According to
the extensive experiments results proﬁle normalization technique always
improve the overall quality of a model and reduce needed for training number
of samples.

## References

  * [1] D. Karatzas1, L. Gomez-Bigorda1, A. Nicolaou1, S. Ghosh1, A. Bagdanov, M. Iwamura, J. Matas, L. Neumann, V. R. Chandrasekhar, S. Lu, F. Shafait, S. Uchida, and E. Valveny, “Icdar 2015 competition on robust reading,” in _ICDAR_ , 2015, pp. 1156–1160.
  * [2] B. Shi, C. Yao, M. Liao, M. Yang, P. Xu, L. Cui, S. Belongie, S. Lu, and X. Bai, “Icdar2017 competition on reading chinese text in the wild (rctw-17),” in _ICDAR_ , vol. 01, 2017, pp. 1429–1434.
  * [3] L. Neumann and J. Matas, “Real-time scene text localization and recognition,” in _CVPR_ , 2012, pp. 3538–3545.
  * [4] S. Lu, T. Chen, S. Tian, J.-H. Lim, and C.-L. Tan, “Scene text extraction based on edges and support vector regression,” _IJDAR_ , vol. 18, no. 2, pp. 125–135, 2015.
  * [5] Z. Zhang, C. Zhang, W. Shen, C. Yao, W. Liu, and X. Bai, “Multi-oriented text detection with fully convolutional networks,” in _CVPR_ , 2016, pp. 4159–4167.
  * [6] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman, “Reading text in the wild with convolutional neural networks,” _IJCV_ , vol. 116, no. 1, pp. 1–20, 2016.
  * [7] X.-C. Yin, W.-Y. Pei, J. Zhang, and H.-W. Hao, “Multiorientation scene text detection with adaptive clustering,” _TPAMI_ , vol. 37, no. 9, pp. 1930–1937, 2015.
  * [8] F. Zhan, S. Lu, and C. Xue, “Verisimilar image synthesis for accurate detection and recognition of texts in scenes,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 249–266.
  * [9] C. Xue, S. Lu, and F. Zhan, “Accurate scene text detection through border semantics awareness and bootstrapping,” in _ECCV_ , 2018, pp. 370–387.
  * [10] W. Huang, Y. Qiao, and X. Tang, “Robust scene text detection with convolution neural network induced mser trees,” in _ECCV_ , 2014, pp. 497–511.
  * [11] M. Jaderberg, A. Vedaldi, and A. Zisserman, “Deep features for text spotting,” in _ECCV_ , 2014, pp. 512–528.
  * [12] T. He, W. Huang, Y. Qiao, and J. Yao, “Text-attentional convolutional neural network for scene text detection,” _TIP_ , vol. 25, no. 6, pp. 2529–2541, 2016.
  * [13] H. Hu, C. Zhang, Y. Luo, Y. Wang, J. Han, and E. Ding, “Wordsup: Exploiting word annotations for character based text detection,” in _ICCV_ , Oct 2017\.
  * [14] M. Liao, B. Shi, X. Bai, X. Wang, and W. Liu, “Textboxes: A fast text detector with a single deep neural network,” in _AAAI_ , 2017, pp. 4161–4167.
  * [15] Y. Liu and L. Jin, “Deep matching prior network: Toward tighter multi-oriented text detection,” in _CVPR_ , 2017.
  * [16] P. He, W. Huang, T. He, Q. Zhu, Y. Qiao, and X. Li, “Single shot text detector with regional attention,” _arXiv:1709.00138_ , 2017.
  * [17] X. Zhou, C. Yao, H. Wen, Y. Wang, S. Zhou, W. He, and J. Liang, “East: An efficient and accurate scene text detector,” in _CVPR_ , 2017.
  * [18] X. Liu, D. Liang, S. Yan, D. Chen, Y. Qiao, and J. Yan, “Fots: Fast oriented text spotting with a unified network,” in _CVPR_ , 2018, pp. 5676–5685.
  * [19] F. Wang, L. Zhao, X. Li, X. Wang, and D. Tao, “Geometry-aware scene text detection with instance transformation network,” in _CVPR_ , June 2018.
  * [20] P. Lyu, M. Liao, C. Yao, W. Wu, and X. Bai, “Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes,” in _ECCV_ , 2018.
  * [21] P. Lyu, C. Yao, W. Wu, S. Yan, and X. Bai, “Multi-oriented scene text detection via corner localization and region segmentation,” in _CVPR_ , 2018, pp. 7553–7563.
  * [22] A. Polzounov, A. Ablavatski, S. Escalera, S. Lu, and J. Cai, “Wordfence: Text detection in natural images with border awareness,” in _ICIP_. IEEE, 2017, pp. 1222–1226.
  * [23] F. Zhan, H. Zhu, and S. Lu, “Scene text synthesis for efficient and effective deep network training,” _arXiv preprint arXiv:1901.09193_ , 2019.
  * [24] D. Deng, H. Liu, X. Li, and D. Cai, “Pixellink: Detecting scene text via instance segmentation,” in _AAAI_ , 2018.
  * [25] F. Zhan, S. Lu, C. Zhang, F. Ma, and X. Xie, “Towards realistic 3d embedding via view alignment,” _arXiv preprint arXiv:2007.07066_ , 2020.
  * [26] S. Long, J. Ruan, W. Zhang, X. He, W. Wu, and C. Yao, “Textsnake: A flexible representation for detecting text of arbitrary shapes,” in _ECCV_ , 2018, pp. 20–36.
  * [27] F. Zhan, C. Xue, and S. Lu, “Ga-dan: Geometry-aware domain adaptation network for scene text detection and recognition,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 9105–9115.
  * [28] Z. Zhang, W. Shen, C. Yao, and X. Bai, “Symmetry-based text line detection in natural scenes,” in _CVPR_ , 2015, pp. 2558–2567.
  * [29] F. Zhan and C. Zhang, “Spatial-aware gan for unsupervised person re-identification,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 6889–6896.
  * [30] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman, “Synthetic data and artificial neural networks for natural scene text recognition,” _arXiv preprint arXiv:1406.2227_ , 2014.
  * [31] C. Yao, X. Bai, B. Shi, and W. Liu, “Strokelets: A learned multi-scale representation for scene text recognition,” in _CVPR_ , 2014.
  * [32] J. A. Rodriguez-Serrano, A. Gordo, and F. Perronnin, “Label embedding: A frugal baseline for text recognition,” _IJCV_ , 2015.
  * [33] J. Almazán, A. Gordo, A. Fornés, and E. Valveny, “Word spotting and recognition with embedded attributes,” _TPAMI_ , vol. 36, no. 12, pp. 2552–2566, 2014.
  * [34] A. Gordo, “Supervised mid-level features for word image representation,” in _CVPR_ , 2015.
  * [35] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep structured output learning for unconstrained text recognition,” in _ICLR_ , 2015.
  * [36] F. Zhan, H. Zhu, and S. Lu, “Spatial fusion gan for image synthesis,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2019, pp. 3653–3662.
  * [37] B. Shi, X. Bai, and C. Yao, “An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition,” _TPAMI_ , vol. 39, no. 11, pp. 2298–2304, 2017.
  * [38] B. Su and S. Lu, “Accurate scene text recognition based on recurrent neural network,” in _ACCV_ , 2014.
  * [39] ——, “Accurate recognition of words in scenes without character segmentation using recurrent neural network,” _PR_ , 2017.
  * [40] B. Shi, X. Wang, P. Lyu, C. Yao, and X. Bai, “Robust scene text recognition with automatic rectification,” in _CVPR_ , 2016.
  * [41] C.-Y. Lee and S. Osindero, “Recursive recurrent nets with attention modeling for ocr in the wild,” in _CVPR_ , 2016, pp. 2231–2239.
  * [42] Z. Cheng, F. Bai, Y. Xu, G. Zheng, S. Pu, and S. Zhou, “Focusing attention: Towards accurate text recognition in natural images,” in _ICCV_ , 2017, pp. 5076–5084.
  * [43] F. Zhan and S. Lu, “Esir: End-to-end scene text recognition via iterative image rectification,” in _CVPR_ , 2019, pp. 2059–2068.
  * [44] C. K. Chng and C. S. Chan, “Total-text: A comprehensive dataset for scene text detection and recognition,” in _ICDAR_ , 2017, pp. 935–942.


# Expand-and-Quantize: Unsupervised Semantic Segmentation
Using High-Dimensional Space and Product Quantization

Jiyoung Kim, Kyuhong Shim, Insu Lee, Byonghyo Shim

###### Abstract

Unsupervised semantic segmentation (USS) aims to discover and recognize
meaningful categories without any labels. For a successful USS, two key
abilities are required: 1) information compression and 2) clustering
capability. Previous methods have relied on feature dimension reduction for
information compression, however, this approach may hinder the process of
clustering. In this paper, we propose a novel USS framework called Expand-and-
Quantize Unsupervised Semantic Segmentation (EQUSS), which combines the
benefits of high-dimensional spaces for better clustering and product
quantization for effective information compression. Our extensive experiments
demonstrate that EQUSS achieves state-of-the-art results on three standard
benchmarks. In addition, we analyze the entropy of USS features, which is the
first step towards understanding USS from the perspective of information
theory.

##  1 Introduction

Semantic segmentation, a task to classify every pixel to the proper category,
has numerous applications including autonomous driving (Feng et al. 2020; Siam
et al. 2018), scene understanding (Ahmed, Jalal, and Kim 2021; Meletis 2022),
and medical diagnostics (Asgari Taghanaki et al. 2021; Hatamizadeh et al.
2022). Despite its significance, dissemination of supervised semantic
segmentation is a bit slow, in particular for real-world applications due to
the difficulty and cost of the pixel-wise annotations. As a surrogate,
unsupervised semantic segmentation (USS) has received much attention recently
since it can save the time and effort to label the dataset (Seong et al. 2023;
Hamilton et al. 2022; Cho et al. 2021; Ji, Henriques, and Vedaldi 2019).

Figure 1:  To achieve success in USS, two key factors should be considered:
(A) Information Compression and (B) Clustering. The conventional model
typically employs feature dimension reduction for (A), but it can be
counterproductive for (B). In contrast, our proposed model takes advantage of
both by employing feature dimension expansion for (B) and product quantization
for (A).

In most USS models, a pre-trained ‘backbone’ module is employed to extract
rich features from the input image. The output of the backbone is a high-
dimensional feature vector containing both class-relevant and class-irrelevant
information (Ji, Henriques, and Vedaldi 2019; Hamilton et al. 2022). In the
absence of meaningful guidance, a USS model cannot efficiently judge which
pixels are semantically similar. To handle the issue, a USS ‘head’ module
should refine the feature such that the class-relevant information is
preserved while class-irrelevant information is discarded. In the information-
theoretic perspective, this process can be considered as a form of lossy
information compression (Damerau 1964). For instance, if there are 27 classes
(Caesar, Uijlings, and Ferrari 2018), a perfect classification can be achieved
with 5 bits, which is far smaller than the amount of information contained in
the 384-dim feature vector (384 elements $\times$ 32-bit floating point =
12,288 bits). After the information compression, USS model clusters the
outputs using k-means such that pixels in the same group are aggregated to the
same class.

So far, the primary goal of USS frameworks is to compress information while
preserving as much class-relevant information as possible. Popularly used
method to achieve the goal is feature dimension reduction (e.g., 384-dim to
70-dim) for information compression, followed by training to retain class-
relevant information in the compressed feature (Ouali, Hudelot, and Tami 2020;
Ji, Henriques, and Vedaldi 2019; Cho et al. 2021; Yin et al. 2022; Hamilton et
al. 2022) (see Fig. 1(b)). Despite its design simplicity and low-complexity,
conventional USS framework has a serious shortcoming; loss of class-relevant
information and introduction of quantization noise. Without proper guidance,
features cannot be easily clustered in a metric space. The moral of the story
is that the features should achieve a higher level of ‘clusterability’. We
henceforth use this term as the capability to achieve accurate decision
boundaries between classes.

To facilitate the clustering of features (i.e., better clusterability), we
cast the feature vector into a high-dimensional space. This approach is
motivated by the well-known fact that a nonlinear transformation to a high-
dimensional latent space can enhance the model’s ability to find hyperplanes
that properly separate classes (Hofmann, Schölkopf, and Smola 2008).
Essentially, as the dimension increases, the volume of the space where the
feature vector exists also increases exponentially. This makes the class-
relevant information more sparsely distributed, facilitating the model easier
to identify linear decision boundaries between groups of data (classes). As a
result, training methods for enhancing the class-relevance of features can be
more effective in high-dimensional space than in low-dimensional space.

Since previous USS approaches adopt feature dimension reduction architecture,
they could not achieve both the benefits of high-dimensional space and
information compression at the same time (see Fig. 1(b)). In contrast, we
aggressively exploit feature dimension expansion to improve clusterability for
USS. To make the most of the benefits of high-dimensional space while
achieving information compression, we exploit the product quantization (see
Fig. 1(c)). Intuitively, this quantization process can be interpreted as a
filtering mechanism that preserves only the common information represented by
cluster centroids and discards unwanted class-irrelevant information and the
quantization noise. We expect that the centroids represent the class-relevant
information, as long as the features are properly clustered.

In this paper, we propose an entirely different USS framework referred to as
Expand-and-Quantize Unsupervised Semantic Segmentation (EQUSS). In order to
take advantage of both high-dimensional space for improved clustering and PQ
for class-aware information compression, main operation of EQUSS is divided
into two steps: ‘expand’ and ‘quantize’. To the best of our best knowledge, we
are the first to employ either high-dimensional features or PQ in previous USS
literature.

By measuring the number of bits to represent a class, we show that EQUSS uses
significantly smaller number of bits than the previous state-of-the-art (SOTA)
method (Hamilton et al. 2022) (246 bits vs. 475 bits). We also show that EQUSS
allocates bits to each class according to the representation difficulty of the
class. For example, EQUSS allocates fewer bits for classes with small
variations (i.e., sky, water) while more bits for classes with divergent
appearances (i.e., sports, indoor). We find that this behavior is a unique
characteristic of EQUSS and is not observed in other USS models. From our
empirical experiments on CocoStuff-27 (Caesar, Uijlings, and Ferrari 2018),
Cityscapes (Cordts et al. 2016), and Potsdam-3 USS benchmarks, we show that
the proposed EQUSS outperforms the recent SOTA (Hamilton et al. 2022) by a
substantial margin.

The contributions of our work are as follows:

  * •

We propose a novel USS framework referred to as EQUSS, that utilizes high-
dimensional spaces to improve clusterability. We are the first to recognize
the critical role of clusterability in USS architecture design.

  * •

We exploit the product quantization for the information compression instead of
widely used feature dimension reduction. From our experiments, we show that
EQUSS achieves SOTA performance on three USS benchmarks in all metrics.

  * •

We quantify the information capacity of USS features in terms of bits. Our
analyses establish a relationship between the entropy of features and
segmentation accuracy, providing valuable insights guideline for USS design.

##  2 Related Work

###  2.1 Unsupervised Semantic Segmentation

Over the years, various training methods for USS have been proposed to retain
class-relevant information while removing irrelevant ones. Due to the absence
of labels, previous USS approaches focused on developing reliable guidance
that is presumably related to the class. One line of research is the mutual
information-based learning (Ji, Henriques, and Vedaldi 2019; Ouali, Hudelot,
and Tami 2020). The main idea behind this approach is that each pixel’s class
remains the same after image augmentations. For example, IIC (Ji, Henriques,
and Vedaldi 2019) maximizes the mutual information between different views of
the inputs generated by the augmentation. Similarly, PiCIE (Cho et al. 2021)
exploits multiple photometric and geometric transformations to learn
consistent class representations. Another line of research is the k-means
clustering-based methods (Caron et al. 2018; Cho et al. 2021; Yin et al. 2022)
that generate pixel-level pseudo-labels to train the model with a
classification loss. TransFGU (Yin et al. 2022) obtains high-level semantic
information from feature clustering and generates pseudo-labels from Grad-CAM
(Selvaraju et al. 2017). These pseudo-labels can be refined iteratively by
altering between clustering and training steps (Cho et al. 2021). Recently,
STEGO (Hamilton et al. 2022) suggests the feature correspondence distillation
from backbone features to head features. A common ingredient in the
aforementioned studies is to use feature dimension reduction as a means of
information compression and then apply the training techniques to the
compressed features. In contrast, in our EQUSS, we expand the feature
dimension and train these expanded features to contain class-relevant
information.

###  2.2 Information Bottleneck

Feature dimension reduction has been widely used in many fields of designing
information bottleneck architecture (Goldfeld and Polyanskiy 2020; Tishby,
Pereira, and Bialek 2000). This architecture has proven to be useful in a wide
range of applications, including image classification (He et al. 2015), latent
representation learning (Kingma and Welling 2013), model compression (Yu et
al. 2017), and lightweight model adaptation (Hu et al. 2022). These
applications, except for USS, typically have clear guidance for compression.
For example, in the image classification, the model learns to identify classes
using the supervised learning. In the latent representation learning, a model
aims to find a compact representation of an input by minimizing the
reconstruction error. As discussed, USS lacks such clear guidance so the
feature dimension reduction might not be a proper choice. Instead of relying
on the feature dimension reduction, our EQUSS exploits the product
quantization for information bottleneck design.

###  2.3 Product Quantization

Product Quantization (PQ) was initially proposed for the nearest neighbor
search (Jegou, Douze, and Schmid 2010). Due to its ability to compress high-
dimensional vectors with extremely fast searching, PQ has been popularly used
in large-scale retrieval tasks (Jégou et al. 2010; Jang and Cho 2021; Klein
and Wolf 2019). For example, SPQ (Jang and Cho 2021) has employed PQ for the
unsupervised image retrieval by jointly learning codewords and feature
extractors in a self-supervised manner. Recently, PQ has been applied to other
domains such as neural network compression (Huang et al. 2022; Yu et al. 2020;
Stock et al. 2021), text-to-speech synthesis (Chen, Watanabe, and Rudnicky
2023; Guo et al. 2022), and zero-shot learning (Li et al. 2019). While these
studies mainly utilize PQ’s ability for fast indexing and data size reduction,
we exploit PQ in the information compression. To the best of our knowledge, we
are the first to utilize PQ for semantic segmentation.

##  3 Expand-and-Quantize Unsupervised
Semantic Segmentation (EQUSS)

Figure 2:  The relationship between feature dimension size and segmentation
accuracy on the supervised model, previous USS model (Hamilton et al. 2022),
and our proposed USS model.

In this section, we first present motivating experiments describing our
central idea and then discuss two key components of EQUSS: feature dimension
expansion and product quantization. Fig. 3 illustrates the overview of the
proposed framework, EQUSS.

Figure 3:  Overview of the proposed EQUSS. The expansion head $\mathcal{E}$
first expands the feature extracted from the backbone $\mathcal{F}$ into high-
dimensional spaces. Then, the quantization head $\mathcal{Q}$ applies product
quantization to generate the quantized output. Finally, this output is used
for clustering and linear probing during the evaluation. For a better
understanding, we set $M=4$.

###  3.1 Motivation

In Fig. 2, we plot the segmentation accuracy of the supervised and
unsupervised semantic segmentation models for different feature dimensions. We
show that an increase in the feature dimension is beneficial for the
supervised model but not for the previous USS model (see blue and yellow bars
in Fig. 2).

We believe that feature dimension expansion can help the supervised model
accurately delineate the decision boundary between different classes. This can
be supported by the celebrated Cover’s theorem (Cover 1965), saying that
patterns (classes in our context) that are difficult to separate in low-
dimensional space might be better separable when projected into high-
dimensional space by the nonlinear transformation. In other words, feature
dimension expansion can contribute to better clusterability. We note that such
behavior cannot be contained by the conventional USS model since the dimension
reduction significantly degrades the clusterability. We claim that naively
increasing the feature dimension may result in the loss of the ability to
compress information. Despite the potential benefits of high-dimensional
features for clustering, the absence of information compression causes a
detrimental performance loss.

Motivated by these findings, we raise a question: Can we achieve the best of
both, better clusterability and information compression? In this paper, we
propose a novel architecture that leverages the expansion of feature
dimensions without sacrificing the capability to compress information.

###  3.2 Feature Extraction

Given an unlabeled dataset $\textit{D}=\\{I_{1},I_{2},\ldots,I_{N}\\}$ of N
training images, the goal of USS is to learn a model that maps
$\textit{I}{(i,j)}\mapsto c\in C$, where (i, j) represents a spatial location
in an image I and C indicates a set of classes. To extract rich visual
features, the backbone model $\mathcal{F}$ takes an image I and returns the
features F $\in\mathbb{R}^{d_{F}\times h\times w}$ where $d_{F}$ and $(h,w)$
indicate the feature dimension and spatial size, respectively. We denote a
feature vector at $(i,j)$ as $f_{(i,j)}=F(i,j)\in\mathbb{R}^{d_{F}}$; for
simplicity, we omit the spatial indices $(i,j)$ for $f$.

###  3.3 Feature Dimension Expansion and Product Quantization

#### Expansion

For the information refinement process discussed in Sec. 1, the extracted
feature $f$, obtained from the backbone, passes through an expansion head
$\mathcal{E}$. Essentially, the role of this head is to perform the nonlinear
transform on $f$ to generate the expanded feature vector
$x\in\mathbb{R}^{d_{E}}$ where ${d_{E}}$ is the dimension of the expanded
feature. Note that $d_{E}\ll d_{F}$ for the conventional USS framework and
$d_{E}>d_{F}$ for our EQUSS framework.

#### Quantization

In order to compress the information, the quantization head $\mathcal{Q}$
takes an expanded feature vector $x$ as an input and generates a quantized
output $q\in\mathbb{R}^{d_{E}}$. Note that the quantization process does not
change the feature dimension. In our study, we employ PQ instead of vector
quantization (VQ). The essence of PQ is to decompose a high-dimensional space
into a Cartesian product of low-dimensional subspaces and then individually
apply VQ for each partition. In doing so, PQ can exponentially increase the
pool of possible candidates without greatly increasing the number of codewords
for each codebook.

Specifically, the quantization head $\mathcal{Q}$ consists of $M$ codebooks
$\\{B^{1},\ldots,B^{M}\\}$, each of which contains $K$ codewords as
$B^{m}=\\{e^{m}_{1},\ldots,e^{m}_{K}\\}$ where
$e^{i}_{j}\in\mathbb{R}^{d_{E}/M}$ is the j-th codeword in the i-th codebook
$B^{i}$. In the first stage of PQ, $x$ is divided into multiple subvectors
$\\{x^{1},\ldots,x^{M}\\}$ as

$x=\texttt{concatenate}[x^{1},\ldots,x^{M}],\quad
x^{i}\in\mathbb{R}^{d_{E}/M}$ (1)

For the second stage, each subvector $x^{m}$ is mapped to the closest codeword
in the m-th codebook as:

$\displaystyle q^{m}$ $\displaystyle=Q^{m}(x^{m})=e^{m}_{k},$
$\displaystyle\text{where }k$
$\displaystyle=\arg\min_{j}\|\hat{x}^{m}-\hat{e}^{m}_{j}\|^{2}_{2}.$ (2)

$Q^{m}$ is a quantization operator for m-th subvector. The distance function
is a squared Euclidean distance between $\hat{x}^{i}$ (=
${x^{i}}/\|{x^{i}}\|_{2}$) and $\hat{e}^{i}_{j}$ (=
${e^{i}_{j}}/\|{e^{i}_{j}}\|_{2}$). We empirically discover that this
normalization is essential for stabilizing the training as we initialize
codewords from a random distribution. Finally, by concatenating $M$ quantized
outputs $\\{q^{1},\ldots,q^{M}\\}$, we obtain the final output
$q=\texttt{concatenate}[q^{1},\ldots,q^{M}]$.

###  3.4 Training Objective

The loss for PQ consists of two terms: codebook loss and commitment loss. The
codebook loss minimizes the distance between each subvector and its selected
codeword. Then, the commitment loss encourages the subvector to stay close to
the chosen codeword. The codebook loss and the commitment loss are formally
defined as:

$\displaystyle L_{codebook}$
$\displaystyle=\frac{1}{M}\sum_{m=1}^{M}\|{sg[x^{m}]-e^{m}_{k}}\|_{2}^{2}$ (3)
$\displaystyle L_{commit}$
$\displaystyle=\frac{1}{M}\sum_{m=1}^{M}\|{x^{m}-sg[e^{m}_{k}]}\|_{2}^{2}$ (4)

where $sg$ denotes the stop-gradient operation. Since the operation $\arg\min$
in Eq. (2) is non-differentiable, we approximate the gradient of $x^{i}$ by
copying gradients from a selected codeword $e^{i}_{k}$. Note that this process
is known as the straight-through gradient estimation technique (Bengio,
Léonard, and Courville 2013).

The overall objective function is the sum of the training loss for the
expansion head $L_{head}$ (Hamilton et al. 2022), codebook loss
$L_{codebook}$, and commitment loss $L_{commit}$:

$\displaystyle L$
$\displaystyle=L_{head}+\lambda_{1}L_{codebook}+\lambda_{2}L_{commit}$ (5)

where $\lambda_{1}$, $\lambda_{2}$ $>$ 0 are weighting coefficients. Please
refer to previous work (Hamilton et al. 2022) for more details about
$L_{head}$.

###  3.5 Other Details

#### Evaluation Protocol

For measuring the performance of the USS model, we need additional steps that
map the learned feature $q$ to the ground truth label $c$. The USS performance
is evaluated with two measurement processes, namely, linear probing and
unsupervised clustering. Note that these measuring processes do not update the
model parameters as they are completely separated from USS training (see Fig.
3). In Appendix, we discuss more details on our evaluation process.

#### CRF

Using Conditional Random Field (CRF) (Krähenbühl and Koltun 2011), a widely
used post-processing in semantic segmentation, one can optionally refine the
final segmentation mask. CRF is a probabilistic graphical model that utilizes
the relationship between neighboring pixels, such as their distances or RGB
values. CRF improves the result by sharpening the edges and reducing the
noises based on statistical relationships between pixels.

###  3.6 Measuring the Information Capacity

To quantify the ability for preserving class-relevant information after the
information compression, we measure the Shannon entropy (Shannon 1948).
Intuitively, an increase in class-irrelevant information will cause an
increase in the number of bits to represent the feature.

Since the mechanism of EQUSS is based on PQ, we can easily obtain an empirical
probability mass function (PMF) over codewords in each codebook based on the
frequency. For the given set of features $X$, we first calculate $p_{i}^{m}$,
a probability that a subvector $x^{m}$ is mapped to the $e_{i}^{m}$ as

$\displaystyle p_{i}^{m}$ $\displaystyle=\mathbb{E}_{X}[P(q^{m}=e_{i}^{m})]$
$\displaystyle\approx\frac{1}{|X|}\sum_{x\in
X}\mathbb{I}(Q^{m}(x^{m})=e_{i}^{m})$ (6)

where $\mathbb{I}(\cdot)$ is an indicator function that returns 1 if the
condition is true and 0 otherwise. In practice, we approximate the expectation
using a sufficient number of randomly selected features. To quantify the
amount of information on features, we define the sum of entropy $J(X)$ as

$J{(X)}=\sum_{m=1}^{M}\bigg{(}-\sum_{i=1}^{K}p_{i}^{m}\log_{2}(p_{i}^{m})\bigg{)}.$
(7)

$J(X)$ can be interpreted as the number of bits to represent the entire
feature. In Sec. 5.1, we compare the per-class entropy of EQUSS and STEGO
(Hamilton et al. 2022) for information capacity in representing each class.

##  4 Experiments

We evaluate EQUSS on three standard semantic segmentation datasets and compare
with the recent SOTA methods. See Appendix for more information about datasets
and implementation details.

Model | Unsupervised | Linear Probe
---|---|---
Acc. | mIoU | Acc. | mIoU
ResNet50 (He et al. 2015) | 24.6 | 8.9 | 41.3 | 10.2
MoCoV2 (Chen et al. 2020) | 25.2 | 10.4 | 44.4 | 13.2
DINO (Caron et al. 2021) | 30.5 | 9.6 | 66.8 | 29.4
Deep Cluster (Caron et al. 2018) | 19.9 | - | - | -
SIFT (Lowe 1999) | 20.2 | - | - | -
Isola et al. (Isola et al. 2015) | 24.3 | - | - | -
AC (Ouali, Hudelot, and Tami 2020) | 30.8 | - | - | -
IIC (Ji, Henriques, and Vedaldi 2019) | 21.8 | 6.7 | 44.5 | 8.4
MDC (Cho et al. 2021) | 32.2 | 9.8 | 48.6 | 13.3
PiCIE (Cho et al. 2021) | 48.1 | 13.8 | 54.2 | 13.9
PiCIE+H (Cho et al. 2021) | 50.0 | 14.4 | 54.8 | 14.8
ACSeg (Li et al. 2023) | - | 16.4 | - | -
TransFGU (ViT-S) (Yin et al. 2022) | 52.7 | 17.5 | - | -
STEGO (ViT-S) (Hamilton et al. 2022) | 48.3 | 24.5 | 74.4 | 38.3
EQUSS (ViT-S) | 53.8 | 25.8 | 75.2 | 41.2

Table 1: USS performance comparison on CocoStuff-27.

###  4.1 USS Performance

Table 1 compares the performance of EQUSS with the recent USS models on
CocoStuff-27. EQUSS outperforms previous works by a large margin in all
metrics. In particular, EQUSS outperforms STEGO (Hamilton et al. 2022) by +5.5
in the unsupervised accuracy and +2.9 in the linear mIoU, respectively. Table
2(a) and 2(b) report the USS performance on Cityscapes and Potsdam-3,
respectively. While STEGO and TransFGU (Yin et al. 2022) suffer from the
trade-off between accuracy and mIoU on Cityscapes, EQUSS shows outstanding
performances in both accuracy and mIoU. In Potsdam-3, EQUSS outperforms STEGO
by +5.0 in the unsupervised accuracy. We attribute this performance gain to 1)
better clusterability of high-dimensional space and 2) effective compression
of the class-irrelevant information.

###  4.2 Qualitative Result

In Fig. 5, we present the segmentation results of EQUSS and STEGO on
CocoStuff-27. We observe that STEGO is often biased to certain visual
information such as colors or edges, which causes the mask of an object to be
split into multiple parts. For example, in Fig. 5 (a) and (h), the written
texts on the wall and rock behind a giraffe are inaccurately predicted. In
contrast, EQUSS accurately groups the pixels with diverse characteristics into
the same class. This is because EQUSS can effectively compress class-
irrelevant information and output features that are highly correlated with
classes. Please see more qualitative comparisons in Appendix.

Model | Unsupervised
---|---
Acc. | mIoU
IIC | 47.9 | 6.4
MDC | 40.7 | 7.1
PiCIE | 65.5 | 12.3
TransFGU (ViT-S) | 77.9 | 16.8
STEGO (ViT-B) | 73.2 | 21.0
EQUSS (ViT-B) | 79.9 | 22.0

(a) Cityscapes

Model | U.Acc.
---|---
Deep Cluster | 41.7
SIFT | 38.2
Doersch et al. | 49.6
Isola et al. | 63.9
IIC | 65.1
STEGO (ViT-B) | 77.0
EQUSS(ViT-B) | 82.0

(b) Potsdam-3

Table 2: Performance comparisons on (a) Cityscapes and (b) Potsdam-3.

##  5 Analysis

Figure 4:  Analyses on EQUSS. (a) Per-class entropy of EQUSS and STEGO
(Hamilton et al. 2022). (b) Frequency of each codeword to be selected, along
with its corresponding codebook. (c) Relationship between entropy and
accuracy. Yellow points with particularly low accuracy correspond to the most
infrequent minority classes (0.1% and 0.2%). (d) Inter-class distance between
the combination of codewords. The darker the color, the closer the distance.
Best viewed in PDF.

###  5.1 Relationship between Entropy and Accuracy

As discussed in Sec. 3.6, we quantify the information capacity of features in
terms of bits. Specifically, we compare the per-class entropy of EQUSS and
STEGO on the CocoStuff-27 (see Fig. 4(a)). Unlike EQUSS, STEGO outputs a
continuous variable, so we cannot directly measure the entropy using Eq. (7).
To deal with the problem, we discretize the continuous distribution by
histogram and calculate entropy for each feature dimension based on the
frequency (see Appendix).

The most noticeable observation is that the entropy of EQUSS is far smaller
than that of STEGO, 246 bits vs. 475 bits, on average. Intuitively, the number
of bits corresponds to the number of distinct representations for each class.
If there are few combinations of codewords for representing a certain class,
we can assume that such candidates contain common core information shared
among the samples in that class. Consequently, when the class is represented
with fewer candidates, the output features are also more likely to contain
class-relevant information, empirically supported by our superior USS
performance.

Interestingly, unlike STEGO, EQUSS shows different entropy values for
different classes. To understand the relationship between entropy and class,
we plot the frequency of each codeword (x-axis) being selected for each
codebook (y-axis) in Fig. 4(b). We observe that classes containing various
objects tend to have high entropy, while those with similar appearances have
low entropy. For example, in classes consisting of various objects such as
‘Sports’ (e.g., surfboard, kite, skis, $\ldots$) and ‘Indoor’ (e.g.,
toothbrush, vase, clock, $\ldots$), a variety of codewords are activated. In
contrast, classes that consistently appear in similar forms such as ‘Sky’ and
‘Water’ tend to concentrate on specific codewords that are assigned more
frequently than others. Our analysis verifies that EQUSS indeed reflects the
diversity of classes, which aligns with our common intuition.

We conjecture that the class with diverse appearance may confuse the model to
learn consistent representation, thus leading to poor performance. To
investigate this, we plot per-class accuracy and its corresponding entropy in
Fig. 4(c). Note that the classes are sorted in ascending order according to
their entropy values. As expected, the accuracy tends to decrease as the
number of bits needed to represent a class increases. From these results, we
establish the quantitative relationship between entropy and accuracy for the
first time.

Figure 5:  Qualitative comparison between the baseline (STEGO) and ours
(EQUSS) on CocoStuff-27.

###  5.2 Distances Between Classes After PQ

As illustrated in the blue box of Fig. 3, the combination of selected
codewords within the same class would exhibit a strong correlation, whereas
those between different classes would have a weak correlation. To investigate
such a correlation, we calculate the distance between codeword combinations of
two pixels. Specifically, we define a codeword combination of $q$ as a
sequence of selected codeword indices:

$\texttt{combination}(q)=[k^{1},k^{2},\ldots,k^{M}]$ (8)

where $k^{i}$ is the $\arg\min$ index of $i$-th subvector (see Eq. (2)). Then,
we define the distance between two combinations as the sum of the number of
different elements as below:

$\texttt{distance}(\texttt{c}(q_{1}),\texttt{c}(q_{2}))=\sum_{i=1}^{M}\mathbb{I}(k_{1}^{i}\neq
k_{2}^{i})$ (9)

where $\texttt{c}(q)$ is an abbreviation of $\texttt{combination}(q)$. Fig.
4(d) shows the average distance of codeword combinations between classes. Note
that we randomly select 10,000 pixels from every class for distance
computation. We observe that the diagonal of the matrix, corresponding to the
distance within the same class, shows much smaller values than other elements
of the matrix. This implies that our quantization head outputs similar results
for the same class, which enables clustering to become easier.

###  5.3 Ablation Study

#### Component Ablation

Table 3 shows the effect of feature dimension expansion and product
quantization. As discussed in Sec. 3.1, naively increasing the feature
dimension hurts the performance, specifically resulting in a consistent
decrease in linear probing accuracy and mIoU (see M1, M2, and M3). When PQ is
applied for compression, the model achieves higher performance in all cases
compared to not using PQ (see M2 vs. Q2, M3 vs. Q3). Furthermore, the
effectiveness of PQ increases as the feature dimension size increases (see Q2
$<$ Q3). By combining both feature dimension expansion and product
quantization, we can achieve the best result (see Q3).

|  Dim. |  Quant. |  |  |  |
---|---|---|---|---|---|---
Model | Unsupervised | Linear Probe
Acc | mIoU | Acc | mIoU
M1 | 70 |  | 48.3 | 24.5 | 74.4 | 38.3
M2 | 384 |  | 48.6 | 24.2 | 67.9 | 31.6
M3 | 1024 |  | 48.1 | 21.9 | 64.6 | 31.5
Q2 | 384 | ✓ | 49.7 | 25.1 | 72.4 | 36.7
Q3 (EQUSS) | 1024 | ✓ | 53.8 | 25.8 | 75.2 | 41.2

Table 3:  Ablation study on CocoStuff-27.

#### PQ Ablation

To study the effect of the number of codebooks (M) and the size of the
codebook (K), we conduct experiments while varying M and K with fixed feature
dimension. Fig. 6 presents unsupervised mIoU performance on Cityscapes. The
results show that the performance generally improves as M increases.
Specifically, M $>1$ outperforms M=1 in all cases. Note that M=1 is identical
to VQ. Furthermore, a larger codebook size (K) usually improves performance
since it expands the number of cases that a single codebook can represent.

However, excessively large values of M or K slightly reduce the performance.
When the fixed input dimension is divided into too many small subspaces (i.e.,
M=64), such small dimensions of each subvector would restrict their ability to
capture necessary representations. Additionally, for overly large K (i.e.,
K=128), each codeword may not receive sufficient updates due to the limited
number of training samples, leading to unstable performance. Nonetheless, in
most cases where $\textit{M}\in\\{8,16,32\\}$ and $\textit{K}\in\\{32,64\\}$,
the performance is consistently higher than our baseline.

Figure 6:  Unsupervised mIoU performance regarding the number and size of the
codebook on the Cityscapes. Note that the dimension of the feature is fixed to
512.

##  6 Conclusion

In this paper, we proposed EQUSS, a novel USS framework that takes both
advantages of high-dimensional space for improved clustering and product
quantization for effective information compression. From this design, we
achieved state-of-the-art performance on three USS benchmark datasets. For the
first time, we quantified the information capacity of USS features and
discovered meaningful analyses such as class-specific tendency and the
relationship between entropy and accuracy. We are delighted to emphasize that
our novelty involves new insights and directions within the field of USS,
contributing to a new information-theoretic understanding of USS.

## Appendix

### Loss Ablation

Table D shows the effect of each component on the overall loss function. We
observe that removing the loss for expansion head, $L_{head}$, results in
unstable performance. Since the expansion head generates the input of the
quantization head, removing this loss makes it difficult for the codebook to
contain the rich representation of codewords. On the other hand, $L_{commit}$
and $L_{codebook}$ improve the performance by +1.6 and +2.1 unsupervised
accuracy, respectively. Finally, CRF post-processing (Krähenbühl and Koltun
2011) slightly improves performance.

|  |  |  | Unsupervised
---|---|---|---|---
$\textit{L}_{head}$ | $\textit{L}_{commit}$ | $\textit{L}_{codebook}$ | CRF | Acc. | mIoU
| ✓ | ✓ |  | 16.8 | -
✓ |  | ✓ |  | 50.7 | 24.1
✓ | ✓ |  |  | 50.2 | 24.3
✓ | ✓ | ✓ |  | 52.3 | 25.3
✓ | ✓ | ✓ | ✓ | 53.8 | 25.8

Table D:  Ablation study on CocoStuff-27.

### Performance Considering Class Imbalance

Many popular USS datasets, such as CocoStuff-27 and Cityscapes, exhibit class
imbalance where the number of pixels per class varies significantly. This can
cause the model to be biased toward the majority classes and fail to learn the
characteristics of minority classes. In other words, the model may falsely
predict most of the pixels to majority classes, even if some of them belong to
the minority classes. However, the current unsupervised accuracy (Acc.) metric
is not suitable for capturing and handling this imbalance problem because Acc
only counts the number of the correct pixels without considering their
classes.

Model | Unsup. mAcc. | Linear. mAcc.
---|---|---
PiCIE | 25.1 | 27.0
TransFGU | 26.7 | 31.7
STEGO | 35.3 | 59.7
EQUSS (Ours) | 37.4 | 61.0

Table E: Mean class accuracy (mAcc) on CocoStuff-27. The scores for other
models are measured using the published pre-trained weights.  Figure G:
Class-wise accuracy of EQUSS (solid) and STEGO (dashed) on CocoStuff-27. The
classes are sorted in descending order based on their proportion (%), denoted
as numbers on top of each bar. The top 7 classes (majority, blue) take more
than 5%, while the bottom 13 classes (minority, red) occupy less than 2%.

To address this issue, we suggest measuring mean class accuracy (mAcc) for USS
metrics. The mAcc is the average between the pixel accuracy over all classes.
Compared to the conventional Acc metric, mAcc can be an effective metric for
representing the overall model performance. This is because mAcc can consider
all classes equally regardless of their class frequency. Table E compares
EQUSS to other recent USS frameworks. We observe that EQUSS surpasses previous
works by a large margin, specifically about +2.1 and +10.7 unsupervised mAcc
gap for the STEGO (Hamilton et al. 2022) and TransFGU (Yin et al. 2022),
respectively. Figure G shows the class-wise accuracy of EQUSS (solid line) and
STEGO (dashed line) in detail. We categorize classes into the majority classes
(blue) and minority classes (red) by frequency; each occupies more than 5% and
less than 2% of the total pixels in the training dataset, respectively. From
the results, we highlight that EQUSS achieves higher accuracy than STEGO for
most of the minority classes. We believe that expanding the dimension of the
feature vector enables fine-grained clustering, which can be helpful for
capturing the unique characteristics of minority classes.

## Additional Analysis

### Higher Information Compression Rate

Due to the novel architecture design, EQUSS can sufficiently compress
information while enjoying the benefits of high-dimensional features for
clustering. In STEGO, for example, a 384-dim feature vector is compressed into
the 70-dim vector and it requires 2,240 bits (32-bit $\times$ 70 elements) to
express the compressed feature. In contrast, our approach casts a feature
vector to high-dimensional space and then partitions the expanded feature
vector into multiple subvectors and quantizes each subvector. Specifically,
EQUSS expands the feature to a 1024-dim vector and partitions to 64 groups
(16-dim for each group) with 256 codewords, each being quantized to 8-bit. We
empirically show that 256 codewords (8-bit) are sufficient for improving
performance. After PQ, the final 1024-dim output only needs 512 bits (8-bit
$\times$ 64 groups) for the representation. In this example, EQUSS could use
approximately 15 ($\approx 1024/70$) times larger feature dimensions while
requiring only a quarter of bits, although the effective number of bits may be
smaller. This aligns with our observation in Fig. 4(a) that EQUSS uses much
fewer bits than STEGO to represent each class.

Figure H:  Histogram that shows the frequency of selected codewords in a
single codebook for top 6 majority classes on CocoStuff-27.

### Class-specific Codewords

We expect that certain codewords will show a strong correlation to specific
classes, as demonstrated in Fig. 4(b). To further investigate this class-
specific tendency, we visualize the frequency of each codeword in the first
codebook for the top-6 majority classes in CocoStuff-27. Fig. H shows the
histogram of each codeword frequency to be selected. For simplicity, we plot
the bar only when the frequency is greater than a given threshold. The results
suggest that certain codewords are particularly activated for specific
classes. In other words, these codewords can be considered unique for each
class without being occupied mutually. For example, the subvector of ground
class pixels is often quantized to codewords whose indices are 43, 220, and
25, whereas the subvector of plant class pixels is often quantized to codeword
number 109. We believe that such codewords can perform as class-specific
attributes 111This attribute is similar to the concept of generalized zero-
shot learning (Kim, Shim, and Shim 2022). Consequently, some classes have
their own representative codeword combination, and their variants do not
change much from it (see Fig. 4(d)).

Figure I:  The average distance of quantized vectors between two classes. Each
axis represents the class index. The brighter the color, the closer the
distance. Best viewed in PDF.

### Distances Between Quantized Vectors

We calculate the Euclidean distance between quantized vectors using the
Cocostuff-27. In Fig. I, we plot the average Euclidean distance between two
classes. We see that a large part of the diagonal component is conspicuous,
indicating high similarity among quantized vectors belonging to the same
class. In our work, instead of measuring the Euclidean distance between two
quantized vectors, we measured the similarity by comparing the number of non-
equal codeword indices in two quantized output vectors. Although the two
approaches seem to be different mathematically, the underlying principle of
both is more or less similar.

### Significance of Performance Improvement

Our major finding is that EQUSS shows excellent performance for both mIoU and
accuracy, but such is not the case for conventional techniques. For example,
while the accuracy difference between EQUSS and TransFGU is a bit marginal,
EQUSS significantly surpasses TransFGU in terms of mIoU (25.8 vs. 17.5, see
Tab. 1). Similarly, mIoU scores of EQUSS and STEGO seem to be comparable but
EQUSS outperforms SETGO in terms of accuracy by a large margin (53.8 vs. 48.3,
see Tab. 1). We believe that the performance gain of EQUSS is substantial and
meaningful.

## Measuring the Information Capacity of Continuous Feature Vector

As discussed in Sec. 3.6, it is straightforward to compute the amount of
information of features, $J(X)$, for EQUSS. However, unlike EQUSS, other USS
frameworks encode information in the continuous feature vector. Therefore, we
cannot directly apply Eq. (6) for calculating $p^{m}_{i}$ in the case of
STEGO. To alleviate this issue, we approximately obtain a discrete
distribution of values by histogram. Specifically, for $m$-th element of the
feature vector, we compute the frequency histogram $H^{m}$ and convert it to
the probability.

We first uniformly divide the min-max range into $B^{m}$ bins, where the edges
of bins are $\\{b^{m}_{0},b^{m}_{1},\ldots,b^{m}_{B^{m}}\\}$. Then, the
probability of each element $x^{m}$ to appear within the $i$-th bin is
calculated as:

$\hat{p}_{i}^{m}=\mathbb{E}_{X}[P(b^{m}_{i-1}<x^{m}\leq
b^{m}_{i})]\approx\frac{|H^{m}_{i}|}{|X|}$ (10)

where $|H^{m}_{i}|$ is the frequency count of histogram corresponds to $i$-th
bin. We can consider this as a uniform quantization from $x^{m}$ to
$(b^{m}_{i-1}+b^{m}_{i})/2$, a center point of $i$-th bin. In doing so, we can
formulate a similar equation to Eq. (7) with the approximated probabilities:

$\hat{J}{(X)}=\sum_{m=1}^{d_{R}}\bigg{(}-\sum_{i=1}^{B^{m}}\hat{p}_{i}^{m}\log_{2}(\hat{p}_{i}^{m})\bigg{)}.$
(11)

$d_{R}$ is a feature dimension after dimensionality reduction. Note that the
number of bins $B^{m}$ can vary for each element. To determine $B^{m}$, we
increase the number of bins while the condition is satisfied: the average
quantization error is less than a threshold $\delta^{m}$. We set $\delta^{m}$
to 0.1% of the total range ($\max(x^{m})-\min(x^{m})$). We empirically found
that a 0.1% error rate is sufficient to maintain the original performance of
STEGO.

## Detail

### Dataset

We evaluate our model on three standard datasets, CocoStuff-27 (Caesar,
Uijlings, and Ferrari 2018), Cityscapes (Cordts et al. 2016), and Potsdam-3.
CocoStuff is a large-scale scene-centric dataset with 80 things categories and
91 stuff categories. We evaluate the model on 27 mid-level classes of the
dataset hierarchy. Cityscapes contains a wide variety of urban street scenes
from 50 different cities in good weather conditions. We utilize 27 classes
excluding void group for the evaluation. Potsdam-3 is a collection of
satellite images obtained by extracting patches of $200\times 200$ from raw
images of high resolution. We evaluate our model on the 3-label variants
including roads, vegetation, and buildings.

Following prior art (Hamilton et al. 2022), we apply a five-crop
transformation on training samples to better handle small objects. We resize
and center-crop the training and validation samples to (224, 224) and (320,
320) resolutions, respectively. For evaluation, we measure mean intersection
over union (mIoU) and per-pixel accuracy.

| Cocostuff-27 |  | Cityscapes | Potsdam-3
---|---|---|---|---
Codebook configuration. |
#Codewords | 256 |  | 32 | 16
#Codebooks | 64 |  | 32 | 64
Codeword dim. | 16 |  | 32 | 16
Loss hyperparameters for $L_{\text{head}}$. |
$\lambda_{rand}$ | 0.63 |  | 0.95 | 0.63
$\lambda_{knn}$ | 0.25 |  | 0.43 | 0.25
$\lambda_{self}$ | 0.67 |  | 1 | 0.67
$b_{rand}$ | 0.66 |  | 0.31 | 0.26
$b_{knn}$ | 0.02 |  | 0.22 | 0.12
$b_{self}$ | 0.08 |  | 0.36 | 0.21

Table F:  Hyperparameters for different datasets.

### Evaluation Protocol

#### Linear Probing

We train a single linear projection layer for the supervised classification
task by utilizing ground truth labels. The main purpose of linear probing is
to evaluate how much class-relevant information is contained in the final
output features.

#### Unsupervised Clustering

Unlike linear probing, we do not access any labels and only cluster the output
features in an unsupervised manner. Then, we use a Hungarian matching
algorithm to compute the optimal assignment from our unlabeled clusters to
ground truth labels.

### Implementation Detail

We train our model with a batch size of 16 on a single NVIDIA A5000 24GB GPU.
Following previous work (Hamilton et al. 2022; Yin et al. 2022), we employ
DINO (Caron et al. 2021) pre-trained on ImageNet-22K (Deng et al. 2009) as a
backbone. For a fair comparison (Hamilton et al. 2022; Yin et al. 2022), we
keep the backbone frozen. We initialize each codebook in the quantization head
$\mathcal{Q}$ by Xavier uniform initialization (Glorot and Bengio 2010). The
expansion head $\mathcal{E}$ is a feed-forward network having a residual path
constructed by a two-layer MLP. For training, we employ Adam optimizer (Kingma
and Ba 2014) with a learning rate of 3e-4. For linear probing and unsupervised
clustering, we use Adam optimizer with a learning rate of 3e-3. We set
$\lambda_{1}=1.0$ and $\lambda_{2}=0.25$ in Eq. (5) for the experiments.

The output dimension of the expansion head is 1024, and the DINO (Caron et al.
2021) backbone feature dimensions of ViT-S and ViT-B architecture (Dosovitskiy
et al. 2020) are 384 and 768, respectively. The expansion head has two
branches, Conv-ReLU-Conv and Conv, that process the backbone feature
separately. The output of the expansion head is the addition of two outputs
from branches. Conv layers employ 1x1 kernel size with 1x1 stride. Feature
dimension expansion to the desired dimension occurs at the first convolutional
layer of each branch.

### Hyperparameter

Table F shows the hyperparameters and architecture (PQ) details used in our
experiments. Note that $\lambda$ and $b$ control $L_{head}$, which is
originated from STEGO (Hamilton et al. 2022).

## Additional Qualitative Result

Figure J shows the additional qualitative comparison between the baseline
(Hamilton et al. 2022) and ours (EQUSS).

## Limitation

Limitations of our methods include: (1) since our work is the first to focus
on the advantage of feature dimension expansion and quantization, we simply
implement grouping by dividing the features into the same channel size. Thus,
we believe that dynamically adjusting the number of channels for each group
can further improve performance and leave it as a future work; (2) currently,
the interpretability between the codewords and human perceivable attributes is
quite low. Thus, training the models to capture intuitively explainable human-
like attributes would be meaningful research.

Figure J:  Qualitative comparison between the baseline (STEGO) and ours
(EQUSS) on CocoStuff-27.

## Discussion

We would like to emphasize that our main contribution is to address the
fundamental weakness of bottleneck architecture in previous studies and
propose a novel framework improving both clustering and information
compression. Additionally, we highlight that our work is the first to analyze
USS from the entropy aspect. Specifically, we have investigated the
relationship between the USS performance and entropy and verified that the
entropy effectively reflects the diversity of classes (see Sec. 5.1). We
believe that our research can contribute to improving the understanding of USS
from an information-theoretic perspective.

## Acknowledgments

This work was supported by the National Research Foundation of Korea (NRF)
grant funded by the Korea government (MSIT) (2022M3C1A3099336) and National
Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT)
(RS-2023-00208985).

## References

  * Ahmed, Jalal, and Kim (2021) Ahmed, A.; Jalal, A.; and Kim, K. 2021.  Multi-objects detection and segmentation for scene understanding based on Texton forest and kernel sliding perceptron.  _Journal of Electrical Engineering & Technology_, 16: 1143–1150.
  * Asgari Taghanaki et al. (2021) Asgari Taghanaki, S.; Abhishek, K.; Cohen, J. P.; Cohen-Adad, J.; and Hamarneh, G. 2021.  Deep semantic segmentation of natural and medical images: a review.  _Artificial Intelligence Review_ , 54: 137–178.
  * Bengio, Léonard, and Courville (2013) Bengio, Y.; Léonard, N.; and Courville, A. 2013.  Estimating or propagating gradients through stochastic neurons for conditional computation.  _arXiv preprint arXiv:1308.3432_.
  * Caesar, Uijlings, and Ferrari (2018) Caesar, H.; Uijlings, J.; and Ferrari, V. 2018.  Coco-stuff: Thing and stuff classes in context.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1209–1218.
  * Caron et al. (2018) Caron, M.; Bojanowski, P.; Joulin, A.; and Douze, M. 2018.  Deep clustering for unsupervised learning of visual features.  In _Proceedings of the European conference on computer vision (ECCV)_ , 132–149.
  * Caron et al. (2021) Caron, M.; Touvron, H.; Misra, I.; Jégou, H.; Mairal, J.; Bojanowski, P.; and Joulin, A. 2021.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 9650–9660.
  * Chen, Watanabe, and Rudnicky (2023) Chen, L.-W.; Watanabe, S.; and Rudnicky, A. 2023.  A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech.  _arXiv preprint arXiv:2302.04215_.
  * Chen et al. (2020) Chen, X.; Fan, H.; Girshick, R.; and He, K. 2020.  Improved baselines with momentum contrastive learning.  _arXiv preprint arXiv:2003.04297_.
  * Cho et al. (2021) Cho, J. H.; Mall, U.; Bala, K.; and Hariharan, B. 2021.  Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 16794–16804.
  * Cordts et al. (2016) Cordts, M.; Omran, M.; Ramos, S.; Rehfeld, T.; Enzweiler, M.; Benenson, R.; Franke, U.; Roth, S.; and Schiele, B. 2016.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 3213–3223.
  * Cover (1965) Cover, T. M. 1965.  Geometrical and statistical properties of systems of linear inequalities with applications in pattern recognition.  _IEEE transactions on electronic computers_ , (3): 326–334.
  * Damerau (1964) Damerau, F. J. 1964.  A technique for computer detection and correction of spelling errors.  _Communications of the ACM_ , 7(3): 171–176.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Feng et al. (2020) Feng, D.; Haase-Schütz, C.; Rosenbaum, L.; Hertlein, H.; Glaeser, C.; Timm, F.; Wiesbeck, W.; and Dietmayer, K. 2020.  Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges.  _IEEE Transactions on Intelligent Transportation Systems_ , 22(3): 1341–1360.
  * Glorot and Bengio (2010) Glorot, X.; and Bengio, Y. 2010.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , 249–256. JMLR Workshop and Conference Proceedings.
  * Goldfeld and Polyanskiy (2020) Goldfeld, Z.; and Polyanskiy, Y. 2020.  The information bottleneck problem and its applications in machine learning.  _IEEE Journal on Selected Areas in Information Theory_ , 1(1): 19–38.
  * Guo et al. (2022) Guo, H.; Xie, F.; Wu, X.; Lu, H.; and Meng, H. 2022.  Towards High-Quality Neural TTS for Low-Resource Languages by Learning Compact Speech Representations.  _arXiv preprint arXiv:2210.15131_.
  * Hamilton et al. (2022) Hamilton, M.; Zhang, Z.; Hariharan, B.; Snavely, N.; and Freeman, W. T. 2022.  Unsupervised Semantic Segmentation by Distilling Feature Correspondences.  In _International Conference on Learning Representations_.
  * Hatamizadeh et al. (2022) Hatamizadeh, A.; Tang, Y.; Nath, V.; Yang, D.; Myronenko, A.; Landman, B.; Roth, H. R.; and Xu, D. 2022.  Unetr: Transformers for 3d medical image segmentation.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , 574–584.
  * He et al. (2015) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015.  Deep residual learning for image recognition.  _CoRR, abs/1512_ , 3385: 2.
  * Hofmann, Schölkopf, and Smola (2008) Hofmann, T.; Schölkopf, B.; and Smola, A. J. 2008.  Kernel methods in machine learning.
  * Hu et al. (2022) Hu, E. J.; yelong shen; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022.  LoRA: Low-Rank Adaptation of Large Language Models.  In _International Conference on Learning Representations_.
  * Huang et al. (2022) Huang, L.; Zhang, Z.; Du, Z.; Li, S.; Zheng, H.; Xie, Y.; and Tan, N. 2022.  EPQuant: A Graph Neural Network compression approach based on product quantization.  _Neurocomputing_ , 503: 49–61.
  * Isola et al. (2015) Isola, P.; Zoran, D.; Krishnan, D.; and Adelson, E. H. 2015.  Learning visual groups from co-occurrences in space and time.  _arXiv preprint arXiv:1511.06811_.
  * Jang and Cho (2021) Jang, Y. K.; and Cho, N. I. 2021.  Self-supervised product quantization for deep unsupervised image retrieval.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 12085–12094.
  * Jegou, Douze, and Schmid (2010) Jegou, H.; Douze, M.; and Schmid, C. 2010.  Product quantization for nearest neighbor search.  _IEEE transactions on pattern analysis and machine intelligence_ , 33(1): 117–128.
  * Jégou et al. (2010) Jégou, H.; Douze, M.; Schmid, C.; and Pérez, P. 2010.  Aggregating local descriptors into a compact image representation.  In _2010 IEEE computer society conference on computer vision and pattern recognition_ , 3304–3311. IEEE.
  * Ji, Henriques, and Vedaldi (2019) Ji, X.; Henriques, J. F.; and Vedaldi, A. 2019.  Invariant information clustering for unsupervised image classification and segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 9865–9874.
  * Kim, Shim, and Shim (2022) Kim, J.; Shim, K.; and Shim, B. 2022.  Semantic feature extraction for generalized zero-shot learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, 1166–1173.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kingma and Welling (2013) Kingma, D. P.; and Welling, M. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_.
  * Klein and Wolf (2019) Klein, B.; and Wolf, L. 2019.  End-to-end supervised product quantization for image search and retrieval.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5041–5050.
  * Krähenbühl and Koltun (2011) Krähenbühl, P.; and Koltun, V. 2011.  Efficient inference in fully connected crfs with gaussian edge potentials.  _Advances in neural information processing systems_ , 24.
  * Li et al. (2019) Li, J.; Lan, X.; Liu, Y.; Wang, L.; and Zheng, N. 2019.  Compressing unknown images with product quantizer for efficient zero-shot classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5463–5472.
  * Li et al. (2023) Li, K.; Wang, Z.; Cheng, Z.; Yu, R.; Zhao, Y.; Song, G.; Liu, C.; Yuan, L.; and Chen, J. 2023.  ACSeg: Adaptive Conceptualization for Unsupervised Semantic Segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7162–7172.
  * Lowe (1999) Lowe, D. G. 1999.  Object recognition from local scale-invariant features.  In _Proceedings of the seventh IEEE international conference on computer vision_ , volume 2, 1150–1157. Ieee.
  * Meletis (2022) Meletis, P. 2022.  Towards holistic scene understanding: Semantic segmentation and beyond.  _arXiv preprint arXiv:2201.07734_.
  * Ouali, Hudelot, and Tami (2020) Ouali, Y.; Hudelot, C.; and Tami, M. 2020.  Autoregressive unsupervised image segmentation.  In _European Conference on Computer Vision_ , 142–158. Springer.
  * Selvaraju et al. (2017) Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 618–626.
  * Seong et al. (2023) Seong, H. S.; Moon, W.; Lee, S.; and Heo, J.-P. 2023.  Leveraging Hidden Positives for Unsupervised Semantic Segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 19540–19549.
  * Shannon (1948) Shannon, C. E. 1948.  A mathematical theory of communication.  _The Bell system technical journal_ , 27(3): 379–423.
  * Siam et al. (2018) Siam, M.; Gamal, M.; Abdel-Razek, M.; Yogamani, S.; Jagersand, M.; and Zhang, H. 2018.  A comparative study of real-time semantic segmentation for autonomous driving.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 587–597.
  * Stock et al. (2021) Stock, P.; Fan, A.; Graham, B.; Grave, E.; Gribonval, R.; Jegou, H.; and Joulin, A. 2021.  Training with Quantization Noise for Extreme Model Compression.  In _International Conference on Learning Representations_.
  * Tishby, Pereira, and Bialek (2000) Tishby, N.; Pereira, F. C.; and Bialek, W. 2000.  The information bottleneck method.  _arXiv preprint physics/0004057_.
  * Yin et al. (2022) Yin, Z.; Wang, P.; Wang, F.; Xu, X.; Zhang, H.; Li, H.; and Jin, R. 2022.  TransFGU: a top-down approach to fine-grained unsupervised semantic segmentation.  In _European Conference on Computer Vision_ , 73–89. Springer.
  * Yu et al. (2020) Yu, K.; Ma, R.; Shi, K.; and Liu, Q. 2020.  Neural network language model compression with product quantization and soft binarization.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28: 2438–2449.
  * Yu et al. (2017) Yu, X.; Liu, T.; Wang, X.; and Tao, D. 2017.  On compressing deep models by low rank and sparse decomposition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 7370–7379.