# CSSL-MHTR: Continual Self-Supervised Learning for Scalable Multi-script
Handwritten Text Recognition

Marwa Dhiaf 1,3,*, $\dagger$ Mohamed Ali Souibgui 2,$\dagger$ Kai Wang 2
Yuyang Liu 4,5 Yousri Kessentini 3 Alicia Fornés 2 Ahmed Cheikh Rouhou 1
1 InstaDeep 2 Computer Vision Center, UAB, Spain
3 Digital Research Center of Sfax, SM@RTS, Tunisia 4 State Key Laboratory of
Robotics,
5 Shenyang Institute of Automation, Chinese Academy of Sciences, China

###### Abstract

Self-supervised learning has recently emerged as a strong alternative in
document analysis. These approaches are now capable of learning high-quality
image representations and overcoming the limitations of supervised methods,
which require a large amount of labeled data. However, these methods are
unable to capture new knowledge in an incremental fashion, i.e. when data is
presented to the model sequentially, which is closer to the realistic
scenario. In this paper, we explore the potential of continual self-supervised
learning to alleviate the catastrophic forgetting problem in handwritten text
recognition, as an example of sequence recognition. Our method consists in
adding intermediate layers called adapters for each task, and efficiently
distilling knowledge from the previous model while learning the current task.
Our proposed framework is efficient in both computation and memory complexity.
To demonstrate its effectiveness, we evaluate our method by transferring the
learned model to diverse text recognition downstream tasks, including Latin
and non-Latin scripts. As far as we know, this is the first application of
continual self-supervised learning for handwritten text recognition. We attain
state-of-the-art performance in English, Italian, and Russian, whilst adding
only a few parameters per task. The code and trained models will be publicly
available111Link available upon acceptance..

##  1 Introduction

††footnotetext: *Work done during an internship at CVC.††footnotetext:
$\dagger$Equal contribution.

Handwritten Text Recognition (HTR) has attracted the interest of the computer
vision and pattern recognition research community for decades [11, 4, 53, 29].
Recent HTR approaches tend to use data-hungry supervised deep learning models,
which require a vast amount of annotated data, especially with the recent
transformer-based methods [28, 34]. To overcome the data scarcity problem,
Self-Supervised Learning (SSL) techniques are nowadays being widely used for
computer vision tasks [9, 10, 16, 21, 23, 24]), and more precisely for text
recognition [2, 52].

Figure 1: The model continually learns representation from sequentially
appearing unlabeled data. The learned representation can be used to fine-tune
HTR models on all the previously learned scripts/languages.

A common assumption in the recent work on self-supervised learning is that all
the training data is available at the same time during the training process.
However, in many real-world scenarios, new tasks and data continuously emerge
over time. Thus, the model should be able to progressively and continuously
learn new tasks without forgetting the previous ones [36], and also, whenever
new unlabeled data is available. In fact, and in order to handle new data, the
model must be retrained on the full dataset, i.e. previous and current data,
to prevent the catastrophic forgetting phenomenon [42] in shifted data
distributions. But, retraining the model with the full dataset is impractical,
costly, and even impossible when previous data is not available anymore.
Indeed, in some realistic scenarios, where privacy and security are
fundamental, the model is not allowed to store the full previous data. More
explicitly, a time frame of data may disappear due to storage constraints or
privacy issues, which requires a dynamic training process to begin upon
receiving the new data. This issue encourages researchers to develop deep
learning models that are able to adapt quickly and re-learn over time.
Inspired by human cognition and its capability to learn incrementally and
sequentially, Continual learning (CL) investigates the ability of neural
networks to continuously learn tasks in a sequential manner, either on
supervised [45, 46], or unsupervised fashion [18, 35, 40, 50], without the
catastrophic forgetting problem. One characteristic of such systems is that
old ideas may be re-examined although there is no need to hold them in mind
[45].

Recent works [20, 8, 19] however, focus on alleviating the problem of
catastrophic forgetting in a self-supervised fashion. The aim is to learn from
a continuous data stream that follows a shifted distribution of visual
representations. This is useful when dealing with unsupervised sequential
representations, as we are no longer bound by the cost of manual labeling
data, which is particularly hard to obtain. This practice is called Continual
Self-Supervised Leaning (CSSL). For handwritten text recognition, we consider
continuously learning new scripts or languages, as illustrated in Figure. 1.
Our self-supervised continual learning decouples into two stages: (1) A
continual learning strategy of an encoder-decoder in a self-supervised
fashion. (2) A fine-tuning stage on top of the encoder for the multi-script
handwriting recognition tasks (downstream tasks). We state that using a
representation learning stream in the continual learning paradigm is
particularly advantageous for practical use cases. Since the encoder learning
stage carries the majority of the computation complexity, earlier tasks can
indeed be trained via the shallow output layer. Therefore, this gives the
additional flexibility to add an indefinite number of classes by fine-tuning
the SSL pre-trained backbone for several downstream tasks, which is useful in
practical real word scenarios. Furthermore, training an encoder with unlabeled
data is beneficial for generalization performance when dealing with data
scarcity, as our experiments demonstrate. For unlabeled text images, [52]
showed that a masking/recovery strategy is more suitable for SSL than other
related approaches. Thus, this work is based on Masked Auto-Encoders (MAE)
[23] for pre-training. Specifically, MAE discards low-level information by
masking a large portion of the image patches and enables the encoder to
extract semantic information by reconstructing the pixels from a very small
number of neighboring patches [6] with a lightweight decoder. However, the MAE
original design does not take continual learning into consideration and thus
can not generalize well both in the previous and current tasks.

With the aim to incorporate representation learning within a continual
learning paradigm, we retrain our model on a new language representation by
utilizing knowledge from an existing representation model. Our model can
retain performance on past tasks while transferring previous knowledge to be
more computationally efficient when training the current task. For this
purpose, we add language/script adapters [25] as new parameters for learning
new languages/scripts. The power of our method is that it can prevent
forgetting completely and speed up the training of a new task by reusing model
components of a past task.

Other continual strategies involve explicit ways to combat catastrophic
forgetting. Some of them proposed the use of a buffer memory [3] with
compressed representations of samples of past tasks. These representations are
then presented to the model while the model is trained with the current task
data. In this manner, the number of images evolves by augmenting the number of
tasks over time. In this work, we alternatively propose a more challenging
strategy when saving images from the previous tasks. Since we use a
masking/recovering pre-training, we do not need to save full images. In fact,
only a few patches of the images are required for the memory replay, since our
model can reconstruct the full images from only the patches. This is a clear
benefit in real scenarios where the storage memory is very limited and also,
where data privacy and security are crucial. Moreover, by using a
language/script adapter combined with the memory replay strategy, no prior
information is required at inference time, which makes our model a task-
agnostic continual learning model. Overall, the main contributions of this
work are the following:

  * •

To the best of our knowledge, our work is the first application of continual
learning to the field of handwritten text recognition, showing its ability to
continuously learn from new scripts or languages.

  * •

An efficient continual self-supervised learning framework in terms of
complexity and storing memory is proposed to address the issue of catastrophic
forgetting with no need for prior knowledge at inference time. We show that
adopting a scalable continual representation learning for HTR is beneficial in
real word scenarios, where the model is mainly trained on unlabeled data, and
only a few labeled data are made available for fine-tuning.

  * •

Extensive experiments show the effectiveness of our approach compared to the
state-of-the-art methods on continual learning in the context of multi-
script/multi-lingual handwritten text recognition.

##  2 Related Work

###  2.1 Continual Learning

Continual learning [48, 55] aims to continuously update the mastered knowledge
based on prior information while preserving performance across all tasks. One
of the major challenges is catastrophic forgetting [42], since relying solely
on new data to update the model leads to the continuous erosion of knowledge
from previous tasks. Recently, some regularization-based methods in continuous
Sequence-to-Sequence models are mainly developed from the perspective of
parameter update [32] and distillation loss [43]. However, these methods
usually suffer from the problem of plasticity, because they constrain the
required updates of parameters specific to different tasks. Moreover,
rehearsal-based methods [33, 54] utilize a small subset of text data or
intermediate features to effectively mitigate forgetting while allowing their
adaptation to new tasks, with encouraging performance in various tasks. The
fixed-size buffer limits the ability of the model to adequately capture the
diversity of the previous tasks. Therefore, the rehearsal-based continual
method faces the following two core challenges: 1) determining how to
effectively select a critical subset of old data for replay from a large
amount of available data, and 2) figuring out how to extract accurate and
effective representations for the rehearsal data within the constraints of
limited memory.

Continual learning in the field of HTR can be seen as performing continual
learning in sequential data. To avoid the rehearsal-based strategies problems
in sequence data, some approaches address the continual learning problem by
reintroducing a modular approach where new parameters are added as language
adapters for learning new languages [25, 30]. Close to our problem, recent
works [37, 39, 20, 8, 19] have attempted to address CSSL. However, these
approaches were designed for isolated classes (not sequences) using
contrastive or barlow-twins losses. Contrary, [52] showed that the use of a
masking/recovery strategy is better for performing SSL on handwritten text.
For this reason, in this work we propose a CSSL approach based on language
adapters (which was shown to work better for sequential data) and knowledge
distillation (with an efficient design that is compatible with the
masking/recovery strategy to overcome the limited memory size by rehearsal-
methods).

###  2.2  Handwritten Text Recognition

Handwritten Text Recognition aims to transform scanned/digitized handwritten
documents into machine-encoded text. HTR is still a hard problem due to the
high variability of writing styles, and often poor image quality
(degradation). Most HTR approaches are fully supervised methods based on
encoder-decoder models [13, 51]. Some recent approaches are based on semi-
supervised and self-supervised learning [53, 58, 52, 2]. Regarding the
unsupervised learning methods, [22] proposes a system for text recognition
that aligns the conditional distribution of the predicted strings with lexical
correct strings sampled from a text database. Concerning self-supervised
learning-based approaches, [2] introduced a self-supervised sequence-to-
sequence model that separates consecutive text features to be later used in a
contrastive loss [9]. By concatenating characters and by detecting spatial
strokes, [57, 38] focused on improving the features resulting from a
contrastive loss. However, these approaches rely on large batches and a
sequential definition of features, which can result in misaligned characters
or n-grams seen in different words. In [52], a degradation-based auto-encoder
was proposed to overcome the issues of contrastive-based methods and to learn
useful representation from text image restoration.

In addition, most existing HTR methods address a single language/script or
domain. Only few attempts address the multi-domain [58] and multi-
script/language [27, 12, 26]. However, these latter models consist of two
stages: 1) identifying the script/language and 2) training task-specific HTR
models. Consequently, it is not efficient for learning consecutive tasks,
because for each new task, the script/language classifier needs to be
retrained, and also, a new HTR model must be trained for the new task.
Moreover, these are supervised HTR/OCR models. Although this limitation is not
present in existing self-supervised methods, such methods are typically
designed for a specific scenario and cannot efficiently generalize to
different and continual tasks. Therefore, to overcome these limitations, an
efficient and scalable continual learning approach is proposed for the CSSL
multi-script HTR problem.

##  3 Methodology

Figure 2: The continual self-supervised learning module: our method can learn
different scripts/languages in a self-supervised way. At each time step $t$,
only the task-specific adapter is learned and the rest of the model is frozen
to overcome the catastrophic forgetting. Also, a set of visual patches from
the data is stored in a memory buffer and the visible patches from previous
tasks are loaded and replayed into the current model for a task-agnostic
inference.

Our continual self-supervised learning based approach for handwritten text
recognition comprises two phases: pre-training and fine-tuning. In this
section, we introduce the proposed building blocks in both phases.

###  3.1 Pre-training Module

The overall pre-training pipeline is depicted in Figure 2. In this work, the
aim is to learn multiple tasks sequentially, where each task consists in
recognizing a particular handwritten script or language. The purpose of each
task is to learn useful representations from its unlabeled data that will be
later used by the HTR model at the fine-tuning stage as the optimal starting
point for learning specific or multiple scripts/languages. In this self-
supervised learning strategy, the model should not forget the previous tasks
when learning a new task. Hence, a continual learning strategy is proposed to
achieve this goal. In what follows, we present the techniques used for both
learning strategies. Then, we detail the components used within our proposed
architecture.

Self-supervised pre-training. As we said before, the goal of the SSL pre-
training is to learn a useful representation $z$ from each unlabeled image
belonging to the pre-training data. To learn this representation, we opt to
use a masking/recovering strategy, as it showed success in handwritten text
images compared to related approaches [52]. As it can be seen from Figure 2,
for every new task with timestep $t$, the unlabeled image is first divided
into a set of patches. Then, a random portion of these patches denoted by
$P_{m}$ is masked to generate a masked image. After that, an auto-encoder
model learns the recovery process by producing an unmasked image. The recovery
is done by predicting the masked patches $P_{m_{pred}}$ based only on the
visible patches $P_{v}$. The auto-encoder model is composed of a ViT encoder
$\mathcal{E}$ that encodes $P_{v}$ into a latent representation $z$, and a
transformer decoder $\mathcal{D}$ that maps $z$ to the predicted patches
$P_{m_{pred}}$. This makes the masking/recovering training process as follows:

$\begin{split}z&=\mathcal{E}(P_{v};\theta_{\mathcal{E}})\\\
P_{m_{pred}}&=\mathcal{D}(z;\theta_{\mathcal{D}})\end{split}$ (1)

Continual Learning Module. As illustrated in Figure 2, the different tasks are
learned sequentially, where at each time-step $t$, a new script or language is
processed. To avoid the catastrophic forgetting of previous tasks, two
techniques are used: a task-specific adapter and a memory buffer.

In a continual learning scenario, the goal is to learn new knowledge for the
new task while preserving the previous knowledge of the model. To accomplish
this, in a time step $t$, a task-specific sub-model called adapter is added to
the architecture. This adapter is the only component that is trained at $t$,
while freezing all the components that were trained in the time-step $t-1$. By
doing this, maximum preservation of the previous knowledge is maintained
(frozen) and also, the learning speed is increased since only the adapter $t$
is trained. The adapter module is added to all the encoder and decoder layers.
In each layer, the adapter modules are placed after the multi-head attention
and feed-forward components. Thus, they shall adapt the network to the new
script/language domain. It is to note that in the first learned task, the
adapter component is set to an identity mapping since there is no previous
knowledge to preserve.

By using the adapters, the model is able to adapt to new script/language
domains. But sequentially plugging the task $t$ adapter can affect the model
performance on the previous tasks. So, to avoid this and improve the
performance of a current adapter on the previous tasks, a memory replay
strategy is used. The idea is to create a memory buffer and store a few
samples of the data belonging to the different learned tasks. At time-step
$t$, the stored samples of the previous tasks $\\{t-1,t-2,...,1\\}$ are
replayed to the model $t$ and its parameters are tuned to have a similar
response to the model $t-1$ for this stored data. This is done by employing a
knowledge distillation L2 loss between the output of the frozen previous model
and the current learning model. An interesting benefit of using a masked auto-
encoder for the SSL task is that we are not obliged to store full images.
Therefore, we propose in this work to store only a few patches of the
unlabeled images. By using those patches and the $t-1$ model, we can produce
the restored image to be considered as the ground-truth for the current model.
Hence, we propose an interesting solution to the problem of memory exploding
when using memory buffers in continual learning.

Next, we detail the components of our method.

###  3.2 Pre-training Module Components

The proposed model is composed of an encoder-decoder architecture with task-
specific adapter components.
Encoder. Our used encoder is vanilla ViT [17] backbone. Given an input image
$I$, it is first split into a set of $N$ patches,
$I^{p}=\\{I^{p_{1}},I^{p_{2}},...,I^{p_{N}}\\}$. Then, a set of these patches
are masked and the remaining visible patches are embedded with a trainable
linear projection layer $E$ to obtain tokens. These tokens are later
concatenated with their 2-D positional information embedded with $E_{pos}$ and
fed to $L$ transformer blocks to map these tokens to the encoded latent
representation $z_{l}$. These blocks are composed of $L$ layers of Multi-head
Self-Attention (MSA) and a feedforward Multi-Layered Perceptron (MLP) as
depicted in Figure 2. Each of these blocks are preceded by a LayerNorm (LN)
and followed by a residual connection and a set of adapter components. The
number of adapters depends on the number of previously learned tasks. In every
learned task $t$, two adapters ${A_{E}^{t}}_{1}$ and ${A_{E}^{t}}_{2}$ and are
added after the MHA and the MLP, respectively. The aforementioned statement is
exemplified in Eqn. 2

$\begin{split}z_{0}&=E(I^{p})+E_{pos}\\\
{z^{\prime}}_{l}&=\text{MSA}(\text{LN}(z_{l-1}))+z_{l-1}\text{, }l=1,\text{
\ldots L}\\\ {z^{\prime\prime}}_{l}&={A_{E}^{t}}_{1}({A_{E}^{t-1}}_{1}(\text{
\ldots}({A_{E}^{1}}_{1}({z^{\prime}}_{l}))))\\\
{z}_{l}&=\text{MLP}(\text{LN}({z^{\prime\prime}}_{l}))+{z^{\prime\prime}}_{l}\text{,
}l=1,\text{ \ldots L}\\\
z_{\mathcal{T}}&={A_{E}^{t}}_{2}({A_{E}^{t-1}}_{2}(\text{
\ldots}({A_{E}^{1}}_{2}({z}_{L}))))\end{split}$ (2)

Decoder. The decoder is a transformer reconstruction module. As in MAE [23],
the encoded tokens $z_{\mathcal{T}}$ (Eqn. 2) are first concatenated with a
set of mask tokens that indicated the presence of the missing patches that
should be predicted. After that, positional embedding is added to the
concatenated tokens and a set of transformer blocks is used to decode these
tokens into patches of pixels. The output of the decoder is a set of vectors
$I_{r}=\\{I_{r}^{p_{1}},I_{r}^{p_{2}}$, …, $I_{r}^{p_{N}}\\}$ where each of
which corresponds to a flattened patch in the predicted (reconstructed)
patches that were initially masked. As in the encoder, the MSA and MLP layers
are used with the task-specific adapters ${A_{D}^{t}}_{1}$ and
${A_{D}^{t}}_{2}$.

$\begin{split}{z^{\prime}}_{l}&=\text{MSA}(\text{LN}(z_{l-1}))+z_{l-1}\text{ ,
}l=1,\text{ \ldots L}\\\
{z^{\prime\prime}}_{l}&={A_{D}^{t}}_{1}({A_{D}^{t-1}}_{1}(\text{
\ldots}({A_{D}^{1}}_{1}({z^{\prime}}_{l}))))\\\
{z}_{l}&=\text{MLP}(\text{LN}({z^{\prime}}_{l})))+{z^{\prime}}_{l}\text{ ,
}l=1,\text{ \ldots L}\\\
I_{r}&=\text{Linear}({A_{D}^{t}}_{2}({A_{D}^{t-1}}_{2}(\text{
\ldots}({A_{D}^{1}}_{2}({z}_{L})))))\end{split}$ (3)

Adapter. The adapter components are designed to efficiently adapt a model for
a new script/language. The idea is to embed them after each MHA and
feedforward layer. During pre-training, only the adapters of the current task
are trained, while freezing the rest of the model. Each adapter is composed of
two Linear layers, where the first Linear layer is preceded by a LayerNorm and
followed by a ReLu activation, as illustrated in Figure 2.

###  3.3 Pre-training Learning Objectives

During the pre-training, at time-step $t$=$1$ (i.e. learning the first
language/script), a reconstruction l2 loss $\mathcal{L}_{r}$ is used between
the patches that were masked $P_{m}$ and the predicted ones $P_{m_{pred}}$ to
make the learning objective as $\mathcal{L}_{r}(P_{m},P_{m_{pred}})$. At
$t>1$, an additional distillation l2 loss $\mathcal{L}_{d}$ is used between
the output of the current model and the output of the previous one ($t$-$1$)
on the data loaded from the memory buffer. This makes the full pre-training
learning objective as:

$\mathcal{L}_{p}=\mathcal{L}_{r}(P_{m},P_{m_{pred}})+\mathcal{L}_{d}(P_{m_{pred}},P_{m_{pred}^{t-1}})$
(4)

###  3.4 Fine-tuning Module

The downstream task for fine-tuning is handwritten text recognition, as an
example of sequence recognition. HTR aims to transform a handwritten text
image from a specific language/script into a machine-encoded form, i.e., a
sequence of characters. Let $I$ be a text image and
$C=\\{c_{{}_{1}},c_{{}_{2}},...,c_{{}_{N}}\\}$ its ground truth label which
corresponds to a sequence of characters, where $N$ is the length of the text.
The training is done by passing $I$ to an encoder function $\mathcal{E}$ to
produce a latent representation $z$. Then, $z$ is later fed to a decoder
function $\mathcal{D^{\prime}}$ multiple times to produce a sequence of
characters $C_{p}=\\{c_{p_{1}},c_{p_{2}},...,c_{p_{M}}\\}$ that should match
the ground truth label sequence. In our approach, the fine-tuning is done by
initializing $\mathcal{E}$ with the learned parameters during pre-training. As
shown in Figure 1, a pre-trained encoder can be used for multi-script fine-
tuning, as the $t^{th}$ encoder can be used to train a model that recognizes
any of the learned scripts/languages during the tasks $t$-1, $t$-2, …, 1.

##  4 Experiments and Results

In this section we evaluate our method’s performance and compare it with some
baselines and state-of-the-art continual learning approaches. In addition, we
compare the obtained HTR accuracy when performing the fine-tuning of multi-
scripts/languages.

Method | FT after Task 2 | FT after Task 3 | # Tr. parms. | # Samples $\uparrow$
---|---|---|---|---
English $\downarrow$ | Italian $\downarrow$ | English $\downarrow$ | Italian $\downarrow$ | Russian $\downarrow$
Multilingual | 12.3 | 17.0 | 12.3 | 17.0 | 4.7 | 68 M | –
Monolingual | 8.0 | 6.0 | 8.0 | 6.0 | 1.4 | 68 M | –
Adapter | 10.6 | 5.7 | 6.1 | 5.4 | 2.8 | 10.2 M | –
Distillation | 9.4 | 9.1 | 8.1 | 9.5 | 6.6 | 68 M | 3200
ER [49] | 19.1 | 5.8 | 14.0 | 22.0 | 5.9 | 68 M | 1920
EWC [31] | 27.0 | 8.6 | 25.9 | 10.0 | 5.7 | 68 M | –
CSSL-MHTR | 5.7 | 5.1 | 4.9 | 5.1 | 2.9 |  10.2 M | 3200
Table 1: Continual representation learning: evaluation of continually pre-
trained models on three tasks (English followed by Italian, followed by
Russian). Then, they are fine-tuned (FT) separately for recognizing English,
Italian, and Russian, which correspond to the downstream HTR tasks, measured
using the Character Error Rate (the lower, the better). Fine-tuning is done
after learning the second task (FT after task 2) or the third task (FT after
task 3). # Tr. parms. is the number of trainable parameters in million. #
Samples indicated the possible stored samples using a fixed-size memory buffer
for the memory-replay-based methods.

###  4.1 Datasets, Tasks and Metrics

We assess the performance of our CSSL framework by sequentially learning three
tasks without forgetting. Each task consists in learning the representation of
a different language/script. For this, we use specific HTR datasets, as
illustrated in Figure 3. The tasks are the following:

English: The first task consists in recognizing English. For this task, the
IAM dataset [41] was used.

Italian: The second task consists in learning another language, concretely
Italian language. Here we maintain the same script as English, but we have a
different domain (i.e. lexicon, language model, probability distribution). For
this task, the LAM dataset [7] was used.

Russian: The third task consists in learning a different language and script,
concretely, the Russian language. This task is the most challenging because it
must not only learn a new language, but also a new alphabet (a different set
of character classes). The HKR [44] was used for this task.

For the evaluation, we use the Character Error Rate (CER) between the produced
text output and the ground truth. More details on the datasets (splits and
size) and the CER metric can be found in the supplementary material.

---
Figure 3: Three examples of the images from the different languages/scripts
datasets that were used for the experiments. Up: IAM, middle: LAM and Down:
HKR. It is noticeable the domain difference between the different tasks.

###  4.2 Continual Learning representations

####  4.2.1 Baselines

The majority of CL approaches require the use of labels, which makes them
inappropriate for CSSL. However, some approaches can be evaluated within our
context by slightly modifying them. In order to demonstrate the effectiveness
of our proposed framework, we compare it to four continual learning methods
consisting of the following: (i) Adapter: language/script adapters only are
inserted after MHSA and feedforward module in all layers of the transformer
architecture. (ii) Distillation: a representation objective is optimized with
a knowledge distillation loss as described in section 3.3; in this method, we
store only some patches of the images (iii) Experience Replay (ER) [49]:
rehearsal-based replay method benefits from storing data (full images), which
contains a lot of information about previous tasks and enables the retraining
on old evaluationclasses. (iv) Elastic Weight Consolidation (EWC): We use the
regularization method from [31] to estimate the diagonal of the Fisher
information matrix that is used to count the parameter’s importance after
every learned task. Details on baseline selection, implementation and tuning
for CSSL can be found in the supplementary material.

For the comparisons, we pre-train the model across languages in a multi-script
fashion and then fine-tune on the respective scripts (Multilingual). Also, we
perform a monolingual pre-training, and we fine-tune on separate scripts.

####  4.2.2 Comparison with Continual Learning Methods

We begin by evaluating the quality of the learned representation within our
continual learning scenario. This is reported in Table 1, where we compare our
method with the baselines and continual learning widely used methods presented
above. The experiment consists in performing self-supervised pre-training to
learn representations from different languages/scripts in a sequential manner.
We begin by pre-training on English (task 1), then we continually pre-train on
Italian (task 2), and then Russian (HKR) (task 3). The goal is, of course, to
not forget the representation of the previously learned tasks while learning
the current task. We evaluate at which extend the model prevents catastrophic
forgetting by taking the encoder of the current task and fine-tuning it on the
current and previously learned tasks. The best model should provide the best
results in terms of Character Error Rate (CER) on the current as well as the
previously learned languages/scripts. Moreover, we evaluate the number of
trainable parameters in each model to highlight the most efficient model in
terms of training time. Also, we show the number of possible stored samples in
the methods that use a memory buffer by fixing its size.

Figure 4: The obtained CER of the English and Italian language using the fine-
tuned models after continually pre-training on the second (Italian) and third
(Russian) tasks.

From Table 1, we notice that the replay-based approach (ER) does not help in
the prevention of forgetting in CSSL. We have identified two prime reasons for
this failure. First, in a supervised CL, replay-based methods benefit from the
storage of labels, which contain some useful information about previous tasks
and allow the model to be retrained on old classes. This is not the case in
CSSL, because labels are not available. Secondly, SSL models need more
training epochs to converge, which means that the samples in the buffer are
also replayed many more times. This leads to significant over-fitting on these
examples and defeats consequently the purpose of the replay buffer. We also
observe that ER stores fewer samples than our method because it requires
storing the full images, while our approach only stores a percentage of
patches from the full image. This explains why the results shown in the
Distillation row are better than ER, since it stores patches instead of full
images.

From the Table, we also observe that the EWC method yields unstable learning
representation and does not prevent forgetting when fine-tuning the model.
This means that the importance of the weights could not be accurately
calculated with the Self-supervised task, i.e. the image recovery. Next, we
can see that the performance of the Adapter is better than the other
approaches. With the adapter, maximum prevention of forgetting is maintained
since the rest of the model components are frozen. This also brings efficient
training since the number of trainable parameters becomes significantly lower
than in the other approaches. However, the adaptation of the current adapter
to the previous task is insufficient, which results in unsatisfactory
performance.

Moreover, we compared the results with monolingual and multilingual pre-
training methods. Although these are not continual learning methods, they can
serve as references to evaluate the different models. In the monolingual
method, the model is pre-trained and fine-tuned on only one script/language.
This should serve as an upper bound because the model only focuses on one
task. In the multilingual pre-training model, all the datasets from the three
different tasks are used to train the model simultaneously. From the results,
we observe that the monolingual pre-train obtains good results, while the
multilingual model achieves poor results. These poor results are caused by the
domain gap between the different languages and the class gap between the
different scripts.

Finally, it must be noted that our CSSL-MHTR model obtains the best results.
Our framework demonstrates efficient forward transfer to downstream tasks,
proving that its features are easier to generalize to other scripts or
languages. This fact is demonstrated by achieving the best CER on both
languages (English and Italian) after learning the second task and by
achieving the best CER in English and Italian after learning the third task
(Russian) as well. Furthermore, our model can prevent the catastrophic
forgetting regarding the fine-tuning stage in all the script/languages. This
is depicted in Figure 4. As it can be seen, our model is stable in the CER of
the different languages after learning the three tasks. Here we observe that
the CER of English and Italian is not only the best among all the compared
approaches, but it also improves after learning more tasks (e.g. the CER for
English is 5.7% after learning Italian, but it decreases to 4.9% after
learning Russian). This can be explained by the assumption that when learning
more scripts/languages in a self-supervised way, the representations are
getting richer and more robust because of the newly learned domains (writing
styles, fonts, backgrounds, etc.). From the Figure, we notice that this gain
in representation quality in both languages is only available when using the
adapter-based approaches, which confirms its utility. In fact, the use of
language adapters makes our model efficient in avoiding the forgetting
problem, and also, it speeds up the training phase because it has fewer
trainable parameters. Moreover, by storing only some patches of the previous
tasks’ images (the full image is not needed) we can have a larger number of
stored samples compared to the ER method. This leads to better performance
when applying knowledge distillation with our approach to adapt the current
model to the previous tasks. In addition, this allows to reuse data from
datasets that are not available anymore or have privacy constraints.

To summarize, with CSSL-MHTR, we can easily transfer knowledge from the
original script (in our experiments, English), to learn another
script/language in a sequential and continuous fashion. Also, we decrease the
time and storage memory that is required to train a new script/language in a
continual learning fashion. The reason is the use of a simple language adapter
with frozen parameters to learn new tasks efficiently, as well as the use of
knowledge distillation to adapt the current model to the previous tasks
(preventing the forgetting issue). Also, the use of the masking/recovering
pre-training strategy enables the model to store only a few patches instead of
the full images that belongs to the previous tasks. This confirms our design
choice for the addressed problem within this paper.

###  4.3 Comparison with HTR Models

After demonstrating the suitability of our proposed approach for learning the
different languages/scripts without forgetting, now we compare our model with
state-of-the-art handwritten text recognition systems on the three
scripts/languages. The results obtained by the models included in the analysis
on the basic split are reported in Table 2. In the first column, we consider
models following different kinds of architectures, including Convolutional-
Recurrent Neural Network Models [47, 14], Transformer Networks such as [28,
34] and Fully-Convolutional Networks [1, 5, 15, 56]. Our CSSL-MHTR holds up on
all datasets after pre-training on other scripts and obtains almost the same
performance when compared to state-of-the-art approaches that use one script
for monolingual training. Nevertheless, we observe a significant margin
between different scripts, due to the writing style, number of characters per
line and their average width, the variation between Latin and non-Latin
scripts, the background of the image, etc. Anyway, our CSSL-MHTR exhibits more
robustness and generalization ability when compared to other systems that were
tuned to obtain the optimal performance on a particular script/language.

System | English | Italian | Russian
---|---|---|---
1D-LSTM[47] | 8.3 | 3.7 | 69.1
CRNN [14] | 6.8 | 3.3 | -
Transformer [28] | 4.6 | 10.2 | -
TrOCR [34] | 3.4 | 3.6 | -
GFCN [15] | 8.0 | 5.2 | -
OrigamiNet [56] | 4.8 | 3.1 | -
Bluche [5] | 3.2 |  | 22.3
G-CNN-BGRU [1] | - | - | 8.3
CSSL-MHTR | 4.9 | 5.1 | 2.9
Table 2: Comparison with state-of-the-art approaches for HTR on multi-
scripts/languages.

###  4.4 Qualitative Evaluation

Input: |
---|---
GT: | was over. ” Again her laughter trilled. ” Marriage
Monolingual: | was over ” Afein ler layghted tiibled. ” Moxoyage
CSSL-MHTR (t=2): | was over. ” Again her laughter tiilled. ” Marriage
CSSL-MHTR (t=3): | was over. ” Again her laughter tribled. ” Marriage
Input: |
GT: | ” Isn’t it ? ” he rose and smoothed himself
Monolingual: | ” tsa’t it ” ” he rose and suoothed himself
CSSL-MHTR (t=2): | ” Isn’t it ’ ” he rose and smoothed himself
CSSL-MHTR (t=3): | ” Isn’t it ? ” he rose and smoothed himself

Figure 5: The output of the different models when recognizing handwritten
English text images. The produced errors of each model are shown in red color.

GT (English): |
---|---
Masked (t=1): |
Recov. (t=1): |
Masked (t=2): |
Recov. (t=2): |
Masked (t=3): |
Recov. (t=3): |
GT (Italian): |
Masked (t=2): |
Recov. (t=2): |
Masked (t=3): |
Recov. (t=3): |

Figure 6: The recovery of masked images from English and Italian using their
tasks models and the models of the next tasks.

For a further demonstration of the suitability of our CSSL-MHTR method for
incrementally learning languages/scripts, in this subsection, we show some
qualitative results. First, Figure 6 shows the reconstruction of the images
for English and Italian. For each image, the model makes the
masking/reconstruction after learning the current as well as the next tasks.
We can notice the ability of the model at task $t$ in reconstructing an image
from a previous task $t-n$ with the same performance as the model $t-n$.n
Hence, it is clear that our method prevents forgetting. Next, we also show the
HTR performance after fine-tuning. Here, we compare the result obtained by our
model after learning the second and third tasks with the result of a
monolingual pre-train model. As it can be seen in Figure 5, when learning more
tasks, our models’ HTR performance improves, with fewer errors. This is due to
the robustness of the representation, which improves over tasks, since the
model is trained to reconstruct images with different handwriting styles,
backgrounds, scripts, etc. Not surprisingly, our model can be successfully
pre-trained on all these different tasks, obtaining more robust
representations.

##  5 Conclusion

In this paper we have presented, to the best of our knowledge, the first
application of continual learning in handwritten text recognition. Our
framework benefits from continual learning and self-supervised learning for
multi-script/language handwritten text recognition, as an example of sequence
recognition. Continual learning can be used in the self-supervised pre-
training of the model to boost the performance of the downstream tasks on
multiple handwritten languages/scripts. Our proposed CSSL-MHTR approach
consists of an encoder-decoder transformer model that includes language/script
adapter components and a memory replay strategy for continual self-supervised
learning on handwritten text images. Extensive experimental results
demonstrated the effectiveness of our approach by obtaining the best
performance compared to state-of-the-art continual learning approaches.
Experiments also proved our method’s efficiency in terms of temporal and
spatial complexity. We believe that our model can serve as a benchmark for
future research in this field.

As future work, we plan to extend this approach to recognize full pages
instead of segmented lines. Also, we will explore the addition of other
document analysis tasks to be learned continually, for instance, layout
analysis, name entity recognition and information extraction.

## References

  * [1] Abdelrahman Abdallah, Mohamed Hamada, and Daniyar Nurseitov.  Attention-based fully gated cnn-bgru for russian handwritten text.  Journal of Imaging, 6(12):141, 2020.
  * [2] Aviad Aberdam, Ron Litman, Shahar Tsiper, Oron Anschel, Ron Slossberg, Shai Mazor, R Manmatha, and Pietro Perona.  Sequence-to-sequence contrastive learning for text recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15302–15312, 2021.
  * [3] Manoj Acharya, Tyler L Hayes, and Christopher Kanan.  Rodeo: Replay for online object detection.  arXiv preprint arXiv:2008.06439, 2020.
  * [4] Théodore Bluche, Jérôome Louradour, and Ronaldo Messina.  Scan, attend and read: End-to-end handwritten paragraph recognition with mdlstm attention.  In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 1050–1055. IEEE, 2017.
  * [5] Théodore Bluche and Ronaldo Messina.  Gated convolutional recurrent neural networks for multilingual handwriting recognition.  In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 646–651. IEEE, 2017.
  * [6] Shuhao Cao, Peng Xu, and David A Clifton.  How to understand masked autoencoders.  arXiv preprint arXiv:2202.03670, 2022.
  * [7] Silvia Cascianelli, Vittorio Pippi, Martin Maarand, Marcella Cornia, Lorenzo Baraldi, Christopher Kermorvant, and Rita Cucchiara.  The lam dataset: A novel benchmark for line-level handwritten text recognition.  In 2022 26th International Conference on Pattern Recognition (ICPR), pages 1506–1513. IEEE, 2022.
  * [8] Hyuntak Cha, Jaeho Lee, and Jinwoo Shin.  Co2l: Contrastive continual learning.  In Proceedings of the IEEE/CVF International conference on computer vision, pages 9516–9525, 2021.
  * [9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [10] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [11] Xiaoxue Chen, Lianwen Jin, Yuanzhi Zhu, Canjie Luo, and Tianwei Wang.  Text recognition in the wild: A survey.  ACM Computing Surveys (CSUR), 54(2):1–35, 2021.
  * [12] Zhuo Chen, Fei Yin, Xu-Yao Zhang, Qing Yang, and Cheng-Lin Liu.  Multrenets: Multilingual text recognition networks for simultaneous script identification and handwriting recognition.  Pattern Recognition, 108:107555, 2020.
  * [13] Zhanzhan Cheng, Fan Bai, Yunlu Xu, Gang Zheng, Shiliang Pu, and Shuigeng Zhou.  Focusing attention: Towards accurate text recognition in natural images.  In Proceedings of the IEEE international conference on computer vision, pages 5076–5084, 2017.
  * [14] Iulian Cojocaru, Silvia Cascianelli, Lorenzo Baraldi, Massimiliano Corsini, and Rita Cucchiara.  Watch your strokes: Improving handwritten text recognition with deformable convolutions.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 6096–6103. IEEE, 2021.
  * [15] Denis Coquenet, Clément Chatelain, and Thierry Paquet.  Recurrence-free unconstrained handwritten text recognition using gated fully convolutional network.  In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 19–24. IEEE, 2020.
  * [16] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised visual representation learning by context prediction.  In Proceedings of the IEEE international conference on computer vision, pages 1422–1430, 2015.
  * [17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  2020\.
  * [18] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra.  Pathnet: Evolution channels gradient descent in super neural networks.  arXiv preprint arXiv:1701.08734, 2017.
  * [19] Enrico Fini, Victor G Turrisi Da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal.  Self-supervised models are continual learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9621–9630, 2022.
  * [20] Alex Gomez-Villa, Bartlomiej Twardowski, Lu Yu, Andrew D Bagdanov, and Joost van de Weijer.  Continually learning self-supervised representations with projected functional regularization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3867–3877, 2022.
  * [21] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent-a new approach to self-supervised learning.  Advances in Neural Information Processing Systems, 33:21271–21284, 2020.
  * [22] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman.  Learning to read by spelling: Towards unsupervised text recognition.  In Proceedings of the 11th Indian Conference on Computer Vision, Graphics and Image Processing, pages 1–10, 2018.
  * [23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [24] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  * [25] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
  * [26] Jing Huang, Kevin J Liang, Rama Kovvuri, and Tal Hassner.  Task grouping for multilingual text recognition.  In Computer Vision–ECCV 2022 Workshops, pages 297–313. Springer, 2023.
  * [27] Jing Huang, Guan Pang, Rama Kovvuri, Mandy Toh, Kevin J Liang, Praveen Krishnan, Xi Yin, and Tal Hassner.  A multiplexed network for end-to-end, multilingual ocr.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4547–4557, 2021.
  * [28] Lei Kang, Pau Riba, Marçal Rusiñol, Alicia Fornés, and Mauricio Villegas.  Pay attention to what you read: non-recurrent handwritten text-line recognition.  Pattern Recognition, 129:108766, 2022.
  * [29] Lei Kang, J Ignacio Toledo, Pau Riba, Mauricio Villegas, Alicia Fornés, and Marçal Rusinol.  Convolve, attend and spell: An attention-based sequence-to-sequence model for handwritten word recognition.  In Pattern Recognition: 40th German Conference, GCPR 2018, Stuttgart, Germany, October 9-12, 2018, Proceedings 40, pages 459–472. Springer, 2019.
  * [30] Samuel Kessler, Bethan Thomas, and Salah Karout.  An adapter based pre-training for efficient and scalable self-supervised speech representation learning.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3179–3183. IEEE, 2022.
  * [31] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [32] Dingcheng Li, Zheng Chen, Eunah Cho, Jie Hao, Xiaohu Liu, Fan Xing, Chenlei Guo, and Yang Liu.  Overcoming catastrophic forgetting during domain adaptation of seq2seq language generation.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5441–5454, 2022.
  * [33] Guodun Li, Yuchen Zhai, Qianglong Chen, Xing Gao, Ji Zhang, and Yin Zhang.  Continual few-shot intent detection.  In Proceedings of the 29th International Conference on Computational Linguistics, pages 333–343, 2022.
  * [34] Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei.  Trocr: Transformer-based optical character recognition with pre-trained models.  arXiv preprint arXiv:2109.10282, 2021.
  * [35] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  In International Conference on Machine Learning, pages 3925–3934. PMLR, 2019.
  * [36] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  * [37] Zhiwei Lin, Yongtao Wang, and Hongxiang Lin.  Continual contrastive learning for image classification.  In 2022 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2022.
  * [38] Hao Liu, Bin Wang, Zhimin Bao, Mobai Xue, Sheng Kang, Deqiang Jiang, Yinsong Liu, and Bo Ren.  Perceiving stroke-semantic context: Hierarchical contrastive learning for robust scene text recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1702–1710, 2022.
  * [39] Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang.  Rethinking the representational continuity: Towards unsupervised continual learning.  arXiv preprint arXiv:2110.06976, 2021.
  * [40] Arun Mallya and Svetlana Lazebnik.  Packnet: Adding multiple tasks to a single network by iterative pruning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7765–7773, 2018.
  * [41] U-V Marti and Horst Bunke.  The iam-database: an english sentence database for offline handwriting recognition.  International Journal on Document Analysis and Recognition, 5:39–46, 2002.
  * [42] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Psychology of learning and motivation, volume 24, pages 109–165. Elsevier, 1989.
  * [43] Natawut Monaikul, Giuseppe Castellucci, Simone Filice, and Oleg Rokhlenko.  Continual learning for named entity recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13570–13577, 2021.
  * [44] Daniyar Nurseitov, Kairat Bostanbekov, Daniyar Kurmankhojayev, Anel Alimova, Abdelrahman Abdallah, and Rassul Tolegenov.  Handwritten kazakh and russian (hkr) database for text recognition.  Multimedia Tools and Applications, 80:33075–33097, 2021.
  * [45] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  Neural networks, 113:54–71, 2019.
  * [46] Benedikt Pfülb and Alexander Gepperth.  A comprehensive, application-oriented study of catastrophic forgetting in dnns.  arXiv preprint arXiv:1905.08101, 2019.
  * [47] Joan Puigcerver.  Are multidimensional recurrent layers really necessary for handwritten text recognition?  In 2017 14th IAPR international conference on document analysis and recognition (ICDAR), volume 1, pages 67–72. IEEE, 2017.
  * [48] Mark B Ring.  Child: A first step towards continual learning.  Machine Learning, 28(1):77–104, 1997.
  * [49] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne.  Experience replay for continual learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [50] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [51] Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai.  Robust scene text recognition with automatic rectification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4168–4176, 2016.
  * [52] Mohamed Ali Souibgui, Sanket Biswas, Andres Mafla, Ali Furkan Biten, Alicia Fornés, Yousri Kessentini, Josep Lladós, Lluis Gomez, and Dimosthenis Karatzas.  Text-diae: A self-supervised degradation invariant autoencoders for text recognition and document enhancement.  arXiv preprint arXiv:2203.04814, 2022.
  * [53] Mohamed Ali Souibgui, Alicia Fornés, Yousri Kessentini, and Beáta Megyesi.  Few shots are all you need: A progressive learning approach for low resource handwritten text recognition.  Pattern Recognition Letters, 160:43–49, 2022.
  * [54] Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee.  Lamol: Language modeling for lifelong language learning.  arXiv preprint arXiv:1909.03329, 2019.
  * [55] Sebastian Thrun.  Lifelong learning algorithms.  Learning to learn, 8:181–209, 1998.
  * [56] Mohamed Yousef and Tom E Bishop.  Origaminet: weakly-supervised, segmentation-free, one-step, full page text recognition by learning to unfold.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14710–14719, 2020.
  * [57] Xinyun Zhang, Binwu Zhu, Xufeng Yao, Qi Sun, Ruiyu Li, and Bei Yu.  Context-based contrastive learning for scene text recognition.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3353–3361, 2022.
  * [58] Yaping Zhang, Shuai Nie, Wenju Liu, Xing Xu, Dongxiang Zhang, and Heng Tao Shen.  Sequence-to-sequence domain adaptation network for robust text image recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2740–2749, 2019.

##  Appendix A Overview

CSSL-MHTR is a Continual SSL framework, aiming to continually learn
representations from handwritten document images. The proposed model is then,
fine-tuned on multi-script handwritten text recognition tasks. In this
supplementary material, we discuss and highlight some essential insights of
CSSL-MHTR, in terms of composition and configuration.

##  Appendix B Datasets and Metrics

###  B.1 Datasets

The datasets used in the experiments are the following:

  * •

The IAM dataset [41] contains forms of handwritten English text created by 657
writers. The training set comprises 747 documents (6,482 lines, 55,081 words),
the validation set is composed of 116 documents (976 lines, 8,895 words) and
the test set contains 336 documents (2,915 lines, 25,920 words). In addition
to that, we used 40k synthetic lines images only for the first pre-trained
model.

  * •

The Ludovico Antonio Muratori (LAM) dataset [7] is a large line-level HTR
dataset of Italian ancient manuscripts edited by a single author over 60
years. The basic split consists of 19,830 lines for training, 2,470 for
validation, and 3,523 lines for testing. We used the same split in our
experiments.

  * •

The HKR dataset [44] is a handwritten text dataset containing Russian Words
(Areas, Cities, Villages, Settlements, Streets) written by a hundred different
writers. It also incorporates the most popular words in the Republic of
Kazakhstan. A few pre-processing and segmentation procedures have been
developed together with the database. They contain free different handwriting
forms. The database is prepared to provide training and testing sets for
Kazakh and Russian word-recognition research. The database consists of more
than 1400 filled forms. There are approximately 63,000 sentences, more than
715,699 symbols, and there are approximately 106,718 words. We used the same
basic split for this dataset: Training (70%), Validation (15%), and Testing
(15%).

###  B.2 Evaluation Metric

We used the standard Character Error Rate (CER) to evaluate the accuracy of
text recognition. For a set of ground truths, the CER is computed using the
Levenshtein distance which is the sum of the character substitutions ($Sc$),
insertions ($Ic$), and deletions ($Dc$), divided by the total number of the
ground truth characters ($Nc$). Formally, the CER is computed as:

$CER=\frac{Dc+Sc+Ic}{Nc}$ (5)

##  Appendix C Implementation Details

The implementation details of our CSSL-MHTR approach are summarized in Table
3. During the pre-training step, we deploy an encoder with 6 layers, 8
attention heads to encode the input, and 768 embedding dimensions. We used the
same number of layers and attention heads for the decoder, with a dimension of
512 in the fine-tuning stage (text recognition). At masking, each input image
with size 64$\times$800$\times$3 is divided into a set of patches with size
8$\times$8$\times$3\. Then, we employ random masking that grows progressively
from 2.5% to attend 40% of the patches in the final pre-training epoch. To
prevent the model from over-fitting, we used regularisation techniques such as
data augmentation and early stopping. In the fine-tuning stage, we use the
same encoder (pre-trained) with a different decoder of 6 layers, 8 attention
heads, and an embedding dimension of 768.

GT (English): |
---|---
Masked (MaskRatio=0.025,epoch=1): |
Pred (MaskRatio=0.025): |
Masked (MaskRatio=0.15,epoch=50): |
Predicted (MaskRatio=0.15): |
Masked (MaskRatio=0.25,epoch=90): |
Predicted (MaskRatio=0.25): |
Masked (MaskRatio=0.4,epoch=150): |
Predicted (MaskRatio=0.4): |
Figure 7: The recovery of masked images from the English dataset with
progressive mask ratio. Table 3: Implementation Details. Implementation
details of CSSL-MHTR during pre-training and fine-tuning.

ConFigure | Pre-Training | Fine-tuning
---|---|---
Optimizer | AdamW | Adam
Learning rate | 1.5 $e^{-4}$ | 5 $e^{-5}$
Weight decay | 0.05 | 0.05
Optimizer momentum |  $\beta_{1}$, $\beta_{2}$=0.9, 0.95 |  $\beta_{1}$, $\beta_{2}$=0.9, 0.95
Batch size | 32 | 32
Learning rate schedule | cosine decay | cosine decay
Warmup epochs | 3 | 3

###  C.1 Masking Strategy

The masking strategy that was used in our model consists in starting with
masking 2.5% of the patches and incrementally increasing this masking ratio by
adding 2.5% every 10 epochs to attend 40% as the maximum used masking ratio.
This allows the model to start easily predicting the masked parts and then
progressively increase the difficulty. We found that without this strategy,
and by keeping a masking ratio of 40% from the beginning, the model is unable
to reconstruct the image during pre-training. In addition to that, in order to
balance the patches classes, we force half of the masking to cover the text
parts of the image and avoid the background pixels, while leaving the ability
of the other half to select random patches (including background). In Figure 7
we report an overview of the masking during different stages of pre-training.

###  C.2 Memory Replay Setting

In our approach, we used a memory replay strategy with a memory
$R_{1,2,3,...t-1}$ dedicated to storing images from the previous datasets.
When learning the task $t$, these images are presented to the model to perform
the memory replay. More precisely, at each iteration, we load a batch from the
current task and a randomly chosen batch from the previous tasks to train the
model. The previous task batch is used to compute the distillation loss. This
is done by feeding the data batch to both models $t-1$ and $t$, then forcing
the model $t$ to behave similarly to the model $t-1$. The current task batch
is fed only to the model $t$ and a reconstruction loss is computed and used to
learn the current task.

Our used memory buffer stores 3200 samples. When learning the second task,
this number is fully dedicated to the first task. This means we have 3200
samples of the first task in our memory. When moving to the third task, the
size of the memory buffer will be equally shared between the first and second
tasks. This means the memory buffer will contain 1600 samples from the first
task and 1600 from the second task. And so on when learning further tasks.


# The Impact of Far-Infrared/Sub-Millimeter Data on the Star Formation Rates
of Massive Dusty Galaxies at Cosmic Noon

Eden Wise, Shardha Jogee, and Yuchen Guo

###### Abstract

We explore how the star formation rate (SFR), stellar mass, and other
properties of massive dusty galaxies at cosmic noon are impacted when far-
infrared (FIR)/sub-millimeter data are added to datasets containing only
ultraviolet (UV) to near-infrared (NIR) data. For a sample of 92 massive
(stellar mass $>4{\times}10^{10}$ M⊙) dusty galaxies at $z\,{\sim}\,1.5$ to
3.0 (corresponding to ${\sim}25$% of cosmic history), we fit the spectral
energy distributions (SEDs) based on DECam UV-to-optical data, VICS82,
NEWFIRM, and Spitzer-IRAC NIR data, and Herschel-SPIRE FIR/sub-millimeter data
using the Bayesian Analysis of Galaxies for Physical Inference and Parameter
Estimation (BAGPIPES) SED-fitting code. We assume a delayed tau star formation
history with a log10 prior on tau and derive the posterior distributions of
stellar mass, SFR, extinction, and specific SFR. We find that adding FIR/sub-
millimeter data leads to SFR estimates that can be both significantly higher
or lower (typically by up to a factor of 10) than estimates based on UV-to-NIR
data alone, depending on the type of galaxies involved. We find that the
changes in SFR scale with changes in extinction. These results highlight the
importance of including FIR/sub-millimeter data in order to accurately derive
the SFRs of massive dusty galaxies at $z\,{\sim}\,2$.

Department of Astronomy, University of Texas at Austin, Austin, Texas, USA

##  1 Introduction

One of the key questions in astronomy is how galaxies—the building blocks of
the Universe—formed their stars over time and assembled their stellar mass.
Most large galaxy surveys trace the star formation rate (SFR) of galaxies by
using readily available rest-frame ultraviolet (UV) data, which map the
unobscured light from hot young massive stars and constrain the unobscured
SFR. In order to get a more complete picture of star formation, we also
ideally need far-infrared/sub-millimeter (FIR/sub-mm) data, which traces the
reprocessed radiation emitted by dust that absorbs part of the UV light and
maps the obscured SFR. This is particularly important at redshift
$z\,{\sim}\,2$ (when the Universe was ${\sim}25$% of its present age) as this
is the period when the cosmic SFR history of the Universe peaked and galaxies
were typically more gas-rich and dusty than today. In this work we explore how
estimates of SFR, stellar mass, and other properties of massive dusty galaxies
at redshifts of $z\,{\sim}\,1.5$ to 3.0 change when FIR/sub-mm data are added
to datasets containing only UV to near-infrared (NIR) data.

##  2 Data and Sample

We consider a sample of 92 massive (stellar mass $M_{\star}>4{\times}10^{10}$
M⊙) dusty galaxies at $z\,{\sim}\,1.5$ to 3.0. Our photometric dataset is
comprised of Dark Energy Camera (DECam) (Wold et al. 2019) $u,g,r,i,z$, VICS82
(Geach et al. 2017) $J$ and $K_{s}$ data, NEWFIRM (Stevans et al. 2021) $K$
data, and Spitzer-IRAC (Papovich et al. 2016) 3.6 and 4.5 $\mu$m NIR data, and
Herschel-SPIRE (Viero et al. 2014) 250, 350, and 500 $\mu$m FIR/sub-mm data.

In order to obtain a sample of massive dusty galaxies, we applied a set of
strict criteria to a previously selected set of galaxies published in Sherman
et al. (2019). The selection performed in Sherman et al. (2019) utilized a
large-area, multi-wavelength survey of SDSS Stripe 82 which aimed to study
objects at $z\,{\sim}\,1.5$ to 3.5. After initial quality cuts provided a
sample of 158,879 galaxies with $M_{\star}>4{\times}10^{10}$ M⊙ at
$z\,{\sim}\,1.5$ to 3.5, the following criteria were applied in order to
extract a sample of massive star forming galaxies. To filter out galaxies with
unreliable masses a $S/N\,{\geq}\,5$ was required for Spitzer-IRAC data.
Additionally, to remove quiescent galaxies a $S/N\,{\geq}\,5$ in the $r$-band
and a specific SFR (sSFR) > 10-11 yr-1 were required, resulting in a sample of
5,352 massive star forming galaxies.

The Herschel-SPIRE data at 250, 350, and 500 $\mu$m have a poor angular
resolution of 18.1, 25.2, and 36.6 arcseconds FWHM respectively, and tend to
blend the emission from several galaxies when they are close together. To
avoid this blending problem, we need to work with isolated galaxies. Starting
from the above-described sample from Sherman et al. (2019), we proceed to
identify isolated galaxies via the following steps. We first matched the
NEWFIRM $K$-band position of each object to within 4 arcseconds ($\sim$28 kpc
at $z\,{\sim}\,2$) of the position of each Herschel-SPIRE band source. Once we
obtained a set of galaxies with a position match and a nonzero flux in all
three Herschel-SPIRE bands, we identified a subset of isolated galaxies by
requiring that no other object falls within 9 arcseconds of the NEWFIRM
$K$-band position of the galaxy. This results in a final sample of 92 massive
dusty isolated galaxies at $z\,{\sim}\,1.5$ to 3.0 with a clear Herschel-SPIRE
counterpart.

##  3 Methodology

In order to derive SFRs, we first built the empirical spectral energy
distribution (SED) of each galaxy based on UV-to-NIR data only, including
DECam $u,g,r,i,z$, VICS82 $J$ and $K_{s}$, NEWFIRM $K$, and Spitzer-IRAC 3.6
and 4.5 $\mu$m photometric data. We then fitted the empirical SED of each
galaxy with the Bayesian Analysis of Galaxies for Physical Inference and
Parameter Estimation (BAGPIPES) SED-fitting code (Carnall et al. 2018).
BAGPIPES has several advantages: it allows for different star formation
histories (e.g., both rising and falling star formation histories) and its
Bayesian approach produces both prior and posterior probability distributions
of important galaxy properties such as SFR, $M_{\star}$, and extinction
($A_{V}$). Figure 1 shows examples of the UV-to-NIR empirical SED fitted with
the theoretical SEDs produced by BAGPIPES. We then added the Herschel-SPIRE
FIR/sub-mm data at 250, 350, and 500 $\mu$m to the empirical SED and repeated
the fit (Figure 2). The same methodologies and prior distributions are used
for both sets of fits.

With BAGPIPES, we assume a delayed tau star formation history with a log10
prior distribution on tau. The log10 prior on tau leads to a distribution
dominated by actively star forming galaxies with high sSFR, but still allows
for a long tail of quiescent galaxies with low sSFR (middle panel in Figures 1
and 2). The delayed tau star formation history is described by the functional
form SFR($t$) $\propto(t-T_{0})\exp{(t-T_{0})/\tau}$.

Figure 1.: Top panel: SED fit of a massive star forming galaxy at redshift
$z\,{\sim}\,2$. The grey band represents the theoretical SEDs within
${\pm}1~{}\sigma$ produced by BAGPIPES. The red circles represent the
empirical SED based on UV-to-NIR data. Middle panel: Prior distributions of
SFR, $M_{\star}$, sSFR and $A_{V}$ for the BAGPIPES SED models. Bottom panel:
Posterior distributions of derived quantities. The three lines in the middle
and bottom panels represent the median values and values $\pm$1$\sigma$ from
the median. Figure 2.: Similar to Figure 1, but the Herschel-SPIRE FIR/sub-mm
data at 250, 350, and 500 $\mu$m has now been added to the empirical SED of
the same galaxy.

##  4 Results and Conclusion

Figure 3.: Median posterior value of the the stellar mass based on UV-to-FIR
data (($M_{\star}/{\rm M}_{\odot}$)UV-FIR) versus the median posterior value
of the stellar mass based on UV-to-NIR data only (($M_{\star}/{\rm
M}_{\odot}$)UV-NIR). The addition of FIR/sub-mm data does not significantly
change estimates of the stellar mass: the changes are typically less than a
factor of three on a case by case basis. Figure 4.: The ratio of (Median
SFRUV-FIR/Median SFRUV-NIR), representing the ratio of (median SFR based on
UV-to-FIR/sub-mm data/median SFR based on UV-to-NIR data only), is plotted
against the median SFR based on UV-to-NIR data only (SFRUV-NIR). The dashed
line marks an SFR ratio of 1. Interestingly, this plot shows two different
regimes: for galaxies with low median SFRUV-NIR, the addition of FIR/sub-mm
data leads to an increase in the estimated SFR by up to a factor of typically
ten, while the reverse is true for galaxies with high median SFRUV-NIR.

As shown in Figure 3, the addition of FIR/sub-mm data does not significantly
change estimates of the stellar mass: the changes are typically less than a
factor of three on a case by case basis. This is not surprising as the latter
is primarily traced by NIR radiation. In contrast, we find that adding
FIR/sub-mm data can lead to SFR estimates that can be both significantly
higher or lower (typically by up to a factor of 10) than estimates based on
UV-to-NIR data alone (Figure 4). In particular, as shown on Figure 4, there
are two regimes: for galaxies with low median SFRUV-NIR, the addition of
FIR/sub-mm data leads to an increase in the estimated SFR by up to a factor of
typically ten, while the reverse is true for galaxies with high median SFRUV-
NIR. We find that the changes in SFR scale with changes in extinction.

When comparing our results to the literature, we note that Pacifici et al.
(2023) also finds that for their sample of galaxies, the addition of FIR data
leads to a decrease in SFR estimates compared to SFRs based on UV-to-NIR data
alone. However, we note that our study and Pacifici et al. (2023) target very
different samples of galaxies: the galaxies in Pacifici et al. (2023) have
much lower UV-to-NIR based SFRs (ranging from 10 to 100 M⊙yr-1) than those in
our sample (ranging from 100 to 1000 M⊙yr-1).

These results highlight the importance of including FIR/sub-mm data in order
to accurately derive the SFRs of massive dusty galaxies at cosmic noon. Future
high angular resolution, high sensitivity surveys in the FIR/sub-mm regime
would greatly benefit future studies of the SFRs of massive dusty galaxies at
early cosmic times.

### Acknowledgments

We gratefully acknowledge support from NSF grant NSF AST-1614798 and the
Roland K. Blumberg Endowment in Astronomy.

## References

  * Carnall et al. (2018) Carnall, A. C., McLure, R. J., Dunlop, J. S., & Davé, R. 2018, Monthly Notices of the Royal Astronomical Society, 480, 4379–4401, doi: 10.1093/mnras/sty2169
  * Geach et al. (2017) Geach, J. E., Lin, Y.-T., Makler, M., et al. 2017, The Astrophysical Journal Supplement Series, 231, 7, doi: 10.3847/1538-4365/aa74b6
  * Pacifici et al. (2023) Pacifici, C., Iyer, K. G., Mobasher, B., et al. 2023, The Astrophysical Journal, 944, 141, doi: 10.3847/1538-4357/acacff
  * Papovich et al. (2016) Papovich, C., Shipley, H. V., Mehrtens, N., et al. 2016, The Astrophysical Journal Supplement Series, 224, 28, doi: 10.3847/0067-0049/224/2/28
  * Sherman et al. (2019) Sherman, S., Jogee, S., Florez, J., et al. 2019, Monthly Notices of the Royal Astronomical Society, 491, 3318–3335, doi: 10.1093/mnras/stz3229
  * Stevans et al. (2021) Stevans, M. L., Finkelstein, S. L., Kawinwanichakij, L., et al. 2021, The Astrophysical Journal, 921, 58, doi: 10.3847/1538-4357/ac0cf6
  * Viero et al. (2014) Viero, M. P., Asboth, V., Roseboom, I. G., et al. 2014, The Astrophysical Journal Supplement Series, 210, 22, doi: 10.1088/0067-0049/210/2/22
  * Wold et al. (2019) Wold, I. G. B., Kawinwanichakij, L., Stevans, M. L., et al. 2019, The Astrophysical Journal Supplement Series, 240, 5, doi: 10.3847/1538-4365/aaee85


11institutetext: Univ. Grenoble Alpes, Inserm, U1216, Grenoble Institut
Neurosciences, 38000, FR 22institutetext: Pixyl, Research and Development
Laboratory, 38000 Grenoble, FR 33institutetext: Univ. Grenoble Alpes, Inria,
CNRS, Grenoble INP, LJK, 38000 Grenoble, FR

# Beyond Voxel Prediction Uncertainty: Identifying brain lesions you can trust

Anonymous  Benjamin Lambert 11 2 2 Florence Forbes 33 Senan Doyle 22 Alan
Tucholka 22 Michel Dojat 11

###### Abstract

Deep neural networks have become the gold-standard approach for the automated
segmentation of 3D medical images. Their full acceptance by clinicians remains
however hampered by the lack of intelligible uncertainty assessment of the
provided results. Most approaches to quantify their uncertainty, such as the
popular Monte Carlo dropout, restrict to some measure of uncertainty in
prediction at the voxel level. In addition not to be clearly related to
genuine medical uncertainty, this is not clinically satisfying as most objects
of interest (e.g. brain lesions) are made of groups of voxels whose overall
relevance may not simply reduce to the sum or mean of their individual
uncertainties. In this work, we propose to go beyond voxel-wise assessment
using an innovative Graph Neural Network approach, trained from the outputs of
a Monte Carlo dropout model. This network allows the fusion of three
estimators of voxel uncertainty: entropy, variance, and model’s confidence;
and can be applied to any lesion, regardless of its shape or size. We
demonstrate the superiority of our approach for uncertainty estimate on a task
of Multiple Sclerosis lesions segmentation.

###### Keywords:

MS lesion Detection Deep LearningInterpretabilty Prediction

##  1 Introduction

Magnetic Resonance Imaging (MRI) is the standard imaging modality for the
diagnosis and follow-up of Multiple Sclerosis (MS). It allows a direct
observation of brain lesions produced by the disease and provides information
about the pathology stage or treatment efficiency. Deep Learning (DL)
approaches, based on a trained U-Net-like neural network, are invaluable tools
to automatically delineate MS lesions [18]. Although powerful and versatile,
these models provide segmentation maps that are typically opaque, with no
indication regarding their certainty. This hinders full acceptance of DL
models in clinical routine, for which uncertainty attached to the computerized
results is essential for their interpretation and to avoid misleading
predictions.

A variety of methods have been proposed to quantify the uncertainty attached
to deep neural networks [1]. Among them, the Monte Carlo (MC) dropout stands
out as one of the simplest approach, as it can be applied to any model trained
with the dropout technique [20]. Such a model can be interpreted as a Bayesian
neural network, giving access to the interesting properties of these
probabilistic models regarding quantification of their uncertainty [4]. More
particularly at inference, for a given input, multiple stochastic forward
passes are computed by keeping dropout activated, corresponding to empirical
samples from the approximated predictive distribution. This produces a set of
softmax probabilities that can further be used to compute uncertainty
estimates. Applied to MRI segmentation, the MC dropout method produces
uncertainty metrics for each voxel in the volume, resulting in so-called
voxel-wise uncertainty maps [17, 6, 12]. The clinically-relevant information,
however, is at a higher level, typically at the instance (lesion, tissue)
level.

Natural ways to obtain such instance-wise uncertainties, meaning the
uncertainties attached to each connected component within the output
segmentation, are through a post hoc aggregation of voxel-wise uncertainty
estimations. Existing approaches include computing the mean uncertainty of
voxels belonging to the same class in the segmentation [16] (thus producing
one uncertainty estimate per class, rather than per connected component). In
the context of MS, lesion-wise uncertainty was also estimated using the logsum
of the connected voxels uncertainties [12]. Using the mean implies that each
component uncertainty contributes equally to the overall instance score, while
the use of the logsum assumes that connected voxels are conditionally
independent, given that they belong to the same instance. These highly
simplified assumptions may degrade the quality of instance uncertainty
computation. To go further, a side-learner called MetaSeg has been proposed to
predict the Intersection Over Unions (IoU) of each individual segmented
instance with the ground truth [15]. For this task, a Linear Regression Model
is trained based on a series of features derived from a standard segmentation
model’s output probabilities. The predicted score is then used as a marker of
instance uncertainty. Yet, the input features of MetaSeg consist in averaged
voxel-wise metrics, leading to the same restrictions than the previously-
described post hoc aggregation methods. Additionally, it has been proposed to
train an auxiliary Graph (Convolutional) Neural Network (GCNN) using the
outputs of a MC dropout U-Net (i.e. voxel-wise segmentation and uncertainty
maps) to refine the predicted masks [19]. This approach, however, remains at
the voxel level and focuses on 2D segmentation tasks.

In this work, we propose to build from the two last methods to overcome their
respective limitations. Indeed, we implement a GCNN at the output of a trained
MC dropout U-Net model. Using the predicted 3D segmentation outputs, each
individual segmented lesion is modeled by a graph whose voxels are the
interconnected nodes. Node features are determined by the input and output of
the U-Net, comprising the voxel image intensities, the voxel predicted label,
and voxel-wise uncertainty maps. We implement two alternative variants of the
proposed GCNN, either classification or regression, to quantify lesions
uncertainty. We test our framework on a task of 3D binary segmentation
segmentation on MS data. Results demonstrate the superiority of our approach
compared to known methods.

##  2 Our Framework: Graph modelization for lesion uncertainty quantification

Overview: Consider an input image X and a trained MC dropout segmentation
model $\mathcal{N}$ with parameters $W$ that produces a segmentation
$Y=\mathcal{N}(X,W)$ and a set of voxel-wise uncertainty maps $U_{i}$ (e.g.
entropy, variance, PCS, etc.). Our objective is to quantify the uncertainty of
each instance (i.e. lesion) in Y. To do so, we propose to train an auxiliary
GCNN to predict this uncertainty directly from X, Y, and $U_{i}$ (see Figure
1).

Figure 1: Illustration of the proposed framework for learning lesion
uncertainty from the outputs of a Monte Carlo dropout model. See the text for
details of each block.

###  2.1 Monte Carlo dropout model and voxel-wise uncertainty

We use a generic 3D U-Net [2] for its simplicity and popularity within the
field, although our method can be employed with any segmentation model trained
with dropout. We add 3D dropout [21] with a rate of $p=0.2$ at the end of each
encoding and decoding block. The model is trained on annotated datasets
composed of pairs of images: (i) input T2-weighted FLAIR MRI sequences $X$ and
(ii) associated ground truth MS lesions segmentation $Y$. At inference,
dropout is kept activated and $T$ forward passes are made for a new input
volume $x^{*}$. We chose $T=20$, as it allows an optimal counterpart between
inference time and quality of uncertainty estimates [13]. From this set of
predictions, several well-known voxel-related uncertainty metrics are
extracted: (see Figure 1, part A): the entropy [5], the variance [7] and the
Predicted Confidence Score (PCS) [23].

###  2.2 Graph dataset generation

####  2.2.1 Inference on Validation Dataset and Connected Component Analysis.

After training, the MC dropout U-Net is subsequently used to generate
segmentation and uncertainty maps on the set-aside validation set of images.
These predictions are used to generate training data for the auxiliary GCNN.
We use a Connected Component Analysis (CCA) to identify each lesion in the
segmentation masks using 26-connectivity — meaning that a lesion is defined by
voxels that are interconnected by their faces, edges, or corner. For each
lesion identified by CCA, we compute the Adjusted Intersection Over Union
($IoU_{adj}$) [15] with the ground truth lesions (see Figure 1, part B). This
variant of the IoU is suited for brain-abnormalities segmentation, where a
connected component in the ground truth can be divided into several pieces in
the predicted segmentation.

Identified lesions can exhibit a wide range of shape and size. To learn from
these data, we must thus design a neural network that can be employed
regardless of the shape and size of the input structure. GCNNs, which can be
interpreted as a generalization of the classic convolutional networks to non-
Euclidean and irregular data, are thus particularly suitable for this task.

####  2.2.2 From voxels to graphs

We first slightly dilate each lesion mask to include surrounding voxels at the
border between classes, which typically convey useful information about
uncertainty. We then convert the dilated mask into a graph by representing its
voxels by nodes and neighborhood relationships by edges. Each node is further
defined by a set of $n+4$ features: (i) the intensity of its corresponding
voxel in each of the $n$ input MRI sequences, (ii) its binarized label (1 for
the observed lesion class and 0 for all other classes), and its 3 voxel-wise
uncertainty estimates: (iii) entropy, (iv) variance and (v) PCS (see Figure 1,
part C). In agreement with the aforementioned 26-connectivity CCA, each node
(i.e. voxel) is connected in the graph to its 26 nearest neighbors.

###  2.3 GCNN architecture and training

Here, we use a lightweight GCNN architecture composed of 2 consecutive Graph
Convolutional layers with a hidden dimension of $h=64$, followed by a Linear
layer (see Figure 1, part D). The model is trained using the graph dataset
generated from the validation images, composed of graphs (transformed
connected components obtained from the segmentation model) along with their
associated ground truth ($IoU_{adj}$). As in [15], we propose two versions of
our model:

  * •

In the classification approach ($\text{GCNN}_{\text{Classif}}$), the
$IoU_{adj}$ labels are first binarized as follows: FP if
$IoU_{adj}(graph)<\epsilon$, and TP if $IoU_{adj}(graph)\geq\epsilon$.
$\epsilon$ is a hyperparameter that we set to $0.1$ in our experiments, so
that lesions with an $IoU_{adj}$ very close to 0 are not wrongly considered as
TP. The network is then trained using the Cross-Entropy Loss. At inference,
structural uncertainty is quantified by the graph FP probability.

  * •

In the regression approach ($\text{GCNN}_{\text{Reg}}$), the model is directly
trained to predict the graph $\widehat{IoU}_{adj}$, using the MSE loss. At
inference, we use $1-\widehat{IoU}_{adj}$ as the structural uncertainty score.

##  3 Material and Method

###  3.1 Data

We combine two open-source MS datasets: from the University Hospital of
Ljubljana (MSLUB) [10] and from the MICCAI 2016 MS segmentation challenge
(MSSEG 2016) [3]. We thus use 83 manually-annotated 3D T2-FLAIR sequences.
Images are resampled to a $1\text{\,}\mathrm{mm}$ isotropic resolution of
$160\times 192\times 160$ to focus on brain tissues, and intensities are
normalized to zero mean and unit variance. We opt for a 4-fold cross-
validation scheme due to the limited number of images. In each fold, we put
aside $25\%$ of the images for testing. From the remaining images, we use
$20\%$ for validation and $80\%$ to train the model. During evaluation,
results are averaged over the 4 folds. Due to the limited number of images, we
extensively use Data Augmentation to train our models, comprising flipping,
rotation, contrast alteration, gaussian noise and blurring.

###  3.2 Comparison with known approaches

To evaluate the relevance of our proposed $\text{GCNN}_{\text{Classif}}$ and
$\text{GCNN}_{\text{Reg}}$ approaches, we implement in parallel known
approaches to obtain instance uncertainty from the U-Net. We use the mean and
logsum of the voxel-wise uncertainty of each lesions, with the 3 different
types of uncertainty. We name these methods $\text{Entropy}_{\text{mean}}$,
$\text{Variance}_{\text{mean}}$, $\text{PCS}_{\text{mean}}$,
$\text{Entropy}_{\text{logsum}}$, $\text{Variance}_{\text{logsum}}$, and
$\text{PCS}_{\text{logsum}}$.

As pointed out in [12], using the logsum assigns a higher uncertainty to
small-size lesions. This appears sub-optimal as small lesions could be
segmented with high confidence, especially in the case of MS lesions. To
verify this point, we implement a naive approach, named Size, which attributes
a lesion uncertainty inversely proportional to its size. The lesion size
(number of voxels composing it) being $S$, its uncertainty is computed as
$1/S$.

Lastly, we implement an approach inspired from the MetaSeg framework [15]. We
extract a series of features from each connected component in the validation
dataset, consisting in the mean entropy, variance and PCS, as well as the size
of the lesion. We then train a Logistic Regression classifier from these 4
features to distinguish between True Positive (TP) and FP lesions
($\text{MetaSeg}_{\text{Classif}}$). Alternatively, we train a Linear
Regression model to directly predict $\widehat{IoU}_{adj}$
($\text{MetaSeg}_{\text{Reg}}$). We use the outputs of these models to obtain
lesion uncertainty as described for the GCNN approach.

###  3.3 Evaluation Setting

For medical applications, the ideal uncertainty quantification should
attribute a higher uncertainty to FP lesions than TP, to allow for proper
interpretation and evaluation of the results. To evaluate this properly, we
use Accuracy-Confidence curves [9]. Briefly, the principle is to set aside the
$\tau\%$ of the most uncertain predicted lesions among the test dataset, and
measure the performance of the model on the remaining lesions by counting the
number of FP and TP lesions. The threshold $\tau$ fluctuates between 0 (all
lesions are kept) and 100 (all lesions are removed). By plotting the couples
(FP, TP) at different thresholds, we obtain an Accuracy-Confidence curve and
compute the AUC (Area Under the Curve) score reflecting the quality of the
estimated lesion uncertainty. FP and TP counts are normalized in the range
$[0,1]$ by dividing by the counts obtained without filtering (at $\tau=0)$.
This metric only depends on the ranking of uncertainties, thus is independent
of the uncertainty ranges of each method ensuring a fair comparison. We
additionally evaluate the segmentation performance of the U-Net on the test
datasets using the Dice coefficient, as well as the total number of TP and FP
lesions. Finally, for each method, we control the correlation between the
estimated uncertainty and the lesion size using the Spearman’s rank
correlation coefficient ($\rho$).

Table 1: U-Net segmentation performance on the MS dataset and number of TP and FP lesions for each fold. Fold | 0 | 1 | 2 | 3
---|---|---|---|---
Dice | 0.672 | 0.645 | 0.705 | 0.693
# TP lesions | 829 | 597 | 715 | 871
# FP lesions | 525 | 294 | 353 | 454

###  3.4 Implementation Details

####  3.4.1 3D Segmentation U-Net

Our segmentation framework was implemented using PyTorch [14]. We opt for a
patch approach to train the segmentation U-Net, meaning that the $160\times
192\times 160$ MRI volumes are split into 3d patches of $160\times 192\times
32$, decreasing the memory cost of training. We use a batch size of 5. The
U-Net is trained with a combination of the Dice [11] and Cross-Entropy loss,
using the ADAM optimizer [8] with a learning rate of $1e^{-4}$ until
convergence. For the training of the segmentation models, a single NVIDIA T4
GPU was used.

####  3.4.2 Graph Neural Networks

We use the Deep Graph Library [22] to implement and train the GCNN models. The
training procedure of our GCNN is standard: we use the ADAM optimizer with a
learning rate of $1e^{-2}$ at the start of training, and progressively
decreasing to $1e^{-5}$. Graphs are presented to the network in the form of
batches, composed of 10 graphs. Due to the small size of the GCNN models, they
were trained on CPU, which took a couple of minutes in our experiments.

Figure 2: Accuracy-Confidence curves for the different methods. The associated
AUC scores are indicated in brackets in the graphs legends.

##  4 Results and Discussion

Accuracy-Confidence curves are presented in Figure 2 along with the
corresponding AUC values. Segmentation performance and correlation
coefficients are presented in Tables 1 and 2. Experimental results show that
both models of the proposed framework outperform the classical methods by a
significant margin, and that their performances are similar with a very small
advantage for the classification version. The naive Size approach achieves the
lowest performance. Similarly, the $logsum$ approaches, also strongly
correlated with the lesion size, have poorer performance than the $mean$
counterparts. Not surprisingly, in the context of MS lesions, the lesion size
is not a satisfying proxy for uncertainty as small lesions can be segmented
with high confidence. In our experimental setting, $MetaSeg$ models do not
outperform simpler methods. This is probably due to the overall simplicity of
these models, failing to fully learn the relationships between the different
input features.

Results show that our graph-based framework can be efficiently used to flag
uncertain lesions, that are more likely to result in False Positives. The
classification variant slightly outperforms regression. We hypothesize that
this is due to the increased difficulty of predicting the exact $IoU_{adj}$,
compared to the binary classification setting. One drawback of our approach is
that it requires an additional validation set containing enough lesions
(typically a few hundreds) to allow GCNN training. However, as most DL
pipelines rely on a set-aside validation set to control overfitting during
training, these data can then be used for this purpose (as it was the case in
this work). The requirement is thus not prohibitive and only necessitates a
sufficiently large validation set.

Overall, our framework is computationally light as CCA is computed only once
per MRI, followed by the graph generation step that can be parallelized among
the lesions. Additionally, in the context of MS, most brain lesions are
relatively small (less than 1000 voxels), which results in small graphs that
are fast to generate. Finally we use 26-connectivity, meaning that a voxel is
only connected to its closest neighbors, which reduces the computational
burden.

Our approach enhances the binary voxel-wise predictions of the segmentation
model with reliable and readable lesion-wise uncertainty estimates. In the
classification setting, uncertainty is cast as the probability of a lesion
being a false positive, which is a straightforward and intelligible
definition. In a real world clinical application, this may help the clinician
examine the automated segmentation in the light of the model’s confidence,
hence allowing a better interpretability of the provided results and a more
trustable usage of the algorithm.

Future work will study the extension to multi-class segmentation, and
inclusion of additional features such as the global location of the lesion
within the MRI volume. Indeed, for brain disorders such as MS, the location of
the lesion within the brain conveys information concerning uncertainty, as
false positives tend to be concentrated in specific brain regions.

Table 2: Evaluation of uncertainty estimates (AUC values). $\rho$ represents Spearman’s rank correlation coefficient $\rho$.  | AUC (%) | Spearman’s $\rho$
---|---|---
$\text{GCNN}_{\text{Classif}}$ | 87.32 | -0.78
$\text{GCNN}_{\text{Reg}}$ | 87.10 | -0.77
$\text{Entropy}_{\text{mean}}$ | 83.80 | -0.42
$\text{Entropy}_{\text{logsum}}$ | 83.72 | -0.97
$\text{Variance}_{\text{mean}}$ | 83.14 | -0.44
$\text{Variance}_{\text{logsum}}$ | 82.99 | -0.99
$\text{PCS}_{\text{mean}}$ | 83.79 | -0.44
$\text{PCS}_{\text{logsum}}$ | 83.88 | -0.98
Size | 80.30 | -1.00
$\text{MetaSeg}_{\text{Classif}}$ | 83.10 | -0.76
$\text{MetaSeg}_{\text{Reg}}$ | 83.42 | -0.77

##  5 Conclusion

This paper presents an innovative graph-based framework to quantify lesion-
wise uncertainty. We demonstrate, with our approach, improvement of the
predicted uncertainty, when compared to various known methods. The strength of
our solution is its generic nature, making it compatible with any segmentation
model trained with dropout.

## References

  * [1] Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U.R., et al.: A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion 76, 243–97 (2021)
  * [2] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. pp. 424–432. Springer (2016)
  * [3] Commowick, O., Kain, M., Casey, R., Ameli, R., Ferré, J.C., Kerbrat, A., Tourdias, T., Cervenansky, F., Camarasu-Pop, S., Glatard, T., et al.: Multiple sclerosis lesions segmentation from multiple experts: The MICCAI 2016 challenge dataset. NeuroImage 244, 118589 (2021)
  * [4] Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In: international conference on machine learning. pp. 1050–1059. PMLR (2016)
  * [5] Gal, Y., Islam, R., Ghahramani, Z.: Deep Bayesian active learning with image data. In: International Conference on Machine Learning. pp. 1183–1192. PMLR (2017)
  * [6] Jungo, A., McKinley, R., Meier, R., Knecht, U., Vera, L., Pérez-Beteta, J., Molina-García, D., Pérez-García, V.M., Wiest, R., Reyes, M.: Towards uncertainty-assisted brain tumor segmentation and survival prediction. In: International MICCAI Brainlesion Workshop. pp. 474–485. Springer (2017)
  * [7] Kendall, A., Badrinarayanan, V., Cipolla, R.: Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding. In: British Machine Vision Conference 2017, BMVC 2017, London, UK, September 4-7, 2017. BMVA Press (2017)
  * [8] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015), http://arxiv.org/abs/1412.6980
  * [9] Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017. pp. 6402–6413 (2017)
  * [10] Lesjak, Ž., Galimzianova, A., Koren, A., Lukin, M., Pernuš, F., Likar, B., Špiclin, Ž.: A novel public MR image dataset of multiple sclerosis patients with lesion segmentations based on multi-rater consensus. Neuroinformatics 16(1), 51–63 (2018)
  * [11] Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international conference on 3D vision (3DV). pp. 565–571. IEEE (2016)
  * [12] Nair, T., Precup, D., Arnold, D.L., Arbel, T.: Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. In: Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 - 21st International Conference, Proceedings, Part I. Lecture Notes in Computer Science, vol. 11070, pp. 655–663. Springer (2018)
  * [13] Orlando, J.I., Seeböck, P., Bogunović, H., Klimscha, S., Grechenig, C., Waldstein, S., Gerendas, B.S., Schmidt-Erfurth, U.: U2-net: A bayesian u-net model with epistemic uncertainty feedback for photoreceptor layer segmentation in pathological oct scans. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). pp. 1441–1445. IEEE (2019)
  * [14] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc. (2019)
  * [15] Rottmann, M., Colling, P., Hack, T.P., Chan, R., Hüger, F., Schlicht, P., Gottschalk, H.: Prediction error meta classification in semantic segmentation: Detection via aggregated dispersion measures of softmax probabilities. In: 2020 International Joint Conference on Neural Networks (IJCNN). pp. 1–9. IEEE (2020)
  * [16] Roy, A.G., Conjeti, S., Navab, N., Wachinger, C., Initiative, A.D.N., et al.: Bayesian QuickNAT: Model uncertainty in deep whole-brain segmentation for structure-wise quality control. NeuroImage 195, 11–22 (2019)
  * [17] Sander, J., de Vos, B.D., Wolterink, J.M., Išgum, I.: Towards increased trustworthiness of deep learning segmentation methods on cardiac MRI. In: Medical Imaging 2019: Image Processing. vol. 10949, p. 1094919. International Society for Optics and Photonics (2019)
  * [18] Shoeibi, A., Khodatars, M., Jafari, M., Moridian, P., Rezaei, M., Alizadehsani, R., Khozeimeh, F., Gorriz, J.M., Heras, J., Panahiazar, M., et al.: Applications of deep learning techniques for automated multiple sclerosis detection using magnetic resonance imaging: A review. Computers in Biology and Medicine 136, 104697 (2021)
  * [19] Soberanis-Mukul, R.D., Navab, N., Albarqouni, S.: Uncertainty-based graph convolutional networks for organ segmentation refinement. In: Medical Imaging with Deep Learning. pp. 755–769. PMLR (2020)
  * [20] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15(1), 1929–1958 (2014)
  * [21] Tompson, J., Goroshin, R., Jain, A., LeCun, Y., Bregler, C.: Efficient object localization using convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 648–656 (2015)
  * [22] Wang, M., Zheng, D., Ye, Z., Gan, Q., Li, M., Song, X., Zhou, J., Ma, C., Yu, L., Gai, Y., et al.: Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.01315 (2019)
  * [23] Zhang, X., Xie, X., Ma, L., Du, X., Hu, Q., Liu, Y., Zhao, J., Sun, M.: Towards characterizing adversarial defects of deep learning software from the lens of uncertainty. In: 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). pp. 739–751. IEEE (2020)