# Exploring Multi-Tasking Learning in Document Attribute Classification

Tanmoy Mondal<EMAIL_ADDRESS>Abhijit Das<EMAIL_ADDRESS>Zuheng Ming SC Team, IMT Atlantique, Brest, France  Thapar University, Punjab,
India  L3i, University of La-Rochelle, France

###### Abstract

In this work, we adhere to explore a Multi-Tasking learning (MTL) based
network to perform document attribute classification such as the font type,
font size, font emphasis and scanning resolution classification of a document
image. To accomplish these tasks, we operate on either segmented word level or
on uniformed size patches randomly cropped out of the document. Furthermore, a
hybrid convolution neural network (CNN) architecture ”MTL+MI”, which is based
on the combination of MTL and Multi-Instance (MI) of patch and word is used to
accomplish joint learning for the classification of the same document
attributes. The contribution of this paper are three fold: firstly, based on
segmented word images and patches, we present a MTL based network for the
classification of a full document image. Secondly, we propose a MTL and MI
(using segmented words and patches) based combined CNN architecture (“MTL+MI”)
for the classification of same document attributes. Thirdly, based on the
multi-tasking classifications of the words and/or patches, we propose an
intelligent voting system which is based on the posterior probabilities of
each words and/or patches to perform the classification of document’s
attributes of complete document image.

###### keywords:

MTL, convolution neural network

††journal: Pattern Recognition Letters

##  1 Introduction

Automatic analysis of document attributes such as font is highly useful for
several document processing tasks such as character recognition (Simard et
al., 2003), document classification, writer and script identification (Shi et
al., 2016). Automatic identification of font type, font size and font emphasis
can highly improve the accuracy of _Optical Character Recognition (OCR)_
systems, especially when the data is processed in multi-script or multi-
language form. If text lines or words are labeled with a font class, then a
specialist OCR system can potentially achieve better recognition rates than an
OCR system trained on many fonts.

Contrary to previous techniques in the literature, which have been mainly
designed for a single task such as _font type_ classification (Tensmeyer et
al., 2017), (Cloppet et al., 2018), this work focuses to learn multiple
attributes in a multi-tasking environment instead of using a single network
for each attribute classification. Given a document image, the motivation of
this work is to automatically identify the Font type (e.g _Arial_ , _Calibri_
, _Courier_ , _Times new roman_ , _Trebuchet_ , _Verdana_), Font size (e.g.
$8$, $10$, $12$), Font emphasis (e.g. _bold_ , _italic_ , _bold-italic_ ,
_none_) and scanning resolution of the document (e.g. $150$ dpi, $300$ dpi,
$600$ dpi) image by using a single _MTL-based convolutional neural network
(CNN)_.

To handle the variations of various font characteristics, we compute deep
features from the segmented word images and also from the extracted patches.
In our fist approach, we have applied a MTL system for the classification of
document attributes by using either word image or patch images as an input. In
our second approach, we have adapted a hybrid CNN of two stream to perform the
same multi-tasks. In the first stream, features are extracted from the input
images from pre-trained CNN (Tensmeyer et al., 2017) and in second stream,
sequential patches are cut from the complete document image and the features
are extracted from these patches These two streams act as multi-instances,
which are combined by vector-wise operation. We term this proposed MTL and MI
based architecture as MTL+MI. The differences among different font types are
subtle or even tiny and to capture these differences, we need to operate on
local level and that’s why we adopted to work on segmented words and/or
patches. In addition, we proposed a voting method for the classification of
each word and/or patch, which are used as the candidates for voting to decide
the final class of the whole document image.

In summary, the contribution of the paper are as follows: i) we present a MTL
based framework to classify the font type, font size, font emphasis and
scanning resolution of document images. ii) a MTL+MI based framework to
jointly learn the tasks with multi-instances i.e. using segmented word and
cropped patch images together, iii) we have proposed an intelligent voting
system based on the posterior probabilities to perform the classification of
the complete document image. The code and dataset used in this research work,
can be found in https://github.com/tanmayGIT/Document-Attribute-
Classification.git and in http://navidomass.univ-lr.fr/TextCopies/
respectively.

##  2 Related Works

We review the literature in two direction: the first one is Font type/family
and Font size classification and the second one is on multi-tasking network,
applied in different domains of computer vision. The existing techniques in
the literature about font type/family recognition can be divided into two main
family. One is _holistic approach_ and other one is _training based
approaches_ , mainly using deep neural network such as CNN and Recurrent
Neural Network (RNN) based techniques. A method proposed by (Ben Moussa et
al., 2010) for Arabic font recognition by using _Fractal geometry_ , which has
resulted in $98\%$ accuracy for $10$ font classes. Another technique is
proposed for optical font recognition by using typographical features and by
using multivariate Bayesian classifier in (Ben Moussa et al., 2010). This
approach reported an accuracy of $97.35\%$ over English text lines for $10$
font classes. More recently, deep learning techniques based on CNN and RNN
have shown very high potential for font classification. By considering font
recognition on a single Chinese character is a sequence of classification
problem, the authors in (Tao et al., 2016) has proposed principal component
based 2D long short-term memory (LSTM) algorithm and were able to classify
single Chinese characters into $7$ font classes with $97.77\%$ accuracy.
Identification of scripts in natural images was proposed in (Shi et al.,
2016). The basic idea is combining deep features and mid-level representations
into a globally trainable deep model. The classification of hand-written
Chinese characters into $5$ calligraphy classes is performed in (Pengcheng et
al., 2017). They have obtained $95\%$ accuracy by using deep features,
extracted from a pre-trained CNN on natural images. A competition on the
classification of medieval handwriting in Latin script was organized in
(Cloppet et al., 2018). The top performing technique obtained an accuracy of
$83.9\%$ among $7$ methods, submitted in the competition. One recent approach
by (Tensmeyer et al., 2017) presents a simple framework based on CNNs. Their
method achieved a state-of-the art performance on challenging data-set of $40$
Arabic computer fonts with $98.8\%$ line level accuracy.

Multi-task Learning (MTL) exploits the task relatedness scenario by learning
the common information that is shared between multiple related tasks and
promotes sharing of model parameters to exploit the shared information across
multiple tasks. The primary issue in MTL setting is to appropriately learn the
relation between the tasks (Happy et al., ) otherwise can lead to negative
performance. Several techniques such as grouped multiple tasks (Kang et al.,
2011), asymmetric MTL (Lee et al., 2016), multi-linear relationship networks
(Long et al., 2017), class relationship (Wu et al., 2014), joint dynamic
weighted (Das et al., 2018) has been proposed in the literature. From the
literature it can be concluded that enforcing MTL to a scenario is challenging
as it depends on the task behavior and the context of MTL based document
analysis has not yet been much explored which encourages us to explore it in
detail.

##  3 Dataset

We have used _L3iTextCopies_ data-set (Eskenazi et al., 2015). This data-set
is consisting of clean, text-only, typewritten documents which has $22$ actual
pages. These pages has following characteristics: $1$ page of a scientific
article with a single column header and a double column body, $3$ pages of
scientific articles with a double column layout, $2$ pages of programming code
with a single column layout, $4$ pages of a novel with a single column layout,
$2$ pages of legal texts with a single column layout, $4$ pages of invoices
with a single column layout, $4$ pages of payslips with a single column
layout, $2$ pages of birth extract with a single column layout. Several
variants of these $22$ pages are created by combining $6$ fonts: _Arial_ ,
_Calibri_ , _Courier_ , _Times New Roman_ , _Trebuchet_ , _Verdana_ ; $3$ font
sizes: $8$, $10$ and $12$ points; $4$ emphasis: _normal_ , _bold_ , and the
combination of _bold_ and _italic_ which makes the total data-set size of
$1584$ documents. Then these documents were printed by three printers (_Konica
Minolta Bizhub 223_ , _Sharp MX M904_ and _Sharp MX M850_) then these ones
were scanned by three scanners at different resolutions between
$150\leavevmode\nobreak\ dpi$, $300\leavevmode\nobreak\ dpi$ and
$600\leavevmode\nobreak\ dpi$. Which finally generates a complete data-set of
total $42768$ document images. To obtain the word images, we apply _Tesseract_
OCR111https://github.com/tesseract-ocr/ to detect the word boundaries and then
these ones are cropped from all the document images. To avoid the noisy
elements, we only have considered the word images, more than $15\times 15$
pixels in dimension. Whereas, to get the patches from a whole image, we crop
patches of window size (standard input image size of ResNet) $224\times 224$
pixels by sliding the window by $112\times 112$ in horizontal and vertical
directions.

##  4 Proposed Methods

In this section, we have explained the architectures of our proposed technique
for document attributes classification.As the base architecture, we propose a
_ResNet50_ (He et al., 2016) based model to perform “single task learning”
i.e. “Font Emphasis”, “Font Type”, “Font Size”, “Scanning Resolution” tasks
separately and independently (see Figure 1(a)). A pre-trained (trained on
_ImageNet_ data-set) _ResNet50_ model is used here and input images normalized
in the same way, i.e. mini-batches of $3$-channel RGB images of shape
($3\times H\times W$), where $H$ and $W$ are taken as $224$. The images are
loaded in a range of $[0,1]$ and then normalized using _mean_ =
$[0.485,0.456,0.406]$ and _std_ = $[0.229,0.224,0.225]$. Hence, we obtain a
pre-trained feature of size $2048$ (see Table. 1 in He et al. (2016)) from
_conv5_x_ layer.

After obtaining the $2048$ features, we add two subsequent fully connected
layers :  FC ($2048\rightarrow 512$) (notation like this represents that it is
a fully connected layer which has $2048$ input nodes and $512$ output nodes)
and  FC ($512\rightarrow 256$). Then, we add a _batch normalization (BN)_
layer which is followed by:  FC ($256\rightarrow 256$). Then we again add
another BN layer which is finally connected to one output head (among $4$
individual output heads). These heads are dedicated for each individual
document attribute related tasks i.e. _Font Emphasis_ , _Font Type_ , _Font
Size_ and _Scanning Resolution_. Each of these heads takes an input of $256$
values and provides an output of $\gamma$ values; where $\gamma=4$ for _Font
Emphasis_ task, $\gamma=6$ for _Font Type_ task, $\gamma=3$ for _Font size_
task and $\gamma=3$ for _Scanning Resolution_ task (see Figure 1(a)). We train
and test this network with either segmented word images or the patches and try
to classify either each word images or each patches.

###  4.1 Multi Task Learning (MTL)

As the first MTL architecture, we propose a multi task learning network where
these four tasks can be performed using a single network (see Fig. 1(b)). The
only difference of this architecture, compared to the previous one is that
here last FC layer is finally connected to four individual heads (instead of
one head in the previous network, shown in Fig.1(a)).

###  4.2 Multi Task and Multi Instance (MI) Learning

As the second MTL based network, we propose to perform a linear combination of
the segmented word based MTL network with the segmented patch based MTL
network (see Fig. 2). In general, a _MTL+MI_ based methods leverage the
information which can be learned from other related tasks and it learns a
general representation from all the available tasks.

(a)

(b)

Fig. 1: (a) The “Single Tasks Learning” based architecture. (b)The “Multi
Tasks Learning” based architecture for document attribute classification.

To combine two _MTL based networks_ i.e. _MTL network_ for word images and
_MTL network_ for patches, the proposed architecture (see Fig. 2) remains same
until last BN layer of the previous _MTL network_ (this block or portion of
network is named as _“Word or Patch block before bottleneck layer”_ in Fig.
1(b)). The last BN layer has $256$ output nodes which are connected to another
FC layer of $256$ nodes for both the word and patch level networks. Now the
block of $256$ output nodes (for both the patch and word networks) are copied
into $4$ heads (i.e. FC layer of word network as well as the patch network are
separately connected to $4$ heads), where each head is consisting of  FC
($256\rightarrow 256$)-RELU-BN. Then the $1^{st}$ head of the word network is
concatenated to the $1^{st}$ head of patch network. Same operation is
performed for $2^{nd}$, $3^{rd}$ and $4^{th}$ heads. By each concatenation
operation, we concatenate the $256$ nodes of word network and $256$ nodes of
patch network to obtain total $512$ nodes. Now each of these $4$ group of
concatenated $512$ nodes are connected to $4$ individual and independent head
of  FC ($512\rightarrow 512$). Which are then finally connected to $4$
individual and independent blocks of  FC ($512\rightarrow 256$)-RELU-BN-
DropOut. Finally, each head is connected to a  FC ($256\rightarrow\gamma$),
followed by SoftMax activation, where $\gamma$ represents the number of output
nodes, dedicated to each individual document attributes classifications i.e.
for _Font Emphasis_ , _Font Type_ , _Font Size_ and _Scanning Resolution_
tasks. The objective of this combined architecture is to get benefited from
multi instance learning i.e. to train by using both the word and patch images
together. Hence, the architecture can get benefited from the equal
participation of both the words and patch features together.

###  4.3 Dynamic Weighed Multi Task & Multi Instance Learning

In the previous architecture (see Section 4.2), we have applied equal weights
(or equal participation) to both the words and patch features. But it could
also be possible that due to this equal weighted fusion, instead of improving,
the accuracy may get decreased. The most probable reason of this kind of
problem is the unequal influence of one instance (either words or patches),
compared to other. Hence, to avoid this problem, we propose to perform multi
instance learning, based on automatically calculated weights which are applied
to both the instances. The architecture is shown in Fig. 3.

Fig. 2: The MTL + MI based architecture

After obtaining $2048$ pre-trained features from word and patch images, we
pass them through  FC ($2048\rightarrow 512$). Then each of these feature set
($512$ features) is passed through several FC layers like :  FC
($512\rightarrow 256$)-RELU-BN;  FC ($256\rightarrow 128$)-RELU-BN;  FC
($128\rightarrow 64$)-RELU-BN;  FC ($64\rightarrow 32$)-RELU-BN;  FC
($32\rightarrow 16$)-RELU-BN; Then $16$ output features of word (as well as
patch) images are get connected to $4$ output heads (see “Word Multi-tasking
block” at the left and “Patch Multi-tasking block” at the right of Fig. 3) The
weights are automatically computed by initially combining $16$ features of
“word” network and $16$ features of “patch” network together. Then these $32$
combined features are individually connected to $4$ heads of  FC
($32\rightarrow 16$)-RELU-BN-SoftMax; As it is visible from the Fig. 3 that
each output head is outputting $2$ weight values. The $1^{st}$ weight value is
dedicated for the output of “word multi-tasking block” and the $2^{nd}$ weight
value is dedicated for “patch multi-tasking block”.

Fig. 3: The weighted MTL + MI based architecture

After obtaining two weight values from each output heads, these ones are
multiplied with the outputs of each output heads of “word multi-tasking block”
as well as “patch multi-tasking block”. For example, the $3$ outputs of
$1^{st}$ output head (dedicated for “Font Emphasis” task) of “word multi-
tasking block” are get multiplied with the $1^{st}$ weight value, coming from
the $1^{st}$ head of “weight computation block”. In the same manner, the
$2^{nd}$ weight from the $1^{st}$ head of “weight computation block” got
multiplied with the $3$ outputs of $1^{st}$ output head (dedicated for “Font
Emphasis” task) of “patch multi-tasking block”. Finally, the pair of $3$
output values are element-wise averaged to generate $3$ final output values,
dedicated for “font emphasis” task. In the same manner, we perform the
weighted average of other three tasks i.e. _Font Type_ , _Font Size_ and
_Scanning Resolution_ tasks to generate the final averaged outputs.

###  4.4 Calculation of Loss in Multi Task & Multi Instance Learning

In this section, we will discuss about the loss function of all the above
defined $4$ different architectures.
a) Single Task Learning Loss: In the case of Single Task Learning (mentioned
in Section 4), the cross entropy loss $\mathcal{L}_{STL}$ is defined as
follows :

$\mathcal{L}_{STL}(\mathbf{X};\Theta)=\sum_{k=1}^{K}-y_{k}\leavevmode\nobreak\
logP(y_{k}|\mathbf{X},\Theta)$ (1)

where $K$ is the total number of output classes, $\mathbf{X}$ and $\Theta$ are
the input and parameters of the network, $y_{k}$ is the true label of input
$\mathbf{X}$ and $P(y_{k}|\mathbf{X},\Theta)$ is the predicted probability of
class $k$ for the given image $\mathbf{X}$ and parameter $\Theta$. As
mentioned in Section 4 that the input could be either segmented word images or
patch images whereas the network could be trained for the classification of
any of these $4$ attributes i.e. for _Font Emphasis_ , _Font Type_ , _Font
Size_ and _Scanning Resolution_ tasks.

b) Multi-Task Learning Loss: Compared to single-task learning, multi-task
learning can gain better performance by jointly learning different tasks.
Multi-task learning is an optimization problem for multiple objectives. The
loss function of multi-task learning (mentioned in Section 4.1) is calculated
by summing up the different tasks in the following manner.

$\mathcal{L}_{MTL}(\mathbf{X};\Theta)=\sum_{t=1}^{T}\mathcal{L}_{t}(\mathbf{X};\Theta_{t})$
(2)

where $T$ is the number of supervised tasks for which we would like to train
in the MTL mode, where the loss of each task
$\mathcal{L}_{t}(\mathbf{X};\Theta_{t})$ is calculated by using same formula
as in Equation 1, $X$ is input of the model and $\Theta=\\{\Theta_{t}\\}$ are
the parameters of the portion of the network, related to each tasks. In our
case, we have $4$ independent tasks i.e. _Font Emphasis_ , _Font Type_ , _Font
Size_ and _Scanning Resolution_ tasks, hence $T=4$.

c) Multi-Task and Multi-Instance Learning Loss: For the case of Multi Instance
and Multi-Task learning (see Section 4.2), the loss function is calculated in
the following manner. Each word image sample, denoted by
$\mathbf{X}_{i}^{word}$ and the patch image sample, denoted by
$\mathbf{X}_{i}^{patch}$, we have $T$ number of label information, where $T$
is the total number of existing tasks. Thus all the tasks, learn together
their features in a joint feature space $f\in\mathcal{F}$ through learning the
weights for individual tasks. The learned features, before the concatenation
operation in Fig. 2 can be represented by $\mathcal{Z}_{t}^{{word}_{256}}$,
$\mathcal{Z}_{t}^{{patch}_{256}}$ as follows:

$\mathcal{Z}_{t}^{{word}_{256}}=\mathcal{F}(\mathbf{X}^{word},\Theta^{word}_{t});\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\
\mathcal{Z}_{t}^{{patch}_{256}}=\mathcal{F}(\mathbf{X}^{patch},\Theta^{patch}_{t})$

(3)

where $\Theta^{word}_{t}$ and $\Theta^{patch}_{t}$ are the parameters of the
word branch and patch branch of each task before the concatenation. Then these
feature vectors i.e. $\mathcal{Z}_{t}^{{word}_{256}}$,
$\mathcal{Z}_{t}^{{patch}_{256}}$ are concatenated in the following manner:

$\mathcal{Z}_{t}^{{concat}_{512}}=Concat(\mathcal{Z}_{t}^{{word}_{256}},\mathcal{Z}_{t}^{{patch}_{256}})$
(4)

where $t\in{1,..,T}$. After that the MTL loss function is calculated in the
same manner as :

$\mathcal{L}_{MTL+MI}^{concat}([\mathbf{X}^{word},\mathbf{X}^{patch}];\Theta)=\sum_{t=1}^{T}\mathcal{L}_{t}(\mathcal{Z}_{t}^{{concat}_{512}};\Theta_{t})$
(5)

where $\Theta_{t}$ is the parameter of the branch of each task after the
concatenation of features.
d) Dynamic Weighed Multi-Task and Multi-Instance Learning Loss: Rather than
applying equal weights to both the words and patch features, the dynamic
weighted version of Multi Instance and Multi-Task learning considers the
different contribution of learned words and patch features by weighting the
output of word instance $y_{k}^{{word}_{t}}$ and the output of patch instance
$y_{k}^{{patch}_{t}}$ to formulate the final output value
$\mathcal{Y}_{t,k}^{{Avg}}$ as described in Section 4.3:

$\mathcal{Y}_{t,k}^{{Avg}}=Elementwise\\_Avg(w^{1}_{t}(\Psi)\times\leavevmode\nobreak\
y_{k}^{{word}_{t}},w^{2}_{t}(\Psi)\times\leavevmode\nobreak\
y_{k}^{{patch}_{t}})$

where ${w^{1}_{t},w^{2}_{t}}$ are the weights, associated to the output of
each instances i.e. the words ($y_{k}^{{word}_{t}}$) and patch
($y_{k}^{{patch}_{t}}$) respectively. These weights are calculated dynamically
during training process. The dynamic weights i.e.
$\\{\alpha_{i}=w_{i}(\Psi)\\}$ are learned automatically by the dynamic
weights learning module. Particularly, since the dynamic task weights
$\\{\alpha_{i}\\}$ are the outputs of the softmax layer, $\sum\alpha_{i}=1$.
$\Psi$ are the parameters of the dynamic weights learning module. Note that
$\Psi\not\subset\Theta$. The parameters of the dynamic weights learning module
$\Psi$ and the parameters of the network $\Theta$ are optimized simultaneously
with the total loss $\mathcal{L}_{MTL+MI}^{weighted}$ given by:

$\mathcal{L}_{MTL+MI}^{weighted}=\sum_{t=1}^{T}\mathcal{L}_{MTL+MI}^{t}([\mathbf{X}^{word},\mathbf{X}^{patch}];\Theta)$
(6)

where $\mathcal{L}^{t}_{MTL+MI}$ is the loss of each task with the dynamic
weighted output $\mathcal{Y}_{t,k}^{{Avg}}$:

$\mathcal{L}^{t}_{MTL+MI}([\mathbf{X}^{word},\mathbf{X}^{patch}];\Theta)=\sum_{k=1}^{K}-\mathcal{Y}_{t,k}^{Avg}\leavevmode\nobreak\
logP(\mathcal{Y}_{t,k}^{Avg}|\mathbf{X},\Theta)$

(7)

###  4.5 Component based majority voting for entire document image
classification

The training, validation and testing are performed by using either word images
or the patch images. The classification of an entire document image is done by
considering a voting mechanism, where the posterior probabilities of all the
word image (or patches) are taken into account to compute the mean posterior
probability of each document page. Such a mechanism is shown in Fig. 4, where
the word images, obtained from each page are arranged row wise and classes are
arranged along columns. The posterior probabilities of each word to belong
into a specific class is noted in each cells. Hence, by taking column wise
mean followed by the maximum of these mean values, we can decide the class of
complete document image. The same principal is applicable by considering the
patches for the classification of complete document image. Let’s say
$P(y_{k}^{i}|\mathbf{X}_{i}^{p},\Theta)$ is the probability to belong in
$k^{th}$ (here in this example, shown in Fig. 4, $k\in{1,...,6}$) class for
the $i^{th}$ word, taken from $p^{th}$ document page of the data-set.

Fig. 4: The linear combination of multi-tasking Res-Net of word images and
multi-tasking Res-Net of patch images.

The average probability of all the words (belongs to $p^{th}$ document page)
for each class is calculated by:

$\bar{\mathcal{P}}=\frac{1}{n}\left(\sum_{i=1}^{n}P(y_{k}^{i}|\mathbf{X}_{i}^{p}\right);\leavevmode\nobreak\
i\in{1,..,n}$ (8)

where there are $n$ number of words exists in $p^{th}$ document page. Now the
document page is belongs to which class is decided by computing the maximum of
$\bar{\mathcal{P}}$ in the following manner :

$\mathcal{P}_{max}=max\left(\bar{\mathcal{P}}\right)$ (9)

Thanks to our proposed voting mechanism, even if the predicted posterior
probabilities of one or multiple components for the correct class is weaker
than the other classes, thanks to the mean based voting scheme, the high
posterior probabilities of the remaining components of full page image will
compensate and finally the mean accuracy of the true class will be superior.

##  5 Experiments

In this section, we explain our experimental setting and the results obtained
are discussed with detailed analysis. The implementation details of all the
above mentioned networks are mentioned in Section S.1 of supplementary
materials.

###  5.1 Results and discussion

In this section, we have discussed the results of different aforementioned
proposed networks. As mentioned in Section 3 that out of $42,768$ images,
$70\%$ images ($29,950$) are taken for training and $10\%$ images ($4278$) are
taken for validation. Hence, by considering only valid word images (see
Section 3), the training data set is consisting of $8,099,561$ images whereas
the validation set is consisting of $1,344,016$ word images. Whereas, by
considering only valid patches, the training data set is consisting of
$7,261,574$ images whereas the validation set is consisting of $1,309,966$
word images. Furthermore, the test set is consisting of $20\%$ of the total
images i.e. $8557$ images from which we could extract $5,990,78$ word image
and $1,324,58$ patch images.

Table 1: Testing accuracy of STL & MTL based network for word and patch level images  | Top-1 accuracy
---|---
|  Font Type |  Font Size |  Font Emphasis |  Scanning Resolution
STL Word |  0.8508 |  0.9095 |  0.9413 |  0.9933
STL Patch |  0.9452 |  0.9661 |  0.9780 |  0.9963
MTL Word |  0.8887 |  0.9137 |  0.9379 |  0.9939
MTL Patch |  0.9512 |  0.9760 |  0.9827 |  0.9965
Table 2: Training (gray colored row), validation (light blue colored row) and testing (orange colored row) accuracies of MTL based network for word and patch level images  |  Font Type |  Font Size |  Font Emphasis |  Scanning Resolution
---|---|---|---|---
Word Level with multiple FC layers |  0.9703 |  0.8917 |  0.9557 |  0.9828
|  0.6073 |  0.6449 |  0.7042 |  0.8431
|  0.6986 |  0.6933 |  0.8359 |  0.9386
Word Level with AlexNet layers |  0.4094 |  0.4199 |  0.5860 |  0.8642
|  0.3447 |  0.4013 |  0.5203 |  0.7677
|  0.3962 |  0.3907 |  0.5992 |  0.8328
Patch Level with multiple FC layers |  0.8631 |  0.8999 |  0.9131 |  0.9933
|  0.7751 |  0.8544 |  0.9501 |  0.9979
|  0.7777 |  0.8563 |  0.9421 |  0.9899
Patch Level with AlexNet layers |  0.3688 |  0.5932 |  0.6368 |  0.9444
|  0.3802 |  0.6167 |  0.7870 |  0.9806
|  0.3600 |  0.5957 |  0.8074 |  0.9562

####  5.1.1 Accuracies of STL & MTL based networks

The testing accuracy of STL based network (see Section 4) and MTL based
network (see Section 4.1) in shown in Table 1, where it can be visible that
the patch level accuracy is better than the word level for all the document
attribute classifications i.e. for _font type_ , _font size_ , _font emphasis_
and _scanning resolution_. It can also be seen that in the case of word and
patch level classification, the Multi-task learning has performed better than
Single-task learning, which was expected. The training accuracies of MTL based
word level as well as patch level networks are shown in Fig. 5 (in the same
manner, the training and validation accuracies of MTL based word and patch
level networks are shown in Fig. 6 and are discussed in Section S.2). It can
be visible that the accuracies of all the $4$ tasks increases with the
iteration of each epochs and patch level network has performed better than
word level network.

(a)

Fig. 5: Training accuracies of word based and and patch based MTL networks for
“Font type”, “Font size”, “Font emphasis” and “Scanning resolution” tasks.

####  5.1.2 Small data-set formation

To test the accuracy of following two architectures i.e. “Multi-task and
Multi-instance Learning” and “Weighted Multi-task and Multi-instance
Learning”, we choose only a part of the complete data-set in the following
manner to obtain faster results and to test various architectural
modifications. The document images (also the cropped word and patch images) in
the data-set are actually labeled (i.e. the ground truth) in one out of $216$
(by combining all the output classes of the four tasks i.e. $4\times 6\times
3\times 3=216$) independent classes. We choose $400$ word & patch images for
training, $100$ word & patch images for validation and $150$ word & patch
images for testing from each of the $216$ classes222if there are not enough
number of word and/or patch images exists in any of these $216$ classes then
we first count total number of word (say $c_{word}$) and patch (say
$c_{patch}$) images in this class and then choose $c_{patch}$ numbers of word
and patch images if $c_{patch}<c_{word}$ or vice-versa.. These limited number
of images are chosen by calculating total number of foreground pixels (we have
applied adaptive threshold based binarization using Otsu’s method) in the
image followed by choosing the images which has high number of foreground
pixels. Hence in this manner, we choose $83,389$ word and $83,389$ patch
images for training and $20,731$ word and $20,731$ patch images for
validation.

Table 3: Training, validation and testing accuracies of MTL and MI based network by combining word and patch level images  |  Font Type |  Font Size |  Font Emphasis |  Scanning Resolution
---|---|---|---|---
Late concat multiple FC layers |  0.9540 |  0.9611 |  0.9766 |  0.9973
|  0.1859 |  0.4090 |  0.6757 |  0.9334
|  0.1885 |  0.4165 |  0.7217 |  0.7759
Early concat multiple FC layers |  0.9680 |  0.9606 |  0.9792 |  0.9975
|  0.5287 |  0.5995 |  0.6228 |  0.9740
|  0.4918 |  0.4805 |  0.6318 |  0.8147
Early concat AlexNet like Conv. layers |  0.2929 |  0.6037 |  0.5895 |  0.9754
|  0.2221 |  0.3756 |  0.5120 |  0.9438
|  0.2197 |  0.3076 |  0.4945 |  0.7998

####  5.1.3 Accuracy of Multi-task and Multi-instance Learning

The first test is performed by using the architecture shown in Fig. 2, where
the fusion of convoluted features from word images and patch images are
performed lately (i.e. the $2048$ number of pre-trained “ResNet-50” are passed
through several block of FC feed forward networks to finally generate $256$
number of features). The results of this network is shown in Table 3 and are
mentioned in first row as “Late concat multiple FC layers”. Although, we have
obtained high training accuracies for all the $4$ tasks but the validation and
testing accuracies got drastically decreased for all the tasks (whereas the
“scanning resolution” task shows comparatively better performance than other
three tasks). This is a classical case of “model over-fitting” where the model
is able to learn very well on training data but it fails to perform on
validation and testing data.

We tried several approaches like increasing and decreasing the percentage of
dropout nodes, completely removing all dropout layers, adding more dropout
layers at every block of FC layers, removing all the batch normalization
layers from the network etc. to overcome this problem. All these trials
doesn’t helped much and the problem of over fitting sustained. We suspected
that the late concatenation of patch and word level features could be the
possible reason for this over-fitting problem. Hence, we decided to concat the
$2048$ number of pre-trained “ResNet-50” features at the early stage of the
network. Hence, we designed another network, where $2048$ number of pre-
trained “ResNet-50” features of word and patch images are concatenated at the
early stage to generate $4096$ features. The architectural details of this
network is mentioned in Section S.4.1 of the supplementary material.

By using this network, although the validation and testing accuracies has
highly improved but still there remains a strong gap between the training
accuracies (shows unconventionally very high) and testing/validation
accuracies of the $4$ tasks. By suspecting that the pipeline connection of
several FC layers could be the possible reason of this gap between the
training and testing/validation accuracies of the $4$ tasks, we tried to
reduce the number of FC layers. This attempt doesn’t helped much and the
problem of “model over fitting” still remained. Please look at Table S7 and
corresponding discussion in supplementary materials for more details.
Furthermore to handle this problem of model over-fitting, we proposed to
replace the FC layers by convolutional layers. Here, we adopt the popular
“AlexNet” (Krizhevsky et al., 2017) like architecture instead of multiple FC
layers, after obtaining $2048$ number of pre-trained “ResNet-50” features from
word and patch images. We have named this network as “Early concat AlexNet
like conv. layers” and it’s results are shown in $3^{rd}$ row of Table 3. The
architectural details of this network is mentioned in Section S.4.3.

It can be visible from the results, mentioned in Table 3 that although “Early
concat AlexNet like conv. layers” network is able to somehow overcome the
problem of model over fitting and the difference between training and
validation/testing accuracy got decreased (still for “Font Size” task, this
difference remains significant), the overall accuracy of the network got
significantly reduced. Instead of “AlexNet” like architecture, we have also
tried “VggNet” (He et al., 2016) like architecture in the same manner to see
whether we can obtain higher accuracies and can also overcome the problem of
model “over-fitting”. This trial successfully overcome the problem of model
“over-fitting” but the overall accuracy of the network also got significantly
decreased. Please see Table. S7 and corresponding discussions in supplementary
materials for more details.

Table 4: Accuracies of Dynamic weighted MTL and MI based network by combining word and patch level images  |  Font Type |  Font Size |  Font Emphasis |  Scanning Resolution
---|---|---|---|---
Late concat multiple FC layers |  0.5265 |  0.7786 |  0.8860 |  0.9866
|  0.2976 |  0.4196 |  0.4683 |  0.7484
|  0.2898 |  0.3794 |  0.4544 |  0.7000
Late concat AlexNet like Conv. layers |  0.2854 |  0.5113 |  0.5501 |  0.9427
|  0.2114 |  0.3572 |  0.3575 |  0.8673
|  0.2038 |  0.3190 |  0.3454 |  0.7679

####  5.1.4 Accuracy of Dynamic Weighted Multi-task and Multi-instance
Learning

The network is shown in Fig. 3 and the corresponding details about this
architecture are mentioned in Section 4.3. It can be visible from the
accuracies, mentioned at the $1^{st}$ row in Table 4 that the accuracies are
not get further improved, compared to the “Multi-task and Multi-instance
Learning” architectures and the problem of “model over-fitting” still remains.
Furthermore, to handle the problem of “model over-fitting”, we tried to adopt
“AlexNet” like architecture by replacing the FC layers with several
convolution layers. The details of this network is mentioned in Section SI.1
in the supplementary materials. The results of this network is shown in
$2^{nd}$ row of Table 4. It can be visible from the results that the problem
of model over fitting are seems to get resolved for “Font Type” and “Scanning
Resolution” tasks but for other two tasks i.e. for “Font Size” and “Font
Emphasis” tasks, the model over fitting problem persists. More importantly,
the overall accuracies of these $4$ tasks become lesser than the “Late concat
multiple FC layers” model.

One interesting fact can be noticed from Table.2 that even when we are using
only word level network (see the $1^{st}$ row of Table.2) on the smaller
dataset (see Section. 5.1.2), the issue of “model over fitting” persists. This
problem of “model over fitting” can be overcome by using “AlexNet” like layers
instead of pipe lined FC layers. The results of this network, named as “Word
Level with AlexNet layers” is shown at the $2^{nd}$ row of Table.2, where we
can see the difference between the training and validation/testing accuracies
for these $4$ tasks got reduced. But the overall accuracies of all the $4$
tasks got decreased than the “Only Word Level” network.

The results of patch level network (denoted as “Patch level with multiple FC
layers”) on small dataset, the problem of “model over fitting” is not visible
(see the $3^{rd}$ row of Table.2). Moreover, the accuracies of all the $4$
tasks are quite higher compared with it’s counter part i.e. “Word level with
multiple FC layers” network. We further tested the performance of “AlexNet”
like layers instead of pipe lined FC layers and this network is named as
“Patch level with AlexNet layers”. It can be visible from the results, (see
the $4^{th}$ row of Table.2) that the problem of “model over fitting” is also
disappeared in this case also but the accuracy got decreased, compared to
“Patch level with multiple FC layers” network.

Among our various experiments with “MTL and MI” based network which combines
word and patch level images, the “Early concat multiple FC layers” network
performed better than others. Whereas, the experiments with various weighted
“MTL and MI” based networks, the “Late concat multiple FC layers” network
performed better. But the accuracies of both these networks i.e. “Early concat
multiple FC layers” and “Late concat multiple FC layers” couldn’t outperform
“Patch Level with multiple FC layers” network. Hence, from all these above
mentioned experiments, we can conclude that “Patch Level with multiple FC
layers” network is the best performer. The result of this network on the
complete dataset is mentioned at the $4^{th}$ row in Table 1. Furthermore, the
page level accuracy (evaluated on test dataset only) of “Patch Level with
multiple FC layers” network by accumulating the patch level (results shown for
word level also) posterior probabilities are mentioned in Table. 5. It can be
seen that thanks to our proposed voting scheme, mentioned in Section. 4.5, we
have achieved very high page level accuracies in the case of “Patch Level with
multiple FC layers” network.

Table 5: Page level testing accuracies of MTL based network using word and patch images  |  Font Type |  Font Size |  Font Emphasis |  Scanning Resolution
---|---|---|---|---
Word Images |  0.8879 |  0.9129 |  0.9392 |  0.9935
Patch Images |  0.9987 |  1.00 |  1.00 |  1.00

##  6 Conclusions

In this work, we adhere to explore MTL to perform for $4$ document attribute
classification tasks i.e. “Font Type”, “Font Size”, “Font Emphasis”, “Scanning
Resolution” recognition. In Table. 1, we have shown that “MTL” based networks
has outperformed “STL” based network. We further tried to combine the “word
and patch” images together by proposing several MTL & MI based network and
weighted MTL & MI based networks. But none of them performed better than “MTL”
based network, using patch images as the input. Hence, from all the above
mentioned experiments, we can conclude that the proposed “MTL” based network,
using patch images can attain high accuracy for these $4$ classification
tasks.

## References

  * (1) , .  Tesseract OCR :.  https://github.com/tesseract-ocr/.
  * (2) , .  TextCopies Dataset : .  http://navidomass.univ-lr.fr/TextCopies/.
  * Ben Moussa et al. (2010) Ben Moussa, S., Zahour, A., Benabdelhafid, A., Alimi, A.M., 2010\.  New features using fractal multi-dimensions for generalized Arabic font recognition.  Pattern Recognition Letters 31, 361–371.
  * Cloppet et al. (2018) Cloppet, F., Eglin, V., Helias-Baron, M., Kieu, C., Vincent, N., Stutzmann, D., 2018\.  ICDAR2017 Competition on the Classification of Medieval Handwritings in Latin Script.  Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 1, 1371–1376.
  * Das et al. (2018) Das, A., Dantcheva, A., Bremond, F., 2018.  Mitigating bias in gender, age and ethnicity classification: a multi-task convolution neural network approach, in: Proceedings of the European Conference on Computer Vision (ECCV), pp. 0–0.
  * Eskenazi et al. (2015) Eskenazi, S., Gomez-Krämer, P., Ogier, J.M., 2015.  When document security brings new challenges to document analysis.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 8915, 104–116.
  * (7) Happy, S., Dantcheva, A., Das, A., Bremond, F., Zeghari, R., Robert, P., .  Apathy classification by exploiting task relatedness, in: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)(FG), pp. 733–738.
  * He et al. (2016) He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.  arXiv:1512.03385.
  * Kang et al. (2011) Kang, Z., Grauman, K., Sha, F., 2011.  Learning with whom to share in multi-task feature learning., in: ICML, pp. 521--528.
  * Krizhevsky et al. (2017) Krizhevsky, A., Sutskever, I., Hinton, G.E., 2017.  ImageNet classification with deep convolutional neural networks, in: NIPs, pp. 84--90.
  * Lee et al. (2016) Lee, G., Yang, E., Hwang, S., 2016.  Asymmetric multi-task learning based on task relatedness and loss, in: International Conference on Machine Learning, pp. 230--238.
  * Long et al. (2017) Long, M., Cao, Z., Wang, J., Philip, S.Y., 2017.  Learning multiple tasks with multilinear relationship networks, in: Advances in Neural Information Processing Systems, pp. 1594--1603.
  * Pengcheng et al. (2017) Pengcheng, G., Gang, G., Jiangqin, W., Baogang, W., 2017\.  Chinese calligraphic style representation for recognition.  International Journal on Document Analysis and Recognition 20, 59--68.
  * Shi et al. (2016) Shi, B., Bai, X., Yao, C., 2016.  Script identification in the wild via discriminative convolutional neural network.  Pattern Recognition 52, 448--458.
  * Simard et al. (2003) Simard, P.Y., Steinkraus, D., Platt, J.C., 2003.  Best practices for convolutional neural networks applied to visual document analysis.  Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 2003-Janua, 958--963.
  * Tao et al. (2016) Tao, D., Lin, X., Jin, L., Li, X., 2016.  Principal Component 2-D Long Short-Term Memory for Font Recognition on Single Chinese Characters.  IEEE Transactions on Cybernetics 46, 756--765.
  * Tensmeyer et al. (2017) Tensmeyer, C., Saunders, D., Martinez, T., 2017.  Convolutional Neural Networks for Font Classification.  Proceedings of the International Conference on Document Analysis and Recognition, ICDAR 1, 985--990.  arXiv:1708.03669.
  * Wu et al. (2014) Wu, Z., Jiang, Y.G., Wang, J., Pu, J., Xue, X., 2014\.  Exploring inter-feature and inter-class relationships with deep neural networks for video classification, in: Proceedings of the 22nd ACM international conference on Multimedia, ACM. pp. 167--176.

The paper is under consideration at ”Pattern Recognition Letters” journal

##  Supplementary Materials

###  S.1 Implementation details of various proposed networks

In this section, we describe the implementation details of each of the
networks which are used for the experiments. We train the proposed network
(mentioned in Section 4) by using a pre-trained (trained on _ImageNet_ data-
set) _ResNet50_ model. The model is trained by using stochastic gradient
descent with momentum takes as $0.9$, multiplicative factor of learning rate
decay (denoted by $\gamma$) is taken as $0.1$, step size is taken as $10$, and
initial learning rate is taken as $0.0001$. We use batches of size of $200$
and weight decay $0.0001$. The model is trained for $20000$ iterations per
epoch and the loss function is cross-entropy with a uniform weighing scheme.
We have used GeForce Titan RTX GPUs with 11GB of RAM capacity per card. For
faster execution, we have used $3$ GPU cards in parallel.

####  S.1.1 Multi task network for segmented words or patches

For this case, almost same network parameters are maintained except the batch
size is taken as $800$ for word images and $500$ for patch images (decided
based on the memory capacity per GPU card).

####  S.1.2 Multi task network for combined segmented words and patches

For this case, almost same network parameters are maintained except the batch
size is taken as $100$ for both the multi task and multi instance learning and
as well as for weighted multi task and multi instance learning.

###  S.2 Training and Validation accuracies of MTL networks

In the following Fig.6, we have shown the training and validation accuracies
of MTL based word level and patch level networks. It can be visible that the
accuracies of the $4$ tasks increases with the iteration of each epochs.

(a)

(b)

Fig. 6: Training and validation accuracies of MTL based network for “Font
type”, “Font size”, “Font emphasis” and “Scanning resolution” tasks : (a)
Taking word images as input (b) Taking patch images as input.

###  S.3 Further experiments on MTL based networks

In continuation with the experiments, mentioned in Table. 1, we further has
performed some more experiments. In another similar kind of network, we adopt
the “VggNet” (He et al., 2016) like architecture instead of “AlexNet” like
architecture (the architectural details remain same as the one, mentioned in
Section S.4.3). In this network, we have treated the $2048$ pre-trained
features as $1D$ channel. Then this $1D$ features are passed through the
following $2$ blocks of:  CONV($in_{ch}^{1}\rightarrow
out_{ch}^{1};k^{1}$)-RELU-CONV ($in_{ch}^{2}\rightarrow
out_{ch}^{2};k^{2}$)-RELU-MaxPool(n) The parameters are taken as follows in
these $2$ consequent blocks i.e. for:

$1^{st}$ block:

$in_{ch}^{1}=2$, $out_{ch}^{1}=64$, $k^{1}=3$;

$in_{ch}^{2}=64$, $out_{ch}^{2}=64$, $k^{2}=3$.

$2^{nd}$ block:

$in_{ch}^{1}=64$, $out_{ch}^{1}=128$, $k^{1}=3$

$in_{ch}^{2}=128$, $out_{ch}^{2}=128$, $k^{2}=3$.

Then the output of this $2^{nd}$ block is passed through the following $3$
blocks of:

CONV($in_{ch}^{1}\rightarrow out_{ch}^{1};k^{1}$)-RELU-CONV
($in_{ch}^{2}\rightarrow out_{ch}^{2};k^{2}$)-RELU-CONV
($in_{ch}^{3}\rightarrow out_{ch}^{3};k^{3}$)-RELU-CONV
($in_{ch}^{4}\rightarrow out_{ch}^{4};k^{4}$)-RELU-MaxPool($\mathcal{K}$) The
parameters are taken as follows in these $3$ consequent blocks i.e.

for $3^{rd}$ block:

$in_{ch}^{1}=128$, $out_{ch}^{1}=256$, $k^{1}=3$;

$in_{ch}^{2}=256$, $out_{ch}^{2}=256$, $k^{2}=3$;

$in_{ch}^{3}=256$, $out_{ch}^{3}=256$, $k^{3}=3$;

$in_{ch}^{4}=256$, $out_{ch}^{4}=256$, $k^{4}=3$; $\mathcal{K}=2$.

For the $4^{th}$ block:

$in_{ch}^{1}=256$, $out_{ch}^{1}=512$, $k^{1}=3$;

$in_{ch}^{2}=512$, $out_{ch}^{2}=512$, $k^{2}=3$;

$in_{ch}^{3}=512$, $out_{ch}^{3}=512$, $k^{3}=3$;

$in_{ch}^{4}=512$, $out_{ch}^{4}=512$, $k^{4}=3$; $\mathcal{K}=2$.

For the $5^{th}$ block:

$in_{ch}^{1}=512$, $out_{ch}^{1}=512$, $k^{1}=3$;

$in_{ch}^{2}=512$, $out_{ch}^{2}=512$, $k^{2}=3$;

$in_{ch}^{3}=512$, $out_{ch}^{3}=512$, $k^{3}=3$;

$in_{ch}^{4}=512$, $out_{ch}^{4}=512$, $k^{4}=3$; $\mathcal{K}=2$.

Table S6: Training, validation and testing accuracies of MTL based network for
word and patch level images  Top-1 accuracy of MTL
---
|  Font Type |  Font Size |  Font Emphasis |  Scanning Resolution
Word Level with VggNet layers |  0.1762 |  0.3485 |  0.2522 |  0.3459
|  0.1740 |  0.3482 |  0.2515 |  0.3479
|  0.1751 |  0.3508 |  0.2484 |  0.3507
Patch Level with VggNet layers |  0.1671 |  0.3314 |  0.2519 |  0.3327
|  0.1660 |  0.3319 |  0.2488 |  0.3319
|  0.1677 |  0.3003 |  0.2446 |  0.3693

The output of the $5^{th}$ block is a $2D$ features of size $512\times 64$
which is passed through the “Average Pooling” layer333here we have used
“Adaptive Average Pooling” algorithm from PyTorch library. For more details,
see :
https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_adaptive_avg_pool1d.html
, having a kernel of size $6$. This makes the $2D$ features to get flattened
and reduced into the $1D$ feature of size $3072$. Then $3072$ features are
passed through following layers:  DropOut-FC($(512\times 6)\rightarrow
768$)-RELU-DropOut-FC($768\rightarrow 384$)-RELU-FC($384\rightarrow 192$)
After passing through the above mentioned layers, we obtain $192$ number of
features which are finally connected to $4$ output heads, corresponding to $4$
individual tasks. We have named this network as “Early concat VggNet like
conv. layers” and it’s results are shown in Table S6.

####  S.3.1 Results and Discussion

The results of above described network mentioned in Table. S6, where either
word images (named as “Word Level with VggNet layers”) or patch images (named
as “Patch Level with VggNet layers”) are taken as input to the network. It can
be visible from these results that the frequently arising model over fitting
problem can be clearly resolved in this network but the accuracy of this
network for both types of input i.e. either word images or patch images as
input, shows inferior accuracies compared to the other networks, shown in
Table. 2.

###  S.4 Further experiments on MTL & MI based networks

In continuation with the experiments, mentioned in Section 5.1.3, where we
have tried to replace the late concatenation of patch and word level features
by the early concatenation of these features. In the Section 5.1.3, we have
discussed such a network where the early concatenated $4096$ features are
passed through several liner layers, before getting connected to $4$ output
heads. The detail of this network is mentioned below in Section S.4.1.

####  S.4.1 Early concat multiple FC layers network

In this network, $2048$ number of pre-trained “ResNet-50” features of word and
patch images are concatenated to generate $4096$ features, which are
sequentially passed through several layers like :  FC ($4096\rightarrow
2048$)-RELU-BN;  FC ($2048\rightarrow 1024$)-RELU-BN;  FC ($1024\rightarrow
512$)-RELU-BN;  FC ($512\rightarrow 256$)-RELU-BN;  FC ($256\rightarrow
128$)-RELU-BN;  FC ($128\rightarrow 64$)-RELU-BN;  FC ($64\rightarrow
32$)-RELU-BN;  FC ($32\rightarrow 16$)-RELU-BN and finally get connected to
$4$ output heads. The results of this network are shown at the $2^{nd}$ row of
Table 3 and are named as “Early concat multiple FC layers” network.

Although, by using this network, the validation and testing accuracies got
highly improved (see Table. 3) but still there remains a strong gap between
the training and validation/testing accuracies. We suspected that may be the
pipeline connection of several FC layers could be the possible reason of this
gap between training and validation/testing accuracies of $4$ tasks, hence we
tried to reduce the number of FC layers.

####  S.4.2 Reduced number of FC layers for MTL & MI based network

Consequently, such a network is explained here. In this network, the pre-
trained concatenated $4096$ features are sequentially passed through smaller
number of FC layers such as:  FC ($4096\rightarrow 2048$)-RELU-BN which is
then directly connected to $4$ output heads. This network is named as “Early
concat less FC layers_1”.

We further modified the network by sequentially passing through the
concatenated $4096$ features by $2$ FC layers as :  FC ($4096\rightarrow
2048$)-RELU-BN;  FC ($2048\rightarrow 1024$)-RELU-BN; which then directly
connected to $4$ output heads and we named this network as “Early concat less
FC layers_2”. The results of these two networks are mentioned in $1^{st}$ and
$2^{nd}$ rows of Table S7 respectively. It can be seen from the validation and
testing results of these two networks that the results doesn’t get improved
for all the $4$ tasks and the problem of “model over fitting” still remains.

Table S7: Training, validation and testing accuracies of MTL and MI based
network by combining word and patch level images  Top-1 accuracy of MTL by
combining word and patch images
---
|  Font Type |  Font Size |  Font Emphasis |  Scanning Resolution
Early concat less FC layers_1 |  0.9669 |  0.9642 |  0.9817 |  0.9977
|  0.4865 |  0.5204 |  0.5644 |  0.9692
|  0.5049 |  0.4258 |  0.5631 |  0.8041
Early concat less FC layers_2 |  0.9715 |  0.9714 |  0.9836 |  0.9976
|  0.4976 |  0.4690 |  0.5661 |  0.9739
|  0.4975 |  0.4052 |  0.6156 |  0.7831
Early concat VggNet like Conv. layers |  0.1708 |  0.3457 |  0.2486 |  0.3448
|  0.1723 |  0.3451 |  0.2497 |  0.3452
|  0.1705 |  0.3683 |  0.2450 |  0.3763

####  S.4.3 AlexNet like convolution layers for MTL & MI based network

To handle the problem of “model over fitting”, in Section 5.1.3, we proposed
to replace the FC layers by convolutional layers, e.g. adapting “AlexNet” like
architecture. We have treated the $2048$ pre-trained features of patch and
$2048$ pre-trained features of word images as two channels of $1D$ vector.
Then these two channels are given as input and are passed through the
following $5$ blocks of: CONV ($in_{ch}\rightarrow out_{ch};k$)-BN-RELU; where
the parameters i.e. $in_{ch}$ represents number of input channels, $out_{ch}$
represents number of output channels and $k$ represents the kernel size. The
parameters are taken as follows in these $5$ consequent blocks i.e. for:

$1^{st}$ block: $in_{ch}=2$, $out_{ch}=64$, $k=11$;

$2^{nd}$ block: $in_{ch}=64$, $out_{ch}=192$, $k=5$;

$3^{rd}$ block: $in_{ch}=192$, $out_{ch}=384$, $k=3$;

$4^{th}$ block: $in_{ch}=384$, $out_{ch}=256$, $k=3$;

$5^{th}$ block: $in_{ch}=256$, $out_{ch}=256$, $k=3$;

Finally, the output of $5^{th}$ block ($192$ features) are finally get
connected to $4$ output heads.

####  S.4.4 VggNet like convolution layers for MTL & MI based network

In another similar kind of network, we adopt the “VggNet” (He et al., 2016)
like architecture instead of “AlexNet” like architecture. In this network
also, we have treated the $2048$ pre-trained features of patch and $2048$ pre-
trained features of word images as two $1D$ channels. Then these two channels
of features are passed through the following $2$ blocks of:
CONV($in_{ch}^{1}\rightarrow out_{ch}^{1};k^{1}$)-RELU-CONV
($in_{ch}^{2}\rightarrow out_{ch}^{2};k^{2}$)-RELU-MaxPool(n) Then the output
of this $2^{nd}$ block is passed through the following $3$ blocks of:

CONV($in_{ch}^{1}\rightarrow out_{ch}^{1};k^{1}$)-RELU-CONV
($in_{ch}^{2}\rightarrow out_{ch}^{2};k^{2}$)-RELU-CONV
($in_{ch}^{3}\rightarrow out_{ch}^{3};k^{3}$)-RELU-CONV
($in_{ch}^{4}\rightarrow out_{ch}^{4};k^{4}$)-RELU-MaxPool($\mathcal{K}$)

Table S8: Training, validation and testing accuracies of MTL based network by
combining normal patch and noisy patch level images  Top-1 accuracy of MTL by
combining normal patch and noisy patch images
---
|  Font Type |  Font Size |  Font Emphasis |  Scanning Resolution
Early concat AlexNet like Conv. layers |  0.2995 |  0.5348 |  0.5389 |  0.9667
|  0.2883 |  0.5312 |  0.6386 |  0.9914
|  0.2592 |  0.5067 |  0.6422 |  0.9691
Early concat VggNet like Conv. layers |  0.1740 |  0.3457 |  0.2521 |  0.3452
|  0.1748 |  0.3495 |  0.2524 |  0.3495
|  0.1713 |  0.3695 |  0.2435 |  0.3762

The parameters of these $5$ blocks remain same as it is mentioned in Section
S.3. In the same fashion, the output of the $5^{th}$ block is a $2D$ features
of $512\times 64$ dimension which is passed through the “Average Pooling”
layer, having a kernel of size $6$. This makes the $2D$ features to get
flattened and reduced into the $1D$ feature of size $3072$. Then $3072$
features are passed through following FC layers:  DropOut-FC($(512\times
6)\rightarrow 768$)-RELU-DropOut-FC($768\rightarrow 384$)-RELU-
FC($384\rightarrow 192$) After passing through the above mentioned layers, we
obtain $192$ number of features which are finally connected to $4$ output
heads, corresponding to $4$ individual tasks. We have named this network as
“Early concat VggNet like conv. layers” and the results are shown in $3^{rd}$
row of Table S7.

It can be visible from results that the annoying problem of “model over
fitting” is also resolved by this network but the overall accuracies of all
the $4$ tasks are inferior to it’s counterpart i.e. “Early concat AlexNet like
Conv. layers” network (see Table. 3).

####  S.4.5 Using noisy patch as input in MTL & MI based network

As it is visible in Table. 3 and are mentioned in Section 5.1.3 that we get
highly annoyed by “model over fitting” problem when we tried to combine word
and patch images together as the inputs in proposed three networks i.e. “Late
concat multiple FC layers”, “Early concat multiple FC layers”, “Early concat
AlexNet like Conv. layers”. We have tried several tricks and strategies to
overcome the problem of “model over fitting”, which are mentioned in Section
5.1.3 and Section S.4.2 and S.4.4. But none of these proposed network
architecture could outperform the results of MTL based networks, which uses
either word images (“Word Level with multiple FC layers” network) or patch
images (“Patch Level with multiple FC layers” network) shown in Table. 2. It
can also be seen from Table. 2 that “Patch Level with multiple FC layers”
network has shown superior accuracy and has overcome the “model over fitting”
problem, compared to “Word Level with multiple FC layers” network. Which
inherently imply that word images as the input may be the possible offender
for this “model over fitting” problem as well as the reason of decline in
accuracy of the results, shown in Table. 3 and Table.S7.

Hence, in the following setup, we have replaced cropped word images by noisy
patch images (get inspired from “denoising auto encoder” 444 I. Goodfellow, Y.
Bengio, A. Courville, Deep Learning (2016), The MIT Press). Hence, we tested
the performance of “Early concat AlexNet like Conv. layers” network (described
in $2^{nd}$ last paragraph in Section 5.1.3) and “Early concat VggNet like
Conv. layers” network (described in Section S.4.4) by considering patch images
as the $1^{st}$ input and noisy version of the same patch images (we have
applied standard Gaussian
noise555https://gist.github.com/Prasad9/28f6a2df8e8d463c6ddd040f4f6a028a#gistcomment-2857098
of mean=$0$ and standard deviation = $\sqrt{0}.1$) as the $2^{nd}$ input of
the network. The data is partially corrupted by noises and are added to the
input vector in a stochastic manner. Then the network is trained to correctly
classify even if it is trained with noisy images.

The results of these networks are shown in Table.S8. It can be visible from
the results that the problem of “model over fitting” is resolved here also.
Moreover, the results of “Early concat AlexNet like Conv. layers” network got
highly improved in comparison with it’s counterpart in Table. 3. Whereas, the
results of “Early concat VggNet like Conv. layers” network from Table.S8
remains same as it’s counter part, mentioned in Table. S7. Most probable
reason could be the inherent architecture of “VggNet”, which is able to
overcome the “model over fitting” problem, irrespective of word images (which
is the most probable offender for “model over fitting” problem) as input to
the network. But in particular, both of these networks couldn’t outperform the
“Patch Level with multiple FC layers” network, mentioend in Table. 2.

Table S9: Training, validation and testing accuracies of weighted MTL based
network by combining normal patch and noisy patch level images  Top-1 accuracy
of weighted MTL by combining word and patch level images
---
|  Font Type |  Font Size |  Font Emphasis |  Scanning Resolution
Late concat multiple FC layers |  0.5025 |  0.7604 |  0.8432 |  0.9905
|  0.4469 |  0.6948 |  0.8364 |  0.9884
|  0.3017 |  0.4366 |  0.5966 |  0.7374
Late concat AlexNet like Conv. layers |  0.2946 |  0.5657 |  0.5791 |  0.9823
|  0.2614 |  0.5224 |  0.6412 |  0.9892
|  0.2012 |  0.4742 |  0.4517 |  0.9302

##  SI Further experiments on weighted MTL & MI based learning

In the following section, we have explained further details about several
experiments regarding weighted MTL & MI based learning.

###  SI.1 AlexNet like Weighted Multi-task and Multi-instance Learning

Here we have mentioned the details about “Late concat AlexNet like Conv.
layers” network, whose results are given in Table. 4. After obtaining $2048$
pretrained “ResNet-50” features from word and patch images, we pass them
through a FC layer of:  FC ($2048\rightarrow 512$) to reduce the dimension of
feature. After obtaining $512$ features of word image network and $512$
features of patch image network, we treat each of them as a channel of $1D$
vector. Then each of these channel are passed through the $5$ blocks of:  CONV
($in_{ch}\rightarrow out_{ch};k$)-BN-RELU; where the parameters i.e. $in_{ch}$
represents number of input channels, $out_{ch}$ represents number of output
channels and $k$ represents the kernel size. The parameters are taken as
before like “Late concat AlexNet like conv. layers” network, mentioned in
Section 5.1.3.

After passing through the $5^{th}$ block, we obtain the $2D$ features of
$256\times 124$ dimension. This $2D$ feature is passed through the “Average
Pooling” layer (having kernel of size $6$), which scale down the feature into
of dimension $256\times 6$. Then this reduced dimensional feature is
flatten666we have used “nn.Flatten()” function of PyTorch library to flatten
the feature. For more details, please see :
https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html to get $1D$
features of size $1536$ which is then passed through 3 blocks of following
layers:  DropOut($\mathcal{P}=50$)-FC ($1536\rightarrow 768$)-RELU;
DropOut($\mathcal{P}=50$)-FC ($768\rightarrow 384$)-RELU;  FC ($384\rightarrow
192$); After passing through these $3$ blocks, we finally obtain $192$
features from the word network (using word images only) as well as from the
patch network (using patch image only). The $192$ output features of word
network get connected to $4$ output heads (for reference, see “Word Multi-
tasking block” at the left of Fig. 3). Another set of $192$ features from
patch images are also get connected to $4$ output heads (for reference, see
“Patch Multi-tasking block” at the right of Fig. 3). The weights are
automatically computed in the same manner by initially combining $192$
features of “word” network and $192$ features of “patch” network together.
Then these $384$ number of combined features are individually connected to $4$
heads of:  FC ($384\rightarrow 192$)-RELU;  FC ($192\rightarrow 96$)-RELU;  FC
($96\rightarrow 48$)-RELU;  FC ($48\rightarrow 24$)-RELU;  FC ($24\rightarrow
2$)-RELU (take reference from the Fig. 3 where you can see that each output
head is outputting $2$ weight values). Hence, from each output head, we can
get two weight values where the $1^{st}$ weight value is dedicated for the
output of “word multi-tasking block” and the $2^{nd}$ weight value is
dedicated for “patch multi-tasking block”. After obtaining two weight values
from each output head, dedicated to _Font Emphasis_ , _Font Type_ , _Font
Size_ and _Scanning Resolution_ tasks, these ones are multiplied and averaged
with the outputs of each output heads of “word multi-tasking block” as well as
“patch multi-tasking block” in the same manner as it is shown and described in
Fig. 3. The results of this network is mentioned in Table. 4 and the
corresponding description is given in Section 5.1.4.

In the following Table. S10, we have also experimented the effect of “soft-
max” layer while computing the weight values from each output heads. Which
means, we have removed the “soft-max” layers in both the network i.e. “Late
concat multiple FC layers” and “Late concat AlexNet like Conv. layers” (see
Table. 4) networks from all $4$ output heads of “Word Multi-tasking Block” and
“Patch Multi-tasking Block”, shown in Fig.3. It can be clearly visible from
the results in Table. S10 in comparison with the results, shown in Table. 4
that when used during the weight computation, the “soft-max” layers plays a
vital role in improving accuracy.

Table S10: Training, validation and testing accuracies of weighted MTL based
network by combining word and patch level images (without soft-max layer for
weight calculation)  Top-1 accuracy of weighted MTL by combining word and
patch level images
---
|  Font Type |  Font Size |  Font Emphasis |  Scanning Resolution
Late concat multiple FC layers |  0.4887 |  0.6970 |  0.6511 |  0.9224
|  0.2558 |  0.4053 |  0.4091 |  0.7032
|  0.2417 |  0.3733 |  0.3789 |  0.5368
Late concat AlexNet like Conv. layers |  0.1824 |  0.3823 |  0.2631 |  0.6090
|  0.1790 |  0.3701 |  0.2476 |  0.5713
|  0.1828 |  0.3517 |  0.2689 |  0.5984