[]✉ For correspondenceLaurent Gatto []Present address Computational Biology
Unit (CBIO), de Duve Institute - UCLouvain, Avenue Hippocrate 75, 1200
Brussels, Belgium []Data availability The code to reproduce the images
presented in the manuscript is available in the Github repository: UCLouvain-
CBIO/2023_scp_na. []Funding This work was funded by a research fellowship of
the Fonds de la Recherche Scientifique-FNRS. []Competing interests The author
declare no competing interests. []Keywords Single-cell, Mass spectrometry,
Proteomics, RNA-Seq, Data analysis, Missing values, Imputation, Reproducible
research. Vanderaa

# Revisiting the thorny issue of missing values in single-cell proteomics

Christophe Vanderaa  Laurent Gatto<EMAIL_ADDRESS>0000-0002-9998-0058

###### Abstract

Missing values are a notable challenge when analysing mass spectrometry-based
proteomics data. While the field is still actively debating on the best
practices, the challenge increased with the emergence of mass spectrometry-
based single-cell proteomics and the dramatic increase in missing values. A
popular approach to deal with missing values is to perform imputation.
Imputation has several drawbacks for which alternatives exist, but currently
imputation is still a practical solution widely adopted in single-cell
proteomics data analysis. This perspective discusses the advantages and
drawbacks of imputation. We also highlight 5 main challenges linked to missing
value management in single-cell proteomics. Future developments should aim to
solve these challenges, whether it is through imputation or data modelling.
The perspective concludes with recommendations for reporting missing values,
for reporting methods that deal with missing values and for proper encoding of
missing values.

##  1 Introduction

The hurdles associated with missing values are a recurring issue in data
analysis and concern a wide range of fields and applications [1]. The handling
of missing values in mass spectrometry (MS)-based proteomics is still actively
debated [2, 3, 4]. While elucidating the best computational approaches to
manage missing values is going on, the field continues pushing the boundaries
of low input acquisitions. Recent technical advances in MS have paved the way
for MS-based single-cell proteomics (SCP) [5, 6, 7, 8, 9], but handling
missing values in SCP data is still a clear challenge to principled data
analysis [10]. We are confident that the amount of missing values will
decrease as future technologies will improve sensitivity [5, 11, 12, 13], but
missing values will always remain unavoidable. It is therefore important to
not only think about how to limit them, but also how to deal with them during
data analysis. In this perspective, we open the discussion on how to manage
missing values in SCP data. We build our perspective on a combination of
computational advances in MS-based proteomics and in single-cell RNA
sequencing (scRNA-Seq). scRNA-Seq is a prolific field with respect to
computational method development [14], and we foresee that many of these
efforts and tools will inspire the field of SCP. We will start by discussing
the advantages and pitfalls of imputation and highlight alternative approaches
to deal with missing values. Next, we describe the main challenges that future
computational development should aim to solve when dealing with missing
values. Finally, we will finish by providing recommendations on how to report
the sensitivity and consistency of an SCP experiment in terms of missing
values, how to report the approach used to deal with these missing values, and
how to encode missing values during data analysis.

##  2 To impute or not to impute?

Ideally, one should avoid imputation when performing SCP data analysis
(imp_args). Imputation of missing values using an unsuitable model can lead to
biased estimates and may introduce false signal [15, 16, 17, 18]. Which model
is suitable for which type of data is still actively benchmarked and debated
in the fields of scRNA-Seq and bulk proteomics [16, 3]. In scRNA-Seq,
imputation can have dramatic effects on clustering and leads to strong
artefacts in the gene expression landscape depicted by t-SNE or UMAP
projections [19, 16]. Another argument against imputation is that imputation
methods may cause over-smoothing [15], i.e. they remove biological
heterogeneity because they directly or indirectly combine observed values.
This means that any prediction is bound within the observed entries and
artificially increase the homogeneity across cells [10, 17]. The issue of
over-smoothing is of particular concern for scRNA-Seq. Numerous methods have
been developed in an attempt to conserve the cell-to-cell relationships in
order to identify rare cell subpopulations [15, 16, 19]. Another important
limitation is that imputation is an estimation process with inherent
uncertainty. Replacing missing values by a single point estimate ignores the
variability associated with the estimation process [16, 17]. For instance,
highly parameterized imputation methods may lead to over-fitting that is
characterized by high variance in the prediction [20]. Multiple imputation
approaches attempt to tackle this issue by computing multiple predictions for
each entry. These predictions are then summarized, typically by averaging,
leading to estimates that are more robust than using a single prediction [21,
22]. Furthermore, variance associated to each imputation and across
imputations can be combined and used for downstream analysis [23], although
this substantially complicates the analytical approach. The performance of
multiple imputation however still relies on the performance of the underlying
imputation method used [24] and does not solve systematic bias. Finally,
imputation can mask contrasts that are inestimable [17]. Proteins quantities
are derived from peptide intensities. However, peptides are often quantified
in a limited fraction of the cells. When two groups of cells share no
quantified peptides and there is no third group of cells that expresses these
peptides to serve as a reference, the contrast is considered inestimable.
Imputation will artificially fill the gaps and mask these inestimable
contrasts, leading to highly biased estimates [17].

Table 1:  Arguments against and for imputation. To impute or not to impute?
---
No |  • Imputation introduces bias • Imputation artificial increases homogeneity across samples • Imputation omits the variance associated to the estimation process • Imputation masks inestimable contrasts
Alternatives |  Use methods that can account for missing values:  • Models dedicated to differential analysis: SMP, hurdle model • Dimension reduction as start point for downstream analyses: NIPALS, missMDA • All-in-one models: ZINB-WaVE, scVI (only scRNA-Seq)
Yes |  • Imputation unlocks many computational tools that break upon missing values • Imputation is easier to develop and assess as a separate step • Imputation is easily transferable across fields

An alternative approach to imputation is to apply models that account for
missing values. In proteomics for example, the selection model for proteomics
(SMP, [17]) or the hurdle model [18] are models that contain a component for
modelling peptide abundance and another to model peptide missing rates, while
taking into account sample and batch effects. While both models provide a
compelling solution to missing data for proteomics, it only applies to label-
free experiments and can only be applied for a single task, differential
analysis. Single-cell downstream analyses also typically include clustering,
dimension reduction or trajectory inference [25, 26]. Moreover, SMP is based
on a Bayesian framework optimized through Gibb’s sampling, and the hurdle
model is fit using a mixed model framework. Hence, these methods will unlikely
scale to large SCP datasets. Another solution is to rely on dimension
reduction output, such as principal component analysis (PCA), to perform the
downstream analysis tasks. Alternating least squares algorithms such as NIPALS
or regularized expectation maximization (EM-)PCA algorithms such as missMDA
are two approaches that perform PCA in presence of missing values [27].
However, the former may suffer from numerical instability and the latter is
computationally expensive and does not scale for large SCP datasets. Moreover,
both approaches assume the missing data pattern is random, which does not hold
for both proteomics and scRNA-Seq data [22, 17, 20], and especially for SCP
[10]. Different methods have been developed for scRNA-Seq data to tackle these
limitations. For instance, the ZINB-WaVE and scVI are modelling approaches
that perform dimension reduction, while accounting for missing values, batch
effects and normalization [28, 29]. These models also go beyond dimension
reduction. ZINB-WaVE estimates the probabilities that missing entries follow a
negative binomial distribution. The probabilities can be used to leverage bulk
differential analysis tools [30]. scVI directly integrates an inference step
in its approach enabling to assess statistical difference between
subpopulations of interest.

Although imputation should ideally be avoided, many SCP analyses include
imputation in their workflow [31]. The reasons are mainly practical
(imp_args). First, many downstream methods cannot directly deal with missing
values. The tools that implement them either return an error when missing
values are present or will output unusable results because the missing values
are propagated to every entry of the data. Using a suitable imputation method
to predict missing values unlocks these methods. Another reason is that it can
be included as a standalone step in a workflow. Although it is wrong to assume
that imputation is independent of other processing steps such as normalization
or batch correction [10], it dramatically facilitates the assessment of its
performance in relation to these other steps [16, 32]. This assessment is much
harder for methods such as ZINB-WaVE or scVI where the quality of the outcome
will depend on the different components of the model without the possibility
to pinpoint performance bottlenecks. Also, different data require different
imputation methods. For instance, [16] and [2] provide decision trees for
which imputation method to select given data characteristics for scRNA-Seq and
proteomics, respectively. Hence, allowing to easily plug a different
imputation method into a workflow is easier and quicker than developing a new
model tailored for each data type and analysis task at hand. Finally,
imputation methods can easily be transferred and reassessed across fields. For
instance, K-nearest neighbour (KNN) imputation is an imputation method used in
a wide range of application and fields [33] and has been rapidly adopted by
the SCP field that still lacks dedicated models to handle missing values [31].

In conclusion, we recommend avoiding imputation, but in practice this can be
difficult to achieve. Suitable methods to handle missing values are still to
be developed for SCP. While imputation leads to inaccuracies, it still
leverages useful information from complex data [16]. In the following section,
we list five main challenges that should be considered when developing
imputation methods or models that handle missing values for SCP data.

##  3 Challenges in managing missing values in SCP data

###  3.1 Challenge 1: High proportion of missing values

The first challenge is to cope with high proportions of missing values
(challenges). A typical SCP dataset can contain on average between 50 and 90 %
missing values. In the context of scRNA-Seq, most imputation methods exhibit a
decreased performance for transcripts with high proportions of missing values
[19]. This is expected since, on the one hand, fewer data are available to
predict the missing value and, on the other hand, more bias is introduced by
performing more estimations. Up to now, the methods used to impute SCP data
were taken from the proteomics field [31], but their suitability to single-
cell data needs to be evaluated. The performance of imputation methods on bulk
data has been evaluated on datasets with, relatively, low numbers of samples
and missing value proportions about 20 to 50 % [2, 3, 24, 34]. Another
question raised by the high proportion of missing values in SCP data is how to
perform peptide or protein filtering. Bulk proteomics pipelines often remove
highly missing peptides, usually allowing at most 90 % of missing values [2].
Depending on the SCP datasets, this filter can lead to removing between 50 and
80 % of all quantified peptides missing. The challenge is therefore to
leverage useful information from these highly missing peptides.

Figure 1:  Distribution of the percentages of missing values for peptides. The
graph shows the cumulative distribution with respect to the percentage of
missing values. Each colour represents a dataset. The red dot is shown as an
example. For the brunner2022 dataset, about 50 % of the peptides have more
than 75 % of missing values.

All data were retrieved from the scpdata package [31] and systematically
processed using the same minimal workflow. Briefly, we start from the
quantified peptide to spectrum match or precursor data and keep only samples
that correspond to a single cell. We then remove any feature that is a
contaminant, matches a decoy peptide, or has an associated false discovery
rate > 1%. We then encode all zeros as missing values and filter out features
that are missing for all cells. Finally, we compute the percentage of missing
values for each peptide (a peptide is missing in a cell if all the
corresponding intensities at the precursors level are missing). The
percentages are ordered by increasing values. The percentage of peptides is
computed as the rank of the percentage of missing values divided by the total
number of peptides. Reference to the source data, in the order they appear in
the legend: [35, 36, 37, 38, 39, 40, 41, 42]

Table 2:  Summary of the main challenges in managing missing values in SCP data. Challenges |
---|---
High proportion of missing values |  SCP data contain between 50 and 90 % missing values. This leads to decreased imputation performance and requires to reassess the thresholds used for filtering peptides or proteins based on the proportion of missing values.
Different types of data |  SCP data can be acquired using DIA or DDA approaches, LFQ or multiplexed protocols, using orbitraps or time of flight. The diversity of technologies leads to different missing value patterns.
Cell to cell heterogeneity |  Every cell is unique, hence no technical replicates are available. Several methods already tackle this challenge when imputing scRNA-Seq data: MAGIC, scISR, Network.
Strong batch effects |  SCP data is acquired through hundreds of MS runs, leading to inevitable batch effects both on the identification and the quantification. The RUV-III-C method is a promising approach, provided adaptation to SCP data.
Different causes of missing values |  Values may be missing because of MNAR or MCAR mechanisms. Most methods to deal with missing values are tailored towards one or the other mechanism. A potential solution is to apply cross-validated ensemble approaches, as suggested for scRNA-Seq, or relying on probability estimations, as as suggested for bulk proteomics.

###  3.2 Challenge 2: Different types of data

The second challenge is the diversity of data acquisition protocols and
instruments to perform SCP (challenges). For instance, we can classify
protocols as label-free (LFQ) or as multiplexed (TMT, mTRAQ). LFQ experiments
acquire one cell per MS run, while multiplexed experiments acquire from 3 to
18 cells per MS run. These two protocols lead to different correlation
structures and batch effects in the data. Furthermore, samples processed by
either of these two protocols can be acquired using data-dependent acquisition
(DDA) or data-independent acquisition (DIA). Also, DDA can be targeted towards
specific peptides known to be present in a sample by using a prioritized
inclusion list for MS2 selection [43, 38]. A thorough description of the
different technologies for acquiring SCP data is provided by others [6]. These
protocols, in combination with the instruments to run them, generate different
types of data and lead to different patterns of missing values, implying that
different imputation assumptions and algorithms might be required. This is not
specific to SCP. In bulk proteomics, benchmarking efforts and literature
reviews focus on one type of data. For instance, [2] focus on DDA LFQ data,
[3] focus on DDA TMT data and [4] focused on LFQ DIA data. Hence, development
and validation of SCP imputation methods should be dataset-specific.

###  3.3 Challenge 3: Cell to cell heterogeneity

The imputation method should ideally account for cell-to-cell heterogeneity
(challenges). Each single cell is a unique sample, hence no technical
replication is possible [44]. Solutions to this challenge have been explored
in the field scRNA-Seq. For instance, MAGIC [45] improves upon sample-wise KNN
imputation by taking into account the similarity between cells (we will
discuss the meaning of sample-wise KNN in the next section). When imputing a
value with sample-wise KNN, we first search a discrete set of K cells that are
closely related to the cell to impute. The average of these neighbouring cells
provides the imputation estimate. Obviously, the number of neighbours K will
influence the imputation results. Instead of searching for a discrete set of
neighbours, MAGIC computes the probability of transitioning from one cell to
another and uses the probabilities as weights to estimate the missing value
for a given cell. This approach takes into account that each cell is different
and may contribute differently to the imputation process. However, MAGIC
borrows information from cell-cell relationships, hence it can potentially
influence these relationships for downstream analysis, such as clustering of
trajectory inference. A similar imputation approach is to predict missing
values using gene-gene relationships, as implemented in the scISR or the
Network algorithms [15, 19]. The drawback of these methods is that building a
reliable gene network is difficult and either relies on fully observed genes
[15] or on external bulk data [19].

###  3.4 Challenge 4: Strong batch effects

The fourth challenge is the presence of strong batch effects in SCP data
(challenges). SCP experiments contain hundreds to thousands of cells that must
be analysed over many MS runs. Slight differences in experimental conditions
across MS acquisitions influence the identification and quantification of
peptides [17]. We also observe a strong influence of batch effects on peptide
quantification and missing rate [10]. An imputation method for SCP should
account for batch effects as well. The ProNorm method was designed to account
for these effects in the context of large bulk proteomics datasets acquired
using a DIA approach [46]. ProNorm first removes batch effects using the RUV-
III-C model and then performs imputation by sampling from a normal
distribution centred around the average of the batch corrected replicates. The
method has however several drawbacks when applied to single-cell data. First,
it relies on the presence of replicates. As stated above, we can no longer
rely on replicates with single-cell data. Second, the model also relies on
negative controls, i.e. peptides that are known to be constant across cells.
The presence of such negative controls is again a strong assumption in the
context of single-cells given the presence of cell-to-cell heterogeneity.
Finally, the RUV-III-C model is specifically designed to remove any variance
that is not associated to biological factors and/or experimental conditions of
interest. In other words, RUV-III-C removes any variability that is not
explained by the experimental design. This property is not desirable for
single-cell analyses as the main objective is to leverage unexplained
variance, for instance, to discover rare cell states or continuous
differentiation processes that are blurred by bulk approaches. This method
could be adapted to SCP similarly to how ZINB-WaVE adapted the RUV model to
scRNA-Seq data [28].

###  3.5 Challenge 5: Different causes of missing values

The fifth challenge is to account for the different mechanisms that generate
missing values (challenges). In proteomics, there are two main types of
missing data: missing completely at random (MCAR) and missing not at random
(MNAR) [2, 24]. MCAR means that the process that generated the missing value
is completely random and unrelated to the true underlying value. Conversely,
MNAR means that the missing value probability is dependent on the underlying
value. In other words, a peptide may be MNAR because its corresponding protein
is not present in a cell or at levels that fall below the MS2 threshold for
selection (DDA) or limit of detection of the instrument. A peptide may be MCAR
because of technical artefacts (e.g. slight variation in ionization
efficiency) or computational limitations (e.g. poor identification score).
Different imputation algorithms are required to correctly impute MNAR or MCAR
values [2, 24]. For instance, the sampling from a down-shifted normal
distribution imputation approach is recommended for missing values that
exhibit MNAR properties, while a dimension reduction-based imputation is
recommended when the missing values are MCAR [2]. However, SCP datasets and
more generally large proteomics datasets exhibit both MCAR and MNAR in unknown
proportions [10, 3, 34]. In line with the second challenge, the prevalence of
MNAR and MCAR will highly depend on the type of datasets, but it will also
depend on the nature of the peptide and protein analysed [22, 17]. This highly
complicates the development and the assessment of imputation methods. The
variety of parameters that would influence the choice of imputation methods is
also a challenge for scRNA-Seq. In this context, Leote et al. conclude that
“there is no universally best performing method that outperforms the others in
all cases”. They therefore propose an ensemble approach. Where the best
imputation method is selected for each transcript separately using a cross
validation approach. If adapted to SCP, this algorithm could for instance
determine which peptides are best imputed using an MNAR-devoted algorithm, a
MCAR-devoted algorithm, or a mixed MCAR/MNAR-devoted algorithm [2, 24].
Another option is to rely on probability theory to assess the proportion of
MNAR and MCAR, as was already investigated for bulk proteomics [22]. This
framework also includes multiple imputation [23, 22], although the
computational scalability to single-cell datasets must be considered as well.

methods or modelling approaches for SCP data should address these
simultaneously. We therefore expect that more complex algorithms than those
currently used for SCP data processing will be required. These challenges are
currently overwhelming and hard to tackle, but we are optimistic. We foresee
that future advances in the technologies and in search engines will reduce the
amount of missing values generated in single cells [11, 47, 12]. Hence,
tackling these challenges will become easier with time as the quality of the
data to impute will improve.

##  4 Recommendations for managing missing values in SCP data

In this section, we extend the initial recommendations recently published for
performing, benchmarking, and reporting SCP experiments [44], focusing on
reporting missing values and the methods to manage them. We will also discuss
how to encode missing values in SCP data.

###  4.1 Reporting sensitivity and consistency

Missing values and sensitivity are two sides of the same coin. If a technology
had perfect sensitivity, no missing data would be generated other than
biologically missing values. We here define sensitivity as the number of
distinct peptide or protein identifications. To facilitate the discussion, we
will not distinguish between sensitivity at the peptide or protein level
because they show the same trends, and we will refer to features instead.
Sensitivity is commonly reported as the number of features per cell [36, 48,
49, 38, 35]. We call this metric the local sensitivity (sensitivitya, dashed
line, and b). Local sensitivity is computed for each cell, hence this metric
should be summarised with the mean and standard deviation. While technological
developments aim to maximize local sensitivity, it should be interpreted with
care. Solely looking at the local sensitivity does not provide the full
picture about the number of features that can be detected in an experiment.
Every cell is unique, and every run is subject to technical variations that
influence feature identification. So, we do not expect to find all detectable
features by looking at individual cells. Therefore, we recommend to also
report sensitivity as the total number of distinct features identified across
the whole experiment. We call this the total sensitivity (sensitivitya, solid
line, c). It allows a more accurate estimation of how many features a given
experiment can potentially detect in low amount samples. Total sensitivity has
also associated caveats. When reporting total sensitivity, we assume that
enough cells were acquired to capture all detectable features. This is the
case for the CD38 negative blast cells acquired by Schoof et al. [40]. By
including thousands of cells into their experiment, the authors were able to
reach a saturation of the set of identified peptides (sensitivitya). If they
had limited their analysis to less than 500 blast cells, they would have
underestimated the total sensitivity. This is the case for the leukaemic stem
cells, that are scarcer than blast cells. About a hundred stem cells do not
allow to reach the total sensitivity for that cell type. We expect that
reaching total sensitivity will be difficult for rare cell populations and
will offer a solution later in the section.

Although local and total sensitivity are useful metrics when reported
together, they provide no information about the consistency of identification.
Consistency of identification relates to the similarity between sets of
features identified across different cells [47]. Consistency of identification
between cells has previously been reported using Venn diagrams, or upset plots
[36, 49, 48, 10, 39, 37]. These approaches nevertheless limit the comparison
to tens of single cells, otherwise these visualizations become difficult to
interpret. We recommend instead to report the data completeness. Data
completeness is the proportion of observed values across the dataset. It
relies on the fact that the lower the consistency between cells, the more
missing values will be incorporated in the dataset, hence the lower the data
completeness. As soon as a cell with new features is added to a dataset, these
new features will introduce missing values for the other cells and therefore
decrease the data completeness. But again, reporting only the data
completeness is not sufficient. It provides no information about sensitivity.
For instance, the Liang et al. dataset showed a relatively high data
completeness (hence high consistency), but at the expense of a limited number
of features (low total sensitivity, sensitivityb, sensitivityc). Another
drawback is that data completeness depends on the accurate estimation of the
total sensitivity, hence the need to acquire sufficient number of cells. The
Liang et al. dataset is composed of only three single cells and hence
underestimates the total sensitivity. This explains the apparently high but
biased data completeness.

In conclusion, we suggest reporting at least the local sensitivity, the total
sensitivity, the proportion of missing values and the number of cells included
in the dataset. No useful assessment of sensitivity and consistency is
possible without reporting these four metrics together. They are simple to
implement and do not require complex data analysis. These metrics represent an
objective way to compare different parameters in an SCP experiment that may
affect sensitivity and consistency, such as sample preparation protocols, MS
settings or peptide identification software. Only one of these parameters
should be varied if the objective is to compare performance. Therefore,
sensitivityc cannot be used to suggest superior performance of a protocol with
respect to the others because the datasets contain different cell types, they
were acquired with different instruments, and they were processed with
different search engines. Still, we can see a correlation between the local
sensitivity and the data completeness, indicating that increased sensitivity
allows retrieving more of the same features across cells. We also recommend
computing these metrics for the full dataset and for each cell type separately
(defined in the experimental design or after data analysis). Different cell
types express different sets of peptides in different quantities which will
inevitably influence the sensitivity. Finally, we focus the discussion solely
on reporting sensitivity of identification. We refer to the initial
recommendations for a thorough discussion about assessment of the quantitative
data [44], and we refer to the recent review by [47] for recommendations about
the computational challenges for peptide and protein quantification in SCP
data.

Figure 2: Reporting sensitivity. a. Cumulative sensitivity curve showing the
total number of distinct peptides as we sample an increasing number of CD38-
blast cells (yellow) or leukaemic stem cells (LCS, brown) from the Schoof et
al. dataset. The dashed line represents the local sensitivity, that is the
average sensitivity of a single cell. The solid grey line is the total
sensitivity. The solid red line shows the fit of the asymptotic regression
model for the LSC data, with extrapolation. b. The median local sensitivity
with respect to data completeness. Each point is a cell type, coloured
according to the corresponding dataset, and sized according to the number of
cells acquired. The vertical lines represent the standard deviation. Empty
diamonds indicate the dataset average. c. Total number of distinct peptides
found in various cell types grouped by dataset (colour). d. Distribution of
the pairwise Jaccard index within each cell type for each dataset.

All data were retrieved from the scpdata package [31] and systematically
processed using the same minimal workflow. Briefly, we start from the
quantified peptide to spectrum match or precursor data and keep only samples
that correspond to a single cell. We then remove any feature that is a
contaminant, matches a decoy peptide, or has an associated false discovery
rate > 1%. We then encode all zeros as missing values and filter out features
that are missing for all cells. We then build the identification matrix where
rows correspond to peptides and columns to single cells, and entries are 1 if
the peptide was found in the corresponding cell and 0 otherwise. The matrix is
then split based on the cell type. Metrics are computed as follows. The number
of cells is the number of columns of the matrix. The local sensitivity is the
sum by column. The total sensitivity is the number of rows and the data
completeness is the average of the matrix. Jaccard indexes are computed for
each pair of column as the size of the union of peptides divided by the size
of the intersection. The cumulative sensitivity plot was constructed from the
identification matrix as described in the body of the text. References to the
source data, in the order they appear in the legend: [35, 36, 37, 38, 39, 40,
41, 42]

We next extend upon the four metrics with two more advanced concepts that may
be harder to implement.

First, we recommend testing whether the number of cells is sufficient to
accurately assess the total sensitivity. We suggest reporting a cumulative
sensitivity plot (sensitivitya), where the sensitivity is reported as a
function of the number of cells. To build this graph, we randomly sample an
increasing number of cells from the dataset and compute the sensitivity for
each pool. We suggest performing several draws with the same number of cells
to include an estimation of the variance associated with the stochastic
selection process. The sensitivity is expected to plateau at the total
sensitivity as we include more cells (sensitivitya). This can be modelled
using an asymptotic regression model. In case the total sensitivity is not
achieved, the total sensitivity may be extrapolated using the fitted model
asymptote (sensitivitya, red line).

Second, inspired by the quality assessment by Derks and colleagues, we also
recommend reporting the Jaccard index between all possible pairs of cells
within a cell type (sensitivityd). The Jaccard index measures the overlap
between the sets of features identified from 2 cells. Contrarily to the data
completeness, the Jaccard index no longer depends on the total sensitivity.
However, computing the Jaccard index between all pairs of cells is more
elaborate and is best performed using a programming language such as R or
python.

###  4.2 Reporting methods for missing values

The minimal requirement for reporting an imputation method or a missing data
model approach is to provide the name of the algorithm used, the name of the
software that implements it and the version of that software. We stress that
these three elements are the bare minimum for a meaningful and useful method
report. Reporting the version of a software is critical, because different
versions of the same software may add, remove or update functionality, hence
possibly changing the output. Next to reporting the version, we have seen
method sections that solely report a piece software or an algorithm [31]. This
leads to ambiguity since software often implement several algorithms, and an
algorithm is often implemented by multiple pieces of software. For instance,
VIM (R), impute (R) and sklearn (python) offer functionality to perform KNN
imputation. These three pieces of software offer different parameters (and
default values), perform different optimizations, solve edge cases
differently, and have different assumptions about the data. We experienced the
impact of these differences when replicating published SCP data analyses [10].
[41] performed KNN imputation using their own implemented in R. We observed
significant numerical differences when comparing their processed data tables
to those generated by our R/Bioconductor package scp [10]. After inspection,
we were able to identify the origin of the problem. The reimplementation by
Specht performs sample-wise KNN (i.e. infers missing values from closely
related cells), while our scp package relies on the impute.knn function from
the R package impute that performs variable-wise KNN (i.e. infers missing
values from closely related peptides or proteins). knn_imputation shows the
impact of both KNN approaches on cell and protein correlations. Because
sample-wise KNN borrows information from other cells, it will on average
increase correlations between cells (knn_imputationa and c). Conversely,
variable-wise KNN will remove correlation between cells because it focuses on
the relationships between features. We also observe a small impact on protein
correlations (knn_imputationb). We observe a reduction in the protein
correlations after both sample-wise and feature-wise KNN imputation, but we
also notice a subtle but systematic shift towards higher correlation when
imputing with sample-wise KNN. Since the proportion of missing values vary
drastically across proteins, we binned the protein correlation based on their
percent missing values (knn_imputationd). KNN imputation, both sample-wise and
variable-wise, has little impact on correlation distribution when proteins
have less than 20 % missing values (knn_imputationd, top). This is expected
since fewer predictions are required. As more observations are missing, the
protein correlations before imputation become inaccurate. Sample-wise KNN
imputation of highly missing proteins restores the correlation distributions
centred around zero, as seen when proteins are less missing. Variable-wise KNN
leads to correlation distributions that are sharper and progressively shifted
towards higher correlations as the percentage of missing values increases.

Sample-wise and variable-wise KNN rely on different assumptions leading to
numerical differences that will propagate through the data processing
workflow. In theory, sample-wise KNN leads to a violation of sample
independence assumption on which most statistical methods rely on. Similarly,
the same violation occurs for proteins when combining variable-wise KNN and
statistical methods to infer regulatory protein networks. However, how these
violations impact the statistical outcome in practice is still to be explored.

To conclude, we strongly recommend reporting the missing value management
approach by mentioning the algorithm, the name of the software and its version
together. However, we stress that the best practice is to provide the code
(e.g. scripts, Rmarkdown files or Jupyter notebooks) that generates the
published results along with the computational environment containing all the
software tools in the state that they were used to run the code [44].

Figure 3: Sample-wise and variable-wise KNN are not the same. a. Distribution
of Pearson correlations computed for each pair of cells within a cell type,
that is melanoma (red) or monocyte (blue). Data were either not imputed
(none), sample-wise KNN imputed or variable-wise KNN imputed. Note that the
observed values before imputation are untouched after imputation. Points
highlight the median of the distribution. Data from Derks et al. [36]. b. Same
as a., but for protein correlation. There is no known protein grouping, so all
pairwise correlations are shown. c. First two principal components computed
using NIPALS to allow for missing values. Data were either not imputed (none),
sample-wise KNN imputed or variable-wise KNN imputed. Each point represents a
single cell, coloured by sample type. Grey circles highlight cell densities.

The nPOP data [38] were retrieved from the scpdata package [10]. We took the
protein data processed before imputation [31]. To avoid complicating the
interpretation with confounding effects, we remove batch effect using the
removeBatchEffect algorithm from the limma R package (version 3.55.1). Note
that we do not recommend this approach for SCP data analysis (c.f. fourth
challenge). Next, we impute the protein data using the impute function from
the QFeatures package (version 1.9.2). We used the KNN algorithm with K = 15,
ensuring that every entry is imputed with KNN, regardless of the proportion of
missing values (rowmax = 1, colmax = 1) and disabling computational shortcuts
(maxp = Inf). Pairwise correlation per cell and per protein were then
computed. When two cells have the same type, their correlation is annotated
with that cell type. The proteins were binned in 5 categories depending on
their percentage missing values. When to proteins are included in the same
bin, their correlation is flagged with that bin.

###  4.3 Encoding missing values

Unobserved entries in scRNA-Seq data are encoded as zeros. This is because
RNA-Seq is a count based approach where zero is a meaningful readout.
Proteomics experiments, however, are based on methods that measure electrical
current intensities. In the absence of any signal, MS instruments still record
noise levels as low values, but zeros are not expected to occur. Any zero
present in SCP data is generated computationally, either because the
intensities below a given threshold were trimmed to zero or because
quantifications for unobserved peptides or proteins were filled with zeros
when joining multiple samples together. Since these computationally derived
zeros are not meaningful, we recommend replacing them with a dedicated data
type, such as the NA value in R or the numpy.nan value in python. Larger
datasets will require sparse data representations that are crucial for
managing and handling the data with limited computational resources. These
data representations, such as implemented in the Matrix R package or the
scipy.sparse python module assume the missing entries are zeros. Future large-
scale analyses might make use of sparse representations to support efficient
data storage. Zeros should then be considered as placeholders that are
disregarded during data processing and downstream analyses. Otherwise, this
would boil down to an implicit zero imputation that is likely ill-suited for
SCP data [2, 24]. We therefore suggest future computational tools for SCP data
to handle missing values and zero values with great care.

##  5 Conclusion

Handling missing values in SCP can be an intimidating task. We prefer to avoid
imputing missing values, but unfortunately no alternatives have yet been
suggested that are easily applicable to SCP. Although imputation methods can
and are applied to SCP data, their suitability still needs to be demonstrated.
Computational benchmarking efforts are required to pinpoint the limitations of
these imputation methods on SCP data, but also to assess the practical
significance of the limitations associated to imputation compared to data
modelling.

We advise that future computational developments consider 5 main challenges
when dealing with missing values. First, the proportion of missing values is
high. Most datasets have over 70 % missing entries at the peptide level. At
these rates of missing values, several imputation methods fail to provide
accurate predictions [2]. Second, there is a diversity of experimental
approaches to perform SCP. Each approach has its strengths and weaknesses and
lead to different missing data patterns. Different algorithms might be
necessary to deal with each pattern, and we do not expect that an algorithm
that performs best for one type of data will perform best for all types of
data. Therefore, future computational innovation must clearly state on which
type of data they were tested and are expected to be best suited. Third, the
imputation methods should take into account that each single cell is unique,
and that replicates are not available. This will require new methods to
evaluate the sweet spot between over-fitting and over-smoothing. Fourth, SCP
inevitably imposes large scale experiments. Therefore, batch effects are and
will be present in SCP data [10]. We cannot imagine a successful method that
handles missing values without handling batch effects as well. Finally,
missing values may be deterministic and/or stochastic, and distinguishing the
two mechanisms is important to avoid bias in the data analysis. This is
probably the toughest challenge.

We provide a few recommendations targeted towards SCP practitioners. First, we
recommend reporting at least four metrics when reporting the sensitivity of an
experiment. The local sensitivity provides information about the amount of
features that are detected in a single cell. Total sensitivity provides
information about the number of features that are potentially detectable in
the experiment. The data completeness provides the global agreement between
the sets of features identified across cells. The number of cells acquired
during the experiment provide essential information because total sensitivity
and data consistency require enough cells to be correctly estimated. We
recommend not interpreting these metrics when less than 100 cells are
acquired, and suggest using a cumulative sensitivity plot to assess whether
the number of cells allow for an accurate estimation of the total sensitivity.
Reporting the pairwise Jaccard index provides also useful information about
data consistency. In addition to sensitivity, we also recommend providing at
least 3 pieces of information when reporting the approach used to deal with
missing values: the name of the algorithm, the name of the software, and the
version of the software. We also strongly recommend providing the code and the
software environment to reproduce any published results. This recommendation
is not new and was already presented previously [44]. We however provide a
comprehensive example showing that small variations in applying an imputation
method can have significant impact on the processed data and downstream
analyses. Finally, we raise concerns regarding how missing values are encoded.
We recommend encoding missing values using an appropriate data type rather
than a numeric value (e.g. NA for R and numpy.nan for python).

The field of SCP is rapidly expanding [50], but it craves for adequate
solutions to deal with missing data. This represents a great opportunity for
computational biologists and bioinformaticians that want to join the exciting
advances of SCP.

###  5.1 Acknowledgment

This preprint was created using the LaPreprint template
(https://github.com/roaldarbol/lapreprint) by Mikkel Roald-Arbøl .

We thank Lieven Clement, Davide Risso, Charlotte Soneson and the members of
the CBIO lab for useful discussions about scRNA-Seq and proteomics data
imputation.

###  5.2 Author contributions

Conceptualization: L.G.; Methodology: C.V., L.G.; Software: C.V., L.G.; Formal
analysis: C.V.; Writing - original draft: C.V., L.G.; Writing - review &
editing: L.G.; Visualization: C.V.; Supervision: L.G.; Project administration:
L.G.; Funding acquisition: L.G.

## References

  * [1] Jiaxu Peng, Jungpil Hahn and Ke-Wei Huang  “Handling Missing Values in Information Systems Research: A Review of Methods and Assumptions”  In _Information Systems Research_ INFORMS, 2022
  * [2] Weijia Kong, Harvard Wai Hann Hui, Hui Peng and Wilson Wen Bin Goh  “Dealing with missing values in proteomics data”  In _Proteomics_ Wiley, 2022, pp. e2200092
  * [3] Lisa M Bramer et al.  “A Review of Imputation Strategies for Isobaric Labeling-Based Shotgun Proteomics”  In _J. Proteome Res._ 20.1, 2021, pp. 1–13
  * [4] Kruttika Dabke, Simion Kreimer, Michelle R Jones and Sarah J Parker  “A Simple Optimization Workflow to Enable Precise and Accurate Imputation of Missing Values in Proteomic Data Sets”  In _J. Proteome Res._ 20.6, 2021, pp. 3214–3229
  * [5] Manuel Matzinger, Rupert L Mayer and Karl Mechtler  “Label-Free Single Cell Proteomics Utilizing Ultrafast LC and MS Instrumentation: A Valuable Complementary Technique to Multiplexing”  In _Proteomics_ Wiley, 2023, pp. e2200162
  * [6] Valdemaras Petrosius and Erwin M Schoof  “Recent advances in the field of single-cell proteomics”  In _Transl. Oncol._ 27, 2022, pp. 101556
  * [7] Nikolai Slavov  “Single-cell protein analysis by mass spectrometry”  In _Curr. Opin. Chem. Biol._ 60, 2021, pp. 1–9
  * [8] Claudia Ctortecka and Karl Mechtler  “The rise of single-cell proteomics”  In _Analytical Science Advances_ 2.3-4 Wiley, 2021, pp. 84–94
  * [9] Ryan T Kelly  “Single-cell Proteomics: Progress and Prospects”  In _Mol. Cell. Proteomics_ 19.11, 2020, pp. 1739–1748
  * [10] Christophe Vanderaa and Laurent Gatto  “Replication of single-cell proteomics data reveals important computational challenges”  In _Expert Rev. Proteomics_ , 2021, pp. 1–9
  * [11] Jason Derks and Nikolai Slavov  “Strategies for increasing the depth and throughput of protein analysis by plexDIA”  In _bioRxiv_ , 2022, pp. 2022.11.05.515287
  * [12] Nikolai Slavov  “Driving Single Cell Proteomics Forward with Innovation”  In _J. Proteome Res._ 20.11, 2021, pp. 4915–4918
  * [13] Harrison Specht and Nikolai Slavov  “Transformative Opportunities for Single-Cell Proteomics”  In _J. Proteome Res._ 17.8, 2018, pp. 2565–2571
  * [14] Luke Zappia and Fabian J Theis  “Over 1000 tools reveal trends in the single-cell RNA-seq analysis landscape”  In _Genome Biol._ 22.1, 2021, pp. 301
  * [15] Jing Qi et al.  “scMTD: a statistical multidimensional imputation method for single-cell RNA-seq data leveraging transcriptome dynamic information”  In _Cell Biosci._ 12.1, 2022, pp. 142
  * [16] Wenpin Hou, Zhicheng Ji, Hongkai Ji and Stephanie C Hicks  “A systematic evaluation of single-cell RNA-sequencing imputation methods”  In _Genome Biol._ 21.1, 2020, pp. 218
  * [17] Jonathon J O’Brien et al.  “The effects of nonignorable missing data on label-free mass spectrometry proteomics experiments”  In _Ann. Appl. Stat._ 12.4, 2018, pp. 2075–2095
  * [18] Ludger J E Goeminne et al.  “MSqRob Takes the Missing Hurdle: Uniting Intensity- and Count-Based Proteomics”  In _Anal. Chem._ 92.9, 2020, pp. 6278–6287
  * [19] Ana Carolina Leote, Xiaohui Wu and Andreas Beyer  “Regulatory network-based imputation of dropouts in single-cell RNA sequencing data”  In _PLoS Comput. Biol._ 18.2, 2022, pp. e1009849
  * [20] Duc Tran, Bang Tran, Hung Nguyen and Tin Nguyen  “A novel method for single-cell data imputation using subspace regression”  In _Sci. Rep._ 12.1, 2022, pp. 2697
  * [21] Wuming Gong et al.  “DrImpute: imputing dropout events in single cell RNA sequencing data”  In _BMC Bioinformatics_ 19.1, 2018, pp. 220
  * [22] Q Giai Gianetto, S Wieczorek, Y Couté and T Burger  “A peptide-level multiple imputation strategy accounting for the different natures of missing values in proteomics data”  In _bioRxiv_ , 2020, pp. 2020.05.29.122770
  * [23] Marie Chion, Christine Carapito and Frédéric Bertrand  “Accounting for multiple imputation-induced variability for differential analysis in mass spectrometry-based label-free quantitative proteomics”  In _PLoS Comput. Biol._ 18.8, 2022, pp. e1010420
  * [24] Cosmin Lazar et al.  “Accounting for the Multiple Natures of Missing Values in Label-Free Quantitative Proteomics Data Sets to Compare Imputation Strategies”  In _J. Proteome Res._ 15.4, 2016, pp. 1116–1125
  * [25] Robert A Amezquita et al.  “Orchestrating single-cell analysis with Bioconductor”  In _Nat. Methods_ 17.2 Nature Publishing Group, 2020, pp. 137–145
  * [26] Malte D Luecken and Fabian J Theis  “Current best practices in single-cell RNA-seq analysis: a tutorial”  In _Mol. Syst. Biol._ 15.6, 2019, pp. e8746
  * [27] Julie Josse and François Husson  “missMDA: A Package for Handling Missing Values in Multivariate Data Analysis”  In _J. Stat. Softw._ 70, 2016, pp. 1–31
  * [28] Davide Risso et al.  “A general and flexible method for signal extraction from single-cell RNA-seq data”  In _Nat. Commun._ 9.1, 2018, pp. 284
  * [29] Romain Lopez et al.  “Deep generative modeling for single-cell transcriptomics”  In _Nat. Methods_ 15.12, 2018, pp. 1053–1058
  * [30] Koen Berge et al.  “Observation weights unlock bulk RNA-seq tools for zero inflation and single-cell applications”  In _Genome Biol._ 19.1, 2018, pp. 24
  * [31] Christophe Vanderaa and Laurent Gatto  “The Current State of Single-Cell Proteomics Data Analysis”  In _Curr Protoc_ 3.1, 2023, pp. e658
  * [32] Luyi Tian et al.  “Benchmarking single cell RNA-sequencing analysis pipelines using mixture control experiments”  In _Nat. Methods_ 16.6, 2019, pp. 479–487
  * [33] Trevor Hastie et al.  “Imputing Missing Data for Gene Expression Arrays”  unknown, 2001
  * [34] Bobbie-Jo M Webb-Robertson et al.  “Review, evaluation, and discussion of the challenges of missing value imputation for mass spectrometry-based label-free global proteomics”  In _J. Proteome Res._ 14.5, 2015, pp. 1993–2001
  * [35] Andreas-David Brunner et al.  “Ultra-high sensitivity mass spectrometry quantifies single-cell proteome changes upon perturbation”  In _Mol. Syst. Biol._ 18.3 EMBO, 2022, pp. e10798
  * [36] Jason Derks et al.  “Increasing the throughput of sensitive proteomics by plexDIA”  In _Nat. Biotechnol._ , 2022, pp. 2021.11.03.467007
  * [37] Maowei Dou et al.  “High-Throughput Single Cell Proteomics Enabled by Multiplex Isobaric Labeling in a Nanodroplet Sample Preparation Platform”  In _Anal. Chem._ 91.20, 2019, pp. 13119–13127
  * [38] Andrew Leduc et al.  “Exploring functional protein covariation across single cells using nPOP”  In _Genome Biol._ 23.1, 2022, pp. 261
  * [39] Yiran Liang et al.  “Fully Automated Sample Processing and Analysis Workflow for Low-Input Proteome Profiling”  In _Anal. Chem._ 93.3, 2021, pp. 1658–1666
  * [40] Erwin M Schoof et al.  “Quantitative single-cell proteomics as a tool to characterize cellular hierarchies”  In _Nat. Commun._ 12.1 Springer ScienceBusiness Media LLC, 2021, pp. 745679
  * [41] Harrison Specht et al.  “Single-cell proteomic and transcriptomic analysis of macrophage heterogeneity using SCoPE2”  In _Genome Biol._ 22.1, 2021, pp. 50
  * [42] Ying Zhu et al.  “Single-cell proteomics reveals changes in expression during hair-cell development”  In _Elife_ 8, 2019
  * [43] Gray R Huffman et al.  “Prioritized single-cell proteomics reveals molecular and functional polarization across primary macrophages”  In _bioRxiv_ , 2022, pp. 2022.03.16.484655
  * [44] Laurent Gatto et al.  “Initial recommendations for performing, benchmarking and reporting single-cell proteomics experiments”  In _Nat. Methods_ , 2023
  * [45] David Dijk et al.  “Recovering Gene Interactions from Single-Cell Data Using Data Diffusion”  In _Cell_ 174.3, 2018, pp. 716–729.e27
  * [46] Rebecca C Poulos et al.  “Strategies to enable large-scale proteomics for reproducible research”  In _Nat. Commun._ 11.1, 2020, pp. 3793
  * [47] Hannah Boekweg and Samuelh Payne  “Challenges and opportunities for single cell computational proteomics”  In _Mol. Cell. Proteomics_ , 2023, pp. 100518
  * [48] Kei G I Webber et al.  “Label-Free Profiling of up to 200 Single-Cell Proteomes per Day Using a Dual-Column Nanoflow Liquid Chromatography Platform”  In _Anal. Chem._ , 2022
  * [49] Jongmin Woo et al.  “Three-dimensional feature matching improves coverage for single-cell proteomics based on ion mobility filtering”  In _cels_ 0.0 Elsevier, 2022
  * [50] “Single-cell proteomics: challenges and prospects”  In _Nat. Methods_ 20.3, 2023, pp. 317–318


# Tracking e-cigarette warning label compliance
on Instagram with deep learning

Chris J. Kennedy, Julia Vassey, Ho-Chun Herbert Chang,
Jennifer B. Unger, Emilio Ferrara

##  1 Introduction

The proportion of the U.S. high school students who report using e-cigarettes
(aka vaping devices) declined in 2020 during the COVID-19 pandemic. In 2020,
19.6% of high school students (3.02 million) reported current e-cigarette use,
compared to 27.5% (4.11 million) of high students who reported using
e-cigarettes in 2019 [41]. However, despite this recent decline, during 2019 -
2020, the use of youth-appealing, low-priced disposable e-cigarette devices
increased approximately 1,000% (from 2.4% to 26.5%) among high school current
e-cigarette users [41]. In addition, more than eight in 10 teenage e-cigarette
users reported consuming flavored e-cigarettes [41]. E-cigarettes can harm the
adolescent brain and increase susceptibility to tobacco addiction [42, 43,
44]. As a result, youth who use e-cigarettes are more likely to subsequently
try combustible cigarettes [45]. Beyond addiction, e-cigarettes pose a risk of
breathing difficulties, inflammatory reactions, lowered defense against
pathogens and lung diseases [46, 47]. Exposure to visual posts featuring
e-cigarette products on social media, including promotional images and videos,
has been associated with increased e-cigarette use among U.S. adolescents [48,
49, 50, 51, 44]. Instagram, one of the most popular social media platforms
among adolescents, is considered the largest source of e-cigarette social
media advertisements (cite). E-cigarette stores, distributors and social media
influencers - users with large followings who post vaping content on behalf of
e-cigarette brands - promote ENDS (Electronic Nicotine Delivery Systems)
products on Instagram and other social media [52].

In August 2018, the U.S. Food & Drug Administration (FDA) introduced a
requirement that e-cigarette advertisements, including social media imagery,
contain a prominent warning label that reminds consumers that nicotine is
addictive. The FDA requires that the warning statement appears on the upper
portion of the advertisement, occupies at least 20 percent of the
advertisement area, and be printed in in conspicuous and legible at least
12-point sans serif (e.g. Helvetica or Arial bold) font size [53]. Several
studies [52, 54] evaluated compliance with the 2018 FDA requirements for
warning labels on Instagram. [52] manually reviewed 2,000 images posted in
2019 and found that only 7% included FDA-mandated warning statements. Posts
uploaded from locations within the U.S. had the highest prevalence of warning
labels, while posts uploaded from other countries were less likely to include
warnings. Most of the international posts featured vaping products distributed
in the U.S. and would therefore still be subject to compliance with the FDA
warning-label regulations [52]. [54] manually reviewed 1,000 posts collected
in late 2018-early 2020 and found that only 13% included warning statements.

Both studies [52, 54] conducted qualitative analysis to assess the presence of
warning statements on a small sample size of Instagram posts and used binary
classification (presence or absence of a warning statement) without reporting
warning labels that were too small or in the wrong place, which would
constitute a partial compliance with the FDA requirements. This study
addresses the limitation of prior research by developing an automated deep
learning image classification capable of quickly and accurately measuring
compliance with the FDA requirements for warning labels on a large sample
size. i.e. thousands of images. We tested whether advanced deep learning
techniques could provide an effective, automated method to track vaping-
related posts on Instagram and evaluate compliance with FDA warning label
requirements.

##  2 Related work

Automated analysis of social media content has exploded in popularity in
recent years, with relevant comparable studies falling into two major groups:
social influence & marketing focused and health focused. Starting with social
influence, which is broader and typically disregards domain, multimodal
learning has been deployed in the recent few years for predicting image
popularity. Before the rise of deep learning, support vector machines were
primarily used, leveraging basic image features such as color and saturation
[55]. In recent years, the focus has turned to the use of deep learning to
synthesize information and learn feature representations directly from the
data. Hu et al. utilize both tags and images [56], leveraging the Yahoo Flickr
Creative Commons 100M dataset. Zohourian and colleagues predicted the number
of likes via context information [57]. De et al. use standard deep neural
networks (DNN) to predict future popularity of the Instagram of a lifestyle
magazine [58]. Each of these cases demonstrate the inclusion of different
features in regards to metadata, context, time, and social indicators.

However, by far the largest specific domain of interest is related to health.
This can be further separated into a) mental health, and b) public health. For
instance, multimodal learning has been used to detect depression on Instagram,
using a convolutional neural network with synthesizes the aforementioned
temporal features [59]. A related, more general study is integrating text and
images to determine the intent of a post [60], including the promotion of
vaping.

Apart from mental health, researchers have also targeted illicit activities
related to public health. Yang and Luo used inductive transfer learning, with
multi-staged regression using post and account related data [61]. The model
was a standard DNN but with lower levels shared among different tasks, typical
of shared learning tasks, and found it more effective than prior approaches
with less modalities. The potential for abuse of social media [62, 63, 64],
especially in the context of the COVID-19 pandemic [65, 66], has also driven
the interest in multimodal machine learning. Mackey and colleagues studied
suspicious COVID-19 related health products by combining natural language
processing and deep learning to produce a binary classifier [67]. In addition
to a useful temporal account of how products related to waves of infection,
they demonstrated the efficacy of deep learning in identifying buying/selling
intent.

##  3 Data and methods

###  3.1 Training data

The dataset consisted of 4,363 images collected from 3,484 distinct posts. Our
prediction targets consisted of three binary labels: the presence of a
correctly placed and sized fully-compliant warning label, a non-fully-
compliant warning label, and whether the post promoted vaping or did not.

###  3.2 Evaluation

The dataset was randomly divided into a 60% training set, a 20% validation set
for early stopping, and a 20% test set for evaluation of model performance.
This splitting procedure was conducted at the post level rather than the image
level, so that multiple images from the same post would all be assigned to the
same split. Models were optimized to minimize cross-entropy loss on the binary
labels, equivalent to negative log-likelihood loss in logistic regression, and
were assessed for accuracy and area under the curve (AUC).

###  3.3 Deep learning models

Our baseline model was a transfer learning-based convolutional neural network
(CNN) built in Tensorflow Keras [68, 69]. The output of the image backbone
model was passed through a global averaging layer, a dropout of 40% or 50% was
applied to mitigate overfitting, and then passed to the output layer. The
model was trained with the Adam optimizer [77] using cross-entropy loss and a
batch size of 32 where possible, or 16 when limited by GPU memory
(EfficientNet B3).

We tested an EfficientNet model [70] as the primary image processing backbone
with pretrained weights from ImageNet [71]. To provide benchmark results, we
compared EfficientNet to popular alternative architectures that have shown
good performance in related research: VGG16 [72], ResNet50 [73], Inceptionv3
[74], and MobileNet [75].

In the bias initialization variation, the bias parameter for each output node
was initialized to $\log(\textrm{positives}/\textrm{negatives})$ to hasten
training convergence with class imbalance [76].

In the progressive unfreezing variation, the image backbone model was
initially frozen and the output head was trained at a relatively higher
learning rate to improve the randomly initialized weights, up to 30 epochs
with a patience of 2 epochs. Then the last 20% of layers of the image backbone
were unfrozen and the model was trained for up to 30 additional epochs with a
reduced learning rate and early stopping with a patience of 3 epochs. Finally
the entire image backbone was unfrozen and the model was trained at an even
lower learning rate for up to 30 epochs, with a patience of 3 epochs.

The multitask design (Figure 1) was implemented by including three nodes with
sigmoid activation in the output layer, one for each of our binary outcomes.

Figure 1: Image-only multitask model architecture.

Given that the warning label in an Instagram image consists entirely of text,
we hypothesized that sufficiently accurate scene text recognition (STR) could
increase the accuracy of our model. Scene text recognition is a variant of
optical character recognition (OCR) designed to detect and extract arbitrary
text in photographs with complex backgrounds, whereas OCR is designed to
extract more standardized text from scanned documents or books [78]. We used
the CRAFT method [79] to detect the location of text in the images, followed
by a convolutional recurrent neural network (CRNN) model to recognize the
specific text characters, which was pretrained on the MJSynth dataset [80].

##  4 Results

Our final model achieved an area under the curve (AUC) and [accuracy] of 0.97
[92%] on vaping classification, 0.99 [99%] on FDA-compliant warning labels,
and 0.94 [97%] on non-compliant warning labels, indicating that our sample
size and modeling approach was sufficient to achieve excellent predictive
accuracy.

We found that the image backbone architecture was the single most important
factor affecting predictive performance. Looking at single-task results, we
found that a VGG16 visual processing backbone led to an overall cross-entropy
loss of 0.53. ResNet50 was comparable at 0.51 loss. By contrast, the simplest
version of EfficientNet (B0) achieved a loss of 0.36. This ranking confirmed
our expectations, mirroring the performance of these models in recent years’
benchmarks.

We did not find a benefit from using higher capacity, more complex versions of
the EfficientNet architecture. Upgrading from the B0 model (224x244
resolution, 5.3M parameters) to the B3 model (300x300 resolution, 12M
parameters) led to an effectively equivalent cross-entropy loss of 0.38. The
B5 and B7 versions showed similar results. This may be a reflection of problem
complexity, sample size limitations, or perhaps suboptimal training strategy
for the larger models. Given its computational efficiency, it was advantageous
for our speed of experimentation that the lower-parameter B0 version yielded
excellent performance.

Initializing the bias weights for each output node to account for class
imbalance was beneficial for both lower cross-entropy loss and faster
convergence, although a less important consideration than the base
architecture with a typical improvement of only 0.02 in cross-entropy.

Many variations that we tested did not show any noticeable performance
benefit. A multitask architecture, learning rate finding, freezing of
batchnorm layers, and progressize layer unfreezing all yielded no
improvements, despite being considered best practices in the deep learning
literature. These findings reinforce the need to verify that common training
practices do show benefit for the particular dataset and task at hand.

##  5 Discussion

Of the three tasks, the identification of FDA-compliant warning labels was
easiest for the deep learning models in terms of area under the receiver
operating characteristic curve (AUC), even when using only raw pixel
information (i.e., no scene text extraction). That is reasonable because the
guideline-compliant warning label is featured prominently in the image, making
it straightforward for convolutional filters to detect. Determining if an
image was promoting vaping proved to be the most difficult task; increasing
our sample size and variety of vaping and non-vaping images should be a
straightforward way to improve model performance on that task.

The critical role of the image backbone in driving performance highlights the
importance of using a recent backbone architecture when accuracy matters. The
rapid innovation in computer vision leads to improved architectures on a
yearly basis, or sometimes even more quickly. It also suggests that accuracy
results should be interpreted as conditional on the selected backbone
architecture, rather than representing a static, final result for a particular
dataset. New innovations in computer vision architectures can be reasonably
expected to reduce overfitting and increase predictive accuracy over time,
even with the same training dataset.

We did not find any images with pseudo-warning text, which would have the
initial appearance of being a compliant warning label but that actually
contained non-warning language. Such adversarial strategies could lead to
false positives in models that rely purely on general convolutional pixels.
Incorporation of scene text recognition that is localized to the warning label
box would provide robustness to such adversarial strategies, and can be
explored in future work. Image augmentation that includes the automatic
generation of adversarial labels could also be used to ensure that models are
not vulnerable to fake warning labels.

A key limitation of the current study is that we used a single test set for
performance evaluation. In future work we plan to use 5-fold cross-validation,
facilitating the construction of confidence intervals and eliminating the
reliance on a single test set.

In future work we plan to incorporate the extracted text from scene text
recognition into the deep learning architecture, which we have successfully
implemented for this dataset. Given our excellent current results, there may
not be much remaining benefit for these three tasks by adding the recognized
scene text. Nevertheless, such extracted text could prove useful for
extracting marketing themes, identifying promoted e-cigarette flavors or
brands, and other more granular analysis. An example Instagram image with
scene text recognition is shown in Figure 2.

(a) Original image

(b) Scene text recognition visualized

Figure 2: Example of scene text recognition in an Instagram image.

The network architecture for our expanded model with scene text recognition is
shown in Figure 3.

Figure 3: Image and scene text multitask model architecture.

##  6 Conclusion

In this study, we demonstrated that deep learning models can successfully
identify FDA-compliant e-cigarette warning labels (or lack thereof) on
Instagram with high accuracy. The models were further able to determine if a
post promoted vaping or did not, and if it contained a non-compliant warning
label. In combination, the models allow the automatic tracking of vaping-
related posts on Instagram and evaluation of their compliance with FDA
marketing policies. Through experimentation across several dimensions, we
identified combinations of model architectures and training strategies that
were most successful at achieving high accuracy. Future research using deep
learning based detection is warranted to evaluate compliance with the FDA
warning statements on images and videos on Instagram and TikTok, the most
popular social media platforms among teenagers. In particular, the
incorporation of scene text recognition has the potential to yield additional
granularity for establishing marketing themes and identifying specific brands
and flavors.

## Funding

The authors did not receive any funding for this research.

[sorting=nyt]

## References

  * [1] Martin Abadi et al.  “Tensorflow: A system for large-scale machine learning”  In _12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16)_, 2016, pp. 265–283
  * [2] Jon-Patrick Allem and Emilio Ferrara  “The importance of debiasing social media data to better understand e-cigarette-related attitudes and behaviors”  In _Journal of medical Internet research_ 18.8 JMIR Publications Inc., Toronto, Canada, 2016, pp. e219
  * [3] Jon-Patrick Allem and Emilio Ferrara  “Could social bots pose a threat to public health?”  In _American journal of public health_ 108.8 American Public Health Association, 2018, pp. 1005–1006
  * [4] Jon-Patrick Allem, Emilio Ferrara, Sree Priyanka Uppu, Tess Boley Cruz and Jennifer B Unger  “E-cigarette surveillance with social media data: social bots, emerging topics, and trends”  In _JMIR public health and surveillance_ 3.4 JMIR Publications Inc., Toronto, Canada, 2017, pp. e98
  * [5] Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun and Hwalsuk Lee  “Character region awareness for text detection”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9365–9374
  * [6] Emily Chen, Kristina Lerman and Emilio Ferrara  “Tracking Social Media Discourse About the COVID-19 Pandemic: Development of a Public Coronavirus Twitter Data Set”  In _JMIR Public Health and Surveillance_ 6.2 JMIR Publications Inc., Toronto, Canada, 2020, pp. e19273
  * [7] Chun Yueh Chiu, Hsien Yuan Lane, Jia Ling Koh and Arbee LP Chen  “Multimodal depression detection on instagram considering time interval of posts”  In _Journal of Intelligent Information Systems_ 56.1 Springer, 2021, pp. 25–47
  * [8] Francois Chollet  “Deep learning with Python”  Manning New York, 2018
  * [9] Shaunak De, Abhishek Maity, Vritti Goel, Sanjay Shitole and Avik Bhattacharya  “Predicting the popularity of instagram posts for a lifestyle magazine using deep learning”  In _2017 2nd International Conference on Communication Systems, Computing and IT Applications (CSCITA)_ , 2017, pp. 174–177  IEEE
  * [10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li and Li Fei-Fei  “Imagenet: A large-scale hierarchical image database”  In _2009 IEEE conference on computer vision and pattern recognition_ , 2009, pp. 248–255  Ieee
  * [11] John-Anthony Fraga  “The Dangers of Juuling”  In _National Center for Health Research. Available online at: http://www.center4research. org/the-dangers-of-juuling_ , 2019
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [13] US Department Health and Human Services  “E-cigarette use among youth and young adults: A report of the Surgeon General”  US Department of HealthHuman Services, Centers for Disease Control and …, 2016
  * [14] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto and Hartwig Adam  “Mobilenets: Efficient convolutional neural networks for mobile vision applications”  In _arXiv preprint arXiv:1704.04861_ , 2017
  * [15] Jiani Hu, Toshihiko Yamasaki and Kiyoharu Aizawa  “Multimodal learning for image popularity prediction on social media”  In _2016 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW)_ , 2016, pp. 1–2  IEEE
  * [16] Max Jaderberg, Karen Simonyan, Andrea Vedaldi and Andrew Zisserman  “Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition”  In _Workshop on Deep Learning, NIPS_ , 2014
  * [17] Andrej Karpathy  “A recipe for training neural networks”  In _Karpathy. github. io_ , 2019
  * [18] Aditya Khosla, Atish Das Sarma and Raffay Hamid  “What makes an image popular?”  In _Proceedings of the 23rd international conference on World wide web_ , 2014, pp. 867–876
  * [19] Minji Kim, Lucy Popova, Bonnie Halpern-Felsher and Pamela M Ling  “Effects of e-cigarette advertisements on adolescents’ perceptions of cigarettes”  In _Health communication_ 34.3 Taylor & Francis, 2019, pp. 290–297
  * [20] Andrea C King, Lia J Smith, Daniel J Fridberg, Alicia K Matthews, Patrick J McNamara and Dingcai Cao  “Exposure to electronic nicotine delivery systems (ENDS) visual imagery increases smoking urge and desire.”  In _Psychology of Addictive Behaviors_ 30.1 American Psychological Association, 2016, pp. 106
  * [21] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _arXiv preprint arXiv:1412.6980_ , 2014
  * [22] Julia Kruk, Jonah Lubin, Karan Sikka, Xiao Lin, Dan Jurafsky and Ajay Divakaran  “Integrating text and image: Determining multimodal document intent in instagram posts”  In _arXiv preprint arXiv:1904.09073_ , 2019
  * [23] Linnea I Laestadius, Megan M Wahl, Julia Vassey and Young Ik Cho  “Compliance with FDA nicotine warning statement provisions in e-liquid promotion posts on Instagram”  In _Nicotine & Tobacco Research_, 2020
  * [24] Shangbang Long, Xin He and Cong Yao  “Scene text detection and recognition: The deep learning era”  In _International Journal of Computer Vision_ 129 Springer, 2021, pp. 161–184
  * [25] Tim Ken Mackey, Jiawei Li, Vidya Purushothaman, Matthew Nali, Neal Shah, Cortni Bardier, Mingxiang Cai and Bryan Liang  “Big Data, Natural Language Processing, and Deep Learning to Detect and Characterize Illicit COVID-19 Product Sales: Infoveillance Study on Twitter and Instagram”  In _JMIR public health and surveillance_ 6.3 JMIR Publications Inc., Toronto, Canada, 2020, pp. e20794
  * [26] Erin K Maloney and Joseph N Cappella  “Does vaping in e-cigarette advertisements affect tobacco smoking urge, intentions, and perceptions in daily, intermittent, and former smokers?”  In _Health communication_ 31.1 Taylor & Francis, 2016, pp. 129–138
  * [27] Pallav Pokhrel, Pebbles Fagan, Thaddeus A Herzog, Linnea Laestadius, Wayne Buente, Crissy T Kawamoto, Hye-Ryeon Lee and Jennifer B Unger  “Social media e-cigarette exposure and e-cigarette expectancies and use among young adults”  In _Addictive behaviors_ 78 Elsevier, 2018, pp. 51–58
  * [28] R Redfield  “CDC director’s statement on the first death related to the outbreak of severe lung disease in people who use e-cigarette or “vaping” devices”  Centers for Disease ControlPrevention, 2019  URL: https://www.cdc.gov/media/releases/2019/s0823-vaping-related-death.html
  * [29] Karen Simonyan and Andrew Zisserman  “Very deep convolutional networks for large-scale image recognition”  In _arXiv preprint arXiv:1409.1556_ , 2014
  * [30] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens and Zbigniew Wojna  “Rethinking the inception architecture for computer vision”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826
  * [31] Mingxing Tan and Quoc Le  “Efficientnet: Rethinking model scaling for convolutional neural networks”  In _International Conference on Machine Learning_ , 2019, pp. 6105–6114  PMLR
  * [32] The U.S. Food and Drug Administration  “FDA launches new, comprehensive campaign to warn kids about the dangers of e-cigarette use as part of agency’s Youth Tobacco Prevention Plan, amid evidence of sharply rising use among kids”, 2018  URL: https://www.fda.gov/news-events/press-announcements/fda-launches-new-comprehensive-campaign-warn-kids-about-dangers-e-cigarette-use-part-agencys-youth
  * [33] The U.S. Food and Drug Administration  “Understanding the Health Impact and Dangers of Smoke and ‘Vapor.”’, 2019  URL: https://www.fda.gov/news-events/fda-voices-perspectives-fda-leadership-and-experts/understanding-health-impact-and-dangers-smoke-and-vapor
  * [34] The U.S. Food and Drug Administration  “Tobacco products: advertising and promotion guidelines”, 2020  URL: https://www.fda.gov/tobacco-products/products-guidance-regulations/advertising-and-promotion
  * [35] Julia Vassey, Catherine Metayer, Chris J Kennedy and Todd P Whitehead  “#Vape: measuring e-cigarette influence on Instagram with deep learning and text analysis”  In _Frontiers in Communication_ 4 Frontiers, 2020, pp. 75
  * [36] Teresa W Wang et al.  “Tobacco product use and associated factors among middle and high school students—United States, 2019”  In _MMWR Surveillance Summaries_ 68.12 Centers for Disease ControlPrevention, 2019, pp. 1
  * [37] Teresa W Wang, Linda J Neff, Eunice Park-Lee, Chunfeng Ren, Karen A Cullen and Brian A King  “E-cigarette use among middle and high school students—United States, 2020”  In _Morbidity and Mortality Weekly Report_ 69.37 Centers for Disease ControlPrevention, 2020, pp. 1310
  * [38] Xitong Yang and Jiebo Luo  “Tracking illicit drug dealing and abuse on Instagram using multimodal analysis”  In _ACM Transactions on Intelligent Systems and Technology (TIST)_ 8.4 ACM New York, NY, USA, 2017, pp. 1–15
  * [39] Lindsay E Young et al.  “Disrupting the COVID-19 Misinfodemic With Network Interventions: Network Solutions for Network Problems”  In _American journal of public health_ American Public Health Association, 2021, pp. e1–e6
  * [40] Alireza Zohourian, Hedieh Sajedi and Arefeh Yavary  “Popularity prediction of images and videos on Instagram”  In _2018 4th International Conference on Web Research (ICWR)_ , 2018, pp. 111–117  IEEE

## References

  * [41] Teresa W Wang, Linda J Neff, Eunice Park-Lee, Chunfeng Ren, Karen A Cullen and Brian A King  “E-cigarette use among middle and high school students—United States, 2020”  In _Morbidity and Mortality Weekly Report_ 69.37 Centers for Disease ControlPrevention, 2020, pp. 1310
  * [42] US Department Health and Human Services  “E-cigarette use among youth and young adults: A report of the Surgeon General”  US Department of HealthHuman Services, Centers for Disease Control and …, 2016
  * [43] John-Anthony Fraga  “The Dangers of Juuling”  In _National Center for Health Research. Available online at: http://www.center4research. org/the-dangers-of-juuling_ , 2019
  * [44] Teresa W Wang et al.  “Tobacco product use and associated factors among middle and high school students—United States, 2019”  In _MMWR Surveillance Summaries_ 68.12 Centers for Disease ControlPrevention, 2019, pp. 1
  * [45] The U.S. Food and Drug Administration  “FDA launches new, comprehensive campaign to warn kids about the dangers of e-cigarette use as part of agency’s Youth Tobacco Prevention Plan, amid evidence of sharply rising use among kids”, 2018  URL: https://www.fda.gov/news-events/press-announcements/fda-launches-new-comprehensive-campaign-warn-kids-about-dangers-e-cigarette-use-part-agencys-youth
  * [46] R Redfield  “CDC director’s statement on the first death related to the outbreak of severe lung disease in people who use e-cigarette or “vaping” devices”  Centers for Disease ControlPrevention, 2019  URL: https://www.cdc.gov/media/releases/2019/s0823-vaping-related-death.html
  * [47] The U.S. Food and Drug Administration  “Understanding the Health Impact and Dangers of Smoke and ‘Vapor.”’, 2019  URL: https://www.fda.gov/news-events/fda-voices-perspectives-fda-leadership-and-experts/understanding-health-impact-and-dangers-smoke-and-vapor
  * [48] Andrea C King, Lia J Smith, Daniel J Fridberg, Alicia K Matthews, Patrick J McNamara and Dingcai Cao  “Exposure to electronic nicotine delivery systems (ENDS) visual imagery increases smoking urge and desire.”  In _Psychology of Addictive Behaviors_ 30.1 American Psychological Association, 2016, pp. 106
  * [49] Erin K Maloney and Joseph N Cappella  “Does vaping in e-cigarette advertisements affect tobacco smoking urge, intentions, and perceptions in daily, intermittent, and former smokers?”  In _Health communication_ 31.1 Taylor & Francis, 2016, pp. 129–138
  * [50] Pallav Pokhrel, Pebbles Fagan, Thaddeus A Herzog, Linnea Laestadius, Wayne Buente, Crissy T Kawamoto, Hye-Ryeon Lee and Jennifer B Unger  “Social media e-cigarette exposure and e-cigarette expectancies and use among young adults”  In _Addictive behaviors_ 78 Elsevier, 2018, pp. 51–58
  * [51] Minji Kim, Lucy Popova, Bonnie Halpern-Felsher and Pamela M Ling  “Effects of e-cigarette advertisements on adolescents’ perceptions of cigarettes”  In _Health communication_ 34.3 Taylor & Francis, 2019, pp. 290–297
  * [52] Julia Vassey, Catherine Metayer, Chris J Kennedy and Todd P Whitehead  “#Vape: measuring e-cigarette influence on Instagram with deep learning and text analysis”  In _Frontiers in Communication_ 4 Frontiers, 2020, pp. 75
  * [53] The U.S. Food and Drug Administration  “Tobacco products: advertising and promotion guidelines”, 2020  URL: https://www.fda.gov/tobacco-products/products-guidance-regulations/advertising-and-promotion
  * [54] Linnea I Laestadius, Megan M Wahl, Julia Vassey and Young Ik Cho  “Compliance with FDA nicotine warning statement provisions in e-liquid promotion posts on Instagram”  In _Nicotine & Tobacco Research_, 2020
  * [55] Aditya Khosla, Atish Das Sarma and Raffay Hamid  “What makes an image popular?”  In _Proceedings of the 23rd international conference on World wide web_ , 2014, pp. 867–876
  * [56] Jiani Hu, Toshihiko Yamasaki and Kiyoharu Aizawa  “Multimodal learning for image popularity prediction on social media”  In _2016 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW)_ , 2016, pp. 1–2  IEEE
  * [57] Alireza Zohourian, Hedieh Sajedi and Arefeh Yavary  “Popularity prediction of images and videos on Instagram”  In _2018 4th International Conference on Web Research (ICWR)_ , 2018, pp. 111–117  IEEE
  * [58] Shaunak De, Abhishek Maity, Vritti Goel, Sanjay Shitole and Avik Bhattacharya  “Predicting the popularity of instagram posts for a lifestyle magazine using deep learning”  In _2017 2nd International Conference on Communication Systems, Computing and IT Applications (CSCITA)_ , 2017, pp. 174–177  IEEE
  * [59] Chun Yueh Chiu, Hsien Yuan Lane, Jia Ling Koh and Arbee LP Chen  “Multimodal depression detection on instagram considering time interval of posts”  In _Journal of Intelligent Information Systems_ 56.1 Springer, 2021, pp. 25–47
  * [60] Julia Kruk, Jonah Lubin, Karan Sikka, Xiao Lin, Dan Jurafsky and Ajay Divakaran  “Integrating text and image: Determining multimodal document intent in instagram posts”  In _arXiv preprint arXiv:1904.09073_ , 2019
  * [61] Xitong Yang and Jiebo Luo  “Tracking illicit drug dealing and abuse on Instagram using multimodal analysis”  In _ACM Transactions on Intelligent Systems and Technology (TIST)_ 8.4 ACM New York, NY, USA, 2017, pp. 1–15
  * [62] Jon-Patrick Allem and Emilio Ferrara  “The importance of debiasing social media data to better understand e-cigarette-related attitudes and behaviors”  In _Journal of medical Internet research_ 18.8 JMIR Publications Inc., Toronto, Canada, 2016, pp. e219
  * [63] Jon-Patrick Allem, Emilio Ferrara, Sree Priyanka Uppu, Tess Boley Cruz and Jennifer B Unger  “E-cigarette surveillance with social media data: social bots, emerging topics, and trends”  In _JMIR public health and surveillance_ 3.4 JMIR Publications Inc., Toronto, Canada, 2017, pp. e98
  * [64] Jon-Patrick Allem and Emilio Ferrara  “Could social bots pose a threat to public health?”  In _American journal of public health_ 108.8 American Public Health Association, 2018, pp. 1005–1006
  * [65] Emily Chen, Kristina Lerman and Emilio Ferrara  “Tracking Social Media Discourse About the COVID-19 Pandemic: Development of a Public Coronavirus Twitter Data Set”  In _JMIR Public Health and Surveillance_ 6.2 JMIR Publications Inc., Toronto, Canada, 2020, pp. e19273
  * [66] Lindsay E Young et al.  “Disrupting the COVID-19 Misinfodemic With Network Interventions: Network Solutions for Network Problems”  In _American journal of public health_ American Public Health Association, 2021, pp. e1–e6
  * [67] Tim Ken Mackey, Jiawei Li, Vidya Purushothaman, Matthew Nali, Neal Shah, Cortni Bardier, Mingxiang Cai and Bryan Liang  “Big Data, Natural Language Processing, and Deep Learning to Detect and Characterize Illicit COVID-19 Product Sales: Infoveillance Study on Twitter and Instagram”  In _JMIR public health and surveillance_ 6.3 JMIR Publications Inc., Toronto, Canada, 2020, pp. e20794
  * [68] Francois Chollet  “Deep learning with Python”  Manning New York, 2018
  * [69] Martin Abadi et al.  “Tensorflow: A system for large-scale machine learning”  In _12th $\\{$USENIX$\\}$ symposium on operating systems design and implementation ($\\{$OSDI$\\}$ 16)_, 2016, pp. 265–283
  * [70] Mingxing Tan and Quoc Le  “Efficientnet: Rethinking model scaling for convolutional neural networks”  In _International Conference on Machine Learning_ , 2019, pp. 6105–6114  PMLR
  * [71] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li and Li Fei-Fei  “Imagenet: A large-scale hierarchical image database”  In _2009 IEEE conference on computer vision and pattern recognition_ , 2009, pp. 248–255  Ieee
  * [72] Karen Simonyan and Andrew Zisserman  “Very deep convolutional networks for large-scale image recognition”  In _arXiv preprint arXiv:1409.1556_ , 2014
  * [73] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [74] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens and Zbigniew Wojna  “Rethinking the inception architecture for computer vision”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 2818–2826
  * [75] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto and Hartwig Adam  “Mobilenets: Efficient convolutional neural networks for mobile vision applications”  In _arXiv preprint arXiv:1704.04861_ , 2017
  * [76] Andrej Karpathy  “A recipe for training neural networks”  In _Karpathy. github. io_ , 2019
  * [77] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _arXiv preprint arXiv:1412.6980_ , 2014
  * [78] Shangbang Long, Xin He and Cong Yao  “Scene text detection and recognition: The deep learning era”  In _International Journal of Computer Vision_ 129 Springer, 2021, pp. 161–184
  * [79] Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun and Hwalsuk Lee  “Character region awareness for text detection”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9365–9374
  * [80] Max Jaderberg, Karen Simonyan, Andrea Vedaldi and Andrew Zisserman  “Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition”  In _Workshop on Deep Learning, NIPS_ , 2014