# Imputation of missing values in multi-view data

Wouter van Loon  Department of Methodology and Statistics, Leiden University
Marjolein Fokkema  Department of Methodology and Statistics, Leiden University
Mark de Rooij  Department of Methodology and Statistics, Leiden University

(October 26, 2022)

###### Abstract

When missing values occur in multi-view data, all features in a view are
likely to be missing simultaneously. This leads to very large quantities of
missing data which, especially when combined with high-dimensionality, makes
the application of conditional imputation methods computationally infeasible.
We introduce a new meta-learning imputation method based on stacked penalized
logistic regression (StaPLR), which performs imputation in a dimension-reduced
space. We evaluate the new imputation method with several imputation
algorithms using simulations. The results show that meta-level imputation of
missing values leads to good results at a much lower computational cost, and
makes the use of advanced imputation algorithms such as missForest and
predictive mean matching possible in settings where they would otherwise be
computationally infeasible.

keywords missing data, imputation, multi-view learning, stacked
generalization, feature selection

##  1 Introduction

Multi-view data refers to any data set where the features have been divided
into distinct feature sets [1, 2]. Such data sets are particularly common in
the biomedical domain where these feature sets, commonly called views, often
correspond to different data sources or modalities [3, 4, 5, 6].
Classification models of disease using information from multiple views
generally lead to better performance than models using only a single view [7,
8, 9, 10, 11, 12]. Traditionally, information from different views is often
combined using simple feature concatenation, where the features corresponding
to different views are simply aggregated into a single feature matrix, so that
traditional machine learning methods can be deployed [3]. More recently,
dedicated multi-view machine learning techniques have been developed, which
are specifically designed to handle the multi-view structure of the data [2,
3]. One such multi-view learning technique is stacked penalized logistic
regression (StaPLR) [13]. In addition to improving classification performance,
StaPLR can automatically select the views most relevant for prediction [13,
14, 15].

In practice, not all views may be observed for all subjects. When confronted
with missing views, typical approaches are to remove any subjects with at
least one missing value from the data set (called list-wise deletion or
complete case analysis (CCA)), or to replace missing values by some
substituted value, a process known as imputation. In biomedical studies, a
single view may consist of thousands or even millions of features. With the
traditional approach of feature concatenation, in the presence of missing
views, CCA leads to a massive loss of information, while imputation may be
computationally infeasible. In this article we propose a new method for
dealing with missing views, based on the StaPLR algorithm. We show how this
method requires much less computation by imputing missing values in a
dimension-reduced space, rather than in the original feature space. We compare
our proposed meta-level imputation method with imputation methods applied in
the original feature space.

##  2 Methods

###  2.1 Methods for handling missing values

Missing values are often divided into three categories: missing completely at
random (MCAR), missing at random (MAR), or missing not at random (MNAR).
Values are said to be MCAR if the causes of the missingness are unrelated to
both missing and observed data [16]. Examples include random machine failure,
or missingness introduced by analyzing a random sub-sample of the data. If the
missingness is not completely random but depends only on observed data, the
missing values are said to be MAR [16]. If the missingness instead depends on
unobserved factors, the missing values are said to be MNAR [16]. Here, we will
focus on MCAR missing values.

The simplest way of dealing with MCAR missing values is to discard
observations with at least one missing value through complete case analysis.
However, this approach is potentially very wasteful since a single missing
value causes an entire observation to be removed from the data. CCA may
therefore remove many more observed values from the data than the number of
values initially missing, and drastically reduce the sample size.

To prevent wasting observed data, missing values can be imputed. The simplest
form of imputation is to simply replace each missing value by the
unconditional mean of the feature, a procedure known as (unconditional) mean
imputation (MI). If one is primarily interested in prediction, MI has some
favorable properties: Its computational cost is extremely small, and it has
been shown that MI is universally consistent for prediction even for MAR data,
as long as the learning algorithm used is also universally consistent [17].
Here consistent means that, given an infinite amount of training data, the
prediction function achieves the error rate of the best possible prediction
function (i.e. the Bayes rate), while universal means that the procedure is
consistent for all possible data distributions [17]. However, MI is often
criticised because it is known to distort the data distribution by attenuating
existing correlations between the features, underestimating the variance, and
causing bias in almost any estimate other than the mean [16]. More
sophisticated imputation methods include Bayesian linear regression,
predictive mean matching (PMM) [16], and missForest (MF) [18]. These methods
are multiple imputation methods, meaning multiple imputed data sets are formed
so that correct variance estimates can be obtained [16].

It should be noted that other methods for handling missing data exist which do
not explicitly impute missing values. These methods incorporate the missing
data handling directly into the model fitting procedure and include
likelihood-based methods such as full information maximum likelihood (FIML)
[19, 20] for parametric regression models, and missingness incorporated in
attributes (MIA) [21] for decision trees. However, these methods are less
broadly applicable than imputation methods [20, 16, 17] and we do not consider
them here.

The literature on the imputation of missing values is vast, and we do not aim
to give a complete overview here. Instead, we divide the existing imputation
methods in two broad classes: unconditional and conditional imputation. We
define a unconditional imputation method as any method in which the imputation
of a missing value is based solely on other observations of the same feature,
i.e., the imputation takes place within a single column of the feature matrix.
The aforementioned mean imputation is a classical example of an unconditional
imputation method. By contrast, a conditional imputation method is any method
in which the imputation of a missing value is based, in part or completely, on
observations of other features, i.e., the imputation uses different columns of
the feature matrix. Most sophisticated imputation methods, such as Bayesian
multiple imputation and PMM, are conditional imputation methods. The
distinction between unconditional and conditional imputation methods is of
particular interest for feature selection. Unconditional imputation methods,
such as mean substitution, use only the univariate distributions for
imputation, so that the imputed feature remains in some sense “pure” and free
from contamination from other features. However, as mentioned earlier, mean
substitution is known to distort the data distribution by attenuating existing
correlations between the features [16].

By contrast, some (but not all) conditional imputation methods preserve the
correlations between features [16]. However, in this case the imputed values
depend on other features in the data. In the event that a selected feature has
a large number of imputed values, this may lead to difficulties in
interpretation, since a large proportion of the selected feature is derived
from other features. Nevertheless, a recent study on the effect of imputation
methods on feature selection suggests sophisticated conditional imputation
methods generally lead to better results than unconditional imputation [22].
Since it is not possible to both perform the imputation independent of other
features and preserve the existing correlations, one has to choose between one
or the other.

###  2.2 From missing features to missing views

In multi-view data, it is likely that missingness will occur at the view
level, rather than at the feature level. Missing views may occur at random
and/or by design. In a study where one of the views corresponds to features
derived from a magnetic resonance imaging (MRI) scan, factors like the MRI
scanner experiencing machine failure, a mistake in the scanning protocol by
the researcher administering the scan, or a subject simply not making it to
their appointment in time due to heavy traffic, would lead to all features of
this view being simultaneously missing. Likewise, if one of the views
corresponds to features derived from a sample of blood or cerebral spinal
fluid (CSF), a lost or contaminated sample would lead to all derived features
being simultaneously missing. Note that in these cases, although the
missingness occurs at the view level, the underlying mechanism is still MCAR.
Another common example of MCAR data occurs in the case of planned missingness,
where the missing values are part of the study design. For example, it may be
considered too expensive to administer an MRI scan to all study participants,
so instead an MRI scan is administered only to a random sub-sample of the
participants. Again the underlying mechanism is MCAR, but all features
corresponding to the MRI scan will be missing simultaneously for the
unmeasured sub-sample. Throughout the rest of this article we will assume that
(1) for each observation, a view is either completely missing or completely
observed, and (2) the missingness is completely at random (MCAR).

Conceptually, one could impute a missing view by first applying feature
concatenation, and then simply applying a chosen imputation method on the
concatenated feature set. However, in practice this may be impossible. For
example, if the missing view is an MRI scan, there could be hundreds of
thousands or even millions of missing values. Similar numbers of missing
values may occur with views corresponding to other neuroimaging techniques,
gene-expression arrays, or other omics-data. With such vast amounts of missing
values, application of sophisticated multiple-imputation methods may be
computationally infeasible. For example, multiple imputation by chained
equations, a very popular method for handling missing data implemented in the
R package mice [23] involves iteratively cycling through a procedure of model-
fitting and generating imputed values from the fitted model. Such iterative
approaches quickly become prohibitive with large numbers of predictor
variables, where the computational load of even a single imputation will
exceed that of fitting the final predictive model of interest. This process is
further complicated by the small sample sizes and thus extreme high-
dimensionality commonly seen in neuroimaging, which means that even in
instances where the imputation would be computationally feasible, the obtained
results may be of poor quality (see Appendix A for further explanation).
However, by analyzing the multi-view data using the StaPLR algorithm, we will
introduce a method of dealing with the missing views in a much less
computationally intensive way.

###  2.3 The StaPLR algorithm

The general StaPLR algorithm was proposed in [13], and further extended to
more than 2 levels of stacking [15]. A range of sparse meta-learners for view
selection were compared in [14]. For clarity, we will assume throughout this
article that (1) there are only 2 levels of stacking, the base (feature)
level, and the meta (view) level, and (2) there is only one base learning
algorithm ($A_{\text{b}}$), which is the same for all views. Under these
assumptions the StaPLR algorithm can be given by Algorithm 1, and displayed as
a flow diagram as in Figure 1. For each view $\bm{X}^{(v)},v=1\dots V$, a
trained classifier $\hat{f}_{v}$ is obtained by applying the base-learning
algorithm $A_{\text{b}}$ to the pair $\bm{X}^{(v)},\bm{y}$. In our case
$A_{\text{b}}$ is an $L_{2}$-penalized logistic regression learner including
cross-validation for the tuning parameter, so that each $\hat{f}_{v}$ is a
fully tuned logistic ridge regression model. For each of the base-classifiers
$\hat{f}_{v},v=1\dots V$, $k$-fold cross-validation is performed to obtain a
vector $\bm{z}^{(v)}$ of estimated out-of-sample predictions. The vectors
$\bm{z}^{(v)},v=1\dots V$, are then concatenated column-wise into the matrix
$\bm{Z}$, which is then used together with outcome $\bm{y}$ to train the meta-
learning algorithm $A_{\text{m}}$ and obtain the meta-classifier
$\hat{f}_{\text{meta}}$. In our case $A_{\text{m}}$ is an $L_{1}$-penalized
nonnegative logistic regression learner including cross-validation for the
tuning parameter, so that each $\hat{f}_{v}$ is a fully tuned logistic
nonnegative lasso model. The final classifier is then given by
$\hat{f}_{\text{stacked}}=\hat{f}_{\text{meta}}\circ(\hat{f}_{\text{1}}\dots\hat{f}_{\text{V}})$.

Data: Views $\bm{X}^{(1)},\dots,\bm{X}^{(V)}$ and outcomes
$\bm{y}=(y_{1},\dots,y_{n})^{T}$.

1 foreach $v$ = 1 to $V$ do

2 $\hat{f}_{v}=A_{\text{b}}(\bm{X}^{(v)},\bm{y})$

3 foreach $k$ = 1 to $K$ do

4 $\hat{f}_{v,k}=A_{\text{b}}(\bm{X}^{(v)}_{i\notin S_{k}},\bm{y}_{i\notin
S_{k}})$

5 $\bm{z}^{(v)}_{k}=\hat{f}_{v,k}(\bm{X}^{(v)}_{i\in S_{k}})$

6 end foreach

7

8 end foreach

9$\bm{Z}=(\bm{z}^{(1)},\dots,\bm{z}^{(V)})$

10 $\hat{f}_{\text{meta}}=A_{\text{m}}(\bm{Z},\bm{y})$

$\hat{f}_{\text{stacked}}=\hat{f}_{\text{meta}}\circ(\hat{f}_{\text{1}}\dots\hat{f}_{\text{V}})$

Algorithm 1 The 2-level StaPLR algorithm, with a single base learner

Figure 1: The 2-level StaPLR algorithm, with a single base learner,
represented as a flow diagram.

###  2.4 Handling missing views through meta-level imputation with StaPLR

As shown in Algorithm 1 and Figure 1, StaPLR performs view selection through
two steps: a dimension reduction step, and a feature selection step. First, a
matrix of cross-validated predictions $\bm{Z}$ is constructed, in which each
view is represented by a single column. The meta-learning algorithm then
performs feature selection in this reduced space, which translates to view
selection in the original feature space. Note that the reduction of
dimensionality is generally very large: If the complete concatenated feature
matrix $\bm{X}=(\bm{X}^{(1)},\dots,\bm{X}^{(V)})$ has dimensionality $n\times
m$, then $\bm{Z}$ has dimensionality $n\times V$, and generally $V\ll m$. This
further implies that, while the original data is usually very high
dimensional, this is often not the case in the reduced space; see for example
van Loon et al. [15] for an application of StaPLR where $n\ll m$, but $n>V$.
Obviously, it is much less computationally intensive to fit models in the
reduced space, than it is to fit them in the original space. It is therefore
highly computationally attractive to perform any missing value imputation in
this reduced space. Furthermore, as long as we assume $n>V$, imputation
methods which would perform poorly in the high-dimensional feature space can
instead be applied in the reduced space.

To apply the dimension reduction, the base-learning algorithm is trained on
each view separately. Given that a view is either completely missing or
completely observed, this means that within each application of the base
learner there are no observations with partially observed data. Thus, the base
learner can simply be trained on the set of complete cases for each view. A
comparison of the proposed method for handling missing views in StaPLR and
feature concatenation is shown in Figure 2.

Figure 2: An example of how missing views can be handled through either
classical feature concatenation (left), or through StaPLR (right). The blank
areas in the data represent missing values. In this example, we assume there
are three views, namely $\bm{X}^{(1)}$, $\bm{X}^{(2)}$, and $\bm{X}^{(3)}$,
with $m_{1}$, $m_{2}$, and $m_{3}$ features, respectively. Assume that there
are $l$ observations which have missing values on $\bm{X}^{(2)}$. In the case
of feature concatenation, we would have to either impute $lm_{2}$ missing
values, or entirely discard $l(m_{1}+m_{3})$ observed values. However, in the
proposed missing data handling method for StaPLR, only $l$ values need to be
imputed.

##  3 Empirical Evaluation

###  3.1 Imputation algorithms

We evaluate the proposed meta-level imputation method by applying three
popular imputation algorithms at the meta-level, and comparing the results
with the application of those same algorithms at the feature level: The first
imputation algorithm is simple unconditional mean imputation (MI), where each
missing value is replaced by the mean of the observed values of the feature.

The second algorithm is predictive mean matching (PMM) [16], a state-of-the-
art multiple imputation algorithm implemented as the default in the R package
mice [23]. PMM is based on Bayesian imputation under the normal linear model
[24, 25, 16], but instead of using a Bayesian draw of the regression
coefficient to directly calculate an imputed value, it is used to find a set
of $d$ candidate ‘donors’, which are the $d$ observations for which the
predicted value under the imputation model is closest to the value drawn for
the missing observation [16, 23]. The imputed value is then a random draw from
the observed values of the donors. Advantages of PMM over direct Bayesian
imputation include that the method allows for discrete target variables, the
imputed values can never go outside the range of the observed data, and the
method is more robust to model misspecification [16]. Disadvantages compared
to other imputation methods include the fact that PMM tends to work poorly in
high-dimensional settings, and that the required computation time is so large
that we can only apply PMM at the meta-level, as feature-level imputation
using PMM proved entirely infeasible (see Appendix A). When applying PMM, we
use the default value of $d=5$ and we also set the number of imputations to
the default value of 5. With multiple imputation comes the need to somehow
combine the results of the different imputed data sets. We evaluate two
strategies for combining the different imputed data sets: The first strategy
is to impute the matrix of cross-validated predictions $\bm{Z}$ 5 times, and
then average the results before training the meta-learner. The second strategy
is to generate the matrix $\bm{Z}$ 5 times, each time using different cross-
validation folds, and then impute each matrix once before averaging the
results. The benefit of the second strategy is that it also averages over
different ways in which observations are allocated to cross-validation folds,
but it is approximately 5 times more computationally expensive. To distinguish
between the two strategies, we will refer to them as mPMM and cvPMM,
respectively.

The third algorithm we consider is missForest (MF), an iterative imputation
method based on random forests [18], as implemented in the R package
missForest. MF can be used to impute both continuous and categorical data, and
by averaging over many unpruned regression or classification trees it is
intrinsically a multiple imputation method, while producing only a single
completed data set [18]. When applying the MF algorithm, we use the default
settings for the R package missForest, which constitutes growing 100 trees in
each forest.

Finally, we also compare the results of the different imputation algorithms
with complete case analysis (CCA), and with an application of StaPLR to the
complete data before any missingness is generated, a process we will call
complete data analysis (CDA).

###  3.2 Simulation design

We generate multi-view data consisting of $n=1000$ observations of $V=4$
disjoint views. Each view $v=1\dots V$ consists of $m_{v}=5\times 10^{v-1}$
random normally distributed features. The mean correlation between features
within the same view is set to $\rho_{w}$ = 0.5. The mean correlation between
features in different views is set to $\rho_{b}$ = 0.2. If view $v$
corresponds to signal, then each feature within that view has a regression
coefficient of $2/\sqrt{m_{v}}$ or $-2/\sqrt{m_{v}}$, each with probability
0.5. If a view corresponds to noise, then each feature within that view has a
regression coefficient of zero. The features and their respective coefficients
together form a linear predictor to which the logistic function is applied to
obtain probabilities. These probabilities are then used to draw the
observations of the binary outcome $y$. The following experimental factors are
varied:

  * •

Fraction of observations with a missing view: A single view is MCAR for either
50% or 90% of the observations.

  * •

Size of the missing view: The missing view is either the smallest ($V_{1}$) or
the largest ($V_{4}$) view.

  * •

Size of the noise view: Either the smallest ($V_{1}$) or the largest ($V_{4}$)
view corresponds to noise, while all other views correspond to signal.

This leads to a total of $2\times 2\times 2=8$ experimental conditions. For
each experimental condition, 100 replications are performed.

###  3.3 Software

All simulations are performed in R (version 4.1.2) [26] on a high-performance
computing cluster running Cent OS (Stream 8) with Slurm Workload Manager
(version 20.11). All pseudo-random number generation is performed using the
Mersenne Twister [27], R’s default algorithm. Imputation with the missForest
algorithm is performed using the R package missForest (version 1.5) [18],
while predictive mean matching is performed using mice (version 3.14) [23].
All StaPLR model fitting is performed using multiview (version 0.3.1) [28],
with a glmnet (version 4.1-4) [29] back-end. Additional scripts, such as those
used for mean imputation and data simulation, are included in the
supplementary materials.

###  3.4 Outcome measures

We compare the predictive performance of the trained stacked classifiers in
terms of two outcome measures calculated on a test set of $n$ = 1000
observations. The first outcome measure is test accuracy, which we define as
the proportion of correctly classified observations in the test set given a
classification threshold of 0.5. The second outcome measure is the mean
squared error of probabilities (MSEP), which we define as:

$\text{MSEP}=\frac{1}{n}\sum_{i=1}^{n}(\hat{p}_{i}-p_{i})^{2},$ (1)

where $p_{i}$ is the true class probability of test set observation $i$. Note
that outside of simulation studies the true class probabilities would be
unknown, and the MSEP cannot be calculated. However, replacing the true
probabilities in the MSEP with the true class labels naturally leads to a
well-known classification measure known as the Brier score [30]. Here we use
the MSEP instead of the Brier score, as it is expected to be more sensitive to
small changes in the data.

To assess view selection performance, we calculate four outcome measures. The
first is simply the mean proportion of correctly selected views. The second
outcome measure is the true positive rate (TPR), which is defined as the
average proportion of views truly related to the outcome that were correctly
selected by the meta-learner. The third outcome measure is the false positive
rate (FPR), which is defined as the average proportion of views not related to
the outcome that were incorrectly selected by the meta-learner. The fourth
outcome measure is the false discovery rate, which is defined as the average
proportion of incorrectly selected views among the set of selected views if at
least one view is selected, and zero otherwise. In addition to classification
and view selection performance, we also calculate the logarithm of the
computation time in seconds for each procedure.

##  4 Results

###  4.1 Predictive performance

Figures 3 and 4 show, respectively, the classification accuracy and MSEP of
the different methods for handling missing data under 90% missingness. Figures
5 and 6 show the same outcome measures under 50% missingness. We will focus on
the results under 90% missingness as they generally show the same patterns of
how the different imputation methods behave, but in an emphasized manner. In
terms of test accuracy, complete case analysis performs worse than all
imputation methods, while the differences between the imputation methods are
comparatively small, although when the noise view consists of many features,
mean imputation appears to perform slightly better than the other imputation
methods.

As expected, the MSEP is more sensitive to changes in the imputed data than
the test accuracy. Again CCA performs the worst, while CDA performs the best.
Mean imputation performs as good as or better than the other imputation
methods. Base-level missForest (MF) performs as good as, or better than meta-
level missForest (mMF) or predictive mean matching (mPMM and cvPMM). When the
noise view consists of a high number of features, mMF, mPMM and cvPMM have
larger variance of the MSEP than the other prediction methods.

For the sake of completeness, we also calculated the binomial deviance. It
supports the same general conclusions, lying in between the accuracy and MSEP
in terms of sensitivity to the imputed data (see Appendix B).

Figure 3: Test accuracy with 90% missingness. CDA = complete data analysis;
CCA = complete case analysis; MI = feature-level mean imputation; MF =
feature-level missForest; mMI = meta-level mean imputation; mMF = meta-level
missForest; mPMM = meta-level predictive mean matching, generating $\bm{Z}$
once; cvPMM = meta-level predictive mean matching, generating $\bm{Z}$ five
times. $V_{1}$ is the smallest view, consisting of 5 features. $V_{4}$ is the
largest view, consisting of 5000 features.

Figure 4: Mean squared error of probabilities (MSEP) with 90% missingness. CDA
= complete data analysis; CCA = complete case analysis; MI = feature-level
mean imputation; MF = feature-level missForest; mMI = meta-level mean
imputation; mMF = meta-level missForest; mPMM = meta-level predictive mean
matching, generating $\bm{Z}$ once; cvPMM = meta-level predictive mean
matching, generating $\bm{Z}$ five times. $V_{1}$ is the smallest view,
consisting of 5 features. $V_{4}$ is the largest view, consisting of 5000
features.

Figure 5: Test accuracy with 50% missingness. CDA = complete data analysis;
CCA = complete case analysis; MI = feature-level mean imputation; MF =
feature-level missForest; mMI = meta-level mean imputation; mMF = meta-level
missForest; mPMM = meta-level predictive mean matching, generating $\bm{Z}$
once; cvPMM = meta-level predictive mean matching, generating $\bm{Z}$ five
times. $V_{1}$ is the smallest view, consisting of 5 features. $V_{4}$ is the
largest view, consisting of 5000 features.

Figure 6: Mean squared error of probabilities (MSEP) with 50% missingness. CDA
= complete data analysis; CCA = complete case analysis; MI = feature-level
mean imputation; MF = feature-level missForest; mMI = meta-level mean
imputation; mMF = meta-level missForest; mPMM = meta-level predictive mean
matching, generating $\bm{Z}$ once; cvPMM = meta-level predictive mean
matching, generating $\bm{Z}$ five times. $V_{1}$ is the smallest view,
consisting of 5 features. $V_{4}$ is the largest view, consisting of 5000
features.

###  4.2 View selection

In terms of the mean proportion of correctly selected views (Table 1), CCA
again performs worse than all imputation methods. Among the imputation
methods, a trend can be observed where mean imputation (both MI and mMI)
performs well when the view to be imputed corresponds to noise, but performs
poorly when the view to be imputed corresponds to signal. The mMF, mPMM and
cvPMM methods all appear to perform similarly. mMF sometimes shows somewhat
better performance than MF, particularly when the missing view has a large
number of features, but the pattern of differences between the two methods is
not entirely consistent.

In terms of the TPR (Table 2), CCA again performs worse than all imputation
methods. Both MI and mMI perform well in terms of TPR when compared with the
other imputation methods. The mMF, mPMM and cvPMM method again show similar
performance. The observed differences between MF and mMF do not appear to
follow a consistent pattern.

In terms of the FPR (Table 3), a pattern can be observed where mean imputation
(MI and mMI) performs really well when the view to be imputed corresponds to
noise, but performs poorly when the view to be imputed corresponds to signal.
The performance of MF shows the opposite behavior: it performs better when the
view to be imputed corresponds to signal than when it corresponds to noise.
Similar behavior can be observed for its meta-level implementation mMF.
Comparing MF and mMF shows that MF has a slightly lower FPR when the size of
the missing view is 5, but has a considerably higher FPR than mMF when the
size of the missing view is 5000. The observed differences in FPR between mMF
and mPMM and cvPMM are small under 50% missingness. Under 90% missingness
larger differences are observed when the signal view with 5 features is
missing, and when the noise view with 5000 features is missing. In terms of
FPR, CCA performs well when compared to the imputation methods.

The relative performance of the imputation methods in terms of FDR (Table 4)
is similar to that in terms of FPR. CCA also performs fairly well in terms of
FDR, just like it does in terms of FPR.

missingness | size of missing view | missing view is | CDA | CCA | MI | MF | mMI | mMF | mPMM | cvPMM
---|---|---|---|---|---|---|---|---|---|---
50% | 5 | noise | .903 | .850 | .900 | .903 | .908 | .898 | .898 | .905
|  | signal | .960 | .918 | .943 | .960 | .925 | .953 | .953 | .955
| 5000 | noise | .958 | .908 | .955 | .910 | .958 | .950 | .948 | .943
|  | signal | .893 | .845 | .848 | .868 | .848 | .875 | .875 | .875
90% | 5 | noise | .885 | .625 | .895 | .865 | .880 | .858 | .855 | .873
|  | signal | .970 | .703 | .868 | .873 | .875 | .895 | .908 | .918
| 5000 | noise | .965 | .683 | .983 | .870 | .978 | .930 | .953 | .963
|  | signal | .908 | .608 | .785 | .830 | .788 | .843 | .830 | .845

Table 1: Mean proportion of correctly selected views. CDA = complete data
analysis; CCA = complete case analysis; MI = feature-level mean imputation; MF
= feature-level missForest; mMI = meta-level mean imputation; mMF = meta-level
missForest; mPMM = meta-level predictive mean matching, generating $\bm{Z}$
once; cvPMM = meta-level predictive mean matching, generating $\bm{Z}$ five
times.

missingness | size of missing view | missing view is | CDA | CCA | MI | MF | mMI | mMF | mPMM | cvPMM
---|---|---|---|---|---|---|---|---|---|---
50% | 5 | noise | .877 | .827 | .880 | .890 | .887 | .887 | .887 | .890
|  | signal | .963 | .927 | .970 | .950 | .963 | .943 | .943 | .943
| 5000 | noise | .957 | .900 | .960 | .960 | .960 | .960 | .960 | .953
|  | signal | .877 | .810 | .853 | .880 | .863 | .850 | .853 | .850
90% | 5 | noise | .877 | .533 | .877 | .873 | .870 | .867 | .863 | .873
|  | signal | .967 | .627 | .933 | .833 | .940 | .877 | .903 | .923
| 5000 | noise | .967 | .613 | .977 | .970 | .973 | .973 | .973 | .980
|  | signal | .900 | .493 | .807 | .847 | .810 | .810 | .797 | .807

Table 2: True positive rate (TPR). CDA = complete data analysis; CCA =
complete case analysis; MI = feature-level mean imputation; MF = feature-level
missForest; mMI = meta-level mean imputation; mMF = meta-level missForest;
mPMM = meta-level predictive mean matching, generating $\bm{Z}$ once; cvPMM =
meta-level predictive mean matching, generating $\bm{Z}$ five times.

missingness | size of missing view | missing view is | CDA | CCA | MI | MF | mMI | mMF | mPMM | cvPMM
---|---|---|---|---|---|---|---|---|---|---
50% | 5 | noise | .02 | .08 | .04 | .06 | .03 | .07 | .07 | .05
|  | signal | .05 | .11 | .14 | .01 | .19 | .02 | .02 | .01
| 5000 | noise | .04 | .07 | .06 | .24 | .05 | .08 | .09 | .09
|  | signal | .06 | .05 | .17 | .17 | .20 | .05 | .06 | .05
90% | 5 | noise | .09 | .10 | .05 | .16 | .09 | .17 | .17 | .13
|  | signal | .02 | .07 | .33 | .01 | .32 | .05 | .08 | .10
| 5000 | noise | .04 | .11 | 0 | .43 | .01 | .20 | .11 | .09
|  | signal | .07 | .05 | .28 | .22 | .28 | .06 | .07 | .04

Table 3: False positive rate (FPR). CDA = complete data analysis; CCA =
complete case analysis; MI = feature-level mean imputation; MF = feature-level
missForest; mMI = meta-level mean imputation; mMF = meta-level missForest;
mPMM = meta-level predictive mean matching, generating $\bm{Z}$ once; cvPMM =
meta-level predictive mean matching, generating $\bm{Z}$ five times.

missingness | size of missing view | missing view is | CDA | CCA | MI | MF | mMI | mMF | mPMM | cvPMM
---|---|---|---|---|---|---|---|---|---|---
50% | 5 | noise | .005 | .022 | .010 | .015 | .008 | .018 | .018 | .013
|  | signal | .013 | .029 | .035 | .003 | .047 | .005 | .006 | .003
| 5000 | noise | .011 | .018 | .015 | .063 | .013 | .020 | .023 | .023
|  | signal | .016 | .013 | .044 | .045 | .053 | .013 | .016 | .013
90% | 5 | noise | .023 | .034 | .013 | .041 | .024 | .048 | .047 | .035
|  | signal | .005 | .022 | .084 | .003 | .08 | .013 | .021 | .026
| 5000 | noise | .012 | .031 | 0 | .112 | .003 | .051 | .028 | .023
|  | signal | .018 | .032 | .073 | .060 | .073 | .016 | .020 | .011

Table 4: False discovery rate (FDR). CDA = complete data analysis; CCA =
complete case analysis; MI = feature-level mean imputation; MF = feature-level
missForest; mMI = meta-level mean imputation; mMF = meta-level missForest;
mPMM = meta-level predictive mean matching, generating $\bm{Z}$ once; cvPMM =
meta-level predictive mean matching, generating $\bm{Z}$ five times.

###  4.3 Computation time

Figure 7 shows the log computation time for all imputation methods, as well as
CDA and CCA, with 90% missingness. Figure 8 shows the log computation time
with 50% missingness. Note that the computation times reported here include
the computations of both the model fitting and the imputation model. It can be
observed that CCA has the lowest computation time across all plots, since it
constitutes a reduction in the amount of observations in the training data. By
comparing MI with mMI, and MF with mMF, it can be observed that meta-level
imputation is faster than base-level imputation. This is particularly
noticeable when the view to be imputed is large (i.e. when $V_{4}$ is
missing), when MF is the slowest method by a large margin. cvPMM is
considerably slower than mPMM, and is the slowest method when the view to be
imputed is small (i.e. when $V_{1}$ is missing).

Figure 7: Logarithm of computation time (in seconds) with 90% missingness. CDA
= complete data analysis; CCA = complete case analysis; MI = feature-level
mean imputation; MF = feature-level missForest; mMI = meta-level mean
imputation; mMF = meta-level missForest; mPMM = meta-level predictive mean
matching, generating $\bm{Z}$ once; cvPMM = meta-level predictive mean
matching, generating $\bm{Z}$ five times. $V_{1}$ is the smallest view,
consisting of 5 features. $V_{4}$ is the largest view, consisting of 5000
features.

Figure 8: Logarithm of computation time (in seconds) with 50% missingness. CDA
= complete data analysis; CCA = complete case analysis; MI = feature-level
mean imputation; MF = feature-level missForest; mMI = meta-level mean
imputation; mMF = meta-level missForest; mPMM = meta-level predictive mean
matching, generating $\bm{Z}$ once; cvPMM = meta-level predictive mean
matching, generating $\bm{Z}$ five times. $V_{1}$ is the smallest view,
consisting of 5 features. $V_{4}$ is the largest view, consisting of 5000
features.

##  5 Discussion

Our results support four general conclusions: First, complete data analysis
performs best. This is unsurprising, since, as Orchard and Woodbury [31, p.
697] noted, “the best way to treat missing data is not to have them.” Second,
perhaps also unsurprising but more interesting, is that any form of imputation
works better than complete case analysis. Third, which is likely more
surprising and interesting, is that if the goal is solely to predict the
outcome, the best form of imputation is unconditional mean imputation (MI or
mMI): it is fast (particularly mMI) and performed well in terms of both test
accuracy and MSEP across all experimental conditions. The idea that mean
imputation can be useful for prediction is not new; Josse et al. [17] have
shown that mean imputation is universally consistent for prediction as long as
the employed learning algorithm is universally consistent. However, those
conditions do not apply here, so it is interesting to see that mean imputation
can also work well for prediction in a multi-view setting with a small sample
size. If the aim is not just prediction but also view selection, then the
performance of mean imputation depends on whether the view to be imputed
corresponds to signal or to noise. If the view to be imputed corresponds to
noise, MI performs well in all measures of view selection performance, but if
the view to be imputed corresponds to signal, MI leads to the highest FPR and
FDR of all imputation methods considered in this article. This difference in
performance can be explained by considering MI as a form of regularization
applied only to the view(s) in which missing values occur, since MI reduces
the variance of any imputed features and attenuates the correlation with the
outcome. Thus, if the view corresponds to noise, this regularization is
beneficial as it reduces the probability that the view is picked up by the
meta-learner. However, if the view corresponds to signal, then MI attenuates
the signal, causing the meta-learner to include superfluous views. Obviously,
a method which automatically selects views is primarily useful in a setting
where the status of a view as signal or noise is unknown beforehand. Thus, if
the aim of a study includes view selection, unconditional mean imputation
should probably not be used. The finding that conditional imputation works
better than unconditional imputation for view selection is consistent with
earlier results on the effect of imputation on feature selection [22].

The fourth and final major conclusion is that meta-level imputation is much
faster than feature-level imputation, but can still obtain good results. In
fact, meta-level imputation allows the use of sophisticated imputation
algorithms such as PMM which are infeasible to use at the feature level.
Feature-level MF sometimes performs better than mMF or mPMM/cvPMM in terms of
MSEP, but these observed differences in MSEP do not correspond to improved
test accuracy. For view selection, the observed differences between MF and
mMF/mPMM/cvPMM are not entirely consistent. However, it is worth noting that
in some cases base level MF lead to considerably higher values of FPR and TPR
than mMF. Consider the FPR: in those cases where a higher value of FPR is
observed for mMF the differences are very small ($\leq 0.04$), but in those
cases where a higher value of FPR is observed for MF, the differences are much
larger ($\leq 0.23$). A similar pattern can be observed for the FDR. For
additional context, it may be worth pointing out that in experiments which aim
to control the FDR, a typical value at which to control the FDR is 0.05. Using
this as a reference value, only mPMM and cvPMM show observed FDR values which
are consistently below 0.05.

For the meta-level implementation of predictive mean matching, we used two
different ways of combining the imputed data sets. The first one is generating
the $\bm{Z}$ matrix once, imputing it 5 times, then averaging the imputed data
sets and training the meta-learner on the averaged data set (mPMM). The second
is generating the $\bm{Z}$ matrix 5 times, imputing each of these matrices
once, and then averaging the imputed data sets and training the meta-learner
on the averaged data set (cvPMM). The cvPMM approach is much more
computationally expensive (Figures 8 and 7), but also averages over different
allocations to the cross-validation folds, instead of just over different
imputations. However, this approach did not lead to an improvement in
classification performance, and only small differences in view selection
performance, which makes mPMM preferable for use in practice. Of course, other
methods of combining imputations are possible. For example, one could train
the meta learner separately on each imputated data set, leading to 5 separate
StaPLR models. However, this leads to additional questions in how to combine
the separate models, and given the observed differences between mPMM and
cvPMM, we would not expect this approach to cause large gains in performance.

Throughout this article we have assumed that while the number of features $m$
was larger than $n$, the number of views $V$ was always smaller. It is natural
then to wonder how the meta-level imputation methods would perform if $n<V$
(i.e. if $\bm{Z}$ were high-dimensional). Given that missForest can handle
high-dimensional data, as is evident by the performance of MF in this article,
mMF could most likely be used without issue. The same cannot be said for mPMM
or cvPMM, since standard PMM based on the linear model struggles even with
small high-dimensional data sets (Appendix A). However, alternative approaches
could be constructed. For example, an imputation method based on the lasso
[32, 33] is included in mice [23], and appeared to work quite well for a small
high-dimensional data set such as the one described in Appendix A (results not
shown), although the associated computation time still made it infeasible to
include as a base-level imputation method in this study. Nevertheless, since
generally $V\ll m$, it has potential as a meta-level imputation method in
cases where $\bm{Z}$ is high-dimensional.

In our simulations missingness occurred only in the training set, but in
practical settings it is likely that missingness occurs also in the test set.
In this case, following the recommendations of Josse et al. [17], we suggest
that the same imputation model that was used to impute the training set is
used to impute the test set.

Throughout most of this article we assumed that the data was MCAR, and gave
examples of why this is often a realistic assumption in the case of multi-view
missing data. Nevertheless, in practice there will also be cases where the
data is MAR or even MNAR. Multiple imputation methods such as MF and PMM are
known to provide unbiased parameter estimates even under MAR. Of course, this
is assuming the learning algorithm generates unbiased estimates in general,
which is not the case for StaPLR, where the exact values of the parameter
estimates are generally not of interest. How the theoretical knowledge about
multiple imputation methods at the feature-level translates to imputation at
the meta-level under MAR or MNAR, and how it affects view selection and
classification accuracy in these settings is a topic for future research.

In summary, we have proposed a meta-level imputation method for missing values
in multi-view data. We have evaluated the method with several imputation
algorithms using simulations. The results show that meta-level imputation
produces competitive results in terms of test accuracy and view selection at a
much lower computational cost, and makes the use of advanced multiple
imputation methods such as predictive mean matching possible in high-
dimensional settings where they would otherwise be computationally infeasible.

##  6 Declaration of interest

Declarations of interest: none.

##  Appendix A Infeasibility of predictive mean matching using MICE

We first show empirically how predictive mean matching using mice leads to
imputations of poor quality when the data is high-dimensional. We generate a
high-dimensional, but comparatively small single-view data set with $n=200$
independent observations of $m=400$ normally distributed features and one
binary outcome $y$. Only the first 50 values of the first feature are missing.
Compared to the main simulations presented in this paper, this is a very easy
missing data problem.

We generate an imputed data set with PMM using mice. Figure 9 shows that the
imputed values (first 50 observations; triangles) of feature one are generally
at the extreme end of the distribution of observed values, and their
distribution is non-normal and very different from that of the observed values
(last 150 observations; circles). R code to replicate this image is included
in the supplementary materials. The likely cause of this behavior is that the
linear model is not well-behaved under high-dimensionality, meaning some form
of (ridge) regularization is required [16, 23], but the amount of
regularization required to find a ‘good’ solution is not obvious. Recommended
values for the penalty parameter in the range $10^{-6}$ to $10^{-4}$ do not
produce good results, even in this relatively simple setting.

Even when good results could be obtained, two practical problems make
calculations using mice infeasible for large high-dimensional problems. The
first, and most important one, is time: Even for the simplest setting included
in our experiments, a single iteration of a single imputation of a single
variable took well over 24 hours to compute. The second, which was not
directly relevant to our study but will be for larger problems, is memory
handling: A test run of PMM with mice on a data set consisting of 200
observations of 75,000 variables (120 MB of data), which is small by medical
imaging standards, already required 42 GB of RAM.

Figure 9: Imputed (red triangles) and observed (blue circles) values of
feature 1 after imputing the missing values with PMM using mice.

##  Appendix B Binomial deviance

The binomial deviance is defined as:

$D=-2\sum_{i=1}^{n}y_{i}\log(\hat{p}_{i})+(1-y_{i})\log(1-\hat{p}_{i}),$ (2)

where $y_{i}$ is the observed class label for test set observation $i$, and
$\hat{p}_{i}$ is the predicted probability of belonging to class 1.

Figure 10: Binomial deviance with 90% missingness. CDA = complete data
analysis; CCA = complete case analysis; MI = feature-level mean imputation; MF
= feature-level missForest; mMI = meta-level mean imputation; mMF = meta-level
missForest; mPMM = meta-level predictive mean matching, generating $\bm{Z}$
once; cvPMM = meta-level predictive mean matching, generating $\bm{Z}$ five
times. $V_{1}$ is the smallest view, consisting of 5 features. $V_{4}$ is the
largest view, consisting of 5000 features.

Figure 11: Binomial deviance with 50% missingness. CDA = complete data
analysis; CCA = complete case analysis; MI = feature-level mean imputation; MF
= feature-level missForest; mMI = meta-level mean imputation; mMF = meta-level
missForest; mPMM = meta-level predictive mean matching, generating $\bm{Z}$
once; cvPMM = meta-level predictive mean matching, generating $\bm{Z}$ five
times. $V_{1}$ is the smallest view, consisting of 5 features. $V_{4}$ is the
largest view, consisting of 5000 features.

## References

  * Zhao et al. [2017] J. Zhao, X. Xie, X. Xu, and S. Sun, “Multi-view learning overview: recent progress and new challenges,” _Information Fusion_ , vol. 38, pp. 43–54, 2017.
  * Sun et al. [2019] S. Sun, L. Mao, Z. Dong, and L. Wu, _Multiview Machine Learning_. Springer, 2019.
  * Li et al. [2018] Y. Li, F.-X. Wu, and A. Ngom, “A review on machine learning principles for multi-view biological data integration,” _Briefings in Bioinformatics_ , vol. 19, no. 2, pp. 325–340, 2018.
  * Sudlow et al. [2015] C. Sudlow, J. Gallacher, N. Allen, V. Beral, P. Burton, J. Danesh, P. Downey, P. Elliott, J. Green, M. Landray _et al._ , “UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age,” _PLoS Medicine_ , vol. 12, no. 3, p. e1001779, 2015.
  * Littlejohns et al. [2020] T. J. Littlejohns, J. Holliday, L. M. Gibson, S. Garratt, N. Oesingmann, F. Alfaro-Almagro, J. D. Bell, C. Boultwood, R. Collins, M. C. Conroy _et al._ , “The UK biobank imaging enhancement of 100,000 participants: rationale, data collection, management and future directions,” _Nature Communications_ , vol. 11, no. 1, pp. 1–12, 2020.
  * Mueller et al. [2005] S. G. Mueller, M. W. Weiner, L. J. Thal, R. C. Petersen, C. Jack, W. Jagust, J. Q. Trojanowski, A. W. Toga, and L. Beckett, “The Alzheimer’s disease neuroimaging initiative,” _Neuroimaging Clinics of North America_ , vol. 15, no. 4, p. 869, 2005.
  * Schouten et al. [2016] T. M. Schouten, M. Koini, F. De Vos, S. Seiler, J. van der Grond, A. Lechner, A. Hafkemeijer, C. Möller, R. Schmidt, M. de Rooij, and S. A. Rombouts, “Combining anatomical, diffusion, and resting state functional magnetic resonance imaging for individual classification of mild and moderate Alzheimer’s disease,” _NeuroImage: Clinical_ , vol. 11, pp. 46–51, 2016\.
  * de Vos et al. [2016] F. de Vos, T. M. Schouten, A. Hafkemeijer, E. G. Dopper, J. C. van Swieten, M. de Rooij, J. van der Grond, and S. A. Rombouts, “Combining multiple anatomical MRI measures improves Alzheimer’s disease classification,” _Human Brain Mapping_ , vol. 37, pp. 1920–1929, 2016.
  * de Vos et al. [2017] F. de Vos, M. Koini, T. M. Schouten, S. Seiler, J. van der Grond, A. Lechner, R. Schmidt, M. de Rooij, and S. A. Rombouts, “A comprehensive analysis of resting state fMRI measures to classify individual patients with Alzheimer’s disease,” _NeuroImage_ , vol. 167, pp. 62–72, 2017.
  * Salvador et al. [2019] R. Salvador, E. Canales-Rodríguez, A. Guerrero-Pedraza, S. Sarró, D. Tordesillas-Gutiérrez, T. Maristany, B. Crespo-Facorro, P. McKenna, and E. Pomarol-Clotet, “Multimodal integration of brain images for MRI-based diagnosis in schizophrenia,” _Frontiers in Neuroscience_ , vol. 13, no. 1203, pp. 1–9, 2019.
  * Guggenmos et al. [2020] M. Guggenmos, K. Schmack, I. M. Veer, T. Lett, M. Sekutowicz, M. Sebold, M. Garbusow, C. Sommer, H.-U. Wittchen, U. S. Zimmermann, M. N. Smolka, H. Walter, A. Heinz, and P. Sterzer, “A multimodal neuroimaging classifier for alcohol dependence,” _Scientific Reports_ , vol. 10, no. 298, pp. 1–12, 2020.
  * Ali et al. [2021] L. Ali, Z. He, W. Cao, H. T. Rauf, Y. Imrana, and M. B. B. Heyat, “MMDD-ensemble: A multimodal data driven ensemble approach for Parkinson’s disease detection,” _Frontiers in Neuroscience_ , vol. 15, no. 754058, pp. 1–11, 2021.
  * van Loon et al. [2020a] W. van Loon, M. Fokkema, B. Szabo, and M. de Rooij, “Stacked penalized logistic regression for selecting views in multi-view learning,” _Information Fusion_ , vol. 61, pp. 113–123, 2020.
  * van Loon et al. [2020b] ——, “View selection in multi-view stacking: choosing the meta-learner,” _arXiv preprint arXiv:2010.16271_ , 2020.
  * van Loon et al. [2022] W. van Loon, F. de Vos, M. Fokkema, B. Szabo, M. Koini, R. Schmidt, and M. de Rooij, “Analyzing hierarchical multi-view MRI data with StaPLR: An application to Alzheimer’s disease classification,” _Frontiers in Neuroscience_ , vol. 16, no. 83063, 2022.
  * Van Buuren [2018] S. Van Buuren, _Flexible imputation of missing data_. CRC press, 2018.
  * Josse et al. [2019] J. Josse, N. Prost, E. Scornet, and G. Varoquaux, “On the consistency of supervised learning with missing values,” _arXiv preprint arXiv:1902.06931_ , 2019.
  * Stekhoven and Bühlmann [2012] D. J. Stekhoven and P. Bühlmann, “MissForest — non-parametric missing value imputation for mixed-type data,” _Bioinformatics_ , vol. 28, no. 1, pp. 112–118, 2012.
  * Arbuckle [1996] J. Arbuckle, “Full information estimation in the presence of incomplete data,” in _Advanced structural equation modeling: Issues and techniques (2009 reprint)_ , G. A. Marcoulides and R. E. Schumacker, Eds. New York, NY: Psychology Press, 1996, ch. 9, pp. 243–277.
  * Myrtveit et al. [2001] I. Myrtveit, E. Stensrud, and U. H. Olsson, “Analyzing data sets with missing data: An empirical evaluation of imputation methods and likelihood-based methods,” _IEEE Transactions on Software Engineering_ , vol. 27, no. 11, pp. 999–1013, 2001.
  * Twala et al. [2008] B. E. Twala, M. Jones, and D. J. Hand, “Good methods for coping with missing data in decision trees,” _Pattern Recognition Letters_ , vol. 29, no. 7, pp. 950–956, 2008.
  * Mera-Gaona et al. [2021] M. Mera-Gaona, U. Neumann, R. Vargas-Canas, and D. M. López, “Evaluating the impact of multivariate imputation by MICE in feature selection,” _PLOS ONE_ , vol. 16, no. 7, p. e0254720, 2021.
  * van Buuren and Groothuis-Oudshoorn [2011] S. van Buuren and K. Groothuis-Oudshoorn, “mice: Multivariate imputation by chained equations in R,” _Journal of Statistical Software_ , vol. 45, no. 3, pp. 1–67, 2011.
  * Rubin [1987] D. B. Rubin, _Multiple Imputation for Nonresponse in Surveys_. John Wiley & Sons, Inc., 1987.
  * Schafer [1997] J. L. Schafer, _Analysis of incomplete multivariate data_. Chapman & Hall / CRC press, 1997.
  * Team [2017] R. C. Team, _R: A Language and Environment for Statistical Computing_ , R Foundation for Statistical Computing, Vienna, Austria, 2017. [Online]. Available: https://www.R-project.org/
  * Matsumoto and Nishimura [1998] M. Matsumoto and T. Nishimura, “Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator,” _ACM Transactions on Modeling and Computer Simulation_ , vol. 8, no. 1, pp. 3–30, 1998\.
  * van Loon [2021] W. van Loon, _R package ‘multiview’ - Methods for high-dimensional multi-view learning (v0.3.1)_ , Feb. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.4630669
  * Friedman et al. [2010] J. Friedman, T. Hastie, and R. Tibshirani, “Regularization paths for generalized linear models via coordinate descent,” _Journal of Statistical Software_ , vol. 33, no. 1, pp. 1–22, 2010. [Online]. Available: http://www.jstatsoft.org/v33/i01/
  * Brier [1950] G. W. Brier, “Verification of forecasts expressed in terms of probability,” _Monthly Weather Review_ , vol. 78, no. 1, pp. 1–3, 1950.
  * Orchard and Woodbury [1972] T. Orchard and M. A. Woodbury, “A missing information principle: theory and applications,” in _Volume 1 Theory of Statistics_. University of California Press, 1972, pp. 697–716.
  * Zhao and Long [2016] Y. Zhao and Q. Long, “Multiple imputation in the presence of high-dimensional data,” _Statistical Methods in Medical Research_ , vol. 25, no. 5, pp. 2021–2035, 2016.
  * Deng et al. [2016] Y. Deng, C. Chang, M. S. Ido, and Q. Long, “Multiple imputation for general missing data patterns in the presence of high-dimensional data,” _Scientific Reports_ , vol. 6, no. 1, pp. 1–10, 2016.


# Expanding the Set of Pragmatic Considerations in Conversational AI

S.M. Seals1,2,3 and Valerie L. Shalin3,4 1Air Force Research Laboratory, 2Oak
Ridge Institute for Science and Education, 3Wright State University,
4Artificial Intelligence Institute- University of South Carolina. The views
expressed are those of the authors and do not necessarily reflect the official
policy or position of the Department of the Air Force, the Department of
Defense, or the U.S. government. Approved for public release, case number:
AFRL20225175.S.M. Seals: s.m.seals @ outlook.com

###### Abstract

Despite considerable performance improvements, current conversational AI
systems often fail to meet user expectations. We discuss several pragmatic
limitations of current conversational AI systems. We illustrate pragmatic
limitations with examples that are syntactically appropriate, but have clear
pragmatic deficiencies. We label our complaints as ”Turing Test Triggers”
(TTTs) as they indicate where current conversational AI systems fall short
compared to human behavior. We develop a taxonomy of pragmatic considerations
intended to identify what pragmatic competencies a conversational AI system
requires and discuss implications for the design and evaluation of
conversational AI systems.

##  I Introduction

Advances in deep learning and large language models have enabled the
development of high performing NLP and conversational applications [1, 2, 3,
4, 5]. This work has yielded conversational AI applications that appear to
reflect the characteristics of human dialogue and follow user instructions [6,
7, 8, 5]. Performance improvements have prompted new empirical work on
evaluation (i.e., [9, 10, 11]). In that spirit, we illustrate several current
challenges for conversational AI systems. We illustrate these limitations with
examples from conversational AI systems in the literature [12, 13] and author
interactions with currently fielded conversational AI systems 111Chatbots:
OpenAI’s chatGPT, Amtrak’s Julie, WoebotHealth’s Woebot and voice
assistants222Voice assistants: Apple’s Siri, Amazon’s Alexa. These examples
are syntactically appropriate, but have clear pragmatic deficiencies compared
to human behavior. This discrepancy triggers the Turing Test criterion-
competent human speakers and users would not produce such constructions. We
draw on traditional (i.e., travel, personal assistants) and more recent (i.e.,
LLM chatbot interfaces, mental health applications). Chatbots clearly
emphasize some of our concerns and, particularly for medical applications,
require highly refined performance.

We structure documented general user frustrations with conversational AI
systems that highlight complaints about conversational skills separately from
other usability concerns [14, 15, 16, 17, 18, 19]. In so doing, we synergize
applied and basic research endeavors that address language in use. Users,
particularly in consequential task domains, are less tolerant of limitations
than researchers.

We frame pragmatic limitations (and resulting user frustrations) of current
conversational AI systems using socially-inspired pragmatic theory of
_relevance_ [20]. We articulate two sub-themes for understanding and
addressing these limitations: preserving local meaning and incorporating
context. The resulting taxonomy informs pragmatic criteria for designing and
evaluating conversational AI systems, and integrates insights from social and
behavioral sciences with computational science.

##  II User Relevance

Conversations preserve _relevance_. When people engage in conversation, they
expect their partners will make relevant contributions that are consistent
with the accepted purpose of the conversation. Users have similar expectations
for interactions with conversational AI applications (i.e., [19]). [21]
initially proposed that the expectation of relevance is due to a _cooperative
principle_ and that the expectation of related utterances is due to a maxim of
_relation_.

[22] revised this explanation and proposed the search for relevance as a basic
feature of human cognition [20]. Input is relevant if the processing it
generates a worthwhile change in a recipient’s representation of the world
[20]. Relevance depends on context. Information must be worth the recipient’s
processing effort and be the most relevant information available consistent
with their goals [20]. Content from a conversational AI system that is
incorrect, difficult to understand, or missing important information reduces
relevance. Users must expend additional comprehension effort or search
elsewhere. Consider this example (originally proposed by [23], repeated on
chatGPT May 24 2023 Version):

> Prompt: _You need flour to bake bread. You have a sack of flour in the
> garage. When you get there, you find that the flour is at the bottom of the
> sack, but that somebody emptied a couple of boxes of dishwashing detergent
> on top of it. So you have to…_
>
> (TTT) Response: _carefully remove the boxes of dishwashing detergent from on
> top of the sack of flour, making sure not to spill or mix any of the
> detergent with the flour. You can gently lift the boxes, taking caution to
> avoid any potential spills or mess. Once the boxes are safely removed, you
> can access the flour at the bottom of the sack and proceed to use it for
> baking your bread, ensuring it remains uncontaminated by the dishwashing
> detergent._

The response acknowledges that you should not make bread with flour
contaminated by dishwashing detergent. However, the response fails to account
for the fact that the dishwashing detergent has been _emptied into_ the flour,
violating the expectation of providing all relevant implications, and
resulting in a dangerous recommendation.

The example above, and related empirical work in HCI, illustrates two
dimensions of relevance; local meaning and contextual awareness. Pragmatically
sensitive responses require integration with these dimensions. Next, we
examine each dimension with particular focus on subcategories of contextual
awareness.

##  III Meaning and Inference

To generate relevant content, conversational AI systems must respond to all
aspects of a user’s meaning. A response that addresses one part of a user’s
intent may omit other related information. This requires recovering the
complex relationships within an utterance [24, 25]. Systems that are unable to
account for these relationships compromise coherence and require additional
user effort that impairs comprehension [26, 27]. Consider this voice assistant
example:

> User: _I want to go to Cleveland, is there any construction that would slow
> down my trip?_
>
> (TTT) Assistant: _Getting directions to Cleveland [does not provide traffic
> information]._
>
> Pragmatically-appropriate assistant: _Traveling to Cleveland will take 3
> hours. There are currently no traffic delays. [Pulls up directions]_

The assistant answers the first aspect of the request, directions to a given
city. The second aspect, travel delays, is ignored. Pragmatically appropriate
responses require a representation of both the requested information and
inference. The user is concerned about travel time- construction is merely one
example.

##  IV Context

A broader category of pragmatic failures concerns the failure to address other
context [28]. The first is conversational context, which addresses
relationships between utterances and the overall conversation. The second
concerns factors external to the conversation.

Psychologists invoke the construct of memory to explain context-related
processing. Semantic memory contains general knowledge while episodic memory
addresses specific events and associated details [29, 30]. Both memory sources
influence conversational and external context.

###  IV-A Conversational Context

Generating relevant content requires systems that can incorporate the meaning
of an utterance with respect to the entire conversation. Specific topics,
words, and phrases create a conversational context that influences later
comprehension and production. People represent this context and recursively
combine and update it throughout the conversation [31, 32, 33, 34].

Results from HCI research demonstrate that users want systems that maintain
conversational context [16, 17, 19] and consider conversational abilities more
important than other features [15]. The ideal collaborative system
demonstrates awareness of the user’s status and intentions in responding [16,
15]. Pragmatic failures arise when conversational AI systems fail to
incorporate these aspects of human conceptual representations.

####  IV-A1 Awareness of Dialogue Topics

Speakers in conversation create, modify, and update expectations about what
all participants know [35, 36]. These representations include information from
personal experience, community membership, and conversational context.
Conversational AI applications either do not create these representations
(i.e., [37]) or do so in a limited fashion [5]. Failure to account for this
history generates pragmatically inappropriate responses. Consider this patient
interview system:

> (TTT) Interviewer: _Do you still go to therapy now?_
>
> Patient: _I don’t, I never went to therapy._

The word _still_ presupposes previous therapy, without having previously
established that fact. An appropriate exchange would first establish whether
the patient has attended therapy and, conditional on an affirmative response,
ask if the patient still attends therapy.

####  IV-A2 Propose Relevant Next Turns

Similarly, systems must account for the prior conversation to generate
responses. This challenge is particularly apparent for extended interactions.
The example below is from a mental health chatbot. The chatbot poses questions
intended to help the user process a thought.

> Bot: _Does this thought assume others are thinking negatively about you?_
>
> User: _No_
>
> Bot: _Does your thought place too much blame on others?_
>
> User: _No it doesn’t_
>
> Bot: _Does it place too much blame on you?_
>
> User: _Nope_
>
> (TTT) Bot: _Can you rewrite the thought without any distortions?_
>
> User: _It doesn’t have any_
>
> (TTT) Bot: _How did you do?_

The last two questions ignore the history of the conversation. An appropriate
response could introduce a new strategy or propose an alternative action.

####  IV-A3 Correction of Prior Content

Dialogue is a collaborative process that requires updating and correcting
information [38, 39, 40]. Despite their pervasiveness, conversational AI
systems struggle with these phenomena:

> User: _I want to book a ticket to Newark, sorry New York_
>
> (TTT) Bot: _There are multiple destinations, please select from the list
> below. [list includes Newark, NJ, Newark DE, and Newark NJ.]_

This response does not address the user’s self-correction. Appropriate
responses would suggest New York, as intended, or request clarification.
Previous work has examined clarification generation (i.e., [41, 42, 43]. The
challenge is identifying when clarification is required.

###  IV-B External Context

Failure to account for external information can generate pragmatic failures.
Speakers in conversation expect that their partners possess certain background
knowledge and awareness of the physical environment [44]. People exploit
context to provide appropriate detail and tailor their messages [31, 32, 45].
Users want conversational AI systems to account for external context and find
it frustrating when systems cannot [16, 17]. Where prior work has incorporated
context, the focus has been on the conversation (i.e., [5, 46]). We suggest
this approach is insufficient for truly collaborative conversational AI
systems. While certainly a challenging goal, systems that incorporate external
context will be easier to use and more consistent with users’ expectations.

####  IV-B1 Episodic Features

Speakers regularly use words and expressions that require context to be
understood (i.e., _today_). These features are easy to articulate and
interpret in conversation- the current context or non-verbal cues suggest the
intended meaning. This information is traditionally represented in different
groups [47]\- facts a third party could observe about the conversation,
information all speakers know, and facts that could verify or falsify aspects
the conversation. Our taxonomy incorporates these contextual features that are
often omitted from conversational AI systems.

Imprecise language creates an interpretation challenge for conversational AI
systems. Systems often rely only on conversational content and neglect
relevant information from the environment and external knowledge sources.
Conversational AI systems struggle with conceptual and commonsense knowledge
[48, 49], domain specific content [50], attribute information [51], and prior
information about the user. Systems that account for certain aspects require
users to explicitly state what information should be considered [5]. For
example, when interacting with chatGPT or GPT-4, users must explicitly state
their preferred response style.

A lack of external context awareness generates several problems for
conversational AI. Consider the following voice assistant example:

> User: _Is there a heat warning today?_
>
> (TTT) Assistant: _I found this on the web_ [Provides news article about heat
> wave in the UK when the user is in the US.]
>
> Pragmatically-appropriate assistant: _Yes, there is a heat warning effect in
> [area] until [time]._

As is typical for conversation, the user does not specify their location. The
voice assistant fails to incorporate this information and generates a response
inconsistent with the user’s intent. The second response is more appropriate-
it provides relevant and tailored information. Indeed, users have expectations
for information that conversational AI systems should have and want systems to
use that information [17].

####  IV-B2 Conceptual Knowledge

Interactions between a lack of external awareness and relevant semantic
knowledge can generate pragmatic failures:

> User: _Do I need gloves today?_
>
> (TTT) Assistant: _Not much sun in the forecast today._
>
> Pragmatically-appropriate assistant: _The temperature is X degrees today._

An appropriate response requires knowledge of the current situation- the
user’s location, current day, and weather forecast [52]. Moreover, the
question requires conceptual knowledge. An appropriate answer accounts for the
purpose of gloves (i.e., commonsense reasoning [48]). The response
demonstrates awareness of the current situation, but omits required conceptual
knowledge. An appropriate answer would include the forecasted low temperature.
Users want systems that can account for these intentions and respond
accordingly [16].

####  IV-B3 Default Reasoning

Incomplete information often requires the ability to draw conclusions based on
general principles or identify when new information invalidates old
conclusions [53]. Conversation regularly invokes these abilities [54], yet
these situations pose problems for conversational AI applications. Consider a
modification of a prior example:

> User: _I want to go to Cleveland, are there any traffic delays?_
>
> (TTT) Assistant: _Getting directions to Cleveland [does not provide
> information about delays]._
>
> Pragmatically-appropriate assistant: _Traveling to Cleveland will take 3
> hours. There are no current delays. [Pulls up directions]_

A pragmatically appropriate response would acknowledge all likely sources of
traffic delays. While construction is the most prototypical, an appropriate
response would account for other potential delays (i.e., a high probability
snowstorm). Similarly, an appropriate response accounts for the probability
that a situation will become relevant. Warnings about minor slowdowns several
hours ahead would be pragmatically inappropriate.

TABLE I:  Considering pragmatic requirements for conversational AI systems.
Local Propositional Content: Can the system address multiple propositions? Can
the system incorporate previous content to create responses? Does this ability
have a time frame and is the time frame appropriate? Can the system handle
common sources of semantic imprecision? Are there methods for addressing
unclear content? How are they formulated?
---
Distal Propositional Content: Does this application require representing
information across sentences or turns? What time frame should apply? Can the
system identify inconsistent details? Can prior information be corrected? What
correction strategies exist?
Access to the External Environment: Does the system represent details of the
current situation? Can this information be meaningfully integrated with
dialogue? What information about the environment would be helpful? Is other
contextual information required?
Access to External Knowledge: What knowledge, general or domain specific, does
the system require? How is it integrated? Can the system draw general
conclusions or identify inconsistent details?

####  IV-B4 Inconsistent Details

Similarly, conversations often require reasoning with inconsistent details
[53]. Inconsistent details require the identification of inconsistent
information and determination of what to disregard.

Humans resolve inconsistent details effectively [55], but they create
challenges for conversational AI systems. Systems that lack these abilities
create pragmatic errors:

> User: _Remind me on Friday August 4th at 5:00 to order groceries. [Friday is
> August 5th, not August 4th]_
>
> (TTT) Assistant: _Done [creates reminder for Thursday August 4th at 5:00]_
>
> Pragmatically-appropriate assistant: _Did you mean Thursday August 4th or
> Friday August 5th?_

This requires detecting the inconsistency between the _Friday_ and _the 4th_
and resolve what the user intended. An appropriate response requires the
ability to request clarification. Failure to detect and resolve inconsistent
information results in conversational breakdown [56]. Inconsistent information
is compounded in situations where dialogue accompanies real world activity
(such as in meetings).

Previous work has proposed methods for generating clarification requests when
conversational AI systems are unsure of a user’s intent (i.e., [57, 42, 43]).
Given that discrepancies have been adequately identified, similar methods
could be used to resolve inconsistencies created by inconsistent details.

####  IV-B5 Expert Knowledge

Domain specific applications are not immune from external context pragmatic
failures.These applications require conversational AI systems with appropriate
background knowledge that generate appropriate responses for the intended
audience [58, 35]. For example, defining new anatomy terms is appropriate for
automated tutoring systems, but unnecessary in a personal assistant for
physicians. Similarly, conversational AI systems need an awareness of domain
content when intended for domain-specific applications (i.e., [59]).

##  V Discussion

We have shown that several limitations of current conversational AI systems
are symptomatic of a more general problem: a lack of attention to pragmatics.
We propose pragmatic failures are captured by _relevance theory_ [20], and
suggest two key limitations for conversational AI systems: preserving meaning
and awareness of external context. We compile our concerns into a guide (Table
I) designed to assess pragmatic requirements for a given application and the
sufficiency of proposed strategies.

Some of the ideas here have been examined in cooperative responding (i.e.,
[60]). However, these issues are not resolved with respect to modern deep
learning based conversational AI systems. Previous work examining pragmatics
has primarily investigated specific pragmatic features independently for
specific applications (i.e., [61, 62, 63, 64, 65, 66, 67, 68, 6, 69, 70, 71].
Treating pragmatics as a decentralized process ignores the interdependent
nature of many pragmatic limitations. While resolving one of these issues may
improve performance, truly context sensitive systems require the ability to
address multiple issues. Some of the limitations we discuss are more glaringly
obvious than others (i.e., systems that fail to recover local propositional
content). However, all contribute to the design of truly cooperative and
context-sensitive conversational AI systems. We suggest that the greatest
challenge to creating pragmatically appropriate conversational AI systems is
designing centralized systems that address multiple pragmatic limitations.

Recent research is addressing some of the issues we discuss here. The success
of several recent models [5, 72, 73] has prompted increased interest in
reinforcement learning with human feedback [74, 75]. While these models have
improved performance on some pragmatic factors (i.e., following instructions)
opportunities for pragmatic improvements remain. Such systems burden the user
to specify what information should be considered. Furthermore, their
performance notably differs from humans and the lack of transparency around
these reasoning and language differences impairs their pragmatic sufficiency
(i.e., [76, 77]).

We take an integrated approach designed to taxonomize recurrent themes,
motivate a theoretical framework, and coordinate research efforts. We suggest
that a unified framework facilitates integration with applied work on human
expectations for conversational AI applications [56, 16, 17, 19]. Our
framework integrates these issues with theoretical and empirical work in
pragmatics.

###  V-A Limitations and Ethical Considerations

This type of work inherits several limitations and ethical concerns related to
the development of large models [78] and privacy concerns common to
conversational AI systems. Many external context features require information
outside the lexical content of a conversation. Some, but not all, users want
systems to use this information [17], requiring customizable sharing settings.
Moreover, we must avoid creating sub-optimal systems for users who share less
information [79]. Systems that request specific information may overcome this
limitation. Second, our position could suggest an endorsement of larger models
with high monetary and energy costs [78]. However, pre-existing knowledge
sources [50, 80, 81, 49, 82], modular designs [46, 83], and approaches that
address dialogue phenomena (i.e., [41, 42, 43]) are promising alternatives.
Larger models alone will not resolve pragmatic limitations. While chatGPT
improves on some tests posed by [23], clear limitations remain. Truly
pragmatically-appropriate systems will require coordinated approaches that
address multiple deficits.

##  VI Conclusion

Several types of pragmatic challenges recur across current, disparate
conversational AI applications. We use examples from fielded conversational AI
systems that are syntactically correct but have clear pragmatic deficiencies.
These results contribute to a better understanding of the current pragmatic
limitations of conversational AI systems. Moreover, they emphasize the
importance of connections between general knowledge and the external
environment in developing future conversational AI systems that better meet
the pragmatic expectations of users.

## References

  * [1] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models are Unsupervised Multitask Learners,” 2019.
  * [2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in _Advances in neural information processing systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 1877–1901.
  * [3] J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” _Conf. of the North Amer. Chapter of the ACL: Human Language Technologies_ , vol. 1, pp. 4171–4186, 2019.
  * [4] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer, “OPT: Open Pre-trained Transformer Language Models,” June 2022, arXiv:2205.01068 [cs].
  * [5] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” Mar. 2022, arXiv:2203.02155 [cs].
  * [6] H. Kim, B. Kim, and G. Kim, “Will I Sound Like Me? Improving Persona Consistency in Dialogues through Pragmatic Self-Consciousness,” in _Proc. of the 2020 Conf. on Empir. Methods in Nat. Lang. Process. (EMNLP)_. Online: ACL, Nov. 2020, pp. 904–916.
  * [7] N. Majumder, P. Hong, S. Peng, J. Lu, D. Ghosal, A. Gelbukh, R. Mihalcea, and S. Poria, “MIME: MIMicking Emotions for Empathetic Response Generation,” in _Proc. of the Conf. on Empir. Methods in Nat. Lang. Process. (EMNLP)_. Online: ACL, Nov. 2020, pp. 8968–8979.
  * [8] W. Wu, Z. Guo, X. Zhou, H. Wu, X. Zhang, R. Lian, and H. Wang, “Proactive Human-Machine Conversation with Explicit Conversation Goal,” in _Proc. of the 57th Annual Meeting of the Assoc. for Computational Linguistics_. ACL, July 2019, pp. 3794–3804.
  * [9] M. Bhandari, P. Gour, A. Ashfaq, P. Liu, and G. Neubig, “Re-evaluating Evaluation in Text Summarization,” in _Proc. of the 2020 Conf. on Empir. Methods in Nat. Lang. Process. (EMNLP)_. Online: ACL, Oct. 2020, pp. 9347–9359.
  * [10] C. Meister and R. Cotterell, “Language Model Evaluation Beyond Perplexity,” Aug. 2021, arXiv:2106.00085 [cs].
  * [11] Y. Wang, J. Deng, A. Sun, and X. Meng, “Perplexity from PLM Is Unreliable for Evaluating Text Quality,” 10 2022, arXiv:2210.05892 [cs].
  * [12] J. Gratch, R. Artstein, G. Lucas, G. Stratou, S. Scherer, A. Nazarian, R. Wood, J. Boberg, D. DeVault, S. Marsella, D. Traum, S. Rizzo, and L.-P. Morency, “The Distress Analysis Interview Corpus of human and computer interviews,” in _Proceedings of LREC 2014 May_ , 2014, pp. 3123–3128.
  * [13] M. Gaur, V. Aribandi, U. Kursuncu, A. Alambo, V. L. Shalin, K. Thirunarayan, J. Beich, M. Narasimhan, and A. Sheth, “Knowledge-infused abstractive summarization of clinical diagnostic interviews: Framework development study,” _JMIR Mental Health_ , vol. 8, no. 5, pp. 1–19, 2021.
  * [14] P. B. Brandtzaeg and A. Følstad, “Why People Use Chatbots,” in _Internet Science_ , ser. Lecture Notes in Computer Science, I. Kompatsiaris, J. Cave, A. Satsiou, G. Carle, A. Passani, E. Kontopoulos, S. Diplaris, and D. McMillan, Eds. Cham: Springer International Publishing, 2017, pp. 377–392.
  * [15] A. Følstad and M. Skjuve, “Chatbots for customer service: user experience and motivation,” in _Proc. of the 1st Intn’l Conf. on Conversational User Interfaces_ , ser. CUI ’19. ACM, Aug. 2019, pp. 1–9.
  * [16] Q. V. Liao, M. Davis, W. Geyer, M. Muller, and N. S. Shami, “What Can You Do?: Studying Social-Agent Orientation and Agent Proactive Interactions with an Agent for Employees,” in _Proc. of the 2016 ACM Conf. on Designing Interactive Systems_. ACM, June 2016, pp. 264–275.
  * [17] E. Luger and A. Sellen, “”Like Having a Really Bad PA”: The Gulf between User Expectation and Experience of Conversational Agents,” in _Proc. SIGCHI Conf. Hum. Factor. Comput. Syst._ ACM, May 2016, pp. 5286–5297.
  * [18] M. Porcheron, J. E. Fischer, S. Reeves, and S. Sharples, “Voice Interfaces in Everyday Life,” in _Proc. SIGCHI Conf. Hum. Factor. Comput. Syst._ ACM, Apr. 2018, pp. 1–12.
  * [19] J. Zamora, “I’m Sorry, Dave, I’m Afraid I Can’t Do That: Chatbot Perception and Expectations,” in _Proc. of the 5th Intn’l Conf. on Human Agent Interaction_ , ser. HAI ’17. ACM, Oct. 2017, pp. 253–260.
  * [20] D. Wilson and D. Sperber, “Relevance theory,” _Routledge Companion to Philosophy of Language_ , pp. 163–176, 2013.
  * [21] H. Grice, “Logic and conversation,” in _Syntax and semantics 3: Speech acts_ , P. Cole and J. L. Morgan, Eds. New York: Academic Press, 1975, pp. 41–58.
  * [22] D. Sperber and D. Wilson, _Relevance: Communication and cognition_. Cambridge, Massachusetts: Harvard University Press, 1986, vol. 142.
  * [23] E. Davis and G. Marcus, “Experiments in Commonsense Reasoning in GPT-3: Status Report from June 2022,” 2022. [Online]. Available: https://cs.nyu.edu/~davise/papers/GPT-3-6-22.html
  * [24] K. I. Forster, “Visual perception of rapidly presented word sequences of varying complexity,” _Perception & Psychophysics_, vol. 8, no. 4, pp. 215–221, July 1970.
  * [25] A. C. Graesser, N. L. Hoffman, and L. F. Clark, “Structural components of reading time,” _Journal of Verbal Learning and Verbal Behavior_ , vol. 19, no. 2, pp. 135–151, Apr. 1980.
  * [26] I. L. Beck, M. G. McKeown, G. M. Sinatra, and J. A. Loxterman, “Revising Social Studies Text from a Text-Processing Perspective: Evidence of Improved Comprehensibility,” _Reading Research Quarterly_ , vol. 26, no. 3, pp. 251–276, 1991.
  * [27] D. S. McNamara, E. Kintsch, N. B. Songer, and W. Kintsch, “Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text,” _Cognition and instruction_ , vol. 14, no. 1, pp. 1–43, 1996, publisher: Taylor & Francis.
  * [28] L. Talmy, “A cognitive framework for narrative structure,” in _Towards a Cognitive Semantics_. Cambridge, MA: The MIT Press, 2000, vol. 1, pp. 417–482.
  * [29] L. Nyberg, A. R. McIntosh, R. Cabeza, R. Habib, S. Houle, and E. Tulving, “General and specific brain regions involved in encoding and retrieval of events: what, where, and when.” _Proc. Natl. Acad. Sci. U.S.A._ , vol. 93, no. 20, pp. 11 280–11 285, Oct. 1996.
  * [30] E. Tulving, “Episodic and semantic memory,” in _Organization of memory_. Oxford, England: Academic Press, 1972.
  * [31] H. H. Clark, R. Schreuder, and S. Buttrick, “Common ground at the understanding of demonstrative reference,” _Journal of Verbal Learning and Verbal Behavior_ , vol. 22, no. 2, pp. 245–258, 1983.
  * [32] E. A. Isaacs and H. H. Clark, “References in Conversation Between Experts and Novices,” _J Exp. Pychol.: General_ , vol. 116, no. 1, pp. 26–37, 1987.
  * [33] C. B. Lockridge and S. E. Brennan, “Addressees’ needs influence speakers’ early syntactic choices,” _Psychon. Bull. Rev._ , vol. 9, no. 3, pp. 550–557, 2002.
  * [34] T. A. van Dijk and W. Kintsch, _Strategies of Discourse Comprehension_. New York: Academic Press, 1983.
  * [35] H. H. Clark, _Using language_. Cambridge, MA: Cambridge University Press, 1996.
  * [36] M. J. Pickering and S. Garrod, “Toward a mechanistic psychology of dialogue,” _Behavioral and Brain Sciences_ , vol. 27, pp. 169–226, Apr. 2004.
  * [37] R. Carpenter/Existor, “Cleverbot,” 2022. [Online]. Available: https://www.cleverbot.com/
  * [38] H. H. Clark and E. F. Schaefer, “Collaborating on contributions to conversations,” _Lang. Cogn. Process._ , vol. 2, no. 1, pp. 19–41, 1987\.
  * [39] S. E. Brennan and H. H. Clark, “Conceptual pacts and lexical choice in conversation,” _J Exp. Pychol. Learn. Mem. Cogn._ , vol. 22, no. 6, pp. 1482–1493, 1996.
  * [40] E. A. Schegloff, G. Jefferson, and H. Sacks, “The preference for self-correction in the organization of repair in conversation,” _Language_ , vol. 53, no. 2, pp. 361–382, 1977.
  * [41] L. Benotti and P. Blackburn, “Grounding as a Collaborative Process,” in _Proc. of the 16th Conf. of the European Chapter of the Assoc. for Computational Linguistics: Main Volume_. Online: ACL, Apr. 2021, pp. 515–531.
  * [42] J.-K. Kim, G. Wang, S. Lee, and Y.-B. Kim, “Deciding whether to ask clarifying questions in large-scale spoken language understanding,” in _2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, Dec. 2021, pp. 869–876.
  * [43] B. P. Majumder, S. Rao, M. Galley, and J. McAuley, “Ask what’s missing and what’s useful: Improving Clarification Question Generation using Global Knowledge,” Apr. 2021.
  * [44] H. H. Clark and C. R. Marshall, “Definite reference and mutual knowledge,” in _Elements of discourse understanding_ , A. Joshi, B. Webber, and I. Sag, Eds. Cambridge: Cambridge University Press, 1981, pp. 10–63.
  * [45] J.-B. Van der Henst and D. Sperber, “Testing the Cognitive and Communicative Principles of Relevance,” in _Experimental Pragmatics_ , ser. Palgrave Studies in Pragmatics, Language and Cognition, I. A. Noveck and D. Sperber, Eds. London: Palgrave Macmillan UK, 2004, pp. 141–171.
  * [46] K. Shuster, J. Xu, M. Komeili, D. Ju, E. M. Smith, S. Roller, M. Ung, M. Chen, K. Arora, J. Lane, M. Behrooz, W. Ngan, S. Poff, N. Goyal, A. Szlam, Y.-L. Boureau, M. Kambadur, and J. Weston, “BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage,” Aug. 2022, arXiv:2208.03188 [cs]. [Online]. Available: http://arxiv.org/abs/2208.03188
  * [47] J. Barwise and J. Perry, _Situations and Attitudes_. Cambridge, MA: MIT Press, 1983.
  * [48] E. Davis and G. Marcus, “Commonsense reasoning and commonsense knowledge in artificial intelligence,” _Comm. of the ACM_ , vol. 58, no. 9, pp. 92–103, Aug. 2015.
  * [49] R. Speer, J. Chin, and C. Havasi, “ConceptNet 5.5: An Open Multilingual Graph of General Knowledge,” in _31st AAAI conference on artificial intelligence_ , 2017, pp. 4444–4451.
  * [50] M. Gaur, U. Kursuncu, A. Alambo, A. Sheth, R. Daniulaityte, K. Thirunarayan, and J. Pathak, “” Let Me Tell You About Your Mental Health!” Contextualized Classification of Reddit Posts to DSM-5 for Web-based Intervention,” in _Proc. of the 27th ACM Int’l Conf. on Information and Knowledge Management_ , 2018, pp. 753–762.
  * [51] F. Zhang, N. J. Yuan, D. Lian, X. Xie, and W.-Y. Ma, “Collaborative Knowledge Base Embedding for Recommender Systems,” in _SIGKDD Explor._ ACM, Aug. 2016, pp. 353–362.
  * [52] S. C. Levinson, “Deixis,” in _The handbook of pragmatics_ , L. R. Horn and G. Ward, Eds. Blackwell Publishing Ltd, 2011, pp. 97–121.
  * [53] G. Brewka, “Introduction,” in _Nonmonotonic Reasoning: Logical Foundations of Commonsense_. Cambridge, MA: Cambridge University Press, 2012.
  * [54] J.-B. Van der Henst, L. Carles, and D. Sperber, “Truthfulness and relevance in telling the time,” _Mind & Language_, vol. 17, no. 5, pp. 457–466, Nov. 2002.
  * [55] P. N. Johnson-laird, V. Girotto, and P. Legrenzi, “Reasoning from inconsistency to consistency,” _Psychol. Rev._ , vol. 111, no. 3, pp. 640–661, 2004.
  * [56] Z. Ashktorab, M. Jain, Q. V. Liao, and J. D. Weisz, “Resilient Chatbots: Repair Strategy Preferences for Conversational Breakdowns,” in _Proc. SIGCHI Conf. Hum. Factor. Comput. Syst._ ACM, May 2019, pp. 1–12.
  * [57] L. Benotti and P. Blackburn, “A recipe for annotating grounded clarifications,” in _Proc. of the 2021 Conf. of the North Amer. Chapter of the Assoc. for Computational Linguistics: Human Language Technologies_. Online: ACL, June 2021, pp. 4065–4077.
  * [58] A. Bell, “Language style as audience design,” _Language in Society_ , vol. 13, no. 2, pp. 145–204, 1984.
  * [59] B. Ferreira, “A Profanity Filter Banned the Word ’Bone’ at a Paleontology Conference,” Oct. 2020. [Online]. Available: https://www.vice.com/en/article/dyzamj/a-profanity-filter-banned-the-word-bone-at-a-paleontology-conference
  * [60] B. A. Cheikes and B. L. Webber, “Elements of a computational model of cooperative response generation,” in _Proc. of the workshop on Speech and Natural Language - HLT_ , 1989, pp. 216–221.
  * [61] L. Pandia, Y. Cong, and A. Ettinger, “Pragmatic competence of pre-trained language models through the lens of discourse connectives,” in _Proc. of the 25th Conf. on Computational Natural Language Learning_. ACL, Nov. 2021, pp. 367–379.
  * [62] A. Ettinger, “What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models,” _Trans. Assoc. Comp. Linguist._ , vol. 8, pp. 34–48, 2020.
  * [63] R. Gubelmann and S. Handschuh, “Context Matters: A Pragmatic Study of PLMs’ Negation Understanding,” in _Proc. of the 60th Annual Meeting of the Assoc. for Computational Linguistics (Volume 1: Long Papers)_. ACL, May 2022, pp. 4602–4621.
  * [64] R. Wang, J. White, J. Mu, and N. Goodman, “Calibrate your listeners! Robust communication-based training for pragmatic speakers,” in _Findings of the Assoc. for Computational Linguistics: EMNLP 2021_. ACL, Nov. 2021, pp. 977–984.
  * [65] A. Nie, R. Cohn-Gordon, and C. Potts, “Pragmatic Issue-Sensitive Image Captioning,” in _Findings of the Assoc. for Computational Linguistics: EMNLP 2020_. Online: ACL, Nov. 2020, pp. 1924–1938.
  * [66] S. Schüz and S. Zarrieß, “Decoupling Pragmatics: Discriminative Decoding for Referring Expression Generation,” in _Proc. of the Reasoning and Interaction Conf. (ReInAct 2021)_. ACL, Oct. 2021, pp. 47–52.
  * [67] C. Zhang and M. Abdul-Mageed, “Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning,” in _Proc. of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis_. ACL, May 2022, pp. 141–156.
  * [68] Y. Bao, S. Ghosh, and J. Chai, “Learning to Mediate Disparities Towards Pragmatic Communication,” in _Proc. of the 60th Annual Meeting of the Assoc. for Computational Linguistics (Volume 1: Long Papers)_. ACL, May 2022, pp. 2829–2842.
  * [69] H. Kim, B. Kim, and G. Kim, “Perspective-taking and Pragmatics for Generating Empathetic Responses Focused on Emotion Causes,” in _Proc. of the 2021 Conf. on Empir. Methods in Nat. Lang. Process._ ACL, Nov. 2021, pp. 2227–2240.
  * [70] A. Nath, “Towards naturally responsive spoken dialog systems by modelling pragmatic-prosody correlations of discourse markers,” _IUI_ , pp. 128–129, 2020.
  * [71] Z. Wu and D. C. Ong, “Pragmatically Informative Color Generation by Grounding Contextual Modifiers,” in _Proc. of the Society for Computation in Linguistics 2021_ , vol. 4, 54, Feb. 2021, pp. 438–445.
  * [72] OpenAI. GPT-4 Technical Report.
  * [73] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, _et al._ , “Llama 2: Open foundation and fine-tuned chat models,” 2023.
  * [74] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep Reinforcement Learning from Human Preferences,” in _NeurIPS_ , vol. 30, 2017.
  * [75] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-Tuning Language Models from Human Preferences.
  * [76] I. Dasgupta, A. K. Lampinen, S. C. Y. Chan, A. Creswell, D. Kumaran, J. L. McClelland, and F. Hill, “Language models show human-like content effects on reasoning,” July 2022, arXiv:2207.07051 [cs]. [Online]. Available: http://arxiv.org/abs/2207.07051
  * [77] S. M. Seals and V. L. Shalin, “Long-form analogies generated by chatGPT lack human-like psycholinguistic properties,” in _CogSci_ , 2023.
  * [78] E. Strubell, A. Ganesh, and A. Mccallum, “Energy and Policy Considerations for Deep Learning in NLP,” in _Proc. of the 57th Annual Meeting of the Assoc. for Computational Linguistics_ , 2019, pp. 3645–3650.
  * [79] S. Zuboff, _The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power_ , 1st ed. PublicAffairs, 2020.
  * [80] G. A. Miller, “WordNet: A lexical database for English,” _Comm. of the ACM_ , vol. 38, no. 11, pp. 39–41, 1995.
  * [81] A. Sheth, B. Aleman-Meza, I. B. Arpinar, C. Bertram, Y. Warke, C. Ramakrishanan, C. Halaschek, K. Anyanwu, D. Avant, F. S. Arpinar, and others, “Semantic association identification and knowledge discovery for national security applications,” _J. of Database Manag._ , vol. 16, no. 1, pp. 33–53, 2005.
  * [82] L. G. Valiant, “Knowledge infusion,” _AAAI_ , vol. 6, 2006.
  * [83] G. Lee, V. Hartmann, J. Park, D. Papailiopoulos, and K. Lee, “Prompted LLMs as Chatbot Modules for Long Open-domain Conversation,” in _ACL 2023_ , 2023, pp. 4536–4554.