# Perturbation-based Active Learning for Question Answering

Fan Luo
University of Arizona
Tucson, AZ, USA
<EMAIL_ADDRESS>
&Mihai Surdeanu
University of Arizona
Tucson, AZ, USA
<EMAIL_ADDRESS>

###### Abstract

Building a question answering (QA) model with less annotation costs can be
achieved by utilizing active learning (AL) training strategy. It selects the
most informative unlabeled training data to update the model effectively.
Acquisition functions for AL are used to determine how informative each
training example is, such as uncertainty or diversity based sampling. In this
work, we propose a perturbation-based active learning acquisition strategy and
demonstrate it is more effective than existing commonly used strategies.

## 1\. Introduction

Contemporary state-of-the-art QA models are deep neural networks. Although
employing deep models has led to impressive performance improvement, but they
are data-hungry. Training and even fine-tuning such models requires a large
amount of high-quality annotated training data to achieve a reasonably good
performance. For question answering tasks, unlabeled data are easily obtained,
but the answer labels are time-consuming and expensive to obtain. With a
limited annotation budget in practice, selecting unlabeled training instances
that are most informative to annotate is an effective way to reduce the
annotation effort. The paradigm of choosing the most informative data to label
is referred to as active learning.

Active learning (AL) is a learning method that aims at minimizing labeling
costs of training data acquisition without sacrificing accuracy (Fu et al.,
2013). With a limited annotation budget in practice, AL selects the most
informative instances and queries their labels through the interaction with
oracles (annotated by experts or apply crowd-sourcing techniques) to learn. In
AL, the learner iteratively selects the most informative data that are helpful
in evolving the model for annotation and updating the model with respect to
the new annotations. Common acquisition functions for active learning are
based on uncertainty and diversity. (Settles, 2009; Dor et al., 2020; Schröder
et al., 2022). The uncertainty-based function select difficult examples
according to probability scores from the model’s predictions, and the
diversity-based function selects heterogeneous data points in the feature
space. The two approaches are orthogonal to each other, since uncertainty
sampling is usually based on the model’s output, while diversity exploits
information from the input feature space.(Dasgupta, 2011) In this work, we
propose PAL (Perturbation Active Learning) acquisition method, utilizing both
the input feature and model’s output predictions to select the most
informative instances. We conduct experiments to compare PAL with multiple AL
strategies by fine-tuning pre-trained language model BERT for answer
prediction and evaluate on the SQuAD dataset(Rajpurkar et al., 2016).

## 2\. Related Works

### 2.1. Answer Extraction with Reader Model

Widely adopted QA approaches use a two-stage retriever-reader pipeline (Chen
et al., 2017): a retriever first gathers relevant passages as evidence
candidates, then a reader performs reading comprehension (RC) to extract an
answer from the retrieved candidates to form an answer.

Figure 1: A typical example of DeepAL model architecture ((Ren et al., 2021))

Typical Reading Comprehension (RC) task is a supervised learning problem that
learns a predictor which takes a question and an input passage of text as
inputs and gives the answer as output. Broadly, RC systems are grouped into
three categories, namely rule-based, machine learning based and deep learning
based models. Rule-based models are based on hand-crafted rules that require
substantial human effort, and are incapable of generalization. A rule can be a
logical combination of any of the language processing tasks such as part-of-
speech tagging, semantic class tagging, and entity recognition. Traditional ML
models transform RC into a supervised learning task, relying on a set of pre-
defined features. After designing a set of features to capture the information
that helps to distinguish answer sentences from non-answer sentences, the
learning algorithm generate a classifier from the training examples for answer
sentence classification. While these methods do not need to design hand-
crafted rules, feature engineering is a critical necessity, demanding
scientists to provide task-driven and insightful feature vector
representations for underlying optimization problems. Besides, most
traditional machine learning methods are also ‘data hungry’, and could be
potentially benefit from AL. Lately, the research progress of RC has diverted
from pure machine learning based models to deep learning based models (Chen et
al., 2017; Vakulenko et al., 2019; Reddy et al., 2019; Lei et al., 2018). DL-
based RC, also called neural reading comprehension, has made significant
progress in recent years. Neural RC models automatically generate
representations to better extract contextual information and dramatically with
attention mechanism (Weston et al., 2014; Antol et al., 2015; Xiong et al.,
2016). The neural attention mechanism allows the system to focus on the most
relevant part of the context paragraph in order to answer a question. The
attention mechanisms help improve the ability of RC systems to model complex
interactions between a context paragraph and a query. A real game-changer is a
new neural network architecture, called Transformer, introduced in the paper
“Attention is All You Need" (Vaswani et al., 2017). It is a type of neural
network architecture that essentially utilizes the concept of self-attention.
The self-attention layer allows each word position in an input sequence to
attend to all positions in the sequence, which captures long-range
dependencies between words, such as syntactic, semantic, and coreference
relations. The transformer also performs multi-head attention, which allows
the model to jointly capture different attentions from different sub-spaces,
e.g., jointly attend to information that might indicate both coreference and
syntactic relations (Storks et al., 2019).

### 2.2. Learning with Limited Annotations

Effective training of deep QA models with very little labeled data can be
achieved by either semi-supervised learning (SSL) or active learning (AL).

#### 2.2.1. Active Learning

AL is a sub-field of machine learning (ML) in which a learning algorithm aims
to achieve good accuracy with fewer training samples by interactively querying
the oracles to label new data points (Zhan et al., 2021). AL aims to
concentrate the expensive labeling process on the most informative instances
from a (typically large) pool of unlabeled data in order to reach high
accuracy with low-cost data labeling. AL has been shown to be effective in
reducing the amount of labeling effort involved in training machine learning
models. A good summary of active learning works prior to the advances in DL
can be found in Settles (2009). With the impressive breakthroughs DL made in
various challenging tasks, the combination of DL and AL, referred as deep
active learning (DeepAL), attracted widespread research interest in recent
years. Both DL and AL are sub-fields of machine learning. DL is also called
representation learning. It realizes the automatic extraction of data
features. DL has strong learning capabilities due to its complex structure,
but this also means that DL requires a large number of labeled samples to
complete the corresponding training. DL is limited by the high cost of sample
labeling in some professional fields that require rich knowledge. AL focuses
on the study of datasets, and it is also known as query learning. An effective
AL algorithm can theoretically achieve exponential acceleration in labeling
efficiency. Therefore, the combination of DL and AL is expected to consider
the complementary advantages of the two methods to achieve superior results
(Ren et al., 2021). Figure 1 illustrates a typical example of DeepAL model
architecture.

#### 2.2.2. Semi-supervised learning

SSL is a type of machine learning that uses both labeled and unlabeled data to
create a model. Whereas the collection of data is often cheap, labeling data
can usually only be achieved at enormous costs because experts have to
annotate the data manually. SSL combines supervised learning and the usage of
unlabeled data (Reinders et al., 2019). It is useful when there is not enough
labeled data to create a model, but there is enough unlabeled large unlabeled
data to provide some information about the model, which is a common case in
many real-world machine learning problems. SSL methods offer a wide range of
methods for leveraging unlabeled data when learning prediction models. Most of
the SSL methods are based on combinations of the supervised loss and an
unsupervised loss. Classical SSL algorithms include the EM based algorithms,
self-training, co-training, semi-supervised SVM (S3VM), graph-based methods,
and boosting based SSL methods (Armoogum and Li, 2019). A batch of novel
models have been recently introduced for SSL based on representation learning
techniques, such as generative models, ladder networks and graph embeddings.

## 3\. Methodology

In this work, we introduce an AL acquisition strategy, Perturbation Active
Learning (PAL), considering the robustness of the model as a signal of
informativeness of active learning. For the investigation of the effectiveness
of PAL in combination with BERT (Devlin et al., 2018) for QA task, we conduct
an empirical study to compare it with several commonly used AL query
strategies with experiments by fine-tuning BERT-based model and evaluate on
the SQuAD dataset (Rajpurkar et al., 2016).

### 3.1. Fine-tuning Reader Model

A reader model performing reading comprehension (RC) to extract an answer from
the earlier retrieved potentially relevant documents. It requires the model
have a semantic understanding of the question and context, and be able to
locate the position of an answer span. Training a transformer-based model from
scratch is computationally expensive. A pre-trained transformer model that has
been already trained on massive amounts of text data (for example, BERT was
trained on both BooksCorpus and Wikipedia) allows it to understand the
language well and capture a surprising amount of common knowledge, which is
crucial for the QA task. And with small changes in their architecture and pre-
training objective, they can be fine-tuned to perform various types of
downstream NLP tasks. Pre-trained transformer models can be treated as
representations encoders by simply dropping the original output decision
layer, with the learned embedding representations of the input text encoded in
the final hidden state vectors. Fine-tuning a pre-trained model on question
answering dataset takes the concatenation of question and context sentences as
a whole input sequence, and utilize the cross-attention between them to
predict the correct answer span. Two probability scores are calculated for
each context token, as the likelihood of the token being the start and end of
the answer span. To compute the probability of each possible answer span, it
is common to assume that the probability of a span being the answer can be
estimated by the probability of tokens of being the start and end of the
answer. The highest scored span is considered an answer. The span with the
highest probability value (i.e., sum of start token and end token
probabilities) is predicted as the answer.

To build a high-performance QA model with less amount of annotated examples,
we explore different acquisition strategies used by active learning to
intelligently select question answer examples to label and learn during the
next iteration.

### 3.2. Active Learning Iteration

An active learner iteratively selects examples for annotation between rounds
of training. As shown in Algorithm 1, initially, one percent of the dataset is
randomly selected to be annotated as $\mathcal{D}_{(0)}l$ for fine-tuning the
already pre-trained BERT-BASE model, and all the rest is a large unlabeled
data pool $\mathcal{D}_{(0)}u$. Then an active learner iteratively selects
examples for annotation between rounds of training. Assume in the current AL
iteration $t$, labeled data with answer annotation for training is
$\mathcal{D}_{(t)}l$ and a pool of unlabeled questions is
$\mathcal{D}_{(t)}u$. After continuing the fine-tune of the pre-trained model
on $\mathcal{D}_{(t)}l$, the active learner applies the acquisition function
to acquire a subset $\mathcal{D}_{(t)}al$, consisting of 10% of unlabeled
questions from the rest of the unlabeled dataset$\mathcal{D}_{(t)}u$. The
newly acquired questions are then labeled, and so they are removed from
$\mathcal{D}_{u}$ and are added to the labeled dataset $\mathcal{D}_{l}$ along
with their gold answer: $\mathcal{D}_{(t+1)})l$ =
$\mathcal{D}_{(t)}l\cup\mathcal{D}_{(t)}al$, and
$\mathcal{D}_{(t+1)}u=\mathcal{D}_{(t)}u-\mathcal{D}_{(t)}al$. This procedure
continues until all unlabeled questions are exhausted.

Input: Unlabeled data pool $\mathcal{U}$, AL acquisition function
$\phi(\cdot,\cdot,\cdot)$, AL selected unlabeled samples $\mathcal{D}_{(t)}al$

while $\lvert\mathcal{D}_{(t)}u\rvert>0$ do

$\mathcal{M}_{(t)}\leftarrow$ Continue fine-tuning $M_{t-1}$ with
$\mathcal{D}_{(t)}al$

$\mathcal{D}_{(t)}al\longleftarrow\text{arg}\max_{x\in\mathcal{D}_{(t)}u}\phi(M_{t},x,10\%)$

$\mathcal{D}_{(t+1)}l\longleftarrow\mathcal{D}_{(t)}l\cup
label(\mathcal{D}_{(t)}al)$

$\mathcal{D}_{(t+1)}u\longleftarrow\mathcal{D}_{(t)}u\setminus\mathcal{D}_{(t)}al$

end while

Algorithm 1 Active Learning Approach

### 3.3. Acquisition Strategies

Instead of randomly choosing examples from unlabelled corpus, AL acquisition
strategies formulate an acquisition function that measures the usefulness of
unlabeled data based on specified criteria. According to the results of the
acquisition function, most informative unlabeled examples are then selected
for annotation.

#### 3.3.1. Common Active Learning strategies

The comparison of several common AL acquisition strategies used to select
informative unlabeled data samples during the active learning process is
illustrated in Figure 2.

Figure 2: Common Active Learning Strategies ((Balcan and Urner, 2015))

##### Least Confidence

This strategy selects instances with least confidence based on the output
predictions of the model. The confidence score is assigned by the acquisition
function to each unlabelled question as a measurement of model’s uncertainty.
For example, in a binary classification task, the instances with probability
value around 0.5 are the ones with least confidence and the model highly
uncertain about. For the QA task, the model outputs the candidate span with
the highest probability value as the predicted answer for each question. The
probability value of the predicted answer is considered as the confidence
score for the question. The least confidence strategy selects questions whose
predicted answer span has the least probability value to annotate.

##### Clustering

The clustering acquisition strategy feeds the BERT output embedding of the
fine-tuned model for all the unlabeled questions concatenated with their
context in Du into clustering algorithm such as K-Means. The k selected
questions for annotation are sampled proportional from each cluster per the
size of the cluster.

##### Maximal Diversity

It projects both unlabelled questions and labelled questions to the feature
space using the embedding of the fine-tuned model. The goal is to query the
unlabelled questions that have maximal distant from the labeled ones.
Considering the heterogeneous of the labeled questions, we cluster the labeled
questions in $\mathcal{D}_{(lab)}$, and return unlabelled questions with
maximum distance to its closest labelled question centroid.

#### 3.3.2. Our Strategy: Perturbation-based AL

We propose this acquisition strategy in considering the robustness of the
model as a signal of informativeness. We hypothesize that a robust model
should product similar probability distributions on the part of the original
context after perturbation the context with an additional distracting
sentence. From another aspect, among unlabelled questions Du, the model
produces very different predictive likelihoods after perturbation should be
good candidates for data acquisition to improve the robustness of the current
model.

##### Creating Perturbation for Unlabeled Candidates

The first step of our PAL acquisition strategy finds the distracting sentence
from the context of the most similar labeled questions using the embedding of
the fine-tuned model. Specifically, we use the [CLS] token embedding from
$Encoder_{t}$ of current fine-tuned model $M_{t}$ to represent the contexts
for all the questions in $\mathcal{D}_{(t)}l$ and $\mathcal{D}_{(t)}u$. We use
a K-Nearest-Neighbors (KNN) implementation in order to query the questions
with most similar contexts in $\mathcal{D}_{(t)}l$ for each candidate
$u\\_i\in\mathcal{D}_{(t)}u$111The benchmark dataset SQuAD contains many
questions with the same context, which we explicitly excluded when choosing
the most similar sentence as the distractor.. Our distance metric $d(.)$ is
Euclidean distance. To find the most similar context in $\mathcal{D}_{(t)}l$
for each $u_{i}$, we select the top k instead of selecting a predefined
threshold. This way, we create a neighborhood set $\mathcal{N}_{(c)}$ that
consists of k most similar contexts in $\mathcal{D}_{(t)}l$ for each $u_{i}$.
We then break the contexts in $\mathcal{N}_{(c)}$ into sentences, and get the
[CLS] token embeddings of each sentence with $Encoder_{t}$. Similarly, we
compute the Euclidean distance between each of these sentences to the context
of $u_{i}$ using embedding, and choose the most similar sentence as the
distractor sentence. A perturbed instance $u^{{}^{\prime}}_{i}$ is generated
by appending the distractor sentence to the original context of $u_{i}$.

##### Scoring Robustness to Perturbation

In the second step, we use the current trained model $\mathcal{M}_{(t)}$ to
obtain the output answer start and end token probabilities for each of the
candidate unlabeled question $u_{i}$ and its corresponding perturbed question
$u^{{}^{\prime}}_{i}$. Then we compute the Kullback-Leibler (KL)
divergences222The vanilla Kullback-Leibler divergence is not a symmetric
metric, instead, we used the symmetrised divergence: $D_{\text{KL}}(P\parallel
Q)+D_{\text{KL}}(Q\parallel P)$.
https://en.wikipedia.org/wiki/Kullback-Leibler_divergence in the model
predictive probabilities between each ($u_{i}$, $u^{{}^{\prime}}_{i}$) pair
for answer start and end tokens separately, and the negative sum of them is
referred as the robustness score $s_{u_{i}}$ for $u_{i}$.

##### Rank Unlabeled Candidates and Select Batch

We apply these steps to all candidate questions $u_{i}\in\mathcal{D}_{(t)}u$
and obtain a score $s_{u_{i}}$ for each. A lowest $s_{u_{i}}$ score indicates
that the unlabeled question $u_{i}$ whose predictive probability distributions
diverge the most from their predictions after adding the distracting sentence,
suggesting that it is more sensitive to the distracting perturbation, making
it a good candidate for data acquisition to choose to improve the robustness
of the current model. To this end, our acquisition function selects $b$
unlabeled questions that have the lowest score $s_{u_{i}}$ from current
unlabeled pool $\mathcal{D}_{(u)}$.

In short, our PAL acquisition strategy finds the distracting sentence from the
context of the most similar labelled question using the embedding of the fine-
tuned model, and then selects unlabeled questions from $D_{u}$, whose
predictive probability distributions diverge the most from their predictions
before adding the distracting sentence measured by Kullback-Leibler
divergence.

## 4\. Evaluation and Results

We conduct our evaluation using the Stanford Question Answering Dataset
(SQuAD) (Rajpurkar et al., 2016). Before we feed the texts of each question-
context pair to the model, we pre-process them with ‘Tokenizer‘ from
HuggingFace’s 333https://huggingface.co Transformers library, which tokenizes
the inputs to convert the tokens to their corresponding IDs in the vocabulary.
Then we compute the offset mappings that map tokens to character positions in
the context, and save the start and end positions of the answer span as the
labels. We call AutoModelForQuestionAnswering to initiate a model that loads
the BERT-base encoder with a question-answering header. This model predicts
the start and the end position of the answer, and calculates the cross-entropy
loss by taking the difference between the predicted and labeled start and the
end positions. At the beginning, all questions in the training dataset are
considered to be unlabeled. To start, 1% of the dataset is selected and
labeled for training the model. Following Algorithm 1, in each iteration, we
continue fine-tuning the model on the newly labelled 10% of the rest of the
unlabeled dataset selected by the active learning acquisition functions. This
process was repeated until $\mathcal{D}_{(i)}u$ is empty.

AL Strategies |  200 |  300 |  400 |  500 |  600 |  700 |  800 |  AUC
---|---|---|---|---|---|---|---|---
Confidence |  52.4 |  66.5 |  71.7 |  74.8 |  77.2 |  77.5 |  79 |  71.3
Clustering |  53.6 |  67.1 |  68.4 |  73.6 |  76.3 |  76.5 |  77.7 |  70.5
Diversity |  50.4 |  65.7 |  71.4 |  71.6 |  75.6 |  74.7 |  78.2 |  70
PAL |  57.7 |  70.1 |  72.6 |  74.1 |  76.2 |  78.5 |  79.9 |  72.7
Table 1: Fine-tuning BERT-base model with various AL acquisition strategies
for the question answering task. The F1 scores are evaluated at every n-th
training step (with batch size of 12) on the SQuAD dataset555SQuAD dataset is
a large annotated QA dataset, containing 100,000+ question-answer pairs on
500+ articles. For a context, there are one or more question-answer pairs are
associated. For the purpose to demonstrating the benefits of active learning
on a smaller annotated dataset in general, we experiment fine-tuning on subset
of the whole training dataset, consisting of about 20,000 unique question-
answer pairs, one pair per context.

We experimented with the several common AL strategies and our novel PAL
strategy for choosing the instances to label in each iteration. Experimental
results show that:

  1. 1.

In general, uncertainty-based strategy outperforms the two other common
sampling strategies as it always searches for the “valuable” samples around
the current decision boundary, but the optimal decision boundary cannot be
found unless a certain number of samples have already been labeled, which
explains its lower performance at the beginning.

  2. 2.

Clustering sampling strategy performs better when the number of labeled
samples is very small, while uncertainty-based criterion usually overtakes the
clustering strategy afterwards. This is expected and consistent with
observations from previous works (Zhan et al., 2021). The main reason is that
the clustering strategy could obtain the entire structure of a database in the
beginning stage, but it is insensitive to the data samples that are close to
the decision boundary, and requires querying a large number of instances
before reaching the optimal decision boundary.

  3. 3.

Both uncertainty-based and clustering sampling strategies can only guarantee
their optimal performance over a period of time in the entire AL processes,
and the optimal period differs.

  4. 4.

The diversity-based sampling is less effective than the other two sampling
strategies for the QA task.

  5. 5.

Our PAL method is effective and outperforms the commonly used AL acquisition
strategies on a fixed annotation budget.

## 5\. Discussion

The main challenge in the development of novel AL methods is that there is no
objectively superior way to determine how informative each data label is. As
clustering and diversity-based methods are highly representation-dependent,
there are two main approaches to acquiring data for active learning -
uncertainty sampling and representation based sampling. Uncertainty sampling
relies on the model’s predictive confidence to select difficult examples,
while representation based sampling exploits heterogeneity in the feature
space. Algorithms based on uncertainty may end up choosing uncertain yet
uninformative repetitive data, while representation-based methods may tend to
select diverse yet easy examples for the model (Roy and McCallum, 2001). The
two approaches are orthogonal to each other, since uncertainty sampling is
usually based on the model’s output, while representation exploits information
from the input (i.e. feature) space (Margatina et al., 2021). Our PAL
acquisition method utilizes both the input feature (embeddings in the feature
space) and model’s output (predictive probability distributions) to select the
most informative instances.

Instead of difficulty or heterogeneity, we consider the robustness of the
model against perturbation as a signal of informativeness for active learning,
by choosing the unlabeled questions where the model produces very different
predictive likelihoods after perturbation. Hybrid data acquisition functions
that combine uncertainty and representation-based sampling have also been
presented in several works (Yuan et al., 2020; Ru et al., 2020; Zhu et al.,
2008; Shen et al., 2004). Incorporating our PAL strategy with both uncertainty
and representation-based sampling is promising, which integrates the
advantages of multi-criteria as they are different informative dimensions.

## 6\. Conclusion

State-of-the-art textual QA systems are commonly based on a deep machine
comprehension model. Such models are not robust enough and are extremely
vulnerable to adversarial examples. To improve system performance, a large
amount of labeled training data is often required, that is, questions for
which the correct answers are known. However, a large and high-quality set of
labeled training data is not always available, and in practice the acquired
labels are not only expensive and also noisy. To improve the practical
applicability of QA systems, the goal is thus to attain the best gain from
this limitation. Active learning is a machine learning methodology in which a
model is trained using a set of most critical labeled data that is selected by
the model with certain acquisition strategies, rather than using a set of all
available data. It is a widely-used training strategy for maximizing
predictive performance subject to a fixed annotation budget. In this work, we
introduce an novel active learning strategy, PAL, for the QA task. The
strategy works by sampling unlabeled questions where the model produces very
different predictive likelihoods after perturbation with a distractor
sentence. We compare our PAL strategy to three other AL acquisition
strategies: model confidence, clustering and question diversity. We used SQuAD
as the benchmark dataset for the QA task. The experimental results confirm the
effectiveness of our PAL strategy, compared to several widely adopted AL
acquisition strategies.

## References

  * Abbasiantaeb and Momtazi (2021) Abbasiantaeb, Z. and S. Momtazi (2021).  Text-based question answering from information retrieval and deep neural network perspectives: A survey.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , 11(6), p. e1412.
  * Agley et al. (2022) Agley, J., Y. Xiao, R. Nolan, and L. Golzarri-Arroyo (2022).  Quality control questions on Amazon’s Mechanical Turk (MTurk): A randomized trial of impact on the USAUDIT, PHQ-9, and GAD-7.  _Behavior research methods_ , 54(2), pp. 885–897.
  * Allam and Haggag (2012) Allam, A. M. N. and M. H. Haggag (2012).  The question answering systems: A survey.  _International Journal of Research and Reviews in Information Sciences (IJRRIS)_ , 2(3).
  * Alromima et al. (2016) Alromima, W., I. F. Moawad, R. Elgohary, and M. Aref (2016).  Ontology-based query expansion for Arabic text retrieval.  _Int. J. Adv. Comput. Sci. Appl_ , 7(8), pp. 223–230.
  * Alvarez-Melis and Jaakkola (2017) Alvarez-Melis, D. and T. S. Jaakkola (2017).  A causal framework for explaining the predictions of black-box sequence-to-sequence models.  _arXiv preprint arXiv:1707.01943_.
  * Antol et al. (2015) Antol, S., A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh (2015).  Vqa: Visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2425–2433.
  * Armoogum and Li (2019) Armoogum, S. and X. Li (2019).  Big data analytics and deep learning in bioinformatics with hadoop.  In _Deep learning and parallel computing environment for bioengineering systems_ , pp. 17–36. Elsevier.
  * Arras et al. (2017) Arras, L., F. Horn, G. Montavon, K.-R. Müller, and W. Samek (2017).  " What is relevant in a text document?": An interpretable machine learning approach.  _PloS one_ , 12(8), p. e0181142.
  * Asghar et al. (2016) Asghar, N., P. Poupart, X. Jiang, and H. Li (2016).  Deep active learning for dialogue generation.  _arXiv preprint arXiv:1612.03929_.
  * Auer et al. (2007) Auer, S., C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives (2007).  Dbpedia: A nucleus for a web of open data.  In _The semantic web_ , pp. 722–735. Springer.
  * Azad and Deepak (2019) Azad, H. K. and A. Deepak (2019).  Query expansion techniques for information retrieval: a survey.  _Information Processing & Management_, 56(5), pp. 1698–1735.
  * Bajaj et al. (2016) Bajaj, P., D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, et al. (2016).  Ms marco: A human generated machine reading comprehension dataset.  _arXiv preprint arXiv:1611.09268_.
  * Balcan and Urner (2015) Balcan, M.-F. and R. Urner (2015).  Active learning.  _Survey in the Encyclopedia of Algorithms_ , 2.
  * Bauer et al. (2018) Bauer, L., Y. Wang, and M. Bansal (2018).  Commonsense for generative multi-hop question answering tasks.  _arXiv preprint arXiv:1809.06309_.
  * Beltagy et al. (2020) Beltagy, I., M. E. Peters, and A. Cohan (2020).  Longformer: The Long-Document Transformer.
  * Berger et al. (2000) Berger, A., R. Caruana, D. Cohn, D. Freitag, and V. Mittal (2000).  Bridging the lexical chasm: statistical approaches to answer-finding.  In _Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval_ , pp. 192–199.
  * Bian et al. (2008) Bian, J., Y. Liu, E. Agichtein, and H. Zha (2008).  Finding the right facts in the crowd: factoid question answering over social media.  In _Proceedings of the 17th international conference on World Wide Web_ , pp. 467–476.
  * Biran and Cotton (2017) Biran, O. and C. Cotton (2017).  Explanation and justification in machine learning: A survey.  In _IJCAI-17 workshop on explainable AI (XAI)_ , volume 8, pp. 8–13.
  * Bollacker et al. (2008) Bollacker, K., C. Evans, P. Paritosh, T. Sturge, and J. Taylor (2008).  Freebase: a collaboratively created graph database for structuring human knowledge.  In _Proceedings of the 2008 ACM SIGMOD international conference on Management of data_ , pp. 1247–1250.
  * Bordes et al. (2014) Bordes, A., S. Chopra, and J. Weston (2014).  Question answering with subgraph embeddings.  _arXiv preprint arXiv:1406.3676_.
  * Boreshban et al. (2021) Boreshban, Y., S. M. Mirbostani, G. Ghassem-Sani, S. A. Mirroshandel, and S. Amiriparian (2021).  Improving question answering performance using knowledge distillation and active learning.  _arXiv preprint arXiv:2109.12662_.
  * Bouadjenek et al. (2016) Bouadjenek, M. R., H. Hacid, M. Bouzeghoub, and A. Vakali (2016).  Persador: personalized social document representation for improving web search.  _Information Sciences_ , 369, pp. 614–633.
  * Bouziane et al. (2015) Bouziane, A., D. Bouchiha, N. Doumi, and M. Malki (2015).  Question answering systems: survey and trends.  _Procedia Computer Science_ , 73, pp. 366–375.
  * Brown et al. (2020) Brown, T., B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020).  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33, pp. 1877–1901.
  * Buchanan and Scofield (2018) Buchanan, E. M. and J. E. Scofield (2018).  Methods to detect low quality data and its implication for psychological research.  _Behavior research methods_ , 50(6), pp. 2586–2596.
  * Burges et al. (2005) Burges, C., T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender (2005).  Learning to rank using gradient descent.  In _Proceedings of the 22nd international conference on Machine learning_ , pp. 89–96.
  * Caballero (2021) Caballero, M. (2021).  A Brief Survey of Question Answering Systems.  _International Journal of Artificial Intelligence & Applications (IJAIA)_, 12(5).
  * Cakir et al. (2019) Cakir, F., K. He, X. Xia, B. Kulis, and S. Sclaroff (2019).  Deep metric learning to rank.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 1861–1870.
  * Callison-Burch and Dredze (2010) Callison-Burch, C. and M. Dredze (2010).  Creating speech and language data with amazon’s mechanical turk.  In _Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s Mechanical Turk_ , pp. 1–12.
  * Cao and Liu (2022) Cao, X. and Y. Liu (2022).  Coarse-grained decomposition and fine-grained interaction for multi-hop question answering.  _Journal of Intelligent Information Systems_ , 58(1), pp. 21–41.
  * Cao et al. (2019) Cao, Y., M. Fang, and D. Tao (2019).  BAG: Bi-directional attention entity graph convolutional network for multi-hop reasoning question answering.  _arXiv preprint arXiv:1904.04969_.
  * Cao et al. (2007) Cao, Z., T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li (2007).  Learning to rank: from pairwise approach to listwise approach.  In _Proceedings of the 24th international conference on Machine learning_ , pp. 129–136.
  * Chen et al. (2017) Chen, D., A. Fisch, J. Weston, and A. Bordes (2017).  Reading wikipedia to answer open-domain questions.  _arXiv preprint arXiv:1704.00051_.
  * Chen et al. (2015) Chen, R.-C., D. Spina, W. B. Croft, M. Sanderson, and F. Scholer (2015).  Harnessing semantics for answer sentence retrieval.  In _Proceedings of the Eighth Workshop on Exploiting Semantic Annotations in Information Retrieval_ , pp. 21–27.
  * Cho et al. (2014) Cho, K., B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio (2014).  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_.
  * Choi et al. (2016) Choi, E., D. Hewlett, A. Lacoste, I. Polosukhin, J. Uszkoreit, and J. Berant (2016).  Hierarchical question answering for long documents.  _arXiv preprint arXiv:1611.01839_.
  * Chowdhary (2020) Chowdhary, K. (2020).  _Fundamentals of artificial intelligence_.  Springer.
  * Chu and Ghahramani (2005) Chu, W. and Z. Ghahramani (2005).  Preference learning with Gaussian processes.  In _Proceedings of the 22nd international conference on Machine learning_ , pp. 137–144.
  * Clark and Gardner (2017) Clark, C. and M. Gardner (2017).  Simple and effective multi-paragraph reading comprehension.  _arXiv preprint arXiv:1710.10723_.
  * Clark and Etzioni (2016) Clark, P. and O. Etzioni (2016).  My computer is an honor student—but how intelligent is it? Standardized tests as a measure of AI.  _AI Magazine_ , 37(1), pp. 5–12.
  * Cossock and Zhang (2006) Cossock, D. and T. Zhang (2006).  Subset ranking using regression.  In _International conference on computational learning theory_ , pp. 605–619. Springer.
  * Crammer and Singer (2001) Crammer, K. and Y. Singer (2001).  Pranking with ranking.  _Advances in neural information processing systems_ , 14.
  * Dasgupta (2011) Dasgupta, S. (2011).  Two faces of active learning.  _Theoretical computer science_ , 412(19), pp. 1767–1781.
  * De Cao et al. (2018) De Cao, N., W. Aziz, and I. Titov (2018).  Question answering by reasoning across documents with graph convolutional networks.  _arXiv preprint arXiv:1808.09920_.
  * Deerwester et al. (1990) Deerwester, S., S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman (1990).  Indexing by latent semantic analysis.  _Journal of the American society for information science_ , 41(6), pp. 391–407.
  * Devlin et al. (2018) Devlin, J., M.-W. Chang, K. Lee, and K. Toutanova (2018).  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dhingra et al. (2018a) Dhingra, B., Q. Jin, Z. Yang, W. W. Cohen, and R. Salakhutdinov (2018a).  Neural models for reasoning over multiple mentions using coreference.  _arXiv preprint arXiv:1804.05922_.
  * Dhingra et al. (2018b) Dhingra, B., D. Pruthi, and D. Rajagopal (2018b).  Simple and effective semi-supervised question answering.  _arXiv preprint arXiv:1804.00720_.
  * Ding et al. (2019) Ding, M., C. Zhou, Q. Chen, H. Yang, and J. Tang (2019).  Cognitive graph for multi-hop reading comprehension at scale.  _arXiv preprint arXiv:1905.05460_.
  * Dor et al. (2020) Dor, L. E., A. Halfon, A. Gera, E. Shnarch, L. Dankin, L. Choshen, M. Danilevsky, R. Aharonov, Y. Katz, and N. Slonim (2020).  Active learning for BERT: an empirical study.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 7949–7962.
  * Esposito et al. (2020) Esposito, M., E. Damiano, A. Minutolo, G. De Pietro, and H. Fujita (2020).  Hybrid query expansion using lexical resources and word embeddings for sentence retrieval in question answering.  _Information Sciences_ , 514, pp. 88–105.
  * Etzioni (2011) Etzioni, O. (2011).  Search needs a shake-up.  _Nature_ , 476(7358), pp. 25–26.
  * Fang et al. (2019) Fang, Y., S. Sun, Z. Gan, R. Pillai, S. Wang, and J. Liu (2019).  Hierarchical graph network for multi-hop question answering.  _arXiv preprint arXiv:1911.03631_.
  * Ferrucci et al. (2010) Ferrucci, D., E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, et al. (2010).  Building Watson: An overview of the DeepQA project.  _AI magazine_ , 31(3), pp. 59–79.
  * Fu et al. (2021) Fu, R., H. Wang, X. Zhang, J. Zhou, and Y. Yan (2021).  Decomposing complex questions makes multi-hop QA easier and more interpretable.  _arXiv preprint arXiv:2110.13472_.
  * Fu et al. (2013) Fu, Y., X. Zhu, and B. Li (2013).  A survey on instance selection for active learning.  _Knowledge and information systems_ , 35(2), pp. 249–283.
  * Gao et al. (2021) Gao, L., Z. Dai, T. Chen, Z. Fan, B. V. Durme, and J. Callan (2021).  Complement lexical retrieval model with semantic residual embeddings.  In _European Conference on Information Retrieval_ , pp. 146–160. Springer.
  * Gillick et al. (2019) Gillick, D., S. Kulkarni, L. Lansing, A. Presta, J. Baldridge, E. Ie, and D. Garcia-Olano (2019).  Learning dense representations for entity retrieval.  _arXiv preprint arXiv:1909.10506_.
  * Gilpin et al. (2018) Gilpin, L. H., D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal (2018).  Explaining explanations: An overview of interpretability of machine learning.  In _2018 IEEE 5th International Conference on data science and advanced analytics (DSAA)_ , pp. 80–89. IEEE.
  * Goldberg (2019) Goldberg, Y. (2019).  Assessing BERT’s syntactic abilities.  _arXiv preprint arXiv:1901.05287_.
  * Gordon et al. (2010) Gordon, J., B. Van Durme, and L. Schubert (2010).  Evaluation of commonsense knowledge with Mechanical Turk.
  * Guo et al. (2018) Guo, L., X. Su, L. Zhang, G. Huang, X. Gao, and Z. Ding (2018).  Query expansion based on semantic related network.  In _Pacific Rim International Conference on Artificial Intelligence_ , pp. 19–28. Springer.
  * Guu et al. (2020a) Guu, K., K. Lee, Z. Tung, P. Pasupat, and M. Chang (2020a).  Retrieval augmented language model pre-training.  In _International Conference on Machine Learning_ , pp. 3929–3938. PMLR.
  * Guu et al. (2020b) Guu, K., K. Lee, Z. Tung, P. Pasupat, and M.-w. Chang (2020b).  REALM: Retrieval-Augmented Language Model Pre.  _Training_.
  * Hartmanis (1982) Hartmanis, J. (1982).  Computers and intractability: a guide to the theory of np-completeness (michael r. garey and david s. johnson).  _Siam Review_ , 24(1), p. 90.
  * He et al. (2015) He, K., X. Zhang, S. Ren, and J. Sun (2015).  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 1026–1034.
  * He et al. (2017) He, W., K. Liu, J. Liu, Y. Lyu, S. Zhao, X. Xiao, Y. Liu, Y. Wang, H. Wu, Q. She, et al. (2017).  Dureader: a chinese machine reading comprehension dataset from real-world applications.  _arXiv preprint arXiv:1711.05073_.
  * Heilman and Smith (2010) Heilman, M. and N. A. Smith (2010).  Rating computer-generated questions with Mechanical Turk.  In _Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s mechanical turk_ , pp. 35–40.
  * Hendrycks et al. (2021) Hendrycks, D., C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021).  Measuring mathematical problem solving with the math dataset.  _arXiv preprint arXiv:2103.03874_.
  * Hermann et al. (2015) Hermann, K. M., T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom (2015).  Teaching machines to read and comprehend.  _Advances in neural information processing systems_ , 28.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S. and J. Schmidhuber (1997).  Long short-term memory.  _Neural computation_ , 9(8), pp. 1735–1780.
  * Honnibal et al. (2020) Honnibal, M., I. Montani, S. Van Landeghem, and A. Boyd (2020).  spaCy: Industrial-strength Natural Language Processing in Python.  doi:10.5281/zenodo.1212303.
  * Huang et al. (2013) Huang, P.-S., X. He, J. Gao, L. Deng, A. Acero, and L. Heck (2013).  Learning deep structured semantic models for web search using clickthrough data.  In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_, pp. 2333–2338.
  * Hwang et al. (2019) Hwang, W., J. Yim, S. Park, and M. Seo (2019).  A comprehensive exploration on wikisql with table-aware word contextualization.  _arXiv preprint arXiv:1902.01069_.
  * Indurthi et al. (2018) Indurthi, S., S. Yu, S. Back, H. Cuayáhuitl, et al. (2018).  Cut to the chase: A context zoom-in network for reading comprehension.  Association for Computational Linguistics.
  * Jhamtani and Clark (2020) Jhamtani, H. and P. Clark (2020).  Learning to explain: Datasets and models for identifying valid reasoning chains in multihop question-answering.  _arXiv preprint arXiv:2010.03274_.
  * Jia and Liang (2017) Jia, R. and P. Liang (2017).  Adversarial examples for evaluating reading comprehension systems.  _arXiv preprint arXiv:1707.07328_.
  * Jia et al. (2018) Jia, Z., A. Abujabal, R. Saha Roy, J. Strötgen, and G. Weikum (2018).  Tempquestions: A benchmark for temporal question answering.  In _Companion Proceedings of the The Web Conference 2018_ , pp. 1057–1062.
  * Jiang and Bansal (2019) Jiang, Y. and M. Bansal (2019).  Self-assembling modular networks for interpretable multi-hop reasoning.  _arXiv preprint arXiv:1909.05803_.
  * Joachims (2002) Joachims, T. (2002).  Optimizing search engines using clickthrough data.  In _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_ , pp. 133–142.
  * Johnson et al. (2019) Johnson, J., M. Douze, and H. Jégou (2019).  Billion-scale similarity search with gpus.  _IEEE Transactions on Big Data_ , 7(3), pp. 535–547.
  * Joshi et al. (2017) Joshi, M., E. Choi, D. S. Weld, and L. Zettlemoyer (2017).  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  _arXiv preprint arXiv:1705.03551_.
  * Jurafsky and Martin (2018) Jurafsky, D. and J. H. Martin (2018).  Speech and language processing (draft).  _preparation [cited 2020 June 1] Available from: https://web. stanford. edu/~ jurafsky/slp3_.
  * Kahaduwa et al. (2017) Kahaduwa, H., D. Pathirana, P. L. Arachchi, V. Dias, S. Ranathunga, and U. Kohomban (2017).  Question Answering system for the travel domain.  In _2017 Moratuwa Engineering Research Conference (MERCon)_ , pp. 449–454. IEEE.
  * Kaisser and Lowe (2008) Kaisser, M. and J. B. Lowe (2008).  Creating a research collection of question answer sentence pairs with amazon’s mechanical turk.  In _Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08)_.
  * Karpukhin et al. (2020) Karpukhin, V., B. Oğuz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih (2020).  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_.
  * Kasai et al. (2019) Kasai, J., K. Qian, S. Gurajada, Y. Li, and L. Popa (2019).  Low-resource deep entity resolution with transfer and active learning.  _arXiv preprint arXiv:1906.08042_.
  * Khot et al. (2020) Khot, T., P. Clark, M. Guerquin, P. Jansen, and A. Sabharwal (2020).  Qasc: A dataset for question answering via sentence composition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 8082–8090.
  * Kim (2019) Kim, Y. (2019).  Convolutional neural networks for sentence classification. arXiv 2014\.  _arXiv preprint arXiv:1408.5882_.
  * Köppel et al. (2019) Köppel, M., A. Segner, M. Wagener, L. Pensel, A. Karwath, and S. Kramer (2019).  Pairwise learning to rank by neural networks revisited: Reconstruction, theoretical analysis and practical performance.  In _Joint European conference on machine learning and knowledge discovery in databases_ , pp. 237–252. Springer.
  * Kramer et al. (2001) Kramer, S., G. Widmer, B. Pfahringer, and M. De Groeve (2001).  Prediction of ordinal classes using regression trees.  _Fundamenta Informaticae_ , 47(1-2), pp. 1–13.
  * Kratzwald and Feuerriegel (2019) Kratzwald, B. and S. Feuerriegel (2019).  Learning from on-line user feedback in neural question answering on the web.  In _The World Wide Web Conference_ , pp. 906–916.
  * Kushman et al. (2014) Kushman, N., Y. Artzi, L. Zettlemoyer, and R. Barzilay (2014).  Learning to automatically solve algebra word problems.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 271–281.
  * Kwon et al. (2018) Kwon, H., H. Trivedi, P. Jansen, M. Surdeanu, and N. Balasubramanian (2018).  Controlling information aggregation for complex question answering.  In _European Conference on Information Retrieval_ , pp. 750–757. Springer.
  * LeCun et al. (2015) LeCun, Y., Y. Bengio, and G. Hinton (2015).  Deep learning.  _nature_ , 521(7553), pp. 436–444.
  * LeCun et al. (1998) LeCun, Y., L. Bottou, Y. Bengio, and P. Haffner (1998).  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11), pp. 2278–2324.
  * Lee et al. (2018) Lee, J., S. Yun, H. Kim, M. Ko, and J. Kang (2018).  Ranking paragraphs for improving answer recall in open-domain question answering.  _arXiv preprint arXiv:1810.00494_.
  * Lee et al. (2019) Lee, K., M.-W. Chang, and K. Toutanova (2019).  Latent retrieval for weakly supervised open domain question answering.  _arXiv preprint arXiv:1906.00300_.
  * Lei et al. (2018) Lei, J., L. Yu, M. Bansal, and T. L. Berg (2018).  Tvqa: Localized, compositional video question answering.  _arXiv preprint arXiv:1809.01696_.
  * Li et al. (2020) Li, B. Z., G. Stanovsky, and L. Zettlemoyer (2020).  Active learning for coreference resolution using discrete annotation.  _arXiv preprint arXiv:2004.13671_.
  * Li et al. (2018) Li, J., B. Li, and X. Lv (2018).  Machine Reading Comprehension Based on the Combination of BIDAF Model and Word Vectors.  In _Proceedings of the 2nd International Conference on Computer Science and Application Engineering_ , pp. 1–4.
  * Li et al. (2007) Li, P., Q. Wu, and C. Burges (2007).  Mcrank: Learning to rank using multiple classification and gradient boosting.  _Advances in neural information processing systems_ , 20.
  * Liang et al. (2020) Liang, Z., Y. Zhao, and M. Surdeanu (2020).  Using the Hammer Only on Nails: A Hybrid Method for Evidence Retrieval for Question Answering.  _arXiv preprint arXiv:2009.10791_.
  * Lin et al. (2019) Lin, B. Y., X. Chen, J. Chen, and X. Ren (2019).  Kagnet: Knowledge-aware graph networks for commonsense reasoning.  _arXiv preprint arXiv:1909.02151_.
  * Lin et al. (2021) Lin, J., R. Nogueira, and A. Yates (2021).  Pretrained transformers for text ranking: Bert and beyond.  _Synthesis Lectures on Human Language Technologies_ , 14(4), pp. 1–325.
  * Lin and Parikh (2017) Lin, X. and D. Parikh (2017).  Active learning for visual question answering: An empirical study.  _arXiv preprint arXiv:1711.01732_.
  * Lin et al. (2018) Lin, Y., H. Ji, Z. Liu, and M. Sun (2018).  Denoising Distantly Supervised Open-Domain Question Answering.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Liu et al. (2015) Liu, D.-R., Y.-H. Chen, M. Shen, and P.-J. Lu (2015).  Complementary QA network analysis for QA retrieval in social question-answering websites.  _Journal of the Association for Information Science and Technology_ , 66(1), pp. 99–116.
  * Liu et al. (2018) Liu, M., W. Buntine, and G. Haffari (2018).  Learning to actively learn neural machine translation.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning_ , pp. 334–344.
  * Liu et al. (2020) Liu, M., Z. Tu, Z. Wang, and X. Xu (2020).  LTP: a new active learning strategy for BERT-CRF based named entity recognition.  _arXiv preprint arXiv:2001.02524_.
  * Liu et al. (2017) Liu, X., Y. Shen, K. Duh, and J. Gao (2017).  Stochastic answer networks for machine reading comprehension.  _arXiv preprint arXiv:1712.03556_.
  * Luo and Surdeanu (2022) Luo, F. and M. Surdeanu (2022).  A STEP towards Interpretable Multi-Hop Reasoning: Bridge Phrase Identification and Query Expansion.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pp. 4552–4560.
  * Ma et al. (2021) Ma, X., K. Sun, R. Pradeep, and J. Lin (2021).  A replication study of dense passage retriever.  _arXiv preprint arXiv:2104.05740_.
  * Margatina et al. (2021) Margatina, K., G. Vernikos, L. Barrault, and N. Aletras (2021).  Active learning by acquiring contrastive examples.  _arXiv preprint arXiv:2109.03764_.
  * Mavi et al. (2022) Mavi, V., A. Jangra, and A. Jatowt (2022).  A Survey on Multi-hop Question Answering and Generation.  _arXiv preprint arXiv:2204.09140_.
  * Mihaylov et al. (2018) Mihaylov, T., P. Clark, T. Khot, and A. Sabharwal (2018).  Can a suit of armor conduct electricity? a new dataset for open book question answering.  _arXiv preprint arXiv:1809.02789_.
  * Min et al. (2019a) Min, S., D. Chen, H. Hajishirzi, and L. Zettlemoyer (2019a).  A discrete hard EM approach for weakly supervised question answering.  _arXiv preprint arXiv:1909.04849_.
  * Min et al. (2019b) Min, S., V. Zhong, L. Zettlemoyer, and H. Hajishirzi (2019b).  Multi-hop Reading Comprehension through Question Decomposition and Rescoring.
  * Molino et al. (2016) Molino, P., L. M. Aiello, and P. Lops (2016).  Social question answering: Textual, user, and network features for best answer prediction.  _ACM Transactions on Information Systems (TOIS)_ , 35(1), pp. 1–40.
  * Nakade et al. (2018) Nakade, V., A. Musaev, and T. Atkison (2018).  Preliminary research on thesaurus-based query expansion for Twitter data extraction.  In _Proceedings of the ACMSE 2018 Conference_ , pp. 1–4.
  * Nallapati (2004) Nallapati, R. (2004).  Discriminative models for information retrieval.  In _Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval_ , pp. 64–71.
  * Ng et al. (2000) Ng, H. T., L. H. Teo, and J. L. P. Kwan (2000).  A machine learning approach to answering questions for reading comprehension tests.  In _2000 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora_ , pp. 124–132.
  * Nguyen et al. (2018) Nguyen, D. T., S. Sharma, H. Schulz, and L. E. Asri (2018).  From film to video: Multi-turn question answering with multi-modal context.  _arXiv preprint arXiv:1812.07023_.
  * Nguyen et al. (2016) Nguyen, T., M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng (2016).  MS MARCO: A human generated machine reading comprehension dataset.  In _CoCo@ NIPS_.
  * Nie et al. (2019) Nie, Y., S. Wang, and M. Bansal (2019).  Revealing the importance of semantic retrieval for machine reading at scale.  _arXiv preprint arXiv:1909.08041_.
  * Nishida et al. (2019) Nishida, K., K. Nishida, M. Nagata, A. Otsuka, I. Saito, H. Asano, and J. Tomita (2019).  Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction.
  * Pandya and Bhatt (2021) Pandya, H. A. and B. S. Bhatt (2021).  Question answering survey: Directions, challenges, datasets, evaluation matrices.  _arXiv preprint arXiv:2112.03572_.
  * Pang et al. (2019) Pang, L., Y. Lan, J. Guo, J. Xu, L. Su, and X. Cheng (2019).  Has-qa: Hierarchical answer spans model for open-domain question answering.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pp. 6875–6882.
  * Pang et al. (2020) Pang, L., J. Xu, Q. Ai, Y. Lan, X. Cheng, and J. Wen (2020).  Setrank: Learning a permutation-invariant ranking model for information retrieval.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pp. 499–508.
  * Panovich et al. (2012) Panovich, K., R. Miller, and D. Karger (2012).  Tie strength in question & answer on social network sites.  In _Proceedings of the ACM 2012 conference on computer supported cooperative work_ , pp. 1057–1066.
  * Patel et al. (2021) Patel, A., S. Bhattamishra, and N. Goyal (2021).  Are NLP Models really able to Solve Simple Math Word Problems?  _arXiv preprint arXiv:2103.07191_.
  * Pei et al. (2019) Pei, C., Y. Zhang, Y. Zhang, F. Sun, X. Lin, H. Sun, J. Wu, P. Jiang, J. Ge, W. Ou, et al. (2019).  Personalized re-ranking for recommendation.  In _Proceedings of the 13th ACM conference on recommender systems_ , pp. 3–11.
  * Perez et al. (2020) Perez, E., P. Lewis, W.-t. Yih, K. Cho, and D. Kiela (2020).  Unsupervised question decomposition for question answering.  _arXiv preprint arXiv:2002.09758_.
  * Peters et al. (2018) Peters, M. E., M. Neumann, L. Zettlemoyer, and W.-t. Yih (2018).  Dissecting contextual word embeddings: Architecture and representation.  _arXiv preprint arXiv:1808.08949_.
  * Petroni et al. (2019) Petroni, F., T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel (2019).  Language models as knowledge bases?  _arXiv preprint arXiv:1909.01066_.
  * Pfannschmidt et al. (2018) Pfannschmidt, K., P. Gupta, and E. Hüllermeier (2018).  Deep architectures for learning context-dependent ranking functions.  _arXiv preprint arXiv:1803.05796_.
  * Pîrtoacă et al. (2019) Pîrtoacă, G.-S., T. Rebedea, and S. Ruseti (2019).  Answering questions by learning to rank–Learning to rank by answering questions.  _arXiv preprint arXiv:1909.00596_.
  * Poon et al. (2010) Poon, H., J. Christensen, P. Domingos, O. Etzioni, R. Hoffmann, C. Kiddon, T. Lin, X. Ling, A. Ritter, S. Schoenmackers, et al. (2010).  Machine reading at the university of washington.  In _Proceedings of the NAACL HLT 2010 First International Workshop on Formalisms and Methodology for Learning by Reading_ , pp. 87–95.
  * Prabhu et al. (2019) Prabhu, A., C. Dognin, and M. Singh (2019).  Sampling bias in deep active classification: An empirical study.  _arXiv preprint arXiv:1909.09389_.
  * Qi et al. (2019) Qi, P., X. Lin, L. Mehr, Z. Wang, and C. D. Manning (2019).  Answering complex open-domain questions through iterative query generation.  _arXiv preprint arXiv:1910.07000_.
  * Qiu et al. (2019) Qiu, L., Y. Xiao, Y. Qu, H. Zhou, L. Li, W. Zhang, and Y. Yu (2019).  Dynamically fused graph network for multi-hop reasoning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 6140–6150.
  * Rajpurkar et al. (2016) Rajpurkar, P., J. Zhang, K. Lopyrev, and P. Liang (2016).  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_.
  * Rao et al. (2017) Rao, Y., J. Lu, and J. Zhou (2017).  Attention-aware deep reinforcement learning for video face recognition.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 3931–3940.
  * Reddy et al. (2019) Reddy, S., D. Chen, and C. D. Manning (2019).  Coqa: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7, pp. 249–266.
  * Reimers and Gurevych (2019) Reimers, N. and I. Gurevych (2019).  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_.
  * Reinders et al. (2019) Reinders, C., H. Ackermann, M. Y. Yang, and B. Rosenhahn (2019).  Learning convolutional neural networks for object detection with very little training data.  In _Multimodal scene understanding_ , pp. 65–100. Elsevier.
  * Ren et al. (2021) Ren, P., Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang (2021).  A survey of deep active learning.  _ACM computing surveys (CSUR)_ , 54(9), pp. 1–40.
  * Riezler et al. (2007) Riezler, S., A. Vasserman, I. Tsochantaridis, V. Mittal, and Y. Liu (2007).  Statistical Machine Translation for Query Expansion in Answer Retrieval.  In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_ , pp. 464–471. Association for Computational Linguistics, Prague, Czech Republic.
  * Riloff and Thelen (2000) Riloff, E. and M. Thelen (2000).  A rule-based question answering system for reading comprehension tests.  In _ANLP-NAACL 2000 Workshop: Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding Systems_.
  * Robertson and Zaragoza (2009) Robertson, S. and H. Zaragoza (2009).  _The probabilistic relevance framework: BM25 and beyond_.  Now Publishers Inc.
  * Robertson et al. (1995) Robertson, S. E., S. Walker, S. Jones, M. M. Hancock-Beaulieu, M. Gatford, et al. (1995).  Okapi at TREC-3.  _Nist Special Publication Sp_ , 109, p. 109.
  * Rolínek et al. (2020) Rolínek, M., V. Musil, A. Paulus, M. Vlastelica, C. Michaelis, and G. Martius (2020).  Optimizing rank-based metrics with blackbox differentiation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7620–7630.
  * Roy and McCallum (2001) Roy, N. and A. McCallum (2001).  Toward optimal active learning through sampling estimation of error reduction. Int. Conf. on Machine Learning.
  * Ru et al. (2020) Ru, D., J. Feng, L. Qiu, H. Zhou, M. Wang, W. Zhang, Y. Yu, and L. Li (2020).  Active sentence learning by adversarial uncertainty sampling in discrete space.  _arXiv preprint arXiv:2004.08046_.
  * Sachan et al. (2021) Sachan, D. S., M. Patwary, M. Shoeybi, N. Kant, W. Ping, W. L. Hamilton, and B. Catanzaro (2021).  End-to-end training of neural retrievers for open-domain question answering.  _arXiv preprint arXiv:2101.00408_.
  * Salant and Berant (2017) Salant, S. and J. Berant (2017).  Contextualized word representations for reading comprehension.  _arXiv preprint arXiv:1712.03609_.
  * Samek et al. (2017) Samek, W., T. Wiegand, and K.-R. Müller (2017).  Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models.  _arXiv preprint arXiv:1708.08296_.
  * Sarker (2021) Sarker, I. H. (2021).  Deep learning: a comprehensive overview on techniques, taxonomy, applications and research directions.  _SN Computer Science_ , 2(6), pp. 1–20.
  * Schröder et al. (2021) Schröder, C., A. Niekler, and M. Potthast (2021).  Uncertainty-based query strategies for active learning with transformers.  _arXiv preprint arXiv:2107.05687_.
  * Schröder et al. (2022) Schröder, C., A. Niekler, and M. Potthast (2022).  Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pp. 2194–2203.
  * Schubotz et al. (2018) Schubotz, M., P. Scharpf, K. Dudhat, Y. Nagar, F. Hamborg, and B. Gipp (2018).  Introducing mathqa: a math-aware question answering system.  _Information Discovery and Delivery_.
  * Seo et al. (2016) Seo, M., A. Kembhavi, A. Farhadi, and H. Hajishirzi (2016).  Bidirectional attention flow for machine comprehension.  _arXiv preprint arXiv:1611.01603_.
  * Settles (2009) Settles, B. (2009).  Active learning literature survey.
  * Shan et al. (2020) Shan, X., C. Liu, Y. Xia, Q. Chen, Y. Zhang, A. Luo, and Y. Luo (2020).  BISON: BM25-weighted selfattention framework for multi-fields document search.  _arXiv preprint arXiv:2007.05186_.
  * Shashua and Levin (2002) Shashua, A. and A. Levin (2002).  Ranking with large margin principle: Two approaches.  _Advances in neural information processing systems_ , 15.
  * Shekarpour et al. (2013) Shekarpour, S., K. Höffner, J. Lehmann, and S. Auer (2013).  Keyword query expansion on linked data using linguistic and semantic features.  In _2013 IEEE Seventh International Conference on Semantic Computing_ , pp. 191–197. IEEE.
  * Shen et al. (2004) Shen, D., J. Zhang, J. Su, G. Zhou, and C. L. Tan (2004).  Multi-criteria-based active learning for named entity recognition.  In _Proceedings of the 42nd annual meeting of the Association for Computational Linguistics (ACL-04)_ , pp. 589–596.
  * Shen et al. (2017) Shen, Y., H. Yun, Z. C. Lipton, Y. Kronrod, and A. Anandkumar (2017).  Deep active learning for named entity recognition.  _arXiv preprint arXiv:1707.05928_.
  * Siddhant and Lipton (2018) Siddhant, A. and Z. C. Lipton (2018).  Deep bayesian active learning for natural language processing: Results of a large-scale empirical study.  _arXiv preprint arXiv:1808.05697_.
  * Simmons et al. (1964) Simmons, R. F., S. Klein, and K. McConlogue (1964).  Indexing and dependency logic for answering English questions.  _American Documentation_ , 15(3), pp. 196–204.
  * Song et al. (2018) Song, L., Z. Wang, M. Yu, Y. Zhang, R. Florian, and D. Gildea (2018).  Exploring graph-structured passage representation for multi-hop reading comprehension with graph neural networks.  _arXiv preprint arXiv:1809.02040_.
  * Storks et al. (2019) Storks, S., Q. Gao, and J. Y. Chai (2019).  Recent advances in natural language inference: A survey of benchmarks, resources, and approaches.  _arXiv preprint arXiv:1904.01172_.
  * Storks et al. (2021) Storks, S., Q. Gao, A. Reganti, and G. Thattai (2021).  Best of Both Worlds: A Hybrid Approach for Multi-Hop Explanation with Declarative Facts.  _arXiv preprint arXiv:2201.02740_.
  * Sun et al. (2021) Sun, H., W. W. Cohen, and R. Salakhutdinov (2021).  Iterative Hierarchical Attention for Answering Complex Questions over Long Documents.  _arXiv preprint arXiv:2106.00200_.
  * Swezey et al. (2020) Swezey, R., A. Grover, B. Charron, and S. Ermon (2020).  Pirank: Learning to rank via differentiable sorting.  _arXiv preprint arXiv:2012.06731_.
  * Takahashi et al. (1980) Takahashi, H. et al. (1980).  An approximate solution for the Steiner problem in graphs.
  * Talmor and Berant (2018) Talmor, A. and J. Berant (2018).  The web as a knowledge-base for answering complex questions.  _arXiv preprint arXiv:1803.06643_.
  * Tang et al. (2022) Tang, J., S. Hu, Z. Chen, H. Xu, and Z. Tan (2022).  Incorporating Phrases in Latent Query Reformulation for Multi-Hop Question Answering.  _Mathematics_ , 10(4), p. 646.
  * Tenney et al. (2019) Tenney, I., P. Xia, B. Chen, A. Wang, A. Poliak, R. T. McCoy, N. Kim, B. Van Durme, S. R. Bowman, D. Das, et al. (2019).  What do you learn from context? probing for sentence structure in contextualized word representations.  _arXiv preprint arXiv:1905.06316_.
  * Terdalkar and Bhattacharya (2019) Terdalkar, H. and A. Bhattacharya (2019).  Framework for question-answering in Sanskrit through automated construction of knowledge graphs.  In _Proceedings of the 6th International Sanskrit Computational Linguistics Symposium_ , pp. 97–116.
  * Trotman et al. (2014) Trotman, A., A. Puurula, and B. Burgess (2014).  Improvements to BM25 and language models examined.  In _Proceedings of the 2014 Australasian Document Computing Symposium_ , pp. 58–65.
  * Tu et al. (2020) Tu, M., K. Huang, G. Wang, J. Huang, X. He, and B. Zhou (2020).  Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pp. 9073–9080.
  * Vakulenko et al. (2019) Vakulenko, S., J. D. Fernandez Garcia, A. Polleres, M. de Rijke, and M. Cochez (2019).  Message passing for complex question answering over knowledge graphs.  In _Proceedings of the 28th acm international conference on information and knowledge management_ , pp. 1431–1440.
  * Vaswani et al. (2017) Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017).  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Voorhees et al. (1999) Voorhees, E. M. et al. (1999).  The trec-8 question answering track report.  In _Trec_ , volume 99, pp. 77–82.
  * Vrandečić (2012) Vrandečić, D. (2012).  Wikidata: A new platform for collaborative data collection.  In _Proceedings of the 21st international conference on world wide web_ , pp. 1063–1064.
  * Wang et al. (2018a) Wang, A., A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman (2018a).  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_.
  * Wang et al. (2017a) Wang, H., Q. Zhang, and J. Yuan (2017a).  Semantically enhanced medical information retrieval system: a tensor factorization based approach.  _Ieee Access_ , 5, pp. 7584–7593.
  * Wang et al. (2018b) Wang, L., S. Li, W. Zhao, K. Shen, M. Sun, R. Jia, and J. Liu (2018b).  Multi-perspective context aggregation for semi-supervised cloze-style reading comprehension.  _arXiv preprint arXiv:1808.06289_.
  * Wang et al. (2018c) Wang, S., M. Yu, X. Guo, Z. Wang, T. Klinger, W. Zhang, S. Chang, G. Tesauro, B. Zhou, and J. Jiang (2018c).  R 3: Reinforced ranker-reader for open-domain question answering.  In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Wang et al. (2000) Wang, W., J. Auer, R. Parasuraman, I. Zubarev, D. Brandyberry, and M. Harper (2000).  A question answering system developed as a project in a natural language processing course.  In _ANLP-NAACL 2000 Workshop: Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding Systems_.
  * Wang et al. (2017b) Wang, W., N. Yang, F. Wei, B. Chang, and M. Zhou (2017b).  Gated self-matching networks for reading comprehension and question answering.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 189–198.
  * Wang et al. (2018d) Wang, X., N. Golbandi, M. Bendersky, D. Metzler, and M. Najork (2018d).  Position bias estimation for unbiased learning to rank in personal search.  In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_ , pp. 610–618.
  * Wang et al. (2021) Wang, Z., P. Ng, R. Nallapati, and B. Xiang (2021).  Retrieval, re-ranking and multi-task learning for knowledge-base question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pp. 347–357.
  * Weston et al. (2014) Weston, J., S. Chopra, and A. Bordes (2014).  Memory networks.  _arXiv preprint arXiv:1410.3916_.
  * Wu et al. (2014) Wu, H., W. Wu, M. Zhou, E. Chen, L. Duan, and H.-Y. Shum (2014).  Improving search relevance for short queries in community question answering.  In _Proceedings of the 7th ACM international conference on Web search and data mining_ , pp. 43–52.
  * Xia et al. (2008) Xia, F., T.-Y. Liu, J. Wang, W. Zhang, and H. Li (2008).  Listwise approach to learning to rank: theory and algorithm.  In _Proceedings of the 25th international conference on Machine learning_ , pp. 1192–1199.
  * Xiong et al. (2016) Xiong, C., S. Merity, and R. Socher (2016).  Dynamic memory networks for visual and textual question answering.  In _International conference on machine learning_ , pp. 2397–2406. PMLR.
  * Yang et al. (2019a) Yang, W., Y. Xie, A. Lin, X. Li, L. Tan, K. Xiong, M. Li, and J. Lin (2019a).  End-to-end open-domain question answering with bertserini.  _arXiv preprint arXiv:1902.01718_.
  * Yang et al. (2019b) Yang, Y., D. Cer, A. Ahmad, M. Guo, J. Law, N. Constant, G. H. Abrego, S. Yuan, C. Tar, Y.-H. Sung, et al. (2019b).  Multilingual universal sentence encoder for semantic retrieval.  _arXiv preprint arXiv:1907.04307_.
  * Yang et al. (2019c) Yang, Z., Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le (2019c).  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32.
  * Yang et al. (2017) Yang, Z., J. Hu, R. Salakhutdinov, and W. W. Cohen (2017).  Semi-supervised qa with generative domain-adaptive nets.  _arXiv preprint arXiv:1702.02206_.
  * Yang et al. (2018) Yang, Z., P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning (2018).  HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering.
  * Yates et al. (2021) Yates, A., R. Nogueira, and J. Lin (2021).  Pretrained Transformers for Text Ranking: BERT and Beyond.  In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_ , pp. 1154–1156.
  * Yih et al. (2011) Yih, W.-t., K. Toutanova, J. C. Platt, and C. Meek (2011).  Learning discriminative projections for text similarity measures.  In _Proceedings of the fifteenth conference on computational natural language learning_ , pp. 247–256.
  * Yu et al. (2018) Yu, A. W., D. Dohan, M.-T. Luong, R. Zhao, K. Chen, M. Norouzi, and Q. V. Le (2018).  Qanet: Combining local convolution with global self-attention for reading comprehension.  _arXiv preprint arXiv:1804.09541_.
  * Yuan et al. (2020) Yuan, M., H.-T. Lin, and J. Boyd-Graber (2020).  Cold-start active learning through self-supervised language modeling.  _arXiv preprint arXiv:2010.09535_.
  * Zhan et al. (2021) Zhan, X., H. Liu, Q. Li, and A. B. Chan (2021).  A Comparative Survey: Benchmarking for Pool-based Active Learning.  In _IJCAI_ , pp. 4679–4686.
  * Zhang et al. (2017) Zhang, Y., M. Lease, and B. Wallace (2017).  Active discriminative text representation learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31.
  * Zhu et al. (2008) Zhu, J., H. Wang, T. Yao, and B. K. Tsou (2008).  Active learning with sampling by uncertainty and density for word sense disambiguation and text classification.  In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_ , pp. 1137–1144.