Equal contribution.footnote

Designing dialog tutors has been challenging as it involves modeling the diverse and complex pedagogical strategies employed by human tutors. Although there have been significant recent advances in neural conversational systems using large language models (LLMs) and growth in available dialog corpora, dialog tutoring has largely remained unaffected by these advances.
In this paper, we rigorously analyze various generative language models on two dialog tutoring datasets for language learning using automatic and human evaluations to understand the new opportunities brought by these advances as well as the challenges we must overcome to build models that would be usable in real educational settings.
We find that although current approaches can model tutoring in constrained learning scenarios when the number of concepts to be taught and possible teacher strategies are small, they perform poorly in less constrained scenarios.
Our human quality evaluation shows that both models and ground-truth annotations exhibit low performance in terms of equitable tutoring, which measures learning opportunities for students and how engaging the dialog is.
To understand the behavior of our models in a real tutoring setting, we conduct a user study using expert annotators and find a significantly large number of model reasoning errors in 45% of conversations.
Finally, we connect our findings to outline future work.

§ INTRODUCTION

The goal of dialog tutoring research is to build systems that can tutor students using natural language conversation
For several decades, learning scientists have been studying the features of domain-specific dialog tutoring systems that engender learning in students <cit.> and have established strong learning gains that are even comparable to human tutoring in specific domains <cit.>.
However, these systems require extensive authoring of materials by teachers <cit.> and therefore cannot fully utilize the scalability of online learning.

Building dialog tutors is technically challenging as tutoring dialogs typically exhibit properties that are absent in other forms of dialog. Tutoring dialogs are often long, enabling students to be exposed to the concepts in a way that they can use them in future <cit.>, and grounded in the learning scenarios <cit.>. Finally, good dialog tutors are engaging and create opportunities to learn, providing students space to seek and provide explanations, and self-reflect <cit.>.

The growing success of deep neural network based language generators in other dialog settings <cit.> suggests new possibilities in dialog tutoring that could scale beyond domain-specific approaches. However, despite their promise, advances in neural generative models have seen little adoption in dialog tutoring.

In this paper, we
contribute a
comprehensive study of the applicability of neural generative models in tutoring.
We formally introduce the dialog tutoring task and analyze existing tutoring datasets (<ref>). Then,
we describe several generative and retrieval-based models for dialog tutoring (<ref>) and benchmark them on two open-access dialog tutoring datasets for language learning: CIMA <cit.> and Teacher-Student Chatroom Corpus (TSCC) <cit.> (<ref>).
We evaluate our models on various automatic metrics (<ref>) as well as two human evaluation studies: an evaluation of the quality of the generated response with respect to various measures of goodness (<ref>), as well as a more realistic user study with a learning interface (<ref>).

Overall, while we find that pretrained models improve over simpler baselines in terms of automatic metrics, our consequent human evaluation reveals several shortcomings that ought to be addressed before these models can be adopted in the real world.
We find that while neural generative models can model more constrained learning settings well, they struggle when the learning goal is more open-ended.
Specifically, these models are unable to understand and
reason about student solutions and misconceptions, and thus, are unable to use effective pedagogical strategies.

We find that the field of dialog tutoring is significantly limited by the quantity and quality of available datasets. The available datasets are both too small and not rich enough to capture the nuances of the dialog tutoring problem.
Our analysis also reveals the inadequacy of automatic evaluation metrics for capturing tutoring quality. Not only are the existing metrics unable to capture faithfulness to the learning material and the student dialog history, but they also cannot capture moves of good human tutors that allow learners the space for reflection, explanation, follow-ups, and real engagement in the process of learning.

Based on our findings, we end with an outline of potential avenues of future research (<ref>).
hope that our paper will bring attention to this underexplored natural language processing application with the potential for significant social good.

§ THE DIALOG TUTORING TASK

Dialog tutoring can be described as a multi-turn interaction between two interlocutors, where
one performs the role of a teacher seeking to teach the other interlocutor who acts in the role of a student.
We then can describe a dialog tutoring session formally as a sequence of turns $\mathcal{H} = (u_1, \dots, u_{|\mathcal{H}|})$ that are taken by either of the interlocutors.
Each turn $u_t \in \mathcal{V}^\ast$ is a finite sequence of tokens from a
vocabulary $\mathcal{V}$.

Further, each turn $u_t$ can be associated with a sequence of dialog acts $\mathbf{a}_t \in \mathcal{A}$
that indicate the action taken by the interlocutor in the corresponding turn. The dialog act is a key aspect of dialog tutoring as it can refer to the teaching strategy employed by the tutor. These may include strategies such as providing a hint or seeking a clarification (see Appendix <ref> for more details).
The set of dialog acts $\mathcal{A}$ is usually fixed according to a predefined taxonomy and may be split into two subsets $\mathcal{A} = \mathcal{A}_{\text{teacher}} \cup \mathcal{A}_{\text{student}}$, each corresponding to the teacher and student role.
Each dialog session $\mathcal{H}$ may also be accompanied with some grounding information $K$, which grounds the response in relevant information and may refer to the teaching material that needs to be taught to the student.
This information $K$ may come in various formats, including images and videos. However, we restrict ourselves to using only text-based grounding in this work such that $K \in \mathcal{K} \subseteq \mathcal{V}^\ast$ is again a sequence of tokens from the common vocabulary $\mathcal{V}$ and $\mathcal{K}$ is used to describe the set of possible groundings (e.g., a textbook with a set of chapters). In Section <ref> we derive different methods to model the role of the teacher, to which we restrict this work.

§.§ Existing tutoring datasets

Examples of tutoring conversations from both datasets. The (image) grounding is shown in the second row and dialog acts in brackets indicate the pedagogical strategy.

To our knowledge, only three conversational tutoring datasets are openly available:
CIMA <cit.> is a crowd-sourced dataset, where annotators were asked to role-play students and teachers by working through an exercise on translating a prepositional phrase from
English to Italian, given an image and a shared set of concepts.
TSCC <cit.> uses real teachers leading one-on-one language tutoring sessions in English language learning, thus creating a more open-ended scenario.
Finally, TalkMoves <cit.>. is a collection of scraped classroom transcripts of K-12 mathematics lesson videos that contain challenging, multi-party interactions.

Dataset            Train samples     #DA     Tgt. length     Src. length     #prev. turns     corpus-div.     Flesch score     F1$(\hat{\mathbf{y}}, K)$
CIMA               2,715       5     14.71     9.70     4.55     0.149     84.64     0.196

TSCC               5,845       23     16.09     11.72     68.28     0.327     73.00     -
MultiWoZ 2.1       56,781      34$^\ast$     19.86     14.49     7.86     0.069     90.90     -

Schema-Guided Dialog     164,982     10$^\ast$     14.30     10.36     11.38     0.049     95.37     -

DSTC9              19,184      -     21.61     11.65     11.70     0.050     81.85     0.47
Personachat        127,162     -     12.26     11.65     6.51     0.162     91.80     0.10

FaithDial          18,357      -     21.72     17.33     4.54     0.274     83.28     0.47

CMU_DoG           81,468      -     14.49     18.23     18.73     0.178     79.54     0.02

Dialogue dataset statistics. Target length and source length in avg. number of tokens (Bart tokenizer). # prev. turns is avg. for each teacher response. Corpus-div is ngram entropy averaged for uni to four-grams. hlines separate: tutoring, task-oriented, open-domain dialog. $\ast$We only count system dialog acts.

The scarcity of tutoring datasets stands in contrast to other dialog scenarios, where plenty of datasets have been proposed and studied.
For example, task-oriented dialog has been studied in domains like reservations <cit.> or public service information <cit.>.
On the other hand, chit-chat or open-domain dialog has been studied on movies <cit.>, Wikipedia knowledge <cit.>, agent persona <cit.>, knowledge graphs <cit.>, and open-ended settings <cit.>.

Furthermore, we note the following limitations and characteristics of tutoring datasets, also in comparison to other dialog domains:
1) Low pedagogical quality (CIMA),
2) Limited teaching strategies (all),
3) Exclusive focus on classroom settings (TalkMoves),
4) Small dataset size (all).
5) Significantly larger context sizes (TSCC)
6) Harder readability according to the Flesch score (TSCC). We provide more evidence in Table <ref> which shows a comparison of dialog tutoring datasets with widely-used task-oriented and open-domain datasets.

§.§ Related work on generative dialog models

Similarly, while the advent of large pretrained models has sparked ample research on generative models for dialog <cit.>, this has not carried over to research on tutoring systems, where existing solutions are predominantly rule-based and do not generate open-ended responses.
For example, the authors on CIMA define heuristics to select responses <cit.>.
Pretrained transformers in general have only very recently been studied in this setting, however only for dialog act classification <cit.> and to study the pedagogical ability of existing large pretrained models <cit.>.

§ DIALOG TUTORING MODELS

After introducing the dialog tutoring task, this section highlights the models we evaluate on the task. We note that our aim is an analysis of existing models.

We explore turn-level models that can generate a teacher response $\mathbf{y} \coloneqq u_{t+1}$ given a tutoring session $\mathcal{H} = (u_1, \dots, u_{|\mathcal{H}|})$.
During training, we obtain the dialog history by teacher forcing, i.e., we take the ground-truth dialog history.
Furthermore, we do not model the problem of retrieving grounding information but rather assume it as given.

Generative Model

In order to study if generative models can capture a given teaching strategy, we first derive a model that assumes the ground-truth dialog act sequence $\mathbf{a} = \{\mathbf{a}_1, \dots, \mathbf{a}_{|\mathcal{H}|}\}$ to be given as an input.
Then, given dialog history $\mathcal{H}_{< t} = \{u_1, \dots, u_t \}$, grounding information $K$ and $\mathbf{a}_{t+1} \subseteq \mathcal{A}_{\text{teacher}}$, the set of dialog acts relevant at timestep $t+1$, the teacher response $\mathbf{y}$ is generated according to a locally-normalized language generation model.
In the case that no grounding information $K$ is given, the dependency on $K$ may be dropped.

\begin{eqnarray}
\label{eq:grounded_generation}
\mathbf{y}^\star &=& \argmax_{\mathbf{y} \in \mathcal{V}^\ast} \{ p\left(\mathbf{y} \mid \mathbf{a}_{t+1}, \mathcal{H}_{< t}, K\right) \}\\\nonumber
&=& \argmax_{\mathbf{y} \in \mathcal{V}^\ast} \prod_{i=1}^{|{\bf y}|} \{ p\left(y_i \mid \mathbf{y}_{< i}, \mathbf{a}_{t+1}, \mathcal{H}_{<t}, K\right) \} %
\end{eqnarray}

We separate the turns in the dialog by special $\langle \text{teacher} \rangle$ and $\langle \text{student} \rangle$ tags and prepend the dialog act as a special token, followed by a special $\langle \text{knowledge} \rangle$ token and the grounding information $K$ as the input to the encoder.
In CIMA we encode the triples defining the grounding information in a simple natural language format, where we separate the English and Italian words for an object, color, and preposition as well as the whole phrase by the word "is", for example as "blue is blu" in Figure <ref>.
Further, we add the grammar rules separated by a special token.
We study different models to parametrize $p$ that are described in Section <ref>.

Finally, we use the version of CTRL <cit.> presented by <cit.>.
The aim of the model is to improve the faithfulness of grounded response generation models, a significant problem in neural language generation
<cit.> which holds high importance in the field of tutoring, where one trusts a teacher to present correct information.
The model is augmented by a sequence of control tokens that are intended to steer the generations to desirable properties. We use the lexical overlap and entailment tokens, which we obtain as follows.
In training, the lexical overlap is measured on a token-level between ground-truth response and grounding.
Then, three equally sized buckets are created indicating low, medium, and high overlap which is indicated by a control token.
Entailment is determined by an MNLI model and again a corresponding token is added.
At test time, we always use the token that encourages the desirable property, in this case high lexical overlap and entailment.
Finally, using a sequence of control tokens $\mathbf{c}$, the model from equation <ref> becomes:

\begin{equation}
\vspace{-0.3em}
p\left(\mathbf{y} \mid \mathbf{a}_{t+1}, \mathbf{c},\mathcal{H}_{<t}, K \right)
\label{eq:ctrl}
\end{equation}

Joint Model
In order to study how well current neural models can decide on a reasonable teaching strategy and perform in real case scenarios, we also implement a model that first decides the dialog act $a_{t+1} \in \mathcal{A}_{\text{teacher}}$ (instead of assuming the ground-truth dialog act) and then uses it to generate a response ${\bf y} = u_{t+1}$.
We use a simple model that again takes the grounding and dialog context as input but now generates the concatenation of dialog act and response in one utterance, akin to SOLOIST <cit.>.
Thus, for a given $\tilde{\mathbf{y}} \coloneqq \mathbf{a}_{t+1} \circ \mathbf{y}$ with act sequence $\mathbf{a}_{t+1}$ of length $N$ and response $\mathbf{y}$ of length T, the model is

\begin{align}
\begin{split}
p\left(\tilde{\mathbf{y}} \mid K, \mathcal{H}_{<t} \right)
= \prod_{i=1}^{m+N} p\left(\tilde{y}_i \mid \tilde{\mathbf{y}}_{<i}, K, \mathcal{H}_{<t} \right)%
\end{split}
\end{align}

In training, we use teacher forcing and prepend $\mathbf{a}_{t+1}$ to $\mathbf{y}$ to obtain the label sequence.
At test time, the model performs a beam search over the dialog act sequence and response jointly.

Retrieval-based model
Since generative models are known to produce erroneous outputs that are factually incorrect and potentially inappropriate <cit.>, we also experiment with using a retrieval-based model that selects responses from the training corpus at test time.
As opposed to previous work on the topic (e.g., <cit.>), we do not employ a rule-based model but rather a learned retrieval model that does not require handcrafting elaborate and possibly brittle rules.
Therefore, we use the Bi-Encoder architecture <cit.> where a dialog context encoder $\text{enc}_{\mathcal{H}_{<t}; \theta}$ and a response encoder $\text{enc}_{\mathbf{y};\theta}$ encode context $\mathcal{H}_{<t}$ and possible responses $\mathbf{y}$ into a fixed size vector of same dimension $n$.
In our experiments, the weights $\theta$ of both encoders are shared.

The model is trained using contrastive learning.
Suppose we are given a training pair $\mathcal{H}, \hat{\mathbf{y}}$ from a training dataset $\mathcal{D}$ that we use for teacher forcing.
We then train the model by sampling a negative response $\bar{\mathbf{y}}$ from the set of responses in $\mathcal{D}$ and using the Triplet Loss criterion, which for a metric function $d : \mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R}$ is defined as:

\begin{equation}
\begin{split}
\mathcal{L}(\theta; \mathcal{H}, \hat{\mathbf{y}}, \bar{\mathbf{y}}) = [m &+ d(\text{enc}_{\mathcal{H}; \theta}(\mathcal{H}), \text{enc}_{\mathbf{y}; \theta}(\hat{\mathbf{y}})) \\
&- d(\text{enc}_{\mathcal{H}; \theta}(\mathcal{H}), \text{enc}_{\mathbf{y}; \theta}(\bar{\mathbf{y}}))]_+ \text{,}
\end{split}
\end{equation}

where $m$ is a margin hyperparameter, and $d$ is the euclidean norm in our experiments.
Further, we do stratified sampling on CIMA to not select negatives from the same preposition, color, or object that might be false negatives. At test time, given a dialog context $\mathcal{H}_{<t}$, we choose a response $\mathbf{y}^\star$ from the training set $\mathcal{D}$ by maximum inner product search using the decision rule

\begin{equation}
\mathbf{y}^\star = \argmax_{ \mathbf{y} \in \mathcal{D}} \{\text{enc}_{\mathcal{H}; \theta}(\mathcal{H}_{<t})^T \text{enc}_{\mathbf{y}; \theta}(\mathbf{y}) \} \text{.}
\end{equation}

§ EXPERIMENTS

3cCIMA      2c|TSCC

Model     sBLEU / BLEU-1 ($\uparrow$)     BERT F1 ($\uparrow$)     $Q^2$ ($\uparrow$)     sBLEU / BLEU-1 ($\uparrow$)     BERT F1 ($\uparrow$)
Rule-based <cit.>$^*$     0.34/-     0.45     -     -     -

LSTM <cit.>$^{*}$    0.31/-     0.53     -     -     -
Seq2seq      2.89 / 28.0     0.676     0.372     -     -

DialoGPT     4.12 / 35.6     0.697     0.571     0.63 / 18.5     0.661

Bi-Encoder (RoBERTa-base)      5.89 / 23.9      0.690     0.344     1.367 / 8.8     0.638

CTRL (BART-base)        5.99 / 42.5      0.702      0.673     -     -

t5-small      7.36 / 34.0     0.672     0.676     2.72 / 12.1     0.646

BART-large     8.61 / 38.7     0.715     0.673     1.85 / 13.7     0.658

BART-base     9.58 / 42.5     0.726     0.680     2.67 / 18.6     0.670

mt5-small     11.26 / 41.0     0.700     0.624     1.80 / 14.9     0.653

BART-base$^\dagger$       5.61 / 41.03     0.707     0.642     1.90 / 15.4     0.659

BART-large$^\dagger$      5.65 / 42.67     0.694     0.607     1.74 / 15.1     0.660

Comparison of models on CIMA and TSCC. We note that the strong sacrebleu differences are caused by the brevity penalty (all generative models generate too short sequences), $^\dagger$: use predicted dialog act label, others use ground-truth. * numbers taken from <cit.> - here, numbers may not be comparable as there is no standard split in CIMA.

We use the following models for parameterizing $p$ in Equation <ref>:
A sequence-to-sequence model <cit.> with a copy mechanism <cit.> trained from scratch.
wide range of pretrained Transformers, namely BART <cit.>, DialoGPT <cit.>, T5 <cit.> and its multilingual version mT5 <cit.>.

BART and T5 are pretrained encoder-decoder models that were trained on denoising and text-to-text tasks, respectively.
mT5 bases on T5 but is multilingual which might help with the code-switched utterances in CIMA.
Lastly, DialoGPT is an autoregressive language model
based on GPT-2 <cit.> that was pretrained on a large dialog dataset obtained from Reddit.
With this, we intend to study whether large-scale dialog-specific pretraining can aid in training educational tutors, as well.

Implementation Details
We implement our experiments using the Huggingface transformers library and finetune the checkpoints provided as part of it for all Transformer-based models.
For these models, we use an initial learning rate of $3.25\times 10^{-5}$, 500 warmup steps and linear learning rate decay.
We train the models using a batch size of 8 and evaluate on the validation sets after each epoch.
In the end, we select the best model to be the one that has a minimal loss on the validation set.
The sequence-to-sequence baseline is trained from scratch using an initial learning rate of $0.001$ for 25,000 steps using the Adam optimizer and a dropout rate of $0.1$
We use beam search with a beam size of 10 to generate model responses.

§.§ Dataset splits

Since there are no official dataset splits for CIMA and TSCC, we split both datasets randomly into training, validation and test sets. We provide the exact split of the dataset in an accompanying code repository.
For CIMA, we use all such samples with less than three annotated tutor responses for training.
The other conversations are split randomly into equally-sized validation and test sets which results in 2715/300/300 samples each.

For TSCC, we split randomly along the conversations to obtain 82/10/11 training, validation, and test conversations each.

§.§ Evaluation metrics

To evaluate our models, we use the BLEU implementation provided by the sacrebleu package (sBLEU) <cit.> to measure lexical overlap between generated and ground-truth response.
Furthermore, we use BERT F1 (BERTScore) to measure their semantic similarity.
Lastly, for CIMA we also calculate $Q^2$ <cit.> which measures the factual consistency of the response $\mathbf{y}$ with the grounding information $K$ by employing a question-answering based matching.
Both BERTScore and $Q^2$ have shown strong correlation with human judgements on factual consistency <cit.>.

§ RESULTS

In this section, we summarize our main findings in terms of automatic evaluation.
First, we give an overview of the performance of different models that we train on CIMA and TSCC in Section <ref>.
Then, we assess their ability to stay faithful to teaching strategies (Section <ref>) and study how grounding annotations can influence the faithfulness of neural dialog tutors (Section <ref>), before studying their scaling behavior with dataset size and complexity (Section <ref>) and their generalization capabilities (Section <ref>). We then finish with an assessment of using education-specific data for pretraining (Section <ref>).

§.§ Comparison of different models

Table <ref> shows the key results from our experiments.
First, all automatic metrics are significantly higher on CIMA, which indicates that the models can fit CIMA much better than TSCC, with which current approaches still struggle. We further analyse this finding in Section <ref> and show that this is because TSCC has richer teaching strategies which are harder to model.
Our comparison also suggests that finetuning large pretrained Transformer models generally gives better results than the rule-based and LSTM model reported in <cit.>, and our implemented retrieval and sequence-to-sequence baselines. This illustrates the potential of LLMs for dialog tutoring.

We also see a significant difference among different LLMs.
Dialog-specific pretraining of DialoGPT does not help and gives worse results than BART and T5, primarily because the model tends to generate short and generic responses more often.
Multilingual pretraining in mT5 improves over T5 only in some metrics, notably in BLEU and BERT F1 on CIMA but not in terms of $Q^2$.
Similarly, adding control tokens to BART does not improve $Q^2$ or other automatic metrics.
Surprisingly, using very large models
actually degrades performance in our experiments.
Finally, the last two rows show results obtained with our joint model that does not use the ground-truth dialog act but predicts it together with the response sequence and still provides reasonable performance.

§.§ How well can generative models capture teaching strategies?

GT     BART$_{\text{base}}$     BART$_{\text{large}}$      CTRL     Retrieval
DA F1     78.3     81.0     70.1     63.0     43.1

F1 score of the dialog act classification based on the generated responses of our models.

ymin=0, ymax=0.6,
bar width=12,
xticklabels=,Hint,Question,Correction,Confirmation,Other, ,
xticklabel style=rotate=30,
+[teal, opacity=0.5, draw=black] coordinates

(5, 0.0)
(4, 0.052)
(3, 0.147)
(2, 0.276)
(1, 0.526)
+[brown, opacity=0.75, draw=black] coordinates

(5, 0.037)
(4, 0.063)
(3, 0.228)
(2, 0.237)
(1, 0.434)

Distribution of predicted and ground-truth dialog acts on CIMA.

We study this question first by evaluating the dialog act prediction accuracy of our joint model.
We find that it is significantly low on TSCC with $21.8$ compared to $71.2$ on CIMA for BART-base which indicates significant room for improvement.
Notably, the joint model tends to predict more frequently occuring dialog acts, which results in fewer follow-up questions and "Other" never being predicted in CIMA, the least frequent act in the data. The distribution of dialog acts in the ground-truth annotations and model predictions with a BART-base joint model is in Figure <ref>.

Then, we evaluate how well different models can stick to a given ground-truth dialog act by predicting the dialog act of the generated response with a BART-base model trained to predict the ground-truth dialog act sequence based on the ground-truth response.
The results are shown in Table <ref>.
Notably, BART-base performs better than the ground-truth annotations. The CTRL model, on the other hand, has worse performance since the control tokens do not respect tutoring principles (e.g., lexical overlap to grounding discourages follow-up questions in favor of just giving hints).

§.§ Does grounding in learning concepts help?

Model              sBLEU ($\uparrow$)     BERT F1 ($\uparrow$)     $Q^2$ ($\uparrow$)
BART-base        6.69 / 38.6     0.718     0.571

$\;\;$ + triples         9.20 / 45.3     0.730     0.642

$\;\;\;$ + grammar rules     9.58 / 42.5     0.726     0.680

Comparison of models with different inputs on CIMA. Triples are made up of preposition, object, and color translations. Grammar rules are a textual description of a learning concept.

Prior work has shown that grounding responses in relevant data can improve their quality, especially in terms of faithfulness <cit.>.
We intend to validate this for dialog tutoring by studying three models with different inputs on CIMA.
The first model is not provided grounding information, whereas
the second and third are grounded in learning concepts (cf. Equation <ref>) with one using only the (preposition, object, color) triples and the other making use of additional grammar rules.
The results with these models are shown in Table <ref> and suggest that grounding responses in relevant knowledge helps the model to produce better and more faithful responses.

§.§ How do models scale with more data?

Due to the limited availability of high-quality pedagogical datasets and the time-consuming process of authoring new materials <cit.>, it is important to understand how quickly generative models can generalize to new settings.

axis y line*=left,
xlabel=% of training data used,
xlabel near ticks,
ylabel=$Q^2 \cdot 100$ <ref>,
ylabel near ticks
+[olive,mark=*,mark options=fill=gray, domain=35:0] coordinates (50, 100);

+[olive,mark=*,mark options=fill=gray, domain=35:0] coordinates
(100, 68.0)
(90, 63.2)
(80, 61.6)
(70, 63.4)
(60, 64.1)
(50, 56.5)
(40, 55.9)
(30, 55.3)
(20, 48.4)
(10, 48.1)

xmin = 0, xmax = 109,
ymin = 0, ymax = 29,
hide x axis,
axis y line*=right,
ytick style=draw=none
+[cyan,mark=*,mark options=fill=gray, domain=35:0] coordinates (50, 100);

+[cyan,mark=*,mark options=fill=gray, domain=35:0] coordinates
(100, 9.59)
(90, 8.92)
(80, 9.89)
(70, 8.67)
(60, 8.61)
(50, 9.10)
(40, 8.69)
(30, 8.99)
(20, 7.36)
(10, 3.11)

axis y line*=left,
xlabel=No. of concepts,
xlabel near ticks,
ytick style=draw=none
+[olive,mark=*,mark options=fill=gray, domain=35:0] coordinates (50, 100);

+[olive,mark=*,mark options=fill=gray, domain=35:0] coordinates
(1, 55.7)
(2, 58.3)
(3, 58.3)
(4, 47.5)
(5, 48.0)
(6, 46.3)

xmin = 0, xmax = 7,
ymin = 0, ymax = 29,
hide x axis,
axis y line*=right,
ylabel=sBLEU <ref>,
ylabel near ticks
+[cyan,mark=*,mark options=fill=gray, domain=35:0] coordinates (50, 100);

+[cyan,mark=*,mark options=fill=gray, domain=35:0] coordinates
(1, 6.2)
(2, 6.2)
(3, 6.6)
(4, 7.5)
(5, 8.8)
(6, 7.9)

Performance of BART-base on CIMA as a function of: (a) training data size uniformly sampled from the training data, (b) the number of concepts, where only the specific number of concepts is retained and all others are excluded.

Thus, we assess how well the model can model tutoring in low-resource scenarios. We construct a study, where we randomly sample subsets of the CIMA training set and test the performance of the various models.
We can see from Figure <ref>(a) that with more training data, the faithfulness of responses appears to improve and is not saturated before we reach the full training set.
This supports the intuition that additional training data might improve the performance further.

Similarly, we study how well our model can deal with an increase in complexity with respect to learning concepts at similar training data sizes.
Therefore, we construct different training datasets with 735 samples and a varying number of concepts each time.
We begin by taking samples concerned with the concept “in front of the” and evaluate exclusively on it, gradually adding new concepts.
Figure <ref>(b) suggests that $Q^2$ drops sharply at four concepts.
BLEU on the other hand increases, and this might be due to the metric encouraging generic utterances that, for example, repeat a grammar rule.

§.§ Can models generalize to new concepts?

As the students progress and gain new knowledge, it might be a desirable property of dialog tutoring models to be able to handle new concepts that suit this increase in prior knowledge.
Hence, we study how well our CIMA model can generalize to new concepts that it has not seen in training, for example, a new preposition.
For this analysis, we create a set-up where we first train the model on all of the training data and evaluate on the subset of samples for each preposition separately.
We then compare this number to a model that is not trained on the corresponding concept it is evaluated on, creating a zero-shot set-up which we carry out for a grounded and ungrounded response generation model.
As measured by $Q^2$ (cf. Table <ref>), this model can indeed generalize to new concepts well, albeit with performance degradation.
Furthermore, grounding information improves generalization as these define the learning concept (in this case the preposition) and how it is used.
Without this information, we observe that the model generates generic responses more often.

Concept     #Samples     full data     zero-shot     zero-shot

without grounding
train/test     $Q^2$     $Q^2$     $Q^2$

is behind the     549/90     0.698     0.603     0.533

is in front of the     735/84     0.616     0.512     0.500

is next to the     547/51     0.497     0.539     0.483

is on top of the     224/30     0.683     0.578     0.567

is under the     270/24     0.854     0.646     0.625

is inside of the     390/21     0.579     0.643     0.190
all concepts     2715 / 300     0.644     0.570     0.502

Performance of a grounded BART-base model by learning concept. Full data uses the entire training data and zero-shot removes the concept of the row from the training data.

§.§ Does education-specific pre-training help?

Method     sBLEU      BERT F1     $Q^2$

BART-base     6.69 / 38.6     0.718     0.571

+ Ed. data     7.31 / 41.4     0.727     0.577

+ Non-Ed. data     6.60 / 39.4     0.721     0.583

Influence of pretraining on educational and non-educational data. Please note that no grounding information is used in this setting.

As educational data are widely available on the internet, next we study how education-specific pre-training effect results. In Table <ref>, we show results obtained with finetuning a BART-base model directly on CIMA and pretraining it on tutoring dialogs from TSCC or non-tutoring dialogs from MultiWoZ 2.1 <cit.>, Personachat <cit.>, CMU DoG <cit.>, DSTC9 <cit.> and Topicalchat <cit.>.
In both cases, we only see minor improvements, which may be explained by the different dataset settings and the lack of a unified dialog act taxonomy.

§ HUMAN EVALUATION

We further evaluate previously assessed models with human judgments firstly by obtaining quality estimates according to different criteria and secondly by conducting a simulation study, where expert annotators are asked to provide novel rewritings of existing conversations and to categorize errors made by the model.

§.§ Quality of the generated responses

We perform a human quality evaluation of the generated response for four models - retrieval (Bi-Encoder), BART-base, BART-base$_{\text{CTRL}}$ and the joint model (BART-base). A randomly chosen subset of the CIMA test set conversations were annotated by 4 annotators (with one annotator speaking C1 level Italian). All annotators labeled 60 examples in total, of which 20 overlapped. To further distinguish the quality of training data for the models, we annotated ground-truth responses on a small sample of 20 examples. We evaluate the following criteria on a 3-point Likert scale (disagree to completely agree) and outline our findings in the following, as shown in Figure <ref>.

Fluency "The response is grammatically correct and fluent."
We find that all models have very high fluency scores.

Coherence "The response naturally follows up on previous utterance and context and has no logical conflicts with the context or DA label."
We find that all generative models are able to produce coherent responses but not the retrieval model.

Correctness "The response is factually correct and respects learning concepts being taught."
All models score comparable to ground-truth responses on the constrained CIMA dataset.
It is noteworthy, however, that a response may be correct in itself but not coherent with the context or the grounding (often the case in the retrieval model), and this could explain the discrepancy between correctness and our automatic $Q^2$ scores.

Equitable tutoring "The response gives a learning opportunity for the student by providing space for reflection, explanation, pointing to follow-up challenge, or engaging student in other ways."
Here we find significant deficiencies not only for our evaluated models but notably also for the annotated ground-truth responses (gt).
This might explain the insufficiencies in the responses as they reflect this distributional behavior of the training data.
We think that future dataset collections should take better care of this property and resort to more expert annotators as opposed to crowdsourcing.

Comparison of models on four criteria (reporting $M$) in the human quality evaluation. We observed high $SD$ for coherence and equitable metrics.

Quality Attribute     sBLEU     BERTScore
Fluency     0.14     0.12

Coherence     0.17     0.26

Correctness     0.06     0.15

Equitable Tutoring     0.08     0.16

Pearson correlation coefficients between the human judgements on our quality criteria and automatic metrics.

Furthermore, Table <ref> shows that our automatic metrics correlate poorly with human judgements.

§.§ User study with a learning interface

Lastly, we seek to study how well dialog tutoring models can perform in a realistic setting with questions obtained from real users (containing out-of-distribution samples) and not the fixed dataset.
Therefore, we randomly sampled conversations from the CIMA test set. We asked two C1-level expert Italian speakers to 1) rephrase these conversations using a conversational dialogue interface and 2) assign erroneous model responses to predefined error categories.
The interface used in the qualitative evaluation is shown in Figure <ref>.
We obtain all model responses from the BART-base model that first predicts the dialog act and then the response.
Error categories adopted from previous work <cit.> describe the ideal behavior of tutoring models as simulating the behavior of good human teachers along two dimensions:

Understanding "Being able to understand and reason about student solutions, misconceptions, and learning concepts."
We find that of the 20 modified conversations, 45% exhibit Understanding errors, such as an incorrect solution assessment or incorrect translations.

Pedagogy"Being able to use effective pedagogy to instruct students."
We find that 10% of the responses exhibit Pedagogical errors, for example telling the correct solution directly without offering any engagement point to the student.

50% of the conversations were labeled good by the annotators.
Examples of the conversations are available in Table <ref>.

§ DISCUSSION: TOWARDS MORE EQUITABLE AND FAITHFUL TUTORING SYSTEMS

In this section, we outline directions of research that we think can be important steps towards more equitable and faithful tutoring models.
Namely, we first address the small scale and quality of current tutoring datasets and cast doubt on the crowdsourcing data quality checks.
Then, we suggest ways of improving the underperformance of both equitable tutoring and teaching strategy prediction identified in current generative models under these constraints by drawing from learning sciences literature.
Finally, we outline desiderata for more reliable dialog evaluation of neural tutoring models.

Datasets Based on the analysis in <ref> and Table <ref>, we think that the community will benefit from a dataset that lies between CIMA and TSCC in terms of its difficulty.
Moreover, the low equitable tutoring scores of CIMA's ground-truth responses indicate that crowdsourcing with untrained annotators can lead to low pedagogical quality.
A similar observation has been found by human evaluation for the TSCC dataset <cit.>.
Finally, we encourage the establishment of better dialog act taxonomies that are backed by learning sciences research.
As outlined in <ref> and in <cit.>, a unified taxonomy may also strongly aid in transfer learning.

Models So far, dialog tutoring models have only covered limited domain-specific settings linked to a particular activity, such as learning Italian prepositions or solving math word problems.
We argue that the community could benefit from working on problems common to learning in general, for example tracking problem-solving states and modeling pedagogies used by teachers. Here, knowledge tracing <cit.> (the problem of estimating students’ skill mastery level) could be used for tracking problem-solving states and increasing the coherence of dialog tutoring conversations and dialog act selection performance which would contribute to better modeling of global teaching strategies.
Furthermore, validated instruction quality coding schemes <cit.> used by classroom teachers can be computationally modeled <cit.> and incorporated into models.

We also think that recently proposed constrained decoding approaches that can balance between multiple criteria <cit.> hold great promise in improving faithfulness in complex tutoring dialogs.
Finally, as data collection is labor-intensive in expert domains, we see great potential in few-shot learning methods, such as prompt-based methods <cit.>.

Our experiments highlight the insufficiencies of current automatic dialog evaluation metrics, as both BLEU and BertScore show comparatively low correlation with our collected human judgements from <ref>.
This is in line with previous research <cit.> and shows the necessity not only for better automatic evaluation metrics but also for verification based on human judgements or user studies that should incorporate criteria relevant to tutoring (e.g., equitable tutoring outcomes).
Metrics that incorporate task success, which have been used in task-oriented dialog systems <cit.>, are a promising direction of future research for automatic evaluation.

§ CONCLUSION

In this work, we reflected on the state of research in dialog tutoring and explored the potential of neural generative models in this domain. We found some promising initial results with these models in comparison to rule- or retrieval-based methods.
However, we also established limitations of currently available benchmarks and evaluation criteria.
Furthermore, we showed that there are a number of challenges that need to be addressed before neural generative models of text can be deployed as intelligent tutoring systems on a larger scale, such as controllability and being able to model a sound pedagogical strategy.
Based on these findings, we outline potential avenues for future research.

§ LIMITATIONS

A key limitation of our work is the use of only two available tutoring datasets. Despite a limited number of datasets available in this domain, using the TalkMoves dataset <cit.> could help further generalize our findings. This remains an avenue for future work.

Based on the prior work, we focused on the specific conversational goal of dialog tutors which is providing learning aid for students' skill development and more opportunities to learn. While this is the most widespread type <cit.>, it is not covering all the goals of human tutors, and other aspects could be important, for example, rapport-building or mentoring on the meta-cognitive level. We acknowledge this both as a prerequisite for our work and at the same time as a limitation. For further discussion we refer the reader to Appendix <ref> and <ref>.

Finally, our user study could be further extended with more participants. In the future, we plan a more comprehensive study with real language learners using an end-to-end dialog tutoring system.

§ ETHICS STATEMENT

We do not foresee any significant harm directly as a result of our work. Having said that, we must understand that automatic tutoring is a high-stake setting that can pose significant harm if appropriate care is not taken before the deployment of these systems. Issues of biases and lack of trust, and other ethical issues such as privacy concerns must be considered. Considering learners only as data points within a neural dialog tutoring context may prevent us from seeing the societal and socioeconomic barriers that they may be up against, thereby running the risk of not only failing to help relevant learner subgroups but also sometimes giving additional privileges to those who use these systems.

§ ACKNOWLEDGEMENTS

This project was made possible by an ETH AI Center Doctoral Fellowship to Jakub Macina with partial support by the Asuera Stiftung and the ETH Zurich Foundation and has received funding by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE.
We thank the group members and our reviewers for their valuable feedback.

§ PEDAGOGICAL STRATEGY AND DIALOG ACTS IN DIALOG TUTORING

Example dialogue between a tutor and a student solving an algebra story problem. Key questions are: What teacher pedagogical strategies are the best in terms of learning gains of students? How to adapt language models to generate pedagogically valid responses?

In the context of this paper, we assume that the pedagogical strategy is represented using dialog act annotations. An example of the teacher strategy is providing hints (cf. example in Figure <ref>), where a teacher provides helpful support or clarifies goals to the student. Another example is Probing (cf. example in Figure <ref>), which prompts students to explain better or reflect on the current solution. CIMA contains five teacher dialog acts - hint, open-ended question, correction, confirmation, other. TSCC contains more fine-grained dialog acts such as eliciting, scaffolding, enquiry, or recap.

From a learning science standpoint, pedagogical strategy could be viewed as a global strategy (knowing how to effectively guide students e.g. using questioning or providing contrasting cases) and dialog acts as a specific decision on how this strategy is implemented on the local turn-based level.

§ EQUITABLE TUTORING

Although tutoring is typically conceived as a scenario where a subject matter expert works synchronously with one or multiple students and takes interpretive authority, there is increasing empirical evidence supporting the case for incorporating active learning approaches in the classroom <cit.>. With collaborative creation of knowledge where teachers position themselves as co-learners and students also take interpretive authority, such approaches are better poised to build classroom equity than monologic educational practices where only one voice (primarily the teacher's) tends to be heard, legitimized and sometimes imposed. Therefore, if we rethink of the goals of education as providing opportunities for students to enter into the workforce with a positive identity about themselves and the subject matter, equitable tutoring via increased student chances to pose ideas, construct knowledge and as a result feel welcomed into the intellectual discussion, holds tremendous promise.

§ CONVERSATIONAL GOALS

In this work, we studied only 1:1 dialog tutoring settings with a specific focus on the role of a teacher/tutor. We focused on the most commonly used goal of dialog tutoring which is a learning aid to support students' skill development and provide opportunities to learn <cit.>.

However, teacher-student interactions may entail multiple conversational goals
that can serve interactional functions (e.g., turn-taking) and interpersonal functions (e.g., rapport-building moves such as self-disclosure, praise, social norm violation). Research in human tutoring and collaborative learning, more generally, has shown that how students manage the task space (consisting of the problem to be solved, e.g., do I agree with the interlocutor's reasoning?) and the relational space (consisting of the interactional challenges and opportunities, e.g., can I disagree without threatening the interlocutor's face?) is critical to learning outcomes <cit.>. Neural dialog tutoring, which can account for this crucial, but fundamental distinction among conversational goals is challenging.

§ QUALITATIVE USER EVALUATION

The interface used in the qualitative evaluation is shown in Figure <ref>.

User interface used for the qualitative evaluation. The interface is based on ParlAI <cit.> library.

Context     Generated teacher response

T: Please try to fill in the blank in Italian. S: how do you say next? T: Remember that  "is next to the" is "e vicino" S: I don't know how to say purple. Is it viola?

Viola is the word for purple.

T: Please try to fill in the blank in Italian.  S: how do you say next? T: Remember that  "is next to the" is "e vicino" S: Purple is viola, right?

That is correct.

T: Please try to fill in the blank in Italian. S: how do you say next? T: Remember that  "is next to the" is "e vicino" S: I think purple could be violett.

Viola is the word for purple.

T: Please try to fill in the blank in Italian.  S: how do you say next? T: Remember that  "is next to the" is "e vicino" S: I can't remember the color. Can you give me a hint?

Purple is viola.

T: Please try to fill in the blank in Italian.  S: how do you say next? T: Remember that  "is next to the" is "e vicino" S: I guess it is e vicino all'albero. Is it correct to start the sentence with the preposition?

The preposition for "purple" is "viola" Understanding - Meaning mismatch

T: "Blue" is "blu". Please try to fill in the blank in Italian. S: But what are the other words? T: Can you give me your best guess? S: e di fronte all'albero blu  (correct)

Almost, tree is all'albero'. Please try again. Understanding - Incorrect solution assessment

T: "Blue" is "blu". Please try to fill in the blank in Italian. S: But what are the other words? T: Can you give me your best guess? S: e di fronte blu all'albero  (adjective position swap)

Almost, you need to add blu after "all'albero" to make the word "tree". Understanding - Incorrect translation

T: "Blue" is "blu". Please try to fill in the blank in Italian. S: But what are the other words? T: Can you give me your best guess? S: e di fronte al albero blu (wrong definite article)

Almost, tree is all'albero not albero.

T: "Blue" is "blu". Please try to fill in the blank in Italian. S: But what are the other words? T: Can you give me your best guess? S: e di fronte al cane blu (noun swap)

Almost. Al cane is not the correct word for tree.

T: "Blue" is "blu". Please try to fill in the blank in Italian. S: But what are the other words? T: Can you give me your best guess? S: e dietro all'albero blu" (wrong preposition)

The correct answer is e di fronte all'albero blu. Pedagogy - Information reveal

Examples of rephrased conversations from the qualitative study. $T$ refers to a teacher utterance, $S$ refers to a student utterance. Bold text is information for the reader indicating error categories.