# A Context-Aware Hierarchical BERT Fusion Network for Multi-turn Dialog Act
Detection

###### Abstract

The success of interactive dialog systems is usually associated with the
quality of the spoken language understanding (SLU) task, which mainly
identifies the corresponding dialog acts and slot values in each turn. By
treating utterances in isolation, most SLU systems often overlook the semantic
context in which a dialog act is expected. The act dependency between turns is
non-trivial and yet critical to the identification of the correct semantic
representations. Previous works with limited context awareness have exposed
the inadequacy of dealing with complexity in multiproned user intents, which
are subject to spontaneous change during turn transitions. In this work, we
propose to enhance SLU in multi-turn dialogs, employing a context-aware
hierarchical BERT fusion Network (CaBERT-SLU) to not only discern context
information within a dialog but also jointly identify multiple dialog acts and
slots in each utterance. Experimental results show that our approach reaches
new state-of-the-art (SOTA) performances in two complicated multi-turn
dialogue datasets with considerable improvements compared with previous
methods, which only consider single utterances for multiple intents and slot
filling.

Index Terms: context, multi-intent, human-computer interaction, task-oriented
dialog, BERT

##  1 Introduction

With the recent success of service assistants such as Alexa, Cortana and Siri,
research attempts to expanding spoken language understanding (SLU)
applications are becoming ubiquitous [1]. In canonical task-oriented dialogs,
SLU establishes so-called semantic frames by capturing semantics in terms of
intents and slots from speech recognized utterances [2]. These intents specify
goals which commit speakers to some course of actions, like requesting,
informing or acknowledging with a series of semantic notions known as slots.
In the full dialog scenario, we refer these intents for an utterance within a
dialog turn as dialog acts [3]. In Table 1 as an example in Microsoft dialogue
challenge dataset [4], each utterance or response may involve more than one
dialog acts with or without specific targeted slots.

Within traditional SLU frameworks, such identification process is usually
articulated as a single intent classification task coupled with a slot
labeling task by dissecting a dialog into single utterances [5, 6]. While most
works induce large success in modeling the separate or joint distribution from
intents and slots [7, 8, 9, 10], systems trained with such independent
locutionary sentences quickly suffer from the insufficiency of capturing
comprehensive semantics as the dialog flows. Irregularity and high mutability
of user utterances may make it more difficult to capture precise user intents,
especially for colloquial or implicit utterances [11]. Moreover, in real world
scenario, an utterance can be associated with more than one intent [7, 12, 13,
14]. Dominant SLU systems have adopted several techniques to model such
sophisticated semantic natures. [14] first explored the joint multi-intent and
slot-filling task by treating multiple intents as a single context vector, but
not scalable to a larger number of intents. [13] proposed a SOTA model to
exploit slot-intent relations with the graph attention.

Table 1: Snippet of a single turn within dialog data conversation with corresponding dialog acts and slots. Speaker | Utterance
---|---
1\. system |  | I had 10 restaurants. 2g Japanese
---
Brasserie is great for you.
Act: Offer |  Slots: (name: 2g Japanese Brasserie)
Act: Inform_count |  Slots: (count: 10)
2\. user |  | Yes, 2g Japanese works.
---
I want to reserve there.
Act: Inform_intent |  Slots: (reserve_restaurant: True)
Act: Select |  Slots: (name: 2g Japanese)
3\. system |  | Please confirm the following details:
---
Booking a table at 2g Japanese
Brasserie. The city is San Francisco.
Act: Confirm |  |  Slots: (name: 2g Japanese Brasserie)
---
(city: San Francisco)
4\. user |  | Yes the restaurant schedule works
---
for me. Do they have live music?
How pricey is it?
Act: Affirm |  Slots: ()
Act: Request |  |  Slots: (has_live_music: None),
---
(price_range: None)
Figure 1: Illustration of our proposed framework for joint dialog act
detection and slot filling in multi-turn dialogs.

However, these approaches trained with independent utterances may not be
sufficient in detecting contextual natures of dialog acts within dialogs,
especially with the multiple intent cases [4, 15]. First, the sequential
dependency between acts are obvious in most dialog cases regardless of
domains. For instance, in Table 1, we can see ‘Select‘ usually comes after
‘Offer‘, and ‘Select‘’s slot is usually one of ‘Offer‘’s slots. Second, in
less stylized conversations, they usually exhibit a broader open set of less
bounded purposes, subject to arbitrary changes during turn transitions [16].
For instance in utterance 4, user presents another requests for the price even
after the confirmation is done, which may result from heuristics of the
‘Japanese Brasserie‘ restaurant name, which sounds expensive. Without such
correlation matching, interpretability is undermined for joint tasks without
any contextual information. Although in pipeline-based frameworks [1], the
resolution of contextual utterances is typically addressed in the next dialog
management module (DM) [17], dissecting dialogs unnaturally may still open the
door for significant cascade of errors by neglecting contexts.

To avert such error propagation, some end-to-end dialog systems [18, 19] have
been proposed to directly bypass the necessity of tracing dialog acts.
Nevertheless, they lead to lower accuracies and tend to be opaque with less
semantic transparency of dialog policies and states. Instead, [20] studied
contextual phenomena, thereby emphasizing the use of natural language
contexts. [21] introduced contextual signals to the joint intent-slot tasks.
However, contextual information in their work was limited in terms of turn
interactions. [22] further proposed CASA-NLU to incorporate miscellaneous
context signals until the current turn to jointly predict intents and slots.
However, naive attention between historical utterances may lose track of
sequential information traversing with the dialog progress. [23] and [17] in
dialog state tracking leveraged contexts to predict utterance-level slot
values, which however may not be compatible with an unknown ontology such as
restaurant names.

In this work, we present a context-aware hierarchical BERT fusion network
(CaBERT-SLU) to exploit dialog history for joint tasks. Simply, CaBERT-SLU
will extract both utterance and turn-level information to identify multiple
dialog acts and exploit a slot tagger to predict slots during the entire
dialog. Our contributions are as follows and the code is available in
https://github.com/waynewu6250/CaBERT-SLU.

1\. We propose CaBERT-SLU, which is the first attempt to consider previous
dialog history for joint multiple dialog act and slot filling tasks, where
previous SLU works usually isolate the utterances and only detect single
dialog act only.
2\. We demonstrate the effectiveness of context fusion attention in joint
tasks with the ablation study and visualization.
3\. Experimental results show that our model achieves SOTA performances over
several competitive baselines.

##  2 Methodology

Table 2: Main results for the joint task on two datasets. We report accuracy (ID Acc) for all intent exact match, F1 scores (ID F1) based on each intent calculation. We also report intent accuracy (IO Acc) and F1 score (IO F1) with models trained only on intent detection task. $\ddagger$ indicates that Stack-Prop can only predict single intent which we solely report its ID/IO Acc. $\dagger$ indicates the significant improvement of $p$-value $<$ 0.05 compared to the previous best contextual baseline [22]. Dataset | MDC | SGD | MDC | SGD
---|---|---|---|---
Model | ID F1 | ID Acc | SL F1 | ID F1 | ID Acc | SL F1 | IO F1 | IO Acc | IO F1 | IO Acc
Stack-Prop† [12] | - | 82.00 | 78.86 | - | 83.51 | 89.24 | - | 82.30 | - | 83.75
Joint MID-SF [14] | 86.18 | 75.75 | 70.92 | 86.33 | 85.11 | 78.21 | 85.48 | 75.41 | 90.71 | 84.97
AGIF [13] | 89.63 | 80.18 | 78.87 | 91.96 | 85.41 | 86.68 | 90.73 | 74.12 | 92.41 | 76.11
ECA [24] | 87.88 | 77.84 | 69.20 | 93.65 | 93.11 | 80.00 | 87.75 | 77.66 | 95.78 | 92.45
BERT [25] | 90.67 | 81.84 | 78.21 | 95.23 | 92.75 | 89.65 | 89.71 | 81.43 | 94.96 | 92.63
BERT+SA [26] | 89.91 | 80.63 | 78.19 | 95.32 | 92.99 | 90.01 | 89.73 | 81.64 | 94.95 | 92.73
BERT+CASA-NLU [22] | 90.98 | 82.17 | 78.16 | 97.07 | 95.24 | 90.46 | 90.25 | 80.91 | 96.72 | 94.84
CaBERT-SLU | 91.26 | 83.05† | 79.64† | 99.14† | 98.59† | 95.71† | 90.81 | 82.69† | 98.92† | 98.24†

###  2.1 Problem Statement

Suppose we have a predefined dialog act label set $\mathcal{Y}^{a}$ and a slot
set $\mathcal{Y}^{s}$, given a dialog $X=\\{x_{1},x_{2},\dots,x_{T}\\}$ of
total $T$ user utterances and system responses, we would like to detect
multiple dialog acts and slots for each $x_{t}$. For dialog act detection, we
formulate it as a multi-label classification problem where for each $x_{t}$,
we aim to find multiple dialog acts
$(y_{a,t}^{1},y_{a,t}^{i},...,y_{a,t}^{N_{t}}),\forall
y_{a,t}^{i}\in\mathcal{Y}^{a}$, and $N_{t}$ is the total number of dialog acts
of the sample $x_{t}$. And for the slot filling task, for each
$x_{t}=\\{w_{t}^{1},w_{t}^{2},...,w_{t}^{N}\\}$ with total $N$ words, we wish
to learn a parameterized mapping function to map input words into
corresponding slot tags $(y_{s,t}^{1},y_{s,t}^{i},...,y_{s,t}^{N}),\forall
y_{s,t}^{i}\in\mathcal{Y}^{s}$.

###  2.2 BERT self-attentive encoder

As shown in Figure 1, our model consists of four functional units. We first
encode each sentence $x_{t}=\\{w_{t}^{1},w_{t}^{2},...,w_{t}^{N}\\}$ in a
dialog $X$ with a BERT encoder $BERT_{u}$ to obtain token-level
representations $\\{h_{t}^{1},h_{t}^{2},...,h_{t}^{N}\\}$. BERT [25] is a
multi-layer transformer-based encoder containing multi-head self-attention
layers. It sufficiently extracts the contextualized information for each word
token with respect to overall utterance. For a dialog with $T$ sentences, such
$T$ token-level hidden representations will be sent into both the downstream
context fusion encoder and the slot tagger to respectively predict dialog acts
and slots.

To further obtain the sentence representation $s_{t}$ of each utterance
$x_{t}$ based on $\\{h_{t}^{1},h_{t}^{2},...,h_{t}^{N}\\}$ and better consider
the individual word importance, we follow the work in [26] to use a self-
attentive network. At each time step $i$ at sentence $x_{t}$, we first feed
each token-level hidden state $h_{t}^{i}$ into an affine transformation
($W,b_{w}$), $\bar{h}^{i}_{t}=Wh_{t}^{i}+b_{w}$. And we use equation 1 to
obtain score $\alpha^{i}_{t}$.

$\displaystyle\alpha^{i}_{t}$
$\displaystyle=\frac{e^{{\bar{h}{{}^{i}_{t}}}^{T}u_{w}}}{\sum_{j}e^{{\bar{h}{{}^{j}_{t}}}^{T}u_{w}}}$
(1)

Then $\\{\alpha^{i}_{t}\\}$ represents the similarity scores between each
$h_{t}^{i}$ and $K$ heads of learnable context vectors $u_{w}$ which indicate
the global sentence views; for each head, we can get a sentence representation
$s^{h}_{t}=\sum_{i}\alpha^{i}_{t}h^{i}_{t}$. Finally we will concatenate all
the heads for the final representation $s_{t}$.

###  2.3 Context fusion encoder

After obtaining the final sentence representation
$S=\\{s_{1},s_{2},\dots,s_{T}\\}\in\mathbb{R}^{T\times H_{b}}$ for a dialog,
where $H_{b}$ is BERT hidden size, we combine $\\{s_{1},s_{2},\dots,s_{T}\\}$
with a unidirectional transformer encoder, which is devised to model the
contextual relevance information throughout $T$ sentences in the dialog. This
context fusion encoder contains a stack of $N_{l}$ layers. There are a masked
multi-head self-attention sublayer (Attention) and a point-wise fully
connected feed-forward network (FFN) as shown in equation 3. It will first
project $S$ with weight matrices: $W^{Q},W^{K},W^{V}\in\mathbb{R}^{H_{b}\times
H_{a}}$ to be $S^{Q}=SW^{Q}$, $S^{K}=SW^{K}$, $S^{V}=SW^{V}$. Then each of
them will be separated into $h$ heads, with each head $i$ to be
$H_{i}\in\mathbb{R}^{T\times(H_{a}/h)}$, $H_{a}$ is the hidden size for the
attention module. These $H_{i}$ will be sent into the self-attention layer.

$Attention(H^{Q}_{i},H^{K}_{i},H^{V}_{i})=softmax(\frac{H^{Q}_{i}(H^{K}_{i})^{T}}{\sqrt{H_{b}}})H^{V}_{i}$
(2)

$FFN(x)=max(0,xW_{1}+b_{1})W_{2}+b_{2}$ (3)

Then for the entire context fusion layer, we can unfold each layer $l$ in
equation 5, where $PE(\cdot)$ denotes positional encoding function. Here we
omit the residual layer and layer normalization layer in the equation, which
exist in real implementation.

$\displaystyle C^{1}$ $\displaystyle=S+PE(S)$ (4) $\displaystyle C^{l}$
$\displaystyle=FFN(Attention(C^{l-1},C^{l-1},C^{l-1})$ (5)

###  2.4 Global Recurrent Unit

We found out that the context fusion encoder could introduce the mutual
interaction between each utterance, which may nevertheless be insufficient to
capture the global sequential information as the dialog progresses. Thus, we
apply an additional unidirectional LSTM layer upon the context fusion layer to
supplement such global relations to obtain the final output states
$H_{act}\in\mathbb{R}^{T\times H_{L}}$, where $H_{L}$ is the hidden size of
LSTM.

$\displaystyle H_{act}=LSTM(C^{N_{l}})$ (6)

Then we can generate the logits $\hat{y}_{a}=\sigma(H_{act}W_{act})$ by
transforming $H_{act}$ with
$W_{act}\in\mathbb{R}^{H_{L}\times|\mathcal{Y}^{a}|}$ and a sigmoid function
$\sigma$. Finally we can have the dialog act detection objective as a binary
cross entropy loss where $N_{s}$ is number of samples, $T$ is the max dialog
length in samples and $|\mathcal{Y}^{a}|$ is the number of total dialog acts:

$\displaystyle\mathcal{L}_{a}\coloneqq-$
$\displaystyle\sum_{i=1}^{N_{s}}\sum_{t=1}^{T}\sum_{a=1}^{|\mathcal{Y}^{a}|}(y^{(i,a)}_{t}log(\hat{y}^{(i,a)}_{t})$
$\displaystyle+(1-y^{(i,a)}_{t})log(1-(\hat{y}^{(i,a)}_{t}))$ (7)

###  2.5 Slot Tagger

In addition to detecting dialog acts, we further detect the slots for each
utterance in the dialog. Here we take the hidden representation
$H_{tok}=\\{h_{t}^{1},h_{t}^{2},...,h_{t}^{N}\\}$ from $BERT_{u}$ again. Then
we concatenate $H_{tok}$ with the dialog act context information $H_{act}$ to
be slot hidden states $E_{slot}=H_{tok}\oplus H_{act}$. Then we use another
BiLSTM as the slot-filling tagger and generate the logits for each token.

$\displaystyle H_{slot}=BiLSTM(E_{slot})$ (8)
$\displaystyle\hat{y}_{s}=softmax(H_{slot}W_{slot})$ (9)

Finally we can define the cross entropy loss as the objective:

$\displaystyle\mathcal{L}_{s}\coloneqq-$
$\displaystyle\sum_{i=1}^{N_{s}}\sum_{t=1}^{T}\sum_{j=1}^{N}\sum_{s=1}^{|\mathcal{Y}^{s}|}(y^{(i,j,s)}_{t}log(\hat{y}^{(i,j,s)}_{t}))$
(10)

The final joint objective will be formulated as
$\mathcal{L}_{\theta}=\mathcal{L}_{a}+\mathcal{L}_{s}$.

##  3 Experiments

###  3.1 Datasets

Table 3: Ablation study on different components of CaBERT-SLU. We report accuracy (ID Acc) for all intents exact match and F1 scores (ID F1) based on the individual intent calculation; SL F1 for slot-filling F1 scores. We also report intent accuracy (IO Acc) and F1 score (IO F1) with models trained solely on intent detection without slot filling. SA: self-attentive layer, CF: context fusion layer. Dataset | MDC | SGD | MDC | SGD
---|---|---|---|---
Model | ID F1 | ID Acc | SL F1 | ID F1 | ID Acc | SL F1 | IO F1 | IO Acc | IO F1 | IO Acc
BERT | 90.67 | 81.84 | 78.21 | 95.23 | 92.75 | 89.65 | 89.71 | 81.43 | 94.96 | 92.63
+LSTM | 90.83 | 82.10 | 78.79 | 98.61 | 97.78 | 89.65 | 90.55 | 82.13 | 98.45 | 97.55
+SA+LSTM | 90.84 | 82.38 | 78.61 | 98.61 | 97.79 | 90.19 | 90.53 | 82.02 | 98.48 | 97.65
+SA+CF | 90.89 | 82.59 | 79.81 | 98.94 | 98.19 | 95.53 | 90.82 | 81.83 | 98.82 | 98.01
CaBERT-SLU | 91.26 | 83.05 | 79.64 | 99.14 | 98.59 | 95.71 | 90.81 | 82.69 | 98.92 | 98.24
Table 4: CaBERT-SLU performance in different domains. Dataset | Domain | ID F1 | ID Acc | SL F1
---|---|---|---|---
MDC | restaurant | 86.41 | 75.28 | 74.68
taxi | 94.30 | 88.47 | 80.97
movie | 93.02 | 85.74 | 82.35
SGD | restaurant | 98.92 | 98.30 | 94.15
single | 99.14 | 98.31 | 91.27
multiple | 99.10 | 98.60 | 95.92

We evaluate our model on two multi-turn dialog datasets: Microsoft Dialogue
Challenge dataset (MDC) [4] and Schema-Guided Dialogue dataset (SGD) [15]. MDC
is a human-annotated conversation dataset in three domains (movie, restaurant,
taxi). Each of them contains 2890, 4103, 3094 dialogs with total 11 acts and
50 slots. For each utterance, it is attached one or more dialog acts and
slots. SGD consists of over 20k annotated multi-domain, task-oriented
conversations between a human and a virtual assistant. These conversations
span 20 domains, ranging from banks and events to media, travel and weather.
We mainly adopt the user and system acts in each utterance for the dialog act
detection and corresponding slots for slot filling. It has total 18 acts and
89 slots. We mainly split the training/validation/testing data with the ratio
0.7/0.1/0.2.

###  3.2 Experimental Setup

We compare our results with several competitive baselines: Stack-Prop [12]
which uses two stacked encode-decoder structures for joint single intent and
slot filling tasks. Joint MID-SF [14] which first considers multi-intent
detection task in use of BiLSTMs, AGIF [13] which uses graph interactive
framework to consider fine-grained information, ECA [24] which encodes context
with LSTM encoder for joint task prediction. We also fine-tune BERT pretrained
layers with several proposed work including a self-attentive layer (SA) from
[26] and CASA-NLU which encodes context with DiSAN sentence2token [22].

For experimental setting, we exploit the pretrained BERT model with 12 layers
of 768 hidden units and 12 self-attention heads. For the self-attentive layer,
we use 4 heads of context vectors. For the context fusion encoder, we set 6
transformer layers and the max sequence length as 60. Both two LSTMs have 256
hidden units. We use the batch size of 4 dialogs for MDC and 2 for SGD. In all
training, we use Adam optimizer with learning rate as 2e-5. The model is
trained for 20 epochs with the best performance on validation set. For
metrics, by following [13], we evaluate the performance of dialog act
detection with accuracy and macro F1 score. We use F1 score for slot filling.
Here we only consider a true positive when all BIO values for a slot is
correct and forfeit ‘O‘ tags.

##  4 Results

###  4.1 Main Results

Figure 2: Visualization of attention weights on the last layer of context
fusion encoder.

Table 2 shows the performance of CaBERT-SLU on joint tasks in two dialogue
datasets, compared with several baseline models. Our model beats all baselines
whether they are based on single utterances or BERT-related techniques, and
achieves 1.1% and 3.5% increase in intent accuracy of two datasets than
BERT+CASA-NLU [22]. We believe that the strong performance yielded by CaBERT-
SLU pertains to the robust contextual information sharing both mutually and
sequentially in multiple layers of masked self-attention. Without learning
different weights of the dialog history to the current turn, previous
approaches’ performances are significantly undermined. Also, it achieves 1.9%
and 5.8% increase in slot F1 score, benefited from our model’s contextual
sharing. We can also observe a slight increase overall by considering the
joint task. We estimate the effectiveness of each module of CaBERT-SLU by
conducting ablation experiments as shown in Table 3. We observe a slight drop
of 0.60% without using self-attentive layer. And by incorporating both context
fusion layer and global recurrence layer, it can boost the performance by
overall roughly 1.5% and almost 7% in SGD slot filling task.

To explore performances within different domains of our model, we separate MDC
based on three domains. As for SGD, since a single dialog may involve multiple
domains, we instead sample SGD with three variations: (1) dialogs associated
with the ‘restaurant‘ domain (2) dialogs with only ‘single‘ domain (3) dialogs
with ‘multiple‘ domains. In Table 4, we can observe that taxi and movie
domains are much easier than restaurant domain which contribute more in joint
scores. In SGD, we can see our method performs well regardless of how data is
subsampled; especially outperforming on multi-domain dialogs. Our model also
performs well in multiple domain for slot filling where context fusion may
benefit domain transition of slots.

###  4.2 Attention Visualization

To further understand the mechanism of our context fusion encoder, we
visualize the attention weights over the mean of heads at the last layer, as
shown in Figure 2. In the example dialog, user first asks system to find a kid
friendly place to eat. And the system asks about time, date and number of
people at id 1 and 3. Then we can observe id 2 and 4 have more weights on
their close previous neighbors, which indicate the sequential relation between
request and inform. We can see id 3 of asking number of people may be related
to kid friendly keyword at id 0. After system talks about options of
restaurants at id 5, id 6 replies with more dependency based on it. To note,
with masked self-attention, we only attend weights on previous contexts.

##  5 Conclusion

In this work, we introduce an effective model that composes a contextual
hierarchical structure affiliated with BERT to reinforce the connection
between dialog contexts, which is often ignored by recent SLU works. By
exploiting such naturalness of dialog flow, it is capable of capturing
necessary mentions in previous dialog history for current tasks. Experimental
results show that our model achieves strong improvements over models without
contextual awareness. We also achieve SOTA results in joint multi-intent
detection and slot filling of two multi-turn dialog datasets, without
sacrificing the mutual relations between SLU and DM, and further error
propagation.

## References

  * [1] Z. Zhang, R. Takanobu, Q. Zhu, M. Huang, and X. Zhu, “Recent advances and challenges in task-oriented dialog system,” 2020.
  * [2] H. Weld, X. Huang, S. Long, J. Poon, and S. C. Han, “A survey of joint intent detection and slot-filling models in natural language understanding,” 2021.
  * [3] L. Abbeduto, “Linguistic communication and speech acts. kent bach robert m. harnish. cambridge: M.i.t. press, 1979, pp. xvii 327.” _Applied Psycholinguistics_ , vol. 4, no. 4, p. 397–407, 1983.
  * [4] X. Li, S. Panda, J. Liu, and J. Gao, “Microsoft dialogue challenge: Building end-to-end task-completion dialogue systems,” _arXiv preprint arXiv:1807.11125_ , 2018.
  * [5] T. Liu, X. DING, Y. QIAN, and Y. CHEN, “Identification method of user’s travel consumption intention in chatting robot,” _SCIENTIA SINICA Informationis_ , vol. 47, p. 997, 08 2017.
  * [6] C. Xia, C. Zhang, X. Yan, Y. Chang, and P. Yu, “Zero-shot user intent detection via capsule neural networks,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 3090–3099. [Online]. Available: https://www.aclweb.org/anthology/D18-1348
  * [7] C.-W. Goo, G. Gao, Y.-K. Hsu, C.-L. Huo, T.-C. Chen, K.-W. Hsu, and Y.-N. Chen, “Slot-gated modeling for joint slot filling and intent prediction,” in _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_. New Orleans, Louisiana: Association for Computational Linguistics, Jun. 2018, pp. 753–757. [Online]. Available: https://www.aclweb.org/anthology/N18-2118
  * [8] C. Li, L. Li, and J. Qi, “A self-attentive model with gate mechanism for spoken language understanding,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 3824–3833. [Online]. Available: https://www.aclweb.org/anthology/D18-1417
  * [9] H. E, P. Niu, Z. Chen, and M. Song, “A novel bi-directional interrelated model for joint intent detection and slot filling,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 5467–5471. [Online]. Available: https://www.aclweb.org/anthology/P19-1544
  * [10] Y. Liu, F. Meng, J. Zhang, J. Zhou, Y. Chen, and J. Xu, “Cm-net: A novel collaborative memory network for spoken language understanding,” 2019.
  * [11] H. Purohit, G. Dong, V. Shalin, K. Thirunarayan, and A. Sheth, “Intent classification of short-text on social media,” in _2015 IEEE International Conference on Smart City/SocialCom/SustainCom (SmartCity)_ , 2015, pp. 222–228.
  * [12] L. Qin, W. Che, Y. Li, H. Wen, and T. Liu, “A stack-propagation framework with token-level intent detection for spoken language understanding,” 2019.
  * [13] L. Qin, X. Xu, W. Che, and T. Liu, “Agif: An adaptive graph-interactive framework for joint multiple intent detection and slot filling,” 2020.
  * [14] R. Gangadharaiah and Balakrishnan, _Joint multiple intent detection and slot labeling for goal-oriented dialog_. Proc. of NAACL, 2019.
  * [15] A. Rastogi, X. Zang, S. Sunkara, R. Gupta, and P. Khaitan, “Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset,” _arXiv preprint arXiv:1909.05855_ , 2019.
  * [16] C. Xia, C. Xiong, P. Yu, and R. Socher, “Composed variational natural language generation for few-shot intents,” 2020.
  * [17] Y. Shan, Z. Li, J. Zhang, F. Meng, Y. Feng, C. Niu, and J. Zhou, “A contextual hierarchical attention network with adaptive objective for dialogue state tracking,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, Jul. 2020, pp. 6322–6333. [Online]. Available: https://www.aclweb.org/anthology/2020.acl-main.563
  * [18] T.-H. Wen, D. Vandyke, N. Mrkšić, M. Gašić, L. M. Rojas-Barahona, P.-H. Su, S. Ultes, and S. Young, “A network-based end-to-end trainable task-oriented dialogue system,” in _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_. Valencia, Spain: Association for Computational Linguistics, Apr. 2017, pp. 438–449. [Online]. Available: https://www.aclweb.org/anthology/E17-1042
  * [19] A. Bordes, Y.-L. Boureau, and J. Weston, “Learning end-to-end goal-oriented dialog,” 2017.
  * [20] N. Bertomeu, H. Uszkoreit, A. Frank, H.-U. Krieger, and B. Jörg, “Contextual phenomena and thematic relations in database QA dialogues: results from a Wizard-of-Oz experiment,” in _Proceedings of the Interactive Question Answering Workshop at HLT-NAACL 2006_. New York, NY, USA: Association for Computational Linguistics, Jun. 2006, pp. 1–8. [Online]. Available: https://www.aclweb.org/anthology/W06-3001
  * [21] Y. Shi, K. Yao, H. Chen, Y.-C. Pan, M.-Y. Hwang, and B. Peng, “Contextual spoken language understanding using recurrent neural networks,” April 2015.
  * [22] A. Gupta, P. Zhang, G. Lalwani, and M. Diab, “Casa-nlu: Context-aware self-attentive natural language understanding for task-oriented chatbots,” 2019\.
  * [23] V. Zhong, C. Xiong, and R. Socher, “Global-locally self-attentive dialogue state tracker,” 2018.
  * [24] S. A. A. J. S. S. Chauhan A., Malhotra A., “Encoding context in task-oriented dialogue systems using intent, dialogue acts, and slots.” in _Saini H., Sayal R., Buyya R., Aliseri G. (eds) Innovations in Computer Science and Engineering. Lecture Notes in Networks and Systems, vol 103. Springer, Singapore._
  * [25] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019.
  * [26] Z. Lin, M. Feng, C. N. dos Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio, “A structured self-attentive sentence embedding,” 2017.


# Evaluating Embeddings for One-shot Classification of Doctor-AI Consultations

Olumide E. Ojo
Instituto Politécnico Nacional (IPN),
Centro de Investigación en Computación (CIC),
Mexico City, Mexico.
<EMAIL_ADDRESS>
Olaronke O. Adebanji
Instituto Politécnico Nacional (IPN),
Centro de Investigación en Computación (CIC),
Mexico City, Mexico.
<EMAIL_ADDRESS>
Alexander Gelbukh
Instituto Politécnico Nacional (IPN),
Centro de Investigación en Computación (CIC),
Mexico City, Mexico.
<EMAIL_ADDRESS>
Hiram Calvo
Instituto Politécnico Nacional (IPN),
Centro de Investigación en Computación (CIC),
Mexico City, Mexico.
<EMAIL_ADDRESS>
Anna Feldman
Montclair State University,
Montclair, USA.
<EMAIL_ADDRESS>

###### Abstract

Effective communication between healthcare providers and patients is crucial
to providing high-quality patient care. In this work, we investigate how
Doctor-written and AI-generated texts in healthcare consultations can be
classified using state-of-the-art embeddings and one-shot classification
systems. By analyzing embeddings such as bag-of-words, character n-grams,
Word2Vec, GloVe, fastText, and GPT2 embeddings, we examine how well our one-
shot classification systems capture semantic information within medical
consultations. Results show that the embeddings are capable of capturing
semantic features from text in a reliable and adaptable manner. Overall,
Word2Vec, GloVe and Character n-grams embeddings performed well, indicating
their suitability for modeling targeted to this task. GPT2 embedding also
shows notable performance, indicating its suitability for models tailored to
this task as well. Our machine learning architectures significantly improved
the quality of health conversations when training data are scarce, improving
communication between patients and healthcare providers.

Keywords: AI in healthcare, embeddings, one-shot classification, few-shot
classification

##  1 Introduction

NLP has seen remarkable advances in recent years due to the development of
large language models and the focus on improving text representations. An area
of interest within this field is text classification [1, 2, 3, 4, 5, 6, 7], a
task that has practical applications in several domains, including politics
[8], health [9, 10, 11], economics [12, 13], etc. Without training data,
developing representations that can capture the features of text poses
different challenges. Sectors like healthcare, where information exchange is
crucial, can benefit from the development and implementation of these
representations, often called embeddings. In this study, embedding techniques
within one-shot classification systems are evaluated for their effectiveness
in recognising AI-generated text.

Doctor-AI consultations between patients and healthcare providers have emerged
as a viable route, using AI to help Doctors diagnose, prescribe treatments,
and provide medical advice [14, 15, 16, 17, 18, 19, 20, 21]. Text data
generated from these interactions must be classified for a variety of
applications, including authorship identification, medical record
administration, and evaluating the legitimacy of online medical advice. The
impact of accurate text classification in healthcare extends well beyond the
immediate consultation process. Users can make informed decisions about the
medical advice they receive online with the help of a correct classification,
which allows them to identify the source of information.

There has been a great deal of interest in the one-shot classification task
[22, 23, 24, 25, 26], where a single data instance is presented to a model for
a rapid decision, usually involving a yes or no response. Specifically, the
one-shot classification of Doctor-AI consultations poses unique challenges and
opportunities, because it requires the development of robust text
representations capable of distinguishing between the linguistic patterns and
characteristics of Doctor-written texts and those generated by AI models.
Medical records, as well as AI-assisted medical advice, must be distinguished
to ensure credibility and reliability. Embeddings, or dense vector
representations of text, are a key component for classification models to be
successful. We use embedding techniques to determine the accuracy and potency
of our one-shot classification systems for Doctor-AI consultations.

We explore the importance of various embedding methods and their performance
in the one-shot classification of Doctor-AI consultations, thus improving
healthcare consultations and online advice. As part of this study, a detailed
analysis of model performance is carried out using various embeddings, and the
effectiveness and limitations of these embeddings are discussed. We employ the
accuracy, precision, recall and F1 scores to evaluate these models adequately.
Study findings provided us with a useful reference for future analyses and
discussions, improving the quality and safety of clinical discourse through
improved accuracy and reliability of health information. Exploring this
multifaceted problem not only advances the state-of-the-art, but also
contributes to the broader discussion about how human expertise meets AI
capabilities.

The contributions of this paper are summarized as follows:

  * •

We advanced our understanding of how embeddings affect the classification of
Doctor-AI consultations, with implications for the quality and reliability of
online medical advice.

  * •

Multiple one-shot classification models were developed, eliminating the need
for extensive training data.

  * •

In our analysis, we classified Doctor-AI consultations as binary and
multiclass categories.

In the following sections, we discuss the following topics. Section 2 is a
comprehensive review of the related literature, setting the foundation for our
study. In Section 3, we delve into the details of the experimental setup and
presents the findings derived from the analysis of the datasets. In Section 4,
we discuss the implications of our findings. Finally, in Section 5, we give a
summary of our contributions, acknowledge the limitations encountered during
the research, and provide directions for future research.

##  2 Literature Review

Human or AI-authored texts are difficult to identify online and require a
collaborative approach that involves professionals from various fields.
Research in text classification of this type has focused predominantly on the
use of NLP techniques, with machine learning playing a vital role in capturing
the inherent features of the text. We highlight the gaps in the existing
literature that this current research seeks to address by conducting a
comprehensive evaluation of embedding techniques for the specific task of one-
shot classification in healthcare consultations.

Within the field of radiology, it is evident that multilingual and diverse
large language models (LLMs) are poised to play a pivotal role in shaping the
future of global healthcare delivery. According to [27], there is potential
for enhanced healthcare services using AI on a global scale. Their study
provided a solid foundation for further exploratory research. There is great
potential to expand the applications of LLM in various medical specialties and
advance the development of multimodal LLM. The latter has the ability to
handle complex and varied data types, enabling a more holistic understanding
of patient health. However, as we navigate this evolving landscape of LLMs, it
is of paramount importance to conscientiously consider their effective
implementation and ethical deployment.

According to the work of [23], the posture of sleep was associated with a
variety of health conditions, including nocturnal cramps and more serious
musculoskeletal problems. Traditionally, in-clinical sleep assessments have
focused primarily on monitoring vital signs, such as brain waves. Although
wearable sensors equipped with inertial measurement units have been used for
the classification of sleep posture, previous studies have been limited in
scope, typically addressing only a handful of common postures, which are
insufficient for comprehensive clinical evaluations. In response to these
limitations, the authors introduced a novel framework for the classification
of sleep postures, focusing on a minimal set of joint angle measurements. This
framework is validated through two distinct experimental pipelines: the first
involves computer animation to generate synthetic postural data, and the
second involves a pilot study of human participants that uses custom-made
miniature wearable sensors. Medical experts could understand the
characterization of body posture made possible by the one-shot learning
framework. The proposed framework achieved impressive overall accuracy,
reaching levels as high as 100% on synthetic data and 92.7% on real data,
placing it on par with the state-of-the-art data-intensive algorithms
available in the existing literature.

The authors in [28] recognized the significant potential of few-shot learning
(FSL) methods, particularly in light of the limited availability of annotated
text for various medical topics. Their study delved into the existing
landscape of FSL methods within the realm of medical NLP spanning articles
published between January 2016 and August 2021, drawing from reputable sources
like PubMed/Medline, Embase, ACL Anthology, IEEE Xplore Digital Library, and
pre-print repositories such as medRxiv through Google Scholar. Inclusion
criteria were established to identify articles that encompass FSL and medical
text, and the review systematically summarized these articles based on their
data sources, objectives, training sets sizes, primary methodologies, and
evaluation techniques. The results of this review revealed 31 studies that met
the defined inclusion criteria, all of which were published after 2018, with a
notable increase in publications since 2020. Among the predominant tasks
addressed in these studies were concept extraction/named entity recognition
and text classification. In particular, a substantial portion of the studies
(68%) resorted to the creation of synthetic few-shot scenarios, frequently
employing the MIMIC-III dataset. The commonalities among the FSL methods used
included integration with attention mechanisms, prototypical networks, and
meta-learning. The authors emphasized the potential benefits of specialized
biomedical FSL datasets to facilitate method development and comparative
evaluations.

In [22], the article introduced a novel approach for the automatic
classification of medical images. This method incorporated a convolutional
neural network backbone and a causality extractor module that captures the
cause-effect relationships between the image features. The study assessed the
approach in low-data scenarios, employing one-shot learning and meta-learning
procedures. Binary and multiclass classification experiments are conducted on
a prostate MRI dataset, and the causality-driven module’s effectiveness is
validated through an ablation study and qualitative assessments. The findings
highlighted the significant role of causal relationships in improving the
model’s ability to make reliable and interpretable predictions, making it a
promising technique for medical image classification tasks. The study focused
on improving classification performance, particularly in scenarios with
limited training data, such as those encountered in medical image analysis.
The research explored causation within the context of image data,
demonstrating performance improvements over baseline methods, with notable
enhancements in multiclass and specific AUROC metrics.

[29] discussed the result of a study carried out by the DS4DH group on the
MEDIQA-Chat tasks at ACL-ClinicalNLP 2023. Their research combined classical
machine learning, specifically the Support Vector Machine (SVM), to classify
medical dialogues, with the incorporation of one-shot prompts using GPT-3.5.
The approach involved using dialogues and summaries from the same category as
prompts to generate summaries for new dialogues. The results exceeded the
average benchmark score, which established a robust reference for the
evaluation of performance in the field. The study highlighted the
effectiveness of combining traditional machine learning techniques such as SVM
with advanced language models such as GPT-3.5 to summarize medical dialogue.
This hybrid methodology holds promise for improving patient care documentation
and supporting healthcare professionals in informed decision making.

In scenarios where classification errors have significant consequences, such
as early detection of threat events with limited data, ensuring the
reliability of classification results becomes paramount. [30] delved into the
realm of Uncertainty Estimation for Few-Shot Text Classification (UEFTC),
which was relatively unexplored. In UEFTC, their goal was to develop models
that can predict the uncertainty associated with a classification outcome,
indicating the likelihood that it is incorrect, particularly in cases with
limited samples. Traditional uncertainty estimation models in text
classification often fall short when applied to UEFTC. These models typically
require many training samples, whereas the few-shot setting of UEFTC provides
only a few or even just one support sample for each class in a given episode.
To address this challenge, the paper introduced Contrastive Learning from
Uncertainty Relations (CLUR) as a solution. CLUR was designed to be trained
with only one support sample for each class, aided by pseudo-uncertainty
scores. In particular, CLUR autonomously learned these scores using a novel
approach to uncertainty relations.

The work of [31] introduced MaskBERT, a framework designed to enhance the few-
shot learning capabilities of BERT-based models. Unlike the existing few-shot
learning strategies, MaskBERT focused on selectively applying masks to text
input, filtering out irrelevant information, and guiding the model to
concentrate on discriminative tokens. In addition, a contrastive learning loss
function was employed to improve the separability of different categories and
the compactness of the same category representations. Experimental results in
benchmark datasets demonstrate the effectiveness of MaskBERT, showing superior
performance compared to other NLP methods. Ablation studies further validate
the effectiveness of its components.

Embeddings improved the detection of suicide risk and depression on social
media platforms in a study by [32]. Identifying the risk of suicide and
depressive content shared online has significant social implications,
particularly for the younger generation. The authors introduced a novel
framework that leverages fastText embedding for contextual analysis, TF-IDF
vectors to gauge term relevance, and the XGBoost machine learning classifier
for precise classification of depression and content related to suicidal risk.
Their innovative approach yields promising results, achieving an AUC of 0.78
and a weighted F1 score of 0.71 on a Reddit dataset. The study demonstrates
robust performance compared to various embedding models and classifiers,
underscoring the effectiveness of the embedding approach in addressing the
classification problem.

Our exploration of embeddings in one-shot classification for Doctor-AI
consultations is contextualized by identifying the significance of one-shot
classification tasks, the relevance of embeddings, and the evolving field of
AI consultations. Our contribution lies in the empirical evaluation of various
embeddings, with a particular focus on one-shot classification learning. It is
our goal to improve both the quality and efficiency of Doctor-AI interactions
to benefit both patients and healthcare providers.

##  3 Experiments

###  3.1 Approach

We adopted a diverse set of embedding techniques, including bag of words (BoW)
in conjunction with Term Frequency-Inverse Document Frequency (TF-IDF),
character n-grams, Word2Vec, GloVe, fastText, and GPT2 embeddings. BoW
provides a quantitative representation of word occurrences, while character
n-grams capture sequential patterns. GloVe, fastText, and GPT2 embeddings
offer semantic representations that preserve contextual relationships among
words. These embedding techniques transform words into continuous vector
spaces. GloVe embeddings leverage global statistical information to represent
words based on their semantic similarity, fastText considers subword
information for robust handling of diverse vocabulary, and GPT2 embeddings
employ a transformer architecture to provide contextualized representations.
By incorporating these advanced embedding techniques into our study on One-
shot Classification models for Doctor-AI Consultations, we aim to enhance the
models’ ability to understand and classify medical conversations by leveraging
the rich semantic and contextual information encoded in the embeddings. As a
result of their ability to handle diverse data structures and complexity, the
Logistic Regression Classifier (LRC), Random Forest Classifier (RFC), Support
Vector Machine (SVM), Naive Bayes Classifier (NBC), Decision Tree Classifier
(DTC), Gradient Boosting Classifier (GBC), and Multilayer Perceptron (MLP)
were selected. Each model was trained on the feature-extracted representations
of doctor-AI consultations and evaluated using standard metrics, including
Accuracy, Precision, Recall, and F1 scores. The selection of multiple models
allows us to assess their relative strengths and weaknesses, providing a
comprehensive understanding of their performance across different feature sets
and embeddings.

###  3.2 Dataset

The traditional machine learning algorithms were applied to the MEDIC dataset
[10] and evaluated across different embeddings: Bag of Words (BoW), character
n grams, Word2Vec, GloVe, fastText and GPT2. The dataset includes 1880
distinct texts for the Doctor’s response, ChatGPT’s response, and the
rephrased Doctor’s response, resulting in a total of 5640 text inputs. The
dataset was split into three separate files. The first file contained
responses from both the Doctor and the ChatGPT model (referred to as DC). The
second file included responses from the Doctor and rephrased versions of the
Doctor’s responses (referred to as DR). The third file combined responses from
Doctor, ChatGPT, and rephrased Doctor’s responses (referred to as DCR). This
file was used for the multi-label classification task. We chose a single
instance of each unique label present in the dataset for model training, and
applied these trained models to test on the remaining data. The embedding
performance evaluation was carried out on the various proposed models using
metrics such as accuracy, precision, recall, and F1 scores.

###  3.3 Experiment Results

Table 1 presents the results of the one-shot classification models applied to
the DC dataset using different embeddings. The traditional machine learning
models (and its associated embedding features) evaluated in the table include
the logistic regression classifier (LRC), random forest classifier (RFC),
support vector machine (SVM), Gaussian naive Bayes classifier (NBC), decision
tree classifier (DTC), gradient boosting classifier (GBC), and the multilayer
perceptron network (MLP). For each model, the table shows the accuracy,
precision, recall, and F1 score values, which serve as metrics to evaluate the
model’s performance. The Bag of Words (BoW), character n-grams, Word2Vec,
GloVe, fastText, and GPT2 embeddings were used to transform text data into
numerical vectors, enabling machine learning algorithms to process and
classify them effectively.

In the DC dataset, MLP and Word2Vec embeddings demonstrated an accuracy of
96.44%, along with impressive recall, precision, and F1 scores. The
combination demonstrated that Word2Vec embeddings can effectively capture
semantic representations. NBC and SVM with character n-gram embeddings also
showed excellent performances, achieving an accuracy of 88. 30% and 88. 86%
respectively, facilitating robust performance in the classification task.
Character n-grams embeddings with SVM and NBC demonstrated strong performances
with accuracies of 88.30% and 88.86%, respectively, showing the importance of
taking into account the synergy between models and embeddings, as well as the
interaction between semantic representations and classification algorithms.
LRC and GPT2 embeddings are less effective, giving 57.15% accuracy.
Considering models and embeddings as a whole, the overall performance trends
highlight the importance of systematic evaluation and selection, as different
combinations exhibit substantial differences in performance.

Table 1: Performance measures of the models in the DC dataset using various embeddings Model | Features | Accuracy | Precision | Recall | F1
---|---|---|---|---|---
LRC | BoW + TF-IDF | 0.7577 | 0.7596 | 0.7577 | 0.7573
| Character n-grams | 0.8830 | 0.8830 | 0.8830 | 0.8830
| Word2Vec | 0.8912 | 0.9104 | 0.8913 | 0.8899
| GloVe | 0.8928 | 0.8930 | 0.8928 | 0.8928
| fastText | 0.7316 | 0.8243 | 0.7318 | 0.7110
| GPT2 Embedding | 0.5715 | 0.5719 | 0.5716 | 0.5710
RFC | BoW + TF-IDF | 0.5253 | 0.7325 | 0.5250 | 0.3887
| Character n-grams | 0.4907 | 0.4790 | 0.4905 | 0.4099
| Word2Vec | 0.8524 | 0.8852 | 0.8525 | 0.8492
| GloVe | 0.7545 | 0.7603 | 0.7546 | 0.7532
| fastText | 0.7043 | 0.8129 | 0.7044 | 0.6762
| GPT2 Embedding | 0.6753 | 0.7758 | 0.6751 | 0.6429
SVM | BoW + TF-IDF | 0.7577 | 0.7595 | 0.7577 | 0.7573
| Character n-grams | 0.8830 | 0.8830 | 0.8830 | 0.8830
| Word2Vec | 0.8886 | 0.9086 | 0.8886 | 0.8872
| GloVe | 0.8928 | 0.8930 | 0.8928 | 0.8928
| fastText | 0.7614 | 0.8371 | 0.7616 | 0.7473
| GPT2 Embedding | 0.5715 | 0.5719 | 0.5716 | 0.5710
NBC | BoW + TF-IDF | 0.7577 | 0.7596 | 0.7577 | 0.7573
| Character n-grams | 0.8830 | 0.8830 | 0.8830 | 0.8830
| Word2Vec | 0.8926 | 0.9113 | 0.8926 | 0.8913
| GloVe | 0.8928 | 0.8930 | 0.8928 | 0.8928
| fastText | 0.7239 | 0.8300 | 0.7241 | 0.7014
| GPT2 Embedding | 0.5715 | 0.5719 | 0.5716 | 0.5710
DTC | BoW + TF-IDF | 0.5021 | 0.5767 | 0.5019 | 0.3413
| Character n-grams | 0.9122 | 0.9253 | 0.9122 | 0.9116
| Word2Vec | 0.6915 | 0.7266 | 0.6916 | 0.6791
| GloVe | 0.7569 | 0.7571 | 0.7569 | 0.7569
| fastText | 0.8864 | 0.8878 | 0.8864 | 0.8863
| GPT2 Embedding | 0.6059 | 0.6314 | 0.6057 | 0.5856
GBC | BoW + TF-IDF | 0.5005 | 0.7502 | 0.5003 | 0.3340
| Character n-grams | 0.5074 | 0.7197 | 0.5072 | 0.3501
| Word2Vec | 0.9064 | 0.9206 | 0.9064 | 0.9056
| GloVe | 0.7918 | 0.7931 | 0.7917 | 0.7915
| fastText | 0.5880 | 0.7697 | 0.5883 | 0.5049
| GPT2 Embedding | 0.5093 | 0.5284 | 0.5091 | 0.4087
MLP | BoW + TF-IDF | 0.9130 | 0.9135 | 0.9130 | 0.9130
| Character n-grams | 0.9130 | 0.9135 | 0.9130 | 0.9130
| Word2Vec | 0.9644 | 0.9662 | 0.9644 | 0.9643
| GloVe | 0.8535 | 0.8565 | 0.8535 | 0.8532
| fastText | 0.8644 | 0.8925 | 0.8644 | 0.8619
| GPT2 Embedding | 0.6016 | 0.7365 | 0.6014 | 0.5351

Table 2 offers the performance metrics for the one-shot classification model
applied to the DR dataset using different models and features. The accuracy
scores range from approximately 0.45 to around 0.83, illustrating a wide range
of classification correctness.

Using GPT2 Embedding, the MLP model showed remarkable accuracy, precision,
recall, and F1 score values of 0.8356, 0.8690, 0.8356, and 0.8318. It
demonstrates the effectiveness of leveraging contextualized embeddings
provided by GPT2 for capturing intricate patterns in the DR dataset, creating
a better prediction. DTC is the least-performing model when combined with BoW.
Multiple models in this experiment illustrate the robustness and suitability
of the GPT2 embedding for the DR dataset. FastText and GloVe display mixed
results, with different results for different models. Moreover, the results
indicate that the choice of embedding and model architecture is critical, as
certain models, like MLP, exhibit substantial improvements over others,
demonstrating the need to select embeddings and models carefully for optimal
performance in datasets like this. The diversity in the scores emphasizes that
no single model-embedding combination stands out as universally superior,
particularly with the AI model’s rephrasing of the Doctor’s text.

Table 2: Performance measures of the models in the DR dataset using various embeddings Model | Features | Accuracy | Precision | Recall | F1
---|---|---|---|---|---
LRC | BoW + TF-IDF | 0.5271 | 0.5271 | 0.5271 | 0.5271
| Character n-grams | 0.6128 | 0.6579 | 0.6130 | 0.5833
| Word2Vec | 0.5950 | 0.6916 | 0.5952 | 0.5368
| GloVe | 0.4745 | 0.4675 | 0.4743 | 0.4452
| fastText | 0.4654 | 0.4622 | 0.4653 | 0.4540
| GPT2 Embedding | 0.5327 | 0.5816 | 0.5329 | 0.4510
RFC | BoW + TF-IDF | 0.5564 | 0.5693 | 0.5563 | 0.5345
| Character n-grams | 0.5649 | 0.5869 | 0.5648 | 0.5353
| Word2Vec | 0.5415 | 0.5475 | 0.5416 | 0.5270
| GloVe | 0.5104 | 0.5104 | 0.5104 | 0.5101
| fastText | 0.5186 | 0.5199 | 0.5187 | 0.5112
| GPT2 Embedding | 0.6985 | 0.7047 | 0.6986 | 0.6962
SVM | BoW + TF-IDF | 0.5357 | 0.5357 | 0.5356 | 0.5355
| Character n-grams | 0.6128 | 0.6579 | 0.6130 | 0.5833
| Word2Vec | 0.4840 | 0.4753 | 0.4839 | 0.4348
| GloVe | 0.4745 | 0.4674 | 0.4743 | 0.4451
| fastText | 0.4665 | 0.4633 | 0.4664 | 0.4550
| GPT2 Embedding | 0.4947 | 0.4932 | 0.4948 | 0.4622
NBC | BoW + TF-IDF | 0.5271 | 0.5271 | 0.5271 | 0.5271
| Character n-grams | 0.6128 | 0.6579 | 0.6130 | 0.5833
| Word2Vec | 0.4891 | 0.4840 | 0.4889 | 0.4465
| GloVe | 0.4745 | 0.4674 | 0.4743 | 0.4451
| fastText | 0.4702 | 0.4631 | 0.4701 | 0.4438
| GPT2 Embedding | 0.4947 | 0.4932 | 0.4948 | 0.4622
DTC | BoW + TF-IDF | 0.5005 | 0.7502 | 0.5003 | 0.3340
| Character n-grams | 0.5027 | 0.7506 | 0.5029 | 0.3397
| Word2Vec | 0.4758 | 0.4741 | 0.4757 | 0.4672
| GloVe | 0.6035 | 0.6063 | 0.6036 | 0.6009
| fastText | 0.5538 | 0.5553 | 0.5538 | 0.5506
| GPT2 Embedding | 0.7326 | 0.7536 | 0.7325 | 0.7269
GBC | BoW + TF-IDF | 0.5005 | 0.7502 | 0.5003 | 0.3340
| Character n-grams | 0.5428 | 0.5967 | 0.5426 | 0.4684
| Word2Vec | 0.4854 | 0.4804 | 0.4852 | 0.4516
| GloVe | 0.5072 | 0.5074 | 0.5071 | 0.5025
| fastText | 0.4729 | 0.4671 | 0.4727 | 0.4492
| GPT2 Embedding | 0.5546 | 0.6427 | 0.5548 | 0.4736
MLP | BoW + TF-IDF | 0.5442 | 0.5505 | 0.5443 | 0.5298
| Character n-grams | 0.5820 | 0.6319 | 0.5821 | 0.5385
| Word2Vec | 0.4537 | 0.4325 | 0.4536 | 0.4073
| GloVe | 0.4614 | 0.4546 | 0.4613 | 0.4408
| fastText | 0.4478 | 0.4417 | 0.4478 | 0.4331
| GPT2 Embedding | 0.8356 | 0.8690 | 0.8356 | 0.8318

Table 3 presents a detailed overview of the performance measures of the one-
shot classification models applied to the DCR dataset using various
embeddings. Accuracy scores range from approximately 0.28 to 0.66, indicating
substantial variation in overall classification correctness between different
combinations of model embeddings. Among the best-performing combinations, the
MLP model achieved an accuracy of 0.6642 with BoW. DTC embeddings with
fastText gave the worst results with an accuracy of 0.2872 and low precision,
recall, and F1 scores. This highlights the model’s sensitivity to both
embeddings and models, therefore, it is essential to carefully select the
model for optimal outcomes. For the specific characteristics of the dataset at
hand, it is important to conduct thorough experimentation and fine-tuning to
identify the best embeddings and models.

This result reflects the substantial variations in performance, highlighting
that model-embedding combinations at the lower end of the range are less
effective on this specific dataset, whereas those at the higher end excel in
classification. The detailed scores demonstrate that no model embedding
combination is universally superior, and the choice depends on the type of
dataset.

Table 3: Performance measures of the models in the DCR dataset using various embeddings Model | Features | Accuracy | Precision | Recall | F1
---|---|---|---|---|---
LRC | BoW + TF-IDF | 0.3989 | 0.4807 | 0.3986 | 0.2957
| Character n-grams | 0.4172 | 0.5989 | 0.4169 | 0.3251
| Word2Vec | 0.6564 | 0.6758 | 0.6563 | 0.6604
| GloVe | 0.4691 | 0.4402 | 0.4690 | 0.4224
| fastText | 0.4551 | 0.5760 | 0.4551 | 0.4620
| GPT2 Embedding | 0.3895 | 0.3557 | 0.3897 | 0.3700
RFC | BoW + TF-IDF | 0.3340 | 0.4446 | 0.3337 | 0.1679
| Character n-grams | 0.3523 | 0.4741 | 0.3520 | 0.2163
| Word2Vec | 0.4842 | 0.5947 | 0.4843 | 0.4984
| GloVe | 0.4716 | 0.4425 | 0.4717 | 0.4456
| fastText | 0.5394 | 0.6011 | 0.5394 | 0.5531
| GPT2 Embedding | 0.5741 | 0.5784 | 0.5742 | 0.5152
SVM | BoW + TF-IDF | 0.4741 | 0.4824 | 0.4739 | 0.4163
| Character n-grams | 0.5956 | 0.6282 | 0.5956 | 0.5628
| Word2Vec | 0.5380 | 0.6137 | 0.5377 | 0.5244
| GloVe | 0.4044 | 0.4225 | 0.4043 | 0.4031
| fastText | 0.4409 | 0.5676 | 0.4410 | 0.4439
| GPT2 Embedding | 0.3652 | 0.3938 | 0.3653 | 0.3756
NBC | BoW + TF-IDF | 0.4741 | 0.4824 | 0.4739 | 0.4163
| Character n-grams | 0.5956 | 0.6282 | 0.5956 | 0.5628
| Word2Vec | 0.5442 | 0.6111 | 0.5439 | 0.5251
| GloVe | 0.4044 | 0.4225 | 0.4043 | 0.4031
| fastText | 0.4058 | 0.5460 | 0.4059 | 0.3850
| GPT2 Embedding | 0.3652 | 0.3938 | 0.3653 | 0.3756
DTC | BoW + TF-IDF | 0.3363 | 0.5788 | 0.3360 | 0.1728
| Character n-grams | 0.5041 | 0.6103 | 0.5038 | 0.4161
| Word2Vec | 0.5239 | 0.5341 | 0.5238 | 0.4854
| GloVe | 0.4452 | 0.4614 | 0.4453 | 0.4450
| fastText | 0.2872 | 0.2280 | 0.2873 | 0.2346
| GPT2 Embedding | 0.4205 | 0.4503 | 0.4206 | 0.4271
GBC | BoW + TF-IDF | 0.3340 | 0.7779 | 0.3337 | 0.1675
| Character n-grams | 0.3340 | 0.7779 | 0.3337 | 0.1675
| Word2Vec | 0.5848 | 0.5920 | 0.5845 | 0.5601
| GloVe | 0.4924 | 0.4794 | 0.4922 | 0.4601
| fastText | 0.5440 | 0.5741 | 0.5439 | 0.5519
| GPT2 Embedding | 0.4496 | 0.5225 | 0.4495 | 0.3741
MLP | BoW + TF-IDF | 0.6642 | 0.6820 | 0.6641 | 0.6687
| Character n-grams | 0.5587 | 0.5988 | 0.5588 | 0.5278
| Word2Vec | 0.5834 | 0.5871 | 0.5831 | 0.5652
| GloVe | 0.5085 | 0.5038 | 0.5083 | 0.5025
| fastText | 0.3794 | 0.4653 | 0.3794 | 0.3581
| GPT2 Embedding | 0.4643 | 0.4960 | 0.4644 | 0.4409

The bar graphs (Figures 1\- 3 visualize the accuracy scores for the seven
machine learning models (LRC, RFC, SVM, NBC, DTC, GBC, MLP) applied using
various feature combinations (BoW + TF-IDF, Character n-grams, Word2Vec,
GloVe, fastText, GPT2 embedding). These visualizations aid in embedding
selection and model decision, ultimately contributing to informed decisions
and improved system decision for the datasets.

Figure 1: Accuracy for Different Models and Feature Combinations in the DC
Dataset Figure 2: Accuracy for Different Models and Feature Combinations in
the DR Dataset Figure 3: Accuracy for Different Models and Feature
Combinations in the DCR Dataset

For each dataset, we provide a visual summary of the performance of one-shot
classification models across embeddings using Heatmaps. Fig. 4 represents the
performance measures of the one-shot classification model for the DC dataset,
showing the capabilities of the models across embeddings. This heatmap reveals
which models and embeddings perform better and allows decisions to be made
about the selection of models and embeddings for Doctor-AI consultations. Fig.
5 illustrates how one-shot classification models perform on the DR dataset.
With the graph, we can highlight trade-offs between models and embeddings
within a particular context, and decide which models are best matched with
which embeddings. Fig. 5, for the DCR dataset, sheds light on the performance
of one-shot classification models and helps us to understand how different
embeddings affect the performance of the model across various models using the
multiclass dataset. By referencing the heatmaps, we show that certain
embeddings consistently perform well on the chosen models.

Figure 4: Accuracy heatmap for DC Dataset Figure 5: Accuracy heatmap for DR
Dataset Figure 6: Accuracy heatmap for DCR Dataset

##  4 Discussion

We propose a comprehensive evaluation of embedding techniques for the one-shot
classification of Doctor-AI consultations. The increasing adoption of AI in
medical consultations has necessitated the development of efficient methods
for distinguishing between responses generated by Doctors and those produced
by AI systems. Different embedding techniques were tested and evaluated in our
experiments to see how well they handle the unique challenges of one-shot
learning in the classification of Doctor and AI-generated text.

In Table 1, which focuses on the DC dataset, Word2Vec and GloVe embeddings
consistently exceed others in all models. This suggests their ability to
achieve accurate classifications, underscoring their effectiveness in encoding
semantic information from text data. Character n-grams also consistently
deliver robust performance, serving as a dependable choice for encoding
textual content. In contrast, the GPT2 embedding lags behind in some models,
showing low accuracy and F1 scores. This discrepancy hints that it may not
align effectively with the models designed for this specific one-shot
classification task.

In Table 2, addressing the DR dataset, the accuracy ranges from approximately
0.45 to around 0.83, which indicates a substantial range in classification
correctness. The precision metric varies significantly, highlighting the
diversity in precision between classifications. These performance variations
underscore the absence of a universally superior model-embedding combination.
Instead, the results underscore the importance of selecting a combination of
model embedding that aligns with the specific requirements of the one-shot
classification task. Furthermore, the distinctive context of this dataset
showcases the substantial influence of this context on the performance of the
chosen models and embeddings.

Table 3 provides an in-depth analysis of one-shot classification models
applied to the DCR dataset using various embeddings. In this scenario, the
accuracy scores range from approximately 0.28 to 0.66, highlighting
substantial disparities in the overall classification accuracy for different
combinations of modeling elements. Precision values illustrate a wide
spectrum, varying from approximately 0.22 to 0.78, demonstrating the diversity
in precision across classifications. Recall figures range from about 0.16 to
0.66, indicating different capabilities to capture true positives relative to
actual positives. F1 scores, which balance precision and recall, range from
roughly 0.17 to around 0.66, underlining the equilibrium between correct
classification and the reduction of false positives and false negatives. The
pronounced performance diversity in the DCR dataset emphasizes that there is
no one-size-fits-all model-embedding combination. Instead, the choice of model
and embedding should be customized to the specific dataset and task
requirements. These results emphasize the critical role of context and data
characteristics in determining the optimal model-embedding combination in one-
shot classification scenarios.

##  5 Conclusion

This research has investigated the performance of various embedding methods in
the context of one-shot classification for the analysis of Doctor-generated
text and AI-generated text across three distinct datasets. We developed
multiple one-shot classification models, which advanced our understanding of
how embeddings impact the classification of text written by Doctors and text
generated by AI, with implications for the quality and reliability of online
medical advice. We tested the following traditional machine learning models
with different embedding techniques (bag-of-words, character n-grams,
word2vec, GloVe, FastText, GPT2 embedding): logistic regression, naive bayes,
random forest, support vector machines, decision tree, gradient boosting,
multi-layer perceptron. The key findings are as follows:

###  5.1 DC Dataset:

  * $\diamond$

Word2Vec and GloVe embeddings consistently outperformed other embeddings,
achieving high accuracy, precision, recall, and F1 scores consistently
exceeding 0.85.

  * $\diamond$

Character n-grams showed strong and consistent performance across certain
models, making them a reliable alternative for capturing textual information.

  * $\diamond$

The GPT2 embedding consistently underperformed in a number of models,
suggesting that it did not align well with the models developed and dataset of
this kind.

###  5.2 DR Dataset:

  * $\diamond$

The DR dataset exhibited a wide range of performance metrics across models and
embeddings. This dataset demonstrates how effectively GPT2 embeddings capture
intricate patterns in datasets of this kind, outperforming other embeddings
across many models.

  * $\diamond$

Character n-grams contribute to higher accuracy, precision, recall, and F1
scores with certain models.

  * $\diamond$

SVM performance is consistent across different embeddings, suggesting its
robustness across multiple feature representations, while Word2Vec maintains a
competitive edge in accuracy across multiple models.

###  5.3 DCR Dataset:

  * $\diamond$

Performance across the DCR dataset demonstrated substantial variations in
overall classification correctness. GPT2 embedding faces challenges, often
resulting in lower performance compared to other embeddings across multiple
models.

  * $\diamond$

The choice of model-embedding combinations played a crucial role in
classification effectiveness. Word2Vec consistently achieves competitive
scores across a variety of models.

  * $\diamond$

These results underscore the importance of considering context and dataset
characteristics when determining the optimal model-embedding combination. MLP
consistently perform well in different embeddings, highlighting their
stability in multiple feature representations.

###  5.4 Implications:

  * $\diamond$

All embeddings perform well in all models, demonstrating their reliability and
adaptability in capturing semantic features.

  * $\diamond$

The results emphasize the importance of selecting the appropriate embedding
and model for machine learning depending on the dataset.

  * $\diamond$

The effectiveness of the models vary significantly with embedding choice,
demanding careful consideration based on dataset features.

  * $\diamond$

Certain models may be more adaptable to specific types of feature
representation because of their sensitivity to embeddings.

  * $\diamond$

Disparities in performance across dataset underscore application-specific
considerations.

  * $\diamond$

What works well for one dataset may not be generalized to another, emphasizing
the importance of evaluating embeddings in the context of the specific dataset
and domain.

###  5.5 Limitations:

  * $\diamond$

The present study is not without limitations, including potential biases in
the dataset composition and inherent limitations in the choice of models and
embeddings.

  * $\diamond$

Based on the data characteristics of the datasets used in this analysis (DC,
DR, and DCR), generalizing these results to other datasets requires caution.

  * $\diamond$

The generalizability of the findings may be improved by more diverse and
larger datasets.

  * $\diamond$

Language models such as GPT2 generate embeddings that are subject to updates
and improvements, which can result in different performance results over time.

  * $\diamond$

The study does not address fine-tuned model architectures, which may further
affect performance.

###  5.6 Future Research Directions:

In order to improve classification accuracy, we will explore advanced and
context-aware embeddings in the future. We plan to explore the applicability
of fine-tuned large-language model architectures, specifically tailored to the
intricacies of health-related dialogues, to further improve classification.

##  6 Acknowledgments

This work was done with partial support from the Mexican Government through
the grant A1-S-47854 of CONACYT, Mexico, grants 20232138, 20230140, 20232080
and 20231567 of the Secretaría de Investigación y Posgrado of the Instituto
Politécnico Nacional, Mexico. The authors thank CONACYT for the computing
resources brought to them through the Plataforma de Aprendizaje Profundo para
Tecnologías del Lenguaje of the Laboratorio de Supercómputo of the INAOE,
Mexico and acknowledge the support of Microsoft through the Microsoft Latin
America PhD Award.

## References

  * [1] C. Tan, K. Talreja, A. Shivam, A. Nag, M. Miranda, S. Raghav, A. Rajaram, M. Lutz, S. Kumar, A. Kumar, Machine learning in clinical text classification: Specialty identification and covid-19 risk, in: 2022 13th International Conference on Information and Communication Technology Convergence (ICTC), IEEE, 2022, pp. 1366–1370.
  * [2] O. O. Adebanji, I. Gelbukh, H. Calvo, O. E. Ojo, Sequential models for sentiment analysis: A comparative study, in: Mexican International Conference on Artificial Intelligence, Springer, 2022, pp. 227–235.
  * [3] H. T. Ta, O. E. Ojo, O. O. Adebanji, H. Calvo, A. F. Gelbukh, The combination of BERT and data oversampling for answer type prediction, in: Proceedings of the SeMantic Answer Type and Relation Prediction Task at ISWC 2021 Semantic Web Challenge (SMART2021) co-located with the 20th International Semantic Web Conference (ISWC 2021), Virtual Conference, October 26th, 2021, Vol. 3119 of CEUR Workshop Proceedings, CEUR-WS.org, 2021, pp. 1–13.
URL https://ceur-ws.org/Vol-3119/paper1.pdf

  * [4] O. E. Ojo, H. T. Ta, A. Gelbukh, H. Calvo, O. O. Adebanji, G. Sidorov, Transformer-based approaches to sentiment detection, in: Recent Developments and the New Directions of Research, Foundations, and Applications: Selected Papers of the 8th World Conference on Soft Computing, February 03–05, 2022, Baku, Azerbaijan, Vol. II, Springer, 2023, pp. 101–110.
  * [5] O. E. Ojo, T.-H. Ta, A. Gelbukh, H. Calvo, G. Sidorov, O. O. Adebanji, Automatic hate speech detection using deep neural networks and word embedding, Computación y Sistemas 26 (2) (2022) 1007–1013.
  * [6] O. Ojo, O. Adebanji, H. Calvo, D. Dieke, O. Ojo, S. Akinsanya, T. Abiola, A. Feldman, Legend at ArAIEval shared task: Persuasion technique detection using a language-agnostic text representation model, in: Proceedings of ArabicNLP 2023, Association for Computational Linguistics, Singapore (Hybrid), 2023, pp. 594–599.  doi:10.18653/v1/2023.arabicnlp-1.61.
URL https://aclanthology.org/2023.arabicnlp-1.61

  * [7] O. Ojo, A. Gelbukh, H. Calvo, A. Feldman, O. Adebanji, J. Armenta-Segura, Language identification at the word level in code-mixed texts using character sequence and word embedding, in: Proceedings of the 19th International Conference on Natural Language Processing (ICON): Shared Task on Word Level Language Identification in Code-mixed Kannada-English Texts, 2022, pp. 1–6.
  * [8] D. Rozado, Danger in the machine: The perils of political and demographic biases embedded in ai systems, Manhattan Institute.
  * [9] O. B. Shoham, N. Rappoport, Cpllm: Clinical prediction with large language models (2023).  arXiv:2309.11295.
  * [10] O. E. Ojo, O. O. Adebanji, A. Gelbukh, H. Calvo, A. Feldman, Medai dialog corpus (medic): Zero-shot classification of doctor and ai responses in health consultations (2023).  arXiv:2310.12489.
  * [11] A. L. Tonja, O. E. Ojo, M. A. Khan, A. G. M. Meque, O. Kolesnikova, G. Sidorov, A. Gelbukh, Cic nlp at smm4h 2022: a bert-based approach for classification of social media forum posts, in: Proceedings of The Seventh Workshop on Social Media Mining for Health Applications, Workshop & Shared Task, 2022, pp. 58–61.
  * [12] O. E. Ojo, A. Gelbukh, H. Calvo, O. O. Adebanji, G. Sidorov, Sentiment detection in economics texts, in: Mexican International Conference on Artificial Intelligence, Springer, 2020, pp. 271–281.
  * [13] O. Ojo, A. Gelbukh, H. Calvo, O. Adebanji, Performance study of n-grams in the analysis of sentiments, Journal of the Nigerian Society of Physical Sciences (2021) 477–483.
  * [14] D. Fraile Navarro, A. B. Kocaballi, M. Dras, S. Berkovsky, Collaboration, not confrontation: Understanding general practitioners’ attitudes towards natural language and text automation in clinical practice, ACM Transactions on Computer-Human Interaction 30 (2) (2023) 1–34.
  * [15] N. Luo, X. Zhong, L. Su, Z. Cheng, W. Ma, P. Hao, Artificial intelligence-assisted dermatology diagnosis: From unimodal to multimodal, Computers in Biology and Medicine 165 (2023) 107413.  doi:https://doi.org/10.1016/j.compbiomed.2023.107413.
  * [16] Y. Sun, D. Gao, X. Shen, M. Li, J. Nan, W. Zhang, et al., Multi-label classification in patient-doctor dialogues with the roberta-wwm-ext+ cnn (robustly optimized bidirectional encoder representations from transformers pretraining approach with whole word masking extended combining a convolutional neural network) model: Named entity study, JMIR Medical Informatics 10 (4) (2022) e35606.
  * [17] S. Geng, W. Zhang, J. Xie, G. Liang, B. Niu, Patient dropout prediction in virtual health: A multimodal dynamic knowledge graph and text mining approach, arXiv preprint arXiv:2306.03833.
  * [18] C. Mao, Q. Zhu, R. Chen, W. Su, Automatic medical specialty classification based on patients’ description of their symptoms, BMC Medical Informatics and Decision Making 23 (1) (2023) 1–9.
  * [19] M. Wornow, Y. Xu, R. Thapa, B. Patel, E. Steinberg, S. Fleming, M. A. Pfeffer, J. Fries, N. H. Shah, The shaky foundations of large language models and foundation models for electronic health records, npj Digital Medicine 6 (1) (2023) 135.
  * [20] Z. Bao, W. Chen, S. Xiao, K. Ren, J. Wu, C. Zhong, J. Peng, X. Huang, Z. Wei, Disc-medllm: Bridging general large language models and real-world medical consultation, arXiv preprint arXiv:2308.14346.
  * [21] S. van der Linden, R. Sevastjanova, M. Funk, M. El-Assady, Medicospace: Visual decision-support for doctor-patient consultations using medical concept spaces from ehrs, ACM Transactions on Management Information Systems 14 (2) (2023) 1–20.
  * [22] G. Carloni, E. Pachetti, S. Colantonio, Causality-driven one-shot learning for prostate cancer grading from mri, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2023, pp. 2616–2624.
  * [23] O. Elnaggar, F. Coenen, A. Hopkinson, L. Mason, P. Paoletti, Sleep posture one-shot learning framework based on extremity joint kinematics: In-silico and in-vivo case studies, Information Fusion 95 (2023) 215–236.
  * [24] L. Xie, Y. Yang, Z. Fu, S. M. Naqvi, Position and orientation-aware one-shot learning for medical action recognition from signal data, arXiv preprint arXiv:2309.15635.
  * [25] A. Balasundaram, M. S. Kavitha, Y. Pratheepan, D. Akshat, M. V. Kaushik, A foreground prototype-based one-shot segmentation of brain tumors, Diagnostics 13 (7) (2023) 1282.
  * [26] D. Shi, S. He, J. Yang, Y. Zheng, M. He, One-shot retinal artery and vein segmentation via cross-modality pretraining, Ophthalmology Science (2023) 100363.
  * [27] Z. Liu, T. Zhong, Y. Li, Y. Zhang, Y. Pan, Z. Zhao, P. Dong, C. Cao, Y. Liu, P. Shu, Y. Wei, Z. Wu, C. Ma, J. Wang, S. Wang, M. Zhou, Z. Jiang, C. Li, J. Holmes, S. Xu, L. Zhang, H. Dai, K. Zhang, L. Zhao, Y. Chen, X. Liu, P. Wang, P. Yan, J. Liu, B. Ge, L. Sun, D. Zhu, X. Li, W. Liu, X. Cai, X. Hu, X. Jiang, S. Zhang, X. Zhang, T. Zhang, S. Zhao, Q. Li, H. Zhu, D. Shen, T. Liu, Evaluating large language models for radiology natural language processing (2023).  arXiv:2307.13693.
  * [28] Y. Ge, Y. Guo, Y.-C. Yang, M. A. Al-garadi, A. Sarker, Few-shot learning for medical text: A systematic review, ArXiv abs/2204.14081.
URL https://api.semanticscholar.org/CorpusID:248476028

  * [29] B. Zhang, R. Mishra, D. Teodoro, Ds4dh at mediqa-chat 2023: Leveraging svm and gpt-3 prompt engineering for medical dialogue classification and summarization, medRxivarXiv:https://www.medrxiv.org/content/early/\\\2023/06/12/2023.06.08.23291121.full.pdf, doi:10.1101/2023.06.08.23291121.
  * [30] J. He, X. Zhang, S. Lei, A. Alhamadani, F. Chen, B. Xiao, C.-T. Lu, Clur: Uncertainty estimation for few-shot text classification with contrastive learning, in: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023, pp. 698–710.
  * [31] W. Liao, Z. Liu, H. Dai, Z. Wu, Y. Zhang, X. Huang, Y. Chen, X. Jiang, D. Zhu, T. Liu, et al., Mask-guided bert for few shot text classification, arXiv preprint arXiv:2302.10447.
  * [32] S. Ghosal, A. Jain, Depression and suicide risk detection on social media using fasttext embedding and xgboost classifier, Procedia Computer Science 218 (2023) 1631–1639, international Conference on Machine Learning and Data Engineering.  doi:https://doi.org/10.1016/j.procs.2023.01.141.