# Give and Take: Federated Transfer Learning for Industrial IoT Network
Intrusion Detection

Lochana Telugu Rajesh1, Tapadhir Das2 Raj Mani Shukla1, and Shamik Sengupta3
2Department of Computer Science, University of the Pacific, USA
2Department of Computer Science and Engineering, University of Nevada, Reno,
USA
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>1School of Computing and Information Science, Anglia Ruskin
University, UK

###### Abstract

The rapid growth in Internet of Things (IoT) technology has become an integral
part of today’s industries forming the Industrial IoT (IIoT) initiative, where
industries are leveraging IoT to improve communication and connectivity via
emerging solutions like data analytics and cloud computing. Unfortunately, the
rapid use of IoT has made it an attractive target for cybercriminals.
Therefore, protecting these systems is of utmost importance. In this paper, we
propose a federated transfer learning (FTL) approach to perform IIoT network
intrusion detection. As part of the research, we also propose a combinational
neural network as the centerpiece for performing FTL. The proposed technique
splits IoT data between the client and server devices to generate
corresponding models, and the weights of the client models are combined to
update the server model. Results showcase high performance for the FTL setup
between iterations on both the IIoT clients and the server. Additionally, the
proposed FTL setup achieves better overall performance than contemporary
machine learning algorithms at performing network intrusion detection.

###### Index Terms:

Internet of Things, Industrial IoT, Network Intrusion Detection, Transfer
Learning, Federated Learning, Cybersecurity

††publicationid: pubid: 978-1-5386-5541-2/18/$31.00 ©2018 IEEE

##  I Introduction

The Internet of Things (IoT) has become an integral part of our lives as it
connects people, things, and processes flawlessly. At a very low cost and with
minimal human intervention, the physical world can meet the digital world.
Certain instances can include smart watches that use sensors to keep track of
your heartbeat rate and footstep count, motion sensor lights that toggle on
when you enter or exit a room, or even smart transit systems that can provide
passengers with predicted wait times for public transportation. IoT can be an
energy-saving resource that promotes automation to help the public. Cisco
stated that by the end of 2023, there will be more than three devices
connected to the internet for every human on the planet [1]. Industrial IoT
(IIoT) is the intersection of information technology (IT) and operational
technology (OT). IT utilizes computing devices to process electronic data
while OT uses programmable devices that gather environmental information
through repeated interactions. IIoT forms an integral part of today’s
industrial applications as the physical infrastructures of various industries
like agriculture, healthcare, and power, can be monitored using this
technology. An illustration of IIoT is provided in Figure 1.

Figure 1: Industrial Internet of Things

A major challenge to the adoption of IIoT is data security. IIoT consists of a
network of interconnected devices. Hence, if a device in the network is
compromised, all other devices in the network could be put in jeopardy.
Therefore, it is essential to secure the sensitive data of all IIoT devices
through measures such as secured data ports, transmission channels, and the
network. Kaspersky reported that within the first six months of 2021, 1.5
billion IoT devices had been attacked, an increase of 639 million from the
previous year [2]. A major hindrance to protecting these networks is the
progressively increasing number of users, resulting in the generation of
sizable and substantial data. Most of this generated data is private and
sensitive, requiring it to be protected from cybercriminals. Unlike
traditional IT, where there are considerable cybersecurity tools to protect
the data, IIoT data is not protected from cyber attacks by simply using
traditional cybersecurity methods.

With the rise of computational capabilities and faster processing power, the
usage of machine learning (ML) for performing IoT network intrusion detection
can be a valuable option for protecting IIoT data from being attacked.
However, certain limitations are currently present with existing ML-based
network intrusion detection systems (NIDS). The first limitation lies in the
scarcity of IIoT data. ML algorithms require high performance to be effective
as a network security mechanism, which requires tremendous amounts of highly-
dimensional data. Unfortunately, many laws such as the General Data Protection
Regulation (GDPR) do not allow the collection of data without the consent of
data subjects. Collection of IoT device data can be challenging as this data
can contain sensitive and private user data. For instance, a personal voice
assistant might contain overheard personal information from a user’s
conversation. Additionally, another limitation is the willingness to share the
data with relevant individuals and organizations. To maintain confidentiality,
organizations prohibit the sharing of their data with the outside world. In
some organizations, data sharing is restricted to a subset of individuals who
work directly with the data and is kept hidden from other employees. To ensure
security, certain practices restrict data sharing across international
borders. Therefore, finding a data source collected from IIoT devices is a
non-trivial task due to the restrictions placed on data-sharing practices by
organizations.

The second limitation lies with the traditional ML algorithms that have been
investigated to create ML-based NIDS for IIoT. Due to the high-dimensional
nature of IoT data, existing ML algorithms tend to lose their efficiency.
Also, most IoT data is collected from heterogenous sources like water sensors
and soil moisture sensors. This can make it difficult to create a
comprehensive ML algorithm capable of analyzing the different robust
statistics and variances of the data sources [3]. Additionally, developing ML
or deep learning (DL) techniques to perform IIoT intrusion detection is
challenging as new attack types and vectors are getting discovered frequently.
These new attack types are not being effectively detected by existing ML and
DL methods and can also require more computation power to operate [4].

To address the issue of data availability, federated learning (FL) has been
seen as an effective mechanism. FL helps to build personalized models for IIoT
devices that do not violate user and organizational privacy through novel
training methods. In addition, FL provides a privacy protection scheme that
utilizes the computing resources of the IIoT device to train the models,
thereby preventing information leakage during transmissions [5]. Similarly, to
address the limitation of heterogeneous IIoT data sources, transfer learning
(TL) can be an effective solution. TL focuses on transferring knowledge on
previously learned IIoT data to another domain to reduce computational
complexity and redundant training of ML models [6]. Through the usage of FL
and TL, ML-NIDS can be generated that protect IoT user privacy and also can
accommodate ML algorithms that can account for heterogeneous data sources.

In this paper, we propose a novel federated transfer learning (FTL) approach
to perform IIoT network intrusion detection. As the centerpiece for our
proposed approach, we propose a novel combinational neural network (Combo-NN)
architecture to perform effective intrusion detection. For this analysis, we
utilize a comprehensive dataset consisting of data collected from diverse IIoT
devices. The main contributions of this work include:

  * •

Performing data processing of the IIoT-based network intrusion detection
dataset to enhance performance.

  * •

Proposing an FTL-based approach to detect data breaches in IIoT environments,
and introducing a novel neural network approach to perform the FTL.

  * •

Analyzing and evaluating the performance of the proposed technique with
traditional ML approaches.

The rest of the paper is organized as follows: Section II introduces the
related work for this research. Our proposed methodology is shown in Section
III. Experimentation, results, and analysis are provided in Section IV.
Finally, conclusions are drawn in Section V.

##  II Related Work

The development of network intrusion detection approaches has been a critical
component in protecting today’s networking infrastructures. With the emergence
of IoT, these infrastructures need security contingencies now more than ever.
Contemporary IoT NIDS surrounded the usage of signature-based mechanisms [7]
[8] [9]. In [7], the authors proposed a signature-based NIDS that involved the
usage of both centralized and distributed intrusion detection models. The work
in [8] developed CBSigIDS, a generic framework of collaborative blockchain
signature-based intrusion detection systems, which can incrementally build and
update a trusted signature database in a collaborative IoT environment.
Researchers in [9] demonstrated the use of an optimized pattern recognition
algorithm to detect such attacks within IoT. The limitation of signature-based
methods is that they are static in operational capability and can be
circumvented if attackers introduce a level of variability in their malware.

With the rise of ML, ML-based NIDS has also become a trending technology to
protect IoT systems [10] [11] [12] [13] [14]. The authors in [10] used a deep
neural network to model various NIDS datasets to assess their performance at
detecting attacks. This work did not take into account the impact of class
imbalance on classifier performance. The work in [11] proposed a bi-
directional long short-term memory neural network on a NIDS dataset to
decipher between normal and attack conditions. The limitation of this work lay
in the small size of the dataset that was used to create the model.
Additionally, the trade-off between the decision parameters was not
considered. Researchers in [12] applied ML to various open-sourced NIDS
datasets. This work did not consider any fog or edge computing devices, which
is not representative of real-life practices. In [13], the researchers
observed multiple ML models on an IoT dataset. Though this work formed the
basis of contemporary research for NIDS for IoT environments, they did not
perform any model hyperparameter optimization that impacts performance. In
[14], authors evaluated six ML models for detecting MQTT-based IoT attacks.
The knock on this work is that it only observed performance after training
only on an MQTT protocol-based dataset.

Additionally, to preserve privacy efforts in IoT, the usage of FL on IoT has
seen some attention from researchers [15] [16] [17]. In [15], the authors
performed centralized FL on an IIoT dataset. The work done in [16] proposed an
FL-based NIDS that trained on multiple views of IoT network data in a
decentralized format to detect, classify, and defend against attacks. Lastly,
the researchers in [17] proposed an FL-based intrusion detection system for
securing agricultural-IoT infrastructures using local learning. The
limitations of these works are that they have not used the advantages of the
transfer learning, FL setup, which is the primary aim of this paper .

##  III Methodology

The proposed approach for the FTL algorithms consists of a client-server
architecture, where the models generated from the client data are utilized to
update the server model in the setup. The steps of this proposed methodology
are illustrated in Figure 2.

###  III-A IoT Dataset

The proposed approach has wide IIoT applications and is agnostic to dataset
features. For this research, any open-sourced IoT dataset can be utilized for
analysis. Let the dataset be denoted using the variable $Z$, consisting of $E$
total samples. $z_{i}$ represents a single data sample and $z_{i}\in
Z,i=\\{0,1...E\\}$. After the aggregation of the dataset, the first step is to
perform data pre-processing where redundant and/or incomplete information from
the datasets can be eliminated to not hamper the performance of any trained ML
model. Additionally, we also perform encoding in this step to ensure that no
categorical features are being fed into the ML model. Once pre-processed,
feature selection is performed to select the features that optimally affect
the performance of the trained ML model. Following this, we split the pre-
processed and feature-selected dataset into train and test samples. Finally,
we parse the trained models into our FTL module.

Figure 2: Proposed technique for the FTL Setup

###  III-B Pre-process

In this stage, we evaluate the contents of the acquired datasets. Any features
that seem to have redundancy or do not actively help in the classification
task are removed. Examples of such features include data features where all
values are the same in magnitude regardless of the labels associated with that
data sample. Additionally, we also evaluate the dataset features and eliminate
columns that consistently have “infinity” or “NaN” type values. These inputs
will not be accepted by an ML classifier and can hamper system performance if
used. Additionally, we also perform feature encoding in this stage, where
categorical features are encoded to have numerical values to ensure that these
inputs are accepted by ML classifiers. After data pre-processing, the original
dataset $Z$ is now denoted by $Z^{\prime}$.

###  III-C Feature Scaling

Once the dataset $Z^{\prime}$ has been pre-processed, it moves into the
feature scaling stage. The aim here is to identify dataset features that can
be essential to the classification task and prioritize them, along with
scaling them to fit within an operational range. To begin, we employ a
correlation matrix. This is an algebraic and statistical tool that can be
utilized to observe the relationship between two variables in a dataset. A
correlation score is given for each pair of features which helps us to
determine features that are not important. If it is positive, the features are
directly correlated. If it is negative, the features are inversely correlated.
The higher the magnitude of the score, the higher the correlation [18]. We
parse the dataset $Z^{\prime}$ through the correlation matrix to unearth which
data features are essential to the classification task. After analysis of the
resultant matrix, the features with the lowest correlation scores get dropped.
The threshold for dropping features is dependent on the variance of the
correlation scores for the data features. These features get dropped as they
result in higher cardinality for the dataset, while not providing any valuable
insights into the data. Finally, we scale the resultant features to ensure
that all feature magnitudes fit within a prescribed range. This makes sure
that any trained model is not biased towards features that have higher
magnitudes. This tactic is essential in datasets that have imbalanced
magnitudes. Post feature scaling step, the resultant dataset is denoted by
$X$.

Figure 3: Overview and Steps in the FTL Process

###  III-D Data Split

After data collection, pre-processing, and feature selection, the full data
suite $X$ is partitioned into the training $X_{train}$ and test $X_{test}$
sets. We assume the total number of training samples is $U$, while the total
number of testing samples is $V$. It is also ensured that $U>V$.

###  III-E Federated Transfer Learning

Once the data set has been split appropriately, the FTL approach can commence.
In this situation, we are utilizing the combination of a deep neural network
(DNN) and a convolutional neural network (CNN) as the base for our FTL in the
IIoT setup. We shall refer to this machine learning architecture as a
combinational neural network (Combo-NN). To conduct our FTL, we split the
existing training data $X_{train}$ into two parts: the client data, denoted by
$X_{c}$, and the server data, denoted by $X_{s}$. The proportion for splitting
the data depends on multiple IoT characteristics like the size of the network
and the number of devices that are in the IoT setup. Next, the server trains a
Combo-NN with $X_{s}$. Once trained, the server models get deployed to all the
IIoT clients. Within each client, the trained Combo-NN model gets retrained
with the local client data. Once all the models are trained, clients send the
models back to the server. The server performs federated averaging on these
models and deploys the resultant model back to all the clients, and the
procedure continues periodically till convergence. An overview of the FTL
procedure is illustrated in Figure 3. The following sections highlight the
important steps in the proposed approach.

Figure 4: Proposed Combo-NN Architecture

####  III-E1 Client Data

The client data, $X_{c}$, is then further split between all the clients in the
IIoT FTL setup. Here we denote the number of IoT devices to be $N$, which each
gets a subset of the client data. The local client data for each IoT client is
denoted by $X_{c_{i}}$, where i represents an IoT client in the
infrastructure.

####  III-E2 Server Data and Model Training

As mentioned above, the training data is also split into the server data,
denoted by $X_{s}$. The server data is essential as this forms the basis for
training our proposed Combo-NN model. The model trained on $X_{s}$ will be
initially deployed to all of the client devices in the IoT infrastructure to
begin the FTL procedure.

The proposed Combo-NN architecture consists of a CNN and a DNN. In our
methodology, we are using Resnet-50 as our CNN [19]. We are utilizing
Resnet-50 as it is an effective tool for generalizing nonlinear data in
literature [19]. However, other CNN architectures can also be utilized,
depending on the proposed application of the FTL. CNNs can be beneficial for
performing network intrusion detection as they tend to train fast due to their
underlying architecture. Also, CNNs have the capability of training multi-
layer networks with gradient descent that is competent to learn complex, high-
dimensional, and nonlinear data [20]. Additionally, we are augmenting multiple
hidden layers of a DNN to Resnet-50 for the proposed FTL approach. A DNN is
chosen as it provides benefits to performing network intrusion detection as
the layers of depth in the network are effective at modeling nonlinear
interactions between the various observable features in the network [21]. The
proposed Combo-NN architecture is illustrated in Figure 4.

Let our proposed untrained Combo-NN in the server be denoted by $M_{s}$. The
network, then, gets trained using the server data $X_{s}$ by:

$T_{s}=train(M_{s},X_{s})$ (1)

where $T_{s}$ denotes the trained Combo-NN in the server device. After
training, $T_{s}$ is duplicated $N$ times, each copy denoted by $T_{c_{i}}$,
where $i$ refers to each of the $N$ IoT clients in the infrastructure.

###  III-F Model Transfer to Clients

After training the server model, $T_{s}$, in the previous stage, $T_{s}$ is
duplicated $N$ times. Each duplicated copy is denoted by $T_{c_{i}}$, where
$i$ refers to each of the $N$ IoT clients in the infrastructure. In this
stage, all the trained duplicated copies are sent to all of the IoT client
devices. Each client device deploys $T_{c_{i}}$ in their environment.

###  III-G Model Training with Local Data

After each model trained model $T_{c_{i}}$ is deployed in each IoT client $i$,
in this stage, the model gets retrained using the local IoT data $X_{c_{i}}$.
The purpose of this is to generalize the model with the local data for the IoT
to provide better network intrusion detection performance for the client. The
network gets trained using the client data by:

$M_{c_{i}}=train(T_{c_{i}},X_{c_{i}})$ (2)

where $M_{c_{i}}$ denotes the trained Combo-NN model that was trained with the
local client data of the IoT device.

###  III-H Model Transfers to Server

Once all the models in all $N$ clients are trained with the local data, these
models are transferred to the IoT server. The purpose of this step is to
accumulate all the trained models into a centralized location to perform the
calibration of the global Combo-NN model located on the server.

###  III-I Federated Averaging

Once all of the trained models $M_{c}$ are transferred to the server, it can
commence the federated averaging step to aggregate all the client models into
one. FL is an iterative procedure and is traditionally conducted in multiple
rounds. Here, $t$ denotes the number of rounds for the FL. In this stage, we
perform the Federated Stochastic Gradient Descent (FedSGD) approach [22]. This
is conducted through the following equation:

$F_{i}(w_{t_{s}})=\frac{1}{n_{i}}\sum_{j\in\rho_{i}}{}f_{i}(w_{t_{s}})$ (3)

where $w_{t_{s}}$ are the weights of the current server model $T_{s}$, $n_{i}$
represents the number of data points on the IoT client $i$, $\rho_{i}$
represents the set of data points on the IoT client $i$, $f_{i}$ denotes the
loss that is achieved for the data sample ($x_{i}$, $y_{i}$) with the current
model weights $w_{t_{s}}$. Next, the FedSGD step is finished using:

$g_{i}=Gradient(F_{i}(w_{t_{s_{t}}}))$ (4)

where $g_{i}$ is the computed gradients for every single IoT client model.

###  III-J Server Model Weight Updates

The final step for our FTL approach is to update the weights of our server
model based on the results achieved from the federated averaging step. In this
stage, the central server aggregates the achieved gradients from the previous
stage and applies the update using:

$\forall i,w_{t_{s_{t+1}^{i}}}\leftarrow w_{t_{s_{t}}}-\eta g_{i}$ (5)

Here, $\eta$ represents the learning rate. In this step, each client $i$ takes
a step of gradient descent $g_{i}$ on the current server model using its local
data with the learning rate $\eta$. The last step in this stage is:

$w_{t_{s_{t+1}}}\leftarrow\sum_{i=1}^{N}\frac{n_{i}}{n}w_{t_{s_{t+1}^{i}}}$
(6)

In this step, the server takes the weighted average of the resulting models
and updates the current server model in the system.

##  IV Experimentation and Results

###  IV-A Experiment Setup

For the experimentation, we utilized Jupyter Notebook as our development
environment, along with Python being the development language. Some of the
essential libraries used were pandas, scikit-learn, matplotlib, seaborn,
keras, tensorflow, and psutil. The dataset that is used in our experiments is
Edge-IIoTset Cyber Security Dataset of IoT & IIoT [15]. For our experiment, we
assume $N=2$. For evaluation, we are using Accuracy (A) and Macro-Average
scores of Precision (MAP), Recall (MAR), and F-Measure (MAF). Additionally, we
are evaluating the FTL approach against Logistic Regression (LR), Gaussian
Naïve Bayes (GNB), Random Forest (RF), and Stochastic Gradient Descent (SGD)
classifiers. These algorithms have been chosen as they are customarily used to
study IoT network intrusion detection in the literature [23] [24] [25].

###  IV-B Performance Evaluation and Analysis

First, we observe the performance achieved by the IIoT client models and the
server model when they experience multiple iterations of the FTL approach. The
results can be illustrated in 6. It should be noted that the server model’s
performance is dependent on that of the client models in the FTL setup. Here,
we note that after the first iteration, Client 2 achieves a high accuracy of
89.8% while Client 1 achieves a high accuracy of 89.7%. Hence, after the first
iteration, the server achieves high performance with an accuracy of 90%. After
the second iteration, we note that Client 2 maintains its performance with an
89.8% accuracy. Client 1 slightly decreases in performance but still maintains
a high accuracy score of 88.2%. The server after the second iteration
diminishes negligibly but achieves a high accuracy score of 86%. In Client 1
and the server, we notice that slight decrease in performance between
iterations, which could be attributed to multiple reasons like inter-client
class imbalance and overall class imbalance in the dataset [26] [27].

Next, we evaluate the performance that the server IoT device experiences when
undergoing the proposed FTL approach. The results from this experiment are
illustrated in Figure 5. Here, we observe that when the server model has
experienced the first iteration of the FTL approach, it achieves high
performance with A, MAP, MAR, and MAF scores of 90%, 89%, 90%, and 88%,
respectively. After the second iteration of the FTL approach, the FTL approach
achieved comparable scores with A, MAP, MAR, and MAF scores of 86%, 84%, 86%,
and 83%, respectively. We observe a slight decrease from the previous
iteration, as previously noted in Figure 6. This can be attributed to multiple
reasons like inter-client class imbalance and overall class imbalance in the
dataset [26].

Figure 5: Progression of Server Performance through Iterations

Figure 6: Accuracy Progression of Client and Server Performances through
Iterations

Finally, we analyze the performance of the proposed FTL approach in performing
multi-class classification. For this, we evaluate the effectiveness of the
proposed approach against traditional ML algorithms that are customarily used
for IoT network intrusion detection. The results are illustrated in Figure 7.
From the observed results, we notice that the GNB classifier achieves the
least performance in being able to detect network intrusions with an A score
of 75%, an MAP, MAR, and MAF scores of 72%, 80%, and 69%, respectively. We
also note that contemporary ML algorithms achieve better performance than the
GNB classifier, but these achieved performances are not the optimal solutions.
The performance achieved by the proposed Combo-NN-based FTL approach provides
the best overall performance compared to the other contemporary ML algorithms.
It achieves a 90% accuracy score, and MAP, MAR, and MAF scores of 89%, 90%,
and 88% scores, respectively. This is because the FTL technique achieves
optimal performance iteratively, making it more suitable for performing
network intrusion detection over a wide set of network data. Additionally, the
usage of a neural network-based architecture ensures that any deeply
correlated network metrics that are not easily visible are identified by the
neural network. This showcases that the proposed FTL techniques can
effectively model the intrusion types in the dataset and better differentiate
between the various labels. Therefore, using the proposed FTL approach can be
a more effective mechanism to detect network intrusions in an IIoT setup
versus other contemporary ML algorithms.

Figure 7: Performance Evaluation of the FTL against Traditional ML Algorithms

##  V Conclusion

In this paper, we propose a new type of intrusion detection mechanism for
Industrial IoT devices using the concept of Federated Transfer Learning. As
part of this proposed work, we have introduced a novel neural network-based
architecture called Combo-NN which conducts network intrusion detection
utilizing a combination of a CNN and DNN. To evaluate the efficacy of our
proposed approach, we analyze its performance on an open-source IIoT network
intrusion detection dataset. Results showcase high performance for the FTL
setup between iterations on both the IIoT clients and the server.
Additionally, we observe that the proposed FTL setup can achieve better
overall performance than contemporary ML algorithms that have previously been
used to perform network intrusion detection in the literature. To the best of
our knowledge, this is the first time FTL is being used to perform network
intrusion detection for IIoT. Future work will focus on analyzing the
performance of the proposed technique to other network intrusion detection
datasets, along with developing potential security features to protect the
approach from adversarial attacks.

## References

  * [1] U. Cisco, “Cisco annual internet report (2018–2023) white paper,” _Cisco: San Jose, CA, USA_ , vol. 10, no. 1, pp. 1–35, 2020.
  * [2] C. Cyrus, “Iot cyberattacks escalate in 2021, according to kaspersky,” _IoT World Today, https://www. iotworldtoday. com/2021/09/17/iot-cyberattacks-escalate-in-2021-according-to-kaspersky/, last accessed December_ , 2021.
  * [3] R. Al-amri, R. K. Murugesan, M. Man, A. F. Abdulateef, M. A. Al-Sharafi, and A. A. Alkahtani, “A review of machine learning and deep learning techniques for anomaly detection in iot data,” _Applied Sciences_ , vol. 11, no. 12, p. 5320, 2021.
  * [4] N. V. Sharma and N. S. Yadav, “An optimal intrusion detection system using recursive feature elimination and ensemble of classifiers,” _Microprocessors and Microsystems_ , vol. 85, p. 104293, 2021.
  * [5] C. Zhang, Y. Xie, H. Bai, B. Yu, W. Li, and Y. Gao, “A survey on federated learning,” _Knowledge-Based Systems_ , vol. 216, p. 106775, 2021.
  * [6] F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He, “A comprehensive survey on transfer learning,” _Proceedings of the IEEE_ , vol. 109, no. 1, pp. 43–76, 2020.
  * [7] P. Ioulianou, V. Vasilakis, I. Moscholios, and M. Logothetis, “A signature-based intrusion detection system for the internet of things,” _Information and Communication Technology Form_ , 2018.
  * [8] W. Li, S. Tug, W. Meng, and Y. Wang, “Designing collaborative blockchained signature-based intrusion detection in iot environments,” _Future Generation Computer Systems_ , vol. 96, pp. 481–489, 2019.
  * [9] N. U. Sheikh, H. Rahman, S. Vikram, and H. AlQahtani, “A lightweight signature-based ids for iot environment,” _arXiv preprint arXiv:1811.04582_ , 2018.
  * [10] B. A. Tama and K.-H. Rhee, “Attack classification analysis of iot network via deep learning approach,” _Res. Briefs Inf. Commun. Technol. Evol.(ReBICTE)_ , vol. 3, pp. 1–9, 2017.
  * [11] B. Roy and H. Cheung, “A deep learning approach for intrusion detection in internet of things using bi-directional long short-term memory recurrent neural network,” in _2018 28th international telecommunication networks and applications conference (ITNAC)_. IEEE, 2018, pp. 1–6.
  * [12] N. Chaabouni, M. Mosbah, A. Zemmari, C. Sauvignac, and P. Faruki, “Network intrusion detection for iot security based on learning techniques,” _IEEE Communications Surveys & Tutorials_, vol. 21, no. 3, pp. 2671–2701, 2019.
  * [13] A. Alsaedi, N. Moustafa, Z. Tari, A. Mahmood, and A. Anwar, “Ton_iot telemetry dataset: A new generation dataset of iot and iiot for data-driven intrusion detection systems,” _Ieee Access_ , vol. 8, pp. 165 130–165 150, 2020.
  * [14] H. Hindy, E. Bayne, M. Bures, R. Atkinson, C. Tachtatzis, and X. Bellekens, “Machine learning based iot intrusion detection system: An mqtt case study (mqtt-iot-ids2020 dataset),” in _Selected Papers from the 12th International Networking Conference: INC 2020_. Springer, 2021, pp. 73–84.
  * [15] M. A. Ferrag, O. Friha, D. Hamouda, L. Maglaras, and H. Janicke, “Edge-iiotset: A new comprehensive realistic cyber security dataset of iot and iiot applications for centralized and federated learning,” _IEEE Access_ , vol. 10, pp. 40 281–40 306, 2022.
  * [16] D. C. Attota, V. Mothukuri, R. M. Parizi, and S. Pouriyeh, “An ensemble multi-view federated learning intrusion detection for iot,” _IEEE Access_ , vol. 9, pp. 117 734–117 745, 2021.
  * [17] O. Friha, M. A. Ferrag, L. Shu, L. Maglaras, K.-K. R. Choo, and M. Nafaa, “Felids: Federated learning-based intrusion detection system for agricultural internet of things,” _Journal of Parallel and Distributed Computing_ , vol. 165, pp. 17–31, 2022.
  * [18] J. H. Steiger, “Tests for comparing elements of a correlation matrix.” _Psychological bulletin_ , vol. 87, no. 2, p. 245, 1980.
  * [19] L. Wen, X. Li, and L. Gao, “A transfer convolutional neural network for fault diagnosis based on resnet-50,” _Neural Computing and Applications_ , vol. 32, pp. 6111–6124, 2020.
  * [20] E. Aminanto and K. Kim, “Deep learning in intrusion detection system: An overview,” in _2016 International Research Conference on Engineering and Technology (2016 IRCET)_. Higher Education Forum, 2016.
  * [21] P. Covington, J. Adams, and E. Sargin, “Deep neural networks for youtube recommendations,” in _Proceedings of the 10th ACM conference on recommender systems_ , 2016, pp. 191–198.
  * [22] S. Singh, “PPML Series #2 - Federated Optimization Algorithms - FedSGD and FedAvg,” Dec. 2021. [Online]. Available: https://shreyansh26.github.io/post/2021-12-18_federated_optimization_fedavg/
  * [23] S. Shitharth, P. R. Kshirsagar, P. K. Balachandran, K. H. Alyoubi, and A. O. Khadidos, “An innovative perceptual pigeon galvanized optimization (ppgo) based likelihood naïve bayes (lnb) classification approach for network intrusion detection system,” _IEEE Access_ , vol. 10, pp. 46 424–46 441, 2022.
  * [24] P. A. A. Resende and A. C. Drummond, “A survey of random forest based methods for intrusion detection systems,” _ACM Computing Surveys (CSUR)_ , vol. 51, no. 3, pp. 1–36, 2018.
  * [25] F. Abbasi, M. Naderan, and S. E. Alavi, “Anomaly detection in internet of things using feature selection and classification based on logistic regression and artificial neural network on n-baiot dataset,” in _2021 5th International Conference on Internet of Things and Applications (IoT)_. IEEE, 2021, pp. 1–7.
  * [26] M. Seol and T. Kim, “Performance enhancement in federated learning by reducing class imbalance of non-iid data,” _Sensors_ , vol. 23, no. 3, p. 1152, 2023\.
  * [27] B. Thiyam and S. Dey, “Efficient feature evaluation approach for a class-imbalanced dataset using machine learning,” _Procedia Computer Science_ , vol. 218, pp. 2520–2532, 2023.


# MedKP: Medical Dialogue with Knowledge Enhancement and Clinical Pathway
Encoding

Jiageng Wu1, Xian Wu2, 🖂, Yefeng Zheng2, Jie Yang3, 🖂
1Zhejiang University, Hangzhou, China
2Tencent Youtu Lab, Jarvis Research Center, Shenzhen, China
3Harvard University, Boston MA, USA
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

With appropriate data selection and training techniques, Large Language Models
(LLMs) have demonstrated exceptional success in various medical examinations
and multiple-choice questions. However, the application of LLMs in medical
dialogue generation—a task more closely aligned with actual medical
practice—has been less explored. This gap is attributed to the insufficient
medical knowledge of LLMs, which leads to inaccuracies and hallucinated
information in the generated medical responses. In this work, we introduce the
Medical dialogue with Knowledge enhancement and clinical Pathway encoding
(MedKP) framework, which integrates an external knowledge enhancement module
through a medical knowledge graph and an internal clinical pathway encoding
via medical entities and physician actions. Evaluated with comprehensive
metrics, our experiments on two large-scale, real-world online medical
consultation datasets (MedDG and KaMed) demonstrate that MedKP surpasses
multiple baselines and mitigates the incidence of hallucinations, achieving a
new state-of-the-art. Extensive ablation studies further reveal the
effectiveness of each component of MedKP. This enhancement advances the
development of reliable, automated medical consultation responses using LLMs,
thereby broadening the potential accessibility of precise and real-time
medical assistance.

##  1 Introduction

Large language models (LLMs) have demonstrated significant potential in the
medical field Tu et al. (2023). For example, several powerful LLMs have passed
medical licensing examinations in various countries, showcasing their
capability to solve medical questions on a par with junior doctors Singhal et
al. (2023a). Consequently, LLMs are being extensively explored in healthcare,
ranging from drafting medical reports to assisting clinical decision-making
Thirunavukarasu et al. (2023a).

Among all potential areas in medical domain, online medical consultation is
probably the most suitable for LLM application. Online medical consultation
has the following advantages: 1) it can increase patients’ accessibility to
medical care, especially for those in rural areas Wang et al. (2023b); 2) it
can let patients feel more relaxed than going to hospital offline, which in
turn increases the accuracy and completeness of collected main compliant; 3)
it can protect the privacy of patients. Due to above advantages, the number of
online medical consultations grows at an explosive speed. The outbreak of
COVID-19 has further boosted the adoption of online medical consultation.
According to a recent statistics
111https://www.globalmarketestimates.com/market-report/global-online-doctor-
consultation-market-2172, the market of online medical consultation is valued
on 3.9 Billion USD in 2020 and is estimated to achieve 16 Billion in 2026.

Specifically, LLMs can improve the efficiency of online medical consultation
from two perspectives: 1) for experienced doctors, LLMs can generate the draft
response automatically, the doctors only need to modify it rather than start
from scratch; 2) for inexperienced doctors, LLMs can reminder the possible
examination to take or further inquiries on symptoms, which in turn avoids the
misdiagnosis and missed diagnosis. Due to these potentials of LLMs in online
medical consultation, growing research efforts have been devoted to this area
Ayers et al. (2023); Sarraju et al. (2023); Lee et al. (2023). However, there
are still two remaining challenges: 1) LLM could produce hallucinations which
is not tolerable in medical domain; 2) LLM is now a black box, and the
inference procedure is hidden. Therefore, doctors are hard to uncover the
chain of thoughts of LLMs.

To address above two challenges, we introduce the Medical dialogue with
Knowledge enhancement and clinical Pathway encoding (MedKP) framework. MedKP
consists of two core modules: 1) External Knowledge Enhancement: this module
extracts related knowledge from a pre-built medical knowledge graph. The
extracted knowledge can help to guide the generation process of LLMs; 2)
Internal Clinical Pathway Encoding: this module mines key points from
historical conversations and the actions taken by doctors. This mined
information ensures the clinical coherence of the entire conversation.

To evaluate whether the proposed MedKP can relieve the hallucination problem,
in addition to the common natural language generation metrics like ROUGE Lin
(2004) and BertScore Zhang et al. (2019), we also introduce two types of new
metrics: 1) entity-based metric which helps to judge whether the key
information can accurately be captured; 2) LLM-judge based metrics which
evaluate the hallucination of generated responses. Overall, our main
contributions are summarized as follows:

  * •

We propose MedKP which enhances the automatic medical dialogue system with two
core modules: External Knowledge Enhancement through a medical knowledge
graph, and Internal Clinical Pathway Encoding via medical entities and
physician actions.

  * •

Integrating these enhancements with a generative Large Language Model (LLM)
for online medical consultations significantly reduces the hallucinations.

  * •

MedKP outperforms baseline models across two datasets, achieving state-of-the-
art results. Comprehensive ablation studies underscore the contribution of
individual components to the overall efficacy of our approach.

Figure 1: Workflow of medical dialogue with knowledge enhancement and clinical
pathway encoding framework.

##  2 Related Work

###  2.1 Large Language Model in Healthcare

Owing to extensive pre-training, LLMs encapsulate a broad spectrum of medical
knowledge Thirunavukarasu et al. (2023b). For general LLMs, GPT-4 Nori et al.
(2023) surpassed the USMLE passing score by more than 20 points. For medical-
specific LLMs, Med-PaLM Singhal et al. (2023a) and Med-PaLM 2 Singhal et al.
(2023b), achieved high scores of 67.6% and 86.5% on USMLE respectively,
indicating their expert-level proficiency in handling medical questions.
Additionally, Wu et al. (2023) quantified Chinese Medical Licensing
Examination by knowledge-enhanced LLMs.

There are emerging studies devoted to applying LLMs in the medical domain.
Jeblick et al. (2023) and Lyu et al. (2023) leverage ChatGPT and GPT-4 to
translate radiology reports into plain language. ChatCAD Wang et al. (2023a)
incorporates the LLMs for an interactive computer-aided diagnosis of radiology
images. However, applying LLM in the healthcare domain also raised concerns
over the generations of hallucination Harrer (2023) or biased results Liu et
al. (2024).

###  2.2 Medical Dialogue System

Medical dialogue systems aim to automatically generate responses to patient
inquiries, streamlining the delivery of medical services Chi et al. (2019).
For disease diagnosis, Wei et al. (2018) and Liu et al. (2022) have developed
systems for symptom collection and diagnosis using task-oriented dialogues and
knowledge graphs, respectively. For general responses, Liu et al. (2022) and
Li et al. (2021) focus on entity-driven dialogue generation for more accurate
responses. Plugmed Dou et al. (2023) exploits LLMs’ in-context learning for
generating physician responses. While studies have integrated medical entity
or knowledge graphs, they often employ additional models for entity prediction
or encoding, leading to a lack of interpretability and fragmented processes
that may omit crucial details, like symptom states (positive/negative).

##  3 Methodology

###  3.1 Problem Formulation

Each medical dialogue consists of inquiries from the patient and responses
from the physician, which we define as $U=\\{P,D\\}$ to represent a whole
medical dialogue. Here, $P=\\{p_{1},p_{2},...,p_{n}\\}$ denotes the patient’s
utterances, while $D=\\{d_{1},d_{2},...,d_{n-1}\\}$ represents the physician’s
utterances. The dialogue between patient and physician alternates in
chronological order. An automated medical dialogue system aims to generate a
physician’s response $d_{n}$ automatically, based on $U$ the patient-physician
dialogue up to the moment $n$ and the current patient inquiry $p_{n}$, thereby
completing the response.

###  3.2 Overall Workflow

To enhance the reliability of automatically generated responses, we introduce
the Medical dialogue with Knowledge enhancement and clinical Pathway encoding
(MedKP) framework, the whole workflow of which is illustrated in Figure 1.
This framework comprises three main components:

  * •

External Knowledge Enhancement: This module identifies medical entities
previously mentioned in historical utterances and retrieves relevant knowledge
from the medical knowledge graph. This process enriches the dialogue with
reliable medical knowledge.

  * •

Internal Clinical Pathway Encoding: It encodes the clinical pathway contained
in historical dialogue using medical entities and physician actions. This aids
in capturing the medical information conveyed in past conversations and
understanding the current state, thereby ensuring a coherent and informed
progression of the medical dialogue.

  * •

Response Generation: The relevant medical knowledge and encoded historical
utterances are formatted with a specified prompt template to leverage the in-
context learning ability of LLMs. We further fine-tuned LLM with the LoRA
framework to augment its ability to utilize external medical knowledge and
internal clinical pathways.

Furthermore, we design a comprehensive automatic evaluation scheme to assess
the response quality, including metrics related to medical entities, Natural
Language Generation (NLG), and judgment of hallucination based on LLM.

###  3.3 External Knowledge Enhancement

In this section, we explore how to integrate the knowledge graph to mine
reliable medical knowledge, serving as the knowledge foundation of medical
dialogue generation. Initially, we identify medical entities contained in each
utterance by patients and physicians, including symptoms, drugs, examinations,
and diseases. We aggregate all entities from historical utterances up to the
current turns $n$, representing as $E=\\{e_{1},e_{2},...,e_{m}\\}$. We
incorporate a large-scale medical knowledge graph $G=\\{K,T\\}$, where $K$
signifies all nodes and $T$ represents all triplets in $G$. Each triplet is
represented as $t_{ij}=<k_{i},k_{j},r_{ij}>$, with $k_{i}$ as the head node,
the $k_{j}$ as the tail node, and $r_{ij}$ denoting their relationship.

##### Direct knowledge among mentioned entities

We first identify the interrelationships existing among the previously
referenced medical entities, establishing a direct knowledge foundation. For
each entity $e_{i}$ within the entity set $E$, we explore connections with the
other entities in $E$, aiming to identify pair $\\{e_{i},e_{j}\\}\in E$ where
a triplet $t_{ij}$ in the knowledge graph denoting the relationship between
$e_{i}$ and $e_{j}$. This process enables the construction of $T_{direct}$,
defined as:

$T_{direct}=\\{t_{ij}\,|\,e_{i},e_{j}\in E\,and\,t_{ij}\in T\\}$ (1)

##### Potential knowledge from related entities

By integrating the interrelationships among medical entities and the network
structure of the knowledge graph, we can also mine medical concepts that are
not yet present in historical dialogue but are significantly related, serving
as potential knowledge supplements. Initially, we retrieve all nodes $K_{E}$
and edges $R_{E}$ from G related to the current entity set $E$. Subsequently,
we identify nodes not in $E$ but frequently connected to entities within $E$.
Specifically, for each $k\in K_{E}$, we calculate the frequency of its
relation to entities in $E$, selecting the top-N nodes most related to
multiple entities in $E$ as potential co-related nodes $K_{potential}$,
formulated as:

$K_{potential}=top\text{-}N\\{k|k\in K_{E},max{\sum_{e_{i}\in
E}}freq(k,e_{i})\\}$ (2)

Here, $freq(k,e_{i})$ denotes the frequency of the relationship between node
$k$ and entity $e_{i}\in E$. Further, we extract triplet set $T_{E}$ that is
related to $K_{potential}$, considering them as potential knowledge supplement
$T_{potential}$. The process is defined as:

$T_{potential}=\\{t_{jk}|e_{k}\in K_{potential},e_{j}\in E,t_{jk}\in T_{E}\\}$
(3)

It systematically identifies and incorporates potentially relevant medical
concepts, enriching the context with unexplored but significant knowledge.

###  3.4 Internal Clinical Pathway Encoding

To accurately represent the dynamic state of medical dialogue, we mine medical
entities together with physician actions, thereby encoding the underlying
clinical pathways. We identify medical entities within each patient and doctor
utterance, labeled as $E_{p_{n}}$ and $E_{d_{n}}$ respectively. Furthermore,
we analyze each physician’s response $d_{n}$ to recognize its $s$ actions into
$A_{d_{n}}=\\{a_{1},a_{2},...,a_{s}\\}$, with each $a$ belonging to a
predefined set of actions $A$. Following previous studies Li et al. (2021) and
Xu et al. (2023), we employ the SOAP note framework Cameron and Turtle-Song
(2002), a widely used method of documentation for physicians, to define seven
types of physician actions $A$: Chitchat, Inform, Inquire, Provide Daily
Precaution, State a Required Medical Test, Make a Diagnosis, and Prescribe
Medications.

Amidst this encoding scheme for the clinical pathway, we concatenate each
utterance with its identified entities and actions. The patient’s utterance is
encoded as:

$p^{\prime}=(E_{p}\parallel p)$ (4)

where $E_{p}$ denotes the entities identified in the patient’s utterance, and
$p$ is the utterance text. Similarly, the physician’s utterance is encoded as:

$d^{\prime}=(E_{d}\parallel A_{d}\parallel d)$ (5)

###  3.5 Response Generation

####  3.5.1 Inference

To generate the response with knowledge enhancement and encoded clinical
pathways, we employ a prompt template to format our input for LLM. As depicted
in Figure 1, we instruct the LLM with a detailed description following the
relevant knowledge and encode historical utterances.

####  3.5.2 Training

Given the encoded historical dialogue
$U^{\prime}=\\{P^{\prime},D^{\prime}\\}$, the model is firstly tasked with
generating the entities $e_{d_{n}}$ to be involved in the response and the
action $A_{d_{n}}$, followed by the actual textual response. The objective of
the generative language model is formalized to maximize the probability of
generating the physician’s response $d_{n}$, which can be represented as:

$\mathop{\arg\max}\limits_{\theta}P(d_{n}\,|\,U^{\prime},e_{d_{n}},A_{d_{n}})$
(6)

where $\theta$ denotes the current model parameters. With the incorporation of
entity and action information, the loss function is enriched to not only
account for the accuracy of the generated text but also the relevance and
correctness of the entities and actions. Hence, the loss function
$\mathcal{L}$ can be formulated as:

$\begin{split}\mathcal{L}=&\mathcal{L}_{g}(E_{d_{t}},\hat{E_{d_{t}}})+\mathcal{L}_{g}(A_{d_{t}},\hat{A_{d_{t}}})\\\
&+\mathcal{L}_{g}(d_{t},\hat{d_{t}})\end{split}$ (7)
$\begin{split}\mathcal{L}_{g}(y,\hat{y})=-\frac{1}{L}\sum_{l=1}^{L}y_{l}\log\hat{y_{l}}\end{split}$
(8)

where $\mathcal{L}_{g}(E_{d_{t}},\hat{E_{d_{t}}})$ penalizes discrepancies
between the predicted and actual entities and
$\mathcal{L}_{g}(A_{d_{t}},\hat{A_{d_{t}}})$ assesses the accuracy of the
predicted actions against the predefined set. These components of the loss
function synergistically guide the response generation, ensuring that the
output not only aligns with the factual content but also adheres to the
appropriate actions, thereby enhancing the reliability of the generated
response.

To expedite the training of LLMs, we employ the LoRA framework Hu et al.
(2021) to implement parameter-efficient fine-tuning. By freezing the
parameters of the base LLM and integrating additional LoRA layers specifically
for training, we can effectively tailor the model. This strategy enables
efficient adaptation of the LLM to incorporate external knowledge and clinical
pathways, providing more reliable responses.

###  3.6 Evaluation

Previous studies Ji et al. (2023); Risch et al. (2021) have observed that
conventional natural language generation (NLG) metrics face challenges in
efficiently measuring the quality of open-domain text generation tasks.
Especially in the medical domain, relying solely on character overlap without
considering the actual semantics and the information conveyed fails to
accurately and objectively evaluate the quality of text generation Pino et al.
(2021). Therefore, we employ comprehensive metrics to assess the quality of
generated responses and whether our method alleviates the hallucinations.

Figure 2: Prompt for LLM judge.

##### NLG metrics

We adopt the ROUGE Lin (2004) and BLEU Papineni et al. (2002) to evaluate the
quality of generated responses at the character level. Specifically, we
utilize BLEU-1/2/3/4 and ROUGE-1/2/L to measure in different n-grams.

##### Text similarity

We use the BertScore Zhang et al. (2019) to measure the overall similarity
between the generated text and the target text, primarily leveraging BERT to
compute the semantic distance between texts.

##### Medical Entity

To better evaluate the accuracy of generated responses in medical contexts, we
assess their performance at the entity level. Specifically, we calculate
recall, precision, and F1-score for entities that should be mentioned in the
responses. Moreover, while previous studies often adopted only micro-metrics
following Liu et al. (2022), this approach may overlook the accuracy of
sentences that are shorter or contain fewer entities. Consequently, we
calculate both macro- and micro-metrics to provide a comprehensive assessment.

##### LLM judge

The previous metrics only evaluate the differences between generated responses
and corresponding ground truth, neglecting the contextual background of
historical dialogue. To address this, we have leveraged the in-context
learning capabilities of advanced models to construct a judge based on LLMs
Zheng et al. (2023). This judge primarily focuses on consistency with previous
context (to mitigate hallucination) and consistency with subsequent responses
(to ensure consistency). Specifically, as illustrated in Figure 2, we have
designed a template for GPT-4 to act as an experienced doctor evaluating the
quality of generated responses. The evaluation is based on the following
criteria, with scoring and reasoning provided to enhance interpretability:

Hallucination (0-10): Measures whether the response introduces information
that conflicts with or is not mentioned in the preceding text. A lower score
indicates fewer hallucinations.

Consistency (0-10): Assess whether the response aligns with subsequent
physician responses, including key information and questions. A higher score
indicates stronger consistency.

Table 1: Performance evaluation of medical entities and LLM judge on MedDG. Rec/Pre/F1 stand for Recall/Precision/F1-score, which together evaluate the accuracy of predicting medical entities, H represents Hallucination, assessing the generation of non-factual information, and C stands for consistency, evaluating the logical coherence between ground-truth and the generated one. Method | Medical Entity-macro | Medical Entity-micro | LLM-Judge
---|---|---|---
Rec | Pre | F1 | Rec | Pre | F1 | H | C
DL-based | Seq2Seq | 13.49 | 15.78 | 13.98 | 10.42 | 27.18 | 15.07 | 1.80 | 3.76
Seq2Seq-Entity | 19.42 | 21.86 | 19.27 | 15.91 | 35.79 | 22.03 | 1.13 | 4.16
HRED | 12.87 | 15.18 | 13.29 | 10.03 | 25.55 | 14.40 | 1.71 | 3.52
HRED-Entity | 19.01 | 20.96 | 18.63 | 15.41 | 33.02 | 21.01 | 1.64 | 3.86
VRBOT | 11.90 | 14.99 | 12.56 | 9.49 | 29.31 | 14.34 | 1.53 | 3.47
PLM-based | GPT-2 | 17.13 | 19.62 | 17.27 | 14.34 | 29.19 | 19.23 | 1.05 | 4.30
GPT-Entity | 20.06 | 22.71 | 19.96 | 16.99 | 32.12 | 22.22 | 1.03 | 4.67
BART | 17.53 | 20.58 | 17.89 | 14.28 | 30.83 | 19.52 | 1.05 | 4.54
BART-Entity | 20.76 | 22.43 | 19.92 | 16.98 | 35.56 | 22.98 | 1.11 | 4.54
DFMed | 27.98 | 26.14 | 24.76 | 24.13 | 32.45 | 27.68 | 1.06 | 5.39
LLM-based | Direct Inference | 13.65 | 13.71 | 12.41 | 12.62 | 17.00 | 14.49 | 2.60 | 3.49
MedKP | 32.38 | 35.11 | 31.41 | 28.12 | 29.62 | 28.85 | 1.03 | 6.10

##  4 Experiments

We conduct experiments on two public datasets of medical dialogue for our
evaluation: (1) MedDG dataset Liu et al. (2022) is sourced from Doctor
Chunyu222https://www.chunyuyisheng.com/. It comprises 17,684 medical
dialogues, primarily focusing on 12 gastrointestinal diseases. This entity-
centric dataset is systematically annotated by physicians and defines 160
normalized medical entities across five types: disease, symptom, medicine,
examination, and attribute. Adhering to the official dataset partition, we
divide the dataset into 14,862/1,999/999 dialogues for the training,
validation, and test sets, respectively. (2) KaMed dataset Li et al. (2021) is
also derived from Doctor Chunyu, caters to diverse clinical scenarios with its
inclusion of over 100 hospital departments (No overlap with MedDG). It is
larger than previous datasets in scale and also features more rounds of
conversation, making it more challenging. Additionally, the original sessions
of KaMed contain multi-modal information such as images and voice recordings,
which have been replaced by the meaningless template "The image/voice is not
available for privacy concern", leading to incomplete information. Following
the filtering rules established by (Xu et al., 2023), we have filtered the raw
dataset to curated 29,159/1,532/1,539 for training/validation/testing.

Table 2: Performance evaluation of NLG metrics and text similarity on MedDG Method | NLG-ROUGE | NLG-BLEU | Text Similarity
---|---|---|---
R-1 | R-2 | R-L | B-1 | B-2 | B-3 | B-4 | BertScore
DL-based | Seq2Seq | 21.90 | 9.13 | 20.95 | 21.86 | 17.33 | 14.55 | 11.84 | 63.35
Seq2Seq-Entity | 22.74 | 9.60 | 21.49 | 23.49 | 18.49 | 15.51 | 12.60 | 64.02
HRED | 21.72 | 8.88 | 20.48 | 25.56 | 20.42 | 17.34 | 14.22 | 63.47
HRED-Entity | 22.55 | 9.03 | 20.99 | 27.63 | 21.78 | 18.36 | 14.86 | 63.87
VRBOT | 20.41 | 8.59 | 19.42 | 22.89 | 18.24 | 15.52 | 12.71 | 62.30
PLM-based | GPT-2 | 25.05 | 11.15 | 23.50 | 28.27 | 22.28 | 18.75 | 15.28 | 65.00
GPT-Entity | 25.51 | 11.30 | 23.79 | 28.31 | 22.21 | 18.62 | 15.13 | 65.20
BART | 25.37 | 11.50 | 23.85 | 27.32 | 21.49 | 17.97 | 14.53 | 65.20
BART-Entity | 24.99 | 11.19 | 23.37 | 27.22 | 21.33 | 17.88 | 14.53 | 65.07
DFMed | 28.22 | 12.81 | 25.07 | 38.93 | 29.81 | 24.82 | 20.00 | 66.58
LLM-based | PlugMed | - | - | 21.10 | - | - | - | - | 64.10
Direct Inference | 18.51 | 5.09 | 15.55 | 37.83 | 30.09 | 25.70 | 20.59 | 60.70
MedKP | 29.50 | 14.25 | 26.86 | 37.41 | 29.08 | 24.24 | 19.64 | 67.35

Notably, real medical consultations often contain many simple sentences, such
as chitchat or greetings, with few words and lacking medical information. To
efficiently develop a robust automatic medical dialogue generation system, our
evaluation primarily focuses on the challenging responses that contain at
least one medical entity.

Table 3: Performance evaluation of medical entities and LLM judge on KaMed. Model | Medical Entity-macro | Medical Entity-micro | LLM-Judge
---|---|---|---
Rec | Pre | F1 | Rec | Pre | F1 | H | C
DL-based | Seq2Seq | 8.70 | 10.13 | 8.91 | 8.48 | 18.93 | 11.72 | 2.32 | 1.72
HRED | 8.94 | 10.26 | 9.11 | 8.40 | 15.62 | 10.93 | 1.81 | 1.44
VRBOT | 6.18 | 7.64 | 6.56 | 5.87 | 16.17 | 8.61 | 2.28 | 1.66
PLM-based | GPT-2 | 14.98 | 16.03 | 14.67 | 13.70 | 21.13 | 16.62 | 0.80 | 2.98
BART | 16.50 | 17.56 | 16.20 | 15.40 | 23.69 | 18.66 | 0.71 | 3.23
DFMed | 27.84 | 26.62 | 25.75 | 25.45 | 23.37 | 24.37 | 0.69 | 4.04
LLM-based | Direct Inference | 20.61 | 21.01 | 19.74 | 19.11 | 23.57 | 21.11 | 0.96 | 4.10
MedKP | 35.25 | 32.49 | 31.71 | 33.09 | 24.12 | 27.90 | 0.20 | 5.38

###  4.1 Baseline models

To fully evaluate the performance of different methods in medical dialogue
generation, we constructed several baselines that cover deep
learning(DL)-based methods, pre-trained language model(PLM)-based methods, and
LLM-based methods.

DL-based method: (1) Seq2Seq Sutskever et al. (2014) is a classical sequence
to sequence model, employing an attention mechanism coupled with RNN-based
architectures for both the encoder and decoder components. (2) HRED Serban et
al. (2016) advanced the conventional Seq2Seq encoder by employing a
hierarchical structure that models a dialogue as a token sequence and an
utterance sequence. (3) VRBot Li et al. (2021) is an end-to-end variational
reasoning model for medical dialogue generation that tracks patient state and
physician action.

PLM-based method: (1) GPT-2 Radford et al. (2019) is a classical transformer-
decoder-based language model. (2) BART Lewis et al. (2019) is a transformer-
based encoder-decoder model. (3) DFMed Xu et al. (2023) employs two sequential
models to predict medical entities and physician actions, respectively. It
introduces an interweaving component designed to integrate those predicted
states to generate a physician’s response.

LLM-based method: (1) PlugMed Dou et al. (2023) retrieves similar dialogues to
guide LLMs to generate responses and fine-tune a small model to discern the
best responses. They employ BLOOM as the foundation model. Due to its code and
dataset is not available, we only record and compare the reported metrics. (2)
Direct Inference represents the direct application of
ChatGLM3-6B333https://github.com/THUDM/ChatGLM3 for response generation
without enhancement or fine-tuning Furthermore, leveraging the high-quality
entities defined by physicians within the MedDG dataset, we have also
augmented various baselines with entity enhancement. Following (Liu et al.,
2022), we entail appending entities predicted by an auxiliary model directly
to the dialogue history. Such augmentation serves as a hint for the generative
language model.

###  4.2 Implementation Details

We select ChatGLM3-6B as our base LLM. The adaptation utilized a LoRA rank $r$
of 8, scaling factor $\alpha$ of 32, and dropout rate of 0.1. The layers
designated for training within the architecture of ChatGLM3 include the self-
attention components and linear layers, specifically: "query_key_value",
"dense", "dense_h_to_4h", "dense_4h_to_h" We set the batch size to 64,
conducting the training over 20 epochs for each dataset. The AdamW optimizer
starts with a learning rate of 5e-4 and decreases to 5e-5. To satisfy $99\%$
of data, the maximum input length is 1,536 tokens and the maximum output
length is 256 tokens.

We select CMEKG444http://cmekg.pcl.ac.cn/ as the knowledge graph and select
top 5 commonly related entities for mining the potential knowledge. For all
baselines, we implement the open-source code and follow their settings
provided by Liu et al. (2022), Li et al. (2021), and Xu et al. (2023). The
MedBERT555https://github.com/trueto/medbert pretrained in the medical domain
is selected as the backbone of PLM-based methods. For LLM Judge, we conduct
tests by calling OpenAI’s official API with the model version
‘GPT4-0125-preview’. Due to the API access rate limitations, we randomly
selected 500 samples in each dataset for testing.

Table 4: Performance evaluation of NLG metrics and text similarity on KaMed. Model | NLG-ROUGE | NLG-BLEU | Text Similarity
---|---|---|---
R-1 | R-2 | R-L | B-1 | B-2 | B-3 | B-4 | BertScore
DL-based | Seq2Seq | 17.85 | 4.71 | 16.96 | 14.16 | 10.99 | 9.14 | 7.23 | 59.86
HRED | 18.35 | 4.81 | 17.02 | 18.59 | 14.53 | 12.17 | 9.66 | 60.53
VRBOT | 16.98 | 5.60 | 14.85 | 27.18 | 21.38 | 18.38 | 14.85 | 57.43
PLM-based | GPT-2 | 22.79 | 7.49 | 20.53 | 26.98 | 20.81 | 17.43 | 13.92 | 62.39
BART | 23.59 | 8.09 | 21.29 | 26.27 | 20.19 | 16.82 | 13.41 | 62.95
DFMed | 26.36 | 9.82 | 22.54 | 36.35 | 27.37 | 22.64 | 17.99 | 64.42
LLM-based | PlugMed | - | - | 14.10 | - | - | - | - | 60.10
Direct Inference | 23.25 | 7.49 | 19.60 | 37.94 | 29.21 | 24.62 | 19.67 | 62.47
MedKP | 27.53 | 10.93 | 23.75 | 38.01 | 28.56 | 23.67 | 18.88 | 64.90
Table 5: Experimental results of ablation study on MedDG. The +KG indicates
the integration of knowledge graph-based enhancement; the +CP refers to the
incorporation of clinical pathways encoding via both medical entities and
physician actions, while +entity signifies encoding solely depends on medical
entity; MedKP denotes the cooperation of KG and CP.

Method | Medical Entity-macro | Medical Entity-micro | LLM-Judge | NLG-ROUGE | NLG-BLEU | Similarity
---|---|---|---|---|---|---
Rec | Pre | F1 | Rec | Pre | F1 | H | C | R-1 | R-2 | R-L | B-1 | B-2 | B-3 | B-4 | BertScore
Direct Inference | 13.65 | 13.71 | 12.41 | 12.62 | 17.00 | 14.49 | 2.60 | 3.49 | 18.51 | 5.09 | 15.55 | 37.83 | 30.09 | 25.70 | 20.59 | 60.70
\+ KG | 28.27 | 31.39 | 27.73 | 24.17 | 28.91 | 26.33 | 0.97 | 5.52 | 28.89 | 14.30 | 26.83 | 33.15 | 25.99 | 21.77 | 17.69 | 67.52
\+ Entity | 32.20 | 34.90 | 31.13 | 27.81 | 28.44 | 28.13 | 1.01 | 5.51 | 28.86 | 13.87 | 26.61 | 33.67 | 26.26 | 21.90 | 17.70 | 67.51
\+ CP | 31.41 | 34.82 | 30.87 | 26.90 | 30.23 | 28.47 | 1.03 | 5.62 | 28.94 | 14.17 | 26.70 | 34.85 | 27.25 | 22.83 | 18.61 | 67.35
\+ MedKP | 32.38 | 35.11 | 31.41 | 28.12 | 29.62 | 28.85 | 1.03 | 6.10 | 29.50 | 14.25 | 26.86 | 37.41 | 29.08 | 24.24 | 19.64 | 67.35

##  5 Results and Analysis

###  5.1 Main result

Table 1 and Table 2 present a detailed performance evaluation of different
methods applied to the MedDG dataset, while Table 3 and Table 4 extend the
evaluation to the KaMed dataset. Overall, the proposed MedKP framework
exhibits a remarkable superiority over competing baselines, yielding new SOTA
results across multiple metrics.

Medical Entity The PLM-based methods demonstrate superior efficacy over DL-
based approaches, with the integration of entity hints also augmenting
performance. Notably, MedKP significantly outperforms all other baselines. On
the MedDG dataset, the LLM equipped with MedKP achieves a substantial increase
in performance, with macro-F1 and micro-F1 scores improving dramatically from
12.41 to 31.41 and 14.49 to 28.85, respectively. Compared to the previous
best-performing baseline, MedKP also yielded considerable gains of 6.65 in
macro-F1 and 1.17 in micro-F1. The pronounced enhancement in macro-metrics
underscores MedKP’s proficiency in precisely delivering pertinent medical
information, highlighting its effectiveness even in concise responses. These
enhancements suggest that the responses generated by MedKP are not only more
informative but also closely mirror the physicians’ responses, thereby
ensuring the effectiveness of medical consultations.

LLM Judge The integration of medical entities effectively facilitates the
understanding of the dialogue state, thereby reducing hallucinations (e.g.,
Seq2Seq and Seq2Seq-entity). Compared to DL-based models, PLM-based methods
showed improved performance, which may be attributed to the proficiency of
medical LLM in understanding medical text. In contrast, LLM applied directly
often introduces irrelevant or conflicting patient information and then
suffers from severe hallucinations. Benefiting from reliable medical knowledge
and the precise understanding of both historical and current states afforded
by pathway encoding, MedKP significantly reduces hallucinations and surpasses
other methods by a notable margin, achieving a 0.71 improvement in consistency
on MedDG.

NLG Metrics and Text Similarity On the KaMed and MedDG datasets, MedKP
achieved the highest ROUGE scores, outperforming the best baseline by 7.14% on
MedDG and 5.36% on KaMed. Similarly, in terms of sentence-level similarity, as
measured by BERTScore, MedKP yielded the best result. However, regarding BLEU
scores, while MedKP’s performance was notably high, the highest scores were
obtained by directly applying LLM. This discrepancy across metrics may be
attributed to the LLM’s tendency to generate long and general suggestions,
leading to high overlaps with standard responses. However such unfocused
responses are meaningless for subsequent consultation. Consequently, metrics
like ROUGE that calculate recall of standard responses and medical entity that
reflect key information tend to be lower. This highlights the potential risks
of using traditional NLG metrics for evaluating rigorous medical text
generation.

###  5.2 Ablation study

Table 5 demonstrates that each component of MedKP significantly enhances its
performance on the MedDG dataset. The integration of external medical KG
notably increases the reliability of responses, achieving the lowest rates of
hallucination and the highest BERTScore. Pathway encoding facilitates an
understanding of the current state, favoring the prediction of medical
entities that should be discussed in subsequent responses. Moreover, the
predicted entities and actions guide the LLM towards generating content that
is more focused and aligned with physician responses, as demonstrated by
improvements in medical entity-metrics and NLG metrics. Cooperation with all
components, MedKP manifests advantages in several metrics, from entity-related
to hallucination, underscoring the effectiveness of our framework.

##  6 Conclusion

In this paper, we present MedKP which generates the response from doctors in
online medical consultations with LLMs. To alleviate the hallucination
problem, on one hand, MedKP introduces an external medical knowledge graph to
guide the generation of LLMs; On the other hand, MedKP identifies the key
point and physician actions within a conversation which ensures clinical
coherence. To evaluate the hallucination problem, we also introduce entity-
based and LLM-judge metrics in addition to the common NLG metrics. Experiments
on two public benchmarks that demonstrated the effectiveness of MedKP.

## Limitations

Several potential limitations should be considered for this study. Firstly,
the parallel tests were not conducted on more LLMs. This stems from the fact
that some advanced LLMs, such as Med-PaLM, have not yet been made available.
It is also due to the high computational resources required for fine-tuning
LLMs. In addition, the detailed responses were not further examined from a
professional perspective, which could better evaluate the quality of the
generated response. We are currently collaborating with clinical physicians to
further this work, and hope to continue refining it in subsequent studies.

## Ethics Statement

While the medical dialogue involves patient information, all cases have been
anonymized, ensuring that no personal information is disclosed. Moreover, the
primary objective of this study is to investigate the effectiveness of LLM in
medical response generation. The results and conclusions will not serve as
medical suggestions. Consequently, they do not have any adverse effect on
human healthcare.

## References

  * Ayers et al. (2023) John W Ayers, Adam Poliak, Mark Dredze, Eric C Leas, Zechariah Zhu, Jessica B Kelley, Dennis J Faix, Aaron M Goodman, Christopher A Longhurst, Michael Hogarth, et al. 2023.  Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum.  _JAMA internal medicine_.
  * Cameron and Turtle-Song (2002) Susan Cameron and Imani Turtle-Song. 2002.  Learning to write case notes using the soap format.  _Journal of Counseling & Development_, 80(3):286–292.
  * Chi et al. (2019) Nai-Ching Chi, Yong Choi, Shih-Yin Lin, Hilaire Thompson, George Demiris, L Laranjo, A Dunn, H Tong, R Jiang, S Provoost, et al. 2019.  A systematic review of health dialog systems.  _Methods of information in medicine_ , 58(06):179–193.
  * Dou et al. (2023) Chengfeng Dou, Zhi Jin, Wenpin Jiao, Haiyan Zhao, Yongqiang Zhao, and Zhengwei Tao. 2023.  Plugmed: Improving specificity in patient-centered medical dialogue generation using in-context learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 5050–5066.
  * Harrer (2023) Stefan Harrer. 2023.  Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine.  _EBioMedicine_ , 90.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Jeblick et al. (2023) Katharina Jeblick, Balthasar Schachtner, Jakob Dexl, Andreas Mittermeier, Anna Theresa Stüber, Johanna Topalis, Tobias Weber, Philipp Wesp, Bastian Oliver Sabel, Jens Ricke, et al. 2023.  Chatgpt makes medicine easy to swallow: an exploratory case study on simplified radiology reports.  _European radiology_ , pages 1–9.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Lee et al. (2023) Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023.  Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine.  _New England Journal of Medicine_ , 388(13):1233–1239.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Li et al. (2021) Dongdong Li, Zhaochun Ren, Pengjie Ren, Zhumin Chen, Miao Fan, Jun Ma, and Maarten de Rijke. 2021.  Semi-supervised variational reasoning for medical dialogue generation.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 544–554.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2022) Wenge Liu, Jianheng Tang, Yi Cheng, Wenjie Li, Yefeng Zheng, and Xiaodan Liang. 2022.  Meddg: an entity-centric medical consultation dataset for entity-aware medical dialogue generation.  In _CCF International Conference on Natural Language Processing and Chinese Computing_ , pages 447–459. Springer.
  * Liu et al. (2024) Xiaocong Liu, Jiageng Wu, An Shao, Wenyue Shen, Panpan Ye, Yao Wang, Juan Ye, Kai Jin, Jie Yang, et al. 2024.  Uncovering language disparity of chatgpt on retinal vascular disease classification: Cross-sectional study.  _Journal of Medical Internet Research_ , 26(1):e51926.
  * Lyu et al. (2023) Qing Lyu, Josh Tan, Michael E Zapadka, Janardhana Ponnatapura, Chuang Niu, Kyle J Myers, Ge Wang, and Christopher T Whitlow. 2023.  Translating radiology reports into plain language using chatgpt and gpt-4 with prompt learning: results, limitations, and potential.  _Visual Computing for Industry, Biomedicine, and Art_ , 6(1):9.
  * Nori et al. (2023) Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. 2023.  Capabilities of gpt-4 on medical challenge problems.  _arXiv preprint arXiv:2303.13375_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Pino et al. (2021) Pablo Pino, Denis Parra, Cecilia Besa, and Claudio Lagos. 2021.  Clinically correct report generation from chest x-rays using templates.  In _Machine Learning in Medical Imaging: 12th International Workshop, MLMI 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, September 27, 2021, Proceedings 12_ , pages 654–663. Springer.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Risch et al. (2021) Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. 2021.  Semantic answer similarity for evaluating question answering models.  _arXiv preprint arXiv:2108.06130_.
  * Sarraju et al. (2023) Ashish Sarraju, Dennis Bruemmer, Erik Van Iterson, Leslie Cho, Fatima Rodriguez, and Luke Laffin. 2023.  Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model.  _JAMA_ , 329(10):842–844.
  * Serban et al. (2016) Iulian Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016.  Building end-to-end dialogue systems using generative hierarchical neural network models.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 30.
  * Singhal et al. (2023a) Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023a.  Large language models encode clinical knowledge.  _Nature_ , 620(7972):172–180.
  * Singhal et al. (2023b) Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023b.  Towards expert-level medical question answering with large language models.  _arXiv preprint arXiv:2305.09617_.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  _Advances in neural information processing systems_ , 27.
  * Thirunavukarasu et al. (2023a) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023a.  Large language models in medicine.  _Nature medicine_ , 29(8):1930–1940.
  * Thirunavukarasu et al. (2023b) Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023b.  Large language models in medicine.  _Nature medicine_ , 29(8):1930–1940.
  * Tu et al. (2023) Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ktena, et al. 2023.  Towards generalist biomedical ai.  _arXiv preprint arXiv:2307.14334_.
  * Wang et al. (2023a) Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. 2023a.  Chatcad: Interactive computer-aided diagnosis on medical image using large language models.  _arXiv preprint arXiv:2302.07257_.
  * Wang et al. (2023b) Xiaofei Wang, Hayley M Sanders, Yuchen Liu, Kennarey Seang, Bach Xuan Tran, Atanas G Atanasov, Yue Qiu, Shenglan Tang, Josip Car, Ya Xing Wang, et al. 2023b.  Chatgpt: promise and challenges for deployment in low-and middle-income countries.  _The Lancet Regional Health–Western Pacific_ , 41.
  * Wei et al. (2018) Zhongyu Wei, Qianlong Liu, Baolin Peng, Huaixiao Tou, Ting Chen, Xuan-Jing Huang, Kam-Fai Wong, and Xiang Dai. 2018.  Task-oriented dialogue system for automatic diagnosis.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 201–207.
  * Wu et al. (2023) Jiageng Wu, Xian Wu, Zhaopeng Qiu, Minghui Li, Yefeng Zheng, and Jie Yang. 2023.  Qualifying chinese medical licensing examination with knowledge enhanced generative pre-training model.  _arXiv preprint arXiv:2305.10163_.
  * Xu et al. (2023) Kaishuai Xu, Wenjun Hou, Yi Cheng, Jian Wang, and Wenjie Li. 2023.  Medical dialogue generation via dual flow modeling.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 6771–6784, Toronto, Canada. Association for Computational Linguistics.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _arXiv preprint arXiv:2306.05685_.