# Learning from Atypical Behavior: Temporary Interest Aware Recommendation
Based on Reinforcement Learning

Ziwen Du, Ning Yang, Zhonghua Yu, Philip S. Yu  Ziwen Du is with the School of
Computer Science, Sichuan University, China. E-mail<EMAIL_ADDRESS>Ning Yang is the corresponding author and with the School of Computer Science,
Sichuan University, China. E-mail<EMAIL_ADDRESS>Zhonghua Yu is with the
School of Computer Science, Sichuan University, China. E-mail:
<EMAIL_ADDRESS>Philip S. Yu is with the Department of Computer Science,
University of Illinois at Chicago, USA. E-mail<EMAIL_ADDRESS>
###### Abstract

Traditional robust recommendation methods view atypical user-item interactions
as noise and aim to reduce their impact with some kind of noise filtering
technique, which often suffers from two challenges. First, in real world,
atypical interactions may signal users’ temporary interest different from
their general preference. Therefore, simply filtering out the atypical
interactions as noise may be inappropriate and degrade the personalization of
recommendations. Second, it is hard to acquire the temporary interest since
there are no explicit supervision signals to indicate whether an interaction
is atypical or not. To address this challenges, we propose a novel model
called Temporary Interest Aware Recommendation (TIARec), which can distinguish
atypical interactions from normal ones without supervision and capture the
temporary interest as well as the general preference of users. Particularly,
we propose a reinforcement learning framework containing a recommender agent
and an auxiliary classifier agent, which are jointly trained with the
objective of maximizing the cumulative return of the recommendations made by
the recommender agent. During the joint training process, the classifier agent
can judge whether the interaction with an item recommended by the recommender
agent is atypical, and the knowledge about learning temporary interest from
atypical interactions can be transferred to the recommender agent, which makes
the recommender agent able to alone make recommendations that balance the
general preference and temporary interest of users. At last, the experiments
conducted on real world datasets verify the effectiveness of TIARec.

###### Index Terms:

Robust Recommendation, Temporary Interest, Deep Reinforcement Learning

##  1 Introduction

In recent years, recommender systems have received much attention due to their
ability to improve user experience in online applications like Amazon,
Netflix, TikTok, etc. By nature, a recommender system makes recommendations
based on the preference learned from users’ historical interactions with
items. However, in reality there always exist atypical behaviors that seldom
appear before and obviously deviate from the general preference revealed by
users’ usual behaviors. For example, a user who is interested in football may
occasionally watch some basketball-related videos during the finals of NBA,
and when the NBA season is over, her attention to football will come back. The
traditional works often consider the atypical interactions as noise that will
impede the learning of the true preference of users and consequently weaken
the robustness of recommender systems [3].

Recently, researchers have proposed a few models for robust recommendation,
whose main idea is to reduce the impact of atypical interactions with some
kind of noise filtering technique [20, 22, 19, 25, 27]. Simply filtering out
atypical interactions, however, may be inappropriate for recommender systems
in real world. On the one hand, due to the dynamic nature of user preference,
even so called atypical interactions do not have to mean nothing but can also
reveal users’ temporary interest. Consider the above NBA example again.
Although the behaviors of watching basketball-related videos seldom happen
before, they can still be a signal of the user’s temporary interest in
basketball due to NBA. Therefore, simply removing atypical interactions as
noise may degrade the learning of user preference, especially the learning of
user temporary interest. On the other hand, it is hard for the existing
recommendation models to acquire the temporary interest directly from users’
historical interactions, since there are no explicit supervision signals to
indicate atypical interactions [26].

To further illustrate our motivation, we conduct an exploration of three real
datasets Movielens-1M, Beauty, and Tianchi, where the interactions are labeled
in terms of the item category. Figure 1 shows the proportions of the users who
have at least one rare interaction label with frequency less than 5% of the
average, while Figure 1 shows the ratios of the users whose at least one rare
interaction is predicted correctly by the robust recommendation models CDAE
[23] and APR [8] in addition to the proposed model TIARec on each dataset. We
can see that although there are more than 70% users with atypical behaviors in
each dataset, the temporary preferences are discovered for no more than 10% of
them by the existing two robust recommendation models CDAE and APR. Therefore,
we need a new method that is able to distinguish whether an interaction is
atypical or not and capture the temporary interest from atypical interactions.

Figure 1: (a) Statistics of the users with rare interactions in real datasets
Movielens-1M, Beauty, and Tianchi. (b) The recalls of the users with atypical
interactions provided by the two state-of-the-art robust recommendation models
CDAE, APR, and the proposed model TIARec.

To address the above challenges, motivated by the success of the reinforcement
learning (RL) based recommendation models [32, 15, 36, 17, 5, 9, 7], we
propose a novel RL-based model, called Temporary Interest Aware Recommendation
(TIARec), which is able to distinguish atypical interactions from normal ones
without annotations and capture the temporary interest as well as the general
preference. TIARec contains a recommender agent and an auxiliary classifier
agent. The recommender agent recommends an item to a user as its action at
each time step, while the classifier agent can help to identify whether the
interaction with the recommended item is atypical in a trial-error fashion. We
also introduce a critic network and employ an Actor-Critic algorithm [13] to
jointly train the recommender agent and the classifier agent with an objective
to maximize the expected cumulative return of the recommendations made by the
recommender agent. During the joint training, the knowledge about how to
capture the temporary interest can be transferred from the classifier agent to
the recommender agent. Once the training is finished, the recommender agent
can alone make recommendations that balance the general preference and
temporary interest of users.

Our contributions can be summarized as follows:

  * •

We propose a novel RL-based model for robust recommendation, called Temporary
Interest Aware Recommendation (TIARec). To our best knowledge, TIARec is the
first reinforcement learning based solution to capture temporary interests for
robust recommendation.

  * •

We propose a deep reinforcement learning network including a recommender agent
and an auxiliary classifier agent. During their joint training, the auxiliary
classifier agent teaches the recommender agent the knowledge about learning
the temporary interest from atypical interactions.

  * •

Extensive experiments conducted on real world datasets verify the superiority
of TIARec over the state-of-the-art methods on robust recommendation and RL-
based recommendation.

The rest of this paper is organized as follows. In Section 2, we introduce the
preliminaries and formally define the target problem. In Section 3, we first
give an overview of TIARec and then describe its details. We empirically
evaluate the performance of TIARec over real-world datasets in Section 4. At
last, we briefly review the related works in Section 5 and conclude in Section
6.

##  2 Preliminaries

In this section, we first give some basic definitions, then formulate the
recommendations as a Markov Decision Process (MDP), and formally define the
target problem of this paper.

###  2.1 Basic Definitions and Vectorization

Let $\mathcal{U}$ be the user set and $\mathcal{V}$ be the item set. Let
$\mathcal{O}=\\{o_{1}$, $\cdots$, $o_{|\mathcal{O}|}\\}$ be an interaction
sequence, where $o_{i}\in\mathcal{V}$, $1\leq i\leq|\mathcal{O}|$, is the item
of the $i$th interaction. Let $\mathcal{O}_{t}$, $\mathcal{M}_{t}$, and
$\mathcal{N}_{t}$ be the sets of all interactions, normal interactions, and
atypical interactions of some user that happen before time $t$, respectively.
For the same user, $\mathcal{M}_{t}\cap\mathcal{N}_{t}=\Phi$ and
$\mathcal{O}_{t}=\mathcal{M}_{t}\cup\mathcal{N}_{t}$. For vectorizing, we
pretrain a $d$-dimensional vector for each item using Skip-gram algorithm,
where an item $o_{i}$ is treated as a word and a historical interaction
sequence $\mathcal{O}_{t}$ as a sentence. For simplicity, in the rest of this
paper we use the pretrained embeddings to represent the corresponding items,
which are denoted by bold lowercase, for example $\bm{o}_{i}$, and reuse
$\mathcal{V}$ to denote the whole set of the item embeddings.

###  2.2 MDP Formulation

Now we formulate the recommendations as an MDP, which is formally defined as
follows:

  * •

State space $\mathcal{S}$ In recommender systems, the environment of the
agents are defined as the users. Therefore the environment state is defined by
a user’s interaction history. Particularly, a state $s$ is defined as any
possible interaction set $\mathcal{O}$, i.e., the state space
$\mathcal{S}=\\{s=\mathcal{O}\\}$.

  * •

Action space $\mathcal{A}$ An action $\bm{a}$ of a recommender system is
defined as an embedding of a recommended item, and therefore the action space
$\mathcal{A}=\mathcal{V}$. In this paper we assume only one item will be
recommended at each time step.

  * •

Transitions $\mathcal{P}$ The transition $p(s^{\prime}|s,\bm{a})$ is the
conditional probability that the state of the environment (user) changes from
$s\in\mathcal{S}$ to $s^{\prime}\in\mathcal{S}$ after an action
$\bm{a}\in\mathcal{A}$ is issued. In recommender systems,
$p(s^{\prime}|s,\bm{a})$ is 1 if the user interacts with the recommended item,
otherwise 0.

  * •

Reward $\mathcal{R}$ For a recommender, the immediate reward $r(s,\bm{a})$
after issuing action $\bm{a}$ at state $s$ will be defined in terms of users’
feedbacks.

  * •

Discount rate $\gamma$ $\gamma\in(0,1]$ is the discount factor for measuring
long-term rewards.

###  2.3 Problem Formulation

Let $\pi:\mathcal{S}\times\mathcal{V}\mapsto[0,1]$ be a recommender policy
that outputs the conditional probability $p(\bm{a}|s)$ of recommending item
$\bm{a}\in\mathcal{V}$ at state $s\in\mathcal{S}$. Let
$\mathcal{V}^{u}=\\{\bm{v}^{u}_{1},\cdots,\bm{v}^{u}_{T}\\}$ be a sequence of
the collected historical interactions of some user $u$ that really happen
before time step $T$, where $T$ is the number of time steps considered and
$\bm{v}^{u}_{i}\in\mathcal{V}$ ($1\leq i\leq T$) is the item embedding of the
$i$th interaction. Given the dataset $\\{\mathcal{V}^{u}\\}$ over all users,
we want to learn an optimal policy $\pi^{*}$ for the recommender agent that
maximizes the expected cumulative reward of the recommender, i.e.,

$\pi^{*}=\operatorname*{argmax}_{\pi}\mathbb{E}\Big{[}\sum_{t=0}^{T}\gamma^{t}r(s_{t},\bm{a}_{t})\Big{]},$
(1)

where $s_{t}={\mathcal{O}_{t}}\in\mathcal{S}$ and $\bm{a}_{t}\in\mathcal{V}$
are the user state and the recommended item at time step $t$, respectively.

##  3 The Proposed Model

In this section, we first give an overview of the architecture of TIARec, and
then describe its components and learning in detail.

Figure 2: The architecture of TIARec

###  3.1 Architecture of TIARec

As shown in Figure 2, TIARec consists of three parts, a recommender agent
$\Pi_{\text{r}}$, an auxiliary classifier agent $\Pi_{\text{c}}$, and a critic
$Q$. The goal of TIARec is to learn an optimal recommender agent
$\Pi_{\text{r}}$ that realizes the objective defined by Equation (1), with the
help of the classifier agent and the critic.

The agents $\Pi_{\text{r}}$ and $\Pi_{\text{c}}$ are two policy networks. At
each time step $t$, based on the current state
$s^{\text{r}}_{t}=\mathcal{O}_{t}$, the recommender agent recommends an item

$\bm{a}^{\text{r}}_{t}=\Pi_{\text{r}}(s^{\text{r}}_{t};\Theta_{\text{r}})\in\mathcal{V}$
(2)

as the action, where $\bm{a}^{\text{r}}_{t}\in\mathbb{R}^{d\times 1}$ is an
item embedding and $\Theta_{\text{r}}$ represents the trainable parameters of
$\Pi_{\text{r}}$. Then the classifier agent judges how likely the interaction
with $\bm{a}^{\text{r}}_{t}$ is atypical. The state $s^{\text{c}}_{t}$
observed by the classifier agent includes three parts, the action
$\bm{a}^{\text{r}}_{t}$ of the recommender agent, the current normal
interaction set $\mathcal{M}_{t}$ and atypical interaction set
$\mathcal{N}_{t}$. Based on its current state
$s^{\text{c}}_{t}=\\{\bm{a}^{\text{r}}_{t},\mathcal{M}_{t},\mathcal{N}_{t}\\}$,
the classifier agent outputs a scalar

$a^{\text{c}}_{t}=\Pi_{\text{c}}\big{(}s^{\text{c}}_{t};\Theta_{\text{c}}\big{)}\in[0,1],$
(3)

where $\Theta_{\text{c}}$ is the trainable parameters of $\Pi_{\text{c}}$. The
real number $a^{\text{c}}_{t}$ is a probability defining the action of the
classifier agent, which is to recognize the interaction with
$\bm{a}^{\text{r}}_{t}$ as atypical and append it to the atypical interaction
set $\mathcal{N}_{t}$ with probability $a^{\text{c}}_{t}$, or as a normal
interaction and append it to the normal interaction set $\mathcal{M}_{t}$ with
probability $1-a^{\text{c}}_{t}$. Essentially, $\mathcal{N}_{t}$ encodes a
user’s temporary interest while $\mathcal{M}_{t}$ encodes the user’s general
preference.

The critic network $Q$ plays the role of the state-action value function in
Actor-Critic algorithm, which helps the recommender agent and the classifier
agent understand how good the states and the actions are in terms of the
expected cumulative return. With the overall state
$s^{\text{q}}_{t}=\\{\mathcal{O}_{t},\mathcal{M}_{t},\mathcal{N}_{t}\\}$ and
the overall action $\bm{a}^{\text{q}}_{t}=\bm{a}^{\text{r}}_{t}\oplus
a^{\text{c}}_{t}$ as input, where $\oplus$ represents concat operation and
$\bm{a}^{\text{q}}_{t}\in\mathbb{R}^{(d+1)\times 1}$, the critic evaluates the
expected cumulative return $\widehat{q}_{t}$ for both the recommender agent
and the classifier agent,

$\widehat{q}_{t}=Q\big{(}s^{\text{q}}_{t},\bm{a}^{\text{q}}_{t};\Theta_{\text{q}}\big{)},$
(4)

where $\Theta_{\text{q}}$ represents the trainable parameters of the critic.
The overall objective of the training is to learn the optimal
$\Theta_{\text{r}}$, $\Theta_{\text{c}}$, and $\Theta_{\text{q}}$ that
maximize the expected cumulative return $\widehat{q}_{t}$.

The insight here is that the classifier agent $\Pi_{\text{c}}$ can help
capture the temporary interest carried by the atypical interactions it
recognizes, due to the trial-error nature of reinforcement learning. As the
two agents are jointly trained, the knowledge about learning temporary
interest from atypical interactions can be transferred from the classifier
agent to the recommender agent along the back propagation path marked by the
red dashed lines in Figure 2. For example, if the classifier agent correctly
recognizes an interaction as atypical (or mistakes it as a normal one), then
the cumulative return will consequently increases (or reduces), and during the
joint training, the supervision signal offered by the critic network will not
only guide the update of the classifier agent, but also enforce the parameters
of the recommender agent to be adjusted such that the recommender agent can
remember that in such case, it should recommend an appropriate item satisfying
the user’s temporary interest rather than her general preference, so as to
maximize the cumulative return.

###  3.2 Recommender Agent $\Pi_{\text{r}}$

At each time step $t$, the task of the recommender agent
$\Pi_{\text{r}}(s^{\text{r}}_{t};\Theta_{\text{r}})$ is to predict an item
$\bm{a}^{\text{r}}_{t}$ that a given user will interact with next time
according to the current state $s_{t}=\mathcal{O}_{t}$ of that user, which is
implemented as an MLP combining an attention network. Particularly, we first
generate the current preference embedding
$\bm{z}^{\text{o}}_{t}\in\mathbb{R}^{d\times 1}$ based on the user’s current
state with following attention network:

$\begin{split}\bm{z}^{\text{o}}_{t}&=\sum_{i=1}^{|\mathcal{O}_{t}|}\alpha^{\text{o}}_{i}\bm{o}_{i},\\\
\alpha^{\text{o}}_{i}&=\frac{\exp\big{(}\bm{q}^{\text{T}}\sigma(\bm{W}^{\text{o}}\bm{o}_{i}+\bm{b}^{\text{o}})\big{)}}{\sum_{j=1}^{|\mathcal{O}_{t}|}\exp\big{(}\bm{q}^{\text{T}}\sigma(\bm{W}^{\text{o}}\bm{o}_{j}+\bm{b}^{\text{o}})\big{)}}\end{split}$
(5)

where $\bm{o}_{i}\in\mathbb{R}^{d\times 1}$ represents the pretrained
embedding of the $i$th item $o_{i}\in\mathcal{O}_{t}$,
$\alpha^{\text{o}}_{i}\in\mathbb{R}$ is the attention coefficient, and
$\bm{q}\in\mathbb{R}^{d\times 1}$, $\bm{W}^{\text{o}}\in\mathbb{R}^{d\times
d}$, and $\bm{b}^{\text{o}}\in\mathbb{R}^{d\times 1}$ are the trainable query
vector, transformation matrix, and bias terms, respectively, and
$\sigma(\cdot)$ is a non-linear activation function.

Then we feed $\bm{z}^{\text{o}}_{t}$ into an MLP network to generate a
prototype item embedding $\bm{a}^{\text{p}}_{t}\in\mathbb{R}^{d\times 1}$,

$\bm{a}^{\text{p}}_{t}=\text{MLP}_{\text{r}}(\bm{z}^{\text{o}}_{t};\theta_{\text{r}}),$
(6)

where $\theta_{\text{r}}$ represents the trainable parameters of
$\text{MLP}_{\text{r}}$. Now the set of the trainable parameters of the
recommender agent is
$\Theta_{\text{r}}=\\{\theta_{\text{r}},\bm{W}^{\text{o}},\bm{b}^{\text{o}},\bm{q}\\}$.

Note that $\bm{a}^{\text{p}}_{t}$ is an embedding of the prototype item that
may not exist in $\mathcal{V}$ but defines how the embedding of the best
recommended item should look. To generate a real item as the action and set
its immediate reward, we generate a top-$k$ list $\mathcal{V}_{k}$ including
the first $k$ closest item embeddings of $\bm{a}^{\text{p}}_{t}$ in terms of
cosine similarity. The item whose embedding is closest to
$\bm{a}^{\text{p}}_{t}$ is chosen as the action of the recommender agent
during the training,

$\bm{a}^{\text{r}}_{t}=\operatorname*{argmax}_{\bm{v}\in\mathcal{V}}\cos(\bm{a}^{\text{p}}_{t},\bm{v}).$
(7)

where $\cos(\cdot,\cdot)$ is the cosine similarity and
$\bm{v}\in\mathbb{R}^{d\times 1}$ is the pretrained embedding of item $v$.

For offline training, we need to obtain the immediate reward
$R^{\text{r}}_{t}$ of the action $\bm{a}^{\text{r}}_{t}$ at time step $t$. The
challenge here is that the logged data $\\{\mathcal{V}^{u}\\}$ were generated
by an uncontrollable recommender system whose recommending policy is unknown,
or in other words, the historical data are user feedbacks to another unknown
agent different from ours. Therefore, during the offline training, we cannot
directly set $R^{\text{r}}_{t}$ with respect to whether or not the recommended
list $\mathcal{V}_{k}$ covers the logged item that the user actually interacts
with at time step $t$. The existing works often apply off-policy strategy to
fulfill the offline training, by which the target recommending policy can be
updated according to the data generated by a different but known policy [2,
31]. However, the traditional offline training strategy is inapplicable to our
case, since in our case, the policy generating the logged data is unknown.
Inspired by the techniques used in [1, 4, 37, 11], our idea to overcome this
dilemma is to mimic the online interactions of the recommender agent with the
user by inferring how likely the user will interact with an item in the
recommended list based on the logged offline data. In particular, before the
training, we first learn a user latent factor $\bm{u}\in\mathbb{R}^{d\times
1}$ for each user $u\in\mathcal{U}$ and an item latent factor
$\bm{v}\in\mathbb{R}^{d\times 1}$ for each item $v\in\mathcal{V}$, by
performing probabilistic matrix factorization (PMF) over the whole dataset in
advance. Then the probability of user $u$ will interact with item $v$ can be
calculated as

$p(\bm{u},\bm{v})=\sigma(\bm{u}^{\text{T}}\bm{v}),$ (8)

where $\sigma(\cdot)$ is sigmoid function. Next, during the training, we set
the immediate reward $R^{\text{r}}_{t}$ to the inferred probability that user
$u$ will interact with at least one item in the recommended list
$\mathcal{V}_{k}$, i.e.,

$R^{\text{r}}_{t}=\frac{1}{k}\sum_{\bm{v}\in\mathcal{V}_{k}}p(\bm{u},\bm{v}).$
(9)

The insight here is that the larger the probability that user will interact
with the recommended list, the higher the quality of it, and the more reward
the agent deserves.

###  3.3 Classifier Agent $\Pi_{\text{c}}$

The task of the classifier agent
$\Pi_{\text{c}}\big{(}s^{\text{c}}_{t}=\\{\bm{a}^{\text{r}}_{t},\mathcal{M}_{t},\mathcal{N}_{t}\\};\Theta_{\text{c}}\big{)}$
is to determine how likely an interaction with the item
$\bm{a}^{\text{r}}_{t}$ recommended by the recommender agent is atypical,
based on current $\\{\mathcal{M}_{t},\mathcal{N}_{t}\\}$. Similar to the
structure of the recommender agent, the classifier agent is also implemented
as an MLP but combining two attention networks.

At each time step $t$, to capture the temporary interest from the atypical
interactions, the classifier agent first generates a temporary interest
embedding $\bm{z}^{\text{n}}_{t}\in\mathbb{R}^{d\times 1}$ based on the
current atypical interaction set $\mathcal{N}_{t}$, with the following
attention network:

$\begin{split}\bm{z}^{\text{n}}_{t}&=\sum_{i=1}^{|\mathcal{N}_{t}|}\alpha^{\text{n}}_{i}\bm{n}_{i},\\\
\alpha^{\text{n}}_{i}&=\frac{\exp\big{[}(\bm{a}^{\text{r}}_{t})^{\text{T}}\sigma(\bm{W}^{\text{n}}\bm{n}_{i}+\bm{b}^{\text{n}})\big{]}}{\sum_{j=1}^{|\mathcal{N}_{t}|}\exp\big{[}(\bm{a}^{\text{r}}_{t})^{\text{T}}\sigma(\bm{W}^{\text{n}}\bm{n}_{j}+\bm{b}^{\text{n}})\big{]}},\end{split}$
(10)

where $\bm{n}_{i}\in\mathbb{R}^{d\times 1}$ represents the pretrained
embedding of the $i$th item in $\mathcal{N}_{t}$,
$\alpha^{\text{n}}_{i}\in\mathbb{R}$ is the attention coefficient,
$\bm{W}^{\text{n}}\in\mathbb{R}^{d\times d}$ and
$\bm{b}^{\text{n}}\in\mathbb{R}^{d\times 1}$ are the learnable transformation
matrix and bias terms, respectively, and $\bm{a}^{\text{r}}_{t}$ serves as the
query vector. Similarly, the classifier agent also generate a general
preference embedding $\bm{z}^{\text{m}}_{t}\in\mathbb{R}^{d\times 1}$ based on
the current normal interaction set $\mathcal{M}_{t}$,

$\begin{split}\bm{z}^{\text{m}}_{t}&=\sum_{i=1}^{|\mathcal{M}_{t}|}\alpha^{\text{m}}_{i}\bm{m}_{i},\\\
\alpha^{\text{m}}_{i}&=\frac{\exp\big{[}(\bm{a}^{\text{r}}_{t})^{\text{T}}\sigma(\bm{W}^{\text{m}}\bm{m}_{i}+\bm{b}^{\text{m}})\big{]}}{\sum_{j=1}^{|\mathcal{M}_{t}|}\exp\big{[}(\bm{a}^{\text{r}}_{t})^{\text{T}}\sigma(\bm{W}^{\text{m}}\bm{m}_{j}+\bm{b}^{\text{m}})\big{]}}.\end{split}$
(11)

Finally, the classifier agent feeds the concatenation of the embeddings
$\bm{z}^{\text{n}}_{t}$ and $\bm{z}^{\text{m}}_{t}$ into an MLP to generate
the probability $a^{\text{c}}_{t}$ that $\bm{a}^{\text{r}}_{t}$ is an atypical
interaction,

$\begin{split}a^{\text{c}}_{t}=\text{MLP}_{\text{c}}(\bm{z}^{\text{n}}_{t}\oplus\bm{z}^{\text{m}}_{t};\theta_{\text{c}})\in[0,1],\end{split}$
(12)

where $\oplus$ is the concatenation operation and $\theta_{\text{c}}$
represents the trainable parameters of $\text{MLP}_{\text{c}}$. Now we can see
that the trainable parameter set of the classifier agent is
$\Theta_{\text{c}}=\\{\theta_{\text{c}},\bm{W}^{\text{n}},\bm{W}^{\text{m}},\bm{b}^{\text{n}},\bm{b}^{\text{m}}\\}$.

As we mentioned before, the probability $a^{\text{c}}_{t}$ actually defines
two alternative actions for the classifier agent, where one is to recognize
$\bm{a}^{\text{r}}_{t}$ as atypical with probability $a^{\text{c}}_{t}$ and
append it to $\mathcal{N}_{t}$, i.e.,

$\mathcal{N}_{t+1}=\mathcal{N}_{t}\cup\\{\bm{a}^{\text{r}}_{t}\\},$ (13)

and the other one is to recognize $\bm{a}^{\text{r}}_{t}$ as a normal
interaction with probability $1-a^{\text{c}}_{t}$ and append it to
$\mathcal{M}_{t}$, i.e.,

$\mathcal{M}_{t+1}=\mathcal{M}_{t}\cup\\{\bm{a}^{\text{r}}_{t}\\}.$ (14)

As we will see later, at the beginning of a training cycle, $\mathcal{N}_{0}$
and $\mathcal{M}_{0}$ are initialized with empty set, and they will gradually
grow with more actions issued by the recommender agent.

The classifier agent also needs to obtain an immediate reward for its action.
Intuitively, atypical behaviors should be dissimilar to each other and lead to
diversity, otherwise many similar atypical behaviors will not be atypical any
more. Therefore, if $\bm{a}^{\text{r}}_{t}$ is recognized as atypical
appropriately, the overall similarity between the items in $\mathcal{N}_{t}$
should reduce after $\bm{a}^{\text{r}}_{t}$ is merged into $\mathcal{N}_{t}$,
which implies that smaller similarity between $\bm{a}^{\text{r}}_{t}$ and the
items in $\mathcal{N}_{t}$ should receive greater reward. On the contrary, if
$\bm{a}^{\text{r}}_{t}$ is recognized as normal appropriately, the overall
similarity between the items in $\mathcal{M}_{t}$ should rise after
$\bm{a}^{\text{r}}_{t}$ is merged into $\mathcal{M}_{t}$, which implies that
larger similarity between $\bm{a}^{\text{r}}_{t}$ and the items in
$\mathcal{M}_{t}$ is better. Based on this idea, we set the immediate reward
$R^{\text{c}}_{t}$ for the classifier agent as

$R^{\text{c}}_{t}=\begin{cases}1-\frac{1}{|\mathcal{N}_{t}|}\sum_{i}^{|\mathcal{N}_{t}|}\cos(\bm{n}_{i},\bm{a}^{\text{r}}_{t}),\text{
if }a^{\text{r}}_{t}\in\mathcal{N}_{t+1},\\\ \\\
\frac{1}{|\mathcal{M}_{t}|}\sum_{i}^{|\mathcal{M}_{t}|}\cos(\bm{m}_{i},\bm{a}^{\text{r}}_{t}),\text{
if }a^{\text{r}}_{t}\in\mathcal{M}_{t+1}.\end{cases}$ (15)

There are two points here we want to further clarify. First, as there are no
supervision signals, how can the classifier agent learn to recognize atypical
behavior? Our idea here is that atypical behaviors cause dissimilarity, and
the immediate reward given by Equation (15) and the expected cumulative return
estimated by the critic (see next subsection) will judge how well the
classifier agent classifies a recommended item as atypical or normal, and
accordingly adjust its parameters through the policy gradient descent during
the training process. Second, where is the temporary interest? As users’
temporary interest hides in their atypical interactions, we argue that the
atypical interaction set $\mathcal{N}_{t}$ reveals what the classifier agent
considers the temporary interest, and during the joint training, such
knowledge will be transferred to the recommender agent.

###  3.4 Critic $Q$

The critic network
$Q(s^{\text{q}}_{t},\bm{a}^{\text{q}}_{t};\Theta_{\text{q}})$ is responsible
for evaluating the expected cumulative return $\widehat{q}_{t}$ based on the
state $s^{\text{q}}_{t}=\\{\mathcal{O}_{t},\mathcal{M}_{t},\mathcal{N}_{t}\\}$
and action $\bm{a}^{\text{q}}_{t}=\bm{a}^{\text{r}}_{t}\oplus
a^{\text{c}}_{t}$. The critic generates the estimation of the expected
cumulative return through an MLP,

$\widehat{q}_{t}=\text{MLP}_{\text{q}}(\bm{a}^{\text{q}}_{t}\oplus\bm{z}^{\text{o}}_{t}\oplus\bm{z}^{\text{n}}_{t}\oplus\bm{z}^{\text{m}}_{t};\Theta_{\text{q}}),$
(16)

where $\bm{z}^{\text{o}}_{t}$, $\bm{z}^{\text{n}}_{t}$, and
$\bm{z}^{\text{m}}_{t}$ are the preference embeddings of $\mathcal{O}_{t}$,
$\mathcal{M}_{t}$, and $\mathcal{N}_{t}$, respectively, which are produced by
Equations (5), (10), and (11), respectively.

###  3.5 Learning of TIARec

As we have mentioned, the recommender agent is trained jointly with the
classifier agent and the critic network using an actor-critic algorithm. In
particular, we employ MADDPG [13] algorithm to learn the parameters
$\Theta_{\text{r}}$, $\Theta_{\text{c}}$, and $\Theta_{\text{q}}$ of TIARec.

####  3.5.1 Target Networks

The target networks are used to provide the supervision signal for the
training. The target networks
$\Pi^{\prime}_{\text{r}}(s^{\text{r}}_{t+1};\Theta^{\prime}_{\text{r}})$ and
$\Pi^{\prime}_{\text{c}}(s^{\text{c}}_{t+1};\Theta^{\prime}_{\text{c}})$ are a
copy of $\Pi_{\text{r}}$ and $\Pi_{\text{c}}$, with their own parameters
$\Theta^{\prime}_{\text{r}}$ and $\Theta^{\prime}_{\text{c}}$, respectively,
which are used to generate the next actions $\bm{a}^{\text{r}}_{t+1}$ and
$a^{\text{c}}_{t+1}$. Similarly, the target network
$Q^{\prime}(s^{\text{q}}_{t+1},\bm{a}^{\text{q}}_{t+1};\Theta^{\prime}_{\text{q}})$
of the critic $Q$ is a copy of $Q$ with parameters
$\Theta^{\prime}_{\text{q}}$, which is used for generating the target
cumulative return when training $Q$.

1

input : The training dataset $\\{\mathcal{V}^{u}\\}$, learning rate $\lambda$,
fusion coefficient $\tau\in[0,1]$

output : The parameters $\Theta_{\text{r}}$, $\Theta_{\text{c}}$, and
$\Theta_{\text{q}}$

2

3Randomly initialize $\Theta_{\text{r}}$, $\Theta_{\text{c}}$, and
$\Theta_{\text{q}}$;

4 Initialize the target networks:
$\Theta^{\prime}_{\text{r}}=\Theta_{\text{r}}$,
$\Theta^{\prime}_{\text{c}}=\Theta_{\text{c}}$,
$\Theta^{\prime}_{\text{q}}=\Theta_{\text{q}}$;

5

6Initialize the capacity of replay buffer $\mathcal{B}$;

7

8while  _not convergent_ do

9 foreach  _$\mathcal{V}^{u}$_ do

10 Initialize $\mathcal{O}_{0}=\mathcal{M}_{0}=\mathcal{N}_{0}=\\{\\}$;

11 for  _$t\in\\{0,1,\cdots,T\\}$_ do

12 Generate the prototype item $\bm{a}^{\text{p}}_{t}$ and the recommended
item $\bm{a}^{\text{r}}_{t}$ using Equations (6) and (7), respectively;

13 Set recommender reward $R^{\text{r}}_{t}$ using Equation (9);

14 Update $\mathcal{O}_{t+1}=\mathcal{O}_{t}\cup\\{\bm{a}^{\text{r}}_{t}\\}$;

15 Generate classifier action $a^{\text{c}}_{t}$ using Equation (12);

16 Update $\mathcal{N}_{t+1}=\mathcal{N}_{t}\cup\\{\bm{a}^{\text{r}}_{t}\\}$,
$\mathcal{M}_{t+1}=\mathcal{M}_{t}$ with probability $a^{\text{c}}_{t}$; or
$\mathcal{N}_{t+1}=\mathcal{N}_{t}$,
$\mathcal{M}_{t+1}=\mathcal{M}_{t}\cup\\{\bm{a}^{\text{r}}_{t}\\}$ with
probability $1-a^{\text{c}}_{t}$;

17 Set classifier reward $R^{\text{c}}_{t}$ using Equation (15);

18 Set overall reward $r_{t}$ using Equation (18);

19 Set $s_{t}=\\{\mathcal{O}_{t},\mathcal{N}_{t},\mathcal{M}_{t}\\}$ and
$s_{t+1}=\\{\mathcal{O}_{t+1},\mathcal{N}_{t+1},\mathcal{M}_{t+1}\\}$;

20 Save transition sample $(s_{t},\bm{a}^{\text{r}}_{t},r_{t},s_{t+1})$ to
$\mathcal{B}$;

21

22 Sample a minibatch from $\mathcal{B}$;

23 Compute the losses $L_{\text{q}}$, $L_{\text{r,c}}$ based on the minibatch,
using Equations (17) and (20), respectively;

24 Update the critic
$\Theta_{\text{q}}=\Theta_{\text{q}}-\lambda\nabla_{\Theta_{\text{q}}}L_{\text{q}}$;

25 Update the recommender
$\Theta_{\text{r}}=\Theta_{\text{r}}-\lambda\nabla_{\Theta_{\text{r}}}L_{\text{r,c}}$;

26 Update the classifier
$\Theta_{\text{c}}=\Theta_{\text{c}}-\lambda\nabla_{\Theta_{\text{q}}}L_{\text{r,c}}$;

27

28 Update the target networks:
$\Theta^{\prime}_{\text{q}}=\tau\Theta_{\text{q}}+(1-\tau)\Theta^{\prime}_{\text{q}}$;
$\Theta^{\prime}_{\text{r}}=\tau\Theta_{\text{r}}+(1-\tau)\Theta^{\prime}_{\text{r}}$;
$\Theta^{\prime}_{\text{c}}=\tau\Theta_{\text{c}}+(1-\tau)\Theta^{\prime}_{\text{c}}$;

29

30 end for

31

32 end foreach

33

34 end while

35

Algorithm 1 Training of TIARec

####  3.5.2 Loss Function of The Critic

We expect the estimated cumulative return $\widehat{q}_{t}$ provided by the
critic $Q$ should be as close to the target cumulative return $q_{t}$ as
possible. Let $r_{t}$ be the total immediate reward and
$\\{s^{\text{q}}_{t},\bm{a}^{\text{q}}_{t},r_{t},s^{\text{q}}_{t+1}\\}$ be a
training sample, then the loss function of the critic network can be defined
as

$L_{\text{q}}(\Theta_{\text{q}})=\mathbb{E}_{s_{t},\bm{a}^{\text{q}}_{t},r_{t},s_{t+1}}[q_{t}-Q(s^{\text{q}}_{t},\bm{a}^{\text{q}}_{t}=\bm{a}^{\text{r}}_{t}\oplus
a^{\text{c}}_{t};\Theta_{\text{q}})].$ (17)

The total immediate reward $r_{t}$ is defined as

$r_{t}=R^{\text{r}}_{t}+\alpha R^{\text{c}}_{t},$ (18)

where $R^{\text{r}}_{t}$ and $R^{\text{c}}_{t}$ are the immediate rewards of
the recommender agent and the classifier agent, respectively, and $\alpha$ is
a hyper-parameter for balancing the contributions of the immediate rewards.

The target cumulative return $q_{t}$ is defined as the sum of the immediate
reward and the cumulative return of
$\\{s^{\text{q}}_{t+1},\bm{a}^{\text{q}}_{t+1}\\}$,

$q_{t}=r_{t}+\gamma
Q^{\prime}(s^{\text{q}}_{t+1},\bm{a}^{\text{q}}_{t+1};\Theta^{\prime}_{\text{q}}),$
(19)

where $\gamma\in[0,1]$ is the discount factor. The next action
$\bm{a}^{\text{q}}_{t+1}$ $=$
$\bm{a}^{\text{r}}_{t+1}\oplus$$a^{\text{c}}_{t+1}$, where
$\bm{a}^{\text{r}}_{t+1}$ $=$ $\Pi^{\prime}_{\text{r}}(s^{\text{r}}_{t+1};$
$\Theta^{\prime}_{\text{r}})$ and $a^{\text{c}}_{t+1}$ $=$
$\Pi^{\prime}_{\text{c}}(s^{\text{c}}_{t+1};\Theta^{\prime}_{\text{c}})$. As
we will see later, in each iteration of the training, the parameters
$\Theta^{\prime}_{\text{q}}$, $\Theta^{\prime}_{\text{r}}$, and
$\Theta^{\prime}_{\text{c}}$ of the target networks are fixed with the model
parameters $\Theta_{\text{q}}$, $\Theta_{\text{r}}$, and $\Theta_{\text{c}}$
of previous iteration.

####  3.5.3 Loss Function of The Agents

As we have seen before, the recommender agent actually generates a prototype
item embedding $\bm{a}^{\text{p}}_{t}$, while a real item embedding
$\bm{a}^{\text{r}}_{t}$ is fed to the critic. To connect the recommender agent
and the critic for joint training, we need to make them as similar as
possible, and at the same time, both agents seek to maximize the expected
cumulative return $\widehat{q}_{t}$. These ideas lead to the following loss
function of the agents:

$L_{\text{r,c}}(\Theta_{\text{r}},\Theta_{\text{c}})=\mathbb{E}_{s_{t}}\Big{[}(\bm{a}^{\text{r}}_{t}-\bm{a}^{\text{p}}_{t})^{2}-Q\big{(}s^{\text{q}}_{t},\bm{a}^{\text{q}}_{t}=\bm{a}^{\text{p}}_{t}\oplus
a^{\text{c}}_{t}\big{)}\Big{]}.$ (20)

Note that since the first term ensures the recommended item embedding
$\bm{a}^{\text{r}}_{t}$ will be close to the prototype embedding
$\bm{a}^{\text{p}}_{t}$, we use the prototype item $\bm{a}^{\text{p}}_{t}$
instead of $\bm{a}^{\text{r}}_{t}$ to compute the expected cumulative return,
which is slightly different from Equation (17) where we use
$\bm{a}^{\text{r}}_{t}$.

####  3.5.4 Training Algorithm

The training of TIARec uses an offline strategy shown in Algorithm 1, where
each iteration is composed of two stages, the sample generating and the policy
updating. In the sample generating stage of each time step $t$ (lines 8-16),
the recommender agent first recommends an item $\bm{a}^{\text{r}}_{t}$ (line
8), and the classifier agent generates a probability $a^{\text{c}}_{t}$ (line
11) with respect to which the interaction with the recommended item is
classified as noise or normal (line 12). Then we set the overall immediate
reward $r_{t}$ (line 14) by fusing the immediate rewards $R^{\text{r}}_{t}$
(line 9) and $R^{\text{c}}_{t}$ (line 13) of the recommender agent and the
classifier agent. At last, the state is updated (line 15) and the transition
sample is stored to the replay buffer (line 16). In the policy updating stage
(lines 17-22), we first sample a minibatch from the replay buffer (line 17),
and then use gradient descent to update the parameters of the critic, the
recommender agent, the classifier agent, and the target networks (lines
19-22).

TABLE I: Statistics of the datasets | Movielens-1M | Beauty | Tianchi
---|---|---|---
The number of users | 6,040 | 1,210,271 | 6,471
The number of items | 3,952 | 249,274 | 554,442
The number of ratings | 1,000,209 | 2,023,096 | 802,757

##  4 EXPERIMENTS

The goals of the experiments are to answer the following three research
questions:

  * •

RQ1 how does TIARec perform as compared to the state-of-the-art baselines?

  * •

RQ2 how does the classifier agent contribute to the performance of TIARec?

  * •

RQ3 How can the superiority of TIARec be illustrated with an intuitive and
visualizable case study?

  * •

RQ4 How is the robustness of TIARec?

  * •

RQ5 how do the hyper-parameters affect the performance?

The implementation of TIARec is available on https://
github.com/dududu123456/TIARec.

###  4.1 Experimental Setup

####  4.1.1 Datasets

We adopt the following three public datasets for the experiments.

  * •

Movielens-1M111https://grouplens.org/datasets/movielens/ Movielens-1M is a
popular dataset for evaluating recommendation models, which contains over 1
million ratings given by 6,040 users to 3,952 movies.

  * •

Beauty222https://www.kaggle.com/skillsmuggler/amazon-ratings Beauty contains
over 2 million ratings given by more than 1 million users to 249,274 products
on Amazon.

  * •

Tianchi333https://tianchi.aliyun.com/dataset/dataDetail?dataId=64345 Tianchi
contains over 802,757 ratings given by 6,471 users to 554,442 products on
TMALL, a well-known Chinese e-commerce website.

The statistics of the datasets is given in Table I. On each dataset, we use
the first 70% of the data as the training set, the middle 10% as the
validation set, and the remaining 20% as the testing set.

####  4.1.2 Baselines

As we have mentioned, TIARec is a RL-based robust recommendation model that
can capture users’ temporary interest. Therefore, to verify our claimed
contributions, we compare TIARec with seven representative methods including
two robust recommendation models (CDAE and APR), four RL-based models (DEERs,
HRL, KGQR, and PDQ), and one interactive recommendation model (I-CARS), which
are briefly introduced as follows:

  * •

CDAE CDAE [23] is a Denoising Auto-encoder based collaborative filtering
model, which can learn robust latent representations of corrupted user-item
interactions for top-$k$ recommendation.

  * •

APR APR [8] is an Adversarial Personalized Ranking framework, which offers the
recommendation robustness by injecting adversarial noise to the parameters of
a pairwise ranking model BPR [16] during an adversarial training.

  * •

DEERs DEERs [32] is a deep reinforcement learning based recommendation model,
which can simultaneously model positive and negative feedbacks using a Deep
Q-network (DQN) [15] with a pairwise training.

  * •

HRL HRL [26] is a hierarchical reinforcement learning model for
recommendations of online courses, which directly remove atypical interactions
through a hierarchical sequential decision process.

  * •

KGQR KGQR [35] is a knowledge graph enhanced Q-learning model for interactive
recommendations, which incorporates the prior knowledge of the item
correlation learned from knowledge graph into the sequential decision making
to improve sample efficiency for learning better representations of items and
users.

  * •

PDQ PDQ [37] is another deep Q-learning based interactive recommendation
model, where the recommendation policy is trained without the requirement of
real customer interactions, but with a pseudo environment model that simulates
the feedbacks of users.

  * •

I-CARS I-CARS [14] is an interactive context-aware recommendation model, which
can continually update its parameters according to a user’s recent preferences
that are revealed by the user’s feedbacks to the deliberately selected items.

Additionally, in order to verify the effectiveness of the classifier agent of
TIARec, we also compare TIARec with one more baseline method TIARec-C, which
is a variant of TIARec where the classifier agent is removed.

####  4.1.3 Evaluation Protocol

We choose the widely used Hit Rate (HR), Recall, and NDCG to evaluate the
performance of TIARec and the baselines. Let
$\\{v^{u}_{1},\cdots,v^{u}_{N_{u}}\\}$ be the testing set consisting of
$N_{u}$ ground-truth items that user $u$ has interacted with. For each testing
instance $v^{u}_{i}$ ($1\leq i\leq N_{u}$), we generate a ranking list
$\bm{l}^{u}_{i}$ of items using TIARec. Then the number of $u$’s cases that a
ground-truth is ranked in the top-$k$ items can be computed as

$S^{u}_{k}=\sum_{i=1}^{N_{u}}{\mathbb{I}\big{(}rank(v^{u}_{i},\bm{l}^{u}_{i})}\leq
k\big{)},$ (21)

where $rank(v,l)$ is the rank of item $v$ in list $l$, and $\mathbb{I}(x)$ is
the indicator function that returns 1 if $x$ is true, otherwise 0. Then the
metrics can be defined as follows:

${\rm
HR}@k=\frac{1}{|\mathcal{U}|}{\sum_{u\in\mathcal{U}}{\mathbb{I}(S^{u}_{k}\geq
1)}},$ (22) ${\rm
Recall}@k=\frac{1}{|\mathcal{U}|}\sum_{u\in\mathcal{U}}{\frac{S^{u}_{k}}{N_{u}}},$
(23) ${\rm
NDCG}@k=\frac{1}{|\mathcal{U}|}\sum_{u\in\mathcal{U}}{\frac{1}{N_{u}}\sum_{i=1}^{N_{u}}{\frac{\mathbb{I}\big{(}rank(v^{u}_{i},\bm{l}^{u}_{i})\leq
k\big{)}}{\log_{2}{\big{(}1+rank(v^{u}_{i},\bm{l}^{u}_{i})\big{)}}}}}.$ (24)

####  4.1.4 Hyper-parameter Setting

The hyper-parameters are tuned on the validation sets. The embedding
dimensionality $d$ is set to 64 for Movielens-1M and 128 for Beauty and
Tianchi. The reward balance factor $\alpha$ in Equation (18) is set to 0.1 for
Movielens-1M, 0.5 for Beauty and 1.0 for Tianchi. On three datasets, the
discount rate $\gamma$, the size of replay buffer $\mathcal{B}$, and the
fusion coefficient $\tau$ are set to 0.99, 2,000 and 0.01, respectively. For
fairness, the hyper-parameters of the baselines are also set to their optimal
values on the validation sets.

TABLE II: Performance comparison on Movielens-1M

Method Metirc HR@k Recall@k NDCG@k k=5 k=10 k=20 k=5 k=10 k=20 k=5 k=10 k=20
CDAE 0.0554 0.0561 0.3014 0.0134 0.0297 0.0674 0.0003 0.0044 0.0187 APR 0.0536
0.0613 0.3387 0.0121 0.0315 0.0752 0.0034 0.0151 0.0304 DEERs 0.0815 0.1166
0.3651 0.0259 0.0477 0.1134 0.0083 0.0174 0.0381 HRL 0.0734 0.1284 0.3433
0.0299 0.0311 0.0998 0.0087 0.0188 0.0377 KGQR 0.1391 0.2765 0.5029 0.0283
0.0519 0.1378 0.0118 0.0165 0.0545 PDQ 0.0829 0.1619 0.3712 0.0174 0.0418
0.1049 0.0126 0.0196 0.0364 I-CARS 0.0629 0.0929 0.3406 0.0150 0.0356 0.1009
0.0025 0.0089 0.0219 TIARec-C 0.1104 0.1916 0.4128 0.0231 0.0505 0.1043 0.0072
0.0170 0.0325 TIARec 0.1706 0.3110 0.5793 0.0322 0.0583 0.1602 0.0151 0.0274
0.0701 Improv. 22.65% 12.48% 15.19% 7.69% 12.33% 12.26% 19.84% 39.80% 28.62%

TABLE III: Performance comparison on Beauty

Method Metirc HR@k Recall@k NDCG@k k=5 k=10 k=20 k=5 k=10 k=20 k=5 k=10 k=20
CDAE 0.0024 0.0936 0.2431 0.0009 0.0161 0.0339 0.0017 0.0054 0.0151 APR 0.0085
0.1077 0.3668 0.0018 0.0137 0.0548 0.0018 0.0081 0.0216 DEERs 0.1025 0.1902
0.3066 0.0054 0.0217 0.0514 0.0041 0.0077 0.0184 HRL 0.1033 0.1178 0.2433
0.0105 0.0233 0.0531 0.0049 0.0118 0.0197 KGQR 0.1120 0.1714 0.3731 0.0114
0.0183 0.0425 0.0056 0.0152 0.0166 PDQ 0.1328 0.2074 0.4378 0.0160 0.0212
0.0548 0.0073 0.0124 0.0183 I-CARS 0.1044 0.1704 0.3728 0.0104 0.0177 0.0357
0.0021 0.0071 0.0163 TIARec-C 0.1009 0.2027 0.3101 0.0151 0.0229 0.0409 0.0071
0.0130 0.0173 TIARec 0.1652 0.2439 0.4699 0.0231 0.0328 0.0701 0.0104 0.0207
0.0304 Improv. 24.40% 17.60% 7.33% 44.38% 40.77% 27.92% 42.47% 36.18% 40.74%

TABLE IV: Performance comparison on Tianchi

Method Metirc HR@k Recall@k NDCG@k k=5 k=10 k=20 k=5 k=10 k=20 k=5 k=10 k=20
CDAE 0.0034 0.0172 0.1669 0.0011 0.0073 0.0218 0.0003 0.0017 0.0083 APR 0.0079
0.0745 0.1967 0.0028 0.0109 0.0332 0.0007 0.0053 0.0088 DEERs 0.0363 0.0818
0.2446 0.0046 0.0102 0.0366 0.0014 0.0037 0.0114 HRL 0.0847 0.0940 0.2327
0.0092 0.0107 0.0344 0.0039 0.0090 0.0117 KGQR 0.0863 0.1417 0.2632 0.0105
0.0165 0.0405 0.0034 0.0069 0.0131 PDQ 0.0771 0.1255 0.2517 0.0085 0.0120
0.0378 0.0027 0.0036 0.0121 I-CARS 0.0238 0.0515 0.1675 0.0006 0.0060 0.0122
0.0003 0.0029 0.0091 TIARec-C 0.0526 0.1037 0.2199 0.0082 0.0160 0.0392 0.0022
0.0074 0.0112 TIARec 0.0922 0.1515 0.2713 0.0119 0.0194 0.0433 0.0062 0.0104
0.0154 Improv. 8.85% 6.92% 3.08% 13.33% 17.58% 6.91% 58.97% 15.56% 17.56%

Figure 3: Training procedure on all datasets.

Figure 4: The effectiveness of the classifier agent.

###  4.2 Comparative Study (RQ1)

Tables II, III, and IV show the results of the comparison between TIARec and
the baselines on Movielens-1M, Beauty, and Tianchi, respectively, where the
last row shows the percentage by which TIARec improves the performance of the
best baseline.

At first, we can note that on both datasets TIARec outperforms the robust
recommendation models. In contrast to the robust models CDAE and APR, which
rely on some kind of noise filtering technique, TIARec can learn from the
atypical interactions rather than filtering them. Particularly, TIARec can
learn the knowledge about how to capture the users’ temporary interest from
the atypical interactions, and transfer it to the recommender agent from the
classifier agent during the joint training, which makes TIARec more powerful
than the traditional robust recommendation models.

At the same time, we can observe that TIARec achieves significant superiority
over the RL-based baselines, which demonstrates the advantages of the
classifier agent of TIARec. In particular, unlike the traditional RL-based
models, which offers no robustness since they indiscriminately treat every
user-item interaction as normal, TIARec can distinguish atypical interactions
from normal ones with the help of the classifier agent during the joint
training with the recommender agent, which brings two advantages. First, the
differential treatment reduces the impact of the atypical interactions for
learning users’ general preference. Second, the joint training implicitly
teaches the recommender agent how to capture users’ temporary interest. Such
two advantages enable the recommender agent of TIARec to make recommendations
with a balance between users’ general preference and temporary interest, which
contrasts sharply with the traditional RL-based models.

We can also see that TIARec performs better than I-CARS on each dataset.
I-CARS aims at capturing a user’s preference shift according to the user’s
feedbacks to the well-chosen probing questions, which is similar to the target
of TIARec. However, there are two points that tell I-CARS apart from TIARec.
First, I-CARS considers each feedback to be a signal of a user’s true
preference, which makes I-CARS more susceptible to the noise incurred by the
user’s feedbacks to the probing questions which cannot reflect the user’s true
preference. In contrast, the recommender agent of TIARec can learn to discern
whether an atypical behavior represents the user’s true temporary interest or
just a noise, by the joint training with the auxiliary classifier agent.
Second, essentially I-CARS is a greedy algorithm which wants to fit every
observation, while TIARec pursues the maximization of the long-term return due
to the nature of reinforcement learning. For the same reason, almost in each
case all the RL-based models achieve a better performance than I-CARS.
However, we can see that I-CARS still outperforms CDAE and APR in most cases,
due to its sensitivity to user’s preference changes.

At last, Figure 3 illustrates the training process of TIARec in terms of the
average cumulative return estimated by the critic over all the users, which is
defined as:

$\text{Average-Q}=\frac{1}{|\mathcal{U}|}\sum_{u\in\mathcal{U}}\frac{1}{T}{\sum_{t=0}^{T}\widehat{q}_{t}^{u}},$
(25)

where $\widehat{q}_{t}^{u}$ is the expected cumulative return estimated by the
critic for user $u$ at time step $t$ (see Equation (16)). From Figure 3 we can
see that as the number of training epochs increases, the Average-Q first
improves, and then approaches convergence after 50 epochs on Movielens-1M, 200
epochs on Beauty, and 30 epochs on Tianchi. Note that it is reasonable that
the training on Beauty takes longer time before convergence since it has much
greater scale of data.

(a) Preference of the sample user

(b) TIARec

(c) TIARec-C

(d) PDQ

(e) KGQR

(f) DEERs

(g) HRL

(h) CDAE

(i) APR

Figure 5: Case study.

###  4.3 Ablation Study (RQ2)

Now we investigate the effectiveness of the classifier agent of TIARec. At
first we compare TIARec with its variant TIARec-C where the classifier agent
is removed. From Tables II, III, and IV we can observe that TIARec
considerably improves the performance of the variant TIARec-C on three
datasets, which demonstrates the indispensable role of the classifier agent
for the robustness of TIARec. As we have mentioned before, the classifier
agent can identify whether an interaction is atypical during the joint
training with the recommender agent with the objective of maximizing the
expected cumulative return estimated by the critic. In fact, once the
classifier agent is removed, TIARec degenerates to a traditional model which
loses the capability of learning users’ temporary interest from atypical
interactions.

The classifier agent plays its role during the training process through its
immediate reward in Equation (18), where its weight is regularized by the
coefficient $\alpha$. To get further insight into the classifier agent, Figure
4 illustrates how the performance of the recommender agent changes with the
different weights $\alpha$ of the classifier agent on both datasets. At first,
when $\alpha=0$, the classifier agent contributes nothing and TIARec
degenerates to TIARec-C. As $\alpha$ increases, the classifier agent accounts
for more proportion of the total immediate reward and more knowledge about
learning temporary interest from atypical interactions is transferred to the
recommender agent, which leads to the better performance of the latter.
However, when $\alpha$ is greater than 0.1 on Movielens-1M, 0.5 on Beauty, and
1.0 on Tianchi, the curves begin dropping. This is because an overlage
$\alpha$ results in the total immediate reward overwhelmed by the classifier
agent and consequently causes the recommender agent to pays excessive
attention to the users’ temporary interest.

(a) Movielens-1M

(b) Beauty

(c) Tianchi

Figure 6: Robustness over different noise levels.

Figure 7: Tuning of the embedding size $d$.

###  4.4 Case Study (RQ3)

Now we conduct a case study to illustrate TIARec’s ability to capture
temporary interest and make recommendations with balance between the general
preference and temporary interest of a user. For this purpose, we sample one
user with ID ”264” from Movielens-1M. Figure 5(a) uses a word cloud to
visualize the tag distribution of the movies watched by the sample user in
2002, where the size of a tag word is proportional to the frequency that the
sample user watched the movies with that tag. From Figure 5(a) we can see that
in 2002, the sample user generally preferred drama and comedy, and seldom
watched movies of other types. However, in October and November the sample
user watched some war movies for some unknown reason, as marked with a red
circle in Figure 5(a). We randomly choose one war movie the sample user
watched in November as the ground-truth for test, and check the top-20 movies
recommended by TIARec and the baselines, whose tag distributions are shown in
Figures 5(b) to 5(i), respectively.

From Figure 5(b) we can see that the drama movies and war movies account for
the vast majority of the top-20 list recommended by TIARec. In fact, 6 war
movies together with 10 drama movies are recommended in the top-20 list,
including 4 movies of both drama and war, which illustrates that TIARec is
able to capture the sample user’s temporary interest in war movie and can
balance it with her general preference to drama when making recommendations.
We also note that as another general preference, the comedy movies are much
less than the drama ones. We argue that this is because as war movie is more
compatible with drama movie, quite a few war movies are also tagged with
drama. To make an easy tradeoff between the temporary interest and the general
preference of the sample user in this case, TIARec tends to combine war movies
with drama rather than comedy ones.

From Figure 5(c) we can see that without the help of the auxiliary classifier
agent, TIARec-C pays most attention to the sample user’s general preference to
comedy movies and can hardly recommend war movies. Compared with Figure 5(b),
Figure 5(c) shows that it is the classifier agent that offers TIARec the
ability to learn a user’s temporary interest from her atypical behaviors.

Figures 5(d) to 5(i) show that similar to TIARec-C, the sample user’s general
preference to drama or comedy dominates the recommendations made by the
baselines. In fact, none of them recommends more than 2 war movies, which
indicates that the baselines can hardly capture the sample user’s temporary
interest in war movie from the recent watching of the war movies, and
excessively rely on the general preference of users when making
recommendations. Compared with Figure 5(b), such results verify the
superiority of TIARec in the temporal interest learning for robust
recommendation.

###  4.5 Robustness Study (RQ4)

Now we investigate the robustness of TIARec by comparing it with the baseline
methods on the three datasets, in terms of the metric HR@10. For this purpose,
in each testing set, with respect to a given ratio (noise level) we randomly
change some consecutive five interactions to the items of the same category
whose frequency is less than the 20% of the average, to simulate the atypical
behaviors that can be regarded as temporary interest. Figure 6 shows the
results at noise levels 0.1, 0.2, 0.4, and 0.8. From Figure 6 we can see that
on each dataset the performances of all methods decline as noise level rises.
However, TIARec still consistently outperforms the baseline methods at
different noise levels, which again shows the advantage of TIARec in capturing
temporary interest of a user. In particular, it is worthy to note that when
the noise level is 0.8, the interest of almost every user becomes very
unstable and changes quite frequently (every five interactions), which results
in a drastic deterioration of the performances of all the methods. However,
even in such extreme case, the HR@10 of TIARec still remains greater than 0.1
which is remarkably higher than that of the baseline methods.

###  4.6 Hyper-parameter Tuning (RQ5)

At last we tune the embedding size $d$ on validation sets, of which the
results are shown in Figure 7. We can observe that the performance curves
first increase and then decrease when we enlarge the embedding size $d$. The
performance reaches its maximum at $d=64$ on Movielens-1M and at $d=128$ on
both Beauty and Tianchi. A larger $d$ means more model parameters needed to be
learned, which may lead to the problem of overfitting.

##  5 RELATED WORK

In this section, we briefly review two domains of the works mostly related to
our study, including robust recommendation and RL-based recommendation.

###  5.1 Robust Recommendation

The goal of robust recommendation is to reduce the impact of atypical
interactions on user preference learning as much as possible to improve the
robustness of the models. The existing methods for robust recommendation can
be roughly divided into two classes. One class of the methods employs auto-
encoder based techniques to generate robust embeddings for users and items
[23, 10, 18]. For example, Wu et al. propose the model CDAE [23] which first
injects random drop-out noise to training data and trains an denoising auto-
encoder [20] on intentionally corrupted input with the objective of minimizing
reconstruction errors. Li et al. [10] propose a collaborative variational
auto-encoder (CVAE) for robust recommendation of multimedia, and Shenbin et
al. [18] propose a Recommender VAE (RecVAE) model with that can be trained
with corrupted implicit user-item interaction vectors. The other class of the
robust recommendation methods introduces adversarial noise as well as
adversarial training to improve the model robustness [8, 19, 21, 25]. For
example, He et al. propose the model APR [8] which enhances the pairwise
ranking method BPR [16] with an adversarial training. Similarly, Wang et al.
[21], Tang et al. [19], and Yuan et al. [25] propose to add perturbations to
the input data and use adversarial learning to strengthen the robustness of
recommendation model. As we have mentioned, the existing robust
recommendations methods often neglect the meaningful signal carried by the
atypical interactions, from which our TIARec can learn users’ temporary
interest for making better recommendations.

###  5.2 RL-based Recommendation

Recently, RL-based recommendation approaches have attracted more attentions
from scholars. RL-based models aim to learn an optimal strategy to maximize
the long terms rewards. RL-based models can be divided into three categories,
the policy-based methods [4, 33, 31, 1, 12, 2, 30], the value-based
methods[32, 34, 37, 11], and the actor-critic based methods [28, 6, 5, 7, 29,
24]. Chen et al. [1] propose to use a balanced hierarchical clustering tree to
tackle the large action space problem. Chen et al. [2] propose to use an off-
policy framework to solve the data bias problem. Liu et al. [12] model the
interactions between users and items to represent users’ state. Zhao et al.
[32] take the negative feedback and positive feedback into consideration to
represent users’ state. Zheng et al. [34] utilize two Q-network to balance the
correlation between exploit and explore. Zou et al. [37] construct a simulator
to model users’ feedback to handle the selection bias of logged data. Zhou et
al. [35] propose to obtain item representations through a graph convolution
network and obtain users’ state embedding vectors through a recurrent neural
network, and then trains the recommendation strategy with the Q-leaning
algorithm. He et al. [7] and Feng et al. [5] treat each agent as a scenario
and use a multi-agent framework to improve the total performance of all
scenarios. Zhang et al. [28] apply a multi-agent reinforcement learning method
to coauthor network analysis for the dynamic collaboration recommendation.
Zhao et al. [33] propose an actor-critic based reinforcement learning method
for list-wise recommendation and introduce a simulator of online user-agent
interacting environment to get more interactions. Zhao et al. [30] also
propose a multi-agent reinforcement learning model to capture the sequential
correlation among different scenarios. The existing RL-based recommendation
methods equally treat user-item interactions as normal, which offer little
robustness against atypical interactions and cannot capture users’ temporary
interest too.

##  6 CONCLUSIONS

In this paper, we propose a novel reinforcement learning based model for
robust recommendation, called Temporary Interest Aware Recommendation
(TIARec), which can learn users’ temporary interest from their atypical
interactions. TIARec contains a recommender agent and an auxiliary classifier
agent, which are jointly trained with the objective of maximizing the expected
cumulative return of the recommendations made by the recommender agent. In
particular, during the joint training, the classifier agent can judge whether
the interaction with a recommended item is atypical and transfer the knowledge
about the temporary interest learning to the recommender agent, which finally
enable the recommender agent to alone make robust recommendations that balance
the general preference and temporary interest of users. The experiments
conducted on real world datasets show that due to the ability to capture
users’ temporary interest, TIARec achieves a significant improvement in
recommendation performance compared with the state-of-the-art baselines.

## Acknowledgment

This work is supported by National Natural Science Foundation of China under
grant 61972270, and in part by NSF under grants III-1763325, III-1909323,
III-2106758, and SaTC-1930941.

## References

  * [1] H. Chen, X. Dai, H. Cai, W. Zhang, X. Wang, R. Tang, Y. Zhang, and Y. Yu.  Large-scale interactive recommendation with tree-structured policy gradient.  In AAAI, 2019.
  * [2] M. Chen, A. Beutel, P. Covington, S. Jain, F. Belletti, and E. H. Chi.  Top-k off-policy correction for a reinforce recommender system.  In RecSys, 2019.
  * [3] Y. Deldjoo, T. Di Noia, E. Di Sciascio, and F. A. Merra.  How dataset characteristics affect the robustness of collaborative recommendation models.  In SIGIR, 2020.
  * [4] G. Dulacarnold, R. Evans, P. Sunehag, and B. Coppin.  Deep reinforcement learning in large discrete action spaces.  CoRR, 2015.
  * [5] J. Feng, H. Li, M. Huang, S. Liu, W. Ou, Z. Wang, and X. Zhu.  Learning to collaborate: Multi-scenario ranking via multi-agent reinforcement learning.  In WWW, 2018.
  * [6] T. Gui, P. Liu, Q. Zhang, L. Zhu, M. Peng, Y. Zhou, and X. Huang.  Mention recommendation in twitter with cooperative multi-agent reinforcement learning.  In SIGIR, 2019.
  * [7] X. He, B. An, Y. Li, H. Chen, R. Wang, X. Wang, R. Yu, X. Li, and Z. Wang.  Learning to collaborate in multi-module recommendation via multi-agent reinforcement learning without communication.  In RecSys, 2020.
  * [8] X. He, Z. He, X. Du, and T.-S. Chua.  Adversarial personalized ranking for recommendation.  In SIGIR, pages 355–364, 2018.
  * [9] E. Ie, V. Jain, J. Wang, S. Narvekar, R. Agarwal, R. Wu, H.-T. Cheng, T. Chandra, and C. Boutilier.  Slateq: A tractable decomposition for reinforcement learning with recommendation sets.  In IJCAI, 2019.
  * [10] X. Li and J. She.  Collaborative variational autoencoder for recommender systems.  In KDD, 2017.
  * [11] F. Liu, H. Guo, X. Li, R. Tang, Y. Ye, and X. He.  End-to-end deep reinforcement learning based recommendation with supervised embedding.  In WSDM, 2020.
  * [12] F. Liu, R. Tang, X. Li, Y. Ye, H. Chen, H. Guo, and Y. Zhang.  Deep reinforcement learning based recommendation with explicit user-item interactions modeling.  arXiv:1810.12027, 2018.
  * [13] R. Lowe, Y. WU, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  In Advances in Neural Information Processing Systems, 2017.
  * [14] R. Lumbantoruan, X. Zhou, Y. Ren, and L. Chen.  I-cars: an interactive context-aware recommender system.  In ICDM, 2019.
  * [15] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller.  Playing atari with deep reinforcement learning.  arXiv:1312.5602, 2013.
  * [16] S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme.  Bpr: Bayesian personalized ranking from implicit feedback.  In UAI, 2009.
  * [17] W. Shang, Y. Yu, Q. Li, Z. Qin, Y. Meng, and J. Ye.  Environment reconstruction with hidden confounders for reinforcement learning based recommendation.  In KDD, 2019.
  * [18] I. Shenbin, A. Alekseev, E. Tutubalina, V. Malykh, and S. I. Nikolenko.  Recvae: A new variational autoencoder for top-n recommendations with implicit feedback.  In WSDM, 2020.
  * [19] J. Tang, X. Du, X. He, F. Yuan, Q. Tian, and T. Chua.  Adversarial training towards robust multimedia recommender system.  TKDE, 2020.
  * [20] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol.  Extracting and composing robust features with denoising autoencoders.  In ICML, 2008.
  * [21] J. Wang, L. Yu, W. Zhang, Y. Gong, Y. Xu, B. Wang, P. Zhang, and D. Zhang.  Irgan: A minimax game for unifying generative and discriminative information retrieval models.  In SIGIR, 2017.
  * [22] Y. Wu, C. DuBois, A. X. Zheng, and M. Ester.  Collaborative denoising auto-encoders for top-n recommender systems.  In WSDM, 2016.
  * [23] Y. Wu, C. DuBois, A. X. Zheng, and M. Ester.  Collaborative denoising auto-encoders for top-n recommender systems.  In WSDM, 2016.
  * [24] X. Xin, A. Karatzoglou, I. Arapakis, and J. M. Jose.  Self-supervised reinforcement learning for recommender systems.  In SIGIR, 2020.
  * [25] F. Yuan, L. Yao, and B. Benatallah.  Adversarial collaborative neural network for robust recommendation.  In SIGIR, 2019.
  * [26] J. Zhang, B. Hao, B. Chen, C. Li, H. Chen, and J. Sun.  Hierarchical reinforcement learning for course recommendation in moocs.  In AAAI, pages 435–442, 2019.
  * [27] S. Zhang, H. Yin, T. Chen, Q. V. N. Hung, Z. Huang, and L. Cui.  Gcn-based user representation learning for unifying robust recommendation and fraudster detection.  In SIGIR, 2020.
  * [28] Y. Zhang, C. Zhang, and X. Liu.  Dynamic scholarly collaborator recommendation via competitive multi-agent reinforcement learning.  In RecSys, 2017.
  * [29] D. Zhao, L. Zhang, B. Zhang, L. Zheng, Y. Bao, and W. Yan.  Mahrl: Multi-goals abstraction based deep hierarchical reinforcement learning for recommendations.  In SIGIR, 2020.
  * [30] X. Zhao, L. Xia, D. Yin, and J. Tang.  Model-based reinforcement learning for whole-chain recommendations.  arXiv:1902.03987, 2019.
  * [31] X. Zhao, L. Xia, L. Zhang, Z. Ding, D. Yin, and J. Tang.  Deep reinforcement learning for page-wise recommendations.  In RecSys, 2018.
  * [32] X. Zhao, L. Zhang, Z. Ding, L. Xia, J. Tang, and D. Yin.  Recommendations with negative feedback via pairwise deep reinforcement learning.  In KDD, pages 1040–1048, 2018.
  * [33] X. Zhao, L. Zhang, L. Xia, Z. Ding, D. Yin, and J. Tang.  Deep reinforcement learning for list-wise recommendations.  In arXiv:1801.00209, 2017.
  * [34] G. Zheng, F. Zhang, Z. Zheng, Y. Xiang, N. J. Yuan, X. Xie, and Z. Li.  Drn: A deep reinforcement learning framework for news recommendation.  In WWW, 2018.
  * [35] S. Zhou, X. Dai, H. Chen, W. Zhang, K. Ren, R. Tang, X. He, and Y. Yu.  Interactive recommender system via knowledge graph-enhanced reinforcement learning.  In SIGIR, 2020.
  * [36] L. Zou, L. Xia, Z. Ding, J. Song, W. Liu, and D. Yin.  Reinforcement learning to optimize long-term user engagement in recommender systems.  In KDD, 2019.
  * [37] L. Zou, L. Xia, P. Du, Z. Zhang, T. Bai, W. Liu, J.-Y. Nie, and D. Yin.  Pseudo dyna-q: A reinforcement learning framework for interactive recommendation.  In WSDM, 2020.

|  Ziwen Du obtained his bachelor’s degree from the School of Computer
Science, Sichuan University, China, in 2018. He is now pursuing the master’s
degree in the School of Computer Science, Sichuan University, China. His
research interests include data mining and recommender systems.
---|---
|  Ning Yang is an associate professor at Sichuan University, China. He
obtained his PhD degree in Computer Science from Sichuan University in 2010.
His research interests include recommender systems and social media mining.
---|---
|  Zhonghua Yu is an associate professor at Sichuan University, China. He
obtained his PhD degree in Computer Science from Belarusian State University
in 1995. He was a post-doctor of the University of Tokyo from 2001 to 2003,
with the supervision of Junichi Tsujii. His research interests include natural
language processing and machine learning.
---|---
|  Philip S. Yu received the PhD degree in electrical engineering from
Stanford University. He is a distinguished professor in computer science at
the University of Illinois at Chicago and is also the Wexler chair in
information technology. His research interests include big data, data mining,
and social computing. He is a fellow of the ACM and the IEEE.
---|---


# Privacy-preserving medical image analysis

Alexander Ziller
Institute of Diagnostic
and Interventional Radiology
Institute for Artificial Intelligence
and Informatics in Medicine
Technical University of Munich
Munich, Germany
<EMAIL_ADDRESS>
Jonathan Passerat-Palmbach
Department of Computing
Imperial College London
London, United Kingdom
<EMAIL_ADDRESS>
Théo Ryffel
Arkhn
INRIA, ENS, PSL University
Paris, France
<EMAIL_ADDRESS>
Dmitrii Usynin
Department of Computing
Imperial College London
London, United Kingdom
<EMAIL_ADDRESS>
Andrew Trask
University of Oxford
Oxford, United Kingdom
<EMAIL_ADDRESS>
Ionésio Da Lima Costa Junior
Universidade Federal de Campina Grande
Campina Grande, Paraíba, Brazil
<EMAIL_ADDRESS>
Jason Mancuso
Cape Privacy
New York, United States of America
<EMAIL_ADDRESS>
Marcus Makowski
Institute of Diagnostic and
Interventional Radiology
Technical University of Munich
Munich, Germany
<EMAIL_ADDRESS>
Daniel Rueckert
Institute for Artificial Intelligence
and Informatics in Medicine
Technical University of Munich
Munich, Germany
<EMAIL_ADDRESS>
Rickmer Braren
Institute of Diagnostic
and Interventional Radiology
Technical University of Munich
Munich, Germany
<EMAIL_ADDRESS>
Georgios Kaissis
Institute of Diagnostic
and Interventional Radiology
Institute for Artificial Intelligence
and Informatics in Medicine
Technical University of Munich
Munich, Germany
<EMAIL_ADDRESS>

###### Abstract

The utilisation of artificial intelligence in medicine and healthcare has led
to successful clinical applications in several domains. The conflict between
data usage and privacy protection requirements in such systems must be
resolved for optimal results as well as ethical and legal compliance. This
calls for innovative solutions such as privacy-preserving machine learning
(PPML).

We present PriMIA (Privacy-preserving Medical Image Analysis), a software
framework designed for PPML in medical imaging. In a real-life case study we
demonstrate significantly better classification performance of a securely
aggregated federated learning model compared to human experts on unseen
datasets. Furthermore, we show an inference-as-a-service scenario for end-to-
end encrypted diagnosis, where neither the data nor the model are revealed.
Lastly, we empirically evaluate the framework’s security against a gradient-
based model inversion attack and demonstrate that no usable information can be
recovered from the model.

##  1 Introduction

Machine Learning (ML) and Artificial Intelligence (AI) are recent approaches
in biomedical data analysis, yielding promising results. These systems can
assist clinicians to improve performance in tasks such as the early detection
of cancers in medical imaging as shown in several applications [10, 1]. The
most prominent challenge regarding AI systems is fulfilling their demands for
large scale datasets. The collection of such datasets is often only achievable
by multi-institutional or multi-national efforts. Data is typically anonymised
or pseudonymised at the source site and stored at the institution performing
data analysis and algorithm training [16]. The assembly and transmission of
these datasets ethically and legally requires to have measures in place to
protect patient privacy [12]. Moreover, information and control about the
storage, transmission and usage of health data is a central patient right.
Conventional techniques of anonymisation and pseudonymisation have been proven
to not sufficiently protect from attacks against privacy [15, 11].

However, such concerns will likely not prevent increased data collection for
future data-driven algorithms. Therefore, AI and ML methods will require
innovations to sustainably reconcile data utilisiation and privacy protection.
One of these approaches is the development of privacy-preserving machine
learning (PPML). It aims to protect data security, privacy and
confidentiality, while allowing the training and development of algorithms, as
well as research in the setting of limited trust and/or data availability.

##  2 Privacy-preserving medical image analysis

In this work, we present PriMIA (Privacy-preserving Medical Image Analysis), a
free-and-open-source software framework, which extends the PySyft/PyGrid
ecosystem of open source privacy-preserving machine learning tools [13].
PriMIA adds functionality to allow medical imaging-specific applications,
facilitating securely aggregated federated learning and secure inference. A
broad variety of applications, including local and distributed model training,
can be triggered from an accessible command-line interface. Furthermore, we
include functional improvements increasing the framework’s flexibility,
usability, performance and security. These include gradient descent and
federated averaging weighted by the dataset size at each site, local early
stopping as a technique to counteract overfitting and catastrophic forgetting,
hyperparameter optimisation over the whole confederation and the secure
aggregation of dataset means/standard deviations.

Our work is clinically validated in a case study of paediatric chest
radiography classification into one of three classes: normal (no signs of
infection), bacterial pneumonia, viral pneumonia. We utilise a publicly
available imaging dataset [8] for model training and two real-life datasets
for model evaluation against clinical experts and ground-truth data. We aim to
collaboratively train the model on three computation nodes without the
disclosure of patient data in an honest-but-curious scenario. The topology of
our federated learning system is shown in Figure 1(a). The classification
model is then made available for performing remote inference without revealing
either it or the data in plain text (end-to-end-encrypted inference-as-a-
service (Figure 1(b))).

0pt

(a) Overview of the federated learning setup. At the beginning of training
(A), the central server (Model Owner) sends the model to the computation nodes
for training. Until convergence is achieved, the locally trained models are
securely aggregated using secure multi-party computation (SMPC Secure
Aggregation) and redistributed for a next round of training (B). Finally (C),
the central model is updated and can be used for inference.

0pt

(b) Overview of the encrypted inference process. Initially (A) the data owner
and model owner respectively obfuscate the data and algorithm using secret
sharing. Inference is then carried out using secure multi-party computation
(B), upon which the data owner receives an encrypted set of predicted labels,
which only they can decrypt (C).

Figure 1: Details on the training and inference process in PriMIA.

##  3 Experiments, Results & Conclusion

In this work we evaluate the performance of a model trained using securely
aggregated federated learning (federated secure model) against the same model
without secure aggregation (federated non-secure) and a locally trained model.
All models are ResNet-18 [6] architectures pre-trained on the ImageNet dataset
[3]. We compare securely aggregated federated learning to models trained on
locally accumulated data to assess the influence of encryption on model
classification performance. Furthermore, we evaluate the performance of expert
radiologists to obtain a human baseline as a benchmark for our models. Two
separate test sets derived from clinical routine with a total of 497 images
(Test Set 1 N=145, Test Set 2 N=352 images) are used. The ground truth is
obtained based on clinical patient records. The evaluation measures employed
are accuracy, sensitivity/specificity, receiver-operator-characteristic-area-
under-the-curve (ROC-AUC) and the Matthews Correlation Coefficient (MCC) [9].
The latter is also the criterion by which we optimise the model
hyperparameters during training. Model and expert classification results can
be found in Table 1(a). The federated secure model performs on par with the
federated non-secure and locally trained models (Mc-Nemar test p>0.05, Table
1(b)). All models significantly outperform both human experts (Table 1(b)).
Inter-observer/model agreement, evaluated using Cohen’s kappa ($\kappa$) is
moderate between experts, substantial between models and experts and almost
perfect between models (Table 1(c)).

Table 1: Statistical evaluation of model and expert classification performance

(a) Classification performance comparison of the federated secure, non-secure
and locally trained models vs. the experts on the Validation set and on Test
Sets 1 and 2. Sensitivity/Specificity metrics refer to normal/bacterial/viral,
respectively. ROC-AUC: Receiver-operator-characteristic-area-under-the-curve
(class-weighted average), MCC: Matthews Correlation Coefficient.

(b) McNemar test results on Test Set 1. Bold text signifies p$<$0.05

(c) Cohen’s $\kappa$ between models and observers on Test Set 1.

Furthermore, we empirically evaluate the models’ resilience against inversion
attacks [4], whereby a reconstruction of input data features is attempted. We
apply the improved deep leakage from gradients method [5] to re-identify
patients from all three models. For the quantification of the reconstruction
success we use the mean squared error (MSE) and Frechet Inception Distance
(FID) [7]. Both metrics are significantly higher in the setting of securely
aggregated federated learning vs. non-secure federated learning and locally
trained models, indicating the highest resistance to model inversion attacks.
The average$\pm$standard deviation MSE was 2.26$\pm$1.26, 2.62$\pm$1.46,
3.22$\pm$1.41; FID 2112$\pm$1207, 2119$\pm$1173, 2651$\pm$1293 for locally
trained, federated non-secure and federated secure, respectively (both one-way
analysis of variance (ANOVA) p$<$0.001, MSE locally trained vs. federated
secure Student’s t-test p$<$0.001, federated non-secure vs. federated secure
Student’s t-test p=0.03; FID locally trained vs. federated secure and
federated non-secure vs. federated secure both Student’s t-test p=0.03).
Lastly, we implement secure inference using the Function Secret Sharing [2]
protocol expanded and adapted for neural networks [14] and observe a
significant reduction in inference latency vs. the current state-of-the-art
SecureNN [17] especially in the high network latency setting (35% average
latency reduction vs. SecureNN at 10ms latency, 47% at 100ms, both Student’s
t-test p<0.001).

In conclusion, we present a free-and-open-source framework for securely
aggregated federated learning and end-to-end encrypted inference in a medical
context. We showcase a clinically relevant real-life case study and outperform
human experts with a model trained in a federated and secure manner. Further
research and development will enable the larger-scale deployment of our
framework, the validation of our findings on diverse cross-institutional data
and further the widespread utilisation of privacy-preserving machine learning
techniques in healthcare and beyond.

## Broader Impact

Our work’s aim is to develop a technical solution providing objective
guarantees of security and privacy to patients and algorithm developers in
medical data science approaches. Concretely, we propose a framework for
privacy-preserving machine learning on confidential patient data, as well as
the provision of algorithmic diagnosis services under a premise of asset
protection.

Our work advances the utilisation of privacy-enhancing tools in medically
focused artificial intelligence. Such developments are warranted both from an
ethical and a legal perspective: The creation of algorithms which assist
clinical decision-making hinges on their training on diverse, large datasets
to ascertain high performance and fairness and to counteract bias. However,
the massive accumulation, transmission or release of private patient health
data for this purpose removes sovereignty and control over the data from its
owners and thus infringes on patient rights. Concurrently, algorithm owners
wish to protect their models from theft or misuse during inference. Our work
proposes a solution that will assert both: the safeguarding of data privacy
and the protection of algorithms developed for clinical decision support. It
promotes the training of models on larger datasets as well as making these
models more broadly accessible.

## Acknowledgments and Disclosure of Funding

Authors would like to thank Bennett Farkas for creating Figures 1 and 2 as
well as the PriMIA logo, Patrick Cason and Héricles Emanuel for helping with
PyGrid debugging, Matthias Lau for his input, the PySyft and PyGrid
development teams for their foundational work and the OpenMined community for
their scientific input, contributions and discussion.

Georgios Kaissis received funding from the Technical University of Munich,
School of Medicine Clinician Scientist Programme (KKF), project reference H14.
Rickmer Braren received funding from the German Research Foundation,
SPP2177/1. Théo Ryffel received funding from the European Community’s Seventh
Framework Programme (FP7/2007-2013 Grant Agreement no. 339563 – CryptoCloud)
and the French project FUI ANBLIC, and is co-founder of ARKHN. The funders
played no role in the design of the study, the preparation of the manuscript
or the decision to publish.

Authors declare no conflict of interest in relation to the study.

## References

  * Ardila et al. [2019] Diego Ardila, Atilla P. Kiraly, Sujeeth Bharadwaj, Bokyung Choi, Joshua J. Reicher, Lily Peng, Daniel Tse, Mozziyar Etemadi, Wenxing Ye, Greg Corrado, David P. Naidich, and Shravya Shetty.  End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography.  _Nature Medicine_ , 25(6):954–961, May 2019.  doi: 10.1038/s41591-019-0447-x.  URL https://doi.org/10.1038/s41591-019-0447-x.
  * Boyle et al. [2015] Elette Boyle, Niv Gilboa, and Yuval Ishai.  Function secret sharing.  In _Annual international conference on the theory and applications of cryptographic techniques_ , pages 337–367. Springer, 2015.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  ImageNet: A Large-Scale Hierarchical Image Database.  In _CVPR09_ , 2009.
  * Fredrikson et al. [2015] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart.  Model inversion attacks that exploit confidence information and basic countermeasures.  In _Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security - CCS 2015_. ACM Press, 2015.  doi: 10.1145/2810103.2813677.  URL https://doi.org/10.1145/2810103.2813677.
  * Geiping et al. [2020] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller.  Inverting Gradients–How easy is it to break privacy in federated learning?  _arXiv preprint arXiv:2003.14053_ , 2020.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium, 2017.
  * Kermany et al. [2018] Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al.  Identifying medical diagnoses and treatable diseases by image-based deep learning.  _Cell_ , 172(5):1122–1131, 2018.
  * Matthews [1975] Brian W Matthews.  Comparison of the predicted and observed secondary structure of t4 phage lysozyme.  _Biochimica et Biophysica Acta (BBA)-Protein Structure_ , 405(2):442–451, 1975.
  * McKinney et al. [2020] Scott Mayer McKinney, Marcin Sieniek, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg C. Corrado, Ara Darzi, Mozziyar Etemadi, Florencia Garcia-Vicente, Fiona J. Gilbert, Mark Halling-Brown, Demis Hassabis, Sunny Jansen, Alan Karthikesalingam, Christopher J. Kelly, Dominic King, Joseph R. Ledsam, David Melnick, Hormuz Mostofi, Lily Peng, Joshua Jay Reicher, Bernardino Romera-Paredes, Richard Sidebottom, Mustafa Suleyman, Daniel Tse, Kenneth C. Young, Jeffrey De Fauw, and Shravya Shetty.  International evaluation of an AI system for breast cancer screening.  _Nature_ , 577(7788):89–94, January 2020.  doi: 10.1038/s41586-019-1799-6.  URL https://doi.org/10.1038/s41586-019-1799-6.
  * Narayanan and Shmatikov [2008] Arvind Narayanan and Vitaly Shmatikov.  Robust de-anonymization of large sparse datasets.  In _2008 IEEE Symposium on Security and Privacy (sp 2008)_ , pages 111–125. IEEE, 2008.
  * Price and Cohen [2019] W Nicholson Price and I Glenn Cohen.  Privacy in the age of medical big data.  _Nature medicine_ , 25(1):37–43, 2019.
  * Ryffel et al. [2018] Theo Ryffel, Andrew Trask, Morten Dahl, Bobby Wagner, Jason Mancuso, Daniel Rueckert, and Jonathan Passerat-Palmbach.  A generic framework for privacy preserving deep learning.  _arXiv preprint arXiv:1811.04017_ , 2018.
  * Ryffel et al. [2020] Théo Ryffel, David Pointcheval, and Francis Bach.  Ariann: Low-interaction privacy-preserving deep learning via function secret sharing.  _arXiv preprint arXiv:2006.04593_ , 2020.
  * Schwarz et al. [2019] Christopher G Schwarz, Walter K Kremers, Terry M Therneau, Richard R Sharp, Jeffrey L Gunter, Prashanthi Vemuri, Arvin Arani, Anthony J Spychalla, Kejal Kantarci, David S Knopman, et al.  Identification of anonymous mri research participants with face-recognition software.  _New England Journal of Medicine_ , 381(17):1684–1686, 2019.
  * Sheller et al. [2020] Micah J. Sheller, Brandon Edwards, G. Anthony Reina, Jason Martin, Sarthak Pati, Aikaterini Kotrotsou, Mikhail Milchenko, Weilin Xu, Daniel Marcus, Rivka R. Colen, and Spyridon Bakas.  Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data.  _Scientific Reports_ , 10(1), July 2020.  doi: 10.1038/s41598-020-69250-1.  URL https://doi.org/10.1038/s41598-020-69250-1.
  * Wagh et al. [2018] Sameer Wagh, Divya Gupta, and Nishanth Chandran.  Securenn: Efficient and private neural network training.  _IACR Cryptol. ePrint Arch._ , 2018:442, 2018.