# Learning to Ask Appropriate Questions in Conversational Recommendation

Xuhui Ren  1234-5678-9012 The University of Queensland4072<EMAIL_ADDRESS>,  Hongzhi Yin  The University of Queensland4072<EMAIL_ADDRESS>,  Tong Chen
The University of Queensland4072<EMAIL_ADDRESS>,  Hao Wang  Alibaba AI
Lab<EMAIL_ADDRESS>,  Zi Huang  The University of Queensland4072
<EMAIL_ADDRESS>and  Kai Zheng  University of Electronic Science and
Technology of China<EMAIL_ADDRESS>
(2021)

###### Abstract.

Conversational recommender systems (CRSs) have revolutionized the conventional
recommendation paradigm by embracing dialogue agents to dynamically capture
the fine-grained user preference. In a typical conversational recommendation
scenario, a CRS firstly generates questions to let the user clarify her/his
demands and then makes suitable recommendations. Hence, the ability to
generate suitable clarifying questions is the key to timely tracing users’
dynamic preferences and achieving successful recommendations. However,
existing CRSs fall short in asking high-quality questions because: (1) system-
generated responses heavily depends on the performance of the dialogue policy
agent, which has to be trained with huge conversation corpus to cover all
circumstances; and (2) current CRSs cannot fully utilize the learned latent
user profiles for generating appropriate and personalized responses.

To mitigate these issues, we propose the Knowledge-Based Question Generation
System (KBQG), a novel framework for conversational recommendation. Distinct
from previous conversational recommender systems, KBQG models a user’s
preference in a finer granularity by identifying the most relevant relations
from a structured knowledge graph (KG). Conditioned on the varied importance
of different relations, the generated clarifying questions could perform
better in impelling users to provide more details on their preferences.
Finially, accurate recommendations can be generated in fewer conversational
turns. Furthermore, the proposed KBQG outperforms all baselines in our
experiments on two real-world datasets.

Conversational recommender systems; knowledge graph; clarifying question;
preference mining

*Corresponding author and having equal contribution with the first author

††copyright: rightsretained††journalyear: 2021††copyright:
acmcopyright††conference: Proceedings of the 44th International ACM SIGIR
Conference on Research and Development in Information Retrieval; July 11–15,
2021; Virtual Event, Canada††booktitle: Proceedings of the 44th International
ACM SIGIR Conference on Research and Development in Information Retrieval
(SIGIR ’21), July 11–15, 2021, Virtual Event, Canada††price: 15.00††doi:
10.1145/3404835.3462839††isbn: 978-1-4503-8037-9/21/07††ccs: Information
systems Users and interactive retrieval††ccs: Computing methodologies
Knowledge representation and reasoning

##  1\. Introduction

With personalized recommendation services, e-commerce platforms can easily
infer users’ preference and generate personalized recommendations based on
their interactions with the platform (e.g., searching, reviewing, and
purchasing) (Polleti et al., 2020). Despite the great success achieved,
traditional recommender systems are inevitably constrained by its requirement
on the passively collected user feedback. This brings information asymmetry
between users and recommender systems, as the system can only recommend items
based on a user’s history instead of her/his real-time intent whenever the
service is used (Lei et al., 2020a).

Recently, the emerging language-based intelligent assistants such as Apple
Siri and Google Home provide a new dimension in recommendation tasks. It
enables an intelligent assistant to actively interact with users via
conversations, so as to guide users to clarify their intent and find items
that can meet their preferences (Bi et al., 2019). This possibility is
envisioned as a holistic composition of a dialogue agent and a recommender
system, and is termed as Conversational Recommender System (Lei et al., 2020a;
Liu et al., 2020). In a general sense, CRS will proactively query a user with
a series of clarifying questions to collect her/his demand on desired items.
Then, the collected information serves as the preference context at the
current timestamp, which helps generate timely and explainable
recommendations.

Increasing attempts to develop CRSs for e-commerce have been put forward in
recent years, and most of them can be categorized into two types: (1) policy-
based approaches (Sun and Zhang, 2018; Christakopoulou et al., 2016; Ren et
al., 2020; Zhang et al., 2018; Bi et al., 2019); and (2) knowledge graph-based
approaches (Lei et al., 2020b; Zou et al., 2020; Li et al., 2018; Chen et al.,
2019a). Policy-based approaches originated from task-oriented dialogue
systems, where a policy agent is designed to automatically react during the
conversational recommendation (e.g., making a recommendation or querying the
user’s preference on one specific attribute type). Such methods can imitate
the dialogue behaviors covered in the training corpus to generate human-like
responses and complete the recommendation. However, their performance heavily
relies on the labor- and knowledge-intensive process (i.e., acquisition and
annotation) in preparing a comprehensive training dataset. Without enough
well-annotated training corpus, most policy-based CRSs cannot learn an
accurate dialogue action distribution for diversified dialogue state, thus
underpeforming in dialogue generation. On the other hand, KG-based approaches
have simplified the conversational recommendation into a “System Ask, User
Answer” fashion with the help of knowledge graphs. By incorporating path-based
reasoning into the conversation process, these methods can effectively offset
the dependence on large training corpus. Each attribute node on the generated
path will be used to formulate a yes/no question (e.g., “Do you like
pop/rock/country music?”) in order to affirm the user’s current interest and
explore the next node to connect. Finally, an explicit path in the KG will
lead to a suitable item for recommendation. However, such exploration on KG
only concerns the connectivity between nodes and overlooks the semantics
behind users’ responses. Apparently, they are less efficient as a node can be
linked to multiple nodes, and the users may need to answer numerous questions
before the correct item is reached.

Figure 1. An example of clarifying questions for a conversational
recommendation.

Distinct from previous KG-based CRSs, we propose to fully account for the
heterogeneous relations when reasoning over the paths in KG, so as to better
understand the rationale behind a user’s decision making process. As shown in
Figure 1, the user recently has viewed several movies. Specifically, we define
attribute types (e.g., genre) as relations that connect items with different
attribute values (e.g., romantic) With the help of the KG, the main genres of
his watched movie can be categorized into three types: legendary, romantic and
horror. Our proposed KBQG could infer “genre” plays an important role in
helping the user find the desired movie instead of other attribute types like
“actor” and “director”. Relations between interacted items are highly
indicative on a user’s intrinsic preferences, thus being beneficial for
understanding fine-grained user interests. Therefore, our KBQG directly
generates the clarifying questions to affirm the user’s preference on the
dominating attribute types for making an efficient and explainable
recommendation. In light of this, we propose a novel framework named
Knowledge-Based Question Generation system (KBQG) as a solution. Specifically,
KBQG is designed to address the following key challenges in conversational
recommendation tasks:

  1. (1)

What personalized question to ask? Different users may focus on different
aspects when consuming items. When generating clarifying questions, a CRS
should prioritize critical item attribute types that influence the user’s
decision the most. This can make a CRS locate users’ demand and avoid
irrelevant questions with substantially fewer conversation turns. In KBQG, we
devise a KG-based recommender component to infer each user’s taste on
different item attributes (represented via relations in KG) from historical
interaction records. A more important relation means the associated attribute
is more crucial to the user, which should be clarified at an early
conversation stage to help the recommendation.

  2. (2)

Which system action should be performed? Though the performance of the policy
agent is directly associated with the recommendation quality, it is non-
trivial to train an accurate policy agent given the potentially large action
space. In this regard, we simplify our policy agent into a question-based
paradigm, where the actions are either generating a recommendation or further
asking a clarifying question. Before the recommendation module gains enough
confidence, the CRS should continuously ask the user about her/his preference
on different types of item attributes. Once the information is considered
sufficient for an accurate recommendation, the CRS should execute the
recommendation action.

  3. (3)

How to react to users’ responses? The user will give their responses after
each turn of conversation. Hence, it is crucial to make full use of the
contexts within users’ responses to further advance the recommendation
performance. In KBQG, we characterize two kinds of user responses. First, if a
user provides an explicit affirmation on a system-generated clarifying
question, KBQG can effectively identify the user’s demand and store it for
future recommendations. Second, once the user rejects the recommendation
generated by the model, KBQG is able to generate more refined questions in
order to further adjust its recommendation.

To the best of our knowledge, KBQG’s ability to generate personalized
clarifying questions by investigating heterogeneous relations within KG is a
brand new take on conversational recommendation. KBQG produces preference-
aware clarifying questions, which help our system find users’ desired items
more efficiently. To validate the effectiveness of KBQG, we conduct extensive
experiments on two real-world datasets, achieving state-of-the-art
performance. Our contributions are summarized as follows:

  * •

We propose a novel framework KBQG for KG-based conversational recommendation,
which models the user preference as well as the diverse KG relations for
generating personalized, preference-aware clarification questions.

  * •

We reformulate the conversational recommendation scheme and simplify the
dialogue action space, releasing the heavy burden for learning a sophisticated
policy agent.

  * •

By simulating human conversations, we build two KG-based conversational
recommendation datasets that can be a stepstone for future research.
Experimental analysis fully showcases the superiority of the proposed KBQG.

##  2\. Related Work

There are two lines of research inspired our proposed work: knowledge graph-
based recommendation and conversational recommendation. We will discuss the
research work in each relevant field.

###  2.1. Knowledge Graph-based Recommendation

The great success of recommender systems makes them prevalent in e-commerce
platforms to offer recommendations that can meet users’ preferences. To
predict the user preference, many approaches have been proposed based on
collaborative filtering (CF), which assume behaviorally similar users exhibit
similar appetites on items (Wang et al., 2019b, 2020; Chen et al., 2019b; Wang
et al., 2019d; Cui et al., 2019; Lu et al., 2019; Yin and Cui, 2016). However,
CF-based methods usually perform poorly in modelling side information, such as
user profiles and item attributes, thus cannot achieve satisfying performance
when users and items have few interactions (Wang et al., 2019a). In real-world
applications, the various side information, including reviews, relational data
and knowledge graphs (KGs) is important to a model’s interpretability, and is
also closely associated with the efficiency and persuasiveness of recommender
systems (Cao et al., 2019). Incorporating side information into recommender
systems to generate explainable recommendations has attracted much attention
over the last few years, in which KG-based methods show significant
performance advancements as KGs can provide rich facets about items.

Based on how graph-structured information is leveraged for recommendation, KG-
based recommenders can be roughly categorized into three types, namely path-
based (Hu et al., 2018; Sun et al., 2018; Xian et al., 2019; Wang et al.,
2019c), propagation-based (Tang et al., 2019; Wang et al., 2018; Wang et al.,
2019a) and embedding-based (Lin et al., 2015; Ji et al., 2015; Cao et al.,
2019; Zhang et al., 2016; Bordes et al., 2013) methods. Path-based methods
mainly explore various path connectivity patterns among entities in the KG to
provide additional guidance for recommendation. The main effort for path-based
methods is to deal with the vast amount of paths between entities, which is
either solved by devising path selection algorithms (Sun et al., 2018; Wang et
al., 2019b) or by defining meta-path patterns (Hu et al., 2018). Meanwhile,
propagation-based methods concentrate on information propagation over the
whole KG to augment the information for recommendation. It strengthens
entities’ semantic representations via the embeddings of their high-order
neighbours in KG (Wang et al., 2018; Tang et al., 2019). The augmented entity
embeddings serve as the exploitation of user interests or item properties, and
are adopted for modelling user-item interactions (Wang et al., 2019a).
Besides, embedding-based methods mainly project entities and relations of a KG
into low-dimensional continuous vector space, in which the rich structured
knowledge is preserved to enhance the expressiveness of user and item
representations. Despite the great success in KG-based recommenders, few of
them can explain which factor influences a user’s decision. (Xin et al., 2019)
models the user preference in a fine-grained manner and devises an attention-
based neural network to analyze the influence of different relations on the
user decision. However, similar to traditional recommenders, it assumes a
user’s interest is static over the long-time horizon (Lei et al., 2020b),
which hinders the recommender from generating accurate recommendations to
satisfy users’ dynamic preferences.

###  2.2. Conversational Recommendation

Conversational recommender systems are proved to be an effective solution for
explainable and accurate recommendation to meet users’ dynamic demand. This
emerging research topic originates from task-oriented dialogue systems that
help the user find her/his desired information with human-like dialogues (Sun
and Zhang, 2018). Through multi-turn conversations with users, the dialogue
system collects the user’s intent and use a retrieval/content-based
recommendation method to generate recommendations directly. The main challenge
for this kind of method is to train an accurate policy agent to guide the
dialogue action during the conversation. Several approaches have been proposed
in the last three years for training a policy agent for conversational
recommendation (Sun and Zhang, 2018; Christakopoulou et al., 2016; Ren et al.,
2020; Zhang et al., 2018; Bi et al., 2019; Chen et al., 2020). However, the
training process for these approaches usually suffers from the scarcity of
training data. Also, a successful conversational recommendation requires a
policy agent to cope with diverse dialogue actions, which brings even more
difficulties to the training process.

Conditioned on the drawbacks of policy-based approaches, a new trend that
employs KGs as an auxiliary information source for conversational
recommendation has attracted much attention (Lei et al., 2020b; Zou et al.,
2020; Li et al., 2018; Chen et al., 2019a; Zhang et al., 2021). Early work on
KG-based CRSs directly generates the recommendation based on connected
neighbours of the user’s mentioned items in the KG during the conversation
(Chen et al., 2019a; Li et al., 2018). They can provide users with
recommendations and persuasive reasons in a single conversation turn. (Lei et
al., 2020b) applies the KG-based method into a multi-round conversation
scenario. It utilizes the structural information of KG to query the user with
yes/no questions about her/his explicit preference on the specific attribute
value (i.e. entity). Through a series of questions and answers between the
user and the dialogue agent, the CRS finds a recommendation path in KG for the
final recommendation. Even though the affirmative answer from the user could
help CRSs find a promising path in KG for recommendation, the entities in a KG
may have multiple relations with other nodes, hence the user may need to
answer numerous questions before reaching the correct recommendation. This
defect of exisiting KG-based recommenders would significantly weaken and
recommendation efficiency and the user experience.

##  3\. Prelimiaries

Following (Lei et al., 2020b), we subsume our model under the multi-round
conversational recommendation (MCR) paradigm but make some adjustments to
improve its real-world applicability. MCR refers to one trial of
recommendation as a round, and the CRS is free to interact with the user via
attribute-related questions or recommendations multiple times until the task
is finished. In this paper, we point out that asking generic questions can
better impel the user to describe her/his specific demand compared with
querying on explicit attributes. For example, asking “What is your preference
on the genre of the movie?” is always more efficient and natural than blindly
asking “Do you like romantic/horrible/art/…?” in capturing user preferences.
So, we bypass the trial-and-error interaction style of the existing MCR
paradigm, thus significantly shrinking the candidate question pool (i.e.,
action space) and making it more practical for e-commerce applications where a
large amount of attributes are usually involved.

Specifically, an item is associated with a set of attribute types (e.g.,
color) and concrete attributes (e.g., blue) to describe its property. In a KG,
an item is connected to an attribute value via an attribute type, e.g.,
$\textnormal{{Jane Eyre}}\xrightarrow[]{genre}romantic$, where the attribute
type (i.e., genre in this case) is regarded as a relation between these two
entities. An item may have multiple descriptions under a same attribute type,
e.g. the genre of book Harry Potter can be described by adventure, fantasy and
fiction. In a conversational recommendation round, KBQG aims to identify a
user’s most cared about attribute types (i.e., relations), then obtain her/his
fine-grained preferences (i.e., attribute values) towards them by asking
customized questions. The obtained preference information will eventually
facilitate accurate recommendation.

Several flies, spanning two columns of text

Figure 2. The overall diagram of KBQG. KBQG is executed with a three-phase
process: Conversational Policy Decision, KG-based Preference Mining, and
Reaction Decision.

##  4\. Methodology

As shown in Figure 2, our proposed KBQG is executed with a three-phase process
consisting of conversational policy decision, preference mining, and reaction
decision. At the beginning, the process is started by a recommendation request
from a user. Conditioned on the current dialogue context, a policy agent is
responsible for deciding the response action $a_{t}$ in the current
conversation turn $t$, that is, whether to ask a clarifying question or to
give a recommendation. The action $a_{t}$ will be passed onto the preference
mining phase. If the policy agent decides to ask a clarifying question, the
model will infer which relation in the KG is more important to the user from
her/his interaction records, and then leverage the identified relations to
guide the generation of the question. If the policy agent decides to provide a
recommendation, then the preference mining module acts as a recommender to
determine the items to be recommended based on the current dialogue belief
state $\mathbf{b}_{t}$, which is a vector storing the user’s current interests
expressed during the conversation. After the user gives feedback to the
system-generated response, KBQG will decide how it should react to the user.
If the user provides explicit expressions for a clarifying question, KBQG will
keep extracting the user demand and updating the dialogue belief state for
conversational policy decision. Once the user accepts the recommendation, or
the conversation reaches the maximum conversation turn, the conversation will
be terminated; otherwise, if the user rejects the recommendation while the
maximum conversation turn is not reached, our KBQG should return to the
question-asking phase to generate further clarifying questions to amend the
recommendation results. In what follows, we will present the design of each
phase in detail.

###  4.1. Conversational Policy Decision

The policy decision component decides which dialogue action should be
conducted in the current conversation turn. As discussed earlier, existing KG-
based CRSs commonly face the difficulty in training a sophisticated policy
agent to cover diversified dialogue situations. The main reason is that, each
dialogue action is defined as one yes/no question about a specific attribute
value (e.g., “Do you like blue?”), which leads to an unnecessarily large
action space for decision making. It requires to train an estimation module to
estimate the information entropy of each attribute value, deducing which
attribute value should be selected as the next dialogue action. However, such
an estimation module usually cannot obtain satisfying performance when the
dataset contains a large quantity of sparse attributes, impeding both the
prediction accuracy and model generalizability. Furthermore, it is not as
efficient and natural as directly enquiring about the user at the attribute
type level (e.g., “Which color do you like?”).

We address this problem by simplifying the dialogue actions at the first
place. Specifically, KBQG has only two actions, which are either asking
clarifying questions or generating recommendations for the user. The questions
asked by KBQG is fully tailored based on a users’ attention towards different
attribute categories, which is fulfilled by the subsequent preference mining
component so that the policy agent can focus on choosing between two actions
at each conversation turn. Thus, we design a policy agent $\pi(\cdot)$ with
multilayer perception, which is formulated as:

(1)
$\pi_{\theta}(a_{t}|\textbf{s}_{t})=\sigma(\textbf{W}_{1}\cdot\tanh(\textbf{W}_{2}\textbf{s}_{t}+\mathbf{b}_{1})+\mathbf{b}_{2}),$

where $\sigma$ is the sigmoid activation function, and
$\theta=\\{\textbf{W}_{1},\textbf{W}_{2},\mathbf{b}_{1},\mathbf{b}_{2}\\}$
denotes the weights and biases to learn in the policy agent. $\textbf{s}_{t}$
is the internal dialogue state of the dialogue system that represents the
current contexts of the dialogue at turn $t$, and it is defined as:

(2) $\textbf{s}_{t}=\textbf{b}_{t}\oplus\textbf{q}_{t}\oplus\textbf{c}_{t},$

where $\textbf{b}_{t}$ is the dialogue belief state which is a latent encoding
of the user utterance (c.f. Section 4.3), $\textbf{q}_{t}$ is a multi-hot
vector indicating which attribute type has been clarified by the user, and
$\textbf{c}_{t}$ is a vector indicating the ratio of candidate items whose
affinity score with the user meets a predefined threshold $M$. The fewer
candidate items means the recommender system has more confidence about the
recommendation. The policy agent is trained in the RL for searching optimal
conversational policy decisions. We adopt deep Q-learning as suggested in (Lei
et al., 2020b), to allow for easier and faster convergence. Correspondingly, a
reinforcement reward will be given after each conversation turn according to
the user feedback. Intuitively, the policy agent needs to keep querying the
user when: (1) few user preferences are captured (i.e., $\textbf{q}_{t}$
contains too many zeros); or (2) there are too many candidate items (i.e.,
$\textbf{c}_{t}$ represents a large ratio) which need to be narrowed down to
ensure accurate recommendation. Naturally, the policy agent will perform the
recommendation action when sufficient preference signals are collected and the
candidate item pool is small yet dedicated to the information provided. Recall
that we define each attribute type as a relation in our KG, and we design the
reward in KBQG as follows: (1) $r_{t,p}$, if the user provides an explicit
demand on the queried relation; (2) $r_{t,n}$, if the user currently does not
provide any detail on the queried relation; (3) $r_{t,a}$, if the user accepts
the recommendation; and (4) $r_{t,m}$, if the user refuses the recommendation
or the conversation reaches the maximum turn with no successful
recommendation. The RL agent selects one of the RL rewards as the evaluation
reward $r_{t}$ to the current action. The policy agent is train towards
maximizing the cumulated reward during the conversation. Finally, a policy
gradient method is adopted to optimize the policy network via:

(3)
$\bigtriangledown_{\theta}J(\theta)=\bigtriangledown_{\theta}\mathbb{E}[\log\pi_{\theta}(a_{t}|\textbf{s}_{t})R_{t}],$

where $R_{t}=\sum_{t^{\prime}=t}^{T_{max}}\eta^{t^{\prime}}r_{t}$ is the
cumulated reward from turn $t$ to the end of conversation at $T_{max}$, and
$\eta\in(0,1)$ is a damping factor that encourages the model to accomplish the
task in fewer conversation turns.

###  4.2. KG-based Preference Mining

The preference mining module in KBQG has two main functionalities. First, when
guiding the generation of clarifying questions, it infers the most important
relations for the user within the KG, which are used for composing the
questions. Second, when performing the recommendation, it is responsible for
generating personalized recommendations based on the user preference.
Different users tend to focus on different attribute types when shopping
online, e.g., some are more concerned on the price while some have preference
on specific brands. Such user preferences can be captured from users’
historical interactions. To better account for each relation’s impact on the
user’s decision process, we propose a translation-based approach in KBQG for
preference mining.

Conventional translation-based methods model the user preference by learning
an additional translation vector p from a set of latent factors $\mathcal{P}$
for each user-item pair, such that given the representations of user u and
item i, we have $\mathbf{u}+\mathbf{p}\approx\mathbf{i}$ (Bordes et al., 2013)
if $u$ prefers $i$. The translation vector p serves as a unique relation
between users and items, and can be interpreted as the user’s item-specific
preference representation. The optimization objective is to minimize the $L1$
distance $||\mathbf{u}+\mathbf{p}-\mathbf{i}||$ for all positive user-item
pairs. However, such method fails to consider the fine-grained user preference
on attribute types when an interaction is made, hence are less suited for our
goal of picking the most important attribute type for question generation.

Correspondingly, we design our preference mining module based on TransH (Wang
et al., 2014), which assumes each relation has its own hyperplane, and the
preference-driven translation from the user to the item is valid only if they
are projected onto the same hyperplane. Hence, in KBQG, we propose the
hyperplane-based preference translation function as follows:

(4) $f(u,i\mid
p_{u})=||\mathbf{u}^{\perp}+\hat{\mathbf{p}}_{u}-\mathbf{i}^{\perp}||,$

where $\mathbf{u}^{\perp}$ and $\mathbf{i}^{\perp}$ are projected user and
item representations, and $\hat{\textbf{p}}_{u}$ is the user’s preference-
driven translation vector. First, $\mathbf{u}^{\perp}$ and
$\mathbf{i}^{\perp}$ are obtained via:

(5) $\begin{split}\hat{\mathbf{u}}&=\mathbf{u}+\mathbf{b},\\\
\mathbf{u}^{\perp}&=\hat{\mathbf{u}}-\mathbf{w}_{u}^{\mathrm{T}}\hat{\mathbf{u}}\mathbf{w}_{u},\\\
\mathbf{i}^{\perp}&=\mathbf{i}-\mathbf{w}_{u}^{\mathrm{T}}\mathbf{u}\mathbf{w}_{u},\end{split}$

where $\hat{\mathbf{u}}$ is the user embedding enhanced by the dialogue belief
state $\mathbf{b}$, thus encoding the user’s current preference during the
conversation. Noticeably, to make this preference translation more suited to
conversational recommendation, we further incorporate conversational contexts
by augmenting the user embedding with dialogue belief state. $\mathbf{w}_{u}$
is the projection weight. Specifically, they are learned from the user’s
previously interacted items, where relations (i.e., attribute types) are
thoroughly modelled to extend their semantic expressiveness. This also
provides an explicit explanation for user’s potential interactions with items.
First, $\hat{\mathbf{p}}_{u}$ can be computed via:

(6) $\hat{\mathbf{p}}_{u}=\sum_{r\in\mathcal{R}}\alpha_{r,u}\mathbf{r},$

where $\mathbf{r}$ is the representation of each relation in the relation set
$\mathcal{R}$, and $\alpha_{r,u}$ is user $u$’s attention weight on each
specific relation. We define $\alpha_{r,u}$ with the standard softmax
function:

(7) $\alpha_{r,u}=\frac{\exp(a(r,u))}{\sum_{r^{\prime}\in
R}\exp(a(r^{\prime},u))},$

where $a(r,u)$ is the attention score of user $u$ towards relation $r$. We
learn it with an attention network:

(8)
$a(r,u)=\mathbf{h}^{\mathrm{T}}(ReLU(\mathbf{W}_{3}(\mathbf{u}\oplus\mathbf{r})+\mathbf{b}_{3}))$

$\mathbf{W}_{3}$ and $\mathbf{b}_{3}$ are weight matrix and bias vector, and
$\mathbf{h}$ projects the calculated latent representation into a scalar
score. A higher attention score means user $u$ exhibits higher interest on $r$
when searching for the next item to interact with. All relations are ranked in
a descending order with their corresponding attention weights. KBQG
sequentially pick an unused attribute type in each conversation turn for
generating clarifying questions. As such, instead of learning a latent
preference to connect items and users as in (Cao et al., 2019), we incorporate
the rich semantics from the relations in KG to infer the user’s most concerned
attribute types, making it possible to generate appropriate questions for
conversational recommendation.

Similarly, the projection vector $\mathbf{w}_{u}$ can be obtained with the
projection vector $\mathbf{w}_{r}$ on each relation hyperplane:

(9) $\mathbf{w}_{u}=\sum_{r\in R}\beta_{r,u}\mathbf{w}_{r},$

where another set of attention weights $\beta_{r,u}$ are computed, and
$\textbf{w}_{r}$ denotes the relation-specific projection weight. By replacing
r in Eq.6 with $\textbf{w}_{r}$ in a parallel attention network, we can
calculate $\beta_{r,u}$ and $\mathbf{w}_{u}$ analogously.

Thus, the auxiliary information from KG and dialogue content could enhance the
semantic representation and learn the connections between users and items. For
each user $u$, we define Bayesian Personalized Ranking (Rendle et al., 2009)
loss to encourage $u$’s translation distance between all interacted items
$i\in\mathcal{I}_{u}^{+}$ to be smaller than randomly selected negative items
$i^{\prime}\in\mathcal{I}_{u}^{-}$:

(10)
$\mathcal{L}_{f}=\sum_{(u,i)\in\mathcal{H}^{+}}\sum_{(u,i^{{}^{\prime}})\in\mathcal{H}^{-}}-\mathrm{ln}\big{(}\sigma(f(u,i^{\prime}\mid
p_{u})-f(u,i\mid p_{u}))\big{)},$

where $\mathcal{H}^{+}=\\{\mathcal{I}_{u}^{+}\\}_{\forall u}$ and
$\mathcal{H}^{-}=\\{\mathcal{I}_{u}^{-}\\}_{\forall u}$ are all positive and
negative interaction instances used for training. In order to effectively
preserve the relations between entities within the KG, we jointly optimize the
recommendation loss with a relation modelling loss in KG. According to the
assumption of TransH, the translation distance between the head entity $e_{h}$
and the tail entity $e_{t}$ is defined as:

(11)
$f(e_{h},e_{t},r)=||\mathbf{e}^{\perp}_{h}+\mathbf{r}-\mathbf{e}^{\perp}_{t}||$

where the projected embeddings of the head and tail entities can be obtained
via:

(12)
$\begin{split}\mathbf{e}^{\perp}_{h}&=\mathbf{e}_{h}-\mathbf{w}_{r}^{\mathrm{T}}\mathbf{e}_{h}\mathbf{w}_{r}\\\
\mathbf{e}^{\perp}_{t}&=\mathbf{e}_{t}-\mathbf{w}_{r}^{\mathrm{T}}\mathbf{e}_{t}\mathbf{w}_{r}\end{split}$

where $\mathbf{w}_{r}$ is the relation-specific projection vector as in Eq.9.
The training objective of TransH is to maximize the margin between the
observed triples in $\mathcal{KG}^{+}$ and randomly sampled false triples in
$\mathcal{KG}^{-}$, which is defined as the following:

(13)
$\mathcal{L}_{g}=\sum_{(e_{h},e_{t},r)\in\mathcal{KG}}\sum_{(e_{h}^{\prime},e_{t}^{\prime},r^{\prime})\in\mathcal{KG}^{-}}[f(e_{h},e_{t},r)-f(e_{h}^{\prime},e_{t}^{\prime},r^{\prime})].$

The final training objective function can be written as:

(14) $\mathcal{L}=\lambda\mathcal{L}_{f}+(1-\lambda)\mathcal{L}_{g}$

where $\lambda$ is used to trade-off the influence of two parts.

###  4.3. Reaction Decision

After the user gives corresponding feedback on the system-generated action
(i.e. clarifying question or recommendation), KBQG is supposed to react
accordingly based on the information carried by the user feedback. When KBQG
has not collected enough information about the user interest, the user will
respond to the generated questions to clarify her/his demand on specific
attribute types. In this condition, we introduce an entity linking algorithm
proposed by (Ferragina and Scaiella, 2010) to accurately identify the
descriptive phrases that appeared in the users’ replies. These identified
entities (i.e., detailed attribute values) $z_{1},z_{2},\dots,z_{t}$ are
recognized as the user’s current explicit interest on the attribute types
asked, whose embeddings are used for composing the dialogue belief state
$\mathbf{b}_{t}$:

(15) $\mathbf{b}_{t}=\sum_{t^{\prime}=1}^{t}\mathbf{z}_{t^{\prime}},$

When the policy agent decides to make recommendation, KBQG will search in the
database with recommendation score $f(u,i\mid p_{u})$ and output top-$K$ items
(i.e., $K$ items with lowest $f(u,i,p_{u})$ scores) for recommendation. The
user can respond to the recommendation list with acceptance or rejection. If
the user accepts the recommendation, this conversation is considered as a
success; otherwise, our system needs to further collect user interest and
narrow down the recommendation candidates until the maximum of the
conversation turn $T$. Meanwhile, the failed recommendations are stored in a
negative set $\mathcal{S}_{u}^{-}$ and are avoided in the next round of
recommendation. That is, the item candidate set for $u$ is
$\mathcal{S}_{u}^{cand}=\mathcal{S}_{u}^{+}\backslash\mathcal{S}_{u}^{-}$,
where $\mathcal{S}_{u}^{+}$ is the set of candidate items whose predicted
distance to $u$ is smaller than the threshold $M$ described in Section 4.1.
Finally, if the system does not find the user’s desired item until the maximum
conversation turn, we regard it as a failed recommendation attempt.

###  4.4. Training

The whole training process includes two parts: (1) offline training for
preference mining; and (2) online training for conversational policy decision.
The training objective for the first part is to minimize the distance between
the embeddings of a user and her/his preferred items. A lower score of
$f(u,i\mid p)$ indicates higher interest of the user to the item. For each
user, we regard all entities connected to her/his interacted items as the
user’s affirmed preference, and store them in the $b_{t}$ to train the
recommender. A multi-task training method is adopted to optimize both the
recommendation and KG modelling losses as mentioned in Section 4.2.

For online training, we introduce a user simulator to interact with KBQG to
train the policy agent. The user simulator is a simplified version of the one
used in (Lei et al., 2020b). In our case, the user simulator provides the
user-side information based on the user’s historical interactions. For each
conversation session, the user’s current preference is predefined by the
attribute types and values associated with her/his interacted items in KG. The
conversation between KBQG and the user simulator includes two parts. Firstly,
KBQG sequentially asks clarifying questions during the conversation. If the
queried relations match the user’s preference on attribute types, the user
simulator will respond with detailed attribute values. Secondly, when KBQG
provides the recommendation list to the user, the user simulator will accept
the results if the ground-truth item $i$ is present. Otherwise, the
recommendation is rejected.

##  5\. Experiments

We evaluate KBQG on two real-world datasets. Specifically, we aim to answer
the following research questions (RQs):

RQ1: How does our proposed KBQG compare with state-of-the-art conversational
recommendation methods?

RQ2: Are our user preference inference scheme and simplified policy agent
really effective?

RQ3: Can KBQG help comprehend user preferences and provide convincing yet
explainable recommendation with conversations?

###  5.1. Datasets

We conduct experiments on two publicly available datasets from two different
domains, namely MovieLens-1M for movie
recommendation111https://grouplens.org/datasets/movielens/ and DBbook2014 for
book recommendation222http://2014.eswc-conferences.org/important-dates/call-
RecSys.html. For both datasets, we follow (Cao et al., 2019) to transform
users’ explicit ratings into implicit feedback where ratings meeting a
predefined threshold are considered as positive interactions. The rating
threshold is 4 for MovieLens-1M and is 1 for DBbook2014 due to data sparsity.
For each user, we also sample an equal amount of negative items to match the
positive interactions.

To build KGs for both datasets, we map items to DBPedia entities (if
available), and filter out relations that are infeasible for generating
clarifying questions, e.g., non-textual data like URLs. We preprocess both
datasets by filtering out infrequent users and items (i.e. retaining users and
items with at least 10 interactions). Table 1 shows the statistics of two
datasets after processing. There are 6,040 users, 3,240 items and 998,539
interaction records in MovieLens-1M dataset. On average, each user has 165
interaction records. DBbook2014 has 5,576 users, 2,680 items and 65,961
interaction records, leading to an average of 12 interactions per user. The KG
triples used in both datasets are at the same scale, where the subgraph for
MovieLens-1M composes of 159,492 triples with 87,492 entities and 13
relations, while the subgraph for DBbook has 194,710 triples with 8,793
entities and 9 relations.

Table 1. Statistics of MovieLens-1M and DBbook2014 | MovieLens | DBbook2014
---|---|---
Interactions | Users | 6,040 | 5,576
#Items | 3,240 | 2,680
#Ratings | 998,539 | 65,961
#Avg. Interactions | 165 | 12
KG | #Entity | 9,457 | 8,793
#Relation | 13 | 9
#Triple | 159,492 | 194,710

###  5.2. Experimental Settings

####  5.2.1. Training Details

We randomly split each dataset for training, validation and test with the
ratio of $7:2:1$. When making recommendations, we set $K=10$ for generating
the top-$K$ item list. For each dataset, the maximum conversation turn is set
to fit the total number of KG relations plus one for recommendation, i.e.,
$T=14$ for MovieLens-1M and $T=10$ for DBbook2014, respectively. The rationale
is that, if a recommender has already obtained users’ preferences on all
relations, it should start making recommendations as no further information
can be acquired from the user. We respectively set the hyperparameters for
offline training and online training phases. For offline training, the
embedding size is 100, and the batch size is 256. The optimizers for the
recommendation module and KG modelling are Adagrad and Adam with the learning
rate of learning rate of 0.003 and 0.001, respectively. For recommendation, we
also impose an $L_{2}$ regularization weighted by $10^{-4}$. We set the trade-
off coefficient $\lambda$ to 0.5 for MovieLens-1M and 0.7 for DBbook2014
following (Cao et al., 2019). For online training, the policy agent is trained
with deep Q-learning using RMSprop. The specific values of RL rewards are
designed as: $\\{r_{t,p}=0.1,r_{t,n}=-0.1,r_{t,a}=1,r_{t,m}=-0.3\\}$. The deep
Q-learning has the batch size of 128, the experience memory size of 100,000,
and the damping factor $\eta$ of 0.9. After performing grid search in
$\\{0.25,0.5,0.75\\}$, we set the recommendation threshold $M$ in the dialogue
belief state to 0.25 and 0.5 on MovieLens-1M and DBbook2014, respectively.

Considering generating extremely human-like responses is not the main research
task in this paper, we currently design a series of response templates for
constructing different clarifying questions. The selected relations are filled
in the vacant position of the templates to conduct the conversation. We leave
the exploration of language generation in the future work. 333The source code
will be release at https://github.com/XuhuiRen/KBQG.

####  5.2.2. Baselines

Studies on CRS are emerging in recent years, which focus on different
application scenarios. To verify the performance of our proposed KBQG, we
select the state-of-the-art methods that have similar task settings as ours.
The baselines are summarized as follows:

CRM::

This is a policy-based CRS that integrates a task-oriented dialogue system
with a recommender (Sun and Zhang, 2018). The main idea is to involve a semi-
structured user query to represent the dialogue history as well as the user’s
current preference. This query is then used for generating personalized
recommendations.

EAR::

This method is designed in MCR scenario and is equipped with an Estimation-
Action-Reflection framework for CRS (Lei et al., 2020a). It estimates the
importance of each attribute with information entropy and utilizes a policy
agent to select a suitable attribute from in each conversation turn to
formulate a yes/no question, acquiring the user preference information for
recommendation.

SCPR::

This method proposes to incorporate path searching on KG into CRS in MCR
scenario (Lei et al., 2020b). It formulates a series of yes/no questions to
collect the user’s feedback and use these feedback as the guidance search on
KG to find the recommendation.

Although there are other recent CRSs (Chen et al., 2019a; Li et al., 2018; Liu
et al., 2020; Zhou et al., 2020; Ren et al., 2020), they are inapplicable for
comparison due to different task settings. For example, (Li et al., 2018; Zhou
et al., 2020) is designed to recommend a similar recommendation to a specific
prototype without clarifying the user’s real-time preference, and (Liu et al.,
2020; Ren et al., 2020) are more concerned on language understanding and
generation.

####  5.2.3. Evaluation Metrics

We follow (Lei et al., 2020b) to evaluate the performance of each CRS with
success rate at turn $T$ (SR$@T$) and average turn (AT) of conversations.
SR$@T$ is the accumulative task success rate by turn $T$, and we set
$T=T_{max}$ by default for overall performance comparison. AT is the average
number of conversational turns needed for a successful recommendation session.
If a user cannot find his desired item till $T_{max}$, we return $T_{max}$ for
computing SR. A higher score on SR represents better recommendation accuracy,
whereas a lower AT score indicates a model could more efficiently deliver a
successful conversational recommendation using fewer conversations.

###  5.3. Performance Comparison (RQ1)

Table 2 shows the overall performance of our proposed system as well as the
baseline systems. It is obvious that our proposed system achieves the best
performance among all methods on both datasets regarding SR and AT.

Table 2. Performance comparison of all methods on two datasets by SR and AT, where the best performance is boldfaced ($p<0.01$ in one-sample paired $t$-tests when comparing all our results with baselines’ scores). | DBbook2014 | MovieLens-1M
---|---|---
SR | AT | SR | AT
CRM | 0.286 | 8.593 | 0.761 | 8.117
EAR | 0.247 | 9.369 | 0.594 | 10.362
SCPR | 0.275 | 9.255 | 0.659 | 10.099
KBQG | $\mathbf{0.323}$ | $\mathbf{8.156}$ | $\mathbf{0.816}$ | $\mathbf{7.335}$

Compared with KG-based methods EAR and SCPR, CRM and our proposed KBQG exhibit
significant advantages on both DBbook2014 and MovieLens-1M. The main reason
behind this phenomenon is the items on both KGs for two datasets are linked
with a lot of attributes. However, the expansion of the item attributes
requires a huge amount of training set to train an estimation module that can
estimate the information entropy on the whole attribute when generating the
clarifying questions, which is a key component in EAR and SCPR. Without an
accurate entropy estimation on the potential attributes, their system cannot
find the suitable attributes to ask for the user preference, leading to a
terrible performance on task completion and conversion efficiency. What’s
more, the clarifying mode for EAR and SCPR collects the user preference with a
series of yes/no questions on the specific attributes. It inherently cannot
achieve a satisfying efficiency when there is a large attribute pool to
describe the items. Our proposed KBQG and CRM clarify the user preference with
a set of generic questions. The user can describe his preference on each
attribute type with the guidance of clarifying questions, achieving a better
conversational recommendation experience.

Compared with policy-based method CRM, our proposed KBQG still presents
remarkable improvement. The reason is that the policy agent of CRM mainly
relies on the dialogue action distribution learned from the training corpus to
generate the next clarifying question. It lacks the ability to analyze the
importance weight of different relations on the user’s decision. Therefore, it
usually requires more user preference and more conversation turn for the
recommendation; and the ultimate process of CRM is decomposed to a traditional
recommender system Factorization Machine (Sun and Zhang, 2018) to recommend
the items. Hence, it fails to utilize the connection information on KG to
enhance the entity embedding.

Figure 3. Cumulative success rate of compared methods at different
conversation turns on DBbook2014 (left) and MovieLens-1M (right).

Several flies, spanning two columns of text

Figure 4. Sample conversations generated by KBQG (left) and SCPR (right) and
attention scores of different relations (middle).

Figure 3 presents a finer-grained performance comparison of the task
cumulative success rate on two datasets. From the figure, we can observe that
our proposed KBQG could reach a higher task success rate at the early stage of
the conversation. The reason is that our proposed KBQG could inference which
relation is more important for the user to make a decision, and give priority
to the user preference on the relations that occupy a high attention weight
for generating the recommendation. It can filter many unimportant relations
during the clarifying period to accelerate the user preference collecting
process. Also, the recommendation performance in our system is benefit from
the relational modelling with TransH, and achieves better performance compared
with a traditional recommender system.

###  5.4. Analysis on Key Designs (RQ2)

The key design in our proposed KBQG is the simplification and optimization of
the conventional policy-based CRS. The policy agent in our proposed KBQG only
accounts for deducing a dialogue action, asking a clarifying question or
presenting the recommendation, which means the dialogue action space is
greatly decreased. Hence, it can achieve superior performance on the dialogue
action generation. KBQG could model the user decision policy from the rich
resource of historical interactions, providing an effective way to infer which
relation is more appropriate for formulating the next clarifying question. To
verify its effectiveness, we conduct ablation experiments by designing two
variants of our KBQG, marked with KBQG-P and KBQG-A. KBQG-P replaces the
current policy agent with a standard policy agent as in (Sun and Zhang, 2018)
that has $R+1$ dialogue actions. It decides the next dialogue action based the
distribution of dialogue actions in the training corpus. The policy agent
behaves similarly compared with other policy-based methods in each
conversation turn when selecting the next relation and deciding
asking/recommending. All other components in KBQG-P are inherited from KBQG.
KBQG-A maintains the policy agent in KBQG, but replaces the attentive relation
aggregation with average pooling, hence all relations share the same
possibility of being selected to formulate the clarifying question.

Figure 5. Cumulative success rate of compared methods at different
conversation turns on DBbook2014 (left) and MovieLens-1M (right).

As can be observed from Figure 5, our proposed KBQG still keeps the best
performance over these two variants. The performance drop on each variant
proves the effectiveness of our proposed method. KBQG-P shows inferior
performance at the early stage of the conversation. This is because KBQG-P
needs to train a policy agent to simulate the dialogue distribution, which is
easily influenced by the quality of the training corpus, especially when the
training corpus is synthesized by a simulated user. Therefore, it cannot
distinguish which relation is more important to the user decision, and can
only deduce the next dialogue action conditioned on the current dialogue
belief state. KBQG-A randomly selects relations to formulate clarifying
questions during the conversation, as each relation share the same attention
weight for influencing the user decision. This schema may let the system ask
clarifying questions about the unimportant relations, which would cripple the
model efficiency.

In addition, we further analyze the influence of joint learning parameter
$\lambda$ in Section 4.2 to the overall model performance. By default, we set
$\lambda$ to 0.5 on MovieLens-1M and 0.7 on DBbook2014. We alternate the value
of $\lambda$ of KBQG, where KBQG-L1 uses $\lambda=0.3$ on both datasets, and
KBQG-L2 uses $\lambda=0.7$ on MovieLense, and $\lambda=0.5$ on DBbook2014. As
we can tell from Figure 5, KBQG-L1 and KBQG-L2 perform worse than our default
setting. KBQG-L1 performs worse on DBbook2014 but obtains better performance
on MovieLens-1M compared with KBQG-L2. The main reason is that MovieLens-1M
contains more interaction records for the recommender to capture implicit user
preferences. Putting a higher weight on the relation modelling could enhance
the entity embeddings to improve the recommendation performance. Whereas
DBbook2014 exhibits fewer historical records for modelling the user
preference, it is more suitable to lay more emphasis on the recommendation
loss, which would benefit the overall recommendation performance. Both the KG
relation modelling and recommendation parts simultaneously take effects on the
eventual recommendation quality, and $\lambda$ should be adjusted according to
the property of each dataset for optimal recommendation performance.

###  5.5. Qualitative Analysis (RQ3)

We further conduct qualitative analysis to show how KBQG is able to improve
its understanding of the user preferences and generate appropriate questions
for explainable recommendations. We randomly select one real conversation
record from KBQG and SCPR on MovieLens based on the same test instance. SCPR
is a representative of graph-based approaches that designs its action space
based on all attribute values. The conversation is initiated by the user (ID
4607) requesting for a movie recommendation.

Figure 4 demonstrates the conversations between the user and KBQG/SCPR. It
also visualizes the user’s attention weights on different relations learned by
KBQG. Clearly, KBQG asks clarifying questions following the rank of attention
weights on different relations to collect user preference information. From
the conversations, we can find KG-based method that relies on the user’s
feedback to find the recommendation path is substantially less efficient when
the dataset has a large attribute pool. Hence, SCPR requires a longer
conversation with the user to find a suitable recommendation. In contrast, our
proposed KBQG is designed to generate the conversation in a totally different
fashion. It substantially uses the recommender system to rank the attention
weight of different relations, formulating the most appropriate clarifying
questions for obtaining a user’s preferences. It is well-suited to the large
KG dataset. With the help of attention scores learned on different relations,
KBQG is able to filter unnecessary clarifying questions to improve the
conversation efficiency.

##  6\. Conclusion

In this paper, we redefine the conversational recommender system and propose a
novel KG-based conversational recommender system, KBQG, where the recommender
system and the dialogue system closely cooperate with each other so as to
efficiently and accurately generate recommendations in a short conversation.
Specifically, the preference mining module in KBQG mainly extracts rich
auxiliary information from the KG to explicitly explore users preferences from
historical records. Conditioned on the explored preference, KBQG can
effectively tailor the clarifying questions by priortizing attribute types
that are important to the user. After multiple conversation turns,
personalized recommendations will be given when the user has sufficiently
clarified her/his real-time interests. With extensive experiments on two real-
world datasets, KBQG presents superior performance compared with state-of-the-
art policy-based and knowledge-based CRSs.

###### Acknowledgements.

This work was supported by ARC Discovery Project (Grant No. DP190101985) and
ARC Training Centre for Information Resilience (Grant No. IC200100022).

## References

  * (1)
  * Bi et al. (2019) Keping Bi, Qingyao Ai, Yongfeng Zhang, and W. Bruce Croft. 2019\.  Conversational Product Search Based on Negative Feedback. In _CIKM’19_. 359–368.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013\.  Translating Embeddings for Modeling Multi-relational Data. In _NIPS_ , Vol. 26. 2787–2795.
  * Cao et al. (2019) Yixin Cao, Xiang Wang, Xiangnan He, Zikun Hu, and Tat-Seng Chua. 2019.  Unifying Knowledge Graph Learning and Recommendation: Towards a Better Understanding of User Preferences. In _WWW_. 151–161.
  * Chen et al. (2019a) Qibin Chen, Junyang Lin, Yichang Zhang, Ming Ding, Yukuo Cen, Hongxia Yang, and Jie Tang. 2019a.  Towards Knowledge-Based Recommender Dialog System. In _EMNLP-IJCNLP_. 1803–1813.
  * Chen et al. (2019b) Tong Chen, Hongzhi Yin, Hongxu Chen, Rui Yan, Quoc Viet Hung Nguyen, and Xue Li. 2019b.  AIR: Attentional Intention-Aware Recommender Systems. In _ICDE_. 304–315.
  * Chen et al. (2020) Tong Chen, Hongzhi Yin, Guanhua Ye, Zi Huang, Yang Wang, and Meng Wang. 2020\.  Try This Instead: Personalized and Interpretable Substitute Recommendation _(SIGIR ’20)_. 891–900.
  * Christakopoulou et al. (2016) Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. 2016\.  Towards Conversational Recommender Systems _(KDD ’16)_. 815–824.
  * Cui et al. (2019) Hui Cui, Lei Zhu, Jingjing Li, Yang Yang, and Liqiang Nie. 2019.  Scalable deep hashing for large-scale social image retrieval.  _IEEE Transactions on image processing_ 29 (2019), 1271–1284.
  * Ferragina and Scaiella (2010) Paolo Ferragina and Ugo Scaiella. 2010.  TAGME: On-the-Fly Annotation of Short Text Fragments (by Wikipedia Entities) _(CIKM ’10)_. 1625–1628.
  * Hu et al. (2018) Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S. Yu. 2018\.  Leveraging Meta-Path Based Context for Top- N Recommendation with A Neural Co-Attention Model _(KDD ’18)_. 1531–1540.
  * Ji et al. (2015) Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015.  Knowledge Graph Embedding via Dynamic Mapping Matrix. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing_. 687–696.
  * Lei et al. (2020a) Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Chua. 2020a.  Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems _(WSDM ’20)_. 304–312.
  * Lei et al. (2020b) Wenqiang Lei, Gangyi Zhang, Xiangnan He, Yisong Miao, Xiang Wang, Liang Chen, and Tat-Seng Chua. 2020b.  Interactive Path Reasoning on Graph for Conversational Recommendation. In _KDD’2020_.
  * Li et al. (2018) Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018.  Towards Deep Conversational Recommendations _(NIPS ’18)_. 9725–9735.
  * Lin et al. (2015) Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015.  Learning Entity and Relation Embeddings for Knowledge Graph Completion _(AAAI’15)_. 2181–2187.
  * Liu et al. (2020) Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, and Ting Liu. 2020\.  Towards Conversational Recommendation over Multi-Type Dialogs. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 1036–1049.
  * Lu et al. (2019) Xu Lu, Lei Zhu, Zhiyong Cheng, Liqiang Nie, and Huaxiang Zhang. 2019.  Online Multi-modal Hashing with Dynamic Query-adaption. In _SIGIR_. 715–724.
  * Polleti et al. (2020) Gustavo Padilha Polleti, Hugo Neri Munhoz, and Fabio Gagliardi Cozman. 2020.  Explanations within Conversational Recommendation Systems: Improving Coverage through Knowledge Graph Embeddings. In _AAAI’2020_.
  * Ren et al. (2020) Xuhui Ren, Hongzhi Yin, Tong Chen, Hao Wang, Nguyen Quoc Viet Hung, Zi Huang, and Xiangliang Zhang. 2020.  CRSAL: Conversational Recommender Systems with Adversarial Learning.  _ACM Trans. Inf. Syst._ 38, 4, Article 34 (June 2020), 40 pages.
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009.  BPR: Bayesian Personalized Ranking from Implicit Feedback.  _UAI_ (2009).
  * Sun and Zhang (2018) Yueming Sun and Yi Zhang. 2018.  Conversational Recommender System. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_ _(SIGIR ’18)_. 235–244.
  * Sun et al. (2018) Zhu Sun, Jie Yang, Jie Zhang, Alessandro Bozzon, Long-Kai Huang, and Chi Xu. 2018\.  Recurrent Knowledge Graph Embedding for Effective Recommendation _(RecSys ’18)_. 297–305.
  * Tang et al. (2019) Xiaoli Tang, Tengyun Wang, Haizhi Yang, and Hengjie Song. 2019\.  AKUPM: Attention-Enhanced Knowledge-Aware User Preference Model for Recommendation _(KDD ’19)_. 1891–1899.
  * Wang et al. (2018) Hongwei Wang, Fuzheng Zhang, Jialin Wang, Miao Zhao, Wenjie Li, Xing Xie, and Minyi Guo. 2018.  RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems _(CIKM ’18)_. 417–426.
  * Wang et al. (2019a) Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019a.  KGAT: Knowledge Graph Attention Network for Recommendation _(KDD ’19)_. 950–958.
  * Wang et al. (2019b) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019b.  Neural Graph Collaborative Filtering _(SIGIR’19)_. 165–174.
  * Wang et al. (2019c) Xiang Wang, Dingxian Wang, Canran Xu, Xiangnan He, Yixin Cao, and Tat-Seng Chua. 2019c.  Explainable Reasoning over Knowledge Graph Paths for Recommendation _(AAAI’19)_. 5329–5336.
  * Wang et al. (2019d) Yuandong Wang, Hongzhi Yin, Hongxu Chen, Tianyu Wo, Jie Xu, and Kai Zheng. 2019d.  Origin-Destination Matrix Prediction via Graph Convolution: A New Perspective of Passenger Demand Modeling _(KDD ’19)_. 1227–1235.
  * Wang et al. (2020) Ze Wang, Guangyan Lin, Huobin Tan, Qinghong Chen, and Xiyang Liu. 2020.  CKAN: Collaborative Knowledge-Aware Attentive Network for Recommender Systems _(SIGIR ’20)_. 219–228.
  * Wang et al. (2014) Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014\.  Knowledge Graph Embedding by Translating on Hyperplanes _(AAAI’14)_. 1112–1119.
  * Xian et al. (2019) Yikun Xian, Zuohui Fu, S. Muthukrishnan, Gerard de Melo, and Yongfeng Zhang. 2019.  Reinforcement Knowledge Graph Reasoning for Explainable Recommendation _(SIGIR’19)_. 285–294.
  * Xin et al. (2019) Xin Xin, Xiangnan He, Yongfeng Zhang, Yongdong Zhang, and Joemon Jose. 2019.  Relational Collaborative Filtering: Modeling Multiple Item Relations for Recommendation _(SIGIR’19)_. 125–134.
  * Yin and Cui (2016) Hongzhi Yin and Bin Cui. 2016.  _Spatio-temporal recommendation in social media_.  Springer.
  * Zhang et al. (2021) Chen Zhang, Hao Wang, Feijun Jian, and Hongzhi Yin. 2021\.  Adapting to Context-Aware Knowledge in Natural Conversation for Multi-Turn Response Selection _(WWW)_.
  * Zhang et al. (2016) Fuzheng Zhang, Nicholas Jing Yuan, Defu Lian, Xing Xie, and Wei-Ying Ma. 2016\.  Collaborative Knowledge Base Embedding for Recommender Systems _(KDD ’16)_. 353–362.
  * Zhang et al. (2018) Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W. Bruce Croft. 2018.  Towards Conversational Search and Recommendation: System Ask, User Respond _(CIKM ’18)_. 177–186.
  * Zhou et al. (2020) Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020\.  Improving Conversational Recommender Systems via Knowledge Graph Based Semantic Fusion _(KDD)_. 1006–1014.
  * Zou et al. (2020) Jie Zou, Yifan Chen, and Evangelos Kanoulas. 2020.  Towards Question-Based Recommender Systems _(SIGIR ’20)_. 881–890.


# Optimizing Latent Graph Representations of Surgical Scenes for Zero-Shot
Domain Transfer

Siddhant Satyanaik Aditya Murali111Corresponding author<EMAIL_ADDRESS>Deepak Alapatt Xin Wang Pietro Mascagni Nicolas Padoy ICube, University of
Strasbourg, CNRS, France  IHU Strasbourg, France  Fondazione Policlinico
Universitario A. Gemelli IRCCS, Rome, Italy  West China Hospital of Sichuan
University, Chengdu, China

###### Abstract

Purpose: Advances in deep learning have resulted in effective models for
surgical video analysis; however, these models often fail to generalize across
medical centers due to domain shift caused by variations in surgical workflow,
camera setups, and patient demographics. Recently, object-centric learning has
emerged as a promising approach for improved surgical scene understanding,
capturing and disentangling visual and semantic properties of surgical tools
and anatomy to improve downstream task performance. In this work, we conduct a
multi-centric performance benchmark of object-centric approaches, focusing on
Critical View of Safety assessment in laparoscopic cholecystectomy, then
propose an improved approach for unseen domain generalization.

Methods: We evaluate four object-centric approaches for domain generalization,
establishing baseline performance. Next, leveraging the disentangled nature of
object-centric representations, we dissect one of these methods through a
series of ablations (e.g. ignoring either visual or semantic features for
downstream classification). Finally, based on the results of these ablations,
we develop an optimized method specifically tailored for domain
generalization, LG-DG, that includes a novel disentanglement loss function.

Results: Our optimized approach, LG-DG, achieves an improvement of 9.28% over
the best baseline approach. More broadly, we show that object-centric
approaches are highly effective for domain generalization thanks to their
modular approach to representation learning.

Conclusion: We investigate the use of object-centric methods for unseen domain
generalization, identify method-agnostic factors critical for performance, and
present an optimized approach that substantially outperforms existing methods.

Keywords: Surgical Video Analysis, Domain Adaptation, Object-Centric Learning,
Graph Neural Networks

Fig. 1: Qualitative examples to illustrate the visual domain gap between
Endoscapes2023 and Endoscapes-WC70. The images differ in aspect ratio, color
distribution, and field of view that could be caused by variations in the
laparoscope used and in surgical workflow.

##  1 Introduction

Surgical video analysis is a rapidly growing field that aims to mine critical
information from unstructured surgical videos at scale, which can then be used
to enhance various aspects of surgical practice. Over the past decade, many
works have harnessed advances in deep learning to achieve proficiency in tasks
like automated phase recognition [20], tool and anatomy segmentation [4, 16],
and fine-grained action recognition [17, 6]. Yet, practical deployment of
these models requires effective domain generalization, or in other words,
generalization to distinct hospital environments characterized by variations
in surgical workflow, instrumentation, camera setups, and patient
demographics.

A straightforward approach to this problem is to simply build training
datasets comprising annotated videos from numerous surgical centers; however,
this is often impractical due to multifaceted difficulties and privacy
concerns regarding data collection and sharing [8]. Moreover, even if these
concerns can be overcome, annotating these datasets and ensuring consistency
across centers is a critical bottleneck, particularly for fine-grained tasks
like semantic segmentation and action recognition. As a result, many works in
medical and surgical computer vision have approached multicentric
generalization as a domain adaptation problem, aiming to adapt models trained
for one center (source domain) to other centers (target domains) [19, 21, 10,
22]. In this context, the diverse aforementioned data and annotation
availability scenarios can be broadly divided into three different problem
settings: (1) Fully Supervised/Supervised Domain Adaptation (SDA), where data
and annotations are available for the source and target domain, (2)
Unsupervised Domain Adaptation (UDA), where data from the target domain is
available but not annotations, and (3) Domain Generalization (DG), where no
information about the target domain is known.

While several works have focused on UDA, the DG setting is relatively under-
explored; nevertheless, it is especially relevant in the surgical domain,
where collecting and sharing data is still extremely challenging.
Consequently, in this work, we focus on the latter, specifically aiming to
improve performance by leveraging object-centric classification approaches.
Object-centric methods learn image representations that are strongly
conditioned on the objects in the scene, often by including an object
detection step prior to classification. These representations can then be
finetuned for different downstream tasks, enabling improved performance. Our
key observation is that such methods could be highly effective for domain
generalization because they explicitly enforce downstream predictions to be
conditioned on detected objects rather than extraneous factors like lighting
or background artifacts.

To explore this hypothesis, we begin by thoroughly benchmarking domain
adaptation performance of four different object-centric methods, starting with
the fully supervised and SDA settings to establish ceiling performances, then
moving to the DG setting. We focus on Critical View of Safety (CVS)
prediction, for two reasons: (1) it is a challenging fine-grained task that
has been successfully tackled using object-centric methods [9, 13, 12], and
(2) exploration of multicentric generalization in the context of such fine-
grained tasks is limited, raising the need for further investigation. To
conduct a multicentric analysis, we employ the Endoscapes2023 dataset [11],
collected in Strasbourg, France, and additionally introduce Endoscapes-WC70, a
dataset of 70 laparoscopic cholecystectomy videos collected in Sichuan, China
annotated with CVS criteria and segmentation masks. Finally, we propose an
object-centric approach for improved domain generalization, LG-DomainGen (LG-
DG), that extends the best single-center method, LG-CVS [13], with a
disentanglement loss function to learn more robust representations and thereby
improve domain generalization.

In our experiments, we find that object-centric models generally outperform
non-object-centric classifiers for domain generalization, and pinpoint
critical attributes across methodologies for effective domain generalization.
Our proposed approach incorporates these various principles into a single
method, thereby outperforming existing approaches.

Our contributions can be summarized as follows:

  1. 1.

We study 4 object-centric methods in the context of domain generalization,
focusing on Critical View of Safety prediction.

  2. 2.

We propose an improved latent graph-based object-centric approach, LG-DG, that
substantially outperforms existing approaches for domain generalization.

  3. 3.

We introduce the Endoscapes-WC70 dataset, which comprises 7,690 images from 70
videos annotated with CVS criteria, of which 510 are additionally annotated
with segmentation masks.

Fig. 2: An illustration of a generic Object-Centric method.

##  2 Related Work

###  2.1 Domain Generalization

Annotating surgical data is extremely expensive due to the need for expert
knowledge. To alleviate this burden, the research community has actively
explored various domain adaptation methodologies, such as unsupervised domain
adaptation (UDA) [19, 21], where target data but not labels are used to refine
source domain-trained models, and semi-supervised domain adaptation (SSDA)
[10, 1], where some of the target data also contains labels. A related
research area is federated learning (FL), which restricts data sharing among
domains, but allows decentralized model training using all available data.
Federated learning has been explored extensively in the medical imaging
community [18], and more recently for surgical phase recognition [8].

Unlike UDA, SSDA, and FL, Domain Generalization (DG) eliminates the need for
target domain data altogether. DG methods aim to build a single model that can
generalize to unseen target domains. Popular approaches include model
ensembling, data augmentation/generation to enhance the training set with
estimations of out-of-domain data, and most commonly, representation learning-
based approaches that seek to learn domain-invariant representations,
generally through custom model or loss function design. [3, 2]

Our work focuses on the under-explored area of Domain Generalization in
surgical computer vision, falling under the umbrella of Representation
Learning-based DG. Unlike previous works, we investigate the use of object-
centric representations, which are already disentangled feature
representations, and additionally propose an auxiliary learning objective to
further boost feature disentanglement, thereby improving generalization.

###  2.2 Object-Centric Learning

Object-centric learning focuses on learning scene representations in terms of
the present objects, then using these representations for downstream tasks. It
has been gaining prominence in the surgical domain, having been applied for
activity recognition [5, 14] and phase recognition [7, 12], action triplet
recognition [17], scene captioning [15], and most relevantly, CVS prediction
[13, 12].

While these works focus on solving specific downstream tasks, we are instead
interested in leveraging object-centric learning for improved domain
generalization.

##  3 Methods

In this section, we begin by detailing the datasets used in our multicentric
study and describe the CVS prediction downstream task. Then, we describe the
four object-centric approaches that we investigate. Finally, we describe our
methodology for optimizing LG-CVS [13], a state-of-the-art object-centric
method for CVS prediction, to develop our proposed approach: LG-DomainGen, or
LG-DG. This consists of preliminary ablations to understand elements of
object-centric methods that aid domain generalization, followed by the
development of a disentanglement loss function to boost latent graph
robustness, and, as a result, domain generalization performance.

###  3.1 Datasets and Downstream Task

The Critical View of Safety, or CVS, is a measure to assess dissection quality
and exposure of key anatomical structures; proper achievement of the CVS is
strongly associated with reduced adverse outcomes, such as bile duct injury.
CVS consists of three different binary criteria, detailed in [11], making CVS
prediction a multi-label classification problem.

To enable a multi-centric study, we use two datasets: (1) Endoscapes2023,
introduced in [11] and collected in Strasbourg, France, and (2) Endoscapes-
WC70, which we introduce here, collected in Sichuan, China. Endoscapes2023
contains 58585 frames from 201 laparoscopic cholecystectomy videos; we use two
subsets of Endoscapes2023: Endoscapes-CVS201, which contains 11090 frames
annotated with CVS, and Endoscapes-Seg201, which contains segmentation masks
for 1933 of the previous 11090 frames. We adopt the official dataset splits,
using 120 videos for training, 41 for validation, and 40 for testing.
Meanwhile, we construct Endoscapes-WC70 by collecting 70 laparoscopic
cholecystectomy videos from the West China Hospital, Sichuan University,
Sichuan, China; following the protocol of Endoscapes2023 [11], we annotate CVS
at 5 second intervals and segmentation masks at 30 second intervals. Finally,
we split the 70 videos into 40 training, 15 validation, and 15 test videos,
applying stratified sampling based on video-level CVS achievement as done in
[13]. Table 1 shows the per-criterion CVS achievement rates for the two
datasets. Endoscapes-WC70 is particularly imbalanced with respect to CVS
achievement; we discuss the impact of this imbalance in our results.

Table 1: Frame-Level Achievement Rates (%) of each CVS Criterion. Criterion | Endoscapes2023 | Endoscapes-WC70
---|---|---
Train | Val | Test | Train | Val | Test
C1: Two Structures | 15.6 | 16.3 | 24.0 | 2.1 | 2.0 | 3.2
C2: HCT Dissection | 11.2 | 12.5 | 17.1 | 7.1 | 7.3 | 11.8
C3: Cystic Plate | 17.9 | 16.7 | 27.1 | 8.4 | 9.0 | 14.9
Fig. 3: Three different examples of the masked latent graph
$\hat{G}_{\text{CVS}}$ that is passed to the downstream classification head
$\phi_{\text{CVS}}$ for CVS prediction. Each node in the graph corresponds to
an object in the image. The masking operation, while pictured for a single
node, is applied to all nodes.

###  3.2 Baseline Models

We evaluate 4 different object-centric classification methods: LG-CVS [13],
DeepCVS [9], LayoutCVS (an ablation of DeepCVS introduced in [13]), and
ResNet50-DetInit (also introduced in [13]); to ensure fair comparisons, we
train a single Mask-RCNN object detector to be used by all the methods. We
additionally evaluate a vanilla ResNet50 model to isolate improvements in
generalization brought by object-centric modeling. We briefly describe each
approach below:

ResNet50: We finetune an ImageNet-pretrained ResNet50 classifier for CVS
prediction, requiring no additional labels in the source domain. This baseline
allows us to study whether object-centric approaches as a whole can enable
better generalization.

DeepCVS [9]: DeepCVS is a two-stage model that consists of first segmenting an
image, then concatenating the predicted segmentation mask with the original
image and processing the result with a convolutional neural network to predict
CVS. In doing so, the downstream CVS predictions are based on both semantic
information (identity and location of various anatomical structures, encoded
as a segmentation mask), and visual information (from the original image). For
fair comparisons, we use the adapted and better performing version of DeepCVS
presented in [13].

LayoutCVS: LayoutCVS is identical to DeepCVS except that it does not
concatenate the original image with the output of the object detector.
Including this model allows us to examine the varying robustness of semantic
and visual features to shifts in input domain for downstream classification,
especially through comparisons with DeepCVS.

ResNet50-DetInit: We initialize a ResNet50 classifier using trained object
detection model weights, then finetune for CVS prediction, thus utilizing the
fine-grained scene information encoded implicitly in the object detector’s
learned visual features. This baseline allows us to isolate the impact of
visual information, but unlike the ResNet50 baseline, the visual information
is sufficiently fine-grained for effective CVS prediction.

LG-CVS: A two-stage approach for object-centric classification: in the first
stage, an encoder $\Phi_{\text{LG}}$ encodes an image $I$ as a latent graph
$G$, wherein nodes represent objects (tools and anatomical structures) and
edges capture 2D geometric relationships between objects. These nodes and
edges contain object/relation-specific visual features as well as semantic
features (bounding box coordinates and class probabilities). The latent graph
$G$ is then passed to a GNN-based classification head $\phi_{\text{CVS}}$ for
downstream CVS prediction, and an auxiliary decoder $\phi_{\mathcal{R}}$ that
reconstructs the original image given $G$ and a backgroundized version of $I$
(foreground regions replaced with noise). This auxiliary reconstruction
objective helps learn more discriminative object features, improving
downstream performance.

###  3.3 Investigating Feature Disentanglement in the Latent Graph

A key facet of LG-CVS is its disentanglement of object-level semantic and
visual properties, encoded in the latent graph nodes, and image-level visual
properties, captured by the backbone feature map. While these properties
contribute synergistically in the single-center setting [13], we note that
this may not be the case for domain generalization, where errors in individual
modes of information can derail model predictions.

To study this effect, we conduct a series of ablations where we mask one or
more information categories from the latent graph for both training and
testing before passing to the downstream classification head
$\phi_{\text{CVS}}$ and the auxiliary reconstruction head
$\phi_{\mathcal{R}}$. Figure 3 illustrates the most extreme ablation settings,
while Table 2 lists the various latent graph configurations that we evaluate.

CVS Head Ablations. In this first study, we experiment with different masking
combinations, but only apply the masking before the downstream head, leaving
the input to the reconstruction head intact. By doing so, we aim to identify
whether (1) certain feature categories are more domain-invariant then others
and (2) the different feature categories contribute synergistically when the
model is exposed to domain shift. Concretely:

$G=\Phi_{\text{LG}}(I);\ \mathcal{M}(G,c)=\hat{G}_{\text{CVS}};\ $

$\hat{y}=\phi_{\text{CVS}}(\hat{G}_{\text{CVS}});\
\hat{I}=\phi_{\mathcal{R}}(G),$ (1)

where $\mathcal{M}$ is a masking function that replaces the original values
with Gaussian noise, $c$ is the feature category to mask,
$\hat{G_{\text{CVS}}}$ is the masked latent graph, and $\hat{y}$ is the final
CVS prediction.

Reconstruction Head Ablations. We follow the same process as above, but leave
the input to the CVS head intact while masking the input to the reconstruction
head, thereby optimizing the auxiliary learning objective to maximize domain-
invariance in the learned latent graph representations:

$G=\Phi_{\text{LG}}(I);\ \mathcal{M}(G,c)=\hat{G}_{\mathcal{R}};\
\hat{y}=\phi_{\text{CVS}}(G);\
\hat{I}=\phi_{\mathcal{R}}(\hat{G}_{\mathcal{R}}).$ (2)

Notably, LG-CVS does not include image visual features in the reconstruction
input, instead constructing an object-centric feature layout $L_{\text{feat}}$
from $G$ alone before decoding with $\phi_{\text{decoder}}$. For completeness,
we introduce a mechanism to incorporate the image features
$H_{\text{backbone}}$ into $L_{\text{feat}}$, which uses predicted object
locations to arrange per-object features into a spatial grid, built on simple
concatenation:

$\hat{L}_{\text{feat}}=[H_{\text{backbone}};L_{\text{feat}}];\
\hat{I}=\phi_{\text{decoder}}(\hat{L}_{\text{feat}}),$ (3)

where we spatially resize $H_{\text{backbone}}$ to match the dimensions of
$\hat{L}_{\text{feat}}$.

###  3.4 Disentanglement Loss

While the feature disentanglement ablation studies can shed light on the
effect of domain shift on different types of features, ultimately, we are
interested in creating a model that uses all available information. Our key
observation is that the masking function $\mathcal{M}$ can also serve as a
form of data augmentation during training, and provide robustness to cases
where one or more feature categories in $G$ are inaccurate. To do so, we
introduce an auxiliary disentanglement loss function
$\mathcal{L}_{\text{DIS}}$, that is a linear combination of binary cross
entropy loss terms each using the prediction from a different masked graph
$\hat{G}$. Concretely:

$\mathcal{L}_{\text{DIS}}=\lambda_{\text{sem}}\mathcal{L}_{\text{CVS}}(\hat{y}_{\text{sem}},y)+\lambda_{\text{viz}}\mathcal{L}_{\text{CVS}}(\hat{y}_{\text{viz}},y)+\lambda_{\text{img}}\mathcal{L}_{\text{CVS}}(\hat{y}_{\text{img}},y),$
(4)

where $\hat{y}_{\text{sem}}=\phi_{\text{CVS}}(\hat{G}_{\text{sem}})$,
$\hat{y}_{\text{viz}}=\phi_{\text{CVS}}(\hat{G}_{\text{viz}})$, and
$\hat{y}_{\text{img}}=\phi_{\text{CVS}}(\hat{G}_{\text{img}})$, $y$ is the
ground-truth CVS label, and $\lambda_{\text{sem}}$, $\lambda_{\text{viz}}$,
and $\lambda_{\text{img}}$ are loss weighting terms.

Table 2: CVS Classifier Head Inputs Ablation Study in the Domain Generalization Transfer Setting: Endoscapes2023 to Endoscapes-WC70 Feature Type | Performance |
---|---|---
| Graph
---
Visual
| Graph
---
Semantic
| Backbone
---
Image
(mAP) |
✓ | ✓ | ✓ | 27.88 ± 0.37 | (LG-CVS [13])
✓ | ✓ | ✗ | 31.37 ± 1.72 |
✓ | ✗ | ✓ | 26.31 ± 5.32 | (LG-Visual)
✗ | ✓ | ✗ | 33.81 ± 1.32 | (LG-Semantic)
✗ | ✓ | ✓ | 34.14 ± 0.72 |
✓ | ✗ | ✗ | 29.36 ± 3.04 |
Table 3: Reconstruction Inputs Ablation Study for LG-CVS in the Domain Generalization Setting, Endoscapes2023 to Endoscapes-WC70. Feature Type | Performance |
---|---|---
| Graph
---
Visual
| Graph
---
Semantic
| Backbone
---
Image
(mAP) |
✓ | ✓ | ✓ | 27.88 ± 0.37 |
✓ | ✓ | ✗ | 23.07 ± 2.22 | (LG-CVS [13])
✓ | ✗ | ✓ | 27.23 ± 0.61 |
✗ | ✓ | ✗ | 28.60 ± 2.41 |
✗ | ✓ | ✓ | 28.29 ± 2.36 |

###  3.5 Training

As in [13, 11], we train all models (except ResNet50) in two-stages, first
finetuning a COCO-pretrained Mask-RCNN instance segmentation model on the
images with segmentation annotations, then freezing this model and training
each object-centric classifier using all images with CVS annotations. To
address class imbalance, we train with an inverse frequency-balanced binary
cross entropy loss. Finally, when training LG-DG, we set
$\lambda_{\text{sem}}=1$, $\lambda_{\text{viz}}=0.3$, and
$\lambda_{\text{img}}=0.3$.

##  4 Experiments and Results

As introduced in Section 1, there are traditionally three paradigms for
evaluating domain transfer: Supervised Domain Adaptation, Unsupervised Domain
Adaptation, and Domain Generalization. Because object-centric models contain
two components: an object detector and a classification model, we also
consider an additional evaluation setting: Partially Supervised Domain
Adaptation, where we assume access to CVS labels in the target domain to train
the classification model but not segmentation labels to train the object
detector222This setting represents a very realistic scenario as collecting
dense bounding box or segmentation labels is orders of magnitude more
expensive than image-level annotations for classification tasks like CVS..
Here, we freeze the Mask-RCNN detector trained on the source domain and train
only the second stage of each object-centric method on the target domain.
Finally, we replace the Supervised Domain Adaptation setting with Fully
Supervised evaluation, as our primary objective is to establish ceiling
performances. We also omit the Unsupervised Domain Adaptation setting, which
warrants a more thorough investigation in future work.

Table 4: Existing Object-Centric Models evaluated in Various Domain Adaptation Settings: Endoscapes2023 to Endoscapes-WC70. Model | Performance (mAP)
---|---
| Fully Supervised | Partially-Supervised | Domain Generalization
LG-CVS [13] | 33.69 ± 4.60 | 35.63 ± 3.34 | 28.06 ± 3.40
DeepCVS | 34.21 ± 2.42 | 37.83 ± 2.87 | 29.54 ± 1.46
LayoutCVS | 35.43 ± 1.68 | 35.95 ± 4.11 | 30.22 ± 1.92
ResNet50-DetInit | 27.38 ± 7.03 | 32.34 ± 5.11 | 23.48 ± 3.13
ResNet50 | 27.58 ± 2.40 | - | 12.77 ± 2.71
LG-DG (Ours) | 38.21 ± 2.03 | 44.18 ± 2.48 | 33.34 ± 2.22
Table 5: Existing Object-Centric Models evaluated in Various Domain Adaptation Settings: Endoscapes-WC70 to Endoscapes2023. Model | Performance (mAP)
---|---
| Fully Supervised | Partially-Supervised | Domain Generalization
LG-CVS [13] | 64.45 ± 1.30 | 55.64 ± 0.59 | 33.83 ± 1.20
DeepCVS | 58.80 ± 2.06 | 42.84 ± 2.32 | 44.30 ± 1.70
LayoutCVS | 58.14 ± 0.76 | 43.76 ± 0.70 | 43.26 ± 1.40
ResNet50-DetInit | 61.86 ± 2.57 | 57.63 ± 0.89 | 44.24 ± 0.39
ResNet50 | 52.27 ± 3.45 | - | 36.11 ± 0.68
LG-DG (Ours) | 67.25 ± 0.90 | 57.91 ± 2.44 | 47.95 ± 2.40
Table 6: Per-Class Breakdown of Object Detection Performance (Mask-RCNN, Instance Segmentation mAP) in Target Domain. WC70 refers to Endoscapes-WC70. Trained On | Tested On |  | Cystic
---
Plate
| Calot
---
Triangle
| Cystic
---
Artery
| Cystic
---
Duct
Gallbladder | Tool | Avg
WC70 | WC70 | 13.1 | 3.0 | 11.9 | 15.2 | 44.7 | 55.2 | 23.8
Endoscapes2023 | WC70 | 17.5 | 3.9 | 14.1 | 16.3 | 39.5 | 49.1 | 23.4
WC70 | Endoscapes2023 | 1.8 | 6.9 | 4.4 | 9.4 | 43.4 | 36.9 | 17.13
Endoscapes2023 | Endoscapes2023 | 9.1 | 22.5 | 13.4 | 19.2 | 65.8 | 57.6 | 31.3

In section 4, we analyze the results of the two latent graph ablation studies:
CVS Head and Reconstruction Head (Tables 2 and 3 respectively). Then, we move
on to the main experiments in subsection 4.2, where we analyze the domain
generalization performance of the four original object-centric methods, a
ResNet50 non-object-centric baseline, and our proposed LG-DG (Tables 4 and 5).
Lastly, Table 6 shows the domain generalization performance of each object
detector for reference. We also show the partially supervised and supervised
performances in Tables 4, 5, and 6 to better contextualize our results.

###  4.1 Latent Graph Ablation Studies

For both ablation studies, we use Endoscapes2023 as the source domain and
Endoscapes-WC70 as the target domain. Table 2 shows the results of the CVS
head ablations (different configurations of $\hat{G}_{\text{CVS}}$); we
identify two particular settings of interest: (1) LG-Semantic, which omits all
visual information, and (2) LG-Visual, which omits semantics and uses all the
visual information.

We observe that, in the domain generalization setting, LG-Semantic attains an
mAP of 33.81 mAP, surpassing LG-CVS by 21.2%. This shows that the various
feature modalities do not contribute synergistically when posed with domain
shift. We attribute the particular effectiveness of LG-Semantic to the fact
that it only uses the detected objects for downstream CVS prediction, thereby
providing robustness to visual domain gap. Studying the domain generalization
performance of the Mask-RCNN object detector (see Table 6) further
substantiates this notion: the object detector trained on Endoscapes2023
performs effectively on par with that trained on WC70 when tested on WC70. It
even outperforms the latter on the four smaller classes, all of which are
critical for CVS assessment. Since the object detector generalizes
effectively, LG-Semantic performs well. Meanwhile, LG-Visual performs markedly
worse than LG-Semantic, and slightly worse than LG-CVS, reiterating the notion
that semantic features are more robust to domain shifts than visual features.

Table 3 shows the results of the reconstruction head ablations (different
configurations of $\hat{G}_{\mathcal{R}}$). Broadly, we find that including
visual features in the reconstruction input, whether graph visual or backbone
image features, has a negative impact on domain generalization performance:
masking both graph visual features and backbone image features in
$G_{\mathcal{R}}$ yields the highest performance. One potential explanation
for this trend is that, when visual features are part of the reconstruction
input, they learn to encode highly domain-specific information like color and
texture distributions, degrading the overall domain invariance of the latent
graph $G$. On the other hand, when only semantic information is included, the
reconstruction objective only enforces encoding general, perhaps even
instance-agnostic, properties of the objects, as the semantic features do not
encode properties like color and texture by construction.

###  4.2 Main Experiments

Table 4 and Table 5 summarize CVS prediction performance for all three domain
adaptation settings, tested on Endoscapes2023 and Endoscapes-WC70
respectively. To recall: Fully Supervised indicates that both the object
detector and object-centric classification model are trained and tested on the
target domain; Partially Supervised indicates that the object detector is
trained on the source domain; Domain Generalization indicates that neither
object detector nor object-centric classification model is trained on the
target domain. We show both Fully Supervised and Partially Supervised
performance to illustrate the ceiling performance, which we use to
contextualize domain generalization performance.

LG-DG consistently outperforms all baseline methods across all settings: for
Domain Generalization, LG-DG demonstrates a mean increase of 9.28% over the
best-performing baseline models, LayoutCVS and DeepCVS respectively.
Interestingly, both LayoutCVS and DeepCVS are primarily based on semantic
features, with LayoutCVS explicitly ignoring visual information and DeepCVS
ineffective at leveraging visual information (shown in [13]); this reinforces
our conclusions from the Latent Graph ablation study. Importantly, LG-DG
vastly outperforms LG-CVS: 41.74% better when tested on Endoscapes2023 and
18.82% better when tested on Endoscapes-WC70 (average 30.28%). This clearly
highlights the effectiveness of our proposed optimizations. Altogether, LG-DG
takes great strides in closing the domain gap with no information about the
target domain: it is 24.54% lower than the ceiling performance for Endoscapes-
WC70 (Partially Supervised LG-DG) and 28.70% lower than the ceiling
performance for Endoscapes (Fully Supervised LG-DG).

##  5 Conclusion

We investigate the capabilities of object-centric approaches for domain
adaptation specifically in the context of fine-grained classification, using
Critical View of Safety prediction to measure performance. We show that
object-centric methods are highly effective for Domain Generalization,
particularly compared to non-object-centric image classifiers. Then,
leveraging the modularity of object-centric approaches, we propose an
optimized method, LG-DomainGen, that includes a novel disentanglement loss to
improve robustness to domain shift. Future work should seek to extensively
validate these findings on various surgical procedures and tasks and to extend
these findings to other settings including UDA and Federated Learning.

##  6 Acknowledgments

This work was supported by French state funds managed by the ANR within the
National AI Chair program under Grant ANR-20-CHIA-0029-01 (Chair AI4ORSafety).
This work was granted access to the HPC resources of IDRIS under the
allocation AD011013523R1 made by GENCI.

Ethical Approval: This article does not contain any studies with human
participants or animals performed by any of the authors.

Competing Interests: The authors declare no conflict of interest.

Informed Consent: This manuscript does not contain any patient data.

Code Availability: The source code is publicly available at
https://github.com/CAMMA-public/SurgLatentGraph.

## References

  * Basak and Yin [2023] Basak H, Yin Z (2023) Semi-supervised domain adaptive medical image segmentation through consistency regularized disentangled contrastive learning. In: MICCAI, Springer, pp 260–270
  * Chen et al [2023] Chen Z, Pan Y, Ye Y, et al (2023) Treasure in distribution: A domain randomization based multi-source domain generalization for 2d medical image segmentation. In: MICCAI. Springer Nature Switzerland, Cham, pp 89–99
  * Choi et al [2021] Choi S, Jung S, Yun H, et al (2021) Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In: CVPR, pp 11580–11590
  * Grammatikopoulou et al [2021] Grammatikopoulou M, Flouty E, Kadkhodamohammadi A, et al (2021) Cadis: Cataract dataset for surgical rgb-image segmentation. Medical Image Analysis 71
  * Hamoud et al [2023] Hamoud I, Jamal MA, Srivastav V, et al (2023) St(or)2: Spatio-temporal object level reasoning for activity recognition in the operating room. In: Medical Imaging with Deep Learning
  * Hao et al [2023] Hao L, Hu Y, Lin W, et al (2023) Act-net: Anchor-context action detection in surgery videos. In: MICCAI, Springer, pp 196–206
  * Holm et al [2023] Holm F, Ghazaei G, Czempiel T, et al (2023) Dynamic scene graph representation for surgical video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 81–87
  * Kassem et al [2022] Kassem H, Alapatt D, Mascagni P, et al (2022) Federated cycling (fedcy): Semi-supervised federated learning of surgical phases. IEEE Transactions on Medical Imaging
  * Mascagni et al [2021] Mascagni P, Vardazaryan A, Alapatt D, et al (2021) Artificial intelligence for surgical safety: automatic assessment of the critical view of safety in laparoscopic cholecystectomy using deep learning. Annals of Surgery
  * Mottaghi et al [2022] Mottaghi A, Sharghi A, Yeung S, et al (2022) Adaptation of surgical activity recognition models across operating rooms. In: MICCAI, Springer, pp 530–540
  * Murali et al [2023a] Murali A, Alapatt D, Mascagni P, et al (2023a) The endoscapes dataset for surgical scene segmentation, object detection, and critical view of safety assessment: Official splits and benchmark. arXiv preprint arXiv:231212429
  * Murali et al [2023b] Murali A, Alapatt D, Mascagni P, et al (2023b) Encoding surgical videos as latent spatiotemporal graphs for object and anatomy-driven reasoning. In: MICCAI, Springer, pp 647–657
  * Murali et al [2023c] Murali A, Alapatt D, Mascagni P, et al (2023c) Latent graph representations for critical view of safety assessment. IEEE Transactions on Medical Imaging pp 1–1
  * Özsoy et al [2023] Özsoy E, Czempiel T, Holm F, et al (2023) Labrad-or: Lightweight memory scene graphs for accurate bimodal reasoning in dynamic operating rooms. arXiv preprint arXiv:230313293
  * Pang et al [2022] Pang W, Islam M, Mitheran S, et al (2022) Rethinking feature extraction: Gradient-based localized feature extraction for end-to-end surgical downstream tasks. IEEE Robotics and Automation Letters 7(4):12623–12630
  * Sestini et al [2023] Sestini L, Rosa B, De Momi E, et al (2023) Fun-sis: A fully unsupervised approach for surgical instrument segmentation. Medical Image Analysis 85:102751
  * Sharma et al [2023] Sharma S, Nwoye CI, Mutter D, et al (2023) Surgical action triplet detection by mixed supervised learning of instrument-tissue interactions. In: MICCAI, Springer, pp 505–514
  * Sohan and Basalamah [2023] Sohan MF, Basalamah A (2023) A systematic review on federated learning in medical image analysis. IEEE Access
  * Srivastav et al [2022] Srivastav V, Gangi A, Padoy N (2022) Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the operating room. In: Medical Image Analysis
  * Twinanda et al [2016] Twinanda AP, Shehata S, Mutter D, et al (2016) Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE transactions on medical imaging 36(1):86–97
  * Wang et al [2019] Wang Q, Bu P, Breckon TP (2019) Unifying unsupervised domain adaptation and zero-shot visual recognition. In: 2019 International Joint Conference on Neural Networks (IJCNN), IEEE, pp 1–8
  * Xu et al [2022] Xu J, Zhang Q, Yu Y, et al (2022) Deep reconstruction-recoding network for unsupervised domain adaptation and multi-center generalization in colonoscopy polyp detection. Computer Methods and Programs in Biomedicine 214:106576