# Towards Proactive Safe Human-Robot Collaborations via
Data-Efficient Conditional Behavior Prediction

Ravi Pandya∗,1, Zhuoyuan Wang∗,2, Yorie Nakahira2, Changliu Liu1 *contributed
equally to this work.1Authors are with the Robotics Institute at Carnegie
Mellon University, Pittsburgh, Pennsylvania, rapandya,
<EMAIL_ADDRESS>are with the Department of Electrical and
Computer Engineering at Carnegie Mellon University, Pittsburgh, Pennsylvania,
zhuoyuaw<EMAIL_ADDRESS>
###### Abstract

We focus on the problem of how we can enable a robot to collaborate seamlessly
with a human partner, specifically in scenarios like collaborative
manufacturing where prexisting data is sparse. Much prior work in human-robot
collaboration uses observational models of humans (i.e. models that treat the
robot purely as an observer) to choose the robot’s behavior, but such models
do not account for the influence the robot has on the human’s actions, which
may lead to inefficient interactions. We instead formulate the problem of
optimally choosing a collaborative robot’s behavior based on a conditional
model of the human that depends on the robot’s future behavior. First, we
propose a novel model-based formulation of conditional behavior prediction
that allows the robot to infer the human’s intentions based on its future plan
in data-sparse environments. We then show how to utilize a conditional model
for proactive goal selection and path generation around human collaborators.
Finally, we use our proposed proactive controller in a collaborative task with
real users to show that it can improve users’ interactions with a robot
collaborator quantitatively and qualitatively.

##  I Introduction

As robots are becoming more common in industrial manufacturing, social and
home environments, it is imperative that they are able to seamlessly interact
with humans. In particular, robots need to be able to collaborate with humans,
since this would allow both agents to take advantage of the speed and
precision of robots as well as the flexibility of humans for completing tasks.
One major challenge in working with people is the closed-loop nature of the
interaction—the robot needs to understand what the human wants to do so it can
assist them, but the robot’s own actions will also influence the human’s
actions because both agents are acting in the same environment. Being able to
reason about this influence is an important factor in both efficient and safe
human-robot interactions, since improved models of human behavior can
additionally enable less conservativeness in safe control. Much existing work
in the field of human-robot interaction has made use of observational models
of humans [1, 2], but generally lacks the ability to predict how the human
will respond to the robot’s ultimate decision.

Autonomous driving researchers have taken note of this and recently developed
predictive models of agents that are conditioned on the ego agent’s future
plan [3], sometimes called conditional behavior prediction (CBP) models. CBP
models rely on large datasets of human-human or human-robot interactions to
reason about how a robot’s actions would change the intentions of humans.
Game-theoretic approaches also do this reasoning, but often assume some
hierarchical information structure [4], can be too restrictive to achieve
optimal safety-performance tradeoffs in human-machine systems [5], and prior
work has shown that Nash equilibria can be poor predictors of humans [6, 7,
8].

In this paper, we are interested in applying the CBP idea more broadly to
human-robot collaborations across data-sparse domains such as home and
manufacturing environments (as opposed to autonomous driving, where many
large, diverse, multi-agent datasets currently exist). In this work, we take a
step towards enabling robots to similarly reason about the closed-loop
collaboration with human partners in a data-efficient manner using a model-
based CBP formulation. We then utilize this CBP model for downstream
collaborative tasks by introducing an objective that proactively switches
between courtesy and influence behavior. We additionally show how to utilize
our CBP model for doing long-term safe control in multi-goal human-robot
collaborations. The system diagram of the proposed framework is shown in Fig.
1.

Figure 1: Overall diagram of the proposed framework.

Our contributions in this work are the following:

  1. 1.

Model-based conditional behavior prediction (CBP) in multi-goal human-robot
collaborations.

  2. 2.

An integrated framework for courtesy and influence behavior in robot goal
selection utilizing a CBP model.

  3. 3.

Validation with human experiments on multi-goal collaboration tasks.

##  II Related Work

Human Goal Prediction: Much prior work has been done that treats humans as
rational agents and tries to predict their intentions [1, 2], often using the
framework of inverse reinforcement learning [9] and Bayesian inference [10,
11]. Generally this line of work treats the robot as a passive observer of the
human’s behavior [12] has the robot pick a best response to the prediction
[13, 14]. Some work has considered the case where both agents have goals in
the same space to complete a collaborative task [15], but still considers
minimizing the expected cost of the robot relative to an observational
predictive model of the human.

Conditional Behavior Prediction: Autonomous agents influencing humans has been
considered recently in autonomous driving and pedestrian prediction settings
[3, 16]. These conditional behavior prediction (CBP) approaches have been
applied in scenarios where there are existing large datasets that allow large
models to be trained [17, 18, 19] (researchers in [3] note that their dataset
consists of 18 years of continuous driving data). We are instead interested in
general human-robot collaborations where data may be limited, so we formulate
a CBP approach that allows the robot to reason in other data-sparse
environments.

Courtesy and Influence: Prior work on courtesy and influence behavior for
robots often relies on human and robot collision avoidance [20, 21] or
minimizes the expected cost incurred by a human [22, 23]. Other work on
influencing humans usually rely on learning-based models, which are shown to
be brittle with real humans [24, 25].

Safe Human-Robot Collaboration (HRC): Safety has long been studied for HRC
with techniques like workplace separation [26], rule-based methods [27] and
control barrier functions [28, 29]. We are interested in ensuring long-term
safety for HRC, so we leverage a long-term safe control strategy for
stochastic systems [30] and discuss how to incorporate our CBP model into the
safe controller.

##  III Problem Formulation

The HRI System. We consider the case of one robot collaborating with one
human. The two agents have a shared set of goals $\Theta$, known to both
agents a priori. The human has noisy dynamics that depend on both agents’
previous states and the human’s true goal, denoted by $\theta_{H}^{*}$:

$x_{H}^{t+1}=m(x_{H}^{t},x_{R}^{t},\theta_{H}^{*})+w^{t},$ (1)

where $w^{t}$ is zero-mean Gaussian noise $w^{t}\sim\mathcal{N}(0,\Sigma_{H})$
with covariance $\Sigma_{H}$. The robot has deterministic dynamics:

$x_{R}^{t+1}=f_{R}(x_{R}^{t},u_{R}^{t}),$ (2)

where the robot’s control policy $u_{R}^{t}$ may arbitrarily depend on the two
agents states $x_{H}^{t},x_{R}^{t}$ and the set of possible goals $\Theta$. We
assume that the robot has access to the form of the human’s dynamics function
$m(x_{H}^{t},x_{R}^{t},\theta)$, but does not know the actual goal of the
human $\theta_{H}^{*}$ or how it might change over time. The human’s dynamics
may be estimated offline from data [31] or from online adaptation [32], but
our focus is on dealing with uncertainty in the human’s goal selection
(particularly how it is affected by the robot).

The Robot’s Objective. The robot’s objective is to allow the team to work
together fluently while keeping the human safe. The robot keeps a
probabilisitic predictive model of the human $p(X_{H}\mid
X_{R}=\mathbf{x_{R}},\mathbf{o})$, where the future trajectories of the human
and robot agents are random variables denoted by $X_{H}$ and $X_{R}$
respectively, and $\mathbf{o}$ contains observations of the environment and
$\mathbf{x_{R}}=[x_{R}^{t},...,x_{R}^{t+T}]$ is a particular robot future
trajectory. Since the robot’s prediction is probabilistic, it needs to stay
safe probabilistically. We thus want the probability of collision with the
human to be less than a small value $\epsilon$, meaning the robot needs to
stay safe with probability $1-\epsilon$. Based on a predefined safety
specification, we define the set of safe states $\mathcal{Z}$. Let the joint
state $x^{t}=[x_{H}^{t,T},x_{R}^{t,T}]^{\top}$, so our probabilistic safety
constraint is:

$p(x^{t}\in\mathcal{Z},\;\forall t={1,...,H})\geq 1-\epsilon,$ (3)

where $t$ is the timestep and $H$ is the trajectory horizon.

Finally, we are given some objective $J$ for the robot to minimize. This
objective function could encompass both efficiency of completing the robot’s
own task and a penalty for affecting the human’s intention. We investigate the
design of $J$ in Sec. V. Our robot’s constrained objective is:

$\displaystyle\min_{\mathbf{x_{R}}}$
$\displaystyle\quad\mathbb{E}_{\mathbf{x_{H}}\sim p(X_{H}\mid
X_{R}=\mathbf{x_{R}},\mathbf{o})}\sum_{t=1}^{H}J(x_{R}^{t},x_{H}^{t})$ (4a)
$\displaystyle\text{s.t.}\quad p(x^{t}\in\mathcal{Z},\;\forall
t={1,...,H})\geq 1-\epsilon$ (4b)

##  IV Model-Based CBP

Figure 2: The robot’s belief probabilities using the derived model-based CBP
formulation for the interaction depicted in the “influence objective” case in
Fig. 4. The differences between the beliefs are exaggerated for illustration.

Key Insight. As previously noted, prior work using observational inference
over a human’s goal does not account for potential effects of a robot
collaborator, since the robot is treated as an external observer. Our key
insight is that we can decouple the human’s prior goal $\theta_{H}^{prior}$
(from observational inference) from their posterior goal they will ultimately
choose around the robot, $\theta_{H}^{post}$. This allows us to formulate a
Bayesian-inference-based CBP model that can be used for HRC without the need
for large datasets.

Derivation. Observational Bayesian inference gives the robot access to

$b^{t}_{R}(\theta_{H}^{prior})=p(\theta_{H}^{prior}\mid
x_{H}^{0:t},x_{R}^{0:t},u_{H}^{0:t}).$ (5)

Given the simulated human’s LQR controller (Sec. VII), we follow [33] to
compute an exact form of this belief (Appendix A).

We want the robot to estimate the distribution of $\theta_{H}^{post}$
conditioned on the robot’s future plan: $b^{t}_{R}(\theta_{H}^{post}\mid
x_{R}^{t+1:T})=p(\theta_{H}^{post}\mid
x_{H}^{0:t},x_{R}^{0:t},u_{H}^{0:t},x_{R}^{t+1:T})$. However, we know that it
may be computationally intractable to integrate over the space of future
trajectories. We instead consider the robot’s goal $\theta_{R}$ as a proxy for
the robot’s full plan:

$b^{t}_{R}(\theta_{H}^{post}\mid\theta_{R})=p(\theta_{H}^{post}\mid
x_{H}^{0:t},x_{R}^{0:t},u_{H}^{0:t},\theta_{R}).$ (6)

We note that one could include $\theta_{R}$ in the human’s reward function in
observational Bayesian inference directly. However, decomposing the inference
as we propose frees the robot to convert any nominal intention predictor to a
conditional one.

For ease of notation, we denote the robot’s observation
$(x_{H}^{0:t},x_{R}^{0:t},u_{H}^{0:t})$ as $\mathbf{o^{0:t}}$. To compute the
conditional distribution, we can integrate out the variable
$\theta_{H}^{prior}$ from the joint distribution
$p(\theta_{H}^{post},\theta_{H}^{prior}\mid\mathbf{o}^{0:t},\theta_{R})$ to
get
$p(\theta_{H}^{post}\mid\mathbf{o^{0:t}},\theta_{R})=\int_{\theta_{H}^{prior}}p(\theta_{H}^{post}\mid\theta_{H}^{prior},\mathbf{o^{0:t}},\theta_{R})p(\theta_{H}^{prior}\mid\mathbf{o^{0:t}},\theta_{R})$.
We know that the human’s prior goal selection $\theta_{H}^{prior}$ will be
independent of the robot’s future goal $\theta_{R}$, so we can simplify this
equation as
$\int_{\theta_{H}^{prior}}p(\theta_{H}^{post}\mid\theta_{H}^{prior},\mathbf{o^{0:t}},\theta_{R})p(\theta_{H}^{prior}\mid\mathbf{o^{0:t}})$.
Since we have a discrete set of goal locations, the final formula is

$\begin{split}&p(\theta_{H}^{post}\mid\mathbf{o^{0:t}},\theta_{R})\\\
&=\sum_{\theta_{H}^{prior}}p(\theta_{H}^{post}\mid\theta_{H}^{prior},\mathbf{o^{0:t}},\theta_{R})p(\theta_{H}^{prior}\mid\mathbf{o^{0:t}}).\end{split}$
(7)

Note that the second term is exactly what comes out of observational Bayesian
goal inference. Now we need a way to compute
$p(\theta_{H}^{post}\mid\theta_{H}^{prior},\mathbf{o^{0:t}},\theta_{R})$.
Inspired by prior work [34, 10], we use a softmax distribution with well-
designed features as a strong prior so we can utilize this formulation in
data-sparse environments:

$p(\theta_{H}^{post}\mid\theta_{H}^{prior},\mathbf{o^{0:t}},\theta_{R})=\frac{e^{-\beta_{cbp}s(\theta_{H}^{post},\theta_{R};\mathbf{o^{0:t}},\theta_{H}^{prior})}}{\sum_{\theta_{H}}e^{-\beta_{cbp}s(\theta_{H},\theta_{R};\mathbf{o^{0:t}},\theta_{H}^{prior})}},$
(8)

where $\beta_{cbp}$ is an additional inverse temperature parameter for this
distribution and $s(\cdot)$ is a score function that encodes how much robot’s
goal selection “interacts” with the human’s goal selection. The robot’s
conditional belief is visualized in Fig. 2 (for the interaction in Fig. 4
Right). The conditional model infers that the human will likely reach the
orange or blue goals if the robot chooses the brown goal.

Discussion of Score Function. For our tasks (Sec. VIII, Sec. IX), we find this
score function to be expressive enough to enable proactive robot goal
selection:

$\begin{split}&s(\theta_{H},\theta_{R};\mathbf{o^{0:t}},\theta_{H}^{prior})=w_{1}||x_{H}^{t}-\theta_{H}||\\\
&-w_{2}||\theta_{H}-\theta_{R}||+w_{3}||\theta_{H}-\theta_{H}^{prior}||,\end{split}$
(9)

where $w_{1},w_{2},w_{3}\in\mathbb{R}$. It encodes the idea that the human is
more likely to choose a goal close to their current state (term 1), farther
from the robot’s goal (term 2), and have some preference to not changing their
current goal (term 3).

##  V The Robot’s Multi-Stage Objective

Design Goal. Equipped with a CBP model, we wish to design an objective
$J(\cdot)$ that will enable efficient collaborations with a human partner. We
first note that many collaborative tasks can be solved in a leader-follower
manner [35, 36] where one agent decides on a general strategy and the other
follows along. When interacting with humans, we want robots to be courteous so
as to not interfere with the human’s intended strategy. However, there may be
times where the human is uncertain about what strategy to choose. At such
times, we want the robot to proactively influence the human to choose an
efficient strategy. We introduce a switching controller that estimates whether
the human is hesitating then utilizes its CBP model to be either proactive or
courteous.

Figure 3: Visualization of interaction and goal selection with (a) KL-
Divergence cost function (b) argmax belief cost function. The Kl-divergence
cost results in chattering of both agents’ goals while the argmax belief
results in a stable interaction.

Detecting Human Uncertainty.  We aim to equip the robot with the ability to
detect when the human is uncertain of their goal and subsequently influence
them towards an efficient strategy. To do this, we turn to prior work on
detecting model misspecificsation in Bayesian models [13, 37]. We jointly keep
a belief over the human’s prior intention $\theta_{H}^{prior}$ and the robot’s
model-confidence parameter $\beta_{R}$:

$b_{R}^{t+1}(\theta_{H}^{prior},\beta_{R})=\frac{p(u_{H}^{t}\mid\mathbf{o^{0:t}};\theta_{H}^{prior},\beta_{R})b_{R}^{t}(\theta_{H}^{prior},\beta_{R})}{\int_{\theta,\beta}p(u_{H}^{t}\mid\mathbf{o^{0:t}};\theta,\beta)b_{R}^{t}(\theta,\beta)d\theta
d\beta}.$ (10)

We focus on collaborative tasks where the set of goals is known to both agents
a priori, so the most likely cause of low model confidence is that the human
has not yet decided on their goal. We follow prior work [37] to detect when
the model confidence is low:
$\forall\theta\in\Theta,\text{argmax}_{\beta}b_{R}(\beta\mid\theta)<\delta$.
If all hypotheses have the most probability mass on $\beta$s that are smaller
than a threshhold $\delta$, the robot will raise a flag saying that the human
is uncertain about their intention.

Following Behavior: Courtesy Objective.  Intuitively, avoiding changing the
human’s intention might be formulated as minimizing the KL-Divergence between
the robot’s nominal belief of the human and the conditional belief of the
human, which would mean that the robot’s future actions have no effect on the
human’s intention:

$J(\theta_{R})=D_{\text{KL}}\left(b^{t+1}_{R}(\theta_{H}^{post}\mid\theta_{R})\parallel
b_{R}^{t}(\theta_{H}^{prior})\right).$ (11)

We actually find that this approach results in chattering of the robot’s goal
(Fig. 3(a)). Since the robot is trying to keep the belief distribution close
to the prior belief, it changes its goal to keep its belief from changing over
time. Instead, we consider maximizing the probability that the human keeps the
same intention before and after considering the robot’s action. The objective
function to minimize is:

$\begin{split}J_{c}(\theta_{R})&=-b_{R}^{t+1}(\theta_{H}^{post}=\hat{\theta}_{H}\mid\theta_{R}),\\\
\text{where }\hat{\theta}_{H}&=\underset{\theta\in\Theta}{\text{argmax
}}b_{R}^{t}(\theta_{H}^{prior}=\theta).\end{split}$ (12)

Using this courtesy objective results in stable goal selection for the robot
(Fig. 3(b)).

Leading Behavior: Influence Objective.

Figure 4: Left: human chooses left goal in isolation. Right: robot chooses a
goal to successfully influence the human’s goal selection by modeling their
goal change using model-based CBP.

When the human is uncertain of which goal to choose, the robot has a chance to
influence them. We want the interaction to be efficient, so the robot can pick
a goal for the human that it thinks is best for the team and try to influence
the human to move towards it. The cost function will be the total distance
that the team would travel:

$\begin{split}J_{i}(\theta_{R})&=||x_{R}^{t}-\theta_{R}||+||x_{H}^{t}-\hat{\theta}_{H}||,\\\
\text{where }\hat{\theta}_{H}&=\underset{\theta\in\Theta}{\text{argmax
}}b_{R}^{t+1}(\theta_{H}^{post}=\theta\mid\theta_{R}).\end{split}$ (13)

Using this objective function and the CBP model, the robot influences the
simulated human to select a different goal than they would have in isolation
(Fig. 4).

Overall Objective. The robot’s objective switches modes between influencing
the human and being courteous to them based on whether the robot believes the
human is unsure about what to do:

$J(\cdot)=\begin{cases}J_{i}(\cdot),&\text{if
}\forall\theta\in\Theta,\text{argmax}_{\beta}b_{R}(\beta\mid\theta)<\delta,\\\
\end{cases}$ (14)

##  VI Safe Trajectory Generation

We still need the robot to stay safe in the long-term with respect to the
actual conditional model of the human, since this will be the distribution
that the human’s goal selection ultimately takes. One major challenge is that
the uncertainty in the human’s intention hinders the direct application of
short-term-based safe control methods to ensure long-term safety [38, 39]. We
instead describe a method for allowing an existing long-term probabilistic
safe controller [30] to sample directly from the CBP model to compute safe
trajectories. We define safety as keeping a minimum distance $d_{\text{min}}$
between human and robot. Mathematically, we can define a barrier function

$\phi_{d_{\text{min}}}(x)=||C_{R}x_{R}-C_{H}x_{H}||_{2}-d_{\text{min}}$ (15)

where $C_{H}$ and $C_{R}$ are matrices that select the $(x,y)$ positions of
agents’ states. The safe set of states for the system is then
$\mathcal{Z}=\\{x:\phi_{d_{\text{min}}}(x)\geq 0\\}$.

To use the CBP model (Sec. IV) for safe control, the robot needs to know how
its low-level action will affect the human’s intention. This can be done by
letting the robot keep a mental model of the human’s inference over the
robot’s goal $\hat{b}_{H}^{t}(\theta_{R})$ (a common paradigm in the
literature [34, 40]). The mental model is essentially an observational
Bayesian inference model of the robot’s goal. The mental model can be
simulated forward and used to keep an overall belief over the human’s
posterior goal $\theta_{H}^{post}$:
$p(\theta_{H}^{post})=\hat{b}_{H}^{t}(\theta_{R})b_{R}^{t}(\theta_{H}^{post}\mid\theta_{R})$.
The robot can sample from $p(\theta_{H}^{post})$ to compute the probability of
safety, which is a distribution that incorporates the proposed CBP model. We
equip a probabilistic long-term safe controller from the literature [30] with
this distribution $p(\theta_{H}^{post})$ to compute safe trajectories for the
robot by sampling from the uncertain human intention instead of just from a
stochastic dynamics model as described in the original paper.

##  VII Simulation Environment

The human and robot are double-integrator LTI systems:

$\displaystyle x_{H}^{t+1}$
$\displaystyle=f_{H}(x_{H}^{t},x_{R}^{t},u_{H}^{t})=Ax_{H}^{t}+Bu_{H}^{t}+w^{t}$
(16) $\displaystyle x_{R}^{t+1}$
$\displaystyle=f_{R}(x_{R}^{t},u_{R}^{t})=Ax_{R}^{t}+Bu_{R}^{t}$ (17)

where $x_{\\{H,R\\}}=[p_{x},v_{x},p_{y},v_{y}]^{T}$ and
$w^{t}\sim\mathcal{N}(0,\Sigma_{H})$. The human’s control is inspired by
social force models of humans [41], where they are attracted to their goal
$\theta$ and repelled from the robot:

$u_{H}^{t}=K(\theta-
x_{H}^{t})+\frac{\gamma}{d^{2}}(C_{H}x_{H}^{t}-C_{R}x_{R}^{t}).$ (18)

$K$ is the control gain computed from LQR,
$d=||C_{R}x_{R}^{t}-C_{H}x_{H}^{t}||_{2}$ is the distance between the two
agents, and $\gamma$ controls how strongly the human is repelled from the
robot.

| % rollouts $H$ changed $\theta$ | # times $H$ changed $\theta$
---|---|---
Naive | $61\%$ | $0.8\pm 0.8$
Reactive | $42\%$ | $0.7\pm 1.1$
Proactive-NN | $\mathbf{17\%}$ | $\mathbf{0.27\pm 0.7}$
Proactive-Model | $\mathit{18\%}$ | $\mathit{0.31\pm 0.8}$
TABLE I:  Average courtesy metrics in simulation (mean $\pm$ SD).

###  VII-A Simulated Human Models

Uncertain Human: The first simulated human infers the robot’s goal online (a
common paradigm from literature [34, 40, 42]). The human keeps its own belief
$b_{H}^{t}(\theta)$ over the robot’s goal. The simulated human chooses the
closest goal that is not the same as the robot’s most likely goal once its
belief is sufficiently low-entropy ($\max_{\theta}b_{H}^{t}(\theta)\geq 0.4$):

$\displaystyle\begin{split}&\theta_{H}^{*}=\underset{\theta\in\Theta}{\text{argmin
}}||x_{H}^{t}-\theta||,\;\>\theta\neq\underset{\theta^{{}^{\prime}}\in\Theta}{\text{argmax
}}b_{H}^{t}(\theta^{{}^{\prime}}).\end{split}$ (19)

Stubborn Human: The second simulated human is a stubborn human that will never
change their goal, which is also a mode of human behavior studied in prior
literature [43, 44]. This human chooses its goal to be the goal closest to
itself $\theta_{H}^{*}=\underset{\theta\in\Theta}{\text{argmin
}}||C_{H}x_{H}-\theta||$.

###  VII-B Baseline Robot Controllers

Naive Robot: This robot simply chooses the closest goal:

$\theta_{R}^{*}=\underset{\theta\in\Theta}{\text{argmin
}}||C_{R}x_{R}-\theta||$ (20)

Reactive Robot: For a less naive baseline, we use a reactive goal selection
method based on observational Bayesian inference, which picks a goal that is
close to the robot but is not the inferred goal of the human:

$\begin{split}\theta_{R}^{*}&=\underset{\theta\in\Theta}{\text{argmin
}}||C_{R}x_{R}-\theta||,\;\>\theta\neq\underset{\theta^{\prime}\in\Theta}{\text{argmax
}}b_{R}^{t}(\theta^{\prime})\end{split}$ (21)

Proactive-NN Robot: We implement a learning-based conditional behavior
prediction model that has a similar structure to other learning-based CBP
models from the literature [3, 16]. The model takes in the agents’ trajectory
history and the robot’s future plan and outputs the probability of the human
reaching each goal. Our training dataset consists of 3.5 million data points
(approximately 97 hours of data for this particular human-robot interaction).
We use the same proactive goal-selection method (14) for this approach.

Figure 5: Long-term safety probabilities in simulation.

##  VIII Simulation Results

| stubborn human | uncertain human | overall
---|---|---|---
Naive | $3.2\pm 3.1$ | $\mathbf{7.68\pm 4.0}$ | $5.44\pm 4.4$
Reactive | $6.4\pm 3.0$ | $4.76\pm 2.1$ | $5.6\pm 2.7$
Proactive-NN | $\mathbf{9.1\pm 3.9}$ | $6.28\pm 3.8$ | $\mathbf{7.7\pm 4.1}$
Proactive-Model | $\mathit{8.6\pm 4.9}$ | $6.4\pm 3.5$ | $\mathit{7.5\pm 4.0}$
TABLE II:  Average number of goals reached in simulation (mean $\pm$ SD).

We tested the proposed controller against a simulated human that is randomly
selected to be either “uncertain” or “stubborn,” (Sec. VII-A), unknown to the
robot, for 100 simulated games. Each game lasts for 30 seconds. For the
proactive-model robot, the weights in (9) are fixed at
$[w_{1},w_{2},w_{3}]=[2,0.9,2]$ for both human models.

Proactive models are courteous. First, we test how courteous each robot agent
is around the uncertain human (Table I). We measure the percentage of
trajectories that the human changed their initial goal and the average number
of times per trajectory that the human changed their goal. The proactive-NN
controller performs the best111The NN-based controller is likely not near 0
because of slight distribution shift between training and testing. with the
proposed proactive-model controller barely behind it. The proactive-model
controller is able to perform just as well as the proactive-NN controller in a
data-sparse environment by just doing online inference.

Proactive models are flexible. We measure the efficiency of the interactions
by counting the total number of goals reached by the team (Table II) over 100
initial conditions. The proactive-NN model performs the best on average with
the proactive-model controller just behind it, showing us that the proactive
models are flexible enough to efficiently interact with very different kinds
of humans without prior knowledge. This also shows that the proposed
controller is able to perform well in a data-sparse environment. We do,
however, see that the naive controller works best with the uncertain human.
Because the naive controller will immediately commit to an action, it’s very
easy for the simulated human to respond immediately.

Long-term safety can be assured. Fig. 5 shows the safety probability under
time horizon $H=20$ and risk tolerance $\epsilon=0.1$ with and without the
long-term safe controller while the robot selects goals with the proactive-
model controller. We can see that with safety the modification, long-term
safety probability is maintained over $90\%$ as desired while the nominal
goal-seeking controller fails to do so.

##  IX User Study

In order to investigate how our proposed proactive robot controller works with
real people, we ran an IRB-approved study with real users interacting with our
2D environment. The study lasted 20 minutes and participants were paid $\$5$.

Figure 6: Shows the setup for the user study where participants control an on-screen avatar directly with their hand to try and collect diamonds in collaboration with different robots. | Naive | Reactive | Proactive
---|---|---|---
Accounted: “[Robot] accounted for |  |  |
the [goal] I wanted to pick when |  |  |
it was picking a [goal].” | $2.1$ | $\mathbf{4.0}$ | $\mathbf{3.7}$
Changed: “I often changed which [goal] |  |  |
I picked initially because of [Robot].” | 3.7 | 1.7 | 2.7
% ranked 1st (forced choice) | 19% | 33% | 47%
TABLE III:  Responses to subjective survey questions (5: Strongly Agree, 1: Strongly Disagree). | # goals reached | avg. hesitation time (sec)
---|---|---
Naive | $\mathbf{18.9\pm 2.1}$ | $\mathbf{0.95\pm 0.34}$
Reactive | $15.2\pm 1.6$ | $1.2\pm 0.55$
Proactive-Model | $17.1\pm 2.3$ | $\mathbf{0.87\pm 0.37}$
TABLE IV:  Objective results for user study.

Experimental Setup: Users control an avatar on a computer screen with their
hand. Their hand position is measured by a camera and mapped onto the 2D space
(Fig. 6). The task is for the user to collect diamonds simultaneously with the
robot in the allotted time without colliding.

Independent Variables: We ran a within-subjects study and manipulated the
robot type with three levels: naive (Sec. VII-B), reactive (Sec. VII-B) and
proactive (Sec. V). We do not include the proactive-NN robot due to the lack
of a large enough dataset for our task.

Objective Measures: We cannot directly measure if the human’s intention
changed over time, so we instead measure the hesitation time, or the amount of
time the human waits before reaching their goal as a proxy for how much mental
energy has to exert to choose their goal. We additionally measure the team’s
efficiency as the number of goals reached.

Subjective Measures: We ask participants a series of 5-point Lkert scale [45]
questions222Full analysis in Appendix D about their experience with each robot
and finally ask for a ranking of all robots.

Hypothesis H1: Participants interacting with the proactive robot will hesitate
less and score more points than when interacting with the reactive robot.

Hypothesis H2: Participants will feel that the proactive robot was easier to
work with because it better accounted for their intentions.

Participants: We recruited 22 participants from the campus community for the
study. Each participant interacted with all three robot types in a randomized
order (four 45-second games per robot). One user’s data was not included due
to technical difficulties during data saving.

Quantitative Results: Running one-way repeated measures ANOVAs tell us the
robot type had a statistically significant effect on the number of goals
reached by the team ($F(2,40)=67.1,p<0.0001$) and on users’ hesitation time
($F(2,40)=14.7,p<0.0001$). A post-hoc Bonferroni test on the score tells us
that the differences in goals reached between all three pairs are significant
($p<0.001$ for all). Users performed the best with the Naive robot, second
best with the Proactive robot and worst with the Reactive robot. This
partially supports H1, though we did not expect users to perform so well with
the naive robot. Users were capable of being followers in the interaction,
even if they did not necessarily enjoy it. A post-hoc Bonferroni test on the
hesitation time tells us that users hesitated significantly less with the
proactive robot than the reactive robot ($p=0.0004$) and less with the naive
robot than the reactive robot ($p=0.004$). This result also supports H1, and
we see a surprisingly positive result with the naive robot, meaning users were
also able to react to it quickly.

We also empirically measured the safety of the interactions to check that they
line up with the theoretical guarantees. Across all interactions with all 21
users, there were 13 collisions out of $45s\times 10Hz\times 4\text{
trials}\times 21\text{ users}=37800$ timesteps, for an empirical safety
probability of $99.96\%$ (most of the collisions occurred because the camera
lost track of the human’s movement for a few timesteps prior).

Qualitative Results: To test H2, we ran a repeated measures ANOVA on the
effect of the robot type on the survey question “The robot was easy to
collaborate with,” but found no significant differences
($F(2,40)=2.9,p=0.06$), since users rated all three as easy to collaborate
with. We did, however, find significant differences between the robots on the
“Accounted” (Table III, $F(2,40)=12.4,p<0.0001$) and “Changed” questions
($F(2,40)=18.5,p<0.0001$). A post-hoc Bonferroni test tells us that, relative
to the naive robot, participants thought the proactive ($p=0.0006$) and
reactive ($p=0.0005$) robots accounted more for their goal selection. This
seems to support H2, although participants tended to think the proactive robot
was about as responsive to their goal selection as the reactive robot. A post-
hoc Bonferroni test tells us that participants felt that they changed their
goals around the naive robot, while they did not change their goals around the
proactive robot ($p=0.01$) or around the reactive robot ($p<0.0001$), also
supporting H2. They also felt that they changed their goal less with the
reactive robot than the proactive robot ($p=0.002$).

We additionally look at users’ rankings of the three robots to test H2. While
not a majority, a plurality of participants (47%) ranked the proactive robot
as the most preferred interaction partner when asked to rank all three robots,
which supports H2. Treating the participants’ rankings as votes for the robots
and running an instant-runoff-voting election shows that the proactive robot
is the winner of the “election.”

##  X Conclusion and Future Work

We proposed a Bayesian-inference-based formulation of conditional behavior
prediction that can allow a robot to reason about the affect its future
actions will have on a human in collaborative multi-goal settings. The CBP
formulation was based on the insight that we can decouple the human’s prior
goal selection $\theta_{H}^{prior}$ from their posterior goal selection
$\theta_{H}^{post}$ that accounts for the robot’s actions. We additionally
discussed how to utilize a CBP model for long-term safety as well as for
switching between courteous and influential behavior. In simulations, we found
that our proposed controller allows the robot to be efficient at interacting
with different kinds of humans (even without tuning the score function
weights) and that it performs very similarly to a learning-based CBP model
without the need for a large dataset. We tested our proposed controller in a
user study and found that although users tended to perform the best with the
naive baseline robot, users tended to prefer interacting with our proactive
controller. Future work may try to improve the model-based CBP approach by
adapting the score function weights online, which would keep the data-
efficiency benefits while potentially increasing the robot’s efficiency at
acting with different kinds of real human partners.

## Acknowledgments

This material is based upon work supported by the National Science Foundation
(NSF) Graduate Research Fellowship under Grant No. DGE1745016 and DGE2140739
and additionally under Grant No. 2144489 as well as the Manufacturing Futures
Institute, Carnegie Mellon University, through a grant from the Richard King
Mellon Foundation. Any opinions, findings, and conclusions or recommendations
expressed in this material are those of the authors and do not necessarily
reflect the views of the NSF.

## References

  * [1] C. L. Baker, J. B. Tenenbaum, and R. R. Saxe, “Goal inference as inverse planning,” in _Proceedings of the Annual Meeting of the Cognitive Science Society_ , vol. 29, no. 29, 2007.
  * [2] C. L. Baker, R. Saxe, and J. B. Tenenbaum, “Action understanding as inverse planning,” _Cognition_ , vol. 113, no. 3, pp. 329–349, 2009.
  * [3] E. Tolstaya, R. Mahjourian, C. Downey, B. Vadarajan, B. Sapp, and D. Anguelov, “Identifying driver interactions via conditional behavior prediction,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 3473–3479.
  * [4] J. F. Fisac, E. Bronstein, E. Stefansson, D. Sadigh, S. S. Sastry, and A. D. Dragan, “Hierarchical game-theoretic planning for autonomous vehicles,” in _2019 International conference on robotics and automation (ICRA)_. IEEE, 2019, pp. 9590–9596.
  * [5] Z. Zhang, A.-S. Maitham, H. Jing, H. Shirado, and Y. Nakahira, “Rethinking safe control in the presence of self-seeking humans,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 12, 2023, pp. 15 331–15 339.
  * [6] F. Bornhorst, A. Ichino, O. Kirchkamp, K. H. Schlag, and E. Winter, “How do people play a repeated trust game?: Experimental evidence,” _None_ , 2004\.
  * [7] L. Bruttel and U. Kamecke, “Infinity in the lab. how do people play repeated games?” _Theory and Decision_ , vol. 72, pp. 205–219, 2012.
  * [8] S. Gächter, “Behavioral game theory,” _Blackwell handbook of judgment and decision making_ , pp. 485–503, 2004.
  * [9] A. Y. Ng, S. Russell _et al._ , “Algorithms for inverse reinforcement learning.” in _Icml_ , vol. 1, 2000, p. 2.
  * [10] A. Bestick, R. Pandya, R. Bajcsy, and A. D. Dragan, “Learning human ergonomic preferences for handovers,” in _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2018, pp. 3257–3264.
  * [11] D. P. Losey and M. K. O’Malley, “Enabling robots to infer how end-users teach and learn through human-robot interaction,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 1956–1963, 2019.
  * [12] D. Vasquez, T. Fraichard, O. Aycard, and C. Laugier, “Intentional motion on-line learning and prediction,” _Machine Vision and Applications_ , vol. 19, no. 5-6, pp. 411–425, 2008.
  * [13] J. F. Fisac, A. Bajcsy, S. L. Herbert, D. Fridovich-Keil, S. Wang, C. J. Tomlin, and A. D. Dragan, “Probabilistically safe robot planning with confidence-based human predictions,” _arXiv preprint arXiv:1806.00109_ , 2018\.
  * [14] J. Mainprice and D. Berenson, “Human-robot collaborative manipulation planning using early prediction of human motion,” in _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2013, pp. 299–306.
  * [15] S. Pellegrinelli, H. Admoni, S. Javdani, and S. Srinivasa, “Human-robot shared workspace collaboration via hindsight optimization,” in _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2016, pp. 831–838.
  * [16] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone, “Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16_. Springer, 2020, pp. 683–700.
  * [17] N. Rhinehart, R. McAllister, K. Kitani, and S. Levine, “Precog: Prediction conditioned on goals in visual multi-agent settings,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 2821–2830.
  * [18] E. Schmerling, K. Leung, W. Vollprecht, and M. Pavone, “Multimodal probabilistic model-based planning for human-robot interaction,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 3399–3406.
  * [19] C. Tang, W. Zhan, and M. Tomizuka, “Interventional behavior prediction: Avoiding overly confident anticipation in interactive prediction,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 11 409–11 415.
  * [20] H. Yu, Z. Kuang, F. Sun, J. Yuan, M. Liu, and M. Xie, “Path planning for mobile robots with courteous behaviors in domestic environments,” in _2021 40th Chinese Control Conference (CCC)_. IEEE, 2021, pp. 3894–3899.
  * [21] D. Sadigh, S. Sastry, S. A. Seshia, and A. D. Dragan, “Planning for autonomous cars that leverage effects on human actions.” in _Robotics: Science and systems_ , vol. 2. Ann Arbor, MI, USA, 2016, pp. 1–9.
  * [22] L. Sun, W. Zhan, M. Tomizuka, and A. D. Dragan, “Courteous autonomous cars,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 663–670.
  * [23] A. Bestick, R. Bajcsy, and A. D. Dragan, “Implicitly assisting humans to choose good grasps in robot to human handovers,” in _2016 International Symposium on Experimental Robotics_. Springer, 2017, pp. 341–354.
  * [24] A. Xie, D. Losey, R. Tolsma, C. Finn, and D. Sadigh, “Learning latent representations to influence multi-agent interaction,” in _Conference on robot learning_. PMLR, 2021, pp. 575–588.
  * [25] W. Z. Wang, A. Shih, A. Xie, and D. Sadigh, “Influencing towards stable multi-agent interactions,” in _Conference on robot learning_. PMLR, 2022, pp. 1132–1143.
  * [26] S. Robla-Gómez, V. M. Becerra, J. R. Llata, E. Gonzalez-Sarabia, C. Torre-Ferrero, and J. Perez-Oria, “Working together: A review on safe human-robot collaboration in industrial environments,” _Ieee Access_ , vol. 5, pp. 26 754–26 773, 2017.
  * [27] A. M. Zanchettin, N. M. Ceriani, P. Rocco, H. Ding, and B. Matthias, “Safety in human-robot collaborative manufacturing environments: Metrics and control,” _IEEE Transactions on Automation Science and Engineering_ , vol. 13, no. 2, pp. 882–893, 2015.
  * [28] F. Ferraguti, C. T. Landi, S. Costi, M. Bonfè, S. Farsoni, C. Secchi, and C. Fantuzzi, “Safety barrier functions and multi-camera tracking for human–robot shared environment,” _Robotics and Autonomous Systems_ , vol. 124, p. 103388, 2020.
  * [29] C. Liu and M. Tomizuka, “Safe exploration: Addressing various uncertainty levels in human robot interactions,” in _2015 American Control Conference (ACC)_. IEEE, 2015, pp. 465–470.
  * [30] Z. Wang, H. Jing, C. Kurniawan, A. Chern, and Y. Nakahira, “Myopically verifiable probabilistic certificate for long-term safety,” in _2022 American Control Conference (ACC)_. IEEE, 2022, pp. 4894–4900.
  * [31] R. Liu and C. Liu, “Human motion prediction using adaptable recurrent neural networks and inverse kinematics,” _IEEE Control Systems Letters_ , vol. 5, no. 5, pp. 1651–1656, 2020.
  * [32] R. Pandya and C. Liu, “Safe and efficient exploration of human models during human-robot interaction,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 6708–6715.
  * [33] R. Tian, M. Tomizuka, A. D. Dragan, and A. Bajcsy, “Towards modeling and influencing the dynamics of human learning,” in _Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ , 2023, pp. 350–358.
  * [34] A. D. Dragan, K. C. Lee, and S. S. Srinivasa, “Legibility and predictability of robot motion,” in _2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 2013, pp. 301–308.
  * [35] E. M. Van Zoelen, E. I. Barakova, and M. Rauterberg, “Adaptive leader-follower behavior in human-robot collaboration,” in _2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 2020, pp. 1259–1265.
  * [36] C. Messeri, A. M. Zanchettin, P. Rocco, E. Gianotti, A. Chirico, S. Magoni, and A. Gaggioli, “On the effects of leader-follower roles in dyadic human-robot synchronisation,” _IEEE Transactions on Cognitive and Developmental Systems_ , 2020.
  * [37] A. Bobu, A. Bajcsy, J. F. Fisac, S. Deglurkar, and A. D. Dragan, “Quantifying hypothesis space misspecification in learning from human–robot demonstrations and physical corrections,” _IEEE Transactions on Robotics_ , vol. 36, no. 3, pp. 835–854, 2020.
  * [38] S. He, W. Zhao, C. Hu, Y. Zhu, and C. Liu, “A hierarchical long short term safety framework for efficient robot manipulation under uncertainty,” _Robotics and Computer-Integrated Manufacturing_ , vol. 82, p. 102522, 2023\.
  * [39] T. Wei and C. Liu, “Safe control algorithms using energy functions: A unified framework, benchmark, and new directions,” in _2019 IEEE 58th Conference on Decision and Control (CDC)_. IEEE, 2019, pp. 238–243.
  * [40] S. H. Huang, I. Huang, R. Pandya, and A. D. Dragan, “Nonverbal robot feedback for human teachers,” _arXiv preprint arXiv:1911.02320_ , 2019.
  * [41] D. Helbing and P. Molnar, “Social force model for pedestrian dynamics,” _Physical review E_ , vol. 51, no. 5, p. 4282, 1995.
  * [42] S. H. Huang, D. Held, P. Abbeel, and A. D. Dragan, “Enabling robots to communicate their objectives,” _Autonomous Robots_ , vol. 43, pp. 309–326, 2019.
  * [43] S. Nikolaidis, D. Hsu, and S. Srinivasa, “Human-robot mutual adaptation in collaborative tasks: Models and experiments,” _The International Journal of Robotics Research_ , vol. 36, no. 5-7, pp. 618–634, 2017.
  * [44] S. Devin, A. Clodic, and R. Alami, “About decisions during human-robot shared plan achievement: Who should act and how?” in _Social Robotics: 9th International Conference, ICSR 2017, Tsukuba, Japan, November 22-24, 2017, Proceedings 9_. Springer, 2017, pp. 453–463.
  * [45] R. Likert, “A technique for the measurement of attitudes.” _Archives of psychology_ , 1932.

##  Appendix A Human’s Bayesian Goal Prediction

The human assumes the robot is noisily rational in choosing its actions for an
intended goal (or that the robot is exponentially more likely to choose an
action if it has a higher Q-value):

$p(u_{R}^{t}\mid
x_{R}^{t};\theta)=\frac{e^{\beta_{H}Q(x_{R}^{t},u_{R}^{t};\theta)}}{\int_{u_{R}^{{}^{\prime}}}e^{\beta_{H}Q(x_{R}^{t},u_{R}^{{}^{\prime}};\theta)}},$
(22)

where $\beta_{H}$ is the rationality coefficient (sometimes called the
“inverse temperature” parameter or the “model confidence”). In general, the
integral in the denominator can be challenging to compute, but the simulated
human makes use of the robot’s baseline LQR controller to define the reward
function as the instantaneous LQR cost:

$r_{R}(x_{R},u_{R};\theta)=-(x_{R}-\theta)^{T}Q(x_{R}-\theta)-u_{R}^{T}Ru_{R},$
(23)

so the $Q-$function is then the negative optimal cost-to-go:

$Q_{R}(x_{R},u_{R};\theta)=r(x_{R},u_{R};\theta)-(x^{\prime}-\theta)^{T}P(x^{\prime}-\theta)$
(24)

where $x^{\prime}=Ax_{R}+Bu_{R}$ and $P$ is the solution to the discrete-time
algebraic Ricatti equation (DARE):

$P=A^{T}PA-A^{T}PB(R+B^{T}PB)^{-1}B^{T}PA+Q.$ (25)

Following prior work [33], this allows us to compute an exact form of the
denominator of (22):

$\begin{split}&\int e^{\beta_{R}Q_{R}(x_{R},u_{R};\theta)}\\\
&=e^{-\beta_{R}(x_{R}-\theta)^{T}P(x_{R}-\theta)}\sqrt{\frac{(2\pi)^{m}}{\text{det}(2\beta_{R}R+2\beta_{R}B^{T}PB)}}.\end{split}$
(26)

The human then uses this likelihood function (and Bayes Rule) to update its
belief given a new observation $(x_{R}^{t},u_{R}^{t})$:

$\begin{split}b_{H}^{t}(\theta_{i})&=p(\theta_{i}\mid
x_{R}^{0:t},u_{R}^{0:t})\\\ &=\frac{p(u_{R}^{t}\mid
x_{R}^{t};\theta_{i})p(\theta_{i}\mid
x_{R}^{0:t-1},u_{R}^{0:t-1})}{\sum_{\theta^{{}^{\prime}}}p(u_{R}^{t}\mid
x_{R}^{t};\theta^{{}^{\prime}})p(\theta^{{}^{\prime}}\mid
x_{R}^{0:t-1},u_{R}^{0:t-1})}\end{split}$ (27)

##  Appendix B Learning-based CBP Details

The basic form of these models is that they take in a trajectory history for
all agents to as well as the future trajectory of the robot and outputs a
prediction about the non-ego agents (the human in our case). We structure our
learning-based model similarly, it takes in the human and robot past
trajectories $x_{H}^{t-k:t},x_{R}^{t-k:t}$ as well as the robot’s future plan
$x_{R}^{t+1:t+T}$. The trajectories are fed into LSTM layers with attention,
and finally all pieces are concatenated with the set of goals
$\\{\theta_{1},\ldots,\theta_{N}\\}$ and passed into linear layers to output
the probability of the human reaching each goal
$[P(\theta_{1}),\ldots,P(\theta_{N})]$. The network is trained to output the
empirical probability of the human reaching each potential goal conditioned on
the robot’s plan; it is trained with an MSE loss.

These papers have generally been in spaces where there are large pre-existing
datasets to train on, like in social navigation and autonomous driving. We’re
instead focusing on human-robot collaborative tasks such as manufacturing and
in-home robots, so such datasets are hard to come by. As a result, we create
our own dataset for our simulated human-robot collaboration task. The dataset
consists of 3.5 million data points, which corresponds to approximately 97
hours of data collected for this particular human-robot interaction. This kind
of data would be impractical to collect on a real human-robot interaction, so
we use this as a baseline in our simulations but not in our user studies. We
use the same goal-selection method (14) for this approach as our proposed
controller, which works since this method is also a conditional prediction
method.

| Naive | Reactive | Proactive | F-score | p-value
---|---|---|---|---|---
“I felt like the robot accounted for the [goal] I wanted to pick when |  |  |  |  |
it was picking a [goal].” | $2.1$ | $\mathbf{4.0}$ | $3.7$ | $F(2,40)=12.4$ | $p<0.01$
“I often changed which [goal] I picked initially because of the robot.” | 3.7 | 1.7 | 2.7 | $F(2,40)=18.5$ | $p<0.01$
“I felt like the robot and I scored as many points as we could.” | 3.6 | 4.1 | 3.0 | $F(2,40)=4.0$ | $p=0.02$
“The robot influenced me to pick good [goals] for the team.” | 3.0 | 2.6 | 3.2 | $F(2,40)=1.8$ | $p=0.18$
“The robot’s choice of [goals] made me choose worse [goals] for the team.” | 3.3 | 1.8 | 2.5 | $F(2,40)=10.0$ | $p<0.01$
“The robot was easy to collaborate with.” | 3.3 | 4.1 | 3.4 | $F(2,40)=2.9$ | $p=0.06$
“I felt like the robot picked the best [goals] to grab for the team.” | 3.2 | 4.0 | 2.7 | $F(2,40)=8.1$ | $p<0.01$
“I felt like the robot hindered the team’s performance.” | 2.8 | 2.2 | 3.0 | $F(2,40)=2.6$ | $p=0.08$
TABLE V:  Responses to subjective survey questions (5: Strongly Agree, 1:
Strongly Disagree).

##  Appendix C Safe Trajectory Generation

Algorithm 1 Safe Robot Control Pipeline with Model-Based CBP

Given: $\\{G_{1},...,G_{N}\\}$ $\triangleright$ goal locations

Given: $x_{H}^{0},x_{R}^{0}$ $\triangleright$ agent starting positions

$b_{R}^{0}(\theta_{H}^{prior})\leftarrow\text{uniform}$
$\hat{b}_{H}^{0}(\theta_{R})\leftarrow\text{uniform}$ $\triangleright$ initial
robot belief and mental model

while $t<H$ do $\triangleright$ $H$ is the trajectory horizon

$b_{R}^{t}(\theta_{H}^{prior})\leftarrow(x_{H}^{t},x_{R}^{t},u_{H}^{t})$,
$\hat{b}_{H}^{t}(\theta_{R})\leftarrow(x_{R}^{t},u_{R}^{t})$ $\triangleright$
robot updates nominal belief and mental model

$b_{R}^{t}(\theta_{H}^{post}\mid\theta_{R})\leftarrow\sum_{\theta_{H}^{prior}}p(\theta_{H}^{post}\mid\theta_{H}^{prior},x_{H}^{0:t},x_{R}^{0:t},u_{H}^{0:t},\theta_{R})b^{t}_{R}(\theta_{H}^{prior})$
$\triangleright$ CBP belief update

$p(\theta_{H}^{post})=\hat{b}_{H}^{t}(\theta_{R})b_{R}^{t}(\theta_{H}^{post}\mid\theta_{R})$
$\triangleright$ robot computes overall goal probabilities

$\mathcal{X}_{R}\leftarrow trajGen(x_{H}^{t},x_{R}^{t},p(\theta_{H}^{post}))$
$\triangleright$ generate candidate safe trajectories

$\mathbf{x}_{R}^{*},\theta_{R}^{*}\leftarrow\underset{(\mathbf{x}_{R},\theta_{R})\in\mathcal{X}_{R}}{\text{argmin
}}J(\mathbf{x}_{R},b_{R}^{t}(\theta_{H}^{post}\mid\theta_{R}),b_{R}^{t}(\theta_{H}^{prior}))$
$\triangleright$ choose (traj, goal) pair that minimizes cost

$u_{R}^{t}\leftarrow\mathbf{x}_{R}^{u,0}$ $\triangleright$ choose first
control action from trajectory

$x_{R}^{t+1}\leftarrow f_{R}(x_{R}^{t},u_{R}^{t})$

end while

We propose a long-term safe controller to ensure the probability of safety
over the time horizon $H$ is always above the desired threshold $1-\epsilon$,
as stated in the objective (3). The controller is summarized in Algorithm 2.
The key procedures at each time step are: 1. estimate the probability of
safety for the current state 2. find the closest state that is safe enough if
the initial state is not 3. execute goal-pursing control plus a potential
field safe control that drives the robot to safe regions. Note that when
estimating the safety probability, we assume a greedy human that has no robot-
avoidance control to cover the worst case.

###### Remark C.1

Assuming the human intention prediction $p(\theta_{H}^{post})$ is correct, the
proposed long-term safe control (Algorithm 2) is guaranteed to meet the design
objective (3). Similar theoretical derivations can be found in [30, 39].

To achieve more efficient real-time implementation, we use Gaussian
distribution to approximate the uncertainty in human’s motion at each time
step $t$. Specifically, the mean of the Gaussian is the expected nominal
motion of the human $x_{H}$ and the variance will be some estimated value
$\sigma^{2}$. For trajectory generation, we are interested in the long-term
safety of the human-robot interaction in the sense that we want the generated
trajectories to be safe at all time within a horizon $H$. Given the initial
level of uncertainty $\sigma^{(i)}$ for each human mode $i$, we know that the
uncertainty at time step $t$ is $\sigma^{t(i)}=\sqrt{t}\sigma^{(i)}$. For each
possible goal $\theta_{H}^{(i)}$ of human, let $x_{H}^{0:H(i)}$ be the nominal
trajectory of the human starting from $x_{H}^{0}$ pursuing $\theta_{H}^{(i)}$.
The probability of human choosing $\theta_{H}^{(i)}$ is given by
$\theta_{H}^{(i){post}}$. The goal is to generate $x_{R}^{0:H}$ such that the
following safety condition is satisfied

$\phi_{d_{\text{min}}+\sigma^{t(i)}}(x^{t})\geq 0,\forall t<H,\;\forall i.$
(28)

For robot trajectory generation, we introduce the potential field controller
to generate candidate trajectories that are more likely to be safe. The idea
of potential field controller is that it will impose repelling forces to the
controlled agent if it is close to the unsafe region. For example, if we want
the robot’s position $x_{R}$ to be away from the human’s position $x_{H}$, we
will have the potential field control for the robot to be

$u_{\text{pf}}=\frac{\gamma}{d^{2}}(C_{H}x_{H}-C_{R}x_{R})=K_{\text{pf}}(x_{H}-x_{R}),$
(29)

where $d$ is the distance between human and robot and $\gamma$ is the repel
force of the potential field controller. For simplicity we assume
$C_{H}=C_{R}$ and use $K_{\text{pf}}$ to denote the coefficient. Similarly, we
can define $o$ to be some obstacles to be avoided and use (29) to find the
safe control by replacing $x_{H}$ with $o$.

The overall safe trajectory generation algorithm in shown in Algorithm 3,
where we use potential field controller to avoid different possible human
motions given different goals, and use synthetic obstacles in the state space
to diverse the trajectory. Safety is identified at each time step $t$ with
uncertainty level $\sigma\sqrt{t}$, which characterizes the growing
uncertainty bound over time. Experiments show that synthetic obstacles close
to the initial position of the robot will give more diverse trajectories.

Algorithm 2 Long-term Safe Control

procedure safeControl($x_{H}^{t},x_{R}^{t},\theta_{R},\theta_{H},H$)

$F\leftarrow$ long_term_safe_prob $(x_{H}^{t},x_{R}^{t},\theta_{H},H)$

if $F>1-\epsilon$ then

$u_{R}^{t}\leftarrow K(x_{R}^{t}-\theta_{R})$ $\triangleright$ goal pursuing
control

else

$x_{\text{safe}}\leftarrow$ find_safe_state
$(x_{H}^{0},x_{R}^{0},\theta_{H},H,\epsilon)$

$u_{R}^{t}\leftarrow
K(x_{R}^{t}-\theta_{R})-K_{\text{pf}}(x_{R}^{t}-x_{\text{safe}})$
$\triangleright$ potential field safe control

end if

return $u_{R}^{t}$ $\triangleright$ return trajectories

end procedure

procedure longTermSafeProb($x_{H}^{0},x_{R}^{0},\theta_{R},\theta_{H},H$)

Given: $n$ $\triangleright$ number of episode to estimate safety probability

Initialize $\\{F^{(1)},F^{(2)},\cdots,F^{(n)}\\}=\mathbf{1}$ $\triangleright$
record safety of each trajectory

for $k=1:n$ do

$\theta_{H}^{*}\leftarrow\text{sample}(\theta_{H})\text{ from
}p(\theta_{H}^{post})$

for $t=1:H$ do

$u_{R}^{t}\leftarrow K(x_{R}^{t}-\theta_{R})$ $\triangleright$ goal pursuing
control for robot

$u_{H}^{t}\leftarrow K(x_{H}^{t}-\theta_{H}^{*})$ $\triangleright$ goal
pursuing control for human

$x_{R}^{t+1}\leftarrow\text{step}(x_{R}^{t},u_{R}^{t})$

$x_{H}^{t+1}\leftarrow\text{step}(x_{H}^{t},u_{H}^{t})$ $\triangleright$ human
dynamics with disturbance

if $\|x_{R}^{t}-x_{H}^{t}\|<d_{\text{min}}$ then

$F^{(k)}=0$

break

end if

end for

end for

return $\text{mean}(F)$

end procedure

procedure
findSafeState($x_{H}^{0},x_{R}^{0},\theta_{R},\theta_{H},H,\epsilon$)

Given: $n$ $\triangleright$ number of sampling states for a certain radius

Initialize $r=1$, $s=0$ $\triangleright$ searching radius and indicator of
safe state found

while $s=0$ do

for $k=1:n$ do

$x_{R}^{\prime}\leftarrow$ uniform_sample $(x_{R},r,k)$ $\triangleright$
sample near robot’s position with radius $r$

$F\leftarrow$ long_term_safe_prob $(x_{H}^{0},x_{R}^{\prime},\theta_{H},H)$

if $F>1-\epsilon$ then

$s=1$ $\triangleright$ mark safe state found

return $\\{x_{R}^{\prime},F\\}$ $\triangleright$ return the safe state and its
safety probability

end if

end for

end while

end procedure

Algorithm 3 Candidate Trajectory Generation

procedure trajGen($x_{H}^{0},x_{R}^{0},\theta_{R},\theta_{H}^{prior}$)

Given: $\\{\sigma^{(1)},\sigma^{(2)},...,\sigma^{(N)}\\}$ $\triangleright$
uncertainties in each mode of the human

$\\{o_{1},o_{2},...,o_{M}\\}\leftarrow$ sample_obstacle$(x_{R})$
$\triangleright$ generate synthetic obstacles

Initialize $\\{s_{1},s_{2},...,s_{M}\\}=\mathbf{1}$ $\triangleright$ record
safety of each trajectory

for $o_{m}\text{ in }\\{o_{1},o_{2},...,o_{M}\\}$ do $\triangleright$ loop
over all synthetic obstacles

for $t=1:H$ do

$u_{R}^{t}\leftarrow K(x_{R}^{t}-\theta_{R})$ $\triangleright$ goal pursuing
control

$u_{R}^{t}\leftarrow u_{R}^{t}+K_{\text{pf}}(x_{R}^{t}-o_{m})$
$\triangleright$ potential field control against obstacle

for $\theta_{H}^{(i)}\text{ in }\\{\theta_{H}^{(1)},...,\theta_{H}^{(N)}\\}$
do $\triangleright$ loop over all human’s goals

if $\|x_{R}^{t}-x_{H}^{t(i)}\|_{2}\leq d_{\text{min}}+\sigma^{(i)}\sqrt{t}$
then $\triangleright$ check safety

$s_{i}\leftarrow 0$

break

else

$u_{R}^{t}\leftarrow
u_{R}^{t}+\theta_{H}^{(i)prior}K_{\text{pf}}(x_{R}^{t}-x_{H}^{t(i)})$
$\triangleright$ potential field control against each human mode

$u_{H}^{t(i)}\leftarrow K(x_{H}^{t(i)}-\theta_{H}^{(i)})$ $\triangleright$
goal pursuing control for human

$x_{H}^{t+1(i)}\leftarrow\text{step}(x_{H}^{t(i)},u_{H}^{t(i)})$

end if

end for

$x_{R}^{t+1}\leftarrow\text{step}(x_{R}^{t},u_{R}^{t})$

end for

if $s_{i}=1$ then

$\\{X_{R},U_{R}\\}\leftarrow[\\{X_{R},U_{R}\\},\\{x_{R},u_{R}\\}]$
$\triangleright$ record safe trajectory

end if

end for

return $\\{X_{R},U_{R}\\}$ $\triangleright$ return trajectories

end procedure

##  Appendix D Subjective Questions for User Study

We ran a one-way repeated measures ANOVA for the effect of the robot type on
each survey question listed in Table V and ran post-hoc tests with Bonferroni
tests. As noted in the main text, we found significant differences for the
“Accounted” and “Changed” questions. We additionally found significant
differences for the questions about scoring as many points as they could,
influencing negatively, and picking the best goals for the team. Participants
rated that they felt like the team scored as many points as possible more with
the reactive robot than the proactive robot ($p<0.01$), which is interesting
given that participants statistically performed the worst with the reactive
robot. Participants also rated that they choose worse goals with the naive
robot than the reactive robot ($p<0.01$) and thought both the CBP and reactive
robots were not making them choose worse goals. They additionally felt like
the reactive robot picked better goals for the team than the CBP robot
($p<0.01$), which is again interesting because it does not align with the
actual performance on the task. This seems to indicate that the peoples’
subjective opinions of a collaborator may not be based precisely on
performance, but on other social factors such as perceived intelligence or
predictability. We also see that on these subjective questions, the reactive
robot is most often the highest rated, but when forced to choose a preferred
collaborator, they tended to choose the CBP model. This may be because they
felt it was the most capable collaborator overall, even though they felt that
the reactive robot was responding well to their strategies.