# Explainable AI in Request-for-Quote

1st Qiqin Zhou  Operation Research and Information Engineering (Cornell
University)
Ithaca, New York
<EMAIL_ADDRESS>
###### Abstract

In the contemporary financial landscape, accurately predicting the probability
of filling a Request-For-Quote (RFQ) is crucial for improving market
efficiency for less liquid asset classes. This paper explores the application
of explainable AI (XAI) models to forecast the likelihood of RFQ fulfillment.
By leveraging advanced algorithms including Logistic Regression, Random
Forest, XGBoost and Bayesian Neural Tree, we are able to improve the accuracy
of RFQ fill rate predictions and generate the most efficient quote price for
market makers. XAI serves as a robust and transparent tool for market
participants to navigate the complexities of RFQs with greater precision.

###### Index Terms:

Explainable AI, Deep Learning, Machine Learning, Neural Network, Boosting
Tree, Random Forest

##  I Introduction

Request-For-Quote (RFQ) is a form of trading where market participants can
obtain electronic quotes on any asset class. RFQ allows for more efficient
price discovery for less liquid asset classes such as To-be-Announcement (TBA)
with a specific coupon. TBA is a forward contract on mortgage-backed
securities that is widely used in the housing finance industry to protect
against interest rate fluctuations during the mortgage lending process.
Providing pricing for broader range of TBA bonds through RFQ is particularly
advantageous for mortgage lenders, as traditional market makers typically only
show the price of the most liquid coupons. The pricing from RFQ enhances the
transparency in the TBA market and it in turn provides crucial data for
generating precise valuation for borrowers in the primary mortgage origination
market, ultimately offering mortgage borrowers better pricing and financing
options in a tough market.

The objective of the paper is to create an algorithm that allows the market-
maker to produce a right price of TBA to fill an RFQ while maximizing market-
maker’s utility and minimizing their inventory risk. The algorithm is
naturally decomposed into two components: (a) evaluate the probability of
filling an RFQ given a price and (b) determine the final price market-maker
should set for the RFQ so as to maximize their utility.

In Section III, we first proposed a simulation algorithm to produce
hypothetical RFQ data that are otherwise not available to the public because
of the regulatory constraints that limit the sharing and distribution of real
RFQ data. Based on the synthesized dataset, we perform data analysis and
feature engineering to disclose the pattern of RFQ fill rate in Section IV.

We address the problem of (a) in Section V-A by experimenting different
machine learning and Explainable Artificial Intelligence (XAI) models to
predict the probability of filling an RFQ. By leveraging a set of
explainability techniques in machine learning and deep learning, we are able
to enhance the transparency and reliability of predicting probability of
filling the RFQ. It helps explain how each model input contributes to the
final outcome. We compared various XAI models including decision trees, random
forests and gradient boosting. We established a base case of traditional
logistic regression models constrained by LASSO regularization. Our findings
indicate that XAI methods can achieve comparable predictive performance while
maintaining explainability, thus facilitating better decision-making in
financial contexts. XAI has strong practical implications in helping financial
institutions to establish a trading and risk management system compliant with
regulatory requirements. It is also pivotal to improve the responsible use of
AI in financial activities.

To address problem (b) in Section VI-A, we proposed a market-maker utility
function that involves both the probability and profit margin of filling an
RFQ. To predict the profit margin, we constructed a parsimonious model to
predict the next middle price (mid-price) given the current information. Note
that we do not intend to put emphasis on this price prediction model
considering the lack of real-world intraday market data and the widely-
acknowledged complexity of predicting the high noise-to-signal intraday price.

##  II Literature Review

###  II-A RFQ

The Request-For-Quote (RFQ) process allows market-makers to respond to trade
requests from a set of counterparties. A counterparty initiates an RFQ
detailing information on the trade request, which will typically include the
instrument identifier, the size of the transaction, the side (buy or sell) and
etc. The RFQ is sent to a selection of market-makers (where the counterparty
will choose the optimal list based on past experience, hit-ratios). Upon
receiving the RFQ, a market-maker will quote back a specific price at which
the are willing to execute the trade (yet each market-maker is not aware of
the prices from their competitor market-makers). When the counterparty is
ready to execute, the market-maker with the best price will win the trade. Of
note, the marketmaker will then have the other side of the trade and will need
to manage the risk and hedge risk appropriately.

###  II-B Explainable Artificial Intelligence

Predictive modeling in finance has evolved significantly with the advent of
machine learning and Artificial Intelligence [1, 4]. Traditional predictive
models often prioritize accuracy over explainability, leading to black-box
solutions that are difficult to understand and trust [5]. Explainable AI (XAI)
addresses this issue by providing transparency in model predictions, which is
essential for compliance with regulatory standards and for gaining the trust
of market participants [6]. Recent advancements in XAI have made it possible
to balance predictive power with explainability [7]. From the existing
literature, we found there are two major categories of XAI: post-hoc
explainability and intrinsic explainability.

Post-hoc explainability refers to model-agnostic techniques we can utilize to
improve explainability. These include Local explainable Model-agnostic
Explanations (LIME) [5], SHAP (Shapley Additive explanations) values [7],
Partial Dependence Plots (PDPs) [8], and Feature Permutation Importance [9].
However, they suffer from several shortcomings. The first problem is the
approximation errors as many of these explainability techniques rely on
approximations or simplifications of the model [10]. The next problem is
scalability as applying these techniques to large datasets or complex models
is extremely computationally challenging [10]. There are also limitations
specific to each technique. For Feature Permutation Importance, the
measurement of feature importance, such as Gini importance in Random Forests,
can be biased towards features with more categories or higher cardinality
[12]. PDPs assume that features are independent of each other, which is often
not true in real-world data, leading to misleading interpretations [13]. The
results of LIME are sensitive to how the local neighborhood around the
prediction is defined, making LIME explanations unstable and providing
different explanations for similar predictions if the local neighborhood
changes slightly [5].

Intrinsically explainable machine learning models are those whose structure
and functioning are inherently understandable and explainable. Intrinsic model
explainability is arguably the most desirable option as it addresses all of
the objectives to justify, control, discover, and improve. These models are
typically simpler, and their predictions can be easily traced back to the
input features. Logistic regression is one such explainable model that can be
used for binary and multiclass classification [16]. The coefficients in a
logistic model indicate the influence of each feature on the model outcome.
Decision Tree or general tree models are also intrinsically explainable, as
their tree nodes represent feature inputs, branches represent the decision
rules, and leaves represent outcomes [17]. The tree structure allows for a
straightforward tracing of the decision-making process from the root to the
leaf. In addition, Naive Bayes is a probabilistic classifier based on Bayes’
theorem with a strong assumption of feature independence. Its decisions can be
understood by examining the contributions of each feature to the posterior
probabilities [18].

The above-mentioned models are all computationally efficient and explainable
but often require careful feature engineering procedure to achieve a good
performance as they are not as good at capturing complex, non-linear
relationships in the data as more complex models can do. Additionally,
unsupervised machine learning algorithms are also utilized as explainable
model in finance such as using K-means to explicitly segment a covariance-
tilted portfolio into sub-portfolios to maintain the stability of optimization
[3].

To balance the trade-off between model performance and model explainability,
we propose a new model framework: Bayesian Neural Tree (BNT). It is a hybrid
model that integrates the explainability of decision trees with the powerful
predictive capabilities of neural networks, framed within a Bayesian context
to manage uncertainty and provide probabilistic predictions. The model
leverages a tree structure where each node represents a neural network,
allowing for complex decision boundaries while maintaining a hierarchical,
explainable form. By incorporating Bayesian inference, BNTs can quantify
uncertainty in predictions, making them particularly useful in applications
requiring robust decision-making under uncertainty [1]. This approach has
shown promise in various fields, including finance and healthcare, where
understanding the confidence in predictions is as crucial as the predictions
themselves [2].

##  III Data Generation Process

Request-for-quote (RFQ) data is often not publicly available because it
pertains to proprietary Information, data privacy and sensitivity. Therefore,
we developed a simulation algorithm to synthetically construct a series of RFQ
data for TBAs. In total we generated 10005 records where the last 5 records
represent real-world RFQs we want to quote.

The variables we generated include: the timestamp of the initiation of the RFQ
(Time), the unique identified the the bond (Bond), the direction of the RFQ:
offer or bid (Side), bond notional (Notional), the id of target counterparty
(CounterParty), the mid-price which is the average of level-1 bid and ask
price (MidPrice), the quoted price of RFQ (QuotedPrice), number of competitors
at the time (Competition), the status of the RFQ: missed or done (Status) and
the mid price at in the following minute (NextMidPrice).

Time is a continuous 5-digit integer that mimics the format of timestamp sent
from the exchange. Notional is an integer generated in $\log 10$ scale from 3
to 7. Side is binary variable and we assume it follows a a Bernoulli
distribution with probability $p$ = 0.5. We assume there are in all 4
participants in RFQ so CounterParty is generated uniformly from $[0,3]$.
Competition is generated uniformly from $[1,4]$ since there are at most 4
participants in the RFQ process.

Status is the target of our prediction and is also a binary variable. We
assume it follows a Bernoulli distribution that is independent of Side. We
used the following steps to generate it: Let $\mathbf{X}$ be defined as

$\mathbf{X}=\begin{pmatrix}(1+0.3N)\cos(2\pi U)\\\ (1+0.3N)\sin(2\pi
U)\end{pmatrix}$

where $U\sim\text{Uniform}(0,1)$ and $N\sim\text{Normal}(0,1)$ are independent
random variables. The outcome $Y$ follows a Bernoulli distribution with
probability $p$. This probability is a function of the Euclidean distance
between $X$ and $S_{1}$, namely

$p=\frac{1}{1+e^{-10\|X\|^{2}}}$

The process for creating prices (QuotedPrice, MidPrice,NextMidPrice) is as
follow:

We assume the initial prices for the bid ($B_{0}$), ask ($A_{0}$), and mid-
price ($M_{0}$) are defined as follows:

$B_{0}=P_{0}-\frac{S_{0}}{2}$ $A_{0}=P_{0}+\frac{S_{0}}{2}$
$M_{0}=\frac{B_{0}+A_{0}}{2}$

where $P_{0}$ is the initial mid-price $124.24\$$ and $S_{0}$ is the initial
spread $0.1\$$.

The spread at each time step $t$ then can be updated using a Geometric
Brownian Motion model:

$S_{t+1}=S_{t}\exp\left(\mu\Delta t+\sigma_{S}\epsilon_{t}\sqrt{\Delta
t}\right)$

where $\mu$ is the drift coefficient, $\sigma_{S}$ is the volatility
coefficient, and $\epsilon_{t}\sim\mathcal{N}(0,1)$ is a random shock drawn
from a standard normal distribution.

At each time step $t$, the bid and ask prices are updated using stochastic
processes:

$B_{t+1}=M_{t}-\frac{S_{t+1}}{2}+\epsilon_{B}$
$A_{t+1}=M_{t}+\frac{S_{t+1}}{2}+\epsilon_{A}$

where $\epsilon_{B}\sim\mathcal{N}(0,\sigma_{B}^{2})$ and
$\epsilon_{A}\sim\mathcal{N}(0,\sigma_{A}^{2})$ are random shocks drawn from
normal distributions with mean zero and variances $\sigma_{B}^{2}$ and
$\sigma_{A}^{2}$, respectively.

The mid-price at each time step $t$, i.e MidPrice is then given by:

$M_{t}=\frac{B_{t}+A_{t}}{2}$

And the next mid-price at time $t+1$, i.e NextMidPrice is given by:

$M_{t+1}=\frac{B_{t+1}+A_{t+1}}{2}$

Finally, we want to generate a quote price that is within $0.01 range of the
market mid-price $P_{mid}$ and followed these steps:

  1. 1.

Let $P_{mid}$ be the market mid price.

  2. 2.

Generate a random number $\delta$ from a uniform distribution within the
interval $[-0.01,0.01]$.

  3. 3.

The quoted price $P_{quote}$ is then given by:

$P_{quote}=P_{mid}+\delta$

##  IV Data Processing and Feature Engineering

The target variable we want to predict is Status. We constructed a feature set
that excludes the unknown variable NextMidPrice, the target Status and we also
created several new features based on exiting variables.

The momentum factor in financial markets refers to the tendency of securities
to continue in their existing trend for a period of time. For intraday data,
momentum can be defined by comparing the current price to a previous price
within the same trading day. If $P_{t}$ represents the price at time $t$ and
$P_{t-k}$ represents the price $k$ minutes earlier, the momentum factor
$M_{t}$ can be defined as:

$M_{t}=\frac{P_{t}-P_{t-k}}{P_{t-k}}$

This formula measures the relative change in price over the intraday period
from $t-k$ to $t$.

We computed momentum factor lagging 5, 10 and 20 periods as the feature: MOM5,
MOM10, MOM20. We also calculated Spread as the difference between MidPrice \-
QuotedPrice. It reflects the relative bid-ask pressure in the market. Spreads
can widen during periods of high market volatility. Finally, we take the $log$
term over bond notional to create LogNotional.

MOM5 $\displaystyle=\frac{\text{MidPrice}(i)}{\text{MidPrice}(i-5)}-1$ MOM10
$\displaystyle=\frac{\text{MidPrice}(i)}{\text{MidPrice}(i-10)}-1$ MOM20
$\displaystyle=\frac{\text{MidPrice}(i)}{\text{MidPrice}(i-20)}-1$ Spread
$\displaystyle=\text{MidPrice}-\text{QuotedPrice}$ Response
$\displaystyle=\text{Spread}\times[1\text{ if bid else }-1]$ LogNotional
$\displaystyle=\log(\text{Notional})$

where $\text{MidPrice}(i)$ is the $i$-th (current) mid-price of a specific
bond. We took logarithm transformation on the Notional.

We intend to keep features intuitive and explainable. A relatively small set
of classic academic features can help us better understand and differentiate
our model performance and their property in explainability.

We used spline regression to calibrate the kernel probability distributions of
each feature shown in figure 1 and figure 2. The calibrated distribution
confirmed our hypothesis that the probability of Getting Filled tend to
decrease when responses or logNotional increases. We found that MOM5 is highly
correlated with the probability of Getting Filled, but the correlation is not
linear and we need to take Response into consideration.

Figure 1: Kernel probability distributions for each feature Figure 2: Kernel
probability distributions for each feature

##  V Probability of Execution a Trade For a Given Price

###  V-A Model and Algorithm

We first recognize that predicting the probability of winning an RFQ serves as
a binary classification problem. Therefore we implemented the following
models: Lasso Logistic, Random Forest, XGBoost and Bayesian Neural Network. In
addition, we applied 3 groups of ensemble learning: each model predicts the
probability of getting RFQ filled and we performed majority vote to generate
the final prediction of the probability. Ensembling successfully mitigates the
problem overfitting and improve accuracy on out-of-sample data.

We evaluated the performance each model by their ability to minimize the log-
loss (cross-entropy). The negative log-loss for a binary outcome $y$ is
defined as:

$-(y\log(p)+(1-y)\log(1-p))$

where $p$ is the model’s predicted probability of $y$ belonging to class 1.

The rationale of our selection of models are as follow: we first use Lasso
Logistics model to establish a baseline. Lasso can automatically select
meaningful features and provides clear explanation of the influence of each
variable to the final outcome. However, Lasso may suffer from under-fitting as
it is a linear model.

Secondly, we implemented both bagging-based Random Forest model and boosting-
based XGBoost model. These tree models are non-linear in nature and can
generally help boost the prediction accuracy compared to a linear model.
However they both suffer from the trade-off between performance and
explainability. Random Forest is an ensemble of many decision trees. While a
single decision tree can be easily visualized and understood, combining many
trees makes the overall model more complex and harder to interpret. XGBoost
builds trees sequentially where each new tree aims to correct the residuals of
the previous trees. This iterative process makes it hard to interpret the
contribution of each feature to the final prediction.

Finally, we input explore the Bayesian Neural Network model. There are two
reasons for the choice of Bayesian Neural Network. Bayesian Neural Network has
a probabilistic construct that can naturally apply to our probability
prediction task. The model can produce an explainable ratio of likelihood and
furthermore, evaluate each new split as a ratio of likelihoods that can
balance the trade-off between statistical improvement and the additional model
complexity. Moreover, the model takes advantage of both neural network and
decision tree. The model has the explainability of decision tree and also has
the plasticity of neural networks that allows the node split to have a soft
margins and slanted boundaries which can be adjusted by using gradient
descent.

###  V-B Bayesian Neural Tree

Suppose we are tasked with estimating the probability of an outcome
$y\in\mathbb{R}$ given an input $x\in\mathbb{R}^{d}$, denoted as $p(y\mid x)$.

We define the Bayesian Neural Tree based on a soft Bayesian Decision Tree in
[14], Adaptive Bayesian Reticulum as detailed [15] and the Bayesian Inference
framework in introduced in [11]. We propose to integrate Bayesian Inference
and Neural Network optimization techniques into the Decision Tree model.

Decision tree on a binary classification problem is a binary tree that can
modeled as directed acyclic graphs (DAG). Decision trees consist of two types
of nodes: internal nodes $n\in\mathcal{N}$ and leaves $\ell\in\mathcal{L}$. We
note the root node as $n_{0}$. The set $\mathcal{L}$ partitions the feature
set $\mathbb{R}^{d}$ based on parameters associated with the internal nodes
$n\in\mathcal{N}$.

Internal nodes create certain rules based on an input feature $x$ that
generate their child nodes and this process can be represented as gate
functions $g_{j}$ at node $n_{j}$ in the Neural Network, where $j$ indexes the
node. Gate functions will produce binary outputs $\\{0,1\\}$. If $g_{j}(x)=1$,
then the left child of an internal node is activated, otherwise the right
child is activated.

We define the following probability:

  * •

$p(x\in\ell)$: The probability that input $x$ falls into leaf $\ell$,
abbreviated as $p(X\in\ell\mid X=x)$.

  * •

$p(y\mid x\in\ell)$: The probability of outcome $y$ given that $x$ is in leaf
$\ell$, abbreviated as $p(y\mid X\in\ell;X=x)$.

Combining these terms with the total law of probability, we can express
$p(y\mid x)$ as:

$p(y\mid x)=\sum_{\ell\in\mathcal{L}}p(y\mid x\in\ell)\cdot p(x\in\ell).$

In a decision tree, $p(x\in\ell)$ is calculated as the product of the gate
functions along the path from the root node $n_{0}$ to leaf $\ell$.

For example:

$p(x\in\ell_{1})=g_{0}(x)\cdot g_{1}(x)$
$p(x\in\ell_{2})=g_{0}(x)\cdot(1-g_{1}(x)),$
$p(x\in\ell_{3})=(1-g_{0}(x))\cdot g_{2}(x)$
$p(x\in\ell_{4})=(1-g_{0}(x))\cdot(1-g_{2}(x)).$

$p(x\in\ell)$ falls between 0 and 1. The probability $p(y\mid x\in\ell)$ is
specified for each leaf.

Decision trees can be seen as single-layer neural networks that maintain a
predefined tree structure constraint. The gate function at node $n_{j}$ is
defined by a manifold $M$ such that $g_{j}(x)=s(d_{M}(x))$. $d_{M}(x)$ is the
signed distance function from $x$ to $M$ and $s$ is the activation function
that maps a real value to the range $[0,1]$. For a hyperplane $M$ with weight
vector $\mathbf{w}$, we will have $d_{M}(x)=\mathbf{w}^{T}\mathbf{x}+w_{0}$.
Traditional decision tree has hard boundary, meaning that it defines
$s(x)=\mathbf{1}_{x>0}$ for gate activation. In the Bayesian Neural Tree, a
smooth activation function $s$ will be better as it allows $p(x\in\ell)$ to be
a continuous variable between 0 and 1 as a function of $\mathbf{w}$,
facilitating the parameter optimization through Nueral Network’s
backpropagation.

In soft decision trees, $p(x\in\ell)$ represents the probability that the
outcome at $x$ comes from the distribution associated with leaf $\ell$. For
convenience, we keep the notation of $x$ belonging to $\ell$.

We then define a Bayesian Neural Network Framework for a binary classification
task. Assume we have data samples $\\{x_{i}\\}_{i=1}^{m}$ in $\mathbb{R}^{d}$
with outcomes $\\{y_{i}\\}_{i=1}^{m}$ in $\\{0,1\\}$. It is impractical to
perform an exhaustive search for the optimal dichotomy for separating
$\\{x_{i}\\}_{i=1}^{m}$ due to the exponential growth of $m$. Therefore, we
need to relax the activation function in decision trees to apply a gradient
ascent algorithm.

We use affine hyperplanes with a sigmoid function $s$ and define the gate
function as:

$g(x)=\frac{1}{1+e^{-(w_{0}+\sum_{k}w_{k}x_{k})}}$

Denote $|\mathcal{N}|$ as the number of internal nodes and $|\mathcal{L}|$ as
the number of leaves. Each data sample is assigned to a unique leaf $\ell$ in
$|\mathcal{L}|$ different configurations. Each configuration has an outcome
$\omega$ that maps point $i$ to leaf $\omega(i)$, so probability
$p(\omega)=\prod_{i}p(x_{i}\in\omega(i))$. Assume that we have beta conjugate
prior parameters $\alpha$ and $\beta$, the posterior parameters for a leaf
$\ell$ and configuration $\omega$ are:

$\alpha^{\prime}_{\ell}(\omega)=\alpha+\sum_{i=1}^{m}\mathbf{1}_{\omega(i)=\ell}(1-y_{i})$
$\beta^{\prime}_{\ell}(\omega)=\beta+\sum_{i=1}^{m}\mathbf{1}_{\omega(i)=\ell}y_{i}$

Therefore the final log-likelihood is

$\sum_{\omega\in\Omega}\sum_{\ell\in\mathcal{L}}\ln\left(B(\alpha^{\prime}_{\ell}(\omega);\beta^{\prime}_{\ell}(\omega))\right)p(\omega)$

, where$B$ is the Beta function.

The expected probabilities at each leaf are:

$p(y=0\mid
x\in\ell)=\frac{\alpha^{\prime}_{\ell}}{\alpha^{\prime}_{\ell}+\beta^{\prime}_{\ell}}$
$p(y=1\mid
x\in\ell)=\frac{\beta^{\prime}_{\ell}}{\alpha^{\prime}_{\ell}+\beta^{\prime}_{\ell}}$

Note that these probabilities are determined by $p(x_{i}\in\ell)$ and $y_{i}$,
unlike standard neural networks where the final layer is parameterized and
optimized through gradient descent.

To find the optimal Bayesian Neural Network, we need to maximize the log-
likelihood. We applies the Jensen Inequality to achieve a lower bound to be
optimized:

$c=\sum_{\ell\in\mathcal{L}}\ln\left(B(\alpha^{\prime}_{\ell};\beta^{\prime}_{\ell})/B(\alpha;\beta)\right)$

with

$\alpha^{\prime}_{\ell}=\alpha+\sum_{i=1}^{m}p(x_{i}\in\ell)(1-y_{i})$
$\beta^{\prime}_{\ell}=\beta+\sum_{i=1}^{m}p(x_{i}\in\ell)y_{i}$

The probabilities at each leaf can also be approximated as:

$p(y=0\mid
x\in\ell)=\frac{\alpha^{\prime}_{\ell}}{\alpha^{\prime}_{\ell}+\beta^{\prime}_{\ell}}$
$p(y=1\mid
x\in\ell)=\frac{\beta^{\prime}_{\ell}}{\alpha^{\prime}_{\ell}+\beta^{\prime}_{\ell}}$

Finally, we define the unexplained potential for each leaf $\ell$ as
$c_{\ell}=\ln\left(\frac{B(\alpha^{\prime}_{\ell};\beta^{\prime}_{\ell})}{B(\alpha;\beta)}\right)$.
This value serves as a lower bound on the log-probability of the data at leaf
$\ell$ and helps determine which leaf to expand further. A smaller $c_{\ell}$
indicates a higher potential for improving the log-likelihood.

When maximizing the bound $c$ defined above, we want to adaptively built from
the data instead of predefining a certian structure using the following steps:

We first begin with a vanilla Bayesian Neural Network structure starting at
the root node. As the network grows, we randomly select a leaf to extend based
on its unexplained potential $c_{\ell}$. The selection is made with
probabilities proportional to $c_{\ell}$, given by:

$\text{Probability}=\frac{c_{\ell}}{\sum_{\ell^{*}\in\mathcal{L}}c_{\ell^{*}}}$

Then for the selected leaf, we sample some new weights uniformly and optimize
them locally for this leaf while keeping others constant. We subsequently
perform a global optimization to fine-tune the entire model. We also adopts
the network post-pruning. After global optimization, we evaluate each leaf to
decide if it should be pruned. Retain leaves that enhance the model’s log-
likelihood compared to their parent leaf.

There are several key hyperparameters that will effect the convergence of
gradient descent:

  * •

Stiff Nodes: Nodes that behave like Heaviside functions, resulting in minimal
gradient changes.

  * •

Soft Nodes: Nodes with gradients that are more sensitive and need to become
stiffer to improve the unexplained potential $c_{\ell}$.

In summary, The entire training process involves two phases: local
optimization focuses on updating the weights of the current leaf while global
optimization adjusts all nodes to enhance the overall model. Our Algorithm 1
integrates local adjustments typical of decision trees with global
optimization methods from neural networks.

Algorithm 1 Construction of Bayesian Neural Network

1: $a=0$, initialize the vanilla network structure

2: while $a<\text{max\\_attempts}$ do

3: $j\leftarrow$ choose the index of node to extend

4: Sample the weights for node $j$

5: Train node $j$’s weights using gradient ascent

6: Train the full tree’s weights using gradient ascent

7: Prune the network

8: $a=a+1$

9: end while

###  V-C Choice of Model Parameter

To optimize the model performance, we first predetermine a range for each
parameter based on financial rationale and rule of thumb in tuning. Then we
perform a 5-fold time series cross-validation and optimize parameters with an
exhaustive grid search. All models are trained in a rolling window fashion on
the the training dataset and then fine-tuned on the validation dataset.

We specially discuss the details of choosing parameter ranges for the Bayesian
Neural Tree.

  1. 1.

Prior: For the binary classification, we can use prior distribution of Beta
distribution with an $(\alpha,\beta)$ prior. We select Beta(1,1) as the prior
to assume that the probability is around 50%, but with little confidence.

  2. 2.

pruning_factor: It is the factor by which the log likelihood of a split has to
be larger than the log likelihood of no split in order to keep the split
during the pruning stage. We choose between [1,1.05] as the pruning factor.

  3. 3.

n_iter: It is the number of attempts to add new nodes by splitting existing
ones. The resulting tree will at most contain n_iter nodes, but it may contain
less due to pruning. Larger values lead to more complicated models at the cost
of potentially overfitting and slower training performance. We choose between
[50,500].

  4. 4.

learning_rate_init: The initial gradient descent learning rate of Adam
optimizer and we set it to be 0.05 as a rule of thumb.

  5. 5.

n_gradient_descent_steps: The number of gradient descent steps to perform in
each iteration. Half of the steps will perform local gradient descent on only
the newly added node and half will be applied during global gradient descent
involving the whole tree. We choose the value of 100.

  6. 6.

initial_relative_stiffness: Small values such as from 0.1 to 5 represent
”soft” splits allowing for further optimization through gradient descent.
Large values (above 20) make the split an almost step function and disable the
weight update. We tested both small and large stiffness values and decide to
choose a small stiffness between 2 to 10 to give a soft split that allows for
further optimization through gradient descent.

###  V-D Out-of-sample Test and Model Comparison

We split the 10000 samples into 70% training and 30% test data. The Bayesian
Neural Network model accuracy on out of sample dataset is around 90%. As shown
in figure 3, the False Positive Rate (FPR) is smaller than False Negative
Rate, which means that the percentage of RFQs predicted as filled (done) but
is actually missed is higher. Further, we found that there are more errors
occur when we have more than 2 competitors. The forecasted execution
probability is well concentrated to 0 and 1 according to figure 4. As we can
see in figure 5, the most importance features are response, MOM5, MOM20 and
counterparty. To analyze all possible splits within a dimension, i.e. produce
a decision boundary, all we need to do is sort the data once for each
dimension. According to the decision boundary in figure 6, the response
feature is an important factor in separating the filled RFQ from unfilled
ones.

Figure 3: Confusion Matrix of Bayesian Neural Network Figure 4: Distribution
of Prediction of Bayesian Neural Network Figure 5: Feature Importance of
Bayesian Neural Network Figure 6: Decision Boundary of Bayesian Neural Network

Then we compared the performance of all 9 models: Logistic Regression with
Lasso regularization (LR), Bayesian Neural Tree with relative stiffness = 2
(ABR2) and relative stiffness = 6 (ABR6), XGBoost (XGB), Random Forest (RF),
Ensemble of LR, ABR2, ABR6 (Ensemble 1), Ensemble of LR, ABR6, XGB (Ensemble
2) and Ensemble of LR, ABR2, ABR6, RF, XGB (Ensemble 3). We finally choose the
model Ensemble 1 as it has the highest accuracy and F1 score:

Figure 7: Model Parameter and Performance Figure 8: Model Parameter and
Performance

##  VI Provide a Quote For a Hypothetical RFQ

###  VI-A Next Mid-Price Prediction

In the intraday trading, NextMidPrice is highly correlated with the current
mid price MidPrice, with a difference followinglog-normal distribution. Assume
that the data generation process is unknown, we can still confirm this fact
using Q-Q plot as the shown in figure 9. Therefore, we fitted a linear
regression model on MidPrice and Side to predict NextMidPrice on the training
set and the adjusted R-squared is 98% on the validation set.

###  VI-B Market-Maker Utility

We define the following rules to simulate a real-world RFQ enviroment:

1\. Each RFQ that would incur a loss (i.e., an exccessively competitive price)
will generate reduce utility by 1 point and be removed from the list of
eligible RFQ responses:

$\text{Utility}=-1\quad\text{if loss}$

2\. Out of all the remaining eligible RFQ responses (i.e., the ones that would
have a positive, or at least zero, profit if executed), only the most
competitive price wins the RFQ and adds 1 to the utility:

$\text{Utility}=+1\quad\text{for the most competitive price}$

3\. If multiple RFQs are filled at the price at the exact same level, these
market-makers will share the equally 1 utility point but each minus an anti-
competition penalty of 0.5:

$\text{Utility}=\frac{1}{n}-0.5\quad\text{if }n\text{ responses have the same
price}$

where $n$ is the number of RFQs priced at the same level.

Putting it all together, we get:

$\text{Utility}=\begin{cases}-1&\text{if loss}\\\ +1&\text{the most
competitive price}\\\ \frac{1}{n}-0.5&\text{if }n\ \text{RFQs have the same
price}\end{cases}$

Based on the rules, our goal is to find the best price to fill an RFQ while
maximizing the expected market-maker’s utility. We convert this problem to an
optimization problem by defining the objective function as the expected pay-
off:

Figure 9: Correlation of Mid-price and Next Mid-price

$E(\text{payoff})=P(\text{Getting Filled})+(-1)\times P(\text{Exceed Limit})$

Note that our expected payoff objective does not take into account the penalty
term when other market-makers produce an RFQ that has the exact same quote
price with us, as it involves constructing other complicated multi-agent
models and is out of the scope of this paper. As an alternative, we assume
that probability of having an RFQ of the same price level is negligible and we
use a qualitative approach to adapt to the penalty..

###  VI-C Out-of-Sample Performance

To optimize the objective function, we can use the Ensemble 1 model in Section
V-D to predict the probability of getting the RFQ filled: $P(\text{Getting
Filled})$. We then use the sample mean of NextMidPrice on the validation data
to estimate the probability of incurring a loss: $P(\text{Exceed Limit})$ for
each TBA bond given a quote price.

Take TBA Bond $0$ as an example, after we predict the NextMidPrice
$\hat{P^{i}}$ of an RFQ $i$ on the validation set using the linear model in
Section VI-A, we start a search for the optimal quote price in an interval:
$[\hat{P^{i}}-1,\hat{P^{i}}-0.99,\ldots,\hat{P^{i}}+0.99,\hat{P^{i}}+1]$ with
an increment of 100bps. As a market-maker, we should have the quote price
below $\hat{P^{i}}$ for a bid RFQ, and for an ask RFQ, the quote price should
be above $\hat{P^{i}}$. If our quote price exceeds the true NextMidPrice
$P^{i}$, we have $I(\text{Exceed Limit})=1$; otherwise, $I(\text{Exceed
Limit})=0$ in the expected payoff function.

The probability of incurring a loss given a quote price $P$: $P(\text{Exceed
Limit}\mid P)=\text{Avg}_{i}[I(\text{Exceed Limit}\mid P^{i})]$ for all sample
RFQ $i$ in the validation dataset. For each TBA bond, we can generate a series
of pairs of this possibility and our quote price. We show the $P(\text{Exceed
Limit})$-to-price curve on validation data in figure 10 and use it for the
test set. The red lines represent our final decision of quote price. According
to the plot, our RFQ with the selected quote price have a high probability of
getting filled according to the Ensemble 1 model we built in Section V-D.

Figure 10: Probability of Filling an RFQ Given Quote Price

To adjust for the penalty of having RFQ at the same price level, we define a
cap for both bid and ask RFQ: the final quote price for a bid RFQ is
$max(\textit{BidPrice},\textit{MidPrice}-0.01)$ and the final quote price for
an ask RFQ is $min(\textit{AskPrice},\textit{MidPrice}+0.01)$. Our final quote
price are: [-0.02, -0.05, 0.05, -0.01, 0.01] for 5 out-of-sample RFQs. As
shown in figure 11, we define the distance to NextMidPrice as our quote price
minus the prediction of next Mid-price. The expected payoff (utility) for
market-maker peaks at a small interval around the true NextMidPrice. The
probability of filling a bid RFQ is at the highest when we set the quote price
far above the true NextMidPrice (it means our quote price is highly
competitive at the price of incurring a loss for market makers). Similary, the
probability of filling an ask RFQ is at the highest when we set the quote
price far below the true NextMidPrice.

We simulate the quote price of other market makers using the same data
generation process in Section III. Among all 5 RFQs out of sample, we are able
to fill 4 RFQ and secure positive utility values on all 5 of them. It shows
that our prediction of RFQ fill rate and mid-price level using XAI is useful
for market makers.

Figure 11: Market-Maker Utility given Quote Price

##  VII Conclusion

In this paper, we explore the formulation of RFQ for TBA bond. We first
establish a simulation algorithm to genereate the RFQ data process. We then
model the probability of getting RFQ filled with multiple machine learning
algorithms and explainable AI models including LASSO Logistic Regression,
XGBoost and Bayesian Neural Tree. We found that Bayesian Neural Tree has
balanced off a good performance and a strong explainability with its
hierarchical tree structure. An ensemble of these models further boost its
prediction accuracy. Finally, we propose a market-maker’s utility function to
optimize the best quote price. The paper makes practical implications on
providing a framework for analyzing RFQ data and promoting pricing efficiency
for TBAs through RFQ to boost the mortgage lending process.

## References

  * [1] Kevin P. Murphy.  Machine Learning: A Probabilistic Perspective.  MIT Press, 2012.
  * [2] Zoubin Ghahramani.  Probabilistic machine learning and artificial intelligence.  Nature, 521(7553):452–459, 2015.
  * [3] Q. Zhou.  Portfolio optimization with robust covariance and conditional value-at-risk constraints.  arXiv preprint arXiv:2406.00610, 2024.
  * [4] Christopher M. Bishop.  Pattern Recognition and Machine Learning.  Springer, 2006.
  * [5] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  “Why should I trust you?”: Explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144. 2016.
  * [6] Amina Adadi and Mohammed Berrada.  Peeking inside the black-box: A survey on explainable artificial intelligence (XAI).  IEEE Access, 6:52138–52160, 2018.
  * [7] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In Advances in Neural Information Processing Systems, pages 4765–4774, 2017.
  * [8] Jerome H. Friedman.  Greedy function approximation: A gradient boosting machine.  Annals of Statistics, pages 1189–1232, 2001.
  * [9] Leo Breiman.  Random forests.  Machine Learning, 45(1):5–32, 2001.
  * [10] Christoph Molnar.  explainable Machine Learning.  2020\.
  * [11] Q. Zhou.  Application of Black-Litterman Bayesian in statistical arbitrage.  arXiv preprint arXiv:2406.06706, 2024.
  * [12] Carolyn Strobl, Anne-Laure Boulesteix, Thomas Kneib, Thomas Augustin, and Achim Zeileis.  Bias in random forest variable importance measures: Illustrations, sources and a solution.  BMC Bioinformatics, 8(1):25, 2007.
  * [13] Alex Goldstein, Adam Kapelner, Justin Bleich, and Emil Pitkin.  Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation.  Journal of Computational and Graphical Statistics, 24(1):44–65, 2015.
  * [14] G. Nuti, L. A. Jiménez Rugama, and I. Cross, A Bayesian Decision Tree Algorithm, arXiv preprint arXiv:1901.03214, 2019.
  * [15] G. Nuti, L. A. Jiménez Rugama, and K. Thommen, Adaptive Bayesian Reticulum, February 27, 2020.
  * [16] David W. Hosmer, Stanley Lemeshow, and Rodney X. Sturdivant.  Applied Logistic Regression.  John Wiley & Sons, 2013.
  * [17] Leo Breiman, Jerome Friedman, Richard Olshen, and Charles Stone.  Classification and Regression Trees.  CRC Press, 1984.
  * [18] Andrew McCallum and Kamal Nigam.  A comparison of event models for naive bayes text classification.  In AAAI-98 Workshop on Learning for Text Categorization, volume 752, pages 41–48. 1998.
  * [19] Radford M. Neal.  Bayesian Learning for Neural Networks.  PhD thesis, University of Toronto, 1992.


# MoDem-V2: Visuo-Motor World Models
for Real-World Robot Manipulation

Patrick Lancaster1 Nicklas Hansen2 Aravind Rajeswaran1 Vikash Kumar1 1Patrick
Lancaster, Aravind Rajeswaran, and Vikash Kumar are with Meta AI {plancaster,
<EMAIL_ADDRESS><EMAIL_ADDRESS>Hansen is with the University
of California San Diego<EMAIL_ADDRESS>
###### Abstract

Robotic systems that aspire to operate in uninstrumented real-world
environments must perceive the world directly via onboard sensing. Vision-
based learning systems aim to eliminate the need for environment
instrumentation by building an implicit understanding of the world based on
raw pixels, but navigating the contact-rich high-dimensional search space from
solely sparse visual reward signals significantly exacerbates the challenge of
exploration. The applicability of such systems is thus typically restricted to
simulated or heavily engineered environments since agent exploration in the
real-world without the guidance of explicit state estimation and dense rewards
can lead to unsafe behavior and safety faults that are catastrophic. In this
study, we isolate the root causes behind these limitations to develop a
system, called MoDem-V2, capable of learning contact-rich manipulation
directly in the uninstrumented real world. Building on the latest algorithmic
advancements in model-based reinforcement learning (MBRL), demo-bootstrapping,
and effective exploration, MoDem-V2 can acquire contact-rich dexterous
manipulation skills directly in the real world. We identify key ingredients
for leveraging demonstrations in model learning while respecting real-world
safety considerations – exploration centering, agency handover, and actor-
critic ensembles. We empirically demonstrate the contribution of these
ingredients in four complex visuo-motor manipulation problems in both
simulation and the real world. To the best of our knowledge, our work presents
the first successful system for demonstration-augmented visual MBRL trained
directly in the real world. Visit sites.google.com/view/modem-v2 for videos
and more details.

##  I Introduction

Robot agents learning manipulation skills directly from raw visual feedback
avoid the need for explicit state estimation and extensive environment
instrumentation for rewards, but face heightened exploration, and thereby
safety, challenges in navigating the contact-rich high-dimensional search
space purely based on sparse visual reward signals. These challenges are
especially critical for agents operating in the real world where inefficiency
can be expensive, and safety faults can be catastrophic. One approach to
developing robot manipulation policies that avoid such safety restrictions is
simulation to reality transfer [1, 2, 3, 4]. However, the creation and
calibration of accurate physics simulations (from first principles) for
contact-rich tasks is extremely challenging and time-consuming. In this work,
we alternatively study the use of visual world model learning [5, 6, 7] for
robot manipulation directly from real-world interaction.

Figure 1: We use MoDem-V2 to train the robot on four contact-rich manipulation
tasks. These tasks cover a wide range of manipulation skills, namely non-
prehensile pushing, object picking, and in-hand manipulation. In recognition
of the difficulty of robust pose tracking and dense reward specification in
the real world, the robot performs these tasks using only raw visual feedback,
proprioceptive signals, and sparse rewards.

Model-Based Reinforcement Learning (MBRL) with visual world models involves
the learning of dynamics models using real-world data, directly from visual
observations. When applied to robot manipulation, visual MBRL can mitigate the
need for detailed physics simulations from first principles, as well as the
need for specialized sensor instrumentation and state estimation pipelines.
However, visual MBRL for real-world robotics still has two major challenges:
(a) sample inefficiency; and (b) sparse/weakly shaped rewards. While a number
of recent algorithms such as RRL [8] and MoDem [9] circumvent these challenges
by leveraging a small number of expert demonstrations to improve sample-
efficiency, they rely on aggressive exploration to compensate for weak reward
supervision that can result in unsafe behaviors, restricting their application
to simulated or heavily engineered scenarios.

We indeed found MoDem to be infeasible for direct application in the real
world due to excessively aggressive exploratory behavior. Even with
significant engineering investments for safety, we found that the built-in
low-level hardware controllers/drivers fault repetitively owing to excessive
velocity, acceleration, and torque in the robot’s joints that exert dangerous
amounts of force on the environment or the robot itself. While some level of
safety can be imposed through hard-coded limits on velocity, acceleration, and
torque (as done in all of our experiments), this is an insufficient solution
for preventing unsafe behavior during online interaction (Figure 2 left).
Tuning these task-specific limits for interaction in the real world is costly
and faced with operational and safety challenges. Balancing the tightness of
these limits presents a further challenge, as aggressive limits can prohibit
the robot from exerting the minimum energy needed to solve the task and weak
limits will fail to inhibit unsafe actions. Finally, these static limits fail
to incorporate the need for dynamic risk sensitivities across relevant time
scales (such as at the scale of a single episode, or even at the scale of
training epochs). Furthermore, simply penalizing the amount of torque exerted
by the agent is an ineffective, retrospective solution that does not prevent
unsafe actions at the onset of exploration as shown in Figure 2 and Figure 8.
How can we get around these challenges?

Figure 2: Agent performance on the inclined pushing task before failure due to
safety violations. An asterisk indicates that agent training was terminated
due to significant safety violations. _Left:_ On a real-world robot, MoDem
violates (robot manufacturer specified) torque limits at the onset of online
interaction and is unable to learn, whereas MoDem-V2’s conservative
exploration allows it to perfect the task. _Right:_ Further evaluation in
simulation reveals that simply penalizing the amount of torque exerted by the
robot does not prevent termination due to significant safety violations. Other
baseline agents are either terminated due to unsafe behavior or achieve
significantly lower success than MoDem-V2.

Our key insight is that conservative exploration can respect the safety
constraints of real-world environments while still allowing the agent to
modulate its strategy according to task progress such that it is able to learn
quickly and efficiently. We translate this insight into implementation in
three steps. First, rather than sampling actions from the entire action space,
warm-starting exploration with actions sampled from a policy learned via
behavioral cloning (BC) [10] prevents the agent from straying far from the
provided demonstrations at the onset of online learning. Second, as our world
model gains better coverage through online exploration, agency-transfer
gradually shifts the agent from executing BC policy actions to short-horizon
planning. Agency transfer provides a mechanism for increased exploration while
stymying over-optimistic evaluation of regions of the observation-action space
far from the agent’s previous experience. Third, we use actor-critic ensembles
to estimate the epistemic uncertainty of the value estimations of these short-
horizon trajectories, allowing the agent to avoid overly optimistic actions.
We integrate these three enhancements into the recently proposed MoDem [9]
algorithm to develop MoDem-V2. Despite the simplicity of the individual
components, their combination, and the resulting effectiveness in transforming
overly aggressive, fault-prone MoDem agents into MoDem-V2 agents that
efficiently and safely learn manipulation behaviors in the real-world is quite
unique. Our work is, to the best of our knowledge, the first successful
demonstration of demonstration-augmented visual MBRL trained directly in the
real world.

To evaluate the effectiveness of our approach, we study four robot
manipulation tasks from visual feedback (Figure 1), and their simulated
counterparts.

Our main contributions are:

  * •

We identify unsafe exploration and over-optimism as the key issues in
leveraging visual MBRL algorithms for real-world applications.

  * •

From this insight, we develop MoDem-V2 by integrating three ingredients into
MoDem, namely policy centering, agency transfer, and actor-critic ensembles.

  * •

We demonstrate that MoDem-v2’s conservative exploration significantly enhances
its safety profile compared to other baselines, while still retaining the
sample-efficient learning capability of MoDem.

  * •

Finally, we demonstrate that MoDem-V2 can quickly learn a variety of contact-
rich manipulation skills, such as pushing, picking, and in-hand manipulation
directly in the real world.

  * •

We contribute towards lowering the barrier of entry for RL in the real world
by open-sourcing our implementation of MoDem-V2 and discussing practical
considerations for training on real hardware.

##  II Preliminaries

We begin by introducing notations and providing an overview of MBRL settings.

Notation: The general setting of an agent interacting with its environment can
be formulated as a Markov Decision Process (MDP) described by the tuple
$\mathcal{M}:=(\mathcal{S},\mathcal{A},\mathcal{T},R,\gamma)$. Here,
$\mathcal{S}$ denotes the state space, $\mathcal{A}$ is the action space, the
conditional probability distribution
$\mathbf{s}_{t+1}\sim\mathcal{T}(\cdot|\mathbf{s}_{t},\mathbf{a}_{t})$ defines
the dynamics of the MDP, and a scalar reward function is given by
$r_{t}=R(\mathbf{s}_{t},\mathbf{a}_{t})$. Finally, $\gamma\in[0,1)$ defines
the discount factor for the MDP to trade-off future rewards to current ones.
The goal for an agent is to learn a policy $\pi:\mathcal{S}\mapsto\mathcal{A}$
that can achieve high long term performance given by
$\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{t}\right]$.

We specifically consider the problem of robotic manipulation from visual
feedback on real hardware. We aim to learn a control policy that controls a
physical robot from RGB observations provided by cameras placed in the scene,
and robot proprioception. We model this setting as a high-dimensional MDP with
sparse rewards. This assumes that while the state space of the MDP (_e.g._
object poses) is not directly observable by the agent, a sufficient
representation of the state can be well-approximated through the combination:
$\mathbf{s}=(\mathbf{x},\mathbf{q})$ where $\mathbf{x}$ denotes stacked RGB
observations from the robot’s camera(s) and $\mathbf{q}$ denotes
proprioception from the robot. Finally, we only assume access to a sparse task
completion reward, which is much easier to obtain via visual inputs compared
to a detailed well shaped reward function. The final goal is to learn a policy
that achieves high performance, using minimal online interactions, while
respecting the safety considerations of hardware.

MoDem - Model-Based Reinforcement Learning with Demonstrations: Our approach
is based on _MoDem_ [9] – a MBRL algorithm that combines _(i)_ model
predictive control (MPC) and the decoder-free world model of TD-MPC [7] with
_(ii)_ a small number of demonstrations to efficiently solve continuous
control problems with limited online interaction. Concretely, MoDem learns the
following five components:

$\begin{array}[]{lll}\text{State
embedding}&\mathbf{z}=h_{\theta}(\mathbf{s})\\\ \text{Latent
dynamics}&\mathbf{z}^{\prime}=d_{\theta}(\mathbf{z},\mathbf{a})\\\
\text{Reward predictor}&\hat{r}=R_{\theta}(\mathbf{z},\mathbf{a})\\\
\text{Terminal value}&\hat{q}=Q_{\theta}(\mathbf{z},\mathbf{a})\\\
\text{Policy guide}&\hat{\mathbf{a}}=\pi_{\theta}(\mathbf{z})\end{array}$ (1)

where $h_{\theta},d_{\theta},R_{\theta}$ and $Q_{\theta}$ are learned end-to-
end using a combination of joint-embedding predictive learning [11], reward
prediction, and Temporal Difference (TD) learning, and $\pi_{\theta}$ is a
deterministic policy that learns to maximize $Q_{\theta}$ conditioned on a
latent state $\mathbf{z}$ (see Equation 2 of the Appendix). Throughout this
work, we will refer to $(h_{\theta},d_{\theta},R_{\theta},Q_{\theta})$ as the
_world model_ , and $\pi_{\theta}$ as the _policy_. See [7, 9] for additional
details on the world model. While _MoDem_ has been shown to be effective in
simulation, owing to aggressive exploration its applicability in domains where
safety constraints can’t be overlooked is limited (Figure 2).

##  III Safety

Agents that seek to learn via continuous operations need to operate respecting
hardware and environmental safety constraints. Such constraints are diverse,
obscure, and unobservable (intrinsic to low-level hardware details, or lack of
appropriate sensing) and therefore cannot be directly accounted for by the
agent during operations. Alternatives such as action penalization and user-
defined safety are also insufficient (Figure 2) requiring extensive human
intervention and monitoring for operations. In this work, we refer to these
unobservable constraints as safety violations. For real-world operations, they
are defined as hardware faults that require human intervention. For simulated
runs, we define them as violations (unobserved by the policy) in either the
robot’s torque limits (as defined by the robot manufacturer) or excessive
contact force applied by the robot’s end effector.

##  IV Method

Algorithm 1 Planning procedure of MoDem-V2
($\bullet$Original MoDem $\bullet$MoDem-V2 modification)

0: $\theta:$ learned network parameters $\mu,\sigma$: initial parameters for
$\mathcal{N}$ $N$: num sample trajectories $\mathbf{s}_{0},h$: current state,
planning horizon $\tau$: trajectory weighting temperature $\alpha$:
probability of using model rollouts $w_{1},w_{2}$: ensemble mixing weights

1: encode state $\mathbf{z}_{0}\leftarrow h_{\theta}(\mathbf{s}_{0})$
$\vartriangleleft$ _State embedding_

2: if $\operatorname{rand}()>\alpha$ then

3: $\Gamma\coloneqq\\{\mathbf{a}_{0}\\}^{N}\sim$ $\pi^{BC}_{\theta}$
$\vartriangleleft$ _Center actions around BC_

4: $\phi_{\Gamma}=Q^{BC}_{\theta}(\mathbf{z}_{0},\\{\mathbf{a}_{0}\\}^{N})$
$\vartriangleleft$ _Critic evaluation_

5: else

6:
$\Gamma\coloneqq\mathbf{a}_{t:t+h}\sim\mathcal{N}(\mu,\mathrm{I}\sigma^{2})$
$\vartriangleleft$ _Prior sampling_

7:
$\Gamma\coloneqq\mathbf{a}_{t:t+h}\sim\pi^{1:M}_{\theta},d_{\theta}$$\vartriangleleft$
_Policy ensemble sampling_

8: for all $N$ trajectories
$\Gamma_{i}=(\mathbf{a}_{t},\mathbf{a}_{t+1},\dots,\mathbf{a}_{t+h})$ do

9: for step $t=0..h-1$ do

10:
$\phi_{\Gamma}=\phi_{\Gamma}+\gamma^{t}R_{\theta}(\mathbf{z}_{t},\mathbf{a}_{t})$
$\vartriangleleft$ _Reward_

11: $\mathbf{z}_{t+1}\leftarrow d_{\theta}(\mathbf{z}_{t},\mathbf{a}_{t})$
$\vartriangleleft$ _Latent transition_

12:
$\phi_{\Gamma}=\phi_{\Gamma}+\gamma^{h}Q_{\theta}(\mathbf{z}_{H},\mathbf{a}_{H})$
$\vartriangleleft$ _Terminal value_

13: # Ensemble of terminal values

14:
$\phi^{1:M}_{\Gamma}=\phi_{\Gamma}+\gamma^{h}Q^{1:M}_{\theta}(\mathbf{z}_{H},\mathbf{a}_{H})$

15: # Epistemic uncertainty estimation

16: $\phi_{\Gamma}=w_{1}$ mean$(\phi^{1:M}_{\Gamma})+w_{2}$ std
$(\phi^{1:M}_{\Gamma})$

17: $\Omega=e^{\tau(\phi_{\Gamma})}$,
$\mu=\frac{\sum_{i=1}^{N}\Omega_{i}\Gamma_{i}}{\sum_{i=1}^{N}\Omega_{i}}$,
$\sigma=\sqrt{\frac{\sum_{i=1}^{N}\Omega_{i}(\Gamma_{i}-\mu)^{2}}{\sum_{i=1}^{N}\Omega_{i}}}$

18: return $\mathbf{a}\sim\mathcal{N}(\mu,\mathrm{I}\sigma^{2})$

In this work, we aim to learn manipulation skills through environment
interaction on _real_ robots, from _visual_ feedback, and with _minimal_ human
supervision and intervention. We first discuss two important metrics for any
agent that aims to quickly and safely learn manipulation skills in the real
world. We then address these limitations with three proposed enhancements to
MoDem in order to develop MoDem-V2.

Strengths and Weaknesses of MoDem: MoDem accelerates learning through a three-
stage framework in which $h_{\theta},\pi_{\theta}$ are first pretrained on a
set of demonstrations using Behavior Cloning (BC), and the resulting policy
$\pi_{\theta}\circ h_{\theta}(\cdot)$ is then used to _seed_ the model, _i.e._
collect a small initial dataset for learning the model (with Gaussian noise
injected into $\pi_{\theta}$ for exploration). After initializing the model on
seeding data, the world model is iteratively used to collect new data via
online interaction, and is optimized on all data: demonstrations, seeding
data, and online interaction data. Using these insights, MoDem demonstrates
accelerated model learning in a collection of simulated tasks. Yet, when
exposed to the real world, MoDem faces a variety of challenges that were
suppressed during its original simulation study.

We find that MoDem exerts excessive forces and torques far beyond those
exemplified in the provided demonstrations (see Section VI). Even at the
beginning of the online interaction phase at which the agent has only observed
data close to the BC policy, MoDem relies on its world model and value
function to discriminate between high and low reward actions. Thus when MoDem
samples actions across the entire action space, the world model and value
function cannot (at least initially) provide good estimates. This can result
in task failure due to poor action selection, or lead to the robot exerting
unsafe forces and torques by choosing consecutive actions that are far apart
in action space. Our primary contribution lies in eliminating these
limitations while also improving the effectiveness of the original method.

###  IV-A MoDem-V2: Real-World Model-Based Reinforcement Learning with
Demonstrations

MoDem generates agent actions by first mapping raw observations into a learned
lower-dimensional space, and then performing efficient short horizon planning
in that latent space with its learned dynamics model and value function. We
propose the following three adaptations to MoDem’s planning procedure in order
to improve its safety while maintaining its strengths in autonomy and data-
efficiency (Algorithm 1).

Policy centered actions: Rather than sampling actions from across the entire
action space, we propose to sample actions from our learned policy. This more
conservative exploration strategy reduces the likelihood of world model and
value function evaluation over unseen regions of the state-action space,
enabling them to better discriminate the quality of the generated actions.

Agency transfer from BC actions to MPC: At the beginning of the online
interaction phase, MoDem immediately begins using its learned world model and
value function to do MPC. Yet both of these components have only seen limited
data near the BC policy, so relying on them to choose actions for multiple
consecutive timesteps at the beginning of interaction can quickly lead the
agent into an unexplored region of the observation-action space from which it
cannot recover. Our remedy is to gradually shift from executing actions
sampled from the BC policy to actions computed by short horizon planning. We
implement this in Algorithm 1 with a hyperparameter $\alpha$ that is
initialized to 0 at the beginning of interaction and linearly increases to 1.0
over a fixed number of interaction steps.

Actor-Critic Ensembles for uncertainty aware planning: The use of actor-critic
ensembles [12] improves the agent’s value estimations in two primary ways.
First, note that each actor is trained by optimizing it to maximize its
corresponding critic. While this provides a solution for efficiently finding
the maximum value of Q over actions, it is subject to significant
overestimation bias [13]. We mitigate this by only evaluating a critic with
final trajectory actions produced by policies not directly optimized to
maximize that particular critic. Actor-critic ensembles also improve value
estimation by providing the agent with a pool of independently trained value
functions, each of which computes its own value estimate. By estimating the
epistemic uncertainty [14] of a trajectory, the agent can make uncertainty-
aware decisions. We incorporate this into MoDem-V2 with weights $w_{1}>0$ and
$w_{2}<0$ in Algorithm 1.

##  V Experimental Design

We design experiments to evaluate the design choices behind MoDem-V2 in
enabling real-world contact-rich manipulation. Our investigation focuses on
the following directions:

  * •

How sample-efficient is MoDem-V2 relative to other methods?

  * •

Is MoDem-V2 safer (_i.e._ fewer safety violations) than other methods
including MoDem?

  * •

Does MoDem-V2 actually enable physical robots to learn real-world manipulation
tasks?

Our first step in answering these questions is to use simulation [15] to
compare our method to both the original MoDem and other strong reinforcement
learning baselines that also uses demonstrations to guide policy learning,
Demonstration Augmented Policy Gradients (DAPG) [16] and the Framework for
Efficient Robot Manipulation (FERM) [17]. We also provide further analysis of
MoDem-V2 by ablating each of its three design decisions. Finally, we deploy
MoDem-V2 onto a physical robot and evaluate its capability to learn four
different manipulation tasks in the real world.

###  V-A Hardware

Figure 3: A view of the in-hand reorientation task as an example of our
hardware setup.

We adopt a set of core hardware components that are common across all of our
experimental tasks. Each task uses a Franka Panda arm. The pushing and picking
tasks use a Robotiq two-fingered gripper, while the in-hand reorientation task
uses a ten-degree of freedom D’Manus hand [18] from the ROBEL ecosystem [19].
For perception, three RealSense D435 cameras are mounted to the left, right
and above the robot. Our hardware setup is depicted in Figure 3.

###  V-B Task Suite

We evaluate MoDem-V2 on four manipulation tasks in both simulation and the
real world. These tasks encompass a variety of manipulation skills, namely
pushing, picking, and in-hand manipulation as shown in Figure 1. We briefly
describe each task below, see Subsection IX-B for more details.

Planar Pushing: This task requires the robot to push an oblong object towards
a fixed goal position on a table top. This task is likely the easiest of all
four tasks, and we view it as base case with which to compare the other tasks.

Inclined Pushing: This task requires the robot to push an object up an incline
to reach a fixed goal position. During execution of the task, the robot must
raise its gripper such that it can progress up the incline while also making
sufficiently precise contact with the block to prevent it from slipping
beneath or around the side of the gripper.

Bin Picking: To complete this task, the robot must grasp a juice container and
then raise it out of the bin. This task requires accurate positioning of the
gripper because the (mostly) non-deformable container has a primary width that
is approximately 65% of the gripper’s maximum aperture. This task also
requires the robot to disambiguate spatially similar states; _e.g._ if the
gripper is above the bin, the robot must understand whether or not the object
is in its grasp so that it can decide to go down towards the bin to pick up
the object or stay where it is in order to receive reward.

In-Hand Reorientation: This task requires the robot to grasp a water bottle
laying on its side and then in-hand manipulate it to an upright position.
Using the multi-fingered D’Manus hand more than doubles the dimensionality of
the action space relative to the previous tasks.

##  VI Experiments

In this section, we evaluate MoDem-V2 against strong baselines in simulation
and measure its performance in real world environments. Our simulation
experiments measure both sample-efficiency and safety (defined in Section
III), two important aspects for any learning method that is used in the real
world. We then use MoDem-V2 to train a robot to perform all four manipulation
tasks in the real world. All experiments train an initial policy with only ten
demos, and each evaluation is aggregated over 30 trials.

Figure 4: The number of safety violations (top row) and success rate (bottom
row) for each of the four manipulation tasks in simulation. Lower is better
for safety violations (top row) while higher is better for episode success
(bottom row). While both MoDem-V2 and MoDem achieve similar or better sample-
efficiency than all of the baselines, MoDem-V2 exhibits significantly safer
learning as evidenced by the drastically lower amount of safety violations.

###  VI-A Simulated Comparison to Baselines

With respect to sample-efficiency, MoDem-V2 and MoDem both significantly
outperform DAPG (State), as shown in Figure 4 (bottom). This is despite DAPG
having access to privileged state information (such as the object’s pose) and
dense rewards (such as rewards based on the distance between the gripper and
the object)! Although FERM achieves similar performance to both versions of
MoDem on the easier pushing tasks, it is unable to learn the bin picking and
in-hand re-orientation tasks. While MoDem achieves similar efficiency to
MoDem-V2 on the first three tasks, it suffers a significant performance drop
in the early stages of training across all tasks. This coincides with the
point at which MoDem begins to aggressively explore the action space,
resulting in the robot applying excessive forces/torques beyond that observed
in the provided demonstrations.

We measured the number of safety violations that occurred throughout training
for MoDem and MoDem-V2, as shown in Figure 4 (top). While both methods
initially commit few violations since their BC policies were trained from the
same demonstrations, MoDem’s safety violations sharply increase as the
interaction phase begins. Thanks to our design decisions, the number of
violations exerted by MoDem-V2 is generally lower throughout online learning
comparatively. Here MoDem-V2 demonstrates that it can achieve similar or
better sample-efficiency than MoDem, while committing significantly fewer
safety violations and thereby safer behavior.

Figure 5: Ablations of the three MoDem-V2 enhancements for all four tasks.
Lower is better for safety violations (top row) while higher is better for
episode success (bottom row). MoDem-V2 achieves both the higher sample-
efficiency of Ensemble and the increased safety profile of Schedule.

###  VI-B Ablation of design choices

We perform ablations of MoDem-V2 by individually adding each improvement
specified in Subsection IV-A to MoDem. As shown in Figure 5, we found that all
of the ablations generally maintained or improved over the sample-efficiency
of MoDem while significantly improving safety by committing fewer violations.
The one exception to this is the bin picking task, for which the Centering and
Ensemble ablations commit a greater number of safety violations whereas
MoDem-v2 exhibits much superior performance. When comparing between ablations
it is clear that each individual modification has both benefits and drawbacks.
First, note that Centering is a necessary sub-component of Schedule. While
Schedule is generally safer than Ensemble, Ensemble typically has better
sample-efficiency. By combining these two ingredients, MoDem-V2 is able to
achieve the improved sample-efficiency of Ensemble while maintaining the
improved safety profile of Schedule.

###  VI-C Real World Results

As suggested by our simulation experiments, the original MoDem is unsafe for
learning real-world manipulation tasks. When we did attempt to run MoDem on
our real robot, we found that its aggressive exploration frequently violated
safety limits at the beginning of online interaction. For example, on the
inclined pushing task, the robot triggered a safety fault within the first two
exploration episodes because it exerted excessive force/torque on the incline.
Due to the unsafe behavior MoDem induces and the significant human
intervention that would be required, it is not feasible to evaluate MoDem in
the real world.

In contrast, MoDem-V2 is capable of safely learning manipulation tasks with
minimal human intervention. MoDem-V2 enabled the robot to significantly exceed
the performance of its initial BC policy with about two hours worth of online
training data or less. Figure 6 shows the success rate of the initial policy
cloned from just ten demonstrations and the best MoDem-V2 agent performance
achieved throughout online training, as well as example trajectories from the
MoDem-V2 agents. Please see sites.google.com/view/modem-v2 for additional
videos and results.

Figure 6: _Left:_ Example rollouts from our MoDem-V2 agents on real-world
manipulation tasks. _Right:_ The success rate of our best performing policy
and its initial BC policy. MoDem-V2 is able to outperform its BC policy in
hours or less.

##  VII Related Work

Visual MBRL: Improving sample-efficiency of visual RL by learning a model of
the environment has been explored extensively in literature [20, 5, 6, 21, 22,
23, 7]. Here we focus on MBRL algorithms that leverage planning. Prior work
typically learns a latent dynamics model from online interaction, and uses a
sampling-based planning technique for action selection with candidate action
sequences evaluated by the learned model. For continuous control, planning can
be formalized as Model Predictive Control (MPC) [20, 6, 7, 9], whereas Monte-
Carlo Tree Search (MCTS) is used for discrete action spaces [22, 23].
Regardless, the majority of work on visual MBRL focus on sample-efficiency in
simulated tasks, where practicality and safety are of limited concern. Our
work extends the MBRL algorithm of [7, 9] which has already been shown to be
very sample-efficient in simulation, and instead focus on the challenges that
arise when training MBRL in the real world.

RL with Demonstrations: RL and imitation learning each have their own
drawbacks: RL is notoriously sample-inefficient due to limited supervision,
and IL is difficult to scale due to the immense cost of data collection. In
recent years, there has been increasing interest in combining the two
paradigms. Depending on the data composition, this can be viewed as either
finetuning an imitation policy with RL [24], or accelerating RL with a small
number of demonstrations [16, 17, 9]. For instance, Baker et al. [24] learns a
general behavioral prior with BC on a large-scale dataset of Minecraft videos,
and finetunes the policy with RL to solve hard-exploration tasks in this
domain. Rajeswaran et al. [16] and Zhan et al. [17] demonstrate that
augmenting model-free policy gradient and Q-learning algorithms, respectively,
effectively alleviate the difficulty of exploration. Hansen et al. [9] show
that model-based methods, too, benefit from a small number of demonstrations,
and learn more efficiently than model-free alternatives in this setting. We
identify shortcomings of [9], propose solutions, and demonstrate the
effectiveness of our method on real hardware.

Real-World Robot Learning: Researchers have explored a wide variety of
approaches for robot learning on real hardware, most of which fall into one of
three categories: learning from human demonstrations [25, 26, 27, 28],
learning from large uncurated datasets [29, 30, 20], learning from online
interaction via RL [31, 32], or any combination of them [17, 33, 34, 35]. Our
work is most similar to Zhan et al. [17] in problem setting (RL with
demonstrations) and experimental setup (robotic manipulation tasks in the real
world). However, we focus on the unique challenges and opportunities of MBRL
for real-world robot learning. Wu et al. [32] study real world MBRL, but
consider simpler tasks with limited variation and do not leaverage
demonstrations.

##  VIII Discussion

In this work, we tackled the challenge of learning manipulation skills in the
real world from only proprioceptive and visual feedback with sparse rewards.
We developed MoDem-V2, a real-world ready adaptation of MoDem, by proposing to
initially center rollouts around the BC policy, gradually increase the
proportion of actions chosen by the learned world model, and implement
uncertainty aware planning with actor-critic ensembles. We evaluated the
sample-efficiency and safety of MoDem-V2 against strong baselines in
simulation and found that it maintained the high sample-efficiency of MoDem
while exhibiting significantly safer behavior through lower contact force
exertion. We found that MoDem-V2 enabled a real, physical robot to learn a
variety of manipulation skills, such as pushing, picking, and in-hand
manipulation, from an hour or less worth of interaction data.

Limitations: One limitation of our work is that it requires a small number of
demonstrations, which may not always be available or easy to collect. Also,
this work assumes that the environment can be reset to a narrow set of
starting states, which may not always be the case in the real-world. In future
work, we hope to explore the reuse of our learned world model across changes
in manipulated object and task goal.

## References

  * [1] A. A. Rusu, M. Vecerík, T. Rothörl, N. M. O. Heess, R. Pascanu, and R. Hadsell, “Sim-to-real robot learning from pixels with progressive nets,” _ArXiv_ , vol. abs/1610.04286, 2016.
  * [2] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 23–30, 2017.
  * [3] A. Kumar, Z. Fu, D. Pathak, and J. Malik, “Rma: Rapid motor adaptation for legged robots,” _ArXiv_ , vol. abs/2107.04034, 2021.
  * [4] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makoviichuk, K. V. Wyk, A. Zhurkevich, B. Sundaralingam, Y. S. Narang, J.-F. Lafleche, D. Fox, and G. State, “Dextreme: Transfer of agile in-hand manipulation from simulation to reality,” _ArXiv_ , vol. abs/2210.13702, 2022.
  * [5] D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” in _Advances in Neural Information Processing Systems 31_. Curran Associates, Inc., 2018, pp. 2451–2463.
  * [6] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson, “Learning latent dynamics for planning from pixels,” in _International Conference on Machine Learning_ , 2019, pp. 2555–2565.
  * [7] N. Hansen, X. Wang, and H. Su, “Temporal difference learning for model predictive control,” _arXiv preprint arXiv:2203.04955_ , 2022.
  * [8] R. M. Shah and V. Kumar, “Rrl: Resnet as representation for reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 9465–9476.
  * [9] N. Hansen, Y. Lin, H. Su, X. Wang, V. Kumar, and A. Rajeswaran, “Modem: Accelerating visual model-based reinforcement learning with demonstrations,” _arXiv preprint arXiv:2212.05698_ , 2022.
  * [10] C. G. Atkeson and S. Schaal, “Robot learning from demonstration,” in _ICML_ , 1997.
  * [11] J.-B. Grill, F. Strub, F. Altch’e, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. Á. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your own latent: A new approach to self-supervised learning,” _ArXiv_ , vol. abs/2006.07733, 2020.
  * [12] Z. Huang, S. Zhou, B. Zhuang, and X. Zhou, “Learning to run with actor-critic ensemble,” _arXiv preprint arXiv:1712.08987_ , 2017.
  * [13] S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in _International conference on machine learning_. PMLR, 2018, pp. 1587–1596.
  * [14] K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep reinforcement learning in a handful of trials using probabilistic dynamics models,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [15] “Robohive – a unified framework for robot learning,” https://sites.google.com/view/robohive, 2020. [Online]. Available: https://sites.google.com/view/robohive
  * [16] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine, “Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations,” in _Proceedings of Robotics: Science and Systems (RSS)_ , 2018.
  * [17] A. Zhan, P. Zhao, L. Pinto, P. Abbeel, and M. Laskin, “A framework for efficient robotic manipulation,” _ArXiv_ , vol. abs/2012.07975, 2020.
  * [18] R. Bhirangi, A. DeFranco, J. Adkins, C. Majidi, A. Gupta, T. Hellebrekers, and V. Kumar, “All the feels: A dexterous hand with large area sensing,” _arXiv preprint arXiv:2210.15658_ , 2022.
  * [19] M. Ahn, H. Zhu, K. Hartikainen, H. Ponte, A. Gupta, S. Levine, and V. Kumar, “Robel: Robotics benchmarks for learning with low-cost robots,” in _Conference on robot learning_. PMLR, 2020, pp. 1300–1313.
  * [20] F. Ebert, C. Finn, S. Dasari, A. Xie, A. X. Lee, and S. Levine, “Visual foresight: Model-based deep reinforcement learning for vision-based robotic control,” _ArXiv_ , vol. abs/1812.00568, 2018.
  * [21] L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, R. Sepassi, G. Tucker, and H. Michalewski, “Model-based reinforcement learning for atari,” _ArXiv_ , vol. abs/1903.00374, 2020.
  * [22] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. P. Lillicrap, and D. Silver, “Mastering atari, go, chess and shogi by planning with a learned model,” _Nature_ , vol. 588 7839, pp. 604–609, 2020.
  * [23] W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y. Gao, “Mastering atari games with limited data,” in _NeurIPS_ , 2021.
  * [24] B. Baker, I. Akkaya, P. Zhokhov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune, “Video pretraining (vpt): Learning to act by watching unlabeled online videos,” _ArXiv_ , vol. abs/2206.11795, 2022.
  * [25] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn, “Bc-z: Zero-shot task generalization with robotic imitation learning,” in _Conference on Robot Learning_ , 2022.
  * [26] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta, “R3m: A universal visual representation for robot manipulation,” in _Conference on Robot Learning_ , 2022.
  * [27] Y. Zhu, A. Joshi, P. Stone, and Y. Zhu, “Viola: Imitation learning for vision-based manipulation with object proposal priors,” _ArXiv_ , vol. abs/2210.11339, 2022.
  * [28] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. C. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manjunath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. S. Ryoo, G. Salazar, P. R. Sanketi, K. Sayed, J. Singh, S. A. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. H. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich, “Rt-1: Robotics transformer for real-world control at scale,” _ArXiv_ , vol. abs/2212.06817, 2022.
  * [29] L. Pinto and A. K. Gupta, “Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours,” _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 3406–3413, 2015.
  * [30] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” _The International Journal of Robotics Research_ , vol. 37, pp. 421 – 436, 2016.
  * [31] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar, “Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost,” _2019 International Conference on Robotics and Automation (ICRA)_ , pp. 3651–3657, 2018.
  * [32] P. Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg, “Daydreamer: World models for physical robot learning,” in _Conference on Robot Learning_. PMLR, 2023, pp. 2226–2240.
  * [33] R. C. Julian, B. Swanson, G. S. Sukhatme, S. Levine, C. Finn, and K. Hausman, “Efficient adaptation for end-to-end vision-based robotic manipulation,” _ArXiv_ , vol. abs/2004.10190, 2020.
  * [34] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman, “Mt-opt: Continuous multi-task robotic reinforcement learning at scale,” _ArXiv_ , vol. abs/2104.08212, 2021.
  * [35] A. Kumar, A. Singh, F. Ebert, Y. Yang, C. Finn, and S. Levine, “Pre-training for robots: Offline rl enables learning new tasks from a handful of trials,” _ArXiv_ , vol. abs/2210.05178, 2022.

##  IX Appendix

###  IX-A MoDem Training Objective

During online interaction, MoDem maintains a replay buffer $\mathcal{B}$ with
trajectories, and the world model learns to minimize the following objective
on length $h$ subtrajectories sampled from the replay buffer:

$\mathcal{L}\left(\theta\right)\doteq\mathop{\mathbb{E}}_{\left(\mathbf{s},\mathbf{a},r,\mathbf{s}^{\prime}\right)_{0:h}\sim\mathcal{B}}\left[\sum_{t=0}^{h}\lambda^{t}\left(\mathcal{L}_{EM}+\mathcal{L}_{RE}+\mathcal{L}_{TD}\right)\right]$
(2)

$\mathcal{L}_{EM}\doteq\mathcolor{black}{\|\
\mathbf{z}_{t}^{\prime}-\operatorname{sg}(h_{\phi}(\mathbf{s}_{t}^{\prime}))\|^{2}_{2}}{\color[rgb]{0.38,0.43,0.77}`\vartriangleleft\emph{Embedding
Prediction}}$ (3)
$\mathcal{L}_{RE}\doteq\mathcolor{black}{\|\hat{r}_{t}-r_{t}\|^{2}_{2}}{\color[rgb]{0.38,0.43,0.77}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\vartriangleleft\emph{Reward
Prediction}}$ (4)
$\mathcal{L}_{TD}\doteq\mathcolor{black}{\|\hat{q}_{t}-q_{t}\|^{2}_{2}}{\color[rgb]{0.38,0.43,0.77}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\vartriangleleft\emph{TD-
learning}}$ (5)

where $\operatorname{sg}$ is the stop-grad operator, $\phi$ is an
exponentially moving average of $\theta$,
$(\mathbf{z}_{t}^{\prime},\hat{r}_{t},\hat{q}_{t})$ are as defined in Equation
1, $q_{t}\doteq
r_{t}+Q_{\phi}(\mathbf{z}_{t}^{\prime},\pi_{\theta}(\mathbf{z}_{t}^{\prime}))$
is the TD-target, and $\lambda\in(0,1]$ is a constant coefficient that assigns
larger weight to temporally close time steps. Similarly, $\pi_{\theta}$ learns
to maximize the objective
$\mathcal{L}_{\pi}(\theta)\doteq\mathbb{E}_{\mathbf{s}_{0:h}\sim\mathcal{B}}\left[\sum_{t=0}^{h}\lambda^{t}Q_{\theta}(\mathbf{z}_{t},\pi_{\theta}(\mathbf{z}_{t}))\right],~{}\mathbf{z}_{t}=h_{\theta}(\mathbf{s}_{t})$
with gradients taken wrt. $\pi_{\theta}$ only.

###  IX-B Additional Task Details

In this subsection, we provide additional details about the reset mechanism
and reward specifications for each of the tasks.

Planar Pushing: In simulation, the robot receives a reward of +1 for each
timestep at which the object is within 5 centimeters of the goal position. For
the real-world task, a computer vision module uses color thresholding in LUV
color space to detect when the green object covers up the red goal area. A
reward of +1 is assigned for each time step at which this occurs. At the end
of each episode, the robot moves back to its starting configuration and the
object is reset by two retractor reels that pull it back towards the starting
position in order to reset it. We collect demonstrations for this task via
teleoperation with an Oculus headset.

Incline Pushing: Here, rewards are specified and demos collected are in the
same manner as the planar pushing task. However this task differs in that it
is reset purely by gravity; when the robot moves back to its starting
configuration the block slides down the incline due to its own weight. This
reset mechanism is less consistent in that it produces a much wider
distribution over starting positions of the object relative to the planar
pushing task, resulting in a higher difficulty.

Bin Picking: In simulation, a reward of +1 is assigned to each time step for
which the object is within 7.5 centimeters away from a goal position above the
table. To detect success at the end of an episode in the real-world, the robot
moves out of the way in order to take a picture of the potentially empty bin,
opens its gripper above the bin (allowing any object in its grasp to fall),
and then takes another image of the bin. These two images are then subtracted
and the result is thresholded to determine if the object was in the robot’s
grasp. If the episode was a success, we iterate through each timestep
backwards in time until we find a timestep at which the robot’s gripper was
less than 10 centimeters above the table. Each timestep later in time than
this timestep is assigned a reward of +1. Note here that simply using gripper
width to detect success would result in false positives because the robot’s
fingertips are somewhat flexible, as well as this particular gripper has two
passive degrees of freedom. We used a hand-coded policy not only to collect
demonstrations for this task, but also as a reset mechanism to ensure that the
object was not pressed up against one of the sides of the bin at the beginning
of an episode.

In-Hand Reorientation: The simulated environment assigns a reward of +1 to
each time step at which the object has a roll and pitch less than 0.03 radians
(i.e. an orientation that corresponds to the bottle being upright). In the
real-world task we use the top-down depth camera to detect if the bottle has
successfully been uprighted at the end of the episode. Given a successful
episode, we iterate backwards through time until we find a time step at which
the hand was less than 10 centimeters above the table. Any time step after
this time step will receive a reward of +1 if all three of the hand’s
metacarpal joints have angles less than a certain threshold corresponding to
the hand being at least partially open and no longer grasping the bottle. Here
the robot arm resets the task by knocking over the bottle if the episode
succeeded and then moving it towards the center of the bin if necessary. We
used a hand-coded policy to collect demonstrations for this task. The high-
level strategy that the robot uses to achieve the task is to initially grasp
the object near the bottle cap with its pinky and thumb fingers. Once it has
lifted the object, it must strike a balance between applying sufficient force
so as to not drop the object but also not too much force so that the bottle
can pivot around the contact axis as the index finger pushes down on the
bottle.

###  IX-C Task Trajectories and Learning Curves

The figure below shows the learning curves of our MoDem-V2 agent on real world
tasks.

Figure 7: MoDem-V2 training performance on real-world manipulation tasks.

###  IX-D Ineffectiveness of Torque Penalization

Our results suggest that simply adding a reward penalty for exerted torque is
ineffective at reducing safety violations and results in a safety profile very
similar to the original MoDem. Here we swept over 3 orders of magnitude for an
additive reward term that penalizes high torques. This reiterates the pivotal
role played by our three proposed modifications in enabling successful
learning on physical robots.

Figure 8: The number of safety violations (top row) and success rate (bottom
row) for each of the four manipulation tasks in simulation. Lower is better
for safety violations (top row) while higher is better for episode success
(bottom row). On top of MoDem-v1, we add an additional term to the reward that
penalizes the use of high torques (additive L2 penalty). While penalization
can help reduce torque in later stages of training, it does not prevent a
large spike in exerted torque at the beginning of online interaction as
indicated by the region shaded in red. This also correlates with a large drop
in performance as indicated by the region shaded in pink. Furthermore, in the
hardest in-hand reorientation task, the use of additive penalty completely
stagnates learning and results in an unsuccessful policy.

###  IX-E MoDem-V2 Hyperparameters

TABLE I: MoDem hyperparameters. We list all relevant hyperparameters for MoDem-V2 below. Highlighted rows are unique to MoDem-V2, while the others are inherited from TD-MPC and MoDem but included for completeness. Hyperparameter | Value
---|---
Discount factor ($\gamma$) | 0.95
Image resolution | $224\times 224$
Frame stack | $2$
Data augmentation |  $\pm 10$ pixel image shifts
Action repeat | $1$
| Seed steps
---
| 7500 (In-hand reorientation)
---
5000 (otherwise)
Pretraining objective | Behavior cloning
Seeding policy | Behavior cloning
Number of demos | $10$
Demo sampling ratio |  $75\%\rightarrow 25\%$ (100K steps)
Replay buffer size | Unlimited
Sampling technique | PER ($\alpha=0.6,\beta=0.4$)
Planning horizon ($H$) | $5$
| MPC action probability ($\alpha$)
---
|  $0.0\rightarrow 1.0$ (25k steps for sim
---
incline pushing)
$0.0\rightarrow 1.0$ (else 100k steps)
Critic mean weight ($w_{1}$) | 1.0
Critic std dev weight ($w_{2}$) | -10.0
Initial parameters ($\mu^{0},\sigma^{0}$) | $(0,2)$
Population size | $510$
Elite fraction | $64$
Iterations | $1$
Policy fraction | $5\%$
Number of particles | $1$
Momentum coefficient | $0.1$
Temperature ($\tau$) | $0.5$
MLP hidden size | $512$
MLP activation | ELU
Latent dimension | $50$
Ensemble size | $5$
Learning rate | 3e-4
Optimizer ($\theta$) | Adam ($\beta_{1}=0.9,\beta_{2}=0.999$)
Temporal coefficient ($\lambda$) | $0.5$
Reward loss coefficient ($c_{1}$) | $0.5$
Value loss coefficient ($c_{2}$) | $0.1$
Consistency loss coefficient ($c_{3}$) | $2$
Exploration schedule ($\epsilon$) |  $0.1\rightarrow 0.05$ (25k steps)
| Batch size
---
| 256 (All sim, and real-world
---
in-hand reorientation)
64 (else due to mem constraints)
Momentum coefficient ($\zeta$) | $0.99$
Steps per gradient update | $1$
$\bar{\theta}$ update frequency | 2