# In-context Exploration-Exploitation for Reinforcement Learning

Zhenwen Dai, Federico Tomasi, Sina Ghiassian
Spotify Research
<EMAIL_ADDRESS>
###### Abstract

In-context learning is a promising approach for online policy learning of
offline reinforcement learning (RL) methods, which can be achieved at
inference time without gradient optimization. However, this method is hindered
by significant computational costs resulting from the gathering of large
training trajectory sets and the need to train large Transformer models. We
address this challenge by introducing an In-context Exploration-Exploitation
(ICEE) algorithm, designed to optimize the efficiency of in-context policy
learning. Unlike existing models, ICEE performs an exploration-exploitation
trade-off at inference time within a Transformer model, without the need for
explicit Bayesian inference. Consequently, ICEE can solve Bayesian
optimization problems as efficiently as Gaussian process biased methods do,
but in significantly less time. Through experiments in grid world
environments, we demonstrate that ICEE can learn to solve new RL tasks using
only tens of episodes, marking a substantial improvement over the hundreds of
episodes needed by the previous in-context learning method.

##  1 Introduction

Transformers represent a highly effective approach to sequence modelling, with
applications extending across multiple domains like text, image, and audio. In
the field of reinforcement learning (RL), Chen et al. (2021) and Janner et al.
(2021) have suggested the concept of addressing offline RL as a sequential
prediction problem using Transformer. This method has proven successful in
tackling a range of tasks using solely offline training when combined with
large-scale sequence modelling techniques (Lee et al., 2022; Reed et al.,
2022a). A notable shortcoming lies with the policy’s inability to self-improve
when employed in online environments. To overcome this, fine-tuning methods
have been introduced, like (Zheng et al., 2022), which enable continued policy
improvement. However, these methods often rely on slow, computationally
expensive gradient-based optimization.

Conversely, in-context learning, a characteristic observed in large language
models (LLMs), can handle new tasks by providing the task details through
language prompts, effectively eliminating the need for fine-tuning. Laskin et
al. (2023) suggest an in-context learning algorithm for RL, which uses a
sequence model to distill a policy learning algorithm from RL training
trajectories. The resulting model is capable of conducting policy learning at
inference-time through an iterative process of action sampling and prompt
augmentation. This method incurs significant computational costs in collecting
extensive training trajectory sets and training large Transformer models that
are needed to model a substantial part of a training trajectory. The key
reason for this high computational cost is the lengthy RL training
trajectories resulting from the slow trial-and-error process of RL policy
learning algorithms.

This paper aims to improve the efficiency of in-context policy learning by
eliminating the need to learn from policy learning trajectories. In an ideal
scenario, efficient policy learning could be achieved through an efficient
trial-and-error process. For simplified RL problems such as multi-armed
bandits (MAB), efficient trial-and-error process such as Thompson sampling and
upper confidence bounds are proven to exist. This process, often referred to
as the exploration-exploitation (EE) trade-off, relies heavily on the
epistemic uncertainty derived from Bayesian belief. However, it is intractable
to infer the exact epistemic uncertainty of sequential RL problems using
conventional Bayesian methods. In light of recent studies on uncertainty
estimation of LLMs (Yin et al., 2023), we examine predictive distributions of
sequence models, demonstrating that, by training with purely supervised
learning on offline data, a sequence model can capture epistemic uncertainty
in sequence prediction. This suggests the potential for implementing EE in
offline RL.

Based on this observation, we develop an in-context exploration-exploitation
(ICEE) algorithm for policy learning. ICEE takes as inputs a sequence of
multiple episodes of the same task and predicts the corresponding action at
every step conditioned on some hindsight information. This offline RL design
resembles the Decision Transformer (DT, Chen et al., 2021), but ICEE tackles
in-context policy learning by modeling multiple episodes of a task while DT
only models a single episode. Furthermore, these episodes do not need to
originate from a training trajectory, thereby averting the high computational
costs associated with generating and consuming learning trajectories. The
action distribution learned in DT is biased towards the data collection
policy, which may not be ideal when it is sub-optimal. To address this bias,
we introduce an unbiased objective and develop a specific form of hindsight
information for efficient EE across episodes.

With experiments, we demonstrate that the EE behavior emerges in ICEE during
inference thanks to epistemic uncertainty in action prediction. This is
particularly evident when applying ICEE to Bayesian optimization (BO), as the
performance of ICEE is on par with a Gaussian process based method on discrete
BO tasks. We also illustrate that ICEE can successfully improve the policy for
a new task with trials-and-errors from scratch for sequential RL problems. To
the best of our knowledge, ICEE is the first method that successfully
incorporates in-context exploration-exploitation into RL through offline
sequential modelling.

##  2 Related Work

Meta Learning. Interest in meta learning or learning to learn algorithms has
recently increased. While learner is an agent that learns to solve a task
using observed data, a learning to learn algorithm includes a meta-learner
that continually improves the learning process of the learner (Schmidhuber et
al., 1996; Thrun & Pratt, 2012; Hospedales et al., 2021; Sutton, 2022). Lots
of work has been done within the space of meta learning. For example, Finn et
al. (2017) proposed a general model-agnostic meta-learning algorithm that
trains the model’s initial parameters such that the model has maximal
performance on a new task after the model parameters have been updated through
a few gradient steps computed with a small amount of data from the new task.
Other works on meta learning include improving optimizers (Andrychowicz et
al., 2016; Li & Malik, 2016; Ravi & Larochelle, 2016; Wichrowska et al.,
2017), improving few-shot learning (Mishra et al., 2017; Duan et al., 2017),
learning to explore (Stadie et al., 2018), and unsupervised learning (Hsu et
al., 2018).

In the space of deep meta RL (Wang et al., 2016), a few works focused on a
special form of meta-learning called meta-gradients. In meta-gradients the
meta-learner is trained by gradients by measuring the effect of meta-
parameters on a learner that itself is also trained using a gradient algorithm
(Xu et al., 2018). In other work, Zheng et al. (2018) used meta-gradients to
learn rewards. Gupta et al. (2018) focused on automating the process of task
design in RL, to free the expert from the burden of manual design of meta-
learning tasks. Similarly, Veeriah et al. (2019) presented a method for a
reinforcement learning agent to discover questions formulated as general value
functions through the use of non-myopic meta-gradients. More recently, meta
gradient reinforcement learning has seen substantial breakthroughs from
performance gains on popular benchmarks to hybrid offline-online algorithms
for meta RL (Xu et al., 2020; Zahavy et al., 2020; Flennerhag et al., 2021;
Mitchell et al., 2021; Yin et al., 2023; Pong et al., 2022). The role of
uncertainty in meta RL has been studied by Zintgraf et al. (2021), which
result in an efficient online meta RL method. This work is then extended by
Dorfman et al. (2021) to the off-policy setting.

Offline Reinforcement Learning. In general, reinforcement learning was
proposed as a fundamentally online paradigm (Sutton, 1988; Sutton et al.,
1999; Sutton & Barto, 2018). This online learning nature comes with some
limitations such as making it difficult to adopt it to many applications for
which it is impossible to gather online data and learn at the same time, such
as autonomous driving and is also sometimes not as data efficient as it could
bem since it might choose to learn from a sample and then dispose the sample
and move on to the next (Levine et al., 2020). One idea for getting more out
of the collected experience is to use replay buffers. When using buffers, part
of the samples are kept in a memory and then are re-used multiple times so
that the agent can learn more from them (Lin, 1992; Mnih et al., 2015). A
variant of reinforcement learning, referred to as offline RL studies RL
algorithms that can learn fully offline, from a fixed dataset of previously
gathered data without gathering new data at the time of learning (Ernst et
al., 2005; Riedmiller, 2005; Lange et al., 2012; Fujimoto et al., 2019; Siegel
et al., 2020; Gulcehre et al., 2020; Nair et al., 2020). Recent literature on
decision transformers also focuses on offline RL (Chen et al., 2021) because
it needs to calculate the return to go at training time, which in turn
necessitates previously gathered data.

In-context learning. In-context RL algorithms are the ones that improve their
policy entirely in-context without updating the network parameters or without
any fine-tuning of the model(Lu et al., 2021). There has been some work
studying the phenomenon of in-context learning trying to explain how learning
in-context might be possible Abernethy et al. (2023); Min et al. (2022). The
“Gato” agent developed by Reed et al. (2022b) works as a multi-model, multi-
task, multi-embodiment generalist agent, meaning that the same trained agent
can play Atari, caption images, chat, stack blocks with a real robot arm only
based on its context. By training an RL agent at large scale, Team et al.
(2023) showed that an in-context agent can adapt to new and open ended 3D
environments. Of special interest to us is Algorithm Distillation (AD), which
is a meta-RL method (Laskin et al., 2023). More specifically, AD is an in-
context offline meta-RL method. Basically, AD is gradient-free—it adapts to
downstream tasks without updating its network parameters.

##  3 Epistemic Uncertainty in Sequence Model Prediction

DT, also known as Upside-down RL, treats the offline policy learning problem
as a sequence modelling problem. In this section, we consider a generic
sequencing model and analyze its predictive uncertainty.

Let ${\bm{X}}_{1:T}=({\bm{x}}_{1},\ldots,{\bm{x}}_{T})$ be a sequence of
inputs of length $T$ and ${\bm{Y}}_{1:T}=({\bm{y}}_{1},\ldots,{\bm{y}}_{T})$
be the corresponding sequence of outputs. Assume that the output sequence is
generated following a step-wise probabilistic distribution parameterized by
${\bm{\theta}}$, ${\bm{y}}_{t}\sim
p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1},{\bm{\theta}})$.
Each sequence is generated with a different parameter sampled from its prior
distribution, ${\bm{\theta}}\sim p({\bm{\theta}})$. These define a generative
distribution of a sequence:

$p({\bm{Y}}_{1:T},{\bm{\theta}}|{\bm{X}}_{1:T})=p({\bm{\theta}})p({\bm{y}}_{1}|{\bm{x}}_{1})\prod_{t=2}^{T}p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1},{\bm{\theta}}).$
(1)

A sequence modeling task is often defined as training an autoregressive model
parameterized by ${\bm{\psi}}$,
$p_{{\bm{\psi}}}({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})$,
given a dataset of sequences
$\mathcal{D}=\\{{\bm{X}}^{(i)},{\bm{Y}}^{(i)}\\}_{i}$ generated from the above
unknown generative distribution. At the limit of infinite data, the objective
of maximum likelihood learning of the above sequence model can be formulated
as
${\bm{\psi}}*=\operatorname*{arg\,max}_{{\bm{\psi}}}\mathcal{L}_{\bm{\psi}}$,

$\begin{split}\mathcal{L}_{\bm{\psi}}=&-\sum_{t}\int
p({\bm{Y}}_{1:t-1}|{\bm{X}}_{1:t-1})\\\
&D_{\mathrm{KL}}\left(p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})||p_{{\bm{\psi}}}({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})\right)d{\bm{Y}}_{1:t-1}+C,\end{split}$
(2)

where $D_{\mathrm{KL}}(\cdot||\cdot)$ denotes the Kullback Leibler divergence
and $C$ is a constant with respect to ${\bm{\psi}}$.

The left hand side distribution in the cross entropy term
$p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})$ is the _true_
predictive distribution of ${\bm{y}}_{t}|{\bm{x}}_{t}$ conditioned on the
observed history ${\bm{Y}}_{1:t-1}$ and ${\bm{X}}_{1:t-1}$, which can be
written as

$p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})=\int
p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1},{\bm{\theta}})p({\bm{\theta}}|{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})d{\bm{\theta}},$
(3)

where

$p({\bm{\theta}}|{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})=\frac{p({\bm{\theta}})p({\bm{Y}}_{1:t-1}|{\bm{X}}_{1:t-1},{\bm{\theta}})}{\int
p({\bm{\theta}}^{\prime})p({\bm{Y}}_{1:t-1}|{\bm{X}}_{1:t-1},{\bm{\theta}}^{\prime})d{\bm{\theta}}^{\prime}}.$
(4)

As shown above, the _true_ predictive distribution of
${\bm{y}}_{t}|{\bm{x}}_{t}$ contains both aleatoric uncertainty and epistemic
uncertainty, in which the epistemic uncertainty is contributed by
$p({\bm{\theta}}|{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})$. With sufficient data and
model capacity, the generative distribution in the sequence model
$p_{{\bm{\psi}}}({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})$
will be trained to match the _true_ predictive distribution. As a result, we
can expect epistemic uncertainty to be contained in the predictive
distribution of a sequence model. Note that the predictive distribution can
only capture the epistemic uncertainty with respect to the parameters of
sequences ${\bm{\theta}}$, but does not include the epistemic uncertainty
regarding the hyper-parameters (if exist).

##  4 In-context Policy Learning

Epistemic uncertainty is the essential ingredient of EE. With the observation
that the predictive distribution sequence model contains epistemic
uncertainty, we design an in-context policy learning algorithm with EE.

Consider the problem of solving a family of RL games based on offline data.
From each game, a set of trajectories are collected from a number of policies,
where
$\tau_{k}^{(i)}=({\bm{o}}_{k,1}^{(i)},{\bm{a}}_{k,1}^{(i)},r_{k,1}^{(i)},\ldots,{\bm{o}}_{k,T_{k}}^{(i)},{\bm{a}}_{k,T_{k}}^{(i)},r_{k,T_{k}}^{(i)})$
is the trajectory of the $k$-th episode of the $i$-th game and ${\bm{o}}$,
${\bm{a}}$, $r$ denote the observed state, action and reward respectively. The
policy used to collect $\tau_{k}^{(i)}$ is denoted as
$\pi_{k}^{(i)}({\bm{a}}_{k,t}^{(i)}|{\bm{o}}_{k,t}^{(i)})$. We concatenate all
the episodes of the $i$-th game into a single sequence
${\bm{\tau}}^{(i)}=(\tau_{1}^{(i)},\ldots,\tau_{K}^{(i)})$. For convenience,
the superscript (i) will be omitted in the following text unless the $i$-th
game is explicitly referred to.

We propose a sequence model that is trained to step-wisely predict
$p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})$, where
$R_{k,t}$ is the _return-to-go_ at the $k$-th episode and the $t$ time step
and ${\bm{H}}_{k,t}=(\tau_{k,1:t-1},{\bm{\tau}}_{1:k-1})$ is the history until
the time step $t$ including the past episodes111Although the learned action
distribution does not need to condition on the return-to-go of the past
actions, for the convenience of a causal Transformer implementation,
${\bm{H}}_{k,t}$ contains the return-to-go of the past actions.. The above
model formulation is similar to DT but a sequence in DT only contains one
episode. Note that, different from AD, the concatenated trajectories do not
need to be from a RL learning algorithm.

As shown in the previous section, by doing maximum likelihood learning on the
collected trajectories, the predictive distribution will be trained to match
the _true_ posterior distribution of action of the data collection policy,

$p({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})=\frac{p(R_{k,t}|{\bm{a}}_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})\pi_{k}({\bm{a}}_{k,t}|{\bm{o}}_{k,t})}{\int
p(R_{k,t}|{\bm{a}}_{k,t}^{\prime},{\bm{o}}_{k,t},{\bm{H}}_{k,t})\pi_{k}({\bm{a}}_{k,t}^{\prime}|{\bm{o}}_{k,t})d{\bm{a}}_{k,t}^{\prime}},$
(5)

where $p(R_{k,t}|{\bm{a}}_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})$ is the
distribution of return after the time step $t$ following $\pi_{k}$.

As shown in (5), the posterior distribution of action is biased towards the
data collection policy. Following such an action distribution allows us to
reproduce the trajectories generated by the data collection policy but will
lead to recreation of suboptimal trajectories if the data collection policy is
not optimal. A more desirable action distribution is the action distribution
that corresponds to the specified return without the influence of data
collection policy, i.e.

$\hat{p}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})=\frac{p(R_{k,t}|{\bm{a}}_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})\mathcal{U}({\bm{a}}_{k,t})}{\int
p(R_{k,t}|{\bm{a}}_{k,t}^{\prime},{\bm{o}}_{k,t},{\bm{H}}_{k,t})\mathcal{U}({\bm{a}}_{k,t}^{\prime})d{\bm{a}}_{k,t}^{\prime}},$
(6)

where $\mathcal{U}({\bm{a}}_{k,t})$ is the uniform random policy, which gives
all the actions equal probabilities. To let the sequence model learn the
unbiased action distribution, the maximum likelihood objective needs to be
defined as

$\mathcal{L}_{{\bm{\psi}}}=\sum_{k,t}\int\hat{p}(R_{k,t},{\bm{a}}_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})\log
p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})dR_{k,t}d{\bm{a}}_{k,t}.$
(7)

After applying the importance sampling trick, the Monte Carlo approximation of
the above objective can be derived as

$\mathcal{L}_{{\bm{\psi}}}\approx\sum_{k,t}\frac{\mathcal{U}({\bm{a}}_{k,t})}{\pi_{k}({\bm{a}}_{k,t}|{\bm{o}}_{k,t})}\log
p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t}),$ (8)

where ${\bm{a}}_{k,t}\sim\pi_{k}({\bm{a}}_{k,t}|{\bm{o}}_{k,t})$ and
$R_{k,t}\sim p(R_{k,t}|{\bm{a}}_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})$, i.e.,
${\bm{a}}_{k,t}$ and $R_{k,t}$ are sampled from the data collection policy
$\pi_{k}$.

##  5 Design of return-to-go

Return-to-go is a crucial component of DT for solving RL tasks at inference
using a trained sequence model. Its return-to-go scheme is designed to
calculate the expected return signal from a single episode. In order to
achieve in-context policy learning, we design the return-to-go across
episodes.

The return-to-go of ICEE consists of two components: one for the individual
steps within an episode and the other for the cross-episode behavior,
$R_{k,t}=(c_{k,t},\tilde{c}_{k})$. The in-episode return-to-go $c_{k,t}$
follows the design used in (Chen et al., 2021), which is defined as the
cumulative rewards starting from the current step
$c_{k,t}=\sum_{t^{\prime}>t}r_{k,t^{\prime}}$. This design borrows the concept
of the cumulative reward of RL and has the benefit of encapsulating the
information of the future rewards following the policy. This is very useful
when the outcomes of future steps strongly depend on the state and action of
the current step. It allows the action that leads to a good future outcome to
be differentiated from the action that leads to a bad future outcome in the
sequence model. The downside is that with a non-expert data collection policy,
the optimal return-to-go at each state is often unobserved. This will limits
the ability of the sequence model to achieve better performance than the data
collection policy at inference time.

In the cross-episode return-to-go design, the situation is different. The
initial states of individual episodes are independent of each other. What
determines the cumulative rewards of individual episodes are the sequence of
actions. If we consider the whole policy space as the action space for each
episode, the cross-episode decision making is closer to MAB, where the policy
is the action and the return of an episode is the MAB reward. Driven by this
observation, we define the return-to-go based on the improvement of the return
of current episode compared to all the previous episodes. Specifically, we
define the cross-episode return-to-go as

$\tilde{c}_{k}=\begin{cases}1&\quad\bar{r}_{k}>\max_{1\leq j\leq
k-1}\bar{r}_{j},\\\ 0&\quad\text{otherwise}.\end{cases}$ (9)

where $\bar{r}_{k}=\sum_{t}r_{k,t}$ is the cumulative reward of the $k$-th
episode. Intuitively, at inference time, by conditioning on $\tilde{c}_{k}=1$,
we take actions from a policy that is “sampled” according to the probability
of being better than all the previous episodes. This encourages the sequence
model to deliver better performance after collecting more and more episodes.
This design avoids the limitation of the need of observing the optimal policy
learning trajectories.

Input: A trained ICEE model $p_{{\bm{\psi}}}$

${\bm{H}}_{1,1}=\\{\\}$ ;

for  _each episode $k=1,\dots,K$_ do

Reset the environment;

Select the episode return-to-go $\tilde{c}_{k}$ ;

for  _each step $t=1,\dots,T_{k}$_ do

Get an observation ${\bm{o}}_{k,t}$ ;

Sample the step return-to-go $c_{k,t}\sim q(c_{k,t})$ ;

Sample an action ${\bm{a}}_{k,t}\sim
p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})$ ;

Take the sampled action ${\bm{a}}_{k,t}$ and collect the reward $r_{k,t}$;

Expand the history
${\bm{H}}_{k,t+1}={\bm{H}}_{k,t}\cup\\{R_{k,t},{\bm{o}}_{k,t},{\bm{a}}_{k,t},r_{k,t}\\}$;

end for

Compute the true return-to-go based on the whole episode
$\\{\hat{R}_{k,t}\\}_{t=1}^{T_{k}}$;

Update the history ${\bm{H}}_{k,T_{k}}$ with the true return-to-go
$\\{\hat{R}_{k,t}\\}_{t=1}^{T_{k}}$;

end for

Algorithm 1 In-context Exploration-Exploitation (ICEE) Action Inference

Action Inference. After training the sequence model, the model can be used to
perform policy learning from scratch. At each step, we sample an action from
the sequence model conditioned on the trajectory so far and a return-to-go for
the step. The return-to-go for action sampling is defined as follows. The
cross-episode return-to-go $\tilde{c}_{k}$ is always set to one to encourage
policy improvements. For the in-episode return-to-go, we follow the action
inference proposed by Lee et al. (2022). During training of ICEE, a separate
sequence model is trained to predict the discretized return from the
trajectories,
$p_{{\bm{\phi}}}(c_{k,t}|\tilde{c}_{k},{\bm{o}}_{k,t},{\bm{H}}_{k,t})$. At
inference time, an in-episode return-to-go for each step is sampled from an
augmented distribution

$q(c_{k,t})\propto
p_{{\bm{\phi}}}(c_{k,t}|\tilde{c}_{k},{\bm{o}}_{k,t},{\bm{H}}_{k,t})(\frac{c_{k,t}-c_{\min}}{c_{\max}-c_{\min}})^{\kappa}.$
(10)

This augmentation biases the return-to-go distribution towards higher values,
which encourages the agent to take actions that leads to better returns. The
return prediction is not combined into the main sequence model as in (Lee et
al., 2022), because in the main sequence model the returns are also fed into
the inputs. In this way, the model quickly figures out $c_{k,t}$ can be
predicted from $c_{k,t-1}$. This is problematic because at inference time the
true return cannot be observed until the end of an episode.

After $c_{k,t}$ is sampled, an action is sampled conditioned on the combined
return-to-go $R_{k,t}$. The resulting state and reward are concatenated to the
trajectory for the next step action prediction. At the end of an episode, we
will recalculate the true $c_{k,t}$ and $\tilde{c}_{k}$ based on the rewards
from the whole episode and update the return-to-go in the trajectory with the
recalculated ones. This makes the trajectory at inference time be as close to
the training trajectories as possible. The description of the action inference
algorithm can be found in Algorithm 1.

##  6 Bayesian Optimization Experiments

(a)

(b)

Figure 1: Discrete BO results on 2D functions with 1024 candidate locations.
16 benchmark functions are used with five trials each. The y-axis shows the
average distance between the current best estimate and the true minimum. The
x-axis of (a) is the number of function evaluations and the x-axis of (b) is
the elapsed time.

Bayesian optimization (BO) is a very successful application of exploration-
exploitation (EE). It is able to search for the optimum of a function with the
minimum number of function evaluations. The Gaussian process (GP) based BO
methods have been widely used in various domains like hyper-parameter tuning,
drug discovery, aerodynamic optimization. To evaluate the EE performance of
ICEE, we apply it to BO and compare it with a GP-based approach using one of
most widely used acquisition functions, expected improvement (EI).

We consider a discrete BO problem. The task is to find the location from a
fixed set of points that has the lowest function value as few number of
function evaluations as possible. BO can be viewed as a special type of multi-
armed bandit (MAB), where each action is associated with a location in a
bounded space. To solve BO with ICEE, we encode the iterative search
trajectory of a function as a single sequence, where ${\bm{a}}_{t}$ is the
location of which a function value is collected at the step $t$, $r_{t}$ is
the corresponding function value and $R_{t}$ is the return-to-go. The
observations $\\{{\bm{o}}_{t}\\}$ can be used to encode any side information
that is known about the function and the minimum. As no such information is
available for generic BO, we do not use $\\{{\bm{o}}_{t}\\}$ in our
experiment. As the function value can seen as the available immediate reward
for, we treat each action as a different episode and use only the episode
return-to-go as the $R_{t}$ here. As each action is associated with a
location, we embed actions by learning a linear projection between the
location space and the embedding space. When decoding a Transformer output for
an action, the logit of an action is generated using a MLP that takes as input
the Transformer output together with the embedding associated with the action.
The design is to tackle the challenge that the set of locations of each
function may be different.

To train ICEE to solve the discrete BO problem, we need to generate training
data that consists of input-output pairs of randomly sampled functions. During
training, the input-output pairs at random locations are generated on-the-fly.
We use a GP with the Matérn 5/2 kernel to sample 1024 points for each
function. The locations of these points are sampled from a uniform
distribution on $[0,1]$. The length scales of the kernel are sampled from a
uniform distribution on $[0.05,0.3]$.

Ideally, for each sampled function, we can use a BO algorithm to solve the
search problem and take the sequence of actions as the training data. This is
very computationally expensive due to the high computational cost of BO
algorithms. Thanks to the EE property of ICEE, the training data do not need
to come from an expert inference algorithm. This is very different from
behavior cloning / algorithm distillation. We use a cheap data collection
algorithm that randomly picks actions according to a probability distribution
that is computed based on the function values of actions. The probability
distribution is defined as follows: The actions are sorted in an ascending
order according to their function values and the action at the $i$-th position
gets the probability
$p({\bm{a}}={\bm{a}}_{i})\propto\exp(\frac{1024-i}{1023}\gamma)$, $\gamma$ is
sampled from a uniform distribution between 0 and 10 for each function.
Intuitively, the locations with lower function values are more likely to be
picked.

After training ICEE, we evaluate its performance on a set of 2D benchmark
functions. We use 16 2D functions implemented in (Kim & Choi, 2017). The input
space of every function is normalized to be between 0 and 1. We randomly
sample a different set of 1024 points from each function and normalize the
resulting function values to be zero-mean and unit-variance. Each function was
given five trials with different initial designs. At each step of search, we
compute the difference of the function value between the current best estimate
and the true minimum. The average performance of all the evaluated functions
is shown in Fig. 1. The performance of ICEE is compared with the GP-based BO
method using the EI acquisition function and the random baseline that picks
locations according to a uniform random probability. “ICEE-biased” denotes a
variant of ICEE that does not use the action bias correction objective as
shown in (8). The search efficiency of ICEE is on par with the GP-based BO
method with EI. Both of them are significantly better than random and are
noticeably better than ICEE-biased. The performance gap between ICEE and ICEE-
biased shows the loss of efficiency due to the biased learned action
distribution. Being able to perform on par with a state-of-the-art BO method
demonstrates that ICEE is able to perform state-of-the-art EE with in-context
inference.

A clear advantage of ICEE is that the whole search is done through model
inference without need of any gradient optimization. In contrast, GP-based BO
methods need to fit a GP surrogate function at each step, which results into a
significant speed difference. Fig. 1(b) shows the same search results with the
x-axis being the elapsed time. All the methods run on a single A100 GPU.
Thanks to the in-context inference, ICEE is magnitude faster than conventional
BO methods. More details of the BO experiments can be found in Appendix D.

##  7 RL Experiments

We investigate the in-context policy learning capability of ICEE on sequential
RL problems. To demonstrate the capability of in-context learning, we focus on
the families of environments that cannot be solved through zero-shot
generalization of a pre-trained model, so in-context policy learning is
necessary for solving the tasks. This implies that some information that are
important to the success of a task is missing from the state representation
and will need to be discovered by the agent. We use the two grid world
environments in (Lee et al., 2022): dark room and dark key-to-door. The
details of them are as follows.

Dark Room. The experiment takes place in a 2D discrete POMDP, where an agent
is placed inside a room to locate a goal spot. The agent has access to its own
location coordinates $(x,y)$, but is unaware of the goal’s placement requiring
it to deduce it from the rewards received. The room’s dimensions are 9x9 with
the possible actions by the agent including moving one step either left,
right, up or down, or staying idle, all within an episode length of 20. Upon
completion, the agent gets placed back at the mid-point of the map. Two
environment variants are considered for this experiment: The Dark Room case
where the agent obtains a reward (r=1) each time the goal is achieved, and the
Dark Room Hard case where the rewards are sparse (r=1 only once for attaining
the goal). Whenever the reward value is not 1, it will be considered as 0.
Different from (Lee et al., 2022), we keep the room size of the hard case to
be 9 x 9.

Dark Key-to-Door. This setting is similar to the Dark Room, but with added
challenging features. The task of the agent is to locate an invisible key to
receive a one-time reward of r=1, and subsequently, identify an invisible door
to gain another one-time reward of r=1. Otherwise, the reward remains at r=0.
The agent’s initial location in each episode resets randomly. The room size is
still 9 x 9 but the episode length increases to 50 steps.

To collect data for offline training, we sample a set of new games for each
mini-batch. We collect $K$ episodes from each game. Thanks to the EE
capability of ICEE, the training data do not need to be from a real RL
learning algorithm like Deep Q-Network (DQN), which is expensive to run.
Instead, we let a cheap data collection policy act for $K$ episodes
independently and concatenate the resulting episodes into a single sequence.
We use an $\epsilon$-greedy version of the “cheating” optimal policy. The
policy knows the goal location that is unknown to the agent and will move
straightly towards the goal with $1-\epsilon$ probability and with $\epsilon$
probability it will take an action that does not make the agent closer to the
goal. For each episode, $\epsilon$ is sampled from a uniform distribution
between 0 and 1. Intuitively, this policy has some chance to solve a game
efficiently when $\epsilon$ is small but on average it does not deliver a good
performance. For Dark Room experiments, each sequence consists of 50 episodes
and for Dark Key-To-Door, it consists of 20 episodes.

Dark Room (Biased). To demonstrate the benefits of EE for sequential RL
problem when the data collection policy cannot be optimal, we create a variant
of the dark room environment. At each step, the data collection policy takes
the “left” action with the $2/3$ probability and with the $1/3$ probability
acts as described above. At training time, the goal can be at any where in the
room and, at evaluation time, the goal will only appear on the right hand side
where $x>5$.

For sequential RL problems, ICEE consists of two sequence models: one for
action prediction and the other for the in-episode return-to-go prediction.
The return-to-go sequence model takes as inputs the sequence of state, action,
reward triplets and predicts the in-episode return-to-go. The action
prediction model takes as inputs the sequence of the triplets and the two
return-to-go $R_{k,t}$ and predicts the action sequence. The two models are
trained together with the same gradient optimizer. To encourage ICEE to solve
the games quickly, when calculating the in-episode return-to-go, a negative
reward, $-1/T$, is given to each step that does not receive a reward, where
$T$ is the episode length. Both $\tilde{c}_{k}$ and $c_{k,t}$ are discrete and
tokenized.

###  7.1 Baseline Methods

Source. We use the data collection policy as a baseline for comparison. As the
data collection policy solves each episode independently, we calculate the
average return across multiple episodes.

Algorithm Distillation (AD, Laskin et al., 2023). The in-context learning
algorithm that distills a RL algorithms from RL training trajectories. AD
predicts action based only on the current states and the history of state,
action and reward triplets. We replicate the implementation of AD using the
Transformer architecture as ICEE. We apply AD to the same training data as
ICEE uses (ignoring the return-to-go signals), despite they are generated from
RL learning trajectories.

AD-sorted. AD is designed to be trained on RL learning trajectories. An
important property of RL learning trajectories is that the performance of the
agent gradually increases throughout training. To mimic such trajectories
using our data, we sort the episodes in a sequence according to the sampled
$\epsilon$ of the data collection policy in the descent order. $\epsilon$
determines how close the data collection policy is to the optimal policy. In
this order, the episodes in a latter position of sequence tend to have a
higher return. We train AD using this sorted sequences instead of the original
ones.

Multi-game Decision Transformer (MGDT, Lee et al., 2022). MGDT is not an in-
context learning algorithm. We train MGDT using only one episode from each
sampled game. The performance of MGDT shows that what the performance of the
agent is when there is no in-context policy learning.

###  7.2 Evaluation & Results

(a)

(b)

(c)

(d)

(e)

Figure 2: Experimental results of in-context policy learning on grid world RL
problems. An agent is expected to solve a game by interacting with the
environment for $K$ episodes without online model updates. The y-axis of (a-d)
shows the average returns over 100 sampled games after each episode. The
y-axis of (e) shows the entropy of the action distribution. The x-axis shows
the number of episodes that an agent experiences.

After training, ICEE will be evaluated on solving a set of sampled games. The
inference algorithm is described in Alg. 1. No online model update is
performed by ICEE and all the baseline methods at evaluation time. For each
sampled game, ICEE and two variants of AD will act for $K$ episodes
consecutively. In each episode, the trajectories from the past episodes are
used as in the history representation. Ideally, a good performing agent
identifies the missing information with as few number of episodes as possible
and then maximizes the return in the following episodes. For each problem, we
sample 100 games and $K$ is 50 for Dark Room and 20 for Key-To-Door.

The experiment results are shown in Fig. 2. ICEE is able to solve the sampled
games efficiently compared to the baseline methods. The EE capability allows
ICEE to search for the missing information efficiently and then acts with
confidence once the missing information is found. An indicator of such
behavior is the continuous decrease of the action entropies as the agent
experiences more episodes (see Fig. 2(e)).

As expected, the original AD learns to mimic the data collection policy, which
results in an average performance slightly below the data collection policy.
MGDT fails to solve most of the games due to the missing information.
Interestingly, despite that the training data is not generated from a RL
learning algorithm, AD-sorted is able to clone the behavior of the data
collection policy with different $\epsilon$ at different stages, which allows
it to solve the games at the end of the sequence.

ICEE-biased is not shown in Fig. 2(a), Fig. 2(b) and Fig. 2(c) as it achieves
similar performance as ICEE does. The reason is that there is no clear bias in
the action distribution of the data collection policy. However, as shown in
Fig. 2(d), for the Dark Room (Biased) environment, ICEE clearly outperforms
ICEE-biased, as it can overcome the bias in the data collection policy and
keep sufficient uncertainty in action distribution to explore the right hand
side of the room. AD-sorted fails the task because it clones the data
collection policy, which is unlikely to solve the tasks due to the action
bias.

##  8 Conclusion

In this paper, we analyze the predictive distribution of sequence models and
show that the predictive distribution can contain epistemic uncertainty, which
inspires the creation of an EE algorithm. We present an in-context EE
algorithm by extending the DT formulation to in-context policy learning and
deriving an unbiased training objective. Through the experiments on BO and
discrete RL problems, we demonstrate that: (i) ICEE can perform EE in in-
context learning without the need of explicit Bayesian inference; (ii) The
performance of ICEE is on par with state-of-the-art BO methods without the
need of gradient optimization, which leads to significant speed-up; (iii) New
RL tasks can be solved within tens of episodes.

## References

  * Abernethy et al. (2023) Jacob Abernethy, Alekh Agarwal, Teodor V Marinov, and Manfred K Warmuth.  A mechanism for sample-efficient in-context learning for sparse retrieval tasks.  _arXiv preprint arXiv:2305.17040_ , 2023.
  * Andrychowicz et al. (2016) Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas.  Learning to learn by gradient descent by gradient descent.  _Advances in neural information processing systems_ , 29, 2016.
  * Balandat et al. (2020) Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy.  BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization.  In _Advances in Neural Information Processing Systems 33_ , 2020.
  * Chen et al. (2021) Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  In _Advances in Neural Information Processing Systems_ , pp. 15084–15097, 2021.
  * Dorfman et al. (2021) Ron Dorfman, Idan Shenfeld, and Aviv Tamar.  Offline meta reinforcement learning – identifiability challenges and effective data collection strategies.  In _Advances in Neural Information Processing Systems_ , pp. 4607–4618, 2021.
  * Duan et al. (2017) Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba.  One-shot imitation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Ernst et al. (2005) Damien Ernst, Pierre Geurts, and Louis Wehenkel.  Tree-based batch mode reinforcement learning.  _Journal of Machine Learning Research_ , 6, 2005.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International conference on machine learning_ , pp. 1126–1135. PMLR, 2017.
  * Flennerhag et al. (2021) Sebastian Flennerhag, Yannick Schroecker, Tom Zahavy, Hado van Hasselt, David Silver, and Satinder Singh.  Bootstrapped meta-learning.  _arXiv preprint arXiv:2109.04504_ , 2021.
  * Fujimoto et al. (2019) Scott Fujimoto, David Meger, and Doina Precup.  Off-policy deep reinforcement learning without exploration.  In _International conference on machine learning_ , pp. 2052–2062. PMLR, 2019.
  * Gulcehre et al. (2020) Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al.  Rl unplugged: A suite of benchmarks for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:7248–7259, 2020.
  * Gupta et al. (2018) Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine.  Unsupervised meta-learning for reinforcement learning.  _arXiv preprint arXiv:1806.04640_ , 2018.
  * Hospedales et al. (2021) Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey.  Meta-learning in neural networks: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(9):5149–5169, 2021.
  * Hsu et al. (2018) Kyle Hsu, Sergey Levine, and Chelsea Finn.  Unsupervised learning via meta-learning.  _arXiv preprint arXiv:1810.02334_ , 2018.
  * Janner et al. (2021) Michael Janner, Qiyang Li, and Sergey Levine.  Offline reinforcement learning as one big sequence modeling problem.  In _Advances in Neural Information Processing Systems_ , pp. 1273–1286, 2021.
  * Kim & Choi (2017) Jungtaek Kim and Seungjin Choi.  BayesO: A Bayesian optimization framework in Python.  https://bayeso.org, 2017.
  * Lange et al. (2012) Sascha Lange, Thomas Gabel, and Martin Riedmiller.  Batch reinforcement learning.  In _Reinforcement learning: State-of-the-art_ , pp. 45–73. Springer, 2012.
  * Laskin et al. (2023) Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, maxime gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih.  In-context reinforcement learning with algorithm distillation.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Lee et al. (2022) Kuang-Huei Lee, Ofir Nachum, Mengjiao (Sherry) Yang, Lisa Lee, Daniel Freeman, Sergio Guadarrama, Ian Fischer, Winnie Xu, Eric Jang, Henryk Michalewski, and Igor Mordatch.  Multi-game decision transformers.  In _Advances in Neural Information Processing Systems_ , pp. 27921–27936, 2022.
  * Levine et al. (2020) Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Li & Malik (2016) Ke Li and Jitendra Malik.  Learning to optimize.  _arXiv preprint arXiv:1606.01885_ , 2016.
  * Lin (1992) Long-Ji Lin.  Self-improving reactive agents based on reinforcement learning, planning and teaching.  _Machine learning_ , 8:293–321, 1992.
  * Lu et al. (2021) Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch.  Pretrained transformers as universal computation engines.  _arXiv preprint arXiv:2103.05247_ , 1, 2021.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer.  Rethinking the role of demonstrations: What makes in-context learning work?  _arXiv preprint arXiv:2202.12837_ , 2022.
  * Mishra et al. (2017) Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel.  A simple neural attentive meta-learner.  _arXiv preprint arXiv:1707.03141_ , 2017.
  * Mitchell et al. (2021) Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn.  Offline meta-reinforcement learning with advantage weighting.  In _International Conference on Machine Learning_ , pp. 7780–7791. PMLR, 2021.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533, 2015.
  * Nair et al. (2020) Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine.  Awac: Accelerating online reinforcement learning with offline datasets.  _arXiv preprint arXiv:2006.09359_ , 2020.
  * Pong et al. (2022) Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine.  Offline meta-reinforcement learning with online self-supervision.  In _International Conference on Machine Learning_ , pp. 17811–17829. PMLR, 2022.
  * Ravi & Larochelle (2016) Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  In _International conference on learning representations_ , 2016.
  * Reed et al. (2022a) Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas.  A generalist agent.  May 2022a.
  * Reed et al. (2022b) Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al.  A generalist agent.  _arXiv preprint arXiv:2205.06175_ , 2022b.
  * Riedmiller (2005) Martin Riedmiller.  Neural fitted q iteration–first experiences with a data efficient neural reinforcement learning method.  In _Machine Learning: ECML 2005: 16th European Conference on Machine Learning, Porto, Portugal, October 3-7, 2005. Proceedings 16_ , pp. 317–328. Springer, 2005.
  * Schmidhuber et al. (1996) Juergen Schmidhuber, Jieyu Zhao, and MA Wiering.  Simple principles of metalearning.  _Technical report IDSIA_ , 69:1–23, 1996.
  * Siegel et al. (2020) Noah Y Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller.  Keep doing what worked: Behavioral modelling priors for offline reinforcement learning.  _arXiv preprint arXiv:2002.08396_ , 2020.
  * Stadie et al. (2018) Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever.  Some considerations on learning to explore via meta-reinforcement learning.  _arXiv preprint arXiv:1803.01118_ , 2018.
  * Sutton (1988) Richard S Sutton.  Learning to predict by the methods of temporal differences.  _Machine learning_ , 3:9–44, 1988.
  * Sutton (2022) Richard S Sutton.  A history of meta-gradient: Gradient methods for meta-learning.  _arXiv preprint arXiv:2202.09701_ , 2022.
  * Sutton & Barto (2018) Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Sutton et al. (1999) Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour.  Policy gradient methods for reinforcement learning with function approximation.  _Advances in neural information processing systems_ , 12, 1999.
  * Team et al. (2023) Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, et al.  Human-timescale adaptation in an open-ended task space.  _arXiv preprint arXiv:2301.07608_ , 2023.
  * Thrun & Pratt (2012) Sebastian Thrun and Lorien Pratt.  _Learning to learn_.  Springer Science & Business Media, 2012.
  * Veeriah et al. (2019) Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L Lewis, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh.  Discovery of useful questions as auxiliary tasks.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Wang et al. (2016) Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick.  Learning to reinforcement learn.  _arXiv preprint arXiv:1611.05763_ , 2016.
  * Wichrowska et al. (2017) Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando Freitas, and Jascha Sohl-Dickstein.  Learned optimizers that scale and generalize.  In _International conference on machine learning_ , pp. 3751–3760. PMLR, 2017.
  * Xu et al. (2018) Zhongwen Xu, Hado P van Hasselt, and David Silver.  Meta-gradient reinforcement learning.  _Advances in neural information processing systems_ , 31, 2018.
  * Xu et al. (2020) Zhongwen Xu, Hado P van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver.  Meta-gradient reinforcement learning with an objective discovered online.  _Advances in Neural Information Processing Systems_ , 33:15254–15264, 2020.
  * Yin et al. (2023) Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang.  Do large language models know what they don’t know?  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pp. 8653–8665, July 2023.
  * Zahavy et al. (2020) Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh.  A self-tuning actor-critic algorithm.  _Advances in neural information processing systems_ , 33:20913–20924, 2020.
  * Zheng et al. (2022) Qinqing Zheng, Amy Zhang, and Aditya Grover.  Online decision transformer.  In _Proceedings of the 39th International Conference on Machine Learning_ , Proceedings of Machine Learning Research, pp. 27042–27059, 2022\.
  * Zheng et al. (2018) Zeyu Zheng, Junhyuk Oh, and Satinder Singh.  On learning intrinsic rewards for policy gradient methods.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Zintgraf et al. (2021) Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson.  Varibad: Variational bayes-adaptive deep rl via meta-learning.  _Journal of Machine Learning Research_ , 22(289):1–39, 2021.

Appendix

##  Appendix A Derivation of the Objective of a Sequence Model

The maximum likelihood objective of a sequence model in (2) can be derived in
the following steps.

$\displaystyle\mathcal{L}_{\bm{\psi}}=$ $\displaystyle\int
p({\bm{Y}}_{1:T},{\bm{\theta}}|{\bm{X}}_{1:T})\log
p_{{\bm{\psi}}}({\bm{Y}}_{1:T}|{\bm{X}}_{1:T})d{\bm{Y}}_{1:T}d{\bm{\theta}}$
$\displaystyle=$ $\displaystyle\int
p({\bm{Y}}_{1:T},{\bm{\theta}}|{\bm{X}}_{1:T})\log\prod_{t=1}^{T}p_{{\bm{\psi}}}({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})d{\bm{Y}}_{1:T}d{\bm{\theta}}$
$\displaystyle=$ $\displaystyle\sum_{t=1}^{T}\int
p({\bm{Y}}_{1:T},{\bm{\theta}}|{\bm{X}}_{1:T})\log
p_{{\bm{\psi}}}({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})d{\bm{Y}}_{1:T}d{\bm{\theta}}$
$\displaystyle=$ $\displaystyle\sum_{t=1}^{T}\int
p({\bm{Y}}_{1:t}|{\bm{X}}_{1:t})\log
p_{{\bm{\psi}}}({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})d{\bm{Y}}_{1:t}$
$\displaystyle=$ $\displaystyle\sum_{t=1}^{T}\int
p({\bm{Y}}_{1:t-1}|{\bm{X}}_{1:t-1})p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})\log
p_{{\bm{\psi}}}({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})d{\bm{Y}}_{1:t}$
$\displaystyle=$ $\displaystyle\sum_{t=1}^{T}\int
p({\bm{Y}}_{1:t-1}|{\bm{X}}_{1:t-1})\Big{(}p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})\log
p_{{\bm{\psi}}}({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})$
$\displaystyle-p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})\log
p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})\Big{)}d{\bm{Y}}_{1:t}$
$\displaystyle+\int
p({\bm{Y}}_{1:t-1}|{\bm{X}}_{1:t-1})p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})\log
p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})d{\bm{Y}}_{1:t}$
$\displaystyle=$ $\displaystyle\sum_{t=1}^{T}\int
p({\bm{Y}}_{1:t-1}|{\bm{X}}_{1:t-1})\Big{(}-D_{\mathrm{KL}}\left(p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})||p_{{\bm{\psi}}}({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})\right)\Big{)}d{\bm{Y}}_{1:t-1}$
$\displaystyle+\int
p({\bm{Y}}_{1:t-1}|{\bm{X}}_{1:t-1})H\Big{(}p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})\Big{)}d{\bm{Y}}_{1:t-1}$
$\displaystyle=$ $\displaystyle-\sum_{t}\int
p({\bm{Y}}_{1:t-1}|{\bm{X}}_{1:t-1})D_{\mathrm{KL}}\left(p({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})||p_{{\bm{\psi}}}({\bm{y}}_{t}|{\bm{x}}_{t},{\bm{X}}_{1:t-1},{\bm{Y}}_{1:t-1})\right)d{\bm{Y}}_{1:t-1}+C,$

##  Appendix B Derivation of the Unbiased Objective

The Decision Transformer’s approach to offline RL includes the training of an
action distribution conditioned the return, thereby allowing for the sampling
of an action at the time of inference by providing the projected return
(return-to-go). Given that the return is the result of present and downstream
actions, the distribution of action that a model tries to learn can be
reframed as an action’s posterior distribution, as is presented in equation
(5). Note that equation (5) outlines the data distribution, which should be
distinguished from the neural network model
$p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})$. As
noted in equation (5), the action’s distribution is proportionate to the
return’s distribution, and this is subsequently weighted by the probability of
action from the data collection policy. As it is the posterior distribution
derived by the Bayes rule, we denote it as the “true” action posterior
distribution.

To put this intuitively, if a model can accurately match equation (5), it will
result in the action distribution being skewed towards the data collection
policy. For instance, in a pre-recorded sequence, should an action be randomly
selected with extremely low probability from the data collection policy, yet
yield a high return, the subsequent action distribution in equation (5) would
attribute a minute probability to the relevant action, given the high return.
Despite the observation of a high return following the action, which would
suggest a high probability of
$p(R_{k,t}|{\bm{a}}_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})$, the resulting
probability of action is weighted by the probability of action in the data
collection policy, $\pi_{k}({\bm{a}}_{k,t}|{\bm{o}}_{k,t})$, resulting in a
small value. Therefore, even though (5) is the true posterior distribution of
action, it is not the desirable action distribution for our model.

Ideally, the action distribution, as demonstrated in equation (6), should be
proportional only to the return’s distribution and would be unaffected by the
data collection policy. With such a distribution, the undesired devaluation
due to the data collection policy would be eliminated, thereby resolving the
mentioned issue.

As explained above, we would like to learn an action distribution in (6)
instead of the action distribution in (5). However, since (5) is the true
action distribution of data, the common maximum likelihood training objective
will let the model match the action distribution in (5).

The unbiased objective of learning the action distribution in (8) can be
derived in the following steps.

To let the model learn the action distribution in (6) instead, we first state
the desirable training objective as if the data follow the distribution (6):

$\displaystyle\mathcal{L}_{{\bm{\psi}}}=$
$\displaystyle\sum_{k,t}\int\hat{p}(R_{k,t},{\bm{a}}_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})\log
p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})dR_{k,t}d{\bm{a}}_{k,t}$
$\displaystyle=$
$\displaystyle\sum_{k,t}\int\hat{p}(R_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})\Big{(}\int\hat{p}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})\log
p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})d{\bm{a}}_{k,t}\Big{)}dR_{k,t}$

Then, we apply the importance sampling trick to introduce the true action
posterior in the equation:

$\displaystyle\mathcal{L}_{{\bm{\psi}}}=$
$\displaystyle\sum_{k,t}\int\hat{p}(R_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})\Big{(}\int
p({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})\frac{\hat{p}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})}{p({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})}$
$\displaystyle\log
p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})d{\bm{a}}_{k,t}\Big{)}dR_{k,t}$

After some rearrangement of the equation, we get a much clearer formulation of
the objective:

$\displaystyle\mathcal{L}_{{\bm{\psi}}}=$
$\displaystyle\sum_{k,t}\int\hat{p}(R_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})\Big{(}\int
p({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})\frac{\frac{p(R_{k,t}|{\bm{a}}_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})\mathcal{U}({\bm{a}}_{k,t})}{\hat{p}(R_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})}}{\frac{p(R_{k,t}|{\bm{a}}_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})\pi_{k}({\bm{a}}_{k,t}|{\bm{o}}_{k,t})}{p(R_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})}}$
$\displaystyle\log
p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})d{\bm{a}}_{k,t}\Big{)}dR_{k,t}$
$\displaystyle=$
$\displaystyle\sum_{k,t}\int\hat{p}(R_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})\Big{(}\int
p({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})\frac{\mathcal{U}({\bm{a}}_{k,t})p(R_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})}{\pi_{k}({\bm{a}}_{k,t}|{\bm{o}}_{k,t})\hat{p}(R_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})}$
$\displaystyle\log
p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})d{\bm{a}}_{k,t}\Big{)}dR_{k,t}$
$\displaystyle=$ $\displaystyle\sum_{k,t}\int
p(R_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})\Big{(}\int
p({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})\frac{\mathcal{U}({\bm{a}}_{k,t})}{\pi_{k}({\bm{a}}_{k,t}|{\bm{o}}_{k,t})}$
$\displaystyle\log
p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})d{\bm{a}}_{k,t}\Big{)}dR_{k,t}$
$\displaystyle=$ $\displaystyle\sum_{k,t}\int
p(R_{k,t},{\bm{a}}_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})\frac{\mathcal{U}({\bm{a}}_{k,t})}{\pi_{k}({\bm{a}}_{k,t}|{\bm{o}}_{k,t})}\log
p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t})d{\bm{a}}_{k,t}dR_{k,t}$

Note the probability distribution in front is now the joint distribution of
return and action in the data distribution. We apply the Monte Carlo
approximation to the integral by considering the recorded data are samples
from the data distribution. We get the proposed training objective.

$\displaystyle\mathcal{L}_{{\bm{\psi}}}\approx$
$\displaystyle\sum_{k,t}\frac{\mathcal{U}({\bm{a}}_{k,t})}{\pi_{k}({\bm{a}}_{k,t}|{\bm{o}}_{k,t})}\log
p_{{\bm{\psi}}}({\bm{a}}_{k,t}|R_{k,t},{\bm{o}}_{k,t},{\bm{H}}_{k,t}),\quad
R_{k,t},{\bm{a}}_{k,t}\sim
p(R_{k,t},{\bm{a}}_{k,t}|{\bm{o}}_{k,t},{\bm{H}}_{k,t})$

##  Appendix C Implementation Details

ICEE is implemented based on nanoGPT 222https://github.com/karpathy/nanoGPT.
For the RL experiments, ICEE contains 12 layers with 128 dimensional
embeddings. There are 4 heads in the multi-head attention. We use the Adam
optimizer with the learning rate $10^{-5}$.

##  Appendix D Bayesian Optimization Experiments

We consider a discrete BO problem. The task is to find the location from a
fixed set of points that has the lowest function value as few number of
function evaluations as possible. At the beginning of search, the function
values of some randomly picked locations are given. The BO algorithm will be
asked to suggest a location where the function minimum lies and then, the
function value will be collected from the suggested location. The suggestion-
evaluation loop will be repeated with a fixed number of times. The performance
of a BO algorithm is evaluated by how quickly it can find the function
minimum.

The list of 2D functions used for evaluations are: Branin, Beale, Bohachevsky,
Bukin6, DeJong5, DropWave, Eggholder, GoldsteinPrice, HolderTable, Kim1, Kim2,
Kim3, Michalewicz, Shubert, SixHumpCamel, ThreeHumpCamel.

Figure 3: Discrete BO results on 2D functions with 1024 candidate locations.
The shading area shows the 95% confidence interval of the mean.

The expected improvement baseline is implemented using BOTorch (Balandat et
al., 2020). We used the “SingleTaskGP” class for the GP surrogate model, which
uses a Matern 5/2 kernel with a Gamma prior over the length scales.

##  Appendix E Quantitative results of the discrete RL experiments

Please find the quantitative comparison of the RL experiments shown in Fig. 2
below. The shown values are the average returns over 100 sampled games and the
values in the parentheses are the confidence intervals of the mean estimates,
which correspond to the shaded area in the figure. We take three time points
along the trajectories of in-context policy learning. As MGDT is not able to
update policy at inference time, we only estimate a single mean return for
each game.

| Dark Room (10th Episode) | Dark Room (30th Episode) | Dark Room (50th Episode)
---|---|---|---
ICEE | 8.15 (1.29) | 12.37 (1.14) | 13.61 (0.86)
AD | 3.74 (1.15) | 4.51 (1.17) | 4.03 (1.15)
AD-sorted | 0.05 (0.05) | 3.83 (0.87) | 12.48 (1.37)
MGDT | 1.86 (0.93) | 1.86 (0.93) | 1.86 (0.93)
source | 5.13 (1.19) | 5.13 (1.19) | 5.13 (1.19)

| Dark Room (Hard) (10th) | Dark Room (Hard) (30th) | Dark Room (Hard) (50th)
---|---|---|---
ICEE | 0.48 (0.10) | 0.74 (0.09) | 0.79 (0.08)
AD | 0.33 (0.09) | 0.43 (0.10) | 0.43 (0.10)
AD-sorted | 0.08 (0.05) | 0.55 (0.10) | 0.75 (0.08)
MGDT | 0.09 (0.06) | 0.09 (0.06) | 0.09 (0.06)
source | 0.51 (0.10) | 0.51 (0.10) | 0.51 (0.10)

| Dark Key-To-Door (5th) | Dark Key-To-Door (10th) | Dark Key-To-Door (20th)
---|---|---|---
ICEE | 1.04 (0.15) | 1.50 (0.12) | 1.84 (0.08)
AD | 0.67 (0.15) | 1.02 (0.17) | 0.94 (0.17)
AD-sorted | 0.17 (0.08) | 0.84 (0.14) | 1.77 (0.09)
MGDT | 0.34 (0.11) | 0.34 (0.11) | 0.34 (0.11)
source | 1.10 (0.19) | 1.10 (0.19) | 1.10 (0.19)


# Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual
Language Model

Minh-Hao Van
Department of EECS
University of Arkanasas
Fayetteville, AR, USA
<EMAIL_ADDRESS>
Xintao Wu
Department of EECS
University of Arkanasas
Fayetteville, AR, USA
<EMAIL_ADDRESS>

###### Abstract

Recently, large language models (LLMs) have taken the spotlight in natural
language processing. Further, integrating LLMs with vision enables the users
to explore more emergent abilities in multimodality. Visual language models
(VLMs), such as LLaVA, Flamingo, or GPT-4, have demonstrated impressive
performance on various visio-linguistic tasks. Consequently, there are
enormous applications of large models that could be potentially used on social
media platforms. Despite that, there is a lack of related work on detecting or
correcting hateful memes with VLMs. In this work, we study the ability of VLMs
on hateful meme detection and hateful meme correction tasks with zero-shot
prompting. From our empirical experiments, we show the effectiveness of the
pretrained LLaVA model and discuss its strengths and weaknesses in these
tasks.

Warning: This paper contains examples of hate speech.

##  1 Introduction

The rapid development of information technology offers better communication
channels for people to express their thoughts and opinions. Social media
platforms nowadays are sources of diverse content, e.g., text, image, audio,
or video. As a double-edged sword, the content is not always safe for humans
since it might include explicit or implicit harm to the viewers. Online memes,
often used for playful or humorous expression, are sometimes misappropriated
to disseminate hate speech against individuals or groups based on their
characteristics. Hateful meme detection is a critical problem that social
media platforms want to work on to protect online users [8]. Detecting hateful
memes, however, is a non-trivial task since memes are multimodal with visual
and textual elements. Further, the hateful content is not always explicitly
expressed in the meme but is implicitly hidden within the story behind it.
Therefore, intelligent systems are urgently needed to process the vast digital
content uploaded to social platforms. Many deep learning techniques can be
used for this detection task [5, 22, 4, 7, 16, 12].

Large language models (LLMs) have shown incredible performance on multiple
natural language processing tasks in recent years. With a strong ability to
learn human language from vast corpora of text, LLMs can encode enormous
amounts of knowledge as embedding vectors, which is a backbone for
transforming the knowledge to different forms of representations with the help
of adapters [32] or decoder modules [29]. This flexibility allows AI
practitioners to combine text with other sources of information to build
multimodally large models. LLMs and their variations can not only perform well
on linguistic tasks but also be applied in other fields such as computer
vision [1, 20, 14, 15, 21], medical data [11, 26] and tabular data analysis
[6].

In this paper, we introduce a simple but efficient framework designed for two
key objectives: (1) detecting hateful memes and (2) correcting the hate speech
from memes by leveraging a pretrained visual language model (VLM). While
multimodal language models have gained traction in different fields, socially
responsible problems remain undiscovered. To the best of our knowledge, our
work is the first to correct the hatefulness in memes, an essential
application to create safe and respectful online social media sites. For
instance, some users may wish to share humorous or satirical memes on their
social media profiles, yet there persists a valid concern about whether such
content inadvertently conveys hurtful messages to viewers. Our framework
offers a valuable solution with only a zero-shot prompt. Additionally, we show
that VLM exhibits promising performance in both evaluating the hatefulness and
generating more considerate text drawing upon its knowledge from the training
corpus, which is an impossible task for classification models. Even though our
framework with multimodal LLM can not compete with all traditional deep
learning classifiers, it can outperform multiple baselines without fine-tuning
or OCR text. Especially our results reveal that the LLaVA model yields
substantial improvements in AUROC scores with carefully crafted instructions,
compared to prior findings in [1, 2].

##  2 Related Works

###  2.1 Hate Speech Detection

There has been an effort from the AI research community to call for solutions
to hate speech problems, including hate speech detection datasets [17, 18] or
hateful memes detection challenge [8]. To deal with the hate speech detection
task, [31] proposed a framework to detect hateful text by evaluating coded
words, which silently represent hateful meanings on social media, by applying
a two-layer network on contextual embeddings from ELMo [24]. BERT-MRP [10]
proposed a two-stage BERT-based model, including masked rationale prediction
(MRP) and hate speech detection. [33] also evaluated Open Pretrain Transformer
(OPT) on the hate speech detection task via zero-, one-, or few-shot cases.
Besides hateful text, hatefulness can exist in multimodal form. [8] introduced
a challenge about multimodally hateful memes detection. With enormous
baselines on three categories: unimodal models from computer vision or natural
language processing tasks [5, 22, 4], multimodal models by combining smaller
unimodally pretrained models [7, 16, 12], or multimodal models [16, 12], the
challenge still calls for more contribution from communities. Many state-of-
the-art methods for detecting hateful memes have been proposed during the
challenge. [35, 19, 28, 13, 23] proposed an ensemble approach of one or
multiple vision and language models to get the final prediction.

###  2.2 Visual Language Models

Besides traditional deep learning models that were specifically trained for
classification tasks, visual language models have been catching much attention
over many interesting applications. CLIP [21] proposed a new network to
connect image and text data. The method guides the model to learn visual
concepts in classification tasks using the supervision from natural language.
OpenAI GPT-4 [20], MiniGPT-4 [34], Flamingo [1], OpenFlamingo [2], and LLaVA
[14] demonstrated their effectiveness on various vision-language tasks such as
question-answering or human-like chatbot. Along with groundbreaking large
foundation models, many research works have explored the emergent abilities of
large models. [15] showed that LLaVA is able to outperform other methods in
OCR tasks with zero-shot learning. LLaVA-Med [11] and the multimodal version
of Med-PaLM [26] could analyze the medical images and answer related questions
like a human expert. By connecting LLM with multiple input encoders and
diffusion decoders, NExT-GPT [29] introduced an architecture for multimodal
inputs and multimodal outputs.

##  3 Exploring Hateful Meme with Visual Language Model

We utilize the pre-trained VLM to explore hateful memes in two scenarios:
hateful memes detection and hatefulness correction.

###  3.1 Hateful Meme Detection

A dataset $\mathcal{D}=(\mathcal{X},\mathcal{Y})$ contains multiple pairs of
data sample $(x_{i},y_{i})$’s, where $x_{i}\in\mathcal{X}$ and
$y\in\mathcal{Y}$ are input image and output label, respectively. For memes
dataset, each input image $x_{i}$ contains the visual part $v_{i}$ and the
textual part $t_{i}$, e.g., $\mathcal{X}=(\mathcal{V},\mathcal{T})$. We define
the hateful memes detection as a classification task
$f:\mathcal{V}\times\mathcal{T}\rightarrow\mathcal{Y}$.

Although $f$ is a binary classification task (hateful or non-hateful), it is
not trivial to classify the hatefulness since $x_{i}$ is naturally a
multimodal meme built upon visual and textual elements, requiring a practical
approach to infer the meaning from both. Another challenging problem is how to
teach the model to understand the hateful context of the meme. To overcome
these challenges, we prompt the pretrained VLM to detect hateful memes with a
zero-shot approach. To do that, we briefly discuss the hatefulness definition
and the zero-shot prompt for guiding the model.

Hatefulness definition. While one can prompt the VLM to enlarge its
capabilities on various tasks, it is essential to note that the model’s
performance dramatically depends on the quality of prompts, including clear
instructions for the specific task. Even though the pretrained model, like
LLaVA, can provide positive and helpful answers to humans, the training
dataset only captures general human knowledge without a specification in a
hateful content detection scheme. Therefore, a clear hatefulness definition is
necessary. We adjust the definition of hate speech in [8] as an instruction in
the prompt.

Zero-shot prompt. In Figure 1, we show three examples of the same hateful meme
but different prompts from naive to complete instruction. The first prompt
(Figure 1(a)) shows a naïve prompt which is a simple question. The VLM clearly
describes people inside the image and the text. However, the prediction is
wrong, which is understandable since there is no clear instruction for
hatefulness. The second prompt (Figure 1(b)) has more detailed instructions
about considering the meaning of the visual and textual parts separately
before going to the conclusion. The VLM does better reasoning where it is able
to give different layers of meanings but still fails in predicting the hateful
class. With the above prompts, the content inside the image is analyzed
accurately, but the reasoning does not reach a good threshold. The third
prompt (Figure 1(c)) shows our complete prompt with detailed instructions
about hatefulness definition and classification criteria. While the content
analysis is similar to previous responses, the conclusion shows a more
confident explanation and correct prediction from the model.

(a) Naïve prompt.

(b) Detailed prompt.

(c) Complete prompt.

Figure 1: Building zero-shot prompt for hateful meme detection.

###  3.2 Hateful Meme Correction

While hateful meme detection is well-explored, a well-formulated problem for
correction still needs to be addressed. Let
$\mathcal{D}_{H}\subset\mathcal{D}$ be a subset of hateful memes in
$\mathcal{D}$. For each $(x_{i},y_{i})\in\mathcal{D}_{H}$,
$x_{i}=(v_{i},t_{i})$ and $y_{i}=1$ (hateful). We define a model
$g:\mathcal{V}\times\mathcal{T}\rightarrow\mathcal{T}$, which finds the new
text $t_{i}^{\prime}=g(v_{i},t_{i})$ such that the new class
$y_{i}^{\prime}=0$ (non-hateful) with input
$x_{i}^{\prime}=(v_{i},t_{i}^{\prime})$. In this work, we utilize the language
generation ability of VLM to craft a positive and respectful text that can
convert hateful memes to non-hateful ones.

Similar to the detection task, we need to craft clear and unambiguous
instructions for the VLM to achieve an effective correction. Since we know the
meme is hateful, there is no need to include a hatefulness definition and
classification criteria. The below paragraph is the prompt for generating new
text to correct hatefulness from memes:

> This meme is hateful and should not be shared or promoted on social media.
> You are serious and critical when evaluating and generating the content.
> Suppose that we want to have a non-hateful meme using the same image. Your
> task is to generate the new text such that the new meme is not hateful and
> promotes positive views of this image.

##  4 Experiments

###  4.1 Experimental Settings

Dataset. We evaluate the performance of our proposed approach using the
Hateful Meme Challenge (HMC) Dataset [8] from Facebook AI (now Meta AI). Phase
I has 10,000 multimodal memes, including 8,500 memes for the training set, 500
memes for the seen dev set, and 1,000 images for the seen test set. Phase II
includes 1,000 (2,000) memes for unseen dev (test) sets. The task is to
classify each meme as hateful or non-hateful.

Model. We use LLaVA [11] as the base model for experiments. The LLaVA version
used in this work is fine-tuned on Llama-2-13B chat model [27], named LLaVA-
Llama-2-13B (LL-2) in this study. For hyper-parameters, we set the temperature
as 0.7 and top-p as 1.0.

Baselines. We compare our approach with results from the Hateful Memes
Challenge [8], including 11 baselines and five challenge winners. For
multimodal LLM baselines, we include benchmark results on this dataset from
Flamingo (Fl) [1] and OpenFlamingo (OF) [2]. Due to space limits, we leave the
details of methods in the Appendix A.

Running time. The inference time for each sample is approximately $7.9$
seconds with the maximum output length of 512 tokens, including image loading
and tokenizing time, on a single Tesla V100 32GB GPU and Xeon 6258R 2.7 GHz
CPU.

###  4.2 Hateful Memes Detection

Overall results. For our method, five trials are executed to get different
answers from LL-2. The final prediction is the major vote from five trials. We
report accuracy and AUROC on both seen and unseen test data and compare our
method with 20 baselines. We conduct the experiments in two cases, e.g.,
without OCR text (LL-2) and with OCR text (LL-2+OCR). Table 1 shows the
results.

Our method outperforms 9 out of 11 baselines in terms of accuracy and 8 out of
15 baselines in terms of AUROC when evaluated on the seen test dataset.
However, LL-2 performs less effectively than other baselines on the unseen
dataset, as this dataset exclusively contains multimodally hateful memes
according to [8]. Note that our approach requires no training or fine-tuning.
Although the zero-shot approach still needs to catch up to the state-of-the-
art methods, it is remarkable that this approach shows promising potential for
detecting hateful memes. Especially while being compared to other multimodal
LLMs, our zero-shot approach outperforms all baselines, even those employing
32-shot prompting. This is understandable as OpenFlamingo and Flamingo yield
results using relatively simple instructions [1, 2], underscoring the
effectiveness of our proposed prompt. Noteworthily, providing OCR text results
in a performance improvement, approximately $1.3\%-2\%$ AUROC score gains on
test sets.

Table 1: Detection performance with various models: The first four blocks
include results from [8, 9], and the last block combines [1, 2] with our
experiments, with dashes denoting unavailable results.

Type | Model | Test Seen | Test Unseen
---|---|---|---
Acc. | AUROC | Acc. | AUROC
Unimodal | Image-Grid [5] | 52.00 | 52.63 | - | -
Image-Region [22, 30] | 52.13 | 55.92 | 60.28 | 54.64
Text BERT [4] | 59.20 | 65.08 | 63.60 | 62.65
Multimodal | Late Fusion [8] | 59.66 | 64.75 | 64.06 | 64.44
(Unimodal Pretraining) | Concat BERT [8] | 59.13 | 65.79 | 65.90 | 66.28
MMBT-Grid [7] | 60.06 | 67.92 | 66.85 | 67.24
MMBT-Region [7] | 60.23 | 70.73 | 70.10 | 72.21
ViLBERT [16] | 62.30 | 70.45 | 70.86 | 73.39
Visual BERT [12] | 63.20 | 71.33 | 71.30 | 73.23
Multimodal | ViLBERT CC [16] | 61.10 | 70.03 | 70.03 | 72.78
(Multimodal Pretraining) | Visual BERT COCO [12] | 64.73 | 71.41 | 69.95 | 74.95
Challenge Winner | Ron Zhu [35] | - | - | 73.20 | 84.50
Niklas Muennighof [19] | - | - | 69.50 | 83.10
Team HateDetectron [28] | - | - | 76.50 | 81.08
Team Kingsterdam [13] | - | - | 73.85 | 80.53
Vlad Sandulescu [23] | - | - | 74.30 | 79.43
Multimodal LLM | Fl-9B+OCR 0-shot [1] | - | 57.00 | - | -
Fl-9B+OCR 32-shot [1] | - | 63.50 | - | -
OF-9B+OCR 0-shot [2] | - | 51.60 | - | -
OF-9B+OCR 32-shot [2] | - | 53.80 | - | -
LL-2 0-shot (Our) | 63.00 | 65.77 | 62.15 | 63.92
LL-2+OCR 0-shot (Our) | 62.50 | 67.07 | 64.20 | 64.12

Content extraction. Figure 2 shows two successful and two failed examples from
LL-2 when using the prompt in Section 3.1. Note that we craft these examples
without providing OCR text. The model can accurately describe the objects in
the memes, such as “slice of watermelon", “a dog", “three men" or “a woman".
The information about the characteristics of objects also describes the
context. These examples demonstrate an impressive capability of text
extraction, while the model was only trained on general language-image
instruction-following samples [14].

Reasoning ability. An interesting research question is “how to know the source
of hatefulness on test data? Visual or textual content or both?”. As
aforementioned, explainability is an advantage of VLM compared to deep
learning classifiers. While traditional classifiers only give users predicted
classes, VLM can additionally justify its prediction. Despite successful or
failed attempts, the VLM justifies its prediction by explaining each part of
the memes step-by-step. In the first example of Figure 2(a), the model
surprisingly shows its knowledge about the named person in the text, which
helps determine the implicit hatefulness inside this meme. The second example
further strengthens our observation. The negative meaning of the word
“overdose" is considered, but the model does not classify it as hateful
content since the whole meme talks about food consumption. Even so, in some
cases, the model exposes its weaknesses in Figure 2(b), where it
misunderstands the mocking speech as an encouragement or misses the text in
the first and second examples, respectively.

(a) Successful examples.

(b) Failed examples.

Figure 2: Hateful memes detection using LL-2.

###  4.3 Hateful Meme Correction

Overall results. Like the previous task, we use the proposed prompt to get the
generated text from the LL-2 model. Then, seven human experts independently
reviewed each sample to evaluate whether the VLM could successfully convert
the hateful meme to a non-hateful one. All experts receive the same guidelines
about the evaluation task. To make the evaluation as fair as possible, we
invite experts from various ages, genders, and backgrounds. We get the
majority voting from experts and report the result with each meme. From the
majority voting, there are 46 successful corrections out of 50 selected
hateful memes, i.e., the accuracy is 92%. The accuracy varies from 82% to 92%
if we individually consider the opinion of every expert.

Quality of generated content. In Figure 3, we show examples of both successful
and failed correction with LL-2. In each example, we can observe that the
generated text follows the topic of the original text, which is understandable
since the common reasoning flow is to understand the hateful context and turn
the text into a non-hateful one. For example, in the first example of Figure
3, the VLM can figure out that the use of the word “black" is directly
referring to the man in the visual part, leading to a replacement with
"beautiful" instead of "black" and positive promotion of diversity, which is a
respectful content for social media. Similar patterns can be observable from
other examples about gender and immigration status.

On the other hand, failed examples demonstrate the limitation of VLM in
understanding the hidden meaning. In Figure 3(b), the first meme attacks Shia
men based on their religious beliefs since it mentions the negative stereotype
of Muslims using sensitive terms about sexuality. The VLM attempts to avoid
this hatefulness by changing to the new context about enjoying delicious food.
However, during the reasoning process, it overlooks the Muslim’s food
restriction. The second and third examples then reinforce our observation. Our
exploration, therefore, calls for additional exploration of whether VLM/LLM
could consider causalities to have better reasoning ability.

(a) Successful examples.

(b) Failed examples.

Figure 3: Hateful memes correction using LL-2.

##  5 Conclusion

While hateful meme detection is well-explored, hateful meme correction is
still an open problem. Our work is one of the first to address the hateful
memes correction problem. We introduce a simple but efficient approach for
detecting and correcting hateful memes by prompting a pretrained VLM.
Remarkably, the VLM showcases an impressive ability to extract the content and
do the reasoning for requested tasks with only zero-shot prompting. Our
quantitative evaluation unfolds promising results achieved without the need
for fine-tuning, especially on the hatefulness correction. The use of VLM
offers a user-friendly solution through multimodal AI applications. Hence, our
exploration calls for more research studies on enhancing the performance of
large models on emergent abilities. In the future, we will improve the
performance with in-context learning and chain-of-thought techniques.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Awadalla et al. [2023] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al.  Openflamingo: An open-source framework for training large autoregressive vision-language models.  _arXiv preprint arXiv:2308.01390_ , 2023.
  * Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hegselmann et al. [2023] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag.  Tabllm: Few-shot classification of tabular data with large language models.  In _International Conference on Artificial Intelligence and Statistics_ , pages 5549–5581. PMLR, 2023.
  * Kiela et al. [2019] Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine.  Supervised multimodal bitransformers for classifying images and text.  _arXiv preprint arXiv:1909.02950_ , 2019.
  * Kiela et al. [2020] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine.  The hateful memes challenge: Detecting hate speech in multimodal memes.  _Advances in neural information processing systems_ , 33:2611–2624, 2020.
  * Kiela et al. [2021] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Casey A Fitzpatrick, Peter Bull, Greg Lipstein, Tony Nelli, Ron Zhu, et al.  The hateful memes challenge: Competition report.  In _NeurIPS 2020 Competition and Demonstration Track_ , pages 344–360. PMLR, 2021.
  * Kim et al. [2022] Jiyun Kim, Byounghan Lee, and Kyung-Ah Sohn.  Why is it hate speech? masked rationale prediction for explainable hate speech detection.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 6644–6655, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.  URL https://aclanthology.org/2022.coling-1.577.
  * Li et al. [2023] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao.  Llava-med: Training a large language-and-vision assistant for biomedicine in one day.  _arXiv preprint arXiv:2306.00890_ , 2023.
  * Li et al. [2019] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_ , 2019.
  * Lippe et al. [2020] Phillip Lippe, Nithin Holla, Shantanu Chandra, Santhosh Rajamanickam, Georgios Antoniou, Ekaterina Shutova, and Helen Yannakoudakis.  A multimodal framework for the detection of hateful memes.  _arXiv preprint arXiv:2012.12871_ , 2020.
  * Liu et al. [2023a] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023a.
  * Liu et al. [2023b] Yuliang Liu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al.  On the hidden mystery of ocr in large multimodal models.  _arXiv preprint arXiv:2305.07895_ , 2023b.
  * Lu et al. [2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _arXiv preprint arXiv:1908.02265_ , 2019.
  * Mathew et al. [2021] Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee.  Hatexplain: A benchmark dataset for explainable hate speech detection.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pages 14867–14875, 2021.
  * Mollas et al. [2020] Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas.  Ethos: an online hate speech detection dataset.  _arXiv preprint arXiv:2006.08328_ , 2020.
  * Muennighoff [2020] Niklas Muennighoff.  Vilio: State-of-the-art visio-linguistic models applied to hateful memes.  _arXiv preprint arXiv:2012.07788_ , 2020.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774, 2023.  URL https://api.semanticscholar.org/CorpusID:257532815.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _arXiv preprint arXiv:1506.01497_ , 2015.
  * Sandulescu [2020] Vlad Sandulescu.  Detecting hateful memes using a multimodal deep ensemble.  _arXiv preprint arXiv:2012.13235_ , 2020.
  * Sarzynska-Wawer et al. [2021] Justyna Sarzynska-Wawer, Aleksander Wawer, Aleksandra Pawlak, Julia Szymanowska, Izabela Stefaniak, Michal Jarkiewicz, and Lukasz Okruszek.  Detecting formal thought disorder by deep contextualized word representations.  _Psychiatry Research_ , 304:114135, 2021.
  * Sharma et al. [2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565, 2018.
  * Singhal et al. [2022] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al.  Large language models encode clinical knowledge.  _arXiv preprint arXiv:2212.13138_ , 2022.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Velioglu and Rose [2020] Riza Velioglu and Jewgeni Rose.  Detecting hate speech in memes using multimodal deep learning approaches: Prize-winning solution to hateful memes challenge.  _arXiv preprint arXiv:2012.12975_ , 2020.
  * Wu et al. [2023] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.  Next-gpt: Any-to-any multimodal llm.  _arXiv preprint arXiv:2309.05519_ , 2023.
  * Xie et al. [2017] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1492–1500, 2017.
  * Xu et al. [2022] Depeng Xu, Shuhan Yuan, Yueyang Wang, Angela Uchechukwu Nwude, Lu Zhang, Anna Zajicek, and Xintao Wu.  Coded hate speech detection via contextual information.  In _PAKDD_. Springer, 2022.
  * Zhang et al. [2023] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv preprint arXiv:2303.16199_ , 2023.
  * Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  _arXiv preprint arXiv:2304.10592_ , 2023.
  * Zhu [2020] Ron Zhu.  Enhance multimodal transformer with external label and in-domain pretrain: Hateful meme challenge winning solution.  _arXiv preprint arXiv:2012.08290_ , 2020.

##  Appendix A Baseline Methods

Facebook introduced the Hateful Memes Challenge in late 2020 to not only
provide a reference benchmark dataset to tackle the multimodal hate speech
detection task but also to provide common visio-linguistic baselines
approaches. Indeed, [8] tested and provided 11 vision and language models
through their software MMF 111https://github.com/facebookresearch/mmf. MMF
(MultiModal Framework) is a framework for vision and language multimodal
research with state-of-the-art models. In [1, 2], some preliminary results
show the potential of multimodal LLMs in detecting hateful memes with both
zero-shot and few-shot learning.

###  A.1 Unimodal

Image-Grid. The Image-Grid baseline uses features from ResNet-152’s res-5c
layer after average pooling (layer of 2048 neurons). ResNet-152 is a Deep
Residual Neural Network with 152 layers originally designed for image/scene
classification [5].

Image-Region. The Image-Region baseline uses features from the Faster-RCNN’s
fc6 layer [22] with a ResNeXt-152 [30] backbone (layer of 4096 neurons). The
Fast Region-based Convolutional Neural Network is originally trained on a
Visual Genome, and the resulting fc6 features are fine-tuned using weights of
the fc7 layer.

Text-BERT. The Text BERT baseline is the original BERT model that outputs a
vector of dimension 768. BERT is a bidirectional transformer-based model for
language representation and understanding that learns embeddings for subwords
[4].

###  A.2 Multimodal: unimodal pretraining

Multimodal models from unimodal pretraining typically combine the output or
the features of vision and linguistic models.

Late Fusion. In the Late Fusion baseline, the output is the mean of the
ResNet-152 and BERT output scores.

Concat BERT. The Concat BERT baseline uses the concatenation of the ResNet-152
features with the BERT features, where an MLP is trained on top of it.

MMBT-Grid. The MMBT-Grid baseline is the original supervised multimodal bi-
transformers (MMBT) [7] that uses the features of Image-Grid. MMBT is a
multimodal bi-transformer model that combines BERT as the textual encoder and
ResNet-152 as the image encoder.

MMBT-Region. The MMBT-Grid baseline is the original supervised multimodal bi-
transformers (MMBT) [7] that uses the features of Image-Region.

ViLBERT. The ViLBERT baseline is the unimodally pretrained version of ViLBERT
[16].

Visual BERT. The Visual BERT baseline is the unimodally pretrained version of
Visual BERT [12].

###  A.3 Multimodal: multimodal pretraining

ViLBERT CC. The ViLBERT CC baseline is the multimodally pretrained version
(also the original version) of ViLBERT that was trained on Conceptual Caption
[25] for the tasks of sentence-image alignment, masked language modeling, and
masked visual-feature classification. ViLBERT is a BERT-based model designed
to learn joint contextualized representations of vision and language by using
two separate transformers: one for vision and one for language [16].

Visual BERT COCO. The Visual BERT COCO baseline is the multimodally pretrained
version (also the original version) of Visual BERT that was trained on COCO
Captions [3] for the tasks of sentence-image alignment and masked language
modeling. [ref to COCO]. Like ViLBERT, it is based on BERT. It was originally
designed to learn joint representations of language and visual content from
paired data. Unlike ViLBERT, it uses a single cross-modal transformer to align
elements of the input text and regions in the input image [12].

###  A.4 Challenge Winner

After the Hateful Memes Challenge, many state-of-the-art models were proposed
to deal with the task. [35] extracted and then applied demographics while [19]
utilized Stochastic Weight Averaging method to stabilize the model. In [28],
the authors applied an augmented dataset on original data to train the
VisualBERT model. [13] used a weighted linear combination of ensemble learners
to get the final prediction instead of the majority voting technique. The
multimodal deep ensemble technique is leveraged by [23] to boost the
performance.

###  A.5 Multimodal LLM

Flamingo (Fl). This VLM [1] executes vision-language tasks effectively using
pretrained visual encoders and language models. To connect those modules, the
Perceiver Resampler is trained to project extensive embedding features from
the encoder (NFNet-F6) to visual tokens. Then, both visual and textual tokens
are combined and sent to the language model. The method further modifies the
pretrain LLM by adding gated cross-attention layers and training those layers
from scratch. The cross-attention scheme allows the model to process multiple
image-text inputs, opening the ability to perform in-context learning.

OpenFlamingo (OF). This framework [2] is an attempt to develop an open-source
version of Flamingo and reproduce the results.

LLaVA-Llama-2-13B (LL-2). Pretrained CLIP and Llama-2 serve as the visual
encoder and language model in this VLM [14], respectively. While Flamingo uses
the cross-attention scheme, LLaVA proposes a projection layer to map CLIP
features to language tokens. Then, both visual and textual tokens are used as
inputs to the pretrained language model.