# World Models with Hints of Large Language Models for Goal Achieving

Zeyuan Liu1 &Ziyu Huan211footnotemark: 1 &Xiyao Wang3 &Jiafei Lyu1 &Jian Tao1
&Xiu Li1† &Furong Huang3† &Huazhe Xu4,5,6† 1Tsinghua Shenzhen International
Graduate School, Tsinghua University
2The Ohio State University, 3University of Maryland, College Park
4IIIS, Tsinghua University, 5Shanghai Qi Zhi Institute, 6Shanghai AI Lab
{liuzeyua22, lvjf20<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{xywang<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal contribution.
† Corresponding Authors.

###### Abstract

Reinforcement learning struggles in the face of long-horizon tasks and sparse
goals due to the difficulty in manual reward specification. While existing
methods address this by adding intrinsic rewards, they may fail to provide
meaningful guidance in long-horizon decision-making tasks with large state and
action spaces, lacking purposeful exploration. Inspired by human cognition, we
propose a new multi-modal model-based RL approach named Dreaming with Large
Language Models (DLLM). DLLM integrates the proposed hinting subgoals from the
LLMs into the model rollouts to encourage goal discovery and reaching in
challenging tasks. By assigning higher intrinsic rewards to samples that align
with the hints outlined by the language model during model rollouts, DLLM
guides the agent toward meaningful and efficient exploration. Extensive
experiments demonstrate that the DLLM outperforms recent methods in various
challenging, sparse-reward environments such as HomeGrid, Crafter, and
Minecraft by 27.7%, 21.1%, and 9.9%, respectively.

##  1 Introduction

Reinforcement learning (RL) is effective when the agents receive rewards that
propel them towards desired behaviors [52, 30]. However, the manual
engineering of suitable reward functions presents substantial challenges,
especially in complex environments [62, 14]. Therefore, solving tasks with
long horizons and sparse rewards has long been desired in RL [2, 60].

Existing RL methods address this issue by supplementing the extrinsic rewards
provided by the environment with an intrinsic reward as an auxiliary objective
such as novelty [7, 67, 68], surprise [1], uncertainty [4, 36, 27], and
prediction errors [53, 40, 5]. Nonetheless, there exist scenarios wherein only
a limited set of elements possess inherent factors that are truly valuable to
the agent’s target objective, rendering the exploration of additional aspects
inconsequential or potentially detrimental to the overall system performance
[14, 11, 13]. Some recent researches employ large language models (LLMs) to
explore new solutions for this issue [13, 70, 69]. Leveraging prior knowledge
from extensive corpus data, these methods aim to encourage the exploration of
meaningful states. While these approaches have demonstrated impressive
results, they depend on querying the LLM for any unknown environmental
conditions, which limits their ability to generalize the acquired language
information to other steps. Additionally, due to their model-free nature,
these approaches cannot capture the underlying relationships between dynamics
and language-based hints. They also fail to leverage planning mechanisms or
synthetic data generation to enhance sample efficiency.

Figure 1: The algorithmic overall structure diagram of DLLM, where WM denotes
the world model, $o_{l}$ represents the natural language caption of the
observation, $u$ denotes the transition, and $i_{k}$ corresponds to the
intrinsic reward for the $k$-th goal.

To address this issue, we draw inspiration from how humans solve long-horizon
tasks efficiently. Humans excel at breaking down overall goals into several
sub-goals and strive to plan a reasonable route to accomplish these goals
sequentially [16]. These goals are often associated with specific actions or
environmental dynamics and can ideally be expressed in concise natural
language. For example, experienced Minecraft players can naturally connect the
action “obtaining iron” with its prerequisite actions “find an iron ore” and
“breaking iron ore”.

Consequently, we propose Dreaming with Large Language Model (DLLM), a multi-
modal model-based RL approach that integrates language hints (i.e., goals)
from LLMs into the rollouts to encourage goal discovery and reaching in
challenging and sparse-reward tasks, as illustrated in Figure 1. DLLM’s world
model processes visual inputs and sentence embeddings of natural language
descriptions for transitions and learns to predict both. It then rewards the
agent when the predicted embeddings are close enough to the goal, facilitating
the agents’ use of inductive bias to achieve task goals. Thanks to the power
of prompt-based LLMs, DLLM can influence agents’ behaviors in distinct manners
based on the prompts provided for identical tasks, resulting in multiple
styles of guidance for the agents. For example, when an agent needs to obtain
iron in Minecraft, it can be guided directly to break iron ore, explore more
for a better policy, or try interpolating both strategies.

Empirically, we evaluate DLLM on various sparse-reward environments, including
Homegrid [31], Crafter [19], and Minecraft [17]. Experimental results
demonstrate that DLLM outperforms recent strong methods in task-oriented and
exploration-oriented environments, showcasing robust performance in guiding
exploration and training of the agent within highly complex scenarios. In
Homegrid, Crafter and Minecraft environments, we successfully improve the
performance by 27.7%, 21.1% and 9.9%, respectively, compared to the strongest
baseline Dynalang [31], Achievement Distillation [38], and DreamerV3 [23]. We
also observe that leveraging more powerful language models and providing the
agent with comprehensive language information results in even better
performance.

Our contributions are as follows: (a) we propose DLLM, a multi-modal model-
based reinforcement learning approach that utilizes human natural language to
describe environmental dynamics, and incorporates LLM’s guidance in model
rollouts to improve the agent’s exploration and goal-completion capabilities;
(b) based on goals extracted by LLMs, DLLM can generate meaningful intrinsic
rewards through an automatic descending mechanism to guide policy learning;
(c) experimental results demonstrate that DLLM outperforms recent strong
baselines across diverse environments.

##  2 Background and Related Work

Model-based RL. Model-based RL (MBRL) trains a world model through online
interactions with the environment to predict rewards and next-step states [49,
51, 50]. With the world model, the agent can plan and optimize its policy from
imagined sequences [25, 34]. Amidst recent advancements, specific contemporary
MBRL methods have acquired a world model that is capable of handling high-
dimensional observations and intricate dynamics, achieving notable milestones
in various domains [18, 46, 20, 22, 23, 25]. Akin to our approach, the work of
Lin et al. [31] constructs a multimodal world model capable of predicting
future visual and textual representations, thereby enabling agents to ground
their language generation capabilities within an imagined, simulated
environment. We employ the same implementation approach, but further integrate
the generated natural language from the LLMs during the planning process into
constructing the intrinsic rewards.

Intrinsically motivated RL. In a sparse-reward environment, agents must take
many steps in a decision sequence before receiving a positive reward signal.
Collecting practical data for training using only random sampling or noisy RL
methods is challenging, especially in complex environments with a large state-
action space [44, 64]. Intrinsically motivated RL is the primary method for
addressing sparse reward problems. It provides extra intrinsic dense rewards
to the agent, encouraging the agent to explore unvisited areas. Pathak et al.
[40] propose using curiosity as an intrinsic reward signal, which measures the
agent’s proficiency in predicting the consequences of its actions within the
latent feature space generated by a self-supervised inverse dynamics model.
Burda et al. propose random network distillation (RND) [6] that leverages the
prediction error of a fixed random neural network on novel states and achieves
outstanding results in Montezuma’s Revenge. Some subsequent studies improve
RND via methods such as distributional modeling [63]. In addition to the
intrinsically motivated method that utilizes state novelty, there are methods
for maximizing the diversity of states [55, 32, 59] and for maximizing the
diversity of skills mastered by the agent [3, 10].

Despite the success of intrinsically motivated RL methods, they may face
challenges when dealing with large state-action space and complex task
scenarios since they are only encouraged to explore novel states, and not all
states are useful for achieving the goal. Purposeless exploration can hinder
the performance of the agent. Hence, it is essential to incorporate meaningful
encouragement to assist the agent, as highlighted in previous studies [14,
13]. This may involve integrating commonsense knowledge, furnishing explicit
subgoals as guides, and employing other relevant strategies to facilitate the
agent’s learning process. DLLM considers these factors with a specific focus
on long-term decision-making. During rollouts, DLLM applies intrinsic rewards
when the agent achieves goals set by LLM in previous steps, strengthening the
understanding of the agent’s contextual connections.

Leveraging large language models (LLMs) for language goals. Pre-trained LLMs
showcase remarkable capabilities, particularly in understanding common human
knowledge. Naturally, LLMs can generate meaningful and human-recognizable
intrinsic rewards for intelligent agents. Choi et al. [9] leverage pre-trained
LLMs as task-specific priors for managing text-based metadata within the
context of supervised learning. Kant et al. [28] utilize LLMs as commonsense
priors for zero-shot planning. Similar efforts are made by Yao et al., Shinn
et al., Wu et al., and Wang et al., [65, 48, 61, 57], who propose diverse
prompt methods and algorithmic structures to mitigate the problems of
hallucination and inaccuracy when employing LLMs directly for decision-making.
Carta et al. [8] examine an approach where an agent utilizes an LLM as a
policy that undergoes progressive updates as the agent engages with the
environment, employing online reinforcement learning to enhance its
performance in achieving objectives. Zhang et al. [66] propose to leverage the
LLMs to guide skill chaining. Du et al. propose ELLM [13], which leverages
LLMs to generate intrinsic rewards for guiding agents, integrating LLM with
RL. However, the guidance obtained using this approach is only effective in
the short term. DLLM draws inspiration from ELLM and endeavors to extend the
guidance from LLMs into long-term decision-making.

##  3 Preliminaries

We consider a partially observable Markov decision process (POMDP) defined by
a tuple $(S,A,O,\Omega,P,\gamma,R)$, where $s\in S$ represents the states of
the environment, $a\in A$ represents the actions, and observation $o\in\Omega$
is obtained from $O(o|s,a)$. $P(s^{\prime}|s,a)$ represents the dynamics of
the environment, $R$ and $\gamma$ are the reward function and discount factor,
respectively. During training, the agent’s goal is to learn a policy $\pi$
that maximizes discounted cumulative rewards, i.e.,
$\max\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t})\right].$

Additionally, we define two sets of natural language sentence embeddings: the
set of sentence embeddings for transitions, denoted as $U$, and the set of
sentence embeddings for goals, denoted as $G$. In this context, each $u\in U$
represents a sentence embedding describing the environmental changes from the
previous step to the current step, while $g\in G$ represents a sentence
embedding of a goal the LLM intends for the agent to achieve. We permit the
LLM to output any content within specified formats, thereby enlarging the
support of the goal distribution to encompass the space of natural language
strings. Thus, $G$ should encompass all possible $u\in U$, i.e., $U\subset G$.

We also define the goal-conditioned intrinsic reward function
$R_{\text{int}}(u|g)$, and the DLLM agent optimizes for an intrinsic reward
$R_{\rm int}$ alongside the reward $R$ from the environment. Assuming that
goals provided by natural language are diverse, common-sense sensitive, and
context-sensitive, we expect that maximizing $R_{\text{int}}$ alongside $R$
ensures that the agent maximizes the general reward function $R$ without
getting stuck in local optima.

##  4 Dreaming with LLMs

This section systematically introduces how DLLM obtains guiding information
(goals) from LLMs and utilizes them to incentivize the agent to manage long-
term decision-making.

###  4.1 Goal Generation by Prompting LLMs

To generate the natural language representations of goals and their
corresponding vector embeddings, DLLM utilizes a pretrained SentenceBert model
[43] and GPT [39]. For GPT, we use two versions including GPT-3.5-turbo-0315
and GPT-4-32k-0315, which we will refer to as GPT-3.5 and GPT-4 respectively
in the following text.

We initially obtain the natural language representation, denoted as $o_{l}$
($l$ denotes language, and $o_{l}$ means natural language description of $o$),
corresponding to the information in the agent’s current observation $o$. This
$o_{l}$ may include details such as the agent’s position, inventory, health
status, and field of view. We use an observation captioner to obtain the
$o_{l}$ following ELLM [13] (see Appendix C for more details of captioners).
Subsequently, we provide $o_{l}$ and other possible language output from the
environment (e.g., the task description in HomeGrid) and the description of
environmental mechanisms to LLMs to get a fixed number of goals $g^{l}_{1:K}$
in the form of natural language, where $K$ is a hyperparameter representing
the expected number of goals returned by the LLM. We utilize SentenceBert to
convert these goals from natural language into vector embeddings $g_{1:K}$.
For different environments, we utilize two specific approaches to obtain
goals: 1) having the LLM generate responses for $K$ arbitrary types of goals
and 2) instructing the LLM to provide a goal for $K$ specified types (e.g.,
determining which room to enter and specifying the corresponding action). The
second approach is designed to standardize responses from the LLM and ensure
that the goals output by the LLM cover all necessary aspects for task
completion in complex scenarios.

###  4.2 Incorporating Decreased Intrinsic Rewards into Dreaming Processes

At each online interaction step, we have a transition captioner that gives a
language description $u_{l}$ of the dynamics between the observation and the
next observation; the language description $u_{l}$ is then embedded into a
vector embedding $u$. Given the sensory representation $x_{0}$, language
description embedding of transition $u_{0}$, embeddings of goals $g_{1:K}$,
and intrinsic rewards for each goal $i_{1:K}$ of replay inputs, the world
model and actor produce a sequence of imagined latent states $\hat{s}_{1:T}$,
actions $\hat{a}_{1:T}$, rewards $\hat{r}_{1:T}$, transitions $\hat{u}_{1:T}$
and continuation flags $\hat{c}_{1:T}$, where $T$ represents the total length
of model rollouts. We use cosine similarity to measure the matching score $w$
between transitions and goals:

$\displaystyle w\left(\hat{u}\mid g\right)=\begin{cases}\frac{\hat{u}\cdot
g}{\left\|\hat{u}\right\|\|g\|}&\text{if }\frac{\hat{u}\cdot
g}{\left\|\hat{u}\right\|\|g\|}>M\\\ 0&\text{ otherwise }\end{cases},$ (1)

where $M$ is a similarity threshold hyperparameter. In this step, we aim to
disregard low cosine similarities to some extent, thereby preventing
misleading guidance. Moreover, within a sequence, a goal may be triggered
multiple times. We aim to avoid assigning intrinsic rewards to the same goal
multiple times during a single rollout process, as it could lead the agent to
perform simple actions repeatedly and eventually diminish the exploration of
more complex behaviors. Hence, we only retain a specific goal’s matching score
when it first exceeds $M$ in the sequence. The method to calculate the
intrinsic reward for step $t$ in one model rollout is written as:

$\displaystyle r_{t}^{\text{int}}=\alpha\cdot\sum_{k=1}^{K}w_{t}^{k}\cdot
i_{k}\cdot\begin{cases}1&\text{if}\ \ t_{k}^{\prime}\ \ \text{exists}\ \
\text{and}\ \ t=t_{k}^{\prime}\\\ 0&\text{otherwise}\end{cases}$ (2)

where $\alpha$ is the hyperparameter that controls the scale of the intrinsic
rewards, $t^{\prime}_{k}$ represents the time step $t$ when the $w^{k}_{t}$
first exceeds $M$ within the range of $1$ to $T$.

Then, we give the method to calculate and decrease $i_{1:K}$. If each goal’s
reward is constant, the agent will tend to repeat learned skills instead of
exploring new ones. We use the novelty measure RND [6] to generate and reduce
the intrinsic rewards from LLMs, which effectively mitigates the issue of
repetitive completion of simple tasks. To be more specific, after sampling a
batch from the replay buffer, we extract the sentence embeddings of the goals
from them: $g_{1:B,1:L,1:K}$, where $B$ is the batch size, and $L$ is the
batch length. Given the target network $f:G\rightarrow{\mathbb{R}}$ and the
predictor neural network $\hat{f_{\theta}}:G\rightarrow{\mathbb{R}}$, we
calculate the prediction error:

$\displaystyle e_{1:B,1:L,1:K}=\|\hat{f}_{\theta}(g)-f(g)\|^{2}.$ (3)

Subsequently, we update the predictor neural network and the running estimates
of reward standard deviation, then standardize the intrinsic reward:

$\displaystyle i_{1:B,1:L,1:K}=(e_{1:B,1:L,1:K}-m)/{\sigma},$ (4)

where $m$ and $\sigma$ stand for the running estimates of the mean and
standard deviation of the intrinsic returns.

###  4.3 World Model and Actor Critic Learning

We implement the world model with Recurrent State-Space Model (RSSM) [21],
with an encoder that maps sensory inputs $x_{t}$ (e.g., image frame or
language) and $u_{t}$ to stochastic representations $z_{t}$. Afterward,
$z_{t}$ is combined with past action $a_{t}$ and recurrent state $h_{t}$ and
fed into a sequence model, denoted as “seq”, to predict $\hat{z}_{t+1}$:

$\displaystyle\hat{z}_{t},h_{t}=\operatorname{seq}\left(z_{t-1},h_{t-1},a_{t-1}\right),$
(5) $\displaystyle
z_{t}\sim\operatorname{encoder}\left(x_{t},u_{t},h_{t}\right),$ (6)
$\displaystyle\hat{x}_{t},\hat{u}_{t},\hat{r}_{t},\hat{c}_{t}=\operatorname{decoder}\left(z_{t},h_{t}\right),$
(7)

where $\hat{z_{t}}$, $\hat{x}_{t}$, $\hat{u}_{t}$, $\hat{r}_{t}$,
$\hat{c}_{t}$ denotes the world model prediction for the stochastic
representation, sensory representation, transition, reward, and the episode
continuation flag. The encoder and decoder employ convolutional neural
networks (CNN) for image inputs and multi-layer perceptrons (MLP) for other
low-dimensional inputs. After obtaining multi-modal representations from the
decoder and sequence model, we employ the following objective to train the
entire world model in an end-to-end manner:

$\displaystyle\mathcal{L}_{\rm
total}={\mathcal{L}_{x}}+\mathcal{L}_{u}+\mathcal{L}_{r}+\mathcal{L}_{c}+{\beta_{1}}\mathcal{L}_{\rm
pred}+{\beta_{2}}\mathcal{L}_{\rm reg},$ (8)

in which $\beta_{1}=0.5$, $\beta_{2}=0.1$, and all sub loss term are written
as:

Sensation Loss:
$\displaystyle\mathcal{L}_{x}=\left\|\hat{x}_{t}-x_{t}\right\|_{2}^{2},$ (9)
Transition Loss:
$\displaystyle\mathcal{L}_{u}=\operatorname{catxent}\left(\hat{u}_{t},{u}_{t}\right),$
Reward Loss:
$\displaystyle\mathcal{L}_{r}=\operatorname{catxent}\left(\hat{r}_{t},\operatorname{twohot}\left(r_{t}\right)\right),$
Continue Loss:
$\displaystyle\mathcal{L}_{c}=\operatorname{binxent}\left(\hat{c}_{t},c_{t}\right),$
Prediction Loss:
$\displaystyle\mathcal{L}_{\text{pred}}=\max\left(1,\operatorname{KL}\left[\operatorname{sg}\left(z_{t}\right)\|\hat{z}_{t}\right]\right),$
Regularizer:
$\displaystyle\mathcal{L}_{\text{reg}}=\max\left(1,\operatorname{KL}\left[z_{t}\|\operatorname{sg}\left(\hat{z}_{t}\right)\right]\right),$

where $\rm catxent$ is the categorical cross-entropy loss, $\rm binxent$ is
the binary cross-entropy loss, $\rm sg$ is the stop gradient operator, KL
refers to the Kullback-Leibler (KL) divergence. Details of the
$\text{twohot}(\cdot)$ can be found in Appendix B.1.

We adopt the widely used actor-critic architecture for learning policies,
where the actor executes actions and collects samples in the environment while
the critic evaluates whether the executed action is good. We denote the model
state as $s_{t}=concat(z_{t},h_{t})$. The actor and the critic give:

Actor: $\displaystyle\pi_{\theta}(a_{t}\mid s_{t}),$ Critic: $\displaystyle
V_{\psi}(s_{t}).$ (10)

Note that both the actor network and the critic network are simple MLPs. The
actor aims to maximize the cumulative returns with the involvement of
intrinsic reward, i.e.,

$\displaystyle
R_{t}\doteq\sum_{\tau=0}^{\infty}\gamma^{\tau}(r_{t+\tau}+r^{\rm
int}_{t+\tau}).$ (11)

The intrinsic rewards beyond the prediction horizon $T$ are inaccessible, and
we set them to zero. The details of bootstrapped $\lambda$-returns [54]. Then
the bootstrapped $\lambda$-returns [54] could be written as:

$\displaystyle R_{t}^{\lambda}\doteq r_{t}+\gamma
c_{t}\left((1-\lambda)V_{\psi}\left(s_{t+1}\right)+\lambda
R_{t+1}^{\lambda}\right),$ $\displaystyle\quad R_{T}^{\lambda}\doteq
V_{\psi}\left(s_{T}\right).$ (12)

The actor and the critic are updated via the following losses:

$\displaystyle\mathcal{L}_{V}=\operatorname{catxent}\left(V_{\psi}(s_{t}),\operatorname{sg}\left(\operatorname{twohot}\left(R_{t}\right)\right)\right),$
(13)
$\displaystyle\mathcal{L}_{\pi}=-\frac{\operatorname{sg}\left(R_{t}-V(s_{t})\right)}{\max(1,S)}\log\pi_{\theta}\left(a_{t}\mid
s_{t}\right)-\eta\mathrm{H}\left[\pi_{\theta}\left(a_{t}\mid
s_{t}\right)\right].$

where $S$ is the exponential moving average between the 5th and 95th
percentile of $R_{t}$. We summarize the detailed pseudocode for DLLM in
Appendix B.2.

##  5 Experiments

The primary goal of our experiments is to substantiate the following claim:
DLLM helps the agent by leveraging the guidance from the LLM during the
dreaming process, thereby achieving improved performance in tasks.
Specifically, our experiments test the following hypotheses:

  * •

(H1) Through proper prompting, DLLM can comprehend complex environments and
generate accurate instructions to assist intelligent agents in multi-task
environments.

  * •

(H2) DLLM can leverage the generative capabilities of LLMs to obtain
reasonable and novel hints, aiding agents in exploration within challenging
environments.

  * •

(H3) DLLM can significantly accelerate the exploration and training of agents
in highly complex, large-scale, near-real environments that necessitate
rational high-dimensional planning.

  * •

(H4) DLLM can be more powerful when leveraging stronger LLMs or receiving
additional language information.

Baselines. Since we include natural language information in our experiments,
we consider employing ELLM [13] and Dynalang [31] as baselines.111For ELLM and
Dynalang, we utilize their official implementations for experimentation. For
ELLM, we prompt the LLM and obtain goals following the same procedure as in
our method. All language information, including the goals obtained from the
LLMs, is encoded into sentence embeddings to feed Dynalang. We also compare
against other recent strong baseline algorithms that do not utilize natural
language in each environment.

Environments. We conduct experiments on three environments: HomeGrid [31],
Crafter [19], and Minecraft based on MineRL [17]. These environments span
first-person to third-person perspectives, 2D or 3D views, and various types
and levels of task complexity.

Captioner and Language Encodings. Within each environment, we deploy an
observation captioner and a transition captioner to caption observations and
transitions, respectively. Transition captions are stored in the replay buffer
for the agent’s predictive learning, while observation captions provide
pertinent information for LLM. For language encoding, we employ SentenceBert
_all-MiniLM-L6-v2_ [58] to convert all natural language inputs into
embeddings.

The Quality of Generated Goals. To measure the quality of goals generated by
LLMs during online interaction in the environment, we selected the following
metrics: novelty, correctness, context sensitivity, and common-sense
sensitivity. See the detailed explanations and experiments in Appendix D.

Cache. As each query to the LLM consists of an object-receptacle combination,
we implement a cache for each experiment to efficiently reuse queries, thereby
reducing both time and monetary cost.

###  5.1 HomeGrid

Environment description. HomeGrid is a multi-task reinforcement learning
environment structured as a grid world, where agents get partial pixel
observations and language hints (e.g., the descriptions of tasks). We reduce
the map size to 10$\times$10 to expedite the overall training of the agent and
incorporate icon signals to indicate actions for opening bins. For each step,
we have the captioners to caption the observation and transition. The other
components of the environment remain unchanged. More details can be seen in
Appendix A.1.1.

To support our claims, we design various settings where the environment
provides different levels of information along with distinct language hints
for each, as outlined in Table 1, to help address hypothesis H4.

Table 1: Description of different environment settings Setting |  Description
---|---
Standard |  The environment provides task descriptions in natural language form.
Key info |  The environment additionally provides task-relevant objects’ location and status information.
Full info |  The environment additionally provides the location and status information of all objects on the map. For bins, the correct opening actions will also be instructed.
Oracle |  The agent will always receive accurate instructions.

Query prompts, LLM choices, and Goals Generated. Each query prompt consists of
the caption of the agent’s current observation and a request for the LLM to
generate a goal for each of the two types: “where to go” and “what to do”,
respectively. For full prompts and examples, please see Appendix A.1.2. We
select GPT-4 as the base LLM for all experiments in HomeGrid. Queries to the
GPT are made every ten steps. We test the quality of the goals generated in
the Appendix D.1.

Figure 2: HomeGrid experiments results. Curves averaged over 5 seeds with
shading representing one-eighth of the standard deviation.

Performance. The overall results are depicted in Figure 2. The baseline
algorithm ELLM fails in HomeGrid, likely because it struggles to comprehend
the sentence embeddings required to describe the task. Our method outperforms
baseline algorithms utilizing the same information in the standard setting,
showing strong evidence for H1 and H3. Moreover, in the Key info, Full info,
and Oracle settings, DLLM demonstrates enhanced performance with increasing
information. In the overall context of reduced error prompts in the Full info,
DLLM consistently demonstrates a more pronounced advantage throughout the
training period. The results of the Full info and Oracle settings show no
significant difference. These findings support hypothesis H4.

###  5.2 Crafter

Environment description. The Crafter environment is a grid world that features
top-down graphics and discrete action space. Crafter is designed similarly to
a 2D Minecraft, featuring a procedurally generated, partially observable world
where players can collect or craft a variety of artifacts. In Crafter, the
player’s goal is to unlock the entire achievement tree, which consists of 22
achievements. As the map is designed with entities capable of harming the
player (e.g., zombies, skeletons), the player must also create weapons or
place barriers to ensure survival.

Extra baselines. We compared three additional types of baselines that do not
utilize language information: (1) LLM-based solutions: SPRING [61], Reflexion
[48], ReAct [65], standalone GPT-4 (step-by-step instructions), (2) model-
based RL baseline: DreamerV3 [23], (3) model-free methods: Achievement
Distillation [38], PPO [47], Rainbow [26]. We also add human experts [19] and
random policy as additional references.

Query prompts, LLM choices, and Goals Generated. Each query prompt contains
the caption of the agent’s current observation description and a request to
have the LLM generate five goals. In this portion, we conduct evaluations
using two popular LLMs, GPT-3.5 and GPT-4. Through these assessments, we
explore whether a more robust LLM contributes to enhanced agent performance,
addressing hypothesis H4. Queries to the GPT are made every ten steps. We test
the quality of the goals generated in the Appendix D.2.

Method | Score | Reward | Steps
---|---|---|---
DLLM (w/ GPT-4) |  38.1$\pm$1.2 | 15.4$\pm$1.1 | 5M
DLLM (w/ GPT-3.5) |  37.6$\pm$1.6 | 14.5$\pm$1.5 | 5M
AdaRefiner (w/ GPT-4) | 28.2$\pm$1.8 | 12.9$\pm$1.2 | 5M
AdaRefiner (w/ GPT-3.5) | 23.4$\pm$2.2 | 11.8$\pm$1.7 | 5M
ELLM | - | 6.0$\pm$0.4 | 5M
DLLM (w/ GPT-4) |  26.4$\pm$1.3 | 12.4$\pm$1.3 | 1M
DLLM (w/ GPT-3.5) |  24.4$\pm$1.8 | 12.2$\pm$1.6 | 1M
Achievement Distillation | 21.8$\pm$1.4 | 12.6$\pm$0.3 | 1M
Dynalang | 16.4$\pm$1.7 | 11.5$\pm$1.4 | 1M
AdaRefiner (w/ GPT-4) | 15.8$\pm$1.4 | 12.3$\pm$1.3 | 1M
PPO (ResNet) | 15.6$\pm$1.6 | 10.3$\pm$0.5 | 1M
DreamerV3 | 14.5$\pm$1.6 | 11.7$\pm$1.9 | 1M
PPO | 4.6$\pm$0.3 | 4.2$\pm$1.2 | 1M
Rainbow | 4.3$\pm$0.2 | 5.0$\pm$1.3 | 1M
SPRING (w/ GPT-4) | 27.3$\pm$1.2 | 12.3$\pm$0.7 | -
Reflexion (w/ GPT-4) | 12.8$\pm$1.0 | 10.3$\pm$1.3 | -
ReAct (w/ GPT-4) | 8.3$\pm$1.2 | 7.4$\pm$0.9 | -
Vanilla GPT-4 | 3.4$\pm$1.5 | 2.5$\pm$1.6 | -
Human Experts | 50.5$\pm$6.8 | 14.3$\pm$2.3 | -
Random | 1.6$\pm$0.0 | 2.1$\pm$1.3 | -

Table 2: The results indicate that DLLM with GPT-4 and GPT-3.5 outperforms
baseline algorithms, achieving superiority at 1M and 5M training steps.

Performance. DLLM outperforms all baseline algorithms at 1M and 5M steps. As
shown in Figure LABEL:fig:crafter_sub_spr and Table 2, DLLM exhibits a
significant advantage compared to baselines. Figure
LABEL:fig:crafter_sub_successes shows that DLLM is good at medium to high
difficulty tasks like “make stone pickaxe/sword” and “collect iron” while
maintaining stable performance in less challenging tasks. When the steps reach
5M, the performance of DLLM significantly surpasses the language-based
algorithm SPRING. These findings show strong evidence for hypotheses H2 and
H3. In all experiments of Crafter, DLLM (w/ GPT-4) demonstrates a more robust
performance than DLLM (w/ GPT-3.5), indicating that DLLM can indeed achieve
better performance with the assistance of a more powerful LLM. This finding
aligns with the results presented in Appendix D.2, where GPT-4 consistently
identifies exploration-beneficial goals, thus confirming hypothesis H4. In
Crafter, We also include ablation studies on the scale of intrinsic rewards in
Appendix E.2, not decreasing intrinsic rewards in Appendix E.3, utilizing
random goals in Appendix E.4, and allowing repeated intrinsic rewards in
Appendix E.5.

###  5.3 Minecraft

Environment description. Several RL environments, e.g., MineRL [17], have been
constructed based on Minecraft, a popular video game that features a randomly
initialized open world with diverse biomes. Minecraft Diamond is a challenging
task based on MineRL, with the primary objective of acquiring a diamond.
Progressing through the game involves the player collecting resources to craft
new items, ensuring his survival, unlocking the technological progress tree,
and ultimately achieving the goal of obtaining a diamond within 36000 steps.
In Minecraft Diamond, we also have captioners to provide the captions of the
observation and transition in natural language form at each step. The
environment settings completely mirror those outlined in DreamerV3 [23], which
includes awarding a +1 reward for each milestone achieved, which encompasses
collecting or crafting a log, plank, stick, crafting table, wooden pickaxe,
cobblestone, stone pickaxe, iron ore, furnace, iron ingot, iron pickaxe, and
diamond.

Method | Reward
---|---
DLLM (w/ GPT-4) |  10.0$\pm$0.3
DreamerV3 | 9.1$\pm$0.3
Dynalang | 8.9$\pm$0.4
IMPALA | 7.4$\pm$0.2
Rainbow | 6.3$\pm$0.3
R2D2 | 5.0$\pm$0.5
PPO | 4.1$\pm$0.2
ELLM | 0.3$\pm$0.0

Table 3: Comparison between DLLM (w/ GPT-4) and baselines in Minecraft at
100M. DLLM (w/ GPT-4) surpasses all baselines, including those that also
involve LLMs or natural languages in policy learning.

Figure 4: The episode returns in Minecraft Diamond. The curves indicate that
DLLM enjoys a consistent advantage throughout the entire learning process,
thanks to its utilization of an LLM for exploration and training. All
algorithms undergo experiments using 5 different seeds.

Extra baselines. To fully compare DLLM with current popular methods from
model-based algorithms to model-free algorithms on Minecraft, we include
DreamerV3 [23], IMPALA [15], R2D2 [33], Rainbow [26] and PPO [47] as our extra
baselines, along with ELLM [13] and Dynalang [31].

Query prompts, LLM choices, and Goals Generated. During each query to the GPT,
we provide it with information about the player’s status, inventory, and
equipment and request the GPT to generate five goals. We choose GPT-4 as our
language model for the DLLM experiment. Please see Appendix A.3.1 for specific
details. We make a query to the GPT every twenty steps. We also test the
quality of the generated goals in Appendix D.3.

Performance. In Figure 4 and Table 3, we present empirical results in
Minecraft Diamond. Baseline algorithm ELLM struggles in this complex
environment, possibly due to high task complexity. DLLM demonstrates higher
data efficiency in the early training stages, facilitating quicker acquisition
of basic skills within fewer training steps compared to baseline methods. DLLM
also maintains a significant advantage in later stages, indicating its ability
to still derive reasonable and practical guidance from the LLM during the
post-exploration training process. These findings underscore the effectiveness
of DLLM in guiding exploration and training in highly complex environments
with the support of the LLM, providing compelling evidence for hypothesis H3.

##  6 Conclusion and Discussion

We propose DLLM, a multi-modal model-based RL method that leverages the
guidance from LLMs to provide hints (goals) and generate intrinsic rewards in
model rollouts. DLLM outperforms recent strong baselines in multiple
challenging tasks with sparse rewards. Our experiments demonstrate that DLLM
effectively utilizes language information from the environment and LLMs, and
enhances its performance by improving language information quality.

Limitations. DLLM relies on the guidance provided by a large language model,
making it susceptible to the inherent instability of LLM outputs. This
introduces a potential risk to the stability of DLLM’s performance, even
though the prompts used in our experiments contributed to relatively stable
model outputs. Unreasonable goals may encourage the agent to make erroneous
attempts, and correcting such misguided behavior may take time. We expect to
address these challenges in future work.

## References

  * [1] Joshua Achiam and Shankar Sastry.  Surprise-based intrinsic motivation for deep reinforcement learning.  arXiv preprint arXiv:1703.01732, 2017.
  * [2] Hui Bai, Ran Cheng, and Yaochu Jin.  Evolutionary reinforcement learning: A survey.  Intelligent Computing, 2, January 2023.
  * [3] Adrien Baranes and Pierre-Yves Oudeyer.  Active learning of inverse models with intrinsically motivated goal exploration in robots.  Robotics and Autonomous Systems, 61(1):49–73, 2013.
  * [4] Marc G Bellemare, Will Dabney, and Rémi Munos.  A distributional perspective on reinforcement learning.  In International conference on machine learning, pages 449–458. PMLR, 2017.
  * [5] Yuri Burda, Harrison Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros.  Large-scale study of curiosity-driven learning.  In Seventh International Conference on Learning Representations, pages 1–17, 2019.
  * [6] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.  Exploration by random network distillation.  In International Conference on Learning Representations, 2018.
  * [7] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.  Exploration by random network distillation.  In Seventh International Conference on Learning Representations, pages 1–17, 2019.
  * [8] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer.  Grounding large language models in interactive environments with online reinforcement learning.  arXiv preprint arXiv:2302.02662, 2023.
  * [9] Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon.  Lmpriors: Pre-trained language models as task-specific priors.  In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022\.
  * [10] Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Yves Oudeyer.  Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey.  Journal of Artificial Intelligence Research, 74:1159–1199, 2022\.
  * [11] Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer.  Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms.  In International conference on machine learning, pages 1039–1048. PMLR, 2018.
  * [12] Rati Devidze, Parameswaran Kamalaruban, and Adish Singla.  Exploration-guided reward shaping for reinforcement learning under sparse rewards.  Advances in Neural Information Processing Systems, 35:5829–5842, 2022.
  * [13] Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas.  Guiding pretraining in reinforcement learning with large language models.  arXiv preprint arXiv:2302.06692, 2023.
  * [14] Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Tom Griffiths, and Alexei Efros.  Investigating human priors for playing video games.  In International Conference on Machine Learning, pages 1349–1357. PMLR, 2018.
  * [15] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al.  Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.  In International conference on machine learning, pages 1407–1416. PMLR, 2018.
  * [16] Tharindu Fernando, Simon Denman, Sridha Sridharan, and Clinton Fookes.  Learning temporal strategic relationships using generative adversarial imitation learning, 2018.
  * [17] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov.  Minerl: A large-scale dataset of minecraft demonstrations.  arXiv preprint arXiv:1907.13440, 2019.
  * [18] David Ha and Jürgen Schmidhuber.  World models.  arXiv preprint arXiv:1803.10122, 2018.
  * [19] Danijar Hafner.  Benchmarking the spectrum of agent capabilities, 2022.
  * [20] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.  Dream to control: Learning behaviors by latent imagination.  arXiv preprint arXiv:1912.01603, 2019.
  * [21] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels.  In International conference on machine learning, pages 2555–2565. PMLR, 2019.
  * [22] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba.  Mastering atari with discrete world models.  arXiv preprint arXiv:2010.02193, 2020.
  * [23] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.  Mastering diverse domains through world models, 2023.
  * [24] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.  Mastering diverse domains through world models, 2024.
  * [25] Nicklas Hansen, Xiaolong Wang, and Hao Su.  Temporal difference learning for model predictive control.  arXiv preprint arXiv:2203.04955, 2022.
  * [26] Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.  Rainbow: Combining improvements in deep reinforcement learning, 2017.
  * [27] David Janz, Jiri Hron, Przemysław Mazur, Katja Hofmann, José Miguel Hernández-Lobato, and Sebastian Tschiatschek.  Successor uncertainties: exploration and uncertainty in temporal difference learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [28] Yash Kant, Arun Ramachandran, Sriram Yenamandra, Igor Gilitschenski, Dhruv Batra, Andrew Szot, and Harsh Agrawal.  Housekeep: Tidying virtual households using commonsense reasoning.  In European Conference on Computer Vision, pages 355–373. Springer, 2022.
  * [29] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling.  Improved variational inference with inverse autoregressive flow.  Advances in neural information processing systems, 29, 2016.
  * [30] Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh.  Exploration in deep reinforcement learning: A survey.  Information Fusion, 85:1–22, 2022.
  * [31] Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan.  Learning to model the world with language.  arXiv preprint arXiv:2308.01399, 2023.
  * [32] Cam Linke, Nadia M Ady, Martha White, Thomas Degris, and Adam White.  Adapting behavior via intrinsic reward: A survey and empirical study.  Journal of artificial intelligence research, 69:1287–1332, 2020\.
  * [33] Qinghua Liu, Tim A Rand, Savitha Kalidas, Fenghe Du, Hyun-Eui Kim, Dean P Smith, and Xiaodong Wang.  R2d2, a bridge between the initiation and effector steps of the drosophila rnai pathway.  Science, 301(5641):1921–1925, 2003.
  * [34] Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch.  Plan online, learn offline: Efficient learning and exploration via model-based control.  arXiv preprint arXiv:1811.01848, 2018.
  * [35] Ziyan Luo, Yijie Zhang, and Zhaoyue Wang.  Does hierarchical reinforcement learning outperform standard reinforcement learning in goal-oriented environments?  In NeurIPS 2023 Workshop on Goal-Conditioned Reinforcement Learning, 2023.
  * [36] TM Moerland, DJ Broekens, and CM Jonker.  Efficient exploration with double uncertain value networks.  In Deep Reinforcement Learning Symposium, NIPS 2017, 2017.
  * [37] Ron Mokady, Amir Hertz, and Amit H Bermano.  Clipcap: Clip prefix for image captioning.  arXiv preprint arXiv:2111.09734, 2021.
  * [38] Seungyong Moon, Junyoung Yeom, Bumsoo Park, and Hyun Oh Song.  Discovering hierarchical achievements in reinforcement learning via contrastive learning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [39] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022\.
  * [40] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell.  Curiosity-driven exploration by self-supervised prediction.  In International conference on machine learning, pages 2778–2787. PMLR, 2017.
  * [41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [42] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [43] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084, 2019.
  * [44] Stuart Ian Reynolds.  Reinforcement learning with exploration.  PhD thesis, University of Birmingham, 2002.
  * [45] Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg.  Learning by playing solving sparse reward tasks from scratch.  In International conference on machine learning, pages 4344–4353. PMLR, 2018.
  * [46] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al.  Mastering atari, go, chess and shogi by planning with a learned model.  Nature, 588(7839):604–609, 2020.
  * [47] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [48] Noah Shinn, Beck Labash, and Ashwin Gopinath.  Reflexion: an autonomous agent with dynamic memory and self-reflection.  arXiv preprint arXiv:2303.11366, 2023.
  * [49] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  nature, 529(7587):484–489, 2016.
  * [50] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science, 362(6419):1140–1144, 2018.
  * [51] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  nature, 550(7676):354–359, 2017.
  * [52] David Silver, Satinder Singh, Doina Precup, and Richard S Sutton.  Reward is enough.  Artificial Intelligence, 299:103535, 2021.
  * [53] Bradly C Stadie, Sergey Levine, and Pieter Abbeel.  Incentivizing exploration in reinforcement learning with deep predictive models.  arXiv preprint arXiv:1507.00814, 2015.
  * [54] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [55] Teck-Hou Teng and Ah-Hwee Tan.  Knowledge-based exploration for reinforcement learning in self-organizing neural networks.  In 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology, volume 2, pages 332–339. IEEE, 2012.
  * [56] Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher.  Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards.  Advances in Neural Information Processing Systems, 32, 2019.
  * [57] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models.  arXiv preprint arXiv:2305.16291, 2023.
  * [58] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  Advances in Neural Information Processing Systems, 33:5776–5788, 2020.
  * [59] Xiyao Wang, Ruijie Zheng, Yanchao Sun, Ruonan Jia, Wichayaporn Wongkamjan, Huazhe Xu, and Furong Huang.  Coplanner: Plan to roll out conservatively but to explore optimistically for model-based rl, 2023.
  * [60] Lisheng Wu and Ke Chen.  Goal exploration augmentation via pre-trained skills for sparse-reward long-horizon goal-conditioned reinforcement learning, 2023.
  * [61] Yue Wu, So Yeon Min, Shrimai Prabhumoye, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, and Yuanzhi Li.  Spring: Gpt-4 out-performs rl algorithms by studying papers and reasoning.  arXiv preprint arXiv:2305.15486, 2023.
  * [62] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu.  Text2reward: Automated dense reward function generation for reinforcement learning.  arXiv preprint arXiv:2309.11489, 2023.
  * [63] Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li.  Exploration and anti-exploration with distributional random network distillation.  arXiv preprint arXiv:2401.09750, 2024.
  * [64] Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Jianye Hao, Zhaopeng Meng, Peng Liu, and Zhen Wang.  Exploration in deep reinforcement learning: a comprehensive survey.  arXiv preprint arXiv:2109.06668, 2021.
  * [65] Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models.  In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022\.
  * [66] Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J Lim.  Bootstrap your own skills: Learning to solve new tasks with large language model guidance.  arXiv preprint arXiv:2310.10021, 2023.
  * [67] Jingwei Zhang, Niklas Wetzel, Nicolai Dorka, Joschka Boedecker, and Wolfram Burgard.  Scheduled intrinsic drive: A hierarchical take on intrinsically motivated exploration.  arXiv preprint arXiv:1903.07400, 2019.
  * [68] Tianjun Zhang, Paria Rashidinejad, Jiantao Jiao, Yuandong Tian, Joseph E Gonzalez, and Stuart Russell.  Made: Exploration via maximizing deviation from explored regions.  Advances in Neural Information Processing Systems, 34:9663–9680, 2021.
  * [69] Wanpeng Zhang and Zongqing Lu.  Adarefiner: Refining decisions of language models with adaptive feedback, 2023.
  * [70] Zihao Zhou, Bin Hu, Pu Zhang, Chenyang Zhao, and Bin Liu.  Large language model is a good policy teacher for training reinforcement learning agents.  arXiv preprint arXiv:2311.13373, 2023.

## References

  * [1] Joshua Achiam and Shankar Sastry.  Surprise-based intrinsic motivation for deep reinforcement learning.  arXiv preprint arXiv:1703.01732, 2017.
  * [2] Hui Bai, Ran Cheng, and Yaochu Jin.  Evolutionary reinforcement learning: A survey.  Intelligent Computing, 2, January 2023.
  * [3] Adrien Baranes and Pierre-Yves Oudeyer.  Active learning of inverse models with intrinsically motivated goal exploration in robots.  Robotics and Autonomous Systems, 61(1):49–73, 2013.
  * [4] Marc G Bellemare, Will Dabney, and Rémi Munos.  A distributional perspective on reinforcement learning.  In International conference on machine learning, pages 449–458. PMLR, 2017.
  * [5] Yuri Burda, Harrison Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros.  Large-scale study of curiosity-driven learning.  In Seventh International Conference on Learning Representations, pages 1–17, 2019.
  * [6] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.  Exploration by random network distillation.  In International Conference on Learning Representations, 2018.
  * [7] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.  Exploration by random network distillation.  In Seventh International Conference on Learning Representations, pages 1–17, 2019.
  * [8] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer.  Grounding large language models in interactive environments with online reinforcement learning.  arXiv preprint arXiv:2302.02662, 2023.
  * [9] Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon.  Lmpriors: Pre-trained language models as task-specific priors.  In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022\.
  * [10] Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Yves Oudeyer.  Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey.  Journal of Artificial Intelligence Research, 74:1159–1199, 2022\.
  * [11] Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer.  Gep-pg: Decoupling exploration and exploitation in deep reinforcement learning algorithms.  In International conference on machine learning, pages 1039–1048. PMLR, 2018.
  * [12] Rati Devidze, Parameswaran Kamalaruban, and Adish Singla.  Exploration-guided reward shaping for reinforcement learning under sparse rewards.  Advances in Neural Information Processing Systems, 35:5829–5842, 2022.
  * [13] Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas.  Guiding pretraining in reinforcement learning with large language models.  arXiv preprint arXiv:2302.06692, 2023.
  * [14] Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Tom Griffiths, and Alexei Efros.  Investigating human priors for playing video games.  In International Conference on Machine Learning, pages 1349–1357. PMLR, 2018.
  * [15] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al.  Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.  In International conference on machine learning, pages 1407–1416. PMLR, 2018.
  * [16] Tharindu Fernando, Simon Denman, Sridha Sridharan, and Clinton Fookes.  Learning temporal strategic relationships using generative adversarial imitation learning, 2018.
  * [17] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov.  Minerl: A large-scale dataset of minecraft demonstrations.  arXiv preprint arXiv:1907.13440, 2019.
  * [18] David Ha and Jürgen Schmidhuber.  World models.  arXiv preprint arXiv:1803.10122, 2018.
  * [19] Danijar Hafner.  Benchmarking the spectrum of agent capabilities, 2022.
  * [20] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi.  Dream to control: Learning behaviors by latent imagination.  arXiv preprint arXiv:1912.01603, 2019.
  * [21] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels.  In International conference on machine learning, pages 2555–2565. PMLR, 2019.
  * [22] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba.  Mastering atari with discrete world models.  arXiv preprint arXiv:2010.02193, 2020.
  * [23] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.  Mastering diverse domains through world models, 2023.
  * [24] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.  Mastering diverse domains through world models, 2024.
  * [25] Nicklas Hansen, Xiaolong Wang, and Hao Su.  Temporal difference learning for model predictive control.  arXiv preprint arXiv:2203.04955, 2022.
  * [26] Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.  Rainbow: Combining improvements in deep reinforcement learning, 2017.
  * [27] David Janz, Jiri Hron, Przemysław Mazur, Katja Hofmann, José Miguel Hernández-Lobato, and Sebastian Tschiatschek.  Successor uncertainties: exploration and uncertainty in temporal difference learning.  Advances in Neural Information Processing Systems, 32, 2019.
  * [28] Yash Kant, Arun Ramachandran, Sriram Yenamandra, Igor Gilitschenski, Dhruv Batra, Andrew Szot, and Harsh Agrawal.  Housekeep: Tidying virtual households using commonsense reasoning.  In European Conference on Computer Vision, pages 355–373. Springer, 2022.
  * [29] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling.  Improved variational inference with inverse autoregressive flow.  Advances in neural information processing systems, 29, 2016.
  * [30] Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh.  Exploration in deep reinforcement learning: A survey.  Information Fusion, 85:1–22, 2022.
  * [31] Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan.  Learning to model the world with language.  arXiv preprint arXiv:2308.01399, 2023.
  * [32] Cam Linke, Nadia M Ady, Martha White, Thomas Degris, and Adam White.  Adapting behavior via intrinsic reward: A survey and empirical study.  Journal of artificial intelligence research, 69:1287–1332, 2020\.
  * [33] Qinghua Liu, Tim A Rand, Savitha Kalidas, Fenghe Du, Hyun-Eui Kim, Dean P Smith, and Xiaodong Wang.  R2d2, a bridge between the initiation and effector steps of the drosophila rnai pathway.  Science, 301(5641):1921–1925, 2003.
  * [34] Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch.  Plan online, learn offline: Efficient learning and exploration via model-based control.  arXiv preprint arXiv:1811.01848, 2018.
  * [35] Ziyan Luo, Yijie Zhang, and Zhaoyue Wang.  Does hierarchical reinforcement learning outperform standard reinforcement learning in goal-oriented environments?  In NeurIPS 2023 Workshop on Goal-Conditioned Reinforcement Learning, 2023.
  * [36] TM Moerland, DJ Broekens, and CM Jonker.  Efficient exploration with double uncertain value networks.  In Deep Reinforcement Learning Symposium, NIPS 2017, 2017.
  * [37] Ron Mokady, Amir Hertz, and Amit H Bermano.  Clipcap: Clip prefix for image captioning.  arXiv preprint arXiv:2111.09734, 2021.
  * [38] Seungyong Moon, Junyoung Yeom, Bumsoo Park, and Hyun Oh Song.  Discovering hierarchical achievements in reinforcement learning via contrastive learning.  Advances in Neural Information Processing Systems, 36, 2024.
  * [39] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022\.
  * [40] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell.  Curiosity-driven exploration by self-supervised prediction.  In International conference on machine learning, pages 2778–2787. PMLR, 2017.
  * [41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [42] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 1(8):9, 2019.
  * [43] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084, 2019.
  * [44] Stuart Ian Reynolds.  Reinforcement learning with exploration.  PhD thesis, University of Birmingham, 2002.
  * [45] Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg.  Learning by playing solving sparse reward tasks from scratch.  In International conference on machine learning, pages 4344–4353. PMLR, 2018.
  * [46] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al.  Mastering atari, go, chess and shogi by planning with a learned model.  Nature, 588(7839):604–609, 2020.
  * [47] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [48] Noah Shinn, Beck Labash, and Ashwin Gopinath.  Reflexion: an autonomous agent with dynamic memory and self-reflection.  arXiv preprint arXiv:2303.11366, 2023.
  * [49] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  nature, 529(7587):484–489, 2016.
  * [50] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science, 362(6419):1140–1144, 2018.
  * [51] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  nature, 550(7676):354–359, 2017.
  * [52] David Silver, Satinder Singh, Doina Precup, and Richard S Sutton.  Reward is enough.  Artificial Intelligence, 299:103535, 2021.
  * [53] Bradly C Stadie, Sergey Levine, and Pieter Abbeel.  Incentivizing exploration in reinforcement learning with deep predictive models.  arXiv preprint arXiv:1507.00814, 2015.
  * [54] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [55] Teck-Hou Teng and Ah-Hwee Tan.  Knowledge-based exploration for reinforcement learning in self-organizing neural networks.  In 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology, volume 2, pages 332–339. IEEE, 2012.
  * [56] Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher.  Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards.  Advances in Neural Information Processing Systems, 32, 2019.
  * [57] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models.  arXiv preprint arXiv:2305.16291, 2023.
  * [58] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.  Advances in Neural Information Processing Systems, 33:5776–5788, 2020.
  * [59] Xiyao Wang, Ruijie Zheng, Yanchao Sun, Ruonan Jia, Wichayaporn Wongkamjan, Huazhe Xu, and Furong Huang.  Coplanner: Plan to roll out conservatively but to explore optimistically for model-based rl, 2023.
  * [60] Lisheng Wu and Ke Chen.  Goal exploration augmentation via pre-trained skills for sparse-reward long-horizon goal-conditioned reinforcement learning, 2023.
  * [61] Yue Wu, So Yeon Min, Shrimai Prabhumoye, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, and Yuanzhi Li.  Spring: Gpt-4 out-performs rl algorithms by studying papers and reasoning.  arXiv preprint arXiv:2305.15486, 2023.
  * [62] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu.  Text2reward: Automated dense reward function generation for reinforcement learning.  arXiv preprint arXiv:2309.11489, 2023.
  * [63] Kai Yang, Jian Tao, Jiafei Lyu, and Xiu Li.  Exploration and anti-exploration with distributional random network distillation.  arXiv preprint arXiv:2401.09750, 2024.
  * [64] Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Jianye Hao, Zhaopeng Meng, Peng Liu, and Zhen Wang.  Exploration in deep reinforcement learning: a comprehensive survey.  arXiv preprint arXiv:2109.06668, 2021.
  * [65] Shunyu Yao, Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models.  In NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022\.
  * [66] Jesse Zhang, Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph J Lim.  Bootstrap your own skills: Learning to solve new tasks with large language model guidance.  arXiv preprint arXiv:2310.10021, 2023.
  * [67] Jingwei Zhang, Niklas Wetzel, Nicolai Dorka, Joschka Boedecker, and Wolfram Burgard.  Scheduled intrinsic drive: A hierarchical take on intrinsically motivated exploration.  arXiv preprint arXiv:1903.07400, 2019.
  * [68] Tianjun Zhang, Paria Rashidinejad, Jiantao Jiao, Yuandong Tian, Joseph E Gonzalez, and Stuart Russell.  Made: Exploration via maximizing deviation from explored regions.  Advances in Neural Information Processing Systems, 34:9663–9680, 2021.
  * [69] Wanpeng Zhang and Zongqing Lu.  Adarefiner: Refining decisions of language models with adaptive feedback, 2023.
  * [70] Zihao Zhou, Bin Hu, Pu Zhang, Chenyang Zhao, and Bin Liu.  Large language model is a good policy teacher for training reinforcement learning agents.  arXiv preprint arXiv:2311.13373, 2023.

##  Appendix A Environment Details

###  A.1 HomeGrid

####  A.1.1 Details of Environmental Adjustments

“HomeGrid” is introduced by Dynalang [31], and our modified version is based
on the “homegrid-task” setting. Aside from the pixel observation, this setting
additionally provides language information describing the task assigned to the
robotic agent. The original map of HomeGrid is a large 14x12 grid, as shown in
Figure LABEL:fig:homegrid_original, and training on such a map would require
an excessively long time. We have reduced the map size to a simplified version
of 10x10 as in Figure LABEL:fig:homegrid_mini. In this smaller map, rooms have
become more compact, but the width of passages between rooms remains
unchanged. In addition to resizing the map, we have adjusted the refresh range
for both the player and items, ensuring that players can always move and items
can always be accessed. HomeGrid does not provide any visual signal when the
robot takes the actions, including “pedal”, “lift”, and “grasp”, representing
the different actions to open the bins, so the trained transition captioner
needs additional information in the pixel observation. We add icons222All
assets of the icons are collected from https://fontawesome.com/. for each of
the three actions and make them appear when the related action is taken and
the robot succeeds in opening any bin, as shown in Figure
LABEL:fig:homegrid_action. Furthermore, there have been no alterations to
HomeGrid’s task assignments, reward mechanisms, or total step count.

####  A.1.2 Full Prompt Details

During each query to the LLM, we provide the agent with a concise overview of
the fundamental aspects of the HomeGrid environment. The observation captioner
interprets the current observational state of the environment into natural
language, and we provide this to the LLM. We then direct the LLM to choose one
goal for “what to do” and another for “where to go.” In order to ensure
consistency in agent responses, we have incorporated mandatory statements and
provided illustrative examples. GPT’s performance can fluctuate, manifesting
as inconsistent quality in generated outputs at different times of the day. We
recommend capitalizing all the warning text. This can help alleviate the
issue.

The actual input provided to the LLM is divided into two parts: system
information and game information. The part of system information is:

You are engaged in a game resembling AI2-THOR. You will receive details about
your task, interactive items in view, carried items, and your current room.
State the goals you wish to achieve from now on. Please select one thing to do
and one room to go, and return them to me, with the format including:

go to the [room],
[action] the [object],
[action] to [change the status of] (e.g., open) the [bin],
[action] the [object] in/to the [bin/room].

Commas should separate goals and should not contain any additional characters.

An example is:
get the bottle, go to the kitchen.

The format for game information is as follows:

Your task is [text],
You see [objects],
Your carrying is [object],
[Extra information based on the setting of standard, key info, and full info].

###  A.2 Crafter

Crafter [19] serves as a platform for reinforcement learning research drawing
inspiration from Minecraft, featuring a 2D world where players engage in
various survival activities. This game simplifies and optimizes familiar
mechanics to enhance research productivity. Players explore a broad world
comprising diverse terrains like forests, lakes, mountains, and caves. The
game challenges players to maintain health, food, and water, with consequences
for neglecting these essentials. The interaction with various creatures, which
vary in behavior based on the time of day, adds to the game’s complexity.

####  A.2.1 Full Prompt Details

During each query to the LLM, we start by presenting the framework of the
Crafter environment, employing Minecraft as an analogy. Subsequently, we
furnish the current observation information to the LLM, encompassing
objects/creatures within the player’s field of view, the details of the
player’s inventory, and the player’s status.

In Crafter, we also divide the prompt for the LLM into two sections: system
information and game information. The system information is as follows:

As a professional game analyst, you oversee an RL agent or a player in a game
resembling Minecraft. You will receive a starting point that includes
information about what the player sees, what the player has in his inventory,
and the player’s status. For this starting point, please provide the top 5 key
goals the player should achieve in the next several steps to maximize its game
exploration.
Consider the feasibility of each action in the current state and its
importance to achieving the achievement. The response should only include
valid actions separated by ’,’. Do not include any other letters, symbols, or
words.
An example is:
collect wood, place table, collect stone, attack cow, attack zombie.

The format for game information is as follows:

The player sees [objects/creatures],
The player has [objects],
The status of the player is [text].

###  A.3 Minecraft

Minecraft Diamond [23] is an innovative environment developed on top of MineRL
[17], gaining significant attention in the research community within the
expansive universe of Minecraft. Minecraft offers a procedurally generated 3D
world with diverse biomes, such as forests, deserts, and mountains, all
composed of one-meter blocks for player interaction. The primary challenge in
this environment is the pursuit of diamonds, a rare and valuable resource
found deep underground [35]. This quest tests players’ abilities to navigate
and survive in the diverse Minecraft world, requiring progression through a
complex technology tree. Players interact with various creatures, gather
resources, and craft items from over 379 recipes, ensuring their survival by
managing food and safety.

Developers have meticulously addressed gameplay nuances identified through
extensive human playtesting in the Minecraft Diamond environment. Key
improvements include modifying the episode termination criteria based on
player death or a fixed number of steps and refining the jump mechanism to
enhance player interaction and strategy development. The environment, built on
MineRL v0.4.415 and Minecraft version 1.11.2, offers a more consistent and
engaging experience. The reward system is thoughtfully structured, encouraging
players to reach 12 significant milestones culminating in acquiring a diamond.
This system, while straightforward, requires strategic planning and resource
management, as each item provides a reward only once per episode. The
environment’s sensory inputs and action space are comprehensive and immersive,
offering players a first-person view and a wide range of actions, from
movement to crafting.

####  A.3.1 Full Prompt Details

In Minecraft, we also split the LLM prompt into system and game info sections.
The system information is as follows:

As a professional game analyst, you oversee an RL agent or a player in
Minecraft, and your final goal is to collect a diamond. You will receive a
starting point that includes information about what the player sees, what the
player has in his inventory, and the player’s status. For this starting point,
please provide the top 5 key goals the player should achieve in the next
several steps to achieve his final goal.

Take note of the game mechanics in Minecraft; you need to progressively
accomplish goals. Each goal should be in the form of an action with an item
after it. Please do not add any extra numbers or words.

An example is:
pick up log, attack creepers, drop cobblestone, craft wooden pickaxe, craft
arrows.

An example of game information is as follows:

You have [objects]
You have equipped [objects]
The status of you is [text].

##  Appendix B Additional Details of DLLM

###  B.1 Two-hot Reward Prediction

We adopt the DreamerV3 approach for reward prediction, utilizing a softmax
classifier with exponentially spaced bins. This classifier is employed to
regress the two-hot encoding of real-valued rewards, ensuring that the
gradient scale remains independent of the arbitrary scale of the rewards.
Additionally, we apply a regularizer with a cap at one free nat [29] to avoid
over-regularization, a phenomenon known as posterior collapse.

###  B.2 Pseudo Code

Algorithm 1 Dreaming with Large Language Models (DLLM)

while acting do

Observe in the environment
$r_{t},c_{t},x_{t},u_{t},o^{l}_{t}\leftarrow\operatorname{env}\left(a_{t-1}\right)$.

Acquire goals
$g^{t}_{1:K}\leftarrow\text{embed}(\operatorname{LLM}\left(o^{l}_{t}\right))$.

Encode observations
$z_{t}\sim\operatorname{enc}\left(x_{t},u_{t},h_{t}\right)$.

Execute action $a_{t}\sim\pi\left(a_{t}\mid h_{t},z_{t}\right)$.

Add $\left(r_{t},c_{t},x_{t},u_{t},a_{t},g^{t}_{1:K}\right)$ to replay buffer.

end while

while training do

Draw batch
$\left\\{\left(r_{t},c_{t},x_{t},u_{t},a_{t},g^{t}_{1:K}\right)\right\\}$ from
replay buffer.

Calculate intrinsic rewards $i_{1:K}$ for each goal using the RND method and
update the RND network.

Use world model to compute representations ${z}_{t}$, future predictions
$\hat{z}_{t+1}$, and decode $\hat{x}_{t},\hat{u}_{t},\hat{r}_{t},\hat{c}_{t}$.

Update world model to minimize $\mathcal{L}_{\rm total}$.

Imagine rollouts from all $z_{t}$ using $\pi$.

Calculate match scores $w$ and the intrinsic reward $r^{\rm int}$ for each
step.

Update actor to minimize $\mathcal{L}_{\pi}$.

Update critic to minimize $\mathcal{L}_{V}$.

end while

##  Appendix C Details of Captioners

For the implementation of the captioners, DLLM generally follows ELLM [13],
except that we use trained transition captioners throughout all our
experiments to get the language description of the dynamics between two
observations. We split the captions into two different parts: semantic parts
and dynamic parts.

###  C.1 Hard-coded Captioner for Semantic Parts

The captioner of semantic parts follows the hard-coded captioner
implementation outlined in Appendix I of ELLM [13]. The overall semantic
captions include the following categories:

  * •

Field of view. In the grid world environments (HomeGrid and Crafter), we
collect the text description of all the interactable objects in the agent’s
view, regardless of the object’s quantity, to form the caption for this
section. Similarly, in the Minecraft environment, we obtain the list of all
visible objects from the simulator’s semantic sensor.

  * •

Inventory. For HomeGrid, this will only include the item the robot carries.
For Crafter and Minecraft, we convert each inventory item to the corresponding
text descriptor. For Minecraft, we get this information directly from
interpreting the observation.

  * •

Health Status. In Crafter and Minecraft, if any health statuses are below the
maximum, we convert each to a corresponding language description (e.g., we say
the agent is “hungry” if the hunger status is less than 9). There is no such
information in HomeGrid, so we do not provide related captions. Note that the
observation directly gives related information in Minecraft, so we simply
translate them into natural language.

###  C.2 Trained Transition Captioner for Dynamics Parts

The captioner for transitions (dynamics parts) is designed to translate the
dynamics between two adjacent observations into natural language form. For
convenience, we modify the original simulator to generate language labels for
the training of the transition captioner. All language labels use a
predetermined and fixed format established by humans. These language labels
succinctly describe the dynamics of the environment in the most
straightforward manner possible. Notably, these human-designed labels aid the
agent in utilizing a similar approach to describe the environment dynamics
with concise key words. The designs of all possible formats of language
descriptions for transitions in each environment are as follows:

  * •

HomeGrid.

    * –

go to the [room].

    * –

[action] the [object]. (e.g., pick up the plates)

    * –

[action] to [change the status of] (e.g., open) the [bin].

    * –

[action] the [object] in/to the [bin/room].

  * •

Crafter.

    * –

[action] (e.g., sleep, wake up)

    * –

[action] the [item/object]. (e.g., attack the zombie)

  * •

Minecraft.

    * –

[action] (e.g., forward, jump, sneak)

    * –

[action] the [object]. (e.g., craft the torch)

The training process of the captioner mainly follows the methodology outlined
in Appendix J of ELLM [13]; we similarly apply a modified ClipCap algorithm
[37] to datasets of trajectories generated by trained agents, with details
provided in Table 4. Specifically, we embed the visual observations at
timestep $t$ and $t+1$ with a pre-trained and frozen CLIP ViT-B-32 model [41];
the embedding is then concatenated together with the difference in semantic
embeddings between the corresponding states. Semantic embeddings encompass the
inventory and a multi-hot embedding of the set of objects/creatures present in
the agent’s local view. The concatenated representation of the transition is
then mapped through a learned mapping function to a sequence of 32 tokens. We
use these tokens as a prefix and decode them with a trained and frozen GPT-2
to generate the caption [42].

Table 4: The algorithm used to generate samples, total steps, and scale of the generated dataset for each environment are as follows. We capture one sample every 1K steps during training. Environment | Algorithm | Steps | Scale
---|---|---|---
HomeGrid | Dynalang | 10M | 10K
Crafter | Achievement Distillation | 1M | 1K
Minecraft | DreamerV3 | 100M | 100K

We employ a reward confusion matrix in Figure 7 to illustrate the accuracy of
our trained transition captioner on HomeGrid, depicting the probability of
each achieved goal being correctly rewarded or incorrectly rewarded for
another goal during real interactions with the environment. Despite being
based on a limited dataset, the captioner demonstrates strong accuracy even
when extrapolated beyond the dataset distribution.

Figure 7: The reward confusion matrix of the trained transition captioner on
HomeGrid. Each square’s color indicates the probability that the action in the
row will be rewarded with the achievement labels on the column. For example,
if all action “go to the dining room” is recognized as the achievement “go to
the dining room”, we will receive a 100% on the square corresponding to this
row and column. The total in each row does not equal 100% because multiple
rewards may be activated by a single achievement, depending on its
description.

##  Appendix D Metrics to Test the Quality of Goals Generated by LLMs and Goal
Analysis.

Despite the superiority of GPT-3.5 and GPT-4, they may still output
impractical or unachievable goals within the game mechanics. This section of
ablation experiments primarily investigates the quality of guidance provided
by different versions of LLMs in all the environments in which DLLM was
conducted. A detailed explanation of the metrics for measuring the generated
goals’ quality is shown in Table 5.

Table 5: Explanation of the metrics. Metrics |  Explanation
---|---
Novelty |  In exploratory environments like Crafter and Minecraft, a goal is “novel” when its prerequisites are fulfilled, but the goal remains unaccomplished in the current episode. For example, in Crafter, the goal “place table” is “novel” when it is unfulfilled when there are sufficient resources available to achieve it.
Correctness |  In task-oriented environments like HomeGrid, each situation has finite correct answers for goals, and other goals may be useless or even lead to task failure. A goal is considered “correct” if it is one of the correct answers. For example, “go to the kitchen” in HomeGrid is correct if the task is to “find the papers” and the papers are located in the kitchen.
Context sensitivity |  A goal is “context sensitivity” when the player’s current field of view and inventory satisfy all the conditions necessary for this goal, regardless of whether these goals are right or novel. e.g., “make wood pickaxe” when you have enough resources but already have a wood pickaxe in your inventory, and you see a table.
Common-sense sensitivity |  A goal is “common-sense sensitive” when it is feasible in the environment in at least one situation. A counterexample is “make path” in Crafter, which is impossible. Sometimes, the LLM may not fully understand a previously unknown environment (such as HomeGrid) through concise descriptions, leading to such situations.

###  D.1 HomeGrid

In HomeGrid, given the current observation (including the information about
the task and the world state), there is a unique correct answer for “where to
go” and “what to do”. To assess the quality of the generated goals, we
conducted tests as shown in Table 6. In this task-oriented environment, we do
not test the novelty of goals. The statistical results for each setting were
obtained using 1M training samples generated from real interactions. The
correctness of goals provided in the standard setting is low since the agent’s
observation may lack relevant information. There is a noticeable improvement
in the Key info setting and extra improvement in the Full info setting. Note
that in the Oracle setting, the goals provided to the agent are always
correct, so we do not include this setting.

Table 6: Testing the quality of goals provided by LLM in each setting of HomeGrid. Ideally, the goals should exhibit high correctness, low context insensitivity, and low commonsense insensitivity. Setting | Standard | Key info | Full info
---|---|---|---
Correctness | 52.55% | 61.64% | 66.92%
Context insensitivity | 24.30% | 18.34% | 18.95%
Common-sense insensitivity | 3.45% | 4.17% | 5.63%

###  D.2 Crafter

Given the exploratory nature of the environment, it is hard to say if a goal
is “correct” or not. Therefore, in evaluating Crafter’s goal quality,
assessing its correctness holds minimal significance. Instead, our evaluation
approach prioritizes novelty over correctness. Through testing various
scenarios, the results presented in Table 7 indicate that GPT-3.5 tends to
offer practical suggestions, demonstrating a context-sensitive ratio of up to
79.41%. Conversely, GPT-4 leans towards proposing more radical and innovative
recommendations, prioritizing novelty. Notably, a goal can exhibit both
novelty and context sensitivity concurrently. Therefore, the proportions of
“context insensitivity” and “common-sense insensitivity” in the table are
acceptable. Despite GPT-4 showing higher ratios in both context insensitivity
and common-sense insensitivity, experimental results underscore its
exceptional assistance in enhancing performance. Statistical results for each
choice of LLMs were derived from 1M training samples generated from real
interactions, with scripts devised to assess these samples without humans in
the loop.

Table 7: Testing the quality of goals provided by GPT-3.5 and GPT-4 in Crafter. LLM | Novelty | Context insensitivity | Common-sense insensitivity
---|---|---|---
GPT-3.5 | 17.44% | 20.59% | 8.26%
GPT-4 | 38.15% | 38.80% | 10.78%

###  D.3 Minecraft

Despite Minecraft’s relative complexity, GPT possesses a wealth of pretrained
knowledge about it due to the abundance of relevant information in its
training data. Similar to Crafter, correctness is not the primary focus in
Minecraft. During the training process of the DLLM, we randomly sampled 1024
steps to collect an equal number of observations, resulting in 5120 goals
(1024 multiplied by 5) aligned with the observations. Due to the complexity of
elements encompassed within Minecraft, writing scripts to label the quality of
goals proves exceedingly challenging. In light of this, we opted for a manual
annotation process. This involved a detailed examination of each goal using
human labeling. The results are presented in Table 8.

Table 8: Testing the quality of goals provided by GPT-4 in Minecraft. Novelty | Context insensitivity | Common-sense insensitivity
---|---|---
73.63% | 7.66% | 0.53%

##  Appendix E Additional Ablation Studies

###  E.1 Token vs Sentence Embedding for Dynalang in HomeGrid

This ablation study compares the performance difference of the Dynalang
baseline when utilizing token or sentence embedding to acquire natural
language information about the task. The results are shown in Figure 8, and we
do not observe significant differences between the two methods. Dynalang with
token embedding does not outperform Dynalang with sentence embedding. We
believe this is because, in our modified environment, Dynalang retrieves task
information using tokens and cannot immediately access the complete task
information compared to Dynalang using sentence embedding. This is because
Dynalang is configured to display only one token per step, requiring time
equal to the number of tokens to display all tokens in a sentence.

We do not attempt a similar experiment for natural language information
related to transitions and goals because each step in the environment may
generate a transition and several goals, and it is impractical to transmit
numerous transition tokens token by token to the agent.

Figure 8: Token vs. sentence embedding performance for Dynalang, averaged
across 5 seeds. Dynalang employs a token-by-token approach by tokenizing
natural language and passing it into the environment token by token. In
contrast, DLLM exclusively utilizes a sentence embedding implementation, as it
helps compress a substantial amount of information into a single time step.
Within the natural language information we use, task-related language can be
separated and still follow Dynalang’s token-by-token format. In the HomeGrid
environment, we have not observed significant differences.

###  E.2 Ablations of the Intrinsic Reward Scale in Crafter

In our work, a Random Network Distillation (RND) network is employed to
progressively reduce the intrinsic reward corresponding to each goal. We
conduct an ablation experiment to illustrate the necessity of this measure. We
set the hyperparameter $\alpha\in${0.5, 2} and perform experiments for each
value. $\alpha$ = 2 resulted in catastrophic outcomes, whereas $\alpha$ = 0.5
only led to a slight performance decrease. We conclude that excessively large
intrinsic rewards tend to mislead the agent, e.g., try to obtain intrinsic
rewards instead of environmental rewards. Conversely, excessively small
intrinsic rewards result in inadequate guidance the DLLM provides, undermining
its effectiveness in directing the agent’s behavior. Please refer to Figures
LABEL:fig:Aplot and LABEL:fig:Aspr for the results.

###  E.3 Decrease or not to decrease intrinsic rewards in Crafter

This ablation study aims to demonstrate our claim in the paper that repeatedly
providing the agent with a constant intrinsic reward for each goal will result
in the agent consistently performing simple tasks [45, 56, 12], thereby
reducing its exploration efficiency and the likelihood of acquiring new
skills. We still use an RND network to provide intrinsic rewards in this
experiment. However, by preventing the RND network from updating throughout
the training process, we ensure that the intrinsic rewards corresponding to
all goals remain constant and do not decrease over time. We observe a slight
increase in performance during the earlier stages and a significant decline in
the later stages, which is consistent with our claim. Please refer to Figures
LABEL:fig:norndspr and LABEL:fig:norndplot for the results.

###  E.4 Random Goals in Crafter

In this ablation study, we investigate the effectiveness of guidance from the
LLM using its pre-trained knowledge compared to randomly sampled goals. In
this experiment, we instruct the LLM to sample goals without providing any
information about the agent, resulting in entirely random goal sampling.
However, we still require the LLM to adhere to the format specified in
AppendixA.2.1. The results are presented in Figure LABEL:fig:randomgoalspr and
LABEL:fig:randomgoalplot. We find that using random goals significantly
reduces the performance of DLLM. Nonetheless, DLLM still maintains a certain
advantage over recent popular algorithms like Dynalang. This is because
providing basic information about the environment to the LLM still generates
some reasonable goals in uncertain player conditions. These goals continue to
provide effective guidance for the agent through the intrinsic rewards
generated in model rollouts.

###  E.5 Allow Repetition in Crafter

In Method, we assert that when rewarding the same goal repeatedly within a
single model rollout, there is a risk that the agent may tend to repetitively
trigger simpler goals instead of attempting to unlock unexplored parts of the
technology tree. Consequently, this may lead to decreased performance within
Crafter environments primarily focused on exploration. This viewpoint aligns
with ELLM [13]. Here, we conducted experiments to substantiate this claim,
with results presented in Figure LABEL:fig:notonepertrajspr and
LABEL:fig:notonepertrajplot. We observed a significant performance decline in
DLLM when repetitive rewards for the same goal were allowed.

##  Appendix F Additional results in Crafter

Figure 10 presents the comparison of success rates on the total 22
achievements between DLLM and other baselines in Crafter at 5M steps. DLLM
exhibits a higher success rate in unlocking fundamental achievements and
outperforms other baselines.

Figure 10: Logarithmic scale success rates for unlocking 22 distinct
achievements at 5M steps.

##  Appendix G Implementation Details

For all the experiments, We employ the default hyperparameters for the XL
DreamerV3 model [24]. Other hyperparameters are specified below. A uniform
learning rate of 3e-4 is applied across all environments for the RND networks.
Regarding the scale for intrinsic reward $\alpha$, we consistently set
$\alpha$ to be 1. We use 1 Nvidia A100 GPU for each single experiment. The
training time includes the total GPT querying time, which should be near zero
when reusing a cache to obtain the goals.

Table 9: Hyperparameters and training information for DLLM. | HomeGrid | Crafter | Minecraft
---|---|---|---
Imagination horizon $T$ | 15 | 15 | 15
Language MLP layers | 5 | 5 | 5
Language MLP units | 1024 | 1024 | 1024
Image Size | (64, 64, 3) | (64, 64, 3) | (64, 64, 3)
Train ratio | 32 | 512 | 32
Batch size | 16 | 16 | 16
Batch length | 256 | 64 | 64
GRU recurrent units | 4096 | 4096 | 8192
Learning rate for RND | 3e-4 | 3e-4 | 3e-4
The scale for intrinsic rewards $\alpha$ | 1.0 | 1.0 | 1.0
Similarity threshold $M$ | 0.5 | 0.5 | 0.5
Max goal numbers $K$ | 2 | 5 | 5
Env steps | 10M | 5M | 100M
Number of envs | 66 | 1 | 64
Training Time (GPU days) | 2.25 | 10.75 | 16.50
Total GPT querying Time (days) | 1.50 | 0.75 | 7.50
temperature of GPT | 0.5
top_p of GPT | 1.0
max_tokens of GPT | 500
CPU device | AMD EPYC 7452 32-Core Processor
CUDA device | Nvidia A100 GPU
RAM | 256G

##  Appendix H Licenses

In our code, we have used the following libraries covered by the corresponding
licenses:

  * •

HomeGrid, with MIT license

  * •

Crafter, with MIT license

  * •

Minecraft, with Attribution-NonCommercial-ShareAlike 4.0 International

  * •

OpenAI GPT, with CC BY-NC-SA 4.0 license

  * •

SentenceTransformer, with Apache-2.0 license

  * •

DreamerV3, with MIT license

##  Appendix I Broader Impacts

LLMs have the potential to produce harmful or biased information. We have not
observed LLMs generating such content in our current experimental
environments, including HomeGrid, Crafter, and Minecraft. However, applying
DLLM in other contexts, especially real-world settings, requires increased
attention to social safety concerns. Implementing necessary safety measures
involves screening LLM outputs, incorporating restrictive statements in LLM
prompts, or fine-tuning with curated data.