# Efficient Non-Parametric Uncertainty Quantification for Black-Box Large
Language Models and Decision Planning

Yao-Hung Hubert Tsai  Walter Talbott  Jian Zhang

###### Abstract

Step-by-step decision planning with large language models (LLMs) is gaining
attention in AI agent development. This paper focuses on decision planning
with uncertainty estimation to address the hallucination problem in language
models. Existing approaches are either white-box or computationally demanding,
limiting use of black-box proprietary LLMs within budgets. The paper’s first
contribution is a non-parametric uncertainty quantification method for LLMs,
efficiently estimating point-wise dependencies between input-decision on the
fly with a single inference, without access to token logits. This estimator
informs the statistical interpretation of decision trustworthiness. The second
contribution outlines a systematic design for a decision-making agent,
generating actions like “turn on the bathroom light” based on user prompts
such as “take a bath”. Users will be asked to provide preferences when more
than one action has high estimated point-wise dependencies. In conclusion, our
uncertainty estimation and decision-making agent design offer a cost-efficient
approach for AI agent development.

Uncertainty Quantification in Language Models, Planning

##  1 Introduction

The key distinction between traditional software and AI agents lies in the AI
agent’s ability to make decisions based on human requests, often conveyed in
natural language. For instance, when prompted to book a trip to Europe, the AI
agent should consider the user’s historical preferences and decide between
destinations like Vienna or Prague. However, in cases where both options are
equally suitable, the AI agent shall seek human input for clarification,
allowing users to disambiguate the decision. This process underscores the
importance of uncertainty quantification, validating the plausibility of one
or more decisions. Furthermore, single-round decision-making is insufficient.
AI agents must engage in multi-round decision-making, also known as step-by-
step decision planning. For example, having chosen Prague as the destination,
the AI agent then needs to decide on accommodation based on nearby tourist
attractions. In this work, we delve into the intricacies of step-by-step
decision-making by AI agents and explore the role of efficient uncertainty
quantification methods in enhancing the decision-making process.

Due to the surge in artificial intelligence, large language models (LLMs)
(Touvron et al., 2023a, b; Jiang et al., 2023, 2024; OpenAI et al., 2023) have
demonstrated their potential to fuel step-by-step decision planning in AI
applications across domains such as robotics (Ahn et al., 2022; Huang et al.,
2022; Ren et al., 2023), workflow automation (Yang et al., 2023; Wu et al.,
2023; Hong et al., 2023), and medicine (Thirunavukarasu et al., 2023). While
uncertainty quantification can mitigate the hallucination problem in LLMs
(i.e., the tendency for LLMs to be overly confident in their output, whether
the decision is correct or not), current approaches either necessitate access
to token logits (Fomicheva et al., 2020; Kuhn et al., 2023; Takayama & Arase,
2019; van der Poel et al., 2022; Malinin & Gales, 2020; Lee et al., 2018; Yoo
et al., 2022; Ren et al., 2022; Vazhentsev et al., 2022; Kadavath et al.,
2022) or involve high computational costs (Lin et al., 2023). Unfortunately,
the requirements hampers the application of building AI agents with black-box
proprietary LLMs within budget constraints, such as using GPT-4 (OpenAI et
al., 2023) as the backend for the AI agent. To address this concern, we
propose an elegant and efficient uncertainty quantification approach for step-
by-step decisions in language models. Then, building on this approach, we
showcase the systematic design of a lightweight yet powerful decision-making
agent.

Figure 1: A decision-making agent design. During the data collection phase, we
curate a dataset comprising 20,000 pairs associating user requests with smart
home actions, recognizing the potential for multiple actions per request. In
the subsequent model training stage, we conduct instruction fine-tuning. The
agent, utilizing a robust language model, generates a comprehensive set of
actions based on user requests and prior actions. Additionally, we train a
point-wise dependency neural estimator, establishing relationships among user
requests, historical actions, and the current action. Moving to the deployment
stage, we integrate the decision-making agent with the neural estimator to
enumerate potential actions guided by point-wise dependencies exceeding a
threshold determined from calibration data. User interaction occurs when
multiple actions are enumerated, prompting user selection. For a single
enumerated action, the agent executes it directly, and in the absence of any
enumerated actions, the agent ceases operation. We use green color to denote
the smart home, blue color to denote the decision-making agent, yellow color
to denote the point-wise dependency neural estimator, gradients of the blue
and the yellow color to denote the combination between the agent and the
neural estimator, and pink color to denote the user.

Our uncertainty quantification approach utilizes point-wise dependency neural
estimation (Tsai et al., 2020) to estimate the point-wise dependency
$\frac{p(x,y)}{p(x)p(y)}$ between $x$ (user prompt and AI agent’s history
decisions) and $y$ (AI agent’s current decision). Notably, this estimation is
efficient, achieved with a single inference from an auxiliary neural network.
In contrast, current black-box approaches (Lin et al., 2023) require the AI
agent to generate multiple decisions to assess decision quality, i.e.,
$p(y|x)$. The point-wise dependency $\frac{p(x,y)}{p(x)p(y)}$ has a clear
statistical interpretation: values greater than $1.0$ indicate correlation,
while values lower than $1.0$ indicate negative correlation. Conformal
prediction (Shafer & Vovk, 2008) can be further incorporated to establish a
threshold for $\frac{p(x,y)}{p(x)p(y)}$, determining whether to trust the
decision. Previous studies have demonstrated that conformal prediction is an
effective tool for uncertainty quantification in language models (Ren et al.,
2023; Quach et al., 2023; Kumar et al., 2023).

Utilizing the introduced uncertainty quantification approach, we devise a
decision-making agent capable of responding to a user prompt such as “writing
in the study” with a series of decisions, including 1) “study room light:
soft”, 2) “study room music player: play quiet music”, and then 3) “desk lamp:
on”. Our system encompasses three stages: data collection, model training, and
deployment. We provide an illustration in Figure 1.

In the data collection stage, we gather $20k$ user prompts and corresponding
potential home agent actions. Examples include {user prompt: time to fix
things; actions: garage light: on, tool drawer: open} and {user prompt: water
the plants; actions: outdoor lights: on, outdoor speaker: play laid-back
music, smart sprinkler: on}.

During the model training stage, we initially conduct instruction fine-tuning
using Mistral-7B-Instruct-v0.1111We treat Mistral-7B-Instruct-v0.1 as a black-
box model by not utilizing its token output logits. One can find the model in
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1.. Note that we
design the fine-tuning to generate all possible actions at once. Subsequently,
we train an auxiliary neural network for point-wise dependency neural
estimation between user prompts, history decisions, and current decisions.

The deployment stage employs an interactive process. We first use conformal
prediction to establish the threshold for point-wise dependency from the
calibration data split. Then, based on our request, we prompt the language
model, which performs a one-shot inference, generating all possible actions.
Our neural point-wise dependency estimator computes scores for all pairs of
user prompts and generated actions. If only one score exceeds the threshold,
the decision-making agent selects that action and appends it to the user
prompt. If multiple scores surpasses the threshold, the decision-making agent
asks the user to choose a preferred action, which is then appended to the user
prompt. The decision-making agent ceases generating actions when none of the
scores exceeded the threshold.

We assess the decision-making agent’s effectiveness by examining the mean
precision, recall, and F1 score of generated actions compared to true actions
on the evaluation data. Our findings highlight that step-by-step decision
planning outperforms all-at-once decision planning in terms of F1 score.
Furthermore, increasing the threshold for estimated point-wise dependency
during action selection enhances mean precision. The best performance is
achieved by consistently selecting the action with the maximum estimated
point-wise dependency in step-by-step decision planning, indicating a higher
likelihood of it being the true action.

##  2 Related Work

This paper covers a broad literature. In this section, our focus is
specifically directed towards intersections between large language models
(LLMs) with uncertainty quantification, AI agents, and decision planning.

Uncertainty Quantification. Uncertainty quantification plays a crucial role in
ensuring the safe and responsible utilization of large language models (LLMs).
Its primary function lies in disambiguating over-confident yet incorrect
responses from LLMs, a challenge commonly known as the hallucination problem
(Zhang et al., 2023). However, the task of uncertainty quantification is
inherently intricate (Smith, 2013; Belghazi et al., 2018), and its complexity
intensifies in the context of text generation using LLMs, particularly due to
the nuances of sampling and beam searches (Lin et al., 2023; Takayama & Arase,
2019; van der Poel et al., 2022; Kuhn et al., 2023; Malinin & Gales, 2020;
Kadavath et al., 2022). Categorizing related approaches reveals two primary
methodologies: the white-box and black-box methods.

On the one hand, the white-box approach necessitates access to logits,
internal layer outputs, or the LLMs themselves. Some approaches involve
forming the probability of output generation given user input (Fomicheva et
al., 2020; Kuhn et al., 2023; Takayama & Arase, 2019; van der Poel et al.,
2022; Malinin & Gales, 2020). Others focus on out-of-distribution detection
(Lee et al., 2018; Yoo et al., 2022; Ren et al., 2022; Vazhentsev et al.,
2022), determining whether a generation aligns with the training distribution.
And some approaches prompt LLMs to self-assess the correctness of their
outputs (Kadavath et al., 2022).

On the other hand, the black-box approach tackles the challenge of utilizing
proprietary large language models (LLMs) where only output texts are
accessible. A method (Lin et al., 2023) suggests estimating the probability of
an output response by generating multiple responses corresponding to the
input. This estimation involves assessing similarities between different
responses, although we contend that defining similarity between responses
introduces another challenge. The primary drawback of existing black-box
approaches is their computational intensity, demanding multiple inferences. In
contrast, our approach offers an elegant and efficient method for quantifying
uncertainty in black-box LLMs. We eliminate the need for multiple generations
given an input and the computation of sentence similarities.

Decision Planning. Step-by-step decision planning utilizing language models
has been demonstrated in various domains, including robotics (Ren et al.,
2023; Brohan et al., 2023; Driess et al., 2023; Tsai et al., 2023; Li et al.,
2022) and self-driving (Sha et al., 2023; Zhou et al., 2023). The overarching
objective is to leverage LLMs for generating actions based on the current
observation, historical observations, and goal specification. Contrasting with
traditional decision-making approaches (Chen et al., 2021), the advantage of
employing LLMs lies in their capacity to comprehend and reason with complex
human languages.

However, the utilization of LLMs in decision planning introduces challenges,
particularly the hallucination problem, which can potentially disrupt the
decision-making process. An early incorrect decision, influenced by
hallucinations, can have a cascading effect on subsequent decisions. In
addition to relying on uncertainty quantification to address the hallucination
problem, other researchers advocate for leveraging explicit (Zhou et al.,
2023) or implicit (Tsai et al., 2023) memory modules, a process known as
augmented retrieval (Guu et al., 2020; Izacard et al., 2022).

Furthermore, an emerging aspect in utilizing LLMs for decision planning
focuses on enhancing inference efficiency (Alizadeh et al., 2023; Sun et al.,
2020), enabling deployment on personal devices with mobile computing
capabilities. This paper specifically covers certain facets of employing LLMs
for step-by-step decision planning. Notably, we introduce an efficient
uncertainty quantification method to mitigate hallucinations and outline the
systematic design of a decision-making agent that incorporates past actions
when making new decisions.

AI Agent. The objective of an AI agent is to design and develop systems
capable of emulating human-like intelligence and abilities (Russell & Norvig,
2010; Wooldridge & Jennings, 1995). The evolution of LLMs (OpenAI et al.,
2023) has significantly enhanced the performance of these agents (Park et al.,
2023; Liu et al., 2023b; Sumers et al., 2023; OpenAI et al., 2023), where LLMs
serve as the cognitive center. Furthermore, the development goes beyond pure
text to encompass diverse perceptual abilities and tool usage (Nakano et al.,
2021; Yao et al., 2022; Schick et al., 2023; Lu et al., 2023). A pivotal
aspect of an AI agent is its capacity to interact with humans (Akyürek et al.,
2023; Peng et al., 2023; Liu et al., 2023a), enabling iterative improvement or
collaborative co-piloting in production. Applications of such agents span
across workflow automation (Wu et al., 2023), software development (Li et al.,
2023; Qian et al., 2023), scientific research (Boiko et al., 2023), and
medicine (Dai et al., 2023). This paper specifically focuses on the design of
an AI agent tailored for a user-interactive decision-making experience.

##  3 Efficient Non-parametric Uncertainty Quantification for Language Models

Notations. We define the training, calibration, and evaluation data splits as
${\mathcal{D}}_{\rm train}$, ${\mathcal{D}}_{\rm calib}$, ${\mathcal{D}}_{\rm
eval}$. Each data split consists of tuples between a user prompt $x$ and the
corresponding actions $\\{a^{j}\\}_{j=1}^{n_{x}}$, where $n_{x}$ denotes the
number of actions associated with the user prompt. For instance, $x$ is water
the plants and $\\{a^{j}\\}$ are {outdoor lights: on, outdoor speaker: play
laid-back music, smart sprinkler: on}. We denote their sample spaces as $X$
and $A$.

###  3.1 Point-wise Dependency Neural Estimation

The objective of uncertainty quantification is to measure uncertainty, such as
the conditional probability $p(a|x)$ (Fomicheva et al., 2020), representing
the relationship between a pair of action $a$ and user prompt $x$. This paper
employs techniques introduced in prior work (Tsai et al., 2020), utilizing a
neural network $\theta$ to estimate the point-wise dependency
$r(a,x)=\frac{p(a,x)}{p(a)p(x)}$, denoted as $r_{\theta}(a,x)$, between $a$
and $x$. The interpretation here is that a higher value of $r_{\theta}(a,x)$
indicates a stronger correlation between $a$ and $x$. Specifically, $r(a,x)<1$
implies negative dependence between $a$ and $x$, $r(a,x)=1$ suggests
independence, and $r(a,x)>1$ suggests positive dependence.

We adopt the relative predictive coding method (Tsai et al., 2021) as a
stabilized version of the density-ratio fitting method (Tsai et al., 2020):

$\begin{split}\underset{\theta}{\rm
sup}\,\,&\mathbb{E}_{P_{AX}}[r^{*}_{\theta}(a,x)]-\alpha\mathbb{E}_{P_{A}P_{X}}[r^{*}_{\theta}(a,x)]\\\
&-\frac{\beta}{2}\mathbb{E}_{P_{AX}}[{r^{*}_{\theta}}^{2}(a,x)]-\frac{\gamma}{2}\mathbb{E}_{P_{A}P_{X}}[{r^{*}_{\theta}}^{2}(a,x)].\end{split}$
(1)

We denote the sampling of an associated pair of action and user prompt as
$(a,x)\sim P_{AX}$, exemplified by smart sprinkler: on and water the plants.
Likewise, $(a,x)\sim P_{A}P_{X}$ signifies the sampling of an unassociated
pair of action and user prompt, such as smart sprinkler: on and feeling
hungry. $\alpha,\beta,\gamma$ are hyper-parameters, and
${r_{\theta}}(a,x)=\frac{\gamma r^{*}_{\theta}(a,x)+\alpha}{1-\beta
r^{*}_{\theta}(a,x)}$ in accordance with Lemma 1 in the prior work (Tsai et
al., 2021).

We select $\alpha=1.0,\beta=0.005,\gamma=0.1$ consistently throughout the
paper, and the design of $r_{\theta}^{*}(a,x)$ is expressed as

$r_{\theta}^{*}(a,x)=\langle f_{a}^{l}\circ f_{a}^{g}\circ
f^{llm}(a),f_{x}^{l}\circ f_{x}^{g}\circ f^{llm}(x)\rangle.$

Here, $\langle\cdot,\cdot\rangle$ denotes the inner product. $f^{llm}$
represents a pre-trained large language model (specifically, Mistral-7B-0.1
(Jiang et al., 2023) is adopted). $f_{a}^{g}$/$f_{x}^{g}$ are gated recurrent
units (GRUs) (Cho et al., 2014) applied over the sequence outputs from
$f^{llm}$, while $f_{a}^{l}$/$f_{x}^{l}$ are fully-connected layers attached
from the last unit in $f_{a}^{g}$/$f_{x}^{g}$. The network $\theta$ exists in
$f_{a}^{g}$/$f_{x}^{g}$ and $f_{a}^{l}$/$f_{x}^{l}$.

###  3.2 Extension to Step-by-step Planning

To establish a point-wise dependency neural estimator between an action and a
user prompt, we can perform gradient updates on the objective (1).
Nonetheless, we also aim to understand the confidence in executing an action
$a$ based on user prompt $x$ and the history of actions ${\bf a}_{\rm
history}:=\\{a_{-1},a_{-2},a_{-3},\cdots\\}$, where
$\\{a_{-1},a_{-2},a_{-3},\cdots\\}$ represent previously executed actions.

To achieve this, we introduce $X^{\prime}$ as the sample space that
encompasses user prompts and any taken actions. In the example,
$x^{\prime}\sim X^{\prime}$ could represent {“water the plants”}, {“water the
plants, outdoor lights: on”}, or {“water the plants, outdoor lights: on,
outdoor speaker: play laid-back music”}. Notably, we achieve this expansion by
simply appending the taken actions to the user prompt for $X^{\prime}$.
Consequently, eq. (1) undergoes modification as follows:

$\begin{split}\underset{\theta}{\rm
sup}\,\,&\mathbb{E}_{P_{AX^{\prime}}}[r^{*}_{\theta}(a,x^{\prime})]-\alpha\mathbb{E}_{P_{A}P_{X^{\prime}}}[r^{*}_{\theta}(a,x^{\prime})]\\\
&-\frac{\beta}{2}\mathbb{E}_{P_{AX^{\prime}}}[{r^{*}_{\theta}}^{2}(a,x^{\prime})]-\frac{\gamma}{2}\mathbb{E}_{P_{A}P_{X^{\prime}}}[{r^{*}_{\theta}}^{2}(a,x^{\prime})].\end{split}$
(2)

The estimated point-wise dependency between an action $a$ and user
prompt/history actions $x^{\prime}$ is then
${r_{\theta}}(a,x^{\prime})=\frac{\gamma
r^{*}_{\theta}(a,x^{\prime})+\alpha}{1-\beta r^{*}_{\theta}(a,x^{\prime})}$.

##  4 Decision-Making Agent

In this section, we present the design of our decision-making AI agent,
highlighting its capabilities in 1) controlling various functions within a
synthetic smart home, 2) comprehending human requests, 3) executing actions in
a step-by-step manner, and 4) soliciting human assistance when confronted with
ambiguous outputs (e.g., multiple plausible actions), particularly when
coupled with our uncertainty quantification technique (see Section 3). The
design is implemented across three key stages: data collection, model
training, and deployment.

###  4.1 Data Collection

We annotate $20k$ data with each data being user prompt and the associated
actions of the smart home. Examples are the following:

  * •

trimming the lawn, {outdoor light : on, yard musicplayer : play nature playing
music }

  * •

unwinding after work, {bedroom light : soft, bedroom musicplayer : play chill
music, diffuser : on}

  * •

video call with friends, {living room light : bright, living room smart
speaker : video call mode}

  * •

watching sunrise in the balcony, {balcony light : dim, outdoor speaker : play
morning raga, coffee maker : on}

  * •

relaxing with a book, {living room light : dim, fireplace : low, reading lamp
: on}

The user prompts and actions encompass a variety of scenarios, and we
prioritize brevity in both user prompts and actions. On average, each user
prompt corresponds to 3.1 actions. Following this, we perform a random
partition of the data into training, calibration, and evaluation sets,
adhering to a ratio of 10:1:2.

The training data serve the purpose of training our agency model and point-
wise dependency neural estimator. Calibration data play a crucial role in
determining the threshold within conformal prediction, providing a statistical
guarantee for our step-by-step planning decision-making process. Lastly, the
evaluation data are utilized to assess the performance of our decision-making
AI agent.

###  4.2 Model Training : Decision-Making Agent

Our objective is to train a decision-making agent capable of generating
plausible actions based on a user prompt and previously executed actions. To
accomplish this, we undertake instruction fine-tuning utilizing large language
models (LLMs). In our case, we choose Mistral-Instruction-7B-v0.1 (Jiang et
al., 2023) as the LLM, although any proprietary or open-sourced LLMs can be
employed for this purpose. It’s crucial to note that we do not use the logits
outputs from Mistral-Instruction-7B-v0.1, treating it as a black-box LLM. In
the following explanation, we will refer to the example in Figure 1, where the
user prompt is water the plants, and the corresponding actions are {outdoor
lights: on, outdoor speaker: play laid-back music, smart sprinkler: on}.

Instruction Fine-tuning between User Prompt and Action. For the instruction,
we instruct the home agent to function as a home assistant, deploying actions
based on an input command from the user:

  * •

[INST] You are a home assistant, and you receive a command  water the plants.

Please deploy your next action: [/INST]

For the output, we surround the action with identifiers $<$ACT$>$ and
$</$ACT$>$:

  * •

$<$ACT$>$ outdoor lights:on $</$ACT$>$

Since $<$ACT$>$ outdoor lights:on $</$ACT$>$ is not the only possible action,
the outputs can also be $<$ACT$>$ outdoor speaker: play laid-back music
$</$ACT$>$ or $<$ACT$>$ smart sprinkler: on $</$ACT$>$.

Extension to Step-by-step Instruction Fine-tuning. The initial fine-tuning, as
described above, does not consider executed actions, leading to a gap in
performing step-by-step decision planning. To integrate the history of
actions, we randomly select some of the corresponding actions with the user
prompt and modify the instruction as follows:

  * •

[INST] You are a home assistant, and you receive a command  water the plants.

You deployed $<$ACT$>$ outdoor lights:on $</$ACT$>$, $<$ACT$>$ outdoor
speaker: play laid-back music $</$ACT$>$.

Please deploy your next action: [/INST]

And the output for this instruction would be $<$ACT$>$ smart sprinkler: on
$</$ACT$>$.

Inference with Multiple Actions at Once. Our setting admits auto-regressive
training (i.e., Mistral-Instruct-7B-0.1 is a decoder-only LLM), meaning that
next tokens are decoded based on history tokens. In the step-by-step
instruction fine-tuning described above, the history actions are decoded one
by one. This results in an inference time where multiple actions can be
generated at once.

For example, when user prompt is take a hot bath, a single inference from our
decision-making agent generates

  * •

$<$ACT$>$ smart tubs : fill with hot water $</$ACT$>$, $<$ACT$>$ bathroom
speaker : play relaxing music $</$ACT$>$, $<$ACT$>$ towel warmer : on
$</$ACT$>$, $<$ACT$>$ bathroom light : soft $</$ACT$>$, $<$ACT$>$ blinds :
down $</$ACT$>$

In contrast, prior approaches on decision planning (Ren et al., 2023; Brohan
et al., 2023; Driess et al., 2023; Li et al., 2022) generate only one single
action per inference, whereas our model is designed to generate all actions in
a single inference.

Figure 2: Distributions of estimated point-wise dependency between user
prompt, taken actions, and current action. Figure 3: Conformal prediction on
calibration data. We define the non-conformity score as $50$ minus the
estimated point-wise dependency. Conformal prediction identifies the $80\%$
quantile value to establish the threshold at $1.627$. Consequently, during
test scenarios, we secure a statistical guarantee that the probability of the
“true action lying in the generated actions with scores greater than $1.627$”
is greater than $80\%$.

###  4.3 Model Training : Point-wise Dependency Neural Estimator

We train the objective (2) on $\mathcal{D}_{\rm train}$ for $3$ epochs to
obtain the estimated point-wise dependency (EPD). In the following, we will
delve into the distributions associated with the EPD and elaborate on our
utilization of conformal prediction (Shafer & Vovk, 2008; Ren et al., 2023) to
derive the threshold for the EPD.

Distributions for Estimated Point-wise Dependency. We visualize the
distributions for EPD on $\mathcal{D}_{\rm train}$, $\mathcal{D}_{\rm calib}$,
$\mathcal{D}_{\rm eval}$ in Figure 2. Notably, we emphasize the value $1$ in
the plots, indicating independence between the current action and the user
prompt/taken actions.

Firstly, given the use of neural networks for direct point-wise dependency
estimation, the presence of negative EPD values (albeit with low probability)
is inevitable.

Secondly, a substantial number of EPD values cluster around the value $1$.
Examples include {“A day of painting” and “living room light : bright”} and
{“A quiet evening with a book” and “electric kettle : on”}. These instances
illustrate that our approach effectively identifies cases where the action and
user prompt are not strongly dependent. Quantitatively, approximately $12\%$
of the data exhibits EPD values smaller than $1.21$, indicating that $12\%$ of
the annotated actions are not strongly dependent on the corresponding user
prompt.

Thirdly, we observe a similar distribution plot across the training,
calibration, and test data. This consistency suggests that our training
process does not overfit to the training data.

Conformal Prediction. Next, we employ conformal prediction (Shafer & Vovk,
2008) to establish the threshold for use in test scenarios. As conformal
prediction is not the primary focus of this paper, we present a simplified
discussion, and interested readers are encouraged to explore the comprehensive
study in the prior work (Ren et al., 2023) .

Let $x^{\prime}_{\rm test}$ be a user prompt along with its taken actions, and
$a_{\rm true}$ be the corresponding true action to be executed given
$x^{\prime}_{\rm test}$. Conformal prediction generates a prediction set
$\mathcal{C}(x^{\prime}_{\rm test})$ with a high probability of containing
$a_{\rm true}$:

$\mathbb{P}\Big{(}a_{\rm true}\in\mathcal{C}(x^{\prime}_{\rm test})\Big{)}\geq
1-\epsilon,$

where $\epsilon$ is user-defined (in our case, we adopt $\epsilon=20\%$). To
determine $\mathcal{C}(x^{\prime}_{\rm test})$, we define the non-conformity
score as $50$ minus the EPD values. Subsequently, we identify the $1-\epsilon$
empirical quantile of the non-conformity scores on calibration data, which is
$1.627$ in our case. Finally, we determine $\mathcal{C}(x^{\prime}_{\rm
test})=\\{a\in A|r_{\theta}(a,x^{\prime}_{\rm test})\geq 1.627\\}$.

This discussion indicates that, given $x^{\prime}_{\rm test}$, conformal
prediction provides a statistical guarantee that the probability of the true
action lying in the generated actions with scores greater than $1.627$ is
greater than $80\%$.

###  4.4 Deployment

Now, we are prepared to deploy our decision-making agent and point-wise
dependency neural estimator. For an illustration, please refer to Figure 1. We
will use the example in the figure for the following explanation.

Greeting and User Prompting. The initial stage of the deployment kicks off
with a greeting from the decision-making agent, followed by the user providing
a prompt. In the example, the user prompt is

  * [User]

I am exhausted today

Actions Generation and User Interaction. In the next stage, the trained
decision-making home agent proceeds to execute the user prompt and generates a
series of actions all at once using a single inference (refer to Section 4.2).
In the example, the decision-making agent generates the following actions:

  * •

living room light : dim, musicplayer : play soft sounds, air purifier : on,
smart blinds : close, living room musicplayer: play nature sound, coffee maker
: on, kitchen light : on

Upon closer inspection, it’s evident that some actions, such as air purifier :
on and kitchen light: on, may not be perfect matches for the user prompts.
Therefore, we rely on the trained point-wise dependency neural estimator to
assist in eliminating these options. By applying the threshold determined
using conformal prediction (refer to Section 4.3), the agent 1) selectively
chooses actions with scores above the threshold and 2) asks the user the
select one of them:

  * [Agent]

Action 1 is (living room light : dim) with point-wise dependency 2.206,
Action 2 is (musicplayer: play soft sounds) with point-wise dependency 1.928,
Action 3 is (living room light: dim) with point-wise dependency 1.837.
Please select an action.

In response, the user will select one of them:

  * [User]

Action 2

If there is only one generated action with an estimated point-wise dependency
larger than the threshold, the agent will automatically select this option.
The interactive process between the agent and the user concludes when no
generated actions have an estimated point-wise dependency above the threshold.

Actions Conclusion. In the final stage, all the selected actions will be
gathered and executed by the smart home, one by one:

  * [Home]

1\. musicplayer: play soft sounds
2\. living room light: warm
3\. living room AC: set temperature 20C
4\. aroma diffuser: on

##  5 Evaluation

This section assesses the performance of our decision-making agent and the
point-wise dependency estimator, addressing the following questions:

  * Q1.

Does step-by-step planning outperform all-at-once action generation?

  * Q2.

What is the optimal action generation strategy without human intervention?

  * Q3.

How does the threshold of the point-wise dependency estimator impact
performance?

Metrics. We evaluate our model on the evaluation data $\mathcal{D}_{\rm eval}$
by assessing the match between generated actions and true annotated actions
using an exact match criterion, which is stringent. For example, musicplayer:
play soft sounds and musicplayer: play soft music are considered distinct in
our evaluation. Our metrics include mean precision, mean recall, and the
corresponding F1 score:

  * •

${\rm
Precision}=\frac{\rm\\#\,\,of\,\,the\,\,generated\,\,actions\,\,that\,\,are\,\,true\,\,actions}{\rm\\#\,\,of\,\,the\,\,generated\,\,actions}$

  * •

${\rm
Recall}=\frac{\rm\\#\,\,of\,\,the\,\,generated\,\,actions\,\,that\,\,are\,\,true\,\,actions}{\rm\\#\,\,of\,\,the\,\,true\,\,actions}$

  * •

${\rm F1\,\,Score}=2\cdot\frac{\rm precision\cdot recall}{\rm
precision+recall}$

Precision and recall scores are computed for each user prompt, and the average
over all user prompts yields mean precision and mean recall. The F1 score is
then calculated based on the mean precision and mean recall. Mean precision
assesses the likelihood that a generated action is the true action. Mean
recall gauges the likelihood that true actions are encompassed by the
generated actions. The F1 score represents the harmonic average of precision
and recall.

Step-by-step Planning vs All-at-Once Generation. We compare actions generated
through step-by-step planning with those generated all-at-once (refer to
Inference with Multiple Actions at Once in Section 4.2). Results are presented
in Table 1. For actions generated all-at-once, we collect them by providing
only the user prompt without history actions to the decision-making agent.
Regarding step-by-step planning, we consider two scenarios: 1) randomly
selecting one action from the suggested actions and 2) always selecting the
action with the highest estimated point-wise dependency. All approaches share
a common threshold for point-wise dependency set at $1.0$, meaning actions
with scores below this threshold will not be generated.

Table results show that actions generated through step-by-step planning
outperform those generated all-at-once. We posit that this is because, in
step-by-step planning, an action is generated not solely based on the user
prompt but also leverages additional information: executed actions. This
observation parallels the chain-of-thought process in large language models
(OpenAI et al., 2023), suggesting that our decision-making agent achieves
better performance by making decisions incrementally rather than directly
making the final decision.

Table 1: Comparison between all-at-once action generation and actions generated through the step-by-step planning process (randomly selecting one action from the suggested actions v.s. always selecting the action with the highest estimated point-wise dependency). The threshold for point-wise dependency is set to $1.0$. metrics | all-at-once | step-by-step | step-by-step
---|---|---|---
(random) | (maximum)
mean precision | 0.108 | 0.126 | 0.134
mean recall | 0.200 | 0.193 | 0.212
F1 score | 0.140 | 0.152 | 0.164

Best Action Generation Strategy without Human in the Loop. Observing Table 1,
we note that employing step-by-step planning and consistently selecting
actions with the highest estimated point-wise dependency yields the best
performance. The fact that random selection performs worse than selecting
actions with the highest point-wise dependency implies that higher estimated
point-wise dependency indicates a higher likelihood of being the true action.
In future work, we aim to conduct a human study wherein actions are selected
one by one by humans, providing a more comprehensive analysis.

Threshold in the Point-wise Dependency Neural Estimator. Now, we explore how
the threshold $t$ for the point-wise dependency neural estimator affects the
overall performance. Actions below the threshold will be discarded during the
generation process. We examine three thresholds: $t=0.0$, $t=1.0$, and
$t=1.627$. A threshold of $0.0$ implies considering all actions with either
negative or positive point-wise dependencies. A threshold of $1.0$ means
considering only actions with positive point-wise dependencies. The threshold
of $1.627$ is determined during the conformal prediction process in Section
4.3 and is used to obtain a statistical guarantee that the probability of the
true action lying in the generated actions is greater than $80\%$. We provide
the results in Table 2.

Table 2: Analysis of the effects of threshold $t$ for estimated point-wise
dependency. Actions below the threshold will be discarded during the
generation process.  all-at-once action generation
---
metrics | t = 0.0 | t = 1.0 | t = 1.627
mean precision | 0.083 | 0.108 | 0.127
mean recall | 0.235 | 0.200 | 0.167
F1 score | 0.127 | 0.140 | 0.144
step-by-step planning (maximum)
metrics | t = 0.0 | t = 1.0 | t = 1.627
mean precision | 0.123 | 0.134 | 0.139
mean recall | 0.232 | 0.212 | 0.209
F1 score | 0.162 | 0.164 | 0.167

Our initial observation indicates that the statistical guarantee obtained from
conformal prediction (Ren et al., 2023; Shafer & Vovk, 2008) does not strongly
correlate with the actual performance in our evaluation. We attribute this to
the strict criterion of exact match used between generated actions and true
actions. Future extensions to our evaluation will include measures of semantic
similarity and human study in addition to exact match.

Our second observation indicates that increasing the threshold leads to an
improved F1 score. As the threshold rises, mean precision increases while mean
recall decreases. This observation is expected, as a higher threshold results
in discarding more generated actions, reducing recall. The increase in mean
precision suggests that our estimated point-wise dependency serves as an
effective selector for actions more likely to be true actions.

##  6 Conclusion

This work explores the design of the AI agent’s ability to make decisions
based on human requests in natural language. It delves into the challenges of
uncertainty quantification in decision-making and proposes an efficient
approach using point-wise dependency neural estimation for black-box large
language models, opening up the possibility of using powerful proprietary
language models. The work further advances the application of this approach in
building a decision-making agent capable of step-by-step decision planning,
detailing the stages of data collection, model training, and deployment.

## References

  * Ahn et al. (2022) Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ , 2022.
  * Akyürek et al. (2023) Akyürek, A. F., Akyürek, E., Madaan, A., Kalyan, A., Clark, P., Wijaya, D., and Tandon, N.  Rl4f: Generating natural language feedback with reinforcement learning for repairing model outputs.  _arXiv preprint arXiv:2305.08844_ , 2023.
  * Alizadeh et al. (2023) Alizadeh, K., Mirzadeh, I., Belenko, D., Khatamifard, K., Cho, M., Del Mundo, C. C., Rastegari, M., and Farajtabar, M.  Llm in a flash: Efficient large language model inference with limited memory.  _arXiv preprint arXiv:2312.11514_ , 2023.
  * Belghazi et al. (2018) Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D.  Mutual information neural estimation.  In _International conference on machine learning_ , pp. 531–540. PMLR, 2018.
  * Boiko et al. (2023) Boiko, D. A., MacKnight, R., and Gomes, G.  Emergent autonomous scientific research capabilities of large language models.  _arXiv preprint arXiv:2304.05332_ , 2023.
  * Brohan et al. (2023) Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al.  Rt-2: Vision-language-action models transfer web knowledge to robotic control.  _arXiv preprint arXiv:2307.15818_ , 2023.
  * Chen et al. (2021) Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I.  Decision transformer: Reinforcement learning via sequence modeling.  _Advances in neural information processing systems_ , 34:15084–15097, 2021.
  * Cho et al. (2014) Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  _arXiv preprint arXiv:1406.1078_ , 2014.
  * Dai et al. (2023) Dai, H., Li, Y., Liu, Z., Zhao, L., Wu, Z., Song, S., Shen, Y., Zhu, D., Li, X., Li, S., et al.  Ad-autogpt: An autonomous gpt for alzheimer’s disease infodemiology.  _arXiv preprint arXiv:2306.10095_ , 2023.
  * Driess et al. (2023) Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_ , 2023.
  * Fomicheva et al. (2020) Fomicheva, M., Sun, S., Yankovskaya, L., Blain, F., Guzmán, F., Fishel, M., Aletras, N., Chaudhary, V., and Specia, L.  Unsupervised quality estimation for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:539–555, 2020.
  * Guu et al. (2020) Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M.  Retrieval augmented language model pre-training.  In _International conference on machine learning_ , pp. 3929–3938. PMLR, 2020.
  * Hong et al. (2023) Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., et al.  Metagpt: Meta programming for multi-agent collaborative framework.  _arXiv preprint arXiv:2308.00352_ , 2023.
  * Huang et al. (2022) Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al.  Inner monologue: Embodied reasoning through planning with language models.  _arXiv preprint arXiv:2207.05608_ , 2022.
  * Izacard et al. (2022) Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and Grave, E.  Few-shot learning with retrieval augmented language models.  _arXiv preprint arXiv:2208.03299_ , 2022.
  * Jiang et al. (2023) Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Jiang et al. (2024) Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E.  Mixtral of experts, 2024.
  * Kadavath et al. (2022) Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al.  Language models (mostly) know what they know.  _arXiv preprint arXiv:2207.05221_ , 2022.
  * Kuhn et al. (2023) Kuhn, L., Gal, Y., and Farquhar, S.  Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.  _arXiv preprint arXiv:2302.09664_ , 2023.
  * Kumar et al. (2023) Kumar, B., Lu, C., Gupta, G., Palepu, A., Bellamy, D., Raskar, R., and Beam, A.  Conformal prediction with large language models for multi-choice question answering.  _arXiv preprint arXiv:2305.18404_ , 2023.
  * Lee et al. (2018) Lee, K., Lee, K., Lee, H., and Shin, J.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  _Advances in neural information processing systems_ , 31, 2018.
  * Li et al. (2023) Li, G., Hammoud, H. A. A. K., Itani, H., Khizbullin, D., and Ghanem, B.  Camel: Communicative agents for” mind” exploration of large scale language model society.  _arXiv preprint arXiv:2303.17760_ , 2023.
  * Li et al. (2022) Li, S., Puig, X., Paxton, C., Du, Y., Wang, C., Fan, L., Chen, T., Huang, D.-A., Akyürek, E., Anandkumar, A., et al.  Pre-trained language models for interactive decision-making.  _Advances in Neural Information Processing Systems_ , 35:31199–31212, 2022.
  * Lin et al. (2023) Lin, Z., Trivedi, S., and Sun, J.  Generating with confidence: Uncertainty quantification for black-box large language models.  _arXiv preprint arXiv:2305.19187_ , 2023.
  * Liu et al. (2023a) Liu, H., Sferrazza, C., and Abbeel, P.  Languages are rewards: Hindsight finetuning using human feedback.  _arXiv preprint arXiv:2302.02676_ , 2023a.
  * Liu et al. (2023b) Liu, R., Yang, R., Jia, C., Zhang, G., Zhou, D., Dai, A. M., Yang, D., and Vosoughi, S.  Training socially aligned language models in simulated human society.  _arXiv preprint arXiv:2305.16960_ , 2023b.
  * Lu et al. (2023) Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y. N., Zhu, S.-C., and Gao, J.  Chameleon: Plug-and-play compositional reasoning with large language models.  _arXiv preprint arXiv:2304.09842_ , 2023.
  * Malinin & Gales (2020) Malinin, A. and Gales, M.  Uncertainty estimation in autoregressive structured prediction.  _arXiv preprint arXiv:2002.07650_ , 2020.
  * Nakano et al. (2021) Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_ , 2021.
  * OpenAI et al. (2023) OpenAI, :, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings, D., Currier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Łukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, J. W., Kim, C., Kim, Y., Kirchner, H., Kiros, J., Knight, M., Kokotajlo, D., Łukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li, C. M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O’Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P., Michael, Pokorny, Pokrass, M., Pong, V., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B.  Gpt-4 technical report, 2023.
  * Park et al. (2023) Park, J. S., O’Brien, J., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S.  Generative agents: Interactive simulacra of human behavior.  In _Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_ , pp. 1–22, 2023.
  * Peng et al. (2023) Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., Chen, W., et al.  Check your facts and try again: Improving large language models with external knowledge and automated feedback.  _arXiv preprint arXiv:2302.12813_ , 2023.
  * Qian et al. (2023) Qian, C., Cong, X., Yang, C., Chen, W., Su, Y., Xu, J., Liu, Z., and Sun, M.  Communicative agents for software development.  _arXiv preprint arXiv:2307.07924_ , 2023.
  * Quach et al. (2023) Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T. S., and Barzilay, R.  Conformal language modeling.  _arXiv preprint arXiv:2306.10193_ , 2023.
  * Ren et al. (2023) Ren, A. Z., Dixit, A., Bodrova, A., Singh, S., Tu, S., Brown, N., Xu, P., Takayama, L., Xia, F., Varley, J., et al.  Robots that ask for help: Uncertainty alignment for large language model planners.  _arXiv preprint arXiv:2307.01928_ , 2023.
  * Ren et al. (2022) Ren, J., Luo, J., Zhao, Y., Krishna, K., Saleh, M., Lakshminarayanan, B., and Liu, P. J.  Out-of-distribution detection and selective generation for conditional language models.  _arXiv preprint arXiv:2209.15558_ , 2022.
  * Russell & Norvig (2010) Russell, S. J. and Norvig, P.  _Artificial intelligence a modern approach_.  London, 2010.
  * Schick et al. (2023) Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., and Scialom, T.  Toolformer: Language models can teach themselves to use tools.  _arXiv preprint arXiv:2302.04761_ , 2023.
  * Sha et al. (2023) Sha, H., Mu, Y., Jiang, Y., Chen, L., Xu, C., Luo, P., Li, S. E., Tomizuka, M., Zhan, W., and Ding, M.  Languagempc: Large language models as decision makers for autonomous driving.  _arXiv preprint arXiv:2310.03026_ , 2023.
  * Shafer & Vovk (2008) Shafer, G. and Vovk, V.  A tutorial on conformal prediction.  _Journal of Machine Learning Research_ , 9(3), 2008.
  * Smith (2013) Smith, R. C.  _Uncertainty quantification: theory, implementation, and applications_ , volume 12.  Siam, 2013.
  * Sumers et al. (2023) Sumers, T. R., Yao, S., Narasimhan, K., and Griffiths, T. L.  Cognitive architectures for language agents.  _arXiv preprint arXiv:2309.02427_ , 2023.
  * Sun et al. (2020) Sun, Z., Yu, H., Song, X., Liu, R., Yang, Y., and Zhou, D.  Mobilebert: a compact task-agnostic bert for resource-limited devices.  _arXiv preprint arXiv:2004.02984_ , 2020.
  * Takayama & Arase (2019) Takayama, J. and Arase, Y.  Relevant and informative response generation using pointwise mutual information.  In _Proceedings of the First Workshop on NLP for Conversational AI_ , pp. 133–138, 2019.
  * Thirunavukarasu et al. (2023) Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., and Ting, D. S. W.  Large language models in medicine.  _Nature medicine_ , 29(8):1930–1940, 2023.
  * Touvron et al. (2023a) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. (2023b) Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Tsai et al. (2020) Tsai, Y.-H. H., Zhao, H., Yamada, M., Morency, L.-P., and Salakhutdinov, R. R.  Neural methods for point-wise dependency estimation.  _Advances in Neural Information Processing Systems_ , 33:62–72, 2020.
  * Tsai et al. (2021) Tsai, Y.-H. H., Ma, M. Q., Yang, M., Zhao, H., Morency, L.-P., and Salakhutdinov, R.  Self-supervised representation learning with relative predictive coding.  _arXiv preprint arXiv:2103.11275_ , 2021.
  * Tsai et al. (2023) Tsai, Y.-H. H., Dhar, V., Li, J., Zhang, B., and Zhang, J.  Multimodal large language model for visual navigation.  _arXiv preprint arXiv:2310.08669_ , 2023.
  * van der Poel et al. (2022) van der Poel, L., Cotterell, R., and Meister, C.  Mutual information alleviates hallucinations in abstractive summarization.  _arXiv preprint arXiv:2210.13210_ , 2022.
  * Vazhentsev et al. (2022) Vazhentsev, A., Kuzmin, G., Shelmanov, A., Tsvigun, A., Tsymbalov, E., Fedyanin, K., Panov, M., Panchenko, A., Gusev, G., Burtsev, M., et al.  Uncertainty estimation of transformer predictions for misclassification detection.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 8237–8252, 2022\.
  * Wooldridge & Jennings (1995) Wooldridge, M. and Jennings, N. R.  Intelligent agents: Theory and practice.  _The knowledge engineering review_ , 10(2):115–152, 1995.
  * Wu et al. (2023) Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., and Wang, C.  Autogen: Enabling next-gen llm applications via multi-agent conversation framework.  _arXiv preprint arXiv:2308.08155_ , 2023.
  * Yang et al. (2023) Yang, H., Yue, S., and He, Y.  Auto-gpt for online decision making: Benchmarks and additional opinions.  _arXiv preprint arXiv:2306.02224_ , 2023.
  * Yao et al. (2022) Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_ , 2022.
  * Yoo et al. (2022) Yoo, K., Kim, J., Jang, J., and Kwak, N.  Detection of word adversarial examples in text classification: Benchmark and baseline via robust density estimation.  _arXiv preprint arXiv:2203.01677_ , 2022.
  * Zhang et al. (2023) Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al.  Siren’s song in the ai ocean: A survey on hallucination in large language models.  _arXiv preprint arXiv:2309.01219_ , 2023.
  * Zhou et al. (2023) Zhou, G., Hong, Y., and Wu, Q.  Navgpt: Explicit reasoning in vision-and-language navigation with large language models.  _arXiv preprint arXiv:2305.16986_ , 2023.