#  Outcome-supervised Verifiers for Planning in Mathematical Reasoning

Fei Yu1,2, Anningzhe Gao2, Benyou Wang1,2
The Chinese Unviersity of Hong Kong, Shenzhen, China
Shenzhen Research Institute of Big Data
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
https://github.com/FreedomIntelligence/OVM

###### Abstract

Large language models (LLMs) often struggle with maintaining accuracy across a
sequence of intermediate reasoning steps in mathematical reasoning, leading to
error propagation that undermines the final result. The current methodology to
mitigate this issue primarily involves using a verifier model to assess the
correctness of generated solution candidates, focusing either on the overall
reasoning path or on an incomplete reasoning path. By rethinking this
approach, we argue that assessing potentials of incomplete reasoning paths
could be more advantageous as it guides towards correct final answers,
transforming the task into a planning problem.

Our proposed verifier, the Outcome-supervision Value Model (OVM), employs
outcome supervision for training, offering an efficient and intuitive method
for planning by prioritizing steps that lead to accurate conclusions over mere
per-step correctness. Furthermore, the OVM eschews the need for labor-
intensive annotations on step-level correctness, enhancing its scalability.
Our experiments on two multi-step mathematical reasoning datasets, GSM8K and
Game of 24, demonstrate the superior performance of the OVM model. Notably, in
GSM8K, our OVM-7B model achieves state-of-the-art results among LLMs up to 13B
parameters; especially it does not utilize GPT-4 or code execution. These
findings offer a novel perspective on the role of outcome supervision in
training verifiers for multi-step reasoning tasks and provide theoretical
justification for its advantage in value estimation for planning.

Outcome-supervised Verifiers for Planning in Mathematical Reasoning

Fei Yu1,2, Anningzhe Gao2, Benyou Wang1,2 The Chinese Unviersity of Hong Kong,
Shenzhen, China Shenzhen Research Institute of Big Data
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>https://github.com/FreedomIntelligence/OVM

##  1 Introduction

Multi-step reasoning problems are challenging for even large language models
(LLMs)(Creswell et al., 2023, Press et al., 2022, Wei et al., 2022).
Especially, it seems that the model capability to implicitly compose multiple
information in an end-to-end manner is not improved as model size increases
(Press et al., 2022). A simple method, Chain of Thought (CoT), which first
outputs a series of intermediate reasoning steps and then finally the answer,
can significantly improve the multi-step reasoning performance(Wei et al.,
2022, Suzgun et al., 2023). However, errors in the previous steps can affect
accuracy of the following steps and the answer in such a single chain, which
is also known as error propagation. For example, Yao et al. (2023) found that
GPT-4 often fails at the first step in Game of 24, blocking the success, while
it actually can solve the task in multiple attempts.

Verifiers for complete reasoning paths To address the error propagation,
(Cobbe et al., 2021, Uesato et al., 2022, Li et al., 2023b, Khalifa et al.,
2023a, Lightman et al., 2023) employ a verifier model to assess generated
solution candidates in terms of correctness on the overall reasoning path.
Typically, there are two types of supervision for verifier training: outcome
supervision and process supervision. (Uesato et al., 2022, Lightman et al.,
2023) have demonstrated a a clear advantage of process supervision over
outcome supervision for training verifiers in terms of verifying complete
reasoning paths, as the former provide more fine-grained supervisions.

Figure 1:  Process supervision and outcome supervision on training verifiers
for complete path verification. Given a question $q$ and a solution path
$[S_{1},\cdots,S_{m},a]$, the verifier is trained to predict path correctness,
which is the circled output scalar on the last token. Training is conducted at
the token level for robustness. Outcome supervision focuses solely on the
final answer’s correctness, replicating this scalar label across all tokens
(indicated by shaded labels), while process supervision assigns labels at each
step. This “copy behavior” accidentally endows outcome supervision with
foresight. Correct steps and answers are colored in yellow and incorrect ones
in grey.

Verifiers for intermediate steps.  This paper, diverging from the traditional
approach of verifying complete reasoning paths, focuses on verifying
intermediate steps. The latter provides a finer granularity of verification,
facilitating the early elimination of unreliable reasoning while fostering
potentially promising inferences Yao et al. (2023). Contrast to complete path
verification, we argue that an outcome supervision verifier might have
advantages over its process supervision counterpart in partial path
verification. The outcome supervision verifier selects intermediate steps that
are more likely to yield accurate answers, unlike the process supervision
verifier that assesses each step in an immediate sense without considering its
potential contribution for the final result. This is analogous to the
difference between reward model and value model in Reinforcement Learning: a
value model forecasts long-term outcomes versus a reward model’s immediate
incentives, see theoretical backing in Sec. 3.1.2

In this paper, we propose to use outcome based supervision to train a verifier
to verify intermediate steps, called OVM. Experiments are conduct on two
popular multi-step mathematical reasoning datasets – GSM8K (Cobbe et al.,
2021) and Game of 24 (Yao et al., 2023). In GSM8K, our OVM-7B model obtains
state-of-the-art performance among models with up to 13B parameters, attaining
a 84.7% accuracy without resorting to supplementary datasets, GPT-4, or
executing programs. In Game of 24, OVM-7B reaches 78.7% success rate with
merely 20 nodes visited per step, in stark contrast to its 11% greedy success
rate and 11.7% with majority voting over 100 reasoning paths. Furthermore, we
demonstrate that our method attains competitive, and often superior,
performance using fewer sampled reasoning paths compared to complete path
verification on both GSM8K and Game of 24. This indicates the effectiveness of
OVM in value estimation, particularly in complex questions requiring longer
steps.

In summary, our contributions are three-fold.

  * •

Planning with outcome-supervised verifiers: we are the first to propose
outcome-supervised verifiers for planning. This method emphasizes the
correctness of the final answer rather than focusing solely on the current
(partial) path’s correctness, which also evidenced by theoretical
understanding. Moreover, OVM also obviates the need for costly step-level
annotations required by process supervision,

  * •

SOTA models in mathematical reasoning: OVM achieves state-of-the-art results
in GSM8K among LLMs up to 13B parameters; especially it does not leverage
additional data, GPT-4, or code execution.

  * •

Systematic rethink on verifiers: We conduct systematic rethink on existing
work regarding verifiers in mathematical reasoning. Especially we discuss the
fine-grained supervision hypothesis in complete path verification and whether
it could be directly applied in partical path verification. We clarify the
fundamental distinction between process supervision (akin to reward) and
outcome supervision (akin to value), proving that "the outcome matters" in
planning contexts, contrary to prior comparisons in literature.

##  2 Rethinking Verifiers in mathematical reasoning

We first give the problem definition of mathmatical reasoning where verifiers
work in two manners: complete path verification and partial path verification.
Then, we introduce two approaches for intermediate state evaluation in partial
path sampling. The main involved notations are shown in Table. 1.

Notation | Description
---|---
$q$ | Multi-step question requiring a sequence of steps
$\mathcal{Q}$ | a question set
$S$ | Solution path for a question, $S=[s_{1},\dots,s_{m},a]$
$S_{i}$ | The i-th step in a solution path
$\mathbb{S}$ | Set of solution path candidates, $\mathbb{S}=\\{S_{1},\dots,S_{k}\\}$
$m$ | Number of steps in a solution path
$a$ | Final answer in a solution path
$\Phi$ | The language model as the generator
$K$ | Sampling size for solution candidates
$f$ | a verifier model that maps a path to a positive number: $\mathbb{S}\rightarrow\mathbb{R}^{+}$
$\hat{S}$ | Predicted solution path with the highest score
$S^{(1:t)}$ | Partial solution path up to step $t$
$\mathbb{S}^{(1:t)}$ | Set of candidate partial paths, $\mathbb{S}_{(1:t)}=\\{S_{1}^{(1:t)},\dots,S_{k}^{(1:t)}\\}$
$v^{(1:t)}$ | Score of a state (partial path)
ORM | Outcome Reward Model, trained with outcome supervision
OVM | Outcome Value Model, trained with outcome supervision
PRM | Process Reward Model, trained with process supervision
Table 1: Summary of Notations Used in the Paper

###  2.1 Problem Definition

####  2.1.1 Mathematical Reasoning Problem

In this paper, we focus on mathematical reasoning problem, typical involves
multiple steps.

######  Definition 1.

A mathematical reasoning question $q\in\mathcal{Q}$ requires a sequence of
steps to be addressed, whose solution path is $S=[s_{1},\dots,s_{m},a]$, where
$s_{i}$ represents the i-th step, $m$ is the number of steps, and $a$ is the
final answer.

##### Generate-and-verify Paradigm

To alleviate the issue of potential error propagation in a single solution
chain, one approach is sampling multiple solutions from the generator $\Phi$
for the question $q$, resulting in a $K$-sized solution candidate set
$\mathbb{S}=\\{S^{k}\\}_{k=1}^{K}$, where $K$ is the sampling size. To relive
the issue of error propagation, this paper focuses on verifiers for reranking
solution candidates, which has been shown to outperform majority-voting (Wang
et al., 2023). There are two verifier usage methods: I) Complete Path
Verification, and II) Partial Path Verification; the former assesses the
correctness of the entire solution path, the latter concentrates on verifying
intermediate steps within a partial path.

####  2.1.2 Scenario I: Complete Path Verification

For a set of complete solution paths $\mathbb{S}=\\{S^{k}\\}_{k=1}^{K}$, a
simple protocol is to verify the complete path: each path’s correctness is
scored by an external verifier model, also known as ‘reward model’ (Li et al.,
2022, Uesato et al., 2022, Lightman et al., 2023), denoted as $f$ that maps a
solution to a positive number $f:\mathbb{S}\rightarrow\mathbb{R}^{+}$. One
could obtain the predicted solution (i.e., a reasoning path) with the highest
score:

$\hat{S}=\mathop{\mathrm{argmax}}\limits_{S\in\mathbb{S}}f(q;S).$ (1)

To train verifiers, one could utilize an outcome supervision or process
supervision demonstrated in Figure 1. Verifiers employing outcome supervision
are trained solely with the correctness of the final answer, whereas verifiers
with process supervision is trained using labels of per-step correctness. See
further details in Appendix A. We conjecture a Fine-grained Supervision
Hypothesis in complete-path verification as below:

######  Definition 2.

Fine-grained Supervision Hypothesis Process supervision -based verifiers
outperform outcome supervision -based verifiers.

(Lightman et al., 2023) empirically demonstrated the Hypothesis in complete-
path verification. The rationale could be that labels of process supervision
are at finer-grained step level instead of a solution level in outcome
supervision (Cobbe et al., 2021), leading to more informative supervisions.

####  2.1.3 Scenario II: Partial Path Verification

Contrast to the complete path verification, partial path verification
dynamically guides generation in each step. Here, the generator $\Phi$
produces tokens based on linguistic probabilities, pausing for evaluation at
each step. Given $t$ steps already generated by the generator, there exists K
candidate solutions
$\mathbb{S}^{(1:t)}=\bigl{\\{}S^{(1:t)}_{k}\bigl{\\}}_{k=1}^{K}$, and each
candidate solution in first t steps is defined as
$S^{(1:t)}_{k}=\\{s^{(1)}_{k},\dots,s_{k}^{(t)}\\}$. Supposed there exists a
verifier model $f_{\textrm{value}}(\cdot)$ that could verify an incomplete
solution within first $t$ steps. The candidate set with top-$n$ selected
incomplete path using verifier $f(\cdot)$ is

$\Bigg{\\{}S^{(1:t)}_{k}\scalebox{1.8}{$\mid$}k\in\text{arg\,top}_{n}\left(\scalebox{1.5}{$\\{$}f(q;S^{(1:t)}_{k})\scalebox{1.5}{$\\}$}_{k=1}^{K}\right)\Bigg{\\}}$
(2)

In general, partial path verification might outperform complete path
verification as the former provides finer-grained verification than the latter
(Khalifa et al., 2023a). In partial path verification, one could also use
either an outcome-based verifier or process-based verifier. A open question
remains as below:

######  Rethinking 1.

Does Fine-grained Supervision Hypothesis for verifiers still hold in partial
path verification?

This is discussed in the following Sec. 2.2.

###  2.2 Reward Model vs. Value Model for Partial Path Verification

Contrast to the complete path verification, here we discuss the scenario when
the path is not complete and needs to be completed. This turns the problem
from  one-shot ranking to multi-step planning. Depending on the type of
supervision (process vs. outcome) employed by verifiers, the planning criteria
in reasoning step planning vary. Verifiers under process supervision assess
per-step correctness during planning, similar to a reward model in
reinforcement learning; whereas those under outcome supervision focus on the
likelihood of achieving a correct final answer, similar to a value model in
reinforcement learning.

  * •

Reward-based Models accesses only the correctness of the current steps, i.e.
$p(S^{(1:t)}\mbox{ is correct}|q)$. Obviously, annotating such step-level
correctness labels are labor-intensive. Recently, Khalifa et al. (2023b)
proposed an auto-labeling algorithm aligning with reference solutions, yet it
requires ground solution paths.

  * •

Value-based Models cares about the correctness of the final answer in the
future, which will be generated in the latter steps. Formally, suppose the
future steps beginning from timestamp $t+1$ is
$[\hat{S}^{(t+1:m)},\hat{a}]=\Phi(S^{(1:t)})$, value-based evaluation assesses
the “value” $p(\hat{a}\mbox{ is correct}|q;S^{(1:t)})$.

Intuitively, in Partial Path Verification, verifiers should adopt a future-
oriented approach that emphasizes the correctness of the final answer rather
than focusing solely on the current (partial) path’s correctness, akin to
prioritizing a value model over a reward model. Here comes another rethinking
point which seems contraction to the Fine-grained Supervision Hypothesis in
Definition 2 but in the context of future-oriented planning, since process
supervision leads to a reward-based model while outcome supervision leads to a
value-based model. See further discussion in Section 3.

######  Rethinking 2.

In reasoning step planning, verifiers with outcome supervision (i.e., a value
model) might outperform that with process supervision (i.e., a reward model)
in partial path verification.

Moreover, verifiers with outcome supervision is easier for scalability since
it does not need labor-intensive annotation on step-level correctness.

###  2.3 Our Solution

A few existing work tried value-based methods (Yao et al., 2023, Hao et al.,
2023, Feng et al., 2023), which takes the future outcomes into consideration.
Those implementation are prompt-based, we focus on training an evaluation
model which can capture the generator’s distribution in this paper. The
challenge of training a value model in partial path verification lies in the
value estimation since the possible continued sequences $\Phi(S^{(1:t)})$ can
be infinite. To approximate $p(\hat{a}\mbox{ is correct}|q;S^{(1:t)})$,
standard reinforcement learning (RL) optimizes a value model with sampled
paths based on algorithms such as TD and MC. Recently, Feng et al. (2023)
trains a value model using standard RL algorithms.

However, we will show in Section 3 that outcome supervision accidentally
enables the model to predict $p(\hat{a}\mbox{ is correct}|q;S^{(1:t)})$ with a
new strategy. We call the process of guided decoding with the value-based
state evaluation, which is future-oriented and guides the model to correct
answers, as “planning”.

##  3 Outcome Supervision for Planning

In this section, we first give an explanation on why we can directly apply
outcome supervision, initially designed to train a reward model (verifier), to
develop a value model, despite reward model and value model are very
different. Then, we detail our implementations. For simplicity, we name the
model as Outcome Value Model (OVM).

###  3.1 Motivation

We first introduce the intuition of outcome-supervised model for value
estimation, then we give an explanation from a theoretical perspective.

####  3.1.1 Intuitive explanation

Although it is originally intended for assessing the correctness of complete
reasoning paths, the token-level extension of outcome supervision labels,
unexpectedly aids each step in learning to predict future outcomes. As shown
in Figure 1, the model is trained to predict the correctness of the final
answer at each step, irrespective of the fact that the final answer is yet to
be produced and remains unseen at that point in time.

Moreover, the model is trained to predict each step’s value, which correlates
with its likelihood of leading to a correct answer. Let’s imagine that, if a
specific step in the training set always leads to correct answers, the model
learns to predict 1 since all of the training labels are 1, indicating it
invariably leads to a correct answer. If it always leads to incorrect answers,
it predicts 0. For a step leading to both correct and incorrect answers, the
model learns with both labels and finally predict a value between 0 and 1,
tending towards the one more frequently observed in the training set.

####  3.1.2 Theoretical Explanation

We will show the theoretical reason why outcome supervision can lead to a
value model in this subsection.

######  Claim 1.

For a OVM model $f_{\bm{\theta}}(\cdot)$ parameterized by the optimal
parameter $\bm{\theta}$, the OVM’s score of a given state within first $t$
steps is the approximated probability to reach a correct answer, i.e.,

$f_{\bm{\theta}}(q;S^{(1:t)})=p(\hat{a}\mbox{ is correct}|q;S)$ (3)

###### Proof.

Suppose for each question $q$, we have the generator producing $n$ solution
paths $\\{S_{i}\\}_{i=1}^{n}$ and the corresponding answers
$\\{a_{i}\\}_{i=1}^{n}$. We define the correctness $y_{i}$ as 1 if $a_{i}$ is
correct otherwise 0. The MSE loss of outcome supervision is

$l(q;S_{i}^{(1:t)},y^{i})=\left(f_{\bm{\theta}}(q;S_{i}^{(1:t)})-y^{i}\right)^{2}$

Given the training question set $\mathcal{Q}$, the overall objective is

$L(q;S_{i}^{(1:t)},y^{i})=\frac{1}{|\mathcal{Q}|}\sum_{q\in\mathcal{Q}}\frac{1}{n}\sum_{i=1}^{n}\sum_{t=1}^{m_{i}}\left(f_{\bm{\theta}}(q;S_{i}^{(1:t)})-y^{i}\right)^{2}$
(4)

Denote $\mathrm{v}^{q}_{S}=f_{\bm{\theta}}(q;S)$, the partial derivation of
$\mathrm{v}^{q}_{S}$ is

$\nabla_{\mathrm{v}^{q}_{S}}L=\frac{1}{|\mathcal{Q}|}\frac{1}{n}\sum_{i=1}^{n}\sum_{t=1}^{m_{i}}2\Gamma(S=S_{i}^{(1:t)})(\mathrm{v}^{q}_{S}-y^{i})$

By optimizing
$\nabla_{\bm{\theta}}L=\nabla_{\mathrm{v}^{q}_{S}}L\cdot\nabla_{\rm{\theta}}\mathrm{v}^{q}_{S}=0$,
we get

$\mathrm{v}^{q}_{S}=\frac{\sum_{i=1}^{n}\sum_{t=1}^{m_{i}}\Gamma(S=S_{i}^{(1:t)})y^{i}}{\sum_{i=1}^{n}\sum_{t=1}^{m_{i}}\Gamma(S=S_{i}^{(1:t)})}$

which is $p(\hat{a}\mbox{ is correct}|q;S)$, whose estimation’s accuracy
depends on the sampling.

∎

###  3.2 Implementation

Algorithm 1 Value-Guided Beam Search

1:Input: Question $q$, Beam size $b$, Sampled steps per state $K$, Maximum
step count $T$

2:Output: Best solution sequence for $q$

3:Model: Generator $\Phi$ and OVM $f$

4:procedure ValueGuidedBeamSearch($q,b,K$)

5: Initialize step sequences $\mathbb{S}\leftarrow\\{\\}$

6: Sample initial steps $\\{s^{1}_{1},\dots,s^{k}_{1}\\}$

7: Evaluate values $\\{v^{1}_{1},\cdots,v^{k}_{1}\\}$ for each step

8: Select top $b$ valued steps and add to $\mathbb{S}$

9: $t\leftarrow 1$

10: while sequences in $\mathbb{S}$ are not complete and $t<T$ do

11: $\mathbb{S}_{\text{new}}\leftarrow\\{\\}$

12: $\mathcal{V}\leftarrow\\{\\}$

13: for each sequence $S^{(1:t)}$ in $\mathbb{S}$ do

14: for $i=1$ to $K/b$ do

15: $S^{(1:t+1)}_{i}=\Phi(q;S^{(1:t)})$

16: $v^{(1:t+1)}_{i}=f(q;S^{(1:t+1)}_{i};)$

17:
$\mathbb{S}_{\text{new}}\leftarrow\mathbb{S}_{\text{new}}+S^{(1:{t+1})}_{i}$

18: $\mathcal{V}\leftarrow\mathcal{V}+v^{(1:t+1)}_{i}$

19: end for

20: end for

21: $\mathbb{S}_{\text{new}}\leftarrow$ top $b$ valued sequences from
$(\mathbb{S}_{\text{new}},\mathcal{V})$

22: $\mathbb{S}\leftarrow\mathbb{S}_{\text{new}}$

23: $t\leftarrow t+1$

24: end while

25: return sequence with highest final value in $\mathbb{S}$

26:end procedure

##### Model architecture

OVM is a decoder-only autoregressive language model, with an additional linear
layer plus a single bias parameter after the model’s final unembedding layer.
It is a model independent of the generator.

##### OVM training

Given a training question set, we generate $n$ solution paths per question
using the generator, then detect answer correctness, such as through string
matching to the ground answer. This allows us to label numerous OVM training
samples using just question-answer pairs, without path annotations. OVM is
then trained on these labels with MSE loss under outcome supervision.

##### Inference - beam search with planning

In this paper, we conduct beam search strategy guided by OVM during inference.
Diverging from the traditional token-level probability-guided beam search, it
is guided by the step-level estimated value. The algorithm is detailed in
Algorithm 1.

Figure 2:  Difference between complete path verification and partial path
verification. Planning is the latter one, evaluate and selects paths based on
value estimation during inference.

##  4 Experiment Results and Analysis

###  4.1 Experimental settings

##### Benchmarks

We conduct experiments on two mathematical reasoning datasets, GSM8K (Cobbe et
al., 2021) and Game of 24 (Yao et al., 2023), in this paper. Each inference
experiment was repeated 3 times, with the average results and their standard
variation reported.

GSM8K is a grade school math word problem dataset consisting of 7473 training
questions and 1319 test questions, each with a single annotated solution path.
The problems require 2-11 reasoning steps, averaging around 3.5 steps. It is
classical for multi-step mathematical reasoning and also the pioneering to
employ the verifier approach in the complete path verification framework
(Cobbe et al., 2021).

Game of 24 is a mathematical reasoning challenge requiring models to combine
four numbers with basic operations (+-*/) to reach a total of 24. Its
expansive solution space, filled with numerous valid yet unproductive
combinations, makes it an ideal benchmark for evaluating planning
capabilities. Following Yao et al. (2023), we use relatively hard problems
index 901-1,000 for testing.

##### Label annotations

In GSM8K, the answer correctness is determined through precise string
matching, where all ground answers are integers. In Game of 24, the answer
correctness is based on the validity of the equation equating to 24 and the
singular usage of input numbers, following (Yao et al., 2023).

##### Baselines

We benchmark our method’s performance against the current state-of-the-art
models in GSM8K, and the notable Tree-of-Thought (ToT) in Game of 24.
Additionally, We compare the effectiveness of OVM planning with non-planning
version of the same model implemented by ours, including greedy, Self-
Consistency (SC) and Outcome-based Reward Model (ORM). For our method, we
report the best results across available beam size, e.g. the result of $k=20$
is the best one among $b=2,b=4,b=5,b=10$.

##### Implementation

We use the newline character as the marker for the end of each step.
Generators: In GSM8K, we fine-tuned Mistral-7B and Llama2-7B on the training
set. Given that GSM8K provides calculation annotations, our models were also
trained to utilize calculators. In Game of 24, we fine-tuned Llama2-7B on
simpler problems (indices 1-900) with enumerated solution paths. For both
datasets, the fine-tuning was carried out for 2 epochs, with a batch size of
128. We set the maximum learning rate to 1e-5, using a linear scheduler with
the AdamW optimizer. OVMs/ORMs 111Since we train the value model with outcome
supervision, the objective originally intended for ORM, the same model is
simultaneously used as OVM and ORM in this paper.: OVMs were initialized from
the corresponding generator checkpoints. We sampled 100 solution paths for
each question from the respective generator (resulting in 747,300 for GSM8K
and 90,000 for Game of 24), with a temperature setting of 0.7. In GSM8K, OVM
was trained for 1 epoch with a batch size of 512. In Game of 24, OVM is
trained for 10 epochs with a batch size of 128, due to its smaller training
set. The optimizer was AdamW and the maximum learning rate was set to 1e-5 for
Llama2-7B and 2e-6 for Mistral-7B respectively, following a linear scheduler.
During inference, we set the maximum number of steps as 10 and the maximum new
token length is 400. We implement FlashAttention for Llama2-7B (Dao et al.,
2022, Dao, 2023).

###  4.2 Overall Performance

##### Benchmark against the current state-of-the-art models

The overall performance of ours in GSM8K and Game of 24 are shown in Table 2
and Table 3 respectively. Significantly, our Mistral-based 7B model
outperforms all other models under 70B in GSM8K, while our Llama2-based 7B
model reaches the best among 7B models, excluding those based on Mistral.
Furthermore, in Game of 24, planning with OVM significantly improves the
performance of fine-tuned Llama2-7B, boosting its accuracy from a mere 11% to
an impressive 78.7%, with 20 sampled solution paths.

##### Comparison with non-planning

The comparisons between our planning with OVM and the non-planning
implementations are demonstrated in Table 4 and Figure 3. The results show
that planning with OVM can generally outperform complete path verification
using ORM when the number of sampled paths is equal. An exception is observed
with Mistral-7B at K=100, where the performance of both planning with OVM and
complete path verification tends to reach saturation.

Table 2: Accuracy on GSM8K. In the third column, we mark models that use GPT for inference or are trained with GPT-generated data. SC denotes ‘Self-Consistency’ and RM denotes ‘Reward Model’. Model | Size |  GPT-3.5/GPT-4 |  Data Augmentation | Accuracy
---|---|---|---|---
Open-Source Models without Code Execution
RFT (Yuan et al., 2023) | 7B |  |  | 51.2%
WizardMath (Luo et al., 2023) | 7B |  |  ✓ | 54.9%
MetaMath (Yu et al., 2023) | 7B |  ✓ |  ✓ | 66.4%
MuggleMATH (Li et al., 2023a) | 7B |  ✓ |  ✓ | 68.4%
Arithmo-Mistral | 7B |  ✓ |  ✓ | 74.7%
MetaMath-Mistral (Yu et al., 2023) | 7B |  ✓ |  ✓ | 77.7%
RFT (Yuan et al., 2023) | 13B |  |  | 55.3%
WizardMath (Luo et al., 2023) | 13B |  |  ✓ | 63.9%
MetaMath (Yu et al., 2023) | 13B |  ✓ |  ✓ | 71.0%
MuggleMATH (Li et al., 2023a) | 13B |  ✓ |  ✓ | 74.0%
RFT (Yuan et al., 2023) | 70B |  |  | 64.8%
WizardMath (Luo et al., 2023) | 70B |  |  ✓ | 81.6%
MuggleMATH (Li et al., 2023a) | 70B |  ✓ |  ✓ | 82.3%
MetaMath (Yu et al., 2023) | 70B |  ✓ |  ✓ | 84.3%
Ours – OVM (Llama2-7B, K=100) | 7B |  |  | 73.7% ± 0.4%
Ours – OVM (Mistral-7B, K=100) | 7B |  |  | 84.7% ± 0.3%
Open-Source Models with Code Execution
ToRA-Code (Gou et al., 2023) | 7B |  ✓ |  ✓ | 72.6%
ToRA-Code (Gou et al., 2023) | 13B |  ✓ |  ✓ | 75.8%
ToRA-Code (SC, K=50) (Gou et al., 2023) | 34B |  ✓ |  ✓ | 85.1%
ToRA (Gou et al., 2023) | 70B |  ✓ |  ✓ | 84.3%
ToRA (SC, K=50) (Gou et al., 2023) | 70B |  ✓ |  ✓ | 88.3%
Closed-Source Models
PaLM (SC, K=32) (Huang et al., 2022) | 540B |  |  | 82.1%
DeepMind (RM, K=96) (Uesato et al., 2022) | 70B |  |  | 87.3%
Codex (RM, K=80) (Xie et al., 2023) | 175B |  ✓ |  | 85.5%
GPT-4 (Bubeck et al., 2023) | - |  ✓ |  | 87.1%
GPT-4 Code+Self-Verification (K=5) (Zhou et al., 2023) | - |  ✓ |  | 97.0%
Table 3: Accuracy on Game of 24. GPT-4’s accuracy are from Yao et al. (2023), and K of ToT is estimated from the Figure 3 in their paper. | Accuracy
---|---
GPT-4 CoT | 4.0%
GPT-4 SC (K=100) | 9.0%
GPT-4 ToT (K=60) | 74.0%
Fine-tuned Llama2-7B | 11.0%
Ours – OVM (Llama2-7B, K=20) | 78.7% ± 1.7%
Table 4: Accuracy on GSM8K and Game of 24. Average and standard variation of 3 runs for multiple path verification are reported. K denotes sampling size. Method | GSM8K | Game of 24
---|---|---
|  |  | Llama2-7B | Mistral-7B | Llama2-7B
Non-Planning | Greedy | 38.6% | 58.4% | 11%
SC | K=20 | 53.3% ± 0.3% | 70.2% ± 0.7% | 10.3% ± 1.7%
K=100 | 57.4% ± 0.8% | 72.6% ± 0.2% | 11.7% ± 1.3%
ORM | K=20 | 65.5% ± 0.7% | 81.8% ± 0.2% | 65.3% ± 5.3%
K=100 | 71.9% ± 0.6% |  84.7% ± 0.4% | 95.3% ± 0.5%
Planning | OVM | K=20 | 69.0% ± 0.4% | 82.6% ± 0.1% | 78.7% ± 1.7%
K=100 | 73.7% ± 0.4% |  84.7% ± 0.3% |  98.3% ± 1.2%

(a) GSM8K (b) Game of 24

Figure 3:  The relationship between the sampling size and accuracy (Llama2-7B)

##  5 Analysis and Discussion

The sections seek to answer the following two Research Questions (RQs).

######  RQ 1.

Can OVM Plan?

######  RQ 2.

When planning benefits more?

The two research questions are crucial to understand the capability of OVM in
estimating furture outcomes for planning and the specific contexts or
scenarios in which planning is most advantageous or effective. We provide some
analysis and discussion for the two questions in Sec. 5.1 and Sec. 5.1
respectively.

###  5.1 RQ1: Can OVM Plan?

##### Evaluation with correct answer proportion

To evaluate the effectiveness of our method’s planning, we measure the
proportion of sampled solution paths with correct answers in the final
sampling stage, just before selecting the final solution. This offers insights
into the accuracy of planning in guiding to correct answers. We compare the
planning performance against direct generation without planning, where the
generator randomly samples solutions based on linguistic probabilities,
denoted as “vanilla sampling”.

##### OVM is an effective planner guiding to the correct answers

We demonstrate the correct answer proportion of OVM planning in comparison
with vanilla sampling in Figure 4. Notably, in GSM8K, less than 35% of the
generator’s randomly sampled solution paths are correct, and this proportion
increases to over 65% with OVM planning. Similarly, in the Game of 24, OVM
planning significantly boosts the correct answer proportion from just 10

(a) GSM8K (b) Game of 24

Figure 4:  The relationship between the sampling size and correct answer
proportion (Llama2-7B)

##### Comparison with reward models

We further compare the performance of reward models and OVM in guiding
generator’s decoding in Game of 24. 222This dataset is simple enough for us to
create step-level annotations automatically using rules. Here we train the
reward models with process supervision, denoted as PRM (Process Reward Model).
We investigate two types of step-level process signals for training 333PRMs
are trained with the same hyper-parameters as OVM.: (1) Steps are rewarded
(i.e. labelled as 1) for logical correctness, (2) Steps are rewarded only if
logically correct and contributes to the correct final answer. The model
trained by the latter rewards is referred to as PRM-O since it implicitly
considers future Outcomes. The result presented in Figure 5 shows that both
OVM and PRM-O can effectively guide towards correct answers, in contrast to
PRM’s failure, highlighting the importance of anticipating outcomes. Notably,
while PRM-O demonstrates quicker convergence than OVM, OVM eventually matches
PRM-O’s performance. This suggests that outcome supervision, relying solely on
final labels, may suffice for models to learn value estimation, though more
detailed step-level rewards can accelerate this learning process.

Figure 5: The comparison between OVM, PRM, and PRM-O in correct answer
proportion in Game of 24 (K=20)

###  5.2 RQ2: When planning benefits more

In this subsection, we conduct experiments to observe the impact of problem
difficulty and the generator’s capability on the benefits derived from
planning. We compare the planning gains of Llama2-7B and Mistral-7B in GSM8K.
First, we define a metric to quantify the degree of improvement a generator
achieves through planning. Then, we give our findings and analysis.

######  Definition 3.

planning gain: relative improvement in correct answer proportion The planning
gain is defined by the gain ratio in correct answer proportion of planning
versus vanilla sampling. Let $p_{planning}$ represent the correct answer
proportion of planning, and $p_{vanilla}$ denote that of vanilla sampling. The
planning gain is $(p_{planning}-p_{vanilla})/p_{vanilla}$.

######  Findings 1.

Planning shows greater improvement in longer-step problems

See results in Figure 6. Overall, both models generally gain more from
planning as problem complexity rises (require more steps to solve) within a
certain range—up to 6 steps for Llama2-7B and 8 steps for Mistral-7B. This
trend is likely due to the exponential increase in potential solution paths in
vanilla sampling as the number of required steps grows. Planning addresses
this by effectively guiding the generation, leading to more gains in longer-
step problems.

######  Findings 2.

Model with a relatively moderate capability may benefit more from planning.

Firstly, stronger models may benefit less from planning. As shown in Figure 6,
Mistral-7B, likely stronger in short-path problems (with correct answer
proportion more than half within 6 steps), gains less from planning than
Llama2-7B in problems solvable within 7 steps. This is intuitive; a stronger
model can easily produce correct answers through vanilla sampling, reducing
the need for planning. Secondly, weaker models may struggle to benefit from
planning. In the same figure, Mistral-7B shows more gain than Llama2-7B in
8-step problems, but both models have negative effects when planning with OVM
in problems over 8 steps. This could be due to OVM’s insufficient training on
more complex problems, as these are less represented in the training set (with
just 13 questions involving more than 8 steps). This implies that both very
capable models (like Mistral-7B in shorter problems) and relatively weaker
ones (like Llama2-7B in 8-step problems and both models in the longest
problems) see diminished returns from planning. However, this interpretation
must be considered cautiously, given the small number of longest-step problems
in the test set (only 3 over 8 steps), potentially leading to biased results.
Further in-depth exploration of this issue is necessary in future studies.

Figure 6: The relationship between the problem difficulty and planning gain in
GSM8K (K=20)

##  6 Related Works

##### Verification in mathematical reasoning

Mathematical reasoning poses challenges in both multi-step reasoning and
arithmetic calculation. The task’s complexity arises from the ease of making
mistakes at each step, which can influence subsequent steps and final answers.
Since autoregressive models cannot correct already-made errors, verification
plays a crucial role in ensuring performance accuracy (Shen et al., 2021,
Cobbe et al., 2021, Weng et al., 2022, Zhou et al., 2023). It has been found
that large language models can self-verify by prompting them with natural
languages (Weng et al., 2022, Xie et al., 2023, Zhou et al., 2023) and even
self-refine their responses (Madaan et al., 2023). Another way of verification
is to train a specific model, which is called as verifier (Cobbe et al., 2021,
Uesato et al., 2022, Li et al., 2022, Khalifa et al., 2023b). Towards how to
train a verifier, there’s a debate between outcome and process supervision,
with recent trends favoring process supervision (Uesato et al., 2022, Lightman
et al., 2023). In this paper, we explore the potential of outcome supervision
in planning.

##### Guided decoding in mathematical reasoning

During decoding, the step can be evaluated using either rewards (Khalifa et
al., 2023b, Hao et al., 2023, Xie et al., 2023) or values (Yao et al., 2023,
Hao et al., 2023, Feng et al., 2023). Rewards can be defined based on logical
correctness of a step (Hao et al., 2023, Xie et al., 2023) or its helpfulness
(Hao et al., 2023). There are two ways of value estimation: simulation-based
and model-based. Simulation-based approaches look ahead, sample answers, and
estimate value from them (Yao et al., 2023, Hao et al., 2023). Model-based
methods train a value model for prediction during inference (Feng et al.,
2023). Additionally, it is popular to apply Monte Carlo Tree Search to balance
exploration and exploitation during planning (Hao et al., 2023, Feng et al.,
2023). In our paper, we just employ the simple beam search strategy.

##  7 Conclusion

In conclusion, this paper presents a novel approaches in verifying
intermediate steps and guiding model generation. This is achieved through the
introduction of the Outcome-based Value Model (OVM), which employs outcome
supervision in training a value model for intermediate steps. Both theoretical
and empirical evidence highlight the effectiveness of outcome supervision for
value estimation in planning, offering a method that is more efficient and
effective than process supervision, which results in a reward-based model. The
OVM, requiring no costly step-level annotations and fewer sampled paths,
demonstrates superior performance in complex multi-step reasoning tasks, as
evidenced by its state-of-the-art results on GSM8K and impressive success rate
improvement in the Game of 24.

## References

  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. 2023.  Sparks of artificial general intelligence: Early experiments with GPT-4.  _CoRR_ , abs/2303.12712.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.  Training verifiers to solve math word problems.  _CoRR_ , abs/2110.14168.
  * Creswell et al. (2023) Antonia Creswell, Murray Shanahan, and Irina Higgins. 2023.  Selection-inference: Exploiting large language models for interpretable logical reasoning.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Dao (2023) Tri Dao. 2023.  Flashattention-2: Faster attention with better parallelism and work partitioning.  _CoRR_ , abs/2307.08691.
  * Dao et al. (2022) Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  In _NeurIPS_.
  * Feng et al. (2023) Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. 2023.  Alphazero-like tree-search can guide large language model decoding and training.  _CoRR_ , abs/2309.17179.
  * Gou et al. (2023) Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. 2023.  Tora: A tool-integrated reasoning agent for mathematical problem solving.  _CoRR_ , abs/2309.17452.
  * Hao et al. (2023) Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. 2023.  Reasoning with language model is planning with world model.  _CoRR_ , abs/2305.14992.
  * Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022.  Large language models can self-improve.  _CoRR_ , abs/2210.11610.
  * Khalifa et al. (2023a) Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. 2023a.  Discriminator-guided multi-step reasoning with language models.  _CoRR_ , abs/2305.14934.
  * Khalifa et al. (2023b) Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. 2023b.  Discriminator-guided multi-step reasoning with language models.  _CoRR_ , abs/2305.14934.
  * Li et al. (2023a) Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. 2023a.  Query and response augmentation cannot help out-of-domain math reasoning generalization.  _CoRR_ , abs/2310.05506.
  * Li et al. (2022) Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2022.  On the advance of making language models better reasoners.  _CoRR_ , abs/2206.02336.
  * Li et al. (2023b) Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. 2023b.  Making language models better reasoners with step-aware verifier.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 5315–5333. Association for Computational Linguistics.
  * Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023.  Let’s verify step by step.  _CoRR_ , abs/2305.20050.
  * Luo et al. (2023) Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.  _CoRR_ , abs/2308.09583.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. 2023.  Self-refine: Iterative refinement with self-feedback.  _CoRR_ , abs/2303.17651.
  * Press et al. (2022) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022.  Measuring and narrowing the compositionality gap in language models.  _CoRR_ , abs/2210.03350.
  * Shen et al. (2021) Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021.  Generate & rank: A multi-task framework for math word problems.  In _Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021_ , pages 2269–2279. Association for Computational Linguistics.
  * Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed Chi, Denny Zhou, and Jason Wei. 2023.  Challenging big-bench tasks and whether chain-of-thought can solve them.  In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 13003–13051. Association for Computational Linguistics.
  * Uesato et al. (2022) Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song, Noah Y. Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022.  Solving math word problems with process- and outcome-based feedback.  _CoRR_ , abs/2211.14275.
  * Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023.  Self-consistency improves chain of thought reasoning in language models.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  In _NeurIPS_.
  * Weng et al. (2022) Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2022.  Large language models are better reasoners with self-verification.  _CoRR_ , abs/2212.09561.
  * Xie et al. (2023) Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. 2023.  Decomposition enhances reasoning via self-evaluation guided decoding.  _CoRR_ , abs/2305.00633.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.  _CoRR_ , abs/2305.10601.
  * Yu et al. (2023) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023.  Metamath: Bootstrap your own mathematical questions for large language models.  _CoRR_ , abs/2309.12284.
  * Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. 2023.  Scaling relationship on learning mathematical reasoning with large language models.  _CoRR_ , abs/2308.01825.
  * Zhou et al. (2023) Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2023.  Solving challenging math word problems using GPT-4 code interpreter with code-based self-verification.  _CoRR_ , abs/2308.07921.

##  Appendix A Train a Verifier

##### Model architecture

The model is a decoder-only autoregressive language model. Following Cobbe et
al. (2021), we add a linear layer with a single bias parameter after the
model’s final unembedding layer.

##### Training set

Given a training question $q$, the generator is queried to generate $n$
solution paths $\mathbb{S}=\\{S_{1},\cdots,S_{n}\\}$. The training labels are
based on either outcome supervision or process supervision, where the former
one can get the labels automatically via rules like string matching to the
ground answer while the latter one is more labor-intensive.

##### Outcome supervision

The label is dependent on the final answer $a$, which is binary
$y_{o}\in\\{0,1\\}$, indicating whether $a$ is correct or not. Since it is
often trained with token-level supervision, the label is expanded into a
consistent vector $\bm{y}^{o}=[y_{o},\dots,y_{o}]$ of token sequence length.

##### Process supervision

The label is a vector with each dimension indicting whether the corresponding
step is correct or not, i.e. $[y_{1},\dots,y_{m}]$, where $m$ is the number of
steps. After expanding into token level, the label becomes
$\bm{y}^{p}=[y_{1},\dots,y_{1},\dots,y_{m},\dots,y_{m}]$, which assigns the
same label $y_{i}$ to all tokens within the i-th step.

##### Training objective

The objective is a MSE loss,

$L(S,\bm{y}^{l};q)=\left(f(q;S)-\bm{y}^{l}\right)^{2},\quad l=o\text{ or }p$

Additionally, the model is jointly trained with language modeling loss
unweighted, following Cobbe et al. (2021).


# Interactive Learning of Physical Object Properties Through Robot
Manipulation and Database of Object Measurements

Andrej Kruzliak1, Jiri Hartvich1, Shubhan P. Patni1, Lukas Rustler1, Jan
Kristof Behrens2, Fares J. Abu-Dakka3,
Krystian Mikolajczyk4, Ville Kyrki5, and Matej Hoffmann1 1Department of
Cybernetics, Faculty of Electrical Engineering, Czech Technical University in
Prague<EMAIL_ADDRESS>Institute of Informatics, Robotics,
and Cybernetics, Czech Technical University in Prague3Electronic and Computing
Department, Faculty of Engineering, Mondragon Unibertsitatea, 20500 Arrasate,
Spain4Department of Electrical and Electronic Engineering, Imperial College
London, London, UK5Intelligent Robotics Group, Department of Electrical
Engineering and Automation, School of Electrical Engineering, Aalto
UniversityThis work was co-funded by the European Union under the project
ROBOPROX (reg. no. CZ.02.01.01/00/22_008/0004590). S.P.P. and L.R. were
additionally supported by the Grant Agency of the CTU in Prague (no.
SGS24/096/OHK3/2T/13). This work originated in the project Interactive
Perception-Action-Learning for Modelling Objects (IPALM) (H2020-FET-ERA-NET
Cofund-CHIST-ERA III / TAČR EPSILON, No. TH05020001). We would like to thank
Tran Nguyen Le for a first code prototype capturing the relationships between
some objects properties.

###### Abstract

This work presents a framework for automatically extracting physical object
properties, such as material composition, mass, volume, and stiffness, through
robot manipulation and a database of object measurements. The framework
involves exploratory action selection to maximize learning about objects on a
table. A Bayesian network models conditional dependencies between object
properties, incorporating prior probability distributions and uncertainty
associated with measurement actions. The algorithm selects optimal exploratory
actions based on expected information gain and updates object properties
through Bayesian inference. Experimental evaluation demonstrates effective
action selection compared to a baseline and correct termination of the
experiments if there is nothing more to be learned. The algorithm proved to
behave intelligently when presented with trick objects with material
properties in conflict with their appearance. The robot pipeline integrates
with a logging module and an online database of objects, containing over
24,000 measurements of 63 objects with different grippers. All code and data
are publicly available, facilitating automatic digitization of objects and
their physical properties through exploratory manipulations.

ACTION
---
SELECTION
---
WEIGH
---
SQUEEZE
---
MATERIAL
---
PRIOR BELIEF
---
MATERIAL
---
SOUND
---
CATEGORY
---
VISION
---
VISION
---
Figure 1: Exploratory action selection to estimate object properties –
schematic overview. Using the current belief, next action is selected to
maximize the expected Information Gain (IG), taking the uncertainty of each
action into account. After each action, new belief is computed using inference
of Bayesian Network, and new action is selected.

##  I Introduction

While rapid progress is being made in automatically extracting information
about objects from large image datasets and text corpora, estimating physical
properties remains a challenge. This work paves the way to automatic
extraction of physical object properties such as material composition, mass,
volume or stiffness, which typically require physically manipulating the
object. We present a theoretical framework and its experimental evaluation
that deals with exploratory action selection to maximize what can be learned
about objects on the table in front of a robot manipulator. In our
demonstrator, there is a partially adversarial mix of 17 household objects of
different shapes and from different materials (incl. a real and a plastic
banana, wine glass from glass and plastic, a soft and a hard sponge with
identical appearance), 6 object properties (2 discrete – category and
material; 4 continuous – elasticity, density, volume, and mass), and 5 actions
the robot system can perform (2 based on vision – estimate the category or
material from the image; 3 manipulation actions – squeeze, poke, and lift). We
expressed the conditional dependencies between the properties using a Bayesian
Network (BN) (see Fig. 2 and added freely available prior probability
distributions as well as uncertainty associated with different measurement
actions. After specifying which object property should be estimated, our
algorithm chooses the best exploratory action based on expected IG, updates
all object properties using inference in the BN, and picks the next optimal
action to be executed. We demonstrate that the overall average performance of
the system outperforms a baseline and analyze interesting runs showing how the
algorithm automatically and efficiently discovers object properties. The robot
pipeline is integrated with a logging module and an online database of
objects, currently containing more than 24000 measurements of 63 different
objects using four different grippers. We make publicly available all code and
data, which provides a starting point for automatic digitization of objects
and their physical properties by exploratory manipulations.

Task and motion planning for robot manipulation, including sequences of
actions, has been extensively studied using symbolic and recently also neuro-
symbolic or neural approaches. In this work, the goal is not to perform a task
(e.g., stack objects) but to learn about the properties of objects. Embodied
reasoning for discovering object properties via manipulation was used in [1];
correct execution of a verbally specified task (e.g., stack objects by weight)
required specific actions to learn about physical object properties. In this
work, the order of actions is chosen not as to produce a cumulative result in
the environment, but so that the action chosen next is optimal with respect to
what can be learned about the object in front of the robot.

Active manipulation for perception can be non-prehensile (e.g. “push to know”
[2, 3]) or prehensile, i.e. grasping the object or exploring it with robot
fingers (e.g., [4]). In most cases, manipulation serves object localization,
shape reconstruction, or recognition; learning physical object properties has
been less studied [5]. Haptic exploration of physical object properties using
several actions and sensory modalities is extremely rare ([6] a notable
exception). The exploratory movements are typically chosen based on some form
of probabilistic model and information theoretic measures like entropy or IG
(see e.g., [7]).

An accompanying video is at https://youtu.be/h_ZIYUmzv-8; the code used in
this work is available at https://github.com/ctu-vras/actsel; The object
database and source code are available at https://cmp.felk.cvut.cz/ipalm/.

##  II Exploratory Action Selection Driven by Information Gain

Our method is based on a Bayesian Network (BN). Beliefs about each of the
variables can be updated through measurements. The action-to-node relation can
be found in Fig. 2. The BN consists of five nodes ordered by their
enumerability types.  (i) categorical (enumerable): Category and Material;
(ii) continuous (non-enumerable): Elasticity, Density, and Volume.  Discrete
variables are represented with a Probability Mass Function (PMF); continuous
variables with a Probability Density Function (PDF).

We consider 10 object Categories: bottle, bowl, box, can, dice, fruit, mug,
plate, sodacan, wineglass.

We consider 8 Material classes: ceramic, glass, metal, soft plastic, hard
plastic, wood, paper, foam.

Elasticity is a continuous variable expressed in Young’s modulus [kPa],
Density in homogeneous density [kg/$m^{3}$], and Volume in the volume of the
object’s material in [m3].

Figure 2: Bayesian Network Structure.

###  II-A Initialization

To establish initial beliefs for each property, first, a uniform PMF is set up
for Category and Material. Those in turn initialize each continuous variable
expressed by the relationships in the BN (Fig. 2), as shown in Fig. 3, giving
rise to a Gaussian Mixture Model (GMM) for each continuous variable.

Figure 3: Initialization of GMM from the reference data by weighting by
corresponding probability mass function.

The Category PMF is translated into the Volume PDF through the values in Table
I. The Material PMF is translated into the Elasticity and Density PDFs through
the means and Standard Deviations in Table II, extracted from technical
literature [8, 9, 10, 11].

---
Volume [cm3] |  Bottle |  Bowl |  Box |  Can |  Dice
$30\pm 50$ |  $40\pm 80$ |  $10\pm 26$ |  $8\pm 60$ |  $37\pm 50$
Fruit |  Mug |  Plate |  Sodacan |  Wineglass
$155\pm 28$ |  $97\pm 27$ |  $0\pm 15$ |  $5\pm 10$ |  $85\pm 20$

TABLE I: Reference values for Volume. The values are means and Standard
Deviations (SDs) scraped from an online store.

---
|  Ceramic |  Glass |  Metal |  Soft Plastic
Density [kg$\cdot$ m-3] |  $2300\pm 100$ |  $2600\pm 200$ |  $7900\pm 600$ |  $1300\pm 300$
Elasticity [kPa] |  $150\pm 10$ |  $140\pm 10$ |  $130\pm 10$ |  $55\pm 30$
|  Hard Plastic |  Wood |  Paper |  Foam
Density [kg$\cdot$ m-3] |  $1500\pm 300$ |  $646\pm 183.6$ |  $756.8\pm 223.8$ |  $91.9\pm 127.7$
Elasticity [kPa] |  $75\pm 30$ |  $120\pm 10$ |  $35\pm 28$ |  $10\pm 15$

TABLE II: Reference values for density and elasticity. The values are means
and Standard Deviations (SDs) extracted from the literature.

The conditional dependence between Category and Volume as well as the
covariance matrix linking Category and Material were initialized using data
extracted from the text descriptions of products available at an online store.

###  II-B Inference

Following the execution of an action, the network is updated based on the
obtained results (measurement update). These data points are then incorporated
as a product into the distribution of the given property. For categorical
properties, the posterior distribution post-update is propagated to all other
nodes in the BN, adhering to the network’s connections as an undirected graph.
This updating process, known as message passing, involves transmitting message
vectors between nodes. The format and order of these message vectors
correspond to the relevant probability functions: for interactions between
nodes related to Material, the format aligns with the material PMF, while for
other interactions, it aligns with the category PMF.

When the message originates from a continuous property node, such as
Elasticity, the continuum must be first transformed into a finite vector to
align with the required message format. Given our knowledge of the original
reference distribution of materials for Density, we only need to estimate the
weights of the GMM using the Expectation–maximization (EM) algorithm. This
approach allows the classification of unknown objects into material categories
without necessitating a class for every known material. Consequently, objects
that fall between two classes can be represented, for instance, as a blend,
like $0.5\times property_{1},0.5\times property_{2}$. Subsequently, this
message vector from Elasticity serves as a virtual measurement for the
Material node. Posterior probabilities are computed using a confusion matrix
to simulate the uncertainty inherent in message passing. Specifically, this
confusion matrix is unique to each connection between nodes; it is computed
beforehand at the initialization step from the reference data for every blue
arrow in the network model depicted in Fig. 2.

When the message originates from a categorical property, i.e., a vector, the
estimation step is bypassed, and message passing occurs directly from the
product of the measurement vector and the associated confusion matrix.

###  II-C Measurement Models

The action selection algorithm needs to estimate the consequences of
performing different actions, i.e. measurements. A measurement model, i.e. the
accuracy of every measurement is required. For categorical variables,
confusion matrices are used to compute the expected measurement. For brevity,
only the average accuracy for each categorical action is given. Complete
confusion matrices are available in the code repository. For continuous
variables, a measurement is modeled as a normal distribution. The measurement
accuracies and SDs are provided in the Table III. The SD for density was
experimentally obtained based on the average performance of extracting a
correct density belief from a mass measurement and a volume belief on the
available experimental data. The mass measurement with our experimental setup
has an accuracy of $\pm$ 6 [grams] for up to a 1 [kg] payload.

Action | Target node | Accuracy |  SD $\sigma$ | Units
---|---|---|---|---
mat-vision | Material (categorical) | 0.2431 | - | -
mat-sound | Material (categorical) | 0.1002 | - | -
cat-vision | Category (categorical) | 0.6345 | - | -
weighing | Density (continuous) | - | 223 | [kg$\cdot$m-3]
squeezing | Elasticity (continuous) | - | 10 | [kPa]

TABLE III: Measurement accuracy. Accuracies and Standard Deviations (SDs) from
experimental data.

###  II-D Expected Information Gain

In our framework, IG quantifies the reduction in entropy of the probability
distribution of one or more variables in our BN after executing an action. It
reflects how much information the action and its associated measurement have
provided. When we compute the difference in entropy before and after
conducting an action, we refer to it as experimental or true IG. While
experimental IG offers insight into the performance of an algorithm, by
definition, it alone cannot be used to decide which action to take. Therefore,
to guide the decision-making, one must create an emulation—an estimation of
how the property distribution will likely appear after each action—to choose
the action most likely to reduce entropy, thus increasing IG. The experimental
IG is formalized as:

$\Delta H(\nu)=H(\nu_{k})-H(\nu_{k+1}),$ (1)

where $\nu$ represents a distribution of a node, either continuous or
categorical (e.g., Elasticity: PDF or Category: PMF), $\Delta H(\nu)$ denotes
the experimental IG, which is the difference in entropies. Here,
$H(\nu{{}_{k}})$ is the entropy of a node distribution at algorithm iteration
$k$, representing the prior, and $H(\nu_{k+1})$ is the entropy of the same
node distribution at the next iteration step $k+1$, representing the posterior
after the measurement. The expected IG is formalized as follows

$\mathbb{E}[\Delta H(\nu)]=H(\nu_{k})-\mathbb{E}[H(\bar{\nu_{k}})],$ (2)

where $\mathbb{E}[\Delta H(\nu)]$ is the expected IG, and
$\mathbb{E}[H(\bar{\nu_{k}})]$ is the expected entropy of the posterior
distribution, which is the entropy of the emulation. Note that the emulation
also contains the index $k$, not $k+1$, to emphasize that it reflects only an
expectation of the posterior based on the current belief. For the continuous
properties, theoretically, the emulation could involve sampling infinitely
many possible scenarios of the action measurements based on the probability
density function of a given property. However, such an approach would require
simulating Bayesian updates for a large number of samples, which consumes
considerable CPU time. Since the variance of the action related to a given
property is available, we chose a more efficient option—convolving the
continuous prior probability density function of a given property $\nu_{k}$
with a normal distribution having a zero mean and variance equal to the
variance of the action measurement in question
$\mathcal{N}(0,\sigma_{a}^{2})$. This procedure propagates the expected error
across the entire continuous probability density function of the property.
Formally:

$\bar{\nu_{k}}=\nu_{k}\ast\mathcal{N}(0,\sigma_{a}^{2}).$ (3)

Subsequently, the expected continuous entropy is numerically calculated as:

$\mathbb{E}[H(\bar{\nu_{k}})]=-\sum_{\mathcal{X}}h_{\mathcal{X}}\bar{\nu_{k}}(x)\log_{2}\bar{\nu_{k}}(x),$
(4)

where the entropy is computed for each step on the support $\mathcal{X}$ of
the property distribution. $h_{\mathcal{X}}$ denotes the normalization factor,
determined by the bin size used for computing the sum-integral approximation.
Strictly mathematically speaking, the Eq. (4) provides an integral
approximation.

To represent the variance of the measurements for categorical properties, such
as the Category vision module, the confusion matrix is employed. From this
confusion matrix, a sample entropy vector is computed to create a categorical
equivalent of the emulation in Eq. (3). Each element of the sample entropy
vector at index $i$ is computed as

$H(\bar{\nu_{k}}|m_{i})=-\sum_{j}C_{a}(i,j)\log_{2}C_{a}(i,j),$ (5)

where $C_{a}$ is a row-unit-sum confusion matrix corresponding to action $a$,
$m_{i}$ signifies the $i$-th member of the categorical property, e.g. ceramic
for the Material node or a mug for the Category node. In the case of a column-
unit-sum confusion matrix, the indices $(i,j)$ in Eq. (5) would interchange.
To compute the expected entropy of the emulation, we use the resultant vector
from Eq. (5) as:

$\mathbb{E}_{m}[H(\bar{\nu_{k}})]=\sum_{m_{i}}C_{a}\nu_{k}(m_{i})H(\bar{\nu_{k}}|m_{i}),$
(6)

where the result is a sum of the sample entropy vector weighted by an expected
measurement computed as a matrix multiplication of the prior distribution and
confusion matrix as $C_{a}\nu_{k}$. The result Eq. (6) is then substituted
into Eq. (2) to obtain the expected IG.

###  II-E Action Selection Algorithm

The approach is schematically illustrated in Fig. 1 and the pseudo-algorithm
is shown in Alg. 1. Formally, the optimal exploratory action is chosen as:

$a^{*}=\text{argmax}_{a\in\mathcal{A}}\mathbb{E}[\Delta H(\nu|a)],$ (7)

where $a$ is a specific action from the total set of available actions
$\mathcal{A}$, and $a^{*}$ is the optimal action that maximizes the expected
IG.

1

2for  _k in iterations_ do

3 $\eta_{k}:=\text{getNetworkEntropy}(N)$

4 for  _a $\in\mathcal{A}$_ do

5 $\nu_{k,a}:=\nu_{k}\in N\;|\;a\rightarrow\nu_{k}$

6 $\overline{\nu_{k}}:=\text{emulatePosterior}(\nu_{k,a},a)$

7 $\mu_{k}:=\text{generateMessages}(\overline{\nu_{k}})$

8 $\overline{N}:=\text{propagateMessages}(N,\mu_{k})$

9 $\overline{\eta_{k}}:=\text{getExpectedNetworkEntropy}(\overline{N})$

10 $\mathbb{E}_{k}[\Delta H(N)]:=\eta_{k}-\overline{\eta_{k}}$

11

12 end for

13 $a_{k}^{*}\in\textit{argmax}_{a\in\mathcal{A}_{k}}\mathbb{E}_{k}[\Delta
H(N)]$

14

15 end for

Algorithm 1 Action selection (ACSEL)

##  III Objects, Robot Setup, Action Repertoire

The setup and robot actions are illustrated in the accompanying video at
https://youtu.be/h_ZIYUmzv-8.

###  III-A Manipulated Objects

We selected 17 objects, most of them from the YCB dataset [12], to be diverse
in size, weight, volume, and material composition—see Fig. 4. We also included
“trick” or adversarial objects where their material composition does not
correspond to the best guess from their appearance (real and plastic bananas,
a soft sponge and an identical but hardened sponge, wine glass from glass or
plastic).

Figure 4: All 17 objects used and experimental setup. Kinova Gen3 robot,
Robotiq 2f-85 gripper, Rode VideoMic Pro microphone, and RealSense D435 camera
(on the right; only the RGB from one camera was used).

###  III-B Robot Setup

The real-world robot setup (Fig. 4) consisted of a Kinova Gen3 robotic
manipulator (7 degrees of freedom), equipped with a Robotiq 2f-85 gripper used
for squeezing and weighing actions. In addition, the gripper was used to poke
objects to generate sounds for material classification (mat-sound). For this
purpose, a professional shotgun microphone Rode VideoMic Pro was used. RGB
images from a RealSense D435 camera were used for the measurements relying on
vision (cat-vision, mat-vision).

###  III-C Action Repertoire

Weigh. The mass of objects is estimated by lifting them using the robot arm
using torque measurements in the robot’s joints in a configuration where a
certain robot joint axis is perpendicular to the force of gravity. The torque
in this joint $\tau_{m}$ can be described by the formula
$\tau_{m}=mgl+\tau_{0}$, where $m$ is the object mass, $l$ is the effective
lever from the joint axis to the object center of mass, and $g$ is the
gravitation constant, and $\tau_{0}$ is a reference torque recorded by moving
the robot to the measuring pose while the gripper is empty.

Squeeze. Grasping or squeezing the object between the jaws of a parallel
gripper provides force and displacement data—the stress-strain response of
every object. We adapted the module we developed in [13] and approximated the
stress-strain curve with a line—a coarse approximation of the object’s
stiffness.

Material from sound. We used sound generated by poking the object with the
closed gripper jaws to recognize the material. Audio recordings from the
interactions were preprocessed and converted to grayscale melodic spectrograms
and then fed into a modified ResNet34 network, with output labels representing
the material classes. More details can be found in [14].

Material and Category from Vision. We used the Detectron2 algorithm [15] with
R50-FPN backbone and trained the model using our dataset consisting of objects
from the YCB dataset [12], SHOP-VRB [16] and our own generated in
Unity.111Dataset available at https://github.com/Hartvi/ipalm-sponges. Code
available at https://github.com/Hartvi/Detectron2-mobilenet.

##  IV Experiments and Results

In our experiments, one of 17 objects was placed in front of the robot. There
were 5 actions or measurements (cat-vision, mat-vision, mat-sound, weighing,
squeezing) in the repertoire that could be selected by our algorithm based on
their expected IG, i.e. their likelihood to maximally reduce the uncertainty
in one or more of the object properties. There were five object properties (2
discrete variables with PMF – Category and Material; 3 continuous with PDF –
Elasticity, Density, Volume/Mass)—see Fig. 2 for an overview. The
action/measurement informing the Category node was cat-vision; mat-vision or
mat-sound could be employed to update the Material node; squeezing measured
Elasticity. Density and Volume nodes did not have a corresponding measurement
that would update them directly and relied on relationships within the BN.

The actions were selected “without replacement”, i.e. once a particular action
was chosen, it was not available in the next step of the algorithm. Thus, with
5 actions in the repertoire, there were at most 5 algorithm steps. After every
measurement, the BN was updated as explained in Section II-B. We ran two
variants of the experiments: (a) with no stopping criterion—5 actions were
always executed; (b) with stopping—if at a given algorithm step, there was no
action available with positive expected IG, the algorithm would terminate.
There were 5 runs of the algorithm for every object.

Optimization modes. To drive the action selection, one of the nodes could be
selected for which the uncertainty should be reduced (say the user is
interested in a specific object property). For Category or Material, the
expected discrete IG for every action was be evaluated and the best action
executed. For the continuous variables (Elasticity, Density, Volume), the
corresponding differential IGs were evaluated. We have also assessed runs
optimizing Category and Material together or the three continuous property
nodes together. Aiming to reduce uncertainty in all five nodes at the same
time would require adding the discrete and differential IG which is not
mathematically rigorous, so this regime was not evaluated. Please note that
all actions have their associated measurement models, i.e. the system has an
estimate of the noise associated with every action. Thus, optimizing one
property does not always lead to executing the action that directly measures
that property.

###  IV-A Qualitative Illustration of the System’s Operation

The operation of the system is illustrated in Fig. 5 on three selected
objects. In the first row (Panels A1-A4), the YCB wineglass, which is plastic,
is explored while the Category variable is optimized. Cat-vision has the
highest expected IG (panel A2) and is always chosen (panel A3). In the single
run (panel A2), additional steps of the algorithm are shown. However, in this
particular run, if termination was on, no more action would be executed as all
actions have negative expected IG at step 1. Panel A4, right, shows the
correct identification of the object Category as wineglass.

The other two rows show exploration of a sponge—a hardened exemplar in the
middle (B1-B4) and a soft one at the bottom (C1-C4) (see accompanying video at
https://youtu.be/h_ZIYUmzv-8);, in both cases with learning about Elasticity
as the objective. Although the goal was to learn about the object stiffness,
Cat-vision was chosen as the first most informative action to serve this
objective. The squeezing action executed in the next step (see B2 and B3) adds
information about the object being stiff, as is apparent in the final
elasticity distribution as well as the material distribution (B4 vs. C4).

Figure 5: System operation and performance – examples. (Left – A1/B1/C1)
Object name and image. (A2/B2/C2) Single run of experiment with termination
criterion off. Expected IG of the actions shown. Triangle markers show which
action was chosen in a given step. The dashed black line shows the
experimental true IG after the action was executed. This action is not shown
anymore in subsequent algorithm steps. (A3/B3/C3) Count of actions chosen
during the runs with the termination on (if all available actions at a given
step had negative expected IG, execution stopped). (A4/B4/C4) Probability
distributions of relevant variables of the single run shown in (A2/B2/C2).

###  IV-B Action Statistics and Intelligent Exploration Termination

Fig. 6 shows how many times different actions were executed when different
object properties, i.e. different nodes in the BN, were selected to be
“learned about”. When reducing the uncertainty in the Category node is the
target, Panel A shows that cat-vision was always selected as the first action.
In about half the cases, the algorithm then terminated. In the other cases,
mat-vision was picked in the second step. When asked to learn about the
Material (Panel C), interestingly, the algorithm picked the Weighing action as
the most effective in all cases, rather than the mat-vision or mat-sound
actions that directly target the Material node through their measurement
update. This illustrates how the algorithm automatically takes the complete BN
and the measurement uncertainties into account. For the continuous variables
(Panels B, D), the algorithm typically performed all five actions. Cat-vision
wass the most informative action to learn about Category (Panel A), Elasticity
(B), and all the continuous together variables (D) but not about Material (C).

Figure 6: Action statistics. Running the algorithm in the active termination
mode across all objects, 5 repetitions each. The bar plots show the count of
actions selected at every step. Shorter bars indicate that some runs
terminated as there was no action with positive expected IG. Optimizing
Network-Continuous jointly optimizes all the continuous variables in the BN.

###  IV-C Average performance across all objects

To demonstrate the effectiveness of the proposed algorithm over all the
objects, we ran the algorithm with 5 repetitions on each of the 17 objects,
using the action selection algorithm proposed optimizing the Category node and
with random action selection as a baseline (i.e., 2 action selection regimes x
17 objects x 5 repetitions x 5 algorithm steps). For comparison purposes, the
termination check was off. Fig. 7 shows the average entropy of the Category
variable after 5 steps of the algorithm in the two action selection regimes,
including the variance. The intelligent action selection (ACSEL) outperforms
RAND. Furthermore, in the ACSEL regime, the plot shows a dramatic drop in
entropy after the execution of the first action/measurement, demonstrating
that the most effective action was chosen in the first step for the majority
of the objects.

Figure 7: Average entropy (lower is better) of the Category node – Optimizing
Category, 17 objects, 5 repetitions each. Comparing action selection based on
IG (ACSEL) and random action selection (RAND).

To complement the analysis in Fig. 8 that shows entropy, which can be measured
autonomously by the system, Fig. 8 displays cross-entropy which considers how
well the current probability distribution of a node corresponds to the ground
truth—in this case the true label for a given Category or Material. For both
properties, ACSEL on average outperforms RAND. The first algorithm step is on
average effective in reducing the cross-entropy. Additional measurements may,
in fact, add noise to the system, increasing the entropy of the PDFs. However,
in these cases, these steps will often not be executed (see Fig. 6 A,C).

Figure 8: Cross-entropy of the Category and Material nodes – Optimizing
Category, 17 objects, 5 repetitions each. Comparing action selection based on
IG (ACSEL) and random action selection (RAND).

##  V Database of object properties and measurements for robot manipulation

Object datasets related to robot manipulation typically contain object models
and visual-based representations like images, point clouds, or meshes (e.g.,
YCB [12]). Datasets containing physical object properties are an exception
(e.g., [17, 18, 19]).

The measurements collected in this work provided the foundation for a database
of measurements of object physical properties. These measurements are
naturally dependent on the objects manipulated and also highly dependent on
the embodiment of the robot (e.g., stiffness measurements on the gripper used
and the force/effort feedback)—unlike for images where the camera that
recorded the images plays less critical role. To be useful, such a database
needs to cover different setups: different robots, grippers, sensors, etc. and
their description in the metadata. The structure of the database we created is
shown in Fig. 9; it is available at https://cmp.felk.cvut.cz/ipalm/.

Figure 9: Database of object properties and measurements for robot
manipulation – structure.

The database contains all the measurements collected in this work (images,
stress/strain curves for stiffness estimation, joint angle and joint torque
measurements for mass estimation, and audio responses after poking objects).
We have already extended the database with elasticity measurements of the same
and different objects using several other grippers (OnRobot RG6, Barrett Hand,
QB SoftHand, professional elasticity measurement setup).

Our contribution is aligned with the efforts of others to share robot data
across platforms and continents [20]. However, while [20] contains mainly
image and joint angle streams, our database is more “strongly embodied”,
allowing for storage of multimodal, in particular haptic data. We invite the
community to contribute. To make data collection easier in the correct format,
we developed a tool that automates this—a Python decorator (Butler), and its
source code is available on the database website. One can use this decorator
to save data in the desired directory structure and upload them to the
database. Using this workflow, we populated the database with more than 24000
measurements for over 60 objects. Ultimately, once enough instances of every
object type appear in the database, category-level description grounded in the
physical measurements will become available. Generalization across the
different devices, which were used to collect the haptic data, remains a
fundamental challenge.

##  VI Conclusion, Discussion, Future Work

We presented an algorithmic framework and experimental setup for learning
physical object properties through robot manipulation. Our approach involved
exploratory action selection to maximize IG about objects on a table in front
of a robot manipulator. At its core, the framework employs a BN model to
represent the probabilistic relationships between different object properties.
The BN consists of nodes corresponding to various object properties, including
categorical properties like Category and Material, as well as continuous
properties such as Elasticity, Density, and Volume. We demonstrated the
effectiveness of our method through experimental evaluation, showcasing its
ability to automatically extract properties such as material composition,
mass, volume, and stiffness. The algorithm outperformed random action
selection baseline and showed promising results in discovering object
properties on a partially adversarial object dataset (with “fake objects” like
a stiff sponge). The algorithm can automatically terminate based on the
expected IG of the available actions. The code used in this work is available
at https://github.com/ctu-vras/actsel. The object database and source code are
available at https://cmp.felk.cvut.cz/ipalm/.

Every measurement type in our framework has its own limitations. For example,
the Young’s modulus for Elasticity is limited by the resolution of current
flowing through the gripper. There is a threshold above which the gripper
cannot provide any specific information on the object deformation. This
threshold was experimentally found to be $95$ [kPa]. In such cases, the output
is binary, providing a half-open set of moduli from the threshold onward. This
is an uninformative measurement over all properties above this experimental
threshold. This brings a discontinuous change of elasticity belief
distribution.

Another source of information degradation is the inference of categorical
weights from the continuous GMMs using the EM algorithm. This stems from
uncertainties on the numerical level when the belief distribution mean moved
too far from the reference labels. Therefore, the only information that the EM
algorithm possesses at the location of reference labels is from the far tails
of the normal distributions. In the process of normalization to unit-sum
probabilities from very small numbers, the numerical errors provide a large
variance in results. We tried to tackle this problem using Monte Carlo Markov
Chains, more specifically Gibbs’ sampling method, although this has shown to
be too computationally expensive to run in real-time.

As part of future work, community engagement and dataset expansion are
essential endeavors. By making our dataset and codebase openly accessible, we
aim to foster collaboration and knowledge-sharing among researchers in the
field of robot learning and manipulation. We encourage contributions from the
community to enrich the dataset with diverse object measurements and
annotations, facilitating a broader range of research applications and
algorithm evaluations. Additionally, active engagement with the community
allows us to gather valuable feedback and insights, guiding the evolution of
our framework and ensuring its relevance and usefulness in real-world
settings. Together, these efforts contribute to the collective advancement of
robot learning and manipulation capabilities, ultimately driving innovation
and progress in the field.

Another promising avenue for advancement lies in the development of adaptive
exploration strategies. By leveraging techniques from reinforcement learning
and advanced decision-making algorithms, robots can autonomously learn and
refine their exploration policies over time. Adaptive exploration strategies
can dynamically adjust the selection of actions based on the robot’s past
experience and the current state of uncertainty in object properties. This
allows robots to prioritize actions that are more likely to yield informative
measurements, leading to more efficient and effective learning of physical
object properties.

## References

  * [1] J. K. Behrens, M. Nazarczuk, K. Stepanova, M. Hoffmann, Y. Demiris, and K. Mikolajczyk, “Embodied reasoning for discovering object properties via manipulation,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 10 139–10 145.
  * [2] A. Dutta, E. Burdet, and M. Kaboli, “Push to know!-visuo-tactile based active object parameter inference with dual differentiable filtering,” in _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2023, pp. 3137–3144.
  * [3] T. N. Le, F. Verdoja, F. J. Abu-Dakka, and V. Kyrki, “Probabilistic surface friction estimation based on visual and haptic measurements,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, 2021.
  * [4] D. Xu, G. E. Loeb, and J. A. Fishel, “Tactile identification of objects using Bayesian exploration,” in _2013 IEEE International Conference on Robotics and Automation_. IEEE, 2013, pp. 3056–3061.
  * [5] A. Petrovskaya and K. Hsiao, “Active manipulation for perception,” _Springer Handbook of Robotics_ , pp. 1037–1062, 2016.
  * [6] V. Chu, I. McMahon, L. Riano, C. G. McDonald, Q. He, J. M. Perez-Tejada, M. Arrigo, N. Fitter, J. C. Nappo, T. Darrell, _et al._ , “Using robotic exploratory procedures to learn the meaning of haptic adjectives,” in _2013 IEEE International Conference on Robotics and Automation_. IEEE, 2013, pp. 3048–3055.
  * [7] J. A. Fishel and G. E. Loeb, “Bayesian Exploration for Intelligent Identification of Textures,” _Frontiers in neurorobotics_ , vol. 6, p. 4, 2012.
  * [8] M. Mahendran, “The Modulus of Elasticity of Steel - Is It 200 GPa?” _International Specialty Conference on Cold-Formed Steel Structures_ , vol. 5, Oct 1996.
  * [9] M. Dondi, E. G., M. M., M. C., and C. Mingazzini, “The chemical composition of porcelain stoneware tiles and its influence on microstructure and mechanical properties,” _InterCeram: International Ceramic Review_ , vol. 48, pp. 75–83, 01 1999.
  * [10] T. Húlan and I. Štubňa, “Young’s modulus of kaolinite-illite mixtures during firing,” _Applied Clay Science_ , vol. 190, p. 105584, 2020.
  * [11] D. C. Giancoli, _Physics: Principles with Applications_. Pearson, 2014.
  * [12] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar, “The YCB object and Model set: Towards common benchmarks for manipulation research,” in _2015 International Conference on Advanced Robotics (ICAR)_ , 2015, pp. 510–517.
  * [13] S. P. Patni, P. Stoudek, H. Chlup, and M. Hoffmann, “Online elasticity estimation and material sorting using standard robot grippers,” _arXiv preprint arXiv:2401.08298_ , 2024.
  * [14] M. Dimiccoli, S. Patni, M. Hoffmann, and F. Moreno-Noguer, “Recognizing object surface material from impact sounds for robot manipulation,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 9280–9287.
  * [15] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2,” https://github.com/facebookresearch/detectron2, 2019.
  * [16] M. Nazarczuk and K. Mikolajczyk, “SHOP-VRB: A Visual Reasoning Benchmark for Object Perception,” _International Conference on Robotics and Automation (ICRA)_ , 2020.
  * [17] Z. Chen, K. Yi, Y. Li, M. Ding, A. Torralba, J. B. Tenenbaum, and C. Gan, “Comphy: Compositional physical reasoning of objects and events from videos,” in _International Conference on Learning Representations_ , 2022\.
  * [18] M. Purri and K. Dana, “Teaching cameras to feel: Estimating tactile physical properties of surfaces from images,” in _European Conference on Computer Vision_. Springer, 2020, pp. 1–20.
  * [19] M. Thosar, C. A. Mueller, G. Jäger, J. Schleiss, N. Pulugu, R. Mallikarjun Chennaboina, S. V. Rao Jeevangekar, A. Birk, M. Pfingsthorn, and S. Zug, “From Multi-Modal Property Dataset to Robot-Centric Conceptual Knowledge About Household Objects,” _Frontiers in Robotics and AI_ , vol. 8, p. 87, 2021.
  * [20] Open X-Embodiment Collaboration, A. Padalkar, _et al._ , “Open X-Embodiment: Robotic learning datasets and RT-X models,” https://arxiv.org/abs/2310.08864, 2023.

  *[IG]: Information Gain
  *[BN]: Bayesian Network
  *[PMF]: Probability Mass Function
  *[PDF]: Probability Density Function
  *[GMM]: Gaussian Mixture Model
  *[PDFs]: Probability Density Function
  *[SDs]: Standard Deviation
  *[EM]: Expectation–maximization
  *[SD]: Standard Deviation
  *[IGs]: Information Gain
  *[GMMs]: Gaussian Mixture Model