#  lilGym: Natural Language Visual Reasoning with Reinforcement Learning

Anne Wu, Kianté Brantley, Noriyuki Kojima, and Yoav Artzi
Department of Computer Science and Cornell Tech, Cornell University
{aw588, kdb82<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present lilGym, a new benchmark for language-conditioned reinforcement
learning in visual environments. lilGym is based on 2,661 highly-compositional
human-written natural language statements grounded in an interactive visual
environment. We introduce a new approach for exact reward computation in every
possible world state by annotating all statements with executable Python
programs. Each statement is paired with multiple start states and reward
functions to form thousands of distinct Markov Decision Processes of varying
difficulty. We experiment with lilGym with different models and learning
regimes. Our results and analysis show that while existing methods are able to
achieve non-trivial performance, lilGym forms a challenging open problem.
lilGym is available at https://lil.nlp.cornell.edu/lilgym/.

##  1 Introduction

The ability to reason about natural language has the potential to transform
how reinforcement learning (RL) is used to train grounded agents. Language
provides an expressive and accessible conduit for task specification, enabling
agents to address a broad set of tasks, rather than to learn single-behavior
policies. At the same time, RL is a promising framework to study core grounded
language learning problems within interactive environments. Prerequisite to
realizing this potential are expressive benchmark environments, as has been
instrumental for progress in RL more broadly. However, natural language poses
unique challenges to building such benchmarks. Beyond the design of the
environment itself, which must support rich linguistic reasoning, accurate
reward computation requires resolving language semantics. Existing approaches
adopt different strategies to address this issue, most often by using
synthetic language (e.g., Côté et al., 2018; Co-Reyes et al., 2019). While
these approaches open new research avenues, their simplification of the
language problem limits their relevance to the complexities of human language.

Figure 1:  An illustrative example of lilGym. Given a natural language
statement with a target truth value, True in this example (omitted from the
figure), and an initial image, the agent needs to manipulate objects in the
environment so that the truth value of the statement with respect to the image
is the target truth value.

We present lilGym,111lilGym stands for Language, Interaction, and Learning
Gym. an RL benchmark where an agent manipulates a visual environment by adding
and removing objects conditioned on a natural language statement. The agent’s
goal is to modify the environment so that a given statement will have a pre-
specified truth-value with regard to the environment (i.e., the constraints
specified in the language are either satisfied or violated). Figure 1
illustrates the scenario. lilGym includes highly-compositional and
semantically-diverse natural language statements and visual environments from
the Natural Language for Visual Reasoning (NLVR) corpus (Suhr et al., 2017),
that combine with a configurable environment backbone to create thousands of
Markov Decision Processes (MDP) of varying complexity.

A key challenge in constructing lilGym is accurate reward computation. Because
of the flexibility of the environments and language, there are many possible
equally correct termination states for each MDP. Correct reward computation at
every possible state requires taking into account the semantic nuances of the
highly-compositional language. We address this by annotating all statements
with executable Python programs representing their meaning, effectively
creating a supervised semantic parsing corpus (e.g., Zelle and Mooney, 1996;
Zettlemoyer and Collins, 2005). The executable programs allow for exact and
reliable reward computation at every possible state.

Our experiments with lilGym show that existing models demonstrate non-trivial
performance given sufficient training time, with multi-modal pre-training
providing further benefit. However, there remains significant room for
improvement. For example, on the simplest configuration, our agent can solve
76.54% of the test environments, but performance drops significantly to only
6.09% on the most complex configuration. Our experiments also confirm the
importance of modeling language meaning for reward computation in learning.
The lilGym benchmark and trained models are available at
https://lil.nlp.cornell.edu/lilgym/.

##  2 Related Work

RL research has benefited greatly from benchmarks such as Atari Bellemare et
al. (2013) and MuJoCo Todorov et al. (2012). Although both are synthetic with
limited potential to train policies directly applicable to realistic
environments, their accessibility and focus on core RL problems have made them
impactful drivers of algorithm development. For example, the first
demonstration of an effective neural network policy Mnih et al. (2013) and the
development of proximal policy optimization (PPO; Schulman et al., 2017) both
used these benchmarks, and both were later shown to generalize to more complex
scenarios. lilGym is inspired by these benchmarks. Rather than aiming for
training models that transfer to realistic domains, it aims to enable
fundamental algorithmic development by emphasizing semantic diversity and
compositionality within an accessible benchmark.

There is significant interest in RL for training policies conditioned on
natural language. This includes multiple efforts at developing RL environments
with language-informed tasks, mainly via grounded language learning in 2D or
3D environments or using text games. Oftentimes, using synthetic language
(Narasimhan et al., 2015; Johnson et al., 2017; Chevalier-Boisvert et al.,
2019; Cao et al., 2020; Côté et al., 2018; Urbanek et al., 2019; Hermann et
al., 2017; Co-Reyes et al., 2019; Hausknecht et al., 2020; Jiang et al.,
2020). Although synthetic language allows studying the problem of learning
high-level concepts, many of the complexities of natural language may be
stripped away, and such approaches run the risk of reducing the language
learning challenge to reverse engineering the hand-crafted generation process.
In contrast, lilGym is based on semantically-diverse human-written language
grounded in a visual environment, and requires both the ability of reasoning
over highly-compositional language including sets and spatial relations, and
precise alignment between statements and states.

Alternatively, other benchmarks are created by first generating the target
sequence of decisions (i.e., task demonstration), and then soliciting post-hoc
instructional language (Shridhar et al., 2020, 2021; Hanjie et al., 2021).
This process uses human-written language, but likely retains the regularities
of the demonstration generation procedure. lilGym uses language from NLVR,
which was crowdsourced via a contrastive task that was shown to elicit high
semantic diversity. Another approach is to simplify the task so only a few
annotated termination states or trajectories are correct (Anderson et al.,
2018; Chen et al., 2019; Ku et al., 2020). lilGym does not adopt such
simplifications. Our experiments show the importance of considering language
meaning for reward computation when there is a large set of valid goal states.

##  3 Background: the NLVR Corpus

lilGym uses data from the NLVR corpus (Suhr et al., 2017). NLVR was initially
created as a supervised learning benchmark. We formalize an interactive task
using the NLVR data and collect additional annotations for reward computation.

NLVR includes human-written natural language statements paired with synthetic
images. Each pair is annotated with the boolean truth-value of the statement
with regard to the image (i.e., True if the statement is true with regard to
the image, or False otherwise). The images are designed to support complex
reasoning, including about spatial and set relations. The original learning
task posed by NLVR is to classify statement-image pairs as True to indicate
the statement is true with regard to the image, or False otherwise. NLVR has
been studied extensively (Suhr et al., 2017; Tan and Bansal, 2018; Goldman et
al., 2018; Pavez et al., 2018; Yao et al., 2018; Hudson and Manning, 2018;
Perez et al., 2018; Dasigi et al., ; Zheng et al., 2020; Gupta et al., 2021),
and a separate version using photos was also released (Suhr et al.,
2019).222We do not use the photographic NLVR2 in this work.

Qualitative analysis of the NLVR data (Table 2 in Suhr et al., 2017) showed it
to provide diverse representation of semantic and compositional phenomena,
including requiring joint visual-linguistic reasoning about spatial relations,
quantities, and set relations. NLVR also provides an underlying structured
representation for every image, which supports image manipulation. The
combination of an interface for image manipulation with complex reasoning via
natural language makes NLVR ideal to support an interactive benchmark
environment.

##  4 The lilGym Benchmark

There is a blue block as the base of a tower with only two
blocks.True$s_{0}$There is no black block as the top of a tower with at most
three blocks.TrueThere is a box with 2 triangles of same color nearly touching
each other.TrueThere is a grey box where none of the black objects are
touching the edgeFalse$c$TOWER-SCRATCHTOWER-FLIPITSCATTER-SCRATCHSCATTER-
FLIPIT$s_{1}$$a_{0}\sim\pi(s_{0},c)$$r_{0}\sim R^{c}(s_{0},a_{0})$$a_{0}$ =
$\mathtt{ADD(}$$\mathtt{MIDDLE,}$$\mathtt{BLUE)}$$a_{0}$ =
$\mathtt{REMOVE(}$$\mathtt{RIGHT)}$$a_{0}$ =
$\mathtt{ADD(}$$\mathtt{x_{0},y_{0},}$$\mathtt{TRIANGLE,}$$\mathtt{YELLOW,LARGE)}$$a_{0}$
=
$\mathtt{ADD(}$$\mathtt{x_{0},y_{0},}$$\mathtt{SQUARE,}$$\mathtt{BLACK,LARGE)}$$s_{2}$$a_{1}\sim\pi(s_{1},c)$$r_{1}\sim
R^{c}(s_{1},a_{1})$$a_{1}$ =
$\mathtt{ADD(}$$\mathtt{MIDDLE,}$$\mathtt{YELLOW)}$$a_{1}$ =
$\mathtt{STOP}()$$a_{1}$ =
$\mathtt{ADD(}$$\mathtt{x_{1},y_{1},}$$\mathtt{TRIANGLE,}$$\mathtt{YELLOW,SMALL)}$$a_{1}$
= $\mathtt{STOP}()$$s_{3}$$a_{2}\sim\pi(s_{2},c)$$r_{2}\sim
R^{c}(s_{2},a_{2})$$a_{2}$ = $\mathtt{STOP}()$$a_{2}$ = $\mathtt{STOP}()$
Figure 2: Examples from the four CMDP configurations. Each example is
conditioned on a context $c=(\bar{x},b)$, and starts with a state $s_{0}$,
sampled from the initial state distribution $\beta^{c}$. For example, for
TOWER-SCRATCH (left column), the context $c$ pairs the statement there is a
blue block as the base of a tower with only two blocks with the target boolean
True. The initial state $s_{0}$ is an image with three empty grey box regions
separated by darker grey separators. The agent $\pi$ is given $(s_{0},c)$, and
samples an action $a_{0}\sim\pi(s_{0},c)$. The environment transitions to the
next state $s_{1}$, while the context remains the same. This process continues
until the agent selects the $\mathtt{STOP}$ action.

lilGym consists of a collection of environments that share a common backbone.
The backbone is a 2D plane that is manipulated by placing and removing objects
of different types. Each environment instance is a Markov Decision Process
(MDP) created by pairing a natural language statement and a target boolean
value with a configuration of the shared backbone. The goal of the agent is to
manipulate the environment by adding and removing objects so that the truth-
value of the statement with regard to the environment is the target boolean.

The learning problem lilGym presents is to induce a policy that generalizes
across MDPs. We split the MDPs to training, development, and held-out testing
sets. The training environments are for parameter estimation, while the two
other sets are for testing during development and for final held-out testing
to report approach performance.333We recommend reporting both development and
held-out test results in future work for easy comparison.

There are two dimensions of configuration: appearance and starting condition.
The appearance determines the state space, transition function, and action
space. The appearance of the environment can be (a) TOWER: the objects include
squares only, and they can be stacked into towers in specific positions only;
or (b) SCATTER: objects of different types can be freely distributed. The two
leftmost examples in Figure 2 are from TOWER, and the two rightmost are from
SCATTER. TOWER gives a more constrained problem with much smaller state and
action spaces compared to SCATTER.

There are two starting conditions, which also determine the agent’s goal: (a)
SCRATCH: the environment starts without any objects and the goal is to modify
it so that the statement’s truth-value is True; or (b) FLIPIT: the environment
starts with a set of objects and the agent’s goal is to flip the truth-value
of the statement. The first row of images in Figure 2 shows start states in
both conditions. SCRATCH generally only requires adding objects, except in
cases of correcting for agent’s errors, while FLIPIT requires both adding and
removing, because there are already objects present.

The four configurations are TOWER-SCRATCH, TOWER-FLIPIT, SCATTER-SCRATCH, and
SCATTER-FLIPIT. In our experiments (Section 6), we observe the different
configurations provide different levels of difficulty. For example, SCATTER
configurations are generally harder than TOWER, due to the larger state and
action spaces.

Formally, each configuration is a Contextual Markov Decision Process (CMDP;
Hallak et al., 2015). CMDP is an abstraction over a set of MDPs to account for
a context that remains constant throughout an interaction with an MDP. The
context includes the statement and the target boolean the interaction is
conditioned on. A CMDP is a tuple
$(\mathcal{C},\mathcal{S},\mathcal{A},\mathcal{M}(c))$, where $\mathcal{C}$ is
the context space, $\mathcal{S}$ the state space, $\mathcal{A}$ the action
space, and $\mathcal{M}$ a function mapping a context $c\in\mathcal{C}$ to an
MDP $\mathcal{M}(c)=(\mathcal{S},\mathcal{A},T,R^{c},\beta^{c})$. Here,
$T:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$ is a transition
function, $R^{c}:\mathcal{S}\times\mathcal{A}\rightarrow$ a reward function,
and $\beta^{c}$ an initial state distribution. This means that a CMDP is a set
of MDPs that share the same states and actions. The policy learning problem is
to estimate parameters $\theta$ of a policy
$\pi_{\theta}:\mathcal{S}\times\mathcal{C}\rightarrow\mathcal{A}$, which maps
the current state and the context underlying the MDP to an action. The policy
must generalize across different contexts from $\mathcal{C}$. Figure 2 shows
example action trajectories in MDPs for each of the four CMDPs. Table 1 shows
the number of MDPs in each configuration.444 NLVR includes 18,322 images. This
allows further expanding the number of initial states to 92,179 initial states
through box element permutations. We do not manipulate this property in this
work, but future work could take advantage of it. Our reward computation is
invariant to such permutations.

#### Contexts $\mathcal{C}$

A context $c\in\mathcal{C}$ is a pair $c=(\bar{x},b)$, where $\bar{x}$ is a
natural language statement and $b\in\\{\texttt{True},\texttt{False}\\}$ is a
target boolean value for the statement $\bar{x}$ with respect to the state
$s$. The set of statements is predefined for TOWER and SCATTER based on the
NLVR data, but identical across the choice of SCRATCH and FLIPIT. The target
boolean value in SCRATCH is always True. In FLIPIT, the target boolean value
is either True or False. Depending on the context, different types of
reasoning are required. For example, in the second column of Figure 2, the
statement there is no black block as the top of a tower with at most three
blocks requires reasoning about negation, soft cardinality, color, and
position, while the statement in the third column there is a box with 2
triangles of same color nearly touching each other requires a comparison and
to reason about several object attributes (shape, color, position). Both
require high-level relational reasoning about single objects or sets.

| TOWER-SCRATCH |  | TOWER-FLIPIT |  | SCATTER-SCRATCH |  | SCATTER-FLIPIT
---|---|---|---|---|---|---|---
| MDPs |  | MDPs | Init. |  | MDPs |  | MDPs | Init.
Train | 989 |  | 1,910 | 5,704 |  | 1,241 |  | 2,340 | 6,696
Dev | 163 |  | 317 | 676 |  | 87 |  | 164 | 313
Test | 324 |  | 619 | 1,383 |  | 155 |  | 285 | 591
Total | 1,476 |  | 2,846 | 7,763 |  | 1,483 |  | 2,789 | 7,600
Table 1: Data statistics per CMDP configuration and data split. The number of
MDPs corresponds to the number of contexts under each CMDP. For FLIPIT,
“Init.” corresponds to the total number of initial states across all MDPs for
this CMDP.${}^{\ref{fn:nlvrexp}}$ The number of initial states and MDPs is
equal for SCRATCH CMDPs.

#### States $\mathcal{S}$

A state $s\in\mathcal{S}$ is an RGB image. Images in lilGym are divided into
three box regions of identical dimensions by two dark gray separators (Figure
2). The objects in lilGym have three properties, each can take multiple
values: shape ($\mathtt{CIRCLE}$, $\mathtt{SQUARE}$, or $\mathtt{TRIANGLE}$),
color ($\mathtt{BLACK}$, $\mathtt{BLUE}$, or $\mathtt{YELLOW}$), and size
($\mathtt{SMALL}$, $\mathtt{MEDIUM}$, or $\mathtt{LARGE}$). In TOWER, states
are constrained to have stacks of up to four $\mathtt{SQUARE}$s of
$\mathtt{MEDIUM}$ size and any color at the center of each box. SCATTER states
support all object shapes, sizes, and colors, and they may be positioned
freely. In both conditions, objects cannot cross image boundaries or into the
separators. The choice of starting condition between SCRATCH or FLIPIT does
not influence the state space.

#### Actions $\mathcal{A}$ and Transitions $T$

There are three action types $\mathtt{STOP}$, $\mathtt{ADD}$, and
$\mathtt{REMOVE}$. $\mathtt{STOP}$ terminates the episode. The truth-value of
the statement is only evaluated and compared to the target boolean after the
$\mathtt{STOP}$ action is taken. $\mathtt{ADD}$ adds objects to the
environment, and $\mathtt{REMOVE}$ removes objects. $\mathtt{ADD}$ and
$\mathtt{REMOVE}$ take arguments that differ between TOWER and SCATTER:

TOWER: Both $\mathtt{ADD}$ and $\mathtt{REMOVE}$ take a position argument,
which has three possible values corresponding to the three box regions.
Objects are added or removed at the top of the stack. Adding an object on top
of a stack of four objects or removing an object from an empty box are both
invalid actions. $\mathtt{ADD}$ also takes a color argument. For example, the
first action on the left trajectory in Figure 2 is adding a blue square in an
empty box. Including $\mathtt{STOP}$, there are $1+(3+1)\times 3=13$ actions.

SCATTER: Unlike TOWER, objects of any type can be placed freely in the box
regions. Both $\mathtt{ADD}$ and $\mathtt{REMOVE}$ take 2D coordinates that
specify pixel location. Adding an object places it so that its top-left
coordinates are the given coordinates. Removing an object will remove the
object at the given coordinates. Adding also requires specifying the shape,
color, and size. The action is invalid if adding results in objects’ overlap
or boundary crossing with the separators or image boundaries. Removing from a
position that does not include an object is also an invalid action. The native
resolution of images in lilGym is 380$\times$100 pixels. Including
$\mathtt{STOP}$, there are $1+(380\times 100)\times((3\times 3\times
3)+1)=1{,}064{,}001$ actions. Because of the extremely large action space,
lilGym also supports a simplification through a coarser grid system for
SCATTER that is automatically mapped to the original resolution (Appendix A).
The grid simplification includes heuristics that assist in identifying
locations to place objects in the original pixel space or objects to remove
once a grid cell is selected. In our experiments (Section 6), we use a grid
simplification of 19$\times$5, giving a total of 2,661 actions. The difficulty
of SCATTER can be adjusted by modifying the grid size, or acting at the
original resolution.

The transition function $T:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S}$
depends on the choice between TOWER and SCATTER configurations, because this
choice determines the action space. Similar to the action space, the
transitions in TOWER are more constrained compared to SCATTER. The transition
function does not modify the context, which is fixed for a given MDP.

#### Reward Function $R^{c}$

The reward function $R^{c}$ is computed with respect to the context
$c=(\bar{x},b)$, and is based on evaluating the truth-value of the natural
language statement $\bar{x}$ with respect to a state $s$, and comparing it to
the target boolean $b$. lilGym includes an evaluation function
$\mathcal{E}^{\bar{x}}:\mathcal{S}\times\mathcal{A}\rightarrow\\{\texttt{True},\texttt{False}\\}$
for every statement $\bar{x}$. Section 5 describes how we create the
evaluation functions.

The agent receives a positive reward for terminating the episode using the
$\mathtt{STOP}$ action if the evaluation $\mathcal{E}^{\bar{x}}(s)$ is equal
to the target boolean $b$. If $\mathcal{E}^{\bar{x}}(s)$ does not equal $b$
when taking the $\mathtt{STOP}$ action, the agent receives a negative reward.
If the episode terminates because the current time step $t$ reached the action
horizon $H$ or because of an invalid action, the agent also receives a
negative reward. Action validity depends on the current state $s$ and on the
configuration, because TOWER and SCATTER have different action spaces. For
example, in TOWER, adding an object to a box (e.g.,
$\mathtt{ADD}(\mathtt{MIDDLE},\mathtt{BLUE})$) is only valid if the box has
less than four objects, because towers have a maximum height of four. There is
also a verbosity penalty of $\delta$. Formally, the reward is:

$\displaystyle R^{c}(s,a)$
$\displaystyle=\begin{cases}1.0&a=\mathtt{STOP}\wedge\mathcal{E}^{\bar{x}}(s)=b\\\
-1.0&a=\mathtt{STOP}\wedge\mathcal{E}^{\bar{x}}(s)\neq b\\\ -1.0&(a\text{ is
invalid in }s)\lor(t=H)\\\ -\delta&\text{otherwise}\end{cases}\;\;.$ (1)

#### Initial State Distribution $\beta^{c}$

The initial state distribution $\beta^{c}$ is parameterized by the context
$c\in\mathcal{C}$, and differs between SCRATCH and FLIPIT. In SCRATCH, the
agent modifies an empty environment to satisfy the truth-condition of the
statement $\bar{x}$ in the context $c$, so the initial state $s_{0}$ is always
an empty image. The set of initial states $\beta^{c}$ for every context
$c\in\mathcal{C}$ is the set of images associated with the statement $\bar{x}$
in the NLVR data. This set includes between 1 to 43 images. Table 1 shows the
total number of initial states in each configuration.

##  5 The lilGym Data and Annotation

We use the NLVR data to create each of the CMDPs (Table 1). SCRATCH CMDPs
include contexts for all natural language statements from NLVR, each paired
with the empty initial state containing no shapes (Figure 2, left and center-
right columns). FLIPIT CMDPs include the natural language statements with
their corresponding images, both from NLVR. The images are used as initial
states. The target boolean is set so that the initial state does not fulfil
it. The split between TOWER and SCATTER also follows from NLVR. Statements
corresponding to TOWER images in NLVR are included in our TOWER CMDPs, and the
same for SCATTER sentences.

NLVR has four splits for training, development, public testing, and hidden
testing. We adopt the original training and development sets splits. Following
the recent public release of the hidden testing set, we merge the public and
hidden testing sets into a single public test split.

The NLVR annotations include the truth-value of each statement with regard to
the images paired with it in the data. Once we manipulate an image (i.e.,
change the state in our environment), the truth-value annotation does not
necessarily hold. A key challenge for creating an interactive environment
using this data is an accurate evaluation of the natural language statement
for every possible state (i.e., image) for reward computation (Section 4). We
address this by annotating each statement $\bar{x}$ with an executable boolean
Python program representing its meaning, $\mathcal{E}^{\bar{x}}$ in Section 4.
This process is inspired by data annotation for supervised semantic parsing
Zelle and Mooney (1996); Zettlemoyer and Collins (2005); Suhr et al. (2018),
where sentences are annotated with formal meaning representations.

The Python programs operate on the underlying structured representation. Each
program returns True for every image that satisfies the constraints specified
in the corresponding statement, and False otherwise. In general, there are
many states that satisfy any given statement, many more than provided with the
original NLVR images.

The programs are written using an API defined over the structured
representations. We base the API design on the ontology designed for NLVR’s
structured representations by Goldman et al. (2018), which we extend to
include 66 functions. Figure 7 in Appendix B shows two examples programs with
their corresponding statements.

We use the freelancing platform Upwork555https://www.upwork.com for
annotation. We recruit three programmers based on preliminary screening of
their fluency in English and competency in Python. We de-duplicate the
naturally occurring sentences in the data, and distribute sentences to
annotators randomly, each with a single example NLVR image. Each program is
evaluated against a corresponding hidden validation set made of all remaining
NLVR images paired with the sentence, and must pass all the tests. Appendix B
provides a screenshot of the interface and more details. We collect 2,666
annotations at a total cost of $3,756, and keep 2,661 valid annotations.

##  6 Experiments

###  6.1 Methods

We experiment with each of the four CMDPs separately, training on the training
split and testing on the development and test splits. We sample a validation
set from the training split for model selection. For SCATTER we use a
simplified grid action space of 19$\times$5 (Section 4). Each grid cell is
20$\times$20 pixels. We set the action horizon $H=12$. Appendix C provides
implementation details.

We use PPO (Schulman et al., 2017) for parameter estimation,666We use the PPO
implementation of Kostrikov (2018). with a separate network as a critic. The
critic network is identical to the policy, except that we add a $\tanh$
activation for the value output. Because of the large action space, especially
for SCATTER, the agent rarely observes positive reward, which requires taking
a $\mathtt{STOP}$ action at an appropriate state. We design a simple variant
of PPO called PPO+SF (PPO with stop forcing) to study this issue. PPO+SF is
identical to PPO, except that during training, we mask all actions except
$\mathtt{STOP}$ when the agent reaches a state where selecting $\mathtt{STOP}$
will give a positive reward. This modification is present only during
training. All testing is done under the same conditions, without stop forcing.

We also study the importance of our annotation for reward computation using a
reward function for SCRATCH that does not require any annotation beyond what
is already in NLVR. The reward uses NLVR images that are associated with each
statement (between 1–43) and are labeled with the target boolean.777We dropped
1% of MDPs without a single target state. Instead of testing the state using a
program, it compares the state to the available NLVR images, and only if it
equals one of them, the learner receives a positive task completion reward.

We experiment with two models: C+BERT and ViLT. In C+BERT, we process the
statement $\bar{x}$ using BERT (Devlin et al., 2019), and do mean pooling
across all layers and tokens to get the statement representation. We use a
three-layer CNN (Fukushima and Miyake, 1982) to embed the image of the current
state $s$. We concatenate the statement and image representations with an
embedding for the target boolean $b$, and use a multi-layer perceptron to
compute the action distribution. ViLT is a pretrained multi-modal Transformer
that jointly processes text and image inputs (Kim et al., 2021). We create a
sequence of tokens by concatenating the statement, a token for the target
boolean, and image patches, separated by special tokens. The image patches are
the same size as the 19$\times$5 grid cells, including in TOWER, where the
action space does not use a grid.

|  | TOWER-SCRATCH |  | TOWER-FLIPIT |  | SCATTER-SCRATCH |  | SCATTER-FLIPIT
---|---|---|---|---|---|---|---|---
|  | Dev | Test |  | Dev | Test |  | Dev | Test |  | Dev | Test
PPO | C+BERT | 71.78 | 63.27 |  | 35.95 | 34.78 |  | 39.08 | 48.39 |  |  00.00 |  00.00
ViLT | 81.60 | 76.54 |  | 67.60 | 65.80 |  | 35.63 | 41.29 |  |  03.51 |  06.09
PPO+SF | C+BERT | 80.98 | 78.70 |  | 27.22 | 26.75 |  | 70.12 | 74.84 |  |  08.31 |  08.46
ViLT | 84.05 | 82.41 |  | 65.09 | 62.91 |  | 64.37 | 70.97 |  | 27.48 | 29.95
Table 2: Mean task-completion accuracies for all four CMDPs.

###  6.2 Results and Analysis

Table 2 shows task-completion accuracies for all CMDPs, and Figure 4 shows
reward statistics. Figure 3 plots training curves for SCATTER.888We plot the
training curves without regard to the patience stopping criteria we use for
model selection. This better reflects long term trends. Figure 5 breaks down
development set accuracies for FLIPIT CMDPs by the target boolean, and Figure
6 shows development rollout statistics for PPO.999Appendix D provides further
analysis, including error analysis, performance breakdown by semantic
phenomena, and rollout statistics for PPO+SF. We sample 50 development
examples for each CMDP, and annotate them with expert101010The expert is an
author of this paper. trajectories to estimate the expert reward and rollout
statistics. All expert rollouts are successful.

Overall, we observe stronger task-completion performance (Table 2) on TOWER
CMDPs compared to SCATTER, especially with ViLT, which shows stronger
performance than C+BERT in most cases. The training curves (Figure 3) and
development rewards show similar trends (Figure 4). The training curve
comparison to the alternative reward that uses NLVR images instead of our
program annotations shows no effective learning. This illustrates the
importance of exact reward computation, such as possible with our program
annotations. The comparison to the estimate of expert rewards shows there
remains significant room for improvement across the board. Even when the
learned policies are able to complete the task, they are doing it
inefficiently, so much so that the mean rewards are often negative. The mean
rewards of the random baseline policy illustrates the task is far from
trivial. Both task accuracies and reward statistics indicate TOWER-SCRATCH is
the easier of the CMPDs, and SCATTER-FLIPIT is the hardest.

The additional guidance of PPO+SF compared to PPO helps with exploration,
especially on SCATTER CMDPs. On SCATTER-FLIPIT, PPO+SF improves performance by
23.86% compared to PPO. This illustrates the exploration challenges SCATTER
CMDPs pose.

ViLT generally outperforms C+BERT, except on SCATTER-SCRATCH. This is
relatively expected given the joint reasoning architecture and multi-modal
pre-training of ViLT. FLIPIT policies generally do better on examples with a
False target boolean, except when learning fails (Figure 5). The other
direction is harder, because the set of states that invalidates a statement is
usually larger than the set that validates it, and it generally requires fewer
actions to invalidate a statement.

We observe more rollouts that are terminated either by reaching the action
horizon $H$ or by taking an invalid action (i.e., without $\mathtt{STOP}$) on
TOWER CMDPs compared to SCATTER (Figure 6, upper left). This difference is
partially explained by a higher rate of invalid actions in SCATTER (Figure 6,
upper right), which cause immediate rollout termination. In general, ViLT has
a higher rate of invalid actions, except on SCATTER-FLIPIT, where overall
performance is extremely low.

There is no consistent difference in the length of rollouts between the two
models (Figure 6, bottom left). Expert trajectory length is similar on TOWER,
where models perform fairly well. However, on SCATTER, where our models are
weaker, expert trajectories are significantly longer. This is partially
explained by the models not learning to effectively avoid invalid actions,
which terminate the execution immediately. Using $\mathtt{REMOVE}$ actions is
generally difficult for the learned policies. TOWER-FLIPIT is an exception
with $\mathtt{REMOVE}$ dominating the rollouts (Figure 6, bottom right),
potentially because removing objects generally provides a more efficient path
to flip the boolean value. While PPO policies generate $\mathtt{REMOVE}$
actions for SCATTER-FLIPIT, the extremely low performance indicates that these
actions are not used effectively. Expert statistics indicate that
$\mathtt{REMOVE}$ actions are beneficial for FLIPIT CMDPs.

Figure 3: Training curves for SCATTER. All curves with NLVR reward are
superposed because accuracy remains zero throughout training.  Figure 4: Mean
development set rewards. Figure 5: Development set accuracies for FLIPIT
CMDPs, reported according to the value of the context target boolean (Red for
True, Gray for False). Dashed gray line: full development set accuracies.
Figure 6: Development statistics for PPO. Clockwise from top left: rollouts
without a $\mathtt{STOP}$ action; rollouts with an invalid action; mean
actions per rollout; relative rate of $\mathtt{ADD}$/$\mathtt{REMOVE}$
actions.

##  7 Conclusion

We introduce lilGym, an RL benchmark that focuses on natural language visual
reasoning. lilGym is designed to be accessible for researchers, while still
displaying the reasoning richness of natural language. It is relatively easy
to deploy using the standard Gymnasium API (Brockman et al., 2016), and has
light compute requirements. Our data annotation approach allows including
expressive and diverse natural language, while still providing accurate and
automatic reward computation. It also exposes the potential connection between
semantic parsing and reward evaluation in RL, thereby outlining how strong
semantic parsers can benefit RL benchmarking. Our strong baselines illustrate
the range of challenges lilGym presents, showing that existing methods can
achieve non-trivial performance, but that there remain significant progress to
be made. Our analysis lays out the framework for studying and reporting these
future results.

lilGym has significant potential beyond the tasks we study. It can be used
without the language, to create thousands of micro RL tasks requiring set and
relational visual reasoning. Our annotations form a new semantic parsing
corpus with annotated executable meaning representations. The semantic
diversity of the data, its executability, and the focus on visual reasoning
make it a unique asset in the landscape of corpora for semantic parsing.
lilGym is also promising for program synthesis guided by natural language
(Wong et al., 2021).

## Acknowledgements

This research was supported by ARO W911NF21-1-0106, NSF under grant No.
1750499, and a gift from Open Philanthropy. KB is supported by NSF under grant
No. 2127309 to the Computing Research Association for the CIFellows Project.
Results presented in this paper were obtained using CloudBank (Norman et al.,
2021), which is supported by the National Science Foundation under award No.
1925001. We thank Alane Suhr, Ge Gao, Justin Chiu, Woojeong Kim, Jack Morris,
Jacob Sharf and the Cornell NLP Group for support, comments, and helpful
discussions.

## References

  * Anderson et al. (2018) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. 2018.  Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments.  In _The IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3674–3683.
  * Bellemare et al. (2013) Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. 2013.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , 47:253–279.
  * Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016.  Openai gym.  _arXiv preprint arXiv:1606.01540_.
  * Cao et al. (2020) Tianshi Cao, Jingkang Wang, Yining Zhang, and Sivabalan Manivasagam. 2020.  Babyai++: Towards grounded-language learning beyond memorization.  _Beyond tabula rasa in RL (BeTR-RL) Workshop held in conjunction with the 8th International Conference on Learning Representations, ICLR_.
  * Chen et al. (2019) Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019.  Touchdown: Natural language navigation and spatial reasoning in visual street environments.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12538–12547.
  * Chevalier-Boisvert et al. (2019) Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. 2019.  Babyai: A platform to study the sample efficiency of grounded language learning.  In _7th International Conference on Learning Representations, ICLR_.
  * Co-Reyes et al. (2019) John D. Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, John DeNero, P. Abbeel, and Sergey Levine. 2019.  Guiding policies with language via meta-learning.  In _7th International Conference on Learning Representations, ICLR_.
  * Côté et al. (2018) Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew J. Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2018.  Textworld: A learning environment for text-based games.  In _Computer Games Workshop (CGW) held in conjunction with the 27th International Conference on Artificial Intelligence, IJCAI_.
  * (9) Pradeep Dasigi, Matt Gardner, Shikhar Murty, Luke Zettlemoyer, and Eduard Hovy.  Iterative search for weakly supervised semantic parsing.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2669–2680.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Fukushima and Miyake (1982) Kunihiko Fukushima and Sei Miyake. 1982.  Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition.  In _Competition and cooperation in neural nets_ , pages 267–285. Springer.
  * Goldman et al. (2018) Omer Goldman, Veronica Latcinnik, Ehud Nave, Amir Globerson, and Jonathan Berant. 2018.  Weakly supervised semantic parsing with abstract examples.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1809–1819.
  * Gupta et al. (2021) Nitish Gupta, Sameer Singh, and Matt Gardner. 2021.  Enforcing consistency in weakly supervised semantic parsing.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 168–174.
  * Hallak et al. (2015) Assaf Hallak, Dotan Di Castro, and Shie Mannor. 2015.  Contextual markov decision processes.  _arXiv preprint arXiv:1502.02259_.
  * Hanjie et al. (2021) Austin W Hanjie, Victor Y Zhong, and Karthik Narasimhan. 2021.  Grounding language to entities and dynamics for generalization in reinforcement learning.  In _International Conference on Machine Learning_ , pages 4051–4062. PMLR.
  * Hausknecht et al. (2020) Matthew J. Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. 2020.  Interactive fiction games: A colossal adventure.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34.
  * Hermann et al. (2017) Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris Apps, Demis Hassabis, and Phil Blunsom. 2017\.  Grounded language learning in a simulated 3d world.  _arXiv preprint arXiv:1706.06551_.
  * Hudson and Manning (2018) Drew A. Hudson and Christopher D. Manning. 2018.  Compositional attention networks for machine reasoning.  In _6th International Conference on Learning Representations, ICLR_.
  * Jiang et al. (2020) Minqi Jiang, Jelena Luketina, Nantas Nardelli, Pasquale Minervini, Philip HS Torr, Shimon Whiteson, and Tim Rocktäschel. 2020.  Wordcraft: An environment for benchmarking commonsense agents.  _arXiv preprint arXiv:2007.09185_.
  * Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017.  Inferring and executing programs for visual reasoning.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 3008–3017.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , pages 5583–5594.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR_.
  * Kostrikov (2018) Ilya Kostrikov. 2018.  Pytorch implementations of reinforcement learning algorithms.
  * Ku et al. (2020) Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020.  Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4392–4412.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR_.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_.
  * Narasimhan et al. (2015) Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. 2015.  Language understanding for text-based games using deep reinforcement learning.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP_ , pages 1–11.
  * Norman et al. (2021) Michael Norman, Vince Kellen, Shava Smallen, Brian DeMeulle, Shawn Strande, Ed Lazowska, Naomi Alterman, Rob Fatland, Sarah Stone, Amanda Tan, Katherine Yelick, Eric Van Dusen, and James Mitchell. 2021.  Cloudbank: Managed services to simplify cloud access for computer science research and education.  In _Practice and Experience in Advanced Research Computing_ , PEARC ’21. Association for Computing Machinery.
  * Pavez et al. (2018) Juan Pavez, Héctor Allende, and Héctor Allende-Cid. 2018.  Working memory networks: Augmenting memory networks with a relational reasoning module.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1000–1009.
  * Perez et al. (2018) Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2018.  Film: Visual reasoning with a general conditioning layer.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017\.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Shridhar et al. (2020) Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020.  ALFRED: A benchmark for interpreting grounded instructions for everyday tasks.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR_ , pages 10740–10749.
  * Shridhar et al. (2021) Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew J. Hausknecht. 2021.  Alfworld: Aligning text and embodied environments for interactive learning.  In _9th International Conference on Learning Representations, ICLR_.
  * Suhr et al. (2018) Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018.  Learning to map context-dependent sentences to executable formal queries.  In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2238–2249.
  * Suhr et al. (2017) Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017.  A corpus of natural language for visual reasoning.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 217–223.
  * Suhr et al. (2019) Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019\.  A corpus for reasoning about natural language grounded in photographs.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6418–6428.
  * Tan and Bansal (2018) Hao Tan and Mohit Bansal. 2018.  Object ordering with bidirectional matchings for visual reasoning.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 444–451.
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 5026–5033.
  * Urbanek et al. (2019) Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019.  Learning to speak and act in a fantasy text adventure game.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019_ , pages 673–683.
  * Wong et al. (2021) Catherine Wong, Kevin Ellis, Joshua B. Tenenbaum, and Jacob Andreas. 2021.  Leveraging language to learn program abstractions and search heuristics.  In _Proceedings of the 38th International Conference on Machine Learning, ICML_ , pages 11193–11204.
  * Yao et al. (2018) Yiqun Yao, Jiaming Xu, Feng Wang, and Bo Xu. 2018.  Cascaded mutual modulation for visual reasoning.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 975–980.
  * Zelle and Mooney (1996) J.M. Zelle and Raymond J. Mooney. 1996.  Learning to parse database queries using inductive logic programming.  In _Proceedings of the National Conference on Artificial Intelligence_.
  * Zettlemoyer and Collins (2005) Luke S. Zettlemoyer and Michael Collins. 2005.  Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars.  In _Proceedings of the Conference on Uncertainty in Artificial Intelligence_.
  * Zheng et al. (2020) Wenbo Zheng, Lan Yan, Chao Gou, and Fei-Yue Wang. 2020.  Webly supervised knowledge embedding model for visual reasoning.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12442–12451.

##  Appendix A SCATTER Grid Simplification

To reduce the large action space of SCATTER, lilGym allows to simplify the
pixel-based action space with a grid that is coarser than the image resolution
of 380$\times$100\. The actions applied in the environment remain in the
original resolution, and the translation between the grid system to pixels is
done heuristically. Without the heuristics, the transition to a grid coarser
than the original image resolution would render many of the MPDs unsolvable.

The heuristics simplify two translation problems: in what pixel exactly to
place an object and which object to remove from a grid cell. Depending on the
grid size, it is possible to add multiple objects in a cell. To find the exact
pixel within a cell to add an object, we search for a pixel in the grid box
where we can add the object starting from the upper left corner. We can add an
object in a pixel if the object fits there without overlapping with other
objects, the image boundaries, or the columns. We also snap objects to touch
each other if the distance between them is below a threshold. This is to allow
adding objects that touch each other, a common constraint in lilGym
statements. When removing an object from a grid cell, we remove the object
with largest overlap with the cell.

##  Appendix B Natural Language Annotation Details

We annotate each natural language statement in the NLVR corpus with a Python
program representing its meaning. The programs return a boolean value, and are
executable given the structured representation underlying each image. Figure 7
shows two examples of text statements with their annotated Python programs.

There are two towers with the same height but their base is not the same in
color.

⬇

exist(filter_obj(

all_boxes, lambda x: x.is_tower() and

exist(filter_obj(

all_boxes, lambda y:

y.is_tower() and

count(x.all_items_in_box()) ==

count(y.all_items_in_box()) and

get_set_colors(filter_obj(

y.all_items_in_box(),

is_bottom)) !=

get_set_colors(filter_obj(

x.all_items_in_box(),

is_bottom))))))

There is a box with all 3 different colors and a black triangle touching the
wall with its top.

⬇

exist(filter_obj(

all_boxes, lambda x:

count(get_set_colors(

x.all_items_in_box())) == 3 and

exist(filter_obj(

x.all_items_in_box(), lambda y:

is_black(y) and

is_triangle(y) and

is_touching_wall(y, Side.TOP)))))

Figure 7: Example sentences with the example images displayed alongside them
during annotation (left), and their annotated Python programs (right). Both
sentences and programs are True for the corresponding image.

Figure 8: The annotation interface for collecting the Python program
annotations in lilGym.

We provide the annotators with a web-based annotation interface (Figure 8), a
tutorial, and an application programming interface (API) presenting a set of
functions, classes and objects that they can use for annotation. We ask the
annotators to prioritize the faithfulness of the program to the natural
language sentence and to prefer shorter annotations. We also provide them with
examples of spurious logical forms and ask them to avoid such expressions.
Annotators can raise questions.

Figure 8 shows the annotation interface for a single sentence. For every
sentence, annotators are provided with a single example image from NLVR and an
associated boolean value. Other images for the same statement from NLVR are
used as hidden validation examples. The annotator never sees these images.

The annotator can validate the program syntax and validate it within the
browser. The validation executes the program against the given image and all
hidden images. Validation passes only once the program returns the expected
boolean value for all examples, including the visible and the hidden ones. The
annotator can only submit their annotation after passing the syntax check and
validation. They can assign a confidence score to their annotation and provide
a comment.

Annotators can skip examples in case of doubt. When skipping, they need to
explicitly provide the reason. We assess the annotations by batch, then
randomly redistribute the skipped examples or examples with problematic
annotations to the annotators after the questions have been solved. We
iteratively communicate with the workers throughout the entire annotation
process.

##  Appendix C Experimental Setup Details

###  C.1 Learning Details

#### Hyperparameters

For C+BERT, we optimize using Adam (Kingma and Ba, 2015) with a learning rate
of 3e-4, except on TOWER-FLIPIT trained with PPO+SF and on SCATTER-FLIPIT,
where we use 3e-5. For ViLT, we use AdamW (Loshchilov and Hutter, 2019) with a
cosine scheduler and a base learning rate of 3e-5 for all experiments, except
on SCATTER-SCRATCH trained with PPO+SF where we use 3e-4. The learning rate is
warmed up for 1% of the maximal total training steps of 4M. We use patience
for early stopping. We set entropy to 0.3 for all our experiments. We use a
mini-batch of 64 actions for gradient updates. At each PPO iteration we sample
2,048 actions (i.e., for the internal update loop).

#### PPO+SF Details

PPO+SF is a simple variant of PPO that applies masking to all the actions
except for $\mathtt{STOP}$ when the agent reaches a state in which it will
receive a positive reward if it would select $\mathtt{STOP}$. PPO+SF allows
the learner to observe $\mathtt{STOP}$ with positive reward with higher
probability than with conventional PPO. A side effect of this masking is that
the learner often samples action with very low probability, which can lead to
exploding gradients. We clip the PPO ratio to address this. Formally, the
original PPO objective is:

$\displaystyle L(\theta)$ $\displaystyle=$ (2)
$\displaystyle\mathbb{E}_{t}\left[\min(r_{t}(\theta)\hat{A}_{t},\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\right]\;\;,$

where
$r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{old}(a_{t}|s_{t})}$,
$\hat{A}$ is the advantage function, and $\epsilon$ is a hyperparameter
(Schulman et al., 2017). In PPO+SF, we clip the ratio term $r_{t}(\theta)$ to
avoid very large value due to “force” sampling of actions with very low
probability:

$\hat{r}_{t}(\theta)=\min\biggl{(}r_{t}(\theta),M\biggr{)}\;\;,$ (3)

where $M$ is a threshold bounding the ratio. We use $\hat{r}_{t}(\theta)$ in
place of $r_{t}(\theta)$ for our experiments.

###  C.2 Inference Details

There are three action types $\mathtt{STOP}$, $\mathtt{ADD}$, and
$\mathtt{REMOVE}$. Each type take a different number of arguments:
$\mathtt{STOP}$ takes no arguments, $\mathtt{ADD}$ takes two arguments in
TOWER and five in SCATTER, and $\mathtt{REMOVE}$ takes one argument in TOWER
and two in SCATTER. During inference, actions $a$ are sampled from the agent
policy $a\sim\pi(\cdot|s,c)$, where $s$ is a state and $c$ is a context. We
decompose the probability of an action to be a product of its type and
arguments. This risks assigning generally lower probability to actions with
more arguments, because of the multiplicative decomposition. We avoid this by
sampling the required arguments as needed. We first sample an action type.
Depending on the action type, we sample the required arguments. In practice,
this means that when an argument slot is not used the probability of that
action marginalizes over all possible assignments to that argument slot.

##  Appendix D Additional Results and Analysis

###  D.1 Development Rollout Statistics

Figure 9 shows development rollout statistics for PPO+SF. The statistics
follow similar trends for these we show for PPO in Figure 6. Compared to PPO,
we observe more non-stopped rollouts for TOWER-FLIPIT when training with
PPO+SF. These rollouts often correspond to the model getting stuck in add-
remove loops.

Figure 9: Development statistics for PPO+SF. Clockwise from top left: rollouts
without a $\mathtt{STOP}$ action; rollouts with an invalid action; mean
actions per rollout; relative rate of $\mathtt{ADD}$/$\mathtt{REMOVE}$
actions.

###  D.2 Error Analysis

We analyze model errors by sampling 50 erroneous development examples for each
of the two SCATTER CMDPs trained with PPO:

SCATTER-SCRATCH with C+BERT

76% of the errors are due to invalid actions, and 24% due to early
termination. Among the invalid actions, 58% are due to trying to put an item
that cannot fit in the box, 24% are due to trying to perform an action on a
separator, and 18% due to trying to remove an object from a position that does
not include an object.

SCATTER-SCRATCH with ViLT

all errors are due to invalid actions. Among the invalid actions, 90% are due
to trying to perform an action on a separator, and 10% due to trying to remove
an object from a position that does not include an object.

SCATTER-FLIPIT with C+BERT

all errors are due to invalid actions. Among the invalid actions, 84% are due
to trying to trying to remove an object from a position that does not include
an object, 8% are due to trying to perform an action on a separator, 4% are
due to trying to put an item that cannot fit in the box, and 4% are due to
trying to add an object on top of an existing one.

SCATTER-FLIPIT with ViLT

90% of the mistakes are due to invalid actions, and 10% due to early
termination. Among the invalid actions, 44% are due to trying to remove an
object from a position that does not include an object, 34% due to trying to
perform an action on a separator, 10% due to trying to add an object on top of
an existing one, and 2% due to trying to put an item that cannot fit in the
box.

###  D.3 Analysis of Action Selection Bias

We observe that the trained models often exhibit bias towards specific action
arguments, which are sampled much more often than others during inference.
Figure 10 illustrates this by visualizing coordinate selection frequencies on
the development set for SCATTER CMDPs. While the presence of bias is
relatively persistent, the exact argument the models are biased towards vary.
This indicates generalization limitations of our learned policies, which
potentially converge to specific argument prematurely, and do not fully
utilize the entire action space. We observe that this bias leads to selecting
invalid actions, for example when attempting to place a large object on the
edge so it crosses image boundaries.

SCATTER-SCRATCH

SCATTER-FLIPIT

Figure 10: Frequency of selecting the grid cell $(x,y)$ in SCATTER CMDPs for
rollouts sampled on the development set. $x$-axis and $y$-axis show the $x$
and $y$ position of the cells in the SCATTER 19$\times$5 grid approximation.

###  D.4 Performance Analysis by Semantic and Syntactic Phenomena

Suhr et al. (2017) manually annotated 200 development examples for semantic
phenomena. Table 3 and Table 4 show the performance on this data of policies
trained with PPO and PPO+SF. We provide an example sentence for each category.
The two models mostly follow similar trends with respect to the categories on
which they perform above and below overall performance. A notable exception is
coordination, where ViLT performs above overall performance on 2/4 CMDPs,
while C+BERT always performs below. We also observe a difference between TOWER
and SCATTER on soft cardinality (e.g., …at least two …), with above average
performance on TOWER and below on SCATTER.

| TOWER-SCRATCH |  | TOWER-FLIPIT |  | SCATTER-SCRATCH |  | SCATTER-FLIPIT |
---|---|---|---|---|---|---|---|---
| Total | Correct % |  | Total | Correct % |  | Total | Correct % |  | Total | Correct % | Example
Semantics |  |  |  |  |  |  |  |  |  |  |  |
Cardinality (hard) | 98 | 67.3 78.6 |  | 480 |  36.7 70.8 |  | 35 | 31.4 37.1 |  | 119 |  00.0 05.0 | There are exactly four objects not touching any edge
Cardinality (soft) | 21 |  81.0 85.7 |  | 82 |  43.9 65.9 |  | 11 |  09.1 18.2 |  | 42 |  00.0 02.4 | There is a box with at least one square and at least three triangles.
Existential | 122 |  73.8 83.6 |  | 577 |  37.6 70.7 |  | 55 | 36.4 32.7 |  | 192 |  00.0 03.6 | There is a tower with yellow base.
Universal | 7 | 14.3 14.3 |  | 28 | 32.1 50.0 |  | 9 |  44.4 55.6 |  | 36 |  00.0 00.0 | There is a black item in every box.
Coordination | 19 | 31.6 52.6 |  | 86 | 29.1 70.9 |  | 15 | 20.0 40.0 |  | 55 |  00.0 00.0 | There are 2 blue circles and 1 blue triangle
Coreference | 3 |  00.0 33.3 |  | 10 | 30.0 30.0 |  | 3 |  00.0 33.3 |  | 9 |  00.0 00.0 | There is a blue triangle touching the wall with its side.
Spatial Relations | 93 |  77.4 87.1 |  | 438 |  39.7 69.2 |  | 39 |  41.0 30.8 |  | 128 |  00.0 05.5 | there is one tower with a yellow block above a yellow block
Comparative | 5 | 40.0 20.0 |  | 20 | 20.0 30.0 |  | 1 |  100.0 100.0 |  | 4 |  00.0 00.0 | There is a box with multiple items and only one item has a different color.
Presupposition | 17 | 47.1 58.8 |  | 74 | 35.1 68.9 |  | 22 |  40.9 40.9 |  | 78 |  00.0 05.1 | There is a box with seven items and the three black items are the same in shape.
Negation | 4 |  75.0 75.0 |  | 15 | 13.3 26.7 |  | 15 | 33.3 28.6 |  | 53 |  00.0 00.0 | there is exactly one black triangle not touching the edge
Syntax |  |  |  |  |  |  |  |  |  |  |  |
Coordination | 4 | 25.0 100.0 |  | 14 | 21.4 71.4 |  | 5 | 20.0 20.0 |  | 20 |  00.0 00.0 | There is a box with at least one square and at least three triangles.
PP Attachment | 44 |  84.1 093.2 |  | 215 |  42.3 72.1 |  | 2 |  00.0 33.3 |  | 7 |  00.0 14.3 | There is a black block on a black block as the base of a tower with three blocks.
Dev performance | 71.78 81.60 |  |  | 35.95 67.60 |  |  | 39.08 35.63 |  |  |  00.00 3.51 |
Table 3: Performance on a set of development examples annotated for semantic and syntactic categories by Suhr et al. (2017) for both models (C+BERT | ViLT) when trained with PPO. Dev performance refers to mean performance on the respective full development set. Results outperforming dev performance are in bold. | TOWER-SCRATCH |  | TOWER-FLIPIT |  | SCATTER-SCRATCH |  | SCATTER-FLIPIT |
---|---|---|---|---|---|---|---|---
| Total | Correct % |  | Total | Correct % |  | Total | Correct % |  | Total | Correct % | Example
Semantics |  |  |  |  |  |  |  |  |  |  |  |
Cardinality (hard) | 98 |  83.7 84.7 |  | 480 | 24.6 62.7 |  | 35 | 68.6 60.0 |  | 119 |  06.7 22.7 | There are exactly four objects not touching any edge
Cardinality (soft) | 21 |  81.0 81.0 |  | 82 |  29.3 63.4 |  | 11 | 27.3 27.3 |  | 42 |  07.1 26.2 | There is a box with at least one square and at least three triangles.
Existential | 122 |  86.9 82.8 |  | 577 | 25.8 64.1 |  | 55 |  70.9 67.3 |  | 192 |  05.2 27.1 | There is a tower with yellow base.
Universal | 7 |  042.9 100.0 |  | 28 | 14.3 57.1 |  | 9 | 44.4 44.4 |  | 36 |  05.6 16.7 | There is a black item in every box.
Coordination | 19 | 63.2 84.2 |  | 86 |  34.9 60.5 |  | 15 | 33.3 33.3 |  | 55 |  03.6 14.5 | There are 2 blue circles and 1 blue triangle
Coreference | 3 | 33.3 66.7 |  | 10 | 20.0 40.0 |  | 3 | 33.3 66.7 |  | 9 |  00.0 33.3 | There is a blue triangle touching the wall with its side.
Spatial Relations | 93 |  83.9 84.0 |  | 438 | 21.0 60.3 |  | 39 |  82.1 76.9 |  | 128 |  03.9 34.4 | there is one tower with a yellow block above a yellow block
Comparative | 5 | 40.0 80.0 |  | 20 |  00.0 40.0 |  | 1 |  100.0 100.0 |  | 4 |  00.0 25.0 | There is a box with multiple items and only one item has a different color.
Presupposition | 17 | 58.8 94.1 |  | 74 | 18.9 58.1 |  | 22 | 68.2 72.7 |  | 78 |  07.7 28.2 | There is a box with seven items and the three black items are the same in shape.
Negation | 4 | 75.0 75.0 |  | 15 | 13.3 53.3 |  | 15 |  80.0 66.7 |  | 53 |  03.8 28.8 | there is exactly one black triangle not touching the edge
Syntax |  |  |  |  |  |  |  |  |  |  |  |
Coordination | 4 | 75.0 75.0 |  | 14 |  28.6 57.1 |  | 5 | 40.0 40.0 |  | 20 |  00.0 10.0 | There is a box with at least one square and at least three triangles.
PP Attachment | 44 |  95.5 84.1 |  | 215 | 22.8 60.0 |  | 2 |  00.0 00.0 |  | 7 |  00.0 12.5 | There is a black block on a black block as the base of a tower with three blocks.
Dev performance | 80.98 84.05 |  |  | 27.22 65.09 |  |  | 70.12 64.37 |  |  |  08.31 27.48 |
Table 4: Performance on a set of development examples annotated for semantic and syntactic categories by Suhr et al. (2017) for both models (C+BERT | ViLT) when trained with PPO+SF. Dev performance refers to mean performance on the respective full development set. Results outperforming dev performance are in bold.


# The case of Quantum Gravity with Spontaneous Collapse of the Wave Function

Lajos Diósi  Wigner Research Centre for Physics, H-1525 Budapest 114, P.O.Box
49, Hungary
Eötvös Loránd University, H-1117 Budapest, Pázmány Péter stny. 1/A
<EMAIL_ADDRESS>
###### Abstract

When about half a century ago the concept of universal spontaneous collapse of
the wave function was conceived it was an attempt to alter standard non-
relativistic quantum physics. As such, it was largely ignored by relativistic
field theory and quantum gravity communities. A central motivation of
spontaneous collapse community has been to replace the standard collapse-by-
measurement that annoyed many. With few exceptions, it did not annoy the field
theory and quantum gravity communities. Concept of certain general-relativity-
related universal irreversibility in quantum field theory had been initiated
very long ago by Wheeler, Hawking and a few others independently from the
concept of spontaneous collapse. Lately the two concepts started to converge
and support each other.

##  1 Introduction

Allow the author to introduce the topics in a self-quotation from 2015 [1].
‘The inception of a universal gravity-related irreversibility took place
originally in quantum cosmology but it turned out soon that a universal non-
unitary dynamics is problematic itself. Independent investigations of the
quantum measurement postulate clarified that a non-unitary dynamics is of
interest already in the non-relativistic context. An intricate relationship
between Newton gravity and quantized bulk matter might result in universal
non-relativistic violation of unitarity - also called spontaneous decoherence.
The corresponding gravity-related spontaneous decoherence model is now on the
verge of detectability in optomechanical experiments. It is also a toy-model
of cosmic quantum gravitational non-unitarity, illuminating that the bottle-
neck of quantum gravity is the quantum measurement postulate instead of
quantum cosmology.’

We want to build our message around the appearance of the concept of
fundamental irreversibility in two separate fields of foundational theoretical
physics. One is quantum cosmology. It has no consistent theory valid down
towards the Planck scale, despite various efforts using various advanced
mathematical models. At the same time, semiclassical and heuristic approaches
imply that fundamental irreversibilities may be part of quantum cosmology. But
quantum irreversibility itself, especially when universal, happens to be non-
trivial both technically and conceptually. And here another research field
gains importance. Its tasks seem opaque compared to quantum cosmology’s. They
include, e.g., the measurement problem, the dynamics of wave function
collapse, the quantum-classical transition and hybrid dynamics, but the
relevant task is the quantum theory of macroscopia, coined as the Schrödinger
cat problem. You should discuss the options of macroobjects’ quantum behavior,
and you can do it non-relativistically, before you go for quantum cosmology!
The hypothesis of spontaneous universal collapse in massive degrees of
freedom, assuming a fundamental gravity-related irreverisibility, may have
perspectives for quantum cosmology as well, especially if fundamental
irreversibility remains part of it.

##  2 Fundamental irreversibility? — Two Communities

The two fundamental issues in question are the following: unified theory of
space-time with quantized matter (cf. quantum gravity, quantum cosmology) and
the physics of quantum measurement (cf. wave function collapse, quantum-
classical transition). These problems were considered unrelated for long time,
studied by two separate research communities. We refer to the first one as
quantum cosmologists and to other as measurement problem solvers or, with a
spectacular touch, as Schrödinger cat killers. Quantum cosmology has always
been part of main stream physics, using heavy artillery of mathematics, while
Schrödinger cat killers used light weapons and sometimes whimsical
identification of their problems.

Table 1:  Comparison of the two research fields and the status of fundamental
irreversibility.

QUANTUM COSMOLOGISTS | SCHRÖDINGER CAT KILLERS∗
---|---
field-string-membrane theorists, | quantum foundation experts,
for full relativistic | for non-relativistic
quantum gravity | spontaneous wave function collapse
of the Universe | of macroscopic bodies
within standard quantum theory | with modified quantum theory
unitary (reversible) | non-unitary (irreversible)
Fundamental Irreversibility? | Fundamental Irreversibility?
Possible: foamy vacuum, black holes | Mandatory: wave function collapse

${}^{*}\\!\\!$ Measurement Problem Solvers

###  2.1 Irreversible Quantum Gravity/Cosmology at the Planck Scale

Emergence of irreversibility _within standard physics_ was not derived by
exact derivations but by _heuristic arguments_. A selection of earliest
theoretical signatures could be the following. In 1936, Bronstein [2] was the
first to point out that space-time metric must have an ambiguity $\delta
g_{ab}$ of different nature from the dictum of Heisenberg uncertainty
relationships. Much later in 1962, Wheeler [3] proposed that space-time has a
certain foamy structure at the Planckian scale. After another decade,
Bekenstein [4] proved that black-holes behave thermodynamically, they have
entropy:

$S_{BH}=\frac{A}{4},$ (1)

where $A$ is the surface aera of the black-hole in Planck units (Boltzmann’s
constant $k_{B}=1$). Then Hawking [5] showed that, yes, in accordance with the
above entropy, black-holes emit thermal radiation of temperature

$T_{BH}=\frac{1}{8\pi M},$ (2)

where $M$ is the black-hole mass. Later, Hawking [6] went deeper and
conjectured that unitarity of standard quantum theory is lost due to the
instanton mechanism. The scattering processes must be non-unitary, the non-
unitary super-scattering operator $\$$ acts on the density operator
$\widehat{\rho}$:

$\widehat{\rho}\rightarrow\$\widehat{\rho}$ (3)

where the r.h.s. does not factorize into
${\widehat{S}}\widehat{\rho}{\widehat{S}}^{\dagger}$ with any unitary
scattering operator $\widehat{S}$. Instead of Hawking’s scattering process,
Banks, Susskind and Peskin [7] considered a detailed dynamical equation

$\frac{d\widehat{\rho}}{dt}=-i[\widehat{H},\widehat{\rho}]-\int\\!\\!\int[\widehat{Q}(x),[\widehat{Q}(y),\widehat{\rho}~{}]]h(x-y)d^{3}xd^{3}y,$
(4)

where $\widehat{H}$ is the Hamiltonian, $\widehat{Q}$ is a relativistic
quantum field and $h$ is a positive kernel. The authors show that the
irreversible term on the r.h.s. violates the energy-momentum conservation
which cannot be restored in local field theories.

###  2.2 Irreversible Quantum Mechanics for Massive Objects

Among diverse incentives, including a metaphysical discontent about von
Neumann’s quantum measurement theory, the only relevant motivation is the
problematic extendibility of quantum theory for massive degrees of freedom.
This led to _heuristic modifications of standard quantum physics_. According
to that, the unitarity is violated in massive degrees of freedom so that
macroscopically different superpositions, aka Schrödinger cat states like

$|\Psi\rangle=\frac{|f_{1}\rangle+|f_{2}\rangle}{\sqrt{2}},$ (5)

collapse spontaneously into one of the components. Milestones of the consept
could be the following.

In 1966, Károlyházy [15] proposed that, due to a conjectured spectrum of
space-time metric fluctuations $\delta g_{ab}$ at the Planck scale,
Schrödinger cat states collapse before they get too large. He outlined a naive
qualitative model of when and how the collapse happens. Independently and much
later in 1986, Ghirardi, Rimini and Weber [18] constructed a simple model of
universal spontaneous collapse in exact mathematical terms. They postulated
extreme rare and weak spontaneous collapses for the elementary constituents’
wave functions which in turn yield robust localization of the macrosopic
center-of-mass, i.e., the collapse of the Schrödinger cat state. The GRW model
has no reference to gravity. However, the present author [19, 20] proposed
gravity-related universal spontaneous collapse of massive superpositions, like
Schrödinger cats, based on a master equation:

$\frac{d\widehat{\rho}}{dt}=-\frac{i}{\hbar}[\widehat{H},\widehat{\rho}]-\frac{G}{2\hbar}\\!\\!\int\\!\\!\\!\int[\widehat{f}(x),[\widehat{f}(y),\widehat{\rho}~{}]]\frac{1}{|x-y|}d^{3}xd^{3}y,$
(6)

where $\widehat{f}$ is the non-relativistic quantized mass density field and
$G$ is Newton’s constant. The master equation corresponds to a certain
ambiguity $\delta\Phi=\tfrac{1}{2}c^{2}\delta g_{00}$ of the Newton potential.

###  2.3 Fundamental irreversibility? — Parallel pursuits

Thus, the desired quantum gravity and the unwanted Schrödinger cats led both
the heavy-armed relativistic and light-horse non-relativistic studies to the
same structure of heuristic master equations. And they suffer of the same
problem of spontaneous creation of energy and momentum. This coincidence is a
spectacular instance to illustrate that the two, apparently distant
theoretical tasks, share similar proposals and problems. Another instance has
been the surprising coincidence between the Schrödinger cat’s collapse rate
predicted by the non-relativistic master equation [19] and by the general
relativistic arguments of Penrose [10]. For possible reasons of coincidence,
see ref. [25].

Mention should be made of Gell-Mann and Hartle [9] who were perhaps the only
ones at the time to recognize that quantum cosmology needed a more general
measurement theory than von Neumann’s. For the relationship of their theory of
decoherent histories to spontaneous collapse theories, see [26]. Most
recently, hybrid quantum-classical master equations have been tested in
semiclassical cosmology by Oppenheim et al. [14], with numerous references to
spontaneous collapse models which are, in fact, the alternative formalism, see
[27].

Table 2:  A selection of parallel concepts related to fundamental irreversibility. | QUANTUM COSMOLOGISTS | SCHRÖDINGER CAT KILLERS
---|---|---
1936 | Bronstein [2]: ambiguity $\delta g_{ab}$ |
1962 | Wheeler [3]: space-time foam |
1966 |  | Károlyh. [15]: ambiguity $\delta g_{ab}$ collapses $\Psi$
1973 | Bekenstein [4]: black hole entropy |
1975 | Hawking [5]: black holes radiate |
1976 |  | Pearle [16]: searching collapse dynamics
1983 | Hawking [6]: instantons break unitarity |
1984 | BPS [7]: energy non-conservation | Gisin [17]: prototype collapse dynamics
1986 |  | GRW [18]: universal collapse
1987 |  | D. [19]: G-related decoherence
1989 | Ellis & al. [8]: wormholes collapse $\Psi$ | D. [20]: G-related collapse
1990 | G-M&H [9]: decoherent histories | GRP [21]: G-unrelated collapse
1996 | Penrose [10]: ambiguity $\delta g_{ab}$ collapses $\Psi$ |
2014 |  | Bedingham & al. [22]: relativisation
2016 | Kwon&Hogan [11]: holographic noise | Tilloy&D [23]: semiclassical G & collapse
2017 |  | Bassi & al. [24]: review of G-collapse
2021 | Sudarsky [12]: cosmology & collapse |
2022 | Anastopoulos&Hu [13]: G-decoherence |
| Oppenheim & al. [14]: hybrid dynamics |

##  3 Summary

The failures of quantum gravity models may not be due to the inadequacy of the
quantization methods tried so far, but the inadequacy of standard quantum
theory for macroobjects, e.g., the spontaneous decay of massive
superpositions, aka Schrödinger-cats. We recalled that the idea of fundamental
irreversibility has also been raised in quantum cosmology. The roots of
irreversibility may be common in quantum cosmology and in speculative models
of spontaneous collapse of the wave function. This may be so despite the fact
that one is predicted at the Planck scale while the other comes from non-
relativistic discussion of massive superpositions [28]. Considerations of
quantum cosmologists and advocates of spontaneous collapse (referred here
extravagantly as Schrödinger cat killers) will have to come closer to each
other. To support this suggestion, here are two quotes. The author (cat
killer) claimed ‘the measurement problem culminates in quantum cosmology”
[29]. The cosmologist claims ‘the principles of general relativity must
influence, and actually change, the very formalism of quantum mechanics’ [30].

This research was funded by the Foundational Questions Institute and Fetzer
Franklin Fund, a donor-advised fund of the Silicon Valley Community Foundation
(Grant No’s. FQXi-RFPCPW-2008, FQXi-MGA-2103), the National Research,
Development and Innovation Office (Hungary) “Frontline” Research Excellence
Program (Grant No. KKP133827), and the John Templeton Foundation (Grant
62099).

## References

## References

  * [1] Diósi L 2015 Gravity related spontaneous decoherence: from Wheeler-Bekenstein-Hawking to optomechanics (talk given in Erice) www.wigner.hu/$\sim$diosi/slides/erice2015.pdf
  * [2] Bronstein M 1936 Phys. Z. Sowjetunion 9 140
  * [3] Wheeler J A 1962 Geometrodynamics (New York: Academic Press)
  * [4] Bekenstein J D 1973 Phys. Rev. D 7 2333
  * [5] Hawking S W 1975 Commun. Math. Phys. 43 199
  * [6] Hawking S W 1982 Commun. Math. Phys. 87 395
  * [7] Banks T, Susskind L and Peskin M E 1984 Nucl. Phys. B 244 125
  * [8] Ellis J, Mohanty S and Nanopoulos D V 1989 Phys. Lett. B 221 113
  * [9] Gell-Mann M and Hartle J B 1993 Phys. Rev. D 47 3345
  * [10] Penrose R 1996 Gen. Relativ. Gravit. 28 581
  * [11] Kwon O and Hogan C J 2016 Class. Quantum Grav. 33 105004
  * [12] Sudarsky D 2021 Do Wave Functions Jump? Perspectives of the Work of GianCarlo Ghirardi, ed V Allori, A Bassi, D Dürr, N Zanghi (Cham: Springer Nature Switzerland) p 291
  * [13] Anastopoulos C and Hu B L 2021 AVS Quantum Science 4 015602
  * [14] Oppenheim J and Weller-Davies 2022 J. High Energ. Phys. 2022 80; Oppenheim J, Sparaciari C, Šoda B and Weller-Davies Z 2022 Preprint arXiv:2203.01982
  * [15] Karolyhazy F 1966 Nuovo Cim. 42 390
  * [16] Pearl P 1976 Phys. Rev. D 13 857
  * [17] Gisin N 1984 Phys. Rev. Lett. 52 1657
  * [18] Ghirardi G C, Rimini A and Weber T 1986 Phys. Rev. D 34 470
  * [19] Diósi L 1987 Phys. Lett. A 120 377
  * [20] Diósi L 1989 Phys. Rev. A 40 1165
  * [21] Ghirardi G C, Rimini A and Pearl P 1990 Phys. Rev. A 42 78
  * [22] Bedingham D, Dürr D, Ghirardi G, Goldstein S, Tumulka R and Zanghi N 2014 J. Stat. Phys. 154 623
  * [23] Tilloy A and Diósi L 2016 Phys. Rev. D 93 024026
  * [24] Bassi A, Grossardt A and Ulbricht H 2017 Class. Quantum Grav. 34 193002
  * [25] Diósi L 2022 AVS Quantum Sci. 4 015605
  * [26] Diósi L, Gisin N, Halliwell J and Percival I C 1995 Phys. Rev. Lett. 74 203
  * [27] Diósi L 2023 Preprint arXiv::2302.13418
  * [28] Diósi L 2018 Entropy 20 496; 2019 J. Phys. Conf. Ser. 1275
  * [29] Diósi L 1992 Quantum Chaos—Quantum Measurement (Springer) p 299
  * [30] Penrose R 2014 Found. Phys. 44 557


| Input
---
parameters
| Percentage
---
(%)
| Cumulative
---
percentage (%)
|
$P_{20}$, $P_{30}$, $c$, $w$ | 67.5 | 67.5 |  |
22.6 | 90.1 |  |
6.9 | 97.0 |  |
| 3.0 | 100 |  |
$c$, $w$ | 77.7 | 77.7 |  |
22.3 | 100 |  |
131313From left to right: input parameters used to realise the PCA, percentage
of variance, and cumulative percentage of variance.

##  Appendix E Ring emission in simulations

In Fig. LABEL:fig:anello is shown the ring emission observed in the central
regions (i.w. $r<$ 40 kpc) of the simulated clusters. As presented in Subsect.
8.4.3, this effect is related to the isotropic model used for the description
of the AGN feedback and has an impact on the estimation of the concentrations
of the simulated sample.