#  Eevee: An Easy Annotation Tool for Natural Language Processing

Axel Sorensen1 Siyao Peng2,3 Barbara Plank1,2,3 Rob van der Goot1
1 Department of Computer Science, IT University of Copenhagen, Denmark
2 Munich Center for Machine Learning (MCML), Munich, Germany
3 MaiNLP, Center for Information and Language Processing, LMU Munich, Germany
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Annotation tools are the starting point for creating Natural Language
Processing (NLP) datasets. There is a wide variety of tools available; setting
up these tools is however a hindrance. We propose Eevee, an annotation tool
focused on simplicity, efficiency, and ease of use. It can run directly in the
browser (no setup required) and uses tab-separated files (as opposed to
character offsets or task-specific formats) for annotation. It allows for
annotation of multiple tasks on a single dataset and supports four task-types:
sequence labeling, span labeling, text classification and seq2seq.111Code,
README and tutorials of Eevee are available on
https://github.com/AxelSorensenDev/Eevee, demo video at
https://www.youtube.com/watch?v=HsOsfckvnQo and the tool itself on
https://axelsorensendev.github.io/Eevee/

Eevee: An Easy Annotation Tool for Natural Language Processing

Axel Sorensen1 Siyao Peng2,3 Barbara Plank1,2,3 Rob van der Goot1 1 Department
of Computer Science, IT University of Copenhagen, Denmark 2 Munich Center for
Machine Learning (MCML), Munich, Germany 3 MaiNLP, Center for Information and
Language Processing, LMU Munich, Germany<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Annotated datasets are of paramount importance to the Natural Language
Processing (NLP) community. Their use is at the core of research, e.g. for
training models, evaluating models, and analyzing trends. One of the first
considerations when creating an annotated dataset is which annotation tool to
choose. There is a variety of (open-source) tools readily available with
extensive feature-sets. We were motivated by the following observed
difficulties with existing tools when designing Eevee:

  * •

Most existing tools use tool-specific data formats, often with the main
annotation happening on the character level. For token-based tasks, the
annotator thus has to make a (tediously) precise selection of the token
boundaries. Furthermore, many NLP tools expect token-level inputs (for
example, for POS tagging, parsing, NER, and relation extraction). To obtain
annotations on the token level, an often cumbersome conversion is necessary.

  * •

Existing tools often require an installation which is especially problematic
on constrained (organization) computers, where there might be no administrator
access.

  * •

Although many of the advanced features (like active learning) can lead to
faster annotation over time, they require some setup time and more time for
the annotators to get used to the tool. Time is costly in annotation; in many
cases, annotators only annotate a small amount of data. Furthermore, most
strategies to increase the speed of annotation (for example active learning)
could lead to an additional bias signal for the annotator (Section 7).

  * •

For many tasks, there are task-specific tools; for example for UD there is
list of available annotation
tools.222https://universaldependencies.org/tools.html#annotation-tools
Instead, we focus on a generalizable and flexible tool. Eevee supports a total
of four task types: sequence labeling, span labeling, text classification, and
sequence to sequence (Section 4).

Figure 1: A screenshot of the setup page of Eevee with multiple tasks. The
user currently configures the NER task.

Based on these observations, we propose Eevee: a simple, free, and flexible
annotation tool built around tab-separated files. It is written in Javascript
and runs directly in the browser. It can also be saved as a desktop
application and run offline. The intuitive interface allows novice users to
import a dataset and set up multiple annotation tasks quickly. The graphical
user interface has two main pages: the setup page (Section 2) and the
annotation page (Section 3). It supports tab-separated files and raw text
input (Section 4.1). We perform a case study on NER annotation with the System
Usability Scale from usability engineering (Section 6). Finally, we compare
Eevee to other toolkits (Section 7).

##  2 Setup page

Figure 1 illustrates the setup page where the user can define the annotation
environment. Tasks can be configured in the task field (Figure 1, top right),
allowing the user to specify the input column (for the input text) and output
column (for the target task), as well as adding the desired labels. Labels can
also be imported automatically from the annotated file (if it already contains
annotations), and a default label can be set for empty annotations. For
utterance-level tasks (i.e. classification), the annotation is stored in a
comment above the text, in the form “# intent = inform” (see also Figure 4).
To facilitate reproducibility and improve the ease of setup, the tool allows
the import and export of all settings to configuration files that users can
create for predefined tasks (more details in Section 4.1).

Once a dataset has been imported, the tabular data field (Figure 1, bottom
right) offers a simple overview of the raw data belonging to each utterance.
The user can add new columns or remove existing ones to achieve the desired
result. This makes Eevee an easy-to-use tool for extending or editing tab-
separated data as well (see Section 4.1). Once the data and tasks are ready,
the user simply clicks “Annotate” (Figure 1, bottom left) to continue to the
“Annotation page” (see Section 3).

Figure 2: Annotation example with the keyboard setting.

##  3 Annotation page

Figure 2 illustrates an example of a NER task in the annotation interface. The
user is presented with a clean, minimal annotation environment. The annotation
process has been designed with efficiency in mind, enabling the user to
navigate the interface also through keyboard shortcuts.

The navigation bar (Figure 2, bottom right) enables navigation between
utterances and, similar to Prodigy Montani and Honnibal (2018), setting the
status of a given task for a given utterance. The status can be set to four
values: completed, wrong, unsure, and cleared (i.e. none). This overall status
is reflected in the progress bar (Figure 2, top right), allowing the user to
spot missing and unsure annotations easily. The progress bar is also useful
when continuing annotation on a previously saved annotation file.

Eevee provides two different annotation modes for label-based tasks: the
keyboard mode and the search mode. With the keyboard mode (Figure 2), the user
can use the number keys to select labels and click/select the part of the
input where the label should apply (for utterance-level tasks, simply pressing
the number key is sufficient). In search mode, a small pop-up appears after
selecting a word or span (see Figure 3), allowing the user to find the desired
label quickly. If there are more than ten labels, Eevee defaults to search
mode. Finally, the annotation file can be exported (Figure 2, bottom left).
The current datetime can be appended to distinguish between different export
versions.

Figure 3: Searching for labels with a navigation bar.

##  4 Tasks

In this section, we will describe the annotation data format used by Eevee
(for import and export, importing text files is also supported), and we will
discuss all the supported task types as well as the configuration files for
the setups.

###  4.1 Data Format

There are many different data formats used in NLP, which are often task-
specific. Eevee is based on the well-established tab-separated files
ubiquitously used in the NLP field. These are also sometimes called conll-like
files, based on the formats used in the CoNLL shared tasks Tjong Kim Sang and
De Meulder (2003); Buchholz and Marsi (2006). This format (example in Figure
4) uses empty lines to separate utterances or sentences and puts one token per
line. Annotations and input tokens are separated by a tab character. Comments
and utterance-level information are included above the texts and are prefixed
with a # character.

⬇

# sent_id = gameboy-1

# intent = inform

1 What PRON O

2 ? PUNCT O

3 Eevee PROPN B-MISC

4 is AUX O

5 evolving VERB O

6 ! PUNCT O

\par# sent_id = gary-1

# intent = goodbye

1 Smell VERB O

2 ya PRON O

3 later ADV O

4 ! PUNCT O

Figure 4: Example of annotated tab-separated file with seq (POS in column 3),
span (NER in column 4), and class (intent classification in the comments)
tasks.

###  4.2 seq task-type

In sequence labeling tasks (seq), we annotate a single label per token, such
as POS tagging or token-level language identification.

###  4.3 span task-type

span-labeling tasks are where spans are annotated as sequences of tokens (e.g.
NER). Most other tools supporting this task type (e.g. Stenetorp et al., 2012;
Nakayama et al., 2018) have character-level annotations, although spans
normally operate on token-borders. An advantage of Eevee is that it
automatically selects the entire token if part of the token is selected,
making annotation easier and faster as the annotators do not have to drag the
mouse to the exact character of the token boundary. The user can simply select
a label (either by clicking or pressing the corresponding number key) and then
click the desired token (i.e. any character within the token) or select a span
of tokens.

###  4.4 class task-type

Eevee also supports classification tasks on the utterance level. Labels are
included as a comment above the text (e.g. intents in Figure 4). The format is
# [UNIQUE NAME] = [LABEL], following typical meta-data format as used in
conll-like formats. Usage is similar to the previous two labeling tasks,
except that the user does not need to select a part of the utterance.
Keyboard-only annotation is thus straightforward: the user can simply press a
number key to select desired class labels and use the arrow keys to navigate
the data.

###  4.5 seq2seq task-type

The seq2seq task type allows for text to text tasks (e.g. translation,
question answering, summarization). This is currently the only task type
without a list of provided labels; the user can directly type the target text
in a text field. The annotations are utterance level and thus also saved in
the comments.

⬇

[{"title":"NER",

"type":

{"name":"seq_bio",

"isWordLevel":true},

"output_index":"4",

"input_index":"1",

"labels":["LOC","MISC","ORG","PER"],

"id":0}]

Figure 5: An example of the configuration file format. The configuration file
is a json file consisting of an array of tasks. Each task has a title, a type,
input and output indices, and finally its corresponding labels.

###  4.6 Config Files

Because Eevee runs entirely in the browser, it will not internally save the
setup for the current annotation task. Therefore, it supports configuration
files. These configuration files are in json format, and can thus easily be
inspected by administrators, and are easy (i.e. small) to be distributed. An
example of the configuration file format for named entity recognition (NER) is
given in Figure 5.

Figure 6: The results from the System Usability Scale Questionaire. The x-axis
shows their agreement with a given statement, while the y-axis shows each
item.

##  5 Compatability with other services

A recent development is the Huggingface datasets library Lhoest et al. (2021),
which has indexed 62K+ datasets in two years. This library does not share the
text directly but through a Python API. We provide a convenient Python script
that automatically downloads data from the datasets library and converts it to
the tab-separated format of Eevee.

One of the toolkits that operates on tab-separated formats is MaChAmp van der
Goot et al. (2021), which is focused on multi-task learning. MaChAmp supports
all the tasks that are included in Eevee. For convenience, we provide a
conversion script that takes Eevee files as input and outputs a MaChAmp
configuration file and the corresponding training command.

| Brat | Potato | Doccano | Prodigy | Eevee
---|---|---|---|---|---
| Stenetorp et al. (2012) | Pei et al. (2022) | Nakayama et al. (2018) | Montani and Honnibal (2018) |
Open Source | ✓ | ✓ | ✓ | ✗ | ✓
Character level | ✓ | ✓ | ✓ | ✓ | ✗
Token level∗ | ✗ | ✗ | ✗ | ✓ | ✓
Utterance level | ✗ | ✓ | ✓ | ✓ | ✓
Data-format | standoff | json | json | json/csv | conll
Runs on | local | local | local | cloud | browser
Active learning | ✗ | ✓ | ✗ | ✓ | ✗
User management | ✗ | ✓ | ✓ | ✓ | ✗

Table 1: We only list the annotation export data files in this table, most
tools (including Eevee) also support importing .txt files. ∗ Note that
character level annotations are commonly used for token/span level tasks. But
as noted in Section 4.3, this requires more efforts for annotation and
conversion of data formats.

##  6 System Usability Study

###  6.1 Procedure

To assess the usability of Eevee, we conduct a case study with two annotators
on two tasks, named entity annotation (span labeling), and German dialect
identification (classification). Before annotating with Eevee, annotators
spent four months labeling named entities (NE) directly on tab-separated text
files in a text editor using BIO encoding and dialect identification (DID)
labels as utterance-level metadata. In this case study, we ask both annotators
to conduct the same NE and DID annotation tasks on a set of new documents,
similar to previous ones but using the newly introduced Eevee.

During Eevee training, we present a 12-minute tutorial video explaining the
setup and annotation pages to the annotators and provide them with tab-
separated unannotated files and the json configuration files. Two annotators
separately annotate the same eight documents, four from Wikipedia (wiki) and
four from Twitter (X, tweet), summing up to 14.2K tokens and 16 working hours
per person.333Annotators are hired student assistants and paid according to
national compensation tables.

###  6.2 Results

The System Usability Scale (SUS) was introduced as a quick and reliable tool
to measure the usability of user interfaces Brooke (1995). It consists of a
10-item questionnaire with 5 responses ranging from ‘Strongly Agree’ to
‘Strongly Disagree’. SUS has become an industry standard and can be validly
used with small sample sizes. Therefore, we evaluate the usability of Eevee
using SUS.

The responses given by both annotators (P1 and P2) are shown in Figure 6. The
ratings of the annotators result in total SUS scores of 75.0 and 87.5, both
above the average of 68.0 Brooke (2013). The standard method for interpreting
these scores is to look at which percentile they fall compared to other
systems. As we are not aware of SUS being used for annotation tools, we can
only compare to more general figures, where our average of 81.25 ranks at the
top 10% and indicates a good (close to excellent) usability Bangor et al.
(2009). We also qualitatively survey annotators’ experience and opinions after
two weeks of annotation. Both annotators appraise that the tool is easy to
learn and use and found it pleasant to work almost exclusively with the
keyboard in a lightweight interface. Both annotators responded that they would
use Eevee for their next annotation jobs.

Since annotators typically spend many hours in an annotation environment, it
is important that an annotation tool is built with user experience in mind. We
encourage existing and future tools to consider usability studies such as SUS.

##  7 Comparison to other annotation toolkits

We compare Eevee to other available toolkits in Table 1. While Eevee does not
have the most functionality, it does clearly allow for a simple setup for
token-level tasks. Also, Eevee provides keyboard shortcuts for annotation
speed.

Other techniques for improving annotation speed need more tuning and setup and
could lead to biases. For example, active learning could lead to model bias
Berzak et al. (2016) and coloring relevant words for a task Pei et al. (2022)
could lead to biases towards these indicators. We leave the user management up
to the organizer of the annotation efforts and prioritize the simplicity in
tool setup. Furthermore, since Eevee does not need installation, it does not
store or send any data to the network, which is beneficial for data privacy.

##  8 Conclusion

We introduce Eevee, an annotation toolkit focused on easy setup and usability.
It runs directly in the browser and allows for annotation of multiple tasks.
In addition, it provides convenience scripts for usage with other libraries.
Eevee’s main distinguishing features, in contrast to other toolkits, are the
simplicity of its setup and use, as well as annotation directly on the token
level (tab-separated files). To evaluate the tool, we conducted a case study
using the System Usability Scale, resulting in high usability scores. We also
qualitatively surveyed the annotators’ experience and noted that they would
prefer to use the tool again for annotation.

## Acknowledgements

We would like to thank Huangyan Shan and Marie Kolm for their invaluable
feedback on Eevee and Mike Zhang for giving feedback on earlier drafts of this
paper. Huangyan Shan, Marie Kolm, Siyao Peng and Barbara Plank are supported
by ERC Consolidator Grant DIALECT 101043235.

## Limitations

We acknowledge that Eevee assumes gold token detection (and annotates on the
token level for seq and span). For languages/datasets where tokenization is
challenging, this would require a first pass of tokenization annotation before
importing the data into Eevee. Furthermore, the input is constrained to text
in Unicode font, which is unavailable for some languages.

## References

  * Bangor et al. (2009) Aaron Bangor, Philip Kortum, and James Miller. 2009.  Determining what individual SUS scores mean: Adding an adjective rating scale.  _Journal of usability studies_ , 4(3):114–123.
  * Berzak et al. (2016) Yevgeni Berzak, Yan Huang, Andrei Barbu, Anna Korhonen, and Boris Katz. 2016.  Anchoring and agreement in syntactic annotations.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2215–2224, Austin, Texas. Association for Computational Linguistics.
  * Brooke (1995) John Brooke. 1995.  SUS: A quick and dirty usability scale.  _Usability Evaluation in Industry_ , 189.
  * Brooke (2013) John Brooke. 2013.  SUS: a retrospective.  _Journal of Usability Studies_ , 8:29–40.
  * Buchholz and Marsi (2006) Sabine Buchholz and Erwin Marsi. 2006.  CoNLL-X shared task on multilingual dependency parsing.  In _Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X)_ , pages 149–164, New York City. Association for Computational Linguistics.
  * Lhoest et al. (2021) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021.  Datasets: A community library for natural language processing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 175–184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Montani and Honnibal (2018) Ines Montani and Matthew Honnibal. 2018.  Prodigy: A new annotation tool for radically efficient machine teaching.
  * Nakayama et al. (2018) Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018\.  doccano: Text annotation tool for human.  Software available from https://github.com/doccano/doccano.
  * Pei et al. (2022) Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. 2022.  POTATO: The portable text annotation tool.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 327–337, Abu Dhabi, UAE. Association for Computational Linguistics.
  * Stenetorp et al. (2012) Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012.  brat: a web-based tool for NLP-assisted text annotation.  In _Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 102–107, Avignon, France. Association for Computational Linguistics.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_ , pages 142–147.
  * van der Goot et al. (2021) Rob van der Goot, Ahmet Üstün, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021.  Massive choice, ample tasks (MaChAmp): A toolkit for multi-task learning in NLP.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , pages 176–197, Online. Association for Computational Linguistics.


# Speaker conditioning of acoustic models using affine transformation for
multi-speaker speech recognition

###### Abstract

This study addresses the problem of single-channel Automatic Speech
Recognition of a target speaker within an overlap speech scenario. In the
proposed method, the hidden representations in the acoustic model are
modulated by speaker auxiliary information to recognize only the desired
speaker. Affine transformation layers are inserted into the acoustic model
network to integrate speaker information with the acoustic features. The
speaker conditioning process allows the acoustic model to perform computation
in the context of target-speaker auxiliary information. The proposed speaker
conditioning method is a general approach and can be applied to any acoustic
model architecture. Here, we employ speaker conditioning on a ResNet acoustic
model. Experiments on the WSJ corpus show that the proposed speaker
conditioning method is an effective solution to fuse speaker auxiliary
information with acoustic features for multi-speaker speech recognition,
achieving +9% and +20% relative WER reduction for clean and overlap speech
scenarios, respectively, compared to the original ResNet acoustic model
baseline.

Index Terms—  Affine transformation, overlap speech recognition, feature-wise
linear modulation, multi-speaker recognition, acoustic modeling

##  1 Introduction

Multi-talker speech recognition is focused on recognizing individual speech
sources from overlap speech, and is one main challenge for current ASR systems
[1, 2, 3, 4, 5, 6, 7, 8]. Current solutions for multi-speaker speech
recognition can be categorized into two main approaches: _(i)_ performing
front-end speech processing based on separation on the overlap speech, then
applying ASR to the separated speech signals [9, 10, 11, 12, 13, 14, 15]; or
_(ii)_ skipping the explicit separation step and developing a multi-speaker
speech recognition system directly using either hybrid [16, 17,
kanda2019simultaneous] or end-to-end [18, 19] ASR frameworks. Recently, an
end-to-end multi-speaker speech recognition system was proposed based on
Transformers [1]. This approach achieved considerable improvement at the
expense of more computational cost for a reasonable temporal resolution. In
another study [20], overlap speech was considered as a mismatch condition of
the clean speech recognition scenario, and teacher-student training was
employed for transfer learning from clean to overlap speech. The main drawback
of this approach is requiring training sets with parallel clean and overlapped
speech, which is difficult to collect in real-world applications [21].
Recently, several studies [16, 22, 23] have used speaker-specific embeddings
to learn a frame-level mask for the target speaker which suppresses
interfering speech. Although these approaches use the additional speaker-
specific information to guide the ASR system, their main limitation is that
they assume only one speaker is active in each Time-Frequency bin.

To address the challenges of single-channel multi-speaker speech recognition,
in this study, we focus on speaker conditioning of the Acoustic Model (AM) by
performing an affine transformation. In contrast to former approaches which
employ speaker embedding to estimate speaker-specific masks, we propose to use
speaker embedding to compute parameters of the affine transformation, allowing
the acoustic model to conduct its computation in the context of the desired
speaker auxiliary information. The proposed speaker conditioning method is a
general approach and can be applied to any AM architecture. In this paper, we
employ speaker conditioning on a ResNet acoustic model in hybrid DNN-HMM
setup. Experiments are performed on WSJ corpus, achieving +9% and +20%
relative WER reduction for clean and overlap speech scenarios, compared to the
original ResNet acoustic model. The contributions of this paper are threefold:

  * •

Proposing speaker conditioning of the ResNet acoustic model using an affine
transformation.

  * •

Comparing the proposed method with alternate feature-wise acoustic model
transformations such as conditional biasing and middle feature-map fusion.

  * •

Evaluating the performance of the proposed speaker conditioned ASR trained on
an alternate input feature called Wav2Vec representation.

The remainder of this paper is organized as follows. In Sec.2, the problem is
outlined and proposed method described. Sec.3, presents experiments and
results. Finally the conclusions are discussed in Sec.4.

Fig. 1: The proposed speaker conditioned ResNet18 acoustic model using Affine
Transformation (AT) blocks.

##  2 Single-channel multi-speaker ASR

Multi-speaker speech recognition can gain substantial improvement by deploying
other sources of information such as speaker identity in addition to acoustic
features [21, 22]. However, designing an efficient method to fuse a
combination of multiple sources (i.e., acoustic features and speaker
embeddings) to obtain higher quality and improved information is still a
challenging task. Additionally, capturing complex interactions between
multiple sources should maintain a balanced compromise between model/network
computational cost and performance. A popular method to address this problem
is to use a feature-wise transformation [24] which can model the complex
relation between speaker-specific characteristics and acoustic features in a
multi-speaker speech scenario to identify and recognize the desired speaker in
the mixed speech recording. This transformation can be performed in several
manners such as _conditional biasing_ , _conditional scaling_ , and
_conditional affine transformation_. In this section, we focus on a
conditional affine transformation which is a more general approach. The
aforementioned conditional biasing and scaling are two specific examples of
conditional affine transformation.

###  2.1 Conditional affine transformation

Affine Transformation (AT) influences the output of the acoustic model network
by applying a linear modulation to the network’s intermediate features. This
modulation is parameterized by shifts and coefficients obtained based on
speaker-specific embedding. Let $x$ be a context-expanded window of acoustic
features for overlap speech, and $y_{s}$ be a phoneme label or a senone
alignment (i.e., from GMM-HMM) for the target-speaker speech signal. DNN
acoustic models are used estimate the posterior probability as:

$p(y_{s}|x,s)=DNN(x,z_{s}),$ (1)

where DNN is typically trained to maximize the log probability of the phoneme
alignment or minimize the cross-entropy error, and $s$ is the target speaker
with an x-vector [25] embedding $z_{s}$. In this study, the original ResNet18
model is considered as our baseline. Next, affine transformation layers are
inserted into ResNet18 network to build the speaker conditioned acoustic
model. The scale and bias factors of AT are estimated by a two-layer fully
connected network $h$ based on x-vector $z_{s}$ as:

$(\alpha_{i,c},\;\beta_{i,c})=h(z_{s})$ (2)

where $i$ and $c$ refer to the $i$-th data sample in the minibatch, and the
$c$-th channel feature map. Once $\alpha_{i,c}$ and $\beta_{i,c}$ are
estimated, they are used to modulate the ResNet’s intermediate activations
$F_{i,c}$ as:

$AT(F_{i,c}^{l}|\alpha_{i,c},\beta_{i,c},F_{i,c}^{l-1})=\alpha_{i,c}\odot
F_{i,c}^{l-1}+\beta_{i,c}$ (3)

where $AT$ and $l$ represent the Affine Transformation, and network’s layer.
The proposed speaker conditioned ResNet18 is shown in Fig.1. The speaker
embedding x-vector is submitted to the network $h$ to estimate a $[B,1920]$
matrix which is $(\alpha_{i,c}$, $\beta_{i,c})$ pairs of AT layers. Each AT
layer receives two inputs: the previous layer output, and the $(\alpha_{i,c}$,
$\beta_{i,c})$ pair. The dimension of $\alpha_{i,c}$ and $\beta_{i,c}$ is
$[B,C]$ each. In the AT layer, each channel of the extracted feature map is
scaled by $\alpha_{i,c}$ and shifted by $\beta_{i,c}$ to modulate the feature-
map distribution of activations based on the target-speaker embedding.

Table 1: Comparing our ResNet18 acoustic model baseline with other approaches
on WSJ (WER in %).

System | Dev-93 | Eval-92
---|---|---
Lee et al. 2021 [26] | 12.0 | 9.9
Higuchi et al. 2020 [27] | 15.4 | 12.1
Chi et al. 2020 [28] | 13.7 | 11.4
Rouhe et al. 2020 [29] | 13.2 | 9.3
Sabour et al. 2018 [30] | - | 9.3
Borgholt et al. 2020 [31] | - | 9.3
Park et al. 2019 [32] | - | 7.8
Our baseline | 12.1 | 7.9

Table 2:  WER of the proposed speaker conditioned ResNet18 acoustic model with Affine Transformation (AT) in different settings. Each experiment is repeated three times and the average WER is reported. | Simulated overlap speech based on Dev-93 | Clean speech
---|---|---
Acoustic model | 0dB | 5dB | 10dB | 15dB | 20dB | 25dB | Dev-93 | Eval-92 | Eval-93
ResNet18 (baseline) | 65.06 | 58.29 | 47.03 | 34.72 | 24.36 | 17.62 | 12.14 | 7.92 | 10.81
ResNet18 + AT (proposed) | 63.83 | 55.83 | 43.30 | 29.90 | 20.34 | 15.15 | 11.50 | 7.66 | 9.64
ResNet18 + AT (bias=0) | 63.69 | 55.23 | 43.57 | 30.15 | 19.95 | 15.19 | 11.75 | 7.66 | 9.57
ResNet18 + AT (scale=1) | 65.10 | 57.66 | 46.39 | 32.79 | 22.29 | 16.65 | 12.25 | 7.91 | 10.57
ResNet18 + AT (sigmoid(scale)) | 64.63 | 56.66 | 45.33 | 31.78 | 21.680 | 16.250 | 11.993 | 7.803 | 10.263
ResNet18 + AT (tanh(scale)) | 64.53 | 56.82 | 45.27 | 31.64 | 21.19 | 16.02 | 11.99 | 7.72 | 9.97
ResNet18 + AT (Block1) | 63.68 | 55.33 | 43.31 | 29.44 | 19.51 | 14.67 | 11.56 | 7.49 | 9.85
ResNet18 + AT (Block 1-2) | 63.50 | 55.19 | 43.60 | 29.33 | 19.79 | 14.85 | 11.33 | 7.50 | 9.88
ResNet18 + AT (Block 1-3) | 63.51 | 55.29 | 43.35 | 29.65 | 19.87 | 14.88 | 11.49 | 7.55 | 9.93
ResNet18 + AT (Block 4) | 64.66 | 57.66 | 47.28 | 33.86 | 23.27 | 16.54 | 11.64 | 8.19 | 10.47

##  3 Experiments and results

In this section, we investigate the performance of the proposed speaker
conditioning method presented in Fig.1 on WSJ corpus. In order to conduct the
experiments, clean tr-si284 is used in the training phase for all acoustic
models. We partitioned tr-si284 into a training set $(90\%)$ and a held-out
cross-validation set $(10\%)$. ASR performance for different acoustic models
are reported in terms of Word-Error-Rate (WER) on clean dev-93, eval-93, and
eval-92. Additionally, overlap speech is generated based on dev-93 by
selecting random utterances from random speakers and adding them with Signal-
to-Interference Ratio (SIR) ranging from $0$ to $25$dB with increments of
$5$dB. The baseline acoustic model is ResNet18 with $3400$ output senones. The
network parameters are updated by the gradients of the cross entropy loss
using Stochastic Gradient Descent (SGD) optimizer with a momentum of $0.9$ and
initial learning rate $0.01$. The training process is completed by performing
early stopping [33]. The maximum number of epochs is set to $100$, batch size
$1024$ context-expanded frames; learning rate is decreased by $50\%$ if the cv
loss improvement is less than $0.01$ for $3$ successive epochs. The early
stopping is performed if no improvement is observed on the cv loss once the
learning rate has decayed $6$ times. 13-dim MFCC computed over a $25ms$ window
with $10ms$ shift with a $20$ frame context ($10$ frames on each side) is used
for training the acoustic model. Consistent with the standard Kaldi recipe for
WSJ, we use the trigram language models provided by LDC for WSJ data. In order
to minimize the effect of parameter initialization on the acoustic model and
final WER, we repeat each experiment three times with different initial
parameters.

Table 3:  WER of the proposed speaker conditioned ResNet18 based on Affine Transformation (AT) compared to other fusion techniques. Each experiment is repeated three times and the average WER is reported. | Simulated overlap speech based on Dev-93 | Clean speech
---|---|---
Signal-to-Interference Ratio | 0dB | 5dB | 10dB | 15dB | 20dB | 25dB | Dev-93 | Eval-92 | Eval-93
ResNet18 + Conditional biasing | 64.66 | 57.20 | 45.74 | 32.75 | 23.54 | 17.87 | 13.10 | 8.44 | 11.22
ResNet18 + Middle fusion | 63.94 | 57.51 | 47.81 | 34.02 | 23.19 | 16.68 | 11.81 | 8.24 | 10.64
ResNet18 + AT (proposed) | 63.68 | 55.33 | 43.31 | 29.44 | 19.51 | 14.67 | 11.56 | 7.49 | 9.85
|
---|---
Fig. 2:  WER of ResNet18 baseline and proposed ResNet18 + AT trained on MFCC
and Wav2Vec input features.

Performance of the ResNet18 baseline is compared with recent studies in Table
1. The main purpose of this comparison is to ensure that our baseline achieves
a competitive performance compared to recent studies, and it is seen we have a
strong starting point for further developing our proposed speaker conditioning
technique. There are other approaches that leverage transfer learning, semi
supervised learning, or more advanced language models to achieve further
improvement. However, since we focus on speaker conditioning of acoustic
model, we train our ResNet18 acoustic model only on tr-si284, and use Kaldi
for training the language model. The first row of Table 2 presents performance
of the baseline on overlap speech, which is severely degraded. Therefore, we
build on the ResNet18 acoustic model baseline and apply our Affine
Transformation (AT) layers as depicted in Fig.1. The results for speaker
conditioned ResNet using AT are presented in the second row of Table 2 which
shows $+2\%$ relative improvement for severe overlap speech recordings (i.e.,
$0$dB) and an average of $+5\%$ relative improvement on clean test sets. Since
AT effectively performs speaker-adaptation, the trained acoustic model is
tuned to the target speaker, therefore, it achieves better performance even on
the clean test sets. The maximum relative improvement is achieved for input
SIR $20\%$ in which the level of overlap speech is neither too severe nor too
easy for the acoustic model; therefore, the target-speaker auxiliary
information can be very helpful in improving performance.

Moreover, the effect of $\alpha$ and $\beta$ is studied separately by setting
$\alpha=1$ and $\beta=0$. The result in Table 2 manifest that the
effectiveness of the conditional Affine Transformation can be mainly
attributed to the scale coefficient rather than the shift parameter.
Therefore, we further investigate the effect of $\alpha$ by restricting its
value to $(0,1)$ using Sigmoid, and $(-1,1)$ using $\tanh$ function.
Nevertheless, the results reported in Table 2 reveal that unrestricted
$\alpha$ achieves better performance which may be due to the flexibility it
provides for the network to learn the range that best suits the data. So far,
the AT layers have been applied to all ResNet18 blocks (each dashed
rectangular in Fig. 1 is considered as a block). To find the best network
depth in which AT layers are most effective, several experiments are conducted
with AT only applied to specific individual blocks. Based on these
experiments, the AT layers are most effective when applied only to the first
block (block1), and least effective when only applied to the last block
(block4). However, applying AT layers to the first two blocks (block1-2) and
the first three blocks (block 1-3) did not improve ASR performance, while it
differently adds computational cost. To summarize our findings based on the
experiments, unrestricted-scale Affine Transformation applied to the initial
blocks of the ResNet18 acoustic model achieves the best overall results while
simultaneously maintaining the lowest computational cost.

Next, the proposed method is compared with other speaker conditioning
techniques in Table 3. Conditional biasing refers to adding speaker
information (x-vector) as a bias to the acoustic features in the first hidden
layer. Middle fusion refers to adding the x-vector to the intermediate
extracted feature map after the second block. Therefore, the intermediate
feature map is conditioned before entering block 3 for extracting further
higher-level features adapted to the target speaker. As shown in Table 3, the
proposed speaker conditioning based on Affine Transformation outperforms all
other approaches in both clean and overlap speech scenarios.

So far, the focus of this study has been on designing the acoustic model.
However, performance of the acoustic model can further improve by deploying
more robust input features other than MFCC. In the final section, we evaluate
the proposed method trained on noise-invariant Wav2Vec features [34]. Wav2Vec
representation has been trained on large amounts of unlabeled audio data in an
unsupervised manner. Fig. 2 (left) shows the WER of the baseline ResNet18
trained on MFCC and Wav2Vec features, which manifests the effectiveness of
Wav2Vec in reducing the WER across all test sets in the absence of speaker
auxiliary information. The highest improvement is achieved for overlap speech
with SIR $15$dB, which is $+11\%$ absolute improvement in WER. Fig.2 (right)
depicts the WER of the proposed speaker conditioned ResNet18 trained on MFCC
and Wav2Vec. Similar to the baseline, the speaker conditioned acoustic model
benefits from the Wav2Vec features by achieving $+6\%$ absolute improvement in
WER for SIR $15$dB. However, due to the availability of speaker information,
the acoustic model is less sensitive to the robustness of the input acoustic
features, and thus, the amount of improvement from Wav2Vec is less in the
proposed speaker conditioned ResNet18 compared to the baseline. In conclusion,
the WER across all test sets is improved by using the proposed speaker
conditioned acoustic model trained on wav2Vec. For example, on the overlap
speech test set with SIR $15$dB, the proposed ResNet18 with Affine
Transformation trained on Wav2Vec gains +33% relative (+11% absolute)
improvement in WER compared to the original ResNet trained on MFCC.

##  4 Conclusion

In this study, we proposed a speaker conditioning method for acoustic modeling
in multi-speaker speech recognition. In the proposed method, Affine
Transformation layers are inserted into the acoustic model architecture to
fuse speaker-specific information with the acoustic model. The proposed
speaker conditioned acoustic model was compared with other fusion techniques
such as early fusion of speaker embedding and middle feature-map fusion.
Additionally, the performance of the proposed method was evaluated on
alternate input features called Wav2Vec. The results on WSJ corpus clearly
demonstrate that the proposed speaker conditioned acoustic model based on
affine transformation achieves consistent WER improvement for clean and
overlap speech scenarios.

## References

  * [1] Xuankai Chang, Wangyou Zhang, Yanmin Qian, Jonathan Le Roux, and Shinji Watanabe,  “End-to-end multi-speaker speech recognition with transformer,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6134–6138.
  * [2] Shinji Watanabe, Michael Mandel, Jon Barker, Emmanuel Vincent, Ashish Arora, Xuankai Chang, Sanjeev Khudanpur, Vimal Manohar, Daniel Povey, Desh Raj, et al.,  “Chime-6 challenge: Tackling multispeaker speech recognition for unsegmented recordings,”  arXiv preprint arXiv:2004.09249, 2020.
  * [3] Midia Yousefi and John H.L. Hansen,  “Real-Time Speaker Counting in a Cocktail Party Scenario Using Attention-Guided Convolutional Neural Network,”  in Proc. Interspeech 2021, 2021, pp. 1484–1488.
  * [4] Midia Yousefi, Navid Shokouhi, and John HL Hansen,  “Assessing speaker engagement in 2-person debates: Overlap detection in united states presidential debates.,”  in Interspeech, 2018, pp. 2117–2121.
  * [5] Jon Barker, Shinji Watanabe, Emmanuel Vincent, and Jan Trmal,  “The fifth’chime’speech separation and recognition challenge: dataset, task and baselines,”  arXiv preprint arXiv:1803.10609, 2018.
  * [6] Yan-min Qian, Chao Weng, Xuan-kai Chang, Shuai Wang, and Dong Yu,  “Past review, current progress, and challenges ahead on the cocktail party problem,”  Frontiers of Information Technology & Electronic Engineering, vol. 19, no. 1, pp. 40–63, 2018.
  * [7] Seyedmahdad Mirsamadi and John HL Hansen,  “Multi-domain adversarial training of neural network acoustic models for distant speech recognition,”  Speech Communication, vol. 106, pp. 21–30, 2019.
  * [8] Midia Yousefi and John HL Hansen,  “Block-based high performance cnn architectures for frame-level overlapping speech detection,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 28–40, 2020.
  * [9] Christoph Boeddeker, Jens Heitkaemper, Joerg Schmalenstroeer, Lukas Drude, Jahn Heymann, and Reinhold Haeb-Umbach,  “Front-end processing for the chime-5 dinner party scenario,”  in CHiME5 Workshop, Hyderabad, India, 2018.
  * [10] Midia Yousefi and Mohammad Hassan Savoji,  “Supervised speech enhancement using online group-sparse convolutive nmf,”  in 2016 8th International Symposium on Telecommunications (IST). IEEE, 2016, pp. 494–499.
  * [11] Midia Yousefi and John HL Hansen,  “Frame-based overlapping speech detection using convolutional neural networks,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6744–6748.
  * [12] Li Deng,  “Front-end, back-end, and hybrid techniques for noise-robust speech recognition,”  in Robust Speech Recognition of Uncertain or Missing Data, pp. 67–99. Springer, 2011.
  * [13] Arun Narayanan and DeLiang Wang,  “Investigation of speech separation as a front-end for noise robust speech recognition,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 22, no. 4, pp. 826–835, 2014.
  * [14] Seyedmahdad Mirsamadi and John HL Hansen,  “Multichannel speech dereverberation based on convolutive nonnegative tensor factorization for asr applications,”  in Fifteenth Annual Conference of the International Speech Communication Association, 2014.
  * [15] Midia Yousefi, Soheil Khorram, and John HL Hansen,  “Probabilistic permutation invariant training for speech separation,”  arXiv preprint arXiv:1908.01768, 2019.
  * [16] Naoyuki Kanda, Yusuke Fujita, Shota Horiguchi, Rintaro Ikeshita, Kenji Nagamatsu, and Shinji Watanabe,  “Acoustic modeling for distant multi-talker speech recognition with single-and multi-channel branches,”  pp. 6630–6634, 2019.
  * [17] Chao Weng, Dong Yu, Michael L Seltzer, and Jasha Droppo,  “Deep neural networks for single-channel multi-talker speech recognition,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 23, no. 10, pp. 1670–1679, 2015.
  * [18] Hiroshi Seki, Takaaki Hori, Shinji Watanabe, Jonathan Le Roux, and John R Hershey,  “A purely end-to-end system for multi-speaker speech recognition,”  in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2620–2630.
  * [19] Liang Lu, Naoyuki Kanda, Jinyu Li, and Yifan Gong,  “Streaming end-to-end multi-talker speech recognition,”  IEEE Signal Processing Letters, vol. 28, pp. 803–807, 2021.
  * [20] Zhehuai Chen, Jasha Droppo, Jinyu Li, and Wayne Xiong,  “Progressive joint modeling in unsupervised single-channel overlapped speech recognition,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 1, pp. 184–196, 2017.
  * [21] Pavel Denisov and Ngoc Thang Vu,  “End-to-end multi-speaker speech recognition using speaker embeddings and transfer learning,”  arXiv preprint arXiv:1908.04737, 2019.
  * [22] Yiming Wang, Xing Fan, I-Fan Chen, Yuzong Liu, Tongfei Chen, and Björn Hoffmeister,  “End-to-end anchored speech recognition,”  in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 7090–7094.
  * [23] Aswin Shanmugam Subramanian, Chao Weng, Meng Yu, Shi-Xiong Zhang, Yong Xu, Shinji Watanabe, and Dong Yu,  “Far-field location guided target speech extraction using end-to-end speech recognition objectives,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7299–7303.
  * [24] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville,  “Film: Visual reasoning with a general conditioning layer,”  in Proceedings of the AAAI Conference on Artificial Intelligence, 2018, vol. 32.
  * [25] David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur,  “X-vectors: Robust dnn embeddings for speaker recognition,”  in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5329–5333.
  * [26] Jaesong Lee and Shinji Watanabe,  “Intermediate loss regularization for ctc-based speech recognition,”  in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 6224–6228.
  * [27] Yosuke Higuchi, Shinji Watanabe, Nanxin Chen, Tetsuji Ogawa, and Tetsunori Kobayashi,  “Mask ctc: Non-autoregressive end-to-end asr with ctc and mask predict,”  arXiv preprint arXiv:2005.08700, 2020.
  * [28] Ethan A Chi, Julian Salazar, and Katrin Kirchhoff,  “Align-refine: Non-autoregressive speech recognition via iterative realignment,”  arXiv preprint arXiv:2020.14233, 2020.
  * [29] Aku Rouhe, Tuomas Kaseva, and Mikko Kurimo,  “Speaker-aware training of attention-based end-to-end speech recognition using neural speaker embeddings,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7064–7068.
  * [30] Sara Sabour, William Chan, and Mohammad Norouzi,  “Optimal completion distillation for sequence learning,”  in International Conference on Learning Representations, 2018.
  * [31] Lasse Borgholt, Jakob D Havtorn, Željko Agić, Anders Søgaard, Lars Maaløe, and Christian Igel,  “Do end-to-end speech recognition models care about context?,”  Proc. Interspeech 2020, pp. 4352–4356, 2020.
  * [32] Lukas Lee, Jinhwan Park, and Wonyong Sung,  “Simple gated convnet for small footprint acoustic modeling,”  in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 122–128.
  * [33] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals,  “Understanding deep learning requires rethinking generalization,”  preprint arXiv:1611.03530, 2016.
  * [34] Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli,  “wav2vec: Unsupervised pre-training for speech recognition,”  arXiv preprint arXiv:1904.05862, 2019.


# Semantic and Geometric Modeling with Neural Message Passing
in 3D Scene Graphs for Hierarchical Mechanical Search

Andrey Kurenkov1, Roberto Martín-Martín1, Jeff Ichnowski2, Ken Goldberg2,
Silvio Savarese1 1Stanford University 2University of California, Berkeley

(112001)

###### Abstract

Searching for objects in indoor organized environments such as homes or
offices is part of our everyday activities. When looking for a desired object,
we reason about the rooms and containers the object is likely to be in; the
same type of container will have a different probability of containing the
target depending on which room it is in. We also combine geometric and
semantic information to infer what container is best to search, or what other
objects are best to move, if the target object is hidden from view. We use a
3D scene graph representation to capture the hierarchical, semantic, and
geometric aspects of this problem. To exploit this representation in a search
process, we introduce Hierarchical Mechanical Search (HMS), a method that
guides an agent’s actions towards finding a target object specified with a
natural language description. HMS is based on a novel neural network
architecture that uses neural message passing of vectors with visual,
geometric, and linguistic information to allow HMS to process data across
layers of the graph while combining semantic and geometric cues. HMS is
trained on 1000 3D scene graphs and evaluated on a novel dataset of 500 3D
scene graphs with dense placements of semantically related objects in storage
locations, and is shown to be significantly better than several baselines at
finding objects. It is also close to the oracle policy in terms of the median
number of actions required. Additional qualitative results can be found at
https://ai.stanford.edu/mech-search/hms

##  I Introduction

In our day-to-day lives we often face the task of looking for a particular
object in organized indoor spaces. When searching for an object, the process
is coarse-to-fine, sequential, and interactive: we first consider the room,
then the container the object can be in, and then, after opening the
container, we may have to move objects to be able to see the target. The
combination of these steps makes this a hierarchical instance of mechanical
search [1].

For example, when looking for cheese in a new house, we would look 1) in the
kitchen, 2) within the fridge, and 3) near or behind other dairy products that
may be occluding it, even if bigger but less related objects (e.g. a big
watermelon) occlude a larger area. Further, while we expect the cheese to be
in a fridge, we would not first search in a fridge in the garage, because that
fridge is more likely to contain other items such as soda. In other words,
each decision is informed by multiple levels (e.g. we expect different content
for the same container in different rooms), and the search process is guided
by both semantic and geometric reasoning of the target, rooms, containers, and
occluding objects.

In this work, we address the problem of searching for objects in hierarchical
organized indoor environments (Fig. 1). The target object is specified by a
short natural language description. We assume that the searching agent is
provided with the layout of the environment and the storage locations that can
contain target objects, as well as visual observations of the objects inside a
container the first time the agent explores it. The goal of the agent is to
find the target with a minimum number of actions (motion to a new room,
selection of a new container, and/or removal of a possibly occluding object).
To efficiently search in this context, the agent needs to exploit information
at different levels (room, container, object) while fusing geometric (size)
and semantic (linguistic description and labels, appearance) information.

Figure 1: Searching for an object in large environments using semantic and
geometric information: assuming knowledge of rooms and containers, the agent
is tasked to find a target object (jasmine green tea). From coarse to fine
location levels, the agent decides what room, container, and section of the
container to search, and, if the target is not visible, what visible object
should be removed to reveal it. Right top/bottom: Content of the same
container in different rooms. Information from different levels (e.g. room)
helps predict and select at others (e.g. container); Right top image: Target
(yellow box) is partially behind semantically related objects that need to be
removed before larger unrelated objects. The mechanical search at the shelf
level is guided by a combination of geometric and semantic information.

Previous learning-based approaches enabled agents to search for objects given
images of the environment. However, these approaches reason only about
semantics [2, 3, 4, 5] or geometry [6, 7, 8] of the objects at a single level,
and ignore the combined nature of the search problem in organized
environments. Furthermore, these works have assumed that the agent starts with
little or no prior knowledge about the environment, which limits their ability
to search as efficiently as possible when such prior knowledge is available.
Several works in robotics [9, 10] have addressed joint reasoning over both
semantic and geometric aspects of objects during search, but they are
restricted to only one room or container, do not enable search for objects
guided by text descriptions, and do not learn to reason about object occlusion
from images; thus, they scale poorly to real world scenarios.

In this work, we propose to represent the environment with an extension of 3D
Scene Graphs [11] that captures the hierarchical structure of the environment
as well as the geometric and semantic information about containers and objects
within it. This graphical structure can represent rooms, containers and
objects as a hierarchy of nodes that can grow dynamically as new objects are
seen by the searching agent when it first searches inside object containers.

To enable an agent to search with joint reasoning about layers of the graph,
as well as semantic and geometric information associated with nodes, we
propose Hierarchical Mechanical Search (HMS). HMS uses neural message passing
to propagate information in the form of feature vectors from individual nodes
across the graph to be spread out to multiple other nodes. Feature vectors are
created by converting labels into word vectors and appearance into the vectors
output by a convolutional neural network, and then concatenating geometric
values such as location and size to these vectors. Given a large dataset of
examples, the model is trained to assess the likelihood of the target object
to be in a given room node, inside a given container node, or behind a given
object node. HMS also includes a search procedure that uses these likelihoods
to take actions that will retrieve the target object. The neural message model
is trained using a large number of procedurally generated 3D Scene Graphs,
created from realistic probabilities of objects to be stored together in
containers, and containers to be within rooms. This process also generates
realistic placements of object models inside containers to generate training
images (Fig. 1, right).

In summary, the contributions of this work are:

  * •

We introduce the problem of object search in organized indoor environments
with partially known layout information and dynamically growing object
information,

  * •

We propose Hierarchical Mechanical Search (HMS), a search procedure guided by
neural message passing that can be recursively applied on 3D scene graphs to
jointly reason across layers of the graph and from semantic and geometric
cues, which enables an agent to efficiently find target objects,

  * •

We experimentally demonstrate that HMS significantly outperforms baselines and
ablations on 500 unique household scene graphs unseen during training, which
have 26 different object categories and 164 unique object types.

##  II Related Work

Using real-world semantic statistics of objects to guide search has been
explored in the past, for example through learning probable spatial relations
between the target object and other objects in the environment [12, 13, 14,
15, 16, 17, 18, 5]. More recent approaches applied Deep Reinforcement Learning
to generate policies that search for a target using image inputs, assuming
access to object detectors. These works have typically embedded co-occurence
statistics between objects into graph structures that are converted into a
vector using graph neural networks [2, 3, 4, 19]. Surveys of these works can
be found in [5] and [20]. Differently, we integrate hierarchical information
of geometric and semantic cues, extracting the statistics of object co-
occurence directly from training data. We also use the information
differently: we actively guide an agent to select the rooms and containers to
search in as well as the occluding objects to remove.

Figure 2: Hierarchical semantic and geometric search: for a given environment
(top left) and a description of a target object (middle left), the agent is
tasked with reasoning about an associated 3D scene graph (bottom) to reason
about which containers (green circles) the object is likely to be in as well
as which objects (red circles) it is likely to be occluded by, so as to move
occluding objects out of the way and make the target object visible (right).

There are other approaches to guide robot manipulation with the environment to
try to make an occluded target object visible. Most of these works reason
about geometry instead of semantics to infer the possible location of an
object, and decide what objects should be moved using graph-based planning
[21, 22] or POMDP planning approaches [6, 7]. We do not plan based on detailed
geometric models, but instead leverage coarser hierarchical geometric
information together with semantic information to guide the search. Recent
works used neural networks to visually reason about piles of objects and learn
how to manipulate them with the objective of finding a target object [1, 8,
23, 24, 25, 26]. Different to ours, those methods manipulate the pile and do
not leverage semantic cues. Our work most closely relates to Wong et al. [9].
They presented a POMDP approach that reasons jointly over object semantics and
geometric properties to guide manipulation actions. However, in our approach
we do not assume a static set of object types and instead learn to perceive
how similar an object is to the target text description, and also address the
problem of jointly reasoning over multiple layers of information.

To model our problem, we extend the 3D Scene Graph representation [11] (Fig.
2). While there are many approaches for scene graph generation from images
[27, 28, 29], our focus is on leveraging an already-generated scene graph to
guide an agent in object search. Prior work has used similar graph
architectures to reason about possible location of objects. Zhou et al. [30]
presented a neural message passing method to infer the most probable missing
object in a 3D indoor scene based on the visible objects. While our approach
also involves message passing on a scene graph, we apply this differently to
reason jointly over multiple layers of the scene graph and about both semantic
and geometric cues information, and to guide mechanical search.

##  III Problem Formulation

In this work, we assume the search task to be defined by a text description of
the target object. Text descriptions are unique characterizations of instances
of a class, e.g. garlic is an instance of class Veggies and
Krill_Oil_Pills_Blue an instance of class Vitamins. We assume the environment
to be composed of two types of elements, containers and objects. In this
formulation, rooms, furniture, compartments inside furniture, …are considered
containers, which contain other containers or objects. We also assume that the
searching agent is provided with prior information about the containers: the
house rooms and the furniture in each room (e.g. cupboards, drawers, fridges,
…), along with their semantic class and size. This is a realistic assumption,
since the robot could just first visit each room to acquire such layout
information. To make the problem more realistic, we assume that the content of
the containers (i.e. objects) is not known until the agent decides to search
them for the first time. Once a container is searched, the agent receives an
image of its content and a simulated object detector will provide the class
label (not text description) and the corresponding image bounding box around
the objects that are at least 70% unoccluded. Objects that are more than 70%
are not visible until the agent removes the objects occluding them.

When searching for a target object, the agent has two types of actions that it
can take: 1) explore a container, or 2) remove an object to reveal what is
behind it. Moving to a room, opening a cupboard, looking at a shelf or picking
an object are counted as individual actions. The goal of the agent is to take
the minimum number of actions to find the target object in the environment. In
this work, we do not directly address the challenges of low-level robot
control for navigation, grasping and manipulation of the objects; we assume
the agent is able to move to desired locations and move desired visible
objects once it chooses to do so.

Figure 3: The neural net architecture we use recursively to search for a
target object. Scene graph nodes are represented as circles, with their colors
representing the contents of their feature vectors. Trapezoidal shapes
correspond to neural net layers. Message passing is performed across
neighboring layers in the graph, such that the parent node’s feature vector is
updated to reflect all of its children, and the children’s nodes are updated
to reflect their parent. As a result, the model is able to jointly reason
across layers of the graph.

##  IV Hierarchical Mechanical Search

We propose to represent the hierarchical structure of containers and objects
with their semantic and geometric properties with a modified 3D Scene Graph
[11]. Our 3D Scene Graph (Fig. 2) represents the house’s rooms and storage
locations as nodes annotated with geometric and semantic information: the
sizes of containers and labels indicating their semantic class (e.g. kitchen,
fridge). When a new container is explored and some objects in its interior are
found and perceived by the provided detector (the ones that are at least 70%
visible), we extend the Scene Graph to include them and their information:
their semantic class (e.g. Veggies, Medicine, Hats), their normalized size in
the image (size of the detector’s given bounding box), their normalized
location in the image, and their appearance (an RGB image cropped to primarily
show the object).

We now need an algorithmic structure that can reason on this hierarchical 3D
Scene Graph and help the agent decide on the most optimal next action to take
to find the target object. To address this, we introduce Hierarchical
Mechanical Search (HMS), a search procedure that is guided by a novel neural
message passing mechanism that enables integrating information across levels
(see Fig. 3). As part of this, we create feature vectors for each of the nodes
in the graph (containers and objects) which can be used as input to our neural
message passing architecture. These feature vectors should contain both the
semantic and geometric information known about each node (Sec. IV-A). Then, we
can propagate this information across layers with the message passing model
and infer the probability of containers to enclose the target, and of objects
to occlude it (Sec. IV-B). This model is optimized in an end-to-end supervised
fashion with a classification objective (Sec IV-C). After it is optimized, the
agent’s searching actions will follow the HMS search procedure based on these
probabilities output by the model (Sec. IV-D).

###  IV-A Featurizing Node Information

Nodes in the scene graph include geometric and semantic information of
containers and objects. This information needs to be “featurized”, or
converted into a numeric vector, $f$, to be used as inputs to our model and
propagated among levels of the graph to make informed decisions. Therefore, we
first compute a feature vector with the information of each node, and of the
target object, $f_{t}$.

Semantic information in the form of language labels (i.e. bedroom, fridge) is
featurized using the corresponding pre-trained 300-dimensional GloVe word
vector [31]; word vectors are known to encode semantic meanings, and so aid in
associating the target object’s description with the appropriate containers
and objects. The feature vector of the target object $f_{t}$ is created by
averaging the GloVe vectors of all the words in its description. For
containers, the Glove vector is concatenated with their known volume.

In the case of objects, the information that needs to be featurized includes
their label, bounding box size, bounding box location, and the RGB image of
the object. We featurize the label in the same way as done for containers, and
concatenate it to the normalized center coordinates and size of the bounding
box. To be able to reason about the appearance of the object, we use the
bounding box coordinates to obtain an image of the object and crop it to show
the object with some of the container surroundings around it in the container
also visible. For extracting features from this image, we fine-tune the
resnet-18 Convolutional Neural Net [32] from weights pretrained on the
ImageNet classification task [33], after replacing its final layers with two
randomly initialized fully connected layers that output a 256-dimensional
vector that can be trained for our task.

Lastly, all vectors are processed by two fully connected layers to obtain
final feature vectors of the same dimension of 100 for containers, the target,
and non-target objects.

###  IV-B Neural Message Passing Model

Given the scene graph and the feature vectors of nodes and the target object,
we propose a single neural net architecture that can be trained to evaluate
each container for whether it contains the target object, and each detected
object for whether it is occluding the target object. To evaluate a single
container in a way that makes use of information about subsequent layers in
the graph (e.g. its contained objects), we utilize neural message passing, a
technique for neural-net based reasoning on graph structures that propagate
information between nodes. Message passing is a better choice for our problem
compared to other forms of learning on graph structures because it can be
applied on dynamically changing graph structures, and on graphs of different
sizes and depths.

Our message passing mechanism is depicted in Fig. 3 and summarized as follows:
first, we combine the parent’s (a container) feature vector, $f_{p}$, with the
feature vectors of each of its $n$ children nodes (other containers or
objects), $f_{c1},\ldots,f_{cn}$, by concatenating them, and further process
the combinations with learnable fully connected layers. This results in a new
set of children vectors, $f^{\prime}_{c1},\ldots,f^{\prime}_{cn}$, that
contain information about both the parent and the child. These new vectors
with parent information are stored as the child’s new feature vector to use in
later processing. All of the new child vectors are aggregated via averaging,
and a new parent feature vector, $f^{\prime}_{p}$, is made by averaging the
original vector with the aggregated parent-child vector. The updated parent
feature vector combines the information of the parent and its children (e.g.
both a room and all the containers within it).

The new parent feature vector, or the stored feature vector of a leaf object
node, is combined with the target object’s vector, $f_{t}$, by element-wise
multiplication. This combined representation encodes information about both
the candidate node and the target. The combined vector is passed through three
fully connected layers, followed by a sigmoid activation. The final output,
$p$, a value between 0 and 1, is optimized to predict the likelihood of the
target object to be in the considered container, or behind the considered
object. While different classification layers are trained for evaluating
containers and objects, the architecture is exactly the same for both,
resulting in a recursive repeating pattern at each parent-child level of the
3D Scene Graph hierarchy.

###  IV-C Model Optimization

In order to utilize the model for search, it needs to be trained to output
correct probabilities, $p$. We will train our neural message passing
architecture using a large dataset of scene graphs of indoor environments with
annotated locations for target objects in a supervised manner. The procedural
generation of this dataset using annotations of plausible object locations is
described in Sec. V. At each optimization step, the following process is
followed:

  1. 1.

We sample batches of scene graphs from the train dataset.

  2. 2.

For each scene graph, we sample an occluded object and set it to be the target
object for that scene graph.

  3. 3.

For each layer in each scene graph, we sample one node having a positive label
(i.e. a container that has the target object, or an object that occludes the
target object) and one node having a negative label with respect to the target
object.

  4. 4.

We compute and store feature vectors for all sampled nodes and their children.

  5. 5.

The model computes probabilities $p$ for all of sampled nodes, starting with
the top-most layer and proceeding to the bottom-most layer. Computation has to
proceed in this way, since for a given parent node $n_{p}$, the updated child
representations $f^{\prime}_{c1},\ldots,f^{\prime}_{cn}$ are used as input to
the model when computing outputs in the children’s layer.

  6. 6.

For each sampled node, we use its known positive or negative label and the
model’s output $p$ to compute the binary cross entropy classification loss.

  7. 7.

We average all losses and optimize all model parameters (resnet-18,
“featurization”, message passing, and classification layers) using the Adam
optimizer [34] end-to-end.

###  IV-D Search Procedure

The above neural net architecture is used to guide a procedure similar to
greedy depth-first-search for finding the target object. This is chosen over a
variant of breadth-first-search to mimic the manner in which an agent would
explore an environment, with the agent not being able to ’teleport’ and
therefore exploring all or most of a given container before proceeding to the
next most promising option.

The search proceeds in the following way: starting at the top-most layer of
the graph, we separately evaluate each node for whether it is likely to
contain the target object. We then sort the nodes by the model’s output, and
search each one in turn by evaluating how likely its children nodes are to
contain or occlude the target. Each child node is evaluated using the same
message passing procedure as in the first layer, with the childrens’ starting
feature vectors, $f^{\prime}_{c1},\ldots,f^{\prime}_{cn}$, carried over from
the message passing step done with the parent. Once evaluated, the children
nodes are again searched in sorted order, until they are all explored and the
search reverts to the next node in the layer above. For containers, “search”
means deciding which of the containers or objects within that container to
search, whereas for objects “search” means moving the object out of the scene.

An issue of the above approach is that it would search nodes even if the model
gives them a low probability $p$ of containing or occluding the target object.
So as to avoid this, for all nodes if $p$ is lower than a threshold $T$, the
node is not searched but rather skipped. The value of $T$ starts the same for
all nodes, but can be updated during the search so as to prevent the correct
nodes for finding the target object from never being searched. This update
happens whenever a node is skipped, at which point the value of $T$ for all
subsequent nodes in that layer is set to to the value of $p$ of that node. If
the target is not found the first time all top-level nodes are either searched
or skipped, the procedure restarts with the updated values of $T$ and removed
objects from the previous iteration being preserved. Thus, nodes can be
revisited if initially skipped, and this procedure is therefore guaranteed
locate the target object. Once the target object is successfully seen, the
search is finished.

##  V Generation of Plausible Scene Graphs

To train and evaluate HMS, we generate 1500 scene graphs with plausible
placements of storage locations within rooms, and dense placements of
semantically related objects that may occlude the target on shelves. Existing
simulation environments for evaluating search and navigation tasks [35, 36],
and annotated 3D Scene Graphs from real-world buildings [11] both lack
annotations for object storage locations and lack dense object arrangements
with object occlusion. Thus, we propose a procedural generation process for
creating synthetic but realistic scene graphs.

We propose that scene graph generation must produce plausible and varied
hierarchies with dense placements of semantically related objects. For
example, in plausible and varied hierarchies, there is usually a fridge in the
kitchen, but sometimes also a fridge in the garage or in rare cases a fridge
in the living room. These hierarchies need also have semantic clusters, in
which objects are stored in containers together with semantically related
objects, with some containers having two or more groups of related objects
(e.g. a pantry having coffee and tea products on one side of a shelf, and
protein products on the other side of the shelf).

To address the need of having plausible but varied hierarchies of containers,
we focus on a hand-picked set of room and storage location types and manually
set the probabilities of each storage location occurring in every type of room
(e.g. a fridge is most likely to be in the kitchen, but has some probability
of being in the living room). Concretely, our procedure makes use of 7 room
types (living room, closet, bedroom, garage, bathroom, closet) and 4 storage
location types (fridge, cabinet, shelves, pantry). Since the same storage type
can be in multiple rooms (e.g. shelves in both the bedroom and bathroom), this
results in at most $4^{7}=16384$ possible combinations. Thus, to create many
plausible but varied scene graphs we sample storage location placements from
this distribution. We also associate each storage location with specific 3D
models so as to have a known volume and number of shelves.

To address the need of having plausible and dense placements of semantically
related objects on the shelves of storage locations, we utilize a second
procedural generation method. First, we manually curate 3D object models and
categorize them into distinct groups based on their semantic identity. We also
annotate each model with a probability per stable orientation for placements
on a support surface, a text description (e.g. “Handmade Brown Hat”), and a
class label (e.g. “Hat”). Each of the groups is annotated with the
probabilities of being in a given storage location. Lastly, to place objects
inside of containers we use the PyBullet [37] physics simulator to sample
collision-free positions for all objects by sequentially dropping them and
waiting for them to come to rest on a given shelf. To generate realistic-
looking RGB images, we use the iGibson renderer [36].

We consider a single scene graph as the hierarchy of containers and the
associated placements of objects within each storage location. Due to the
variety of possible ways to assign storage locations to rooms and place
objects on shelves, this procedure can generate a large number of plausible
but varied scene graphs. We use this procedure as the basis for our
experiments.

##  VI Experiments

Table I: Accuracy Results | Container Accuracy | Object Accuracy
---|---|---
Word Vector Similarity* | 75 | 56
Nearest* | - | 64
Largest* | - | 69
Most Likely* | 82 | -
HMS, Object Context Vector | 90 | 69
HMS, No Message Passing | 76 | 89
HMS, No Object Label | 91 | 86
HMS, No Object Image | 94 | 75
HMS | 92 | 89
Average classification accuracies on the entire test set of scene graphs.
Values are rounded to nearest integer. *has access to information not
available to our model
Table II: Search Efficiency Results | Oracle* | (1)RND | (4)NR* | (5)LR* | (6)CV | HMS
---|---|---|---|---|---|---
Oracle* | 4.3 | 7.4 | 5.8 | 7.0 | 7.3 | 5.4
(1)RND | 72.5 | 76.8 | 81.0 | 75.0 | 79.2 | 73.5
(2)WV* | 55.7 | 57.7 | 58.4 | 57.6 | 57.7 | 60.9
(3)MS* | 15.3 | 17.2 | 16.0 | 17.5 | 18.5 | 15.6
HMS | 13.6 | 14.5 | 14.3 | 15.6 | 16.2 | 13.8
Average test scene graph action counts (# containers searched + # objects
moved). Rows correspond to container selection method, columns correspond to
object selection method. *has access to information not available to our agent

We evaluate HMS on 1500 synthetic scene graphs (Sec. V), and analyze its
performance to address the following questions:

  * •

How well does our approach perform object search?

  * •

To what extent does integrating information across different levels of the
scene graph guide our model?

  * •

To what extent does our model leverage the visual input of objects to decide
on occlusion probability?

  * •

Does our model correctly prioritize semantically and geometrically appropriate
objects when evaluating occlusion?

#### Experimental Setup

For our object set, we curate 135 3D object models from the Google Scanned
Objects set [38], and 29 3D object models bought online, and group these
models into 26 categories such a fruits, dairy products, hats, board games,
etc. We use the same object models when generating both train and test scene
graphs. Each non-empty shelf of a storage container has either 1 or 2 object
categories and between 4 and 15 objects placed in plausible arrangements that
often result in semantically related objects occluding each other. We generate
1000 unique scene graphs for training and 500 unique scene graphs for testing;
train and test scene graphs have slightly different probabilities of container
and object placement. When sampling target objects during training, we hold
out 26 objects (1 from each category) to use as a search target at test time.
We implement our model using the Pytorch framework [39], and train for 50
epochs of 100 batches, sampling 10 scene graphs per batch.

#### Baselines and Ablations

We compare to a set baselines, which all output a value in $[0,1]$ instead of
$p$:

  1. 1.

Random (RND): a uniformly random value in $[0,1]$

  2. 2.

Word Vector Similarity (WV): the cosine similarity of the node word vector
features and those of the target object.

  3. 3.

Most Likely (MS): the ground truth probability of a container having the class
of the target object, as used in the procedural generation of scene graphs.

  4. 4.

Nearest (NR): how near the object is to the camera, normalized so that the
farthest object on a shelf has a value of 0 and the nearest a value of 1.

  5. 5.

Largest (LR): the size of the object, normalized so that the smallest object
on a shelf has a value of 0 and the largest has a value of 1.

  6. 6.

Object Context Vector (CV): similar to [4], each object is associated with a
5-dimensional “context vector” which contains its bounding box information and
word vector similarity to the target. We train a new model for which each
object input representation is a context vector, and the model is otherwise
the same as that of HMS.

We also evaluate ablations of the model: no object labels, no object images,
and no message passing.

#### Metrics

We focus on two metrics: classification accuracy and search efficiency.
Classification accuracy is the measure of how often our model’s classification
network correctly predicts whether a container has the target object or an
object occludes the target object. Different to this, search efficiency refers
to the number of actions our approach requires to find the target object
within a scene graph. Every decision with regards to a container or object is
counted as one action. Because search involves reasoning about both containers
and objects, we evaluate the search efficiency metric in terms of each
possible combination of how containers are chosen and how objects are chosen.
For search, the value of $T$ is set to $0.1$ initially for all nodes.

Figure 4: Percent of test set searches that succeeded in finding the target
object in at most $x$ actions, for different search policies. HMS can find
objects with significantly fewer actions compared to the baselines. The ’Word
Vector’ policy does object selection randomly.

#### Results

Results in Table I suggests that the HMS approach outperforms baselines and
ablations in accuracy by correctly predicting containers and occluders of the
target object. Further, it relies on the image input for occlusion
predictions. Due to the higher accuracy in predictions, an agent using HMS
finds the target object with significantly fewer actions when compared to the
baselines (Table II). While the average action count is higher than that of
the oracle policy, this is primarily due to outlier scene graphs that require
large action counts (Fig. 4). These scene graphs have rare container
placements that are underrepresented in the training set.

HMS usually correctly combines semantic and geometric cues to select the
correct occluding objects, but is occasionally incorrect. Fig. 5 shows
qualitative examples. However, flawed occlusion classification does not
strongly effect search efficiency; the incorrect choice of container results
in most of the unnecessary actions in searches.

Figure 5: Qualitative examples of correct and flawed outputs by HMS. Yellow
bounding boxes correspond to the visible portion of the target object, and
greener bounding boxes correspond to higher model estimates that the object is
occluding the target object. (left) Good predictions with target object canned
capers: the model picks the more semantically relevant objects, despite the
larger unrelated dairy products. (middle) Good predictions with target object
hair product: the geometrically and semantically likely objects are preferred.
(right) Flawed predictions with target object nescafe coffee: the products on
the right are similarly scored as the more semantically relevant products on
the left.

##  VII Conclusion

We presented Hierarchical Mechanical Search (HMS), an approach to search for a
target object in organized indoors environments leveraging prior information
about the layout of the environment in the form of 3D Scene Graphs, and
reasoning jointly about semantics and geometry. Through a novel neural message
passing architecture trained on 1000 3D scene graphs, and a novel search
procedure, HMS recursively reasons across layers of the graph to help an
interactive agent to find a target object specified with natural language
descriptions. Evaluation on 500 3D scene graphs with dense placements of
semantically related objects shows that HMS is significantly superior to
several baselines at finding objects efficiently. In the future, we will work
to make a non-synthetic dataset for this problem, evaluate HMS in real indoor
environments, and extend HMS with low level control implementations for the
desired navigation and manipulation actions on containers and objects [40, 1,
23].

## References

  * [1] Michael Danielczuk et al.  “Mechanical search: Multi-step retrieval of a target object occluded by clutter”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 1614–1621  IEEE
  * [2] Wei Yang et al.  “Visual semantic navigation using scene priors”  In _arXiv preprint arXiv:1810.06543_ , 2018
  * [3] Yi Wu et al.  “Bayesian relational memory for semantic visual navigation”  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 2769–2779
  * [4] Yiding Qiu, Anwesan Pal and Henrik I Christensen  “Target driven visual navigation exploiting object relationships”  In _arXiv preprint arXiv:2003.06749_ , 2020
  * [5] Jonathan Crespo, Jose Carlos Castillo, Oscar Martinez Mozos and Ramon Barber  “Semantic Information for Robot Navigation: A Survey”  In _Applied Sciences_ 10.2 Multidisciplinary Digital Publishing Institute, 2020, pp. 497
  * [6] Jue Kun Li, David Hsu and Wee Sun Lee  “Act to see and see to act: POMDP planning for objects search in clutter”  In _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2016, pp. 5701–5707  IEEE
  * [7] Yuchen Xiao et al.  “Online planning for target object search in clutter under partial observability”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 8241–8247  IEEE
  * [8] Michael Danielczuk, Anelia Angelova, Vincent Vanhoucke and Ken Goldberg  “X-Ray: Mechanical Search for an Occluded Object by Minimizing Support of Learned Occupancy Distributions”  In _arXiv preprint arXiv:2004.09039_ , 2020
  * [9] Lawson LS Wong, Leslie Pack Kaelbling and Tomás Lozano-Pérez  “Manipulation-based active search for occluded objects”  In _2013 IEEE International Conference on Robotics and Automation_ , 2013, pp. 2814–2819  IEEE
  * [10] Bogdan Moldovan and Luc De Raedt  “Occluded object search by relational affordances”  In _2014 IEEE International Conference on Robotics and Automation (ICRA)_ , 2014, pp. 169–174  IEEE
  * [11] Iro Armeni et al.  “3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera”  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 5664–5673
  * [12] Kristoffer Sjöö, Alper Aydemir and Patric Jensfelt  “Topological spatial relations for active visual search”  In _Robotics and Autonomous Systems_ 60.9 Elsevier, 2012, pp. 1093–1107
  * [13] Alper Aydemir et al.  “Search in the real world: Active visual object search based on spatial relations”  In _2011 IEEE International Conference on Robotics and Automation_ , 2011, pp. 2818–2824  IEEE
  * [14] Walter Toro, Fábio Gagliardi Cozman, Kate Revoredo and Anna Helena Reali Costa  “Probabilistic Relational Reasoning in Semantic Robot Navigation.”  In _URSW_ , 2014, pp. 37–48
  * [15] Malte Lorbach, Sebastian Höfer and Oliver Brock  “Prior-assisted propagation of spatial information for object search”  In _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2014, pp. 2904–2909  IEEE
  * [16] Lars Kunze, Chris Burbridge and Nick Hawes  “Bootstrapping Probabilistic Models of Qualitative Spatial Relations for Active Visual Object Search.”  In _AAAI Spring Symposia_ , 2014  Citeseer
  * [17] Mingu Kim and Il Hong Suh  “Active object search in an unknown large-scale environment using commonsense knowledge and spatial relations”  In _Intelligent Service Robotics_ 12.4 Springer, 2019, pp. 371–380
  * [18] Zhen Zeng, Adrian Röfer and Odest Chadwicke Jenkins  “Semantic Linking Maps for Active Visual Object Search”  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 1984–1990  IEEE
  * [19] Raphael Druon, Yusuke Yoshiyasu, Asako Kanezaki and Alassane Watt  “Visual object search by learning spatial context”  In _IEEE Robotics and Automation Letters_ 5.2 IEEE, 2020, pp. 1279–1286
  * [20] Fanyu Zeng, Chen Wang and Shuzhi Sam Ge  “A survey on visual navigation for artificial agents with deep reinforcement learning”  In _IEEE Access_ 8 IEEE, 2020, pp. 135426–135442
  * [21] Mehmet R Dogar, Michael C Koval, Abhijeet Tallavajhula and Siddhartha S Srinivasa  “Object search by manipulation”  In _Autonomous Robots_ 36.1-2 Springer, 2014, pp. 153–167
  * [22] Changjoo Nam et al.  “Planning for target retrieval using a robotic manipulator in cluttered and occluded environments”  In _arXiv preprint arXiv:1907.03956_ , 2019
  * [23] Andrey Kurenkov et al.  “Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter”  In _IEEE/RSJ Int. Conference. on Intelligent Robots and Systems (IROS)_ , 2020
  * [24] Ricson Cheng, Arpit Agarwal and Katerina Fragkiadaki  “Reinforcement learning of active vision for manipulating objects under occlusions”  In _arXiv preprint arXiv:1811.08067_ , 2018
  * [25] Tonci Novkovic et al.  “Object finding in cluttered scenes using interactive perception”  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 8338–8344  IEEE
  * [26] Tara Boroushaki et al.  “Robotic Grasping of Fully-Occluded Objects using RF Perception”  In _arXiv preprint arXiv:2012.15436_ , 2020
  * [27] Danfei Xu, Yuke Zhu, Christopher B Choy and Li Fei-Fei  “Scene graph generation by iterative message passing”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5410–5419
  * [28] Roei Herzig et al.  “Mapping images to scene graphs with permutation-invariant structured prediction”  In _arXiv preprint arXiv:1802.05451_ , 2018
  * [29] Jianwei Yang et al.  “Graph r-cnn for scene graph generation”  In _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 670–685
  * [30] Yang Zhou, Zachary While and Evangelos Kalogerakis  “SceneGraphNet: Neural Message Passing for 3D Indoor Scene Augmentation”  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 7384–7392
  * [31] Jeffrey Pennington, Richard Socher and Christopher D Manning  “Glove: Global vectors for word representation”  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , 2014, pp. 1532–1543
  * [32] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778
  * [33] Jia Deng et al.  “Imagenet: A large-scale hierarchical image database”  In _2009 IEEE conference on computer vision and pattern recognition_ , 2009, pp. 248–255  Ieee
  * [34] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _arXiv preprint arXiv:1412.6980_ , 2014
  * [35] Eric Kolve et al.  “Ai2-thor: An interactive 3d environment for visual ai”  In _arXiv preprint arXiv:1712.05474_ , 2017
  * [36] Fei Xia et al.  “Interactive gibson: A benchmark for interactive navigation in cluttered environments”  In _arXiv preprint arXiv:1910.14442_ , 2019
  * [37] Erwin Coumans and Yunfei Bai  “PyBullet, a Python module for physics simulation for games, robotics and machine learning”, http://pybullet.org, 2016
  * [38] GoogleResearch  “Google Scanned Objects”, 2020  Open Robotics  URL: https://app.ignitionrobotics.org/GoogleResearch/fuel/collections/Google%5C%20Scanned%5C%20Objects
  * [39] Adam Paszke et al.  “PyTorch: An Imperative Style, High-Performance Deep Learning Library”  In _Advances in Neural Information Processing Systems 32_ Curran Associates, Inc., 2019, pp. 8024–8035  URL: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [40] Roberto Martín-Martín and Oliver Brock  “Cross-modal interpretation of multi-modal sensor streams in interactive perception based on coupled recursion”  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2017, pp. 3289–3295  IEEE