(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: 1Technical University of Munich 2Ludwig Maximilian University
of Munich 3Google
11email<EMAIL_ADDRESS>
https://sites.google.com/view/echoscene

#  EchoScene: Indoor Scene Generation via Information Echo over Scene Graph
Diffusion

Guangyao Zhai1 Evin Pınar Örnek1 Dave Zhenyu Chen1 Ruotong Liao
Yan Di1 Nassir Navab1 Federico Tombari1,3 Benjamin Busam1 22

###### Abstract

We present _EchoScene_ , an interactive and controllable generative model that
generates 3D indoor scenes on scene graphs. EchoScene leverages a dual-branch
diffusion model that dynamically adapts to scene graphs. Existing methods
struggle to handle scene graphs due to varying numbers of nodes, multiple edge
combinations, and manipulator-induced node-edge operations. EchoScene
overcomes this by associating each node with a denoising process and enables
collaborative information exchange, enhancing controllable and consistent
generation aware of global constraints. This is achieved through an
information echo scheme in both shape and layout branches. At every denoising
step, all processes share their denoising data with an information exchange
unit that combines these updates using graph convolution. The scheme ensures
that the denoising processes are influenced by a holistic understanding of the
scene graph, facilitating the generation of globally coherent scenes. The
resulting scenes can be manipulated during inference by editing the input
scene graph and sampling the noise in the diffusion model. Extensive
experiments validate our approach, which maintains scene controllability and
surpasses previous methods in generation fidelity. Moreover, the generated
scenes are of high quality and thus directly compatible with off-the-shelf
texture generation. Code and trained models are open-sourced.

Figure 1: EchoScene Schematic. EchoScene uses a dual-branch diffusion model to
generate 3D scenes from scene graphs. In both branches, each node is allocated
a denoising process, and different processes are aware of global states
through layout and shape "echoes" (waves in different colors) with an
information exchange unit (grey block) along the denoising steps.

##  1 Introduction

Controllable Scene Generation (CSG) refers to synthesizing realistic 3D scenes
according to input prompts while enabling specific entities within the scene
to be user-interactive [6, 5, 47]. It has successfully been applied in
robotics [55, 36], Virtual Reality / Augmented Reality [2], and autonomous
driving [43, 29]. Recently, combining CSG with scene graph diffusion models
has attracted significant research interest [57, 51, 30], since on the one
hand, diffusion models empower more realistic and diverse 3D content
generation [9, 10, 46, 42], on the other hand scene graphs capture the
structure of the scene in a compact manner and facilitate intuitive user
manipulation [15, 57]. Importantly, users can modify the input scene graph to
dynamically change the generated scene.

Despite its significant progress so far, CSG with scene graph diffusion still
suffers from two open challenges. First, due to varying numbers of graph nodes
and manipulator-induced node-edge operations, the input scene graphs
dynamically describe global scene states, thus demanding adaptability from
networks to accurately represent changing states. This status includes nodes
appearing or disappearing and edges altering during scene creation. Second, it
is crucial yet difficult when encapsulating both fine-grained node classes and
diverse edge combinations into a network to be aware of global constraints.
Existing solutions either simplify graphs by preserving only node sets,
avoiding complex edge encoding in the diffusion while losing key structural
relationships [51], or convert nodes and edges into tokens for transformer-
based denoiser [30], which is effective for the layout generation from simple
graphs but not scalable for larger ones due to exponential token growth. For a
graph with $P$ nodes and $Q$ types of edges, the maximum token count grows
with $Q\cdot P!$, making such a strategy hard to model intricate relations and
computationally infeasible for large graphs. CommonScenes [57] collapses
graphs to triplets (subject, predicate, object) and employs graph convolution
to aggregate features for each node, which are then conditioned on individual
denoising processes for shape generation. While this approach adapts well to
varying graph sizes, each denoising process is isolated from the other,
resulting in insufficient awareness of global shape states and, thus,
undesired inter-object inconsistency. Additionally, its reliance on VAE and
GAN for layout generation complicates synchronized training.

To fully utilize the controllability of scene graphs for scene generation and
manipulation, the feature aggregation of multiple edges to the respective
nodes is essential. In an attempt to flexibly encapsulate information from an
indefinite number of nodes without losing global graph constraints, it can be
helpful to allocate an individual denoising process per node and encourage
information communication among all processes to achieve a common goal state.
Integrating both concepts, CSG becomes promising with complex scene graphs by
adhering to graph constraints and manipulating object-level diffusion. With
_EchoScene_ we move a step further and introduce a scene-generative framework
equipped with dual-branch diffusion models capable of producing object shapes
and scene layouts simultaneously (see Fig. 2 for the pipeline). Each branch
contains multiple denoising processes, with an equal number of nodes in the
scene graph. The denoiser is weight-shared for all processes without
introducing additional costs. To consolidate global awareness in every
denoising process, we introduce an information echo scheme to exchange
information at each denoising step in the respective branches, as illustrated
in the middle of Fig. 1. Here, an information echo in the graph diffusion
happens when a node first sends the denoising data and other features to an
information exchange unit, which echoes back the aggregated features to the
node as the condition signal for its denoiser. More clearly, for a single
denoising process, the echo route is: {current denoising input
$\longrightarrow$ information exchange unit $\longrightarrow$ denoising
conditioner}.

In this design, EchoScene is able to generate and edit scenes from complex
scene graphs with an indefinite number of nodes and edges. More importantly,
the information echo scheme supports temporally global information exchange
within both branches, thus continuously making the generation compliant with
the scene graph description. For example, in the shape branch, shape echoes
enable steady awareness of object appearance along the denoising steps for
each generation process, bringing more inter-object consistency to the global
style. Last but not least, the pure diffusion-based model dynamics enables
synchronized training. EchoScene outperforms the previous state-of-the-art
method on generation fidelity by a large margin and is more robust when
handling graph manipulation. Moreover, we show that scenes generated by
EchoScene are compatible with off-the-shelf texture generators, e.g., SceneTex
[9], facilitating potential downstream applications.

The main contributions of this work can be summarized as follows:

  1. 1.

We present _EchoScene_ , a scene generation method with a dual-branch
diffusion model on dynamic scene graphs, to simultaneously generate layouts
and shapes with more controllability.

  2. 2.

We introduce an information echo scheme inside each branch of EchoScene that
allows multiple denoising processes to exchange their denoising status among
each other at each time step, bringing global awareness to each individual
process.

  3. 3.

We show in the experiments that the proposed generative framework achieves
more generation fidelity, more robustness in handling graph manipulation, and
effectively handles inter-object style consistency.

##  2 Related work

Semantic Scene Graph. Scene graphs have become a fundamental tool for semantic
scene understanding, offering a structured and symbolic representation of
scenes through nodes (objects) and edges (relationships) [21]. Their
versatility spans various domains, including image manipulation, caption
generation, and visual question answering, demonstrating their capacity to
enrich 2D image generation [20, 14, 25]. Beyond 2D, scene graphs have been
extended to 3D scene understanding [54, 61], dynamic modeling [45], and
notably, controllable scene synthesis [27], showcasing their adaptability in
representing complex spatial and semantic relations in multi-dimensional
spaces [54, 45, 27, 60, 38, 57]. EchoScene proposes an indoor 3D scene
generation method conditioned fully on semantic scene graphs coupled with a
scene controlling ability.

Diffusion Models. Applications of diffusion models [48] have witnessed rapid
expansion, from generating intricate images to modeling complex distributions
in various data types [19, 50]. Recent advances have focused on improving
their flexibility and realism, with significant efforts to enhance conditional
generation and refine the models’ understanding of input conditions [37, 58,
8, 4]. Score Based Distillation (SDS) was proposed to enable training Neural
Radiance Fields through pretrained diffusion models [42]. A line of work
addressed text-to-3D via SDS[46, 32], studied the compositionality of scenes
via SDS[31, 13, 17, 41], and more recently via Gaussian Splatting [59]. These
methods can create realistic 3D assets from text via a small number of images;
however, they are limited in terms of number of objects within a scene while
capturing their relationships. Comparably, a set of works used diffusion
models to learn 3D shapes explicitly [12, 33]. Finally, diffusion models have
recently been shown to be useful for graph generative processes [53, 52, 24].
EchoScene draws from the recent advances in diffusion models, where both shape
and layout generation are through graph diffusion processes.

Controllability in Scene Synthesis. The quest for controllable scene synthesis
has led to diverse methodologies, ranging from text descriptions to spatial
layouts and probabilistic grammars for creating 3D scenes. Methods have
explored generating scenes from images [11], texts [5], and layouts [1],
employing various strategies such as autoregressive models and deep priors to
achieve controllable outputs [35, 22, 40]. Recent work has also seen graph-
based conditioning emerge as a powerful approach for controlled scene
synthesis, enabling more precise manipulation of scene elements and their
relations [27, 60, 57, 51, 30]. CommonScenes [57] came up with a scene graph
conditioned scene generation method, which is based on a VAE and a GAN for
capturing the layout and creating shapes via a latent diffusion model. Even
though these methods have high quality in layout generation, their shape
generation abilities are limited, as well as their manipulation quality still
has room for improvement. EchoScene addresses these challenges through a
diffusion-based layout generation with a novel information echo scheme.

##  3 Preliminary

Scene Graph. The scene graph we use is semantic scene graph [7], denoted as
$\mathcal{G}=\left\\{\mathcal{V},\mathcal{E}\right\\}$, which serves as a
structured representation of a visual scene. In such representation,
$\mathcal{V}=\\{v_{i}~{}|~{}i={1,\ldots,N\\}}$ refers to the set of object
nodes, while $\mathcal{E}=\\{e_{i\to j}~{}|~{}i,j={1,\ldots,N},i\neq j\\}$
represents the set of directed edges connecting each pair of nodes
$v_{i}\rightarrow v_{j}$. As structured in the Fig. 2.A.1, each node $v_{i}$
and edge $e_{i\to j}$ can encompass various extensible attributes, e.g.,
object categorical information.

Scene Graph Convolution. The scene graph convolution we use in this work,
originally from [20] and applied in [34, 15, 57], is the fundamental module
processing semantic scene graphs, as known as triplet-GCN. A typical $K$-layer
triplet-GCN module is:

$\displaystyle(\alpha_{v_{i}}^{(k)},\beta_{e_{i\to
j}}^{(k+1)},\alpha_{v_{j}}^{(k)})=h_{1}(\beta_{v_{i}}^{(k)},\beta_{e_{i\to
j}}^{(k)},\beta_{v_{j}}^{(k)}),k=0,\ldots,K-1,$ (1)
$\displaystyle\beta_{v_{i}}^{(k+1)}=\alpha_{v_{i}}^{(k)}+h_{2}\Big{(}\texttt{Avg}\big{(}\alpha_{{v}_{j}}^{(k)}~{}|~{}v_{j}\in
N_{\mathcal{G}}(v_{i})\big{)}\Big{)},$

where $k$ represents an individual layer in triplet-GCN, and $h_{1},h_{2}$ are
MLP. $N_{\mathcal{G}}(v_{i})$ encompasses all nodes that are directly
connected to $v_{i}$, and Avg denotes the process of average pooling. The
initial attributes of nodes and edges, denoted by
$(\beta_{v_{i}}^{(0)},\beta_{e_{i\to j}}^{(0)},\beta_{v_{j}}^{(0)})$, able to
be customized for specific usages.

Contextual Graph. CLIP features are proven to bring strong inter-object
information to the scene graph [57]. Thus, a semantic scene graph evolves to a
contextual graph by using the readily available CLIP text encoder to infer the
semantic information for each node and each triplet in the graph to provide
semantic anchors ($p_{i}$, $p_{i\to j}$), and each node and edge have their
own learnable vectors ($o_{i}$, $\tau_{i\to j}$). As shown in the Fig. 2.A.1,
the node and edge feature of a contextual graph is $v_{i}:=\\{p_{i},o_{i}\\}$
and $e_{i\to j}:=\\{p_{i\to j},\tau_{i\to j}\\}$, respectively.

Conditional Diffusion Models. Diffusion models learn to estimate a target
distribution through a progressive Markov Process of length $T$ where a
denoiser $\varepsilon_{\theta}$ is trained to gradually remove noise from
diffused noisy versions ${\bf d}_{t}$ with $t\in\\{1,\ldots,T\\}$ of a data
sample ${\bf d}$ [19, 50]. Latent Diffusion Models (LDMs) [44] efficiently
establish this process in latent space where a conditioner ${\bf c}$ can guide
the generation process [12]. Learning the denoising is reached by minimization
of the objective

$\mathcal{L}_{LDM}=\mathbb{E}_{{\bf
d},\varepsilon\sim\mathcal{N}(0,1),t}\left[||\varepsilon-\varepsilon_{\theta}({\bf
d}_{t},t,{\bf c})||_{2}^{2}\right].$ (2)

Once the denoiser $\varepsilon_{\theta}$ is learned, the inverse process can
be established using Langevin dynamics [49].

##  4 Method

Overview. We present _EchoScene_ , a method that accomplishes scene generation
through layout and shape generation from scene graphs. EchoScene evolves the
contextual graph to the latent space utilizing an encoder and a manipulator
based on triplet-GCN, as shown in Fig. 2.A and as introduced in Sec. 4.1.
Then, as shown in Fig. 2.B, it conditions latent nodes to layout branch (Fig.
2.B.1) and shape branch (Fig. 2.B.2) separately based on an information echo
scheme defined in Sec. 4.2. In the layout branch, each diffusion process
interacts with each other through layout echo to acquire global layout
awareness at each denoising step, so that the final layout generation is
compliant with the scene graph description (see Sec. 4.3 and Fig. 3.A).
Similarly, in the shape branch, each diffusion process interacts with each
other through shape echo to be aware of other shape appearances at each
denoising step, so that the final generated shapes in the scene are consistent
(see Sec. 4.4 and Fig. 3.B). Moreover, our generated scenes are compatible
with off-the-shelf texture generators, e.g., SceneTex [9], making more
photorealistic appearances.

###  4.1 Graph Preprocessing

Graph Encoding. To make the layout and shape branches aware of the semantic
and spatial information among the objects, we first encode the contextual
graph to have latent relation embedding for each node by a triplet-GCN-based
encoder $E_{r}$. In this case, we initialize the input embeddings of layer $0$
with the features from the contextual graph, thus,
$(\beta_{v_{i}}^{(0)},\beta_{e_{i\to
j}}^{(0)},\beta_{v_{j}}^{(0)})=(v_{i},e_{i\to j},v_{j})$ in Eq. 1. After the
encoding, node features evolve to
$\mathcal{V}_{\mathcal{Z}}=\\{v_{i}^{z}~{}|~{}i=1,\ldots,N\\}$, where
$v_{i}^{z}$ consists of an explicit semantic representation $v_{i}$ with an
implicit embedding $z_{i}$ encapsulating relation information with other
nodes.

Graph Manipulation. The manipulation includes node addition and relation
change, mimicking the user interaction. An example of manipulation is shown in
Fig. 2. We inherit the manipulation functionality of Graph-to-3D [15] and
CommonScenes [57] by setting up a graph manipulator, another triplet-GCN
module, to adjust the graph in the latent space, but with an orthogonal
strategy during training. We illustrate more details in the Supplementary
Materials. After the manipulation, Node features in the scene graph are
updated to $\mathcal{V}_{\mathcal{Z}}=\\{v_{i}^{z}~{}|~{}i=1,\ldots,M,M\geq
N$}. Subsequently, we send the updated graph to the diffusion-based layout
branch and shape branch, where we set each node with an individual denoising
process.

Figure 2: Overview of EchoScene. Our pipeline consists of graph preprocessing
and two collaborative branches Layout Branch and Shape Branch. The details of
two branches in one step are shown in Fig. 3. During inference, EchoScene
evolves the contextual graph to the latent space, where a manipulator
optionally adjust the graph by editing nodes and edges. Then, EchoScene
samples a random noise from Gaussian Distribution $\mathcal{B}$ and
$\mathcal{S}$ for both branches conditioned on the latent graph to generate
shapes and layouts. Finally, the generated shapes are populated into layouts
to synthesize the scenes. Moreover, an external texture generator [9] can be
optionally utilized to provide a more photorealistic appearance.

###  4.2 Information Echo Scheme

Before introducing the layout and shape branch, we illustrate the core of the
two branches–the information echo scheme in graph diffusion.

Challenges from a Dynamic Graph. While "Denoising Diffusion Probabilistic
Models" (DDPM) [19] has significantly advanced the field of generative models
by producing realistic and diverse content, their application to dynamic data
structures, such as scene graphs, is limited. Scene graphs contain indefinite
nodes and multiple edge combinations between nodes, describing compact and
flexible relationships among objects. Some concurrent work based on DDPM [51,
30] to ours either models the object graph as a set or preserves it with a
single edge between two nodes, where the representation ability of a scene
graph degrades. Besides, they set a maximum number of tokens and model them in
a single DDPM. As such, the local controllability of the generation is limited
when users want to manipulate the specific elements. We believe a better
solution is for each node in the graph to be allocated with an individual
denoising process, jointly forming a diffusion model for specific tasks,
which, in our case, are shape and layout generation. Thus, the content
generation is fully controllable by node and edge manipulation.

Echo-Formed Condition. However, even though each denoising process
theoretically realizes more controllability, it also brings isolation
problems. As each generation proceeds individually, there is no awareness of
scene content during the denoising steps, which makes the generation
inconsistent with global constraints in the graph. To achieve both highly
controllable and consistent generation at the same time, we propose to couple
the inverse iterative conditional diffusion mechanism on graph node features
with a recurrent intermediate message exchange strategy over the graph
$\mathcal{G}=\left\\{\mathcal{V},\mathcal{E}\right\\}$. In analogy to an echo
in the real world, a node can send its information and receive it back along
with information from other nodes during the steps of the inverse diffusion
process.

The cornerstone of echo is the introduction of an information exchange unit
$U$ that enables dynamic and interactive diffusion processes among the
elements of a dynamic graph. The incorporation of $U$ on the conditioner
during the inverse diffusion process allows each element within the group to
share and receive information. At every denoising step, each diffusion process
assembles the current denoising data ${\bf d}_{t}^{i}$ with other available
node attributes, where we choose latent feature $v_{i}^{z}$, to construct a
new node set $\mathcal{V}_{\mathcal{D}_{t}}:=\\{f\left({\bf
d}_{t}^{i},v_{i}^{z},\pi(t)\right)\ |\ i=1,\ldots,M\\}$, where $f(\cdot)$
depicts a feature assembling function, and we choose simple concatenation
here. $\pi(\cdot)$ illustrates the temporal process embedding function. Then,
the process sends $\mathcal{V}_{\mathcal{D}_{t}}$ to $U$, which
comprehensively understands group dynamics according to graph edges
$\mathcal{E}$, by subscribing and aggregating information from all processes.
In our task, we set $U$ based on triplet-GCN. The feature aggregation function
$U(\cdot)$ is based on Eq. 1, with input $(\beta_{v_{i}}^{(0)},\beta_{e_{i\to
j}}^{(0)},\beta_{v_{j}}^{(0)})=(v^{{\bf d}_{t}}_{i},e_{i\to j},v^{{\bf
d}_{t}}_{j})$, where $v^{{\bf d}_{t}}_{i}$ and $v^{{\bf d}_{t}}_{j}$ are
elements in $\mathcal{V}_{\mathcal{D}_{t}}$. The denoiser of each process
receives the aggregated features for conditioning. In this form, one sending
and one receiving step constitute an ‘ information echo.’ Note that the
Langevin dynamics here are different from the ones in a normal diffusion
model, where we introduce the relationship of a group of denoising data into
the condition $\mathcal{C}_{\mathcal{D}_{t}}$, making it temporarily changed
and steadily introducing constraints to generate next states. With forward
process variances $\beta_{t}$ and $\sigma_{t}=\sqrt{\beta_{t}}$ as well as the
shorthand notations $\alpha_{t}\coloneqq 1-\beta_{t}$ and
$\bar{\alpha}_{t}\coloneqq\prod_{s=1}^{t}\alpha_{s}$, we define one
conditional step in the denoising procedure as:

$\displaystyle{\bf d}_{t-1}^{i}$
$\displaystyle=\frac{1}{\sqrt{\alpha_{t}}}\left({\bf
d}_{t}^{i}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\varepsilon_{\theta}({\bf
d}_{t}^{i},\pi(t),\mathcal{C}_{\mathcal{D}_{t}})\right)+\sigma_{t}\mathbf{z},$
(3) $\displaystyle\mathcal{C}_{\mathcal{D}_{t}}$
$\displaystyle=U\left(\mathcal{G}_{\mathcal{D}_{t}}\right),\mathcal{G}_{\mathcal{D}_{t}}=\left\\{\mathcal{V}_{\mathcal{D}_{t}},\mathcal{E}\right\\},$

where $\mathbf{z}\sim\mathcal{N}(0,1)$ if $t>1$, else $\mathbf{z}=0$. The
information echo is repeated at every timestep, collectively forming the
information over the scene graph diffusion.

###  4.3 Layout Branch

As described in Sec. 4.2, we model the layout generation by setting each node
with its own denoising process and encouraging them to interact with their
diffused layout with each other at every denoising step.

Layout Parametrization. The scene layout is represented by object bounding
boxes. Initially, each bounding box ${\bf b}_{0}^{i}$ has 7 parameters, e.g.,
location $(x,y,z)$, size $(l,h,w)$, and a yaw angle $\theta$. Before training,
we normalize location and size by their maximum and minimum values on each
axis. For angles, we calculate its sine and cosine values. Thereby, the final
representation contains 8 parameters: ${\bf
b}_{0}^{i}=\\{x,y,z,l,h,w,\sin{\theta},\cos{\theta}\\}$, as shown in Fig.
3.A.3.

Layout Echoes. Since the bounding box generation needs to be compliant with
the spatial constraints described in the scene graph, state observation from
other nodes is needed to determine the bounding box of a specific node. In
this branch, we encourage all nodes to communicate their current diffused
bounding boxes ${\bf B}_{t}:=\\{{\bf b}_{t}^{i}~{}|~{}i=1,...,M\\}$ in the
graph by a layout exchange unit $U\mapsto U_{l}$ to achieve global awareness
at each time step. As shown in Fig. 3.A.1, in the implementation, we
substitute denoising data ${\bf d}_{t}^{i}$ to the diffused bounding box ${\bf
b}_{t}^{i}$, resulting in
$\mathcal{V}_{\mathcal{D}_{t}}\mapsto\mathcal{V}_{\mathcal{B}_{t}}$ and
$\mathcal{G}_{\mathcal{D}_{t}}\mapsto\mathcal{G}_{\mathcal{B}_{t}}$. Thus,
layout echoes happen at each time step by using Eq. 3, which are essential for
the functionality of this branch.

Training objective. We follow a normal DDPM training routine, in which we set
1000 time steps for all diffusion processes with weight-shared
$\gamma_{\theta}$. In each time step, the objective is to minimize the noise
prediction errors supervised by $\gamma$ sampled from Gaussian Distribution:

$\mathcal{L}_{layout}=\mathbb{E}_{{\bf
B},\gamma\sim\mathcal{N}(0,1),t}\left[||\gamma-\gamma_{\theta}({\bf
B}_{t},\pi(t),U_{l}(\mathcal{G}_{\mathcal{B}_{t}})||_{2}^{2}\right],\mathcal{G}_{\mathcal{B}_{t}}=\\{\mathcal{V}_{\mathcal{B}_{t}},\mathcal{E}\\}.$
(4)

Figure 3: One Step of Dual-Branch Information Echo. For each time step, we
encourage the layout (left) and shape (right) branches to exchange information
within each branch for all objects in the same scene.

###  4.4 Shape Branch

As shown in Fig. 2.B.2, we pretrain a VQ-VAE as a shape en-decoder and treat
the latent codes ${\bf X}_{0}$ in the bottleneck of the VQ-VAE as the ground
truth of the LDM, as deployed in CommonScenes [57]. Similar to our layout
branch, we introduce the information echo scheme to shape generation,
enhancing the generation consistency.

Isolation Problems. Unlike layout generation, shape generation can be
conditioned solely with the semantic information [12, 28] or relation
information [57], which means the shape branch is already functional when
allocating each node to a denoising process driven by semantics or latent
relation embeddings. However, each process still lacks consistency control
from the shape appearance aspect. In other words, each generation process only
focuses on its own, while ignoring others’ shape generation state throughout
the whole denoising. Thus, the generations are isolated from each other,
resulting in suboptimal inter-object consistency of the global style.

Shape Echoes. We solve the problem by introducing shape observation of other
processes for each process, which is achieved by shape echoes. As shown in
Fig. 3.B.3, before shape echoes happen, we first feed diffused shape codes
${\bf X}_{t}:=\\{{\bf x}_{t}^{i},~{}|~{}i=1,...,M\\}$ to 3D convolutional
layers and flatten to ${\bf S}_{t}:=\\{{\bf s}_{t}^{i},~{}|~{}i=1,...,M\\}$,
aligning the dimension with the node’s attribute. The meaning of the ${\bf
X}_{t}$ and ${\bf S}_{t}$ becomes more obvious when the time step is closer to
0. Similar to the layout branch, we substitute denoising data ${\bf
d}_{t}^{i}$ to the diffused shape code ${\bf s}_{t}^{i}$, resulting in
$\mathcal{V}_{\mathcal{D}_{t}}\mapsto\mathcal{V}_{\mathcal{S}_{t}}$ and
$\mathcal{G}_{\mathcal{D}_{t}}\mapsto\mathcal{G}_{\mathcal{S}_{t}}$. Thus,
with a shape exchange unit $U\mapsto U_{s}$, shape echoes happen at each time
step by using Eq. 3.

Training objective. For each latent diffusion process, the weight-shared
denoiser $\varepsilon_{\theta}$ takes time step $t$, shape codes ${\bf
X}_{t}$, and shape-aggregated feature as input to predict the noise, which is
supervised by $\varepsilon$ sampled from Gaussian Distribution. The objective
of the training is to minimize the noise prediction errors:

$\mathcal{L}_{shape}=\mathbb{E}_{{\bf
X},\varepsilon\sim\mathcal{N}(0,1),t}\left[||\varepsilon-\varepsilon_{\theta}({\bf
X}_{t},\pi(t),U_{s}(\mathcal{G}_{\mathcal{S}_{t}})||_{2}^{2}\right],\mathcal{G}_{\mathcal{S}_{t}}=\\{\mathcal{V}_{\mathcal{S}_{t}},\mathcal{E}\\}$
(5)

###  4.5 Dual-Branch Joint Training

Since both branches are based on diffusion models, the framework supports end-
to-end synchronized training optimized by two losses weighted by
${\lambda_{1},\lambda_{2}}$:

$\mathcal{L}=\lambda_{1}\mathcal{L}_{layout}+\lambda_{2}\mathcal{L}_{shape}.$
(6)

##  5 Experiments

Dataset. We conduct our experiments on SG-FRONT dataset [57], which provides
scene-graph annotations for the high-quality 3D-FRONT [16] with household
environments. SG-FRONT contains 15 relationship types and 45K object instances
from three types of scenes.

Metrics. We follow the metrics used in [57, 51, 30], to measure the _fidelity
and diversity_ of generated scenes, where we adopt Fréchet Inception Distance
(FID) [18], $\text{FID}_{\text{CLIP}}$ [26] and Kernel Inception Distance
(KID) [3] metrics [40]. To measure the _scene graph consistency_ , we follow
the scene graph constraints [15], which measure the accuracy of a set of
relations on a generated layout. To measure the _shape consistency and
diversity_ following [57], we test dining rooms, a typical scenario in which
tables and chairs should be in suits.

Baselines. For retrieval-based methods _i.e_., a layout generation network
_3D-SLN_ [34], a progressive method to add objects one-by-one designed in
[15], _Graph-to-Box_ from [15], _DiffuScene_ [51] with text conditioning, and
_InstructScene_ [30] with graph conditioning. For generative methods, _Graph-
to-3D_[15], modeling shape and layouts both in VAE. CommonScenes [57], which
model scene layouts through VAE and generate each object via LDM conditioned
on relation embeddings. _EchoLayout/CommonLayout+SDFusion_ that we design
stacking the layout branch and a text-to-shape generation model SDFusion [12]
sequentially. More details are in Supplementary Material.

Implementation Details. The training, evaluation, and visualization are
carried out on a single NVIDIA A40 GPU with 40 GB of memory. Training is
optimized with AdamW with an initial learning rate of 1e-4. The values of
${\lambda_{1},\lambda_{2}}$ are set to 1.0. More details are provided in
Supplementary Material.

###  5.1 Scene Fidelity

Method | Shape | Bedroom | Living room | Dining room
---|---|---|---|---
Representation | FID | $\text{FID}_{\text{CLIP}}$ | KID | FID | $\text{FID}_{\text{CLIP}}$ | KID | FID | $\text{FID}_{\text{CLIP}}$ | KID
3D-SLN [34] |  Retrieval | 57.90 |  05.45 |  03.85 | 77.82 |  07.02 |  03.65 | 69.13 |  07.99 |  06.23
Progressive [15] | 58.01 |  05.67 |  07.36 | 79.84 |  07.41 |  04.24 | 71.35 |  08.28 |  06.21
Graph-to-Box [15] | 54.61 |  05.26 |  02.93 | 78.53 |  06.88 |  03.32 | 67.80 |  07.75 |  06.30
CommonLayout [57] | 52.69 |  05.22 |  02.82 | 76.52 |  06.58 |  02.08 | 65.10 |  07.55 |  06.11
DiffuScene [51] | 52.02 |  05.01 |  02.52 | 81.61 |  07.52 |  01.23 | 65.90 |  07.39 |  00.09
InstructScene∗ [30] | 45.40 |  03.87 |  01.06 | 75.83 |  06.98 |  04.15 | 61.56 |  06.49 |  04.90
EchoLayout (Ours) | 46.53 |  04.24 |  00.33 | 75.54 |  06.35 | 1.60 | 59.66 |  06.24 |  02.63
Graph-to-3D [15] | DeepSDF [39] | 63.72 |  06.01 | 17.02 | 82.96 |  07.80 | 11.07 | 72.51 |  07.25 | 12.74
CommonLayout+SDFusion [12] | txt2shape | 68.08 |  05.61 | 18.64 | 85.38 |  07.23 | 10.04 | 64.02 |  06.92 |  05.08
EchoLayout+SDFusion [12] | txt2shape | 57.68 |  04.96 | 10.54 | 83.66 |  06.83 |  09.62 | 65.55 |  07.02 |  04.99
CommonScenes [57] | rel2shape | 57.68 |  04.86 |  06.59 | 80.99 |  07.05 |  06.39 | 65.71 |  07.04 |  05.47
EchoScene (Ours) | echo2shape | 48.85 |  04.26 |  01.77 | 75.95 |  06.73 |  00.60 | 62.85 |  06.28 |  01.72

Table 1: Scene generation realism as measured by FID,
$\text{FID}_{\text{CLIP}}$ and KID $(\times 0.001)$ scores at $256^{2}$ pixels
between the top-down rendering of generated and real scenes, following [51,
57] (lower is better). We color the best and the second. Two main rows are
separated with respect to the reliance on an external shape database.
_InstructScene ∗_ represents its 3D layout decoder solely. Both _EchoLayout_
and _CommonLayout_ refer to the single layout branch.

As a generation task, the fidelity of scene synthesis is essential to measure.

Quantitative results. We perform two types of scene synthesis, which are shape
retrieval-based and generation-based synthesis, and provide FID /
$\text{FID}_{\text{CLIP}}$ / KID in Tab. 1. For retrieval, our layout branch
$EchoLayout$ performs better than the layout branch of the previous SoTA
CommonLayout [57] across all metrics, and concurrent work DiffuScene [51] on
most of the metrics. In terms of shape generation, our 3D scene generation
version $EchoScene$ shows clear advantages against CommonScenes [57], for
example, improving FID by $15\%$, $\text{FID}_{\text{CLIP}}$ by $12\%$ and KID
by $73\%$ in the bedroom generation. Notably, although retrieval-based methods
overall score better due to the mesh alignment with the test database,
EchoScene remains competitive, showcasing generative effectiveness. Moreover,
our text-to-shape baseline _Echolayout+SDFusion_ outperforms the equivalent
CommonLayout+SDFusion. Despite under the issue of shape inconsistency, it
still achieves similar results as CommonScenes. This also underscores the
effectiveness of our layout branch, driven by layout echoes, in generating
more coherent scene layouts. For shape consistency, we test the Chamfer
Distance (CD) between two generated objects that are supposed to be identical
(_i.e_. shape consistency [57]). We report common objects in three room types
in Tab. 3, showing that our shape echoes alleviate the isolation problem in
CommonScenes and thus bring better inter-object consistency.

Figure 4: Comparisons with other generative methods. Input scene graphs have
more edges between two nodes than the ones visualized here. Red rectangles
highlight the inconsistent generation. (Zoom for details)

Qualitative results. Graph-to-3D [15] and CommonScenes [57] have the same
settings as EchoScene, which belong to (semi-/full-)generative models
conditioned on scene graphs for the whole scene generation. We provide a
comparison of these methods for each different room type in Fig. 4. For
example, in the bedroom, beds and nightstands of Graph-to-3D and CommonScenes
are in twisted poses, indicating VAE-based layout generator does not learn
angles well, while our diffusion-based architecture can provide neat results.
In the dining room, Graph-to-3D generates chairs in different styles, which
breaks the inter-object consistency. Although CommonScenes can generate chairs
in a suit, the generated table on the left is problematic, having a chair-like
shape, which makes an unsatisfactory global appearance as well. Ours achieves
higher shape consistency.

Method |  Shape Representation |  Mode |  left/ right |  front/ behind |  smaller/ larger |  taller/ shorter |  close by |  symmetrical
---|---|---|---|---|---|---|---|---
3D-SLN [34] |  Retrieval |  Change | 0.89 | 0.90 | 0.55 | 0.58 | 0.10 | 0.09
Progressive [15] | 0.89 | 0.89 | 0.52 | 0.55 | 0.08 | 0.09
Graph-to-Box [15] | 0.91 | 0.91 | 0.86 | 0.91 | 0.66 | 0.53
CommonLayout [57] | 0.91 | 0.92 | 0.86 | 0.92 | 0.70 | 0.53
EchoLayout (Ours) | 0.94 | 0.93 | 0.92 | 0.92 | 0.72 | 0.56
Graph-to-3D [15] | DeepSDF [39] | 0.91 | 0.92 | 0.86 | 0.89 | 0.69 | 0.46
CommonScenes [57] | rel2shape | 0.91 | 0.92 | 0.86 | 0.91 | 0.69 | 0.59
EchoScene (Ours) | echo2shape | 0.94 | 0.96 | 0.92 | 0.93 | 0.74 | 0.50
3D-SLN [34] |  Retrieval |  Addition | 0.92 | 0.92 | 0.56 | 0.58 | 0.05 | 0.05
Progressive [15] | 0.92 | 0.91 | 0.53 | 0.54 | 0.02 | 0.06
Graph-to-Box [15] | 0.94 | 0.93 | 0.90 | 0.94 | 0.67 | 0.58
CommonLayout [57] | 0.95 | 0.95 | 0.90 | 0.94 | 0.73 | 0.63
EchoLayout (Ours) | 0.98 | 0.99 | 0.97 | 0.96 | 0.74 | 0.58
Graph-to-3D [15] | DeepSDF [39] | 0.94 | 0.95 | 0.91 | 0.93 | 0.63 | 0.47
CommonScenes [57] | rel2shape | 0.95 | 0.95 | 0.91 | 0.95 | 0.70 | 0.61
EchoScene (Ours) | echo2shape | 0.98 | 0.99 | 0.96 | 0.96 | 0.76 | 0.49
3D-SLN [34] |  Retrieval |  None | 0.97 | 0.99 | 0.95 | 0.91 | 0.72 | 0.47
Progressive [15] | 0.97 | 0.99 | 0.95 | 0.82 | 0.69 | 0.46
Graph-to-Box [15] | 0.98 | 0.99 | 0.96 | 0.95 | 0.72 | 0.45
CommonLayout [57] | 0.98 | 0.99 | 0.97 | 0.95 | 0.74 | 0.63
InstructScene∗ [30] | 0.99 | 0.99 | 0.85 | 0.86 | 0.52 | 0.58
EchoLayout (Ours) | 1.00 | 0.99 | 0.95 | 0.96 | 0.74 | 0.67
Graph-to-3D [15] | DeepSDF [39] | 0.98 | 0.99 | 0.97 | 0.95 | 0.74 | 0.57
CommonScenes [57] | rel2shape | 0.98 | 1.00 | 0.97 | 0.95 | 0.77 | 0.60
EchoScene (Ours) | echo2shape | 0.98 | 0.99 | 0.96 | 0.96 | 0.74 | 0.55

Table 2: Scene graph constraints (higher is better). Top: Relationship change
mode. Middle: Node addition mode. Bottom: No manipulation (_i.e_., generation
only).

###  5.2 Graph Constraints

In this part, we evaluate the layout generation performance with respect to
scene graph constraints. We compare against the closest methods: 3D-SLN [34],
Graph-to-3D [15], InstructScene [30], and CommonScenes [57].

We optionally manipulate scene graphs in the latent by either adding a node
and relevant edges to the graph or changing relations between two nodes, as
illustrated in Fig. 2. As shown in the top and middle row of Tab. 2, for both
node addition and relation change, EchoLayout and EchoScene perform better on
most metrics. 3D-SLN, CommonScenes, and Graph-to-3D, for instance, loose the
constraints smaller/bigger than and close by after graph manipulation, while
ours largely keeps it. We believe that the improvement is caused by our scene
graph diffusion strategy, where a graph-consistent sample is generated within
a realistic appearance distribution. The decrease in symmertical category
compared with CommonScenes is likely caused by the same effect. Inspection of
the data reveals that symmertical is an extremely rare label in the
supplementary of CommonScenes [56], with an occurrence of only 0.9%. Thus, the
learned part of the distribution is underrepresented and less accurate. The
residual echo correction diffuses the latent code towards the learned
distribution, resulting in a slight variation. In the bottom row, we report
the results without manipulation. All baselines show good performance, yet our
EchoLayout and EchoScene are still competitive on their own track. Notably,
even though InstructScene shows impressive performance on the global
generation fidelity shown in Tab. 1, it struggles to maintain the multiple
graph constraints simultaneously, as we described in Sec. 1, while our
information echo scheme can sufficiently aggregate relations and condition
each process to achieve better results. Distribution-dependant effects from
data equally manifest in the results where we can see some drop for
symmertical again. Interestingly, the effect is less pronounced for a lower
parametric final task such as layout generation. This can be explained by the
fact that the variability of bounding boxes is smaller compared to shape
variations.

###  5.3 Ablation study

We ablate three parts of the full version and report the performance in Tab.
4. Firstly, we ablate the time embeddings $\pi(t)$ to check the influence on
the model without explicit time information. We observe some marginal decrease
compared with the full version, indicating even without $\pi(t)$, the model
still learns temporal information through the denoising data at every step.
Secondly, we exclude the shape echoes to degrade the shape branch as the one
in CommonScenes. We clearly see a drop in the scene fidelity evaluation, where
‘mSG’ focusing on the layouts is not influenced. Thirdly, we substitute cross-
attention for concatenation to check if the way of the condition influences
the information echo scheme. We also observe a drop in all metrics, especially
for fidelity.

Figure 5: Off-the-shelf texture creation. A bedroom (top) and living room
(bottom) generated by EchoScene and textured in different styles by SceneTex
[9].

Method | Bedroom | Living room | Dining room
---|---|---|---
Ward. | N.stand | Chair | Table | Chair | Table
CommonScenes [57] | 0.61 | 2.69 | 6.64 | 11.75 | 1.96 | 9.04
EchoScene | 0.14 | 1.68 | 0.99 | 3.02 | 1.75 | 1.26

Table 3: Inter-object Consistency. The consistent object shapes within a scene
are indicated by low CD values ($\times 0.001$).

Ablation | FID | $\text{FID}_{\text{CLIP}}$ | KID | mSG
---|---|---|---|---
Ours w/o $\pi(t)$ | 40.55 | 3.14 | 1.69 | 0.87
Ours w/o shape echoes | 46.88 | 3.81 | 4.17 | 0.88
Ours with concat | 48.32 | 3.82 | 6.87 | 0.87
Ours | 39.74 | 3.14 | 1.24 | 0.88

Table 4: Ablations under three circumstances. mSG means average graph
constraints.

###  5.4 Application

EchoScene can serve multiple purposes, including but not limited to robotic
imagination and manipulation [55]. However, the limitation of EchoScene is
that it can generate semantic yet only textureless scenes, which prohibits it
from downstream tasks requiring photorealistic textures. Yet, the generated
scenes are of high quality, so that the limitation can be alleviated by
directly equipping EchoScene with a subsequent off-the-shelf texture
generator. Here, we demonstrate that textured scenes can be generated via
SceneTex [9] in Fig. 5.

##  6 Conclusion

We have introduced EchoScene, a scene-generative method based on dual-branch
diffusion models, which are layout branch and shape branch. To make two
branches functional, we have proposed an information scheme encouraging
information exchange within both branches at each denoising step, generating
final scenes that are compliant with the scene graph description. Experiments
show that EchoScene can achieve higher generation fidelity and more inter-
object consistency of the global scene style.

## References

  * [1] Bahmani, S., Park, J.J., Paschalidou, D., Yan, X., Wetzstein, G., Guibas, L., Tagliasacchi, A.: CC3D: Layout-conditioned generation of compositional 3D scenes. In: ICCV (2023)
  * [2] Bautista, M.A., Guo, P., Abnar, S., Talbott, W., Toshev, A., Chen, Z., Dinh, L., Zhai, S., Goh, H., Ulbricht, D., et al.: Gaudi: A neural architect for immersive 3D scene generation. In: NeurIPS (2022)
  * [3] Bińkowski, M., Sutherland, D.J., Arbel, M., Gretton, A.: Demystifying MMD GANs. In: ICLR (2018)
  * [4] Brooks, T., Holynski, A., Efros, A.A.: InstructPix2Pix: Learning to follow image editing instructions. In: CVPR (2023)
  * [5] Chang, A., Monroe, W., Savva, M., Potts, C., Manning, C.D.: Text to 3D scene generation with rich lexical grounding. In: ACL (2015)
  * [6] Chang, A., Savva, M., Manning, C.D.: Learning spatial knowledge for text to 3D scene generation. In: EMNLP (2014)
  * [7] Chang, X., Ren, P., Xu, P., Li, Z., Chen, X., Hauptmann, A.: A comprehensive survey of scene graphs: Generation and application. T-PAMI (2021)
  * [8] Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-Excite: Attention-based semantic guidance for text-to-image diffusion models. In: SIGGRAPH (2023)
  * [9] Chen, D.Z., Li, H., Lee, H.Y., Tulyakov, S., Nießner, M.: SceneTex: High-quality texture synthesis for indoor scenes via diffusion priors. In: CVPR (2024)
  * [10] Chen, D.Z., Siddiqui, Y., Lee, H.Y., Tulyakov, S., Nießner, M.: Text2Tex: Text-driven texture synthesis via diffusion models. In: ICCV (2023)
  * [11] Chen, Z., Wang, G., Liu, Z.: SceneDreamer: Unbounded 3D scene generation from 2d image collections. T-PAMI (2023)
  * [12] Cheng, Y.C., Lee, H.Y., Tuyakov, S., Schwing, A., Gui, L.: SDFusion: Multimodal 3D shape completion, reconstruction, and generation. In: CVPR (2023)
  * [13] Cohen-Bar, D., Richardson, E., Metzer, G., Giryes, R., Cohen-Or, D.: Set-the-Scene: Global-local training for generating controllable nerf scenes. In: ICCV Workshop (2023)
  * [14] Dhamo, H., Farshad, A., Laina, I., Navab, N., Hager, G.D., Tombari, F., Rupprecht, C.: Semantic image manipulation using scene graphs. In: CVPR (2020)
  * [15] Dhamo, H., Manhardt, F., Navab, N., Tombari, F.: Graph-to-3D: End-to-end generation and manipulation of 3D scenes using scene graphs. In: ICCV (2021)
  * [16] Fu, H., Cai, B., Gao, L., Zhang, L.X., Wang, J., Li, C., Zeng, Q., Sun, C., Jia, R., Zhao, B., et al.: 3D-FRONT: 3D furnished rooms with layouts and semantics. In: ICCV (2021)
  * [17] Gao, G., Liu, W., Chen, A., Geiger, A., Schölkopf, B.: GraphDreamer: Compositional 3D scene synthesis from scene graphs. In: CVPR (2024)
  * [18] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: NeurIPS (2017)
  * [19] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS (2020)
  * [20] Johnson, J., Gupta, A., Fei-Fei, L.: Image generation from scene graphs. In: CVPR (2018)
  * [21] Johnson, J., Krishna, R., Stark, M., Li, L.J., Shamma, D.A., Bernstein, M.S., Fei-Fei, L.: Image retrieval using scene graphs. In: CVPR (2015)
  * [22] Jyothi, A.A., Durand, T., He, J., Sigal, L., Mori, G.: LayoutVAE: Stochastic scene layout generation from a label set. In: ICCV (2019)
  * [23] Kenton, J.D.M.W.C., Toutanova, L.K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (2019)
  * [24] Kong, L., Cui, J., Sun, H., Zhuang, Y., Prakash, B.A., Zhang, C.: Autoregressive diffusion model for graph generation. In: ICML (2023)
  * [25] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual Genome: Connecting language and vision using crowdsourced dense image annotations. IJCV (2017)
  * [26] Kynkäänniemi, T., Karras, T., Aittala, M., Aila, T., Lehtinen, J.: The role of imagenet classes in fréchet inception distance. In: ICLR (2023)
  * [27] Li, M., Patil, A.G., Xu, K., Chaudhuri, S., Khan, O., Shamir, A., Tu, C., Chen, B., Cohen-Or, D., Zhang, H.: GRAINS: Generative recursive autoencoders for indoor scenes. TOG (2019)
  * [28] Li, M., Duan, Y., Zhou, J., Lu, J.: Diffusion-SDF: Text-to-shape via voxelized diffusion. In: CVPR (2023)
  * [29] Li, X., Zhang, Y., Ye, X.: DrivingDiffusion: Layout-guided multi-view driving scene video generation with latent diffusion model (2023)
  * [30] Lin, C., MU, Y.: InstructScene: Instruction-driven 3D indoor scene synthesis with semantic graph prior. In: ICLR (2024)
  * [31] Lin, Y., Bai, H., Li, S., Lu, H., Lin, X., Xiong, H., Wang, L.: CompoNeRF: Text-guided multi-object compositional nerf with editable 3D scene layout (2023)
  * [32] Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-1-to-3: Zero-shot one image to 3D object. In: ICCV (2023)
  * [33] Liu, Z., Feng, Y., Black, M.J., Nowrouzezahrai, D., Paull, L., Liu, W.: MeshDiffusion: Score-based generative 3D mesh modeling. In: ICLR (2023)
  * [34] Luo, A., Zhang, Z., Wu, J., Tenenbaum, J.B.: End-to-end optimization of scene layout. In: CVPR (2020)
  * [35] Ma, R., Patil, A.G., Fisher, M., Li, M., Pirk, S., Hua, B.S., Yeung, S.K., Tong, X., Guibas, L., Zhang, H.: Language-driven synthesis of 3D scenes from scene databases. TOG (2018)
  * [36] Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y., Fan, L., Zhu, Y., Fox, D.: MimicGen: A data generation system for scalable robot learning using human demonstrations. In: CoRL (2023)
  * [37] Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.Y., Ermon, S.: SDEdit: Guided image synthesis and editing with stochastic differential equations. In: ICLR (2022)
  * [38] Özsoy, E., Örnek, E.P., Eck, U., Tombari, F., Navab, N.: Multimodal semantic scene graphs for holistic modeling of surgical procedures. arXiv preprint (2021)
  * [39] Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: Learning continuous signed distance functions for shape representation. In: CVPR (2019)
  * [40] Paschalidou, D., Kar, A., Shugrina, M., Kreis, K., Geiger, A., Fidler, S.: ATISS: Autoregressive transformers for indoor scene synthesis. In: NeurIPS (2021)
  * [41] Po, R., Wetzstein, G.: Compositional 3D scene generation using locally conditioned diffusion. In: 3DV (2023)
  * [42] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: DreamFusion: Text-to-3D using 2d diffusion. In: ICLR (2023)
  * [43] Pronovost, E., Ganesina, M.R., Hendy, N., Wang, Z., Morales, A., Wang, K., Roy, N.: Scenario Diffusion: Controllable driving scenario generation with diffusion. In: NeurIPS (2023)
  * [44] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022)
  * [45] Rosinol, A., Gupta, A., Abate, M., Shi, J., Carlone, L.: 3D Dynamic Scene Graphs: Actionable spatial perception with places, objects, and humans. In: RSS (2020)
  * [46] Shi, Y., Wang, P., Ye, J., Mai, L., Li, K., Yang, X.: MVDream: Multi-view diffusion for 3D generation. In: ICLR (2024)
  * [47] Simsar, E., Tonioni, A., Örnek, E.P., Tombari, F.: LatentSwap3D: Semantic edits on 3D image gans. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops. pp. 2899–2909 (October 2023)
  * [48] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: ICML (2015)
  * [49] Song, Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems 32 (2019)
  * [50] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. In: ICLR (2021)
  * [51] Tang, J., Nie, Y., Markhasin, L., Dai, A., Thies, J., Nießner, M.: DiffuScene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis. In: CVPR (2024)
  * [52] Verma, T., De, A., Agrawal, Y., Vinay, V., Chakrabarti, S.: VarScene: A deep generative model for realistic scene graph synthesis. In: ICML (2022)
  * [53] Vignac, C., Krawczuk, I., Siraudin, A., Wang, B., Cevher, V., Frossard, P.: DiGress: Discrete denoising diffusion for graph generation. In: International Conference on Learning Representations (ICLR) (2023)
  * [54] Wald, J., Dhamo, H., Navab, N., Tombari, F.: Learning 3D semantic scene graphs from 3D indoor reconstructions. In: CVPR (2020)
  * [55] Zhai, G., Cai, X., Huang, D., Di, Y., Manhardt, F., Tombari, F., Navab, N., Busam, B.: SG-Bot: Object rearrangement via coarse-to-fine robotic imagination on scene graphs. In: ICRA (2024)
  * [56] Zhai, G., Örnek, E.P., Wu, S.C., Di, Y., Tombari, F., Navab, N., Busam, B.: Supplementary material CommonScenes: Generating commonsense 3D indoor scenes with scene graph diffusion
  * [57] Zhai, G., Örnek, E.P., Wu, S.C., Di, Y., Tombari, F., Navab, N., Busam, B.: CommonScenes: Generating commonsense 3D indoor scenes with scene graphs. In: NeurIPS (2023)
  * [58] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: ICCV (2023)
  * [59] Zhou, X., Ran, X., Xiong, Y., He, J., Lin, Z., Wang, Y., Sun, D., Yang, M.H.: GALA3D: Towards text-to-3D complex scene generation via layout-guided generative gaussian splatting (2024)
  * [60] Zhou, Y., While, Z., Kalogerakis, E.: SceneGraphNet: Neural message passing for 3D indoor scene augmentation. In: ICCV (2019)
  * [61] Özsoy, E., Örnek, E.P., Eck, U., Czempiel, T., Tombari, F., Navab, N.: 4D-OR: Semantic scene graphs for or domain modeling. In: MICCAI (2022)

## Supplementary Material of EchoScene

In this Supplementary Material, we report the following:

  * •

Section 0.A: Manipulation Strategy.

  * •

Section 0.B: Manipulation Qualitatives.

  * •

Section 0.C: Additional Results.

  * •

Section 0.D: Additional Qualitatives.

  * •

Section 0.E: Additional Experimental Details.

##  Appendix 0.A Manipulation Strategy

We propose a novel training strategy for node and edge manipulation tailored
for scene graph diffusion, thus editing the generated scenes.

VAE$+$GAN-based strategy. In the previous methods [57, 15], the layout branch
is modeled by a VAE architecture. During training, the input scene graph must
be augmented by ground truth bounding boxes. In this case, pseudo layouts are
created from the output side after the manipulation, and there is no matching
ground truth to supervise them. To make the pipeline functional, they set an
additional GAN module to help train the pseudo label. For example, as shown in
Fig. 6.A, the original relationship is {Bed $\longrightarrow$ Left of
$\longrightarrow$ TV stand }, with a manipulator changing it to {Bed
$\longrightarrow$ Right of $\longrightarrow$ TV stand }. Only the bounding
boxes on the input side exist in the dataset, while the ones on the output
side are generated and need to be discriminated. Such a manipulation strategy
relies on the performance of GAN, whose training scheme is not stable and
requires large-scale training, consequently affecting the training quality of
layout VAE. Thus, the results shown in Tab. 2 in the main paper are always
unsatisfactory.

Figure 6: Manipulation workflow. (Top) Previous VAE-based methods set the
input as real and use a GAN-based module to facilitate the output prediction.
(Bottom) Our Diffusion-based framework inverse the workflow by starting with a
pseudo graph and ending with ground truth, without redundant modules.

Our strategy. In contrast, we set both branches through diffusion processes,
avoiding the necessity of input ground truth bounding boxes during training.
Instead, we are able to simulate an inverted workflow by setting input as
pseudo graphs with fake relations, and we manipulate them back to the existing
graphs in the dataset. For example, as shown in Fig. 6.B, the ground truth
relation label is {Bed $\longrightarrow$ Right of $\longrightarrow$
Nightstand}, we simulate the relation changing by setting an arbitrary label,
e.g., {Bed $\longrightarrow$ Front of $\longrightarrow$ Nightstand}. In this
case, we do not need to provide real data as input and further do not set an
additional GAN module to facilitate the training. Thus, the arbitrary pseudo
side enables large-scale training, maintaining a good manipulation
performance. More importantly, we do not need to discriminate the final layout
prediction. With our proposed strategy, we maintain the typical diffusion
training routine, which has Gaussian noise as supervision during training
without further exhausting inference. Last but not least, pure diffusion
dynamics guarantee stable training.

Figure 7: Scene manipulation. (Upper main row) Object addition to the scene
graph. (Bottom main row) Partial relation change in the graph. (Zoom for
details)

##  Appendix 0.B Manipulation Qualitatives

We provide several qualitative results of the scenes before and after
manipulation in Fig. 7. The procedure includes EchoScene generating a scene
based on a provided scene graph, followed by scene modification according to
adjustments made to the graph.

Object addition. We first show object addition in the scene through the
incorporation of new nodes and their corresponding edges into the scene graph.
For instance, the second row illustrates the addition of a sofa node to the
graph. Upon this addition, EchoScene adeptly reconfigures the scene, modifying
the pose and appearance of pre-existing objects to seamlessly integrate the
sofa into the scene.

Relation change. Next, we show object rearrangement within the scene through
modifications to specific graph relations. The depiction is organized row-
wise, showcasing various types of edge manipulations: the first row highlights
the front of/behind relationship, the second focuses on bigger/smaller than,
and the final row on left/right of. An interesting case is observed in the
second row, where the bed is adjusted to be bigger than the wardrobe from the
volume perspective. This particular relationship between the bed and wardrobe
is comparatively rare within the dataset. Yet, EchoScene can jointly adjust
the bounding box sizes of both the wardrobe and bed to achieve the goal. In
the last row, EchoScene successfully alters the relationship between the desk
and chair, while reorienting the chair’s pose to face the desk, thereby
maintaining inter-object consistency within the scene.

##  Appendix 0.C Additional Results

We believe our shape branch driven by shape echoes can bring more object
generation compliance to the global scenes. Thus, we further conduct object-
level analysis, following [57], to report the MMD ($\times 0.01$), COV (%),
and 1-nearest neighbor accuracy (1-NNA, %) for evaluating per-object
generation. As shown in the first two rows of Table 5, our method shows better
performance on most of the categories in both MMD and COV, which highlights
the object-level shape generation ability of EchoScene. 1-NNA directly
measures distributional similarity to the ground truth objects in both
diversity and quality. The closer 1-NNA is to 50%, the better the shape
distribution is captured. It can be observed that in most of the categories,
our method surpasses CommonScenes in the evaluation of distributional
similarity. Overall, EchoScene exhibits more plausible object-level generation
than the previous state-of-the-art.

Method | Metric | Bed | N.stand | Ward. | Chair | Table | Cabinet | Lamp | Shelf | Sofa | TV stand
---|---|---|---|---|---|---|---|---|---|---|---
Graph-to-3D [15] | MMD ($\downarrow$) | 1.56 | 3.91 | 1.66 | 2.68 | 5.77 | 3.67 | 6.53 | 6.66 | 1.30 | 1.08
CommonScenes [57] | 0.49 | 0.92 | 0.54 | 0.99 | 1.91 | 0.96 | 1.50 | 2.73 | 0.57 | 0.29
EchoScene (Ours) | 0.37 | 0.75 | 0.39 | 0.62 | 1.47 | 0.83 | 0.66 | 2.52 | 0.48 | 0.35
Graph-to-3D [15] | COV ($\%,\uparrow$) | 4.32 | 1.42 | 5.04 | 6.90 | 6.03 | 3.45 | 2.59 | 13.33 | 0.86 | 1.86
CommonScenes [57] | 24.07 | 24.17 | 26.62 | 26.72 | 40.52 | 28.45 | 36.21 | 40.00 | 28.45 | 33.62
EchoScene (Ours) | 39.51 | 25.59 | 37.07 | 17.25 | 35.05 | 43.21 | 33.33 | 50.00 | 41.94 | 40.70
Graph-to-3D [15] | 1-NNA ($\%,\downarrow)$ | 98.15 | 99.76 | 98.20 | 97.84 | 98.28 | 98.71 | 99.14 | 93.33 | 99.14 | 99.57
CommonScenes [57] | 85.49 | 95.26 | 88.13 | 86.21 | 75.00 | 80.17 | 71.55 | 66.67 | 85.34 | 78.88
EchoScene (Ours) | 72.84 | 91.00 | 81.90 | 92.67 | 75.74 | 69.14 | 78.90 | 35.00 | 69.35 | 78.49

Table 5: Object-level generation performance. We report MMD($\times 0.01$),
COV and 1-NNA for evaluating shapes by means of quality and diversity. Figure
8: More comparisons with other methods. Red rectangles highlight the
inconsistent generation. (Zoom for details)  Figure 9: Comparisons with other
retrieval methods. Input scene graphs have more edges between two nodes than
the ones visualized here. Red rectangles highlight the inaccurate graph
constraints. (Zoom for details)  Figure 10: Off-the-shelf texture creation. A
bedroom with a complex structure bed inside generated by EchoScene and
textured in different styles by SceneTex [9].

##  Appendix 0.D Additional Qualitatives

Generative methods. We first show more quantitative results in Fig. 8. Our
method can achieve more inter-object consistency and generation quality. For
example, in the bedroom, Graph-to-3D fails to achieve desk-and-chair
consistency, while EchoScene can generate a desk whose appearance is closer to
a desk in the real scenario with a chair suitable for it. In the living and
dining rooms, Graph-to-3D either fails on shape consistency or angle
predictions of chairs. CommonScenes cannot guarantee a fine-grained shape
consistency, while the chairs coarsely look similar. In contrast, EchoScene
can generate coherent shapes and make pose prediction more accurate.

Retrieval methods. Retrieval methods select objects from the database based on
how closely their bounding box sizes match those of the generated layouts.
Therefore, this line of work suffers from inter-object inconsistency; for
example, chairs are not recalled in suits. Such inconsistencies often stem
from even minor misalignment in the size of generated bounding boxes, leading
to the selection of entirely different objects than intended. Despite this,
our focus in Figure 9 is to illustrate the effectiveness of graph constraints,
instead of consistency. It is observable that even though InstructScene [30]
demonstrates the capability to generate objects in a decent manner, it fails
to adhere to the partial graph constraints. On the contrary, both CommonLayout
[57] and EchoLayout exhibit proficiency in complying with these constraints.

Texture Generation. We finally show additional texture generation on a
relatively complex structured bedroom in Fig. 10. EchoScene can provide well-
generated scenes that are compatible with an off-the-shelf SceneTex [9] to
generate textures.

##  Appendix 0.E Additional Experimental Details

###  0.E.1 Baselines.

Graph-to-3D series [15]. This series includes one generative method and three
object retrieval methods. First, the full generative version _Graph-to-3D_ ,
stacking two VAE-based branches for object and layout generation,
respectively. Second, _Graph-to-Box_ , the single layout branch focusing on
the object retrieval task. Third, _Progressive_ , a modified baseline upon
Graph-to-Box, specifically adding objects one by one in an autoregressive
manner. Fourth, _3D-SLN_ [34], sharing the same architecture as Graph-to-Box,
but without layout standardization during training. We follow the illustration
in the supplementary of CommonScenes [56].

CommonScenes series [57]. This series includes the fully generative version
_CommonScenes_ , and its layout branch for object retrieval, _CommonLayout_.
We follow the illustration in the supplementary of CommonScenes [56].

Text-to-shape series. This series includes two generative baselines. One is
built upon CommonScenes called _CommonLayout+SDFusion_ , and the other is
_EchoLayout+SDFusion_ , with EchoLayout as our layout branch. Both methods
achieve the fully generative ability by first generating the bounding boxes
and further generating shapes upon a text-to-shape method SDFusion [12] within
each bounding box, according to the textual information in the graph nodes.

DiffuScene [51]. DiffuScene is a diffusion-based retrieval method, which can
be both unconditional and text-conditional. It uses a diffusion process to
generate bounding boxes of an object set as the scene layout. We test its
text-conditional version to perform scene synthesis on the SG-FRONT dataset.
Specifically, we transfer scene graph description to sentences based on the
script provided by DiffuScene. Then, we feed the sentences to the BERT [23]
encoder to have textual embeddings for conditioning the denoising process. The
experimental settings are kept the same as the original ones. DiffuScene is
explicitly designed for scene completion tasks, where it uses partial inputs
as a basis and generates additional content. Our task, however, concentrates
on achieving fully controllable scene synthesis. This means we aim to produce
scenes that precisely match the descriptions provided in the scene graph.
Consequently, the evaluation of DiffuScene focuses solely on the fidelity of
the generated scenes, assessing how closely the distribution of the generated
content aligns with the original data. We do not enforce strict adherence to
the graph constraints, recognizing that a significant portion of the content
is creatively inferred or ‘hallucinated.’

InstructScene [30]. InstructScene is another retrieval method with a closer
setting to us. It stacks two diffusion-based stages: Firstly, it transfers
sentences using a graph transformer denoiser to a semantic graph containing
shape prior as nodes and allocates each two nodes a single edge. Secondly,
another graph transformer denoiser serves as a 3D layout decoder, taking the
graph and steadily denoising the 3D bounding boxes as the scene layouts. Our
task focuses on graph-conditioned scene synthesis. Therefore, we conduct the
experiment solely in the second stage of InstructScene. We train the graph
transformer denoiser with scene graphs in SG-FRONT until its convergence. As
the synthesized scenes are fully controllable by the scene graph, we are able
to evaluate both scene fidelity and the performance of graph constraints. When
synthesis the scenes, we retrieve the objects whose sizes are closest to the
ones of generated bounding boxes. This retrieval strategy is kept the same for
all methods for comparison fairness.

###  0.E.2 Implementation Details.

Trainval and test splits. We use the settings in DiffuScene [51] and
CommonScenes [57] to train and test all methods on SG-FRONT and 3D-FRONT. The
whole dataset contains 4,041 bedrooms, 900 dining rooms, and 813 living rooms.
The training split contains 3,879 bedrooms, 614 dining rooms, and 544 living
rooms, with the rest as the test split.

Batch size definition. The two branches use individual batches in terms of the
different training objectives. The layout branch uses a scene batch $B_{s}$
during one training step, containing all bounding boxes in scenes. There are
two ways to determine the batch size for the shape branch.

First, if it is the ablated version where the shape branch is without shape
echoes, we can straightforwardly sample $B_{o}$ objects (nodes) out of the
scene batch to train, as illustrated in CommonScenes [57]. This method allows
for random sampling of objects since the lack of shape echoes means there’s no
requirement for the objects to originate from the same scene.

Second, if it is the full version, we set up a maximum batch size $B_{o}^{*}$,
and select scenes where the total number of objects $B_{o}$ closely approaches
but does not exceed $B_{o}^{*}$. This method ensures that we efficiently
utilize the batch capacity while adhering to the constraint of keeping the sum
of objects within the predetermined maximum batch size. In this case, the
batch size in shape branch $B_{o}$ slightly fluctuates, as the object numbers
are not fixed in the scene.

Training procedure. We train _EchoLayout_ (layout branch) for 2000 epochs with
$B_{s}=64$. The learning rate is set to [1e-4, 5e-5, 1e-5, 5e-6] at [0,
35,000, 70,000, 140,000] step. For the full _EchoScene_ , which integrates
shape information via a shape branch into the pipeline—where both branches
utilize a shared latent graph representation—we extend the training by an
additional 50 epochs. The maximum batch size $B_{o}^{*}$ for the shape branch
is set to 64. The learning rate is kept in the same fashion.