# ISS: Image as Stepping Stone
for Text-Guided 3D Shape Generation

Zhengzhe Liu1 Peng Dai2 Ruihui Li3 Xiaojuan Qi2∗ Chi-Wing Fu1∗
1The Chinese University of Hong Kong 2The University of Hong Kong 3Hunan
University

###### Abstract

Text-guided 3D shape generation remains challenging due to the absence of
large paired text-shape dataset, the substantial semantic gap between these
two modalities, and the structural complexity of 3D shapes. This paper
presents a new framework called Image as Stepping Stone (ISS) for the task by
introducing 2D image as a stepping stone to connect the two modalities and to
eliminate the need for paired text-shape data. Our key contribution is a two-
stage feature-space-alignment approach that maps CLIP features to shapes by
harnessing a pre-trained single-view reconstruction (SVR) model with multi-
view supervisions: first map the CLIP image feature to the detail-rich shape
space in the SVR model, then map the CLIP text feature to the shape space and
optimize the mapping by encouraging CLIP consistency between the input text
and the rendered images. Further, we formulate a text-guided shape stylization
module to dress up the output shapes with novel structures and textures.
Beyond existing works on 3D shape generation from text, our new approach is
general for creating shapes in a broad range of categories, without requiring
paired text-shape data. Experimental results manifest that our approach
outperforms the state-of-the-arts and our baselines in terms of fidelity and
consistency with text. Further, our approach can stylize the generated shapes
with both realistic and fantasy structures and textures. Codes are available
at https://github.com/liuzhengzhe/ISS-Image-as-Stepping-Stone-for-Text-
Guided-3D-Shape-Generation.

Figure 1: Our novel “Image as Stepping Stone” framework (a) is able to connect
the text space (the CLIP Text feature) and the 3D shape space (the SVR
feature) through our two-stage feature-space alignment, such that we can
generate plausible 3D shapes from text (b) beyond the capabilities of the
existing works (CLIP-Forge and Dream Fields), without requiring paired text-
shape data.

##  1 Introduction

3D shape generation has a broad range of applications, e.g., in Metaverse,
CAD, games, animations, etc. Among various ways to generate 3D shapes, a user-
friendly approach is to generate shapes from natural language or text
descriptions. By this means, users can readily create shapes, e.g., to
add/modify objects in VR/AR worlds, to design shapes for 3D printing, etc.
Yet, generating shapes from texts is very challenging, due to the lack of
large-scale paired text-shape data, the large semantic gap between the text
and shape modalities, and the structural complexity of 3D shapes.

Existing works (Chen et al., 2018; Jahan et al., 2021; Liu et al., 2022)
typically rely on paired text-shape data for model training. Yet, collecting
3D shapes is already very challenging on its own, let alone the tedious manual
annotations needed to construct the text-shape pairs. To our best knowledge,
the largest existing paired text-shape dataset (Chen et al., 2018) contains
only two categories, i.e., table and chair, thus severely limiting the
applicability of the existing works.

Very recently, two annotation-free approaches, CLIP-Forge (Sanghi et al.,
2022) and Dream Fields (Jain et al., 2022), were proposed to address the
dataset limitation. These two state-of-the-art approaches attempt to utilize
the joint text-image embedding from the large-scale pre-trained language
vision model, i.e., CLIP (Radford et al., 2021), to eliminate the need of
requiring paired text-shape data in model training. However, it is still
extremely challenging to generate 3D shapes from text without paired texts and
shapes for the following reasons. First, the range of object categories that
can be generated are still limited due to the scarcity of 3D datasets. For
example, Clip-Forge (Sanghi et al., 2022) is built upon a shape auto-encoder;
it can hardly generate plausible shapes beyond the ShapeNet categories. Also,
it is challenging to learn 3D prior of the desired shape from texts. For
instance, Dream Field (Jain et al., 2022) cannot generate 3D shapes like our
approach due to the lack of 3D prior, as it is trained to produce only multi-
view images with a neural radiance field. Further, with over an hour of
optimization for each shape instance from scratch, there is still no guarantee
that the multi-view consistency constraint of Dream Field (Jain et al., 2022)
can enforce the model for producing shapes that match the given text; we will
provide further investigation in our experiments. Last, the visual quality of
the generated shapes is far from satisfactory due to the substantial semantic
gap between the unpaired texts and shapes. As shown in Figure 1 (b), the
results generated by Dream Field typically look surrealistic (rather than
real), due to insufficient information extracted from text for the shape
structures and details. On the other hand, CLIP-Forge (Sanghi et al., 2022) is
highly restricted by the limited $64^{3}$ resolution and it lacks colors and
textures, further manifesting the difficulty of generating 3D shapes from
unpaired text-shape data.

Going beyond the existing works, we propose a new approach to 3D shape
generation from text without needing paired text-shape data. Our key idea is
to implicitly leverage 2D image as a stepping stone (ISS) to connect the text
and shape modalities. Specifically, we employ the joint text-image embedding
in CLIP and train a CLIP2Shape mapper to map the CLIP image features to a pre-
trained detail-rich 3D shape space with multi-view supervisions; see Figure 1
(a): stage 1. Thanks to the joint text-image embedding from CLIP, our trained
mapper is able to connect the CLIP text features with the shape space for
text-guided 3D shape generation. Yet, due to the gap between the CLIP text and
CLIP image features, the mapped text feature may not align well with the
destination shape feature; see the empirical analysis in Section 3.2. Hence,
we further fine-tune the mapper specific to each text input by encouraging
CLIP consistency between the rendered images and the input text to enhance the
consistency between the input text and the generated shape; see Figure 1 (a):
stage 2.

Our new approach advances the frontier of 3D shape generation from text in the
following aspects. First, by taking image as a stepping stone, we make the
challenging text-guided 3D shape generation task more approachable and cast it
as a single-view reconstruction (SVR) task. Having said that, we learn 3D
shape priors from the adopted SVR model directly in the feature space. Second,
benefiting from the learned 3D priors from the SVR model and the joint text-
image embeddings, our approach can produce 3D shapes in only 85 seconds vs. 72
minutes of Dream Fields (Jain et al., 2022). More importantly, our approach is
able to produce plausible 3D shapes, not multi-view images, beyond the
generation capabilities of the state-of-the-art approaches; see Figure 1 (b).

With our two-stage feature-space alignment, we already can generate shapes
with good fidelity from texts. To further enrich the generated shapes with
vivid textures and structures beyond the generative space of the pre-trained
SVR model, we additionally design a text-guided stylization module to generate
novel textures and shapes by encouraging consistency between the rendered
images and the text description of the target style. We then can effectively
fuse with the two-stage feature-space alignment to enable the generation of
both realistic and fantasy textures and also shapes beyond the generation
capability of the SVR model; see Figure 1 (b) for examples. Furthermore, our
approach is compatible with different SVR models (Niemeyer et al., 2020;
Alwala et al., 2022). For example, we can adopt SS3D (Alwala et al., 2022) to
generate shapes from single-view in-the-wild images to broaden the range of
categorical 3D shapes that our approach can generate, going beyond Sanghi et
al. (2022), which can only generate 13 categories of ShapeNet. Besides, our
approach can also work with the very recent approach GET3D (Gao et al., 2022)
to generate high-quality 3D shapes from text; see our results in Section 10.

##  2 Related Works

##### Text-guided image generation.

Existing text-guided image generation approaches can be roughly cast into two
branches: (i) direct image synthesis (Reed et al., 2016a; b; Zhang et al.,
2017; 2018; Xu et al., 2018; Li et al., 2019; 2020; Qiao et al., 2019; Wang et
al., 2021) and (ii) image generation with a pre-trained GAN (Stap et al.,
2020; Yuan & Peng, 2019; Souza et al., 2020; Wang et al., 2020; Rombach et
al., 2020; Patashnik et al., 2021; Xia et al., 2021). Yet, the above works can
only generate images for limited categories. To address this issue, some
recent works explore zero-shot text-guided image generation (Ramesh et al.,
2021; Ding et al., 2021; Nichol et al., 2021; Liu et al., 2021; Ramesh et al.,
2022) to learn to produce images of any category. Recently, Zhou et al. (2022)
and Wang et al. (2022b) leverage CLIP for text-free text-to-image generation.
Text-guided shape generation is more challenging compared with text-to-image
generation. First, it is far more labor-intensive and difficult to prepare a
large amount of paired text-shape data than paired text-image data, which can
be collected from the Internet on a large scale. Second, the text-to-shape
task requires one to predict full 3D structures that should be plausible
geometrically and consistently in all views, beyond the needs in single-view
image generation. Third, 3D shapes may exhibit more complex spatial structures
and topology, beyond regular grid-based 2D images.

##### Text-guided 3D generation.

To generate shapes from text, several works (Chen et al., 2018; Jahan et al.,
2021; Liu et al., 2022) rely on paired text-shape data for training. To avoid
paired text-shape data, two very recent works, CLIP-Forge (Sanghi et al.,
2022) and Dream Fields (Jain et al., 2022), attempt to leverage the large-
scale pre-trained vision-language model CLIP. Yet, they still suffer from
various limitations, as discussed in the third paragraph of Section 1. Besides
3D shape generation, some recent works utilize CLIP to manipulate a shape or
NeRF with text (Michel et al., 2022; Jetchev, 2021; Wang et al., 2022a) and to
generate 3D avatars (Hong et al., 2022). In this work, we present a new
framework for generating 3D shape from text without paired text-shape data by
our novel two-stage feature-space alignment. Our experimental results
demonstrate the superiority of this work beyond the existing ones in terms of
fidelity and text-shape consistency.

##### Single-view reconstruction.

Another topic related to this work is single-view reconstruction (SVR).
Recently, researchers have explored SVR with meshes (Agarwal & Gopi, 2020),
voxels (Zubić & Liò, 2021), and 3D shapes (Niemeyer et al., 2020). Further, to
extend SVR to in-the-wild categories, Alwala et al. (2022) propose SS3D to
learn 3D shape reconstruction using single-view images in hundreds of
categories. In our work, we propose to harness an SVR model to map images to
shapes, such that we can take 2D image as a stepping stone for producing
shapes from texts. Yet, we perform the mapping and feature alignment
implicitly in the latent space rather than explicitly.

##  3 Methodology

Figure 2: Overview of our text-guided 3D shape generation framework, which has
three major stages. (a) Leveraging a pre-trained SVR model, in stage-1
feature-space alignment, we train the CLIP2Shape mapper $M$ to map the CLIP
image feature space $\Omega_{\text{I}}$ to shape space $\Omega_{\text{S}}$ of
the SVR model with $E_{\text{S}}$, $E_{\text{I}}$ frozen, and fine-tune
decoder $D$ with an additional background loss $L_{\text{bg}}$. $M$ and $D$
are trained with their own losses separately at the same time by stopping the
gradients from SVR loss $L_{D}$ and background loss $L_{bg}$ to propagate to
$M$. (b) In stage-2 feature-space alignment, we fix $D$ and fine-tune $M$ into
$M^{\prime}$ by encouraging CLIP consistency between input text $T$ and the
rendered images at test time. (c) Last, we optimize the style of the generated
shape and texture of $S$ for $T$. At the inference, we use stage 2 to generate
3D shape from $T$ and (c) is optional.

###  3.1 Overview

This work aims to generate 3D shape $S$ from text $T$. Overall, our idea is to
map the CLIP features to the shape space of a pre-trained SVR model, such that
we can leverage the joint text-image embeddings from CLIP and also the 3D
generation capability of the SVR model to enhance the generation of 3D shape
from text. Hence, our method only needs to be trained with multi-view RGB or
RGBD images and the associated camera poses without paired text-shape data. As
Figure 2 shows, our framework includes (i) image encoder $E_{\text{S}}$, which
maps input image $I$ to SVR shape space $\Omega_{\text{S}}$, (ii) pre-trained
CLIP text and image encoders $E_{\text{T}}$ and $E_{\text{I}}$, which map text
$T$ and image $I$ to CLIP spaces $\Omega_{\text{T}}$ and $\Omega_{\text{I}}$,
respectively, (iii) mapper $M$ with 12 fully-connected layers, each followed
by a Leaky-ReLU, and (iv) decoder $D$ to generate the final shape $S$.
Specifically, we use DVR (Niemeyer et al., 2020) as the SVR model when
presenting our method, unless otherwise specified.

Overall, we introduce a novel two-stage feature-space-alignment approach to
connect the text, image, and shape modalities. In detail, we first train
CLIP2Shape mapper $M$ to connect the CLIP image space $\Omega_{\text{I}}$ and
the shape space $\Omega_{\text{S}}$ from the pre-trained SVR model (see Figure
2 (a)). Then, we fine-tune $M$ at test time using a CLIP consistency loss
$L_{c}$ to further connect the CLIP text space $\Omega_{\text{T}}$ with
$\Omega_{\text{S}}$ (see Figure 2 (b)). Last, we may further optimize the
texture and structure style of $S$ by fine-tuning the decoders (see Figure 2
(c)).

In the following, we first introduce two empirical studies on the CLIP feature
space in Section 3.2, then present our two-stage feature-space-alignment
approach in Section 3.3. Further, Section 9 presents our text-guided shape
stylization method and Section 3.5 discusses the compatibility of our approach
with different SVR models and our extension to generate a broad range of
categories.

###  3.2 Empirical Studies and Motivations

Existing works (Sanghi et al., 2022; Zhou et al., 2022; Wang et al., 2022b)
mostly utilize CLIP directly without analyzing how it works and discussing its
limitations. To start, we investigate the merits and drawbacks of leveraging
CLIP for text-guided 3D shape generation by conducting the following two
empirical studies to gain more insight into the CLIP feature space.

Figure 3: Empirical studies on the CLIP feature space for text-guided 3D shape
generation.

####  3.2.1  Whether CLIP feature space sufficiently good for 3D shape
generation?

First, we study the representative capability of the CLIP image feature space
$\Omega_{\text{I}}$ by trying to generate 3D shapes from this space.
Specifically, we replace the SVR image encoder $E_{\text{S}}$ with the CLIP
image encoder $E_{\text{I}}$, and optimize implicit decoder $D$ using multi-
view losses like DVR (Niemeyer et al., 2020) with $E_{\text{I}}$ frozen. This
approach can be extended to text-to-shape generation by replacing
$E_{\text{I}}$ with CLIP text encoder $E_{\text{T}}$ during the inference. To
compare the performance of $E_{\text{S}}$ and $E_{\text{I}}$, we evaluate 3D
mIoU between their generated shapes and GTs. The results are as follows: the
standard SVR pipeline $E_{\text{S}}$+$D$ achieves 47.86% mIoU while replacing
the SVR encoder $E_{\text{S}}$ with CLIP encoder $E_{\text{I}}$
($E_{\text{I}}$+$D$) degrades the performance to 40.09%. From the results and
qualitative comparison shown in Figures 3 (a, b), we can see that the CLIP
image space $\Omega_{\text{I}}$ has inferior representative capability to
capture details of the input image for 3D shape generation. This is not
surprising, since the pre-trained $E_{\text{I}}$ from CLIP is targeted to
extract semantic-aligned features from texts rather than extracting details
from images. Hence, image details relevant to 3D reconstruction are lost,
e.g., textures. On the contrary, $E_{\text{S}}$ from the SVR model is
optimized for 3D generation from images, so it maintains more necessary
details. The above result motivates us to design a mapper $M$ from
$\Omega_{\text{I}}$ to $\Omega_{\text{S}}$ and then generate shapes from
$\Omega_{\text{S}}$ instead of $\Omega_{\text{I}}$ for better generative
fidelity.

####  3.2.2 How the CLIP image and text feature gap influences 3D shape
generation?

Second, we investigate the gap between the normalized CLIP image feature
$f_{\text{I}}\in\Omega_{\text{I}}$ and normalized CLIP text feature
$f_{\text{T}}\in\Omega_{\text{T}}$; (see also the CLIP image and text feature
spaces in Figure 1 (a)) and how such gap influences text-guided 3D shape
generation. Specifically, we randomly sample $300$ text-shape pairs from the
text-shape dataset (Chen et al., 2018), then evaluate the cosine distance
between $f_{\text{I}}$ and $f_{\text{T}}$, i.e.,
$d=1-\text{cosine\\_similarity}(f_{\text{I}},f_{\text{T}})$, where
$f_{\text{I}}$ is the CLIP feature of the rendered images from the
corresponding shape. We repeat the experiment and obtain
$d(f_{\text{T}},f_{\text{I}})=0.783\pm 0.004$. The result reveals _a certain
gap between the CLIP text and image features in this dataset, even though they
are paired._ Also, the angle in the feature space between the two features is
around $\arccos(1-0.783)=1.35$ rad in this dataset (Chen et al., 2018). Having
said that, directly replacing $f_{\text{I}}$ with $f_{\text{T}}$ like Sanghi
et al. (2022); Zhou et al. (2022) in inference may harm the consistency
between the output shape and the input text. As demonstrated in Figure 3 (c),
directly replacing $f_{\text{I}}$ with $f_{\text{T}}$ causes a cosine distance
of 0.45 to $f_{\text{S}}\in\Omega_{\text{S}}$ (see Figure 3 (c)), which is
significantly larger than the distance between $M(f_{\text{I}})$ and
$f_{\text{S}}$ (0.21). Our finding is consistent with the findings in Liang et
al. (2022). It motivates us to further fine-tune $M$ into $M^{\prime}$ at test
time, such that we can produce feature $M^{\prime}(f_{\text{T}})$, which is
closer to $f_{\text{S}}$ than $M(f_{\text{T}})$.

###  3.3 Two-Stage Feature-Space Alignment

Following the above findings, we propose a two-stage feature-space-alignment
approach to first connect image space $\Omega_{\text{I}}$ and shape space
$\Omega_{\text{S}}$ and further connect text space $\Omega_{\text{T}}$ to
shape space $\Omega_{\text{S}}$ with the image space $\Omega_{\text{I}}$ as
the stepping stone.

##### Stage-1 alignment: CLIP image-to-shape mapping.

Given multi-view RGB or RGBD images for training, the stage-1 alignment is
illustrated in Figure 2 (a). Considering that shape space $\Omega_{\text{S}}$
contains richer object details than the image space $\Omega_{\text{I}}$, while
$\Omega_{\text{I}}$ provides a joint text-image embedding with the input text
space $\Omega_{\text{T}}$, we introduce a fully-connected CLIP2Shape mapper
$M$ to map image feature $f_{\text{I}}$ to shape space $\Omega_{\text{S}}$.
Taking a rendered image $I$ as input, $M$ is optimized with an $L_{2}$
regression between $M(f_{\text{I}})$ and standard SVR feature
$f_{\text{{S}}}=E_{\text{S}}(I)$ according to Equation (1) below:

$\displaystyle
L_{M}=\sum^{N}_{i=1}||M(f_{\text{I,i}})-E_{\text{S}}(I_{i})||^{2}_{2}$ (1)

where $N$ is the number of images in the training set and $f_{\text{I},i}$ is
the normalized CLIP feature of $I_{i}$.

Also, we fine-tune decoder $D$ to encourage it to predict a white background,
which helps the model to ignore the background and extract object-centric
feature (see Figure 4), while maintaining its 3D shape generation capability.
To this end, we propose a new background loss $L_{\text{{bg}}}$ in Equation
(2) below to enhance the model’s foreground object awareness to prepare for
the second-stage alignment.

$\displaystyle
L_{\text{bg}}=\sum_{p}||D_{c}(p)-1||_{2}^{2}\mathbbm{1}(\text{ray}(o,p)\cap
F=\emptyset)$ (2)

where $\mathbbm{1}$ is the indicator function; $F=\\{p:D_{o}(p)>t\\}$
indicates the foreground region, in which the occupancy prediction $D_{o}(p)$
is larger than threshold $t$; $p$ is a query point; $\text{ray}(o,p)\cap
F=\emptyset$ means the ray from camera center $o$ through $p$ does not
intersect the foreground object marked by $F$; and $D_{c}(p)$ is the color
prediction at query point $p$. In a word, $L_{\text{bg}}$ encourages $D$ to
predict the background region as white color (value 1), such that
$E_{\text{I}}$ can focus on and better capture the foreground object. In
addition, to preserve the 3D shape generation capability of $D$, we follow the
loss $L_{\text{D}}$, with which $D$ has been optimized in the SVR training. In
this work, we adopt Niemeyer et al. (2020).

Hence, the overall loss in stage-1 training is $\lambda_{M}L_{M}$ for mapper
$M$ and $\lambda_{\text{bg}}L_{\text{bg}}+L_{D}$ for decoder $D$, where
$\lambda_{M}$ and $\lambda_{\text{bg}}$ are weights. The stage-1 alignment
provides a good initialization for the test-time optimization of stage 2.

##### Stage-2 alignment: text-to-shape optimization.

Given a piece of text, the stage-2 alignment aims to further connect the text
and shape modalities. Specifically, it searches for shape $S$ that best
matches the input text $T$. To this end, we formulate a fast test-time
optimization to reduce the gap between the text and image CLIP features
$f_{\text{T}}$ and $f_{\text{I}}$, as discussed earlier in the second
empirical study.

As shown in Figure 2 (b), given input text $T$, we replace image encoder
$E_{\text{I}}$ with text encoder $E_{\text{T}}$ to extract CLIP text feature
$f_{\text{T}}$, then fine-tune $M$ with CLIP consistency loss between input
text $T$ and $m$ images $\\{R_{i}\\}_{i=1}^{m}$ rendered with random camera
poses from output shape $S$; see Equation 3:

$\displaystyle
L_{\text{C}}=\sum_{i=1}^{m}{\langle{f_{\text{T}}}\cdot\frac{E_{\text{I}}(R_{i})}{||E_{\text{I}}(R_{i})||}\rangle}$
(3)

where $\langle\cdot\rangle$ indicates the inner-product.

Figure 4: Effect of generating shapes from the same text with/without
background loss $L_{\text{bg}}$.

In stage-2 alignment, we still adopt $L_{\text{bg}}$ to enhance the model’s
foreground awareness. Comparing Figures 4 (a) and (b), we can see that the
stage-2 alignment is able to find a rough shape with $L_{\text{{bg}}}$ in
around five iterations, yet failing to produce a reasonable output without
$L_{\text{{bg}}}$, since having the same color prediction on both foreground
and background hinders the object awareness of the model.

Thanks to the joint text-image embedding of CLIP, the gap between text feature
$f_{\text{T}}$ and shape feature $f_{\text{S}}$ has already been largely
narrowed by $M$. Therefore, the stage-2 alignment only needs to fine-tune $M$
with 20 iterations using the input text, taking only around 85 seconds on a
single GeForce RTX 3090 Ti, compared with 72 minutes taken by Dream Fields
(Jain et al., 2022) at test time. After this fine-tuning, we can readily
obtain a plausible result; see, e.g., the “result” shown in Figure 4 (b). Our
ISS is a novel and efficient approach for 3D shape generation from text.

##### Diversified generation.

In general, shape generation from text is one-to-many. Hence, we further
extend our approach with diversified 3D shape generation from the same piece
of input text. Unlike the existing works, which require additional and complex
modules, e.g., GANs (Chen et al., 2018), IMLE (Liu et al., 2022), and
normalizing flow network (Sanghi et al., 2022), we can simply perturb the
image and text features for diversified generation. Specifically, after
stage-1 alignment, we randomly permute $f_{\text{I}}$ as an initialization and
$f_{\text{T}}$ as the ground truth by adding normalized Gaussian noises
$z_{1}=h_{1}/||h_{1}||,z_{2}=h_{2}/||h_{2}||$, where $h_{1},h_{2}\sim N(0,1)$
to derive diversified features

$\hat{f}_{\text{I}}=\tau_{1}f_{\text{I}}+(1-\tau_{1})z_{1}\ \ \text{and}\ \
\hat{f}_{\text{T}}=\tau_{2}f_{\text{T}}+(1-\tau_{2})z_{2},$ (4)

where $\tau_{1},\tau_{2}$ are hyperparameters to control the degrees of
permutation. With permuted $\hat{f}_{\text{I}}$ and $\hat{f}_{\text{T}}$ in
stage-2 alignment, our model can converge to different 3D shapes for different
noise.

###  3.4 Text-Guided Stylization

The two-stage feature-space alignment is already able to generate plausible 3D
shapes; see, e.g., Figures 2 (b) and 4 (b). However, the generative space is
limited by the representation capability of the employed SVR model, e.g., DVR
(Niemeyer et al., 2020) can only generate shapes with limited synthetic
patterns as those in ShapeNet. However, a richer and wider range of structures
and textures are highly desired. To this end, we equip our model with a text-
guided stylization module to enhance the generated shapes with novel structure
and texture appearances, as shown in Figures 2 (c) and 1.

Specifically, for texture stylization, we first duplicate $D$ (except for the
output layer) to be $D_{o}$ and $D_{c}$, then put the output occupancy
prediction layer and color prediction layer on top of $D_{o}$ and $D_{c}$,
respectively. Further, we fine-tune $D_{c}$ with the same CLIP consistency
loss as in Equation (3), encouraging the consistency between input text $T$
and the $m$ rendered images $\\{R_{i}\\}_{i=1}^{m}$.

Besides textures, novel structures are also desirable for shape stylization.
Hence, we further incorporate a shape-and-texture stylization strategy to
create novel structures. To enable shape sculpting, we fine-tune $D$ with the
same CLIP consistency loss in Equation 3. At the same time, to maintain the
overall structure of the initial shape $S$, we propose a 3D prior loss $L_{P}$
shown in Equation (5), aiming at preserving the 3D shape prior learned by the
two-stage feature-space alignment.

$L_{\text{P}}=\sum_{p}|D_{o}(p)-D_{o}^{\prime}(p)|$ (5)

where $p$ is the query point, and $D_{o}$, $D_{o}^{\prime}$ are the occupancy
predictions of the initial $D$ and the fine-tuned $D$ in the stylization
process, respectively. To improve the consistency between the generated
texture and the generated shape, we augment the background color of $R_{i}$
with a random RGB value in each iteration. Please find more details in the
supplementary material.

###  3.5 Compatibility with Different SVR models

Besides DVR (Niemeyer et al., 2020), our ISS framework is compatible with
different SVR models. For example, we can adapt it with the most recent SVR
approach SS3D (Alwala et al., 2022) that leverages in-the-wild single images
for 3D generation. With this model, our framework can generate a wider range
of shape categories by using SS3D’s encoder and decoder as shape encoder
$E_{\text{S}}$ and decoder $D$ in our framework, respectively. Here, we simply
follow the same pipeline as in Figure 2 to derive a text-guided shape
generation model for the in-the-wild categories; see our results in Section
4.4. Notably, we follow the losses in Alwala et al. (2022) in place of
$L_{\text{D}}$ (see Section 3.3) in stage-1 training, requiring only single-
view images without camera poses. More importantly, our approach’s high
compatibility suggests that it is orthogonal to SVR, so its performance can
potentially be further upgraded with more advanced SVR approaches in the
future.

##  4 Experiments

###  4.1 Datasets, Implementation Details, and Metrics

With multi-view RGB or RGBD images and camera poses, we can train ISS on the
synthetic dataset ShapeNet (Chang et al., 2015) (13 categories) and the real-
world dataset CO3D (Reizenstein et al., 2021) (50 categories). To evaluate our
generative performance, we create a text description set with four texts per
category on ShapeNet and two texts per category on CO3D. SS3D (Alwala et al.,
2022) takes single-view in-the-wild images in training; as their data has not
been released, we only evaluate our method on some of their categories. To
evaluate the performance, we employ Fréchet Inception Distance (FID) (Heusel
et al., 2017), Fréchet Point Distance (FPD) (Liu et al., 2022) to measure
shape generation quality, and conduct a human perceptual evaluation to further
assess text-shape consistency. Please refer to the supplementary material for
more details on the metrics and implementation details.

###  4.2 Comparisons with State-of-the-art Methods

We compare our approach with existing works (Sanghi et al., 2022; Jain et al.,
2022) both qualitatively and quantitatively. For a fair comparison, we use
their official codes on GitHub to generate shapes on our text set. Table 1
shows quantitative comparisons, whereas Figure 5 shows the qualitative
comparisons. Comparing existing works and ours in Table 1, we can see that our
approach outperforms two state-of-the-art works by a large margin for both
generative quality and text-shape consistency scores. On the other hand, the
qualitative comparisons in Figure 5 (a,b) show that CLIP-Forge (Sanghi et al.,
2022) produces low-resolution shapes without texture, and some generated
shapes are inconsistent with the input text, e.g., “a wooden boat.” Dream
Fields (Jain et al., 2022) cannot generate reasonable shapes from the input
text on the top row and its generated shape on the bottom row is also
inconsistent with the associated input text. On the contrary, our approach
(Figure 5 (i)) can generate high-fidelity shapes that better match the input
text. Note that we only utilize two-stage feature-space alignment without
stylization in producing our results. Please refer to the supplementary file
for more results and visual comparisons.

###  4.3 Ablation Studies

To manifest the effectiveness of our approach, we conduct ablation studies on
the following baselines (see Table 1 and Figure 5): generate shapes from
$\Omega_{I}$ ($E_{\text{I}}+D$), optimize stage-2 alignment without stage-1
(w/o stage 1), conduct stage-1 alignment without stage-2 (w/o stage 2),
disable the background loss in stage 1, stage 2 and both (w/o
$L_{\text{bg\\_1}}$, w/o $L_{\text{bg\\_2}}$, w/o $L_{\text{bg}}$), and two
additional baselines that first create images and then 3D shapes (GLIDE+DVR,
LAFITE+DVR). More details on the setup and analysis are provided in the
supplementary material Section 3. Our results outperform all the existing
works and baselines in terms of fidelity and text-shape consistency by a large
margin.

###  4.4 More Analysis on Generative Results of ISS

Next, we present evaluations on the generative novelty and diversity, as well
as the scalability of our two-stage feature-space alignment. Then, we show
more text-guided stylization results and how our ISS approach generalizes to a
wide range of categories and generates shapes with better fidelity.

Table 1: Quantitative comparisons with existing works and baselines.

Method Type | Method | FID ($\downarrow$) | Consistency Score (%) ($\uparrow$) | FPD ($\downarrow$) | A/B/C Test
---|---|---|---|---|---
Existing works | CLIP-Forge | 162.87 | 41.83 $\pm$ 17.62 | 37.43 | 8.90 $\pm$ 4.12
Dream Fields | 181.25 | 25.38 $\pm$ 12.33 | N.A. | N.A.
Ablation Studies |  $E_{\text{I}}$+$D$ | 181.88 | 20.97 $\pm$ 13.59 | 38.61 | N.A.
w/o Stage 1 | 222.96 | 1.92 $\pm$ 2.22 | 79.41 | N.A.
w/o Stage 2 | 202.33 | 29.52 $\pm$ 14.86 | 41.71 | N.A.
w/o $L_{\text{bg\\_1}}$ | 149.45 | 29.45 $\pm$ 14.67 | 40.85 | N.A.
w/o $L_{\text{bg\\_2}}$ | 156.52 | 31.55 $\pm$ 8.87 | 38.31 | N.A.
w/o $L_{\text{bg}}$ | 178.34 | 30.96 $\pm$ 15.49 | 40.98 | N.A.
Text2Image+SVR | GLIDE+DVR | 212.41 | 8.85 $\pm$ 7.94 | 41.33 | N.A.
LAFITE+DVR | 135.01 | 52.12 $\pm$ 11.05 | 37.55 | 11.70 $\pm$ 4.11
Ours | ISS | 124.42 $\pm$ 5.11 | 60.0 $\pm$ 10.94 | 35.67 $\pm$ 1.09 | 21.70 $\pm$ 5.19

Figure 5: Qualitative comparisons with existing works and baselines.  Figure
6: Our approach is able to generate novel shapes, not in the training set. (a)
shows our results and (b,c,d) are the top-three shapes retrieved from the
training set.  Figure 7: Our approach can generate diversified results from
the same input text. Figure 8: Results on CO3D dataset. We show two different
views of each result. Figure 9: Text-guided stylization. Left: texture
stylization. Right: shape-and-texture stylization. Figure 10: With single
images for training (without camera poses), our approach can produce results
for a broad range of categories, by adopting (Alwala et al., 2022). Two
different views are rendered.  Figure 11: Results of ISS built upon the SVR
model of IM-Net (left two) and GET3D (right four).

##### Generation novelty.

Our approach is able to generate novel shapes beyond simple retrieval from the
training data. As shown in Figure 6, from the input text, we first generate
our result in (a) and then take our generated shape to retrieve the top-three
closest shapes in the associated training set based on the cosine similarity
between the CLIP features of the rendered images $f_{\text{I}}$ and input text
$f_{\text{T}}$ as the retrieval metric. Our results with the two-stage
feature-space alignment are not in the training sets, showing that our ISS
approach can generate novel shapes beyond the training set, even without
stylization. It is not surprising, as our approach shares the generative space
with the pre-trained SVR model and can potentially generate all shapes that
the pre-trained SVR model can produce.

##### Generation diversity.

By perturbing the features and injecting randomness on initialization, ISS is
able to generate diversified results from the same text input. As shown in
Figure 7, ISS produces various cars and chairs from the same piece of text.
Quantitative results are presented in the supplementary material Section 3.4
where our method achieves a high PS (Point Score).

##### Generation fidelity.

To assess the capability of ISS in generating realistic real-world 3D shapes,
we train the SVR model on the CO3D dataset (Reizenstein et al., 2021) which is
a real-world dataset, and ISS leverages the learned feature space for text to
shape generation without paired data. As shown in Figure 8, our model is able
to generate real-world shapes. As far as know, this is first work that
investigates text-guided shape generation and on real-world datasets can
generate realistic 3D shapes.

##### Generation beyond the capability of SVR model.

Our text-guided stylization module equips our model with the capability to
generate 3D shapes beyond the SVR model. As shown in Figure 9 and Figure 2
(c), our model is able to create realistic and fantasy novel structures and
textures that match text descriptions. Please refer to the supplementary
material Section 4 for more details.

##### Generality and Scalability of ISS on other SVR models.

Our model is generic and can work together with other SVR models. To evaluate
the generality and scalablity of our model, we employ SS3D (Alwala et al.,
2022), IM-Net (Chen & Zhang, 2019), and GET3D (Gao et al., 2022) as SVR models
to provide the feature space. It is worth noting that SS3D is capable of
generating shapes of more categories and IM-Net, GET3D can produce high
fidelity results. First, as shown in Figure 10, built upon SS3D, our approach
can generate shapes of more real-world categories, e.g., bird. Note that our
model can generate shapes with comparable or even better qualities compared
with initial SS3D model that takes an image as input. Second, when combined
with IM-Net and GET3D, our model can fully exploit their generative
capabilities and produces high-quality 3D shapes as shown in Figure 11. The
above manifests that ISS is generic and compatible to advanced models for
generating shapes of more categories and higher qualities.

##  5 Conclusion

In this paper, we present a novel approach for text-guided 3D shape generation
by leveraging the image modality as a stepping stone. Leveraging the joint
text-image embeddings from CLIP and 3D shape priors from a pre-trained SVR
model, our approach eliminates the need for the paired text and shape data.
Technically, we have the following contributions. First, we step-by-step
reduce the semantic gap among the text, image and shape modalities through our
two-stage feature-space alignment approach. Second, our text-guided
stylization technique effectively enriches our generated shapes with novel
structures and textures in various styles. Third, our approach is compatible
with various single-view reconstruction approaches and can be further extended
to generate a wide range of categories with only single images without camera
poses in training. Experiments on ShapeNet, CO3D, and multiple single-image
categories manifest the superiority of our framework over the two state-of-
the-art methods and various baselines. Limitations are discussed in the
supplementary files.

## ACKNOWLEDGEMENTS

The work has been supported in part by the Research Grants Council of the Hong
Kong Special Administrative Region (Project no. CUHK 14206320), General
Research Fund of Hong Kong (No. 17202422), Hong Kong Research Grant Council -
Early Career Scheme (Grant No. 27209621), and National Natural Science
Foundation of China (No. 62202151).

Supplementary Material

In this supplementary material, we first introduce the background augmentation
in text-guided shape stylization (Section 6). Next, we present the
implementation details and evaluation metrics, and provide the details of
human perceptual evaluation results (Section 7). Then, we introduce the setup
and analysis of our ablation studies (Section 8). After that, we discuss text-
guided stylization and provide more results (Section 9). Then we show
additional generative results of our approach (Section 10). Further, we
discuss two alternative training strategies (Section 11). Then, we show how
feature is mapped in the latent space (Section 12) and also review some
related literatures about SVR and differentiable rendering (Section 13).
Afterwards we compare the number of parameters in our model and existing works
(Section 14). We also discuss failure cases (Section 15) and limitations of
this work (Section 16). Finally, we provide our text sets (Section 17) and
summarize the notations in the paper (Section 18).

##  6 Background Augmentation in Text-Guided Shape Stylization

One important thing in text-guided shape stylization is that the generated
texture should align with the given shape. However, it cannot be ensured with
a simple white or black background during training since the generated
textures can be affected by the background color. As shown in Figure 12 (a),
the shape in white color may confuse with the white background; thus, the
model would struggle to capture the boundary of objects, hence cannot generate
textures that well-align with the table. In Figure 12 (c), the generated
texture is severely affected by the black background color, causing low-
quality stylization results.

To address the above issue, we propose a background augmentation strategy to
improve the alignment between texture and shape. Specifically, the background
color is replaced with random RGB values in each training iteration. By this
means, the foreground shape may be more easily captured in training as shown
in Figure 12 (b,d), improving the texture-shape consistency and the
stylization quality.

##### Discussion on $L_{\text{bg}}$ and background augmentation.

In two-stage feature space alignment (Section 3.3 in the main paper), we
introduce a background loss $L_{\text{bg}}$ to encourage the color prediction
on the background region to be white. A natural question is whether we can use
background augmentation as a replacement, and our answer is no. As shown in
Figure 13 (a), the background color can affect the cosine similarity of CLIP
features between the image and input text; thus, using different background
color in each iteration makes stage-2 alignment unstable and affects shape
generation. As a result, the two-stage alignment can only benefit from
$L_{bg}$, but not from the background augmentation, for producing a plausible
shape. Besides, we empirically found that the two-stage feature space
alignment with $L_{\text{bg}}$ performs well, even if a white shape is being
considered, see the bottom row in Figure 16 (i).

##  7 Implementation Details, Metrics, and Human Perceptual Evaluation Details

##### Implementation details.

Our framework is implemented using PyTorch (Paszke et al., 2019). We first
train the stage-1 CLIP-image-to-shape mapping for 400 epochs with learning
rate $1e^{-4}$, and then train the stage-2 text-to-shape module at test time
for 20 iterations which takes only $85$ seconds on average on a single GeForce
RTX 3090 Ti. Optionally, we can further train text-guided stylization with the
same learning rate. We empirically set hyperparameters $\lambda_{M}$,
$\lambda_{bg}$, $t$, $m$, $\tau_{1}$, $\tau_{2}$ to be 0.5, 10, 0.5, 10, 0.2,
0.95, respectively, according to a small validation set.

##### Details on camera poses.

In Stage 1, we follow Niemeyer et al. (2020) to set the camera poses to
encourage the background to be white. Specifically, we randomly sample the
distance of the camera and the viewpoint on the northern hemisphere.

In Stage 2, compared with Niemeyer et al. (2020), we sample the camera
distance to be 1.5 times further compared with Niemeyer et al. (2020). It
helps to encourage sampling more global views instead of only local ones, so
that the CLIP image encoder can capture the whole shape and yield a better
CLIP feature.

In Stage 3, we also sample the camera distance to be 1.5 times. Since this
stage aims to generate textures instead of searching for a target shape like
Stages 1 and 2, only sampling view points on the northern hemisphere of the
view space cannot ensure good generation quality in the bottom regions. Thus,
we further randomly sample viewpoints on the southern hemisphere for random
10% training iterations to encourage the stylized results to be consistent
with the text in various viewpoints.

##### Decoder duplication

Except for the output layer, we simply duplicate them to be $D_{o}$ and
$D_{c}$. For the output layer, it takes $d$-dimensional features as input and
outputs one value for occupancy and three values for RGB. We then copy
$d\times 1$ weights to $D_{o}$ and copy $d\times 3$ weights to $D_{c}$. See
Figure 15 for more details.

##### Metric: Shape generation quality.

To measure the shape generation quality, we employ Fréchet Inception Distance
(FID) (Heusel et al., 2017) between five rendered images of the generated
shape with different camera poses and a set of ground-truth ShapeNet or CO3D
images. We adopt the official model with Inception Net trained on ImageNet,
which is widely used to evaluate generative quality and realism. We do not
train a model on ShapeNet, since it is too small to train a better network for
evaluating the FID than models trained on ImageNet. In addition, we randomly
sample 2600 images in the ShapeNet dataset as ground truths for FID
evaluation, instead of using images from ImageNet. It helps to measure the
similarity between the generated shapes and ground truths of ShapeNet.

Besides adopting FID, we further utilize the metric Fréchet Point Distance
(FPD) proposed in (Liu et al., 2022) to measure the shape generation quality
without texture. We first convert the generated shapes to 3D point clouds
without color (see Figure 14) and then evaluate FPD. Note that Dream Field
(Jain et al., 2022) does not produce 3D shapes directly, so that we cannot
evaluate this work in this regard.

To further assess the text-shape consistency, we conduct a human perceptual
evaluation which is detailed as follows.

##### Human perceptual evaluation Setup.

First, we prepare generated results for human evaluation. For each input text,
we produce nine results from state-of-the-art methods, including CLIP-Forge
(Sanghi et al., 2022) and Dream Fields (Jain et al., 2022), six baseline
methods, and our full method; see Section 4.2 in the main paper and Section
8.1 in this supplementary material for details of each baseline. Second, we
invite 10 volunteers (3 females and 7 males; aged from 19 to 58; all with
normal vision) to evaluate the results. We show these results to the
participants in random order without revealing how each result is produced.
Then, they are asked to give a score from $\\{1,0.5,0\\}$ (1: perfect match,
0.5: partial match; and 0: don’t match) on the degree of match between the
generated shapes and input text. Then, for each method, we gather the
evaluation scores from all participants and obtain the “Consistency Score” as
$s/n$, where $s$ is the total score and $n$ is the number of samples.

Figure 12: Effective of text-guided shape stylization with/without background
augmentation.  Figure 13: An investigation on background loss and background
augmentation. (a) Background color affects the cosine similarity of the CLIP
features between the image and the text “a red car”, i.e., (i) 0.292 (ii)
0.303 and (iii) 0.285. (b) Effect of generating shapes with background
augmentation, but without background loss $L_{\text{bg}}$. Comparing with
Figure 4 (b) in the main paper, the two-stage feature space alignment works
well with $L_{\text{bg}}$, but fails with the background augmentation.  Figure
14: Visualization of point clouds of different methods for FPD evaluation.
Figure 15: Visualization to duplicate the decoder $D$ to $D_{o}$, $D_{c}$.

##### Details of human perceptual evaluation results.

We show Consistency Score and the preferences in the A/B/C test from each
volunteer in Tables 2 and 3. As shown in Table 2, all ten volunteers
consistently give the highest Consistency Score to our approach, and in the
A/B/C test (see Table 3), all ten volunteers prefer results from our approach.
The above further manifest the superiority of our model.

Table 2: Total scores $s$ from the ten volunteers out of 52 generated shapes.

Method | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | mean $\pm$ std | Consistency Score (%) $\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---|---
CLIP-Forge | 21 | 35 | 34 | 16.5 | 18.5 | 29 | 21.5 | 4 | 21 | 17 | 21.75 $\pm$ 9.16 | 41.83 $\pm$ 17.62
Dream Fields | 13 | 17.5 | 7 | 19 | 6.5 | 22 | 22 | 9 | 10 | 6 | 13.2 $\pm$ 6.41 | 25.38 $\pm$ 12.33
$E_{\text{I}}$+$D$ | 14 | 9.5 | 5 | 18 | 4.5 | 25 | 15.5 | 4.05 | 4.5 | 9 | 10.91 $\pm$ 7.06 | 20.97 $\pm$ 13.59
w/o stage-1 | 1.5 | 1 | 0 | 1.5 | 0 | 3.5 | 2 | 0 | 0 | 0.5 | 1.00 $\pm$ 1.15 | 1.92 $\pm$ 2.22
w/o stage-2 | 20 | 14.5 | 8.5 | 23.5 | 7.5 | 27 | 19 | 8.5 | 4.5 | 20.5 | 15.35 $\pm$ 7.73 | 29.52 $\pm$ 14.86
w/o $L_{\text{bg}}$ | 17.5 | 15.5 | 8.5 | 21 | 8 | 27 | 26.5 | 9.5 | 5 | 22.5 | 16.1 $\pm$ 8.06 | 30.96 $\pm$ 15.49
GLIDE+DVR | 5 | 3.5 | 1.5 | 10 | 2 | 13 | 6.5 | 0.5 | 1 | 3 | 4.60 $\pm$ 4.13 | 8.85 $\pm$ 7.94
LAFITE+DVR | 23 | 33.5 | 31 | 23 | 27 | 31.5 | 27.5 | 14.5 | 32.5 | 27.5 | 27.10 $\pm$ 5.75 | 52.12 $\pm$ 11.05
ISS (ours) | 32 | 37 | 35.5 | 29.5 | 29 | 37 | 29 | 17.5 | 32.5 | 33 |  31.20 $\pm$ 5.69 |  60.00 $\pm$ 10.94

Table 3: A/B/C Test results of the ten volunteers. The numbers in the table
indicate the number of shapes from the corresponding method he/she likes most
out of the three candidates. Volunteers can optionally select “pass” instead
of “A/B/C” if he cannot decide which one is the best.

Category | Method | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | mean $\pm$ std $\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---|---
Existing works | CLIP-Forge | 9 | 17 | 12 | 13 | 6 | 9 | 3 | 6 | 6 | 8 | 8.9 $\pm$ 4.12
SOTA Text2Image+SVR | LAFITE+DVR | 9 | 16 | 9 | 12 | 7 | 13 | 9 | 20 | 8 | 14 | 11.7 $\pm$ 4.11
Ours | ISS | 27 | 19 | 21 | 20 | 17 | 25 | 19 | 26 | 13 | 30 |  21.7 $\pm$ 5.19

##  8 Ablation Studies

###  8.1 Baseline setups

We create the following baselines in our ablation study. The first six
baselines aim to assess the effectiveness of key modules in our approach,
whereas the last two adopt state-of-the-art text-to-image generation
approaches to first create images then adopt DVR (Niemeyer et al., 2020) to
generate shapes for a fair comparison with our approach. Note that we do not
adopt the most recent SVR model SS3D (Alwala et al., 2022) (which aims to work
with in-the-wild images), due to its inferior generative quality and lack of
texture generation.

  * •

$E_{\text{I}}+D$: As the first empirical study in Section 3.2 in the main
paper, $E_{\text{I}}$ is adopted to extract the image feature $f_{\text{I}}$
and $D$ is trained for 3D shape generation from $f_{\text{I}}$ without the
two-stage feature-space alignment.

  * •

w/o Stage 1: Stage-2 alignment is optimized from randomly initialized $M$,
without stage 1.

  * •

w/o Stage 2: Generate with $M$ after stage 1, without the test-time
optimization of stage 2.

  * •

w/o $L_{\text{bg\\_1}}$: Remove $L_{\text{bg}}$ in stage-1 alignment.

  * •

w/o $L_{\text{bg\\_2}}$: Remove $L_{\text{bg}}$ in stage-2 alignment.

  * •

w/o $L_{\text{bg}}$: Remove $L_{\text{bg}}$ in both stage-1 and stage-2
alignment.

  * •

GLIDE+DVR: Use a recent zero-shot text-guided image generation approach GLIDE
(Nichol et al., 2021) to generate image $I$ from $T$, then use DVR (Niemeyer
et al., 2020) to generate $S$ from $I$.

  * •

LAFITE+DVR: Train a recent text-guided image generation approach LAFITE (Zhou
et al., 2022) with ShapeNet images, then generate image $I$ from $T$. Further
generate $S$ from $I$ with DVR (Niemeyer et al., 2020).

###  8.2 Quantitative and qualitative comparisons

In this section, we analyze the results of the above baselines one by one.

  * •

$E_{\text{I}}+D$: column (c) shows that the results generated from CLIP space
$\Omega_{\text{I}}$ have inferior fidelity in terms of the texture (top row in
Figure 16) and shape structure (bottom two rows in Figure 16) due to the
limited capability of $E_{\text{I}}$ to capture details of the image.

  * •

w/o Stage 1: column (d) of Figure 16 indicates that without stage-1 alignment,
the generated shapes are almost the same whatever text $T$ is adopted as
input, since $M$ tends to map text feature $f_{\text{T}}$ to almost the same
feature even though stage-2 alignment is enabled. It shows that a good
initialization provided by stage-1 alignment is necessary for the test-time
optimization of stage 2.

  * •

w/o Stage 2: as shown in column (e) of Figure 16, without Stage 2, may not
align well with $f_{S}$ due to the semantic gap between $f_{\text{I}}$ and
$f_{\text{T}}$. Now, we use Figure 17 (a) to illustrate their associated
results: the model in “w/o stage 2” can generate reasonable shapes from a
single image (see “SVR” in Figure 17 (a)) but fails with text as input (see
“stage 1” in Figure 17 (a)); further with the stage-2 optimization, a
plausible phone can be generated (see “stage 2 (ours)” in Figure 17 (a)). w/o
$L_{\text{bg\\_1}}$, w/o $L_{\text{bg\\_2}}$, w/o $L_{\text{bg}}$: stage-2
alignment cannot work properly without $L_{\text{bg}}$ in either stage-1 or
stage-2 alignment or both (see column (f, g, h) of Figure 16) due to the lack
of foreground awareness. Even though stage-1 alignment has encouraged the
background to be white, we still need this loss in stage 2 to get satisfying
results.

  * •

GLIDE+DVR: the images created by GLIDE (Nichol et al., 2021) have a large
domain gap from the training data of DVR (see Figure 17 (b)), severely
limiting the performance of GLIDE+DVR (see Figure 16 (i)).

  * •

LAFITE+DVR: as shown in Figure 16 (j), some generative results of LAFITE+DVR
can be coarse (first row of the main paper and the last row of Figure 16 in
this supplementary material) due to the error accumulation of the isolated two
steps, i.e., LAFITE (see Figure 17 (b) “image from LAFITE”) and DVR, and some
do not match the input text due to the semantic gap between $f_{\text{I}}$ and
$f_{\text{T}}$ (two bottom rows of Figure 5 in the main paper and the last row
of Figure 16 in this supplementary material). Despite the above, subsequently
generating images then shapes is still a strong baseline that is a valuable
research direction in the future.

  * •

ours (ISS): column (k) of Figure 5 in the main paper and Figure 16 in this
supplementary material show that our approach can generate shapes and textures
with good text-shape consistency (see “small wings” in the top row, “water
craft” in the middle row of Figure 16) and fidelity, beyond all the above
baselines and the existing works CLIP-Forge (Sanghi et al., 2022) and Dream
Field (Jain et al., 2022).

Figure 16: Additional qualitative results compared with existing works and
baselines.  Figure 17: A further investigation on baselines “w/o stage 2”,
“GLIDE+DVR”, and “LAFITE+DVR”. (a) “w/o stage 2” produces a plausible shape
(“SVR”) from image but a low-quality shape (“stage 1”) from text; further
fine-tuning using stage-2 alignment enables us to produce a more plausible
shape from text (“stage 2 (ours)”). (b) GLIDE / LAFITE generate out-of-domain
and inferior-quality images, limiting the performance of subsequent 3D
generation.

###  8.3 A/B/C test

To further compare our approach with the strongest baselines CLIP-Forge
(Sanghi et al., 2022) and “LAFITE+DVR”, we perform an A/B/C test with 10
volunteers to compare these two baselines with ours. Specifically, the results
from the three approaches (per input text) in random order for all the $52$
texts. Then, they were instructed to choose a most preferred one. The results
in Table 1 “A/B/C Test” in the main paper show that our results are more
preferred than others, outperforming (Sanghi et al., 2022) by 143.8%
($(21.70-8.90)/8.90$) and “LAFITE+DVR” by 85.5% ($(21.70-11.70)/11.70$).

###  8.4 Diversified Generation

In addition, we evaluate the diversified generation results discussed in
Section 3.3 in the main paper. Specifically, we generate additional two
samples per input text, and then adopt FID (Heusel et al., 2017), FPD (Liu et
al., 2022) (the lower, the better) for the fidelity evaluation and Point Score
(PS) (Liu et al., 2022) (the higher, the better) for the diversity evaluation.
The results are: FID: 113.98, FPD: 35.37, PS: 3.11, which is even better than
our one-text-one-shape generative results (FID: 124.42 $\pm$ 5.11, FPD: 35.67
$\pm$ 1.0, PS: 3.18 $\pm$ 0.11), manifesting the superior performance of
diversified generation capability of ISS.

##  9 Discussions on Text-Guided Shape Stylization

##### Additional stylization results.

As shown in Figure 18 (a), our text-guided shape stylization is able to
generate vivid landscape and flower textures on the sofa shape. Note that the
sofa is generated from the text “A sofa with black backrest” (see Figure 20
top right) with totally different initial color from the stylization results.
Further, in Figure 18 (b), we show four additional texture stylization results
in addition to the Figure 9 in the main paper. In addition, as shown in Figure
18 (c), our shape-and-texture stylization is able to generate novel shapes and
textures beyond the dataset, and create imaginary shapes with diversified
structures. Note that our results achieve a good balance on the stylization
and the functionality. For example, our result of “avocado chair” possesses
both the style of “avocado” and the functionality of “chair”.

As shown in Figure 19, shape-and-texture stylization consistently produces the
cars that is consistent with the input text “a blue car”, with proper
variation in terms of the color and shape. In addition, the degree of the
shape variations can be controlled by the loss weight $\lambda_{P}$ of the 3D
prior loss $L_{p}$.

Figure 18: Additional stylization results.  Figure 19: Results of shape-and-
texture stylization with the same text “A blue car” and different 3D prior
loss $L_{P}$.

##### Why does shape-and-texture stylization need an initial shape?

Shape-and-texture stylization is initialized by the two-stage feature-space
alignment result, since it provides the 3D prior.

##### Relationship of texture stylization and shape-and-texture stylization.

Texture stylization and shape-and-texture stylization have their own merits.
Texture stylization keeps the shape unchanged and is able to guarantee the
functionality of the shape. In addition, it can take some abstract text
descriptions as input like “sunset” in Figure 18 (a). Shape-and-texture
stylization is able to generate novel and imaginary structures beyond the
training dataset. However, there is a tradeoff between the stylization and the
functionality.

##  10 Additional Results

##### ShapeNet.

As shown in Figure 20, our approach can generate view-consistent 3D shapes on
ShapeNet (Chang et al., 2015) that well match the input texts.

Figure 20: Additional generative results on the ShapeNet (Chang et al., 2015)
dataset.

##### CO3D.

Further, we show more text-guided shape generation results on the CO3D
(Reizenstein et al., 2021) dataset in Figure 21. These results again manifest
the capability of our approach on real-world 3D shape generation, beyond the
existing works (Chen et al., 2018; Sanghi et al., 2022; Liu et al., 2022) that
focus only on the synthetic shape generation on ShapeNet (Chang et al., 2015).

Figure 21: Additional generative results on the CO3D (Reizenstein et al.,
2021) dataset.

##### Working with GET3D

When working with GET3D (Gao et al., 2022), our approach can generate 3D
shapes of good fidelity from texts, as shown in Figure 22 in this
supplementary file.

Figure 22: Additional generative results built upon GET3D (Gao et al., 2022).

##### Single-image categories.

In Figure 23, we present more text-guided generations for more categories
using single images in training without camera pose, built upon Alwala et al.
(2022). The results further demonstrate the compatibility of our approach to
various SVR approaches, particularly generating plausible 3D shapes from text
with single images in training.

Figure 23: Additional generative results built upon SS3D (Alwala et al., 2022)
using single images in training without camera pose.

##### More generative results.

Further, we present more generative results of our approach in Figure 24.
Using ISS, we are able to effectively generate a wide variety of 3D shapes
from texts.

Figure 24: Generative results of ISS. Using our new approach, we are able to
effectively produce 3D shapes for a wide variety of categories from texts.
Figure 25: Results of optimizing the decoder, instead of the two-stage
feature-space alignment.  Figure 26: Results of optimizing the feature in
shape space, instead of the stage-2 feature-space alignment.

##  11 Discussions on Two Alternative Training Strategies

##### Fine-tune decoder.

In this section, we discuss an alternative training strategy to optimize the
decoder directly with the CLIP consistency loss, instead of using two-stage
feature space alignment.

First, we provide the results of the above training strategy. It produces
unsatisfactory 3D shapes such as the one shown in Figure 25 (c) with more than
20 minutes of training. Besides, Dream Field (Jain et al., 2022) also utilizes
the same idea to optimize the decoder directly; yet, the generated shape (see
Figure 25 (d)) is also far from satisfactory even though Dream Fields is
trained only to produce multi-view images with an NeRF-like architecture,
unlike ours that is capable for 3D shape generation.

Then we try to analyze why the above strategy fails to produce desired shapes.
Leveraging the pre-trained decoder that has already incorporated the 3D shape
prior, our approach is able to search for the desired shape in the shape
feature space $\Omega_{S}$ efficiently. On the contrary, fine-tuning the
decoder in stage 2 without the explicit 2D/3D supervision will destroy the
pre-trained shape feature space $\Omega_{S}$, which is used to introduce 3D
priors. Specifically, in stage 2 (“alignment stage”), the model is trained at
test time with the user-provided text without any explicit 2D/3D supervision,
so it is hard for the model to gain the knowledge about what the desired 3D
shape should be like. In other words, it is extremely challenging for the
model to learn the 3D shape prior with only CLIP consistency supervision from
text.

Further, we analyze why the decoder can be fine-tuned in shape-and-texture
stylization. First, note that 3D shape generation is a very different task
from shape-and-texture stylization, since shape-and-texture stylization is
initialized by the two-stage feature-space alignment result, which has already
learned the 3D prior. Therefore, we can fine-tune the decoder in shape-and-
texture stylization.

##### Update the feature in the shape space.

In this section, we discuss another alternative training strategy to optimize
the feature in shape space $\Omega_{S}$, instead of optimizing the mapper $M$
with stage-2 feature space alignment.

However, the shape feature has only 256 dimensions, which is far smaller than
the number of weights in the mapping network. Hence, directly optimizing the
shape feature in $\Omega_{S}$ is not capable for generating the desired 3D
shape as shown in Figure 26, and the stage-2 feature space alignment is
necessary in our approach.

##  12 Analysis on Feature Space Mapping

To provide more insights on explaining how the latent space is mapped, we
measure the distance of features at different stages on all the samples in our
test set on ShapeNet in Table 4. The notations follow Figure 3 (c) of the main
paper, $M$ means the mapper, and $d$ means cosine distance. Our ultimate goal
is to obtain a text mapper $M^{\prime}$ (Figure 2 in the main paper) to map
the text feature space $f_{T}$ to shape feature space $f_{S}$ using image with
features $f_{I}$ as a stepping stone to gradually narrow their distances using
two stage mapping. Note that the image $f_{I}$ and text features $f_{T}$ are
obtained using pre-trained CLIP models.

In the stage-1 alignment process, we train a mapper $M$ to map image features
$f_{I}$ to a space $M(f_{I})$ close to the shape space $f_{S}$ using image
data and the regression loss $L_{M}$. Note that the text feature $f_{T}$ and
image feature $f_{I}$ are all from the CLIP model in a shared embedding space.
It’s natural that the trained mapper can be used to map the text feature
$f_{T}$ to $M(f_{T})$, making text features closer to the shape space.
However, when measuring the distance among $M(f_{T})$, $M(f_{S})$, we find the
average distance among all samples is $d(M(f_{I}),M(f_{T}))=0.58\pm 0.23$, the
average distance between $M(f_{I})$ and $f_{S}$ is $0.21\pm 0.10$ , and the
average distance between shape and text is $d(M(f_{T}),f_{S}))=0.45\pm 0.20$.
This implies that there is a gap between CLIP image and text feature after the
first step mapper and further motivates our stage-2 alignment. Note that there
is no GT shape for our task on 3D generation, so we manually select a shape in
the ShapeNet dataset that matches the input text as the GT.

In the stage-2 alignment, $M$ is further updated and the final delivered
mapper is called $M^{\prime}$, which is to further narrow down the gap between
mapped text features and shape features. The average distance between the
mapped text feature $M\textquoteright(f_{T})$ and shape feature space $f_{S}$:
$d(M\textquoteright(f_{T}),f_{S}))=0.17\pm 0.08$ which is much smaller than
the corresponding distance after stage-1 alignment. It shows that stage 2
alignment can significantly reduce the difference between the mapped text and
the GT shape feature from 0.45 to 0.17 on average.

Table 4: Distance changes in the feature space mapping of all the 52 samples
in the test set. $d$ means cosine distance. Almost all distances are
consistently reduced after our stage-2 alignment.

Text | $d(M(f_{I}),M(f_{T}))$ | $d(M(f_{I}),f_{S}))$ | $d(M(f_{T}),f_{S}))$ | $d(M\textquoteright(f_{T}),f_{S}))$ | $d(M(f_{T}),M\textquoteright(f_{T}))$
---|---|---|---|---|---
a glass single leg circular table | 0.63 | 0.31 | 0.58 | 0.14 | 0.34
a wooden double layers table | 0.72 | 0.10 | 0.64 | 0.14 | 0.65
a square metal table | 0.76 | 0.32 | 0.44 | 0.21 | 0.30
a round shaped single legged wooden table | 0.47 | 0.24 | 0.21 | 0.21 | 0.30
this is a bar stool with metal arches as a design feature | 0.43 | 0.33 | 0.35 | 0.18 | 0.17
a children chair with little legs | 0.79 | 0.20 | 0.43 | 0.12 | 0.35
a swivel chair with wheels | 0.61 | 0.30 | 0.38 | 0.22 | 0.11
a red recliner seems confortable | 0.33 | 0.20 | 0.23 | 0.24 | 0.05
a red car | 1.02 | 0.19 | 0.78 | 0.10 | 0.69
a green SUV | 0.49 | 0.12 | 0.24 | 0.19 | 0.25
a large black truck | 0.74 | 0.19 | 0.50 | 0.15 | 0.53
a long luxury black car | 0.67 | 0.22 | 0.54 | 0.09 | 0.54
army fighter jet | 0.43 | 0.54 | 0.43 | 0.25 | 0.28
a black airplane with long white wings | 0.62 | 0.12 | 0.32 | 0.16 | 0.10
a blue airplane with short wings | 0.51 | 0.14 | 0.55 | 0.33 | 0.40
boeing 747 | 0.35 | 0.13 | 0.23 | 0.06 | 0.20
a big ship for transportation | 0.95 | 0.17 | 0.72 | 0.35 | 0.21
a boat with sail | 0.39 | 0.19 | 0.46 | 0.12 | 0.45
a watercraft | 0.12 | 0.14 | 0.27 | 0.22 | 0.06
a wooden boat | 0.76 | 0.29 | 0.52 | 0.12 | 0.49
a blue sofa | 0.46 | 0.20 | 0.34 | 0.12 | 0.27
sofa with legs | 0.67 | 0.13 | 0.55 | 0.10 | 0.35
a sofa with black backrest | 0.59 | 0.14 | 0.33 | 0.08 | 0.16
a small sofa | 0.49 | 0.20 | 0.33 | 0.10 | 0.28
a long brown bench | 0.57 | 0.29 | 0.34 | 0.18 | 0.27
a marble bench | 0.61 | 0.16 | 0.46 | 0.12 | 0.29
a metal bench | 0.18 | 0.29 | 0.20 | 0.20 | 0.09
concrete bench | 0.48 | 0.07 | 0.46 | 0.11 | 0.36
a military sniper rifle with long barrel | 0.47 | 0.14 | 0.29 | 0.07 | 0.21
a rifle with magazines | 0.71 | 0.48 | 0.51 | 0.17 | 0.43
a short rifle | 0.75 | 0.15 | 0.81 | 0.35 | 0.52
rifle shotgun | 0.38 | 0.13 | 0.25 | 0.10 | 0.09
a computer monitor | 0.28 | 0.07 | 0.22 | 0.07 | 0.20
a display | 0.22 | 0.18 | 0.15 | 0.13 | 0.02
a monitor with square base | 0.56 | 0.17 | 0.75 | 0.33 | 0.34
a TV monitor | 0.70 | 0.16 | 0.41 | 0.11 | 0.27
a cabinet with cylindrical legs | 0.53 | 0.24 | 0.27 | 0.18 | 0.11
a cupboard | 0.54 | 0.37 | 0.42 | 0.19 | 0.45
a long brown cabinet | 0.57 | 0.29 | 0.34 | 0.18 | 0.27
a wardrobe | 0.41 | 0.13 | 0.32 | 0.16 | 0.26
a desk lamp | 0.92 | 0.35 | 0.51 | 0.16 | 0.45
bedside lamp | 0.71 | 0.48 | 0.55 | 0.17 | 0.43
lamp supported by a long pillar | 0.19 | 0.12 | 0.18 | 0.05 | 0.14
mushroom-like lamp | 1.20 | 0.17 | 1.06 | 0.32 | 0.46
a mobile phone | 0.75 | 0.09 | 0.85 | 0.28 | 0.55
a small cell phone | 0.49 | 0.20 | 0.33 | 0.10 | 0.28
a mobile phone with black screen | 0.94 | 0.39 | 0.61 | 0.14 | 0.68
an iphone | 0.75 | 0.25 | 0.54 | 0.22 | 0.57
a columnar loudspeaker | 0.66 | 0.25 | 0.54 | 0.15 | 0.40
a loudspeaker with metal surface | 0.34 | 0.17 | 0.44 | 0.16 | 0.30
a wooden loudspeaker | 0.72 | 0.10 | 0.64 | 0.14 | 0.65
a cylindrical loudspeaker | 1.06 | 0.19 | 0.82 | 0.31 | 0.27
mean $\pm$ std | 0.58 $\pm$ 0.234 | 0.21 $\pm$ 0.10 | 0.45 $\pm$ 0.20 | 0.17 $\pm$ 0.08 | 0.32 $\pm$ 0.17

##  13 Related Works on Single-View Reconstruction and Differentiable
Rendering

##### Single-view reconstruction (SVR)

This task aims to reconstruct the 3D shape from a single-view image of it.
Recently, many approaches have been proposed for meshes (Agarwal & Gopi,
2020), voxels (Zubić & Liò, 2021), and 3D shapes (Niemeyer et al., 2020).
Recently, to extend SVR to in-the-wild categories, Alwala et al. (2022)
proposed a new approach called SS3D to learn 3D shape reconstruction using
single-view images for the reconstruction of hundreds of categories.

As 3D-to-2D projections, single-view 2D images are more closely related to
shapes than texts, since they reveal many attributes of 3D shapes, e.g.,
structure, details, appearance, etc. The strong correlation between 2D images
and 3D shapes motivates us to reduce the challenging text-to-shape generation
task to text-to-image and then SVR, by connecting the CLIP features from text
with the shape features in SVR using images as an intermediate step to
gradually bridge the gap between text and shape. Specifically, we extend the
pre-trained SVR model to be compatible with text input, transforming the
challenging text-to-shape task into SVR. Our framework can work with different
SVR approaches to extend them for 3D shape generation from texts. So, our
approach is orthogonal to the SVR approaches.

##### Differentiable Rendering

3D rendering is an important topic in computer vision and graphics. It takes a
3D scene as input and predicts the 2D view of it given a camera pose. Beyond
3D rendering, differentiable rendering further aims to derive the
differentiations of the rendering function. With differentiable rendering, a
renderer can be integrated into an optimization framework, thus a 3D shape can
be reconstructed from multi-view 2D images. Neural Volume Rendering
(Mildenhall et al., 2020) and its following works (Jain et al., 2021; Barron
et al., 2021) aim to synthesize novel view images of a 3D scene. Besides,
recent works (Niemeyer et al., 2020; Munkberg et al., 2022; Gao et al., 2022)
leverage differentiable rendering for 3D shape generation using 2D images. In
this work, we derive 2D images of the generated 3D shape using differentiable
rendering and use a pre-trained large-scale image-language model CLIP to
encourage the 2D images to be consistent with the input text. Thanks to
differentiable rendering, we can update the generated 3D shape indirectly
using the rendered images.

##  14 Analysis on the Number of Parameters

Our model incorporates only a small number of learnable parameters for shape
generation in $M$. Note that the parameters of the SVR model are not tuned for
the specific text-to-shape generation task. Therefore, they are not counted in
the total number of parameters.

We provide the comparison of learnable model parameters responsible for shape
generation and compare it with existing methods in Table 5.

Table 5: Number of parameters and performance of existing works and our
method.

Method | Number of parameters (M) | FID | FPD | Consistency Score
---|---|---|---|---
CLIP-Forge (Sanghi et al., 2022) | Normalized flow network: 18.37 | 162.87 | 37.43 | 41.83±17.62
Dream Fields (Jain et al., 2022) | 0.61 | 181.25 | N.A. | 25.38±12.33
Ours | Mapper: 2.43 | 124.42±5.11 | 35.67±1.09 | 60.0±10.94
Ours with a lightweight mapper | Mapper: 0.46 | 129.01 | 34.26 | 67.21±10.64

The number of learnable parameters of CLIP-forge (Sanghi et al., 2022) is 8
times larger than ours. Note that we do not include the parameters of the
CLIP-Forge auto-encoder for a fair comparison. With much fewer parameters, our
model outperforms CLIP-Forge in all evaluated metrics (see Table 5). This
demonstrates that our performance gain is not purely from the learnable
parameters.

To better compare with Dream Fields (Jain et al., 2022) (0.61 M parameters),
we design a lightweight mapper to match the total number of parameters.
Specifically, the mapper is composed of three fully connected layers with 512,
256, and 256 output dimensions, yielding a total of 0.46M parameters which is
smaller than Dream Field. With this new mapper, we still outperform Dream
Fields in terms of metrics. Please see Figure 27 for quantitative comparison.
This further demonstrates our major performance gain is not from the learnable
parameters.

From the results in Table 27, we can consistently observe that our model can
achieve much better performance with much fewer parameters, manifesting the
efficiency of our proposed image as a stepping-stone pipeline that allows us
to leverage the 3D priors in pre-trained SVR models to enable text to 3D shape
synthesis without requiring paired text and 3D data.

Note that we exclude the number of parameters of the decoder $D$ in our
comparison because the only effect of fine-tuning $D$ is to make the
background white and does not contribute to the generative capability of our
model. As shown in Figure 2 of the main paper, mapper $M$ and decoder $D$ are
trained with their own losses separately at the same time by stopping the
gradients from $L_{D}$ and $L_{bg}$ to propagate to $M$.

Also, to show that fine-tuning $D$ does not improve its generative capability,
we feed the same input feature $f_{s}$ to $D$ before and after fine-tuning,
they generate almost the exact same 3D shape as the original SVR model, as
shown in Figure 28 in the supplementary material. And in stage 2, $D$ is not
optimized. Admittedly, our generative capability benefits from the SVR model
including $D$. However, $D$ is not optimized for our text-to-shape generation
task. Therefore, we exclude the number of parameters of decoder $D$ for fair
comparison.

Figure 27: (a) Our generative result with lightweight mapper vs. (b) Dream
Fields (Jain et al., 2022).  Figure 28: The generative results of the original
SVR model and after our stage-1 feature space alignment using the same feature
$f_{S}$.

##  15 Failure Cases

Here are some examples of failure cases of our approach.

##### The complex and unusual shapes, e.g., “an oval table with 3 legs. ”

First, our model fails to generate the shape from the text “an oval table with
3 legs. ”. Our approach leverages the CLIP consistency loss in the rendered 2D
image; however, in the rendered image shown in Figure 29 (a), only three legs
can be seen and the remaining one is occluded, which confuses the model
training.

##### The given text is very long, e.g., “it is grey in color, circular in
shape with four legs and back support, material used is wood and overall
appearance looks like unique design armless chair. ”

Our model fails to generate the shape from the above long description as shown
in Figure 29 (b). Some attributes are missing including “armless”, “four
legs”. This is partially due to the limited representative capability of a
single CLIP feature for such a long sentence. We may incorporate an additional
local feature like Liu et al. (2022) to handle the long text in the future.

##### The shapes with multiple fine-grained descriptions, e.g., “A chair with
a red back and a green cushion.”

As shown in Figure 29 (c), our model fails to generate the shape “A chair with
a red back and a green cushion. ” As studied in some recent works (Yao et al.,
2022; Li et al., 2022), CLIP mainly address on the global image feature, but
has inferior capability to capture fine-grained features. Therefore, our
approach may fail to generate shapes where multiple fine-grained descriptions
are given. In the future, we may try more recent pre-trained text-and-image
embedding models to enhance the model’s capability to handle the fine-grained
descriptions.

##  16 Limitations

This work still has some limitations. First, our performance is limited by the
SVR model that our approach is built upon, e.g., some results in Figure 10 of
the main paper and Figure 23 in this supplementary material are still not very
satisfactory, because SS3D (Alwala et al., 2022) itself is struggling to
create shapes with fine details. Second, we cannot generate the categories
outside the image dataset due to the lack of 3D prior of the unseen category.
That is why our model needs images as the stepping stone to learn what the
particular category is like. However, we want to highlight the following.
First, built upon SS3D (Alwala et al., 2022), our approach can generate a wide
range of categories with single-view images in the wild as training data.
Second, with our shape-and-texture stylization, our approach can generate
imaginary and uncommon shapes outside the image dataset (in the same
category). Third, it is extremely challenging to generate arbitrary category
shapes from text. As far as we know, there is only one existing work, Dream
Field (Jain et al., 2022), that can generate more categories than ours.
However, Dream Fields only generate multi-view images instead of directly
generate 3D shapes, and it cannot generate reasonable shapes in many cases as
shown in Figure 5 in our main paper and Figure 8 in this supplementary
material.

Figure 29: Failure cases of our approach.

##  17 Text Set in the Experiments

Recent works (Chen et al., 2018; Sanghi et al., 2022; Jain et al., 2022)
proposed their own text sets. However, their datasets have some limitations
and are not suitable to evaluate our approach. The dataset of Text2shape (Chen
et al., 2018) contains text descriptions in only two categories, i.e., Table
and Chair; CLIP-Forge (Sanghi et al., 2022) lacks of descriptions on the color
and texture; while Dream Fields (Jain et al., 2022) utilizes text descriptions
containing complex scenes and actions. To fairly evaluate our approach, we
propose two text datasets on the ShapeNet (Chang et al., 2015) and CO3D
(Reizenstein et al., 2021) categories, respectively, shown in Tables 7 and 8.

##  18 List of Notations

In this section, we summarize symbols and notations used in the paper to
facilitate readers to follow up. Please refer to Table 6 for more details.

Table 6: Summary of symbols used in paper.

Notation | Description | Notation | Description
---|---|---|---
$E_{S}$ | encoder of SVR model | $E_{I}$ | CLIP image encoder
$E_{T}$ | CLIP text encoder | $D$ | Decoder of SVR model
$M$ | Mapper | $M^{\prime}$ | Mapper after stage-2 alignment
$D_{o}$ | occupancy decoder | $D_{c}$ | color decoder
$R$ | rendered images | $f_{I}$ | CLIP image feature
$f_{T}$ | CLIP text feature | $f_{S}$ | shape feature in SVR model
$o$ | camera center | $p$ | query point
$d$ | cosine distance | $L_{M}$ | regression loss in stage 1
$L_{D}$ | original loss in SVR model | $L_{bg}$ | background loss
$L_{bg\\_1}$ | background loss in stage 1 | $L_{bg\\_2}$ | background loss in stage 2
$L_{C}$ | CLIP consistency loss | $L_{P}$ | 3D prior loss
$\Omega_{T}$ | CLIP text feature space | $\Omega_{S}$ | shape feature space from the SVR model
$\Omega_{I}$ | CLIP image feature space |  |

Table 7: Texts on ShapeNet (Chang et al., 2015). They are utilized to measure
FID (Table 1 of the main paper), and employed in Human Perceptual Evaluation
(Table 2) and A/B/C Test(Table 3).

a glass single leg circular table | a wooden double layers table
---|---
a square metal table | a round shaped single legged wooden table
this is a bar stool with metal arches as a design feature | a children chair with little legs
a swivel chair with wheels | a red recliner seems comfortable
a red car | a green SUV
a large black truck | a long luxury black car
army fighter jet | a black airplane with long white wings
a blue airplane with short wings | boeing 747
a big ship for transportation | a boat with sail
a watercraft | a wooden boat
a blue sofa | sofa with legs
a sofa with black backrest | a small sofa
a long brown bench | a marble bench
a metal bench | concrete bench
a military sniper rifle with long barrel | a rifle with magazines
a short rifle | rifle shotgun
a computer monitor | a display
a monitor with square base | a TV monitor
a cabinet with cylindrical legs | a cupboard
a long brown cabinet | a wardrobe
a desk lamp | bedside lamp
lamp supported by a long pillar | mushroom-like lamp
a mobile phone | a small cell phone
a mobile phone with black screen | an iphone
a columnar loudspeaker | a loudspeaker with metal surface
a wooden loudspeaker | a cylindrical loudspeaker

Table 8: Texts on CO3D (Reizenstein et al., 2021).

A big apple | A red apple | A green bottle | A tall cylindrical bottle
---|---|---|---
A white cup | A wooden cup | A large black microwave | A white cuboid microwave
A black skateboard | A green long skateboard | A cute toytruck | A large toy truck
A blue backpack | A red big backpack | A white bowl | A big wooden bowl
A red round frisbee | A blue large frisbee | A big blue motorcycle | A black large wheels motorcycle
A circular stop sign | A triangle stop sign | Tv screen | A grey big tv screen
A basketball | A tennis ball | A large broccoli | A green broccoli
A hairdryer | A yellow hairdryer | A black mouse | A white mouse
A cuboid big suitcase | A large size tall suitcase | A round umbrella | A big black umbrella
A big banana | A long banana | A cream round cake | A chocolate mooncake
A blue handbag | A red big handbag | An orange | A large round orange
A teddybear | A cute teddybear | A blue fat vase | A blue tall vase
A black baseball bat | A long wooden baseball bat | A blue car | A red car
An egg hotdog | A sausage hotdog | A black parkingmeter | A white tall parkingmeter
A black toaster | A round toaster | Tall wineglass | Single leg big wineglass
A brown baseball glove | A black big baseball glove | A big carrot | A long carrot
A red hydrant | A yellow hydrant | A large round pizza | A tomato meat pizza
A white toilet | A fat white toilet | A stone bench | A wooden long bench
A gray iphone | A black phone | A long black keyboard | A short white keyboard
A short tree | A tall green tree | A toy bus | One decker toy bus
A blue bicycle | A black bicycle | A blue chair | A wooden chair
A red kite | A long blue kite | A TV remote | A long white remote
A book with blue cover | A black book | Brown couch | A long brown couch
A open laptop | A black laptop | An egg sandwich | A meat sandwich
A cute toy train | A short blue toy train | Chocolate donut | Big circular donut

## References

  * Agarwal & Gopi (2020) Nitin Agarwal and M Gopi.  Gamesh: Guided and augmented meshing for deep point networks.  In _3DV_ , 2020.
  * Alwala et al. (2022) Kalyan Vasudev Alwala, Abhinav Gupta, and Shubham Tulsiani.  Pre-train, self-train, distill: A simple recipe for supersizing 3D reconstruction.  _CVPR_ , 2022.
  * Barron et al. (2021) Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields.  In _ICCV_ , 2021.
  * Chang et al. (2015) Angel X. Chang, Thomas Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], 2015.
  * Chen et al. (2018) Kevin Chen, Christopher B Choy, Manolis Savva, Angel X Chang, Thomas Funkhouser, and Silvio Savarese.  Text2shape: Generating shapes from natural language by learning joint embeddings.  In _ACCV_ , 2018.
  * Chen & Zhang (2019) Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In _CVPR_ , 2019.
  * Ding et al. (2021) Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  _NeurIPS_ , 2021.
  * Gao et al. (2022) Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler.  GET3D: A generative model of high quality 3D textured shapes learned from images.  _NeurIPS_ , 2022.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs trained by a two time-scale update rule converge to a local Nash equilibrium.  _NIPS_ , 2017.
  * Hong et al. (2022) Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu.  Avatarclip: Zero-shot text-driven generation and animation of 3d avatars.  _ACM TOG (SIGGRAPH)_ , 2022.
  * Jahan et al. (2021) Tansin Jahan, Yanran Guan, and Oliver van Kaick.  Semantics-guided latent space exploration for shape generation.  In _COMPUT GRAPH FORUM_ , 2021.
  * Jain et al. (2021) Ajay Jain, Matthew Tancik, and Pieter Abbeel.  Putting nerf on a diet: Semantically consistent few-shot view synthesis.  In _ICCV_ , 2021.
  * Jain et al. (2022) Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with Drefam Fields.  In _CVPR_ , 2022.
  * Jetchev (2021) Nikolay Jetchev.  ClipMatrix: Text-controlled creation of 3D textured meshes.  _arXiv preprint arXiv:2109.12922_ , 2021.
  * Li et al. (2019) Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H. S. Torr.  Controllable text-to-image generation.  _NeurIPS_ , 2019.
  * Li et al. (2020) Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H. S. Torr.  ManiGAN: Text-guided image manipulation.  In _CVPR_ , 2020.
  * Li et al. (2022) Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang.  Fine-grained semantically aligned vision-language pre-training.  _arXiv preprint arXiv:2208.02515_ , 2022.
  * Liang et al. (2022) Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou.  Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.  _arXiv preprint arXiv:2203.02053_ , 2022.
  * Liu et al. (2021) Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu.  FuseDream: Training-free text-to-image generation with improved CLIP+ GAN space optimization.  _arXiv preprint arXiv:2112.01573_ , 2021.
  * Liu et al. (2022) Zhengzhe Liu, Yi Wang, Xiaojuan Qi, and Chi-Wing Fu.  Towards implicit text-guided 3D shape generation.  In _CVPR_ , 2022.
  * Michel et al. (2022) Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka.  Text2mesh: Text-driven neural stylization for meshes.  In _CVPR_ , 2022.
  * Mildenhall et al. (2020) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  NERF: Representing scenes as neural radiance fields for view synthesis.  In _ECCV_ , 2020.
  * Munkberg et al. (2022) Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler.  Extracting triangular 3D models, materials, and lighting from images.  In _CVPR_ , 2022.
  * Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Niemeyer et al. (2020) Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger.  Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision.  In _CVPR_ , 2020.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  PyTorch: An imperative style, high-performance deep learning library.  _NeurIPS_ , 2019.
  * Patashnik et al. (2021) Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  StyleCLIP: Text-driven manipulation of StyleGAN imagery.  _ICCV_ , 2021.
  * Qiao et al. (2019) Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao.  MirrorGAN: Learning text-to-image generation by redescription.  In _CVPR_ , 2019.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _ICML_ , 2021.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with CLIP latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Reed et al. (2016a) Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee.  Generative adversarial text to image synthesis.  In _ICML_ , 2016a.
  * Reed et al. (2016b) Scott E. Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee.  Learning what and where to draw.  _NIPS_ , 2016b.
  * Reizenstein et al. (2021) Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny.  Common objects in 3D: Large-scale learning and evaluation of real-life 3D category reconstruction.  In _ICCV_ , 2021.
  * Rombach et al. (2020) Robin Rombach, Patrick Esser, and Björn Ommer.  Network-to-network translation with conditional invertible neural networks.  _NeurIPS_ , 2020.
  * Sanghi et al. (2022) Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, and Marco Fumero.  CLIP-Forge: Towards zero-shot text-to-shape generation.  In _CVPR_ , 2022.
  * Souza et al. (2020) Douglas M. Souza, Jônatas Wehrmann, and Duncan D. Ruiz.  Efficient neural architecture for text-to-image synthesis.  In _IJCNN_ , 2020.
  * Stap et al. (2020) David Stap, Maurits Bleeker, Sarah Ibrahimi, and Maartje ter Hoeve.  Conditional image generation and manipulation for user-specified content.  _CVPRW_ , 2020.
  * Wang et al. (2022a) Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao.  CLIP-NeRF: Text-and-image driven manipulation of neural radiance fields.  In _CVPR_ , 2022a.
  * Wang et al. (2021) Hao Wang, Guosheng Lin, Steven Hoi, and Chunyan Miao.  Cycle-consistent inverse GAN for text-to-image synthesis.  _ACM MM_ , 2021.
  * Wang et al. (2022b) Zihao Wang, Wei Liu, Qian He, Xinglong Wu, and Zili Yi.  CLIP-GEN: Language-free training of a text-to-image generator with CLIP.  In _arXiv preprint arXiv:2203.00386_ , 2022b.
  * Wang et al. (2020) Zixu Wang, Zhe Quan, Zhi-Jie Wang, Xinjian Hu, and Yangyang Chen.  Text to image synthesis with bidirectional generative adversarial network.  In _ICME_ , 2020.
  * Xia et al. (2021) Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu.  TediGAN: Text-guided diverse face image generation and manipulation.  In _CVPR_ , 2021.
  * Xu et al. (2018) Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks.  In _CVPR_ , 2018.
  * Yao et al. (2022) Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: Fine-grained interactive language-image pre-training.  _ICLR_ , 2022.
  * Yuan & Peng (2019) Mingkuan Yuan and Yuxin Peng.  Bridge-GAN: Interpretable representation learning for text-to-image synthesis.  _IEEE TCSVT_ , 2019.
  * Zhang et al. (2017) Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas.  StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks.  In _ICCV_ , 2017.
  * Zhang et al. (2018) Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N. Metaxas.  StackGAN++: Realistic image synthesis with stacked generative adversarial networks.  _IEEE TPAMI_ , 2018.
  * Zhou et al. (2022) Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun.  LAFITE: Towards language-free training for text-to-image generation.  In _CVPR_ , 2022.
  * Zubić & Liò (2021) Nikola Zubić and Pietro Liò.  An effective loss function for generating 3D models from single 2D image without rendering.  _arXiv preprint arXiv:2103.03390_ , 2021.