# Farm3D: Learning Articulated 3D Animals by Distilling 2D Diffusion

Tomas Jakab Ruining Li∗ Shangzhe Wu Christian Rupprecht Andrea Vedaldi
Visual Geometry Group, University of Oxford
{tomj, ruining, szwu, chrisr<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

We present Farm3D, a method for learning category-specific 3D reconstructors
for articulated objects, relying solely on “free” virtual supervision from a
pre-trained 2D diffusion-based image generator. Recent approaches can learn a
monocular network that predicts the 3D shape, albedo, illumination, and
viewpoint of any object occurrence, given a collection of single-view images
of an object category. However, these approaches heavily rely on manually
curated clean training data, which are expensive to obtain. We propose a
framework that uses an image generator, such as Stable Diffusion, to generate
synthetic training data that are sufficiently clean and do not require further
manual curation, enabling the learning of such a reconstruction network from
scratch. Additionally, we incorporate the diffusion model as a score to
enhance the learning process. The idea involves randomizing certain aspects of
the reconstruction, such as viewpoint and illumination, generating virtual
views of the reconstructed 3D object, and allowing the 2D network to assess
the quality of the resulting image, thus providing feedback to the
reconstructor. Unlike work based on distillation, which produces a single 3D
asset for each textual prompt, our approach yields a monocular reconstruction
network capable of outputting a controllable 3D asset from any given image,
whether real or generated, in a single forward pass in a matter of seconds.
Our network can be used for analysis, including monocular reconstruction, or
for synthesis, generating articulated assets for real-time applications such
as video games. The code can be found on the project page at
https://farm3d.github.io/.

Figure 1:  Learning to Reconstruct 3D Animal Categories Purely from Synthetic
Images. Our method learns to reconstruct articulated and textured animals from
single images, using only virtual supervision from an off-the-shelf diffusion-
based 2D image generator, without the curation of any real training images.
The method generalizes to a wide range of animal categories, such as cows,
horses, sheep, pigs, and dogs. Moreover, it can be used for controllable 3D
synthesis. For instance, we can relight our generated 3D assets, swap their
textures by conditioning on another input, and animate our articulated shapes,
giving us more precise control over the generated assets.  **footnotetext:
Equal contribution.

##  1 Introduction

The success of generative AI has led to significant progress in image
generation. Methods like DALL-E [22], Imagen [26], and Stable Diffusion [23]
can produce high-quality images from textual prompts. While these image
generators are not trained with 3D capabilities, they can be used to recover
3D shapes by distillation [21, 19].

A limitation of these distillation methods, is that they require test-time
optimization to recover each 3D output. Not only is this slow, but it also
does not learn a true statistical model of the 3D objects. In this paper, we
thus ask whether it is possible to extract from a 2D image generator, not
individual 3D instances, but a _feed-forward model for an articulated 3D
object category_. A category-level model, say for cows, sheep or horses,
captures an entire family of articulated 3D objects. Once learned, it can
reconstruct a specific object from a single real or generated image in a feed-
forward manner, in seconds rather than in hours.

Besides the theoretical interest, answering this question has a practical
interest too. Recent works [15, 41] have learned analogous models of 3D object
categories using real images, but they are sensitive to excessive occlusions,
truncations, background clutter, and extreme poses. Such noise is present even
in relatively curated datasets like ImageNet ([4], Fig. 2) – our estimate is
that 41% of its images contain occlusions – and manually filtering such images
to train one of these models would require about 9 hours per category.111We
assume 2 seconds/image, 41% of noisy images, aim for $10$k clean images, as in
[41].

In this paper, we show that we can _wholly replace_ this real training data
with synthetic images obtained from a 2D image generator. Remarkably, while
these generators are in their own right trained on billions of uncurated
images, the images that they output are on average much cleaner – we estimate
that only 4% of Stable Diffusion [24] images that we generate contain
occlusions. For each category, we can thus simply specify a textual
prompt222The prompt is engineered once for all categories. and obtain clean
training data without manual filtering. Notably, the learned 3D models still
generalize to real images at test time.

Besides generating training images, we also extend Score Distillation Sampling
(SDS) [21] to provide virtual supervision not for a specific 3D object, but
for the category-level model itself. The advantage is that, while the training
images only capture a single view of the object, distillation can provide
virtual supervision for arbitrary viewpoints, thus providing multi-view
constraints that cannot be obtained from monocular real single-view images. We
also make an important technical point and show how to match the energy of the
noise in SDS to the learning process in order to improve convergence.

We assess Farm3D qualitatively and quantitatively. For the latter, we find
prior work lacking a dataset with 3D ground truth to be used for direct
evaluation of 3D reconstructions, resorting instead to evaluating on 2D
proxies like keypoint transfer. To remedy this problem, we introduce a new
synthetic 3D benchmark dataset designed for monocular articulated 3D object
reconstructions. In this way, we are the first to evaluate this class of
methods directly on their task of 3D geometry recovery. Using this benchmark,
despite not using any real images for training, thus avoiding tedious data
collection and curation, our method achieves performance comparable to state-
of-the-art baselines.

To summarise, we make the following contributions: (1) we show that we can
learn models of articulated 3D object categories using only a 2D image
generator, without real data; (2) at test time, these models can reconstruct
individual 3D objects from a single real or synthetic image in seconds instead
of hours required by distillation; (3) we show that 2D image generators
implicitly “curate” billions of noisy training images and produce clean
training data without manual curation; (4) we extend SDS to supervise not
individual objects, but a statistical model of an entire category of such
objects; (5) we show how to improve SDS by matching the noise energy to the
learning process; (6) we introduce a new synthetic dataset for a more rigorous
evaluation of articulated 3D reconstruction methods.

Figure 2: Examples of typical unsuitable images from ImageNet.

##  2 Related Work

#### Weakly-Supervised 3D Object Learning.

While reconstructing deformable 3D objects typically requires simultaneous
multi-view captures [7], recent works have shown that it is possible to learn
3D models of deformable objects purely from monocular image collections, with
a form of geometric supervision in addition to segmentation masks, like
keypoint annotations [11, 10], category-specific template shapes [6, 14, 13],
semantic correspondences distilled from image features [15, 46, 41, 47] and/or
strong assumptions like symmetry [38, 40]. Alternatively, with a known prior
viewpoint distribution, a generative adversarial framework can be used to
learn simpler 3D objects, such as faces and cars [20, 28, 2, 3]. Monocular
videos have also been used for training, incorporating additional temporal
signals for learning [42, 43, 39, 44, 45]. Although impressive results have
been shown, most still rely on heavily curated category-specific data for
training, limiting the model to only few categories. Here, we introduce a
method for distilling 3D objects from large 2D diffusion models, which can
potentially generalise to a wide range of object categories.

Figure 3: Training Pipeline. We prompt Stable Diffusion for synthetic images
of an object category that are then used to train a monocular articulated
object reconstruction model that factorises the input image of an object
instance into articulated shape, appearance (albedo and diffuse and ambient
intensities), viewpoint, and light direction. During training, we also sample
virtual instance views that are then “critiqued” by Stable Diffusion to guide
the learning.

#### Diffusion Models.

Diffusion models in recent years [31, 32, 8, 33] became the keystone of text-
to-X generative models, where X can be images [22, 26, 23], videos [29],
vector graphs [9], audio [18] and so on. These models generate complex high-
fidelity samples by learning to reverse a diffusion process, _i.e_.,
progressively removing synthetically added noise until the image is recovered.
Text-to-image diffusion models [22, 26, 23], in particular, introduce textual
conditioning to these generative models, which offers a powerful interface for
general controllable image generation. Although these models have demonstrated
some level of compositionality and controllability, it is unclear what kind of
3D information is encoded in these learned image synthesis models. Recently,
diffusion models have been investigated for generating training data for 2D
tasks [27, 36]. Here, we use them to generate training data for learning a
category-specific 3D reconstructor.

#### Distilling 3D Models from Image Diffusion Models.

Several recent works explored the extraction of 3D information from large pre-
trained 2D image diffusion models. DreamFusion [21] and Magic3D [16] have
demonstrated the possibility of generating diverse full 3D objects from text
queries by prompting pre-trained image diffusion models. Make-A-Video3D [30]
adopts a similar strategy for generating 4D dynamic scenes; RealFusion [19]
extends the pipeline to reconstructing 3D objects in photos.

Our method differs from these approaches by learning an articulated category-
level model. This has several advantages. First, it predicts 3D shapes in a
single forward pass, eliminating the need for lengthy optimisation as required
by other methods. Second, the category-level model enables us to directly
relate semantically corresponding points on the surface of objects within the
same category, which enables numerous applications, such as texture swapping
(Fig. 9) and image understanding (Tab. 1). Third, our method learns
articulated shapes, which can be readily animated.

##  3 Method

Our goal is to learn an articulated 3D model of an object category, such as
cows, sheep or horses, using exclusively a pre-trained 2D image generator
(Sec. 3.2). We start from a method designed to learn 3D models from real data
(Sec. 3.1) and (1) replace the real data with images obtained by applying to a
2D generator to category-agnostic textual prompts (Sec. 3.3) and (2) modify
the 3D model’s training objective to use the 2D diffusion model as a critic
(Sec. 3.4). We provide an overview of the method in Fig. 3.

###  3.1 Background: Articulated Category Model

We base our model on MagicPony [41], which learns articulated 3D objects from
real monocular images. It does so by training a monocular reconstruction
network as a photo-geometric auto-encoder. This auto-encoder, $f_{\theta}$,
receives as input a single RGB image $I\in\mathbb{R}^{3\times H\times W}$ of
the object and outputs a set of photo-geometric parameters
$(s,a,v,l)=f_{\theta}(I)$ describing it. Here, $\theta$ are the parameters of
the autoencoder, $s$ is the object shape (which incorporates a category-level
shape prior, an instance-specific deformation, and the image-specific skeleton
pose), $a$ is the appearance (accounting for albedo and diffuse and ambient
illumination), $v\in SE(3)$ is the object viewpoint (expressed as a rotation
and a translation with respect to the camera), and $l\in\mathbb{S}^{2}$ is the
dominant direction of the illuminant.

The photo-geometric encoder $f_{\theta}$ is paired with a rendering function
$\hat{I}=R(s,a,v,l)$ which outputs the image of the object. Key to the method
is the fact that $R$ is a _handcrafted_ (not learned) differentiable renderer,
which implicitly assigns $s,a,v,l$ their photo-geometric meaning.

The model is learned from a collection of single-view images, assuming that
images are cropped around the objects with no occlusions or truncation.
Learning also assumes knowledge of the object masks $M\in\\{0,1\\}^{H\times
W}$, obtained using a segmenter such as PointRend [12]. Given the resulting
set $\mathcal{D}$ of training image-mask pairs $(I,M)$, MagicPony minimises
the objective function

$\mathcal{L}(\theta|\mathcal{D})=\frac{1}{|\mathcal{D}|}\sum_{(I,M)\in\mathcal{D}}\mathcal{L}(f_{\theta}(I)|I,M)+\mathcal{R}(f_{\theta}(I)),$

where the term $\mathcal{L}(f_{\theta}(I)|I,M)$ measures how well the
predicted 3D object reconstructs the input image $I$, the input mask $M$, and
off-the-shelf image features $\Phi(I)$ (ViT-DINO), and the regulariser
$\mathcal{R}(f_{\theta}(I))$ (Eikonal loss for an SDF) smooths the prior shape
and controls the amount of instance-specific deformation and articulation.

###  3.2 Background: Image Generation with Diffusion

For the image generator we use Stable Diffusion (SD) [23]. SD uses an auto-
encoder $h$ to reduce the image $I$ to a smaller latent code
$z_{0}=h(I)\in\mathbb{R}^{D^{\prime}\times H_{h}\times W_{h}}$. It then learns
a conditional distribution $p(z_{0}|y)$ of the latent code conditioned on a
textual prompt $y$. It does so by considering the sequence of noised signals
$z_{t}=\alpha_{t}z_{0}+\sigma_{t}\epsilon_{t}=\alpha_{t}h(I)+\sigma_{t}\epsilon_{t}$
where $\epsilon_{t}$ is normally distributed noise,
$\alpha_{t}=\sqrt{1-\sigma_{t}^{2}}$, and $\sigma_{t}\in(0,1)$,
$t=0,1,\dots,T$ is an increasing sequence of noise standard deviations (from
$\sigma_{0}\approx 0$ to $\sigma_{T}\approx 1$). The model consists of a
‘denoising network’ $\hat{\epsilon}_{t}(z_{t}|y)$ that approximates the noise
content $\epsilon_{t}$ of $z_{t}$. The latter is trained to minimise a loss of
the type

$\mathcal{L}(\hat{\epsilon})=\mathbb{E}_{t,\epsilon,I,y}[\|\hat{\epsilon}_{t}(\alpha_{t}h(I)+\sigma_{t}\epsilon_{t}|y)-\epsilon_{t}\|^{2}],$
(1)

averaged over noise levels $t$, noise samples $\epsilon$ and empirical samples
$(I,y)$ (captioned images).

In order to draw an image sample $I$ from the learned distribution, one draws
a random sample $z_{T}$ from a normal distribution and then progressively
denoise it with
$z_{t-1}=\alpha_{t}^{-1}(z_{t}-\sigma_{t}\hat{\epsilon}_{t}(z_{t}))$ to obtain
$z_{0}$ and eventually $I=h^{-1}(z_{0})$.

###  3.3 Generating Training Images via Prompting

Figure 4: Synthetic Training Images Generated with Stable Diffusion. The
generated animals are typically without occlusions but sometimes anatomically
incorrect (_e.g_., columns 2 and 3), but our model is robust to this and
learns plausible 3D shapes.

In Sec. 3.1, the training data $\mathcal{D}$ comprises curated real images. We
propose replacing it with _synthetic_ images from SD, prompted with text
specific to each object category, derived from the same category-agnostic
textual template, with the only variation being the category name.

When prompted using only the name of the object category, _e.g_., ‘cow’, SD
generates mostly frontal and side views of the object. We hypothesise that
this is due to the bias contained in its training data, an observation similar
to that of [21]. We found that naive view-dependent prompting [21] for {_side,
front, back_} does not work well with SD (as opposed to Imagen [26] used in
[21]). Instead, we found that prompting SD for an animal category “walking
away from the camera” biases the generation well enough to obtain images that
have diverse view coverage. More details on the textual prompt design together
with qualitative analysis are included in the supp. mat.

Figure 5: Noise Scheduling and SDS Gradient. We show four rendered images
$\hat{I}$, obtained by first sampling a random viewpoint and illumination as
for training our model. Then, we pick a fixed noise sample $\epsilon$ and show
the SDS gradient $(\hat{\epsilon}_{t}(z_{t}|y)-\epsilon)({\partial
h}/{\partial\hat{I}})$ used to update $\hat{I}$ in Eq. 2 for different values
of $\sigma_{t}$. Because $\epsilon$ is fixed,
$z_{t}=\alpha_{t}h(\hat{I})+\sigma_{t}\epsilon$ only depends on $\sigma_{t}$.
Large noise levels ($t=0.9$) generate an update which is essentially
independent of the input image $\hat{I}$. Lower noise levels provide more
meaningful gradients and lead to more stable training as shown in Fig. 11.
Figure 6: Single-View Real Image Reconstruction. We visualize the predicted
textured 3D meshes from both input and novel views. Our method learns 3D
models of a wide range of categories with fine details, _e.g_. legs and ears,
despite not being trained on any real images.

###  3.4 Distilling a 3D Reconstructor

We can use the SD image generator not only to obtain training images but also
for _virtual multi-view supervision_ to further constrain the reconstruction
network $f_{\theta}$. Given an example image $I$, we first obtain an estimate
$(s,a,v,l)=f_{\theta}(I)$ of the object’s photo-geometric parameters. Then, we
randomly sample a _new camera viewpoint_ $\tilde{v}$ and a _new light
direction_ $\tilde{l}$, and render the corresponding image as follows:

$\tilde{I}=R(s,a,\tilde{v},\tilde{l})=R\circ\operatornamewithlimits{subst}_{\tilde{v},\tilde{l}}\circ
f_{\theta}(I).$

Here the operator $\operatorname{subst}_{(\tilde{v},\tilde{l})}$ replaces the
predicted viewpoint $v$ and light $l$ from the output of $f_{\theta}(I)$ with
the new values $\tilde{v}$ and $\tilde{l}$. We have no direct supervision for
the resulting image $\tilde{I}$, but, we can use the 2D image generator as a
critic to judge whether $\tilde{I}$ ‘looks correct’.

Concretely, this is achieved through a new way of using the SDS loss [21].
Given a training image $I$, the corresponding caption $y$, and randomly-
sampled viewpoint $\tilde{v}$ and lighting $\tilde{l}$ parameters, the
_gradient_ of the distillation loss is:

$\nabla_{\theta}\mathcal{L}_{\text{SDS}}(\theta|\tilde{v},\tilde{l},I,y)=\mathbb{E}_{t,\epsilon}\left[w_{t}\cdot\left(\hat{\epsilon}_{t}\left(z_{t}|y\right)-\epsilon_{t}\right)\frac{\partial
z_{t}}{\partial\theta}\right],$ (2)

where $w_{t}>0$ is a weight factor, $\epsilon_{t}$ a noise sample, and

$z_{t}=\alpha_{t}\cdot\left(h\circ
R\circ\operatornamewithlimits{subst}_{\tilde{v},\tilde{l}}\circ
f_{\theta}\right)(I)+\sigma_{t}\epsilon_{t}$

is the noised version of the object image obtained from the new viewpoint and
lighting. Based on this definition, the derivative in Eq. 2 is, with a slight
abuse of notation,

$\frac{\partial z_{t}}{\partial\theta}=\alpha_{t}\cdot\left.\frac{\partial
h}{\partial
I}\right|_{\tilde{I}}\cdot\left.\frac{\partial{R}}{\partial(s,a)}\right|_{(s,a,\tilde{v},\tilde{l})}\cdot\left.\frac{\partial[f_{\theta}]_{sa}}{\partial\theta}\right|_{I}.$

The first factor $\alpha_{t}$ comes from the diffusion scaling. The second
factor is the derivative of the Stable Diffusion latent code $z_{0}$ with
respect to the coded image $I$. The third factor is the derivative of the
rendering function with respect to the shape and appearance components only.
The last factor is the derivative of the shape $s$ and appearance $a$
predictions by the model $f_{\theta}$. Note that the viewpoint $\tilde{v}$ and
the lighting $\tilde{l}$ are not included in the derivative because they are
not estimated by $f_{\theta}$ but sampled.

#### Noise Scheduling.

We found that scheduling the noise properly in Eq. 2 is critical. If the noise
is too large, the noised latent image bears little to no relation to the one
that is actually reconstructed; as a result, the feedback from the SDS loss
points in a novel direction, reconstructing ‘any’ object compatible with the
prompt $y$ (as in DreamFusion [21]), instead of improving the reconstruction
of the _specific_ object contained in the input image $I$. As motivated in
Fig. 5, in practice, we sample $t\sim\mathcal{U}(0.02,t_{\mathrm{max}})$
where$\enspace t_{\mathrm{max}}=0.6$ in the definition of Eq. 2, as opposed to
DreamFusion that uses $t_{\mathrm{max}}=0.98$. The choice of
$t_{\mathrm{max}}$ is further supported by an ablation study in Fig. 11.

###  3.5 Discussion

While our approach borrows several ideas from DreamFusion [21], the
application is rather different. DreamFusion distils from the image generator
$\epsilon$ a _single_ 3D object $V$ (a neural radiance field), corresponding
to a single textual prompt $y$. We learn instead a _monocular reconstruction
network_ $f_{\theta}$ which embodies a category-level articulated 3D model. At
test time, this network can infer a 3D object $(s,a,v,l)=f_{\theta}(I)$ from a
single image $I$ of a new instance in seconds, through feed-forward
calculations. This should be contrasted to inference in DreamFusion, which
requires hours of test-time optimization for each 3D object. Our method can
also be used to reconstruct photographs of real objects, which DreamFusion is
not capable of, and the output is not a radiance field, but an explicit
_articulated mesh_ , which can be easily used in applications.

The main shortcoming of our model compared to test-time distillation is the
fact that generality is limited to a single category. Furthermore, we make
assumptions about the object categories we learn, including the topology of
the articulated skeleton (_e.g_., 4 legs).

##  4 Experiments

Figure 7: Qualitative Comparison. Comparison with MagicPony [41] (Horse and
Cow) and MagicPony* trained on ImageNet on real Horse, Cow and Sheep images.
Our model predicts more plausible 3D shapes than MagicPony without being
trained on any real images. When MagicPony* is trained on uncurated data, the
predicted shape is less plausible, exhibiting many artefacts and inaccuracies.
This especially pronounced with cows and sheep, whose training images are
noisier than those of horses.

We conduct experiments on both single image 3D reconstruction (Sec. 4.2) and
3D asset generation (Sec. 4.3) on several animal categories, including cows,
sheep, horses, pigs, and Dalmatian dogs, showcasing the generalisation
capability of our method across a variety of articulated animal species. We
also compare our approach both qualitatively and quantitatively to prior work,
demonstrating that even when training solely on synthetic images, we can still
achieve comparable results.

###  4.1 Datasets

We generate $30$k synthetic images per object category using the procedure
described in Sec. 3.3 to train our method without any real images. The
segmentation masks are obtained with PointRend [12] and the samples containing
likely occluded instances are automatically removed using simple heuristics,
as detailed in the sup. mat. Fig. 4 shows a few examples of the pre-processed
synthetic training data.

For cows, sheep and horses, we evaluate our model qualitatively using PASCAL
[5] and COCO [17] images and quantitatively on the keypoint transfer task
using annotations from PASCAL. For pigs and Dalmatian dogs, we show
qualitative results using ImageNet [4] and MagicPony images, respectively.
Since MagicPony [41] does not provide a model for sheep, we collect images
from ImageNet sheep synsets to fine-tune their pre-trained horse model.

For all images, we follow the same pre-processing procedure as for our
synthetic data to obtain segmentation masks and remove truncated objects.

#### 3D Articulated Animals Dataset.

We introduce a new 3D Articulated Animals Dataset to directly evaluate the
quality of single-view 3D reconstruction of articulated animals. This dataset
includes textured 3D meshes of articulated animals, including horses, cows,
and sheep, crafted by a professional 3D artist, and is accompanied by
realistic articulated animations. For each object category, we generated a
randomly sampled set of images, each paired with the corresponding 3D ground-
truth mesh. To render the images, we use Blender333http://www.blender.org, and
for each asset, we randomly select an animation frame, a viewpoint, and an
HDRI environmental map from a publicly available
collection444https://polyhaven.com, which is used to simulate the background
and lighting. Rendered examples can be found in Fig. 8. We intend to release
this dataset for future benchmarking purposes. Additional dataset details are
provided in the supplementary material.

Figure 8:  3D Articulated Animals Dataset. Examples of rendered images from
our new dataset used for evaluation.  Table 1: Quantitative Comparison. We
report<EMAIL_ADDRESS>($\uparrow$ better) on PASCAL VOC and bi-directional Chamfer
Distance ($\downarrow$ better), with GT objects normalized in a 1m cube, on 3D
Articulated Animals. Our method matches MagicPony without training on real
images. † A-CSM starts with an articulated 3D model, learning camera and
articulations, not shape. ‡ MagicPony does not offer a model for sheep.
Therefore, we fine-tuned their horse model using sheep images from ImageNet. *
MagicPony trained on ImageNet from scratch. Extended version in the sup. mat.

Method |<EMAIL_ADDRESS>(%) $\uparrow$ |  Chamfer Distance (cm) $\downarrow$
---|---|---
Horses | Cows | Sheep | Horses | Cows | Sheep
A-CSM [14]† | 32.9 | 26.3 | 28.6 |  2.93 $\pm 1.16$ |  2.35 $\pm 0.68$ |  2.48 $\pm 0.70$
UMR [15] | 24.4 | — | — |  3.51 $\pm 1.51$ | — | —
MagicPony [41] | 42.9 | 42.5 |  41.2‡ |  2.50 $\pm 0.69$ |  2.53 $\pm 0.59$ |  3.00 $\pm 0.68$‡
MagicPony* | 49.1 | 39.9 | 38.3 |  3.48 $\pm 1.06$ |  2.54 $\pm 0.62$ |  3.93 $\pm 0.62$
Ours (full model) | 42.5 | 40.2 | 36.1 |  2.85 $\pm 0.83$ |  2.41 $\pm 0.54$ |  3.31 $\pm 0.49$
Ours w/o SDS | 41.4 | 37.0 | 37.6 |  3.54 $\pm 1.00$ |  2.50 $\pm 0.60$ |  3.47 $\pm 0.84$

###  4.2 Single Image 3D Reconstruction

Figure 9: Controllable 3D Synthesis. Our method enables the generation of
controllable 3D assets from either a real image or an image synthesised using
Stable Diffusion. Once generated, we have the ability to adjust lighting, swap
textures between models of the same category, and even animate the shape. For
shapes out of the training distribution, in this case “cow made out of fruits”
(top row), we use the optional test time shape adaptation (Sec. 4.3).

We evaluate the effectiveness of our method on the monocular 3D reconstruction
task. The results, as shown in Fig. 6, demonstrate that our method is able to
generalise across several animal categories, producing high-quality 3D meshes
with faithful texture and accurate articulation, including reconstruction of
fine-grained shapes such as legs and ears. We also provide additional results
in the sup. mat to further support the effectiveness of our approach.

#### Qualitative Comparisons.

We compare our model against MagicPony [41], which is trained on a large
collection of curated real images, including video frames, whereas our model
learns purely from synthetic training data generated by Stable Diffusion. To
further illustrate the vulnerability of weakly-supervised methods to uncurated
data, we train MagicPony [41] from scratch on ImageNet images from the horse,
cow, and sheep synsets, applying the same automatic pre-processing we use for
the synthetic images. Fig. 7 shows a qualitative comparison of single image 3D
reconstruction results. Despite not seeing any real images at training, our
model reconstructs high-quality articulated shapes, as visualized from novel
views. Note that the overall shapes look more natural compared to the
MagicPony baseline (_e.g_. no bumps on the horseback), owing to the virtual
multi-view supervision from the distillation loss. This observation is further
supported by the ablation study presented in Fig. 10. The quality of
MagicPony’s reconstructions significantly deteriorates when trained from
scratch using uncurated ImageNet images, which highlights the importance of
the time-consuming data curation procedures.

#### Quantitative Comparisons.

To evaluate our approach quantitatively, we employ two metrics: keypoint
transfer and Chamfer distance between the predicted 3D shape and the ground
truth (Tab. 1). Keypoint transfer is commonly used in weakly-supervised object
reconstruction [15, 14, 41], specifically on PASCAL [5], and it primarily
assesses the quality of the predicted pose and articulation.

We also compute Chamfer distance between the predicted and ground-truth shapes
using our 3D Articulated Animals Dataset, which provides a direct assessment
of the 3D reconstruction accuracy. We follow the procedure in [39].
Specifically, we first scale the predicted shape to match the volume of the 3D
ground-truth shape and roughly align the canonical pose manually for each
method. We then run the iterative closest point (ICP) algorithm [1] to
automatically align each predicted shape to the ground-truth, and report the
bi-directional Chamfer distance in cm, with objects normalized into a 1m cube.

In addition to MagicPony [41], we also compare our method with two other
weakly-supervised 3D object learning methods: UMR [15] and A-CSM [14]. Note
that only the horse model from UMR has been made publicly available. UMR does
not model complex articulations and, as a result, often produces shapes with
reduced accuracy.

Our method significantly outperforms UMR on both metrics. A-CSM predicts
articulated poses given a category-specific articulated template shape, and
results in a lower accuracy in the keypoint transfer metric. On the Chamfer
distance metric, it performs reasonably as it relies on an input template
shape and learns only the poses.

Even without training on real images, our method achieves comparable results
to MagicPony which relies on heavy data curation. The qualitative comparison,
as shown in Fig. 7, demonstrates a sharp deterioration in MagicPony’s
performance when trained on uncurated ImageNet. However, this is not mirrored
in the quantitative evaluation. This is because the keypoint transfer metric
only looks at a few points and is not sensitive to artefacts on the surface.
On the Chamfer distance metric, as all existing methods are not yet capable of
predicting accurate shapes well aligned with the ground-truth meshes, the
errors are still dominated by the inaccurate poses (_e.g_. confusing left and
right legs). Nonetheless, we reach similar performance of prior work by
training only on synthetic data. This also suggests that monocular 3D
reconstruction is still far from solved. We hope the new benchmark will
facilitate assessing the progress in this field.

###  4.3 Controllable 3D Shape Synthesis

In addition to controlling articulation via the underlying skeleton model, our
approach enables controllable 3D synthesis, as demonstrated in Fig. 9. The
model can be conditioned on either an image or text. For text conditioning,
such as “sheep in baroque armor”, Stable Diffusion is first used to generate
an example image of the input prompt. The image is then fed into the model as
done with real images.

Unlike distillation approaches that require hours of test-time optimisation,
our approach produces the 3D shape in a single pass, enabling faster
iterations when producing 3D assets. Notably, we can explicitly control
articulation, lighting, and albedo (_i.e_., texture swapping) using our
method.

#### Test Time Shape Adaptation (Optional).

During test time, we can optionally fine-tune our category model on a single
image. This allows us to adapt the model to shapes that are out of the
training distribution, such as “cow made out of fruits” as demonstrated in
Fig. 9. As this process is very fast (less than 30 seconds) this allows users
to control the outputs of generative models in 3D, which is an exciting
capability with many potential applications.

###  4.4 Ablations

We conduct ablations to evaluate the effect of the SDS guidance on the quality
of the learned pose and shape, both quantitatively in Tab. 1 and qualitatively
in Fig. 10. Our qualitative results illustrate that using SDS helps the method
learn more plausible shapes and articulation.

We also perform an ablation study on the noise scheduling in Fig. 11,
demonstrating that it is critical to training stability. Without our noise
scheduling, the learning is unstable and often collapses multiple times during
training.

Figure 10: Ablation Study on the SDS Loss. The SDS loss utilizes rendered
virtual views of an object instance under random directional light. These
views are encouraged to resemble real images, which helps to achieve more
consistent object pose and articulation. However, this may lead to side
effects, such as simulating a sheep’s ’fluffiness’ through surface wrinkles.
Figure 11: Effect of our SDS Noise Scheduling on the Category-Specific Prior
Shape during Training. Sampling with $t_{max}=0.6$ helps the SDS guidance stay
more faithful to the input image, and stabilises training as can be seen from
the learned prior shape.

##  5 Conclusions

We introduced Farm3D, a method for learning a monocular 3D reconstructor of an
object category entirely from synthetic data from an off-the-shelf text-to-
image generator. There are many enticing aspects of this approach: it reduces
the need for collecting and curating real data for learning 3D objects, it
learns 3D reconstruction networks that produce outputs in seconds at test
time, and these networks can be used for monocular reconstruction (analysis)
or for 3D asset generation (synthesis), producing articulated meshes ready for
animation and rendering in applications.

#### Data Ethics.

We use the DOVE, Weizmann Horse, PASCAL, Horse-10, and MS-COCO datasets in a
manner compatible with their terms. Some of these images may accidentally
contain faces or other personal information, but we do not make use of these
images or image regions. For further details on ethics, data protection, and
copyright please see
https://www.robots.ox.ac.uk/~vedaldi/research/union/ethics.html.

#### Acknowledgements.

Shangzhe Wu is supported by a research gift from Meta Research. Andrea Vedaldi
and Christian Rupprecht are supported by ERC-CoG UNION 101001212. Tomas Jakab
and Christian Rupprecht are (also) supported by VisualAI EP/T028572/1.

## References

  * Besl and McKay [1992] Paul J. Besl and Neil D. McKay.  A method for registration of 3-D shapes.  _PAMI_ , 14(2), 1992.
  * Chan et al. [2021] Eric Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein.  pi-GAN: Periodic implicit generative adversarial networks for 3d-aware image synthesis.  In _CVPR_ , 2021.
  * Chan et al. [2022] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein.  Efficient geometry-aware 3D generative adversarial networks.  In _CVPR_ , 2022.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 2009.
  * Everingham et al. [2015] Mark Everingham, S. M. Ali Eslami, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  _IJCV_ , 2015.
  * Goel et al. [2020] Shubham Goel, Angjoo Kanazawa, and Jitendra Malik.  Shape and viewpoints without keypoints.  In _ECCV_ , 2020.
  * Hartley and Zisserman [2004] Richard Hartley and Andrew Zisserman.  _Multiple View Geometry in Computer Vision_.  Cambridge University Press, ISBN: 0521540518, second edition, 2004.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , 2020.
  * Jain et al. [2023] Ajay Jain, Amber Xie, and Pieter Abbeel.  VectorFusion: Text-to-svg by abstracting pixel-based diffusion models.  In _CVPR_ , 2023.
  * Kanazawa et al. [2018] Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik.  Learning category-specific mesh reconstruction from image collections.  In _ECCV_ , 2018.
  * Kar et al. [2015] Abhishek Kar, Shubham Tulsiani, João Carreira, and Jitendra Malik.  Category-specific object reconstruction from a single image.  In _CVPR_ , 2015.
  * Kirillov et al. [2020] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick.  PointRend: Image segmentation as rendering.  In _CVPR_ , 2020.
  * Kokkinos and Kokkinos [2021] Filippos Kokkinos and Iasonas Kokkinos.  To the point: Correspondence-driven monocular 3d category reconstruction.  In _NeurIPS_ , 2021.
  * Kulkarni et al. [2020] Nilesh Kulkarni, Abhinav Gupta, David F. Fouhey, and Shubham Tulsiani.  Articulation-aware canonical surface mapping.  In _Proc. CVPR_ , pages 449–458, 2020.
  * Li et al. [2020] Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang, and Jan Kautz.  Self-supervised single-view 3d reconstruction via semantic consistency.  In _ECCV_ , 2020.
  * Lin et al. [2023] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In _CVPR_ , 2023.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In _ECCV_ , 2014.
  * Liu et al. [2023] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley.  Audioldm: Text-to-audio generation with latent diffusion models.  _arXiv preprint arXiv:2301.12503_ , 2023.
  * Melas-Kyriazi et al. [2023] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi.  RealFusion: 360 reconstruction of any object from a single image.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Nguyen-Phuoc et al. [2019] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang.  HoloGAN: Unsupervised learning of 3d representations from natural images.  In _ICCV_ , 2019.
  * Poole et al. [2023] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  DreamFusion: Text-to-3d using 2d diffusion.  In _ICLR_ , 2023.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _ICML_ , 2021.
  * Rombach et al. [2022a] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022a.
  * Rombach et al. [2022b] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proc. CVPR_ , 2022b.
  * Rother et al. [2004] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake.  “GrabCut”: interactive foreground extraction using iterated graph cuts.  _ACM TOG_ , 23(3):309–314, 2004.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  In _NeurIPS_ , 2022.
  * Sarıyıldız et al. [2023] Mert Bülent Sarıyıldız, Karteek Alahari, Diane Larlus, and Yannis Kalantidis.  Fake it till you make it: Learning transferable representations from synthetic imagenet clones.  In _CVPR_ , 2023.
  * Schwarz et al. [2020] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  GRAF: Generative radiance fields for 3d-aware image synthesis.  In _NeurIPS_ , 2020.
  * Singer et al. [2023a] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman.  Make-a-video: Text-to-video generation without text-video data.  In _ICLR_ , 2023a.
  * Singer et al. [2023b] Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, and Yaniv Taigman.  Text-to-4d dynamic scene generation.  _arXiv preprint arXiv:2301.11280_ , 2023b.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _ICML_ , 2015.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  In _NeurIPS_ , 2019.
  * Song et al. [2021] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _ICLR_ , 2021.
  * Tang [2022] Jiaxiang Tang.  Stable-dreamfusion: Text-to-3d with stable-diffusion.  https://github.com/ashawkey/stable-dreamfusion, 2022.
  * Tang et al. [2022] Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture.  What the DAAM: Interpreting stable diffusion using cross attention.  _arXiv preprint arXiv:2210.04885_ , 2022.
  * Tian et al. [2023] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan.  Stablerep: Synthetic images from text-to-image models make strong visual representation learners.  _arXiv preprint arXiv:2306.00984_ , 2023.
  * von Platen et al. [2022] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf.  Diffusers: State-of-the-art diffusion models.  https://github.com/huggingface/diffusers, 2022.
  * Wu et al. [2020] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Unsupervised learning of probably symmetric deformable 3D objects from images in the wild.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Wu et al. [2021a] Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi.  DOVE: Learning deformable 3d objects by watching videos.  _arXiv preprint arXiv:2107.10844_ , 2021a.
  * Wu et al. [2021b] Shangzhe Wu, Ameesh Makadia, Jiajun Wu, Noah Snavely, Richard Tucker, and Angjoo Kanazawa.  De-rendering the world’s revolutionary artefacts.  In _CVPR_ , 2021b.
  * Wu et al. [2023] Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi.  MagicPony: Learning articulated 3d animals in the wild.  In _CVPR_ , 2023.
  * Yang et al. [2021a] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T. Freeman, and Ce Liu.  LASR: Learning articulated shape reconstruction from a monocular video.  In _CVPR_ , 2021a.
  * Yang et al. [2021b] Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Ce Liu, and Deva Ramanan.  ViSER: Video-specific surface embeddings for articulated 3d shape reconstruction.  In _NeurIPS_ , 2021b.
  * Yang et al. [2022] Gengshan Yang, Minh Vo, Neverova Natalia, Deva Ramanan, Vedaldi Andrea, and Joo Hanbyul.  BANMo: Building animatable 3d neural models from many casual videos.  In _CVPR_ , 2022.
  * Yang et al. [2023] Gengshan Yang, Chaoyang Wang, N. Dinesh Reddy, and Deva Ramanan.  Reconstructing animatable categories from videos.  In _CVPR_ , 2023.
  * Yao et al. [2022] Chun-Han Yao, Wei-Chih Hung, Michael Rubinstein, Yuanzhen Lee, Varun Jampani, and Ming-Hsuan Yang.  Lassie: Learning articulated shape from sparse image ensemble via 3d part discovery.  In _NeurIPS_ , 2022.
  * Yao et al. [2023] Chun-Han Yao, Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming-Hsuan Yang, and Varun Jampani.  Hi-lassie: High-fidelity articulated shape and skeleton discovery from sparse image ensemble.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4853–4862, 2023.

## Appendix

Table 2: Evaluation on 3D Articulated Animals Dataset. We report bi-
directional Chamfer Distance in cm ($\downarrow$ better) on the 3D Articulated
Animals dataset. We evaluate the methods using both the normalized versions of
the objects (GT objects fitted inside a 1m cube) and their real-sized
counterparts. Our method achieves results comparable to those of MagicPony
without training on any real images. Test-time optimization-based methods
directly optimize the test images; in contrast, our method is not trained on
these images and reconstructs the 3D shape in seconds. † A-CSM starts from an
existing articulated 3D model of the object category and only learns camera
and articulations, but not shape. ‡ MagicPony does not offer a model for
sheep. Therefore, we fine-tuned their horse model using sheep images from
ImageNet. * MagicPony trained on ImageNet from scratch.

Method | 1k samples (full dataset) | 30 samples (subset)
---|---|---
Normalized | Real-Sized | Normalized | Real-Sized
Horses | Cows | Sheep | Horses | Cows | Sheep | Horses | Cows | Sheep | Horses | Cows | Sheep
_Test-Time Optimization-Based Methods_
LASSIE [46] | — | — | — | — | — | — |  2.66 $\pm 0.80$ |  2.17 $\pm 0.76$ |  2.34 $\pm 0.63$ |  7.45 $\pm 2.23$ |  6.15 $\pm 1.82$ |  2.68 $\pm 0.65$
RealFusion [19] | — | — | — | — | — | — |  5.07 $\pm 1.35$ |  4.28 $\pm 0.79$ |  5.62 $\pm 1.47$ |  14.16 $\pm 3.62$ |  12.29 $\pm 2.17$ |  6.43 $\pm 1.50$
_Learning-Based Methods_
A-CSM [14]† |  2.93 $\pm 1.16$ |  2.35 $\pm 0.68$ |  2.48 $\pm 0.70$ |  8.15 $\pm 3.09$ |  6.71 $\pm 1.81$ |  2.84 $\pm 0.77$ |  2.77 $\pm 1.11$ |  2.52 $\pm 0.90$ |  2.49 $\pm 0.66$ |  7.74 $\pm 2.97$ |  7.14 $\pm 2.10$ |  2.85 $\pm 0.70$
UMR [15] |  3.51 $\pm 1.51$ | — | — |  9.75 $\pm 4.11$ | — | — |  3.23 $\pm 1.09$ | — | — |  9.05 $\pm 3.02$ | — | —
MagicPony [41] |  2.50 $\pm 0.69$ |  2.53 $\pm 0.59$ |  3.00 $\pm 0.68$ |  6.95 $\pm 1.90$ |  7.22 $\pm 1.53$ |  3.43 $\pm 0.73$ |  2.39 $\pm 0.52$ |  2.56 $\pm 0.64$ |  2.87 $\pm 0.48$ |  6.72 $\pm 1.62$ |  7.30 $\pm 1.45$ |  3.29 $\pm 0.46$
MagicPony* |  3.48 $\pm 1.06$ |  2.54 $\pm 0.62$ |  3.93 $\pm 0.62$ |  9.69 $\pm 2.85$ |  7.23 $\pm 1.56$ |  4.49 $\pm 0.62$ |  3.34 $\pm 1.13$ |  2.63 $\pm 0.79$ |  3.77 $\pm 0.66$ |  9.35 $\pm 3.15$ |  7.46 $\pm 1.74$ |  4.32 $\pm 0.65$
Ours (full model) |  2.85 $\pm 0.83$ |  2.41 $\pm 0.54$ |  3.31 $\pm 0.49$ |  7.92 $\pm 2.13$ |  6.91 $\pm 1.49$ |  3.79 $\pm 0.55$ |  2.63 $\pm 0.51$ |  2.57 $\pm 0.63$ |  3.32 $\pm 0.48$ |  7.37 $\pm 1.44$ |  7.34 $\pm 1.56$ |  3.80 $\pm 0.46$
Ours w/o SDS |  3.54 $\pm 1.00$ |  2.50 $\pm 0.60$ |  3.47 $\pm 0.84$ |  9.85 $\pm 2.56$ |  7.12 $\pm 1.51$ |  3.96 $\pm 0.90$ |  3.44 $\pm 1.04$ |  2.59 $\pm 0.70$ |  3.30 $\pm 0.72$ |  9.64 $\pm 2.84$ |  7.36 $\pm 1.57$ |  3.78 $\pm 0.74$

Table 3: Evaluation on Keypoint Transfer. We report<EMAIL_ADDRESS>($\uparrow$ better) on PASCAL VOC. Our method achieves results comparable to those of MagicPony without training on any real images. RealFusion does not output 3D meshes in canonical space and thus cannot be used for keypoint transfer. † A-CSM starts from an existing articulated 3D model of the object category and only learns camera and articulations, but not shape. ‡ MagicPony does not offer a model for sheep. Therefore, we fine-tuned their horse model using sheep images from ImageNet. * MagicPony trained on ImageNet from scratch.  Method | Horses | Cows | Sheep
---|---|---|---
_Test-Time Optimization-Based Methods_
LASSIE [46] | 42.2 | 37.5 | 27.5
_Learning-Based Methods_
A-CSM [14]† | 32.9 | 26.3 | 28.6
UMR [15] | 24.4 | — | —
MagicPony [41] | 42.9 | 42.5 |  41.2‡
MagicPony* | 49.1 | 39.9 | 38.3
Ours (full model) | 42.5 | 40.2 | 36.1
Ours w/o SDS | 41.4 | 37.0 | 37.6
Figure 12: Comparison of shape initialisation against LASSIE. Our model begins
with a generic ellipsoid with a simple heuristic description of the bone
topology, whereas LASSIE [46] initiates with a skeleton featuring bones
initialised as cylinders which are reminiscent of the final object structure.
LASSIE employs a much stronger hand-crafted initial shape, which simplifies
their optimisation process.  Figure 13: Comparison with LASSIE and RealFusion.
Our model outputs more realistic 3D shapes than LASSIE [46] and RealFusion
[19] without optimising the shape on the exact same input images. Note that
when compared to LASSIE our model disentangles lighting from albedo when
predicting appearance, resulting in higher-quality texture when the shape is
rendered from a novel view – in the top row, the texture is darker from the
un-illuminated side. RealFusion tends to predict incorrect geometry that can
be flat, full of holes, or with incorrect number of limbs similarly to
DreamFusion.

This supplementary material contains extended comparisons with prior work
(Secs. A.1, A.2 and A.3), a failure case analysis (Sec. A.4), and more
technical details (Appendix B), which also include additional results with
automatically obtained segmentation masks (Sec. B.4).

##  Appendix A Additional Results

###  A.1 Extended Comparison

We include an extended version of quantitative comparison on 3D Articulated
Animals Dataset in Tab. 2. Additionally, we provide extended qualitative
comparisons with prior work of MagicPony [41] and MagicPony trained on
uncurated images from ImageNet in Fig. 14, showcasing the vulnerability of
weakly-supervised methods to uncurated data.

Figure 14: Qualitative Comparison. Comparison with MagicPony [41] (Horse and
Cow) and MagicPony* trained on ImageNet on real Horse, Cow and Sheep images.
Our model predicts more plausible 3D shapes than MagicPony without being
trained on any real images. When MagicPony* is trained on uncurated data, the
predicted shape is less plausible, exhibiting many artefacts and inaccuracies.
This especially pronounced with Cows and Sheep, whose training images are
noisier than those of Horses.

###  A.2 Comparison with Test-time Optimization Methods

We also compare our method with recent test-time optimization-based methods,
namely RealFusion [19] and LASSIE [46]. A later iteration of LASSIE, Hi-LASSIE
[47], does not have publicly available code, which prevents us from evaluating
against it. As LASSIE optimizes over an image ensemble consisting of a few
(10-30) images, we use a subset of 30 images for each category from the 3D
Articulated Animals dataset. Since RealFusion optimizes over a single image
and thus is computationally expensive, we also use the same subset of 30
images.

We show qualitative comparisons in Fig. 13, and report quantitative
comparisons in Tab. 3 and Tab. 2. The key advantage of our method is that it
is orders of magnitude faster than test-time optimization-based methods,
producing a 3D shape in seconds, whereas LASSIE takes roughly an hour and
RealFusion takes roughly $1.5\text{-}3$ hours for each image.

#### RealFusion.

RealFusion [19] extends DreamFusion [21] by conditioning on an input image,
but it inherits similar shortcomings, namely, a long optimization time and not
learning the 3D mesh in canonical space, which is crucial for downstream tasks
such as keypoint transfer. Additionally, it requires an initial textual
inversion step that adds another $60\text{-}120$ minutes to the optimization
time, bringing the total time needed to reconstruct shape to $1.5\text{-}3$
hours. Both the qualitative and quantitative results demonstrate that the
method is highly unstable and produces shapes with highly distorted geometry.

#### LASSIE.

LASSIE [46] deforms geometric primitive shapes (spheres, cylinders, cones,
etc.) for each body bone/part, yielding unnatural bone junctions, as
demonstrated in Fig. 13. The initialization also requires more human input
than our method, as detailed in Fig. 12. Note that LASSIE directly optimizes
the reconstruction for each input image, whereas our model has not seen any of
these input images during training (nor any real images), showcasing its
superb generalizability.

This also highlights the benefit of learning-based methods in unsupervised
single-view 3D reconstruction. Given that this problem is highly ill-posed and
under-constrained, optimization-based approaches necessitate strong
regularization, which can limit their ability to generate complex shapes.
Conversely, learning-based methods can capitalize on multi-view knowledge
acquired from their training data. This work demonstrates that one can gain
comparable, if not better, multi-view prior knowledge from pre-trained 2D
diffusion models, which reduces the need to collect datasets for 3D training
and boosts training stability.

###  A.3 Comparison with DreamFusion

Figure 15: Comparison with DreamFusion. We run Stable-DreamFusion [34], an
open-source implementation of DreamFusion [21]. Stable-DreamFusion produces
shapes with holes and wrong anatomies (for ‘cow’, half of the left front leg
is missing; for ‘horse’, there are 5 legs).

To elaborate on how our method can be a preferable candidate for 3D asset
generation over category-agnostic text-to-3D methods such as DreamFusion [21],
we provide an additional qualitative comparison with Stable-DreamFusion [34],
an open-source implementation of DreamFusion that uses Stable Diffusion [23]
(rather than the non-public Imagen [26]) as the 2D prior. Given a prompt, for
Stable-DreamFusion, we extract the mesh from the optimized radiance field,
using its implementation out of the box; for our method, we first sample 4
images from Stable Diffusion conditioned on the same prompt and remove the
images which are likely to contain truncated instances (using the same
automatic pipeline as for pre-processing our virtual training images), and
then randomly select one image to feed into our category-specific model to get
the mesh. We show the generated 3D contents, with their corresponding prompts,
in Fig. 15. While Stable-DreamFusion takes $30\text{-}60$ minutes to produce
the mesh, our method uses less than $10$ seconds ($5$ seconds for generating
and pre-processing images and $1$ second for a single forward pass to predict
the shape). In addition, users have more control over the synthesized 3D asset
when using our method, as our model can be conditioned on images of the same
category; by contrast, even to get a different mesh with (Stable-)DreamFusion,
one has to tweak the prompt, change the random seed, and/or modify the
guidance scale and wait for another $30\text{-}60$ minutes.

###  A.4 Failure Cases

As shown in Fig. 16, our method is more likely to predict wrong viewpoints
when the animal instance in the input image is facing away from the camera. We
hypothesize that this is due to two key reasons. First, Stable Diffusion
sometimes generates anatomically incorrect images (examples are shown in Fig.
17), especially when the instance is facing away. Second, even with our
engineered prompts, instances facing away from the camera are still
underrepresented in the virtual training set. In the case of extreme
articulation, the viewpoint and bone predictions are also less stable
similarly to MagicPony [41].

Figure 16: Failure Cases. Typical failure modes include incorrect viewpoint
prediction and inaccurate articulation.  Figure 17: Stable Diffusion Failure
Cases. Stable Diffusion can generate anatomically wrong images. We observe
that this is typically correlated with which is particularly significant for
instances facing away.

##  Appendix B Additional Technical Details

###  B.1 Virtual Multi-View Supervision

Computing the SDS gradient, Eq. 2 in the main paper, has a relatively large
memory footprint and requires a halved batch size, which deteriorates training
stability. We found it helpful in practice to apply SDS guidance every second
iteration. Specifically, at every second iteration, camera positions
$\tilde{v}$ are randomly sampled in spherical coordinates, with an elevation
angle $\phi_{\text{cam}}\in\left[-10^{\circ},90^{\circ}\right]$, an azimuth
angle $\theta_{\text{cam}}\in\left[0^{\circ},360^{\circ}\right]$, and a
distance from the origin in $\left[9,11\right]$. For reference, we initialize
the prior shape as an ellipsoid with axis lengths 2.1, 2.1 and 1.05. We found
that randomly sampling the camera field of view (as in DreamFusion) did not
improve results. Instead of sampling point lights (as in DreamFusion), we
assume a distant directional light following MagicPony, and sample a random
light direction
$\tilde{l}\sim\mathcal{N}({\tilde{v}}/{\norm{\tilde{v}}},\textbf{I})$, where
$\tilde{v}\in\mathbb{R}^{3}$ is the sampled camera viewpoint. This sampling
strategy, first used in DreamFusion, ensures that the side of the object
facing towards the camera is mostly illuminated.

We render $256\times 256$ images and, like in DreamFusion, randomly alternate
between a shaded texture, texture-less shading, and albedo-only texture
(_i.e_., no lighting). We also use the same classifier-free guidance (with a
classifier guidance strength of 100) as it improves the regularity of the
learned shape.

###  B.2 Optional Texture Finetuning

We follow the implementation of MagicPony [41] for the underlying articulated
3D representation. In particular, MagicPony represents the appearance of an
object using a single feature vector and is unable to model high-fidelity
details in the input image. Following [41], for the visualizations with
texture rendering, we fine-tune the appearance network for 100 iterations on
each test image to obtain sharper textures. This step is optional but results
in a more faithful texture at a modest additional cost (the 3D shape
prediction takes 0.5 seconds and the optional texture finetuning requires up
to 30 seconds, vs. hours of distillation approaches like DreamFusion).
Notably, these textures still generalize to the unobserved parts of the object
owing to the underlying symmetric neural field representation.

###  B.3 Segmentation Masks

After generating the images $I$, we follow MagicPony and obtain masks using
PointRend [12], which is trained on COCO [17]. Unlike MagicPony, instead of
focusing on a single category (_i.e_., horse), our method works on a variety
of quadruped animal categories, some of which (_i.e_., pig) do not exist in
COCO. In practice, we find PointRend also produces reasonably good masks for
pigs when asked to detect its training categories (_e.g_., cow, elephant,
sheep, or horse), despite the semantic mismatch. We, therefore, use the mask
with the highest confidence score from detections of these categories.

###  B.4 Automatic Object Segmentation via Cross-attention

We also test our model on unreal “fictional” objects, such as steampunk cows,
generated by Stable Diffusion. Since PointRend cannot detect such objects in
general, we adapt a technique based on cross-attention originally developed
for interpreting Stable Diffusion [35], and automatically obtain reasonably
accurate masks without manual supervision. This method produces attribution
heatmaps for a given word in the textual prompt used to generate the image. We
condition the generation of the attribution heatmap on the name of the object
we are generating (_e.g_. ‘cow’, ‘horse’, ‘sheep’, _etc_.), and threshold the
generated heatmap to obtain an initial segmentation map, which is then refined
using the classic GrabCut algorithm [25]. This simple unsupervised
segmentation method can also replace PointRend in our training pipeline as
demonstrated in Fig. 18.

Figure 18: Qualitative Results Obtained from Automatic Object Segmentation on Cows. Despite using segmentation masks automatically obtained via cross-attention, our model can still produce 3D shapes that are faithful to the input images and realistic when rendered from novel views.  Figure 19: Training Images Generated with Stable Diffusion. Additional examples of our generated training images (after cropping).  Table 4: Effect of prompt engineering (% of back-views $\uparrow$, % of unsuitable images $\downarrow$<EMAIL_ADDRESS>in keypoint transfer $\uparrow$). | horse | cow | sheep
---|---|---|---
_a[V]_ | 3 / 26 / 0.215 | 1 / 29 / 0.185 | 3 / 30 / 0.170
_a photograph of a single[V]_ | 1 / 25 / 0.230 | 1 / 22 / 0.193 | 3 / 5 / 0.183
_+,[V] is walking away from the camera_ (ours) |  17 / 4 / 0.425 |  10 / 10 / 0.402 |  23 / 4 / 0.361

###  B.5 Removal of Truncated Images

Stable Diffusion sometimes generates portraits or truncated images of an
instance, not showing its full body and hence not useful for our purpose of
learning a statistical model of an articulated object category. We filter out
such images automatically with a simple heuristic: if any of the four borders
of a generated image, with a 10-pixel margin, contain more than 25% of masked
pixels, we consider the image to be truncated and exclude it from the training
dataset.

###  B.6 Prompt Design

To generate training images using Stable Diffusion, we needed to create an
appropriate textual prompt. Starting with a simple prompt, a [V] (where [V]
represents the word for an animal category, such as horse, cow, or sheep), we
iteratively adapted the prompt until we observed a reasonable quality of
generated images, displaying good viewpoint coverage. This process took about
eight iterations. The final prompt is as follows: a photograph of a single
[V], [V] is walking away from the camera. Additionally, we utilize the
following negative prompt: wrong anatomy, animal is lying on the ground,
lying, dead, black and white photography, human, person. This single prompt,
in conjunction with the negative prompt, generalizes effectively across all
categories. We show examples of generated images in Fig. 19. Note this
engineered prompt is only used for data generation. For SDS gradient
computation, we condition Stable Diffusion simply on the prompt a [V] without
negative prompts.

We also assess the effectiveness of this prompt design, considering both the
percentage of generated back-views of an animal, the percentage of unsuitable
images (e.g. cartoons, truncated parts) and the end-to-end performance on the
keypoint transfer metric in Tab. 4. We start with the initial naive prompt [V]
and we gradually increase the complexity by each row. Each added component
improves the quality of generated data. Most importantly, our final prompt
significantly increases the number of back-views.

###  B.7 Implementation Details

Here we provide additional implementation details not covered in the main
text. We employ the open-source HuggingFace Diffusers library [37] for the
Stable Diffusion implementation. We use Stable Diffusion version 1.5, as we
observed that version 2.1 more frequently results in incorrect anatomies. When
generating training images with Stable Diffusion, we set the guidance scale to
10 and perform 50 inference steps using half-precision floating point. The SDS
loss used during training, Eq. (2) in the main text, is weighted with $1\times
10^{-4}$. The complete training takes 20 hours for 120 iterations on a single
NVIDIA A40 GPU.

##  Appendix C Datasets

###  C.1 3D Articulated Animals Dataset

Our dataset consists of two models per category (male and female), with
featuring 1-2 textures, created by a professional 3D artist. We have also
collected a set of 28 HDRI backgrounds used to simulate both the background
and lighting. The sources of the random HDRI backgrounds are listed in Tab. 5.
We render $1k$ images per category, using a random model from the category and
with randomised animation frames, viewpoints, and HDRI backgrounds with
randomly sampled azimuth orientation.

Table 5: 3D Articulated Animals Dataset HDRI Files URLs

https://polyhaven.com/a/alps_field
---
https://polyhaven.com/a/belfast_farmhouse
https://polyhaven.com/a/belfast_open_field
https://polyhaven.com/a/belfast_sunset
https://polyhaven.com/a/bismarckturm_hillside
https://polyhaven.com/a/clarens_midday
https://polyhaven.com/a/drackenstein_quarry
https://polyhaven.com/a/etzwihl
https://polyhaven.com/a/evening_meadow
https://polyhaven.com/a/farm_sunset
https://polyhaven.com/a/forest_grove
https://polyhaven.com/a/graveyard_pathways
https://polyhaven.com/a/green_point_park
https://polyhaven.com/a/je_gray_02
https://polyhaven.com/a/je_gray_park
https://polyhaven.com/a/kloofendal_43d_clear
https://polyhaven.com/a/lilienstein
https://polyhaven.com/a/meadow
https://polyhaven.com/a/meadow_2
https://polyhaven.com/a/promenade_de_vidy
https://polyhaven.com/a/resting_place
https://polyhaven.com/a/resting_place_2
https://polyhaven.com/a/rosendal_mountain_midmorning
https://polyhaven.com/a/rosendal_park_sunset
https://polyhaven.com/a/rural_asphalt_road
https://polyhaven.com/a/rural_crossroads
https://polyhaven.com/a/scythian_tombs_2

###  C.2 ImageNet Images

In order to train MagicPony on data without manual curation, we automatically
collect images from ImageNet [4] for synsets corresponding to our categories.
We then apply the same pre-processing procedure as for our Stable Diffusion
generated data to obtain segmentation masks and remove truncated objects. This
method allows us to obtain $18.4k$, $6.6k$, and $7.7k$ images for horse, cow,
and sheep respectively.


the optimality gap defined as $100\%\times(UB-LB)/UB$. Thus, the former
quantity is equal to the number of times that the upper bounding problem in
line 14 of Algorithm 8 is solved. We observe that although the larger class of
instances with $T=24$ shifts is difficult to solve using either method, our
proposed algorithm is able to solve more instances and with a smaller
optimality gap, with an average speedup (in total run time) of roughly 1.75.
Finally, although not shown in this table, the number of outer-level
iterations for $T=24$ never exceeded 10 for either method and for any value of
$k$. This is because the solution of these instances is limited more by the
lower bounding problem (2) in the column-and-constraint generation algorithm,
than by the computation of worst-case parameter realizations.

##  6 Conclusions

This paper presented new Lagrangian dual formulations for two-stage robust
optimization problems with binary-valued uncertain data. Their crucial
features are the use of apropriately defined penalty functions that move all
uncertainties from the constraints to the objective function using a finite
scalar-valued Lagrange multiplier. The proposed formulations satisfy strong
duality properties even in the presence of discrete first- and second-stage
decisions, uncertain objective functions, and in the absence of relatively
complete recourse. We showed how they can be used to efficiently calculate
worst-case parameter realizations that can be readily integrated in existing
high-level exact algorithms. Notably, they circumvent many of the numerical
issues that plague existing methods, including smaller MILP formulations and a
lack of reliance on decision-independent bounds on dual variables. The
proposed methods can exploit problem structure in applications where the
binary parameters switch on or off constraints, and they can potentially
enable existing algorithms that address uncertainty only in the objective
function to solve also problems with uncertainty-affected constraints.
Numerical experiments on a diverse set of network design, facility location
and staff rostering test instances showed that they can provide computational
speedups over existing methods that have been tailored for those specific
applications.

We believe that this work can be extended along several directions. First, we
need to build extensions of the proposed Lagrangian dual formulations and
associated exact methods for problems featuring random recourse or mixed-
integer uncertainties (including single-stage problems with mixed-integer
decisions). Looking beyond exact algorithms, it would also be interesting to
assess whether the proposed ideas can be useful in other contexts as well,
including but not limited to approximations schemes for multi-stage robust
optimization problems, and worst-case uncertainty quantification for nonlinear
models.

## References

  * [1] Maria Albareda-Sambola, Elena Fernández, and Francisco Saldanha-da Gama.  The facility location problem with bernoulli demands.  Omega, 39(3):335–345, 2011.
  * [2] Amir Ardestani-Jaafari and Erick Delage.  Linearized robust counterparts of two-stage robust optimization problems with applications in operations management.  INFORMS Journal on Computing, 2020.
  * [3] Ayşe Arslan and Boris Detienne.  Decomposition-based approaches for a class of two-stage robust binary optimization problems.  INFORMS Journal on Computing, 2021.
  * [4] Josette Ayoub and Michael Poss.  Decomposition for adjustable robust linear optimization subject to uncertainty polytope.  Computational Management Science, 13(2):219–239, 2016.
  * [5] Luiz André Barroso, Jimmy Clidaras, and Urs Hölzle.  The datacenter as a computer: An introduction to the design of warehouse-scale machines.  Synthesis lectures on computer architecture, 8(3):1–154, 2013.
  * [6] Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski.  Robust optimization.  Princeton university press, 2009.
  * [7] Aharon Ben-Tal, Alexander Goryashko, Elana Guslitzer, and Arkadi Nemirovski.  Adjustable robust solutions of uncertain linear programs.  Mathematical programming, 99(2):351–376, 2004.
  * [8] Dimitris Bertsimas and Constantine Caramanis.  Finite adaptability in multistage linear optimization.  IEEE Transactions on Automatic Control, 55(12):2751–2766, 2010\.
  * [9] Dimitris Bertsimas and Iain Dunning.  Multistage robust mixed-integer optimization with adaptive partitions.  Operations Research, 64(4):980–998, 2016.
  * [10] Dimitris Bertsimas and Angelos Georghiou.  Design of near optimal decision rules in multistage adaptive mixed-integer optimization.  Operations Research, 63(3):610–627, 2015.
  * [11] Dimitris Bertsimas and Melvyn Sim.  The price of robustness.  Operations research, 52(1):35–53, 2004.
  * [12] Hans-Georg Beyer and Bernhard Sendhoff.  Robust optimization–a comprehensive survey.  Computer methods in applied mechanics and engineering, 196(33-34):3190–3218, 2007.
  * [13] Daniel Bienstock and Abhinav Verma.  The $N-k$ Problem in Power Grids: New Models, Formulations, and Numerical Experiments.  SIAM Journal on Optimization, 20(5):2352–2380, 2010.
  * [14] Christoph Buchheim and Jannis Kurtz.  Min–max–min robust combinatorial optimization.  Mathematical Programming, 163(1-2):1–23, 2017.
  * [15] Constantine Caramanis, Shie Mannor, and Huan Xu.  14 robust optimization in machine learning.  Optimization for machine learning, page 369, 2012.
  * [16] Xin Chen and Yuhan Zhang.  Uncertain linear programs: Extended affinely adjustable robust counterparts.  Operations Research, 57(6):1469–1482, 2009.
  * [17] Chun Cheng, Yossiri Adulyasak, and Louis-Martin Rousseau.  Robust facility location under demand uncertainty and facility disruptions.  Omega, 103:102429, 2021.
  * [18] Antonio J Conejo and Xuan Wu.  Robust optimization in power systems: a tutorial overview.  Optimization and Engineering, pages 1–23, 2021.
  * [19] Jing Dong and Rouba Ibrahim.  Managing supply in the on-demand economy: Flexible workers, full-time employees, or both?  Operations Research, 68(4):1238–1264, 2020.
  * [20] Iain Dunning, Joey Huchette, and Miles Lubin.  Jump: A modeling language for mathematical optimization.  SIAM Review, 59(2):295–320, 2017.
  * [21] Frank J Fabozzi, Petter N Kolm, Dessislava A Pachamanova, and Sergio M Focardi.  Robust portfolio optimization and management.  John Wiley & Sons, 2007.
  * [22] Wei Feng, Yi Zhang, Gang Rong, and Yiping Feng.  Finite adaptability in data-driven robust optimization for production scheduling: a case study of the ethylene plant.  Industrial & Engineering Chemistry Research, 58(16):6505–6518, 2019.
  * [23] Virginie Gabrel, Cécile Murat, and Aurélie Thiele.  Recent advances in robust optimization: An overview.  European Journal of Operational Research, 235(3):471–483, 2014\.
  * [24] Francis Garuba, Marc Goerigk, and Peter Jacko.  A comparison of discrete and polyhedral uncertainty sets for robust network design.  arXiv preprint arXiv:2003.10507, 2020.
  * [25] Angelos Georghiou, Daniel Kuhn, and Wolfram Wiesemann.  The decision rule approach to optimization under uncertainty: methodology and applications.  Computational Management Science, 16(4):545–576, 2019.
  * [26] Fred Glover.  Improved linear integer programming formulations of nonlinear integer problems.  Management Science, 22(4):455–460, 1975.
  * [27] Marc Goerigk, Jannis Kurtz, and Michael Poss.  Min–max–min robustness for combinatorial problems with discrete budgeted uncertainty.  Discrete Applied Mathematics, 285:707–725, 2020.
  * [28] Selcuk Goren and Ihsan Sabuncuoglu.  Robustness and stability measures for scheduling: single-machine environment.  IIE Transactions, 40(1):66–83, 2008.
  * [29] Gurobi Optimization, LLC.  Gurobi Optimizer Reference Manual, 2021.
  * [30] Grani A Hanasusanto and Daniel Kuhn.  Conic programming reformulations of two-stage distributionally robust linear programs over wasserstein balls.  Operations Research, 66(3):849–869, 2018.
  * [31] Grani A Hanasusanto, Daniel Kuhn, and Wolfram Wiesemann.  K-adaptability in two-stage robust binary programming.  Operations Research, 63(4):877–891, 2015.
  * [32] Hossein Hashemi Doulabi, Patrick Jaillet, Gilles Pesant, and Louis-Martin Rousseau.  Exploiting the structure of two-stage robust optimization models with exponential scenarios.  INFORMS Journal on Computing, 33(1):143–162, 2021.
  * [33] Rabih A Jabr, Izudin Džafić, and Bikash C Pal.  Robust optimization of storage investment on transmission networks.  IEEE Transactions on Power Systems, 30(1):531–539, 2014.
  * [34] Ruiwei Jiang, Minseok Ryu, and Guanglin Xu.  Data-driven distributionally robust appointment scheduling over wasserstein balls.  arXiv preprint arXiv:1907.03219, 2019.
  * [35] Ruiwei Jiang, Siqian Shen, and Yiling Zhang.  Integer programming approaches for appointment scheduling with random no-shows and service durations.  Operations Research, 65(6):1638–1656, 2017.
  * [36] Nicolas Kämmerling and Jannis Kurtz.  Oracle-based algorithms for binary two-stage robust optimization.  Computational Optimization and Applications, 77(2):539–569, 2020\.
  * [37] Hervé Kerivin and A Ridha Mahjoub.  Design of survivable networks: A survey.  Networks: An International Journal, 46(1):1–21, 2005.
  * [38] Arie MCA Koster, Manuel Kutschka, and Christian Raack.  Robust network design: Formulations, valid inequalities, and computations.  Networks, 61(2):128–149, 2013.
  * [39] Panos Kouvelis and Gang Yu.  Robust discrete optimization and its applications, volume 14.  Springer Science & Business Media, 2013.
  * [40] Hongqiang Harry Liu, Srikanth Kandula, Ratul Mahajan, Ming Zhang, and David Gelernter.  Traffic engineering with forward fault correction.  In Proceedings of the 2014 ACM Conference on SIGCOMM, pages 527–538, 2014.
  * [41] Mengshi Lu, Lun Ran, and Zuo-Jun Max Shen.  Reliable facility location design under uncertain correlated disruptions.  Manufacturing & Service Operations Management, 17(4):445–455, 2015\.
  * [42] Logan R Matthews, Chrysanthos E Gounaris, and Ioannis G Kevrekidis.  Designing networks with resiliency to edge failures using two-stage robust optimization.  European Journal of Operational Research, 279(3):704–720, 2019\.
  * [43] Areesh Mittal, Can Gokalp, and Grani A Hanasusanto.  Robust quadratic programming with mixed-integer uncertainty.  INFORMS Journal on Computing, 32(2):201–218, 2020.
  * [44] Tri-Dung Nguyen and Andrew W Lo.  Robust ranking and portfolio optimization.  European Journal of Operational Research, 221(2):407–416, 2012\.
  * [45] Krzysztof Postek and Dick den Hertog.  Multistage adjustable robust mixed-integer optimization via iterative splitting of the uncertainty set.  INFORMS Journal on Computing, 28(3):553–574, 2016.
  * [46] Soroosh Shafieezadeh-Abadeh, Peyman Mohajerin Esfahani, and Daniel Kuhn.  Distributionally robust logistic regression.  arXiv preprint arXiv:1509.09259, 2015.
  * [47] J Cole Smith, Mike Prince, and Joseph Geunes.  Modern network interdiction problems and algorithms.  In Handbook of combinatorial optimization, pages 1949–1987. Springer New York, 2013.
  * [48] Anirudh Subramanyam.  Robust optimization of vehicle routing problems under uncertainty.  PhD thesis, Carnegie Mellon University, 2018.
  * [49] Anirudh Subramanyam, Chrysanthos E Gounaris, and Wolfram Wiesemann.  K-adaptability in two-stage mixed-integer robust optimization.  Mathematical Programming Computation, 12(2):193–224, 2020.
  * [50] Anirudh Subramanyam, Frank Mufalli, José M Laínez-Aguirre, Jose M Pinto, and Chrysanthos E Gounaris.  Robust multiperiod vehicle routing under customer order uncertainty.  Operations Research, 69(1):30–60, 2021.
  * [51] Anirudh Subramanyam, Mohamed El Tonbari, and Kibaek Kim.  Data-driven two-stage conic optimization with zero-one uncertainties.  arXiv preprint arXiv:2001.04934, 2020.
  * [52] Aurélie Thiele, Tara Terry, and Marina Epelman.  Robust linear optimization with recourse.  Technical report, Lehigh University and University of Michigan, 2009.
  * [53] Zidong Wang, Fuwen Yang, Daniel W. C. Ho, and Xiaohui Liu.  Robust $h_{\infty}$ control for networked systems with random packet losses.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 37(4):916–924, 2007.
  * [54] Guanglin Xu and Samuel Burer.  A copositive approach for two-stage adjustable robust optimization with uncertain right-hand sides.  Computational Optimization and Applications, 70(1):33–59, 2018\.
  * [55] Lei Xu and Tsan Sheng Adam Ng.  A robust mixed-integer linear programming model for mitigating rail transit disruptions under uncertainty.  Transportation Science, 54(5):1388–1407, 2020.
  * [56] Wei Yuan, Jianhui Wang, Feng Qiu, Chen Chen, Chongqing Kang, and Bo Zeng.  Robust optimization-based resilient distribution network planning against natural disasters.  IEEE Transactions on Smart Grid, 7(6):2817–2826, 2016.
  * [57] Bo Zeng and Long Zhao.  Solving two-stage robust optimization problems using a column-and-constraint generation method.  Operations Research Letters, 41(5):457–461, 2013.
  * [58] Long Zhao and Bo Zeng.  An exact algorithm for two-stage robust optimization with mixed integer recourse problems.  Technical report, University of South Florida, 2012.
  * [59] Jianzhe Zhen, Dick Den Hertog, and Melvyn Sim.  Adjustable robust optimization via fourier–motzkin elimination.  Operations Research, 66(4):1086–1100, 2018.

The submitted manuscript has been created by UChicago Argonne, LLC, Operator
of Argonne National Laboratory (“Argonne”). Argonne, a U.S. Department of
Energy Office of Science laboratory, is operated under Contract No. DE-
AC02-06CH11357. The U.S. Government retains for itself, and others acting on
its behalf, a paid-up nonexclusive, irrevocable worldwide license in said
article to reproduce, prepare derivative works, distribute copies to the
public, and perform publicly and display publicly, by or on behalf of the
Government. The Department of Energy will provide public access to these
results of federally sponsored research in accordance with the DOE Public
Access Plan (http://energy.gov/downloads/doe-public-access-plan).