,

# One Noise to Rule Them All:
Learning a Unified Model of Spatially-Varying Noise Patterns

Arman Maesumi<EMAIL_ADDRESS>0000-0001-7898-8061 Brown
UniversityProvidenceRIUSA ,  Dylan Hu<EMAIL_ADDRESS>0009-0001-9456-9470 ,
Krishi Saripalli<EMAIL_ADDRESS>0009-0000-5224-2632 Brown
UniversityProvidenceRIUSA ,  Vladimir G. Kim<EMAIL_ADDRESS>0000-0002-3996-6588 Adobe ResearchUSA ,  Matthew Fisher<EMAIL_ADDRESS>0000-0002-8908-3417 Adobe ResearchUSA ,  Sören Pirk<EMAIL_ADDRESS>0000-0003-1937-9797 CAUKielGermany and  Daniel Ritchie
<EMAIL_ADDRESS>0000-0002-8253-0069 Brown UniversityProvidenceRIUSA

(2024; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Procedural noise is a fundamental component of computer graphics pipelines,
offering a flexible way to generate textures that exhibit “natural” random
variation. Many different types of noise exist, each produced by a separate
algorithm. In this paper, we present a single generative model which can learn
to generate multiple types of noise as well as blend between them. In
addition, it is capable of producing spatially-varying noise blends despite
not having access to such data for training. These features are enabled by
training a denoising diffusion model using a novel combination of data
augmentation and network conditioning techniques. Like procedural noise
generators, the model’s behavior is controllable via interpretable parameters
plus a source of randomness. We use our model to produce a variety of visually
compelling noise textures. We also present an application of our model to
improving inverse procedural material design; using our model in place of
fixed-type noise nodes in a procedural material graph results in higher-
fidelity material reconstructions without needing to know the type of noise in
advance. Open-sourced materials can be found at
https://armanmaesumi.github.io/onenoise/

Procedural noise, texture synthesis, texture acquisition, deep generative
model

††copyright: acmlicensed††journalyear: 2024††doi: xxxxxxx.xxxxxxx††conference:
Make sure to enter the correct conference title from your rights confirmation
emai; Article 1; Publication date: July 2024††booktitle: ACM Trans. on
Graphics††submissionid: 739††ccs: Computing methodologies Texturing††ccs:
Computing methodologies Neural networks Figure 1. Our method enables the
synthesis of a wide range of noise patterns with spatially-varying
characteristics. Here we show the flexibility of our unified noise model,
allowing one to art-direct their noise in a granular fashion. Our model
creates semantically meaningful interpolations between noise configurations;
above we see the Siggraph logo written with hay fibers that are nested inside
of Damascus steel striations – the scale and distortion of the steel pattern
naturally interpolates into a denser pattern before transitioning into fibers.
We also show renderings of a clay shader that incorporates our spatially-
varying noise patterns. The first three images make use of class-interpolated
noise, the final image uses parameter-interpolated noise. Please zoom into the
figures for full visual detail.

##  1\. Introduction

Procedural noise has long been a fundamental building block in computer
graphics, serving as a versatile tool for modeling fine, naturalistic details
in a range of applications. It finds widespread use in creating albedo
textures, bump and normal maps, terrain height fields, and density fields for
volumetric phenomena such as clouds and smoke. There exists a zoo of
algorithms for generating such noise patterns, for instance Perlin, Worley,
and Gabor noise (Perlin, 1985; Worley, 1996; Lagae et al., 2009), along with
systems for composing and transforming these noise patterns to synthesis more
complex shaders (e.g. Blender, Adobe Substance 3D Designer, and other material
editors (Blender, 2023; Adobe, 2023a)).

Despite the advancement of these tools, a fundamental limitation remains in
the design process: the necessity to make discrete choices regarding the types
of noise to employ. This constraint poses challenges in creative design, where
the ideal noise type may not be immediately present among the provided “zoo”,
or where desired visual characteristics fall between the behaviors of existing
noise types. Designers might envision patterns exhibiting different noise
characteristics in various regions, transitioning smoothly without abrupt or
artificial blending. The standard technique for blending between noise
types—alpha blending—often yields unsatisfactory results, where features do
not naturally interpolate in the image (see Figure 2).

This limitation also complicates inverse design tasks, such as recovering a
procedural representation (i.e. a material node graph) of a texture from
observed data. Such graphs often use multiple noise generator nodes; if the
types of these nodes aren’t known in advance, the search space for the inverse
design problem grows combinatorially, and the need to search over discrete
possible noise types limits the applicability of gradient-based optimization.

In this paper, we address these challenges by introducing a method for
learning a continuous space of spatially-varying noise patterns from data that
lacks any spatially-varying observations. We leverage a denoising diffusion
probabilistic model (DDPM) (Ho et al., 2020) equipped with a spatially-varying
conditioning module, which is trained via a novel application of CutMix data
augmentation (Yun et al., 2019). Our method enables the generation of noise
that can vary smoothly across the canvas, offering a greater level of control
over the resulting noise pattern. Just as with traditional procedural noise
generators, the model can be controlled by changing a set of interpretable
parameters or by changing the input source of randomness (i.e. random seed).
Importantly, our method can generate noise images at any requested image size
(beyond the size of the training data) and can also produce seamlessly
tileable images.

We evaluate our model by using it to generate a variety of noise blends and
spatially-varying noise patterns (including spatial variation driven by image-
based masks). We also present a user interface for painting spatially-varying
noise patterns, videos of this interface are included in the supplemental
material. Finally, we show an application of our method to inverse procedural
material design, demonstrating that using our model as a noise generator node
in a material graph to be optimized can result in higher-fidelity material
reconstructions without needing to know the particular noise type for that
node in advance.

In summary, our contributions are:

  1. (1)

A generative model which can learn to generate spatially-varying noise
patterns.

  2. (2)

A training scheme for the model using a novel application of CutMix
augmentation, allowing the model to learn to generate spatially-varying noise
patterns without having any spatially-varying training data.

  3. (3)

An application of our model to inverse procedural material design, showing
improved material reconstructions without pre-specified discrete noise types.

Figure 2. Our model produces noise patterns whose characteristics (i.e. scale,
ripples, distortion, etc.) interpolate naturally, creating seamless and
coherent transitions. By contrast, traditional alpha-blending results in
images with overlapping features, inconsistent feature opacity, and a lack of
sensible transitions between the noise characteristics.

##  2\. Related Work

##### Procedural noise

“Noise is the random number generator of computer graphics” (Lagae et al.,
2010). Procedural noise functions collectively form a body of techniques for
algorithmically synthesizing patterns that mimic the randomness and
irregularities in natural phenomena. These noise functions have become
foundational to many applications within computer graphics, facilitating the
synthesis of natural materials (Perlin, 1985; Dorsey and Hanrahany, 2006),
terrain (Génevaux et al., 2013; Galin et al., 2019; Van Der Linden et al.,
2013; Fournier et al., 1982), and motion (Hinsinger et al., 2002).

There are a wide variety of noise functions that synthesize distinct patterns,
for example, Perlin (Perlin, 1985, 2002), Wavelet (Cook and DeRose, 2005), and
Gabor noise (Lagae et al., 2009). Such noises are often composed and
transformed using various primitive operations (e.g. scalar functions and
domain warping), forming patterns that are colloquially referred to as _noise_
in computer graphics — while these resulting patterns may not strictly fall
under the formal definition of noise (Ebert et al., 2002), we continue to use
the term in this established, albeit technically imprecise sense, throughout
the paper. A selection of various noise patterns is shown in Figure 5. For
further background, we refer to the survey by Lagae et al. (2010).

##### Neural parametric texture synthesis

As noise is a central building block for textures, our work is related to
methods for texture synthesis. Texture synthesis has a deep, decades-long
history, a full discussion of which is outside the scope of this paper. The
most relevant methods to our work are parametric texture synthesis methods
which use neural networks as their parametric model. Most of this prior work
focuses on _example-based texture synthesis_ , i.e. generating textures which
are similar in some statistical sense to an input exemplar. Some methods do
this by optimizing an image such that the features produced by passing it
through a certain pre-trained neural network match those produced by the
exemplar image (Gatys et al., 2015; Sendik and Cohen-Or, 2017). Others train
feedforward neural networks to directly generate texture similar to an input
exemplar (Ulyanov et al., 2016; Rodriguez-Pardo and Garces, 2023; Li and Wand,
2016; Zhou et al., 2018, 2023a). The closest work to ours is a generative
adversarial network (GAN) that does not perform example-based synthesis, but
rather learns a continuous latent space of textures (Bergmann et al., 2017).
This model supports interpolation between textures, but it is not controllable
by a combination of interpretable noise parameters and a source of randomness
(its latent space entangles texture “style” with spatial randomness). We
initially experimented with a GAN-based architecture and found that it
struggled to capture a data distribution with the many different modes
represented by a set of highly distinct noise types. Prior work suggests that
diffusion models such as ours fare better at modeling such distributions (Xiao
et al., 2022).

We also acknowledge the role of non-parametric methods in this field. These
approaches synthesize textures by using the structures present within an
image, without relying on a learned parametric model. For instance, Matusik et
al. (2005) frame the texture interpolation problem through _morphable models_
(Jones and Poggio, 1998), and define a space of interpolatable textures via a
simplicial complex induced by a texture similarity metric. ImageMelding
(Darabi et al., 2012), on the other hand, proposes a patch-based method that
smoothly “in-fills” regions of a texture via a regularized screened Poisson
equation. For a more comprehensive review of this body of work, we refer to
the survey by Raad et al. (2018).

##### Inverse procedural material design

Recovering procedural representations of materials from observations (i.e.
photographs) has become an increasingly promising area of research. Material
graphs—the procedural representation of choice—feature nodes that generate,
transform, and filter various signals that ultimately become spatially-varying
material maps (SVBRDFs). Authoring and editing these material graphs is not
only time consuming, but also requires mastery of complex software (Adobe,
2023a; Blender, 2023). _Inverse procedural material design_ aims to alleviate
this difficulty by recovering a material graph from a given image. Recent
works tackle this problem in various ways, either by 1) directly predicting
the parameters of a given material graph (Hu et al., 2019; Guo et al., 2020),
2) optimizing the continuous parameters of the graph via gradient-based
optimization (Shi et al., 2020; Hu et al., 2022), or 3) by synthesizing a
material graph from scratch (Guerrero et al., 2022; Zhou et al., 2023b).

Most relevant to our work are learned differentiable proxies, which serve as
continuous relaxations for gradient-based optimization strategies to the
inverse design problem. Hu et al. (2022) train several GAN-based generative
models to act as differentiable proxies for _pattern_ generators (i.e. a brick
tiling generator node). However, these proxies are deterministic, meaning they
cannot model noise functions, and critically, each pattern is represented by a
separate generative model. Our unified noise DDPM is able to simultaneously
capture many non-deterministic noise functions in the same continuous space,
acting as a relaxation over an entire space of unique noise functions.

Figure 3. Our DDPM is trained using CutMix data augmentation. We first
transform the current data sample (highlighted in blue) by cutting and
patching together a set of other random samples from the dataset, resulting in
a training image $\mathbf{x}_{0}$. The noise parameters for each image patch
are passed to an MLP, which projects the parameter sets into an embedding
space that encodes both the noise type (class) and the noise parameters. The
resulting feature vectors are tiled to form a feature grid, which is used as a
conditioning signal in the U-Net’s SPADE blocks, as outlined in Section 3.1.
Figure 4. At inference time, we query our network using artificially
constructed feature grids, enabling a flexible way to synthesize spatially-
varying noise patterns. Here we embed four sets of noise parameters,
pictorially shown as one of four colors. We blend the feature vectors using
bilinear interpolation, creating a smoothly-varying feature grid, which our
U-Net is able to transform into a Voronoi noise pattern with non-uniform scale
and distortion characteristics.

##  3\. Method

Given a collection of procedural noise textures sampled from various
parametric noise generating functions, we seek to learn a conditional
generative model that captures the noise textures given their accompanying
parameter configurations, enabling the synthesis of a wide range of noises
from a single universal function. Our conditional generative model will be
formulated in a way that facilitates synthesizing spatially-varying noise
textures (i.e. spatial blends between the categorical noise type as well as
noise parameters), even though the training collection does not contain such
samples. That is, we assume our given collection of noise samples only
exhibits globally uniform semantic properties, as shown in Figure 5.

We make use of a denoising diffusion probabilistic model (DDPM) that
incorporates a spatially-varying conditioning mechanism. The model is trained
using a data augmentation scheme that regulates the behavior of this
conditioning module, ensuring that it behaves appropriately when given
granular conditioning signals. In the following sections, we first detail the
spatially-varying conditioning module, followed by the aforementioned data
augmentation strategy. We defer details pertaining to our DDPM training to
Section 4.

###  3.1. Spatially-varying conditioning

At training time, our conditioning signal is given by two vectors,
$\mathbf{f}_{c}$, a class embedding that encodes the categorical label of a
particular noise type, and $\mathbf{f}_{p}$, a list of parameters that
determine semantic properties of the generated noise image (i.e. scale,
distortion, etc.). Providing these parameters as explicit input to the model
allows interpretable user control; it also disentangles control over the
“style” of the generated noise (via these parameters) from the “seed” (i.e.
stochastic component) of the noise functions via the initial Gaussian noise
provided to the DDPM.

To facilitate synthesis of noise maps with spatially-varying properties, we
employ spatially-adaptive denormalization (SPADE) (Park et al., 2019). We
first map the class and parameter vectors into a shared feature space with a
small MLP, call the resulting feature vector $\mathbf{f}$. The feature vector
is tiled into a spatial grid, $\mathbf{F}$, of shape $|\mathbf{f}|\times
H\times W$, where $H$, $W$ are the height and width respectively. This feature
grid is used as input to the SPADE module. Following Dhariwal et al. (2021),
we modulate the intermediate feature maps of our network’s Group Normalization
layers, making the SPADE conditioning function

$\begin{gathered}S(\mathbf{h},\mathbf{F})=\gamma(\mathbf{F})\odot\mathrm{GroupNorm}(\mathbf{h})+\beta(\mathbf{F})\\\
\mathrm{GroupNorm}_{g}(\mathbf{h})=\frac{\mathbf{h}_{g}-\mu_{g}}{\sqrt{\sigma_{g}^{2}+\epsilon}}\end{gathered}$

where $\gamma,\beta$ are convolutional layers that transform the feature grid
into element-wise scales and shifts respectively, and $\mathbf{h}$ is the
incoming activations of the previous layer. The scales and shifts act on the
output of a Group Normalization (Wu and He, 2018) block, which is computed
using the mean, $\mu$, and standard deviation, $\sigma$, of each group of
channels $g$.

Our training data only contains noise textures that exhibit spatially
_uniform_ properties, meaning that all feature entries in the grid
$\mathbf{F}$ are identical at training time. However, at inference time we may
query the network with an artificially constructed feature grid – for
instance, we can spatially blend between multiple feature vectors to produce a
conditioning signal that smoothly interpolates between noise types and noise
parameters, as shown in Figure 4.

####  3.1.1. Spherical class embeddings 111The following subsection introduces
an enhancement to our original method, identified post-publication, which
improves results in many cases.

As previously mentioned, the network is conditioned on a class embedding,
$\mathbf{f}_{c}$, that is learned for each noise type during training. We find
that regularizing this embedding space leads to substantial improvements when
interpolating between classes. In particular, we penalize the deviation of the
norms of these embeddings from a target norm. For embedding vectors of
dimension $d$ that are initialized from
$\mathbf{f}_{c}\sim\mathcal{N}^{d}(0,\mathbf{I}_{d})$, we define the target
norm, $T$, as the expected squared norm of a d-dimensional Gaussian vector
according to the identity,

$T_{d}^{n}\coloneq\mathbb{E}_{\,\mathbf{f}_{c}\sim\mathcal{N}^{d}(0,\mathbf{I}_{d})}\left[||\mathbf{f}_{c}||_{2}^{n}\right]=2^{n/2}\frac{\Gamma((d+n)/2)}{\Gamma(d/2)},$

where $\Gamma$ denotes Euler’s gamma function. During training we then
penalize the embeddings according to

(1) $\mathcal{L}_{\mathrm{reg}}=\frac{1}{|C|}\sum_{c\in
C}(\|\mathbf{f}_{c}\|_{2}^{2}-T_{d}^{2})^{2},$

this imposes a _spherical structure_ on the embeddings, which means that
interpolating between them can be done via spherical linear interpolation
(Shoemake, 1985). We find that imposing this structure greatly improves
texture blending between classes. This is likely because the MLP (which acts
on these embeddings) can now learn a smoother mapping into the final
conditioning signal. As mentioned in note 1, this regularization was
identified post-publication; hence, our primary results do not make use of
this change. We demonstrate its effectiveness in Figure 15 specifically.

###  3.2. Enhancing localized conditioning

Ideally, locally modifying the conditioning feature grid, $\mathbf{F}$, within
a confined region should correspondingly alter the generated noise texture
solely within that region. However, our empirical observations revealed that
such localized adjustments in the conditioning signal often lead to global (or
near-global) changes in the resultant noise textures. This phenomenon can be
primarily attributed to the architectural design of the employed U-Net neural
network – in particular, its bottleneck shape in combination with having many
convolutional layers causes the receptive field of each output pixel to be
relatively wide. This enables the conditioning signal to spatially propagate
across the majority of the canvas, which is undesirable for the quality and
controllability of our synthesized noise textures.

To address this, we incorporate a modified version of CutMix data augmentation
(Yun et al., 2019) into our training procedure. CutMix, a technique that is
used to enhance performance in image classification tasks, involves creating
synthetic training examples by stitching together axis-aligned crops of
different dataset samples. Notably, the application of CutMix has been
predominantly in classification tasks, with its use in generative models being
relatively unexplored and confined to augmenting the performance of GAN
discriminators (Schonfeld et al., 2020; Huang et al., 2021). In our context,
we apply CutMix by combining noise textures and their corresponding
conditioning feature grids, as illustrated in Figure 3. This approach not only
enriches the diversity of our training dataset, but also imparts a crucial
capability to our model: the ability to respond correctly to spatially
localized conditioning signals.

We train our network with CutMix data augmentation applied with a probability
of $0.5$, i.e. for half of the training samples we train without applying
CutMix. When augmenting a training image, we sample one to four (uniformly at
random) auxiliary noise textures from the dataset, which are then randomly
cropped (_cut_) into rectangular patches with a randomly sampled rotation
$\theta\sim\mathcal{U}(0,2\pi)$. Note that only the crop mask is rotated, not
the texture itself. The resulting _mixed_ sample is then a composition of a
base noise texture, and one to four texture patches. It is important that all
sampled patches belong to unique noise types, otherwise the resulting training
image would be invalid, that is, the image would no longer belong to the
distribution that we are trying to capture.

Figure 5. Samples from our noise dataset, procured from Adobe Substance 3D
Designer. We sample 18 noises along with a variety of their parameters. Note
that our dataset does not contain samples with spatially-varying properties.
Figure 6. We compare our method to a neural texture synthesizer, PSGAN (2017),
as well as a non-parametric texture blending method, Image Melding (2012). In
the case of Image Melding, the first and last quarter of the image are given,
only the remaining interior region is filled in. Both prior methods suffer
from artifacts and repeated visual details, whereas our method is able to
blend smoothly while synthesizing novel details throughout the canvas. We note
that PSGAN produces anisotropic features that are not characteristic of the
data distribution (e.g. horizontal streaks in bottom left example).

##  4\. Implementation Details

##### Noise dataset

We procure a dataset of ${\sim}1.2$ million noise images, covering 18 unique
noise functions, along with dense samplings of their respective parameter sets
(listed in Appendix Table 1). In particular, we sample the following noise
functions from Adobe Substance 3D Designer: cells 1, cells 4, voronoi,
microscope view, grunge galvanic small, liquid, bnw spots 1, grunge leaky
paint, grunge rust fine, grunge map 002, grunge map 005, grunge damas, messy
fibers 3, perlin, gaussian, clouds 1, clouds 2, clouds 3. A preview of our
dataset is shown in Figure 5. For each noise type, we deterministically sample
$\num{16384}$ parameter sets using the low-discrepancy Halton number sequence,
ensuring better coverage over the space of parameters. Each parameter set is
sub-sampled four times, i.e. we query the noise functions with four different
seeds for each parameter set, resulting in $\num{65536}$ image samples per
noise type, and $\num{1179648}$ samples across the entire dataset. We sample
at a resolution of $512\times 512$px.

##### Network architecture

Our U-Net model is composed of three spatial levels in the encoder and
decoder, each level (including the bottleneck) consists of two ResNet blocks
that are conditioned on the diffusion timestep, $t$, and the block’s
respective SPADE module. The two conditioning signals are composed together as

$\gamma_{2}(\mathbf{Z})\odot(\gamma_{1}(t)\cdot\mathrm{GroupNorm}(\mathbf{h})+\beta_{1}(t))+\beta_{2}(\mathbf{Z})$

where $\gamma_{1},\beta_{1}$ operate on $t$ and $\gamma_{2},\beta_{2}$ belong
to a SPADE block. Note that the former functions produce scalar quantities,
whereas the latter functions produce spatial modulation maps. Time values are
first encoded using sinusoidal positional encoding. The noise parameter MLP is
defined by three layers of size 128. The parameter vector $\mathbf{f}_{p}$ is
a concatenation of all noise parameters present in the dataset, with entries
zeroed out when not applicable. In total our U-Net has ${\sim}5.1$ million
parameters.

##### Training

Following Song et al. (2020) we formulate the U-Net as a noise predictor
$\epsilon_{\theta}(\mathbf{x}_{t},t,\mathbf{f}_{c},\mathbf{f}_{p})$ making the
training objective

(2)
$\mathcal{L}=\mathbb{E}_{\epsilon\sim\mathcal{N}(0,1),t\sim\mathcal{U}(0,1)}\|\epsilon-\epsilon_{\theta}(\mathbf{x}_{t},t,\mathbf{f}_{c},\mathbf{f}_{p})\|_{2}+\lambda\mathcal{L}_{\mathrm{reg}}$

We make use of offset noise (Guttenberg, 2023), which replaces the noise
sampling, $\epsilon\sim\mathcal{N}(0,1)$, with a slightly modified
distribution, $\mathcal{N}(0.1\delta,1)$, where $\delta\sim\mathcal{N}(0,1)$.
We find that offset noise greatly helps our network resolve noise types that
are extremely dark or bright (see Appendix Figure 16). We also note that more
principled alternatives to offset noise are now available (Lin et al., 2024).
In practice the timestep sampling, $t\sim\mathcal{U}(0,1)$, is discretized –
we make use of a standard number of timesteps, i.e. $\num{1000}$. We employ
the cosine-beta schedule proposed by Nichol et al. (2021). The spherical
embedding regularization loss is weighted by $\lambda=0.02$ in Equation 2.
Training is done in PyTorch; we use the AdamW optimizer (Loshchilov and
Hutter, 2017) with a learning rate of $8\cdot 10^{-5}$,
$\beta_{1}=0.9,\beta_{2}=0.99$, and a weight decay of $0.01$. We train on 8
NVIDIA RTX 3090 GPUs for ${\sim}\num{300000}$ optimization steps with batches
of size 128. Noise images are downsampled to $256\times 256$px resolution for
training.

##### Inference details & performance.

Our model can perform 80 diffusion steps per second on a single NVIDIA RTX
3090 GPU at $256\times 256$ resolution at full precision (fp32), scaling
quadratically with the resolution. We use the DDIM sampler proposed by Song et
al. (2020) – unless otherwise specified, our noise figures and visualizations
use 30 diffusion steps. To produce tileable noise maps, we modify all conv2d
layers in our network to use a _circular_ padding mode, making the canvas
topologically toroidal.

##  5\. Results and evaluation

##### Spatially and temporally varying noise

We demonstrate the capabilities of our noise generator by synthesizing noise
patterns using an assortment of blending maps, shown in Figures 1 and 7. Our
noise generator is able to be conditioned flexibly, allowing one to generate
large noise patterns with diverse patterns and blends between them. We
additionally show that our model is able to synthesize continuously-varying
noise by interpolating the conditioning maps and/or diffusion noise (see
Figure 11 and supp. video). Finally, in Figure 6 we qualitatively compare our
spatially-varying noise to a GAN-based texture synthesis method, PSGAN (2017),
as well as a non-parametric texture blending method, ImageMelding (2012).

Figure 7. Several instances of spatially-varying noise, which arise from
various blended feature grids. Blend maps are shown in the top right of each
panel. Figure 8. We demonstrate the utility of our method in material graph
optimization. Given a procedural material graph and a target photograph, our
noise generative model serves as a useful prior over the space of noise
functions, facilitating the recovery of non-trivial patterns that are present
in the target photos, and improving the baseline result given by MATch (2020).
Two similarity scores are reported for each result with respect to the target
image: MATch’s feature-based texture similarity metric (left) and LPIPS (2018)
(right). Lower is better for both scores.

##### Quantitative evaluation

We evaluate our noise generator’s spatially-uniform noise textures against the
ground truth textures from Adobe Substance 3D Designer using Frechet Inception
Distance (FID), which measures the distributional similarity between synthetic
samples and the data distribution (Heusel et al., 2017). We sample 20000
synthetic images for each noise type and evaluate the FID of each noise type
separately. FID scores are reported in Appendix Table 2, alongside analogous
scores for PSGAN. Our method achieves a mean FID of 20.9 and median of 13.1,
while PSGAN scores 99.2 and 87.5 respectively (lower is better). We note that,
for many noise types, PSGAN suffers from mode collapse, which causes virtually
all of the synthesized results to have similar visual features and hence its
FID is severely impacted. As mentioned in Section 2, GAN-based methods
struggle to faithfully capture our dataset due to the presence of many
disjoint data modes (Xiao et al., 2022). Finally, we show additional results
with modified U-Net architectures in Appendix C.

##### Inverse material graph design

We incorporate our noise generator into the differentiable material graph
library MATch (Shi et al., 2020). Given a target photo, we select a template
material graph from a selection of 88 graphs using the MATch-provided texture
descriptors. In the graph, we replace one noise generator node with our model,
and expose optimizable parameters for the class vector $\mathbf{f}_{c}$,
parameter vector $\mathbf{f}_{p}$, and the diffusion model’s latent noise
$\mathbf{z}$. We include an L1 regularization on the former vectors to
encourage sparsity in the optimized parameters. In Figure 8 we show that our
noise generator provides a valuable prior over the space of noise functions,
allowing end-to-end optimization to recover more accurate reconstructions of
the target photo. We additionally show results of editing operations on an
optimized material graph in Figure 9. Target photos are sampled from the
Glossy dataset by Zhou et al. (2023b). Further details for this application
are in Appendix A.3.

Figure 9. After optimization, we can easily modify the resulting procedural
material graphs. Here a marble material graph is optimized (first row), and
edited in various ways (second row). We show the results of editing the noise
colorization, resampling our diffusion noise, and modifying our model’s
conditioning inputs. Figure 10. Our model is capable of synthesizing noise
maps at resolutions that far exceed the training resolution, thus facilitating
one to paint “infinite” canvases. Here we see a $2048\times 2048$px Damascus
steel noise pattern with intricate fine details, which was generated via a
single diffusion process. Please zoom into the figure for full details. Figure
11. Our model is able to synthesize continuously-varying textures via
interpolation of the conditioning parameters and diffusion noise (analogous to
the “seed” in a traditional noise function). Here we show three noise types
with 1) parameters interpolating (horizontal axis), and 2) the noise seed
interpolating (vertical axis). We also note that, when using the same noise
seed across different noise types, we observe that similar structures appear
in the output – we suggest zooming out to see this effect clearly. Noise
interpolation videos are included in supplemental material.

##### Tileability and texture size agnosticism

Our model is able to produce noise images that are tileable, which is crucial
for many downstream graphics applications. Additionally, we are able to
synthesize noise at arbitrary sizes (within computational limits). In tandem,
these two properties are highly desired, as it allows one to generate large
tileable images that do not contain excessive repeated visual patterns, which
is a common problem with tileable images. We demonstrate these capabilities in
Figure 12, where we show tileable grunge map 5 and microscope noise images, as
well as the same noises synthesized on canvases that are twice as large,
eliminating the repeated visual content. Additionally, in Figure 10 we show a
much larger Damascus pattern that fills a $2048\times 2048$ canvas – we
highlight that fine-grained details are not at this size.

Figure 12. Our model is able to produce seamless tileable images, as shown by
the green regions, which are tiled into 2-by-2 noise maps. We also support
synthesizing noise on an arbitrary canvas, avoiding noticeable repeated
patterns, which is a common problem with tileable textures.

##### CutMix augmentation

We demonstrate the affect of our modified CutMix data augmentation strategy by
training models with: no augmentation, augmentation with only one patch
(CutMix $n=1$), and augmentation with one to four patches sampled uniformly at
random (_CutMix $n\in[1,4]$_). In Figure 13, we see that removing the
augmentation strategy severely hinders our model’s ability to respond to the
conditioning signal in a local manner, producing noise patterns whose
characteristics are not well-pronounced. Intuitively, this is because the
conditioning signal becomes “blurred” as it propagates through the network’s
layers. Due to the lack of meaningful ground-truth data, we qualitatively
observe that Cutmix $n=1$ and Cutmix $n=4$ produce similar results, with the
latter being able to interpolate noise characteristics more smoothly in some
cases.

Figure 13. Ablation study of CutMix data augmentation. Without CutMix, the
U-Net fails to resolve noise maps that contain non-uniform characteristics.
For instance, in row 3 of the _class blending_ panel, the network cannot
adequately blend between noise classes, causing the galvanic noise pattern to
disappear entirely. We include models with one and four applications of
CutMix, as detailed in Section 5. Finally, the _uniform_ panel (no blending)
acts as a control group – as expected, all outputs are similar.

##  6\. Conclusion

Figure 14. Our method is unable to smoothly blend between some noise types. In
the configurations above, we see artifacts in the transition regions.

We presented a new method for learning a single generative model for a wide
range of procedural noise types. The continuous space of noise patterns
learned by our model allows for smooth blending between noise types; the
characteristics of the output noise are still controllable by interpretable
parameters provided as input to the model. In addition, our novel approach for
spatially-varying conditioning allows the model to generate spatially-varying
blends between different noises despite never having seen such data at
training time. We used this model to produce a variety of visually compelling
textures, and we showed a proof-of-concept application of how it can be used
to improve inverse procedural material design.

Our model is not without limitations. Not all pairs of noise types can be
interpolated well: if the geometric features of two noise patterns are too
dissimilar, the intermediate regions of a blend between them can look blurry
or otherwise visually awkward (see Fig. 14). It may be possible to reduce
these artefacts with further improvements to our CutMix-based data
augmentation (e.g. non-rectangular patches; adaptive patch sampling to focus
more training time on “difficult” transitions). But to some extent, such
behavior is inevitable, and finding good noise pairs for interpolation is an
artistic decision. It may be possible to leverage the structure of our noise
embedding space to suggest good candidates for blending. Finally, diffusion
models are known to model low-density modes of the data distribution less
accurately (Song, 2021; Um et al., 2024); in Appendix B we demonstrate our
model on additional noise types, and observe that some low-density modes of
these noise functions are poorly captured. More careful data sampling
techniques may be needed when the desired noise distribution exhibits such
modes; e.g. by sampling some regions of the parameter space more frequently.

Procedural material authoring tools such as Adobe Substance Designer often
have deterministic pattern generators in addition to stochastic noise
generators. Could our model include these in its learned space of blendable
output? The deterministic nature of these patterns makes them an unnatural fit
for generation via denoising diffusion model; modifications to our model
and/or training procedure would be necessary to achieve this goal.

It would also be interesting to explore more techniques for interactively
creating spatially-vary noise textures. For example, one might consider an
interaction metaphor where the user places ‘droplets’ of noise on a canvas,
and those droplets spread an interact with one another by e.g. solving a
diffusion equation.

Finally, the application to inverse procedural material design that we
presented is an early proof-of-concept; much more could be done in this
direction. Distilling our model into a one-shot diffusion model (Liu et al.,
2023) would make it tractable to replace _every_ noise generator node in a
procedural material graph with our model. Provided other graph operations are
differentiable, this would open up the possibility of solving for the
structure of a material graph via continuous optimization: initializing the
graph to be over-complete, assigning weights to edges, and imposing a sparsity
prior on edge weights. Similar approaches have been successfully employed for
inferring procedural representations of 3D shapes (Kania et al., 2020; Ren et
al., 2021). In addition, allowing the optimization of spatially-varying noise
generators within material graphs might allow for recovering smaller, easier-
to-use graphs while still retaining the benefits of interpretable parameters.

Figure 15. Noise interpolations that arise from our spherically regularized
embedding space, as described in Section 3.1.1. The blending between noise
types appears organic and is largely free of the artifacts noted in Figure 14.

###### Acknowledgements.

This material is based upon work that was supported by the National Science
Foundation Graduate Research Fellowship under Grant No. 2040433. Part of this
work was done while Arman Maesumi was an intern at Adobe Research.

## References

  * (1)
  * Adobe (2023a) Adobe. 2023a.  Adobe Substance 3D Designer.  https://www.adobe.com/products/substance3d-designer.html.
  * Adobe (2023b) Adobe. 2023b.  Adobe Substance 3D Documentation.  https://helpx.adobe.com/substance-3d-designer/substance-compositing-graphs/nodes-reference-for-substance-compositing-graphs/node-library.html.
  * Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017.  Wasserstein Generative Adversarial Networks. In _Proceedings of the 34th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 70)_ , Doina Precup and Yee Whye Teh (Eds.). PMLR, 214–223.  https://proceedings.mlr.press/v70/arjovsky17a.html
  * Bergmann et al. (2017) Urs Bergmann, Nikolay Jetchev, and Roland Vollgraf. 2017.  Learning Texture Manifolds with the Periodic Spatial GAN. In _Proceedings of the 34th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 70)_ , Doina Precup and Yee Whye Teh (Eds.). PMLR, 469–477.  https://proceedings.mlr.press/v70/bergmann17a.html
  * Blender (2023) Blender. 2023.  Blender.  https://www.blender.org.
  * Cook and DeRose (2005) Robert L Cook and Tony DeRose. 2005.  Wavelet noise.  _ACM Transactions on Graphics (TOG)_ 24, 3 (2005), 803–811.
  * Darabi et al. (2012) Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan B Goldman, and Pradeep Sen. 2012.  Image Melding: Combining Inconsistent Images using Patch-based Synthesis.  _ACM Transactions on Graphics (TOG) (Proceedings of SIGGRAPH 2012)_ 31, 4, Article 82 (2012), 82:1–82:10 pages.
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol. 2021.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ 34 (2021), 8780–8794.
  * Dorsey and Hanrahany (2006) Julie Dorsey and Pat Hanrahany. 2006.  Modeling and rendering of metallic patinas.  In _ACM SIGGRAPH 2006 Courses_. 2–es.
  * Ebert et al. (2002) David Ebert, Kenton Musgrave, Darwyn Peachey, Ken Perlin, and Steve Worley. 2002.  _Texturing And Modeling. A Procedural Approach_ (3 ed.).  Morgan Kaufmann.
  * Fournier et al. (1982) Alain Fournier, Don Fussell, and Loren Carpenter. 1982.  Computer rendering of stochastic models.  _Commun. ACM_ 25, 6 (1982), 371–384.
  * Galin et al. (2019) Eric Galin, Eric Guérin, Adrien Peytavie, Guillaume Cordonnier, Marie-Paule Cani, Bedrich Benes, and James Gain. 2019.  A review of digital terrain modeling. In _Computer Graphics Forum_ , Vol. 38. Wiley Online Library, 553–577.
  * Gatys et al. (2015) Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2015.  Texture Synthesis Using Convolutional Neural Networks. In _NeurIPS_.
  * Génevaux et al. (2013) Jean-David Génevaux, Éric Galin, Eric Guérin, Adrien Peytavie, and Bedrich Benes. 2013.  Terrain generation using procedural models based on hydrology.  _ACM Transactions on Graphics (TOG)_ 32, 4 (2013), 1–13.
  * Guerrero et al. (2022) Paul Guerrero, Miloš Hašan, Kalyan Sunkavalli, Radomír Měch, Tamy Boubekeur, and Niloy J Mitra. 2022.  MatFormer: A generative model for procedural materials.  _arXiv preprint arXiv:2207.01044_ (2022).
  * Guo et al. (2020) Yu Guo, Miloš Hašan, Lingqi Yan, and Shuang Zhao. 2020.  A bayesian inference framework for procedural material parameter estimation. In _Computer Graphics Forum_ , Vol. 39. Wiley Online Library, 255–266.
  * Guttenberg (2023) Nicholas Guttenberg. 2023.  Diffusion with Offset Noise.  (2023).
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ 30 (2017).
  * Hinsinger et al. (2002) Damien Hinsinger, Fabrice Neyret, and Marie-Paule Cani. 2002.  Interactive animation of ocean waves. In _Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation_. 161–166.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ 33 (2020), 6840–6851.
  * Hu et al. (2019) Yiwei Hu, Julie Dorsey, and Holly Rushmeier. 2019.  A novel framework for inverse procedural texture modeling.  _ACM Transactions on Graphics (ToG)_ 38, 6 (2019), 1–14.
  * Hu et al. (2022) Yiwei Hu, Paul Guerrero, Milos Hasan, Holly Rushmeier, and Valentin Deschaintre. 2022.  Node graph optimization using differentiable proxies. In _ACM SIGGRAPH 2022 conference proceedings_. 1–9.
  * Huang et al. (2021) Zhizhong Huang, Junping Zhang, Yi Zhang, and Hongming Shan. 2021.  DU-GAN: Generative adversarial networks with dual-domain U-Net-based discriminators for low-dose CT denoising.  _IEEE Transactions on Instrumentation and Measurement_ 71 (2021), 1–12.
  * Jones and Poggio (1998) Michael J Jones and Tomaso Poggio. 1998.  Multidimensional morphable models. In _Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271)_. IEEE, 683–688.
  * Kania et al. (2020) Kacper Kania, Maciej Zieba, and Tomasz Kajdanowicz. 2020.  UCSG-NET- Unsupervised Discovering of Constructive Solid Geometry Tree. In _Advances in Neural Information Processing Systems_.
  * Lagae et al. (2010) Ares Lagae, Sylvain Lefebvre, Rob Cook, Tony DeRose, George Drettakis, David S Ebert, John P Lewis, Ken Perlin, and Matthias Zwicker. 2010.  A survey of procedural noise functions. In _Computer Graphics Forum_ , Vol. 29. Wiley Online Library, 2579–2600.
  * Lagae et al. (2009) Ares Lagae, Sylvain Lefebvre, George Drettakis, and Philip Dutré. 2009.  Procedural noise using sparse Gabor convolution.  _ACM Transactions on Graphics (TOG)_ 28, 3 (2009), 1–10.
  * Li and Wand (2016) Chuan Li and Michael Wand. 2016.  Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks. In _Computer Vision – ECCV 2016_ , Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 702–716.
  * Lin et al. (2024) Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. 2024.  Common diffusion noise schedules and sample steps are flawed. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 5404–5411.
  * Liu et al. (2023) Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu. 2023.  InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation.  arXiv:2309.06380 [cs.LG]
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ (2017).
  * Matusik et al. (2005) Wojciech Matusik, Matthias Zwicker, and Frédo Durand. 2005.  Texture design using a simplicial complex of morphable textures.  _ACM Transactions on Graphics (TOG)_ 24, 3 (2005), 787–794.
  * Nichol and Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal. 2021.  Improved denoising diffusion probabilistic models. In _International Conference on Machine Learning_. PMLR, 8162–8171.
  * Park et al. (2019) Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019.  Semantic Image Synthesis with Spatially-Adaptive Normalization. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Perlin (1985) Ken Perlin. 1985.  An Image Synthesizer. In _Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’85)_. Association for Computing Machinery, New York, NY, USA, 287–296.  https://doi.org/10.1145/325334.325247
  * Perlin (2002) Ken Perlin. 2002.  Improving noise. In _Proceedings of the 29th annual conference on Computer graphics and interactive techniques_. 681–682.
  * Raad et al. (2018) Lara Raad, Axel Davy, Agnès Desolneux, and Jean-Michel Morel. 2018.  A survey of exemplar-based texture synthesis.  _Annals of Mathematical Sciences and Applications_ 3, 1 (2018), 89–148.
  * Ren et al. (2021) Daxuan Ren, Jianmin Zheng, Jianfei Cai, Jiatong Li, Haiyong Jiang, Zhongang Cai, Junzhe Zhang, Liang Pan, Mingyuan Zhang, Haiyu Zhao, et al. 2021\.  CSG-Stump: A Learning Friendly CSG-Like Representation for Interpretable Shape Parsing. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 12478–12487.
  * Rodriguez-Pardo and Garces (2023) Carlos Rodriguez-Pardo and Elena Garces. 2023.  SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps.  _IEEE Transactions on Visualization and Computer Graphics_ 29, 6 (jun 2023), 2914–2925.  https://doi.org/10.1109/TVCG.2022.3143615
  * Schonfeld et al. (2020) Edgar Schonfeld, Bernt Schiele, and Anna Khoreva. 2020.  A u-net based discriminator for generative adversarial networks. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 8207–8216.
  * Sendik and Cohen-Or (2017) Omry Sendik and Daniel Cohen-Or. 2017.  Deep Correlations for Texture Synthesis.  _ACM Trans. Graph._ 36, 5, Article 161 (jul 2017), 15 pages.  https://doi.org/10.1145/3015461
  * Shen et al. (2021) Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. 2021.  Efficient attention: Attention with linear complexities. In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_. 3531–3539.
  * Shi et al. (2020) Liang Shi, Beichen Li, Miloš Hašan, Kalyan Sunkavalli, Tamy Boubekeur, Radomir Mech, and Wojciech Matusik. 2020.  MATch: Differentiable Material Graphs for Procedural Material Capture.  _ACM Trans. Graph._ 39, 6 (Dec. 2020), 1–15.
  * Shoemake (1985) Ken Shoemake. 1985.  Animating rotation with quaternion curves. In _Proceedings of the 12th annual conference on Computer graphics and interactive techniques_. 245–254.
  * Song et al. (2020) Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020.  Denoising Diffusion Implicit Models.  _arXiv:2010.02502_ (October 2020).  https://arxiv.org/abs/2010.02502
  * Song (2021) Yang Song. 2021.  Generative Modeling by Estimating Gradients of the Data Distribution.  https://yang-song.net/blog/2021/score/.
  * Tricard et al. (2019) Thibault Tricard, Semyon Efremov, Cédric Zanni, Fabrice Neyret, Jonàs Martínez, and Sylvain Lefebvre. 2019.  Procedural phasor noise.  _ACM Transactions on Graphics (TOG)_ 38, 4 (2019), 1–13.
  * Ulyanov et al. (2016) Dmitry Ulyanov, Vadim Lebedev, Vedaldi Andrea, and Victor Lempitsky. 2016.  Texture Networks: Feed-forward Synthesis of Textures and Stylized Images. In _Proceedings of The 33rd International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 48)_ , Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, New York, USA, 1349–1357.
  * Um et al. (2024) Soobin Um, Suhyeon Lee, and Jong Chul Ye. 2024.  Don’t Play Favorites: Minority Guidance for Diffusion Models. In _The Twelfth International Conference on Learning Representations_.  https://openreview.net/forum?id=3NmO9lY4Jn
  * Van Der Linden et al. (2013) Roland Van Der Linden, Ricardo Lopes, and Rafael Bidarra. 2013.  Procedural generation of dungeons.  _IEEE Transactions on Computational Intelligence and AI in Games_ 6, 1 (2013), 78–89.
  * Worley (1996) Steven Worley. 1996.  A cellular texture basis function. In _Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’96)_. Association for Computing Machinery, New York, NY, USA, 291–294.  https://doi.org/10.1145/237170.237267
  * Wu and He (2018) Yuxin Wu and Kaiming He. 2018.  Group Normalization. In _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII_ _(Lecture Notes in Computer Science, Vol. 11217)_ , Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer, 3–19.  https://doi.org/10.1007/978-3-030-01261-8_1
  * Xiao et al. (2022) Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. 2022.  Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In _ICLR_.
  * Yun et al. (2019) Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019.  CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In _International Conference on Computer Vision (ICCV)_.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In _CVPR_.
  * Zhou et al. (2023b) Xilong Zhou, Milos Hasan, Valentin Deschaintre, Paul Guerrero, Yannick Hold-Geoffroy, Kalyan Sunkavalli, and Nima Khademi Kalantari. 2023b.  PhotoMat: A Material Generator Learned from Single Flash Photos. In _ACM SIGGRAPH 2023 Conference Proceedings_. 1–11.
  * Zhou et al. (2023a) Yang Zhou, Kaijian Chen, Rongjun Xiao, and Hui Huang. 2023a.  Neural Texture Synthesis with Guided Correspondence. In _Conference on Computer Vision and Pattern Recognition (CVPR)_. 18095–18104.
  * Zhou et al. (2018) Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2018.  Non-stationary Texture Synthesis by Adversarial Expansion.  _ACM Transactions on Graphics (Proc. SIGGRAPH)_ 37, 4 (2018), 49:1–49:13.

Noise function | Sampled parameters | Range
---|---|---
cells 4 | scale | $i[5,50]$
cells 1 | scale | $i[10,50]$
voronoi | scale | $f[5.0,15.0]$
| distortion intensity | $f[0.0,1.0]$
| distortion scale multiplier | $f[1.0,2.0]$
microscope view | scale | $i[25,64]$
| warp intensity | $f[0.0,1.0]$
bnw spots 1 | scale | $i[1,3]$
liquid | scale | $i[10,45]$
| warp intensity | $f[0.1,0.8]$
grunge galvanic small | crispness | $f[0.0,0.75]$
| dirt | $f[0.0,1.0]$
| micro distortion | $f[0.0,0.6]$
grunge leaky paint | leak intensity | $f[0.0,1.0]$
| leak scale | $i[1,8]$
| leak crispness | $f[0.0,0.8]$

Noise function | Sampled parameters | Range
---|---|---
grunge rust fine | base grunge contrast | $f[-0.3,0.3]$
| base warp intensity | $f[0.0,256.0]$
grunge damas | distortion | $f[0.0,1.0]$
| divisions | $i[4,16]$
| waves | $i[1,3]$
| details | $f[0.0,0.75]$
| rotation random | $f[0.05,1.0]$
grunge map 002 | N/A | N/A
grunge map 005 | N/A | N/A
messy fibers 3 | scale | $i[1,3]$
perlin | scale | $i[10,50]$
gaussian | scale | $i[10,50]$
clouds 1 | scale | $i[1,5]$
clouds 2 | scale | $i[1,5]$
clouds 3 | scale | $i[1,3]$

Table 1. We enumerate the noise functions that are sampled from Adobe
Substance 3D Designer, along with their parameters and accompanying ranges –
integer and real ranges are denoted by $i$ and $f$ respectively.

##  Appendix A Experimental and Implementation Details

| FID$\downarrow$ |  | FID$\downarrow$
---|---|---|---
Noise | PSGAN | Ours | Noise | PSGAN | Ours
cells 4 | $218.8$ | $33.6$ | rust fine | $88.5$ | $12.3$
cells 1 | $171.3$ | $2.4$ | damas | $56.1$ | $71.0$
voronoi | $149.3$ | $12.5$ | map 002 | $37.3$ | $13.7$
microscope | $133.1$ | $29.7$ | map 005 | $155.4$ | $34.5$
bnw spots 1 | $22.4$ | $4.4$ | fibers | $86.4$ | $34.6$
liquid | $163.7$ | $38.0$ | perlin | $47.8$ | $4.8$
galvanic small | $79.2$ | $24.9$ | gaussian | $45.2$ | $1.7$
leaky paint | $155.1$ | $44.3$ | clouds 1 | $20.4$ | $1.5$
clouds 3 | $38.4$ | $2.9$ | clouds 2 | $110.8$ | $9.0$
Mean | $99.2$ | $20.9$ | Median | $87.5$ | $13.1$
Table 2. We compare our FID scores for each noise type alongside PSGAN. The
mean and median of all values are shown as well.

###  A.1. Noise dataset details

We include a table enumerating all sampled noise types, the parameters that we
sample, as well as the parameter ranges (see Table 1). In general, we choose
to include any parameters that lead to noticeable changes in the resulting
noise, with exceptions to parameters that simply act as color correction (i.e.
grunge map 005’s _contrast_ parameter). We note that, despite their names,
some of these noises correspond to those that belong to graphics literature;
for instance, cells 4, cells 1, and voronoi are variants of Worley noise
(Worley, 1996). For more details about these noises, please refer to the Adobe
Substance documentation (2023b).

The conditioning vector $\mathbf{f}_{p}$ contains an entry for each unique
noise parameter – we treat identical parameter names as separate, with the
exception of scale, which is treated as a single entry in the vector.
Parameters are independently normalized to the range $[0,1]$ before being
passed to our conditioning MLP. Below we show many samples of our model’s
outputs with all listed noise parameters being sampled randomly (see Figs. 19,
20, 21, 22 and 23).

Figure 16. Ablation of offset noise. Without offset noise, the network
occasionally fails to synthesize noise images with extreme intensity
distributions (i.e. intensely dark and bright images). We show representative
examples above.

###  A.2. PSGAN Baseline

We slightly modify the PSGAN (2017) architecture for the results shown in
Figure 6. Since PSGAN is a purely unconditional generative model (i.e. it has
no class or parameter conditioning beyond randomized latent code inputs), we
endow the PSGAN model with SPADE conditioning blocks that are identical to the
ones used throughout our method. We made the discriminator slightly larger to
compensate for the added parameters in the generator. Finally, we use the WGAN
loss (Arjovsky et al., 2017). The remaining parts of the network and training
are largely kept as-is from the open implementation. In total the generator
has 30 million parameters.

###  A.3. Inverse Material Design Details

We expose three parameters to the MATch differentiable material graph
optimizer; a soft-class vector $\mathbf{v}_{c}$, the parameter vector
$\mathbf{f}_{p}$, and the diffusion Gaussian noise image $\mathbf{z}$. The
soft-class vector represents a list of $[0,1]$ values that are used to index
into our class embeddings – this amounts to taking a convex combination over
the class embeddings. During optimization, we apply the SoftMax function to
$\mathbf{v}_{c}$ to ensure the values are valid. In order to encourage
sparsity, we include an L1 regularization term on both $\mathbf{v}_{c}$ and
$\mathbf{f}_{p}$, using a weight of $0.1$ for both L1 terms in the final
optimization objective. Additionally, we include a scheduled temperature
parameter $\tau$ into the SoftMax operator by performing element-wise division
of $\mathbf{v}_{c}$ by $\tau$,
$\text{softmax}(v,t)=\frac{e^{v_{i}/t}}{\sum_{j=1}^{K}e^{v_{j}/t}}$. The
temperature is initialized to $0.25$ and is updated at every optimization step
via the update rule $\tau^{\prime}\leftarrow\tau\cdot 0.97$. Finally we clamp
$\tau$ to be at minimum $0.01$. This scheduled temperature forces the
optimization to “hone in” on a single class. We use a learning rate of $0.01$
for all graphs. After $\tau=0.01$ we perform a warm-restart of the optimizer
and add noise to the exposed parameters to avoid local minima.

##  Appendix B Training with additional noise types

We train our model on two additional noise types that are commonly used in
graphics literature: Phasor noise (2019) and Gabor noise (2009). We utilize
the same data sampling method as mentioned in Section 4. The parameters that
we sampled for both noise types, as well as their ranges, are enumerated in
Table 3. We use the released implementations for both noise types to accrue
our training data. Examples of spatially-varying images using parameter
interpolation and class interpolation for both Phasor and Gabor noise are
shown in Figs. 17 and 18.

We note that our model exhibits an FID of 93.6 and 129.3 on Phasor and Gabor
noise respectively, which is notably higher than the FID scores for other
noise types. Upon inspection of our model’s outputs, we see that some
parameter configurations are not well captured by our training. For example,
Gabor noise exhibits a significantly different visual appearance when its
principal frequency and kernel width are very small; however, since this
appearance is only captured by a narrow subset of the parameter space, the
dataset thus contains much fewer of such samples. One drawback of diffusion
models is that they model low-density regions of the data distribution less
accurately (compared to higher-density regions) (Song, 2021; Um et al., 2024),
and hence our performance is worse in such situations. More careful data
sampling methods may be needed when the desired noise distribution exhibits
low-density modes; e.g. by sampling some regions of the parameter space more
frequently.

Noise function | Sampled parameters | Range
---|---|---
phasor noise | principal frequency, $F$ | $f[8.0,32.0]$
| num cells | $i[1,12]$
| phasor density | $f[0.3,0.5]$
| factor angle spread, $\theta$ | $f[0.0,1.0]$
gabor noise | principal frequency, $F$ | $f[0.02,0.08]$
| kernel width, $\alpha$ | $f[0.01,0.35]$
| kernel orientation, $\omega$ | $f[0.0,2\pi]$
Table 3. Parameters and sampling ranges for additional noise types. Figure 17.
Examples of our model’s Phasor noise. From top to bottom: isotropic Phasor to
anisotropic Phasor (parameter interpolation), anisotropic Phasor to Messy
Fibers, and anisotropic Gabor to anisotropic Phasor (class interpolation).
Figure 18. Examples of our model’s Gabor noise. From top to bottom: Gabor to
Gabor (parameter interpolation), Gabor to Microscope View, and Gabor to Grunge
Rust Fine (class interpolation).

##  Appendix C Model architectures

The U-Net model detailed in Section 4 has just ${\sim}5.1$ million parameters
and is constructed using two downsampling blocks, a bottleneck block, and two
upsampling blocks. Each block contains two ResNet sub-blocks. An additional
ResNet sub-block is placed at the end of the network. We use channel
dimensions of 32 and 64 for the outer blocks, and 128 for the bottleneck. The
ResNet sub-blocks are globally conditioned on the diffusion time $t$, and
spatially conditioned (via SPADE) by 128-dim noise embeddings. We will refer
to this model as Model-XS (extra small) below.

We train two additional model architectures to evaluate the potential
performance of larger and more complex models. Model-S is identical to Model-
XS, with the exception of added _linear attention_ layers inside of each block
(Shen et al., 2021), and the addition of an extra set of
downsampling/upsampling blocks. Similarly, Model-M is identical to Model-S,
with the exception of larger channel dimensions. In Table 4 we summarize these
architectures as well as their FID scores and inference performance. The
models were trained for the same number of optimization steps following the
details in Section 4.

For our inverse material design application, we found that using Model-XS was
most suitable due to the added memory cost of attention layers. For
consistency, we used this model for all figures in the main text; however, in
applications that do not require such light-weight networks, the larger models
may be suitable.

Architecture | Block dimensions | Attention | Num. Parameters | Mean FID$\downarrow$ | Median FID$\downarrow$ | Steps per second$\uparrow$
---|---|---|---|---|---|---
Model-XS | (32, 64, 128) | ✗ | 5.1M | 20.9 | 13.1 | 79.5/s
Model-S | (32, 64, 64, 128) |  $\mathtt{heads}=4$, $\mathtt{dim}=32$ | 6.5M | 14.0 | 10.8 | 28.6/s
Model-M | (64, 128, 128, 256) |  $\mathtt{heads}=4$, $\mathtt{dim}=32$ | 22.5M | 10.3 | 10.4 | 20.1/s
Table 4. Three model architectures with varying parameter counts and design
choices are shown. The _Block dimensions_ column shows the dimension counts
for the downsampling blocks followed by the bottleneck block (the networks are
symmetric; i.e. the upsampling blocks follow the same dimensions in reverse
order). Our primary model (first row) is significantly faster than the other
models, but compromises slightly on FID. Figure 19. Random samples of our
model’s cells 4, cells 1, and voronoi noises at $256\times 256$ resolution.
Figure 20. Random samples of our model’s microscope view, bnw spots1, and
liquid noises at $256\times 256$ resolution. Figure 21. Random samples of our
model’s grunge galvanic small, grunge leaky paint, and grunge rust fine noises
at $256\times 256$ resolution. Figure 22. Random samples of our model’s grunge
damas, grunge map 002, and grunge map 005 noises at $256\times 256$
resolution. Figure 23. Random samples of our model’s messy fibers 3, perlin,
and gaussian noises at $256\times 256$ resolution. Figure 24. Random samples
of our model’s clouds 1, clouds 2, and clouds 3 noises at $256\times 256$
resolution.