# A Multi-Implicit Neural Representation for Fonts

Pradyumna Reddy1 Zhifei Zhang2 Matthew Fisher2 Hailin Jin2 Zhaowen Wang2 Niloy
J. Mitra1,2 1University College London 2Adobe Research

###### Abstract

Fonts are ubiquitous across documents and come in a variety of styles. They
are either represented in a native vector format or rasterized to produce
fixed resolution images. In the first case, the non-standard representation
prevents benefiting from latest network architectures for neural
representations; while, in the latter case, the rasterized representation,
when encoded via networks, results in loss of data fidelity, as font-specific
discontinuities like edges and corners are difficult to represent using neural
networks. Based on the observation that complex fonts can be represented by a
superposition of a set of simpler occupancy functions, we introduce multi-
implicits to represent fonts as a permutation-invariant set of learned
implicit functions, without losing features (e.g., edges and corners).
However, while multi-implicits locally preserve font features, obtaining
supervision in the form of ground truth multi-channel signals is a problem in
itself. Instead, we propose how to train such a representation with only local
supervision, while the proposed neural architecture directly finds globally
consistent multi-implicits for font families. We extensively evaluate the
proposed representation for various tasks including reconstruction,
interpolation, and synthesis to demonstrate clear advantages with existing
alternatives. Additionally, the representation naturally enables glyph
completion, wherein a single characteristic font is used to synthesize a whole
font family in the target style.

##  1 Introduction

Fonts constitute the vast majority of documents. They come in a variety of
styles spanning a range of topologies representing a mixture of smooth curves
and sharp features. Although fonts vary significantly across families, they
remain stylistic coherent across the different alphabets/symbols inside any
chosen font family.

Fonts are most commonly stored in a vector form (e.g., a collection of spline
curves) that is compact, efficient, and can be resampled at arbitrary
resolutions without loss of features. This specialized representation,
however, prevents the adaptation of many deep learning setups optimized for
regular structures (e.g., image grids). In order to avoid this problem, custom
deep learning architectures have been developed for directly producing vector
output but they typically require access to ground truth vector data for
training. This is problematic: first, collecting sufficient volume of vector
data for training is non trivial; and second, vector representations are not
canonical (i.e., same fonts can be represented by different sequence of
vectors), which in turn requires hand-coded network parameters to account for
varying number of vector instructions.

Figure 1: Multi-implicit neural representation for high fidelity font
reconstruction and generation. Note that while ours perform similar to
ImageVAE at lower/training resolution, the advantage of ours become clear when
we test at higher resolution (e.g, how corners continue to be preserved).

An alternate approach is to rasterize vectorized fonts and simply treat them
as images. While this readily allows using image-based deep learning methods,
the approach inherits problems of discretized representations leading to
aliasing artifacts and loss of sharp features. Further, the resultant images
are optimized for particular resolutions and cannot be resampled without
introducing additional artifacts.

Recently, implicit representation has emerged as an attractive representation
for deep learning as, once trained, they can resampled at different
resolutions without introducing artifacts. Unfortunately, implicit
representations (e.g., signed distance fields) for fonts are often too complex
to be represented accurately by neural networks. As a results, although deep
implicits work for simple fonts, they can fail to retain characteristic
features (i.e., edges and corners) for complex fonts.

Drawing inspiration from multi-channel SDFs Chlumskỳ et al. (2018), we observe
that complex fonts can be expressed as a composition of multiple simple
regions. For example, a local corner can be represented as a suitable
composition of two half-planes, each of which can easily be individually
encoded as deep implicit functions. We build on this idea by hypothesizing
that complex fonts can also be encoded as suitable composition of global
implicit functions. We call such a representation to be a multi-implicit
neural representation, as each (global) implicit function is neurally encoded.

Figure 2: Corner preserving capability of different sampling methods.

Thus, multi-implicits provide a simple representation that is amenable for
processing by neural networks and the output fidelity remains comparable, even
under resampling, to vector representations without losing edge or corner
features. A remaining challenge is how to supervise such a network as there is
no dataset with reference multi-implicits that be directly used. In this
paper, we present a network structure and training procedure that allow multi-
implicts to be trained using only local supervision. We describe how to
extract necessary local supervision from vector input and to adaptively obtain
training information for the multi-implicits.

We extensively evaluated the proposed multi-implicits representation for
various tasks including reconstruction, interpolation, and synthesis to
demonstrate clear advantages over several existing state-of-the-art
alternatives. Additionally, the representation naturally enables glyph
completion, wherein a single characteristic font glyph is used to synthesize a
whole font family in consistent style.

##  2 Related Work

Raster-based representation. One of the most intuitive representation of
shapes is raster, i.e., representing a 2D shape by pixels or a 3D shape by
voxels, which provides the grid-format data that is perfectly compatible with
(regular) CNN models. Hence it has acted as a catalyst for many deep learning
based methods for semantic editing of raster-based shapes. A general idea is
to learn an encoding-decoding model and then manipulate the shape in the
latent space. Sharing a similar spirit, many generative models Sinha et al.
(2017); Choi et al. (2018); Liu et al. (2019) engaged in the raster-based
generation and attribute transfer. Azadi et al. (2018); Yizhi Wang* (2020)
narrowed down the scope to fonts specifically, which focus more on the shape
instead of texture. While the raster-based representation has shown strong
semantic editability as incorporated with deep models, it is still limited by
its intrinsic resolution. Since image super-resolution is an ill-posed
problem, details cannot be fully recovered after upscaling.

Deep learning based vector graphs. Vector graphics (e.g., SVG) have been
widely adopted as a scalable representation. Typically, it is constructed by
sequences of Bézier curves, which are difficult to be modeled by traditional
CNN models since the diverse sequence length across different shapes, as well
as diverse types and numbers of Bézier curves for the similar or even the same
shape. Therefore, RNN-based models (e.g., LSTM) are commonly adopted in recent
works Lopes et al. (2019); Carlier et al. (2020) to learn the dynamic sequence
of curves. Although the sequential modeling could achieve semantic editing
and/or interpolation between shapes, it is still lagging behind CNN-based
models (raster-based methods) in terms of reconstruction accuracy because it
has to model much longer temporal dependency. Also, there is no specific
attention given to features like edges and corners.

Transformation from raster to vector. Instead of directly modeling raster or
vector, an idea of achieving both scalability and editability is to model the
transfer from raster to vector Reddy et al. (2021) (we will not consider
vectorization methods that purely transfer images to vectors). Such an
approach inherits the advantages of raster-based methods and supports easy
editability. In addition, they output scalable vectors directly. However, this
method would be still limited by the raster resolution, i.e., the output
vector graph cannot capture enough details since the corresponding raster
input may have already lost those finer details. Another challenge for such
methods is the high complexity of the shape, e.g., interpolation between
shapes with different topologies leads to undesirable intermediate shapes.

Deep implicit representation. Deep implicit functions Groueix et al. (2018);
Chen and Zhang (2019); Genova et al. (2020); Sitzmann et al. (2020);
Mildenhall et al. (2020); Takikawa et al. (2021) have achieved great success
in shape representation. They take advantage of deep learning techniques to
fit an implicit function, which provides a continuous representation breaking
the grid limitation of a raster domain. In addition, deep implicit functions
model shapes spatially instead of modeling sequentially as aforementioned in
deep learning based vector graphs. Therefore, deep implicit functions could
preserve the editability like raster-based representation and potentially
achieves scalable representation like vectors. Unfortunately, existing works
seldom explore the scalability and fidelity of shapes in their
representations, especially during editing, interpolation, and upscaling. For
instance, sharp corners always suffer from editing and scaling. This is
particularly problematic in the domain of fonts, and our work addresses this
limitation via the proposed multi-implicit representations.

##  3 Methods

We represent fonts as the composition of multiple global implicit functions.
An implicit representation has two key advantages: it allows for rendering at
arbitrary resolution; it can be locally supervised in characteristic areas
such as sharp edges and corners. Unlike a single implicit function, our multi-
implicit representation can faithfully reconstruct sharp features with low
reconstruction error (Figure 2).

We train a generative neural network, as shown in the inset, that models fonts
using this representation. Instead of directly learning the inside-outside
status of the font image Chen and Zhang (2019), we predict a set of distance
fields of 2D shapes. Their composition is then fed into a differentiable
rasterizer to produce the final image. We will show in Section 4 how this
generative model enables a diverse set of font reconstruction and editing
operations. Section 3.1 details differentiable rasterization of 2D distance
fields, and corresponding corner preservation is discussed in Section 3.2.
Finally, training losses and details are given in Section 3.3.

###  3.1 Differentiable Rasterization of Distance Field

As aforementioned, we use the signed distance field (SDF) to model 2D shapes.
The common supervisions for SDF are SDF labels from the ground truth curves or
distance transform on the silhouette Lin et al. (2020). Figure 3 compares the
results between training with SDF and training with raster, where training
only on the ground truth SDF does not always result in a good raster image in
terms of boundary smoothness. In contrast, training with the supervision of
raster yields smoother boundary. Therefore, we will draw supervision on
rasterized SDF to achieve better shape.

Figure 3:  Reconstruction with different training signals. Training using SDF
(2nd column) does not ensure good reconstruction in raster domain, while
training with both SDF and rasterized image (3rd column) and only rasterized
SDF (ours) yields smoother boundary.

Sharing the spirit from vector graphics rasterization works Fabris and Forrest
(1997); Gupta and Sproull (1981); Loop and Blinn (2005); Nehab and Hoppe
(2008); Turkowski (1982), we simplify general vector graphics rendering by
analytically approximating the point to curve distance as,

$I(x,y)\ =\ K\left(\min_{i\in\pazocal{F}}d_{i}(x,y)\right)g\left(x,y\right),$
(1)

$K(d)=\left\\{\begin{array}[]{lll}1&\text{if}\ \ d>\ \gamma,\\\
k\left(\frac{d}{\gamma}\right)=\frac{1}{2}+\frac{1}{4}\left(\left(\frac{d}{\gamma}\right)^{3}-3\left(\frac{d}{\gamma}\right)\right)&\text{if
}-\gamma\leq d\leq\gamma,\\\ 0&\text{if}\ \ d<-\gamma,\end{array}\right.$ (2)

where $d_{i}(x,y)$ indicates an SDF of distance from pixel $(x,y)$ center to
the closest point on the $i$-th curve corresponding the font $\pazocal{F}$,
and $K$ denotes a function that approximates the opacity based on the distance
value. There are many works for estimating $d_{i}(x,y)$ from scene parameters
but most of them are constrained by the choice of the parameterization. In
this paper, we model each $d_{i}(x,y)$ using an implicit neural network. The
function $g$ models the spatially-varying texture of the shape. For solid
fonts, we set $g=1$. In the supplemental we show examples of textured fonts
where we make use of spatially varying $g$. In the function $K$, $\gamma$ is
the anti-alias range, and the kernel $k$ is a radially symmetric continuous
filter that satisfies the constraint $k(1)=0$ and $k(-1)=1$. In our work, we
approximate $k(\cdot)$ using a parabolic kernel similar to Nehab and Hoppe
(2008); Li et al. (2020). Note that the rasterization function $I(x,y)$ has
non-zero gradients only if $(x,y)$ falls inside of the anti-aliasing range. We
use a progressively decreasing anti-aliasing range strategy for better
convergence and fidelity (see Sec. 3.3).

###  3.2 Multi-Curve Representation for Sharp Corners

We will lose details like sharp corners when upscaling bitmaps or sign
distance functions. Resampling an implicit model that encodes the pixel values
or signed distance values of a shape similarly suffers from blurry corners. An
brute force solution is to train the implicit model with extremely high-
resolution images, but this would drastically increase the burden of training,
and still limited by the training resolution. Rather than directly modeling
corners, we represent corners as the intersection of multiple curves (e.g.,
lines or parabolas), drawing inspiration from traditional representations
Green (2007). Note that even though these individual sub-curves may be
smoothed after encoded by a deep model, the sharpness of their intersection
will be preserved. With this insight, we construct sharp corners from multiple
smooth curves predicted by the implicit model. More specifically, the implicit
model is designed to predict multiple SDFs (and rasterization of distance
fields), each of which carries smooth curves/shapes decoupled from corners and
edges as illustrated in Figure 4(a).

Assume a shape $\pazocal{F}$ is represented by a set of curves
$\pazocal{C}=\\{C_{1},C_{2},\cdots,C_{n}\\}$, where $C_{n}$ is a binary map
indicating whether a pixel is inside (i.e., 1) or outside (i.e., 0) the $n$-th
curve, like the example in Figure 4(a). In our scenario, $C_{n}=K(d_{n})$,
where $d_{n}$ is the $n$-th SDF channel estimated by the implicit model. Then,
a function over all curves $F(\pazocal{C})$ will fuse those curves to
reconstruct the shape, preserving sharp corners. As illustrated in Figure
4(a), two curves can sufficiently represent a corner, either convex or concave
by adopting maximum or minimum as the function $F$. To represent a shape with
arbitrary corners, however, it requires three curves at least. For example,
$F(\pazocal{C})=\min(\max(C_{1},C_{2}),C_{3})$ can model all corners in a
shape. Therefore, we set $n=3$, i.e., $\pazocal{C}=\\{C_{1},C_{2},C_{3}\\}$.
Since deep models are sensitive to the permutation of guidance signals during
training, we use the median function as $F$, which achieves sharp corners and
permutational invariance to the order of these curves. Thus, a corner-related
loss on $F(\pazocal{C})$ could be robust to the ordering ambiguity.

Based on $F(\pazocal{C})$ (median function on three curves), a typical corner
$O$ is presented in Figure 4(b), where intersection of two curves divides its
local space into four quadrants, i.e., from $Q_{1}$ to $Q_{4}$. There are
always two opposite quadrants that one is inside area (i.e., $Q_{1}$) where
corresponding values from $F(\pazocal{C})$ are 1, and the other is outside
area (i.e., $Q4$) where corresponding values from $F(\pazocal{C})$ are 0. The
rest two opposite quadrants (i.e., $Q_{2}$ and $Q_{3}$) are equal on
$F(\pazocal{C})$ but different on $\pazocal{C}$. For example, the values on
$\\{C_{1},C_{2},C_{3}\\}$ corresponding to the $Q_{2}$ area is $(1,0,0)$, so
$F(\pazocal{C})$ on $Q_{2}$ is 0. Then, $F(\pazocal{C})$ on $Q_{3}$ should be
0 as well. However, the values on $\\{C_{1},C_{2},C_{3}\\}$ corresponding to
the $Q_{3}$ region must be different from $(1,0,0)$, i.e., could be $(0,1,0)$
or $(0,0,1)$. If $F(\pazocal{C})$ on $Q_{2}$ and $Q_{3}$ is 1, $O$ is a
concave corner. Otherwise, $O$ is a convex corner. Such distribution of
$\pazocal{C}$ along the four quadrants around a corner $O$ is referred to as
corner template.

(a) Possible corner preserving shapes represented with n=2.

(b) Corner template.

Figure 4: Intersection of two curves to encode concave and convex corners.

The shapes are encoded as the multi-curve representation through an implicit
model. The implicit model takes sample $(x,y,l)$ (i.e., 2D spatial location
$(x,y)$ and glyph label $l$), as well as embedding $z$ that indicates the font
style, and outputs three channels of SDF $\\{d_{1},d_{2},d_{3}\\}$. Then, the
rasterization approach discussed in Differentiable Rasterization of Distance
Field converts each $d_{i}$ to $C_{i}$ ($i=1,2,3$). Finally, the median
function $F(\pazocal{C})$ renders the final shape. We optimize the global
shape using the final render output and (locally) supervise each corner to be
locally represented as an intersection of two curves. One could use the output
of Chlumskỳ et al. (2018) to train a network. However, the edge coloring
approach presented in Chlumskỳ et al. (2018) has no canonical form, which
prevents a neural network from learning a continuous latent space between
shapes. Please note that since we focus on corner supervision to ensure sharp
corners at higher resolution resampling, it is unnecessary to constraint the
model by multi-channel supervision globally.

In the optimization of rendered global shape, the median operation
$F(\pazocal{C})$ would route the gradients to the correspondingly active value
only, i.e., only update a single channel at each location of $\pazocal{C}$.
However, at least two of the three channels of $\pazocal{C}$ at a certain
location need to be updated to approaching the ground truth because of the
nature of the median operator. Therefore, in the training stage, we use an
approximation $\hat{F}(\pazocal{C})$ defined as the average of the median and
the closest value to the median, thus two channels will be updated.

###  3.3 Training Details

We use three losses on the shape of glyph: (i) a global shape loss that
captures glyph shapes globally, (ii) corner template loss that supervises
intersection of curves locally to make the shape robust against resampling and
editing; and (iii) Eikonal loss to adhere to true SDFs.

For global shape training, since gradients are non-zero only at the anti-
aliasing range, i.e., edges of the shape, we sample the edges of the
rasterized glyph to train the implicit model. We shape $3\times 3$
neighborhoods around the anti-alias pixel, where we have a sample of one value
outside the shape (i.e., $0$), one value inside the shape (i.e., $1$), and
everything in between. Meanwhile, we sample from homogeneous areas inside and
outside of the glyph, such that the model does not fit a degenerate solution.
The edge/corner-aware sampling is referred to as importance sampling (see
inset). Such non-standard sampling further motivates the use of implicit
models, and importance sampling would significantly reduce the computational
complexity for training on higher resolution shapes as compared to the
traditional training on grid images.

We measure global shape loss between the final rendering from $F(\pazocal{C})$
and rasterized glyph $I$. We use a differential approximation
$\hat{F}(\pazocal{C})$ at training time. All the edges, area, and corner
samples from the raster image are used to train the implicit model via mean
square error as,

$\pazocal{L}_{global}=\mathbb{E}\left(\hat{F}(\pazocal{C})-I\right)^{2}.$ (3)

For local corner template loss, we first perform corner detection. A corner is
defined as a local where two curves intersect at an angle less than a
threshold (the threshold is 3rad or 171∘ in our experiments). For each corner,
we generate the corner template as discussed in section 3.2. The template size
is $7\times 7$ corresponding to the image size of $128\times 128$. The size of
the corner template is scaled based on the size of the image, but the size of
the sampling neighborhood for the global shape training remains the same. We
densely sample the edges and corners, and sparsely sample the homogeneous
areas.

To represent glyph corners as the intersection of two curves, we supervise the
corner samples by the corresponding corner templates. Since the render
function $F(\pazocal{C})$ is invariant to the order of the SDF/raster
channels, the corner template loss inherits the permutation invariance to the
channel order and, in order to avoid unnecessarily constraining the network,
we only supervise $Q_{2}$ and $Q_{3}$ of the template using the loss,

$\pazocal{L}_{local}=\mathbb{E}_{O\in\text{corner
samples}}\sum_{i=2}^{n}\min_{j\in\\{2,\cdots,n\\}}\left(C_{i}^{O}-T_{j}(O)\right)^{2},$
(4)

where $O$ indicates a corner sample from ground truth, and $n$ is the number
of channels (i.e., $n=3$ in our setting). The correspondingly predicted curves
of the corner $O$ are denoted by $C_{i}^{O}$, and the corresponding corner
template is $T(O)$ that has $n$ channels indexed by $j$.

The gradient loss aims to constraint the output of the implicit network to
resemble a distance field this is so that the implicit re-sampling is more
well behaved and resembles a closed continuous shape. A special case of
Eikonal partial differential equations Crandall and Lions (1983) posits that
the solution to $d(x,y;\theta)$ must satisfy $\mathbb{E}||\nabla
d(x,y;\theta)||\ =\ 1$, where $d(x,y;\theta)$ denotes a simplified implicit
model parameterized by $\theta$. Since satisfying this constraint is not
completely necessary for our desired solution, we loosen it to be greater than
or equal to 1 as Eq. 5, which intuitively encourages the function to be
monotonic.

$\pazocal{L}_{grad}=\left\\{\begin{array}[]{ll}\mathbb{E}|1-\|\nabla
d(x,y;\theta)\|_{2}|&\text{if }\|\nabla d(x,y;\theta)\|_{2}<1,\\\ 0&\text{if}\
\|\nabla d(x,y;\theta)\|_{2}\geq 1,\end{array}\right.$ (5)

where $\|\cdot\|_{2}$ represents the $\ell_{2}$-norm, and $|\cdot|$ calculates
the absolute values. Finally, the total loss is

$\pazocal{L}=\pazocal{L}_{global}\ +\ \alpha\pazocal{L}_{local}\ +\
\beta\pazocal{L}_{grad}+\gamma\|z\|_{2},$ (6)

where $\alpha$, $\beta$, and $\gamma$ are weights to balance these terms
during the training.

#### Training Warm-up.

Since the gradients mainly fall into the anti-aliasing range, network
initialization would significantly affect the convergence. To this
perspective, we set the initial anti-aliasing range to be the whole image
range and slowly shrink it to $k\cdot w^{-1}$ during the training, where $w$
is image width, and $k=4$ in our experiments. Such warm-up helps the model
converge more consistently, and the estimated SDF is more well behaved.
Comparison of SDF with and without warm-up is conducted in the supplementary.

##  4 Experiments

We evaluate our method against the tasks of reconstruction, interpolation, and
generation. In reconstruction and interpolation, we compare our method to
ImageVAE Kingma and Welling (2013), DeepSVG Carlier et al. (2020), and Im2Vec
Reddy et al. (2021), while we compare to DeepSVG and Attr2Font Yizhi Wang*
(2020) in the generation task. The metrics for evaluating the rendered glyphs
are mean squared error (MSE) and soft IoU (s-IoU), which is defined as

$\operatorname{s-IoU}(I_{1},I_{2})={\|I_{1}I_{2}\|_{1}}/{\|(I_{1}+I_{2})_{|0,1|}\|_{1}},$
(7)

where $I_{1}$ and $I_{2}$ are the images to compare, $\|\cdot\|_{1}$ denotes
the $\ell_{1}$-norm, and $|0,1|$ clips the values to the interval of [0,1]. To
evaluate the fidelity of glyph rendering in larger scales, glyphs are rendered
at the resolution of 128, 256, 512, and 1024 from each method without changing
the training resolution ($64\times 64$).

#### Reconstruction and Interpolation.

Figure 5: Reconstruction examples (baseline vs. ours) with zoom-in box
highlighting corners. We have vectorized the zero-level-set of the SDF output
in a piece-wise linear way. Please use digital-zoom to take a closer look at
the difference in reconstruction quality.

We compute MSE and s-IoU over the training dataset at different resolutions to
quantify how different algorithms capture the input training dataset. For a
fair comparison, we train all the algorithms on the same dataset used by
Im2Vec Reddy et al. (2021), which consists of 12,505 images. Table 1 displays
MSE and s-IoU metrics on the training set. For ImageVAE, we perform bilinear
interpolation to obtain higher resolution outputs. Our method outperforms the
others on s-IoU, indicating better reconstruction of the glyph shapes.
ImageVAE gets higher scores on MSE because its training objective aligns with
the MSE metric. However, ImageVAE would show blurry shapes in interpolation
and editing as demonstrated in Figure 6(a), where we achieve more continuous
interpolation/latent space. Even in reconstruction, as visualized in Figure.
5, ImageVAE cannot preserve sharp boundary and corners as compared to the
other methods. Since DeepSVG and Im2Vec directly output vectors, they can
always render shapes with clear boundaries, but they are limited in capturing
the global shapes as compared to our method. Another advantage of our method
over DeepSVG and Im2Vec is that we learn a smoother latent space, achieving
better performance on interpolation as demonstrated in Table 2. Even
interpolating between complex shapes, as shown in Figure 6(b), our method
performs better than the state-of-the-art Im2Vec. In Table 2 we present the
MSE and s-IOU calculated between a random interpolated glyph and its nearest
neighbour in the training dataset. This helps quantify similarity between
training versus generation distribution.

Table 1: Comparison with baselines on reconstructing training samples at different resolutions. In training, we use resolution of $64\times 64$. In testing, to achieve target resolution, we bilinearly upsample ImageVAE output from $64$, rasterize vector from DeepSVG and Im2Vec, and directly query ours. | MSE $\downarrow$ | s-IoU $\uparrow$
---|---|---
Methods | 128 | 256 | 512 | 1024 | 128 | 256 | 512 | 1024
ImageVAE | .0072 | .0120 | .0160 | .0186 | .8252 | .8416 | .8482 | .8494
DeepSVG | .1022 | .1081 | .1108 | .1121 | .3073 | .3124 | .3162 | .3164
Im2Vec | .0435 | .0518 | .0557 | .0571 | .7279 | .7293 | .7294 | .7294
Ours | .0118 | .0170 | .0201 | .0218 | .8750 | .8978 | .9035 | .9049

We have also experimented with Fourier features, Sine activation and Sawtooth
activation function with a feed forward network, due to the high frequency
nature of these functions the latent space learned by such a network is not
continuous. Since learning a continuous latent space is essential for
applications like interpolation and font family generation from a complete or
partial glyph, we resorted to using a LeakyReLU based activation function.
However if the users main requirement is only faithful reconstruction of the
input dataset, single channel fitting with fourier features trained using our
differential rasterization function should yield similar results as multi
channel fitting with LeakyReLU activation.

Table 2: Comparison with baselines on interpolation at different resolutions. | MSE $\downarrow$ | s-IoU $\uparrow$
---|---|---
Methods | 128 | 256 | 512 | 1024 | 128 | 256 | 512 | 1024
ImageVAE | .0181 | .0183 | .0185 | .0185 | .7715 | .7721 | .7731 | .7734
DeepSVG | .0544 | .0556 | .0569 | .0575 | .6337 | .6347 | .6365 | .6372
Im2Vec | .0434 | .0445 | .0463 | .0473 | .7213 | .7218 | .7232 | .7238
Ours | .0279 | .0297 | .0316 | .0343 | .8181 | .8184 | .8222 | .8234

(a) Comparison of all methods.

(b) A challenging example where baselines fail.

Figure 6: Comparison of interpolation between two random font styles. We color
different curves in the vector output to highlight details.

#### Generation.

To generate new fonts and the corresponding glyphs, first the auto-decoder
implicit model is trained with latent vector $z$ and glyph label (i.e., one-
hot encoding) concatenated to spatial locations as the input. We train on
1,000 font families, i.e., 52,000 images, and test on 100 font families. In
the inference stage, given an unseen glyph, we first find the optimal latent
vector (i.e., font style) that makes the rendered glyph closest to the given
glyph. More specifically, fixing the glyph label based on the given glyph, its
font latent vector $\hat{z}$ can be obtained by minimizing the distance
between the raster of the given glyph and the predicted glyph using gradient
descent. With the optimal $\hat{z}$, all the other glyphs with the same font
style can be generated by iterating the glyph label. Figure 7 compares the
font completion results between the baselines and ours, where a glyph “A” is
given with unseen font style. Our results outperform the others in terms of
global shape and sharpness of boundaries and corners. In general, raster-based
methods (e.g., Attr2Font) tend to generate better shape but get blurry at
corners. By contrast, vector-based methods (e.g., DeepSVG) eliminate the
blurry effect while difficult to achieve good global shapes. It is a dilemma
of generating better global shapes or better local corners in recent works.
Our method is achieving both good shapes and corners. Table 3 provides
statistical results that further demonstrates the superior generation capacity
of our method.

We conduct a more challenging task, i.e., glyph completion, to explore the
potential generation capacity of our method. As illustrated in Figure 8, given
a partial glyph, it can still recover the whole glyph, as well as other glyphs
with the same font style. In addition, sharp corners are still preserved.

Figure 7: Font completion examples (baseline vs. ours) with zoom-in boxes highlighting the corners. Figure 8: Glyph completion example. Given a partial glyph unseen in the training set, our method can complete the given glyph and other glyphs with the same font style. The zoom-in boxes highlight the corners. The mask region in the input is ignored during optimizing the latent vector $\hat{z}$. Table 3: Comparison with baselines on font generation task at different resolutions. | MSE $\downarrow$ | s-IoU $\uparrow$
---|---|---
Methods | 128 | 256 | 512 | 1024 | 128 | 256 | 512 | 1024
DeepSVG | .2597 | .2768 | .2854 | .2911 | .3584 | .3613 | .3651 | .3672
Attr2Font | .2004 | .2231 | .2481 | .2563 | .6204 | .6451 | .6523 | .6560
Ours | .0946 | .1027 | .1065 | .1083 | .8429 | .8462 | .8469 | .8471

##  5 Conclusion

We have presented multi-implicits – a new vector representation that is easy
to process with neural networks and maintains 2D shape fidelity under
arbitrary resampling. The representation is learned in a locally supervised
manner allowing high precision recovery of corners and curves. The proposed
multi-implicits representation is extensively evaluated in various font
applications, including high-resolution reconstruction, font style
interpolation, font family completion, and glyph completion, and demonstrates
clear advantages over prior image based and curve based approaches.

#### Broader Impact

The proposed representation has the potential to be applied to other 2D vector
objects, such as icons and animations, which can empower artists’ creativity
and productivity. If trained on a handwriting dataset such a method could
possibly be used for emulating a person’s handwriting for forgery.

## References

  * Azadi et al. [2018] Samaneh Azadi, Matthew Fisher, Vladimir G Kim, Zhaowen Wang, Eli Shechtman, and Trevor Darrell.  Multi-content gan for few-shot font style transfer.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7564–7573, 2018.
  * Carlier et al. [2020] Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte.  Deepsvg: A hierarchical generative network for vector graphics animation.  _arXiv preprint arXiv:2007.11301_ , 2020.
  * Chen and Zhang [2019] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5939–5948, 2019.
  * Chlumskỳ et al. [2018] Viktor Chlumskỳ, Jaroslav Sloup, and I Šimeček.  Improved corners with multi-channel signed distance fields.  In _Computer Graphics Forum_ , volume 37, pages 273–287. Wiley Online Library, 2018.
  * Choi et al. [2018] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8789–8797, 2018.
  * Crandall and Lions [1983] Michael G Crandall and Pierre-Louis Lions.  Viscosity solutions of hamilton-jacobi equations.  _Transactions of the American mathematical society_ , 277(1):1–42, 1983.
  * Fabris and Forrest [1997] Antonio Elias Fabris and A Robin Forrest.  Antialiasing of curves by discrete pre-filtering.  In _Proceedings of the 24th annual conference on Computer graphics and interactive techniques_ , pages 317–326, 1997.
  * Genova et al. [2020] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser.  Local deep implicit functions for 3d shape.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4857–4866, 2020.
  * Green [2007] Chris Green.  Improved alpha-tested magnification for vector textures and special effects.  In _ACM SIGGRAPH 2007 courses_ , pages 9–18. 2007.
  * Groueix et al. [2018] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.  A papier-mâché approach to learning 3d surface generation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 216–224, 2018.
  * Gupta and Sproull [1981] Satish Gupta and Robert F Sproull.  Filtering edges for gray-scale displays.  _ACM SIGGRAPH Computer Graphics_ , 15(3):1–5, 1981.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Li et al. [2020] Tzu-Mao Li, Michal Lukáč, Gharbi Michaël, and Jonathan Ragan-Kelley.  Differentiable vector graphics rasterization for editing and learning.  _ACM Trans. Graph. (Proc. SIGGRAPH Asia)_ , 39(6):193:1–193:15, 2020.
  * Lin et al. [2020] Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey.  Sdf-srn: Learning signed distance 3d object reconstruction from static images.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Liu et al. [2019] Ming Liu, Yukang Ding, Min Xia, Xiao Liu, Errui Ding, Wangmeng Zuo, and Shilei Wen.  Stgan: A unified selective transfer network for arbitrary image attribute editing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3673–3682, 2019.
  * Loop and Blinn [2005] Charles Loop and Jim Blinn.  Resolution independent curve rendering using programmable graphics hardware.  In _ACM SIGGRAPH 2005 Papers_ , pages 1000–1009. 2005.
  * Lopes et al. [2019] Raphael Gontijo Lopes, David Ha, Douglas Eck, and Jonathon Shlens.  A learned representation for scalable vector graphics.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7930–7939, 2019.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _European Conference on Computer Vision_ , pages 405–421. Springer, 2020.
  * Nehab and Hoppe [2008] Diego Nehab and Hugues Hoppe.  Random-access rendering of general vector graphics.  _ACM Transactions on Graphics (TOG)_ , 27(5):1–10, 2008.
  * Reddy et al. [2021] Pradyumna Reddy, Michael Gharbi, Michal Lukac, and Niloy J Mitra.  Im2vec: Synthesizing vector graphics without vector supervision.  _arXiv preprint arXiv:2102.02798_ , 2021.
  * Sinha et al. [2017] Ayan Sinha, Asim Unmesh, Qixing Huang, and Karthik Ramani.  Surfnet: Generating 3d shape surfaces using deep residual networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6040–6049, 2017.
  * Sitzmann et al. [2020] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Takikawa et al. [2021] Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, and Sanja Fidler.  Neural geometric level of detail: Real-time rendering with implicit 3D shapes.  2021\.
  * Turkowski [1982] Kenneth Turkowski.  Anti-aliasing through the use of coordinate transformations.  _ACM Transactions on Graphics (TOG)_ , 1(3):215–234, 1982.
  * Yizhi Wang* [2020] Zhouhui Lian Yizhi Wang*, Yue Gao*.  Attribute2font: Creating fonts you want from attributes.  _ACM Trans. Graph._ , 2020.


# Phase separation on surfaces in presence of matter exchange

Nirvana Caballero<EMAIL_ADDRESS>Department of Quantum Matter
Physics, University of Geneva, 24 Quai Ernest-Ansermet, CH-1211 Geneva,
Switzerland  Karsten Kruse  Department of Biochemistry, University of Geneva,
1211 Geneva, Switzerland  Department of Theoretical Physics, University of
Geneva, 1211 Geneva, Switzerland  NCCR Chemical Biology, University of Geneva,
1211 Geneva, Switzerland  Thierry Giamarchi  Department of Quantum Matter
Physics, University of Geneva, 24 Quai Ernest-Ansermet, CH-1211 Geneva,
Switzerland

(September 3, 2024)

###### Abstract

We present a field theory to describe the composition of a surface
spontaneously exchanging matter with its bulk environment. By only assuming
matter conservation in the system, we show with extensive numerical
simulations that, depending on the matter exchange rates, a complex patterned
composition distribution emerges in the surface. For one-dimensional systems
we show analytically and numerically that coarsening is arrested and as a
consequence domains have a characteristic length scale. Our results show that
the causes of heterogeneous lipid composition in cellular membranes may be
justified in simple physical terms.

††preprint: APS/123-QED

Living cells are full of fluid lipid membranes Alberts  _et al._ (2008). The
primary function of these membranes is to compartmentalize the cell interior
and to separate the cell from its environment. At the same time, diverse
patterns that play essential roles in vital processes form on their surfaces.
For example, protein clusters acting as units for sensing extra- or
intracellular signals Maddock and Shapiro (1993); Varma and Mayor (1998);
Plowman  _et al._ (2005); Fujita  _et al._ (2007). These protein clusters can
be transient or not and are often associated with domains rich in specific
kinds of lipids, commonly designated as lipid rafts Simons and Ikonen (1997).
Another spectacular example of membrane-associated patterns are protein waves
Beta and Kruse (2017). Such waves can be standing Raskin and Boer (1999) or
traveling Vicker (2000), which can lead to turbulent dynamics Tan  _et al._
(2020). Some of the surface-associated patterns could be reproduced in
reconstitution experiments in vitro Loose  _et al._ (2008); Landino  _et al._
(2021).

The physical principles underlying the formation of these patterns are still
not fully understood and simple reaction-diffusion systems as pioneered by
Turing Turing (1952) can miss essential aspects. For example, convective
transport along the membrane surface can play an important role Fan  _et al._
(2008). This holds notably for patterns associated with the so-called
cytoskeleton, a cellular polymer network in which chemical energy is
transformed into mechanical stress Alberts  _et al._ (2008). Gradients in this
stress lead to flows along the membrane surface Bois  _et al._ (2011); Kumar
_et al._ (2014); Hannezo  _et al._ (2015).

Alternatively, in-plane rearrangements in or on a membrane surface can result
from phase separation. This is of particular importance for the formation of
lipid domains. Lipid phase separation usually results in complete de-mixing,
although the coupling between line tension at the interface between different
phases and membrane bending can lead to stable domain patterns Baumgart  _et
al._ (2003). Alternatively, membrane-associated patterns can be formed when
the system is kept out of thermodynamic equilibrium Fang  _et al._ (2019), for
example, through the exchange of matter between the surface and the
surrounding medium Foret (2005); Turner  _et al._ (2005) as shown
schematically on Fig. 1. Such an exchange is also essential for the formation
of some protein patterns Wettmann and Kruse (2018).

Figure 1: Illustration of the dynamics considered. Two types of particles
(blue and yellow) on a membrane phase, in equilibrium with their respective
bulk reservoirs, phase separate into blue and yellow domains. Each species
attaches to and detaches from the membrane with the respective rates $k_{A}$
and $k_{D}$.

Pattern formation in presence of matter exchange between a surface and the
surroundings is often studied theoretically using descriptions in which the
distribution of particles in the bulk around the surface is assumed ad hoc to
be homogenous, see for example Refs. Foret (2005); Loose  _et al._ (2008);
Bement  _et al._ (2015). As a consequence, effective descriptions that only
consider the surface are used to analyze the ensuing dynamics. Even though
many patterns observed experimentally could be qualitatively reproduced in
this way, several observations show that these results have to be taken with
care. First, formally the assumption of a homogeneous bulk is not justified.
Second, although the patterns may look qualitatively similar to patterns
obtained in the full description that also accounts for the bulk, important
features may be missed Wettmann and Kruse (2018); Levernier and Kruse (2020).

In this work, we consider the case of phase-separating dynamics on a surface
in presence of spontaneous exchange of particles between the surface and the
bulk, Fig. 1. By integrating out the bulk dynamics we obtain an effective
model with memory for the time evolution on the surface. We show that the
resulting dynamics leads to stationary phase-separated patterns with an
intrinsic length scale. We determine this length scale both numerically and
via variational arguments. We also compare our results to a previous
phenomenological model with a simple instantaneous effective kernel Foret
(2005). Our work sheds light on the effects of coupling bulk and surface
dynamics for pattern formation.

We consider a surface, which coincides with the plane $z=0$, such that we
neglect surface fluctuations, exchanging matter (from both sides for
simplicity) with a bulk. The system is schematized in Fig. 1. We denote by
$\vec{r}=(x,y)$ a position on the surface and by $\tilde{m}(\vec{r})$ and
$n(\vec{r},z)$ the densities of particles adhering to the surface and in the
bulk respectively. Alternatively, we can interpret $\tilde{m}$ and $n$ as the
surface and volume fractions of one component of a two-component system (see
Fig. 1), where the two components could be, for example, protein species that
can adhere to a membrane or lipid species that constitute the membrane.

Conservation of matter implies that the bulk density $n(\vec{r},z)$ obeys

$\partial_{t}n+\vec{\nabla}\cdot\vec{j}=\delta(z)[k_{D}\tilde{m}(\vec{r},t)-k_{A}n(\vec{r},z=0,t)],$
(1)

where $\vec{j}(\vec{r},z,t)=-D\vec{\nabla}n(\vec{r},z,t)$ is the bulk particle
current, which we assume to be purely diffusive with diffusion constant $D$.
Particles close to the membrane attach to the surface at rate $k_{A}$, whereas
particles on the surface detach at rate $k_{D}$. We neglect possible
cooperative effects during particle attachment and detachment.

The dynamics of the surface is characterized by either the difference in
density of two species $m=\tilde{m}_{1}-\tilde{m}_{2}$ (as shown in Fig. 1) or
by the fluctuations of a single species around some average value
$m=\tilde{m}-\tilde{m}_{0}$. In both cases the free energy describing the
particle interactions is essentially symmetric in $m$. We use the
Ginzburg–Landau (GL) free energy
$\mathcal{F}=\int\mathrm{d}^{2}\vec{r}\left\\{-\frac{\alpha}{2}m^{2}+\frac{\delta}{4}m^{4}+\frac{\gamma}{2}\left(\nabla
m\right)^{2}\right\\}$ with constants $\alpha$, $\beta$, and $\gamma$. For
$\alpha>0$, it has two minima at $m_{1,2}=\pm\sqrt{\frac{\alpha}{\delta}}$
showing the tendency for phase separation. The dynamics of $m(\vec{r},t)$ is
ruled by a generalized form of the Cahn-Hilliard equation Chaikin  _et al._
(1995); Cugliandolo (2015):

$\partial_{t}m+\vec{\nabla}\cdot[-\mu\vec{\nabla}\frac{\delta\mathcal{F}}{\delta
m}]=k_{A}n(\vec{r},z=0,t)-k_{D}m(\vec{r},t)$ (2)

where $-\mu\vec{\nabla}\frac{\delta\mathcal{F}}{\delta m}$ is the surface
matter current, $\mu$ a mobility, and $n$ denotes in this equation either the
difference of the two species $n_{1}-n_{2}$ or the density in the bulk shifted
by $-\frac{k_{D}}{k_{A}}m_{0}$.

By integrating out the bulk we reduce the two-equations system (1) and (2) to
a single equation for $m(\vec{r},t)$ (see the Supplementary Material (SM)):

$\frac{\partial m}{\partial t}=-\mu\nabla^{2}(\alpha m-\delta
m^{3}+\gamma\nabla^{2}m)\\\ +\int_{0}^{t}dt^{\prime}\int
d\vec{r}^{\prime}K(\vec{r}-\vec{r}^{\prime},t-t^{\prime})m(\vec{r}^{\prime},t^{\prime}).$
(3)

The exchanges of matter with the bulk now manifest in this equation as the
kernel $K$

$K(r,t)=(\frac{\pi}{2})^{1/4}\frac{k_{D}\kappa}{\sqrt{Dt}}e^{-\frac{r^{2}}{4Dt}}\Big{(}\frac{1}{\sqrt{\pi\kappa
t}}-e^{\kappa t}\text{Erfc}(\sqrt{\kappa t})\Big{)}\\\
-k_{D}\delta(\vec{r}-\vec{r}^{\prime})\delta(t-t^{\prime}),$ (4)

where ${\kappa}=\pi\frac{k^{2}_{A}}{2D}$.

The kernel is non-local in time representing a memory in the dynamics coming
from the diffusion in the bulk. The result (3) is thus a microscopically
rooted description of the dynamics of the surface. It must be compared to more
phenomenological approaches Foret (2005) where the bulk was modelized by a
simple ad hoc relaxation term, local in space and time

$K_{\tau}=-\tau^{-1}\delta(\vec{r}-\vec{r}^{\prime})\delta(t-t^{\prime}),$ (5)

where the parameter $\tau$ is a typical matter exchange time. Then, the
dynamics is similar to that of phase separating and reacting chemical mixtures
Huberman (1976); Glotzer  _et al._ (1995). We examine below the physical
properties of (3) and show that depending on parameters this dynamics, which
leads to pattern exhibiting characteristic lengthscales, can differ markedly
from the phenomenological case.

Because of the kernel memory, solving (3) is much more complicated than for
the instantaneous kernel (5). To do so we take advantage of massive
parallelization in Graphical Processing Units (GPUs). We consider a system of
dimensions $L_{x}\times L_{y}$ with $L_{x}=L_{y}=128$ unless stated otherwise
and with periodic boundary conditions in both directions. We integrate the
dynamical equation by using a semi-implicit Fourier-spectral method Chen and
Shen (1998), adapted from Caballero  _et al._ (2018); Caballero (2021).
Without loss of generality, we choose $\mu=\alpha=\delta=\gamma=1$ and space
discretization equal to 1. We use $D$ to fix the time-scale by choosing
$D=0.1$ and vary $k_{A}$ and $k_{D}$. We approximate the integral in (3) by a
Riemann sum that requires the configurations of the system $m(\vec{q},t)$ in
the previous $M$ simulation steps (see SM for more details). For an
integration time-step $\Delta t=10^{-1}$ the difference in $m(\vec{r},t)$ for
simulations with $M=100$ and $M=10$ is of the order of $10^{-2}$. For our
purpose, this is a reasonable numerical error so we fix $\Delta t=10^{-1}$ and
$M=10$. Typical results obtained from a random initial condition are shown in
Fig. 2 together with results for the phenomenological kernel (5) for several
values of $\tau$.

Figure 2:  Snapshots of the distribution $m$ in a domain of 128$\times$128
after $10^{6}$ simulations steps starting from a random initial condition with
$\Sigma_{\vec{r}\in\text{system}}m(\vec{r},t)=0$. The pure states $m_{1}$ and
$m_{2}$ are given by the minima of the Ginzburg–Landau energy. Top rows 1 to
3: configurations for the full kernel (4). Row 4: configurations for the
instantaneous kernel (5). We use $D=0.1$. The values of the attachment rate
$k_{A}$, the detachment rate $k_{D}$, and the exchange time $\tau$ are given
in the figure. In absence of particle exchange with an environment, $K=0$, the
system completely phase separates (inset). On the contrary in presence of a
kernel $K$ coarsening is limited and a natural scale for the patterns appear.

In absence of matter exchange with the bath, $K=0$, and Eq. (3) reduces to the
Cahn-Hilliard equation for which coarsening leads to macroscopic phase
separation with eventually two domains of the pure phases $m_{1}$ and $m_{2}$.
Starting from a random initial condition with
$\Sigma_{\vec{r}\in\text{system}}m(\vec{r},t)=0$, we observe macroscopic
separation after $\sim 10^{5}$ simulation steps, see insert of Fig. 2.

On the contrary, in the presence of the full kernel (4) coarsening is
interrupted and a natural length scale of the pattern emerges. This can be
seen on Fig. 2, where configurations, which have evolved from the same random
initial condition, but for a much longer time ($10^{6}$ steps), exhibit a
characteristic pattern. To ensure that we were not tricked by slowing down of
the dynamics towards complete phase separation, we also considered initial
conditions with fully separated phases. In the presence of the full kernel
(4), stripe or bubble configurations evolved into multiple domains, indicating
that patterns with a characteristic length scale are indeed stable fixed
points of the dynamics, see SM.

The labyrinthine patterns we observe in steady state exhibit a characteristic
length scale determining the width of the meandering stripes. This length
scale decreases with increasing detachment rate $k_{D}$ and shows a non-
monotonic dependence on the attachment rate $k_{A}$. In case of the
instantaneous kernel (5) we observe a similar dependence of the characteristic
length scale on $\tau^{-1}$.

In order to rationalize this dependence of the patterns on the matter exchange
rates and to quantify the differences between the full and the instantaneous
kernels, we first examine the full kernel as a function of momentum and the
parameter $s$ resulting from a Laplace transform of the temporal coordinate.
In these variables, our kernel (4) reads

$K(q,s)=-\frac{k_{D}}{1+\sqrt{\frac{\kappa}{Dq^{2}+s}}}.$ (6)

We see that for small values of the parameter $\kappa$, that is small values
of the attachment rate $k_{A}$ or large diffusion constants $D$, this
expression becomes essentially independent of $s$ and thus an instantaneous
kernel of the form (5) with the identification $\tau=1/k_{D}$. Our microscopic
calculation thus validates the use of the instantaneous kernel (5) in such a
limit and gives a microscopic value for the effective lifetime $\tau$. In the
opposite limit, on the contrary, we see that the non local dependence in time
has a strong effect on the kernel and we can thus expect different physical
behaviors, at least quantitatively.

Figure 3:  Pseudo free energy $\mathcal{F}_{m}$ for profiles with $n$ evenly
spaced kinks. Dashed line: instantaneous kernel, full lines: full kernel,
dotted line: full kernel for $s=1/t_{0}=1$. Kernel parameters $\tau^{-1}$,
$k_{D}$, and $k_{A}$ as indicated in the legend. The minimum $n_{min}$ of each
energy function is indicated with a symbol. Inset: profile with $n=12$ kinks.

Let us consider the instantaneous kernel (5) in one spatial dimension. Its
dynamics is given by

$\displaystyle\partial_{t}m(q,t)$
$\displaystyle=-q^{2}\mu\frac{\delta\mathcal{F}_{m}}{\delta m^{*}({q},t)},$
(7)

where the star indicates the complex conjugate, and $q=\frac{2\pi}{L}k$ for a
system size $L$, and $k=1,\dots,L$. In this expression, we have introduced the
pseudo-free energy $\mathcal{F}_{m}=\mathcal{F}+\mathcal{F}_{\tau}$, where
$\mathcal{F}$ is the GL free energy and $\mathcal{F}_{\tau}$ satisfies

$\frac{\delta\mathcal{F}_{\tau}}{\delta m^{*}({q},t)}=\frac{1}{\tau
q^{2}\mu}m({q},t),$ (8)

From Eq. (7) it is easy to show that $\mathcal{F}_{m}$ monotonically decreases
under the time evolution and thus that the fixed point of the dynamic
evolution must correspond to the minimum of $\mathcal{F}_{m}$ if reachable
from the initial configuration. For the parameters of the GL free energy used
in this work, we can approximate these states as regions of uniform
concentration, where $m$ takes one of the minimal values $m_{1}$ or $m_{2}$,
separated by narrow transition regions or ’kinks’. In the case of the Cahn-
Hilliard equation, where $\mathcal{F}_{\tau}=0$, these kinks take the form of
a hyperbolic tangent,
$\sqrt{\frac{\alpha}{\delta}}\tanh(\sqrt{\frac{\alpha}{2\gamma}}x)$, and
$\mathcal{F}\approx\mathcal{F}_{0}+\epsilon_{0}n$. Here, $n$ is the (even)
number of kinks, $\epsilon_{0}$ the energy associated with a kink in GL, and
$\mathcal{F}_{0}$ the energy associated with the uniform regions. The term
$\mathcal{F}_{0}$ depends only weakly on $n$. The configuration with minimal
energy is thus the one with the minimal number of kinks, i.e., $n=2$, and
corresponds to macroscopic phase separation.

The presence of $\mathcal{F}_{\tau}$ changes this minimum. We can estimate the
corresponding number of domains through a variational approach. First, we
construct one-dimensional profiles by combining $n$ evenly spaced kinks and
probe the value of the pseudo free energy as a function of $n$ as shown in
Fig. 3. For our parameters, kink (and anti-kink) $j$ with $j=0,\cdots,n/2$ is
given by $\theta\tanh(\frac{x-x_{j}(\theta)}{\sqrt{2}})$, where $\theta=1$ for
$x\in[2jL/n,(2j+1)L/n]$, $\theta=-1$ for $x\in[(2j+1)L/n,2(j+1)L/n]$ and
$x_{j}(\theta)=(4j+3/2-\theta/2)L/(2n)$. See Fig. 3, inset for an example of
the resulting profile. Then, we compute the pseudo-free energy
$\mathcal{F}_{m}$ for this profile and minimize with respect to $n$.

Figure 4:  a) Part of the profiles used as initial condition (dashed lines) to
simulate systems evolving with (5) ($\tau^{-1}=10^{-4}$) and our kernel (4)
for $k_{A}=10^{-4}$ and $k_{D}=\tau^{-1}$. In continuous lines we show the
profiles after dynamic evolution. For $k_{A}=1$ the number of kinks
significantly differs from the number used as initial condition, as shown in
b). For reference in this case we show in dotted lines a constructed profile
with the observed $n_{min}$ and amplitude 0.8. c) Scaling relation between the
optimal number of kinks $n_{min}$ and the kernel parameters $k_{D}$ for fixed
values of $k_{A}$ or $\tau^{-1}$ depending on the kernel used. Stars and
crosses represent respectively for the microscopic and phenomenological kernel
an initial condition obtained by minimizing the effective free energy, as
shown in Fig. 3. Squares indicate the number of kinks observed in simulations
of systems that evolved for up to $10^{6}$ steps from this initial condition
and thus correspond up to numerical limitations to the fixed point of the
dynamics (the two circles corresponding to $k_{A}=1$ and
$k_{D}=10^{-4},10^{-3}$ indicate that the configuration observed for these set
of parameters is not regular in contrast with all the other observed
configurations). The scaling of $n_{min}$ appears to be always a power law but
with strong quantitative differences between the microscopic and
phenomenological kernels as expected when $k_{A}$ is large. Fits of the data
with a power law with exponent $\zeta_{dis}$ are shown in dashed and
continuous lines for their respective kernels.

Extending this analysis to the case of the full kernel (4) is more involved. A
naive attempt could be made from Eq. (6) by assuming that at large times we
can approximate this expression by taking $s\to 0$ and then identifying the
resulting $q$-dependent prefactor with $1/\tau$ in Eq. (8). The pseudo-free
energy has in this case a qualitatively similar shape as for the instantaneous
kernel, Fig. 3.

We test the configurations obtained with the minimization strategy by
constructing profiles with $n_{min}$ number of kinks that serve as initial
condition for the full evolution of the dynamic Eq. (3) for both kernels. As
shown in Fig. 4a, the effect of both kernels on the profiles is to slightly
distort the shape of the valleys and peaks. As depicted in Fig. 4b we see that
for the instantaneous kernel, the variational solution is essentially stable
under time evolution, showing that the variational principle is indeed
predicting correctly the fixed point of the dynamics. A simple estimate can be
given by noting that $\mathcal{F}_{\tau}$ scales as $L^{2}\tau^{-1}n^{-2}$. In
presence of matter exchange in the instantaneous system, we find that the
minimum of $\mathcal{F}_{m}$ is reached for
$n=const\left(\frac{L^{2}}{\tau\epsilon_{0}}\right)^{1/3}$. This scaling
relation is in agreement with our numerical results as shown in Fig. 4c.
However, for the full kernel, the final state differs strongly from the
initial configuration derived from the argument above when $k_{A}$ is large.
In particular, the number of kinks is largely different between the
variational estimate and the full evolution, Fig. 4b. This shows that the
naive substitution of the $s=0$-kernel in the pseudo-free energy is not
sufficient and that a more precise method must be found. Putting
phenomenologically a finite $s$, as shown in Fig. 3, to mimic a finite time
cutoff in the memory of the full kernel does push the minimum of the pseudo
free-energy to a larger number of kinks but does not allow for a reliable
prediction of the fixed point of the time evolution. Another possibility is
that there is more than just one characteristic scale for the domains which
could explain why deformed bubble domains were observed in liquid-liquid phase
separation of intracellular condensates Riback  _et al._ (2020). Finding the
equivalent of a predictive variational approach, if at all possible, for the
full kernel is a very interesting but challenging question for future studies.
Extension of these methods to the case of the two dimensional patterns
computed numerically in Fig. 2 is also interesting since it would provide
direct access to the pattern formation bypassing the need for the full
dynamical analysis.

Beyond the analytical approximations, our simulations show clearly that an
optimal length scale exists, revealed by the optimal number of kinks
$n_{min}$. This number scaling algebraically with the parameters of the
kernel. As expected the full and instantaneous kernels essentially coincide at
small $k_{A}$, whereas largely different behaviors with different exponents
are observed when $k_{A}$ is large; at least for the time available in our
simulations. This conveys the importance of having a properly defined full
kernel to identify quantitatively the pattern formed.

In this work, we have presented a field theory to describe matter distribution
in a membrane exchanging matter with its environment. Our theory predicts
arrested phase separation with domains characterized by typical sizes
determined by the absorption and expulsion matter exchange rates. Based on
semi-microscopical equations for the composition of a surface and its
environment, we integrate the environment contribution in a single equation
for the surface composition dynamics. The matter exchange effect induces
spatio-temporal memory effects with non-trivial consequences for the typical
domain sizes for large absorption rates. On the contrary, when the absorption
rate is low ($k_{A}\ll 1$) our theory behaves very similarly to a
instantaneous kernel that was previously phenomenologically proposed. In this
case, we show with semi-analytical arguments that matter exchange induces
phase separation in the membrane with domains characterized by a typical
length. We compute its scaling as a function of the parameters of the problem
for the one-dimensional case. Our theory provides a physical justification for
the functional form of the instantaneous kernel.

###### Acknowledgements.

This work was supported in part by the Swiss National Science Foundation under
Division II. All numerical simulations were performed at the University of
Geneva on the Mafalda cluster of GPUs.

## References

  * Alberts  _et al._ (2008) B. Alberts, A. Johnson, J. Lewis, M. Raff, K. Roberts, and P. Walter,  _Molecular Biology of The Cell_ , 5th ed. (2008).
  * Maddock and Shapiro (1993) J. R. Maddock and L. Shapiro, Science 259, 1717 (1993).
  * Varma and Mayor (1998) R. Varma and S. Mayor, Nature 394, 798 (1998).
  * Plowman  _et al._ (2005) S. J. Plowman, C. Muncke, R. G. Parton, and J. F. Hancock, Proc. Natl. Acad. Sci. USA 102, 15500 (2005).
  * Fujita  _et al._ (2007) A. Fujita, J. Cheng, M. Hirakawa, K. Furukawa, S. Kusunoki, and T. Fujimoto, Mol. Biol. Cell 18, 2112 (2007).
  * Simons and Ikonen (1997) K. Simons and E. Ikonen, Nature 387, 569 (1997).
  * Beta and Kruse (2017) C. Beta and K. Kruse, Ann. Rev. Cond. Mat. Phys. 8, 239 (2017).
  * Raskin and Boer (1999) D. M. Raskin and P. A. J. d. Boer, Proc. Natl. Acad. Sci. USA 96, 4971 (1999).
  * Vicker (2000) M. G. Vicker, Biophysical Chemistry 84, 87 (2000).
  * Tan  _et al._ (2020) T. H. Tan, J. Liu, P. W. Miller, M. Tekant, J. Dunkel, and N. Fakhri, Nat. Phys. 16, 657 (2020).
  * Loose  _et al._ (2008) M. Loose, E. Fischer-Friedrich, J. Ries, K. Kruse, and P. Schwille, Science 320, 789 (2008).
  * Landino  _et al._ (2021) J. Landino, M. Leda, A. Michaud, Z. T. Swider, M. Prom, C. M. Field, W. M. Bement, A. G. Vecchiarelli, A. B. Goryachev, and A. L. Miller, Current Biology (2021), 10.1016/j.cub.2021.10.021.
  * Turing (1952) A. M. Turing, Phil. Trans. Roy. Soc. B 237, 37 (1952).
  * Fan  _et al._ (2008) J. Fan, M. Sammalkorpi, and M. Haataja, Phys. Rev. Lett. 100, 178102 (2008).
  * Bois  _et al._ (2011) J. S. Bois, F. Jülicher, and S. W. Grill, Phys. Rev. Lett. 106, 028103 (2011).
  * Kumar  _et al._ (2014) K. V. Kumar, J. S. Bois, F. Juelicher, and S. W. Grill, Phys. Rev. Lett. 112, 208101 (2014).
  * Hannezo  _et al._ (2015) E. Hannezo, B. Dong, P. Recho, J.-F. Joanny, and S. Hayashi, Proc. Natl. Acad. Sci. USA 112, 8620 (2015).
  * Baumgart  _et al._ (2003) T. Baumgart, S. T. Hess, and W. W. Webb, Nature 425, 821 (2003).
  * Fang  _et al._ (2019) X. Fang, K. Kruse, T. Lu, and J. Wang, Rev. Mod. Phys. 91, 137 (2019).
  * Foret (2005) L. Foret, EPL (Europhysics Letters) 71, 508 (2005).
  * Turner  _et al._ (2005) M. S. Turner, P. Sens, and N. D. Socci, Phys. Rev. Lett. 95, 168301 (2005).
  * Wettmann and Kruse (2018) L. Wettmann and K. Kruse, Phil. Trans. Roy. Soc. B 373, 20170111 (2018).
  * Bement  _et al._ (2015) W. M. Bement, M. Leda, A. M. Moe, A. M. Kita, M. E. Larson, A. E. Golding, C. Pfeuti, K.-C. Su, A. L. Miller, A. B. Goryachev, and G. v. Dassow, Nature Cell Biology 17, 1471 (2015).
  * Levernier and Kruse (2020) N. Levernier and K. Kruse, New J. Phys. 22, 013003 (2020).
  * Chaikin  _et al._ (1995) P. M. Chaikin, T. C. Lubensky, and T. A. Witten,  _Principles of condensed matter physics_ , Vol. 10 (Cambridge university press Cambridge, 1995).
  * Cugliandolo (2015) L. F. Cugliandolo, Comptes Rendus Physique 16, 257 (2015).
  * Huberman (1976) B. A. Huberman, J. Chem. Phys. 65, 2013 (1976).
  * Glotzer  _et al._ (1995) S. C. Glotzer, E. A. D. Marzio, and M. Muthukumar, Phys. Rev. Lett. 74, 2034 (1995).
  * Chen and Shen (1998) L. Q. Chen and J. Shen, Computer Physics Communications 108, 147 (1998).
  * Caballero  _et al._ (2018) N. B. Caballero, E. E. Ferrero, A. B. Kolton, J. Curiale, V. Jeudy, and S. Bustingorry, Physical Review E 97, 062122 (2018).
  * Caballero (2021) N. Caballero, J. Stat. Mech.: Th. Exp. 2021, 103207 (2021).
  * Riback  _et al._ (2020) J. A. Riback, L. Zhu, M. C. Ferrolino, M. Tolbert, D. M. Mitrea, D. W. Sanders, M.-T. Wei, R. W. Kriwacki, and C. P. Brangwynne, Nature 581, 209 (2020).

Phase separation in surfaces in presence of matter exchange
Supplementary material

## Reduction of the coupled dynamic equations

The system of two equations for the surface membrane and the reservoir

$\systeme[][:]{\partial_{t}n=D\Delta
n+\delta(z)[k_{D}m-k_{A}n(\vec{r},z=0,t)]:\partial_{t}m=\mu\nabla^{2}\frac{\delta\mathcal{F}}{\delta
m}+k_{A}n(\vec{r},z=0,t)-k_{D}m},\\\ $ (S.1)

may be reduced to a single equation. By defining the Fourier transform of a
function $A_{x}$ as
$A_{x}=\int\frac{d{q}}{\sqrt{2\pi}}e^{i{x}{q}}\tilde{A}_{q}$, we write

$\tilde{n}(\vec{q},z=0,w)=\frac{k_{D}\tilde{m}(\vec{q},w)\tilde{I}(\vec{q},w)}{\sqrt{2\pi}+k_{A}\tilde{I}(\vec{q},w)},\\\
$ (S.2)

where $\tilde{I}(\vec{q},w)=\int dk_{z}\frac{1}{iw+D(k_{z}^{2}+\vec{q}^{2})}$.
Replacing (S.2) in (S.1) for $\tilde{I}\neq 0$ we obtain

$\partial_{t}m=\mu\nabla^{2}\frac{\delta\mathcal{F}}{\delta
m}+\mathcal{F}^{-1}\Big{[}\Big{(}\tilde{K}(\vec{q},w)-1\Big{)}k_{D}\tilde{m}(\vec{q},w))\Big{]},$

with
$\tilde{K}(\vec{q},w)=\frac{1}{\frac{\sqrt{2\pi}}{\tilde{I}(\vec{q},w)}+k_{A}}.$

The space-Fourier transform of this quantity is

$\hat{K}(\vec{q},t)=\kappa e^{-q^{2}Dt}\Big{(}\frac{1}{\sqrt{\pi\kappa
t}}-\text{Erfcx}(\sqrt{\kappa t})\Big{)},\\\ $ (S.3)

where ${\kappa}=\pi\frac{k^{2}_{A}}{2D}$, and $\text{Erfcx}(t)$ is the
standard error function
$\text{Erfcx}(t)=e^{t^{2}}\frac{2}{\sqrt{\pi}}\int_{t}^{\infty}e^{-z^{2}}dz$.

## Numerical solution

We discretize Eq. 3 so that the surface particles’ density $m$ is a matrix of
elements $(i,j)$ at each simulation step $p$, with a given initial condition
$m(i,j,p=0)$. To obtain the evolution of $m$ we use the standard Euler semi-
implicit integration method for the time variable and a five-point
discretization of the Laplacian terms in Fourier space for the spatial
coordinates.

Compared to a standard Cahn-Hilliard equation we have to consider an extra
term involving the integral of our kernel. We approximate the integral in Eq.
3 at a simulation step $p$ by the Riemann sum
$\int_{0}^{t}dt^{\prime}\hat{K}(\vec{q},t-t^{\prime})m(\vec{q},t^{\prime})\simeq\sum_{p^{\prime}=M}^{p-1}\hat{K}(q_{i},q_{j},p-p^{\prime})m(q_{i},q_{j},p^{\prime})\Delta
t$, where $\hat{K}$ is given by Eq. S.3, and $M$ is the number of previous
configurations that will be taken into account in the calculation.

Figure SF1: Snapshot of systems that evolved during $10^{4}$ simulation steps
for different values of the absorption and desorption rates, $k_{A}$ and
$k_{D}$, respectively, at fixed diffusion constant $D=0.1$. We consider two
different initial conditions (shown on the insets). The effect of matter
exchange between the membrane and the bath is observed at relatively short
times, especially for larger values of $k_{D}$. The images highlighted with
dashed gray lines correspond to systems that evolved under the instantaneous
kernel (Eq. 5).

## Short-time effects

To further verify the multi-domain state of the system when matter exchange is
considered, we studied the cases where the initial condition is a single
domain, either a bubble or a stripe, as shown in Fig. SF1. The effect of the
kernel is immediately observed in the simulations for both considered kernels.


# ExpertMatcher: Automating ML Model Selection for Clients using Hidden
Representations

Vivek Sharma
MIT
<EMAIL_ADDRESS>
&Praneeth Vepakomma
MIT
<EMAIL_ADDRESS>
&Tristan Swedish
MIT
<EMAIL_ADDRESS>
Ken Chang
MIT
<EMAIL_ADDRESS>
&Jayashree Kalpathy-Cramer
MGH/Harvard Medical School
<EMAIL_ADDRESS>
&Ramesh Raskar
MIT
<EMAIL_ADDRESS>

###### Abstract

Recently, there has been the development of Split Learning, a framework for
distributed computation where model components are split between the client
and server (Vepakomma et al. ,, 2018b). As Split Learning scales to include
many different model components, there needs to be a method of matching
client-side model components with the best server-side model components. A
solution to this problem was introduced in the ExpertMatcher (Sharma et al. ,,
2019) framework, which uses autoencoders to match raw data to models. In this
work we propose an extension of ExpertMatcher, where matching can be performed
without the need to share the client’s raw data representation. The technique
is applicable to situations where there are local clients and centralized
expert ML models, but sharing of raw data is constrained.

##  1 Introduction

The Expert matching problem as described by Sharma et al. , (2019) is to
assign input data from a client to the most appropriate expert model without
any information other than the input data itself. In essence, the goal is to
assign input data based on its likelihood of being drawn from the distribution
of the expert model training data. We are interested in assigning an expert
model to a given sample such that an improved performance is achieved. The
expert matching problem is well-studied: (Jacobs et al. ,, 1991; Jacobs,,
1995, 1997), starting from the seminal work of Jacobs et al. , (1991), where
the authors trained a versatile blend of experts for speaker vowel recognition
and utilized a gating system to determine which of the systems should be
utilized for each sample. Other lines of work try to avoid using a gating
function (Hinton et al. ,, 2015; Ahmed et al. ,, 2016) by training one oracle
model. Aljundi et al. , (2017) learns a gating function to make expert network
assignments. More recently, with the ongoing advances in machine learning and
big data, Sharma et al. , (2019) proposed a landscape for the expert matching
problem and shows that by using an Autoencoder (AE), it becomes possible to
assign the most relevant expert model(s) given a sample data representation in
a client-server setting. However, in their setting the client shares the raw
data with the server.

For many reasons raw data cannot always be shared. First, there is oftentimes
a need to protect the privacy of the individual from whom the data originated
(such as patient health data). Further, data is a valuable resource and many
clients prefer not to have their data open to the public. Lastly, if data is
shared, there will be additional storage requirements which may be cumbersome
and expensive. Motivated by this observation, our approach does not require
sharing of raw data directly. Instead, independent autoencoders are trained on
both the server and client sides and only the intermediate hidden
representation of the data is shared for expert network(s) assignment.

We evaluate our proposed method on six datasets, such as objects, text,
digits, biological and sensor, namely STl-10 (Coates et al. ,, 2011), MNIST
(LeCun et al. ,, 1998), HAR (Anguita et al. ,, 2013), Reuters (Lewis et al. ,,
2004), Non Line of Sight (Tancik et al. ,, 2018) and Diabetic Retinopathy
(Graham,, 2015). We are inspired from Sharma et al. , (2019), and we
experimentally show that we can achieve similar performance as Sharma et al. ,
(2019) to match the task (coarse-Level) assignment and reasonably good
performance for class (fine-grained) assignment in a Split Learning
distributed computation framework (Vepakomma et al. ,, 2018b, 2019, a).

Figure 1: Pipeline of ExpertMatcher with hidden representations at the client
and server for automatic model selection from a repository of server models
based on their relevance to query data set hosted by the client. The matching
is done based on encoded intermediate representations. Figure 2: Landscape of
ExpertMatcher, Source: Sharma et al. , (2019). This paper is relevant to the
fourth setup where raw data or model is not shared between client and server,
but rather some intermediate encoded representation is shared.

##  2 Hidden Representation based ExpertMatcher

The expert matching problem aims to assign a given sample the best model at
hand for a given task. Figure 2 shows the landscape of the ExpertMatching
problem proposed by Sharma et al. , (2019). In this work, we consider the no
sharing scenario (row fourth in Figure 2), where the client and server do not
share the data and model, but just the low-dimensional intermediate
representation. Furthermore, within Figure 2 we consider (1) Resolution:
coarse and fine level assignment; (2) Fusion: top-1; and (3) Metric: adhoc (
i.e. cosine similarity).

Figure 3: Overview of the coarse level dataset assignment (CA) and fine-level
model assignment (FA) of the client’s data in a hierarchical fashion. In
contrast to Sharma et al. , (2019), for both CA and FA we use maximum cosine-
similarity as a metric for model assignment. Further, the client shares the
encoder representation rather than the raw samples (Sharma et al. ,, 2019).

Now, we describe our proposed method to solve the ExpertMatcher problem
without sharing raw data. In Sharma et al. , (2019), the raw data is shared
with the Server. In our proposed method, the Client shares only a encoded
hidden representation for both Coarse-level (CA) and Fine-grained (FA) expert
matching. Figure 1 and Figure 3 sketches the proposed method.

Implementation: We assume that we have $K$ pre-trained expert networks on the
centralized server, each of these networks has its corresponding pre-trained
autoencoders (AE) $\phi_{K}$ trained on a task-specific dataset. Given the
dataset on which the AE was trained on, we extract the hidden representations
of the whole dataset and compute an average representation of the dataset
$\mu_{k}\in\mathbb{R}^{d},k\in\\{1,\ldots,K\\}$, where $d$ is the feature
dimension. Assuming the dataset consist of $N$ object classes, we also compute
the average representation of each class in the dataset
$\mu_{k}^{n}\in\mathbb{R}^{d},n\in\\{1,\ldots,N\\},k\in\\{1,\ldots,K\\}$.

The clients (Client A and Client B) utilize a similar approach as the server,
where the clients train their unique AE’s one for each $p^{th}$ and $q^{th}$
datasets, Client A: $p\in\\{1,\ldots,P\\}$ and Client B:
$q\in\\{1,\ldots,Q\\}$. Lets assume for Client A, the intermediate features
extracted from a hidden layer for a sample $X_{p}^{1}$ is given as
$x_{p}^{1}=\phi_{p}^{1}(X_{p}^{1})$, similarly for Client B,
$x_{q}^{2}=\phi_{q}^{2}(X_{q}^{2})$. For brevity, we denote the intermediate
representation coming from any client as $x^{{}^{\prime}}$.

For coarse assignment (CA) of clients data: hidden representation
$x^{{}^{\prime}}$, we assign an server AE, $k^{\ast}\in\\{1,\ldots,K\\}$ which
has maximum cosine similarity of $x^{{}^{\prime}}$ with $\mu_{k}$.

For fine-grained (FA) of clients data: hidden representation
$x^{{}^{\prime}}$, we assign an expert network $M_{n},n\in\\{1,\ldots,N\\}$
that has the maximum cosine similarity of $x_{k^{\ast}}^{{}^{\prime}}$ with
$\mu_{k}^{n}$.

Finally after the assignment of the given sample to the model, one can easily
train a SplitNN type architecture Vepakomma et al. , (2018b, 2019, a).

In the current setup, a weak level of privacy is preserved as the server does
not have access to the raw client’s data, but rather a very low dimensional
encoded representation.

Note that there is a shortcoming of this approach. If the server has no AE
model dedicated to the client data, the wrong assignment of client data takes
place because of maximum cosine similarity criteria - this can be resolved by
adding an additional model on the server that performs a binary
classification: if the client data matches the server data or not.

##  3 Experiments

We present our evaluation of several different datasets. We first describe the
dataset, metric, implementation details, followed by a thorough analysis of
the proposed methods and comparison to Sharma et al. , (2019) for both coarse
assignment (CA) and fine-grained assignment (FA).

Datasets and Implementation Details. We demonstrate our method on challenging
datasets which cover domains such as objects, text, digits, biological and raw
sensors. The datasets are summarized in Table 1. For a fair comparison with
Sharma et al. , (2019), we use the same splits for Server, client A and Client
B.

For all the image data types, we resize them to $28\times 28$, and then
flattening it to 784 dimensions. While for other datasets, we use 1D adaptive
average pooling (AdaptiveAvgPool1d) to transform the input data to 784
dimensions. AE: All the AEs use a single-layer MLP encoder-decoder
($\mathbb{R}^{784}\rightarrow\mathbb{R}^{128}\rightarrow\mathbb{R}^{784}$). We
use Adam optimizer for model training. We train the AE models for 45 epochs
with a learning rate of $10^{-2}$, which is then manually decreased by a
factor of 10 every 15 epochs.

Note that, in this work, the server shares the pseudorandom number generator
seeds with the clients to make the same random initialization for PyTorch
model training (e.g. torch.manual_seed(0)). In future work, we plan to study
the impact of random initialisation when the seed is not shared between server
and clients.

Table 1: Datasets. “#S” denotes the number of samples, “#C” denotes the true
number classes, and LC/SC is the class balance largest class (LC) to smallest
class (SC).

| STL-10 | MNIST | HAR | REUTERS | NLOS | DB | ALL
---|---|---|---|---|---|---|---
Type | Object | Digits | Sensors | Text | Sensor | Biological |
#C | 10 | 10 | 6 | 4 | 3 | 3 |
#S | 13k | 10k | 10299 | 10k | 45096 | 3540 |
Dim. | 32px | 28px | 561 | 2000 | $640\times 480$ | 512px |
LC/SC (%) | 10/10 | 11.35/8.92 | 19/14 | 43.12/8.14 | 33.33/33.33 | 33.33/33.33 |
Server | 6500 | 5000 | 5151 | 5000 | 22548 | 1770 |
Client A | 3250 | 2500 | 2574 | 2500 | 11274 | 885 | 22983
Client B | 3250 | 2500 | 2574 | 2500 | 11274 | 885 | 22983

Evaluation Metric. For both CA and FA, we use maximum cosine similarity for
making a model assignment (CA) / class assignment (FA) and then computing the
accuracy between predicted class and target class.

Multiple Clients. In the current work, we consider two clients (Client A and
Client B) for evaluation. However, the amount data available to each client is
the same. In future work we plan to study the impact of the varying proportion
of samples assigned to the clients for model training and assignment. Our idea
is easily scalable to many more clients, and is not limited to the two client
scenario presented here.

###  3.1 Coarse-level dataset assignment (CA)

In Table 2, we compare the results for the coarse level dataset assignment
with Sharma et al. , (2019). In case of Sharma et al. , (2019), the authors
use minimum reconstruction error (i.e. MSE) as the criteria for making the
sample assignment when the client shares the raw data with the server.

These results suggest that autoencoders trained on non-overlapping datasets at
the client or server side perform similarly to Sharma et al. , (2019) \- where
the client shares the raw data with the server. Note we are not sharing our
raw data - but only hidden representation. Further, we note that autoencoders
are effective in learning the underlying representation of the dataset even if
the client and server AE’s are trained on different datasets. This is a
surprising result, as training with non-overlapping data seems to lead to
representations that are close via cosine similarity. In this work, we ensure
models are initialized with the same seed, which encourages the models to
converge to similar representations as long as the training samples are coming
from the same underlying distribution. Otherwise, we expect the order of
learned filters would not be preserved and cosine similarity would fail. We
note that both Client A and Client B are 99% accurately assigned to their
corresponding autoencoders.

Note that in the financial services industry, we have a similar situation
where we are interested in finding an expert method for handling our specific
problem at hand without loss of confidential information. A similar example
dataset we have our experimental setup is REUTERS that contains newswire
articles, which may be similar to financial analysts reports or other valuable
text-based data that may not want to be shared with third parties. We think
our approach to finding an expert model without sharing the raw data can
easily be deployed to the financial industry where confidential information
may not want to be shared in raw form. Sharing encoded representations
provides some weak privacy, but also has other benefits such as lower
bandwidth requirements leading to decreased latency and more efficient storage
of data being shared between client and server.

Table 2: Coarse-level dataset assignment using cosine-similarity as the
assignment metric for computing accuracy (%). Note that Sharma et al. , (2019)
use MSE loss as the assignment metric.

| MNIST | STL-10 | HAR | REUTERS | NLOS | DB | Average
---|---|---|---|---|---|---|---
Client A (Sharma et al. ,, 2019) | 100.0 | 100.0 | 100.0 | 99.64 | 99.92 | 96.49 | 99.34
Client A (ours) | 99.36 | 99.93 | 99.80 | 96.36 | 99.59 | 100.0 | 99.30
Client B (Sharma et al. ,, 2019) | 100.0 | 100.0 | 100.0 | 99.56 | 99.89 | 95.36 | 99.13
Client B (ours) | 99.80 | 99.96 | 99.80 | 96.40 | 98.28 | 100.0 | 98.72

###  3.2 Fine-grained class assignment (FA)

In Table 3, we compare the results for the fine-grained class assignment with
Sharma et al. , (2019). It is interesting to observe that the autoencoder
trained on the client-side and the server-side seems to perform well even if
they are trained independently with different samples although coming from the
same underlying distribution - this shows that autoencoders are effective at
learning the class identity representation. A drop in performance for our
method is expected, as the client’s AE model has access to much smaller set of
examples than the server.

Table 3: Fine-grained class assignment using cosine-similarity as the
assignment metric for computing accuracy (%).

Dataset | #C | Client A | Client B
---|---|---|---
MNIST (Sharma et al. ,, 2019) | 10 | 84.36 | 83.40
MNIST (ours) | 10 | 71.4 | 71.64
NLOS (Sharma et al. ,, 2019) | 3 | 71.78 | 71.26
NLOS (ours) | 3 | 59.40 | 52.40
DB (Sharma et al. ,, 2019) | 3 | 41.47 | 44.41
DB (ours) | 3 | 44.30 | 53.33

##  4 Conclusion

In this work, we propose a novel ExpertMatcher based model selection where the
raw data of remote clients are not shared with the central server. In contrast
to the previous method, the server and client both train a model independently
and the intermediate hidden representations are shared between them for either
coarse or fine-grained matching respectively. We believe our approach to
finding an expert model via hidden representations provides a path for a new
paradigm of distributed machine learning architectures where sharing of raw
data is not feasible.

Acknowledgements V. Sharma would like to thank Karlsruhe House of Young
Scientists (KHYS) for funding his research stay at MIT.

## References

  * Ahmed et al. , (2016) Ahmed, Karim, Baig, Mohammad Haris, & Torresani, Lorenzo. 2016.  Network of experts for large-scale image categorization.  Pages 516–532 of: European Conference on Computer Vision.  Springer.
  * Aljundi et al. , (2017) Aljundi, Rahaf, Chakravarty, Punarjay, & Tuytelaars, Tinne. 2017.  Expert gate: Lifelong learning with a network of experts.  Pages 3366–3375 of: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
  * Anguita et al. , (2013) Anguita, Davide, Ghio, Alessandro, Oneto, Luca, Parra, Xavier, & Reyes-Ortiz, Jorge Luis. 2013.  A public domain dataset for human activity recognition using smartphones.  In: Esann.
  * Coates et al. , (2011) Coates, Adam, Ng, Andrew, & Lee, Honglak. 2011.  An analysis of single-layer networks in unsupervised feature learning.  Pages 215–223 of: Proceedings of the fourteenth international conference on artificial intelligence and statistics.
  * Graham, (2015) Graham, Ben. 2015.  Kaggle diabetic retinopathy detection competition report.  University of Warwick.
  * Hinton et al. , (2015) Hinton, Geoffrey, Vinyals, Oriol, & Dean, Jeff. 2015.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531.
  * Jacobs, (1995) Jacobs, Robert A. 1995.  Methods for combining experts’ probability assessments.  Neural computation, 7(5), 867–888.
  * Jacobs, (1997) Jacobs, Robert A. 1997.  Bias/variance analyses of mixtures-of-experts architectures.  Neural computation, 9(2), 369–383.
  * Jacobs et al. , (1991) Jacobs, Robert A, Jordan, Michael I, Nowlan, Steven J, Hinton, Geoffrey E, et al. . 1991.  Adaptive mixtures of local experts.  Neural computation, 3(1), 79–87.
  * LeCun et al. , (1998) LeCun, Yann, Bottou, Léon, Bengio, Yoshua, Haffner, Patrick, et al. . 1998.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11), 2278–2324.
  * Lewis et al. , (2004) Lewis, David D, Yang, Yiming, Rose, Tony G, & Li, Fan. 2004.  Rcv1: A new benchmark collection for text categorization research.  Journal of machine learning research, 5(Apr), 361–397.
  * Sharma et al. , (2019) Sharma, Vivek, Vepakomma, Praneeth, Swedish, Tristan, Chang, Ken, Kalpathy-Cramer, Jayashree, & Raskar, Ramesh. 2019.  ExpertMatcher: Automating ML Model Selection for Users in Resource Constrained Countries.  In: NeurIPS workshop on Machine Learning for the Developing World (ML4D).
  * Tancik et al. , (2018) Tancik, Matthew, Satat, Guy, & Raskar, Ramesh. 2018.  Flash photography for data-driven hidden scene recovery.  arXiv preprint arXiv:1810.11710.
  * Vepakomma et al. , (2018a) Vepakomma, Praneeth, Swedish, Tristan, Raskar, Ramesh, Gupta, Otkrist, & Dubey, Abhimanyu. 2018a.  No Peek: A Survey of private distributed deep learning.  arXiv preprint arXiv:1812.03288.
  * Vepakomma et al. , (2018b) Vepakomma, Praneeth, Gupta, Otkrist, Swedish, Tristan, & Raskar, Ramesh. 2018b.  Split learning for health: Distributed deep learning without sharing raw patient data.  arXiv preprint arXiv:1812.00564.
  * Vepakomma et al. , (2019) Vepakomma, Praneeth, Gupta, Otkrist, Dubey, Abhimanyu, & Raskar, Ramesh. 2019.  Reducing leakage in distributed deep learning for sensitive health data.  arXiv preprint arXiv:1812.00564.