# 3D shape reconstruction of semi-transparent worms

Thomas P. Ilett Omer Yuval11footnotemark: 1 Thomas Ranner11footnotemark: 1
Netta Cohen11footnotemark: 1 † David C. Hogg11footnotemark: 1 †
University of Leeds, Leeds, United Kingdom {T.Ilett, O.Yuval, T.Ranner,
N.Cohen<EMAIL_ADDRESS>
Funding This work was supported by University of Leeds and EPSRC.
Author contributions Conceptualisation, Methodology, Formal analysis,
Investigation, Software, Visualisation: TPI. Data curation, Validation: TPI,
OY. Writing: TPI (original), all (review and editing). Funding acquisition,
Supervision: NC, DCH, TR. $\dagger$ Equal contribution.
Acknowledgements Additional thanks to Matan Braunstein (for help with
LABEL:fig:overview), Robert I. Holbrook (data), Felix Salfelder (discussions
and data), Lukas Deutz (discussions) and Jen Kruger (proof reading).
Data availability Supplementary movies are available here:
https://doi.org/10.6084/m9.figshare.22310650.

###### Abstract

3D shape reconstruction typically requires identifying object features or
textures in multiple images of a subject. This approach is not viable when the
subject is semi-transparent and moving in and out of focus. Here we overcome
these challenges by rendering a candidate shape with adaptive blurring and
transparency for comparison with the images. We use the microscopic nematode
_Caenorhabditis elegans_ as a case study as it freely explores a 3D complex
fluid with constantly changing optical properties. We model the slender worm
as a 3D curve using an intrinsic parametrisation that naturally admits
biologically-informed constraints and regularisation. To account for the
changing optics we develop a novel differentiable renderer to construct images
from 2D projections and compare against raw images to generate a pixel-wise
error to jointly update the curve, camera and renderer parameters using
gradient descent. The method is robust to interference such as bubbles and
dirt trapped in the fluid, stays consistent through complex sequences of
postures, recovers reliable estimates from blurry images and provides a
significant improvement on previous attempts to track _C. elegans_ in 3D. Our
results demonstrate the potential of direct approaches to shape estimation in
complex physical environments in the absence of ground-truth data.

##  1 Introduction

Many creatures such as fish, birds and insects move in all directions to
search and navigate volumetric environments. Acquiring 3D data of their motion
has informed models of locomotion, behaviour and neural and mechanical control
[22, 3]. While technological advances have made the collection of large
quantities of multi-viewpoint visual data more attainable, methods for
extracting and modelling 3D information remain largely domain-dependant as few
species share common geometric models or exist within the same spatial and
temporal scales [41, 65, 54, 50, 11, 37, 14, 4, 26]. Furthermore, while humans
and some domesticated animals [60, 30] may act naturally while wearing special
markers, marker-less observations of many species makes feature extraction
more challenging and means pose estimation generally lacks ground-truth data
[48].

As a case study in marker-less 3D shape reconstruction, we consider _C.
elegans_ , a hair-thick, $\mathord{\sim}$1\text{\,}\mathrm{mm}$$ long animal
with a simple tapered cylinder shape, which can be constructed from a midline
“skeleton”. In the wild, _C. elegans_ can be found in a wide range of complex
3D environments, _e.g_. decomposing organic matter, with continually changing
physical properties [17, 15, 46]. However, to date, experiments have focused
nearly exclusively on locomotion on a plane, limiting insight to the
constrained, planar behaviours.

We obtained a large dataset (4 hours 53 minutes $\simeq$ 440,000 frames at
25Hz) of experimental recordings of individual worms moving freely inside a
glass cube filled with a gelatin solution. The cube is positioned between
three nearly-orthogonal static cameras fitted with telecentric lenses. Initial
pinhole camera model parameter estimates are provided [45] but are imprecise
and require continuous adjustment across the course of a recording to account
for small vibrations and optical changes to the gel. We aim to simultaneously
reconstruct a 3D shape and find corrected camera parameters to match these
recordings in a process akin to bundle adjustment [56].

3D reconstruction typically involves the identification and triangulation of
common features from multiple viewpoints or the synthesis of full images
including texture and shading information to match given scenes [21, 47, 66,
16]. Imaging animals with length $\mathord{\sim}$1\text{\,}\mathrm{mm}$$
requires sufficient magnification, but simultaneously capturing long-term
trajectories up to 25 minutes requires a large volume of view (10-20 worm
lengths per axis). As the worm explores the cube it frequently appears out of
focus in one or more of the cameras. Air bubbles and dirt trapped in the gel
along with old tracks are difficult to differentiate from the transparent
worm, particularly at the tapered ends. Self occlusion invariably appears in a
least one view, where hidden parts darken the foreground while the ordering of
fore/back-parts is not discernible. As the semi-transparent and self-occluding
subject moves in the volume, photometric information in one view bears little
relevance to the appearance in the others making feature identification and
photometric matching particularly challenging. We found that standard
approaches may suffice for limited sub-clips, but lose parts of the object or
fail catastrophically for much of the data and the solution requires a degree
of adaptation.

We present an integrated “project-render-score” algorithm to obtain a midline
curve for each image-triplet (LABEL:fig:overview). Discrete curve vertices are
_projected_ through a triplet of pinhole camera models, _rendered_ to produce
an image-triplet for direct comparison against the recorded images and
_scored_ according to their intersection with worm-like pixels in all three
views. The differentiable renderer stacks 2D super-Gaussian blobs at the
projected locations of each vertex to approximate the transparency along the
worm, accounting for the variable focus and providing soft edges that direct
the geometric model towards the midline. The scoring allows the detection of
incongruities and keeps the curve aligned to the worm in all views.
Regularisation terms ensure smoothness along the body and in time. Curve,
camera and rendering parameters are jointly optimised using gradient descent
to convergence. Once the worm shape has been resolved, it is generally only
lost during image degradation or significant self-occlusions that make the
posture unresolvable by eye.

In summary, our main contributions are:

  * •

A robust pipeline for 3D posture reconstruction of a freely deforming semi-
transparent object from noisy images.

  * •

A novel viewpoint renderer to capture optical distortions and transparency.

  * •

A feature-free bundle adjustment algorithm using direct image comparison and
gradient descent.

##  2 Related work

#### Bundle adjustment (BA)

is a procedure to jointly optimise 3D geometry and camera parameters [56, 21].
BA typically identifies common features of an object from multiple viewpoints
in order to minimise a prediction error between projections of the
corresponding 3D points and their 2D observations. BA is frequently used in
conjunction with other methods to find camera parameters using multiple images
of a 3D calibration object with known control points or for fine-tuning
results [36, 57, 59, 23, 13, 40].

Feature detection converts photometric information into image coordinates. In
BA, coordinates of common features are used to solve a geometric optimisation
problem. Photometric bundle adjustment methods additionally require objects to
have the same appearance in all views [18, 12]. Our method is entirely
photometric, as such differing from BA. As our objects appear differently
across views, all pixel information is used and the geometry is solved
intrinsically.

#### Pose estimation

Deep network approaches have proved well-suited to 2D human-pose estimation as
they are potent feature extractors and large annotated training sets are
available [1, 55, 51]. For 3D postures, ground truth multi-view datasets are
less common. Recent progress [35] relies on end-to-end architectures [29, 42,
19, 27, 32, 61] or splitting the problem into 2D pose estimation and then
constructing the 3D pose [38, 10]. Despite similar approaches used for non-
human pose estimation, the huge variability in scales and shapes among species
introduces a variety of challenges [26]. Motion capture in controlled settings
with markers (providing ground truth skeleton and joint angle data for humans,
horses and dogs [60, 30]), are not available for most animals. Generalised
mesh surfaces may be used, but often require multiple views and thousands of
parameters, and do not guarantee consistency through time. In contrast,
approximating an animal shape using a few-parameter morphable model can be
both tractable and robust. Successful examples include swimmers [43, 9], birds
[58, 27], mammals [2, 6, 28, 39] and generic quadrupeds [67, 7]. However,
these methods expect opaque subjects with consistent textural appearances
between views.

_C. elegans_ has a simple geometric shape that can be well reconstructed from
a midline skeleton and parametrised by curvature values along the body (see
Sec. 3). This is the deformable template we look to fit to the data. Despite
the apparent simplicity, each vertex of the discretised curve has two degrees
of freedom (two curvature values) and as we use 128 vertices, our model is
highly deformable and requires many parameters (although smoothness
regularisation simplifies the problem somewhat). In contrast to deep-learning
approaches, our model includes only a small number of explainable parameters
and direct optimisation avoids lengthy training and dataset requirements.

####  _C. elegans_

Numerous freely available software packages are capable of simultaneous
tracking and skeletonising single or multiple worms in 2D using inexpensive
microscopic imaging [52, 62, 44, 25, 53, 5] (see [24] for a review). Most of
these skeletonisers combine image segmentation to separate the animal from the
background with thinning of the mask to some midline pixels and fitting a
spline.

The 3D reconstruction problem has received relatively little attention. Using
at first two views [34] and then three, Kwon _et al_. [33] designed a
motorised stage coupled with a real-time tracker to keep a worm in focus under
high magnification in a 3D environment while capturing trajectories of up to 3
minutes. Thresholded images are lifted into 3D, intersected in voxel space and
thinned [20] to produce a final skeleton. Kwon _et al_. omit camera modelling
and assume perfectly parallel projections – assumptions that result in large
errors for the data we use. Shaw _et al_. [49] employed light field microscopy
to generate depth maps alongside images from a single viewpoint. A midline
skeleton is generated by fitting a spline to the 3D coordinates of the central
voxels. However, self-occlusions cannot be resolved and only relatively planar
postures were investigated.

Salfelder _et al_. [45] and Yuval [63] both present 3D reconstruction
algorithms using the three-camera set up and calibration described in [45]. In
Salfelder _et al_. [45], a neural network is trained to identify 2D midlines
from individual camera images before lifting into 3D voxel space. To account
for changing camera parameters, a relative axial shift $(dx,dy,dz)$ is
optimised for each frame-triplet to maximise the voxel intersection before
thinning. Remaining voxel coordinates are used as control points to fit a
curve using a finite-element formulation. This approach works well when the
midline is well detected in each of the views, but can fail on occluded
postures or low-resolution, blurry images.

Yuval [63] uses a neural network to track head and tail points in 3D lab
coordinates and a curve is fit between these fixed end points using a hill-
climbing optimisation algorithm. Scoring is based on curve smoothness and
pixel intensities at the projected curve points. This method works well when
the head and tail are correctly identified but struggles, or requires manual
correction, otherwise.

In our approach we find that incorporating the camera model parameters into
the optimisation results in more robust and accurate results. This extends the
idea proposed in Salfelder _et al_. [45] that adjusting the relative positions
of the cameras could result in large gains in accuracy. It is likely that the
relative shift adjustments, presented there, account for the changing optical
properties.

Figure 1: The 3D curve is traced out from initial point $P_{n_{0}}$ and
orientation frame $(T_{n_{0}},M^{1}_{n_{0}},M^{2}_{n_{0}})$. The index $n_{0}$
of the initial point is drawn from a normal distribution at each iteration to
prevent kinks developing through repeated use of the same starting point. The
final curve $\hat{P}$ is computed in two parts by integrating the Bishop
equations with curvature $K$ towards the head and tail separately.

##  3 Geometric model

Nematode shapes can be well approximated by a tapered cylinder and computed
from a midline. We construct the midline curve in 3D using an object-centric
parametrisation, separating shape from position and orientation to allow us to
easily constrain and regularise the shape to stay within biologically-
reasonable bounds. We discretise the curve into $N$ equidistant vertices and
encode the posture in curvature $K\in\mathbb{R}^{N\times 2}$ and length
$l\in\mathbb{R}$ that fully define the shape up to a rigid-body
transformation.

We express the 3D curve using the Bishop frame [8], given by $TM^{1}M^{2}$
where $T$ is the normalised tangent of the curve and $M^{1},M^{2}$ form an
orthogonal basis along the midline. At vertex $n$, the curvature is
$K_{n}=(m^{1}_{n},m^{2}_{n})$, where $m_{n}^{1},m_{n}^{2}\in\mathbb{R}$ are
the curvature components along $M^{1},M^{2}$. (The more familiar Frenet frame
is less stable as it is undefined at zero-curvature points.) Numerical
integration of a system of difference equations from starting point
$P_{\mathrm{init}}$ and initial orientation
$(T_{\mathrm{init}},M^{1}_{\mathrm{init}},M^{2}_{\mathrm{init}})$ yields the
curve path $P\in\mathbb{R}^{N\times 3}$. See Appendix A for details.

During optimisation, errors accumulate near the starting point,
$P_{\mathrm{init}}$, resulting in either parts of the curve moving faster than
other or kinks developing (even with strong regularisation). To resolve this
we sample an initial vertex index $n_{0}$ from a Gaussian distribution
(subject to rounding) centred at the middle index at every optimisation step.
Setting the starting point $P_{\mathrm{init}}=P_{n_{0}}$ has the effect of
continually shifting the discontinuity so kinks are never given the
opportunity to develop (Fig. 1). Summarising the integration as $F$, the 3D
curve is generated from the parameters:

$\displaystyle(\hat{P},\hat{T},\hat{M}^{1})$
$\displaystyle=F\left(P_{n_{0}},T_{n_{0}},M^{1}_{n_{0}},K,l,n_{0}\right).$ (1)

Each gradient update adjusts all curvature values $K$ but the position and
orientation only at the randomly selected $n_{0}$ vertex
$(P_{n_{0}},T_{n_{0}},M^{1}_{n_{0}})$. Updating $(P,T,M^{1})$ at only this
vertex produces a $P$ that is inconsistent with the updated $K$. Therefore,
after applying gradient updates we re-compute the full curve and orientation
from $n_{0}$ and set $(P,T,M^{1})$ to the output
$(\hat{P},\hat{T},\hat{M^{1}})$.

Since the curve describes a biological creature, we constrain the length $l$
to $(l_{\text{min}},l_{\text{max}})$ and limit the curvature by $|K_{n}|<2\pi
k_{\text{max}}$. The values of $(l_{\text{min}},l_{\text{max}})$ we use vary
depending on magnification but the bounds do not need to be tight and are in
the range $0.5$–$2\text{\,}\mathrm{mm}$. The curvature constraint
$k_{\text{max}}$ is set by considering the number of circle achieved by a
constant curvature curve and is fixed at $3$.

##  4 Project, Render, Score

The core of the optimisation pipeline is separable into three main stages;
project, render and score. The 3D curve $\hat{P}$ generated in Eq. 1 is
_projected_ through the camera models into 2D points that are _rendered_ into
images and then _scored_ against the three views.

###  4.1 Project

The cameras are modelled using a triplet of pinhole camera models with
tangential and radial distortion that project 3D points into image planes
using perspective transformations (see Appendix B). Each pinhole camera model
offers a simple (15 parameters, $\\{\eta_{c}\\}$), tractable, approximation to
the optical transformation. We also include relative shifts along the local
coordinate axes, $\eta^{s}=(dx,dy,dz)$, shared between the three models, as
proposed by Salfelder _et al_. [45]. Initial camera coefficients for the
triplet-model are provided along with the recordings and typically give root
mean squared reprojection errors up to 10 pixels ($\sim\mathcal{O}(\text{worm
radius})$).

Due to the initial calibration errors and changes in optical properties as the
gelatin sets and is disturbed by the worms we re-calibrate the cameras at
every frame by including the camera parameters in the optimisation step. To
avoid an under-determined problem, after we have found a configuration that
supports good reconstructions for a recording we fix all but the $\eta^{s}$
parameters. Interestingly, we still see changes (up to
$30$\mathrm{p}\mathrm{x}$\sim$0.15\text{\,}\mathrm{mm}$$) in $\eta^{s}$ but as
this relates to the relative positioning it does not affect the posture
reconstruction or long-term trajectories.

Projecting the 3D curve $\hat{P}$ through the camera-triplet model $\Gamma$
with parameters $\eta=\\{\eta_{0},\eta_{1},\eta_{2},\eta^{s}\\}$ generates 2D
image points per view, which we combine as
$Q=\Gamma(\hat{P},\eta)\in\mathbb{R}^{3\times N\times 2}$.

###  4.2 Render

Figure 2: The rendering stage generates super-Gaussian blobs at each vertex
position on the image. The shape of the blobs depends on the optimisable
parameters: the scale $\sigma$, the intensity $\iota$ and the exponent used in
the Gaussian $\rho$. $\sigma$ and $\iota$ are tapered down to fixed minimum
values at the head and tail. The effects of varying these parameters from a
converged solution (blue curves) are shown above (green curves) and below
(orange curves) each.

In order to evaluate the reconstruction directly against the raw data, we
render the projected 2D midline points into images using optimisable shape and
rendering parameters. Since worm bodies are well approximated by tapered
cylinders, in theory we only require maximum and minimum radius values and a
tapering function. However, _C. elegans_ are semi-transparent – increasingly
so at the head and tail – and their internal anatomy has varying optical
properties that diffract and distort the light. These challenges are further
exacerbated by the worms often being out of focus in at least one of the
views, therefore even an anatomically accurate model stands little chance of
being correctly resolved.

We render realistic images by combining 2D super-Gaussian functions centred on
each projected vertex. Crucially, we allow the rendering parameters to differ
between cameras since the animal seldom has the same photometric qualities in
different views. We optimise three parameters for each camera view $c$:
$\sigma_{c}\in\mathbb{R}$ controls the spread, $\iota_{c}\in\mathbb{R}$ scales
the intensity, and $\rho_{c}\in\mathbb{R}$ sharpens or softens the edges (Fig.
2). To capture the tapered shape we weight $\sigma_{c}$ and $\iota_{c}$ from
their optimisable values along the middle $60\%$ to minimum values
$\sigma_{\text{min}}$ and $\iota_{\text{min}}$ at the ends and define the
tapered outputs $\bar{\sigma}_{c}\in\mathbb{R}^{N}$ and
$\bar{\iota}_{c}\in\mathbb{R}^{N}$ (Appendix C). $\sigma_{\text{min}}$ and
$\iota_{\text{min}}$ are manually fixed for each recording to account for
different magnification factors and worm size variability.

For each camera index $c$ and vertex index $n$ we define the rendered blob
$B_{c,n}\in\mathbb{R}^{w\times w}$ (image size $w$) for pixel $(i,j)$ as:

$B_{c,n}(i,j)=\bar{\iota}_{c,n}\exp{\left[-\left(\frac{(i-Q_{c,n,0})^{2}+(j-Q_{c,n,1})^{2}}{2\bar{\sigma}_{c,n}^{2}}\right)^{\rho_{c}}\right]}.$
(2)

The stacks of blobs are combined to generate the complete renderings
$R\in\mathbb{R}^{3\times w\times w}$ by taking the maximum pixel value across
all blobs: for pixel $(i,j)$,

$R_{c}(i,j)=\max\left\\{B_{c,n}(i,j)\right\\}_{n=0,\dots,N-1}.$ (3)

The orientation of the body directly affects the pixel intensity of both raw
and rendered images. When pointing directly at a camera the peaks of the blobs
cluster closely together and appear as a high-intensity (opaque) circle.
Pointing laterally causes the peaks to spread out on the image revealing more
of the lower-intensity tails. In both situations our blob-rendering approach
approximates transparency effects in the raw images without the need to model
complex intensity-orientation responses. Moreover, super-Gaussian blobs allow
sharp outlines to be produced in one view by using a large exponent and flat-
top blobs, and blurry images to be produced for another, using low intensity
and high variance.

###  4.3 Score

Figure 3: The 3D curve points are scored individually according to how well
they match the three views. The triplet of blobs associated with vertex $n$
($B_{.,n}$) are multiplied with the images $I$ and summed. We take the minimum
of the three sums and then taper these values from the midpoint-out.

In order to evaluate how well the curve represents the worm we require a way
of distinguishing between worm-pixels and non-worm pixels such as dirt,
bubbles, old tracks and even other worms. When the animal truly intersects
with environmental interference it can be impossible to differentiate between
the two, but in the majority of cases there exists a gap between the worm and
the noise that is visible in at least one of the views. By ensuring that the
curve corresponds to a single contiguous pixel mass in _all_ of the images we
are able to safely ignore other artefacts (Fig. 3).

To detect if the curve is bridging a gap, each vertex $\hat{P}_{n}$ is scored
by correlating its corresponding blobs $B_{.,n}$ (Sec. 4.2) with the images
$I$. The raw score $S_{n}\in\mathbb{R}$ is defined:

$S_{n}=\min\left\\{\frac{\sum_{i,j}B_{c,n}\cdot
I_{c}}{\bar{\sigma}_{c,n}\bar{\iota}_{c,n}}\right\\}_{c=0,1,2}$ (4)

where $\cdot$ is element-wise multiplication and the sum is taken over the
image dimensions. By taking the minimum we ensure that vertices failing to
match pixels in any one of the views will receive low scores regardless of how
well they match pixels in the other views.

If the curve is bridging two disjoint groups of pixels that are visible in all
three views this will present as two peaks in $S$. Since we are only
interested in finding one object we restrict the scores to contain just one
peak by tapering $S$ from the middle-out to form the intermediate
$S^{\prime}$. Finally we normalise $S^{\prime}$ to get scores $\hat{S}$
relative to the peak:

$\displaystyle S^{\prime}_{n}$
$\displaystyle=\begin{cases}\min\\{S_{n},S^{\prime}_{n+1}\\}&0\leq n<N/2\\\
S_{n}&n=N/2\\\ \min\\{S_{n},S^{\prime}_{n-1}\\}&N/2<n<N\end{cases}$ (5)
$\displaystyle\hat{S}$
$\displaystyle=\frac{S^{\prime}}{\max_{n}\\{S^{\prime}\\}}.$ (6)

The final score profile $\hat{S}$ provides insight into how well the curve
matches a contiguous pixel mass across all three views and how evenly that
mass is distributed.

#### Masking

Figure 4: The noisy input images are cleaned by applying masks that force
pixel-errors to be local to the current estimate. The blobs $B$ are scaled by
the relative scores $\hat{S}$, combined using the maximum pixel value across
blobs and thresholded to form the masks $M$. The masks are applied to the raw
input images $I$ to generate the targets: $I^{\star}$. Masking ensures only a
single contiguous pixel mass is detected. Without it, parts of the
reconstruction can “stick” to nearby bubbles and other artefacts as shown
below.

From the score profile $\hat{S}$ we identify image areas that are more likely
to contain the pixel masses that correspond to the worm. Masks
$M\in\mathbb{R}^{3\times w\times w}$ applied to the input, $I^{\star}=M\cdot
I$, focuses attention (and gradient) to only these areas of interest,
consistently across all three views and exclude interference outside the masks
(Figs. 4 and D). Pixel intensities outside the masks are significantly
reduced, but not zeroed in order to avoid stagnation in case the
reconstruction completely misses the worm.

#### Centre-shifting

Figure 5: As the animal moves along the path of its midline the tail may be
left behind (left column). This can be identified from an unbalanced score
profile $\hat{S}$. By periodically shifting the curve along its length (adding
new curvature values at one end and discarding from the other) the centroid
index ($\bar{n}$) of the scores can be centred. Gradient descent optimisation
then updates the new curvature values so the curve matches the target (right
column).

The scores $\hat{S}$ also indicate the relative positioning of the curve over
the target object. As the curve aligns with a pixel mass, vertices with high
scores (apparently “converged”) tend to lock into place thus hindering
convergence of the rest of the object. For each frame, we use the previous
frame solution as the starting point, so the majority of points rapidly
converge. However, errors introduced at the tips remain as they are
insufficient to generate the collective shift required. The effect can easily
be identified from an unbalanced score profile (Fig. 5) and rectified by
periodically shifting the curve along its length between gradient descent
optimisation steps (Appendix E).

##  5 Optimisation

The main pixel-loss to be minimised is defined as:

$\mathcal{L}_{\text{px}}=\frac{1}{3w^{2}}\sum_{c,i,j}(R_{c}(i,j)-I^{\star}_{c}(i,j))^{2}.$
(7)

To improve head and tail detection we also minimise a scores-loss,

$\displaystyle\mathcal{L}_{\text{sc}}$
$\displaystyle=\frac{\max(S^{\prime})N}{\sum_{n}S^{\prime\prime}_{n}},\text{
where}$ (8) $\displaystyle S^{\prime\prime}_{n}$
$\displaystyle=S^{\prime}_{n}\left(\frac{2n-(N-1)}{N-1}\right)^{2},$ (9)

that is quadratically weighted towards the tips where the scores are naturally
lower due to the transparency.

In addition we include a number of regularisation terms. To keep the curve
smooth we define

$\mathcal{L}_{\text{sm}}=\sum_{n=1}^{N-1}|K_{n}-K_{n-1}|^{2},$ (10)

where $|\cdot|$ is the $l^{2}\text{-norm}$. To ensure all parameters change
smoothly between frames we set

$\mathcal{L}_{\text{t}}=\sum_{x\in\\{l,K,\hat{P},\eta,\sigma,\iota,\rho\\}}|x^{\text{prev}}-x|^{2},$
(11)

where $x^{\text{prev}}$ refers to the frozen value of the variable from the
previous frame. And to avoid self-intersections, we use

$\displaystyle d_{n,m}$ $\displaystyle=|\hat{P}_{n}-\hat{P}_{m}|,$ (12)
$\displaystyle d^{\prime}_{n,m}$
$\displaystyle=\frac{1}{3}\sum_{c}\bar{\sigma}_{c,n}+\frac{1}{3}\sum_{c}\bar{\sigma}_{c,m},\text{
and }$ (13) $\displaystyle\mathcal{L}_{\text{i}}$
$\displaystyle=\sum_{n=0}^{N-N/k_{\text{max}}-1}\sum_{m=n+N/k_{\text{max}}}^{N-1}\begin{cases}\frac{d^{\prime}_{n,m}}{d_{n,m}},&\text{if
}d_{n,m}<d^{\prime}_{n,m}\\\ 0,&\text{otherwise.}\end{cases}$ (14)

A loss is incurred, $\mathcal{L}_{\text{i}}>0$, when two points which are
sufficiently far apart ($>N/k_{\text{max}}$) along the curve come within a
distance defined by the sum of their mean rendering variances (since these
approximate the worm’s radius). Eq. 14 forces the algorithm to find postures
that are always feasible even during self-occlusions and complex manoeuvres.

The losses are combined in a weighted sum to yield the final optimisation
target:

$\displaystyle\mathcal{L}=\omega_{\text{px}}\mathcal{L}_{\text{px}}+\omega_{\text{sc}}\mathcal{L}_{\text{sc}}+\omega_{\text{sm}}\mathcal{L}_{\text{sm}}+\omega_{\text{t}}\mathcal{L}_{\text{t}}+\omega_{\text{i}}\mathcal{L}_{\text{i}}.$
(15)

Values of $\omega$ used in our experiments are listed in Tab. S8.

To achieve robust reconstructions it is important that the curve parameters
learn fastest, then the rendering parameters and finally the camera
parameters. Imposing this hierarchy of rates ensures camera model stability
and prevents the renderer from over-blurring the edges (as it tries to “reach”
the pixels). Thus, movement between frames is primarily captured through curve
deformations. We use learning rates $\lambda_{p}=$110-3$$ for the curve
parameters $\\{P,T,M^{1},K,l\\}$, $\lambda_{r}=$110-4$$ for the rendering
parameters $\\{\sigma,\iota,\rho\\}$ and $\lambda_{\eta}=$110-5$$ for the
camera parameters $\eta$.

The curve is initialised as a small
($\mathord{\sim}$0.2\text{\,}\mathrm{mm}$$), randomly oriented straight line
centred in the field of view of all three cameras. We slowly increase the
length to $l_{\text{min}}$ over the first $200\text{-}500$ steps as the curve
gets positioned and orientated.

The pipeline is constructed using PyTorch [64] and the loss minimised is using
Adam [31] with periodic centre-shifting of the curve vertices. Learning rates
are decreased by a factor of $0.8$ for every $5$ steps taken without
improvement in $\mathcal{L}$ to a minimum of $110-6$ until convergence is
detected. Subsequent frames are instantiated with the solution from the
previous frame for efficiency and to maintain consistency through complex
sequences of self-occluding postures. Example videos showing the effects of
varying some of the options on the optimisation are described in Appendix I.

##  6 Results

Figure 6: Validation against 487 manual annotations. At the top we show an
example of an annotated frame (left, orange) alongside a projection of our
matching 3D midline (right, blue). Below we plot the sample averages $\pm
2\text{std}$. We find our midlines are consistently close to annotated points
(blue curve), but annotations typically extend further into the head and tail
regions (orange curve). Figure 7: A comparison between our Midline Finder
(MF), Yuval’s Worm-Tracker 3D (WT3D) [63] and Salfelder _et al_.’s ‘reconst’
[45] methods across a single trial
($\mathord{\sim}$13\text{\,}\mathrm{min}$$). In the majority of cases our
method generates midlines that better match the data (lower pixel losses,
$\mathcal{L}_{\text{px}}$). We show moving averages over 25 frames
($\mathord{\sim}$1\text{\,}\mathrm{s}$$) with shaded areas indicating $\pm
2\text{std}$. Figure 8: The rendering parameters change continually over the
course of a recording to capture optical changes. Clear images (_e.g_. early
frames in cameras 0 and 1, switching to late frames in camera 2) are
consistent with small values of $\sigma$ and large values of $\rho$. Blurry
images (early camera 2, late camera 1) use high $\sigma$ and small $\rho$. We
show moving averages over 25 frames ($\mathord{\sim}$1\text{\,}\mathrm{s}$$)
with shaded areas indicating $\pm 2\text{std}$. Example comparisons between
the renders (red) and raw images (grey) are shown on either side.

Using our method we generate high quality 3D midline reconstructions for 43 of
44 recordings. One fails due to excessive coiling of the worm. Significant
occlusions also occur during successful reconstructions and when combined with
loss of focus can cause the shape to be lost. Video clips of good and poor
reconstructions through challenging environmental conditions are described in
Appendix I along with ablation results to show benefits of each component.

We compare 2D reprojections of our midlines against 487 manual annotations
that were produced from single images in isolation and contain a varying
number of unordered points. We calculate the minimum distance from each
annotated point to any reconstructed point and vice-versa and find that our
midlines consistently come close ($\mathord{\sim}2\text{px}$) to hand-
annotated points (Fig. 6). Annotated points at the ends show an increased
distance ($\mathord{\sim}10\text{px}$) to our midline points. This shows that
our curves generally fall short of reaching the very tips of the worm by
$\sim\mathcal{O}(\text{worm radius})$.

Our method significantly outperforms previous methods developed using the same
dataset [45, 63] when evaluated against the manual annotations (Tab. S1), but
these only cover a selection of hand-picked examples. For a large-scale
comparison we take 3D midlines and camera parameters found by each method and,
using our pipeline, render them to generate comparable images (re-optimising
the render parameters for their midlines, see Appendix G). We skip the scoring
and masking and calculate $\mathcal{L}_{\text{px}}$. The results (Fig. 7) show
our method consistently produces shapes that more closely match the raw
images. The biggest advantage over previous approaches is the improvement in
robustness; we recover $4\text{\,}\mathrm{h}$ $37\text{\,}\mathrm{min}$ (ours)
versus $1\text{\,}\mathrm{h}$ $32\text{\,}\mathrm{min}$ [45] and
$45\text{\,}\mathrm{min}$ [63].

Fig. 8 shows the rendering parameters during a trial as the worm moves in and
out of focus in the different cameras. Clearer images result in smaller values
of $\sigma$ and larger values of $\rho$. The fluctuations in intensity $\iota$
are due in part to the posture of the worm in relation to the camera; when it
is pointing directly towards the camera we see higher values of $\iota$ used
to capture the darker image observed and when the shape is perpendicular to
the camera we see lower values of $\iota$ to emulate the worm’s transparency.
All three parameters work in tandem to produce the final effect.

##  7 Conclusion

We present a robust and reliable framework for the 3D reconstruction of a
microscopic, semi-transparent subject moving through a fluid and evaluate
against two other algorithms and manually annotations. The key contribution of
our approach – constructing unique differentiable renderings for each view –
allows us to solve shape reconstruction and camera parameter optimisation by
direct image comparison. This avoids feature extraction and correspondence
matching, and hence offers a powerful alternative when those approaches are
not well-suited, _e.g_. due to the variation in appearance between views.

Multi-view microscopic camera calibration, imaging through fluids and
parametric model fitting of semi-transparent subjects are challenges that have
received little attention in the literature. While we have focused here on
constructing a curve to fit a microscopic worm from three views, our method
could be applied to the 3D reconstruction of arbitrary shape models at any
scale using any number of viewpoints. Rendering points with adaptable super-
Gaussian functions presents an effective solution to transparency and focal
issues, but more generally, our results indicate that our direct optimisation
approach may offer an effective alternative to contemporary methods for 3D
approximation of generic objects from a limited number of silhouette-like
images.

## References

  * [1] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele.  2D human pose estimation: New benchmark and state of the art analysis.  In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 3686–3693. IEEE, June 2014.
  * [2] Praneet C. Bala, Benjamin R. Eisenreich, Seng Bum Michael Yoo, Benjamin Y. Hayden, Hyun Soo Park, and Jan Zimmermann.  Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio.  Nat Commun, 11(1):4560, Sept. 2020.
  * [3] Jerrold L Belant, Joshua J Millspaugh, James A Martin, and Robert A Gitzen.  Multi-dimensional space use: The final frontier.  Front. Ecol. Environ., 10(1):11–12, Feb. 2012.
  * [4] Florian Berlinger, Melvin Gauci, and Radhika Nagpal.  Implicit coordination for 3D underwater collective behaviors in a fish-inspired robot swarm.  Sci. Robot., 6(50):eabd8668, Jan. 2021.
  * [5] Stefano Berri, Jordan H. Boyle, Manlio Tassieri, Ian A. Hope, and Netta Cohen.  Forward locomotion of the nematode C. elegans is achieved through modulation of a single gait.  Hfsp J., 3(3):186–193, June 2009.
  * [6] Benjamin Biggs, Oliver Boyne, James Charles, Andrew Fitzgibbon, and Roberto Cipolla.  Who left the dogs out? 3d animal reconstruction with expectation maximization in the loop.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16, pages 195–211. Springer, 2020.
  * [7] Benjamin Biggs, Thomas Roddick, Andrew Fitzgibbon, and Roberto Cipolla.  Creatures great and smal: Recovering the shape and motion of animals from video.  In Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part V 14, pages 3–19. Springer, 2019.
  * [8] Richard L. Bishop.  There is more than one way to frame a curve.  Amer. Math. Monthly, 82(3):246–251, Mar. 1975.
  * [9] Thomas J. Cashman and Andrew W. Fitzgibbon.  What shape are dolphins? building 3D morphable models from 2D images.  IEEE Trans. Pattern Anal. Mach. Intell., 35(1):232–244, Jan. 2013\.
  * [10] Ching-Hang Chen and Deva Ramanan.  3D human pose estimation = 2D pose estimation + matching.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7035–7043. IEEE, July 2017.
  * [11] Nathan W. Cooper, Thomas W. Sherry, and Peter P. Marra.  Modeling three-dimensional space use and overlap in birds.  Auk, 131(4):681–693, Oct. 2014.
  * [12] Amael Delaunoy and Marc Pollefeys.  Photometric bundle adjustment for dense multi-view 3D modeling.  In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 1486–1493. IEEE, June 2014.
  * [13] Olivier Faugeras and Quang-Tuan Luong.  The Geometry of Multiple Images.  The MIT Press, 2001.
  * [14] Alessandro Ferrarini, Giuseppe Giglio, Stefania Caterina Pellegrino, Anna Grazia Frassanito, and Marco Gustin.  A new methodology for computing birds’ 3D home ranges.  Avian Res, 9(1):1–6, May 2018.
  * [15] Lise Frézal and Marie-Anne Félix.  The natural history of model organisms: C. elegans outside the petri dish.  eLife, 4:e05849, Mar. 2015.
  * [16] Kui Fu, Jiansheng Peng, Qiwen He, and Hanxiao Zhang.  Single image 3D object reconstruction based on deep learning: A review.  Multimed Tools Appl, 80(1):463–498, Sept. 2020.
  * [17] Marie-Anne Félix and Christian Braendle.  The natural history of caenorhabditis elegans.  Curr. Biol., 20(22):R965–R969, Nov. 2010.
  * [18] P. Georgel, S. Benhimane, and N. Navab.  A unified approach combining photometric and geometric information for pose estimation.  In Procedings of the British Machine Vision Conference 2008, pages 1–10. Citeseer, British Machine Vision Association, 2008.
  * [19] Riza Alp Guler, Natalia Neverova, and Iasonas Kokkinos.  DensePose: Dense human pose estimation in the wild.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7297–7306. IEEE, June 2018.
  * [20] Zicheng Guo and Richard W. Hall.  Parallel thinning with two-subiteration algorithms.  Commun. ACM, 32(3):359–373, Mar. 1989.
  * [21] Richard Hartley and Andrew Zisserman.  Multiple View Geometry in Computer Vision.  Cambridge University Press, Mar. 2004.
  * [22] Robert I. Holbrook and Theresa Burt de Perera.  Three-dimensional spatial cognition: Information in the vertical dimension overrides information from the horizontal.  Anim Cogn, 14(4):613–619, Mar. 2011.
  * [23] C.T. Huang and O.R. Mitchell.  Dynamic camera calibration.  In Proceedings of International Symposium on Computer Vision - ISCV, pages 169–174. IEEE, IEEE Comput. Soc. Press, 1995.
  * [24] Steven J. Husson, Wagner S. Costa, Cornelia Schmitt, and Alexander Gottschalk.  Keeping track of worm trackers.  WormBook, pages 1–17, Sept. 2012.
  * [25] Avelino Javer, Michael Currie, Chee Wai Lee, Jim Hokanson, Kezhi Li, Céline N. Martineau, Eviatar Yemini, Laura J. Grundy, Chris Li, QueeLim Ch’ng, William R. Schafer, Ellen A. A. Nollen, Rex Kerr, and André E. X. Brown.  An open-source platform for analyzing and sharing worm-behavior data.  Nat Methods, 15(9):645–646, Aug. 2018.
  * [26] Le Jiang, Caleb Lee, Divyang Teotia, and Sarah Ostadabbas.  Animal pose estimation: A closer look at the state-of-the-art, existing gaps and opportunities.  Comput. Vis. Image Und., 222:103483, Sept. 2022.
  * [27] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7122–7131. IEEE, June 2018.
  * [28] Angjoo Kanazawa, Shahar Kovalsky, Ronen Basri, and David Jacobs.  Learning 3d deformation of animals from 2d images.  In Computer Graphics Forum, volume 35, pages 365–374. Wiley Online Library, 2016.
  * [29] Isinsu Katircioglu, Bugra Tekin, Mathieu Salzmann, Vincent Lepetit, and Pascal Fua.  Learning latent representations of 3D human pose with deep neural networks.  Int J Comput Vis, 126(12):1326–1341, Jan. 2018.
  * [30] Sinead Kearney, Wenbin Li, Martin Parsons, Kwang In Kim, and Darren Cosker.  RGBD-dog: Predicting canine pose from RGBD sensors.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8336–8345. IEEE, June 2020.
  * [31] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [32] Nikos Kolotouros, Georgios Pavlakos, Michael Black, and Kostas Daniilidis.  Learning to reconstruct 3D human pose and shape via model-fitting in the loop.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2252–2261. IEEE, Oct. 2019.
  * [33] Namseop Kwon, Ara B. Hwang, Young-Jai You, Seung-Jae V. Lee, and Jung Ho Je.  Dissection of C. elegans behavioral genetics in 3-d environments.  Sci Rep, 5(1):1–9, May 2015.
  * [34] Namseop Kwon, Jaeyeon Pyo, Seung-Jae Lee, and Jung Ho Je.  3-d worm tracker for freely moving C. elegans.  PLoS ONE, 8(2):e57484, Feb. 2013.
  * [35] Wu Liu, Qian Bao, Yu Sun, and Tao Mei.  Recent advances of monocular 2D and 3D human pose estimation: A deep learning perspective.  ACM Comput. Surv., 55(4):1–41, Nov. 2022.
  * [36] H. C. Longuet-Higgins.  A computer algorithm for reconstructing a scene from two projections.  Nature, 293(5828):133–135, Sept. 1981.
  * [37] Simone Macrì, Daniele Neri, Tommaso Ruberto, Violet Mwaffo, Sachit Butail, and Maurizio Porfiri.  Three-dimensional scoring of zebrafish behavior unveils biological phenomena hidden by two-dimensional analyses.  Sci Rep, 7(1):1–10, May 2017.
  * [38] Julieta Martinez, Rayat Hossain, Javier Romero, and James J. Little.  A simple yet effective baseline for 3d human pose estimation.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2640–2649. IEEE, Oct. 2017.
  * [39] Valsamis Ntouskos, Marta Sanzari, Bruno Cafaro, Federico Nardi, Fabrizio Natola, Fiora Pirri, and Manuel Ruiz.  Component-wise modeling of articulated objects.  In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2327–2335. IEEE, Dec. 2015.
  * [40] Onur Özyeşil, Vladislav Voroninski, Ronen Basri, and Amit Singer.  A survey of structure from motion.  Acta Numer., 26:305–364, May 2017.
  * [41] Brian L. Partridge, Tony Pitcher, J. Michael Cullen, and John Wilson.  The three-dimensional structure of fish schools.  Behav Ecol Sociobiol, 6(4):277–288, Mar. 1980.
  * [42] Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, and Kostas Daniilidis.  Coarse-to-fine volumetric prediction for single-image 3D human pose.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7025–7034. IEEE, July 2017.
  * [43] Mukta Prasad, Andrew Fitzgibbon, Andrew Zisserman, and Luc Van Gool.  Finding nemo: Deformable object class modelling using curve matching.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1720–1727. IEEE, IEEE, June 2010.
  * [44] Daniel Ramot, Brandon E. Johnson, Tommie L. Berry, Lucinda Carnell, and Miriam B. Goodman.  The parallel worm tracker: A platform for measuring average speed and drug-induced paralysis in nematodes.  PLoS ONE, 3(5):e2208, May 2008.
  * [45] Felix Salfelder, Omer Yuval, Thomas P Ilett, David C Hogg, Thomas Ranner, and Netta Cohen.  Markerless 3D spatio-temporal reconstruction of microscopic swimmers from video.  In Visual observation and analysis of Vertebrate And Insect Behavior 2020, 2021.
  * [46] Hinrich Schulenburg and Marie-Anne Félix.  The natural biotic environment of Caenorhabditis elegans.  Genetics, 206(1):55–86, May 2017.
  * [47] S.M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski.  A comparison and evaluation of multi-view stereo reconstruction algorithms.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR’06), volume 1, pages 519–528. IEEE, IEEE, 2006.
  * [48] William Irvin Sellers and Eishi Hirasaki.  Markerless 3D motion capture for animal locomotion studies.  Biology Open, 3(7):656–668, June 2014.
  * [49] Michael Shaw, Haoyun Zhan, Muna Elmi, Vijay Pawar, Clara Essmann, and Mandayam A. Srinivasan.  Three-dimensional behavioural phenotyping of freely moving C. elegans using quantitative light field microscopy.  PLoS ONE, 13(7):e0200108, July 2018.
  * [50] Colin A. Simpfendorfer, Esben M. Olsen, Michelle R. Heupel, and Even Moland.  Three-dimensional kernel utilization distributions improve estimates of space use in aquatic animals.  Can. J. Fish. Aquat. Sci., 69(3):565–572, Mar. 2012.
  * [51] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5693–5703. IEEE, June 2019.
  * [52] Nicholas A Swierczek, Andrew C Giles, Catharine H Rankin, and Rex A Kerr.  High-throughput behavioral analysis in C. elegans.  Nat Methods, 8(7):592–598, June 2011.
  * [53] Raphael Sznitman, Manaswi Gupta, Gregory D. Hager, Paulo E. Arratia, and Josué Sznitman.  Multi-environment model estimation for motility analysis of caenorhabditis elegans.  PLoS ONE, 5(7):e11631, July 2010.
  * [54] Diane Theriault, Zheng Wu, Nickolay I Hristov, Sharon M Swartz, Kenneth S Breuer, Thomas H Kunz, and Margrit Betke.  Reconstruction and analysis of 3D trajectories of Brazilian free-tailed bats in flight.  In 20th Int. Conf. on Pattern Recognition, pages 1–4, 2010.
  * [55] Alexander Toshev and Christian Szegedy.  DeepPose: Human pose estimation via deep neural networks.  In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 1653–1660. IEEE, June 2014.
  * [56] Bill Triggs, Philip F. McLauchlan, Richard I. Hartley, and Andrew W. Fitzgibbon.  Bundle adjustment — a modern synthesis.  In Vision Algorithms: Theory and Practice, pages 298–372. Springer Berlin Heidelberg, 2000.
  * [57] R. Tsai.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses.  IEEE J. Robot. Automat., 3(4):323–344, Aug. 1987.
  * [58] Sara Vicente and Lourdes Agapito.  Balloon shapes: Reconstructing and deforming objects with volume from images.  In 2013 International Conference on 3D Vision, pages 223–230. IEEE, IEEE, June 2013.
  * [59] J. Weng, P. Cohen, and M. Herniou.  Camera calibration with distortion models and accuracy evaluation.  IEEE Trans. Pattern Anal. Machine Intell., 14(10):965–980, 1992\.
  * [60] Nils Wilhelm, Anna Vögele, Rebeka Zsoldos, Theresia Licka, Björn Krüger, and Jürgen Bernard.  FuryExplorer: Visual-interactive exploration of horse motion capture data.  In SPIE Proceedings, volume 9397, pages 148–162. SPIE, SPIE, Feb. 2015.
  * [61] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Unsupervised learning of probably symmetric deformable 3D objects from images in the wild.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1–10. IEEE, June 2020.
  * [62] Eviatar Yemini, Rex A. Kerr, and William R. Schafer.  Tracking movement behavior of multiple worms on food.  Cold Spring Harb Protoc, 2011(12):pdb.prot067025, Dec. 2011.
  * [63] Omer Yuval.  The neuromechanical control of Caenorhabditis elegans head motor behaviour in 3D environments.  PhD thesis, University of Leeds, 2022.
  * [64] Sergey Zagoruyko, Adam Lerer, Tsung-Yi Lin, PedroO. Pinheiro, Sam Gross, Soumith Chintala, and Piotr Dollar.  A MultiPath network for object detection.  In Procedings of the British Machine Vision Conference 2016. British Machine Vision Association, 2016.
  * [65] Liqun Zhu and Wei Weng.  Catadioptric stereo-vision system for the real-time monitoring of 3D behavior in aquatic animals.  Physiology & Behavior, 91(1):106–119, May 2007.
  * [66] Michael Zollhöfer, Patrick Stotko, Andreas Görlitz, Christian Theobalt, Matthias Nießner, Reinhard Klein, and Andreas Kolb.  State of the art on 3D reconstruction with RGB-d cameras.  In Computer graphics forum, volume 37, pages 625–652. Wiley Online Library, 2018.
  * [67] Silvia Zuffi, Angjoo Kanazawa, David W. Jacobs, and Michael J. Black.  3D menagerie: Modeling the 3D shape and pose of animals.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6365–6373. IEEE, July 2017.

## Supplementary material

In this supplementary material we provide additional detail for some of the
steps in our method and include the ranges and values for the various
parameters and hyper-parameters. In Appendix G we provide a more detailed
comparison between our method and the two other methods available. We report
the results of an ablation study in Appendix H and in Appendix I we describe
the three supporting videos (available here:
https://doi.org/10.6084/m9.figshare.22310650) that demonstrate the method and
showcase some of the results.

##  Appendix A Geometric model: The Bishop frame

3D curves are typically expressed in the Frenet frame $TNB$ where $T$ refers
to the normalised tangent of the curve, $N$ is the ‘normal’ vector defined as
the normalised arc-length derivative of $T$ and $B$ is the ‘binormal’ vector
obtained through the cross product $B=T\times N$. This frame is defined along
the curve using the Frenet-Serret formulas:

$\displaystyle\dot{T}$ $\displaystyle=\kappa N,$ (S16) $\displaystyle\dot{N}$
$\displaystyle=-\kappa T+\tau B,$ (S17) $\displaystyle\dot{B}$
$\displaystyle=-\tau N,$ (S18)

where dot denotes the arc-length derivative $d/ds$ and $\kappa$ and $\tau$ are
scalar fields generally called curvature and torsion respectively. For
simplicity we leave the arc length parameter $s$ implicit in all equations.

A difficulty with the Frenet formulation is that the torsion, $\tau$, is
strictly undefined for straight curves, or locally wherever $\kappa=0$. Zero
(or near-zero) curvature is expected in an animal that propagates sinusoidal
waves along its body and at these points we cannot guarantee a unique and
consistent parametrisation. To overcome this ambiguity we use the Bishop frame
[8], given by $TM^{1}M^{2}$ where $T$ again refers to the normalised tangent
of the curve and $M^{1},M^{2}$ form an orthogonal basis. The Bishop equations
define how the frame changes along the curve:

$\displaystyle\dot{T}$ $\displaystyle=m^{1}M^{1}+m^{2}M^{2},$ (S19)
$\displaystyle\dot{M}^{1}$ $\displaystyle=-m^{1}T,$ (S20)
$\displaystyle\dot{M}^{2}$ $\displaystyle=-m^{2}T,$ (S21)

where $m^{1},m^{2}$ are scalar fields analogous to $\kappa,\tau$ that express
the curvature in the $M^{1}$ and $M^{2}$ directions respectively.

While the Bishop frame improves the zero-curvature problem, it does leave a
degree of freedom in the choice of the initial value of $M^{1}$
($M^{1}_{\mathrm{init}}$) that can point in any direction perpendicular to the
initial tangent $T_{\mathrm{init}}$. Any rotation of $M^{1}_{\mathrm{init}}$
around $T_{\mathrm{init}}$ will result in a different $(m^{1},m^{2})$
representation of the curvature, but this rotation angle can easily be
recovered and different representations subsequently aligned.

As Bishop describes in [8] (and expanded here for completeness) the two frames
are related through their scalar field components. $\kappa$ can be recovered
from $m^{1},m^{2}$ using Eqs. S16 and S19 as:

$\displaystyle\kappa=\left\lvert\frac{dT}{ds}\right\rvert=\left\lvert
m^{1}M^{1}+m^{2}M^{2}\right\rvert=\sqrt{(m^{1})^{2}+(m^{2})^{2}}.$ (S22)

To recover the torsion $\tau$ that describes the rotation of the Frenet frame
around $N$ let $\theta$ be the angle between $N$ and $M^{1}$, then

$\displaystyle N$ $\displaystyle=M^{1}\cos{\theta}+M^{2}\sin{\theta},$ (S23)
$\displaystyle B$ $\displaystyle=-M^{1}\sin{\theta}+M^{2}\cos{\theta},$ (S24)
$\displaystyle m^{1}$ $\displaystyle=\kappa\cos{\theta}\text{ and}$ (S25)
$\displaystyle m^{2}$ $\displaystyle=\kappa\sin{\theta}.$ (S26)

Differentiating Eq. S23 with respect to arc length and substituting from Eqs.
S20 and S21 we have:

$\displaystyle\dot{N}$
$\displaystyle=\dot{\theta}\left(-M^{1}\sin{\theta}+M^{2}\cos{\theta}\right)$
(S27)
$\displaystyle\quad\quad+T\left(m^{1}\cos{\theta}-m^{2}\sin{\theta}\right)$
(S28) $\displaystyle=\dot{\theta}B-\kappa
T(\cos^{2}{\theta}+\sin^{2}{\theta})$ (S29)
$\displaystyle\implies\tau=\dot{\theta}.$ (S30)

Thus, in the words of Bishop, “$\kappa$ and an indefinite integral $\int\tau
ds$ are polar coordinates for the curve $(m^{1},m^{2})$”.

##  Appendix B Project: Pinhole camera model

The imaging setup is modelled using a triplet of pinhole camera models with
tangential and radial distortion [21]. A single pinhole camera model is used
to project 3D points into an image plane using a perspective transformation.
The 15 parameters required for each camera model are summarised in Table S5.
These are divided into intrinsic, extrinsic and distortion parameters. The
intrinsic parameters are $(f_{x},f_{y},c_{x},c_{y})$, where $f_{x}$ and
$f_{y}$ are the focal lengths and $(c_{x},c_{y})$ is a principal point usually
set to the image centre. The extrinsic parameters – angles
$(\phi_{0},\phi_{1},\phi_{2})$ and a translation vector $t$ – define the
extrinsic transformation $M=[R|t]$, where

$\displaystyle R$
$\displaystyle=R_{z}(\phi_{0})R_{y}(\phi_{1})R_{x}(\phi_{2}),$ (S31)
$\displaystyle t$ $\displaystyle=\begin{pmatrix}t_{0}\\\ t_{1}\\\
t_{2}\end{pmatrix}$ (S32)

and $R$ is a rotation matrix composed of three axial rotations:

$\displaystyle R_{z}(\phi_{0})$
$\displaystyle=\begin{pmatrix}\cos{\phi_{0}}&-\sin{\phi_{0}}&0\\\
\sin{\phi_{0}}&\cos{\phi_{0}}&0\\\ 0&0&1\end{pmatrix},$ (S33) $\displaystyle
R_{y}(\phi_{1})$
$\displaystyle=\begin{pmatrix}\cos{\phi_{1}}&0&\sin{\phi_{1}}\\\ 0&1&0\\\
-\sin{\phi_{1}}&0&\cos{\phi_{0}}\end{pmatrix}\text{ and }$ (S34)
$\displaystyle R_{x}(\phi_{2})$ $\displaystyle=\begin{pmatrix}1&0&0\\\
0&\cos{\phi_{2}}&-\sin{\phi_{2}}\\\
0&\sin{\phi_{2}}&\cos{\phi_{2}}\end{pmatrix}.$ (S35)

The radial and tangential distortion coefficients, $(k_{1},k_{2},k_{3})$ and
$(p_{1},p_{2})$ respectively, complete the parametrisation.

Due to imperfections in the camera model (and possibly environmental
vibrations of the setup), using fixed pinhole camera model parameters yields
results with errors that vary over time and space (_i.e_. the accuracy depends
on the worm position). Allowing the camera parameters to change freely between
frames resolves the reconstruction errors, but the problem becomes under-
determined and introduces drift into both the camera and curve parameters
resulting in incorrect tracking. To compensate for these errors without
allowing full freedom of movement we use fixed pinhole camera parameters and
introduce frame-dependent variables that emulate relative movement between the
cameras, hence limiting drift and providing stable reconstructions.

To this end, the standard pinhole camera model is extended to include
$\eta^{s}=(dx,dy,dz)$, relative shifts along the local coordinate axes (see
Sec. 4.1). These parameters approximate changes in the relative positions and
rotations of the cameras by applying pixel translations after the perspective
transformation. Without loss of generality $\eta^{s}$ can be limited to one
direction per camera, thus capturing only relative shifts. The shifts used in
camera index $c$ are given by $(s_{x},s_{y})_{c}$ where:

$\displaystyle(s_{x},s_{y})_{0}$ $\displaystyle=(dx,0),$ (S36)
$\displaystyle(s_{x},s_{y})_{1}$ $\displaystyle=(0,-dy),\text{ and }$ (S37)
$\displaystyle(s_{x},s_{y})_{2}$ $\displaystyle=(0,dz).$ (S38)

For 3D object point $(X,Y,Z)$, the corresponding projected image point $(u,v)$
is generated using the following procedure (when $z\neq 0$):

$\displaystyle\begin{pmatrix}x\\\ y\\\ z\end{pmatrix}$
$\displaystyle=R\begin{pmatrix}X\\\ Y\\\ Z\end{pmatrix}+t,$ (S39)
$\displaystyle x^{\prime}$ $\displaystyle=\frac{x}{z}+\frac{s_{x}}{f_{x}},$
(S40) $\displaystyle y^{\prime}$
$\displaystyle=\frac{y}{z}+\frac{s_{y}}{f_{y}},$ (S41) $\displaystyle r^{2}$
$\displaystyle=x^{\prime 2}+y^{\prime 2},$ (S42) $\displaystyle k$
$\displaystyle=1+k_{1}r^{2}+k_{2}r^{4}+k_{3}r^{6},$ (S43) $\displaystyle
x^{\prime\prime}$
$\displaystyle=kx^{\prime}+2p_{1}x^{\prime}y^{\prime}+p_{2}(r^{2}+2x^{\prime
2}),$ (S44) $\displaystyle y^{\prime\prime}$
$\displaystyle=ky^{\prime}+p_{1}(r^{2}+2y^{\prime
2})+2p_{2}x^{\prime}y^{\prime},$ (S45) $\displaystyle\begin{pmatrix}u\\\
v\end{pmatrix}$ $\displaystyle=\begin{pmatrix}f_{x}x^{\prime\prime}+c_{x}\\\
f_{y}y^{\prime\prime}+c_{y}\end{pmatrix}.$ (S46)

Note the inclusion of the shift parameters in Eqs. S40 and S41.

##  Appendix C Rendering parameters: Tapering

The rendering stage generates super-Gaussian blobs at the projected image
locations of each curve vertex ($n$). The shape of the blobs in camera $c$
depends on the optimisable parameters: the scale $\sigma_{c}$, the intensity
$\iota_{c}$ and the exponent used in the Gaussian $\rho_{c}$. To capture the
worm shape we taper the values of $\sigma_{c}$ and $\iota_{c}$ from their
optimisable values along the middle $60\%$ down to fixed minimum values
$\sigma_{\text{min}}$ and $\iota_{\text{min}}$ respectively at the ends. The
tapered outputs $\bar{\sigma}_{c},\bar{\iota}_{c}\in\mathbb{R}^{N}$ are
calculated thus:

$\displaystyle\bar{\sigma}_{c,n}$
$\displaystyle=\begin{cases}\sigma_{\text{min}}(1-\frac{5n}{N})+\sigma_{c}\frac{5n}{N}&0\leq
n<N/5\\\ \sigma_{c}&N/5\leq n<4N/5\\\
\sigma_{c}(1-\frac{n-4N/5}{N-4N/5})+\sigma_{\text{min}}\frac{n-4N/5}{N-4N/5}&4N/5\leq
n<N,\end{cases}$ (S47)

and

$\displaystyle\bar{\iota}_{c,n}$
$\displaystyle=\begin{cases}\iota_{\text{min}}(1-\frac{5n}{N})+\iota_{c}\frac{5n}{N}&0\leq
n<N/5\\\ \iota_{c}&N/5\leq n<4N/5\\\
\iota_{c}(1-\frac{n-4N/5}{N-4N/5})+\iota_{\text{min}}\frac{n-4N/5}{N-4N/5}&4N/5\leq
n<N.\end{cases}$ (S48)

These values are used in Eq. 2.

##  Appendix D Mask generation

The input images are masked to focus the pixel-errors to a single region,
local to the predicted curve, that is consistent across all three views and
excludes any interference that does not correspond to the same mass. The masks
$M\in\mathbb{R}^{3\times w\times w}$ are generated in a similar way to the
renders $R$ (see Sec. 4.2), but with a few notable differences. First, the
blobs $B$ are normalised and weighted by the relative scores,

$B^{\prime}_{c,n}=\frac{B_{c,n}}{\sum_{i,j}B_{c,n}}\cdot\hat{S}_{n},$ (S49)

then combined by taking the maximum values as before,

$M^{\prime}_{c,i,j}=\max\\{B^{\prime}_{c,n,i,j}\\}_{n=0,\dots,N-1},$ (S50)

and finally passed through a threshold:

$M_{c,i,j}=\begin{cases}1&M^{\prime}_{c,i,j}\geq\Theta\\\
0.2&M^{\prime}_{c,i,j}<\Theta.\end{cases}$ (S51)

For $\Theta=0$ we have $M=1$ everywhere and no masking occurs. For $\Theta\sim
1$ the mask shrinks around the blobs that correspond to the highest scoring
vertices, making $M=0.2$ almost everywhere. In all our experiments we fix
$\Theta=0.1$ as this appears to produce a good balance. Note that we do not
completely exclude the remaining points, but just reduce their intensity. This
allows some gradient to flow from outside the detection region which is
especially important in the early stages when none of the curve may be
intersecting the correct pixel mass. It is also important to detach the masks
from the gradient computation at this stage otherwise the curve will simply
shrink and fade away from the high-intensity pixels thus minimising pixel
errors simply by detecting fewer pixels.

##  Appendix E Centre-shifting

The curve is periodically shifted along its length to centre it over the pixel
mass in all three views. An unbalanced alignment can be seen from the score
profile when the centre-of-mass index of $\hat{S}$ ($\bar{n}$) is not in the
middle of the curve (_i.e_. $\bar{n}\neq N/2$). We can then shift the curve
along its length using $\bar{n}$ as the new midpoint, removing vertices from
the low-scoring end and adding new vertices to the high-scoring end. The low-
scoring end will consequently improve, and since there is no expectation that
the new vertices will match the images this typically means the high-scoring
end worsens; rectifying the imbalance.

To perform a centre-shift we calculate the centre of mass of the score profile
and the degree of imbalance as:

$\displaystyle\bar{n}$
$\displaystyle=\frac{\sum_{n}n\hat{S}_{n}}{\sum_{n}\hat{S}_{n}},\text{ and}$
(S52) $\displaystyle n_{s}$ $\displaystyle=\bar{n}-N/2.$ (S53)

Then we update the curvature by shifting the values and decreasing linearly to
zero at the ends. I.e. for $n_{s}>0$,

$K_{n}\leftarrow\begin{cases}K_{n+n_{s}}&0\leq n<N-n_{s},\\\
K_{N-n_{s}-1}(1-\frac{N-n_{s}-n+1}{n_{s}})&N-n_{s}\leq n<N,\end{cases}$ (S54)

and similarly for $n_{s}<0$. Finally, new position and orientation parameters
are calculated from the adjusted midpoint and updated curvatures using Eq. 1:

$(P,T,M^{1})\leftarrow
F\left(P_{\bar{n}},T_{\bar{n}},M^{1}_{\bar{n}},K,l,\bar{n}\right).$ (S55)

This process is illustrated in Fig. 5.

Centre-shifting the curve occurs between gradient descent optimisation steps.
In practise, shifting after every step quickly leads to instabilities as the
new points are not afforded the time required to align them with the images.
Furthermore, it is unrealistic to expect a perfect balance can be sustained
and an unconstrained $n_{s}$ means large shifts may be applied, possibly due
to a change in the camera parameters or some transient interference, that
would destroy extensive sections of the curve. To mitigate these problems we
only apply centre-shifting every $\alpha$ steps when $|n_{s}|>\beta N$ and
then restrict the shift size to $\gamma$ (_i.e_.
$n_{s}\leftarrow\min\\{n_{s},\gamma\\}$). In our experiments we find values of
$\alpha\in[3,6]$, $\beta\in(0.05,0.1)$ and $\gamma\in[1,2]$ provide the
necessary stabilisations (Table S8).

##  Appendix F Optimisation

The non-optimisable parameter values and ranges that are used in our
experiments are outlined in Table S8. The biggest factors affecting the choice
of parameters are the magnification and individual worm size – both of which
vary between experiments. These determine the required image size, $w$, and
inform the estimates for the length bounds, $l_{\text{min}}$ and
$l_{\text{max}}$. The super-Gaussian blobs are generated in corresponding
$w\times w$ images, so the minimum scales and intensities at the tips,
$\sigma_{\text{min}}$ and $\iota_{\text{min}}$, must also change accordingly
with the image and worm size.

Table S8 lists the weighting coefficients used in the combined loss
calculation (Eq. 15). Values of $\omega_{\text{sm}}$ and $\omega_{\text{t}}$
may vary between experiments to capture the different dynamics observed in the
different environmental conditions (specifically, concentration of the
gelatin). For example, when the worm is deforming quickly (in low-viscosity
experiments) there are large postural changes between frames and therefore the
temporal loss $\mathcal{L}_{\text{t}}$ is relatively big. In this case a
smaller value of $\omega_{\text{t}}$ is used to prevent the reconstruction
lagging behind the worm. Similarly, when the worm is deforming slowly (in
high-viscosity experiments) it frequently forms tightly coiled postures in
which case the smoothness loss $\mathcal{L}_{\text{t}}$ is large and a smaller
value for $\omega_{\text{sm}}$ is more suitable. As discussed in the main
text, when reconstructing full sequences the initial curve and parameters are
used for the initial guesses to the subsequent frame. This preserves head-tail
orientation and consistency through complex manoeuvres.

The learning rates are shown in Table S8 – these are fixed for all
experiments. Optimisable camera, curve and render parameters are summarised in
Tables S5, S5 and S5 respectively.

##  Appendix G Comparisons with previous methods

In Fig. 7 we compare results against two existing methods [45, 63]. These
methods only provide projected midline coordinates, so in order to use our
pipeline to generate renders and calculate $\mathcal{L}_{\text{px}}$ we need
to provide renderer (blob) parameters. We can use values found for our
midlines, but this introduces a bias towards our method as these parameters
are only optimal for our midlines. To mitigate against this we initialise with
our values but re-optimise for each frame until convergence (keeping the curve
and camera parameters fixed) to ensure optimal rendering parameters are found
for each midline. In Fig. S9 we show the effect of the re-optimisation across
the same clip. As expected, re-optimisation reduces the loss for both methods,
but the improvement is fairly marginal. The improved losses are used for
comparison in the main text.

Figure S9: A comparison between the losses obtained when using our renderer
parameters for Yuval’s Worm-Tracker 3D (WT3D) [63] and Salfelder _et al_.’s
‘reconst’ [45] midlines _vs_. the losses obtained after re-optimising the
renderer parameters to better suit their midlines. Our method (MF) is shown
for reference. We show moving averages over 25 frames
($1\text{\,}\mathrm{s}$).

In Table S1 we evaluate midline quality of all three methods against ground-
truth manual annotations. This unbiased evaluation (as only projected midline
points are used) shows our method to be more accurate. However, the sample
size is limited, more so as the other methods only provide results for roughly
half of the available annotations.

Method |  # poses |  Score |  Total
---|---|---|---
MF (ours) |  487 |  1.53 (0.51) |  $4\text{\,}\mathrm{h}$ $37\text{\,}\mathrm{min}$
reconst [45] |  226 |  1.54 (0.69) |  $1\text{\,}\mathrm{h}$ $32\text{\,}\mathrm{min}$
MF (ours) |  226 |  1.34 (0.53) |
WT3D [63] |  237 |  2.64 (0.92) |  $45\text{\,}\mathrm{min}$
MF (ours) |  237 |  1.46 (0.59) |
Table S1:  Mean (and standard deviation) pixel distances between predicted
points and hand-annotated points (see Fig. 6). Total refers to the overall
reconstructed duration using each method.

##  Appendix H Ablation study

We demonstrate some of the effects of masking (Fig. 4), centre-shifting (Fig.
5) and varying parameters (Movie 1) throughout the main text. In Table S2 we
present the results from a more thorough ablation study conducted over a
typical $\sim$ $5\text{\,}\mathrm{min}$ clip to clarify the importance of the
different components of our method.

Optimisable camera (a), rendering parameters (b) and centre-shifting (c) yield
considerable benefit; setting $\omega_{\text{sc}}>0$ (d) and regularisation
(f) incur a marginal cost ($\tilde{\mathcal{L}}>0.99$) but recover poses with
high transparency (d) and ensure realistic (smooth) poses (f); finally,
masking (e) incurs a small cost but provides robustness to sequences with
interference (_e.g_. Movie 1).

Variant |  $<\mathcal{L}_{\text{px}}>$ |  $\tilde{\mathcal{L}}=<\mathcal{L}_{\text{px}}>/\text{ Ref}$
---|---|---
Ref |  $3.9910-03$ ($1.1310-03$) |  1.000
a |  $7.2110-03$ ($1.9710-03$) |  1.807
b |  $4.6410-03$ ($1.4210-03$) |  1.163
c |  $4.2910-03$ ($1.7110-03$) |  1.075
d |  $3.9910-03$ ($1.1210-03$) |  1.000
e |  $3.6910-03$ ($9.0010-04$) |  0.925
f |  $3.9610-03$ ($1.0810-03$) |  0.992
Table S2:  Ablation results (mean (and standard deviation) pixel errors and
normalised mean) across a typical 7200-frame clip. Variants: a) no camera
parameter optimisation, b) rendering parameters ($\sigma,\iota,\rho$) fixed to
averages from the reference results, c) no centre-shifting, d) no scores-loss
($\omega_{\text{sc}}=0$), e) no input masking, f) no regularisation losses
($\omega_{\text{sm}}=\omega_{\text{t}}=0$).

##  Appendix I Supplementary results

Three supporting videos are available here:
https://doi.org/10.6084/m9.figshare.22310650.

In the accompanying Movie 1, the effects of adjusting some of the parameters
listed in Tables S8 and S8 on the solution are demonstrated. In this video we
show the optimisation process using paired examples. The same frame and
randomised initial guess are used for each pair and the optimisation is run
for a fixed 2000 steps. The frames are selected to demonstrate a range of
challenging conditions – especially for achieving convergence from a random
guess. The first of each pair shows successful optimisation using parameter
values in the ranges specified in Tables S8 and S8. The second of each pair
shows the effect that changing one of the parameters has on the converged
solution.

We include two further videos demonstrating examples of both successful and
less-successful sequence reconstructions. In Movie 2 we showcase successful
examples. First, when the worm is well resolved in all three views. Next, when
there is interference from dirt or bubbles and/or poor focus in one or more
views. Lastly, through complex coiling manoeuvres that include significant
self-occlusion.

The limitations of our method are illustrated in Movie 3. These examples,
taken from otherwise successful reconstructions, demonstrate that when
significant loss of focus is combined with coiling, heavily occluded, postures
the reconstruction can fail. The exact degree of failure is difficult to
ascertain for the exact same reasons and only by watching the full sequences
can we be convinced that the reconstruction is incorrect. This suggests that
incorporating more temporal information may help to resolve these fail-cases,
but we leave this for future investigation.

Parameter |  Purpose |  Domain
---|---|---
$f_{x}$,$f_{y}$ |  Focal lengths |  $\mathbb{R}^{+}$
$(c_{x},c_{y})$ |  Principal image point |  $\mathbb{R}^{+}$
$\phi_{0},\phi_{1},\phi_{2}$ |  Rotation angles |  $[0,2\pi)$
$t$ |  Translation (position) vector |  $\mathbb{R}^{3}$
$(k_{1},k_{2},k_{3})$ |  Radial distortion coefficients |  $\mathbb{R}^{+}$
$(p_{1},p_{2})$ |  Tangential distortion coefficients |  $\mathbb{R}^{+}$
$\eta^{s}$ |  Relative shifts |  $\mathbb{R}^{3}$

Table S3: Camera model parameters. With the exception of $\eta^{s}$ these are
defined for each camera. $\eta^{s}$ is shared between the models as per Eqs.
S36, S37 and S38.

Parameter |  Purpose |  Domain
---|---|---
$P$ |  3D curve vertex coordinates |  $\mathbb{R}^{N\times 3}$
$T$ |  Normalised curve tangent vectors at each vertex location |  $\mathbb{R}^{N\times 3}$
$M^{1}$ |  Normalised curvature orientation vectors at each vertex location |  $\mathbb{R}^{N\times 3}$
$K$ |  Vector curvature |  $\mathbb{R}^{N\times 2}$
$l$ |  Curve length |  $(l_{\text{min}},l_{\text{max}})$

Table S4: Curve and Bishop frame parameters.

Parameter |  Purpose |  Domain
---|---|---
$\sigma_{c}$ |  Standard deviation of the super-Gaussian blobs along the untapered middle 60% of the worm in camera $c$ |  $[\sigma_{\text{min}},\infty)$
$\iota_{c}$ |  Intensity scaling factor for the super-Gaussian blobs along the untapered middle 60% of the worm in camera $c$ |  $[\iota_{\text{min}},\infty)$
$\rho_{c}$ |  Exponent used in the super-Gaussian blobs in camera $c$ |  $(0,\infty)$

Table S5: Rendering parameters.

Parameter |  Purpose |  Value/Range
---|---|---
$w$ |  (Square) image size |  $200$–$350~{}$\mathrm{p}\mathrm{x}$$
$N$ |  Number of discrete curve vertices |  $128$
$l_{\text{min}}$ |  Minimum curve length |  $0.5$–$1~{}$\mathrm{m}\mathrm{m}$$
$l_{\text{max}}$ |  Maximum curve length |  $1$–$2~{}$\mathrm{m}\mathrm{m}$$
$k_{\text{max}}$ |  Maximum curvature constraint |  $3$ osculating circles
$\sigma_{\text{min}}$ |  Standard deviation of the super-Gaussian functions at the tips |  $2$–$4~{}$\mathrm{p}\mathrm{x}$$
$\iota_{\text{min}}$ |  Intensity scaling factor of the super-Gaussian functions at the tips |  $0.15$–$0.3$
$\Theta$ |  Mask threshold |  $0.1$
$\alpha$ |  Frequency of centre-shift adjustments (number of gradient descent steps) |  $3$–$6$ steps
$\beta$ |  Centre-shift adjustment sensitivity |  $0.05$–$0.1$
$\gamma$ |  Maximum centre-shift adjustment |  $1$–$2$ vertices

Table S6: Non-optimisable parameter values and ranges used in our experiments.
Listed in the order they appear in the text.

Parameter |  Purpose |  Value/Range
---|---|---
$\omega_{\text{px}}$ |  Weighting of the pixel loss $\mathcal{L}_{\text{px}}$ |  $0.1$
$\omega_{\text{sc}}$ |  Weighting of the scores loss $\mathcal{L}_{\text{sc}}$ |  $0.01$
$\omega_{\text{sm}}$ |  Weighting of the smoothness loss $\mathcal{L}_{\text{sm}}$ |  $10$–$100$
$\omega_{\text{t}}$ |  Weighting of the temporal loss $\mathcal{L}_{\text{t}}$ |  $10$–$100$
$\omega_{\text{i}}$ |  Weighting of the intersections loss $\mathcal{L}_{\text{i}}$ |  $0.1$–$1$

Table S7: Weighting coefficients for the different loss terms.

Parameter |  Purpose |  Value
---|---|---
$\lambda_{p}$ |  Learning rate for the curve parameters $\\{P,T,M^{1},K,l\\}$ |  $110-3$
$\lambda_{r}$ |  Learning rate for the rendering parameters $\\{\sigma,\iota,\rho\\}$ |  $110-4$
$\lambda_{\eta}$ |  Learning rate for the camera parameters $\eta$ |  $110-5$
$\lambda_{\text{min}}$ |  Minimum learning rate for all parameters |  $110-6$

Table S8: Learning rates for the different parameter groups.