(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Carnegie Mellon University
11email: {manam, jinhyun1, siwase, rkhirodk<EMAIL_ADDRESS>

# Generalizable Neural Human Renderer

Mana Masuda  Jinhyung Park  Shun Iwase  Rawal Khirodkar  Kris Kitani

###### Abstract

While recent advancements in animatable human rendering have achieved
remarkable results, they require test-time optimization for each subject which
can be a significant limitation for real-world applications. To address this,
we tackle the challenging task of learning a Generalizable Neural Human
Renderer (GNH), a novel method for rendering animatable humans from monocular
video without any test-time optimization. Our core method focuses on
transferring appearance information from the input video to the output image
plane by utilizing explicit body priors and multi-view geometry. To render the
subject in the intended pose, we utilize a straightforward CNN-based image
renderer, foregoing the more common ray-sampling or rasterizing-based
rendering modules. Our GNH achieves remarkable generalizable, photorealistic
rendering with unseen subjects with a three-stage process. We quantitatively
and qualitatively demonstrate that GNH significantly surpasses current state-
of-the-art methods, notably achieving a $31.3\%$ improvement in LPIPS.

###### Keywords:

Neural rendering, Novel view synthesis, Human Rendering

Figure 1:  Given only a monocular video as input, our novel generalizable
human rendering framework outputs high-fidelity animatable human rendering
without any test-time optimization.

##  1 Introduction

Recent advances in free-viewpoint rendering of humans in motion have achieved
remarkable breakthroughs by combining parametric human body models [60, 73,
82] and neural radiance fields (NeRF) [64, 4, 6] or 3D Gaussian Splatting [78,
31]. Nevertheless, in practical scenarios requiring quick generation, current
methods [55, 101] encounter limitations due to the necessity of hours of test-
time optimization for each subject. Although some recent studies utilize empty
space skipping schemes [37] or 3D Gaussian splatting [78] to speed up
rendering, the extensive test-time optimization needed for every subtle
variation in subjects or clothing severely hinders its wide applicability in
dynamic human rendering.

Synthesizing novel views from single-view inputs without the need for subject-
specific fine-tuning poses a significant challenge. This is due to the
complexities involved in transforming geometry-aware features from the input
to the target view, depending on human pose and appearance. Existing
generalizable methods [48, 70, 51, 30] address this challenge by either
reconstructing the complete human body in 3D from multi-view inputs [48, 70]
or by inferring missing information from a single input image [30]. GHuNeRF
[51] can reconstruct the human body from a monocular video, but it still
suffers from severe blurriness and takes several seconds for rendering,
because it adheres to NeRF’s conventional ray-sampling rendering process.

To enable high-fidelity animatable human rendering from monocular video
without the need for test-time optimization for each subject, we present the
Generalizable Neural Human Renderer (GNH). This achievement is facilitated by
a streamlined three-stage process: 1) appearance feature extraction, 2)
feature transformation, and 3) multi-frame fusion, followed by CNN-based image
rendering. Specifically, GNH processes monocular video with known poses and
camera views, extracting generalizable appearance features and projecting them
into 3D space utilizing an explicit SMPL body prior. This step allows the
transfer of appearance information from the input videos to the output image
plane. In the second stage, appearance features are transformed to the output
image plane using an SMPL-based transformation. GNH then fuses these features
into a CNN-compatible image feature through multi-view geometry, to render the
subject with a CNN-based image renderer. These components enable GNH to
produce high-fidelity images of moving subjects, circumventing the extensive
test-time optimization characteristic of many previous methods and achieving
faster rendering speeds.

To assess the performance of GNH, we conduct experiments on three widely-used
human datasets: the ZJU-MoCap dataset [75], the People Snapshot dataset [1],
and the AIST++ dancing dataset [53]. These datasets include multi-view RGB
videos of various subjects, showcasing diversity in subject appearance and
clothing. The results demonstrate that GNH surpasses contemporary
generalizable human rendering methods, improving LPIPS by $31.3\%$ and
reducing average rendering error by $17.4\%$ compared to the state-of-the-art.
In summary, our contributions are as follows:

  * •

We introduce the Generalizable Neural Human Renderer (GNH), an innovative
approach designed to render animatable humans from monocular video inputs,
eliminating the need for subject-specific test-time optimization.

  * •

By transferring appearance features from the input video to the output image
plane, GNH yielded superior high-fidelity rendering results compared to
previous methods of animatable human rendering.

  * •

Comprehensive evaluations reveal that GNH sets a new benchmark in performance,
achieving state-of-the-art results. Notably, GNH improves upon previous
generalizable human NeRF methods [30, 51] by reducing LPIPS by $31.3\%$ while
also achieving a fourfold increase in rendering speed. Furthermore, GNH
surpasses the existing non-generalizable human NeRF approach [101] by lowering
LPIPS by $4.7\%$.

##  2 Related Works

###  2.1 Novel View Synthesis for Humans

Free-viewpoint rendering of humans is a challenging task that has been
investigated over the last couple of decades [42, 62, 10, 88, 20, 11]. One
active stream for rendering humans in novel poses is human pose retargeting
[61, 3, 87, 67, 24, 12, 84, 99] by using 2D human pose parameters [8, 25]. Liu
_et al_. [56] and following works [85, 32, 27, 28, 106, 2, 33] start using a
pre-captured 3D body model. Recent advancements in the field of novel view
synthesis have been marked by the introduction of the Neural Radiance Field
(NeRF) [64] and its follow-up works [4, 77, 72, 5]. Following its
developments, many NeRF-based animatable human rendering methods have been
proposed for both body part rendering [18, 26, 43, 15, 7, 95] and full body
rendering [75, 74, 107, 68, 47, 55, 90, 115, 100, 38, 98, 89, 63, 37, 86, 91,
104, 17, 112]. Peng _et al_. [75] used learned structured latent codes
embodied for reposed mesh vertices from the SMPL model [60] and introduced a
NeRF-based neural renderer. Weng _et al_. [101] is one of the first works that
takes only monocular video as input. They achieved high-fidelity full 3D
rendering by mapping multiple frames of a moving person into a canonical,
volumetric T-pose. InstantAvater[37] achieved in a minute of training and fast
rendering speed from an empty space skipping scheme designed for dynamic
scenes with known articulation patterns. Recent advancements in 3D Gaussian
Splitting achieved fast training and rendering of animatable head avatars
[117, 83, 109, 19, 16, 96, 103, 49], hand avatar [39] and full-body avatars
from multi-view inputs [119, 65, 54, 71] or even a monocular video input [46,
29, 50, 52, 35, 78, 41, 31].

###  2.2 Generalizable Novel View Synthesis

As a downstream task of NeRF, many scene-agnostic (generalizable) novel view
synthesis methods have been proposed [111, 13, 97, 40, 57, 93, 59, 80, 94,
116, 108, 110]. PixelNeRF [111] employs an image encoder to condition NeRF on
2D image features. FeatureNeRF [110] solved this problem by distilling vision
foundation models [9, 81]. Zhao _et al_. [116] firstly worked on a
generalizable method for general dynamic scenes from a monocular video, though
this approach assumes small motions and is incapable of expressing dynamic
human motions.

###  2.3 Generalizable Novel View Synthesis for humans

Synthesizing novel views of moving humans in a scene-agnostic way is an
especially challenging task due to the complexity and diversity of human
appearances and poses. Most of the existing works [79, 47, 14, 22, 48, 21, 70]
take multiple camera inputs and render novel views of unseen subjects in
unseen poses by integrating the features of input camera views by using body
model as a geometric prior. ActorsNeRF [66] trains the network on multiple
subjects and synthesizes unseen poses of subjects from monocular video with
few-shot optimization on that subject. Geng _et al_. [23] also presents a
generalizable NeRF model to leverage learned prior to reducing the
optimization time to minutes. GHuNeRF [51] achieved generalizable human
rendering from a monocular video by introducing a visibility-aware aggregation
scheme. DINAR [92] creates full-body avatars from a single image by utilizing
a diffusion model to inpainting the unseen part of the body. SHERF [30] also
recovers the full-body avatars from a single image from a combination of three
different features. GPS-Gaussian [118] achieved generalizable and fast full-
body rendering from multi-view inputs by utilizing Gaussian-based rendering,
but they do not support pose changes.

##  3 Method: Generalizable Neural Human Renderer

We propose the Generalizable Neural Human Rnederer (GNH) to achieve animatable
human rendering solely from a monocular video of the subject. The overall
pipeline of our GNH is shown in Fig. 2.

Problem Setting Given a monocular video, estimated camera pose, subject pose,
target camera pose, and target subject pose, we first sample a set of $N$
frames as source input ${\\{\mathbf{I}_{s}\in\mathbb{R}^{H\times W\times
3},\mathbf{P}_{s}\in\mathbb{R}^{3\times
4},\boldsymbol{\theta}_{s}\in\mathbb{R}^{24\times 3}\\}}_{s=1}^{N}$, where
$\mathbf{I}_{s}$ represents the source image, $\mathbf{P}_{s}$ the camera
extrinsic, and $\boldsymbol{\theta}_{s}$ the SMPL body pose parameters [60].
Given this input, we render the subject in a specified pose
$\boldsymbol{\theta}_{t}$ from a queried target camera pose $\mathbf{P}_{t}$.

Figure 2:  Overview of our Generalizable Neural Human Renderer (GNH). Stage 1
- 3D Source feature extraction obtains 3D source feature using 2D feature
extraction and lifting it to 3D using the body mesh vertices and camera
parameters. Stage 2 - Source to target mapping converts the 3D source features
to the target domain and projects them into 2D. Stage 3 - Multi-frame
aggregation and rendering consolidates information from all the source frames
using multi-view geometry and renders the target image using a CNN-based
renderer.

###  3.1 Source Feature Extraction

2D Source Image Feature Extraction. As seen in Fig.2(a), the architecture of
our source feature extraction begins with a convolutional feature extraction
backbone. We leverage a pre-trained DINO-v2 [69] as a robust extractor for
low-resolution image features, $\boldsymbol{\phi}_{\text{coarse}}$. To better
align this feature to our task, we then refine and upsample this image feature
by feeding it into a vision transformer $\boldsymbol{\phi}_{\text{refine}}$,
which produces a refined coarse image feature [105, 95]:

$\mathbf{F}_{\text{coarse}}=\boldsymbol{\phi}_{\text{refine}}(\boldsymbol{\phi}_{\text{coarse}}(\mathbf{I}_{\text{s}})).$
(1)

This coarse feature extraction stream has a parallel fine-grained feature
extraction stream, $\boldsymbol{\phi}_{\text{fine}}$, using convolutional
layers with a single downsampling step:

$\mathbf{F}_{\text{fine}}=\boldsymbol{\phi}_{\text{fine}}(\mathbf{I}_{\text{s}}),$
(2)

which aims to maintain a high-resolution feature map to preserve high-fidelity
details for the target image rendering. The final per-source frame feature
$\mathbf{F}_{\text{s}}$ is generated by channel-wise concatenation of the
coarse image feature and fine image feature:

$\mathbf{F}_{\text{s}}=\mathbf{F}_{\text{coarse}}\oplus\mathbf{F}_{\text{fine}}.$
(3)

2D-to-3D Feature Lifting. Following the 2D feature extraction module, our
source feature extraction module lifts this 2D feature into 3D space. To
handle large human motion while keeping generalizability, we leverage the SMPL
body template to seamlessly transfer the source frame feature into the 3D
space. We extract a 3D mesh $\mathcal{M}_{s}$ from the source human pose
parameters $\boldsymbol{\theta}_{s}$, containing mesh vertices
$\\{\boldsymbol{v}_{s}^{i}\\}_{i=1}^{6890}$. Note that we set SMPL body shape
parameters as $\boldsymbol{\beta}=\boldsymbol{0}$ to keep generalizability for
various human shapes. Each mesh vertex is then projected onto the source image
feature plane using the source camera’s projection function, $\Pi_{s}$. This
projection, $\Pi_{s}(\boldsymbol{v}_{s}^{i})$, enables the extraction of an
appearance feature $\boldsymbol{F}^{3D}_{s}(\boldsymbol{v}_{s}^{i})$ from the
estimated source image feature $\mathbf{F}_{\text{s}}$. Thus, each 3D source
vertex $\boldsymbol{v}_{s}^{i}$ is associated with the following appearance
feature:

$\boldsymbol{F}^{3D}_{s}(\boldsymbol{v}_{s}^{i})=\mathbf{F}_{s}\big{(}\Pi_{s}(\boldsymbol{v}_{s}^{i})\big{)}.$
(4)

###  3.2 Source-to-Target Feature Mapping

Mapping to Target Space. As shown in Fig. 2(b), our second step consists of
transforming the 3D source appearance feature $\boldsymbol{F}^{3D}_{s}$ into
the target pose. We transform the source SMPL mesh vertices to the target
space using the given target subject pose parameters. After preparing target
3D mesh $\mathcal{M}_{t}$ from the given target human pose parameters
$\boldsymbol{\theta}_{t}$ and template body shape parameter
$\boldsymbol{\beta}=\boldsymbol{0}$, each target mesh vertex
$\boldsymbol{v}_{t}^{i}$ carries a latent feature vector derived from the
source feature $\boldsymbol{F}^{3D}_{s}(\boldsymbol{v}_{s}^{i})$.

Target Feature Projection. After transforming each source appearance feature
related to $N$ source frame images, the 3D appearance features populate their
own 2D target features using the projection function
$\Pi_{t}(\boldsymbol{v}_{t}^{i})$ of the target camera parameter. At this
stage, only half of the SMPL vertices are visible from the target camera view.
To account for occlusions in the target features, we only project the vertices
visible from the target camera view to the 2D target image plane. The 2D
target feature map is formalized as

$\mathbf{F}_{\text{s}\rightarrow\text{t}}\big{(}\Pi_{t}(\boldsymbol{v}_{t}^{i})\big{)}=\boldsymbol{F}^{3D}_{s}(\boldsymbol{v}_{t}^{i}).$
(5)

After this stage, we obtain a target feature map for each source frame,
setting the stage for the subsequent rendering process.

###  3.3 Multi-Frame Aggregation and Rendering

The third and final stage of GNH focuses on fusing information from all source
images in the context of the target pose.

Multi-Frame Fusion. As shown in Fig. 2(c), our feature fusion transformer
$\Psi_{\text{multi\\_frame}}$ fuses the target features
$\mathbf{F}_{\text{s}_{i}\rightarrow\text{t}}$ from each source frame
$s_{i}\in\\{s_{1}\dots s_{N}\\}$ into a single target feature map which is
compatible with CNN-based image rendering. Intuitively, this transformer
highlights features from source images relevant to the target pose and
attenuates features from distant source frames. For each pixel $(u,v)$ in the
target image, $\Psi_{\text{multi\\_frame}}$ accepts $N$ tokens as input—one
from each source frame. Each token corresponds to the latent feature
$\mathbf{F}_{\text{s}_{i}\rightarrow\text{t}}(u,v)$ at that specific pixel
location $(u,v)$. This is formally represented as follows:

$\displaystyle\boldsymbol{\tau}_{s_{i}}$
$\displaystyle=\mathbf{F}_{\text{s}_{i}\rightarrow\text{t}}(u,v),$ (6)
$\displaystyle\boldsymbol{\Omega}_{u,v}$
$\displaystyle=\Psi_{\text{multi\\_frame}}(\boldsymbol{\tau}_{s_{1}},\boldsymbol{\tau}_{s_{2}},\dots,\boldsymbol{\tau}_{s_{N}}),$
(7)

where $\boldsymbol{\tau}_{s_{i}}$ is the $i^{\text{th}}$ input token to
$\Psi_{\text{multi\\_frame}}$, indicating the feature from source $s_{i}$
mapped to the pixel location in target $t$. $\boldsymbol{\Omega}_{u,v}$
signifies the fused multi-frame feature at each pixel.

Image Rendering. The multi-frame fusion transformer outputs a single CNN-
compatible feature map $\boldsymbol{\Omega}$ for the target pose, having
adaptively aggregated features from each source frame. GNH processes this
feature map with a CNN-based image rendering network to generate the target
output image. Specifically, the image rendering network within GNH adopts a
deep residual U-Net architecture, denoted as $\mathcal{R}$:

$\displaystyle\hat{\mathbf{I}}_{t}$
$\displaystyle=\mathcal{R}(\boldsymbol{\Omega}).$ (8)

###  3.4 Optimizing a Generalizable Neural Human Renderer

The learnable modules within our proposed GNH framework include the feature
refinement module $\boldsymbol{\phi}_{\text{refine}}$, the fine image feature
extractor $\boldsymbol{\phi}_{\text{fine}}$, the multi-frame fusion
transformer $\Psi_{\text{multi\\_frame}}$, and the image renderer
$\mathcal{R}$. During training, we randomly sample the pairs of $N$ source
frames and a target frame set for each training subject. By inputting the
source frames into the above-described three-stage process, we aim to
reconstruct the subject in the target pose. Four loss functions are used to
supervise the training.

Photometric Loss. Given the ground truth target image $\mathbf{I}_{t}$ and
predicted image $\hat{\mathbf{I}_{t}}$, we apply the photometric loss as
follows:

$\mathcal{L}_{\text{color}}=||\mathbf{I}_{t}-\hat{\mathbf{I}}_{t}||_{1}^{1}.$
(9)

Perceptual Loss. We also leverage the perceptual loss LPIPS to ensure the
quality of the rendered image:

$\mathcal{L}_{\text{LPIPS}}=\text{LPIPS}(\mathbf{I}_{t},\hat{\mathbf{I}}_{t}).$
(10)

Adversarial Loss. We employed adversarial training for our training procedure.
This adversarial training contributes to rendering detailed parts such as
clothing patterns and facial features, helping to the enhancement of image
resolution:

$\mathcal{L}_{\text{adv}}=\log(1-\mathcal{D}(\hat{\mathbf{I}}_{t})),$ (11)

where $\mathcal{D}$ denotes our discriminator module.

Anti-Bias Loss. We also employed an anti-bias loss. This anti-bias loss
effectively eliminates color shifts and is robust to minor misalignments.
Although perceptual and adversarial losses improve high-frequency details,
they may introduce color shifts [76]:

$\mathcal{L}_{\text{ab}}=||\text{DS}_{k}(\hat{I}_{t})-\text{DS}_{k}(I_{t})||_{1}^{1},$
(12)

where $\text{DS}_{k}(\cdot)$ denotes $k$-fold downsampling.

In summary, the overall loss function is as follows:

$\mathcal{L}=\lambda_{1}\mathcal{L}_{\text{color}}+\lambda_{2}\mathcal{L}_{\text{LPIPS}}+\lambda_{3}\mathcal{L}_{\text{adv}}+\lambda_{4}\mathcal{L}_{\text{ab}},$

where $\lambda_{i}$ denotes the weights of each loss.

##  4 Experiments

###  4.1 Experimental Setup

Datasets. We evaluated GNH using three widely used datasets: the ZJU-MoCap
[75], the People Snapshot [1], and the AIST++ dance motion [53]. For ZJU-
MoCap, we used six subjects recorded with “camera 1” for training, leaving
three subjects (387, 393, 394) for evaluation. In the People Snapshot dataset,
21 videos were utilized for training with three videos (female-3-casual,
male-1-casual, male-3-casual) set aside for evaluation. The AIST++ dataset,
featuring nine multi-view videos of 30 subjects in dynamic dance scenes, was
used by selecting one action sequence per subject for training with 25
subjects and evaluating on five subjects (16-20). We employed PointRend [45]
to generate foreground masks for AIST++. The test set involved sampling the
initial part of each video for source input frames and the remainder for
evaluation in the task of animatable human rendering. For further details,
please refer to the supplemental materials.

Baselines. We compare our method against state-of-the-art methods for modeling
humans in motion, including HumanNeRF [101] (test-time optimization required,
single video), ActorsNeRF [66] (generalizable training and test-time
optimization required, single video), Neural Human Performer (NHP) [48]
(multi-view video required), SHERF [30] (fully generalizable method, single
image input), and GHuNeRF [51] (fully generalizable method, single video
input).

Evaluation Metrics. To quantitatively evaluate the quality of the rendered
images, we report on three widely-used metrics: Peak Signal-to-Noise Ratio
(PSNR), Structural Similarity Index (SSIM) [100], and Learned Perceptual Image
Patch Similarity (LPIPS) [113]. To facilitate easier comparison in certain
evaluation setups, we set an “average” error metric [4] that summarizes all
three metrics: the geometric mean of three metrics: Mean Squared Error
$\mathrm{MSE}=10^{-\mathrm{PSNR}/10}$, dissimilarity measure
$\mathrm{dssim}=\sqrt{1-\mathrm{SSIM}}$ [58], and LPIPS. We report LPIPS and
“average” errors as multiplied by $\times 10^{3}$.

Implementation Details. We use the ResUnet++ [36] with group normalization
layers [102] for our image rendering network $\mathcal{R}$. For our
adversarial training, we employed the discriminator from Pix2Pix [34]. For the
coarse image feature, we utilize DINO-v2 [69] with ViT-S/14 architecture and
set the patch size as 64.

Optimization Details. Our framework contains five different modules to
optimize: the coarse feature refinement module, the fine image feature
extractor, the multi-frame feature fusion transformer, the image rendering
module, and the discriminative module for our adversarial training. We adopt
the Adam optimizer [44] for training with a learning rate of $1e^{-3}$ for the
feature encoder module, $5e^{-4}$ for the feature fusion module and the image
rendering module, and $1e^{-5}$ for the discriminator module with a batch size
of five (other Adam hyperparameters are left at default values of
$\beta_{1}=0.9$, $\beta_{2}=0.999$, and $\epsilon=10^{-7}$). We train our
network for 4k epochs for the ZJU-MoCap dataset, 7k epochs for the People
Snapshot dataset, and 6k epochs for the AIST++ dataset on one RTX 3090 GPU
(about 6-7 hours). For all datasets, we set
$\\{\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4}\\}=\\{0.2,0.1,0.05,0.8\\}$
for our objective function. We used $k=2$ for our anti-bias loss. For the
training set, we sampled one frame from the training video as the target and
randomly sampled source frames from the rest of the frames. We set $N=5$
during the training for all datasets.

Evaluation Details. During the evaluation, we sampled the source frames from
the first part of the evaluation video. To use the best feature for the target
pose, we sampled the most closely posed frame for the source frames by
calculating the torso rotation difference and Procrustes pose difference. We
also set $N=5$ for all evaluation setups except for Table. 4 and Fig. 7.
Following previous works, the evaluations are conducted with an image size of
$512\times 512$ for the ZJU-MoCap dataset and People Snapshot dataset,
$540\times 960$ for the AIST++ dataset.

Table 1: Quantitative comparison on the ZJU-MoCap dataset [75]. (*indicates
the numbers are from the original paper.)

| Multi | Test-time | Person387 | Person393 | Person394
---|---|---|---|---|---
| View | Opt. | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$
HumanNeRF [101] | ✗ | ✓ | 27.25 | 0.960 | 44.68 | 25.83 | 28.05 | 0.960 | 39.97 | 23.38 | 28.51 | 0.962 | 36.02 | 21.65
ActorsNeRF* [66] | ✗ | ✓ | 27.61 | 0.961 | 36.18 | - | 27.59 | 0.957 | 39.36 | - | 28.98 | 0.961 | 34.17 | -
NHP [48] | ✓ | ✗ | 28.28 | 0.965 | 59.77 | 25.60 | 29.86 | 0.967 | 54.41 | 21.75 | 29.93 | 0.966 | 52.11 | 21.56
SHERF [30] | ✗ | ✗ | 25.69 | 0.957 | 56.31 | 31.57 | 27.74 | 0.965 | 39.88 | 23.23 | 28.27 | 0.964 | 42.67 | 23.05
GHuNeRF [51] | ✗ | ✗ | 27.08 | 0.960 | 59.59 | 28.76 | 28.59 | 0.965 | 57.01 | 24.60 | 28.22 | 0.963 | 53.32 | 25.06
GNH (Ours) | ✗ | ✗ | 27.58 | 0.963 | 41.26 | 23.95 | 29.15 | 0.968 | 39.08 | 20.52 | 29.58 | 0.969 | 34.22 | 18.89

Figure 3:  Qualitative comparison of animatable rendering of unseen identity
on the ZJU-MoCap dataset [75].  Table 2: Quantitative comparison on the People
Snapshot dataset [1].

| Female-3-Casual | Male-1-Casual | Male-3-Casual
---|---|---|---
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$
GHuNeRF [51] | 29.63 | 0.962 | 90.97 | 26.89 | 26.15 | 0.939 | 99.77 | 39.10 | 27.93 | 0.952 | 99.47 | 32.09
GNH (Ours) | 29.60 | 0.964 | 67.41 | 24.18 | 27.50 | 0.944 | 75.00 | 31.74 | 27.20 | 0.950 | 76.06 | 32.29

Figure 4:  Qualitative comparison of animatable rendering of unseen identity
on the People Snapshot dataset [1].  Table 3: Quantitative comparison on the
AIST++ dataset [53]. (*indicates the numbers are from the original paper.)

| Test-time | Person 16 | Person 17 | Person18
---|---|---|---|---
| Opt. | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$
ActorsNeRF* [66] | ✓ | 25.73 | 0.981 | 18.93 | - | 26.14 | 0.983 | 18.37 | - | 25.03 | 0.983 | 18.52 | -
GHuNeRF [51] | ✗ | 24.14 | 0.981 | 24.67 | 23.66 | 24.79 | 0.982 | 27.08 | 23.13 | 23.12 | 0.980 | 28.14 | 27.28
GNH (Ours) | ✗ | 27.56 | 0.987 | 14.40 | 14.31 | 27.84 | 0.988 | 15.87 | 14.27 | 27.12 | 0.987 | 14.89 | 14.98
| Test-time | Person 19 | Person 20 |  |  |  |
| Opt. | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$ |  |  |  |
ActorsNeRF* | ✓ | 25.88 | 0.983 | 18.58 | - | 25.78 | 0.985 | 17.44 | - |  |  |  |
GHuNeRF [51] | ✗ | 24.68 | 0.981 | 23.69 | 22.39 | 26.20 | 0.958 | 22.21 | 18.78 |  |  |  |
GNH (Ours) | ✗ | 28.44 | 0.987 | 13.13 | 12.90 | 30.05 | 0.992 | 10.61 | 9.89 |  |  |  |

Figure 5:  Qualitative comparison of animatable rendering of unseen identity
on the AIST++ dataset [53].

###  4.2 Result

Qualitative Evaluation. The qualitative comparisons with baseline methods are
shown in Fig. 3, Fig. 4, and Fig. 5. Our GNH demonstrates superior rendering
performance both overall and in detail for all subjects compared to SHERF and
GHuNeRF. In Fig. 3, we observe that SHERF and GHuNeRF tend to overfit the
training set and render incorrect individuals during inference. In contrast,
GNH produces results that are visually on par with or superior to HumanNeRF
[101] and NHP [48], despite the latter requiring test-time optimization and
multi-viewpoint inputs, respectively. Fig. 4 demonstrates that our GNH can
render a more detailed structure precisely than GHuNeRF. We also found that
GHuNeRF cannot render meaningful output in our experiments as shown in Fig. 5,
likely because the AIST++ dataset is more difficult compared to the other two
datasets since it contains dynamic dancing motions.

Quantitative Evaluation. Our GNH significantly surpasses baseline methods in
quantitative evaluations across three key datasets as shown in Table 1, Table
2, and Table 3. In LPIPS scores, GNH achieves substantial improvements:
$31.5\%$ better than GHuNeRF on ZJU-MoCap, $24.6\%$ on People Snapshot, and
$45.2\%$ on AIST++. Against SHERF, GNH shows a $17.5\%$ improvement on ZJU-
MoCAP. Remarkably, GNH also outperforms HumanNeRF by $4.7\%$ on ZJU-MoCap,
demonstrating its efficiency without necessitating test-time optimization.

Number of Input Source Frames. Our quantitative evaluation on the ZJU-MoCap
dataset, considering a range of source frames (1, 3, 5, 7), demonstrates a
clear trend in Table 4: as the number of input frames increases, the average
rendering error notably decreases. This trend, calculated from the averages
for three test subjects (People 387, 393, 394), demonstrates our method’s
efficiency in handling multiple frames. Notably, even with a single frame
input, our method outperforms SHERF, a technique that aims to reconstruct
unseen parts for animating humans from a single image. A qualitative
comparison in Fig. 6 further illustrates this advantage, showing our method’s
superior capability in rendering animatable humans with greater accuracy and
detail.

Rendering Time. A key advantage of generalizable rendering is its ability to
complete rendering in a single forward pass, which is crucial for reducing
inference time in downstream applications. We report and compare the inference
frames per second (FPS) of GNH across various numbers of source frames with
the comparison to baseline generalizable human NeRF methods [30, 51] in Table
4. Although the number of input frames and rendering performance are inversely
related, GNH achieves a rendering speed that is $2\sim 7$ times faster than
baseline methods. Fig. 7 shows the trade-off relationship between the
rendering speed and the “average” rendering error. Increasing the number of
input frames improves rendering performance, but it results in slower
rendering speeds. Conversely, reducing the number of input frames speeds up
the rendering process but at the expense of performance.

Figure 6:  Qualitative comparison across various numbers of source frames.
Table 4: Quantitative evaluation on the ZJU-MoCap dataset across various
numbers of source frames. The numbers in parentheses indicate the difference
compared to GNH with a 5-frame input result.

Model | #Frames | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS*$\downarrow$ | Avg.* $\downarrow$ | FPS $\uparrow$
---|---|---|---|---|---|---
SHERF [30] | 1 | 27.23 | 0.962 | 46.29 | 25.95 | (+4.83) | 1.33 | ($\times$0.40)
GHuNeRF [51] | 15 | 28.10 | 0.963 | 55.75 | 25.58 | (+4.46) | 0.76 | ($\times$0.23)
GNH (Ours) | 1 | 28.66 | 0.966 | 38.19 | 21.39 | (+0.27) | 5.98 | ($\times$1.81)
3 | 28.73 | 0.966 | 38.05 | 21.19 | (+0.07) | 4.28 | ($\times$1.30)
5 | 28.77 | 0.967 | 38.18 | 21.12 | (+0.00) | 3.30 | ($\times$1.00)
7 | 28.80 | 0.967 | 38.32 | 21.08 | (-0.04) | 2.68 | ($\times$0.81)

Figure 7:  Rendering speed vs. “average” rendering error across various
numbers of source frames on the ZJU-MoCap dataset.  Figure 8:  Qualitative
evaluation of seen subject in unseen pose result on the ZJU-MoCap dataset.
Table 5: Quantitative evaluation of the rendering result of seen subject (the
six subject in train set) in unseen pose on the ZJU-MoCap dataset.

Model | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$
---|---|---|---|---
GHuNeRF [51] | 29.42 | 0.969 | 51.07 | 21.99
GNH (Ours) | 29.62 | 0.969 | 33.28 | 18.68

Table 6: Ablations and comparisons. Our final model is underlined.

Experiment | Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | Avg. $\downarrow$
---|---|---|---|---|---
Objective Function | w/o $\mathcal{L}_{\text{color}}$ | 27.14 | 0.963 | 41.08 | 24.78 | (+0.83)
w/o $\mathcal{L}_{\text{LPIPS}}$ | 26.45 | 0.956 | 56.71 | 29.94 | (+5.99)
w/o $\mathcal{L}_{\text{adv}}$ | 27.35 | 0.965 | 41.70 | 24.30 | (+0.35)
w/o $\mathcal{L}_{\text{ab}}$ | 27.24 | 0.963 | 42.00 | 24.81 | (+0.86)
Full Model | 27.58 | 0.963 | 41.26 | 23.95 |
Feature Encoder | $\mathbf{F}_{\text{fine}}$ | 27.32 | 0.961 | 41.91 | 24.56 | (+0.61)
$\mathbf{F}_{\text{coarse}}$ | 24.84 | 0.954 | 51.79 | 33.25 | (+9.30)
$\mathbf{F}_{\text{coarse}}\oplus\mathbf{F}_{\text{fine}}$ | 27.58 | 0.963 | 41.26 | 23.95 |
Metadata | w/o camera diff. | 27.16 | 0.962 | 42.38 | 25.09 | (+1.14)
w/o pose diff. | 27.51 | 0.961 | 42.35 | 24.52 | (+0.57)
w/o target pose | 27.16 | 0.963 | 42.11 | 24.97 | (+1.02)
Full Model | 27.58 | 0.963 | 41.26 | 23.95 |
Normalization | Batch Norm | 27.10 | 0.960 | 53.67 | 27.56 | (+ 3.61)
Group Norm | 27.58 | 0.963 | 41.26 | 23.95 |
Rendering Module | ResUNet[114] | 27.36 | 0.964 | 41.81 | 24.42 | (+0.47)
ResUNet++[36] | 27.58 | 0.963 | 41.26 | 23.95 |

Rendering Result of Seen Identities. We additionally present an evaluation of
animatable rendering for seen identities. In this experiment, we utilized
videos from six subjects (313, 315, 377, 386, 390, 392) within our training
set and assessed frames from each video’s final part, which were not used in
our generalizable training. The quantitative and qualitative results are shown
in Table 5 and Fig. 8 respectively. For both metrics, our GNH surpassed the
baseline method, showcasing superior performance.

Figure 9:  Ablation study of the objective function.

Figure 10: Ablation study of the feature encoder module.

Figure 11: Ablation study of the metadata for multi-frame fusion transformer.

###  4.3 Ablations and Other Analysis

We conduct a series of ablation studies on the ZJU-MoCap dataset to validate
the effectiveness of our architectural designs. The scores reported reflect
the 5-shot results for subject 387 from the ZJU-MoCap dataset. The results are
in Table 6.

Objective Function. To validate the effectiveness of our proposed optimization
function, we demonstrate the effect of each element. In addition to the
quantitative evaluation demonstrating that each element contributes to
performance improvement, we also report the qualitative comparison in Fig. 9.
The photometric loss $\mathcal{L}_{\text{L1}}$, the discriminative loss
$\mathcal{L}_{\text{Disc}}$, and the anti-bias loss $\mathcal{L}_{\text{ab}}$
are contributing to the representation of detailed parts and the perspective
loss $\mathcal{L}_{\text{VGG}}$ is contributing to the overall visual quality.

Feature Encoder Design. To validate the effectiveness of our feature encoder
which extracts appearance features from source frames, we demonstrate the
effect of each element by removing elements. In addition to the quantitative
evaluation showing that both course image feature $\mathbf{F}_{\text{coarse}}$
and fine image feature $\mathbf{F}_{\text{fine}}$ contribute to the
performance improvement, we also report the qualitative comparison in Fig. 11.

Metadata for the Multi-Frame Fusion Transformer. We ablate the metadata input
for the multi-frame fusion transformer by demonstrating the effect of each
element. In addition to the quantitative evaluation showing that all of the
metadata contribute to improving the performance, we also report the
qualitative comparison in Fig. 11.

Normalization Layer. Since we utilize the group normalization[102] instead of
the batch normalization used in the official implementation of ResUNet++[36],
we compare the performance of these normalization layers. The quantitative
evaluation shows that the group normalization improves the rendering
performance by $23.1\%$ in LPIPS in our experiment.

Rendering Module. We implement a variant of the image rendering module with
ResUNet [114]. The result shows the overall performance of this version is
lower than our final model.

##  5 Discussion and Conclusion

We introduce the Generalizable Neural Human Renderer (GNH), a pioneering
approach for rendering high-fidelity, animatable humans from monocular video
inputs, eliminating the need for test-time optimization. GNH leverages a
streamlined three-stage rendering process that not only simplifies the
workflow but also significantly enhances the quality of the output. This
method has been rigorously tested and has demonstrated superior performance,
both qualitatively and quantitatively, across three dynamic human motion
datasets, underscoring its robustness and versatility in handling complex
animations.

Limitations and Future Work. GNH skillfully utilizes source views to render
accurate target views and poses, yet it relies on precise pose and mask
estimations in input views. Although our approach is proficient in handling
SMPL-based human topology transformations, it does not account for dynamic
lighting changes, which could compromise its performance under varied lighting
conditions. Acknowledging these limitations, we see substantial opportunities
to improve pose and mask estimation accuracy and to enhance adaptability to
diverse lighting conditions. These areas offer fertile ground for future
research, aiming to enhance the robustness and applicability of GNH and
similar methodologies.

## References

  * [1] Alldieck, T., Magnor, M., Xu, W., Theobalt, C., Pons-Moll, G.: Video based reconstruction of 3d people models. In: CVPR. pp. 8387–8397 (2018)
  * [2] Alldieck, T., Zanfir, M., Sminchisescu, C.: Photorealistic monocular 3d reconstruction of humans wearing clothing. In: CVPR. pp. 1506–1515 (2022)
  * [3] Balakrishnan, G., Zhao, A., Dalca, A.V., Durand, F., Guttag, J.: Synthesizing images of humans in unseen poses. In: CVPR. pp. 8340–8348 (2018)
  * [4] Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In: ICCV. pp. 5855–5864 (2021)
  * [5] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In: CVPR. pp. 5470–5479 (2022)
  * [6] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Zip-nerf: Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.06706 (2023)
  * [7] Bühler, M.C., Sarkar, K., Shah, T., Li, G., Wang, D., Helminger, L., Orts-Escolano, S., Lagun, D., Hilliges, O., Beeler, T., et al.: Preface: A data-driven volumetric prior for few-shot ultra high-resolution face synthesis. In: ICCV. pp. 3402–3413 (2023)
  * [8] Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2d pose estimation using part affinity fields. In: CVPR. pp. 7291–7299 (2017)
  * [9] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV. pp. 9650–9660 (2021)
  * [10] Carranza, J., Theobalt, C., Magnor, M.A., Seidel, H.P.: Free-viewpoint video of human actors. ACM transactions on graphics 22(3), 569–577 (2003)
  * [11] Casas, D., Volino, M., Collomosse, J., Hilton, A.: 4d video textures for interactive character appearance. In: Computer Graphics Forum. vol. 33, pp. 371–380 (2014)
  * [12] Chan, C., Ginosar, S., Zhou, T., Efros, A.A.: Everybody dance now. In: ICCV. pp. 5933–5942 (2019)
  * [13] Chen, A., Xu, Z., Zhao, F., Zhang, X., Xiang, F., Yu, J., Su, H.: Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In: ICCV. pp. 14124–14133 (2021)
  * [14] Chen, M., Zhang, J., Xu, X., Liu, L., Cai, Y., Feng, J., Yan, S.: Geometry-guided progressive nerf for generalizable and efficient neural human rendering. In: ECCV. pp. 222–239 (2022)
  * [15] Chen, X., Wang, B., Shum, H.Y.: Hand avatar: Free-pose hand animation and rendering from monocular video. In: CVPR. pp. 8683–8693 (2023)
  * [16] Chen, Y., Wang, L., Li, Q., Xiao, H., Zhang, S., Yao, H., Liu, Y.: Monogaussianavatar: Monocular gaussian point-based head avatar. arXiv preprint arXiv:2312.04558 (2023)
  * [17] Choi, H., Moon, G., Armando, M., Leroy, V., Lee, K.M., Rogez, G.: Mononhr: Monocular neural human renderer. In: 3DV. pp. 242–251 (2022)
  * [18] Corona, E., Hodan, T., Vo, M., Moreno-Noguer, F., Sweeney, C., Newcombe, R., Ma, L.: Lisa: Learning implicit shape and appearance of hands. In: CVPR. pp. 20533–20543 (2022)
  * [19] Dhamo, H., Nie, Y., Moreau, A., Song, J., Shaw, R., Zhou, Y., Pérez-Pellitero, E.: Headgas: Real-time animatable head avatars via 3d gaussian splatting. arXiv preprint arXiv:2312.02902 (2023)
  * [20] Flagg, M., Nakazawa, A., Zhang, Q., Kang, S.B., Ryu, Y.K., Essa, I., Rehg, J.M.: Human video textures. In: Proceedings of the 2009 symposium on Interactive 3D graphics and games. pp. 199–206 (2009)
  * [21] Gao, Q., Wang, Y., Liu, L., Liu, L., Theobalt, C., Chen, B.: Neural novel actor: Learning a generalized animatable neural representation for human actors. IEEE Transactions on Visualization and Computer Graphics (2023)
  * [22] Gao, X., Yang, J., Kim, J., Peng, S., Liu, Z., Tong, X.: Mps-nerf: Generalizable 3d human rendering from multiview images. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  * [23] Geng, C., Peng, S., Xu, Z., Bao, H., Zhou, X.: Learning neural volumetric representations of dynamic humans in minutes. In: CVPR. pp. 8759–8770 (2023)
  * [24] Grigorev, A., Sevastopolsky, A., Vakhitov, A., Lempitsky, V.: Coordinate-based texture inpainting for pose-guided human image generation. In: CVPR. pp. 12135–12144 (2019)
  * [25] Güler, R.A., Neverova, N., Kokkinos, I.: Densepose: Dense human pose estimation in the wild. In: CVPR. pp. 7297–7306 (2018)
  * [26] Guo, Z., Zhou, W., Wang, M., Li, L., Li, H.: Handnerf: Neural radiance fields for animatable interacting hands. In: CVPR. pp. 21078–21087 (2023)
  * [27] He, T., Collomosse, J., Jin, H., Soatto, S.: Geo-pifu: Geometry and pixel aligned implicit functions for single-view human reconstruction. NeurIPS 33, 9276–9287 (2020)
  * [28] He, T., Xu, Y., Saito, S., Soatto, S., Tung, T.: Arch++: Animation-ready clothed human reconstruction revisited. In: ICCV. pp. 11046–11056 (2021)
  * [29] Hu, L., Zhang, H., Zhang, Y., Zhou, B., Liu, B., Zhang, S., Nie, L.: Gaussianavatar: Towards realistic human avatar modeling from a single video via animatable 3d gaussians. In: CVPR (2024)
  * [30] Hu, S., Hong, F., Pan, L., Mei, H., Yang, L., Liu, Z.: Sherf: Generalizable human nerf from a single image. In: ICCV. pp. 9352–9364 (2023)
  * [31] Hu, S., Liu, Z.: Gauhuman: Articulated gaussian splatting from monocular human videos. arXiv preprint arXiv:2312.02973 (2023)
  * [32] Huang, Z., Xu, Y., Lassner, C., Li, H., Tung, T.: Arch: Animatable reconstruction of clothed humans. In: CVPR. pp. 3093–3102 (2020)
  * [33] Iqbal, U., Caliskan, A., Nagano, K., Khamis, S., Molchanov, P., Kautz, J.: Rana: Relightable articulated neural avatars. In: ICCV. pp. 23142–23153 (2023)
  * [34] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: CVPR. pp. 1125–1134 (2017)
  * [35] Jena, R., Iyer, G.S., Choudhary, S., Smith, B., Chaudhari, P., Gee, J.: Splatarmor: Articulated gaussian splatting for animatable humans from monocular rgb videos. arXiv preprint arXiv:2311.10812 (2023)
  * [36] Jha, D., Smedsrud, P.H., Riegler, M.A., Johansen, D., De Lange, T., Halvorsen, P., Johansen, H.D.: Resunet++: An advanced architecture for medical image segmentation. In: ISM. pp. 225–2255 (2019)
  * [37] Jiang, T., Chen, X., Song, J., Hilliges, O.: Instantavatar: Learning avatars from monocular video in 60 seconds. In: CVPR. pp. 16922–16932 (2023)
  * [38] Jiang, W., Yi, K.M., Samei, G., Tuzel, O., Ranjan, A.: Neuman: Neural human radiance field from a single video. In: ECCV. pp. 402–418. Springer (2022)
  * [39] Jiang, Z., Rahmani, H., Black, S., Williams, B.M.: 3d points splatting for real-time dynamic hand reconstruction. arXiv preprint arXiv:2312.13770 (2023)
  * [40] Johari, M.M., Lepoittevin, Y., Fleuret, F.: Geonerf: Generalizing nerf with geometry priors. In: CVPR. pp. 18365–18375 (2022)
  * [41] Jung, H., Brasch, N., Song, J., Perez-Pellitero, E., Zhou, Y., Li, Z., Navab, N., Busam, B.: Deformable 3d gaussian splatting for animatable human avatars. arXiv preprint arXiv:2312.15059 (2023)
  * [42] Kanade, T., Rander, P., Narayanan, P.: Virtualized reality: constructing virtual worlds from real scenes. IEEE MultiMedia 4(1), 34–47 (1997)
  * [43] Kania, K., Garbin, S.J., Tagliasacchi, A., Estellers, V., Yi, K.M., Valentin, J., Trzciński, T., Kowalski, M.: Blendfields: Few-shot example-driven facial modeling. In: CVPR. pp. 404–415 (2023)
  * [44] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [45] Kirillov, A., Wu, Y., He, K., Girshick, R.: Pointrend: Image segmentation as rendering. In: CVPR. pp. 9799–9808 (2020)
  * [46] Kocabas, M., Chang, J.H.R., Gabriel, J., Tuzel, O., Ranjan, A.: Hugs: Human gaussian splats. arXiv preprint arXiv:2311.17910 (2023)
  * [47] Kwon, Y., Kim, D., Ceylan, D., Fuchs, H.: Neural human performer: Learning generalizable radiance fields for human performance rendering. NeurIPS 34, 24741–24752 (2021)
  * [48] Kwon, Y., Kim, D., Ceylan, D., Fuchs, H.: Neural image-based avatars: Generalizable radiance fields for human avatar modeling. In: ICLR (2022)
  * [49] Lan, Y., Tan, F., Qiu, D., Xu, Q., Genova, K., Huang, Z., Fanello, S., Pandey, R., Funkhouser, T., Loy, C.C., et al.: Gaussian3diff: 3d gaussian diffusion for 3d full head synthesis and editing. arXiv preprint arXiv:2312.03763 (2023)
  * [50] Lei, J., Wang, Y., Pavlakos, G., Liu, L., Daniilidis, K.: Gart: Gaussian articulated template models. In: CVPR (2024)
  * [51] Li, C., Lin, J., Lee, G.H.: Ghunerf: Generalizable human nerf from a monocular video. 3DV (2024)
  * [52] Li, M., Tao, J., Yang, Z., Yang, Y.: Human101: Training 100+ fps human gaussians in 100s from 1 view. arXiv preprint arXiv:2312.15258 (2023)
  * [53] Li, R., Yang, S., Ross, D.A., Kanazawa, A.: Ai choreographer: Music conditioned 3d dance generation with aist++. In: ICCV. pp. 13401–13412 (2021)
  * [54] Li, Z., Zheng, Z., Wang, L., Liu, Y.: Animatable gaussians: Learning pose-dependent gaussian maps for high-fidelity human avatar modeling. In: CVPR (2024)
  * [55] Liu, L., Habermann, M., Rudnev, V., Sarkar, K., Gu, J., Theobalt, C.: Neural actor: Neural free-view synthesis of human actors with pose control. ACM TOG 40(6), 1–16 (2021)
  * [56] Liu, L., Xu, W., Zollhoefer, M., Kim, H., Bernard, F., Habermann, M., Wang, W., Theobalt, C.: Neural rendering and reenactment of human actor videos. ACM TOG 38(5), 1–14 (2019)
  * [57] Liu, Y., Peng, S., Liu, L., Wang, Q., Wang, P., Theobalt, C., Zhou, X., Wang, W.: Neural rays for occlusion-aware image-based rendering. In: CVPR. pp. 7824–7833 (2022)
  * [58] Lombardi, S., Simon, T., Saragih, J., Schwartz, G., Lehrmann, A., Sheikh, Y.: Neural volumes: Learning dynamic renderable volumes from images. ACM TOG 38 (2019)
  * [59] Long, X., Lin, C., Wang, P., Komura, T., Wang, W.: Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In: ECCV. pp. 210–227 (2022)
  * [60] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: A skinned multi-person linear model. ACM TOG 34(6) (2015)
  * [61] Ma, L., Jia, X., Sun, Q., Schiele, B., Tuytelaars, T., Van Gool, L.: Pose guided person image generation. NeurIPS 30 (2017)
  * [62] Matusik, W., Buehler, C., Raskar, R., Gortler, S.J., McMillan, L.: Image-based visual hulls. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques. pp. 369–374 (2000)
  * [63] Mihajlovic, M., Bansal, A., Zollhoefer, M., Tang, S., Saito, S.: Keypointnerf: Generalizing image-based volumetric avatars using relative spatial encoding of keypoints. In: ECCV. pp. 179–197 (2022)
  * [64] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65(1), 99–106 (2021)
  * [65] Moreau, A., Song, J., Dhamo, H., Shaw, R., Zhou, Y., Pérez-Pellitero, E.: Human gaussian splatting: Real-time rendering of animatable avatars. arXiv preprint arXiv:2311.17113 (2023)
  * [66] Mu, J., Sang, S., Vasconcelos, N., Wang, X.: Actorsnerf: Animatable few-shot human rendering with generalizable nerfs. In: ICCV. pp. 18391–18401 (2023)
  * [67] Neverova, N., Guler, R.A., Kokkinos, I.: Dense pose transfer. In: ECCV. pp. 123–138 (2018)
  * [68] Noguchi, A., Sun, X., Lin, S., Harada, T.: Neural articulated radiance field. In: ICCV. pp. 5762–5772 (2021)
  * [69] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
  * [70] Pan, X., Yang, Z., Ma, J., Zhou, C., Yang, Y.: Transhuman: A transformer-based human representation for generalizable neural human rendering. In: ICCV. pp. 3544–3555 (2023)
  * [71] Pang, H., Zhu, H., Kortylewski, A., Theobalt, C., Habermann, M.: Ash: Animatable gaussian splats for efficient and photoreal human rendering. arXiv preprint arXiv:2312.05941 (2023)
  * [72] Park, K., Sinha, U., Barron, J.T., Bouaziz, S., Goldman, D.B., Seitz, S.M., Martin-Brualla, R.: Nerfies: Deformable neural radiance fields. In: ICCV. pp. 5865–5874 (2021)
  * [73] Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A., Tzionas, D., Black, M.J.: Expressive body capture: 3d hands, face, and body from a single image. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10975–10985 (2019)
  * [74] Peng, S., Dong, J., Wang, Q., Zhang, S., Shuai, Q., Zhou, X., Bao, H.: Animatable neural radiance fields for modeling dynamic human bodies. In: ICCV. pp. 14314–14323 (2021)
  * [75] Peng, S., Zhang, Y., Xu, Y., Wang, Q., Shuai, Q., Bao, H., Zhou, X.: Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In: CVPR. pp. 9054–9063 (2021)
  * [76] Prinzler, M., Hilliges, O., Thies, J.: Diner: Depth-aware image-based neural radiance fields. In: CVPR. pp. 12449–12459 (2023)
  * [77] Pumarola, A., Corona, E., Pons-Moll, G., Moreno-Noguer, F.: D-nerf: Neural radiance fields for dynamic scenes. In: CVPR. pp. 10318–10327 (2021)
  * [78] Qian, Z., Wang, S., Mihajlovic, M., Geiger, A., Tang, S.: 3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting. In: CVPR (2024)
  * [79] Raj, A., Zollhofer, M., Simon, T., Saragih, J., Saito, S., Hays, J., Lombardi, S.: Pixel-aligned volumetric avatars. In: CVPR. pp. 11733–11742 (2021)
  * [80] Ren, Y., Zhang, T., Pollefeys, M., Süsstrunk, S., Wang, F.: Volrecon: Volume rendering of signed ray distance functions for generalizable multi-view reconstruction. In: CVPR. pp. 16685–16695 (2023)
  * [81] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR. pp. 10684–10695 (2022)
  * [82] Romero, J., Tzionas, D., Black, M.J.: Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610 (2022)
  * [83] Saito, S., Schwartz, G., Simon, T., Li, J., Nam, G.: Relightable gaussian codec avatars. In: CVPR (2024)
  * [84] Sarkar, K., Golyanik, V., Liu, L., Theobalt, C.: Style and pose control for image synthesis of humans from a single monocular view. arXiv preprint arXiv:2102.11263 (2021)
  * [85] Sarkar, K., Mehta, D., Xu, W., Golyanik, V., Theobalt, C.: Neural re-rendering of humans from a single image. In: ECCV. pp. 596–613 (2020)
  * [86] Shen, K., Guo, C., Kaufmann, M., Zarate, J.J., Valentin, J., Song, J., Hilliges, O.: X-avatar: Expressive human avatars. In: CVPR. pp. 16911–16921 (2023)
  * [87] Siarohin, A., Sangineto, E., Lathuiliere, S., Sebe, N.: Deformable gans for pose-based human image generation. In: CVPR. pp. 3408–3416 (2018)
  * [88] Starck, J., Hilton, A.: Surface capture for performance-based animation. IEEE computer graphics and applications 27(3), 21–31 (2007)
  * [89] Su, S.Y., Bagautdinov, T., Rhodin, H.: Danbo: Disentangled articulated neural body representations via graph neural networks. In: ECCV. pp. 107–124 (2022)
  * [90] Su, S.Y., Yu, F., Zollhöfer, M., Rhodin, H.: A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. NeurIPS 34, 12278–12291 (2021)
  * [91] Sun, W., Che, Y., Huang, H., Guo, Y.: Neural reconstruction of relightable human model from monocular video. In: ICCV. pp. 397–407 (2023)
  * [92] Svitov, D., Gudkov, D., Bashirov, R., Lempitsky, V.: Dinar: Diffusion inpainting of neural textures for one-shot human avatars. In: ICCV. pp. 7062–7072 (2023)
  * [93] T, M.V., Wang, P., Chen, X., Chen, T., Venugopalan, S., Wang, Z.: Is attention all that nerf needs? In: ICLR (2023)
  * [94] Tian, F., Du, S., Duan, Y.: Mononerf: Learning a generalizable dynamic radiance field from monocular videos. In: ICCV. pp. 17903–17913 (2023)
  * [95] Trevithick, A., Chan, M., Stengel, M., Chan, E., Liu, C., Yu, Z., Khamis, S., Chandraker, M., Ramamoorthi, R., Nagano, K.: Real-time radiance fields for single-image portrait view synthesis. TOG 42(4), 1–15 (2023)
  * [96] Wang, J., Xie, J.C., Li, X., Xu, F., Pun, C.M., Gao, H.: Gaussianhead: Impressive head avatars with learnable gaussian diffusion. arXiv preprint arXiv:2312.01632 (2023)
  * [97] Wang, Q., Wang, Z., Genova, K., Srinivasan, P.P., Zhou, H., Barron, J.T., Martin-Brualla, R., Snavely, N., Funkhouser, T.: Ibrnet: Learning multi-view image-based rendering. In: CVPR. pp. 4690–4699 (2021)
  * [98] Wang, S., Schwarz, K., Geiger, A., Tang, S.: Arah: Animatable volume rendering of articulated human sdfs. In: ECCV. pp. 1–19 (2022)
  * [99] Wang, T.Y., Ceylan, D., Singh, K.K., Mitra, N.J.: Dance in the wild: Monocular human animation with neural dynamic appearance synthesis. In: 3DV. pp. 268–277 (2021)
  * [100] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600–612 (2004)
  * [101] Weng, C.Y., Curless, B., Srinivasan, P.P., Barron, J.T., Kemelmacher-Shlizerman, I.: Humannerf: Free-viewpoint rendering of moving people from monocular video. In: CVPR. pp. 16210–16220 (2022)
  * [102] Wu, Y., He, K.: Group normalization. In: ECCV. pp. 3–19 (2018)
  * [103] Xiang, J., Gao, X., Guo, Y., Zhang, J.: Flashavatar: High-fidelity digital avatar rendering at 300fps. In: CVPR (2024)
  * [104] Xiang, T., Sun, A., Wu, J., Adeli, E., Fei-Fei, L.: Rendering humans from object-occluded monocular videos. In: ICCV. pp. 3239–3250 (2023)
  * [105] Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: Segformer: Simple and efficient design for semantic segmentation with transformers. NeurIPS 34, 12077–12090 (2021)
  * [106] Xiu, Y., Yang, J., Tzionas, D., Black, M.J.: Icon: Implicit clothed humans obtained from normals. In: CVPR. pp. 13286–13296. IEEE (2022)
  * [107] Xu, H., Alldieck, T., Sminchisescu, C.: H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. NeurIPS 34, 14955–14966 (2021)
  * [108] Xu, M., Zhan, F., Zhang, J., Yu, Y., Zhang, X., Theobalt, C., Shao, L., Lu, S.: Wavenerf: Wavelet-based generalizable neural radiance fields. In: ICCV. pp. 18195–18204 (2023)
  * [109] Xu, Y., Chen, B., Li, Z., Zhang, H., Wang, L., Zheng, Z., Liu, Y.: Gaussian head avatar: Ultra high-fidelity head avatar via dynamic gaussians. In: CVPR (2024)
  * [110] Ye, J., Wang, N., Wang, X.: Featurenerf: Learning generalizable nerfs by distilling foundation models. In: ICCV. pp. 8962–8973 (2023)
  * [111] Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelnerf: Neural radiance fields from one or few images. In: CVPR. pp. 4578–4587 (2021)
  * [112] Yu, Z., Cheng, W., Liu, X., Wu, W., Lin, K.Y.: Monohuman: Animatable human neural field from monocular video. In: CVPR. pp. 16943–16953 (2023)
  * [113] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR. pp. 586–595 (2018)
  * [114] Zhang, Z., Liu, Q., Wang, Y.: Road extraction by deep residual u-net. IEEE Geoscience and Remote Sensing Letters 15(5), 749–753 (2018)
  * [115] Zhao, F., Yang, W., Zhang, J., Lin, P., Zhang, Y., Yu, J., Xu, L.: Humannerf: Efficiently generated human radiance field from sparse inputs. In: CVPR. pp. 7743–7753 (2022)
  * [116] Zhao, X., Colburn, A., Ma, F., Bautista, M.A., Susskind, J.M., Schwing, A.G.: Is generalized dynamic novel view synthesis from monocular videos possible today? (2023)
  * [117] Zhao, Z., Bao, Z., Li, Q., Qiu, G., Liu, K.: Psavatar: A point-based morphable shape model for real-time head avatar creation with 3d gaussian splatting. arXiv preprint arXiv:2401.12900 (2024)
  * [118] Zheng, S., Zhou, B., Shao, R., Liu, B., Zhang, S., Nie, L., Liu, Y.: Gps-gaussian: Generalizable pixel-wise 3d gaussian splatting for real-time human novel view synthesis. arXiv preprint arXiv:2312.02155 (2023)
  * [119] Zielonka, W., Bagautdinov, T., Saito, S., Zollhöfer, M., Thies, J., Romero, J.: Drivable 3d gaussian avatars. arXiv preprint arXiv:2311.08581 (2023)

##  A Dataset Details

This section includes the detailed datasets training setup.

ZJU-MoCap Dataset. For the ZJU-MoCap dataset, we utilized the first 550 frames
for the training. For the unseen subject evaluation, we used the first 300
frames as the source input video and evaluated our model on the next 250
frames for every 10 frames. For seen subject evaluation, we utilized the first
550 frames which is used for training as the source input video, and evaluated
our model on the rest of the video for every 10 frames.

People Snapshot Dataset. For the People Snapshot dataset, we utilized every
five frames of the whole video for the training. For the evaluation, we used
every five frames from the first 300 frames for the source input video and
evaluated our model on every 10 frames of the rest of the video.

AIST++ Dataset. For the AIST++ dataset, we utilized every four frames from
200-800 frames of each video for the training. For the evaluation, we used
every four frames from 200-1400 frames for the source input video and
evaluated our model on every four frames of the rest of the video.

Figure B1:  Structure of our feature encoder module.

##  B Network Structure

The detailed structure of our feature encoder module is shown in Fig. B1.

Coarse Feature Refinement Net $\phi_{\text{refine}}$. We process the output
coarse feature $\mathbf{F}_{\text{coarse}}$ with a convolutional transformer
architecture. We use patch embedding and four layers of efficient self-
attention blocks from Segformer [105]. Three convolutional layers follow this
transformer structure and produce final coarse feature maps
$\mathbf{F}_{\text{coarse}}$ with dimensions of $256\times 256\times 96$.

Fine Feature Encoder $\phi_{\text{fine}}$. In parallel to the coarse feature
extraction process, we extract fine-grained features using a five-layer
convolutional network, which includes only one downsampling layer. We use the
Leaky Rectified Linear Unit (LeakyReLU) for activation. Following the
convolutional network, the feature image is upsampled using bilinear
interpolation, resulting in refined feature maps $\mathbf{F}_{\text{fine}}$
with dimensions of $256\times 256\times 96$.

Multi-Frame Fusion Transformer $\Psi_{\text{multi\\_frame}}$. To fuse feature
maps from each source image into a single feature map, we use a transformer
network equipped with cross-view attention [93]. Considering that not all
pixels possess distinct features, our approach carefully targets only those
pixels with projected features, enhancing computational efficiency. The
architecture of our transformer blocks is uniformly designed with a dimension
of 256 and we set the transformer depth as four.