# Flocking to Mastodon: Tracking the Great Twitter Migration

Haris Bin Zia,1 Jiahui He,2 Aravindh Raman,3 Ignacio Castro,1 Nishanth
Sastry,4 Gareth Tyson2,1
1Queen Mary University of London 2Hong Kong University of Science and
Technology (GZ)
3Telefonica Research 4University of Surrey

###### Abstract

The acquisition of Twitter by Elon Musk has spurred controversy and
uncertainty among Twitter users. The move raised as many praises as concerns,
particularly regarding Musk’s views on free speech. As a result, a large
number of Twitter users have looked for alternatives to Twitter. Mastodon, a
decentralized micro-blogging social network, has attracted the attention of
many users and the general media. In this paper, we track and analyze the
migration of 136,009 users from Twitter to Mastodon. Our analysis sheds light
on the user-driven pressure towards centralization in a decentralized
ecosystem and identifies the strong influence of the social network in
platform migration. We also characterize the activity of migrated users on
both Twitter and Mastodon.

##  1 Introduction

In October 2022, Elon Musk, a self-declared “free speech absolutist” acquired
Twitter — the social network that he regarded as the “de facto town square”
where public debate takes place. Musk’s takeover has been controversial and
highly publicized. Some users admire Musk and his takeover, regarding it as
crucial for free speech; others have expressed concerns over increased
misinformation and toxicity.

Regardless of one’s stance, it is undeniable that the acquisition has led to a
series of noteworthy events. On November 04, 2022, Musk fired half of the
7,500 employees previously working at Twitter. Two weeks later (November 17,
2022), hundreds of employees resigned in response to an ultimatum to commit to
“extremely hardcore” work or leave. These events and the associated public
backlash, prompted many users to search for alternatives. Figure 1(a) presents
a time series of Google trend search interest for “Twitter alternatives”. We
observe a large spike on October 28, 2022, the day after Musk’s takeover.
Similarly, Figure 1(b) shows equivalent search interest for other popular
alternatives to Twitter, _e.g._ Koo (an Indian micro-blogging and social
networking service), and Hive (a micro-blogging service that permits NSFW
mature content).

One platform that stands out as being particularly prominent is _Mastodon_ , a
decentralized micro-blogging platform. Although released in 2016, Mastodon has
anecdotally gathered significant attention since October 2022. It is part of
the wider _fediverse_ , in which any person can create and operate their own
Mastodon server (aka “instance”). Each Mastodon instance operates as an
independent microblogging service, where users can create local accounts and
enjoy similar functions to Twitter (_e.g._ posting, following). Importantly,
these instances can also federate together, allowing users on one instance to
follow users on another. This means that Mastodon operates in a decentralized
fashion (with people joining independent instances), while retaining the
ability to interact across the entire globe. This new paradigm has attracted
significant attention and has made it an obvious candidate for users who are
unhappy with the Musk acquisition (and the associated centralization of power
in the hands of one individual).

(a)

(b)

Figure 1: Interest over time for the search terms (a) Twitter alternatives and
(b) Mastodon, Koo & Hive Social.

This sudden interest in Mastodon offers a unique opportunity to study the
migration of users between social networks. This is particularly the case due
to the differing value propositions of the two platforms, with clear contrasts
in the governance and ownership of Twitter vs. Mastodon. The unusual
circumstances of the migration create further dimensions of analysis. With
this context in mind, we explore the following three research questions:

  * •

RQ1 How are new users spread across Mastodon instances, and are there any
consequences for decentralization?

  * •

RQ2 How much (if at all) does a user’s ego-centric Twitter follower network
influence their migration to Mastodon?

  * •

RQ3 What are usage patterns of migrated users across both platforms?

To address these questions, we track 136,009 unique twitter users who moved to
2,879 unique Mastodon instances. The main findings related to our three RQs
are as follows:

  * •

There is a user-driven pressure towards centralization on Mastodon (the top
25% most populous instances contain 96% of the users). This pressure is
counterbalanced by the greater activity of the users on smaller instances. On
average, users of single-user instances post 121% more statuses than users on
bigger instances.

  * •

The social network of users on Twitter influences their choice of an instance
on Mastodon _e.g._ 4.09% of users changed the instance on which they created
an account (when they first migrated to Mastodon) and moved to the instance of
choice of their Twitter followees who migrated to Mastodon as well.

  * •

Users tend to post different content across the two platforms. On average,
only 1.53% of a user’s Mastodon posts are identical to their Twitter posts.
Twitter hosts more diverse topics ranging from Entertainment to Politics,
whereas discussions around Fediverse and Migration dominate on Mastodon.

##  2 Mastodon Primer

Mastodon is an open-source [15] federated server platform released in 2016. It
offers micro-blogging functionality, allowing administrators to create their
own independent Mastodon servers, aka instances. Each unique Mastodon instance
works much like Twitter, allowing users to register new accounts and share
statuses with their followers – equivalent to tweeting on Twitter. Users can
also boost others’ statuses – equivalent to retweeting on Twitter.

Instances can work in isolation, only allowing locally registered users to
follow each other. However, Mastodon instances can also federate, whereby
users registered on one instance can follow users registered on another
instance. This results in the instance subscribing to posts performed on the
remote instance, such that they can be pushed across and presented to local
users. For simplicity, we refer to users registered on the same instance as
local, and users registered on different instances as remote. Note that a user
registered on their local instance does _not_ need to register with the remote
instance to follow the remote user. Instead, a user just creates a single
account with their local instance; when the user wants to follow a user on a
remote instance, the user’s local instance performs the subscription on the
user’s behalf. This process is implemented using an underlying subscription
protocol, ActivityPub [1]. This makes Mastodon compatible with other
decentralised micro-blogging implementations (notably, Pleroma). The
Fediverse, refers to the growing group of ActivityPub compatible, and
therefore interconnected, applications.

When a user logs in to their local instance, they are presented with three
timelines: (i) a home timeline, with statuses shared by the accounts whom the
user follows; (ii) a local timeline, listing the statuses generated within the
same instance; and (iii) a federated timeline, with _all_ statuses that have
been retrieved from remote instances. The latter is not limited to remote
statuses that the user follows; rather, it is the union of remote statuses
retrieved by all users on the instance.

##  3 Data Collection

###  3.1 Mastodon Accounts from Twitter

Figure 2: Temporal distribution of tweets containing (i) links to Mastodon
instances and (ii) migration related keywords/ hashtags.

We collect a global list of Mastodon instances from instances.social, which
contains a comprehensive index of Mastodon instances. We compile a set of
15,886 unique instances. We then collect all tweets containing a link to these
Mastodon instances using Twitter’s Search
API.111https://api.twitter.com/2/tweets/search/all Additionally, we collect
all tweets containing the following list of keywords related to the migration
from Twitter: ‘mastodon’, ‘bye bye twitter’, ‘good bye twitter’; and hashtags
#Mastodon, #MastodonMigration, #ByeByeTwitter, #GoodByeTwitter,
#TwitterMigration, #MastodonSocial, #RIPTwitter. In total, we collect
2,090,940 tweets posted by 1,024,577 users between October 26, 2022 (_i.e._ a
day before Musk’s takeover) and November 21, 2022. Figure 2 shows the temporal
distribution of these tweets.

We next search for Mastodon usernames in these tweets and the accompanying
metadata of any account that posted a tweet (_i.e._ display name, location,
description, URLs, pinned tweet text). Mastodon usernames take the form
<EMAIL_ADDRESS>and https://example.com/@alice, where alice is a username
and example.com is an instance. To map a Mastodon handle to a Twitter account,
we do this search in a hierarchical fashion: We first look for Mastodon
usernames in user metadata (_e.g._ bio) and create a mapping between Twitter
account & Mastodon account if one is found. If the search is unsuccessful at
the first step, we then look for Mastodon usernames in the tweet text. To
ensure mapping accuracy, we only map a Twitter account to a Mastodon account
identified from a tweet text if both the Twitter and Mastodon usernames are
identical.

Using this methodology, we identify the Mastodon accounts of 136,009 Twitter
users, which are created across 2,879 unique Mastodon instances. We find that
72% of Twitter users that migrated created a Mastodon account with the same
username that they use on Twitter. 4% of the Twitter users who create a
Mastodon account, have a (legacy) verified status (_i.e._ authentic, notable,
and active) on Twitter, suggesting that even well-established users have been
migrating.

Figure 3: Weekly activity on Mastodon instances.

While we track and analyze the migration of a large number of Twitter users
(136,009), the takeover of Twitter by Musk, is likely to have pushed even more
users to migrate who the above methodology cannot identify. Indeed, on
November 12, 2022, Mastodon announced that over 1 million users had registered
since October 27, 2022 [16], significantly more than our methodology
identifies. To understand the wider activities on the Mastodon instances, we
cross-check the new registrations on the 2,879 instances by crawling their
weekly activity from Mastodon’s Weekly Activity
Endpoint.222https://docs.joinmastodon.org/methods/instance/#activity Figure 3
shows the weekly number of registrations, logins and statuses. We notice a
large increase in all three activity metrics after the Twitter acquisition. Of
course, we cannot confirm that all these users migrated directly from Twitter.
However, given the timeline of registrations, we believe that it is very
likely that a large share of these new users migrated from Twitter.

###  3.2 Twitter and Mastodon Timelines.

We next crawl both the Twitter and Mastodon timelines of the migrating users
identified in the previous section. We use Twitter’s Search API and Mastodon’s
Account’s Statuses
Endpoint.333https://docs.joinmastodon.org/methods/accounts/#statuses For each
user, we crawl all tweets/statuses from October 01, 2022 to November 30, 2022.
In total, we gather Twitter timelines for 94.88% of the users. The rest were
suspended (0.08%), deleted/deactivated (2.26%), or the tweets were protected
(2.78%). We crawl the timelines of 79.22% of Mastodon users: the rest have
either not posted a single status (9.20%) or their instances were down at the
time of crawl (11.58%). In total, we gather 16,163,600 tweets, and 5,746,052
Mastodon statues.

###  3.3 Followees

We also crawl the user’s followees for both Twitter and Mastodon accounts. We
use the Twitter Follows API444https://api.twitter.com/2/users/:id/following
and the Mastodon Account’s Following
Endpoint555https://docs.joinmastodon.org/methods/accounts/#following
respectively. Due to the rate limitations of the Twitter’s API we crawl a sub-
sample of 10% of the migrated users. For representativity, our sample relies
on the followees distribution takes 5% from above the median value and 5% from
below. In total, we gather followee data for 13,068 users. This covers
11,453,484 followee relationships.

###  3.4 Ethical Considerations

The datasets in this paper include both user and post information, and
therefore might have privacy implications. To overcome any data mishandling,
we have exclusively collected the publicly available data following well-
established ethical procedures for social data. We have obtained a waiver from
the ethics committee at the author’s institution.666anonymised for double-
blind submission We anonymize the data before use and store it in a secure
silo. Upon acceptance of the paper, anonymized data will be made available to
the public, which we hope will help further works.

##  4 RQ1: The Centralization Paradox

Figure 4: Top 30 Mastodon instances Twitter users migrated to. Figure 5:
Percentage of users on top 25% instances (w.r.t number of users).

Mastodon is a decentralized platform, in which users are (notionally) spread
across thousands of independent instances. We therefore first test if the
migration has resulted in true decentralization or if Mastodon experiences a
paradox, whereby the majority of users centralize upon a small number of
servers. Overall, the Twitter users in our dataset migrate to 2,879 unique
Mastodon instances. Figure 4 presents a histogram of the number of users who
have joined the top 30 Mastodon instances. The plot divide accounts into those
created before the acquisition, and those created after. Interestingly, not
all the Mastodon accounts advertised on Twitter in response to Elon Musk’s
acquisition are new though. 21% of the Mastodon accounts were created before
the Musk’s takeover. Despite Mastodon’s decentralization efforts, we observe a
clear trend towards centralization: a large number of users migrate to a small
set of instances. In particular, mastodon.social, a flagship Mastodon instance
operated by Mastodon gGmbH, receives the largest fraction of migrated Twitter
users.

We next explore the pressure towards Mastodon centralization by comparing the
percentage of migrated users with the percentage of instances they join.
Figure 5 plots the distribution of users across the top % of instances. We
find that nearly 96% of users join the top 25% of largest instances (w.r.t
number of users). This centralization trend implies that a small number of
instance owners, administrators and moderators have an impact on a large
fraction of migrated users. Arguably, this means that Mastodon is at risk of
becoming another (semi-)centralized service.

Figure 6: (a) Distribution of instances w.r.t to number of users. (b) CDF of
number of followers of users on different-sized instances. (c) CDF of number
of followees of users on different-sized instances. (d) CDF of number of
statuses of users on different-sized instances.

One possible explanation for this trend is that users intend to build a large
social network by joining large and well-known instances. We therefore examine
the relationship between the size of instances and social networks by
analyzing the number of followers and followees of users joining different-
sized instances. We analyze all migrated users who join Mastodon after the
Twitter acquisition and have 30 days old account (to ensure a fair
comparison). This covers 50.59% of all migrated users. We divide the instances
based on quantiles w.r.t number of users. Figure 6 presents the distribution
of instances by the number of users, CDFs of the number of followers,
followees, and statuses of users on different-sized instances. Contrary to our
hypothesis, users in the bigger instances tend to have smaller social
networks. 13.16% of instances have just one user, who tends to have more
followers, followees, and statuses than users in more populated instances.
Paradoxically, the single user instances, have 64.88% more followers, follows
99.04% more users, and posts 121.14% more statuses (on average) than the users
of the bigger instances. This implies that the size of an instance has a
limited impact on the size of a user’s social network. Rather it mainly
depends on the user’s activeness, engagement and networking. Hence, while
large instances have more users, small instances attract more active users.
Manual inspection suggests that this is because smaller instances have more
dedicated and proactive user bases (whereas larger ones accumulate more
experimental users).

##  5 RQ 2: Social Network Influence

There are at least two possible reasons for platform migration from Twitter to
Mastodon, particularly after the Musk takeover: (i) A user might have decided
to move for ideological reasons, if they disagree with Musk’s actions after he
gained control of Twitter; and (ii) A user might have decided to move because
a large fraction of the accounts they follow moved (and therefore Twitter has
become irrelevant as a platform for them). Of course, these two reasons are
not contradictory or mutually exclusive. In this section, we attempt to
distinguish between these reasons based on the observation that if a user
moves because their immediate social network moves, a large proportion of
their ego network neighbourhood would also have moved with them. We argue this
offers an interesting example of social contagion.

###  5.1 Twitter vs. Mastodon Social Network

Figure 7: CDF of number of followers and followees of migrated users on
Twitter and Mastodon.

We first analyze the size of the social network (_i.e._ number of followers &
followees) that the migrated users have on both Twitter and Mastodon. Figure 7
plots the CDF of the number of followers and followees of migrated users on
both platforms. The median followers and followees that migrated users have on
Twitter are 744 and 787, respectively. Just 152 users (0.11% of total
migrated) have no Twitter followers, and 465 (0.35% of total migrated) have no
Twitter followees. In contrast, on Mastodon, 6.01% of users have no followers,
and 3.6% do not follow anyone. The median followers and followees on Mastodon
were 38 and 48, respectively. Interestingly, 1.65% of migrated users gained a
median of 33 _more_ followers on Mastodon than their Twitter followers. This
confirms that these new users are yet to bootstrap a significant social
network on Mastodon. However, we emphasize that the median age of migrated
accounts on Twitter is 11.5 years, in contrast to just 35 days on Mastodon.
Hence, due to these disproportionate ages, the size of the social networks on
the two platforms are not directly comparable.

###  5.2 Social Network Driven Migration

Figure 8: CDFs of the fraction of Twitter followees of each migrated user that
(i) moved to Mastodon (blue) (ii) moved to Mastodon before the user (orange)
and (iii) moved to the same instances on Mastodon as the user (green).

We next conjecture that a user’s (Twitter) social network may have an impact
on their likelihood of migration. For example, if a user’s friends migrate to
Mastodon, this may encourage them to do the same.

To inspect this, we analyze the followees data from both Twitter and Mastodon
for 10% of the migrated users (see §3.3). Figure 8 shows CDFs of the fraction
of Twitter followees of each migrated user that (i) moved to Mastodon (blue);
(ii) moved to Mastodon before the user (orange); and (iii) moved to the same
Mastodon instances as the user (green). We notice that just 5.99% of each
user’s followees also migrate (on average). In fact, for 3.94% of the migrated
users, none of their Twitter followees move to Mastodon. Thus, the majority of
the social network of the migrated users seems indeed reluctant to migrate,
and sometimes they are the first in taking this step.

To better understand this, we compare the date on which each migrated user
joined Mastodon with that of their Twitter followees who migrated as well. We
find that, out of their social network (_i.e._ their followees), 4.98% of the
migrated users were the first and 4.58% were the last to migrate from Twitter
to Mastodon. On average, 45.76% of the followees of a user migrated to
Mastodon before the user actually did.

We are also curious to understand if users select the same Mastodon instance
as their social network. We therefore compare the instance of each migrated
user with that of its Twitter followees. On average, 14.72% of each migrated
user’s followees (that move to Mastodon) join the same instance. With 15K+
Mastodon instances, this is a considerable proportion, suggesting a clear
network effect. However, we also notice that this average is highly impacted
by one flagship instance: mastodon.social. This is the largest instance
available, and is probably the best known. Of all the migrated users whose
Twitter followees move to the same instance, 30.68% are on mastodon.social.
That said, we also find small instances that attract significant proportions
of a given user’s Twitter followers. For example, 4.5% of the migrated users
whose Twitter followees join them on the same instance are on
mastodon.gamedev.place (a Mastodon server focused on game development and
related topics)

Figure 9: Chord plot of switching within Mastodon instances. Figure 10: CDFs
of the fraction of Twitter followees of each switched user that (i) moved to
first instance (blue) (ii) moved to second instance (orange) and (iii) moved
to second instance before the user (green).

###  5.3 Instance Switching

A unique feature of Mastodon is that users can easily ‘switch’ instance. This
involves migrating their data from one instance to another. We are curious to
see if this is also driven by network effects. Overall, 4.09% of the users
have switched from the Mastodon instance they initially created an account on
(hereinafter first instance) to a new instance (hereinafter second instance).
Curiously, 97.22% of these switches happened after Musk’s Twitter takeover.
This suggests that users may join initial instances, but migrate to a more
suitable one once they are more experienced.

Figure 9 shows the chord plot of switches from each user’s first Mastodon
instance to their second. A common pattern across these switches is that users
move from general purpose/ flagship instances (_e.g._ mastodon.social,
mastodon.online) to more topic specific instances, _e.g._ sigmoid.social (a
Mastodon instance for people researching and working in Artificial
Intelligence) and historians.social (a Mastodon server for people interested
in history).

Interestingly, we notice a strong social network influence behind these
switches. Figure 10 shows the CDFs of the fraction of Twitter followees of
each switched user that (i) moved to the first instance (blue); (ii) moved to
the second instance (orange); and (iii) moved to second instance before the
user (green). On average, 46.98% of each user’s followees (who moved to
Mastodon) at some point also join the second instance. In contrast to just
11.4% who join the first instance. Interestingly, 77.42% of each switching
user’s followees (on average) joined the second instance before the user. This
suggests that the users switched from the first instance because a large
fraction of their Twitter followees moved to the second one.

##  6 RQ3: Timelines Analysis

We are next curious to understand how people use their (two) accounts after
migration.

###  6.1 Twitter vs. Mastodon Activity

Figure 11: Temporal distribution of tweets and statuses posted by migrated
users on Twitter and Mastodon respectively. Figure 12: Top 30 sources of
tweets. Note the log scale on the y-axis. Figure 13: Number of users that use
cross-posting tools daily.

We first analyze the timelines of migrated users from both Twitter and
Mastodon. Figure 11 shows the number of tweets on Twitter and the number of
statuses on Mastodon posted by migrated users each day from October 01, 2022
to November 30, 2022. We observe a continuous growth in user activity on
Mastodon after the acquisition of Twitter. However, the activity of migrated
users on Twitter do not decrease in parallel, _i.e._ our migrated users are
using both their Twitter and Mastodon accounts simultaneously.

Figure 14: CDFs of fraction of each migrated user’s Mastodon statuses that are
identical or similar to its tweets.

We next check if people are generating identical content across both platforms
or are, instead, projecting multiple ‘personas’. Figure 14 plots the CDFs of
the fraction of each migrated user’s Mastodon statuses that are identical or
similar to its tweets. We consider the Mastodon status similar to a tweet if
the cosine-similarity of their sentence embeddings [20] is greater than 0.7.
Surprisingly, just 1.53% of each migrated user’s Mastodon statuses are
identical. On average, just 16.57% of each user’s Mastodon statues are similar
to their tweets. Instead, 84.45% of the migrated users use the two platforms
to post completely different content. This suggests a mix of users, some of
whom create different personas on the two platforms, and a smaller subset who
mirror all their content.

A potential explanation for the latter is the use of cross-posting tools. Such
tools allow users to automatically mirror their Mastodon statues on Twitter,
and vice versa. To examine this, we compare the number of tweets posted via
different sources before and after Musk’s takeover in Figure 12. Naturally,
the majority are posted by official Twitter clients such as the Twitter Web
App. The two sources that increase most dramatically, however, are two well-
known cross-posters, Mastodon-Twitter Crossposter and Moa Bridge — by 1128.95%
and 1732.26%, respectively. Of all migrated users, 5.73% use one of the two
cross-posters at least once. This suggested such users see both Twitter and
Mastodon as viable platforms, and have limited intention of creating multiple
’personas’. Figure 13 also plots the number of users using cross-posters over
time. We see that their usage increases rapidly after Musk’s takeover. The
downward trend towards the end of November is likely a result of the posting
issues that cross-posters faced after their posting rate limit was revoked by
Twitter [21].

###  6.2 Hashtags

Figure 15: Top 30 hashtags along with their frequencies on Twitter and
Mastodon.

Given that 84.45% of the migrated users post completely different content on
the two platforms, we next inspect the hashtags used. This gives us a flavour
of the parallel discussions taking place on Mastodon and Twitter. Figure 15
presents the top 30 most frequent hashtags used over the two platforms by the
migrated users. We notice that users discuss more diverse topics on Twitter
such as Entertainment (#NowPlaying, #BBC6Music), Celebrities (#BarbaraHolzer),
and Politics (#StandWithUkraine, #GeneralElectionNow), whereas Mastodon seems
dominated by Fediverse related discussion (#fediverse) and the migration to it
(#TwitterMigration). We conjecture that we might see more diverse discussions
on Mastodon once the migrated users make themselves familiar with the
platform.

Figure 16: CDFs of fraction of each migrated user’s toxic posts on Twitter and
Mastodon.

###  6.3 Toxicity Analysis

Moderation on Mastodon has received significant attention in recent months [5,
4]. This is because the administrators of Mastodon instances do not
universally have the resources to moderate malicius content. To shed light on
this, we study the extent to which toxic content is shared by migrated users
on both platforms. To do this, we label all tweets and statuses using Google
Jigsaw’s Perspective API.777https://www.perspectiveapi.com For a given post,
Perspective returns a score between 0 and 1 for its toxicity (0 $=$ non-
toxic). Specifically, we use the API’s TOXICITY attribute that defines
toxicity as “a rude, disrespectful, or unreasonable comment that is likely to
make people leave a discussion”. In the literature, 0.5 is the most common
choice to threshold the perspective scores [5, 22, 17], however, higher values
such as 0.8 are also used [2]. Here, we use 0.5 as a threshold and consider a
post to be toxic if its toxicity score is greater than 0.5 (and vice versa).

Figure 16 shows the CDFs of the fraction of each migrated user’s toxic posts
on Twitter and Mastodon. Overall, just 5.49% of tweets are toxic. Mastodon is
substantially less toxic, with just 2.80%. On average, each user posts 4.02%
toxic tweets on Twitter vs. just 2.07% toxic statuses on Mastodon. Even though
the discourse is non-toxic over both platforms, we notice that 14.26% of
migrated users post at least one toxic post on both the platforms. While this
may not be problematic for Twitter which has its own moderation team, it might
present challenges for Mastodon, where volunteer administrators are
responsible for content moderation [11].

##  7 Related Work

Decentralised Social Networks. Many previous efforts have been made to build
decentralized online social platforms. In the earliest days, there were many
peer-to-peer online social networks, such as Safebook [7], PeerSoN [6],
LotusNet [3], and LifeSocial.KOM [10]. However, performance and security
limitations [18], limited their adoption and success. New decentralized social
networks, such as Mastodon, Pleroma, Pixelfed, and PeerTube, have since
emerged. In sum, these platforms are referred to as the _Fediverse_. These
social network applications use ActivityPub, a W3C protocol, to implement
server federation. Some recent work has looked into these new decentralized
social networks. For instance, a large-scale measurement study of Mastodon
[19], found centralization trends in Mastodon. Paradoxically, we found that
while centralization occurs in terms of how many users are attracted to an
instance, smaller instances attract more active users. Other works focus on
user behavior across instances [13, 12]. Our work also touches upon the need
for decentralised moderation. This has been investigated in prior work on
Pleroma (another Fediverse microblog. Hassan et al identify novel challenges
[11] and propose a strawman solution. Zia et al. [5] also propose model
sharing solution to help automate moderation. Our work confirms the presence
of toxic content in Mastodon, though the numbers identified do not show a
trend towards greater toxicity than Twitter.

Social Network Migration. There have been a number of measurement studies on
social network migration. For example, [8] measured the migration activity of
fandom, tracking migrating users and the reasons behind their migration. The
authors find that policy and value-based aspects are determinant in the
migration. Gerhart et al. [9] analyze user migration from traditional social
networks to anonymous social networks perspective. They identify that social
norms drive migration. Otala et al. [14] study the migration of Twitter users
to Parler. The results show that, although Parler is not widely used, it has a
significant impact on political polarization. Our work also studies the
migration of Twitter users. However, to the best of our knowledge, it is the
first to systematically measure and analyze the migration of users from
centralised Twitter to a decentralised platform.

##  8 Conclusion

In this paper, we have explored the migration of users from Twitter to
Mastodon, prompted by Elon Musk’s acquisition of Twitter. We have focused on
three RQs: (i) How are new users spread across Mastodon instances, and are
there any consequences for decentralization? (ii) How much (if at all) does a
user’s ego-centric Twitter network influence their migration to Mastodon?
(iii) What are usage patterns of migrated users across both platforms? To
answer RQ1, we have found that 2.26% of users completely left Twitter,
deleting their account. Despite Mastodon’s decentralized architecture, we
found that the 25% largest instance on Mastodon contains 96% of the users.
Paradoxically, while larger instances attract more users, smaller ones attract
more active users, reinforcing Mastodon’s decentralization. To answer RQ2, we
showed that the size of the Mastodon instance had limited effect on the size
of the user’s social network. We observed the impact of social network in
migration, with an average of 14.72% of Twitter followees per user migrating
to the exact same Mastodon instance as user. To answer RQ3, we found that
users tend to post _different_ content across platforms. On average, only
1.53% of Mastodon posts per user were identical to Twitter. In terms of
toxicity, most of the user’s content on both platforms was non-toxic. Mastodon
appears to be less toxic than Twitter though. Overall, just 5.49% of tweets
and 2.80% of statuses posted by migrated users on Twitter and Mastodon
respectively were toxic.

There are a number of lines of future works. We would like to further
investigate whether migrating users retain their Mastodon accounts or return
to Twitter, and whether new users are joining the migration wave. It will be
interesting to see what the future holds for these user-driven centralized
Mastodon instances. This study provides the first step in the migration of
Twitter to Mastodon. We hope that it will inspire further exploration of
follow-up work.

## References

  * [1] ActivityPub.  https://www.w3.org/TR/activitypub/, 2018.
  * [2] P. Agarwal, O. Hawkins, M. Amaxopoulou, N. Dempsey, N. Sastry, and E. Wood.  Hate speech in political discourse: A case study of uk mps on twitter.  In Proceedings of the 32nd ACM Conference on Hypertext and Social Media, pages 5–16, 2021.
  * [3] L. M. Aiello and G. Ruffo.  Lotusnet: Tunable privacy for distributed online social network services.  Computer Communications, 35(1):75–88, 2012.
  * [4] I. H. Anaobi, A. Raman, I. Castro, H. B. Zia, D. Ibosiola, and G. Tyson.  Will admins cope? decentralized moderation in the fediverse.  ACM Web Conference, 2023.
  * [5] H. Bin Zia, A. Raman, I. Castro, I. Hassan Anaobi, E. De Cristofaro, N. Sastry, and G. Tyson.  Toxicity in the decentralized web and the potential for model sharing.  Proceedings of the ACM on Measurement and Analysis of Computing Systems, 6(2):1–25, 2022.
  * [6] S. Buchegger, D. Schiöberg, L.-H. Vu, and A. Datta.  Peerson: P2p social networking: early experiences and insights.  In Proceedings of the Second ACM EuroSys Workshop on Social Network Systems, pages 46–52, 2009.
  * [7] L. A. Cutillo, R. Molva, and T. Strufe.  Safebook: A privacy-preserving online social network leveraging on real-life trust.  IEEE Communications Magazine, 47(12):94–101, 2009.
  * [8] C. Fiesler and B. Dym.  Moving across lands: Online platform migration in fandom communities.  Proceedings of the ACM on Human-Computer Interaction, 4(CSCW1):1–25, 2020.
  * [9] N. Gerhart and M. Koohikamali.  Social network migration and anonymity expectations: What anonymous social network apps offer.  Computers in Human Behavior, 95:101–113, 2019.
  * [10] K. Graffi, C. Gross, D. Stingl, D. Hartung, A. Kovacevic, and R. Steinmetz.  Lifesocial. kom: A secure and p2p-based solution for online social networks.  In 2011 IEEE Consumer Communications and Networking Conference (CCNC), pages 554–558. IEEE, 2011.
  * [11] A. I. Hassan, A. Raman, I. Castro, H. B. Zia, E. De Cristofaro, N. Sastry, and G. Tyson.  Exploring content moderation in the decentralised web: The pleroma case.  In Proceedings of the 17th International Conference on emerging Networking EXperiments and Technologies, pages 328–335, 2021.
  * [12] L. La Cava, S. Greco, and A. Tagarelli.  Information consumption and boundary spanning in decentralized online social networks: the case of mastodon users.  Online Social Networks and Media, 30:100220, 2022.
  * [13] L. La Cava, S. Greco, and A. Tagarelli.  Network analysis of the information consumption-production dichotomy in mastodon user behaviors.  In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 1378–1382, 2022.
  * [14] J. M. Otala, G. Kurtic, I. Grasso, Y. Liu, J. Matthews, and G. Madraki.  Political polarization and platform migration: a study of parler and twitter usage by united states of america congress members.  In Companion Proceedings of the Web Conference 2021, pages 224–231, 2021.
  * [15] Mastodon.  Mastodon.  hhttps://github.com/mastodon/mastodon, 2022.
  * [16] Mastodon.  Over 1 million people have joined mastodon since october 27.  https://twitter.com/joinmastodon/status/1591519312338210816, 2022\.  Accessed: 2023-01-04.
  * [17] A. Papasavva, S. Zannettou, E. De Cristofaro, G. Stringhini, and J. Blackburn.  Raiders of the lost kek: 3.5 years of augmented 4chan posts from the politically incorrect board.  In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 885–894, 2020.
  * [18] T. Paul, A. Famulari, and T. Strufe.  A survey on decentralized online social networks.  Computer Networks, 75:437–452, 2014.
  * [19] A. Raman, S. Joglekar, E. D. Cristofaro, N. Sastry, and G. Tyson.  Challenges in the decentralised web: The mastodon case.  In Proceedings of the Internet Measurement Conference, pages 217–229, 2019.
  * [20] N. Reimers and I. Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019\.
  * [21] Renato Lond Cerqueira.  Timeline for the shutdown of the mastodon-twitter crossposter instance at crossposter.masto.donte.com.br.  https://write.as/renatolond/timeline-for-the-shutdown-of-the-mastodon-twitter-crossposter-instance-at, 2022\.  Accessed: 2023-01-04.
  * [22] P. Röttger, B. Vidgen, D. Nguyen, Z. Waseem, H. Margetts, and J. Pierrehumbert.  Hatecheck: Functional tests for hate speech detection models.  arXiv preprint arXiv:2012.15606, 2020.


# Efficient View Synthesis and 3D-based Multi-Frame Denoising
with Multiplane Feature Representations

Thomas Tanay Aleš Leonardis Matteo Maggioni
Huawei Noah’s Ark Lab
<EMAIL_ADDRESS>
###### Abstract

While current multi-frame restoration methods combine information from
multiple input images using 2D alignment techniques, recent advances in novel
view synthesis are paving the way for a new paradigm relying on volumetric
scene representations. In this work, we introduce the first 3D-based multi-
frame denoising method that significantly outperforms its 2D-based
counterparts with lower computational requirements. Our method extends the
multiplane image (MPI) framework for novel view synthesis by introducing a
learnable encoder-renderer pair manipulating multiplane representations in
feature space. The encoder fuses information across views and operates in a
depth-wise manner while the renderer fuses information across depths and
operates in a view-wise manner. The two modules are trained end-to-end and
learn to separate depths in an unsupervised way, giving rise to Multiplane
Feature (MPF) representations. Experiments on the Spaces and Real Forward-
Facing datasets as well as on raw burst data validate our approach for view
synthesis, multi-frame denoising, and view synthesis under noisy conditions.

##  1 Introduction

Multi-frame denoising is a classical problem of computer vision where a noise
process affecting a set of images must be inverted. The main challenge is to
extract consistent information across images effectively and the current state
of the art relies on optical flow-based 2D alignment [45, 7, 3]. Novel view
synthesis, on the other hand, is a classical problem of computer graphics
where a scene is viewed from one or more camera positions and the task is to
predict novel views from target camera positions. This problem requires to
reason about the 3D structure of the scene and is typically solved using some
form of volumetric representation [32, 55, 28]. Although the two problems are
traditionally considered distinct, some novel view synthesis approaches have
recently been observed to handle noisy inputs well, and to have a denoising
effect in synthesized views by discarding inconsistent information across
input views [18, 26]. This observation opens the door to 3D-based multi-frame
denoising, by recasting the problem as a special case of novel view synthesis
where the input views are noisy and the target views are the clean input views
[26, 31].

Noisy inputs IBRNet [48] NAN [31] MPFER (ours)

Figure 1: _Top:_ Our Multiplane Features Encoder-Renderer (MPFER) reimagines
the MPI pipeline by moving the multiplane representation to feature space.
_Bottom:_ MPFER significantly outperforms existing methods in multiple
challenging scenarios, including here, novel view synthesis from 8 highly
degraded inputs.

Recently, novel view synthesis has been approached as an encoding-rendering
process where a scene representation is first _encoded_ from a set of input
images and an arbitrary number of novel views are then _rendered_ from this
scene representation. In the Neural Radiance Field (NeRF) framework for
instance, the scene representation is a radiance field function encoded by
training a neural network on the input views. Novel views are then rendered by
querying and integrating this radiance field function over light rays
originating from a target camera position [28, 2, 24]. In the Multiplane Image
(MPI) framework on the other hand, the scene representation is a stack of
semi-transparent colored layers arranged at various depths, encoded by feeding
the input views to a neural network trained on a large number of scenes. Novel
views are then rendered by warping and overcompositing the semi-transparent
layers [41, 55, 8].

In the present work, we adopt the MPI framework because it is much lighter
than the NeRF framework computationally. The encoding stage only requires one
inference pass on a network that generalizes to new scenes instead of training
one neural network per-scene, and the rendering stage is essentially free
instead of requiring a large number of inference passes. However, the standard
MPI pipeline struggles to predict multiplane representations that are self-
consistent across depths from multiple viewpoints. This problem can lead to
depth-discretization artifacts in synthesized views [40] and has previously
been addressed at the encoding stage using computationally expensive
mechanisms and a large number of depth planes [40, 27, 8, 11]. Here, we
propose to enforce cross-depth consistency at the rendering stage by replacing
the fixed overcompositing operator with a learnable renderer. This change of
approach has three important implications. First, the encoder module can now
process depths independently from each other and focus on fusing information
across views. This significantly reduces the computational load of the
encoding stage. Second, the scene representation changes from a static MPI to
Multiplane Features (MPF) rendered dynamically. This significantly increases
the expressive power of the scene encoding. Finally, the framework’s overall
performance is greatly improved, making it suitable for novel scenarios
including multi-frame denoising where it outperforms standard 2D-based
approaches at a fraction of their computational cost. Our main contributions
are as follow:

  * •

We solve the cross-depth consistency problem for multiplane representations at
the rendering stage, by introducing a learnable renderer.

  * •

We introduce the Multiplane Feature (MPF) representation, a generalization of
the multiplane image with higher representational power.

  * •

We re-purpose the multiplane image framework originally developed for novel
view synthesis to perform 3D-based multi-frame denoising.

  * •

We validate the approach with experiments on 3 tasks and 3 datasets and
significantly outperform existing 2D-based and 3D-based methods for multi-
frame denoising.

##  2 Related work

#### Multi-frame denoising

Multi-frame restoration methods are frequently divided into two categories,
depending on the type of image alignment employed. _Explicit alignment_ refers
to the direct warping of images using optical flows predicted by a motion
compensation module [4, 43, 44, 54]. In contrast, _implicit alignment_ refers
to local, data-driven deformations implemented using dynamic upsampling
filters [16, 17], deformable convolutions [45, 49], kernel prediction networks
[25] or their extension, basis prediction networks [53]. Explicit alignment is
better at dealing with large motion while implicit alignment is better at
dealing with residual motion, and state-of-the-art performance can be achieved
by combining both in the form of flow-guided deformable convolutions [6, 7].
Another distinction between multi-frame restoration methods is the type of
processing used. A common approach is to concatenate the input frames together
along the channel dimension [4, 43, 44, 54, 49] but recurrent processing is
more efficient [10, 34, 9, 42], especially when implemented in a bidirectional
way [14, 5, 7]. BasicVSR++ achieves state-of-the-art performance by combining
flow-guided deformable alignment with bidirectional recurrent processing
iterated multiple times [7]. In a different spirit, the recent DeepRep method
[3] introduces a deep reparameterization in feature space of the maximum a
posteriori formulation of multi-frame restoration. Similarly to the previous
methods however, it still uses a form of explicit 2D alignment, and lacks any
ability to reason about the 3D structure of the scene.

#### View synthesis

The idea to decompose a scene into a set of semi-transparent planes can be
traced back to the use of mattes and blue screens in special effects film-
making [47, 38]. This scene representation was first applied to view
interpolation in [41], and recently gained popularity under the name of
Multiplane Image (MPI) [55]. It is particularly powerful to generate novel
views from a small set of forward facing views [55, 27, 8], and can even be
used to generate novels views from a single image [46, 20, 11]. The rendering
of view dependent-effects and non-Lambertian surfaces is challenging due to
the use of a single set of RGB$\alpha$ images, and can be improved by
predicting multiple MPIs combined as a weighted average of the distance from
the input views [27], or as a set of basis components [51]. The simplicity of
this representation is appealing, but it can still be computationally heavy
when the number of depth planes grows [8, 51], and the rendered views can
suffer from depth discretization artifacts [40]. A number of alternative
layered scene representations exist, including the Layered Depth Image (LDI)
consisting in one RGB image with an extra depth channel [36], and variants of
MPIs and LDIs [21, 13, 19, 39]. So far however, all these methods use a fixed
overcompositing operator at the rendering stage. The idea to perform view
synthesis by applying 3D operations in the feature space of an encoder-decoder
architecture was explored on simple geometries in [52], and used with success
on a point-cloud scene representation for view synthesis from a single image
in [50]. Recently, Neural Radiance Fields (NeRFs) have become highly popular
for their ability to produce high quality renderings of complex scenes from
arbitrary viewpoints [28, 2, 24, 29, 30]. However, they tend to be very heavy
computationally, require a large number of input views, and lack the ability
to generalize to novel scenes. IBRNet [48] improves generalizability by
learning a generic view interpolation function, but the approach remains
computationally heavy. The application of view synthesis approaches to multi-
frame restoration has been limited so far, and exclusively based on NeRF.
RawNeRF [26] explores novel view synthesis in low-light conditions, and
reports strong denoising effects in the synthesized views. Deblur-NeRF [22]
augments the ability of NeRFs to deal with blur by including Deformable Sparse
Kernels. Noise-aware-NeRFs [31] improves the IBRNet architecture to explicitly
deal with noise. However, these different restoration approaches still suffer
from the limitations affecting their underlying NeRF representations.

##  3 Method

We start by describing the standard MPI processing pipeline, before discussing
the cross-depth consistency problem. We then introduce our MPF Encoder-
Renderer and its adaptation to multi-frame denoising.

###  3.1 Standard MPI processing pipeline

The standard MPI processing pipeline turns a set of input views into an
arbitrary set of rendered novel views by applying 4 main transformations:
forward-warping, MPI prediction, backward-warping, overcompositing (See Figure
2(a)). We describe this pipeline in more details below.

#### Input views

The inputs of the pipeline are a set of $V$ views of a scene, consisting of
images and camera parameters. The images are of height $H$ and width $W$, with
red-green-blue color channels, and can be stacked into a 4D tensor
$\bm{I}=\\{\\{\\{\\{\bm{I}_{vchw}\\}_{w=1}^{W}\\}_{h=1}^{H}\\}_{c=1}^{3}\\}_{v=1}^{V}$.
To simplify notations, we omit the dimensions $c,h,w$ and refer to an
individual image as $\bm{I}_{v}$. The camera parameters consist of an
intrinsic tensor $\bm{K}$ of size $V\\!\times\\!3\\!\times\\!3$ containing
information about the focal lengths and principal point of the cameras, and an
extrinsic tensor containing information about the camera orientations in the
world coordinate system, that can be split into a rotation tensor $\bm{R}$ of
size $V\\!\times\\!3\\!\times\\!3$ and a translation tensor $\bm{t}$ of size
$V\\!\times\\!3\\!\times\\!1$. A reference view $i$ is defined a priori, and
the positions of all the cameras are assumed to be expressed relatively to it.
The intrinsic matrix $\bm{K}_{i}$ of the reference camera is defined such that
the corresponding field of view covers the region of space visible to all the
input views. Finally, a set of $D$ depth planes is distributed orthogonally to
the reference viewing direction such that their normal is
$\bm{n}=(0,0,1)^{\top}$, and their distances $\\{a_{d}\\}_{d=1}^{D}$ from the
reference camera center are sampled uniformly in disparity. The camera
parameters and the depth planes are used to define a set of $D\\!\times\\!V$
homography projections, represented by a tensor $\bm{H}$ of size
$D\\!\times\\!V\\!\times\\!3\\!\times\\!3$. Each homography is between one of
the input views and the reference view, and is induced by one of the depth
planes, such that its matrix is expressed as [12]:

$\bm{H}_{dv}=\bm{K}_{v}\left(\bm{R}_{v}-\frac{\bm{t}_{v}\,\bm{n}^{\top}}{a_{d}}\right)\bm{K}_{i}^{-1}$
(1)

#### Plane Sweep Volumes

The first transformation in the MPI pipeline is the computation of Plane Sweep
Volumes (PSVs), obtained by forward-warping each input image $D$ times,
according to the homography $\bm{H}_{dv}$. The sampling rate of the warping
operator $\mathcal{W}$ is a hyperparameter that can be controlled using an up-
scaling factor $s$. Each transformation can thus be written as
$\bm{X}_{dv}=\mathcal{W}(\bm{I}_{v},\,\bm{H}_{dv},\,s)$ and the PSV tensor
$\bm{X}$ is of size $D\\!\times\\!V\\!\times\\!3\\!\times\\!sH\\!\times\\!sW$.

#### Multiplane Image

The main processing block of the pipeline is a neural network MPINet, turning
the set of PSVs into a multiplane image representation of the scene
$\bm{Y}=\text{MPINet}(\bm{X})$ where $\bm{Y}$ is a set of $D$ semi-transparent
RGB images of size $D\\!\times\\!4\\!\times\\!sH\\!\times\\!sW$, constrained
to the [0,1] range by using a sigmoid activation function.

#### Projected MPIs

The MPI is then backward-warped to a set of $R$ novel views, defined by an
homography tensor $\bm{G}$ of size $R\\!\times\\!D\\!\times\\!3\\!\times\\!3$
following Eq. 1 with the depth and view dimensions transposed. The backward-
warping operation is defined as
$\bm{Z}_{rd}=\mathcal{W}\left(\bm{Y}_{d},\,\bm{G}_{rd}^{-1},\,1/s\right)$
obtaining a tensor of projected MPIs $\bm{Z}$ of size
$R\\!\times\\!D\\!\times\\!4\\!\times\\!H\\!\times\\!W$.

#### Rendered views

The projected MPIs are finally collapsed into single RGB images by applying
the overcompositing operator $\mathcal{O}$. This operator splits each
RGB$\alpha$ image $\bm{Z}_{rd}$ into a colour component $\bm{C}_{rd}$ and an
alpha component $\bm{A}_{rd}$ and computes
$\bm{\tilde{J}}_{r}=\sum_{d=1}^{D}\left(\bm{C}_{rd}\bm{A}_{rd}\prod_{k=d+1}^{D}\left(1-\bm{A}_{rk}\right)\right)$
obtaining the rendered views $\bm{\tilde{J}}$ of size
$R\\!\times\\!3\\!\times\\!H\\!\times\\!W$.

#### Training

The pipeline is typically trained end-to-end in a supervised way by minimizing
a loss $\mathcal{L}(\bm{\tilde{J}},\bm{J})$ between the rendered views
$\bm{\tilde{J}}$ and the corresponding ground-truth images $\bm{J}$. In
practice, $\mathcal{L}$ is often an $\mathcal{L}_{1}$ loss applied to low
level features of a VGG network [37].

(a) Standard MPI processing pipeline.

(b) Our MPF Encoder-Renderer.

Figure 2: In the standard MPI processing pipeline, all the learning and most
of the processing happens in the MPINet module. We propose to move the
multiplane representation to feature space, by giving some processing power to
the warping operators, and replacing the overcompositing operator with a
learnable renderer.

###  3.2 Cross-depth consistency

The main and only learnable module of the standard MPI pipeline is the
prediction network MPINet, transforming a 5D PSV tensor into a 4D MPI scene
representation. Its task is challenging because multiplane images are hyper-
constrained representations: their semi-transparent layers interact with each
other in non-trivial and view-dependent ways, and missing or redundant
information across layers can result in depth discretization artifacts after
applying the overcompositing operator [40]. To perform well, MPINet requires a
mechanism enforcing _cross-depth consistency_ ; and several approaches have
been considered before.

In the original case of stereo-magnification [55], there is one reference
input and one secondary input which is turned into a single PSV. Both are
concatenated along the channel dimension and fed to an MPINet module
predicting the full MPI in one shot as a 3D tensor of size
$(D\\!\times\\!4)\\!\times\\!sH\\!\times\\!sW$, such that cross-depth
consistency is enforced within the convolutional layers of MPINet. A similar
solution can be used in the case of single-view view synthesis [46], where
there is a single reference input image and no PSV. In the general case with
$V$ inputs, however, the PSV tensor $\bm{X}$ becomes very large and there are
two main ways to process it.

Option 1

The first solution is to generalize the approach of [55] and concatenate
$\bm{X}$ across views and depths before feeding it to a network predicting the
full MPI in one shot:
$\bm{Y}=\text{MPINet}\\!\left(\left\\{\\{\bm{X}_{dv}\\}_{v=1}^{V}\right\\}_{d=1}^{D}\right)$.

Option 2

The second solution is to concatenate $\bm{X}$ across views, and process each
depth separately—effectively running the MPINet block $D$ times in parallel:
$\bm{Y}=\left\\{\text{MPINet}\\!\left(\\{\bm{X}_{dv}\\}_{v=1}^{V}\right)\right\\}_{d=1}^{D}$.

Option 1 tends to work poorly in practice [8], as it requires to either use
very large convolutional layers with intractable memory requirements, or
discard most of the information contained in the input PSVs after the first
convolutional layer. Option 2 is appealing as it fuses information across
views more effectively, but the resulting MPI typically suffers from a lack of
cross-depth consistency as each depth is processed separately. Most previous
works adopt Option 2 as a starting point, and augment it with various
mechanisms allowing some information to flow across depths. For instance, some
methods implement MPINet with 3D convolutions [40, 27], such that each depth
is treated semi-independently within a local depth neighborhood dependent on
the size of the kernel, which is typically 3. By design however, this solution
cannot handle interactions between distant depth planes. DeepView [8] proposes
to solve the recursive cross-depth constraint by iteratively refining the
prediction of the MPINet block, effectively performing a form of learned
gradient descent. However, this solution requires to run the MPI network
multiple times, which is computationally heavy. Finally, the method of [11]
uses a feature masking strategy to deal with inter-plane interactions
explicitly. This solution is both complex (multiple networks are required to
predict the masks) and rigid (the masking operations are fixed and still work
on a per-depth basis).

Figure 3: Our Multiplane Features Encoder-Renderer (MPFER). Input views are
forward-warped into plane sweep volumes (PSVs) which are processed depthwise
by the Encoder Unet64. The resulting multiplane feature representation (MPF)
can then be back-projected to an arbitrary number of novel views, or to the
same views as the inputs—allowing the integration of a skip connection
(illustrated here). The Renderer Unet64 processes the projected MPFs on a per-
view basis, producing the final synthesised or denoised outputs.

###  3.3 Our MPF Encoder-Renderer

Here, we propose to solve the cross-depth consistency problem in a novel way,
by addressing it at the rendering stage. Specifically, we replace the fixed
overcompositing operator with a learnable renderer enforcing consistency
directly at the output level on a per-view basis. This design change greatly
simplifies the task of the multiplane representation encoder, which can now
focus on fusing information across views in a depth-independent way. It also
promotes the multiplane representation to feature space, by relaxing existing
constraints on the number of channels and scaling in the [0,1] range. The four
transformations of the pipeline are modified as follows (See Figure 2(b)).

#### Plane Sweep Volumes

To decrease the amount of information loss through image warping and promote
the PSVs to feature space, we now apply a convolution to the images before
warping them:
$\bm{X}_{dv}=\mathcal{W}(\text{Conv}(\bm{I}_{v}),\,\bm{H}_{dv},\,s)$. The PSV
tensor $\bm{X}$ is now of size
$D\\!\times\\!V\\!\times\\!C_{1}\\!\times\\!sH\\!\times\\!sW$ where the number
of channels $C_{1}$ is a hyperparameter.

#### Multiplane Features

We then replace the MPINet module with an encoder applied to each depth
independently:
$\bm{Y}=\left\\{\text{Encoder}\left(\\{\bm{X}_{dv}\\}_{v=1}^{V}\right)\right\\}_{d=1}^{D}$.
The multiplane representation $\bm{Y}$ is now in feature space: its range is
not constrained by a sigmoid function anymore and its size is
$D\\!\times\\!C_{2}\\!\times\\!sH\\!\times\\!sW$ where the number of channels
$C_{2}$ is a second hyperparameter.

#### Projected MPFs

The MPF is still backward-warped to a set of $R$ novel views defined by an
homography tensor $\bm{G}$, but this is now followed by a convolution
collapsing the depth and channel dimensions into a single dimension:
$\bm{Z}_{r}=\text{Conv}\\!\left(\left\\{\mathcal{W}\left(\bm{Y}_{d},\,\bm{G}_{dr}^{-1},\,1/s\right)\right\\}_{d=1}^{D}\right)$.
The tensor $\bm{Z}$ is of size $R\\!\times\\!C_{3}\\!\times\\!H\\!\times\\!W$
where $C_{3}$ is a third hyperparameter.

#### Rendered views

Finally, the projected MPFs are turned into the final rendered views by a
simple CNN renderer operating on each view separately:
$\bm{\tilde{J}}=\left\\{\text{Renderer}(\bm{Z}_{r})\right\\}_{r=1}^{R}$. The
rendered views $\bm{\tilde{J}}$ are of size
$R\\!\times\\!3\\!\times\\!H\\!\times\\!W$.

#### Training

The pipeline is still trained end-to-end by minimizing a loss
$\mathcal{L}(\bm{\tilde{J}},\bm{J})$ between the rendered views
$\bm{\tilde{J}}$ and the corresponding ground-truth images $\bm{J}$, but the
renderer and the warping operators are now learnable. When the input views
$\bm{I}$ are noisy versions of ground-truth images $\bm{I}^{\ast}$, the
pipeline can be turned into a multi-frame denoising method by using the input
homographies $\bm{H}$ during backward-warping, obtaining denoised outputs
$\bm{\tilde{I}}$, and by minimizing the loss
$\mathcal{L}(\bm{\tilde{I}},\bm{I}^{\ast})$. There is then a one-to-one
mapping between the input views and the rendered views, and it is possible to
integrate a skip connection feeding the noisy inputs directly to the renderer
to guide its final predictions. In all our experiments, we use Unets [33] with
a base of 64 channels to implement both the encoder and the renderer. We set
$C_{1}=C$, $C_{2}=V\times C$ and $C_{3}=64$ such that there is a single
hyperparameter $C$ to vary. Our Multiplane Features Encoder-Render (MPFER) is
illustrated in Figure 3 for 3 inputs views and 3 depth planes. A MindSpore
[15] implementation of our method is available111https://github.com/mindspore-
lab/mindediting.

Noisy input BPN [53] BasicVSR++ [7] DeepRep [3] MPFER-64 (ours) Ground truth
Noisy input IBRNet-N [31] NAN [31] MPFER-N (ours) MPFER-C (ours) Ground truth

Figure 4: Qualitative evaluation for multi-frame denoising with Gain 20 (best
viewed zoomed in). We compare MPFER to 2D-based methods on Spaces (top) and to
3D-based methods on the Real Forward-Facing dataset (bottom).

##  4 Experiments

We first consider the Spaces dataset [8] and validate our approach on novel
view synthesis. We then focus on a denoising setup and perform extensive
comparisons to state-of-the-art 2D-based methods. Finally, we compare our
approach to the 3D-based multi-frame denoising method of [31] by replicating
their experimental setup on the Real Forward-Facing dataset. In all cases, our
method outperforms competitors at a fraction of the computational cost.

###  4.1 Spaces

The Spaces dataset [8] consists of 100 indoor and outdoor scenes, captured 5
to 10 times each using a 16-camera rig placed at slightly different locations.
90 scenes are used for training and 10 scenes are held-out for evaluation. The
resolution of the images is 480$\times$800.

#### Novel view synthesis

We start by replicating the novel view synthesis setup of DeepView [8] with
four scenarios: one with 12 input views and three with 4 input views.
Similarly to DeepView, we use a VGG loss and train our models for 100k steps
using the Adam optimizer, with a learning rate of 1.5e-3. We reduce the
learning rate to 1.5e-4 after 80k steps, and use a batch size of 4. Memory
usage was reported to be a major challenge in DeepView, and all our models are
kept at a significantly smaller size to avoid this issue. While DeepView uses
a sophisticated strategy to only generate enough of the MPI to render a
32$\times$32 crop in the target image, we use a large patch size of 192 and
only apply the loss on the region of the patch that contains more than 80% of
the depths planes after backward warping. We compute all metrics by averaging
over the validation scenes and target views of each setup, and after cropping
a boundary of 16 pixels on all images as done in [8]. We compare to DeepView
[8] and Soft3D [32] by using the synthesised images provided with the Spaces
dataset. We also consider three variants of the standard MPI pipeline using
the same Unet backbone as our MPFER method and trained in the same conditions,
but processing the input PSV in different ways. MPINet implements _Option 1_
from Sec. 3.2. The views and depths dimensions of the PSV tensor are stacked
along the channel dimension and fed to the Unet backbone to predict the output
MPI in one shot. MPINet-dw implements _Option 2_ from Sec. 3.2. The Unet
backbone runs depthwise on slices of the PSV to predict each depth plane of
the MPI separately, without communication mechanism across depths. Finally,
MPINet-dw-it implements a one-step version of the learned gradient descent
algorithm of DeepView. A first estimate of the MPI is predicted by a Unet
backbone running depthwise, and this estimate is fed to a second Unet backbone
also running depthwise, along with the input PSV and gradient components (R),
which are PSV-projected current estimates of the target views.

For our MPFER method and the MPINet ablations, we use a number of depth planes
$D=64$ distributed between 100 and 0.5 meters away from the reference camera,
placed at the average position of the input views. We use a number of channels
$C=8$ and a PSV/MPF upscaling factor $s=1.5$. Since the Unet backbone is not
agnostic to the number of input views, we train one version of each model for
the setup with 12 input views and one version for the three setups with 4
input views. The results are presented in Table 1. We observe a clear
progression between the performances of MPINet, MPINet-dw and MPINet-dw-it,
illustrating the benefit of each design change. Our MPFER method outperforms
MPINet-dw-it by up to 4dBs in PSNR at a similar computational complexity, and
outperforms DeepView by up to 1.8dB at a fraction of the complexity, clearly
motivating the use of a learnt renderer for efficient depth fusion.

#### Multi-frame denoising

We now consider a different setup where the inputs are 16 views from one rig
position with images degraded with noise, and the targets are the same 16
views denoised. Similarly to previous works [25, 53, 3, 31], we apply
synthetic noise with a signal dependent Gaussian distribution
$\bm{I}_{vchw}\sim\mathcal{N}\left(\bm{I}_{vchw}^{\ast}\;,\;\sigma_{r}^{2}+\sigma_{s}\bm{I}_{vchw}^{\ast}\right)$
where $\bm{I}$ is the tensor of noisy inputs, $\bm{I}^{\ast}$ is the ground
truth signal, and $\sigma_{r}$ and $\sigma_{s}$ are noise parameters that are
fixed for each sequence. We focus in particular on challenging scenarios with
moderate to high gain levels [4, 8, 16, 20], corresponding to the
$\left(\log(\sigma_{r}),\log(\sigma_{s})\right)$ values [(-1.44, -1.84),
(-1.08, -1.48), (-0.72, -1.12), (-0.6, -1.0)] respectively.

We consider two patch-based approaches: VBM4D [23] and VNLB [1], as well as
four state-of-the-art learning-based methods: BPN [53], BasicVSR [5] and its
extension BasicVSR++ [7], and DeepRep [3]. To evaluate the influence of the
model size and in particular the number of depth planes, we train three MPFER
models: MPFER-16 with $(D,C,s)\\!=\\!(16,8,1.)$, MPFER-32 with
$(D,C,s)\\!=\\!(32,16,1.25)$, and MPFER-64 with $(D,C,s)\\!=\\!(64,8,1.25)$.
MPFER-16 has the particularity of using the same number of depth planes as
there are input images, meaning that the number of Unet passes per frame to
denoise the sequence is $(D+V)/V=2$. This observation motivates us to perform
a comparison with three other architectures, using a strict computational
budget of 2 Unet passes per frame. Unet-SF (for Single-Frame) is constituted
of two Unet blocks without temporal connection, therefore processing the
sequence as a disjoint set of single frames. Unet-BR (for Bidirectional-
Recurrent) is constituted of two Unet blocks with bidirectional recurrent
connections: the lower Unet processes the sequence in a backward way, and the
higher Unet processes the sequence in a feedforward way. Finally, Unet-BR-OF
(for Bidirectional-Recurrent with Optical-Flow alignment) is constituted of
two Unet blocks with bidirectional recurrent connections, and the recurrent
hidden-state is aligned using a SpyNet module, as done in basicVSR [5]. We
train all the models in the same conditions as for the novel view synthesis
setup, except for the patch size which we increase to 256, and the loss which
we replace with a simple L1 loss. During training, we vary the gain level
randomly and concatenate an estimate of the standard deviation of the noise to
the input, as in [25, 53, 31]. We evaluate on the first rig position of the 10
validation scenes of the Spaces dataset for the 4 gain levels without
boundary-cropping, and present the results in Table 2. Each model receives 16
noisy images as input and produces 16 restored images as output, except for
BPN and DeepRep which are burst processing methods and only produce one
output. For these methods, we choose the view number 6 at the center of the
camera rig as the target output, and compare the performances of all methods
on this frame. Our MPFER method clearly outperforms all the other methods at a
fraction of the computational cost. It performs particularly strongly at high
noise levels, with improvements over other methods of more that 2dBs in PSNR.
MPFER-16 also performs remarkably well, despite using only 16 depth planes.
This suggests that the high representational power of multiplane features
allows to significantly reduce the number of depth planes—and therefore the
computational cost—compared to standard MPI approaches, which typically use a
very high number of planes (80 in the case of DeepView [8], up to 192 in the
case of NeX [51]). A qualitative evaluation is available in Figure 9 (top),
and we observe that MPFER is able to reconstruct scenes with much better
details. We also present a visualization of multiplane features in Figure 6,
illustrating how the model learns to separate depths in an unsupervised way.

###  4.2 LLFF-N

The LLFF-N dataset [31] is a variant of the Real Forward -Facing dataset [27]
where images are linearized by applying inverse gamma correction and random
inverse white balancing, and synthetic noise is applied following the same
signal dependent Gaussian distribution as used in the previous section with
the six gain levels $[1,2,4,8,16,20]$. The dataset contains 35 scenes for
training and 8 scenes for testing, and the resolution of the images is
756$\times$1008.

#### Denoising

In this setup, the model receives 8 frames in input: the target frame plus its
7 nearest neighbors in terms of camera distances. We train one MPFER model
with $(D,C,s)\\!=\\!(64,8,1.25)$, using an $\mathcal{L}_{1}$ loss applied to
the target frame. We evaluate on the 43 bursts used in [31] (every 8th frame
of the test set) and present the results in the first half of Table 3. A
qualitative evaluation is also available in Figure 9 (bottom). To assess the
robustness of our method to noisy camera positions, we evaluate it using
camera positions computed on clean images (MPFER-C) and on noisy images
(MPFER-N) using COLMAP [35]. Our method outperforms IBRNet-N and NAN in both
scenarios by large margins, but the evaluation using clean camera poses
performs significantly better at high noise levels.

#### Synthesis under noisy conditions

In this setup, the model receives as input the 8 nearest neighbors to a held-
out target. Again, we train one MPFER model with $(D,C,s)\\!=\\!(64,8,1.25)$,
using an $\mathcal{L}_{1}$ loss applied to the target frame. We evaluate on
the same 43 bursts as before and report the results in the second half of
Table 3. Our method performs on par with IBRNet and NAN at very low noise
levels (close to a pure synthesis problem), and significantly outperforms the
other methods at larger noise levels. MPFER only requires $D$ Unet passes to
produce an MPF, and 1 Unet pass to render a novel view, which is significantly
lighter than IBRNet and NAN. At inference time, the Unet pass requires 0.6
Mflops per pixel, compared to 45 Mflops for IBRNet [48]. A qualitative
evaluation is available in Figure 1 for Gain 20.

#### Low-Light Scenes

Finally, we qualitatively evaluate our denoising model trained on LLFF-N on
sequences with real noise captured with a Google Pixel 4 under low-light
conditions. We use the sequences from [31] and estimate camera poses using
COLMAP [35] on noisy images. We compare our results to those of
[31]—downloaded from their project page where more baselines can be found—in
Figure 5.

Noisy input NAN [31] MPFER (Ours)

Figure 5: Qualitative evaluation on sequences with real noise from [31],
captured with a Google Pixel 4.

##  5 Conclusion

We proposed to approach multi-frame denoising as a view synthesis problem and
argued in favor of using multiplane representations for their low
computational cost and generalizability. We introduced a powerful
generalization of multiplane images to feature space, and demonstrated its
effectiveness in multiple challenging scenarios.

Noisy inputPlane 1Plane 3Plane 5Plane 7Plane 9Plane 11Plane 13Plane 15

Figure 6: Visualization of Multiplane Features for 3 scenes. We plot the first
3 channels of planes 1, 3, 5, 7, 9, 11, 13, 15 from MPFER-16. Our method
learns to separate depths in an unsupervised way, in this case from a pure
denoising problem.

12 input views (dense) 4 input views (small) 4 input views (medium) 4 input
views (large) GFlops@ PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$
PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$
LPIPS$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$
500$\times$800 Soft3D* 31.93 0.940 0.052 30.29 0.925 0.064 30.84 0.930 0.060
30.57 0.931 0.054 n/a DeepView* 34.23 0.965 0.015 31.42 0.954 0.026 32.38
0.957 0.021 31.00 0.952 0.024 45800 MPINet 27.43 0.914 0.035 27.00 0.906 0.054
26.16 0.896 0.062 24.93 0.865 0.085 450 MPINet-dw 30.70 0.963 0.021 29.39
0.951 0.027 28.47 0.948 0.030 26.83 0.937 0.040 7890 MPINet-dw-it 30.85 0.966
0.017 30.22 0.955 0.024 29.37 0.953 0.026 28.00 0.943 0.034 14800 MPFER-64
35.73 0.972 0.012 33.20 0.959 0.018 33.47 0.959 0.018 32.38 0.953 0.021 8490

Table 1: Novel view synthesis on Spaces. All metrics were computed on
predicted images with a 16-pixel boundary cropped, as done in [8]. Stared
methods were evaluated using the predicted images released with the Spaces
dataset.

Gain 4 Gain 8 Gain 16 Gain 20 GFlops@ PSNR$\uparrow$ SSIM$\uparrow$
LPIPS$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$
PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$
LPIPS$\downarrow$ 500$\times$800 VBM4D 32.00 0.900 0.108 29.94 0.850 0.172
27.48 0.769 0.280 26.55 0.730 0.331 n/a VNLB 33.41 0.918 0.089 30.30 0.871
0.144 25.74 0.793 0.283 23.51 0.743 0.366 n/a BPN 34.52 0.934 0.048 32.10
0.900 0.082 29.45 0.846 0.144 28.56 0.824 0.168 810 BasicVSR 36.87 0.959 0.027
34.52 0.937 0.049 31.73 0.898 0.095 30.68 0.879 0.119 2090 BasicVSR++ 36.98
0.959 0.026 34.66 0.938 0.045 31.97 0.902 0.083 30.92 0.883 0.102 4300 DeepRep
37.37 0.963 0.024 35.13 0.943 0.043 32.37 0.906 0.085 31.32 0.888 0.107 3230
UNet-SF 35.10 0.942 0.043 32.62 0.909 0.075 29.81 0.857 0.134 28.81 0.834
0.161 440 UNet-BR 35.19 0.943 0.040 32.67 0.912 0.070 29.90 0.861 0.124 28.94
0.840 0.148 470 UNet-BR-OF 36.41 0.956 0.029 34.27 0.936 0.048 31.77 0.899
0.089 30.85 0.882 0.110 710 MPFER-16 37.56 0.968 0.020 35.80 0.955 0.030 33.70
0.933 0.051 32.89 0.921 0.063 470 MPFER-32 37.94 0.970 0.019 36.17 0.958 0.028
33.99 0.936 0.047 33.14 0.924 0.058 1210 MPFER-64 38.00 0.970 0.018 36.25
0.959 0.027 34.08 0.936 0.047 33.23 0.925 0.057 1810

Table 2: Denoising on Spaces. All metrics were computed on frame 6, as BPN and
DeepRep are burst-denoising methods producing only one output. For the other
methods, average performances over the entire sequence are provided in
Supplementary Material.

Gain 1 Gain 2 Gain 4 Gain 8 Gain 16 Gain 20 PSNR$\uparrow$ SSIM$\uparrow$
LPIPS$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$
PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$
LPIPS$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$
PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$ _Denoising of Synthetic Noise_
IBRNet-N* 33.50 0.915 0.039 31.29 0.877 0.070 29.01 0.822 0.123 26.57 0.741
0.210 24.19 0.634 0.331 23.40 0.591 0.380 NAN* 35.84 0.955 0.018 33.67 0.930
0.034 31.26 0.892 0.068 28.64 0.834 0.132 25.95 0.749 0.231 25.07 0.715 0.271
MPFER-N 37.90 0.969 0.013 35.61 0.951 0.025 33.02 0.921 0.048 30.21 0.872
0.091 27.24 0.797 0.164 26.23 0.765 0.198 MPFER-C 38.06 0.971 0.011 35.95
0.956 0.020 33.65 0.934 0.036 31.21 0.898 0.065 28.61 0.843 0.115 27.71 0.819
0.138 _Novel View Synthesis Under Noisy Conditions_ IBRNet* 24.53 0.774 0.135
24.20 0.730 0.159 23.44 0.653 0.217 22.02 0.536 0.327 19.76 0.377 0.492 18.80
0.319 0.553 IBRNet-N* 23.86 0.763 0.170 23.73 0.744 0.178 23.38 0.703 0.208
22.68 0.638 0.275 21.67 0.549 0.377 21.29 0.514 0.418 NAN* 24.52 0.799 0.132
24.41 0.787 0.145 24.18 0.765 0.171 23.70 0.726 0.221 22.79 0.666 0.305 22.37
0.641 0.342 MPFER 24.52 0.798 0.157 24.51 0.796 0.158 24.47 0.789 0.164 24.33
0.775 0.178 23.94 0.746 0.212 23.72 0.731 0.230

Table 3: LLFF-N. We consider the two scenarios introduced in [31]: Denoising
of Synthetic Noise, where the noisy target is accessible, and Novel View
Synthesis Under Noisy Conditions, where the noisy target is held-out. The
numbers for the stared methods correspond to Figure 9 in [31], and were
provided by the authors.

## References

  * [1] Pablo Arias and Jean-Michel Morel.  Video denoising via empirical bayesian estimation of space-time patches.  Journal of Mathematical Imaging and Vision, 60(1):70–93, 2018.
  * [2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In ICCV, pages 5855–5864, 2021.
  * [3] Goutam Bhat, Martin Danelljan, Fisher Yu, Luc Van Gool, and Radu Timofte.  Deep reparametrization of multi-frame super-resolution and denoising.  In ICCV, pages 2460–2470, 2021.
  * [4] Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi.  Real-time video super-resolution with spatio-temporal networks and motion compensation.  In CVPR, pages 4778–4787, 2017.
  * [5] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Basicvsr: The search for essential components in video super-resolution and beyond.  In CVPR, pages 4947–4956, 2021.
  * [6] Kelvin CK Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Understanding deformable alignment in video super-resolution.  In AAAI, pages 973–981, 2021.
  * [7] Kelvin CK Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy.  Basicvsr++: Improving video super-resolution with enhanced propagation and alignment.  In CVPR, pages 5972–5981, 2022.
  * [8] John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker.  Deepview: View synthesis with learned gradient descent.  In CVPR, pages 2367–2376, 2019.
  * [9] Dario Fuoli, Shuhang Gu, and Radu Timofte.  Efficient video super-resolution through recurrent latent space propagation.  In ICCVW, pages 3476–3485. IEEE, 2019.
  * [10] Clément Godard, Kevin Matzen, and Matt Uyttendaele.  Deep burst denoising.  In ECCV, pages 538–554, 2018.
  * [11] Yuxuan Han, Ruicheng Wang, and Jiaolong Yang.  Single-view view synthesis in the wild with learned adaptive multiplane images.  In ACM SIGGRAPH, 2022.
  * [12] Richard Hartley and Andrew Zisserman.  Multiple view geometry in computer vision.  Cambridge university press, 2003.
  * [13] Ronghang Hu, Nikhila Ravi, Alexander C Berg, and Deepak Pathak.  Worldsheet: Wrapping the world in a 3d sheet for view synthesis from a single image.  In ICCV, pages 12528–12537, 2021.
  * [14] Yan Huang, Wei Wang, and Liang Wang.  Bidirectional recurrent convolutional networks for multi-frame super-resolution.  In NeurIPS, 2015.
  * [15] Huawei.  Mindspore.  https://www.mindspore.cn/en, 2020.
  * [16] Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc Van Gool.  Dynamic filter networks.  In NeurIPS, 2016.
  * [17] Younghyun Jo, Seoung Wug Oh, Jaeyeon Kang, and Seon Joo Kim.  Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation.  In CVPR, pages 3224–3232, 2018.
  * [18] Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi.  Learning-based view synthesis for light field cameras.  ACM TOG, 35(6):1–10, 2016.
  * [19] Taras Khakhulin, Denis Korzhenkov, Pavel Solovev, Gleb Sterkin, Andrei-Timotei Ardelean, and Victor Lempitsky.  Stereo magnification with multi-layer images.  In CVPR, pages 8687–8696, 2022.
  * [20] Jiaxin Li, Zijian Feng, Qi She, Henghui Ding, Changhu Wang, and Gim Hee Lee.  Mine: Towards continuous depth mpi with nerf for novel view synthesis.  In ICCV, pages 12578–12588, 2021.
  * [21] Kai-En Lin, Zexiang Xu, Ben Mildenhall, Pratul P Srinivasan, Yannick Hold-Geoffroy, Stephen DiVerdi, Qi Sun, Kalyan Sunkavalli, and Ravi Ramamoorthi.  Deep multi depth panoramas for view synthesis.  In ECCV, pages 328–344. Springer, 2020.
  * [22] Li Ma, Xiaoyu Li, Jing Liao, Qi Zhang, Xuan Wang, Jue Wang, and Pedro V Sander.  Deblur-nerf: Neural radiance fields from blurry images.  In CVPR, pages 12861–12870, 2022.
  * [23] Matteo Maggioni, Giacomo Boracchi, Alessandro Foi, and Karen Egiazarian.  Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms.  IEEE TIP, 21(9):3952–3966, 2012.
  * [24] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In CVPR, pages 7210–7219, 2021.
  * [25] Ben Mildenhall, Jonathan T Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, and Robert Carroll.  Burst denoising with kernel prediction networks.  In CVPR, pages 2502–2510, 2018.
  * [26] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P Srinivasan, and Jonathan T Barron.  Nerf in the dark: High dynamic range view synthesis from noisy raw images.  In CVPR, pages 16190–16199, 2022.
  * [27] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  ACM TOG, 38(4):1–14, 2019.
  * [28] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [29] Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan.  Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.  In CVPR, pages 5480–5490, 2022.
  * [30] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In ICCV, pages 5865–5874, 2021.
  * [31] Naama Pearl, Tali Treibitz, and Simon Korman.  Nan: Noise-aware nerfs for burst-denoising.  In CVPR, pages 12672–12681, 2022.
  * [32] Eric Penner and Li Zhang.  Soft 3d reconstruction for view synthesis.  ACM TOG, 36(6):1–11, 2017.
  * [33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [34] Mehdi SM Sajjadi, Raviteja Vemulapalli, and Matthew Brown.  Frame-recurrent video super-resolution.  In CVPR, pages 6626–6634, 2018.
  * [35] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In CVPR, pages 4104–4113, 2016.
  * [36] Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski.  Layered depth images.  In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pages 231–242, 1998.
  * [37] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [38] Alvy Ray Smith and James F Blinn.  Blue screen matting.  In ACM SIGGRAPH, pages 259–268, 1996.
  * [39] Pavel Solovev, Taras Khakhulin, and Denis Korzhenkov.  Self-improving multiplane-to-layer images for novel view synthesis.  In WACV, pages 4309–4318, 2023.
  * [40] Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely.  Pushing the boundaries of view extrapolation with multiplane images.  In CVPR, pages 175–184, 2019.
  * [41] Richard Szeliski and Polina Golland.  Stereo matching with transparency and matting.  In ICCV, pages 517–524, 1998.
  * [42] Thomas Tanay, Aivar Sootla, Matteo Maggioni, Puneet K Dokania, Philip HS Torr, Ales Leonardis, and Greg Slabaugh.  Diagnosing and preventing instabilities in recurrent video processing.  IEEE TPAMI, 2022.
  * [43] Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia.  Detail-revealing deep video super-resolution.  In ICCV, pages 4472–4480, 2017.
  * [44] Matias Tassano, Julie Delon, and Thomas Veit.  Dvdnet: A fast network for deep video denoising.  In ICIP, pages 1805–1809, 2019.
  * [45] Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu.  Tdan: Temporally-deformable alignment network for video super-resolution.  In CVPR, pages 3360–3369, 2020.
  * [46] Richard Tucker and Noah Snavely.  Single-view view synthesis with multiplane images.  In CVPR, pages 551–560, 2020.
  * [47] Petro Vlahos and B Taylor.  Traveling matte composite photography.  American Cinematographer Manual, pages 430–445, 1993.
  * [48] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In CVPR, pages 4690–4699, 2021.
  * [49] Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy.  Edvr: Video restoration with enhanced deformable convolutional networks.  In CVPRW, 2019.
  * [50] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson.  Synsin: End-to-end view synthesis from a single image.  In CVPR, pages 7467–7477, 2020.
  * [51] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn.  Nex: Real-time view synthesis with neural basis expansion.  In CVPR, pages 8534–8543, 2021.
  * [52] Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow.  Interpretable transformations with encoder-decoder networks.  In ICCV, pages 5726–5735, 2017.
  * [53] Zhihao Xia, Federico Perazzi, Michaël Gharbi, Kalyan Sunkavalli, and Ayan Chakrabarti.  Basis prediction networks for effective burst denoising with large kernels.  In CVPR, pages 11844–11853, 2020.
  * [54] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman.  Video enhancement with task-oriented flow.  IJCV, 127(8):1106–1125, 2019.
  * [55] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: learning view synthesis using multiplane images.  ACM TOG, 37(4):1–12, 2018.

##  Appendix A Network architecture

The architecture of our Multiplane Feature Encoder-Renderer (MPFER) is
described in the main paper and illustrated in Figure 3. The Encoder and the
Renderer consist in two identical Unets with a base of 64 channels,
illustrated in more details in Figure 7.

Figure 7: Unet architecture used for the Encoder and for the Renderer in all
our MPFER experiments.

##  Appendix B Average metrics

In Section 4.1, Table 2, We compare our MPFER model to various 2D-based video
restoration methods for denoising of synthetic noise on the Spaces dataset.
Two of the methods we consider, BPN [53] and DeepRep [3], are burst denoising
methods producing only one denoised output for the entire set of noisy inputs.
By default, we chose this output to be frame number 6 at the center of the
camera rig and compared the performances of all the methods on that frame.
However, MPFER as well as BasicVSR [5] and BasicVSR++ [7], are multi-frame
denoising methods producing one denoised output per noisy input. We compare
their average performances over the 16 frames of the validation sequences in
Table 4. We see that the overall performances are comparable to those on frame
6 from Table 2. In particular, MPFER outperforms all other methods by large
margins on all noise levels. To qualitatively evaluate the cross-view
consistency of different methods, we also plot V$\times$W slices computed on
scene 52 in Figure 8. We run BPN and DeepRep 16 times (once per frame) to
obtain these profiles. Our method qualitatively matches the ground-truth
better than other methods.

Noisy

BPN

BasicVSR++

DeepRep

MPFER-64 (ours)

Ground truth

Figure 8: V$\times$W slices computed on scene 52 of Spaces.

##  Appendix C Ablations

Our MPFER method depends on three hyperparameters: the number of depth planes
$D$, the number of channels in the multiplane representation $C$, and the
upscaling factor of the PSV/MPF representation $s$. In Table 2 of the main
paper, we evaluated the influence of model size by varying these three
hyperparameters simultaneously. We now evaluate the influence of each
hyperparameter independently in Table 5. We see that the performance of the
method increases with $D$, $C$ and $s$, and so does the computational
complexity. Interestingly, the performance improvement is higher when $C$
increases from 4 to 16 (+0.4dB at Gain 20), than when $D$ increases from 16 to
64 (+0.23dB at Gain 20), while the increase in computational complexity is
significantly lower ($\times 1.33$ vs $\times 2.97$ respectively). This
observation confirms that multiplane features are inherently more powerful
representations than multiplane images, allowing to perform efficient 3D-based
video restoration with fewer depth planes.

##  Appendix D Qualitative evaluations

We consider 4 experimental setups in the main paper: (1) Novel View Synthesis
on Spaces, (2) Denoising on Spaces, (3) Denoising on the Real Forward Facing
dataset, and (4) Novel View Synthesis under Noise Conditions on the Real
Forward Facing dataset. We present some visual comparisons with state-of-the-
art methods for the setups (2) and (3) in Figure 4, and for the setup (4) in
Figure 1. We present some additional visual comparisons for the setup (1) in
Figure 9 here.

Gain 4 Gain 8 Gain 16 Gain 20 GFlops@ PSNR$\uparrow$ SSIM$\uparrow$
LPIPS$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$
PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$
LPIPS$\downarrow$ 500$\times$800 VBM4D 32.30 0.90 n/a 30.12 0.849 n/a 27.53
0.763 n/a 26.58 0.723 n/a n/a VNLB 33.41 0.917 0.089 30.31 0.869 0.144 25.79
0.794 0.279 23.58 0.746 0.363 n/a BasicVSR 36.86 0.957 0.029 34.45 0.935 0.052
31.62 0.895 0.099 30.59 0.875 0.124 2090 BasicVSR++ 36.81 0.957 0.030 34.39
0.934 0.051 31.62 0.895 0.091 30.60 0.875 0.111 4300 UNet-SF 35.15 0.942 0.042
32.67 0.910 0.074 29.86 0.857 0.134 28.87 0.834 0.160 440 UNet-BR 35.23 0.943
0.040 32.72 0.912 0.070 29.97 0.861 0.124 29.02 0.840 0.148 470 UNet-BR-OF
36.37 0.955 0.029 34.18 0.934 0.049 31.65 0.896 0.091 30.71 0.878 0.112 710
MPFER-16 37.20 0.965 0.021 35.37 0.952 0.033 33.22 0.927 0.055 32.41 0.915
0.067 470 MPFER-32 37.52 0.967 0.020 35.69 0.954 0.030 33.50 0.931 0.051 32.66
0.919 0.063 1210 MPFER-64 37.60 0.968 0.020 35.78 0.955 0.030 33.58 0.932
0.050 32.74 0.920 0.061 1810

Table 4: Denoising on Spaces. Average metrics over the 16 frames in the
validation sequences. Best results in bold, second best underlined.

Gain 4 Gain 8 Gain 16 Gain 20 GFlops@ $(D,C,s)$ PSNR$\uparrow$ SSIM$\uparrow$
LPIPS$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$
PSNR$\uparrow$ SSIM$\uparrow$ LPIPS$\downarrow$ PSNR$\uparrow$ SSIM$\uparrow$
LPIPS$\downarrow$ 500$\times$800 _Influence of the number of depth planes_
$(16,8,1.25)$ 37.75 0.969 0.019 35.98 0.957 0.028 33.83 0.934 0.049 33.00
0.922 0.061 610 $(32,8,1.25)$ 37.86 0.970 0.018 36.11 0.958 0.027 33.95 0.935
0.047 33.10 0.924 0.059 1010 $(64,8,1.25)$ 38.00 0.970 0.018 36.25 0.959 0.027
34.08 0.936 0.047 33.23 0.925 0.057 1810 _Influence of the number of channels
in the multiplane representation_ $(32,4,1.25)$ 37.64 0.969 0.021 35.81 0.957
0.031 33.59 0.935 0.051 32.74 0.923 0.062 910 $(32,8,1.25)$ 37.86 0.970 0.018
36.11 0.958 0.027 33.95 0.935 0.047 33.10 0.924 0.059 1010 $(32,16,1.25)$
37.94 0.970 0.019 36.17 0.958 0.028 33.99 0.936 0.047 33.14 0.924 0.058 1210
_Influence of the upscaling factor_ $(16,8,1.0)$ 37.56 0.968 0.020 35.80 0.955
0.030 33.70 0.933 0.051 32.89 0.921 0.063 470 $(16,8,1.25)$ 37.75 0.969 0.019
35.98 0.957 0.028 33.83 0.934 0.049 33.00 0.922 0.061 610 $(16,8,1.5)$ 37.86
0.969 0.019 36.08 0.957 0.029 33.92 0.935 0.050 33.08 0.923 0.061 780

Table 5: Denoising on Spaces. Influence of hyperparameters $(D,C,s)$: number
of depth planes, number of channels in the multiplane representation,
upscaling factor.

MPINet MPINet-dw MPINet-dw-it

DeepView MPFER-64 (ours) Ground truth

Figure 9: Qualitative evaluation for novel view synthesis on Spaces (best
viewed zoomed in).