# When would online platforms pay data dividends?

Sukanya Kudva and Anil Aswani *This material is based upon work supported by
the National Science Foundation under Grant CMMI-1847666.S Kudva and A Aswani
are with the Department of Industrial Engineering and Operations Research,
University of California, Berkeley, CA 94720, USA<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Online platforms, including social media and search platforms, have routinely
used their users’ data for targeted ads, to improve their services, and to
sell to third-party buyers. But an increasing awareness of the importance of
users’ data privacy has led to new laws that regulate data-sharing by
platforms. Further, there have been political discussions on introducing data
dividends, that is paying users for their data. Three interesting questions
are then: When would these online platforms be incentivized to pay data
dividends? How does their decision depend on whether users value their privacy
more than the platform’s free services? And should platforms invest in
protecting users’ data? This paper considers various factors affecting the
users’ and platform’s decisions through utility functions. We construct a
principal-agent model using a Stackelberg game to calculate their optimal
decisions and qualitatively discuss the implications. Our results could inform
a policymaker trying to understand the consequences of mandating data
dividends.

##  I INTRODUCTION

The revenue models of many online platforms depend on collecting, analyzing,
and selling users’ data. A free-service and advertising-based revenue model
can cause conflicts in the users’ and platform’s interests. Users may be
concerned about their privacy and possible misuse of data, while platforms
want to maximize their profits. Further, users’ perception of a platform’s
ethics and their willingness to participate can be affected by the platform’s
revenue model, pricing decisions, and privacy practices[1, 2, 3].

###  I-A Cybersecurity on online platforms

After the onset of the Internet of Things (IoT), there has been an increase in
the variety, speed, and volume of users’ data collected. Information from
multiple sources including devices, sensors, and social networks is being used
by platforms to assist users and collect data [4].

Though users have become more aware of the rampant collection of their data,
they often do not know about the proliferation of IoT devices in their
everyday lives. For instance, in August 2022 the Australian federal court
convicted a major search platform for collecting users’ location data without
their knowledge [5]. When users give consent and permissions to apps and
platforms, they tend to underestimate the implications of it [6]. Reforms to
protect users’ privacy are very much needed across the world [7]. Users should
be able to control, delete and transfer their data across different platforms
and service providers. They should be asked for explicit consent every time a
platform wants to use their data for a new purpose [4].

###  I-B Privacy and data dividends

With growing user concerns, consumer privacy legislation has become an
important topic for public discussion, and multiple new data privacy laws have
been introduced [8]. In Europe, the General Data Protection Regulation (GDPR)
was introduced in 2016 to give consumers more control over their data [9]. In
2019, legislators in California announced their intent to introduce data
dividends, which is a model in which platforms would pay users in exchange for
use of users’ data [10]. The same year, Oregon legislators introduced a bill,
called the Health Information Property Act, to compensate consumers for
monetizing their health data [11].

Implementing data dividends comes with its own challenges as companies holding
the data are far more powerful than individual users. Further, there is a huge
information asymmetry and only companies know the actual value of the users’
data. The critics of data dividends argue that selling data would make it a
commodity and be counter-productive in protecting users’ privacy. They also
feel that vulnerable groups – such as people of color and the poor – who are
currently discriminated against should not be incentivized to pour more data
into the system and further reduce their privacy [12, 13]. On the other hand,
the proponents of data dividends argue that today’s technology economy is
hugely driven by monetizing users’ data, and paying users a share of these
benefits is only fair.

Recent studies have explored different ways of pricing data dividends for each
user based on the value of their individual data [14, 15]. Using the idea of
Shapley and Owen values, they calculated the contribution made by each user to
the platform’s profits. Some scholars have also proposed different ways of
charging data dividends and how they could be used for the greater public good
[16]. For our work, we ask different questions: Should platforms pay data
dividends at all, and why? In this paper, we do our analysis with homogeneous
users but our methods can be extended to analyze heterogeneous users too.

###  I-C Contributions and outline

Our paper is organized as follows: Sect. II outlines our utility functions for
an online platform and its users and our principal-agent model, Sect. III
analytically solves the principal-agent model in order to derive their optimal
choices and Sect. IV discusses insights from our model.

Our paper comes in the context of rising debates on data dividends. We try to
understand when and how much online platforms would pay as data dividends. We
consider users’ privacy concerns as an important factor, for which a platform
can invest in data protection. The platform can also pay users to incentivize
taking risks and sharing their data. Our main contributions are to introduce a
principal-agent model using a Stackelberg game to capture the platform-users
dynamics and then use this to gain a better understanding of when the platform
would pay users with data dividends.

##  II OUR MODEL

An online platform provides its users with free services and a data dividend
in exchange for their data. The users are allowed to choose between two levels
– high or low – of data sharing. For each level, the platform provides a
different data dividend and set of services. As the platform collects users’
data, it faces the risk of a possible data breach. If a data breach occurs,
the platform loses its reputation and faces possible legal and financial
complications, and the users are harmed by the misuse of their data. Hence,
the platforms consider investing in protecting their users’ data. This could
include the costs of building better technological infrastructure and signing
insurance contracts.

###  II-A Platform’s utility

The platform has a fixed cost of $S$ to maintain and provide its services. It
invests $I$ in users’ data protection, reducing the probability of a data
breach to $B(I)$. Here, $B$ is assumed to be a twice differentiable function
such that $B(I)>0$ with $\lim_{I\rightarrow\infty}B(I)=0$, $B^{\prime}(I)<0$
and $B^{\prime\prime}(I)>0\;\forall I$. In case of a data breach, the platform
has a loss of $F$ from legal cases and a lost reputation. When having access
to $k$ users’ data, the platform makes a revenue of $U(k,b)$, where $b$ of the
total $k$ users chose the low level of data sharing. The revenue may be due to
selling the data to a third-party, selling advertisements to display to users
or other sources. The platform pays a share of this revenue to users as data
dividends, which are priced at two scales – $p_{0}$ and $p_{1}$ – for the low
and high levels of data sharing respectively.

Considering these costs, revenue, and risks, the platform has a total expected
utility of :

$\displaystyle U(k,b)\\--B(I)F\\--I\\--S\\--p_{0}b\\--p_{1}(k\\--b).$ (1)

###  II-B User’s utility

We consider $k$ homogeneous users who have the same behavior and parameters.
Each user values their personal data at $V$ and faces an additional personal
loss of $L$ on a data breach. They also benefit from the platform’s free
services, which amount to a value of $W$. When a user chooses the low level of
data sharing, these benefits and losses scale down by a factor
$\alpha\in(0,1)$. The platform pays users with data dividends at two scales of
pay and thereby encourages a particular level of data sharing.

Let $c_{i}$ be user $i$’s decision variable so that $c_{i}=0$ and $1$ for the
low and high levels of data sharing respectively. Then a user $i$’s total
expected utility is:

$\displaystyle
c_{i}(p_{1}\\--\mathcal{V})+(1\\--c_{i})(p_{0}-\alpha\mathcal{V}),$ (2)

where $\mathcal{V}=\bar{V}+B(I)L$ and $\bar{V}=V\\--W$.

###  II-C Principal-agent formulation

We construct a Stackelberg game [17, 18, 19], in which players act
sequentially with follower(s) acting after a leader. In our model, the
platform first prices data dividends for different levels of data-sharing, and
decides on investment for users’ data protection. Given this information, the
users decide how much data to share.

The optimal, equilibrium choices of the platform and the users are their
Stackelberg strategies. We capture this using an optimization problem, which
maximizes the platform’s utility when the users are maximizing their
utilities[20]. It is formulated as follows:

$\displaystyle\begin{split}\max&{\;\;U(k,b)\\--B(I)F\\--I\\--S\\--p_{0}b\\--p_{1}(k\\--b)}\\\
s.t&\;\;c_{i}^{*}=\text{arg}\max_{\;c_{i}\in\\{0,1\\}}c_{i}(p_{1}\\--\mathcal{V})+(1\\--c_{i})(p_{0}\\--\alpha\mathcal{V})\\\
&\hskip 150.0pt\forall i=\\{1,\cdots,k\\}\\\
&\;\;c_{i}^{*}(p_{1}\\--\mathcal{V})+(1\\--c_{i}^{*})(p_{0}\\--\alpha\mathcal{V})\geq
0\\\ &\hskip 150.0pt\forall i=\\{1,\cdots,k\\}\\\ &\;\;I,\>p_{0}\>,p_{1}\geq
0.\end{split}$ (3)

The second constraint in (3) ensures that the users do not have a net loss
from using the platform, without which they would leave the platform.

##  III Optimal choices

Since the users are homogeneous, they make similar choices: either
$c_{i}^{*}=1$ or $c_{i}^{*}=0$ for every user $i$. If users find both levels
of data sharing to be utility-maximizing, then we assume that they all choose
the level that most benefits the platform. We solve these two cases of
$c_{i}^{*}$ separately. The optimal solution is the best of the two cases and
can vary with the numerical values of the parameters of the model.

###  III-A Case 1: $c_{i}^{*}=1\;\;\forall i=\\{1,\cdots,k\\}$

To ensure users choose high level of data sharing, the platform must price the
data dividends such that $p_{1}\\--\mathcal{V}\geq p_{0}-\alpha\mathcal{V}$.
Note that $b=0$ in this case. The optimization problem then reduces to:

$\displaystyle\begin{split}\max&{\;U(k,0)\\--B(I)F\\--I\\--p_{1}k\\--S}\\\
s.t\;\;&p_{1}\\--\mathcal{V}\geq p_{0}\\--\alpha\mathcal{V}\\\
&p_{1}\\--\mathcal{V}\geq 0\\\ &I,\>p_{0}\>,p_{1}\geq 0.\end{split}$ (4)

Since $p_{0}\geq 0$, $p_{0}^{*}=0$ is optimal for the problem. Given this, one
can conclude $p_{1}^{*}=\mathcal{V}$ when $\mathcal{V}\geq 0$ and
$p_{1}^{*}=0$ when $\mathcal{V}\leq 0$. Also, $U(k,0)-S$ can be treated as a
constant. These observations further reduce (4) to two sub-cases with
optimization problems in a single variable $I$.

####  III-A1 Sub-case 1 - If $\mathcal{V}\geq 0$ then $p_{1}^{*}=\mathcal{V}$

Set $p_{1}^{*}=\mathcal{V}=\bar{V}+B(I)L$ and add an additional constraint
$\mathcal{V}\geq 0$ in (4).

$\displaystyle\begin{split}\min&{\;B(I)F_{1}+I+\bar{V}k}\\\
s.t\;\;&\bar{V}+B(I)L\geq 0\\\ &I\geq 0,\end{split}$ (5)

where $F_{1}=F+Lk$.

Let $I_{1}=B^{\prime-1}(\frac{\\--1}{F_{1}})$ and
$I_{2}=B^{-1}(\frac{\\--\bar{V}}{L})$.

######  Proposition III.1.

When $\bar{V}\geq 0$, $I^{*}=I_{1}$ if $I_{1}$ exists. Else, $I^{*}=0$.

###### Proof.

$B(I)L\geq\\--\bar{V}$ holds $\forall I$ because $\bar{V}\geq 0$. When $I_{1}$
exists, it’s the optimal solution to the unconstrained problem. Since it also
satisfies $I_{1}\geq 0$, it’s the optimal solution for the constrained problem
too. When $I_{1}$ doesn’t exist, the objective in (5) is an increasing
function for $I\geq 0$. So the smallest feasible value of $I$, i.e $0$, is
optimal. ∎

######  Proposition III.2.

When $\bar{V}\leq 0$,

  1. 1.

If $I_{1}$ exists, $I^{*}=I_{1}$ when $B(I_{1})L\geq\\--\bar{V}$. Else when
$B(I_{1})L<\\--\bar{V}$, $I^{*}=I_{2}$ if $I_{2}$ exists and no solution
otherwise.

  2. 2.

If $I_{1}$ doesn’t exist, $I^{*}=0$ when $B(0)L\geq\\--\bar{V}$ and no
solution otherwise.

###### Proof.

When $I_{1}$ exists and $B(I_{1})L\geq\\--\bar{V}$ holds, $I_{1}$, which is
the unconstrained optimal solution, satisfies both constraints and is optimal.
When $B(I_{1})L<\\--\bar{V}$, $I\geq I_{1}$ doesn’t satisfy the first
constraint in (5) and the objective is a decreasing function for $I<I_{1}$. If
$I_{2}$ exists, it is the largest $I<I_{1}$ satisfying $\bar{V}+B(I_{2})L\geq
0$ and is optimal. If $I_{2}$, which satisfies $\bar{V}+B(I_{2})L=0$, doesn’t
exist then there is no solution as $B(I)$ is a strictly decreasing function
and the constraint $B(I)L\geq-\bar{V}$ can never be satisfied.

When $I_{1}$ doesn’t exist, the objective is an increasing function for $I\geq
0$. If $B(0)L\geq\\--\bar{V}$ then $I=0$ satisfies both constraints of the
problem and is optimal. If $B(0)L<\\--\bar{V}$ then there is no solution as
$B(I)$ is strictly decreasing and the constraint $B(I)L\geq\\--\bar{V}$ can
never hold. ∎

####  III-A2 Sub-case 2 - If $\mathcal{V}\leq 0$ then $p_{1}^{*}=0$

Set $p_{1}=0$ and add an additional constraint $\mathcal{V}\leq 0$ in (4).

$\displaystyle\begin{split}\min&{\;B(I)F_{0}+I}\\\ s.t\;\;&\bar{V}+B(I)L\leq
0\\\ &I\geq 0\end{split}$ (6)

where $F_{0}=F$.

Let $I_{3}=B^{\prime-1}(\frac{\\--1}{F_{0}})$ and as before,
$I_{2}=B^{-1}(\frac{\\--\bar{V}}{L})$.

######  Proposition III.3.

When $\bar{V}>0$, there is no solution.

###### Proof.

$\\--\bar{V}\geq B(I)L$ can never hold for any $I$ as $\bar{V}>0$. ∎

######  Proposition III.4.

When $\bar{V}\leq 0$,

  1. 1.

If $I_{3}$ exists, $I^{*}=I_{3}$ when $\\--\bar{V}\geq B(I_{3})L$. Else when
$\\--\bar{V}<B(I_{3})L$, $I^{*}=I_{2}$.

  2. 2.

If $I_{3}$ doesn’t exist, $I^{*}=0$ when $\\--\bar{V}\geq B(0)L$ and
$I^{*}=I_{2}$ otherwise.

###### Proof.

When $I_{3}$ exists and $\\--\bar{V}\geq B(I_{3})L$ holds, $I_{3}$, which is
the unconstrained optimal solution, satisfies both constraints and is optimal.
When $\\--\bar{V}<B(I_{3})L$, $I\leq I_{3}$ doesn’t satisfy $\\--\bar{V}\geq
B(I)L$ as $B(I)$ is strictly decreasing. Since the objective is increasing for
$I>I_{3}$, $I_{2}$ which is the smallest $I$ satisfying $\\--\bar{V}\geq
B(I)L$ is optimal. Here, $I_{2}$ exists because $I_{2}>I_{3}$ and $B(I)$ is
strictly decreasing.

When $I_{3}$ doesn’t exist, the objective is increasing for $I\geq 0$ and
hence the smallest feasible $I$ is optimal. If $\\--\bar{V}\geq B(0)L$ then
$I=0$ satisfies both the constraints and is optimal. If $\\--\bar{V}<B(0)\>L$
then $I=I_{2}$ is the smallest feasible $I$ with $\bar{V}+B(I)L=0$ and is
optimal. ∎

The results from the two sub-cases can be combined by finding the minimum
objective value of the two sub-cases.

####  III-A3 Final results of Case 1

######  Theorem III.5.

For the case when all users choose the high level of data-sharing, when
$\bar{V}\leq 0$ the optimal choices of the platform are as follows:

  1. 1.

If $I_{1}$ and $I_{3}$ exist,
$(I^{*}=I_{1},\>p_{1}^{*}=\bar{V}+B(I_{1})L,\>p_{0}^{*}=0)$ when
$L\geq\frac{\\--\bar{V}}{B(I_{1})}$,
$(I^{*}=I_{2},\>p_{1}^{*}=0,\>p_{0}^{*}=0)$ when
$\frac{\\--\bar{V}}{B(I_{1})}\geq L\geq\frac{\\--\bar{V}}{B(I_{3})}$ and
$(I^{*}=I_{3},\>p_{1}^{*}=0,\>p_{0}^{*}=0)$ when
$\frac{\\--\bar{V}}{B(I_{3})}\geq L$.

  2. 2.

If $I_{1}$ exists but not $I_{3}$,
$(I^{*}=I_{1},\>p_{1}^{*}=\bar{V}+B(I_{1})L,\>p_{0}^{*}=0)$ when
$L\geq\frac{\\--\bar{V}}{B(I_{1})}$,
$(I^{*}=I_{2},\>p_{1}^{*}=0,\>p_{0}^{*}=0)$ when
$\frac{\\--\bar{V}}{B(I_{1})}\geq L\geq\frac{\\--\bar{V}}{B(0)}$ and
$(I^{*}=0,\>p_{1}^{*}=0,\>p_{0}^{*}=0)$ when $\frac{\\--\bar{V}}{B(0)}\geq L$.

  3. 3.

If both $I_{1}$ and $I_{3}$ don’t exist,
$(I^{*}=0,\>p_{1}^{*}=\bar{V}+B(0)L,\>p_{0}^{*}=0)$ when
$L\geq\frac{\\--\bar{V}}{B(0)}$ and $(I^{*}=0,\>p_{1}^{*}=0,\>p_{0}^{*}=0)$
when $\frac{\\--\bar{V}}{B(0)}\geq L$.

###### Proof.

The results from Propositions III.1-III.4 are combined by finding the optimal
of the two sub-cases.

  1. 1.

When $I_{1}$ and $I_{3}$ exist :

If $L\geq\frac{\\--\bar{V}}{B(I_{1})}$, the optimal solution is one of
$(I=I_{1},\>p_{1}=\bar{V}+B(I_{1})L,\>p_{0}=0)$ and
$(I=I_{2},\>p_{1}=0,\>p_{0}=0)$ with objective function values
$B(I_{1})F_{1}+I_{1}+\bar{V}k$ and $B(I_{2})F_{0}+I_{2}$ respectively. The
former solution is optimal if:

$\displaystyle\begin{split}B(I_{2})F_{0}+I_{2}&\geq
B(I_{1})F_{1}+I_{1}+\bar{V}k\\\ \iff I_{2}\\--I_{1}&\geq
F_{1}(B(I_{1})\\--B(I_{2}))\\\
\iff\frac{\\--1}{F_{1}}&\leq\frac{B(I_{1})\\--B(I_{2})}{I_{1}\\--I_{2}}\end{split}$
(7)

Since $B(I_{1})\geq\frac{\\--\bar{V}}{L}=B(I_{2})$, $I_{1}\leq I_{2}$ as $B$
is strictly decreasing. Using the mean value theorem, $\exists I^{\prime}$ s.t
$I_{1}\leq I^{\prime}\leq I_{2}$ and
$B^{\prime}(I^{\prime})=\frac{B(I_{1})\\--B(I_{2})}{I_{1}\\--I_{2}}$. As
$B^{\prime\prime}>0$, $B^{\prime}(I_{1})\geq
B^{\prime}(I^{\prime})\implies\frac{\\--1}{F_{1}}\leq\frac{B(I_{1})\\--B(I_{2})}{I_{1}\\--I_{2}}$.
Hence, (7) is true.

If $\frac{\\--\bar{V}}{B(I_{1})}\geq L\geq\frac{\\--\bar{V}}{B(I_{3})}$ then
both sub-cases give the same optimal solution
$(I^{*}=I_{2},\>p_{1}^{*}=0,\>p_{0}^{*}=0)$.

If $\frac{\\--\bar{V}}{B(I_{3})}\geq L\geq\frac{\\--\bar{V}}{B(0)}$ then the
optimal solution is one of $(I=I_{3},\>p_{1}=0,\>p_{0}=0)$ and
$(I=I_{2},\>p_{1}=0,\>p_{0}=0)$ with objective function values
$B(I_{3})F_{0}+I_{3}$ and $B(I_{2})F_{0}+I_{2}$ respectively. The former
solution is optimal if:

$\displaystyle\begin{split}B(I_{2})F_{0}+I_{2}&\geq B(I_{3})F_{0}+I_{3}\\\
\iff\frac{\\--1}{F_{0}}&\geq\frac{B(I_{3})\\--B(I_{2})}{I_{3}\\--I_{2}}\end{split}$
(8)

Since $B(I_{3})\leq\frac{\\--\bar{V}}{L}=B(I_{2})$, $I_{3}\geq I_{2}$ as $B$
is strictly decreasing. Using the mean value theorem, $\exists I^{\prime}$ s.t
$I_{2}\leq I^{\prime}\leq I_{3}$ and
$B^{\prime}(I^{\prime})=\frac{B(I_{3})\\--B(I_{2})}{I_{3}\\--I_{2}}$. As
$B^{\prime\prime}>0$, $B^{\prime}(I_{3})\geq
B^{\prime}(I^{\prime})\implies\frac{\\--1}{F_{0}}\geq\frac{B(I_{3})\\--B(I_{2})}{I_{3}\\--I_{2}}$.
Hence, (8) is true.

If $\frac{\\--\bar{V}}{B(0)}\geq L$ then
$(I^{*}=I_{3},\>p_{1}^{*}=0,\>p_{0}^{*}=0)$ is only solution from the two sub-
cases as one of them was infeasible.

  1. 2.

When $I_{1}$ exists but not $I_{3}$:

If $L\geq\frac{\\--\bar{V}}{B(I_{1})}$ then
$(I=I_{1},\>p_{1}=\bar{V}+B(I_{1})L,\>p_{0}=0)$ and
$(I=I_{2},\>p_{1}=0,\>p_{0}=0)$ are compared and (7) holds. If
$\frac{\\--\bar{V}}{B(I_{1})}\geq L\geq\frac{\\--\bar{V}}{B(0)}$ then both
sub-cases give the same optimal solution
$(I^{*}=I_{2},\>p_{1}^{*}=0,\>p_{0}^{*}=0)$. If $\frac{\\--\bar{V}}{B(0)}\geq
L$ then $(I^{*}=0,\>p_{1}^{*}=0,\>p_{0}^{*}=0)$ is only solution from the two
sub-cases as one of them was infeasible.

  1. 3.

When both $I_{1}$ and $I_{3}$ don’t exist:

If $L\geq\frac{\\--\bar{V}}{B(0)}$ then
$(I=0,\>p_{1}=\bar{V}+B(0)L,\>p_{0}=0)$ and $(I=I_{2},\>p_{1}=0,\>p_{0}=0)$
with objective function values $B(0)F_{1}+\bar{V}k$ and $B(I_{2})F_{0}+I_{2}$
respectively are compared. The former solution is optimal if:

$\displaystyle\begin{split}B(I_{2})F_{0}+I_{2}&\geq B(0)F_{1}+\bar{V}k\\\
\iff(B(I_{2})\\--B(0))F_{1}&\geq\\--I_{2}\\\
\iff\frac{B(I_{2})\\--B(0)}{I_{2}}&\geq\frac{\\--1}{F_{1}}\end{split}$ (9)

Using the mean value theorem, $\exists I^{\prime}$ s.t $I_{2}\leq
I^{\prime}\leq 0$ and $B^{\prime}(I^{\prime})=\frac{B(I_{2})\\--B(0)}{I_{2}}$.
And $B^{\prime}(I^{\prime})\geq\frac{\\--1}{F_{1}}$ as $I_{1}$ doesn’t exist.
Hence, (9) is true.

If $\frac{\\--\bar{V}}{B(0)}\geq L$ then
$(I^{*}=0,\>p_{1}^{*}=0,\>p_{0}^{*}=0)$ is only solution from the two sub-
cases as one of them was infeasible. ∎

######  Theorem III.6.

For the case when all users choose the high level of data-sharing, when
$\bar{V}\geq 0$ the optimal choices of the platform are as follows:

  1. 1.

If $I_{1}$ exists then
$(I^{*}=I_{1},\>p_{1}^{*}=\bar{V}+B(I_{1})L,\>p_{0}^{*}=0)$ is optimal.

  2. 2.

If $I_{1}$ doesn’t exist then
$(I^{*}=0,\>p_{1}^{*}=\bar{V}+B(0)L,\>p_{0}^{*}=0)$

###### Proof.

The results follow from Propositions III.1-III.4, and the above solutions are
the only feasible solutions from the two sub-cases. ∎

The final results of case 1 are summarized in table I.

###  III-B Case 2: $c_{i}^{*}=0\;\;\forall i=\\{1,\cdots,k\\}$

In this case, platform must price the data dividends such that
$p_{0}\\--\alpha\mathcal{V}\geq p_{1}\\--\mathcal{V}$. Here, $b=k$. The
optimization problem becomes:

$\displaystyle\begin{split}\max&{\;U(k,k)\\--B(I)F\\--I\\--p_{0}k\\--S}\\\
s.t\;\;&p_{0}\\--\alpha\mathcal{V}\geq p_{1}\\--\mathcal{V}\\\
&p_{0}\\--\alpha\mathcal{V}\geq 0\\\ &I,\>p_{0}\>,p_{1}\geq 0.\end{split}$
(10)

Similar to case 1, it can be observed that $p_{1}^{*}=0$ is optimal and
$p_{0}^{*}=\alpha\mathcal{V}$ when $\mathcal{V}\geq 0$ and
$p_{0}^{*}=(\alpha\\--1)\mathcal{V}$ when $\mathcal{V}\leq 0$.

Let $I_{4}=B^{\prime\\--1}\left(\frac{\\--1}{F+\alpha Lk}\right)$,
$I_{5}=B^{\prime-1}\left(\frac{\\--1}{F+(\alpha-1)Lk}\right)$ and as before,
$I_{2}=B^{-1}(\frac{\\--\bar{V}}{L})$.

######  Theorem III.7.

For the case when all users choose the low level of data-sharing, when
$\bar{V}\leq 0$ the optimal choices of the platform are as follows:

  1. 1.

If $I_{4}$ and $I_{5}$ exist,
$(I^{*}=I_{4},\>p_{1}^{*}=0,\>p_{0}^{*}=\alpha(\bar{V}+B(I_{4})L))$ when
$L\geq\frac{\\--\bar{V}}{B(I_{4})}$,
$(I^{*}=I_{2},\>p_{1}^{*}=0,\>p_{0}^{*}=0)$ when
$\frac{\\--\bar{V}}{B(I_{4})}\geq L\geq\frac{\\--\bar{V}}{B(I_{5})}$ and
$(I^{*}=I_{5},\>p_{1}^{*}=0,\>p_{0}^{*}=(\alpha\\--1)(\bar{V}+B(I_{5})L)$ when
$\frac{\\--\bar{V}}{B(I_{5})}\geq L$.

  2. 2.

If $I_{4}$ exists but not $I_{5}$,
$(I^{*}=I_{4},\>p_{1}^{*}=0,\>p_{0}^{*}=\alpha(\bar{V}+B(I_{4})L))$ when
$L\geq\frac{\\--\bar{V}}{B(I_{4})}$,
$(I^{*}=I_{2},\>p_{1}^{*}=0,\>p_{0}^{*}=0)$ when
$\frac{\\--\bar{V}}{B(I_{4})}\geq L\geq\frac{\\--\bar{V}}{B(0)}$ and
$(I^{*}=0,\>p_{1}^{*}=0,\>p_{0}^{*}=(\alpha\\--1)(\bar{V}+B(0)L))$ when
$\frac{\\--\bar{V}}{B(0)}\geq L$.

  3. 3.

If both $I_{4}$ and $I_{5}$ don’t exist,
$(I^{*}=0,\>p_{1}^{*}=0,\>p_{0}^{*}=\alpha(\bar{V}+B(0)L))$ when
$L\geq\frac{\\--\bar{V}}{B(0)}$ and
$(I^{*}=0,\>p_{1}^{*}=0,\>p_{0}^{*}=(\alpha\\--1)(\bar{V}+B(0)L))$ when
$\frac{\\--\bar{V}}{B(0)}\geq L$.

###### Proof.

Propositions similar to III.1-III.4 can be derived for case 2. The proof is
then similar to Theorem III.5. ∎

######  Theorem III.8.

For the case when all users choose the low level of data-sharing, when
$\bar{V}\geq 0$ the optimal choices of the platform are as follows:

  1. 1.

If $I_{4}$ exists then
$(I^{*}=I_{4},\>p_{1}^{*}=0,\>p_{0}^{*}=\alpha(\bar{V}+B(I_{4})L))$ is
optimal.

  2. 2.

If $I_{4}$ doesn’t exist then
$(I^{*}=0,\>p_{1}^{*}=0,\>p_{0}^{*}=\alpha(\bar{V}+B(0)L))$

###### Proof.

The proof is similar to Theorem III.6. ∎

The final results of case 2 are summarized in table I.

TABLE I:  Final results of Case 1 when $\bar{V}\leq 0$ : Here, $p_{0}^{*}=0$
always.
---
$I_{1}$ and $I_{3}$ exist |  $I_{1}$ exists but not $I_{3}$ |  $I_{1}$ and $I_{3}$ don’t exist
$\\!\begin{aligned} \text{If }L\geq\frac{\\--\bar{V}}{B(I_{1})}:\;&I^{*}=I_{1},\\\ &\;p_{1}^{*}=\bar{V}+B(I_{1})L.\\\ \text{If }\frac{\\--\bar{V}}{B(I_{1})}\geq L\geq\frac{\\--\bar{V}}{B(I_{3})}:&\;I^{*}=I_{2},\\\ &\;p_{1}^{*}=0.\\\ \text{If }\frac{\\--\bar{V}}{B(I_{3})}\geq L:&\;I^{*}=I_{3},\\\ &\;p_{1}^{*}=0.\\\ \end{aligned}$ | $\\!\begin{aligned} \text{If }L\geq\frac{\\--\bar{V}}{B(I_{1})}:\;&I^{*}=I_{1},\\\ &\;p_{1}^{*}=\bar{V}+B(I_{1})L.\\\ \text{If }\frac{\\--\bar{V}}{B(I_{1})}\geq L\geq\frac{\\--\bar{V}}{B(0)}:&\;I^{*}=I_{2},\\\ &\;p_{1}^{*}=0.\\\ \text{If }\frac{\\--\bar{V}}{B(0)}\geq L:&\;I^{*}=0,\\\ &\;p_{1}^{*}=0.\\\ \end{aligned}$ | $\\!\begin{aligned} \text{If }L\geq\frac{\\--\bar{V}}{B(0)}:\;&I^{*}=0,\\\ &\;p_{1}^{*}=\bar{V}+B(0)L.\\\ \text{If }\frac{\\--\bar{V}}{B(0)}\geq L:&\;I^{*}=0,\\\ &\;p_{1}^{*}=0.\\\ \end{aligned}$
Final results of Case 1 when $\bar{V}\geq 0$ : Here, $p_{0}^{*}=0$ always.
$I_{1}$ exists |  $I_{1}$ doesn’t exist
$I^{*}=I_{1}$ | $I^{*}=0$
$p_{1}^{*}=\bar{V}+B(I_{1})L$ | $p_{1}^{*}=\bar{V}+B(0)L$
Final results of Case 2 when $\bar{V}\leq 0$ : Here, $p_{1}^{*}=0$ always.
$I_{4}$ and $I_{5}$ exist |  $I_{4}$ exists but not $I_{5}$ |  $I_{4}$ and $I_{5}$ don’t exist
$\\!\begin{aligned} \text{If }L\geq&\frac{\\--\bar{V}}{B(I_{4})}:\;I^{*}=I_{4},\\\ &\;p_{0}^{*}=\alpha(\bar{V}+B(I_{4})L).\\\ \text{If }\frac{\\--\bar{V}}{B(I_{4})}\geq L\geq&\frac{\\--\bar{V}}{B(I_{5})}:\;I^{*}=I_{2},\\\ &\;p_{0}^{*}=0.\\\ \text{If }\frac{\\--\bar{V}}{B(I_{5})}&\geq L:\;I^{*}=I_{5},\\\ &\;p_{0}^{*}=(\alpha\\--1)(\bar{V}+B(I_{5})L).\\\ \end{aligned}$ | $\\!\begin{aligned} \text{If }L\geq&\frac{\\--\bar{V}}{B(I_{4})}:\;I^{*}=I_{4},\\\ &\;p_{0}^{*}=\alpha(\bar{V}+B(I_{4})L).\\\ \text{If }\frac{\\--\bar{V}}{B(I_{4})}\geq L\geq&\frac{\\--\bar{V}}{B(0)}:\;I^{*}=I_{2},\\\ &\;p_{0}^{*}=0.\\\ \text{If }\frac{\\--\bar{V}}{B(0)}&\geq L:\;I^{*}=0,\\\ &\;p_{0}^{*}=(\alpha\\--1)(\bar{V}+B(0)L).\\\ \end{aligned}$ | $\\!\begin{aligned} \text{If }L\geq&\frac{\\--\bar{V}}{B(0)}:\;I^{*}=0,\\\ &\;p_{0}^{*}=\alpha(\bar{V}+B(0)L).\\\ \text{If }\frac{\\--\bar{V}}{B(0)}&\geq L:\;I^{*}=0,\\\ &\;p_{0}^{*}=(\alpha\\--1)(\bar{V}+B(0)L).\\\ \end{aligned}$
Final results of Case 2 when $\bar{V}\geq 0$ : Here, $p_{1}^{*}=0$ always.
$I_{4}$ exists |  $I_{4}$ doesn’t exist
$I^{*}=I_{4}$ | $I^{*}=0$
$p_{0}^{*}=\alpha(\bar{V}+B(I_{4})L)$ | $p_{0}^{*}=\alpha(\bar{V}+B(0)L)$

##  IV Insights from optimal choices

The platform uses data dividends to direct users to choose a particular level
of data sharing. In general, the platform pays a higher data dividend for the
high level of data sharing to compensate for users’ larger risks. But it also
earns a larger revenue when it collects more data. This trade-off between
revenue and data dividend decides the optimal level of sharing.

It is optimal for the platform to pay for only one level of data sharing which
maximizes its profit. However, sometimes it may prefer to not pay at all. The
cases when this happens are outlined below.

###  IV-A Can $p_{1}^{*}=0$ while encouraging high level of data sharing?

Firstly, $p_{1}^{*}\neq 0$ when $V>W$. That is, the platform always pays a
data dividend whenever users value their data more than the platform’s free
services. When $V\leq W$, the users value the platform’s free services more
than their data. In this case, if the platform ensures that a data breach does
not occur then it does not pay any data dividend. So if $B(I)\rightarrow
0\;\forall I$, meaning there is a low possibility of a data breach, or
$L\rightarrow 0$ so that users face a small loss from a data breach, then no
data dividend is paid. Interestingly, even if $F\rightarrow\infty$, which is
if the platform has heavy losses from a data breach, then they do not pay
users with data dividends and instead invest heavily in data protection
($I\rightarrow\infty$). In general, the platform does not pay a data dividend
when $W\\--V-B(I^{*})L\geq 0$, that is, the net users’ utility before pay is
positive. Currently, major technology companies do not pay data dividends. Our
model suggests that these companies believe they are in the region described
by the above discussion.

###  IV-B Can $p_{0}^{*}=0$ while encouraging low level of data sharing?

Like for $p_{1}$, if $V>W$ then $p_{0}^{*}\neq 0$. But when $V\geq W$,
$p_{0}^{*}=0$ only if $W\\--V-B(I^{*})L=0$, that is, the net users’ utility is
zero. This is strange because the platform pays a data dividend when
$W\\--V-B(I^{*})L>0$, that is, the net users’ utility before pay is positive.
The reason is that in this case the users naturally prefer a high level of
data sharing. So the platform has to pay a data dividend to incentivize the
low level of data sharing. One could argue that the platform should rather
remove the option of a high level of data sharing, and it probably would.

Finally, we also analyze the trends in the investment for users’ data
protection. If $B(I)\rightarrow 0\;\forall I$, meaning the possibility of a
data breach is small, or if $F,L\rightarrow 0$ so that no party loses from a
data breach, then the platform does not invest in data protection. If
$B^{\prime}(I)\rightarrow 0\;\forall I$, that is when the likelihood of a data
breach is not very sensitive to investment, then the platform chooses to not
invest at all.

Figure 1: Here, users choose the high level of data sharing, $I_{1}$ exists,
and $\bar{V}\leq 0$. Data dividend $p_{1}$ is plotted for increasing values of
$L$, that is users’ loss from a data breach. The platform pays a data dividend
beyond a threshold when $\mathcal{V}>0$, that is users face a total loss.
Figure 2: Here, users choose the high level of data sharing and $\bar{V}\geq
0$. Data dividend $p_{1}$ is plotted for increasing values of $L$, that is
users’ loss from a data breach. The platform always pays a data dividend of at
least $\bar{V}$. Figure 3: Here, users choose the low level of data sharing,
$I_{4}$ exists, and $\bar{V}\leq 0$. Data dividend $p_{0}$ is plotted for
increasing values of $L$, that is users’ loss from a data breach.
Interestingly, the platform doesn’t pay a data dividend only when
$\mathcal{V}=0$, that is users neither have a net loss nor gain.

##  V Conclusion

We studied the dynamics between an online platform and its users when the
latter are paid with data dividends. We demonstrated how an online platform
could use data dividends to incentivize its users to share more data and take
more risks, while also giving them an option to share as much data as they
want. If the platform wants to gather more data, it pays a data dividend only
when the users perceive their risks and losses to be higher than their
benefits. Further, when users feel that they do not lose much from a data
breach, the platform prefers to invest in data protection rather than pay
users for risking a breach. On the other hand, when users perceive a high loss
from a breach, the platform has to both invest and pay. It would be
interesting to analyze multiple (more than 2) levels of data sharing, each
with a different data dividend, and where heterogeneous users are incentivized
to share more or fewer data as is optimal for the platform. We leave this for
future work.

It is debatable whether data dividends could treat data as a commodity and
exploit the poor and vulnerable. But if data dividends are introduced, our
model shows that platforms cannot always use it to shirk responsibility for
data protection. When users feel considerable harm from a data breach,
platforms would both invest in data protection and pay higher data dividends.
We believe that our insights could help a policymaker understand data
dividends better.

## References

  * [1] L. Xia, K. B. Monroe, and J. L. Cox, “The Price is Unfair! A Conceptual Framework of Price Fairness Perceptions,” Journal of Marketing, vol. 68, no. 4, pp. 1–15, 2004, DOI: 10.1509/jmkg.68.4.1.42733.
  * [2] Y. Su and L. Jin, “The Impact of Online Platforms’ Revenue Model on Consumers’ Ethical Inferences,” J Bus Ethics, vol. 178, no. 2, pp. 555–569, Jun. 2022, DOI: 10.1007/s10551-021-04798-0.
  * [3] L. Yu, H. Li, W. He, F.-K. Wang, and S. Jiao, “A meta-analysis to explore privacy cognition and information disclosure of internet users,” International Journal of Information Management, vol. 51, p. 102015, Apr. 2020, DOI: 10.1016/j.ijinfomgt.2019.09.011.
  * [4] C. Perera, R. Ranjan, L. Wang, S. U. Khan, and A. Y. Zomaya, “Big Data Privacy in the Internet of Things Era,” IT Professional, vol. 17, no. 3, pp. 32–39, May 2015, DOI: 10.1109/MITP.2015.34.
  * [5] A. A. Press, “Google to pay $60m fine for misleading Australians about collecting location data,” The Guardian, Aug. 12, 2022. Accessed: Sep. 24, 2022. [Online] Available: https://www.theguardian.com/technology/2022/aug/12/google-to-pay-60m-fine-for-misleading-australians-about-collecting-location-data.
  * [6] J. Golbeck and M.L. Mauriello, “User Perception of Facebook App Data Access: A Comparison of Methods and Privacy Concerns”, 2014, [Online] Available: http://hcil2.cs.umd.edu/trs/2014-19/2014-19.pdf.
  * [7] L. Na, C. Yang, C.-C. Lo, F. Zhao, Y. Fukuoka, and A. Aswani, “Feasibility of reidentifying individuals in large national physical activity data sets from which protected health information has been removed with use of machine learning”, JAMA Network Open, vol. 1, no. 8, pp. e186040–e186040, 2018.
  * [8] “2022 Consumer Privacy Legislation.” https://www.ncsl.org/research/telecommunications-and-information-technology/2022-consumer-privacy-legislation.aspx.
  * [9] “Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016.” http://cnpd.public.lu/en/legislation/droit-europ/union-europeenne/rgpd.html.
  * [10] “Newsom wants companies collecting personal data to share the wealth with Californians,” Los Angeles Times, May 05, 2019. https://www.latimes.com/politics/la-pol-ca-gavin-newsom-california-data-dividend-20190505-story.html.
  * [11] H. Journal, “Oregon Health Information Property Act Proposes Paying Patients to Share Their Healthcare Data,” HIPAA Journal, Jan. 31, 2019. https://www.hipaajournal.com/oregon-health-information-property-act-proposes-paying-patients-to-share-their-healthcare-data/.
  * [12] H. Tsukayama, “Why Getting Paid for Your Data Is a Bad Deal,” Electronic Frontier Foundation, Oct. 26, 2020. https://www.eff.org/deeplinks/2020/10/why-getting-paid-your-data-bad-deal.
  * [13] R. Bartlett, A. Morse, R. Stanton, and N. Wallace, “Consumer-lending discrimination in the FinTech Era,” Journal of Financial Economics, vol. 143, no. 1, pp. 30–56, Jan. 2022, DOI: 10.1016/j.jfineco.2021.05.047.
  * [14] E. Bax, “Computing a Data Dividend.” arXiv, Jun. 27, 2019. DOI: 10.48550/arXiv.1905.01805.
  * [15] E. Bax et al., “Data Consortia,” in Proceedings of the Future Technologies Conference (FTC) 2020, Volume 2, Cham, 2021, pp. 489–498. DOI: $10.1007/978-3-030-63089-8\\_31.$
  * [16] “A Data Dividend that Works: Steps Toward Building an Equitable Data Economy - Ideas - Berggruen Institute,” May 05, 2021. https://www.berggruen.org/ideas/articles/a-data-dividend-that-works-steps-toward-building-an-equitable-data-economy/, https://www.berggruen.org/ideas/articles/a-data-dividend-that-works-steps-toward-building-an-equitable-data-economy/.
  * [17] M. Simaan, “Equilibrium properties of the Nash and Stackelberg strategies,” Automatica, vol. 13, no. 6, pp. 635–636, Nov. 1977, DOI: 10.1016/0005-1098(77)90086-3.
  * [18] M. Simaan and J. B. Cruz, “On the Stackelberg strategy in nonzero-sum games,” J Optim Theory Appl, vol. 11, no. 5, pp. 533–555, May 1973, DOI: 10.1007/BF00935665.
  * [19] T. Basar, “On the relative leadership property of Stackelberg strategies,” J Optim Theory Appl, vol. 11, no. 6, pp. 655–661, 1973, DOI: 10.1007/BF00935564.
  * [20] Y. Lee and A. Aswani, “Optimally Designing Cybersecurity Insurance Contracts to Encourage the Sharing of Medical Data,” in Proceedings of the Conference on Decision and Control, 2022. To appear.


# A Two-Stage Masked Autoencoder Based Network for Indoor Depth Completion

Kailai Sun 1,2 Zhou Yang 1 Qianchuan Zhao 1
1 Tsinghua University 2 National University of Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Depth images have a wide range of applications, such as 3D reconstruction,
autonomous driving, augmented reality, robot navigation, and scene
understanding. Commodity-grade depth cameras are hard to sense depth for
bright, glossy, transparent, and distant surfaces. Although existing depth
completion methods have achieved remarkable progress, their performance is
limited when applied to complex indoor scenarios. To address these problems,
we propose a two-step Transformer-based network for indoor depth completion.
Unlike existing depth completion approaches, we adopt a self-supervision pre-
training encoder based on the masked autoencoder to learn an effective latent
representation for the missing depth value; then we propose a decoder based on
a token fusion mechanism to complete (i.e., reconstruct) the full depth from
the jointly RGB and incomplete depth image. Compared to the existing methods,
our proposed network, achieves the state-of-the-art performance on the
Matterport3D dataset. In addition, to validate the importance of the depth
completion task, we apply our methods to indoor 3D reconstruction. The code,
dataset, and demo are available at https://github.com/kailaisun/Indoor-Depth-
Completion.

##  1 Introduction

An investigation by the Environmental Protection Agency (EPA) has found that
over 75% of the global population resides in urban areas and spends
approximately 90% of their time indoors in buildings [31]. With the urban
population and the proliferation of buildings, there is an increasing demand
for the spatial layout of indoor environments and the information about
objects within them [34]. In order to meet the demands of human activities,
indoor spatial information representation is of paramount importance.

Indoor 3D reconstruction [25], which aims to create a digital spatial
information representation of the interior of a building in three dimensions,
has gained increasing attention recently. Indoor 3D reconstruction has wide-
ranging implications and significance in architecture [30], virtual reality,
navigation guidance, and robotics. Indoor 3D reconstruction typically uses
sensors (e.g., RGB-D camera [12], laser scanning [5], IMUs [16]) to collect
indoor scene data and develop artificial intelligence (AI) methods to
effectively capture the geometric details, spatial layout, and semantic
understanding of the indoor scene.

Figure 1: Pipelines of depth completion.

In indoor 3D reconstruction, depth completion is an important task. The depth
completion task focuses on using part of the depth data measured in the real
scene to obtain more dense and complete depth data. While existing depth
completion studies have made notable advancements, their performance is
limited when applied to complex indoor scenarios. Traditional methods [29, 27,
33, 17, 18] often need many hand-tuning hyper-parameters, and they struggle to
fill high-quality depth holes and complete complex depth structures
effectively. Existing deep learning-based methods in depth completion can be
divided into two categories: only taking the depth map as input [4, 13, 2,
22], and combining the depth map with the RGB image as input. The latter often
performs better with more scene information. Early methods in this category
typically directly input the depth map and RGB image as four-channel data into
the model [23, 24, 26, 20]. To improve them, recent data fusion methods employ
two encoders to process the depth map and RGB image separately. The dual
encoders extract multiscale features while performing data fusion [28, 36, 15,
7]. However, the latest methods often suffer from sensitivity to dynamic
environmental lighting, failing to predict accurate depth completion results
under demanding lighting conditions. Moreover, While many depth completion
algorithms are tailored for sparse depth images such as LiDAR, they are
difficult to directly apply to RGBD cameras. Thus, further research on the
network design, and adaptation in complex indoor scenarios and dynamic
environments is needed.

To address these problems, we notice that masks in computer vision could be
applied to simulate the missing data. Mask-based methods, e.g., Masked
Autoencoders (MAE) [10] only apply partial observation to reconstruct the
entire image, making the model learn robust features and improving the
generalization ability. Inspired by this, we consider the missing depth
patches as masks, then train an encoder-decoder model to complete (i.e.,
reconstruct) full depth images. We propose a Vision Transformer-based two-
stage network for indoor depth completion: an MAE-based self-supervision pre-
training encoder to learn an effective latent representation from the jointly
masked RGB and depth images; a decoder based on token fusion to complete
(reconstruct) the full depth from an incomplete depth image. To our knowledge,
it is the first work to apply MAE and Vision Transformers (ViT) in an indoor
depth completion task. The contributions: we introduce an MAE-based self-
supervision pre-training mechanism to learn a latent representation for the
missing depth value, a supervised fine-tuning mechanism for accurate indoor
depth completion. We demonstrate our method on the Matterport3D dataset with
SOTA performance. By applying on the indoor 3D reconstruction task, we
highlight the importance of the depth completion task.

##  2 Methods

Like autoencoder-based depth completion approaches, our method has an encoder
that maps the measured depth to a latent representation, and a decoder that
completes (i.e., reconstructs) the full depth from the latent representation.
Unlike existing depth completion approaches, we adopt a mask-based self-
supervision pre-training mechanism to extract an effective latent
representation from part depth data; we employ a Vision Transformer encoder,
and merge depth tokens to achieve the SOTA performance. Our method includes
self-supervision pre-training and supervised fine-tuning. Figure 1 illustrates
the idea, introduced next.

###  2.1 MAE-based Self-supervision Pre-training

Input In Figure 1(a), we follow a self-supervision learning pipeline to learn
an effective latent representation from partial observation. We divide a raw
RGB image and a true depth image (ground truth) into regular non-overlapping
patches, simultaneously. Then, we sample a subset of patches and mask the
remaining ones. We sample random patches from a uniform distribution to
prevent more masked patches near the image centre [10]. We mask these patches
to simulate the missing depth in the real scene. The masks in the raw RGB
image and the true depth image are spatially aligned. Then, we concatenate the
unmasked regions of the RGB image and true depth image to a four-channel RGB-D
image for further feature extraction.
Encoder The remaining RGB-D image patches after masking are fed into the
encoder along with position encodings. Since the masking operation reduces the
size of the input data, it allows larger encoder modules to be trained with
the same computational cost as the original ViT. We adopt a self-attention
mechanism and a Vision Transformer to learn global structural information. Our
encoder is based on a Vision Transformer but applied only on unmasked patches.
Like a standard Vision Transformer, our encoder embeds patches by a linear
projection with added position embeddings. To represent the position of each
patch, we use absolute position embedding, a learnable parameter. We
straightway add this parameter to the image tokens. Then, we process the
tokens via many Transformer blocks.
Decoder Before the decoder, because our encoder is applied only on unmasked
patches, we need to concatenate the mask tokens with the encoder’s output. The
masks in RGB-D images are encoded as a shared, learned parameter, which
indicates the presence of missing patches to be predicted. We add this
parameter and position embeddings to obtain mask tokens. We concatenate the
mask tokens with the encoder’s output. Then, we process the tokens via our
decoder, including many Transformer blocks.
Loss function The tokens after the decoder are then mapped to the same
dimensions as the original image through a fully connected layer. The purpose
of the task is to reconstruct the masked image patches, similar to the depth
completion task, which aims to reconstruct/complete the depth image. In the
true depth image, partial depth incompleteness is inevitable. When the model
needs to reconstruct parts that contain this incomplete depth, we do not wish
the missing depth to mislead the model. Unlike MAE, in the loss function, we
only consider the parts where the depth value exceeds zero. In particular, we
employ the Root Mean Square Error (RMSE) as the loss function in Equation (1):

$RMSE\left(D,D_{T}\right)=\sqrt{\frac{1}{\lvert O\rvert}\sum_{p\in
O}\|D\left(p\right)-D_{T}\left(p\right)\|^{2}},$ (1)

where $D$ represents the predicted depth image, $D_{T}$ represents the true
depth image, $p$ represents a pixel in the depth image, and $O$ represents
non-zero pixels in $D_{T}$.

###  2.2 Indoor Depth Completion based on Supervised Fine-tuning

Using this pretrained encoder and decoder directly for depth image completion
still has many limitations. The masked RGB image patches have yet to be
utilized for depth prediction (i.e., underuse the full RGB image). Prior
information from the RGB image has significant guiding implications [11].
Besides, the patches are square, while missing depth areas are irregular,
resulting in the information loss of RGB-D images. Therefore, to overcome
these limitations, we consider our encoder as a pre-training model to extract
implicit features and finetune the pretrained model to complete the entire
image. Figure 1(b) illustrates the whole algorithm pipeline. The algorithm(b)
is similar to the algorithm(a); the main differences:
Input In the fine-tuning process, following the self-supervised learning
method, we feed the masked true depth image into the encoder and reconstruct
the true depth image conditional on the masked RGB image. In contrast, we use
a supervised learning method to complete the depth image using the raw RGB and
depth images. Thus, the input only contains RGB-D image patches without masks.
Token Fusion Before the pre-training encoder, the RGB/depth image patches are
fused by concatenation. We perform multi-modal data fusion at both the input
level and the token level. After the pre-training encoder, the encoder’s
output and depth tokens are fused. We add a residual connection to connect the
raw depth image with the intermediate tokens from the encoder. In this step,
the features extracted from the RGB image and the depth image can be fused
with the depth image for the second time.
Loss function After the decoder, the tokens are mapped to the same size as the
raw depth image through a fully connected layer. Different from (1), we
compute the loss in all patches, including non-zero pixels and zero pixels.
Through the above improvements, our model can utilize the full RGB image
information and carry out two-stage data fusion in the whole process.

##  3 Experiments and Discussion

###  3.1 EXPERIMENTAL SETUP

Dataset We use the famous dataset from work [35], which is modified on the
original MatterPort3D [3]. The original data set takes RGB and depth panoramas
at fixed points indoors and then aligns RGB and depth images. We follow the
method [35] to generate true depth images. As a result, the dataset includes
104699 images for training and 474 images for testing. The size of each image
is $320\times 256$.
Evaluation Indicators We use the RMSE in the section 2.1, Mean Error(ME),
Structure Similarity Index Measure(SSIM [32]), and $\delta_{t}$ for
performance comparison.
Hyperparameters The input images are resized to $224\times 224$. The channel
is 4. We set the mask rate in our pre-training model at 75%. The encoder
comprises 24 layers with 16 attention heads and an embedding dimension of
1024. The decoder comprises 8 layers with 16 attention heads and an embedding
dimension of 512. Our pre-training model is trained for 200 epochs while our
fine-tuning model is trained for 20 epochs.

Table 1: Quantitative comparison of our proposed methods against the existing SOTA methods on the Matterport3D dataset. RMSE and MAE are measured in meters. Methods |  RMSE$\downarrow$ |  ME$\downarrow$ |  SSIM$\uparrow$ |  $\delta_{1.25}$ $\uparrow$ |  $\delta_{1.25^{2}}\uparrow$
---|---|---|---|---|---
Joint Bilateral Filter |  1.978 |  0.774 |  0.507 |  0.613 |  0.689
MRF[9] |  1.675 |  0.618 |  0.692 |  0.651 |  0.780
AD[19] |  1.653 |  0.610 |  0.696 |  0.663 |  0.792
FCN |  1.262 |  0.517 |  0.605 |  0.681 |  0.808
Zhang[35] |  1.316 |  0.461 |  0.762 |  0.781 |  0.851
Huang[15] |  1.092 |  0.342 |  $\bm{0.799}$ |  0.850 |  0.911
Struct-MDC[14] |  $1.060$ |  0.503 |  0.534 |  0.656 |  0.713
Pre-training |  1.216 |  0.675 |  0.642 |  0.705 |  0.800
Fine-tuning w/o Pre-training |  $\bm{0.660}$ |  0.243 |  0.654 |  0.794 |  0.904
Fine-tuning w/ Pre-training |  $\bm{0.690}$ |  $\bm{0.206}$ |  $\bm{0.765}$ |  $\bm{0.852}$ |  $\bm{0.912}$
Figure 2: Qualitative comparison of our proposed methods against the existing
depth completion methods on the Matterport3D dataset.

###  3.2 Quantitative and Qualitative comparison

Table 2: 3D Reconstruction Errors Methods | Mean (m)$\downarrow$ | Median (m)$\downarrow$ | Standard Deviation (m)$\downarrow$ | Minimum (m) | Maximum (m)$\downarrow$
---|---|---|---|---|---
Depth uncompletion | 0.138 | 0.053 | 0.200 | 0.0 | 1.106
Depth completion | $\bm{0.086}$ | 0.057 | $\bm{0.101}$ | $\bm{0.0}$ | $\bm{1.100}$

In Table 1, we compare our proposed methods with some existing depth
completion methods, including traditional methods (e.g., joint bilateral
filter, JBF), FCN-based methods (ResNet18), the methods of Zhang [35] and
[15], and Struct-MDC[14]. Compared to the existing methods, our proposed
methods, achieve the state-of-the-art (SOTA) performance on the Matterport3D
dataset. The result shows our significant improvement in most of the
evaluation metrics (e.g., RMSE, ME and $\delta_{t}$).

The pre-training model is better than the traditional methods (e.g., MRF) and
the method of Zhang [35] on RMSE, and only better than the Joint bilateral
filter on ME, and better than Joint bilateral filter and FCN on SSIM.

Our fine-tuning model achieves a significant improvement. More importantly,
our fine-tuning model achieves superior performance on the Matterport3D
dataset. In particular, on $\delta_{t}$ and ME, our fine-tuning model performs
best. In Figure 2, our method learns the structural features in the scene
effectively, reconstructing the edge structure instead of only smoothing the
missing depth values.

Our fine-tuning model also has limitations. It is slightly inferior to the
method of Huang[15] on SSIM. SSIM focuses on the overall structure of graphics
and the intuitive feeling of human eyes[32]. And SSIM can be easily affected
by variance and mean. For example, in Figure 2, the depth images generated by
our Transformer models have unsmooth grids, which need to be improved in the
future.

##  4 An application:Indoor 3D Reconstruction

Indoor 3D reconstruction by RGB-D cameras has many errors because of camera
limitations (e.g., bright lighting conditions, limited measured range, and
depth images often contain holes and noise). We claim that deep completion
plays a crucial role in ensuring the accuracy and reliability of indoor 3D
reconstruction. We use a living room scene in the ICL-NUIM dataset [8] to
confirm.

As for the input, for a fair comparison, we adopt an image fusion strategy in
Equation 2 to merge the depth image from our depth completion network with the
original image.

${D_{R}}\left(p\right)=\left\\{\begin{array}[]{lll}D_{O}\left(p\right)&&{D_{O}\left(p\right)\neq
0},\\\ D\left(p\right)&&{D_{O}\left(p\right)=0},\end{array}\right.$ (2)

where $D_{R}$ represents the fused depth image, $D_{O}$ represents the
original depth image, and $D$ represents the depth image, which is outputted
by our depth completion network.

As for evaluation indicators, we followed the evaluation method recommended by
the ICL-NUIM dataset, which uses the point cloud distance measurement in the
open-source software (i.e., CloudCompare [6]) to quantify the reconstruction
quality of 3D models.

We choose ORB SLAM3 [1] to estimate pose estimation and then use TSDF volume
and marching cubes algorithm[21] to get 3D reconstruction models. Since the
estimated trajectories exhibit minimal disparity, we conduct separate
experiments to compare the 3D reconstruction results using the completed depth
image $D_{R}$ or the original, incomplete depth image $D_{O}$. The errors are
shown in Table 2. It can be observed that the depth completion method leads to
a more minor mean error, standard deviation, and maximum error. Only the
median error slightly increases. Overall, ORB SLAM3 with completed depth
images performs better than incompleted depth images.

##  5 Conclusion

Although depth cameras can sense the distance information between objects,
many depth values are often lost because of the bright, glossy, transparent,
and distant surfaces. In this paper, we propose a two-stage Transformer-based
network to complete an indoor depth image given a single RGB-D image. Our
proposed network achieves the SOTA performance on the Matterport3D dataset. An
indoor 3D reconstruction task application highlights the depth completion’s
importance. We believe our method can be widely applied in other areas,
including 3D understanding, building renovation or construction, etc.

## References

  * Campos et al. [Dec. 2021] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. M. Montiel, and J. D. Tardós.  Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam.  _IEEE Transactions on Robotics_ , 37(6):1874–1890, Dec. 2021.
  * Cao et al. [2021] Z. Cao, A. Li, and Z. Yuan.  Self-supervised depth completion via adaptive sampling and relative consistency.  In _2021 IEEE International Conference on Image Processing (ICIP)_ , pages 3263–3267. IEEE, 2021.
  * Chang et al. [2017] A. Chang et al.  Matterport3d: Learning from rgb-d data in indoor environments.  In _2017 International Conference on 3D Vision (3DV)_ , pages 667–676, 2017.
  * Cheng et al. [2019] X. Cheng, P. Wang, and R. Yang.  Learning depth with convolutional spatial propagation network.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 42(10):2361–2379, 2019.
  * Cui et al. [2019] Y. Cui, Q. Li, B. Yang, W. Xiao, C. Chen, and Z. Dong.  Automatic 3-d reconstruction of indoor environment with mobile laser scanning point clouds.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , 12(8):3117–3130, 2019.
  * Girardeau-Montaut [2016] D. Girardeau-Montaut.  Cloudcompare.  France: EDF R&D Telecom ParisTech, 2016.
  * Gu et al. [2021] J. Gu, Z. Xiang, Y. Ye, and L. Wang.  Denselidar: A real-time pseudo dense depth guided depth completion network.  _IEEE Robotics and Automation Letters_ , 6(2):1808–1815, 2021.
  * Handa et al. [2014] A. Handa, T. Whelan, J.B. McDonald, and A.J. Davison.  A benchmark for rgb-d visual odometry, 3d reconstruction and slam.  In _IEEE Intl. Conf. on Robotics and Automation, ICRA_ , 2014.
  * Harrison and Newman [2010] Harrison and P. Newman.  Image and sparse laser fusion for dense scene reconstruction.  page 219–228, 2010.
  * He et al. [2022] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick.  Masked autoencoders are scalable vision learners.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15979–15988, 2022.
  * Hu et al. [2022] Junjie Hu, Chenyu Bao, Mete Ozay, Chenyou Fan, Qing Gao, Honghai Liu, and Tin Lun Lam.  Deep depth completion from extremely sparse data: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pages 1–20, 2022.
  * Huan et al. [2022] L. Huan, X. Zheng, and J. Gong.  Georec: Geometry-enhanced semantic 3d reconstruction of rgb-d indoor scenes.  _ISPRS Journal of Photogrammetry and Remote Sensing_ , 186:301–314, 2022.
  * Huang et al. [2019] Z. Huang, J. Fan, S. Cheng, S. Yi, X. Wang, and H. Li.  Hms-net: Hierarchical multi-scale sparsity-invariant network for sparse depth completion.  _IEEE Transactions on Image Processing_ , 29:3429–3441, 2019.
  * Jeon et al. [2022] Jinwoo Jeon, Hyunjun Lim, Dong-Uk Seo, and Hyun Myung.  Struct-mdc: Mesh-refined unsupervised depth completion leveraging structural regularities from visual slam.  _IEEE Robotics and Automation Letters_ , 7(3):6391–6398, 2022.
  * K et al. [2019] Huang Y K, Wu T H, Liu Y C, et al.  Indoor depth completion with boundary consistency and self-attention.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 2019.
  * Karam et al. [2020] S. Karam, V. Lehtola, and G. Vosselman.  Strategies to integrate imu and lidar slam for indoor mapping.  In _ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences_ , pages 223–230, 2020.
  * Kopf et al. [2007] J. Kopf, M. F. Cohen, D. Lischinski, and M. Uyttendaele.  Joint bilateral upsampling.  _ACM Transactions on Graphics (ToG)_ , 26(3):96, 2007.
  * Ku et al. [2018] J. Ku, A. Harakeh, and S. L. Waslander.  In defense of classical image processing: Fast depth completion on the cpu.  In _2018 15th Conference on Computer and Robot Vision (CRV)_ , pages 16–22. IEEE, 2018.
  * Liu and Gong [2013] J. Liu and X. Gong.  Guided depth enhancement via anisotropic diffusion.  In _Pacific-Rim Conference on Multimedia_ , page 408–417. Springer, 2013.
  * Long et al. [2021] Y. Long, H. Yu, and B. Liu.  Depth completion towards different sensor configurations via relative depth map estimation and scale recovery.  _Journal of Visual Communication and Image Representation_ , 80:103272, 2021.
  * Lorensen and Cline [1998] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In _Seminal graphics: pioneering efforts that shaped the field_ , pages 347–353. 1998.
  * Lu et al. [2020] K. Lu, N. Barnes, S. Anwar, and L. Zheng.  From depth what can you see? depth completion via auxiliary image reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11306–11315, 2020.
  * Ma and Karaman [2018] F. Ma and S. Karaman.  Sparse-to-dense: Depth prediction from sparse depth samples and a single image.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 4796–4803. IEEE, 2018.
  * Ma et al. [2019] F. Ma, G. V. Cavalheiro, and S. Karaman.  Self-supervised sparse-to-dense “‘bibtex : Self-supervised depth completion from lidar and monocular camera.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 3288–3295. IEEE, 2019.
  * Nikoohemat et al. [2020] Shayan Nikoohemat, Abdoulaye A. Diakité, Sisi Zlatanova, and George Vosselman.  Indoor 3d reconstruction from point clouds for optimal routing in complex buildings to support disaster management.  _Automation in Construction_ , 113, 2020.
  * Qu et al. [2020] C. Qu, T. Nguyen, and C. Taylor.  Depth completion via deep basis fitting.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 71–80, 2020.
  * Silberman et al. [2012] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus.  Indoor segmentation and support inference from rgbd images.  2012\.
  * Teixeira et al. [2020] L. Teixeira, M. R. Oswald, M. Pollefeys, and M. Chli.  Aerial single-view depth completion with image-guided uncertainty estimation.  _IEEE Robotics and Automation Letters_ , 5(2):1055–1062, 2020.
  * Tomasi and Manduchi [1998] C. Tomasi and R. Manduchi.  Bilateral filtering for gray and color images.  In _Sixth international conference on computer vision (IEEE Cat. No. 98CH36271)_ , pages 839–846. IEEE, 1998.
  * Trzeciak and Brilakis [2023] M. Trzeciak and I. Brilakis.  Dense 3d reconstruction of building scenes by ai-based camera–lidar fusion and odometry.  _Journal of Computing in Civil Engineering_ , 37(Article 4), 2023.
  * US Environmental Protection Agency [2009] US Environmental Protection Agency.  Buildings and their impact on the environment: A statistical summary.  US Environmental Protection Agency Green Building Workgroup, 2009.
  * Wang et al. [2004] Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ , 13(4):600–612, 2004.
  * Xiaiming [2013] C. Xiaiming.  Research of 3d reconstruction and filtering algorithm based on depth information of kinect, 2013.
  * Z et al. [2020] Kang Z, Yang J, Yang Z, and Cheng S.  A review of techniques for 3d reconstruction of indoor environments.  _ISPRS International Journal of Geo-Information_ , 2020.
  * Zhang and Funkhouser [2018] Y. Zhang and T. Funkhouser.  Deep depth completion of a single rgb-d image.  In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 175–185, 2018.
  * Zhang et al. [2020] Y. Zhang, P. Wei, H. Li, and N. Zheng.  Multiscale adaptation fusion networks for depth completion.  In _2020 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–7. IEEE, 2020.