# Federated Self-Supervised Learning of Monocular Depth Estimators for
Autonomous Vehicles

Elton F. de S. Soares
IBM Research & UNIRIO
Rio de Janeiro
<EMAIL_ADDRESS><EMAIL_ADDRESS>
&Carlos Alberto V. Campos
UNIRIO
Rio de Janeiro
<EMAIL_ADDRESS>

###### Abstract

Image-based depth estimation has gained significant attention in recent
research on computer vision for autonomous vehicles in intelligent
transportation systems. This focus stems from its cost-effectiveness and wide
range of potential applications. Unlike binocular depth estimation methods
that require two fixed cameras, monocular depth estimation methods only rely
on a single camera, making them highly versatile. While state-of-the-art
approaches for this task leverage self-supervised learning of deep neural
networks in conjunction with tasks like pose estimation and semantic
segmentation, none of them have explored the combination of federated learning
and self-supervision to train models using unlabeled and private data captured
by autonomous vehicles. The utilization of federated learning offers notable
benefits, including enhanced privacy protection, reduced network consumption,
and improved resilience to connectivity issues. To address this gap, we
propose FedSCDepth, a novel method that combines federated learning and deep
self-supervision to enable the learning of monocular depth estimators with
comparable effectiveness and superior efficiency compared to the current
state-of-the-art methods. Our evaluation experiments conducted on Eigen’s
Split of the KITTI dataset demonstrate that our proposed method achieves near
state-of-the-art performance, with a test loss below 0.13 and requiring, on
average, only 1.5k training steps and up to 0.415 GB of weight data transfer
per autonomous vehicle on each round.

_K_ eywords Monocular Depth Estimation $\cdot$ Self-Supervised Learning
$\cdot$ Federated Learning

##  1 Introduction

Because of the adverse impact of a poorly managed mobility system on the
quality of life, Smart Mobility is often presented as one of the main options
to seek more sustainable transport systems [1]. It could also be seen as a set
of coordinated actions aimed at improving cities’ efficiency, effectiveness,
and environmental sustainability. One of these actions is the development of
Intelligent Transportation Systems (ITS), which has been occurring since the
beginning of the 1970s and can be seen as the integration of advanced
technologies, which include electronic sensor technologies, data transmission
technologies, and intelligent control technologies, into the transportation
systems [2]. Nonetheless, the primary purpose of ITS is to provide better
services for drivers and riders [3].

In the last few years, a large amount of research effort has been made to
apply Big Data Analytics and other advanced Artificial Intelligence (AI)
techniques to improve ITS [4]. In contrast, a smaller amount has been focused
on developing intelligent agents to support ITS. The primary efforts made in
that sense are those focused on developing Autonomous Vehicles (AVs), which
are now one of the most prominent topics in the ITS initiative [5]. Research
on AVs has also applied advanced AI techniques to tackle its most critical
tasks, such as Computer Vision (CV). Scene Depth Estimation (DE) plays an
essential role in CV as it enables the perception and understanding of three-
dimensional scenes [6]. Lasers, structured light, and other reflections on the
object surface have traditionally been applied in active DE methods [6]. To
enable these approaches, elevated costs of human labor and computational
resources are usually required for obtaining dense and accurate depth maps
[7].

Thus, image-based DE has become one of the main focuses of recent research in
CV for AVs due to its lower deployment cost and a wider range of application
scenarios [6]. Image-based DE methods traditionally calculate the disparity
between two 2D images a binocular camera takes to obtain a depth map [8].
However, binocular DE methods require at least two fixed cameras, and it is
difficult to capture enough features in the image to match when the scene has
less or no texture [9].

Therefore, research began focusing on Monocular DE (MDE) [10]. Since MDE uses
a single camera to obtain an image or video sequence, which does not require
additional specialized equipment, it has an even wider applicability [6].
Nonetheless, as monocular images lack a reliable stereoscopic visual
relationship, the regression of depth in 3D space from it is an ill-posed
problem [6]. More specifically, monocular images adopt a 2D form to reflect
the 3D world. However, the depth of the scene is not captured by the imaging
process, making it impossible to judge the size and distance of an object in
the scene or whether it is occluded by another object [6].

Thus, we need to estimate the depth of each pixel from the monocular image.
Based on the pixel depth map, we can judge the size and distance of the
objects contained in that scene. When the estimated depth map can accurately
reflect the 3D structure of the scene, we can consider the estimation method
used to be effective [6]. Several State-of-The-Art (SoTA) solutions for MDE
make use of Self-Supervised Learning (SSL) of Deep Neural Networks (DNNs) for
this task in combination with other CV tasks, such as ego-motion/pose
estimation (PE) and semantic segmentation (SS) [11, 12]. Nonetheless, to the
best of our knowledge, none of the SoTA solutions for MDE combines the use of
Federated Learning (FL) [13] with SSL to learn MDE models from unlabeled and
private data captured by AVs.

The use of FL has been explored in many recent works on ITS and AVs [14, 15].
The main advantages of FL [16] include: (1) increased privacy protection, as
there is no longer the need to share the raw data collected by each vehicle
with a central server or other vehicles; (2) reduced network consumption, as
the size of the model updates that need to be shared in the FL process is
significantly smaller than the raw datasets; (3) increased resiliency to
connectivity loss when compared to the centralized approach; and (4) increased
robustness to Non-IID (independent and identically distributed) data [17].
Thus, we hypothesize that combining FL and SSL can enable learning models with
comparable effectiveness and superior efficiency to the SoTA methods in MDE
for AVs. Also, several works have explored the combination of SSL and FL on CV
tasks with promising results [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29,
30, 31, 32]. Nonetheless, none of them were evaluated on datasets of images
collected by vehicles, such as the SoTA benchmarks for MDE models [33].

Thus, this work’s main objective is to develop a solution for the problem of
MDE for AVs. This solution must be able to generate depth maps of images
captured by monocular cameras in moving AVs with high effectiveness and
efficiency. MDE effectiveness is essential for scene understanding by AVs, as
the depth information will help identify the distance of obstacles as well as
estimate the speed and acceleration of other moving vehicles [34]. Meanwhile,
high efficiency is another critical requirement of the ideal solution because
it cannot consume a high proportion of the computational resources available
on the vehicle, as these are already disputed by the other tasks the vehicle
must do in real-time. Besides, the ITS network infrastructure might be unable
to support the sharing of all the training data between AVs and/or a central
server; therefore, mitigating the bandwidth consumption can increase the ITS
infrastructure’s scalability.

To the best of our knowledge, this is the first work to present and discuss
empirical evidence of the applicability of Self-Supervised Federated Learning
(SSFL) to MDE for AVs.

This work tackles the following Research Questions (RQs):

  * •

RQ1 - Is the efficiency of the SSL of MDE models higher when applying FL (with
IID and Non-IID data) or a centralized approach 111Training a model on a
central server with data collected by all vehicles., in the AVs use case?

  * •

RQ2 - Is the effectiveness of SSL MDE models equivalent when applying FL (with
IID and Non-IID data) instead of a centralized approach in the AVs use case?

To answer the RQs, we provide the following contributions:

  1. 1.

We propose the FedSCDepth method to solve the problem of MDE in AVs using SSFL
for collaboratively training a depth estimator using unlabeled data captured
by vehicles with high effectiveness, efficiency, and privacy;

  2. 2.

We present an empirical evaluation of a prototype of the proposed method using
a real dataset for MDE in AVs.

  3. 3.

We show that FedSCDepth reaches comparable performance with the SoTA on MDE,
with lower computation and communication costs per vehicle per round than
centralized training, using both IID and Non-IID data;

Section 2 discusses the related work. Section 3 presents the proposed method,
and Section 4 details the evaluation experiments. Section 5 discusses the
results obtained, and Section 6 presents conclusions and future work.

##  2 Related Work

In this section, we present the theoretical background of the methods proposed
for solving the MDE problem, the methods that leveraged FL in the ITS and AV
domains, and the recent works that combined SSL and FL for CV tasks.

###  2.1 Evolution of Monocular Depth Estimation Methods

During the early phase of DE research, depth maps were primarily estimated
using various depth cues such as vanishing points, focus and defocus, and
shadows. However, most of these methods were limited to constrained scenes
[6]. In the subsequent Machine Learning (ML) period of DE research,
researchers proposed several handcrafted features and probabilistic graph
models. These models were utilized for MDE using parametric and non-parametric
learning within the ML framework [6]. The emergence of Deep Learning (DL)
marked a new period in DE research in which MDE became a task of inferring
depth maps from single 2D color images using DNNs. Eigen et al. [35] pioneered
this approach by introducing a coarse-to-fine framework.

DL techniques for MDE commonly employ encoder-decoder networks to generate
depth maps from RGB images. The encoder captures depth features using
convolution and pooling layers, while the decoder estimates pixel-level depth
maps using deconvolution layers. Skip connections preserve features at
different scales. Training involves minimizing a depth loss function until a
predefined threshold is reached [6]. Gradient descent variants are commonly
used, but their quality depends on hyperparameters and network initialization.
Image resizing is often necessary during initialization.

Supervised and semi-supervised approaches to MDE will typically require some
amount of labeled data [6], which might not represent training truly general
models for DE in the heterogeneous domains where AVs will be deployed. To
address this problem, several unsupervised methods have been proposed for
learning visual features from large datasets of unlabeled images or videos
without relying on human annotations [11]. These methods, often called self-
supervised, utilize pseudo-labels generated from raw data. Typically, they
employ one or more pretext tasks to learn from unlabeled data. By optimizing
the objective functions of pretext tasks, DNNs acquire higher-order
representational features, enabling them to predict desired visual features
such as image depth [11].

###  2.2 Self-Supervised Learning for Monocular Depth Estimation

SSL has introduced various pretext tasks, including colorizing grayscale
images, image inpainting, and image jigsaw puzzles [6]. These pretext tasks
have been explored in conjunction with other training paradigms. Similarly, in
addition to single-task learning, which involves training a single network for
DE, combining DE with other tasks such as PE, SS, and optical flow prediction
can lead to the acquisition of shared representations beneficial for multiple
related tasks [6, 11].

A notable series of works that incrementally enhanced SSL for MDE was the SC-
Depth series methods [36, 37, 38]. In SC-DepthV1 [36], authors focused on the
scale inconsistency issue of preexisting solutions and proposed a method to
enable scale-consistent DE over video. In their following work, SC-DepthV2
[37], they focused on the rotation issue in videos that are captured by
handheld cameras and proposed an auto-rectify network to handle large
rotations. Finally, in SC-DepthV3 [38], they focused on the issue of dynamic
objects and blurred object boundaries. Provided that, they proposed a method
that leverages an externally pretrained MDE model for generating single-image
depth prior, namely pseudo-depth, based on which novel losses are computed to
boost SSL. As a result, the models trained through this method can predict
sharp and accurate depth maps, even when trained from monocular videos of
highly dynamic scenes.

In the present work, we use SC-DepthV3 [38] as our baseline method for
centralized SSL of MDE models since it presented great results on two popular
benchmarking datasets for MDE in AVs: KITTI [39] and DDAD [40]. Additionally,
the well-documented source code222https://github.com/JiawangBian/sc_depth_pl
provided by its authors enabled us to quickly reproduce their experiments and
integrate them within our FL solution.Besides SC-DepthV3, we will also compare
our results with DepthFormer [41] and MonoFormer [42], two recent transformer-
based method that, to the best of our knowledge, currently hold the best
results on KITTI Eigen’s Split among the SSL-based methods. DephFormer’s main
characteristic is that it performs multi-frame SSL-based MDE by improving
feature matching across images during cost volume generation [41], while
MonoFormer uses a CNN-Transformer hybrid network to increase shape bias by
employing Transformers while compensating for the weak locality bias of
Transformers by adaptively fusing multi-level representations [42].

###  2.3 Federated Learning (FL)

Big Data-based ML systems usually collect, clean, and aggregate data into one
or multiple central servers deployed in the cloud for model training [15].
However, privacy has become a critical aspect of deploying these platforms in
recent years. The data used for training typically belongs to different
parties that might require different policies and privacy restrictions for
sharing data with the platform. In addition, while cloud servers provide
highly scalable computational power and storage, transferring data from
distributed agents to the cloud might demand high bandwidth from the network
infrastructure and incur high communication delays [15].

To tackle these issues, Google proposed FL to allow joint model training by
multiple parties [13]. In their approach, the model is assumed to be a neural
network whose parameter updates can be shared with a central server without
transferring the raw data through the network [15]. Usually, the central
server, also called Aggregator Agent (AA) [14], orchestrates the training
process and determines how often and how many distributed agents, also called
Federated Nodes (FN) [14], will contribute to the global model update.

###  2.4 Federated Learning with Non-IID data

The problem of Non-IID data (or heterogeneous data) exists in many ML
applications and distributed learning methods [17]. ML models are usually
trained under the assumption that the training data is IID [17]. Thus, when
the data of the FL clients or participants is Non-IID with regards to feature
values, categorical labels, or even just the quantity of samples, the trained
models’ performance might be reduced [17].

To comprehend the challenge posed by Non-IID data to FL, we need to consider
the SGD algorithm. Many DNN training algorithms depend largely on SGD for
optimization [17]. SGD updates the gradient of each sample every time [17].
Thus, the SGD algorithm converges faster to a local minimum, has a faster
update speed, and can be seamlessly applied to FL [17].

In Google’s seminal work [13], its authors claimed that FedAvg could make FL
more robust to Non-IID data, which was put in check by subsequent research
that presented evidence that, in some Non-IID data scenarios, FedAvg might be
unstable or even divergent [17]. Nonetheless, FedAvg is still regarded as a
baseline aggregation algorithm for FL, with good results on recent Non-IID
data experiments [22].

###  2.5 Self-Supervised Federated Learning

Table 1: Characterization of SSFL works with regards to their applicability to AVs, evaluation on CV tasks, experimentation with IID and NIID, and Datasets used. | Ref.
---
| AV
---
| CV
---
| IID
---
| NIID
---
Datasets
[43] | ✗ | ✗ | ✓ | ✗ |  | Sleep-EDF, HHAR, MobiAct, WiFi-CSI, WESAD
---
[44] | ✗ | ✗ | ✓ | ✗ |  | HHAR, MobiAct, HAPT
---
[45] | ✗ | ✗ | ✗ | ✓ | Custom
[18, 20] | ✗ | ✓ | ✓ | ✓ |  | CIFAR, Mini-ImageNet
---
[19, 21, 26] | ✗ | ✓ | ✓ | ✓ | CIFAR
[22] | ✗ | ✓ | ✓ | ✓ |  | ImageNet, CIFAR, MS-COCO, Amazon
---
[23] | ✗ | ✓ | ✓ | ✓ |  | CIFAR, Tiny-ImageNet, LEAF
---
[24] | ✗ | ✓ | ✗ | ✓ |  | CIFAR, SVHN
---
[25] | ✗ | ✓ | ✓ | ✓ |  | CIFAR, Fashion-MNIST
---
[27] | ✗ | ✓ | ✓ | ✓ |  | Retina, CIFAR, CelebA
---
[28] | ✗ | ✓ | ✗ | ✓ |  | CIFAR, Tiny-ImageNet
---
[29] | ✗ | ✓ | ✗ | ✓ |  | TCIA PET-CT, MICCAI 2015, CTI ICH D&S
---
[30] | ✗ | ✓ | ✓ | ✓ |  | MNIST, CIFAR
---
[31] | ✗ | ✓ | ✓ | ✓ |  | (Mini-)ImageNet, CIFAR, Mini-INAT2021
---
[32] | ✗ | ✓ | ✓ | ✓ |  | CIFAR, SVHN, STL-10, COVID-19, Mini-ImageNet
---
Ours | ✓ | ✓ | ✓ | ✓ |  | KITTI
---

Several recent works have explored the combination of SSL and FL with
promising results. In Table 1, we characterize those works concerning key
aspects, including their applicability for AV use cases, based on the datasets
in which they were evaluated. Although most of them tackled CV tasks [18, 19,
20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32], none of them were
evaluated on vehicular datasets, such as the SoTA MDE benchmarks. Thus,
although the approaches proposed by those works could be adapted to AV use
cases, none of them provided empirical evidence that SSFL could be adopted
successfully in AV use cases, which is precisely the gap we intend to fill in
the literature.

Regarding the CV tasks for which SSFL was used, most works focused on image
classification tasks. The most frequent datasets were variations of CIFAR [46]
and ImageNet [47]. Also, most works were evaluated with IID and Non-IID data.
Non-IID data is usually generated synthetically based on the number of images
containing a given object class. That is another aspect in which the present
work differentiates itself since we preserve the natural unbalance of samples
inherent to the data collection instead of synthetically generating one based
on some assumed distribution skew [22].

##  3 Proposed Method

In this section, we detail the proposed method, namely FedSCDepth, which
combines an SSL-based MDE component and an FL component, presented in the
following subsections.

###  3.1 Self-Supervised Monocular Depth Estimation

In this section, we describe the MDE model and the formalization of frame
warping and self-supervision losses.

####  3.1.1 Model Architecture

As in [38, 37, 36, 48], the core of the model architecture is composed of an
MDE network (DepthNet) and a PE network (PoseNet).Both the DepthNet and
PoseNet used ResNet18 [49] as their backbone.A fully convolutional U-Net
architecture is used for DepthNet [50] with a DispNet [51] as the decoder. The
activations are ELU nonlinearities at every layer except the output, where
sigmoids are used. The sigmoid output $\sigma$ is converted to depth with
$D=\frac{1}{(a\sigma+b)}$, where $a$ and $b$ are chosen to constrain $D$
between $0.1$ and $100$ units [48]. The PoseNet is a ResNet18 [52] modified to
accept a pair of color images (or six channels) as input and to predict a
single 6-DoF (Degrees of Freedom) relative pose [53, 38]. Also, as proposed in
SC-DepthV3 [38] we leverage a pre-trained MDE network (PseudoDepthNet) to
generate pseudo-depth. During the training of the DepthNet and PoseNet, the
PseudoDepthNet generates a single-image depth prior, which is used to boost
SSL.

Figure 1: DepthNet architecture. Illustration adapted from [50] and [54].
Figure 2: PoseNet architecture. Illustration adapted from [55] and [56].

An overview of the DepthNet and PoseNet architectures and their combination
with PseudoDepthNet in the SSL component is presented in Fig. 1, Fig. 2, and
Fig. 3, respectively.

Figure 3: SSL component overview. Given a training sample (i.e., image pair
$I_{a}$ and $I_{b}$) the combined self-supervision loss ($L_{self}$) is
computed. Meanwhile, a pseudo-depth map ($PD_{a}$) is generated using
PseudoDepthNet, while depth maps ($D_{a}$ and $D_{b}$) are produced by
DepthNet, and PoseNet outputs the pose estimate ($P_{a}b$). $PD_{a}$ and
$D_{a}$ are also fed to the Dynamic Region-Refinement (DRR) and Local
Structure Refinement (LSR) modules.

####  3.1.2 Frame Warping

Given a sequence of image frames captured by a moving monocular camera, the
reconstruction $I^{\prime}_{s\rightarrow t}$ of a target frame $I_{t}$ at time
$t$ can be obtained from a source frame $I_{s}$ at time $s$ by performing a
bi-linear interpolation over the reprojected frame coordinates. This
interpolation, also referred to as warping flow ($W$), can be formalized as
[57]:

$W=I^{\prime}_{s\rightarrow t}(p_{t})=I_{s}(\hat{p}_{s})$ (1)

where $\hat{p}_{s}$ is the reprojection of point $p_{t}$ into frame $I_{s}$.
To obtain the mapping from $p_{t}$ to $\hat{p}_{s}$, $p_{t}$ needs to be back-
projected into 3D point $X$ using the camera’s intrinsic matrix $K$ and the
depth map $D_{t}$ corresponding to $I_{t}$. Then $X$ is transformed to account
for camera movement $C_{t\rightarrow s}$ and projected onto the image plain
[57]. This transformation is formalized as:

$\hat{p}_{s}\sim KC_{t\rightarrow
s}\underbrace{D_{t}(p_{t})K^{-1}}_{\text{X}}.$ (2)

####  3.1.3 Photometric Loss

For a consecutive pair of images ($I_{a}$, $I_{b}$) randomly sampled from a
sequence of monocular images, their depths ($D_{a}$, $D_{b}$) and their 6-DoF
camera pose ($P_{ab}$) are predicted by forwarding the DepthNet and PoseNet,
respectively [38]. Provided that, the warping flow ($W_{ab}$) between $I_{a}$
and $I_{b}$ can be generated using $D_{a}$, $D_{b}$, and $P_{a}b$, and a
synthetic reconstruction of $I_{a}$ ($I^{\prime}_{a}$) can be generated using
$W_{ab}$ and $I_{b}$ via bi-linear interpolation [51]. Thus, the photometric
loss ($L_{p}$) between $I_{a}$ and $I^{\prime}_{a}$ can be used as a self-
supervision signal for both networks [36]. Formally,

$L_{p}=\frac{1}{|V|}\sum_{p\in V}||I_{a}(p)-I^{\prime}_{a}(p)||_{1},$ (3)

where $V$ corresponds to the valid points that are successfully projected from
$I_{a}$ to the image plane of $I_{b}$, and $|V|$ stands for the number of
points in $V$ [36]. $L_{1}$ loss is used to reduce the impact of outliers,
nonetheless, as it is not invariant to illumination changes, an additional
image dissimilarity loss (SSIM [58]) is used, as it normalizes the pixel
illumination [36]. The modified $L_{p}$ is formally defined as,

$L_{p}=\frac{1}{|V|}\sum_{p\in
V}(\lambda_{i}||I_{a}(p)-I^{\prime}_{a}(p)||_{1}+\lambda_{s}\frac{1-SSIM_{aa^{\prime}}(p)}{2}),$
(4)

where $SSIM_{aa^{\prime}}$ is the element-wise similarity between $I_{a}$ and
$I^{\prime}_{a}$ by the SSIM function [58].
$\lambda_{i}=0.15,\lambda_{s}=0.85$ [48].

####  3.1.4 Mask-Weighted Photometric Loss

To mitigate the adverse impact of moving objects and occlusions, a weight mask
$M=1-D_{diff}$ is used to assign low weights to inconsistent pixels and high
weights to consistent pixels [36]. Thus, a mask-weighted photometric loss
($L^{M}_{p}$) can be formalized as,

$L^{M}_{p}=\frac{1}{|V|}\sum_{p\in V}(M(p)\cdot L_{p}(p)).$ (5)

By replacing $L_{p}$ with $L^{M}_{p}$, the gradients of inaccurately predicted
regions have a lower impact on back-propagation [36].

####  3.1.5 Combined Self-Supervision Loss Function

As in [38], the edge-aware smoothness loss ($L_{s}$) [59] is used to
regularize the estimated depth maps since $L_{p}$ is neither very informative
in low-texture images nor in homogeneous regions. Also, to enforce that
$D_{a}$ and $D_{b}$ conform to the same 3D scene structure, another loss was
introduced in [36], based on a depth inconsistency map. In addition, to
mitigate the impact of moving objects and occlusions, a weight mask is used to
assign low weights to inconsistent pixels and high weights to consistent ones
[36]. Thus, by replacing $L_{p}$ with a mask-weighted photometric loss
($L^{M}_{p}$), the gradients of inaccurately predicted regions have less
impact in back-propagation [36].

Finally, as in [38] the signals produced by the PseudoDetphNet are used to
compute additional losses that help regularize the SSL: the Confident Depth
Ranking Loss ($L_{cdr}$) and the normal matching loss ($L_{n}$) that replaces
$L_{s}$ [38]. Also, the edge-aware relative normal loss ($L_{ern}$) helps
constrain the relative normal angles of sampled point pairs to be consistent
with pseudo-depth [38]. Thus, by combining these losses, a robust self-
supervision signal is obtained. Formally,

$L_{Self}=\alpha L^{M}_{p}+\beta L_{g}+\gamma L_{n}+\delta L_{cdr}+\epsilon
L_{ern}.$ (6)

As in [38], $\alpha=1$, $\beta=0.5$, $\gamma=0.1$. Auto-masking and per-pixel
minimum reprojection loss are used to filter stationary and non-best points
during training [53] and $\gamma=\delta=\epsilon$ [38].

###  3.2 Federated Learning

The main goal of FL is to learn a global model from highly distributed and
heterogeneous data by aggregating locally trained models on remote devices
[43], such as AVs. Considering that our MDE model (DepthNet) is represented as
$\varepsilon^{\theta}_{D}(I)=D$, our FL goal can be formally defined as:

$\underset{\theta}{min}\varepsilon^{\theta}_{D},\textnormal{ where
}\varepsilon^{\theta}_{D}:=\sum\limits_{c}^{C}{\frac{m_{c}}{m}\varepsilon^{\theta}_{D_{c}}}.$
(7)

where $C$ represents the number of participating client devices (participants)
in an FL round, $m_{c}$ is the total number of instances available for client
$c$ with $m=\sum\nolimits_{c}{m_{c}}$. Lastly, $\varepsilon^{\theta}_{D_{c}}$
denotes the local MDE model parameterized with weights $\theta$. To produce a
global model, FedAvg [13] is applied to accumulate client updates after every
FL round. Fig. 4 illustrates the FL of a self-supervised MDE model proposed in
this work. At the same time, Algorithm 1 details how FedAvg is applied for
generating the global models for MDE and PE.

Figure 4: Federated SSL of an MDE estimator for connected AVs. The aggregation
server dispatches the initial global models to the participating AVs, as
depicted by dashed blue lines. The AVs perform the SSL on their local copies
of the global models on their private data and send the model updates back to
the server, illustrated with dashed orange lines. The models are aggregated to
produce a new version of the global model that is sent back to the AVs. This
process is repeated until a stop condition is reached. Algorithm 1 FedAvg of
DepthNet and PoseNet. The $C$ participating AVs are indexed by $c$. $F$ is the
fraction of AVs active on each FL round. $E$ is the number of training passes
each AV makes over its local dataset on each round (local epochs), and $l$ is
the number of AVs selected for each round.

1:

2:DepthNet $\varepsilon^{\theta}_{D}(I)=D$;

3:PoseNet $\varepsilon^{\theta}_{P}(I_{i},I_{j})=P_{i\rightarrow j}$;

4:

5:for each round $r=1,2,...$ do

6:

7: $l=max(F*C,1)$;

8: $C_{l}$ = Random Set of $l$ AVs;

9:

10: for each AV $c\in C_{l}$ in parallel do

11:

12: $\varepsilon^{\theta^{c}}_{D_{r+1}}$ and
$\varepsilon^{\theta^{c}}_{P_{r+1}}=LocalUpdate(c,\varepsilon^{\theta}_{D_{r}},\varepsilon^{\theta}_{P_{r}})$;

13:

14: end for

15:

16:
$\varepsilon^{\theta}_{D_{r+1}}=\sum\limits_{c}^{C}{\frac{m_{c}}{m}\varepsilon^{\theta}_{D_{c}}}$;

17:
$\varepsilon^{\theta}_{P_{r+1}}=\sum\limits_{c}^{C}{\frac{m_{c}}{m}\varepsilon^{\theta}_{P_{c}}}$;

18:

19:end for

20:

21:procedure $LocalUpdate$($c$,
$\varepsilon^{\theta}_{D}$,$\varepsilon^{\theta}_{P}$)

22:

23: for each local epoch $i$ from 1 to $E$ do

24:

25: Execute Self-Supervised Training with $\varepsilon^{\theta}_{D}$ and
$\varepsilon^{\theta}_{P}$ on AV $c$;

26: Evaluate updated $\varepsilon^{\theta}_{D}$ and $\varepsilon^{\theta}_{P}$
with Validation Dataset on AV $c$;

27:

28: end for

29:

30: return updated $\varepsilon^{\theta}_{D}$ and $\varepsilon^{\theta}_{P}$;

31:

32:end procedure

33:

34:Updated Global DepthNet $\varepsilon^{\theta}_{D}(I)=D$;

35:Updated Global PoseNet
$\varepsilon^{\theta}_{P}(I_{i},I_{j})=P_{i\rightarrow j}$;

36:

##  4 Evaluation Experiments

###  4.1 Datasets and Scenarios

Our evaluation experiments are conducted with the publicly available KITTI
dataset [33], which contains monocular images and 3D scans from scenes
captured by cameras and sensors mounted on top of a moving vehicle. Following
the approach of [38, 37, 36], we also adopt Eigen’s split [35], with the
maximum depth set to 80 meters and images resized to a resolution of 832 x 256
pixels for training.

In our experiments, we assume that the 34 drives present in the training
dataset correspond to distinct AVs (although some were collected by the same
vehicle in different drives). Based on this assumption, we characterize three
base experimental scenarios: Centralized Training, Federated Training with IID
samples, and Federated Training with Non-IID samples.

####  4.1.1 Centralized Training (CT)

All vehicles upload their samples to a central server that will train the
depth prediction model and distribute the final version to all participants.

####  4.1.2 Federated Training with IID samples (FT-IID)

The train samples are randomly distributed across the participants, preserving
an equal number of samples across all participants, as depicted in Fig. 5. All
participants share all validation samples, acting as a gold standard. Each
participant trains their local model, which is initialized with the downloaded
global model, using their random subset of train samples and computes the
validation losses against the gold standard at the end of every epoch. After
each FL round, each participant (that was selected for that round) uploads its
local model to the aggregation server, which computes the FedAvg, and then
distributes the updated global model to all participants.

####  4.1.3 Federated Training with Non-IID samples (FT-NIID)

This scenario is similar to the previous, except for the fact that the train
samples are distributed according to the drives in which they were collected,
reflecting the natural unbalance of the data collection. Also, since the
number of samples per drive was highly skewed when selecting a subset of the
34 drives, we first picked the ones with the most train samples. Nonetheless,
the participants selected for each FL round were picked randomly, without
replacement, assuring that the training would go over every participant at
least once, given a sufficient number of FL rounds. In addition, to avoid
creating too much advantage for the IID scenario, we randomly redistributed
the remaining samples (from the participants with the least number of train
samples) across the selected participants. Thus, both the IID and the Non-IID
scenarios had access to all the samples, changing only their distribution
across participants. As shown in Fig. 5, this redistribution did not remove
the great unbalance present in the original distribution by drive. However, it
substantially increased the lower bound for the number of train samples.

(a)  (b)  (c)

Figure 5: Number of samples by a participant in each of the sample
distribution strategies considered across 34 (a), 10 (b), and 9 (c)
participants, with and without random redistribution of remaining samples.

###  4.2 Metrics

For assessing the effectiveness of the final models, we adopt standard depth
evaluation metrics [38, 37, 36] that include the mean absolute relative error
($AbsRel$), root mean squared error ($RMS$), root means squared log error
($RMSlog$), and accuracy under threshold ($\delta_{i}<1.25^{i},i=1,2,3$),
which are defined in detail in Eigens’s seminal work [35]. Also, as in [38],
the predicted depth maps are multiplied by a scalar matching the median with
the ground truth for evaluation.

For assessing the efficiency of the training methods, we consider the $AbsRel$
computed over the validation set during the training as our reference for
effective learning, and, adapting a communication cost estimation approach
proposed in [60] for FL, we formally compute its upper bound ($W_{max}$) as

$W_{max}=2T(C\times\omega^{*}_{B}).$ (8)

where $C$ is the total number of participants, $T$ is the total number of
communication rounds (or FL rounds), and $\omega^{*}$ is the number of model
parameters. In our formulation, we replace $\omega^{*}$ with $\omega^{*}_{B}$
to make it explicit that what we consider is the number of model parameters in
Bytes (B). The main motivation for this is to make the comparisons with the
estimated cost for CT more direct since these will be estimated based on the
dataset size, which is also measured in Bytes. Additionally, we estimate its
lower bound ($W_{min}$) considering only the participants selected for
training the model as,

$W_{min}=2T(C\times F\times\omega^{*}_{B}).$ (9)

where $F$ is the fraction of participants that were selected for training the
model locally on each FL round. In this estimate, we assume that instead of
updating the global model for every participant on every round, only those
that will perform local training will have access to the latest global model
version.

Finally, we also analyze the number of training steps as a proxy for a
computational cost estimate since these represent the number of batches the
model has "seen" during the learning process (including images repeated across
epochs). Formally, the number of training steps at a given epoch of the CT
($\\#Steps$) can be computed as,

$\\#Steps=\\#Epochs\times\\#Batches$ (10)

where $\\#Epochs$ is the number of epochs the model has already been trained
on and $\\#Batches$ is the number of batches per epoch, assuming every epoch
was trained over the same number of batches. $\\#Batches$ was fixed as $1k$,
to enable comparison with SC-DepthV3’s original results [38].

For the FL scenarios, the computation of the training steps is adjusted to
account for the number of participants. Thus, we define the steps in a given
FT round ($\\#Steps_{FL}$) as,

$\\#Steps_{FL}=\sum^{\hat{T}}_{t=1}\\#Steps^{t}_{FL},$ (11)
$\\#Steps^{t}_{FL}=\sum_{p\in P_{t}}\\#Steps(p)$ (12)
$\\#Steps(p)=\\#Epochs_{p}\times\\#Batches_{p}$ (13)

where $\hat{T}$ is the number of FL rounds elapsed, $P_{t}$ are the
participants that performed training on the round $t$ (which are not
necessarily all $P$), and $\\#Epochs_{p}$ is the number of epochs through
which participant $p$ iterated over $\\#Batches_{p}$ batches.

Thus, $\\#Steps(p)$ is the number of training steps for participant $p$, and
$\\#Steps^{t}_{FL}$ is the total number of training steps for FL round $t$.
Although there might be fewer batches available for a given $p$, in the Non-
IID scenarios, we have configured the training to resample the available
batches randomly until the maximum number of $\\#Batches_{p}=\\#Steps$ is
reached.

###  4.3 Implementation Details

Our FedSCDepth prototype implementation was based on SC-DepthV3 [38] and Dec-
SSL [22] source codes, which were made publicly available on GitHub by their
authors. Our implementation was also shared publicly on GitHub
333https://github.com/eltonfss/federated-sc-depth.

As in SC-DepthV3 [38], the DNN implementation used PyTorch Lighting
444https://www.pytorchlightning.ai/, with Adam optimizer, and learning rate
set to $10^{-4}$. The DNN encoder was initialized using ImageNet [47] pre-
trained weights. As previously mentioned, the maximum number of batches per
epoch was set as $1k$ and the batch size as $4$ to allow comparing CT and FT
results.

FT experiments ran for 12 rounds with a total of 18 different setups resulting
from the combination of the following parameter values: $C=\\{10,9\\}$;
$F=\\{1,\frac{1}{2},\frac{1}{3}\\}$; $E=\\{1,2,3\\}$.

As in Dec-SSL [22], the local updates were simulated on the same process in
which the FedAvg aggregation was computed. Although this simulation strategy
might not be sufficiently realistic for estimating all possible metrics, it
does not impact the metrics we adopt for estimating the effectiveness of the
trained models and the efficiency of the training.

The experiments were deployed and executed on a bare metal server with 1 x CPU
i3-12100F 4,3 GHz (4 cores, 8 threads), 2 x 16GB DDR4 RAM (3200 MHz), 1 x
GeForce RTX 2060 GPU with 12 GB GDDR6 RAM, and 1 x SSD M.2 2280 1TB (93GB
configured as swap memory). The server was configured with Ubuntu 22.04.2 LTS,
Python 3.8.15, Conda 4.12.0, Pip 22.3.1, and CUDA version 12.1.

The experiments were executed directly on the server without using any
virtualization. Python dependencies were installed in a Conda environment, as
described in the sources.

###  4.4 Ablation Studies

In this section, we analyze the impact of the number of participants per round
($C*F$), the number of local training epochs ($E$), the number of
communication/federation rounds ($T$), and the data heterogeneity (IID x NIID)
on the lowest global validation losses ($AbsRel$) and their corresponding
communication and computational costs, depicted in Figure 6.

(a)  (b)  (c)  (d)  (e)  (f)  (g)  (h)

Figure 6: Lowest global validation loss (a), its estimated communication cost
upper bound (c), and lower bound (e), and training steps to obtain it (g), by
number of participants. Lowest global validation loss (b), its estimated
communication cost upper bound (d), and lower bound (f), and training steps to
obtain it (h), by number of local epochs.

####  4.4.1 Impact of number of participants per round

In Fig. 6 (a), we find that the Validation Loss (VL) is lower when $C\times
F=5$, between $4.2\%$ to $5.5\%$ lower than the highest. Meanwhile, in Fig. 6
(c), the $W_{max}$ corresponding to the best VL is lower when $C\times F=10$,
$34\%$ to $42\%$ lower than the highest, while in Fig. 6 (e), $W_{min}$ is
lower when $C\times F=3$, $51.7\%$ to $66.7\%$ lower than the highest.
Finally, in Fig. 6 (g), the number of training steps corresponding to the best
VL is lower when $C\times F=3$, $52.9\%$ to $77.5\%$ lower than the highest.

####  4.4.2 Impact of number of local epochs

In Fig. 6 (b), we find that the best VL is lower when $E=3$, $7.2\%$ to
$8.6\%$ lower than the highest. Meanwhile, in Fig. 6 (d), the $W_{max}$
corresponding to the best VLs is lower when $E=1$, $8\%$ to $10\%$ lower than
the highest, while in Fig. 6 (f), the $W_{min}$ is also lower when $E=1$,
$40\%$ to $54\%$ lower than the highest. Finally, in Fig. 6 (h), the number of
training steps corresponding to the best VLs is also lower when $E=1$,
$77.1\%$ to $80\%$ lower than the highest.

####  4.4.3 Impact of number of federation rounds

In Fig. 7 (a), we find that the best VL decreases rapidly until $T=3$, with a
modest decrease afterwards. Nonetheless, the lowest values are observed with
$T=12$. Meanwhile, in Fig. 7 (b), the number of training steps corresponding
to the best VLs increases almost linearly with $T$ up to about 240k steps, at
$T=12$, while in Fig. 7 (c) and (d), we observe that the $W_{max}$ and
$W_{min}$ corresponding to the best VLs follow a similar trend, scaling up to
about 50GB and 33GB, respectively.

(a)  (b)  (c)  (d)

Figure 7: Lowest global validation loss (a), its communication cost upper
bound (b), and lower bound (c), and steps to obtain it (d), by number of
rounds.

####  4.4.4 Impact of data heterogeneity

Analyzing Fig. 6 and 7, we find that the proposed method showed robustness to
the data heterogeneity inherent to the data collection, obtaining the lowest
VL with NIID data (about $3\%$ lower than the lowest VL with IID data).
Meanwhile, the impact on communication costs was not very high since the
maximum cost (about 50GB) was the same for both. Also, the maximum number of
training steps when using NIID data was the same as IID, about 240k.

###  4.5 Comparison with Centralized Training

After analyzing the different FT configurations, we concluded that the one
that showed the best results was the one with $C\times F=5$, $E=3$, $T=12$ and
IID data. This was especially due to the fact that it produced the lowest VL
and, although the additional communication and computation cost required for
it was not negligible, it was within a reasonable value for AV use cases.
Therefore, in Fig. 8 (a), we observe that the selected FT configuration
reached a VL about $7\%$ worse than the best VL obtained with CT with an
additional total computational cost of about $80\%$. One thing to note here is
that, as the VLs are computed at the end of every epoch for the CT, its first
data point was obtained at 1k training steps. Meanwhile, in the FT, the first
VL is computed after the first FL round is complete. Thus, the number of steps
of its first data point will depend on the values of $C\times F$, $E$, and
$\\#Steps=1000$.

(a)  (b)  (c)

Figure 8: Lowest validation loss (a) and its estimated communication cost
upper bound (b), and lower bound (c), by number of training steps.

Meanwhile, in Fig. 8 (b) and (c), we find that the final $W_{min}$ and
$W_{max}$ were about $1.93\times$ and $2.85\times$ higher with FT than CT,
respectively. Nonetheless, the total CT communication cost needs to be paid
right at the first round, while in FT, this cost is split across 12 rounds.
Considering that there would be 10 AVs involved in the data collection and
training, we would have to transmit, on average, about 1.293GB per AV with CT
(in the first round only) and something between 0.208GB and 0.415GB per AV
with FT (on each round), which indicates that the communication cost paid by
each AV on the first round would be on average $67.9\%$ to $83.9\%$ lower with
FT. Also, if we consider that in CT, the computational cost of 100k training
steps has to be paid by the central server at the first round, while in FT,
the computational cost of 180k training steps is shared by the 10 AVs, with an
average of about 1.5k training steps being performed by each AV on each round,
we conclude that FT promotes a more efficient cost distribution overall.

Finally, in Table 2 we observe that the efficacy metrics obtained on the test
set with the best model obtained with FT were very close to the ones obtained
with CT, even matching the $RMSlog$ and obtaining a slight advantage on the
$SqlRel$ metrics calculated over dynamic regions (image regions classified as
vehicles or pedestrians [38]).

Table 2: Effectiveness Comparison with CT on KITTI (Eigen Split). Scn. | Dynamic | Static
---|---|---
$AbsRel$ | $SqRel$ | $RMS$ | $RMSlog$ | $AbsRel$ | $SqRel$ | $RMS$ | $RMSlog$
FT | 0.202 | 1.933 | 7.248 | 0.282 | 0.119 | 0.723 | 4.861 | 0.177
CT | 0.191 | 2.072 | 7.111 | 0.282 | 0.106 | 0.638 | 4.275 | 0.159

###  4.6 Comparison with State-of-the-Art on SSL-based MDE

In Table 3, we present the MDE efficacy metrics obtained in the test dataset
with the best-performing FT configuration of the FedSCDepth. We also compare
its results with the results reported by three SoTA SSL-based (centralized)
MDE methods: SC-DepthV3 (SCD) [38], which was the baseline of our SSL MDE
component; MonoFormer (MF) [42] and DepthFormer (DF) [41], the best performing
SSL-based MDE methods in the KITTI Eigen Split [35].

Table 3: Effectiveness Comparison with SoTA on KITTI (Eigen Split). Method | Resolution | $AbsRel$ | $SqRel$ | $RMS$ | $RMSlog$ | $\delta_{1}$ | $\delta_{2}$ | $\delta_{3}$
---|---|---|---|---|---|---|---|---
DF [41] | $640\times 192$ | 0.090 | 0.661 | 4.149 | 0.175 | 0.905 | 0.967 | 0.984
MF [42] | $640\times 192$ | 0.104 | 0.846 | 4.580 | 0.183 | 0.891 | 0.962 | 0.982
SCD [38] | $832\times 256$ | 0.118 | 0.756 | 4.709 | 0.188 | 0.864 | 0.960 | 0.984
Ours | $832\times 256$ | 0.128 | 0.803 | 5.015 | 0.197 | 0.836 | 0.956 | 0.984

Analyzing Table 3, we can observe that the $AbsRel$ obtained by FedSCDepth is
about $8.5\%$, $23.1\%$, and $42.2\%$ worse than SCD, MF, and DF,
respectively. Meanwhile, $\delta_{3}$ was the same for all except MF, and the
$RMSlog$ with FedSCDepth is about $4.8\%$, $7.6\%$, and $12.6\%$ worse than
with SCD, MF, and DF, respectively. Based on those results, we conclude that
most of the difference between FedSCDepth and DF, which was the best
performing overall, is due to the transformer-based technique employed by the
latter, which produced highly superior efficacy than SCD. This difference is
also visible when we compare them with MF, which presents a much closer
performance to DF than SCD. Thus, our results were very close to SCD, which
represents the pre-transformer SSL-based MDE SoTA and is considered our main
baseline.

##  5 Discussion

After analyzing the different efficiency metrics of the federated and
centralized training methods, we consider the proper answer for RQ1 to be the
following: FT is more efficient than CT when we accept a less strict loss
threshold (such as $AbsRel$ below $0.13$). Meanwhile, to reach optimal depth
prediction loss (below $0.12$), the CT will be more efficient concerning the
total computational and communication costs. Nonetheless, it should be noted
that while in CT, the communication cost has to be paid right at the
beginning, in FT, this cost is paid across several rounds (at most, 4.15 GB of
data are transferred on each round, totaling an average of 0.415 GB per
participant on each round and 4.98 GB per participant after 12 rounds).

Also, while the computational cost is entirely paid by the central server in
the CT (100k training steps), this cost is shared by the participating AVs in
the FT scenarios (on average, 1.5k training steps are performed by each
participant at each round, totaling 18k steps of training by each participant
at the 12th round). Finally, when comparing the FT efficiency with IID and
Non-IID data, it was better with IID data in most scenarios. Nonetheless,
there were no significant differences in efficiency in those two FT setups
overall, which is an indication that the proposed solution would perform well
in realistic FT with AVs, which usually presents Non-IID data.

Meanwhile, after comparing the effectiveness of the best FT model with the
SoTA, we consider the proper answer for RQ2 to be the following: MDE models
obtained with CT are more effective than those learned with FT. Nonetheless,
the effectiveness lost when using FT is minimal, with the models obtained with
FT reaching near SoTA performance in only 12 rounds. Also, the effectiveness
obtained by the models obtained with FT is significantly better when working
with NIID data for most scenarios, which indicates that this approach is
highly applicable to realistic AV deployments, where data collection is
typically unbalanced.

##  6 Conclusion

In this paper, we tackle the problem of monocular depth estimation for
autonomous vehicles. The key to our method is using federated and self-
supervised learning to collaboratively train a depth estimator using unlabeled
data captured by vehicles with high effectiveness, efficiency, and privacy
preservation. We evaluate a prototype implementation of this method using the
KITTI dataset and show that it can achieve near-SoTA performance with a low
computation cost per vehicle and a lower communication cost per round per
vehicle than centralized training. Additionally, the experimental results
indicate that the proposed method is robust to Non-IID data, even using simple
FedAvg aggregation. Future work includes exploring other aggregation functions
and optimization strategies to further reduce the proposed method’s
computational and communication costs, as well as evaluating its
generalizability with other public benchmark datasets.

## References

  * [1] A Khamis.  Smart mobility: Exploring foundational technologies and wider impacts. apress, 2021.
  * [2] Li Zhu, Fei Richard Yu, Yige Wang, Bin Ning, and Tao Tang.  Big data analytics in intelligent transportation systems: A survey.  IEEE Transactions on Intelligent Transportation Systems, 2018.
  * [3] Sheng-hai An, Byung-Hyug Lee, and Dong-Ryeol Shin.  A survey of intelligent transportation systems.  In IEEE CICSyN, 2011.
  * [4] Hexuan Hu, Bo Tang, Xuejiao Gong, Wei Wei, and Huihui Wang.  Intelligent fault diagnosis of the high-speed train with big data based on deep neural networks.  IEEE Transactions on Industrial Informatics, 2017.
  * [5] Abdulaziz Aldakkhelallah and Milan Simic.  Autonomous vehicles in intelligent transportation systems.  In HCIS, pages 185–198. Springer, 2021.
  * [6] Yue Ming, Xuyang Meng, Chunxiao Fan, and Hui Yu.  Deep learning for monocular depth estimation: A review.  Neurocomputing, 438:14–33, 2021.
  * [7] Junning Zhang, Qunxing Su, Cheng Wang, and Hongqiang Gu.  Monocular 3d vehicle detection with multi-instance depth and geometry reasoning for autonomous driving.  Neurocomputing, 403:182–192, 2020.
  * [8] Alexander N Gorban, Evgeny M Mirkes, and Ivan Y Tyukin.  How deep should be the depth of convolutional neural networks: a backyard dog case study.  Cognitive Computation, 12(2):388–397, 2020.
  * [9] Fei Liu, Shubo Zhou, Yunlong Wang, Guangqi Hou, Zhenan Sun, and Tieniu Tan.  Binocular light-field: Imaging theory and occlusion-robust depth perception application.  IEEE Transactions on Image Processing, 29:1628–1640, 2019.
  * [10] Hamid Laga.  A survey on deep learning architectures for image-based depth reconstruction.  arXiv preprint arXiv:1906.06113, 2019.
  * [11] Longlong Jing and Yingli Tian.  Self-supervised visual feature learning with deep neural networks: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
  * [12] Jierui Liu, Zhiqiang Cao, Xilong Liu, Shuo Wang, and Junzhi Yu.  Self-supervised monocular depth estimation with geometric prior and pixel-level sensitivity.  IEEE Transactions on Intelligent Vehicles, 2022.
  * [13] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Artificial intelligence and statistics. PMLR, 2017.
  * [14] Dimitrios Michael Manias and Abdallah Shami.  Making a case for federated learning in the internet of vehicles and intelligent transportation systems.  IEEE Network, 35(3):88–94, 2021.
  * [15] Zhaoyang Du, Celimuge Wu, Tsutomu Yoshinaga, Kok-Lim Alvin Yau, Yusheng Ji, and Jie Li.  Federated learning for vehicular internet of things: Recent advances and open issues.  IEEE Open Journal of the Computer Society, 1, 2020.
  * [16] Stefano Savazzi, Monica Nicoli, Mehdi Bennis, Sanaz Kianoush, and Luca Barbieri.  Opportunities of federated learning in connected, cooperative, and automated industrial systems.  IEEE Communications Magazine, 59, 2021.
  * [17] Xiaodong Ma, Jia Zhu, Zhihao Lin, Shanxuan Chen, and Yangjie Qin.  A state-of-the-art survey on solving non-iid data in federated learning.  Future Generation Computer Systems, 135:244–258, 2022.
  * [18] Fengda Zhang, Kun Kuang, Zhaoyang You, Tao Shen, Jun Xiao, Yin Zhang, Chao Wu, Yueting Zhuang, and Xiaolin Li.  Federated unsupervised representation learning.  arXiv preprint arXiv:2010.08982, 2020.
  * [19] Juan Zhao, Ruixuan Li, Haozhao Wang, and Zijun Xu.  Hotfed: Hot start through self-supervised learning in federated learning.  In 2021 IEEE HPCC/DSS/SmartCity/DependSys, pages 149–156. IEEE, 2021.
  * [20] Weiming Zhuang, Xin Gan, Yonggang Wen, Shuai Zhang, and Shuai Yi.  Collaborative unsupervised visual representation learning from decentralized data.  In IEEE/CVF international conference on computer vision, 2021.
  * [21] Simou Li, Yuxing Mao, Jian Li, Yihang Xu, Jinsen Li, Xueshuo Chen, Siyang Liu, and Xianping Zhao.  Fedutn: federated self-supervised learning with updating target network.  Applied Intelligence, pages 1–14, 2022.
  * [22] Lirui Wang, Kaiqing Zhang, Yunzhu Li, Yonglong Tian, and Russ Tedrake.  Does decentralized learning with non-iid unlabeled data benefit from self supervision?  arXiv preprint arXiv:2210.10947, 2022.
  * [23] Disha Makhija, Nhat Ho, and Joydeep Ghosh.  Federated self-supervised learning for heterogeneous clients.  arXiv preprint arXiv:2205.12493, 2022.
  * [24] Jiahe Shi, Yawen Wu, Dewen Zeng, Jingtong Hu, and Yiyu Shi.  Fedcoco: A memory efficient federated self-supervised framework for on-device visual representation learning.  arXiv preprint arXiv:2212.01006, 2022.
  * [25] Yawen Wu, Zhepeng Wang, Dewen Zeng, Meng Li, Yiyu Shi, and Jingtong Hu.  Decentralized unsupervised learning of visual representations.  In IJCAI, 2022.
  * [26] Weiming Zhuang, Yonggang Wen, and Shuai Zhang.  Divergence-aware federated self-supervised learning.  arXiv preprint arXiv:2204.04385, 2022.
  * [27] Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Ehsan Adeli, Li Fei-Fei, and Daniel Rubin.  Rethinking architecture design for tackling data heterogeneity in federated learning.  In IEEE/CVF CVPR, 2022.
  * [28] Xutong Mu, Yulong Shen, Ke Cheng, Xueli Geng, Jiaxuan Fu, Tao Zhang, and Zhiwei Zhang.  Fedproc: Prototypical contrastive federated learning on non-iid data.  Future Generation Computer Systems, 143:93–104, 2023.
  * [29] Sangjoon Park, Ik-Jae Lee, Jun Won Kim, and Jong Chul Ye.  Ms-dino: Efficient distributed training of vision transformer foundation model in medical domain through masked sampling.  arXiv preprint arXiv:2301.02064, 2023.
  * [30] Nan Yang, Dong Yuan, Charles Z Liu, Yongkun Deng, and Wei Bao.  Fedil: Federated incremental learning from decentralized unlabeled data with convergence analysis.  arXiv preprint arXiv:2302.11823, 2023.
  * [31] Nan Yang, Xuanyu Chen, Charles Z Liu, Dong Yuan, Wei Bao, and Lizhen Cui.  Fedmae: Federated self-supervised learning with one-block masked auto-encoder.  arXiv preprint arXiv:2303.11339, 2023.
  * [32] Chen Zhao, Zhipeng Gao, Yang Yang, Qian Wang, Zijia Mo, and Xinlei Yu.  Fedusc: Collaborative unsupervised representation learning from decentralized data for internet of things.  IEEE Internet of Things Journal, 2023.
  * [33] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger.  Sparsity invariant cnns.  In IEEE 3DV, 2017.
  * [34] Shoufeng Jin, Jiajie Yin, Mingrui Tian, Shizhe Feng, Sarkodie-Gyan Thompson, and Zhixiong Li.  Practical speed measurement for an intelligent vehicle based on double radon transform in urban traffic scenarios.  MST, 32(2), 2020.
  * [35] David Eigen, Christian Puhrsch, and Rob Fergus.  Depth map prediction from a single image using a multi-scale deep network.  Advances in neural information processing systems, 27, 2014.
  * [36] Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian Reid.  Unsupervised scale-consistent depth and ego-motion learning from monocular video.  In NeurIPS, 2019.
  * [37] Jia-Wang Bian, Huangying Zhan, Naiyan Wang, Tat-Jin Chin, Chunhua Shen, and Ian Reid.  Auto-rectify network for unsupervised indoor depth estimation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  * [38] Libo Sun, Jia-Wang Bian, Huangying Zhan, Wei Yin, Ian Reid, and Chunhua Shen.  Sc-depthv3: Robust self-supervised monocular depth estimation for dynamic scenes.  arXiv:2211.03660, 2022.
  * [39] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.  Vision meets robotics: The kitti dataset.  The International Journal of Robotics Research, 32(11):1231–1237, 2013.
  * [40] Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon.  3d packing for self-supervised monocular depth estimation.  In IEEE/CVF CVPR, pages 2485–2494, 2020.
  * [41] Vitor Guizilini, Rares Ambrus, Dian Chen, Sergey Zakharov, and Adrien Gaidon.  Multi-frame self-supervised depth with transformers.  In CVPR, 2022.
  * [42] Jinwoo Bae, Sungho Moon, and Sunghoon Im.  Deep digging into the generalization of self-supervised monocular depth estimation.  In AAAI, 2023.
  * [43] Aaqib Saeed, Flora D Salim, Tanir Ozcelebi, and Johan Lukkien.  Federated self-supervised learning of multisensor representations for embedded intelligence.  IEEE Internet of Things Journal, 8(2), 2020.
  * [44] Bram van Berlo, Aaqib Saeed, and Tanir Ozcelebi.  Towards federated unsupervised representation learning.  In EdgeSys, pages 31–36, 2020.
  * [45] Mykola Servetnyk, Carrson C Fung, and Zhu Han.  Unsupervised federated learning for unbalanced data.  In IEEE GLOBECOM, pages 1–6, 2020.
  * [46] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  University of Toronto, 05 2012.
  * [47] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [48] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In IEEE/CVF CVPR, 2017.
  * [49] Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab.  Deeper depth prediction with fully convolutional residual networks.  In IEEE 3DV, pages 239–248, 2016.
  * [50] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI. Springer, 2015.
  * [51] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe.  Unsupervised learning of depth and ego-motion from video.  In CVPR, 2017.
  * [52] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In IEEE/CVF CVPR, pages 770–778, 2016.
  * [53] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow.  Digging into self-supervised monocular depth estimation.  In CVPR, 2019.
  * [54] Sakshi Ahuja, Bijaya Ketan Panigrahi, Nilanjan Dey, Venkatesan Rajinikanth, and Tapan Kumar Gandhi.  Deep transfer learning-based automated detection of covid-19 from lung ct scan slices.  Applied Intelligence, 51(1), 2021.
  * [55] Farheen Ramzan, Muhammad Usman Ghani Khan, Asim Rehmat, Sajid Iqbal, Tanzila Saba, Amjad Rehman, and Zahid Mehmood.  A deep learning approach for automated diagnosis and multi-class classification of alzheimer’s disease stages using resting-state fmri and residual neural networks.  Journal of medical systems, 44(2):1–16, 2020.
  * [56] David Berga, Marc Masana, and Joost Van de Weijer.  Disentanglement of color and shape representations for continual learning.  arXiv preprint arXiv:2007.06356, 2020.
  * [57] Yevhen Kuznietsov, Marc Proesmans, and Luc Van Gool.  Comoda: Continuous monocular depth adaptation using past experiences.  In IEEE/CVF WACV, pages 2907–2917, 2021.
  * [58] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600–612, 2004.
  * [59] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black.  Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation.  In CVPR, 2019.
  * [60] Su Liu, Jiong Yu, Xiaoheng Deng, and Shaohua Wan.  Fedcpf: An efficient-communication federated learning approach for vehicular edge computing in 6g communication networks.  IEEE Transactions on Intelligent Transportation Systems, 23(2):1616–1629, 2021.