# Digging into contrastive learning for robust depth estimation with diffusion
models

Jiyuan Wang  0009-0001-2525-7473 Beijing Jiaotong UniversityBeijingChina
<EMAIL_ADDRESS>,  Chunyu Lin  Beijing Jiaotong UniversityBeijingChina
<EMAIL_ADDRESS>,  Lang Nie  Beijing Jiaotong UniversityBeijingChina
<EMAIL_ADDRESS>,  Kang Liao  Nanyang Technological
UniversitySingaporeSingapore<EMAIL_ADDRESS>,  Shuwei Shao  Beihang
UniversityBeijingChina<EMAIL_ADDRESS>and  Yao Zhao  Beijing Jiaotong
UniversityBeijingChina<EMAIL_ADDRESS>
(2022)

###### Abstract.

Recently, diffusion-based depth estimation methods have drawn widespread
attention due to their elegant denoising patterns and promising performance.
However, they are typically unreliable under adverse conditions prevalent in
real-world scenarios, such as rainy, snowy, etc. In this paper, we propose a
novel robust depth estimation method called D4RD, featuring a custom
contrastive learning mode tailored for diffusion models to mitigate
performance degradation in complex environments. Concretely, we integrate the
strength of knowledge distillation into contrastive learning, building the
‘trinity’ contrastive scheme. This scheme utilizes the sampled noise of the
forward diffusion process as a natural reference, guiding the predicted noise
in diverse scenes toward a more stable and precise optimum. Moreover, we
extend noise-level trinity to encompass more generic feature and image levels,
establishing a multi-level contrast to distribute the burden of robust
perception across the overall network. Before addressing complex scenarios, we
enhance the stability of the baseline diffusion model with three
straightforward yet effective improvements, which facilitate convergence and
remove depth outliers. Extensive experiments demonstrate that D4RD surpasses
existing state-of-the-art solutions on synthetic corruption datasets and real-
world weather conditions. The code for D4RD will be made available for further
exploration and adoption.

Depth Estimation, Robust Perception, Self-supervised Learning, Diffusion
Methods

††copyright: acmcopyright††journalyear: 2022††doi:
XXXXXXX.XXXXXXX††submissionid: 2846††conference: Conference; Some Days;
Anywhere on earth††booktitle: Proceedings of the 32th ACM International
Conference on Multimedia (MM ’24), October 29, 2024 – November 3, 2024,
Canada, Ottawa††price: 15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing
methodologies Reconstruction

##  1\. Introduction

Estimating accurate depth from a single image is highly attractive because
this task arises whenever the 3D structure is needed. Since the supervised
Monocular Depth Estimation (MDE) methods(Ranftl et al., 2020, 2021) require
high-cost ground truth (GT) depth labels. To address this limitation,
researchers have explored self-supervised approaches that leverage adjacent
frame pose information and photometric consistency from video sequences.
However, most existing self-supervised MDE models can only handle clear and
ideal settings but are highly unreliable under challenging conditions. This
limitation significantly hampers the practical applicability of these methods.

Recently, diffusion models have demonstrated their superiority in MDE(Ke et
al., 2024; Patni et al., 2024) with remarkable performance. Moreover, some
works from other domains(Liu et al., 2023; Nam et al., 2023) have revealed
that the denoising diffusion process allows the potential of robust
perception. These findings have inspired us to explore how diffusion models
can address the challenges faced in MDE and enhance robustness. To this end,
we propose Diffusion for(4) Robust Depth (D4RD), a novel diffusion-based MDE
framework. D4RD incorporates a new multi-level ‘trinity’ contrastive scheme
designed to enhance robustness and mitigate performance degradation in
challenging conditions.

Figure 1.  Comparisons with RMDE method(Wang et al., 2023b; Saunders et al.,
2023; Song et al., 2023) on KITTI and WeatherKITTI. The length of the Line
represents the performance degradation magnitude under adverse environments.
All the methods are trained on the same dataset WeatherKITTI for fairness.

To better understand the concept of the ‘trinity’, we need to revisit the
previous efforts aimed at resolving the challenges of Robust Monocular Depth
Estimation (RMDE). Generally, these approaches fall into two main categories:
contrastive learning-based alignment approaches(Song et al., 2023; Liu et al.,
2021; Zhao et al., 2022a; Wang et al., 2023b) and knowledge distillation-based
pseudo-supervised methods(Saunders et al., 2023; Gasperini et al., 2023; Mao
et al., 2024). Considering that corruption does not affect the physical scene
depth, the former approach typically involves the clear image along with its
various augmentations into the network, enforcing consistency among different
output depths to provide extra supervision. However, a common issue of these
methods is that directly maximizing the depth similarity can easily fall into
a collapsing solution(Chen and He, 2020). To address this issue, additional
guidance is required, which poses a challenge in a self-supervised manner
since the photometric consistency cannot provide a perfect convergence anchor.

The latter usually employs a pre-trained teacher model to estimate the depths
of clear images and uses these estimates as pseudo-labels to train the student
model on augmented images. Obviously, there is a performance upper bound for
the student model as the teacher model is not entirely accurate.

In short, we urgently need a perfect label that can not only serve as an
anchor for guiding optimization in contrastive learning but also eliminate
performance bottlenecks in knowledge distillation. This label seems impossible
in self-supervised learning but we ingeniously find it in the diffusion
process—the sampled noise. In the forward diffusion process, we obtain noisy
images by adding a sampled noise that follows a normal distribution to the
original image. During the training process, sampled noise is considered as
the GT label for supervising noise prediction. When the prediction is
accurate, the original image will be accurately restored(Ho et al., 2020; Song
et al., 2022). Therefore, we incorporate this noise as supplementary guidance
for contrastive learning to establish a ‘trinity’ contrast pattern.
Concretely, we maintain the original consistency constraints unchanged, but on
this basis, we apply sampled noise constraints to clear/augment input noise
predictions, gathering the pair of estimated noise towards a more stable and
precise optimum. If the model is completely robust and accurate, then the
three types of noise(i.e., clear/augmented/sampled noise) should be identical,
that is, the ‘trinity’.

Moreover, (Kalb and Beyerer, 2023) have found that the burden of robust dense
perception is mainly concentrated on a specific component of the network
(e.g., the lower layers), which limits the network potential. Based on this,
we intuitively try to distribute these responsibilities across the network by
extending the ‘trinity’ paradigm to more generic feature and image levels.
Naturally, finding suitable reference points like sampled noise is challenging
on these occasions, so we rely on clear scenes and teacher models to provide
guidance to the model. Despite that, we found our trinity is still better than
the traditional distillation and contrast methods. In addition, to further
explore how multi-level contrastive learning promotes network performance, we
visualize the perceptual results after each level. The improvement, as
depicted in Fig. 4, is likely attributed to each level addressing distinct
aspects of image enhancement, ultimately leading to more robust and accurate
depth results.

D4RD takes MonoDiffusion (Shao et al., 2023) as the baseline framework. Before
incorporating the multi-level trinity contrastive paradigm, we also enhance
the stability of the baseline with three simple but effective improvements. It
strengthens convergence and eliminates depth outliers, which lay the
foundation for handling complex scenarios. Finally, extensive experiments show
that our proposed method, D4RD, achieves SoTA performance among all RMDE
competitors across two syntheses dataset and five real weather conditions.
Moreover, as shown in Fig. 1, D4RD not only surpasses other methods with a
significant margin but also demonstrates the minimum performance degradation
in challenging conditions. In summary, the contributions of this paper are as
follows:

  * •

We enhance the diffusion-based depth estimation architecture with better
stability and convergence, and further establish a novel robust depth
estimation framework, named D4RD, with a customized contrastive learning
scheme for diffusion models.

  * •

Benefiting from the natural guidance of sampled noise, we ingeniously
integrate the strength of distillation learning into contrastive learning to
form a ‘trinity’ contrast pattern, which facilitates noise prediction accuracy
and robustness in D4RD.

  * •

The noise-level trinity is then expanded to more generic feature and image
levels, building the multi-level trinity scheme. It evenly distributes the
pressure of handling domain variances across different model components.

##  2\. Related Work

###  2.1. Monocular Depth Estimation

MDE aims to predict the accurate depth from a single image, which is an ill-
posed problem. Intuitively, this challenge was initially seen as a dense
regression task(Eigen et al., 2014; Eigen and Fergus, 2015) based on GT depth
labels. Four years later, Fu et al.(Fu et al., 2018) discovered that
reconstructing MDE into a depth bins classification task can significantly
improve performance. Still about four years later, as the diffusion model
showcases its talents in generative task(Ho et al., 2020; Song et al., 2022),
detection(Chen et al., 2022), and segmentation(Wang et al., 2023a), DDP(Ji et
al., 2023) first reformulates this task as a depth map denoising task, and
lead to giant progress again. Followers like Diffusiondepth(Duan et al.,
2023), DDVM(Saxena et al., 2023), VPD(Zhao et al., 2023), TAPD(Kondapaneni et
al., 2023), EcoDepth(Patni et al., 2024), Marigold(Ke et al., 2024), and
MonoDiffusion(Shao et al., 2023) all demonstrate the advantages of this
paradigm in various MDE sub-tasks. D4RD also adopts this promising scheme and
extends it on robust depth estimation.

###  2.2. Robust Depth Estimation

As mentioned before, although MDE is an ill-posed problem, with the
development of research and task optimization in the past decade, MDE on clear
data sets has achieved excellent performance. Therefore, for RMDE tasks, an
intuitive and effective method is to use the results of clear scenes, which
can be divided into two major categories, knowledge distillation and
contrastive learning.

Distillation-based methods focus on leveraging the well-predicted depth map
(typically estimated from clear scenes) as pseudo-labels to assist in training
the RMDE model. MD4all(Gasperini et al., 2023) trained a base model on sunny
scenes first, and then employed it as a teacher to train another network for
weather scenarios with a specifically designed loss function. SSD(Mao et al.,
2024) may be the most similar work to ours because it also takes diffusion to
solve RMDE tasks. However, it follows the distillation paradigm of MD4all and
introduces an enhanced dataset generated by diffusion, even altering the scene
itself when rendering weather or night enhancements.

Contrastive-based methods typically improve the model’s robustness by ensuring
consistency between the depth predictions of a clear image and its augmented
versions. Robust-Depth(Saunders et al., 2023) introduces the semi-augmented
warp and bi-directional contrast, which ensures the consistency assumption and
improve the accuracy. WeatherDepth considers the domain gap between clear
scenes and complex scenes and solves this problem by applying a gradual
adaptation scheme based on curriculum contrastive learning. Additionally, EC-
Depth(Song et al., 2023) combines contrast and distillation with double-stage
training. The first stage uses contrast, creating a KL divergence-like
contrast to align the depth of harsh scenes, and the second stage uses
distillation, introducing EMA for teacher-student joint learning.

##  3\. Method

In this section, we will introduce the foundational knowledge of the self-
supervised MDE and diffusion-based MDE (Sec. 3.1), the improvements to enhance
the stability of the baseline (Sec. 3.2), the ‘trinity’ contrast paradigm
(Sec. 3.3), the multi-level contrasting scheme (Sec. 3.4), and the two-stage
training strategy (Sec. 3.5). An overview of the whole framework is shown in
Fig. 2.

Figure 2.  The training framework of D4RD is depicted at the upper part of the
image. Below that, multi-level trinity learning (i.e. images, depth features,
and noise prediction) are presented through (a), (b), and (c), respectively.

###  3.1. Preliminaries

Self-Supervised MDE uses an auxiliary image $\textbf{I}_{a}$ from the
neighboring frame to constrains the output depth with a view-synthesis
training pattern. Denoting the MDE model as
$\mathcal{F}:\textbf{I}\rightarrow\textbf{d}{\in}\mathbb{R}^{W\times H}$, we
can reach the warped target frame $\textbf{I}_{a\rightarrow t}$ with:

(1) $\textbf{I}_{a\rightarrow
t}=\textbf{I}_{a}\left\langle\operatorname{proj}\left(\textbf{d},\mathcal{T}_{t\rightarrow
a},K\right)\right\rangle,$

where$\mathcal{T}_{t\rightarrow a}$ denotes the relative camera poses obtained
from the pose network, and K denotes the camera intrinsics. Afterwards, we can
compute the photometric reconstruction loss between I and
$\textbf{I}_{a\rightarrow t}$ to constrain the depth:

(2)
$L_{ph}=\omega_{1}\frac{1-\operatorname{SSIM}\left(\textbf{I},\textbf{I}_{a\rightarrow
t}\right)}{2}+\omega_{2}\left\|\textbf{I}-\textbf{I}_{a\rightarrow
t}\right\|.$

In our implementation, DR4D takes the semi-augmented synthesis from Robust-
Depth(Saunders et al., 2023) to replace Eq. 1 with:

(3) $\textbf{I}_{a\rightarrow
t}^{\prime}=\textbf{I}_{a}\left\langle\operatorname{proj}\left(\textbf{d}_{aug},T_{t\rightarrow
a},K\right)\right\rangle,$

where $\textbf{d}_{aug}$ is the depth map inferred form the augmented images
and $\textbf{I}_{a\rightarrow t}^{\prime}$ will replace
$\textbf{I}_{a\rightarrow t}$ in Eq. 2.

Diffusion-based MDE redefines the depth estimation task as a conditional
denoising process. Given the image $\textbf{I}\ {\in}\ \mathbb{R}^{W\times
H\times 3}$ and the corresponding depth distribution $D$, we can add the
Gaussian noise $\epsilon$ to get the noisy sample $\textbf{d}_{\tau}$:

(4)
${\textbf{d}_{\tau}}=\sqrt{{{\overline{\alpha}}_{\tau}}}{D}+\sqrt{1-{{\overline{\alpha}}_{\tau}}}\epsilon,\quad\epsilon\sim\mathcal{N}\left({0,I}\right),$

and iteratively reverse it by removing the predict-added noise
$\epsilon_{\theta}$:

(5)
${\epsilon_{\theta}}\left({{\textbf{d}_{\tau-1}}\left|{{\textbf{d}_{\tau}},\textbf{{c}}}\right.}\right):=\mathcal{N}\left({{{\mu_{\theta}}}\left({{\textbf{d}_{\tau}},\tau,\textbf{c}}\right),\sigma_{\tau}^{2}\textbf{n}}\right),\textbf{c}=\mathcal{F}_{f}(\textbf{I}).$

Here $\tau$ consists of $\mathbb{T}$ steps,
$\bar{\alpha}_{\tau}:=\prod_{s=1}^{\tau}{1\\!-\\!\beta_{s}}$, and
$\\{\beta_{1},\ldots,\beta_{\mathbb{T}}\\}$ is the variance noise schedule for
$\mathbb{T}$ steps (Ho et al., 2020). The $\mathcal{N}(\cdot;\cdot)$ stands
for gaussian sampling, $\mathcal{F}_{f}(\cdot)$ is a feature extractor, and
$\sigma_{\tau}^{2}\textbf{n}$ is typically set to 0 to make the estimation
process deterministic.

Obviously, if $\epsilon_{\theta}$ is completely consistent with the
$\epsilon$, we can reverse the depth result precisely, so we have the
classical training loss:

(6)
$L_{ddim}=\mathbb{E}_{\textbf{d}_{0},\epsilon\sim\mathcal{N}(0,1),\tau\sim\mathcal{U}(T)}\left\|\epsilon-{\epsilon_{\theta}}\left({{\textbf{d}_{\tau-1}}\left|{{\textbf{d}_{\tau}},\textbf{{c}}}\right.}\right)\right\|^{2}_{2},$

where $\mathcal{U}(\cdot)$ is the uniformly sampling. For self-supervised
learning, D4RD adopts the pseudo-GT diffusion as used in MonoDiffusion (Shao
et al., 2023), which introduces a pre-trained model $\mathcal{F}_{T}$ to infer
the pseudo-depth $\textbf{d}_{T}$ and it will replace $D$ in Eq. 4.

###  3.2. Stable Depth with Diffusion Models

Pseudo-depth knowledge distillation. To improve the performance,
MonoDiffusion(Shao et al., 2023) adopted a weighted $L_{1}$ loss as the
distillation constraint on the predicted depth. The weights are calculated
based on the photometric difference using a fixed threshold. However, such a
fixed threshold may stick the network to a performance bottleneck. To this
end, we replace the fixed filter with an adaptive one:

(7) $M=\left[\min_{a}ph\left(\textbf{I},\textbf{I}_{a\rightarrow
t}^{T}\right)<\frac{\lambda}{\text{ epoch }}\right],$

where $\lambda$ is a constant set to 1.5, and $\textbf{I}_{a\rightarrow
t}^{T}$ is the warped target image using $\textbf{d}_{T}$. This dynamic weight
initially enables D4RD to converge with the entire depth map and subsequently
filters out the less accurate regions to reduce the negative effects brought
by inaccurate pseudo-depth labels. Moreover, we find that BerHu loss(Laina et
al., 2016) yields a lower error than the $L_{1}$ loss. It imposes a greater
penalty on pixels with higher errors using $L_{2}$ loss, while maintaining the
advantage of L1 loss at small errors. In short, we conclude with a dynamically
weighted BerHu loss as our constraint for knowledge distillation:

(8) $L_{\text{dis }}=M\odot L_{\text{berhu
}}(\textbf{d}_{S,aug},\textbf{d}_{T,clr}),$

where $\textbf{d}_{T,clr}$ and $\textbf{d}_{S,aug}$ denote the predicted
result of the clear or augmented images from $\mathcal{F}_{T}$ and
$\mathcal{F}_{S}$, respectively.

Outlier depth removal. As mentioned before, since the iterative denoising
process may produce unnatural depths (e.g., negative values), previous
methods(Ke et al., 2024) intend to implement diffusion in the latent space:

(9)
${\textbf{d}_{\tau}}=\sqrt{{{\overline{\alpha}}_{\tau}}}\mathcal{V}(\textbf{d})+\sqrt{1-{{\overline{\alpha}}_{\tau}}}\epsilon,\textbf{d}=\mathcal{V}^{-1}(\textbf{d}_{0}),$

where $\mathcal{V}(\cdot)$ is a pre-trained variational-auto-encoder (VAE) and
$\mathcal{V}^{-1}(\cdot)$ is the reverse decoder. This manner undoubtedly
introduces extra calculations. In this paper, we simplify the above solution
through a non-linear activation function (and its inverse):

(10)
$\textbf{d}_{\tau}=\sqrt{{{\overline{\alpha}}_{\tau}}}\textbf{sigmoid}^{-1}(\textbf{d}_{T})+\sqrt{1-{{\overline{\alpha}}_{\tau}}}\epsilon,\textbf{d}=\textbf{sigmoid}(\textbf{d}_{0}),$

where $\textbf{sigmoid}(\cdot)$ will transform the outputs into a limited
range $[0,1]$ and our experiments show that this simple manner can effectively
remove the depth outliers. Eventually, D4RD adopts Eq. 10 in the denoising
manner.

Feature-image joint condition. Our experiments have revealed that using the
concatenation of the input image and its depth feature as the diffusion
condition can bring better performance. It can be elaborated as:

(11)
$\textbf{c}=\textbf{I}\oplus\mathcal{F}_{f}(\textbf{I}),\quad\mathcal{F}_{f}(\textbf{I})\in\mathbb{R}^{1\times
H\times W},$

where $\oplus$ represents the channel-dim concatenation. We visualize the
feature map in Fig. 4 to analyze the reasons and reveal that
$\mathcal{F}(\textbf{I})$ is a coarse ‘depth map’ with some additional
incorrect edges. In other words, $\mathcal{F}_{f}$ focuses solely on depth-
relevant information, and the combination in Eq. 11 provides higher-level
contextual and semantic information, which helps the diffusion architecture in
understanding the scenes, predicting noises, and ultimately generating better
estimations.

###  3.3. Trinity Contrastive learning

Figure 3.  Visual comparisons among three types of robust learning methods.
Our trinity contrast method is more close to our expected effect and achieves
better actual performance.

The proposed trinity contrastive scheme integrates both knowledge distillation
and contrastive learning. For the first one, the general pattern is:

(12)
$L_{dis}=|\mathcal{F}_{S}(\textbf{I}_{aug})-\underline{\mathcal{F}_{T}(\textbf{I})}|,$

where $\underline{\text{underline}}$ denotes the predicted from a frozen
model. However, Fig. 3(a) has shown that this strategy struggles to teach
$\mathcal{F}_{S}$ when the augmented image $\textbf{I}_{aug}$ has a
significant domain difference. Moreover, there is an evident performance upper
bound for $\mathcal{F}_{S}$.
For the latter, the usual form is:

(13) $L_{cst}=|\mathcal{F}(\textbf{I})-\mathcal{F}(\textbf{I}_{aug})|.$

As shown in Fig. 3 (b), without further guidance, simply gathering the depth
map can easily lead to network collapse (Chen and He, 2020). The self-
supervised methods (Zhao et al., 2022a; Saunders et al., 2023) solve this by
providing additional photometric guidance, but this guidance is not accurate
enough and requires twice the training time (Eq.2 is calculated twice for
$\textbf{I}_{a\rightarrow t}$ and $\textbf{I}_{a\rightarrow t}^{\prime}$
separately).

In diffusion models, we discovered that in Eq. 4, the added noise $\epsilon$,
which is sampled from $\mathcal{N}\left({0,I}\right)$, can serve as a perfect
guiding anchor for the estimated noise $\epsilon_{\theta}$ in Eq. 6 and do not
require any extra annotation. Let $\epsilon_{\theta}$ and
$\epsilon_{\theta,\text{aug }}$ represent the noise prediction for the clear
image I and augment image $\textbf{I}_{aug}$. Once these noises are identical
and accurate, the final denoising depth will demonstrate consistency and
robustness. Hence, as shown in Fig. 2(c), we reformulate a novel noise-level
trinity contrast scheme as:

(14) $L_{\text{nis}}=\eta_{1}\|\epsilon_{\theta,\text{aug
}}-\epsilon_{\theta}\|_{2}^{2}+\\\ \eta_{2}(\|\epsilon_{\theta,\text{aug
}}-\epsilon\|_{2}^{2}+\|\epsilon_{\theta}-\epsilon\|_{2}^{2}),$

where $\eta_{1}$ and $\eta_{2}$ are set to 0.5 and 1, respectively . With the
natural and perfect sampled noise label, the proposed trinity can stably
gather the pair of estimated noises toward an optimal anchor position. We also
experimentally prove its superiority to the other two traditional strategies
in Section 5.

###  3.4. Multi-level Contrast

Figure 4. Visual results of each level trinity. Compared to the origin image,
as shown in the highlight regions in the red dashed rectangle, the image level
trinity can assist in handling water surface artifacts and ground snow. The
feature level trinity builds a coarse depth map with some wrong edges but the
noise trinity fixes them all. Better viewed when zooming in.

Prior studies(Kalb and Beyerer, 2023) have revealed that the burden of
handling complex conditions mainly concentrates on a specific part of the
network. Such an unbalanced burden may hinder the potential of robust
algorithms. To this end, as shown in Fig. 2, we further extend the noise-level
‘trinity’ approach to more generic targets (e.g., feature and image levels),
through which we distribute the responsibility of robust perception across the
whole network.

Feature level. We first extend it to the depth feature level, which exploits
the robust ability of feature network $\mathcal{F}$. However, at this time, we
lack the guidance as perfect as the added noise. Instead, as depicted in Fig.
2 (b), we adopt a teacher model $\mathcal{F}_{T}$ to get the suboptimal
guiding label and build the feature-level ‘trinity’ loss as:

(15)
$\begin{array}[]{l}L_{feat}=\omega_{1}\|\mathcal{F}_{S}(\textbf{I})-\mathcal{F}_{S}(\textbf{I}_{aug})\|_{1}+\\\
\omega_{2}(\|\mathcal{F}_{T}(\textbf{I})-\mathcal{F}_{S}(\textbf{I})\|_{1}+\|\mathcal{F}_{T}(\textbf{I})-\mathcal{F}_{S}(\textbf{I}_{aug})\|_{1}),\end{array}$

where $\omega_{1}$ and $\omega_{2}$ are empirically set to 1 and 0.5,
respectively.

Image level: In addition to noise and feature, we also implement an image-
level trinity contrast by setting the guiding label as the clear image I. As
depicted in Fig. 2(a), we design a simple CNN network that performs down-up
sampling (denoted as $\mathcal{F}_{R}$) to get the enhanced image via:

(16) $\left\\{\begin{array}[]{l}\textbf{I}_{\text{aug
}}^{\prime}=\mathcal{F}_{R}\left(\textbf{I}_{\text{aug
}}\right)+\textbf{I}_{\text{aug }}\\\
\textbf{I}^{\prime}=\mathcal{F}_{R}(\textbf{I})+\textbf{I}\end{array}\right..$

Then the image-level trinity is formulated as:

(17) $L_{img}=\beta_{1}(\|\textbf{I}_{\text{aug
}}^{\prime}-\textbf{I}\|_{1}+\|\textbf{I}^{\prime}-\textbf{I}\|_{1})+\beta_{2}\|\textbf{I}_{\text{aug
}}^{\prime}-\textbf{I}^{\prime}\|_{1},$

where the values of $\beta_{1}$ and $\beta_{2}$ are same as those of
$\omega_{1}$ and $\omega_{2}$.

As shown in Fig. 4, through three trinity contrasts from different levels, the
stresses of robust perception are evenly distributed among three parts of the
network. They encourage the model to recognize and resist various types of
scene degeneration at multiple stages, thus boosting the overall prediction
reliability.

Table 1. Quantitative results on WeatherKITTI and KITTI dataset. For the
error-based metrics, the lower value is better; and for the accuracy-based
metrics, the higher value is better. The best and second best results are
marked in bold and underline.

| WeatherKITTI | KITTI
---|---|---
Method | AbsRel | SqRel | RMSE | RMSElog | $a_{1}$ | $a_{2}$ | $a_{3}$ | AbsRel | SqRel | RMSE | RMSElog | $a_{1}$ | $a_{2}$ | $a_{3}$
MonoDiffusion(Shao et al., 2023) | 0.133 | 0.960 | 5.212 | 0.213 | 0.828 | 0.945 | 0.977 | 0.103 | 0.718 | 4.411 | 0.178 | 0.894 | 0.965 | 0.984
MonoViT(Zhao et al., 2022b) | 0.120 | 0.899 | 5.111 | 0.200 | 0.857 | 0.953 | 0.980 | 0.099 | 0.708 | 4.372 | 0.175 | 0.900 | 0.967 | 0.984
Robust-Depth(Saunders et al., 2023) | 0.107 | 0.791 | 4.604 | 0.183 | 0.883 | 0.963 | 0.983 | 0.100 | 0.747 | 4.455 | 0.177 | 0.895 | 0.966 | 0.984
EC-Depth(Song et al., 2023) | 0.110 | 0.790 | 4.744 | 0.185 | 0.875 | 0.960 | 0.983 | 0.100 | 0.689 | 4.315 | 0.173 | 0.896 | 0.967 | 0.985
Robust-Depth*(Saunders et al., 2023) | 0.103 | 0.806 | 4.517 | 0.180 | 0.894 | 0.964 | 0.983 | 0.100 | 0.776 | 4.440 | 0.177 | 0.900 | 0.965 | 0.983
WeatherDepth(Wang et al., 2023b) | 0.103 | 0.738 | 4.414 | 0.178 | 0.892 | 0.965 | 0.984 | 0.099 | 0.698 | 4.330 | 0.174 | 0.897 | 0.967 | 0.984
EC-Depth*(Song et al., 2023) | 0.102 | 0.762 | 4.400 | 0.177 | 0.895 | 0.967 | 0.984 | 0.098 | 0.732 | 4.326 | 0.174 | 0.902 | 0.968 | 0.984
D4RD | 0.099 | 0.688 | 4.377 | 0.174 | 0.897 | 0.966 | 0.984 | 0.097 | 0.665 | 4.312 | 0.171 | 0.900 | 0.967 | 0.985

Figure 5. Qualitative results for WeatherKITTI dataset. We compare D4RD with
the current SoTA RMDE methods in the adverse rain and snow subsets. The part
marked with ‘Clear’ is the correspondence sunny image (processed for more
clarity). Regions with prominent differences are highlighted using dashed
boxes.

###  3.5. Two stage training

Motivations. We implement a two-step training strategy for two reasons. First,
the proposed multi-level trinity contrastive paradigm requires the condition
labels from the teacher model, but it does not exist initially. On the other
hand, the teacher model $\mathcal{F}_{T}$ from the second stage can integrate
the historical knowledge of stage one, yielding more accuracy and reliable
pseudo-depth.

The first training phase. As shown in Fig. 2, We feed the clear image I and
its augmentation $\textbf{I}_{aug}$ as a mini-batch to the model and obtain
their depth feature maps. Then we concat them with their corresponding images
with Eq. 11 to get the $\textbf{c}_{aug}$ and c. Next, both of them will be
delivered to diffusion UNet to predict
$\epsilon_{\theta}$/$\epsilon_{\theta,\text{aug }}$ and calculate the noise-
level trinity loss $L_{nis}$. Note only $\epsilon_{\theta,\text{aug}}$ will be
used for the denoising process and $L_{dis}$, $L_{ph}$ will be imposed on d,
which implies the proposed trinity will not double the computational burden as
the traditional contrastive learning. With an extra edge-aware constraint
$L_{e}$ from (Godard et al., 2019), we finally conclude the total loss at the
first stage:

(18) $L_{stage1}=L_{nis}+L_{dis}+L_{ph}+\rho L_{e},$

where $\rho$ is empirically set to 1e-3.

The second training phase. Building upon the optimization objective of stage
one, we add the $L_{feat}$ and $L_{img}$ to build the multi-level contrasts as
mentioned in Section 3.4. The total loss of the second stage can be written
as:

(19) $L_{stage2}=L_{nis}+L_{dis}+L_{ph}+\rho L_{e}+\theta(L_{feat}+L_{img}),$

where $\theta$ is empirically set to 0.5.

Figure 6. Qualitative results for real weather dataset. Besides the RMDE
methods, we also compare D4RD with the standard method MonoViT(Zhao et al.,
2022b) in snowy and rainy environments to show our method’s practicality. The
part marked with Magnification is processed for more clarity.

##  4\. Experiment

###  4.1. Dataset

WeatherKITTI(Wang et al., 2023b) is built on the KITTI dataset and includes 6
kinds of weather augmentation. We train D4RD on this dataset because, compared
with other synthetic datasets(Saunders et al., 2023; Mao et al., 2024), it is
more realistic in simulating real-world complex scenarios. WeatherKITTI shares
the same train/validate/test splits as KITTI. Therefore, for training, we
follow Zhou’s split(Zhou et al., 2017) (containing 19,905 training images and
2,212 validation images), and for testing, we adopt Eigen’s spilt(Eigen et
al., 2014) with 697 test images.

KITTI-C(Kong et al., 2023) is a synthetic dataset building for the RoboDepth
Competition. In total, the benchmark contains 18 types of perturbations with 5
levels of corruption magnitudes, most of which are stronger than WeatherKITTI.
We use it to fairly compare D4RD with their competition champions(Song et al.,
2023) to further demonstrate our robust performance.

DrivingStereo(Yang et al., 2019) is a large-scale real-world dataset. It
contains four subsets of images under four weather conditions (i.e., fog,
cloudy, rainy, and sunny), with each subset containing 500 images. We test
RMDE models on this dataset to characterize their ability to handle complex
real-world conditions.

Dense(Bijelic et al., 2020) is one of the few real snowy datasets and it has
approximately 1,500 snowy images in total. We introduce this benchmark to test
the snowy scenes. To ensure consistency with the above dataset, we followed
(Wang et al., 2023b) and sampled 1 image every 3 to obtain 500 images as test
data. To generate accurate depth GT in challenging snowy conditions, we use
the DROR(Charron et al., 2018) algorithm to remove inaccurate depth readings
from LiDARs affected by snowflakes. In addition, a small portion of distorted
areas has been trimmed, resulting in a final resolution of $1880\times 924$.
We will explain the reasons for not using the snowy part of CADC dataset in
the supplementary materials.

###  4.2. Implementation Details

In this paper, D4RD is based on MonoDiffusion(Shao et al., 2023), which is
currently the only diffusion-based self-supervised MDE pipeline that combines
the diffusion architecture with Lite-Mono(Zhang et al., 2023). To maintain
consistency with previous RMDE researches(Saunders et al., 2023; Wang et al.,
2023b; Song et al., 2023), we replace Lite-Mono with MonoViT(Zhao et al.,
2022b) in our D4RD.

D4RD is implemented in PyTorch and trained on a single NVIDIA RTX 3090. We use
the AdamW optimizer(Loshchilov and Hutter, 2019) and other similar
hyperparameters as MonoViT. The batch size is set to 6, and the training
process runs 30 epochs in each stage. Considering the resolution difference
across the datasets, we resize them to $640\times 192$ for both training and
testing. For the diffusion process, we follow MonoDiffusion(Shao et al., 2023)
and set it with 1000 training steps and 20 inference steps.

For fair comparisons, we retrained Robost-Depth(Saunders et al., 2023) and EC-
Depth(Song et al., 2023) on the WeatherKITTI dataset. Furthermore, to
demonstrate D4RD’s ability to handle robust scenes, we also retrain our model
with the same training data used by (Song et al., 2023), named D4RD†, to
compare the performance on the KITTI-C dataset.

###  4.3. Quantitative and Qualitative Comparison

Here, we adopt the same evaluation metrics and errors up to 80m for all
comparisons as (Godard et al., 2019). Please refer to the supplementary for
more details. The comparative methods include the baseline MonoDiffusion,
traditional RMDE models’ baseline MonoViT(Zhao et al., 2022b), previous SoTA
RMDE models WeatherDepth(Saunders et al., 2023), EC-Depth(Song et al., 2023),
Robust-Depth(Saunders et al., 2023), and their retained models. Besides, (Wang
et al., 2023b) and (Wang et al., 2023b) apply their approaches to various
baselines, we want to clarify that we choose the one that uses MonoViT(Zhao et
al., 2022b). In (Song et al., 2023), they take a two-stage training pipeline
and we select the second stage model for the comparison.

WeatherKITTI result. We show detailed comparative experiments on the KITTI and
WeatherKITTI datasets in Table 1, where our method outperforms other SoTA
methods in all metrics by a large margin. Specifically, D4RD demonstrates a
25.6% and 17.5% decrease in AbsRel errors compared to the
baselines(MonoDiffusion and MonoViT) under challenging conditions
(WeatherKITTI). Furthermore, its performance either exceeds or closely matches
that of these baselines in standard conditions (KITTI). The qualitative
results are shown in Fig. 5. From the regions highlighted in green boxes, the
compared schemes are affected by weather particle occlusion and blurs. In
contrast, D4RD can robustly mine the consistent depth plane of the truck body
and accurately estimate the correct contour of the road sign.

Table 2. Zero-shot evaluation on the Dense and DrivingStereo dataset.

Method | AbsRel | SqRel | RMSE | RMSElog | $a_{1}$ | $a_{2}$ | $a_{3}$
---|---|---|---|---|---|---|---
(a)DrivingStereo Sunny
MonoDiffusion(Shao et al., 2023) | 0.162 | 1.738 | 7.535 | 0.220 | 0.799 | 0.936 | 0.975
MonoViT(Zhao et al., 2022b) | 0.150 | 1.615 | 7.657 | 0.211 | 0.815 | 0.943 | 0.979
Robust-Depth(Saunders et al., 2023) | 0.152 | 1.574 | 7.293 | 0.210 | 0.812 | 0.944 | 0.979
EC-Depth(Song et al., 2023) | 0.151 | 1.436 | 7.213 | 0.209 | 0.808 | 0.944 | 0.979
Robust-Depth*(Saunders et al., 2023) | 0.152 | 1.651 | 7.369 | 0.212 | 0.810 | 0.944 | 0.978
WeatherDepth(Wang et al., 2023b) | 0.155 | 1.562 | 7.356 | 0.213 | 0.803 | 0.941 | 0.978
EC-Depth*(Song et al., 2023) | 0.155 | 1.538 | 7.301 | 0.211 | 0.804 | 0.941 | 0.980
D4RD | 0.149 | 1.437 | 7.121 | 0.207 | 0.815 | 0.946 | 0.980
(b)DrivingStereo Cloudy
MonoDiffusion(Shao et al., 2023) | 0.155 | 1.853 | 7.822 | 0.214 | 0.807 | 0.939 | 0.978
MonoViT(Zhao et al., 2022b) | 0.141 | 1.626 | 7.550 | 0.201 | 0.831 | 0.948 | 0.981
Robust-Depth(Saunders et al., 2023) | 0.148 | 1.781 | 7.472 | 0.204 | 0.825 | 0.947 | 0.981
EC-Depth(Song et al., 2023) | 0.147 | 1.561 | 7.301 | 0.201 | 0.825 | 0.947 | 0.983
Robust-Depth*(Saunders et al., 2023) | 0.147 | 1.749 | 7.486 | 0.205 | 0.823 | 0.946 | 0.982
WeatherDepth(Wang et al., 2023b) | 0.144 | 1.549 | 7.349 | 0.201 | 0.822 | 0.949 | 0.983
EC-Depth*(Song et al., 2023) | 0.150 | 1.655 | 7.517 | 0.204 | 0.819 | 0.945 | 0.982
D4RD | 0.141 | 1.560 | 7.271 | 0.198 | 0.830 | 0.948 | 0.983
(c)DrivingStereo Rainy
MonoDiffusion(Shao et al., 2023) | 0.196 | 2.629 | 10.546 | 0.254 | 0.691 | 0.905 | 0.973
MonoViT(Zhao et al., 2022b) | 0.175 | 2.136 | 9.618 | 0.232 | 0.730 | 0.931 | 0.979
Robust-Depth(Saunders et al., 2023) | 0.167 | 2.019 | 9.157 | 0.221 | 0.755 | 0.938 | 0.982
EC-Depth(Song et al., 2023) | 0.162 | 1.746 | 8.538 | 0.212 | 0.755 | 0.947 | 0.986
Robust-Depth*(Saunders et al., 2023) | 0.173 | 2.154 | 9.452 | 0.226 | 0.733 | 0.934 | 0.982
WeatherDepth(Wang et al., 2023b) | 0.158 | 1.833 | 8.837 | 0.211 | 0.764 | 0.945 | 0.985
EC-Depth*(Song et al., 2023) | 0.162 | 1.810 | 8.792 | 0.215 | 0.744 | 0.943 | 0.985
D4RD | 0.158 | 1.722 | 8.584 | 0.208 | 0.773 | 0.946 | 0.985
(d)DrivingStereo Foggy
MonoDiffusion(Shao et al., 2023) | 0.128 | 1.540 | 8.687 | 0.191 | 0.831 | 0.955 | 0.986
MonoViT(Zhao et al., 2022b) | 0.109 | 1.204 | 7.760 | 0.167 | 0.870 | 0.967 | 0.990
Robust-Depth(Saunders et al., 2023) | 0.105 | 1.132 | 7.273 | 0.158 | 0.882 | 0.974 | 0.992
EC-Depth(Song et al., 2023) | 0.105 | 1.061 | 7.121 | 0.155 | 0.880 | 0.974 | 0.994
Robust-Depth*(Saunders et al., 2023) | 0.111 | 1.240 | 7.536 | 0.163 | 0.873 | 0.971 | 0.992
WeatherDepth(Wang et al., 2023b) | 0.110 | 1.195 | 7.323 | 0.160 | 0.878 | 0.973 | 0.992
EC-Depth* (Song et al., 2023) | 0.111 | 1.177 | 7.315 | 0.160 | 0.870 | 0.974 | 0.993
D4RD | 0.105 | 1.061 | 7.102 | 0.154 | 0.883 | 0.975 | 0.994
(e)Dense Snowy
MonoDiffusion(Shao et al., 2023) | 0.173 | 2.169 | 10.029 | 0.273 | 0.744 | 0.898 | 0.955
MonoViT(Zhao et al., 2022b) | 0.162 | 2.063 | 9.787 | 0.262 | 0.762 | 0.904 | 0.960
Robust-Depth(Saunders et al., 2023) | 0.157 | 1.992 | 8.945 | 0.240 | 0.786 | 0.923 | 0.971
EC-Depth(Song et al., 2023) | 0.155 | 1.866 | 8.828 | 0.237 | 0.780 | 0.922 | 0.972
Robust-Depth*(Saunders et al., 2023) | 0.157 | 2.050 | 8.951 | 0.243 | 0.785 | 0.921 | 0.969
WeatherDepth(Wang et al., 2023b) | 0.157 | 2.000 | 9.021 | 0.243 | 0.781 | 0.919 | 0.968
EC-Depth*(Song et al., 2023) | 0.154 | 1.984 | 8.806 | 0.235 | 0.782 | 0.923 | 0.972
D4RD | 0.154 | 1.868 | 8.780 | 0.236 | 0.789 | 0.925 | 0.972
(f)Average
MonoDiffusion(Shao et al., 2023) | 0.163 | 1.986 | 8.924 | 0.230 | 0.774 | 0.927 | 0.973
MonoViT(Zhao et al., 2022b) | 0.147 | 1.729 | 8.474 | 0.215 | 0.802 | 0.939 | 0.978
Robust-Depth(Saunders et al., 2023) | 0.146 | 1.700 | 8.028 | 0.207 | 0.812 | 0.945 | 0.981
EC-Depth(Song et al., 2023) | 0.144 | 1.534 | 7.800 | 0.203 | 0.810 | 0.947 | 0.983
Robust-Depth*(Saunders et al., 2023) | 0.148 | 1.769 | 8.159 | 0.210 | 0.805 | 0.943 | 0.981
WeatherDepth(Wang et al., 2023b) | 0.145 | 1.628 | 7.977 | 0.206 | 0.810 | 0.945 | 0.981
EC-Depth*(Song et al., 2023) | 0.146 | 1.633 | 7.946 | 0.205 | 0.804 | 0.945 | 0.983
D4RD | 0.141 | 1.530 | 7.772 | 0.201 | 0.818 | 0.948 | 0.983

Table 3. Evaluation on the KITTI-C dataset.

Method | AbsRel | SqRel | RMSE | RMSElog | $a_{1}$ | $a_{2}$ | $a_{3}$
---|---|---|---|---|---|---|---
MonoViT(Zhao et al., 2022b) | 0.161 | 1.292 | 6.029 | 0.247 | 0.768 | 0.915 | 0.964
Robust-Depth(Saunders et al., 2023) | 0.123 | 0.957 | 5.093 | 0.202 | 0.851 | 0.951 | 0.979
EC-Depth(Song et al., 2023) | 0.111 | 0.807 | 4.561 | 0.185 | 0.874 | 0.960 | 0.983
D4RD† | 0.108 | 0.778 | 4.652 | 0.183 | 0.880 | 0.961 | 0.983

Real Weather Scenes Results. To validate the practicality of D4RD, we conduct
zero-shot evaluations on real-world challenging datasets. As shown in Table 2
(a), D4RD still exhibits the best performance on the out-of-distribution clear
dataset, demonstrating that our model has a stronger zero-shot ability than
previous RMDE methods. Subsequently, we test the robustness of our model using
the most common real-world changing conditions, including cloudy, rainy,
foggy, and snowy scenes. As Table 2 shows, D4RD consistently demonstrates the
SoTA performance in every scenario (b-e), and (f) further reports that our
average performance has improved by a level compared to other methods in
AbsRel( 1.41 to around 1.45), which strongly proves the superiority and
practicality of our method. Fig. 6 displays the qualitative results of our
method on real rainy and snowy environments. It can be observed that our
method distinguishes the snow scene from the white bus (its windows even
reflect some snow scenes) and better identifies artifacts caused by water
surface reflections.

KITT-C Result. In addition to dealing with weather conditions, another role of
RMDE is to resist image corruption and perturbations. The first track of
RoboDepth competitions(Kong et al., 2023) focused on solving this problem. The
solution from EC-Depth(Song et al., 2023) won 1st place among dozens of
participants and Table 3 shows its giant improvement from MonoViT. However,
D4RD† further defeats the EC-Depth solution by simply replacing the training
dataset with EC-Depth’s dataset, which powerfully proves that D4RD is
currently the best RMDE framework.

###  4.4. Ablation Study

In this section, we conduct detailed ablation studies on WeatherKITTI and
real-world benchmarks to demonstrate the effectiveness of the proposed
components.

Table 4. Abladation study on stable depth with diffusion. All experiments are
implemented in stage one and the meaning of the abbreviation is as follows.
PDE: Pseudo-depth distillation enhancement; ODR: Outlier depth removing; FIC:
Feature-image joint condition.

Benchmark | ID | PDE | ODR | FIC | Trinity | AbsRel | SqRel | RMSE | RMSElog | $a_{1}$ | $a_{2}$ | $a_{3}$
---|---|---|---|---|---|---|---|---|---|---|---|---
WeatherKITTI | 1 |  |  |  |  | 0.106 | 0.780 | 4.548 | 0.181 | 0.888 | 0.964 | 0.983
2 | $\checkmark$ |  |  |  | 0.104 | 0.775 | 4.502 | 0.179 | 0.892 | 0.965 | 0.983
3 | $\checkmark$ | $\checkmark$ |  |  | 0.103 | 0.708 | 4.433 | 0.178 | 0.891 | 0.965 | 0.984
4 | $\checkmark$ | $\checkmark$ | $\checkmark$ |  | 0.101 | 0.712 | 4.423 | 0.177 | 0.895 | 0.965 | 0.984
5 | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 0.100 | 0.696 | 4.375 | 0.175 | 0.896 | 0.966 | 0.984

Table 5. Abladation study on contrast manner and level. All experiments are
implemented in stage second and the meaning of the abbreviation is as follows.
NIS: the noise level contrast; DF: the depth feature level contrast; IMG: the
image level contrast. Real Weather shows the average test result across all
kinds of real weather conditions in Tab.2

Benchmark | ID | NIS | DF | IMG | Method | AbsRel | SqRel | RMSE | RMSElog | $a_{1}$ | $a_{2}$ | $a_{3}$
---|---|---|---|---|---|---|---|---|---|---|---|---
WeatherKITTI | 1 |  |  |  |  | 0.100 | 0.696 | 4.375 | 0.175 | 0.896 | 0.966 | 0.984
2 | $\checkmark$ |  |  | Trinity | 0.100 | 0.700 | 4.367 | 0.175 | 0.897 | 0.966 | 0.984
3 | $\checkmark$ | $\checkmark$ |  | Trinity | 0.099 | 0.698 | 4.364 | 0.175 | 0.898 | 0.966 | 0.984
4 | $\checkmark$ | $\checkmark$ | $\checkmark$ | Distill | 0.100 | 0.704 | 4.384 | 0.175 | 0.896 | 0.966 | 0.984
5 |  | $\checkmark$ | $\checkmark$ | Contrast | 0.100 | 0.712 | 4.425 | 0.176 | 0.895 | 0.965 | 0.984
6 | $\checkmark$ | $\checkmark$ | $\checkmark$ | Trinity | 0.099 | 0.688 | 4.377 | 0.174 | 0.897 | 0.966 | 0.984
Real Weather | 7 |  |  |  |  | 0.143 | 1.566 | 7.853 | 0.202 | 0.815 | 0.947 | 0.982
8 | $\checkmark$ |  |  | Trinity | 0.141 | 1.540 | 7.851 | 0.201 | 0.817 | 0.948 | 0.983
9 | $\checkmark$ | $\checkmark$ |  | Trinity | 0.141 | 1.535 | 7.850 | 0.202 | 0.817 | 0.947 | 0.982
10 | $\checkmark$ | $\checkmark$ | $\checkmark$ | Distill | 0.145 | 1.588 | 7.894 | 0.204 | 0.812 | 0.946 | 0.981
11 |  | $\checkmark$ | $\checkmark$ | Contrast | 0.142 | 1.538 | 7.835 | 0.202 | 0.815 | 0.948 | 0.982
12 | $\checkmark$ | $\checkmark$ | $\checkmark$ | Trinity | 0.141 | 1.530 | 7.772 | 0.201 | 0.818 | 0.948 | 0.983

Effectivness of Diffusion Enhancement. Table 4 shows the effectiveness of each
enhancement based on the first stage of training. We take the MonoDiffusion
(replaced with MonoViT) as the baseline (ID 1). Firstly, the distillation
enhancement (ID 2) is added and produces a significant improvement. This gain
is reasonable because, compared with MonoDiffusion which only considers clear
scenes, the depth estimation in complex scenes typically requires more
powerful distillation. Next, we add outlier depth removal (ID 3), which brings
a notable improvement in RMSE and SqRel metrics because the unnatural depth
values have more significant impact on the second-order indicators. Finally,
we improved the condition (ID 4). With the introduction of image context
information, the diffusion-based model further shows an improvement in the
$a_{1}$ metric representing estimation stability.

Effectiveness of Trinity Contrast. As reported in Table 4 (ID 5), the
incorporation of trinity in the first stage model produces overall performance
gains in all aspects. Meanwhile, in Table 5, we can compare the performance of
three paradigms using IDs 4-6 and IDs 10-12. The results show that direct
distillation did not enhance the performance and even led to a decrease in
stability (RMSE, $a_{1}$). For the contrastive method, although we applied the
constraint of Eq. 2 to the contrastive objects, directly contrasting the
predicted noises would lead to network crashes (i.e., in which all predicted
depth values are zeros, which are not included in Table 5). Therefore, we
still maintain the noise constraint from Eq. 6, only taking the feature and
image levels comparison, but there is only a minor improvement. The proposed
trinity contrast is the only approach that significantly improves convergence
on WeatherKITTI and demonstrates increased robustness and accuracy on real
datasets.

Effectiveness of Multi-Level Contrasts The comparison of ablation with multi-
level contrasts is presented in Table 5 (IDs 2,3,6 and IDs 8,9,12). While the
performance of different models is close in the WeatherKITTI dataset, in real-
world weather situations, we notice that the results become more reliable as
we gradually include the feature and image level trinity. The enhancements in
these metrics, coupled with Fig. 4, apparently prove the effectiveness of the
multi-level contrast.

##  5\. Conclusion

In this paper, we enhance the stability and convergence of diffusion-based MDE
and further introduce a novel robust depth estimation framework named D4RD,
which integrates a customized contrastive learning scheme. This method
combines the strength of distillation with contrastive learning and repurposes
the sampled noise in the diffusion forward process, creating a ”trinity”
contrastive mode, thereby improving the accuracy and robustness of noise
prediction. Furthermore, we extend the noise level’s trinity to the more
general feature and image levels, constructing a multi-level trinity scheme to
balance the robust perception pressure across different model components.
Through extensive quantitative and qualitative experiments, we demonstrate the
effectiveness of D4RD against various architectures and its superior
performance over existing SoTA solutions.

## References

  * (1)
  * Bijelic et al. (2020) Mario Bijelic, Tobias Gruber, Fahim Mannan, Florian Kraus, Werner Ritter, Klaus Dietmayer, and Felix Heide. 2020.  Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Charron et al. (2018) Nicholas Charron, Stephen Phillips, and Steven Waslander. 2018.  De-noising of Lidar Point Clouds Corrupted by Snowfall. 254–261.  https://doi.org/10.1109/CRV.2018.00043
  * Chen et al. (2022) Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. 2022.  Diffusiondet: Diffusion model for object detection.  _arXiv preprint arXiv:2211.09788_ (2022).
  * Chen and He (2020) Xinlei Chen and Kaiming He. 2020.  Exploring Simple Siamese Representation Learning.  arXiv:2011.10566 [cs.CV]
  * Duan et al. (2023) Yiqun Duan, Xianda Guo, and Zheng Zhu. 2023.  Diffusiondepth: Diffusion denoising approach for monocular depth estimation.  _arXiv preprint arXiv:2303.05021_ (2023).
  * Eigen and Fergus (2015) David Eigen and Rob Fergus. 2015.  Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In _Proceedings of the IEEE international conference on computer vision_. 2650–2658.
  * Eigen et al. (2014) David Eigen, Christian Puhrsch, and Rob Fergus. 2014.  Depth map prediction from a single image using a multi-scale deep network.  _Advances in neural information processing systems_ 27 (2014).
  * Fu et al. (2018) Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. 2018.  Deep Ordinal Regression Network for Monocular Depth Estimation. In _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_. IEEE.  https://doi.org/10.1109/cvpr.2018.00214
  * Gasperini et al. (2023) Stefano Gasperini, Nils Morbitzer, HyunJun Jung, Nassir Navab, and Federico Tombari. 2023.  Robust Monocular Depth Estimation under Challenging Conditions. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_.
  * Godard et al. (2019) Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. 2019.  Digging Into Self-Supervised Monocular Depth Estimation.  arXiv:1806.01260 [cs.CV]
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising Diffusion Probabilistic Models.  arXiv:2006.11239 [cs.LG]
  * Ji et al. (2023) Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. 2023.  DDP: Diffusion Model for Dense Visual Prediction. In _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_. IEEE.  https://doi.org/10.1109/iccv51070.2023.01987
  * Kalb and Beyerer (2023) Tobias Kalb and Jürgen Beyerer. 2023.  Principles of Forgetting in Domain-Incremental Semantic Segmentation in Adverse Weather Conditions. In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE.  https://doi.org/10.1109/cvpr52729.2023.01869
  * Ke et al. (2024) Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. 2024.  Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kondapaneni et al. (2023) Neehar Kondapaneni, Markus Marks, Manuel Knott, Rogério Guimarães, and Pietro Perona. 2023.  Text-image Alignment for Diffusion-based Perception.
  * Kong et al. (2023) Lingdong Kong, Shaoyuan Xie, Hanjiang Hu, Lai Xing Ng, Benoit R. Cottereau, and Wei Tsang Ooi. 2023.  RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions.  _ArXiv_ abs/2310.15171 (2023).  https://api.semanticscholar.org/CorpusID:264436593
  * Laina et al. (2016) Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. 2016.  Deeper Depth Prediction with Fully Convolutional Residual Networks.  arXiv:1606.00373 [cs.CV]
  * Liu et al. (2023) Jiuming Liu, Guangming Wang, Weicai Ye, Chaokang Jiang, Jinru Han, Zhe Liu, Guofeng Zhang, Dalong Du, and Hesheng Wang. 2023.  DifFlow3D: Toward Robust Uncertainty-Aware Scene Flow Estimation with Diffusion Model.  _arXiv preprint arXiv:2311.17456_ (2023).
  * Liu et al. (2021) Lina Liu, Xibin Song, Mengmeng Wang, Yong Liu, and Liangjun Zhang. 2021.  Self-supervised Monocular Depth Estimation for All Day Images using Domain Separation.  arXiv:2108.07628 [cs.CV]
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled Weight Decay Regularization.  arXiv:1711.05101 [cs.LG]
  * Mao et al. (2024) Yifan Mao, Jian Liu, and Xianming Liu. 2024.  Stealing Stable Diffusion Prior for Robust Monocular Depth Estimation.  arXiv:2403.05056 [cs.CV]
  * Nam et al. (2023) Jisu Nam, Gyuseong Lee, Sunwoo Kim, Hyeonsu Kim, Hyoungwon Cho, Seyeon Kim, and Seungryong Kim. 2023.  DiffMatch: Diffusion Model for Dense Matching.  arXiv:2305.19094 [cs.CV]
  * Patni et al. (2024) Suraj Patni, Aradhye Agarwal, and Chetan Arora. 2024.  ECoDepth: Effective Conditioning of Diffusion Models for Monocular Depth Estimation.  _arXiv preprint arXiv:2403.18807_ (2024).
  * Ranftl et al. (2021) René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. 2021.  Vision transformers for dense prediction. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 12179–12188.
  * Ranftl et al. (2020) René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. 2020.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  _IEEE transactions on pattern analysis and machine intelligence_ 44, 03 (2020), 1623–1637.
  * Saunders et al. (2023) Kieran Saunders, George Vogiatzis, and Luis Manso. 2023.  Self-supervised Monocular Depth Estimation: Let’s Talk About The Weather.  arXiv:2307.08357 [cs.CV]
  * Saxena et al. (2023) Saurabh Saxena, Charles Herrmann, Junhwa Hur, Abhishek Kar, Mohammad Norouzi, Deqing Sun, and David J. Fleet. 2023.  The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation. In _Thirty-seventh Conference on Neural Information Processing Systems_.  https://openreview.net/forum?id=jDIlzSU8wJ
  * Shao et al. (2023) Shuwei Shao, Zhongcai Pei, Weihai Chen, Dingchi Sun, Peter C. Y. Chen, and Zhengguo Li. 2023.  MonoDiffusion: Self-Supervised Monocular Depth Estimation Using Diffusion Model.  arXiv:2311.07198 [cs.CV]
  * Song et al. (2022) Jiaming Song, Chenlin Meng, and Stefano Ermon. 2022.  Denoising Diffusion Implicit Models.  arXiv:2010.02502 [cs.LG]
  * Song et al. (2023) Ziyang Song, Ruijie Zhu, Chuxin Wang, Jiacheng Deng, Jianfeng He, and Tianzhu Zhang. 2023.  EC-Depth: Exploring the consistency of self-supervised monocular depth estimation in challenging scenes.  https://api.semanticscholar.org/CorpusID:268513620
  * Wang et al. (2023b) Jiyuan Wang, Chunyu Lin, Lang Nie, Shujun Huang, Yao Zhao, Xing Pan, and Rui Ai. 2023b.  WeatherDepth: Curriculum Contrastive Learning for Self-Supervised Depth Estimation under Adverse Weather Conditions.  _ArXiv_ abs/2310.05556 (2023).  https://api.semanticscholar.org/CorpusID:263831385
  * Wang et al. (2023a) Mengyu Wang, Henghui Ding, Jun Hao Liew, Jiajun Liu, Yao Zhao, and Yunchao Wei. 2023a.  SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process. In _NeurIPS_.
  * Yang et al. (2019) Guorun Yang, Xiao Song, Chaoqin Huang, Zhidong Deng, Jianping Shi, and Bolei Zhou. 2019.  DrivingStereo: A Large-Scale Dataset for Stereo Matching in Autonomous Driving Scenarios. In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zhang et al. (2023) Ning Zhang, Francesco Nex, George Vosselman, and Norman Kerle. 2023.  Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation. In _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE.  https://doi.org/10.1109/cvpr52729.2023.01778
  * Zhao et al. (2022a) Chaoqiang Zhao, Yang Tang, and Qiyu Sun. 2022a.  Unsupervised Monocular Depth Estimation in Highly Complex Environments.  arXiv:2107.13137 [cs.CV]
  * Zhao et al. (2022b) Chaoqiang Zhao, Youmin Zhang, Matteo Poggi, Fabio Tosi, Xianda Guo, Zheng Zhu, Guan Huang, Yang Tang, and Stefano Mattoccia. 2022b.  MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer. In _2022 International Conference on 3D Vision (3DV)_. IEEE.  https://doi.org/10.1109/3dv57658.2022.00077
  * Zhao et al. (2023) Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. 2023.  Unleashing Text-to-Image Diffusion Models for Visual Perception.  _ICCV_ (2023).
  * Zhou et al. (2017) Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. 2017.  Unsupervised Learning of Depth and Ego-Motion from Video.  arXiv:1704.07813 [cs.CV]


11institutetext: Georgia Institute of Technology, Atlanta GA 30332, USA
11email<EMAIL_ADDRESS>
# Root-Cause Analysis of Activation Cascade Differences in Brain Networks

Qihang Yao  0000-0001-5091-3850 Manoj Chandrasekaran  0000-0002-8625-0013
Constantine Dovrolis  0000-0002-4491-861X

###### Abstract

Diffusion MRI imaging and tractography algorithms have enabled the mapping of
the macro-scale connectome of the entire brain. At the functional level,
probably the simplest way to study the dynamics of macro-scale brain activity
is to compute the “activation cascade” that follows the artificial stimulation
of a source region. Such cascades can be computed using the Linear Threshold
model on a weighted graph representation of the connectome. The question we
focus on is: if we are given such activation cascades for two groups, say A
and B (e.g. Controls versus a mental disorder), what is the smallest set of
brain connectivity (graph edge weight) changes that are sufficient to explain
the observed differences in the activation cascades between the two groups? We
have developed and computationally validated an efficient algorithm, TRACED,
to solve the previous problem. We argue that this approach to compare the
connectomes of two groups, based on activation cascades, is more insightful
than simply identifying ”static” network differences (such as edges with large
weight or centrality differences). We have also applied the proposed method in
the comparison between a Major Depressive Disorder (MDD) group versus healthy
controls and briefly report the resulting set of connections that cause most
of the observed cascade differences.

###### Keywords:

Connectome Structural brain networks Activation cascade Root-cause analysis.

##  1 Introduction

Diffusion MRI imaging and tractography algorithms have enabled the mapping of
the macro-scale connectome of the entire brain [23]. This network
representation enables the application of powerful tools from graph theory and
graph algorithms in the study of the brain’s structure and function. Earlier
work has focused on various important network properties of the brain such as
small worldness [1], presence of hubs [12], modularity [22], etc. These
studies have revealed that seemingly local pathologies in specific regions can
have far-reaching global effects on other parts of the brain [24, 19].

Probably the simplest way to study the dynamics of brain activity at the
macro-scale is to compute the “activation cascade” that is generated by the
artificial stimulation of a source region. Activation cascades, represented in
the form of directed acyclic graphs (DAGs), describe how an activation
starting from one region (i.e., source node) propagates to the rest of the
brain, activating other brain regions along the way. Previous work has applied
the Asynchronous Linear Threshold (ALT) model on the mouse meso-scale
connectome to simulate the propagation and integration of sensory signals
through activation cascades [21]. Those modeling results were validated with
functional data from cortical voltage-sensitive dye imaging, showing that the
order of node activations in the model matches quite well with the empirical
activation order observed experimentally [21].

The question that we focus on in this study is: suppose we are given two
groups with significant differences in the activation cascades generated in
their brain networks. What is the smallest set of brain connectivity (i.e.,
graph edge weight) changes that are sufficient to explain the observed
differences in the activation cascades between the two groups? Answering this
question can be valuable in many studies when two groups should be compared,
not only in terms of structural connectome differences, but also in terms of
functional dynamics. For example, we can identify a (generally small) set of
brain connectivity changes that appear to cause the functional activation
differences in a given disorder, by comparing the corresponding activation
cascades with healthy controls. Further, the corresponding connections can be
used as possible targets in interventions and treatments such as deep brain
stimulation [26, 20].

We have developed an algorithm named TRACED (The Root-cause of Activation
Cascade Differences) to solve the previous problem. TRACED starts by
identifying node membership differences between the two groups (say A and B)
within the activation cascade of each source. Then, for each source, we
identify the smallest set of edges that, if their weights in group A are
modified to be equal to the weights in group B, the corresponding activation
cascades will be the same in both groups. We have computationally validated
TRACED across many test cases. Additionally, we have applied TRACED in the
comparison between a group of patients with major depressive disorder (MDD)
and a group of controls. This paper focuses on the proposed computational
method – a more comprehensive MDD-focused study of the two groups will be
presented in a different article.

Previous work detected significant topological differences in terms of network
metrics such as edge weights and centrality measures for various neurological
disorders, including multiple sclerosis [7, 15], Alzheimer’s disease [6],
Parkinson’s disease [27], and schizophrenia [8]. We argue that the activation
cascade approach to comparing the connectomes of two groups is more insightful
than simply identifying such ”static” network differences. The former makes
some clear and simple assumptions about the processing and propagation of
information in the brain, and it creates a causal connection between
structural changes and functional effects. Therefore, the identified
abnormalities are more interpretable and robust to subject variability.

##  2 Linear threshold model and activation cascades

Our starting point is a structural macro-scale brain network. In this network
representation, the graph is denoted by $G=(V,E)$, each node in $V$
corresponds to a brain region, and $E$ contains edges that correspond to
connectivity between brain regions. For DTI-based structural networks, the
edges are undirected. Each edge $(x,y)$ in $E$ is associated with a weight
$w(x,y)$ that represents the strength of the corresponding connection.

In the linear threshold model, each node can be either active or inactive.
Initially, all nodes are inactive, except a single source node. If a neighbor
$y$ of a node $x$ is active, then we say that $x$ “receives an activation”
from $y$ with strength $w(y,x)$. Node $x$ becomes active if it receives a
cumulative activation from all its active neighbors that is more than a
threshold $\theta$.

More formally, a node $x$ at time $t$ is associated with a binary state
variable $A(x,t)$ indicating whether $x$ is active (1) or not (0). For the
source node $s$, we have that $A(s,t=0)=1$ and for all other nodes:

$A(x,t+1)=1\text{ if }\sum_{y\mid(y,x)\in E}{w(y,x)A(y,t)}\geq\theta\\\ $ (1)

for $t\geq 0$. If $x$ becomes active in the cascade of source $s$, $t_{s}(x)$
is the time of its activation. By convention, $t_{s}(x)=\infty$ if node $x$
never gets active.

An activation cascade, in the form of a directed acyclic graph (DAG), shows
whether as well as how each node becomes active. The nodes in the activation
cascade of source $s$ form the following set:

$U(s)=\left\\{x\in V\mid t_{s}(x)<\infty\right\\}$ (2)

The edges in the activation cascade include $(x,y)\in E$ if node $x$ becomes
active before $y$. So, the presence of this edge in the cascade DAG means that
$x$ participates in the activation of $y$. Mathematically,

$F(s)=\\{(x,y)\in E\mid t_{s}(y)<t_{s}(x)\\}$ (3)

We denote the activation cascade as $H(s)=\\{U(s),F(s)\\}$. In Fig. 1 we show
a simple example illustrating an activation cascade generated in a toy network
using the linear threshold model.

Figure 1: An illustrative example of an activation cascade obtained using the
linear threshold model. Node B is the source of the cascade. The threshold
$\theta=2$. Node A gets active through the edge (A, B), and node C becomes
active after both A and B are active. The rest of the nodes stay inactive in
this cascade.

For a given $\theta$, different source nodes may give different cascade sizes.
Some source nodes do not activate any other node giving rise to empty
cascades, while other source nodes may activate every node in the network
causing a full cascade. The third case is that of a partial cascade, which is
more likely in practice. It would be unrealistic to set the threshold $\theta$
so high that we get many empty cascades – that would correspond to a comatose
brain! However, it would also be unrealistic to set $\theta$ so low that we
get many full cascades. The previous observations guide us to choose a range
of $\theta$ values that result in more partial cascades, across different
source nodes.

When comparing the structural brain networks of two subjects, or two groups,
we rely on the membership of each source’s cascade: If a node $x$ is active in
the cascade of source $s$ in one network, is $x$ also active in the
corresponding cascade of the other network? The similarity between the node
membership of two cascades is quantified using the Jaccard similarity metric,
applied on the set of active nodes in the two cascades. A small Jaccard
similarity represents a large difference between the two cascades. Therefore,
using $U(s)$ and $U^{\prime}(s)$ to denote the set of active nodes in networks
$G$ and $G^{\prime}$, respectively, after the activation of source $s$, the
difference between the two cascades is quantified by:

$d\\{U(s),U^{\prime}(s)\\}=1-J\\{U(s),U^{\prime}(s)\\}=1-\frac{|U(s)\cap
U^{\prime}(s)|}{|U(s)\cup U^{\prime}(s)|}$ (4)

where $J\\{U(s),U^{\prime}(s)\\}$ is the Jaccard similarity of the two
cascades.

##  3 TRACED algorithm

Figure 2: Method overview: The abnormal and control networks may have several
edges with different weights. We generate the activation cascade for each
source using the linear threshold model, and identify the cascade membership
differences across the two networks. Then we identify a subset of edges
(containing only edge BD in this example) whose weight change can explain the
majority of the observed cascade differences. In other words, if we restore
the weights of this subset of edges in the abnormal network to be equal to the
corresponding weights in the control network, the majority of the cascade
differences between two networks no longer exist.

We expect that a mental disorder (or any other genuine distinction in the
structural brain networks of a group) would cause cascade membership
differences for several different sources [28, 25]. Additionally, it is
reasonable to expect that these cascade membership differences will be caused
by a rather small set of brain connectivity abnormalities (a larger set of
abnormalities would probably be lethal). Under these assumptions, we aim to
detect the smallest set of edge weight changes that can explain the observed
cascade membership differences between the two groups.

#### The case of a single source node:

The problem of finding the root-cause for the activation cascade differences
of a single source $s$ can be formulated as follows: We are given the cascade
of $s$ in the control and the abnormal networks. Compute the minimum set of
edges $C$ in the abnormal network so that, if we restore the weights of those
edges to be equal to the corresponding weights in the control network, the
activation cascade of $s$ will be identical in the two networks. The
$C$-restored network is created by replacing the weight of edge $e$ ($e\in
C$), in the abnormal network with the weight of $e$ in the control network.

The mathematical formulation of the previous problem is:

$\hat{C}=\operatorname*{arg\,min}_{C\in\\{E\cup E^{\prime}\\}}|C|\
\text{s.t.}\ U^{\prime}_{C}(s)=U(s)$ (5)

where the set of active nodes in the control cascade of $s$ is denoted by
$U(s)$, the set of active nodes in the abnormal cascade of $s$ is
$U^{\prime}(s)$, and the set of active nodes in the $C$-restored network of
$s$ is $U^{\prime}_{C}(s)$. By convention, we take the weight of any edges
that are not present as 0.

A naive algorithm would be to search among all $2^{m}$ solutions ($m=|E\cup
E^{\prime}|$) but that would be computationally infeasible for the scale of
structural brain networks.

Instead, the TRACED algorithm starts from an empty set $C$ and gradually
“grows” the solution by adding one edge at a time. The original empty set $C$
can grow into $m$ different sets, each with a distinct edge. In the next step,
each of these $m$ sets can include one of the remaining $m-1$ edges, creating
a total of $m(m-1)$ sets with two edges each. This way, when $\hat{C}$ is
found, the number of candidate solutions is $m^{k}$, where $k=|\hat{C}|$.
Since we are adding edges step by step following an approach similar to
breadth-first-search, the solution is guaranteed to be optimal. Note that even
though the run-time of this approach grows exponentially with the solution
size $k$, we expect (as previously mentioned) that $k$ will be small in
practice.

The run-time of the algorithm can be improved however based on the following
observation. Let us define as “candidate edges” the edges that point from
$U(s)\cap U^{\prime}_{C}(s)$ (nodes active in both cascades) to $U(s)\triangle
U^{\prime}_{C}(s)$ (nodes active in one cascade but not the other). We know
that at each “growth” step at least one of the candidate edges should be added
to the solution. Otherwise, it is impossible to change the activation status
of the nodes in $U(s)\triangle U^{\prime}_{C}(s)$. Therefore, in each step we
only consider candidate edges, and thus limit the number of new possible
solutions created. If $b$ is an upper bound on the number of candidate edges,
the number of total solutions generated during the search is at most $b^{k}$.

Fig. 3 illustrates the execution of the TRACED algorithm with a small example.
We start with an empty solution $C$ and with the two activation cascades
(control and abnormal) for a single source $s$. Then, we identify the
candidate edges between the two cascades. For each candidate edge we “grow” a
new branch of the solution tree. We repeat these steps until
$U(s)=U^{\prime}_{C}(s)$. TRACED has a time complexity of
$O(b^{k}(|V|+|E^{\prime}|))$ because it iterates through $b^{k}$ candidate
solutions and executes the linear threshold model once for each possible
solution.

In Section 0.A.1 we introduce an improvement that further reduces the average
run-time and allows multiple optimal solutions to be found, by adding more
than one edge into a candidate solution at each step. That improvement does
not change the algorithm’s main idea or its worst-case run time.

Figure 3: Illustration of TRACED: The tree structure shows how the solution is
gradually computed one edge at a time – different branches of the tree can
lead to different solutions. The final solutions are marked in red. Along with
each candidate solution $C$, we present the corresponding cascade
$H^{\prime}_{C}(s)$. In this example, two solutions can explain equally well
the observed differences between the two cascades that originate from source
$C$.

To computationally validate the correctness of the algorithm, we created pairs
of small-scale graphs for which we know the edges that cause activation
cascade differences between the two networks. These examples are designed so
that they vary in several factors: they can have one or multiple optimal
solutions, only one edge or multiple edges in one solution, and edges in a
solution that are dependent on each other (i.e. an edge included in the
cascades only when the weight of another edge is restored). TRACED results in
the correct results in all cases, identifying one or multiple optimal
solutions correctly.

#### Aggregation across different source nodes:

The previous algorithm may produce different sets of edges for different
source nodes. Some of these edges may be the result of noise in the data or
other artifacts. We select a subset of these edges based on the following
argument: if TRACED identifies a certain edge as causal, not only for one
source but for multiple, it is likely that that edge represents a genuine and
important difference between the control and abnormal networks.

We use the coverage metric to measure the number of sources for which an edge
$e$ has been identified as causal for the cascade membership differences.
Edges with higher coverage play a more central role in the observed
differences between the two networks.

To test if the coverage of an edge is significant or not, we construct a null
hypothesis that all edges in the network have the same probability
($\frac{|\hat{C}(s)|}{|E|}$, where $\hat{C}(s)$ refers to the set of edges
identified to be causal to cascade membership differences with source node
$s$) to be reported as causal for source $s$. Under that assumption, the
coverage metric follows a binomial distribution:

$coverage^{\prime}(e)\sim B(\sum_{s}|\hat{C}(s)|,\frac{1}{|E|})$ (6)

So, the final output of TRACED is the set of edges for which the coverage
value is much higher than expected based on chance ($p<0.05$ in the binomial
distribution).

This final step makes the TRACED algorithm heuristic - the set of edges that
we finally report is no longer guaranteed to explain all differences in the
activation cascades of all sources. Nevertheless, the result captures edges
that have influenced the activation cascades across many source nodes, and is
therefore more reliable.

##  4 A case study on major depressive disorder

The focus of this paper is on the analysis method presented in the previous
section, rather than a specific application. To illustrate one potential
application of this method, however, we summarize here the results of a
comparison between a group of severe MDD patients and a group of healthy
controls. The DTI data for this comparison was provided to us by Dr. Helen
Mayberg’s group and they were originally used in the PReDICT study [3, 4]. The
PReDICT study was approved by Emory’s Institutional Review Board and the Grady
Hospital Research Oversight Committee. We constructed structural brain
networks applying probabilistic tractography on diffusion MRI scans of 90 MDD
patients and 18 control subjects. The brain was parcellated into 396 regions
(198 regions for each hemisphere) using the multi-modal cortical parcellation
of Glasser et al. [9], and the Brainnetome Atlas [5] for sub-cortical regions.
We applied the linear threshold model and generated an activation cascade for
each source node, and measured the cascade membership differences between the
two groups. The threshold that we used ranges from 0.1 to 0.3 among different
source nodes, and is determined for each source node as the one associated
with most significant cascade membership differences. We then applied TRACED
to identify the minimal set of connections that can explain the observed
cascade differences.

Table 1 lists the connections that we identified as causal for the cascade
membership differences between the two groups. These connections have a
significant overlap with findings of earlier studies reporting MDD-related
structural/functional changes. The connections identified as causal are
adjacent to parts of Brodmann area 24 [14], area 32 [10], area 9 [13], area 10
[16], and the orbitofrontal region [18]. All of these regions have been
reported to be pathologically relevant for MDD in earlier studies. Some of the
reported connections are also in the default mode network (DMN), which has
been shown to be heavily affected by MDD [14], with increased functional
connectivity [11]. We are going to further analyze this dataset and also
compare our findings with those of other network analysis methods in a follow-
up MDD-specific article.

Table 1: The connections that can explain the cascade differences between a group of MDD patients and a group of Controls. The name of each node is based on the parcellation of Glasser et al. [9], followed with a brief description of the location of that region (L: left hemisphere, R: right hemisphere).  Node 1 | description | – | Node 2 | description
---|---|---|---|---
p24 (L) | area-24 posterior | – | a24 (L) | area-24 anterior
10v (L) | area-10 ventral | – | 10pp (L) | medial polar area-10
a24 (L) | area-24 anterior | – | 9m (L) | area-9 medial
Pir (L) | piriform olfactory cortex | – | pOFC (L) | posterior OFC
13l (L) | area-13 lateral | – | OFC (L) | orbital frontal complex
p32 (L) | area-32 posterior | – | 10d (L) | area-10 dorsal
p32 (L) | area-32 posterior | – | 9m (L) | area-9 medial
10v (R) | area-10 ventral | – | 10pp (R) | polar 10p
pOFC (L) | posterior OFC | – | 13l (L) | area-13 larteral
10pp (L) | medial polar area-10 | – | OFC (L) | orbital frontal complex
p32 (L) | area-32 posterior | – | 10pp (L) | medial polar area-10

##  5 Discussion

Various network analysis metrics and methods have been proposed in the past to
compare structural brain networks. For instance, earlier work has investigated
the differences between brain networks in terms of small-worldness [1],
efficiency [2], and modularity [22]. At the node level, the clustering
coefficient, participant coefficient, and different node centrality metrics
(especially the betweenness centrality) have been widely adopted [17, 29]. At
the edge level, researchers have investigated the edges with significant
weight differences and the subnetwork they form [14].

TRACED falls in the spectrum of the edge-level analysis, and the resulting set
of connections is a subset of edges that have significant weight differences
between the two groups. Additionally however TRACED also incorporates the
information flow across the entire network in varied paths (because of all the
source nodes considered). We aggregate this topological information across the
entire network to describe the role that a specific network element (node or
edge) plays in the network, and how that role is different between the two
groups.

Figure 4: Earlier work has mostly focused on brain connectivity differences
using graph-theoretic metrics (e.g. node centrality metrics). TRACED
associates connectivity changes with their impact on information transfer in
the brain. It measures the impact of such changes on activation cascade
differences, and identifies the specific connections that cause these
differences through root-cause analysis.

Fig. 4 illustrates typical node-level and edge-level network analysis metrics
and compares them with TRACED. Compared to identifying solely edges with
significant weight changes, TRACED associates a structural change (i.e.,
restoring the weight of a connection to its value in the other group) with
functional changes (the node membership of the corresponding activation
cascades). This is favorable for two reasons: it makes the results more
interpretable, and less sensitive to variability across subjects. A
significant difference in the weight of a connection between two networks may
be simply due to subject variability. With TRACED, a connection is identified
as causal not only based on its weight but also based on the topological role
of that edge in the propagation of information (activation cascades) from
different source nodes.

Compared to node-level analysis metrics, TRACED can provide higher spatial
resolution because it identifies specific connections instead of entire brain
regions. Additionally, some network analysis metrics often make implicit
assumptions about information transfer in the brain (e.g. the betweenness
centrality metric assumes that information travels through shortest paths,
while the communicability metric assumes that information follows random-
walks). These assumptions may not be realistic (e.g. shortest path routing
requires information about the complete network stored in every node). It is
also harder to interpret these metrics in terms of their associated localities
in the brain (e.g. a node may have much lower communicability in one group but
what is the corresponding set of affected information pathways?). TRACED makes
an explicit assumption about information transfer, namely activation cascades
based on the linear threshold model, and it associates structural connectivity
changes with corresponding functional changes, making the results more
transparent and informative.

## References

  * [1] Bassett, D.S., Bullmore, E.T.: Small-world brain networks revisited. The Neuroscientist 23(5), 499–516 (2017)
  * [2] Berlot, R., Metzler-Baddeley, C., Ikram, M.A., Jones, D.K., O’Sullivan, M.J.: Global efficiency of structural networks mediates cognitive control in mild cognitive impairment. Frontiers in aging neuroscience 8, 292 (2016)
  * [3] Choi, K.S., Holtzheimer, P.E., Franco, A.R., Kelley, M.E., Dunlop, B.W., Hu, X.P., Mayberg, H.S.: Reconciling variable findings of white matter integrity in major depressive disorder. Neuropsychopharmacology 39(6), 1332–1339 (2014)
  * [4] Dunlop, B.W., Binder, E.B., Cubells, J.F., Goodman, M.M., Kelley, M.E., Kinkead, B., Kutner, M., Nemeroff, C.B., Newport, D.J., Owens, M.J., et al.: Predictors of remission in depression to individual and combined treatments (predict): study protocol for a randomized controlled trial. Trials 13(1), 1–18 (2012)
  * [5] Fan, L., Li, H., Zhuo, J., Zhang, Y., Wang, J., Chen, L., Yang, Z., Chu, C., Xie, S., Laird, A.R., et al.: The human brainnetome atlas: a new brain atlas based on connectional architecture. Cerebral cortex 26(8), 3508–3526 (2016)
  * [6] Fischer, F.U., Wolf, D., Scheurich, A., Fellgiebel, A., Initiative, A.D.N., et al.: Altered whole-brain white matter networks in preclinical alzheimer’s disease. NeuroImage: clinical 8, 660–666 (2015)
  * [7] Fleischer, V., Radetz, A., Ciolac, D., Muthuraman, M., Gonzalez-Escamilla, G., Zipp, F., Groppa, S.: Graph theoretical framework of brain networks in multiple sclerosis: a review of concepts. Neuroscience 403, 35–53 (2019)
  * [8] Fornito, A., Zalesky, A., Pantelis, C., Bullmore, E.T.: Schizophrenia, neuroimaging and connectomics. Neuroimage 62(4), 2296–2314 (2012)
  * [9] Glasser, M.F., Coalson, T.S., Robinson, E.C., Hacker, C.D., Harwell, J., Yacoub, E., Ugurbil, K., Andersson, J., Beckmann, C.F., Jenkinson, M., et al.: A multi-modal parcellation of human cerebral cortex. Nature 536(7615), 171–178 (2016)
  * [10] Grieve, S.M., Korgaonkar, M.S., Koslow, S.H., Gordon, E., Williams, L.M.: Widespread reductions in gray matter volume in depression. NeuroImage: Clinical 3, 332–339 (2013)
  * [11] Hamilton, J.P., Farmer, M., Fogelman, P., Gotlib, I.H.: Depressive rumination, the default-mode network, and the dark matter of clinical neuroscience. Biological psychiatry 78(4), 224–230 (2015)
  * [12] Hwang, K., Hallquist, M.N., Luna, B.: The development of hub architecture in the human functional brain network. Cerebral Cortex 23(10), 2380–2393 (2013)
  * [13] Kerestes, R., Ladouceur, C.D., Meda, S., Nathan, P.J., Blumberg, H.P., Maloney, K., Ruf, B., Saricicek, A., Pearlson, G.D., Bhagwagar, Z., et al.: Abnormal prefrontal activity subserving attentional control of emotion in remitted depressed patients during a working memory task with emotional distracters. Psychological Medicine 42(1), 29–40 (2012). https://doi.org/10.1017/S0033291711001097
  * [14] Korgaonkar, M.S., Fornito, A., Williams, L.M., Grieve, S.M.: Abnormal structural networks characterize major depressive disorder: a connectome analysis. Biological psychiatry 76(7), 567–574 (2014)
  * [15] Llufriu, S., Martinez-Heras, E., Solana, E., Sola-Valls, N., Sepulveda, M., Blanco, Y., Martinez-Lapiscina, E.H., Andorra, M., Villoslada, P., Prats-Galino, A., et al.: Structural networks involved in attention and executive functions in multiple sclerosis. NeuroImage: Clinical 13, 288–296 (2017)
  * [16] Long, Z., Duan, X., Wang, Y., Liu, F., Zeng, L., Zhao, J.p., Chen, H.: Disrupted structural connectivity network in treatment-naive depression. Progress in Neuro-Psychopharmacology and Biological Psychiatry 56, 18–26 (2015)
  * [17] Qin, J., Wei, M., Liu, H., Yan, R., Luo, G., Yao, Z., Lu, Q.: Abnormal brain anatomical topological organization of the cognitive-emotional and the frontoparietal circuitry in major depressive disorder. Magnetic resonance in medicine 72(5), 1397–1407 (2014)
  * [18] Rajkowska, G., Miguel-Hidalgo, J.J., Wei, J., Dilley, G., Pittman, S.D., Meltzer, H.Y., Overholser, J.C., Roth, B.L., Stockmeier, C.A.: Morphometric evidence for neuronal and glial prefrontal cell pathology in major depression. Biological psychiatry 45(9), 1085–1098 (1999)
  * [19] Rehme, A.K., Grefkes, C.: Cerebral network disorders after stroke: evidence from imaging-based connectivity analyses of active and resting brain states in humans. The Journal of physiology 591(1), 17–31 (2013)
  * [20] Riva-Posse, P., Choi, K., Holtzheimer, P.E., Crowell, A.L., Garlow, S.J., Rajendra, J.K., McIntyre, C.C., Gross, R.E., Mayberg, H.S.: A connectomic approach for subcallosal cingulate deep brain stimulation surgery: prospective targeting in treatment-resistant depression. Molecular psychiatry 23(4), 843–849 (2018)
  * [21] Shadi, K., Dyer, E., Dovrolis, C.: Multisensory integration in the mouse cortical connectome using a network diffusion model. Network Neuroscience 4(4), 1030–1054 (2020)
  * [22] Sporns, O., Betzel, R.F.: Modular brain networks. Annual review of psychology 67, 613–640 (2016)
  * [23] Sporns, O., Tononi, G., Kötter, R.: The human connectome: a structural description of the human brain. PLoS Comput Biol 1(4), e42 (2005)
  * [24] Stam, C.J.: Modern network science of neurological disorders. Nature Reviews Neuroscience 15(10), 683–695 (2014)
  * [25] Stam, C.J., Jones, B., Nolte, G., Breakspear, M., Scheltens, P.: Small-world networks and functional connectivity in alzheimer’s disease. Cerebral cortex 17(1), 92–99 (2007)
  * [26] Van Hartevelt, T.J., Cabral, J., Møller, A., FitzGerald, J.J., Green, A.L., Aziz, T.Z., Deco, G., Kringelbach, M.L.: Evidence from a rare case study for hebbian-like changes in structural connectivity induced by long-term deep brain stimulation. Frontiers in behavioral neuroscience 9, 167 (2015)
  * [27] Wen, M.C., Heng, H.S., Hsu, J.L., Xu, Z., Liew, G.M., Au, W.L., Chan, L.L., Tan, L.C., Tan, E.K.: Structural connectome alterations in prodromal and de novo parkinson’s disease patients. Parkinsonism & related disorders 45, 21–27 (2017)
  * [28] Zeng, L.L., Shen, H., Liu, L., Wang, L., Li, B., Fang, P., Zhou, Z., Li, Y., Hu, D.: Identifying major depression using whole-brain functional connectivity: a multivariate pattern analysis. Brain 135(5), 1498–1507 (2012)
  * [29] Zhang, R., Kranz, G.S., Zou, W., Deng, Y., Huang, X., Lin, K., Lee, T.M.: Rumination network dysfunction in major depression: A brain connectome study. Progress in Neuro-Psychopharmacology and Biological Psychiatry 98, 109819 (2020)

##  Appendix 0.A Appendix

###  0.A.1 Optimization of TRACED

A key observation is that if adding a single edge $(x,y)$ into a solution set
does not change the activation status of node $y$, we will inevitably need to
add additional edges pointing to $y$ to build a final solution. Otherwise, for
a solution $C$ with $(x,y)$, we can always find a better solution
$C^{\prime}=C-\\{(x,y)\\}$ with
$U^{\prime}_{C}(s)=U^{\prime}_{C^{\prime}}(s)$.

Therefore, we can improve the original TRACED algorithm, by adding a
collection of edges in each iteration, so that $U^{\prime}_{C}(s)$ changes
when we create a new partial solution. This way we can reduce the number of
partial solutions that we create during the search for the optimal solution.
How do we find the collection of edges that can cause the change in
$U_{C}(s)$? We know that we focus on change of activation status of nodes in
$U^{\prime}_{C}(s)\triangle U(s)$, and so we can discuss the case of nodes
$U(s)\setminus U^{\prime}_{C}(s)$ and $U^{\prime}_{C}(s)\setminus U(s)$
separately.

  1. 1.

For each node $v$ in $U(s)\setminus U^{\prime}_{C}(s)$, we can check if there
is an ensemble of edges from $U(s)\cap U^{\prime}_{C}(s)$ pointing to this
node, so that if we include the ensemble into the solution, $v$ would be
active in the updated $U^{\prime}_{C}(s)$. It is guaranteed that we can find
at least one such collection of edges. Otherwise, we cannot explain why this
$v$ could be active in $U(s)$.

  2. 2.

For nodes in $U^{\prime}_{C}(s)\setminus U(s)$, we can further find its subset
$T_{C}(s)$ so that for each node $v\in T_{C}(s)$, $\sum_{u\in U(s)\cap
U^{\prime}_{C}(s)}w(u,v)\geq\theta$. We can prove that
$U^{\prime}_{C}(s)\setminus U(s)$ will no longer be in $U^{\prime}_{C}(s)$ if
and only if we add an ensemble of edges for each node in $T_{C}(s)$ into $C$.
If for a node $v$ in $T_{C}(s)$ we do not add edges connecting to $v$ into
$C$, $v$ will remain active and present in $U^{\prime}_{C}(s)$. If we add
edges connecting to $v$ for every node $v$ in $T_{C}(s)$, none of the nodes in
$U^{\prime}_{C}(s)\setminus U(s)$ receive an activation more than $\theta$, so
that they will no longer be active.

With this modification, each partial solution $C$ corresponds to a state
$U^{\prime}_{C}(s)$, and it is guaranteed that there are no edges that can be
removed from $C$ without changing that state. Therefore, all partial solutions
corresponding to one state are equivalent, in terms of the edges that need to
be added to the solution to reach another state. Therefore, we can construct a
graph of solutions, where each node $x$ corresponds to a state, and each edge
$(x,y,\\{e_{1},\dots\\})$ corresponds to an ensemble of edges
$\\{e_{1},\dots\\}$ needed to be added into the partial solutions
corresponding to state $x$ so that the new solution leads to state $y$. Such
an edge is also weighted, with a weight that is equal to the number of edges
in the collection. Notice that there can be multiple edges between two nodes,
each corresponding to one collection of edges and may have a different weight
different than other edges.

With such a graph of solutions, our goal is equivalent to finding the weighted
shortest path between the initial state $U^{\prime}(s)$ and the final state
$U=U^{\prime}_{\hat{C}}(s)$ in the graph. This is because the sum of the
weights of edges along a path in the graph of solutions would be the number of
actual edges we include in the final solution. We can find the shortest path
using Dijkstra’s algorithm since we have only positive weights. The major
benefit of having this graph of solutions is that we can deal with the case of
multiple optimal solutions more explicitly. They will be represented as
multiple shortest paths from the initial state to the final state.