# Domain Generalization Deep Graph Transformation

Shiyu Wang
Department of Biostatistics and Bioinformatics
Emory University
<EMAIL_ADDRESS>
&Guangji Bai
Department of Computer Science
Emory University
<EMAIL_ADDRESS>
Qingyang Zhu
Department of Environmental Health
Emory University
<EMAIL_ADDRESS>
&Zhaohui Qin
Department of Biostatistics and Bioinformatics
Emory University
<EMAIL_ADDRESS>
&Liang Zhao
Department of Computer Science
Emory University
<EMAIL_ADDRESS>

###### Abstract

Graph transformation that predicts graph transition from one mode to another
is an important and common problem. Despite much progress in developing
advanced graph transformation techniques in recent years, the fundamental
assumption typically required in machine-learning models that the testing and
training data preserve the same distribution does not always hold. As a
result, domain generalization graph transformation that predicts graphs not
available in the training data is under-explored, with multiple key challenges
to be addressed including (1) the extreme space complexity when training on
all input-output mode combinations, (2) difference of graph topologies between
the input and the output modes, and (3) how to generalize the model to
(unseen) target domains that are not in the training data. To fill the gap, we
propose a multi-input, multi-output, hypernetwork-based graph neural network
(MultiHyperGNN) that employs a encoder and a decoder to encode topologies of
both input and output modes and semi-supervised link prediction to enhance the
graph transformation task. Instead of training on all mode combinations,
MultiHyperGNN preserves a constant space complexity with the encoder and the
decoder produced by two novel hypernetworks. Comprehensive experiments show
that MultiHyperGNN has a superior performance than competing models in both
prediction and domain generalization tasks. The code of MultiHyperGNN is in
https://github.com/shi-yu-wang/MultiHyperGNN.

##  1 Introduction

Graph is a ubiquitous data structure characterized by node attributes and the
graph topology that describe objects and their relationships. Many tasks on
graphs ask for predicting a graph (i.e., graph topology or node attributes)
from another one. Applications of such graph transformation include traffic
forecasting between two time stamps based on traffic flow (Li et al., 2018; Yu
et al., 2018), fraud detection between transactional periods (Van Belle et
al., 2022), and chemical reaction prediction according to molecular structures
(Guo et al., 2019; Pan et al., 2022; Wang et al., 2022).

Figure 1: Graph transformation predicts gene expression of the output tissue
(e.g., brain and muscle) from gene expression of the input tissue (e.g., heart
and whole blood). The yellow and the blue network represents the gene-gene
network in the input tissue and the output tissue, respectively. Node colors
indicate different expression values. During training we train the model on
input-target tissue pairs, whereas during testing we generalize the model to
unseen mode pairs or even unseen modes.

Despite of a wide spectrum of applications, graph transformation still faces
major issues such as insufficient samples of graph pairs for training the
model. For instance, as shown in Figure 1, if the model is trained to predict
gene-gene network on specific tissue pairs (e.g., from heart and blood to
brain, from blood to muscle), but in testing process, one may want to
generalize the model to unseen tissue pairs (e.g., from heart to muscle) or
even to tissues unavailable in the training data. If so, the performance of
the graph transformation model may deteriorate due to domain distribution gaps
(Quinonero-Candela et al., 2008). Therefore, it is imperative and crucial to
improve the generalization ability of graph transformation models to
generalize the learned graph transformation to other (unseen) graph
transformations, namely domain generalization graph transformation.

Domain generalization graph transformation, nevertheless, is still under-
explored by the machine-learning community due to the following challenges:
(1) High complexity in the training process. To learn the distribution of
graph (or mode) pairs in training data, we need to learn the model by
traversing on all combinations of input modes to predict all combinations of
output modes. In this case, the training complexity would be exponential if we
train a single model for all possible input-output mode combinations; (2)
Graph transformation between topologically different modes. The existing works
regarding graph transformation predict node attributes conditioning on either
the same topology or the same set of nodes of input and output modes
(Battaglia et al., 2016; Yu et al., 2018; Guo et al., 2019). Performing graph
transformation across modes with varying topologies, including different edges
and even varying graph sizes, is a difficult task. Main challenges include how
to learn the mapping between distinct topologies and how to incorporate the
topology of each mode to enhance the prediction task; (3) Learning graph
transformation involving unseen domains and lack of training data. Graph
transformation usually requires both the source and target domains to be
visible and have adequate training data to train a sophisticated model.
However, during the prediction phase, we may be asked to predict a graph in an
unseen target domain. Learning such transformation mapping without any
training data is an exceedingly challenging task.

To fill the gap, we propose a novel framework for domain generalization graph
transformation via a multi-input, multi-output hypernetwork-based graph neural
networks (MultiHyperGNN). Our contributions are summarized as follows:

  * •

A novel multi-input, multi-output framework of graph transformation is
proposed. We aim at graph transformation for predicting node attributes across
multiple input and output modes, by introducing a novel framework based on the
multi-input, multi-output training strategy. The space complexity of the model
is reduced from exponential to constant in training process.

  * •

An encoder and a decoder are developed for graph transformation between
topologically different modes. To achieve the graph transformation between
topologically different modes, MultiHyperGNN has an encoder and a decoder to
encode the graph in the input and output mode, respectively. Additionally,
MultiHyperGNN performs semi-supervised link prediction to complete the output
graph, enabling the model to generalize to all nodes in the output mode.

  * •

Two hypernetworks are used to produce the encoder and the decoder for domain
generalization. We design two novel hypernetworks that produce the encoder and
the decoder. Mode-specific meta information serves as the input to guide the
hypernetwork to produce the corresponding encoder or decoder, and generalize
to unseen target domains.

  * •

The performance of MultiHyperGNN is experimentally superior. We conduct
extensive experiments to demonstrate the effectiveness of MultiHyperGNN on two
real-world datasets. The experimental results show that MultiHyperGNN is
superior than competing models.

This paper introduces the existing works on domain generalization graph
transformation in Section 2. Next, the problem is formally defined in Section
3. Details of the proposed model, MultiHyperGNN, is discussed in Section 4,
followed by the experiments in Section 5 and conclusion in Section 6.

##  2 Related works

###  2.1 Graph transformation

Graph transformation maps graph from one mode to another (Du et al., 2021b).
Some of the existing works predict node attributes given fixed graph topology.
Li et al. Li et al. (2018) predicted traffic forecasting by incorporating both
spatial and temporal dependency. Battaglia et al. Battaglia et al. (2016)
predicted velocities of objects on the subsequent time step. Some works
instead predict graph topology. Guo et al. Guo et al. (2022) learned the
global and local translation with graph convolution and deconvolution layers.
Other works instead simultaneously predict node attributes and graph topology.
Guo et al. Guo et al. (2019) solved node-edge joint translation with a multi-
block network. Lin et al. (Lin et al., 2020) applied graph attention to the
co-evolution of node and edge states. When predicting node attributes,
nevertheless, the assumption of fixed graph topology in both input and output
modes may not always hold. Graph transformation that can handle topologies of
both modes remains to be explored.

###  2.2 Domain generalization

Machine learning systems usually assume the same distribution between the
training and the testing data, whereas generalizing trained models to unseen
data is significant in fields such as semantic segmentation (Gong et al.,
2019; Dou et al., 2019), fault diagnosis (Li et al., 2020; Zheng et al.,
2020), natural language processing (Wang et al., 2020; Garg et al., 2021), etc
(Du et al., 2021a; Qian et al., 2021). Du et al. Du et al. (2021a) applied
domain generalization to time series modeling by an RNN-based model to solve
the temporal covariate shift. Qian et al. (Qian et al., 2021) applied domain
generalization to sensor-based human activity recognition by learning the
domain-invariant modules to disentangle different persons. Gong et al. (Gong
et al., 2019) translated images from the input to the output mode while
producing a sequence of intermediate modes for domain generalization. Wang et
al. (Wang et al., 2020) used meta learning that targets zero-shot domain
generalization for semantic parsing. Chen et al. (Chen et al., 2022) assumed
that the domain label is unavailable for training and the model needs to
identify latent domain structure and their semantic correlations, which may
expect sufficient expressiveness of the representation learning process.

###  2.3 Hypernetworks

A hypernetwork is a neural network that generates weights of another neural
network (Ha et al., 2017). Hypernetwork has a broad spectrum of applications,
including image classification tasks (Sun et al., 2017; Sendera et al., 2023),
image editing (Alaluf et al., 2022), robotic control (Huang et al., 2021;
Rezaei-Shoshtari et al., 2023) and language models (Volk et al., 2022; Zhang
et al., 2022). Noticeably, hypernetwork has also been employed for domain
generalization. Qu et al. (Qu et al., 2022) used hypernetworks to generate
weights of experts while allowing experts to share meta-knowledge. This model
needs to generate multiple classifiers and take the weighted sum for the final
prediction, whose training space complexity is linear to the number of
classifiers. Sendera et al. (Sendera et al., 2023) proposed HyperShot where
the kernel-based representation of the support examples is fed to hypernetwork
to create the classifier for few-shot learning. Bai et al. Bai et al. (2022)
utilized hypernetworks to produce graph classifiers, but with only time stamps
as the input of hypernetworks to focus on temporal domain generalization.
Despite of the wide use of hypernetworks for domain generalization, the
research of hypernetworks to generate GNNs is limited. In our work, we design
two novel hypernetworks to guide the domain generalization on graph
transformation tasks.

##  3 Problem formulation

Suppose we have $N$ modes of graphs composed of $p$ nodes:
$\mathcal{G}=\\{\mathcal{G}^{(1)},\mathcal{G}^{(2)},...,\mathcal{G}^{(N)}\\}$,
where each mode contains graphs with the same topology. Specifically, suppose
there are $n$ independent samples in the dataset, and for each sample $i$ in
the mode $j$, denote
$G_{i}^{(j)}=\\{A^{(j)},X_{i}^{(j)}\\}\in\mathcal{G}^{(j)}$, where
$A^{(j)}\in\mathbb{R}^{p_{j}\times p_{j}}$ is the graph of size $p_{j}\leq p$
and $X_{i}^{(j)}\in\mathbb{R}^{p_{j}\times d}$ is the node attributes with $d$
features. Note that the graph of $\mathcal{G}^{(j)}$ may not contain all $p$
nodes in mode $j$, all other nodes are disjointed with each other and with
$p_{j}$ nodes in $\mathcal{G}^{(j)}$. We further assume each mode $j$ can be
characterized by its meta information $m^{(j)}$. For instance, the mode
$\mathcal{G}^{(j)}$ can be a specific human tissue $j$ that has the gene-gene
expression network $G_{i}^{(j)}$ for a patient $i$. There are $p$ human genes
expressed in various human tissues but $G_{i}^{(j)}$ only contains $p_{j}$ of
them.

Next, we formally formulate the task domain generalization graph
transformation as below:

######  Definition 1 (Domain generalization graph transformation).

Let
$\mathcal{S}=\\{\mathcal{X}\times\mathcal{Y}:\mathcal{X}\in\mathcal{P}(\mathcal{G}),\mathcal{Y}\in\mathcal{P}(\mathcal{G}-\mathcal{X})\\}$
be the source domain where we train the graph-transformation model
$f:\mathcal{X}\rightarrow\mathcal{Y}$, which predicts node attributes in
$\mathcal{Y}$ from node attributes in $\mathcal{X}$. $\mathcal{P}(\cdot)$ is
the power set excluding the empty set. Domain generalization graph
transformation learns the $f$ so that the prediction error on
$f:\mathcal{X}^{\mathcal{T}}\rightarrow\mathcal{Y}^{\mathcal{T}}$ is
minimized, where $\mathcal{X}^{\mathcal{T}}\times\mathcal{Y}^{\mathcal{T}}$ is
the target domain s.t.
$\mathcal{X}^{\mathcal{T}}\times\mathcal{Y}^{\mathcal{T}}\notin\mathcal{S}$.

Domain generalization graph transformation is exceptionally difficult due to
the following challenges:

Challenge 1: High complexity in training process. For training the graph
transformation $f:\mathcal{X}\rightarrow\mathcal{Y}$, where
$\mathcal{X}\times\mathcal{Y}\in\mathcal{S}$, conventionally we need to train
$O(3^{N})$ models to handle all possible mode combinations in $\mathcal{S}$,
which is rather computationally intensive.

Challenge 2: Topological difference between input and output domains. When the
input mode and the output mode have different topologies, how to utilize
topologies of both modes to jointly contribute to graph transformation remains
to be explored. An intuitive way is to employ two graph encoders to
respectively encode the graph topology of both modes, but how to form the
graph-transformation model on $\mathcal{S}$ with only two such encoders is
still challenging.

Challenge 3: Generalization to unseen domains. Even if it is possible that we
train the model on all combinations of modes in $\mathcal{S}$, how to learn
the graph transformation that can efficiently predict the graph in an unseen
target domain is still challenging.

##  4 Domain generalization deep graph transformation

###  4.1 Overview of MultiHyperGNN

For the first challenge, instead of including all the exponentially many modes
by separately training all their combinations, we collectively train all the
modes together to avoid the duplication of modes and reduce the time
complexity to linearity (Figure 2). The details are given in this section. To
address the heterogeneity of node set and topology, we propose a novel
encoder-decoder framework in Section 4.2 (Figure 2 (A)). Moreover, each input
mode requires an encoder while each output mode needs a decoder, which can be
any type of GNNs such as Graph Convolutional Network (GCN), Graph Isomorphism
Network (GIN) and Graph Attention Network (GAT). To learn the encoder and the
decoder for unseen modes, we propose to train two hypernetworks that can
respectively generate any encoder or decoder given the meta information of the
mode in Section 4.3 (Figure 2 (B)). Furthermore, we provide a theoretical
assurance that an ample amount of meta-information will result in improved
generalization accuracy when extrapolating to unexplored domains.

Figure 2: Overview of MultiHyperGNN for the graph transformation from $N$
input to $M$ output modes. $\\{f_{e}^{(j)}\\}_{j\in\mathcal{X}}$ are encoders
generated by the encoder hypernetwork $\beta_{e}$.
$\\{f_{d}^{(k)}\\}_{k\in\mathcal{Y}}$ are decoders generated by the decoder
hypernetwork $\beta_{d}$. $m^{\mathcal{X}}$ and $m^{\mathcal{Y}}$ are mode-
specific meta information. "Link pred." is semi-supervised link prediction.
Blue dotted line is the flow of back-propagation.

Let $j\in\mathcal{X}$ be the $j$-th mode in $\mathcal{X}$ and
$k\in\mathcal{Y}$ be the $k$-th mode in $\mathcal{Y}$, where
$\mathcal{X}\in\mathcal{P}(\mathcal{G})$ and
$\mathcal{Y}\in\mathcal{P}(\mathcal{G}-\mathcal{X})$ as in Def. 1. As shown in
Figure 2 (A), to predict $X_{i}^{(k)}$, we employ encoders (i.e.,
$f_{e}=\\{f_{e}^{(j)}\\}_{j\in\mathcal{X}}$) and decoders (i.e.,
$f_{d}=\\{f_{d}^{(k)}\\}_{k\in\mathcal{Y}}$) to encode the topology of both
input and output modes:

$\displaystyle\hat{X}_{i}^{(k)}=f_{d}^{(k)}(A^{(k)},\sigma_{1}(\\{f_{e}^{(j)}(G_{i}^{(j)};\beta_{e}^{(j)})\\}_{j\in\mathcal{X}}));\beta_{d}^{(k)})),\
\ k\in\mathcal{Y}$ (1)

Namely, to predict node attributes of any $k\in\mathcal{Y}$ from modes in
$\mathcal{X}$, we first encode topologies of modes in $\mathcal{X}$ via
$f_{e}^{(j)}(G_{i}^{(j)};\beta_{e}^{(j)})$, $j\in\mathcal{X}$. Then we
aggregate embeddings of all these modes via the pooling function
$\sigma_{1}(\cdot)$ and feed it with the graph topology $A^{(k)}$ into
$f_{d}^{(k)}$ to predict $X_{i}^{(k)}$, where $k\in\mathcal{Y}$. To reduce the
heavy complexity of the training process due to the exponential number of
choices of $\mathcal{X}\times\mathcal{Y}\in\mathcal{S}$ (Def. 1) and
generalize the mode to unseen domains, instead of training separately for each
$\mathcal{X}\times\mathcal{Y}$, as shown in Figure 2 (B), we borrow two
hypernetworks (i.e., $\beta_{e}$ and $\beta_{d}$) to produce all encoders and
decoders with the corresponding mode-specific meta information:

$\displaystyle\beta_{e}^{(j)}=\beta_{e}(m^{(j)};\gamma_{e}),\ \
j\in\mathcal{X};\ \ \beta_{d}^{(k)}=\beta_{d}(m^{(k)};\gamma_{d}),\ \
k\in\mathcal{Y},$ (2)

where $\gamma_{e}$ and $\gamma_{d}$ parameterize $\beta_{e}$ and $\beta_{d}$,
respectively, and are learned during training process. Therefore, Eq. 1 is re-
parameterized by
$\gamma_{\mathcal{X}\rightarrow\mathcal{Y}}=\\{\gamma_{e},\gamma_{d}\\}$:

$\displaystyle\hat{X}_{i}^{(k)}$ $\displaystyle=$ $\displaystyle
f_{d}^{(k)}(A^{(k)},\sigma_{1}(\\{f_{e}^{(j)}(G_{i}^{(j)};\beta_{e}(m^{(j)};\gamma_{e}))\\}_{j\in\mathcal{X}});\beta_{d}(m^{(k)};\gamma_{d})),\
\ k\in\mathcal{Y}$ (3) $\displaystyle=$ $\displaystyle
f_{\gamma_{\mathcal{X}\rightarrow\mathcal{Y}}}(A^{(k)},\\{G_{i}^{(j)},m^{(j)}\\}_{j\in\mathcal{X}};\gamma_{\mathcal{X}\rightarrow\mathcal{Y}}),\
\ k\in\mathcal{Y},$

where
$f_{\gamma_{\mathcal{X}\rightarrow\mathcal{Y}}}=\\{f_{d}^{(k)}*\\{f_{e}^{(j)}\\}_{j\in\mathcal{X}}\\}_{k\in\mathcal{Y}}:\mathbb{R}^{N\times
p}\rightarrow\mathbb{R}^{M\times p}$ formularizes the graph transformation
that predicts node attributes of $M$ modes in $\mathcal{Y}$ from $N$ modes in
$\mathcal{X}$.

As long as $\hat{X}_{i}^{(k)}$ is predicted via Eq. 3, we mathematically
formulate the first term of the learning objective of MultiHyperGNN as
follows:

$\displaystyle\mathcal{L}_{1}(\gamma_{e},\gamma_{d})=\sum_{i=1}^{n}\ell(\\{\hat{X}_{i}^{(k)}\\}_{k\in\mathcal{Y}},\\{X_{i}^{(k)}\\}_{k\in\mathcal{Y}})$
(4)

where $\ell(\cdot)$ measures the prediction error of
$f_{\gamma_{\mathcal{X}\rightarrow\mathcal{Y}}}$ of each sample, such as mean
squared error (MSE), mean absolute error (MAE), etc. $n$ is the total number
of samples in training data.

Since the size of the source domain $\mathcal{S}$ is $O(3^{N})$, leading to an
exponential space complexity of $O(3^{N})$ with the space of trainable
parameters as
$\\{\mathcal{P}(\\{\beta_{e}^{(j)}\\}_{j\in\mathcal{X}})\times\mathcal{P}(\\{\beta_{d}^{(k)}\\}_{k\in\mathcal{Y}}):\mathcal{X}\in\mathcal{P}(\mathcal{G}),\mathcal{Y}\in\mathcal{P}(\mathcal{G}-\mathcal{X})\\}$.
MultihyperGNN reduces the space complexity to $O(1)$.

###  4.2 Graph transformation on topologically different domains

Traditional graph-transformation models encounter significant challenges when
attempting to handle modes with different graph topologies (i.e., $A^{(j)}\neq
A^{(k)},p_{j}\neq p_{k}$). To address this issue, as shown in Figure 2 (A), we
propose GNN-based encoder $f_{e}^{(j)}$ and decoder $f_{d}^{(k)}$ that encode
the graph of modes $j\in\mathcal{X}$ and $k\in\mathcal{Y}$, perform semi-
supervised link prediction to complete the topology of the output mode
$\mathcal{G}^{(k)}$ and enable the model to predict all $p$ nodes. Let
${\mathcal{V}}^{(j)}$ and ${\mathcal{V}}^{(k)}$ be sets of nodes contained in
the graph of modes $j$ and $k$, respectively, and
$|{\mathcal{V}}^{(j)}|=p_{j}$, $|{\mathcal{V}}^{(k)}|=p_{k}$. Since $p_{j}\neq
p_{k}$, to match the input dimension of $f_{e}^{(j)}$ and $f_{d}^{(k)}$, we
expand $\mathcal{G}^{(j)}$ and $\mathcal{G}^{(k)}$ by the union of their nodes
and obtain $\tilde{\mathcal{G}}^{(j)}$ and $\tilde{\mathcal{G}}^{(k)}$ with
node sets:
$\tilde{\mathcal{V}}^{(j)}=\tilde{{\mathcal{V}}}^{(k)}={\mathcal{V}}^{(j)}\bigcup{\mathcal{V}}^{(k)}$,
and
$|\tilde{\mathcal{V}}^{(j)}|=\tilde{p}_{j}=|\tilde{\mathcal{V}}^{(k)}|=\tilde{p}_{k}\leq
p$. Those newly added nodes are self-connected and are disjointed with other
nodes.

####  4.2.1 Encoder

For the $i$-th sample, the encoder $f_{e}^{(j)}$ encodes the topology and node
attributes of the mode $j$ into the latent embedding ${\bf
z}_{i}^{(j)}\in\mathbb{R}^{l}$, where $l$ is the hidden dimension:

$\displaystyle{\bf
z}_{i}^{(j)}=f_{e}^{(j)}(\tilde{G}_{i}^{(j)};\beta_{e}^{(j)})=\text{GNN}(\tilde{G}_{i}^{(j)};\beta_{e}^{(j)}).$
(5)

Based on Eq. 2, the encoder $f_{e}^{(j)}$ is generated by the hypernetwork
$\beta_{e}$ guided by the mode-specific meta information $m^{(j)}$. Therefore,
Eq. 5 becomes ${\bf
z}_{i}^{(j)}=\text{GNN}(\tilde{G}_{i}^{(j)};\beta_{e}(m^{(j)};\gamma_{e}))$,
where $\gamma_{e}$ is mode-invariant and parameterizes all encoders
$\\{f_{e}^{(j)}\\}_{j\in\mathcal{X}}$.

####  4.2.2 Decoder

Once $\\{{\bf z}_{i}^{(j)}\\}_{j\in\mathcal{X}}$ is obtained for all modes in
$\mathcal{X}$, we apply the decoder $f_{d}^{(k)}$ that decodes $\\{{\bf
z}_{i}^{(j)}\\}_{j\in\mathcal{X}}$ and encodes the topology $\tilde{A}^{(k)}$
of the output mode $k\in\mathcal{Y}$ to predict node attributes of
$\tilde{\mathcal{V}}^{(k)}$:

$\displaystyle\hat{\tilde{X}}_{i}^{(k)}=f_{d}^{(k)}(\tilde{A}^{(k)},\sigma_{1}(\\{{\bf
z}_{i}^{(j)}\\}_{j\in\mathcal{X}});\beta_{d}^{(k)})=\text{MLP}(\text{GNN}(\tilde{A}^{(k)},\sigma_{1}(\\{{\bf
z}_{i}^{(j)}\\}_{j\in\mathcal{X}});\beta_{d,\text{GNN}}^{(k)});\beta_{d,\text{MLP}}^{(k)}),$
(6)

where the Multilayer Perceptron (MLP) serves as the prediction layer and
$\beta_{d}^{(k)}=\\{\beta_{d,\text{GNN}}^{(k)},\beta_{d,\text{MLP}}^{(k)}\\}$,
generated by the hypernetwork $\beta_{d}$ with mode-specific meta information
$m^{(k)}$. Then Eq. 6 becomes:

$\displaystyle\hat{\tilde{X}}_{i}^{(k)}=\text{MLP}(\text{GNN}(\tilde{A}^{(k)},\sigma_{1}(\\{{\bf
z}_{i}^{(j)}\\}_{j\in\mathcal{X}});\beta_{d}(m^{(k)};\gamma_{d}));\beta_{d}(m^{(k)};\gamma_{d})),$
(7)

where $\gamma_{d}$ is model-invariant and parameterizes all target decoders
$\\{f_{d}^{(k)}\\}_{k\in\mathcal{Y}}$. We further define $\mathcal{V}$ as the
set of all nodes contained in $\mathcal{G}$ so that $|\mathcal{V}|=p$. Since
$p\geq\tilde{p}_{j}=\tilde{p}_{k}$, now we have only predicted node attributes
of $\tilde{\mathcal{V}}^{(k)}$, and the attributes of the remaining nodes
$\mathcal{V}\setminus\tilde{\mathcal{V}}^{(k)}$ still need to be predicted.

####  4.2.3 Semi-supervised link prediction

We adopt the semi-supervised link prediction to complete the topology of the
mode $k$ using graph auto-encoder (Kipf and Welling, 2016) under the
supervision of $\tilde{A}^{(k)}$:

$\displaystyle{\bf
h}_{i}^{(k)}=\text{GNN}(\tilde{A}^{(k)},\hat{\tilde{X}}_{i}^{(k)};\phi),\quad\hat{A}^{(k)}=\text{Sigmoid}({\bf
h}_{i}^{(k)}\cdot({\bf h}_{i}^{(k)})^{T})$ (8)

Then we compute the Binary Cross Entropy (BCE) between $\tilde{A}^{(k)}$ and
$\hat{A}^{(k)}$ as the second term of the learning objective:

$\displaystyle\mathcal{L}_{2}(\phi)=\sum_{k\in\mathcal{Y}}\text{BCE}(\tilde{A}^{(k)},\hat{A}^{(k)})=-\sum_{k\in\mathcal{Y}}\sum_{s=1}^{p_{k}}\sum_{t=1}^{p_{k}}\\{\tilde{A}_{st}^{(k)}\log\hat{A}_{st}^{(k)}+(1-\tilde{A}_{st}^{(k)})\log(1-\hat{A}_{st}^{(k)})\\}$
(9)

Once Eq. 8 is trained and $\hat{\phi}$ is learned, we perform link prediction
and update $\tilde{A}^{(k)}$ as follows:

$\displaystyle\bar{\bf
h}_{i}^{(k)}=\text{GNN}(\bar{A}^{(k)},\bar{X}_{i}^{(j\rightarrow
k)};\hat{\phi}),\ \ \tilde{A}^{(k)}\leftarrow\text{Sigmoid}(\bar{\bf
h}^{(k)}\cdot(\bar{\bf h}^{(k)})^{T}),$ (10)

where $\bar{A}^{(k)}$ is the diagonal block matrix with $\tilde{A}^{(k)}$ and
the identity matrix
$I\in\mathbb{R}^{(p-\tilde{p}_{k})\times(p-\tilde{p}_{k})}$ as diagonal
blocks. Since the node attributes of $\tilde{\mathcal{V}}^{(k)}$ has been
predicted in Eq. 7, we only need to impute the attributes of
$\mathcal{V}\setminus\tilde{\mathcal{V}}^{(k)}$ with the corresponding
attributes of modes in $\mathcal{X}$ as the input of GNN$(\cdot)$. Therefore,
$\bar{X}_{i}^{(j\rightarrow
k)}=[\hat{\tilde{X}}_{i}^{(k)},\sigma_{2}(\\{X_{i}^{(j)}[\tilde{p}_{k}:]\\}_{j\in\mathcal{X}})]$
is the concatenation of previously predicted attributes
$\hat{\tilde{X}}^{(k)}$ (Eq. 7) and the aggregated attributes of
$\mathcal{V}\setminus\tilde{\mathcal{V}}^{(k)}$ in modes $j\in\mathcal{X}$ via
the pooling function $\sigma_{2}(\cdot)$. $\sigma_{2}(\cdot)$ is the mean
pooling function across modes in $\mathcal{X}$ in implementation.

Once $\tilde{A}^{(k)}$ is updated by Eq. 10, we apply the decoder again in Eq.
7 to predict attributes of $\mathcal{V}\setminus\tilde{\mathcal{V}}^{(k)}$:

$\displaystyle\hat{\bar{X}}_{i}^{(k)}=\text{MLP}(\text{GNN}(\tilde{A}^{(k)},\bar{X}_{i}^{(j\rightarrow
k)};\beta_{d}(m^{(k)};\gamma_{d}));\beta_{d}(m^{(k)};\gamma_{d}))[\tilde{p}_{k}:],$
(11)

Finally, the predicted node attributes in $k\in\mathcal{Y}$ are
$\hat{X}_{i}^{(k)}=[\hat{\tilde{X}}_{i}^{(k)},\hat{\bar{X}}_{i}^{(k)}]$.

###  4.3 Domain generalization via hypernetworks

In this section, we propose the encoder hypernetwork ($\beta_{e}$ in Figure 2
(B)), the decoder hypernetwork ($\beta_{d}$ in Figure 2 (B)), and the
algorithm to learn them. The similarity among input and output modes is
captured by meta information $m^{\mathcal{X}}=\\{m^{(j)}\\}_{j\in\mathcal{X}}$
and $m^{\mathcal{Y}}=\\{m^{(k)}\\}_{j\in\mathcal{Y}}$, respectively, which
guide the encoder and the decoder hypernetwork to produce mode-specific
encoders (i.e., $\\{f_{e}^{(j)}\\}_{j\in\mathcal{X}}$) and decoders (i.e.,
$\\{f_{d}^{(k)}\\}_{k\in\mathcal{Y}}$). When generalizing to unseen target
domains, $\beta_{e}$ and $\beta_{d}$ can produce encoders and decoders of
unseen modes given their meta information.

####  4.3.1 Learning phase

In the training process, we learn parameters $\gamma_{e}$, $\gamma_{d}$ of the
encoder hypernetwork $\beta_{e}$ and the decoder hypernetwork $\beta_{d}$,
respectively, on the source domain
$\mathcal{S}=\\{\mathcal{X}\times\mathcal{Y}:\mathcal{X}\in\mathcal{P}(\mathcal{G}),\mathcal{Y}\in\mathcal{P}(\mathcal{G}-\mathcal{X})\\}$.
Specifically, we minimize the learning objective $\mathcal{L}$ of
MultiHyperGNN:

$\displaystyle\mathcal{L}=\mathcal{L}_{1}(\gamma_{e},\gamma_{d})+\rho\cdot\mathcal{L}_{2}(\phi),$
(12)

where $\mathcal{L}_{1}$ and $\mathcal{L}_{2}$ are obtained from Eq. 4 and Eq.
9, respectively, $\rho$ is the hyperparameter, and $\phi$ is another trainable
paratemer for semi-supervised link prediction. In implementation, $\beta_{e}$
and $\beta_{d}$ are approximated by MLPs. The learning phase is also depicted
in Algorithm 1.

Input: $\\{\mathcal{\tilde{G}}^{(j)},m^{(j)}\\}_{j\in\mathcal{X}}$: topology,
node attributes and meta information of source modes

Input: $\\{\tilde{A}^{(k)},m^{(k)}\\}_{k\in\mathcal{Y}}$: topology and meta
information of target modes

Input: Initialized parameters $\gamma_{e}$, $\gamma_{d}$ and $\phi$

Output: $\\{\hat{X}_{i}^{(k)}\\}_{k\in\mathcal{Y}}$, i=1,2,…,n

1 while  _Converge_ do

2 for  _$k\in\mathcal{Y}$_ do

3 Compute attributes $\hat{\tilde{X}}_{i}^{(k)}$ of
$\tilde{\mathcal{G}}^{(k)}$ via Eq. 7 for each sample $i$ in mode $k$

4 Assemble $\bar{A}^{(k)}$ and $\bar{X}_{i}^{j\rightarrow k}$ as in Eq. 10

5 Perform link prediction and update $\tilde{A}^{(k)}$ via Eq. 10

6 Compute attributes of the remaining nodes $\hat{\bar{X}}_{i}^{(k)}$ in
$\mathcal{V}\setminus\tilde{\mathcal{V}}^{(k)}$ via Eq. 11

7 Concatenate predicted attributes of all nodes:
$\hat{X}_{i}^{(k)}=[\hat{\tilde{X}}_{i}^{(k)},\hat{\bar{X}}_{i}^{(k)}]$

8 Compute $\mathcal{L}_{1}(\gamma_{e},\gamma_{d})$ via Eq. 4,
$\mathcal{L}_{2}(\phi)$ via Eq. 9 and $\mathcal{L}$ via Eq. 12

9 Update $\gamma_{e}$, $\gamma_{d}$ and $\phi$ by stochastic gradient descent
on $\mathcal{L}$

Algorithm 1 Algorithm of learning phase

####  4.3.2 Generalization phase

Once $\hat{\gamma}_{e}$, $\hat{\gamma}_{d}$ are learned as parameters of
$\beta_{e}$ and $\beta_{d}$, respectively, we generalize the model to the
unseen target domain
$\mathcal{T}=\mathcal{X}^{\mathcal{T}}\times\mathcal{Y}^{\mathcal{T}}$ by
guiding $\beta_{e}$ and $\beta_{d}$ with the meta information of unseen modes
$\\{m^{(j)}\\}_{j\in\mathcal{X}^{\mathcal{T}}}$ and
$\\{m^{(k)}\\}_{k\in\mathcal{Y}^{\mathcal{T}}}$. Following Eq. 1 and Eq. 3, we
have:

$\displaystyle\beta_{e}^{(j)}=\beta_{e}(m^{(j)};\hat{\gamma}_{e}),\
j\in\mathcal{X}^{\mathcal{T}},\quad\beta_{d}^{(k)}=\beta_{d}(m^{(k)};\hat{\gamma}_{d}),\
k\in\mathcal{Y}^{\mathcal{T}}$
$\displaystyle\hat{X}_{i}^{(k)}=f_{d}^{(k)}(A^{(k)},\sigma(\\{f_{e}^{(j)}(G_{i}^{(j)};\beta_{e}^{(j)})\\}_{j\in\mathcal{Y}^{\mathcal{T}}});\beta_{d}^{(k)}),\
\ k\in\mathcal{Y}^{\mathcal{T}}$ (13)

We theoretically prove that in the generalization phase our model can
generalize to $\mathcal{T}$ given sufficient mode-specific meta information.

######  Definition 2 (Generalization error).

Suppose
$X_{i}^{(k)}=f_{\hat{\gamma}_{{\mathcal{X}}\rightarrow{\mathcal{Y}}}}(A^{(k)},\\{G_{i}^{(j)},m^{(j)}\\}_{j\in{\mathcal{X}}^{\mathcal{T}}};\hat{\gamma}_{{\mathcal{X}}\rightarrow{\mathcal{Y}}})+\bm{\epsilon_{i}}$
following Eq. 3, where $k\in{\mathcal{Y}}^{\mathcal{T}}$ and
$\hat{\gamma}_{{\mathcal{X}}\rightarrow{\mathcal{Y}}}$ is estimated during
training process on ${\mathcal{S}}$. We define $\|\bm{\epsilon}_{i}\|_{2}^{2}$
as the generalization error of sample $i$.

######  Definition 3 (Sufficient meta information).

We define $m^{(j)}$ and $m^{(k)}$ as the sufficient meta information of the
prediction $f_{\gamma_{{\mathcal{X}}\rightarrow{\mathcal{Y}}}}$ if
$j\in{\mathcal{X}}$, $k\in{\mathcal{Y}}$, $m^{(j)}$ belongs to the space
$\mathcal{M}$ that is a bijective mapping of the space of sufficient statistic
of ${\mathcal{G}}^{(j)}$, and
$m^{(k)}=\operatorname*{arg\,max}_{m}p({\mathcal{G}}^{(k)}|{\mathcal{G}}^{(j)},m)$.

######  Theorem 1.

For the mode $j\in\mathcal{X}^{T}$ and the mode $k\in\mathcal{Y}^{T}$,
$\mathcal{X}^{\mathcal{T}}\times\mathcal{Y}^{\mathcal{T}}\in\mathcal{T}$,
$m^{(j)}$ and $m^{(k)}$ are sufficient meta information of $j$ and $k$,
compute $\hat{X}_{i}^{(k)}$ following Eq. 3 using $\mathcal{G}^{(j)}$,
$A^{(k)}$, $m^{(j)}$ and $m^{(k)}$ and calculate the generalization error
$\|\bm{\epsilon}_{i}\|_{2}^{2}$ as in Def. 2. Then compute
$\\{\hat{X}_{i}^{(k)^{\prime}}\\}_{k\in\mathcal{Y}^{\mathcal{T}}}$ following
Eq. 3 using the same input but with $\forall
m^{(j^{\prime})},j^{\prime}\in\mathcal{X}$, $\forall
m^{(k^{\prime})},k^{\prime}\in\mathcal{Y}$ and
$\mathcal{X}\times\mathcal{Y}\in\mathcal{S}$ as the input of $\beta_{e}$ and
$\beta_{d}$. This leads to the generalization error
$\|\bm{\epsilon}_{i}^{\prime}\|_{2}^{2}$. Assume $\bm{\epsilon}_{i}$ in Def. 2
has a Gaussian distribution
$\bm{\epsilon}_{i}\sim\mathcal{N}(\bm{0},\bm{\sigma}^{2})$, then we have
$\|\bm{\epsilon}_{i}\|_{2}^{2}\leq\|\bm{\epsilon}_{i}^{\prime}\|_{2}^{2}$.

The proof of the above theory is in Appendix A. Meta information is especially
critical when producing encoders. The more informative it is, the more
accurate the domain generalization is.

##  5 Experiments

This section reports the results of both quantitative and qualitative
experiments that were performed to evaluate MultiHyperGNN and other competing
models.

###  5.1 Dataset

We conducted experiments on two real-world datasets: (1) Genes. We used gene
expression data from Genotype-Tissue Expression Consortium (Lonsdale et al.,
2013), in which five tissues, whole blood (WB), lung (L), muscle skeletal
(MS), sun-exposed skin (lower leg, LG), not-sun-exposed skin (suprapubic, S)
were used and the gene-gene network was constructed by weighted correlation
network analysis (Langfelder and Horvath, 2008) with the expression values as
the node attributes. Meta information includes tissue type (lung, muscle,
skin), location (trunk, leg, arm), structure (dense, rigid, spongy), function
(movement, protection, gas exchange) and cell types (alveoli and bronchioles,
cylindrical muscle fibers, epithelial cells); (2) Climate. We extracted data
from the Goddard Earth Observing System Composition Forecasting across the US
from 2019-2021. We collected the air temperature (T) for each state capital
and then splitted a day into four modes: early morning (0:00AM-6:00AM), late
morning (6:00AM-12:00PM), afternoon (12:00PM-18:00PM) and night
(18:00PM-0:00AM). To construct the network, we used cities as graph nodes and
air temperature in each city as node attributes. In each time period, two
cities are connected if air temperatures between them have a high Pearson
Correlation. We used the time period indicator (four-element, one-hot vector
to indicate four periods) and various time stamps when collecting data as the
meta information. Detailed introduction and summary statistics of datasets
used are in Appendix B.

Table 1: Evaluation on prediction accuracy.

Model Genes-L Genes-LG Genes-S T-Afternoon T-Night MSE PCC MSE PCC MSE PCC MSE
PCC MSE PCC ED-GNN 1.9810 0.6072 2.1289 0.5795 2.1925 0.5764 59.3010 0.4539
84.0824 0.4187 MHM 2.0126 0.5913 2.0153 0.5312 2.0384 0.5816 61.2798 0.4300
69.8599 0.4207 IN 2.0182 0.6026 2.2019 0.5683 2.1304 0.5377 60.8755 0.4650
71.0456 0.4210 EERM 1.8624 0.6493 1.9035 0.6325 2.1187 0.5931 84.0604 0.4259
83.2518 0.4101 DRAIN 1.9798 0.6132 1.9969 0.6009 2.2100 0.5741 91.4561 0.3987
104.3200 0.4085 HyperGNN-1 2.7566 0.2574 2.8543 0.2494 2.8863 0.2501 129.6152
0.3566 101.0478 0.4095 HyperGNN-2 2.9383 0.2654 3.0230 0.2565 3.0467 0.2574
280.5912 0.3557 400.0514 0.3125 HyperGNN 1.9720 0.6144 2.2040 0.5700 2.1930
0.5799 69.3157 0.4405 70.0319 0.4299 MultiHyperGNN-MLP 2.8958 0.2608 3.4251
0.2736 3.6073 0.2814 104.0525 0.3764 81.9324 0.4122 MultiHyperGNN-S 2.0023
0.6492 2.2420 0.6018 2.2723 0.6153 89.1604 0.4027 75.6518 0.4151
MultiHyperGNN-GCN 1.8023 0.6511 1.9426 0.6340 1.9539 0.6337 89.1321 0.4395
68.7137 0.4216 MultiHyperGNN-GIN 1.7101 0.6654 1.8913 0.6450 1.9046 0.6455
43.5142 0.5155 49.0168 0.4878 MultiHyperGNN-GAT 1.7695 0.6583 1.9107 0.6347
1.8951 0.6470 54.2913 0.4937 60.1922 0.4561

Table 2: Evaluation on domain-generalization accuracy. Each time we train to
predict two output modes and treat another mode as the unseen target mode in
testing process.

Model Genes-L Genes-LG Genes-S MSE PCC MSE PCC MSE PCC ED-GNN 2.2387 0.4752
2.0573 0.5229 2.0425 0.5511 IN 2.1017 0.5312 2.1539 0.5249 2.3746 0.4795 EERM
2.2148 0.5193 2.3536 0.4583 2.5792 0.4669 DRAIN 2.8155 0.5123 3.2461 0.4016
3.2777 0.4230 HyperGNN-1 3.7586 0.2359 3.3152 0.2614 3.3011 0.2537 HyperGNN-2
3.1516 0.2338 3.3064 0.2572 3.5869 0.2629 HyperGNN 1.9025 0.6003 2.0471 0.6427
1.9913 0.6236 MultiHyperGNN-MLP 3.0812 0.2150 3.1519 0.2963 3.6322 0.3049
MultiHyperGNN-GCN 1.8513 0.6495 2.0086 0.6410 1.9965 0.6127 MultiHyperGNN-GIN
1.8005 0.6600 1.9852 0.6479 1.9031 0.6471 MultiHyperGNN-GAT 1.8069 0.6562
2.0123 0.6455 1.8921 0.6425

###  5.2 Evaluation metrics

We evaluated the model performance both quantitatively and qualitatively. For
quantitative evaluation, we measured prediction accuracy based on Mean Squared
Error (MSE) and Pearson Correlation Coefficients (PCC). To evaluate the
efficiency, we theoretically analyzed the space complexity of MultiHyperGNN
and other models. For qualitative evaluation, we visualized the distribution
between predicted and ground-truth node attributes in unseen modes during the
testing process.

###  5.3 Competing models and ablation studies

We employed five competing models to compare with MultiHyperGNN regarding
prediction and domain generalization: (1) ED-GNN. We modified MultiHyperGNN to
a naive encoder-decoder-based graph transformation model by directly training
the encoder and the decoder for each mode combination. A single model is
trained for all mode combinations; (2) Multi-Head Model (MHM). Following
(Vandenhende et al., 2021), we modified ED-GNN into a multi-task learning
framework by simultaneously training multiple decoders with the same encoder.
This model can only be used for the prediction purpose instead of domain
generalization since each decoder deals with a specific output mode; (3)
Interaction Networks (IN) (Battaglia et al., 2016). IN models the interactions
and dynamics of nodes in the graph for node-level graph transformation.
Particularly, IN uses only fixed graph topology from the input mode; (4)
Explore-to-Extrapolate Risk Minimization (EERM) (Wu et al., 2022). EERM
employs $p$ context explorers that undergo adversarial training to maximize
the variance of risks across multiple virtual environments. This design
enables the model to extrapolate from a single observed environment; (4) DRAIN
(Bai et al., 2022). DRAIN utilizes a recurrent graph generation approach to
generate dynamic graph-structured neural networks using hypernetworks trained
on various time points. This framework can capture the temporal drift of both
model parameters and data distributions, enabling it to make future
predictions. In addition, we modified MultiHyperGNN to evaluate four different
aspects: (1) HyperGNN. HyperGNN is a simpler version of MultiHyperGNN by
predicting multiple output modes from one single input mode. In this case,
only $\beta_{d}$ was trained; (2) HyperGNN-1. To explore whether a single MLP
prediction layer can predict for all output modes, for HyperGNN-1, we will not
produce MLP layers but only produce GAT layers by hypernetwork; (3)
HyperGNN-2. In our experimental setting the meta information is composed of
mode types (one-hot vector) and other mode-related features. For HyperGNN-2,
we reduced the meta information by only feeding the mode type to
hypernetworks; (4) MultiHyperGNN-S. Graph transformation from multiple input
modes is expected to power the prediction by aggregating from these input
modes. To validate this assumption, during the testing process of
MultiHyperGNN, we will not use only a single input mode as the input data.

###  5.4 Quantitative evaluation

####  5.4.1 Prediction accuracy

Figure 3: Density plot to visualize the gene expression in lung, sun-exposed
skin and not-sun-exposed skin in the testing data.

On Genes, we trained MultiHyperGNN that predicts gene expression in lung
(Genes-L), sun-exposed skin (Genes-LG) and not-sun-exposed skin (Genes-S)
using gene expression of whole blood and muscle skeletal. For MultiHyperGNN-S,
during testing, we used gene expression from whole blood as a single input
mode. We trained HyperGNN and its variations (i.e., HyperGNN-1, HyperGNN-2) to
predict in the same output tissues but only from whole blood as the single
input mode. EERM and DRAIN were also trained from one single mode. For the
dataset Climate, we trained MultiHyperGNN from the air temperature in the
early morning and late morning to predict air temperature in the afternoon
(T-Afternoon) and at night (T-Night). To train HyperGNN and its variations, we
predicted T-Afternoon and T-Night from only late night. EERM and DRAIN were
also trained from one single mode. For ED-GNN and IN, we trained them on all
input-output mode combinations. To train MHM, we followed the same training
strategy as we trained HyperGNN.

As shown in Table 1, MultiHyperGNN achieves superior performance on both
datasets. The MSE of MultiHyperGNN-GIN is 0.1262 (6.43$\%$) smaller than the
second best model, EERM, by average. The PCC of MultiHyperGNN-GIN is 0.0270
(4.32$\%$) higher than the second best model, EERM, by average. This is
expected since MultiHyperGNN involves two input modes so that it is more
expressive than EERM. HyperGNN, MultiHyperGNN-S and other competing models
have comparable results since they all predict from a single input mode. The
performance of HyperGNN-1 is worse, indicating that a mode-specific prediction
layer is still needed. In addition, the deployment of MultiHyperGNN hinges
upon the accessibility of mode-specific meta-information. As evidenced in
Table 1 and Table 2, the utilization of HyperGNN-2, which condenses meta-
information to only the mode type, results in suboptimal prediction accuracy
across almost all settings.

####  5.4.2 Domain generalization

We evaluated the performance of MultiHyperGNN and other models regarding
domain generalization using the Genes dataset. To evaluate the generalization
ability on a specific output mode (e.g., Genes-L), each time we trained the
model to predict another two modes (e.g., Genes-LG, Genes-S) using data of
whole blood and muscle skeletal as input modes. During testing time, we
applied the trained model to the output mode (e.g., Genes-L) and calculated
the prediction accuracy.

Based on Table 2, MultiHyperGNNs shows consistently better performance
compared with other models. Specifically, MultiHyperGNN-GIN has the MSE 0.0840
(4.24$\%$) smaller by average than the second model, HyperGNN, which has the
MSE of 1.9803 by average. MultiHyperGNN-GIN has the PCC 0.0295 (4.74$\%$)
higher by average than the second model, HyperGNN, which has the PCC of 0.6222
by average. The better performance of MultiHyperGNN results from the fact that
MultiHyperGNN predicts from multiple input modes, which is more expressive
than HyperGNN that only achieves single-input, multi-output mode prediction.
The superior performance of MultiHyperGNN and HyperGNN compared with other
models results from the meta information that guides hypernetworks to
generalize the model to unseen domains.

####  5.4.3 Space complexity and implementation details

We compared MultiHyperGNN with other models by the theoretical space
complexity analysis. To train a predictive mapping that covers all mode
combinations in ${\mathcal{S}}$, ED-GNN, MHM and IN have $O(3^{N})$ encoders
and decoders in total that need to be trained, leading to $O(3^{N})$ space
complexity. EERM requires to train $p$ classifiers whereas DRAIN only needs a
hypernetwork to produces classifiers at each time point. Therefore, EERM and
DRAIN have the space complexity of $O(p)$ and $O(1)$, respectively. To train
MultiHyperGNN, instead, we only need to train two hypernetworks, whose space
complexity is $O(1)$ which is much smaller than competing models except DRAIN.

Figure 4: Density plots to visualize the distribution of predicted and ground-
truth gene expression in testing data, including density plot for (A) lung,
(B) sun-exposed skin and (C) not-sun-exposed skin.

###  5.5 Qualitative evaluation

We visualized the distribution of node attributes in different modes of Genes.
As shown in Figure 3, in the testing data the distribution of sun-exposed skin
is similar to the not-sun-exposed skin. This is reasonable since both are skin
tissues and they share similar meta information. By contrast, lung is
different from skin, so that its distribution is different from two skin
tissues. This also confirms the necessity to design the model to handle mode
similarities. We also visualized via density plots the alignment of the
distribution of predicted values with the ground-truth distribution in unseen
testing data (Figure 4) corresponding to the results in Table 2. Based on the
results in Figure 4, in all three human tissues, MultiHyperGNN achieves
roughly the same distribution with the ground-truth distribution of the
testing data, which is much better than other competing models. This is
aligned with the superior prediction accuracy in domain generation of
MultiHyperGNN as shown in Table 2.

##  6 Conclusion

In this paper, we attempt to tackle challenges regarding domain generalization
deep graph transformation. Firstly, we identify three challenges in domain
generalization graph generalization. Then we propose MultiHyperGNN that
includes a encoder and a decoder to respectively encode graph topologies in
input and output modes. Two novel hypernetworks are designed to produce the
encoder and the decoder, guided by the mode-specific meta information for
domain generalization. Comprehensive experiments were conducted on real-world
datasets and our model shows superior performance than competing models.
Further exploration is warranted to determine the crucial components of meta-
information that should be incorporated to optimize the performance of
MultiHyperGNN.

## References

  * (1)
  * Alaluf et al. (2022) Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. 2022.  Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In _Proceedings of the IEEE/CVF conference on computer Vision and pattern recognition (ICCV)_. 18511–18521.
  * Bai et al. (2022) Guangji Bai, Chen Ling, and Liang Zhao. 2022.  Temporal Domain Generalization with Drift-Aware Dynamic Neural Networks.  _arXiv preprint arXiv:2205.10664_ (2022).
  * Battaglia et al. (2016) Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. 2016\.  Interaction networks for learning about objects, relations and physics.  _Conference on Neural Information Processing Systems (NeurIPS)_ 29 (2016).
  * Chen et al. (2022) Chaoqi Chen, Jiongcheng Li, Xiaoguang Han, Xiaoqing Liu, and Yizhou Yu. 2022.  Compound domain generalization via meta-knowledge encoding. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (ICCV)_. 7119–7129.
  * Dou et al. (2019) Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. 2019.  Domain generalization via model-agnostic learning of semantic features.  _Conference on Neural Information Processing Systems (NeurIPS)_ 32 (2019).
  * Du et al. (2021a) Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang. 2021a.  Adarnn: Adaptive learning and forecasting of time series. In _Proceedings of the 30th ACM international conference on information & knowledge management (CIKM)_. 402–411.
  * Du et al. (2021b) Yuanqi Du, Shiyu Wang, Xiaojie Guo, Hengning Cao, Shujie Hu, Junji Jiang, Aishwarya Varala, Abhinav Angirekula, and Liang Zhao. 2021b.  Graphgt: Machine learning datasets for graph generation and transformation. In _Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Garg et al. (2021) Vikas Garg, Adam Tauman Kalai, Katrina Ligett, and Steven Wu. 2021\.  Learn to expect the unexpected: Probably approximately correct domain generalization. In _International Conference on Artificial Intelligence and Statistics (AISTATS)_. PMLR, 3574–3582.
  * Gong et al. (2019) Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool. 2019\.  Dlow: Domain flow for adaptation and generalization. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (ICCV)_. 2477–2486.
  * Guo et al. (2022) Xiaojie Guo, Lingfei Wu, and Liang Zhao. 2022.  Deep graph translation.  _IEEE Transactions on Neural Networks and Learning Systems (TNNLS)_ (2022).
  * Guo et al. (2019) Xiaojie Guo, Liang Zhao, Cameron Nowzari, Setareh Rafatirad, Houman Homayoun, and Sai Manoj Pudukotai Dinakarrao. 2019.  Deep multi-attributed graph translation with node-edge co-evolution. In _2019 IEEE International Conference on Data Mining (ICDM)_. IEEE, 250–259.
  * Ha et al. (2017) David Ha, Andrew Dai, and Quoc V Le. 2017.  Hypernetworks.  _The International Conference on Learning Representations (ICLR)_ (2017).
  * Huang et al. (2021) Yizhou Huang, Kevin Xie, Homanga Bharadhwaj, and Florian Shkurti. 2021.  Continual model-based reinforcement learning with hypernetworks. In _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 799–805.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Variational graph auto-encoders.  _arXiv preprint arXiv:1611.07308_ (2016).
  * Langfelder and Horvath (2008) Peter Langfelder and Steve Horvath. 2008.  WGCNA: an R package for weighted correlation network analysis.  _BMC bioinformatics_ 9, 1 (2008), 1–13.
  * Li et al. (2020) Xiang Li, Wei Zhang, Hui Ma, Zhong Luo, and Xu Li. 2020.  Domain generalization in rotating machinery fault diagnostics using deep neural networks.  _Neurocomputing_ 403 (2020), 409–420.
  * Li et al. (2018) Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. 2018\.  Diffusion convolutional recurrent neural network: Data-driven traffic forecasting.  _The International Conference on Learning Representations (ICLR)_ (2018).
  * Lin et al. (2020) Yucheng Lin, Huiting Hong, Xiaoqing Yang, Xiaodi Yang, Pinghua Gong, and Jieping Ye. 2020\.  Meta Graph Attention on Heterogeneous Graph with Node-Edge Co-evolution.  _arXiv preprint arXiv:2010.04554_ (2020).
  * Lonsdale et al. (2013) John Lonsdale, Jeffrey Thomas, Mike Salvatore, Rebecca Phillips, Edmund Lo, Saboor Shad, Richard Hasz, Gary Walters, Fernando Garcia, Nancy Young, et al. 2013\.  The genotype-tissue expression (GTEx) project.  _Nature genetics_ 45, 6 (2013), 580–585.
  * Pan et al. (2022) Bo Pan, Yinkai Wang, Xuanyang Lin, Muran Qin, Yuanqi Du, Shiva Ghaemi, Aowei Ding, Shiyu Wang, Saleh Alkhalifa, Kevin Minbiole, et al. 2022\.  Property-Controllable Generation of Quaternary Ammonium Compounds. In _2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_. IEEE, 3462–3469.
  * Qian et al. (2021) Hangwei Qian, Sinno Jialin Pan, and Chunyan Miao. 2021.  Latent independent excitation for generalizable sensor-based cross-person activity recognition. In _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , Vol. 35. 11921–11929.
  * Qu et al. (2022) Jingang Qu, Thibault Faney, Ze Wang, Patrick Gallinari, Soleiman Yousef, and Jean-Charles de Hemptinne. 2022.  HMOE: Hypernetwork-based Mixture of Experts for Domain Generalization.  _arXiv preprint arXiv:2211.08253_ (2022).
  * Quinonero-Candela et al. (2008) Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. 2008.  _Dataset shift in machine learning_.  Mit Press.
  * Rezaei-Shoshtari et al. (2023) Sahand Rezaei-Shoshtari, Charlotte Morissette, Francois Robert Hogan, Gregory Dudek, and David Meger. 2023.  Hypernetworks for Zero-shot Transfer in Reinforcement Learning.  _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ (2023).
  * Sendera et al. (2023) Marcin Sendera, Marcin Przewięźlikowski, Konrad Karanowski, Maciej Zięba, Jacek Tabor, and Przemysław Spurek. 2023.  Hypershot: Few-shot learning by kernel hypernetworks. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (ICCV)_. 2469–2478.
  * Sun et al. (2017) Zhun Sun, Mete Ozay, and Takayuki Okatani. 2017.  HyperNetworks with statistical filtering for defending adversarial examples.  _arXiv preprint arXiv:1711.01791_ (2017).
  * Van Belle et al. (2022) Rafaël Van Belle, Charles Van Damme, Hendrik Tytgat, and Jochen De Weerdt. 2022.  Inductive graph representation learning for fraud detection.  _Expert Systems with Applications_ 193 (2022), 116463.
  * Vandenhende et al. (2021) Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. 2021.  Multi-task learning for dense prediction tasks: A survey.  _IEEE transactions on pattern analysis and machine intelligence (TPAMI)_ (2021).
  * Volk et al. (2022) Tomer Volk, Eyal Ben-David, Ohad Amosy, Gal Chechik, and Roi Reichart. 2022.  Example-based hypernetworks for out-of-distribution generalization.  _arXiv preprint arXiv:2203.14276_ (2022).
  * Wang et al. (2020) Bailin Wang, Mirella Lapata, and Ivan Titov. 2020\.  Meta-learning for domain generalization in semantic parsing.  _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ (2020).
  * Wang et al. (2022) Shiyu Wang, Xiaojie Guo, Xuanyang Lin, Bo Pan, Yuanqi Du, Yinkai Wang, Yanfang Ye, Ashley Petersen, Austin Leitgeb, Saleh AlKhalifa, et al. 2022\.  Multi-objective Deep Data Generation with Correlated Property Control.  _Conference on Neural Information Processing Systems (NeurIPS)_ 35 (2022), 28889–28901.
  * Wu et al. (2022) Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. 2022\.  Handling distribution shifts on graphs: An invariance perspective.  _The International Conference on Learning Representations (ICLR)_ (2022).
  * Yu et al. (2018) Bing Yu, Haoteng Yin, and Zhanxing Zhu. 2018.  Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting.  _International Joint Conference on Neural Networks (IJCNN)_ (2018).
  * Zhang et al. (2022) Zhengkun Zhang, Wenya Guo, Xiaojun Meng, Yasheng Wang, Yadao Wang, Xin Jiang, Qun Liu, and Zhenglu Yang. 2022\.  Hyperpelt: Unified parameter-efficient language model tuning for both language and vision-and-language tasks.  _arXiv preprint arXiv:2203.03878_ (2022).
  * Zheng et al. (2020) Huailiang Zheng, Yuantao Yang, Jiancheng Yin, Yuqing Li, Rixin Wang, and Minqiang Xu. 2020\.  Deep domain generalization combining a priori diagnosis knowledge toward cross-domain fault diagnosis of rolling bearing.  _IEEE Transactions on Instrumentation and Measurement_ 70 (2020), 1–11.