# SHINE: SubHypergraph Inductive Neural nEtwork

Yuan Luo
Feinberg School of Medicine
Northwestern University
Chicago, IL 60611
<EMAIL_ADDRESS>

###### Abstract

Hypergraph neural networks can model multi-way connections among nodes of the
graphs, which are common in real-world applications such as genetic medicine.
In particular, genetic pathways or gene sets encode molecular functions driven
by multiple genes, naturally represented as hyperedges. Thus, hypergraph-
guided embedding can capture functional relations in learned representations.
Existing hypergraph neural network models often focus on node-level or graph-
level inference. There is an unmet need in learning powerful representations
of subgraphs of hypergraphs in real-world applications. For example, a cancer
patient can be viewed as a subgraph of genes harboring mutations in the
patient, while all the genes are connected by hyperedges that correspond to
pathways representing specific molecular functions. For accurate inductive
subgraph prediction, we propose SubHypergraph Inductive Neural nEtwork
(SHINE). SHINE uses informative genetic pathways that encode molecular
functions as hyperedges to connect genes as nodes. SHINE jointly optimizes the
objectives of end-to-end subgraph classification and hypergraph nodes’
similarity regularization. SHINE simultaneously learns representations for
both genes and pathways using strongly dual attention message passing. The
learned representations are aggregated via a subgraph attention layer and used
to train a multilayer perceptron for inductive subgraph inferencing. We
evaluated SHINE against a wide array of state-of-the-art (hyper)graph neural
networks, XGBoost, NMF and polygenic risk score models, using large scale NGS
and curated datasets. SHINE outperformed all comparison models significantly,
and yielded interpretable disease models with functional insights.

##  1 Introduction

Hypergraph neural networks have recently emerged as a series of successful
methods to model multi-way connections that are beyond pairwise associations
among nodes of the graphs. Multi-way connections are common in many real-world
applications and, in particular, genetic medicine. From genetic medicine’s
perspective, pathways or broadly speaking gene sets encode the relationship
among multiple genes that collectively correspond to a molecular function [1],
which can be used in machine learning models to account for disease mechanisms
more intuitively and accurately than individual genes [2]. Genetic pathways or
gene sets encode functional relations among multiple genes (see Appendix for
detailed explanations), which can be naturally modeled as hyperedges
connecting all involved nodes (e.g., genes). Thus, hypergraph-guided embedding
can capture functional relations in learned representations.

Existing hypergraph neural network models often adopt the semi-supervised
learning (SSL) paradigm to assign labels to initially unlabeled nodes in a
hypergraph [3, 4, 5]. Other methods have focused on learning graph
representations [6, 7]. Node-level and graph-level representations give either
local or overarching views of the graphs, i.e., at the two extremes of
hypergraph topological structures. There is an unmet need in learning powerful
representations of subgraphs in hypergraphs. Such capabilities are important
in real-world applications such as genetic medicine. For example, cancer
patients can be viewed as subgraphs of genes that harbor mutations, while all
the genes are connected by hyperedges that correspond to pathways or gene sets
representing specific molecular functions. Powerful subgraph representations
will lead to the capability to more accurately account for the patient’s
pathophysiology. For regular graphs where edges connect node pairs, several
subgraph representation learning algorithms were proposed, including methods
that can use the learned representations to make predictions for subgraphs
with fixed sizes [8] or varying sizes [9]. There are currently few if any work
on inductive inference for varying-sized subhypergraphs. In this work, we
propose a new framework named SHINE: SubHypergraph Inductive Neural nEtwork.
We share our source code at https://github.com/luoyuanlab/SHINE. Our
contributions are as follows:

  * •

To the best of our knowledge, SHINE is the first model to effectively learn
subgraph representations for hypergraphs, use the learned representations (for
seen subgraphs) and inductively infer representations (for unseen subgraphs)
for downstream subgraph predictions.

  * •

Novel applications in the field of genetic medicine on Next Generation
Sequencing (NGS) datasets across diverse diseases show significant performance
improvements by SHINE over a wide array of state-of-the-art baselines.

  * •

In addition to learning and inductively inferring subgraph representations,
SHINE simultaneously learns the representations of nodes and hyperedges. This
brings interpretation advantages, allowing assessing pathways (hyperedges)
correlations and reasoning on multiple molecular functions mutually
interacting and collectively contributing to disease onset and progression.

##  2 Related Work

Graph Neural Networks. Graph representation learning maps graphs or their
components to vector representations and has attracted growing attention over
the past decade. Recently, graph neural networks (GNNs), which can learn a
distributed representation for a graph or a node in a graph, are widely
applied to a variety of areas including computer vision and image processing
[10, 11], molecular structure inference [12, 13], natural language processing
[14, 15, 16], and healthcare [17, 18]. GNN recursively updates the
representation of a node in a graph by aggregating the feature vectors of its
neighbors and itself, e.g. [19]. The graph-level representations can then be
obtained through set pooling (e.g., [20]) or graph coarsening (e.g., [21]) to
aggregate the node representations in the graph. The reader is referred to a
comprehensive book [22] on the topic of graph neural networks.

Hypergraph neural network. Hypergraph neural networks [6, 4, 3, 7] have become
a popular approach for learning on multi-way relations from data. Early work
on hypergraph learning, e.g., [23], formulated hypergraph message passing
using spectral theory of hypergraphs. This formulation and its variants [3, 4,
24, 25, 26, 27] essentially adopted clique expansion to extend graph
convolutional network (GCN) for hypergraph learning. Others methods applied
attention mechanism to aggregate the information across the hypergraph [6, 7]
or directly learned node representations to preserve the proximity of nodes
sharing a hyperedge or having similar neighborhoods [28]. In both
formulations, messages were passed to the node of interest from its immediate
neighbors, and added layers allow propagation of messages to a farther
neighborhood. A very recent model implements hypergraph neural network layers
in a generalized way as compositions of two multiset functions that are
approximated by neural networks [29].

Subgraph representation learning and prediction. Recent studies on subgraph
embeddings and prediction starts with learning representations of small
subgraphs. [8] encoded small fixed-sized subgraphs for subgraph evolution
prediction. SubGNN [9] learned representations for varying-sized subgraphs
through neighborhood, position and structure channels using random patches
distributed throughout the graph. [30, 31] learned subgraph representations by
pooling local structures to aid the predictions of the entire graphs. Note
modeling hyperedges as another type of nodes turns hypergraphs into bipartite
(and heterogeneous) graphs, making SubGNN a potential baseline for
subhypergraph inferencing. On the other hand, most of the existing general
heterogeneous graph neural network models do not support subgraph inferencing
[32, 33, 34, 35, 36].

The intersection of hypergraph neural network and subgraph representation
learning is currently underexplored. While the above methods focus on either
hypergraph learning or subgraph learning, none of the methods consider
subgraph prediction for hypergraphs. Technically, subgraphs can be viewed as a
hyperedge and studies on link prediction could predict the existence of a
hyperedge [37]. However, few if any such studies addressed the problems of
differentiating the classes of the subgraphs, which is especially important in
genetic medicine where subgraphs and hyperedges have different real-world
meanings. For example, a hyperedge corresponds to a genetic pathway from
curated knowledge and a subgraph corresponds to a patient with mutated genes
as its nodes.

To sum, there is a major unmet need regarding varying-sized subgraph inference
for hypergraphs, and even more so in the inductive learning setting. Our
proposed framework SHINE provides an end-to-end framework that operates on
hypergraphs and performs inductive subgraph inferencing.

##  3 Methods

We first outline the workflow of SHINE, see Table 1 for symbol definitions. We
develop a strongly dual attention message passing algorithm to propagate
information between nodes and hyperedges, and across layers. We develop a
weighted subgraph attention mechanism to learn the subgraph representation by
integrating representations of hypergraph nodes. We next explain each step.

Table 1: Common notations used throughout the paper. Symbol | Definition | Symbol | Definition
---|---|---|---
$\mathbf{\boldsymbol{\mathcal{H}}}$ | An undirected hypergraph |  $\mathbf{\boldsymbol{\mathcal{N}}}$, $|\mathbf{\boldsymbol{\mathcal{N}}}|$ | Set of hypergraph nodes and size
$\mathbf{\boldsymbol{\mathcal{E}}}$, $|\mathbf{\boldsymbol{\mathcal{E}}}|$ | Set of hypergraph hyperedges and size | $d$ | Hidden layer size
$\mathbf{H}$ | Hyperedge incidence matrix | $\circ$ | Operation composition
$n$ | Number of subgraphs | * | Element-wise multiplication

###  3.1 Collecting Genetic Pathways

We use the Molecular Signatures Database (MSigDB) [1] and focus on MSigDB’s
curated pathway (gene set) collection, which contains human gene sets that are
canonical representations of a biological process compiled by domain experts.
There are 21,587 genes in MSigDB Pathways. Some pathways may overlap with
others and have been filtered by MSigDB to remove interset redundancy. Genes
with unknown functions are not included in the pathways and not used for
classification, as our focus here is on interpretable modeling through
inferencing with known molecular functions. Adding genes with unknown
functions to study their impact will be our future work.

###  3.2 Hypergraph Learning

We first review the hypergraph analysis basics. Different from a simple graph,
a hyperedge in a hypergraph connects two or more vertices. A hypergraph is
defined as $\mathcal{H}=(\mathcal{N},\mathcal{E})$, which includes a set of
nodes $\mathcal{N}$, a set of hyperedges $\mathcal{E}$. In the case of genetic
medicine, we model genes as hypergraph nodes, i.e.,
$\mathcal{N}=\\{g_{1},...,g_{|\mathcal{N}|}\\}$, and pathways as hyperedges,
i.e., $\mathcal{E}=\\{p_{1},...,p_{|\mathcal{E}|}\\}$, where
$|\mathcal{N}|,|\mathcal{E}|$ are sizes of the nodes and hyperedges
respectively. The hypergraph’s topological structure can be denoted by an
$|\mathcal{N}|\times|\mathcal{E}|$ incidence matrix $\mathbf{H}$, whose
entries are defined as

$\displaystyle\mathbf{H}_{ij}=\begin{cases}\quad 1&\text{if node
}g_{i}\in\text{ hyperedge }p_{j}\\\ \quad 0&\text{if node }g_{i}\notin\text{
hyperedge }p_{j}\\\ \end{cases}$ (1)

Generally speaking, each node in the hypergraph may be accompanied by a
$d$-dimensional node feature/embedding matrix $\mathbf{N}\in
R^{|\mathcal{N}|\times d}$, where each row corresponds to a node’s
feature/embedding. The hypergraph with its topological structure and node
features can be represented succinctly as
$\mathcal{H}=(\mathbf{H},\mathbf{N})$.

Fig. 1 (a) shows a schematic of the constructed genome hypergraph with nodes
denoted by circles and hyperedges denoted by colored arcs. While a pathway can
contain multiple genes, a gene can also contribute to multiple pathways. That
is, we can have multiple hyperedges incident on the same node (gene), as can
be seen in Fig. 1 (a) nodes $n_{2},n_{5},n_{10}$.

Figure 1: SHINE’s strongly dual attention mechanism for message passing for
the genome hypergraph, and its use of subgraph attention to integrate gene
nodes in the feature learning for subgraphs.

###  3.3 Strongly Dual Attention Message Passing

Hyperedge attention over nodes. For a hyperedge $p_{j}\in\mathcal{E}$, in
order to update its hidden representation at layer $k$, we aggregate the
information from its incident nodes using the following attention mechanism.
We first calculate the hyperedge attention over nodes as in

$a_{E}(p_{j},g_{i})=\text{exp}\left(\mathbf{c}^{T}\mathbf{s}(p_{j},g_{i})\right)\bigg{/}\left(\textstyle\sum\nolimits_{g_{i^{\prime}}\in
p_{j}}\text{exp}\left(\mathbf{c}^{T}\mathbf{s}(p_{j},g_{i^{\prime}})\right)\right)$
(2)

where $\mathbf{c}$ is a trainable context vector and the attention ready state
$\mathbf{s}(p_{j},g_{i})$ for a hyperedge-node pair $(p_{j},g_{i})$ is
calculated from the $(k-1)$th layer as in

$\mathbf{s}(p_{j},g_{i})=\text{LeakyReLU}\Big{(}\big{(}\mathbf{W}_{N}\mathbf{h}^{k-1}_{N}(g_{i})+\mathbf{b}_{N}\big{)}*\big{(}\mathbf{W}_{E}\mathbf{h}^{k-1}_{E}(p_{j})+\mathbf{b}_{E}\big{)}\Big{)}$
(3)

where $*$ denotes element-wise product, $\mathbf{W}_{N}$ and $\mathbf{b}_{N}$
($\mathbf{W}_{E}$ and $\mathbf{b}_{E}$) are the transformation weights and
bias of the nodes (the hyperedges) for the attention ready state. This is
motivated by the observation that different nodes (genes) contribute
differently to the hyperedges (pathways), thus we need proper attentions
across the nodes to up- or down-weight their contributions when aggregating
their representations to compute the representation of the hyperedge. Once we
have the hyperedge attention over nodes, we calculate the hyperedge’s
representation in layer $k$ from the nodes’ representations in layer $k-1$
(equation 4) where $\sigma$ is the nonlinearity layer (ReLU in our
experiment).

$\mathbf{h}^{k}_{E}(p_{j})=\sigma\left(\textstyle\sum\nolimits_{g_{i}\in
p_{j}}a_{E}(p_{j},g_{i})\;\mathbf{h}^{k-1}_{N}(g_{i})\right)$ (4)

Node attention over hyperedges. For a node $g_{i}\in\mathcal{N}$, in order to
update its hidden representation at layer $k$, we aggregate the information
from its incident hyperedges using the following attention mechanism. We first
calculate the node attention over hyperedges as in

$a_{N}(g_{i},p_{j})=\text{exp}\left(\mathbf{c}^{T}\mathbf{s}(p_{j},g_{i})\right)\bigg{/}\left(\textstyle\sum\nolimits_{p_{j^{\prime}}\ni
g_{i}}\text{exp}\left(\mathbf{c}^{T}\mathbf{s}(p_{j^{\prime}},g_{i})\right)\right)$
(5)

where $\mathbf{c}$ is the same trainable context vector as used in hyperedge
attention calculation and the attention ready state $\mathbf{s}(p_{j},g_{i})$
for hyperedge-node pair $(p_{j},g_{i})$ is calculated as in equation 3. This
allows us to have the node’s attention over the hyperedges, i.e., we can
weight hyperedges’ contributions when aggregating their representations to
compute the representation of the node. We calculate the node’s representation
in layer $k$ from the hyperedges’ representations in layer $k-1$ as in

$\mathbf{h}^{k}_{N}(g_{i})=\sigma\left(\textstyle\sum\nolimits_{p_{j}\ni
g_{i}}a_{N}(g_{i},p_{j})\;\mathbf{h}_{E}^{k-1}(p_{j})\right)$ (6)

Note that different from HyperGAT, here the calculation of hyperedge’s and
node’s attentions share the same underlying dual-attention matrix as shown in
Fig. 1 (b), which is essentially unstandardized covariance matrix. Such
parameter sharing across hyperedges and nodes allows us to cross-regulate the
learning of their mutual attentions to prevent overfitting. This difference
not only allows for simplification of the model, but also is more consistent
with the notion of duality. The dual $\mathcal{H}^{*}$ of the hypergraph
$\mathcal{H}$ is a hypergraph with $\mathcal{H}$’s vertices and edges
interchanged, and we should have $(\mathcal{H}^{*})^{*}=\mathcal{H}$. It is
easily provable that the dual-attentions for $(\mathcal{H}^{*})^{*}$ are the
same as those for $\mathcal{H}$. Such a self-dual statement is generally not
true for the attentions proposed in HyperGAT due to their unsymmetrical way of
calculating the node-level and the edge-level attentions, despite that the
HyperGAT attention was termed as “dual” attention. For this reason, we term
our attention message passing scheme as strongly dual attention message
passing.

Hypergraph regularization. One important intuition about graph and hypergraph
convolutional network is that the learned representations for nodes with
similar context of (hyper)edges should be similar. In the case of a simple
graph $G=(V,E)$, this is to minimize the summed distance $\sum_{(u,v)\in
E}\left\lVert h_{u}-h_{v}\right\rVert^{2}$ or its weighted variants. Instead
of using it as an explicit regularizer, graph or hypergraph convolutional
networks leverage an appropriately defined graph or hypergraph Laplacian. As
noted in [23], the hypergraph Laplacian is
$\mathbf{\Delta}=\mathbf{I}-\mathbf{\Theta}$ where $\mathbf{I}$ is the
identity matrix and $\mathbf{\Theta}$ is defined as (let $\mathbf{W}$ be a
diagonal matrix with diagonal entries as hyperedge weights)

$\displaystyle\mathbf{\Theta}=\mathbf{D}_{v}^{-1/2}\mathbf{H}\mathbf{W}\mathbf{D}_{e}^{-1}\mathbf{H}^{T}\mathbf{D}_{v}^{-1/2}$
(7)

Here, different from hypergraph convolutional networks, we use explicit
regularization on the similarity of representations of nodes with similar
hyperedge context. Let $\mathbf{X}$ be the matrix of the learned nodes’
representations, where row $\mathbf{X}_{i}=\mathbf{h}^{K}_{N}(g_{i})$ and the
$K$th layer is the last hypergraph message passing layer. We can define the
regularizer as

$\displaystyle\mathcal{L}_{reg}=\textstyle\sum\nolimits_{i,j}\left((\mathbf{X}_{i}\mathbf{X}_{i}^{T}-2\mathbf{X}_{i}\mathbf{X}_{j}^{T}+\mathbf{X}_{j}\mathbf{X}_{j}^{T})*\mathbf{\Theta}_{ij}\right)$
(8)

Intuitively, the more hyperedges are incident on the node pair $i,j$, the more
we should penalize their representational differences. On the other hand, the
regularizer down-weights the penalization if a hyperedge connects many nodes
or if a node has many incident hyperedges, indicating lack of specificity for
hyperedges and nodes, respectively.

###  3.4 Weighted Subgraph Attention

The multiple layers of strongly dual attention message passing allow learning
the nodes’ and hyperedges’ representations. However, the instance for the
classification algorithm is a subgraph (e.g., a patient, who has mutations in
multiple genes (nodes)). From the hypergraph perspective, a patient $j(1\leq
j\leq n)$ can be considered as a subhypergraph $\mathcal{G}_{j}$ whose nodes
(genes) have mutations in $j$ and are a subset of those of $\mathcal{H}$. This
is shown in Fig. 1 (c) where different node colors in the hypergraph denote
different patients. In order to calculate the subgraph’s representation from
its component nodes’ representations at the $K$th layer, we use the following
weighted subgraph attention (WSA) mechanism, inspired by [38]. In fact, none
of the previous hypergraph methods support subgraph inferencing, and we had to
add our WSA module to those models for subgraph inferencing as well. We first
compute the subgraph attention over nodes (e.g., $g_{i}$’s) as in

$a(\mathcal{G}_{j},g_{i})=\text{exp}\left(\mathbf{M}_{ji}\mathbf{b}^{T}\mathbf{h}^{K}_{N}(g_{i})\right)\bigg{/}\left(\textstyle\sum\nolimits_{g_{i^{\prime}}\in\mathcal{G}_{j}}\text{exp}\left(\mathbf{M}_{ji^{\prime}}\mathbf{b}^{T}\mathbf{h}^{K}_{N}(g_{i^{\prime}})\right)\right)$
(9)

where $\mathbf{b}$ is a trainable context vector, $\mathbf{M}$ is the mutation
rate feature matrix with each row corresponding to a patient and each column
corresponding to a gene. Thus, equation 9 is a mutation rate weighted subgraph
attention mechanism. This choice conforms to the intuition that the rate of a
mutation is more informative than a categorical indicator of the mutation’s
occurrence. With these subgraph level attentions, we compute the patient
(subgraph) representation from the $K$th layer’s gene representations as in

$\mathbf{h}(\mathcal{G}_{j})=\sigma\left(\textstyle\sum\nolimits_{g_{i}\in\mathcal{G}_{j}}a(\mathcal{G}_{j},g_{i})\;\mathbf{h}^{K}_{N}(g_{i})\right)$
(10)

We then stacked the learned patient representations to form the new patient
feature matrix, as in

$\mathbf{S}=[\mathbf{h}(\mathcal{G}_{1})^{T}\;|\;\mathbf{h}(\mathcal{G}_{2})^{T}\;|\;...\;|\;\mathbf{h}(\mathcal{G}_{n})^{T}]^{T}$
(11)

where each row is a patient (subgraph) embedding.

###  3.5 Inductive Classification on Subgraphs

Let the learned feature matrix be $\mathbf{S}$ and feed it into a softmax
classifier

$\mathbf{Z}=\text{softmax}(\mathbf{W}^{(1)}\;(ReLU\circ
FC)^{(2)}(\mathbf{S})\>+\>\mathbf{W}^{(0)})$ (12)

where (2) in the superscript indicates two MLP layers ($FC$=Fully Connected
layer). The loss function is defined as the cross-entropy error over all
subjects in all classes as in

$\mathcal{L}=-\sum\nolimits_{j\in\mathcal{Y}_{D}}\sum\nolimits_{f=1}^{F}\mathbf{Y}_{jf}\ln\mathbf{Z}_{jf}+\mathcal{L}_{reg}$
(13)

where $\mathcal{Y}_{D}$ is the training set of subjects that have labels and
$F$ is the dimension of the output labels, which is equal to the number of
classes. $\mathbf{Y}$ is the label indicator matrix. Note that the subgraph
attention layer allows us to compute any patient’s representation, which
effectively eliminates the need for access to test set patient features during
training, making the model inductive. Existing models such as HyperGCN and
HGNN are transductive, we cascade our subgraph attention layer on top of these
models and make them inductive to serve as our comparison models.

##  4 Experiments

We conducted experiments on real-world datasets in genetic medicine. Both
datasets have more than 20 different classes, indicating significant
complexity of the prediction tasks. These datasets are different in nature,
e.g., curated from literature vs. obtained directly from high-throughput
sequencing, and multi-class vs. multi-class multi-label. Our experiments are
motivated by the fact that massive genomic data call for novel methods and
present unique technical challenges, in this case inductive subgraph
inferencing on hypergraph. The summary statistics for each dataset is shown in
Table 2, and the description of each dataset is as follows. Most pathways have
small to medium sizes, see Table 2 pathway sizes IQR. In fact, even at the
95th percentile, the pathway size is just over 200. On the other hand, we
observed that the larger the pathway (hyperedge), the more subgraph it is
incident on, and the less attention our model will give it as a discriminative
feature. The DisGeNet and the TCGA-MC3 datasets are publicly
available111DisGeNet: https://www.disgenet.org/; TCGA-MC3:
https://gdc.cancer.gov/about-data/publications/mc3-2017, and this study is
approved by Northwestern University Institutional Review Board.

###  4.1 Disease Type Prediction with DisGeNet Data

In this experiment, we have used the DisGeNet dataset [39] that is a
collection of mutated genes involved in human diseases compiled from expert
curated repositories, GWAS catalogs, animal models and the scientific
literature. In the following text, we abuse terminology to use “gene” to
really mean "variants in the gene". We model genes as hypergraph nodes and
diseases as hyperedges. Each disease is labeled with one or more of 22 MeSH
codes, and the task is a multi-class multi-label classification problem. We
used 6:2:2 train:validation:test partition, and the split distribution is
shown in Appendix. The DisGeNet dataset has 6226 pathways and 9133 genes
involved in 8383 diseases.

###  4.2 Cancer Type Prediction with NGS Somatic Mutations Data

In this experiment, we have used the consensus somatic mutations for TCGA
subjects produced by the Multi-Center Mutation Calling in Multiple Cancers
(MC3) project [40]. Aiming to enable robust cross-tumor-type analyses, the MC3
approach applied an ensemble of 7 mutation-calling algorithms and assigned a
PASS identifier to a mutation that was called by 2 or more variant callers out
of the total 7 callers [40]. The MC3 approach accounted for variance and batch
effects introduced by the rapid advancement of DNA extraction, hybridization-
capture, and sequencing over time. Following this approach, we restricted our
analysis to PASS calls in order to maintain sample sizes and uniformity in
mutation calling. Each subject is labeled with one of 25 cancer types, and the
task is a multi-class classification problem. We used 6:2:2
train:validation:test partition, stratified by cancer types, and the split
distribution is shown in Appendix. The TCGA-MC3 dataset has 6229 pathways and
18059 genes involved in 9012 subjects in total.

Table 2: Real-world hypergraph datasets for subgraph inference used in our work. For hyperedge sizes, shown are medians and interquartile ranges. From the distribution, it is clear that the hyperedge size has a positive skewness. Dataset | # hypernodes | # hyperedges | Hyperedge size | # classes | # subgraphs
---|---|---|---|---|---
DisGeNet | 9133 | 6226 | 25 (12 - 57) | 22 | 8383
TCGA-MC3 | 18059 | 6229 | 33 (15 - 77) | 25 | 9012

###  4.3 Baselines

We compared SHINE with the following state-of-the-art baselines, we use
validation datasets to tune parameters and hyperparameters, please see
Appendix for details.

  * •

Hypergraph neural networks (HGNN) [3] uses clique expansion to transform the
hypergraph to graph and uses Chebyshev approximation to derive a simplified
hypergraph convolution operation.

  * •

HyperGCN [4] represents a hyperedge by a selected pairwise simple edge
connecting two most unlike nodes, and adds the remaining nodes in the
hyperedge as mediators.

  * •

HyperGAT [7] learns node representations by aggregating information from nodes
to edges and vice versa. Different from SHINE, HyperGAT uses alternating
attention instead of strictly dual attention and has no regularization on
nodes with similar context of hyperedges.

  * •

AllSetTransformer and AllDeepSets are two variants (attention-based and MLP-
based respectively) of set-based methods (e.g., compositions of two multiset
functions that are permutation invariant on their input multisets) for
exploiting hyperedges in hypergraphs [29].

  * •

SubGNN [9] was applied to the hypergraph by viewing the nodes and hyperedges
as two types of vertices of a bipartite graph [41].

  * •

The multilayer perceptron (MLP) baseline evaluates how a simple feed-forward
neural network with hypergraph regularization and subgraph attention performs,
by replacing dual attention with MLP.

  * •

Polygenic risk score (PRS) is currently a widely used standard practice in
genetic medicine and calculates disease risk from genotype profile using
regression [42].

  * •

Non-negative matrix factorization (NMF) discovers low-dimensional structure
from high-dimensional multi-omic data and enables inference of complex
biological processes [43].

  * •

XGBoost is an end-to-end tree boosting system and a state-of-the-art machine
learning method [44] that frequently achieves the top results on many machine
learning challenges.

To assess whether performance changes are due to added information (e.g.,
pathway information) and/or better utilization of the added information, we
run PRS, NMF, and XGBoost in the following three settings: gene features only,
pathway features only, and both gene and pathway features.

##  5 Results

The held-out test set micro-averaged F1 scores (micro-F1) for our proposed
method SHINE and all the other comparison models are in Table 3. Comparing all
the models, we can see that SHINE clearly outperforms a comprehensive array of
state-of-the-art baselines in various configurations, with non-overlapping
standard deviation intervals. PRS is indeed a competitive baseline, as can be
seen from its close performance compared with XGBoost that frequently topped
many machine learning challenges’ leaderboards. Previously state-of-the-art
hypergraph neural network models (HyperGCN, HGNN, HyperGAT) do not always
outperform PRS and XGBoost (e.g., on the TCGA-MC3 dataset). On the other hand,
pathway as features could improve performance if used properly, whether alone
or jointly with genes. This comparison shows that genetic pathway information
is useful to disease type classification, consistent to the intuition that
pathways encode molecular functional mechanisms that underlie the disease
etiology. However, properly utilizing such information is non-trivial, as
evidenced by the difficulty to outperform PRS and XGBoost models by NMF models
and hypergraph models including HyperGCN, HGNN and HyperGAT. Given that
difficulty, SHINE still attained the best performance on each dataset.
Intuitively speaking, HyperGCN and HGNN focus on similarity regularization:
hypergraph nodes with similar context of hyperedges should have similar
representations. HyperGAT’s attention mechanism gears more towards minimizing
the classification loss. SHINE, in an attempt to balance the similarity
regularization with the end-to-end classification task via strongly dual
attention mechanism, achieved better trade-off between the two objectives and
effectively integrated the functional pathway’s (hyperedge) information with
individual gene’s (node) information.

The importance of the strong duality follows naturally from that SHINE
outperforms SubGNN (bipartite) with non-overlapping standard deviation
intervals and wide separation. In addition, although MLP has been frequently
used to approximate a target function, in the setting of large hypergraph
(e.g., both hypergraphs have a few thousand-nodes hyperedges), it can still be
quite challenging to approximate an ideal target function and explicit dual
attention formulation wins out. The results from both AllDeepSets and
AllSetTransformer have non-overlapping standard deviation intervals, in fact
wide separation, with their counterparts from SHINE. These results echo with
our observations that strongly dual attention explores the hypergraph
propagation from a different angle than both AllDeepSets and
AllSetTransformer, and suggest that effectively combining both angles could be
an interesting future direction. Also note that in general, we have some
performance drop when moving from the DisGeNet dataset to the TCGA-MC3
dataset, likely due to the fact that the former uses genetic features from
curated literatures and the latter is from high throughput sequencing intended
for data-driven discovery. Both complex classification tasks ($>$20 classes)
are uniquely challenging because diseases may have overlapping disrupted
molecular functions (genetic pathways, hyperedges), especially for the TCGA-
MC3 experiment that is distinguishing subcategories of similar diseases as
they are loosely all cancers. In addition, both tasks exhibit class
distributional shift between the train and the test datasets, as shown in
Appendix Tables 5 and 6, and have been designed to require inductive inference
on subgraphs with highly variable hyperedge sizes. Strong performance of SHINE
on these tasks thus suggests that our model can leverage its relational
inductive biases for more robust generalization. Ablation studies further
confirmed the contributions from each of the key components, including
strictly dual attention massage passing, weighted subgraph attention and
hypergraph regularization (see Appendix for details).

Table 3: Held-out test set micro-F1 on real-world datasets. Standard deviations are provided from runs with 10 random seeds. SHINE significantly outperforms all the state-of-the-art comparison models. PRS: polygenic risk score. NMF: non-negative matrix factorization. Best model in bold. Model | Feature | DisGeNet Dataset | TCGA-MC3 Dataset
---|---|---|---
Metrics |  | Test Micro F1 | Test Micro F1
PRS | gene | 0.6303 | 0.4981
PRS | pathway | 0.6461 | 0.5047
PRS | gene+pathway | 0.6512 | 0.5042
XGBoost | gene |  0.6259 $\pm$ 0.0012 |  0.4927 $\pm$ 0.0058
XGBoost | pathway |  0.6467 $\pm$ 0.0035 |  0.4936 $\pm$ 0.0092
XGBoost | gene+pathway |  0.6486 $\pm$ 0.0036 |  0.5117 $\pm$ 0.0084
NMF | gene |  0.6167 $\pm$ 0.0040 |  0.4181 $\pm$ 0.0125
NMF | pathway |  0.5867 $\pm$ 0.0039 |  0.4842 $\pm$ 0.0057
NMF | gene+pathway |  0.5847 $\pm$ 0.0045 |  0.4839 $\pm$ 0.0032
SubGNN (bipartite) | gene+pathway |  0.6137 $\pm$ 0.0097 |  0.4025 $\pm$ 0.0049
HyperGCN | gene+pathway |  0.6638 $\pm$ 0.0028 |  0.4384 $\pm$ 0.0095
HGNN | gene+pathway |  0.6809 $\pm$ 0.0027 |  0.4504 $\pm$ 0.0042
HyperGAT | gene+pathway |  0.6495 $\pm$ 0.0050 |  0.4721 $\pm$ 0.0032
MLP | gene+pathway |  0.6331 $\pm$ 0.0056 |  0.4249 $\pm$ 0.0165
AllDeepSets | gene+pathway |  0.6309 $\pm$ 0.0147 |  0.4324 $\pm$ 0.0220
AllSetTransformer | gene+pathway |  0.6355 $\pm$ 0.0160 |  0.4904 $\pm$ 0.0158
SHINE | gene+pathway | 0.6955 $\pm$ 0.0034 | 0.5319 $\pm$ 0.0049

#### Model interpretation.

SHINE simultaneously learns the representations of nodes and hyperedges, which
are then used to learn and inductively infer subgraph representations. This
brings some interpretation advantages as it allows assessing pathways
(hyperedges) correlations and reasoning multiple molecular functions mutually
interacting and collectively contributing to the disease onset and
progression. In addition, SHINE has built-in measures to prevent or discourage
genes belonging to the same functional class (e.g., promoting immune
reactions) from having drastically different representations (e.g., opposite
directions), a phenomenon that will pose interpretation difficulty to other
models that do not employ SHINE’s hypergraph regularization. We identify the
top pathways that are enriched in different cancers using the attention
weights learned for SHINE, as shown in Table 4. From the table, we see that
many of the listed pathways reflect innate key events in the development of
individual or multiple types of cancers, consistent with genetic and medical
knowledge from wet lab (e.g., TNF/Stress Related Signaling [45]). We showcase
interpretations for breast cancer and lung cancer here, and refer the reader
to the Appendix for full interpretation of Table 4. For breast cancer,
TNF$\alpha$ is not only closely involved in its onset, progression and in
metastasis formation, but also linked to therapy resistance [45]. Regarding
the 4-1BB pathway, studies have suggested HER2/4-1BB bispecific molecule as a
candidate of alternative therapeutic strategy to patients in HER2-positive
breast cancer [46]. VIP/PACAP and their receptors have prominent roles in
transactivation of the Epidermal growth factor (EGF) family and growth effects
in breast cancer [47]. For lung cancer, the ErbB3 receptor recycling
controlled by neuroregulin receptor degradation protein-1 is linked to lung
cancer and small inhibitory RNA (siRNA) to ErbB3 shows promise as a
therapeutic approach to treatment of lung adenocarcinoma [48]. Lung cancer is
also modulated by multiple miRNAs interacting with the TFAP2 family [49].

Table 4: Top enriched genetic pathways associated with different cancer risks. The text color indicates the source database for pathways that MSigDB integrated: BioCarta, Reactome, WikiPathways, Pathway Interaction Database, KEGG. BRCA | LUAD | LGG | HNSC
---|---|---|---
Stress pathway |  PTK6 stabilizes HIF1$\alpha$ | Citrate cycle TCA cycle | Apoptotic factor response
4-1BB pathway | ErbB3 pathway | Cytosine methylation | Programmed cell death
VIP pathway | Hypertrophic | TCA cycle and deficiency | MECP2 regulates neuronal
| cardiomyopathy | of pyruvate dehydrogenase | receptors and channels
CD40 pathway | Diseases of metabolism | Glutathione metabolism | FRA pathway
TOLL pathway | TFAP2 regulates growth | Digestion of | Caspase activation via
| factors transcription | dietary carbohydrate | extrinsic apoptotic signalling

##  6 Discussion, Limitation and Future Work

In addition to being significantly more accurate and interpretable, SHINE uses
inductive subgraph inferencing that works well with minibatch, and scales well
to large scale problems, as showcased by real-world experiments. It is known
that GNN suffers from over-smoothing when the number of layers increases, as
increasingly globally uniform representation of nodes may be developed. On the
other hand, attention could limit this phenomenon by limiting to a restricted
set of nodes. The effect of hypergraph regularization, while also smoothing,
happens on a local scale as part of a direct optimization objective and does
not accumulate with increasing number of layers. Such decoupling between
attention and local smoothing allows SHINE to better explore the optimization
landscape.

Our work has limitations. We assumed that the hyperedges are known in advance.
However, in reality, as our domain knowledge increases and evolves, we need to
account for unknown hyperedges and, better, simultaneously discover novel
hyperedges from data while predicting disease classes. Such a task has
important clinical utilities in genetic medicine to discover new genetic
pathways that may underlie disease etiology, and will be our future work.
Moreover, strongly dual attention explores the hypergraph propagation from a
different angle than both AllDeepSets and AllSetTransformer, and effectively
combining both angles could be an interesting future direction. Another line
of future work is to derive a hypergraph coarsening model on top of SHINE.
SHINE currently has flat hypergraph layout and does not learn hierarchical
representations of hypergraphs. The emerging technique of spatial
transcriptomics can enable discovery of localized and hierarchical gene
expression patterns [50, 51]. A flexible hypergraph coarsening model that can
effectively learn hierarchical network structure out of the hypergraphs can
shed light on the organizations of the hyperedges (e.g., pathways representing
synergistic molecular functions in certain tissue context).

From the application point of view, detecting tumor subtypes is often
interesting, and we expect to extend our method to such detections using
multi-modal data when large shared datasets will become available [52]. To
certain extent, the TCGA labels we used reflect subtypes of organ-specific
primary tumors, e.g., LUAD vs. LUSC in lung cancer, KIRC vs. KIRP in kidney
cancer. On the other hand, identifying drivers genes and pathways for cancer
types and other disease subtypes continue to be biologically important [53]
and will be increasingly fruitful with simultaneously collected deep genetic
and phenotypic data on the same patients [54, 55].

The field of genetic medicine encompasses areas of molecular biology and
clinical phenotyping to explore new relationships between disease
susceptibility and human genetics. Though appearing as a single field, it
revolutionizes the practice of medicine in preventing, modifying and treating
many diseases such as cardiovascular disease and cancer [56]. We expect SHINE
to be a useful tool in the quest of broadly advancing the knowledge on disease
susceptibility. In these real-world applications, a subject’s genetic profile
may contain individual characterizing information. Thus, this work should
never be used in violation of an individual’s privacy, and the necessary steps
of IRB review and execution of data user agreement need to be properly
completed prior to the study.

##  7 Conclusions

We proposed a novel framework termed SubHypergraph Inductive Neural nEtwork
(SHINE) for inductive subgraph inferencing on hypergraphs, designed for
jointly optimizing the objectives of end-to-end subgraph classification and
similarity regularization for representations of hypergraph nodes with similar
context of hyperedges. We showed that SHINE improved the performance
(micro-F1) of the learned model for disease type prediction for complex ($>$20
classes) genetic medicine datasets of different characteristics and under
different settings (e.g., multi-class and/or multi-label). Genetic pathways
directly correspond to molecular mechanisms and functions, which are more
informative than individual genes and are represented as hyperedges in SHINE.
The novel formulation of disease classification as a subgraph inferencing
problem allows a hypergraph neural network to link correlated pathways, i.e.,
interacting molecular mechanisms, to disease etiology. This leads to better
performance with added interpretability. We compared SHINE with a wide array
of state-of-the-art (hyper)graph neural networks, XGBoost, NMF, and PRS models
with different configurations of genes and pathways as features. SHINE
consistently outperformed all state-of-the-art baselines significantly in each
of the disease classification and cancer classification tasks. Feature
analysis of the learned pathway groups that are automatically identified by
SHINE in a data-driven fashion offered significant clinical insights about
multiple molecular mechanisms that interact and are associated with disease
types and status.

## Acknowledgments and Disclosure of Funding

This work was supported in part by NIH grants R01LM013337 and U01TR003528.

##  Appendix A Appendix for SHINE: SubHypergraph Inductive Neural nEtwork

#### Datasets details.

In this section, we give additional details on the datasets used in this
paper. The DisGeNet dataset is a collection of mutated genes involved in human
diseases compiled from expert curated repositories, GWAS catalogs, animal
models and the scientific literature. Each disease is labeled with one or more
of 22 MeSH codes, and the task is a multi-class multi-label classification
problem. We used 6:2:2 train:validation:test partition, and the split
distribution is shown in Table 5. The DisGeNet dataset has 6226 pathways and
9133 genes involved in 8383 diseases in total. The TCGA-MC3 dataset records
somatic mutations for subjects in The Cancer Genome Atlas (TCGA). The genetic
variants are stored in a specially formatted file. A row in the file specifies
a particular variant (e.g., Single Nucleotide Polymorphism or
insertion/deletion), its chromosomal location, and what proportion of the
sequencing reads covering that chromosomal location have that variant, among
other characteristics. Each subject is labeled with one or more of 25 cancer
types, and the task is a multi-class classification problem. We used 6:2:2
train:validation:test partition, stratified by cancer types, and the split
distribution is shown in Table 6. The TCGA-MC3 dataset has 6229 pathways and
18059 genes involved in 9012 subjects in total.

#### Genetic pathways.

Genetic pathways are a valuable tool to assist in representing, understanding,
and analyzing the complex interactions between molecular functions. The
pathways contain multiple genes (can be modeled using hyperedges) and
correspond to genetic functions, including regulations, genetic signaling, and
metabolic interactions. They have a wide range of applications, including
predicting cellular activity and inferring disease types and status [57]. For
a simplified and illustrative example, a signaling pathway p1 (having 20
genes) sensing the environment may govern (the governing function embodied as
a pathway p2 having 15 genes) the expression of transcription factors in
another signaling pathway p3 (having 23 genes), which then controls (the
controlling function embodied as a pathway p4 having 34 genes) the expression
of proteins that play roles as enzymes in a metabolic pathway p5 (having 57
genes). In general, there will be partial overlap between pathways p1 and p2,
p2 and p3, p3 and p4, p4 and p5, and other potential partial overlaps
corresponding to partial overlaps between their corresponding hyperedges.

Table 5: Statistics of DisGeNet experiment data. The table includes the distribution of the 22 MeSH categories with more than 100 diseases. The dataset is split into a training set, a validation set and a test set according to a 6:2:2 ratio. MeSH | Description | Total | Train | Val | Test
---|---|---|---|---|---
C01 | Infections | 221 | 135 | 45 | 41
C04 | Neoplasms | 1010 | 626 | 190 | 194
C05 | Musculoskeletal Diseases | 1266 | 765 | 239 | 262
C06 | Digestive System Diseases | 430 | 238 | 91 | 101
C07 | Stomatognathic Diseases | 242 | 156 | 50 | 36
C08 | Respiratory Tract Diseases | 235 | 137 | 52 | 46
C09 | Otorhinolaryngologic Diseases | 299 | 188 | 55 | 56
C10 | Nervous System Diseases | 2960 | 1769 | 619 | 572
C11 | Eye Diseases | 756 | 470 | 150 | 136
C12 | Male Urogenital Diseases | 537 | 337 | 102 | 98
C13 | Female Urogenital Diseases and | 640 | 402 | 118 | 120
| Pregnancy Complications |  |  |  |
C14 | Cardiovascular Diseases | 746 | 441 | 147 | 158
C15 | Hemic and Lymphatic Diseases | 624 | 392 | 108 | 124
C16 | Congenital, Hereditary, and Neonatal | 3648 | 2168 | 725 | 755
| Diseases and Abnormalities |  |  |  |
C17 | Skin and Connective Tissue Diseases | 789 | 459 | 142 | 188
C18 | Nutritional and Metabolic Diseases | 1277 | 725 | 271 | 281
C19 | Endocrine System Diseases | 535 | 327 | 107 | 101
C20 | Immune System Diseases | 415 | 249 | 87 | 79
C23 | Pathological Conditions, Signs and Symptoms | 1795 | 1065 | 387 | 343
C25 | Chemically-Induced Disorders | 135 | 80 | 29 | 26
F01 | Behavior and Behavior Mechanisms | 267 | 164 | 62 | 41
F03 | Mental Disorders | 501 | 295 | 123 | 83
Table 6: Statistics of TCGA-MC3 experiment data. The table includes the distribution of the 25 cancer types with more than 100 subjects. The dataset is split into a training set, a validation set and a test set according to a 6:2:2 ratio. Cancer | Description | Total | Train | Val | Test
---|---|---|---|---|---
BLCA | Bladder Urothelial Carcinoma | 411 | 247 | 82 | 82
BRCA | Breast invasive carcinoma | 791 | 475 | 158 | 158
CESC | Cervical squamous cell carcinoma | 289 | 173 | 58 | 58
| and endocervical adenocarcinoma |  |  |  |
COAD | Colon adenocarcinoma | 288 | 173 | 57 | 58
ESCA | Esophageal carcinoma | 184 | 110 | 37 | 37
GBM | Glioblastoma multiforme | 309 | 185 | 62 | 62
HNSC | Head and Neck squamous cell carcinoma | 507 | 304 | 102 | 101
KIRC | Kidney renal clear cell carcinoma | 368 | 220 | 74 | 74
KIRP | Kidney renal papillary cell carcinoma | 281 | 169 | 56 | 56
LAML | Acute Myeloid Leukemia | 137 | 83 | 27 | 27
LGG | Brain Lower Grade Glioma | 510 | 306 | 102 | 102
LIHC | Liver hepatocellular carcinoma | 363 | 217 | 73 | 73
LUAD | Lung adenocarcinoma | 512 | 307 | 103 | 102
LUSC | Lung squamous cell carcinoma | 480 | 288 | 96 | 96
OV | Ovarian serous cystadenocarcinoma | 409 | 245 | 82 | 82
PAAD | Pancreatic adenocarcinoma | 175 | 105 | 35 | 35
PCPG | Pheochromocytoma and Paraganglioma | 178 | 107 | 35 | 36
PRAD | Prostate adenocarcinoma | 493 | 295 | 99 | 99
SARC | Sarcoma | 236 | 142 | 47 | 47
SKCM | Skin Cutaneous Melanoma | 466 | 280 | 93 | 93
STAD | Stomach adenocarcinoma | 438 | 262 | 88 | 88
TGCT | Testicular Germ Cell Tumors | 128 | 77 | 25 | 26
THCA | Thyroid carcinoma | 490 | 294 | 98 | 98
THYM | Thymoma | 122 | 74 | 24 | 24
UCEC | Uterine Corpus Endometrial Carcinoma | 447 | 268 | 90 | 89

#### Genetic variant calling and filtering for TCGA-MC3 dataset.

The variants are usually of high dimensionality. For example, in the TCGA-MC3
dataset, even after we retain only the variants that received PASS
identifiers, there are still around 3 million variants. Thus, we choose to
aggregate their counts according to the affected genes to avoid impractically
large matrices. We aggregate genetic variant count at gene level and sum up
all the alternative allele counts and reference allele counts in a gene. We
calculate the mutation rate for a gene $g$ as in equation 14,

$\mu(g)=\frac{\sum_{v\in g}C_{ALT}(v)}{\sum_{v\in g}C_{ALT}(v)+\sum_{v\in
g}C_{REF}(v)}$ (14)

where variant $v$ belongs to the gene $g$, $C_{ALT}(v)$ is the read depth
supporting the variant (alternative) allele in tumor sequencing data and
$C_{REF}(v)$ is the read depth supporting the reference allele (non-mutated)
in tumor sequencing data.

#### Parameter and hyperparameter tuning for models.

For SHINE and other hypergraph methods, the hyperparameter of hidden dimension
$d$ is tuned on the validation dataset with choices from 100 to 1000, at
increments of 100. Deep neural network models are often randomly initialized,
thus we also run initialization 10 times and report the averages and standard
deviations. For the comparison hypergraph neural network models, we used the
implementations by the original authors. The hyperparameters were tuned on the
validation set using choice grids according to respective papers, or when
unspecified, from default grids as with our proposed method (learning rate
$\in[0.001,0.002,0.005]$, weight decay $\in[0.0001,0.0005]$, dropout rate
$\in[0.4,0.5,0.6]$). For PRS, the regularization coefficient $C$ is tuned on
the validation dataset with choices from a geometric sequence from 0.001 to
1000 at a multiplying ratio of 10. For NMF, the number of factors is tuned on
the validation dataset with choices from 100 to 1000, at increments of 100.
For XGBoost, we tuned max tree depth (3, 5, 10), the number of estimators
(from 100 to 1000, at increments of 100), and min child weight (0.01, 0.1, 1,
10, 100), using the validation set. For models requiring random
initialization, we run initializations 10 times with different seeds and
report the averages and standard deviations. We varied the number K of layers
from 1 to 4, and found that 2 layers to give the best results for SHINE.

Regarding sensitivity to the hidden dimension, in general, the performance is
less sensitive to the hidden dimensions when it is sufficiently big
($\geq$300), with <0.05 change in micro-F1 score. Smaller hidden dimensions
(100-200) can lead to >0.05 micro-F1 drop, likely due to insufficient
representation power. The optimal hidden dimension is 300 for the TCGA-MC3
dataset and 600 for the DisGeNet dataset. The performance also shows <0.05
change in micro-F1 score when varying other hyperparameters including learning
rate, weight decay, dropout rate in their respective grids as specified above.

#### Computational complexity.

The complexity of SHINE scales as the following factors grow: the numbers of
layers and nodes, the number and size of hyperedges, the size of hidden
dimensions, and finally the number and size of subhypergraphs. We implement
SHINE on PyTorch, and run it on NVIDIA V100 GPUs. We train SHINE for up to
6000 epochs using Adam [58] and stop training if the validation loss does not
decrease for 10 consecutive epochs. The TCGA-MC3 dataset’s training times are:
MLP $\sim$5 min, HyperGCN $\sim$7 min, AllSetTransformer $\sim$20 min,
AllDeepSet $\sim$20 min, SHINE $\sim$30 min, HGNN $\sim$30 min, HyperGAT
$\sim$30 min, SubGNN >1 day (excluding prebuild time). The DisGeNet dataset’s
training times are: MLP $\sim$5 min, HyperGCN $\sim$6 min, AllDeepSet $\sim$13
min, HGNN $\sim$15 min, HyperGAT $\sim$15 min, AllSetTransformer $\sim$16 min,
SHINE $\sim$20 min, SubGNN >1 day (excluding prebuild time).

#### Ablation study.

To investigate the contribution of key components (e.g., the strictly dual
attention massage passing, the usage of hypergraph regularization) in the
proposed algorithm to the overall method, we performed an ablation analysis.
The previous state-of-the-art hypergraph neural network models in fact serve
as some of the steps in the ablation. For example, HyperGAT does not have
strict dual attention message passing and does not employ hypergraph
regularization. HGNN and HyperGCN apply hypergraph convolution instead of
attention message passing. HyperGCN, compared to HGNN, applies approximate
hypergraph convolution by representing a hyperedge by a selected pairwise
simple edge connecting two most unlike nodes, and adding the remaining nodes
in the hyperedge as mediators. To evaluate the efficacy of the weighted
subgraph attention (WSA), we consider a subgraph simply the sum of the nodes
(genes) that are of interest (with mutations) for each patient (subgraph).
Finally, we added SHINE with no hypergraph regularization to evaluate the
regularization effectiveness. The ablation analysis results are shown in Table
7. From the results, it is clear that SHINE’s strictly dual attention message
passing outperforms HyperGAT without strictly dual attention message passing.
We can see that adding hypergraph regularization further improves performance,
in fact, with improvement beyond standard deviation intervals of the
regularization-ablated model on both datasets. The weighted subgraph attention
(WSA) ablation leads to a larger performance drop than hypergraph
regularization ablation, which corroborates the importance of the WSA step. We
also notice that the performance drop due to WSA ablation on the TCGA-MC3
dataset is larger than that on the DisGeNet dataset. This is consistent with
the fact that the TCGA-MC3 dataset has denser hypergraph and larger subgraphs
than the DisGeNet dataset. This is also consistent with the fact that
differentiating among cancer subtypes is a more complex and nuanced task than
differentiating among disease categories. These observations collectively
argue for the benefits of weighted subgraph attention over direct aggregation
such as sum, and more increasingly so for larger datasets and more complex
tasks.

Table 7: Ablation Analysis: Held-out test set micro-F1 on real-world datasets. Standard deviations are provided from runs with 10 random seeds. SHINE significantly outperforms all the state-of-the-art comparison models. Best model in bold. Model | DisGeNet Dataset | TCGA-MC3 Dataset
---|---|---
Metrics | Test Micro F1 | Test Micro F1
HyperGCN (approx. hypergraph convolution) | 0.6638 $\pm$ 0.0028 | 0.4384 $\pm$ 0.0095
HGNN (hypergraph convolution) | 0.6809 $\pm$ 0.0027 | 0.4504 $\pm$ 0.0042
HyperGAT (not strictly dual attention) | 0.6495 $\pm$ 0.0050 | 0.4721 $\pm$ 0.0032
SHINE without weighted subgraph attention | 0.6472 $\pm$ 0.0053 | 0.4388 $\pm$ 0.0091
SHINE without hypergraph regularization | 0.6829 $\pm$ 0.0059 | 0.5247 $\pm$ 0.0048
SHINE | 0.6955 $\pm$ 0.0034 | 0.5319 $\pm$ 0.0049

#### Model interpretation.

SHINE simultaneously learns the representations of nodes and hyperedges, which
are then used to learn and inductively infer subgraph representations. This
brings model interpretation advantages as it allows assessing pathways
(hyperedges) correlations and reasoning multiple molecular functions mutually
interacting and collectively contributing to the disease onset. We identify
the top pathways that are enriched in different cancers using the attention
weights learned for SHINE, as shown in Table 4. From the table, we see that
many of the listed pathways reflect innate key events in the development of
individual or multiple types of cancers, consistent with genetic and medical
knowledge from wet lab (e.g., TNF/Stress Related Signaling [45]).

For breast cancer, TNF$\alpha$ is not only closely involved in its onset,
progression and in metastasis formation, but also linked to therapy resistance
[45]. Regarding the 4-1BB pathway, studies have suggested HER2/4-1BB
bispecific molecule as a candidate of alternative therapeutic strategy to
patients in HER2-positive breast cancer [46]. VIP/PACAP and their receptors
have prominent roles in transactivation of the Epidermal growth factor (EGF)
family and growth effects in breast cancer [47]. For lung cancer, the ErbB3
receptor recycling controlled by neuroregulin receptor degradation protein-1
is linked to lung cancer and small inhibitory RNA (siRNA) to ErbB3 shows
promise as a therapeutic approach to treatment of lung adenocarcinoma [48].
Lung cancer is also modulated by multiple miRNAs interacting with the TFAP2
family [49]. For lower-grade gliomas, recent studies have reported the
association between DNA demethylation and their malignant progressions [59].
Emerging evidence has also linked the citric acid (TCA) cycle for energy
production to fuel the development of certain cancer types, especially those
with deregulated oncogene and tumor suppressor expression [60]. For head and
neck cancer, studies have reported a high percentage of cases with MECP2 copy-
number gain and in combination with RAS mutation or amplification [61]. The
apoptotic signaling and response pathways involving the mitochondrial pro-
apoptotic protein SMAC/Diablo have also been suggested to regulate lipid
synthesis that is essential for cancer growth and development [62].

Of note, the pathways listed in Table 4 for each cancer type play roles in
different phases of cancer onset, growth or metastasis, and likely function
together in tumorigenesis and progression, as discovered by SHINE. These
analyses suggest that besides providing useful and discriminative features,
SHINE integrates gene and pathway data to provide insights into functional and
molecular mechanisms by linking together multiple pathways that may function
together and contribute to cancer development and progression.

#### Relevance and impact.

The techniques and results presented in the paper could apply to many diseases
through informing genetic medicine practice. In these real-world applications,
a subject’s genetic profile may contain individual characterizing information.
Thus, this work, or derivatives of it, should never be used in violation of an
individual’s privacy. For using individual level dataset such as the TCGA-MC3,
the proper steps of IRB review of study and execution of data user agreement
need to be properly completed prior to the study, such as done by this study.

It is important for the machine learning (ML) community to continue being
informed about the problems arising in critical application domains such as
healthcare and biomedicine that can guide model design. More specifically,
explicitly treating hyperedges as first class citizens in the GNN modelling is
important, since in this way hyperedges can be the subjects of notions of
regularization or attention. This article demonstrated the feasibility to
address those needs with our practical considerations of design and
implementation choices by SHINE to advance modern genetic medicine study. We
have demonstrated successful applications of SHINE on large-scale genetic
medicine datasets, including the TCGA-MC3 dataset that is one of the largest
NIH dbGaP datasets. Genetic medicine revolutionizes the practice of medicine
in preventing, modifying and treating many diseases such as cardiovascular
disease and cancer. In the future, as even larger genetic datasets will be
collected through NIH programs such as All of Us and TopMed, we expect SHINE
to be a useful tool in the quest of broadly advancing the knowledge on disease
susceptibility.

## References

  * Liberzon et al. [2015] Arthur Liberzon, Chet Birger, Helga Thorvaldsdóttir, Mahmoud Ghandi, Jill P Mesirov, and Pablo Tamayo.  The molecular signatures database hallmark gene set collection.  _Cell systems_ , 1(6):417–425, 2015.
  * Luo and Mao [2021] Yuan Luo and Chengsheng Mao.  Panther: pathway augmented nonnegative tensor factorization for higher-order feature learning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pages 371–380, 2021.
  * Feng et al. [2019] Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao.  Hypergraph neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 3558–3565, 2019.
  * Yadati et al. [2018] Naganand Yadati, Madhav Nimishakavi, Prateek Yadav, Vikram Nitin, Anand Louis, and Partha Talukdar.  Hypergcn: A new method of training graph convolutional networks on hypergraphs.  _arXiv preprint arXiv:1809.02589_ , 2018.
  * Yadati [2020] Naganand Yadati.  Neural message passing for multi-relational ordered and recursive hypergraphs.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Zhang et al. [2020] Ruochi Zhang, Yuesong Zou, and Jian Ma.  Hyper-sagnn: a self-attention based graph neural network for hypergraphs.  In _International Conference on Learning Representations_ , 2020.
  * Ding et al. [2020] Kaize Ding, Jianling Wang, Jundong Li, Dingcheng Li, and Huan Liu.  Be more with less: Hypergraph attention networks for inductive text classification.  _arXiv preprint arXiv:2011.00387_ , 2020.
  * Meng et al. [2018] Changping Meng, S Chandra Mouli, Bruno Ribeiro, and Jennifer Neville.  Subgraph pattern neural networks for high-order graph evolution prediction.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Alsentzer et al. [2020] Emily Alsentzer, Samuel G Finlayson, Michelle M Li, and Marinka Zitnik.  Subgraph neural networks.  _arXiv preprint arXiv:2006.10538_ , 2020.
  * Wei et al. [2020] Xin Wei, Ruixuan Yu, and Jian Sun.  View-gcn: View-based graph convolutional network for 3d shape analysis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1850–1859, 2020.
  * Mao et al. [2022] Chengsheng Mao, Liang Yao, and Yuan Luo.  Imagegcn: Multi-relational image graph convolutional networks for disease identification with chest x-rays.  _IEEE Transactions on Medical Imaging_ , 2022.
  * Duvenaud et al. [2015] David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams.  Convolutional networks on graphs for learning molecular fingerprints.  In _NeurIPS_ , pages 2224–2232, 2015.
  * Gilmer et al. [2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl.  Neural message passing for quantum chemistry.  In _ICML_ , pages 1263–1272. JMLR. org, 2017.
  * Yao et al. [2019] Liang Yao, Chengsheng Mao, and Yuan Luo.  Graph convolutional networks for text classification.  In _AAAI_ , 2019.
  * Peng et al. [2018] Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Lihong Wang, Yangqiu Song, and Qiang Yang.  Large-scale hierarchical text classification with recursively regularized deep graph-cnn.  In _WWW_ , pages 1063–1072, 2018.
  * Li et al. [2019] Yifu Li, Ran Jin, and Yuan Luo.  Classifying relations in clinical narratives using segment graph convolutional and recurrent neural networks (seg-gcrns).  _Journal of the American Medical Informatics Association_ , 26(3):262–268, 2019.
  * Zitnik et al. [2018] Marinka Zitnik, Monica Agrawal, and Jure Leskovec.  Modeling polypharmacy side effects with graph convolutional networks.  _Bioinformatics_ , 34(13):i457–i466, 2018.
  * Mao et al. [2019] Chengsheng Mao, Liang Yao, and Yuan Luo.  Medgcn: Graph convolutional networks for multiple medical tasks.  _arXiv preprint arXiv:1904.00326_ , 2019.
  * Kipf and Welling [2016] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ , 2016.
  * Vinyals et al. [2015] Oriol Vinyals, Samy Bengio, and Manjunath Kudlur.  Order matters: Sequence to sequence for sets.  _arXiv preprint arXiv:1511.06391_ , 2015.
  * Ying et al. [2018] Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L Hamilton, and Jure Leskovec.  Hierarchical graph representation learning with differentiable pooling.  _arXiv preprint arXiv:1806.08804_ , 2018.
  * Hamilton [2020] William L Hamilton.  Graph representation learning.  _Synthesis Lectures on Artifical Intelligence and Machine Learning_ , 14(3):1–159, 2020.
  * Zhou et al. [2006] Dengyong Zhou, Jiayuan Huang, and Bernhard Schölkopf.  Learning with hypergraphs: Clustering, classification, and embedding.  _Advances in neural information processing systems_ , 19:1601–1608, 2006.
  * Jin et al. [2019] Taisong Jin, Liujuan Cao, Baochang Zhang, Xiaoshuai Sun, Cheng Deng, and Rongrong Ji.  Hypergraph induced convolutional manifold networks.  In _IJCAI_ , pages 2670–2676, 2019.
  * Jiang et al. [2019] Jianwen Jiang, Yuxuan Wei, Yifan Feng, Jingxuan Cao, and Yue Gao.  Dynamic hypergraph neural networks.  In _IJCAI_ , pages 2635–2641, 2019.
  * Satchidanand et al. [2015] Sai Nageswar Satchidanand, Harini Ananthapadmanaban, and Balaraman Ravindran.  Extended discriminative random walk: A hypergraph approach to multi-view multi-relational transductive learning.  In _IJCAI_ , pages 3791–3797, 2015.
  * Feng et al. [2018] Fuli Feng, Xiangnan He, Yiqun Liu, Liqiang Nie, and Tat-Seng Chua.  Learning on partial-order hypergraphs.  In _Proceedings of the 2018 World Wide Web Conference_ , pages 1523–1532, 2018.
  * Tu et al. [2018] Ke Tu, Peng Cui, Xiao Wang, Fei Wang, and Wenwu Zhu.  Structural deep embedding for hyper-networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Chien et al. [2021] Eli Chien, Chao Pan, Jianhao Peng, and Olgica Milenkovic.  You are allset: A multiset function framework for hypergraph neural networks.  _arXiv preprint arXiv:2106.13264_ , 2021.
  * Huang and Zitnik [2020] Kexin Huang and Marinka Zitnik.  Graph meta learning via local subgraphs.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Sun et al. [2021] Qingyun Sun, Jianxin Li, Hao Peng, Jia Wu, Yuanxing Ning, Phillip S Yu, and Lifang He.  Sugar: Subgraph neural network with reinforcement pooling and self-supervised mutual information mechanism.  _arXiv preprint arXiv:2101.08170_ , 2021.
  * Wang et al. [2019] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu.  Heterogeneous graph attention network.  In _The world wide web conference_ , pages 2022–2032, 2019.
  * Zhang et al. [2019] Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla.  Heterogeneous graph neural network.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 793–803, 2019.
  * Fu et al. [2019] Yuwei Fu, Yun Xiong, S Yu Philip, Tianyi Tao, and Yangyong Zhu.  Metapath enhanced graph attention encoder for hins representation learning.  In _2019 IEEE International Conference on Big Data (Big Data)_ , pages 1103–1110. IEEE, 2019.
  * Fu et al. [2020] Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King.  Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding.  In _Proceedings of The Web Conference 2020_ , pages 2331–2341, 2020\.
  * Hu et al. [2020] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun.  Heterogeneous graph transformer.  In _Proceedings of The Web Conference 2020_ , pages 2704–2710, 2020\.
  * Zhang et al. [2018] Muhan Zhang, Zhicheng Cui, Shali Jiang, and Yixin Chen.  Beyond link prediction: Predicting hyperlinks in adjacency space.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Li et al. [2015] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel.  Gated graph sequence neural networks.  _arXiv preprint arXiv:1511.05493_ , 2015.
  * Piñero et al. [2016] Janet Piñero, Àlex Bravo, Núria Queralt-Rosinach, Alba Gutiérrez-Sacristán, Jordi Deu-Pons, Emilio Centeno, Javier García-García, Ferran Sanz, and Laura I. Furlong.  DisGeNET: a comprehensive platform integrating information on human disease-associated genes and variants.  _Nucleic Acids Research_ , 45(D1):D833–D839, 10 2016.  ISSN 0305-1048.  doi: 10.1093/nar/gkw943.  URL https://doi.org/10.1093/nar/gkw943.
  * Ellrott et al. [2018] Kyle Ellrott, Matthew H Bailey, Gordon Saksena, Kyle R Covington, Cyriac Kandoth, Chip Stewart, Julian Hess, Singer Ma, Kami E Chiotti, Michael McLellan, et al.  Scalable open science approach for mutation calling of tumor exomes using multiple genomic pipelines.  _Cell systems_ , 6(3):271–281, 2018.
  * Wang et al. [2022] Xiao Wang, Deyu Bo, Chuan Shi, Shaohua Fan, Yanfang Ye, and S Yu Philip.  A survey on heterogeneous graph embedding: methods, techniques, applications and sources.  _IEEE Transactions on Big Data_ , 2022.
  * Choi et al. [2020] Shing Wan Choi, Timothy Shin-Heng Mak, and Paul F O’Reilly.  Tutorial: a guide to performing polygenic risk score analyses.  _Nature Protocols_ , 15(9):2759–2772, 2020.
  * Stein-O’Brien et al. [2018] Genevieve L Stein-O’Brien, Raman Arora, Aedin C Culhane, Alexander V Favorov, Lana X Garmire, Casey S Greene, Loyal A Goff, Yifeng Li, Aloune Ngom, Michael F Ochs, et al.  Enter the matrix: factorization uncovers knowledge from omics.  _Trends in Genetics_ , 34(10):790–805, 2018.
  * Chen and Guestrin [2016] Tianqi Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , pages 785–794, 2016.
  * Mercogliano et al. [2020] María Florencia Mercogliano, Sofía Bruni, Patricia V Elizalde, and Roxana Schillaci.  Tumor necrosis factor $\alpha$ blockade: an opportunity to tackle breast cancer.  _Frontiers in oncology_ , 10, 2020.
  * Hinner et al. [2019] Marlon J Hinner, Rachida Siham Bel Aiba, Thomas J Jaquin, Sven Berger, Manuela Carola Dürr, Corinna Schlosser, Andrea Allersdorfer, Alexander Wiedenmann, Gabriele Matschiner, Julia Schüler, et al.  Tumor-localized costimulatory t-cell engagement by the 4-1bb/her2 bispecific antibody-anticalin fusion prs-343.  _Clinical Cancer Research_ , 25(19):5878–5889, 2019.
  * Moody et al. [2016] Terry W Moody, Bernardo Nuche-Berenguer, and Robert T Jensen.  Vip/pacap, and their receptors and cancer.  _Current opinion in endocrinology, diabetes, and obesity_ , 23(1):38, 2016.
  * Sithanandam and Anderson [2008] G Sithanandam and LM Anderson.  The erbb3 receptor in cancer and cancer gene therapy.  _Cancer gene therapy_ , 15(7):413–448, 2008.
  * Kołat et al. [2019] Damian Kołat, Żaneta Kałuzińska, Andrzej K Bednarek, and Elżbieta Płuciennik.  The biological characteristics of transcription factors ap-2$\alpha$ and ap-2$\gamma$ and their importance in various types of cancers.  _Bioscience reports_ , 39(3), 2019.
  * Zeng et al. [2022] Zexian Zeng, Yawei Li, Yiming Li, and Yuan Luo.  Statistical and machine learning methods for spatially resolved transcriptomics data analysis.  _Genome biology_ , 23(1):1–23, 2022.
  * Rao et al. [2021] Anjali Rao, Dalia Barkley, Gustavo S França, and Itai Yanai.  Exploring tissue architecture using spatial transcriptomics.  _Nature_ , 596(7871):211–220, 2021.
  * Kline et al. [2022] Adrienne Kline, Hanyin Wang, Yikuan Li, Saya Dennis, Meghan Hutch, Zhenxing Xu, Fei Wang, Feixiong Cheng, and Yuan Luo.  Multimodal machine learning in precision health.  _arXiv preprint arXiv:2204.04777_ , 2022.
  * Bailey et al. [2018] Matthew H Bailey, Collin Tokheim, Eduard Porta-Pardo, Sohini Sengupta, Denis Bertrand, Amila Weerasinghe, Antonio Colaprico, Michael C Wendl, Jaegil Kim, Brendan Reardon, et al.  Comprehensive characterization of cancer driver genes and mutations.  _Cell_ , 173(2):371–385, 2018.
  * Luo et al. [2019] Y Luo, C Mao, Y Yang, F Wang, FS Ahmad, D Arnett, MR Irvin, and SJ Shah.  Integrating hypertension phenotype and genotype with hybrid non-negative matrix factorization.  _Bioinformatics (Oxford, England)_ , 35(8):1395–1403, 2019.
  * Ritchie et al. [2015] Marylyn D Ritchie, Emily R Holzinger, Ruowang Li, Sarah A Pendergrass, and Dokyoon Kim.  Methods of integrating data to uncover genotype–phenotype interactions.  _Nature Reviews Genetics_ , 16(2):85–97, 2015\.
  * Green et al. [2020] Eric D Green, Chris Gunter, Leslie G Biesecker, Valentina Di Francesco, Carla L Easter, Elise A Feingold, Adam L Felsenfeld, David J Kaufman, Elaine A Ostrander, William J Pavan, et al.  Strategic vision for improving human health at the forefront of genomics.  _Nature_ , 586(7831):683–692, 2020.
  * Alon [2006] Uri Alon.  _An introduction to systems biology: design principles of biological circuits_.  Chapman and Hall/CRC, 2006.
  * Kingma and Ba [2015] DP Kingma and JL Ba.  Adam: A method for stochastic optimization.  In _ICLR_ , 2015.
  * Nomura et al. [2019] Masashi Nomura, Kuniaki Saito, Koki Aihara, Genta Nagae, Shogo Yamamoto, Kenji Tatsuno, Hiroki Ueda, Shiro Fukuda, Takayoshi Umeda, Shota Tanaka, et al.  Dna demethylation is associated with malignant progression of lower-grade gliomas.  _Scientific reports_ , 9(1):1–12, 2019.
  * Anderson et al. [2018] Nicole M Anderson, Patrick Mucka, Joseph G Kern, and Hui Feng.  The emerging role and targetability of the tca cycle in cancer metabolism.  _Protein & cell_, 9(2):216–237, 2018.
  * Neupane et al. [2016] Manish Neupane, Allison P Clark, Serena Landini, Nicolai J Birkbak, Aron C Eklund, Elgene Lim, Aedin C Culhane, William T Barry, Steven E Schumacher, Rameen Beroukhim, et al.  Mecp2 is a frequently amplified oncogene with a novel epigenetic mechanism that mimics the role of activated ras in malignancy.  _Cancer discovery_ , 6(1):45–58, 2016.
  * Paul et al. [2018] Avijit Paul, Yakov Krelin, Tasleem Arif, Rina Jeger, and Varda Shoshan-Barmatz.  A new role for the mitochondrial pro-apoptotic protein smac/diablo in phospholipid synthesis associated with tumorigenesis.  _Molecular Therapy_ , 26(3):680–694, 2018.


* (27) Y. Nambu, Phys. Rev. 117, 648 (1960).
  * (28) J. Goldstone, Nuovo Cimento 19, 154 (1961).
  * (29) J. Goldstone, A. Salam, and S. Weinberg, Phys. Rev. 127, 965 (1962).
  * (30) V. Ambegaokar and L. P. Kadanoff, Nuovo Cimento 22, 914 (1961).
  * (31) J. R. Schrieffer, Theory of Superconductivity (W. A. Benjamin, New York, 1964).
  * (32) P. B. Littlewood and C. M. Varma, Phys. Rev. Lett. 47, 811 (1981); Phys. Rev. B 26, 4883 (1982).
  * (33) P. W. Anderson, Phys. Rev. 130, 439 (1963).
  * (34) M. Randeria, N. Trivedi, A. Moreo, and R. T. Scalettar, Phys. Rev. Lett. 69, 2001 (1992).
  * (35) V. B. Geshkenbein, L. B. Iofe, and A. I. Larkin, Phys. Rev. B 55, 3173 (1997).
  * (36) H. B. Yang, J. D. Rameau, P. D. Johnson, T. Valla, A. Tsvelik, and G. D. Gu, Nature (London) 456, 77 (2008).
  * (37) Y. I. Seo, W. J. Choi, S. I. Kimura, Y. Bang, and Y. S. Kwon, Phys. Rev. B 95, 094510 (2017).
  * (38) Y. I. Seo, W. J. Choi, S. I. Kimura, and Y. S. Kwon, Sci. Rep. 9, 3987 (2019).
  * (39) L. Li, Y. Wang, M. J. Naughton, S. Ono, Y. Ando, and N. P. Ong, Europhys. Lett. 72, 451 (2005).
  * (40) Y. Wang, L. Li, M. J. Naughton, G. D. Gu, S. Uchida, and N. P. Ong, Phys. Rev. Lett. 95, 247002 (2005).
  * (41) L. Li, Y. Wang, S. Komiya, S. Ono, Y. Ando, G. D. Gu, and N. P. Ong, Phys. Rev. B 81, 054510 (2010).
  * (42) G. Yu, D.-D. Xia, D. Pelc, R.-H. He, N.-H. Kaneko, T. Sasagawa, Y. Li, X. Zhao, N. Barišić, A. Shekhter, and M. Greven, Phys. Rev. B 99, 214502 (2019).
  * (43) J. L. Tallon, J. G. Storey, and J. W. Loram, Phys. Rev. B 83, 092502 (2011).
  * (44) Y. Ando, S. Komiya, K. Segawa, S. Ono, and Y. Kurita, Phys. Rev. Lett. 93, 267001 (2004).
  * (45) O. C.-Choiniére, R. Daou, F. Laliberté, C. Collignon, S. Badoux, D. LeBoeuf, J. Chang, B. J. Ramshaw, D. A. Bonn, W. N. Hardy, R. Liang, J.-Q. Yan, J.-G. Cheng, J.-S. Zhou, J. B. Goodenough, S. Pyon, T. Takayama, H. Takagi, N. D. Leyraud, and L. Taillefer, Phys. Rev. B 97, 064502 (2018).
  * (46) W. Zhang, C. L. Smallwood, C. Jozwiak, T. L. Miller, Y. Yoshida, H. Eisaki, D.-H. Lee, and A. Lanzara, Phys. Rev. B 88, 245132 (2013).
  * (47) L. Perfetti, B. Sciolla, G. Biroli, C. J. Van der Beek, C. Piovera, M. Wolf, and T. Kampfrath, Phys. Rev. Lett. 114, 067003 (2015).
  * (48) T. Kondo, Y. Hamaya, A. D. Palczewski, T. Takeuchi, J. S. Wen, Z. J. Xu, G. Gu, J. Schmalian, and A. Kaminski, Nat. Phys. 7, 21 (2011).
  * (49) T. Kondo, A. D. Palczewski, Y. Hamaya, T. Takeuchi, J. S. Wen, Z. J. Xu, G. D. Gu, and A. Kaminski, Phys. Rev. Lett. 111, 157003 (2013).
  * (50) T. Kondo, W. Malaeb, Y. Ishida, T. Sasagawa, H. Sakamoto, T. Takeuchi, T. Tohyama, and S. Shin, Nat. Commun. 6, 7699 (2015).
  * (51) S. D. Chen, M. Hashimoto, Y. He, D. Song, K.-J. Xu, J.-F. He, T. P. Devereaux, H. Eisaki, D.-H. Lu, J. Zaanen, and Z. X. Shen, Science 366, 1099 (2019).
  * (52) A. Dubroka, M. Rössle, K. W. Kim, V. K. Malik, D. Munzar, D. N. Basov, A. A. Schafgans, S. J. Moon, C. T. Lin, D. Haug, V. Hinkov, B. Keimer, T. Wolf, J. G. Storey, J. L. Tallon, and C. Bernhard, Phys. Rev. Lett. 106, 047006 (2011).
  * (53) E. Uykur, K. Tanaka, T. Masui, S. Miyasaka, and S. Tajima, Phys. Rev. Lett. 112, 127003 (2014).
  * (54) K. Lee, K. Kamiya, M. Nakajima, S. Miyasaka, and S. Tajima, J. Phys. Soc. Jpn. 86, 023701 (2017).
  * (55) H. Kitano, T. Ohashi, A. Maeda, and I. Tsukada, Phys. Rev. B 73, 092504 (2006).
  * (56) M. S. Grbić, N. Barišić, A. Dulčić, I. Kupčić, Y. Li, X. Zhao, G. Yu, M. Dressel, M. Greven, and M. Požek, Phys. Rev. B 80, 094511 (2009).
  * (57) M. S. Grbić, M. Požek, D. Paar, V. Hinkov, M. Raichle, D. Haug, B. Keimer, N. Barišić, and A. Dulčić, Phys. Rev. B 83, 144508 (2011).
  * (58) J. Corson, R. Mallozzi, J. Orenstein, J. N. Eckstein, and I. Božović, Nature (London) 398, 221 (1999).
  * (59) J. Orenstein, J. Corson, S. Oh, and J. N. Eckstein, Ann. Phys. 15, 596 (2006).
  * (60) L. S. Bilbro, R. V. Aguilar, G. Logvenov, O. Pelleg, I. Božović, and N. P. Armitage, Nat. Phys. 7, 298 (2011).
  * (61) D. Nakamura, Y. Imai, A. Maeda, and I. Tsukada, J. Phys. Soc. Jpn. 81, 044709 (2012).
  * (62) K. Katsumi, N. Tsuji, Y. I. Hamada, R. Matsunaga, J. Schneeloch, R. D. Zhong, G. D. Gu, H. Aoki, Y. Gallais, and R. Shimano, Phys. Rev. Lett. 120, 117001 (2018).
  * (63) H. Chu, M. J. Kim, K. Katsumi, S. Kovalev, R. D. Dawson, L. Schwarz, N. Yoshikawa, G. Kim, D. Putzky, Z. Z. Li, H. Raffy, S. Germanskiy, J. C. Deinert, N. Awari, I. Ilyakov, B. Green, M. Chen, M. Bawatna, G. Christiani, G. Logvenov, Y. Gallais, A. V. Boris, B. Keimer, A. Schnyder, D. Manske, M. Gensch, Z. Wang, R. Shimano, and S. Kaiser, Nat. Commun. 11, 1793 (2020).
  * (64) K. Katsumi, Z. Z. Li, H. Raffy, Y. Gallais, and R. Shimano, Phys. Rev. B 102, 054510 (2020).
  * (65) J. M. Wade, J. W. Loram, K. A. Mirza, J. R. Cooper, and J. L. Tallon, J. Supercond. 7, 261 (1994).
  * (66) F. Mahmood, X. He, I. Božović, and N. P. Armitage, Phys. Rev. Lett. 122, 027003 (2019).
  * (67) In the original phenomenological two-fluid model which was first proposed by Tisza and LondonTisza  in superconductors and then developed by LandauLandau  in Bosonic liquid helium, the superfluid and normal fluid in superconductors describe the excitations in superconducting and normal statesBa0 , respectively. From our recent study in microscopic viewGIKE1 , we followed the idea of Fulde-Ferrell-Larkin-Ovchinnikov stateFF1 ; FF2  and then divided the BCS spherical shell in momentum space into the pairing and unpairing regions with finite and vanishing anomalous correlation, respectively. It was further shown that the particles in unpairing region behave like normal ones, whereas the dynamics in the pairing region consists of the contributions from Meissner density and Bogoliubov quasiparticlesGIKE1 . We therefore considered the excitations in the pairing and unpairing regions as superfluid and normal fluid, and then, recovered the optical conductivity of the two-fluid model at low temperatureGIKE1 .
  * (68) I. Ussishkin, S. L. Sondhi, and D. A. Huse, Phys. Rev. Lett. 89, 287001 (2002).
  * (69) M. N. Serbyn, M. A. Skvortsov, A. A. Varlamov, and V. Galitski, Phys. Rev. Lett. 102, 067001 (2009).
  * (70) A. I. Larkin and A. A. Varlamov, Theory of Fluctuations in Superconductors, (Oxford University Press, New York, 2002).
  * (71) P. Curty and H. Beck, Phys. Rev. Lett. 91, 257002 (2003).
  * (72) Z. X. Li, S. A. Kivelson, and D. H. Lee, npj Quantum Mater. 6, 36 (2021).
  * (73) F. Yang and M. W. Wu, Phys. Rev. B 98, 094507 (2018).
  * (74) F. Yang and M. W. Wu, Phys. Rev. B 100, 104513 (2019).
  * (75) F. Yang and M. W. Wu, Phys. Rev. B 102, 144508 (2020).
  * (76) F. Yang and M. W. Wu, Phys. Rev. B 102, 014511 (2020).
  * (77) A. A. Abrikosov, L. P. Gor’kov, and I. E. Dzyaloshinski, Methods of Quantum Field Theory in Statistical Physics (Prentice Hall, Englewood Cliffs, 1963).
  * (78) I. Khavkine, H. Y. Kee, and K. Maki, Phys. Rev. B 70, 184521 (2004).
  * (79) G. Tkachov and V. I. Fal’ko, Phys. Rev. B 69, 092503 (2004).
  * (80) F. Rohlfing, G. Tkachov, F. Otto, K. Richter, D. Weiss, G. Borghs, and C. Strunk, Phys. Rev. B 80, 220507(R) (2009).
  * (81) B. D. Josephson, Rev. Mod. Phys. 46, 251 (1974).
  * (82) P. Fulde and R. A. Ferrell, Phys. Rev. 135, 550 (1964).
  * (83) A. I. Larkin and Y. N. Ovchinnikov, Zh. Eksp. Teor. Fiz. 47, 1136 (1964) [Sov. Phys. JETP 20, 762 (1965)].
  * (84) T. Yu and M. W. Wu, Phys. Rev. B 94, 205305 (2016).
  * (85) F. Yang and M. W. Wu, Phys. Rev. B 95, 075304 (2017).
  * (86) F. Yang and M. W. Wu, J. Low Temp. Phys. 192, 241 (2018).
  * (87) The translational symmetry leads to $g_{\bf kk^{\prime}}=g_{\bf k-k^{\prime}}$, whereas $g_{\bf k-k^{\prime}}=g_{\bf k^{\prime}-k}$ by time-reversal symmetry.
  * (88) J. Wu, A. T. Bollinger, X. He, and I. Božović, Nature 547, 432 (2017).
  * (89) Y. Dagan, R. Krupke, and G. Deutscher, Phys. Rev. B 62, 146 (2000).
  * (90) S. E. Sebastian, N. Harrison, M. M. Altarawneh, C. H. Mielke, R. Liang, D. A. Bonn, W. N. Hardy, and G. G. Lonzarich, Proc. Natl. Acad. Sci. USA 107, 6175 (2010).
  * (91) B. J. Ramshaw, S. E. Sebastian, R. D. McDonald, J. Day, B. S. Tan, Z. Zhu, J. B. Betts, R. Liang, D. A. Bonn, W. N. Hardy, and N. Harrison, Science 348, 317 (2015).
  * (92) The prefactor $1/2$ in Nambu-space BdG Hamiltonian [Eq. (1)] in Refs. GIKE1,  and GIKE2,  should be removed.
  * (93) H. Haug and A. P. Jauho, Quantum Kinetics in Transport and Optics of Semiconductors (Springer, Berlin, 1996).
  * (94) M. W. Wu, J. H. Jiang, and M. Q. Weng, Phys. Rep. 493, 61 (2010).
  * (95) S. Tsuchiya, D. Yamamoto, R. Yoshii, and M. Nitta, Phys. Rev. B 98, 094503 (2018).
  * (96) M. E. Peskin and D. V. Schroeder, An Introduction to Quantum Field Theory (Addison-Wesley, New York, 1995).
  * (97) I. J. R. Aitchison, P. Ao, D. J. Thouless, and X. M. Zhu, Phys. Rev. B 51, 6531 (1995).
  * (98) I. J. R. Aitchison, G. Metikas, and D. J. Lee, Phys. Rev. B 62, 6638 (2000).
  * (99) S. Y. Qin, J. Kim, Q. Niu, and C. K. Shih, Science 324, 1314 (2009).
  * (100) T. Zhang, P. Cheng, W. J. Li, Y. J. Sun, G. Wang, X. G. Zhu, K. He, L. L. Wang, X. C. Ma, X. Chen, Y. Y. Wang, Y. Liu, H. Q. Lin, J. F. Jia, and Q. K. Xue, Nat. Phys. 6, 104 (2010).
  * (101) M. H. Liao, Y. Y. Zang, Z. Y. Guan, H. W. Li, Y. Gong, K. J. Zhu, X. P. Hu, D. Zhang, Y. Xu, Y. Y. Wang, K. He, X. C. Ma, S. C. Zhang, and Q. K. Xue, Nat. Phys. 14, 344 (2018).
  * (102) H. Nam, H. Chen, P. W. Adams, S. Y. Guan, T. M. Chuang, C. S. Chang, A. H. MacDonald, and C. K. Shih, Nat. Commun. 9, 5431 (2018).
  * (103) M. Liu, H. Nam, J. Kim, G. A. Fiete, and C. K. Shih, arXiv:2008.12733.
  * (104) M. Mondal, A. Kamlapure, M. Chand, G. Saraswat, S. Kumar, J. Jesudasan, L. Benfatto, V. Tripathi, and P. Raychaudhuri, Phys. Rev. Lett. 106, 047001 (2011).
  * (105) M. Chand, G. Saraswat, A. Kamlapure, M. Mondal, S. Kumar, J. Jesudasan, V. Bagwe, L. Benfatto, V. Tripathi, and P. Raychaudhuri, Phys. Rev. B 85, 014508 (2012).
  * (106) M. Mondal, A. Kamlapure, S. C. Ganguli, J. Jesudasan, V. Bagwe, L. Benfatto, and P. Raychaudhuri, Sci. Rep. 3, 1357 (2013).
  * (107) T. Dubouchet, B. Sacépé, J. Seidemann, D. Shahar, M. Sanquer, and C. Chapelier, Nat. Phys. 15, 233 (2019).
  * (108) J. Chang, E. Blackburn, A. T. Holmes, N. B. Christensen, J. Larsen, J. Mesot, R. X. Liang, D. A. Bonn, W. N. Hardy, A. Watenphul, M. v. Zimmermann, E. M. Forgan, and S. M. Hayden, Nat. Phys. 8, 871 (2012).
  * (109) R. Comin, A. Frano, M. M. Yee, Y. Yoshida, H. Eisaki, E. Schierle, E. Weschke, R. Sutarto, F. He, A. Soumyanarayanan, Y. He, M. L. Tacon, I. S. Elfimov, J. E. Hoffman, G. A. Sawatzky, B. Keimer, and A. Damascelli, Sci. Rep. 343, 390 (2014).
  * (110) X. M. Chen, C. Mazzoli, Y. Cao, V. Thampy, A. M. Barbour, W. Hu, M. Lu, T. A. Assefa, H. Miao, G. Fabbris, G. D. Gu, J. M. Tranquada, M. P. M. Dean, S. B. Wilkins, and I. K. Robinson, Nat. Commun. 10, 1435 (2019).
  * (111) H. Miao, G. Fabbris, R. J. Koch, D. G. Mazzone, C. S. Nelson, R. A. Esteves, G. D. Gu, Y. Li, T. Yilimaz, K. Kaznatcheev, E. Vescovo, M. Oda, T. Kurosawa, N. Momono, T. Assefa, I. K. Robinson, E. S. Bozin, J. M. Tranquada, P. D. Johnson, and M. P. M. Dean, npj Quantum Mater. 6, 31 (2021).
  * (112) J. M. Tranquada, H. Woo, T. G. Perring, H. Goka, G. D. Gu, G. Xu, M. Fujita, and K. Yamada, Nature 429, 534 (2004).
  * (113) S. M. Hayden, H. A. Mook, P. C. Dai, T. G. Perring, and F. Doǧan, Nature 429, 531 (2004).
  * (114) B. Khaykovich, S. Wakimoto, R. J. Birgeneau, M. A. Kastner, Y. S. Lee, P. Smeibidl, P. Vorderwisch, and K. Yamada, Phys. Rev. B 71, 220508 (2005).
  * (115) C. V. Parker, P. Aynajian, E. H. D. S. Neto, A. Pushp, S. Ono, J. S. Wen, Z. J. Xu, G. Gu, and A. Yazdani, Nature 468, 677 (2010).
  * (116) M. J. Lawler, K. Fujita, J. Lee, A. R. Schmidt, Y. Kohsaka, C. K. Kim, H. Eisaki, S. Uchida, J. C. Davis, J. P. Sethna, and E.-A. Kim, Nature 466, 347 (2010).
  * (117) E. H. D. S. Neto, P. Aynajian, R. E. Baumbach, E. D. Bauer, J. Mydosh, S. Ono, and A. Yazdani, Phys. Rev. B 87, 161117 (2013).
  * (118) Z. Y. Weng and V. N. Muthukumar, Phys. Rev. B 66, 094509 (2002).
  * (119) J.-H. Zhang, S. Li, Y. Ma, Y. G. Zhong, H. Ding, and Z. Y. Weng, Phys. Rev. Res. 2, 023398 (2020).
  * (120) J. Y. Zhao, S. A. Chen, H. K. Zhang, Z. Y. Weng, arXiv:2106.14898.
  * (121) Y. He, S. D. Chen, Z. X. Li, D. Zhao, D. Song, Y. Yoshida, H. Eisaki, T. Wu, X. H. Chen, D. H. Lu, C. Meingast, R. J. Birgeneau, T. P. Devereaux, M. Hashimoto, D. H. Lee, and Z. X. Shen, arXiv:2009.10932.
  * (122) N. R. Lee-Hone, J. S. Dodge, and D. M. Broun, Phys. Rev. B 96, 024501 (2017).
  * (123) N. R. Lee-Hone, H. U. Özdemir, V. Mishra, D. M. Broun, and P. J. Hirschfeld, Phys. Rev. Res. 2, 013228 (2020).
  * (124) P. W. Anderson, J. Phys. Chem. Solids 11, 26 (1959).
  * (125) H. Suhl and B. T. Matthias, Phys. Rev. 114, 977 (1959).
  * (126) S. Skalski, O. B. Matibet, and P. R. Weiss, Phys. Rev. 136, A1500 (1964).
  * (127) L. Tisza, C. R. Acad. Sci. 207, 1035 (1938); 207, 1186 (1938); F. London, Phys. Rev. 54, 947 (1938).
  * (128) L. D. Landau, Zh. Eksp. Teor. Fiz. 11, 592 (1941); J. Phys. USSR 5, 71 (1941).